Kai Zhang

Ph.D. student @ Lehigh, focusing on solving medical problems with AI techniques.

[Kai Zhang @kai_z99]

EditGRPO: https://t.co/7lxgolqPRC

RadAgents: https://t.co/iHYjdIfzp7

[Kai Zhang @kai_z99]

Excited to share our two new works on chest X-ray image interpretation!

1️⃣ EditGRPO: first work applying GRPO on medical report generation.

2️⃣ RadAgents: A multi-agent framework that mimics radiologist workflows.

Papers and codebase are detailed in comments!

[Kai Zhang @kai_z99]

Excited to share that our work has been accepted! A big thank you to all my co-authors!

Preprint version: https://t.co/7216zJczrH

Check out the GitHub: https://t.co/kkHp1DYAUx. Hugging Face-compatible code is currently under development and will be released soon!

[Nature Medicine @NatureMedicine]

An open source and computing-friendly vision-language model is shown to achieve state-of-the-art accuracy in 16 out of 25 biomedical tasks, with promising performance in a series of potential clinical applications @slc137 @james_y_zou @hongfangliu

https://t.co/H2dbhRwcuh

[Kai Zhang @kai_z99]

A great work from our Lehigh AI Lab @slc137 -- Mora: More like a Sora, a multi-agent framework designed to facilitate video generation tasks. It aims to replicate and extend the capabilities of OpenAI's Sora, and totally open-sourced!:
Github: https://t.co/wF2uXHCjAG

[Kai Zhang @kai_z99]

Our new paper regrading speculation of techniques used in Sora, the corresponding applications and social impact, insights about how to develop (open-sourced) Sora-like models.

#Lehigh #MSR #Sora

[AK @_akhaliq]

Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models

Sora is a text-to-video generative AI model, released by OpenAI in February 2024. The model is trained to generate videos of realistic or imaginative scenes from text instructions and show potential in simulating the physical world. Based on public technical reports and reverse engineering, this paper presents a comprehensive review of the model's background, related technologies, applications, remaining challenges, and future directions of text-to-video AI models. We first trace Sora's development and investigate the underlying technologies used to build this "world simulator". Then, we describe in detail the applications and potential impact of Sora in multiple industries ranging from film-making and education to marketing. We discuss the main challenges and limitations that need to be addressed to widely deploy Sora, such as ensuring safe and unbiased video generation. Lastly, we discuss the future development of Sora and video generation models in general, and how advancements in the field could enable new ways of human-AI interaction, boosting productivity and creativity of video generation.

[Kai Zhang @kai_z99]

@cwolferesearch @readwise save

[AK @_akhaliq]

Instruction Mining: High-Quality Instruction Data Selection for Large Language Models

paper page: https://t.co/yenjsG5LdE

Large language models typically undergo two training stages, pretraining and finetuning. Despite that large-scale pretraining endows the model with strong capabilities to generate natural language responses, these pretrained models can still fail to understand human instructions at times. To enhance language models' ability of interpreting and responding to instructions, instruction finetuning has emerged as a critical method in this area. Recent studies found that large language models can be finetuned to perform well even with a small amount of high-quality instruction-following data. However, the selection of high-quality datasets for finetuning language models still lacks clear guidelines to follow. In this paper, we propose InstructMining, a linear rule for evaluating instruction-following data quality. We formulate InstructMining using specific natural language indicators. To investigate the relationship between data quality and these indicators, we further conduct extensive finetuning experiments. The experiment results are then applied to estimating parameters in InstructMining. To further investigate its performance, we use InstructMining to select high-quality data from unseen datasets. Results demonstrate that InstructMining can help select relatively high-quality samples from various instruction-following datasets. Compared to models finetuned on unfiltered datasets, models finetuned on InstructMining selected datasets perform better on 42.5% cases.

[elvis @omarsar0]

GPT-4 is awesome, but...

...during these times, I like to remind myself that we are just at the beginning. There is so much to improve.

One approach I have been relying on is implementing ideas from scratch. One of my favorite examples is the nanoGPT repository by @karpathy.

Once you start to get into the details of these models, you start to realize how many areas we can improve. Ranging from better tokenization to evaluation to reliability to explainability.

The unbelievable capabilities of these systems give the illusion that all the work is done and we can all go home. However, you don't see OpenAI and these other LLM companies stopped hiring. I actually believe that with the recent progress we have given ourselves even more challenging work. This is the reason why we see disciplines like prompt engineering and others emerging.

I hear NLP students/practitioners discouraged about the recent progress of LLMs. I think that feeling should be temporary. There is a lot of work to get done. It helps to think what are the high-priority features/capabilities you would like to see that currently don't exist in these AI systems -- because there are so many. It also helps to play around with open-source code and implementations.

I would love to see a nanoChatGPT and nanoMultimodalGPT at some point. Let's keep pushing.

https://t.co/XbKIDuGAQy

[Kai Zhang @kai_z99]

最近每个人都很感兴趣的ChatGPT并��是突然发生的。我们经过1年半的时间细心打磨，大模型预训练综述总算preprint了。综述包括了大模型变化的历史，不同算法和应用场景，提供了视觉，语言，图，多模态，统一模型的深度讨论。希望可以帮助感兴趣的大家更加好的了解大模型的预训练和ChatGPT相关的技术。

[elvis @omarsar0]

A ~100 pages survey on pretrained foundation models: from BERT to ChatGPT.

Has a nice structure for reviewing pretrained models in addition to challenges and open problems.

https://t.co/uXA4NrzV0w