Kevin Lin

Senior Researcher @Microsoft; PhD @UW; Computer Vision, Machine Learning

[Kevin Lin @linkeyun2]

MAI-Code-1-Flash in VSCode⚡️
Very proud of what the team has built!
Checkout https://t.co/fHIBG3Xb3p

[Weizhu Chen @WeizhuChen]

We just released MAI-Code-1-Flash, a 5B-active small model. The team just finished this from a pretrained model to VSC and Copilot CLI production within a few weeks, so proud of their achievements. More details are here https://t.co/jx85ssYAea and a bigger model is on the way.

[Cheng Han Chiang (姜成翰) @dcml0714]

1/7 🔗 Introducing STITCH: our new method to make Spoken Language Models (SLMs) think and talk at the same time.
Paper link 👉 https://t.co/LjmlAf0AEr

[Kevin Lin @linkeyun2]

Happy to share our recent work on GenXD!

[Yuyang Zhao @yuyangzhao_]

🚀🚀Excited to introduce GenXD: Generating Any 3D and 4D Scenes! A joint framework for general 3D and 4D generation, supporting both object-level and scene-level generation.

Project Page: https://t.co/ndDPY448MG
Arxiv: https://t.co/Ci7Y54Duoq

[Yan @AnYan_ai]

I am attending #COLM2024 in Philly!

Will present our paper “List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMs” on Monday morning ⏰

Come and chat if you are interested in multimodal LLMs, synthetic data and training recipes!

[OpenAI @OpenAI]

We’re rolling out the ability for ChatGPT Free users to create up to two images per day with DALL·E 3.

Just ask ChatGPT to create an image for a slide deck, personalize a card for a friend, or show you what something looks like.

[OpenAI @OpenAI]

You can now edit DALL·E images in ChatGPT across web, iOS, and Android.

[Kevin Lin @linkeyun2]

Our work was accepted at ICLR 2024!

Paper and reviews: https://t.co/qRTt2RTwvE
Code & data: https://t.co/67Hw2mkBJo

[AK @_akhaliq]

Aligning Large Multi-Modal Model with Robust Instruction Tuning

paper page: https://t.co/nA0MjAb4B9

Despite the promising progress in multi-modal tasks, current large multi-modal models (LMM) are prone to hallucinating inconsistent descriptions with respect to the associated image and human instructions. This paper addresses this issue by introducing the first large and diverse visual instruction tuning dataset, named Large-scale Robust Visual (LRV)-Instruction. Our dataset consists of 120k visual instructions generated by GPT4, covering 16 vision-and-language tasks with open-ended instructions and answers. Unlike existing studies that primarily focus on positive instruction samples, we design LRV-Instruction to include both positive and negative instructions for more robust visual instruction tuning. Our negative instructions are designed at two semantic levels: (i) Nonexistent Element Manipulation and (ii) Existent Element Manipulation. To efficiently measure the hallucination generated by LMMs, we propose GPT4-Assisted Visual Instruction Evaluation (GAVIE), a novel approach to evaluate visual instruction tuning without the need for human-annotated groundtruth answers and can adapt to diverse instruction formats. We conduct comprehensive experiments to investigate the hallucination of LMMs. Our results demonstrate that existing LMMs exhibit significant hallucination when presented with our negative instructions, particularly with Existent Element Manipulation instructions. Moreover, by finetuning MiniGPT4 on LRV-Instruction, we successfully mitigate hallucination while improving performance on public datasets using less training data compared to state-of-the-art methods. Additionally, we observed that a balanced ratio of positive and negative instances in the training data leads to a more robust model.

[Kevin Lin @linkeyun2]

Thanks for sharing! @_akhaliq
📲We present MM-Navigator, an agent system built on GPT-4V for smartphone GUI navigation.
📲MM-Navigator incorporates action histories and set-of-mark tags to produce precise executable actions.
Project page: https://t.co/evZrLQx0Ge

[AK @_akhaliq]

GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation

paper page: https://t.co/INFoyShhBM

present MM-Navigator, a GPT-4V-based agent for the smartphone graphical user interface (GUI) navigation task. MM-Navigator can interact with a smartphone screen as human users, and determine subsequent actions to fulfill given instructions. Our findings demonstrate that large multimodal models (LMMs), specifically GPT-4V, excel in zero-shot GUI navigation through its advanced screen interpretation, action reasoning, and precise action localization capabilities. We first benchmark MM-Navigator on our collected iOS screen dataset. According to human assessments, the system exhibited a 91\% accuracy rate in generating reasonable action descriptions and a 75\% accuracy rate in executing the correct actions for single-step instructions on iOS. Additionally, we evaluate the model on a subset of an Android screen navigation dataset, where the model outperforms previous GUI navigators in a zero-shot fashion. Our benchmark and detailed analyses aim to lay a robust groundwork for future research into the GUI navigation task.

[Kevin Lin @linkeyun2]

@YamanKSingla @_akhaliq Great work! Thank you for sharing @YamanKSingla

[Kevin Lin @linkeyun2]

Thanks for featuring our work! @_akhaliq
We explore GPT-4V for many interesting video tasks

Check out our recent works:
📢 MM-VID: https://t.co/lT9JK4il9g
📢 DEsignBench: https://t.co/9vDbifvcuh
📢 Idea2Img: https://t.co/FUfbNTxUmK
📢 The Dawn of LMMs: https://t.co/whRh4vrNj0

[AK @_akhaliq]

MM-VID: Advancing Video Understanding with GPT-4V(ision)

paper page: https://t.co/6CDr4Lygzw

present MM-VID, an integrated system that harnesses the capabilities of GPT-4V, combined with specialized tools in vision, audio, and speech, to facilitate advanced video understanding. MM-VID is designed to address the challenges posed by long-form videos and intricate tasks such as reasoning within hour-long content and grasping storylines spanning multiple episodes. MM-VID uses a video-to-script generation with GPT-4V to transcribe multimodal elements into a long textual script. The generated script details character movements, actions, expressions, and dialogues, paving the way for large language models (LLMs) to achieve video understanding. This enables advanced capabilities, including audio description, character identification, and multimodal high-level comprehension. Experimental results demonstrate the effectiveness of MM-VID in handling distinct video genres with various video lengths. Additionally, we showcase its potential when applied to interactive environments, such as video games and graphic user interfaces.

[AK @_akhaliq]

MM-VID: Advancing Video Understanding with GPT-4V(ision)

paper page: https://t.co/6CDr4Lygzw

present MM-VID, an integrated system that harnesses the capabilities of GPT-4V, combined with specialized tools in vision, audio, and speech, to facilitate advanced video understanding. MM-VID is designed to address the challenges posed by long-form videos and intricate tasks such as reasoning within hour-long content and grasping storylines spanning multiple episodes. MM-VID uses a video-to-script generation with GPT-4V to transcribe multimodal elements into a long textual script. The generated script details character movements, actions, expressions, and dialogues, paving the way for large language models (LLMs) to achieve video understanding. This enables advanced capabilities, including audio description, character identification, and multimodal high-level comprehension. Experimental results demonstrate the effectiveness of MM-VID in handling distinct video genres with various video lengths. Additionally, we showcase its potential when applied to interactive environments, such as video games and graphic user interfaces.

[AK @_akhaliq]

Idea2Img: Iterative Self-Refinement with GPT-4V(ision) for Automatic Image Design and Generation

paper page: https://t.co/LdZv9APEvJ

introduce ``Idea to Image,'' a system that enables multimodal iterative self-refinement with GPT-4V(ision) for automatic image design and generation. Humans can quickly identify the characteristics of different text-to-image (T2I) models via iterative explorations. This enables them to efficiently convert their high-level generation ideas into effective T2I prompts that can produce good images. We investigate if systems based on large multimodal models (LMMs) can develop analogous multimodal self-refinement abilities that enable exploring unknown models or environments via self-refining tries. Idea2Img cyclically generates revised T2I prompts to synthesize draft images, and provides directional feedback for prompt revision, both conditioned on its memory of the probed T2I model's characteristics. The iterative self-refinement brings Idea2Img various advantages over vanilla T2I models. Notably, Idea2Img can process input ideas with interleaved image-text sequences, follow ideas with design instructions, and generate images of better semantic and visual qualities. The user preference study validates the efficacy of multimodal iterative self-refinement on automatic image design and generation.

[Tanishq Mathew Abraham, Ph.D. @iScienceLuvr]

The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)

Link: https://t.co/m8u4wSGGzN

A 166-page report from Microsoft qualitatively exploring GPT-4V capabilities and usage. Describes visual+text prompting techniques, few-shot learning, reasoning, etc.

Looks like it will be a must-read for GPT-4V power users 👀

[Yi Lin Sung @yilin_sung]

🚨 Multimodal Model Merging is accepted to #EMNLP2023 findings!

Check out our camera-ready version with exps on more tasks & architectures, and a new metric to better predict whether two models are mergeable --> https://t.co/gRiRQC1HcG

@linjiefun @linkeyun2 @zhegan4 @mohitban47

[AK @_akhaliq]

MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities

paper page: https://t.co/oaXdDJKWgG

propose MM-Vet, an evaluation benchmark that examines large multimodal models (LMMs) on complicated multimodal tasks. Recent LMMs have shown various intriguing abilities, such as solving math problems written on the blackboard, reasoning about events and celebrities in news images, and explaining visual jokes. Rapid model advancements pose challenges to evaluation benchmark development. Problems include: (1) How to systematically structure and evaluate the complicated multimodal tasks; (2) How to design evaluation metrics that work well across question and answer types; and (3) How to give model insights beyond a simple performance ranking. To this end, we present MM-Vet, designed based on the insight that the intriguing ability to solve complicated tasks is often achieved by a generalist model being able to integrate different core vision-language (VL) capabilities. MM-Vet defines 6 core VL capabilities and examines the 16 integrations of interest derived from the capability combination. For evaluation metrics, we propose an LLM-based evaluator for open-ended outputs. The evaluator enables the evaluation across different question types and answer styles, resulting in a unified scoring metric. We evaluate representative LMMs on MM-Vet, providing insights into the capabilities of different LMM system paradigms and models.

[Tan Wang @Wangt97]

Excited to share our EqBen/EqSim, a new benchmark/algorithm focusing on evaluating and improving the similarity measure of V&L foundation models, to be presented in Oral session at #ICCV2023 in Paris!
Joint work w/ @linkeyun2 @LINJIEFUN CC Lin, ZY Yang, HW Zhang, ZC Liu, LJ Wang.

[Tan Wang @Wangt97]

I will be in @icvss for the upcoming week presenting our DisCo (https://t.co/0LdY3y11qy), with interactive demo to turn static images into human dancing videos! Big thanks to @GMFarinella , @robertocipolla , @sebattiato for organizing ICVSS.

[Linjie (Lindsey) Li @LINJIEFUN]

We are hiring full-time/part-time research interns all year round. If you are interested, please send your resume to [email protected].

[Kevin Lin @linkeyun2]

Thanks @_akhaliq for sharing our work! Please find more details below.
Github Page: https://t.co/BiGYlnS0qP
Arxiv: https://t.co/gUNBAwxjdO
Huggingface: https://t.co/jHJvGnii7S
Youtube: https://t.co/9qZIKcfGTP

[AK @_akhaliq]

DisCo: Disentangled Control for Referring Human Dance Generation in Real World

paper page: https://t.co/0YsK8v2tc5

Generative AI has made significant strides in computer vision, particularly in image/video synthesis conditioned on text descriptions. Despite the advancements, it remains challenging especially in the generation of human-centric content such as dance synthesis. Existing dance synthesis methods struggle with the gap between synthesized content and real-world dance scenarios. In this paper, we define a new problem setting: Referring Human Dance Generation, which focuses on real-world dance scenarios with three important properties: (i) Faithfulness: the synthesis should retain the appearance of both human subject foreground and background from the reference image, and precisely follow the target pose; (ii) Generalizability: the model should generalize to unseen human subjects, backgrounds, and poses; (iii) Compositionality: it should allow for composition of seen/unseen subjects, backgrounds, and poses from different sources. To address these challenges, we introduce a novel approach, DISCO, which includes a novel model architecture with disentangled control to improve the faithfulness and compositionality of dance synthesis, and an effective human attribute pre-training for better generalizability to unseen humans. Extensive qualitative and quantitative results demonstrate that DISCO can generate high-quality human dance images and videos with diverse appearances and flexible motions.

[Tan Wang @Wangt97]

Thx @_akhaliq! Check out our DisCo at https://t.co/OsBieyXZ0h.🔥🔥🔥

🧙‍♂️High Generalizability. No need human-specific fine-tuning!

💃Extensive human-related applications with disentangled control!

👨‍💻Easy-to-follow framework and totally opensource code!