Video models can generate impressive motion โ but can they actually reason?
๐ข Introducing VideoRLVR: a systematic RL recipe that turns video models into visual reasoners using verifiable rewards.
๐งฉ Tasks: Maze, FlowFree, Sokoban
โก Early-Step Focus: ~40% faster RL training
๐ Better in-domain + OOD reasoning than SFT and strong video baselines
๐ฐ Paper: https://t.co/CkCqFymR4l
๐ฅ๏ธ Project: https://t.co/Wq2e59Q7gC
๐ฉ Instead of asking: โDoes this video look realistic?โ
VideoRLVR asks: โDoes this video form a valid visual reasoning trajectory?โ
๐ก Our starting point is a simple analogy:
- LLMs learn to reason better by moving beyond imitation: Pre-training โ SFT โ RLVR
- But video reasoning is stuck at: Pre-training โ SFT
๐ง So what would the RLVR stage look like for video models?
A video is not just pixels changing over time. For reasoning tasks, a video is a chain of visual states. If each state represents progress toward a goal, then the full video can be checked, scored, and optimized. That turns video generation into something closer to visual problem solving.
โ๏ธ Our recipe
1โฃ Verifiable reasoning tasks
Include 30,000 video samples on Maze, FlowFree, and Sokoban, ranging over various levels of difficulty.
2โฃ SDE-GRPO for video diffusion with Early-Step Focus
Only inject SDE noise and optimize at early denoise steps. They are the key to the global plan and let the latter steps do the rendering refinement they are already good at.
3โฃ Verifiable visual rewards
Define rewards over the generated trajectory, optimizing for valid transitions, constraint satisfaction, and goal completion.
๐ฏ What we achieve
- In-domain improvement over proprietary and open-source models, including strong video reasoning SFT baselines.
- OOD performance matches 14B models with a 5B model.
โฐ The broader message
Future video models should not only answer: โCan I synthesize this motion?โ but also: โCan I generate a valid process that solves this problem?โ
VideoRLVR is a step toward video models as visual reasoners, not just visual imitators.
Thanks to all the collaborators: @sheng_zh@JamesYHuang36@Selena3187@Xiaofei_Wen_Mk @ Yuankai Li @hoifungpoon@muhao_chen
#VideoGeneration #VideoReasoning #AI
LLMs can't see.
How can we build effective multi-agent systems with vision capabilities?
Building multimodal models from scratch is expensive. Training joint vision-language architectures requires massive compute, specialized datasets, and careful optimization.
But there's another way.
This new research introduces "Be My Eyes," a framework where vision models become literal eyes for LLMs.
The key idea: multi-agent collaboration through natural language.
Vision agents analyze images and describe what they see. Language agents receive these descriptions and reason about them. Communication happens entirely through text.
No joint training. No architectural modifications. Just agents talking to each other.
The system is modular. Swap in better vision models as they emerge. Upgrade the LLM independently. Each component improves without retraining the whole system.
Results on MMMU, MMMU-Pro, and video understanding benchmarks show competitive performance with specialized multimodal models.
What makes this powerful: it challenges the assumption that multimodal AI requires unified architectures. Agent collaboration through language provides an efficient alternative.
Paper: https://t.co/VqnwooAGkJ
Learn to build with AI Agents in our academy: https://t.co/Y5kVy5iKiQ
๐What if DeepSeek-R1 can see?
๐Introducing Be My Eyes, a multi-agent framework for extending LLMs to new modalities without modifying their parameters.
โ With a 7B/8B VLM as eyes, DeepSeek-R1 can beat GPT-4o across multimodal benchmarks.
๐https://t.co/k9zMsJVPvL
๐งต[1/n]
Be My Eyes unlocks LLMs' capabilities on visual inputs, leading to substantial performance gain across multimodal reasoning benchmarks. Notably, with a 7B/8B VLM as eyes, DeepSeek-R1 can consistently outperform large-scale VLMs like GPT-4o.
๐งต[4/n]
๐ Excited to share that our paper, "MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding", will be presented at #ICLR2025!โ
๐ Date: April 24
๐ Time: 3:00 PM
๐ Location: Hall 3 + Hall 2B #11
MuirBench challenges multimodal LLMs with diverse multi-image tasks, highlighting the need for models that can reason beyond single images.
๐ Learn more: https://t.co/AGL17b4HEC
๐ผ๏ธ Poster: https://t.co/KXK7bzymMp
#MuirBench #MultimodalAI #ICLR2025
๐ Excited to share MetaScale, our latest work advancing LLM reasoning capabilities! MetaScale empowers GPT-4o to match or even surpass frontier reasoning models like o1, Claude-3.5-Sonnet, and o1-mini on the challenging Arena-Hard benchmark (@lmarena_ai). Additionally, MetaScale significantly boosts GPT-4o and Llama-3.1 performance on MMLU-Pro, scaling impressively with increased computational budgets.
The idea of MetaScale is to introduce an adaptive test-time scaling framework built on the concept of meta-thoughts โ dynamic cognitive strategies tailored specifically to each task. MetaScale begins by establishing a diverse pool of candidate meta-thoughts, then iteratively selects and evaluates these candidates through a multi-armed bandit algorithm utilizing upper confidence bound (UCB) selection, guided by a dedicated reward model. Additionally, a genetic algorithm continually evolves high-performing meta-thoughts, enhancing adaptability by refining and expanding the available strategy pool over time. This dynamic inference-time optimization allows MetaScale to significantly improve both accuracy and generalization across diverse tasks.
Explore adaptive thinking transforms LLM reasoning: https://t.co/gC5e08Hbjm
Great work by @QinLiu_NLP, @WenxuanZhou_96, @xunannancy, @JamesYHuang36, @fwang_nlp, @hoifungpoon, @muhao_chen
๐จ DPO works well on LLMs, but what happens when we apply it to VLMs/MLLMs?
๐ Excited to introduce our #EMNLP2024 work on DPO for Multimodal LLMs! We identify the unconditional preference issues (where the model ๐จ๐ฏ๐๐ซ๐ฅ๐จ๐จ๐ค๐ฌ ๐ฏ๐ข๐ฌ๐ฎ๐๐ฅ ๐๐จ๐ง๐ญ๐๐ฑ๐ญ), leading to severe hallucinations, and address it with แดDPO in collaboration with @MSFTResearch.
๐ ๐๐ก๐ ๐๐ข๐ญ๐๐๐ฅ๐ฅ: ๐๐ง๐๐จ๐ง๐๐ข๐ญ๐ข๐จ๐ง๐๐ฅ ๐๐ซ๐๐๐๐ซ๐๐ง๐๐
Through controlled experiments, we find that MLLMs can perform similarly even if all images are removed from DPO training. While DPO aims to compute implicit rewards conditioned on all input modalities, it may prioritize language-only preferences and overlook the image condition, leading to hallucinations.
๐ง ๐๐ฎ๐ซ ๐๐จ๐ฅ๐ฎ๐ญ๐ข๐จ๐ง: แด๐๐๐
We propose แดDPO, a multimodal DPO objective that jointly optimizes text and image preferences, with absolute reward anchors to address problems caused by relative preferences.
๐ก ๐๐๐ฒ ๐๐๐ค๐๐๐ฐ๐๐ฒ๐ฌ
โข Unconditional preference is common and harmful in DPO for multimodal scenarios, where the model often overlooks visual signals.
โข It is essential to jointly optimize preferences across different modalities.
โข แดDPO effectively addresses the unconditional preference issue and significantly improves model performance, particularly in reducing hallucinations.
Check out our paper, code, and more:
๐ Homepage: https://t.co/EGzSE6Mdbt
๐ Paper: https://t.co/f1AIlp5gBV
๐ป Code: https://t.co/82LlhoMaS3
#AI #MachineLearning #VLM #FoundationModels #MultimodalAI #AIAlignment #Hallucination
With the fast-growing interest in multi-image understanding, where can we find a high-quality, comprehensive & robust benchmark? A group of talented students have created แดแดษชสสแดษดแดส, a long-awaited multi-image benchmark, with 2.6k multi-choice Qs and over 10k images. (1/6)
Can Text-to-Image models understand common sense? ๐ค
Can they generate images that fit everyday common sense? ๐ค
tldr; NO, they are far less intelligent than us ๐๐ปโโ๏ธ
Introducing Commonsense-T2I ๐ก https://t.co/CTcmCbGUhX, a novel evaluation and benchmark designed to measure commonsense reasoning in T2I models ๐ฅ๐ฅ
Paper: https://t.co/Csu2Q0453s
(1/n)
๐คจ Frustrated when LLMs can give different or even wrong answers because you phrased your instruction differently?
๐ Introducing Contrastive Instruction Tuning (CoIN) - our solution to make models less sensitive to the exact phrasing of instructions! (https://t.co/hnUQ4vUQts)
DeAL
Decoding-time Alignment for Large Language Models
paper page: https://t.co/6ena1OiOcI
Large Language Models (LLMs) are nowadays expected to generate content aligned with human preferences. Current work focuses on alignment at model training time, through techniques such as Reinforcement Learning with Human Feedback (RLHF). However, it is unclear if such methods are an effective choice to teach alignment objectives to the model. First, the inability to incorporate multiple, custom rewards and reliance on a model developer's view of universal and static principles are key limitations. Second, the residual gaps in model training and the reliability of such approaches are also questionable (e.g. susceptibility to jail-breaking even after safety training). To address these, we propose DeAL, a framework that allows the user to customize reward functions and enables Decoding-time Alignment of LLMs (DeAL). At its core, we view decoding as a heuristic-guided search process and facilitate the use of a wide variety of alignment objectives. Our experiments with programmatic constraints such as keyword and length constraints (studied widely in the pre-LLM era) and abstract objectives such as harmlessness and helpfulness (proposed in the post-LLM era) show that we can DeAL with fine-grained trade-offs, improve adherence to alignment objectives, and address residual gaps in LLMs. Lastly, while DeAL can be effectively paired with RLHF and prompting techniques, its generality makes decoding slower, an optimization we leave for future work.
My awesome student @JamesYHuang36 just received an outstanding paper award at #EMNLP2023! He is looking for summer research intern. Please interview him.