🚨 There are too many models to choose!
Whether you're shortlisting candidates before fine-tuning, picking among 500+ models on OpenRouter, or searching through hundreds of thousands of checkpoints on HuggingFace, the question is the same:
Which model should you actually use for a new task?
💡 Introducing ModelLens — a model recommendation system for the open-source model ecosystem.
Instead of asking users to evaluate or fine-tune every candidate model, ModelLens asks a different question:
👉 Can we leverage large-scale model–dataset interaction patterns from public leaderboards to enable model selection in the wild, without requiring direct evaluation or fine-tuning?
Our key insight:
The millions of (model, dataset, performance) records accumulating on all platforms are not just leaderboard noise. They encode how model capabilities align with task characteristics across the open-source ecosystem.
⚙️ How it works:
We collect large-scale model–dataset performance interactions from public leaderboards (HuggingFace, Open LLM Leaderboard, Papers with Code) and learn transferable model–task compatibility, jointly modeling:
- 47K models across 348 architecture families
- 9.6K datasets across 2,551 task types
- 8,420 heterogeneous evaluation metrics
- 1.62M historical performance records as implicit supervision
Given any task description, ModelLens then recommends a ranked Top-K pool of candidates likely to perform well, generalizing to newly released models and never-benchmarked datasets, before any of them is ever run on the target task.
task description → learned model–task compatibility → Top-K recommended models
📊 What ModelLens enables:
✅ Recommend Top-K models for a new dataset
✅ Rank newly released models against existing models
✅ Build model pools before routing or cascading
✅ Reduce costly brute-force evaluation
✅ Scale model selection in the open-source AI ecosystem
📰 Paper: https://t.co/5SyNfY3OlE
💻 Code: https://t.co/YUepL9ntpv
🌐 Demo: https://t.co/UnMt0zt55q
Try the demo with your own task description — we’d love feedback from people working on model selection, routing, AutoML, and open-source LLM ecosystems.
Thanks to all coauthors: @Wenjie_Jacky_Mo@Xiaofei_Wen_Mk@qiyao_ma@WenhuiZhu8@XiwenChen37@muhao_chen@zpegt
#MachineLearning #OpenSource #AI #ModelSelection @OpenRouter@huggingface
(1/9)Suppose you're training an LLM on web data with limited math data. Should you place all your math data at the end of training, or mix it with web data earlier? The answer flips with the learning rate (LR) schedule!
At fixed compute and mid-training (MT) token budgets, earlier mixing gives lower MT loss when LR decays throughout training. With constant LR, placeing all MT data for the end works better.
We then vary when LR decay begins. The best tested strategy: keep LR high after MT data enters, then decay near the end.
See our new paper: "Data Schedules and Learning Rate Schedules Should Be Designed Together" !
I’m also excited to share that our paper, “Diagnosing and Mitigating Modality Interference in Multimodal Large Language Models,” has been accepted to #NeurIPS2026! 🎉 @NeurIPSConf
Why can an irrelevant caption mislead an MLLM looking at an image—or an unrelated image distort its answer to a text-only question? We call this modality interference: signals from a modality the task doesn’t need can still change the model’s answer.
We use causal perturbations to diagnose modality interference and propose an effective method to mitigate it. Across image, text, and multimodal tasks, this improves robustness and multimodal performance.
📄 Paper: https://t.co/1CXYpWOvfL
#Multimodal #MLLM
🚨 Check out our latest paper!
Diagnosing and Mitigating Modality Interference in MLLMs:
📉 MLLMs struggle on unimodal tasks
🧠 We call this Modality Interference, a symptom of Cross-Modality Competency Problem
🛠️ Causal Analysis and Solution provided
📄 https://t.co/TfX4FbORPt
Excited to share that “ModelLens: Finding the Best for Your Task from Myriads of Models” has been accepted to NeurIPS 2026 @NeurIPSConf#NeurIPS2026 ! 🎉🍻
📰 Paper: https://t.co/VwpijxNuWO
💻 Code: https://t.co/YUepL9ntpv
🌐 Demo: https://t.co/UnMt0zt55q
🗺️ ModelLens-Atlas: https://t.co/dkcTnkNMSq
#MachineLearning @NeurIPSConf@huggingface@OpenRouter
🚨 There are too many models to choose!
Whether you're shortlisting candidates before fine-tuning, picking among 500+ models on OpenRouter, or searching through hundreds of thousands of checkpoints on HuggingFace, the question is the same:
Which model should you actually use for a new task?
💡 Introducing ModelLens — a model recommendation system for the open-source model ecosystem.
Instead of asking users to evaluate or fine-tune every candidate model, ModelLens asks a different question:
👉 Can we leverage large-scale model–dataset interaction patterns from public leaderboards to enable model selection in the wild, without requiring direct evaluation or fine-tuning?
Our key insight:
The millions of (model, dataset, performance) records accumulating on all platforms are not just leaderboard noise. They encode how model capabilities align with task characteristics across the open-source ecosystem.
⚙️ How it works:
We collect large-scale model–dataset performance interactions from public leaderboards (HuggingFace, Open LLM Leaderboard, Papers with Code) and learn transferable model–task compatibility, jointly modeling:
- 47K models across 348 architecture families
- 9.6K datasets across 2,551 task types
- 8,420 heterogeneous evaluation metrics
- 1.62M historical performance records as implicit supervision
Given any task description, ModelLens then recommends a ranked Top-K pool of candidates likely to perform well, generalizing to newly released models and never-benchmarked datasets, before any of them is ever run on the target task.
task description → learned model–task compatibility → Top-K recommended models
📊 What ModelLens enables:
✅ Recommend Top-K models for a new dataset
✅ Rank newly released models against existing models
✅ Build model pools before routing or cascading
✅ Reduce costly brute-force evaluation
✅ Scale model selection in the open-source AI ecosystem
📰 Paper: https://t.co/5SyNfY3OlE
💻 Code: https://t.co/YUepL9ntpv
🌐 Demo: https://t.co/UnMt0zt55q
Try the demo with your own task description — we’d love feedback from people working on model selection, routing, AutoML, and open-source LLM ecosystems.
Thanks to all coauthors: @Wenjie_Jacky_Mo@Xiaofei_Wen_Mk@qiyao_ma@WenhuiZhu8@XiwenChen37@muhao_chen@zpegt
#MachineLearning #OpenSource #AI #ModelSelection @OpenRouter@huggingface
Excited to share that our paper "Rethinking Personalized Generation: Test-Time Alignment via Factorized Ranking Models" has been accepted to NeurIPS 2026 🍻
Our claim: LLMs already generate well-personalized responses. The bottleneck is picking them out.
Generalist reward models are poorly calibrated for personalization, and at billions of parameters they're too slow to score large pools. So we train a million-parameter personalized ranking model directly on the generator's own hidden states. Across 9 datasets and 3 personalization settings, it beats billion-parameter reward models on every one, with <0.4% of their parameters and ~10,000× faster scoring.
paper: https://t.co/MMdTE5eTlP
If you find it useful, a ⭐️ on the repo (https://t.co/tY8fq2UcBm) and an upvote ⬆️ on Hugging Face (https://t.co/HrYBJWcOTU) would mean a lot
#NeurIPS2026 #Personalization
Excited to share that Personalized RewardBench has been accepted to COLM2026! 🎉🍻
Reward models know what's good — but do they know what you like?
Our answer: not really. By controlling for general quality, we force models to compete strictly on personal preferences. Come see if your RM can actually read the room 👀
@COLM_conf #COLM2026
What if LLM agents could reason silently and think out loud only when needed?
💡Adaptive Latent Agentic Reasoning (ALAR)💡
Agents often generate verbose CoT for every action. ALAR makes this adaptive:
💭 latent reasoning for routine turns
💬 explicit CoT for harder decisions
We also built an interactive ModelLens Atlas 🌍
For each model or dataset, you can inspect its most compatible counterparts ranked by real benchmark performance.
This gives a more intuitive way to explore:
• Which models behave similarly
• Which datasets require similar capabilities
• Which models are strong candidates for a target task
• How the open-source model ecosystem is structured
Try it here:
🌐 https://t.co/dkcTnkNMSq
@Xiaofei_Wen_Mk Great work! Really sharp insight into the audio-visual Clever Hans effect in current video models. The finding that models “hear” sounds purely from visual priors is fascinating.
turns out, today's video models are mostly just pretending to listen.
👁️🎙️When Vision Speaks for Sound 👁️🎙️
We tested top models (including the new Gemini 3.5 Flash). They all suffer from the audio-visual Clever Hans effect: hallucinating sounds from visual cues instead of actually verifying audio. 🤯
🔇 Mute the audio? They still “hear” the crash.
⏱️ Shift the sound? They still say it is synced.
🔄 Swap the track? They often accept the mismatch.
🛠️ The fix? 10K pairs of targeted preference alignment. We boosted performance by 28 points—with zero drop in general video skills.
More details on how we diagnose and fix this here 👇
📄 Paper: https://t.co/SpQh0PIfbv
🔗 Website: https://t.co/nRZmH9Mec5
🤗 Collection: https://t.co/g5uNJ0Qkie
#videoaudio #videoaudioalignment #omnimodel
Video models can generate impressive motion — but can they actually reason?
📢 Introducing VideoRLVR: a systematic RL recipe that turns video models into visual reasoners using verifiable rewards.
🧩 Tasks: Maze, FlowFree, Sokoban
⚡ Early-Step Focus: ~40% faster RL training
🏆 Better in-domain + OOD reasoning than SFT and strong video baselines
📰 Paper: https://t.co/CkCqFymR4l
🖥️ Project: https://t.co/Wq2e59Q7gC
🚩 Instead of asking: “Does this video look realistic?”
VideoRLVR asks: “Does this video form a valid visual reasoning trajectory?”
💡 Our starting point is a simple analogy:
- LLMs learn to reason better by moving beyond imitation: Pre-training → SFT → RLVR
- But video reasoning is stuck at: Pre-training → SFT
🧐 So what would the RLVR stage look like for video models?
A video is not just pixels changing over time. For reasoning tasks, a video is a chain of visual states. If each state represents progress toward a goal, then the full video can be checked, scored, and optimized. That turns video generation into something closer to visual problem solving.
⚙️ Our recipe
1⃣ Verifiable reasoning tasks
Include 30,000 video samples on Maze, FlowFree, and Sokoban, ranging over various levels of difficulty.
2⃣ SDE-GRPO for video diffusion with Early-Step Focus
Only inject SDE noise and optimize at early denoise steps. They are the key to the global plan and let the latter steps do the rendering refinement they are already good at.
3⃣ Verifiable visual rewards
Define rewards over the generated trajectory, optimizing for valid transitions, constraint satisfaction, and goal completion.
🎯 What we achieve
- In-domain improvement over proprietary and open-source models, including strong video reasoning SFT baselines.
- OOD performance matches 14B models with a 5B model.
⏰ The broader message
Future video models should not only answer: “Can I synthesize this motion?” but also: “Can I generate a valid process that solves this problem?”
VideoRLVR is a step toward video models as visual reasoners, not just visual imitators.
Thanks to all the collaborators: @sheng_zh@JamesYHuang36@Selena3187@Xiaofei_Wen_Mk @ Yuankai Li @hoifungpoon@muhao_chen
#VideoGeneration #VideoReasoning #AI
We also built an interactive ModelLens Atlas 🌍
For each model or dataset, you can inspect its most compatible counterparts ranked by real benchmark performance.
This gives a more intuitive way to explore:
• Which models behave similarly
• Which datasets require similar capabilities
• Which models are strong candidates for a target task
• How the open-source model ecosystem is structured
Try it here:
🌐 https://t.co/dkcTnkNMSq
🚨 There are too many models to choose!
Whether you're shortlisting candidates before fine-tuning, picking among 500+ models on OpenRouter, or searching through hundreds of thousands of checkpoints on HuggingFace, the question is the same:
Which model should you actually use for a new task?
💡 Introducing ModelLens — a model recommendation system for the open-source model ecosystem.
Instead of asking users to evaluate or fine-tune every candidate model, ModelLens asks a different question:
👉 Can we leverage large-scale model–dataset interaction patterns from public leaderboards to enable model selection in the wild, without requiring direct evaluation or fine-tuning?
Our key insight:
The millions of (model, dataset, performance) records accumulating on all platforms are not just leaderboard noise. They encode how model capabilities align with task characteristics across the open-source ecosystem.
⚙️ How it works:
We collect large-scale model–dataset performance interactions from public leaderboards (HuggingFace, Open LLM Leaderboard, Papers with Code) and learn transferable model–task compatibility, jointly modeling:
- 47K models across 348 architecture families
- 9.6K datasets across 2,551 task types
- 8,420 heterogeneous evaluation metrics
- 1.62M historical performance records as implicit supervision
Given any task description, ModelLens then recommends a ranked Top-K pool of candidates likely to perform well, generalizing to newly released models and never-benchmarked datasets, before any of them is ever run on the target task.
task description → learned model–task compatibility → Top-K recommended models
📊 What ModelLens enables:
✅ Recommend Top-K models for a new dataset
✅ Rank newly released models against existing models
✅ Build model pools before routing or cascading
✅ Reduce costly brute-force evaluation
✅ Scale model selection in the open-source AI ecosystem
📰 Paper: https://t.co/5SyNfY3OlE
💻 Code: https://t.co/YUepL9ntpv
🌐 Demo: https://t.co/UnMt0zt55q
Try the demo with your own task description — we’d love feedback from people working on model selection, routing, AutoML, and open-source LLM ecosystems.
Thanks to all coauthors: @Wenjie_Jacky_Mo@Xiaofei_Wen_Mk@qiyao_ma@WenhuiZhu8@XiwenChen37@muhao_chen@zpegt
#MachineLearning #OpenSource #AI #ModelSelection @OpenRouter@huggingface
🌐 Want to try ModelLens yourself?
We built a demo where you can paste any task description and get back a ranked list of recommended models in seconds.
Try it: https://t.co/UnMt0zt55q
For example, describe a new benchmark you're building, an internal dataset you can't share, or a niche task no current leaderboard covers — and see which models ModelLens thinks fit best.
Would love feedback from anyone working on:
🔧 model selection
🔀 model routing or cascading
🤖 AutoML
🌍 open-source LLM tooling
Can SOTA reward models truly handle "pluralistic alignment"? Spoiler: Not really.
Introducing Personalized RewardBench! By controlling for general quality, we force models to compete strictly on user-specific rubrics.
See if your RM can actually read the room! 😎👇
🚨 Tokens are the surface.
Attention is the mechanism.
What if RL learned the latter?
💡 Introducing Reinforced Attention Learning
🧠 Key idea: Current RL for LLMs shapes "what is generated". We instead optimize "where the model focuses", to influence how it actually reasons.
📰 Paper: https://t.co/UnZ7txssG6
(with @Google and @GoogleDeepMind)
Standard PPO/GRPO perform importance sampling over token distributions, using advantages to up- or down-weight token probabilities.
⏳ The flip: Instead of "what token to generate", we optimize "where to attend"
At each generation step, we measure the divergence between attention distributions of the current and old policy—across all previous tokens.
- High-reward samples → keep attention close
- Low-reward samples → push attention away
The policy learns where to allocate computation, not just which surface-level token to generate.
⚙️ Plug-and-play
Our attention-level objective pairs with any policy gradient method, including PPO and GRPO.
📊 Results
RAL > Vanilla GRPO.
We’re seeing consistent accuracy gains across a wide range of image and video QA benchmarks when experimenting on multimodal LLMs.
✨ One more thing
Attention can be learned from a teacher model via on-policy attention distillation—going beyond knowledge transfer to inherit latent attention behaviors. This On-policy Attention Distillation takes standard On-Policy Distillation to the next level.
Thanks to all my colleagues: @jianmo_ni@Chen_Qu1@ianmiao@liuyang_irnlp@XingyuFu2@infolaber🙏
Your agent just broke a rule again 😅
Time to upgrade:
✅ Executable policies
✅ Trajectory-level safety
✅ Cross-domain control
Read how 🛡️:
https://t.co/AVe1wzHng9
More details 🌐:
https://t.co/7Rm3ZUoHdx