Olivia
Online
Yifan Song
@99Solaris
Ph.D. student@Peking University
Joined March 2020
162 Following
169 Followers
25 Posts
Quiet ambush🦄
MiMo-V2-Pro & Omni & TTS is out. Our first full-stack model family built truly for the Agent era.
I call this a quiet ambush — not because we planned it, but because the shift from Chat to Agent paradigm happened so fast, even we barely believed it. Somewhere in between was a process that was thrilling, painful, and fascinating all at once.
The 1T base model started training months ago. The original goal was long-context reasoning efficiency. Hybrid Attention carries real innovation, without overreaching — and it turns out to be exactly the right foundation for the Agent era. 1M context window. MTP inference for ultra-low latency and cost. These architectural decisions weren't trendy. They were a structural advantage we built before we needed it.
What changed everything was experiencing a complex agentic scaffold — what I'd call orchestrated Context — for the first time. I was shocked on day one. I tried to convince the team to use it. That didn't work. So I gave a hard mandate: anyone on MiMo Team with fewer than 100 conversations tomorrow can quit. It worked. Once the team's imagination was ignited by what agentic systems could do, that imagination converted directly into research velocity.
People ask why we move so fast. I saw it firsthand building DeepSeek R1. My honest summary:
— Backbone and Infra research has long cycles. You need strategic conviction a year before it pays off.
— Posttrain agility is a different muscle: product intuition driving evaluation, iteration cycles compressed, paradigm shifts caught early.
— And the constant: curiosity, sharp technical instinct, decisive execution, full commitment — and something that's easy to underestimate: a genuine love for the world you're building for.
We will open-source — when the models are stable enough to deserve it.
From Beijing, very late, not quite awake.
Introducing MiMo-V2-Pro & Omni & TTS
https://t.co/qdufHaBJWY
⚡️Frontier intelligence. Blazing speed. Unbeatable price!
MiMo-V2-Flash scores 66 on the @ArtificialAnlys Intelligence Index — #2 among open-source models and #8 overall! 🎉🎉🎉
Designed for Agentic AI — now with the benchmarks to prove it: #1 on τ²-Bench Telecom for agentic tool-use among all evaluated models. ⚡⚡⚡
Frontier intelligence. Blazing speed. Unbeatable price.
Try it out 👇
🤗 Model: https://t.co/4Etm0z0iJj
🎨 AI Studio: https://t.co/nSReUs7o6u
🌐 API: https://t.co/MJ8GAIJGPW
Who to follow
Ning Ding 
@stingning
Researcher of AI. Assistant Professor @Tsinghua_Uni. Working on scalable methods of language and physical models.
Piotr Nawrot
@p_nawrot
LLM Efficiency @NVIDIA - views have always been only my own
🥇🥈 @ Flunkyball Polish Championships
Liang Chen
@liangchen5518
Cofounder of @UniPat_AI. I worked at Moonshot AI, Alibaba Qwen and Microsoft Research Asia.
Faster than fast. Stronger than strong.
⚡ Faster than Fast. Designed for Agentic AI.
Introducing Xiaomi MiMo-V2-Flash — our new open-source MoE model: 309B total params, 15B active.
Blazing speed meets frontier performance.
🔥 Highlights:
🏗️ Hybrid Attention: 5:1 interleaved 128-window SWA + Global | 256K context
📈 Performance:
⚔️ Matches DeepSeek-V3.2 on general benchmarks — at a fraction of the latency
🏆 SWE-Bench Verified: 73.4% | SWE-Bench Multilingual: 71.7% — new SOTA for open-source models
🚀 Speed: 150 output tokens/s with Day-0 support from @lmsysorg🤝
🤗 Model: https://t.co/4Etm0yZKTL
📝 Blog Post: https://t.co/5zxmcDuB6o
📄 Technical Report: https://t.co/crac1YTLYl
🎨 AI Studio: https://t.co/nSReUs6QgW
Fighting for AGI together!
Intelligence will inevitably evolve from language to the physical world, unlocking spatial intelligence for multi-modal perception, reasoning, generation, and action—essential for true AGI.
I'm working on building this at @XiaomiMiMo, spearheading a creative and talented team!
MiMo-VL technical report, models, and evaluation suite are out!
🤗 Models: https://t.co/Qb2zYTVfzS (or RL)
Report: https://t.co/AqTpy0r2bI
Evaluation Suite: https://t.co/s0rU38DoyU
Looking back, it's incredible that we delivered such compact yet powerful vision-language models in under six months.
Here are my key takeaways from our journey:
Reasoning is now essential for VLMs. Adding long chain-of-thought data to our training produced clear performance gains across all benchmarks. What's fascinating is watching our model actually examine different parts of images, checking various details before working through its reasoning to reach an answer.
Mixed reward learning was our biggest challenge and most inspiring discovery. We saw comprehensive improvements on almost every task with objective rewards like document perception, visual grounding, and multimodal math. MiMo-VL-RL is now the best open-source VLM on the InfoVQA test set. But subjective rewards like human preference data proved much trickier—models learn to game these signals surprisingly quickly. Finding the right balance is truly an art.
We're committed to reproducible VLM research. Throughout development, we experienced firsthand how difficult it is to reproduce results from other papers. Different prompts, temperature settings, and evaluation processes make fair comparisons nearly impossible. That's why we're releasing our complete evaluation suite covering 50+ tasks, built on lmms-eval, with fully reproducible results. We might be the first to do this comprehensively, and we hope it helps advance the field by making research more transparent and comparable.
Today, Xiaomi releases MiMo, our first open-source reasoning model. At 7B parameters, it’s optimized for reasoning through pre-training and post-training, surpassing OpenAI’s o1-mini and QwQ-32B-Preview on AIME 2024-2025 and LiveCodeBench v5 benchmarks.
#XiaomiMiMo #MiMo7B
📢 Introducing VL-RewardBench - A new benchmark for vision-language generative reward models (VL-GenRMs)!
📊Even SOTA models struggle: Gemini-1.5-Pro (67.2%) & GPT-4o (65.8%).
🧐The best open-source model LLaMA-3.2-90B only hits 56.2%, most others below random chance!
👉Project page: https://t.co/KvAingNQPa
🚀Introducing MixEval-X, the first real-world, any-to-any benchmark. https://t.co/XJeUAYMDhQ
It extends all benefits of MixEval to multi-modal evaluations, including real-world query distribution, fast yet accurate model ranking, high standards evaluation across modalities!
🏇 Frontier players are racing to solve modality puzzles in the quest for AGI.
But to get there, we need consistent, high-standard evaluations across all modalities!
🚀 Introducing MixEval-X, the first real-world, any-to-any benchmark.
Inheriting the philosophy from MixEval, MixEval-X optimizes the benchmark data mixtures to reflect real-world task distributions, across diverse input and output modalities.
Several key features setting it apart:
🥇 Real-world task distribution, ensuring the evaluation results generalizable to real-world use cases;
🥇 Accurate model ranking (up to 0.98 correlation with Arena-like multi-modal evals);
🥇 Consistent, high standards across modalities, preventing any from lagging behind;
🥇 Decoupled evaluations for different modalities, compatible with models having any numbers of input / output modalities;
🥇 Fast, cost-effective, reproducible execution;
🥇 Dynamic data refresh;
🥇 Challenging tasks;
🥇 Organization-level benchmarking with balanced dimensions (modalities)–a new level of evaluation.
Project Page: https://t.co/oLPM4fyu5O
Paper: https://t.co/GEY2JAopb3
Github Repo: https://t.co/hPrxDBvo6u
Data: https://t.co/isny327hWQ
Working on multimodal instruction tuning and finding it hard to scale? Building Web/GUI agents but data is too narrow?
Introducing 🚀MultiUI: 7.3M multimodal instructions from 1M webpage UIs, offering diverse data to boost text-rich visual understanding.
Key takeaways:
🌟WebUI-trained models show major gains in visual web understanding and agent tasks. 💻
🌟Models also generalize well to non-UI tasks like DocVQA/OCR. 📄
How it works:
We generate multimodal instructions with a text LLM using structured text from webpage accessibility trees. We then pair them with UI screenshots, to train multimodal models.
Homepage: https://t.co/pxEU9iMkqc
Paper: https://t.co/2yLqAlhNxA
Dataset: https://t.co/e2Kir53W8K
Model: https://t.co/2WBgGnU2Xc
Congrats to the student lead @jeepliu1212 and the team @tianyue_01 @99Solaris @QuYuxiao @XiongChenyan @WenhuChen @gneubig !
More details are in the following threads ⬇️
How to get ⚔️Chatbot Arena⚔️ model rankings with 2000× less time (5 minutes) and 5000× less cost ($0.6)?
Maybe simply mix the classic benchmarks.
🚀 Introducing MixEval, a new 🥇gold-standard🥇 LLM evaluation paradigm standing on the shoulder of giants (classic benchmarks).
https://t.co/igeaQuZASt
🕶️LLM Benchmark Mixture:
We mine comprehensive and well-distributed 🌎real-world user queries from the web and match them with similar queries from off-the-shelf 💯ground-truth-based benchmarks.
🤔Why to Use MixEval?
(1) 🎯 Accurate model ranking (0.96 correlation with Chatbot Arena)
(2) ⚡️ Fast, cheap, and reproducible execution, requiring only 6% the time and cost of MMLU
(3) 🌊 Dynamic benchmarking enabled by low-effort and stable updating mechanism
(4) 🏔️ Challenging question set (GPT-4o, the top model on MixEval leaderboard, achieves 64.7% accuracy)
(5) 🌌 Comprehensive and highly impartial query distribution, as it is deeply grounded in real-world user queries
(6) ⚖️ Fair grading process without preference bias, ensured by its ground-truth-based nature
❌ What's wrong with the current LLM evaluation?
(1)❓Query Bias: evaluation queries falling short of
comprehensiveness or appropriate distribution
a) ground-truth-based benchmarks
b) LLM-judged benchmarks
(2)👨⚖️Grading Bias: the grading process involving
significant bias or error
a) LLM-judged benchmarks
(3)🔬Generalization Bias: models overfitting the
evaluation data (contamination)
a) ground-truth-based benchmark
b) LLM-judged benchmarks
🤔 Any current benchmarks that are not so biased?
☑️ Yes. Large-scale user-facing benchmarks, e.g., ⚔️Chatbot Arena⚔️, solve
(1) query bias by collecting a large number of real-world user queries,
(2) grading bias by collecting a large number of real-world user preferences, and
(3) generalization bias by continuously receiving fresh queries.
But they are prohibitively 💰expensive (around $2936 for a single model, see the below figure), ⌚slow, and 🚫irreproducible!
✅MixEval addresses all these issues.
It's not only highly unbiased in query, grading, and generalization, but also fast, cheap, and reproducible.
📊We provide extensive meta-evaluation and insights for MixEval and existing LLM benchmarks in our paper. 🔥We hope this will deepen the community’s understanding of LLM evaluation and guide future research directions!
🏆Our dynamic leaderboard is now available at: https://t.co/VrkgE2DZRq
🚀Join us in revolutionizing LLM evaluation! Test your model on MixEval and see where you stand on our dynamic leaderboard.
🌊 We will update the data points on a monthly basis.
🚀 Moving forward, we'll continuously add new benchmarks to our pool as they release. This will refine our mixtures and enhance dynamism at a higher level.
This work is done by @NiJinjie, @XueFz, @xiangyue96, @yuntiandeng, Mahir Shah, Kabir Jain, @gneubig, and @YangYou1991. Kudos to the team!
We also sincerely thank @Francis_YAO_ @gblazex @zhansheng @_jasonwei @p_nawrot @soldni @guanzhi_wang @deepanwayx @BoLi68567011 @JunhaoZHANG19 @99Solaris @ZangweiZheng @zian_andy_zheng @KevinQHLin @WenhuChen @billyuchenlin and colleagues from NUS HPC-AI Lab & CMU NeuLab for insightful discussions and pointers!
🚀Excited to share our new paper "LongEmbed: Extending Embedding Models for Long Context Retrieval". We introduce the LongEmbed benchmark, explore context extension of existing embedding models, and release E5-Base-4k & E5-RoPE-Base.
Paper: https://t.co/QM0XpnhPEQ
🚀Introducing VisualWebBench: A Comprehensive Benchmark for Multimodal Web Page Understanding and Grounding. https://t.co/BWGqWSP90a
🤔What's this all about? Why this benchmark?
> Back in Nov 2023, when we released MMMU (https://t.co/FXbFDDa09T), a comprehensive multimodal understanding benchmark, we received feedback that it included very few UI screenshots. Considering the growing importance of UI understanding, especially with the rise of powerful agents like Devin (https://t.co/o6R9AiLGbk), which is built on the strong vision capability of #GPT4, we recognized the need for a benchmark focused on UI screenshot understanding.📸👀
> Multimodal #LLMs have significantly boosted web agents' performance on benchmarks like Mind2Web and WebArena. For instance, the SeeAct agent (https://t.co/8tgTmaiAdB) showcases the power of integrating vision into web agents. However, these benchmarks primarily evaluate the end-to-end task execution ability of web agents rather than their understanding of web pages.
🌉 Bridging the Gap with VisualWebBench
> To provide a comprehensive evaluation of multimodal LLMs' web page understanding capabilities, we introduce VisualWebBench. Our benchmark spans 139 websites 🌐 across 12 domains 🏷️ and 87 sub-domains 🔍, ensuring a diverse and representative dataset. It assesses MLLMs at three levels: website-level, element-level, and action-level 📊, and encompasses seven tasks designed to evaluate understanding, OCR, grounding, and reasoning abilities 🧠💡.
😮 Surprising Findings
> 🎉 Open-source models are catching up: Even though closed-source MLLMs are still leading the leaderboard, we are happy to see open-source models like LLaVA 1.6 34B achieve comparable performance to Gemini Pro.
> 🧠 Grounding ability, crucial for developing MLLM-based web applications, is a weakness for most MLLMs.
> 🖼️ Importance of Image Resolution: The limited image resolution handling capabilities of most open-source MLLMs restrict their utility in web scenarios, where rich text and elements are prevalent.
> 🧱 Relatively strong correlation with general understanding benchmarks like MMMU but weak correlation with web agent benchmarks like Mind2Web. Web agent benchmarks primarily evaluate the end-to-end task execution ability of web agents, which involves a series of actions to accomplish a goal. In contrast, VisualWebBench emphasizes evaluating the foundational skills of MLLMs such as understanding and grounding web page elements.
💡Fun Fact
> Claude Sonnet is better than Opus on our benchmark :)
🎓 Conclusion
> VisualWebBench serves as a valuable resource for the community, driving research and development in the field of multimodal web page understanding and grounding. As MLLMs continue to evolve and improve, we look forward to seeing new applications and breakthroughs. We believe that our benchmark will contribute to the development of more powerful MLLMs in the web domain, ultimately leading to a more intuitive and efficient user experience on the web.
Kudos to the student leads @jeepliu1212 @99Solaris and the team @billyuchenlin, Wai Lam, @gneubig, Yuanzhi Li! 👏
Check out more details in the Junpeng's thread👇
Introducing AI2 𝕎𝕚𝕝𝕕𝔹𝕖𝕟𝕔𝕙 ! We aim to benchmark LLMs with challenging tasks from real users in the wild. 🤗 Link: https://t.co/eoSWvyMaLP
🤩 What great features does it offer? 🌟x9 ⬇️
🌟1. 𝐂𝐡𝐚𝐥𝐥𝐞𝐧𝐠𝐢𝐧𝐠 & 𝐑𝐞𝐚𝐥: We carefully curate a collection of 1024 hard tasks from real users, which cover common use cases such as code debugging, creative writing, and data analysis.
🌟2. 𝐀𝐮𝐭𝐨𝐄𝐯𝐚𝐥 𝐰/ 𝐂𝐡𝐞𝐜𝐤𝐥𝐢𝐬𝐭𝐬: Instead of merely asking GPT-4 to choose between A and B, we provide an instance-specific 𝘊𝘩𝘦𝘤𝘬𝘭𝘪𝘴𝘵 (i.e., a list of evaluation questions) for it to reason before making a judgment. It’s quite similar to 𝗖𝗼𝗧. Thus, our eval is highly interpretable and easy-to-verify.
🌟3. 𝐋𝐞𝐧𝐠𝐭𝐡 𝐏𝐞𝐧𝐚𝐥𝐭𝐲: GPT-4 judges tend to prefer longer outputs (although humans do too); to avoid this, we devise a simple method to add length penalty on Elo. You can even slide it on our leaderboard UI!
🌟4. 𝐓𝐚𝐬𝐤 𝐂𝐚𝐭𝐞𝐠𝐨𝐫𝐢𝐳𝐚𝐭𝐢𝐨𝐧: We tag each example with 12 task types, so we can analyze task-specific performance of LLMs, in addition to their overall ranking.
🌟5. 𝐅𝐚𝐢𝐫 𝐂𝐨𝐦𝐩𝐚𝐫𝐢𝐬𝐨𝐧𝐬: WildBench tests all examples on all LLMs. This is different from arena-style evaluation, where one example is only tested on a single pair of models and never seen again.
🌟6. 𝐄𝐚𝐬𝐲-𝐭𝐨-𝐮𝐬𝐞: WildBench (v1.0) contains 1024 examples now, and it is extremely easy to add your own LLMs to our leaderboard! We will do the work for you!
🌟7. 𝐃𝐲𝐧𝐚𝐦𝐢𝐜: WildBench will not be a static dataset. We will continue adding new examples and updating evaluation methods based on community feedback.
🌟8. 𝐇𝐮𝐦𝐚𝐧 𝐄𝐯𝐚𝐥𝐮𝐚𝐭𝐢𝐨𝐧 (ongoing): We are collecting human preferences via our Leaderboard UI (check the 🔍 🆚 tab). Please help us vote! (We’re planning to recruit domain experts too.)
🌟9. 𝐂𝐨𝐦𝐦𝐮𝐧𝐢𝐭𝐲 𝐝𝐫𝐢𝐯𝐞𝐧: We welcome everyone to contribute to human evaluation and create challenging examples. We also value your feedback and suggestions, and will continue enhancing our benchmark leaderboard accordingly.
To sum up, 𝕎𝕚𝕝𝕕𝔹𝕖𝕟𝕔𝕙 aims to dynamically benchmark LLMs with challenging real-world tasks and provide quick yet reliable evaluations.
Thank you to our amazing team @allen_ai: @khyathi_chandu @faeze_brh @yuntiandeng @lasha_nlp @valentina__py @Ronan_LeBras @YejinChoinka!
How does a baby learn to navigate the world around them? 🚶♂️👶 Through exploration and learning from each little stumble and triumph. The ETO framework applies this very essence of human learning to AI, emphasizing the importance of both success and failure in developing better AI agents.
ETO has following features:
🎲 Learning from Failure Trajectories. Contrary to previous approaches that exclusively train on successful expert trajectories, ETO allows agents to learn from their exploration failures.
🎭 Contrastive Trajectory Optimization. ETO applies DPO loss to perform policy learning from failure-success trajectory pairs.
🌏 Iterative Policy Learning. ETO can be expanded to multiple rounds for further policy enhancement.
⚔️ Effectiveness on Three Datasets. ETO significantly outperforms strong baselines, such as RFT, PPO, on WebShop, ScienceWorld, and ALFWorld.
🦾 Generalization on Unseen Scenarios. ETO demonstrates an impressive performance improvement of 22% over SFT on the challenging out-of-distribution test set in ScienceWorld.
Great work led by Yifan Song and @billyuchenlin! Kudos to the team @Wade_Yin9712 @jefffhj!
Check out our work for more details 👇
Obstacles and failures are the most stable stepping stones to success💪. This also holds true for LLM agents🤖.
🚀Check out ETO, a framework helps LLM agents learn by trial and error.
💖Great thanks for @Wade_Yin9712, @xiangyue96, @jefffhj, Prof. Sujian Li, and @billyuchenlin!
Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents
Presents an exploration-based trajectory optimization approach, which consistently surpasses baseline performance by a large margin
repo: https://t.co/uRk3Yb9pWK
abs: https://t.co/sGLOnTtmmz
⚕️RestGPT
New paper (with code) connecting LLMs with Real-World RESTful APIs by @99Solaris
Code here: https://t.co/vT6tDt0JJB
Always fun to use LangSmith to get a better sense of what's going on under the hood!
Traces:
https://t.co/ZTVjT2i6Sn
https://t.co/YRHjOI74Sg
@showhnposts The data in our repo is for the evaluation of RestGPT. Since RestGPT rely on GPT-3.5 and do not need training, just use the data to test our method.
How to connect large language models with real-world applications?🤔
🤯We're releasing RestGPT, a framework to connect LLMs with real-world applications, such as Spotify, via RESTful APIs.
Paper: https://t.co/kVEDMXNL1y
The code and demo will be released soon.
@mkrupskis1 The access tokens are placed in the HTTP headers. You can see line 35-57 in https://t.co/ZPhioJ3wIk for more details.
Last Seen Users on Sotwe
Trends for you
Most Popular Users
Elon Musk
@elonmusk
240.9M followers
Barack Obama
@barackobama
119.2M followers
Donald J. Trump
@realdonaldtrump
111.8M followers
Cristiano Ronaldo
@cristiano
111.7M followers
Narendra Modi
@narendramodi
107.1M followers
Rihanna
@rihanna
97.9M followers
NASA
@nasa
92.2M followers
Justin Bieber
@justinbieber
91.1M followers
KATY PERRY
@katyperry
88.3M followers
Taylor Swift
@taylorswift13
82.1M followers
Lady Gaga
@ladygaga
73.6M followers
Virat Kohli
@imvkohli
70.8M followers
Kim Kardashian
@kimkardashian
70.1M followers
YouTube
@youtube
68.7M followers
Bill Gates
@billgates
64.2M followers
Neymar Jr
@neymarjr
63.7M followers
The Ellen Show
@theellenshow
62.4M followers
CNN
@cnn
61.8M followers
Selena Gomez
@selenagomez
61.3M followers
X
@x
60.8M followers