Zefan Cai

Now Ph.D student @UWMadison
Previous @PKU1898

[Cheng Luo @ChengLuo_lc]

Attention over tokens is multi-head. Attention over depth is not. We fixed that.

Attention residuals turn the residual stream into a depth memory: every sublayer attends over all earlier layers' outputs. But the read uses ONE query shared across the entire width — every feature subspace forced through the same softmax, however much they disagree about which layers to read. And disagreement grows with width.

Multi-Head Attention Residuals (MHAR): split the routing query into H per-subspace heads, each with its own softmax over depth.

✅ Zero new parameters — it's literally a reshape
✅ H=1 recovers attention residuals exact
✅ Best of four methods at 100M / 350M / 1B; the gain over a standard Transformer grows from 100M to the larger
scales (−0.061 → −0.149 / −0.140)
✅ Fused Triton kernels make depth routing practical, and an identity-preserving conversion brings it to 8B
mid-training: +3.2 GSM8K, +3.1 GPQA

📄 Paper: https://t.co/JFs9OTmIeA
📝 Blog: https://t.co/tNht32FvRs
💻 Code: https://t.co/uApmGlc93h

[Cheng Luo @ChengLuo_lc]

MHAR v2 is now on arXiv 📄 https://t.co/JFs9OTmIeA

Multi-Head Attention Residuals: attention residuals read the depth history through ONE query shared across the whole width — we split it into H per-subspace heads. Zero new parameters, H=1 recovers AttnRes exactly.

What changed from v1:

🔁 Main results re-based on a stronger corpus — deduplicated, quality-filtered, STEM- & code-heavy Nemotron-based anneal data (FineWeb-Edu results moved to the appendix). The gains grew: −0.061 / −0.149 / −0.140 val loss vs a standard Transformer at 100M / 350M / 1B, best of four methods in every setting.

📊 The 1B head-count study is now complete: full kv×H grid with a verified H=16 column (v1's figure had an incomplete 1B point). Val loss is U-shaped in H with a flat optimum at H=4–8; we adopt H=8 at scale.

Unchanged: fused Triton kernels (0.2–0.5× → 0.55–0.88× baseline throughput), identity-preserving 8B mid-training (+3.2 GSM8K, +3.1 GPQA).

💻 Code: https://t.co/uApmGlc93h

[Hanwen Jiang @hanwenjiang1]

Interested in how frontier labs pre-train image/video generation models?
We were too.

Since those recipes are rarely made public in full, we started from the most mature pretraining playbook available in the open: how modern LLMs are built.
Introducing Chimera: a visual generation model family that brings LLM-style hybrid linear attention and scaling co-design to visual generation.

In large-scale pretraining, nearly every design choice eventually shows up.
That means solving architecture design and scaling as one coupled problem: every architectural choice changes how the model scales, and scaling behavior determines which choices actually survive.
Chimera approaches both jointly, building a model family that remains predictable as model size, compute budget, and data distribution change.

Our key architectural observation is a simple division of labor: a single raster-ordered KDA stream carries long-range state, while modality-aware short convolutions preserve native local geometry.
Together, they form an effective and elegant linear-attention backbone for multidimensional visual data, with periodic MLA providing direct global interaction and sparse MoE expanding capacity at controlled activated compute.

This design comes with a useful effect: NoPE.
In Chimera, position is represented by the computation itself. KDA’s ordered recurrence and learned state decay encode order and recency, while the short convolutions encode local spatial-temporal structure.
Explicit positional embeddings are not needed in our design.
Because these mechanisms are not tied to a fixed training grid or sequence length, Chimera shows strong zero-shot extrapolation in both space and time.
Trained exclusively on 1K images and 5-second videos, it directly generates coherent 4K images with little visible quality degradation and 30-second videos with only 6.5% FID degradation over the final five seconds, all zero-shot, without resolution- or length-specific finetuning.

But architecture alone is not a pretraining recipe unless it scales predictably.
Thus, scaling should not be treated as an afterthought: fitting a curve over model sizes is easy; making that curve meaningful is much harder.
If every model size is differently under-tuned, your scaling law may simply be measuring optimization error.
We propose HeteroP to transfer proxy-tuned hyperparameters module by module across width and depth, giving us a consistently tuned model family. This allows us to fit Chinchilla-style laws over activated model size, training tokens, and the image-video data mixture.

The laws not only provide the reference for compute-optimal model and data size, but also suggest that visual generation may be more model-hungry than we tend to assume.
Under the same parametric loss-fitting method used in Chinchilla, compute-optimal model size grows as FLOPs^0.516 for images, compared with FLOPs^0.46 for language. Video is even more model-hungry, with the exponent rising from 0.516 for images to 0.544 for video.
Guided by these laws, we trained an 11B-parameter Chimera that activates only 2B parameters per token, using ~600 H100 days.
- It matches Wan-2.1 2B pretraining loss with 7.3x fewer FLOPs, and runs 2.14x faster than full attention at 255K tokens.
- It matches FLUX.1-dev and Z-Image-Turbo on GenEval and outperforms both on DPG-Bench, using roughly 20× less training compute than Z-Image-Turbo.

Ultimately, Chimera indicates that once you pretrain at scale, every decision shows up.
And if the Kimi K3 recipe caught your attention, Chimera may look oddly familiar, except the tokens are pixels and frames with diffusion models.

A team effort from team @ChongjianG30781 , me, @VisionSteve , Jiuxiang Gu, @Xu_Arthas , @chenziwee , @ShaotengLiu , @Jingorz , @YicongHong , @Zefan_Cai , @HaoTan5 ; supported by Hailin Jin and @kalyank_s at @Adobe @AdobeResearch

[Cheng Luo @ChengLuo_lc]

Kimi K3 is A GREAT WORK 👏 Congrats @Kimi_Moonshot — great to see Attention Residuals in the 2.8T backbone.

Take this chance to introduce our concurrent work MHAR: give attention residuals 8 routing heads instead of one. and it's a simple drop-in upgrade that wins at every scale we tested (100M/350M/1B).

📄 https://t.co/aNrnXua7n9

[Zefan Cai @Zefan_Cai]

Announcing the 2nd Workshop on Efficient Reasoning (ER) at @colm2026 — Oct 9!

[Cheng Luo @ChengLuo_lc]

🌟 Announcing the 2nd Workshop on Efficient Reasoning (ER) at @colm2026 — Oct 9!

📣 We welcome submissions! Submit your work here: https://t.co/loVmlunK87
🗓️ Deadline: July 12, 2026 (AoE)
🔗 Website: https://t.co/FRgQ95CcAd

💬 Topics include (but aren't limited to):
🔹 Multimodal, spatial & embodied reasoning under efficiency constraints
🔹 Curating high-quality reasoning datasets under resource constraints
🔹 Algorithmic innovations for efficient training & RL fine-tuning
🔹 Fast inference: pruning, compression, progressive generation, KV-cache tricks
🔹 Benchmarks & theory on time-/space-complexity and faithfulness
🔹 Systems to deploy long-CoT or on-device reasoning in the wild
🔹 Safety & robustness of efficient reasoning pipelines
🔹 Real-time applications in healthcare, robotics, autonomy, and more

🤝 We invite perspectives from ML, systems, natural & social sciences, and industry practitioners to rethink reasoning under tight compute, memory, latency, and cost budgets.

Hope to see you there! 🚀

[Zefan Cai @Zefan_Cai]

Delta Attention Residuals is the most beautiful work I did this year.

[Cheng Luo @ChengLuo_lc]

We're excited to release 𝐃𝐞𝐥𝐭𝐚 𝐀𝐭𝐭𝐞𝐧𝐭𝐢𝐨𝐧 𝐑𝐞𝐬𝐢𝐝𝐮𝐚𝐥𝐬, a drop-in upgrade to residual connections that
learns which past layers to route from — without the routing collapse that breaks prior cross-layer
attention at scale. 🚀

Attention Residuals route over cumulative hidden states, but those are highly redundant, so routing
collapses to near-uniform (max weight ~0.2) in deep layers. Delta Attention Residuals route over
𝐝𝐞𝐥𝐭𝐚𝐬 (vᵢ = hᵢ₊₁ − hᵢ) — what each sublayer actually contributed — and natively enable:

⚡ 𝟏.𝟖× 𝐬𝐡𝐚𝐫𝐩𝐞𝐫 𝐜𝐫𝐨𝐬𝐬-𝐥𝐚𝐲𝐞𝐫 𝐫𝐨𝐮𝐭𝐢𝐧𝐠
Deltas are structurally diverse, lifting max attention weight from ~0.2 → ~0.6 (0.62 vs 0.35 avg)
and curing routing collapse in deep layers.

📉 −𝟖.𝟐% 𝐯𝐚𝐥𝐢𝐝𝐚𝐭𝐢𝐨𝐧 𝐏𝐏𝐋 𝐚𝐭 𝟕.𝟔𝐁
Consistent gains from 220M → 7.6B (1.7–8.2% lower PPL), beating both standard residuals and
Attention Residuals — the latter actually degrades below baseline at scale (18.58 vs 17.43).

🔌 𝐃𝐫𝐨𝐩-𝐢𝐧 𝐟𝐢𝐧𝐞-𝐭𝐮𝐧𝐢𝐧𝐠 𝐨𝐟 𝐩𝐫𝐞𝐭𝐫𝐚𝐢𝐧𝐞𝐝 𝐦𝐨𝐝𝐞𝐥𝐬
Additive, zero-init routing is identity at initialization, so you can convert pretrained
checkpoints (e.g. Qwen3-0.6B) into Delta Attention Residuals via standard fine-tuning — beating the
original on 8 downstream benchmarks (55.6 vs 55.0).

🪶 ≤𝟎.𝟎𝟏% 𝐩𝐚𝐫𝐚𝐦𝐞𝐭𝐞𝐫 𝐨𝐯𝐞𝐫𝐡𝐞𝐚𝐝
Delta Block adds just 589K params (0.008% at 8B) and ~3% memory — and runs faster + lighter than
Attention Residuals (14.0k vs 12.5k tok/s, 42.7 vs 44.0 GB).

💻 Code: https://t.co/c8E4NXCZWn
📄 Paper: https://t.co/Mj1W07qOm2

[Cheng Luo @ChengLuo_lc]

We open-source Attention Residuals — replacing standard additive residuals with learned cross-layer attention in transformers.

Block AttnRes reduces WikiText-2 perplexity by 7.7% with only 0.03% extra parameters.

Includes visualization of how layers route information across depth.

Code: https://t.co/6aBEDlIn1Y
Blog: https://t.co/3VMF7u1PlW

[Zefan Cai @Zefan_Cai]

Attention sinks also exists in attention residual😂

[Cheng Luo @ChengLuo_lc]

We open-source Attention Residuals — replacing standard additive residuals with learned cross-layer attention in transformers.

Block AttnRes reduces WikiText-2 perplexity by 7.7% with only 0.03% extra parameters.

Includes visualization of how layers route information across depth.

Code: https://t.co/6aBEDlIn1Y
Blog: https://t.co/3VMF7u1PlW

[Zihan "Zenus" Wang @wzenus]

I keep coming back to John Schulman’s line that “value functions are underrated.” I think we still haven’t figured out how to use them for LLM agents.

For token-level LLMs, the horizon is huge. If you naively put a value function on tokens, variance blows up and credit assignment smears across thousands of sub-decisions. You get a beautiful value head on paper, but very noisy learning signals in practice.

In our recent VAGEN work we’re trying a bi-level view: estimate value both at the environment step level and at the reasoning token level (bi-level GAE), so the agent learns which thoughts and which interactions actually matter. It’s still early, but I’m convinced “good value functions for agents” is a wide-open design space — curious what others are trying here.

[Salesforce AI Research @SFResearch]

🚨 Can video models actually REASON, or just look good?

Introducing MMGR: comprehensive benchmark testing Veo-3, Sora-2, Wan-2.2, Nano-banana, GPT-4o & Qwen on physical, logical, spatial & temporal reasoning.

📄 https://t.co/u76Dc84fxa
🌐 https://t.co/aP5vYA5tIT

Key findings:
- Models excel at physics (~60%) but FAIL at abstract logic (<10% on ARC-AGI)
- Video models struggle with long-horizon planning
- Huge gap between "looking real" and "acting real"

Authors: @Zefan_Cai @HaoyiQiu @TianyiM15933 Haozhe Zhao, @GengzeZhou @steeve__huang @Kordjamshidi @Minjia_Zhang @wendyxiao06091 @joshua_jxgu @VioletNPeng @JunjieHu12 @uwmadison @UCLA @michiganstateu @UIUC @SFResearch @Microsoft @AdobeResearch

[Christopher Nguyen ⽗ @pentagoniac]

TL;DR: these video models excel at generating plausible next frames that look like spatio-temporal reasoning, but are terrible at abstract logical reasoning and global consistency—moving through walls, for example.

Cc @ylecun @demishassabis

[Junjie Hu @JunjieHu12]

🚀 Excited to share our new research project MMGR (Multi-Modal Generative Reasoning)!
STOA video models look stunning—but do they actually reason? Our large-scale benchmark puts leading video and image models to the test across physical, logical, spatial, and temporal reasoning.

[Tianyi Ma @ACL2026 @TianyiM15933]

Using notebooklm to introduce our MMGR

[GZ Zhou @GengzeZhou]

In MMGR, we find that modern video generation models (Veo2, Sora-2, WAN) exhibit surprisingly strong capabilities for generating ego-centric view navigation episodes. We will open-source our Habitat/IsaacSim data pipelines to facilitate community-driven, rigorous evaluation.

[Rohan Paul @rohanpaul_ai]

MMGR is a benchmark checking if video and image generators follow real world rules, and they often do not.

On a grade school math test, Veo 3 reached 74% correct final answers, but only 12% of its steps made sense.

Popular video scores reward pretty frames, so physics breaks and broken logic often go unnoticed.

MMGR, short for Multi Modal Generative Reasoning, scores 5 reasoning skills, physical cause and effect, logical rules, 3D space, 2D layout, and time order.

It tests them on 3 domains, abstract puzzles like mazes and Sudoku, navigation tasks, and commonsense physics scenes.

Each task gives a picture plus a prompt, and the model must generate either a single image or a multi frame video.

Another model that can read images and text acts as a grader by checking goal reached, correct path, and believable motion.

The results show a gap between looking coherent and being coherent, especially for videos that must keep logic consistent over time.

That matters for tutorials, simulations, and robot training, because a mistake can send people, or agents, the wrong way.

----

Paper Link – arxiv. org/abs/2512.14691

Paper Title: "MMGR: Multi-Modal Generative Reasoning"

[Haoyi Qiu @HaoyiQiu]

Video models like Sora-2 and Veo-3 look incredible, but can they actually 𝗿𝗲𝗮𝘀𝗼𝗻? 🤔

📣 Introducing 𝗠𝗠𝗚𝗥: 𝗠𝘂𝗹𝘁𝗶-𝗠𝗼𝗱𝗮𝗹 𝗚𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝘃𝗲 𝗥𝗲𝗮𝘀𝗼𝗻𝗶𝗻𝗴, a new benchmark evaluating 𝘃𝗶𝗱𝗲𝗼 (Veo-3, Sora-2, and Wan-2.2) AND 𝗶𝗺𝗮𝗴𝗲 models (Nano-banana/Pro, GPT-4o-image, Qwen-image) on 5 core abilities: Physical, Logical, 3D/2D Spatial, and Temporal reasoning across three domains: Abstract Logic, Embodied Navigation, and Physical Commonsense.

🥁 Our findings? We have conducted an extensive 𝟴𝟬+ 𝗽𝗮𝗴𝗲 deep-dive analysis: While models excel at visual physics, they fail catastrophically at 𝗮𝗯𝘀𝘁𝗿𝗮𝗰𝘁 𝗹𝗼𝗴𝗶𝗰 (<10% on ARC-AGI for most video models) and 𝗹𝗼𝗻𝗴-𝗵𝗼𝗿𝗶𝘇𝗼𝗻 𝗽𝗹𝗮𝗻𝗻𝗶𝗻𝗴. It’s time to move from “looking real” to “acting real.” 🧵👇

[Zefan Cai @Zefan_Cai]

See Multi-Modal Generative Reasoning!

[Junjie Hu @JunjieHu12]

🚀 Excited to share our new research project MMGR (Multi-Modal Generative Reasoning)!
STOA video models look stunning—but do they actually reason? Our large-scale benchmark puts leading video and image models to the test across physical, logical, spatial, and temporal reasoning.

[Griffintaur @griffintaur]

🧠 New #AI Research: MMGR: Multi-Modal Generative Reasoning

📄 https://t.co/aZi52WecCQ

#MachineLearning #AIResearch #DeepLearning

[AI 原生基金会 @AINativeF_zh]

📚 AI Native 每日论文摘要 - 2025-12-17🌟

关注我们 @AINativeF_zh，获取AI原生领域的最新洞察。

本期介绍下图中来自Hugging Face的AI研究论文，帮助您及时了解最新研究趋势，让我们一起探索AI的未来！

#AI #HuggingFace #AIPaper #AINative #AINF
— 附录：今日AI研究论文 —

1. MMGR: 多模态生成推理

2. 视频现实测试：AI生成的ASMR视频能否欺骗视觉语言模型和人类？

3. WorldPlay：迈向长期几何一致性的实时交互世界建模

4. Scone：通过统一理解-生成建模在主体驱动图像生成中弥合组合与区分

5. RoboTracer：在视觉语言模型中通过推理掌握空间轨迹以用于机器人

6. OpenDataArena：一个用于基准测试训练后数据集价值的公平开放竞技场

7. 揭示隐藏的陷阱并从任务中心视角引领下一代向量相似性搜索

8. Vector Prism: 通过分层语义结构为矢量图形赋予动画效果

9. ShowTable: 通过协作反思与完善释放创意表格可视化的潜力

10. RecGPT-V2 技术报告

11. MemFlow：自适应内存流动以实现一致且高效的长视频叙述

12. 通过文本可控的图像到3D实现前馈3D编辑

13. 可微分进化强化学习

[AI 原生基金会 @AINativeF_zh]

1. MMGR: Multi-Modal Generative Reasoning
    MMGR: 多模态生成推理

🔑 关键词: MMGR, 生成推理, 感知质量, 因果关系, 空间规划

💡 类别: 多模态学习

🌟 研究目标:
- 本研究介绍了MMGR，作为一个新的评估框架，用于评估视频和图像生成模型在多种推理领域的能力。

🛠️ 研究方法:
- 本研究使用MMGR基于五种推理能力来评估模型: 物理推理、逻辑推理、三维空间推理、二维空间推理和时间推理，并将其应用于如抽象推理、体感导航和物理常识等多样领域。

💬 研究结论:
- 当前视频和图像模型的主要局限性被识别出来，包括过度依赖感知数据和整体一致性较弱。MMGR基准标出了显著的性能差距，模型在物理常识任务中表现较好，但在抽象推理和长远视角的空间规划方面表现不佳。

👉论文地址: https://t.co/SqaBxb8Z9f