Zhouhan Lin

Associate Professor at Shanghai Jiao Tong University @sjtu1896. Formerly @ Facebook AI Research @MetaAI. Ph.D. @Mila_Quebec with Yoshua Bengio.

[Zhouhan Lin @zhouhan_lin]

Top tech giants iterate LLMs at breakneck speed behind closed doors, only releasing final model weights without revealing the internal trade-offs, trial-and-error, or architectural decisions. Meanwhile, the traditional academic publishing cycle—from paper submission to peer review—moves too slowly to keep up. As a result, the most valuable frontier insights are left circulating through word-of-mouth and industry rumors.

We desperately need a shared, transparent space to track the real science of LLM pre-training.

That’s why we launched ArchSpace—a platform pioneering "Crowdsourced Architecture Exploration" for LLMs.

Here is how the new game is played:
🔹 Complete Pre-training Transparency: ArchSpace will open up full checkpoints, training logs, and experimental results for each of the architectural ideas, in the form of GitHub issues and pull requests. Both successful and negative trials will be logged. A PR including a terse description, a change log of code, and the corresponding experimental results makes it complete. No more need to write a 10 page paper and wait for several months for acceptance!
🔹 The ArchSpace (repo at https://t.co/UB5dQuptbP) starts from a standard Decoder-only Transformer, specifically the Olmo-3 architecture released by @allen_ai , and accepts ideas of architectural innovation from the community.
🔹Anyone in the community can propose their own ideas through GitHub Issues, and submit their experiment results through Pull Requests. If the idea turns out to work well, it gets scaled up, and can get merged into the original repo. Every idea and experiment is tracked and credited in this way, thus enables to disseminate proper recognition to every contributor.
🔹All changes can be visually traced through the interactive ArchSpace website (https://t.co/wJ9Z7Kwcy0)

Let’s shift LLM research from industry rumors to transparent, reproducible science!

🌐 Explore the platform: https://t.co/wJ9Z7Kwcy0
And GitHub repo: https://t.co/UB5dQuptbP

[Jason Zhu @GoSailGlobal]

Stanford CS336 上，Tatsu 讲了一节 LLM 架构课，把过去 3 年所有主流 LLM 拆开，看它们的共通模板

结论挺爆：90% 的架构选择已经收敛，你随便挑一个开源大模型，它跟其他模型在这些维度上几乎一模一样

讲师的原话
- 2024 年大家都在 cosplay Llama2
- 2025 年的主题是「怎么训得不崩」
- 2026 年的主题是「怎么扛住长上下文」

下面是 2026 年开源 LLM 的标准模板 你训自己的模型可以直接抄

【架构层 已经收敛的 7 件事】
1）Layer Norm 挪出残差流（pre-norm）
原版 Transformer 把 LN 放在残差里 几乎所有现代模型都挪到外面
原因：keep your residual stream clean 梯度反传更稳

2）RMS Norm 替代 LayerNorm
LayerNorm 的减均值 + 加 bias 那部分实际没怎么帮上忙
丢掉之后 flops 只省 0.17% 但运行时省到 25%
（瓶颈在数据搬运 计算反而次要）

3）所有 bias 项全删
跟 RMS Norm 一个道理 系统层省内存搬运

4）激活函数用 SwiGLU 或 GeGLU
gated linear unit 几乎所有现代模型都用
Llama 系 / Qwen / Mistral 用 SwiGLU
Google 系（Gemma / T5）用 GeGLU
区别极小 选哪个都行

5）位置编码用 RoPE
2024 年之后基本统一了
原理：把每对维度按位置旋转一个角度 让 inner product 只依赖相对位置

6）Transformer block 串联（不是并联）
GPT-J / Palm 试过并联 现在基本被放弃
串联的实现优化得太好了 并联省的那点系统开销不值得损失表达力

7）Layer norm 可以「撒」
哪儿不稳就在哪儿加 LN
attention 之前能加 之后能加 两边都加（double norm）也可以
现代模型很多这样做

【超参数 已经收敛的 5 个数】
1）feedforward 维度 / hidden 维度
- 非 GLU 模型：4 倍
- GLU 模型：8/3 ≈ 2.67 倍（因为 GLU 多一组矩阵 要保持总参数量）
- Llama 系：3.5 倍
- T5 1.0 试过 64 倍 后来 T5 1.1 改回标准 别学

2）head 数 × head 维度 ≈ hidden 维度
几乎所有模型都遵守 T5 是为数不多的例外

3）模型纵横比（hidden / 层数）≈ 100
太深 pipeline parallel 难做
太宽 表达力受限
100 这个数字是系统约束 + 表达力的平衡点

4）vocab size
单语模型：30K 左右（早期 GPT-2 那种）
多语 / 通用模型：100K-200K（GPT-4 / Llama 3 / Gemma 都在这个范围）
现代基本都是后者

5）weight decay
仍然普遍使用
但研究发现它在 LLM 里干的事其实是优化器干预 让你最终能收敛到更深的最优点
跟你想的「防过拟合」没什么关系
所以别因为「单 epoch 不会过拟合」就把它关掉

【稳定性 三个救命 trick】
训练大模型最怕中途 loss 突然飙升 然后 NaN 全军覆没
现代模型用三个 trick 防这件事

1）Z-loss
output softmax 的 normalizer 容易爆
加一个 (log Z)² 的正则项 让 Z 始终接近 1
DCLM / Olmo 都用

2）QK norm
attention 的 Q 和 K 在矩阵乘之前各加一个 LN
让 softmax 的输入永远是单位尺度
multimodal 圈先用起来 现在所有大模型都加

3）Logit soft cap（仅 Google 系）
attention logit 用 tanh 硬封顶
Gemma 2/3/4 都在用 但会损失一点点性能 慎用

【Attention 两个新趋势】
1）GQA（Grouped Query Attention）几乎统一
原版 multi-head 推理时 KV cache 会让算术强度崩到 1/h
GQA 共享 K 和 V 但保留多个 Q
表达力几乎不损失 推理成本砍掉 80%
现在所有要做生产部署的大模型 没有不用 GQA 的

2）局部 + 全局 attention 交替
处理长上下文的新方式
Cohere Command A 起头 现在 Llama 4 / Gemma 4 / Olmo 3 全在用
比如每 4 层有 1 层 full attention 其他 3 层是 sliding window 只看附近的 token
比纯 SSM 更稳 比纯 full attention 便宜得多
（Qwen 3.5 做了变体 把 sliding window 那 3 层换成 SSM）

收尾一句
如果你正在训自己的 LLM，上面这一套就是 2026 年的「默认配置」 不需要重新发明，直接抄

如果你只是想看懂 GitHub 上那些 modeling_xxx.py
这一份足够你不再被术语吓住

[Zhouhan Lin @zhouhan_lin]

@caglarml @Yoshua_Bengio @ODET_UN @UNESCO @ITU Ah, sorry about that! Hope all is well with you too. We should definitely catch up. Are you going to ICML this year?

[Zhouhan Lin @zhouhan_lin]

Sick of LLMs over-optimizing for a single reasoning path? 📉

Meet FlowRL at #ICLR2026! We use flow balancing to capture the full reward distribution, ensuring more diverse and robust reasoning. 🌈It moves beyond the reward maximization objective of PPO/GRPO, and introduces a new objective of reward distribution matching, to promote diverse & generalizable reasoning paths via flow balancing. 🌊

📍 Come chat at our ICLR 2026 poster!
⏰ Apr 24, 11:15 AM – 1:45 PM
🏢 Pavilion 4, P4-#4607
📄 Paper: https://t.co/tn2BrXrUTu
💻 Code: https://t.co/Xtvx9Gxg2K
#ICLR2026 #LLM #ReinforcementLearning #FlowRL @hongyuan_mei @zhu_xuekai

[Zhouhan Lin @zhouhan_lin]

🚨 Is that "new" model actually yours? 🕵️‍♂️

Stop by our ICLR poster to see AWM (Accurate Weight-Matrix Fingerprint) in action. We’ve built a high-fidelity metric to verify LLM lineage—even after intensive post-training or structural changes.
✅ No more worrying about model "washing" or fine-tuning obfuscation.
✅ Robust against pruning, scaling & permutations via LAP + Unbiased CKA.
✅ 100% detection rate.

📍 Pavilion 4, P4-#4109
⏰ Apr 24, 9:30 PM - 12:00 AM CST
🔗 Paper: https://t.co/87CCjIbVIt
🔗 Code: https://t.co/JMNfcgeIqn
See you there! 👋 #ICLR2026 #LLM #IPProtection

[Zhouhan Lin @zhouhan_lin]

🚨 Is that "new" model actually yours? 🕵️‍♂️

Stop by our ICLR poster to see AWM (Accurate Weight-Matrix Fingerprint) in action. We’ve built a high-fidelity metric to verify LLM lineage—even after intensive post-training or structural changes.
✅ No more worrying about model "washing" or fine-tuning obfuscation.
✅ Robust against pruning, scaling & permutations via LAP + Unbiased CKA.
✅ 100% detection rate.

📍 Pavilion 4, P4-#4109
⏰ Apr 24, 9:30 PM - 12:00 AM CST
🔗 Paper: https://t.co/87CCjIbVIt
🔗 Code: https://t.co/JMNfcgeIqn
See you there! 👋 #ICLR2026 #LLM #IPProtection

[Zhouhan Lin @zhouhan_lin]

Presenting our #ICLR2026 work on Apr. 23: "MLP Memory: A Retriever-Pretrained Memory for LLMs." We are trying to solve the Long-term Memory challenge by shifting from external document access to a differentiable parametric module.🧠✨
✅ True Parametric Memory: An external neural network pre-trained to imitate retriever patterns.
✅ Unlimited Scaling: Handles large-scale, long-term knowledge that context windows can't reach. Theoretically, we could memorize the whole pretraining set into the external memory!
✅ Seamless Integration: Zero-latency retrieval for deep knowledge synthesis.
Stop by to see how we’re giving LLMs a permanent, scalable memory!
📍 Pavilion 4 #P3-135
📅 Thu, Apr 23 | 3:15 PM – 5:45 PM
🔗 Paper: https://t.co/iKJMiMdGUE
🔗 Code: https://t.co/c6vWYoQskC

[Zhouhan Lin @zhouhan_lin]

🚀 Excited to present our latest work #PonderLM at #ICLR2026!Why just scale parameters when you can scale Latent Thoughts? 🧠Inspired by the pondering process, PonderLM repeatedly invokes the forward pass in continuous space within a single token step—refining predictions without any human labels. Highlights:
✅ Efficiency: PonderPythia-2.8B rivals Pythia-12B with 4x fewer params.
✅ Data-Savvy: PonderPythia-1B matches TinyLlama-1.1B using 90% LESS training data.
✅ Flexible: Performance scales dynamically with inference-time pondering steps.
Come chat with us at the Poster Session!
📍 Location: Pavilion 3 Poster Location: P3-#717
📅 Time: April 23, 10.30 am-1.00 pm
🔗 Paper: https://t.co/hMSFgsJXLe
🔗 Code: https://t.co/8bb2Wb1esC
#LLMs #ICLR #MachineLearning @iclr_conf

[Zhouhan Lin @zhouhan_lin]

Standard LMs are confined to token-by-token generations and form a tree-structured trajectory. We introduce Flow of Spans (FoSS), a novel framework that generalizes LMs to generate text in dynamic spans via GFlowNets.

Key highlights:
✅ Modeling generation trajectory as a DAG
✅ Flexible span sampling for better expressiveness
✅ Superior performance in retrieval-augmented tasks
📍 Come chat with us at ICLR 2026 Poster Session!
📅 Thu, Apr 23, 2026 | ⏰ 3:30 PM – 6:00 PM CEST | 📌 Pavilion 3 Poster Location: P3-#705
#ICLR2026 #LLM #GFlowNets #DeepLearning @iclr_conf

[Zhouhan Lin @zhouhan_lin]

Looks like this recent paper is closely related to what you described: https://t.co/QfYvbiMivh . The model learns from its own experiences without updating its internal weights or using gradients. It introspects its actions, identifies successful strategies, and stores them as "semantic advantage" in natural language.

[Zhouhan Lin @zhouhan_lin]

@Yuchenj_UW Not sure why Elon proposed this. It'd be very easy to think of some tasks in coding that LLMs fail. What Andrej said in the interview is just one of them. Grok-5 is definitely going to lose if such a contest happens.

[Zhouhan Lin @zhouhan_lin]

@haider1 Really True! AI agents have mastered the low-hanging fruit in various applications. The hard, unsolved problems in those applications won't be fixed with more tweaks. To solve them, you'll need to go all the way down to fundamental model architectures and pretraining tasks!

[Zhouhan Lin @zhouhan_lin]

Really True! AI agents have mastered the low-hanging fruit in various applications. The hard, unsolved problems in those applications won't be fixed with more tweaks. To solve them, you'll need to go all the way down to fundamental model architectures and pretraining tasks!

[Haider. @haider1]

Andrej Karpathy says today's agents aren't ready to work like real coworkers or interns

They lack intelligence, can't use computers, aren't multimodal, lack continual learning, and forget what you tell them

Fixing these gaps will take about a decade

[Zhouhan Lin @zhouhan_lin]

And fight with reviewer 2 to explain why this simple method is worth publication.

[Xin Eric Wang @xwang_lk]

Unpopular opinion: Finding a simple idea that actually works is way harder than publishing a fancy one that kinda works. You have to fight the urge to overcomplicate, give up many fancier ideas, fail and pivot again and again until you hit the first principle that truly holds.