..

Hi!

[Sangmin Bae @raymin0223]

Nice to see this idea getting attention!
This is essentially what we called "depth-wise batching": overlapping requests at different recurrent depths (much higher utilization with adaptive compute).

Recursive Models: https://t.co/abCjYgcfGx
MoR: https://t.co/q2BGHi6f77

[Max For AI @MaxForAI]

🔥太疯狂了，现在你甚至可以直播围观一个 535B 大模型是怎么被训练出来的。

斯坦福教授兼 @simile_ai 创始人 @percyliang 刚刚宣布，他们的新模型 Marin 535B-A23B 本周正式开训，而且整个训练过程公开。

Marin 共有 535B 总参数，其中 23B 激活参数。

为此Percy他们一共准备了18.75T tokens的数据，动用了 11 套 GB200 NVL72，约 792 张 GB200显卡。

预计连续训练 3 个月，总训练计算量约 2.7e24 FLOPs。

训完还会继续做 post-training。

请注意，Marin 真的全部开源了整个训练过程。

正式开跑前，他们先训练了一套 4 级 Scaling Ladder，从 1.6B-A61M 一路跑到 27.7B-A1.2B，用小模型提前预测这次 535B 主模型的 loss 和训练状态。

现在主模型正式开跑，所有人都可以直接围观实时训练曲线，后面的数据、实验记录、工程问题也会持续公开。

类似的事情以前有人做过。

2022 年 BigScience 训练 176B 的 BLOOM 时，就公开过训练进度和 TensorBoard。

但 Marin 这次直接把公开训练的规模推到了 535B。

目前来看，这应该是人类公开围观过的最大规模大模型训练之一。

接下来三个月应该会很好看。

模型最后能不能打是一回事，我更想看的是一个 500B+ 模型的MoE到底会怎么训、怎么调，以及中间到底会炸几次。

这种机会太少见了，等于把一场原本只有顶级 AI Lab 内部才能看到、成本可能高达数千万美元的训练实验，直接摊在全世界面前。

[waterloo intern @waterloo_intern]

https://t.co/aivc7B4A7Z

[BSPK @BSPK_]

LLM이 뭘 잘하는지 생각하면 당연한 일이다. LLM은 언어세계를 기반으로 하고, 대부분의 인간들 보다 언어를 더 능숙하게 다룬다. 특히 창의적 글쓰기 보다는 정리와 요약을 잘한다.

그러니 내가 정리하는 것보다 내 아이디어를 더 잘 이해하고 정리할 수 있다. (많은 사람들이 혼자 고민하면서는 정리가 잘 안되던 생각이 대화를 통해 명확해지는 경험을 가지고 있을 것이다.)

프롬프트 엔지니어링 페러다임도 바뀔 것이다. 인간의 커뮤니케이션을 생각해보면 앞으로 프롬프트를 잘 설계하기 위해 노력하기 보다는, 회의에 에이전트가 함께 참여하거나, 긴 시간 토론을 하는 방식이 사용될 가능성이 더 크다.
---------------------------------------------------
LLM과 작업할 때 유용하다고 느끼는 패턴 하나는, 길게 늘어놓는 '횡설수설 세션'이다. 가끔은 내가 뭘 하려는지 LLM이 이해하려면 정보가 더 필요한데, 그걸 다 타이핑하기는 귀찮을 때가 있다. 그럴 때 나는 등을 기대고 앉아 /voice로 전환한 다음, 그냥 10분쯤 떠든다. 완전히 뒤죽박죽, 뭐든 상관없이, 순수한 의식의 흐름으로. 어떨 땐 시작할 때 미리 선언해두기도 한다. "음성 인식으로 전환합니다, 오타가 있으면 미안해요..." 같은 식으로. 또 어떨 땐 몇 번 주고받는 짧은 인터뷰 형식으로 만들기도 한다.

그런데 내가 발견한 건, LLM이 길고 앞뒤 없는 이 횡설수설을 재구성하는 데 어쩐지 아주 능하다는 것이다. 그리고 뒤엉킨 내 생각을 되돌려주는 그 결과물이 애초에 내가 시작했던 것보다 훨씬 깔끔하게 나오는 경우가 많다. 결과적으로 '마인드 멜드(생각의 합치)'가 개선되고, 그 시점 이후로는 바로잡아야 할 일이 줄어든다.

[Bhavin Jawade @BhavinJawade]

𝗢𝗻-𝗽𝗼𝗹𝗶𝗰𝘆 𝗱𝗶𝘀𝘁𝗶𝗹𝗹𝗮𝘁𝗶𝗼𝗻 𝗶𝘀𝗻'𝘁 𝗮 𝗳𝗿𝗲𝗲-𝗹𝘂𝗻𝗰𝗵

On-policy distillation has become a default post-training tool in many open-source frontier model training recipes. Recent releases lean on it heavily: DeepSeek v4, MiMO, and Nemotron-Cascade-2 use MOPD, and GLM 5.x uses on-policy cross-stage self-distillation. It provides RL's on-policy nature reducing exposure bias, while providing token level supervision like SFT.
But OPD and OPSD have their own failure modes. In this post I discuss a few of them:

1. 𝗘𝗮𝗿𝗹𝘆 𝗺𝗶𝘀𝘁𝗮𝗸𝗲𝘀 𝗮𝗿𝗲 𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗮𝗹𝗹𝘆 𝘂𝗻𝗰𝗼𝗿𝗿𝗲𝗰𝘁𝗮𝗯𝗹𝗲. When the student samples a rollout and takes an early wrong turn, the per-token KL computed along that frozen rollout cannot pull it back onto a correct path. TRD proves that this failure is built into the objective rather than being a matter of noisy gradients. Even with a perfect teacher, the gradient obtained from token-level KL on the student's own rollout agrees with the ideal corrective gradient at exactly one point, the token where the student first diverged, and disagrees everywhere after it. Every later supervision target is therefore anchored to a context that the student should never have entered. Because reweighting or clipping only rescales the magnitude of each token's gradient, and here the terms point in the wrong direction, no per-token adjustment can recover the correct update. TRD's proposed fix is to distill along a teacher-refined trajectory rather than the raw student rollout, which restores a target the student can actually follow.

2. 𝗔 𝘀𝘁𝗿𝗼𝗻𝗴𝗲𝗿 𝘁𝗲𝗮𝗰𝗵𝗲𝗿 𝗰𝗮𝗻 𝗯𝗲 𝗮 𝘄𝗼𝗿𝘀𝗲 𝘁𝗲𝗮𝗰𝗵𝗲𝗿. On-policy distillation can only teach the student at states the student itself visits, and the usable signal at each of those states lives in the overlap between the student's and teacher's next-token distributions. Rethinking OPD shows that a bigger, higher-scoring teacher can fail to move a student while a weaker one succeeds, because if the teacher's token distribution places its mass on tokens the student rarely produces, the overlap is small and almost nothing transfers, no matter how capable the teacher is in absolute terms. What actually predicts success is early top-k thinking-pattern overlap. In runs that work, the shared top-k tokens carry 97 to 99% of the probability mass and the overlap ratio climbs steadily during training, whereas a run that starts with low overlap never recovers it. A teacher trained on the same recipe as the student also converges toward the student's own distribution, so its higher benchmark score does not correspond to any new knowledge it can transfer. The practical rule is to pick teachers by distributional closeness to the student, not by leaderboard rank.

3. 𝗣𝗿𝗶𝘃𝗶𝗹𝗲𝗴𝗲𝗱-𝗶𝗻𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻-𝗰𝗼𝗻𝗱𝗶𝘁𝗶𝗼𝗻𝗲𝗱 𝗢𝗣𝗦𝗗 𝗰𝗮𝗻 𝗳𝗮𝗶𝗹 𝘁𝗼 𝘁𝗿𝗮𝗻𝘀𝗳𝗲𝗿. In OPSD you distill a teacher that was conditioned on privileged information, such as the gold answer, into a student that will never have it. The Many Faces of OPD shows what goes wrong when that information is instance-specific. The student cannot recover the teacher's per-instance reasoning, since it never sees the answer, so it instead learns a single answer-free policy that effectively averages the teacher's behavior across all problems, and that averaged policy is too generic to solve any particular one. The signature is initial gains followed by collapse: rollouts grow long, fill with hedging tokens, and accuracy craters toward zero. The approach works only when the privileged information is a shared rule that applies across all instances, such as a system prompt or an alignment preference, and not when it is a per-problem answer.

4. 𝗧𝗵𝗶𝗻𝗸𝗶𝗻𝗴 𝗰𝗼𝗹𝗹𝗮𝗽𝘀𝗲: 𝗱𝗲𝗻𝘀𝗲 𝘀𝘂𝗽𝗲𝗿𝘃𝗶𝘀𝗶𝗼𝗻 𝘀𝘂𝗽𝗽𝗿𝗲𝘀𝘀𝗲𝘀 𝘁𝗵𝗲 𝗺𝗼𝗱𝗲𝗹'𝘀 𝗼𝘄𝗻 𝗱𝗲𝗹𝗶𝗯𝗲𝗿𝗮𝘁𝗶𝗼𝗻. A teacher conditioned on the answer has no reason to hesitate, backtrack, or explore, so its per-token targets quietly push down the student's deliberation tokens. Diagnosing and Mitigating Thinking Collapse names this phenomenon thinking collapse: over training, the student's native reasoning behavior erodes as the exploratory tokens that carry it, words like wait, maybe, and alternatively, become progressively less frequent. The mechanism is local rather than global. The damage concentrates at high-entropy decision forks, the branch points where the student is genuinely uncertain and would normally deliberate. Exactly there, the student's top-1 token is often an exploratory marker while the answer-conditioned teacher's top-1 token is not, so the mismatch produces a strong gradient that suppresses the very tokens that make reasoning work. The result is a model whose native reasoning behavior is measurably suppressed, and downstream reasoning accuracy falls in step with it.

[licheng @liulicheng10]

probably the best blog i have read for some time

viewing SFT, RL, and OPD as different ways of reshaping a model's distribution makes their tradeoffs super intuitive.

- SFT pulls toward a fixed external target
- RL moves along the reward gradient on on-policy samples
- OPD sits in between, using a teacher signal but on student-generated data, which is why it inherits RL's anti-forgetting properties even when the teacher itself was an overtrained SFT model.

the post is heavily grounded in recent literature and uses the distributional perspective as a unifying bridge across all three paradigms, i really like the point it argues the load-bearing ingredient is on-policy data and OPD's convergence to RL-like outcomes is the strongest evidence

[Seth Karten @sethkarten]

We've solved another piece of the generalist gaming agent puzzle!

Mario requires different skills than Pokemon: reactive navigation, spatial reasoning, and safe exploration rather than long-term memory and zero-sum reasoning.

Check out our new paper on finetuning VLMs to beat Mario via PPO. 👇

[Jason Zhu @GoSailGlobal]

Stanford CS336 上，Tatsu 讲了一节 LLM 架构课，把过去 3 年所有主流 LLM 拆开，看它们的共通模板

结论挺爆：90% 的架构选择已经收敛，你随便挑一个开源大模型，它跟其他模型在这些维度上几乎一模一样

讲师的原话
- 2024 年大家都在 cosplay Llama2
- 2025 年的主题是「怎么训得不崩」
- 2026 年的主题是「怎么扛住长上下文」

下面是 2026 年开源 LLM 的标准模板 你训自己的模型可以直接抄

【架构层 已经收敛的 7 件事】
1）Layer Norm 挪出残差流（pre-norm）
原版 Transformer 把 LN 放在残差里 几乎所有现代模型都挪到外面
原因：keep your residual stream clean 梯度反传更稳

2）RMS Norm 替代 LayerNorm
LayerNorm 的减均值 + 加 bias 那部分实际没怎么帮上忙
丢掉之后 flops 只省 0.17% 但运行时省到 25%
（瓶颈在数据搬运 计算反而次要）

3）所有 bias 项全删
跟 RMS Norm 一个道理 系统层省内存搬运

4）激活函数用 SwiGLU 或 GeGLU
gated linear unit 几乎所有现代模型都用
Llama 系 / Qwen / Mistral 用 SwiGLU
Google 系（Gemma / T5）用 GeGLU
区别极小 选哪个都行

5）位置编码用 RoPE
2024 年之后基本统一了
原理：把每对维度按位置旋转一个角度 让 inner product 只依赖相对位置

6）Transformer block 串联（不是并联）
GPT-J / Palm 试过并联 现在基本被放弃
串联的实现优化得太好了 并联省的那点系统开销不值得损失表达力

7）Layer norm 可以「撒」
哪儿不稳就在哪儿加 LN
attention 之前能加 之后能加 两边都加（double norm）也可以
现代模型很多这样做

【超参数 已经收敛的 5 个数】
1）feedforward 维度 / hidden 维度
- 非 GLU 模型：4 倍
- GLU 模型：8/3 ≈ 2.67 倍（因为 GLU 多一组矩阵 要保持总参数量）
- Llama 系：3.5 倍
- T5 1.0 试过 64 倍 后来 T5 1.1 改回标准 别学

2）head 数 × head 维度 ≈ hidden 维度
几乎所有模型都遵守 T5 是为数不多的例外

3）模型纵横比（hidden / 层数）≈ 100
太深 pipeline parallel 难做
太宽 表达力受限
100 这个数字是系统约束 + 表达力的平衡点

4）vocab size
单语模型：30K 左右（早期 GPT-2 那种）
多语 / 通用模型：100K-200K（GPT-4 / Llama 3 / Gemma 都在这个范围）
现代基本都是后者

5）weight decay
仍然普遍使用
但研究发现它在 LLM 里干的事其实是优化器干预 让你最终能收敛到更深的最优点
跟你想的「防过拟合」没什么关系
所以别因为「单 epoch 不会过拟合」就把它关掉

【稳定性 三个救命 trick】
训练大模型最怕中途 loss 突然飙升 然后 NaN 全军覆没
现代模型用三个 trick 防这件事

1）Z-loss
output softmax 的 normalizer 容易爆
加一个 (log Z)² 的正则项 让 Z 始终接近 1
DCLM / Olmo 都用

2）QK norm
attention 的 Q 和 K 在矩阵乘之前各加一个 LN
让 softmax 的输入永远是单位尺度
multimodal 圈先用起来 现在所有大模型都加

3）Logit soft cap（仅 Google 系）
attention logit 用 tanh 硬封顶
Gemma 2/3/4 都在用 但会损失一点点性能 慎用

【Attention 两个新趋势】
1）GQA（Grouped Query Attention）几乎统一
原版 multi-head 推理时 KV cache 会让算术强度崩到 1/h
GQA 共享 K 和 V 但保留多个 Q
表达力几乎不损失 推理成本砍掉 80%
现在所有要做生产部署的大模型 没有不用 GQA 的

2）局部 + 全局 attention 交替
处理长上下文的新方式
Cohere Command A 起头 现在 Llama 4 / Gemma 4 / Olmo 3 全在用
比如每 4 层有 1 层 full attention 其他 3 层是 sliding window 只看附近的 token
比纯 SSM 更稳 比纯 full attention 便宜得多
（Qwen 3.5 做了变体 把 sliding window 那 3 层换成 SSM）

收尾一句
如果你正在训自己的 LLM，上面这一套就是 2026 年的「默认配置」 不需要重新发明，直接抄

如果你只是想看懂 GitHub 上那些 modeling_xxx.py
这一份足够你不再被术语吓住

[Ordine Vecchio @gramsci_destra]

대통령을 사칭하는 자가
대놓고 거짓말을 하고 있네
1주택장특공제=40%보유공제+40%거주공제
로 설계되어있고
거주를 요건으로 적용되는 감면제도는
현행세법에 존재하지 않음
그리고 장특공제를 폐지하면
실제로 세금 폭탄임
원래 장특공제는
애초에 거주와는 상관없이 설계된 제도임
오랜기간 자산가치 상승에서 인플레이션 효과를 제거해주는 제도일뿐.
비슷한 세법적 장치가 퇴직소득세에도 있음.
뭘 알지도 못하면서 떠들어 대는건 지지자들이 그 수준이라서...

[Yulu Gan @yule_gan]

Simply adding Gaussian noise to LLMs (one step—no iterations, no learning rate, no gradients) and ensembling them can achieve performance comparable to or even better than standard GRPO/PPO on math reasoning, coding, writing, and chemistry tasks. We call this algorithm RandOpt.

To verify that this is not limited to specific models, we tested it on Qwen, Llama, OLMo3, and VLMs.

What's behind this? We find that in the Gaussian search neighborhood around pretrained LLMs, diverse task experts are densely distributed — a regime we term Neural Thickets.

Paper: https://t.co/rFJz2kVEOA
Code: https://t.co/HAmonfpXIA
Website: https://t.co/QZ6AMIsKCw

[Yu Zhang 🐙🌘 @yzhang_cs]

The idea of rotating attention by 90° is sooooooo cool (credits to @Jianlin_S 's insights), and it surprisingly works.
We (w/ the amazing @nathan) are so excited about this— been working on the paper for months and couldn't stop.

Go give it a try. It's a drop-in replacement for standard residuals, born in 2015.

really like the figs btw :-)

[Amil Dravid @ COLM2026 @_AmilDravid]

I recently gave a talk at the AI@MIT reading group on our NeurIPS 2025 paper: https://t.co/cXDuIDkkks

We identify the neural mechanism behind attention sinks and propose a training-free mitigation.

Video: https://t.co/TrprZSRgY6
Slides: https://t.co/3L1Mud9c2M

[Nathan Godey @nthngdy]

🧵New paper: "Lost in Backpropagation: The LM Head is a Gradient Bottleneck"
The output layer of LLMs destroys 95-99% of your training signal during backpropagation, and this significantly slows down pretraining 👇

[Tri Dao @tri_dao]

FlashAttention is widely used to accelerate Transformers, already making attention 4-8x faster, but has yet to take advantage of modern GPUs. We’re releasing FlashAttention-3: 1.5-2x faster on FP16, up to 740 TFLOPS on H100 (75% util), and FP8 gets close to 1.2 PFLOPS!
1/

[Yining Hong @yining_hong]

Test-time errors aren't dead ends. They're training data — for both our world model and decision model.

Yet embodied systems are stuck with: Fixed world models; Repeated mistakes; Zero growth.

We broke the cycle by introducing Reflective Test-Time Planning — embodied agents that reflect like human reflective practitioners.

🧠 Reflection-in-Action: We engage in internal simulation, questioning whether our planned approach will actually work given what we currently understand — before committing to any action.
🔄 Reflection-on-Action: We use actual outcomes to reshape both our beliefs about the environment and our strategies for acting within it — updating the world model and decision policy in real time.
⏪ Retro-Reflection: Rewind past decisions with hindsight — so long-horizon failures get the credit assignment they deserve.

🌐 https://t.co/CAtCYOIsiW
💻 https://t.co/GDSYrMagJG

[Sakana AI @SakanaAILabs]

We’re excited to introduce Doc-to-LoRA and Text-to-LoRA, two related research exploring how to make LLM customization faster and more accessible.

https://t.co/ApVzVsBuv1

By training a Hypernetwork to generate LoRA adapters on the fly, these methods allow models to instantly internalize new information or adapt to new tasks.

Biological systems naturally rely on two key cognitive abilities: durable long-term memory to store facts, and rapid adaptation to handle new tasks given limited sensory cues. While modern LLMs are highly capable, they still lack this flexibility. Traditionally, adding long-term memory or adapting an LLM to a specific downstream task requires an expensive and time-consuming model update, such as fine-tuning or context distillation, or relies on memory-intensive long prompts.

To bypass these limitations, our work focuses on the concept of cost amortization. We pay the meta-training cost once to train a hypernetwork capable of producing tasks or document specific LoRAs on demand. This turns what used to be a heavy engineering pipeline into a single, inexpensive forward pass. Instead of performing per-task optimization, the hypernetwork meta-learns update rules to instantly modify an LLM given a new task description or a long document.

In our experiments, Text-to-LoRA successfully specializes models to unseen tasks using just a natural language description. Building on this, Doc-to-LoRA is able to internalize factual documents. On a needle-in-a-haystack task, Doc-to-LoRA achieves near-perfect accuracy on instances five times longer than the base model's context window. It can even generalize to transfer visual information from a vision-language model into a text-only LLM, allowing it to classify images purely through internalized weights.

Importantly, both methods run with sub-second latency, enabling rapid experimentation while avoiding the overhead of traditional model updates. This approach is a step towards lowering the technical barriers of model customization, allowing end-users to specialize foundation models via simple text inputs. We have released our code and papers for the community to explore.

Doc-to-LoRA
Paper: https://t.co/87xEEpf0GN
Code: https://t.co/zBfQi2L9LW

Text-to-LoRA
Paper: https://t.co/emLRZ4Vdvo
Code: https://t.co/b9mrdoWWRB

[Jorge Bravo Abad @bravo_abad]

An ADMM-based optimizer that outperforms state-of-the-art methods across vision models, LLMs, and GANs

Nearly every deep learning model trained today relies on some variant of SGD. Adam, AdamW, Muon, Shampoo—they differ in how they estimate gradient moments or precondition updates, but they share the same theoretical baggage: bounded gradient assumptions, bounded variance, unbiased gradient estimation, and the expectation that training data are IID. When data are heterogeneous—the norm in federated and distributed settings—these assumptions break down and convergence guarantees evaporate.

Shenglong Zhou and coauthors take a fundamentally different path. They build PISA, a preconditioned inexact stochastic ADMM framework that decomposes the training objective into parallelizable subproblems linked through Lagrange multipliers, then solves each inexactly using stochastic gradients and adaptive preconditioning matrices. PISA converges at a linear rate under a single assumption—Lipschitz continuity of the gradient on a bounded region—without requiring bounded variance, bounded gradients, or IID data. Among all stochastic optimizers surveyed, only PISA achieves this combination.

The framework supports pluggable preconditioners, yielding two practical variants: SISA (second-moment preconditioning, analogous to Adam-style adaptive rates) and NSISA (Newton–Schulz orthogonalized momentum, inspired by Muon). Both retain the same convergence guarantees.

The empirical breadth is notable. Under extreme label skew in federated learning (each client holding one class), SISA reaches 95% on MNIST where FedAvg, FedProx, and Scaffold plateau around 54%. On CIFAR-10, SISA matches or exceeds ten optimizers across ResNet-34, VGG-11, and DenseNet-121. For LLM training, NSISA opens an increasing validation loss gap over Adam, Muon, Shampoo, SOAP, and Adam-mini as model size grows from GPT2-Nano to GPT2-XL, with clear wall-clock advantages at the largest scale. On GAN training—notoriously unstable—SISA achieves the lowest FID scores on both WGAN and WGAN-GP.

What makes this compelling beyond any single benchmark is the theoretical clarity. Most optimizer comparisons are purely empirical; here the convergence theory explains why the algorithm handles heterogeneous data gracefully—the ADMM decomposition naturally accommodates non-IID batches without variance reduction over full datasets. The optimization framework itself, not just the learning rate schedule or momentum scheme, can be a decisive design choice for robust training across architectures and data distributions.

Paper: https://t.co/5vTe7kufSR

[Alex Weers @a_weers]

Finished reading, here is the summary:

SFT before RL usually improves performance or at least saves time/compute. However, when SFT isn't the last stage, measuring a "good" or "successful" SFT becomes non-obvious.

Instead of optimizing for best performance on downstream tasks, SFT should prepare the model for the subsequent RL stage.
This paper shows that strong SFT performance is indeed a bad indicator for success in the subsequent RL stage, with stronger SFT checkpoints often underperforming weaker ones after RL.

The authors introduce a simple weighting scheme during SFT based on the similarity between the behavior policy (the one that generated the SFT data) and the model policy.
The key insight: if the continuation after a token is implausible under the current model, learning from that token provides little useful signal, since RL will sample from its own distribution and rarely visit those trajectories anyway. PEAR down-weights such tokens.

This weighting can operate at the sequence, block, or token level, trading granularity for stability.

They successfully demonstrate that this weighting helps. Interestingly, their experiments use a Qwen model for SFT data generation while training other Qwen models. I was surprised by this, since I expected those policies to already be well aligned.

The only drawback is that the method requires access to the SFT data generation policy. This is only available for synthetic data, and even then, it incurs a non-negligible cost.

However, I like the idea and think optimizing SFT with respect to subsequent RL stages is a good motivation.

Great paper by @dylan_works_ et al.!

[Percy Liang @percyliang]

Don't use weight decay.
Just normalize everything you see (updates + parameters).
Works on top of your favorite optimizer (e.g., Muon).
Result: 33% speedup + better hyperparameter transfer.