Carlos M.

[Bhavin Jawade @BhavinJawade]

𝗢𝗻-𝗽𝗼𝗹𝗶𝗰𝘆 𝗱𝗶𝘀𝘁𝗶𝗹𝗹𝗮𝘁𝗶𝗼𝗻 𝗶𝘀𝗻'𝘁 𝗮 𝗳𝗿𝗲𝗲-𝗹𝘂𝗻𝗰𝗵

On-policy distillation has become a default post-training tool in many open-source frontier model training recipes. Recent releases lean on it heavily: DeepSeek v4, MiMO, and Nemotron-Cascade-2 use MOPD, and GLM 5.x uses on-policy cross-stage self-distillation. It provides RL's on-policy nature reducing exposure bias, while providing token level supervision like SFT.
But OPD and OPSD have their own failure modes. In this post I discuss a few of them:

1. 𝗘𝗮𝗿𝗹𝘆 𝗺𝗶𝘀𝘁𝗮𝗸𝗲𝘀 𝗮𝗿𝗲 𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗮𝗹𝗹𝘆 𝘂𝗻𝗰𝗼𝗿𝗿𝗲𝗰𝘁𝗮𝗯𝗹𝗲. When the student samples a rollout and takes an early wrong turn, the per-token KL computed along that frozen rollout cannot pull it back onto a correct path. TRD proves that this failure is built into the objective rather than being a matter of noisy gradients. Even with a perfect teacher, the gradient obtained from token-level KL on the student's own rollout agrees with the ideal corrective gradient at exactly one point, the token where the student first diverged, and disagrees everywhere after it. Every later supervision target is therefore anchored to a context that the student should never have entered. Because reweighting or clipping only rescales the magnitude of each token's gradient, and here the terms point in the wrong direction, no per-token adjustment can recover the correct update. TRD's proposed fix is to distill along a teacher-refined trajectory rather than the raw student rollout, which restores a target the student can actually follow.

2. 𝗔 𝘀𝘁𝗿𝗼𝗻𝗴𝗲𝗿 𝘁𝗲𝗮𝗰𝗵𝗲𝗿 𝗰𝗮𝗻 𝗯𝗲 𝗮 𝘄𝗼𝗿𝘀𝗲 𝘁𝗲𝗮𝗰𝗵𝗲𝗿. On-policy distillation can only teach the student at states the student itself visits, and the usable signal at each of those states lives in the overlap between the student's and teacher's next-token distributions. Rethinking OPD shows that a bigger, higher-scoring teacher can fail to move a student while a weaker one succeeds, because if the teacher's token distribution places its mass on tokens the student rarely produces, the overlap is small and almost nothing transfers, no matter how capable the teacher is in absolute terms. What actually predicts success is early top-k thinking-pattern overlap. In runs that work, the shared top-k tokens carry 97 to 99% of the probability mass and the overlap ratio climbs steadily during training, whereas a run that starts with low overlap never recovers it. A teacher trained on the same recipe as the student also converges toward the student's own distribution, so its higher benchmark score does not correspond to any new knowledge it can transfer. The practical rule is to pick teachers by distributional closeness to the student, not by leaderboard rank.

3. 𝗣𝗿𝗶𝘃𝗶𝗹𝗲𝗴𝗲𝗱-𝗶𝗻𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻-𝗰𝗼𝗻𝗱𝗶𝘁𝗶𝗼𝗻𝗲𝗱 𝗢𝗣𝗦𝗗 𝗰𝗮𝗻 𝗳𝗮𝗶𝗹 𝘁𝗼 𝘁𝗿𝗮𝗻𝘀𝗳𝗲𝗿. In OPSD you distill a teacher that was conditioned on privileged information, such as the gold answer, into a student that will never have it. The Many Faces of OPD shows what goes wrong when that information is instance-specific. The student cannot recover the teacher's per-instance reasoning, since it never sees the answer, so it instead learns a single answer-free policy that effectively averages the teacher's behavior across all problems, and that averaged policy is too generic to solve any particular one. The signature is initial gains followed by collapse: rollouts grow long, fill with hedging tokens, and accuracy craters toward zero. The approach works only when the privileged information is a shared rule that applies across all instances, such as a system prompt or an alignment preference, and not when it is a per-problem answer.

4. 𝗧𝗵𝗶𝗻𝗸𝗶𝗻𝗴 𝗰𝗼𝗹𝗹𝗮𝗽𝘀𝗲: 𝗱𝗲𝗻𝘀𝗲 𝘀𝘂𝗽𝗲𝗿𝘃𝗶𝘀𝗶𝗼𝗻 𝘀𝘂𝗽𝗽𝗿𝗲𝘀𝘀𝗲𝘀 𝘁𝗵𝗲 𝗺𝗼𝗱𝗲𝗹'𝘀 𝗼𝘄𝗻 𝗱𝗲𝗹𝗶𝗯𝗲𝗿𝗮𝘁𝗶𝗼𝗻. A teacher conditioned on the answer has no reason to hesitate, backtrack, or explore, so its per-token targets quietly push down the student's deliberation tokens. Diagnosing and Mitigating Thinking Collapse names this phenomenon thinking collapse: over training, the student's native reasoning behavior erodes as the exploratory tokens that carry it, words like wait, maybe, and alternatively, become progressively less frequent. The mechanism is local rather than global. The damage concentrates at high-entropy decision forks, the branch points where the student is genuinely uncertain and would normally deliberate. Exactly there, the student's top-1 token is often an exploratory marker while the answer-conditioned teacher's top-1 token is not, so the mismatch produces a strong gradient that suppresses the very tokens that make reasoning work. The result is a model whose native reasoning behavior is measurably suppressed, and downstream reasoning accuracy falls in step with it.

[Iberia @Iberia]

¡Bienvenidos a casa, campeones! 🇪🇸🏆

Despegamos juntos con ilusión y volvemos con la Copa del Mundo. Gracias, @SEFutbol, por dejarnos ser parte de vuestra historia.

Un orgullo traeros de vuelta, equipo. ¡Ahora la celebración continúa en tierra!

[Demis Hassabis @demishassabis]

https://t.co/PTeDiv1b6L

[Phong Nguyen-Ha @PhongStormVN]

🚀 Excited to share that our paper "Cross-Space Distillation: Teaching One-Step Students with Modern Diffusion Teachers" has been accepted to ECCV 2026! 🎉

Modern one-step diffusion models are fast and efficient, but existing distillation methods assume that the teacher and student share the same latent space. In practice, this prevents powerful models such as SD 3.5 and FLUX from directly transferring their knowledge to compact and deployment-friendly models like SD 1.5, which operate in different latent spaces and resolutions.

In this work, we introduce Cross-Space Distillation, a new setting where teacher and student differ in both latent resolution and VAE representation. To bridge this gap, we propose a lightweight module called Bridge, which maps student latents into the teacher space while keeping the student backbone unchanged.

✅ Distill knowledge from heterogeneous modern diffusion teachers
✅ Preserve one-step generation speed and low latency
✅ Maintain compatibility with the extensive SD 1.5 ecosystem
✅ Significantly improve generation quality (e.g., HPSv3 improves from 5.4 → 9.4)

Our results suggest that high-capacity diffusion models can effectively teach compact generators even when their latent spaces are fundamentally different, opening a new path toward efficient and deployable generative AI.

Huge thanks to all collaborators @anhttran_usc @ngducminhkhoi at Qualcomm AI Research Viet Nam and others for making this work possible!

📄 Paper: https://t.co/3PTsFvBRqP

#ECCV2026 #ComputerVision #GenerativeAI #DiffusionModels #StableDiffusion #ModelCompression #EfficientAI #Research #MachineLearning

[SAIR @SAIRfoundation]

Our co-founder Terence Tao spoke at UCR on what’s next for SAIR Competitions.

He discussed how SAIR Competitions can help build open, cumulative, and verifiable progress in mathematics and science.

Watch the talk:

[Kirk Borne @KirkDBorne]

MIT's Introduction to Computational Thinking and Data Science — course slides and videos

HT @Riazi_Cafe_en

[Google Research @GoogleResearch]

Today we present a study on how reasoning unlocks parametric knowledge in LLMs. We identify two key driving mechanisms, a computational buffer effect and factual priming, and suggest ways that can help build more reliable models. Learn more: https://t.co/CjIKqyoG4N

[ar0cket1 @ar0cket1]

I am almost certain that if you mix GRPO with positive pressure OPSD (I was able to generate hints very easily, by just passing a problem and solution through a model to create a hint for each problem but a human and the RL environment companies can probably do this as well) and you get a massively better RL algorithm.

I suspect positive OPSD works as well, but mixing is negligible risk.

experiments will be run later this week (can't run them at this moment in time unfortunately)

[Jeonghye Kim @beanie0__0]

Great blog post, @ar0cket1. I enjoyed discussing RLRT with you.

One question kept circling in my head while reading. How do we post-train an LLM agent that keeps improving, and what signal can drive that improvement?

(I tried to organize my thoughts by connecting this question to exploration in the RL literature. My first paper, LESSON@ICML23, was about exploration in classical RL. It has been especially fun to revisit OP(S)D from this perspective.)

What we ultimately want is an agent that keeps improving, what @ar0cket1 called "RL like infinite upper bound."

RL offers a useful lens for thinking about this. Sustained improvement requires a balance between exploration and exploitation. An agent should explore broadly at first, then gradually focus on the most promising directions. Too little exploration can lead to a suboptimal solution, while too much can prevent the agent from reinforcing what already works.

From this perspective, the blog post and RLRT emphasize different engines of improvement.
- “Preserving only positive” emphasizes improvement through stable exploitation. If negative teacher-student disagreement contains little task-relevant information, suppressing it can stabilize learning.
- RLRT emphasizes reward-gated exploration. It gives greater weight to deviations from the privileged teacher when they lead to verified success.

Ultimately, both exploration and exploitation are necessary. Stable exploitation helps preserve and reinforce what already works, while exploration is needed to discover better solutions and push the frontier. The important question is: How do we distinguish productive exploration from unproductive deviation?

We usually cannot know in advance whether an exploratory decision will actually be useful. Therefore, it is crucial to develop effective exploration proxies. We rely on signals such as uncertainty, optimism, novelty, information gain, structural consistency, and outcome verification. Each captures only part of what we care about, so progress often comes from combining signals with complementary failure modes.

Prior work on RLVR exploration has largely focused on token-level entropy regulation and sequence-level diversity.
RLRT opens up a new axis of exploration based on privileged-information gaps: deviations from PI, filtered by verifiable outcomes.

At the individual token level, the disagreement signal (discussed in the blog) can be noisy. However, practical exploration signals inevitably involve some uncertainty and imperfection, since they aim to identify promising behaviors before their value is fully known. This does not make an exploration signal meaningless. The more relevant question is whether it guides the agent toward genuinely valuable regions and improves learning overall. (And it works very well!)

(The observation that negative signals become more destructive for stronger models can also be interpreted through the exploration-exploitation trade-off. As a model approaches optimality, the marginal value of exploration may decrease, while the same exploratory deviation incurs greater regret.)

Of course, RLRT does not establish an “infinite upper bound.” But I think it opens an important question.

- Can the signals used in OPD, OPSD, and related methods be systematically understood as exploration or exploitation signals for continual improvement?
- Can we design their schedules so that exploration continues where it remains valuable and becomes more focused over time?

I believe this is an important and still underexplored perspective. I am proud that RLRT opened up an important new axis of investigation!

[ar0cket1 @ar0cket1]

https://t.co/QL4SkUecDr

[ar0cket1 @ar0cket1]

https://t.co/XSmftb6kfC

[Dwarkesh Patel @dwarkesh_sp]

Recently met @srush_nlp and he started giving me an impromptu lecture on how targeted on-policy self-distillation works.

I asked him if I could record it on my iPhone.

The basic idea is this: if the model made a mistake at some point in the rollout (for example, calling a tool that doesn't exist), we want to discourage this specific error, but we don't want to just learn from the final reward, because it's a very noisy signal spread out over the whole trajectory.

So we have another model read this trajectory and figure where the error was made. It simply inserts some hint tokens to the part of the trajectory right above where the mistake was made.

Now with these injected hint tokens, have the model run a forward pass. You're not having to regenerate a new rollout - aka no new decode required.

The hint causes the model to assign lower probabilities to the error tokens. You then trains the original model to match these new probabilities, teaching it to downweight that specific mistake.

[Demis Hassabis @demishassabis]

I’ve always believed the No.1 application of AI should be to improve human health.

That work started with AlphaFold, and now at @IsomorphicLabs with the mission to reimagine drug discovery and one day solve all disease!

We are turbocharging that goal with $2.1B in new funding.

[SpaceX @SpaceX]

Watch Starship's twelfth flight test https://t.co/caRB1thMlg

[Fred Peng @pengzhangzhi1]

How to Train Diffusion LLM more efficiently?

Our paper has an answer for you:
Don’t Retrain, Align: Adapting Autoregressive LMs to Diffusion LMs via Representation Alignment

Diffusion language models are becoming increasingly attractive: they support bidirectional generation, non-sequential decoding, and flexible editing.

But training them from scratch is expensive.

So a natural question is:

If we already have strong pretrained autoregressive LMs, do we really need to relearn all language representations for diffusion LMs?

We argue: probably not.

Our view is that AR→DLM conversion should not be treated as learning language from scratch again. Much of the semantic structure is already inside the AR model. What changes is the generation order and denoising behavior.

So instead of only continuing denoising training, we explicitly preserve the representation geometry of the AR model.

We introduce REPR-ALIGN: during masked diffusion training, we align the hidden states of the DLM to a frozen AR teacher of the same architecture, layer by layer, using cosine similarity.

No adapters.
No architectural changes beyond the attention mask.
Just representation alignment + masked denoising.
The result: up to 4× training acceleration in our setting, with especially strong gains in low-data regimes.
The main takeaway is simple:

Don’t retrain the representation space from scratch.
Align it, and let the model relearn the decoding path.

Paper: https://t.co/djnfVWYRco
Code: https://t.co/SKYEBToFFW

Work done with an amazing undergrad @alexisfox and advisors @Anru_Zhang @AlexanderTong7

[Philosophy Of Physics @PhilosophyOfPhy]

Albert Einstein himself never believed black holes were real when they first appeared in his equations.

The actual black hole solution was found by Karl Schwarzschild in 1916, just months after Einstein published general relativity. Schwarzschild solved the equations exactly while serving on the Russian front during World War I.

Einstein considered the singularities unphysical and later tried to show that such objects could not exist in reality.

[Gabriele Berton @gabriberton]

Matrioshka Representation Learning (2024, Google Research)

A common problem with embedding models is retrieval cost. Embeddings can be costly to store and retrieve, especially if you store embeddings for huge databases (e.g. 100B documents) [1/N]

[Scite @scite]

https://t.co/DRcG09AHbK

[SpaceX @SpaceX]

Announcing TERAFAB: the next step towards becoming a galactic civilization https://t.co/xTA70LOU0e