We found a problem with on-policy self-distillation. The student makes mistakes early in a rollout, and the teacher has to continue from that flawed prefix, so its learning signal gets a lot weaker.
So we asked, "What if the teacher could step in before those mistakes happen?" Our method, Speculative Self-Distillation, lets the student generate while the teacher intervenes when their predictions diverge a lot.
We got better accuracy and training efficiency with our results, plus less forgetting in our tests. See @talaei_shayan’s thread for more!
Thanks to my amazing co-authors @chewing_a_gum, @jonashubotter, Amin Saberi and @Azaliamirh.
Introducing Speculative Self-Distillation (SSD, NeurIPS 2026), which pushes the accuracy–efficiency–forgetting frontier of self-distillation.
In SSD, student and teacher (the same model with additional information) write the rollouts together. Like a private tutor, the teacher lets the student write and takes over only when the student drifts too far. As a result, SSD trains on rollouts that stay as close to the student's own writing (on-policy) as possible while the teacher keeps them on track.
📉 38–69% fewer training tokens than on-policy, matching its accuracy
⏱️ Reaches on-policy accuracy in ~130 vs ~414 min (Science Q&A)
🧠 Forgets less in a stress test: −1.4 vs −4.5 points for on-policy
(1/4) why do image models chop every picture into the same square patches? 🤔 (hint: they shouldn't)
following the bitter lesson, we let diffusion models *choose* how to tokenize an image. it spends fewer tokens on the boring background, and more tokens on our faces! 🧵
(1/4) why do image models chop every picture into the same square patches? 🤔 (hint: they shouldn't)
following the bitter lesson, we let diffusion models *choose* how to tokenize an image. it spends fewer tokens on the boring background, and more tokens on our faces! 🧵
Trick #3: Merge, merge, merge.
In pretraining, we skipped LR decay for a running weight average.
Then we merged pre-context-extension models into post-extension checkpoints (see fig. for ablation study gains).
Merging was also an expedient stand-in for MOPD in our RLVR recipe.
Our Theo system formalized all 22 of Robertson and Seymour's Graph Minors papers, deepest theory in combinatorics: all 838 numbered results, no extra axioms.
Including graph minor theorem (Wagner's conjecture), excluded-minor structure theorem, and O(n³) k-disjoint paths algo.
We found a problem with on-policy self-distillation. The student makes mistakes early in a rollout, and the teacher has to continue from that flawed prefix, so its learning signal gets a lot weaker.
So we asked, "What if the teacher could step in before those mistakes happen?" Our method, Speculative Self-Distillation, lets the student generate while the teacher intervenes when their predictions diverge a lot.
We got better accuracy and training efficiency with our results, plus less forgetting in our tests. See @talaei_shayan’s thread for more!
Thanks to my amazing co-authors @chewing_a_gum, @jonashubotter, Amin Saberi and @Azaliamirh.
Introducing Speculative Self-Distillation (SSD, NeurIPS 2026), which pushes the accuracy–efficiency–forgetting frontier of self-distillation.
In SSD, student and teacher (the same model with additional information) write the rollouts together. Like a private tutor, the teacher lets the student write and takes over only when the student drifts too far. As a result, SSD trains on rollouts that stay as close to the student's own writing (on-policy) as possible while the teacher keeps them on track.
📉 38–69% fewer training tokens than on-policy, matching its accuracy
⏱️ Reaches on-policy accuracy in ~130 vs ~414 min (Science Q&A)
🧠 Forgets less in a stress test: −1.4 vs −4.5 points for on-policy
On-policy learning is wasteful. Trajectories following an early mistake often don’t recover and waste many tokens on non-useful signal.
We study speculative self-distillation: sampling as on-policy as possible while correcting “stupid” mistakes early.
This improves the efficiency-accuracy-forgetting frontier.
Every token counts!
Awesome work with @talaei_shayan, @chewing_a_gum, @ckodser, Amin Saberi, and @Azaliamirh.
Speculative Self-Distillation (SSD) lets AI models learn new knowledge from their own outputs, with a teacher stepping in only when predictions significantly diverge.
The result: 38–69% fewer training tokens, near on-policy accuracy, and less catastrophic forgetting.
Great work from @talaei_shayan 🚀
New paper!
On-policy self distillation is super wasteful. The teacher is only useful early in the rollout, and the student cannot course correct later.
With SSD, we handhold the student, and see much faster convergence and lesser forgetting!
Introducing Speculative Self-Distillation (SSD), which improves accuracy and effiecieny and reduces forgetting in self distillation.
It turns out only a few tokens in a rollout actually depend on the new knowledge. Letting the teacher generate just those tokens results in nearly matching on-policy self-distillation with 38–69% fewer training tokens, and the model forgets less.
Great work co-led by @talaei_shayan, @chewing_a_gum and @ckodser, with @jonashubotter and Amin Saberi (NeurIPS 2026). Check out Shayan’s thread for more.
Introducing Speculative Self-Distillation (SSD, NeurIPS 2026), which pushes the accuracy–efficiency–forgetting frontier of self-distillation.
In SSD, student and teacher (the same model with additional information) write the rollouts together. Like a private tutor, the teacher lets the student write and takes over only when the student drifts too far. As a result, SSD trains on rollouts that stay as close to the student's own writing (on-policy) as possible while the teacher keeps them on track.
📉 38–69% fewer training tokens than on-policy, matching its accuracy
⏱️ Reaches on-policy accuracy in ~130 vs ~414 min (Science Q&A)
🧠 Forgets less in a stress test: −1.4 vs −4.5 points for on-policy
Really happy to finally share Ripple-65B Preview! We trained it from scratch with a core team of five researchers, and it sets a new U.S. open-weight SOTA on Terminal-Bench 2.1 for agentic coding.
Three choices that shaped how we built it, and a few things I learned along the way.
1. We deliberately chose a dense model.
Many leading open-weight models are MoEs. For our use case, we wanted strong reasoning and low latency for agents going through long sequences of model and tool calls. Going dense also gave us a simpler training and serving stack and avoided expert-routing communication. That combination mattered a lot to us.
2. We bought no posttraining data.
A lot of the work went into building our own data generation and filtering pipelines, and making sure we covered the tasks we cared about. We built on open-source training and inference stacks throughout. I’m particularly happy about how much we were able to do as a small team by building on the community’s work.
3. We worked on the RL system and curriculum together.
Mixed RL meant dealing with very different rollout lengths, tool-call patterns and shares of the batch across domains. We tuned oversampling per domain and per problem to keep batch latency down, while managing policy staleness and trainer-inference mismatch. Score centering worked best among the corrections we tried; we used it alongside truncated importance sampling.
We also kept adjusting task selection and introducing harder problems as the model improved. Getting batches ready faster and keeping updates stable had to go hand in hand with giving the model useful problems to learn from.
Loved building this with @DZhang50, @esonghori2, @ethanboneh and @amiwilliamhuang. Huge thanks to our cofounders @annadgoldie and @Azaliamirh, and the whole @RicursiveAI family for making it possible. Really proud of what we built together.
Today we release Ripple-65B Preview, our 65B dense LLM we pretrained from scratch.
Ripple sets a new state-of-the-art for U.S. open-weight models in agentic coding tasks, surpassing significantly larger models including Muse Spark Open-816B, Beam-501B, Inkling-975B, and Nemotron 3 Ultra-550B.
Learn more: https://t.co/Pe04ZKhkE9
Scaling self-verification with DeepSeek V4 Flash beats Claude Fable 5 on Terminal-Bench 2.1, while being 11x cheaper 💰
As open-source models become more capable, they can now generate large numbers of high-quality candidate solutions and verify their own outputs at very low cost.
For example, we find that sampling just 5 solutions with DeepSeek V4 Flash and ranking them using the same model with LLM-as-a-Verifier can lead to a significant boost in accuracy (79% → 88%), outperforming closed frontier models on Terminal-Bench.
Try it out today: https://t.co/UhudjKDaYI
More on verification scaling in my previous post.
@JasonRute@MTHajiaghayi Same as Aleph Prover, the formalization assumes two classical theorems. First the Golod–Shafarevich, “On the Class Field Tower,” 1964 and Shafarevich, and “Extensions with Given Points of Ramification,” 1963. Theo independently assumed the exact two theorems as Aleph Prover.
Recently, OpenAI disproved planar unit distance problem, first posed by Erdős. We tested our pipeline (Theo) on it and Theo formalized the paper using under 15k lines of Lean, a big improvement over Aleph Prover's 33k lines & Open AI 1M lines. See https://t.co/SryZ82a0a7
The author just confirms the bug our system detected in the STOC 2025 paper. AI reviews are getting real and even better than the STOC-level review process.
🚨We built an AI system that turns research-level math into Lean and machine-checks it.
We formalized 5 complex STOC papers to Lean, and even found a gap in one published proof.
Paper: https://t.co/Ck8p3AIU7v.
Thanks to @DARPA (expMath program) & @patrickshafto for their support.