Budhaditya Deb

NLP researcher, DL practitioner and a compulsive philomath

[William Dalrymple @DalrympleWill]

"Unless You Have Studied History to an Advanced Level It is Not Possible to Understand Britain’s Past"
https://t.co/vyTToC8pzR

[John Schulman @johnschulman2]

PPO had a second wave in the LLM era for reasons unanticipated by the original paper
- the importance-ratio objective fixes biases from numeric error, async training, and forward pass noise
- the clipping objective affects entropy through a mechanism that we didn't know about at the time of publication (DAPO, https://t.co/sBo9DeFS5Y)

[Nathan Lambert @natolambert]

New podcast with @finbarrtimbers! We survey the latest post-training recipes, from GLM 5.1, Kimi K2.6, DeepSeek V4, Xiaomi MiMo V2.5, Nemotron Ultra, etc. and discuss:
- Why the industry slowly shifted to multi-teacher on-policy distillation (MOPD).
- What an Olmo-style recipe would need improvements in
- How post-training works / suits larger organizational efforts
- Career advice in the foothills of the singularity
- and other topics

I heard y'all wanted me to start doing this, so making some time when I'm in funemployment!

Chapters:

00:00 Introduction & Olmo reflections
06:28 Post-train recipes review (history)
23:00 2026’s model recipes (MiMo Flash, DeepSeek V4, GLM 5, Kimi K2.6, etc.)
39:05 Open-ended post-training discussions
48:22 Career advice in the LLM race

Links below, please follow @interconnectsai and like and subscribe and buy my book?

[Rishabh Agarwal @agarwl_]

Self-distillation does not work for thinking models YET

https://t.co/yEMIADSJis
https://t.co/prFFWPpWAo
https://t.co/ChozypYWPD

[Satya Nadella @satyanadella]

https://t.co/vLmiBKTtX3

[Anthropic @AnthropicAI]

The US government, citing national security authorities, has issued an export control directive to suspend all access to Fable 5 and Mythos 5 by any foreign national, whether inside or outside the United States, including foreign national Anthropic employees.

The net effect of this order is that we must abruptly disable Fable 5 and Mythos 5 for all our customers to ensure compliance.

Access to all other Claude models is not affected.

We apologize for this disruption to our customers. We believe this is a misunderstanding and are working to restore access as soon as possible.

Read our full statement: https://t.co/bwn0sximKZ

[Yinghui He@ICML @yinghui_he_]

RLVR gives sparse supervision; On-Policy Self-Distillation often requires high-quality demonstrations. Our new method, ✨SD-Zero✨, gets the best of both worlds – we use model’s self-revision to turn binary rewards into dense token-level supervision. No external teacher. No curated demonstrations.

🚨 Introducing Self-Distillation Zero (SD-Zero), which trains one model to play two roles: (1) “Generator” that makes attempts, and (2) “Reviser” that conditions on the generator’s failed/successful attempt + binary reward to produce a better answer. ‼️Even WRONG attempts can become the training signal.‼️

🔗Paper: https://t.co/LwboIqHE11

🏆 SD-Zero brings 10%+ improvement over base models (Qwen3,4B; Olmo3,7B) on math & code reasoning, beating GRPO and vanilla On-Policy Self-Distillation under the same training budget. SD-Zero also enables iterative self-evolution.

[Dwarkesh Patel @dwarkesh_sp]

Recently met @srush_nlp and he started giving me an impromptu lecture on how targeted on-policy self-distillation works.

I asked him if I could record it on my iPhone.

The basic idea is this: if the model made a mistake at some point in the rollout (for example, calling a tool that doesn't exist), we want to discourage this specific error, but we don't want to just learn from the final reward, because it's a very noisy signal spread out over the whole trajectory.

So we have another model read this trajectory and figure where the error was made. It simply inserts some hint tokens to the part of the trajectory right above where the mistake was made.

Now with these injected hint tokens, have the model run a forward pass. You're not having to regenerate a new rollout - aka no new decode required.

The hint causes the model to assign lower probabilities to the error tokens. You then trains the original model to match these new probabilities, teaching it to downweight that specific mistake.

[Gary Marcus @GaryMarcus]

this is legit funny.

[Dan @DanTalkSports]

World Cup Group schedule in BST, for anyone who wants it

[Hanna Hajishirzi @HannaHajishirzi]

MAI-Thinking-1 is out!

Excited to share what we are building and how climbing from scratch (no distillation) actually works: simple recipes, rigorous science, self-distillation, patience, and great infra.

Check out our tech report has the full story of our RL climbs.
https://t.co/aLW40sWz4d

[Tony Lee @tonyh_lee]

Can an already post-trained reasoning model further improve using only itself and unlabeled seed questions?

@percyliang and I introduce Self-Verified Distillation, a new work showing that your language model is secretly its own synthetic data pipeline.

🧵

[Turing Post @TheTuringPost]

https://t.co/dQosaSZYx2

[Arsenal @Arsenal]

The Arsenal. Your Premier League champions.

[Jeremy Corbyn @jeremycorbyn]

North London Forever.

[Parimal @Fintech03]

Gandhi was frustrated that even while protesting the British, every time a freedom fighter signed a document, they were using Quink (Parker)/Stephens (British) ink. He saw it as a symbolic defeat. Gandhi personally asked Satish Chandra Dasgupta (a legendary chemist, I will write separately about him) to create an ink that would not betray Indian paper.

Satish Chandra did not want to run a business, so he passed the formula (Krishna Dhara) to 2 brothers, Nanigopal & Sankaracharya Maitra in Rajshahi (now Bangladesh). They started Sulekha (named by Tagore) in a small room with a single stove.

British inks of the 1930s were notorious for being Iron-Gall based. These were highly acidic. If you look at British-era documents today, many have holes where the ink was the acid literally ate through the paper over 50 yrs. The Maitra brothers realized that Indian paper (often handmade/low-quality) would disintegrate under British ink.

They engineered Sulekha to be pH-neutral. They used a secret infusion of tannins from indigenous sources & refined them to ensure the ink was archival. The reason many 1940s Indian revolutionary pamphlets are still readable today, w/o the paper crumbling, is specifically due to the non-corrosive chemistry of Sulekha.

Also, Satyajit Ray, who was a master calligrapher (he designed his own fonts like Ray Roman), preferred Sulekha because of its Matte-Finish density.

[Students For Liberty @sfliberty]

Four months after George Orwell published 1984, his former teacher sent him a letter.

Aldous Huxley had one message: you described the wrong dystopia. 🧵

[Vasundhara Sirnate @vsirnate]

Sigh! In the light of what Donald Trump has posted about Indian and Chinese immigrants and birthright citizenship, I am once again doing a thread on Bhagat Singh Thind and what happened in the US a century ago.

Bhagat Singh Thind was born in 1892 in Taragarh, Amritsar, Punjab, colonial India. He migrated to the US on board a ship called the Minnesota which docked at Seattle on July 14, 1913.

Photo: Wikimedia Commons.

[William Dalrymple @DalrympleWill]

Lovely map showing the first letter of the Sanskrit alphabet 'ka' in every Brahmi-derived script... It vividly shows the reach of the Golden Road

[Gary Marcus @GaryMarcus]

This – “The CEO of Google DeepMind (@demishassabis) just admitted that if the decision had been his, we would've cured cancer before anyone ever used ChatGPT.” is exactly what i am trying to say in my pinned tweet, in different words.

The fact that we left things to Altman instead will haunt us.