Zafir Stojanovski

Core contributor:
• 🏋🏻‍♀️ Reasoning Gym – NeurIPS (Spotlight)
• 🌊 MCL – Interpolate @ NeurIPS (Best Paper)
• 📚 RLHF Book

[Zafir Stojanovski @zafstojano]

Added this neat little figure to @natolambert's RLHF Book showing the conditions under which we backprop through the PPO-style loss:

1. [Green region] Advantage is positive and π_new(a) < (1+ε) * π_old(a) -- the action we performed was good, so boost the probability of the action under the current policy, BUT not so much that we make it substantially more likely.

2. [Red region] Advantage is negative and π_new(a) > (1-ε) * π_old(a) -- the action we performed was bad, so decrease the probability of the action under the current policy, BUT not so much that we over-suppress it.

3. [Grey region] The action is appropriately expressed under the current policy -- therefore do nothing.

[Alexander Panfilov @kotekjedi_ml]

We can finally talk about it:

We found a way to extract hidden reasoning of frontier models using a vulnerability in the APIs of every frontier AI company.

We verified that our reasoning token count matches billed API thinking tokens 1:1 for most of the prompts we queried.

[Nathan Lambert @natolambert]

My summer project is done! A 20 video, free course on post-training to accompany my book is all on YouTube with slides open for modification & re-use.

~12 hours of content covers the core foundations and some research areas I think will grow in importance. It was a fun time to review all the fundamentals again, as it is clear in the next 1-3 people the amount of people wanting to learn post training will likely 100X again from today, as we have already 100X'ed from two years ago.

As AI agents get increasingly capable at coding and discussing these fundamentals (see the code exercises accompanying the book that I am refining with the community) I think developing clear intuitions for how models work and why is one of the most important skills going forward in AI. Still, learning the post-training math is the best way to battle test them. I personally just in this course am starting to master how forward/reverse KL relates to post-training topics.

Thanks to all my viewers, and I'm happy to answer questions in the book discord or understand how to better teach the various reward models, on-policy distillation, new RL algorithms, etc.

Plus, the book is 50% off right now with the code PBLambert on Manning to celebrate the launch.

I'll share the relevant links below.
Who's going to make this course for pretraining?

[Zafir Stojanovski @zafstojano]

@deteodskopje 🩵

[Zafir Stojanovski @zafstojano]

@neurosp1ke Good luck Andreas!

[Zafir Stojanovski @zafstojano]

@natolambert @Shekswess who needs more than that :D

[Zafir Stojanovski @zafstojano]

@marioxpet 🩵

[Bojan Jakimovski @Shekswess]

We strongly believe that science must be open and accessible to everyone!

Today, we are proud to announce our collaboration with @arcee_ai and @awscloud (& @PrimeIntellect's prime-rl), exploring what open-source models can achieve for science and for the future of humanity.

This is only the beginning of what open models can do for science!

Open for the win !!!

[Arcee.ai @arcee_ai]

The @lokahq team post-trained our 26B Trinity Mini into an agentic harness specialized for scientific work.

Working with @PrimeIntellect and @AWS, Loka taught Trinity Mini to execute multi-step research workflows, coordinate tools, query databases, and log an auditable trail at every step.

Here is a look inside the harness. 🧵

[Zafir Stojanovski @zafstojano]

@natolambert Massive, congrats!

[Nathan Lambert @natolambert]

My book, Reinforcement Learning from Human Feedback is done!

This is the book I wish I had when learning to fine-tune, align, & now post-train models since ChatGPT. The resource has been built by me finding time to study and document the fundamentals on nights and weekends since 2024.

Transferring as much of the intuitions of building Olmo as I possibly can in the book format.

The book is launching with an over 10 hour, full course with slidedecks, functional code for the training chapters, an example model completions library, and of course the free online web version.

Physical orders from Manning will ship in 1-2 weeks, and Amazon a week or so after. Thanks for your support!

[Jean Kaddour @jeankaddour]

CISPO outperforms GRPO/DAPO by capping a token's IS weight instead of zeroing out its update, so rare "fork" tokens ("Wait", "However") keep contributing gradient across updates.

Using CISPO, @lorenz_wlf reaches a new Sokoban Speedrun record; for the first time under 20min!!

[Zafir Stojanovski @zafstojano]

@marioxpet @Shekswess 🩵

[Zafir Stojanovski @zafstojano]

@Shekswess 🩵

[Nathan Lambert @natolambert]

Happy to say @zafstojano - an added maintainer who helps me with the RLHF Book code - added a simple on-policy self-distillation example to the codebase, which can work on some toy problems.

Excited to dig into this more, happy to see the repo fleshed out!

[Zafir Stojanovski @zafstojano]

@_Suresh2 @natolambert Exactly, otherwise we end up over-expressing/suppressing an already correctly calibrated action

[Zafir Stojanovski @zafstojano]

Added this neat little figure to @natolambert's RLHF Book showing the conditions under which we backprop through the PPO-style loss:

1. [Green region] Advantage is positive and π_new(a) < (1+ε) * π_old(a) -- the action we performed was good, so boost the probability of the action under the current policy, BUT not so much that we make it substantially more likely.

2. [Red region] Advantage is negative and π_new(a) > (1-ε) * π_old(a) -- the action we performed was bad, so decrease the probability of the action under the current policy, BUT not so much that we over-suppress it.

3. [Grey region] The action is appropriately expressed under the current policy -- therefore do nothing.

[Zafir Stojanovski @zafstojano]

@Shekswess 🩵

[Zafir Stojanovski @zafstojano]

@natolambert Read the chapter here: https://t.co/rlA7ovF8Sc

[Zafir Stojanovski @zafstojano]

@Shekswess 💙

[Nathan Lambert @natolambert]

Banning open-source AI in any form would be a mistake. A general audience PSA with @kevinsxu on why open source upholds American values.

Managing frontier risks is hard, but reducing transparency, innovation, and education from kneecapping the open frontier would be worse.