Jay Hineman

Mathematics Ph.D., dad, etc.

[Nathan Lambert @natolambert]

My summer project is done! A 20 video, free course on post-training to accompany my book is all on YouTube with slides open for modification & re-use.

~12 hours of content covers the core foundations and some research areas I think will grow in importance. It was a fun time to review all the fundamentals again, as it is clear in the next 1-3 people the amount of people wanting to learn post training will likely 100X again from today, as we have already 100X'ed from two years ago.

As AI agents get increasingly capable at coding and discussing these fundamentals (see the code exercises accompanying the book that I am refining with the community) I think developing clear intuitions for how models work and why is one of the most important skills going forward in AI. Still, learning the post-training math is the best way to battle test them. I personally just in this course am starting to master how forward/reverse KL relates to post-training topics.

Thanks to all my viewers, and I'm happy to answer questions in the book discord or understand how to better teach the various reward models, on-policy distillation, new RL algorithms, etc.

Plus, the book is 50% off right now with the code PBLambert on Manning to celebrate the launch.

I'll share the relevant links below.
Who's going to make this course for pretraining?

[Lenka Zdeborova @zdeborova]

❓ Ever wondered what drives the rank of keys & queries in attention layers?
❓ Why do we train them factorized instead of their product directly?
❓ What explains the spectra of key&query weights after training?
We tackle these in a new paper:
👉 https://t.co/iMRKXj0AFC

[Lenka Zdeborova @zdeborova]

Congrats to the great #NeurIPS best paper (https://t.co/junOr0NASh) on memorization in diffusion models. But note: producing non-memorized samples is not the same as generalization in the sense of learning structure or rules. Understanding generalization in DMs remains open.

[elvis @omarsar0]

Interesting research from Meta on hardware scaling trends.

More GPUs doesn't always mean faster training.

The default approach to scaling LLM training today remains throwing more hardware at the problem. More accelerators, more parallelism, more compute.

However, there's a ceiling that most teams don't see until they hit it.

This new research demonstrates that scaling the total number of accelerators for large model training quickly yields diminishing returns, even with optimized hardware and parallelization strategies.

The researchers tested Llama-2 models (1B to 70B parameters) across 8 to 2,048 GPUs spanning V100, A100, and H100 hardware. What did they find? When scaling from 128 to 2,048 GPUs, throughput decreased by 37.22% while per-GPU power draw only dropped 5.87%.

The culprit is communication overhead. At large scales, AllGather and ReduceScatter (two MPI primitives) operations become bottlenecks. The majority of communication becomes exposed, and computation can't hide the latency anymore.

Counter-intuitively, model parallelism strategies (tensor and pipeline parallelism at degrees 2-4) that were previously thought to reduce hardware utilization actually become preferable at scale. They reduce exposed communication compared to pure data parallelism.

On newer hardware, utilization gets worse, not better. Model FLOPS Utilization dropped from 59.67% on A100 to 40.77% on H100; faster chips expose more communication overhead.

Why it matters: Adding more GPUs provides poor marginal performance per additional unit of power or GPU-hour. Teams scaling to thousands of accelerators need to carefully reconsider parallelization strategies rather than assuming more hardware equals faster training.

[Chelsea Finn @chelseabfinn]

For robots to be actually useful, they need to be reliable.

We’re sharing an RL recipe for VLA models that takes a step in this direction, allowing robots to operate autonomously for hours at a time.

Blog & paper: https://t.co/SICNEhnXis

[Chelsea Finn @chelseabfinn]

RL fine-tuning often prematurely collapses policy entropy.

We consider a general framework, called set RL, i.e. RL over a set of trajectories from a policy.

We use it to incentivize diverse solutions & optimize for inference time performance.

Paper: https://t.co/YF32rah9tS

[Subbarao Kambhampati (కంభంపాటి సుబ్బారావు) @rao2z]

Yes, I peer reviewed DeepSeek-R1 paper for @nature and hope to see more frontier model developers follow their lead of sharing peer-reviewed technical details of their work, and go beyond buzzy blogs and patronizing "yeah, we have done it this way too" post facto claims..🤞

[Julian Togelius @togelius]

Can large language models play simple arcade games? Kind of. Sometimes. Slowly, and not as well as a simple search algorithm. And only if you format the input right. Of course, we made a benchmark to investigate this in more detail, because that's what we do.

[The Associated Press @AP]

More than 500 robots from 16 countries compete in sports like soccer, running and boxing at the first World Humanoid Robot Games in Beijing.

[Jeff Clune @jeffclune]

Thrilled to introduce Foundation Model Self-Play, led by @_aadharna. FMSPs combine the intelligence & code generation of foundation models with the curriculum of self-play & principles of open-endedness to explore diverse strategies in multi-agent games, like the one below 🧵👇

[Andrew White 🐦‍⬛ @andrewwhite01]

Feel the AGI moment right here

[Sergey Levine @svlevine]

Warm-start RL (WSRL) can learn to control a real robot in under 20 minutes! Deep RL is getting really fast. Warm-start from offline data + super-efficient online learning is increasingly making real world RL not just practical but pretty easy.

[Olexandr Isayev 🇺🇦🇺🇸 @olexandr]

Our new @ChemRxiv preprint: AIMNet2-rxn: A #MachineLearning Potential for Generalized Reaction Modeling on a Millions-of-Pathways Scale! #compchem Collaboration with @ProfSavoie group. Funded by @USNavyResearch under #MURI program

https://t.co/OMwymZi2eb

[Andrew White 🐦‍⬛ @andrewwhite01]

I have written up a 3.5k word/10 figure essay on how to write a reward function while avoiding reward hacking for chemistry. It covers all the ridiculous ways we had to avoid reward hacking for training ether0, our scientific reasoning model.

[Sergey Levine @svlevine]

Self-supervised representation learning looks a bit like RL. What if we literally use RL as a SSL method for visual representations? Turns out that it works quite well. In new work by @its_dibya, we show how this can be done: https://t.co/JTTbqIW2Pv

[Kyle Corbitt @corbtt]

"RL from a single example works"
"RL with random rewards works"
"Base model pass@256 can match RL model pass@1"
"RL updates a small % of params"

Recent papers all point in the same direction: RL is mostly just eliciting latent behavior already learned in pretraining, not teaching new behavior. Yann Lecun's "RL is the cherry on top" was right after all.

Is this bearish for RL? Perhaps not! Maybe we should think about RL as the last mile of on-the-job training for your specific task.

When you hire a customer support rep, you start with someone who's already smart and capable. But you *also* watch them take a few calls and give them feedback. That helps them align them to the nuances of your org and task, and can improves their performance dramatically.

RL is exactly this, but for LLMs. I expect it to become standard practice across orgs serious about deploying reliable agents.

[Subbarao Kambhampati (కంభంపాటి సుబ్బారావు) @rao2z]

Anthropomorphization of intermediate tokens as reasoning/thinking traces isn't quite a harmless fad, and may be pushing LRM research into questionable directions.. So we decided to put together a more complete argument.. 👇🧵 1/

[Mathieu Blondel @mblondel_ml]

We just released a new approach for turning local search heuristics used to solve NP-hard combinatorial problems in OR into differentiable layers. The key idea is to use the neighborhoods used by these algorithms for creating MCMC proposal distributions https://t.co/b07ODJfmGG