New blog post - Chasing the Karpathy Peak (and Falling Off Slop Cliff): Confessions from Slopaholics Anonymous
My story of becoming a slopaholic, who’s most at risk, and how to avoid it (or recover)
https://t.co/9RHumEXfMM
Small bird, fast wings, Kolibri is here.
78B parameters. 3.46B active. Up to 1M tokens of context. Built in Europe.
Now the weights are yours. Run it on your own hardware, under Apache 2.0.
Our work on improving exploration for RLVR has been accepted at #NeurIPS2026!
Thanks a lot to my collaborators @ndaheim_ and @IGurevych
Project page: https://t.co/Hmvh6ecqIs
Paper: https://t.co/mCe58Ozt0X
Your GRPO run is burning compute on rollouts that teach it nothing.
When all rollouts get the same reward, the advantage is zero. No gradient. You paid full price and got nothing.
We fix this by studying opening a new axis of exploration in RLVR: the parameter space.🧵
For AI to work with us, it needs to understand us
Today, we're introducing Persimmon, the first large-scale model designed to realistically simulate how people talk and interact
Ok, so I can finally talk about this!
We spent the last year (actually a bit longer) training an LLM with recurrent depth at scale.
The model has an internal latent space in which it can adaptively spend more compute to think longer.
I think the tech report ...🐦⬛
🚢 Marin 535B-A23B started training this week! As usual, the whole process is open.
Voyage plan: pretraining (80%) + midtraining (20%) on 18.75T tokens on 11 x GB200 NVL72 for ~3 months (2.7e24 FLOPs). Post-training will follow.
Before kicking off the run, we trained a 4-rung scaling ladder from 1.6B-A61M (48B tokens) to 27.7B-A1.2B (926B tokens) to debug issues, and to make a forecast of our hero run. This is by far our biggest run, so definitely expecting the unexpected.
We can now RL large MoEs with 0 train-infer mismatch! And doing so can improve performance (pictured task: teach Qwen3.6-35B-A3B to play Wordle). Everything is open-source and we did a bunch of ablations. 🧵
Your GRPO run is burning compute on rollouts that teach it nothing.
When all rollouts get the same reward, the advantage is zero. No gradient. You paid full price and got nothing.
We fix this by studying opening a new axis of exploration in RLVR: the parameter space.🧵
Our methods are implemented in verl and compose with everything you’re already doing in action space! So feel free to try it out :)
Thanks to my amazing collaborators @ndaheim_ and @IGurevych
💻 https://t.co/VSFgnaWtco
🤗 https://t.co/dqLkHaM9HI
📄 https://t.co/p8Ml2euf3n
Some practical guidance if you wanna try this out on your datasets:
A chunk size of N=2-4 works quite well for C3PO, and λ=1e9 is a good starting point but should be tuned. M3PO’s variance reduction only pays off if you don’t shrink the group size, but costs extra.