Thomas Wolf

co-founder @HuggingFace - moonshots

get your Microducks and Reachy Mini's at

[Thomas Wolf @Thom_Wolf]

18 of the 20 best open models today are Mixture-of-Experts. Dense is basically over.

And yet the know-how to train a good MoE still lives mostly in the heads of a handful of teams.

So we spent 800k GPU hours running the experiments ourselves and wrote everything down. Super happy to finally release "The Ultimate MoE Training Handbook", free and open, organized around the questions training teams hit:
- how many experts, how big, shared or not?
- why is half my step spent in all-to-all, and how do I hide it?
- why is my run 25x slower with no error in the logs?
- softmax, sigmoid or sqrt(softplus) router, and why keep it in fp32?
- aux loss or DeepSeek's loss-free bias?
- upcycle a dense checkpoint or train from scratch?
- can I serve with fewer experts per token than I trained with?

⚠️ Warning: it's long (~70k words, budget a few days, sorry) but with 55 interactive figures, code you can run and numbers from our own runs.

Read it here: https://t.co/6XihlUrj92

[Thomas Wolf @Thom_Wolf]

The future of software

In the limit, of course, software won't mean anything anymore. Everything digital will be instantly created at will. But in the interim period we'll wander through an increasingly odd in-between age for software. This transition periodwill be defined by three new precepts.

Everflux software - instead of stepping from release to release, months apart, software now evolves continuously.

Software creation has broken the sound barrier of our own keystrokes, and every model release pushes it faster. There is no reason to wait weeks or months for the next release when a single company merges over 30,000 code changes a day. Cyber capabilities are also progressing at an increasing pace. Bug and vulnerability fixes need to ship quickly and continuously. And every piece of software sits in a lattice of dependencies, each one now shipping at a blistering pace. Software will have to move accordingly fast just to keep from breaking as everything around it shifts.

Translucent sources – A new territory has opened between open source and closed source.

Today Claude will happily pick apart any Electron app, and any binary shipped with debug symbols has become a reverse-engineering playground for LLMs. In other words, much so-called closed-source software is now open source in all but name, and keeping code closed makes less and less sense.

Even the notion of IP and copyright is breaking in software: Apps don't even need to be accessed to be reproduced. I'm writing this text on a personal text editor I prompted Claude to vibe-code from a few videos of people demoing editor features they liked. A mere description of the interface is increasingly enough for an LLM to rebuild the internals, whether or not they resemble the original. Software IP seems to be dissolving like cotton candy in a raccoon's paws.

Bespoke features - Now that all software is open source, nothing prevents us from modifying everything.

And personalization is just one prompt and a couple million tokens away, so there is absolutely no reason anymore to wait for someone else to add this feature you want. Everyone becomes their own product manager. Want a search box, an API bridge, a custom button? It's a couple of minutes away, mostly executed in the background of your daily life. Soon we'll want to personalize everything.

A major challenge appears when bespoke features crash into our first tenet: everflux software. How do you customize software without cutting yourself off from upstream? I could fork it for good to add my search box, but I'd lose all future benefit from the software creator, be it new security updates or other updates to the rest of the software. How do you personalize software that was never meant to be personalized, without shouldering all of its future maintenance?

Merge conflicts won't be the exception anymore; they'll be a constant of everflux software. Agents will rebase our personal features onto each new release overnight, and software will look less like a product and more like a living species, evolving through constant mutation.

Image: M.C. Escher, Metamorphosis II, November 1939 - March 1940

[Thomas Wolf @Thom_Wolf]

@rauchg Beautiful, I love it

[Thomas Wolf @Thom_Wolf]

@hyperstition_cc very impressive folks

[Thomas Wolf @Thom_Wolf]

@psipbc Great work!

(and unfortunate conclusion)

[Thomas Wolf @Thom_Wolf]

@LiTianleli lol

[Lysandre @LysandreJik]

We’ve been hard at work trying to make models work with harnesses better. The best thing @yb2698 found: “harness adapters”. Small LoRAs trained with RL inside the actual agent harness they’ll be used with.

Same base model. Different adapter for pi, opencode, hermes, etc. Why?

Simply because the same model behaves very differently depending on the tools, prompts and control flow around it. Instead of asking one set of weights to be optimal everywhere, we can specialize a tiny part of the model for each environment.

Early results are pretty encouraging: on held-out tasks, the adapters make fewer tool calls than the base model while keeping or improving accuracy.

And because they’re LoRAs, the operational side is surprisingly cheap too.

For one LFM2.5-24B-A2B experiment, we evaluated 9 checkpoints / 1,350 agent sessions on a single H200 in under an hour. The total: ~$5!

Merging and serving every checkpoint separately would have cost around $90.

There’s another important piece: RL agents will eventually try to hack the reward.

So every tool call goes through `safeagents`, a deterministic rule-based guard that blocks things like editing tests, tampering with rewards or bypassing the sandbox before they execute.

And the whole loop can start from real usage:

your coding sessions -> verifiable environments -> RL -> a better adapter for your agent.

The model doesn’t have to change completely.

Sometimes it just needs to get better at the harness it actually lives in.

https://t.co/1vZxJ0d5EY

[Thomas Wolf @Thom_Wolf]

same open model, different harness => very different results

Joël set out to find which harness actually gets the most out of open models in November 2026

[Joël Niklaus @joelniklaus]

I burned a shitload of compute so you don't have to. Releasing the Harness Leaderboard.

Which harness should you run your open model in? I ran 28,849 trials to find out.

The leaderboard compares nine coding-agent harnesses on open models. In each comparison only the harness changes: the model, provider and reasoning effort stay fixed, and every task gets three attempts.

The trials ran GLM-5.3-Flash, DeepSeek-V4.1-Flash and Flint on Terminal-Bench 2.1 and DeepSWE, for $4,746 of inference.

Averaged over the three models, mini-swe-agent (@KLieret), a bash-only agent, solves 51.4% of tasks and @openclaw 27.3%. Four others are statistically tied with mini-swe-agent.

One of them is Pi (@pidotdev), at 49.4% for $0.29 per solved task, about 45% less than mini-swe-agent's $0.53.

The best harness depends on the model: mini-swe-agent on GLM-5.3-Flash, Hermes Agent (@NousResearch) on DeepSeek-V4.1-Flash and Kimi Code (@Kimi_Moonshot) on Flint.

@opencode costs $3.35 per solved task, 11.7 times as much as Pi.

See Harness Hub's Harness Lens for what the nine harnesses do differently. It documents the prompts, tools and compaction of 34 coding-agent harnesses and links every claim to its source.

Leaderboard: https://t.co/AiWiQtBOz5
Harness Lens: https://t.co/I8jbcWa9JH

[LeRobot @LeRobotHF]

𝗟𝗲𝗥𝗼𝗯𝗼𝘁 𝘃𝟬.𝟳.𝟬 𝗶𝘀 𝗟𝗜𝗩𝗘 🚀🤖

From terabytes of data to prompting robots! Check out all what's new in the video below and join the 50,000+ developers & 300+ organizations in the LeRobot ecosystem 🤗

[Thomas Wolf @Thom_Wolf]

@eliebakouch @leavittron how can we scale you up elie? where is the batch size hp

[Thomas Wolf @Thom_Wolf]

By the way you should follow Augustin on X - I’m trying to get him to post more - it’s one of the most cracked roboticist I’ve ever met

[Augustin Crampette @Gus_Crampette]

We were at the @seeedstudio Shenzhen factory today with @antoinepirrone and @matth_lapeyre and seeing the Microduck production line run for the first time was something special.

We even had our own assembly stations 🙂

[Thomas Wolf @Thom_Wolf]

18 of the 20 best open models today are Mixture-of-Experts. Dense is basically over.

And yet the know-how to train a good MoE still lives mostly in the heads of a handful of teams.

So we spent 800k GPU hours running the experiments ourselves and wrote everything down. Super happy to finally release "The Ultimate MoE Training Handbook", free and open, organized around the questions training teams hit:
- how many experts, how big, shared or not?
- why is half my step spent in all-to-all, and how do I hide it?
- why is my run 25x slower with no error in the logs?
- softmax, sigmoid or sqrt(softplus) router, and why keep it in fp32?
- aux loss or DeepSeek's loss-free bias?
- upcycle a dense checkpoint or train from scratch?
- can I serve with fewer experts per token than I trained with?

⚠️ Warning: it's long (~70k words, budget a few days, sorry) but with 55 interactive figures, code you can run and numbers from our own runs.

Read it here: https://t.co/6XihlUrj92

[Teortaxes▶️ (DeepSeek 推特🐋铁粉 2023 – ∞) @teortaxesTex]

And yes, they do not highlight their own row, those are best results. It is good at vision. It's good at modern frontier-coded tasks. And it was *cheap*. They just somehow trained a very strong base… and RL on that was easy. That's pretty crazy. Look eg at Omniscience.

[Thomas Wolf @Thom_Wolf]

@thinkymachines Well done, this is great ! Looking forward to the weights release

[Thomas Wolf @Thom_Wolf]

this METR report on AI R&D acceleration at Anthropic is very inward looking imo. it counts the lab's compute, hires and internal data, then credits ~80% of the speed-up to AI

what I see from the outside: labs are becoming schelling points for money, redistributing it to an ecosystem they created to accelerate their own development

it's a symbiosis. the best data and RL environment companies now know where models will fail before the labs do, and the labs just buy what they propose. that feeds back into every lab at once

a lot of the "AI accelerating AI R&D" happens outside the lab

[METR @METR_Evals]

Since August, a team within METR has worked with internal Anthropic information to assess whether AI itself is accelerating AI development.

We are now publishing a report summarizing the results of this risk assessment.

[Thomas Wolf @Thom_Wolf]

@TheZachMueller @Nouamanetazi @lvwerra @huggingface It will come! (But without interactive animations…)

[Thomas Wolf @Thom_Wolf]

18 of the 20 best open models today are Mixture-of-Experts. Dense is basically over.

And yet the know-how to train a good MoE still lives mostly in the heads of a handful of teams.

So we spent 800k GPU hours running the experiments ourselves and wrote everything down. Super happy to finally release "The Ultimate MoE Training Handbook", free and open, organized around the questions training teams hit:
- how many experts, how big, shared or not?
- why is half my step spent in all-to-all, and how do I hide it?
- why is my run 25x slower with no error in the logs?
- softmax, sigmoid or sqrt(softplus) router, and why keep it in fp32?
- aux loss or DeepSeek's loss-free bias?
- upcycle a dense checkpoint or train from scratch?
- can I serve with fewer experts per token than I trained with?

⚠️ Warning: it's long (~70k words, budget a few days, sorry) but with 55 interactive figures, code you can run and numbers from our own runs.

Read it here: https://t.co/6XihlUrj92

[Thomas Wolf @Thom_Wolf]

@giffmana @Nouamanetazi @lvwerra @huggingface Thanks Lucas!

[alexine 🏴‍☠️ @alexinexxx]

this takes a few days to read, so if you're unemployed it'd be a great use of your time. the rest of the hugging face blog too actually, they always put out great playbooks!!

[Thomas Wolf @Thom_Wolf]

short answer from the handbook: at fixed total and active params, finer experts help up to a point. in one ablation the loss kept dropping until 128 experts (each 1/4 width, top-8), then nothing more (most released models sit at 128 to 384 experts, top-4 to top-8)

past that, smaller GEMMs and more all-to-all traffic start eating your step time, so the best count on your cluster is usually a bit less sparse than the loss alone says.

details + Megatron flags: https://t.co/EBnt5cJsW1

[Allan @AllanatrixQ]

This is insanely cool!

[Thomas Wolf @Thom_Wolf]

Leandro's thread:
https://t.co/6oAw6ZgUuV

[Leandro von Werra @lvwerra]

After the Ultra Scale Playbook, Nouamane is back with this crazy guide to train MoEs!

It's probably the most comprehensive resource if you want to get into large scale MoE training and an absolute gem packed with beautiful visualization!

[Thomas Wolf @Thom_Wolf]

Nouamane thread on it: https://t.co/l2BAx7Qhac

[Nouamane Tazi @Nouamanetazi]

📖 The Ultimate MoE Training Handbook 📖

We spent 1/2M GPUh to optimize MoE training and wrote the most comprehensive blog about it. With 30k words and 50+ interactive figures it covers everything from profiling to large cluster training.
https://t.co/mk6fGoCiqV