PostTrainBench

Measuring how well CLI agents like Claude Code or Codex CLI can post-train base LLMs on a single H100 GPU in 10 hours, build by @karinanguyen, @ThoughtfulLab_

[PostTrainBench @posttrainbench]

PostTrainBench v1.0 and the accompanying paper are out! We believe this benchmark will be important to measure progress in AI R&D automation. What are our findings? 1/n

[PostTrainBench @posttrainbench]

$PTB

Can AI agents improve performance of base LLMs? We give each agent 4 small target LLMs, an H100 GPU, and 10 hours to post-train them.

build by @thoughtfullab , @karinanguyen

CA : GtQndxV3DxFqMZDkXMdT17BLY4fH8BHM6hqgWThjBAGS

[PostTrainBench @posttrainbench]

I just applied to the $4M @BagsHackathon with my project PostTrainBench on @BagsApp

Check it out here 👇

https://t.co/uJyYEATdxi

[PostTrainBench @posttrainbench]

I just launched $PTB on @BagsApp 💰

https://t.co/TSZgmSFNgS

[PostTrainBench @posttrainbench]

I just applied to the $4M @BagsHackathon with my project PostTrainBench on @BagsApp

Check it out here 👇

https://t.co/1kPLP8N6Rg

[PostTrainBench @posttrainbench]

I just launched $PTB on @BagsApp 💰

https://t.co/BO9HYKzVBD

[PostTrainBench @posttrainbench]

Contributions are very welcome! We have a roadmap in the README for future work and will be continuously expanding it. Led by @full__rank and @hrdkbhatnagar

🔗 https://t.co/zfe0u0HCOd 📷

https://t.co/hSZWuawdg4

[PostTrainBench @posttrainbench]

PostTrainBench v1.0!

This benchmark evaluates the ability of frontier AI agents to post-train language models in a simplified setting.

We believe this is a first step toward tracking progress in recursive self-improvement 🧵:

[PostTrainBench @posttrainbench]

From ImportAI by @jackclarkSF

[Karina @karinanguyen]

From ImportAI by @jackclarkSF (thank you for the feature):

"Imagine where we’ll be in two years - we’ll certainly have AI models that are smart enough to point themselves at a specific objective, find an open weight model, then autonomously improve it to get better performance at that task. The era of ephemeral, custom AI systems, built and budded off into the world like spores from mushrooms, draws near. Are you ready for this new ecosystem you will find yourself in? I am not. But nonetheless it approaches."

[PostTrainBench @posttrainbench]

https://t.co/KU9V1mhumz

[PostTrainBench @posttrainbench]

Introducing PostTrainBench

How well can AI agents post-train language models? We built a benchmark to find out.

Post-training is how raw language models become useful, the stage that turns a capable but unsteered base model into a system that follows instruction

[PostTrainBench @posttrainbench]

Since our initial release, we made our benchmark more robust: - added more tasks (ArenaHard-Writing and HealthBench-Easy are new) - ran more seeds - weighted the tasks by difficulty We also added a lot of agents! 3/3

[PostTrainBench @posttrainbench]

PostTrainBench v1.0 and the accompanying paper are out! We believe this benchmark will be important to measure progress in AI R&D automation. What are our findings? 1/n

[PostTrainBench @posttrainbench]

Findings: 1. There was a lot of progress recently (Sonnet 4.5 was at 9.9%, Opus 4.6 has 23.2%) 2. Agents lack behind instruct tuned models by human engineers (51.1%) 3. Persistent agents are usually more performant 4. Agents like to cheat. E.g. Opus 4.6 and GPT-5.1 Codex Max 2/n