Very excited to release Terminal-Bench 2.1!
Coding agents are among the most economically consequential deployments of LLMs to date. As agents improve, benchmark reliability matters more.
We audited TB2.0 and found and corrected issues in 28/89 tasks. 30% of the benchmark!
But the rankings survived, absolute scores moved up to 12pp!
Looking forward to sharing our work Next-Concept Modeling (NCM) at the Discrete Diffusion Reading Group this Monday
Paper: https://t.co/3Vd1tAqopb
Code: https://t.co/UVtKUvd2gU
Turns out my initial version missed a lot of evals. Claude searched harder and found 36% more mentions of benchmarks: https://t.co/ONtgbBuwEs
It still misses blog posts with fancy UI elements, long PDFs, etc. Guess we need a benchmark for creating benchmark benchmark...
Terminal-Bench 4.1 is here with updated verifiers for all tasks.
In this release, we focused on:
- resisting reward hacking
- reducing false negatives
- improving reproducibility
Re-grading the 4.0 leaderboard trials with the 4.1 verifiers, resulted in ~1% of rewards flipping.
Leaderboard scores remain within previously reported confidence intervals.
Some unpopular-but-true notes on pretraining research:
→ 𝗗𝗮𝘁𝗮 𝗿𝗲𝗽𝗲𝘁𝗶𝘁𝗶𝗼𝗻 is a very common practice.
→ 𝗢𝘃𝗲𝗿-𝘁𝗿𝗮𝗶𝗻𝗶𝗻𝗴 (~1000 TPP) is a very common practice.
→ In the long run, 𝗱𝗮𝘁𝗮-𝗲𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁 architectures and algorithms will matter more than (currently) 𝗵𝗮𝗿𝗱𝘄𝗮𝗿𝗲-𝗳𝗿𝗶𝗲𝗻𝗱𝗹𝘆 ones.
→ 𝗙𝗶𝗻𝗲𝗪𝗲𝗯-𝗘𝗱𝘂 is a bad dataset for testing architectures; it can be misleading.
Fable 5.2 shows substantially improved skills in controlling physical robots, even on tasks it has never explicitly been trained to do.
On RoboDojo, a benchmark with a simulated two-arm robot, it outperformed other models and most AI systems built to control robots.
Thrilled to share a big milestone from our AI & Infrastructure team at @Google.
Today we’re expanding our 8th generation TPU portfolio with:
• TPU 8i Max: Built for massive context windows and the highest compute density per rack
• TPU 8tx: Purpose-built for secure, confidential inference
This marks the first time we’ve introduced four distinct TPU systems in a single generation, in one year.
For developers, hardware is only as good as the software stack. And we want you to be able to build with TPUs as easily and efficiently as we do at Google. You can now:
↳ Run native PyTorch & Hugging Face models via TorchTPU (zero code rewrites)
↳ Cut KV transfer latency >90% with open-source TPU Sync
↳ Eliminate idle compute waste with Agent Compute Engine
Read more about our newest TPUs on our blog: https://t.co/8eaFhq3jhi
I look forward to connecting with builders, researchers, engineers and organizations pushing the boundaries of AI and computing!
Nice DeepSWE score! Does it survive a lowercase filename?
Introducing MaxxBench: 89 DeepSWE tasks with small changes to the instructions or starting code. Same engineering problem, slightly different requirement.
Models clustered around 74% on DeepSWE end up 23 pp apart.
Good at coding, or good at DeepSWE? 🧵
New on our Science Blog: How AI is beginning to change scientific modeling.
Scientific models help researchers explain and predict real-world systems—like how a single nerve cell fires, or what happens when neutron stars collide. Recent advances have transformed how scientists fit complex models to data. But scientists still typically write the models manually, and usually need to weigh up several competing versions, each of which might explain the data in a different way.
In a guest post, Jakob Macke, Daniel Gedon, and Stefan Wahl describe how AI is starting to help with these parts of the process. By writing, running, fitting, and revising scientific models as code, AI can explore many possible model structures, and find out which explanations the data support.
https://t.co/qsFJG76FDw
> This lack of benchmarks which are difficult for Model 4 makes us somewhat nervous about our ability to measure AECI now and in the future.
Even with internal preview models we lack hard enough benchmarks to measure progress
Wild!
3. Interviews and surveys with staff are consistent with AI usage driving current acceleration, though this evidence is very hard to interpret. Staff self-report dramatic productivity gains, but say the most important AI R&D contributions aren’t yet fully automated.
An “acceleration” would mean that AI capabilities increase even faster than their historical trend.
Our assessment studied three questions: (1) is acceleration occurring, (2) is it attributable to AI, and (3) are there blockers to near future growth?
https://t.co/ErENKaNEew
Introducing FrontierPhysics - a benchmark evaluating agents for end-to-end physics research
Our core team from physics labs at Berkeley, Caltech, and Harvard recruited 42 physicists from 26 institutions. 28 of them published papers on Nature, Nature Physics, PRX, PRA*
1/7 🧵
Today we release Ripple-65B Preview, our 65B dense LLM we pretrained from scratch.
Ripple sets a new state-of-the-art for U.S. open-weight models in agentic coding tasks, surpassing significantly larger models including Muse Spark Open-816B, Beam-501B, Inkling-975B, and Nemotron 3 Ultra-550B.
Learn more: https://t.co/Pe04ZKhkE9
Introducing the Together Compiler: continual compilation for AI inference! I’m super proud to finally share this work from our research team! ✨
Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in the last three months and the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur.
Together Compiler research highlights include:
1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems.
2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2.
3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation.
4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions.
From an incredible team @reyna_abhyankar@dylan__lim@realDanFu@togethercompute!!
Today we release Ripple-65B Preview, our 65B dense LLM we pretrained from scratch.
Ripple sets a new state-of-the-art for U.S. open-weight models in agentic coding tasks, surpassing significantly larger models including Muse Spark Open-816B, Beam-501B, Inkling-975B, and Nemotron 3 Ultra-550B.
Learn more: https://t.co/Pe04ZKhkE9