For folks who haven’t been following Sierra’s τ³-Bench, it’s a tough test of what AI agents can actually do. Frontier models have been benchmaxxed on it, which makes this result especially striking.
What catches my attention is the architectural shift behind Cortex and the leap it made beyond the previous state of the art.
Sharing an early result: Sierra verified Cortex at 73.2% on τ³-benchmark.
We build algorithmic complements to LLMs. We don’t fight what models do well. We fill in the rest.
https://t.co/MmTS0HVc3E
Strong thesis. Relay races are linear, while most real-world problems are dynamic - goals shift, new actors enter, and the path changes. The breakthrough isn’t just agents that run longer, but agents that continuously adapt without losing context or intent.
The three GTM paths aren't really alternatives; they look like a sequence a founder could follow.
@Aleph__Alpha German engineering went from setting the global standard to needing its own affirmative-action category: “fantastic, for a German model.”
How the mighty have fallen.
Benchmarks are basically standardized tests.
Everyone knows you can learn to test well, and that the score is an imperfect proxy for actual ability.
But you can’t just opt out. Someone else will show up tomorrow with a better number and claim to beat you at your own game.
Benchmarks are gameable, but indispensable.
The real call to action: we need a better way to measure actual capability.
@GoogleDeepMind The last few Gemini models were benchmaxed poorly and the results didn't match their capabilities.
@benchmarkheaven looking forward to your verdict on Argon!
@saranormous How much of the “cost collapse” survives contact with production?
An 80–90% gain on a prototype can shrink to ~10% once you include production-ready software.
Before we reprice the software industry, I’d want to see the savings on building production-ready systems.
We entered a crowded market and everyone warned us about competition. We however went deep with problem- and productioninzed. Along the way we uncovered what we had solved for one the truly hardest problem that exists and it was not just horizontal but also super aligned to us as founders!
The lens you come in makes huge difference as well as how deep you go! Looking at competition would have distracted us and adding value helped us unlock to this big pivot!
@OpenAI Well done Astra!
However, this is not a singularity; it just proves we have gotten really good at benchmaxxing!
Really love the thought behind @arcprize; time to redefine AGI bench and not let hyperscalers post-train on it!
@jeremyberman Congrats and thanks for sharing the code!
Also makes me wonder, should AGI be removed from ARC.
Also, what is the next benchmark to look at if you care about AGI?