And here it is, our GPT-6.2 Sol benchmark, all the details and link to traces available.
Proud to be an independent benchmark that (so far) doesn't seem to be used by any lab to benchmaxx against.
Lots of good signal in this one, GPT-6.2 Sol is definitely a strong choice for routine daily coding work, congrats to the OpenAI team, very impressed with this one on cost and performance esp. at lower effort levels.
Okay, woke up to lots of snow, and the results of my GPT-6.2 Sol benchmark on @VulcanBench ๐
Very impressed with this model, it is clearly a strong daily driver, and saw noticeable improvements at low and medium effort compared to GPT-6.1 Sol.
As I've said many times, but worth saying again. A score between 80 - 90 on VulcanBench Frontier v4 is the range you want to be for routine daily work.
And, once you compare it to Fable 5.2 and Opus 5.5, it's hard not to see how cost-effective GPT-6.2 Sol is.
I'm impressed, will definitely make it into my stack for daily coding tasks. Congrats to the whole team at OpenAI, really neat to see the improvement specifically at lower effort levels too.
Everything is now live on the VulcanBench site:
https://t.co/JtEEXkYWyX
And of course, as a free, open source, independent benchmark, I also share the traces so you can see everything the model did across every task, every run:
https://t.co/ycHkmfhkio
Live long and benchmark ๐
Weโre releasing Multi Agent Coordination Bench, a benchmark of twelve natively multi-agent long-horizon tasks that measure the coordination capabilities of frontier models.
We find that current frontier agents coordinate poorly. A team of eight agents beats a single agent, but the gain comes from having more agents, not from coordination. Of Opus 5.5, GPT-6.1 Sol and Gemini 4 Argon, only Opus shows signs of a gain from coordinating.
@AdamHoltererer very interesting behavior, are you sharing the traces? Would be super interested to look at them and see what the heck happens at high ๐
Okay, so I am making (some) progress with my Computer Use v1 eval suite for @VulcanBench
It's still not there yet, more tasks to add, but already getting some really interesting signal, and experimenting with different ways to share the results.
Here's an early look, note, still need to add harder tasks, review traces in more detail, etc. but as many of you know, I like sharing wayyyyy too early, so here ya go.
Please give me feedback, how do you like this format for a model card? What kind of tasks am I missing from this eval suite that you'd want to know about?
We like giving the power to the people to pick ๐
But not to worry, youโll always have an option to have someone else run a model router for you, lots of company happy to charge for that!
But people like Morgan have fun optimizing and not paying someone else to make the routing decision.
And here it is, our Fable 5.2 benchmark, and a new feature, a thinking report! ๐ง
Can confirm, no better way to start a Saturday than a fresh benchmark ๐
Okay, the day has come, my Fable 5.2 Benchmark with @VulcanBench is now complete ๐
This is the first model to clear every task, and thus, am calling it, "the Frontier v4 killer"
And honestly, if something was going to kill this eval suite, it's an honor to have Fable be the one to do it.
That being said, VulcanBench is all about testing models on real engineering tasks, that teams like mine would use a model for every day, to really understand first, if you need the model at all, and second, what effort level you need if you do.
I say anything in the 80 - 90 range on Frontier v4 is strong enough for daily work, and that is why, if you use Fable 5.2, Low effort is likely all you need for routine daily engineering tasks.
But you'll pay a price, as you can see from the cost-per-task chart, it isn't cheap. But it does think in amazing ways.
Which is why I'm also introducing a brand new section on VulcanBench where I share analysis on how a model thinks...and yeah, hard to think of a better model to kick this off with than Fable.
With that, here's how you access the benchmark and the thinking report if you want to do a deeper dive into either or both.
VulcanBench on Frontier v4:
https://t.co/6BOEw6QQaA
How Fable 5.2 Thinks Report:
https://t.co/NCWzDh2BHm
Live long and benchmark ๐
Might delay this by a day or two, still reviewing traces, and putting together a new deliverable from VulcanBench that I'm probably going to call:
Thinking Reports ๐ง ๐
Looking at the traces for a model like Fable is something that for me, has become in many ways like appreciating art.
That might sound overly dramatic, but it really does feel that way.
And while I know, of course the most interesting thing to most ppl is the actual benchmark report, i.e. showing how a model performs from an accuracy, cost, and timing standpoint...maybe there are other people like me that want to understand more about how the model thinks when it's solving a task in an eval suite, and how this varies from model-to-model.
I can think of no better model to start this "Thinking Reports" project with, than Fable 5.2.
So expect to see this sometime over the weekend, going to take more time to really do this right.
Live long and study how models think ๐
Fable 5.2 benchmark results are in, reviewing the traces in detail now.
Publishing model cards in the morning.
You thought Ocarina of Time was a masterpiece, wait until you see the traces for this one ๐