Okay, woke up this morning and my Sol benchmark was finished, so I have now completed my full run, on the five core OpenAI models available today, across all effort levels.
Through this process as I shared a bit last week, I also decided to create a third eval suite, focused on routine engineering tasks.
This current suite, which I'm now calling VulcanBench Frontier, is really, most likely, harder tasks that regular engineers on engineering teams are giving models on a normal day.
What I'm testing with this eval suite is how these models do with hard stuff, things you might give a model, but not likely on a daily basis. Which means I would look at these results as providing signal for what model/effort to use for your hardest tasks.
Also as a quick reminder, or heads up for those just joining me here. With this v4 eval suite I also included a code quality scoring system that uses both Muse and Grok to score the code quality and include this as 33% of the score.
I think code quality output for models is going to matter more and more over time, not because models need to write good code for humans, but because models need to write code other agents can understand and work on too.
As for key insights I got from this, here's three:
1. Increasing effort level does not seem to impact code quality. So anyone thinking Low effort writes crappy code and Max writes beautiful code, that doesn't seem to be the case. Low and Max in pretty much every OpenAI model writes the same quality code.
2. Terra Max matches Astra at half the price. I'll just leave that here 👀
3. You never need to use GPT 5.5, and if you are, in any workflows, stop, you're wasting money.
Comparison model cards below, and if you want to see detailed reports on each specific model run, you can find those here:
https://t.co/zM6xmollED
So... we decided to hack WordPress Core, AGAIN! 🔥 ⛓️ Click2Shell is a one-click unauthenticated remote command execution chain (Preauth RCE) affecting every WordPress website. Wordpress rolled out a fix yesterday! The story about how one preview link made WordPress click Install, load an inactive theme's PHP, and hand us RCE is below. Happy Friday!
https://t.co/91D2F2lSdb
Here is the answer I was looking for
Buried in the report
Bonsai 2 27B's accuracy on terminal bench and swe bench drops by nearly 20 points.
It's all I needed to know and what they should have published from the beginning in their main benchmark table.
Today, we’re announcing Ternary Bonsai 2 27B.
Based on Qwen3.8 27B, Bonsai 2 27B is 9x smaller than its full-precision counterpart while retaining 98.2% of its aggregate benchmark performance.
Two months after the first Bonsai 27B release, the biggest change is quality. The footprint remains 5.9 GB, but the gap to full precision has narrowed materially, with particularly strong gains in agentic coding, multimodal reasoning, and long-horizon tool use.
Ternary Bonsai 2 27B is available today under Apache 2.0.
We've had a number of Go vs Rust debates over the years on our team. We think a lot about performance since we're building an infra service, and the performance question basically comes down to: where does your software spend the most time?
It's classic Amdahl's law: for our case, 90% of our time is spent in the database layer. Even if we got a 2x improvement on the 10% of the time we spend in the actual application runtime, which is written in Go, we're still only improve a little over 5%. Our total performance improvements from an expensive rewrite would be bounded at a little over +10%.
On the other hand, if we throw all of our resources into optimizing the database layer, we could get massive improvements, up to 9x! It would even make more sense to throw tons of resources at something like pg_rust from @mmalisper -- which we're watching closely -- over rewriting our Go app in Rust. We're obsessive about the database layer for this very reason, and less obsessive about the language.
This type of 90/10 split, where 90% of the time is spent on the durability layer, is the sign of the healthy application which needs no rewrite.