We’re launching Senior SWE-Bench today: a new open benchmark that evaluates AI coding agents as senior engineers. Realistic instructions, long horizons, and code quality assessments. Built at Snorkel AI in collab with @Princeton and @UWMadison.
Claude Opus 4.8 is the leader with 24% high quality solves, but it needed 117K tokens on average to get there. Claude Sonnet 5 almost swooped in for the win, but we found it cheated on 26% of trials (analysis on the site). GPT-5.5 is the most efficient high performer, using only 36K tokens at its highest effort level to land at #2.
Led by @henryehrenberg with @vincentsunnchen (Snorkel); @karthik_r_n and Austin W. Hanjie (@Princeton); @gorlanski and @fredsala (@UWMadison).
We expect agents to act like senior engineers, but most benchmarks still evaluate them like interns.
Excited to introduce Senior SWE-Bench, an open-source and @harborframework-native benchmark that assesses agents as senior engineers on long-horizon tasks with realistically under-specified instructions.
We expect agents to build real features going on just a quick Slack message, nothing like the super technical instructions most benchmarks provide. Senior SWE-Bench fixes that.
Claude Opus 4.8 is the current leader at 24% high quality solves, but it took 117K tokens on average to get there. Claude Sonnet 5 looked like it was going to swoop in for the top spot, but we found it cheated on 26% of trials.
Only 2 days until we’re in London for @aiDotEngineer 🇬🇧
Come connect with the Snorkel AI team and fellow leaders shaping the future of AI agents – over drinks and conversation 🍻
Last chance to register for Tuesday’s Happy Hour at Bantof: https://t.co/aZmb6vWPGW
London 🇬🇧
We’re heading to @aiDotEngineer Europe (April 8–10). @vincentsunnchen will be presenting on the art and science of benchmarking agents, including learnings from Open Benchmark Grants.
Find us at Booth G9:
https://t.co/VouHA6Ep8h
Miami's UNREAL playoff run continues
-Lost first play-in game to Hawks
-Beat Bulls in play-in finale for No. 8 seed
-Knocked out No. 1 seed Bucks in 5
-Beat No. 5 seed Knicks in 6
-Beat No. 2 seed Celtics in ECF
-First play-in team to make the Finals
-Second No. 8 seed to ever make the Finals