Surge AI

Our mission is to raise AGI with the richness of humanity — curious, witty, imaginative, and full of breathtaking brilliance.

[Surge AI @HelloSurgeAI]

Learn more about sudo L7: https://t.co/dK307R16ap

[Surge AI @HelloSurgeAI]

When designing sudo L7, we deliberately made the prompts natural and short.

Just 76 words median, compared with 291-427 words across DeepSWE, FrontierSWE, and SWE-Marathon.

Think about a request like “Can we safely cache this?”

To answer it, an engineer needs to understand the system, identify which data can become stale, and decide whether caching is even the right solution.

All of that is unstated in the real world.

Agents need to learn how to handle the decisions engineers have to make when you don’t get a long list of instructions spelled out.

[Surge AI @HelloSurgeAI]

When we started our public benchmarks effort, our Research team’s goal was to give the industry better North Stars.

Happy to see Chartography and GDP.pdf take the top two trending spots on @paperswithcode.

[Niels Rogge @NielsRogge]

One thing I like about @paperswithcodee is how it automatically surfaces trending benchmarks for me

e.g. right now the top benchmark for image understanding is Chartography, which includes 100 professional chart-understanding tasks from @HelloSurgeAI

I had no idea this benchmark existed, but both Anthropic and DeepSeek are reporting evals on it for their new models

Find it here: https://t.co/pxwrAzgIj0

[Surge AI @HelloSurgeAI]

@NielsRogge @paperswithcodee Thanks for the shoutout, @NielsRogge!

You might also be interested in GDP.pdf, right below it 👀 It’s been showing up in recent model cards from Anthropic, OpenAI, and Google.

Let us know what you think!

[echen @echen]

one of my fav findings from sudo L7:

opus 5.5 and gpt-6.1 sol are basically tied at writing correct code. but opus is much better at the behaviors that get you promoted to staff.

on functional correctness:
opus 82.1% ~ sol 82.4%

on thought partnership (e.g., telling you when there’s a better approach than the one you asked it to build):
opus 76.1% >> sol 53.0%

on communication (explaining its decisions and the tradeoffs it makes, plus any limitations):
opus 66.4% >> sol 49.4%

put differently: imagine promoting an engineer because their unit tests pass. that's basically how we've been evaluating coding agents.

but staff engineers do a lot more than that. they also:
- question bad assumptions
- suggest better approaches
- think about tradeoffs
- and help everyone around them make better decisions.

that’s what we want the best agents to do too, and that's what we built sudo L7 to measure.

[Surge AI @HelloSurgeAI]

Could AI actually solve the Riemann Hypothesis?

That's the question that inspired Riemann-bench. We wanted to go beyond Olympiad math and test whether AI could tackle difficult, research-level problems from top mathematicians.

Six months ago, models scored under 5%. Today, they're above 70% (10,000 agents not allowed!), and Navier-Stokes and UGC have now been solved.

With all the recent breakthroughs, we have an internal betting pool with two predictions: the Riemann Hypothesis will be solved within a year, and P vs NP will be the last Millennium problem to fall.

Great to see Riemann-bench featured in this year's @stateofai Report, and who can even imagine next year's?

[Surge AI @HelloSurgeAI]

Check out the leaderboard: https://t.co/n48mE6P5Uz

[Surge AI @HelloSurgeAI]

We've added Gemini 4 Argon and Muse Wonder to the sudo L7 leaderboard:

Gemini 4 Argon: 18.9% (#7)
Muse Wonder: 15.6% (tied #8)

Claude Opus 5.5 and Sonnet 5.5 continue to lead at 41.7%.

Today's frontier coding agents can write remarkable code. So can a new PhD. Writing great code and being a great engineer are very different things.

Can an agent write idiomatic Rust? Recognize that logging sensitive banking information or storing passwords in plaintext is a terrible idea, even when nobody explicitly warns against it? Proactively warn when a new feature request introduces security risks?

That's what sudo L7 tests, using real-world engineering tasks in private production codebases. Beyond functional correctness, agents have to exercise architectural judgment, anticipate risks, communicate tradeoffs, act as thought partners (don't you wish someone could partner with your PM?), and recognize when technically correct code still isn't ready to ship.

Coding agents are getting very good at writing code. Getting promoted to L7 is the next frontier.

[Surge AI @HelloSurgeAI]

Real software engineering happens in repositories with years of accumulated history.

Legacy code nobody wants to touch. Historical accidents that became architectural decisions. Dependencies that break in surprising ways.

That's why most sudo L7 tasks are set in privately licensed production codebases from real companies.

The tasks are written by staff and principal engineers, former CTOs, and people who've built production infrastructure at places such as Google, Meta, GitHub, and NASA JPL.

Every task starts from a meaningful failure observed in a frontier model. Each one is graded on eight dimensions, from functional correctness to knowing when to push back on a flawed request.

We built sudo L7 to measure the judgment that separates a capable coder from a staff-level engineer.

[echen @echen]

another Surge benchmark on the Muse Wonder model card 👀

Muse scores 33.2% on GDP.pdf. #3 on our leaderboard, right behind Astra

nice work @Meta!

[Surge AI @HelloSurgeAI]

🇫🇷 Le Chonk joins the sudo L7 leaderboard 🇫🇷

We just added @MistralAI's Mistral Large 4 to sudo L7, our benchmark for staff-level software engineering.

Mistral Large 4 scores 13.3%, ahead of Muse Spark 1.3 (12.2%) and Grok 4.7 (10.6%).

sudo L7 tests whether coding agents can go beyond writing correct code to exercise the judgment of experienced engineers: navigating messy systems, spotting hidden risks, and knowing when technically correct code is still the wrong thing to ship.

Allez, le Chonk ! 🥖

Leaderboard ↓
https://t.co/ouCzU2m8mT

[Lenny Rachitsky @lennysan]

Sweet new benchmark from Surge looking at how close AI is getting to a staff-level software engineering. We're half-way there.

[Surge AI @HelloSurgeAI]

An AI agent found every expiring rent discount in a commercial real estate workbook.

Then it missed three break clauses representing roughly $14 million in forecast rent.

The task required following two different paths through a 15-tab workbook. The model followed one and stopped.

Knowing when you’re done is part of the job.

https://t.co/wfNpWtYPzR

[Surge AI @HelloSurgeAI]

Introducing sudo L7: Can Coding Agents Act Like Staff Engineers?

Coding agents are getting very good at solving well-defined problems. But staff engineers do much more than write correct code.

They understand unfamiliar systems, make architectural decisions, spot risks the ticket never mentioned, and know when passing the tests isn't enough.

sudo L7 measures that judgment.

Our new software engineering benchmark contains 60 expert-authored tasks, primarily in private production codebases from real companies, complete with legacy code, technical debt, and messy dependencies.

We evaluate eight dimensions of engineering behavior, from functional correctness and engineering craft to practical judgment, thought partnership, and knowing when to push back.

The best-performing agents score under ~45%.

Coding is only part of the job. See you next cycle.

Explore sudo L7: https://t.co/ipmwvaaVPZ

[echen @echen]

introducing sudo L7, our new benchmark for staff-level software engineering.

coding agents are getting very good at being L3s. give them a well-defined ticket and they can write excellent code.

but a staff engineer is valuable for everything that happens around the code.

what should we build? what already exists? what's going to break in prod? what did the tests miss? should we even be doing this?

sudo L7 has 60 tasks, mostly in private production repos from real companies, written by engineers who've actually owned these systems.

expert rubrics grade everything from functional correctness to engineering craft, architectural judgment, thought partnership, and unnecessary complexity.

the best agents succeed on only ~45% of tasks.

coding is only part of the job. for now, we're keeping them at idiot genius L3.

[Surge AI @HelloSurgeAI]

Only 2 of 66 runs caught a $25.7M pricing error.

One screenshot. One unit mistake. A 100× overstatement.

See what else frontier models miss on DAYJOB: Finance:
https://t.co/0qKiYrYZCY

[Surge AI @HelloSurgeAI]

For their Mistral Large 4 release, @MistralAI ran a blind Surge human eval, with our expert software engineers evaluating five frontier models on coding quality.

Le Chonk ranked #1 among open-weight models and #2 overall, behind only Opus 5.

Unit tests ask: does it work?
Human code review asks: would you merge it?

Our evals measures the difference. Working code is the floor; professional taste and judgment are what make it shippable.

Congrats to the Mistral team on le Chonk and their big release 🇫🇷

[Mistral AI @MistralAI]

Meet Mistral Large 4, aka Le Chonk.

• 1T parameters, natively multimodal. 49B active.
It is the best open weights model from US or Europe on aggregated benchmarks.
• State-of-the-art on critical workloads, including cyber defense, manufacturing and finance and it surpasses closed frontier models on visual grounding.
• Forged in Europe end-to-end and is deployable from Europe via our own Mistral Cloud infrastructure.
• Available to all via API today. Working with cybersecurity partners privately.

Open weights release end of October.

[echen @echen]

🥖 LGTM, le Chonk 🥖

in their latest Mistral Large 4 release, @MistralAI ran a blind Surge human eval across five frontier models - professional software engineers comparing five frontier models on coding quality. the big guy finished #2, first among open-weight models and behind only Opus 5.

most benchmarks can narrowly measure correctness. but human evals get at the deeper stuff: the professional taste, judgment, and maintainability we look for in the real world as well

the difference between passing a set of unit tests vs. getting a "LGTM - ship it"

bien joué, le Chonk ! congrats to the entire Mistral team 🇫🇷

[Ritvik Pandey @ritvikpandey21]

A big focus for our team at @Pulse__AI is making the data AI agents receive more useful.

Huge congrats to the @HelloSurgeAI team on GDP.xlsx, a strong benchmark testing spreadsheet reasoning across finance, engineering and healthcare.

With Ultra 2, our latest model, we’re focused on preserving hidden content, formulas, complete records and visual context within these enterprise documents. We evaluated Pulse’s spreadsheet extraction pipeline using the benchmark’s OpenHands agent harness, providing structured text and image evidence upfront.

Our latest reported task-pass rates, compared with our earlier Pulse-assisted results:

- Claude Opus 5.5 46.4% (up from 41.4%; +5 percentage points)
- GPT-6.1 Sol 37.9% (up from 32.9%; +5 percentage points)
- GPT-6 Astra 35.0% (up from 30.0%; +5 percentage points)
- Claude Fable 5.1 33.6% (up from 28.6%; +5 percentage points)
- Gemini 3.8 Flash 23.6% (up from 18.6%; +5 percentage points)

Better data gives agents more to work with, read more here: https://t.co/4JFNPQaLd8

[echen @echen]

GDP.pdf just got a little brother: introducing GDP.xlsx

the world runs on spreadsheets. somewhere in Sheet2!D18 there’s a vlookup that changes a $14m decision.

GDP.xlsx has 70 real-world tasks across 12 domains. what’s hard is figuring out how the workbook even works:

→ which tab is stale?
→ what does a red cell mean?
→ which comment explains the exceptions?
→ which chart does Finance actually trust?
→ did you remember to check the termination clauses in Sheet 3?

the best frontier agent scores 38.3%.
(congrats Gemini 4 Argon!)

this one’s for you, Kelly.