Jasmin Kern

AI Benchmark Research PM at Mercor

[Jasmin Kern @JasminLovesAI]

@logorrhythmic Fascinating work! I wonder how we can incentivize researchers to create benchmarks to cover parts of the economy that are currently less affected or accepting of AI.

Proving usefulness in professions like Kindergarden teachers or Aircraft producers may spur more AI adoption.

[Mercor @mercor]

Mercor Research mapped AI benchmarks to actual US jobs.

Only 208 of 923 occupations in the US economy are represented by existing AI benchmarks. 90% of hours worked in the US aren't covered well by a relevant AI benchmark.

Today, we’re releasing the 𝗕𝗲𝗻𝗰𝗵𝗺𝗮𝗿𝗸 𝗠𝗮𝗽 𝗼𝗳 𝘁𝗵𝗲 𝗨𝗦 𝗘𝗰𝗼𝗻𝗼𝗺𝘆, an interactive breakdown of which economic tasks are covered by AI benchmarks.

Benchmarks are how we measure AI’s progress and anticipate its impact on real work. They’re crucial to understand which jobs AI can automate, where the biggest gaps are, and how those capabilities are changing over time.

The Benchmark Map is our first step toward building a resource that makes it easier to answer: 𝘋𝘰𝘦𝘴 𝘵𝘩𝘪𝘴 𝘈𝘐 𝘮𝘰𝘥𝘦𝘭 𝘸𝘰𝘳𝘬 𝘧𝘰𝘳 𝘮𝘺 𝘰𝘤𝘤𝘶𝘱𝘢𝘵𝘪𝘰𝘯?

Explore it here: https://t.co/cDmR4S156o

[Chamath Palihapitiya @chamath]

Model Benchmarks Are Worthless

How can a model score 24% and 59% on the same 113 coding tasks?

It happens when you vary the harness that wraps it.

I’ve increasingly ignored most of the recent model leaderboard results. Partly from exhaustion and partly because improvements in a ranking don’t show up in the actual work.

So I asked the 8090 Labs team to measure what the leaderboards skip: the model and the harness together, on real bug fixes and features from open-source projects.

8090 Labs ran 7 models through 5 harnesses.

Here are a few results I didn't expect:

- Anthropic's Claude Fable 5.1 scored 41% in Claude Code and 61% in OpenAI's Codex. It made 28 tool calls at the median in one and 66 in the other.

- The cheapest runs weren't the cheapest work. GLM in Codex cost $0.22 a run and $0.93 per success. In Pi it cost $0.25 a run and $0.43 per success.

- GLM passed a task on both attempts 51 times in one harness and 16 times in another.

- GPT-5.6 Sol and Grok 4.6 barely cared which harness they were in.

The bottom line is that if you're picking a model/harness combo off a model-only leaderboard, you're making a critical decision based on only half the information.

https://t.co/ivLJ02gLLz

[Tibo @thsottiaux]

We are going to need a different AGI benchmark. Where is the goalpost moving next?

[Mercor @mercor]

Where do frontier models fail?

Working with accounting experts, we categorized every failed run: reasoning, information gathering, instruction following, and planning and reflection.

Roughly seven in ten failures by the top three models are reasoning failures. Information gathering accounts for far fewer. Agents usually find the right numbers, reach a correct intermediate answer, then drop or contradict it later in the workflow.

Better retrieval won't fix this. It takes stronger accounting judgment and more discipline in carrying conclusions through to the final entry.

[Mercor @mercor]

Spending more on compute doesn't guarantee a higher score.

We tested every model at budgets of $1, $5, $10, and $50 per task. At $50, Fable 5 spends about $32 per run and Muse Spark 1.1 spends about $5, yet their scores land within 4 points of each other. Cut Fable 5's budget to $1 and it collapses to 11.8%. Muse Spark barely moves.

On Pass@8, whether a model solves a task at least once in eight tries, Muse Spark 1.1 actually edges ahead: 21.5% to Fable 5's 20.1%.

For accounting work, the most expensive model is not the most reliable one.

[Mercor @mercor]

It's month close. AI models can pass the CPA exam, but do you trust them to close the books?

Introducing APEX-Accounting, built in collaboration with @tryramp and @RampLabs, to test whether frontier AI can do real accounting work.

It’s a new benchmark comprising 160 tasks across 10 simulated companies, authored and graded by accounting professionals with experience at firms like Deloitte, PwC, EY, and KPMG.

[Jasmin Kern @JasminLovesAI]

@Manu_Sisti Claude