Jonathan Roberts

Evals @MistralAI | Previously, PhD @Cambridge_Uni

[Jonathan Roberts @JRobertsAI]

Is computer vision “solved”?

Not yet

Current models score 0% on ZeroBench

🧵1/6

[Jonathan Roberts @JRobertsAI]

📢 Muse Spark Open w/tools scores 50% pass@5 on ZeroBench

[Alexandr Wang @alexandr_wang]

1/ muse spark open is the most capable US open-weight model to date.

816B total params, 42B active. 1M-token context. natively multimodal. built for coding and agentic work like long-horizon tasks, hard software engineering, multi-step reasoning.

america! 🇺🇲

[Jonathan Roberts @JRobertsAI]

👀 DeepSeek-V4.1-Flash-1017 scores 51% pass@5 on ZeroBench w/ tools

[DeepSeek @deepseek_ai]

🚀 DeepSeek-V4.1-Flash-1017 is here!

Stronger benchmark performance, faster task completion, and shorter outputs than the previous version—a Pareto improvement in performance and cost. Now live on our website, app, and API.

🛠️ Optimized for @DeepSeekHarness, Tencent CodeBuddy, WorkBuddy, and more. Upgrade to DeepSeek Harness v0.2.1-rc.1 or later for the best experience with the new model.

1/4

[Samuel Albanie 🇬🇧 @SamuelAlbanie]

nothing to report

opus 5.5 max looks on/above trend for the ai futures model

(added to the eci graph from the august update)

wide CIs yada yada

[Mistral AI @MistralAI]

Mistral Large 4 is one of the world's strongest AI models for cybersecurity.

[Mistral AI @MistralAI]

Meet Mistral Large 4, aka Le Chonk.

• 1T parameters, natively multimodal. 49B active.
It is the best open weights model from US or Europe on aggregated benchmarks.
• State-of-the-art on critical workloads, including cyber defense, manufacturing and finance and it surpasses closed frontier models on visual grounding.
• Forged in Europe end-to-end and is deployable from Europe via our own Mistral Cloud infrastructure.
• Available to all via API today. Working with cybersecurity partners privately.

Open weights release end of October.

[Jonathan Roberts @JRobertsAI]

More details on the project page: https://t.co/E4noN7yDDM

[Jonathan Roberts @JRobertsAI]

Opus 5.5 (max) is no.2 on ZeroBench

43% pass@5
25% pass^5

Now, 77% of questions have been solved at least once

[Jonathan Roberts @JRobertsAI]

Project page: https://t.co/E4noN7yDDM

[Jonathan Roberts @JRobertsAI]

All three GPT-6 models sit on the ZeroBench performance-cost pareto frontier

pass@5 | pass^5

GPT-6 Astra (max)
51% | 35%

GPT-6 Sol (max)
41% | 17%

GPT-6 Luna (max)
20% | 7%

[Jonathan Roberts @JRobertsAI]

@Aging_Scientist The "if" is cautionary because the jump is so large. Given the benchmark is only softly gated, optimisation/benchmaxxing is possible. Qualitatively the responses do look genuine though

[Jonathan Roberts @JRobertsAI]

GPT-6 Astra (max) on ZeroBench:

pass@5: 51% (prev. SOTA: 30%)
pass^5: 35% (prev. SOTA: 13%)

If this reflects a genuine capability improvement, it is a significant step forward in both performance and consistency

[Jonathan Roberts @JRobertsAI]

@anh_ng8 Thanks @anh_ng8 - also curious to see this

[Jonathan Roberts @JRobertsAI]

@flauneun Great suggestion - thank you!

[Jonathan Roberts @JRobertsAI]

@AkashGu30808281 Yeah - in the Leaderboards tab of the project page select tokens/question. Astra is pretty efficient

[Jonathan Roberts @JRobertsAI]

@AkashGu30808281 Yes

[Jonathan Roberts @JRobertsAI]

More details: https://t.co/E4noN7yDDM

[AI at Meta @AIatMeta]

Now let’s take a look at how Muse Spark 1.2 performs across visual reasoning, chart understanding, and knowledge-intensive tasks.

[Jonathan Roberts @JRobertsAI]

Impressive! Muse Spark 1.2 w/ tools surpasses 50% pass@5 on ZeroBench!

[Alexandr Wang @alexandr_wang]

2/ muse spark 1.2 performs quite strongly across a wide variety of multimodal capabilities and evals.

[Jonathan Roberts @JRobertsAI]

Project page: https://t.co/E4noN7yDDM

[Jonathan Roberts @JRobertsAI]

Grok 4.6 (xhigh) on ZeroBench (no tool use):

pass@5 / pass^5

Grok 4.6: 17% / 7%
GPT-5.6 Sol (SOTA): 30% / 13%

More evaluation details available on the project page