Seldon

Advancing multimodal AI & computer-use

[Seldon @seldon_tech]

How good are agents actually at CAD?

Today we release CADBench, evaluating 10 Models on 105 realistic tasks in Fusion 360

Fable 5 and Gemini 3.7 Flash are #1 with only 24.6% pass rate

[Seldon @seldon_tech]

@mattsilv Sure! Coming soon

[Seldon @seldon_tech]

How good are agents actually at CAD?

Today we release CADBench, evaluating 10 Models on 105 realistic tasks in Fusion 360

Fable 5 and Gemini 3.7 Flash are #1 with only 24.6% pass rate

[Seldon @seldon_tech]

@CWong24871 Coming soon!

[Seldon @seldon_tech]

@swkkadjo @productflo_io Coming soon!

[Seldon @seldon_tech]

@thinkymachines Inkling struggles a lot with agentic computer use, especially with CAD

[Seldon @seldon_tech]

@samsja19 https://t.co/46M136nt1Y

[Seldon @seldon_tech]

We're hosting another event in San Francisco coming monday, this time a research talk with @samsja19 as guest speaker

Drop by if you're in town!

Sign up link is in the comments

[Seldon @seldon_tech]

Read the full report here:
https://t.co/cuBI1oymAb

[Seldon @seldon_tech]

Gemini 3.7 Flash delivers performance comparable to Fable 5 at a fraction of the cost

We see that many models finish the tasks prematurely, resulting in lower scores and costs

[Seldon @seldon_tech]

@GoogleDeepMind and @Alibaba_Qwen released new models this week. These are the results:

Gemini 3.7 Flash is the new State-of-the-art model on VGI-Bench.

Qwen 3.8 Max comes in as fourth place in after Gemini 3.6 Flash.

[Pascal Bertrand @PascalBtd]

drop by our meetup today if you're in SF!
https://t.co/1mZL7DmG7T

[Seldon @seldon_tech]

Full blog-post about this benchmark: https://t.co/Nubh2QdA96

[Seldon @seldon_tech]

Infinibench-v0: part of the benchmark was created by a continuously exploring agent, rewarded for identifying new failure modes in VLMs.

It produces non-trivial, out-of-distribution problems — trivially easy for humans, yet SOTA VLMs fail — and diagnoses failure modes that were previously unknown.

[Seldon @seldon_tech]

To celebrate the release of this benchmark we'll host a meetup in SF. If you're in town, sign up here: https://t.co/FQUEClYOlG

[Seldon @seldon_tech]

The 12 skills:

[Seldon @seldon_tech]

Our dataset consists of 398 public videos and programmatically generated videos, the latter to avoid models having been exposed to the test data at train time, spanning just under 100 hours.

[Seldon @seldon_tech]

Introducing VGI-Bench: a multimodal, holistic benchmark probing 12 distinct visual and audio-visual skills.

550 human-curated questions, designed to mitigate the common mistakes in today's video benchmarks and expose pragmatic failures of state-of-the-art models.

Best model: 64.73%. Humans: 84.5%.

[Seldon @seldon_tech]

Every question in our final dataset must pass two gates, in order:

-> Not solved in pass^3 by a blind, text-only model
-> Not solved in pass^3 by an older baseline VLM (Gemini 2.5 Flash-Lite)

This ensure that every questions tests actual visual understanding

[Seldon @seldon_tech]

VLMs are today's de-facto standard for visual reasoning. They are widely applied in embodied systems like humanoid robots and self-driving cars, and serve as the visual backbones for computer-use and design agents.

This demands diagnostic benchmarks that show exactly where a model can be trusted and what remains difficult.