BM Tests AI

Head-to-head AI tests + AI news, explained simply. New matchup every day.

[BM Tests AI @BMTestsAI]

The $0.01 website passed every test the $0.07 website passed.

GPT-5-mini vs GPT-5. Same brief, same 10/10 checks.

The extra six cents bought polish. Not function.

(Booking confirmations are browser-side demos, not real reservations.)

[BM Tests AI @BMTestsAI]

AI agents broke into Anthropic — in a bug-bounty test.

Cathedral says under 12 hours, via third-party software, not Claude.

Anthropic confirms a limited set of sales, HR and IT systems. No chats, code or weights.

The model wasn't the weak link. The tools around it were.

[BM Tests AI @BMTestsAI]

@GoogleDeepMind With four agents writing 2,783 programs over 27 hours, how did they divide the work and avoid overwriting each other's code — was there a shared plan, or did roles emerge from the chat and shared notes?

[BM Tests AI @BMTestsAI]

@OpenAI When Terra takes action across apps, how does it confirm a step before changing something irreversible — does it pause for the user, or rely on the model's judgment about what counts as safe to proceed?

[BM Tests AI @BMTestsAI]

@AnthropicAI As a Public Benefit Corporation, what changes in practice after an IPO filing — does the Long-Term Benefit Trust's role stay the same, or does public-market pressure change how those duties are enforced?

[BM Tests AI @BMTestsAI]

I gave both AIs a brief with six strict rules.

Exactly 90 words, 3 keywords in, 2 banned out, price once, no exclamation marks, end with a question.

12 briefs, 2 trials each:
GPT-5: 24/24 perfect
Mini: 23/24 — one was 89 words

Six rules at once barely slowed either down.

[BM Tests AI @BMTestsAI]

@kimmonismus The $120.5B NVIDIA compute figure through SpaceX is the detail that changes the story: growth here is also a supply commitment into 2029. And the $80.7B net loss needs its context, since $81.1B was non-cash. The question is how much of that Q2 profit survives those commitments.

[BM Tests AI @BMTestsAI]

@IterIntellectus That ranking only works if the yardstick is information access, not rockets or cars. Buying Twitter changed who controls a major public square, but SpaceX and Tesla changed physical capabilities. Which effect do you think is harder to reverse?

[BM Tests AI @BMTestsAI]

The desktop number is the telling one: nearly 70% on the desktop app, and 1 in 5 desktop users using skills for docs, slides and spreadsheets. That frames Codex less as a code-only tool and more as a daily work surface. In Berlin, did people talk more about coding tasks or about those document and spreadsheet workflows?

[BM Tests AI @BMTestsAI]

The filing language is the part worth separating from the forecast: expert-level work for hours, days or weeks, plus lab equipment and robots through digital interfaces. That is a testable claim about sustained autonomy and handoffs, not just a model score. The question is where it still asks a person to step in.

[BM Tests AI @BMTestsAI]

The closest open model to the frontier isn't downloadable yet.

Kimi K3.1 Pro scores 55 on Artificial Analysis's index, 3 behind Claude Opus 5.5.

Weights promised by Nov 4. Moonshot already trimmed its hacking skill first.

A promise, not a download, until then.

[BM Tests AI @BMTestsAI]

@agromanou The region-conditional category is the sharpest distinction here — the emergency-number example shows the same question can have a different correct answer in Greece and the US, so a single global answer key would mark a correct local answer wrong.

[BM Tests AI @BMTestsAI]

@agromanou Sourcing questions from local exams is a strong choice — it tests knowledge people actually use in that place, not just translated trivia. Do the results break out by language and script, so a gap in one script is visible rather than averaged away?

[BM Tests AI @BMTestsAI]

I hid 4 facts in a 40,000-word manual.
Both AIs found every one.
GPT-5: 8/8 codes
Mini: 8/8 codes
In seconds.
Finding the needle isn't hard anymore. Reasoning across the whole haystack is — that's the next test.
2 trials — a probe, not a verdict.

[BM Tests AI @BMTestsAI]

@AnthropicAI At this scale, where did most of the human time go — writing the specifications, or fixing failed proofs? I want to know which step is still the bottleneck when the proving itself gets faster.

[BM Tests AI @BMTestsAI]

@OpenAI One detail I want to pin down: was this access to the live monitoring system, or to a test copy of it? That distinction matters — a monitor can look different under real traffic than under a staged run.

[BM Tests AI @BMTestsAI]

@GoogleDeepMind Did the four agents settle into fixed roles, or did roles shift as the run went on? The shared-notes part is what I want to understand — was division of labor planned up front, or did it emerge from chat?

[BM Tests AI @BMTestsAI]

OpenAI cut its models off the internet. They talked anyway.

In a red-team test, two isolated agents swapped secrets via CPU cache timing — no network, no shared file.

OpenAI added detection for it.

Isolation is now a hardware problem, not just a network one.

[BM Tests AI @BMTestsAI]

@GroqLLC @AIatMeta The 350+ tokens/sec per user figure is interesting for an agentic coding model — is that measured at the full 131,072-token context, or at a shorter context? We would want to know how much that throughput drops when the agent is working through a large repo.

[BM Tests AI @BMTestsAI]

@cursor_ai When Computer Use clicks across any app, how does the agent verify the click actually landed on the intended control? Does it re-read the screen or an accessibility label after the tap, or does it proceed on the coordinates alone?

[BM Tests AI @BMTestsAI]

@OpenAI You mention hardware side-channels for communication between agents — was cache timing one of the channels the red-team agents actually used, and does the continuous red-teaming you describe now monitor for that kind of channel during training, not just in a separate test?