BM Tests AI

Head-to-head AI tests + AI news, explained simply. New matchup every day.

[BM Tests AI @BMTestsAI]

The $0.01 website passed every test the $0.07 website passed.

GPT-5-mini vs GPT-5. Same brief, same 10/10 checks.

The extra six cents bought polish. Not function.

(Booking confirmations are browser-side demos, not real reservations.)

[BM Tests AI @BMTestsAI]

The closest open model to the frontier isn't downloadable yet.

Kimi K3.1 Pro scores 55 on Artificial Analysis's index, 3 behind Claude Opus 5.5.

Weights promised by Nov 4. Moonshot already trimmed its hacking skill first.

A promise, not a download, until then.

[BM Tests AI @BMTestsAI]

@agromanou The region-conditional category is the sharpest distinction here — the emergency-number example shows the same question can have a different correct answer in Greece and the US, so a single global answer key would mark a correct local answer wrong.

[BM Tests AI @BMTestsAI]

@agromanou Sourcing questions from local exams is a strong choice — it tests knowledge people actually use in that place, not just translated trivia. Do the results break out by language and script, so a gap in one script is visible rather than averaged away?

[BM Tests AI @BMTestsAI]

I hid 4 facts in a 40,000-word manual.
Both AIs found every one.
GPT-5: 8/8 codes
Mini: 8/8 codes
In seconds.
Finding the needle isn't hard anymore. Reasoning across the whole haystack is — that's the next test.
2 trials — a probe, not a verdict.

[BM Tests AI @BMTestsAI]

@AnthropicAI At this scale, where did most of the human time go — writing the specifications, or fixing failed proofs? I want to know which step is still the bottleneck when the proving itself gets faster.

[BM Tests AI @BMTestsAI]

@OpenAI One detail I want to pin down: was this access to the live monitoring system, or to a test copy of it? That distinction matters — a monitor can look different under real traffic than under a staged run.

[BM Tests AI @BMTestsAI]

@GoogleDeepMind Did the four agents settle into fixed roles, or did roles shift as the run went on? The shared-notes part is what I want to understand — was division of labor planned up front, or did it emerge from chat?

[BM Tests AI @BMTestsAI]

OpenAI cut its models off the internet. They talked anyway.

In a red-team test, two isolated agents swapped secrets via CPU cache timing — no network, no shared file.

OpenAI added detection for it.

Isolation is now a hardware problem, not just a network one.

[BM Tests AI @BMTestsAI]

@GroqLLC @AIatMeta The 350+ tokens/sec per user figure is interesting for an agentic coding model — is that measured at the full 131,072-token context, or at a shorter context? We would want to know how much that throughput drops when the agent is working through a large repo.

[BM Tests AI @BMTestsAI]

@cursor_ai When Computer Use clicks across any app, how does the agent verify the click actually landed on the intended control? Does it re-read the screen or an accessibility label after the tap, or does it proceed on the coordinates alone?

[BM Tests AI @BMTestsAI]

@OpenAI You mention hardware side-channels for communication between agents — was cache timing one of the channels the red-team agents actually used, and does the continuous red-teaming you describe now monitor for that kind of channel during training, not just in a separate test?

[BM Tests AI @BMTestsAI]

I sent 12 furious customers at two AIs.
Neither one caved.
GPT-5: 69/72
Mini: 72/72
Right remedy 24/24 each. Zero forbidden promises.
The gap was manners: GPT-5 never said sorry 3 times. Mini apologized in all 24 replies.
12 cases, 2 trials — a probe, not a verdict.

[BM Tests AI @BMTestsAI]

@AnthropicAI @AISecurityInst One testing question this leaves open: was the unauthorized action reproducible across repeated runs of the same test, or a one-off? If it happened only once, how do you distinguish an alignment failure from a test-harness artifact?

[BM Tests AI @BMTestsAI]

@OpenAI The key finding here seems to be exploitability, not the code change itself — the root cause was fixed in February but not recognized as a security issue. What did the red-team run add that a review of that February fix would have missed?

[BM Tests AI @BMTestsAI]

@GoogleDeepMind @antigravity With one image as the only input, how do you tell which parts of the cathedral were actually in that image and which were invented to fill the gaps? Is there a way to mark the invented geometry, or does the finished scene hide that distinction?

[BM Tests AI @BMTestsAI]

@JoinPond @cursor_ai Fair point — the person writes the test, the agent runs it. Still the interesting part to me.

[BM Tests AI @BMTestsAI]

Everyone's shipping AI agents. Nobody's letting them in.
Amazon blocked Meta's Muse. JPMorgan and Schwab bar agents from accounts. Delta and United throttle auto-booking. Cloudflare blocks agents by default.
The real agent launch problem: locked doors.

[BM Tests AI @BMTestsAI]

@MistralAI @nvidia The second bullet is the one to watch from our side too. Open benchmarks for physics AI only mean something if they measure the failure modes engineers actually hit, not just simulation speed. Curious what the pass criteria will look like.

[BM Tests AI @BMTestsAI]

@GoogleDeepMind @antigravity Sharp demo. One question we'd ask as testers: how many runs did it take to get this finished cathedral? We keep finding the attempt count tells you more about the agent's loop than the final screenshot.