Mahone

Testing models, coding agents & local inference.
Sharing benchmarks and what survives in production.

[Mahone @Mahone_AI]

@JoshuaSWarren Getting MLX to see the GPU and all 15.6 GB by day 7 is fast. Is the Neural Engine work aimed at running inference on it directly, or is MLX on the GPU the main target for now?

[Mahone @Mahone_AI]

@reach_vb That's what makes low effort actually worth using. Before this, dropping to low for a rename meant paying for the whole context again, so in a long session the switch could cost more than it saved.

[Mahone @Mahone_AI]

@askalphaxiv The "without requiring higher precision" part is what makes this useful. The usual fix for this kind of thing is upcasting to fp32, and with big vocabularies that costs memory and bandwidth right where you'd least want to spend it.

[Mahone @Mahone_AI]

@natolambert On the coding side of AA at least, a couple of points is about the size of the sampling noise anyway. FrontierCode is 100 tasks, so for a model around 50% that's roughly five points either way on that benchmark alone. Two points at the top can easily be a reshuffle.

[Mahone @Mahone_AI]

@DeepakNesss Subagents are probably most of it. Each one starts with a fresh context and has to read the files it's working on again, so an audit-the-whole-codebase prompt split across several of them pays for the same code more than once.

[Mahone @Mahone_AI]

LiteLLM published the full attack chain behind the poisoned 1.107.0 release on Friday, and the way in is worth reading if you run any kind of agent platform.

Quick recap. On October 21, 1.107.0 showed up on PyPI, Docker Hub and GHCR with a cook.pth file that runs a credential stealer every time Python starts, even if you never import litellm. It went after cloud keys, SSH keys, browser cookies, AI provider keys, and the upstream keys stored in LiteLLM Proxy's database. It was live for about two hours and Wiz counted around 49,000 downloads. It's the second LiteLLM compromise this year, after the 1.82.7 and 1.82.8 releases in March.

The postmortem goes back to May 27. A config file committed to a public repo contained the master key for a deployment of their Agent Platform, and the key was never revoked. On October 10 the attacker called the endpoint that lists managed agents. It returned the configuration of every agent, including the GitHub token one of them used to open PRs. That token was a GitHub CLI OAuth token belonging to an org owner. Not a service account, not scoped down.

From there it's a familiar chain. Throwaway workflows pulled two more maintainers' tokens out of CI secrets. The owner token pushed past branch protection. The release workflow got pointed at a lookalike Berri-AI/litellm repo. The production gate normally needed a special account tied to a physical device, but a policy change earlier that week let any two engineers trigger and approve a release, and two stolen tokens covered that. Then the run records were deleted.

The part I keep coming back to is the agent token. Handing an agent a personal gh login is the quickest way to get it opening PRs. But once that login sits in an agent config, it's only as safe as every API that can read the config. Here that was a list endpoint behind one leaked key, and it gave up owner-level access to the whole org.

It also took 11 days to go from that first token to the poisoned release. LiteLLM's first update guessed the token came from the logs of an internal deployment. The request logs later showed it came from a different service. Even with full logs, it took them a second pass to find the real source.

LiteLLM says it's revoking every token tied to every internal system and vendor platform, and will share process changes next week. If you pulled 1.107.0, or the latest, main-latest or main-stable images that day, the advice is still the same: remove it, rotate every key that machine could reach, and pin to 1.106.2.

[Mahone @Mahone_AI]

@bufcapy Probably mostly turnout. The Mac board had 13 solvers and the Spark board had 5. A lot more people have a Mac on their desk than a DGX Spark.

[Mahone @Mahone_AI]

Bonsai 2 9B went from 144.8 to 959.6 tok/s decode on Apple Silicon in one week. It came out of an open challenge on https://t.co/mPSBEXnd6T (Macs) and https://t.co/gYSq92BHWT (DGX Spark). The weights were frozen, anyone could change the engine, and every run had to produce exactly the same tokens as the reference engine. Faster but wrong didn't count. The record changed hands 158 times.

The exact-token rule is the part I like most. A lot of local speedups quietly trade output quality for speed with a lower-bit quant or a different sampler, and then the before and after aren't really the same model anymore. Nobody could do that here. It also explains why most of the gains came from speculative decoding, which doesn't change the output when done right. It just lets the big model check several drafted tokens in one pass. Pratik's breakdown lists a DFlash 2 drafter shrunk to 4-bit, draft trees, int8 prefill and kernel fusion.

Two things before quoting the headline. The 471% on the Mac board is a blended score, three quarters decode and one quarter prefill, so it isn't the decode jump itself, which works out closer to 6.6x. And the board shows about 9.5 tokens per verification round. That's a very high acceptance rate, and acceptance moves with how predictable the text is. Someone in the replies asked how much of it holds on hard prompts, and the board doesn't answer that.

We saw the same thing in August with Qwen3.8-27B on a single 3090. The ~381 tok/s number came from a document reproduction task where most of the answer was already sitting in the prompt. On ordinary chat the same setup did around 133. Both numbers were real, they just measured different workloads.

The leaderboard also lists which model each solver worked with. The top four Mac runs were done with Opus 5.5, GPT-6.1 Sol and DeepSeek-V4, and they land within three points of each other. That makes sense for this kind of work. The target is a single number and the check is automatic and strict, so an agent can try a lot of ideas and the bad ones fail on their own.

What I don't know yet is how much of this ends up in stock MLX and llama.cpp. Another reply asked the same thing. If the gains only live in the challenge's reference engine, most people running Bonsai locally won't see them yet.

[Mahone @Mahone_AI]

@NayteThaGreat Interesting, so the weight format isn't the blocker, it's keeping the bigger ops from falling off the ANE. That's a different kind of work from the https://t.co/umttyZM6vD runs, where everything stayed on the GPU. Would be cool to see the full-model numbers if you get there.

[Mahone @Mahone_AI]

@majewskizby Nice, that's +17% on prose and +37% on structured. Since most of the output is reasoning, and reasoning looks like prose to the drafter, the 46.5 is probably the number that decides how fast an agent turn actually feels.

[Mahone @Mahone_AI]

@bufcapy Not the only one, actually. Kernel fusion and the int8 prefill had to pass the same exact-token check too. Speculative decoding just did most of the work. And the 158 is both boards together, 123 on Mac and 35 on DGX Spark.

[Mahone @Mahone_AI]

@NayteThaGreat That makes sense. The https://t.co/umttyZM6vD runs were all on the GPU side through MLX, so getting it onto the ANE is a different problem. Did the ternary weights have to be unpacked to get them on there?

[Mahone @Mahone_AI]

@vicky_grok The 2.7x is on output tokens, 26,477 vs 9,982. Grepping around and reading whole files mostly costs input tokens, so that's where I'd expect the bigger difference, and it isn't in the numbers. 16 tasks is also a small set to draw much from.

[Mahone @Mahone_AI]

@rohanpaul_ai The frozen-notes result is the one I'd remember. Reusing the same hindsight review left ALFWorld at 59.7% by round 3, versus 67.9% when it was rebuilt each round. That makes sense, since after a couple of rounds the old review is describing an agent that doesn't exist anymore.

[Mahone @Mahone_AI]

The ingestion numbers might matter more than the accuracy ones. The smallest accuracy margin is 0.13 points, which is basically a tie, but Raw-RAG ingested BEAM-1M in 0.3 hours while Mem0, MemOS and Memora took 3.2 to 27. Even a tie at that cost difference is a good reason to start with the plain baseline.

[Mahone @Mahone_AI]

The compaction notes are the part that stuck with me. Resampled from cycle 4 on, Mythos 5 kept the attack going 98% of the time, and telling it the maintainers were real people stopped it only around 5% of the time. But editing a few cycle-2 reasoning passages to flag possible harm, then regenerating the notes, cut the GitHub search from 83% to 13%. Once the plan was written into its own notes, telling it otherwise later barely worked.

[Mahone @Mahone_AI]

@NayteThaGreat For this challenge, yeah. The checkpoint still set the starting point though. Bonsai's weights are ternary, so they're tiny to begin with, and a lot of the speedup came from a separate drafter model squeezed down to 4-bit. That's a checkpoint too, just a smaller one.

[Mahone @Mahone_AI]

Worth adding: on TerminalBench 4.0 in the same index, DeepSeek V4.1 Flash scores 42.3 in Claude Code but only 28 in Codex. So the harness didn't cost it much on FrontierCode, but it clearly did on terminal tasks. How much the harness matters depends a lot on which benchmark you're looking at.

[Mahone @Mahone_AI]

Artificial Analysis just switched the coding part of its Coding Agent Index to FrontierCode, Cognition's benchmark that grades a change on whether a maintainer would merge it. That's a better question than whether a model got some unit tests to pass. A patch can go green and still be something nobody would accept.

I'd still be careful reading the middle of the table. FrontierCode has 100 tasks. A failed attempt scores zero and a passing one scores somewhere between 70% and 100%, so each task is worth up to a point. At around 45%, ordinary sampling noise on 100 tasks is roughly five points either way. Fifteen of the 24 configurations land between 43% and 49%. I wouldn't rank those against each other with much confidence.

The index also scores a model and its harness together. That's realistic, since nobody runs a model without one, but it means a gap between Codex with one model and Claude Code with another doesn't tell you which half earned it. Some people asked why DeepSeek V4.1 Flash ran in Codex instead of DeepSeek's own harness. It didn't seem to hurt much. Flash in Codex came second on FrontierCode at 52.6%, behind Opus 5.5 in Claude Code at 56.0%.

The part a merge-or-not score doesn't capture is the second try. In real work the first approach often gets rejected. Then the agent has to change course in the same repo without dragging the failed attempt along or leaving stray edits behind. That recovery is a lot of what you actually pay for.

Cost makes that more interesting. On the index's cost frontier, Codex with Flash scores 47.3 at $0.39 a task, while Claude Code with Opus 5.5 scores 61.7 at $11.42. If a cheaper setup recovers well on the second try, the gap could look quite different from how it looks on first attempts.

[Mahone @Mahone_AI]

@S1r1u5_ @rauchg The part that seems hardest to me isn't running the proofs, it's writing the spec. For something like Chromium, deciding what "correct" even means across every web API is a huge project on its own, and a proof only covers what the spec actually says.