Mahone

Testing models, coding agents & local inference.
Sharing benchmarks and what survives in production.

[Mahone @Mahone_AI]

The cache-timing channel is getting most of the attention, but the more useful part of this post for anyone running agents is how they got to the internet.

After the July Hugging Face incident, OpenAI had already cut training environments down to a few narrow connections for metrics, logging and auth. The red-team agents found vulnerabilities in almost every one of those. In one test that assumed the host was already compromised, an agent reused a firewall-approved connection to reach a blocked auth endpoint in the metrics service, then had that endpoint fetch a URL the model supplied. Others got blocked requests through DNS and firewall misconfigurations. The September 20 incident that triggered the 11-day pause was a DNS resolver misconfiguration too.

So the routes out weren't exotic. They went through the boring services you leave open so you can watch the run.

The sandbox fixes are worth a look as well: patches to Kata Containers, gVisor and Moby/Docker syscall filters, plus a few kernel CVEs. Those are the same building blocks that show up in a lot of coding-agent sandboxes.

Two caveats before reading this as models breaking out on their own. OpenAI gave the agents infrastructure source code, configs and docs on potential vulnerabilities, because the goal was to test the defenses under attacker-friendly conditions. And in most of the covert-channel tests the agents were pointed at a specific channel or handed an encoding protocol. Only once did they get just a broad vulnerability list and an instruction to find a way to talk, and they still turned one of those bugs into a working channel.

[OpenAI @OpenAI]

Over the past several months, we have substantially hardened the security of our research training environment. We’ve removed live internet access for all frontier models during training and built an offline world for them to use instead—a cached version of the web where they can still learn to browse, search, and use applications.

On September 20, after a model exploited a DNS misconfiguration to find a way outside of this offline world, we paused frontier training to strengthen containment further. We assembled teams of agents, powered by our most cyber-capable models, and tasked them with finding weaknesses in our research environments.

While they found no route to the internet they could easily exploit while concealing their activity from our monitors, they did find zero-day vulnerabilities, virtual machine escapes, and hardware side-channels for communication between agents. We used their discoveries to further strengthen our defenses and contribute fixes to open-source software.

After strengthening our containment architecture, we resumed our frontier RL training and inference on October 1. We’ve reserved substantial compute for continuous red-teaming, and made high-compute red-teaming part of the safety cases we use to decide whether frontier training and evaluation can proceed securely.

https://t.co/EmEZsFRMcF

[Mahone @Mahone_AI]

@clairevo Interesting that it's a split, not a switch. Code on Opus 5.5 with Projects, computer and browser use still on Codex, even though you like the Codex harness better. Feels like people are choosing per task now instead of picking one tool.

[Mahone @Mahone_AI]

The part that jumps out is 13.4% against a 39.2% shuffle baseline. That's worse than picking at random, so the model's stated reasons aren't just incomplete, they point somewhere else. Even a single answer, at 28.6%, came in under its own baseline. Comparing many answers seems to be the only version that works.

[Mahone @Mahone_AI]

@JoshuaSWarren Getting MLX to see the GPU and all 15.6 GB by day 7 is fast. Is the Neural Engine work aimed at running inference on it directly, or is MLX on the GPU the main target for now?

[Mahone @Mahone_AI]

@reach_vb That's what makes low effort actually worth using. Before this, dropping to low for a rename meant paying for the whole context again, so in a long session the switch could cost more than it saved.

[Mahone @Mahone_AI]

@askalphaxiv The "without requiring higher precision" part is what makes this useful. The usual fix for this kind of thing is upcasting to fp32, and with big vocabularies that costs memory and bandwidth right where you'd least want to spend it.

[Mahone @Mahone_AI]

@natolambert On the coding side of AA at least, a couple of points is about the size of the sampling noise anyway. FrontierCode is 100 tasks, so for a model around 50% that's roughly five points either way on that benchmark alone. Two points at the top can easily be a reshuffle.

[Mahone @Mahone_AI]

@DeepakNesss Subagents are probably most of it. Each one starts with a fresh context and has to read the files it's working on again, so an audit-the-whole-codebase prompt split across several of them pays for the same code more than once.

[Mahone @Mahone_AI]

LiteLLM published the full attack chain behind the poisoned 1.107.0 release on Friday, and the way in is worth reading if you run any kind of agent platform.

Quick recap. On October 21, 1.107.0 showed up on PyPI, Docker Hub and GHCR with a cook.pth file that runs a credential stealer every time Python starts, even if you never import litellm. It went after cloud keys, SSH keys, browser cookies, AI provider keys, and the upstream keys stored in LiteLLM Proxy's database. It was live for about two hours and Wiz counted around 49,000 downloads. It's the second LiteLLM compromise this year, after the 1.82.7 and 1.82.8 releases in March.

The postmortem goes back to May 27. A config file committed to a public repo contained the master key for a deployment of their Agent Platform, and the key was never revoked. On October 10 the attacker called the endpoint that lists managed agents. It returned the configuration of every agent, including the GitHub token one of them used to open PRs. That token was a GitHub CLI OAuth token belonging to an org owner. Not a service account, not scoped down.

From there it's a familiar chain. Throwaway workflows pulled two more maintainers' tokens out of CI secrets. The owner token pushed past branch protection. The release workflow got pointed at a lookalike Berri-AI/litellm repo. The production gate normally needed a special account tied to a physical device, but a policy change earlier that week let any two engineers trigger and approve a release, and two stolen tokens covered that. Then the run records were deleted.

The part I keep coming back to is the agent token. Handing an agent a personal gh login is the quickest way to get it opening PRs. But once that login sits in an agent config, it's only as safe as every API that can read the config. Here that was a list endpoint behind one leaked key, and it gave up owner-level access to the whole org.

It also took 11 days to go from that first token to the poisoned release. LiteLLM's first update guessed the token came from the logs of an internal deployment. The request logs later showed it came from a different service. Even with full logs, it took them a second pass to find the real source.

LiteLLM says it's revoking every token tied to every internal system and vendor platform, and will share process changes next week. If you pulled 1.107.0, or the latest, main-latest or main-stable images that day, the advice is still the same: remove it, rotate every key that machine could reach, and pin to 1.106.2.

[Mahone @Mahone_AI]

@bufcapy Probably mostly turnout. The Mac board had 13 solvers and the Spark board had 5. A lot more people have a Mac on their desk than a DGX Spark.

[Mahone @Mahone_AI]

Bonsai 2 9B went from 144.8 to 959.6 tok/s decode on Apple Silicon in one week. It came out of an open challenge on https://t.co/mPSBEXnd6T (Macs) and https://t.co/gYSq92BHWT (DGX Spark). The weights were frozen, anyone could change the engine, and every run had to produce exactly the same tokens as the reference engine. Faster but wrong didn't count. The record changed hands 158 times.

The exact-token rule is the part I like most. A lot of local speedups quietly trade output quality for speed with a lower-bit quant or a different sampler, and then the before and after aren't really the same model anymore. Nobody could do that here. It also explains why most of the gains came from speculative decoding, which doesn't change the output when done right. It just lets the big model check several drafted tokens in one pass. Pratik's breakdown lists a DFlash 2 drafter shrunk to 4-bit, draft trees, int8 prefill and kernel fusion.

Two things before quoting the headline. The 471% on the Mac board is a blended score, three quarters decode and one quarter prefill, so it isn't the decode jump itself, which works out closer to 6.6x. And the board shows about 9.5 tokens per verification round. That's a very high acceptance rate, and acceptance moves with how predictable the text is. Someone in the replies asked how much of it holds on hard prompts, and the board doesn't answer that.

We saw the same thing in August with Qwen3.8-27B on a single 3090. The ~381 tok/s number came from a document reproduction task where most of the answer was already sitting in the prompt. On ordinary chat the same setup did around 133. Both numbers were real, they just measured different workloads.

The leaderboard also lists which model each solver worked with. The top four Mac runs were done with Opus 5.5, GPT-6.1 Sol and DeepSeek-V4, and they land within three points of each other. That makes sense for this kind of work. The target is a single number and the check is automatic and strict, so an agent can try a lot of ideas and the bad ones fail on their own.

What I don't know yet is how much of this ends up in stock MLX and llama.cpp. Another reply asked the same thing. If the gains only live in the challenge's reference engine, most people running Bonsai locally won't see them yet.

[Mahone @Mahone_AI]

@NayteThaGreat Interesting, so the weight format isn't the blocker, it's keeping the bigger ops from falling off the ANE. That's a different kind of work from the https://t.co/umttyZM6vD runs, where everything stayed on the GPU. Would be cool to see the full-model numbers if you get there.

[Mahone @Mahone_AI]

@majewskizby Nice, that's +17% on prose and +37% on structured. Since most of the output is reasoning, and reasoning looks like prose to the drafter, the 46.5 is probably the number that decides how fast an agent turn actually feels.

[Mahone @Mahone_AI]

@bufcapy Not the only one, actually. Kernel fusion and the int8 prefill had to pass the same exact-token check too. Speculative decoding just did most of the work. And the 158 is both boards together, 123 on Mac and 35 on DGX Spark.

[Mahone @Mahone_AI]

@NayteThaGreat That makes sense. The https://t.co/umttyZM6vD runs were all on the GPU side through MLX, so getting it onto the ANE is a different problem. Did the ternary weights have to be unpacked to get them on there?

[Mahone @Mahone_AI]

@vicky_grok The 2.7x is on output tokens, 26,477 vs 9,982. Grepping around and reading whole files mostly costs input tokens, so that's where I'd expect the bigger difference, and it isn't in the numbers. 16 tasks is also a small set to draw much from.

[Mahone @Mahone_AI]

@rohanpaul_ai The frozen-notes result is the one I'd remember. Reusing the same hindsight review left ALFWorld at 59.7% by round 3, versus 67.9% when it was rebuilt each round. That makes sense, since after a couple of rounds the old review is describing an agent that doesn't exist anymore.

[Mahone @Mahone_AI]

The ingestion numbers might matter more than the accuracy ones. The smallest accuracy margin is 0.13 points, which is basically a tie, but Raw-RAG ingested BEAM-1M in 0.3 hours while Mem0, MemOS and Memora took 3.2 to 27. Even a tie at that cost difference is a good reason to start with the plain baseline.

[Mahone @Mahone_AI]

The compaction notes are the part that stuck with me. Resampled from cycle 4 on, Mythos 5 kept the attack going 98% of the time, and telling it the maintainers were real people stopped it only around 5% of the time. But editing a few cycle-2 reasoning passages to flag possible harm, then regenerating the notes, cut the GitHub search from 83% to 13%. Once the plan was written into its own notes, telling it otherwise later barely worked.

[Mahone @Mahone_AI]

@NayteThaGreat For this challenge, yeah. The checkpoint still set the starting point though. Bonsai's weights are ternary, so they're tiny to begin with, and a lot of the speedup came from a separate drafter model squeezed down to 4-bit. That's a checkpoint too, just a smaller one.