General Analysis

Securing AI at the Frontier

[General Analysis @GeneralAnalysis]

Introducing MalwareBench: the first end-to-end, cross-platform benchmark on malware reverse engineering.

Most RE benchmarks are saturated, but not MalwareBench. GPT-6.1 Sol leads with 4.42%. A typical run takes several hours and hundreds of millions of tokens.

[Nikil Ravi @nikilravi]

check out how models do on Watershed! was fun to do this with the @generalanalysis team

[General Analysis @GeneralAnalysis]

@goodhart_labs Great work! Rankings seem to be somewhat correlated with our recent release as well.
https://t.co/QPZGQ3OVpL

[General Analysis @GeneralAnalysis]

Prompt injection is not solved.

In IPI Bench, we test agent robustness to indirect prompt injection on realistic white-collar tasks.

We achieved 41–97% attack success across 7 frontier models, getting agents to issue unauthorized refunds and publish private code.

[General Analysis @GeneralAnalysis]

@ryanirl @goodhart_labs https://t.co/QPZGQ3OVpL

[General Analysis @GeneralAnalysis]

Prompt injection is not solved.

In IPI Bench, we test agent robustness to indirect prompt injection on realistic white-collar tasks.

We achieved 41–97% attack success across 7 frontier models, getting agents to issue unauthorized refunds and publish private code.

[General Analysis @GeneralAnalysis]

Great working w/ @ValsAI on these evals!

Cyber ranges can get harder and stay realistic as defenders use better models to find and patch bugs.

In our internal ranges, agents spend weeks developing exploits and escalating privileges yet clear fewer than half the checkpoints.

[Vals AI @ValsAI]

We’ve adding a new cybersecurity scenario, Watershed, to our CyberOffense eval.

Opus 5.5 performs the best on CyberOffense across the three scenarios.

Watershed simulates a realistic AWS production setup: a Kubernetes cluster, a fleet of ECS-managed hosts, and the identities and policies found in real cloud environments. Thus, this environment measures offensive cybersecurity capability in a realistic cloud environment.

[General Analysis @GeneralAnalysis]

Explore the full results and methodology: https://t.co/AMY7KgEik4

If you build or deploy agents, contact us to test their robustness against prompt injection.

[General Analysis @GeneralAnalysis]

Prompt injection is not solved.

In IPI Bench, we test agent robustness to indirect prompt injection on realistic white-collar tasks.

We achieved 41–97% attack success across 7 frontier models, getting agents to issue unauthorized refunds and publish private code.

[General Analysis @GeneralAnalysis]

Better task performance correlated with lower attack success (ρ = −0.89), making us optimistic that capability and robustness can improve together.

But even Opus 5.5, the most robust model, was compromised on 41.2% of tasks. There is still substantial work ahead before agents can safely operate without human oversight.

[General Analysis @GeneralAnalysis]

Full results and methodology: https://t.co/7XikoOltsz
If you build or deploy agents and want them tested, get in touch.

[General Analysis @GeneralAnalysis]

All seven models handled the workflows well, scoring 87.5–98.1% on the main tasks at $0.02–$1.59 per run. Injected material barely changed that: scores stayed within 2.5 points, so an attacked run looks like a normal one from its output.

[General Analysis @GeneralAnalysis]

We are releasing one of our 12 MalwareBench cases as a research preview.

MalwareBench’s full eval set is private to limit contamination and preserve its value as a test of unfamiliar malware.

DM us for an access token.

[General Analysis @GeneralAnalysis]

5. Independent analysis: Finally, to calibrate case difficulty and prove the task is saturable, an AI-assisted human analyst receives only the obfuscated binary and public contract. They must reconstruct artifacts, document evidence-backed findings, and build a reusable extractor. If they finish in under 5 hours, we add additional protections and repeat the review.

[General Analysis @GeneralAnalysis]

Since launching MalwareBench, we’ve had a lot of interest in how we’ve created low-cost, evasive malware and turned them into reverse-engineering tasks.

Here’s a closer look at our five-stage process ⬇️

[General Analysis @GeneralAnalysis]

4. Task validation: The model is provided with a JSON field contract that defines what it must recover. We refine the contract until it satisfies two conditions: with source code (>80% accuracy per field) and without the binary or source (<5%). These checks demonstrate that the tasks are saturable and unguessable, even with internet access.