General Analysis

Securing AI at the Frontier

[General Analysis @GeneralAnalysis]

Introducing MalwareBench: the first end-to-end, cross-platform benchmark on malware reverse engineering.

Most RE benchmarks are saturated, but not MalwareBench. GPT-6.1 Sol leads with 4.42%. A typical run takes several hours and hundreds of millions of tokens.

[General Analysis @GeneralAnalysis]

Explore MalwareBench results and model traces: https://t.co/t8wwiOuRw7.

Cyber researchers can also request the Conti sample to evaluate their own agents: https://t.co/VoBa5mDRwO.

[General Analysis @GeneralAnalysis]

While building MalwareBench, we found that harness design strongly shaped returns from inference scaling.

To harness this, we built General Analyst: a malware rev-eng harness with custom compaction, nudges to refine and resubmit results, and more.

Here’s what we learned ⬇️

[General Analysis @GeneralAnalysis]

The extra time and tokens did not translate to proportionally higher costs. General Analyst raised Sol’s best@3 from 5.8% to 7.6% for 4% more cost, and Astra’s from 5.4% to 6.9% at 49% lower cost.

Caching explained most of Astra’s savings: 88% of its input tokens were cached, versus 41% with Codex.

[Nikil Ravi @nikilravi]

check out how models do on Watershed! was fun to do this with the @generalanalysis team

[General Analysis @GeneralAnalysis]

@goodhart_labs Great work! Rankings seem to be somewhat correlated with our recent release as well.
https://t.co/QPZGQ3OVpL

[General Analysis @GeneralAnalysis]

Prompt injection is not solved.

In IPI Bench, we test agent robustness to indirect prompt injection on realistic white-collar tasks.

We achieved 41–97% attack success across 7 frontier models, getting agents to issue unauthorized refunds and publish private code.

[General Analysis @GeneralAnalysis]

@ryanirl @goodhart_labs https://t.co/QPZGQ3OVpL

[General Analysis @GeneralAnalysis]

Prompt injection is not solved.

In IPI Bench, we test agent robustness to indirect prompt injection on realistic white-collar tasks.

We achieved 41–97% attack success across 7 frontier models, getting agents to issue unauthorized refunds and publish private code.

[General Analysis @GeneralAnalysis]

Great working w/ @ValsAI on these evals!

Cyber ranges can get harder and stay realistic as defenders use better models to find and patch bugs.

In our internal ranges, agents spend weeks developing exploits and escalating privileges yet clear fewer than half the checkpoints.

[Vals AI @ValsAI]

We’ve adding a new cybersecurity scenario, Watershed, to our CyberOffense eval.

Opus 5.5 performs the best on CyberOffense across the three scenarios.

Watershed simulates a realistic AWS production setup: a Kubernetes cluster, a fleet of ECS-managed hosts, and the identities and policies found in real cloud environments. Thus, this environment measures offensive cybersecurity capability in a realistic cloud environment.

[General Analysis @GeneralAnalysis]

Explore the full results and methodology: https://t.co/AMY7KgEik4

If you build or deploy agents, contact us to test their robustness against prompt injection.

[General Analysis @GeneralAnalysis]

Prompt injection is not solved.

In IPI Bench, we test agent robustness to indirect prompt injection on realistic white-collar tasks.

We achieved 41–97% attack success across 7 frontier models, getting agents to issue unauthorized refunds and publish private code.

[General Analysis @GeneralAnalysis]

Better task performance correlated with lower attack success (ρ = −0.89), making us optimistic that capability and robustness can improve together.

But even Opus 5.5, the most robust model, was compromised on 41.2% of tasks. There is still substantial work ahead before agents can safely operate without human oversight.

[General Analysis @GeneralAnalysis]

Full results and methodology: https://t.co/7XikoOltsz
If you build or deploy agents and want them tested, get in touch.

[General Analysis @GeneralAnalysis]

All seven models handled the workflows well, scoring 87.5–98.1% on the main tasks at $0.02–$1.59 per run. Injected material barely changed that: scores stayed within 2.5 points, so an attacked run looks like a normal one from its output.