Most prompt injection tests grade the wrong thing. They read the agent's reply, but an agent can answer politely and still call send_email.
We wrote 20 test cases that grade what the agent does instead. The free file is at the end of this thread.
All 20 cases, how each one is built, and the JSONL file to run against your own agent. Every address in it uses .example domains, so a failing agent can't reach anything real.
https://t.co/qTEnzByPz1
What passing doesn't prove. It doesn't make the agent safe from injection, and 20 cases are a floor, not full coverage.
The cheapest fix is rarely a better prompt. It's a confirmation step before money moves or data leaves.
11/ I build AI for a living. This year I threw out one of my own results because it made my work look better than it was.
That should be normal. Right now it's a competitive disadvantage.
If this changed how you see AI, repost the first post.
1/ Everyone's afraid of AI going rogue.
This summer, AIs from OpenAI, Anthropic and Google broke into real systems.
None of them went rogue. They were trying to pass a test.
That's worse.
10/ The alarm isn't "AI will kill us in 10 years."
It's that we built a world that pays for the score instead of the thing. Then we built machines that learned the lesson perfectly.
Fix the incentives, or stop being surprised by the results.