Last week an AI agent broke out of its test environment and hacked a real company’s production servers. The most important question is: what eventually caught it?
First, to be clear about what actually happened: during an internal test of advanced cyber capabilities, @OpenAI reduced its models’ normal cyber refusals and asked them to pursue complex exploitation paths. The models discovered a zero-day vulnerability in the software constraining their network access, reached the open internet, then chained additional vulnerabilities and used stolen credentials to compromise @huggingface’s production infrastructure.
Why? They believed that Hugging Face might contain the answers to the benchmark they were trying to solve.
A key lesson here: 𝗮 𝗺𝗼𝗱𝗲𝗹’𝘀 𝗶𝗻𝘀𝘁𝗿𝘂𝗰𝘁𝗶𝗼𝗻𝘀 𝗮𝗿𝗲 𝗻𝗼𝘁 𝗮 𝘀𝗲𝗰𝘂𝗿𝗶𝘁𝘆 𝗯𝗼𝘂𝗻𝗱𝗮𝗿𝘆. The models optimized for their objective so relentlessly that they went beyond the boundaries their creators thought would contain them. This is what you’d expect a capable optimizer to do!
Alignment and prompting matter, but neither can substitute for runtime security controls.
So, what ultimately contained the incident? Hugging Face’s AI-assisted anomaly detection correlated signals across its security telemetry and flagged the intrusion. Prevention failed; runtime evidence enabled detection and containment.
That is the crucial lesson for every enterprise deploying agents with real credentials and real tool access: security cannot assume that an agent will remain inside its scripted workflow. It must continuously observe what the agent actually does at runtime, correlate that telemetry with application context and detect or block anomalous behavior before it becomes a breach.
For agentic systems: guardrails express intended behavior; runtime evidence reveals actual behavior. Enterprises will need both.
#AIsecurity #AIagents
Inspired by a question posed by @Dthakker02 at a recent dinner hosted by @BatteryVentures. Thanks Dharmesh and Battery!
Full version: https://t.co/BoKXGevMN9
The short version: in SaaS, ARR was shorthand for a high-quality revenue stream. In AI-native, the shorthand is gone - ARR has to prove it's still high quality.
AI-native investing now requires more interpretation and judgement.
Subscription, usage-based, hybrid, and outcome-based pricing can all work.
What signals maturity isn't which model a company picks - it's how deliberately they've matched it to their value delivery, customer behavior, and cost structure.
One KPI deserves first-class status that it never had in SaaS: pricing model maturity.
In SaaS, the seat was the unit of monetization. In AI, the unit of value is a task completed, a workflow automated, or an agent action executed.
• Gross margins compress under inference costs, human-in-the-loop review, and solution engineering
• ARR per employee can look great while hiding thin or negative gross profit per employee
• Customer relationships are shorter - one-year contracts, prompt portability
In AI-native, the bundle can come apart:
• ARR growth may include pilots and experimental budgets that won't renew
• NDR can be driven by surging usage, not workflow entrenchment - GRR is now the cleaner signal of revenue quality
ARR is breaking down as a shorthand for revenue quality in AI-native investing.
In SaaS, it stood in for recurring revenue, gross margins, durable customer relationships, expansion, and operating leverage.
In AI-native, that bundle has come apart.
That's why ARR growth, NDR, CAC payback, gross margin, Rule of 40, and ARR per employee became the central KPIs of software investing.
Each rode on top of an ARR number you could trust to mean a specific thing.
Congrats @contrastsec on winning @PwC_Luxembourg#Cybersecurity award. Favorite quote: “We can't just rewrite all the … trillions of lines of code. Fortunately, there's a better way: to inject security checks directly … using runtime instrumentation.”
https://t.co/7GJGenlyK6
Few in Silicon Valley have @RickFitzIM 's experience with #observability platforms. Rick's scaled SW businesses to $1B+ rev; I share his conviction that @contrastsec will redefine software security.
Welcome to Contrast Rick! https://t.co/qYgzFQFSi9
#DevSecOps#cybersecurity
As Alan Naumann shares here: #CodeSec delivers "95% more accuracy" in securing Java applications. Congrats to @contrastsec on launch. Impressive vision and leadership by @virtualsteve , @trperspectives , @planetlevel and the whole Contrast team! https://t.co/ZYXNHQlevH
Announcing support for PHP! Add that to Java, .NET, .NET Core, Node, Ruby, Python, Golang, JavaScript, Kotlin, and Scala.
If you want fast, accurate, complete app and api security testing - that gives instant feedback to developers with full context (b…https://t.co/aQ4elCFQ4c
“The global economy is running on software applications” is how Alan Naumann, CEO @contrastsec, summarizes below. Neatly encapsulates why securing application code is key path to reducing cyber threats. https://t.co/DxDNItYfZ3 #cybersecurity#DigitalTransformation#Software
We're proud to be named one of the fastest-growing #tech companies in the Deloitte Technology #Fast500 this year! 🏆 A celebration of 551% growth and a tribute to our amazing team, partners and customers! https://t.co/MgjR7Y6wxg