Introducing Firedrill, a simulation and testing platform for AI agents. Run thousands of simulations to find where your AI agents break before production.
Before an agent handles customer work, you need to test how it behaves with different data, permissions and tool failures. Doing that across apps and tools means managing test accounts, recreating scenarios and cleaning up after every run.
Firedrill spins up synthetic tools with data you control, so you can test your existing agent without using production accounts. Choose from the library, including Gmail, HubSpot, Stripe and more.
Run workflows under different conditions, inspect what happened and test your fixes. Turn those scenarios into repeatable checks in your PR workflow to catch regressions before they reach customers.
What does your AI agent do when a tool goes offline mid-task?
In Firedrill, take a synthetic tool offline and see how your agent handles a 503. Does it retry, explain what it couldn’t finish, or say “done” anyway?
Your personal agent should know what you’re working towards, who matters to you, and how you make decisions.
The better it understands you, the more it can do for you.
This Friday, we’re getting together the best builders to build personal agents with @typesafeai@Meta@MongoDB@Amplitude_HQ@statsig and @superlinked
📍 SF
📅 Fri, Nov 20
🏆 $9,000+ in cash, gift cards, credits, and other prizes
More partners include @AirbyteHQ@arcee_ai@RunFiredrill@GuildAI@withpersona@SingleStoreDB@cognee_
Limited spots. Apply: https://t.co/NvLjznJ2A3
We’re hyped for the Personal Agents Hackathon in SF on Nov 20th!
If you’re building a personal agent, come hack with us. We’ll have Firedrill there so you can test what you build in a synthetic world of apps and data before demo time.
Apply: https://t.co/4pVXdt3uin
Your AI agent can now sign itself up for Firedrill using auth.md from @WorkOS.
All you have to do is approve the request.
Thanks to Michael (@grinich) and the WorkOS team for building this in the open.
@sachinmour@newtonasareII The two-connections case is a good one, most people stop at firing the same webhook twice. Would love to get you on Firedrill and see how those three hold up against your setup.
@JamesWalker1979 That’s why simulations matter alongside tests: we need to check the whole agent run, including what it leaves out, even when the total looks right.
When you score an AI-agent simulation, decide which requirements must be satisfied exactly.
Firedrill lets you simulate your AI agent in a synthetic world with controlled scenarios, using synthetic tools and test data when needed. Your existing AI application runs in your own runner.
If you add an assessment, keep quality scores and exact checks separate.
Consider a hypothetical invoice-summary agent. Grade how clearly it explains the charges. Separately, check the structured total and currency against the supplied invoice data.
A polished explanation should not compensate for the wrong total. Keep that failed check visible even when the quality score is high.
Firedrill’s reports retain these as separate results. You define the criteria and the exact requirements for the task.
Use graded criteria for meaning, and exact checks for values that must match.
Scoring guide: https://t.co/J6j2MC4XSE
Firedrill has been selected for the @MetaforDevs Global AI Developer Hackathon.
We'll be exploring how Meta's Muse models can expand our simulation capabilities, helping us create more realistic environments and a wider range of scenarios for AI agents.
Firedrill is a training ground where AI agents can practice, learn from failures, and improve themselves.
We're excited to be working with Meta and all the amazing partners @baseten@cerebral_valley@modal and @FireworksAI_HQ.
@speedrun Simulations can unlock that self-improvement loop: agents explore changing conditions and scenarios, learn from outcomes, and adapt how they work. Firedrill gives them a synthetic world to do that.
BTW we applied to @speedrun. Would love to chat!
We will be at AGNTCon + MCPCon in San Jose, Oct 22–23 hosted by @linuxfoundation. Firedrill helps you simulate AI agent workflows with synthetic tools and test data to catch failures before they reach users.
Building agents? DM us to meet.