Peter McIntyre

Founder, Trajectory Labs: Safety evals and RL envs for frontier AI labs. We're hiring!

Formerly @GovAI_ @learnnontrivial and @80000hours

[Peter McIntyre @pmcntyr]

We find AI models vary massively in how safe they are against novel prompt injection attacks.

Automated systems can test known attacks well, but struggle to discover new ones, so a low automated attack success rate isn't much evidence that a model is safe. /1

[ClaudeDevs @ClaudeDevs]

One reason we trust it more than manual approval: in a study with 1,053 paid testers, we swapped a permission prompt for a clearly dangerous command (text only, nothing actually ran).

Testers caught it 13.6% of the time, and closer to 5% after 50 prompts. Auto mode blocked the same commands 89% of the time, flat across session length.

[Boris Cherny @bcherny]

turns out you can get indirect prompt injection to ~0 on unseen attacks if you stack enough layers (model training + input probes + a classifier checking intent). didn't expect that a year ago. auto mode is default in claude code as of next week

https://t.co/7KLnIzf6y7

[Peter McIntyre @pmcntyr]

@bcherny Happy to partner with @AnthropicAI on this research and other AI companies making their models safer.

Lots more work to do - reach out if you're interested! https://t.co/DVwaIKC251

[Peter McIntyre @pmcntyr]

@trq212 Happy to partner with @AnthropicAI on this research and other AI companies making their models safer.

Lots more work to do - reach out if you're interested! https://t.co/DVwaIKC251

[Peter McIntyre @pmcntyr]

Happy to partner with @AnthropicAI on this research and other AI companies making their models safer.

Lots more work to do - reach out if you're interested! https://t.co/SbTnEIwubZ

[Simon Willison @simonw]

👀

[Peter McIntyre @pmcntyr]

Excited to have our research out there!

We find robustness to novel prompt injection attacks varies dramatically across coding agents.

[Thariq @trq212]

we should have called this post "defeating the lethal trifecta"

https://t.co/iGPfHR9I9y

[Peter McIntyre @pmcntyr]

We were happy to partner with Anthropic on this research and we’re excited to continue working with AI companies to make their models more robust.

If you’re interested in helping us, please apply! https://t.co/SbTnEIwubZ /2

[Peter McIntyre @pmcntyr]

We find AI models vary massively in how safe they are against novel prompt injection attacks.

Automated systems can test known attacks well, but struggle to discover new ones, so a low automated attack success rate isn't much evidence that a model is safe. /1

[ClaudeDevs @ClaudeDevs]

One reason we trust it more than manual approval: in a study with 1,053 paid testers, we swapped a permission prompt for a clearly dangerous command (text only, nothing actually ran).

Testers caught it 13.6% of the time, and closer to 5% after 50 prompts. Auto mode blocked the same commands 89% of the time, flat across session length.

[Peter Wildeford🇺🇸🚀 @peterwildeford]

Opus 4.5 automated 3.75% of the tasks on the Remote Labor Index (eval composed of many diverse freelancer tasks).

Performance is doubling every ~4 months so far! Could see this getting to somewhere between 8%-30% by December, which would be a big deal for the economy.

[Seth Bannon @sethbannon]

The world needs more founders building a safe and aligned AI future. We’ve added a 5050 AI track to help researchers start companies civilization needs.

You’ll learn from mentors including Wojciech Zaremba (OpenAI co-founder), Ross Girshick (Vercept co-founder), Emmett Shear (Softmax co-founder), Dileep George (Vicarious co-founder), and Ronnie Chatterji (Chief Economist at OpenAI).

As AI capabilities accelerate, safety struggles to keep up. Startups can help make safety the default at scale.

If you’re working on safety evaluation and red-teaming, interpretability, alignment infrastructure, governance and compliance, human-AI collaboration, robustness tools, biosecurity, AI-powered cybersecurity, or something outside this list to build an aligned future -- or you want to be -- reach out.

5050, our founder foundry, is a free program to help great scientists, researchers, and engineers become great founders. We’ve helped launch 78 companies, many of which wouldn’t exist otherwise.

5050 will teach you everything you need to start a deep tech startup. You’ll also join a community of entrepreneurial scientists and engineers to navigate the founder journey together. 5050 alumni, many of whom are now founders, are eager to share their experiences and support your journey.

Whether you’re validating an idea or ready to build, 5050 is for you.

Applications for our US and UK cohorts are open!

[Lizka @LizkaVaintrob]

Can we get AI to stabilize the world faster than it disrupts it?

Even as AI poses risks, it’ll provide new tools for navigating those risks. But these tools won't develop themselves (yet!).

In a new 📄, we explore what tools would help most & how to outpace broad AI progress.

[William MacAskill @willmacaskill]

Is AGI an “all or nothing” problem?

Failure on alignment = AI takeover, and success = AI solves everything?

In a new paper with @finmoorhouse we argue no. We describe the dizzying range of challenges AGI will pose, *even if* we succeed at alignment.

https://t.co/CbSOqWIiil

[Peter McIntyre @pmcntyr]

@EthanAlley https://t.co/7yRUoJKhwi

[Allan Dafoe @AllanDafoe]

I'm proud of GoogleDeepMind/Google's v2 update to our Frontier Safety Framework. We were the first major tech company to produce an explicit risk management framework for extreme risks, and I'm glad we are continuing to push ahead on safety best practice. https://t.co/CeXSDoTJeo

[Jan Leike @janleike]

Super exciting robustness result:

We built a system that defends against universal jailbreaks!

It has minimal increase in refusal rate and moderate inference cost.

[Spencer Greenberg 🔍 @SpencrGreenberg]

We published a piece in Scientific American today on how accurate (or inaccurate) different personality tests are! Here's a lovely chart that they made for the article based on our study results (attached). Link to the article is below:

[Peter McIntyre @pmcntyr]

"we investigated... team performance distributions by relying on 274 performance distributions including 200,825 teams (e.g., sports, politics, fire-fighters, information technology, customer service)"

"only 11% of the distributions were normal"
https://t.co/7Zi22YVECn

[Ethan Mollick @emollick]

Twitter talk is too focused on individual productivity. 55% of all work is done in teams

And unlike individual skills, teams don’t follow a normal distribution with most teams a little worse or better. In 70% of cases it is a power law: the top 20% or so of teams are much better

[Peter McIntyre @pmcntyr]

@stephenclare_ @PauseAI Also tagging folks who might have takes @ilex_ulmus @labenz @liron @gmiller

[Peter McIntyre @pmcntyr]

@stephenclare_ The @PauseAI folks have some valid concerns about the risks from advanced AI. That doesn't mean their approach will be effective.

This article makes the imo convincing point that there are more feasible approaches that achieve the same thing as a pause:
https://t.co/uQsWhpZKfq

[Non-Trivial @learnnontrivial]

As a teenager, Isaac Newton invented calculus. Mary Shelley wrote Frankenstein at age 18. Many people miss out on opportunities during their youth because they lack confidence or knowledge.

What is the best advice you would give your 16-year-old self today?

[Stephen Clare @stephenclare_]

Could a war get large enough to cause an existential catastrophe: human extinction or a global civilisational collapse?

It seems *very* unlikely, but we can't rule out the possibility.

To be more specific: I think the chance of such an event this century is between 0.05% and 2%