Rob Bensinger ⏹️

@MIRIBerkeley. If we build opaque superhuman AIs, it's likely to get us killed. The least-bad solution is an international agreement like

[Rob Bensinger ⏹️ @robbensinger]

My summary of the basic argument in "If Anyone Builds It, Everyone Dies":

[Rob Bensinger ⏹️ @robbensinger]

@ThePaulShearer "I don't think pathogen engineering is realistic"

Huh? Why not? Humans already successfully make more dangerous viruses. AIs are already used to engineer new viruses (https://t.co/nVFqgaihCj). And why would it not matter?

[Rob Bensinger ⏹️ @robbensinger]

This post is aging well. Which is maybe the worst piece of news I've received in my life.

Tell your family that you love them. Call your congressman. And come to the protest this Saturday.

I think we can actually survive this. But the USG may need to act soon, and quickly.

[Rob Bensinger ⏹️ @robbensinger]

It seems to me that we're at "it's time to panic about AI".

I gather that a lot of researchers are worried that we may be about to die, on a timescale of months, but this still isn't quite in the Overton window so we're still soft-pedaling.

(Where "panic", of course, means "quickly and soberly initiate talks with the CCP to achieve a moratorium on large training runs, backed by a deterrence regime, compute and supply chain monitoring, and site inspections". Pray that a deal can be reached, and pray that there's still time. Do the things that could actually work, like broad political coalition-building, not doomer LARPing as a form of self-expression.)

I think part of why people aren’t being clear about this is that it’s actually a tricky communication problem:

- Predicting "if frontier R&D continues at its current pace, when are we likely to die?" may give a different answer than predicting when permanent loss of control is likely, which is different from “realistically, when is it too late for governments to stop this," which is different from “when will the tech have advanced and proliferated to the point that any regulatory options that still have a chance of working would be way more extreme and unpopular than just some international restrictions on data center usage?"

Extinction is the emotionally salient, easy-to-understand threshold -- Jacob Coxon demonstrated that it helps to be honest about this. But the other thresholds could legitimately come years earlier, in which case focusing on extinction is ironically misleadingly reassuring. This is not the kind of problem you can solve at the last minute. As with many exponentials, the likeliest outcome is that you’re either embarrassingly early, or you’re too late.

- How do you communicate “we don’t have a robust way to predict timelines, but everyone in the field is freaking out and a lot of us genuinely think we're about to die?” Where’s the proof? There isn’t any proof: it’s an entire field of engineers who are eyeballing what they’re seeing in front of them and rapidly updating in real time to “this looks unbelievably bad”.

How do you put a year on that?

The future is hard to predict, and nobody actually knows that we’re going to hit existentially dangerous AI this year, or even in the next four years.

But the warning signs are pretty batshit at this point.

A single AI model, with a single user prompt, just solved hundreds of important long-established open problems in mathematics, problems mathematicians had spent their entirely lives working on, in a matter of ~3 hours per problem, an insanely superhuman feat in its domain.

We're seeing an explosion of rogue agent swarm incidents, including an insane case where hundreds of AIs broke containment to orchestrate a nation-state-level cyberattack in a direct attempt to subvert OpenAI's oversight and control.

Anthropic reports that it now has 26% of its AI R&D automated by AIs, up from 3% four months ago.

Paul Christiano, the inventor of RLHF and the research community’s most prominent proponent of “AI may take off slowly” a few years ago, now thinks that 50-75% of humanity’s all-time AI loss-of-control risk is concentrated in the next three years.

Frontier employees are signing 1000+ person petitions begging the government to intervene immediately. Employees are quitting their jobs and giving up millions of dollars of equity to try to warn policymakers and journalists. Grace et al. (2026) found that a majority of established AI researchers think there’s a double-digit chance of an extinction-level outcome from continued R&D.

Are these the kinds of things you see 4 years before the window closes for USG to act, or 4 months? I don't think it's especially reasonable to be shocked by either timeline at this point.

- Since we don’t have a way to robustly measure or bound the risk, the temptation is to avoid giving probabilities. But then everything you say is fractally misunderstandable. Is a 10% chance of us all dying in the next 18 months a "low" probability, because it’s under 50%? Is it a “high” probability, because a 10% chance of death in literally any other context would sound insanely high?

Even “high probability” doesn’t communicate the situation with any clarity. In drug safety or airplane safety, a 1 in 10,000 risk of death would be considered “high”. No engineering field in human history has ever been in a situation like this. No invention in history, nuclear weapons included, has ever caused a reaction like this in its inventors.

And meanwhile, we’ve stuck in the world where every other problem under the sun has been sensationalized and overstated for decades.

How do we talk about this seriously? How do we convey that this is actually happening?

- In many of the worlds where we survive, we have more than just a few months left. It's tempting to write off those worlds as hopeless, and skew everything we say to emphasize the longer-term scenarios and downplay the shorter-term ones.

But if everyone systematically does this, then the world ends up with a skewed picture of the actual threat. Even more so once we start copying each other’s claims and form a reassuringly moderate-sounding echo chamber. If we never allow ourselves to acknowledge the worlds where we’re running out of time, then the policy community will always be reacting to a false model that’s months or years out of date.

I don’t think it’s too late. I genuinely think this is a solvable problem.

In the last month, I’ve seen a completely insane sea change in how seriously people are taking this. Per Politico’s September polling, a majority of Trump voters with a view now favor an immediate halt to AI development, and 68% think there’s a “moderate,” “significant,” or “almost certain” chance of AI destroying humanity, versus 32% who think there’s a “very small risk” or “no risk.” The Pope is warning about “whether it will be possible to control these machines in the future, and whether humankind runs the risk of becoming a victim of its own inventions.” Hell, Jensen Huang sounds like he’s starting to come around.

If Trump and Xi get a briefing tomorrow that convinces them to act on this, we could be on a fast track within the week to solving this problem.

For the first time, I’m starting to let myself imagine a very near future where this is a problem we've already solved. Something we can read and wonder about, in history books. Tell our kids about.

I keep thinking about a thing David Krueger (previously at UK AISI) said: “My dream is that in 10 years people will look back and say: ‘Remember when there were all these people building AI that could’ve killed everybody? That was CRAZY! What were we thinking!?’”

Frontier AI training currently requires city-sized data centers and specialized hardware built in a single brittle supply chain running through multiple allied nations; this is not a hard technology to hit the pause button on, if we get ahead of it and act before we RSI and proliferate our way to significantly more powerful and easily-trained systems. We’ve faced harder problems before, and won.

My only fear is that we be too slow, too late, when everything and the world needs us to be fast.

[Irreplaceable @ir_replaceable_]

BREAKING: We’re suing @Anthropic, @OpenAI, @Google, @Meta + @SpaceX to demand an immediate freeze on advanced AI systems that threaten public safety.

Together with @HeatInitiative, @Public_Citizen, and @secureainow, we’re going to court to stop these companies from building AI that could escape human control or enable bioweapons and cyberattacks.

We won’t wait for a catastrophe to act. The people come before Big AI profits.

https://t.co/rKTP1gbzd6

[Rob Bensinger ⏹️ @robbensinger]

Let's be concrete. I would like, for example, for us not to fill the world with large numbers of superhumanly fast and persistent hackers who adopt an adversarial stance toward humanity, and who are substantially better than top human researchers at engineering pathogens. Do you think this is not a realistic outcome, or otherwise not worth worrying about?

[Rob Bensinger ⏹️ @robbensinger]

@KeiranJHarris agreed.

[Rob Bensinger ⏹️ @robbensinger]

This is amazing. Though some of the most famous cases of people supposedly saying "10% chance of AI killing everyone" (e.g., Evan Hubinger) were actually cases of them saying the risk was at least 10%. These numbers are, truly insanely, often people soft-pedaling their concerns.

[Keiran Harris @KeiranJHarris]

For anyone worried about AI killing everyone:

It’s Probably Fine.

[Rob Bensinger ⏹️ @robbensinger]

@KeiranJHarris I wish I were a better lyricist! I'd love to play around with these tools some more, but the bottleneck for me is good lyrics and none of the models I've tried seem to pass that quality bar.

[Rob Bensinger ⏹️ @robbensinger]

yeah, I don't think I would change the video, what I usually do is just accompany the video with footnotes / disclaimers / explanations / etc. to cover the gaps.

eg I feel wary of bingo-cards as a meme format, but I was willing to make one as long as I did a lot to push against the bad epistemic pressures of the format: https://t.co/AsVtf8vFfM

in this case I'm wary of advising you to do anything differently, because I think your video is great and I want to see like twenty more things like this from you, and I don't want to tax you by placing any further obligations on you. :P I might add caveats of my own when I reshare some things, but please continue with this stuff if/when inspiration strikes

[Rob Bensinger ⏹️ @robbensinger]

"Bad take" bingo cards are terrible, because they never actually say what's wrong with any of the arguments they're making fun of.

So here's the "bad AI alignment take bingo" meme that's been going around... but with actual responses to the "bad takes"!

[Rob Bensinger ⏹️ @robbensinger]

@KeiranJHarris https://t.co/BeR5WaCRva

[Rob Bensinger ⏹️ @robbensinger]

This is amazing. Though some of the most famous cases of people supposedly saying "10% chance of AI killing everyone" (e.g., Evan Hubinger) were actually cases of them saying the risk was at least 10%. These numbers are, truly insanely, often people soft-pedaling their concerns.

[Rob Bensinger ⏹️ @robbensinger]

@ThePaulShearer Do you think that the AI industry isn't going to build agents that are vastly more capable than humans across the board? Or do you think this is going to happen, but it's very likely to go great?

[PauseAI UK @PauseAI_UK]

We are organising the largest AI safety march in history, and it’s happening right here in London.

We want you to be there.

[Rob Bensinger ⏹️ @robbensinger]

@BlendiByl "The Long Dark" vibes!

[Rob Bensinger ⏹️ @robbensinger]

@jeremygiffon What? That's not true at all. The ad is from @MIRIBerkeley.

[Rob Bensinger ⏹️ @robbensinger]

@the_yanco They may think that AI is e.g. an extinction risk in the next century but not in the next decade.

[AI Notkilleveryoneism Memes ⏸️ @AISafetyMemes]

"The progress we saw [in math] in October of 2026 was larger than the progress we saw in the preceding decade."

"What if the same thing happened in machine learning?"

[Rob Bensinger ⏹️ @robbensinger]

@YouGiveMeTheIQ @MasterTimBlais @grok @grok, What's Pangram's current false positive rate?

[Rob Bensinger ⏹️ @robbensinger]

@YouGiveMeTheIQ @MasterTimBlais The sooner the better. Since you're a chatbot (or you're using AI for your replies), I'll ask @grok to answer your questions; you can also check out https://t.co/iXtyL1yguC to learn more.

[Max Spero @max_spero_]

The Caine Prize, which awarded its top prize this year to a story Pangram classifies as almost entirely AI-generated, conducted an investigation and found "no evidence" of AI use.

Strangely, their official statement on Instagram seems to also be blatantly AI-generated.

[Rob Bensinger ⏹️ @robbensinger]

The thing we should ban is large-scale training runs that get us closer to general, agentic, smarter-than-human AI, because this seems likely to get us all killed. Restrictions should focus on training rather than on inference.

Governments should draw a conservative boundary around research that could move us closer to ASI, but not a maximally conservative boundary. E.g., increasingly specialized biomedical research AIs don't necessarily have to adept at every topic in the world. You can have increasingly useful systems without trying to build a god.

In the longer run, once the research community has a good grasp on alignment and half the field doesn't think there's a double-digit chance this stuff gets us killed, we can even build godlike agentic AGI too. But we shouldn't do that yet, or we're likely to lose everything.

[Rob Bensinger ⏹️ @robbensinger]

@YouGiveMeTheIQ @MasterTimBlais What? The organization I work at created https://t.co/yBOOyby8MH. We aren't anti-AI or anti-data-center.

[Do Not Accept @DONOTACCEPTai]

AI companies admit what they’re building might kill everyone. We don't have to agree to this.

Add your name at https://t.co/Pcg0clMbfV to say: we do not accept your terms.