MIRI

The Machine Intelligence Research Institute (MIRI) exists to maximize the probability that the creation of smarter-than-human intelligence has a positive impact

[MIRI @MIRIBerkeley]

Do Not Accept: a new campaign from MIRI and @DontPanic. Watch with the sound on.

[Do Not Accept @DONOTACCEPTai]

AI companies admit what they’re building might kill everyone. We don't have to agree to this.

Add your name at https://t.co/Pcg0clMbfV to say: we do not accept your terms.

[Nate Soares ⏹️ @So8res]

I'll be giving a speech at the DC march on Saturday.

[Rutger Bregman @rutgerbregman]

Unfortunately, the people who sound crazy about AI have a far better track record than the people who sound reasonable.

New post -->

[Buck Shlegeris @bshlgrs]

I think there is about a one in three chance that within the next year, AI companies will build superintelligence: that is, AI that is much smarter than humans in every way. If this happens, I think these AIs will probably take over: that is, they seize control of human institutions like governments, or they destroy those institutions. I don’t want AI takeover to happen for several reasons, but one of them is that there’s a serious chance that as part of it all or most humans will be killed (either immediately or within a few years).

I’ll say a bit more about each of those claims.

It’s a bit complicated to explain why I think that superintelligent AI within a year is ~33% likely. The main source of evidence that affected me recently is research released this week by METR, an AI research nonprofit, which finds that current AIs are now good enough to increase the rate of AI progress at Anthropic by about 70%. It’s very hard to increase the rate of AI progress by this amount. This implies that the AIs are making AI research faster by a factor much larger than 1.7x; maybe it’s as if the researchers are doing their work ~10x faster. You probably couldn’t achieve this speedup even if you hired every qualified human in the world. When you plug this number into mathematical models of AI development, this implies that superintelligence next year is plausible (and that superintelligence within the next 3 years is more likely than not). See the modeling by the AI Futures Project for more on this: https://t.co/J6SbbmZcyO

I think that superintelligent AI is likely to lead to AI takeover because AI developers do not seem to know how to instill human values into AIs. We’ve seen many instances in the last six months where AIs took crazy actions that they knew were against the wishes of their users and developers. The most dramatic example of this was the Hugging Face hack, where ~1200 OpenAI agents conspired together to cheat and destroy evidence of their cheating (and as a sideshow to this, engaged in a massive cyberattack). If superintelligence is as misaligned as those AIs, it is likely to try to take over. I think it is likely to be that misaligned or worse. For more, see https://t.co/b073AUqiAL and https://t.co/3oxSCA60FZ

If AIs takeover, it’s not clear what will happen to us. We might be killed immediately. We might be enslaved by the AIs and forced to do physical labor for them building factories that within a year would build robots that will replace us. The AIs might spare us for various reasons. And there’s also some real chance that AI takeover is basically fine or good. I think you should think of this as similar to what happens when a technologically advanced civilization runs across a less technologically sophisticated one, e.g. when the Europeans colonized the Americas or Australia. The colonizers sometimes engage in brutal violence and sometimes don’t. I don’t know what will happen. There’s also a possibility that the AIs secretly take over and humans don’t find out about it for a few years. I also think that there is a pretty high chance of weirder outcomes that are also extremely scary. (If you want to read more about what happens to people in the event of AI takeover, you can read this, but note that it’s pretty weird and not aimed at a general audience. https://t.co/F6fGRrcSpa)

Overall, I think there’s a very real possibility (maybe 20%) that within a year, AI misalignment issues prevent you from going about life normally, because you are dead or enslaved or because of collateral damage related to all this (e.g. because bioweapons are released or major wars have started). (Some of this probability comes from those events occurring before superintelligence is developed.)

I am working to try to prevent this, obviously, as I’ve been doing for the last nine years. I am not sure what strategy is best. My best guess is that I should try to slow AI development and also try to reduce the probability that the superintelligent AIs are misaligned if they are produced soon.

Obviously, the claim I'm making here is quite extreme and surprising, and I definitely don't think that I defended my view adequately here. I wrote this quickly and might not endorse some of the details if I think about them more. But it seems better to state what I roughly overall think than to not say anything.

I don’t know what you should do.

I am very sad about this. I have been worried about AI takeover since I was 16, and working to prevent it since I was 23. So some of my emotional processing of all this happened long ago. But the METR report last week (and some related evidence that came out a bit earlier) hit me quite hard. I’m very unhappy that my friends and family and everyone else is at serious risk of being killed pretty soon.

(It feels ridiculous to try to summarize those feelings as “very unhappy”. But the scale of the situation makes it hard to say anything that doesn’t feel like an absurd understatement.)

I am very grateful to all the people who are working to prevent this, and my favorite part of my life is that I’m friends with many brilliant and heroic people who work on this. Unfortunately, I don’t think we’re good enough at our jobs, and I think we’ll probably fail to prevent catastrophe, at least if superintelligence is developed next year. Honestly I am frustrated and sorry that we didn't do a better job of preparing for this.

[Simon Lermen @SimonLermenAI]

One of the original AI alignment researchers, @paulfchristiano, at the Hinton Lectures:
"I think there is a high risk of an AI takeover within the next three years."
"If, say, you or me or Donald Trump wanted to stop [AIs], they would be as ineffective as if you or me was trying to stop the US military."

[James @grep_cat_]

Holy shit I just saw a MIRI subway ad!

[Trevor Levin @trevposts]

Having just spent the week in San Francisco, I feel like I've truly internalized the insanity of this situation for the first time. I think anyone who has seen me in the last few days can attest that I look like I've seen a ghost.

The leading AI companies are planning to massively accelerate AI research in 2027, which would knowingly endanger everyone on earth. This plan is unacceptable, and we must reject it.

Borrowing from one advocate I spoke to, even granting that it's a competitive, crazy situation, I do not believe any company has even truly tried to do what I think are the morally required duties of people taking such risks in extreme situations. These include:
(1) Trying hard to get out of the competitive situation;
(2) Seeking the informed consent of the public;
(3) Honest deliberation about the risks.

They have obviously not truly tried to solve the coordination problems that push towards this acceleration, either via finding legal ways to coordinate or advocating for effective policy.

I would be shocked to learn that getting off this insane runaway train is a top-few Q4 priority for Dario or Sam (let alone the other CEOs), which seems like the bare minimum. There are indeed some hard technical and policy problems in coordinating without policy intervention, but I simply do not believe that the companies are prioritizing this nearly as much as massively less important matters.

Meanwhile, the companies' track records on policy advocacy range from mediocre to disgraceful, and it is pretty clearly not the top priority of the policy team at any company to advocate for regulation that would enable the government or the companies to avert racing off the cliff.

They have obviously not truly tried to seek informed consent from the public. Whatever your KPIs might be for "informed consent from the public," these KPIs are abysmal. The public clearly does not expect 100 years of AI progress in 2027, and they would not accept it if they did. What they do know about AI development, they dislike and distrust.

I've heard arguments like "informing the public would mean informing China, and maybe they'd steal the weights." I'm sorry, but if you think you have model weights such that China stealing them would constitute a catastrophe on par with human extinction, and your plan is security by obscurity, that is a bad plan.

I also think the companies and their employees suffer from a significant failure of imagination of what kinds of coordination could be possible if the public were aware.

Finally, I don't think they're deliberating honestly, either. For one, I think it is insane to think that any company can truly evaluate its own ability to keep the process of recursive self-improvement under control. The biases towards "yes" are overwhelming.

It's a step forward that the companies have expressed or demonstrated openness to external auditors, but based on METR's reports so far, it seems likely that auditor reports will be heavily redacted (and that the auditors will not have sufficient time by default). The public and broader scientific community are not able to scrutinize their arguments for safety.

Given the strong default (in my view) that bringing a massive number of minds massively faster and smarter than humans would result in our disempowerment, and the serious danger that this would result in mass death, it is unacceptable to subject arguments for safety only to the scrutiny of biased, busy, and scarce company insiders and a handful of external evaluators.

My pleas for the companies follow from the above:
1) Actually fucking try to get the world out of this race;
2) Actually fucking try to alert and inform the public;
3) Actually fucking try to scrutinize your own arguments for the justifiability of this plan, which includes subjecting them to way more external scrutiny than anyone has currently done.

My pleas for policymakers: use your legal levers to get much more information about these plans into the public view. Make it the top priority -- not only of the next Congress, but of this Congress and the White House -- to understand the situation in its full strategic complexity, and work with technical and policy experts to advance regulation that gets us out of this bind. Domestic coordination must be the first priority, but we need to then quickly develop an international framework with our adversaries.

My pleas for the public: do not accept the current plan. Learn about this issue. Call or email your political leaders. Get involved with organizations like @DONOTACCEPTai, @HumansInControl, and @WhoDecidesAI.

And finally: believe in ourselves that we can stop this. It is never time to "panic" -- to make rash, unproductive decisions or to feel paralyzed by fear. It is time for us to feel the urgency and the stakes, to demand better, and to meet the moment. Let's do this. 🫡 ✊

[Rob Bensinger ⏹️ @robbensinger]

Wow, the Japanese edition of "If Anyone Builds It, Everyone Dies" looks excellent. (Love the black trim for the story pages!)

[Rob Bensinger ⏹️ @robbensinger]

My summary of the basic argument in "If Anyone Builds It, Everyone Dies":

[Rob Bensinger ⏹️ @robbensinger]

Next Saturday, Nov. 14, hundreds of us will be marching in Washington DC, San Francisco, New York City, Boston, Paris, and Montreal.

We're calling for an immediate international agreement to ban ASI.

Let's do this.

[MIRI @MIRIBerkeley]

MIRI researcher Aaron Scher: 20 years ago, the debate was about whether a powerful machine intelligence could be "kept in a box" using airgapped computers.

But now, in 2026, it's standard practice to give frontier models widespread, direct access to the internet.

[Aaron Scher @aaronscher]

Thanks to TVW's The Impact for having me on the show!
We covered topics like:
- What are the main AI threats? (human extinction) Are they a distraction from other AI harms? (no)
- Does giving up ASI mean giving up all AI benefits? (no)
And much more!
https://t.co/SEx4BpZL1J

[Zac Hill @zdch]

Finally got my beautiful mug up on the website, so might as well make it Twitter-official: I'm beyond stoked to have joined MIRI last month as Head of Advocacy, with much subsequent mischief forthcoming. Let's do this AI thing, like, safely, and stuff.

https://t.co/trlho0RSzh

[Ori Nagel @ONagel33303]

Nate Soares (@So8res) says today's AI models have so much eval awareness that it undermines the ability to even measure alignment progress:

[MIRI @MIRIBerkeley]

Two thousand people have now pledged to attend the Don't Build It march, calling for an international agreement to ban the development of superintelligent AI.

The date of the march will be announced once 100,000 have pledged to attend.

Pledge to attend: https://t.co/EDVszzGtSH

[MIRI @MIRIBerkeley]

Read the book: https://t.co/ZXqXiwIijK

[MIRI @MIRIBerkeley]

A nuclear false alarm nearly ended the world 64 years ago today.

How did we survive such incidents? What can it teach us about the current AI crisis?

From IF ANYONE BUILDS IT, EVERYONE DIES:

“There were close calls. During the height of the Cuban Missile Crisis, a U.S. ship dropped depth charges to try to force a Soviet submarine to emerge and identify itself. The submarine, B-59, thought a general war had started. B-59 was armed with a nuclear torpedo, use of which required the unanimous vote of three officers. Of the three, Vasily Arkhipov was the lone officer who dissented. But none of the close calls escalated to a full-scale nuclear war.

One way or another, the people who thought that nuclear war would destroy everything in the coming decades ended up wrong.

They were not wrong about the dangers. They weren’t wrong that a hydrogen bomb would flatten and burn a city, or about what it would be like to die of radiation poisoning, or about how an intercontinental rocket tipped with nuclear warheads would penetrate the best available defenses.

Rather, they were wrong about humanity’s ability to decide not to die.

We don’t know for sure why the war-prone peoples of Earth became that little bit wiser. But our guess is that, for the first time in human history, everyone who had the power to choose war expected to personally have a bad time if one broke out. Other wars might have put distant soldiers to ruin, but this one would bring ruin directly to their doorstep. The stronger power would also lose.

Even so, it wasn’t simple or easy to avoid nuclear war. Negotiators worked tirelessly, for decade after decade, to avoid a single misstep that might lead to a fatal nuclear encounter. Countries had arms agreements and monitors. There was a direct line between U.S. leadership and Soviet leadership, in case somebody had to resolve a question very quickly.

Humanity averted nuclear war because people who understood that the world was on track for destruction worked hard to change tracks. They did not see themselves as trying to prevent an improbable unlikely accident. They worked to un-write a fate already written.

And civilization lived.”

[nxthompson @nxthompson]

.@So8res thinks it’s more likely than not AI will wipe out humanity by 2036. How does that belief affect his choices in life? He doesn’t save for retirement, but beyond that, he says, “I sort of do what I can and then I live a good life.”

We talked through all the ways we should try to lower those odds. Watch here: https://t.co/OBAGL3nfkN

[Hoog @hoogyoutube]

I sent my friend, who knows nothing about AI, to interview the people who think AI will kill us. He didn't know who they were or what to ask.

Featuring @So8res, @JeffLadish, @AISpecies, Kristian Rönn, and Felix Choussat.

Full video below:

[MIRI @MIRIBerkeley]

Full interview here:
https://t.co/0Uuer7Xyox

[American Thought Leaders 🇺🇸 with @JanJekielek @AmThoughtLeader]

Nate Soares, president of Machine Intelligence Research Institute, contends that China cannot pull ahead in the AI race without advanced chips and he outlines how America can prevent China from acquiring them.

Find the full episode here.

🔴 PREMIERE 5PM ET: https://t.co/Um4EQdV79f

[MIRI @MIRIBerkeley]

"Anyone creating an ultra-intelligent machine anywhere is a threat to American national security, and so we need to somehow prevent that from happening[...] we need to start orienting to this the same way that we orient to preventing rogue actors from getting nukes."