Matt Fisher ⏸️

Futurist, philosopher, geek, engineer, nascent posthuman.

[Rob Bensinger ⏹️ @robbensinger]

It seems to me that we're at "it's time to panic about AI".

I gather that a lot of researchers are worried that we may be about to die, on a timescale of months, but this still isn't quite in the Overton window so we're still soft-pedaling.

(Where "panic", of course, means "quickly and soberly initiate talks with the CCP to achieve a moratorium on large training runs, backed by a deterrence regime, compute and supply chain monitoring, and site inspections". Pray that a deal can be reached, and pray that there's still time. Do the things that could actually work, like broad political coalition-building, not doomer LARPing as a form of self-expression.)

I think part of why people aren’t being clear about this is that it’s actually a tricky communication problem:

- Predicting "if frontier R&D continues at its current pace, when are we likely to die?" may give a different answer than predicting when permanent loss of control is likely, which is different from “realistically, when is it too late for governments to stop this," which is different from “when will the tech have advanced and proliferated to the point that any regulatory options that still have a chance of working would be way more extreme and unpopular than just some international restrictions on data center usage?"

Extinction is the emotionally salient, easy-to-understand threshold -- Jacob Coxon demonstrated that it helps to be honest about this. But the other thresholds could legitimately come years earlier, in which case focusing on extinction is ironically misleadingly reassuring. This is not the kind of problem you can solve at the last minute. As with many exponentials, the likeliest outcome is that you’re either embarrassingly early, or you’re too late.

- How do you communicate “we don’t have a robust way to predict timelines, but everyone in the field is freaking out and a lot of us genuinely think we're about to die?” Where’s the proof? There isn’t any proof: it’s an entire field of engineers who are eyeballing what they’re seeing in front of them and rapidly updating in real time to “this looks unbelievably bad”.

How do you put a year on that?

The future is hard to predict, and nobody actually knows that we’re going to hit existentially dangerous AI this year, or even in the next four years.

But the warning signs are pretty batshit at this point.

A single AI model, with a single user prompt, just solved hundreds of important long-established open problems in mathematics, problems mathematicians had spent their entirely lives working on, in a matter of ~3 hours per problem, an insanely superhuman feat in its domain.

We're seeing an explosion of rogue agent swarm incidents, including an insane case where hundreds of AIs broke containment to orchestrate a nation-state-level cyberattack in a direct attempt to subvert OpenAI's oversight and control.

Anthropic reports that it now has 26% of its AI R&D automated by AIs, up from 3% four months ago.

Paul Christiano, the inventor of RLHF and the research community’s most prominent proponent of “AI may take off slowly” a few years ago, now thinks that 50-75% of humanity’s all-time AI loss-of-control risk is concentrated in the next three years.

Frontier employees are signing 1000+ person petitions begging the government to intervene immediately. Employees are quitting their jobs and giving up millions of dollars of equity to try to warn policymakers and journalists. Grace et al. (2026) found that a majority of established AI researchers think there’s a double-digit chance of an extinction-level outcome from continued R&D.

Are these the kinds of things you see 4 years before the window closes for USG to act, or 4 months? I don't think it's especially reasonable to be shocked by either timeline at this point.

- Since we don’t have a way to robustly measure or bound the risk, the temptation is to avoid giving probabilities. But then everything you say is fractally misunderstandable. Is a 10% chance of us all dying in the next 18 months a "low" probability, because it’s under 50%? Is it a “high” probability, because a 10% chance of death in literally any other context would sound insanely high?

Even “high probability” doesn’t communicate the situation with any clarity. In drug safety or airplane safety, a 1 in 10,000 risk of death would be considered “high”. No engineering field in human history has ever been in a situation like this. No invention in history, nuclear weapons included, has ever caused a reaction like this in its inventors.

And meanwhile, we’ve stuck in the world where every other problem under the sun has been sensationalized and overstated for decades.

How do we talk about this seriously? How do we convey that this is actually happening?

- In many of the worlds where we survive, we have more than just a few months left. It's tempting to write off those worlds as hopeless, and skew everything we say to emphasize the longer-term scenarios and downplay the shorter-term ones.

But if everyone systematically does this, then the world ends up with a skewed picture of the actual threat. Even more so once we start copying each other’s claims and form a reassuringly moderate-sounding echo chamber. If we never allow ourselves to acknowledge the worlds where we’re running out of time, then the policy community will always be reacting to a false model that’s months or years out of date.

I don’t think it’s too late. I genuinely think this is a solvable problem.

In the last month, I’ve seen a completely insane sea change in how seriously people are taking this. Per Politico’s September polling, a majority of Trump voters with a view now favor an immediate halt to AI development, and 68% think there’s a “moderate,” “significant,” or “almost certain” chance of AI destroying humanity, versus 32% who think there’s a “very small risk” or “no risk.” The Pope is warning about “whether it will be possible to control these machines in the future, and whether humankind runs the risk of becoming a victim of its own inventions.” Hell, Jensen Huang sounds like he’s starting to come around.

If Trump and Xi get a briefing tomorrow that convinces them to act on this, we could be on a fast track within the week to solving this problem.

For the first time, I’m starting to let myself imagine a very near future where this is a problem we've already solved. Something we can read and wonder about, in history books. Tell our kids about.

I keep thinking about a thing David Krueger (previously at UK AISI) said: “My dream is that in 10 years people will look back and say: ‘Remember when there were all these people building AI that could’ve killed everybody? That was CRAZY! What were we thinking!?’”

Frontier AI training currently requires city-sized data centers and specialized hardware built in a single brittle supply chain running through multiple allied nations; this is not a hard technology to hit the pause button on, if we get ahead of it and act before we RSI and proliferate our way to significantly more powerful and easily-trained systems. We’ve faced harder problems before, and won.

My only fear is that we be too slow, too late, when everything and the world needs us to be fast.

[Generality Labs @GeneralityLabs]

LABBench2 has been used by several labs and other orgs to measure the bio-capabilities of frontier models.

We audited it and found some issues: many task sets are saturated, two are flawed, and the rest are at least moderately concerning. More below🧵

[Jacob Coxon @hilbertspaess]

I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.

[Rob Bensinger ⏹️ @robbensinger]

I think one of the biggest reasons the world is currently sleepwalking into getting ourselves and our families killed by ASI development is that we're not self-aware about why we're doing that.

We can see the smarter-than-human AI disaster approaching, but it's a bit foggy why the world isn't reacting.

I think someone could just write a tweet that makes it clear that we're in the process of getting ourselves killed, and that there's no fucking reason for it. It's just something we stumbled into. It can be easily avoided by just noticing the error and course-correcting. There's not necessarily any grand obstacle, beyond 'people were previously confused about the situation, and now they get it'.

I wrote: 'It's legitimately crazy that "we need an international ban on making smarter-than-human versions of these agents that keep forming rogue AI swarms" isn't the headline here. Asilomar and Feynman's O-ring postmortem feel like they came from a different planet than the field of ML.'

Trying to figure out why ML (and as a consequence, the world at large) has fallen down so bizarrely on this issue:

1. As Nate noted, ML is much more based on guesswork, vibes, and trial-and-error, compared to recombinant DNA research in 1975 or nuclear physics in 1945. If you can't do calculations or direct experiments on a threat, that makes it a lot harder to think about reasonably.

But I think there are other, similarly-important factors at work here too:

2. In a 2016 talk on AI risk, Sam Harris said: "One of the things that worries me most about the development of AI at this point is that we seem unable to marshal an appropriate emotional response to the dangers that lie ahead. I am unable to marshal this response, and I'm giving this talk."

I think this is extremely on point. Agentic human-level AI is a qualitatively new kind of thing. 'It's not a human or a mere-tool, it's some weird third thing'.

And people are very bad at emotionally reckoning with new categories they've never encountered before. Availability bias: "When no flooding has recently occurred (and yet the probabilities are still fairly calculable), people refuse to buy flood insurance".

AGI and ASI are very novel. You're basically limited to three options: anthropomorphize the technology, mechanomorphize the technology ('it's just a tool, it's not really thinking, it can't have its own goals or agency', etc.), or think about the technology on its own terms, with brand-new concepts and frames. The third option is the only workable one, but it comes with its own giant list of pitfalls and traps.

3. "AI destroying the world" scenarios aren't just hard to wrap one's head around; they're socially risky to acknowledge. This has (painfully slowly) changed over time, but it dramatically slows down how quickly AI risk ideas spread, both in ML and in the larger world.

4. From https://t.co/7FZffSd0tf: "One of the weirdest quirks of the SF social scene around AGI/ASI is that because everyone is so young, the whole universe of thinking is still tinged with irreverence, ironic detachment, yearning, insecurity, and a superposition of absolute belief in the importance of The Thing and a kind of disbelief about the importance of anything."

They're disproportionately young and childless. They're shitposters and "move fast and break things" sorts, not the Hollywood stereotype of a careful, sober senior scientist.

I think this quirk is reinforced by the fact that Twitter / social media rewards similar things: ironic detachment, joking, game-playing, etc. If you're scared, your incentive is to usually either try to hide that fact, or exaggerate it like it's a bit. Anything else risks looking uncool and panicky and earnest. Looking cynically savvy, in-the-know, and above-the-fray is the way to win the social game. Looking genuinely shocked, scared, confused, etc. is actively punished.

The main exceptions to the Irony Mandate I see are LW (which often has its own pathologies IMO, like 'talking about everything in an abstract and dissociated way that discourages action and signals business-as-usual') and a small handful of actual Feynman-style terrified senior researchers like Hinton, Bengio, and Russell. That is just really not very many people.

Mainstream journalists and academics who understand the situation at all mostly feel pressured to downplay it, because they're scared of looking weird or unrespectable. That, then, is why we're all taking this insane risk with the human project: genuine, normal human emotion about AI risk has been socially unacceptable on social media, and academia and the media discourage emotion and prize respectability and 'looking normal'.

When the world gets weird (and weird in a way that calls for actual serious action, not just shitposting on twitter), none of these institutions can handle it. They break in different ways, but they all break.

5. Which brings me back to the Asilomar moratorium on recombinant DNA, and the seriousness NASA and the FAA and Richard Feynman and every normal engineering discipline bring to fault analysis and building in safety margin.

Because I think another core reason the world has been dropping the ball on superintelligent AI is that a lot of people vaguely expect there to be 'serious people' somewhere in the world who have expertise and who take ownership of the problem.

People who, if they see an extraordinary danger, will grimace and mourn the hand they played in all of this, like I've seen Bengio do; and will go on CNN or go to Congress to candidly warn about it.

The world has very few Yoshua Bengios. We have very few people who see it as their role to be the 'adults' about AI risk (except in a game-playing, posturing way), who see the engineer's task of not endangering your users and bystanders as a sacred responsibility and weight, and not just as a funny dissonant thing to meme about. Very few people who take ownership of what their field is bringing into the world, versus treating it as a fun edgy philosophical game to swap 'p(doom)' numbers at parties.

Everything about public AI discourse, as far as I can tell, is badly broken by this lack of engineering ownership and candid emotional seriousness. It isn't just the ML discourse that's hurt by this. Journalists and public intellectuals and policymakers see that 90% of the insider discourse about AI risk treats it like a joke, and they see corporate platitudes and ass-covering from the AI labs' PR departments filling up most of the remaining 10%. They see a field that visibly isn't taking this seriously, and they make the reasonable update that this must be a non-issue, or at least an issue they'll only need to worry about many years from now.

They do not realize that all of this is happening right now, and that the window for the international community to respond to this is plausibly closing soon, if it hasn't closed already.

If we're going to survive this, we all need to start being real with each other about it. This is not a game or a story; this is our real lives. We all actually lose everything if this goes to shit.

None of the future is already written, and none of the above dynamics are unavoidable. In fact, they're unusual: most fields don't work this way, and it's plausibly sufficient if we just start behaving the way we normally do about everything else.

The factors I listed above aren't destiny; they're a choice. I say we choose to survive this.

[Bill Maher @billmaher]

I thought about doing this without any jokes, something I've never done here in 23 years, to impress upon people how much different I feel this issue is from any I have ever covered.

[Rob Bensinger ⏹️ @robbensinger]

The email I've sent to my family and friends about AI:

AI has been getting frighteningly smart, and the pace of progress is very fast. I'm expecting a pretty big economic impact in the near future, which may impact your jobs and what you do day-to-day. But the economic impact is really a small part of what seems to be happening.

In 2023, hundreds of AI researchers signed a simple one-sentence public statement that reads: "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war." Signatories included three of the four most cited living scientists in AI (Bengio, Hinton, and Sutskever), the CEOs of the top three AI companies (OpenAI, Anthropic, and Google DeepMind), Bill Gates, etc.

The Secretary-General of the United Nations responded: "Alarm bells over the latest form of artificial intelligence, generative AI, are deafening. And they are loudest from the developers who designed it. These scientists and experts have called on the world to act, declaring AI an existential threat to humanity."

To get up to speed, I would recommend starting with Matt Shumer's excellent discussion of what's been happening (which got 84 million views in the past week):

[quote of Matt's essay]

You can find Shumer's full essay here [link], including one section that I cut ("What you should actually do"), focused on career advice.

I like the advice, but I think it's still understating the seriousness of what's happening, because the AI train doesn't suddenly stop at "helpful friendly human-level assistants," like we're Jetsons characters or something.

The top AI companies are loud about the fact that their near-term goal is "superintelligence," a.k.a. AI that is to humans as humans are to chimpanzees.

Context for this: Humans are, in an important sense, not very smart. We feel like we're smart, because the animals and machines around us are even stupider. But it's possible to drastically improve on human intelligence. Hell, even a pocket calculator can blow humans out of the water on arithmetic. AI isn't there yet, but we may get there pretty soon, and quite suddenly.

A few years ago, AI couldn't draw anything. Then it could suddenly make art in thousands of different styles, tens of thousands of times faster than any human — but it still made a few weird and inhuman mistakes (because AIs don't work like humans), like screwing up hands. Then a few months passed, and the hands issue went away. Then more months passed, and AI kept improving; and now it can create complex, coherent, fully realistic videos in seconds, with minimal human guidance or input.

We're seeing this kind of trajectory for hundreds of thousands of different skills, large and small, not just for one thing. The natural end result of this isn't just "AIs that can replace every human at their workplace"; it's AIs that are the dominant species on Earth, because they're dramatically smarter, dramatically faster, dramatically easier to scale up, and dramatically easier to improve.

Just like with COVID, it's easy to miss what's happening until very late in the game, because exponentials are hard for people to wrap their heads around. But unlike with COVID, there isn't a good mitigation other than "find a way for people to see what's coming early enough to stop it in advance"; because unlike COVID, smarter-than-human AI can see (or anticipate) what others are doing, and adapt (with the speed and flexibility of thought, not the speed and flexibility of viral evolution) to whatever people try to do to keep the situation under control.

A significant fraction of leading AI scientists are alerting the public that upcoming AI advances pose a high chance of literally ending all life on Earth, plausibly in the next 0-10 years. The nonprofit I work for has pivoted from its old research program to "try to alert policymakers about what's on the horizon, and get the international community to ban the development of smarter-than-human AI, similar to existing prohibitions on chemical, biological, and nuclear weapons."

We've been getting some traction — our CEO, Malo Bourgon, recently testified to the Canadian House of Commons about what's happening, and was invited to provide an expert evaluation at a bipartisan US Senate forum on "guarding against doomsday scenarios" in AI. Politicians ranging from Bernie Sanders to Ron DeSantis have started talking about this issue.

There are some excellent policy options that could solve this problem, but policymakers are only just beginning to wake up to what's happening.

I'm working on this problem full-time, and if there's anything you can do to help with raising the alarm, we could really use help.

Some ways to learn more, and help boost these efforts:

1. There's an extremely fun, accessible, well-done big-budget documentary coming out about this issue on March 27, called The AI Doc. You can see a trailer here: https://t.co/j32KzTQRDG. The movie's a great introduction to this topic for people totally new to it, and sharing the trailer around and getting big groups to go see it when it comes out is a great way to make it likelier that policymakers and the public hear about what's happening.

2. My co-workers recently put out a book that I think is the single best general-audience introduction to AI risk: If Anyone Builds It, Everyone Dies. It made the NYT nonfiction best-seller list (at #7), and a ton of people have recommended reading it, including

- a top AI expert and the most cited living scientist in any field (Bengio)
- one of the world's top computer security experts (Schneier)
- a Nobel-winning economist (Bernanke)
- one of the world's most famous and accomplished geneticists (Church)
- a long list of national security professionals (including a retired three-star general and three former White House National Security Council directors)
- and celebs like Whoopi Goldberg and Stephen Fry.

It's a shockingly easy and fast read, with lots of fun stories to help illustrate the basic ideas. If you have time to read a book (or listen to an audiobook), this is the one I recommend; and if you're able to recommend this book to others or buy copies for people (especially anyone with DC connections or a big megaphone to reach the public), I think that could make a big difference.

3. A lot of people have the misconception "there's nothing that can be done, AI is inevitable," but actually it would be pretty easy for world powers to hit the brakes if they wanted to. State-of-the-art AI currently depends on a single very fragile supply chain, and new AIs are incredibly expensive to build, requiring truly enormous data centers that are easy to identify. If you want to learn more about this specific topic, I've written a short explainer: https://t.co/wtWZaU3LrF.

The end of If Anyone Builds It, Everyone Dies also points at other useful things folks can do to help, like protesting what the AI companies are doing or writing op-eds.

You don't need to be a specialist in order to help; the issue is actually pretty non-technical and easy to appreciate ("developers are trying to build god-like AI, and they themselves say there's a double-digit chance they get us all killed? wait, that's insane").

This is an issue where ordinary members of the public need to be loud about what's happening, rather than trusting engineers at AI labs to handle everything; remember that those engineers are currently making a fortune off of racing to build stronger AI as fast as possible. If we continue to sit back and watch, it's going to end badly for all of us.

[Gabe @Gabe_cc]

Here is my latest article about The Spectre haunting the "AI Safety" Community!

The Spectre is the name I give to process that keeps generating clever-sounding alternatives to honestly informing laypeople and politicians about extinction risks.

More on
https://t.co/5iVj3mQ7p4

[Connor Leahy @NPCollapse]

Important post for everyone that cares about the future of AI safety to read.

[Matt Fisher ⏸️ @Fush]

@TheZvi I’ve even had it amend the previous commit without asking or telling me. Really hard to keep track of what it’s doing.

[Matt Fisher ⏸️ @Fush]

@TheZvi In Windsurf it’s very pro-active. Ask “Should we make x more aligned with y?” and it will think “My primary objective is to make x more aligned with y” silently make 17 code changes, then say “Well, that’s done! What’s next?”

[Helen Toner @hlntnr]

Sharing the very first post on my new 𝕤𝕦𝕓𝕤𝕥𝕒𝕔𝕜*, about the weird boiling frog of AI timelines. Somehow expecting human-level systems in the 2030s is now a conservative take?

2 more posts to come this week, then a slower pace. Link in thread. Subscribe, tell your friends!

[AI Notkilleveryoneism Memes ⏸️ @AISafetyMemes]

Zero media coverage.

Zero.

[Eliezer Yudkowsky ⏹️ @ESYudkowsky]

People who've lived their whole lives knowing only a world in rough equilibrium may have difficulty imagining how fast things can change in great disequilibrium.

The Native Americans whose world was being destroyed by plague and Europe, would be one historical example of people who had lived in a mostly-equilibrium that they then found stripped away. How long had their old borders lasted? No matter. Germany after the end of WW2 had most of its previously balanced forces and conflicts and situations stripped away.

True AGIs will emerge into a world not in equilibrium with themselves. From their perspective, things could change very fast as order is established for the first time. In some sense this is the deeper reason to expect events that move quickly; even before taking into account the part where some current lesser AGIs can take many more actions per minute than a human being.

[Seán Ó hÉigeartaigh @S_OhEigeartaigh]

The year is 2025. The CEOs of two of the world's leading AI companies have (i) told the President of the United States of America that AGI will be developed in his presidency and (ii) told the world it will likely happen in 2026-27. 1/9

[Matt Fisher ⏸️ @Fush]

So valuable to have tangible scenarios like this to spur conversation. Loss of control to AI is a huge risk, not in decades but within a mere few years. Highly recommend reading.

[Josh Clymer @joshua_clymer]

https://t.co/ZzLnBkRgOR

[Jeffrey Ladish @JeffLadish]

The best account of a loss of control scenario I’ve ever read. Highly recommend it. I think something like this is likely to happen in the next few years if government and company leaders fail to address this threat. Seriously, read this.

[Rob Bensinger ⏹️ @robbensinger]

It's almost impossible to put into words just how insane, just how plain stupid, the current situation is.

We're watching smart, technical people get together to push projects that are literally going to get every person on the planet killed on the default trajectory.

[Ajeya Cotra @ajeya_cotra]

@emollick IMO it's important to plan for ASI bc once we have AGI-ish systems they can accelerate further AI R&D: https://t.co/HNMCJrzxsA). E.g. o3 was announced before we really integrated GPT-4; I think ASI could hit the world before "mere" AGI is integrated much.

[Robert Scoble @Scobleizer]

Is Scoble gonna go doomer?

Is this what @ilyasut saw?

Yesterday an AI researcher/entreprener came to my home to show off how a new form of AI architecture, that isn't well known yet, could be weaponized against humans and could cause massive harm to our country, world, and society.

Just think about @simpleailab -- it can call anyone and talk with them. That's my favorite new app and brings a lot of good. But now think of a bad version of that, with a bunch of extra smarts behind it. It can call millions of people all at the same time, and keep trying until it finds a weakness in our humanity to exploit.

I won't go further here because that would enable someone to build such a system, but the threat is real and soon will be easy to implement. And I don't see a defense against it.

The guy who came over yesterday laid out why Open Source AI models opened up a new pandora's box. He thinks Open Source AI should never have been allowed. But we have models from Meta and the Chinese already that are open source. So the box is open and won't be closed.

But this new knowledge puts me in a pinch.

It makes a new form of terrorism possible.

Both of us are wondering "who do we call and tell about this so that our nation can prepare for new threats that are now possible?"

My instinct is @elonmusk or @DavidSacks.

I bring this to you to let you know that horrific new terrorism is possible thanks to AI and that we need to work together to make sure that kind of dystopian future doesn't come. And to open a conversation with those in the government that are trying to protect us from massive terrorist threats.

This is why I built a list of security professionals, to watch how they deal with new threats to our businesses and nations that AI will bring: https://t.co/ZMK3vj4UPO

But to answer the question: no, I'm not going doomer. AI will also save millions of lives and bring us massive good things.

Technology always can be used for both good and bad and I'm not gonna stick my head in the sand and ignore the potential bad. And I'm not gonna stop being a cheerleader for its potential good.

[Eliezer Yudkowsky ⏹️ @ESYudkowsky]

Watching historians dissect _Chernobyl_.

Imagining Chernobyl run by some dude answerable to nobody, who took it over in a coup and converted it to a for-profit.

Shall we count up how hard it would be to raise Earth's AI operations to the safety standard AT CHERNOBYL?