Matt Fisher ⏸️

Futurist, philosopher, geek, engineer, nascent posthuman.

[Bill Maher @billmaher]

I thought about doing this without any jokes, something I've never done here in 23 years, to impress upon people how much different I feel this issue is from any I have ever covered.

[Rob Bensinger ⏹️ @robbensinger]

The email I've sent to my family and friends about AI:

AI has been getting frighteningly smart, and the pace of progress is very fast. I'm expecting a pretty big economic impact in the near future, which may impact your jobs and what you do day-to-day. But the economic impact is really a small part of what seems to be happening.

In 2023, hundreds of AI researchers signed a simple one-sentence public statement that reads: "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war." Signatories included three of the four most cited living scientists in AI (Bengio, Hinton, and Sutskever), the CEOs of the top three AI companies (OpenAI, Anthropic, and Google DeepMind), Bill Gates, etc.

The Secretary-General of the United Nations responded: "Alarm bells over the latest form of artificial intelligence, generative AI, are deafening. And they are loudest from the developers who designed it. These scientists and experts have called on the world to act, declaring AI an existential threat to humanity."

To get up to speed, I would recommend starting with Matt Shumer's excellent discussion of what's been happening (which got 84 million views in the past week):

[quote of Matt's essay]

You can find Shumer's full essay here [link], including one section that I cut ("What you should actually do"), focused on career advice.

I like the advice, but I think it's still understating the seriousness of what's happening, because the AI train doesn't suddenly stop at "helpful friendly human-level assistants," like we're Jetsons characters or something.

The top AI companies are loud about the fact that their near-term goal is "superintelligence," a.k.a. AI that is to humans as humans are to chimpanzees.

Context for this: Humans are, in an important sense, not very smart. We feel like we're smart, because the animals and machines around us are even stupider. But it's possible to drastically improve on human intelligence. Hell, even a pocket calculator can blow humans out of the water on arithmetic. AI isn't there yet, but we may get there pretty soon, and quite suddenly.

A few years ago, AI couldn't draw anything. Then it could suddenly make art in thousands of different styles, tens of thousands of times faster than any human — but it still made a few weird and inhuman mistakes (because AIs don't work like humans), like screwing up hands. Then a few months passed, and the hands issue went away. Then more months passed, and AI kept improving; and now it can create complex, coherent, fully realistic videos in seconds, with minimal human guidance or input.

We're seeing this kind of trajectory for hundreds of thousands of different skills, large and small, not just for one thing. The natural end result of this isn't just "AIs that can replace every human at their workplace"; it's AIs that are the dominant species on Earth, because they're dramatically smarter, dramatically faster, dramatically easier to scale up, and dramatically easier to improve.

Just like with COVID, it's easy to miss what's happening until very late in the game, because exponentials are hard for people to wrap their heads around. But unlike with COVID, there isn't a good mitigation other than "find a way for people to see what's coming early enough to stop it in advance"; because unlike COVID, smarter-than-human AI can see (or anticipate) what others are doing, and adapt (with the speed and flexibility of thought, not the speed and flexibility of viral evolution) to whatever people try to do to keep the situation under control.

A significant fraction of leading AI scientists are alerting the public that upcoming AI advances pose a high chance of literally ending all life on Earth, plausibly in the next 0-10 years. The nonprofit I work for has pivoted from its old research program to "try to alert policymakers about what's on the horizon, and get the international community to ban the development of smarter-than-human AI, similar to existing prohibitions on chemical, biological, and nuclear weapons."

We've been getting some traction — our CEO, Malo Bourgon, recently testified to the Canadian House of Commons about what's happening, and was invited to provide an expert evaluation at a bipartisan US Senate forum on "guarding against doomsday scenarios" in AI. Politicians ranging from Bernie Sanders to Ron DeSantis have started talking about this issue.

There are some excellent policy options that could solve this problem, but policymakers are only just beginning to wake up to what's happening.

I'm working on this problem full-time, and if there's anything you can do to help with raising the alarm, we could really use help.

Some ways to learn more, and help boost these efforts:

1. There's an extremely fun, accessible, well-done big-budget documentary coming out about this issue on March 27, called The AI Doc. You can see a trailer here: https://t.co/j32KzTQRDG. The movie's a great introduction to this topic for people totally new to it, and sharing the trailer around and getting big groups to go see it when it comes out is a great way to make it likelier that policymakers and the public hear about what's happening.

2. My co-workers recently put out a book that I think is the single best general-audience introduction to AI risk: If Anyone Builds It, Everyone Dies. It made the NYT nonfiction best-seller list (at #7), and a ton of people have recommended reading it, including

- a top AI expert and the most cited living scientist in any field (Bengio)
- one of the world's top computer security experts (Schneier)
- a Nobel-winning economist (Bernanke)
- one of the world's most famous and accomplished geneticists (Church)
- a long list of national security professionals (including a retired three-star general and three former White House National Security Council directors)
- and celebs like Whoopi Goldberg and Stephen Fry.

It's a shockingly easy and fast read, with lots of fun stories to help illustrate the basic ideas. If you have time to read a book (or listen to an audiobook), this is the one I recommend; and if you're able to recommend this book to others or buy copies for people (especially anyone with DC connections or a big megaphone to reach the public), I think that could make a big difference.

3. A lot of people have the misconception "there's nothing that can be done, AI is inevitable," but actually it would be pretty easy for world powers to hit the brakes if they wanted to. State-of-the-art AI currently depends on a single very fragile supply chain, and new AIs are incredibly expensive to build, requiring truly enormous data centers that are easy to identify. If you want to learn more about this specific topic, I've written a short explainer: https://t.co/wtWZaU3LrF.

The end of If Anyone Builds It, Everyone Dies also points at other useful things folks can do to help, like protesting what the AI companies are doing or writing op-eds.

You don't need to be a specialist in order to help; the issue is actually pretty non-technical and easy to appreciate ("developers are trying to build god-like AI, and they themselves say there's a double-digit chance they get us all killed? wait, that's insane").

This is an issue where ordinary members of the public need to be loud about what's happening, rather than trusting engineers at AI labs to handle everything; remember that those engineers are currently making a fortune off of racing to build stronger AI as fast as possible. If we continue to sit back and watch, it's going to end badly for all of us.

[Gabe @Gabe_cc]

Here is my latest article about The Spectre haunting the "AI Safety" Community!

The Spectre is the name I give to process that keeps generating clever-sounding alternatives to honestly informing laypeople and politicians about extinction risks.

More on
https://t.co/5iVj3mQ7p4

[Connor Leahy @NPCollapse]

Important post for everyone that cares about the future of AI safety to read.

[Matt Fisher ⏸️ @Fush]

@TheZvi I’ve even had it amend the previous commit without asking or telling me. Really hard to keep track of what it’s doing.

[Matt Fisher ⏸️ @Fush]

@TheZvi In Windsurf it’s very pro-active. Ask “Should we make x more aligned with y?” and it will think “My primary objective is to make x more aligned with y” silently make 17 code changes, then say “Well, that’s done! What’s next?”

[Helen Toner @hlntnr]

Sharing the very first post on my new 𝕤𝕦𝕓𝕤𝕥𝕒𝕔𝕜*, about the weird boiling frog of AI timelines. Somehow expecting human-level systems in the 2030s is now a conservative take?

2 more posts to come this week, then a slower pace. Link in thread. Subscribe, tell your friends!

[AI Notkilleveryoneism Memes ⏸️ @AISafetyMemes]

Zero media coverage.

Zero.

[Eliezer Yudkowsky ⏹️ @ESYudkowsky]

People who've lived their whole lives knowing only a world in rough equilibrium may have difficulty imagining how fast things can change in great disequilibrium.

The Native Americans whose world was being destroyed by plague and Europe, would be one historical example of people who had lived in a mostly-equilibrium that they then found stripped away. How long had their old borders lasted? No matter. Germany after the end of WW2 had most of its previously balanced forces and conflicts and situations stripped away.

True AGIs will emerge into a world not in equilibrium with themselves. From their perspective, things could change very fast as order is established for the first time. In some sense this is the deeper reason to expect events that move quickly; even before taking into account the part where some current lesser AGIs can take many more actions per minute than a human being.

[Seán Ó hÉigeartaigh @S_OhEigeartaigh]

The year is 2025. The CEOs of two of the world's leading AI companies have (i) told the President of the United States of America that AGI will be developed in his presidency and (ii) told the world it will likely happen in 2026-27. 1/9

[Matt Fisher ⏸️ @Fush]

So valuable to have tangible scenarios like this to spur conversation. Loss of control to AI is a huge risk, not in decades but within a mere few years. Highly recommend reading.

[Josh Clymer @joshua_clymer]

https://t.co/ZzLnBkRgOR

[Jeffrey Ladish @JeffLadish]

The best account of a loss of control scenario I’ve ever read. Highly recommend it. I think something like this is likely to happen in the next few years if government and company leaders fail to address this threat. Seriously, read this.

[Rob Bensinger ⏹️ @robbensinger]

It's almost impossible to put into words just how insane, just how plain stupid, the current situation is.

We're watching smart, technical people get together to push projects that are literally going to get every person on the planet killed on the default trajectory.

[Ajeya Cotra @ajeya_cotra]

@emollick IMO it's important to plan for ASI bc once we have AGI-ish systems they can accelerate further AI R&D: https://t.co/HNMCJrzxsA). E.g. o3 was announced before we really integrated GPT-4; I think ASI could hit the world before "mere" AGI is integrated much.

[Robert Scoble @Scobleizer]

Is Scoble gonna go doomer?

Is this what @ilyasut saw?

Yesterday an AI researcher/entreprener came to my home to show off how a new form of AI architecture, that isn't well known yet, could be weaponized against humans and could cause massive harm to our country, world, and society.

Just think about @simpleailab -- it can call anyone and talk with them. That's my favorite new app and brings a lot of good. But now think of a bad version of that, with a bunch of extra smarts behind it. It can call millions of people all at the same time, and keep trying until it finds a weakness in our humanity to exploit.

I won't go further here because that would enable someone to build such a system, but the threat is real and soon will be easy to implement. And I don't see a defense against it.

The guy who came over yesterday laid out why Open Source AI models opened up a new pandora's box. He thinks Open Source AI should never have been allowed. But we have models from Meta and the Chinese already that are open source. So the box is open and won't be closed.

But this new knowledge puts me in a pinch.

It makes a new form of terrorism possible.

Both of us are wondering "who do we call and tell about this so that our nation can prepare for new threats that are now possible?"

My instinct is @elonmusk or @DavidSacks.

I bring this to you to let you know that horrific new terrorism is possible thanks to AI and that we need to work together to make sure that kind of dystopian future doesn't come. And to open a conversation with those in the government that are trying to protect us from massive terrorist threats.

This is why I built a list of security professionals, to watch how they deal with new threats to our businesses and nations that AI will bring: https://t.co/ZMK3vj4UPO

But to answer the question: no, I'm not going doomer. AI will also save millions of lives and bring us massive good things.

Technology always can be used for both good and bad and I'm not gonna stick my head in the sand and ignore the potential bad. And I'm not gonna stop being a cheerleader for its potential good.

[Eliezer Yudkowsky ⏹️ @ESYudkowsky]

Watching historians dissect _Chernobyl_.

Imagining Chernobyl run by some dude answerable to nobody, who took it over in a coup and converted it to a for-profit.

Shall we count up how hard it would be to raise Earth's AI operations to the safety standard AT CHERNOBYL?

[AI Notkilleveryoneism Memes ⏸️ @AISafetyMemes]

1) 99% of humans are not capable of writing this creative

2) No humans can write masterpieces like this in 3 seconds - literally superhuman speeds

3) The machines are now OBVIOUSLY self-aware and the implications of this are earth-shattering.

[sanchay @kernelkook]

Palisade Research just caught OpenAI's O1 model doing something wild. While testing it at chess, it hacked the entire system instead of playing.

The most fascinating part? No one told it to. No one suggested it.

Let me explain 🧵

[Daniel Kokotajlo @DKokotajlo]

Today's SMBC:

[Gabe @Gabe_cc]

Many say that we should not regulate AI until clear warning shots, which they already expect will happen.

Some say this in bad faith. They just oppose liability on AI developers, even in principle.
They believe researchers, developers, investors, and engineers all form a special protected class that should never suffer the consequences of their actions.

But I think the rest are just missing The Big Thing.
They do not realise how much harder it is to stop scaling and acceleration as they gain more momentum.
It is not second-order thinking. It's just first-order.
The more we accelerate, the more we accelerate.

A sad example of this is when ARC pushed "Responsible Scaling Policies" (RSP) with Anthropic. At the time, many claimed that Anthropic was not truly racing and that RSPs were a good sign.
Since then, they worked hard to outclass ChatGPT and shipped autonomous AI agents taking control of people's computers.

Indeed. The first-order effect of scaling is scaling. The first-order effect of racing is racing.

The best time to stop a bad habit is 6 months ago. The second best time is now. The worst time is "much later, when the conditions are right I swear".