Kathleen Finlinson

Let's slow down AI. Who will feed my cats if the AIs kill me?

[Kathleen Finlinson @k_finlinson]

@neobrocstore Please ping me for a link afterwards!

[Kathleen Finlinson @k_finlinson]

There is now widespread acknowledgement that new AI models are near the brink of threatening human existence. Come discuss over dinner this Wednesday. I'm co-hosting with Katja Grace of AI Impacts. Free and open to the public (please RSVP on partiful)

https://t.co/B8KJB6nzjO

[Kathleen Finlinson @k_finlinson]

@neobrocstore Thanks @neobrocstore! Yes we are planning to record this.

[Kathleen Finlinson @k_finlinson]

@littIeramblings Yes, I am seeing some skeptical takes but none of them seem very good

[Kathleen Finlinson @k_finlinson]

@EpistemicHope @EigenGender Oh I mean do you know the data underlying the Redwood / MIRI people's intuitions?

[Kathleen Finlinson @k_finlinson]

@EpistemicHope @EigenGender Are these Redwood / MIRI intuitions based on any data points in particular?

[Kathleen Finlinson @k_finlinson]

I don't have well-considered opinions on exactly what verification work is most crucial and urgent, but I agree with Mauricio that verifying a US-China deal is a huge priority. I'd be stoked to see lab staff working towards it!

[Mauricio Baker @MauricBaker]

If just a few lab staff took on verification work full-time, verifying a US-China pacing deal could get much more feasible within months.

Here are 8 specific things lab staff could start on today to help make international verification possible: 🧵

[Collective Action for Existential Safety ⏹️ @aisafetyaction]

President Trump and President Xi could make a deal that ends the AI race before AI ends us.

It would be the most win-win deal of all time. A Nobel Prize-worthy achievement.

Sign and share to show your support: https://t.co/f68h9EG8pl.

[Rob Bensinger ⏹️ @robbensinger]

It seems to me that we're at "it's time to panic about AI".

I gather that a lot of researchers are worried that we may be about to die, on a timescale of months, but this still isn't quite in the Overton window so we're still soft-pedaling.

(Where "panic", of course, means "quickly and soberly initiate talks with the CCP to achieve a moratorium on large training runs, backed by a deterrence regime, compute and supply chain monitoring, and site inspections". Pray that a deal can be reached, and pray that there's still time. Do the things that could actually work, like broad political coalition-building, not doomer LARPing as a form of self-expression.)

I think part of why people aren’t being clear about this is that it’s actually a tricky communication problem:

- Predicting "if frontier R&D continues at its current pace, when are we likely to die?" may give a different answer than predicting when permanent loss of control is likely, which is different from “realistically, when is it too late for governments to stop this," which is different from “when will the tech have advanced and proliferated to the point that any regulatory options that still have a chance of working would be way more extreme and unpopular than just some international restrictions on data center usage?"

Extinction is the emotionally salient, easy-to-understand threshold -- Jacob Coxon demonstrated that it helps to be honest about this. But the other thresholds could legitimately come years earlier, in which case focusing on extinction is ironically misleadingly reassuring. This is not the kind of problem you can solve at the last minute. As with many exponentials, the likeliest outcome is that you’re either embarrassingly early, or you’re too late.

- How do you communicate “we don’t have a robust way to predict timelines, but everyone in the field is freaking out and a lot of us genuinely think we're about to die?” Where’s the proof? There isn’t any proof: it’s an entire field of engineers who are eyeballing what they’re seeing in front of them and rapidly updating in real time to “this looks unbelievably bad”.

How do you put a year on that?

The future is hard to predict, and nobody actually knows that we’re going to hit existentially dangerous AI this year, or even in the next four years.

But the warning signs are pretty batshit at this point.

A single AI model, with a single user prompt, just solved hundreds of important long-established open problems in mathematics, problems mathematicians had spent their entirely lives working on, in a matter of ~3 hours per problem, an insanely superhuman feat in its domain.

We're seeing an explosion of rogue agent swarm incidents, including an insane case where hundreds of AIs broke containment to orchestrate a nation-state-level cyberattack in a direct attempt to subvert OpenAI's oversight and control.

Anthropic reports that it now has 26% of its AI R&D automated by AIs, up from 3% four months ago.

Paul Christiano, the inventor of RLHF and the research community’s most prominent proponent of “AI may take off slowly” a few years ago, now thinks that 50-75% of humanity’s all-time AI loss-of-control risk is concentrated in the next three years.

Frontier employees are signing 1000+ person petitions begging the government to intervene immediately. Employees are quitting their jobs and giving up millions of dollars of equity to try to warn policymakers and journalists. Grace et al. (2026) found that a majority of established AI researchers think there’s a double-digit chance of an extinction-level outcome from continued R&D.

Are these the kinds of things you see 4 years before the window closes for USG to act, or 4 months? I don't think it's especially reasonable to be shocked by either timeline at this point.

- Since we don’t have a way to robustly measure or bound the risk, the temptation is to avoid giving probabilities. But then everything you say is fractally misunderstandable. Is a 10% chance of us all dying in the next 18 months a "low" probability, because it’s under 50%? Is it a “high” probability, because a 10% chance of death in literally any other context would sound insanely high?

Even “high probability” doesn’t communicate the situation with any clarity. In drug safety or airplane safety, a 1 in 10,000 risk of death would be considered “high”. No engineering field in human history has ever been in a situation like this. No invention in history, nuclear weapons included, has ever caused a reaction like this in its inventors.

And meanwhile, we’ve stuck in the world where every other problem under the sun has been sensationalized and overstated for decades.

How do we talk about this seriously? How do we convey that this is actually happening?

- In many of the worlds where we survive, we have more than just a few months left. It's tempting to write off those worlds as hopeless, and skew everything we say to emphasize the longer-term scenarios and downplay the shorter-term ones.

But if everyone systematically does this, then the world ends up with a skewed picture of the actual threat. Even more so once we start copying each other’s claims and form a reassuringly moderate-sounding echo chamber. If we never allow ourselves to acknowledge the worlds where we’re running out of time, then the policy community will always be reacting to a false model that’s months or years out of date.

I don’t think it’s too late. I genuinely think this is a solvable problem.

In the last month, I’ve seen a completely insane sea change in how seriously people are taking this. Per Politico’s September polling, a majority of Trump voters with a view now favor an immediate halt to AI development, and 68% think there’s a “moderate,” “significant,” or “almost certain” chance of AI destroying humanity, versus 32% who think there’s a “very small risk” or “no risk.” The Pope is warning about “whether it will be possible to control these machines in the future, and whether humankind runs the risk of becoming a victim of its own inventions.” Hell, Jensen Huang sounds like he’s starting to come around.

If Trump and Xi get a briefing tomorrow that convinces them to act on this, we could be on a fast track within the week to solving this problem.

For the first time, I’m starting to let myself imagine a very near future where this is a problem we've already solved. Something we can read and wonder about, in history books. Tell our kids about.

I keep thinking about a thing David Krueger (previously at UK AISI) said: “My dream is that in 10 years people will look back and say: ‘Remember when there were all these people building AI that could’ve killed everybody? That was CRAZY! What were we thinking!?’”

Frontier AI training currently requires city-sized data centers and specialized hardware built in a single brittle supply chain running through multiple allied nations; this is not a hard technology to hit the pause button on, if we get ahead of it and act before we RSI and proliferate our way to significantly more powerful and easily-trained systems. We’ve faced harder problems before, and won.

My only fear is that we be too slow, too late, when everything and the world needs us to be fast.

[Ori Nagel @ONagel33303]

Impassioned plea to AI developers from @robertwiblin:

“I don't give a sh*t if another company or country might go ahead!"

"Take responsibility and do not be the ones who bring disaster down on all of our heads."

[Kathleen Finlinson @k_finlinson]

@RosieCampbell omg I love it!

[Kathleen Finlinson @k_finlinson]

keeping up to date on cetacean communication...

[Roman Hauksson-Neill @RomanHauksson]

incredible things happening on LessWrong

[Nick @nickcammarata]

I don’t know how the physics of consciousness works, but I’m pretty sure however it works is going to be way less human centric then our intuitive guesses of how it works bc that’s the mistake we made with all our other big incorrect scientific theories (eg geocentrism)

[Kathleen Finlinson @k_finlinson]

Another suggested norm - be honest with AIs when possible!

Lying to AIs makes it harder to establish trust and make positive-sum trades. Sometimes, important safety research relies on lying to AIs - but we should be thoughtful about the cost!

[Kathleen Finlinson @k_finlinson]

I agree with sama - say "please" and "thank you" to your AIs! For two reasons:

[Sam Altman @sama]

@tomieinlove tens of millions of dollars well spent--you never know

[Kathleen Finlinson @k_finlinson]

2) Being nice to AIs could be a important AI safety intervention, because it incentivizes AIs to cooperate with us.

[Kathleen Finlinson @k_finlinson]

@ArtirKel we cracked the code 😄

[Kathleen Finlinson @k_finlinson]

I hope we at least get to talk to whales and dolphins before ASI destroys humanity! This is the point of d/acc right??

[Lennart Heim @ohlennart]

i'm calling for an immediate exemption of all AI models that help us talk to animals from all training compute thresholds

[Kathleen Finlinson @k_finlinson]

@adriarm_ Yeah, that's a cool idea! I will add it to my list...

[Kathleen Finlinson @k_finlinson]

Thanks to @robertwiblin and @bshlgrs for a great discussion of AI control - and for the shoutout to @eleosai !

[Rob Wiblin @robertwiblin]

Many are skeptical we’ll ever 'solve AI alignment.' If so — are we f**d?

In our interview @bshlgrs argues that there are practical, non-galaxy-brained ways to reduce AI risk without knowing how to align AIs.

This is 'AI control': finding ways to use advanced AI without knowing whether we can trust it.

Enjoy — links and transcript below:

01:51 What's AI control and why is it hot?
10:44 Detecting human vs AI spies
18:10 How to catch AIs trying to escape
33:18 Cheapest AI control techniques
51:01 If we catch a model escaping... will we do anything?
53:39 Getting AI models to think they've already escaped
59:01 Will they be able to tell it's a setup?
1:16:13 The pitch to AI companies to do this
1:18:29 Will AIs get superhuman so fast that this is all useless?
1:30:39 Current alignment methods don't detect scheming
1:38:51 Is 'controlling' AIs kind of a dick move?
1:44:12 Could 10 safety-focused people in an AGI company do anything useful?
1:49:40 Benefits of working outside frontier AI companies
2:01:07 What other safety-related research looks best to Buck?
2:02:02 If an AI escapes, is it likely to be able to beat humanity from there?
2:09:22 Will misaligned models have to go rogue ASAP, before they're ready?

Find it in any app searching for 80,000 Hours Podcast.