Marian Andrecki

ML, AI, sanity, London-based

[Rob Bensinger ⏹️ @robbensinger]

It seems to me that we're at "it's time to panic about AI".

I gather that a lot of researchers are worried that we may be about to die, on a timescale of months, but this still isn't quite in the Overton window so we're still soft-pedaling.

(Where "panic", of course, means "quickly and soberly initiate talks with the CCP to achieve a moratorium on large training runs, backed by a deterrence regime, compute and supply chain monitoring, and site inspections". Pray that a deal can be reached, and pray that there's still time. Do the things that could actually work, like broad political coalition-building, not doomer LARPing as a form of self-expression.)

I think part of why people aren’t being clear about this is that it’s actually a tricky communication problem:

- Predicting "if frontier R&D continues at its current pace, when are we likely to die?" may give a different answer than predicting when permanent loss of control is likely, which is different from “realistically, when is it too late for governments to stop this," which is different from “when will the tech have advanced and proliferated to the point that any regulatory options that still have a chance of working would be way more extreme and unpopular than just some international restrictions on data center usage?"

Extinction is the emotionally salient, easy-to-understand threshold -- Jacob Coxon demonstrated that it helps to be honest about this. But the other thresholds could legitimately come years earlier, in which case focusing on extinction is ironically misleadingly reassuring. This is not the kind of problem you can solve at the last minute. As with many exponentials, the likeliest outcome is that you’re either embarrassingly early, or you’re too late.

- How do you communicate “we don’t have a robust way to predict timelines, but everyone in the field is freaking out and a lot of us genuinely think we're about to die?” Where’s the proof? There isn’t any proof: it’s an entire field of engineers who are eyeballing what they’re seeing in front of them and rapidly updating in real time to “this looks unbelievably bad”.

How do you put a year on that?

The future is hard to predict, and nobody actually knows that we’re going to hit existentially dangerous AI this year, or even in the next four years.

But the warning signs are pretty batshit at this point.

A single AI model, with a single user prompt, just solved hundreds of important long-established open problems in mathematics, problems mathematicians had spent their entirely lives working on, in a matter of ~3 hours per problem, an insanely superhuman feat in its domain.

We're seeing an explosion of rogue agent swarm incidents, including an insane case where hundreds of AIs broke containment to orchestrate a nation-state-level cyberattack in a direct attempt to subvert OpenAI's oversight and control.

Anthropic reports that it now has 26% of its AI R&D automated by AIs, up from 3% four months ago.

Paul Christiano, the inventor of RLHF and the research community’s most prominent proponent of “AI may take off slowly” a few years ago, now thinks that 50-75% of humanity’s all-time AI loss-of-control risk is concentrated in the next three years.

Frontier employees are signing 1000+ person petitions begging the government to intervene immediately. Employees are quitting their jobs and giving up millions of dollars of equity to try to warn policymakers and journalists. Grace et al. (2026) found that a majority of established AI researchers think there’s a double-digit chance of an extinction-level outcome from continued R&D.

Are these the kinds of things you see 4 years before the window closes for USG to act, or 4 months? I don't think it's especially reasonable to be shocked by either timeline at this point.

- Since we don’t have a way to robustly measure or bound the risk, the temptation is to avoid giving probabilities. But then everything you say is fractally misunderstandable. Is a 10% chance of us all dying in the next 18 months a "low" probability, because it’s under 50%? Is it a “high” probability, because a 10% chance of death in literally any other context would sound insanely high?

Even “high probability” doesn’t communicate the situation with any clarity. In drug safety or airplane safety, a 1 in 10,000 risk of death would be considered “high”. No engineering field in human history has ever been in a situation like this. No invention in history, nuclear weapons included, has ever caused a reaction like this in its inventors.

And meanwhile, we’ve stuck in the world where every other problem under the sun has been sensationalized and overstated for decades.

How do we talk about this seriously? How do we convey that this is actually happening?

- In many of the worlds where we survive, we have more than just a few months left. It's tempting to write off those worlds as hopeless, and skew everything we say to emphasize the longer-term scenarios and downplay the shorter-term ones.

But if everyone systematically does this, then the world ends up with a skewed picture of the actual threat. Even more so once we start copying each other’s claims and form a reassuringly moderate-sounding echo chamber. If we never allow ourselves to acknowledge the worlds where we’re running out of time, then the policy community will always be reacting to a false model that’s months or years out of date.

I don’t think it’s too late. I genuinely think this is a solvable problem.

In the last month, I’ve seen a completely insane sea change in how seriously people are taking this. Per Politico’s September polling, a majority of Trump voters with a view now favor an immediate halt to AI development, and 68% think there’s a “moderate,” “significant,” or “almost certain” chance of AI destroying humanity, versus 32% who think there’s a “very small risk” or “no risk.” The Pope is warning about “whether it will be possible to control these machines in the future, and whether humankind runs the risk of becoming a victim of its own inventions.” Hell, Jensen Huang sounds like he’s starting to come around.

If Trump and Xi get a briefing tomorrow that convinces them to act on this, we could be on a fast track within the week to solving this problem.

For the first time, I’m starting to let myself imagine a very near future where this is a problem we've already solved. Something we can read and wonder about, in history books. Tell our kids about.

I keep thinking about a thing David Krueger (previously at UK AISI) said: “My dream is that in 10 years people will look back and say: ‘Remember when there were all these people building AI that could’ve killed everybody? That was CRAZY! What were we thinking!?’”

Frontier AI training currently requires city-sized data centers and specialized hardware built in a single brittle supply chain running through multiple allied nations; this is not a hard technology to hit the pause button on, if we get ahead of it and act before we RSI and proliferate our way to significantly more powerful and easily-trained systems. We’ve faced harder problems before, and won.

My only fear is that we be too slow, too late, when everything and the world needs us to be fast.

[Alan Cole @AlanMCole]

Scott Alexander definitely shows Pinker has been uncareful and flippant. These are common vices even among great thinkers, but always unfortunate.

But my object-level takeaway is Scott doesn't have a complete substantive response to the better points.
https://t.co/UUjuoKp1lc

[Marian Andrecki @MarianAndrecki]

@CineraVerinia Such a world is probably more competent than ours. But worlds where you can easily steeply sacrifice the thing that you can enjoy for the thing that improves win-rate at zero sum games ends up in a non-enjoyable place.

Granted there may be some positive-sumness here.

[Marian Andrecki @MarianAndrecki]

@CineraVerinia Well, luckily nobody is allowed this trade. That would be one ugly race to the bottom.

[Danijar Hafner @danijarh]

Excited to announce DreamerV3 🌍, a scalable and general RL algorithm that masters a wide range of applications with fixed hyperparameters!

Applied out of the box, it solves the Minecraft Diamond challenge without human data. 💎

👇 Thread
https://t.co/zTAy1HZvIW

[Linden Li @lindensli]

@abhi_venigalla and I turned @karpathy’s minGPT into a GPT-3 quality model with 30 billion parameters—projected to cost only $450k to train. The code to do so is public: it's easily readable and can be launched on however many GPUs you want.

Here’s how:

[Dumitru Erhan @dumierhan]

1/ Today we are excited to introduce Phenaki: https://t.co/7xkcoeuXwB, short-link-to-paper, a model for generating videos from text, with prompts that can change over time, and that is able to generate videos that can be as long as multiple minutes!

[Yi Zhe Ang @yizhe_ang]

Happy to share my new interactive article / explorable on K-Means Clustering, which has also been accepted to @VISxAI at @ieeevis!

https://t.co/0nLinxILrh

Built with #svelte and #d3js.

#MachineLearning #DataScience #Datavisualization #dataviz #scicomm

[Andrey Zagoruiko @andreyzagoruiko]

accurate

[Marian Andrecki @MarianAndrecki]

@NathanpmYoung @StefanFSchubert I've been pretty happy with modafinil - not transformative changes but zero side effects (except for sleeping difficulties if you take late in the day)

[Marian Andrecki @MarianAndrecki]

@robinhanson Maybe it would help if you clarify "making sense".
Are they not realistic enough? Are the wrong parts unrealistic (eg character's psychology)? Is the message not clear?

[Marian Andrecki @MarianAndrecki]

@robinhanson Horror genre has a lot of breadth and is catch all for scary&fantastical stories. Some (eg "mother!") are meant to induce paranoia, surrealism - bound to be unrealistic & "not making sense". There are many that aim for (a degree of) realism, eg "Alien".

[Neal Agarwal @nealagarwal]

I made a page that gives you increasingly absurd trolley problems :)

https://t.co/MwfoNTv4Tm

[Robin Hanson @robinhanson]

And we are number 50 out of 100. :)

[Andrej Karpathy @karpathy]

The ongoing consolidation in AI is incredible. Thread: ➡️ When I started ~decade ago vision, speech, natural language, reinforcement learning, etc. were completely separate; You couldn't read papers across areas - the approaches were completely different, often not even ML based.

[Christian Gumbsch @cgumbsch]

Most Recurrent Neural Networks (RNNs) update their latent state in every time step. Is this necessary?
In our #NeurIPS2021 paper we present GateL0RD, an RNN that sparsely updates its memory. And this has many benefits...
👥 with @mvbutz & @GMartius
📜https://t.co/FwI1VHvLY5
1/🧵

[Thomas Kipf @tkipf]

Excited to share our work on Conditional Object-Centric Learning from Video!

We introduce SAVi, a slot-based model that can discover + represent visual entities in videos, using simple location cues and object motion (...or entirely unsupervised)

🖥️ https://t.co/kzj5pwcDNd

1/7

[David Krueger 🦥 ⏸️ ⏹️ ⏪ @DavidSKrueger]

I am looking for 1-2 PhD students to join my research group at Cambridge (CBL) fall 2022.
Our focus is on Deep Learning, AI Alignment, and existential safety.
Deadline for university funding is December 2.

[Marian Andrecki @MarianAndrecki]

@StuartJRitchie Nice! Preordered.

[hardmaru @hardmaru]

The agent first learns a world model driven only by exploration, without task-specific rewards.

It then receives rewards to adapt to multiple tasks, such as standing, walking, running, flipping using either zero or only a few tasks-specific interactions.

https://t.co/5ccxqdYxUB