Existential Risk Observatory ⏸

Reducing AI x-risk by informing the public. We propose a Conditional AI Safety Treaty:

[Existential Risk Observatory ⏸ @XRobservatory]

But how can it kill us all?

Together with @MITFutureTech and @FLI_org, our research found eight AI existential risk scenarios.

We will present them at our event at @cwclub SF 19 Nov with @geoffreyhinton, @DKokotajlo, @bshlgrs, and @m_bourgon.

RSVP now! https://t.co/ckTzwduaQg

[Existential Risk Observatory ⏸ @XRobservatory]

More on the treaty we proposed:
https://t.co/2Ccz1uBb21
https://t.co/UP47hbfmcC

More on enforcement and which sub-problems are the ones that may break it:
https://t.co/yj9Hzgt1M6

[Existential Risk Observatory ⏸ @XRobservatory]

Trump signals he may be willing to work with the Democrats to implement AI guardrails.

What obviously needs to happen, and what will happen, is what we and many others have said for years: an AI safety treaty (or, if that's easier, coordinated national regulation) between the US and China, where public institutes vet models for risks we can't take both during and after development.

This needs to be enforced via hardware regulation. One outcome of our research is that the hardest part of this is making sure we track and physically control all the chips that can train dangerous AI, globally. Regulators should waste no time making that happen.

In addition, AI capabilities research is currently lowering the threshold of how many chips are needed for building such dangerous AI. It is crucial that this research stops, as it is undermining regulatability.

The AI safety treaty will come anyway. But whether it is effective at saving us from extinction or other loss of control scenarios may well be determined by how soon and how effectively these measures are being taken.

[Reuters @Reuters]

Trump says AI risks can be managed, signals openness to guardrails https://t.co/Mjsn0ApURI

[Existential Risk Observatory ⏸ @XRobservatory]

@GabrielDunin Yes we're working hard to get this done. In addition, do join the waitlist as not all attendees are necessarily showing up and there may still be spots available.

[Existential Risk Observatory ⏸ @XRobservatory]

We were planning to promote our event "How can AI kill us all? Eight existential risk scenarios with Prof. Geoffrey Hinton". But it organically sold out 😲 The topic is clearly current!

If you're in AI safety or AI reporting and you really want to be there, reach out to us through X and we can see if we can fit you in.

We're very excited!

https://t.co/ckTzwdtD0I

[Congresswoman Yassamin Ansari @RepYassAnsari]

Top of the Democratic agenda on day 1 needs to be a pause on advanced AI development, regulation of this out of control industry, and a ban on artificial superintelligence.

[News from Science @NewsfromScience]

It’s happened again.

Just 3 weeks after OpenAI dumped hundreds of mathematical results onto the internet—a move that came 1 month after OpenAI’s solution to the famed Navier-Stokes problem—the San Francisco–based company is back with more.

The frontier artificial intelligence developer released solutions to three new problems spanning math, computer science, and physics.

Widely regarded as most consequential among them was an apparent refutation of the exponential time hypothesis, a pillar of theoretical computer science that has stood for almost 30 years.

Learn more: https://t.co/tESZ6kfwvq

[Charbel-Raphael @CRSegerie]

If you want to start doing AI safety advocacy, read this post by Josh.

He cold-emailed a Finnish MP, the MP asked the government a question, and Finland ended up co-launching an international coalition.

It’s not the whole story, but everyone who pushed the Finnish government had counterfactual impact.

[Existential Risk Observatory ⏸ @XRobservatory]

The impressive and original scientific results that keep being churned out should scare us much more than the Hugging Face attack.

That they are wildly underreported in the media keeps the public unnecessarily uninformed.

[Dan Roberts @danintheory]

We are announcing three new discoveries in mathematics, computer science, and physics about harmonic analysis, complexity theory, and quantum field theory.

https://t.co/9BX8HWQDTJ

[Existential Risk Observatory ⏸ @XRobservatory]

This is so good.

Leading voices of the societal debate like @tomfriedman have an actual shot at making people listen. And he's right: we rise together, or we fall together. We can learn that lesson early, or (too) late.

For ideas on what we should do: https://t.co/p6XSKbjF4C

[Thomas L. Friedman @tomfriedman]

We Just Set Off the Second Big Bang: https://t.co/2YYb2CnRiq

[Existential Risk Observatory ⏸ @XRobservatory]

https://t.co/2Ccz1uBb21
https://t.co/UP47hbfmcC

[Existential Risk Observatory ⏸ @XRobservatory]

This is obvious nonsense: frontier pace can and will be controlled by the US and China working together and implementing an AI Safety Treaty, as we and others proposed for years. It is wilful negligence that got us into this mess.

[Beff (e/acc) @beffjezos]

Yeah there's no going back.

The only way to secure systems is frontier labs out-accelerating open source.

Legislation won't stop black hat hackers.

Acceleration was always asymptotically the only option.

Security through multipartite adversarial non-equilibrium steady state.

[Cas (Stephen Casper) @StephenLCasper]

🚨New paper (possibly the most important one I have written yet).

We think that as early as 2027 or 2028, there might be a major AI-enabled terrorism event. We conducted a pre-mortem, made 9 falsifiable predictions, and wrote 2 concrete scenarios.

Led by @KamacheeMax.

[Existential Risk Observatory ⏸ @XRobservatory]

"Melting everyone else's GPUs" (Altman) has always been an integral part of solving xrisk via alignment.

Such a 'pivotal act' would be a blatant infringement on property rights and other nations' sovereignty and may escalate into war or durable suppression.

We need to go down a different path: get an AI Safety Treaty implemented now!

[Existential Risk Observatory ⏸ @XRobservatory]

Today, we propose the Conditional AI Safety Treaty in @TIME as a solution to AI's existential risks.

AI poses a risk of human extinction, but this problem is not unsolvable. The Conditional AI Safety Treaty is a global response to avoid losing control over AI.

How does it work?

[Existential Risk Observatory ⏸ @XRobservatory]

https://t.co/4th07BACGE

[Existential Risk Observatory ⏸ @XRobservatory]

"But how would AI kill us all?"

If recursive self-improvement leads to superintelligence (ASI), we believe AI can in fact take over power, and may quite possibly kill us all. In a new piece for the South China Morning Post SCMP, our @BartenOtto explains a number of ways:

"ASI could exchange data over the internet at the speed of light. It could spin up a vast amount of agents, creating AI armies where every soldier is superhumanly intelligent. It could run at superhuman speeds, making its relation to us perhaps comparable to our relation to plants.

"Unleashed rogue ASI could make money, either legally by trading on the stock market or illegally by extortion or hacking entire banks or cryptocurrency platforms. It could also act in the physical world by using any of the internet-connected devices that we have, including weapons, or manipulating humans.

"An ASI could plausibly hide itself on unguarded servers for a long time, overpowering us before we know it exists. Even more fundamental, a superintelligence might rapidly invent new science and technology. Once humans understood relativity theory, we were able to weaponise this understanding into atomic weapons. Which insights are still hidden to us but can be discovered and weaponised by a superintelligence, leaving us as exposed as a medieval castle to a nuclear bomb?

"Besides AI strengths, there are also profound sources of human weakness. There are more than 12,000 nuclear warheads. Fragile human bodies are easy prey for chemical attacks or viruses, as we saw during Covid. We have even created companies which can synthesise these viruses and, absurdly, respond to email orders. Water or food could be poisoned, and hacked online communication networks could start blocking anti-AI coordination messages while sending messages that serve the AI’s interests while we don’t realise what’s happening.

"If an AI needed us to maintain the electricity grid and server infrastructure, it could simply pay humans to do that, which many would accept because they need to pay the rent. Humans in powerful positions, for example in politics, could likewise be tempted to adopt pro-AI policies.

"As if all of this is not enough, we are handing AIs huge advantages on a plate. When a new model is released, it gets used by customers including the military, government, financial industry, utility companies, media and many other powerful institutions. All that is then needed for a takeover is collusion by these AIs against us.

"All these advantages and the fact humans are handing over total control to AI could well add up to the latter taking over power and, if it chooses, killing all humans."

[Existential Risk Observatory ⏸ @XRobservatory]

Time will tell whether these results are a crucial step towards intent-alignment, or overhyped incremental progress. In any case, the problem we face is *not* to create aligned AI. It is instead to make sure no one can create an unaligned, takeover-level AI.

How could creating an aligned AI, or solving the technical part of the alignment problem, contribute to solving this more difficult, larger problem?

- Historically, Yudkowsky presumed there would be a pivotal act, of which a toy example was to make the aligned AI "destroy all GPUs" or subtly alter all GPUs to make sure no one can build unaligned, takeover-level AI with them. It is however important to remember that the pivotal act would be illegal at home, extrajudicial abroad, would likely lead to pushback which would then need to be durably suppressed, that no one currently seems to be planning one, and that success is therefore at best uncertain.
- If our goal is to make sure GPUs cannot be used to create unaligned, takeover-level AI, why don't we simply put regulations in place to make sure this is achieved? Building aligned AI seems a strange, needlessly complex, needlessly failure-prone, and, combined with a pivotal act, illegal way to make this happen.
- If we need to regulate hardware anyway, what is the additional use of having aligned, takeover-level AI? It could be used to make useful products and make AI companies rich, but in a world where we need to regulate hardware anyway, AI alignment is unrelated to reducing existential risk.

Different researchers have voiced different existential risk scenarios, such as Paul Christiano's What failure looks like, Kulveit's Gradual disempowerment, and others. For some of these scenarios, there may be a positive offense-defense balance (defense wins), and in these cases solving alignment may actually reduce existential risk even without regulation. But in the canonical Yudkowsky/Bostrom risk scenario, which also seems the one researchers worry most about at the moment, offense-defense balance is probably negative (offense wins), meaning it would be very non-obvious how, if at all, progress in mechanistic interpretability reduces existential risk.

[Nick @nickcammarata]

interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had

with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works

https://t.co/zHij5zwKGT

[Existential Risk Observatory ⏸ @XRobservatory]

It is so important for leading AI authorities to keep informing the public about the existential risks we are all facing. Professors Hinton, Bengio, Russell, and many others are doing an amazing job.

[Geoffrey Hinton @geoffreyhinton]

Penguin has moved up the publication date of my new book with Patchen Barss called "Smarter Than Us". The book explains how AI works for a non-technical reader and then talks about the myriad risks and what to do about them.

[Congresswoman Yassamin Ansari @RepYassAnsari]

I’ve heard some Members of Congress say they “don’t know enough about AI to regulate it.” That’s absurd.

Just as we regulate airplanes, pharmaceutical drugs, cars and nuclear weapons, we need guardrails on advanced AI and we must ban artificial superintelligence - urgently.

Anything other than that is an abdication of our responsibility to the American people. And the consequences of continued inaction will be nothing short of catastrophic.

[Existential Risk Observatory ⏸ @XRobservatory]

https://t.co/ubbDHNPDtT

[Existential Risk Observatory ⏸ @XRobservatory]

There are so many strange and surprising ways in which a digital system can make a large impact on our physical world.