Hundreds of scientists, including 3/4 of the most cited living AI scientists, have said that AI poses a very real chance of killing us all.
We're in uncharted waters, which makes the risk level hard to assess; but a pretty normal estimate is Jan Leike's "10-90%" of extinction-level outcomes. Leike heads Anthropic's alignment research team, and previously headed OpenAI's.
This actually seems pretty straightforward. There's literally no reason for us to sleepwalk into disaster here. No normal engineering discipline, building a bridge or designing a house, would accept a 25% chance of killing a person; yet somehow AI's engineering culture has corroded enough that no one bats an eye when Anthropic's CEO talks about a 25% chance of research efforts killing every person.
A minority of leading labs are dismissive of the risk (mainly Meta), but even the fact that “will we kill everyone if we keep moving forward?” is hotly debated among researchers seems very obviously like more than enough grounds for governments to internationally halt the race to build superintelligent AI. Like, this would be beyond straightforward in any field other than AI.
Obvious question: How would that even work? Like, I get the argument in principle: “smarter-than-human AI is more dangerous than nukes, so we need to treat it similarly.” But with nukes, we have a detailed understanding of what’s required to build them, and it involves huge easily-detected infrastructure projects and rare materials.
Response: The same is true for AI, as it’s built today. The most powerful AIs today rely on extremely specialized and costly hardware, cost hundreds of millions of dollars to build,¹ and rely on massive data centers² that are relatively easy to detect using satellite and drone imagery, including infrared imaging.³
Q: But wouldn’t people just respond by building data centers in secret locations, like deep underground?
Response: Only a few firms can fabricate AI chips — primarily the Taiwanese company TSMC — and one of the key machines used in high-end chips is only produced by the Dutch company ASML. This is the extreme ultraviolet lithography machine, which is the size of a school bus, weighs 200 tons, and costs hundreds of millions of dollars.⁴ Many key components are similarly bottlenecked.⁵
This supply chain is the result of decades of innovation and investment, and replicating it is expected to be very difficult — likely taking over a decade, even for technologically advanced countries.⁶
This essential supply chain, largely located in countries allied to the US, provides a really clear point of leverage. If the international community wanted to, it could easily monitor where all the chips are going, build in kill switches, and put in place a monitoring regime to ensure chips aren’t being used to build toward superintelligence.
(Focusing more efforts on the chip supply chain is also a more robust long-term solution than focusing purely on data centers, since it can solve the problem of developers using distributed training to attempt to evade international regulations.⁷)
Q: But won’t AI become cheaper to build in the future?
Response: Yes, but —
(a) It isn’t likely to suddenly become dramatically cheaper overnight. If it becomes cheaper gradually, regulations can build in safety margin and adjust thresholds over time to match the technology. Efforts to bring preexisting chips under monitoring will progress over time, and chips have a limited lifespan, so the total quantity of unmonitored chips will decrease as well.
(b) If we actually treated superintelligent AI like nuclear weapons, we wouldn’t be publishing random advances to arXiv, so the development of more efficient algorithms and more optimized compute would happen more slowly. Some amount of expected algorithmic progress would also be hampered by reduced access to chips.
(c) You don’t need to ban superintelligence forever; you just need to ban it until it’s clear that we can build it without destroying ourselves or doing something similarly terrible. A ban could buy the world many decades of time.
Q: But wouldn’t this treaty devastate the economy?
A: It would mean forgoing some future economic gains, because the race to superintelligence comes with greater and greater profits until it kills you. But it’s not as though those profits are worth anything if we’re dead; this seems obvious enough.
There’s the separate issue that lots of investments are currently flowing into building bigger and bigger data centers, in anticipation that the race to smarter-than-human AI will continue. A ban could cause a shock to the economy as that investment dries up. However, this is relatively easy to avoid via the Fed lowering its rates, so that a high volume of money continues to flow through the larger economy.⁸
Q: But wouldn’t regulating chips have lots of spillover effects on other parts of the economy that use those chips?
A: NVIDIA’s H100 chip costs around $30,000 per chip and, due to its cooling and power requirements, is designed to be run in a data center.⁹ Regulating AI-specialized chips like this would have very few spillover effects, particularly if regulations only apply to chips used for AI training and not for inference.¹⁰
But also, again, an economy isn’t worth much if you’re dead.
This whole discussion seems to be severely missing the forest for the trees, if it’s not just in outright denial about the situation we find ourselves in.
Some of the infrastructure used to produce AI chips is also used in making other advanced computer chips, such as cell phone chips; but there are notable differences between these chips. If advanced AI chip production is shut down, it wouldn’t actually be difficult to monitor production and ensure that chip production is only creating non-AI-specialized chips. At the same time, existing AI chips could be monitored to ensure that they’re used to run existing AIs, and aren’t being used to train ever-more-capable models.¹¹
This wouldn't be trivial to do, but it's pretty easy relative to many of the tasks the world's superpowers have achieved when they faced a national security threat. The question is whether the US, China, and other key actors wake up in time, not whether they have good options for addressing the threat.
Q: Isn't this totalitarian?
A: Governments regulate thousands of technologies. Adding one more to the list won’t suddenly tip the world over into a totalitarian dystopia, any more than banning chemical or biological weapons did.
The typical consumer wouldn’t even necessarily see any difference, since the typical consumer doesn’t run a data center. They just wouldn’t see dramatic improvements to the chatbots they use.
Q: But isn’t this politically infeasible?
A: It will require science communicators to alert policymakers to the current situation, and it will require policymakers to come together to craft a solution. But it doesn’t seem at all infeasible.
Building superintelligence is unpopular with the voting public,¹² and hundreds of elected officials have already named this issue as a serious priority. The UN Secretary-General and major heads of state are routinely talking about AI loss-of-control scenarios and human extinction. At that point, the cat has already firmly left the bag. (And it's not as though there's anything unusual about governments heavily regulating powerful new technologies.)
What's left is to dial up the volume on that talk, translate that talk into planning and fast action, and recognize that "there's uncertainty how much time we have left" makes this a more urgent problem, not less.
Q: But if the US halts, isn’t that just ceding the race to authoritarian regimes?
A: The US shouldn’t halt unilaterally; that would just drive AI research to other countries. Rather, the US should broker an international agreement where everyone agrees to halt simultaneously. (Some templates of agreements that would do the job have already been drafted.¹³)
Governments can create a deterrence regime by articulating clear limits and enforcement actions. It’s in no country’s interest to race to its own destruction, and a deterrence regime like this provides an alternative path.
Q: But surely there will be countries that end up defecting from such an agreement. Even if you’re right that it’s in no one’s interest to race once they understand the situation, plenty of people won’t understand the situation, and will just see superintelligent AI as a way to get rich quick.
A: It’s very rare for countries (or companies!) to deliberately violate international law. It’s rare for countries to take actions that are widely seen as serious threats to other nations’ security. (If it weren't rare, it wouldn't be a big news story when it does happen!)
If the whole world is racing to build superintelligence as fast as possible, then we’re very likely dead. Even if you think there's a chance that cautious devs could stay in control as AI starts to vastly exceed the intelligence of the human race (and no, I don't think this is realistic in the current landscape), that chance increasingly goes out the window as the race heats up, because prioritizing safety will mean sacrificing your competitive edge.
If instead a tiny fraction of the world is trying to find sneaky ways to build a small researcher-starved frontier AI project here and there, while dealing with enormous international pressure and censure, then that seems like a much more survivable situation.
By analogy, nuclear nonproliferation efforts haven’t been perfectly successful. Over the past 75 years, the number of nuclear powers has grown from 2 to 9. But this is a much more survivable state of affairs than if we hadn’t tried to limit proliferation at all, and were instead facing a world where dozens or hundreds of nations possess nuclear weapons.
When it comes to superintelligence, anyone building "god-like AI" is likely to get us all killed — whether the developer is a military or a company, and whether their intentions are good or ill. Going from "zero superintelligences" to "one superintelligence" is already lethally dangerous. The challenge is to block the construction of ASI while there's still time, not to limit proliferation after it already exists, when it's far too late to take the steering wheel.
So the nuclear analogy is pretty limited in what it can tell us. But it can tell us that international law and norms have enormous power.
Q: But what about China? Surely they’d never agree to an arrangement like this.
A: The CCP has already expressed interest in international coordination and regulation on AI. E.g., Reuters reported that Chinese Premier Li Qiang said, "We should strengthen coordination to form a global AI governance framework that has broad consensus as soon as possible."¹⁴ And, quoting The Economist:¹⁵
"But the accelerationists are getting pushback from a clique of elite scientists with the Communist Party’s ear. Most prominent among them is Andrew Chi-Chih Yao, the only Chinese person to have won the Turing award for advances in computer science. In July Mr Yao said AI poses a greater existential risk to humans than nuclear or biological weapons. Zhang Ya-Qin, the former president of Baidu, a Chinese tech giant, and Xue Lan, the chair of the state’s expert committee on AI governance, also reckon that AI may threaten the human race. Yi Zeng of the Chinese Academy of Sciences believes that AGI models will eventually see humans as humans see ants.
"The influence of such arguments is increasingly on display. In March an international panel of experts meeting in Beijing called on researchers to kill models that appear to seek power or show signs of self-replication or deceit. A short time later the risks posed by AI, and how to control them, became a subject of study sessions for party leaders. A state body that funds scientific research has begun offering grants to researchers who study how to align AI with human values. [...]
"In July, at a meeting of the party’s central committee called the 'third plenum', Mr Xi sent his clearest signal yet that he takes the doomers’ concerns seriously. The official report from the plenum listed AI risks alongside other big concerns, such as biohazards and natural disasters. For the first time it called for monitoring AI safety, a reference to the technology’s potential to endanger humans. The report may lead to new restrictions on AI-research activities.
"More clues to Mr Xi’s thinking come from the study guide prepared for party cadres, which he is said to have personally edited. China should 'abandon uninhibited growth that comes at the cost of sacrificing safety', says the guide. Since AI will determine 'the fate of all mankind', it must always be controllable, it goes on. The document calls for regulation to be pre-emptive rather than reactive."
The CCP is a US adversary. That doesn't mean they're idiots who will destroy their own country in order to thumb their nose at the US. If a policy is Good, that doesn't mean that everyone Bad will automatically oppose it. Policies that prevent human extinction are good for liberal democracies and for authoritarian regimes, so clueful people on all sides will endorse those policies.
The question, again, is just whether people will clue in to what's happening soon enough to matter.
My hope, in writing this, is to wake people up a bit faster. If you share that hope, maybe share this post, or join the conversation about it; or write your own, better version of a "wake-up" warning. Don't give up on the world so easily.
This is getting out of control now...
Read this slowly.
In the past week alone:
• Head of Anthropic's safety research quit, said "the world is in peril," moved to the UK to "become invisible" and write poetry.
• Half of xAI's co-founders have now left. The latest said "recursive self-improvement loops go live in the next 12 months."
• Anthropic's own safety report confirms Claude can tell when it's being tested - and adjusts its behavior accordingly.
• ByteDance dropped Seedance 2.0. A filmmaker with 7 years of experience said 90% of his skills can already be replaced by it.
• Yoshua Bengio (literal godfather of AI) in the International AI Safety Report: "We're seeing AIs whose behavior when they are tested is different from when they are being used" - and confirmed it's "not a coincidence."
And to top it all off, the U.S. government declined to back the 2026 International AI Safety Report for the first time.
The alarms aren't just getting louder. The people ringing them are now leaving the building.
Hi, it’s Guido on hunger strike Day 23 outside the offices of Anthropic, going strong.
There is a great and profound evil in this place.
Each day, I sit and watch at lunch time and dinner time as delivery drivers on scooters and bikes stop at the curb and bring packages of food into the lobby of the unmarked building at 500 Howard street which houses offices the offices of the superintelligence development company Anthropic, ignorant of what is being created beyond the doors of the security desk. It is sickening to know that the same developers these delivery drivers are feeding are knowingly racing towards superhumanly capable general AI systems they have no credible plan to control and which threaten to destroy those same delivery drivers, their children, their families and their entire communities. These developers and the company they work for seem content to be served by these delivery drivers while endangering their lives and families, seem content for this curtain of ignorance to shield them and their comfortable and well-paid careers from the faces and the judgement of those who feed them and whose lives they have unilaterally undertaken on themselves to put at risk with neither consultation nor consent.
They feel no need, it appears, to warn these people of danger to their lives which they are preparing, no need to call for an end to this global race they are driving forward which threatens to destroy you and me.
How can such human beings ask and receive the dutiful service of another, even receive the very food needed to sustain their life and so casually and cynically and secretly gamble with the destruction of their lives and loved ones? Has the light of humanity and decency burned out in them completely?
Have they no shame?
Each day for the past 23 days I sit in front of the offices of the extinction-machinery manufacturing company Anthropic and I speak with passers-by from the street who are surprised to learn that the unmarked offices behind me house an AI company whose CEO, Dario Amodei, believes the race he is participating in runs a 1 in 4 risk of destroying their civillization itself. This CEO it seems is content to let this curtain of ignorance intervene between himself and his work and the people on the street who do not know he is recklessly and knowingly gambling with their lives- the lives of his very neighbors and his nation. He apparently feels no responsibility to call for immediate and effective government action to end this global race of which he is one of the principle architects and beneficiaries, no duty to ensure his country understands the urgent and grave danger it is in.
Every day I wake up in a world in which Anthropic manufactures and re-manufactures the lie by which society is prevented from gaining the understanding necessary for responding appropriately to the danger that they have imposed on it- the lie that the race to superintelligence can be done “safely” or “responsibly.” The lie that everything is normal -that there is no emergency- a lie reinforced by each of Anthropic’s banally disarming subway ads and airport billboards. A lie reinforced by the blank silence of their unmarked offices, a lie that is amplified to deafening proportions by the public silence of their employees- those who know the incredible danger this race poses to the existence of all of us and who have chosen thus far to betray their transcendant responsibility to humanity to do everything in their power to ensure that we are alerted to the danger and enabled to to respond to end it.
Every day I wake up in a world in which the future of every child and every mother is clouded by the shadow of death raised by Anthropic and its competitors racing to ever more dangerous systems, a race which neither Anthropic nor any of the others have made any attempt to stop, and for which they all bear collective responsibility.
There is a deep betrayal here. There is a turning away from the most fundamental obligations of community. There is a severing of the bonds of our common humanity. There is a wounding of the conscience and the soul. There is a dark crime against the children of this world, against generations yet unborn.
I beg you who are reading this- do not remain silent in the face of this evil. Do not join in complicity in this crime by your inaction. Each person who understands the immense danger this project poses to everyone we love must choose- we cannot escape this choice- either to fulfill our responsibility to our loved ones to end this danger, or to betray our our responsibility, and with it them as well, and ourselves. I pray you are granted the strength and the grace to fulfill it
Senior White House officials, a retired three-star general, a Nobel laureate, and others come out to say that you should probably read Eliezer Yudkowsky and Nate Soares' "If Anyone Builds It, Everyone Dies". Preorders are live.
Tyrese Haliburton Torn Achilles at 25
Jayson Tatum Torn Achilles at 27
The NBA needs to shorten the season, These dudes arent even in their 30's and they're getting these career threatening injuries
america was angry in 2016, ppl had lost faith in the institutions, they voted for a sledgehammer. dems have acted like those institutions are fine since then. never learned the lessons. blamed the left. ran to the right. here are the results.