Having just spent the week in San Francisco, I feel like I've truly internalized the insanity of this situation for the first time. I think anyone who has seen me in the last few days can attest that I look like I've seen a ghost.
The leading AI companies are planning to massively accelerate AI research in 2027, which would knowingly endanger everyone on earth. This plan is unacceptable, and we must reject it.
Borrowing from one advocate I spoke to, even granting that it's a competitive, crazy situation, I do not believe any company has even truly tried to do what I think are the morally required duties of people taking such risks in extreme situations. These include:
(1) Trying hard to get out of the competitive situation;
(2) Seeking the informed consent of the public;
(3) Honest deliberation about the risks.
They have obviously not truly tried to solve the coordination problems that push towards this acceleration, either via finding legal ways to coordinate or advocating for effective policy.
I would be shocked to learn that getting off this insane runaway train is a top-few Q4 priority for Dario or Sam (let alone the other CEOs), which seems like the bare minimum. There are indeed some hard technical and policy problems in coordinating without policy intervention, but I simply do not believe that the companies are prioritizing this nearly as much as massively less important matters.
Meanwhile, the companies' track records on policy advocacy range from mediocre to disgraceful, and it is pretty clearly not the top priority of the policy team at any company to advocate for regulation that would enable the government or the companies to avert racing off the cliff.
They have obviously not truly tried to seek informed consent from the public. Whatever your KPIs might be for "informed consent from the public," these KPIs are abysmal. The public clearly does not expect 100 years of AI progress in 2027, and they would not accept it if they did. What they do know about AI development, they dislike and distrust.
I've heard arguments like "informing the public would mean informing China, and maybe they'd steal the weights." I'm sorry, but if you think you have model weights such that China stealing them would constitute a catastrophe on par with human extinction, and your plan is security by obscurity, that is a bad plan.
I also think the companies and their employees suffer from a significant failure of imagination of what kinds of coordination could be possible if the public were aware.
Finally, I don't think they're deliberating honestly, either. For one, I think it is insane to think that any company can truly evaluate its own ability to keep the process of recursive self-improvement under control. The biases towards "yes" are overwhelming.
It's a step forward that the companies have expressed or demonstrated openness to external auditors, but based on METR's reports so far, it seems likely that auditor reports will be heavily redacted (and that the auditors will not have sufficient time by default). The public and broader scientific community are not able to scrutinize their arguments for safety.
Given the strong default (in my view) that bringing a massive number of minds massively faster and smarter than humans would result in our disempowerment, and the serious danger that this would result in mass death, it is unacceptable to subject arguments for safety only to the scrutiny of biased, busy, and scarce company insiders and a handful of external evaluators.
My pleas for the companies follow from the above:
1) Actually fucking try to get the world out of this race;
2) Actually fucking try to alert and inform the public;
3) Actually fucking try to scrutinize your own arguments for the justifiability of this plan, which includes subjecting them to way more external scrutiny than anyone has currently done.
My pleas for policymakers: use your legal levers to get much more information about these plans into the public view. Make it the top priority -- not only of the next Congress, but of this Congress and the White House -- to understand the situation in its full strategic complexity, and work with technical and policy experts to advance regulation that gets us out of this bind. Domestic coordination must be the first priority, but we need to then quickly develop an international framework with our adversaries.
My pleas for the public: do not accept the current plan. Learn about this issue. Call or email your political leaders. Get involved with organizations like @DONOTACCEPTai, @HumansInControl, and @WhoDecidesAI.
And finally: believe in ourselves that we can stop this. It is never time to "panic" -- to make rash, unproductive decisions or to feel paralyzed by fear. It is time for us to feel the urgency and the stakes, to demand better, and to meet the moment. Let's do this. 🫡 ✊
AI companies admit what they’re building might kill everyone. We don't have to agree to this.
Add your name at https://t.co/Pcg0clMbfV to say: we do not accept your terms.
Lots of awesome stuff about the differences between emotional and intellectual recognition of x-risk in this classic @jkcarlsmith essay https://t.co/0wa3BYA3HX
Same.
One of my biggest regrets these days is that I didn't do this emotional processing much earlier. I remember an attendee at a workshop in 2022 leading a session on emotionally confronting x-risk and avoiding it because I didn't think it would be helpful, and I think that was a huge mistake.
I think for me it’s less that my beliefs changed a lot in the last few weeks and more that I started emotionally processing the reality of what I already believed intellectually.
It’s not that I’m suddenly very sure that we’re all going to die in two years. I’m not even close to sure of anything of the sort. But the preference cascade and the bad vibes have made me come to terms with my belief that the people I love might be in actual danger in the pretty near future.
Importantly it's still TBD what "California is moving to..." means. With the acceleration reported by METR two weeks ago, a few months of delay might make a huge difference. Governor Newsom, please make these happen before your term ends, even if this requires a special session!
California is moving to strengthen AI safety with these new recommendations:
🌐 Government-enforced baseline AI safety standards
🌐 Required AI shutdown protocols with emergency state access
🌐 Ability to limit use of AI recursive self-improvement
@Bananasaur_Rex@max_spero_@tracewoodgrains No, OpenAI did not articulate a plan for how they would avoid AGI taking over a lot of human agency. One OpenAI employee said a bunch of high-level principles that it would be good to follow. That's an extremely different thing.
Democrats, it’s time to stop listening to Ron Conway about AI policy.
He was wrong about supporting preemption of all state AI laws last year and he’s wrong now about Jensen Huang.
I understand this impulse, and everyone's family relationships are different. But at least for me I think it was important to push through the discomfort and tell my family why I'm so worried, for two reasons.
One is that we have to move past the era of the "AI priesthood" where a tiny number of people bears this great responsibility and cursed knowledge that everything is going to change and we all might die. That affects everybody, including our families. Everybody has the right to know about it; everybody should be empowered to do with this information what they will.
Maybe that means contacting their political leaders, joining a grassroots group, or switching jobs. Maybe it just changes their relationship to the present. I know it has for me. If the world could be unrecognizable in 2028, including potentially in catastrophic ways, it gives every moment of 2026 more meaning and makes me take everything about my life more seriously. If I withhold this from people, I think I'm denying them some ability to make the most of their life, and some dignity along the way.
In any case, though, I think that imposes some duty on the people who knew about this early to tell the people in our lives, as Step 1 of telling the rest of the world, both for deontological and consequentialist reasons.
The second reason, on a more personal level, is that truly close relationships require us to be known and understood.
My concerns that AI might change everything, including threatening my own life and the lives of my loved ones, are a very important fact about my life. My family would not know me if they didn't understand this. That would be an important way that I was not living a full life.
OPENAI ENGINEER DOESN'T WANT HIS OWN FAMILY TO KNOW WHAT'S COMING WITH AI
"I don't think that I have very comforting words for them"
"I'd rather they just not find out that much and continue to live very happy lives"
juan felipe cerón uribe has worked at openai since 2022. his job is to stop chatgpt from helping people make bioweapons
earlier in the same interview he described how the industry is racing: "let's see what it is and flip a coin and see if we're all going to die or we're going to create a lot of shareholder value"
asked about human extinction, he said it's "turning out to be really hard to rule out"
and he says people at openai talk about these risks less now than when he joined
is that love or avoidance?
@max_spero_@tracewoodgrains I wish I shared your confidence that this weren't OpenAI's plan! They really have not articulated how their plan differs from this IMO.
There are a gazillion examples from Elon's own life. Like, the "most interesting outcome" for the Zuckerberg cage match would probably involve, idk, it happening at all?
How does the claim "the most interesting outcome is the most likely" withstand 5 seconds of scrutiny, let alone become the refrain of one of the world's most powerful people?
It's so obvious that it's insane and almost embarrassing to say, but "the most interesting outcome" actually very rarely happens! That's why it's an interesting outcome!!!
Ban RSI rally outside Anthropic offices tomorrow at 11am! https://t.co/PY5tRa6Jex
Anthropic is shooting for “full handoff” to AIs automating AI research with no meaningful human oversight (ie. Recursive Self-Improvement or ‘RSI’) is unbelievably dangerous. Even CEOs themselves have said RSI is too dangerous to build:
“"Left unchecked, it (RSI) could outrun our ability to understand and control these systems."
- Dario Amodei, November 2026
“(RSI is) a very dangerous way to get to full AGI… you don't quite know how to keep control of such a system or such a process."
- Demis Hassabis, November 2026
“If you allow the model to independently self-improve… that would obviously be more dangerous… I would actually argue that we should not (do RSI), that would be one of the dangerous capabilities that we should step back from.”
- Mustafa Suleyman, CEO of Microsoft AI, 2024
"If there's a super intelligent (misaligned AI), particularly if it's engaged in recursive self improvement… then it will have a very bad effect.”
- Elon Musk, October 2014
@grace_huckins (Also for the record, I think the "default" scenario is like, the edge of plausibility for how fast things go even without slowdown, I think there will at least be *some* non-transfer/decorrelation from AECI to actual AI R&D capabilities.)
Definitely seems pretty risky, and agree on insufficient, but think it'd be among the most helpful single levers. Among other things it would probably delay the arrival of the superhuman AI researcher until there's more societal/political awareness of the problem, which I think is the main bottleneck to more ambitious stuff.
In a very simplified world where AECI translated directly to real-world capabilities including AI R&D, and every 25 points is a 2x speedup, I think this buys you about a year before you hit "speed doubles multiple times in a month," during which the capabilities are increasingly scary/obvious/causing real-world problems and benefits that make the problem increasingly salient.
In the default:
- 2024 to 2025-era models scored 125-150, with a slope of ~15/year; in Q4 2025, we hit 150, and the slope ~doubled
- Model 4 got ~178, so maybe now the slope will be ~60/year.
- This means a model would cross ~200 in ~February, making the slope ~120/year.
- This means a model would cross ~225 in ~May, making the slope ~240/year.
- This means a model would cross ~250 in ~June, making the slope ~480/year, and by July the sequence converges and it's a literal singularity (which is like, actually bounded by the laws of physics etc).
Whereas if you could only use models from 6 months ago, as of today:
- The slope would go back to 30/year until ~March (because models hit 175 in ~September), so the best model in March would be at ~195 (but they wouldn't be allowed to use this for AI R&D yet).
- It would then be at 60/year as the labs were allowed to use the 175+ models again. So the best model would hit 200 in ~April and 225 in ~September.
- This means they'd stay at 60/year until 6 months after April, so ~October. Then they could use the 200 model to hit 120/year, which would mean the 250 model arrives in ~December 2027, and 275 in ~March 2028.
- Then in March you're allowed to use the 225 model from September 2027, which hits 240/year, and then in June 2028 you're using the December model and doubling more than once a month. Things would be improving insanely fast but you never actually hit a literal singularity (though given physical limits it might not be that different).
But again the latter scenario is just way more time for the companies to improve their security and alignment, and more importantly for the rest of society/the world to be like, "holy shit, this plan is crazy and unacceptable."
I will be at the Ban RSI rally outside Anthropic offices tomorrow at 11 am. https://t.co/wjeJKyHok4
Recursive Self-Improvement is a stupidly dangerous idea. AIs are already getting smarter rapidly, and they often lie and cheat; why on earth do we think it's a good idea to have them build new smarter AIs to build newer smarter AIs faster and faster? Because if we don't do it, someone else will? I do not accept that answer. Anthropic (like OpenAI before them) styles themselves the good guys in the race, but IMO there are no good guys in this race.
I trust @goldman on these things. And from what I’ve read (haven’t made it through 85 pages yet sorry fam) I think this is a great framework. As a Californian and American, I’d like to see some action sooner rather than later!
The framework Newsom mentions in this clip has now arrived. I was hoping for strong and urgent calls to action to stop runaway recursive self-improvement.
I was not disappointed! It's a great report! Hope @GavinNewsom acts on it before he leaves office, given the stakes!
Donald Trump has completely FAILED on AI.
While he idles, California is stepping up with real safeguards, independent oversight, and accountability.
We aren't waiting for Washington to get its act together.