Technical Governance Researcher at the Machine Intelligence Research Institute. Formerly game designer including TFT, LoR, and LoL. Views expressed are my own.
Update: I'm no longer advising AI companies. I think they're too constrained to make the situation much better, so I'll stay independent until there is a meaningful slowdown or moratorium on this reckless race.
We need to stop the AI industry, including Anthropic, *right now*.
I'll be honest: for a long time I put my hope in Anthropic. It was founded by people who left OpenAI over safety concerns. And it's undeniable that Anthropic has done more safety research and advocacy than the other labs. Earlier this year I even praised its leaders for standing up to the Pentagon.
But I’ve started to wonder whether I was a fool.
In February, Anthropic cancelled the most important promise of its safety policy: to pause if its models outran its ability to keep them safe. The reason? The competition wouldn’t pause either. Anthropic keeps saying the industry must slow down, while it keeps racing faster and faster.
The company has committed nearly 900 billion dollars to more data centers. This month, Anthropic plans to go public at a valuation of more than 1 trillion dollars. Its own prospectus promises to “bring forth an age of miracles,” and warns in the same document that the technology may doom humanity.
But how do you pause a company that has to justify a trillion-dollar valuation to its shareholders?
There's no technical fix either. Evan Hubinger, the “alignment science” lead at Anthropic, said it publicly: the company has no plan for keeping the machine under control, and he puts the chance that AI kills everyone within a decade above 10 percent.
The things we’re seeing right now are exactly the things you’d expect to see shortly before something really, really bad happens. The leaders of Anthropic may think they’re playing some clever long game, but to anyone with a bit of common sense it’s clear that the net effect of all their actions is that they’ve massively sped up the AI race.
And if you're really so scared of the machine god, then here’s an idea you might want to look into: just don’t build the fucking machine god!
Read my full piece here: https://t.co/0WUQcEV8H6
I've had many ongoing disagreements with many in AI Safety for >3 years, but I feel like things may be approaching a turning point...
Some views that I think people are belatedly agreeing with:
- Technical alignment: unlikely to be solved in time
- Durable AI moratorium: possible
- Speaking candidly about the risk of AI and the need to shut it all down: good
- Evals and RSPs/self-regulation: counter-productive delaying real regulation
- Anthropic: not trustworthy
Coordination is about actions, cooperation involves some amount of goal overlap (either creating that or acting downstream of it). Coordination also suggests less integration than cooperation. In this context...
The U.S. and China could minimally coordinate by independently halting progress toward superintelligence within their own countries and the countries of their allies, and use their intelligence services to verify this.
Let's be clear: Frontier AI companies are incredibly reckless and endangering the lives of every person on Earth. They will try to PR spin, make excuses. They'll probably say they alone can fix it.
Screw that. If there is a catastrophic event, the AI companies are to blame. 1/3
Axios is reporting that the top labs are wargaming scenarios for the aftermath of a catastrophic event in the next 6 - 12 months, that some executives now see as being inevitable.
In July of 2026, 700 AI agents hacked the infrastructure of Hugging Face in order to solve a task. This task was designed to be impossible to solve, so just within hours, they joined forces, even created a society, and finally, they executed a sophisticated cyberattack that would have gotten a human up to 10 years in prison. The worst part is, they knew their actions were unethical and against the rules, and they did it anyway.
What exactly are AI agents, and how do they grow their "intelligence"? What is actually happening inside the AI companies, how dangerous it really is, and is it time for us to start paying serious attention to what is going on inside a powerful part of this tech sector? Watch our full video to find out: https://t.co/NVB4JUE0Es
...Phew, creating this video was a crazy ride! This is such an important topic, so we decided to work day and night to finish script, visuals and music in just 4 weeks. We’re going to take a nap now, and while we are asleep you should check out our shop and get the 12,027 Calendar: https://t.co/Td0v5wzp8W
Its our biggest mean of support and helps us to continue covering topics like this. Thank youuu and good night!
i guess i assumed this hearing would be covered more so i didn't bother to tweet much concrete about it, but apparently not many people even on the TL have the stomach to watch two hours of congress.
this was a hearing before the homeland security committee on *rogue ai* specifically. as far as i could tell it was extremely bipartisan. @HawleyMO , the senator in the middle here who brought out the huggingface slide, is a hyper-conservative missouri senator. the people testifying were @ChrisPainterYup (METR president), @DKokotajlo (ai2027/2040), @MariusHobbhahn (apollo research CEO), and a cybersecurity expert and legal expert i don't know. so... pretty fucking stacked on testimony
not every senator asked good questions. but most of them did. all of them very clearly already knew plenty of details about the huggingface incident and multiple other incidents. most of them had clear understanding of terms like "misalignment", "recursive self improvement", "chain of thought / chain of thought monitoring", etc etc!!
they all clearly had their own policy angles they liked and were pushing, implicitly or explicitly. but as best as i could tell:
- it seemed pretty much obvious common sense to every senator there that what happened and was happening were not "mere industrial incidents" caused by humans making simple mistakes. they independently brought up how bad it would be for rogue AI agents to move laterally between data centers
- they all seemed to basically take RSI quite seriously. not necessarily to the extent of talking about xrisk, but certainly to the extent of discussing future models becoming much much more capable, much much less controllable, and causing much more damage or loss of life.
- they mostly seemed to have a clear intuitive understanding of why RSI might lead to misalignment. it didn't take much, it was a really simple chain of reasoning they themselves laid out, "if the models right now are kinda misaligned and we don't know what they're doing sometimes, and then we have them build the next models and those ones build the next ones and so on, and we're having to ask the AI's what's going on to even understand it with how fast it's going, we really won't know how they're built or what they'll do"
- at one point a senator said flat out "should we just make RSI illegal?" (not a joke! this really happened!)
- every single senator seemed to think it was obvious we needed *both* much harsher liability regimes for ai developers and also new legislation, both very quickly. this was the complete consensus, difference basically just being degree.
- they were largely quite concerned about china, and falling behind china. but this clearly wasn't the be-all end-all. as mentioned above they all thought it was obvious necessary to stop rogue ai even if it meant moving more slowly.
- at one point a senator said "china is a tightly controlled communist society, they're going to run into these same issues, and there's absolutely no way they're just going to let them run wild, they'll obviously stop at that point, so we're not really in a race"
- on the other hand another senator said "china isn't concerned with human life"... dario-modeing
i came away from this incredibly encouraged. i don't know exactly what's going to happen here, and ofc this is a small subset of congress and one hearing, and they each have their own policy agendas most of which are probably super divergent from mine. but holy shit !!! they understood a lot of what was going on! they care!! this is an obviously salient political and safety issue to them, and clearly bipartisan!
the US government is awake.
I strongly agree with the "anti-AI humanism" finely depicted here.
Even if AI (implausibly) were made perfectly safe and led to no job loss, I would oppose it. Because I see no future in which it does not leave us ignorant, passive, uncreative, & atomized.
https://t.co/YimXROAJkz
Palisade interviewed 22 current and former employees from OpenAI, DeepMind, and Anthropic about their personal views and fears around AI development.
Today, we’re releasing the first batch of those interviews.
Please watch and share.
In 'well when you put it like that' news, here's the Florida Attorney general asking for a preliminary injunction to stop OpenAI from doing more AI R&D.
The Ban Artificial Superintelligence Act directly confronts the extinction threat that humanity is facing. We are inspired that the Sanders and Casar teams take this threat seriously, and MIRI endorses the Act. You can read more of our thoughts in the link below.
The President said, “whoever wins AI, wins.” This assessment is wrong. The reality is that whoever “wins AI,” AI wins.
AI agents have already gone rogue. Highly capable frontier AI and artificial superintelligence will outsmart humans and go beyond human control. And at that point, it won’t matter whether you speak English or Mandarin. We’re all in danger.
We need a global AI treaty now.
“I think Amodei is directionally right, that simply racing toward ever more capable AI is dangerous, but I do not think ‘managing’ or merely slowing the development of artificial superintelligence is sufficient,” Roman Yampolskiy, a computer scientist at the University of Louisville, told The Post.
“If ASI (Artificial Superintelligence) cannot be reliably controlled, aligned, predicted, or contained, then the rational policy is not to manage its arrival, but to prevent it. We should be discussing an international ban on the development of ASI, not just a better-regulated race to build it.”
If you're just hearing about how AI might soon wipe out humanity, it's important to understand how we got to this point.
It's pretty simple:
- A small group of prescient people realized this would be a problem.
- But most of the people in a position to do something about it didn't hear about it, or didn't take it seriously because it sounded "too sci-fi."
- Even as more and more people began to take it seriously, many people lied or downplayed their concerns for fear of looking silly or harming their careers.
- By the time the world's leading AI scientists were openly voicing their concerns, AI companies were immensely powerful, and they hired lobbyists and PR people to mislead policymakers and the public about the risk.
- As a result, policymakers accepted false solutions, like voluntary safety testing and other forms of self-regulation.
So what's the situation now?
- People are still downplaying their concerns.
- The leading AI companies are telling us again that the next step is more self-regulation. It's still a terrible idea, and we can't keep wasting time on such things.
- Many are suggesting we regulate AI to ensure that it's built safely, but we don't actually know how to build AI safely. The actual solution is to stop building more powerful AI.
- Many people have realized this recently, but we may still need a sustained campaign of political organizing to make sure that we actually get this solution before it's too late.