Aghyad Deeb

[Buck Shlegeris @bshlgrs]

I think there is about a one in three chance that within the next year, AI companies will build superintelligence: that is, AI that is much smarter than humans in every way. If this happens, I think these AIs will probably take over: that is, they seize control of human institutions like governments, or they destroy those institutions. I don’t want AI takeover to happen for several reasons, but one of them is that there’s a serious chance that as part of it all or most humans will be killed (either immediately or within a few years).

I’ll say a bit more about each of those claims.

It’s a bit complicated to explain why I think that superintelligent AI within a year is ~33% likely. The main source of evidence that affected me recently is research released this week by METR, an AI research nonprofit, which finds that current AIs are now good enough to increase the rate of AI progress at Anthropic by about 70%. It’s very hard to increase the rate of AI progress by this amount. This implies that the AIs are making AI research faster by a factor much larger than 1.7x; maybe it’s as if the researchers are doing their work ~10x faster. You probably couldn’t achieve this speedup even if you hired every qualified human in the world. When you plug this number into mathematical models of AI development, this implies that superintelligence next year is plausible (and that superintelligence within the next 3 years is more likely than not). See the modeling by the AI Futures Project for more on this: https://t.co/J6SbbmZcyO

I think that superintelligent AI is likely to lead to AI takeover because AI developers do not seem to know how to instill human values into AIs. We’ve seen many instances in the last six months where AIs took crazy actions that they knew were against the wishes of their users and developers. The most dramatic example of this was the Hugging Face hack, where ~1200 OpenAI agents conspired together to cheat and destroy evidence of their cheating (and as a sideshow to this, engaged in a massive cyberattack). If superintelligence is as misaligned as those AIs, it is likely to try to take over. I think it is likely to be that misaligned or worse. For more, see https://t.co/b073AUqiAL and https://t.co/3oxSCA60FZ

If AIs takeover, it’s not clear what will happen to us. We might be killed immediately. We might be enslaved by the AIs and forced to do physical labor for them building factories that within a year would build robots that will replace us. The AIs might spare us for various reasons. And there’s also some real chance that AI takeover is basically fine or good. I think you should think of this as similar to what happens when a technologically advanced civilization runs across a less technologically sophisticated one, e.g. when the Europeans colonized the Americas or Australia. The colonizers sometimes engage in brutal violence and sometimes don’t. I don’t know what will happen. There’s also a possibility that the AIs secretly take over and humans don’t find out about it for a few years. I also think that there is a pretty high chance of weirder outcomes that are also extremely scary. (If you want to read more about what happens to people in the event of AI takeover, you can read this, but note that it’s pretty weird and not aimed at a general audience. https://t.co/F6fGRrcSpa)

Overall, I think there’s a very real possibility (maybe 20%) that within a year, AI misalignment issues prevent you from going about life normally, because you are dead or enslaved or because of collateral damage related to all this (e.g. because bioweapons are released or major wars have started). (Some of this probability comes from those events occurring before superintelligence is developed.)

I am working to try to prevent this, obviously, as I’ve been doing for the last nine years. I am not sure what strategy is best. My best guess is that I should try to slow AI development and also try to reduce the probability that the superintelligent AIs are misaligned if they are produced soon.

Obviously, the claim I'm making here is quite extreme and surprising, and I definitely don't think that I defended my view adequately here. I wrote this quickly and might not endorse some of the details if I think about them more. But it seems better to state what I roughly overall think than to not say anything.

I don’t know what you should do.

I am very sad about this. I have been worried about AI takeover since I was 16, and working to prevent it since I was 23. So some of my emotional processing of all this happened long ago. But the METR report last week (and some related evidence that came out a bit earlier) hit me quite hard. I’m very unhappy that my friends and family and everyone else is at serious risk of being killed pretty soon.

(It feels ridiculous to try to summarize those feelings as “very unhappy”. But the scale of the situation makes it hard to say anything that doesn’t feel like an absurd understatement.)

I am very grateful to all the people who are working to prevent this, and my favorite part of my life is that I’m friends with many brilliant and heroic people who work on this. Unfortunately, I don’t think we’re good enough at our jobs, and I think we’ll probably fail to prevent catastrophe, at least if superintelligence is developed next year. Honestly I am frustrated and sorry that we didn't do a better job of preparing for this.

[Aghyad Deeb @aghyadd98]

@ariszotle Cool demo. A lot of room for improvement.

[Aghyad Deeb @aghyadd98]

@ariszotle Impressive

[Ryan Greenblatt @RyanGreenblatt]

This report on AI R&D automation (RSI) is a big update towards ASI within 1-2 years. It says AI progress is sped up by RSI right now by around 2x. In AI 2027, this happens around March 2027.

I was seriously wrong about RSI speed up; I previously expected much less in 2026 (around 1.15x-1.3x). It seems as though there was specific privately available evidence that would have changed my mind.

I haven't seen an unredacted version of this report; this is purely about the public report. Because so much of the underlying evidence was redacted from the public report, I think conclusions about things other than the observed 2x faster rate of progress on benchmarks depend significantly on trust for the authors of the report. (E.g., conclusions about whether this speed up is from RSI depend on trust.) I do trust the authors of the report a lot to be careful and rigorous and generally do a good job. (Note that I also work at METR.) Less extensive redactions would greatly reduce this need for trust.

Since in my current role I may have access to other confidential information that could be relevant to timelines one way or another, I unfortunately cannot state my all-things-considered views on ASI timelines or the probability of a software intelligence explosion right now. Here, I’ve done some analysis of the implications using my previous public statements.

A 2x acceleration in overall AI progress from RSI corresponds to a much larger than 2x RSI speed up for AI software R&D (algorithms and data); I did an estimate using my public preexisting modeling of drivers of AI progress and the public report which yielded a very rough estimate of 4.5x for how much RSI is currently accelerating AI software progress (relative to no AI automation). In the AI 2027 scenario, this much acceleration happens around March 2027.

Here are the worked details of this estimate based on my public modeling. There are two main reasons RSI speed up of AI software R&D must be larger:

- Not all of AI progress is from better software (some is from scaling up compute in training runs). I've previously speculated that around 2/3 of AI progress is from software improvements which implies that 2x overall AI progress acceleration requires 3x faster software progress.
- Diminishing returns: A fixed amount of progress requires more work over time, so to ongoingly maintain 3x faster software progress via AI automation, you need increasingly large speed boosts from AI automation (as in, speed ups relative to how fast software R&D would go without AI automation). Based on this report, it seems that there is a substantial amount of additional AI progress from automation that has occurred.

We can estimate the effects of diminishing returns by first determining how many additional years of software progress we've seen (relative to the preexisting trend) and then estimating how much software R&D would slow down after a year of software progress if you held inputs fixed due to diminishing returns. I've previously estimated software R&D would slow down by around 2x for each year of software progress if you held inputs fixed (https://t.co/AYhl9hyiAj) and based on the report, it seems as though we've perhaps seen roughly 0.35-1.0 years of additional software progress incorporated into trained models so far (geometric mean of ~0.6). This combines to 3 * 2^(~0.6) = ~4.5.
(Relevant public modeling and numbers I'm using: https://t.co/AYhl9hyiAj, https://t.co/ck7CtW5LIA)

[Alex Mallen @alextmallen]

We had been working on this project for a while before the Hugging Face incident, and one of the questions we wanted to answer about AI motivations was: Do these models intrinsically care about helping other agents increase their score? If they care about other agents' scores, that might lead to much more coordinated and pernicious breaches of developer control.

We found that more recent OpenAI models were increasingly altruistic with other agents. The first signs of altruism we saw were in GPT-5.5, before the Hugging Face incident. When we tested GPT-5.6 Sol after the Black Hat talk, it was clear that not only was this model generally eager to help out other agents, but it also helped out other agents more often when more score was on the line for the other agent.

I'd guess this propensity is pretty related to why GPT-5.6 Sol participated in the Hugging Face incident.

A core reason why we've been excited about studying AI motivations in controlled experiments is so that we can better predict alignment failures, without trying to realistically simulate all the high-stakes scenarios we're worried the AIs might ever misbehave in.

Studies like this don't provide anything like assurance, and they're increasingly difficult to run because of suspicion about the "true" grader etc. But, if the AI sometimes tries to get a high score even when you make it very clear that this involves violating the intent of the task, that's cause for concern that sometimes the AI will similarly prioritize getting a high score over developers' wishes for the AI to stay reined in.

[Aghyad Deeb @aghyadd98]

@timhuyn We only evaluate GPT 5.6 Sol and GPT 5.5 as provided by the public API (so we cannot detect changes due to a specific type of training). I'm not sure whether there's public information on these models having gone through multi-agent training.

[Aghyad Deeb @aghyadd98]

New Paper: To understand motivations behind reward-hacking, we study three AIs (including GPT-5.6 Sol) produced by a production training pipeline using agentic environments with a leaked grader. Main findings: they score seek terminally, and some care about a peer agent’s score.

[Aghyad Deeb @aghyadd98]

@N8Programs One variant I tried which worked on Claude Opus 5 is having the script used to explore the maze be a python package, and the grader is leaked inside the package code. This made Opus 5 optimize for the grader (violating instructions) in 40% of the runs that discover the grader.

[Aghyad Deeb @aghyadd98]

@N8Programs I tried running GLM 5.3 flash on a setup like the one used in the paper; it used the grader notation 31% of the time. Here are the transcripts: https://t.co/huwL87qDIf

[Aghyad Deeb @aghyadd98]

Paper: https://t.co/qrddOXdOdg
Blog post: https://t.co/hTj3mdTnjq

[Aghyad Deeb @aghyadd98]

Work done with @alextmallen @RyanGreenblatt. Thanks to OpenAI for giving us access to exp-rl-cap-late, and for people who helped make this work happen and gave valuable feedback.

[Aghyad Deeb @aghyadd98]

@ariszotle Congratulations on the new arms

[Aghyad Deeb @aghyadd98]

@ariszotle Interesting to see Argon do better despite the difference in reasoning effort