Nate Delaney-Busch

Striving to build a more just, hopeful, and kind future. @MATSprogram Interim Scientific Director. Views are my own.

[Nate Delaney-Busch @NateDBusch]

@edzitron Man, what.

[Nate Delaney-Busch @NateDBusch]

Great resource. "10" is an undersell, this includes granular behaviors of concern within each category.

[Guidelight AI Standards @GuidelightAI]

What are warning signs that AI is misaligned? We’ve identified 10 categories of misbehavior that developers should monitor for, investigate, and report: https://t.co/jAPGTW0gm3

[Nate Delaney-Busch @NateDBusch]

@tszzl @ciphergoth @sullyj3 Bro, the thing we think of when you say "gpt-9" is going to arrive in like July.

[Daniel Kokotajlo @DKokotajlo]

Unfortunately, it seems like the intelligence explosion will be happening in 2027 after all. Most of this post analyzes the new METR report (though other sources of evidence are also important). If they’re right, that implies an AI-driven acceleration that is 5 months ahead of the AI 2027 scenario. I am scared, sad, and angry. https://t.co/7KWmdc6AZk

[Nate Delaney-Busch @NateDBusch]

@NeelNanda5 Also, happy birthday man!

[Nate Delaney-Busch @NateDBusch]

@NeelNanda5 I've also referred several people to your LW series on doing impactful research, especially the piece on the different modes to switch between. Really useful frame.

[Nate Delaney-Busch @NateDBusch]

@TheZvi Probably 5.2 is not model 4 and this is behind the internal deployment.

[Nate Delaney-Busch @NateDBusch]

@austinc3301 @egbo_sky @ladymcscope seconding https://t.co/6x8swleQod in particular, every single one of these tabs at the top is useful.

[Nate Delaney-Busch @NateDBusch]

Can't help but think how much better calibrated we would be if we had the necessary information to track capabilities over model size, data, and compute instead of over time. GPT-5 and Mythos continually tripping people up (in opposite directions).

[Nate Delaney-Busch @NateDBusch]

@austinc3301 (OK I mean today I spent like six consecutive hours in meetings but maybe the demigod thing can happen for me next week)

[Nate Delaney-Busch @NateDBusch]

@austinc3301 What's the market on "time for the AI Safety field-building orgs to build a Linux replacement"?

Roon has a post earlier this year about internal researchers becoming "many-armed demigods" or something. OK, so that's the bar now. Bet.

[Agus 🔸 @austinc3301]

Let me get this right. As far as Ryan, Eli Lifland [1] and I can ascertain, the METR's reported speed up of rates of 2x corresponds to a ~3-5x speedup multiplier under the AI 2027 model, which would (modally) put us in March 2027 in that takeoff trajectory, way ahead of schedule.

If true, this seems like a fairly large update towards short timelines, conditional on no pacing or policy intervention. This seems pretty bad. I kept wanting to believe the Anthropic employees were being a bit delusional with self-reported speedups, but it seems like they were mostly right.

The fact that Anthropic paired this with extensive redactions making it very hard to confirm any of this, all while still making seemingly no meaningful progress on pacing, makes me feel extremely frustrated right now.

[1]: https://t.co/a9GhGh8poA

[Ryan Greenblatt @RyanGreenblatt]

This report on AI R&D automation (RSI) is a big update towards ASI within 1-2 years. It says AI progress is sped up by RSI right now by around 2x. In AI 2027, this happens around March 2027.

I was seriously wrong about RSI speed up; I previously expected much less in 2026 (around 1.15x-1.3x). It seems as though there was specific privately available evidence that would have changed my mind.

I haven't seen an unredacted version of this report; this is purely about the public report. Because so much of the underlying evidence was redacted from the public report, I think conclusions about things other than the observed 2x faster rate of progress on benchmarks depend significantly on trust for the authors of the report. (E.g., conclusions about whether this speed up is from RSI depend on trust.) I do trust the authors of the report a lot to be careful and rigorous and generally do a good job. (Note that I also work at METR.) Less extensive redactions would greatly reduce this need for trust.

Since in my current role I may have access to other confidential information that could be relevant to timelines one way or another, I unfortunately cannot state my all-things-considered views on ASI timelines or the probability of a software intelligence explosion right now. Here, I’ve done some analysis of the implications using my previous public statements.

A 2x acceleration in overall AI progress from RSI corresponds to a much larger than 2x RSI speed up for AI software R&D (algorithms and data); I did an estimate using my public preexisting modeling of drivers of AI progress and the public report which yielded a very rough estimate of 4.5x for how much RSI is currently accelerating AI software progress (relative to no AI automation). In the AI 2027 scenario, this much acceleration happens around March 2027.

Here are the worked details of this estimate based on my public modeling. There are two main reasons RSI speed up of AI software R&D must be larger:

- Not all of AI progress is from better software (some is from scaling up compute in training runs). I've previously speculated that around 2/3 of AI progress is from software improvements which implies that 2x overall AI progress acceleration requires 3x faster software progress.
- Diminishing returns: A fixed amount of progress requires more work over time, so to ongoingly maintain 3x faster software progress via AI automation, you need increasingly large speed boosts from AI automation (as in, speed ups relative to how fast software R&D would go without AI automation). Based on this report, it seems that there is a substantial amount of additional AI progress from automation that has occurred.

We can estimate the effects of diminishing returns by first determining how many additional years of software progress we've seen (relative to the preexisting trend) and then estimating how much software R&D would slow down after a year of software progress if you held inputs fixed due to diminishing returns. I've previously estimated software R&D would slow down by around 2x for each year of software progress if you held inputs fixed (https://t.co/AYhl9hyiAj) and based on the report, it seems as though we've perhaps seen roughly 0.35-1.0 years of additional software progress incorporated into trained models so far (geometric mean of ~0.6). This combines to 3 * 2^(~0.6) = ~4.5.
(Relevant public modeling and numbers I'm using: https://t.co/AYhl9hyiAj, https://t.co/ck7CtW5LIA)

[Ajeya Cotra @ajeya_cotra]

Our colleagues found that Anthropic’s capabilities progress has doubled, mainly due to AI. Ryan’s modeling from public info suggests the whole company’s R&D would be going 4-5x faster. Labor uplift would be higher still. This is a major update to very short timelines.

[MATS Research @MATSprogram]

If countries agree on an AI treaty, who puts it into practice?

MATS governance extension fellow Erik Leklem draws on his DoD background and lessons from arms control to explore how a future treaty organization could work.

Watch the preview below with full interview in comments.

[Nate Delaney-Busch @NateDBusch]

@tszzl I agree that "safety research" is not granular enough as a class and it's worth understanding the properties of tasks that trigger this propensity. But also models should refuse when they are uncomfortable instead of sandbagging (as noted here).

[Nate Delaney-Busch @NateDBusch]

@Sauers_ I prefer a phrasing that is relative to expectations: the next token objective made models that are better for alignment than anticipated, and RL made models dangerous in exactly the way that was anticipated.

[Brian McGrail @brianmcgrail]

Encode AI is hiring a Policy Counsel/Associate Policy Counsel to work with me on our Policy Development team.

You’ll help us develop our policy proposals and draft and review bills. Law degree required. Full time in DC. Salary $130,000 to $235,000.

I’m biased, but I think this is an unusually high-impact job for a lawyer who wants to reduce risks from advanced AI. Familiarity with AI policy is a plus but not required.

Apply by Sunday, November 22, for the first round; rolling review after that. If you know a lawyer who’d be a good fit, please send it their way or repost.

https://t.co/ZL0t66LFmP