Ryan Greenblatt

Chief scientist at Redwood Research (@redwood_ai), focused on technical AI safety research to reduce risks from rogue AIs

[Ryan Greenblatt @RyanGreenblatt]

Plan A seems like a good plan for handling powerful AI, or at least the best plan anyone's written up.

Many choices initially seem crazy, but are actually pretty carefully considered.

Plan A isn't likely to happen, but pushing for something like this seems worthwhile.

[Daniel Kokotajlo @DKokotajlo]

In AI 2027, we predicted that AI would take over the world or irreversibly concentrate power.

In AI 2040: Plan A, we've laid out our positive vision for what should happen instead.

[Ryan Greenblatt @RyanGreenblatt]

Would be great if there were consistently community notes saying that training on only outputs (without logits) works well. Or saying that the term "distillation" is now commonly used to include training on samples without logits.

Maybe there should be a canonical write up...

[Migel Tissera @migtissera]

How do you distill a model without logits?!

Like, why do you guys keep saying this? It's one thing for some non-technical person to keep bashing Chinese labs. But even the technical folks? Why?!

If your idea of Chinese models' improvement is because of "distillation (without logits!) using API data" -- (also, note that the reasoning-traces are held out at the server -- you only get a hash from Anthropic) -- and you're a technical person, you're either,

1. Not that technical, or
2. You're trying to get likes on Twitter.

[Ryan Greenblatt @RyanGreenblatt]

@migtissera There is a lot of evidence demonstrating that training on just outputs (without logits) can transfer capability improvements from post training. E.g. work on reaching similar perf to R1 via SFT on transcripts.

There are known ways to get around hidden reasoning for Ant models.

[Ryan Greenblatt @RyanGreenblatt]

@joshua_saxe @BronsonSchoen @sebkrier @Miles_Brundage I don't think gpt-2 was handled very reasonably, but I think the situation was significantly "we should try having a different release process that's more cautious as this might be needed going forward", as in, people weren't really that worried about this specific AI (I think).

[Ryan Greenblatt @RyanGreenblatt]

@joshua_saxe This seems straightforwardly inaccurate about historical risk claims by individuals under discussion.

[Ryan Greenblatt @RyanGreenblatt]

@yonashav I think this is good, but not that sufficient. E.g., it's hard to know what to look for without some understanding of the situation at the company and current AIs aren't smart enough to figure this out themselves. And AIs aren't that robust. Other issues too.

[Ryan Greenblatt @RyanGreenblatt]

@Simon__Grimm @Miles_Brundage I'd guess 5.6 Sol is somewhat worse than Mythos Preview for the applications that matter most and Ant had this model much earlier.

[Ryan Greenblatt @RyanGreenblatt]

FWIW, I personally don't have this theory of "victory".

I do think small leads could yield permanent advantage, but this is an insanely scary situation. I'm worried about misaligned AI takeover, human powergrabs, and a bunch of other shit.

See (e.g.) https://t.co/0urmZxjRBQ to understand my perspective (TBC, I don't agree with everything; there are multiple authors).

[Ryan Greenblatt @RyanGreenblatt]

@daniel_271828 Chip advantage seems much more durable / relevant anyway, especially in the regime where AI R&D is automated so relevant labor is commoditized.

[Ryan Greenblatt @RyanGreenblatt]

@teortaxesTex Yeah, continued pretrain would def count as a different base model (with how I'm using these terms). And totally possible that 4.1 is a continued pretrain on the 4 base model in which case I'd expect better perf on this sort of thing. Haven't checked though.

[Ryan Greenblatt @RyanGreenblatt]

@JoshPurtell Source?

[Ryan Greenblatt @RyanGreenblatt]

I did some quick tests that indicated that the Kimi K3 pretrain is around halfway between Opus 4 and Opus 4.5. So ~10 months behind Anthropic. These tests probably understate data improvements, so overall I think it's a similarly good pretrain to Opus 4.5 (~8 months behind). These tests are better at measuring "general pretrain capability" than at incorporating (coding-specific) data quality.

This was prompted by me thinking more and realizing that my claim that "As a pretrain, it's probably somewhere between 4.8 and Mythos (around halfway between?)" was probably too bullish on the model and that I might as well test and find out. (And yep, this was very wrong.)

I think Mythos is a pretty big step up in pretraining, so K3 might be more than 8 months behind on the historical pretraining trend relative to Mythos (as in, Mythos is >>3 months ahead of K3 and Mythos was fully done training ~5 months ago).

Overall, this makes me suspect more of the improvements are due to distillation-type effects and makes me think the full catch-up times would be somewhat longer (if Ant/OpenAI stopped but investment still followed current trends). Minimally, more of the improvement probably lives in post-training/mid-training.

For reference, the same test indicates K2.6 is around halfway between Sonnet 4 and Sonnet 4.5. (And this roughly corresponds to some other similar measures.)

Sorry about the error.

[Ryan Greenblatt @RyanGreenblatt]

Kimi K3 was significantly but not massively above my expectations. I'd tentatively guess it's similar in overall usefulness/usability to Opus 4.8 and in overall capability somewhat above Opus 4.8 (while also being somewhat more benchmaxxed). As a pretrain, it's probably somewhere between 4.8 and Mythos (around halfway between?). Maybe this implies Kimi is like 8 or so months behind Anthropic in overall model strength/goodness (including usability) and like 6 or so months behind on overall capability (somewhat below Mythos Preview).

This gap is presumably reduced by distillation (and more generally using OpenAI/Anthropic models) and algorithm leakage/diffusion, so I think that hypothetically if the US completely stopped and recent algos didn't diffuse, it would maybe take Kimi like 10 months to fully catch up to the best internal (including in development) Anthropic model. (I think this notion might be a better measure of where Anthropic/OpenAI are relative to Kimi, even though this hypothetical won't happen.) And if the US completely stopped, it might take Kimi around 27 months to reach the level the US would otherwise have reached one year from now (as in, with a year of further progress).

My views here are pretty sensitive to how much benchmark performance is representative to overall usability.

I think I now expect an open-weight AI which is straightforwardly "Mythos-level at cyber" (including usability etc.) in like 5 months supposing Kimi and others don't change their open-weight model policy. (I don't have a strong view about how big of a deal this is for cyber, but it may cause significant political consequences. This could be a significant overestimate of the time required.)

I wonder what's driving Kimi being closer than I would have expected. Options include:

- Experiment compute is significantly less important than labor (and labor at Kimi is competitive, which seems super plausible)
- Implies more of a speedup from AI automating AI R&D and a bigger software-only intelligence explosion.
- Or possibly Kimi is just doing much better than US companies and this is overcoming experiment compute disadvantages.
- Algorithms are diffusing a lot / quickly (from e.g. OpenAI to Kimi).
- Perf is overstated / benchmaxxed a lot.
- Distillation / using OpenAI or Anthropic frontier AIs in AI development is very helpful for catching up. (But I'd guess Kimi K3 is a competitive pretrain which distillation doesn't help with?)
- US companies aren't going as fast as they could for whatever reason.

[Ryan Greenblatt @RyanGreenblatt]

@tyler_m_john @deanwball I'd guess less than a year behind tbc and it will depend on how you weight various usability dimensions.

[Ryan Greenblatt @RyanGreenblatt]

I don't think this particularly supports "the primary driver of capabilities isn't distillation". It just implies that most of the action is in mid-training/post-training (or possibly data quality).

The evidence I've seen seems consistent with "it would be over a year behind (worse than Opus 4.5) without distillation".

[Ryan Greenblatt @RyanGreenblatt]

@stanislavfort > Distilling without logits is crazy inefficient

This is very false for obtaining perf improvements from post training / RL. E.g., many experiments indicated you could train DeepSeek v3 to match r1 on (narrow) math with just 1000s of transcripts from r1.

[Ryan Greenblatt @RyanGreenblatt]

@tenobrus I'd guess it's similar to (or somewhat worse than) unsafeguarded opus 4.8 for cyber offense. Not sure though and current evidence also seems consistent with more like halfway between opus 4.8 and Mythos.

While the worse pretraining matters, post training is a big deal!

[Ryan Greenblatt @RyanGreenblatt]

@tszzl Ultimately the wildly superhuman ASI will help us or there are bigger problems. So I'm not worried.

[Ryan Greenblatt @RyanGreenblatt]

@celestepoasts The steel man (not necessarily endorsed): current cyber (and bio) risks are dominated by the longer run impacts and open weight models are good for safety research. Further (the argument would be) the precedent setting is either positive (because good for research) or smaller.

[Ryan Greenblatt @RyanGreenblatt]

Oops, I linked the wrong thing. This is a better canonical paper, but I just did quick and dirty experiments using this dataset: https://t.co/5Tt0Tm0rYA

[Ryan Greenblatt @RyanGreenblatt]

This is based on single forward pass perf (on math). This is imperfect, but corresponds to pretrain quality. (Minimally, it isn't improved by post-training.) Loss would be better, but we don't have the base and can't measure loss on closed weight models. https://t.co/QdlftmWosV

[Ryan Greenblatt @RyanGreenblatt]

@renatomoraesp @tszzl Mythos had an atypically long delay from internal deployment to public deployment. I think the public deployment lag for (e.g.) 4.1 Opus was probably more like 1/2 weeks.

I think releasing fast is desirable in both US and China, it's just slower now for compliance.

[Ryan Greenblatt @RyanGreenblatt]

Kimi K3 was significantly but not massively above my expectations. I'd tentatively guess it's similar in overall usefulness/usability to Opus 4.8 and in overall capability somewhat above Opus 4.8 (while also being somewhat more benchmaxxed). As a pretrain, it's probably somewhere between 4.8 and Mythos (around halfway between?). Maybe this implies Kimi is like 8 or so months behind Anthropic in overall model strength/goodness (including usability) and like 6 or so months behind on overall capability (somewhat below Mythos Preview).

This gap is presumably reduced by distillation (and more generally using OpenAI/Anthropic models) and algorithm leakage/diffusion, so I think that hypothetically if the US completely stopped and recent algos didn't diffuse, it would maybe take Kimi like 10 months to fully catch up to the best internal (including in development) Anthropic model. (I think this notion might be a better measure of where Anthropic/OpenAI are relative to Kimi, even though this hypothetical won't happen.) And if the US completely stopped, it might take Kimi around 27 months to reach the level the US would otherwise have reached one year from now (as in, with a year of further progress).

My views here are pretty sensitive to how much benchmark performance is representative to overall usability.

I think I now expect an open-weight AI which is straightforwardly "Mythos-level at cyber" (including usability etc.) in like 5 months supposing Kimi and others don't change their open-weight model policy. (I don't have a strong view about how big of a deal this is for cyber, but it may cause significant political consequences. This could be a significant overestimate of the time required.)

I wonder what's driving Kimi being closer than I would have expected. Options include:

- Experiment compute is significantly less important than labor (and labor at Kimi is competitive, which seems super plausible)
- Implies more of a speedup from AI automating AI R&D and a bigger software-only intelligence explosion.
- Or possibly Kimi is just doing much better than US companies and this is overcoming experiment compute disadvantages.
- Algorithms are diffusing a lot / quickly (from e.g. OpenAI to Kimi).
- Perf is overstated / benchmaxxed a lot.
- Distillation / using OpenAI or Anthropic frontier AIs in AI development is very helpful for catching up. (But I'd guess Kimi K3 is a competitive pretrain which distillation doesn't help with?)
- US companies aren't going as fast as they could for whatever reason.