@tszzl Not so sure. Extremely powerful models will have to be managed by security-state-like infrastructures. Maybe there can be diversity within these but I don't think we'll have a private, competitive AI market in 4 years
I wanted to be an astrophysicist. Instead, I've spent the last three years building a mass movement to pause AI, dedicating to this cause every ounce of my patience and ability, desperately trying to convince politicians to ban superhuman AI development before it kills us.
Do you think talking to politicians is fun for me? Believe me, I'd much rather be staying awake at night scrutinizing exoplanet data. Instead, I'm staying awake at night working to prevent my loved ones from being killed.
Redwood Research CEO @bshlgrs responds to Google DeepMind researcher @EgeErdil2 asking what would prove his short AI timelines wrong:
"Sorry, Ege, but I'm not gonna hang my head in shame if there's no superintelligence by the end of next year."
"One of the things where I would feel dumbest, if it turned out that recent increases in the rate of AI progress are almost entirely determined by factors that are not AIs contributing to AI R&D."
"If the progress has just been from increasing amounts of training compute, then I would feel pretty embarrassed."
"If superintelligence is developed and there's absolutely no misalignment issues and the whole thing was super chill, I will feel somewhat embarrassed to have been wrong about that one, though overall very glad."
@redwood_ai
@danfaggella The violence valence will turn, and multiply. It won’t be you getting the threats, it’ll be those trying to keep the research going on unfettered
In 2020, @harris_edouard and I quit a profitable YC-backed startup we founded, which had become the world's largest marketplace for income share agreements. It was fun and we'd rather have kept doing it.
We went for almost 2 years without a salary. I got married later than I would have, had kids later than I would have, made far fewer investments than I would have. Just because we read the original scaling laws and GPT-3 papers and realized the writing was on the wall, and that AI was about to become the most important national security problem in world history.
We turned down funding from external sources to maintain our independence for the last 6 years as we did this. We turned down job offers at frontier labs. We literally had to spin up an entire for-profit business just to cover our expenses as we tried to warn the U.S. government of what was coming.
We spoke with dozens of politicians, briefed cabinet members all unpaid. We took on wildly below-market work when we thought it was mission-aligned.
In 2025, we spent over a year researching and writing a report, which was completely, 100% unfunded. It flagged concerns about Chinese espionage at frontier labs, and some of our friends in the intelligence community said that publishing it might put us and our families at some risk. Not huge risk, but some risk. We published it anyway.
I could go on, as could so many people in the AI community who've sacrificed for this. @DKokotajlo did in fact put most of his net worth at risk to blow the whistle. @So8res could be a frontier lab equity millionaire many times over. Many such cases.
a number of respected researchers (e.g. @DKokotajlo@bshlgrs@ajeya_cotra@RyanGreenblatt) have suggested that the METR-documented ~2x speedup in AI progress at anthropic over the last year supports the hypothesis that there will be a near-term (e.g. 2027) intelligence explosion (IE). how exactly does this reasoning work?
one way to think about this: perhaps the leading alternative to the near-term IE hypothesis is what we might call the factor F hypothesis, which says that there's at least one key factor F [e.g. creativity, research vision, planning ability, ...] such that (i) high F is required for an AI system to reach the superhuman levels required for full-scale IE [e.g. ASI, artificial superhuman general intelligence or at least SAR, superhuman AI researcher], and (ii) F is so difficult to build into AI systems that leading AI systems won't have high F in the near term. i don't know whether the factor F hypothesis is true, but it's not obvious to me why the 2x speedup makes it much less likely.
it's widely agreed that the 2x speedup reflects an even larger (perhaps 10x) speedup in other capacities G such as coding and experimentation. under the F hypothesis, G could explode locally, leading to major progress in sub-AGI capacities without much progress in F (think of G as the easy part of AGI and F as the hard part). that tends to lead just to a weak form of recursive self-improvement (call it an intelligence subexplosion) below AGI levels.
the AI futures document responding to the METR report is usefully explicit about nearby issues. the authors report believing in a high-ish growth rate for "research taste" (AI research factors beyond coding and experimentation) relative to increases in general capacity. presumably they take potential Fs to be aspects of research taste that will grow fast too. but i don't really see where the evidence counts again a relatively-low-growth F hypothesis where F stays below threshold for years. maybe one could argue that the explosion in G provides bayesian evidence against this F hypothesis (G's exploding is antecedently less likely under the F hypothesis than under its negation), but that's far from clear, and even if true there will remain a solid mass of credence under the F hypothesis.
maybe there are other reasons to reject the factor F hypothesis: e.g. because AI research is relevantly like math research, and AI math research is becoming superhuman already. but if we have reasons like that, the 2x speedup isn't really needed to make the case for near-term SAR and IE. maybe the 2x speedup could speed up an expected IE slightly from 2028 (two years away) to 2027 (one year away), and/or maybe it could be extrapolated into ever-faster speedups (another 2x in 6 months, 3 months, and so on) converging in 2027. but those are very different arguments that require huge premises to work.
am i missing something here? i'm curious to hear which evidential routes from 2x speedup to near-term IE people have in mind.
N.B. i'm not saying you shouldn't worry about near-term ASI. i'm worried!