It's so tragic that universal love is all you need and it's already right there but you also need to proactively defend against those who don't believe either of those claims to be true.
In Rotterdam, @MaximeStauffer@MopTheTerritory@chiaragerosa and I lost our minds over his stuff. In particular this hyperreal painting of NYC from 2016. A masterpiece, without hyperbole. Just sublime. https://t.co/RhEgGAjFWB
@nandabeniniii thank you for reminding me of this!!
I’ve read this a couple times now, and while there is, frankly, a lot of sloppy argumentation (which I will get to), the charitable, simplified read of Dwarkesh's position seems to be:
1) AI needs major breakthroughs to achieve AGI/ASI (continual learning, sample efficiency, etc.) — this is a view he’s held for some time now, in line with Ilya
2) These breakthroughs, and the ML research skills required to obtain them are verifiable, and thus current, pre-breakthrough AIs can hill-climb on these sorts of problems and capabilities to solve them (like we saw in math)
3) Once these breakthroughs occur, we *may* see rapid, recursively compounding gains to intelligence, leading to ASI in a very short period of time
There's a ton of other stuff in the essay, but I think this is the basic crux. Christiano basically articulated this same argument in his Dwarkesh appearance, and if you look at AI-2027, it essentially follows the same trajectory: Agent 2/3 makes a ton of algorithmic progress, allowing Agent 4/5 to explode.
In general, I think the high-level narrative here is...fine? It’s certainly one plausible outcome. But, imo, it is a much narrower path than I think many are giving it credit for—a lot of things actually have to go just right for this to play out. And here's where I return to the other, messier arguments, because many of the intuitions and arguments driving Dwarkesh's view in favor of near-term ASI (e.g., EO2027) as a highly probability outcome (~50%) are, imo, quite bad:
- “[We'd see] a jump as big as GPT-3.5 to GPT-6 Astra in just 12 months” -> No, under this scenario we’ll see ~4 years of AI progress in 1 year. If there are diminishing returns, then this won’t be as extreme as the jump we saw from GPT-3 to GPT-6. It very well could be, but that would require innovation to overcome the diminishing returns we'd expect in the base case.
- “The other big piece of evidence for transfer is human evolution” -> Except if AI was following the sort of pattern we saw for human evolution, we should have seen better transfer long before AI started solving millennium problems. This spikiness has been one of the big surprises of AI!
- “Rather, my claim is that scaling up RL training will give AIs the abilities necessary to ramp quickly in new contexts…it’ll go through an evolutionary process that involves centuries of corporate competition to figure out the optimal organizational techniques.” -> These are in direct conflict! If ramping requires RL runs that simulate centuries of competition within “digital economy full of millions of other AIs” then that’s quite bearish for ramp speed (relatively) given the compute and wall-clock requirements!
- “I have a lot of 20-something-year-old friends whom I’ve watched accomplish incredible things and build world-leading companies and ventures in domains they entered into with no previous experience.” -> lol. rip leopold. But also, again, these are humans, with human sample efficiency. If AI gets there, sure. But how soon it gets there is the big crux.
- “Obviously, some pieces of feedback—about how your actions impacted the quarterly earnings, say—require months of serial time. But even granting that, we should expect these future AIs to be able to get to top human performance in almost all domains within months of being deployed.” -> …no that’s not how that works at all because one set of earnings isn't enough to robustly learn from and you can’t re-run the same real-world conditions.
- “[Sample efficiency & online learning are] AI’s equivalent of a Millennium Prize problem. And mathematics has already shown us what happens to such problems” -> Sure, but there’s a difference between “AI can solve a Millennium-level problem” and “AI can solve *this specific Millennium-level problem*” -- if the blocker is a specific problem, the timing spread becomes much wider.
- “My answer to this [experiment/wall clock/compute] objection is that each agent will still have access to a ton of compute. Frontier labs have been more than quadrupling their compute every single year.” -> But that’s what they’ve needed to do keep up with their current pace! Even if the labs maintain their current pace of compute growth (and like, will they even? capital markets really do not seem accommodating and Dwarkesh's roommate just lowered his compute projections), we'd still need a step change in pace above that!
And that's not even covering the biggest crux of all: whether the returns to intelligence continue to recursively compound faster than diminishing returns eat into them.
Now, are any of these objections fatal? No. Do they “disprove” hard RSI/an intelligence explosion? No, ofc not. Am I kind of nitpicking here? Yeah, 100%.
And to Dwarkesh's credit, he does anticipate and acknowledge a scenario (the “spark”) where these objections play out.
But the reason I've bothered to rant this long is that I see a lot of AI/RSI discourse follow this pattern: where 50% of the arguments are reasonable and grounded and appropriately hedged, and then the other 50% are these sorts of sloppy intuition pumps and what-ifs that include a ton of detail but a noticeable lack of rigor. And I can’t help but get the sense that it is the latter 50% that is really driving people’s predictions and expectations here, as it leads to sentences like this:
“In the spark scenario…it takes years to get to the same endpoint that the software-only singularity would have achieved in months: a superintelligence that is to us what we are to chimps.”
Note that Dwarkesh thinks these two outcomes (spark vs. software-only singularity) are equally likely. 50/50. I’m sorry, but that’s absolutely nuts. Even Christiano put the likelihood of ASI by in 2027 *absent any slowing down* at only 1 in 3, and I still think that's too bullish.
Like, yes, I fully anticipate that we will have automated AI researchers (or something close to them) in 2027, and the universe where we go from that to a software-only singularity is not entirely out of the question. But obtaining that universe relies on a confluence of various frictions, diminishing returns, and wall-clocks being overcome, a somewhat specific set of algorithmic innovations being made or obviated, and an assortment of other factors playing out just-so to enable that perfect storm. Even a fairly rapid exponential doesn't qualify when you're talking about full-blown ASI in 13 months. It may not be impossible, but it's certainly not a 50/50 outcome.
I don't know if you actually need legislation to get real commitments out of these companies. But you definitely need the threat of legislation. You need them to believe that the alternative to doing this is MORE transparency and oversight.
There's so much reason to be so hopeful for humanity. We have come so far in so little time. It's hard to think about how it could have gone much better much faster.
I have believed this for a very long time. But the first two days with my daughter have made it even clearer.
It's so tragic that universal love is all you need and it's already right there but you also need to proactively defend against those who don't believe either of those claims to be true.
@__drewface I think there's a lot you have on offer for a subset of experts, as well as the wider scene :)
Girl dads shall unite on a podcast asap! I gotta get my mic from the office but that is feasible
The panic vibes around first the hugging face incident and now the ai r&d acceleration report are really confusing to me.
Either, a bunch of people who've been making accurate predictions are being fake dramatic, hoping it doesn't look like being dishonest to pretend to update drastically in light of roughly on par events OR they didn't take their forecasts seriously themselves OR they feel powerless and desperate (or any combination of these).
I feel you guys. But the panic vibes are really not helping us. Ground your nervous systems and get to work, there are lots of people looking for support right now who have woken up to the dangers of the game we're in. The last thing they need is more advocacy noise. If you cannot access any such people, consider that you might be best off building technical solutions. Otherwise, help people understand the game we're in to defeat the odds.
But telling decision-makers the odds has approximately 0 marginal value. And maybe more importantly, will backfire hard whenever you're wrong. And you will be wrong. That's fine. Nobody cares if you can make yourself concretely useful. It's not most people's job to be right, that's the job of a tiny sliver of humanity.
i would love to do a podcast with you on this. if you have a setup, maybe we can just sit down and yap. there's so much to say here and i feel very heartened by your reaction and reflections. also i'm on paternity leave now for a good moment.
i think there's something you and i have that is a highly detailed map of a part of reality that the people i'm talking about in op don't seem to have. this is why despite having made the right forecasts at a technical level, they seem to now not know what to do with having been right.