Tom Davidson

Senior Research Fellow @forethought_org

Understanding the intelligence explosion and how to prepare

[Tom Davidson @TomDavidsonX]

So, exactly how big will the intelligence explosion be?

…Ten years of AI progress in a year? In a month?

Our new paper tackles this question head-on.

I've researched AI takeoff speeds for many years. This is my best stab at an answer.🧵

[William MacAskill @willmacaskill]

New paper out! With @ElliottThornley:

We explore the case for training AIs to be risk-averse in resources.

The idea is that risk aversion acts as a failsafe against misalignment.

If AIs turn out misaligned but risk-averse, then, at least before the point at which they are far beyond human capability, we can pay them to cooperate with us - e.g. to reveal misalignment, or to do useful work - because they prefer the payment-and-cooperation to a low chance of successful takeover.

What’s more, we think there are some reasons for thinking training for risk-aversion may be easier than getting full alignment.

[Brangus🔍⏹️ @RatOrthodox]

My current best guess after thinking about it for a while is that the effective banning of claude fable was bad from an x-risk pov? This mostly causes the labs to have a larger tech lead over the rest of the world, which seems mostly like bad news about the future to me. It also gives civilization less of an ability to adapt to gradual changes in released capabilities. If you model the lead up to the point of no return in detail, one of the key parameters to pay attention to is the extent to which the distribution of intelligence is highly concentrated, and lead ups with more concentrated intelligence generally seem worse to me. There's some positive precedent here, and it maybe makes people realize that governments might sometimes do things, but I'm not sure that's worth it. It's certainly not obvious.

This is why I really strongly prefer advocacy strats aimed at helping people reason sanely about the fundamentals of AI X-risk to advocacy strats that doggedly push anti AI sentiment for any reason available. The history of governance is riddled with terrible, counterintuitive outcomes caused by policies which were championed by people with the best of intentions. Popular movements lead by the morally certain and epistemically unscrupulous seem to me to have a pretty bad track record when it comes to picking policy especially within strategically complex domains, perhaps worse than random, which is itself an interesting puzzle.

[Tom Davidson @TomDavidsonX]

Cool, i look forward to reading. I agree this stuff is complicated. Apologies for wading in without reading, but i thought i should say that my impression from the tweet thread was that you were strawmanning pro-handoff ppl (i.e. mentioning ppl who think humans should die rather than the obv alternative that AIs control stuff in distant space) and leaning into a frame which shames ppl sympathetic to this position (i.e. painting them as crazy techies who have abandoned humanity, rather than eg ppl who feel it's fucked up for humans to permanently deny rights to digital beings).

[Tyler John @tyler_m_john]

New RFP! 'Checks and balances to empower citizens in an automated society.'

It's an 11,000-word RFP, research agenda, and funder's guide that we hope can serve as a launch pad for this new field.

Loved working with Ashwin Acharya, @JoalStein, and @divyasiddarth on this.

[Tom Davidson @TomDavidsonX]

@boazbaraktcs @peterwildeford @sama For mass surveillance I would have thought you need to look at the patterns of use across many interactions, rather than having a safeguard that applies on a per-interaction basis.

Can you do that?

[Tom Davidson @TomDavidsonX]

@LRudL_ @softminus Do you think there are versions of handoff that are good?

[Fin Moorhouse @finmoorhouse]

Wei Dai has been thinking about AI safety longer than I've been alive, and his recent ideas deserve more attention. As far as I know this is his first podcast interview (of sorts!)

[Tom Davidson @TomDavidsonX]

@DavidSacks They believe it. And the implication for you, that you highlight, is that you should support a pause/slowdown

[Yo Shavit @yonashav]

Twitter community, I would like you to perform a public service. Please take a moment to calibrate how unhinged Dean’s posting has been.
Now, resolve that you will dunk on his ass if he lets incentives make him noticeably less unhinged.

Dean, we’re watching, and rooting for you.

[Jaime Sevilla @Jsevillamol]

A couple of years ago, we published a taxonomy of AI R&D tasks that was surprisingly popular. My colleagues have now significantly expanded on that work, breaking down the AI development process into 60+ tasks.

[Tom Davidson @TomDavidsonX]

@JeffLadish We don't know for sure, but it's pretty likely targeting

[Andrew Koh @andrewjkoh]

Fourth, long-run growth: data and capital (compute) accumulation feed into each other. Data simultaneously augments task-specific productivity + gradually shrinks need for labor => learning-by-doing singularity (infinite growth in finite time!) But this happens pretty slowly vs ‘software only singularities’ (@TomDavidsonX @BasilHalperin Tom H @akorinek)

These views aren't mutually exclusive! In a way we’re being pretty conservative – the economy is messy, but we know how to do RL so this data channel is lower bound on how crazy things can get! Basil and I think it's worth unifying both views and taking it to data!

[Kobi Hackenburg @KobiHackenburg]

New w/ @AISecurityInst & @UniofOxford:

Frontier AI can now out-persuade expert humans in conversation - incl. world-champ debaters and professional canvassers.

This held even when humans chose their topics, prepared in advance, and competed for £1,000 prizes 🧵

[Tom Davidson @TomDavidsonX]

disagree. how is "stop ai now" the lesson from Fable events? It may be true! but how in any way is it a lesson from those events?

The govt just showed it has no idea about ai and is willing to capriciously abuse its powers. Seems v reasonable to take away a lesson about govt incompetence and power concentration!

i agree with you: we need to prepare to pause. but i find it frustrating when doomers insist that this has to be the bottom line of every post and the takeaway from every new development

[David Krueger 🦥 ⏸️ ⏹️ ⏪ @DavidSKrueger]

No, the lesson is not:
"try to fix government before the singularity"
instead of:
"try to fix AI before the singularity"

The lesson is:
For the love of God, stop the singularity so we have some time to figure shit out!

Stop AI.
Now.
Everywhere.

Or risk losing everything.

[Tom Davidson @TomDavidsonX]

agreed. really shocking how many completely open antisemitic accounts with thousands of likes on their posts. really scary. they're not even saying "zionist" anymore

[Tom Davidson @TomDavidsonX]

@EgeErdil2 seems like an overreaction

[Dean W. Ball @deanwball]

Precisely as I predicted, the recent cyber EO, which admin officials insisted was not a licensing regime, ends up in practice being a licensing regime. Forget “voluntary,” forget “permissionless.”

AI is licensed now, but the requirements change constantly and are always a secret, even to the administration itself, which will discover the rules spontaneously in real time as it reacts to events. This means also that the rules are in practice stricter and more roughly enforced for organizations the administration does not like.

Can you blame Anthropic for making itself so disliked? In a sense, sure. The problem is that this childish “he said, she said” is all we have to go on in our analysis of the situation. And because there is no transparency (it is all calls and texts between “White House officials” and “Anthropic executives”), in practice it comes down to who you trust more.

This is why we create laws! To abstract away from personal power struggles and grudges, to submit to the steady application of rules so that complex human activity can unfold with predictability.

The rule of law has been being eroded in the U.S. for my entire life, but it is especially acute in AI because of both the lack of much preexisting law to serve as bulwark, and because of this admin’s insistence that it is Not Regulating AI. This has become an excuse for vagueness and evasiveness in rule-drafting (see the cyber EO), and this in turn makes the lawlessness worse.

The government wants to apply its force to frontier AI, that much is clear. It wants to make the industry submit. And in service of that goal, it has discovered that “not regulating AI” is in fact a great excuse for refusing to support laws that could constrain the admin’s exercise of power. In other words, “not regulating AI” is a *justification* for the tyrannical control of AI by the state.

This should alarm you regardless of what party you are in. What you are seeing now will be used against you one day soon, if not by this admin then by its successors. This is the antithesis of the rule of law.

The administration cannot and will not fix this problem alone. We need Congress to step in and impose rules on this mess.

[Rob Wiblin @robertwiblin]

I can understand someone who thinks Amodei 'deserves' bad outcomes because they believe he's straight up lying when he says he thinks AI will become dangerous.

But some commentators seem to think he 'deserves' punishment for sincerely warning the public (and them personally) about AI's dangers – notwithstanding that he believes it and may be correct.

It's an extraordinary loser mindset.

Implies you think people are totally wasting their time to inform you of things because you're too lazy or incompetent to use information to defend your interests in any case.

What fools they are to think telling you something could be of any use!

[Simon Grimm @Simon__Grimm]

Say Europe decided to build a frontier model. We do the following:

- Poaching founders from top US AI companies, paying them $100 million per year.
- Securing a datacenter site the size of Manhattan.
- Spending $125 billion on data center construction.

Would Europe succeed? Probably not.

Mark Zuckerberg is doing all of the above, and his company has not built a frontier model or made much money from selling AI models. Few count Meta as a serious contender in the AI race.

Similarly, Elon Musk tried and failed to build a leading AI model, has lost most of xAI's researchers, and is now leasing compute to Anthropic.

Instead of pursuing a bureaucratic, multinational 'CERN for AI' project, we need to consider more realistic plans for strengthening Europe’s position.

A lot of knowledge on how to strengthen Europe lies within individual European nations' capitals. There, national leaders can think about AI from the perspective of their national interest. The steps they take to make their countries grow, capture future industries, and indigenize parts of the AI stack will do a lot to secure Europe’s position.

Those thinking seriously about Europe's position on AI should, in the meantime, develop opinionated policies that individual European nations can pursue to make AI go better for Europe and the world.

@pietergaricano and I have written many more thoughts in our latest Substack piece.
https://t.co/meMDGaGBVK

[Eliezer Yudkowsky @allTheYud]

@TomDavidsonX Basically agree.