Robert Wright

Author of Nonzero, The Moral Animal, Why Buddhism Is True, and my new book on AI, The God Test: EIC @NonzeroPods @NonzeroNews

[Robert Wright @robertwrighter]

Happy pub date to me! My book on AI is now available. If you want to ask questions about it before buying, just click reply. (And feel free to retweet!) The link to Amazon is in my bio.

[Nonzero Podcasts @NonzeroPods]

Humanity: An Earth’s-Eye View | @robertwrighter and Boris Shoshitaishvili discuss Boris’s new book, Our Planetary Mirror, which draws on three big ideas—Gaia, the noosphere, and the Anthropocene—along with philosophy and myth to reassess who we are and the problems we face as a species. Overtime: Are humans AI’s fossil fuel? https://t.co/Wbd5CdridV

[Robert Wright @robertwrighter]

@jamesgibney there's no g in 'splainin

[Robert Wright @robertwrighter]

@ChrisRMcGuire Are there parallel cases where the MSS has failed to do the right thing? (Just curious as to why you're so confident that they wouldn't have done it in this case.)

[Nonzero Podcasts @NonzeroPods]

“This is like saying you’re going to create a one-world government.”

@NPCollapse on aligning superintelligence

[Robert Wright @robertwrighter]

@marcar yeah. it's like saying we'll stock all the fraternity houses with free liquor but tell the fraternity brothers not to overindulge.

[Robert Wright @robertwrighter]

If you’re among those who doubted the sincerity of Dario Amodei’s call last month for an AI slowdown, this week brought new grounds for your skepticism: Jack Clark, an Anthropic co-founder who serves as a kind of co-visionary-in-chief at the company, gave a talk at the IMF that I’d call de facto accelerationist.

One of the talk’s most salient themes is that we need to speed up the construction of AI infrastructure. A big reason, says Clark, is that then we’ll get the benefits of scientific advance sooner (which in turn means that building more data centers ASAP isn’t the only priority: “Infrastructure alone will not be enough. If discovery becomes abundant, we need to avoid bottlenecking ourselves downstream…”)

Clark does of course mention the dangers of AI, and near the end of the talk he alludes to the possible need for a slowdown. But here he is even vaguer than Dario was. Clark says, “We will need to build the infrastructure to take advantage of this technology, but we will also need to go slowly and deliberately on developing the technology itself, to pace or otherwise control its development.“

What does that mean in practical terms? For example: Since market dynamics encourage Anthropic and OpenAI to race each other recklessly, maybe the government should step in and make them slow down—like, maybe, mandate a pause on large training runs? No, there’s no mention of such a thing, and it sounds like Clark wants us to mainly just count on the companies to exercise good judgment. As we move toward recursive self-improvement, “labs like Anthropic will need to make decisions about how much oversight we apply to AI-driven research… We will need to move extremely carefully here, because otherwise we could lose control of the technology. We will also likely need rules to govern this.”

So apparently it’s only “likely,” not certain, that we’ll need actual rules—and even then it’s not clear that they’ll come from the government? I guess we’ll just let Dario and Sam come up with the rules at one of their many friendly meetups!

Clark’s lack of practical urgency would be remarkable in any event, but what makes it downright weird is his talk’s billboard theme: He declares that “the singularity is here.” He calls the singularity a time when “AI systems drastically increase the pace of overall scientific and technical progress, making things move so quickly that it is hard for humans to keep up.” Well if it’s already getting hard for humans to keep up, then maybe we should… slow things down? Apparently not.

“The singularity” doesn’t just denote fast technological advance; it denotes faster and faster and faster advance—a positive feedback loop that draws us into a vortex of change that could without warning pull us past the point of no return. It’s kind of like walking down an icy slope that gets steeper and steeper and steeper: Things are under control until they’re not. The idea that we’re in the singularity but it’s too soon to take dramatic precautionary action is a contradiction in terms.

I have a very different take on the singularity and its implications, and I lay it out in an essay called “The Singularity is Clear”: https://t.co/8aylVM7qAc.

And the text of Clark’s IMF talk can be found here: https://t.co/lFUDuFTrio

[Robert Wright @robertwrighter]

@ThatEricAlper I think you should allow this so long as the students can find an AI that reliably translates cuneiform.

[Robert Wright @robertwrighter]

@vmanancourt @calder_mchugh @MuiChristine @samjsutton Actually there's a lot of rightward-in-the-sense-of-libertarian lean in the AI community (plus in the sense of China-hawk).

[Alex Bores @AlexBores]

Anthropic is pitching a bill that closely matches its framework from June.

How can you watch Hugging Face, Coxon, & rogue AI swarms, and conclude that the bill you cooked up in June is sufficient?

This comes nowhere close to the speed of developments nor the speed we need to keep people safe.

[Robert Wright @robertwrighter]

I'm not saying it will happen. I'm just saying that if it doesn't happen we're toast (and that this is only now becoming the case). If you want to assess the argument, read my book (or at least this essay). If you'd rather just ridicule the argument without first assimilating it, well... have a good time.

[Robert Wright @robertwrighter]

@neil_chilson Yes. And again, I didn't mean to minimize the natural tendency of cooperation to emerge. In my book on AI I (https://t.co/3NVhVnAXG4) I discuss an experiment in which two LLMs readily colluded in a price-fixing scheme.

[John Horgan @Horganism]

"End of war"? Yeah, tell it Bob.

[Robert Wright @robertwrighter]

Daniel Seidemann is a Jewish Israeli who has been fighting the good fight for a long, long time. Americans who want to stop the ethnic cleansing of the West Bank should follow him and support him.

[Daniel Seidemann @DanielSeidemann]

Simultaneously pathetic and outrageous.

Under international law, as accepted even by Israel, the IDF Commander of the Central Command is the sovereign in the occupied West Bank.

The IDF regime backs settler terrorists and abandons to their fate Palestinians under our rule.

The savage attacks on Palestinians have been going on for months and years with the IDF, at best, turning a blind eye to the terror, and at times participating in "the festivities".

The IDF has now been taught that the world will give us a pass on pogroms against Palestinians, but is less tolerant when the victims are European diplomats and members of the press. That does not speak well of "the world".

It's long overdue to conclude that we will not stop until we are stopped.

[Daniel Seidemann @DanielSeidemann]

Simultaneously pathetic and outrageous.

Under international law, as accepted even by Israel, the IDF Commander of the Central Command is the sovereign in the occupied West Bank.

The IDF regime backs settler terrorists and abandons to their fate Palestinians under our rule.

The savage attacks on Palestinians have been going on for months and years with the IDF, at best, turning a blind eye to the terror, and at times participating in "the festivities".

The IDF has now been taught that the world will give us a pass on pogroms against Palestinians, but is less tolerant when the victims are European diplomats and members of the press. That does not speak well of "the world".

It's long overdue to conclude that we will not stop until we are stopped.

[Robert Wright @robertwrighter]

This is from @TheZvi, who is definitely not a reflexive Anthropic basher. https://t.co/IDBtxQJhvK

[Robert Wright @robertwrighter]

@JSchanzer You sound hopeful. Yet as I recall the Arab Spring didn't turn out very well.

[Robert Wright @robertwrighter]

National security in the age of AI is incompatible with the level of international conflict we're used to. I think controlling this technology will ultimately call for nothing less than the end of war. In the final installment of my five-part essay "The Singularity Is Clear," just posted on @NonzeroNews, I elaborate:

Part V: The Epic Challenge of International Governance

For years the term “warning shot” has had a special meaning within the AI safety community. It refers to a high-profile event that could focus attention on the catastrophic potential of artificial intelligence and thus on the need for preventive measures. Typically, the envisioned event involves significant death and suffering. “Chernobyl-sized” is the way some put it, referring to the 1986 nuclear accident that killed dozens of people in the short run and an estimated tens of thousands via cancer in the long run. Others imagine a higher death toll—maybe millions—as the necessary price for enlightenment about the dangers of AI.

Against that backdrop, some of the most unsettling AI stories of recent months look like good news. We learned in April about the superhacking abilities of Anthropic’s Mythos, and three months later came the Hugging Face story—about the swarm of OpenAI agents that escaped their “sandbox” and then broke various other rules. Neither of these developments, so far as we know, entailed any deaths, and both significantly expanded awareness of AI risk.
Indeed, in the wake of the Mythos story, President Trump—a skeptic of international governance, to say the least—agreed with Chinese leader Xi Jinping to initiate a formal dialogue on AI safety between the two countries. And the aftermath of the Hugging Face incident saw not only the “pacing the frontier” letter, but a subsequent statement by Dario Amodei embracing—in principle at least—the idea of a slowdown.

Still… there’s a ways to go. It’s great that the phrase “international coordination” is now more common in the context of AI, but even the more optimistic observers have modest expectations about the fruits of US-China dialogue. The wildest commonly expressed hope is for the eventual emergence of something comparable to a US-Soviet arms accord during the Cold War—a narrow agreement about specific capabilities that doesn’t involve very intrusive monitoring and doesn’t much alter the fundamentally adversarial nature of the relationship between the two nations.
That would be progress, but not as much progress as the US and China, and the whole world, are going to need in the long run. This technology is only beginning to demonstrate its power—or, more to the point, its powers. Given the number of dimensions along which its capability will grow, and the novel nature of the policy challenges it will pose, preserving international and intranational stability will demand much more thorough and creative policy collaboration between the US and its superpower rival than materialized during the Cold War.

No issue more clearly highlights the degree of innovation in governance that AI will eventually demand than the issue of “open source” (or, strictly speaking, “open weights”) large language models—models whose internal parameters are publicly available…
[You can read the rest of Part V at https://t.co/eJTqUsTMlA, and you can read the whole essay, from the beginning, at https://t.co/14dwLjajp6]

[Robert Wright @robertwrighter]

@TheTadDaley Why are bilateral agreements unenforceable? If verification of compliance is possible, you can establish incentives that maintain compliance. (A chief incentive being that the other side's continued compliance is conditional on your compliance.)