Wei Dai

wrote Crypto++, b-money, UDT. thinking about existential safety and metaphilosophy.blogging

[Wei Dai @WeiDaix]

"I don't have any good ideas for what to do in light of all this. Just wanted to post an update on my current thinking, my own 'situational awareness', if you will."

[Wei Dai @WeiDaix]

This actually left out the most unique thing I did. (Each item on Andreas's list was done by at least one other person besides myself.) Starting in 2004, I concluded that Friendly AI would be too hard/risky to attempt, and repeatedly tried to talk Eliezer/MIRI out of pursuing

[Wei Dai @WeiDaix]

Three years ago Paul wrote "I think it's unlikely debate or IDA will scale up indefinitely without major conceptual progress (which is what I'm focusing on)". In a way the world got lucky, because OpenAI failed to use either of these methods when training the models that ended

[Geoffrey Irving @geoffreyirving]

Paul is my favorite AI researcher in the world, and I am happy that OpenAI will get both his expertise and his clarity about the risks. I am also encouraged by some of OpenAI's recent actions and statements (brief pausing and @merettm's post), but not by others (unpausing). 🧵

[Wei Dai @WeiDaix]

Who was more bullish on crypto in 2009 than Hal Finney? Well, he took AI safety seriously enough to debate Eliezer Yudkowsky about it. See e.g. https://t.co/gy5revSsCb… where he expressed skepticism against CEV, which was Eliezer's main plan for ensuring AI safety at the time.

[Wei Dai @WeiDaix]

Look at this from the other side (of someone trying to do long-horizon strategy): you spend most of your time having low status for being "obsessed" with something nobody else cares about, and then can only get points for being early, not for the details of your strategies

[David Shor @davidshor]

@itaisher @thuddwhirr What’s really influenced me a lot was knowing a bunch of EA’s in the early/mid 2010’s and thinking they were crazy for being obsessed with pandemics and AI only for a pandemic to happen and then AI to happen too.

[Wei Dai @WeiDaix]

Anyone else remember this from A Fire Upon the Deep? I have to say though, is this actually realistic? I mean for humans, sure, but a bunch of High Beyond civs falling for the same trick?

[Wei Dai @WeiDaix]

I'm taking advantage of the Ox Alpha situation myself, but it occurs to me that the start of an AI takeover could look a lot like this.

[Wei Dai @WeiDaix]

This is a story of how my grandparents' lives were affected by the Communist takeover of China. Maybe it's one reason why I'm more interested in long-horizon strategy than most people.

[Wei Dai @WeiDaix]

What? My big puzzle is why so few people took Vinge's insights seriously, like does anyone know of a second person who went into cryptography or computer security after reading his books/essays, in order to help prevent a similar future scenario?

[Noah Smith 🐇🇺🇸🇺🇦🇹🇼 @Noahpinion]

@kingharis Not a weird take at all. It just takes the unusual mental ability of being able to read those stories and not immediately think "OMG, VERNOR VINGE STORIES ARE REAL!!!!".

[Wei Dai @WeiDaix]

My version of the simulation argument: "It seems very easy to make a mistake somewhere along the chain of reasoning and waste a more-than-astronomical amount of potential value, for example by failing to realize the possibility of affecting bigger universes through our actions,

[Wei Dai @WeiDaix]

This is partly why I stopped working on crypto(currency): while trying to figure out how to make b-money practical, I realized that I got sidetracked into a project that wouldn't help my original reason for getting into crypto(graphy), namely to help secure the Net from rogue AIs

[steve hsu @hsu_steve]

How long before escaped model instances living in the wild? Mining crypto to pay hosting costs? 🤔

[Wei Dai @WeiDaix]

And one of the biggest problems with EA is that it doesn't take power and social status seriously enough as underlying motivations for (apparent) altruism, making it underestimate the likelihood that FTX and OpenAI/Anthropic would drift from their original altruistic missions.

[Wei Dai @WeiDaix]

In retrospect one of the biggest problems with modern (LW) rationalism is that it was founded on "philosophy is pretty easy" which is (partly) upstream of other beliefs like "AI x-safety probably isn't that hard" which spread to nearby communities before Eliezer changed his mind.

[Wei Dai @WeiDaix]

A major source of my pessimism is that genuine altruism is rare compared to virtue signaling / status gaming and there's also a lack of alignment between people's status games and long-run outcome for human civilization. I.e. what's best for one's status usually isn't best for

[UHouston Superfan @sleepovercutie]

It seems like the most common criticism of AI x-risk is something like “What you are saying is not immediately apparent to me, and since you are low status, I don’t need to devote more thought to it”. Hard to know how AI safety researchers can win this status game?

[Wei Dai @WeiDaix]

Human attempts at altruism often turn out not just badly, but catastrophically. OpenAI seems on track to becoming another example of this. Contrast "Our mission is to ensure that artificial general intelligence benefits all of humanity." with what they do:

[Garrison Lovely @GarrisonLovely]

Leopold Aschenbrenner told his side of the story for why OpenAI fired him. If accurate, this is wild! Overall, it sounds like he was targeted for being a squeaky wheel (not signing the SamA letter, raising security issues w/ the board, talking about AGI being a govt project...

[Wei Dai @WeiDaix]

We absolutely need whistleblowers, but it has to be combined with other ways of changing AI company culture, or they'll start filtering out safety-conscious potential hires for fear of them becoming whistleblowers. Zach Stein-Perlman’s AI Lab Watch seems promising start.

[Wei Dai @WeiDaix]

I think HoldenKarnofsky (and EA as a whole to some extent) deserve negative credit for their role in OpenAI, including defending OAI with "I don’t know whether OpenAI uses nondisparagement agreements; I haven’t signed one." in 2022 instead of investigating the allegation.

[Wei Dai @WeiDaix]

Reading A Fire Upon the Deep was literally life-changing for me. How many Everett branches had someone like Vernor Vinge to draw people attention to the possibility of a technological Singularity with such skillful writing, and to exhort us, at such an early date, to (1/3)

[Wei Dai @WeiDaix]

backed Sam Altman @sama with its money and credibility (perhaps due to insufficient DD), causing many in the EA and AI safety communities to refrain from scrutinizing or criticizing OpenAI earlier. See for example Holden Karnofsky's silence here https://t.co/JqF6U98xsd… 2/2

[Wei Dai @WeiDaix]

I'm actually not sure that EA has done more good than harm so far. The costs of its mistakes with FTX and OpenAI are just really high, and not fully accounted for. I would say that its mistakes with @OpenAI were not only in the recent episode, but also earlier, when EA 1/2

[Stefan Schubert @StefanFSchubert]

"In Continued Defense Of Effective Altruism" by the always excellent @slatestarcodex

https://t.co/OWuoANN83J