Chris van Merwijk

AI Alignment researcher. ex @FHIOxford, ex AI PhD student @CarnegieMellon.

[Chris van Merwijk @ChrisvanMerwijk]

@tryfectaa @MatthewJBar I agree

[Chris van Merwijk @ChrisvanMerwijk]

@AISafetyMemes Do u know what newspaper this is from? If its a real photo at all?

[Chris van Merwijk @ChrisvanMerwijk]

@AISafetyMemes Makes sense :)

[Chris van Merwijk @ChrisvanMerwijk]

@xriskology Would this be much worse than if people were tortured until proton decay, except the person who was being tortured was replaced every 80 years with a new person? That would still be possible if we didnt have immortality

[Chris van Merwijk @ChrisvanMerwijk]

Lol all of this gives such a misleading picture.
1. They target smart young people because they have valuable skills, which is fine?
2. I am very monogomous, much more so than even the average person, and I've never been pressured into being polyamorous or anything like that. Yeah there is a much higher % of polyamorous people there but this is analogous to universities being predominantly progressive: if you select for one thing, you get correlated things as well (which then gets reinforced via network effects). It's not some kind of core part of the movement. (Btw I dislike polyamory).
3. "Push group housing"? I've never heard anyone getting punished somehow for not joining. And group housing seems like a reasonable thing to do. I'm sure some people get isolated, but the base rate for isolation/loneliness among college students is high.
4. Sexual harassment is very prevalent in society, especially now that the bar for it has been lowered (which I largely think is good). Most EAs I think just straightforwardly think it's very bad when it happened? I do at least. I think those stories are probably just more memetically fit bc EA has a specific brand and the polyamory angle. I could be wrong about this.
5. Regarding SBF, the statement is true as stated, but it seems pretty important to mention that SBF lied to the people around him and constantly lied about things, and that no one apart from his co-conspirators was aware of or endorsed what he did. EAs bave almost universally rejected it as immoral.
6. "Secretly place operatives throughout the government"??? Wtf. Maybe I missed a news story, but EAs seem prety open about this? And its very normal for social/political/activist movements to try to influence policy by joining think thanks, government agencies.
7. Well, ok EA is also an activist movement, and those do in fact try to do that all the time. It's a very standard and accepted thing to do.

So I agree EA is not "just" a philosophy, in the modern sense of the term, because nowadays "philosophy" is thought to be a purely academic thing. But in history it's been very normal for philosophies to say something about how people should act. I think one of the core parts of EA is to say we should actually take certain ethical responsibilities seriously.

Re. The bed nets. This is another very misleading claim. EA literally started out being mostly about global health (bed nets etc). This was never a pretense. People at OpenPhil and CEA were skeptical of AI risk at first. Then they were slowly convinced by arguments and evidence that the risk is real. And IF the risk is real then obviously we need lots of smart kids to direct their careers into AI safety...

Repeatedly calling it a "cult obsesion" is not evidence that it's wrong. And even if it is wrong, it's still not a cult obsession, because: most EAs are not certain about the danger of misaligned AI. Given uncertainty it's very rational for humanity to allocate the careers of a significant amount of smart young people to it.

I think you wouldn't make this point about "the only thing that matters" if you actually believed like I do that AI poses an x-risk. If so, this is more about an empirical/factual disagreement than a view of EA as a social movement. But maybe I'm wrong about that. (I don't think it's the "only thing that matters" btw. Though I do agree people have said this at times, I'm not sure they'd endorse that literal statement).

[Chris van Merwijk @ChrisvanMerwijk]

From wikipedia:
In the 1970s, with the rise of secular anti-cult movements, scholars (though not the general public) began to abandon the use of the term cult, regarding it as pejorative. By the end of the 1970s, the term cult was largely replaced in academia with the term "new religion" or "new religious movement".

Is that what you mean?

[Chris van Merwijk @ChrisvanMerwijk]

@perrymetzger Can you give a definition of cult that
1. Applies to EA
2. Doesn't apply to basically all new practical philosophical movements that aren't widespread.
3. Implies that it's bad

[Chris van Merwijk @ChrisvanMerwijk]

@perrymetzger Susannah Crockford argues that the least subjective definition of cult refers to a religion or religion-like group "self-consciously building a new form of society", but that the rest of society rejects as unacceptable.

Does that match what you mean?

[Chris van Merwijk @ChrisvanMerwijk]

I asked Claude for the definition of "cult":

1. Excessive devotion to a particular person, idea, or thing, using unethically manipulative techniques of persuasion and control
2. Authoritarian leadership - usually centered around a charismatic leader who claims special knowledge or divine authority
3. Isolation and control - separating members from outside influences, controlling information, and discouraging questioning
4. Exploitation - financial, sexual, or labor exploitation of members for the benefit of leaders
5. Punishment for dissent - harsh consequences for questioning doctrine or leaving the group

MAYBE you could argue "excessive devotion to an idea", namely that people should use their careers to do good. I'd say it's not at all excessive but then I would say that wouldn't I.

The other ones so obviously don't apply I don't need to discuss it.

[Chris van Merwijk @ChrisvanMerwijk]

@perrymetzger If all the major governments wanted, they COULD probably shut down almost all or at least most of the internet. It'd be a bad idea and almost no one wants it so it doesnt happen.

So I'm not really sure what your point is?

[Chris van Merwijk @ChrisvanMerwijk]

It is not at all "inverse". It is a signal that it will probably be even easier to pursuade certain humans than they expected. If anything it is mild evidence in favor of the "superpursuasion" proposition.

But even if you disagree, this doesn't answer my question. My question was:

What is your point? What do you concretely conclude from this?

That superintelligent AI, when it's created (which it hasn't yet) won't be able to do so?

How does that follow?

[Chris van Merwijk @ChrisvanMerwijk]

Most people dont have their attention on these debates. If "Doomer" is the first word they hear that shapes their immediate first impression.

My guess is that reclaiming a slur only works if the slur is already very widespread and people are used to it.

I'd guess it takes too much time for this to have effect, and it's better to call ourselves "anti-superintelligence", or something, and try to frame the "doomers" as unhinged and intellectually dishonest, which it is.

[Chris van Merwijk @ChrisvanMerwijk]

@ronax69 @DKokotajlo @romeovdean By which time 99% of readers are no longer paying attention

[Chris van Merwijk @ChrisvanMerwijk]

Regarding this:
"If humans mostly adopt the belief that having preferences over 'base reality' is old-school and stressful, and locking ourselves in experience machines is fine, it has many consequences: need for resources becomes smaller - in the limit, a brain in a vat can be fed and entertained for about 40W. Human conflict over resources also decreases - everyone can be in their own reality"

I think there is a counter-pressure: The people who decide to live in VR will have less control over the world (bc they care less). The people who don't do so will have a larger share of the influence over the lightcone.

So doesn't this precisely cause evolutionary pressure in the opposite reaction? Going into VR might be analogous to "dying", in the sense of no longer influencing base reality as much.

[Chris van Merwijk @ChrisvanMerwijk]

Btw just to clarify, I am mostly worried about this interpretation being made by the vast majority of people who don't read all the carefully phrased caveats in detail, and make a snap judgment in a few seconds or minutes. So I think the solution is ideally a simple one.

I don't have perfect solutions, but here are some ideas:

1. Release 2030 and 2040 at the same time, in the same publication.
2. Frame the articles as part of a "Possible AI Timelines Series" or something. Make this a central part of the framing when it's published. Retroactively fit AI 2027 into it. Make it so clear that a single image in an https://t.co/rQaLRHEOgj post makes the intention obvious to someone who only looks at that image. I think "AI Futures Project" doesn't convey it as directly so it'd be nice if there's also a name for the series itself. But maybe there's something better than "Possible AI Timelines Series".
3. Don't put a caveat in the form of a long-winded sentence in the third paragraph. Instead write a snappy tagline below the "AI 2030" headline that frames it clearly from the beginning. E.g. something like "One possible timeline", or "Our Probability of superintelligence before 2030: XX%". (or better: XX to YY%)
4. Or maybe a slightly longer but still simple sentence directly below the headline (in a way that is easily visibly screenshottable for in an https://t.co/awFkkuDH16 post) that just directly contradicts the interpretation: "No one can be confident when superintelligence will arrive. We are thinking through the possible timelines."

---
But basically what would be nice is: If someone (possibly intellectually dishonest) adversarially posts something on https://t.co/awFkkuDH16 like that hypothetical reply I wrote, it'd be nice if it's possible to respond with a single screenshot of the AI 2027/2030/2040 article headline that makes that poster look stupid (ideally even to people who don't think probabilistically).

Currently with AI 2027, that's not really the case. E.g. there is this caveat:
"We disagree somewhat amongst ourselves about AI timelines; our median AGI arrival date is somewhat longer than what this scenario depicts. This scenario depicts something like our mode."

But it's long-winded/complex AND it's hidden in a footnote.

[Chris van Merwijk @ChrisvanMerwijk]

@DKokotajlo @romeovdean I agree it's laughable, but I don't trust the rationality of our adversaries. Let me think about your question.

[Chris van Merwijk @ChrisvanMerwijk]

@RokoMijic @DKokotajlo @romeovdean Roko just straight up advertising his intellectual dishonesty