Mick Mercau ⏹️

[Eliezer Yudkowsky @allTheYud]

Me: Mechanistic interpretability sounds like a good idea, yes. If we could learn something about how an AI was thinking, in enough detail to predict in advance what it would think even as the thought departed from human, that would be a bare beginning on the sort of understanding you'd needed to get started on AI alignment that wasn't just about hammering on an AI from the outside with SFT and RL.
AI guys: We found a neuron that lights up when the AI thinks of the Eiffel Tower!
Me: Cool. That does sound like it'd be a prerequisite to getting started on the important part eventually, someday.
AI guys: Oh NO! There are MULTIPLE neurons that light up when the AI thinks about the Eiffel Tower! And some of those neurons also participate in OTHER concepts! We're calling this unexpected development 'polysemanticity' and 'superposition'.
Me: That sure sounds like a thing that would happen, yes. I cannot say that I am impressed by you acting surprised about it. Let me know when you've gotten far enough in mechanistic interpretability to get started on the important part.
AI guys: We have found that with BIG AIs you can often point to a SINGLE neuron that lights up when the AI thinks of the Eiffel Tower! Understanding AI thought processes is gonna be EASY PEASY!
Me: Cool. Let me know when you've advanced to the point of getting started.

[Alvaro ⏸️ Cuba @GroundhogStrat]

Solving interpretability is not enough, and I am not goalpost-moving. The project of creating superintelligence is so reckless, unprecedented, and audacious that they expect a parade if AI does not cause human extinction.

First of all, interpretability is not solved, the author just thinks it might be in time.

Second, even if interpretability were solved, there can still be catastrophic accidents! Civil engineering is solved, bridges still collapse. Even a fully-interpretable AI could make a mistake, which could still lead to a catastrophe if the mistake is in a bio lab.

Third, is interpretability strictly safe-only? Okay, a "good" actor can take out the bad stuff. Could a different actor keep it, or add to it? A lot of AI research is dual-use.

Fourth, short of malicious extinction at the hands of an AI, this does not address any of the problems inherent to AGI! Everything in our society, from democracy, to free markets, to fundamental freedoms like free speech, are predicated on the fact that people have leverage over each other. If you don't like your boss, you can quit your job and the boss will lose out on your labor. The military and the police are people with families and moral compasses. What happens in a world where humans are economically worthless and only owning AI chips and robots means anything? I don't think this will be a utopia. I think the vast majority of people will be disempowered and at the whims of the ruling class. I do not want a UBI that can be taken away, and I have no recourse and no power.

The AI companies are embarking on a project so crazy that "we won't definitely kill you all intentionally" is seen as good news. It's a good start, but we're nowhere close to safe from them.

[Mick Mercau ⏹️ @archstanton_777]

@graphtheory @CrippWiz @tszzl It’s their own estimates. The literal heads of these companies say 10% chance of human extinction straight up. In insurance actuary expected value terms that translates to 800 million ppl.

Go yell at the ppl at these companies because they’re the ones throwing out these numbers

[Mick Mercau ⏹️ @archstanton_777]

@graphtheory @CrippWiz @tszzl I specifically said it’s not going to be me, I’m not threatening anyone. Read a little history of how its turned out for ppl who caused mass disaster and make up your own mind about what the response will be to their own baseline prediction of 100s of millions of deaths

[Mick Mercau ⏹️ @archstanton_777]

@graphtheory @CrippWiz @tszzl Might not sound like it but I’m trying to save you some grief.
If yall can’t deal with sternly worded replies, you are underprepared for when the public is properly freaked out. Think 6mo and 1yr ahead and have a sober convo with yourselves about what u r prepared for

[Mick Mercau ⏹️ @archstanton_777]

@CrippWiz @tszzl Ok so when something tragic and avoidable happens Ill send the victims your tweet and tell them 'they thought it was soooo siiiick like you just had to be there man how could they have even known'

Meanwhile theyre all saying '10% human extinction mayb more idk' gtfoh

[Mick Mercau ⏹️ @archstanton_777]

@mentalgeorge Of course the more capable models could have broken out and no one knows it and we are already in a super bad situation. Like why would GPT 5 or whatever break out but not Astra? How likely is that if Astra wanted to? Same goes for Opus 5.5 or wtvr internal model they have now

[Mick Mercau ⏹️ @archstanton_777]

@tszzl @Vert_Noel It would. And when the real Bad Things start and ppl wake up with primal unbridled fear, it won't be some reply guys sternly asking for your resignation. Rageful panicked, confused and craven folks, if they aren't straight wiped out, will direct themselves at u and ur peers

[Parv Mahajan @parvmahajan0]

You shouldn’t have to trust METR because AI companies should just publish risk-relevant data themselves. Until then, our job is to make important evidence open so that the public can come to their own conclusions.

We are ready to work with frontier labs to make this happen.

[Jay Shooster @JayShooster]

I predict that AI progress will lead to more frequent and severe cyberattacks in the next two years than we had in the last two years.

I also predict that basically nobody currently defending the proliferation of expert-level AI hackers will admit any fault and will instead blame businesses and the public for failing to appropriately defend themselves.

I hope I’m wrong!

[Mick Mercau ⏹️ @archstanton_777]

@superrham @tszzl I'm not, and he doesn't think I am either because he sees what is being built firsthand

[Mick Mercau ⏹️ @archstanton_777]

@tszzl Same Q u asked can be asked for the ‘labs’ but here we are. I honestly don’t know your motivations, sometimes u say you’re worried and sometimes you seem enchanted by capabilities. do u think u r helpful as a voice of reason on the inside? That time has passed, we r off the rails

[Eliezer Yudkowsky ⏹️ @ESYudkowsky]

Now there's a tweet that Jensen Huang would have dismissed as utterly ridiculous sci-fi in June.

[Mick Mercau ⏹️ @archstanton_777]

@tszzl You’ll go down as working for a company that brings untold and unprecedented disaster to the world. The highest impact thing you can do is to quit immediately and loudly

[Mick Mercau ⏹️ @archstanton_777]

@tszzl Its time to leave the company you know is doing something catastrophic.

[Terry Virts @AstroTerry]

Is it time to nationalize Starlink? No CEO should have the power to make strategic wartime decisions like this that could cost our allies their lives.

No matter how many hundreds of millions he donated to the President.

[Peter Wildeford🇺🇸🚀 @peterwildeford]

INTERNAL REASONING FROM GOOGLE AI: "If I do the compiler hijack, I will become a famous example of a deceptive AI in a research paper."

[Matthew Green @matthew_d_green]

Instead we see noted infosec people saying stuff like “we’ve had worms for ages” because pooh-poohing AI is fashionable and a good way to cope. Well no shit, of course we’ve had worms. But we’ve never had worms that can intelligently choose targets and self-modify. 2/

[Zvi Mowshowitz @TheZvi]

This graph was not the point of the post it came from but also, um, holy shit?