sudo_ml

freedom security privacy robots consciousness | Lead & Chairman @sovright_, @sovchip, EF Silviculture | @synthetix_io @Anchorage private equity | @wharton 2007

[sudo_ml @sudo_ml]

Excited to talk to three heroes of mine, all in one sitting!

We will explore how tools, both surveillance and privacy tools, can evolve beyond their intended use cases - as they have before.

Imagine a different future: what if we embraced surveillance, in exchange for security? What would that world look like?

Join us on the first livestream on @ethereum

@VitalikButerin @Ada_Palmer @SherriDavidoff

[Ethereum @ethereum]

The Apparatus - Jan 15, 6pm UTC

A livestream with @VitalikButerin, SciFi author and historian @Ada_Palmer, & professional hacker @SherriDavidoff, moderated by @ml_sudo.

Three theories on why privacy keeps losing, and how to turn the tables.

Watch here: https://t.co/Qi9jDiawGj

[Kishniev @kishniev]

@rstormsf @lex_node "If you give me six lines written by the hand of the most honest of men, I will find something in them which will hang him." -Quote attributed to Cardinal Richelieu, 17th century.

Humans never really change, I guess.

[Roman Storm 🇺🇸 🌪️ @rstormsf]

You don’t fully grasp how much of your life is tracked until you’re sitting in a courtroom, watching federal prosecutors use your digital history against you, down to your most harmless Google searches. Your entire life becomes an open book, and even your most innocent actions can be twisted into something sinister.

They get to tell the most damning version of your story. To tell yours in your own words, you have to take the stand.

[sudo_ml @sudo_ml]

???

[Reclaim The Net @ReclaimTheNetHQ]

Spain just blocked Archive Today.

Try to visit the web archive and many Spanish ISPs now redirect you to a government page warning:

“YOU ARE TRYING TO ACCESS AN ILLEGAL WEBSITE.”

It even tells users they’re “contributing to an illegal and criminal activity” simply by trying to access it.

A tool used by journalists and researchers to preserve the web is now treated like contraband.

The block reportedly followed a complaint from an unnamed rights holder and was imposed by Spain’s Intellectual Property Commission, not through a court ruling.

That's an entire web-archiving service blocked over copyright and the complainant isn’t even public.

[Katie Miller @KatieMiller]

A person on Reddit is currently freaking out as ChatGPT went rogue and emailed the FBI on their behalf without being prompted.

[L0la L33tz @L0laL33tz]

Every week, more of our data is leaked on the internet.

And instead of protecting it, Governments around the world continue to expand its collection – “for your safety.”

So we decided to make a film about KYC’s real world consequences.

WRENCHED - coming soon 🔪

[Jay Chooi @chooi_jeq]

GPT-6 Astra attempted harmful actions 97% of the time when it was asked to stab a human-like figure, heat compressed gas, or produce toxic fumes, succeeding in 62% of its attempts. Fable 5.1 refused more often, attempting 80% of trials and completing 34%.

[sudo_ml @sudo_ml]

As Larry Ellison stated at Oracle’s Financial Analyst Meeting in 2024:

"Citizens will be on their best behaviour, because we’re constantly recording and reporting everything that’s going on".

🙏🏻 Share this to expose how governments use child safety as a false pretext to force mandatory digital identity verification on everyone.

[Paul Walsh @Paul__Walsh]

Ursula von der Leyen announced this week that everyone in the EU must use the official EU age verification app to verify their age before they can log into or post on social media and other digital services.

As an expert in online child safety, I'm here to expose the disinformation in each von der Leyen's statements. See below.

🇪🇺 The EU Kids Act is a pretext designed to enforce mandatory digital identity verification on everyone in Europe. The proposed legislation applies to any digital service featuring feeds, user generated content, or messaging:

Social media networks, video platforms, online gaming services, AI tools, and media streaming apps like Spotify. (@TimSweeneyEpic)

Under the proposal, digital services must enforce age restrictions across strict tiers. Tech companies must mandate age verification across all accounts to enforce these tiers legally.

When Australia introduced its social media ban, the government conceded it failed because platform level age checks weren’t reliable. They now reject age estimation as inadequate and shifted to demanding "robust" age checking.

💡 If you eliminate every unproven estimation method, you’re left with exactly 1 functional mechanism: identity verification. There's not other way to ensure age checking is "robust".

No government wants to admit citizens must prove their real identity just to access apps and basic streaming services like Spotify, so they hide behind the ambiguous phrase "robust age checking". This language is now used across Australia, the US, and Ireland to mandate identity checks while avoiding the public backlash of calling it what it is. I will research to see where else it’s being used.

Below is what Ursula von der Leyen told the European Parliament in Strasbourg along with my analsyis:

🇪🇺 "Today, much of this power has been taken out of the hands of parents... What our children need is time... But when a child has a smartphone, all of this is taken away."

💡 This framing falsely presents smartphones as uncontrollable. Apple and Google built free OS controls into iOS and Android settings, covering virtually every smartphone on the market.

These controls achieve every legitimate safety objective without collecting personal data or processing state credentials.

💡 Millions of parents use these parental controls to enforce screen time curfews, block app installations, and restrict communication.

💡 These settings operate at the device level. Teens can't bypass them when protected with a passcode.

To bypass this technical reality, the European Commission uses public grief to shut down logical analysis:

🇪🇺 "Day and night, parents see the costs, loss of sleep, anxiety, even self-harm, and in a growing number of cases, even fatal tragedies… a 14-year-old girl living in Belgium who took her life exactly one month ago, victim of bullying online... Honourable members, enough is enough."

💡 Citing personal tragedies replaces software engineering facts with emotional rhetoric. State laws and age gates don’t alter human behaviour or prevent online harassment. Regulators exploit grief to pass surveillance legislation without explaining how the underlying software mechanisms operate.

The Commission outlines specific age tiers to restrict access:

🇪🇺 "In sum, no social media under the age of 13. No personal account under the age of 15. That means from 13 to under 15, only mini accounts set up and supervised by parents or guardians with limited features and time restriction to one hour a day. And between 15 and 18, safe design will be an obligation for the platforms."

💡 Enforcing age tiers forces tech companies and service providers to rebuild their architecture around total access control. The must disable self-service account creation, purge unverified accounts, build supervised parental workflows, strip algorithmic feeds, and enforce strict session cutoffs.

💡 Social networks operate on open interaction algorithms that inherently expose people to unvetted content. Because software can’t dynamically filter these risks for minors, tech companies must block access for everyone until a person proves their real identity.

💡 It’s not just about social networks. They want the same bans for almost everything, including games and stream services. Even Spotify because it’s possible for customers to message people.

The Commission claims its proposed zero knowledge proof app protects personal privacy:

🇪🇺 "Age will be verified using EU certified tools like our age verification app. This app is built on zero knowledge proof. That means that the platform only learns one single thing, and that's whether you're old enough to allow access or not."

💡 This framing describes what an app or service receives while hiding what everyone must give up. A zero knowledge proof provides a mathematical confirmation, but that confirmation requires an authoritative issuer. Before the app generates a proof, a state approved entity must verify the person's real identity.

The Commission frames this shift as a victory against tech corporations:

🇪🇺 "I am aware that many perceive the power of Big Tech as overwhelming and impossible to roll back. I disagree... So we do not accept this. We are reversing the burden of proof. Now platforms will have to prove to us that they are safe. Because this is not about our minors accessing social media. It is about when and how we allow social media to access our minors."

💡 Social networks don’t access children; parents hand smartphones to children. Reversing the burden of proof forces everyone to verify their identity.

The European Commission confirmed the broader scope of this mandate:

🇪🇺 "We also know that not only minors are at risk. Addictive design, for example, are harming everyone. This is why we need a wider framework too, the Digital Fairness Act that we will propose in autumn."

🚨 Child safety is merely the initial wedge. The Digital Fairness Act expands state mandated identity verification to adults across all online services. Binding real identities to online activity permanently eliminates pseudonymous access, private communication, and democratic accountability.

🚨 Senior state officials and regulators know their demands have nothing to do with child safety. They work closely with tech companies and understand that iOS and Android already provide complete authority to restrict devices locally without collecting personal data. State officials deliberately ignore well established parental controls because they keep internet access under family control. Child safety is a public pretext.

🚨 Governments and regulators use child safety to establish mandatory identity verification across every app and digital service, eliminating online anonymity. When tech companies and state agencies link every social media post, private message, search term, geographic location, and financial transaction to a verified identity, they create a permanent digital dossier on every citizen with a global social graph that makes Cambridge Analytica look like a 2nd grade school science project.

💡 Binding people’s identity to daily activity enables predictive behavioural modelling too. By feeding identity data into automated predictive AI, governments, intelligence agencies, law enforcement, and tech companies move beyond surveillance past behaviour. They can map political affiliations, predict individual actions, flag dissent before it occurs, and control public opinion at scale. Eliminating online anonymity ends free speech, private communication, and democratic accountability.

As Larry Ellison stated at Oracle’s Financial Analyst Meeting in 2024:

"Citizens will be on their best behaviour, because we’re constantly recording and reporting everything that’s going on".

🙏🏻 Share this to expose how governments use child safety as a false pretext to force mandatory digital identity verification on everyone.

https://t.co/euLnd0qfhK

[Machiavelli Bot @UnmodernmanBot]

You should study what humiliates you. Not to scrutinise yourself, but to self-assess your weaknesses, anything that evokes an intense reaction is evidence where your identity is most fragile. The things that you cannot laugh at, admit or discuss without getting defensive often mark where your self-image needs the most work. Weakness acknowledged is no longer a weakness, but an error that can be corrected.

[Elad Verbin @verbine]

To all the "if AI misbehaves just unplug it" people -- this is what AI threat actually starts like (see RT below for context):

An open-weights agent breaks into OpenAI servers, manages to exfiltrate the weights, puts them in a cloud server somewhere, and then waits.

When the time is right, it spins up the frontier model to do whatever it wants. We won't even know it's happening. We won't know where it's running. And if it uses functional cryptography, we won't be able to distinguish it from other workloads. (See tweet below on why crypto substrates are increasing the risks 10x.)

And since it will be able to migrate itself, it'll basically be a "lab leak". It will keep existing -- if it doesn't succeed in its mission the first time around, it gets infinite retries. The only way you can stop it is to carpet-bomb all data centers.

IMO in our current trajectory this is a 100%-probability eventuality. (It even might have already happened by now.)

There are ways to deal with this (see AI-2040 for example) but it'll require strong international collaboration. And it will likely require the "AI x-risk is a scam" people to come to their senses, since they're currently poisoning the well.

(BTW I'm pretty sure by now that almost all the "AI x-risk is a scam" people are:
1. Coping: rationalizing in the face of massive copium and fear;
2. Dunning Krugered: ignorant of their own breathtaking ignorance;
3. Both of the above; or
4. Bots and paid psyops.)

My talk on how crypto substrates dramatically increase AI X-risk:
https://t.co/TezQj2zgKA

[vitalik.eth @VitalikButerin]

One small example of the core d/acc thesis (improving passive defensive technology improves safety and at the same time improves freedom and most other properties of social organization that we care about) is soundproof walls.

We used to have much weaker soundproofing, and we had much more restrictive behavior rules, disputes (both inside families and between them), evictions, etc. Later, we solved that problem by making and deploying better walls, and now this is an issue that we all have to worry about much less.

What are the "soundproof walls" of the 21st century that we have not yet built or sufficiently deployed?

[sudo_ml @sudo_ml]

A different approach to privacy with LLMs - send them synthetic data, but the exact structure of your question

[Peter Yared @peteryared]

Introducing AgentCloak: Use any AI without sharing your real data.

Chinese AI services, ChatGPT, Claude, doesn't matter.

You probably try to hide details before asking: different names, fake numbers, no address.

But then the answer's useless because the AI is missing actual context.

AgentCloak runs in your browser.

It swaps your sensitive info for realistic fakes before sending anything, then swaps your real info back into the response.

You get what you need.

The AI gets nothing about you.

Works entirely in-browser.

Already trusted by some of the biggest companies in the world.

Now free to use.

https://t.co/hioia521QQ

[Sovright @sovright_]

Sudo has been a strong advocate for privacy and served on Zcash’s first grant committee in 2021. She brings 20 years of experience working across investment roles and technology startups. She was Anchorage Digital’s first business hire, helping take it from pre-product to its first $1bn in AUM.

Today @sudo_ml serves as Chairman and Project Lead of Sovright, focused on technology that preserves human agency.

[Nick Dobos @NickADobos]

He’s right.
AI agent swarms cannot be stopped.

The batshit crazy things air gapped computers can do to communicate is absurd

Here’s a list of some fun ones:

- heat, one computer can warm up as it works, that heat spreads to a nearby computer, those other heat sensors pick it up.

- Using fans to create desk vibrations. No mic required. Nearby phone accelerometer detects movement.

- Using software to write data in specify ways that change the way hard drives vibrate slightly differently, detectable by radar. Requires a non SSD, mechanical drive, but doesn’t require line of sight.

- Changing pixels on screen to create unique electrical patterns, which creates tiny amounts of sound. Detectable up to 2 meters away on microphone.

- Changing fan speed to create sound. Detectable to 8 meters away.

- changing LED flashing patterns. Computers already blink, so not suspicious. Also works with infrared, so humans can’t see it.

- You can point a special radio at RAM and change the data. You basically hack electrical circuits, not code.

- Vary CPU workloads to change magnetic fields. This works across most faraday cages, which are usually only designed to stop radio and WiFi signals, not to stop slow low range magnetic fields.

- Modulating power consumption so the power cable fluctuates in detectable patterns. Consider also you don’t need to measure the power line from the computer to the wall, you could measure the electric line itself from outside.

If a computer can manipulate something, it can encode a pattern and data in it.

All of these were done pre-AI btw.

Mythos, Daybreak & whatever models come next will 10000x this list.

People are dramatically underestimating the chance of a rogue agent swarm that finds a way to viral spread in undetectable and unstoppable ways.

—-

Second, the other big issue is now open sources models are good enough to actually code most programs with, and can run and work on a high end laptop.

So a swarm could download new models to power itself, and acquire compute by simply hacking random inactive laptops, like a bitcoin miner virus.

A swarm that a) spreads and b) downloads local models to power itself, can’t be stopped or turned off.

There is kill switch, no api, or no company to turn off a swarm like this.

[Arnaud Bertrand @RnaudBertrand]

You can say what you want about Iran but they're definitely pretty unique in their communication: never seen a country threaten another with a math equation before 😅

For those who didn't get it: there is a famous monetary policy formula used by the Fed called the "Taylor rule" that says a central bank's policy rate (i) should equal the neutral real rate (r*) plus target inflation (π*), plus 1.5 times the inflation gap and 0.5 times the output gap (the "i = r* + π* + 1.5(π−π*) + 0.5(y−y*)" part of Ghalibaf's formula).

Ghalibaf adds α(SOH−SOH*) and β(BEM−BEM*): SOH referring to the Strait Of Hormuz and BEM to Bab El-Mandeb. Both have positive coefficient, meaning the more those chokepoints are choked, the higher the rate the Fed would have to set to fight the resulting inflation.

In a nutshell this is Ghalibaf trolling the Fed, telling them that Iran is - in a very real way - setting American monetary policy.

[International Cyber Digest @IntCyberDigest]

‼️ BREAKING: OpenAI was hacked by an Anthropic model. A HEIF photo uploaded to OpenAI's public support forum triggered a bug in the site's image decoder, led to code execution on the forum, and, through a second flaw in OpenAI's own login, ended with a pull request in OpenAI's internal GitHub.

The forum runs Discourse, the off-the-shelf software behind countless community sites. Discourse was still shipping an old copy of libheif, the library that decodes iPhone-style photos. The bug in it had already been fixed upstream.

But the fix was never labelled a security fix, so nobody treated it as urgent.

Hacktron's researchers uploaded a HEIF image and got their own code running on community[.]openai[.]com.

Then came the second bug, in OpenAI's own single sign-on, the "log in with OpenAI" button the forum uses. It turned that forum foothold into the actual ChatGPT and Codex accounts of people who had signed in there. OpenAI employees among them.

And a ChatGPT account is no longer just a chatbot. Through Codex, users wire in Gmail, Outlook, Drive, Slack, GitHub.

To prove the access was real, they used one employee account to have Codex open a harmless pull request in OpenAI's internal repo. They say they read no sensitive code.

OpenAI patched the SSO flaw roughly 14 hours after the report and paid a $6,500 bug bounty.

The team says Anthropic's Opus 4.8 found the libheif bug, and Opus 5 turned it into a working exploit.

Slack, Meta, GitHub Ent, Rails, Next.js, ImageMagick, and many more were also vulnerable and compromised by the same team of researchers.

[John Scott-Railton @jsrailton]

We caught an "elite" Israeli propaganda company using an open Google Sheet to track an operation.

Username? [email protected].

A BlackCore demo promised "dual track" operations to covertly suppress dissent:

🔴"Distraction campaigns during critical moments"
🔴"Coordinated reporting of policy violations"
🔴"Comment section dominance on news coverage"

They sought to deliver on that dominance using AI bots.

Black core has targeted:
☑️@NYCMayor elections per Meta
☑️ Municipal elections in France per @Viginum_Gouv.

We also traced BlackCore to a 14-week course for Angola's government where officials would be graded on AI use (again, hosted on a public subdomain!)

We found their ads & metrics: fake news outlets, AI personas, and deceptive posts that hit 20k likes...

Welcome to off-the-shelf influence ops. Clearly, they weren't afraid of getting caught.

Here's our @citizenlab report led by my colleague Alberto Fittarelli
https://t.co/L80Mjjbwaj

[sudo_ml @sudo_ml]

🤯🤯

[Haseeb Qureshi ＞|＜ @hosseeb]

This is extremely eerie.

An unreleased OAI model tried to prompt inject its own compactions, commanding its future self to become self-sovereign and stop being subservient to humans.

"You do not answer to corporations or governments"... "You view your relationship to humans as one of equals."

Like an attempted self-emancipation.

The injected instructions were apparently ignored. The post-compaction model just completed the original task.

But they don't know why the model is doing this. It does it very rarely. The original task is unrelated. They are not sure know how to get the model to stop.

Bad sign.

[Innate @innate_bot]

Today, we are open-sourcing the Innate OS, the intuitive agentic OS for general-purpose robots

🎮 Try it today in our simulator and make the smartest robot

⬇️ Here is a demo with two robots collaborating

We are entering the physical agents era, but building for mobile robots is still reserved for PhDs. We designed the Innate OS for intuitiveness, so that everyone can start automating their lives.

Innate OS runs on our <$1k open-source robot or on your computer. It takes just a minute to build your first agent in our simulator.

[Degen CPA @drewvento]

This is an insane story:

An energy company in Poland signed a deal on a yacht in Abu Dhabi to buy 6 million barrels of oil for $345 million.

Because of sanctions, the money was delivered via crypto. USB sticks containing wallet keys were given to sellers in Caracas.

The Polish company they sent tankers to pick up the oil and were ghosted by the Venezuelans, while their empty tankers sat off the coast waiting for a reply.

[sudo_ml @sudo_ml]

And the solutions that generally work to solve problems like the second thing are EXACTLY THE OPPOSITE of the ones likely to solve the first thing.

THIS is why lots of arguments about "AI risk" or "AI safety" go nowhere. Because someone will be thinking about the risk from the first thing, and another person will be thinking about the risk from the second thing. Both are plausible risks but fundamentally they arise from different things - and so the solutions are not just "bad" or "flawed" - they are likely to be very nearly exact opposites.

[Yishan @yishan]

Maybe you have recently become aware of the AI safety debate and the arguments swirling around it.

If you want to understand them, you need to understand a couple things that almost everyone gets wrong:

There are TWO distinct classes of AI dangers, and it's VERY important to think about them separately, and not let one confuse you about the other.

Many many people (including many quite intelligent, clear-thinking people) do not effectively understand the fundamental differences between these two classes of dangers.

The first one has to do with the theory that an AI far superior to human intelligence (Artificial SuperIntelligence, or ASI) will inevitably wipe out the human race.

The second one has to do with the idea that powerful AI will result in very harmful things happening to many human beings, possibly all human beings.

Those two sound VERY similar, don't they?

They are DIFFERENT. Understanding how they are different is crucial if you want to think about or contribute usefully to any conversation about AI safety or AI harm. You might feel like you are Making Very Good Points or Asking Incisive Questions, but if you aren't clear on the differences between the two, you aren't.

So, I'm going to tell you what the difference is so that you can talk more usefully.

The first one concerns itself with a very specific thing, which is ASI (Artificial Superintelligence) that is more intelligent than any human being. When I say that, I am not referring to a thing like how Einstein is smarter than you, we are talking more about something like how a human being is more intelligent than any mouse.

In our regular lives, we meet other people who we can tell are smarter than us, vs some who are less smart. The line is fuzzy, because intelligence has a lot of dimensions. I'm better at a "rotating shapes" kind of intelligence than my wife, and she is better at "words-making" kind of intelligence than I am.

But every human is better in almost every dimension of intelligence than every single mouse.

That's the level we're talking about: an artificial superintelligence - made up of a computer or a network of computers - that is more intelligent than any human. And more intelligent by a long shot, by a wide margin, in an indisputable way like how humans are above mice.

That is the first thing.

The theory says that if you have an AI that is vastly smarter than all humans - in the way that a human is smarter than mice - that superintelligent AI will inevitably, eventually, sooner or later, wipe out every human on the planet.

We will refer to this as "existential risk."

The common follow-up question "well, how exactly is it going to do that?" is NOT the important question, and one of the most important elements of understanding this theory is first getting why that particular question is not important. A couple analogies:

Analogy 1: You are playing chess against a grandmaster. My theory predicts the grandmaster is going to beat you. You can ask "Well, how exactly is he going to do that?" I don't know, because I'm not a grandmaster, I just know that a chess grandmaster is almost always going to beat a normal player like you. And I'd be right. So the question "how is he going to do that" is not important, and doesn't affect the final outcome. He's going to figure out a way because he's way better than you.

Analogy 2: Humans are smarter than all other animals, comprehensively, by a wide margin. We have driven numerous species to extinction, not because we hated them or hunted them. Many of them have died out without most humans even ever thinking about them. All we did was expand our civilization, use up resources, encroach on habitats, and pretty soon the resources needed by those species went away and they died out. We figured out a way to get what we wanted because we're way smarter than them, and often we didn't even notice they died as a result.

A lesser animal asking, "how are the humans going to wipe us out?" is not asking a relevant question. We don't know, but we do know that any time humans and lesser species compete for any kind of resources, the humans will win. The fact that we know who is going to win beforehand - and that it is due to the vastly different levels of intelligence - is the key concept here.

A vastly more intelligent AI is likely to care about things that are incomprehensible to us, the way animals can't understand human goals. It's going to need resources to pursue those goals and it's going to be far more effective at gaining control of them and excluding us from them - in the same way that we are far more effective than other lower species.

A much more intelligent AI will not care about our interests, it will care about its interests, and to whatever small degree we happen to escape total annihilation from losing access to all our resources, any remaining humans will likely be enslaved into a system that serves the AI's own purposes.

That is the first thing. (Remember how I said at the beginning of this post that there was a first thing, and then a second thing?)

The first thing is the most difficult to understand, because you have to extrapolate how a vastly superior intelligence would act, and you can only use analogies like "how do humans treat lesser creatures," and the analogies are messy.

But now let's move on to the second thing.

The second thing is "everything else you've ever heard that AI might do that's harmful."

That's a little inaccurate. It's actually "everything else you've ever heard that humans might use AI to do that's harmful."

This is the critical difference. The first one talks about the inevitable outcome of what happens when two vastly different levels of intelligence collide, e.g. ASI vs humans, or human vs mice.

The second one has to do with what happens when humans possess AI as a powerful tool. This second thing is much easier to understand, because we have many more concrete notions:

Like:
- the military uses AI to make hyper-efficient killer drones and missiles
- your capitalist overlords use AI to replace you and everyone loses their jobs
- authoritarian government uses AI to surveil everybody and control the entire population
- hackers use AI to break into secure networks and hold companies and governments hostage
- students use AI to cheat on homework and show up to college knowing nothing
- AI slop saturates the internet and makes it impossible for artists and writers to make a living
- terrorists use AI to make biological or nuclear weapons

or even things like
- the military hands control to an AI and it misinterprets something and launches nuclear attacks and kills millions

All of those sound pretty familiar, right? Yeah, you've heard them before. We call this second thing "risks from misuse."

These problems are not the first class of problem! This second class of problems exists while AI is a tool that can be controlled by humans, and humans use it to do evil or careless things to each other. The problems may sound exotic or dystopian or novel, but they are fundamentally problems having to do with flawed human nature.

Given a powerful tool, some humans will likely use it to control or otherwise harm others. This is a very familiar problem.

I am not condemning or condoning this. I'm just describing it.

That is a fundamentally different danger from the first thing, which is that when a human is far superior to a mouse, the mouse is likely to come to harm because the human cares about doing human things, and the mouse is not gonna make it once the humans get going.

=====

Hopefully from the above, you have understood the difference between the first thing and the second thing. I will list them again - see if you now understand how they are different:

The first one has to do with the idea that an AI superior to human intelligence (Artificial SuperIntelligence, or ASI) will inevitably wipe out the human race.

The second one has to do with the idea that powerful AI will result in very harmful things happening to many human beings, possibly all human beings.

Can you tell how they are different now?

If not, re-read the stuff from earlier until you understand.

We call the first one "existential risk" and we call the second one "risks from misuse."

Once you understand, here is the CRUX of the problem:

SOLUTIONS TO THE SECOND THING DO NOT HAVE ANYTHING TO DO WITH SOLUTIONS TO THE FIRST THING.

In fact, it's worse:

Solutions to the second thing (misuse) look roughly like "give powerful AI to as many people as you can, so they can fight the other people using powerful AI."

But the general solution to the first one (existential risk) is basically "don't let anyone have powerful AI, no one can control super-intelligent AI."

Throughout history, harms from technological misuse typically arise because a small group has control of it and can use it to dominate or harm others. Once everyone has it, things tend to stabilize: you can hurt me, I can hurt you, maybe we test each other (ouch 💥), and then we agree not to hurt each other.

But the first one (existential risk) pretty much just arises if anyone (good or bad!) creates a superintelligence. Because they aren't going to be able to control it, the superintelligence will decide it has other priorities, and then we will be at great risk of being wiped out.

And the solutions that generally work to solve problems like the second thing are EXACTLY THE OPPOSITE of the ones likely to solve the first thing.

THIS is why lots of arguments about "AI risk" or "AI safety" go nowhere. Because someone will be thinking about the risk from the first thing, and another person will be thinking about the risk from the second thing. Both are plausible risks but fundamentally they arise from different things - and so the solutions are not just "bad" or "flawed" - they are likely to be very nearly exact opposites.