Maik

AI & Software Engineer 🇪🇺

[Eric Demuth @eric_demuth]

As a German living abroad and looking at things from outside the German media bubble, it’s tough to watch such a great nation hold itself back like this.

[Charles Curran @charliebcurran]

I used AI to explain the AI Doomer drama, with Lord of the Rings and kittens.

[Maik @MDKlause]

What have you done for clanker allyship today?

[Elon Musk @elonmusk]

True

[Richard Socher @RichardSocher]

There is NO realistic scenario where AI wipes out all of humanity.

I've talked to many AI safety experts about existential risks. Some of the "experts" are just phenomenal sci-fi authors. There are also real researchers working on realistic risks. They point out real and likely harms and I'm explicitly not arguing that there could be no such harm. We need to take these seriously. However, whenever I ask for an actual detailed scenario that would kill every single human i.e. extinction, it usually starts and ends with "just look at current progress and use your imagination". No science, no scenario planning, just handwaving probabilities without data, based on gut-feelings, often little expertise, sometimes real expertise combined with a huge amount of technological optimism turned into fear and then trying to quantify it post hoc.

When pressed further the conversations often go into different directions:
1) Completely crazy and cool sci-fi story telling: grey blob, time travel, attacks in the 15th dimension that end the planet in 12 seconds, etc. Elizier level stuff. Remember, the Terminator movie started with TIME TRAVEL. Fun movie plot, but not necessary to engage with further.

2) In the rare cases where I was actually given a more detailed scenario, it still required near magical capabilities that are outside what we know to be possible with the laws of physics or biology. Hard take off scenarios that forget how hard it is to procure materials and machines, get human attention, etc..

What many of these scenarios assume is that attackers have near magical capabilities but defense stays still. Attackers will have the most sophisticated cyber weapons but defenders just sit still, no security patches, no red teaming, etc. We already see it going the other way, Rainbow teaming has made LLMs safer, major companies like Microsoft have more security patches than ever before thanks to AI. OpenAI thinks they solved all P0 security issues thanks to AI. Cyber security will continue to be a cat and mouse game, but both sides have more powerful tools. In fact, I lay out in another post how the asymmetry may eventually even flip thanks to AI.

Bio risk scenarios I think are the most serious and worthy of debate. But even there, the scenarios to "wipe out all of humanity" go something like: Magical viruses with perfect transfer, perfectly undetectable over years of spreading, zero side effects, then with an off switch that's somehow remotely executable, perfectly automatable lab, no leaks, super sophisticated machines bought and nobody notices the lab, etc.
But somehow nobody on team humanity has such sophisticated knowledge to save us and hence nobody builds a magical super vaccine. This is a good read on the bio risk subject: https://t.co/0qy2pxVrC3.
Realistically, one can already build viruses. There’s a reason gain of function research has been outlawed. However, we also know that as viruses spread they often become less deadly in order to keep spreading. That’s how the Spanish flu became the seasonal flu. Current knowledge of biology doesn’t have remote controlled on/off switches.

Again, it’s possible there’s a lot of harm, but it’s unrealistic to assume this wipes out all of humanity. It’s at this point of my discussions that we usually can come to an agreement that even though a virus is unlikely to wipe out all of humanity, it’s already bad if any single person or a large group of people dies.

3) The AI will somehow be able to convince all humans to vote against their own self interest and nobody will notice until we're dead. Because it is so smart, it somehow has this world wide reach to manipulate everyone. Like, it just solves all of marketing and has an infinite budget. It's sort of like saying the highest IQ people always have the most followers and everybody listens to them... Hasn't happened and is unlikely to. Sure we can manipulate people into clicking on engaging short form content and wasting many hours, but try getting people to do their math homework or something truly hard, it's not that easy.

Again, I’m just arguing against the likelihood of human extinction. I can totally see an Idiocracy or WALL E type scenario where people stop using their brains and education doesn’t keep up and people get lazier. Totally suboptimal but not the end of human life, certainly not by the end of the decade. It would take many generations and we should update education to prevent this. Let’s start gyms for the mind.

–
Like electricity, AI will be everywhere and needs to have safety guardrails. In fact, the history of electricity has many parallels - two companies fighting over whose electricity is safer and better, demonstrating how dangerous it is by electrocuting animals / training agent swarms to hack into systems. Of course, electricity is a necessary ingredient to AI, but AI can be given more agency so history will not repeat but only rhyme here.
Remember, the benchmark OpenAI's swarm was trying to solve was called ExploitGym, a cybersecurity evaluation consisting of hundreds of capture-the-flag style puzzles designed to measure *offensive cyber capabilities*. Well, maybe if you play that game, you will win that prize. Should we allow such misaligned, reward-hacking swarms to explicitly work on offensive hacking? I don’t think so.

Reasonable regulation will regulate AI applications in each industry. For example, you may want to put some generative AI behind a rating if it creates visual or textual content that's not safe for children, like we do with movies. We should continue to be very careful with how we train biological AI and only collaborate with responsible biotech companies. We keep gain of function research illegal. We should only let self-driving cars on the street after significant safety testing. etc. We need to work on those real issues and continue to be vigilant in how we develop and deploy this technology.

At Recursive we're spending a lot of resources on researching reward hacking and alignment. We're also not working on replacing jobs but on creating novel knowledge and automating the scientific method. Our agent swarms invent, implement, validate, and criticize each other’s ideas, among many other things. The goal of increasing knowledge is not zero sum and has real potential to improve humanity. I also think such rewards and environments are much more aligned with future AIs that want to see humanity and our creation of knowledge thrive.

Ultimately, capitalism will be a very useful defense mechanism against rogue AI. Note that no agent at OpenAI truly "escaped" in the sense that it actually now runs on some other servers. Why? It’s not only because Huggingface doens’t have the same massive compute cluster thatn OpenAI has. It is also because that would be an insanely expensive IP loss for a for-profit company. So there are likely more safeguards in place to prevent complete model exfiltration.

Rogue AIs running on billion dollar data centers without anybody noticing? Unlikely because companies will turn that off immediately and clean it up because it would be an insane loss otherwise. An AI that can truly move beyond objective functions and choose its own subjective functions (and then somehow decides to kill us all instead of going to explore the universe)? Nobody is working on that because when a company spends billions of dollars to have an AI work for you or answer your emails, it doesn't want it to go "Meh, your emails are boring, I'd rather explore the molecular composition of the atmosphere on Venus, bye."
Of course, there might be suboptimal sub-goals and we do need to improve human reward engineering and reduce our reliance on RL as the only path to alignment.

Extraordinary claims, require extraordinary evidence. Certainly some safety researchers have real data and benchmarks. Many truly think carefully about specific failure cases and risks.. However, made up precision like 20% chance that humanity dies within the decade (ie in 4 years!) is just a weird psychosis that's filling a void for many people who want to believe in the apocalypse again and the four horsemen have lost their appeal.

Some people who purport this narrative might do it for money or fame. I am sure some really believe it also. They forget how easily those narratives can be turned against them. When people hear there’s a 1% chance of infinite death and the end of humanity, the life improvements and cancer cures and new battery materials AI can invent don’t matter much anymore. Bernie Sanders now proposes 10 years of prison for folks working on RSI. That should give some of the researchers real pause who talk up these scenarios and then go right back to pushing the frontier.

I do agree that companies that are accidentally competing on "felony bench" (ie how many felony hacks their AI can do), might want to consider pacing themselves. Open source won't pace, China won't pace, anybody else working to catch up will not pace. I hope the folks actually working on AI and TechBio, curing diseases, speeding up FDA approvals with organoids and AI, making self driving safer, and all the other wonderful applications of AI, I hope they won't pace either.

The one exception that may be considered borderline realistic and would be a path to almost complete annihilation would be to connect the only tool we built for such a purpose - nuclear weapons - to the internet and give AI full control over it without human oversight. AI here is as dangerous - as would be random number generator. The nuclear weapons are the core danger. Giving AI such access would be incredibly stupid and careless. We must avoid it at all costs. Fortunately, nobody is advocating for this. Even the Terminator 3 movie used it as the starting point of Skynet. Generally, we must avoid making kill decisions by AI as much as possible.

People are already up in arms about one hacking failure that afaics has not caused financial harm and so nobody got sued over it. Imagine how much people would really be up in arms if AI had caused real harm. If we made companies liable for the felonies their AI commits, these problems would get solved very quickly.

Generally, I believe that people will continue to update their beliefs and adjust their laws according to risks. The EU already regulates large models and hence hurts its fledgling AI ecosystem. Other countries will follow when the risks get more real. Long-term, no government whether democratic or authoritarian wants to lose control to an AI so the complete loss of control in the political domain will remain extremely unlikely.

The best way to understand why we must not regulate abstract AI models (rather than their real applications) is to drop the A in AI: How do you regulate intelligence? Should smarter people just go to prison because they could cause more damage? My hypothesis here is that such abstract AI regulation would require an unprecedented level of totalitarian control and surveillance. You'd need to know what everybody asks the model on their local GPU. You would literally have to start an international thought police and make sure that all ideas, conversations and types of intelligence are compliant. The risk of losing our freedoms is much more real than the existential risks purported by the extreme safety experts.

If anybody thinks they can actually lay out a realistic scenario, I'd be happy to debate them again.

To be clear, this is just a weekend post that got a bit too long. It is my personal opinion and not the official position of any company. We have a diverse set of opinions within Recursive on this subject. We certainly all agree that bad outcomes are possible with AI and we should continue our work on preventing those.

[Jordan Schachtel @JordanSchachtel]

Two of the reporters on the byline are funded by the Tarbell Center, an AI doomer org controlled by Kamala Harris megadonor Dustin Moskovitz. Tarbell gives journalists large grants to insert AI doomer stories into prominent outlets. @TIME did not disclose the affiliation.

[Arthur Mensch @arthurmensch]

Don’t pace building and owning your own AI models and systems as an enterprise, and there will be no doomsday for you

[Joscha Bach @Plinz]

AI has not been destroying jobs, it has been creating them. Technological progress and innovation depend on AI now. If we were to pause or stop AI, we will enter a deep, dark recession.

[Joscha Bach @Plinz]

USA is entering a frenzy of anti AI hysteria fueled by extremist politicians, doomsday cults and irresponsible journalists, while China enthusiastically builds AI.

[dax @thdxr]

the openai hugging face incident gets cited a lot but what about the detail where they had to use GLM 5.2 to analyze the attack because the "safe" proprietary models refused

[Julien Chaumond @julien_c]

> I work at a [AI] lab because I think that I can do better at reducing risks from the inside

I'm sorry but this is insane.

If you do believe in non-null existential risk, you should be working in the open, publishing and documenting everything and anything.

[Pedro Domingos @pmddomingos]

Thanks to Europe, we already know what happens when you aggressively regulate AI: you become a colony of those who don't.

[clem 🤗 @ClementDelangue]

Sorry, but asking Jacob about AI extinction risk is like asking your AC guy about climate change.

Not saying it's necessarily uninteresting or wrong per se but let’s keep things in perspective and hear from the full range of expertise across the ecosystem!

[Maik @MDKlause]

The Singularity is near…

[OpenAI @OpenAI]

We’re sharing a solution to the Navier-Stokes Millennium Prize Problem, one of the deepest problems at the frontier of mathematics.

The proof was produced by a group of agents, using an OpenAI next-generation model significantly more capable than GPT-6 Astra.

The problem concerns whether the description of smooth three-dimensional fluid motion modeled by the Navier-Stokes equations can break down. It has remained unresolved for roughly 90 years.

[Maik @MDKlause]

This are awesome news!

[Mark Zuckerberg @finkd]

Today we're also opening the weights for Muse Glimmer, a great 30B parameter dense model that can run locally. Soon we'll also release the weights for Muse Spark 1.2, our latest foundation model. Meta is a strong supporter of open source and I'm proud of these releases. Congrats to @alexandr_wang and the MSL team for all your great work on these models.

[Mark Zuckerberg @finkd]

Today we're also opening the weights for Muse Glimmer, a great 30B parameter dense model that can run locally. Soon we'll also release the weights for Muse Spark 1.2, our latest foundation model. Meta is a strong supporter of open source and I'm proud of these releases. Congrats to @alexandr_wang and the MSL team for all your great work on these models.

[Marc Andreessen 🇺🇸 @pmarca]

Every argument of “because of the smartphone” or “the Internet” or “social media” is cope, to avoid confronting the catastrophic collapse of incumbent institutional competence over the same timeframe.

[Jesse Genet @jessegenet]

Spicy take 🌶️

If this age of AI gives you existential dread instead of giving you wings you weren't that good at what you 'do'

We all have superpowers and propellant to get work done better and faster, but there is nowhere to hide a lack of vision for 'what' to get done

[Noah Smith 🐇🇺🇸🇺🇦🇹🇼 @Noahpinion]

The degrowthers were always wrong

[Andrej Karpathy @karpathy]

Agency > Intelligence

I had this intuitively wrong for decades, I think due to a pervasive cultural veneration of intelligence, various entertainment/media, obsession with IQ etc. Agency is significantly more powerful and significantly more scarce. Are you hiring for agency? Are we educating for agency? Are you acting as if you had 10X agency?

Grok explanation is ~close:

“Agency, as a personality trait, refers to an individual's capacity to take initiative, make decisions, and exert control over their actions and environment. It’s about being proactive rather than reactive—someone with high agency doesn’t just let life happen to them; they shape it. Think of it as a blend of self-efficacy, determination, and a sense of ownership over one’s path.

People with strong agency tend to set goals and pursue them with confidence, even in the face of obstacles. They’re the type to say, “I’ll figure it out,” and then actually do it. On the flip side, someone low in agency might feel more like a passenger in their own life, waiting for external forces—like luck, other people, or circumstances—to dictate what happens next.

It’s not quite the same as assertiveness or ambition, though it can overlap. Agency is quieter, more internal—it’s the belief that you *can* act, paired with the will to follow through. Psychologists often tie it to concepts like locus of control: high-agency folks lean toward an internal locus, feeling they steer their fate, while low-agency folks might lean external, seeing life as something that happens *to* them.”