xeldarie

Physicist and former PhD Student in AI & Robotics
Interested in Risk Mitigation of Superintelligence
Interested in Bitcoin

[xeldarie @xeldarie]

If a sufficiently strong AI will be allowed to reprogram itself without any limits, it will never be controllable. How do we know that? Human intelligence is also out of control, we are one example of that.

[Timothy Ash @tashecon]

Finally one billionaire with some balls.

[JohnfromPGH @johnis44]

This woman is 104 years old.

She’s seen 18 presidents in her life.

Listen to what she says about Trump.

[xeldarie @xeldarie]

puh

[0xMarioNawfal @RoundtableSpace]

Closed source is dead.

REA lets Claude Code or Codex take apart any app, all the way down to the binary, and explain exactly how a feature works.

- Native apps, Electron apps, websites, APKs, firmware, even smart contract bytecode
- Shows the evidence for every claim
- Then builds its own version of the feature

25,000+ GitHub stars in a single day. 100% open-source.

https://t.co/0u2Zc9TxmR

[vitalik.eth @VitalikButerin]

Doing a bit of a self-experiment.

Goal: use my personal health and travel data to provide personalized diet and exercise recommendations for me, using frontier models but in a way that avoids leaking to them any private information.

Strategy: use a local model (Qwen 3.8 Flash Next) to orchestrate, and use powerful remote models as a tool call to benefit from higher-level thinking and knowledge that the local model does not have.

Three-layer approach to privacy:

Avoid leaking personally identifiable information, or leaking my identity through writing style -> local model writes the queries to the frontier model, not me
Avoid leaking who I am through the payment channel -> zkAPI
Avoid leaking who I am through networking / IP -> Tor

You need all three (and finally we have all three, at least to some extent)

A skill file teaches the local model when and how to construct minimally-data-revealing requests to remote models. Use zkAPI-wrapped-with-Tor as a CLI tool.

And everything works! I got the recommendations back, info from frontier models helped to improve them.

Main deficiencies:

* Tor is really not optimized for request-by-request de-linking, which is the only form of network-layer privacy that really makes sense in today's world (long-running identifiers are too fragile). Probably not private enough, and latency 10-100x higher than it could be, at the same time.
* The skill file's request construction strategies are definitely far from optimal.
* Qwen 3.8 Flash Next is still too slow for comfort. It's comfortably running at 20-30 TPS, but it would only really start to feel fast at 100+
* There is a tradeoff: the more careful you are about what data you give to a remote model, the less it can help you

https://t.co/1g81Mp1sWW

[Charbel-Raphael @CRSegerie]

How does AI actually kill a billion people? Here is one of the many ways.

An MIT team led by Kevin Esvelt took the DNA of the 1918 influenza virus, split it into fragments, and ordered them from 38 synthesis providers. 36 shipped. No screening triggered. Pieces of a pandemic virus, arriving in the post.

That barrier was already lower than you think. AI lowers the rest.

AI does not build a pathogen end-to-end, at least not yet. But it considerably lowers the expertise you used to need. In August, a Stanford team used an AI model to design an entirely new virus from scratch. Out of nearly 300 synthetic genomes, 16 were viable. A mix of these AI-made viruses wiped out their bacterial target, which had already become immune to natural viruses.

Nature does not optimize viruses to kill us. Ebola, for example, kills about half the people it infects, but because it makes people visibly sick very fast, it is relatively easy to contain and quickly runs out of targets. An artificial pathogen has no such constraint. A synthetic virus could be optimized to combine extreme lethality with a long, hidden incubation period, a combination that rarely happens in nature. And an attacker does not have to stop at one. They could release a hundred at once.

Every year, if not every month, the hard part gets less hard.

I don't think what I just said is sufficient for extinction. Someone is always in a submarine, and some AI CEOs have allegedly invested in bunkers. But this could end in billions of casualties.

Every year, if not month, the hard part gets less hard. This is an emergency. We need to urgently scale up pandemic preparedness and implement basic measures like robust DNA synthesis screening, while also establishing strong governance for this technology.

[Peter Steinberger 🦞 @steipete]

"the Waymo effect is what happens when a technology removes the friction of dealing with another human being" https://t.co/dOVghMfF1L

[xeldarie @xeldarie]

leviathan to the rescue

[Max Tegmark @tegmark]

I'm excited to share our new film Moloch, about how we all get disempowered without AI going rogue. It’s the 4th (and IMHO least discussed) dystopian AI scenario:
1) Rogue AI
2) Obedient AI misused
3) Obedient AI causes 1984
4) Race-to-replace by obedient AI
Synopsis: As AI triggers worldwide job loss, an engineer who built the tech struggles with dark forces, her family & her CEO over pulling the plug.
Starring Fiona Hampton, Parker Sawyers, David Buttle & Buddy Wignall-Ho.
Written & directed by Tom Cozens.
It's an Owl In Space production in collaboration with FLI.
We worked really hard on this and hope you find it thought-provoking!

[xeldarie @xeldarie]

@thejefflutz @JensenHuang Alarmists are entitled to their opinion just as much as he is to his.

Allow people to be scared so they start to build their own opinion.

At least some of them :).

[Yoshua Bengio @Yoshua_Bengio]

I was invited to address the @UN Security Council this afternoon to discuss the unprecedented threat posed by uncontrolled frontier AI agents.

Humanity has summoned the courage and wisdom to manage catastrophic risks before. This is one of those moments, far bigger than any personal, commercial, or national interest. The choices we make about this technology today will shape generations to come. We must act now.

[DHH @dhh]

My positivity for this magical moment in computing is not born out of blind optimism for a possible future, but from awe of the present potential. Everything we've ever wanted to do with software is now possible. We can make anything. We can fix everything.

[vitalik.eth @VitalikButerin]

Qwen 3.8 flash is truly impressive, and llama.cpp has been rapidly getting better and better at processing it

columns are: pre-existing prompt, new prompt, generated, input tok/s, output tok/s

This is on my laptop (strix halo). I think we're very close to the point where you can just use local models for a large share of tasks, and for anything more advanced, workflows like "use your local model to orchestrate queries to powerful models so your queries don't leak your personal information" actually become viable.

[Jacob Coxon @hilbertspaess]

I resigned from Anthropic today. I spent the last three years doing pretraining research at both OpenAI and Anthropic. Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives. More thoughts below.

[Maxime Fournes⏸️ @FournesMaxime]

Here is a normal reaction by a journalist learning that all humans face a high probability of imminent death because of AI.

"oh my god"

[Nate Soares ⏹️ @So8res]

Finally figured out how to talk to people about AI danger:

[Demis Hassabis @demishassabis]

https://t.co/PTeDiv1b6L

[xeldarie @xeldarie]

@0xSchnitzel I think humans together with machines for quite some time, eventually machines autonomously also. As long as they innovate for us we will be the beneficiaries, so they gotta ask us. Let's hope they stay tools.

[xeldarie @xeldarie]

@Megatron_ron Natürlich, damit die Politiker und Scheinökonomen wieder mehr Geld drucken können. Es ist ein Fass ohne Boden Leute.

[Aakash Gupta @aakashgupta]

This is being read as a philosophical farewell. It’s a resignation letter from the head of Anthropic’s Safeguards Research Team, and the most important sentence is buried in paragraph three.

“I’ve repeatedly seen how hard it is to truly let our values govern our actions. I’ve seen this within myself, within the organization, where we constantly face pressures to set aside what matters most.”

That’s the person responsible for keeping Claude safe telling you the pressures to ship are winning.

Mrinank Sharma built the Constitutional Classifiers system, developed defenses against AI-assisted bioterrorism, and authored one of the first AI safety cases ever written. Two years of work at the exact intersection of “make the model safe” and “ship the model fast.” And he just walked away.

Now zoom out. Dylan Scandinaro, another Anthropic AI safety researcher, left last week to become OpenAI’s Head of Preparedness. Harsh Mehta and Behnam Neyshabur, both senior technical staff, also departed in the past two weeks. Four notable exits in a single month from the company that sells itself as the responsible AI lab.

Meanwhile, Anthropic is in talks to raise at a $350B valuation and just launched Opus 4.6 last Thursday. The commercial engine is accelerating. The safety talent is dispersing.

This is the core tension of every AI company right now: the people building the guardrails and the people building the revenue targets occupy the same org chart, but they optimize for different variables. When the pressure to scale wins enough internal battles, the safety people don’t fight forever. They leave and write beautifully worded letters about integrity.

Sharma’s next move tells you everything. He’s pursuing a poetry degree. When your head of safeguards research decides the most authentic use of his time is writing poems instead of writing safety cases, that’s a signal about what he believes the safety cases were actually accomplishing.

[xeldarie @xeldarie]

Good point.

[Luke McNally @l_mc_nally]

I would contend that it is a more objective statement to say "@karpathy is an amazing and experienced engineer, you should listen to his vibes on how impressive AI coding agents are", than it is to say "the METR time horizon eval is a calibrated measure of how close we are to AI R&D". Also, what I say is not coming from vibes, it is coming from my experience of talking to lawmakers and experience of others at @JoinTorchbearer and chatting to people @ControlAI . Lawmakers don't care about the charts, they want to hear what the experts think.
More fundamentally, we already have real world examples that illustrate the progress. Anthropic say Claude code is more than 2xing productivity, Claude largely autonomously carried out cyber espionage, etc. This is more convincing than a benchmark. A lot of brilliant people, who are concerned with AI risk, are sinking all this time into measuring something that many people think is just measuring a good thing, while they think it's a scary thing! Imagine if they sunk that brainpower into talking to lawmakers, building coordination technology, or working on pause implementation plans - this would clearly be better.
Putting intelligence on a chart is hard. METR have made the best effort and it's still not at all clearly beneficial. This is not the bottleneck - educating lawmakers and the public, and figuring out how to implement a pause/moratorium are the bottlenecks.