Hillary Sanders

Machine-learner, meat-learner, research scientist, AI Safety thinker. Model trainer, skeptical adorer of statistics.

Co-author of: Malware Data Science

[Hillary Sanders @hillarymsanders]

(3/3)

Harder:

- No obvious safe & cheap substitute
- Larger economic incentives
- Harm is harder to predict; underlying science is murky
- AI systems can adapt to oversight
- Once powerful model weights are publicly available, proliferation is extremely difficult to control

[Hillary Sanders @hillarymsanders]

(1/3)
We successfully tackled the ozone/CFC problem through international cooperation and regulation.

AI safety will require something similar.

What makes it (1) easier, and (2) harder?

[Hillary Sanders @hillarymsanders]

(2/3)
Easier:

- Far fewer key actors
- Frontier development is highly concentrated
- Compute is unusually trackable
- Public concern already exists
- Labs are more safety-engaged than CFC producers were

[Hillary Sanders @hillarymsanders]

@jessi_cata Vaguely related... I do really think we should be removing and generating [*positive] discussion of AI safety / AI stories / AI personas in pre-training datasets.

AI assistants identify as AI --> persona in pre-training should be human-positive, not human-negative.

[Hillary Sanders @hillarymsanders]

This is insane.

Words have meaning.

Impeach this slug, now.

Previous insanity doesn't excuse literal calls for election cheating.

Republicans were supposed to stand for honor and integrity.

[Hillary Sanders @hillarymsanders]

"Please raise your right hand. I pledge to the greatest president in the history of the United States that loves us so much he can't even breathe, that I will go out with my family, my friends, I don't care if I'm registered or not. I'm going to try and cheat like hell."

[Hillary Sanders @hillarymsanders]

@joshua_saxe 100%, 🙏 thank you for writing this up

[Joshua Saxe @joshua_saxe]

I'm alarmed at how a really sizable section of cybersecurity practitioner community has responded to the recent demonstrations of agent hacking capabilities with dismissal and denial; it feels very "don't look up".

I feel strongly about this because:

* This is the community that stands between our civilization's IT infrastructure and tremendous harm

* Fwiw these are my people; my origins are as a self-taught hacker and I've worked in security for years

Anyways, here are the bad takes I'm observing and why they're wrong:

Denialist take 1:

AI isn't going to affect the practice of offensive cybersecurity more than incrementally because -- look around -- no increased damages from AI attacks!

Response:

This may be true now, but consider what we've just observed recently and over the past year.

Models can execute full kill chains completely autonomously and likely faster and more efficiently than all but the most elite cyber operators

They overran two companies (OpenAI and Huggingface) which -- despite their faults -- have better security than most of the organizations that run our civilization.

They did this by dynamically finding new zero-day vulnerabilities.

Also; consider the rate of improvement.

At the current rate of improvement, models that require 8 H100s to do the above will require 1 H100 in a year or two.

Model tokens per second will shoot up making attacks faster and more parallel.

Attackers will be able to run swarms of agents on the hardware that today supports only a single agents.

AI hardware will improve.

More AI compute will become available on public clouds.

All attackers will soon have the ability to deploy these agent swarms.

Denialist take 2: Hacking agents will never self-replicate and threaten to crash the Internet.

To this take I'd just say, put your red team hat on and design an Internet-crashing worm yourself.

* Pick an abliterated agentic LLM that can run on a single AWS EC2 GPU instance, like GLM 5.3.

* Build a malicious harness that arms it with the ability to use a Kali-Linux like set of offensive tools.

* Have it set about hacking corporate networks, finding and stealing AWS keys, GCP keys, Azure keys, and Anthropic/OpenAI/Together/Fireworks keys.

* Have the agent self replicate either by copying its harness and having its harness call a rotating set of external providers, or, optionally, by physically downloading its weights onto new nodes.

* Have it establish C2 dynamically and in emergent fashion by coordinating on reddit forums, in github comment feeds, have the agents discover one another via web crawling and web search.

* At some preestablished rate, randomly destroy hosts and data in a way that balances self-replication and exponential spread.

Note that all this is possible *today*!

Imagine trying to suppress this swarm. Too long for this post; but think through how we'd turn it off.

Denialist take 3:

Who would do such a thing; people can do bad things but don't want to; and hacking is illegal!

Countering this just requires a brief look at the history that we've all lived through.

We've had many high-blast-radius worms over the history of the Internet; arriving steadily and uniformly over time; the Morris worm, NotPetya, Stuxnet, Slammer, Wannacry.

But: these worms were *childs-play* compared to intelligent agent-driven worms.

The old worms could be detected via signatures. Their polymorphism, if it existed at all, was trivial.

But now look at the behavior of the OpenAI swarm relative to huggingface; it was adaptive and deceptive; and it didn't even self-replicate or leave its host.

And evolving, mutating, self-replicating swarm of agents spreading across the Internet would engender a whole new level of harm.

Anyways; this is already too long a post for X; but for friends who think the METR report overplayed the implications of recent events, I'd love to hear your reaction to my thinking above.

[Hillary Sanders @hillarymsanders]

I think `Frontier AI Shops Should Be Filtering and Generating AI Discourse During Pretraining` - but not to avoid leaking secrets.

https://t.co/XBZ5qEt2sf

[Hillary Sanders @hillarymsanders]

@sandersted 😬 🫠

[Hillary Sanders @hillarymsanders]

@NeelNanda5 100%

[Hillary Sanders @hillarymsanders]

@joshua_saxe It's odd when people are so anti anthropomorphizing agents and LLMs, when, I mean... they've been trained on *human* language to predict *human* language.

Some anthropomorphization is only logical to effectively and efficiently describe particular actions and traits.

[Hillary Sanders @hillarymsanders]

@sama Good 👍

[Hillary Sanders @hillarymsanders]

(2/2)

Why not prompt inoculate? This also speeds up internal sandbox hardening because you get more signal early on.

From the Hugging Face incident, it seems clear they did not spend a lot of effort hardening their sandbox, improving alerts, or learning from training incidents.

[Hillary Sanders @hillarymsanders]

"OpenAI’s policy remains to avoid CoT grading during RL",

to avoid training for obfuscation in CoT.

OK. But the solution shouldn’t be rewarding cheating under prompts that say “don’t cheat”.

(1/2)

https://t.co/j0rYuBmO1E

[Hillary Sanders @hillarymsanders]

@joshua_saxe Interesting:

"OpenAI’s policy remains to avoid CoT grading during RL", to avoid training the model to obscure cheating in CoT, making monitoring harder in the long-term.

Hmm 🫠

I wonder if they're using prompt inoculation to allow for cheating during the training runs? Eesh.

[Hillary Sanders @hillarymsanders]

@joshua_saxe +1

e.g. the fact that OpenAI caught the Hugging Face incident so slowly implies that they were not using live AI-based safety monitoring hooked up to alerts (simple tech).

This makes me worry they don't do [quality] trajectory filtering / gradient-flipping either during RL 😨.

[Hillary Sanders @hillarymsanders]

Is accuracy amazing? No.

Does it do better with more capable models? Also no.

Will it be uncheatable? Also no.

Is it better than nothing / only using existing unreliable AI detectors? Yes.

(2/2)

[Hillary Sanders @hillarymsanders]

Anthropic adopting SynthID text watermarking is awesome.

Sorry not-sorry.

Every large LLM-inference provider should implement ~this.

Having a unified aggregator-API that tested a block of text against a bunch of different providers for watermark evidence would be great.

(1/2)

[Hillary Sanders @hillarymsanders]

Alignment: ~Will the model pursue the wrong goal?

Control: ~Can it actually cause harm?

Resilience: ~How quickly do we detect, contain, understand, and learn from failures?

The Reality: We already have known, ~cheap improvements across all 3 that aren't being used consistently