Cassidy Nelson

Director of Biosecurity Policy at the Centre for Long-Term Resilience

[Rob Wiblin @robertwiblin]

In 2025 AI coded full bacteriophage genomes that work better than anything found in nature.

And crushed top virologists in a test of lab troubleshooting – beating human specialists in their area of expertise 45% to 22%.

What could go wrong?

But destroying the 'tacit knowledge' barrier to bioweapons development is just the beginning.

AI can design crypto-ricin, likely as deadly as normal ricin, which evades the best gene synthesis screening available.

PhDs with Claude nearly breached Anthropic's 'critical danger threshold' for bioweapons development. But most tests are still on undergrads: they're cheaper to hire.

Russia, Iran, and North Korea all have active biological weapons programmes. And even the most dangerous biological AI tools are so open that limiting proliferation is near impossible in the near-term.

The situation feel hopeless so I spoke to one of the world's top authorities on 'AI-Bio', Dr Richard Moulange, to find out why this situation isn't as bleak as it sounds. He proposes a range of shovel-ready projects, none of which fully solve the problem, but all of which would help move the needle:

1. Screen DNA synthesis orders. (Incredibly this is voluntary now.)
2. Give defenders a preview of models before they're fully public.
3. Def/acc: Stockpile broad-spectrum vaccines that cover entire virus families.
4. Prevent AI expertise in 'mirror bacteria', as their only use is mass murder.
5. Surveil waste-water to spot engineered pathogens before they spread.
6. Invest in intelligence so if a state releases a bioweapon, that can be proven, and the state punished.
7. Limit access and improve refusals to buy time.

(FWIW to me the situation still seems very bad.)

On the 80,000 Hours Podcast, wherever you get podcasts. Links below — enjoy!

01:11 AI can now design novel bacteriophage genomes
04:42 The end of the 'tacit knowledge' barrier
18:50 Are risks from bioterrorists overstated?
23:14 The 3 key disasters AI makes more likely
30:43 Which bad actors does AI help the most?
42:07 Experts are more scary than amateurs
47:32 Barriers to bioterrorists using AI
49:43 AI biorisks are sometimes dismissed (and that’s a huge mistake)
1:05:12 Advanced AI biology tools we already have or will soon
1:10:57 Rob argues that the situation is hopeless
1:19:38 Intervention #1: Limit access
1:34:28 Intervention #2: Get AIs to refuse to help
1:44:18 Intervention #3: Surveillance and attribution
1:58:28 Intervention #4: Universal vaccines and antivirals
2:12:01 Intervention #5: Screen all orders for DNA
2:21:57 AI companies talk about def/acc more than they fund it
2:28:44 Can you build a profitable business solving this problem?
2:33:08 This doesn't have to interfere with useful science (much)
2:35:16 What are the best low-tech interventions?
2:40:17 Richard's top request for AI companies
2:55:44 Grok shows governments lack legal levers
2:58:54 Best ways listeners can help fix AI-Bio

[Rob Wiblin @robertwiblin]

New Lawfare analysis: the designation of Anthropic as a 'supply chain risk' is doomed and "won’t survive first contact with the legal system".

1. The authors argue every layer of the government's legal position has severe problems, any one of which could be fatal on its own.

2. The most obvious concern: can Anthropic even get into court? The statute bars judicial review, which sounds absolute. But it isn't. The bar only applies when the government limits disclosure for national security reasons. Hegseth publicly broadcast his rationale on X: "arrogance and betrayal," "defective altruism," "corporate virtue-signaling." Hard to claim classification when you're tweeting the reasoning. And even if the bar held, constitutional claims survive it under Supreme Court precedent. And actions that exceed a statute's authority aren't shielded by that statute at all. The authors identify multiple independent paths into court; any one is sufficient.

3. There's direct precedent for courts striking down unsupported DoD designations. In 2021 a federal court granted a preliminary injunction against the Pentagon's designation of Xiaomi as a Chinese military company, finding it likely arbitrary and capricious — no notice, no explanation, no opportunity to be heard. Anthropic's situation is closely analogous.

4. The supply chain risk statutes were built to address foreign adversary threats and the legislative history points exclusively to foreign threats. Anthropic is a domestic company in a contract dispute. There's no precedent for using these authorities against a US firm. Wrong statute, wrong target.

5. The authors argue this is a strong candidate for the 'major questions doctrine'. The Supreme Court just last month struck down Trump's IEEPA tariffs on exactly this logic, holding you can't read transformative power into a statute never designed for it.

6. Hegseth ordered that no military contractor may conduct any commercial activity with Anthropic — including cloud providers like Amazon and Google, which could cut Anthropic off from the compute it needs to operate. This secondary boycott has no statutory basis. The statute authorizes 3 specific procurement actions, none of which cover contractors' private commercial relationships. When the government wanted comparable restrictions on Huawei, it took an act of Congress.

7. The government's own statements are devastating to their likely defense. Trump called Anthropic "RADICAL LEFT, WOKE" and "Leftwing nut jobs." Hegseth cited "arrogance and betrayal" and "defective altruism." The statute requires a technical, intelligence-driven finding — not ideological punishment.

8. Federal procurement regulations explicitly state that exclusion "shall be imposed only in the public interest for the Government's protection and not for purposes of punishment." The Supreme Court's pretext doctrine says courts need not "exhibit a naiveté from which ordinary citizens are free."

9. You can't simultaneously claim a vendor is indispensable, safe to keep using in the battlefield, and an acute national security threat, as the Pentagon is currently doing. As the authors put it, it's the joke from Annie Hall: the food is terrible and the portions are too small. To determine if a decision is 'arbitrary and capricious' a court will look for logical coherence and notice these propositions can't all be true at once.

10. OpenAI immediately announced its own classified Pentagon contract, claiming it has restrictions that mirror Anthropic's: no mass surveillance, no autonomous weapons. If OpenAI's restrictions are genuinely comparable, the government is punishing one company for terms it tolerates from a competitor. If they aren't comparable, one has to ask why OpenAI is publicly claiming they are without pushback from the Pentagon.

The authors' bottom line: "This is designation as political theater: a show of force that will not stick."

https://t.co/NkXpxXNzFd

[Cassidy Nelson @cassidyknelson]

Read our full piece here: https://t.co/2gqwWT3GTc

[Cassidy Nelson @cassidyknelson]

1/ The AI safety ecosystem has mobilised around AI-enabled biological attacks. That's the right instinct. But we've converged on too narrow a model of what that threat looks like — and it's creating dangerous blind spots.
New piece in TIME with @rebeccahersman 👇

[Cassidy Nelson @cassidyknelson]

7/ This isn't a call to look away from extreme biological risks. It's a call for a posture that can actually detect and prevent them.

[Cassidy Nelson @cassidyknelson]

6/ And we need public-private partnerships that merge classified intelligence with proprietary AI data. Cross-cleared personnel from companies and government need a shared space to spot threats and capability horizons that neither side can see alone.

[Cassidy Nelson @cassidyknelson]

5/ We need distinct safety tests and reporting across the CBRNe spectrum, not only bio ones.

[TIME @TIME]

"While it is good that companies are focusing on pandemics, the ecosystem is overly focused on a single 'lone wolf virus terrorist' model as the most serious threat. Significantly less attention is being paid to all other risk scenarios."

Rebecca Hersman and Cassidy Nelson: The weapons of mass destruction AI security gap https://t.co/H6TIn4apVD

[Cassidy Nelson @cassidyknelson]

See here: https://t.co/SauJXVPjAW

And read here: https://t.co/37IQVWjTUi

[Cassidy Nelson @cassidyknelson]

Rare speech at the BWC capturing the frustration of these efforts that has hit a precipice.
"If we take this path, mechanisms will one day – at best – be born old. This delay is not harmless. It comes at a cost, a huge cost." - Ambassador Frederico S Duque Estrada Meyer (Brazil)

[Cassidy Nelson @cassidyknelson]

"The biosecurity landscape is evolving rapidly. The BWC risks falling even further behind. A single lapse in vigilance could spark consequences that reverberate across continents and generations."

[J. Craig Venter Institute @JCVenterInst]

A new @ScienceMagazine article from 30+ leading international scientists including several @JCVenterInst researchers, examines the potential dangers of building ‘mirror life’ — organisms composed entirely of mirror-image biological molecules.