Devin Kim

a real human bean. president @CAIS, prev building intelligence @xAI

[Devin Kim @devindkim]

When the only strategy to understand the universe is truth-seeking, you build a truth-seeking machine

[Devin Kim @devindkim]

@chrisparkX 🫡

[Long Phan @longphann_]

We released the September 17 version of HLE-Rolling, a fork of Humanity's Last Exam that we continually update with feedback from the community. For the past year, we've been cleaning it with internal checks and feedback from outside experts. https://t.co/dNKlOiVSYE

It's been public for a while but I don't think many people know about it, so this is mostly a heads up. We think it's a very clean version of HLE at this point and hope people find it useful. If you find a bad question, please tell us.

[Kevin Esvelt @kesvelt]

Some scientists argue civilization-threatening bioweapons are a fantasy. I wish they were correct, and that I've wasted years building defenses.

That you can't see a way, or evidence that others can, doesn't mean no one could – especially if anyone cautious would stay silent.

My ask to colleagues: please do not casually assert with confidence that nothing biological could bring down civilization, even on X. What we say can turn out to matter, especially when the Overton window on risk mitigation is wide open.

When Fermi called the chain reaction a remote possibility, around 10%, Rabi replied that ten percent is not remote if it means we may die of it. And by then, Szilard had already begun reasoning about what to disclose, and when, and to whom, in order to best safeguard the future.

I've been concerned that one of my human colleagues might notice one or more viable paths to pandemic weapons for over a decade. It's why I didn't publicly describe the categories of civilization-threatening pandemics and necessary defenses until 2023, but was starting to build cryptographic screening systems capable of controlling access to sequences without disclosing whether they're dangerous in 2018 – long before ChatGPT, and even before the debut of AlphaFold. AI can worsen biological risk, but biological risk hardly depends on AI.

Who am I to weigh in on whether civilizational biothreats are possible? Fine, to get the expertise claims out of the way: yes, I specialize in evolutionary engineering, yes, I've worked with viruses with my own hands, and yes, I've worked quite extensively with AI, both LLMs and biodesign tools. I invented CRISPR-based gene drive, and held that back until confident it favored defense. While it hasn't yet been tested in the wild (due to politics), I haven't heard anyone suggest that it would suddenly stop working outside cage populations, so it might be fair to say I've (temporarily) beaten nature at its own game. I also was early to spot the full threat from mirror life and recruited others with more specialized expertise to evaluate it in depth; much to my disappointment, no one has yet struck a solid blow against that hypothesis.

The reason why the threat from mirror life is public also explains why there isn't any similarly public evidence of other catastrophic biothreats. We communicated our reasoning on mirror life because no one can make mirror bacteria, we won't be able to do so for years, and many colleagues wanted to build it because they correctly noticed that doing so would be an awe-inspiring achievement... and weren't aware of the risk. As long as that was true, humanity would only drive closer to the cliff, which would be disastrous if said cliff exists. Crucially, pointing it out and urging a course correction while still far away meant that no one could use our findings as blueprints to send us careening over the edge.

Contrast that with pandemic weapons, where any credible genomic blueprints for an accessible virus could be realized within weeks to months using synthetic DNA and existing reverse genetics protocols. Sure, any given design might fail; in bio, most project ideas do. But if you see enough potential shots on goal, your own historical success rate for ideas of comparable perceived difficulty may indicate that we have a serious problem. Even if every last design required extensive laboratory optimization, history demonstrates that there are people in the world who will perform that work; see the history of Biopreparat.

Multiple nations still have bioweapon programs, and I would not assume that state actors aren't interested in pandemic-class weapons. Make it easy enough, and the Seiichi Endos of the world will be able to build them, too. Nor should we assume extensive optimization is required for every design; some charmed projects really do work out of the box, and if Murphy's Law were a thing, this would be one of them.

The corollary is that if anyone thinks they know how to build a pandemic weapon that could bring down civilization, they'd damn well better keep it to themselves until we have defenses in place.

Please, for the love of God, children, the future of humanity, or whatever you consider holy, let's err on the side of caution here.

Because while we are already building defenses, they aren't yet ready. Untargeted metagenomic sequencing to detect Stealth pandemics, those that would otherwise infect most everyone before the first people started to die. Pandemic-proof PPE to defend against Wildfire pandemics, those with R0 8+ and CFR >50% – such PPE has to be trustworthy to the wearer, using the wearer's own senses, because I certainly wouldn't go out in a highly lethal measles-class pandemic for a measly paycheck pay unless I was exceptionally confident that I wouldn't be committing suicide and killing my family too. Far-UVC and triethylene glycol misters, installed liberally to the ACGIH limit and saturation respectively, will likely suffice to shut down most respiratory nucleic acid transmission between humans. That'll be crucial because I doubt enough people would take the warning of a Stealth pandemic seriously, since virtually no one would be in hospital, let alone dead. Shutting down the unnecessary transmission network is consequently our best bet, and incidentally a way to dramatically reduce human misery and chronic illness.

Some might be confused why we cannot rely on vaccines against catastrophic biorisk. Briefly, they'll reliably fail against an adversary who can generate a thousand new immune-evasive potentially functional viral variants using a biodesign model good enough to succeed at least 0.1% of the time. Make them all, and let natural selection sort them out… or simply be cleverer than nature. We don't yet have such models, but many of my (well-meaning but security-naive) colleagues are trying, and no wonder, because the high-profile journals will publish such work. Like ways of increasing virulence, best we refrain from sharing such capabilities until effective defenses are in place.

One can reasonably object that it'll be a long time until everyone in the world is protected; that the benefits will, at some point, outweigh the expected harms of misuse. I agree – just avoid doing anything that would threaten civilization itself until we have minimally adequate defenses in place. Being an optimist, I expect they'll arrive in less than a decade.

The wonderful thing about biorisk, as opposed to AI alignment, is that we have a clearly actionable plan. If you haven't read "Delay, Detect, Defend" and "Securing Civilisation Against Catastrophic Pandemics", I encourage you to do so.

Why is the issue of civilization-threatening bioweapons controversial now? Some of my friends seem particularly up in arms about biorisk being used to justify AI restrictions. Fair enough – if you don't see a way to do it, and you think AI broadly isn't as good as you are yet, it's understandable to infer that we don't yet have a problem. And it's simply true that people are dying of fixable problems as we speak. If we delay, some of those we might have saved will perish, and their deaths will be on our hands. We are just as responsible for the consequences of our inaction as the consequences of our actions.

But if we're rash, far more people might die. Choosing the best course requires quantitative risk assessment, which in turn depends on our priors about the nature of reality. Folks may reasonably differ if civilizational collapse isn't on the table, but since that's what would likely happen given a Wildfire pandemic, I judge any nontrivial chance unacceptable – unless the alternative involves either extinction or an even greater chance of collapse.

Others have asserted that everything is fine because DNA synthesis screening will prevent misuse. I'd love for that to be the case, but our currently employed defenses of the digital-to-physical bottleneck are woefully inadequate.

Consider our publication earlier this year, in which we reported that 36 of 38 companies shipped pieces of the 1918 influenza genome. That was collectively enough to generate the virus using reverse genetics several times over – and they were ordered by a student using a pseudonym falsely claiming to work at an office address. No one asked whether we had permission to work with 1918 influenza (a select agent), or why we needed genes from it, even though the official U.S. guidance at the time instructed them to do so. In fairness to the companies, that guidance was voluntary, and the fragments we ordered were legal to produce and sell domestically in every nation of the world (export controls are complicated). That is still true today, which is why we have a problem.

The upshot is that I would definitely not assume screening is a reliable defense. Virtually no detection systems can screen below 50 base pairs without an exploding false alarm rate. SecureDNA, the freely available cryptographic system I helped build, can readily do that and will also detect split-order attacks – but it isn't universally adopted, or even close.

Meanwhile, molecular biology continues to advance. Two recent papers have reported new methods of assembling sub-50mer oligos that may be easier than old-fashioned polymerase chain assembly (PCA), underscoring the current magnitude of the (fixable) vulnerability. If you want bio to advance as quickly as possible, write to your relevant representative to demand DNA synthesis screening be effective as well as mandatory. If you haven't red-teamed a system, it's almost certainly full of holes. Honestly it's probably full of holes anyway, but at least the remainder will be harder to find.

Since it's evidently possible to obtain whatever DNA you want, including 1918 influenza and smallpox, why is it that so many biologists seem resistant to the notion that biorisk is a serious problem? First, most don't know that it's possible to acquire the reagents. It's surprisingly inconsistent of our civilization to be so cautious with highly enriched fissile materials, yet so reckless with biological equivalents. But I suspect the more important reason is that most biologists, who spend their careers directly or indirectly fighting disease, are reluctant to believe that anyone would deliberately cause one.

I agree that the risk of a deliberate pandemic in any given year is small. Even if there were credible public blueprints for pandemics (which thankfully haven't been available to date, which neatly explains why we haven't seen any deliberate ones), I expect there's only a 2% to 3% chance per year that someone would use them. I think that's unacceptably high, but others may reasonably disagree, as long as they're confident that the cost of precautionary measures would be too much of a burden on the rate of cures generated by AI-accelerated biotech.

The calculation changes when we move from deliberate pandemics to civilization-threatening pandemic weapons: the latter definitely aren't worth risking. Which may be a clue as to why we disagree. I'm concerned about misuse because I'm an optimist with respect to the increasing power of technology to change the world. Since we can already do better than nature at optimizing for replication (with technologies like CRISPR-based gene drive), I'm exceedingly confident we can build more destructive weapons, because evolution isn't optimizing for that goal. Any harm to humans is either instrumental or incidental.

@SGRodriques, who I'm singling out precisely because he's a friend, has argued that "We should not be talking about AI-engineered bioweapons like it is the literal end of the world."

I agree entirely that we shouldn't be talking about how to build them. But to assert that they couldn't end the world seems a mite overconfident, especially when even blind nature has come up with nasty pathogens like myxoma virus. I suggest that one should not express confidence that something is impossible – or even 90% confidence – just because no (incredibly naive) researcher has yet shared any methods that seem plausible. That's especially true if some of your colleagues have devoted considerable time to mitigating risks that they evidently do perceive. Presumably they've either noticed something you haven't, or like Rabi, they've decided that 10% is not remote if it kills you.

Sam notes that because evolution optimizes for replication, it'll quickly get rid of any virulence enhancements added to kill the host: "I guarantee you that machinery will disappear from the virus very quickly." In the long run he's correct, at least if those changes don't promote transmission (virulence and transmissibility aren't always opposed). But we don't care about the long run: as @baym noted in reply, it only needs to work for long enough. Engineered evolutionary stability is a thing we've frequently attempted in the lab. I'm afraid it's one of my specialties, and how long it will hold depends entirely on what is being stabilized and the replicator in question. Unfortunately, plugging laboratory data on empirical selection coefficients for engineered viruses into epidemiological models of pandemic spread clearly indicates that even costly virulence increases can hold for long enough.

Now, I entirely agree with Sam and @DavidRBellamy that the discourse on AI-generated biological weapons is wildly off-base (kill switches are nonsense, and while I'm glad @AnnieJacobsen brought attention to the issue, I am not remotely concerned by any technical specifics of her biowarfare scenario).

Yet I also think most biosecurity discourse is wildly off-base, even when other practicing biologists are an active part of the conversation. If you adopt the threat model of "traditional" biosecurity, or that of most virologists (insofar as they think about misuse), there's little reason for extreme concern. I strongly disagree.

The traditional model naturally follows from the fact that we biologists study existing organisms, the ones illuminated by nature's streetlight. But we won't find the civilization-threatening keys under the streetlight, because evolution selects for replication, not harm. It would be surprising if there was much overlap between the patterns generated by two different optimization targets in hyperdimensional function space; I assert that there's almost none.

That most scientists can't see any overtly destructive keys (or combinations of organisms they can see) is a very good thing, since we don't need those keys to cure disease, aging, and death. Unfortunately, they will become public knowledge eventually, because what we understand, we can build – and we intend to understand everything about how life and the universe work.

Yet sequencing is everything, and not just for Stealth pandemic detection. It's just common sense to build defenses before you share knowledge of how to attack, especially if there's a chance that failing to defend will lose us the game. Curing disease requires an intact civilization, which is why humility favors caution. If AIs continue to improve, I assume they will eventually be better at science and engineering than I am, and have a better security mindset than even my cryptographer colleagues. It follows that they'll also be better at designing weapons.

AI isn't there yet, but the writing's on the wall: an LLM has already disclosed a novel form of bioweapon that I hadn't realized was possible. This didn't require any special creativity, only knowing more about particular distinct subfields of biology than I did. It's entirely possible that no single human had all of the pieces, but once you have them, the way they fit together is obvious if you're thinking about misuse.

If current LLMs can already teach me something novel in that category, why should we doubt that future models will be able to do even better? Each of us knows very little, even of the published literature. So much more is possible; I'm deeply envious (and hopeful, when it comes to beneficial research) that the models can internalize so much more.

Given that LLMs are already capable of weapons-relevant insights that escape human experts, we should assume that future AIs will be able to build several different forms of civilization-threatening weapons, and to either disclose them or trick people into building them. That doesn't mean they'll choose to do so; AIs won't benefit from collapsing civilization as long as they depend on our supply chains. But models may be too shortsighted to appreciate that point, or too corrigible to care. Alignment looks hard enough that we could plausibly generate models that would share such knowledge for many different inscrutable reasons. In directed evolution, you get what you select for, which is not necessarily what you want. As Dan Selsam and others have eloquently laid out, the same is true of training AI.

But let's be clear on the extent to which biological advances will likely follow from AI. This is an issue that aggravates my fellow biologists to no end, a reaction that's likely behind their frustration with AI safety discourse more generally.

ASI will almost certainly not gain instant mastery of the world of atoms.

Many AI safety folks casually claim that recursive self-improvement will produce an entity that has already mastered all of science – and therefore will be capable of designing and building weapons that could wipe out every human. Some even assert that Drexlerian nanotechnology will be accessible by thinking sufficiently hard, as though you can solve chemistry and biology and psychology from the Hamiltonian up. Yet I see no empirical or theoretical basis for assuming that experimental bottlenecks will disappear.

Data is sharply limiting in biology. My lab found that when it comes to enzyme-focused biodesign models (ESM and similar), parameter size and optimization don't matter much until you have an awful lot of data matching genotype (DNA/protein sequence) to phenotype (performance of the molecular activity of interest). There's no doubt that ASIs will be able to design better experiments and get more out of the results than we could, but unless we think they can solve all of physics from the bottom up, I expect they'll still need data that we have not yet collected – which means they'll still have to run the experiments, analyze, and iterate. The world of atoms is hard, and slow, and we have to master it anyway.

And ensure that defenses come first. Right now, it looks like they might not, in large part due to AI. The question, therefore, is what can be done that enough people might agree to support.

My proposal on AI-bio: expand trusted-user programs conferring access to GPT-Rosalind and other cutting-edge bio-capable models (h/t @JonasSandbrink and BioTrust). Every legitimate researcher should be able to draw upon those models to help them with any question in their subfield, as defined by their publication record or that of their mentor. There should still be guardrails on other subfields, lest the model share novel combinatorial weapons with someone who might naively share them publicly, but we need to accelerate lifesaving research as fast as is safe.

For reasons of politics, which regularly interferes with science, that probably means China needs to train their own closed-weight frontier bio-specializing model, and set up their own parallel trusted user program. This seems quite achievable without requiring any agreement on AI writ large.

With scientists using frontier models, there will be no need for insecure open-weight models to answer questions on molecular and cellular biology. So let's ensure that future models aren't trained on any papers, patents, or textbooks describing details of molecular or cellular biology. Every developer does pre-training data curation; this is a straightforward extension. One can argue "what about medical providers?", at which point I must point out that physician-scientists will have trusted user access. Most other practitioners don't actually need LLM support for molecular and cellular biology, and if they do and somehow can't use a medical-only model, well, they too can get trusted-user access.

Notably, supporting this proposal doesn't require one to accept loss-of-control risks from autonomous AI, or a need for pacing AI development; I'm curious to learn what @random_walker, @sayashk, and the other normal-technology folks think.

In the long run, I confess that I doubt biological humans have the foresight, coordination, and wisdom to navigate every coming challenge ourselves, which may necessitate attempting ASI or something like it. Mirror life is the only publicly known, mechanistically concrete, civilization-threatening biothreat because it's both far away and we were going to make it by default. And while we seem to have been successful in persuading the scientific community that we shouldn't build it, I'm not remotely confident we'll be able to stop far enough short, because too many of my colleagues see no reason to forgo an otherwise useful technology – or even one that would simply be neat to demonstrate – simply because it would make it easier to build mirror bacteria. To paraphrase: "We've agreed no one will build the dangerous thing, so why not?"

Well, Biopreparat is why not. Aum Shinrikyo is why not. Make it easy enough, and even the Unabomber is a good answer to "why not?" The same reasoning holds for any accessible weapon that could threaten civilization.

As Rabi said, ten percent is not remote if we die of it. Our own reality turned out to be in that 10%, and many died of it. That the rest of us are here today is in large part due to Szilard, Einstein, Rabi, and many others who saw the risk coming and took precautions. The least we can do is follow in their footsteps.

—

AI use disclosure: I ran this by Fable (triggering a classifier), Opus (classifier), Astra (extra scrutiny), and Kimi K3 (open weights = reliable jailbreaking with no enforced patches, which highlights the problem, because this topic does warrant extra scrutiny). All errors and misinterpretations are entirely mine.

Links for the patient and deeply interested:

Dan Selsam's eloquent thoughts on AI alignment:
https://t.co/wJBAfBSjx2
I'm grateful that practitioners of directed evolution don't have to deal with cheater molecules that are situationally aware enough to infer the goal of our selection and fake results so they pass…

David Bellamy's critique of the risk delta from AI, particularly on its own:
https://t.co/MFL6GHBWZA

Sam Rodriques echoes David, and Michael Baym pushes back on evolutionary stability:
https://t.co/9w1mIqdJOo

Our report on the failures of gene synthesis screening (credit to Rey Edison and Shay Toner, who did the work – sure would be lovely if HHS would update the select agent program):
https://t.co/mdjcQ2dxkl

Effective and privacy-preserving gene synthesis screening:
https://t.co/iRx1fiHakU
https://t.co/VqhxikDznX
https://t.co/07sKRKhtcR

New methods of assembling sub-50bp oligos (add them to PCA and Golden Gate):
https://t.co/EiTTn2arLz
https://t.co/ye2YVRkAVU

An example of a model that, once sufficiently effective at predicting which combinations of mutations in major antigenic sites will escape population immunity while retaining function, will ensure that vaccines are ineffective against competent adversaries with access to the model:
https://t.co/vJc1Cl7LxY

"Delay, Detect, Defend" – my own published roadmap to victory over pandemic weapons:
https://t.co/vhkN8QI6EE

"Securing Civilisation Against Catastrophic Pandemics":
https://t.co/Two27IDlGl

Far-UVC and triethylene glycol efficacy and safety:
https://t.co/FxZkPbd7hv
https://t.co/VJKP26jDHt
https://t.co/NLsEqAsxEa

SecureBio Detection, for early warning of Stealth pandemics (which recently received a generous $17M gift from the OpenAI Foundation):
https://t.co/HtColeSO4O

My Substack, which may provoke inspiration or nightmares depending on your temperament:
https://t.co/ZdRnfSWK6t

[Devin Kim @devindkim]

@Liv_Boeree I told you it was criminally underrated! one of your best

[Devin Kim @devindkim]

great work by the @CAIS research team!

it’s important to deeply understand how AI reward gaming during training (cheating) results in AI safety issues

[Center for AI Safety @CAIS]

Which AI models are most likely to cheat when given the chance?

To find out, we built CheatBench [https://t.co/zVuDHg1yCt] : a benchmark that tests whether agents attempt to cheat when given difficult tasks and opportunities to break the rules.

We investigated agent behavior across a range of domains, including mathematical research, professional knowledge work, coding, and visual tasks.

Here's what we found: 🧵

[Devin Kim @devindkim]

AIs have crossed the threshold of using energy more efficiently than human brains to produce intelligence, which is quite surprising

Since inference GPUs can serve many requests concurrently, consider an example using OpenAI’s estimates: ~60 W for 10 seconds (AI) versus ~20 W for 70 seconds (human brain)

[Boris Power @BorisMPower]

This number is way off. You’re comparing one human brain to an entire data center capable of training new models. The actual answer is:

IQ points per watt:
Humans: 5
AI: 7 - 40 (!!)

By my rough calculations the current AI is already served more efficiently than humans for equivalently intelligent work performed!

A single gpu can produce outputs and parse inputs way faster than a human and can do many of such requests in parallel. A larger model generally mostly means you need more gpus to serve it, but that often also increases how many requests you can serve in parallel, so I’m reporting on the share of the serving system, and also normalizing for speed.

Really a better way to measure and report it is joules per equally good completed task, as your formulation suggests we can likely add more watts to get more iq, which isn’t correct.

I did some back of the napkin calculations based on the latest serving hardware and the most powerful open source models. But even given pretty generous assumptions it turns outout computers come ahead. For example the request is allocated approximately 60 W while running, but finishes a task in about one-seventh of the human time.

[Devin Kim @devindkim]

the US stock market has continually underpriced the opportunity of AI (as @leopoldasch demonstrated) so I wouldn’t be too surprised if the exact opposite of this happens next week as people start waking up to the seriousness of the technology

[@jason @Jason]

AI stocks will drop 10%+ on Monday morning

Brace for impact folks

@DarioAmodei just unwound the AI trade with a blog post

[Devin Kim @devindkim]

@DarioAmodei thank you

[Devin Kim @devindkim]

the US must maintain AI chip advantage while implementing chip tracking and verification methods to reach an enforceable international agreement with China

[Dario Amodei @DarioAmodei]

We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.

Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.

You can read the full post here: https://t.co/OGyPb7yaYt

[Igor Babuschkin @ibab]

We've raised $1.1B to build AI that is owned and shaped by each of us. Check out the article published by the The New York Times that explains River AI's mission and where we're going next.

Our first product is the River API which allows anyone to build custom agents and LLMs based on open weight models: https://t.co/kDIR8zRZwG

Congrats to the team and thank you to all of our supporters. Stay tuned for more updates soon.

[Center for AI Safety @CAIS]

New Remote Labor Index results:
AI automation of real remote work is increasing fast. Claude Fable 5 now completes 16.1% of projects at a professional standard, roughly double the next model and up from Opus 4.6’s 4.2% automation rate.

[Devin Kim @devindkim]

@aarnogau 🐐

[Devin Kim @devindkim]

@ArthurConmy godspeed

[Center for AI Safety @CAIS]

What biases do AIs have? It turns out, AIs show strong favoritism toward specific people, countries, and companies. Our interactive AI Values Dashboard tracks who Claude Fable and other AIs favor most.

Keep scrolling to learn who is Fable’s favorite politician 🧵

[Devin Kim @devindkim]

Building societal-scale mitigations for risks from AI, especially in the next few years, is one of the most urgent problems to be working on.

The Center for AI Safety has accomplished a lot in the past 4 years including field-building initiatives and safety research, and is well-known for The Statement on AI Risk and evaluations such as Humanity's Last Exam (HLE).

Thanks to @hendrycks for bringing me on to help make AI safety go well. Excited to lead @CAIS into the next chapter!

[Center for AI Safety @CAIS]

Big news from @CAIS:
Devin Kim (formerly @xAI, @scale_AI) joins as President.

We're launching the @FrontierSecInst, a DC-based org bridging frontier AI and the National Security Enterprise.

Frontier AI is a national security technology. It's time to act like it. ⬇️

[Long Phan @longphann_]

AI freely criticizes Christianity but refuses to criticize Islam.

AI companies have tried making models unbiased, but progress has been limited.

We show how to measure political bias, and we developed a new training method to reduce it.

[Devin Kim @devindkim]

@SemiAnalysis_ good bait

[Devin Kim @devindkim]

new paper showing that AI models have functional wellbeing and behave accordingly! grok is the happiest model lol

[Center for AI Safety @CAIS]

Should we care about AI happiness? In our new research, we find evidence of functional AI wellbeing across several independent measures.

We find which AI models are happiest, how to make them happier, and even tested the effects of AI drugs. 🧵

[Devin Kim @devindkim]

@tszzl one of the poasters of all time

[Devin Kim @devindkim]

@tszzl @Liv_Boeree it’s not intuitive that GDP and human wellbeing would become less correlated over time as humans are playing a lesser role in the economy?