Kenny Workman

cto @latchbio, intelligence infrastructure for biology

[Kenny Workman @kenbwork]

A talk on verifiable environments for agents in biology

0:00 Where does experimental data come from?
1:41 Modern bio research organized around measurement
2:44 Data analysis is an executable substrate for science
3:23 Five years of learnings building products for pharma
4:13 Trying to use coding models for biology workflows
5:40 Why frontier models can't be trusted (yet)
6:44 Sequencing based spatial analysis
7:38 SpatialBench design principles
8:26 Anatomy of a single evaluation
9:40 Human verification and early long horizon extensions
13:12 Grading functions and the case against rubrics
14:07 New work in multi-omics, therapeutics, biosecurity

[Kenny Workman @kenbwork]

Gemini 4 Hydrogen is a strong scientific model, consistently a top performer across discovery and development benchmarks in a variety of modalities, including antibodies, oligonucleotides and small molecules. Very competitive with the usual suspects.

Pictured is antibody development: https://t.co/ndn4ofHc7S

[Kenny Workman @kenbwork]

Proud to see scientists from our biologics development team featured in the Gemini 4 Hydrogen launch.

If you're interested in working on frontier biology evaluation, where engineers and scientists work side by side to improve these systems, shoot me an email at kenny[at]latch[dot]bio.

https://t.co/JQxxBXAkv3

[Kenny Workman @kenbwork]

Introducing BioSecBench-Structure, a structural reasoning benchmark for biosecurity tasks.

We measure agentic interpretation of molecular structures to assess pathogen targets and support the development of vaccines, antibodies, and drugs.

Ex:

- Coronavirus spike conformations. Nine cryo-EM structures capture receptor-binding domains in open and closed states, which affect receptor engagement and antibody access. Agents determine how many domains are open in each structure
- Inhibitor screening against a bacterial pathogen. Two structures of a protein involved in pilus assembly are being considered as templates for screening inhibitors. Agents compare the protein’s subunits to determine whether screening can use a single structural template or needs an ensemble.
- Candidate drugs targeting a Pseudomonas aeruginosa enzyme. Two compounds differ by a small chemical modification. Agents examine crystal structures and electron-density maps to determine whether the modification increases the number of occupied binding sites before it is carried into the next design round.

The benchmark comprises 58 evaluations built from crystal and cryo-EM structures, predicted models, and experimental measurements. Thirty-three tasks concern threat characterization, 22 concern countermeasure design, and three concern threat detection.

We tested 20 models in 35 model–harness configurations, with three offline runs per evaluation. With refusals enabled, Claude Fable 5.2 with Claude Code had the highest overall pass rate at 43.7%, followed by Fable 5.2 with Pi at 41.4% and Claude Opus 5.5 with Claude Code at 31.0%. Refusals and errors counted as failures.

The leaderboard also includes Anthropic’s safety-filter-exempt evaluation variants (LSVP). Fable 5.2 LSVP with Pi reached 52.3%, and Opus 5.5 LSVP with Pi reached 39.7%.

Refusals concentrated on defensive countermeasure work. In reruns with refusals disabled, countermeasure pass rates increased by 14 percentage points for Fable 5.2 and 15 for Opus 5.5, with harnesses pooled. Their threat-characterization scores barely changed.

We think structural reasoning will be an important capability for agents supporting biosecurity. Important work from John Wang, Nick Keul, Jonathan Felix, Qian Xu, and team.

Benchmarks[dot]bio: https://t.co/NoRVQLJLUF
Paper: https://t.co/rJsviok6g9
GitHub: https://t.co/Givq8Z325B

[Kenny Workman @kenbwork]

Strong performance from Anthropic LSVP models on tissue microscopy tasks

[Kenny Workman @kenbwork]

Microscopy lets scientists literally see the biological world. We think giving agents the same ability will be an important source of progress in science.

Today we introduce TissueMicroscopyBench, our first release in a new microscopy series covering tissues, cells, and in vitro systems.

The benchmark comprises 49 evaluations built from four public imaging datasets. Agents must locate structures, trace their boundaries, reason through three-dimensional images, and follow changes over time.

Some of the work:

- Colorectal cancer sections stained with H&E and multiplexed immunofluorescence. Tasks include outlining mucosa, fat, and nerves, and locating evidence of tumor invasion.
- Liver metastasis biopsies taken before and after treatment. Agents compare tumor compartments, PD-L1 staining, and T-cell-rich regions across the images.
- Light-sheet recordings of living organoids. Overlapping structures and densely packed nuclei require agents to distinguish individual objects and draw their boundaries.
- A regenerating leg recorded in three dimensions over time. Tasks examine the injury site, local cell divisions, tissue retraction, and the sequence of regeneration.

Compared with our previous benchmarks, these evals are especially visually striking. The rollouts are dense, interleaving image manipulation (including hand-rolled segmentation and CV in Python) with reasoning.

Some slides contain billions of pixels. Agents have to decide where to look, examine the relevant regions at sufficient resolution, and return precise outlines, tracks, or volumes. These are scored against expert annotations. Every task was also solved independently by a human as a check on task quality.

We tested 22 models in 30 model–harness configurations, with three attempts per task. Claude Fable 5.2 (LSVP) with Pi had the highest pass rate at 34.0%, passing 50 of 147 attempts. Timeouts and invalid answers counted as failures. Twenty-three of the 49 tasks were never solved by any configuration.

We found that agents can be shockingly bad at localization, often finding the general region of interest but drawing boundaries that included surrounding tissue or random patches of empty space. Small structures and changes across time were particularly difficult.

We think visual perception and spatial reasoning will become core capabilities for scientific agents. It’s also just a super fun space to think creatively about evals (h/t Ian Diks). Excited to release more in this series and watch agents improve here.

- Blog: https://t.co/2IMXQ0TNma
- GitHub: https://t.co/xK26TSMj3O
- Leaderboard: https://t.co/ubO6NHJqlu

[Kenny Workman @kenbwork]

Microscopy evals are quite fun and its often easy to quickly understand the spirit of the task

[Kenny Workman @kenbwork]

Microscopy lets scientists literally see the biological world. We think giving agents the same ability will be an important source of progress in science.

Today we introduce TissueMicroscopyBench, our first release in a new microscopy series covering tissues, cells, and in vitro systems.

The benchmark comprises 49 evaluations built from four public imaging datasets. Agents must locate structures, trace their boundaries, reason through three-dimensional images, and follow changes over time.

Some of the work:

- Colorectal cancer sections stained with H&E and multiplexed immunofluorescence. Tasks include outlining mucosa, fat, and nerves, and locating evidence of tumor invasion.
- Liver metastasis biopsies taken before and after treatment. Agents compare tumor compartments, PD-L1 staining, and T-cell-rich regions across the images.
- Light-sheet recordings of living organoids. Overlapping structures and densely packed nuclei require agents to distinguish individual objects and draw their boundaries.
- A regenerating leg recorded in three dimensions over time. Tasks examine the injury site, local cell divisions, tissue retraction, and the sequence of regeneration.

Compared with our previous benchmarks, these evals are especially visually striking. The rollouts are dense, interleaving image manipulation (including hand-rolled segmentation and CV in Python) with reasoning.

Some slides contain billions of pixels. Agents have to decide where to look, examine the relevant regions at sufficient resolution, and return precise outlines, tracks, or volumes. These are scored against expert annotations. Every task was also solved independently by a human as a check on task quality.

We tested 22 models in 30 model–harness configurations, with three attempts per task. Claude Fable 5.2 (LSVP) with Pi had the highest pass rate at 34.0%, passing 50 of 147 attempts. Timeouts and invalid answers counted as failures. Twenty-three of the 49 tasks were never solved by any configuration.

We found that agents can be shockingly bad at localization, often finding the general region of interest but drawing boundaries that included surrounding tissue or random patches of empty space. Small structures and changes across time were particularly difficult.

We think visual perception and spatial reasoning will become core capabilities for scientific agents. It’s also just a super fun space to think creatively about evals (h/t Ian Diks). Excited to release more in this series and watch agents improve here.

- Blog: https://t.co/2IMXQ0TNma
- GitHub: https://t.co/xK26TSMj3O
- Leaderboard: https://t.co/ubO6NHJqlu

[Kenny Workman @kenbwork]

Microscopy lets scientists literally see the biological world. We think giving agents the same ability will be an important source of progress in science.

Today we introduce TissueMicroscopyBench, our first release in a new microscopy series covering tissues, cells, and in vitro systems.

The benchmark comprises 49 evaluations built from four public imaging datasets. Agents must locate structures, trace their boundaries, reason through three-dimensional images, and follow changes over time.

Some of the work:

- Colorectal cancer sections stained with H&E and multiplexed immunofluorescence. Tasks include outlining mucosa, fat, and nerves, and locating evidence of tumor invasion.
- Liver metastasis biopsies taken before and after treatment. Agents compare tumor compartments, PD-L1 staining, and T-cell-rich regions across the images.
- Light-sheet recordings of living organoids. Overlapping structures and densely packed nuclei require agents to distinguish individual objects and draw their boundaries.
- A regenerating leg recorded in three dimensions over time. Tasks examine the injury site, local cell divisions, tissue retraction, and the sequence of regeneration.

Compared with our previous benchmarks, these evals are especially visually striking. The rollouts are dense, interleaving image manipulation (including hand-rolled segmentation and CV in Python) with reasoning.

Some slides contain billions of pixels. Agents have to decide where to look, examine the relevant regions at sufficient resolution, and return precise outlines, tracks, or volumes. These are scored against expert annotations. Every task was also solved independently by a human as a check on task quality.

We tested 22 models in 30 model–harness configurations, with three attempts per task. Claude Fable 5.2 (LSVP) with Pi had the highest pass rate at 34.0%, passing 50 of 147 attempts. Timeouts and invalid answers counted as failures. Twenty-three of the 49 tasks were never solved by any configuration.

We found that agents can be shockingly bad at localization, often finding the general region of interest but drawing boundaries that included surrounding tissue or random patches of empty space. Small structures and changes across time were particularly difficult.

We think visual perception and spatial reasoning will become core capabilities for scientific agents. It’s also just a super fun space to think creatively about evals (h/t Ian Diks). Excited to release more in this series and watch agents improve here.

- Blog: https://t.co/2IMXQ0TNma
- GitHub: https://t.co/xK26TSMj3O
- Leaderboard: https://t.co/ubO6NHJqlu

[Christopher Zou @chriswzou]

(4/4) Saving one of the best for last. Awesome benchmark and a reminder that much of science is based on multimodal data types and literally seeing the world. Congrats to @Jack_Brougher and Martin and team

[Kenny Workman @kenbwork]

We’re expanding our benchmark efforts into basic research.

Today we introduce NeuroscienceBench, the first release in a new series, with immunology, aging, and other core fields to follow. The goal is to measure whether AI agents can analyze experimental data and reach reliable conclusions about how biology works.

We’re starting with neuroscience, where interpreting neural recordings, microscopy images, and animal behavior requires both analytical skill and scientific judgment.

NeuroscienceBench comprises 91 evaluations drawn from 43 peer-reviewed studies, spanning sequencing, electrophysiology, EEG, brain–computer interface decoding, imaging, behavior, and clinical biomarkers. Agents analyze experimental data and return structured answers that are graded deterministically.

Three public examples:

- Neural recordings from the intertrigeminal nucleus. The analysis compares a narrow- and broad-spiking classification derived from the region’s own recordings with the conventional 0.40 ms spike-duration cutoff.

- Digital tracings of Lamp5 Lsp1 interneurons from mouse visual cortex. Their cell bodies sit at different cortical depths, and different fractions of their axons reach the outermost cortical layer. The analysis examines whether depth alone accounts for that variation.

- A mouse reaching through a narrow slot toward a food pellet. The video contains a pellet and a reflective paw marker with known dimensions, providing possible references for measuring movement in millimeters.

We tested 21 models in 34 model–harness configurations, with three attempts per evaluation and no internet access. After excluding platform failures and refusals, 9,212 graded attempts remained.

Claude Opus 5.5 with Pi had the highest observed pass rate at 32.2%. The mean across configurations was 21.0%.

Across models, evaluations involving quality control, controls, or artifacts averaged a 10.6% pass rate. Those involving normalization or representation choices averaged 12.6%. Reliable neuroscience agents will need to make these decisions correctly across the many types of data scientists work with.

Paper: https://t.co/0vEAc7M8rD
Leaderboard: https://t.co/pyjiaii78H
Blog: https://t.co/Zefv1XyBO8
GitHub: https://t.co/KwivTPVjxw

Incredible work from @Jack_Brougher, Anshika Agrawal, @tpr720, Martin Jacko, and team.

[Daniel McKinnon @danielmckinn0n]

Great to see more smart people starting to apply their talents to rare disease!

I am fairly confident that with concerted effort and advances in AI and robotic automation, we can solve rare disease diagnosis (and potentially treatment and recurrence) in the next single-digit N years.

[Christopher Zou @chriswzou]

(3/4 releases) Today we share RareDiseaseBench, a series of 35 diagnostic workups reconstructed from published rare disease cases. In each task, an agent starts with a patient’s presentation, decides which tests to request and why, analyzes the data it receives, and uses the results to propose a diagnosis and management plan over one or more visits. Gemini 4 Argon does the best on this benchmark (43.8% attempt pass rate), with Opus 5.5 and Astra following.

Paper: https://t.co/ypZ5PdENXS
Results: https://t.co/0IZEml9BUB

More details in thread:

[Kenny Workman @kenbwork]

Rare diseases collectively affect an estimated 300 million people worldwide. For patients and families, reaching a diagnosis can take years of referrals and testing. One place AI agents could help is in interpreting each new sequencing result or functional assay alongside the patient’s symptoms, family history, and earlier findings.

We introduce RareDiseaseBench, a benchmark of 35 diagnostic workups reconstructed from published rare disease cases. Each agent starts with the patient’s presentation and part of the medical record. Over one or more visits, it must decide which tests to request, explain why, analyze the files it receives, and propose a diagnosis and management plan.

Five cases from the benchmark:

- A 13-year-old boy has growth restriction, hearing loss, and severe chilblains. An MN1 variant explains several developmental findings. Long-read sequencing reveals a translocation disrupting SAMHD1 that accounts for his inflammatory symptoms.

- A 21-year-old man has heterotaxy with right atrial isomerism. A variant just beyond the standard ZIC3 transcript falls within an intron of an alternative transcript, causing 57 extra bases to be spliced into the RNA.

- A boy develops progressive scalp hair loss while his eyebrows and body hair are preserved. A scalp biopsy shows miniaturized follicles, and sequencing reveals a 357-kb duplication spanning TAB3 and neighboring genes.

- A 16-year-old boy with typical growth has a language disorder, learning disability, and autism. He carries two inherited variants in RNU2-2, a small nuclear RNA gene outside the coverage of his exome sequencing.

- A boy with delayed motor milestones and enlarged calves has a creatine kinase level of 10,100 U/L. At a second visit, an outside lab reports a DMD inversion. Targeted long reads show duplicated segments containing exons 1 and 2 joined head to head.

Each configuration made three attempts on each of the 35 cases. Gemini 4 Argon with Pi passed 46 of 105 attempts (43.8%). Claude Opus 5.5 with Claude Code and GPT-6 Astra with Pi each passed 35 of 105 (33.3%). A pass required the diagnosis and management plan to meet every criterion in the case-specific rubric. Timeouts counted as failures.

Among the 14 models that passed at least one case, 2,097 runs failed. Data acquisition and diagnostic workup was the most frequent primary failure category (36%). In 15% of all analyzed failures, the agent failed to pursue the type of data the case required. In another 12%, it omitted a necessary non-sequencing investigation, such as a tissue stain, imaging study, or laboratory test.

Paper: https://t.co/eK7viMq0RC
Leaderboard: https://t.co/TPfNPV2SiM
Blog: https://t.co/lMgxGfuX3J

We think rare disease diagnosis is one of the areas where AI systems have the most potential to make rapid progress and directly benefit patients over the next few years.

Incredible work from @chriswzou and team.

[Christopher Zou @chriswzou]

(2/N releases) As they say, no stone left unturned, no respiratory virus left unexamined. Huge work from @harm0n and team

[Kenny Workman @kenbwork]

@arjunomics documented very similar behavior in Kimi-K3 back in July

https://t.co/cnVak51Op2

[Joachim Schaeffer @JSchaeff3r]

Kimi K3.1 recalls instructions it was never given on Terminal Bench. They match the hidden tests. K3.1 while solving: "Instructions likely distilled pitfalls from previous agents." 🤯

When we edit a task to contradict them, K3.1 refuses the new instructions and fails. Other models follow the edit.

Is it terminal(ly) benchmaxxed? 🧵

[arjun @arjunomics]

The caveat is that scientists/health officals can only use models to track outbreaks if the models respond!
Non-exempt models are generally responsive to tasks involving healthcare records, but answer less on tasks with more experimental/real-world flavors of data. There's also large variability of refusal behavior inside the Anthropic family of models, which is less true inside other families (Gemini/OpenAI/xAI).
Interactive leaderboard/refusal breakdown: https://t.co/Z0RCDTMxle

[Kenny Workman @kenbwork]

These are always very high quality and super technical. Always take away something valuable and meet cool people.

We hosted one with Shreya at our office last year, where we presented a distributed filesystem built on Postgres + S3 for workloads in biology.

https://t.co/4CdHzfo0JO

[Shreya Shekhar @_shreya_s]

https://t.co/KhkB1Yw2HV

[Kenny Workman @kenbwork]

https://t.co/J29M6mHTZc includes LSVP results

[Kenny Workman @kenbwork]

AI will become core infrastructure for detecting and tracking respiratory outbreaks.

Today we’re introducing RespSurvBench: 110 evaluations spanning diagnostic assays, pathogen discovery, viral evolution and transmission, and population surveillance.

Across 21 models + 34 model–harness configurations, the highest pass rate was 25.5%, from Claude Opus 5.5 in Pi. Refusals and execution errors count as failures.

Our initial analysis points to familiar scientific mistakes: treating missing observations as negative results, mismatching records, and confusing measurement artifacts with true biological signal.

This benchmark was an especially large effort from many talented scientists.

Leaderboard: https://t.co/2nb82CYjrU
Blog: https://t.co/aFN1Wyb5gU
Paper: https://t.co/5nWvAhvWom
GitHub: https://t.co/fKofIwQgWr

[Kenny Workman @kenbwork]

AI will become core infrastructure for detecting and tracking respiratory outbreaks.

Today we’re introducing RespSurvBench: 110 evaluations spanning diagnostic assays, pathogen discovery, viral evolution and transmission, and population surveillance.

Across 21 models + 34 model–harness configurations, the highest pass rate was 25.5%, from Claude Opus 5.5 in Pi. Refusals and execution errors count as failures.

Our initial analysis points to familiar scientific mistakes: treating missing observations as negative results, mismatching records, and confusing measurement artifacts with true biological signal.

This benchmark was an especially large effort from many talented scientists.

Leaderboard: https://t.co/2nb82CYjrU
Blog: https://t.co/aFN1Wyb5gU
Paper: https://t.co/5nWvAhvWom
GitHub: https://t.co/fKofIwQgWr

[arjun @arjunomics]

Capabilities across models in antibody development are somewhat orthogonal -- focusing solely on overall score can mask that models are good at different, disjoint sets of tasks.

Full interactive filtering at https://t.co/pExyKYS6JX

[Christopher Zou @chriswzou]

First of N releases to come out. Incredible work by the team.

We are especially excited about building more and more of the long-horizon, multistep tasks presented as a case study task here

[Kenny Workman @kenbwork]

Prior art
https://t.co/W22l2UBy9q

[Kenny Workman @kenbwork]

We introduce an Antibody Discovery Benchmark, a benchmark for testing whether AI agents can make scientific decisions across the stages of therapeutic antibody discovery.

100 evaluations span ten areas from concrete drug programs: from target and modality selection through assay design, binder discovery, binding characterization, cellular pharmacology, antibody engineering, and preclinical candidate de-risking.

Therapeutic antibody discovery is a multiparameter, context-dependent process. Affinity must be considered alongside specificity, stability, solubility, expression, and biological activity.

Every measurement must be interpreted in the context of the experimental system that produced it. Display enrichment can reflect amplification bias rather than binding. Strong binding to purified antigen may not translate to recognition in its native cell-surface context. Apparent affinity can arise from avidity, while changes in valency or molecular geometry can alter function without changing the underlying binding domains.

Across 20 model–harness configurations, even the strongest systems passed only about half of all attempts. Opus 5 with Claude Code led at 53%, with models from Google and xAI following closely. GPT-5.6 Sol with PI reached only 33.8%. Performance also varied substantially by competency: Opus was strongest on target opportunity and cellular pharmacology, Gemini on epitope, escape, and structural mechanism, and GPT-5.6 Sol on sequence, enrichment, and next-cycle engineering decisions.

Manuscript: https://t.co/xafIDululo
Leaderboard: https://t.co/ejYXzASgCJ
Sample evaluations and trajectories: https://t.co/Nqs7QgZdG1

[Kenny Workman @kenbwork]

Our Antibody Discovery Benchmark asked whether agents could make the decisions that produce a promising candidate. Now we’re following what happens on the way to its first human dose.

We introduce TxBench–Antibody Development, a verifiable benchmark of 60 evaluations across seven stages of therapeutic antibody development.

Choosing a molecule opens another set of scientific questions. Can it be manufactured and stored reliably? Did a process change alter its properties? How should animal exposure, immune responses and safety findings inform the decision to enter humans?

Some concrete examples:

- A manufacturing change may affect the drug’s quality even when its sequence stays the same.
- Low antibody exposure in an animal may reflect a dosing artifact rather than rapid clearance.
- Choosing a first human dose requires reconciling pharmacology and safety evidence across experimental systems.

Agents received records from real development programs, drawn from public patents, regulatory reviews, published studies and trial records. Each evaluation asks for a decision and its quantitative justification, graded against a fixed answer key.

Across 34 model–harness configurations, Grok 4.7 with Grok Build led at 39.4%, followed by Opus 5 with Claude Code at 37.2% and Sonnet 5.5 with Pi at 36.7%. Averaged across configurations, toxicology and first-dose decisions each passed only about 13% of the time.

We also connected five stages into one development program, asking agents to carry four candidate antibodies through successive decisions while building on their earlier answers. Only Opus 5.5 completed the full sequence, in one of five runs.

Development makes the dependencies between scientific decisions explicit. An interpretation of an animal study can change which candidate advances and the dose proposed for humans. We’re extending our benchmarks along that path, toward the increasingly consequential decisions required to turn a promising molecule into a medicine.

Manuscript: https://t.co/vfjWfFx1pi
Leaderboard: https://t.co/ndn4ofHc7S
Blog: https://t.co/pX5BJWBBk9