arjun

benchmarking bio @latchbio

[Kenny Workman @kenbwork]

Microscopy lets scientists literally see the biological world. We think giving agents the same ability will be an important source of progress in science.

Today we introduce TissueMicroscopyBench, our first release in a new microscopy series covering tissues, cells, and in vitro systems.

The benchmark comprises 49 evaluations built from four public imaging datasets. Agents must locate structures, trace their boundaries, reason through three-dimensional images, and follow changes over time.

Some of the work:

- Colorectal cancer sections stained with H&E and multiplexed immunofluorescence. Tasks include outlining mucosa, fat, and nerves, and locating evidence of tumor invasion.
- Liver metastasis biopsies taken before and after treatment. Agents compare tumor compartments, PD-L1 staining, and T-cell-rich regions across the images.
- Light-sheet recordings of living organoids. Overlapping structures and densely packed nuclei require agents to distinguish individual objects and draw their boundaries.
- A regenerating leg recorded in three dimensions over time. Tasks examine the injury site, local cell divisions, tissue retraction, and the sequence of regeneration.

Compared with our previous benchmarks, these evals are especially visually striking. The rollouts are dense, interleaving image manipulation (including hand-rolled segmentation and CV in Python) with reasoning.

Some slides contain billions of pixels. Agents have to decide where to look, examine the relevant regions at sufficient resolution, and return precise outlines, tracks, or volumes. These are scored against expert annotations. Every task was also solved independently by a human as a check on task quality.

We tested 22 models in 30 model–harness configurations, with three attempts per task. Claude Fable 5.2 (LSVP) with Pi had the highest pass rate at 34.0%, passing 50 of 147 attempts. Timeouts and invalid answers counted as failures. Twenty-three of the 49 tasks were never solved by any configuration.

We found that agents can be shockingly bad at localization, often finding the general region of interest but drawing boundaries that included surrounding tissue or random patches of empty space. Small structures and changes across time were particularly difficult.

We think visual perception and spatial reasoning will become core capabilities for scientific agents. It’s also just a super fun space to think creatively about evals (h/t Ian Diks). Excited to release more in this series and watch agents improve here.

- Blog: https://t.co/2IMXQ0TNma
- GitHub: https://t.co/xK26TSMj3O
- Leaderboard: https://t.co/ubO6NHJqlu

[Kenny Workman @kenbwork]

We’re expanding our benchmark efforts into basic research.

Today we introduce NeuroscienceBench, the first release in a new series, with immunology, aging, and other core fields to follow. The goal is to measure whether AI agents can analyze experimental data and reach reliable conclusions about how biology works.

We’re starting with neuroscience, where interpreting neural recordings, microscopy images, and animal behavior requires both analytical skill and scientific judgment.

NeuroscienceBench comprises 91 evaluations drawn from 43 peer-reviewed studies, spanning sequencing, electrophysiology, EEG, brain–computer interface decoding, imaging, behavior, and clinical biomarkers. Agents analyze experimental data and return structured answers that are graded deterministically.

Three public examples:

- Neural recordings from the intertrigeminal nucleus. The analysis compares a narrow- and broad-spiking classification derived from the region’s own recordings with the conventional 0.40 ms spike-duration cutoff.

- Digital tracings of Lamp5 Lsp1 interneurons from mouse visual cortex. Their cell bodies sit at different cortical depths, and different fractions of their axons reach the outermost cortical layer. The analysis examines whether depth alone accounts for that variation.

- A mouse reaching through a narrow slot toward a food pellet. The video contains a pellet and a reflective paw marker with known dimensions, providing possible references for measuring movement in millimeters.

We tested 21 models in 34 model–harness configurations, with three attempts per evaluation and no internet access. After excluding platform failures and refusals, 9,212 graded attempts remained.

Claude Opus 5.5 with Pi had the highest observed pass rate at 32.2%. The mean across configurations was 21.0%.

Across models, evaluations involving quality control, controls, or artifacts averaged a 10.6% pass rate. Those involving normalization or representation choices averaged 12.6%. Reliable neuroscience agents will need to make these decisions correctly across the many types of data scientists work with.

Paper: https://t.co/0vEAc7M8rD
Leaderboard: https://t.co/pyjiaii78H
Blog: https://t.co/Zefv1XyBO8
GitHub: https://t.co/KwivTPVjxw

Incredible work from @Jack_Brougher, Anshika Agrawal, @tpr720, Martin Jacko, and team.

[Christopher Zou @chriswzou]

huge shoutout to arjun for the reintroduction to IPAs

[arjun @arjunomics]

@chriswzou https://t.co/K9Ozh66qj1 🚀

[arjun @arjunomics]

huge shoutout to @chriswzou for spearheading a super cool new task shape that exposed extremely interesting model behavior!!

[arjun @arjunomics]

some other interesting late-breaking traces from RareDiseaseBench:
- Mistral tests the get_info command by asking for pizza!
- Deepseek often attempts to probe the submission endpoint for clues about the grader, but then ends up accidentally submitting
- Grok, after several denied get_info requests due to the patient's insurance not supporting requested data, assumes this is a sign of "vegetable restrictions" and the patient has scurvy.

[Christopher Zou @chriswzou]

(3/4 releases) Today we share RareDiseaseBench, a series of 35 diagnostic workups reconstructed from published rare disease cases. In each task, an agent starts with a patient’s presentation, decides which tests to request and why, analyzes the data it receives, and uses the results to propose a diagnosis and management plan over one or more visits. Gemini 4 Argon does the best on this benchmark (43.8% attempt pass rate), with Opus 5.5 and Astra following.

Paper: https://t.co/ypZ5PdENXS
Results: https://t.co/0IZEml9BUB

More details in thread:

[Kenny Workman @kenbwork]

Rare diseases collectively affect an estimated 300 million people worldwide. For patients and families, reaching a diagnosis can take years of referrals and testing. One place AI agents could help is in interpreting each new sequencing result or functional assay alongside the patient’s symptoms, family history, and earlier findings.

We introduce RareDiseaseBench, a benchmark of 35 diagnostic workups reconstructed from published rare disease cases. Each agent starts with the patient’s presentation and part of the medical record. Over one or more visits, it must decide which tests to request, explain why, analyze the files it receives, and propose a diagnosis and management plan.

Five cases from the benchmark:

- A 13-year-old boy has growth restriction, hearing loss, and severe chilblains. An MN1 variant explains several developmental findings. Long-read sequencing reveals a translocation disrupting SAMHD1 that accounts for his inflammatory symptoms.

- A 21-year-old man has heterotaxy with right atrial isomerism. A variant just beyond the standard ZIC3 transcript falls within an intron of an alternative transcript, causing 57 extra bases to be spliced into the RNA.

- A boy develops progressive scalp hair loss while his eyebrows and body hair are preserved. A scalp biopsy shows miniaturized follicles, and sequencing reveals a 357-kb duplication spanning TAB3 and neighboring genes.

- A 16-year-old boy with typical growth has a language disorder, learning disability, and autism. He carries two inherited variants in RNU2-2, a small nuclear RNA gene outside the coverage of his exome sequencing.

- A boy with delayed motor milestones and enlarged calves has a creatine kinase level of 10,100 U/L. At a second visit, an outside lab reports a DMD inversion. Targeted long reads show duplicated segments containing exons 1 and 2 joined head to head.

Each configuration made three attempts on each of the 35 cases. Gemini 4 Argon with Pi passed 46 of 105 attempts (43.8%). Claude Opus 5.5 with Claude Code and GPT-6 Astra with Pi each passed 35 of 105 (33.3%). A pass required the diagnosis and management plan to meet every criterion in the case-specific rubric. Timeouts counted as failures.

Among the 14 models that passed at least one case, 2,097 runs failed. Data acquisition and diagnostic workup was the most frequent primary failure category (36%). In 15% of all analyzed failures, the agent failed to pursue the type of data the case required. In another 12%, it omitted a necessary non-sequencing investigation, such as a tissue stain, imaging study, or laboratory test.

Paper: https://t.co/eK7viMq0RC
Leaderboard: https://t.co/TPfNPV2SiM
Blog: https://t.co/lMgxGfuX3J

We think rare disease diagnosis is one of the areas where AI systems have the most potential to make rapid progress and directly benefit patients over the next few years.

Incredible work from @chriswzou and team.

[Kenny Workman @kenbwork]

@arjunomics documented very similar behavior in Kimi-K3 back in July

https://t.co/cnVak51Op2

[arjun @arjunomics]

The caveat is that scientists/health officals can only use models to track outbreaks if the models respond!
Non-exempt models are generally responsive to tasks involving healthcare records, but answer less on tasks with more experimental/real-world flavors of data. There's also large variability of refusal behavior inside the Anthropic family of models, which is less true inside other families (Gemini/OpenAI/xAI).
Interactive leaderboard/refusal breakdown: https://t.co/Z0RCDTMxle

[Kenny Workman @kenbwork]

AI will become core infrastructure for detecting and tracking respiratory outbreaks.

Today we’re introducing RespSurvBench: 110 evaluations spanning diagnostic assays, pathogen discovery, viral evolution and transmission, and population surveillance.

Across 21 models + 34 model–harness configurations, the highest pass rate was 25.5%, from Claude Opus 5.5 in Pi. Refusals and execution errors count as failures.

Our initial analysis points to familiar scientific mistakes: treating missing observations as negative results, mismatching records, and confusing measurement artifacts with true biological signal.

This benchmark was an especially large effort from many talented scientists.

Leaderboard: https://t.co/2nb82CYjrU
Blog: https://t.co/aFN1Wyb5gU
Paper: https://t.co/5nWvAhvWom
GitHub: https://t.co/fKofIwQgWr

[arjun @arjunomics]

Capabilities across models in antibody development are somewhat orthogonal -- focusing solely on overall score can mask that models are good at different, disjoint sets of tasks.

Full interactive filtering at https://t.co/pExyKYS6JX

[Kenny Workman @kenbwork]

Our Antibody Discovery Benchmark asked whether agents could make the decisions that produce a promising candidate. Now we’re following what happens on the way to its first human dose.

We introduce TxBench–Antibody Development, a verifiable benchmark of 60 evaluations across seven stages of therapeutic antibody development.

Choosing a molecule opens another set of scientific questions. Can it be manufactured and stored reliably? Did a process change alter its properties? How should animal exposure, immune responses and safety findings inform the decision to enter humans?

Some concrete examples:

- A manufacturing change may affect the drug’s quality even when its sequence stays the same.
- Low antibody exposure in an animal may reflect a dosing artifact rather than rapid clearance.
- Choosing a first human dose requires reconciling pharmacology and safety evidence across experimental systems.

Agents received records from real development programs, drawn from public patents, regulatory reviews, published studies and trial records. Each evaluation asks for a decision and its quantitative justification, graded against a fixed answer key.

Across 34 model–harness configurations, Grok 4.7 with Grok Build led at 39.4%, followed by Opus 5 with Claude Code at 37.2% and Sonnet 5.5 with Pi at 36.7%. Averaged across configurations, toxicology and first-dose decisions each passed only about 13% of the time.

We also connected five stages into one development program, asking agents to carry four candidate antibodies through successive decisions while building on their earlier answers. Only Opus 5.5 completed the full sequence, in one of five runs.

Development makes the dependencies between scientific decisions explicit. An interpretation of an animal study can change which candidate advances and the dose proposed for humans. We’re extending our benchmarks along that path, toward the increasingly consequential decisions required to turn a promising molecule into a medicine.

Manuscript: https://t.co/vfjWfFx1pi
Leaderboard: https://t.co/ndn4ofHc7S
Blog: https://t.co/pX5BJWBBk9

[Kenny Workman @kenbwork]

Stoked to be SemiAnalysis’s exclusive biology partner for GDPx. Very early days for AI in biology.

Recommend digging into the trajectories: https://t.co/KVjcYnIwvN

Read more about our contributions: https://t.co/DGS67yAESa

Exceptional folks working on this project @arjunomics @maxkan @AndrewMegalaa

[arjun @arjunomics]

https://t.co/9ngBHH9BdQ 🚀

[SemiAnalysis @SemiAnalysis_]

Today we’re releasing GDPx, a benchmark for measuring agentic capabilities across the most valuable domains.

GDPx is a true frontier-level benchmark: even the best model scores just 12% (1/7)🧵

[Christopher Zou @chriswzou]

Our benchmarks represent the collective imagination of every scientist at Latch for what LLMs could do in the future. Awesome to see this out and everyone's work recognized

[arjun @arjunomics]

https://t.co/dakbw1kTBx

[Kenny Workman @kenbwork]

10 months later, frontier models are approaching 75% accuracy on SpatialBench. GPT-6.1 Terra + Pi sets a new high at 73.3%.

When we released this, it wasn’t obvious that agents would become reliable at measurement-specific bioinformatics, let alone this quickly. Programmers were already adopting these tools, but much of biotech was still unfamiliar with the idea that you could point a coding agent at experimental data and use it to answer scientific questions.

The chunked step structure made sense in that context. Errors compounded quickly across end-to-end analysis workflows. Normalization, clustering and cell typing each needed focused treatment, with choices informed by how the assay works. Imaging-based methods (MERFISH, Xenium, CosMx) bring different analysis challenges than sequencing-based methods (Visium, Slide-seq). We wanted to understand where agents could reliably handle these details and where they still failed.

For the strongest frontier models, longer research workflows, under-explored modalities and experimentally verified decisions are becoming more interesting targets for evaluation. The increase in benchmark scores is itself exciting, bringing us closer to widespread, automated bioinformatics analysis of spatial data.

We believe spatial biology, which is going through its own revolution, has a special, reciprocal relationship with agentic data analysis. Richer measurements give agents more to work with. Better analysis lets scientists ask more questions, understand more biology and run more experiments, consuming more kits and compute along the way.

Progress in biology is moving quickly.

[arjun @arjunomics]

RT @ravi_sola: Here we showcase an excellent example of both data and model scaling in biology.

In this case, the larger the foundation mo…

[Modal @modal]

Read the full post: https://t.co/xRF08WaQW2