@ltm I sent this to my friends at Apple and their feedback is “why is he saying eggs with such a hard g” which makes me think they are a little embarrassed about this bad product
Introducing BioResearchBench, a new benchmark focused on scientific reasoning in biology from @OfficeHours built by 100+ PhD biologists in our network.
For models and agents to materially speed up scientific discovery, scientists need to be able hand off work that would take them hours to complete, come back, and find it done right without mistakes.
This work shows that frontier models aren’t there yet. They still struggle to stitch multiple steps together, get the computations right, interpret the data correctly, handle uncertainty, and draw the right conclusions.
We built 50 long-horizon research tasks across 10 subdomains, from genomics and cancer biology to neuroscience and immunology. Each uses multiple data files, takes an expert hours to several days to complete, and is independently reviewed by at least 2 PhD biologists for verifiability and scientific accuracy.
Every frontier model passes fewer than 1 in 5 tasks:
Claude Opus 5.5: 18%
Gemini 4 Argon: 13%
Muse Wonder: 11%
GPT-6 Astra*: 4%
Grok 4.7: 4%
The top models met ~80% of individual rubric criteria, but completed the full end-to-end workflow correctly <20% of the time. Half of the tasks were not solved by any model.
The biggest gaps point to where models need to improve next: computation and data handling (failures on 67% of tasks), handling uncertainty (55%), and reporting complete answers (55%).
An important note on scoring: GPT-6 Astra’s score is skewed lower because every task had to stump Astra (SOTA at the time) during development before it made the final set.
Paper, leaderboard, and per-task results:
https://t.co/RAye2J4cXh
@samar_abedrabbo@alexpryhoda
@conductor_build is there an easy way to get conductor working with the new claude channels? would love if it just spun up on random discord channels on my server
@conductor_build could you make terminal searchable too? love the product but i find myself copy and pasting terminal to vs code to search old logs in the terminal