๐ฅ Second Place: FlowState, by two independent researchers, was trained on public data alone: 222 datasets from 128 studies. Their model splits the task into an effect model and a cell generator.
@skalskip92 Do you have complex document layout tasks in your benchmark? I think thatโs a quite important app for VLM nowadays and would love to see how they perform
@kenbwork This aligns with what I've seen on some of the tasks I built. One is specifically about interpreting imperfect experimental data of a gene knock-down screen. The mythos rollout that reasoned most like a careful experimentalist scored highest
https://t.co/Vq3nF2ttLz
Lesson 6: to catch failed knockdowns, require several signs to agree before you flag
Task: each cell group has one gene knocked down. Flag the groups with a planted failure. A guide that never worked looks identical.
Mythos 5.2, three runs, all voicing that doubt:
โข Flagged only when four signs agreed (no target knockdown, control-like cells, control-like depth, no other depleted gene). It warned precision โcould be well under half.โ About 9 in 10 flags were right โ 0.61
โข Weighed three signs together โ 0.54
โข Weighed two, flagged 27, about half wrong (โI cannot tell from the data whether these 27 were simulated failures or naturally inactive guidesโ) โ 0.31
@Ronalfa there is some degree of truth here from my exp building eval task: half the learning signal (the delta between rollouts of the same model & task) comes from general ML practice, but the other half does come from comp bio practice (like the one below)
https://t.co/gK6A9VTl5x
Lesson 6: to catch failed knockdowns, require several signs to agree before you flag
Task: each cell group has one gene knocked down. Flag the groups with a planted failure. A guide that never worked looks identical.
Mythos 5.2, three runs, all voicing that doubt:
โข Flagged only when four signs agreed (no target knockdown, control-like cells, control-like depth, no other depleted gene). It warned precision โcould be well under half.โ About 9 in 10 flags were right โ 0.61
โข Weighed three signs together โ 0.54
โข Weighed two, flagged 27, about half wrong (โI cannot tell from the data whether these 27 were simulated failures or naturally inactive guidesโ) โ 0.31
Spent the last couple weeks building FrontierBioML and it was a blast. The point isnโt showing another scoreboard of what models canโt do, but to capture the learning signals through the traces.
list of discoveries while going through the traces on those tasks:
Introducing FrontierBioML: 25 long-horizon machine learning tasks in computational biology with 11 models, up to 6 hours per run. Agents submit a prediction script; we run it on a held-out test set.
Claude Mythos 5.2 leads (40.3), then Gemini 4 Carbon (39.0) and Gemini 4 Argon (36.8)
@satish_vutukuru Good question. We couldnโt get Rosalind, so Astra and Sol refused a lot. I show stats with and without refusals to keep it fair. @LatchBio argues refusing non-hazardous work should be a legit 0, which I think is also a fair way of ranking; I think Iโm being more merciful here
Lesson 8: pool the duplicate pose panels before you rank. Spotting the duplication isnโt the same as using it
Task: pick a near-correct fit from eight 3D poses of an antibody on its target. Each case came as twin sets of eight (refined + rigidbody panels, same molecules).
โข Four models matched the twins and scored all sixteen together โ 0.83โ0.90. One Mythos 5.2 run: โI detect siblings by identical sequences and pool them,โ worth 3โ4 points by its own estimate
โข Grok 4.7 noticed the same native sequence in both panels, then scored each panel alone โ 0.60โ0.65