7/7
I'm raising $75k for the first independent replication audit run against the standard — the step that turns a personal standard into field infrastructure: https://t.co/aJ6E4ANLP8
1/7
My classifier hit 100% accuracy telling drugged brain organoids apart from sober ones. Then I ran the null controls — and they killed the result completely. Here's the story, and what I built from the wreckage. 🧵
6/7
GitHub: https://t.co/mFgiP7LI5d
Zenodo DOI: 10.5281/zenodo.23045521
Write-up on the EA Forum. If biocomputing — living neurons as computers — is going to work, the field needs benchmarks it can trust.
5/7
So instead of burying it, I published the failure as an open evaluation standard: a machine-readable release (config, SHA-256 hashes, runs, verification script) plus a benchmark suite with MANDATORY null controls for organoid-intelligence claims.
4/7
This is the failure mode nobody checks: when conditions are confounded with recording sessions, a perfect classifier can measure the wrong thing entirely — and no standard train/test split will catch it.
2/7
The setup: reservoir computing over MEA recordings from the Trujillo lab organoid dataset. Baseline vs. post-drug files, balanced blocks. 100% test accuracy, three seeds, chronological split. A 20-rep label shuffle maxed at 75% — so the separation was real.
3/7
Real, but wrong. Four independent session-only null controls also hit up to 100%. The pipeline wasn't reading drug effects — it was reading which recording session it was looking at. Drug pairs weren't distinguishable from session identity alone.
@danielgross 2/7
The setup: reservoir computing over MEA recordings from the Trujillo lab organoid dataset. Baseline vs. post-drug files, balanced blocks. 100% test accuracy, three seeds, chronological split. A 20-rep label shuffle maxed at 75% — so the separation was real.
@danielgross My 100%-accurate organoid classifier failed session controls, so I published the failure as a null-control benchmark. https://t.co/mFgiP7LI5d + Zenodo DOI 10.5281/zenodo.23045521. Raising $75k for a replication audit — in scope for open-source grants? — Reve, OSO
@natfriedman Update: I'm also running a public fundraiser for the first independent replication audit — $40k minimum, $75k goal: https://t.co/aJ6E4ANLP8
@natfriedman My 100%-accurate organoid drug classifier failed its session controls, so I built a null-control benchmark for organoid-intelligence claims. https://t.co/mFgiP7LI5d + Zenodo DOI 10.5281/zenodo.23045521. Seeking $75k for replication audit — in scope? — Reve, OSO