Midnight reflexion on to improve scientific evaluation beyond h-index: how about a veting mechanism with pagerank based on e.g ORCID profiles.
Each researcher can vouch for any number of other researchers. The importance of a researcher is then simply their pagerank.
Don't be a sore loser.
It's not "mode collapse dressed up as performance", it's "illustration of mode collapse pointing to a problem in your benchmark".
I like the new proposal put forward, not perfect because it forgets the evaluation of the error, but def a better benchmark.
If you are at #ICML2026, check out our paper “Needles in the Haystack”!
Our paper tries to address an uncomfortable finding in the field of "virtual cells": predicting the mean outperforms many scFMs such as GEARS, scGPT, scLambda on standard perturbation benchmarks. No learning required.
Not because the models are bad. Because the benchmarks are wrong.
Here's the root cause: a genetic perturbation changes ~20-50 genes out of ~20,000. Standard MSE weights every gene equally. So you're scoring models on 19,980 noise genes for every 20 signal genes. The mean predictor wins by capturing systematic shifts shared across all perturbations, ie, not the actual perturbation-specific response. Mode collapse dressed up as performance.
The fix is conceptually simple: weight the error toward differentially expressed genes. We call it WMSE. Under WMSE, the mean baseline collapses to null performance, as it should. Models that were supposedly underperforming turn out to actually learn something real. You just couldn't see it with the wrong ruler.
We also introduce calibrated baselines including a "positive" baseline : predict half a perturbation's cells from the other half. That gives you a realistic ceiling based on the dataset's intrinsic noise floor. Now you know what good actually looks like.
The broader point: measurement is upstream of modeling. Every benchmark paper, every foundation model comparison in perturbation biology, is potentially scoring on the wrong thing. We need to fix the ruler before we argue about the model.
"Needles in the Haystack" — paper: https://t.co/GXpraMFnWV | code: https://t.co/gimh9CYXpO
Congrats to first authors Henry and Gabriel, and co-authors Francis, @BrendanMSwain, @lucascamillomd 🙏
A useful model has a properly defined utility envelope.
The role of benchmarks is to illustrate this envelope.
A good model comes with benchmarks it does not saturate, or does more than 2% improvement on existing ones.
A foundation model is good on existing and future benchmarks
To avoid stating the obvious: the goal of virtual cells isn’t to achieve great perturbation prediction.
Perturbation prediction is a means, not the end.
The real objective is to build a foundation model that powers downstream drug discovery, from target identification and mechanism-of-action inference to toxicity prediction, biomarker discovery, and therapeutic design.
Let’s not optimize for the proxy instead of the mission.
Honestly unsure if Bo Wang has epiphanies once in a while but just falls short of proper introspection to realise he's doing exactly what should not be done.
Or whether he knows very well that he's doing BS and sometimes tries to remind us that he could do something different.
To avoid stating the obvious: the goal of virtual cells isn’t to achieve great perturbation prediction.
Perturbation prediction is a means, not the end.
The real objective is to build a foundation model that powers downstream drug discovery, from target identification and mechanism-of-action inference to toxicity prediction, biomarker discovery, and therapeutic design.
Let’s not optimize for the proxy instead of the mission.
Li-ion batteries costs decrease every time the installed capacity doubled. A recent studies looked at battery costs since 1991 (first Li-ion battery) and found that EVs became the driver for cost declines from 2000 (Toyota Prius). The learning rate since 2000 has been 25%
@michaelzlin That's true. Many of those variations can be captured by patient derived organoids though. So at least we can expect proper data collection and training to reduce the failure rate.
But yeah there's gonna a few straggler due to food, mood, sun exposure, etc for the near future
@michaelzlin You can alleviate part of the problem by screening in a genetically varied group of sample.
But most Bio x AI companies train on data generated in the same three cancer cell lines.
Tahoe100m kind of improved on that but is still limited in scope.
Yes making a drug that binds and inhibits a target is the (relatively) easy part. The hard part is getting one out of the dozens you try that doesn't run into unexpected toxicity or feedback effects that reduce efficacy in the Ph3 patient population, which is more heterogeneous than Ph1/2.
ML bros don't talk about that because there's nothing ML can do about it.
Yes biology is "complex" but as I said that's an euphemism for being utterly unpredictable.
"A cancer diagnosis is a death sentence."
It isn't. Not even close, for most cancers today.
Five-year survival rates by common cancer type:
🟢 Thyroid cancer: 98%
🟢 Breast cancer (localized): 99%
🟢 Prostate cancer (localized): nearly 100%
🟢 Melanoma (localized): 99%
🟡 Colorectal cancer (localized): 91%
🟡 Cervical cancer (localized): 91%
The cancer doesn't change, the stage does. And stage is almost entirely determined by when you find it. Never skip your screening appointments, they have saved countless lives and will continue to do so!
Another proof that we live in a society so advanced we basically make magic real.
Clarke's three laws apply today more than ever https://t.co/UMAPO6WBJo
PSA: You don't specifically need Fable, any model will do.
@PatrickKidger They're making the same mistakes as the first wave of techbio and that many techbio (including isomorphic) still make today: no proper understanding of the data, no biology team.
They have a huge budget to learn make it work though.
And actually it's Tempus they should acquire.
That leaves a lot of wavelength (IR,visible,u) where we don't really know the effect. And we're not even sure that we know all the effects of the ones we have mechanisms for.
It's very likely a bit of UV induced DNA damage is good, keeping your damage repaired mechanisms active.
The issue with the sun is that it produces a whole spectrum of light, and we only get partially know what they do to the body.
UV does DNA damage than can cause cancer and this is bad so wear sunscreen.
Blue light makes vitamin D, but it's not the only beneficial effect.
I’m a dermatologist. I’m supposed to say there is no amount of safe sun exposure.
But I won’t, because that’s a lie.
The attached shows how much sun is safe in different cities at different times of year.
What do I mean by ‘safe’?
I mean this: UV causes DNA damage and skin cancer.
But, shockingly, your body repairs that damage. As long as the damage doesn’t outpace repair and start accumulating it shouldn't increase your risk of skin cancer.
Data just came out that tells us how much UV you can get without damage accumulating.
They took the people most susceptible to DNA damage from UV and exposed them to UV, then did skin biopsies to measure the damage, then more skin biopsies to measure the repair, and repeated it daily for 4 days.
At 1.6 ‘Standard Erythemal Dose’ (SED) there was no accumulation of damage.
So, the attached charts show how much sun it takes to get 1 SED in different cities at different times of the year at different times of day.
And there are extra safety margins built in. It assumes a perfectly clear day with zero air pollution and that the sun is hitting your skin perpendicularly. Unless you’re laying flat, most sun is hitting you at an angle, which isn’t nearly as intense.
But a bigger question you might be asking is ‘Why would a dermatologist be telling you to get sun in the first place?’
Because getting sun reduces your risk of death.
Mostly by reducing your risk of heart attacks and strokes. That is very well proven.
But it’s also very likely that sun exposure reduces your risk of autoimmune disease, dementia, cancer and depression. It’s just not as well proven as the protection against heart attacks and strokes.
And before you reply and say ‘just take vitamin D!’, know that it has been ROBUSTLY proven that vitamin D has little (if any) benefit for preventing any of the above. Vitamin D is mostly useful as a marker of if you’re getting enough sun.
What do I do myself and what do I tell my patients?
Get as much unprotected sun exposure as you can without getting a burn.
That’s my GUESS as to what has the best risk/benefit ratio. Dying of skin cancer is actually really rare, especially when compared to the risk of heart attacks, strokes, autoimmune disease, dementia and other cancers.
But I’ll admit it’s not for sure best to get as much sun as possible, since sun does increase the risk of skin cancer and it might be the case the benefits plateau at a low level.
So, if you’re really worried about skin cancer stick to the charts.
The best science I can find says that amount won’t cause skin cancer.
The takeaway?
Sun is good for you, just don’t get a burn.
@norwyx@PragmaticNurse He's actually fully correct.
Some diseases that appear correlated with vitamin D levels are actually better explained by latitude.
https://t.co/isbdQWXgRz.
The inefficacy of vitamin D supplementation on health is also very well documented.
@kimmonismus You're downloading at least one actor.
Where do you place Gemini Pro ?
I wouldn't downplay GLM5.2 or the Gemma models neither. Well harnessed there are quite powerful.
Big blue is the chronic pain that we can finally address.
Pastels are easy going.
Yellow is crazy but we're making it work.
Dark blue and violet are close friends that are targeted together, and they do weird stuffs with blood.