Yes! We looked at a few. No RMSD per se, since there's no experimental structure for these designs to measure against, but we did break confidence down to the site level. In some cases the binding-site-specific metrics did outperform the global ones, pocket-minus-protein pLDDT hit 0.77 AUC in one panel against 0.53 for pTM, but it fell to 0.49 in another. So they didn't generalize.
Put another way: they might tell you whether a site is completely malformed, but they don't seem to recover specifics on function or selectivity
After 6 years in stealth, Aether's first paper is out.
We designed and measured hundreds of proteins. The ones that leading models were most confident in were least likely to work: 84% didn't bind.
A confident predicted shape is not evidence of function. https://t.co/fJIFn5uqAv
@OenoneZer0@curiouswavefn Agreed on communication! IMO I thought this was well summarized after some work with our agents, so always open to better communication techniques but whether it was written by AI or not I care less about :)
Yes, many of these studies are susceptible to assay bias and I suspect a ton of results are contaminated by them. The only way through is better assays, secondary validation, and large scale production of functional proteins to confirm that they work as intended in the context they’re designed for!
@AjasjaLjubetic Couldn’t agree more! Students should know that proteins are made of ribbons and arrows, otherwise how will they know what proteins truly are and how they function?
@GregPreibisch@SylvainGariel ^^^ exactly, we thought this was a good test bed for exploring this distinction, we've got some cool work underway using our models to predict ensemble states of complex partially or completely unstructured proteins as well!
@SylvainGariel yeah absolutely, we're planning on running open benchmarks over time on a wide range of functional assays we've been running, it'll be interesting to see how the structural models evolve over time but I suspect they'll struggle across the board
Anthropic lists Thursday. It bought a biotech and runs a wet lab.
We published a result today: the protein designs the structure models were most confident in were the ones that didn’t work. 84% of them didn’t bind anything.
Structure confidence does not predict function. A model trained on measured function does.
I’m on @MTSLive later today to get into it.
After 6 years in stealth, Aether's first paper is out.
We designed and measured hundreds of proteins. The ones that leading models were most confident in were least likely to work: 84% didn't bind.
A confident predicted shape is not evidence of function. https://t.co/fJIFn5uqAv
I do not at foreclose the possibility of various sorts of political shenanigans even after Vucic’s concession but this extraordinary result in Serbia is what happens when people show up: in the streets, at the ballot box. That’s the lesson. If you care about democracy, show up.
This is a very poor one line summary of the article by @DaphneKoller IMO.
The point is not just data volume. The right kind of data scaled in the right axis with the right experimental designs for well defined tasks + using the models to discover biology (not eval maxxing)
Yes! These people have made building in San Francisco so restrictive that OVER HALF of the homes there now are illegal to build today
For SF to get in compliance, 130,860 homes would have to be destroyed
This self-inflicted housing shortage is the city's biggest problem—by far