Jeffrey Chan

ai4science @lilasciences | previously @dyno_tx @googleai @apple @uberailabs | phd @berkeley_eecs + bs @mit. views are my own.

[Jeffrey Chan @jeffrey_d_chan]

Always been bullish on the relative value of retrieval in biology. You get to cheaply bring all the latest most useful sequences "in-context" rather than scaling the memorization in the weights.

[Sergey Ovchinnikov @sokrypton]

Upper-left - academia: still struggling to raise $100K to support the project (grants, fellowships rejected). 😅

Upper-right - industry: raised $142 million in seed funding (company sold for much much more). 🙃

[Jeffrey Chan @jeffrey_d_chan]

@pastramimachine Imo hard to find the right intersection of useful application and clear value in dedicating FLOPs to inferring phylogenies over the more implicit approaches

[Jeffrey Chan @jeffrey_d_chan]

@samsinai Still waiting for my other 999999 to develop

[Jeffrey Chan @jeffrey_d_chan]

@nlarusstone The closest thing to a scaling law ablation is was by Align for pLMs showing unsupervised loss which is well correlated with contact prediction was reasonably stable but growing from 2013 onwards. https://t.co/rhDIGoZzv9

[Jeffrey Chan @jeffrey_d_chan]

The historic order of events was roughly: first you had enough structures, then you had enough sequences/MSAs, then enough compute, then right arch. The last 3 probably all happened mid to late 2010s while you probably had enough structures way before.

[Nicholas Larus-Stone @nlarusstone]

I see this claim a lot but the really interesting question here is what is the smallest version of the PDB that would have allowed us to get alphafold2 level performance?

[Jeffrey Chan @jeffrey_d_chan]

@nlarusstone My back-of-the-envelope guess would be < 40M sequences. All of the coevolution and Potts model stuff was starting to work around then for contact prediction around 2013.

[Jeffrey Chan @jeffrey_d_chan]

Both of these seem correct to me. Humans don't operate at the raw seq data layer for most tasks so most of their reasoning occurs at the analysis/knowledge level of abstraction so LLMs shouldnt be any different. Also the expt data you can scale are usually not the ones you want

[Chris Hayduk @ChrisHayduk]

I'm pretty amenable to this position as well.

If a text model is able to understand the current state of bio knowledge, formulate novel hypotheses, suggest experiments and associated protocols to assess these hypotheses, and then analyze the produced data and make inferences based on the analysis, then it should be able to produce new biological knowledge in the same way humans can.

I think harnesses like Codex are already getting us part of the way here by exposing plugins that text models can use as repeatable tools to understand/interpret biological data.

It's possible that just getting the models much better at these things is enough to get us accelerated bio progress.

[Jeffrey Chan @jeffrey_d_chan]

@anshulkundaje So from that perspective the bitter lesson still applies. Its just that it doesn't make economic sense to pursue. Spending say $100M in compute to get a comparable model without MSA, tri attention, complex losses likely isn't worth it.

[Jeffrey Chan @jeffrey_d_chan]

I somewhat disagree while it doesn’t map cleanly AF has a lot of bitter lesson elements. Inference time scaling for Abs in AF has been demonstrated in the AF3 paper. Synth data can reduce reliance on MSAs and with enough data and FLOPs eliminate it

[Anshul Kundaje @anshulkundaje]

Firstly, AlphaFold is not a great example of the "Bitter lesson" IMO. It is trained on a relatively small labeled dataset & primarily transforms MSAs that encode structural constraints into structure with carefully crafted architectures with many inductive biases. 1/

[Jeffrey Chan @jeffrey_d_chan]

@anshulkundaje Youre right in that no model exists that fits your specs! My main point is papers have shown that you can hillclimb effectively with more FLOPs, synthetic data, and simpler architecture (eg no tri attention) but the $$ to match AF3 would be so costly its not worth it economically

[Jeffrey Chan @jeffrey_d_chan]

My main point is I think the technical principles behind the bitter lesson still applies to bio but the economic motivation (or scientific reason in the case of DNALMs) to use that path as the way to better models is more unclear.

[Jeffrey Chan @jeffrey_d_chan]

Perhaps the right analogy with the bitter lesson world is what if instead of needing to scale up params/compute/data to train Opus then distill down to a one generation behind Sonnet, you could instead just use retrieval(MSAs) to distill from for much cheaper.

[Markus Covert Lab @MarkusCovertLab]

How would we go about spearheading a Human Cell Project? By request I’ll post some thoughts on this topic over the next few weeks.

First, we need a clear idea of what the goals are. For the sake of discussion, let me suggest two: (1) we want at least one *strong* annotation for 95% of the known functional (coding and non-coding) regions of the genome, and (2) the totality of this information needs to be made openly available, machine readable and encoded in a computational model that can make actionable predictions about cell behavior, including in response to different environmental or genomic conditions. As in the Human Genome Project, it makes sense to start with simpler (microbial) cells to develop the tech and intuition, and keep moving forward in complexity.

Today I’ll offer some thoughts on annotation and the data challenge, which is a huge effort that I think is sometimes misdirected. Specifically, a unified experimental effort would need to be more specifically directed to what is unknown – that’s where the uncharacterized functional regions lie.

For example, we and others have shown that a large fraction of the unknown genes in bacterial cells are related to interactions with other biological entities. Viruses, other microbes and mammalian cells or factors – adding any of these to the culture media will cause a number of uncharacterized genes to be upregulated and expressed. In order to characterize and annotate this large set of genes, we need new, more complex means of recreating the full spectrum of what cells experience in the wild!

This will help us to annotate, even across species -- for example, a few years ago we reported a system in which we prompted mammalian macrophages (part of the immune system) to swallow bacteria which were themselves harboring a virus called lambda phage. Around a quarter of the bacteria in your gut right now contain viruses like this.

The mammalian cells didn’t kill the bacteria directly. Instead, they released a peptide that activated the lambda phage, which then went on a rampage and killed all of the engulfed bacteria (as they say, "the enemy of my enemy is my friend"). This unexpected finding led to a number of novel annotations that you would never find in experimental data derived using simple media conditions, or interpolate from a model generated from such data.

So one thing we need to think about is how to systematically capture these interactions, ideally at scale but definitely quantitatively and over time. Innovators wanted, we need ideas here!

Maybe in the future I can talk more about dynamics, quantification, causality, curation and a detailed strategy for annotating those 95% of genes in future posts if people are interested.

https://t.co/S897V06Aus

[Jeffrey Chan @jeffrey_d_chan]

My bet is that scientific reasoning won’t be achieved purely via the o1 era RLVR paradigm but will still have some RL characteristics.

[Amy Lu @amyxlu]

@LiamFedus If the key is that verifiable rewards for scientific reasoning will be too sparse / too unreliable for the foreseeable future, curious if you think RL can still be the right tool, or should we look elsewhere to cope with the uncertainty you mention?

[Jeffrey Chan @jeffrey_d_chan]

Congrats to the OF3 team! Open-source models on top of preexisting open-weight models will be a step change for the broader AI bio community

[Mohammed AlQuraishi @MoAlQuraishi]

New OpenFold3 preview out! (OF3p2)

It closes the gap to AlphaFold3 for most modalities.

Most critically, we're releasing everything, including training sets & configs, making OF3p2 the only current AF3-based model that is functionally trainable & reproducible from scratch🧵1/9

[Jeffrey Chan @jeffrey_d_chan]

@rishabh16_ AI4Bio models still haven’t found sufficient financial upside that makes it obvious everyone should put in the $$ to make their own. Still remains just a handful of groups invested in owning the stack but maybe that changes as Chinese big tech dips into making OS protein models

[Jeffrey Chan @jeffrey_d_chan]

https://t.co/bYaFsSvSxH

[Jeffrey Chan @jeffrey_d_chan]

If you wanna hang out with me and other researchers at Lila, come help us celebrate and settle into our new SF office.

Register below to come by our SF Open House on the evening of March 18th.

[Jeffrey Chan @jeffrey_d_chan]

@anshulkundaje @SashaGusevPosts You often see this effect in the protein world as well. Alignment between evolution/self supervision objective and fitness matter a lot. Self supervision helps with many structure tasks but small supervised models match finetuned pretrained models on many functional tasks