Nicholas Sofroniew

math/neuroscience - AI

[Nicholas Sofroniew @sofroniewn]

Incredibly excited to share what I’ve been researching since joining @AnthropicAI

We found emotion concepts in Claude and studied their function!

[Anthropic @AnthropicAI]

New Anthropic research: Emotion concepts and their function in a large language model.

All LLMs sometimes act like they have emotions. But why? We found internal representations of emotion concepts that can drive Claude’s behavior, sometimes in surprising ways.

[Zeming Lin @ebetica]

I'm so excited to show the world what we've been working on the for the past months!! I'm going to highlight some of the fun results from this paper that I find particularly exciting.

[Nicholas Sofroniew @sofroniewn]

@alexrives Congrats @alexrives @salcandido and the whole team! What an incredible effort and paper!!!

[Nicholas Sofroniew @sofroniewn]

Very excited to have played a part in this while I still at ES

I recommend reading the paper, there is some really cool stuff in it including interpretability on ESMC

I think the next wave of biological discovery will come through understanding the internals of language models!

[Alex Rives @alexrives]

Today we're announcing ESMFold2, an open scientific engine to power prediction, design, and discovery across protein biology.

The new model delivers state of the art performance on protein interactions, especially antibodies, a critical modality for therapeutics.

We have designed and validated miniprotein binders and single chain antibodies across five therapeutic targets that are important in cancer and immunology. We are seeing very high success rates, and affinities at levels consistent with therapeutic activity.

We’re also releasing an atlas of 6.8 billion proteins, and 1.1 billion predicted structures.

ESMFold2 is built on a state of the art language model that has been trained on billions of protein sequences.

A world model of protein biology emerges through language modeling.

We’ve used the techniques of mechanistic interpretability developed to understand large language models to understand the concepts ESM uses to represent proteins.

The model’s representation space has a compositional organization of features across scales, levels of complexity, and abstraction, that reflects and mirrors the understanding of protein biology developed through a century of empirical science.

This understanding emerges without prior knowledge, just from language modeling of protein sequences.

Language models are becoming a powerful substrate to understand and program biology.

The design of protein interactions is one of the most fundamental problems in biophysics, and has critical implications for the discovery of new medicines. A simple gradient based search with the model was able to discover high-affinity protein binders.

I'm excited by the potential this has to accelerate basic science and the understanding of proteins. And especially for the new avenues it opens up for therapeutic design and medicine.

[Nicholas Sofroniew @sofroniewn]

Two things, first an update that I've been working at Anthropic on Interpretability. For me it's a wonderful combination of maths, neuroscience, and AI, and I love it

Second I want to express my support for the company and its leadership for acting with integrity and principles

[Anthropic @AnthropicAI]

A statement on the comments from Secretary of War Pete Hegseth.

https://t.co/Gg7Zb09IMR

[Science Magazine @ScienceMagazine]

Researchers have developed a deep learning protein language model, ESM3, that enables programmable protein design.

Learn more in this week's issue of Science: https://t.co/PndjVQWjT5

[Nicholas Sofroniew @sofroniewn]

@pdhsu @arcinstitute @nvidia Congrats @pdhsu super exciting!

[Nicholas Sofroniew @sofroniewn]

@MillionInt and best way to do that is to make them digital!

[Nicholas Sofroniew @sofroniewn]

So excited to be part of this work and programming biology! 🤖🧬🧫

[Alex Rives @alexrives]

We're thrilled to present ESM3 in @ScienceMagazine. ESM3 is a generative language model that reasons over the three fundamental properties of proteins: sequence, structure, and function. Today we're making ESM3 available free to researchers worldwide via the public beta of an API for biological intelligence.

Trained with over a trillion teraflops of compute, this is the first time a model of this scale has been trained for biology, pushing the frontier of AI for biological discovery and engineering.

ESM3 learns to represent the immense complexity of protein biology, learning from billions of natural proteins. From this training it developed the capability to design proteins, responding to complex prompts combining atomic level details and high level instructions to generate new proteins.

ESM3 can explore protein space far beyond natural evolution. We prompted ESM3 to generate a fluorescent protein at a far distance from any known fluorescent proteins, searching an unknown region of protein space, to discover a new fluorescent protein.

We estimate this is equivalent to simulating five hundred million years of evolution.

[Alex Rives @alexrives]

We're thrilled to present ESM3 in @ScienceMagazine. ESM3 is a generative language model that reasons over the three fundamental properties of proteins: sequence, structure, and function. Today we're making ESM3 available free to researchers worldwide via the public beta of an API for biological intelligence.

Trained with over a trillion teraflops of compute, this is the first time a model of this scale has been trained for biology, pushing the frontier of AI for biological discovery and engineering.

ESM3 learns to represent the immense complexity of protein biology, learning from billions of natural proteins. From this training it developed the capability to design proteins, responding to complex prompts combining atomic level details and high level instructions to generate new proteins.

ESM3 can explore protein space far beyond natural evolution. We prompted ESM3 to generate a fluorescent protein at a far distance from any known fluorescent proteins, searching an unknown region of protein space, to discover a new fluorescent protein.

We estimate this is equivalent to simulating five hundred million years of evolution.

[Roshan Rao @proteinrosh]

i love this plot

[Nicholas Sofroniew @sofroniewn]

Super excited about ESMC and the quality of the protein representations! Can't wait to see what people build on top of it

[Alex Rives @alexrives]

Introducing ESM Cambrian.

Unsupervised learning can invert biology at scale to reveal the hidden structure of the natural world.

We’ve scaled up compute and data to train a new generation of protein language models. ESM C defines a new state of the art for protein representation learning.

[Hannes Stark @HannesStaerk]

New paper with Bowen :)
"Generative Modeling of Molecular Dynamics Trajectories" https://t.co/utCWRYBMtV

A "video diffusion" model but for MD trajectories.
Different conditioning solves different tasks.

E.g. condition on first and last frame => transition path sampling
1/4

[Yunha Hwang @Micro_Yunha]

What does gLM2 learn in non-protein-coding sequences?🧬 Using the Categorical Jacobian and the latest gLM2, we detect incredible regulatory signals in the non-protein coding regions -- all without any supervision!🪄 a quick 🧵

[Sergey Ovchinnikov @sokrypton]

Exciting new work from Qian Cong's group on predicting human protein interactome. Leveraging new eukaryotic genomes, new RoseTTAFold2 trained on +/- pairs of PPI and large distilled dataset of domain-domain interactions! 🤩

https://t.co/vsbJWdNze6

[Martin Pacesa @MartinPacesa]

Have you ever wanted to design protein binders with ease? Today we present 𝑩𝒊𝒏𝒅𝑪𝒓𝒂𝒇𝒕, a user-friendly and open-source pipeline that allows to anyone to create protein binders de novo with high experimental success rates. @befcorreia @sokrypton

https://t.co/IPhMFpRgHh

[Nicholas Sofroniew @sofroniewn]

@TrackingActions It took ~3 hours, and was the only time in my life where I really felt I taught a mouse anything and saw a mouse think. There was a clear transition of < 1 minute where it went from not getting it to getting it which was pretty magical

[Nicholas Sofroniew @sofroniewn]

@TrackingActions Not sure if I ever told you about this one, but it's unpublished (>10 years old now!), but I taught a mouse to do a navigate my winding corridor task with full "-1" gain inversion - e.g. think wearing prism glasses, but with the whisker system

[John Cumbers @johncumbers]

Great AI meets Bio event in Boston with @mkoeris @geochurch and @alexrives 🧬🧬🧬🔥🔥🔥

[OpenAI @OpenAI]

OpenAI o1 solves a complex logic puzzle.