Mathew Barber

ED HumGen DataSci @EliLillyandCo | Past: @InscriptaInc @RegeneronDNA @TempusLabs @Ancestry DNA sci team | @UChicago @Cambridge_Uni | All opinions are my own

[Julia Bauman @JuliaBauman2]

The 5 biology papers & areas of progress I found most impressive in 2024 (with video explainers for each!):
1/n

[Jonathan Pritchard @jkpritch]

Very excited about this paper. Tweetorial coming soon!

[Theofanis Karaletsos @Tkaraletsos]

Our AI for virtual cells article is now published in @CellCellPress with lots of amazing co-authors who gathered with us at @cziscience and @_bunnech, Aviv, @Prof_Lundberg , @jure , and @StephenQuake

https://t.co/NQqQZrwT89

[Jonathan Marchini @marchini]

Introducing a new method called REMETA from the Regeneron Genetics Center (RGC) @RegeneronDNA for meta-analysis of gene-based tests using summary statistics https://t.co/Bv3pqMB1iy This is great work from Tyler Joseph. Pre-print is here​
https://t.co/CffQXSZPv2

[Veera Rajagopal  @doctorveera]

Analysis of sequencing data of 320k individuals (75k cases and 245k controls) shows heterozygous carriers of cystic fibrosis mutations are protected from inflammatory bowel disease. The protection mechanism could be due to an altered gut mucosal barrier that resists penetration by bacterial or other toxins. Prior animal studies have indeed shown that heterozygous state of cystic fibrosis mutation protects against cholera and typhoid bacterial toxins. It's amazing to see validation using human genetics with IBD as outcome. This was one of the ASHG abstracts this year that stood out for me. Nice to see it in preprint.

Yu et al. medRxiv (from International IBD consortium)
https://t.co/gMBNZhlhYe

[Andrej Karpathy @karpathy]

The (true) story of development and inspiration behind the "attention" operator, the one in "Attention is All you Need" that introduced the Transformer. From personal email correspondence with the author @DBahdanau ~2 years ago, published here and now (with permission) following some fake news about how it was developed that circulated here over the last few days.

Attention is a brilliant (data-dependent) weighted average operation. It is a form of global pooling, a reduction, communication. It is a way to aggregate relevant information from multiple nodes (tokens, image patches, or etc.). It is expressive, powerful, has plenty of parallelism, and is efficiently optimizable. Even the Multilayer Perceptron (MLP) can actually be almost re-written as Attention over data-indepedent weights (1st layer weights are the queries, 2nd layer weights are the values, the keys are just input, and softmax becomes elementwise, deleting the normalization). TLDR Attention is awesome and a *major* unlock in neural network architecture design.

It's always been a little surprising to me that the paper "Attention is All You Need" gets ~100X more err ... attention... than the paper that actually introduced Attention ~3 years earlier, by Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio: "Neural Machine Translation by Jointly Learning to Align and Translate". As the name suggests, the core contribution of the Attention is All You Need paper that introduced the Transformer neural net is deleting everything *except* Attention, and basically just stacking it in a ResNet with MLPs (which can also be seen as ~attention per the above). But I do think the Transformer paper stands on its own because it adds many additional amazing ideas bundled up all together at once - positional encodings, scaled attention, multi-headed attention, the isotropic simple design, etc. And the Transformer has imo stuck around basically in its 2017 form to this day ~7 years later, with relatively few and minor modifications, maybe with the exception better positional encoding schemes (RoPE and friends).

Anyway, pasting the full email below, which also hints at why this operation is called "attention" in the first place - it comes from attending to words of a source sentence while emitting the words of the translation in a sequential manner, and was introduced as a term late in the process by Yoshua Bengio in place of RNNSearch (thank god? :D). It's also interesting that the design was inspired by a human cognitive process/strategy, of attending back and forth over some data sequentially. Lastly the story is quite interesting from the perspective of nature of progress, with similar ideas and formulations "in the air", with a particular mentions to the work of Alex Graves (NMT) and Jason Weston (Memory Networks) around that time.

Thank you for the story @DBahdanau !

[Leland McInnes @leland_mcinnes]

I recently wrote a primer on UMAP for Nature Reviews Primers. If you are looking for an overview of the method, a getting started primer, or best practices it is a good place to start.

https://t.co/O85kc1yZKN

[Stephen Bates @stats_stephen]

Our new textbook Theoretical Foundations of Conformal Prediction is out!

Conformal prediction is a a statistical technique that augments ML systems with uncertainty information for safe deployment. This book lays out the core theory.

https://t.co/ZaDAl1Hnft

[Anastasios Nikolas Angelopoulos @ml_angelopoulos]

🚨 New Textbook on Conformal Prediction 🚨

https://t.co/n2gcyiOM7k

“The goal of this book is to teach the reader about the fundamental technical arguments that arise when researching conformal prediction and related questions in distribution-free inference.

Many of these proof strategies, especially the more recent ones, are scattered among research papers, making it difficult for researchers to understand where to look, which results are important, and how exactly the proofs work.

We hope to bridge this gap by curating what we believe to be some of the most important results in the literature and presenting their proofs in a unified language, with illustrations, and with an eye towards pedagogy.”

We are looking for feedback — and this is only a draft, with Part 4 coming soon! Please reach out!

With Rina Foygel Barber and @stats_stephen

[Patrick Hsu @pdhsu]

Today we report Evo, a generalist foundation model for biology, on the cover of @ScienceMagazine. Trained on billions of DNA nucleotides, Evo was designed to capture two key aspects of biology: the multi-modality of the central dogma and the multi-scale nature of evolution.

[Jack Kosmicki @jakphd]

Maybe everyone already knew about this, but the @AllofUsResearch results browser is up so you can look up your favorite genetic associations (I assume this is the browser @konradjk presented at #ASHG24 that is now ready?).

https://t.co/DqlLpxVg5W

[Adam Auton @adamauton]

Yesterday was a hard day at 23andMe, and we said goodbye to a number of tremendously talented colleagues. If people have job openings in the genetics space that they'd like me to share with the impacted folks, please do post here.

[Jack Kosmicki @jakphd]

"I spent too much time on QC to not have a slide about it" 😂☺️ #ASHG24

[Mathew Barber @mathartic]

We have an exciting new opportunity to join the data science team at Inscripta!
We are working on many exciting projects using deep learning with high-content imaging and gene editing to help shape the future of synthetic biology. Pls RT
https://t.co/JQBlExcNBb

[Gael Varoquaux 🦋 @GaelVaroquaux]

Data preparation!
It's crucial for machine learning, and we all hate it.

Tools and techniques to reduce this burden?
A quick summary of 10 years of R&D on this, from cheap tricks to LLMs and graph neural networks

1/9

[Adaptyv Bio @adaptyvbio]

After testing in our lab, we validated that 5 of those proteins actually show binding affinity to EGFR. That's 5 completely novel proteins that don't exist in nature or scientific literature and have passed the first validation to potentially become new cancer drugs.

To put this in context, just a couple of years ago, designing and validating a protein binder for a human receptor was reserved for a few pharma companies with lots of experts and specialized facilities.

Today, we show that individuals from around the world were able to contribute meaningful designs within just a one-month timeframe.

Let’s take a closer look at these proteins and the designers behind them:

[Vince Buffalo @vsbuffalo]

I'm looking to change my career path and am quite excited about exploring jobs in biotechs in the Bay Area (or remote). I'm very keen to start a career that offers stability so my partner and I can finally live in the same state. Does anyone know of organizations that are hiring?

[Institute for Protein Design @UWproteindesign]

Nobel Laureate David Baker on the phone with fellow awardees Demis Hassabis and John Jumper. The trio discussed how their teams have inspired one another and what the future may hold.

[Institute for Protein Design @UWproteindesign]

The first episode of @BakerLabPodcast is live. Meet Mohamad Abedi, who went from being a Palestinian refugee to a postdoctoral researcher. His journey proves that science can be a creative and transformative process.
Apple: https://t.co/kBk9SW7kDE
Spotify: https://t.co/WiozZ2GYR2

[Lior Pachter @lpachter]

So this plagiarism thing has happened to our lab.. again. This time it's plagiarism of our poseidon syringe pump paper @booeshaghi et al., 2019 in @SciReports: https://t.co/JLWrOK2Lr4
Text has been plagiarized, as well as figures copied directly here: https://t.co/qzAdvj9H44 1/🧵