Dylan Paiton

web3 data pipelining & AI

Former: head of research @delv_tech; dev @weights_biases; phd neuroscience uc berkeley @redwood_neuro

[David Klindt @klindt_david]

🧵 New paper! We explore sparse coding, superposition, and the Linear Representation Hypothesis (LRH) through identifiability theory, compressed sensing, and interpretability research. If neural representations intrigue you, read on! 🤓
https://t.co/3bkO9doSNi

[ToposBio @ToposBio]

Announcing Topos-1, an all-atom generative model that sets a new benchmark for predicting structural ensembles of intrinsically disordered proteins. Topos-1 outperforms existing models by a large margin, with implications for designing drugs for challenging targets ranging from cancer to neurodegenerative disease.

Read the full technical report on our website: https://t.co/3UtTSdmddf

[Delip Rao e/σ @deliprao]

Adversarial attacks on vision language action models. https://t.co/auWICLcpZ3

[Dylan Paiton @DylanPaiton]

@karpathy you have no idea how validated i feel reading this

[Andrej Karpathy @karpathy]

This is interesting as a first large diffusion-based LLM.

Most of the LLMs you've been seeing are ~clones as far as the core modeling approach goes. They're all trained "autoregressively", i.e. predicting tokens from left to right. Diffusion is different - it doesn't go left to right, but all at once. You start with noise and gradually denoise into a token stream.

Most of the image / video generation AI tools actually work this way and use Diffusion, not Autoregression. It's only text (and sometimes audio!) that have resisted. So it's been a bit of a mystery to me and many others why, for some reason, text prefers Autoregression, but images/videos prefer Diffusion. This turns out to be a fairly deep rabbit hole that has to do with the distribution of information and noise and our own perception of them, in these domains. If you look close enough, a lot of interesting connections emerge between the two as well.

All that to say that this model has the potential to be different, and possibly showcase new, unique psychology, or new strengths and weaknesses. I encourage people to try it out!

[Dylan Paiton @DylanPaiton]

@joce_breton @smithcollege Yay that's awesome! Congrats!!

[Hynek Kydlíček @HKydlicek]

🚀 We've boosted MATH benchmark scores for popular models by 65% —no training or model changes needed! The secret?

Math-Verify, our new math evaluator.

Turns out current math benchmark evaluations are broken. Here's what's wrong and how we fixed it:

[Mark Riedl @mark_riedl]

Open Source Initiative (OSI) says AI models aren’t “open source” unless data, weights, hyperparameters, and executable code to build and run the model are released

[Ben Recht @beenwrekt]

The dirty secret of machine learning is benchmarks are all we have.

[Alex Naka @gottapatchemall]

1/ I did my own little hackathon last weekend designing EGFR binders for @adaptyvbio's protein design competition. I was really excited to see that my submissions took the top 10 spots in the virtual scoring phase! I got some DMs asking about my process so here's a thread:

[Charles 🎉 Frye @ ICML @charles_irl]

since folks care about sparse autoencoders and friends now thanks to Golden Gate Claude, TBT to a poster from a few years back comparing SAEs to neurally-inspired recurrent autoencoders

[Dylan Paiton @DylanPaiton]

This is awesome -- checking ML training validity in GH actions!

[Charles 🎉 Frye @ ICML @charles_irl]

Added a fun new feature to the @modal_labs LLM fine-tuning repo's CI yesterday, bringing together my @wandb logging days and ideas from @full_stack_dl 2022 with serverless infrastructure:

👩‍🏫 memorization testing 👩‍🏫

[Databricks @databricks]

#DBRX sets a new standard for efficient open source LLMs. While it has 132B total parameters, with its fine-grained MoE architecture, DBRX only uses 36B at any given time.

Learn more about how we built #DBRX & benchmarked its performance. https://t.co/oNiffY3bqF

[Cecilia Ziniti @CeciliaZin]

🧵 The historic NYT v. @OpenAI lawsuit filed this morning, as broken down by me, an IP and AI lawyer, general counsel, and longtime tech person and enthusiast.

Tl;dr - It's the best case yet alleging that generative AI is copyright infringement. Thread. 👇

[@emilymbender.bsky.social @emilymbender]

You know what's most striking about this graphic? It's not that mentions of people/cities/etc from different continents cluster together in terms of word co-occurrences. It's just how sparse the data from the Global South are.

[Dylan Paiton @DylanPaiton]

@jesseengel I want an easy way to generate dungeon / tavern / etc music for my DnD games :)

[Charles 🎉 Frye @ ICML @charles_irl]

Over the past month, I've been working to grok RWKV, one of the most successful challengers to Transformers for language modeling.

I untangled numerical tricks from load-bearing math, assigned semantic names to one-letter variables, and debugged weird NaNs so you don't have to!

[Grant Sanderson @3blue1brown]

Here's a short dynamics-based explanation of the equation e^(πi)=-1

[Charles 🎉 Frye @ ICML @charles_irl]

I keep coming back to the "Python is Two Languages Now" article by @tintvrtkovic from February. It has durably changed my thoughts about types and about Python, two things I think about a lot!

https://t.co/YQVOJMIOQ4

[Dylan Paiton @DylanPaiton]

These lectures are a super valuable resource. Great job FSDL team, and thanks for releasing them!

[The Full Stack @full_stack_dl]

🥞🦜 Full Stack LLM Bootcamp 🦜🥞

tl;dr We're releasing our lectures on building LLM-powered apps, for FREE.

🚀 Launch an LLM App in One Hour
✨ Prompt Engineering
🗿 LLM Foundations
🔨 Augmented LLMs
🤷 UX for LUIs
🏎️ LLMOps
🔮 What's Next?
👷 Project Walkthrough

Learn more: