Alan Baade

CS PhD Student at Stanford

[Alan Baade @BaadeAlan]

1/ The tokenizer is a new scaling axis!

We introduce an approach for diffusable representation learning that's principled, scalable, and works when DINO/LeJEPA collapse.

We get joint tokenizer+diffusion scaling laws and it's compute-optimal vs raw diffusion

w/ @drfeifei @GordonWetzstein @jcjohnss @ericryanchan @vincesitzmann @KyleSargentAI

Thread, Paper, and Blogpost below

[Alan Baade @BaadeAlan]

@yubel426 @drfeifei It's like using warmup-stable-decay. It's convenient, especially for scaling laws, to at any time be able to decay and get an evaluable model. And removing a training stage is convenient. There is no fundamental reason to not train separately.

[Alan Baade @BaadeAlan]

1/ The tokenizer is a new scaling axis!

We introduce an approach for diffusable representation learning that's principled, scalable, and works when DINO/LeJEPA collapse.

We get joint tokenizer+diffusion scaling laws and it's compute-optimal vs raw diffusion

w/ @drfeifei @GordonWetzstein @jcjohnss @ericryanchan @vincesitzmann @KyleSargentAI

Thread, Paper, and Blogpost below

[Sander Dieleman @sedielem]

Another new scaling axis? How many are we at now?😵‍💫

In all seriousness, this is a very cool piece of work, and as far as I can tell, a completely novel way to create representations whose inner products capture mutual information. Check out the blog post: https://t.co/V9WkPjkFVq

[Shivam Duggal @ShivamDuggal4]

Really insightful work — congratulations Alan @BaadeAlan and team!

Exciting to see more exploration in this direction.
We've also been curious about related questions (studying different scaling laws) through our work ––

ALIT (looped tokenizer), KARL (single-pass adaptive tokenizer), and UNITE (generative tokenization).

Lots to learn from this work! 🙌

[Alan Baade @BaadeAlan]

Yeah I totally get this! And I very much dislike the fact that "tokenizer" sounds like it should be discrete

Here's the issue I try to highlight
I want a term for the space that a model generates in, regardless of whether it's continuous or discrete.

Crucially, it needs to be "modelable" (or "diffusable" if that's the downstream architecture). The outputs of regular encoders (e.g. MAE) don't necessarily satisfy this. Although they are useful for later layers, this doesn't mean they are easy to generate in (i.e. express uncertainty in during prediction).

An adversarial case here could be that the output of my encoder is an encrypted function of the data. A decoder could learn to decrypt this and use that for e.g. really good MAE prediction. And the encrypted data could even be extremely useful (e.g. a CoT reasoning trace to solve an IMO problem). But modeling this space is horrible.

Another way to put this is that I use "tokenizer" when I really mean that we have (effectively) a learned loss function + information order during generation.

But I think the real takeaway here is that, yes, I'm sorry for only knowing zoomer ML.

[Alan Baade @BaadeAlan]

@sedielem Thank you so much!
I've been trying to address the semantic gap issue in generation that you've been highlighting for a while. I think this paper is a strong attack on the problem, and hopefully we'll get a complete solution sometime soon!

[Alan Baade @BaadeAlan]

@RyanJ_KOL @drfeifei We do both:
Tokenizer Encoder FLOPs Only on the X axis, Fixed Diffusion Budget
Tokenizer Encoder + Diffusion FLOPs on the X axis, Varying Diffusion Budget

[Alan Baade @BaadeAlan]

@TimDarcet @giffmana @vincesitzmann I have no allegiance to the term "tokenizer"
But I can't use just "encoder" in the paper
We both a tokenizer encoder and a MAE encoder
"Encoder" just can't disambiguate enough

[Alan Baade @BaadeAlan]

@giffmana @vincesitzmann @TimDarcet And I'm guessing you'll probably hate this, but I'd also say "tokenizer encoder" for the CNN input as well. I agree the downstream architecture shouldn't matter here.

[Alan Baade @BaadeAlan]

@giffmana @vincesitzmann @TimDarcet No, "encoder" is too overloaded of a term
Encoder/Decoder Transformers and the Encoder/Decoder in MAE/MARs are different. They don't need to have modelable or predictable outputs. So to specify the space the model generates in we use "tokenizer encoder."

[Alan Baade @BaadeAlan]

I agree it's a little annoying that tokenization has become the standard catchall term for latent generation spaces, both continuous and discrete. That's why I use the term "diffusable representation learning" on the second line.

But tokenization is certainly a key use of representation learning, and people have been using current-paradigm representation learning for speech tokenizers since 2021!

[Alan Baade @BaadeAlan]

@giffmana @vincesitzmann @TimDarcet Yes, in the paper we explicitly disambiguate that it's the tokenizer encoder

I don't think of the tokenizer decoder as it's own stage. I only see 1) tokenizer encoding and 2) generative modeling.

The decoder is just the subset of the generative model that samples the pixels.

[Eric Chan @ericryanchan]

Excited to share @BaadeAlan's latest work: a new approach to representation learning designed specifically for generative modeling!

Semantic representations like DINO have become remarkably effective tokenizers for image and video generation. But DINO was never designed for this task. Its success in generative modeling is almost accidental. What if we learned representations specifically to make generation easier?

[Alan Baade @BaadeAlan]

@AlexHe00880585 Thanks so much @AlexHe00880585 !

[Alan Baade @BaadeAlan]

@eps_greedy tysm!

[Alan Baade @BaadeAlan]

@astridwilde Yes this approach is incomplete for sure -- representation learning isn't solved yet!
But I think the real critical thing is to focus on being principled and scalable, not satisfaction. Satisfaction can only come after we have a strong foundation for what we're actually doing.

[Alan Baade @BaadeAlan]

@1rreverant @drfeifei It's an great work by @ShivamDuggal4
And also related to DeTok by @JiaweiYang118
I think that backprop through added noise is the main other attack for scalable tokenization, however backpropagating through an information bottleneck (e.g. noise) hasn't yet been shown to scale.

[Alan Baade @BaadeAlan]

@aliteracy Thanks so much! Also tysm for the goated 10/10 social media edits.

[Alan Baade @BaadeAlan]

I think representation learning is pre-paradigmatic. Hopefully, this is a real step forward.

Arxiv: https://t.co/TOLufworeA

Blog: https://t.co/pPkR3wPzj4

Code: https://t.co/mZUNhgLgBi

[Alan Baade @BaadeAlan]

TL;DR: We have

* Same theory as contrastive learning without negatives, batch statistics, or non-masking augmentations
* Tokenizer encoder scaling laws
* Compute optimal performance versus raw diffusion