ninad

debugging intelligence / masters @uofillinois

[ninad @ninaddaithankar]

Can a vision model learn to see with no augmentations, no masking, no cropping, no reconstruction?

🎬 It can! Introducing Temporal Difference in Vision (TDV), a new visual representation learning paradigm built on a single assumption: the past causes the future.

TL;DR :
- We introduce TDV, the first approach to learn useful representations without any augmentations, masking, cropping or pixel based reconstruction.
- TDV matches SOTA recipes like DINO and iBOT on dense spatial tasks
- We also show that as data scales up, weaker assumptions work better.

🧵Thread:

[Lukas Kuhn @lukaskuhn77]

🔥 We introduce LeVLJEPA: the first fully non-contrastive end-to-end vision-language pretraining method competitive with CLIP & SigLIP 💪🏼

👀 No negatives. No temperature. No momentum encoder. No teacher-student.

TL;DR: LeVLJEPA learns image to text structure by prediction: each modality predicts the other's embedding, while SIGReg keeps each embedding isotropic Gaussian. 🧵

📄 https://t.co/1qBXor8qTf

[ninad @ninaddaithankar]

@juliarturc visualisations!

[ninad @ninaddaithankar]

@KyeGomezB a model that takes in the current state of the environment + an action, and gives you the next state of the environment

[artem kirsanov @ArtemKRSV]

Anthropic is missing an opportunity to release an image generation model and name it Claude Monet

[Unconventional AI @unconvAI]

Introducing our first model, Un-0!

We trained an image generator powered by a backbone of coupled oscillators in place of a more traditional conventional neural network.

[Hisku @ZikuD_s]

Can you read a model's belief state?
Two transformers navigate a grid blindfolded. A linear probe tries to read each one's mind.
One is all over the board and backs the wrong cell. The other is correct ~100% of the time.
The only difference is how they were trained. 🧵

[ninad @ninaddaithankar]

nice work @affon44 & @aditya_potnis! 🎉

[Francisco Affonso @faffonso2]

How do you get perceptive locomotion over rough terrain without brittle terrain classifiers?

Excited to share CTS-MoE, a framework for implicit terrain adaptation via Mixture-of-Experts for perceptive locomotion. No selectors or per-task policies; the policy adapts end-to-end straight from vision.

TL;DR:
→ Perception-driven routing handles diverse, discontinuous terrain implicitly; no high-level task selector or per-task policy distillation.
→ Big gains on hard tasks (climbing, gaps) under MTRL, with smooth transitions on both seen and unseen terrain.

🧵Thread:

[ninad @ninaddaithankar]

@ID_AA_Carmack @AlexiGlad checkout our thread on TDV for more info!

[ninad @ninaddaithankar]

Can a vision model learn to see with no augmentations, no masking, no cropping, no reconstruction?

🎬 It can! Introducing Temporal Difference in Vision (TDV), a new visual representation learning paradigm built on a single assumption: the past causes the future.

TL;DR :
- We introduce TDV, the first approach to learn useful representations without any augmentations, masking, cropping or pixel based reconstruction.
- TDV matches SOTA recipes like DINO and iBOT on dense spatial tasks
- We also show that as data scales up, weaker assumptions work better.

🧵Thread:

[John Carmack @ID_AA_Carmack]

Paper review:
You Don’t Need Strong Assumptions: Visual Representation Learning via Temporal Differences
https://t.co/w4zUfV0ZkX
https://t.co/ji2imtMQNv
@AlexiGlad @ninaddaithankar

The premise is that the more data you can use, the fewer inductive biases you should have. Starting with strong priors is helpful with limited data, but eventually, architectural priors will hinder learning true knowledge buried in sufficiently large datasets. That sounds correct.

Concretely, the ad hoc image cropping / masking / augmentations used in self supervised representation learning all make assumptions about what is important in the images, and appendix A gives examples where they can be harmful.

Figure 3 looks very compelling for this argument, but if you notice that the X axis is log scale, it is sketchier; the anchoring values on the left are relative values from experiments on 0.1% of Imagenet, which I would expect to be quite high variance.

Instead of making multiple augmentations of an image for self supervision, this work uses sequential video frames as related image pairs for representation learning. They train two separate models: a frame encoder, and a “motion encoder” that takes the RGB subtraction between the sequential video frames to produce a delta vector. The models are jointly trained so that the first frame’s representation vector, added to the delta vector, will equal the second frame’s representation vector.

They use a fairly substantial 0.25 second stride between images in the pair, noting that too small of a stride results in near-zero differences in slow moving scenes, while too large of a stride gives incoherent pixel jumps.

It looks to me like the model should be stride independent, and they could simultaneously train on many different strides, increasing the dataset diversity.

I don’t like the DINO EMA teacher approach for avoiding collapse, I think SigReg would have been more direct.

The LeWorldModel work uses sequential video frames and SigReg, but it just minimizes latent distances between neighboring frames; you really want to predict (the ‘P’ in JEPA) from one latent to the next. Linear extrapolation based on the previous frame kind of works, but some level of conditioning on the current latent should be better.

Still, I’m unsure about the soundness of using frame subtraction to create the delta, since it has both frames entangled in it, so it really isn’t doing any kind of causal prediction. The architectural prior here is “only represent things that can be disentangled from a delta frame”, and I’m not sure that is universally valuable.

In their limitations section, they note that scaling to larger video datasets did not help their performance, but they expect better datasets and hyperparameter tuning will.

[davinci @leothecurious]

this is cool work that i'm really excited about. honestly, looks to me like latent space predictive coding slowly being rediscovered.

regardless, i think that "motion encoder" here can simply be a compressed representation (preferrably spatiotemporally multiscale) of whatever the model itself thinks will "happen next" given the previous frame encoding (no lookahead cheating during training, the motion encoding prediction task itself offers very rich learning signal).

in another direction, jepa does get one aspect of its assumed inductive biases right, and that is assuming spatial correlations. while this work simply assumes rich correlations across the temporal dimension, which are valid, it overcorrects by completely ignoring rich correlations across the spatial dimensions (i.e. predict parts of image from encodings of other parts). i hypothesize that this misses out on an entire dimension of learning signal and will likely degrade sample efficiency relative to a spatiotemporal prediction alternative. while it is true that the jepa/dino augmentations are often convoluted or constraining, that doesn't justify throwing the baby out with the bathwater by entirely stripping away the "learn from spatial correlations" aspect of latent prediction SSL methods.

u can elegantly do both TDV and SDV, as the brain seems to already do.

[ninad @ninaddaithankar]

you're welcome :)

1. without the motion encoder, the training boils down to the standard DINO recipe but with only a very small augmentation due to time diff between adjacent frames. as shown in the ablation for DINO table it collapses when there's no augmentation between the two views.

2. DINO head objective was added as a collapse prevention mechanism + it helps with learning semantic representations due to the clustering effect of the loss (although SIGReg might be a better choice now)

3. because we think just predicting delta z_t keeps the role of the motion encoder simpler and interpretable. it translates RGB diff into the representation diff. having it directly predict the next frames latent would tangle it up with world modeling

[ninad @ninaddaithankar]

Can a vision model learn to see with no augmentations, no masking, no cropping, no reconstruction?

🎬 It can! Introducing Temporal Difference in Vision (TDV), a new visual representation learning paradigm built on a single assumption: the past causes the future.

TL;DR :
- We introduce TDV, the first approach to learn useful representations without any augmentations, masking, cropping or pixel based reconstruction.
- TDV matches SOTA recipes like DINO and iBOT on dense spatial tasks
- We also show that as data scales up, weaker assumptions work better.

🧵Thread:

[ninad @ninaddaithankar]

agree on the different strides (supporting that in a stable arch needs more work). about causal prediction, the RGB delta exists as TDV is an architecture for representation learning rather than world modeling (i.e., learning a good visual encoder, not necessarily a world model), so the model needs some info about the next frame.

[ninad @ninaddaithankar]

@zongze_wu @AlexiGlad @ylecun we haven't released any yet but we plan to. we've only trained on ssv2 tho for our experiments

[ninad @ninaddaithankar]

@ID_AA_Carmack @AlexiGlad thank you so much for carefully reviewing our work John! :)

[ninad @ninaddaithankar]

@seanpianka @ID_AA_Carmack @AlexiGlad true that xD

[ninad @ninaddaithankar]

still processing the fact that john carmack carefully reviewed my first paper!

we genuinely appreciate your thoughts @ID_AA_Carmack. all points noted for tdv v2 :)

[John Carmack @ID_AA_Carmack]

Paper review:
You Don’t Need Strong Assumptions: Visual Representation Learning via Temporal Differences
https://t.co/w4zUfV0ZkX
https://t.co/ji2imtMQNv
@AlexiGlad @ninaddaithankar

The premise is that the more data you can use, the fewer inductive biases you should have. Starting with strong priors is helpful with limited data, but eventually, architectural priors will hinder learning true knowledge buried in sufficiently large datasets. That sounds correct.

Concretely, the ad hoc image cropping / masking / augmentations used in self supervised representation learning all make assumptions about what is important in the images, and appendix A gives examples where they can be harmful.

Figure 3 looks very compelling for this argument, but if you notice that the X axis is log scale, it is sketchier; the anchoring values on the left are relative values from experiments on 0.1% of Imagenet, which I would expect to be quite high variance.

Instead of making multiple augmentations of an image for self supervision, this work uses sequential video frames as related image pairs for representation learning. They train two separate models: a frame encoder, and a “motion encoder” that takes the RGB subtraction between the sequential video frames to produce a delta vector. The models are jointly trained so that the first frame’s representation vector, added to the delta vector, will equal the second frame’s representation vector.

They use a fairly substantial 0.25 second stride between images in the pair, noting that too small of a stride results in near-zero differences in slow moving scenes, while too large of a stride gives incoherent pixel jumps.

It looks to me like the model should be stride independent, and they could simultaneously train on many different strides, increasing the dataset diversity.

I don’t like the DINO EMA teacher approach for avoiding collapse, I think SigReg would have been more direct.

The LeWorldModel work uses sequential video frames and SigReg, but it just minimizes latent distances between neighboring frames; you really want to predict (the ‘P’ in JEPA) from one latent to the next. Linear extrapolation based on the previous frame kind of works, but some level of conditioning on the current latent should be better.

Still, I’m unsure about the soundness of using frame subtraction to create the delta, since it has both frames entangled in it, so it really isn’t doing any kind of causal prediction. The architectural prior here is “only represent things that can be disentangled from a delta frame”, and I’m not sure that is universally valuable.

In their limitations section, they note that scaling to larger video datasets did not help their performance, but they expect better datasets and hyperparameter tuning will.

[Alexi Gladstone @AlexiGlad]

Thanks for the paper review (I'm co-first author) :) I agree with most of what you said. We'll definitely try out more seeds for the 0.1% runs (good catch on that)!

Regarding swapping dino EMA for sigreg---totally agree, we conducted most experiments pre sigreg being a thing, but this is first on our todo list

On the temporal stride, I think this would be interesting to try more comprehensively (we did try this a bit), but you'd probably need to provide conditioning for it, which makes encoding a single image hard. If you don't provide conditioning, I'd guess collapse happens based on our previous experiments

On the last point, the purpose is more about representation learning than world modeling (i.e., learning a good visual encoder, not necessarily a world model), but it would be interesting to use TDV with something like LeJepa as another SSL learning signal. Or for something like DINO-WM as a pretrained encoder

For the delta, I think it's mostly valuable in domains where temporal consistency is common (like video, audio, tactile, etc). I agree it's not universally valuable, since some modalities change abruptly (like text). Although, many real-world modalities have high temporal consistency

Thanks for the feedback though, we'll definitely try some of these ideas!

[ninad @ninaddaithankar]

Good question! The frame (t+1) goes through a frozen frame encoder, which is ema of the online student frame encoder. So frame (t+1) doesn’t contribute any gradients to the encoder update, which minimizes information leak. The only info about t+1 comes from the RGB diff that’s input to the motion encoder, which we found min required info about frame (t+1) since we want to learn good representations, not a world model.

[ninad @ninaddaithankar]

appendix B.1 documents the struggle!

[Alexi Gladstone @AlexiGlad]

P.S. This project was insanely difficult implementation-wise---pushing on an entirely new paradigm for representation learning is not easy!

We therefore see TDV as laying the foundation for future representation learning approaches that aren't reliant on strong assumptions

Huge shoutout to @ninaddaithankar for being able to push through these challenges

[ninad @ninaddaithankar]

@M33pinator @HamidMaei @AlexiGlad @ylecun @RichardSSutton that’s correct!