Are EMA teachers commonly used in video SSL (๐V-JEPA family) actually necessary?
Hot take: your EMA teacher is a waste of compute
Introducing SALT. ICLR 2026 ๐ง๐ท | Led by @XianhangLi at Apple MLR.๐งต
NM State lets go of Women's Soccer Coach Rob Baarts after NM State made a significant run in the CUSA Tourney this week, ultimately falling to Liberty on Tuesday. While the last couple of years have been a struggle, the Women's Soccer team is full of amazing and talented women who almost took out Liberty the other day.
Thank you, Coach Baarts...
Story on AGGREGATOR. Link in Bio
#AggieUp
Our first major update on https://t.co/v8KSoNBKqM
- venue confirmed in the heart of Manhattan
- paper deadline is December 22nd
- single-blind, free-form (just a page limit), use AI however you want--focus on the science
- everything public (openreview), < 3 papers per reviewer
@WangYw251 Reviewers are allowed to be wrong. Do whatever you need in order to convince them that your work is good, not just what they want. I mean pumping up their (my) ego is fine but it will just leave you bitter and exhausted.
@iclr_conf desk rejection + scoring system seems to be doing its job in my assigned batch of papers to review. One paper desk rejected just as I was getting started. A paper rated 1 was withdrawn by the authors. The papers that remain in my lot are two 3s and a single two.
Several comments:
1. image/video SSL methods differ by whether they use reconstruction or not. Joint embedding architectures Amy or may not have a predictor (JEA vs JEPA). They also differ by their anti-collapse method and data corruption/augmentation methods.
2. DINO *is* a kind of JEPA (technically a JEA). The difference with other JEPA models is the anti-collapse criterion (based on contrative clustering+ stopgradient/EMA), and the data augmentation scheme (image transformations, which is task specific),
3. V-JEPA 2.1, like DINO, is from Meta-FAIR (corporate lab), involved significant engineering teams, large/proprietary datasets, considerable computing resources, lots of fine-tuning, and number-pushing on many benchmarks.
4. Many JEPA papers are applications and/or fine-tuned versions of DINO, V-JEPA or I-JEPA.
5. LeJEPA, LeWM, and most other papers with "JEPA" in the title are *research* papers focused on the methodology, often done by students in academic labs, involving no engineering support, very little computing resources, and small/public training sets. They are generally tested on toy problems, simulated environments, or small/public benchmarks.
6. You simply cannot compare 5 with 2/3/4. 5 gives you ideas and new methods, not models. 2/3 give you well-engineered models, with lots of empirically validated tricks. Only rarely will 2/3 give you a new, well-principled method with room to spare. 5 give you empirical results on new applications.
7. What is published in 5 are methods that will eventually become like 2 and 3, whenever someone with resources picks it up and scales it.
DO NOT CONFUSE UPSTREAM FOUNDATIONAL RESEARCH ON METHODS AND DOWNSTREAM SCALING -> ENGINEERING -> NUMBER PUSHING -> MODELS -> PRODUCTS.
The former makes the rest possible, but it takes time.
2/N Spikes do not only appear in LLM training, but can also occur in much simpler systems -- e.g., learning modular arithmetic. The slingshot mechanism paper (https://t.co/r4RWU4AN6B) argues that training spikes lead to grokking.
This is the model organism we're interested in studying, given its simplicity and controllability.
I love this paper. But I can easily see R2s rejecting by using the last statement in justification. Paraphrasing Astra, โitโs just seen on modular arithmetic data which is a O(thousands) samples per operation.
Still salty about neurips 2022 rejecting slingshot that this paper solved ๐
@_onionesque@EtaiLittwin@zhaisf@Roni_Paiss@jmsusskind My speculation is that grokking, especially with weight decay=0, is not that interesting a setup for capable folks to look and root cause + fix.
With prevalence of low-precision being necessary, it was time for this issue to be unearthed and fixed. A capable team fixed it
Very happy to see this paper fix the Slingshot Effect that we observed in our paper (slight preference for https://t.co/wbW2om7TQ8). That work was a first of sorts for me. Enjoyed working with @EtaiLittwin, @zhaisf , Omid, @Roni_Paiss & @jmsusskind. We knew it was likely precision related but I was never able to identify conditions that caused training spikes to occur. Well done @jon_barron & Gemini :)
Hi all, I have a new result I'm excited to share. It turns out that *all existing implementations* of softmax and cross-entropy (that's JAX, PyTorch, textbooks, etc) have a subtle bug --- two bugs, actually. A simple rewrite fixes both, and eliminates "slingshots" when grokking:
@giffmana Two lessons from that paper:
1. Check their abstract. They solved a problem that we didnโt know we had in our Slingshot paper.
2. Going all in on representation learning now that the field has moved on from it ๐. Wonโt rest until eljepa is solved or dead