@StefanABaumann@WangYw251@XiaoyuW9401147 Thanks for the feedback! I'd also love to hear more about which choices you found questionable. I’d really appreciate your thoughts, as I want to address these issues and improve the work.
@BNarayanaReddyX Each TEM epoch costs more, since every training step runs 40 Langevin steps (~27× DiT’s per-step cost), so DiT starts faster. But TEM needs far fewer epochs so it scales better at the end. Training is pretty stable.
Energy-Based Models can now scale better than Diffusion models.
Introducing TEM: Transformer Energy-based Models
TLDR:
• TEM outscales Diffusion without guidance
• First scalable contrastive Energy-Based Models
• Better generalization, test-time scaling, and OOD detection than Diffusion
🧵
@Thomas_ensc@du_yilun@AlexiGlad Thanks! I think complex conditioning is still an open challenge. While EBMs support compositional conditioning, scaling this to complex text prompts needs further exploration.
@PHuembeli@AlexiGlad Thanks! Increasing the noise improves mixing, but comes at the cost of sample quality. >1000 steps they tend to remain at sharp sample region. Occasionally injecting a large noise, followed by another round of Langevin sampling, can help them jump between modes.
@peter9863@AlexiGlad Awesome work! We should definitely consider comparing against it. Since our work is not focusing SOTA generation, we focused on representative baselines rather than covering all methods. Consistency models often rely on pretrained diffusion models, different with our settings.
@vanraj_narula@AlexiGlad That's indeed one of the biggest challenges! We use replay buffers. We also think representation alignment helps regularize the energy landscape, keeping samples close to the data manifold while allowing exploration.
I wanted to share some of my personal excitement surrounding this paper.
In 2002, Geoff Hinton proposed Contrastive Divergence as a way of training EBMs. The goal then was to model p(x) in an unsupervised manner. Modeling log p(x) was easier to do, where log p(x) acts as the negative energy in a Boltzmann distribution.
But the problem with contrastive divergence back then was that it caused the energy landscape to be too sharp on the data manifold. Because images live on a thin, low-dimensional manifold in high-dimensional space, probabilistic methods like these caused the density to explode on the manifold and drop to zero everywhere else.
As a result, the community diverged and opted toward adding noise to prevent sharpness and smooth out the energy landscape, which ultimately led down the diffusion denoising route.
But we are finally back to Hinton's original vision. Very excited about what's next!
Congrats @AlexiGlad and team!
@jatingargiitk@du_yilun Yes! TEM can score and rank images from other generators without retraining on their samples. Lower energy means the image is more compatible with the learned distribution.
@aliteracy@AlexiGlad Good question. We haven’t trained gray line longer or compared it to a compute-matched diffusion baseline yet. That comparison would also show whether representation alignment helps EBMs more or less than diffusion.