Rodrigo Casado Noguerales

PhD student @ETH_en · discrete diffusion

[Rodrigo Casado Noguerales @rodcasnog]

1/ If you have been reading about discrete diffusion, you might have asked yourself the following question:

What does a discrete diffusion model actually learn?
A denoiser? A concrete score? A bridge plug-in?

The literature treats them as interchangeable. They are not! 🧵

[Tiago Pimentel @tpimentelms]

I'm looking for emergency ACs for some interpretability + reasoning papers in EMNLP :) If you're available and think you qualify, please reach out! 📷 Ideally, you should be a postdoc (or senior PhD) for this! If you don't know if you qualify, feel free to ask me about it :)

[Antonio Orvieto @orvieto_antonio]

On my Seoul -> Beijing i will read this! DLMs are just optimizing a loss.. but they also happen to be generative. The connection is much less trivial than for next token prediction.

[Rodrigo Casado Noguerales @rodcasnog]

@rbuit_ Thanks a lot Ricardo! We will follow up digging more into the many practical implications. Stay tuned!

[Rodrigo Casado Noguerales @rodcasnog]

1/ If you have been reading about discrete diffusion, you might have asked yourself the following question:

What does a discrete diffusion model actually learn?
A denoiser? A concrete score? A bridge plug-in?

The literature treats them as interchangeable. They are not! 🧵

[Rodrigo Casado Noguerales @rodcasnog]

@dvruette Thanks a lot Dimitri! More to come 🫡

[Rodrigo Casado Noguerales @rodcasnog]

@jramapuram Thanks, that means a lot!!

[Jason Ramapuram @jramapuram]

Some good science on discrete diffusion here.

[Dimitri von Rütte @dvruette]

Treasure trove of insights for anyone looking to develop an intuition for WTF is going on with discrete diffusion models! Super exciting work by @rodcasnog 👏

[Rodrigo Casado Noguerales @rodcasnog]

9/ Joint work with Bernhard Schölkopf @bschoelkopf, Thomas Hofmann, and Aran Raoufi.

Paper: https://t.co/iIQ4HK78uO
Blog post with the interactive figures: https://t.co/HmF8MryaEr

Questions and feedback very welcome. This is my first paper! 🎉

[Rodrigo Casado Noguerales @rodcasnog]

8/ Practical rules:

1. Match the NN head to its loss at sampling time, or convert it.
2. Don't use a denoiser head for UDM/GIDD noise: the ELBO diverges at init.
3. Unit test: a cavity NN at init gives per-token NELBO = log V, always.
4. Keep the boundary terms before comparing!