David

[Discrete Diffusion Reading Group @diffusion_llms]

📢 June 1 (Mon): ELF: Embedded Language Flows

🤔Unlike their image-domain counterparts, today’s leading diffusion language models (DLMs) primarily operate over discrete tokens.

💡The authors show that continuous DLMs can be made effective with minimal adaptation to the discrete domain. They propose Embedded Language Flows (ELF), a class of diffusion models in continuous embedding space based on continuous-time Flow Matching. Unlike existing DLMs, ELF predominantly stays within the continuous embedding space until the final time step, where it maps to discrete tokens using a shared-weight network.

🔧This formulation makes it straightforward to adapt established techniques from image-domain diffusion models, e.g., classifier-free guidance (CFG).

📈Experiments show that ELF substantially outperforms leading discrete and continuous DLMs, achieving better generation quality with fewer sampling steps. These results suggest that ELF offers a promising path toward effective continuous DLMs.

This Monday, Keya Hu (@HuLillian39250) and Linlu Qiu (@linluqiu) will present their jointly led paper ELF.

[David @LiDavid2002]

Very exciting direction. Eager to dig into the preprint and see where this leads.

[Yazid Janati @yjelid]

Happy to share our recent preprint:

Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation

Takeaways:
- standard implementations of uniform diffusion do not learn a denoiser, but rather a
"leave-one-out" denoiser
- uniform diffusion and masked diffusion are two sides of the same coin

paper: https://t.co/lMNCuyKhyb
code: https://t.co/HpSoYz0DFR (we release 9 models trained on owt)

with @samsongvch @dario_sha @umutsimsekli @EricMoulines @ericxing
@AlainDurmus

A short thread on our findings:
(1/5)

[David @LiDavid2002]

So eager to see this!

[Discrete Diffusion Reading Group @diffusion_llms]

📢 May 25 (Mon): Language Modeling with Spherical Geometry 📷

💡Join us to hear Justin (@jdeschena) and Jannis (@JChemseddine) present their recent work on (Hyper)spherical language modeling!

⚖️Discrete Diffusion and Continuous Flow Language Models (DLMs / FLMs) have emerged as interesting alternatives to autoregressive models. Yet they face fundamental tensions: discrete diffusion samples from a factorized distribution that is strictly less expressive than AR. FLMs avoid factorized sampling but typically add Gaussian noise on one-hot vectors or embeddings. It is far from clear that this kind of noise is well suited to text generation.

🤔Both papers ask the same question: what if the natural geometry for language flows isn't Euclidean space or the probability simplex, but the sphere? The hint has been there for a while: prior work like CDCD (@sedielem et al.) already operates on normalized vectors, and empirically, the cosine distance outperforms the Euclidean one for comparing word embeddings (think of word2vec, GloVe, or retrieval systems).

🧭By lifting tokens onto Sᵈ⁻¹, the authors develop tools for spherical language modeling via SLERP and vMF paths. The vMF path has the added benefit of a closed-form score, enabling principled predictor–corrector samplers on the sphere.

📈 Working with the sphere leads to concrete performance improvements: on code generation with TinyGSM, prior FLMs reach roughly 0% accuracy, while flows on the hypersphere reach 12–18% 🚀. This still lags behind the AR and discrete diffusion baselines, but it strongly suggests that spherical embedding geometry is a natural noise model for tokens. At matched NFE, a properly tuned PC sampler with vMF paths clearly improves the accuracy on Sudoku. And as a bonus, training with rotations avoids materializing one-hot vectors, making it cheaper than standard FLM training⚡️.

🔗 Language Modeling with Hyperspherical Flows: https://t.co/FERfcYZKfZ
🔗 Spherical Flows for Sampling Categorical Data: https://t.co/tKc5EthyIv

🤝 Joint work with Caglar Gulcehre (@caglarml), Gregor Kornhardt (@gregorkornhardt), and Gabriele Steidl (https://t.co/VKLrbhpFVX)

[David @LiDavid2002]

Amazing work and interesting results🔥. The next step is to evaluate them in benchmarks

[Oscar Davis @osclsd]

We were all wondering whether Categorical Flow Maps (CFMs) could scale... 🤔

I couldn't help trying it out...

So we scaled CFMs to 1.7B parameters over 2.1T tokens 🚀🔥

Short summary 🧵⬇️

[Oscar Davis @osclsd]

We were all wondering whether Categorical Flow Maps (CFMs) could scale... 🤔

I couldn't help trying it out...

So we scaled CFMs to 1.7B parameters over 2.1T tokens 🚀🔥

Short summary 🧵⬇️

[David @LiDavid2002]

@jdeschena huge thanks for the recording!

[Justin Deschenaux @jdeschena]

📢 Missed the talk? Check out the recording on YouTube: https://t.co/NMVCulBSNH

[Zhihan Yang @zhihanyang_]

📢Excited to share our new paper:

Continuous Diffusion Scales Competitively with Discrete Diffusion for Language

We introduce RePlaid 🌊, a continuous diffusion language model (DLM) with
🏅Discrete likelihood bound
🏅Scaling laws competitive with SOTA discrete DLMs

How? Dive in👇[🧵1/12]

Paper: https://t.co/Q00fmvtHmy

Work done with my amazing collaborators: @WeiGuo01 @ShuibaiZ69721 @ssahoo_ @YongxinChen1 @ArashVahdat @MardaniMorteza @jwthickstun

[David @LiDavid2002]

📄 Paper: https://t.co/ATpyG6rYwn
💻 Code: https://t.co/Vad4bnNXjN
📢 Invitation post: https://t.co/1Ny00y2nHN

Many thanks to the organizers for the invitation,
@jdeschena, @ssahoo_, @zhihanyang_! 🙌

#ICML2026 #DiffusionModels #LanguageModels #GenerativeAI #MachineLearning

[Discrete Diffusion Reading Group @diffusion_llms]

📢 May 18 (Mon): IDLM: Inverse-distilled Diffusion Language Models

🤔Diffusion Language Models (DLMs) have recently achieved strong results in text generation. However, their multi-step sampling leads to slow inference, limiting practical use.

💡To address this, the authors extend Inverse Distillation, a technique originally developed to accelerate continuous diffusion models, to the discrete setting. However, this extension introduces both theoretical and practical challenges.

🔧To overcome these challenges, the authors first provide a theoretical result demonstrating that their inverse formulation admits a unique solution, thereby ensuring valid optimization. They then introduce gradient-stable relaxations to support effective training.

📊As a result, experiments on multiple DLMs show that their method, Inverse-distilled Diffusion Language Models (IDLM), reduces the number of inference steps by 4×—64×, while preserving the teacher model’s entropy and generative perplexity.

This Monday, David Li (https://t.co/8geY1UUQ72) and Nikita Gushchin (https://t.co/L4nc4Gz7ZC) will present their jointly led paper, which was recently accepted at ICML 2026.

Collaborators of this work include: Dmitry Abulkhanov (@dabulkhanov_), Eric Moulines (https://t.co/tK9ctSzw0r), Ivan Oseledets (@oseledetsivan), Maxim Panov (@maxim_panov), Alexander Korotin (https://t.co/RTSk7HAc3v)

Paper link: https://t.co/L8MniD5GBu

[David @LiDavid2002]

🚀 Today, @iNikitaGushchin and I will be presenting our work on IDLM as part of the dLLM Reading Group!

We’ll discuss the main ideas behind the paper, what motivated this direction, and why we believe it is an exciting step for diffusion-based language modeling.

[David @LiDavid2002]

I also shared a brief overview of the core idea in my Telegram channel for anyone curious, you can check it out here: https://t.co/iszxS5iiWf

We’ll be happy to see everyone interested in diffusion LMs, generative modeling, or just curious about where this field is heading.

[Emiel Hoogeboom @emiel_hoogeboom]

Since I'm between jobs, I've been having a lot of fun vibe-coding with public tooling.

First drop: a clean PyTorch impl of the Gradient Moment metric from our recent paper (arXiv:2603.20155).

https://t.co/mZ5a8raSTk

[Discrete Diffusion Reading Group @diffusion_llms]

📢 May 18 (Mon): IDLM: Inverse-distilled Diffusion Language Models

🤔Diffusion Language Models (DLMs) have recently achieved strong results in text generation. However, their multi-step sampling leads to slow inference, limiting practical use.

💡To address this, the authors extend Inverse Distillation, a technique originally developed to accelerate continuous diffusion models, to the discrete setting. However, this extension introduces both theoretical and practical challenges.

🔧To overcome these challenges, the authors first provide a theoretical result demonstrating that their inverse formulation admits a unique solution, thereby ensuring valid optimization. They then introduce gradient-stable relaxations to support effective training.

📊As a result, experiments on multiple DLMs show that their method, Inverse-distilled Diffusion Language Models (IDLM), reduces the number of inference steps by 4×—64×, while preserving the teacher model’s entropy and generative perplexity.

This Monday, David Li (https://t.co/8geY1UUQ72) and Nikita Gushchin (https://t.co/L4nc4Gz7ZC) will present their jointly led paper, which was recently accepted at ICML 2026.

Collaborators of this work include: Dmitry Abulkhanov (@dabulkhanov_), Eric Moulines (https://t.co/tK9ctSzw0r), Ivan Oseledets (@oseledetsivan), Maxim Panov (@maxim_panov), Alexander Korotin (https://t.co/RTSk7HAc3v)

Paper link: https://t.co/L8MniD5GBu

[Justin Deschenaux @jdeschena]

🔥 New paper: Language Modeling with Hyperspherical Flows

Recent flow language models (FLMs) all use Gaussian noise. Makes sense for images, but not necessarily for text 🫠 We propose to add noise by rotating embeddings on 𝕊^{d−1} instead 🌐

w/ @caglarml
(1/9)

[David @LiDavid2002]

@Sam_Acqua Yeah, I think evaluating gen. ppl with matching entropy would provide a fairer comparison. Otherwise, gen. ppl can always be reduced by lowering the entropy of the output distribution, for example through temperature scaling

[David @LiDavid2002]

@Sam_Acqua Also, reporting perplexity bounds is not always straightforward. Some models, like distillation-based ones, are not trained with an ELBO or a clear likelihood objective, so it is unclear how to compare perplexity fairly.

[David @LiDavid2002]

@Sam_Acqua I agree with the criticism of these metrics, but I guess we just do not have any options.

The main issue is that academia often cannot train large models, so benchmark results on small models may be less meaningful, even if downstream metrics are better in principle.

[David @LiDavid2002]

@emiel_hoogeboom Thank you for citing our work, we really appreciate it. I think our works are quite closely related. We already have some ideas for pushing this further and would be excited to explore a collaboration.