Alexander Shabalin

PhD student at @constructor_uni | Research scientist at @bayesgroup
Developing text diffusion models

[Alexander Shabalin @a__shabalin]

Why Gaussian diffusion models fail on text data and how to prevent it?

☝️ We find that discrete-like latent spaces are fundamentally bad for continuous diffusions.
☝️ We explain what happens inside, and why self-conditioning and other heuristics improve generation.

🧵👇 1/7

[Sander Dieleman @sedielem]

New blog post: continuous diffusion for language is back!

This research direction receded into the background for a while, but as of this year, it is once again a hot topic. I wrote down a historical perspective and some thoughts on the recent revival.

https://t.co/wNafED6aDB

[Ildus Sadrtdinov @irsadrtdinov]

💨 Did you know neural networks can behave like literal ideal gases? 💨
In our #IJCAI2026 paper, we show that at stationarity, scale-invariant NNs surprisingly obey... the ideal gas law 🤯
Thermodynamics helps us to understand how train hyperparams shape final solutions!
🧵👇1/8

[Alexander Shabalin @a__shabalin]

Our paper on continuous text diffusion analysis was accepted to @COLM_conf!

🥲The problem is that none of the authors will probably be able to get a US visa. If you're going to COLM and you can help with poster hanging, please write to me.

[Alexander Shabalin @a__shabalin]

Why Gaussian diffusion models fail on text data and how to prevent it?

☝️ We find that discrete-like latent spaces are fundamentally bad for continuous diffusions.
☝️ We explain what happens inside, and why self-conditioning and other heuristics improve generation.

🧵👇 1/7

[Sander Dieleman @sedielem]

📢 Diffusion circle at @icmlconf 2026: join us Thursday July 9 at 3:30PM at the information desk / job board, we'll head out from there and find a spot to sit. No agenda, just get together and talk shop.

Please tell your friends and tag people who might be interested!

[Alexander Shabalin @a__shabalin]

@osclsd Thank you!

[Alexander Shabalin @a__shabalin]

Why Gaussian diffusion models fail on text data and how to prevent it?

☝️ We find that discrete-like latent spaces are fundamentally bad for continuous diffusions.
☝️ We explain what happens inside, and why self-conditioning and other heuristics improve generation.

🧵👇 1/7

[Alexander Shabalin @a__shabalin]

@AlanNawzadAmin Thanks! It seems like we discovered similar trends! Your observation about the deterministic path should be related to what we call the critical interval.

I see that you only measure Gen PPL. Do you have any ideas whether your solution to Gaussian diffusion hurts diversity?

[Alexander Shabalin @a__shabalin]

Thank you!
We ran experiments because, in theory, there is no difference between image and text diffusion; both utilise the same diffusion process, just in different spaces. In fact, we show that the optimal denoiser for RHM works perfectly.

The problem arises when we constrain the denoiser's expressivity by choosing a fixed-size model. Therefore, everything can only be analysed empirically.

[Alexander Shabalin @a__shabalin]

@eb1aexperts Thanks!

[Alexander Shabalin @a__shabalin]

@giladturok Thank you! We didn't run any experiments with flow-matching, but most probably the same problem will persist if the initial distribution is a mixture of delta functions. It's just too easy for the latent to become OOD closer to the end of generation.

[Alexander Shabalin @a__shabalin]

Seq2seq is the exception: the model leans on the source sequence, so OOD barely hurts, and the heuristics help much less.

📍 The stronger the conditioning, the smaller the win from SC & QS.

📄 Paper: https://t.co/pftGewiJEu

w/ @semenelist, @Viacheslav91112, @irsadrtdinov, Dmitry Vetrov

7/7

[Alexander Shabalin @a__shabalin]

Does any of this transfer to real data? Yes.

3 diffusion types, across ROCStories, Wikipedia, OpenWebText + protein generation.

Two takeaways:
📍 Latent space geometry is key: the more continuous it is, the better.
📍 You can't stop the failure mode without paying in sample diversity.

6/7

[Alexander Shabalin @a__shabalin]

🥝🍑 SMOOTHIE was accepted to ICML 2026!

A new text diffusion method that generalises simplex diffusion by gradually smoothing the distances to all vocab embeddings.

This makes denoising more natural for text data, resulting in improved generation.

📄 https://t.co/drWvR3UOpq

[Viacheslav Meshchaninov @Viacheslav91112]

Continuous diffusion for text is taking off.

But most methods still diffuse in spaces that were never learned for denoising and this limits their performance.

In LDLM, we train the encoder, diffusion model, and decoder jointly, so the denoising objective shapes the representation directly.

❌ Not: learn a space → run diffusion on it.

✅ Instead: learn a diffusable space.

On OpenWebText & LM1B, LDLM achieves a better quality–speed trade-off than MDLM, Duo, CANDI, and FLM.

Up to 13× faster sampling.

🧵 Recipe and insights below ↓