Ilia Sudakov

GenAI Researcher @YandexResearch
MSc Student at HSE University

Seeking PhD

[Discrete Diffusion Reading Group @diffusion_llms]

This Monday, Nov 2, Dmitry Baranchuk (@DmitryBaranchuk) will present his paper:

Representation-Space MMD for Diffusion Language Models

Project also co-led by Ilya Drobyshevskiy (@free001style) and Ilia Sudakov (@sudakov_il)!

[Ilia Sudakov @sudakov_il]

MMD (Maximum Mean Discrepancy) has shown promise in image generation. Can it improve diffusion language models too?

We post-train DLMs to match generated and reference distributions in a frozen DLM’s feature space. It works for both discrete and continuous models, and scales to 16B! 🚀

The results are strong across text generation, math reasoning, and code! Here’s how it works 👇

[Ilya Drobyshevskiy @free001style]

Introducing a simple method to post-train both discrete and continuous diffusion LMs.

It's based on Maximum Mean Discrepancy (MMD) that matches token-level feature distributions. Our method improves generation and speed on OWT/GSM8K and scales to 16B models. 🧵