Ilya Drobyshevskiy

ML Research Resident @YandexResearch

Seeking PhD

[Ilya Drobyshevskiy @free001style]

Introducing a simple method to post-train both discrete and continuous diffusion LMs.

It's based on Maximum Mean Discrepancy (MMD) that matches token-level feature distributions. Our method improves generation and speed on OWT/GSM8K and scales to 16B models. 🧵

[Justin Deschenaux @jdeschena]

Missed the talk? Check out the recording on YouTube: https://t.co/synGr2JqNs

[Dmitry Baranchuk @DmitryBaranchuk]

Really enjoyed @anvilapex’s talks at our diffusion seminars: a clear overview of post-training visual diffusion models (DPO, RL, and reward modeling).

The recordings are in Russian, but he’s now written up the material in English blog posts. Well worth a read!

[Yura Gorishniy @YuraFiveTwo]

Double announcement!

🤖 TabPack: an effective, practical baseline for medium-to-large tabular datasets. Competitive out of the box on benchmarks and trainable on a modern MacBook

📦 TabPackage: a Python package for training TabPack and building similar models

👇 Links & Thread

[Discrete Diffusion Reading Group @diffusion_llms]

This Monday, Nov 2, Dmitry Baranchuk (@DmitryBaranchuk) will present his paper:

Representation-Space MMD for Diffusion Language Models

Project also co-led by Ilya Drobyshevskiy (@free001style) and Ilia Sudakov (@sudakov_il)!

[Ilya Drobyshevskiy @free001style]

@peter9863 Nice work! We observe similar behaviour in continuous diffusion language models: our post-trained model iteratively refines predictions via self-conditioning at inference (Eq. 12). We also distill this into a single prediction (Appendix B.1, Eqs. 13–14).
https://t.co/3z4gN6rCba

[Ilya Drobyshevskiy @free001style]

Introducing a simple method to post-train both discrete and continuous diffusion LMs.

It's based on Maximum Mean Discrepancy (MMD) that matches token-level feature distributions. Our method improves generation and speed on OWT/GSM8K and scales to 16B models. 🧵

[Ilya Drobyshevskiy @free001style]

@emiel_hoogeboom @djjruhe @TimSalimans @JonathanHeek @cdomingoenrich @ShamKakade6 @ssahoo_ @jwthickstun @nmboffi @aaron_lou @JiachengYe15 @ValentinDeBort1 @StefanoErmon Also tagging @Goodeat258 @lihe50hz @TianhongLi6
@HuLillian39250 @linluqiu @Lyy_iiis @Hope7Happiness @yoonrkim @jacobandreas
@chandavidlee @wognsfjq96 @mananag_007

[Ilya Drobyshevskiy @free001style]

Introducing a simple method to post-train both discrete and continuous diffusion LMs.

It's based on Maximum Mean Discrepancy (MMD) that matches token-level feature distributions. Our method improves generation and speed on OWT/GSM8K and scales to 16B models. 🧵

[Ilya Drobyshevskiy @free001style]

Tagging people who might be interested @emiel_hoogeboom @djjruhe @TimSalimans @JonathanHeek @cdomingoenrich @ShamKakade6 @ssahoo_ @jwthickstun @nmboffi @aaron_lou @JiachengYe15 @ValentinDeBort1 @StefanoErmon

[Ilya Drobyshevskiy @free001style]

[🧵7/7] Check out the paper and code for details! 🔗

Paper: https://t.co/PtKEUW9HBZ
Code: https://t.co/Qdf1JsqeLu

[Ilya Drobyshevskiy @free001style]

[🧵6/7] Our approach scales! 🚀 We apply MMD to 16B DMax-Math and DMax-Coder. Just 400 post-training updates per model and the results are exciting!

• Math: 10.3–16.5% more tokens per forward pass, with similar or higher accuracy.

• Code: MBPP-Instruct pass@1 jumps from 79.2% to 83.0%, while tokens per forward pass rise from 5.86 to 6.10.