Taekyun Lee

UTAustin WNCG / ECE Ph.D. student

[Discrete Diffusion Reading Group @diffusion_llms]

📢Aug 31 (Mon): From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models

🤔Many discrete reasoning tasks, such as code generation, are inherently non-causal. Programmers naturally move between high-level structure and local details in a process called any-order inference. While masked diffusion models offer a native any-order prediction interface, maximizing their potential is difficult due to a core limitation:

1️⃣ This any-order interface does not automatically yield actual any-order inference
2️⃣ Fixed-canvas, token-level models suffer from "positional uncertainty"—they may know what semantic component should appear, but not where to place it

🔑To address the interface-inference gap, the authors propose two complementary approaches designed to natively support any-order inference without relying on hand-designed mechanisms.

🔧The framework tackles positional uncertainty through two methods: (1) Insertion-based masked diffusion (building on FlexMDM), which relaxes fixed-position commitments via insertions to enable generation across non-contiguous regions, and (2) Latent-space masked diffusion (LatentMDM), which shifts prediction to coarser semantic segments to enable search over latent generation orders.

📈The resulting models—a 7B FlexMDM trained for Python coding and a 125M LatentMDM for GSM8K—successfully induce distinct any-order inference behaviors. The study demonstrates that overcoming positional uncertainty through these approaches significantly improves downstream performance.

This Monday, the joint first authors Seunggeun Kim (@seunggeun_kr), Jaeyeon Kim (@Jaeyeon_Kim_0), Taekyun Lee (https://t.co/aL55CjRNkf), and Yuyuan Chen (https://t.co/D1jETDku4r) will present their recent paper on the FlexMDM and LatentMDM frameworks.

[Seunggeun Kim @seunggeun_kr]

There are a bunch of other interesting observations and results in our paper, but I especially want to advertise LatentMDM, probably my favorite part of this work 🔥
The core idea is simple: instead of any-order generation directly in token space, do it in latent segment space.

[Jaeyeon (Jay) Kim @Jaeyeon_Kim_0]

Excited to share what we have worked on for the past 6 months.

Can we really call masked diffusion an "any-order" model?
In our new work, we argue that off-the-shelf masked diffusion models have any-order predictive power but fail to reason in a 'genuinely any-order fashion'.

We introduce two models that do: LatentMDM and 7B-scale FlexMDM, extending our prior work. LatentMDM, our new latent-space masked diffusion, outperforms AR (with KV caching) on TinyGSM, at a matched inference budget. To the best of our knowledge, LatentMDM is the first model on the same setup to do so.

Joint work with co-first authors @seunggeun_kr, @tklll_tx, and Yuyuan Chen, and advisors @du_yilun, @ShamKakade6, and @sitanch.

[ainta @kaizero_ainta]

Anthropic published a paper today, written by Claude, proving that at least 67.25% of the zeros of the Riemann zeta function are simple and on the critical line.

I gave Claude's paper to GPT-5.6-Sol Pro. It's now 67.30%.

https://t.co/xSJUmNJ3GY

[Dimitris Papailiopoulos @DimitrisPapail]

https://t.co/cpwPlGKWpD

[Jaeyeon (Jay) Kim @Jaeyeon_Kim_0]

🚨🚨🚨 Now you can stop training your masked diffusion models ''for the worst''.

We propose 🐆PUMA🐆--Progressive UnMAsking, a simple modification of the forward masking process that speeds up the masked diffusion training.

[Jaeyeon (Jay) Kim @Jaeyeon_Kim_0]

🚨🚨🚨 Now your Masked Diffusion Model can self-correct! We propose PRISM, a plug-and-play approach fine-tuning method that adds self-correction ability to any pretrained MDM! (1/N)

[Jaeyeon (Jay) Kim @Jaeyeon_Kim_0]

Huge thanks to my collaborators, including co-first authors @seunggeun_kr, who especially led all the experiments, and @tklll_tx / also professors David Z. Pan, Hyeji Kim, @ShamKakade6 , @sitanch !!
(18/N)

[Jay Alammar @JayAlammar]

The Illustrated NeurIPS 2025: A Visual Map of the AI Frontier

New blog post!

NeurIPS 2025 papers are out—and it’s a lot to take in. This visualization lets you explore the entire research landscape interactively, with clusters, summaries, and @cohere LLM-generated explanations that make the field easier to grasp.

Link in thread!

[Kangwook Lee @Kangwook_Lee]

DLLMs seem promising... but parallel generation is not always possible

Diffusion-based LLMs can generate many tokens at different positions at once, while most autoregressive LLMs generate tokens one by one.

This makes diffusion-based LLMs highly attractive when we need fast generation with less compute.

A big question is … is parallel generation possible without losing modeling accuracy?

The answer is no. There are fundamental limits on how much parallelism we can achieve.

Consider this example:

“Pick one city uniformly at random from the following four cities:

New York, New Orleans, Mexico City, or Panama City.”

Then,
P(Y₁ = New, Y₂ = York) = 1/4,
P(Y₁ = New, Y₂ = Orleans) = 1/4, and so on.

Thus, P(Y₁ = New) = 1/2, P(Y₂ = City) = 1/2.

If you choose to generate Y₁ and Y₂ in parallel, no matter which decoding algorithm you use …
You’re doomed to sample “New City.”

None of today’s DLLMs can generate these two words correctly without giving up parallelism.

-----

Why is this the case?

In fact, we never train LLMs to learn the joint distribution over multiple tokens in one forward iteration.

We always teach a single-token marginal distribution conditioned on context.

(The same holds for autoregressive models too.)

Therefore, sampling multiple tokens at once is only possible when those tokens are mutually independent given the current context.

And this limitation of parallel sampling can be precisely formalized.

One can derive an information-theoretic limit that’s decoding-strategy agnostic, and also derive strategy-specific limits.

-----

So are DLLMs doomed? No!

They have huge potential to save compute and time.

But:
(1) we need to be aware of their fundamental limitations, and
(2) we need to design better training and decoding strategies.

In particular, there’s huge room for improvement in decoding.

Why?
Ideally, we want the model to control the degree of parallelism during generation.
At the same time, it should choose a subset of future tokens that are almost mutually independent given the current context.

Are current decoding strategies good at this?

Hard to tell.
Most DLLMs were never stress-tested for it.

-----

That’s why we introduced a synthetic benchmark to stress-test DLLMs.

We call it ParallelBench.

The idea is simple: these are natural language tasks, but carefully designed so that parallel generation is inherently difficult.
(Think “New City”, but more natural, real tasks.)

What did we find?

We tested popular DLLMs with various decoding algorithms, and none came close to “oracle” performance, the ideal performance you’d get if the model could optimally adjust its parallelism during decoding.

-----

Takeaway:

(1) Parallel generation is not always possible and check out our paper for more details :)

(2) If you can design a DLLM that matches oracle performance on our benchmark, well, who knows, you might just get a call from someone in Menlo Park. 😉

[Jaeyeon (Jay) Kim @Jaeyeon_Kim_0]

(Website: https://t.co/x2AsT7cgHi)
Masked Diffusion Models (MDMs) have recently emerged as a promising alternative to autoregressive models. Despite these successes, they
❌ struggle to model variable-length sequence distributions, and
❌ insert new tokens during generation.

[Jaeyeon (Jay) Kim @Jaeyeon_Kim_0]

We introduce a new ''rule'' for understanding diffusion models: Selective Underfitting.

It explains:
🚨 How diffusion models generalize beyond training data
🚨 Why popular training recipes (e.g., DiT, REPA) are effective and scale well

Co-led with @kiwhansong0!
(1/n)

[Ernest Ryu @ErnestRyu]

New lecture recordings on RL+LLM! 📺

This spring, I gave a lecture series titled **Reinforcement Learning of Large Language Models**. I have decided to re-record these lectures and share them on YouTube. (1/7)

[Litu Rout @litu_rout_]

Continuous diffusion had a good run—now it’s time for Discrete diffusion!

Introducing Anchored Posterior Sampling (APS)

APS outperforms discrete and continuous baselines in terms of performance & scaling on inverse problems, stylization, and text-guided editing.