Wei Deng

Research Interests: Sampling and Diffusion (Language) Models

AI/ML researcher @ Morgan Stanley, 21-. Purdue AI/ML PhD ’21. Ex-Baidu, 12–15.

[Wei Deng @dwgreyman]

@Adrian_Z_Wang @zdhnarsil I believe many of them are impressive engineering achievements, but there are still plenty of interesting problems that can be tackled with eight H100s or B100s. Moreover, this resource constraint is likely to become less significant over time.

[Wei Deng @dwgreyman]

Maybe it’s mostly a compute gap—big labs simply have 100x-1M more GPUs, so they naturally focus on different problems. As compute becomes more accessible, I think this gap will shrink. Huge respect for big labs pushing scaling and bringing AGI closer, but algorithmically, the progress has still been fairly incremental.

[Wei Deng @dwgreyman]

@iampanxu Congrats, well deserved!

[Wei Deng @dwgreyman]

@madprizm0 @ssahoo_ good choice for PhD study indeed

[Wei Deng @dwgreyman]

Reposting this as well — hopefully it brings more confidence to the community.

[Subham Sahoo @ssahoo_]

Sneak peek at the Diffusion LLM that will be released soon

I’m at #ICML2026 in Korea presenting two papers (details in the comments) and joining a panel discussion at the SPIGM workshop.

Drop by my posters or DM me if you’d like to chat more.

[Wei Deng @dwgreyman]

@ssahoo_ Thanks — really looking forward to the paper!

Also, huge appreciation for the work you have done. This brings a lot of confidence and momentum to the community, and I’m excited to see the details.

[Wei Deng @dwgreyman]

The background figure is so similar to binomial/trinomial tree methods in option pricing, except that it differs in: 1. low versus high dimensionality, and 2. continuous versus discrete dynamics.

[Recursive @Recursive_SI]

https://t.co/IlLyWptydX

[Wei Deng @dwgreyman]

@atomic_chat_hq pure uniform is costly, mask+uniorm is more practical.

[Wei Deng @dwgreyman]

@StefanoErmon @_inception_ai @wef Huge congrats to Inception — Mercury is one of the most exciting bets in coding agents.

Being rusty on basic ML details is humbling 😅

Rooting for Mercury.

[Wei Deng @dwgreyman]

@MrDeGao Thanks so much for your kind words, DeGao! We’re really looking forward to scaling it up!

[Wei Deng @dwgreyman]

1/N: Best base model for discrete diffusion LM? Masked diffusion is cheap, but it has weak native self-correction ability. Remasking introduces ad-hoc parameters, making the pipeline hard to tune and generalize. Pure uniform diffusion has a stronger correction potential, but it comes with a quadratic state space over the vocabulary, making it unscalable and much more expensive to train.

We therefore revisit the mask+uniform direction previously explored by GIDD and present Self-Correction Discrete Diffusion (SCDD), which can be viewed as a real multi-step BERT. We exploit the absorbing structure not only for the mask state, but also for uniform token-to-token transitions, making our formulation much simpler than GIDD. We do not rely on ReMask for self-correction; instead, everything is learned during pre-/mid-training to maximize generalization potential. Here is how SCDD compares with GIDD and MDLM.

[Wei Deng @dwgreyman]

4/ N: Free feel to check https://t.co/UaduFgZqB8 and https://t.co/RTn8MWCT4u

[Wei Deng @dwgreyman]

3/ N: In GPT-2 parallel decoding, SCDD exhibits the smallest quality drop. We are working on scaling it up, with the goal of enabling stronger base models to develop stronger self-correction ability.

[Wei Deng @dwgreyman]

@aniketdxsh Thanks for your kind words. Writing blogs is a good way for me to learn and glad you enjoy them as well

[Wei Deng @dwgreyman]

@baaadas Discrete diffusion (mask + uniform), feels like a cheap and effective way to extend AR models to parallel decoding. Continuous dLLMs may work better in images, but for LMs I’m not convinced they’re empirically worth it.

[Wei Deng @dwgreyman]

@dvruette @jdeschena Thanks, great suggestion! We kept the v1 objective in an expanded ELBO form, but I agree a cleaner KL + pointwise IS-style marginal-matching view should be possible. Will keep you updated when we have that extension.

[Wei Deng @dwgreyman]

@jdeschena @dvruette @dvruette appreciate any comments on SCDD. I love the pioneering work of GIDD, and I believe we can make formulation even cleaner to be more scalable.

[Wei Deng @dwgreyman]

@jdeschena @dvruette GIDD is a nice work, but the formulation is a bit complex and hard to scale; the mask and uniform signal-to-noise ratios are coupled. We can do it better (check how simple our generator is compared to GIDD). the arXiv v1 is here: https://t.co/ZwVlLoO4tT, updating ICML v2 soon