QIWEI DI

CS PhD candidate at UCLA || Outstanding reviewer for @ICLR 2024, Top reviewer for @Neurips 2024 || 2025 Amazon Fellow UCLA

[Xuheng Li @xuhengli_]

Another piece of AI-assisted research where AI helped us to think out of the box. Our initial idea was to characterize the high-probability event where the sampled reward function f is close to f* point-wise, similar to UCB; AI said it was unnecessary: Markov' inequality solves everything neatly.

It was a hard time understanding the key technical contributions in the initial version of AI output though. In the age of AI-assisted research, we theorists are "priests" to understand the "oracle".

We are working on automating the process of understanding and presentation of AI-generated math. There's more to come.

[Qingyue Zhao @ZhaoQingyue]

Feel-good Thompson sampling (FGTS) achieves Õ(d√T) expected regret in linear contextual bandits, but does single-run regret concentrate? Our new paper proves the first high-probability regret bounds for FGTS: https://t.co/jT6ah3mP3h

[Francesco Orabona @bremen79]

https://t.co/0zmf7jQaeK

[QIWEI DI @di_qiwei]

Had a great time working on this. I’ve never been happy treating training as a black box, or stopping at a better loss curve. Working out why Muon learns facts more evenly was the fun part, and also the hard part.

AI helped us a bit on the proofs along the way, too. And with recent developments like the ETH result this week, AI is really proving its ability to solve difficult problems. Honestly feels like the best time to be doing learning theory.

[Xuheng Li @xuhengli_]

Muon now trains frontier LLMs. But how does its spectral orthogonalization shape feature learning?

New paper: on factual recall, Muon learns facts in a more balanced way and is far less slowed by softmax saturation.
🔗https://t.co/hPm5VI0SG2
[1/8]

[Lijie Chen @wjmzbmr1]

The exponential time hypothesis is FALSE!

[Aditya Grover @adityagrover_]

Over the past two years of building @_inception_ai, one question has come up for me more than any other across countless conversations: Does the diffusion advantage scale?

Today, we're answering that question with Mercury 3, our most capable diffusion LLM yet. Mercury 3 surpasses GPT-6-Luna and Claude 5.5 Haiku while delivering blazing-fast inference at 1000+ tokens/sec.

More importantly, the Mercury factory is up and running: Compared to Mercury 1, Mercury 3 is 40x larger and 3x more intelligent, all while maintaining the same industry-leading speed.

The diffusion advantage doesn't just scale. It compounds.

[QIWEI DI @di_qiwei]

7/ Joint work with @xuhengli_ , @Kaixuan_Ji_19 , @ChenggongAlex , @HeyangZhao and @QuanquanGu.
📄Paper: https://t.co/HwrS9TMkzI

[QIWEI DI @di_qiwei]

6/ ⚠️ Third, OPD struggles on long rollouts. Its next-token target carries a to-go term summing over all continuations. Diffuse continuations inflate it, more so with length. The expert is 99% sure of the first token, yet past H≈67 OPD's target picks the wrong one.

[QIWEI DI @di_qiwei]

5/ ⚠️ Second, a mismatched teacher can break OPD. Its target averages log-probs, so one low-weight teacher giving the right answer ~0 mass can veto it. Expert 0.99 (weight 0.9), mismatched teacher 1e-30 (weight 0.1): SFT's target keeps 0.89, OPD's falls to 0.06.

[QIWEI DI @di_qiwei]

4/ Why this view helps: comparing the targets explains patterns seen in practice. ✅ First, OPD forgets less. When later teachers know nothing about an expert's prompts, SFT's target dilutes its right answer (0.95 → 0.865); OPD's barely does (0.919), if the expert is confident.

[QIWEI DI @di_qiwei]

3/ Each round the student hears from one teacher, yet it must learn the aggregate of all of them. We prove it does: from teacher-written responses (SFT) it converges to the arithmetic mixture; from teacher scores on its own outputs (OPD), to the geometric mean. Regret: Õ(log T).

[QIWEI DI @di_qiwei]

2/ We frame this as sequential distillation: each round, one teacher (a domain expert or an earlier checkpoint) trains the student on its own data. Goal: one student that minimizes its average divergence to all teachers. Fitting only the current teacher is forgetting.

[QIWEI DI @di_qiwei]

OPD forgets less than SFT, yet breaks under teacher mismatch and on long rollouts. Why? Our answer: they optimize different targets. With multiple teachers, SFT (forward KL) learns their arithmetic mixture; OPD (reverse KL) learns their normalized geometric mean. 🧵

[QIWEI DI @di_qiwei]

@saa1605 @xwang_lk @OpenAI same

[Guitar Cat + LLM 「Will @ SGLang Summit in SF」 @GenAI_is_real]

The mathematical breakthrough I most want from @OpenAI is a better theory of why deep learning works.

Why can gradient descent find solutions that generalize when there are so many ways to fit the training data? What determines whether a learned skill transfers or falls apart in a new setting?

We already have important pieces of the theory. I'd love something predictive enough to guide the next training run, rather than just help explain the last one. Ideally before we spend the GPU budget.

A neural network helping explain why neural networks work would be a pretty satisfying plot twist.

[Quanquan Gu @QuanquanGu]

Biology could be AI’s next recursive self-improvement frontier.

Design in silico → test in the lab → turn experimental feedback into semi-synthetic training data → improve the model → repeat.

At @Geodesiclab , we’re building toward that flywheel.

The wet lab becomes part of the learning algorithm.

[Chenggong "Alex" Zhang @ChenggongAlex]

1/5

🚀 New paper: Saturation-Insensitive Dueling Bandits with General Function Approximation

Can we learn from pairwise preferences without paying exponentially for BTL saturation? Answer: YES!

@ChenggongAlex @xuhengli_ @di_qiwei @WeitongZhang @QuanquanGu

🔗https://t.co/WASIeCfYza

[Guitar Cat + LLM 「Will @ SGLang Summit in SF」 @GenAI_is_real]

Hi everyone, today we released SGLang Omni v0.1.7. This release includes 75 merged PRs and welcomes 8 new contributors, with 8 first-time contributions. We added MiniCPM-o 4.5, NVIDIA PersonaPlex-7B, and OmniTyper powered by MLX streaming ASR, while further improving realtime and stateful Omni serving.

1.Performance: continued optimizations for Qwen3-TTS, Qwen3-Omni, CosyVoice3, MOSS-TTS, and AuK, covering Prefill CUDA Graph, speaker/reference encoding, kernel fusion, batching, and vocoder hot paths.

2.Serving: added Omni session lifecycle, the SGLang streaming session bridge, and a shared /v1/realtime WebSocket runtime, while further improving realtime ASR and streaming serving.

3.Models & hardware: added MiniCPM-o 4.5 multimodal input and speech output, plus PersonaPlex-7B offline speech-to-speech. MiniCPM-o and MiniMax-Music3 now support Intel XPU, with further MUSA support for Qwen3-TTS.

4.Runtime: improved breakable Prefill CUDA Graph, Talker / Code2Wav colocation, priority CUDA streams, scheduler admission, and profiling infrastructure to reduce host overhead and improve high-concurrency stability.

https://t.co/UkddEuyACN

https://t.co/KAyUTf8BRb

[QIWEI DI @di_qiwei]

@iampanxu Your students are lucky to have you. Good work will get its recognition.

[Sitan Chen @sitanch]

🚨 We finally have a better understanding of how the different paradigms for discrete diffusion compare!

In new work with Liye Wang, we prove a quadratic separation between the parallel generation capabilities of uniform and Gaussian diffusion relative to unmasking! 1/