Hanqing Zeng

PhD candidate in computer engineering at USC

[Jacqueline He @jcqln_h]

Knowledge distillation (KD) is a powerful training objective—but when should we use it?

We find that KD behaves differently across training stages: vs. vanilla NTP, it improves reasoning + factual recall in pre-training, but trades off recall for reasoning in mid-training.

We introduce Switch Distillation, which improves this tradeoff using teacher uncertainty.

Paper: https://t.co/ZZd8pC5m03
1/🧵

[Zhuokai Zhao @zhuokaiz]

Excited to share that FusionRoute has been accepted to #ICML2026

Grateful to all our collaborators @XiongNuoya69368 @YuhangZhou2 @zimplex4 @ZRChen_AISafety @furongh @shuchaobi @LizhuZhang

[Hanqing Zeng @zimplex4]

@zhuokaiz Huge congrats! Really grateful for all the hard work and inspiring discussions 💪

[Zhuokai Zhao @zhuokaiz]

Meta × TBD Lab × CMU × UChicago × UMaryland

In our latest work, we introduce
Token-Level LLM Collaboration via FusionRoute
📝: https://t.co/L0bCn3yHP9

LLMs have come a long way, but we continue to face the same trade-off:
– one huge model that kind of does everything, but is expensive and inefficient, or
– many small specialist models that are cheap, but brittle outside their comfort zones

We’ve tried a lot of things in between — model merging, MoE, sequence-level agents, token-level routing, controlled decoding, etc.

Each helps a bit, but all come with real limitations.

A key realization behind FusionRoute is:
Pure token-level model selection is fundamentally limited, unless you assume unrealistically strong global coverage.

We show this formally. And then we fix it by letting the same router also generate.

Concretely, FusionRoute is a lightweight router LLM that
– performs token-level model selection, and
– directly contributes complementary logits to refine or correct the selected specialist when it fails

So it's not "routing + another model" — the router itself is part of the decoding policy as well.

This turns token-level collaboration from a brittle "pick-an-expert" problem into a strictly more expressive policy.

No joint training of specialized models.
No model merging.
No full multi-agent rollouts.

In our experiments, FusionRoute works across math, coding, instruction following, and consistently outperforms sequence-level collaboration, prior token-level methods, model merging, and even direct fine-tuning.

Feeling especially timely as LLM systems (e.g., GPT-5) move toward routing-based, heterogeneous model stacks (whether prompt-level or test-time).

[Zhuokai Zhao @zhuokaiz]

Want more experts without the cost of increasing, under-utilized parameters?

Check out our new paper: S’MoRE — Structural Mixture of Residual Experts for LLM fine-tuning, which extends the concept of routing flexibility into structural flexibility.

How?

💡 Instead of selecting an expert index, S’MoRE assembles a custom sub-tree of experts per token, enabling diverse, non-isomorphic expert compositions from the same expert pool.

📄 Read the paper: https://t.co/c4nonGXHlK

#MixtureOfExperts #LLM #SparseModels #AIResearch #DeepLearning #arXiv #MachineLearning

[Hanqing Zeng @zimplex4]

Our paper “GraphSAINT: Graph Sampling Based Inductive Learning Method” has been accepted by #ICLR2020! To efficiently train deep GCNs (e.g., GraphSAGE, JK-net, GAT, etc.), we construct minibatches by graph sampling to avoid neighbor explosion. https://t.co/ZBV2PcgNMN