Jianliang He

PhD Student @ Yale S&DS. Undergrad @ Fudan.

[Fengzhuo Zhang @FengzhuoZhang]

How do optimizers, architectures, and data shape LLM generalization differently?

🚀 We study this through Relative Generalization Invariance (RGI): two models satisfy RGI if their token-wise losses differ by a constant gap.

1. RGI largely holds across optimizers and moderate architecture changes—an approximately uniform shift in token-wise loss.

2. Changing the training data stream breaks RGI.

Data, optimization, and architecture leave distinct signatures on generalization.

Paper Link: https://t.co/sKZY38EqxX

A thread 🧵

[Siyu Chen @siyuc3141]

How do we get LLMs to self-improve using unstructured documents without relying on human curation or naive "difficulty" heuristics?

Excited to share INFUSER, our latest work that turns self-evolution into influence-guided curriculum learning! 🚀 https://t.co/weDJ9Fjrbw

[Zhuoran Yang @zhuoran_yang]

Followup Paper -- "Neural Networks Provably Learn Group Representations: From Cyclic to Non-Abelian"

Our prior work (https://t.co/uCU4eNr7tH) showed that NNs trained on modular addition learn **sinusoidal features** with precise **phase alignment**. But why sinusoidal? Why those phases?

Because sine and cosine are the **irreducible representations** of the cyclic group Z_p. The network is discovering the representation theory of the group it computes on.

This is not a coincidence. In our new paper, we show that all of these findings can be generalized to Abelian groups, and there is some surprise for non-Abelian groups.

In particular, we study how NNs learn the group composition task: given two group elements g1 and g2, predict g1*g2. The key insight is that NN learns to leverage **group fourier features** to solve this task.

arxiv: https://t.co/WdNIuC9nPj
blog: https://t.co/MUfWOnSICj
code: https://t.co/XUp37Fhjqc

@JLiangHe @LedaW77625 @FengzhuoZhang @siyuc3141

[Fengzhuo Zhang @FengzhuoZhang]

Why do DeepSeek and Kimi use Muon instead of Adam?

🚀 Reasons from a curvature perspective:

1⃣ Under a second-order approx., Muon incurs a much smaller curvature penalty than Adam while maintaining the same first-order decrease.

2⃣ This advantage does not come from a smaller update norm. Instead, it comes from Muon having lower Normalized Directional Sharpness (NDS).

3⃣ Muon’s NDS advantage becomes larger when the training data is more imbalanced.

Paper Link: https://t.co/nGy2FeF538

A thread 🧵

[Zhuoran Yang @zhuoran_yang]

New Paper -- "On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokking"

We give a complete mechanistic and dynamic picture of how neural networks learn modular addition f(x,y) = (x+y) mod p. We answer three questions:
(1) What does the trained network compute?
(2) How do Fourier features emerge during training?
(3) Why does grokking happen?
Each answer comes with a mathematical characterization backed by theory and experiments.

Paper: https://t.co/uCU4eNr7tH
Blog: https://t.co/Ok9dDQcYvw
Demo: https://t.co/SX4Q6IsLHB
Code: https://t.co/03Adj6E3Bt

[Zhuoran Yang @zhuoran_yang]

[New paper on in-context learning] "In-Context Linear Regression Demystified" (link: https://t.co/7Z1ubhXQQB). Joint work @JLiangHe, @xintianpan, @siyuc3141. We establish a rather complete understanding of how one-layer multi-head attention solves in-context linear regression,