Zihang Liu

Work on ML. ICSI, UC Berkeley

[Tianyu Pang @TianyuPang327]

(1/6) Happy to share our ICML 2026 paper:
Balancing Learning Rates Across Layers: Exact Two-Step Dynamics and Optimal Scaling in Linear Neural Networks
Paper: https://t.co/fTtmyrnqmu

How should learning rates across layers evolve during training?

Our answer: training can undergo a transition from asymmetry to balance, as cross-layer feature learning make balanced learning increasingly important over time.

[Tianyu Pang @TianyuPang327]

🎉 Excited to share our recent Findings of ACL 2026 paper, HTMuon!

Muon has recently shown promising results in LLM training. But can we further improve its update rule? In our new work, we study Muon from the perspective of Heavy-Tailed Self-Regularization (HT-SR) theory and introduce HTMuon, a simple yet effective spectral correction for Muon.

Our key contributions are:

1. Understanding a limitation of Muon. Muon’s orthogonalized update rule can over-emphasize noise-dominated directions and suppress the emergence of heavy-tailed eigenspectral distributions in the model’s weight matrices, potentially limiting performance under HT-SR theory.

2. Introducing HTMuon. While Muon uses the orthogonalized update UV^T, HTMuon considers the more general form U\Sigma^pV^T, introducing a spectral correction. This enables HTMuon to produce heavier-tailed updates while preserving Muon’s strength in capturing parameter interdependencies. Across LLM pretraining and image classification, HTMuon consistently improves over Muon and other strong optimizers. It can also be used as a plug-in correction for existing Muon variants. For example, HTMuon reduces perplexity by up to 0.98 over Muon in LLaMA pretraining on C4. We further develop accelerated implementations and demonstrate improvements over Muon on LLaMA-1B.

3. Providing a theoretical characterization. We show that HTMuon is equivalent to steepest descent under a Schatten-q norm constraint and provide a convergence analysis in smooth non-convex settings. The results show that HTMuon retains competitive convergence guarantees while improving practical training performance.

📄 Paper: https://t.co/7yqov5p3jP
💻 Code: https://t.co/iWVtOBspcS

Many thanks to my collaborators Yujie Fang, @HenryLiu0820, @DengShenyang24, @twweeb , Shuhua Yu and @nsfzyzz !

[Zihang Liu @zhliu02]

@tianylin But it’s interesting to compare RMNP with Adam. In some way RMNP is a more fine-grained Adam mini (under the same ema setting). It would be more straightforward to do an lr sweep and compare the curves.

[Zihang Liu @zhliu02]

@tianylin Even though diag(VV^T) cannot exactly recover VV^T, using the diagonal to approximate exact preconditioner is still nontrivial and widely used. For example Adam itself is a diagonal preconditioner, and also Jacobi preconditioner for spd systems.

[Zihang Liu @zhliu02]

@tianylin In the end it’s about precision-computation tradeoff, more accurate preconditioner needs more numerical operations, and row-normalization is one way to balance it.

[Shenyang Deng ✈️ ICML2026 @DengShenyang24]

1/n Please stop by👋. This is not just another ICML 2026 optimizer paper. We have rich intuition to share on why simple preconditioners like orthogonalization and row-normalization specifically benefit NNs optimization. Quick overview below 🧵

[Zihang Liu @zhliu02]

@yule_gan Very cool insights from the weight space! It also reminds me of merging the checkpoints of the model at final steps of pre-training. The base model arrives at a basin with a lot of expert models

[Shenyang Deng ✈️ ICML2026 @DengShenyang24]

It‘s an honor to receive the Best Student Paper Award at #ALT2026 (37th Algorithmic Learning Theory) ! 🏆

Huge thanks to my amazing collaborators Boyao，@Collapsar0000 ，@Tianyu0628 ，@MinhakSong ，@nsfzyzz ！

Had a great time at the Fields Institute in Toronto. 🇨🇦 Looking forward to attending ALT again next time! ✨

[Zihang Liu @zhliu02]

Our work provides insights for determining the critical components of model weights, and inspires future research on the sparse and low-rank structures in LLM reasoning. [5/5] Link: https://t.co/taBR9YuOZU

[Zihang Liu @zhliu02]

Excited to share our #ICML2025 paper "LIFT the Veil for the Truth: Principal Weights Emerge After Rank Reduction for Reasoning-focused Supervised Fine-tuning"

The paper will be presented on Tue Jul 15, 11 am - 1:30 pm, East Exhibition Hall A-B, No. 2604. Come check it out! [1/5]

[Zihang Liu @zhliu02]

Based on this insight, we present LIFT, a sparse optimizer that fine-tunes Principal Weights. LIFT achieves wide success on LLM reasoning tasks. By “lifting the veil” with low-rank approximation, LIFT finds the “truth” within model weights that are critical to fine-tuning. [4/5]

[Lucerne Festival @LucerneFestival]

In Memoriam Claudio Abbado (1933 – 2014)
On the tenth anniversary of his passing, we remember the great Italian conductor Claudio Abbado. His unparalleled contributions to musical life and to our Festival in particular never cease to fill us with admiration and gratitude.

[Jan Leike @janleike]

Super excited about our new research direction for aligning smarter-than-human AI:

We finetune large models to generalize from weak supervision—using small models instead of humans as weak supervisors.

Check out our new paper:
https://t.co/8zxKf5LcAL

[Yaoqing Yang @yaoqing_yang]

NeurIPS 2023 is around the corner, and I feel excited to introduce our spotlight paper, “Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Training.”
https://t.co/GubYhNznS0

This is a long story, so please bear with me. 👇👇👇

[Google DeepMind @GoogleDeepMind]

We’re excited to announce 𝗚𝗲𝗺𝗶𝗻𝗶: @Google’s largest and most capable AI model.

Built to be natively multimodal, it can understand and operate across text, code, audio, image and video - and achieves state-of-the-art performance across many tasks. 🧵 https://t.co/mwHZTDTBuG

[Stat.ML Papers @StatMLPapers]

Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Models. (arXiv:2311.00871v1 [cs.LG]) https://t.co/9OxFRKz773

[Konstantin Mishchenko @konstmish]

Why do we need warm-up, cosine annealing, and other learning rate schedules when training with gradient descent? It turns out it's all about how gradient norms change over time. E.g., large norms at the start => warm-up. Slow decrease => cosine.
Paper: https://t.co/2RggKkvUnm
0/4