Top Tweets for #HTMuon
Introducing #HTMuon, a simple spectral correction to Muon inspired by Heavy-Tailed Self-Regularization, improving performance across LLM pretraining and image classification while retaining strong theoretical guarantees.
#Muon #Optimization
🎉 Excited to share our recent Findings of ACL 2026 paper, HTMuon!
Muon has recently shown promising results in LLM training. But can we further improve its update rule? In our new work, we study Muon from the perspective of Heavy-Tailed Self-Regularization (HT-SR) theory and introduce HTMuon, a simple yet effective spectral correction for Muon.
Our key contributions are:
1. Understanding a limitation of Muon. Muon’s orthogonalized update rule can over-emphasize noise-dominated directions and suppress the emergence of heavy-tailed eigenspectral distributions in the model’s weight matrices, potentially limiting performance under HT-SR theory.
2. Introducing HTMuon. While Muon uses the orthogonalized update UV^T, HTMuon considers the more general form U\Sigma^pV^T, introducing a spectral correction. This enables HTMuon to produce heavier-tailed updates while preserving Muon’s strength in capturing parameter interdependencies. Across LLM pretraining and image classification, HTMuon consistently improves over Muon and other strong optimizers. It can also be used as a plug-in correction for existing Muon variants. For example, HTMuon reduces perplexity by up to 0.98 over Muon in LLaMA pretraining on C4. We further develop accelerated implementations and demonstrate improvements over Muon on LLaMA-1B.
3. Providing a theoretical characterization. We show that HTMuon is equivalent to steepest descent under a Schatten-q norm constraint and provide a convergence analysis in smooth non-convex settings. The results show that HTMuon retains competitive convergence guarantees while improving practical training performance.
📄 Paper: https://t.co/7yqov5p3jP
💻 Code: https://t.co/iWVtOBspcS
Many thanks to my collaborators Yujie Fang, @HenryLiu0820, @DengShenyang24, @twweeb , Shuhua Yu and @nsfzyzz !

Last Seen Hashtags on Sotwe
Trends for you
Most Popular Users

Elon Musk 
@elonmusk
240.7M followers

Barack Obama 
@barackobama
119.2M followers

Donald J. Trump 
@realdonaldtrump
111.7M followers

Cristiano Ronaldo 
@cristiano
110.8M followers

Narendra Modi 
@narendramodi
107M followers

Rihanna 
@rihanna
97.7M followers

NASA 
@nasa
92.2M followers

Justin Bieber 
@justinbieber
90.9M followers

KATY PERRY 
@katyperry
87.8M followers

Taylor Swift 
@taylorswift13
81.6M followers

Lady Gaga 
@ladygaga
73.2M followers

Virat Kohli 
@imvkohli
70.1M followers

Kim Kardashian 
@kimkardashian
69.8M followers

YouTube 
@youtube
68.7M followers

Bill Gates 
@billgates
64M followers

Neymar Jr 
@neymarjr
62.8M followers

The Ellen Show
@theellenshow
62.4M followers

CNN 
@cnn
61.9M followers

Selena Gomez 
@selenagomez
60.9M followers

X 
@x
60.8M followers

