jianlin.su

Grad&Clip&EM is all you need @Kimi_Moonshot
Blog: , Cool Papers: link:

[jianlin.su @Jianlin_S]

A Perturbation Analysis Approach to Equality-Constrained Optimization
https://t.co/RWkcXdvyCm

[jianlin.su @Jianlin_S]

Slow is Fast: Is Muon the end of Shampoo?
https://t.co/9zo5vGKbcH

Orthogonal Slowest Descent — from a convex optimization convergence bound — washes away any Kronecker preconditioner's eigenvectors, leaving only eigenvalues to balance the gradient's spectrum — leads to Muon.

[jianlin.su @Jianlin_S]

@alessio__serra I see what you mean. Actually, even in fp32, there are some acceleration tricks for computing exact quantiles, but they make the implementation more complex. So my main point is that histogram approximation is a trade-off between being 'good enough' and 'simple.'

[jianlin.su @Jianlin_S]

@alessio__serra Furthermore, EQB seems to assume that the router logits are in bf16, which is often not the case in practice.

[jianlin.su @Jianlin_S]

@alessio__serra Good job! However, for the histogram approximation, we compared 1000 bins with 10000 bins and found that the latter brings no gains — so the 1000-bin histogram approximation is indeed sufficient, which is a comparison EQB did not perform.

[jianlin.su @Jianlin_S]

The Rearrangement Inequality and Its Generalizations https://t.co/vkIrEV31kP

Starting from the rearrangement inequality, this article follows two lines of development — "matricization" and "multi-sequence extension" — to survey its principal generalizations.

[jianlin.su @Jianlin_S]

Revisiting Convergence Results in Convex Optimization (X)
https://t.co/9A85wAhUv1

drop the monotonicity assumption: dynamic learning rates get average- and last-iterate convergence (why LR decay works), and non-monotone preconditioners get minimal fixes with guarantee

[jianlin.su @Jianlin_S]

Revisiting Convergence Results in Convex Optimization (IX)
https://t.co/bmxSZwxbX4

Revisits AdaGrad, the seminal work on adaptive gradient algorithms, reproducing its full derivation via "convergence analysis → minimizing the upper bound → optimal preconditioning matrix."

[jianlin.su @Jianlin_S]

Revisiting Convergence Results in Convex Optimization (VIII)
https://t.co/z3p7R4Q2I9

This article re-examines “schedule-free learning rates” from the perspective of multi-stage training, shifting the scheduling objective to “approaching optimality by the end of each stage” .

[jianlin.su @Jianlin_S]

A New Understanding of Momentum: Toward Feature-Level Gradient Descent
https://t.co/f8qNhnpGmp

[jianlin.su @Jianlin_S]

Steepest Descent on Manifolds: 7. The Close-form Solution of Stiefel
https://t.co/aDFj69vlu3

[jianlin.su @Jianlin_S]

@JingyuanLiu123 signsgd式的loss-free，主要是个别层压不住。

[jianlin.su @Jianlin_S]

@JingyuanLiu123 因为attnres最后的输出加了个rmsnorm，所以softmax的分母其实被消去了，结果等价于exp(q*k)加权求和，换言之分母是L1还是L2都不影响结果

[jianlin.su @Jianlin_S]

@MayankMish98 @hsu_byron Most f-divergences cannot be used as LM loss, as the blog has already illustrated with the example of Total Variation.

[jianlin.su @Jianlin_S]

What Loss Functions Can LMs Use Besides Cross-Entropy?
https://t.co/K56sJ8cjkB

[jianlin.su @Jianlin_S]

A Brief Look at K3's MoE and Attention
https://t.co/LIF6OnT7wV

[jianlin.su @Jianlin_S]

@latentumx 一般比较适合pretrain，但推导上其实是没区分的，所以原则post-train也可以用，但可能各种系数、指数要重新搜

[jianlin.su @Jianlin_S]

Deconstructing Scaling Laws: The Triad of Optimization, Architecture, and Data
https://t.co/jbqiFXsIJN

[jianlin.su @Jianlin_S]

@tonysilveti I've read this paper too—it finds that optimal batch size scales as data size^(2/3). I haven't fully digested it yet, but I'll study it properly when I get the chance.

[jianlin.su @Jianlin_S]

@cedric_chee Thanks for your interest. Whether KDA+MLA uses RoPE is purely a matter of following the experimental results, and has nothing to do with whether I'm at Kimi or not. Just as we continue to use MLA, this too is a result of respecting what the experiments tell us.