1/ Let me chip in on the recent “which optimizer rules them all” discussion with a somewhat more moderate take, asking:
What Schatten-p norm to use?
Turns out the answer is regime dependent! Specifically, even when smooth in Schatten-∞, Muon is not necessarily the best choice.
@kothasuhas Will also take a look - thanks for the precise pointer! By equilibrium you mean the perspective that eg AdamC also takes arriving at lr^2 for wd? What I find curious is that convex+smooth with noise immediately gives the same scaling as MAI
@xidulu@laurence_ai@ShaneBergsma This looks exactly like the O(1/steps) schedule we theoretically arrive at with SODA https://t.co/f25xr2KEMX
I should check out your papers!
@kalomaze It might be interesting for you to look at https://t.co/f25xr2LcCv where we weight decay towards the initialization - we'll soon put out the updated accepted version which applies this idea to fine-tuning (where init=pre-trained model)
@ruuustem_10@tonysilveti Ah I see, it’s the muKL regardless of normalization that does it, e.g., the O(k^-1/3) achieved here for SGD https://t.co/vKPruFPC7c
@ruuustem_10@tonysilveti You mean in their experiment and without normalization/clipping? AdamW would be ~consistent with your theory, but SGDM does not fit into FW so how do we explain?
@stochasticchasm It might be interesting for you to check out angular muown and magnitude-direction decoupling that came out concurrently
https://t.co/0rXE5GCpiK
https://t.co/tedUHWtOCw
@SmartPig_Joe There’s a simple argument for why aggressive wd/normalization hurts for the last layer (if one does not compensate by introducing a learnable scalar) https://t.co/Rui3D1KEIY
It is common to exclude the unembedding layer from using weight decay, but why? Essentially, applying weight decay to the last layer will constrain the logits of the model and consequently prevent high-confident output. 👇
https://t.co/WZ6bIilUH3
1/ Let me chip in on the recent “which optimizer rules them all” discussion with a somewhat more moderate take, asking:
What Schatten-p norm to use?
Turns out the answer is regime dependent! Specifically, even when smooth in Schatten-∞, Muon is not necessarily the best choice.
1/ Let me chip in on the recent “which optimizer rules them all” discussion with a somewhat more moderate take, asking:
What Schatten-p norm to use?
Turns out the answer is regime dependent! Specifically, even when smooth in Schatten-∞, Muon is not necessarily the best choice.
@TedNoNumbers@Creative_Math_ Yes, I think there is something interesting to be said here: take a look at @leloykun's blogpost https://t.co/r2D5RhzhWw "So, in a sense, you can interpret Muon with the old coefficients as doing steepest descent under the Schatten-32 norm"
1/ Let me chip in on the recent “which optimizer rules them all” discussion with a somewhat more moderate take, asking:
What Schatten-p norm to use?
Turns out the answer is regime dependent! Specifically, even when smooth in Schatten-∞, Muon is not necessarily the best choice.
@TedNoNumbers@Creative_Math_ Yes, I think there is something interesting to be said here: take a look at @leloykun's blogpost https://t.co/r2D5RhzhWw "So, in a sense, you can interpret Muon with the old coefficients as doing steepest descent under the Schatten-32 norm"