Neural nets don’t just forget. Sometimes, after long training, they lose the ability to learn at all.
In our #ICLR2026 poster, we model Loss of Plasticity as gradient dynamics trapped in invariant manifolds: 🔴 frozen units, 🔵 cloned units.
The video makes the traps visible.
Speedrunning GPT-2 is now routine thanks to @karpathy.
But can we speedrun GPT3-175B?
We attempted to match accuracy on a <$10K budget; while we didn't quite reach it, our first results show that quality data, engineering, and native FP4 can get close.
Details in 🧵
A thorough reproduction and evaluation of $\mu$P
Georgios Vlassis, David Belius, Volodymyr Fomichov.
Action editor: Anastasios Kyrillidis.
https://t.co/kJ3egckB6u
#hyperparameters#parameters#yang2021tuning
@JamesWhate89993@_arohan_ However, when you use Shampoo, you never actually use (1), or (2). And, in practice, the behavior that you get is very different from Muon, both in terms of loss, as well as in terms of error propagation behavior (eg see my figure above)
@JamesWhate89993@_arohan_ E.g. There are two conditions under which Shampoo is exactly the same as Muon. 1) If you assume the one sided version with β2=0, 2) If you use the two sided version with an exponent of 1/4 instead of 1/2.
@JamesWhate89993 Which makes sense if you realize that networks trained with different optimizers might propagate noise differently. A nice visualization of this is Figure 3 (X might compress this).
@JamesWhate89993 Nevertheless, to me, the most interesting observation is that the Max-to-median ratio of the activations, which is used in a lot of quantization studies, is a bad predictor of quantization performance when you use different optimizers.
@evaninwords@HessianFree Nevertheless, we are meeting Omead later today to see which newer version of PSGD we should try. If you have any input/feedback/suggestions we would be more than happy to hear it too :).
@evaninwords@HessianFree ii) Most of the quantization error actually comes from activation quantization, not weight quantization. In our paper, we find that the spectral norm of the weights (which we link to quantization error propagation) barely changes after INT4 weight quantization for all optimizers.
@evaninwords IMO the logical next step would be to design an optimizer/architecture/method that explicitly takes that into account. If you go through the maths in section 3.2, you will see that the quantity of interest is the "gain".
@evaninwords Hello Evan!
Glad you like the idea! I completely agree that the most interesting finding is that the quantization error propagation profiles are different.
@HessianFree @AshkboosSaleh From what I saw, it just uses most of the code of Evan Walters (https://t.co/a34iAhRuan), which is in turn based on Xi-Lin's original PSGD repo.
@harold_matmul @AshkboosSaleh In the meantime, if you want to play with some code/train some networks yourself, you can use this: https://t.co/RkWYIUtAOc (this is what we used for the full-precision training)
@harold_matmul @AshkboosSaleh What I can tell you about the distributions of the weights, is that they indeed differ a lot from optimizer to optimizer (e.g. the Max-to-Median ratio is very different). We will maybe explore this in a future version :).