Kwangjun Ahn

Researcher at NVIDIA // ex-Researcher at Microsoft, PhD from MIT EECS

[Kwangjun Ahn @KwangjunA]

@JohnCLangford Updates on the Dion codebase (https://t.co/jVz7Fxv6B1), please check them out!
- Dion2 (https://t.co/VtAoq2eH01), which has much simpler math than Dion.
- NorMuon (https://t.co/ZHaNLWQO2y) thanks to @li_zichong.

[Seunghyun Seo @SeunghyunSEO7]

sacled up to 12.7B dense, 5.5T tokens.
- polynorm (optimized kernel)
- grouped diff attn (their work)
- parallel muonclip (adopt alltoall like mainhorse, essential, dion)
- 80M batch
it's still non-reasoning, also not moe though... keep pushing guys!
https://t.co/XheiAS8TBO

[Kwangjun Ahn @KwangjunA]

@ionutmodo Thanks! Looks great, let me read through it

[Kwangjun Ahn @KwangjunA]

New improvement in Dion leads to a speedup that makes orthonormal updates (eg. Muon) more scalable for larger matrices. The trick: carefully using Newton-Schulz (on smaller matrices) as Dion's backend. Updates to our microsoft/dion codebase are coming soon---stay tuned!

[Microsoft Research @MSFTResearch]

Join us on Sept 24 at 8 AM PT for Microsoft Research Forum Season 2 – a virtual series highlighting purposeful research and its real-world impact, from fundamental exploration to advancing AI responsibly, scaling innovation through products and open source, and driving positive change for society. Register now: https://t.co/eWh5h1NZ7N

[Andrej Karpathy @karpathy]

@jxbz love the repo! clean code, good practices but still not overly over-engineered, triton kernels, well documented, simple reference implementations alongside optimized code. nice

[elie @eliebakouch]

Lot, lot of alpha here

[Kwangjun Ahn @KwangjunA]

@eliebakouch @jxbz @JohnCLangford @GagMagakyan Thank you!! Glad to hear it

[Kwangjun Ahn @KwangjunA]

@jxbz @JohnCLangford @GagMagakyan Thank you for the kind words! Hope this proves useful to the community!

[Jeremy Bernstein @jxbz]

Looks like extremely exciting and useful work by @KwangjunA, Byron Xu, Natalie Abreu, @JohnCLangford and @GagMagakyan

https://t.co/8WzCbdljDS

(2/2)

[Jeremy Bernstein @jxbz]

I had wondered why there was no official Dion implementation by the authors... I guess now we know. This repository looks dope: FSDP Muon and Dion implementations, triton kernels for Newton-Schulz, and lots of practical advice

(1/2)

[Laker Newhouse @LakerNewhouse]

[1/6] Curious about Muon, but not sure where to start? I wrote a 3-part blog series called “Understanding Muon” designed to get you up to speed—with The Matrix references, annotated source code, and thoughts on where Muon might be going.

[John Langford @JohnCLangford]

Apparently Dion is now being worked on for Torch Titan: https://t.co/QeuRFDTyan :-)

[Kwangjun Ahn @KwangjunA]

@MParakhin Thanks for advertising Dion! :)

[Mikhail Parakhin @MParakhin]

Since nobody asked :-), here is my list of papers not to be missed from ICML:
1) Dion: distributed orthonormalized updates (well, technically not at ICML, but everyone's talking about it).
2) MARS: Unleashing the Power of Variance Reduction for Training Large Models
3) ...

[Kwangjun Ahn @KwangjunA]

@orvieto_antonio @micahgoldblum @teodorasrec @jonasgeiping Nice results! One question: wouldn’t large (global-)batch size be more practical for distributed training? Does that mean still SGD is not effective for large scale?

[Kwangjun Ahn @KwangjunA]

@seungwookh @jxbz Go Jeremy and Laker!!

[Seungwook Han @seungwookh]

But actually this is the og way of doing it and should stop by E-2103 to see @jxbz and Laker Newhouse whiteboard the whole paper.

[Kwangjun Ahn @KwangjunA]

@konstmish @aaron_defazio Thanks Konstantin!

[Konstantin Mishchenko @konstmish]

Schedule-Free methods, which forgo cosine/linear schedulers by averaging iterates and computing gradients at interpolated points, yield smoother training curves. It's still unclear why they work well, and this paper explains the phenomenon through the river-valley loss landscape.