Great to see our Spectral Scaling Laws work discussed from the perspective of FFN/MoE design!
This work was started out of curiosity, to know how wide should an FFN be, and how do we know whether that width is actually being utilized in LLMs?
And it was based on the fact that, FFN expansion ratios vary substantially across LLM families -- 2.67x (LLaMA), 4x (GPT-2, PaLM), 8x (Gemma-2), 64x (T5-11B). However, the nominal width does not necessarily tell us the dimensionality actually realized during pre-training.
More recently, we further investigate if the observed asymmetric spectral scaling is intrinsic to the transformer architecture, or also depends on which pre-training optimizer is used? For this we use Muon, NorMuon, Dion, along with AdamW variants. (In our EMNLP 2025 paper, we had used only AdamW)
We find that changing the optimizer alone can substantially change the asymmetry between soft- and hard-spectral scaling, and even models with the same architecture and similar validation loss can realize very different spectral scaling exponents.
Project: https://t.co/oHPKQgpuaA
paper reading thread!
i've been exploring spectral scaling laws for FFNs and MOEs recently and hence my thoughts on one of the papers that i found really interesting in the way the authors recast width selection as a spectral utilization optimization problem. (1/n)
"Deep Learning with Differential Privacy" is one of the ACM CCS 2026 test of time awards this year. Congratulations to my co-authors, and also to fellow adversarial example researchers
@mahmoods01 et al for "Accessorize to a crime"
A lot of the most successful people I know spent years of their life just bumming around, before they suddenly exploded into action and did something great.
I don't think that's a coincidence.
https://t.co/KT4vsg5txg
Today we release Ripple-65B Preview, our 65B dense LLM we pretrained from scratch.
Ripple sets a new state-of-the-art for U.S. open-weight models in agentic coding tasks, surpassing significantly larger models including Muse Spark Open-816B, Beam-501B, Inkling-975B, and Nemotron 3 Ultra-550B.
Learn more: https://t.co/Pe04ZKhkE9
We've posted a 156-page tech report on Beam! We have good systems and it's worth reading. It covers all phases through pre-training, mid-training, SFT, RL, and OPD and goes into the infra, data, and scientific decisions we made throughout the process
📚 https://t.co/UPwkbAnlb3
📖 The Ultimate MoE Training Handbook 📖
We spent 1/2M GPUh to optimize MoE training and wrote the most comprehensive blog about it. With 30k words and 50+ interactive figures it covers everything from profiling to large cluster training.
https://t.co/mk6fGoCiqV
Introducing Beam: a highly efficient agentic open model with 501B total parameters and 23B active.
- Frontier reasoning efficiency
- Advances the Western open frontier on coding & agentic tasks
- Trained end-to-end from scratch
Full weights release this month.
Learn more about Beam: https://t.co/c3Qx2cpM8G
One of the worst forms of brainrot AI has cultivated is pessimism about basic research, ie the idea that important work can only happen inside a frontier/neo lab and only with 10k+ GPUs, so the rest should not even bother. What a bleak way to think about science. And it's false.
We are having our first @SimonsFdn yearly meeting on the Physics of Learning and Neural Computation Sept 24th-25th and all are welcome to join on this zoom:
https://t.co/nj0Aex9af0
We have an excellent set of speakers. Here is the agenda:
https://t.co/zpvY99bWMn
Join our webinar starting Sept 24th 9:30am ET to learn how the structure of data, architectures, and learning dynamics conspire to empower AI to learn, scale, compute, imagine, reason and collaborate!
See https://t.co/TfOKlQxCrE for more about our collaboration.
🚢 Marin 535B-A23B started training this week! As usual, the whole process is open.
Voyage plan: pretraining (80%) + midtraining (20%) on 18.75T tokens on 11 x GB200 NVL72 for ~3 months (2.7e24 FLOPs). Post-training will follow.
Before kicking off the run, we trained a 4-rung scaling ladder from 1.6B-A61M (48B tokens) to 27.7B-A1.2B (926B tokens) to debug issues, and to make a forecast of our hero run. This is by far our biggest run, so definitely expecting the unexpected.
Also wrote a (bit longer) blog post to unpack the main results, and describing our intuition. We've also included a discussion on the implications of realized-spectral capacity for future-learning and adaption, mainly plasticity-stability trade-off.
Blog: https://t.co/pReJfMgtsu
Great to see our Spectral Scaling Laws work discussed from the perspective of FFN/MoE design!
This work was started out of curiosity, to know how wide should an FFN be, and how do we know whether that width is actually being utilized in LLMs?
And it was based on the fact that, FFN expansion ratios vary substantially across LLM families -- 2.67x (LLaMA), 4x (GPT-2, PaLM), 8x (Gemma-2), 64x (T5-11B). However, the nominal width does not necessarily tell us the dimensionality actually realized during pre-training.
More recently, we further investigate if the observed asymmetric spectral scaling is intrinsic to the transformer architecture, or also depends on which pre-training optimizer is used? For this we use Muon, NorMuon, Dion, along with AdamW variants. (In our EMNLP 2025 paper, we had used only AdamW)
We find that changing the optimizer alone can substantially change the asymmetry between soft- and hard-spectral scaling, and even models with the same architecture and similar validation loss can realize very different spectral scaling exponents.
Project: https://t.co/oHPKQgpuaA
paper reading thread!
i've been exploring spectral scaling laws for FFNs and MOEs recently and hence my thoughts on one of the papers that i found really interesting in the way the authors recast width selection as a spectral utilization optimization problem. (1/n)
The architectural interaction goes beyond RoPE, and in fact the most-interesting part of this work.
We compare scaling effects of architectural interventions vs optimizer selection across token-regimes. Architectural intervention is most competitive on frequent (HEAD) tokens, whereas optimizer choice dominates rare-token (TAIL) scaling. The same intervention shifts β by 0.65 under Muon vs 0.04 under Dion.
We've unpacked the results and discuss more on forward-looking implications (e.g., optimizer's role in preserving plasticity for continual learning) in the blog: https://t.co/Ajme2Vddqt
Author here, thanks for reading & posting!
On behavioral analysis: we think the connection to the tail associative-memory result is plausible, they observed Muon storing tail facts better; our work present a representational counterpart, TAIL hard-rank scaling exponent β=1.02 for Muon vs 0.44 for AdamW with same architecture and training-data.
Behavioral changes at matched loss is a natural extension to this work, and (hopefully!) will do in coming iteration. Since perplexity is dominated by frequent tokens, TAIL-token capacity scaling is hidden by loss; hence, behaviorally the matched loss effects would be more distinguishing for rare-token recall, long-tail knowledge, etc.
https://t.co/fDryGaQQwP
Scaling of effective rank of post activation over FFN hidden dimensions, per token frequency, is quite different for Adam and Muon even when loss is matched. And this interacts with architectural choices like RoPE. Maybe related to tail knowledge behavior of Muon? (https://t.co/0lW3HAFlw3) How would this lead to behavioral changes when loss is matched?
Our takeaway: realized capacity emerges from the architecture-optimizer interaction, not from architecture alone.
Limitations and open questions (e.g., behavioral validation) are discussed in the paper and blog.
📄 Paper: https://t.co/RddtOEeuHT
📝 Blog: https://t.co/Ajme2Vddqt
The optimizer-architecture interaction shows a distinctive pattern across token frequency.
For frequent (HEAD) tokens, architectural intervention is most effective. For rare (TAIL) tokens, optimizer choice dominates hard-rank scaling.
Our interpretation: where data is dense, architectural solutions are more-effective; where data is sparse, the optimizer's implicit bias matters more.
We also compared optimizer effects against a controlled architectural intervention (12 → 6 attention heads, fixed parameter count).
In 28 of 30 regime–optimizer comparisons, switching the optimizer shifted spectral scaling more than the intervention did.
Same intervention shifts β by 0.65 under Muon and 0.04 under Dion. That is, architectural effects are not optimizer-independent.
Orthonormalized update rank acts as a knob for controlling spectral capacity at training.
Varying Dion's update rank from 1/2 to 1/16, lowers TAIL hard-rank scaling from β=0.88 to 0.40. The rank of the update itself appears to determine how much added width becomes utilized capacity.
This difference is not visible in validation loss.
Under 2× longer training, AdamW matches the perplexity of a low-rank Dion optimizer, while its hard-rank scaling nearly vanishes (β: 0.29 → 0.03).
Matched loss ≠ Matched representation
🚨 New paper!
Optimizer-Induced Spectral Scaling Laws
We investigate how the optimizer geometry modulates representation geometry and affect latent space utilization across token-regimes.
Holding architecture and training-data fixed, changing optimizer changes the scaling exponents that govern how FFN width becomes usable capacity.
On rare-token (TAIL) representations, hard-rank scaling is β=0.44 under AdamW vs β=1.02 under Muon.
A thread 🧵👇