Nandan Kumar Jha

PhD @nyuniversity | LLM pre-training, representation geometry & optimization | Spectral scaling laws | Prev: @IITHyderabad @Seagate

[Nandan Kumar Jha @TheRegularizer]

Great to see our Spectral Scaling Laws work discussed from the perspective of FFN/MoE design!

This work was started out of curiosity, to know how wide should an FFN be, and how do we know whether that width is actually being utilized in LLMs?

And it was based on the fact that, FFN expansion ratios vary substantially across LLM families -- 2.67x (LLaMA), 4x (GPT-2, PaLM), 8x (Gemma-2), 64x (T5-11B). However, the nominal width does not necessarily tell us the dimensionality actually realized during pre-training.

More recently, we further investigate if the observed asymmetric spectral scaling is intrinsic to the transformer architecture, or also depends on which pre-training optimizer is used? For this we use Muon, NorMuon, Dion, along with AdamW variants. (In our EMNLP 2025 paper, we had used only AdamW)

We find that changing the optimizer alone can substantially change the asymmetry between soft- and hard-spectral scaling, and even models with the same architecture and similar validation loss can realize very different spectral scaling exponents.

Project: https://t.co/oHPKQgpuaA

[Andy @prompt_Tunes]

paper reading thread!
i've been exploring spectral scaling laws for FFNs and MOEs recently and hence my thoughts on one of the papers that i found really interesting in the way the authors recast width selection as a spectral utilization optimization problem. (1/n)

[Percy Liang @percyliang]

🚢 Marin 535B-A23B started training this week! As usual, the whole process is open.
Voyage plan: pretraining (80%) + midtraining (20%) on 18.75T tokens on 11 x GB200 NVL72 for ~3 months (2.7e24 FLOPs). Post-training will follow.
Before kicking off the run, we trained a 4-rung scaling ladder from 1.6B-A61M (48B tokens) to 27.7B-A1.2B (926B tokens) to debug issues, and to make a forecast of our hero run. This is by far our biggest run, so definitely expecting the unexpected.

[Nandan Kumar Jha @TheRegularizer]

Also wrote a (bit longer) blog post to unpack the main results, and describing our intuition. We've also included a discussion on the implications of realized-spectral capacity for future-learning and adaption, mainly plasticity-stability trade-off.

Blog: https://t.co/pReJfMgtsu

[Nandan Kumar Jha @TheRegularizer]

Great to see our Spectral Scaling Laws work discussed from the perspective of FFN/MoE design!

This work was started out of curiosity, to know how wide should an FFN be, and how do we know whether that width is actually being utilized in LLMs?

And it was based on the fact that, FFN expansion ratios vary substantially across LLM families -- 2.67x (LLaMA), 4x (GPT-2, PaLM), 8x (Gemma-2), 64x (T5-11B). However, the nominal width does not necessarily tell us the dimensionality actually realized during pre-training.

More recently, we further investigate if the observed asymmetric spectral scaling is intrinsic to the transformer architecture, or also depends on which pre-training optimizer is used? For this we use Muon, NorMuon, Dion, along with AdamW variants. (In our EMNLP 2025 paper, we had used only AdamW)

We find that changing the optimizer alone can substantially change the asymmetry between soft- and hard-spectral scaling, and even models with the same architecture and similar validation loss can realize very different spectral scaling exponents.

Project: https://t.co/oHPKQgpuaA

[Andy @prompt_Tunes]

paper reading thread!
i've been exploring spectral scaling laws for FFNs and MOEs recently and hence my thoughts on one of the papers that i found really interesting in the way the authors recast width selection as a spectral utilization optimization problem. (1/n)

[Petar Veličković @PetarV_93]

happening right now in hall a, #3702
come & chat to us!

[Nandan Kumar Jha @TheRegularizer]

The architectural interaction goes beyond RoPE, and in fact the most-interesting part of this work.

We compare scaling effects of architectural interventions vs optimizer selection across token-regimes. Architectural intervention is most competitive on frequent (HEAD) tokens, whereas optimizer choice dominates rare-token (TAIL) scaling. The same intervention shifts β by 0.65 under Muon vs 0.04 under Dion.

We've unpacked the results and discuss more on forward-looking implications (e.g., optimizer's role in preserving plasticity for continual learning) in the blog: https://t.co/Ajme2Vddqt

[Nandan Kumar Jha @TheRegularizer]

Author here, thanks for reading & posting!

On behavioral analysis: we think the connection to the tail associative-memory result is plausible, they observed Muon storing tail facts better; our work present a representational counterpart, TAIL hard-rank scaling exponent β=1.02 for Muon vs 0.44 for AdamW with same architecture and training-data.

Behavioral changes at matched loss is a natural extension to this work, and (hopefully!) will do in coming iteration. Since perplexity is dominated by frequent tokens, TAIL-token capacity scaling is hidden by loss; hence, behaviorally the matched loss effects would be more distinguishing for rare-token recall, long-tail knowledge, etc.

[Rosinality @rosinality]

https://t.co/fDryGaQQwP

Scaling of effective rank of post activation over FFN hidden dimensions, per token frequency, is quite different for Adam and Muon even when loss is matched. And this interacts with architectural choices like RoPE. Maybe related to tail knowledge behavior of Muon? (https://t.co/0lW3HAFlw3) How would this lead to behavioral changes when loss is matched?

[Nandan Kumar Jha @TheRegularizer]

Our takeaway: realized capacity emerges from the architecture-optimizer interaction, not from architecture alone.

Limitations and open questions (e.g., behavioral validation) are discussed in the paper and blog.

📄 Paper: https://t.co/RddtOEeuHT
📝 Blog: https://t.co/Ajme2Vddqt

[Nandan Kumar Jha @TheRegularizer]

The optimizer-architecture interaction shows a distinctive pattern across token frequency.

For frequent (HEAD) tokens, architectural intervention is most effective. For rare (TAIL) tokens, optimizer choice dominates hard-rank scaling.

Our interpretation: where data is dense, architectural solutions are more-effective; where data is sparse, the optimizer's implicit bias matters more.

[Nandan Kumar Jha @TheRegularizer]

We also compared optimizer effects against a controlled architectural intervention (12 → 6 attention heads, fixed parameter count).

In 28 of 30 regime–optimizer comparisons, switching the optimizer shifted spectral scaling more than the intervention did.

Same intervention shifts β by 0.65 under Muon and 0.04 under Dion. That is, architectural effects are not optimizer-independent.

[Nandan Kumar Jha @TheRegularizer]

Orthonormalized update rank acts as a knob for controlling spectral capacity at training.

Varying Dion's update rank from 1/2 to 1/16, lowers TAIL hard-rank scaling from β=0.88 to 0.40. The rank of the update itself appears to determine how much added width becomes utilized capacity.

[Nandan Kumar Jha @TheRegularizer]

This difference is not visible in validation loss.

Under 2× longer training, AdamW matches the perplexity of a low-rank Dion optimizer, while its hard-rank scaling nearly vanishes (β: 0.29 → 0.03).

Matched loss ≠ Matched representation

[Nandan Kumar Jha @TheRegularizer]

🚨 New paper!

Optimizer-Induced Spectral Scaling Laws

We investigate how the optimizer geometry modulates representation geometry and affect latent space utilization across token-regimes.

Holding architecture and training-data fixed, changing optimizer changes the scaling exponents that govern how FFN width becomes usable capacity.

On rare-token (TAIL) representations, hard-rank scaling is β=0.44 under AdamW vs β=1.02 under Muon.

A thread 🧵👇

[Google Research @GoogleResearch]

Congratulations to the @GoogleDeepMind authors of "Asynchronous Methods for Deep Reinforcement Learning", recipient of the #ICML2026 Test of Time Award. This work shows that asynchronous actor-critic succeeds on a wide variety of continuous motor control problems as well as on a new task of navigating random 3D mazes using a visual input.

[Ananyo Bhattacharya @Ananyo]

Our @Nature comment this week on the use of AI in maths and theoretical physics - and why the community should embrace it!
Authors @London_Inst & @GoogleDeepMind.
First draft 8 months ago but edited many times as the field steamed ahead!
Free-to-read link at the end of 🧵1/

[OpenAI @OpenAI]

In conversation with OpenAI’s @markchen90, Terence reflects on a future where AI reduces the cognitive friction of research, helps preserve the paths behind discovery, and expands what mathematicians and scientists can attempt.
https://t.co/0elJITP8XT

[François Chollet @fchollet]

There are only two honest metrics when it comes to benchmarking intelligence: novelty and efficiency.

You don't need intelligence to solve a known problem (only memory). And you don't need intelligence to solve a problem via brute force. But to solve a novel problem efficiently, intelligence is the only way.

[Brandon Luu, MD @BrandonLuuMD]

Literally just having a delusional golden retriever mindset measurably changes outcomes and physiology.

Sleep badly? Convince yourself you're well rested.
Stressful day? Convince yourself it's fuel.
Failed? Convince yourself it's useful data.

[Han Xiao @hxiao]

Katherine Bouman talking about blackhole while Yann Lecun walking into the room!

[ICLR @iclr_conf]

Day 1 afternoon keynote talk given by Max Welling @wellingmax #ICLR2026

From Physics to AI to Materials; A Journey from Foundations to Impact

"Do we reward strange new, potentially paradigm-shifting ideas or do we focus on engineering, scaling and bold numbers?"