Starting a business in materials industry /ex-ML engineer @Airbnb / AI researcher by passion/ worked on time series, conformal prediction, optimization
50+ pages of prose and formulas, zero experiment, but code in the github.
Is this a slop-grenade, an unvalidated theoretical proposal, or a rushed flag-plant to stand out in 50k+ iclr submissions?
We are at the dawn of Superintelligence.
Introducing the Recurrent Looped Transformer (RLT),
We now have Transformers with Infinite Reasoning depth.
From now on, we should pace progress at the Open Frontier of Superintelligence,
Until Safe Superintelligence is achieved.
https://t.co/yMWIWU4upo
@timlautk Weird choice indeed. The use of Sinkgd will break the symmetry condition I believe. Also don’t understand their decision here, Adamw won’t be an issue if you shard your states properly, sinkgd is not necessary.
Somewhere along the way, optimization papers started acting like every result needs a looksmaxxed numbers.
The same noisy curves can be dressed up as “headline breakthrough” or “marginal improvement”, depending on what they had for breakfast and which two dots they circle that morning.
The paper is already good enough without looksmaxxed marketing; and this is not helping.
People keep asking me: what's different about optimization in RL?
Seemingly nothing — the pre-training stack just works (Adam, even SGD 👀 @saagnikkk).
Bringing some answers from my last work (sorry for the delay — been cooking 🚀).
We introduce ISO: Isospectral Optimization: an RLVR-native optimization stack.
Built on one simple observation, spectral inheritance: RLVR can reuse the base model's spectrum and acquire new behavior purely through the singular frames.
🧩 Offline: ISO-Merger — consolidates RL experts into one model with no data, no rollouts, no OPD. Checkpoints only.
⚙️ Online: ISO-Optimizer — a drop-in wrapper on AdamW / Muon that matches AdamW's accuracy with ~2.7× fewer steps on Qwen3-8B-Base.
📄 https://t.co/ty1FrAveA5
🌐 https://t.co/xzXf6V8ukO
🧵👇
I helped Prof Volkan Cevher tune it, remember the initial results for spectral descent was much worse, like way worse.
Volkan’s orginal paper from 2015 is something I unfortunately didn’t make the connection
I am not sure what the right solution is but providing generous credit help with the raising more compute in academic settings.
Fun fact: Jeremy’s first Muon research thread (https://t.co/kbtxZ4VuHF ) back in Aug 2024 exactly started from Shampoo winning AlgoPerf, then gave the key observation: “Shampoo preconditioning exactly orthogonalizes the gradient matrices.”
Then the Old Optimizer, New Norm paper derived Newton–Schulz, the key innovation of Muon, as a “computational strategy for Shampoo”.
So “Muon is not exactly Shampoo” is a dodge. Of course it isn’t byte-for-byte Shampoo. That was never the point.
The point is that Muon was born downstream of Shampoo, as a practical Shampoo-line variant, then got hype-maxxed into “the first big optimizer breakthrough after Adam.”
No. Shampoo was the breakthrough. The Shampoo line led to amazing optimizers like SOAP, which was invented before Muon, and btw still leads the current ModdedNanoGPT optimizer track record
.@kellerjordan0 posts results saying X is half way between. I educate him on optimization details, and hyper parameter tuning, and literally changes his code. I did not write any of this code.
Now he has mixed feelings and claims didn’t know Shampoo would work when he was in the same discord channel where they discussed Muon as fancy shampoo or practical shampoo in 2024. @HessianFree back me up?
When I say @_arohan_ vs. keller disagreement is deeply laudable (and edifying), here’s what i mean:
Keller has been leading a righteous crusade against optimizers that claim to improve on sota, but turn out to be fake —compared to untuned or detuned baselines— with his speedrun project.
Meanwhile, Rohan has been quietly pointing out that —ironically— Muon is just such an optimizer. ie, it was a small variation on his preexisting SOTA optimizer (Shampoo) long in use at Google, winner of AlgoPerf, etc. that only looked inferior to Keller’s Muon because it was untuned. But didn’t (perhaps due to his BigLab employment, couldn’t) conclusively set the record straight on the speedrun…until yesterday.
Keller is now being hoisted by his own petard: the exemplary standards he loudly, angrily set about optimizers that incorrectly claim new SOTA, enforced by the de facto industry standard for first-pass optimizer evaluation he virtuously created.
And you’ll notice he’s handling this with —maybe not perfect poise, there’s been a bit of cope— but taking the lesson with more humility and grace than most would.
This paper from Tsinghua University and Shanghai Jiao Tong University received perfect scores (6, 6, 6, 6) at NeurIPS 2025!
It aims to answer a key question: Does reinforcement learning really make large language models better reasoners?
The authors study Reinforcement Learning with Verifiable Rewards (RLVR) and find that while it improves accuracy for small k, it doesn’t create new reasoning patterns—meaning the base model still determines the upper limit of reasoning ability.
Across six RLVR variants, performance gains plateau, suggesting that current RL setups mainly refine reasoning rather than reinvent it.
Interestingly, it’s distillation, not RL, that shows genuine signs of emergent reasoning.
This research points to the next frontier for truly self-improving large language models.
Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
Paper: https://t.co/jOvAn6eGIZ
Page: https://t.co/o951L62oR6
Our report: https://t.co/nnMLg0FC5p
📬 #PapersAccepted by Jiqizhixin
1/8 Second Order Optimizers like SOAP and Muon have shown impressive performance on LLM optimization. But are we fully utilizing the potential of second order information? New work: we show that a full second order optimizer is much better than existing optimizers in terms of iteration complexity (~5x over SOAP and ~15x over Muon).
Hmm...what I was suggesting is: it is fine that you don't scale every baseline to 1T. Everyone knows it's difficult. However, if you didn't do the ablation, it is not scientific to claim that Muon/MuonClip is the key factor that enabled zero spike loss. Just be humble and say "our evidence on 16B suggest that Muon might be the key for 1T training, but we are not sure what roles it plays in our case and it needs future work for verification". Honestly, a good model like K2 already speaks for itself already and you don't need more than that.
OK let's talk about it. As someone with a physics background who cares about controlled studies, I find the Kimi-Muon paper inconsistent (but it's a good paper still!). It doesn't substantiate the stronger claims - especially those referenced in K2. Here is why.
- In the Moonlight paper, Muon experiments were run on a max 16B model, not 1T. You can't repeat the Moonlight results and use them to support 1T claims (which are questionable given that zero/few spikes were achieved in Puangu/Qwen3Max/DSV3 without Muon/MuonClip).
- The reported 2X speedups in Moonlight come from close-sourced Llama runs with unclear Adam settings (e.g., unknown betas). I haven’t seen these results reproduced in a fair, transparent setup.
- In their core 16B runs, Muon’s gains are much smaller - under somewhat between 1.15 and 1.2X - likely because the larger model is beginning to overtrain. The trend isn’t that terribly encouraging.
- The 16B experiment isn't 100% careful: the Adam learning rate wasn't tuned. Using an untuned Adam LR while applying a 0.2 rescaling for Muon makes the comparison unfair and may inadvertently pick a better LR for Muon.
- A more careful study from Stanford (@wen_kaiyue / @percyliang 's team) shows Muon's speedups shrink with scale - down to ~1.1X at 1B with 8X Chinchilla-and it only ties Nesterov-Adam (see figure). This time Adam's LR was tuned (and the authors say they tried their best to tune Muon as well).
- Concurrent work from EPFL (@AndreiSemenov17) reports Muon underperforming Adam in certain setups.
- @EssentialAI reports only 10–15% improvement in sample efficiency at a regime that barely exceeds the Chichilla law (~1.25X Chinchilla at 2B and ~0.6X at 4B), and they already observed lower speedups. This is a negative result for me, considering that we already have Nestrov AdamW which can be 10% faster than AdamW (according to stanford paper, still need to be very careful), and Muon has additional overhead.
I'm not saying K2's claim is false; I'm saying it's unfalsifiable as stated. Simply repeating "X showed Y is more efficient than Z" without checking assumptions, baselines, and setups is more likely to mislead than help.
@JamesWhate89993 Hmm they did publish a full paper with many ablation to show that muon is more flops efficient than Adam in their setup. They did the work proprely and publish all their results
https://t.co/pXHUNMct3l