Chen Liang

MTS @MicrosoftAI. PhD @mlatgt. Feed and train large models.

[Chen Liang @chenliang1_]

Checkout NorMuon, neuron-wise normalized muon that combines orthogonalization with adaptive lr.

https://t.co/m1xeOK90B0

[Tuo Zhao @tourzhao]

🚀 NorMuon: Muon + Neuron-wise adaptive learning rates: +21.7% training efficiency vs Adam, +11.3% vs Muon on 1.1B pretrain.

🚀 Distributed Normuon: A highly efficient FSDP2 implementation.

Paper 👉 https://t.co/4OuWAdYbzB
#LLM #AI #DeepLearning #Optimizer

[jianlin.su @Jianlin_S]

arrived as promised

[Jensen Huang @JensenHuang]

For my first post, I’m sharing a letter @NVIDIA signed on why open models matter.

AI will transform every industry, power every company, and be built by every country.

Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty.

The world needs both frontier closed models and frontier open models.

https://t.co/AUKzoQ5Ikb

[Kimi.ai @Kimi_Moonshot]

Introducing Kimi K3: Open Frontier Intelligence

🔹 2.8 Trillion Parameters, 1 Million Context, Native Multimodal
🔹 Kimi Delta Attention enables up to 6.3x faster decoding in million-token contexts
🔹 Attention Residuals deliver ~25% higher training efficiency at <2% additional cost
🔹 Built for long-horizon agentic coding and self-evolving workflows

Kimi K3 is now live on on https://t.co/zrk6zZxZUo, Kimi Work, Kimi Code, and the Kimi API.
Open Weights by July 27, 2026.

🔗 API: https://t.co/XCrgjXAqMw
🔗 Tech blog: https://t.co/YTfiMSNM1f

[jietang @jietang]

Our Single-rollout Asynchronous Optimization (SAO), is able to train stably for one thousand steps and consistently outperform GRPO and its variants on agentic coding and reasoning benchmarks,
such as SWE-Bench Verified, BeyondAIME, and IMOAnswerBench. https://t.co/oca1DvDvdB

[humans& @humansand]

At humans&amp;, we train models from the long-term impacts of their interactions with people. This requires prioritizing long-horizon multi-agent RL. We've developed and are excited to share an open-source, hardware-native 4-bit RL recipe, significantly accelerating training

[Yang Liu @nlpyang]

Excited to introduce our MAI Code model at Microsoft Build. As shared in the session, this is a MoE (5B active / 137B total) initialized from an MAI pretrained model and trained for real user scenarios with product harnesses. I’m proud to have served as the research lead for this effort, and even prouder of what the team has achieved. It’s a beast for its size. Stay tuned — a larger model could come :)

[Hanna Hajishirzi @HannaHajishirzi]

MAI-Thinking-1 is out!

Excited to share what we are building and how climbing from scratch (no distillation) actually works: simple recipes, rigorous science, self-distillation, patience, and great infra.

Check out our tech report has the full story of our RL climbs.
https://t.co/aLW40sWz4d

[Chen Liang @chenliang1_]

Checkout MAI-Code-1-Flash in VS Code ⚡️

https://t.co/V0Xj1gZfbO

[Weizhu Chen @WeizhuChen]

We just released MAI-Code-1-Flash, a 5B-active small model. The team just finished this from a pretrained model to VSC and Copilot CLI production within a few weeks, so proud of their achievements. More details are here https://t.co/jx85ssYAea and a bigger model is on the way.

[Keller Jordan @kellerjordan0]

Three new modded-NanoGPT optimization benchmark results, all of them using NorMuon, have near-concurrently improved the benchmark record from 3325 to 3250 steps.

1) Kumar Krishna Agrawal (gh:kumarkrishna) used NorMuon with an update-clamping strategy (no weight decay, like hyperball) to reach the target loss in 3250 steps.
2) @wen_kaiyue used NorMuon with hyperball optimization.
3) Subsequently, Liming Liu (gh:lliu606) (one of the NorMuon authors!) used standard NorMuon with weight decay.

NorMuon is also used in the main NanoGPT speedrun track. Ali Naeimi did the earliest NorMuon runs on the optimization benchmark, albeit below stat sig.

NorMuon authors: @li_zichong, Liming Liu, @chenliang1_, @WeizhuChen, and @tourzhao

[Chen Liang @chenliang1_]

The method was inspired by a bug in ctx parallelism, a very interesting journey with @li_zichong 🤣. Also, many thanks to great discussions w/ @liliang_ren, Yelong and @WeizhuChen along the way.

[Chen Liang @chenliang1_]

Introducing RoPE-Perturbed Self-Distillation: perturb RoPE indices to create alternative views of the same sequence, then train the model to make consistent predictions across views

https://t.co/TKy7Ctr4ro

[Kaizhao Liang @KyleLiang5]

A fun project putting ideas brainstormed with @chenliang1_ into reality, combining deep research and a childhood classic 🐍

you have to bring your own key so I don't go broke, supporting @claudeai @OpenAI @GeminiApp

demo: https://t.co/kY1JZDFzni
code: https://t.co/9mL7GNXA9j

[Anthropic @AnthropicAI]

New on the Anthropic Engineering Blog:

How we use a multi-agent harness to push Claude further in frontend design and long-running autonomous software engineering.

Read more: https://t.co/HWvmXk1ykn

[Andrej Karpathy @karpathy]

Three days ago I left autoresearch tuning nanochat for ~2 days on depth=12 model. It found ~20 changes that improved the validation loss. I tested these changes yesterday and all of them were additive and transferred to larger (depth=24) models. Stacking up all of these changes, today I measured that the leaderboard's "Time to GPT-2" drops from 2.02 hours to 1.80 hours (~11% improvement), this will be the new leaderboard entry. So yes, these are real improvements and they make an actual difference. I am mildly surprised that my very first naive attempt already worked this well on top of what I thought was already a fairly manually well-tuned project.

This is a first for me because I am very used to doing the iterative optimization of neural network training manually. You come up with ideas, you implement them, you check if they work (better validation loss), you come up with new ideas based on that, you read some papers for inspiration, etc etc. This is the bread and butter of what I do daily for 2 decades. Seeing the agent do this entire workflow end-to-end and all by itself as it worked through approx. 700 changes autonomously is wild. It really looked at the sequence of results of experiments and used that to plan the next ones. It's not novel, ground-breaking "research" (yet), but all the adjustments are "real", I didn't find them manually previously, and they stack up and actually improved nanochat. Among the bigger things e.g.:

- It noticed an oversight that my parameterless QKnorm didn't have a scaler multiplier attached, so my attention was too diffuse. The agent found multipliers to sharpen it, pointing to future work.
- It found that the Value Embeddings really like regularization and I wasn't applying any (oops).
- It found that my banded attention was too conservative (i forgot to tune it).
- It found that AdamW betas were all messed up.
- It tuned the weight decay schedule.
- It tuned the network initialization.

This is on top of all the tuning I've already done over a good amount of time. The exact commit is here, from this "round 1" of autoresearch. I am going to kick off "round 2", and in parallel I am looking at how multiple agents can collaborate to unlock parallelism.
https://t.co/WAz8aIztKT

All LLM frontier labs will do this. It's the final boss battle. It's a lot more complex at scale of course - you don't just have a single train. py file to tune. But doing it is "just engineering" and it's going to work. You spin up a swarm of agents, you have them collaborate to tune smaller models, you promote the most promising ideas to increasingly larger scales, and humans (optionally) contribute on the edges.

And more generally, *any* metric you care about that is reasonably efficient to evaluate (or that has more efficient proxy metrics such as training a smaller network) can be autoresearched by an agent swarm. It's worth thinking about whether your problem falls into this bucket too.

[Chen Liang @chenliang1_]

🙏

[jianlin.su @Jianlin_S]

My 2025: https://t.co/0FGtHL9gJb

[Larry Dial @classiclarryd]

New NanoGPT WR from @ChrisJMcCormick at 130.2s, a 1.4s improvement! He has somehow found a way to make Muon even faster, along with several other optimizations to pre-multiply lambdas, update Normuon axis on gates, and reshape matrices. https://t.co/WsiPzddGGJ.

[Chen Liang @chenliang1_]

We will present SlimMoE at #COLM2025 #COLM in Poster Session 3 (Wednesday Morning 11:am-1pm)

[Chen Liang @chenliang1_]

We released two small yet competitive MoE models, compressed from the 42B Phi-3.5-MoE.

Phi-mini-MoE-instruct (2.4B activated, 7.6B total): https://t.co/ce8BNNf4TC

Phi-tiny-MoE-instruct (1.1B activated, 3.8B total): https://t.co/VUUJ7kZPgx

[Thinking Machines @thinkymachines]

Efficient training of neural networks is difficult. Our second Connectionism post introduces Modular Manifolds, a theoretical step toward more stable and performant training by co-designing neural net optimizers with manifold constraints on weight matrices.

https://t.co/PGG4zy3u23

We explore a fundamental understanding of the geometry of neural network optimization.

[Thinking Machines @thinkymachines]

Today Thinking Machines Lab is launching our research blog, Connectionism. Our first blog post is “Defeating Nondeterminism in LLM Inference”

We believe that science is better when shared. Connectionism will cover topics as varied as our research is: from kernel numerics to prompt engineering. Here we share what we are working on and connect with the research community frequently and openly.

The name Connectionism is a throwback to an earlier era of AI; it was the name of the subfield in the 1980s that studied neural networks and their similarity to biological brains.

https://t.co/lrJioBmpbT