Zhimeng Wang

Long Context LLM and Memory.

[jianlin.su @Jianlin_S]

Revisiting Convergence Results in Convex Optimization (VIII)
https://t.co/z3p7R4Q2I9

This article re-examines “schedule-free learning rates” from the perspective of multi-stage training, shifting the scheduling objective to “approaching optimality by the end of each stage” .

[Sean Lie @seanlie]

Ten years ago, we started @cerebras around an approach many believed was impossible.

As a computer architect, it is hard for me to imagine a more exciting time. Model releases are accelerating, and hardware tapeout is compressing from multi-year roadmaps to annual launches.

Hot Chips is my favorite conference, and it’s where I launched Cerebras 7 years ago. This year’s conference was especially exciting, and so much innovation was shared. I am watching the industry recreate itself: SRAM is mainstream, DRAM is moving into the third dimension, networks are being fundamentally redesigned, and AI is helping design and program the chips themselves.

The industry has never moved faster and some of the hardest architectural questions are still wide open.

[elie @eliebakouch]

every chinese frontier model now uses linear attention (except deepseek)

they all use (except kimi) sparse attention with similar indexer/compression designs to maximize efficiency

they all use "fancy" residuals (mHC, attention residual, gated residual) to maximize signal propagation

they all use Muon

very exciting time for frontier (and efficient) oss models, the beauty of open research :)

[elie @eliebakouch]

this plot is even more insane than it looks

vera rubin here uses a somewhat optimized config with P/D disagg and speculative decoding while jalapenos uses none of this and is still better. P/D disagg also favors the "per W" normalization since you can separate and only count the power of the decode engine

another very impressive thing in this plot is that there are only 5 days between the light purple and dark purple curve, shows the speed of software development (they also changed the rack disposition to allow more parallelism)

about P/D disagg, the article talks about it in more details but from what i understand the jalapenos chip is designed to need less of it (maybe it's similar for speculative decoding?). extract from oai blog

> "The result is a balanced and fungible accelerator that can support changing model architectures, excel at both prefill and decode, and adapt as the balance between them changes, a defining feature of agentic workloads."

other caveats they mention in the blog is that this is on 8K prefill / 1K output which is not agentic setting, this is also on previous generation models (no 3T kimi K3), also important to have in mind that this plot is normalized by "unit" power (not only chip) so to get the same curve vera rubin needs to generate 3x more output tokens (i think this is a fair metric, just need to be aware of this imo).

scale up domain details are also interesting, slower local bandwidth (and fit less chips and less "power" i think) but they have a with a "global" tier (local 128 / global 2048 chips) that would let bigger models run without paying too much "scale out price" for it

[jietang @jietang]

Thoughts About Scaling Law

Scaling, but not only of parameters. Every model release now ends with the same question: how many parameters? It isn't a question that can be answered on its own. Parameter count is only meaningful alongside three others — how much data you have, where you intend to spend your compute, and who will run the model, under what conditions.

The field learned this the hard way. Kaplan et al. (2020) fit an exponent that told everyone to grow parameters faster than data — roughly 2.7:1 — and the industry complied: GPT-3, Gopher, MT-NLG. Hoffmann et al. (2022) redid the experiment across four hundred models and found the compute-optimal split is closer to 20 tokens per parameter, and that with sufficient compute the two should grow at the same rate rather than drifting apart. The error in the earlier fit compounded with every order of magnitude of compute, which is why the largest models of that generation were the most misallocated. The trillion-parameter round was, in retrospect, a detour the whole field took together and then reversed.

Chinchilla wasn't the end either. It optimized training compute for models that would be trained once and evaluated. Today a model is called billions of times a day and inference dominates lifetime cost. Put inference into the objective and the optimum moves toward smaller models trained far longer — deliberate over-training, which is what Llama-2-7B and Gemma-2-9B were doing at roughly 290 and 889 tokens per parameter.

Sparsity moved the target again. In a MoE model two quantities have to be kept apart: total parameters govern roughly how much the model can hold — knowledge, facts, the long tail — while activated parameters and effective depth govern roughly how far it can think, how many steps of a causal chain it can carry before it comes apart. A dense 20:1 ratio does not transfer. And the ratio isn't a single number at all: Roberts et al. (2025) find the optimal tokens-per-parameter is task-dependent, with memorization favoring more parameters and reasoning favoring more data. Follow-up work on MoE observes that at fixed TPP, pushing total parameters higher actually degrades reasoning, while activating more experts reliably helps it.

This matters for what we are building toward. Finding a vulnerability is not a retrieval problem. It doesn't come from having memorized more CVEs; it comes from carrying a twenty-step chain of inference to the end without losing the thread. That capability does not live in total parameter count.

Which brings us to this release. Total parameters appear to matter up to a threshold — enough to hold the world — after which additional capability comes from scaling elsewhere: effective depth per forward pass, and above all post-training. GLM-5.3 is our controlled experiment on that claim. Same base, same architecture, same total and activated parameters as GLM-5.2. One month of scaling long-horizon environments and RL. The gains are not marginal. Well, scaling has more than one dial. We turned the post-training one this time because it had the most slack left in it — not because the others are finished. Base model size, pretraining data, compute spent per forward pass: all of them are still on the table, and we will come back to each. What this experiment taught us is that the dials do not have to be turned together, and that the one worth turning next is rarely the one that was worth turning last. We are not done scaling. Next time, maybe mid-training, pre-training, and even more.

[Junyang Lin @JustinLin610]

a life update: i started a new company called Pragmatik (p7k) Labs (语用科技) in shanghai, focusing on the research of next-generation agents across digital and physical worlds. thanks to Gaorong Ventures and HSG (红杉中国 & 高榕创投) for co-leading this round, and to Tencent (腾讯) and Shanghai Engine Fund (上海未来产业基金) for the support. @pragmatik_labs · https://t.co/01J3uSbxmX

[Thinking Machines @thinkymachines]

Today, we are releasing Inkling-Small.

Inkling-Small achieves comparable performance to Inkling at a quarter of its size. It features 276B total parameters, 12B active. We are making the full weights available.

https://t.co/BtYNcpkDRA

Fine-tune it on Tinker today, or chat with it in text, image, and audio on Tinker Playground.

[Yu Zhang 🐙🌘 @yzhang_cs]

lost for words rn.

[Kimi.ai @Kimi_Moonshot]

Releasing the model weights and technical report of Kimi K3.

Kimi K3 is our most capable model: a 2.8T MoE model with native visual understanding and a 1M-token context window.

New model architecture: 2.5x the intelligence per unit of compute, not just more params.

Alongside Kimi K3, we're opening up more of the stack behind it — high-performance attention kernels, MoE communication library, and infrastructure for running agent environments at scale.

Model weights: https://t.co/7m7eEg6Y0B
Tech report: https://t.co/yeu6cjpMCT
Tech blog: https://t.co/YTfiMSNM1f

[Julian Schrittwieser @Mononofu]

I’m so excited that @JensenHuang is a believer in open source now, looking forward to the CUDA and GPU driver open source release!

[Jensen Huang @JensenHuang]

For my first post, I’m sharing a letter @NVIDIA signed on why open models matter.

AI will transform every industry, power every company, and be built by every country.

Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty.

The world needs both frontier closed models and frontier open models.

https://t.co/AUKzoQ5Ikb

[Yu Zhang 🐙🌘 @yzhang_cs]

In the early morning, @nathancgy4, @Xinyu2ML, @Yulun_Du and I were preparing some showcases for the blog while watching the World Cup. The moment Argentina beat England, I felt something. I looked up, and saw the most unforgettable Beijing sunrise. So I took the photo.

I knew this was no ordinary day, and now it comes and says, "Hello, world!"

Happy hatch day, K3!

Please enjoy it, I have felt 「眩晕瘫坐」 for weeks.
It is a monster model, yet clean in architecture; elegant in optimization, beautiful in MoE desigins.
It has crossed a certain singularity, it is a year of work, in one release.

🚀🚀🚀 This is a Sputnik moment for OSS.

[jianlin.su @Jianlin_S]

Hello, K3!
https://t.co/D1NJeIky5x

[jianlin.su @Jianlin_S]

Taylor Expansion of LogSumExp and Softmax
https://t.co/HPk88biP8n

[elie @eliebakouch]

mythos will be bad ON PURPOSE on ai "frontier llm research" tasks, this is very very sad for the research community

also the fact that this is un purpose not visible to the user is crazy

[rohan anil @_arohan_]

Here you go, sir. Muon is a good optimizer.

I think Keller's attempt at implementing it is great -- this primarily helped me look at why his effort produced much worse looking curve than what I could get.

This ended up being a nerdsnipe into hyper parameters, grafting, and eigh calls in the distributed shampoo package from Meta in my car ride home.

The main delta's are below:

[jianlin.su @Jianlin_S]

Steepest Descent on Manifolds: 6. Muon + Double Rotation
https://t.co/tGfnlzfYv6

Introduces MuonR — a Muon variant that constrains updates to left & right rotation matrices. This preserves the singular value distribution of weights, providing a clean, elegant way to maintain training stability.

[Tilde @tilderesearch]

Introducing Compositional Muon, an optimizer that extends Muon from individual matrices to composed transformer circuits.

Modern optimizers usually draw trust regions around individual parameters. But in attention, the loss often sees compositions like QK^T and OV. Updating each factor independently can therefore control the wrong object.
Compositional Muon closes this gap by deriving partner-whitened update rules. Each factor’s update is shaped by the spectral geometry of the matrix it is composed with, producing more stable composed updates and better effective learning-rate allocation across heads and layers.

For QK, this gives a head-local half-split rule. For OV, the circuit geometry selects a hybrid rule: (V) is optimized per-head, while (W_O) is optimized as the single matrix that aggregates all heads back into the residual stream.

CM improves over Muon at 340M and 1B scale, transfers to the modded-nanoGPT optimization benchmark, and can be approximated cheaply as partner-rescaled Muon via the isotropic rule.

The broader point is optimizer-architecture co-design: better optimizers should not only ask how to update a parameter, but what composed circuit that parameter participates in. CM is one step toward optimizers that respect the functional structure the loss actually sees.

[Z.ai @Zai_org]

https://t.co/jaOMnP7Yud

[elie @eliebakouch]

we let opus 4.7 and gpt 5.5 run on the nanogpt optimizer speedrun: ~10k runs, 14k H200 hours, 23.9B tokens. opus hits 2930, codex 2950, both beating the human baseline of 2990. we cover claude autonomy failures, codex high compute usage, and much more

https://t.co/3sf1fUD7PU