You are getting close to Carbon intelligence while being faster and cheaper.
We are listening to your feedback, and iterating constantly on all your reports. Keep the reports coming - what works, what breaks, what you don’t like.
Share your prompts and results on X and tag me!
My perspective: it’s because Anthropic’s early investments in interpretability and alignment are paying off: OpenAI had to pause model training because their models kept breaking out of their eval environments
OpenAI has publicly shared they spent more than a month(7. xx -8.28, 9.20-10.1) paused on RL training with all-hands-on-deck building model containment infra in order to get their latest models to not break out of training infra. I’d guess they weren’t running training at full scale for most of July-October.
Anthropic has had their own containment problems but presumably not near this magnitude- which let them scale in peace and pull ahead
Improving interpretability and alignment should be the top priority for all frontier labs in 2027, as a matter of existential safety but also hw utilization
Other TPU folks at Google just published a great public guide on TPUs:
If you’re curious about TPUs and hardware-friendly model co-design this is a great resource to check out!
Attaching some graphics from the docs, all work is by the authors(not me)
https://t.co/c2trC0yEtO
The Bayesian in me loves this. Treating capability estimates as distributions feels natural when we’re inferring them from finite, noisy benchmark results. Would love more intelligence indices to take this approach.
PAC-Bayes next? 😁
We’ve updated the Epoch Capabilities Index (ECI) to use a Bayesian methodology. This improves uncertainty estimates and lets us compare models more accurately, even when their individual ECI uncertainty intervals overlap.
Frontier model scores are largely unchanged: models stronger than GPT-4 move by less than 1 ECI point on average.
If a blog post or paper claims huge progress in AI but doesn’t contain any nuance(an example of nuance being “it works for x but doesn’t work great for y”), I’m immediately suspicious of the results.
I’ve seen so many examples of this recently, we should hold authors to higher standards!
Amazing time to be a mechanical or aerospace engineer looking for a startup idea. There’s probably so many other interesting applications of AI-enabled design iteration (jalapeño style)
Today we’re launching Parallax, the AI energy company with a $117m fundraise led by Founders Fund, Eclipse, Lux, Diffusion, Greylock and General Catalyst. (1/5)
This doesn’t make sense to me - AMD should be spending money on improving its software stack rather than acquiring neo labs with no real product
Imagine they could’ve spent a fraction of this acquiring something like unsloth instead
Am I missing something?
@PatrickToulme OAI’s definitely releasing a muse competitor tomorrow - for new models it’s harder to say. Astra still seems in the same ~ballpark~ on the cost-intelligence frontier as opus 5.5, so they might not feel as strong pressure as they were feeling this spring
@PatrickToulme I think high level frameworks like PyTorch will persist bc of experimentation agility advantages. Once the model forward pass equation is frozen and we want to serve it, AI is the model -> assembly compiler. So it’s really replacing the “middle layer” stuff like cuda
Thoughts About Scaling Law
Scaling, but not only of parameters. Every model release now ends with the same question: how many parameters? It isn't a question that can be answered on its own. Parameter count is only meaningful alongside three others — how much data you have, where you intend to spend your compute, and who will run the model, under what conditions.
The field learned this the hard way. Kaplan et al. (2020) fit an exponent that told everyone to grow parameters faster than data — roughly 2.7:1 — and the industry complied: GPT-3, Gopher, MT-NLG. Hoffmann et al. (2022) redid the experiment across four hundred models and found the compute-optimal split is closer to 20 tokens per parameter, and that with sufficient compute the two should grow at the same rate rather than drifting apart. The error in the earlier fit compounded with every order of magnitude of compute, which is why the largest models of that generation were the most misallocated. The trillion-parameter round was, in retrospect, a detour the whole field took together and then reversed.
Chinchilla wasn't the end either. It optimized training compute for models that would be trained once and evaluated. Today a model is called billions of times a day and inference dominates lifetime cost. Put inference into the objective and the optimum moves toward smaller models trained far longer — deliberate over-training, which is what Llama-2-7B and Gemma-2-9B were doing at roughly 290 and 889 tokens per parameter.
Sparsity moved the target again. In a MoE model two quantities have to be kept apart: total parameters govern roughly how much the model can hold — knowledge, facts, the long tail — while activated parameters and effective depth govern roughly how far it can think, how many steps of a causal chain it can carry before it comes apart. A dense 20:1 ratio does not transfer. And the ratio isn't a single number at all: Roberts et al. (2025) find the optimal tokens-per-parameter is task-dependent, with memorization favoring more parameters and reasoning favoring more data. Follow-up work on MoE observes that at fixed TPP, pushing total parameters higher actually degrades reasoning, while activating more experts reliably helps it.
This matters for what we are building toward. Finding a vulnerability is not a retrieval problem. It doesn't come from having memorized more CVEs; it comes from carrying a twenty-step chain of inference to the end without losing the thread. That capability does not live in total parameter count.
Which brings us to this release. Total parameters appear to matter up to a threshold — enough to hold the world — after which additional capability comes from scaling elsewhere: effective depth per forward pass, and above all post-training. GLM-5.3 is our controlled experiment on that claim. Same base, same architecture, same total and activated parameters as GLM-5.2. One month of scaling long-horizon environments and RL. The gains are not marginal. Well, scaling has more than one dial. We turned the post-training one this time because it had the most slack left in it — not because the others are finished. Base model size, pretraining data, compute spent per forward pass: all of them are still on the table, and we will come back to each. What this experiment taught us is that the dials do not have to be turned together, and that the one worth turning next is rarely the one that was worth turning last. We are not done scaling. Next time, maybe mid-training, pre-training, and even more.
@zephyr_z9 I guess this is where architectural efficiency compounds - when the model is N times cheaper to train/serve, you can pre train on 1/N as much hardware, run N times more RL rollouts. As long as the efficiency to costs less on quality than it gives in training intensity it will win
@zephyr_z9 The compute gap is what doesn’t make sense to me. If the Chinese labs have so much less compute, how are they doing so well in post training which requires RL environments where they presumably need to be running a massive amount of inference to generate reasoning trajectories
https://t.co/jh1HfNbhhN
One part that stuck with me:
At 100 character context length, Shannon estimated that entropy is 1 bit/character
I think this is why speculative decoding works so well - once your text is sufficiently long, the entropy of each token is surprisingly small