Since I started getting interested in ML I got it in my head that all I wanted to do was one smart thing that I could look back on and be satisfied that I did. Most papers are kinda bad even if they get accepted - the idea is very incremental, or it's just not that good an idea, or it doesn't really matter.
I never was able to do this all through PhD or my time at Amazon. All the papers I did there got into various places, but I never really thought they were actually that good. And I'd pretty much given up on this because Pluralis meant I couldn't really devote enough time to research myself. But in February I decided I didn't care and spend two months focused on a specific problem that had been going round in my head for about a year that I felt we needed to solve, and the solution came to me, and @ChaminHewa picked it up and generalised the approach and ran a bunch of novel experiments I hadn't thought of, and pulled everything together into an actual paper. And yesterday we presented this work at NeurIPS.
This is the first and probably only work I will ever do that for me feels like "ok that was GOOD". I don't care if it racks up a bunch of citations and disperses into the field or not, I don't care if someone repackages the ideas and takes all the credit for it, I don't care. For me there is an internal checkbox that just got ticked after more than ten years of trying. Anyone in ML will understand what I'm trying to say. Special day I'm going to remember for a long time.
"Outside of NVIDIA, the biggest loser is the open-source ecosystem." — @dylan522p
"Open-source in America would not exist without Jensen. Both Reflection and Thinking Machines are pushing the open-source frontier in America, and both have massive investments and backing from NVIDIA. As do many other companies across the ecosystem."
"And all these application companies rely on open models. Whether it's the Harveys, the Instincts, the Sierras — they all use open models."
"All the inference providers, which include multiple of the 10+ billion dollar companies, rely on open models from China."
"And if we're slamming the hammer down on China way harder, and the biggest defender against restrictions was Jensen and his team, what happens now? Given their credibility is out the window and they're being investigated and subpoenaed, etc."
"If there are going to be a lot more restrictions, do Chinese models continue to progress at the current pace? And if they do, do they open-source everything? These are real questions. But I think the open-source ecosystem is the biggest loser, for sure, if you're talking about outside of NVIDIA."
Saw Aaron's mention of his work on AdamC, thought to put the gist of it together in a thread. This work is about the contrast in learning rate going down vs. the gradients getting larger. The paper shows how weight decay and normalisation can cause this, and how a small change to decay helps mitigate this.
1/12
Here is the math of "Why Gradients Rapidly Increase Near the End of Training".
During training, gradient update increases weight norm and weight decay decreases weight norm, together they can reach an equilibrium where the weight norm stays roughly constant.
But LR schedule decreases LR near the end of training, weight decay becomes relatively stronger, causing weight norms to shrink.
In a scale invariant network f(cw) = f(w), gradients are inversely proportional to the model weights. And because of this, smaller weights at the end of training can produce larger gradients.
The fix in AdamC(https://t.co/BFlHUYDCAz) is quite simple, proportionally decrease weight decay as LR gets smaller.
Derivation on weight norm geometry: https://t.co/lMhVQrSvbU
1. Benchmarks provide one piece of evidence for acceleration. Since Oct 2025, AECI scores for Anthropic’s models have grown at 1.9x [95%: 1.5–2.8] the earlier rate. Analysis choices affect this estimate, and it remains unclear how benchmark scores map to real-world capabilities.
"We opted not to buy any posttraining data, instead devising our own synthetic data pipelines, quality filtering, and coverage metrics." has quite large implications when combined with the fact the whole thing cost 32M.
Today we release Ripple-65B Preview, our 65B dense LLM we pretrained from scratch.
Ripple sets a new state-of-the-art for U.S. open-weight models in agentic coding tasks, surpassing significantly larger models including Muse Spark Open-816B, Beam-501B, Inkling-975B, and Nemotron 3 Ultra-550B.
Learn more: https://t.co/Pe04ZKhkE9
@iScienceLuvr For a long time it was just flat out denial. Kept running confs like EMNLP, had gary marcus come and explain how LLMs were a dead end in 2022 and kept publishing papers on knowledge graphs, neural coreference and entity recognition.
"“What do you expect?" he said. "For generations they've been built up to worship competition and the market, productivity and economic usefulness, and the envy of their fellow men-and boom! it's all yanked out from under them. They can't participate, can't be useful any more. Their whole culture's been shot to hell.”"
Received a moving message from an aspiring mid-20s professional that, I think, captures an increasingly widespread despair among younger cohorts:
"It really saddens me when I hear about the work [senior people in the field] are doing. Their work seems interesting, and I'll never get to do it.
The people in their fifties and sixties, they're lucky. I'm even lucky. I feel bad for the kids who are fourteen. I at least got a chance to use my brain for one or two minutes, and that's still priceless. I got to be human. I got to be more than just some puppet who presses Enter so something can think for me.
It's very depressing looking at people who are sixty, who I would love to be in twenty years. But to know that I won't get to do that because Dario said so, right? And, like, that's actually heartbreaking.
It's heartbreaking to know that no matter how hard you try and no matter what you do, you cannot add value as a human being in a year or two's time.
It's like hitting a brick wall. You feel no motivation to build the skill, to get better, to attain wisdom and knowledge. It seems worthless to you. And all the older people are telling you all your efforts might be worthless, and they don't need you anymore. And they can just sit there and, you know, make money and leave you to drown."
It's extremely rare to encounter such a thoughtful blog on pretraining and scaling. I really liked the profiling plots depicting a clear yet sufficient picture.
1501: Pope Alexander VI criticizes Gutenberg’s printing press safety
“The art of printing can be of great service in so far as it furthers the circulation of useful & tested books; but it can bring about serious evils l..it will, therefore, be necessary to maintain full control”
We are fortunate to have 3 papers accepted to this years NeurIPS.
The first is AsyncMesh - a technique for asynchronous updates across both data and pipeline parallelism simultaneously.
Second is NuMuon, which shows the low-rank structure we observe in Subspace Networks is also present and can be exploited under Muon-style optimisers.
The third introduces a technique to finetune open-weight models in the Protocol Learning setup. This is significant as prior to this work, this kind of distributed training required architectural changes, and hence starting from scratch.
My position is typically to focus on getting things to work rather than making noise about what needs to be done, but enough people have messaged me in last 24 hours that felt I should write some brief thoughts. The tl;dr is I completely agree with the sentiment but think no-one aside from Pluralis is earnestly moving towards a realistic solution.
The reality is without some mechanism to pool resources for the creation of the models - with hardware that can't be taken away - the situation remains exactly the same, with a faint hope that Nvidia pours enough money into various challengers that the open <> closed model gap stays narrower for longer. I see very few scenarios where an individual group can pull ahead to the frontier (and even if they do, they will be subject to the same dystopian regulation the labs are now speedrunning towards). It must be a community driven effort to achieve this and hence we require a substrate to pool compute, data, and expertise effectively. This looks like p2p/distributed/multiparty training using hardware that has already been purchased to do local AI, thats sitting idle half the time, is physically controlled by individuals and can't be taken away. Repurposing this hardware for creation of the models and not just consumption is absolutely critical.
I don't view most of the other things in this list as meaningfully moving the needle towards a good outcome. RL post-trains as a service.. ok great you made it easier for enterprise clients to use open models. Inference providers serving openweights... ok you host models and make money I don't see what you've changed. Open harnesses... my concern was never that I'd be locked out of the harness layer. Local inference... I don't really care about qwen3.8 27b running at 200 tok/s I want deepseek v4 pro running locally, and I want a better model than deepseek v4 pro. I wanna own and impact the process not the output.
We don't have this yet because its hugely technically challenging - it requires solving fundamental research problems around low-bandwidth training and a re-write of most layers of the training stack. This is what we've been doing with extreme urgency for the last year and what we will continue to do until its working. Once such a system exists however, we will look back and find it strange that private companies ever felt they would be able to monopolize and control something this fundamental.
If you are reading this, the only possible resistance is to build decentralized, permissionless, and sovereign AI.
Open source, self-hosting tools, RL post-training, local inference, open harnesses, we must build it all.
We must fight overcentralization with an open stack.
Yeah that's my view - networks of many participants +pooled hardware is the way this will all play out long term. The problem has always been combining hardware from many people in one run was just not possible technically. This is what we've been working on.
There's actually a test run going on right now if you'd like to push a GPU into it - will stay up another day or so https://t.co/40o6hp5JiS