James

🇦🇺 engineering @pluralis

[Pluralis Research @Pluralis]

We are fortunate to have 3 papers accepted to this years NeurIPS.

The first is AsyncMesh - a technique for asynchronous updates across both data and pipeline parallelism simultaneously.

Second is NuMuon, which shows the low-rank structure we observe in Subspace Networks is also present and can be exploited under Muon-style optimisers.

The third introduces a technique to finetune open-weight models in the Protocol Learning setup. This is significant as prior to this work, this kind of distributed training required architectural changes, and hence starting from scratch.

[Alexander Long @AlexanderLong]

My position is typically to focus on getting things to work rather than making noise about what needs to be done, but enough people have messaged me in last 24 hours that felt I should write some brief thoughts. The tl;dr is I completely agree with the sentiment but think no-one aside from Pluralis is earnestly moving towards a realistic solution.

The reality is without some mechanism to pool resources for the creation of the models - with hardware that can't be taken away - the situation remains exactly the same, with a faint hope that Nvidia pours enough money into various challengers that the open <> closed model gap stays narrower for longer. I see very few scenarios where an individual group can pull ahead to the frontier (and even if they do, they will be subject to the same dystopian regulation the labs are now speedrunning towards). It must be a community driven effort to achieve this and hence we require a substrate to pool compute, data, and expertise effectively. This looks like p2p/distributed/multiparty training using hardware that has already been purchased to do local AI, thats sitting idle half the time, is physically controlled by individuals and can't be taken away. Repurposing this hardware for creation of the models and not just consumption is absolutely critical.

I don't view most of the other things in this list as meaningfully moving the needle towards a good outcome. RL post-trains as a service.. ok great you made it easier for enterprise clients to use open models. Inference providers serving openweights... ok you host models and make money I don't see what you've changed. Open harnesses... my concern was never that I'd be locked out of the harness layer. Local inference... I don't really care about qwen3.8 27b running at 200 tok/s I want deepseek v4 pro running locally, and I want a better model than deepseek v4 pro. I wanna own and impact the process not the output.

We don't have this yet because its hugely technically challenging - it requires solving fundamental research problems around low-bandwidth training and a re-write of most layers of the training stack. This is what we've been doing with extreme urgency for the last year and what we will continue to do until its working. Once such a system exists however, we will look back and find it strange that private companies ever felt they would be able to monopolize and control something this fundamental.

[Pluralis Research @Pluralis]

This exceeded all expectations and continued to train with 40+ nodes in a single stage, 370 across the swarm and GPUs in multiple continents. TPS reached over 250k tok/s.

Compute wise, this local hardware swarm is equivalent to 40 colocated B200's and would run $150k per month, potentially more depending on the provider.

Thankyou to omk, mehmet and everyone else who helped us track down bottlenecks mid-run.

[Hadi M. Dolatabadi @hmdolatabadi]

PyTorch can run out of memory with 60 GB reserved and only 6 GB used by tensors. I was looking into its native CUDA allocator to understand where the other 54 GB can get stuck and thought to put this together in a visual guide.

1/n 🧵

[Pluralis Research @Pluralis]

A 13B stress-test has just finished in Agora and instead of winding down we're going to ramp the contributor count until it blows up. If you have access to good local ML hardware, point your agent at https://t.co/u81XkfssK0 and join the pre-train for a few hours. Currently at 165 consumer GPU's.

[Hadi M. Dolatabadi @hmdolatabadi]

After hosting two satellite events at ICLR and ICML this year, we'll be organising an official @NeurIPSConf workshop on collaborative, open and decentralised training with an stellar panel of speakers!

Check our call for papers here:
https://t.co/9KQ3ZUPq8N

Please consider submitting your work to our workshop. Feel free to reach out if you have any questions. Hope to see y'all in Sydney!

[Pluralis Research @Pluralis]

60 page tech report detailing Agora and the run it just powered now on Arxiv. Spans many different fields, from communication-efficient model parallelism, asynchronous optimization, to fault-tolerant systems design.

[Alexander Long @AlexanderLong]

I think the logic is extremely simple:
1. The model is the product. Will capture massive value and hand unprecedented power to whichever group(s) control it.
2. Closed models put everyone at the mercy of labs... the problems there are obvious. Open-weight models leak value, require constant subsidization, and will be blocked/restricted in the exact the same way closed releases are.
3. What is required is a neutral, independent model supply chain that cannot be seized, coerced, turned off or modified, distributes value fairly among its participants and users, and respects their data and IP, and is self-sustaining.
4. The ONLY way to achieve this is in a protocol where models are trained and served by many participants on pooled global compute and owned by no one entity. All other efforts fail because they just move control from one group to another.
5. Until now this was a nice idea but impossible; you lose the high-bandwidth links of the datacenter and the entire training stack was designed for single-participant, centrally controlled development.
6. It is now basically solved. Several major research results from us in ICML, ICLR, NeurIPS + has been implemented and permissionless training with hundreds of real participants and thousands of GPUs. The only remaining question is will this scale to 100b/trillion parameter models and every piece of evidence indicates it it will.
7. Low-bandwidth PP unlocks a massive compute advantage; consumer silicon that is far cheaper per FLOP, couldn't previously be used for training, and scales without buildout or power hookups. Means you can scale to utility very fast which is important.
8. It also unlocks the incentive layer required to make the whole thing self-sustaining: a model split across participants is unextractable, so ownership and revenue can be enforced on an open model for the first time.
9. The result is a setup that is structurally advantaged to closed or openweight; you get a sovereign, independent, uninterruptible, neutral protocol that naturally develops into its own ecosystem.

That's basically the entire Pluralis thesis.

[Pluralis Research @Pluralis]

Pluralis-8B is done, the first pipeline-parallel training over WAN with our new Agora system. We maintained approximately 20% MFU throughout the run, with a completely dynamic pool of devices utilising networking speeds 100-1000x slower than a datacenter setup

[Chamin Hewa Koneputugodage @ChaminHewa]

Our #ICML2026 paper: "𝐅𝐚𝐜𝐭𝐨𝐫𝐞𝐝 𝐆𝐨𝐬𝐬𝐢𝐩 𝐃𝐢𝐋𝐨𝐂𝐨: 𝐑𝐞𝐝𝐮𝐜𝐢𝐧𝐠 𝐁𝐥𝐨𝐜𝐤𝐢𝐧𝐠 𝐂𝐨𝐦𝐦𝐮𝐧𝐢𝐜𝐚𝐭𝐢𝐨𝐧 𝐢𝐧 𝐃𝐢𝐋𝐨𝐂𝐨" from @Pluralis now has a blog post! It is also available on ArXiv, and we'll be presenting it 𝐭𝐨𝐦𝐨𝐫𝐫𝐨𝐰 in the Morning Poster Session (Poster Session 1, Poster #3811).

We introduce a data-parallel training formulation for low-bandwidth (consumer internet) links.

Thread 🧵

[Alexander Long @AlexanderLong]

It's now clear both labs will restrict access to the best modes to a small group of trusted parties. Unlike other technology, not having access fundamentally reduces your value/effectiveness as an individual or company, and you have effectively no recourse. The impact of this is hence very severe. However, there are reasons to be very optimistic - the engineering and research required to coordinate large swathes of independently controlled compute and to construct an independent and a sovereign model supply chain is hitting the tipping point of feasibility at the exact same time the world is waking up to what is effectively an existential threat. Many people are completely unaware of how progressed the research in decentralized training/training in protocols is - things are looking extremely promising. If we were starting from scratch today, the situation would be far worse.

The primary problem these approaches have to address is that compute cannot be pooled for training without compression (as you don't have fast interconnects you have in a datacenter). However, if you have big nodes, and every node keeps a copy of the model, DiLoCo (@Ar_Douillard ) and variants (SparseLoCo from @amir_sarfi , Factored Gossip DiLoCo from @ChaminHewa) solve this problem. SPARTA from @exolabs is also very promising, and these two classes of approaches can also be combined. The common argument is then "ok so everyone training needs multiple 8xB200 nodes which only exist in datacenters that doesn't seem very decentralized". Subspace Networks from @SameeraRamasin1 address this compressing activation and activation gradients, which then allows a large model to be split over participants (pipelined with each holding a layer or sequence of layers). This is the fundamental technology that really allows anything to work; SSNs + SPARTA is whats live in Agora right now with each participant serving a small shard. It allows consumer cards to be productive (of which there are A LOT - far more than people think), and also lets the approach scale to large models.

Even with this level of compression - which unlocks this class of approaches in theory, there remains many other problems. Verification is often considered very difficult - however it's actually easier than the deterministic verification normally required - the models are inherently robust - a large amount of training is actually injecting randomness and pertubations to induce generalisation. Methods such as SENTINEL by @hmdolatabadi can hence be extremely effective here and are specific to large model training - you can't do this for arbritary computation.

There is much more work than what I've listed, and there is also so much work to do - but how can this not be exiting rather than distressing to people - this can work! From an ML research perspective it's incredible - suddenly there is a new substrate to work on - every modelling and arch design decision can be rethought. Growing the networks within swarms actually feels MORE correct to me than in monolithic blocks of datacenter compute. There are scenarios here where you actually decouple productive capacity from the old control structures - that's never happened before - it's never even been close to possible... if you have any ability to work on these area's how could you be spending a second doing anything else.

[Pluralis Research @Pluralis]

We are hosting a workshop during ICML on Protocol Learning to examine the key open challenges in collaborative training across distributed networks.

Talks by @Ana_koloskova @sam_hrvth @hmdolatabadi &amp; more speakers to be announced

Register here: https://t.co/slAqXcuxjP

[Pluralis Research @Pluralis]

The 8B model currently training on Agora is 350B tokens in and continuing to converge. The top level metrics and evals look almost exactly like a centralised run. But;
- 133 external contributors total bringing 4090's, 5090's, L40S/RTX 6000 and RTX 6000 Pros. These are cards that people actually own - there are no H100, B200's etc.
- The max number of nodes the system can support (104) was filled almost immediately. The authorization layer is receiving approximately 100 requests/minute to join.
- The total tokens/per second processed moves directly with amount of compute in the swarm, with Agora constantly optimising to make most efficient use of what hardware is present.
- MFU is approximately 20%, TPS is 170k tok/s. There are near constant communication failures which Agora is completely absorbing without slowdown.
- The system is effectively on auto-pilot, requiring very little intervention from us. Bad nodes are purged immediately before training is affected and new nodes take their place.

[Alexander Long @AlexanderLong]

I would like to make a few brief points;
- Opensource ai is not the same thing as opensource software. The models cost tens to hundreds of millions to make. This is not gonna be a volunteer effort from people doing stuff after work for free.
- the second you release a weight set, you lose any ability to make money serving your own model and recoup the training cost. This very simple property means open-weights is unsustainable.
- the things you ACTUALLY want from opensource ai is: transparent behaviour, dispersed ownership and control, a guarantee of access, the ability to build on it/modify it, and privacy.

protocol learning gets you all 4 and is the only alternative to closed models that makes any kind of sense.

By protocol learning I mean a very specific, novel thing; collaborative training and development of the models without anyone ever being able to see the complete weight set.

[Riccardo Patana @RiccardoPatana]

The path we're on should now be evident to everyone.
Access to AI will be used as instrument of coercion by the USA and Chinese governments.
We do need a third path, #ProtocolLearning, #DecentralizedAI for a Collective, Trustless, Sovereign AI.

[Alexander Long @AlexanderLong]

Seems like a good time to repost this. tl;dr extreme power asymmetries emerge if models stay closed (we're seeing start of this now), and open-weight releases are not sustainable and we cannot rely on them. But there is a third path that makes sense.

[Hadi M. Dolatabadi @hmdolatabadi]

We’re live! Glad to have been part of the Agora effort within @Pluralis. We’ve come a long way since Node0, making the infra layer more fault-tolerant while speeding up training by 10x with less compute.

At the core, we’re using SSNs plus AsyncSparta to enable LLM pretraining over the internet. Under the hood, however, we had to tackle many technical challenges caused by the stochastic nature of the underlying hardware.

To make this happen, we had to resolve conflicts that arise when combining PP training with stochastic hardware: enabling each replica to join/hold its own set of weights without derailing the run, keeping AR communication lightweight, and not wasting bandwidth on parts of the model that don’t need it after explicitly baking PP compression into the architecture itself.

Glad to have contributed meaningfully to building this, and honestly, super excited for what the future holds. There are many technical challenges here that haven’t really surfaced anywhere before. A lot of them come from the uncharted territory of decentralized training; problems that big labs haven’t had to resolve because they’ve had access to massive amounts of datacenter compute.

[Pluralis Research @Pluralis]

The first Protocol Learning workshop in Rio today!

A new field is emerging: decentralized training of foundation models. Because the world’s intelligence belongs in open, collectively owned systems.

Thanks for joining @oguzer90 @niclane7 @m_ryabinin @namhoonlee09 @itsmaddox_j

[Hadi M. Dolatabadi @hmdolatabadi]

Exactly. The issue is not that labs are becoming "less open". The issue is that open-weight foundation models are not financially sustainable.

If we want open AI to win, we need architectures and incentive systems that make it viable: distributed compute, shared ownership, and shared revenue.

That is the idea behind protocol-native, unextractable pipeline-parallel models we've been cooking @Pluralis.

[James @tsnewnami_]

Again, the companies releasing open weight models are at an economic disadvantage. The dynamic here feels fragile: model training dominates CapEx for these companies but other companies capture this margin while remaining dependent on them.

[Aakash Gupta @aakashgupta]

Cursor is raising at a $50 billion valuation on the claim that its “in-house models generate more code than almost any other LLMs in the world.” Less than 24 hours after launching Composer 2, a developer found the model ID in the API response: kimi-k2p5-rl-0317-s515-fast.

That’s Moonshot AI’s Kimi K2.5 with reinforcement learning appended. A developer named Fynn was testing Cursor’s OpenAI-compatible base URL when the identifier leaked through the response headers. Moonshot’s head of pretraining, Yulun Du, confirmed on X that the tokenizer is identical to Kimi’s and questioned Cursor’s license compliance. Two other Moonshot employees posted confirmations. All three posts have since been deleted.

This is the second time. When Cursor launched Composer 1 in October 2025, users across multiple countries reported the model spontaneously switching its inner monologue to Chinese mid-session. Kenneth Auchenberg, a partner at Alley Corp, posted a screenshot calling it a smoking gun. KR-Asia and 36Kr confirmed both Cursor and Windsurf were running fine-tuned Chinese open-weight models underneath. Cursor never disclosed what Composer 1 was built on. They shipped Composer 1.5 in February and moved on.

The pattern: take a Chinese open-weight model, run RL on coding tasks, ship it as a proprietary breakthrough, publish a cost-performance chart comparing yourself against Opus 4.6 and GPT-5.4 without disclosing that your base model was free, then raise another round.

That chart from the Composer 2 announcement deserves its own paragraph. Cursor plotted Composer 2 against frontier models on a price-vs-quality axis to argue they’d hit a superior tradeoff. What the chart doesn’t show is that Anthropic and OpenAI trained their models from scratch. Cursor took an open-weight model that Moonshot spent hundreds of millions developing, ran RL on top, and presented the output as evidence of in-house research. That’s margin arbitrage on someone else’s R&D dressed up as a benchmark slide.

The license makes this more than an attribution oversight. Kimi K2.5 ships under a Modified MIT License with one clause designed for exactly this scenario: if your product exceeds $20 million in monthly revenue, you must prominently display “Kimi K2.5” on the user interface. Cursor’s ARR crossed $2 billion in February. That’s roughly $167 million per month, 8x the threshold. The clause covers derivative works explicitly.

Cursor is valued at $29.3 billion and raising at $50 billion. Moonshot’s last reported valuation was $4.3 billion. The company worth 12x more took the smaller company’s model and shipped it as proprietary technology to justify a valuation built on the frontier lab narrative.

Three Composer releases in five months. Composer 1 caught speaking Chinese. Composer 2 caught with a Kimi model ID in the API. A P0 incident this year. And a benchmark chart that compares an RL fine-tune against models requiring billions in training compute without disclosing the base was free.

The question for investors in the $50 billion round: what exactly are you buying? A VS Code fork with strong distribution, or a frontier research lab? The model ID in the API answers that.

If Moonshot doesn’t enforce this license against a company generating $2 billion annually from a derivative of their model, the attribution clause becomes decoration for every future open-weight release. Every AI lab watching this is running the same math: why open-source your model if companies with better distribution can strip attribution, call it proprietary, and raise at 12x your valuation?

kimi-k2p5-rl-0317-s515-fast is the most expensive model ID leak in the history of AI licensing.