Yan Zuo

Machine Learner

[Akash Network @akashnet]

Thrilled to report Decentralized Training has arrived on Akash.

Training AI on globally distributed heterogeneous compute allows you to use GPUs from anywhere in the world to train AI, which is key to scaling AI and solving the energy crisis created by Big Cloud.

We at Akash are incredibly excited about Pluralis Research for releasing the first-ever permissionless training framework to achieve this goal — it works flawlessly on Akash.

Needless to say, this opens up new demand opportunities for low-powered GPUs on the network.

SDL to deploy your own Pluralis node and guide coming soon!

[Greg Osuri 🇺🇸 @gregosuri]

Training AI on globally distributed heterogeneous compute is the key missing piece for DeAI to reach the masses.

It has now arrived and works flawlessly on Akash.

This is a new era for the network, a moment I've been waiting for 5 years.

[Alexander Long @AlexanderLong]

This is kinda crazy

[Thalaiyasingam Ajanthan @tha_ajanthan]

Happy to share 4 papers (3 posters, 1 spotlight) accepted at NeurIPS'25. Congrats to all the co-authors. Three of them on realising scalable and economically sustainable decentralised training @PluralisHQ.
Details below:

[Pluralis Research @Pluralis]

NeurIPS results are out; Pluralis has three accepted papers. The first is the core research result which currently is powering our 7.5B open pretraining run node0. The second two are also major results which we will integrate into subsequent runs. This is with a small team of <10 people, done while building node0.

1) Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism: Scaling models has led to significant advancements in deep learning, but training these models in decentralized settings remains challenging due to communication bottlenecks. While existing compression techniques are effective in data-parallel, they do not extend to model parallelism. Unlike data-parallel training, where weight gradients are exchanged, model-parallel requires compressing activations and activation gradients as they propagate through layers, accumulating compression errors. We propose a novel compression algorithm that compresses both forward and backward passes, enabling up to 99% compression with no convergence degradation with negligible memory/compute overhead. By leveraging a recursive structure in transformer networks, we predefine a low-dimensional subspace to confine the activations and gradients, allowing full reconstruction in subsequent layers. Our method achieves up to 100x improvement in communication efficiency and enables training billion-parameter-scale models over low-end GPUs connected via consumer-grade internet speeds as low as 80Mbps, matching the convergence of centralized datacenter systems with 100Gbps connections with model parallel.

2) Mixtures of Subspaces for Bandwidth Efficient Context Parallel Training: Pretraining language models with extended context windows enhances their ability to leverage rich information during generation. Existing methods split input sequences into chunks, broadcast them across multiple devices, and compute attention block by block which incurs significant communication overhead. While feasible in high-speed clusters, these methods are impractical for decentralized training over low-bandwidth connections. We propose a compression method for communication-efficient context parallelism in decentralized settings, achieving a remarkable compression rate of over 95% with negligible overhead and no loss in convergence. Our key insight is to exploit the intrinsic low-rank structure of activation outputs by dynamically constraining them to learned mixtures of subspaces via efficient reparameterizations. We demonstrate scaling billion-parameter decentralized models to context lengths exceeding 100K tokens on networks as slow as 300Mbps, matching the wall-clock convergence speed of centralized models on 100Gbps interconnects.

3) Unextractable Protocol Models: Collaborative Training and Inference without Weight Materialization:
We consider a decentralized training setup in which the participants collaboratively train and serve a large neural network, and where each participant only processes a subset of the model. In this setup, we explore the possibility of unmaterializable weights, where a full weight set is never available to any one participant. We introduce Unextractable Protocol Models (UPMs): a training and inference framework that leverages the sharded model setup to ensure model shards (ie, subsets) held by participants are incompatible at different time steps. UPMs periodically inject time-varying, random, invertible transforms at participant boundaries; preserving the overall network function yet rendering cross-time assemblies incoherent. On Qwen-2.5-0.5B and Llama-3.2-1B, 10 000 transforms leave FP 32 perplexity unchanged (PPL Jensen–Shannon drift). Applying a transform every 30s adds 3% latency, 0.1% bandwidth, and 10% GPU-memory overhead at inference, while training overhead falls to 1.6% time and < 1% memory. We consider several attacks, showing that the requirements of direct attacks are impractical and easy to defend against, and that gradient-based fine-tuning of stitched partitions consumes of the tokens required to train from scratch. By enabling models to be collaboratively trained yet not extracted, UPMs make it practical to embed programmatic incentive mechanisms in community-driven decentralized training.

[Jake Brukhman @jbrukh]

You can learn more about this permissionless model parallel training run in the FAQ at https://t.co/9uzT0nM4WU.

[ᴅᴀᴠɪᴅ_ᴅꜰᴏʀᴛʜ ✪ @David_dforth]

My Pluralis Node-0-7.5B is live

This is a permissionless, multi-participant, model-parallel pretraining run over the open internet

If you have a GPU with 16GB+ VRAM, you can join and run a node.

No CPU option here it requires a GPU

Participants are collaboratively training a model much larger than any single person could train alone.

Guide: https://t.co/DJUhdDUgDA (click Join to start)

Currently ~265 active node runners vs. Gensyn’s 20k+ runners.

[Hadi M. Dolatabadi @hmdolatabadi]

No RDMA 100GB+, no problem! @PluralisHQ, we do LLM training with &lt;30 Mbps comms :)

Join the Node0 run: https://t.co/g9UOUGlxKV

[Thalaiyasingam Ajanthan @tha_ajanthan]

5 Continents, 25 Countries, and 80 Cities in day 1. Steady influx of nodes, more than what we anticipated.

[USV @usv]

Pluralis is pulling off something really big: a decentralized AI training run where anyone can contribute compute + earn rewards.

This matters because it shows how AI can be built as an open network, not locked inside a few giant labs.

We’re excited because this aligns with our belief that networks > hierarchies, and that the future of AI should be shared, permissionless, and owned by many.

[Alexander Long @AlexanderLong]

What a wild map to wake up to. External node count growing as quick as join queue can get people in. Loss marching down entirely unaffected.

[Max Ryabinin @m_ryabinin]

With SWARM, we've shown back in 2023 that pipeline-parallel training over the Internet is feasible: https://t.co/m0I8OYqMSb

IMO pure data parallelism isn't enough if we want to train truly big models outside of clusters. To the best of my knowledge, @PluralisHQ are the only ones pursuing this research direction (and making great progress!) right now

[Alexander Long @AlexanderLong]

Max's papers were the major reason I got convinced any of this was possible at all. I cannot emphasise enough how non-accepted/contrarian this whole line of research was. So you think your missing something because all these other smart people dismiss it. But then Max's papers were in at ICML. So you think hmm maybe there is a chance everyone else is wrong here.

[Alexander Long @AlexanderLong]

320 active nodes in &lt;24hrs. Completely heterogeneous GPU mix. Loss still on track.

[Greg Osuri 🇺🇸 @gregosuri]

Pluralis is one of the most exciting projects for me in decentralized AI training.

Very proud they chose to launch permissionless and open-source from day one, unlike many grifters we have seen launch closed source in the name of "protection."

We're adding support to Pluralis as we speak.

[Max Ryabinin @m_ryabinin]

Fault-tolerant pipeline parallelism (SWARM/Petals-style) with compression, scaled to an open pretraining run!

On a personal note, I'm really glad that Hivemind powers a few parts of this project — both for pipelines and for robust data parallelism

[Alexander Long @AlexanderLong]

This is a functioning system. https://t.co/TmGcsYZ8Vr to join.

[Jake Brukhman @jbrukh]

Major congrats to @alexanderlong & @PluralisHQ team, this has been over a year in the making and is literally a Google moment for AI.

This run is a live demonstration of a 7.5B model parallel training process running on commodity hardware.

The future of AI training is in giant permissionless swarms of heterogeneous, unreliable hardware.

[Sam Lehman @SPLehman]

If you had told me just 1yr ago that a permissionless, model-parallel pre-training run would’ve happened in 2025, I would have thought you’re crazy.

This is a really big deal!

[Alexander Long @AlexanderLong]

Join queue saturated fast - the system is letting nodes in as quick as it can. For support/discussion: channel is node-0 on discord. Pluralis team in there helping out.