gil avraham

AI Researcher. Robotics enthusiast. @MetacogAI

[Metacognition @MetacogAI]

Hello world, we’re here! While today’s LLMs are primarily text prediction machines that are unreliable and have short attention spans, Metacognition is building a new architecture, which includes memory that persists, reasoning that adapts, and trust you can inspect.

Read more:
https://t.co/AGPvvnQope
#DeepTechWeek #AI #AustralianTech #Metacognition

[Pluralis Research @Pluralis]

60 page tech report detailing Agora and the run it just powered now on Arxiv. Spans many different fields, from communication-efficient model parallelism, asynchronous optimization, to fault-tolerant systems design.

[gil avraham @GilA1337]

@bcherny Very eloquent

[gil avraham @GilA1337]

@AISystemGuy @jbrukh @Pluralis @MacrocosmosAI - Failure recovery, the events column on the left + the Averaging telemetry. Though I don't think these couple of metrics aren't doing justice to how resilient this system is. Taking into account the frequency of these events and how stable the throughput is, it's quite amazing.

[gil avraham @GilA1337]

@AISystemGuy @jbrukh @Pluralis @MacrocosmosAI Under the same bandwidth regime Agora is 11x over the state-of-the-art publicly available framework Megatron-LM.
See:
- MFU Overview, "Mean MFU" is hovering around 20%.
- Quality-adjusted wall-clock, in the Overview see the wall-clock "Elapsed" toggle and the Evals

[Pluralis Research @Pluralis]

Today we're releasing Agora: the first ever pretraining stack that allows non-collocated consumer GPUs to be competitive with centralized clusters

Agora is 15x faster than Megatron-LM in this setting and is only 1.5x less efficient in terms of tokens per unit compute than TorchTitan on H100s, despite running on devices that have no NVLink or InfiniBand support.

[Pluralis Research @Pluralis]

Factored Gossip DiLoCo (by @ChaminHewa) has been accepted to ICML 2026. It removes the all-reduce required to compute the outer-optimiser step, improving robustness to failed nodes. In a collective training setting, this allows nodes to leave arbritarily with minimal impact.

[Alexander Long @AlexanderLong]

insane sequence of statements buried in an Alibaba tech report

[Sameera Ramasinghe @SameeraRamasin1]

Glad to share that our paper “Taming Curvature: Architecture Warm-up for Stable Transformer Training” has been accepted to ICLR 2026. This is a recent work we’ve been developing at Pluralis Research.

Billion-parameter Transformer training is still surprisingly brittle: transient loss spikes and occasional divergence can burn days (millions of dollars worth) of compute. These problems are more pronounced in decentralized training where the innate stochasticity of the system amplifies these issues.

A really interesting lens to look at this problem is the Edge of Stability (EoS) theory. That is, stability is governed by the curvature (top Hessian eigenvalue) of the model. But so far, tracking curvature online has been too expensive to be practical for models at this scale. In this paper, we propose a cheaper algorithm to compute the curvature of the networks, built on Hessian–vector products (HVPs), which dramatically simplifies the cost.

We then use this curvature-tracking signal as a control knob during training by dynamically growing the model’s effective depth. Intuitively, begin with a shallower network (typically lower curvature → more stable dynamics), then progressively unlock depth on a schedule so the run doesn't cross the stable boundary.

This has especially attractive implications in decentralized training: as new participants join, you can grow the network mid-run in a principled way, without destabilizing optimization. We call this "architecture warm-up", and in our experiments it remains consistently stable precisely in regimes where common stabilization baselines become spiky or diverge.

There could be multiple other advantages of this: if we can track the effective curvature online, we can start thinking of training recipes as closed-loop control systems. For instance, instead of fixed heuristics, adapt depth / LR / preconditioning to keep the system inside a stability envelope.

Have a read if interested (preprint): https://t.co/1vDDFbzdJC

[Thalaiyasingam Ajanthan @tha_ajanthan]

Amazing week at the KAUST Workshop on Distributed Training. I learned a lot from each talk and also had the pleasure of presenting our recent work @Pluralis. Explored the Red Sea and the Desert too!
Many thanks to @peter_richtarik and the organizers for the wonderful hospitality!

[Pluralis Research @Pluralis]

Today, you need huge amounts of capital to train a large base model, meaning a handful of corporations control this layer. Creation of the base models in decentralized protocols is an alternative to this, democratizing access - but until recently there have been major technical hurdles preventing feasibility.

Today we are launching our technical blog with the first post covering our work on Asynchronous Pipeline Parallel. Async training is a key technical pillar that must be addressed for Protocol Learning and community created and owned models to be practical.

This work removes pipeline bubbles which are very damaging in the decentralized case, especially so when heterogenous hardware is present.

https://t.co/0QzAoKeVd7

[Alexander Long @AlexanderLong]

10GW of productive compute would cost on the order of $100B to produce today. But we know Decentralized networks can accumulate enormous computational power if incentivized correctly. Great report from epoch where they discuss how linking datacenters could train a 72T param model. My view is this gets even more interesting when you can use non-datacenter compute, low-power sources and you have a truly open network. I wanna know what a 72T model does.

[Pluralis Research @Pluralis]

Node-0-7.5B has concluded training after 36B tokens over 3 weeks. This was (as far as we know) the first multi-party pretrain that ran in an open, dynamic, heterogeneous GPU swarm. The loss marched down to 2.75 on fineweb-edu and was continuing to fall, perfectly inline with a model of this size and training duration. In total there were over 300 active participants, 78 of which contributed more than an exoFLOP each to the run. This was a model-parallel pre-train where each device held one block of the larger transformer. Contributions followed a classic power-law, with a total of 1642 individual GPU's contributing from over 198 cities around the globe, many of which were small consumer cards not in datacenters. A massive thankyou to everyone who participated - initial summary and high level metrics below.

[Greg Osuri 🇺🇸 @gregosuri]

Training AI on globally distributed heterogeneous compute is the key missing piece for DeAI to reach the masses.

It has now arrived and works flawlessly on Akash.

This is a new era for the network, a moment I've been waiting for 5 years.

[Pluralis Research @Pluralis]

NeurIPS results are out; Pluralis has three accepted papers. The first is the core research result which currently is powering our 7.5B open pretraining run node0. The second two are also major results which we will integrate into subsequent runs. This is with a small team of <10 people, done while building node0.

1) Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism: Scaling models has led to significant advancements in deep learning, but training these models in decentralized settings remains challenging due to communication bottlenecks. While existing compression techniques are effective in data-parallel, they do not extend to model parallelism. Unlike data-parallel training, where weight gradients are exchanged, model-parallel requires compressing activations and activation gradients as they propagate through layers, accumulating compression errors. We propose a novel compression algorithm that compresses both forward and backward passes, enabling up to 99% compression with no convergence degradation with negligible memory/compute overhead. By leveraging a recursive structure in transformer networks, we predefine a low-dimensional subspace to confine the activations and gradients, allowing full reconstruction in subsequent layers. Our method achieves up to 100x improvement in communication efficiency and enables training billion-parameter-scale models over low-end GPUs connected via consumer-grade internet speeds as low as 80Mbps, matching the convergence of centralized datacenter systems with 100Gbps connections with model parallel.

2) Mixtures of Subspaces for Bandwidth Efficient Context Parallel Training: Pretraining language models with extended context windows enhances their ability to leverage rich information during generation. Existing methods split input sequences into chunks, broadcast them across multiple devices, and compute attention block by block which incurs significant communication overhead. While feasible in high-speed clusters, these methods are impractical for decentralized training over low-bandwidth connections. We propose a compression method for communication-efficient context parallelism in decentralized settings, achieving a remarkable compression rate of over 95% with negligible overhead and no loss in convergence. Our key insight is to exploit the intrinsic low-rank structure of activation outputs by dynamically constraining them to learned mixtures of subspaces via efficient reparameterizations. We demonstrate scaling billion-parameter decentralized models to context lengths exceeding 100K tokens on networks as slow as 300Mbps, matching the wall-clock convergence speed of centralized models on 100Gbps interconnects.

3) Unextractable Protocol Models: Collaborative Training and Inference without Weight Materialization:
We consider a decentralized training setup in which the participants collaboratively train and serve a large neural network, and where each participant only processes a subset of the model. In this setup, we explore the possibility of unmaterializable weights, where a full weight set is never available to any one participant. We introduce Unextractable Protocol Models (UPMs): a training and inference framework that leverages the sharded model setup to ensure model shards (ie, subsets) held by participants are incompatible at different time steps. UPMs periodically inject time-varying, random, invertible transforms at participant boundaries; preserving the overall network function yet rendering cross-time assemblies incoherent. On Qwen-2.5-0.5B and Llama-3.2-1B, 10 000 transforms leave FP 32 perplexity unchanged (PPL Jensen–Shannon drift). Applying a transform every 30s adds 3% latency, 0.1% bandwidth, and 10% GPU-memory overhead at inference, while training overhead falls to 1.6% time and < 1% memory. We consider several attacks, showing that the requirements of direct attacks are impractical and easy to defend against, and that gradient-based fine-tuning of stitched partitions consumes of the tokens required to train from scratch. By enabling models to be collaboratively trained yet not extracted, UPMs make it practical to embed programmatic incentive mechanisms in community-driven decentralized training.

[Tim Khoury @tkhoury]

It really is that easy to get started!

[Pluralis Research @Pluralis]

Node-0-7.5B is live. It is a permissionless, multi-participant, model-parallel pretraining run over the open internet. Anyone with a 16GB+ GPU can join. Node-0 allows participants to collaboratively train a model far larger than could be done as individuals.

[Alexander Long @AlexanderLong]

This is a functioning system. https://t.co/TmGcsYZ8Vr to join.

[Pluralis Research @Pluralis]

We also made a @3blue1brown style video explaining the paper at a high level.

[Pluralis Research @Pluralis]

We've reached a major milestone in fully decentralized training: for the first time, we've demonstrated that a large language model can be split and trained across consumer devices connected over the internet - with no loss in speed or performance.