β BENCHMARKED: IOTA Decide 4B v0.4, an open-weight decision model from @MacrocosmosAI, running on my 64GB M5 Pro Mac mini
You hand it a state (a ticket, a policy, an agent trace) plus questions and one forward pass returns a probability for every answer. With their bundled loader on the Mac's GPU in full bf16, it loaded in 12.8 seconds and peaked at 10.7 GB.
Short ticket, about 130 tokens:
- yes/no: 88 ms
- 3-way choice: 105 ms
- 3-level score: 88 ms
That's about 11 decisions a second on a desktop Mac.
Their own example, a refund asked for 45 days after purchase on a 30-day policy, answered all 3 questions in 277 ms with the same answers on every repeat:
- eligible: 2.2% yes
- route: refunds team, 86%
- urgency: Low, 63%
Longer states take about 0.6 s per decision at 1k tokens and 2.6 to 3.0 s at 4k.
These numbers are the floor. On Mac its linear-attention layers still run on PyTorch's generic kernels, so an optimized Metal path should make it faster. Their 33 ms figure is on a dedicated GPU.
If you're building agents, about 90 ms a decision is fast enough to check a policy before an agent acts or route tickets on your own machine, with no per-call bill. I didn't benchmark accuracy, but it gave the same answers on their refund example every run.
Happy to have trained this model with @v_truba. Trained across 16 disaggregated single GPUs using our IotaSDK we created a leaderboard jev model in under one week of dev time. Thinking about offering jev finetuning as a service with custom data in the near future.
Introducing IOTA Decide 4B, our first public model.
Pareto optimal. Built different.
Trained across globally distributed GPUs with our SDK.
Open weights, Apache-2.0.
π€ https://t.co/cFlqLmcnop
Congrats to our colleagues at @Pluralis for finishing their Agora-8B run, although we of course launched the first pipeline parallel training run over the internet over one year ago now on @IOTA_SN9.
Regardless, this represents a great win for decentralised AI.
Two challenges remain. Each Mac holds the whole model, so we can only train models that fit on one Mac, and the trainer sits in a single cluster.
Agora gets past both. It just pretrained Pluralis-8B across hundreds of consumer GPUs, the first pipeline-parallel training run over the open internet as far as we can tell.
After 1 year and almost 1000 runs we are proud to finally present the largest, fastest and most resilient DPP LLM training system to date. This is the cheapest possible way to pre-train an LLM - both compared to centralized as well as other decentralized systems.
A 100B model is trained using single A100s at a cost of 1.4$/hour/participant, trainable on spot instances. For comparison, the previous large decentralized training project (SN3) required 8xB200 @ 50$/hour. By allowing the use of spot instances whilst preserving up to ~80% of the speed of centralized, we are able to train at the lowest cost/FLOP ever demonstrated in both centralised and decentralised AI.
Once we finish the various proof points of Project Orion, we will begin first trials of working with clients and will start to go down the revenue-generation path.
Today, we are launching the first stage of Project Orion.
Our early pre-training run of Orion-100B achieves upward of 65% of data-center training efficiency on hardware costing a fraction of the price.
Orion-100B is the first proof point for a simple idea: that underutilized compute around the world can be turned into frontier training capacity.
We believe that this work presents, for the first time, an economically compelling case for training large models using distributed approaches.
I'd like to think that usually past performance is a predictor of future performance - in all likelihood the team that's been running SN1, SN9 & SN13 from the inception of bittensor subnets and exited SN25 & SN39 gracefully without negative impact to investors is one of the least likely teams to rugpull.
The entire reason we use one wallet was to allow >256 miners and bittensor simply imposes various limitations that occasionally make it useful to think out of the box.
We're currently actually working on a liquid compute stack that allows us to use the cheapest GPUs across all providers. Chute alone doesn't have enough GPUs to fulfill our requirements so we're also going to have to hook up to paperspace/lyceum/targon etc., but this is in the works. Not necessarily as a full partnership initially but more as a client/provider model, but as we're starting to productise and are doing more serious training runs with external partners we have to be very conscious of data ownership.
Certain companies want full control over their GPU locations, have assurances on data integrity etc., so we have to work out on a case-by-case basis whether attestation meets the necessary security standards.
We're currently actually working on a liquid compute stack that allows us to use the cheapest GPUs across all providers. Chute alone doesn't have enough GPUs to fulfill our requirements so we're also going to have to hook up to paperspace/lyceum/targon etc., but this is in the works. Not necessarily as a full partnership initially but more as a client/provider model, but as we're starting to productise and are doing more serious training runs with external partners we have to be very conscious of data ownership.
Certain companies want full control over their GPU locations, have assurances on data integrity etc., so we have to work out on a case-by-case basis whether attestation meets the necessary security standards.
We're currently actually working on a liquid compute stack that allows us to use the cheapest GPUs across all providers. Chute alone doesn't have enough GPUs to fulfill our requirements so we're also going to have to hook up to paperspace/lyceum/targon etc., but this is in the works. Not necessarily as a full partnership initially but more as a client/provider model, but as we're starting to productise and are doing more serious training runs with external partners we have to be very conscious of data ownership.
Certain companies want full control over their GPU locations, have assurances on data integrity etc., so we have to work out on a case-by-case basis whether attestation meets the necessary security standards.
Great to see more serious capital flowing into IOTA. With 2 of the major TAO funds being invested, IOTA has launched increasingly ambitious runs over the last few weeks with seriously impressive results to be published at proof of talk.
Onwards to becoming the world's most cost-efficient way of pretraining a model.
It's amazing to see the level of community engagement - it's a really cool problem to have if your community suddenly starts building better dashboards than we do ourselves
Train at Home now has a beautiful web UI for managing multiple miners. View your machines, watch your earnings, set up alerts and even remotely deploy.
Coolest part is that we didn't build this, one of our cracked community members cooked it to solve his own problem. @trainathome_ai literally beat us to our own roadmap. Accelerate!
Check it out https://t.co/U6LJgdtwt8
The first Protocol Learning workshop in Rio today!
A new field is emerging: decentralized training of foundation models. Because the worldβs intelligence belongs in open, collectively owned systems.
Thanks for joining @oguzer90@niclane7@m_ryabinin@namhoonlee09@itsmaddox_j