Sam Dare

Founder / Member of Non Technical Staff @covenant_ai ( @tplr_ai : @basilic_ai : @grail_ai )

[Sam Dare @DistStateAndMe]

A small step for mankind, a massive leap for decentralised training... for agency.

In the space of 9 months, @tplr_ai went from 1.2B -> 72B.

It's never been easy, and has broken everyone on the team multiple times. But I speak for all of us when I say it is the most rewarding thing we have ever done.

We have a fraction of the resources. We don't have the PhDs. But Bittensor shows you it doesn't matter. Innovation happens at the edge. We innovate through scarcity.

The ones who rewrite the rules are never the ones with the most. They're the ones who refuse to accept the limits they were handed.

Bittensor is prophecy. Subnets (@covenant_ai and others) are the tools through which that prophecy is manifested.

Next stop: TRILLIONS.

[templar @tplr_ai]

We just completed the largest decentralised LLM pre-training run in history: Covenant-72B. Permissionless, on Bittensor subnet 3.

72B parameters. ~1.1T tokens. Commodity internet. No centralized cluster. No whitelist. Anyone with GPUs could join or leave freely.

1/n

[Martin Marek @mrtnm]

Tinker's new StableReinforce objective is Reinforce + Score Centering. Here's how I reverse-engineered it 🧵

[λux @novasarc01]

turns out that human slop is much dangerous than ai slop

[Sam Dare @DistStateAndMe]

@HotAisle wen supply ?

[Laude Institute @LaudeInstitute]

Minimal harness + strong model = flexible code review agent. That’s the idea behind RevEngine, Laude’s open-source tool now topping @GitHub’s ReviewBench. @NickAJalbert shares how it works, which model the harness helped most, and what else the team learned in the process: https://t.co/1CSP9QJXMI

[Sam Dare @DistStateAndMe]

@sloppenheimer Moria

[Benjamin Thérien @benjamintherien]

Your neo-lab wants to pre-train a frontier model, but no provider will sell you enough co-located chips?

We got you.

Our new work shows that SOAP & SPlus achieve strong worker scaling to 96+ DiLoCo workers (e.g., 96 providers) by whitening the update’s covariance. 🧵1/9

[Sam Dare @DistStateAndMe]

I believe this school of thought is premised on :

- Chinese models been able to distill the closed source frontier
- Chinese models keep been open.

Rumors are that China is placing restrictions on the size of frontier models Chinese companies can release (sub 1T).

I predict get a pretraining resurgence

[Braden Hancock @bradenjhancock]

Well said. “We got lucky with pretraining. The people who built Wikipedia and Stack Exchange weren’t making training data; they were making things they wanted to exist. But no one is going to inscribe the work of a tax attorney into a reproducible environment for fun.”

As a field we’re essentially rapidly constructing the internet-sized corpus for post-training on the fly.

[Sam Dare @DistStateAndMe]

@romainlanz I see where you are coming from , and currently have the same nits.

Thinking of having them work together in a local worktree , then ship that to main

[Sam Dare @DistStateAndMe]

Kill your priors

[Thorsten Ball @thorstenball]

"My colleague pushed code that doesn't match what I had in mind."

How bad is that really? Consider: it's now much faster and easier to change code.

Years ago a wrong decision could cause days and weeks of refactoring.

Now? Probably less. Adjust.

[marwan @_marwanad]

Scout’s hardest eval: explain the Datadog bill

[Tianyin Xu @tianyin_xu]

Congrats @bthecohen and @datadoghq on releasing #Scout 1.0! I've been waiting for a post-trained SRE model for a while and #Scout makes my day.

Thanks for trying #SREGym !

and thanks for writing a very informative tech report!
https://t.co/8gSmrO3Yla

[Omer Shlomovits @OmerShlomovits]

Worth checking out this model

[Sam Dare @DistStateAndMe]

@esaliya @dMatrix_AI Thanks for the transparency . Does tok/s stay the same at 64?

[Sam Dare @DistStateAndMe]

@willcb Delta weight sync : March 2026

https://t.co/nx96nMGRqY

[Sam Dare @DistStateAndMe]

@qinzytech @brian_armstrong He’s the CEO of a billion dollar company why does his explanation need to be scientifically rigorous? Not like he’s submitting the tweet to NeurIPS

[Sam Dare @DistStateAndMe]

I think they operate at different ends of the spectrum , exploration v exploitation.

If you wanna rewrite an app, you don’t want your agents trying to figure out how to from first principles .

Here the orchestrator model seems to be the right choice .

However , for research / open ended tasks , I think evolving swarms might be spot on.

This paper seemed pretty interesting on that :

https://t.co/wS9ohB92Pa

[slime @slime_framework]

OpenAI's Navier-Stokes effort used ~10,000 concurrent agents and ~130B output tokens. It made us rethink the scale ahead for RL infrastructure.

slime v0.4.0 is a first step toward greater scale, with four architectural changes:

• Focus: synchronous training for algorithmic exploration and correctness; fully async training for throughput at scale.

• Distributed rollout: spread orchestration across cluster CPUs, with independent generation pools and event loops.

• Persistent storage: introduce straw, our new storage middleware for rollout and training data, providing durable queues and shared tensor storage.

• Independent recovery: restart Megatron while healthy SGLang engines stay running, preserving accepted rollout work.

More on the architecture below 👇

https://t.co/4PR50r1AmR

1/5

[Zyphra @ZyphraAI]

In Mixture of Expert (MoE) models the cost of moving tokens to their experts can dominate total runtime.

At Zyphra research, we use patterns in how tokens are routed to experts to make that communication faster by up to 2.63x on @AMD MI300X GPUs, with the model unchanged.