Sam Dauncey

wandb technical analyst @SID_AI, PhD candidate at ETH Zurich

[Sam Dauncey @SJDauncey]

Tokenization has resisted the bitter lesson for 10 years. Bringing the tokenizer inside the model is the obvious path forward, but the key question is: how do we train this tokenizer?

[Susan Zhang @suchenzang]

everyone getting sniped by the personal drama, but missed the more interesting unanswered question:

can these labs see all your work and scoop you when the stakes are "high enough"?

🤷🏻‍♀️

[Sam Dauncey @SJDauncey]

Prediction: the long context RL will become turn-level PPO with:
- token-defined value function & reasoning (solves the "expensive to train value head" problem)
- privileged information for the value rollout (solves the "discovering the task is harder than expected" problem)

[Sam Dauncey @SJDauncey]

Very cool approach - the 2026 update to the token-defined "P" / "N" value function from Polu & Sutskever's "Generative Language Modelling for Automated Theorem Proving" (2020)

[Chao Qiao @ChaoQiao42]

How do you train an agent when a single rollout takes tens of hours?

Alongside the open-source release of dots3-note Preview, we’re sharing more details about TEMPO: Test-Time-Scaled Value Estimation with Macro-Step Policy Optimization.

Long-horizon agent RL faces two fundamental problems:

• Training signals arrive only when an extremely long trajectory ends
• Final rewards make it difficult to identify which intermediate decisions actually mattered

TEMPO breaks a long trajectory into macro-steps. At each boundary, the same model switches from actor to generative critic: it reviews the interaction history, checks the agent’s hypotheses, reasons about possible futures, and estimates the remaining return.

The agent therefore learns not only how to act, but also how to evaluate itself.

One result surprised us: two branches can receive exactly the same environment reward, while one is trapped by an incorrect understanding of the task and the other has already found a viable direction. A reasoning critic can distinguish these states before the task ends.

On the public ARC-AGI-3 set, TEMPO improves the average Score by 31.6% over the base checkpoint and 20.7% over GRPO, while reaching comparable progress with fewer environment interactions.

TEMPO is still early work, but we believe test-time-scaled self-evaluation may be an important ingredient for training agents on tasks that last hours, days, or eventually much longer.

Technical blog:
https://t.co/ErAowzmDfs

Open-source dots3-note Preview:
https://t.co/OOeoS7KMB8

[Max Rumpf @maxrumpf]

The Multi-Model Future is Here.

That's why we're partnering with Baseten to bring agentic search to every dataset.

People often misinterpret The Bitter Lesson, which is about using general methods, not about creating general models. One of Sutton's examples of a Bitter-Lesson model is AlphaGo, which is the opposite of general (it only plays Go) but was trained using general methods. That the most general method to pretrain a text transformer happened to create a very general model is less a law of nature than a property of the data: the biggest dataset (the internet) also happens to be the most general.

But even the most general language models aren't very good at playing Go. There is value to general models, but this generality comes at the cost of outright performance. What you really want to maximize is leverage on computation: capability per FLOP, whether spent in training or at inference. A better training objective gives you more leverage. A way to use less human data gives you leverage (albeit indirectly). For Go, AlphaZero training has orders of magnitude more leverage than retrofitting a language model.

At SID, we have found ways to get much more leverage for the task of search; incidentally, one of the two tasks Sutton believes scales arbitrarily (yes, he meant search as in MCTS, but we'll take it). We don't know the full extent of our leverage, but SID-1 gives a lower bound: it matches frontier models on retrieval accuracy at roughly 1/500th the inference compute, more than 2 OOMs.

Unlike Go, search is very useful. We’re thrilled to partner with Baseten to bring this utility to everyone. Together we offer agentic search at 20x lower latency and 100x lower cost than a general frontier model, at equal or better accuracy.

We expect many other tasks to go this way. The future will be multi-model not as a compromise to save money, but because specialized models are simply better: faster, cheaper, and more accurate at the thing you actually need.

[will brown @willcb]

MOPD is just git for RL

[Sam Dauncey @SJDauncey]

@DefTiger Yes! As the head of the model is byte-level, it can output “supertokens”

[Sam Dauncey @SJDauncey]

Tokenization has resisted the bitter lesson for 10 years. Bringing the tokenizer inside the model is the obvious path forward, but the key question is: how do we train this tokenizer?

[Sam Dauncey @SJDauncey]

Come find me at ICML in Hall A Poster #4000 to talk!

[Sam Dauncey @SJDauncey]

Leveraging this connection to RL lets us side-step the problem of prior approaches such as dynamic token pooling (Nawrot et al. 2023) and HNet (Hwang et al. 2026) how do you differentiate through sampling and selection with respect to token boundaries?

[Sam Dauncey @SJDauncey]

In fact, we can borrow many techniques from RL, such as time discounting, value functions and group-relative advantages to boost performance.

[Sam Dauncey @SJDauncey]

@maxrumpf When I tried to add privileged information to my value model people called it “insider trading” and “unethical” smh

[Max Rumpf @maxrumpf]

bitter-lessoning multi-agent orchestration is extremely effective. especially as soon as you get out of the "human-legible" communication requirement. more things to come soon.

models have different scaling axes than human intelligence and we should exploit that fact.

[Sumanth Hegde @sumanthrh]

Excited to share some of our work on improving vLLM for RL!

A number of RL frameworks, including SkyRL, use vLLM for inference, and we’ve noticed some common problems:
1. Weight syncing between training and inference is implemented in an ad-hoc fashion and duplicated across frameworks.
2. Asynchronous RL is prone to break at scale, especially in P/D and DPEP deployments.

We’ve been working on improving both!

[Simon Eskildsen @Sirupsen]

we are seeing more and more RL puffin', here's a model specifically trained for it on tpuf from @SID_AI

[Max Rumpf @maxrumpf]

@SID_AI We won't stop until they need to increase their limits.

[Sam Dauncey @SJDauncey]

@turbopuffer @SID_AI Don’t unnecessarily puff your context 🐡

[turbopuffer @turbopuffer]

SID-1 is an agentic search model by @SID_AI

→ 1.9x recall over RAG + rerank
→ 24x faster, 99% cheaper than GPT-5.1

trained using large-scale RL on turbopuffer at 1k+ QPS bursts over 10M+ document corpora across thousands of steps

https://t.co/hqmdPUmdLt