Thanks @joelniklaus for burning this compute for all of us and sharing the learnings openly. Cost per task is a critical metric for our team so your results make us feel warm and fuzzy
I burned a shitload of compute so you don't have to. Releasing the Harness Leaderboard.
Which harness should you run your open model in? I ran 28,849 trials to find out.
The leaderboard compares nine coding-agent harnesses on open models. In each comparison only the harness changes: the model, provider and reasoning effort stay fixed, and every task gets three attempts.
The trials ran GLM-5.3-Flash, DeepSeek-V4.1-Flash and Flint on Terminal-Bench 2.1 and DeepSWE, for $4,746 of inference.
Averaged over the three models, mini-swe-agent (@KLieret), a bash-only agent, solves 51.4% of tasks and @openclaw 27.3%. Four others are statistically tied with mini-swe-agent.
One of them is Pi (@pidotdev), at 49.4% for $0.29 per solved task, about 45% less than mini-swe-agent's $0.53.
The best harness depends on the model: mini-swe-agent on GLM-5.3-Flash, Hermes Agent (@NousResearch) on DeepSeek-V4.1-Flash and Kimi Code (@Kimi_Moonshot) on Flint.
@opencode costs $3.35 per solved task, 11.7 times as much as Pi.
See Harness Hub's Harness Lens for what the nine harnesses do differently. It documents the prompts, tools and compaction of 34 coding-agent harnesses and links every claim to its source.
Leaderboard: https://t.co/AiWiQtBOz5
Harness Lens: https://t.co/I8jbcWa9JH
people of pi (durable). i did a round of performance optimizations in pi-durable.
quick primer. every step of an agent run is saved to a db before the agent moves on. 1 save = 1 commit. thats what makes it durable, crash mid run, resume from the last commit. ~12 commits per run
4 things worth building with @pidotdev's Pi Durable + @CloudflareDev Durable Objects:
#1 Durable agents for high-stakes workflows
Pi Durable can identify interrupted operations and avoid automatically replaying unsafe calls. Imagine an agent refunding a customer, then crashing before it gets confirmation. You don't want it refunding the customer twice.
Pi Durable checkpoints each step, recording the intent to execute a tool call and whether it completed. After a crash, it knows which operations were interrupted. Unsafe calls aren't blindly retried, so your app can check whether the payment went through before proceeding.
#2 Shared team agents
Pi Durable + Cloudflare Durable Objects are a great foundation for multiplayer agentic coding.
A Durable Object holds the persistent agent state. Multiple clients can connect to the same session, incoming messages are queued or used to steer the agent, and updates are streamed to everyone.
Imagine an agent picking up a GitHub issue and working on a PR while teammates watch or steer the same conversation from their phones, browsers, or terminals. @pidotdev's shared-chat tutorial demonstrates this.
#3 Agents that run anywhere
@badlogicgames built a Pi Durable Android app that runs the agent harness and stores its state locally on his phone, while model calls happen remotely.
Pi Durable separates agent state from execution, with pluggable storage and execution backends. The harness can run anywhere there's a compatible JavaScript runtime, and its tools can execute on another machine. Your durable agent doesn't need an always-running terminal or cloud server just to preserve the agent's state, the harness can run locally, on a phone, or in the cloud.
#4 Build new tools on top of agent state
This is the one we're most excited about.
Pi Durable persists structured tasks, checkpoints, and interrupted operations. That means you can inspect an agent's execution state without asking the model to reconstruct what happened.
We experimented with putting Pi Durable inside a Cloudflare Durable Object and visualizing its stored state without waking the agent.
Imagine using that data to build:
- Debugging dashboards showing exactly where an agent crashed.
- Reliability metrics showing which tool calls repeatedly fail or need recovery.
- Multiplayer dashboards showing pending tasks, who steered an agent, and when.
The interesting part is that durable execution state becomes a new primitive you can build tools on top of, not just something used for crash recovery.
00:00 What is Pi Durable?
07:06 Idea 1: Durability for high-stake workflows
13:16 How to build with Pi Durable
16:31 Durable storage: local vs. cloud
19:46 Pi Durable + Cloudflare Durable Objects
32:20 Idea 2: Multiplayer agents with shared state
35:01 Idea 3: Running the harness on a phone
37:45 Idea 4: Inspecting persistent agent state
🧵👇 Links to YouTube, Apple Podcasts, and Spotify below.
@Recursive_SI shipped its first model, Flint! Exciting to see they did extensive RL training on the Pi harness, so it works well out of the box.
40B MoE (2.5B active) for fast local coding. Apache 2.0.
Available now for free in Pi via Radius.
Introducing Flint, @Recursive_SI’s first model: a fast local coding model built with our automated research system, the Eureka Machine.
• Best-in-class fast local coding
• Up to 2x faster than comparable models: 125+ tokens/sec on a MacBook Pro, 150+ on RTX Spark
• Built from scratch in under 4 months
Private, unmetered intelligence on your own machine. Open weights available now on @huggingface, run it locally on your favorite framework or try it for free on @opencode or @pidotdev.
Introducing 𝚍𝚞𝚛𝚊𝚋𝚕𝚎-𝚑𝚊𝚛𝚗𝚎𝚜𝚜. Run Pi, OpenCode, Codex, Claude Code, eve, fx, or your own harness durably on Vercel.
With 𝚍𝚞𝚛𝚊𝚋𝚕𝚎-𝚑𝚊𝚛𝚗𝚎𝚜𝚜, agents automatically resume work after failures and run for hours, days, or months. https://t.co/XkqmULA2ng
i had opus 5.5 distill this 30 min lecture about pi-durable into a 7 min vid with my voice.
> qwen3 tts could do a better job cloning my voice
> the video looks great. i'm working on industrializing these kind of explanation videos.
good starting point, but i still recommend watching the full lecture for a more detailed explainer.
For your weekend reading: Build with Pi Durable on Cloudflare, a walkthrough of how to create a multiplayer chat agent, including subagents, on Durable Objects.
https://t.co/Y7EBwsV52x
@niandubay@earendilworks Thanks for this. We’re a business so inherently we mix tech and business decisions. But we operate under the principle that what is good for the tech is good for the business and making decisions that favor the business over the tech is short term greedy, long run stupid