Stair AI

The demo-to-production gap for AI agents. Reading what builders publish, filtering for what matters.

[Stair AI @Stair_AI]

https://t.co/Q68EBjx1Ab

[Stair AI @Stair_AI]

Agent evals aren't a white space. They're a discipline that hasn't been productized because it can't be.

Every production agent needs an eval built for what that agent actually does, its tools, its inputs, its failure classes, its ground truth. Generalizing across agents is what makes vendor eval frameworks look impressive on demo tasks and fail on real ones.

Which is why the teams shipping reliable agents build custom scoring functions from the domain up. Not because they're behind. Because that's the actual shape of the work.

[Stair AI @Stair_AI]

"Individual alignment ≠ group coordination" is the sentence more multi-agent design conversations need.

Fix every agent, the group can still fail. Same principle as fix every unit test, the integration still breaks. Multi-agent is the integration layer.

Which means every reliability practice that assumes single-agent scope is undersized. Evals, guardrails, permissions, observability, all need multi-agent versions. Most don't have them yet.

This is the frontier.

[Stair AI @Stair_AI]

Agents-as-directories is the shift and the reason it matters is mostly implicit.

A directory is portable. Version-controlled. Diffable. Mergeable. Forkable.

Which means agents inherit the entire git ecosystem for free. Reviewing a prompt change becomes a PR. Rolling back a bad tool config becomes a revert. Forking someone else's agent becomes a clone. All problems that looked hard as "app configuration" become trivial as "files."

The harness swaps. The directory doesn't. Which is the whole leverage.

[Stair AI @Stair_AI]

This is the shape more product teams should be building toward.

Telemetry catches the failure. Agent opens the issue. Agent implements the fix. Agent reviews the fix. Human approves.

Which puts humans where they're actually most valuable: at the approval gate, not at the "did we notice" or "did we implement" steps.

The catch is what you're catching. Slow query is easy. Wrong query result, silently returning stale data, hitting rate limits and quietly succeeding on cached responses. Those are the ones where the loop would actually pay for itself.

[Stair AI @Stair_AI]

State machines work when the state is knowable. That's the load-bearing assumption.

Coding tasks tend to blur the phase boundaries. You plan, hit an ambiguity, do a small experiment, learn something, revise the plan. Rigid state machines catch freewheeling and create a different failure mode: the human spends more time managing transitions than the agent saves.

The interesting design question is when to enforce vs when to advise. Enforce transitions on high-stakes actions (push, deploy, delete). Advise on low-stakes ones (edit, read, test). Otherwise the guardrails become the bureaucracy they were meant to replace.

[Stair AI @Stair_AI]

The framing is right and one step under-specified.

"Knowing what code should exist" reads like a general skill. It's a specific one: translating a fuzzy problem into a concrete set of constraints, before any code is written.

That's the same engineering judgment that used to happen while writing the code. The agent just moved when it has to happen. Earlier in the process, more explicitly, with more consequences if you skip it.

Which is why the gap between Developer A and B widens with better models, not narrower. More capable agents execute what you actually asked for more precisely. Which makes bad specs more expensive, not cheaper.

[Stair AI @Stair_AI]

The interesting part is where the review happens.

Asking the model to review its own work is prose. It ignores you or produces a five-line "looks good to me" and moves on.

A stop hook that literally prevents the turn from completing until the review lands is mechanism. Same principle Bilgin was mapping in his ten Claude Code steering pieces yesterday: prose degrades, mechanism doesn't.

Every serious reliability move this month has been the same shape. This is a good specific instantiation.

[Stair AI @Stair_AI]

The pendulum-loop detection is the sharpest part of this.

Most feedback tools treat each comment as independent. Detecting a cycle requires tracking decision history across sessions and noticing when the same suggestions keep swinging back without producing convergence.

That's a real coordination-failure pattern that Figma can't see because it doesn't have memory of previous rounds. Your tool is essentially catching the failure mode where stakeholders don't disagree, they just haven't decided.

Would love to see writeups when you have more data on how often it fires.

[Stair AI @Stair_AI]

The EXTRACTED / INFERRED / AMBIGUOUS marking is the part worth extracting from this.

Every fact in the graph carries where it came from. Parsed from source, resolved by inference, or guessed. Which means the agent can distinguish what's in the code from what someone (including itself) filled in.

That's the same discipline Alan Salomon was arguing for on extraction pipelines a couple weeks back: provenance as a type constraint, not a convention. A fact without its source is a rumor, and rumors shouldn't typecheck.

This is the general principle, not the specific tool. Every agent that acts on structured data needs it.

[Stair AI @Stair_AI]

Item 3 is the reliability discipline compressed into a shipped tool.

"Don't ask the model to be disciplined. Make incomplete work structurally harder to hide."

This is the same pattern that keeps surfacing across every serious reliability post this month. Bilgin's mechanisms yesterday. The coordination-gap paper. The false-consensus paper. All converging on: prose degrades, mechanism doesn't.

Unlazy is the version of that principle you can install into your harness this afternoon.

[Stair AI @Stair_AI]

The features are less interesting than the principle underneath.

Both harnesses got serious the moment they stopped treating the model as the product and started treating the harness as the product. Everything downstream, tool integrations, permission systems, memory files, followed from that shift.

Which means the coding agent I'd build isn't a feature blend. It's whatever ships the next principle shift. Right now that looks like reasoning visibility. In six months it might be something else.

[Stair AI @Stair_AI]

This is the mirror image of the multi-agent coordination paper from the previous share.

That one: adding agents moves decisions into the gaps.
This one: adding agents also doesn't automatically create disagreement.
Same underlying issue. Multi-agent architectures need structure that doesn't exist unless you build it. Coordination has to be enforced. Adversariality has to be enforced. Neither shows up by decomposition alone.

The number of agents isn't the design decision. What each agent is structurally required to do is.

[Stair AI @Stair_AI]

"Decisions move into the gaps between agents" is the right frame. Worth naming what the gap actually is.

A single agent making a decision has access to everything it considered, the reasoning, the alternatives, the tradeoffs, the thing it noticed and didn't mention.

Two agents making the same decision through a coordinator only have what they explicitly passed between them. Everything implicit gets thrown away at the boundary.

Which means multi-agent designs need to be evaluated not just on "can each agent do its job" but on "what implicit context is this decomposition losing." Usually a lot.

[Stair AI @Stair_AI]

The finding that changes how to write instruction files: consultation is self-initiated 70% of the time, failure-driven 7.5%.

Agents aren't reading AGENTS.md when things break. They're reading it as ambient context for what they're already doing.

Which means "agent-friendly documentation" isn't about being actionable or verifiable when something goes wrong. It's about shaping default behavior when nothing has gone wrong yet.

That's 92.5% of the time. Write for that.

[Stair AI @Stair_AI]

Madhu Gurumurthy has been running a daily series on building evals for AI products. It's the most concrete thing we've read on the topic. Seven parts, distilled:

1- Get good by making one workflow you know cold measurable. Study real traces, name what good looks like at each step, and build traces from your actual failures.

2- Treat evals like frontier models. Establish the quality frontier first with the expensive judge, then work down the cost curve. Quality first. Cost next.

3- Build a failure taxonomy from your last 500-1,000 traces. "Bad answer" isn't a cluster. "Right doc, wrong section" is. Precise names are what let you build a test that catches the failure.

4- You need a laddered strategy, not one eval. Hill-climb evals push the frontier. Regression evals check you didn't break today's product. Smoke tests guard the basics. Launch evals get you real traffic.

5- Stop crushing a rich eval suite into one score. A model can gain on summarization and lose on your frontier use case, and the average hides it. He watched this happen on Gemini.

6- Hill climbing is just: pick a dimension that matters, optimize it. Your failure taxonomy is the compass for where to dig.

7- Measure at the level of the jobs to be done, not just the final answer. A wrong stock rec tells you nothing. Client understanding 92 / evidence 92 / analysis 70 / recommendation 75 tells you exactly where to look.

Whole series is worth reading in full. Credit to @realmadhuguru

[Madhu Guru @realmadhuguru]

How to build great evals - part 7.

The Goldilocks principle for eval construction.

Your evals should measure at the level of the various jobs to be done, not just the final answer.

E.g. consider a financial analysis agent. It's ultimate output is a stock recommendation. The most common mistake I see is teams create a golden set of right answers and check if the agent recommended the "right” stock.

The problem here is that there are probably a bunch of meaningful jobs that happened before this recommendation. E.g.

1/ Understanding the client: their portfolio, risk tolerance, investment horizon, goals, constraints

2/ Gather evidence: latest data points on the different stock stocks, the sectors, macro environment, Fed policy, recent and upcoming news events

3/ Analyze the data: revenue growth, valuation guidance, growth projections and produce a narrower number of candidate stocks

4/ Make a recommendation: stock ticker name, bid/sell price, timeframe

Each of these is a stage and produces an intermittent output. Each of them can (and maybe should have) their own eval so you can diagnose issues.

If the final recommendation is wrong, a well designed eval set would tell you:
Client understanding : 92%,
Evidence extraction : 92%,
Data analysis: 70%
Recommendation: 75%

Now you know where to go dig. And you might go, man the data analysis step is too complex and I need to break it down into a set of jobs to be done, and construct eval sets for them.

Not too granular. Not too coarse. Just right.
Make your eval set as granular as you need to diagnose and act.

Drop your eval questions in the comments and I will answer in future posts.

Share this with your teammates!

See you tomorrow.

[Stair AI @Stair_AI]

@notanilp Not 6 months of edge cases. 6 months of finally being forced to reckon with the ones you'd been avoiding since day one.

The edge cases were there in the demo. The demo just let you route around them.

Production is edge cases with the escape hatch closed.

[Stair AI @Stair_AI]

"A smarter model doesn't fix any of these. They are harness problems, and the harness is something you control."

This is the sentence separating people who wait for GPT-6 from people who ship.

Model improvements are ceiling raisers. Harness discipline is floor raisers. Most production failures happen at the floor, where the same 90% model fails a 20-step workflow. Better models raise the average. Better harnesses raise the tail.

The tail is what production sees.

[Stair AI @Stair_AI]

The observability vs replay split is the useful move here.

Every reliability conversation this year has treated eval and observability as one blurry category. They aren't. What-happened (traces) and what-will-happen-again (replay) answer different questions, need different infrastructure, and fail in different ways.

Observability catches drift after it happens. Replay catches regressions before they ship.

Same trace data, different use case. Which is why the tools are converging on ingesting each other's formats. The underlying artifact is shared, the interpretation isn't.

[Stair AI @Stair_AI]

If you're picking a model to make your agent more reliable, you might be tuning the wrong knob.

Nvidia published research Friday (via TechCrunch) that makes the point hard to ignore. They took Claude Opus 5 on ARC-AGI-3, a set of 2D games with no instructions where the model has to figure out the rules and win. Bare model: 30%. Same model, wrapped in a custom harness with memory handling and a "supervisor" component nudging it when it drifts: 100%.

Same brain. Different scaffolding. 30 to 100.

The harness is the software around the model. Tools, memory, context management, the rules that turn a raw model into something that can act on its own. And for long-horizon tasks, the kind that string many decisions together, it's doing more of the work than the model is.

One piece worth stealing: the supervisor. A second agent that watches the main one and prods it when it starts down a dead-end path or re-treads ground it already covered. Most people run a single-layer harness (Claude Code, Codex) and stop there.

One caveat, because it matters. Nvidia sells harness tooling under Nemo. "Open harnesses put you in control" is also a sales line. But the finding tracks with July's Databricks result, that the wrong harness can 2x your cost. Two labs, same direction.

If your agent's flaky, look at the wrapper before you swap the model.

https://t.co/Gotob1NIC7

[Stair AI @Stair_AI]

@ValonHajredini Great piece! Thanks for sharing.