Ryan

CTO @heynoah. Built ARC-AGI 2 evals @gregkamrad.

[Ryan @_PaperMoose_]

When you deploy an LLM-as-a-Judge, you’re shipping a classifier into production.

Each new version is a hypothesis about how the model interprets the world.

It’s data science, just expressed in natural language.

Here’s what that looked like for a recent client project where we trained an evaluator to detect a specific agent error type (labeled Category 1 failures) before release.

Dataset

Dev: 104 labeled traces (46 failures, 58 clean)

Eval: 95 labeled traces (34 failures, 61 clean)

What We Saw

v1 established a clear baseline.

v2 drove recall higher but overfit to the dev set, collapsing generalization.

v3 made surgical adjustments that clarified “when not to trigger,” improving specificity and stability.

v10 is when started to see a step change in the eval set performance, a sign the judge was beginning to generalize.

Why It Matters

I find that teams often fall into the trap of assuming the llm works without verifying it through hard data. This is a big mistake! Look at the numbers below and see for yourself. Even with careful preparation, the model still fails to correctly classify more than 80 percent of actual labeled errors.

A few percent of overfit recall here, a small generalization gap there, and suddenly your CI isn’t filtering what you think it is.

Treat them like classifiers: versioned, measured, and tuned against held-out data.

That’s how you keep agents honest in production.

@HamelHusain @sh_reya

[Ryan @_PaperMoose_]

@levelsio total pyscho

[Ryan @_PaperMoose_]

@SahilBloom I can definitely relate to this. How did you do it?

[Ryan @_PaperMoose_]

@thorstenball Been living in coding agents all year. The thing that decides which ones stick for me isn't raw capability, it's how fast I can review what they produced. That's where my bottleneck moved. Curious how Neo handles the review surface.

[Ryan @_PaperMoose_]

reading user transcripts on a sunday is the closest thing founders have to therapy.

not because the users say nice things. because you stop being defensive about your product and start hearing what they actually said.

distance is a feature. plan for it.

[Ryan @_PaperMoose_]

ai chat ux failure mode i keep seeing:

the agent has too much context and uses all of it.

users don't want a system prompt rolled into the response. they want the answer. context should be invisible. the moment the agent says "based on what you told me earlier..." you've shown your hand and lost the magic.

[Ryan @_PaperMoose_]

startup math:

100% of 1 thing > 75% of 7 things.

the second is dead in the water. the math feels wrong because adding features feels like progress. it's the opposite. each new feature dilutes the one thing you were almost great at.

[Ryan @_PaperMoose_]

the questions you ask politely get answered politely.

the questions you ask after four hours of sitting in a room together get answered honestly.

most product research stops at hour one. the good stuff starts at hour three. founders who skip the slog get the polite answer and miss the real one.

[Ryan @_PaperMoose_]

every great agent product i've used has the same property:

i forget there's a model behind it.

the moment you remember it's an llm, the product has lost. the work is making the seams invisible. most teams optimize for showing the seams off.

[Ryan @_PaperMoose_]

scheduling isn't the wedge into executive workflow because scheduling is hard.

it's the wedge because the meeting that didn't happen is more expensive than the meeting that ran ten minutes long.

founders building "calendly for X" miss this. it's not the meeting. it's the meeting NOT happening.

[Ryan @_PaperMoose_]

LLM costs are now the variable cost.
infrastructure is the fixed cost.

we've inverted the SaaS economics in two years and most of the playbooks haven't caught up. usage-based pricing isn't a billing model anymore. it's a margin survival strategy.

[Ryan @_PaperMoose_]

founders ship two products at the same time:

the one the user pays for.
the one they sell to themselves at night to keep going.

if those drift apart for too long you quit. the work is keeping them aligned without lying about either.

[Ryan @_PaperMoose_]

calling it review surface area: how fast your team can comment on what claude code just made.

markdown is bad at this. html with inline comments (like google docs) is great. specs, evals, prompts, feature proposals, code reviews. all rendered + commentable now. comments feed back into claude code as direct edits.

ai for implementation. humans for judgment. that second one breaks the moment review gets painful.

[Ryan @_PaperMoose_]

the eval suite that matters is the one built from user complaints, not the one designed upfront.

upfront evals test what you imagined the product would do. complaint evals test what the product actually does in users' hands.

these are different surfaces. one is a vanity metric. the other is the product.

[Ryan @_PaperMoose_]

i'm a context-switcher. it took me years to stop apologizing for it.

the productivity advice industry is built for momentum people. deep blocks. monk mode. one thing at a time. it's good advice. it's just not for me.

i do my best work jumping between threads. each one charges the next. the cost of forcing myself into a single-track day is bigger than the cost of switching.

play to your wiring. half the productivity gospel is for someone else's brain.

[Ryan @_PaperMoose_]

@businessbarista AI-native

[Ryan @_PaperMoose_]

agents that over-explain what they did are showing their training.

agents that just did the thing and moved on are showing taste.

"here's the draft, does this look okay?" is a model output. "done, you're booked thursday at 2pm" is a product output. these are different things.

[Ryan @_PaperMoose_]

qa suite that runs once is a smoke test.

qa suite that runs and then auto-updates the prompt against the failures is the actual product.

the eval system that improves itself is worth 10x the eval system that just grades.

we built the second one and i'd never go back.

[Ryan @_PaperMoose_]

agent ux insight i didn't see coming:

users tolerate 60+ seconds of latency once they trust the agent.

not because they got more patient. because they stopped watching. trust = put the phone away. the product target isn't speed. it's earning enough trust that users are doing something else while you work.

[Ryan @_PaperMoose_]

@simonw @jarredsumner @bcherny we see this with noah all the time. the exec doesn't get replaced. they just stop doing the stuff they were putting off anyway. robobun same pattern. jarred has more time to think about bun instead of writing it. not a threat, just a reallocation.

[Ryan @_PaperMoose_]

@pmarca yes this is concerning