Sebfox

Building AI evals | Previously AI at McKinsey & QuantumBlack | Medical Doctor

[Boris Cherny @bcherny]

Something I have been thinking about: in the past, the best engineers I knew spent a lot of time automating their work in various ways. Better vim/emacs automations, writing lint rules to catch repeat code issues, building up a suite of e2e tests so they don't need to smoke test the app manually. These kinds of things were the highest leverage activities an engineer could do, because it multiplied their own output, which in turn meant they could build more things.

I think many of these automations have become even more important now. This is true for a number of reasons.

First, infra and DevX automation speeds you up. And if you are running an army of agents, each of those agents will be sped up also. More automation == more output per unit of time.

Second, moving things to code improves efficiency. Your agent could fix an issue every time it sees that issue happen, but that uses tokens and might miss cases. If Claude instead writes a lint rule, CI step, or routine, that class of issue can be fully automated forever. This is really what people are talking about when they talk about loops -- it's about automating entire types of busywork rather than solving them one off. This isn't a new idea at all. Engineers have been doing this for a long time!

Third and most importantly, automation makes it possible for others to contribute to the codebase more easily. Increasingly what I am seeing is engineers are contributing to codebases on day one because Claude can navigate the codebase for them, and that non-engineers are able to contribute to a codebase as effectively as engineers can. What gets in the way of both of these is domain knowledge that lives in peoples' heads rather than in automation -- the stuff you used to have to learn when ramping up. What has changed thanks to agents is the domain knowledge that can be encoded as infrastructure is no longer limited to what is expressible in lint rules and types and tests; it can now capture nearly all domain knowledge, encoded as code comments and skills and CLAUDE.md rules and memories. If I put up a PR for an iOS codebase I don't know and a code reviewer rejects it because it doesn't use the right framework, or if a designer builds a new feature and it gets rejected because it doesn't follow the right architectural patterns, these are failures of automation.

Every team should be writing the CLAUDE.md's, REVIEW.md's, skills, and docs that enable agents to productively work in their codebase with zero additional context from the prompter. This sounds crazy, and at the same time is a natural extension of the stuff engineers have always done: automate, and encode domain knowledge as infrastructure. As the model gets smarter and as the harness matures, this task becomes easier. In the meantime, it is on every team to look for ways to convert their domain knowledge to infra so that Claude can write code better, so that code review catches issues automatically, and so the next person working on your codebase can contribute more easily.

[Sebfox @Sebfox1]

The real question isn't "is our eval process good enough?"

It's "at what scale does our current approach become dangerous?"

[Sebfox @Sebfox1]

470+ conversations with teams building AI. Here's how they actually evaluate quality in production:

- Top law firm: Google Sheets, one tab per document, manual true/false scoring
- Precision medicine co: one person reviewing 50-100 queries/day by hand
- Healthcare co: rotating doctors through call reviews (creates more variance, not less)
- Browser automation startup: Mechanical Turk at $1/eval
- Multiple enterprise teams: dumping logs into Claude and asking "was this good?"

[Sebfox @Sebfox1]

If you're evaluating AI with spreadsheets and spot-checks — you're not behind. You're normal.

The "best practices" people post about are aspirational, not widespread. Everyone is building the plane while flying it.

[Sebfox @Sebfox1]

@chipro Very interesting, does the CCP have a stated desire to encourage this I wonder, or is it independent of that?

[Sebfox @Sebfox1]

@chipro @lennysan super interesting, thanks!

[Sebfox @Sebfox1]

The question I wanted answered: do different starting assumptions converge?

They did. More on that in the next post.

[Sebfox @Sebfox1]

More autostrategist experimenting. Set up 4 variants overnight — same evidence base, same loop, different strategic seeds: services-heavy, product-led, OSS-first, moonshot.

Different critic panels on each. 7 critics per variant.

[Sebfox @Sebfox1]

Tournament mode every 5 iterations: independent agent writes a fresh strategy from scratch. Writer can adopt or cherry-pick. Stops local optima.

All on Claude Max CLI — free to experiment. Tell it to pause until limits reset, then pick back up with cross-variant analysis.

[Sebfox @Sebfox1]

@sh_reya been doing the exact same myself! Plugging in qualitative critic agents quantitative eval scoring as feedback for a central writer agent that iterates on product ideas, strategy, marketing etc

[Sebfox @Sebfox1]

@sh_reya great point - and not well at all, I can see! Look forward to trying this out

[Sebfox @Sebfox1]

@karpathy yeah, ive had the exact same. Perplexity is obsessed with the fact that i once said i like wine, and now any recommendation i ask for is completely wine oriented

[Sebfox @Sebfox1]

@karpathy feels similar to LLM Judge for evals, can be very confident in an answer being high or low quality, regardless of its actual quality. Seems to more often be over optimistic in favour of high-quality

[Sebfox @Sebfox1]

@Robsnyder_ The most useful output isn't the strategy document — it's the critic feedback transcripts. Where each persona pushes, iteration after iteration. That's the real map of strategic tensions.

[Sebfox @Sebfox1]

What if you had a sceptical customer, your CTO, a Datadog-scale founder, top practitioners in your field, @RobSnyder_ as sales advisor, and Paul Graham — all critiquing your strategy at once?

What if they iterated on it overnight and you woke up to a better version?

[Sebfox @Sebfox1]

@Robsnyder_ PG and @RobSnyder_ kept landing on the same conclusion from completely different reasoning. PG because focus is everything. Rob because strategy doesn't matter until someone's signed.

The customer and Datadog founder are structurally at odds. That's the whole point.