ben redmond

you don't measure your coding agent's performance.
it's time to start.

[ben redmond @ben__redmond]

https://t.co/cisp9tFpoO

[ben redmond @ben__redmond]

Strong agree with the need to benchmark on your own tasks - I've been building on this idea for a while (coding agent evals on your own code)

One relevant learning I've had is that passing held out tests and being code you'd actually merge are two different bars. Identical pass rates can hide a big quality gap. Binary pass/fail on tests dodges judge bias / issues with LLMs but has its own ceiling in discriminating model quality.

Super interested to hear how yall plan on using the results. I've been thinking along the lines of:
- Model / reasoning effort routing based on task difficulty
- Recursive improvement on the harness (eg optimize AGENTS.md using private eval as a fitness function)
- Org-wide release decisions / CI for changes in defaults (models, harness, context)

[ben redmond @ben__redmond]

for coding evals, I've actually been exploring a mixture of deterministic (does it pass tests + compile) and subjective (llm-as-a-judge using rubrics to assess code) as I've found that the determinist gates don't fully capture the difference between a good and great response.

agree that agents dont seem quite capable enough to apply judgement on what harness lever to pull - it feels like they still need the feedback loop to check their work and iterate

not sure I understand the predictive model point - mind explaining further :)

[ben redmond @ben__redmond]

@jxnlco ideally product would make the choice, but it's hard to trust that an llm will be able to make an informed choice without having eval data on the reasoning effort -> performance curve. especially bad if product under-provisions effort and task ends up failing / introducing bug

[ben redmond @ben__redmond]

@AndrewCurran_ I just ran a small GLM 5.2 eval on some open-source repos (go, rust) and it wasn't super impressive - below Composer 2.5 performance while costing about 2x as much. Obviously ymmv on task shape, but it feels like the type of model that needs to be carefully watched

[ben redmond @ben__redmond]

*how* engineers should route tasks to models/reasoning efforts is super hard and requires lots of time with the model

I'm exploring more measured approaches - testing the models at various reasoning efforts on real tasks to determine reasonable routing policies

Without data, choosing between Opus medium and xhigh can be a 2x cost increase that's purely based on vibes of task difficulty

[ben redmond @ben__redmond]

I've realized that most public benchmarks are not super representative of how I actually use the model. like, who prompts like that...

There's an increasing need for
1 real world benchmarks like deepswe or cursorbench
2 personalized benchmarks on your repo using your context / prompting style

2 primarily becomes valuable at scale, i.e. when there are many developers using agents in one repo. but in that case, I think that local measurement and optimization can produce tangible cost / performance gains

[ben redmond @ben__redmond]

@GergelyOrosz adoption is way up, but we're still treating the tools like it's late 2024 - relying entirely on vibes. if you can save 10% by switching to another model, or through some other technique, you also have to show me what the quality tradeoff is / new failure modes are.

[ben redmond @ben__redmond]

@garrytan this is directionally right, but IMO the measurement function part of SkillOpt is lacking. in other words, in order to gradient descent a skill, we need a reliable and high-signal way to actually test how that skill affects the agent behavior, and a metric to optimize towards

[ben redmond @ben__redmond]

@RhysSullivan This also addresses a failure pattern I've seen where opus 4.7 tends to "overthink" at higher reasoning levels and ends up overcomplicating things. Using lower reasoning efforts seems to help

[ben redmond @ben__redmond]

@RhysSullivan Biggest thing to tune per-task is reasoning-effort - you retain the base model intelligence, but can get huge cost savings by reducing the amount of thinking tokens. e.g. use opus 4.7 xhigh to plan, and opus 4.7 medium to execute

[ben redmond @ben__redmond]

@hiarun02 recently ran an analysis of 5.5 reasoning effort levels, and came out that high is the sweet spot between cost/intelligence. xhigh does post non-trivial gains, but also uses 1.5x - 2x as many tokens

[ben redmond @ben__redmond]

A bit late to this but I've been dealing with this exact problem, and feel like it's been underdiscussed.
I've built a beta product to do exactly this - have been running this loop on a few OSS repos (Zod, graphql-go-tools) to A/B test things such as model choice, reasoning effort, AGENTS.md changes, SKILL.md changes, etc on PR-derived tasks.

An example report is here, looking at the effect of reasoning effort on gpt-5.5 https://t.co/xZ9oVoyS5K

If you want a sanity check against the framework you're building, I'd be happy to help run a one-shot eval on your repo (entirely local on your machine) and compare notes - I'm always looking to learn and improve what I;m measuring.

[ben redmond @ben__redmond]

@morganlinton it's a completely different intelligence profile, and seems like it requires re-architecting CLAUDE.md / skills / other context to fit it. (had to do a similar thing when I adopted Codex and found it to be much more instruction-following than Claude previously)

[ben redmond @ben__redmond]

@aliceisplaying model is the same, it's just thinking less

[ben redmond @ben__redmond]

I'm working on repo-specific coding agent evals and have thought about multi-turn a decent amount. however, I came out that there's enough of a gap in measuring single-turn that we should focus on that as a proxy for multi-turn performance for the short term.

there are already sooo many variables affecting performance that we're not able to effectively measure (model, harness version, claude.md, skills, reasining effort, time of day, ...)

[ben redmond @ben__redmond]

@dejavucoder cursorbench + composer 2 report touch on this a bit
https://t.co/qjiE48Lphs https://t.co/0HTVOqgGQw

although feels like doing good evals for single turn is already hard enough, let alone multi-turn

[ben redmond @ben__redmond]

@kunchenguid cool benchmark, tracks with my experience of codex being great at invoking/following skills (almost to a fault sometimes), whereas claude really doesn't do it unless explicitly asked.

wonder how tweaking CLAUDE.md / other context affects this

[ben redmond @ben__redmond]

there are certainly classes of tasks that cheaper models excel at - the problem is understanding what those tasks are on your codebase. ideally we use OSS for the 80% of tasks where agent performance is similar to closed source, and closed source for the 20% truly hard/unique tasks.

the important bit here is gaining the ability to actually measure the agent performance on your tasks, and using that telemetry to make decisions about "right sizing" models for tasks