Nick Lemoff

I have a toxic relationship with agent evals

[Nick Lemoff @nlemoff]

Aligning agent evals with your personal judgment is essential for building improvement loops you can trust.

Otherwise, you’ll steer your coding agents towards problems that don’t exist, and miss the ones that do.

A common approach for agent evals is LLM-as-a-Judge or Agent-as-a-Judge that labels your traces. The foundation of the eval is the prompt you define as a rubric. However, most people do not realize the first prompt they create doesn’t cover all cases that their judge will have to evaluate.

Say you want to flag instances where users are frustrated with your agent. What happens if a user is still frustrated after an issue is resolved? If your initial prompt did not include clarification on whether to flag this or not, the judge might or might not label this trace as frustrated. When this ambiguity scales to thousands of traces, you'll be making changes to your agent based on false positives or negatives.

Therefore, having a human in the loop to verify the evals is a natural prereq to creating scalable evals you can trust. This has traditionally looked like the following: hand-label agent traces, compare your labels with the judge’s, and continue iterating on the rubric until the judge aligns with your ground-truth labels.

I work directly with agent engineers who have historically found this tedious and avoided it entirely, but I’d argue that aligning against human-reviewed traces is well worth the time. Labeling even just 10-20 traces will make the quality of your rubric significantly better, allowing you to reliably grade tens of thousands of traces.

I’ve personally seen teams that neglected judge alignment and ended up spending entire sprints’ worth of engineering time making choices based on a false understanding, only to have to do a complete 180 after realizing their evals weren’t giving them accurate insights.

This is why at @JudgmentLabs, we spent a lot of time streamlining this process in our product. Once you provide the ground-truth labels, our agents iteratively refine and backtest the rubric until it matches your labels. The product also reminds you to regularly label new traces so that your judges stay accurate as your agent evolves.

Whether you’re using Judgment Labs or not, aligning your judges’ rubrics will not only improve the quality of their evals, but will eventually help create an eval system you can trust and scale.

[turbopuffer @turbopuffer]

we’re serving up scoops of puffmallow in SF with @JudgmentLabs to celebrate their GA

come grab a cup or cone at 1 Montgomery, now til 4 pm

[Mercor @mercor]

Congratulations @judgmentlabs on launching general access and thanks again for the @mercor ice cream. 🍨

[Judgment Labs @JudgmentLabs]

~1 min and 1 prompt away = Your first issue and judges from your own data.

[Nick Lemoff @nlemoff]

@seedelano what if I’m not a candidate but wanna work on your evals 👀

[Nick Lemoff @nlemoff]

@seedelano honored to have you all as customers :) been a pleasure working with the @trymelius team!

[Nick Lemoff @nlemoff]

@alexshander03 bro hasn't tried the judgment-writing-voice skill and it shows

[Nick Lemoff @nlemoff]

@hmartenjoyer @typesafeai yall actually don't miss. every time I see anyone from typesafe in the elevator, they're always dripped out

[Mercor @mercor]

Join us tomorrow in San Francisco for a free scoop of @Mercor mocha ice cream.🍦

Congratulations @JudgmentLabs on releasing general access and thanks for the sweet treat.

500 Howard St.
San Francisco
Wednesday, October 28, 11AM-1:15PM

[Brad Johnson 百面包 @johnson_brad]

This was a sweet idea from @JudgmentLabs. Excited for a scoop of @mercor mocha tomorrow! 🍦

Thanks for including us!

[ahh @abhishekg999]

quick osint

[Nick Lemoff @nlemoff]

@999toba @bernirov Always spot check with humans
https://t.co/fBXh46fETb

[Nick Lemoff @nlemoff]

Aligning agent evals with your personal judgment is essential for building improvement loops you can trust.

Otherwise, you’ll steer your coding agents towards problems that don’t exist, and miss the ones that do.

A common approach for agent evals is LLM-as-a-Judge or Agent-as-a-Judge that labels your traces. The foundation of the eval is the prompt you define as a rubric. However, most people do not realize the first prompt they create doesn’t cover all cases that their judge will have to evaluate.

Say you want to flag instances where users are frustrated with your agent. What happens if a user is still frustrated after an issue is resolved? If your initial prompt did not include clarification on whether to flag this or not, the judge might or might not label this trace as frustrated. When this ambiguity scales to thousands of traces, you'll be making changes to your agent based on false positives or negatives.

Therefore, having a human in the loop to verify the evals is a natural prereq to creating scalable evals you can trust. This has traditionally looked like the following: hand-label agent traces, compare your labels with the judge’s, and continue iterating on the rubric until the judge aligns with your ground-truth labels.

I work directly with agent engineers who have historically found this tedious and avoided it entirely, but I’d argue that aligning against human-reviewed traces is well worth the time. Labeling even just 10-20 traces will make the quality of your rubric significantly better, allowing you to reliably grade tens of thousands of traces.

I’ve personally seen teams that neglected judge alignment and ended up spending entire sprints’ worth of engineering time making choices based on a false understanding, only to have to do a complete 180 after realizing their evals weren’t giving them accurate insights.

This is why at @JudgmentLabs, we spent a lot of time streamlining this process in our product. Once you provide the ground-truth labels, our agents iteratively refine and backtest the rubric until it matches your labels. The product also reminds you to regularly label new traces so that your judges stay accurate as your agent evolves.

Whether you’re using Judgment Labs or not, aligning your judges’ rubrics will not only improve the quality of their evals, but will eventually help create an eval system you can trust and scale.

[Nick Lemoff @nlemoff]

@liambraus @bernirov Totally agree, I wrote about this a couple week back. Judgment is a great tool for gaining visibility into these exact types of failures
https://t.co/fBXh46fETb

[Nick Lemoff @nlemoff]

Aligning agent evals with your personal judgment is essential for building improvement loops you can trust.

Otherwise, you’ll steer your coding agents towards problems that don’t exist, and miss the ones that do.

A common approach for agent evals is LLM-as-a-Judge or Agent-as-a-Judge that labels your traces. The foundation of the eval is the prompt you define as a rubric. However, most people do not realize the first prompt they create doesn’t cover all cases that their judge will have to evaluate.

Say you want to flag instances where users are frustrated with your agent. What happens if a user is still frustrated after an issue is resolved? If your initial prompt did not include clarification on whether to flag this or not, the judge might or might not label this trace as frustrated. When this ambiguity scales to thousands of traces, you'll be making changes to your agent based on false positives or negatives.

Therefore, having a human in the loop to verify the evals is a natural prereq to creating scalable evals you can trust. This has traditionally looked like the following: hand-label agent traces, compare your labels with the judge’s, and continue iterating on the rubric until the judge aligns with your ground-truth labels.

I work directly with agent engineers who have historically found this tedious and avoided it entirely, but I’d argue that aligning against human-reviewed traces is well worth the time. Labeling even just 10-20 traces will make the quality of your rubric significantly better, allowing you to reliably grade tens of thousands of traces.

I’ve personally seen teams that neglected judge alignment and ended up spending entire sprints’ worth of engineering time making choices based on a false understanding, only to have to do a complete 180 after realizing their evals weren’t giving them accurate insights.

This is why at @JudgmentLabs, we spent a lot of time streamlining this process in our product. Once you provide the ground-truth labels, our agents iteratively refine and backtest the rubric until it matches your labels. The product also reminds you to regularly label new traces so that your judges stay accurate as your agent evolves.

Whether you’re using Judgment Labs or not, aligning your judges’ rubrics will not only improve the quality of their evals, but will eventually help create an eval system you can trust and scale.

[Nick Lemoff @nlemoff]

@arvarik_ classic 5th place most PF

[Ara @arafatkatze]

Agent trace telemetry is one of the most critical parts of the RSI loop.

Back when I did an RSI loop on Cline to make it automagically improve the harness over a 17 hour long running agent run I found that parsing good signals and iterating on it with the analyser agent could fix an agent response and rerun to hill climb agentic evals. Until it reaches the models peak possible score without some tricky reward hacking.

This is just one part, now imagine if the agent failures at the peak become filtered to become proper post training signals and trajectories of growth, wouldn't that be amazing?

Now lets go one step further, if you are an enterprise with tons of confidential data that AI just messes up because its too much context to get it all right at once what do you do? You take a smaller model and get agent telemetry and post train models and get higher perf.

The future is a world of smaller post trained models curtailed to specific enterprise use cases working in tandem with behemoth trillion dollar models as agent orchestrators but it all has to start somewhere.

[kirandarisi @kirandarisi]

Congrats @JudgmentLabs and @alexshander03 on GA!

The team was in our Slack shipping features at 4 AM. That's the kind of partner you want. Great integrating Judgment into our stack at Atomicwork, and looking forward to building more together.

[澔澔 @JahowYu]

.@Guidepoint @TegusHQ @AlphaSights I am open and willing to take customer reference calls for judgement labs

[Luciano Arroyo @LucianoArroyo_]

Back in January, @carloagostinel2 introduced me to @alexshander03, @JSCamyre, and @andrew_li03.

From the first conversation I knew this team was onto something special. 10 months in and it’s been an incredible journey helping customers build evals across all kinds of agents. This team works harder than any group I’ve been part of, and I’m incredibly proud of what we’ve built. 🧡

Now we’re live for general availability. Give the platform a try: https://t.co/hGW74bZRSR

[Andy Fang @andyfang]

@alexshander03 @JudgmentLabs congrats Alex on the launch!

[Nick Lemoff @nlemoff]

guys, it's literally free ice cream. pull up

[Alex Shan @alexshander03]

The @JudgmentLabs free ice cream truck is back in SF.

Tue, Wed, Thu this week. Go get some 🍦

Thank you to our partners! Today, we're serving @typesafeai jev jam, @modal À La Modal, @ExaAILabs EXA-presso.

[Nick Lemoff @nlemoff]

@bruceradley @aresotik We work with customers who have agents that run significantly longer than this