Most evals can tell you when an LLM judge is right, but not when it's silently wrong.
At Forus, our LLM judge reads prior authorization forms and decides whether a patient's medical record supports them. A missed error delays someone's medication.
To catch those failures, we built a way to generate realistic wrong answers and test the judge against them.
Here's how 🧵
Congratulations to @CoachDustyMay, Elliot Cadeau, and @UMichBBall on winning their first title since 1989! This team dominated the tournament from start to finish. Well deserved. Go Blue!