@hannahhaina The Moody's for data probably ends up being a benchmark. Does the dataset move scores on real work, graded against what an expert actually delivered?
@Mehdiyac Point 4 is the big one. An operator correcting the model is about the best ground truth a company will ever get, and it usually dies in a chat thread.
@elliotarledge A check run on random inputs ends up grading the part that doesn't matter. Great audit, and a good argument for testing against real cases.
@itamar_mar Replaying past tickets is the best part, since they come with what actually shipped as the answer key. The factory editing its own evals is where I'd get nervous.
@osanseviero Regression vs climb is a smart split. Would love to see how the experts' own deliverables score on it, since up-front rubrics often ask for things a pro wouldn't bother with.
@code_star That o1-traces-in-context result kind of answers it already. The rerank would need evals none of them were tuned on, or it just measures which recipe saw the test.
Today we're releasing MatterBench, the first legal benchmark built from real work and graded against what lawyers actually delivered.
The best model scores just 8.8%. Not one attempt passed a long horizon task.
Other legal benchmarks are written to mirror real work. MatterBench is built from the real work: client matters that lawyers delivered. The real instruction, the real matter file, the lawyers' reasoning, graded against what went to the client.
Can AI run a corporate due diligence, or take a fintech through its license application? Not yet.
tldr: Lawyers aren't going anywhere. Yet.
Leaderboard: https://t.co/k5cE7Uv6TQ (1/5)๐งต
@alexgshaw Works nicely when the right answer is an iced latte. For most professional work the truth only exists in what an expert actually handed over.
@nikogrupen Agreed. The bottleneck is the answer key, since for most knowledge work it's whatever a good professional actually handed over, and someone has to go collect it.
@bitwjg Keeping the eval grader frozen while the agent edits the training one is the right call. I'd love to see whether it can catch a grader that's wrong on substance, not just parsing.
@michaelrbock Same experience here. The grader is the easy part, and making each case truly right takes someone qualified, which doesn't get cheaper with more agents.
@Ronny_MiniMax Trust per kind of task, weighed by what a mistake costs, is the right frame. The expensive tasks stay manual mostly because nobody has built a check a professional would actually trust.
@Vtrivedy10 Seeding teaches agents what a good eval looks like. What they still can't invent is the answer key, which for real work has to come from what an expert actually delivered.
@nicbstme The ownership math grades itself. The 'what's missing or inconsistent' task is the one buyers actually want, and the answer key there is what the analyst flagged, which usually isn't in the database.