Today we're releasing MatterBench, the first legal benchmark built from real work and graded against what lawyers actually delivered.
The best model scores just 8.8%. Not one attempt passed a long horizon task.
Other legal benchmarks are written to mirror real work. MatterBench is built from the real work: client matters that lawyers delivered. The real instruction, the real matter file, the lawyers' reasoning, graded against what went to the client.
Can AI run a corporate due diligence, or take a fintech through its license application? Not yet.
tldr: Lawyers aren't going anywhere. Yet.
Leaderboard: https://t.co/k5cE7Uv6TQ (1/5)🧵