@prysmfoundry What surprised me building this: top models get most checks right (~65% partial credit) but almost never all of them. In legal work, one missed critical point means it can't go to the client. That gap is the job. Agents aren't there yet, but they will be.
Today we're releasing MatterBench, the first legal benchmark built from real work and graded against what lawyers actually delivered.
The best model scores just 8.8%. Not one attempt passed a long horizon task.
Other legal benchmarks are written to mirror real work. MatterBench is built from the real work: client matters that lawyers delivered. The real instruction, the real matter file, the lawyers' reasoning, graded against what went to the client.
Can AI run a corporate due diligence, or take a fintech through its license application? Not yet.
tldr: Lawyers aren't going anywhere. Yet.
Leaderboard: https://t.co/k5cE7Uv6TQ (1/5)๐งต