Overfitting Dicta

Senior patent and trade secret litigation associate. Evaluating and improving legal AI. Writing at Not legal advice.

[Overfitting Dicta @Overfit_Dicta]

What if we assessed legal AI on how it works, not just what it generates?

AI is typically assessed on its results. Legal AI fixates on memos, contract redlines, and other generated outputs because those are what a reviewing attorney sees. Most commonly, such outputs are judged against a rubric of scoring criteria or a sample answer, or compared pairwise against other AI-generated outputs. Results, however, are generated at the end of multi-component AI analysis involving workflow structure, research methods, verification, adaptation, draft assembly, review strategies, and revision procedures. Cumulatively, these processes define an AI agent’s behavioral profile, and that profile is arguably more informative for assessing and improving legal AI than the result it produced.

Behavioral analysis uses the trajectory records of AI sessions to reconstruct their behavioral profiles. Every agentic session produces a trajectory, a recorded event log of the AI's processes in pursuing a specific task. The trajectory of a legal agent might specify which case documents were reviewed in what order, how important facts were verified, how conflicting information was resolved, and how results were drafted, reviewed, and revised. Compared across multiple sessions, these processes can reveal the behaviors of AI models as well as how those behaviors are affected by effort level and skills. Additionally, if certain behaviors are associated with superior analysis, behavioral analysis can help identify the configuration best suited to a particular legal task.

[Overfitting Dicta @Overfit_Dicta]

@ipezyGJ Twelve.

The sampling error between the model/effort/task combination and the trajectory is going to be far more significant than that between the trajectory and the profile. So this feels like an argument for re-running the sessions.

[Overfitting Dicta @Overfit_Dicta]

Categorical Analysis and LAB Rubric Scoring Misattribute Regressions to Claude-for-Legal Skills https://t.co/tplHTZxROm

[Overfitting Dicta @Overfit_Dicta]

I'm planning to publish behavioral profiles for models, tasks, and skills, but not the raw trajectories because there are 216 of them. Is there an analysis where having the raw trajectories would be useful?

For example, attached is the executive summary for native Fable-5 on the compare-asserted-patent-claims-against-accused-product task. The full profile identifies 14 behavioral characteristics derived from pairwise analysis across multiple session trajectories.

I've redesigned this analysis four times, but I think I'm finally happy with it. The results are starting to capture behavioral nuance that was abstracted by the previous categorical analysis.

[Overfitting Dicta @Overfit_Dicta]

@adamdavidlong Agreed. Getting up to speed and staying current on legal AI is exhausting and time-consuming in the short term. But falling behind feels more consequential in the long term.

[Overfitting Dicta @Overfit_Dicta]

Legal-AI behavioral analysis results are still coming, starting with Sonnet-5, Opus-5, and Fable-5. I’ve redesigned the methodology a couple of times to make sure it captures the behavioral nuance.

These are actually the opening paragraphs of an “Introducing Behavioral Analysis for Legal AI” article that’s waiting on updated results.

[Overfitting Dicta @Overfit_Dicta]

What if we assessed legal AI on how it works, not just what it generates?

AI is typically assessed on its results. Legal AI fixates on memos, contract redlines, and other generated outputs because those are what a reviewing attorney sees. Most commonly, such outputs are judged against a rubric of scoring criteria or a sample answer, or compared pairwise against other AI-generated outputs. Results, however, are generated at the end of multi-component AI analysis involving workflow structure, research methods, verification, adaptation, draft assembly, review strategies, and revision procedures. Cumulatively, these processes define an AI agent’s behavioral profile, and that profile is arguably more informative for assessing and improving legal AI than the result it produced.

Behavioral analysis uses the trajectory records of AI sessions to reconstruct their behavioral profiles. Every agentic session produces a trajectory, a recorded event log of the AI's processes in pursuing a specific task. The trajectory of a legal agent might specify which case documents were reviewed in what order, how important facts were verified, how conflicting information was resolved, and how results were drafted, reviewed, and revised. Compared across multiple sessions, these processes can reveal the behaviors of AI models as well as how those behaviors are affected by effort level and skills. Additionally, if certain behaviors are associated with superior analysis, behavioral analysis can help identify the configuration best suited to a particular legal task.

[Overfitting Dicta @Overfit_Dicta]

https://t.co/RswRKtAgdc

[Overfitting Dicta @Overfit_Dicta]

Has anyone tried out Gemini Enterprise for Legal? ⬇️

It looks like it’s currently in preview and will only be available to law firms. Given its focus on purpose-built skills for legal work, I’d love to try it out and see how its capabilities and behavior compare with other legal AI platforms. @googlecloud

[Overfitting Dicta @Overfit_Dicta]

Exactly, that’s the issue. Gemini 3.1 Pro may have been competitive in February, but using it as comparison for the contemporary Thomson-1 creates a pretty weak baseline. February 2026 was around the Claude 4.6 / GPT-5.3 era, roughly 3–4 generations back from today’s frontier models.

I’ve actually heard good things about Gemini 3.7 Flash, but haven’t tested it myself since I cancelled my Gemini/Antigravity subscription.

[Overfitting Dicta @Overfit_Dicta]

I’m thinking the prospective generation risk is more concerning than the provenance issue: Presumably the training data is public… or somebody messed up entering confidential data into a public model.

Harvey is already using LAB to train models on synthetic tasks. So eventually someone uses one of those models to draft a brief without sufficiently verifying the authorities, and we get:

“Here, as in Reyes v. Creston Industrial Coatings, where Judge Harwick of the Northern District of Ohio granted summary judgment that … Reyes v. Creston, Case No. 1:24-cv-00613-EMH (N.D. Ohio Jan. 22, 2025).”

Except the case, parties, judge, and summary-judgment opinion all come from a synthetic LAB task: https://t.co/hZvNzbfiT1

At least then, we’ll know which model generated the hallucinations. 🤣

[Overfitting Dicta @Overfit_Dicta]

@juliesaltman Could this just be training data leakage from whatever model Harvey used to generate LAB (suspect Opus 4.7)?

[Overfitting Dicta @Overfit_Dicta]

So, by your article, measuring Tenet against LAB feels like Level 1.5.

- You’ve got multiple models tested against the same benchmark, which is the Level 2 feature.
- But the AI-generated benchmark is structurally and functionally similar to the model’s development environment, potentially reflecting the same synthetic-generation process and design assumptions.

So there’s an added concern that the model is being developed against one set of synthetic tasks and then tested against close analogs of those same tasks.

[Overfitting Dicta @Overfit_Dicta]

Certainly, LAB (and other AI-generated benchmarks) are useful training environments for legal-AI development, both proprietary and open-source. I don't think a benchmark necessarily needs to be perfect to be useful, particularly where its limitations are understood and the people using it know that an all-pass score on an AI-generated benchmark comes with substantial caveats.

But I think the equities shift when LAB all-pass scores are presented to a broader audience as a measure of legal-AI performance. Unfortunately, I think we're seeing that more and more.

For example, the following is easy to misinterpret as a measure of legal-analysis quality rather than compliance with a particular AI-generated benchmark environment that has known imperfections:

[Overfitting Dicta @Overfit_Dicta]

Yeah, this is one pitfall of a vibe-coded benchmark.

AI-Refusal Artifacts: https://t.co/lwJC9i1kkm

Hallucinated Scoring Criteria: https://t.co/B5A7D2VJ5I

I think the utility of Harvey’s LAB (and other AI-generated benchmarks) for measuring and training legal AI remains an open question.

https://t.co/cMHgE2wNFa

[Overfitting Dicta @Overfit_Dicta]

My RAG pipeline surfaced some odd artifacts in Harvey’s new RL environments for M&A diligence.

I sincerely hope that legal agents are not being trained on vibe-coded AI refusals.

https://t.co/RWSNJJgVnd

[Overfitting Dicta @Overfit_Dicta]

@gabepereyra "The model was trained on approximately 150 NVIDIA B300 GPUs over the course of 2 months."

👆That's a TON of compute. How much did the model cost to train?

[Overfitting Dicta @Overfit_Dicta]

I’m trying to solve two problems. The first originates with parsing flowchart images before anything is chunked or embedded. For those, OCR loses the spatial structure before the document ever reaches chunking.

And if I’m building a parsing solution anyway, I might as well preserve structural information all the way through chunking, embedding, and retrieval. Probably too expensive for a commercial solution, but maybe not for a custom pipeline handling a single case.

I’m actually working on the parser’s geometry renderer this evening, and PDFium just bombed the first test... so back to the drawing board on the layout issue.

[Overfitting Dicta @Overfit_Dicta]

I’m building a parsing solution for judging documents that include embedded flowchart images, which should also support structurally aware chunking for RAG across Harvey LAB and Legora BAR tasks. So far, the parser has been validated on Harvey LAB’s “media-recap” diligence task, which does not include PDFs or visually encoded artifacts like flowchart images.*

The chart below was generated from reconnaissance of the parser’s current architecture. I would appreciate any suggestions or advice from others who have attempted a similar solution.

I tried Docling and Unstructured, but their parsing outputs did not preserve enough source-level structural and relational metadata, and their architectures are not readily adaptable to LLM-based OCR and rendering for unparsable artifacts. I want to take advantage of recently released models like GPT-5.6 Luna make LLM-based OCR and rendering economically feasible. Docling, for example, routes documents through an intermediate document representation (DoclingDocument) that makes source-level adaptation difficult.

The parser is being coded in Codex using GPT-5.6 Sol at Max reasoning as the primary worker. Each phase is audited by Fable-5 at Max reasoning in Claude Code, with human review of each worker/auditor handoff.

*So far, flowchart images and similar visually encoded artifacts have appeared only in documents generated by newer Fable-5, Opus-5, and GPT-5.6 models. The working hypothesis is that this reflects the stronger multimodal document-generation capabilities of newer models and that LAB tasks without these artifacts may have been generated by earlier models that did not embed them, such as Opus 4.7 or 4.8.

https://t.co/YsZx5HMmQ0

[Overfitting Dicta @Overfit_Dicta]

Reposting this because far more people saw the original results from Legora Bar’s open-source Tax Case than saw the subsequent retraction.

The results are being retracted because grading judge models are not reliably parsing flowchart images embedded in the generated Word documents. Newer Opus 5, Fable 5, and GPT-5.6 models appear to be trained to generate flowchart images as .png files and are embedding these flowcharts into their generate word document (.docx) files, which are not being scored by the graders.

Fixing the grading pipeline is not straightforward. The images do not contain embedded, machine-readable text, and conventional OCR often produces word salad because flowcharts are spatial diagrams rather than single-column documents.

https://t.co/1OCAFxvkaQ

[Overfitting Dicta @Overfit_Dicta]

@jkubicki This narrative assumes the senior partner is the most AI-capable attorney. The counter narrative would see AI-capable associates undercut the senior partner with superior AI solutions and lower rates.

[Overfitting Dicta @Overfit_Dicta]

Gotcha, how does Grounded DI work?And why is a perfect Harvey-LAB score good?

I’ve discussed this with a couple of people, and I think a small consensus is forming that a perfect model would correctly fail a small number of criteria, such that the ideal score would be less than 100%.

[Overfitting Dicta @Overfit_Dicta]

The development of a parsing solution capable of running tasks and judging outputs that contain non-text artifacts (e.g., flowchart images) continues. ⬇️ In the meantime, the next project will focus on legal-skill optimization.

Before the identification of hallucinated scoring criteria and AI-refusal artifacts in certain LAB tasks, and Harvey’s subsequent remediation of those issues, a 12-task, skill-matched Claude-for-Legal benchmark was introduced to assess how Claude-for-Legal skills affect legal analysis as measured by Harvey’s Legal Agent Benchmark. ⬇️

The benchmark pairs 12 Claude-for-Legal skills with the Harvey LAB tasks that most directly exercise them. Below are the aggregate results for 15 model/effort configurations, each run both natively and with its matched Claude-for-Legal skill.

The results show that Claude-for-Legal skills cause roughly a 4 percentage-point regression in LAB scores relative to native runs, with regressions broadly distributed across all 15 model/effort combinations.

This project is analyzing the trajectories of native and Claude-for-Legal runs to identify which specific model behaviors lead to regression and which behaviors lead to incremental criteria passes. The objective is to develop a method for optimizing legal skills against LAB performance, then use that method to create optimized versions of the 12 task-matched Claude-for-Legal skills.

*The 12 Harvey LAB tasks included in this Claude-for-Legal benchmark have been reviewed and were not found to contain the non-text artifacts that currently create parsing problems. To further reduce the possibility of generating unparsable artifacts during this analysis, the task prompts were modified to require textual Markdown output rather than DOCX output.

[Overfitting Dicta @Overfit_Dicta]

@Grounded_DI That were changed from fail->pass by the rubric-aware remediation?

[Overfitting Dicta @Overfit_Dicta]

@Grounded_DI And there's a list of "Fourteen Target Remediation"?