Vasudha Rengarajan

[Aatish Nayak @nayakkayak]

https://t.co/OfierUuZaZ

[Harvey @harvey]

Introducing Harvey II: smarter agents from the start.

- Featuring Harvey Tenet, our first model trained for legal work
- Built around matters and projects
- Agents start with the files, context, permissions, and history they need
- Assign tasks to lawyers or agents, then track and review the work
- Harvey remembers how you work and writes like you

[Christina Nguyen White @ChristinaNWhite]

and we're just getting started...

[Julio Pereyra @ItsJulioPereyra]

Great working with the @EngramLab team to explore model memory as a way to improve personalization and effective knowledge recall

We're particularly excited about how models improve trajectories through memory. Rather than just learning to use search tools better, memory enables more directed, efficient, and effective searches that result in far higher intelligence per token.

Promising for a future where agents never start from scratch!

[Siva Gurumurthy @sgurumur]

Presenting our first post-trained model effort, that maximized intelligence per token. This is humble beginnings and wouldn’t be possible with all our research partners and a mighty team of @nikogrupen @ItsJulioPereyra @calvincongelado @vtrengarajan @gabepereyra

Summary:
- Tenet is a Kimi K3 base that we post-trained for long-horizon, agentic legal tasks.
- We co-optimized harness to make training and task execution more effective.
- It completes 2x held out tasks on LAB and 20% more on LAB contracts than base Kimi K3
- Not just lab, the results transferred other legal benchmarks: e.g. APEX-v1,CUAD, and MAUD.

The idea is simple:
we sample multiple ways to solve the same legal task, grade them, and train the model toward the best approaches. More technically, we use GSPO to sample groups of independent rollouts, score them with rubric-based rewards, and compute advantages within each group.

When we co-optimize harness and model, we get really good results on key legal-specific product areas:
• M&A Diligence: pass rate jumped from 46% to 60% and could tackle datarooms of up to 80M tokens
• Review table: Answer quality improved by 3.6 points and citation quality by 12.1 points at roughly 1/10th the cost per cell.
• Firm Knowledge: improved criteria pass rate by 15%+, and reduced cost per query by 90%

Follow Gabe for full details:

[Vasudha Rengarajan @vtrengarajan]

An exciting milestone for @harvey research! Introducing Tenet, our custom model post-trained for legal.

We also share details of our investments in data gen, RLMs, reward shaping, long-horizon agents, memory, search, & more.

A big thank you to the many incredible collaborators who helped make this happen!

[Gabe Pereyra @gabepereyra]

https://t.co/1c2Dn0vuoX

[Siva Gurumurthy @sgurumur]

Review Table is a fascinating fan-out problem: M documents and N columns. Naively, each table can require O(MN) model calls. We wanted to fix that.

For our first foray into specialized model training, we worked with Applied Compute to build a model that beat frontier models on answer and citation quality while cutting cost per cell by 55%. Agentic retrieval also used 50% fewer input tokens at the same quality.

[Harvey @harvey]

We post-trained a model for Harvey's Review Tables with @appliedcompute, reducing costs by 50% while improving answer and citation quality.

Review Tables allow lawyers to upload up to 10,000 documents and ask up to 500 questions over each document - yielding up to 5 million model queries.

As a result they comprise a large fraction of our inference workloads. In July, the largest review table query on Harvey cost $26,000 in inference costs over a single run.

We partnered with @appliedcompute to post-train GLM-5.2 to optimize both quality and cost for review table queries.

For training data, we started with a corpus of open-source legal data like filings, contracts and emails.

We then used frontier models to simulate different user personas and generate realistic review table queries over this corpus, alongside generating ground-truth answers to these queries sourced from the documents. Our applied legal research team did several rounds of review to ensure this data matched our quality bar.

With GLM-5.2 as a base, we post-trained a custom review table model in Harvey's review table harness on Applied Compute's AC2 platform.

This custom model beat all frontier models by 4-17% on answer quality and 11-19% on citation quality for review table queries.

It's also much cheaper: less than half the cost of Sonnet 5, and 1/10th the price of leading frontier models.

We also experimented with harness engineering alongside model training. We trained two Qwen 3.6-35B-A3B models in two separate harnesses: one using single-turn RAG and the other using agentic search. The agentic harness matched answer quality but drove down input tokens by 50% and output tokens down by 29%.

These results are promising early steps in improving quality and cost across Harvey using custom models.

Deep dive from @vtrengarajan, @nikogrupen, @itsjuliopereyra, @srice120, and Karl de la Roche at Harvey, and @rhythmrg @caenopy @jacob_dphillips at Applied Compute:

[Vasudha Rengarajan @vtrengarajan]

In conversation with the @appliedcompute research team about post-training models for high-scale legal tasks.

Learn more about our collaboration here! https://t.co/4JxzdL2TAL

[Applied Compute @appliedcompute]

We partnered with @Harvey to train a model for Review Table, achieving state-of-the-art accuracy at a fraction of the cost and latency of frontier alternatives.

Review Table is one of Harvey’s highest inference volume products. Training an open source model for production traffic required building a representative eval and understanding how real lawyers use the product. Our teams discuss the research process.

[Vasudha Rengarajan @vtrengarajan]

Together with @appliedcompute, we post-trained a model for @harvey Review Table, pushing the Pareto frontier on quality & cost. Was great to collaborate with @ypatil125, @caenopy, @jacob_dphillips, @rhythmrg, and the AC team. Learn more about our results here!

[Vasudha Rengarajan @vtrengarajan]

https://t.co/FLzObAJnod

[Vasudha Rengarajan @vtrengarajan]

https://t.co/FLzObAJnod

[Gabe Pereyra @gabepereyra]

Last week we held Harvey Hacks, our internal hackathon.

27 projects total across our 200-person eng team.

Wanted to highlight a few hackathon projects:

[rahul @rahulgs]

yes things are changing fast, but also I see companies (even faang) way behind the frontier for no reason.

you are guaranteed to lose if you fall behind.

the no unforced-errors ai leader playbook:

For your team:
- use coding agents. give all engineers their pick of harnesses, models, background agents: Claude code, Cursor, Devin, with closed/open models. Hearing Meta engineers are forced to use Llama 4. Opus 4.5 is the baseline now.

- give your agents tools to ALL dev tooling: Linear, GitHub, Datadog, Sentry, any Internal tooling. If agents are being held back because of lack of context that’s your fault.

- invest in your codebase specific agent docs. stop saying “doesn’t do X well”. If that’s an issue, try better prompting, https://t.co/SOjpn47yxo, linting, and code rules. Tell it how you want things. Every manual edit you make is an opportunity for https://t.co/S1ZvtYQwta improvement

- invest in robust background agent infra - get a full development stack working on VM/sandboxes. yes it’s hard to set up but it will be worth it, your engineers can run multiple in parallel. Code review will be the bottleneck soon.

- figure out security issues. stop being risk averse and do what is needed to unblock access to tools.

in your product:

- always use the latest generation models in your features (move things off of last gen models asap, unless robust evals indicate otherwise). Requires changes every 1-2 weeks - eg: GitHub copilot mobile still offers code review with gpt 4.1 and Sonnet 3.5 @jaredpalmer. You are leaving money on the table by being on Sonnet 4, or gpt 4o

- Use embedding semantic search instead of fuzzy search. Any general embedding model will do better than Levenshtein / fuzzy heuristics.

- leave no form unfilled. use structured outputs and whatever context you have on the user to do a best-effort pre-fill

- allow unstructured inputs on all product surfaces - must accept freeform text and documents. Forms are dead.

- custom finetuning is dead. Stop wasting time on it. Frontier is moving too fast to invest 8 weeks into finetuning. Costs are dropping too quickly for price to matter. Better prompting will take you very far and this will only become more true as instruction following improves

- build evals to make quick model-upgrade decisions. they don’t need to be perfect but at least need to allow you to compare models relative to each other. most decisions become clear on a Pareto cost vs benchmark perf plot

- encourage all engineers to build with ai: build primitives to call models from all code bases / models: structured output, semantic similarity endpoints, sandbox code execution. etc

What else am I missing?

[Ricky @rickyrobinett]

What can an 8-year-old build in 45 minutes with the assistance of AI?

My daughter has been learning to code with @cursor_ai and it's mind-blowing🤯

Here are highlights from her second coding session. In 45 minutes she built a chatbot powered by @CloudflareDev Workers AI 👀

[Darren Markland @drdagly]

More and more this shows me how disparities drive despair. Without financial support essential workers and the marginalized don’t have a real chance at avoiding these risks. These are the people in our unit. Privilege is protective. I’ve known this, but damn, this hits me.

[Darren Markland @drdagly]

Last night before I intubated a lovely man with COVID, he insisted on calling his boss to explain why he wouldn’t be into work on Mon. He was so short of breath I had to finish the call for him. In the aftermath I realized how this job was a lifeline for him. It left me unnerved.

[Vasudha Rengarajan @vtrengarajan]

Looking forward to speaking with @UCBerkeley students about ML!

[A.J. Bauer @ajbauer]

Biden, Blinken, Yellen — what is this, a gerundocracy?

[Vasudha Rengarajan @vtrengarajan]

I had the best time collaborating with NYU ITP students (in a cross-registered class with Columbia University). ITP students are so bright and creative! Can’t wait to see all the value they bring to the world through art+tech.

[Bo Ren @Bosefina]

I’d love to pair NYU ITP grad students with SV grow marketers to incubate consumer social products. ITP grads understand how to turn art into pixels into sound. SV growth strategists know how to get your first 20K DAUs. The next big social product needs to embody art & science.