Update on @harvey’s model training effort.
We post-trained a model we are calling Tenet:
- Achieves SOTA on LAB
- Generalizes 3rd party legal benchmarks
- Uses sub-agents for domain specific capabilities
Tenet uses Kimi K3 as base and was post-trained in collaboration with @FireworksAI_HQ:
- Rank-64 LoRA over the full network
- GSPO with importance-ratio masking
- 134 B300 GPUs for 2 months
Despite not being trained on 3rd party legal datasets we found improvements on:
- @mercor’s Apex Agents - Corporate Law
- @crosbylegal’s Redline Bench
- LegalBench
Tenet also learned how to use domain-specific subagents (separate post-trained models) for complex tasks:
- M&A Diligence: training in an RLM harness for long-horizon tasks (with @baseten)
- Review Table: specialist models for high-volume structured data extraction (with @appliedcompute)
- Firm Knowledge: parametric memory and structured notes for more efficient enterprise search (with @engram)
These results suggest we can significantly scale training and we plan to:
- Scale both human and synthetic data significantly and scale training to 1K and then 10K GPUs
- This scale will let us move to full parameter fine-tuning and larger models
- We are now starting to post train models in our production harnesses
- Post-train other open-source base models to provide customers with model choice
If these problems sound interesting we are hiring for our post-training team
Here’s the raw breakdown of how-we created our product release video. Clear-cut answer to all your anxieties with AI video. I’m listing down what worked, the messy middle of it all.
Check the thread:
today is @square’s biggest release in our company’s history.
over 100 products and features. including the start of SquareAI.
we are collecting all of them, and will keep our roadmap updated, here: https://t.co/t0hkrgXfaJ
San Francisco has the highest % of people who need 45 minutes of Calm meditation, 25 minutes in an Equinox sauna, a $17 smoothie, and 8.5 hours of rest in their Eight Sleep to recover from filing 4 Jira tickets but also want to leave this high-pressure life and become a farmer.