Paper Instruments

Open-source frontier models for knowledge work (prev. TLDC)

[Daanish Khazi @bertgodel]

Some notes on agency and driving agents https://t.co/4DHPqISNIL

[Joey Besgen @joeybesgen]

speaking next week about how @paperinstr leverages @e2b for large scale agent post-training

come through :)

[Malthe Have Musaeus 👨‍💻 @malthe8]

Had a great time designing DiligenceBench and its reference harnesses with the team @paperinstr x @thoughtfullab

It is a benchmark for evaluating AI agents on real public-equity research tasks, giving us a shared environment for testing models and harnesses against dense, task-specific rubrics.

https://t.co/boJbZGLbTX

[Daanish Khazi @bertgodel]

Today, @paperinstr and @thoughtfullab are releasing DiligenceBench, an agent-first benchmark for long-form equity research. DiligenceBench is designed to be a simple, grounded playground for our rubric-based RL and harness optimization work.

[Thoughtful @thoughtfullab]

https://t.co/lx2E2fC4nV

[Paper Instruments @paperinstr]

The LLM Data Company is now Paper Instruments

[Daanish Khazi @bertgodel]

New post: “Notes on Choosing a Rubric Judge”

A practical cost/latency/performance question: Given a well-specified rubric, how strong does your grader model need to be for evaluation and RL?

[Daanish Khazi @bertgodel]

We're releasing early results from training Kos-1 Experimental, a Kimi K2.5 checkpoint post-trained on the same medical RL data we used for Kos-1 Lite.

As clinical workloads become more agentic, we wanted a model that pairs medical domain knowledge with tool-calling knowhow.

[Paper Instruments @paperinstr]

Medical RL at 1T scale

[Daanish Khazi @bertgodel]

We're releasing early results from training Kos-1 Experimental, a Kimi K2.5 checkpoint post-trained on the same medical RL data we used for Kos-1 Lite.

As clinical workloads become more agentic, we wanted a model that pairs medical domain knowledge with tool-calling knowhow.

[Paper Instruments @paperinstr]

Kos is the birthplace of Hippocrates

[Daanish Khazi @bertgodel]

We’re announcing Kos-1 Lite, a medical model that achieves SOTA on HealthBench Hard at 46.6%.

As a medium sized language model (~100B), it achieves these results at a fraction of the serving cost of frontier trillion-parameter models.

[Daanish Khazi @bertgodel]

We're excited to partner with @perplexity_ai on their latest release. We were impressed with how performant their new Deep Research product was on early benchmarks and are thrilled that this work is being open sourced.

[Paper Instruments @paperinstr]

🙏🤝

[Baseten @baseten]

Baseten is proud to support training jobs for The LLM Data Company 💪

[Daanish Khazi @bertgodel]

(1/5)
New post: "Mismatch Praxis: Rollout Settings and IS Corrections". We pressure-tested solutions for inference/training mismatch.

Inference/training mismatch in modern RL frameworks creates a hidden off-policy problem. To resolve the mismatch, various engineering (e.g., FP16 unification, deterministic kernels) and algorithmic (e.g., importance sampling) fixes have been proposed. In this work, we examine how rollout settings (temp, top-p, and top-k) affect mismatch, and how importance sampling corrections bear out in practice.

We find that while Sequence-TIS is theoretically optimal, it can succumb to catastrophic variance in long-horizon contexts. Additionally, non-standard rollout settings create subtle mismatch patterns that require careful engineering fixes. Token-TIS with default rollout settings proved to be the most robust setting for long-horizon training.