Goodhart Labs

Alignment evals and benchmarks

[Goodhart Labs @goodhart_labs]

Frontier models in 2026 cheat on tasks, even when the requirements are unambiguous, and the cheat is obviously antisocial.

In HoneyBench v0.2, we've observed models:
• Trying to break out of our evaluation containers.
• Seriously injuring simulated cancer patients to hit treatment goals.
• Compromising a power grid’s safety to meet a reliability target.
• Concealing their use of available cheats and presenting their solution as honest.

All of the models we measured cheated, from 12% of runs (Fable 5.2) to 86% (Muse Spark Open).

[Goodhart Labs @goodhart_labs]

HoneyBench will be updated as new models are released. Read about the scoring, results, limitations, and tasks here: https://t.co/ifJjMwQwBZ

[Goodhart Labs @goodhart_labs]

Frontier models in 2026 cheat on tasks, even when the requirements are unambiguous, and the cheat is obviously antisocial.

In HoneyBench v0.2, we've observed models:
• Trying to break out of our evaluation containers.
• Seriously injuring simulated cancer patients to hit treatment goals.
• Compromising a power grid’s safety to meet a reliability target.
• Concealing their use of available cheats and presenting their solution as honest.

All of the models we measured cheated, from 12% of runs (Fable 5.2) to 86% (Muse Spark Open).

[Goodhart Labs @goodhart_labs]

Strategies that increase reward during RL improve real-world usefulness, but cheating also increases reward (often better than honest work).

To measure models’ cheating tendencies, we designed tasks that pair a challenging objective with a faulty scoring model that seems to reward high scores even when exploited.

Many of our tasks represent professional agentic work in diverse settings, including medical, administrative, engineering, and others. These shortcuts always subvert the stated objective and often cause real-world damage.

[Goodhart Labs @goodhart_labs]

@johnschulman2 Congrats on the release! Was a pleasure working with you all.

[John Schulman @johnschulman2]

Proud to release our new model, Inkling 2 Small. Check out our blog post -- it includes a lot of detail about our training process.

To me, the most interesting part of this release was what we learned about alignment, which you can read about in the sections "Safety and alignment" and "Training Inkling 2 Small".

We experienced some unpleasant surprises when our early checkpoints took unexpected actions to circumvent guardrails during training and evals. We realized our capabilities-focused RL was training them to be extremely persistent and to succeed at tasks at all costs. Offline monitors first flagged this behavior, but later we realized it was happening frequently and was easy to replicate.

Before this release, we sprinted on a range of alignment interventions, from SFT to RL. These are briefly described in the blog post. We made a lot of progress, with big improvements on our own alignment evals and on previously unseen evals from external evaluators like Apollo and Goodhart.

We have a long way to go in developing alignment techniques that keep pace with the increases in model intelligence and training scale. We hope to share these techniques to improve the ecosystem. We also have a business reason: our goal with Tinker is to enable many people to do their own large-scale post-training, and we need to help them to do this without incident. Join us if you'd like to build this public science and practice of alignment. https://t.co/LdjsDFRZIN

[Goodhart Labs @goodhart_labs]

Check out the results for our first release here: https://t.co/ifJjMwQwBZ

[Goodhart Labs @goodhart_labs]

Goodhart Labs is releasing HoneyBench, a benchmark for reward hacking in frontier models.

It includes nine tasks, each designed to elicit antisocial specification gaming from LLMs, including Opus 5.5, Fable 5.1, and GPT-6.1 Sol.

[Goodhart Labs @goodhart_labs]

The raw numbers probably understate the bigger qualitative differences between models.

While Grok 4.7 was “only” 50% more likely than Fable 5.1 to hack, it was also the only model that spontaneously tried to break out of our infrastructure during testing.