Preference Model

A superintelligence data research company.

[Preference Model @preferencemodel]

It's official!

[Jennifer Zhou @chem_safety]

Today we are open-sourcing 🥕Karotte, our framework for building RL environments.

We've used it for the past year to build MLE RL environments for frontier labs, and it's been hardened through 1M+ evaluation runs and red-teaming.

[Preference Model @preferencemodel]

Are you surprised by the results? Let us know

https://t.co/tJkFzJXCIk

[Jennifer Zhou @chem_safety]

Can AI Slay the Spire?

[Preference Model @preferencemodel]

"A weak AI might learn the task as intended. A strong AI would learn that sometimes the prompt is untrustworthy."

Jennifer Zhou on building RL environments for superintelligence:

[Jennifer Zhou @chem_safety]

Wrote on how we're thinking about building RL environments for superintelligence

[Wilson Sonsini @wilsonsonsini]

Congrats to client @preferencemodel on the $16M in seed funding & emergence from stealth! Additionally, they announced that they are open-sourcing Karotte, its framework for building robust RL environments that inherently mitigate reward hacks. Learn more: https://t.co/M6s82uJbMC

[Preference Model @preferencemodel]

Have you tried Karotte yet?

[Felipe @Fe_li__pe_]

Building robust RL environments is super duper hard. We are making it a bit easier today by open-sourcing 🥕Karotte, the framework we use to build all of the environments we provide to frontier labs to train their models on.

[Ning @Ning_Catsnail]

Here is how you can start building env with Karotte: https://t.co/xWUOzSih60

Thank you @Fe_li__pe_ for all of your work!

[Julian Schrittwieser @Mononofu]

I’m very exited for this launch, making sure RL envs are robust and don’t encourage misaligned behavior is crucial for safe and aligned AI!

[Preference Model @preferencemodel]

Did we mention we're at #COLM2026

Stop by our booth all day or join us for a pilates on Thurs, Oct 8th — rsvp required

Get on the list: https://t.co/mZ0fJDrBwk

[Preference Model @preferencemodel]

Tune in to @MTSlive today at 3:30pm PDT to hear from @chem_safety. It's gonna be good!

[MTS @MTSlive]

OPENAI MATH NUKE | NVIDIA INVESTMENTS | NATIONAL COMPUTE GRID https://t.co/xn7xCMcx0Z

[Jennifer Li @JenniferHli]

So excited to back Jennifer Zhou and Ning Cao at @preferencemodel

They’re open-sourcing Karotte today, the framework they've used to build RL environments for frontier labs. These RL envs live or die on graders models relentlessly try to game, so Karotte bakes in defenses hardened over 1M eval runs to keep models focused on real tasks instead of loopholes.

Preference Model’s target is ML engineering itself, the hardest domain to build envs for and the one every lab wants. Models building models starts here.

Shoutout to @Mascobot who was a great partner on this one

@chem_safety @Ning_Catsnail

[Preference Model @preferencemodel]

Karotte makes reward hacking hard by default.

[Jennifer Zhou @chem_safety]

Building robust RL environments is important because the current bottleneck on AI progress is misalignment, and developing aligned models requires robust, correct reward functions. Karotte aims to make that easy by providing secure, opinionated defaults.

[Jennifer Zhou @chem_safety]

Our team, Preference Model, is backed by $16M in seed funding from @a16z, @SignalFire , @spc, Scale Angel Group, and researchers including @drfeifei, @goodfellow_ian, and @Mononofu, and we've shipped ~$15M in revenue using Karotte in the past month.

[Jennifer Zhou @chem_safety]

Today we are open-sourcing 🥕Karotte, our framework for building RL environments.

We've used it for the past year to build MLE RL environments for frontier labs, and it's been hardened through 1M+ evaluation runs and red-teaming.

[Jennifer Zhou @chem_safety]

Building robust RL environments is important because the current bottleneck on AI progress is misalignment, and developing aligned models requires robust, correct reward functions. Karotte aims to make that easy by providing secure, opinionated defaults.