Jennifer Zhou

im doing a startup!

[Jennifer Zhou @chem_safety]

We observe that there are clear reasoning deficiencies across models. They often make very obviously poor decisions within a run and fail to learn in-context from their mistakes. They misunderstand game mechanics even after being shown how they work.

[Jennifer Zhou @chem_safety]

Can AI Slay the Spire?

[Jennifer Zhou @chem_safety]

A good long-horizon benchmark needs clear rules, an easy-to-measure score, and lots of decisions where early choices have impact later. A sts run involves hundreds of decisions at different time scales.

[Jennifer Zhou @chem_safety]

The only correct terms for this are moving a meeting "clockwise" or "counter-clockwise"

[Chana @ChanaMessinger]

Is moving a meeting back moving it

[Jennifer Zhou @chem_safety]

https://t.co/NLh0aAN2s6

[Jennifer Zhou @chem_safety]

Wrote on how we're thinking about building RL environments for superintelligence

[Jennifer Zhou @chem_safety]

@xlr8harder I think the whole industry could do better. From what I've seen, the bar really is on the floor. One step at a time.

[Jennifer Zhou @chem_safety]

@xlr8harder Is this crash-out directed at us or at the labs whose specs we have to follow? If the later, I certainly agree! It would be a lot better for the grader to be a thing that lives outside the task container that's able to spin up isolated VMs containing tainted agent code.

[Jennifer Zhou @chem_safety]

@dubrovnik07 Env-ishness trades off against security guarantees. I think the later is more important as models get smarter.

[Jennifer Zhou @chem_safety]

Today we are open-sourcing 🥕Karotte, our framework for building RL environments.

We've used it for the past year to build MLE RL environments for frontier labs, and it's been hardened through 1M+ evaluation runs and red-teaming.

[Jennifer Zhou @chem_safety]

@dubrovnik07 Strong opinion loosely held: I don't think env-unawareness is possible. Even a few generations ago these models could identify authors from writing style. Identifying whether you're in an env is way easier than mapping writing style -> author.

[Jennifer Zhou @chem_safety]

@Mononofu Thanks Julian!!

[Jennifer Zhou @chem_safety]

@hampterfi We generally recommend using Firecracker for sandboxing, but apart from that, these are security issues best solved at a lower level of the stack e.g. by patching bugs in the Linux kernel and by automated monitoring of rollouts.

[Jennifer Zhou @chem_safety]

@yanjo115 Yesss, let me know how it goes!

[Jennifer Zhou @chem_safety]

@earakely Yes, the two packages are aimed at solving different problems. AgentEnv is more about building out a realistic virtual world, whereas Karotte is more about getting the graders and reward signals to be correct and free from reward hacks. You can see this reflected in the docs

[Felipe @Fe_li__pe_]

Building robust RL environments is super duper hard. We are making it a bit easier today by open-sourcing 🥕Karotte, the framework we use to build all of the environments we provide to frontier labs to train their models on.

[Jennifer Zhou @chem_safety]

docs: https://t.co/Gb5ZCd8bGY
technical deep dive: https://t.co/ol9IqITqaS
join us! https://t.co/S0QnnqimLQ

[Jennifer Zhou @chem_safety]

Training on even small amounts of faulty RL envs can misalign your models. Let's stop doing that!

Karotte will be the first of a few components in our stack that we are releasing over the next weeks. Our hope is that these tools will help the world produce better, safer RL envs.