Felipe

[Felipe @Fe_li__pe_]

Building robust RL environments is super duper hard. We are making it a bit easier today by open-sourcing 🥕Karotte, the framework we use to build all of the environments we provide to frontier labs to train their models on.

[Felipe @Fe_li__pe_]

Insight 5: Slay the Spire is so good

[Felipe @Fe_li__pe_]

A fun one to start the week! Testing frontier models' long-horizon reasoning by letting them play Slay the Spire!

https://t.co/zsxG45ug9r

There's also a cool companion site with all the run recordings and transcripts:
https://t.co/18ZNkSUkD8

[Jennifer Zhou @chem_safety]

Can AI Slay the Spire?

[Felipe @Fe_li__pe_]

Insight 4: Nobody knows how to play the Watcher

[Jennifer Zhou @chem_safety]

Can AI Slay the Spire?

[Felipe @Fe_li__pe_]

@willcb @jdchawla29 @ChinmayKak That's correct

[Felipe @Fe_li__pe_]

I wish Firecracker had support for GPUs/TPUs. I've been banging my head against how to harden those kinds of tasks for a while. In prod, we currently use gVisor for accelerator tasks but I'm not very happy with the kinds of tools it gives us to confine the agent. PID namespaces work but cgroups don't :\

[Felipe @Fe_li__pe_]

@jdchawla29 I think the better approach is to tell the agent how its stuff should be started, then kill all processes before grading, and then run the agent's submission using binaries that the grader controls etc.
Haven't looked into live-database-cutover in detail tbh

[Jennifer Zhou @chem_safety]

Wrote on how we're thinking about building RL environments for superintelligence

[Felipe @Fe_li__pe_]

@sanyamsatia There are certainly things in Harbor that Karotte can learn from! So far it's been mostly driven by our internal demand, and internet access is on the list but hasn't made it to the highest priority yet. The chart is about the defaults though and that seems about right?

[Felipe @Fe_li__pe_]

First off, AgentEnv looks awesome for building virtual worlds (that seems to be your focus?). Will have to look into it more but I'm sure there's something we can learn from you guys.

Regarding the chart, please correct me if I'm wrong but it seems that
- NetworkPolicy defaults to ALLOW_ALL in a local sandbox
- verify_sandbox runs whatever is one the path, so the agent could hijack it
- env_outcome_verifier reads from an MCP server that doesn't have auth, so the agent can mess around with it by setting the role header
- there's no limit on number of processes, so a fork bomb crashes the run

[Felipe @Fe_li__pe_]

@maxisawesome538 I know, right 😂 Karotte is v3 of what we started building envs with a year ago, just a lot less janky lol

[Felipe @Fe_li__pe_]

Building robust RL environments is super duper hard. We are making it a bit easier today by open-sourcing 🥕Karotte, the framework we use to build all of the environments we provide to frontier labs to train their models on.

[Felipe @Fe_li__pe_]

@jatingargiitk @a16z @preferencemodel Yeah, maintaining envs over time is certainly a challenge. Karotte comes with an updater (`uvx karotte update`) that automatically updates existing environments to get Karotte's newest hardening features.

[Felipe @Fe_li__pe_]

@alvind319 🥕🚢

[Felipe @Fe_li__pe_]

Individually hardening every environment against all possible hacks is a non-starter if you are building environments at scale, like we do at Preference Model. That’s why we created Karotte, an opinionated env building framework that automatically mitigates many of the most egregious hacks. Try it out, it’s pretty cool! https://t.co/ZxyYslhsao 🥕

[Felipe @Fe_li__pe_]

Concretely, when the agent is done with its work and you start your grader, you have to assume that the agent left behind an absolute mine field. That file you asked it to submit? Might be a symlink to the hidden solution or a 100GB sparse file to OOM your grader. The possibilities are endless 🫠