Eric Gan

[Eric Gan @ejcgan]

@w01fe @abhayesian Yeah if the model isn't robust enough at reward laundering then it would get selected against. But some pressure during RL could also push the reward laundering propensity upwards.

I think I'm most worried if the propensity happens rarely to remove rare undesired behaviors.

[Abhay Sheshadri @abhayesian]

New Redwood Research post

LLMs can use RL training on one task to teach themselves a different, unrelated capability. In a form of exploration hacking, the model outputs answers that get reward only when it succeeds at a side task it wants to learn. We call it reward laundering.

[Eric Gan @ejcgan]

Paper: https://t.co/Q1EKhHPSht
Code: https://t.co/utvVw2RGEH
Blog post: https://t.co/5c550z1tG7

[Eric Gan @ejcgan]

Can frontier LLMs and humans catch sabotage in ML research code?

In Auditing Sabotage Bench, I added subtle sabotages to 9 existing ML codebases which change a key finding of the research. Neither LLMs nor LLM-assisted humans reliably caught them.

[Eric Gan @ejcgan]

Most surprising finding: the hardest red-team skill isn't making sabotages, it's predicting which single change actually flips a result. Many of my attempted sabotages didn't move the outcome of the experiment.

[Eric Gan @ejcgan]

If Tinker didn't exist I likely wouldn't work on this project at all. I'd tried renting GPUs before but scaling multi-node training is always a pain. I was basically waiting for something reliable to exist before diving into RL work

[Eric Gan @ejcgan]

I've been using Tinker at Redwood Research to RL-train long-context models like Qwen3-32B on difficult AI control tasks - specifically teaching models to write unsuspicious backdoors in code similar to the AI control paper. Early stages but seeing some interesting backdoors 👀

[Eric Gan @ejcgan]

What I really like about Tinker is that you can directly look at the code, understand it, and modify it. It's much more flexible than other RL APIs, and you don't have to deal with the infra yourself. It's also just really nicely written.