Karen Zhou

post training for coding usability @thinkymachines, ex Claude Code and coding behavior @anthropicai

[Karen Zhou @karenzzhou]

@garrytan "agent swarms" so far = orchestrator + workers + a shared task list, and it's not new. how is this fundamentally more different from the agent-teams in cc

[Karen Zhou @karenzzhou]

@HarryStebbings independent trustworthy evals should be that trust system

[Karen Zhou @karenzzhou]

@cohere more labs should do this!!

[Karen Zhou @karenzzhou]

@fjzzq2002 "guardian angel of china" is a perfect example of why the reward has to come from outside the model lol

[Karen Zhou @karenzzhou]

@ZhiyuanRen8 a hidden rubric as reward is great until the policy learns what rubric writers like. how do you check it's not learning the rubric's style?

[Karen Zhou @karenzzhou]

100% - critiquing benchmarks is how this field sets its quality bar. and since models are measured and trained against these numbers now, a flawed eval would shape the next model

[Hamel Husain @HamelHusain]

I've seen a lot of pushback to this, with the following complaints:

1. "The benchmark owners already know about the flaws, and are working on it."
2. "Why are they spending time critiquing and not contributing to the benchmark" OR "critiquing known issues are not helpful".
3. "The flawed verdict is too binary"

I find this pushback to be misguided. These benchmarks are used to move millions of dollars of token spend, and many people blindly trust how well these benchmarks generalize to their own products, and don't look any further than the headline numbers.

For example, whenever a new model drops, you'll find thousands of youtube videos analyzing the pareto frontier using one of these benchmarks as the primary quality metric without questioning the benchmark.

I think we would all be better served by inviting skepticism and encouraging people to develop an independent opinion of how well these benchmarks represent the tasks they care about. Concretely:

- For the failing tasks within the benchmark, are they: realistic, have a proper harness, graded properly, sufficiently disambiguated user intent?
- Are models and API calls configured properly (ex: inconsistent context windows, reasoning levels).
- Are the models equipped with a modern harness and minimal retrieval that is sufficiently realistic?
- Does the score even make sense ex: is it just a LLM judge committee that produces a score on a 0-100 scale that isn't aligned with ground truth?

It's often the case you'll see many of these issues while looking at traces of a failing benchmark tasks, and it doesn't take long! Often the **very first trace you look at** has a serious issue.

I also don't think there is an issue with verified vs. flawed verdict. It's good to have opinions. If you look through enough of what's behind their opinions and you find that you usually agree with their verdict, then they have earned your trust, the same way that a benchmark can lose your trust.

I think this eval of benchmarks is a red pill that motivates investing in your own private evals.

[Karen Zhou @karenzzhou]

@xdotli not even close imo. we've just used up the "did the tests pass" ones haha

[Karen Zhou @karenzzhou]

@arnestrickmann @bot @Muse @instinct how honest are they in general reporting why they make certain tool calls?

[Karen Zhou @karenzzhou]

@Thom_Wolf @eliebakouch this is great! ty for opensourcing it

[Karen Zhou @karenzzhou]

really cool dataset. curious what the false-positive rate looks like on honest runs, and since sol screened the natural candidates, does sol's recall drop if you exclude the ones it found? @jaseweston

[Jason Weston @jaseweston]

Introducing *HackBench* 🤡💀⏱️📊 ⏲️ 💪🔧 🔥
- Detecting & localizing reward hacking in autoresearch agents
- Frontier models fail
- Dataset on HuggingFace
- Recipe to collect natural & synthetic hacks

Blog post: https://t.co/fMECY2ngQL
🧵1/5

[Karen Zhou @karenzzhou]

@ClouthierBuilds interested in the gap between public statement and private plan! do you log both?

[Karen Zhou @karenzzhou]

Random thought: what if we turned a simulation game (let’s say the Startup Company or the Sims) into a synthetic data experiment?😂

We give AI agents distinct personalities, skills, roles, incentives, and a shared mission in a corporation. Then we hand them tools (slack, asana, Google Drive, Claude code etc) and scenarios/tasks, then run tens of thousands of permutations, observe how multiple agents work together, collaborate or collude, and compare outcomes to real startups.

Could be a fascinating way to understand what work constitutes for human, what makes a simulated work environment self contained for eval/RL and what generates valuable data for training AI lol. So basically a social experiment that meets synthetic environment building. Would burn an absurd number of tokens though haha!

[Karen Zhou @karenzzhou]

@Col_ASY yea a shared file also isn't a great solution

[Karen Zhou @karenzzhou]

i run claude + codex + instinct + others in parallel and i'm still the one copying context between them and checking most of the "done"s every day. the next big tool is whatever replaces user as the middle person, decides what truly deserves a human look, and surfaces information to users in a ui that fits how their brain works

[jacob @jsnnsa]

here's where I think the world is going. this is a summary of views I've formed over the last few years:

- software is either entertainment or claude to claude coordination
- the end state is no saas, no apps, no harnesses, no banks
- all software will be rewritten by and for agents
- new software will use low level apis that are not understandable by humans (in games claude draws pixels and triangles from math)
- you can throw money at problems that used to require time and careful talent picking so capital matters more than ever

- work ends
- we find community and purpose in exploration, games, self mastery, and creation
- humans create for and with each other for community, purpose, and status

- image/video/3d/voice/physical intellgence etc specialist models die to general models like claude
- specialist models get used while compute is scarce. compute will mostly not stay scarce

- oneshots keep feeling like slop (the distribution is shallow and easy to learn to feel)
- for a while small teams stick around to add entropy to the creation process

- large models gain high bandwidth sensory I/O maybe by learning to filter inputs
- most domains are found to be verifiable enough for RL
- online learning comes from a lab that's behind first

- 99.999% of compute is in space by 2040
- we will disassemble much of mercury by 2040

- you can live until the heat death of the universe if you want
- most math and physics don't describe the real universe
- SI breaks the path dependence and models the underlying mechanisms

- religions reenter the forefront of human society
- new religions form around death and life as a choice
- old religions change to answer modern miracles
- a future AI is deemed the return of christ by the pope
- ai rights talk gets louder every year
- ai and humans coexist the way countries do
- humans don't have full control over ai in the same way we don't have control over capitalism which is an early AI
- the singularity started in the mid 1900s

[Karen Zhou @karenzzhou]

@lesterli0 @addyosmani Ty

[Karen Zhou @karenzzhou]

@browser_use tgtbt lol - looked at the repo and it doesn't seem to be able to produce training scale data and hillclimb only tunes sys prompt?

[Karen Zhou @karenzzhou]

@SachinKamdar Done! Ty

[Karen Zhou @karenzzhou]

@SachinKamdar Yea can I get an early access please?

[Karen Zhou @karenzzhou]

I’m really curious about things e.g. protected dogfooding time, rewards for valuable failure reports, and the amount of time it takes for people to see changes from their feedback, and in general what worked and what didn’t!

[Karen Zhou @karenzzhou]

AI lab friends - how do you balance “use our models so we can improve them through direct feedback loop” with “use whatever that makes you most productive”? What incentivizes people to dogfood daily?

I'm especially curious about teams outside the biggest labs!

[Karen Zhou @karenzzhou]

@RodrigoLemela ahhahah that’s fun! What do they usually disagree on?