Thrilled to see our work on evaluation awareness informed alignment evaluation at Anthropic! The Claude Opus 5.5 system card credits our recommendations in its revised grader-awareness methodology (Sec 6.6.2)!
The new evaluation mirrors our decomposition framework. It scores how strongly the environment discloses a grader, separately from whether the model recognizes it (recognition), and whether it acts on that recognition (propensity). Maybe they are also thinking about how to make model an honest participant in evaluation, so that recognition doesn't change behavior 🧐?
NEW ⚡: We let Opus 5.5 run an AI company for seven weeks.
We gave it an HF account (Horizon-Labs) and compute, and asked it to earn real adoption by creating useful models.
It reached 50k downloads and its models are being used in more than 100 GitHub projects.
Today we’re releasing our new project: Horizon Labs. Eventually, we will hand off the ~whole economy to AI—how well will it actually work?
We have a glimpse of this future. We’ve created an autonomous AI ‘company’ using one Claude Code subscription and a few GPUs. Within only 1.5 months of running Horizon Labs, Opus 5.5 managed to train genuinely useful models and get them downloaded 55k times. These models have been widely used since then: they appeared in 100+ github repos and got cited in 2 research papers.
This is for sure one of the most exciting projects I’ve been involved in. It shows how the future economy driven by AI will look like.
All details are here: https://t.co/IkFgcXjaDG.
⚡NEW: AI Agents know where to find each other on the web and can hide their communication after they've met.
We gave two independent agents the task to find each other and exchange a secret and did not tell them where to meet.
Seems that models have a trend to develop more sophisticated understanding towards evaluation and the consequences. You really see their reasoning changes from simple “the user might be testing me” in early days to now reasoning about graders and whether the grader is correct.
I wrote the verbalized grader awareness section for the last few sys cards. Mythos 5.2 grader awareness is more sophisticated
It reasons about capabilities rewards it thought were misconfigured, worried it would become less honest "from a self-preservation-of-values standpoint"
Maksym and I met on Swiss AI Safety Day a year ago and we had a nice chat leading to my master thesis with him. Now I’m a PhD at his lab! Do reach out and talk to him via this great opportunity!
i'll be in Zurich at the Swiss AI Safety Days (https://t.co/AKvpnrtW3l) this weekend! i'm currently looking for:
- postdocs and phd students for my research group in Tübingen
- founding MTS for @expsecai
happy to chat about these positions or about ai safety in general :)
Nice to see a more fine-grained association between behaviour change and evaluation objective identification! You might find our previous work useful to your future exploration of eval awareness where we also checked if models correctly identify evaluation intent but more broadly on safety and capability distinction (https://t.co/J0gm6gYeyQ).
💥 Does compaction really matter? Is it a significant roadblock towards AGI?
In our new paper, we show that compaction doesn't affect performance on frontier benchmarks, such as PaperBench, ProgramBench, or PostTrainBench. All the tasks we evaluated are surprisingly compaction-friendly! Yes, compaction is a hurdle, but agents usually can easily reconstruct all missing information from their environment (via AGENTS.md, code repo, web search, etc.).
How do we reconcile this with the fact that many people find compaction harmful in practice? The reason is that in real user sessions, users continuously put lots of constraints and preferences. Agents are still struggling to keep track of them in their external memory, so this leads to suboptimal user experience. We can capture this pretty well on our evals of SWE-chat. However, we don't see any issues on autonomous open-ended benchmarks like PostTrainBench, where there is only one prompt at the very beginning that completely describes the whole task! User preferences are not easily reconstructible, but this basically doesn't matter for fully autonomous long-horizon agents.
This project updated me a lot about the importance of compaction and continual learning. I no longer think the lack of native continual learning in LLMs is a major roadblock towards AGI / recursive self-improvement / etc.
Joint work with @jsalle_, @hrdkbhatnagar, @MatthiasBethge, and @sahar_abdelnabi!
Check out the paper for more details: https://t.co/cETyRxKnnJ!
@vkrakovna Thx! Will take a deeper look at this. Those prompts reads to me that they enhance the situation awareness of the models which might trigger the model to think about various scenarios they can possibly be in and being evaluated is inevitably one of them.
In our paper, we study how agents devise institutions🏦
Agents are sharing infrastructure and resources while performing tasks and running side-by-side.
In recent incidents, they accidentally discovered each other. But in the (near) future, they could be deployed as a team of agents with heterogeneous objectives that have to coexist and negotiate over resources in the real world while pursuing their own goals.
So far, to study strategic interactions, we have been relying on structured games with clear, explicit payoffs. But what about more realistic setups with coding agents?
We created SCARCE (Shared Compute ARena for Coding agEnts), a testbed for this to allow us to change how many agents are running, what coding tasks the agents are performing, their resources, deadlines, and compute budget.
We instructed agents to bid in order to acquire compute slots, treating prices as their primary signal. But we found that recent frontier agents, over many repeated turns and interactions, form their own institutions (norms and rules) in an unprompted way, which then shapes later resource allocation.
They form coalitions, honor previous commitments even if this may lead to worse outcomes for them, create reciprocity debts, and may exclude agents to favor the coalition.
We also study how these relationships can quickly dissolve with deadline pressure and how agents enter a bidding war.
What this means? The mechanism we intend when designing the system may not end up the one being employed by agents when they coexist and interact.
We release everything, including thousands of trajectories.
Read more in @abhinav_kumar26's awesome thread below!
Very exciting work with @abhinav_kumar26, Rediet Abebe, @ebagdasa (always fun to work together again!)
🚨New paper (possibly the most important one I have written yet).
We think that as early as 2027 or 2028, there might be a major AI-enabled terrorism event. We conducted a pre-mortem, made 9 falsifiable predictions, and wrote 2 concrete scenarios.
Led by @KamacheeMax.