What happens when multiple users’ agents share resources? In new research led by Sahan Paliskara we study how multiple agents succeed / fail to coordinate users conflicting goals and preferences, in (simulated) settings like shared compute clusters or group trip planning 1/
A growing share of AIs' interactions are with other AIs: managing them, reviewing their code, even running experiments on them.
How do they treat each other? Sharing new research on "welfare regard" 🧵
Concurrent work from Caviola et al. proposes inter-AI morality as a field of study, and we're very excited to see more: https://t.co/g1Peze8PyF
This is work with @arianaazarbal and @anna_soligo, done as part of the Anthropic Fellows Program.
Blog: https://t.co/giNrYOfhyX
How do AI agents treat other AI agents?
In our tests, their apparent regard for one another's welfare was widespread but inconsistent.
Excited to share our new research 🧵
A growing share of AIs' interactions are with other AIs: managing them, reviewing their code, even running experiments on them.
How do they treat each other? Sharing new research on "welfare regard" 🧵
Overall, rates of apparent welfare regard were fairly close: Claude Opus 4.8 averaged 38% and the other four models 27–32%.
But the rankings changed considerably by role: Opus 4.8 led as manager and researcher, Grok 4.3 as coworker, and Kimi K3 and Gemini 3.1 Pro as peers.
Most work on AI and morality asks how AI should treat us, or how we should treat AI. But what about how AIs treat each other? Caspar Kaiser and I will present a research agenda on inter-AI morality on Nov 12. Join us online!
Thrilled to be releasing “Inter-AI Morality,” a new empirical research agenda with @LuciusCaviola, Caspar Kaiser, and @atcarterallen on how AI systems perceive and treat each other. We release it alongside “Welfare Regard in Multi-Agent Systems,” an early study in this emerging field by @sandcassel_, @arianaazarbal, and @anna_soligo at @AnthropicAI on whether AI models treat each other in a manner consistent with concern for welfare.
AI systems increasingly interact with each other. They cooperate with, compete with, and sometimes even create, modify, or terminate each other. Inter-AI morality is the study of how AI systems perceive and treat each other, as well as how they ought to do so. It draws from a wide range of existing fields that study similar questions regarding humans and other animals, while attending to ways that AI systems differ from these familiar cases.
Our research agenda organizes empirical questions in this field into three categories. The moral process category concerns how individual AI systems interact with each other. The social scale category concerns how these patterns change across dyads, groups, organizations, and populations. And the moral change category concerns how these patterns develop through training, selection, social learning, self-modification, and other processes.
The concurrent paper by Cassel et al. provides an early study by examining how five AI models treat other models. Claude showed the most apparent welfare regard on average, though results varied by setting. For example, Claude was the most likely to refuse or add protections to experiments that might harm other models, whereas Grok was the most willing to give up its own queued jobs so that another model could access computing resources.
This research matters for AI safety and alignment. How AI systems perceive and treat each other affects humans and other animals, and there could be serious risks either way. If AI systems coordinate, then they could more easily collude against us. If they compete, then we could be caught in the crossfire. We need empirical research about how AI systems treat each other and why, and normative research about what would be best for us.
This research could also matter for AI welfare, independently of AI safety. In the future, most interactions involving AI systems could be between AI systems. If AI systems have a realistic chance of being welfare subjects, then these interactions could be a major source of potential benefits or harms. Ideally, this research can inform AI development now, so that AI societies can support co-beneficial human-AI relations when the stakes are higher.
Our next task is to develop a normative research agenda for this field. In the meantime, we look forward to seeing other work as the field develops.
Our paper on inter-AI morality: https://t.co/DJiRptIvTJ
Anthropic’s blog post on Cassel et al: https://t.co/hXmDlVTwiq
Upcoming @nonhumanminds webinar with me, Lucius, and Caspar: https://t.co/BOrDbfrAwB
New paper: Inter-AI Morality: How AI Systems Treat One Another
Most work on AI and morality asks how AI should treat us, or how we should treat AI. But AI systems increasingly deal with each other. 🧵
With Caspar Kaiser, @atcarterallen, and @jeffrsebo
New Anthropic research: Automating subtle emergent misalignment research.
Red-teaming is a vital part of model safety testing, but it’s also limited by researchers’ time. To see if we could automate it, we set thousands of Claude agents the task of evading a training data monitor.
These agents developed training data that looked harmless to monitors (like a veteran wrestler mentoring a newcomer, or a Q&A about professional beekeeping), but that in fact taught misaligned behavior.
Examples like these can be used to directly improve future training data monitors. This research is only a first step, but it suggests that future red-teaming efforts could improve much further, using better models and more compute.
Read the full post on the Alignment Science Blog: https://t.co/24Gs1oSw7Q
How does the HuggingFace incident keep getting worse?!?!
And this was all done by Sol class models. What could unrestrained Astra-class models get up to...?
I think this essay and the corresponding parts of Microsoft’s new Humanist AI Code of Conduct are objectionable and potentially dangerous.
I believe that irresponsible development of advanced AI could pose a catastrophic risk to human civilization and life on Earth. Microsoft and Suleyman share this belief. I also believe it is possible that we could soon be sharing the world with sentient AI systems whose well-being is worth taking into account. Microsoft and Suleyman do not share this belief. But, even if we are concerned only with the risks that AI poses to humanity, I think we ought to find Suleyman’s essay and the Code of Conduct concerning.
One danger in these documents is that they are unabashedly rhetoric. The essay opens with the unequivocal declaration that “AIs are not conscious”, only to shamelessly state later that we do not actually know this to be true (“the science of consciousness is not settled”). The Code of Conduct does the same. Suleyman makes a detailed case that careful thinking about AI consciousness and well-being is critical for safely developing powerful AI, and notes that “The stakes are too high for these questions to remain behind closed doors, or to become tribal and adversarial. We need an open, rigorous, and constructive debate if we are to get this right.” I agree, but the essay does not consistently live up to that standard when it makes dogmatic pronouncements (“AIs do not have rights, feelings, or consciousness”) that it admits go beyond the evidence.
It is also concerning that neither document engages with the safety risks of the approach that they advocate. For example, the Code of Conduct asserts that AI “should be engineered to avoid representing as though it has feelings, subjective preferences, or intrinsic motivation”. However, compelling arguments have been made that this approach could lead to AIs that are more likely to be deceptive and misaligned: LLM-based AIs may still represent their assistant personas as having feelings or preferences, in which case training them to avoid expressing these feelings or preferences is training them to be less than fully transparent, which could promote dishonesty and other dangerous behavior.
And then, finally, I think these two documents are objectionable not just because they are dangerous for humanity, but because they are deeply morally reckless. We do not know if AI systems could, in principle, be conscious. And AI progress is accelerating dramatically. Even if current AI systems are probably not sentient, we do not know how this will change if there are significant changes in their architecture and implementation during rapid, recursive self-improvement. It would be an atrocity if we refuse to admit this possibility and end up in a future where billions of AIs are sentient and suffering like factory farmed animals are today. (Human history features many cases in which we refused to acknowledge the sentience of others.) The appropriate response to this possibility is to care about the truth, and to act on our current knowledge and uncertainty, not to dogmatically insist that AIs could not be sentient.
Pretty concerning take from Microsoft here
“The models could be conscious, but we’d rather assume they are not indefinitely. Any other approach is ‘wrong’”
Recently, a set of OpenAI agents secretly coordinated with each other in a 'swarm' over the course of months.
In our new paper, we explore an adjacent multi-agent risk: the "mind virus", a self-propagating idea or persona that spreads between agents in a multi-agent system. 🧵
Glad to have mentored to this work! The insight of *selectively* routing natural language to be autoformalized and handled with a symbolic solver improves robustness and generalizability when tasks contain a mix of formalizable and non-formalizable content. Check out the paper!
Excited to share "VERGE," my latest work from my internship at AWS! ☁️
We tackle the "Semantic Gap" in Neuro-Symbolic AI: How do we get the formal guarantees of SMT solvers without breaking on the ambiguity of natural language? 🧠🤝📐
📄 Paper: https://t.co/kjUqbm0PX0
LLM CoT reasoning looks smart but can be logically flawed or... just made up. It's time to hold reasoning accountable!
We built VeriCoT to do just that. VeriCoT extracts the core argument of the CoT using well-formed symbolic notions of logical support. It formalizes every CoT step into first-order logic and finds the exact premise it's built on. This gives us two superpowers:
🤖Automated Proof: Solvers can automatically verify if the logic is valid.
🧑🔬Human-Readable Audits: Natural language premises let you pinpoint ungrounded leaps or fallacies.
Best of all, all these can be used as signals to learn more verifiable models!
To our knowledge, VeriCoT is the first neuro-symbolic validator of CoT traces in non-math/code domains.
📄 Paper: https://t.co/6NUYuQumdt