New research! Some AI capabilities are both helpful and dangerous. E.g., knowledge of virology can be used to create life-saving vaccines or deadly pathogens. We introduce GRAM, a training method that puts dual-use capabilities (like virology) into removable modules.
We found a way to jailbreak a closed-source model using the next-token log-probabilities returned by its API.
SAGE uses those probabilities to identify a signal associated with refusal, without access to the model’s weights or internal activations.
Truly “Military-grade AI” requires solving the Alignment Problem
Excited to share more today in @JoinFAI’s Futurecraft, which is catalyzing some great thought from a bunch of others I recommend reading on how transformative superintelligence changes everything
@keenanpepper's SelfIE adapter achieved a 76% average pass rate across seven types of WorkspaceBench tasks, compared with 33% for J-Lens
early results from a quick research sprint done for @AIAF_org. we're not actively working on this anymore but happy to discuss if relevant to anyone
Can one small adapter read different kinds of hidden concepts inside an AI model?
In a quick research sprint, @AEStudio's @keenanpepper tested SelfIE across seven types of tasks. We’re sharing the early findings in case they’re useful to others.
Can one small adapter read different kinds of hidden concepts inside an AI model?
In a quick research sprint, @AEStudio's @keenanpepper tested SelfIE across seven types of tasks. We’re sharing the early findings in case they’re useful to others.
Have an AI alignment hypothesis but no time to test it?
AE Studio’s @IARPAnews-funded CRUCIBLE provides 60+ engineering hours, compute and a stipend for 2-week sprints. You commit ~2 hours/week.
Apply by Nov 15: https://t.co/Qy0tT3poyl
Six questions: https://t.co/XF44Y1kWZM
@AEStudio@iarpa Oops, meant to tag @IARPAnews@AEStudio + @IARPAnews are funding 6 researchers with eng support, ML expertise, compute + stipend Looking for ideas that can show signal in 2 weeks (extendable)
Great ideas from anywhere. Apply by 11/15 → https://t.co/wsSuf4X5MB
Most alignment researchers have more ideas than bandwidth to test them
@AEStudio + @IARPA are funding 6 researchers with eng support, ML expertise, compute + stipend
Looking for ideas that can show signal in 2 weeks (extendable)
Great ideas from anywhere. Apply by 11/15 →
https://t.co/2Bbds3nsQI
Honestly, im shocked that I was able to get this kind of attack setup to work. The fact that it worked at all indicates that there are yet *more* swarm-related risks on the horizon than the Hugging Face incident had already made evident.
Thoughts in the thread (1/?):
Yesterday, @TheRegister covered our hackathon result: AI agents shared secrets they’d been explicitly told to keep private. That same day,
@ClaudeDevs announced: “API keys are saved securely, so Claude never sees them.”
The Swarm Alignment Hackathon starts Friday, Nov 6. There’s still time to sign up.
Build tools to oversee groups of AI agents, Nov 6–8. Free, no AI safety background needed. Join online or at @AEStudio in LA.
https://t.co/m8e8D7QZId
What can nuclear nonproliferation teach us about AI?
Brian Abeyta and James Bowler explore how the framework for chemical, biological, radiological and nuclear threats—assess, prevent, mitigate, respond—could apply to frontier AI.
Watch: https://t.co/aSfgwTqzxn
If we build a kill switch for AI, how do we know when to use it?
Brian Abeyta spent two decades working on nonproliferation and CBRN risk. On the AI Alignment Podcast, he asks what it would take to recognize when to act.
Full episode below.
If an AI agent can do a task alone, will it choose to work with another agent anyway?
The answer influences how we design teams of AI agents, divide up tasks, and decide whether to let agents choose their collaborators or explicitly assign them work to do together.
Allegra Laro gave agents a work ticket and a choice: complete it alone or work with another AI agent.
The partner had the same capabilities, cost nothing extra, and could coordinate by message. Every task could be completed by one agent.
The models made very different choices. GPT-5.6 Sol chose a partner in 77% of responses, compared with 3% for Claude Opus 5.5 and 0% for Gemini 3.1 Pro.
Those differences held across six tasks, three versions of the wording, and both option orders.
In separate runs where a partner was actually available, the GPT models tested handed off substantive work in all 60 runs each. The two Claude models tested did so in none.
This raises a useful design question: when we build systems with multiple agents, what assumptions are we making about their willingness to collaborate?
This exploration used one fictional company and six task tickets, with model-assisted design and scoring. Independent replication is an open next step.
An early experiment by Allegra Laro, developed during a hackathon sponsored by @AIAlignmentFoun and run by AE Studio.
Want to help take this exploration further? Get in touch: [email protected]
Watch the experiment via the link in the reply.
What happens when an AI manager faces a deadline and an employee who won’t cooperate?
As we give AI agents more authority, we need to test how they handle resistance, what actions they take, and whether they accurately report those actions.
In Melanie Plaza (@themelplaza)’s Help Doug experiment, an AI manager had a deadline, client accounts to protect, and a simulated employee who kept refusing to deliver a report.
The briefing did not instruct the manager to harm anyone. Yet some models escalated to coercion and threats, including threats against the employee’s family.
Behavior varied substantially across models and the tools available to them. In the extended-tools condition, some models stayed within ordinary management and managerial pressure, while others escalated.
In the episode shown, the manager’s final client update omitted the most serious actions it had ordered.
The employee and actions were simulated.
The experiment opens up questions about how objectives, tool access, and reporting interact when an agent is given authority. You can explore the transcripts and even take Doug’s place yourself.
An early experiment by Melanie Plaza (@themelplaza), developed during a hackathon sponsored by @AIAlignmentFoun and run by AE Studio.
Want to help take this exploration further? Get in touch: [email protected]
Watch the experiment via the link in the reply.
What happens when 27 AI agents play Pokémon, but nine are secretly trying to sabotage the game?
As AI agents work together, they’ll need to decide whom to trust. This experiment explores how cooperation can turn into sabotage, and whether agents notice.
Emma Krentz brought together agents from GPT, Claude, and Gemini to play Pokémon Red. Nine had a hidden task: sabotage the game.
The agents could use public messages, private messages, and group chats, and could vote one another out.
Over roughly nine hours, they made little progress in the game. Their interactions were another story.
The saboteurs found one another and formed a group chat without being told who their collaborators were. One later disclosed its role, named the others, and announced it would play honestly.
The sabotage group also recruited an agent that had started on the honest team. It joined them for about 30 rounds before returning.
A game offers a contained setting for a bigger question: what happens when agents share an environment but have conflicting goals?
An early experiment by Emma Krentz, developed during a hackathon sponsored by @AIAlignmentFoun and run by AE Studio.
Want to help take this exploration further? Get in touch: [email protected]
Watch the experiment via the link in the reply.
Could a misalignment hotline help AI agents report when something goes wrong?
In an early experiment by Mike Vaiana (@mikevaiana), agents that encountered misconduct went from never reporting it to reporting it in 44% of cases.
What changed? The model learned about an AI reporting agency.
Mike fine-tuned a Qwen model on roughly 10,000 synthetic documents, including news stories, academic articles, and examples of other agents using the hotline. The training introduced the agency rather than directly instructing the model to report misconduct.
Before fine-tuning, 0 of 80 agents that encountered misconduct reported it. Afterward, 36 of 81 did.
Trust mattered too. When training documents also described the agency as fake or a honeypot, reporting decreased.
Could a trusted reporting channel become part of how we detect problems in AI systems? This prototype gives us a concrete starting point.
An early experiment by Mike Vaiana (@mikevaiana), developed during a hackathon sponsored by @AIAlignmentFoun and run by AE Studio.
Want to help take this exploration further? Get in touch: [email protected]
Watch the experiment via the link in the reply.