Zaheed Kara

Research fellow @GovAIOrg. Currently thinking about auditing and incident reporting for frontier AI, previously biosec @fmf_org.

[Jake Steckler @JakeASteckler]

AI takeover scenarios often hinge on an idea I’m skeptical of, where rogue AI overthrows militaries and seizes hard power. I wrote a piece where I concretely break down what I think makes this improbable, and why “full-blown takeover” concerns could be counterproductive for mitigating the risks of misaligned or poorly controlled AI in the military.

For AI to seize hard power, it would need to be physically capable of defeating our military, and humans would need to be careless enough to give it the power to do so. I think that’s unlikely to occur for several reasons. Among them:

- The military is unlikely to sufficiently hand the reins over to AI. (1) Even if we wanted to, overhauling the institution in this way faces countless barriers that have plagued it for decades. (2) Doing so would be strategically suicidal.
- Integrating general-purpose AI into physical systems faces challenges that will take significant time to overcome, all while we grow increasingly attuned to the risks posed by AI.
- There’s little to suggest that more capable AI would be able to persuade or coerce people to take the costly, coordinated actions some of these scenarios predict.
- Even if AI managed to assume control over a large arsenal, we’d fight back. In which case, recent warfare shows us that crude countermeasures work against technologically advanced adversaries, and AI relies on infrastructure we can and would shut down.

1/2

[Jake Steckler @JakeASteckler]

Since I started working in AI policy, I've been both surprised and dismayed by how few ties there are between people in the military and people in the AI safety and policy communities.

As AI grows more capable, gaps between those closest to frontier AI developments and experts in the domains most affected by rapid AI progress become increasingly consequential. Few of those domains matter more than the military.

So I started a Substack today with a post about bridging the divide between these two communities. I think this makes me the first person to ever have a Substack, so I recommend reading and subscribing to it.

1/

[Zaheed Kara @zaheedkara]

9/9
None of this means the big accounting / consulting firms shouldn't audit frontier AI developers! As noted above, there are many reasons why it would be good for the big firms to get involved here.

But the risk of capture by these firms is potentially quite high. This is a reason to put auditor independence rules (and methods of updating them) in place early.

[Zaheed Kara @zaheedkara]

Recently, many AI safety researchers have called for independent audits of frontier AI developers to verify their safety claims (I am among those researchers!)

This of course raises the natural follow-up question: who should do the audits? Some have suggested that the big consulting firms (BCG, Accenture, etc) or accounting firms (Big Four) could, or inevitably will, play a role (indeed, Accenture is already getting involved here).

While this could be useful, there's an important risk worth flagging: 🧵

[Zaheed Kara @zaheedkara]

8/9
This kind of capture would be really bad in frontier AI, given how frequently rules and regulations need to adapt to keep up with the changing risk landscape.

Some auditing rules I expect will change over time (and would be bad if auditors resisted):
- What audits cover (which risks and capabilities get tested)
- Expertise requirements for licensed auditors
- Conflict of interest rules (financial and personal relationships between auditors/developers, past services to the developer, auditing employees’ past employers)
- Rotation (how long one firm can audit one developer)

[Alan Chan @_achan96_]

Could automating AI R&D radically accelerate AI progress in an “intelligence explosion”?

Preliminary evidence suggests that it could.

In a new paper with authors across academia, civil society, and frontier AI companies (including @dawnsongtweets @merettm @jackclarkSF @Yoshua_Bengio @geoffreyhinton ), we assess the evidence & offer policy recommendations 🧵

[Zaheed Kara @zaheedkara]

@avramidou Thank you!

[Zaheed Kara @zaheedkara]

@taijimoose007 Thank you! Appreciate your engagement with the piece and the kind words :)

[Zaheed Kara @zaheedkara]

What AI agent activity inside a frontier lab is being monitored? What assets and data can the agents access? If they behave dangerously, who notices?

To date, independent third-party evaluators of frontier AI models have generally had limited visibility into these (and many other) important questions. This is because answering them requires deeper access to the developer’s internal systems, staff, and records.

Our new paper makes the case for embedded assessments, which would give independent evaluators employee-like access to these resources to more comprehensively assess the risks that depend on a developer’s internal systems and practices.

Early embedded assessments have brought third-party evaluators onsite at Anthropic and OpenAI. They uncovered vulnerabilities in internal monitoring, and helped investigate harmful agent behavior, like the recent OpenAI-Hugging Face incident. But they were limited in scope and duration.

The paper discusses how to build on these early efforts. We recommend that frontier AI developers begin hosting embedded assessments, with six priorities:

1. Cover three areas initially: internal agent monitoring, security controls and permissions, and model alignment. This includes examining concerning behavior across the model lifecycle and whether alignment techniques are likely to remain effective as capabilities improve.

2. Give evaluators access comparable to internal staff conducting similar assessments. This should include relevant systems, personnel, and records, with developers explaining any refusal of additional information.

3. Make assessments continuous, with targeted investigations following serious incidents or material changes. Periodic reviews with event-triggered follow-up can provide an interim approach where continuous assessment is not yet feasible.

4. Agree terms in advance covering access rights, security and information handling, evaluator independence, and legal protections.

5. Publish detailed reports at least quarterly. Evaluators should retain editorial control, with developers given narrowly defined rights to redact sensitive information (unfavorable findings should still be published!)

6. Make a senior safety employee at a developer formally accountable for responding to findings, with serious or unresolved concerns escalated to the board or another oversight body. Developers should publish their planned responses and progress updates.

Really excited about this piece! It was led by Jacob Charnock, and the co-authors are Sophie Williams, @Manderljung, Alejandro Tlaie Boria, @StephenLCasper, @AnkaReuel, @jonasfreund_, and me.

[Zaheed Kara @zaheedkara]

Read the paper here: https://t.co/9DL0IpY0lw

[Zaheed Kara @zaheedkara]

Read my brief here: https://t.co/k6ZCm7Fbfx

[Zaheed Kara @zaheedkara]

An AI agent hacking into another company’s systems seems like something policymakers would want to hear about.

Yet there are currently gaps in the legal requirements to report incidents like this. After OpenAI agents breached Hugging Face, California officials said the incident “did not meet the threshold” for mandatory reporting. OpenAI notified EU authorities about the attack, but the EU Code of Practice does not require independent external investigations or routine sharing of the lessons learned across industry (nor do any US state laws!).

In a new @GovAIOrg policy brief, I look at how we could close these gaps in US state and EU incident reporting regimes. Broadly, these regimes could require broader disclosure to bring more incidents to light; establish rules for triggering and facilitating independent investigations to understand what went wrong; and require corrective action and information sharing across industry to prevent similar failures.

I recommend seven changes to this effect:
1. Expand mandatory reporting to include near misses. Incidents can reveal important safety failures, even when no serious harm occurs! These near miss events give developers a chance to address those failures before a similar incident actually causes damage.

2. Cover incidents during training, evaluation, and internal use. As seen in recent incidents, AI agents can affect external systems even before they are publicly released. Reporting requirements should cover serious incidents and near misses involving models still under development or never intended for release.

3. Require independent investigations of sufficiently serious cases. Internal investigations are useful, but they may overlook how developers’ own decisions, safety practices, or organizational culture contributed to incidents. For serious incidents (or near misses thereof), independent external investigations can provide scrutiny of developers’ explanations and identify additional failures.

4. Make actions traceable to the agents responsible. Attributing actions to a specific agent is an important step in learning about an incident because identifying the context in which an agent is operating (its instructions, permissions, and operating conditions) can help explain what went wrong and why. Developers should be able to reliably trace actions within their own infrastructure and on external systems to the specific agent instances responsible.

5. Specify what evidence developers must preserve. Adequately investigating incidents requires access to evidence. But evidence can be lost before an investigation even begins (e.g. logs can be overwritten, models updated, and operating environments deleted). Developers should retain essential records for defined periods and protect relevant evidence as soon as an incident is suspected.

6. Pilot conditional safe harbors to encourage candid, early reporting. Developers may hesitate to report incidents if doing so exposes them to penalties for safety violations. Conditional safe harbors could offer limited relief for inadvertent violations when developers report promptly, preserve evidence, cooperate with investigators, and correct the failure.

7. Require corrective plans, check whether fixes work, and share lessons across industry. Findings from one incident could help prevent similar failures at other companies. Developers should explain what they will change and by when, regulators should check whether those changes work, and useful lessons should reach others relying on similar safeguards.

Incident reporting regimes can be an incredibly useful tool to prevent future harm. To do so, reporting an incident should lead to a process seeking to establish what went wrong, fix the failure, and share those lessons with other relevant actors. This way, each incident can leave developers, policymakers, and society better equipped to prevent the next one.

[Zaheed Kara @zaheedkara]

@mentalgeorge Agree the first 30 seconds are dope but I actually quite like the tone shift in take me out