Jam Kraprayoon

Senior Researcher @iapsAI Oxford/LSE. AI agents and national security. Fmr international civil servant. Also poet.

[Jam Kraprayoon @JKraprayoon]

Using the buffer well means three things
(1) Triage: harden the most exposed defenders first
(2) Translation: turn raw model capability into tools a hospital or utility can actually run
(3) Distribution: get those tools the last mile, to operators who can't integrate them alone

[Jam Kraprayoon @JKraprayoon]

@shaunkeee and I have a new piece in @lawfare arguing that managing access to powerful AI away from attackers isn't enough to ensure cyber resilience. We have to actively push it into defenders' hands. 🧵
https://t.co/c49jyaQw96

[Jam Kraprayoon @JKraprayoon]

Adaptation buffers buy time, but society needs to make sure this time is used well. Open-weight models trail the frontier by ~4-8 months. Adversaries steal capabilities via distillation. The threat advances no matter who you lock out.

[Lawfare @lawfare]

"Just as Operation Warp Speed defined the U.S. pandemic response, cyber defense needs a bold push to make the best of this critical window: triaging defenders, translating models into deployable tools, and distributing those tools at scale," writes Shaun Ee and @JKraprayoon.

[Theo Bearman @theobearman]

My @iapsAI colleagues and I have come together for a Researchers React piece on today's EO on AI Innovation and Security. A few reflections from me:

- The focus on advanced cyber capabilities is understandable, but future frontier AI models may also exhibit advanced capabilities in other dual-use national security-relevant domains. Determinations of what models qualify as “covered frontier models” should take into account capabilities in relevant domains beyond cyber–including chemical, biological, radiological, or nuclear domains and autonomous R&D acceleration–to avoid a situation where the government lacks visibility into other sources of frontier AI risk before they materialize.

- The benchmarking process directed by Section 3(a) of the EO is entirely government-internal, with no apparent role for independent third-party evaluators. Government evaluators alone may not be able to keep pace with the volume and velocity of frontier model development. To address this, the administration should consider establishing model access standards that allow qualified third parties to conduct rigorous evaluations. Doing so would distribute the testing burden, improve assessment quality, and create an institutional infrastructure that outlasts any single EO.

- Agencies evaluating frontier models’ capabilities, propensities, and safeguards must be able to make full use of the “up to 30-day” pre-release access window directed by Section 3(b)(ii) of the EO, alongside the government leveraging the model’s capabilities for defensive purposes during this period. We have already seen examples of third-party evaluators who work with frontier AI companies facing difficulties making conclusive assessments due to the limited time they have been given to do their work. Longer pre-release access windows can support a more rigorous and comprehensive understanding of the risks frontier models may pose, reporting of findings as appropriate, and the design and implementation of mitigations as needed. Whatever the agreed length of the access window, the government should ensure that evaluating bodies are appropriately resourced and that the final voluntary framework includes clear expectations for how frontier AI companies will act on evaluation findings before releasing the model onward.

... and one from my colleague @__J0E___ which I think is very important on the back of our recent work on risk reporting for developers’ internal AI model use (https://t.co/GCyv0E3yzh):

- As part of the process of identifying covered frontier models, the NSA should explicitly consider internal models which are never intended for public release. Frontier AI companies first deploy their most capable models internally, often long before public release–and in some cases, models are not released publicly at all. Moving forward, companies may choose to keep highly-capable models for internal use only, because (1) public deployment may present outsized misuse risks, and (2) they may want to deploy these models internally to accelerate their own AI R&D. The administration should build on the EO by ensuring that powerful internal models fall within the benchmarking process regardless of eventual deployment plans.

Glad to be able to share these reflections alongside my colleagues @rosen_br, @__J0E___, @covinstantinop @MattInThemittel, and @JKraprayoon!

[Matt Mittelsteadt @MattInThemittel]

A key provision in the new AI EO is the vulnerability-discovery "clearinghouse." A good start — but the admin must build on it to meet the defensive-coordination challenges that offensive cyber agents will create. In a just-published @IAPS and @aisafetysg report, we detail how 🧵https://t.co/l3TZwUwAOY

[Dave Banerjee @DaveRBanerjee]

New post from @iapsAI on Cyber Superstorms

My colleagues argue that counting zero-days is not the way to measure the consequences of AI-accelerated vulnerablility

Instead, they propose that the community should focus on how often AI-accelerated discovery produces crises that overwhelm the defensive ecosystem

Read the full post here: https://t.co/0gFIzbKqqy

[Institute for AI Policy and Strategy (IAPS) @iapsAI]

The IAPS Frontier Security team is launching a newsletter – “The Attack Surface” – covering national security challenges created by the most powerful AI systems and what policymakers, developers, and defenders can do about them.

[Matt Mittelsteadt @MattInThemittel]

🚨Just published a big new report! Mythos showed AIxCyber risks are real. Offensive cyber agents are the next challenge and detection must be a priority. In our work we frame the challenge, present a strategic path forward, and introduces 5 mechanisms to implement today.

[Institute for AI Policy and Strategy (IAPS) @iapsAI]

The UK @AISecurityInst has proven AI agents can orchestrate cyberattacks. To defend against these emerging threats, we must detect them. New research from @iapsAI and @aisafetysg shows why detecting offensive cyber agents will be difficult — and what we should do about it.

[Foreign Affairs @ForeignAffairs]

“The United States and its allies have years, not decades, before autonomous cyber-capabilities proliferate,” write @rosen_br and @JKraprayoon. What should Washington do now to mitigate potential security risks?
https://t.co/ilAsiD2M3J

[AI Policy Bulletin @AIPB_org]

While initiatives like Project Glasswing provide tech companies with access to cyber-capable models, "access is only the starting point".

A timely policy memo on AI cyber defence, by @covinstantinop, @JKraprayoon, & @MattInThemittel @iapsAI

[Institute for AI Policy and Strategy (IAPS) @iapsAI]

Mythos is a leap in AI cyber capabilities, but the bigger story is the accelerating trajectory and proliferation.

@iapsAI researcher Christopher Covino explores what the preview means for national security and what policymakers should do about it: https://t.co/J5JiyHhN9C

[Foreign Affairs @ForeignAffairs]

“Autonomous cyber-agents are already operational, and policymakers are unprepared.”

Read @rosen_br and @JKraprayoon on the threat that AI agents pose to global security—and how governments should respond:
https://t.co/ilAsiD2M3J

[Foreign Affairs @ForeignAffairs]

.@rosen_br and @JKraprayoon examine the security risks posed by autonomous cyber-agents—and urge Washington to shore up its defenses “before AI goes rogue.”
https://t.co/mQKSfabIZD

[Foreign Affairs @ForeignAffairs]

“The policy choices made today will determine whether autonomous cyber-agents become a manageable risk or an uncontrollable one,” write @rosen_br and @JKraprayoon.
https://t.co/D639zm1Hhc

[Institute for AI Policy and Strategy (IAPS) @iapsAI]

Autonomous cyber-agents can already execute in minutes what would take hours of expert human labor — with no off switch and no capacity to judge when to stop.

Today in @ForeignAffairs, IAPS's @rosen_br and @JKraprayoon discuss what this means for national security: https://t.co/ROocgivHwZ

[Dave Banerjee @DaveRBanerjee]

New post on Substack: Securing AI Infrastructure to Prevent Backdoors and Sabotage

In an intense AI race, we may expect adversaries (nation-states, insiders, maybe even misaligned AIs) to target our frontier AI models through data poisoning and other integrity attacks

One of the best ways to stop these kinds of attacks is securing our AI infra

My new post explores open problems in securing AI infrastructure and outlines a concrete research agenda

https://t.co/RY8inFtE9q