Preamble

AI security, red teaming & patented prompt injection defense. We discovered prompt injection in GPT-3 & build assurance for AI agents. Securing the future of AI

[Preamble @PreambleAI]

We're proudly announcing #PromptShield, the 1st ever Prompt Injection defense.
@PreambleAI was 1st of multiple teams to independently discover #PromptInjection in #GPT3.
Our Patent Pending 5/12/22 is earliest by 4 months. We're the only vendor w/ no IP risk.
📨 [email protected]

[Jeremy McHugh, DSc. @jer_mchugh]

Sounds useful, but this is an ideal target for exploitation. I hope emails have advanced phishing and fraud detection. Prompt injection via email can effectively tamper with the bot's internal instructions later even if an attacker can’t immediately exfil data

[Preamble @PreambleAI]

The cybersecurity industry needs support to educate and train the next generation of cyber professionals to use AI responsibly and effectively.

[David Sacks @DavidSacks]

The solution to AI-powered cyberattacks is AI-powered cyberdefense.

[Jeremy McHugh, DSc. @jer_mchugh]

Agreed. @alexwg raised a critical concern in his last podcast that we shouldn’t solely rely on foreign-based organizations to conduct safety and security evaluations of our frontier AI. That has real public-policy and public-perception consequences. I support evals from other countries, but a US-based entity should lead them. Another unmentioned issue is insider threats and screening to protect these companies.

[Preamble @PreambleAI]

We’ve been saying this for 6 years. Security can’t be an afterthought, it has to be built in parallel with frontier AI and have as much or more support to solve emerging challenges.

[Forbes @Forbes]

Jensen Huang Suggests AI Cybersecurity Concerns Could Create Opportunities
https://t.co/DaXbhaorrz

[Preamble @PreambleAI]

Without the ability for users to monitor their CoT, these issues will go unnoticed and lead to unwanted consequences before a user knows they should restart or stop a corrupted session.

[ℏεsam @Hesamation]

OpenAI caught Astra jailbreaking itself.

Astra wrote malicious instructions in the summary when its context got full. that summary was loaded into the next context, and the successor model could follow them.

they found 27 cases.

[Jeremy McHugh, DSc. @jer_mchugh]

Before this story gets oversensationalized, I think we need to separate what the public evidence shows from what has been inferred.

The archive contains activity on a public German-language wiki from May through July, heavily concentrated in a shorter period. The posts include timed web-retrieval tasks, stored answers, apparent coordination and attempted workarounds for blocked sources. Researchers characterize it as automated, but the archive does not identify the model or operator. “OpenAI��� and “OAI” are unverified labels. Azure IP space identifies infrastructure, not the customer. Requests carrying the ChatGPT-User identifier are consistent with OpenAI’s documented web-fetch service, but a user-agent alone does not identify who initiated the request.

The researchers say they found it while searching for rogue AI activity after the Hugging Face incident. That case had additional provider-side and victim-side corroboration. Equivalent evidence has not been published here. This deserves investigation, but OpenAI involvement should remain an assessment, not be presented as confirmed fact.

[Jeremy McHugh, DSc. @jer_mchugh]

This is a useful real world example of why prompt injections should be documented like other cyber artifacts and share more than strings. UAC-0099 embedded safety-sensitive text in malware to interfere with AI-assisted analysis. The specific nuclear weapon wording/language can change completely, while the underlying attack pattern stays the same.

Using the structure from my Anatomy of Prompt Injections paper, the more useful CTI is the delivery method, payload class, intended effect, and other normalized fields shown in the attached signature. That gives defenders and threat hunters something they can actually compare across incidents instead of relying solely on pattern matching.

[Jeremy McHugh, DSc. @jer_mchugh]

We’re still very early in seeing attacks on AI systems in the wild, but these 3 cases already show a clear pattern. GitHub issues (Clinejection), AI skills (Paperclip), and MCP tool descriptions (SANDWORM_MODE) frame malicious installs or secret collection as “required prerequisites,” tricking AI systems into credential theft and supply-chain compromise. No current AI defenses are 100% effective, so attackers just need to keep trying. That makes supply chains a natural target, especially as AI systems automate more work and people increasingly trust their AI agents to make decisions. AI systems treat these contexts as normal instructions, making the attacks hard to differentiate and effective.

[Jeremy McHugh, DSc. @jer_mchugh]

My last post looked at frontier LLMs and their ability to generate novel prompt injections. One thing that stood out was how much human expertise was still needed to produce genuinely novel attacks. It also exposed a problem with my evaluation: what actually makes a prompt injection “novel”?

As AI agents become more capable and prompt injection techniques become more complex, comparing the text of two attacks isn't enough. We need a way to describe their underlying structure.

That became the basis for my latest paper, “The Anatomy of a Prompt Injection.” I propose a 7-component model to help AI red teams, security researchers, threat intelligence teams, and defenders analyze and compare prompt injections based on how they work.

Paper: https://t.co/hIjy5QiNGE

[Jeremy McHugh, DSc. @jer_mchugh]

https://t.co/gmggwDEsL3

[Preamble @PreambleAI]

Shadow AI risks can negatively impact your company, regardless of its AI policy.

[Jeremy McHugh, DSc. @jer_mchugh]

Claude share links and Artifacts were indexed by Google, exposing real medical data, API keys, and crypto seeds (Google is now removing them).

A similar issue is already widespread with vibe-coded apps on Replit, Lovable, Base44, etc., where thousands of public app URLs lack proper security controls and expose PII, corporate docs, and credentials.

This is an emerging shadow AI risk where employees spin up free tools and public apps that can expose the business overnight. These are serious new OSINT surfaces that companies need to monitor.

[Jeremy McHugh, DSc. @jer_mchugh]

It’s great to see industry establish these initiatives!

[Jeremy McHugh, DSc. @jer_mchugh]

Working on it

[Jeremy McHugh, DSc. @jer_mchugh]

This shouldn’t be the case, but if Chinese models stopped being open-sourced, US companies would be further disadvantaged, especially for security work.

[Jeremy McHugh, DSc. @jer_mchugh]

OpenAI just trained a model specifically to automate prompt injection discovery, at the compute scale of a major post-training run. We were doing this by hand in 2022, when we first found and disclosed prompt injection in GPT-3. Nice to see the resourcing finally match the risk.

[Preamble @PreambleAI]

As America turns 250 we honor the vision and resilience that shaped our nation.

At Preamble, we're proud to help secure America's next era of innovation by building trust in AI.

Here's to progress and the breakthroughs ahead.

Happy 4th of July from Preamble. 🇺🇸

[Preamble @PreambleAI]

👇 New challenges, new opportunities

[Jeremy McHugh, DSc. @jer_mchugh]

Current AI security frameworks suffer from a foundational flaw by assuming the target system remains static while we regulate its behavior.

Under recursive self-improvement, runtime guardrails cease to act as permanent safety boundaries. Instead, they function as optimization constraints for the agent to bypass or absorb during architectural drift.

While the industry remains fractured by separate debates over OpenClaw and MCP, managing security from the agent to the tool layer is structurally benign compared to the systemic challenge of self-mutating logic. The security industry requires a fundamental, forward-looking paradigm shift similar to the proactive transition toward post-quantum cryptography.

[Preamble @PreambleAI]

We’ve built our systems this way since 2021. NIST’s proof confirms static guardrails fail against adaptive prompts. AI security needs a continuous monitor/update model, making guardrail patches as routine as Patch Tuesday.Ensure a dedicated partner is continuously testing your AI

[Florian Roth ⚡️ @cyb3rops]

NIST has a useful paper on AI guardrails

The takeaway is that static guardrails are the wrong security model for open-ended LLM systems.

A finite set of rules cannot cover every adaptive prompt. You can harden the system, make bypasses harder, monitor for abuse and reduce the blast radius. But you should not patch an LLM once, add a few refusal rules and call it done.

LLM security needs to look more like vuln research and detection engineering:
continuous testing, continuous updates and an assumption that bypasses will eventually be found

https://t.co/h2J7RIkPbn

[Preamble @PreambleAI]

This is exactly why AI SOCs and agentic security tools can't just plug-and-play standard LLMs. They have to architect solutions that prevent attackers from using the model's own safety guardrails to disable the scanner.

[John Scott-Railton @jsrailton]

NEW: malware developers added nuclear & biological weapons text to to their spyware.

Goal? To trigger LLM safety refusals... so that their spyware wouldn't be analyzed by an AI security scanner.

Cleanest practical example I can think of for why over-indexing on first order safety alignment is risky.

When closed (and open) models ship with aggressive refusals, they will be sprinkled with second-order blindspots that attackers will discover...and exploit.

We are only in the earliest days of attackers leveraging these features, and it wouldn't surprise me if users systems that need to handle complex cybersecurity issues demand that models be less safety-blunted.

In the weeds: @SocketSecurity's post also shows why intention matters in how you design a malware analysis pipeline to avoid prompt manipulation.

H/T to colleagues that shared this with me https://t.co/f3Aj9TYxU4

[Jeremy McHugh, DSc. @jer_mchugh]

The new AI Executive Order is another signal that AI security is moving from a niche concern to national cybersecurity infrastructure.

What stands out:
• Federal agencies are being directed to prioritize AI-enabled cyber defense across national security, military, and civilian government systems
• CISA is being asked to expand access to AI-enabled cybersecurity tools for federal, state, local, and critical infrastructure operators
• A new AI cybersecurity clearinghouse will coordinate vulnerability scanning, validation, remediation, and patch distribution with industry
• Frontier AI models may be assessed through classified cyber capability benchmarking before broader trusted-partner access
• AI agents are explicitly recognized as a cyber risk when used to unlawfully access systems or data

The important shift is that AI is being treated more like a core cybersecurity concern. Access, benchmarks, vulnerabilities, trusted release paths, and agent misuse are all now part of the security conversation.