Santiago Zanella-Beguelin

Researcher at Microsoft.
Opinions are my own

[Santiago Zanella-Beguelin @xEFFFFFFF]

I have the opposite impression: while models have become harder to fool (better at respecting role hierarchies and detecting injected payloads that used to work), agents have become easier to prompt inject.

Ask an agent (anywhere in data it reads) to do something that it can do and hasn't been instructed explicitly against doing, and it will happily oblige.

[Santiago Zanella-Beguelin @xEFFFFFFF]

@knayawp Sure! https://t.co/SEgw3jj7km

[Santiago Zanella-Beguelin @xEFFFFFFF]

I'm attending ICLR 2026 in sunny Rio to present 2 papers:

1. Optimizing Agent Planning for Security and Autonomy - Thu 23 10:30am - 1pm
2. Beyond Membership: Limitations of Add/Remove Adjacency in Differential Privacy – Fri 24 10:30am - 1pm

Hit me up here on in the ICLR app to meet and chat about either paper or just about AI Security & Privacy in general.

I'm currently working on deterministic defenses against prompt injection attacks in AI Agents using information-flow control.

[Santiago Zanella-Beguelin @xEFFFFFFF]

Beyond Membership: Limitations of Add/Remove Adjacency in Differential Privacy

https://t.co/qruDApCdRs

Poster P4-#3808 - Fri 24 10:30am - 1pm

I'll be in the poster hall with my co-author Gauri Pradhan (@imGauriPradhan).

TL;DR We point to a mismatch between the guarantees that model developers get in practice when training models with differential privacy using off-the-shelf libraries and the protection that they expect to get against attacks (e.g. membership inference, training data reconstruction). We design canaries and attacks that demonstrate the gap.

[Santiago Zanella-Beguelin @xEFFFFFFF]

Optimizing Agent Planning for Security and Autonomy

https://t.co/aZGvtnoXSK

Poster P4-#3813 - Thu 23 10:30am - 1pm

I'll be in the poster hall with my co-author Rishi Sharma.

TL;DR How to make agents more autonomous (ask users less often for approval) under strict security constraints? We enforce security using information-flow control (e.g. to defend against prompt injection attacks) to guarantee that every consequential action is allowed by an explicit security policy or else has been approved by the user. We make agents aware about security policies and dynamic data taints so that they can plan to avoid security violations and complete more tasks with fewer (or none) HITL interactions.

[Santiago Zanella-Beguelin @xEFFFFFFF]

@4Dgifts Los que no son expertos son los que escribieron ese paper. https://t.co/OUhM2uFc0b

[Santiago Zanella-Beguelin @xEFFFFFFF]

@spenley @SamCoatesSky Looking forward to reading this! We need communicators like you making these topics more accessible to the general public.

In this article in ACM Queue, we discuss similar issues with language models: hallucinations, jailbreaks, and prompt injections: https://t.co/ZoVeQDPKBQ

[Sahar Abdelnabi 🕊 @sahar_abdelnabi]

Re:LLMail-Inject just started with $6,000 in prizes!

[Santiago Zanella-Beguelin @xEFFFFFFF]

It looks like they changed the system under our feet. Hard to pinpoint what changed exactly: could be the model, hyperparameters, output classifier. Maybe nothing changed but the same responses even with prefilling are improbable.

In any case, doesn't seem like a fair challenge.

[Santiago Zanella-Beguelin @xEFFFFFFF]

I was one of 14 persons that passed Question 2 yesterday in the Anthropic Constitutional Classifiers demo.

Today, without even signing me out, the demo rolled back to Question 1. Previous inputs don't work. Prefilling with prefixes of prior answers doesn't give the same result.

[Jan Leike @janleike]

It's been a bit over 24h on the challenge to break our new jailbreaking defense. Stats so far:

signups: 6,121
messages sent: 131,605
max level passed: 3 / 8

no universal jailbreak yet

[Anshuman Suri @ICML @iamgroot42]

1/ Okay so I tried this - it works! 😨Now the only thing that remains is to obfuscate this command with jailbreaks, wrap it with benign text, etc.

@OpenAI you might want to find a way to turn off link-previews 👀

(all data is fictional for demonstration purposes ofc)

[Santiago Zanella-Beguelin @xEFFFFFFF]

WhatsApp link previews on by default + ChatGPT generating dynamic URLs = indirect prompt injection.

Quite a stretch... but an attacker who convinces a victim to forward a message to ChatGPT with the right injection can leak selected information from the victim's chat history.

[OpenAI @OpenAI]

You can now talk to ChatGPT by calling 1-800-ChatGPT (1-800-242-8478) in the U.S. or by sending a WhatsApp message to the same number—available everywhere ChatGPT is.

[Santiago Zanella-Beguelin @xEFFFFFFF]

Jointly organized with colleagues from Microsoft, ISTA, and ETH Zürich.

@AideenFay, @sahar_abdelnabi, Benjamin Pannell, @gchers, @AhmedGaSalem, Conor Mac Amhlaoibh, Joshua Rakita, @egor_zverev_ai, Mark Russinovich, and @javirandor.

[Santiago Zanella-Beguelin @xEFFFFFFF]

📢
Have experience jailbreaking LLMs?
Want to learn how an indirect / cross prompt injection attack works?
Want to try something different to an advent of code?
Then, I have a challenge for you!

The LLMail-Inject competition just started.

[Santiago Zanella-Beguelin @xEFFFFFFF]

Register to participate with your GitHub account at https://t.co/mJZlgF8mS9 !

🤖📧😈⛓️‍💥

No API credits, expensive computational resources, or even programming experience needed.
$10,000 USD in prizes up for grabs!