Chawin Sitawarin

Research Scientist @GoogleDeepMind. Postdoc @Meta. PhD @UCBerkeley. ML security 👹 privacy 👀 robustness 🛡️

[Chawin Sitawarin @csitawarin]

🇧🇷 Going to ICLR!

Will present a poster on data poisoning attack at Agents in the Wild workshop.

Great work by @davidhuang33176 Jaewon Chang, @Avidan_Shah56, @prateekmittal_

📄 https://t.co/ZEi11IvDxt (full paper, website, code soon). But please come visit and chat!

[Chawin Sitawarin @csitawarin]

@kotekjedi_ml @javirandor @alxndrdavies @usmananwar391 @_zifan_wang @edoardo_debe @GraySwanAI Thanks a lot for sharing. Cool stuff!! Was gonna try something similar, but I guess now we didn’t have to :)

[Andrew Gordon Wilson @andrewgwils]

To be honest, I was initially confused and reserved about AI alignment. It's not that I was against the research direction, quite the opposite. For 15 years, I'd been developing the foundations of what had been rebranded as alignment. But, I've changed my mind. 1/6

[Nils Walter @nilspwalter]

It is notoriously hard to defend LLMs against prompt injections. Most defenses show good performance on static benchmarks but fall apart against stronger adaptive attackers.
In our latest work, we present an almost embarrassingly simple defense that delivers ~3× better robustness against the strongest adaptive prompt injection attacks to date - while keeping utility degradation acceptable.

Joint work with @csitawarin, Jamie Hayes, @davidstutz92, @iliaishacked.

[Federico Barbero @fedzbar]

Feel free to check out the paper here :)
https://t.co/N91jQN5Zoz

Special thanks to the amazing co-authors!

w/ @gu_xiangming @Chris_Choquette @csitawarin Matthew Jagielski @itay__yona @PetarV_93 @iliaishacked Jamie Hayes

[Chawin Sitawarin @csitawarin]

@_alyxya @SallyHZhu Sorry typo: "the explanation here does not seem convincing"

[Chawin Sitawarin @csitawarin]

@_alyxya @SallyHZhu I didn't read the paper but sort of have the same question as @_alyxya. This seems like an obvious test so I assume I miss something that's covered in the paper? But the explanation here does seem convincing to me... There might be FPs sure, but it seems like

[Chawin Sitawarin @csitawarin]

@_alyxya @SallyHZhu a lot stronger statistical test than checking against the training data (regardless of whether Bob fine-tunes the ?

[Florian Tramèr @florian_tramer]

5 years ago, I wrote a paper with @wielandbr @aleks_madry and Nicholas Carlini that showed that most published defenses in adversarial ML (for adversarial examples at the time) failed against properly designed attacks.

Has anything changed?

Nope...

[Konrad Rieck 🌈 @mlsec]

🚨 Got a great idea for an AI + Security competition?

@satml_conf is now accepting proposals for its Competition Track! Showcase your challenge and engage the community.

👉 https://t.co/3g3nvv3yqa
🗓️ Deadline: Aug 6

[Chawin Sitawarin @csitawarin]

Very cool thought-provoking piece! In practice, computation units are much more nuanced than what theories capture. But just trying to identify classes of problems that benefit from sequential computation (or is unsolvable without it) seems very useful!

[Konpat Ta Preechakul @konpatp]

Some problems can’t be rushed—they can only be done step by step, no matter how many people or processors you throw at them.

We’ve scaled AI by making everything bigger and more parallel: Our models are parallel. Our scaling is parallel. Our GPUs are parallel.

But what if the real bottleneck isn’t size—but depth?What if the model just didn’t have enough serial steps to get it right? Some problems need depth, not width.

This is the Serial Scaling Hypothesis.

This is not the same as recent studies in scaling test-time compute, which focus on train vs. test and are agnostic to parallel vs. serial.
For example: test-time majority voting increases compute by running models in parallel — but doesn’t help when the task itself is serial.

We argue: what really matters is how the compute is structured. And for many real-world problems, it must be serial.

Read more at: https://t.co/msytYszWK0 or 🧵.

(In collaboration with: @layer07_yuxi , Kananart Kuwaranancharoen and @YutongBAI1002 )

[Konpat Ta Preechakul @konpatp]

Some problems can’t be rushed—they can only be done step by step, no matter how many people or processors you throw at them.

We’ve scaled AI by making everything bigger and more parallel: Our models are parallel. Our scaling is parallel. Our GPUs are parallel.

But what if the real bottleneck isn’t size—but depth?What if the model just didn’t have enough serial steps to get it right? Some problems need depth, not width.

This is the Serial Scaling Hypothesis.

This is not the same as recent studies in scaling test-time compute, which focus on train vs. test and are agnostic to parallel vs. serial.
For example: test-time majority voting increases compute by running models in parallel — but doesn’t help when the task itself is serial.

We argue: what really matters is how the compute is structured. And for many real-world problems, it must be serial.

Read more at: https://t.co/msytYszWK0 or 🧵.

(In collaboration with: @layer07_yuxi , Kananart Kuwaranancharoen and @YutongBAI1002 )

[Chawin Sitawarin @csitawarin]

@edoardo_debe Awesome! Congrats 🎉
You gonna be in Menlo Park?

[Chawin Sitawarin @csitawarin]

I will be at ICML this year after a full long year of not attending any conference :)

Happy to chat, and please don’t hesitate to reach out here, email, on Whova, or in person 🥳

[Andreas Terzis @aterzis]

We are starting our journey on making Gemini robust to prompt injections and in this paper we present the steps we have taken so far.

A collective effort by the GDM Security & Privacy Research team spanning over > 1 year.

[Jack Morris @jxmnop]

new paper from our work at Meta!

**GPT-style language models memorize 3.6 bits per param**

we compute capacity by measuring total bits memorized, using some theory from Shannon (1953)

shockingly, the memorization-datasize curves look like this:
___________
/
/

(🧵)

[Tong Wu @TongWu_Pton]

🛠️ Still doing prompt engineering for R1 reasoning models?
🧩 Why not do some "engineering" in reasoning as well?
Introducing our new paper, Effectively Controlling Reasoning Models through Thinking Intervention.
🧵[1/n]

[Edoardo Debenedetti @edoardo_debe]

1/🔒Worried about giving your agent advanced capabilities due to prompt injection risks and rogue actions? Worry no more! Here's CaMeL: a robust defense against prompt injection attacks in LLM agents that provides formal security guarantees without modifying the underlying model!

[Max Nadeau @MaxNadeau_]

🧵 Announcing @open_phil's Technical AI Safety RFP!

We're seeking proposals across 21 research areas to help make AI systems more trustworthy, rule-following, and aligned, even as they become more capable.

[Sicheng Zhu @sichengzhuml]

Using GCG to jailbreak Llama 3 yields only a 14% attack success rate. Is GCG hitting a wall, or is Llama 3 just safer? We found that simply replacing the generic "Sure, here is***" target prefix with our tailored prefix boosts success rates to 80%. (1/8)