Tinghao Xie

~Last year ECE PhD candidate @Princeton | Interning @TikTok_US | Prev Intern @Meta GenAI | Worked on AI safety & security, now exploring VLM training

[Tinghao Xie @VitusXie]

🏖️First time attending ACL! Will be around San Diego 7/2-7/7.

Let's meet up for ☕️/🍱/🧗‍♀️!!! (Or just join my poster session on 7/6 11:00-12:30.)

[Tinghao Xie @VitusXie]

Spent 2d vibe-designing this ACL poster.

Pipeline: LM notebook (create initial infographic)-> ChatGPT (iterate on details) -> Gemini (super resolution) -> Photoshop (manual polishing).

✨This paper came out of my fun internship project at Meta. Hope the poster can speak for me!

[Yueqi Xie @XieYueqi]

🎊Excited to share that our research on validating AI-generated social science data is now online at @PNASNews !

Check it out: https://t.co/ljqoTBVBxB
Great thanks to my collaborators!

💡In this work, we highlight population-level statistical realism as a core criterion.

[Tinghao Xie @VitusXie]

@elonmusk Well... got ya? tho I'll say Grok moderation feels quite strong to me🦾

(not fully tested, just based on the experience of my last research project https://t.co/ESYIx24Zkb)

[Tinghao Xie @VitusXie]

Nano Banana Pro🍌, superior text rendering😈🧨!!

@GeminiApp #nanobanana

[Tinghao Xie @VitusXie]

I cannot make it to ICLR in person this year🥺 (as well as most other authors on this paper). But please feel free to stop by [Hall 3 + Hall 2B #541] for our poster of "🥺SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal" this afternoon (3-5:30pm)! #ICLR2025

[Xiangyu Qi @xiangyuqi_pton]

Thrilled to know that our paper, `Safety Alignment Should be Made More Than Just a Few Tokens Deep`, received the ICLR 2025 Outstanding Paper Award.

We sincerely thank the ICLR committee for awarding one of this year's Outstanding Paper Awards to AI Safety / Adversarial ML. Special thanks go to the reviewers and area chairs for their strong support and recommendations. Throughout the rebuttal period, the reviewers remained deeply engaged, raising thoughtful questions that helped enhance the rigor of our experiments and manuscript. I am also profoundly grateful to my collaborators (@PandaAshwinee @vfleaking @infoxiao @sroy_subhrajit @abeirami) for their joint efforts and my advisors (@prateekmittal_ @PeterHndrsn) for their invaluable guidance and support.

+ On a personal note, I also defended my PhD at Princeton in February and joined OpenAI last month, where I will continue working on AI safety and adversarial robustness. I'm looking forward to catching up with old friends and meeting new friends around the Bay!)

------

Below are some of my reflections and thoughts on our awarded paper:

Adversarial robustness has been an ongoing topic since the early rise of deep learning in 2013 (https://t.co/Squ5sX8GCz). Over the years, we've observed the community swing from pessimism—epitomized by Nicholas Carlini's adaptive attacks (https://t.co/EE9O5aLcRd) systematically dismantling various defenses, fostering the sentiment "adversarial examples are hard"—to skepticism, as adversarial examples appeared to have limited impact on practical AI applications for a while, prompting the notion "adversarial examples are not even important."

With the emergence of ChatGPT at the end of 2022, deep learning entered a new era towards AGI, shifting AI safety from theoretical speculation to mainstream practical concern. This is also when adversarial robustness again gets more attention. For example, following our 2023 demonstrations that adversarial examples pose fundamental threats to AI safety alignment (https://t.co/WZM7wfYqa3, https://t.co/g7IQA8D1nY, https://t.co/e5jOcIvFzU), adversarial examples reemerged as the "Sword of Damocles" hanging over AI safety (memorably illustrated by Zico Kolter at ICML 2023 in Hawaii, who humorously preempted his talk on the GCG attack with a Terminator slide captioned, "adversarial examples are back"). More concerningly, in the context of AI safety, disrupting safety alignment through fine-tuning is even simpler and harder to mitigate than adversarial examples (https://t.co/P9o5KGH9mM, https://t.co/2D0HYXcnCb, https://t.co/gH8wW6Nx9V, https://t.co/PK1nlICXzo, https://t.co/jrqD8XQfxB).

In 2023, conducting attack research was enjoyable—simply formulating and demonstrating the existence of vulnerabilities sufficed, as the effectiveness of an attack is inherently compelling. However, in 2024, my advisors started to heavily push me toward working on robustness defense, asserting that identifying problems without striving for solutions is not ambitious enough. While I wholeheartedly agreed, I was acutely aware of the profound challenge in achieving genuine robustness. After a lot of struggle, we eventually still developed this paper. Initially, our exploration focused on constrained supervised fine-tuning (SFT) against fine-tuning attacks. During this process, we discovered a critical bias—models exhibit substantial "first-few-tokens bias" concerning safety (here we acknowledge similar findings by https://t.co/irDKwTmsNb and https://t.co/ZceJPx37yg, despite differences in our ultimate directions). Using this bias as a technical trick, we impose strong constraints on the losses of only the initial tokens, relaxing constraints for later tokens. This achieved robustness with significantly lower utility regression. Nevertheless, we soon recognized that this bias is not merely a technical trick but represents a fundamental issue. Consequently, we shifted our focus to exploring the broader implications of this phenomenon itself, ultimately shaping the current paper. In writing this paper, I intentionally echoed the style of two seminal works: "Adversarial Examples Are Not Bugs, They Are Features" (https://t.co/lOmPqbo6B3) and "Shortcut Learning in Deep Neural Networks" (https://t.co/IdeWjpNygQ). The two papers deeply influenced my research style, and receiving the Outstanding Paper award at the culmination of my PhD journey, using a similar writing style, feels both fulfilling and like a tribute to these classics.

Frankly, our work still stands far from fully resolving adversarial robustness. In fact, during writing, we deliberately reduced/avoided using the term "defense," resulting in some critique that our paper reads more like a position paper. Rather, our contribution primarily provides just a simple yet concrete explanation (shallow alignment) for a broadly exploited class of vulnerabilities, enabling causal interventions on models to explore the counterfactual of shallow alignment—deep alignment—and demonstrating that such interventions genuinely improve robustness. Fundamentally, our intervention underscores that model alignment must span the entire generation process rather than being confined to the first few token distributions—a principle articulated explicitly in our paper's title. This concept resonates with several other studies, such as Andy Zou et al.’s Circuit Breakers (https://t.co/ul9h7tWVXC), Youliang Yuan et al.’s refusal at every position (https://t.co/DEOBQpOF3i), and Yiming Zhang et al.’s backtracking (fri). To some extent, improved robustness in reasoning models’ safety alignment (https://t.co/F4fNc7BLOu) might also be related to this principle, as large-scale reinforcement learning for reasoning spontaneously enhances self-correction and recovery.

Yet, adversarial robustness remains unresolved. Adaptive attacks will continuously emerge, potentially perpetuating many cycles of a cat-and-mouse game again. Furthermore, our challenges extend beyond AI safety and jailbreak issues. As frontier models rapidly advance in agentic capabilities, we eagerly anticipate their large-scale deployment to automate numerous tasks. However, currently, robustness and prompt injection significantly hinder this vision. As AI increasingly manages critical workloads and computational systems, robustness failures could pose severe systemic security risks.

Finally, we again extend our sincere appreciation to all friends in the AI safety and AdvML research communities for their ongoing support and encouragement. Let’s continue working together to advance the research on AI safety and adversarial machine learning.

[ICLR @iclr_conf]

Honorable Mentions

Data Shapley in One Training Run.
Jiachen T. Wang, et al.

SAM 2: Segment Anything in Images and Videos.
Nikhila Ravi, et al.

Faster Cascades via Speculative Decoding.
Harikrishna Narasimhan, et al.

[ICLR @iclr_conf]

Outstanding Papers

Safety Alignment Should be Made More Than Just a Few Tokens Deep. Xiangyu Qi, et al.

Learning Dynamics of LLM Finetuning.
Yi Ren and Danica J. Sutherland.

AlphaEdit: Null-Space Constrained Model Editing for Language Models.
Junfeng Fang, et al.

[Tong Wu @TongWu_Pton]

🛠️ Still doing prompt engineering for R1 reasoning models?
🧩 Why not do some "engineering" in reasoning as well?
Introducing our new paper, Effectively Controlling Reasoning Models through Thinking Intervention.
🧵[1/n]

[Javier Rando @javirandor]

Running out of good benchmarks? We introduce AutoAdvExBench, a real-world security research benchmark for AI agents.

Unlike existing benchmarks that often use simplified objectives, AutoAdvExBench directly evaluates AI agents on the messy, real-world research tasks.

[Pin-Yu Chen @pinyuchenTW]

As a signal processing researcher, I believe it's the best time ever to contribute our methods and knowledge to improve computational AI safety - problems that can be reformulated as detection and estimation tasks - Jailbreak, AI-generated content, etc

📰https://t.co/vVjmPTmwGj

[Zhenxing Mi @Mifucius1]

Excited to share our new paper "ThinkDiff" on arxiv.

I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning in Diffusion Models

It can make the diffusion models take "IQ tests"!

It empowers diffusion models with multimodal in-context understanding and reasoning capabilities of vision-language models without diffusion training and reasoning datasets.

It can generate images and videos based on the multimodal context.

ThinkDiff significantly improves accuracy from 19.2% to 46.3% on the challenging CoBSAT benchmark for multimodal in-context reasoning generation, with only 5 hours of training on 4 A100 GPUs. Additionally, it demonstrates exceptional performance in composing multiple images and texts into logically coherent images.

Many thanks to all the collaborators for their great help!

@kcjacksonwang @guocheng_qian @leoyerrrr @rleobest @SergeyTulyakov @AbermanKfir @danxuhk

Project page: https://t.co/QKjEHeiIBy
Code to be released at: https://t.co/NVMi3x02gV
Arxiv: https://t.co/M6JxdgcS3H
Huggingface paper page: https://t.co/u5VvAP1k44

[Kaixuan Huang @KaixuanHuang1]

Do LLMs have true generalizable mathematical reasoning capability or are they merely memorizing problem-solving skills? 🤨

We present MATH-Perturb, modified level-5 problems from MATH dataset to benchmark LLMs' generalizability to slightly perturbed problems.

🔗 https://t.co/HP2WPZ9k1G
🧵 [1/n]

[Javier Rando @javirandor]

Adversarial ML research is evolving, but not necessarily for the better. In our new paper, we argue that LLMs have made problems harder to solve, and even tougher to evaluate. Here’s why another decade of work might still leave us without meaningful progress. 👇

[Tinghao Xie @VitusXie]

@xiangyuqi_pton The case is even worse for image safety eval & attack..

[Tinghao Xie @VitusXie]

@prateekmittal_ @thegrugq @SwiftOnSecurity @kashhill Not exactly on topic, but the recent poisoning attack against Bytedance model training was an explosive example for ML security community🤔

https://t.co/fNuYMfdRG7

[Boyi Wei @wei_boyi]

Open-sourced models suffer from dual-use risks via fine-tuning. Recently, several new defenses have been proposed to counter these attacks. But how do we properly characterize the depth of the defense? Our paper in ICLR 25 shows that correct evaluation is hard—beware of common pitfalls!
https://t.co/Gaqew8u7kA [1/n] #ICLR2025

[Furong Huang @furongh]

I saw a slide circulating on social media last night while working on a deadline. I didn’t comment immediately because I wanted to understand the full context before speaking. After learning more, I feel compelled to address what I witnessed during an invited talk at NeurIPS 2024 by Professor Rosalind Picard.

I deeply respect Professor Picard’s scholarship and contributions to the field. However, her comments during the talk reflected a deeply troubling and racist view of Chinese scholars. This was not just inappropriate but also profoundly disheartening.

First, it was entirely unnecessary to mention the student’s nationality when discussing an incident of cheating. The point about academic integrity could have been made without emphasizing nationality. Yet, Professor Picard chose to highlight it. This choice perpetuates harmful stereotypes about Chinese scholars and reflects a broader bias against Asians, often rooted in the assumption that we “work hard, avoid conflict, and don’t push back.”

This needs to change. Asians, like everyone else, have the right to speak out and demand accountability when racism occurs. We will ensure that being racist against Asians has consequences, including here, Professor Picard.

What made this incident worse was how it unfolded during the Q&A session. A Chinese attendee asked a professional and thoughtfully articulated question. She began by thanking Professor Picard for her talk and posed this question:
Are you calling out the student’s nationality because you find most Chinese scholars honest, and the fact that the cheating student was Chinese is rare? Is that why you emphasized nationality?

This was a generous and high-EQ question, offering Professor Picard an opportunity to reconsider or clarify her comments. Unfortunately, she doubled down instead.

Professor Picard reinforced her remarks by quoting the student’s excuse —that ethics wasn’t taught in their school—and generalized this as a broader issue with Chinese education. This statement is both factually incorrect and deeply offensive.

There are glaring logical flaws in this argument:
1.If the student cheated, why would their excuse about ethics education be taken at face value? A serious scholar would investigate the claim before making it a central part of their argument.
2.Even if the student’s school didn’t teach ethics (which is false for schools in China), other sources like family and community often instill strong ethical values. Ignoring this nuance is careless and reinforces stereotypes.

What is most heartbreaking is that Professor Picard couldn’t even acknowledge something as simple as: “Most Chinese scholars are honest and upright.” Instead, she focused on the singular exception and added, “Of course, with this one exception in this case” in her response.

I regret that this happened at NeurIPS. I regret that this happened in my research community—a place I have cherished and contributed to for over 14 years. I regret that this happened at MIT, an institution of excellence and aspiration for many Chinese scholars.

Racism has no place in academia, and incidents like this tarnish the principles of inclusion and respect that we, as a global research community, should uphold.

I hope NeurIPS and the broader academic community take this as a wake-up call to address the biases and systemic issues that enable such comments to go unchallenged. We must do better.

@MIT_CSAIL @NeurIPSConf

[Xiangyu Qi @xiangyuqi_pton]

A very important result.
In the past, interpretability easily failed in addressing adversarial examples because adversaries can easily manipulate the model's internal activations to be anything.

This paper clearly shows the same fate for interoperability in the LLM context.