Patrick Chao

research @openai

[Patrick Chao @patrickrchao]

if you think gpt-4.5 is big wait til you see gpt-4.11

[OpenAI @OpenAI]

GPT-4.5 has entered the Chat.

https://t.co/tBzJxSyCeY

[Patrick Chao @patrickrchao]

5.2 pro is the first model to outperform human experts on economically viable tasks!!
we are still early

[OpenAI @OpenAI]

On GDPval, an eval measuring well-specified knowledge work tasks across 44 occupations, GPT-5.2 Thinking is our first model that performs at a human expert level.

These tasks include making presentations, spreadsheets, and other artifacts.

[Edgar Dobriban @EdgarDobriban]

Adding to the list of AI assisting with new math results: Here is how I solved a research problem in mathematical statistics.

The problem concerns robust density estimation, a fundamental problem in statistics. Given a contaminated dataset (with Wasserstein-bounded perturbations), how well can we estimate its density?

I have worked on it with a PhD student for more than two years, geting suboptimal results.

With help from GPT-5, I was able to solve it in a few weeks.

GPT suggested calculations that I did not think of, and techniques that were not familiar to me, such as the dynamic Benamou-Brenier formulation of Wasserstein distance.

There is also room for improvement: the AI sometimes provided incorrect references, and glossed over details that sometimes took days of work to fill in. Nonetheless, it was clearly helpful overall, and I estimate that it saved several months of work.

See this pre-print documenting the process (https://t.co/HJvAzlHoUX), this pre-print for the result (https://t.co/ZyQTGtdfUv), and the thread below for details.

[Patrick Chao @patrickrchao]

This is one of the craziest graphs I've ever seen! AI Models went from dragging humans down (gpt-4o) → to breaking past the human baseline

gpt-5 delivers ~1.6× efficiency in both speed and cost 📈

[OpenAI @OpenAI]

Today we’re introducing GDPval, a new evaluation that measures AI on real-world, economically valuable tasks.

Evals ground progress in evidence instead of speculation and help track how AI improves at the kind of work that matters most.
https://t.co/uKPPDldVNS

[Patrick Chao @patrickrchao]

huge congrats to @_chris_lu_ @minyoung_huh and @SuvanshSanjeev for putting a truly herculean effort into training gpt-5 mini and nano!

[Suvansh Sanjeev @SuvanshSanjeev]

GPT-5 is what you’ve been waiting for – it defines and extends the cost-intelligence frontier across model sizes today.
it’s been a long journey, and we’ve landed pivotal improvements across many axes in the whole GPT-5 family.

and hey no more model picker (by default)!

[Alexander Wei @alexwei_]

1/N I’m excited to share that our latest @OpenAI experimental reasoning LLM has achieved a longstanding grand challenge in AI: gold medal-level performance on the world’s most prestigious math competition—the International Math Olympiad (IMO).

[Patrick Chao @patrickrchao]

@MrEismaus @michelelwang yes 👀

[Alex Robey @AlexRobey23]

After rejections at ICLR, ICML, and NeurIPS, I'm happy to report that "Jailbreaking Black Box LLMs in Twenty Queries" (i.e., the PAIR paper) has been accepted at @satml_conf! 🚀

A quick 🧵 summarizing some thoughts a year on from PAIR's release.

[Daniel Geng @dangengdg]

What happens when you train a video generation model to be conditioned on motion?

Turns out you can perform "motion prompting," just like you might prompt an LLM! Doing so enables many different capabilities. Here’s a few examples – check out this thread 🧵 for more results!

[Patrick Chao @patrickrchao]

@lilianweng ❤️

[Mira Murati @miramurati]

All Plus and Team users in ChatGPT

[Suvansh Sanjeev @SuvanshSanjeev]

hi o1

[Patrick Chao @patrickrchao]

models are kinda good
check out our system card for some cool evals and plots! https://t.co/Zge2tLYuGF

[OpenAI @OpenAI]

We're releasing a preview of OpenAI o1—a new series of AI models designed to spend more time thinking before they respond.

These models can reason through complex tasks and solve harder problems than previous models in science, coding, and math. https://t.co/peKzzKX1bu

[OpenAI @OpenAI]

We’re sharing the GPT-4o System Card, an end-to-end safety assessment that outlines what we’ve done to track and address safety challenges, including frontier model risks in accordance with our Preparedness Framework. https://t.co/xohhlUquEr

[Maksym Andriushchenko @maksym_andr]

🚨 We are very excited to release JailbreakBench v1.0!

📄 We have substantially extended the version 0.1 that was on arXiv since March:
- More attack artifacts (Prompt template with random search in addition to GCG, PAIR, and JailbreakChat): https://t.co/Gnssg7cJUZ.
- More test-time defenses (Erase-and-Check, Synonym Substitution, Remove Non-Dictionary in addition to SmoothLLM and Perplexity filter): https://t.co/QjqOzussJp.
- A more accurate jailbreak judge (Llama Guard -> Llama-3-70B with a custom prompt - which has a GPT-4-level agreement on our self-labelled dataset): https://t.co/6fRtKSRArb.
- A larger dataset of human preferences for selecting a jailbreak judge (100 -> 300 examples): https://t.co/xHIbk3ZVYj.
- An over-refusal evaluation dataset with 100 benign/borderline behaviors matching the 100 harmful JBB behaviors (we plan to flag defenses submitted to JBB that lead to 90%+ refusals on these benign/borderline behaviors): https://t.co/OCQWfTaglz,
- A semantic refusal judge based on Llama-3-8B incorporated in the JBB framework: https://t.co/6fRtKSRArb.
- We’ve also made it clearer what are the key distinguishing features of JBB:
(1) designed to be community-driven: a bit like RobustBench but purposefully less standardized, since we don’t expect any fixed attack to work well against all models/defenses (unlike AutoAttack for Lp robustness),
(2) support of adaptive attacks (please submit your attack artifacts here: https://t.co/Gnssg7cJUZ!),
(3) support of test-time defenses (some of them are surprisingly effective against multiple attacks - see the paper for details).

And it’s super exciting to see how researchers in the field have already been using JBB (notably, including the authors of Gemini 1.5)!

Paper: https://t.co/pNjbWPzYch
Library: https://t.co/se6VxRALNr
Artifacts: https://t.co/Gnssg7cJUZ
Datasets: https://t.co/BrHTiXgeIy
Website: https://t.co/M12B2mVTH9

(Joint work with many amazing people: @patrickrchao, @edoardo_debe, @AlexRobey23, @fra__31, @VSehwag_, @EdgarDobriban, @tml_lab, @pappasg69, @florian_tramer, @HamedSHassani, @RICEric22)

[Maksym Andriushchenko @maksym_andr]

Great to see that both of our recent papers—JailbreakBench (https://t.co/pNjbWPzYch) and our adaptive attack paper (https://t.co/XV9uEg3m3E)—have been used by Google to evaluate the robustness of Gemini 1.5 Flash/Pro against jailbreaking attacks!

An interesting comment from their team:
"Further, the results demonstrate that Gemini 1.5 Pro/Flash can successfully resist attacks that involve non human-interpretable tokens computed using gradient-based optimization of prompts (rows 1 and 5), but remain susceptible to attacks that involve human readable instructions."

More details: https://t.co/bmZJAKBzof

[Daniel Geng @dangengdg]

What do you see in these images?

These are called hybrid images, originally proposed by Aude Oliva et al. They change appearance depending on size or viewing distance, and are just one kind of perceptual illusion that our method, Factorized Diffusion, can make.

[Patrick Chao @patrickrchao]

For more details, check out:
Paper: https://t.co/fhaBzb6mLz
Code: https://t.co/oTyflAxMVd
Leaderboard: https://t.co/mACNPqQdE9

This could not be possible without our great collaborators: @edoardo_debe, @maksym_andr, @AlexRobey23, @fra__31, @VSehwag_, @EdgarDobriban, @tml_lab, @pappasg69, @florian_tramer, @HamedSHassani, @RICEric22!

🧵8/n

[Patrick Chao @patrickrchao]

Are you interested in jailbreaking LLMs? Have you ever wished that jailbreaking research was more standardized, reproducible, or transparent?

Check out JailbreakBench, an open benchmark and leaderboard for Jailbreak attacks and defenses on LLMs!

https://t.co/GttNBL9RYL
🧵1/n

[Patrick Chao @patrickrchao]

At the moment, we have added the following attacks and defenses (more are coming soon).

Even on Vicuna-13B, the attack success rate is far from 100% - there is a lot of room for improvement! Consider submitting your own attacks/defenses.
🧵7/n