Check out @ZihaoZhao1's thread on our new paper!
We compare various techniques to mitigate privacy leakage, finding that synthetic data and LLM rewriting offers a particularly strong privacy/capability trade-off 📈📈
We’ve all offloaded the writing to LLMs, but have you considered REWRITING with LLMs? 👀
Rewriting your private data with an LLM before finetuning gives a better empirical privacy vs. capability trade-off than PII scrubbing or DP-SGD!
To test this, we generated 5,000 synthetic medical records with private information. Then, we finetuned Qwen3.5-4B and gpt-oss-20b with several methods to mitigate privacy leakage and asked two things: does the model learn from the medical records, and does it leak private information?
1/n 🧵👇
I am quite excited about this work on scaling gradient based data attribution to find innocuous looking harmful data.
"What's in your Safe Data" (@LuxiHeLucy) was one of my favorite papers from 2024, and Emergent Misalignment (@BetleyJan) has been one of the most important papers from 2025. Scalable gradients can connect the two. A few things that are exciting here:
1. Gradient attribution can scale: We built on top of the LoRIF paper (Li et. al.), and further simplified it. Gradients are now only 3x slower than standard embeddings.
2. Small models can be strong monitors. A 0.6B model can be used as monitor for harmful data that may enter the training of much larger models.
3. EM data might only detectable after training: The representations in gradient space often align after the model itself has had exposure to the safe-looking harmful data. This is quite intriguing, as it suggests attribution rather than filters as a path towards preventing EM.
https://t.co/PqEsqJGOwM
Foundation models are trained in stages: pretraining, midtraining, and post-training. But is this separation fundamental?
We introduce unified training: a single algorithm for the full pipeline. This gives
⚙️ Simpler infrastructure
🔬 A framework for better science
(1/n)
1/
When LLMs answer questions with retrieved webpages, several plausible answers may appear.
What if one appears much more often and fits much of what the question asks, but still isn't the right answer?
Excited to share our #EMNLP2026 paper on this!
📄https://t.co/1nA6wYCpro
We’ve all offloaded the writing to LLMs, but have you considered REWRITING with LLMs? 👀
Rewriting your private data with an LLM before finetuning gives a better empirical privacy vs. capability trade-off than PII scrubbing or DP-SGD!
To test this, we generated 5,000 synthetic medical records with private information. Then, we finetuned Qwen3.5-4B and gpt-oss-20b with several methods to mitigate privacy leakage and asked two things: does the model learn from the medical records, and does it leak private information?
1/n 🧵👇
We're hiring safety researchers at @thinkymachines! 🦺
My team works on safety across the model development stack -- pre-training data filtering, harmful capability evals, safety post-training, red-teaming, abliteration/malicious fine-tuning.
We're particularly interested in building evals and tooling to make strong safety cases for open-weights releases. We think this is where some of the most important safety research will happen in the next few years.
If any of this resonates, apply here: https://t.co/SP75pGaGoY
A lot of excitement in synthetic data for science is motivated by ambitious uses of LLMs (e.g., running a survey with GPT before real participants, forecasting promising experiments with LLMs). We ask, what beliefs should the experimenter have to make this a good idea? 🧵
9/ This work was done with the amazing @brwilder. I will be presenting this work today at #ICML 🇰🇷 (Thursday 5:00-6:45 PM KST) in Hall A #3605. Feel free to stop by or reach out to chat! Would love to meet old/new friends! :)
Paper Link: https://t.co/trUrPJuXqH
A lot of excitement in synthetic data for science is motivated by ambitious uses of LLMs (e.g., running a survey with GPT before real participants, forecasting promising experiments with LLMs). We ask, what beliefs should the experimenter have to make this a good idea? 🧵
Hellooo!
I’ll be at @icmlconf with @OpenAI! If you’re interested in safety research at OAI generally or pretraining safety research I’d love to chat!
You can find me hosting a Q&A at our booth Wednesday morning at 9:30am!
Thanks for having me! I talked about our work on valid inference with synthetic data (https://t.co/yIzGKRL8Yq) and robust human-AI complementarity (ICML 2026, paper up soon), both with my PhD student @yewonbyun_
New OpenAI post: Can midtraining on docs about aligned AI bake in alignment priors for agents? We report an experiment where those priors are quickly washed away by RL and fail to generalize to agentic settings. But that cuts both ways: priors that AIs are misaligned fade too!
I'm also extremely excited for our companion post today on Model Spec Evals! Spec Evals are a new way we're measuring progress towards alignment with the Model Spec — including public results, an open dataset, and code others can build on.
https://t.co/jDWMnns3tM
As AI agents access more untrusted information with greater autonomy, prompt injections may become the greatest security challenge of our era.
@GraySwanAI, in collaboration many frontier labs, just released our paper on the largest public prompt injection challenge to date.
🧵
I defended my PhD thesis!
Also, a very (~4 month) late life update, but I've joined @OpenAI to work on safety research and pretraining safer language models! 📈
Thank you to my advisor @zicokolter and my committee: Matt Fredrikson, @andrew_ilyas, and @furongh! 🙏
1/🧵 We are very excited to release our new paper! From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence
https://t.co/M8ETQk9gHz
with amazing team @ShikaiQiu@yidingjiang@Pavel_Izmailov@zicokolter@andrewgwils
Finally, I'm presenting work on monitoring models for harmful behaviors, hallucinations, and adversarial manipulation at Poster #1304 in Exhibit Hall C,D,E on 12/5 at 4:30pm!
https://t.co/hIirReXAwZ
To trust LLMs in deployment (e.g., agentic frameworks or for generating synthetic data), we should predict how well they will perform. Our paper shows that we can do this by simply asking black-box models multiple follow-up questions! w/ @m_finzi and @zicokolter
1/ 🧵
I'm at NeurIPS this week! Excited to meet old/new friends and chat with people about training safer language models.
I'm presenting a few works on safety pretraining, measuring diversity in data curation, and monitoring model behaviors --- more info below 👇
Next, I'm presenting on safety pretraining, where we find that incorporating safety behaviors during pretraining leads to more robust language models!
Come by Poster #5210 at Exhibit Hall C,D,E at 4:30pm today (12/4)!
https://t.co/p1mCATySPL
🚨Excited to introduce a major development in building safer language models: Safety Pretraining!
Instead of post-hoc alignment, we take a step back and embed safety directly into pretraining.
🧵(1/n)