Dylan Sam

safety research @openai | formerly phd @mldcmu, @BrownUniversity

[Dylan Sam @dylanjsam]

I defended my PhD thesis!

Also, a very (~4 month) late life update, but I've joined @OpenAI to work on safety research and pretraining safer language models! 📈

Thank you to my advisor @zicokolter and my committee: Matt Fredrikson, @andrew_ilyas, and @furongh! 🙏

[Tomek Korbak @tomekkorbak]

New OpenAI post: Can midtraining on docs about aligned AI bake in alignment priors for agents? We report an experiment where those priors are quickly washed away by RL and fail to generalize to agentic settings. But that cuts both ways: priors that AIs are misaligned fade too!

[Jason Wolfe @w01fe]

I'm also extremely excited for our companion post today on Model Spec Evals! Spec Evals are a new way we're measuring progress towards alignment with the Model Spec — including public results, an open dataset, and code others can build on.

https://t.co/jDWMnns3tM

[Zico Kolter @zicokolter]

As AI agents access more untrusted information with greater autonomy, prompt injections may become the greatest security challenge of our era.

@GraySwanAI, in collaboration many frontier labs, just released our paper on the largest public prompt injection challenge to date.

🧵

[Karan Singhal @thekaransinghal]

https://t.co/sbZncI4gF9

[Marc Finzi @m_finzi]

1/🧵 We are very excited to release our new paper! From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence
https://t.co/M8ETQk9gHz
with amazing team @ShikaiQiu @yidingjiang @Pavel_Izmailov @zicokolter @andrewgwils

[Dylan Sam @dylanjsam]

Finally, I'm presenting work on monitoring models for harmful behaviors, hallucinations, and adversarial manipulation at Poster #1304 in Exhibit Hall C,D,E on 12/5 at 4:30pm!

https://t.co/hIirReXAwZ

[Dylan Sam @dylanjsam]

To trust LLMs in deployment (e.g., agentic frameworks or for generating synthetic data), we should predict how well they will perform. Our paper shows that we can do this by simply asking black-box models multiple follow-up questions! w/ @m_finzi and @zicokolter

1/ 🧵

[Dylan Sam @dylanjsam]

I'm at NeurIPS this week! Excited to meet old/new friends and chat with people about training safer language models.

I'm presenting a few works on safety pretraining, measuring diversity in data curation, and monitoring model behaviors --- more info below 👇

[Dylan Sam @dylanjsam]

Next, I'm presenting on safety pretraining, where we find that incorporating safety behaviors during pretraining leads to more robust language models!

Come by Poster #5210 at Exhibit Hall C,D,E at 4:30pm today (12/4)!

https://t.co/p1mCATySPL

[Dylan Sam @dylanjsam]

🚨Excited to introduce a major development in building safer language models: Safety Pretraining!

Instead of post-hoc alignment, we take a step back and embed safety directly into pretraining.

🧵(1/n)

[Emily Byun @yewonbyun_]

I’m at NeurIPS this week (12/2-12/8) to present our work on when/how synthetic data (e.g., LLM simulations) can help scientists make inferences with less real data, improving the efficiency of costly experiments. Come by Poster #904 on Thursday 4:30PM (Exhibit Hall C,D,E)!🙂

[Pratyush Maini @pratyushmaini]

Excited about our NeurIPS'25 tutorial
Data Privacy, Memorization & Copyright in GenAI

with Cooper (co-founder, GenLaw) & Joe (represents OpenAI, Stability in all US copyright litigations)

We bring together ML researchers, with those who understand its legal implications. Pls RT

[Bryan Wilder @brwilder]

I gave talks at MIT and Harvard this week about "Science with synthetic data". How can generative models help us learn about the world (e.g., social systems) in a principled way? Lots of interesting conversations; more convinced than ever that there's nuanced issues to navigate

[Sachin Goyal @goyalsachin007]

📢 Multi-token prediction has long struggled with defining the right “auxiliary target,” leading to tons of heuristics. We show a core limitation of these and propose a simple & sweet idea: future summary prediction.

Introducing what I call
🚀TL;DR token pretraining🚀

[Yuda Song @yus167]

🤖 Robots rarely see the true world's state—they operate on partial, noisy visual observations.
How should we design algorithms under this partial observability?
Should we decide (end-to-end RL) or distill (from a privileged expert)?
We study this trade-off in locomotion. 🧵(1/n)

[Bryan Wilder @brwilder]

How can synthetic data from LLMs be used, e.g. for social science, in a principled way? Check out Emily's thread on our NeurIPS paper. The key is to generate each synthetic sample by prompting with a real example -- enables debiased estimates that wouldn't be possible otherwise!

[Emily Byun @yewonbyun_]

14/ I’ll be giving a talk on our work at the #COLM2025 Social Simulations workshop tomorrow (Friday 10/10) at 10AM. Come by Room 523AB!🙂

Paper Link: https://t.co/XH2wk06MRA
Code: https://t.co/5NZyzT25p0

[Emily Byun @yewonbyun_]

13/ I really enjoyed working on this project with the brilliant and kindest @shantanug7 and great mentors @zacharylipton, @DonskerClass and @brwilder

[Emily Byun @yewonbyun_]

💡Can we trust synthetic data for statistical inference?

We show that synthetic data (e.g. LLM simulations) can significantly improve the performance of inference tasks. The key intuition lies in the interactions between the moments of synthetic data and those of real data

[Dylan Sam @dylanjsam]

Very interesting insights into understanding when and why synthetic data (although imperfect and biased) can boost the performance of statistical inference!! 📈📈

[Emily Byun @yewonbyun_]

💡Can we trust synthetic data for statistical inference?

We show that synthetic data (e.g. LLM simulations) can significantly improve the performance of inference tasks. The key intuition lies in the interactions between the moments of synthetic data and those of real data