Can LLMs control their chains of thought (CoT)? If so, they could evade CoT monitors 🚨
We introduce the CoT Controllability eval suite to find out.
Our results leave us cautiously optimistic that today’s models struggle to obfuscate their CoT in ways that undermine monitorability.
In this thread, I explain additional findings that I find interesting
Joint work w @OpenAI
if you truly believe in the bitter lesson, then why hand design scaling laws?
introducing: neural neural scaling laws (NeuNeu), a neural network
- trained on open-source LM trajectories
- that predicts LMs' future downstream task performance
🧵👇
Our work on LLM novelty as the frontier of original and high-quality output was accepted to #ICLR26! Come talk to us about how model scale, SFT, and RL affect this trade-off! See you in Brazil!🇧🇷h/t to my awesome collaborators @hhexiy@valeriechen_@JanePan_@jcyhc_ai
🚨Prompt Curriculum Learning (PCL)
- Efficient LLM RL training algo!
- We investigate factors that affect convergence: bsz, # prompt, # gen, prompt selection
- We propose PCL: lightweight algo that *dynamically selects intermediate-difficulty prompts* using a learned value model
Bored of seeing pristine, perfect posters? Come see me at Hall X5, Board 105 at 6pm to witness my masterpiece, featuring bonus Sharpie scribbles and a QR code that betrayed me at the last moment 😤
I'll be at ACL Vienna 🇦🇹 next week presenting this work! If you're around, come say hi on Monday (7/28) from 18:00–19:30 in Hall 4/5. Would love to chat about code model benchmarks 🧠, simulating user interactions 🤝, and human-centered NLP in general!
I'll be at ACL Vienna 🇦🇹 next week presenting this work! If you're around, come say hi on Monday (7/28) from 18:00–19:30 in Hall 4/5. Would love to chat about code model benchmarks 🧠, simulating user interactions 🤝, and human-centered NLP in general!
When benchmarks talk, do LLMs listen?
Our new paper shows that evaluating that code LLMs with interactive feedback significantly affects model performance compared to standard static benchmarks!
Work w/ @RyanShar01, @jacob_pfau, @atalwalkar, @hhexiy, and @valeriechen_!
[1/6]
What does it mean for #LLM output to be novel?
In work w/ @jcyhc_ai, @JanePan_, @valeriechen_, @hhexiy we argue it needs to be both original and high quality. While prompting tricks trade one for the other, better models (scaling/post-training) can shift the novelty frontier 🧵
We're excited to receive wide attention from the community—thank you for your support!
We release code, trained probes, and the generated CoT data👇
https://t.co/Rkw6LJtAyj
We have labeled answer data on its way. Stay tuned!
Reasoning models overthink, generating multiple answers during reasoning. Is it because they can’t tell which ones are right?
No! We find while reasoning models encode strong correctness signals during chain-of-thought, they may not use them optimally.
🧵 below
Our work bridges the gap between existing static benchmarks and real-world usage, and we hope to inspire future work on scalable methods for evaluating models in a collaborative setting.
Read our preprint at https://t.co/gwPhIJDmqR!
[6/6]
When benchmarks talk, do LLMs listen?
Our new paper shows that evaluating that code LLMs with interactive feedback significantly affects model performance compared to standard static benchmarks!
Work w/ @RyanShar01, @jacob_pfau, @atalwalkar, @hhexiy, and @valeriechen_!
[1/6]
We also investigate how much a code model adjusts its solution in response to feedback. Weaker models tend to make many surface-level changes that do not greatly change code behavior; stronger models may make relatively small edits that highly affect code behavior.
[5/6]
@tallinzen@akyurekekin@yoavartzi@NeelNanda5 I really like the paper from Jane Pan (w @danqi_chen) abt this: https://t.co/rnbu4Mnfab. ICL in big models is clearly a mix of task recognition and "real learning" (you're not learning to translate from 3 examples, but you're not getting an arbitrary label mapping from the prior)
Do LLMs exploit imperfect proxies of human preference in context? Yes!
In fact, they do it so severely that iterative refinement can make outputs worse when judged by actual humans. In other words, reward hacking can occur even without gradient updates!
w/ @hhexiy, @sleepinyourhat, @ihsgnef
[1/7]
We follow the canonical definition of reward hacking, observing a divergence between the ground-truth reward (human expert judgment) and its proxy (an LLM judge following the same scoring criteria as the humans).
Our results complement recent work on output degradation via iterative refinement when measured with secondary objectives (https://t.co/x4WpZ8sky4) or with reference-based metrics (https://t.co/31ugQmUcZq).
[6/7]