Adam Block

AsstProf@Columbia CS | ML/AI

[Adam Block @adam_block66985]

@thecekbote @NghiMDao @vijaylingam08 Sorry we missed this! It does seem related, but my understanding is that that paper uses a more complicated rollout procedure involving natural language feedback on incorrect solutions as well as a different credit assignment strategy? Moreover, in our paper, eval is single turn.

[Adam Block @adam_block66985]

🚨New paper led by my amazing master's student @NghiMDao! 🚨
Introducing Retry-Aware Policy Optimization (ReAPO). Conditioning retries on failures improves RLVR exploration: ReAPO improves accuracy and training efficiency relative to GRPO.

1/12

[Adam Block @adam_block66985]

@kimtaehyeon610 Good question! In our experiments we always let it see the full failed attempt, but you could definitely imagine using a compressed summary to scale up with larger retry budgets or on harder problems requiring longer CoTs.

[Adam Block @adam_block66985]

@nanjiang_cs @NghiMDao In particular, we primarily care about what you call accuracy at Turn1, not about ensuring that the model is learning to correct itself in order to maximize the final answer (after many attempts) being correct.

[Adam Block @adam_block66985]

@nanjiang_cs @NghiMDao Thanks! Sorry we missed that. I think one key difference is that ReAPO is using external feedback and the serial sampling is intended to aid exploration while training, whereas my understanding is that paper wants to help with instill self-correction during inference?

[Adam Block @adam_block66985]

Next up: longer retry budgets and harder problems.
Thanks to Nghi and our collaborators @caldermf and @GolowichNoah!
Paper: https://t.co/5TXazqIF5f
Code: https://t.co/AOWmAFhkv4
12/12

[Adam Block @adam_block66985]

Credit assignment is important, too. With the same sampling, assigning one advantage to the full chain performs worse. So does retrying after success. The prompt changes little, and a third attempt offers limited benefit on 0.6B, but early results look promising on 1.7B.

11/12

[John Hewitt @johnhewtt]

New from the lab: @LorenaYannnnn finds that reward misspecification during RL causes unmonitorable chain-of-thought. Models talk about the goals in their prompt, even as they learn throughout RL to optimize the misspecified reward. We also propose an intuitive fix; see below.

[Adam Block @adam_block66985]

Really excited to announce this work led by our student @rahulth03as (joint with @micahgoldblum)! We had a great time going back to fundamentals and really thinking about the correct way to run evaluations and what to actually evaluate.

[Micah Goldblum @micahgoldblum]

🚨 LLMs are trained on past data and deployed in the future, but we evaluate them like time doesn't exist. By treating LLM evaluation as a forecasting problem, we can better predict how models will actually perform in the future. 📈 1/n

[Micah Goldblum @micahgoldblum]

Announcing Exploration-Distillation, a RL framework that unlocks aggressive exploration and large performance boosts over standard algorithms like DAPO. In principle, an LLM can discover new things during RL, but existing algorithms struggle. Here’s how we do it. 🧵1/n