Jason Wei

studying @scsatcmu | researching @scale_ai

[Jason Wei @jaswwei]

1/ As agentic tasks grow longer and rollouts become more expensive, the choice between critic-based PPO and critic-free GRPO becomes increasingly important.

Critics can reduce gradient variance, allowing models to learn from fewer trajectories, but they also introduce additional compute costs and potential bias from bootstrapping.

Our new research @ScaleAILabs compares PPO and GRPO under a shared actor objective across 1,200+ training runs in control tasks, short-horizon math, and long-horizon terminal agents. We studied how three factors influence this tradeoff: batch size, critic reliance, and task complexity. 🧵

[Jason Wei @jaswwei]

@BetaTomorrow Thanks! I would imagine the general findings would hold. But since larger models are more sample efficient, I would expect a smaller crossover batch size

[Jason Wei @jaswwei]

1/ As agentic tasks grow longer and rollouts become more expensive, the choice between critic-based PPO and critic-free GRPO becomes increasingly important.

Critics can reduce gradient variance, allowing models to learn from fewer trajectories, but they also introduce additional compute costs and potential bias from bootstrapping.

Our new research @ScaleAILabs compares PPO and GRPO under a shared actor objective across 1,200+ training runs in control tasks, short-horizon math, and long-horizon terminal agents. We studied how three factors influence this tradeoff: batch size, critic reliance, and task complexity. 🧵

[头雁 @alacheng]

好文：什么时候该用带 Critic 的 PPO，什么时候用无 Critic 的 GRPO。⁠
https://t.co/5vcZyFyPRO

[Jason Wei @jaswwei]

@emilianogl you're incredible

[Jason Wei @jaswwei]

@DarshK_2023 Thanks Darsh!

[Jason Wei @jaswwei]

@tanaytndn Thanks Tanay!

[Jason Wei @jaswwei]

@AIchanyi Thanks! yeah I think the crossover point shifts down slightly with a smaller critic from a few experiments I've done

[Jason Wei @jaswwei]

@ShawnSYFeng Thanks Shengyu!

[Xudong Han @Xudong07452910]

Scale AI 这个 GRPO vs PPO 的研究提供了一个很好的 insight。

现在的 Agent 任务越来越长、Rollout 成本越来越高，Critic 的价值可能被低估了。实验发现，小 Batch 下 PPO 更有优势，而 GRPO 通常需要更大的 Batch 才能追上。

我记得之前智谱发布 GLM-5.2 时，就已经明确转向 Critic-based PPO，后续的 SAO 论文也进一步验证了这条路线的可行性。

感觉随着 Agent RL 向长时任务发展，如何提高单条 Rollout 的学习效率，会变得越来越重要。

[Yuan He @lawhy_X]

Nice post! Would also be interesting to see how the gap between GRPO and PPO changes with score centering.

[Chinmay @ChinmayKak]

This is a generally good read on the whole PPO vs GRPO debate. The main take-aways are:
> batch size and critic dependence decide the win
> surprisingly, PPO at small scale is worth the extra compute to train the critic.
> As task complexity grows, GRPO needs larger batch size to reduce variance (as also seen in the SAO paper)

Also very basic and important notes on bias vs variance tradeoff and formulations of how variance scales inversely with batch size.

[Jason Wei @jaswwei]

@teortaxesTex https://t.co/jwRbj7Fo9U

[Jason Wei @jaswwei]

1/ As agentic tasks grow longer and rollouts become more expensive, the choice between critic-based PPO and critic-free GRPO becomes increasingly important.

Critics can reduce gradient variance, allowing models to learn from fewer trajectories, but they also introduce additional compute costs and potential bias from bootstrapping.

Our new research @ScaleAILabs compares PPO and GRPO under a shared actor objective across 1,200+ training runs in control tasks, short-horizon math, and long-horizon terminal agents. We studied how three factors influence this tradeoff: batch size, critic reliance, and task complexity. 🧵

[Teortaxes▶️ (DeepSeek 推特🐋铁粉 2023 – ∞) @teortaxesTex]

1) Lightly patched GRPO was clearly optimal until the long-horizon agent era, just no good reason to turn to anything more complicated.
2) in this era, we really want more granular rewards.
3) my own guess is that most of the critic benefit can be harvested with ≈10% of steps

[stochasm @stochasticchasm]

this is gonna be a good read. finally

[khazzz1c @Chonghan_Liu]

they’ve ran every test I had in mind lol.

https://t.co/0WNcH0IJw0

[Jason Wei @jaswwei]

@Chonghan_Liu Thank you! I wrote more about it here!

https://t.co/6VFsdWqTRB

[Jason Wei @jaswwei]

1/ As agentic tasks grow longer and rollouts become more expensive, the choice between critic-based PPO and critic-free GRPO becomes increasingly important.

Critics can reduce gradient variance, allowing models to learn from fewer trajectories, but they also introduce additional compute costs and potential bias from bootstrapping.

Our new research @ScaleAILabs compares PPO and GRPO under a shared actor objective across 1,200+ training runs in control tasks, short-horizon math, and long-horizon terminal agents. We studied how three factors influence this tradeoff: batch size, critic reliance, and task complexity. 🧵

[Penghui Qi @QPHutu]

This is really a nice study and the results make a lot of sense to me.🫡

Only one question: if the bootstrapping bias matters, why we must need bootstrap?🤣 Will lambda=1 (no bootstrap) be better for critic-based PPO?

[Jingxuan Fan ➡️ NeurIPS 2026 @fjxdaisy]

GRPO variants have been dominating for math reasoning tasks but PPO starts to be revisited more and more for agentic post-training. In this work, a comprehensive characterizations of PPO–GRPO tradeoff across 1,200+ runs reveals that PPO leads at small batch sizes, and GRPO needs increasingly larger batches to catch up as tasks get longer and harder. Our findings guide algorithm selection based on factors like batch size, task complexity and compute budget. Check out the amazing work from @jaswwei

[Lei Li @_TobiasLee]

wow :)

[Jason Wei @jaswwei]

@QPHutu Thank you @QPHutu! Big fan of your work!

[Jason Wei @jaswwei]

We actually tried PPO lambda = 1 (gray line) in our control settings, and it performed very similarly to PPO alpha = 1.5, so alpha 1.5 doesn't actually introduce much bias.

Another interesting finding:
We also compared GRPO with and without std normalization (Dr. GRPO). With continuous rewards, std normalization can go horribly wrong if group stds differ by orders of magnitude. Normalizing gives groups whose returns barely vary just as much weight as the ones with the most to learn. This case only appeared in Pendulum at H = 25.

[Penghui Qi @QPHutu]

This is really a nice study and the results make a lot of sense to me.🫡

Only one question: if the bootstrapping bias matters, why we must need bootstrap?🤣 Will lambda=1 (no bootstrap) be better for critic-based PPO?