Zhicheng Cai

PhD @ THU-AIR & Intern @ Bytedance Seed

[Zhicheng Cai @AiolusZero]

Proximal Policy Optimization (PPO) has been the foundational standard for LLM post-training, yet it quietly suffers from a fatal flaw in long-horizon reasoning: exploration collapse.

🧐 We explore the root cause: PPO suffers from a decade-old Geometric Fallacy!

Excited to share our paper published in ICML 2026:
"Beyond Euclidean Clipping: Overcoming Exploration Collapse in LLM RL via Riemannian Isometric Policy Optimization"
paper: https://t.co/fgxDMeJPtr
blog: https://t.co/21lhLP2n43
work with: @hello_gensi @ericguoxy @yaqinzhang @haozhou_ai

🚨 PPO’s Geometric Mismatch
🔻 PPO-Clip implicitly adopts a Euclidean metric, resulting in a geometric mismatch with the non-uniform Riemannian policy manifold.
🔻 This mismatch pathologically suffocates trust regions for low-probability tokens while overly expanding high-probability ones, polarizing the policy and triggering exploration collapse!

🚀 Introducing RIPO (Riemannian Isometric Policy Optimization)
🔹 Method: RIPO dynamically aligns isometric trust regions with local geometry, allowing larger updates for low-probability actions while tightening updates for high-probability ones to achieve a balanced exploration-exploitation trade-off.
🔹 Bonus: Geometric isometry mathematically induces statistical homoscedasticity, balancing bias-variance trade-off for fast, stable training!

📌 Key Results
🔹 35% Avg Gain over GRPO:
RIPO consistently refresh SOTA across 7 competition-level mathematical reasoning benchmarks.

🔹 5x Token Efficiency:
RIPO reaches GRPO's 200-step performance in just 40 steps with highly stable gradient norms.

🔹 Sustained Exploration:
RIPO maintains policy entropy at a healthy level throughout late-stage training, preserving policy diversity.

🔹 Unlocks Pass@K Scaling Ceiling:
On HMMT25, RIPO outperforms GRPO by a massive 50% of Pass@128, breaking the intrinsic ceiling of base models!

🌟 Summary
RIPO is simple, principled, and highly effective. By replacing PPO's heuristic Euclidean clip with a dynamic, geometrically aligned clip, it offers a new foundation for RL scaling in LLM reasoning.

[Zhicheng Cai @AiolusZero]

Appreciate the nice video explaining RIPO (Riemannian Isometric Policy Optimization), thanks! @vinhnx https://t.co/erHGwVNLjq

[alphaXiv @askalphaxiv]

"Beyond Euclidean Clipping: Overcoming Exploration Collapse in LLM RL via Riemannian Isometric Policy Optimization"

PPO-Clip is the default for LLM RL, but it accidentally kills exploration by treating all probability ratio changes as equally large.

This paper shows the bug is geometric. Rare tokens and common tokens live in very different parts of the policy manifold, so fixed clipping under-updates rare but useful reasoning paths and over-updates already dominant ones.

They fix this with probability-aware Riemannian clipping, giving rare actions more room and common actions less.

This gives smoother RL, less exploration collapse, and up to 60% better AIME24 performance over GRPO.

[Zhicheng Cai @AiolusZero]

@fabianfranz Good job and nice visualization! Looking forward to more results!

[Zhicheng Cai @AiolusZero]

Proximal Policy Optimization (PPO) has been the foundational standard for LLM post-training, yet it quietly suffers from a fatal flaw in long-horizon reasoning: exploration collapse.

🧐 We explore the root cause: PPO suffers from a decade-old Geometric Fallacy!

Excited to share our paper published in ICML 2026:
"Beyond Euclidean Clipping: Overcoming Exploration Collapse in LLM RL via Riemannian Isometric Policy Optimization"
paper: https://t.co/fgxDMeJPtr
blog: https://t.co/21lhLP2n43
work with: @hello_gensi @ericguoxy @yaqinzhang @haozhou_ai

🚨 PPO’s Geometric Mismatch
🔻 PPO-Clip implicitly adopts a Euclidean metric, resulting in a geometric mismatch with the non-uniform Riemannian policy manifold.
🔻 This mismatch pathologically suffocates trust regions for low-probability tokens while overly expanding high-probability ones, polarizing the policy and triggering exploration collapse!

🚀 Introducing RIPO (Riemannian Isometric Policy Optimization)
🔹 Method: RIPO dynamically aligns isometric trust regions with local geometry, allowing larger updates for low-probability actions while tightening updates for high-probability ones to achieve a balanced exploration-exploitation trade-off.
🔹 Bonus: Geometric isometry mathematically induces statistical homoscedasticity, balancing bias-variance trade-off for fast, stable training!

📌 Key Results
🔹 35% Avg Gain over GRPO:
RIPO consistently refresh SOTA across 7 competition-level mathematical reasoning benchmarks.

🔹 5x Token Efficiency:
RIPO reaches GRPO's 200-step performance in just 40 steps with highly stable gradient norms.

🔹 Sustained Exploration:
RIPO maintains policy entropy at a healthy level throughout late-stage training, preserving policy diversity.

🔹 Unlocks Pass@K Scaling Ceiling:
On HMMT25, RIPO outperforms GRPO by a massive 50% of Pass@128, breaking the intrinsic ceiling of base models!

🌟 Summary
RIPO is simple, principled, and highly effective. By replacing PPO's heuristic Euclidean clip with a dynamic, geometrically aligned clip, it offers a new foundation for RL scaling in LLM reasoning.

[Zhicheng Cai @AiolusZero]

@xinbinjian Yeah, that's right.
And RIPO's computational complexity is negligible compared to the extremely large computation of LLM's rollout and gradient-update

[Zhicheng Cai @AiolusZero]

Thanks for highlighting our work!

PPO’s Euclidean Clip is geometrically mismatched with the Riemannian policy manifold, causing exploration collapse.
RIPO aligns trust regions with local geometry, achieving superior LLM RL performance!

Happy to discuss!
https://t.co/4lGdwjF8vs

[Xiuyu Li @sheriyuo]

RIPO replaces PPO's fixed importance-ratio clip with a probability-dependent Riemannian trust region, giving rare tokens room to grow while constraining dominant ones and improving RL.

Beyond Euclidean Clipping: Overcoming Exploration Collapse in LLM RL via Riemannian Isometric Policy Optimization
Paper: https://t.co/kO0d4jiqs6

[Zhicheng Cai @AiolusZero]

@bjoern_janson Very insightful interpretation, thanks!

[Zhicheng Cai @AiolusZero]

@sheriyuo Thanks for highlighting our work!

PPO’s Euclidean Clip is geometrically mismatched with the Riemannian policy manifold, causing exploration collapse.
RIPO aligns trust regions with local geometry, achieving superior LLM RL performance!

Happy to discuss!
https://t.co/4lGdwjF8vs

[Zhicheng Cai @AiolusZero]

Proximal Policy Optimization (PPO) has been the foundational standard for LLM post-training, yet it quietly suffers from a fatal flaw in long-horizon reasoning: exploration collapse.

🧐 We explore the root cause: PPO suffers from a decade-old Geometric Fallacy!

Excited to share our paper published in ICML 2026:
"Beyond Euclidean Clipping: Overcoming Exploration Collapse in LLM RL via Riemannian Isometric Policy Optimization"
paper: https://t.co/fgxDMeJPtr
blog: https://t.co/21lhLP2n43
work with: @hello_gensi @ericguoxy @yaqinzhang @haozhou_ai

🚨 PPO’s Geometric Mismatch
🔻 PPO-Clip implicitly adopts a Euclidean metric, resulting in a geometric mismatch with the non-uniform Riemannian policy manifold.
🔻 This mismatch pathologically suffocates trust regions for low-probability tokens while overly expanding high-probability ones, polarizing the policy and triggering exploration collapse!

🚀 Introducing RIPO (Riemannian Isometric Policy Optimization)
🔹 Method: RIPO dynamically aligns isometric trust regions with local geometry, allowing larger updates for low-probability actions while tightening updates for high-probability ones to achieve a balanced exploration-exploitation trade-off.
🔹 Bonus: Geometric isometry mathematically induces statistical homoscedasticity, balancing bias-variance trade-off for fast, stable training!

📌 Key Results
🔹 35% Avg Gain over GRPO:
RIPO consistently refresh SOTA across 7 competition-level mathematical reasoning benchmarks.

🔹 5x Token Efficiency:
RIPO reaches GRPO's 200-step performance in just 40 steps with highly stable gradient norms.

🔹 Sustained Exploration:
RIPO maintains policy entropy at a healthy level throughout late-stage training, preserving policy diversity.

🔹 Unlocks Pass@K Scaling Ceiling:
On HMMT25, RIPO outperforms GRPO by a massive 50% of Pass@128, breaking the intrinsic ceiling of base models!

🌟 Summary
RIPO is simple, principled, and highly effective. By replacing PPO's heuristic Euclidean clip with a dynamic, geometrically aligned clip, it offers a new foundation for RL scaling in LLM reasoning.