Sarvesh Patil

Your friendly neighborhood PhD in Robotics @CMU.
Dexterous Manipulation | Generative Control | Reinforcement Learning

[Sarvesh Patil @servo97]

Interaction with the real world is the major bottleneck in robot learning. So what would robot RL look like if we didn’t need to limit compute per interaction? Our latest work, Off-Policy Generative Policy Optimization (OGPO, accepted to ICML26) embarks on answering this question (spoiler alert: when done correctly, it helps massively!).
🧵(1/N)

[Sarvesh Patil @servo97]

Henlo Frens,
I'll be presenting OGPO at #ICML2026 tomorrow (9 July)
Time ⏰: 2:30 to 4:15 PM KST
Location 📍: Hall A, Poster #202

Feel free to stop by for a chat on RL-finetuning, diffusion/flow policies, dexterous manipulation, or existential philosophy! ⛵️

https://t.co/iU54WPYQMx

[Prasanna Sriganesh @realprassi007]

Do we still need Uncertainty Quantification for robotics? What does uncertainty mean in the modern robotics paradigm?

💡 We are organizing a workshop “Rethinking Uncertainty for Modern Robotics Paradigms” @ IROS 2026

Submit your work by Aug 23! 👇 (https://t.co/AUhtTyDmnK)

[Sarvesh Patil @servo97]

@kvablack Hahaha DDPO and DPPO were my first foray and massive sources of motivation into this entire domain! It's an honor to be Schimidded by the 1st author of Black et al.! :D

[Sarvesh Patil @servo97]

@kartiksharmma9 Indeed! I think there might be a need for understanding the role of adapters in VLA/WAM post-training, and to see if we can get non-degenerate likelihoods purely through the adapter networks. i.e. consider the rest of the VLA like a blackbox, and only update the adapters via IS.

[Sarvesh Patil @servo97]

Interaction with the real world is the major bottleneck in robot learning. So what would robot RL look like if we didn’t need to limit compute per interaction? Our latest work, Off-Policy Generative Policy Optimization (OGPO, accepted to ICML26) embarks on answering this question (spoiler alert: when done correctly, it helps massively!).
🧵(1/N)

[Sarvesh Patil @servo97]

This was an amazing collaboration with @mitsuhiko_nm, with tremendous help from Manan Agarwal, Shashwat Saxena, and @Jesse_Y_Zhang! I would like to thank all the collaborators for their insightful discussions that helped us think widely about the scope of the project. Special thanks to @abhishekunique7, Hongkai Dai, Paarth Shah for all their guidance and insights throughout the project. And finally I would like to express gratitude to my advisor @max_simchowitz for his incredible leadership, mentorship, and encouragement to pursue our intellectual curiosity on understanding the myriad reasons behind OGPO’s performance. I would especially like to thank @max_simchowitz for all his advice and intuition-development-exercises that I hope helps make reinforcement learning with generative models more accessible! :D

Paper: https://t.co/eDyxGSBrg4
Code: https://t.co/rvPFbL8LRj
Website: https://t.co/iU54WPYiWZ
(10/10)

[Sarvesh Patil @servo97]

When done naively, aggressive PPO policy extraction tends to over-optimize Q-functions which can result in an offline-to-online policy “dip” that reduces sample efficiency. We overcome this with two key innovations:(1) An online Success Buffer for supervised finetuning (SFT) on successful actions, (2) a new technique we call “conservative advantages” which selects the “least extreme” advantages for PPO optimization. Both improve OGPO’s overall stability during online RL.
(6/N)

[Sarvesh Patil @servo97]

To summarize, we highlight 6 key takeaways from the success and design of OGPO:

Takeaway #1: Even within the standard Actor-Critic template, simply increasing training time computation can improve sample efficiency by an order of magnitude.

Takeaway #2: Better policy extraction alone can yield substantial improvements in training stability and sample efficiency.

Takeaway #3: Surprisingly, full policy finetuning can increase action diversity and enhance policy exploration, despite the fact that one is only trying to maximize sparse reward (with no explicit entropy penalties!). Understanding this phenomenon is an exciting direction for future work

Takeaway #4: Zero-order policy policy optimization can be incredibly effective given sufficient computation as it avoids unstable gradients through denoising steps or critics, improving performance on high-precision tasks. It also enables advantage-curation techniques, such as the conservative advantages

Takeaway #5: Targeted interventions, like imitating successful trajectories and the modified “conservative advantages” are both reliable, preserve training efficiency, and ameliorate the need for task-specific hyperparameter tuning.

Takeaway #6: While there are many options for fine-tuning a multi-step GCP (AWR, FPO, etc), simple PPO is the most effective, most stable, and requires the least amount of hyperparameter finetuning.
(9/N)

[Sarvesh Patil @servo97]

OGPO leverages the observation that in real world RL, env interactions are costly but compute can be cheaper. Given a learned critic, we sample many denoising trajectories from a given state purely in the robots’ imagination, without needing more time in the real world.

Taking the averages of the critic values gives us a value baseline estimate (V), and we perform a batch gradient update with a PPO loss using A(s, a) = Q(s, a) - V as the advantages. This removes the need for value estimates, and batchwise updates reduce gradient variance (in the same spirit as GRPO).
(5/N)

[Sarvesh Patil @servo97]

Off-policy methods reuse data for sample-efficiency, but usually avoid full-policy updates. As a consequence, many off-policy methods rely on initial noise steering, residual policies, Best-of-N, or single-step policy distillation. The implicit claim is that full-policy off-policy fine tuning might not be necessary for GCPs, especially if they are initialized from strong behavior cloned (BC) policies. Steering or residual learning might be enough.
(3/N)

[Sarvesh Patil @servo97]

In OGPO, we ask: how much can we benefit from full-policy finetuning, and if so, how can it be done stably, reliably and efficiently even with weak BC pretraining? (some emojis)

The core algorithm is simple: use TD learning to learn a critic, then optimize through the denoising chain with PPO-based extraction.
(4/N)

[Sarvesh Patil @servo97]

Up until now there has been a tension in finetuning Generative Control Policies (GCPs), (flow/diffusion). On-policy RL methods like DPPO/ReinFlow show that finetuning the entire denoising process works, but they require prohibitive amounts of environmental interaction data.
(2/N)

[Sarvesh Patil @servo97]

Validation loss can be a red herring measure of model inference with naive optimizers. I have observed this since 2016 (🦕). Awesome work on an approach to mitigate this dissonance which further brings better representation learning at test time as an additional benefit!

[Thomas Zhang @ThomasTCKZhang]

Super excited to finally announce our new paper “Double Preconditioning (DoPr): Optimization for Test-Time Performance, not Validation Loss”

Tl;dr: from LLMs to robotics, on-policy deployment causes a mismatch between validation loss over the training distribution and “downstream performance”.

We propose Double Preconditioning (DoPr), which drops in an activation-based preconditioner into your favorite gradient-wise whitener/normalizer, such as AdamW or Muon, to help close this gap.

1/12 🧵

[dax @thdxr]

gov keeps handing anthropic marketing Ws

[Jason Liu @JasonJZLiu]

💥Introducing FACTR 2, learning external force sensing on commodity robot arms without needing dedicated sensors.

We show that learned force signals enable force-feedback teleop on low-cost arms and improve BC policies.

FACTR 2 consists of:
1. Neural External Torque (NEXT): learns external forces without needing dedicated force sensors.
2. Force-Informed Re-Sampling Training (FIRST): uses the learned force signal to identify task-critical regions and upsample them during training.

w/ @StevenOh_ @_tonytao_

🧵(1/N)

[恒星 @vintcessun]

终于有人把LLM的推理黑箱拆开了：只看最终答案和token数根本不知道模型是在真推理还是瞎绕路，准确率相同的模型推理结构可能天差地别，这才是评估的盲区。
论文从根源下手——构建逻辑谜题基准LRM，把原始推理链条解析成有向无环图，节点是论断，边是依赖关系，再定义推理效率指标量化逻辑流的集中程度。
分析DeepSeek-R1等模型发现：同等准确率下推理图复杂度竟能相差数倍，哪个更可靠一目了然。
https://t.co/oD47nDoF3F

[Peter Holderrieth @peholderrieth]

We release Diamond Maps💎 unlocking accurate and efficient guidance for diffusion models. Our experiments show that our methods scale incredibly well. Excited to see what people will build with this!

Accurate guidance has been a notoriously hard problem, but in this work, we’re bringing TWO (!) solutions to the table. The recipe for success:
1️⃣ Speed: Use distilled models (flow maps, mean flows, consistency models).
2️⃣ Exploration: Inject stochasticity to properly explore your search space.

Because this fundamentally improves anything using flow matching and diffusion, we see a lot of potential for applications across audio, robotics, molecules, and beyond.

Paper: https://t.co/wxtWWRrnw7
Code: https://t.co/WocPtT6orn

Huge thanks to an amazing team: Douglas Chen, @LucaEyring, @ishin_shah, Giri Anantharaman, @electronickale, @zeynepakata, Tommi Jaakkola, @nmboffi, and @max_simchowitz. It was awesome bringing this to life together!