Zhou Zihan

CS PhD student at UofT.

[Zhou Zihan @zzhouredo]

SPIRE was the result of a wonderful collaboration with @animesh_garg, Dieter Fox, @CaelanGarrett, and @AjayMandlekar.

Website: https://t.co/BEi3hQViQY
arXiv: https://t.co/UsAq1qovy3

[Zhou Zihan @zzhouredo]

One common issue with using RL-based finetuning is that new behavior does not preserve safety considerations implicit in the human teleoperation data. In contrast, our RL finetuning scheme allows SPIRE agents to preserve human behavior, allowing for safe deployments.

🧵 6/

[Zhou Zihan @zzhouredo]

SPIRE agents reach an 80% success rate in 8 out of 9 challenging long-horizon tasks while only using 60% of the time needed by BC agents to finish the task. SPIRE achieves the same level of proficiency as alternatives using only 17% of the demonstrations.

🧵 8/

[Zhou Zihan @zzhouredo]

With RL fine-tuning, SPIRE massively improves both the success rate and task completion speed over a naive BC agent, allowing us to train superior policies with as few as 10 demos.

🧵 5/

[Zhou Zihan @zzhouredo]

SPIRE trains a residual policy with RL to fine-tune the BC policy. A KL penalty term prevents it from deviating from the BC policy. This allows for structured exploration and use of sparse task completion rewards - no more painful reward tuning!

🧵 4/

[Zhou Zihan @zzhouredo]

SPIRE agents consistently solve long-horizon tasks, including those with 5 sequential sections, using only a handful of demonstrations.

🧵 3/

[Zhou Zihan @zzhouredo]

Want your robot to make a cup of coffee but don’t want to spend hours collecting demos?

We introduce SPIRE, a system that solves long-horizon manipulation tasks with limited demos through planning, Behavior Cloning, and Reinforcement Learning.
#CoRL2024 #NVIDIAResearch 👇

🧵 1/

[Zhou Zihan @zzhouredo]

SPIRE decomposes long-horizon tasks into shorter subtasks using task and motion planning. Challenging subtasks not solvable by planning are deferred to a human teleoperator. Then, it trains behavior cloning policies with the collected data and fine-tunes them with RL.

🧵 2/

[Zhou Zihan @zzhouredo]

This project is done under the supervision of @animesh_garg. Check out our paper and code repo to learn more!

Paper: https://t.co/aorkFcrAhi
Code: https://t.co/nPXcaQB8ly

(10/10)

[Zhou Zihan @zzhouredo]

Exploration in complex environments has always been a major challenge in RL, especially in procedurally-generated environments where state-oriented methods can be ineffective. In our ICLR 2023 submission, we propose to explore the concept of achievements as a solution. (1/n)

[Zhou Zihan @zzhouredo]

This is a video of a SEA agent unlocking collect_diamond, the hardest achievement in Crafter. (9/n)