Yunhai Han

Robotics PhD at Georgia Tech

[Yunhai Han @HanYunhai]

Can a robot acquire real-world dexterous manipulation skills from just human videos?

Meet Video2Sim2Real: full-stack autonomous dexterous skill acquisition from a single RGB-D human manipulation video — without robot data or expert intervention.

Project: https://t.co/gegklCcs5A
Paper: https://t.co/5AcK8NFQAM

[Kelin Yu @ColinYu14116982]

1/ 👀Vision tells robots where to go. 👋Touch tells robots about the interactions.
2/ Visual policies from teleoperated robot demos and human videos are scaling fast — but they don't have paired tactile data, so they still fail at the last millimeter of contact-rich manipulation.

❓So here is the question:
How can we adapt tactile feedback into pretrained visual policies?

Humans solve this naturally.
1️⃣ Learn from demonstrations through vision: we understand the task structure and the motion priors.
2️⃣ Then practice with touch: we interact with the world, feel what happens, and refine the motion.

🚀 We introduce OmniTacTune: Policy-Agnostic Real-World RL for Tactile Residual Adaptation of Visual Policies ⬇️
✦ Adapting tactile feedback into existing visual policies
✦ No offline tactile demonstrations
✦ Real-world RL in 40–80 minutes, 5–40% → 85–100%
✦ Works across human flow policies, ACT, DP, π0.5
✦ Works across different tactile representations

🌐 Website: https://t.co/I6GOpdUrhP
📖 Paper: https://t.co/YzKRucxxnu
📷 Video: https://t.co/dryarXliCK

In this demo, we show four challenging contact-rich manipulation tasks and a 40-mins, one-take recording of an online RL training demo.

1/n

[Nadun Ranawaka @NadunRanawakaA]

Real robot data is expensive. Real robot evaluations are slow.

Excited to share SimFoundry - a system that turns real scenes into sim-ready worlds for training and benchmarking robots at scale -

✅Automated Scene Reconstruction with asset generation
✅Handles clutter, articulated objects, multiple robot embodiments
✅High Correlation Real-to-Sim Evals
✅Zero-shot Sim-to-Real
✅Generates diverse digital cousins

Less manual environment authoring, more scalable feedback for robot learning.

🌐https://t.co/jmCno7eslU

🧵1/9

[Haonan Chen @HaonanChen_]

Check out our paper on learning forceful manipulation by separating task planning from interaction control!

[Yunhai Han @HanYunhai]

@SanSanychUA Yes all codes will be released soon!

[Yunhai Han @HanYunhai]

@HondaInvestor Yes and no. Yes - learning from human video is getting hot! No - there is significant difference in the method and tasks, meaning that there is still no unified solution yet. But I believe with more ideas exchanged we can make the learning framework more and more powerful.

[Yunhai Han @HanYunhai]

Can a robot acquire real-world dexterous manipulation skills from just human videos?

Meet Video2Sim2Real: full-stack autonomous dexterous skill acquisition from a single RGB-D human manipulation video — without robot data or expert intervention.

Project: https://t.co/gegklCcs5A
Paper: https://t.co/5AcK8NFQAM

[Yangcen Liu @Randle_Liu]

Beyond EgoEngine and Video2Sim2Real: Turning Human Videos into Robot Experience: https://t.co/TzfFAthe9I

[Yunhai Han @HanYunhai]

Takeaways:
- Retargeting a robot trajectory from a human video alone is insufficient; effective execution requires object-centric refinement that leverages hand-object interaction cues extracted from the video.
- Such refinement does not need to be applied continuously over the entire trajectory; instead, it can focus on a small set of key manipulation frames that capture the most critical interaction effects.
- Since this refinement is performed in a digital-twin simulator, reliable real-world deployment also requires effective sim-to-real transfer. To this end, we introduce a decoupled transfer strategy in which global IL adapts to geometric variations, while residual RL handles contact and physics discrepancies.

Big thanks to the team! Yunhai Han, Jianuo Qiu @JianuoQiu, Linhao Bai @lbai46, Ziyu Xiao, Zihang Zeng, Yangcen Liu @Randle_Liu, Zhaodong Yang, Shalin Jain, Wenrui Ma, Jiaqi Fu @JiaqiFu17693, Yuqian Zheng, Manisha Natarajan, Muhammad Zubair Irshad @mzubairirshad, Kenneth Shaw @kenny__shaw, Matthew Gombolay @MatthewGombolay, Zsolt Kira @zsoltkira, and Harish Ravichandar @h_ravichandar.

[Yunhai Han @HanYunhai]

We evaluate on 7 everyday dexterous tasks, including fruit placement, steak seasoning, toy rearrangement, tissue handover, book passing, and tray retrieval.

In real-world trials with object-pose variations, Video2Sim2Real achieves a 95.7% success rate, significantly outperforming both RL-only and IL-only sim-to-real methods.

More task videos and results are available on our project website.

[Yunhai Han @HanYunhai]

@tomssilver Hi Tom, I really appreciate that you like our work! We’re also working on extensions to incorporate visual inputs and other capabilities, which we hope to publish soon.

[Yangcen Liu @Randle_Liu]

Project website: https://t.co/mrBAjewrlg
Arxiv Link: https://t.co/L61pKejIFD
Big thanks to: @wshin1726 @HanYunhai @DanielZhenyang @h_ravichandar @danfei_xu

[Yangcen Liu @Randle_Liu]

What if one unified method helps robots learn from human videos across many tasks, many robots?
Meet ImMimic: Cross-Domain Imitation from Human Videos via Mapping and Interpolation (CoRL 2025 Oral Presentation🏆) @ICatGT
Check it here https://t.co/mrBAjewrlg!

[Yangcen Liu @Randle_Liu]

Can robots learn from human videos for different embodiments?
ImMimic: leveraging DTW mapping and MixUp interpolation to co-train from retargeted human hand poses and robot demonstrations.
RSS Dexterous Manipulation Workshop: https://t.co/ipCCvdZRiV.

[Hongyi Chen @chen_hongyi_]

#CoRL2024 accepted!🌈

Our work KOROL developed a linear dynamics model using object features that capture key information for robotic manipulation, outperforming models that rely on GT object states.

Code: https://t.co/3WWJZGsH1S

[Kelin Yu @ColinYu14116982]

Introducing MimicTouch, our new paper accepted by #CoRL2024 (also the Best Paper Award at the #NIPS2024 TouchProcessing Workshop).

MimicTouch learns tactile-only policies (no visual feedback) for contact-rich manipulation directly from human hand demonstrations. (1/6)

[Yunhai Han @HanYunhai]

@bauzavillalonga Just applied!