Chen Wang

Final-year CS PhD @Stanford. Prev @GoogleDeepMind @NVIDIA @MIT_CSAIL. Robotics/Manipulation

[Chen Wang @chenwang_j]

Can we use wearable devices to collect robot data without actual robots?

Yes! With a pair of gloves🧤!

Introducing DexCap, a portable hand motion capture system that collects 3D data (point cloud + finger motion) for training robots with dexterous hands

Everything open-sourced

[Ruohan Zhang @RuohanZhang76]

I will join Northwestern University Computer Science as an Assistant Professor in Fall 2026!

I am actively recruiting PhD students and seeking collaborations in robotics, human-robot interaction, brain-computer interfaces, cognitive science, societal impact of AI & automation, and AI for art & design.

Please see the recruitment announcement on my personal website, and feel free to reach out!

[Chen Wang @chenwang_j]

@HaozhiQ @YiMaTweets @JitendraMalikCV Congrats Haozhi!

[Yunfan Jiang @YunfanJiang]

🤖 Ever wondered what robots need to truly help humans around the house?

🏡 Introducing 𝗕𝗘𝗛𝗔𝗩𝗜𝗢𝗥 𝗥𝗼𝗯𝗼𝘁 𝗦𝘂𝗶𝘁𝗲 (𝗕𝗥𝗦)—a comprehensive framework for mastering mobile whole-body manipulation across diverse household tasks!

🧹🫧 From taking out the trash to laying out clothes and cleaning toilets—𝗕𝗥𝗦 equips robots to handle practical, everyday activities.

🌐 Explore more: https://t.co/CwU7xj6n8Z

Let's dive in! 🤿🧵

[Toru @ToruO_O]

Sim2Real RL for Vision-Based Dexterous Manipulation on Humanoids

https://t.co/9yJqaOxhs9

TLDR - we train a humanoid robot with two multifingered hands to perform a range of dexterous manipulation tasks

robust generalization and high performance without human demonstration :D

[Chen Wang @chenwang_j]

@agrimgupta92 Congrats Agrim!🤩

[Hao-Shu Fang @haoshu_fang]

A good hand can push intelligence development. Introducing Eyesight Hand, equipped with full-hand high-res tactile sensors and proprioceptive actuators. It is compliant, agile, and powerful. Good tactile sensing makes learning more efficient and robust. Shout out to Branden!

[Simar Kareer @simar_kareer]

Introducing EgoMimic - just wear a pair of Project Aria @meta_aria smart glasses 👓 to scale up your imitation learning datasets!

Check out what our robot can do.

A thread below👇

[Chen Wang @chenwang_j]

@DJiafei Thanks Jiafei for first introducing AR for data collection! We learned a lot from AR2-D2 and EVE

[Chen Wang @chenwang_j]

Excited to introduce ARCap! We found that visual feedback is crucial for high-quality data collection, and AR can greatly help! We invited 20 novice users to each gather a small amount of data using only AR—no robot hardware required. The combined data can successfully train a robot policy!

[Sirui Chen @eric_srchen]

How can we collect high-quality robot data without teleoperation? AR can help!

Introducing ARCap, a fully open-sourced AR solution for collecting cross-embodiment robot data (gripper and dex hand) directly using human hands.

🌐:https://t.co/GPiSqxvD5M
📜:https://t.co/NMmziT6FQr

[Tianyuan Dai @RogerDai1217]

Why hand-engineer digital twins when digital cousins are free?

Check out ACDC: Automated Creation of Digital Cousins 👭 for Robust Policy Learning, accepted at @corl2024! 🎉

📸 Single image -> 🏡 Interactive scene

⏩ Fully automatic (no annotations needed!)
🦾 Robot policies deployed zero-shot in original scene

🌐: https://t.co/GRsPGLuEaV

[AK @_akhaliq]

Synchronize Dual Hands for Physics-Based Dexterous Guitar Playing

discuss: https://t.co/Or54pmQmw6

We present a novel approach to synthesize dexterous motions for physically simulated hands in tasks that require coordination between the control of two hands with high temporal precision. Instead of directly learning a joint policy to control two hands, our approach performs bimanual control through cooperative learning where each hand is treated as an individual agent. The individual policies for each hand are first trained separately, and then synchronized through latent space manipulation in a centralized environment to serve as a joint policy for two-hand control. By doing so, we avoid directly performing policy learning in the joint state-action space of two hands with higher dimensions, greatly improving the overall training efficiency. We demonstrate the effectiveness of our proposed approach in the challenging guitar-playing task. The virtual guitarist trained by our approach can synthesize motions from unstructured reference data of general guitar-playing practice motions, and accurately play diverse rhythms with complex chord pressing and string picking patterns based on the input guitar tabs that do not exist in the references. Along with this paper, we provide the motion capture data that we collected as the reference for policy training.

[Chen Wang @chenwang_j]

@shahdhruv_ @Princeton Congrats!!

[Chen Wang @chenwang_j]

@xiaolonw Congrats!

[Chen Wang @chenwang_j]

@elliottszwu @Cambridge_Eng @_atewari Congrats Elliott!

[Chen Wang @chenwang_j]

We found that the relations between keypoints are a powerful way to represent tasks. What’s more exciting is that these keypoint relations can be formulated as constraint satisfaction problems, allowing us to use off-the-shelf optimization solvers to generate complex robot actions. Check out @wenlong_huang's thread for more on how we automate the keypoint constraints generation process and how this enables robots to perform reactive, bimanual, and long-horizon tasks!

[Wenlong Huang @wenlong_huang]

What structural task representation enables multi-stage, in-the-wild, bimanual, reactive manipulation?

Introducing ReKep: LVM to label keypoints & VLM to write keypoint-based constraints, solve w/ optimization for diverse tasks, w/o task-specific training or env models.

🧵👇

[Chen Wang @chenwang_j]

We are hosting the dexterous manipulation workshop at CoRL this year 🤖! We'll dive into topics like visual & tactile perception, skill learning, and control. Don’t miss the opportunity to share your amazing works and participate! https://t.co/WqBbFdDx7L

[Haozhi Qi @HaozhiQ]

🎺 Announcing our CoRL 2024 “Learning Robot Fine and Dexterous Manipulation:
Perception and Control” in Munich.

Join us to hear from an incredible lineup of speakers!

And don’t miss the opportunity to submit your work and participate!

Checkout: https://t.co/QGVJvu2Rhv

[Google DeepMind @GoogleDeepMind]

Meet our AI-powered robot that’s ready to play table tennis. 🤖🏓

It’s the first agent to achieve amateur human level performance in this sport. Here’s how it works. 🧵

[Chen Wang @chenwang_j]

@zhijianliu_ @UCSanDiego @HDSIUCSD huge congrats!

[Chen Wang @chenwang_j]

@shahdhruv_ @svlevine @berkeley_ai @GoogleDeepMind Congrats!

[Qiayuan Liao @qiayuanliao]

Excited to share a new humanoid robot platform we’ve been working on. Berkeley Humanoid is a reliable and low-cost mid-scale research platform for learning-based control. We demonstrate the robot walking on various terrains and dynamic hopping with a simple RL controller.