๐ Robots in the real world have to move around. Teleoperating a robot to do both at once is still slow and awkward
Today we're excited to release HALO, Human-Aligned Data Collection for Dexterous Whole-body Mobile Manipulation!
HALO lets you hold the robot's grippers directly and just walk, and the platform rolls with you ๐ถ You get natural whole-body motion and fine bimanual control at the same time.
Three key results:
๐ Demo collection approaching natural human speed: operators walk at a normal pace (~0.7 m/s), up to 2.9ร faster than Mobile ALOHA with comparable or higher success
๐ Dexterous skills: the policy model trained on HALO data fastens a short zip tie and disassembles Lego, while policies trained on Mobile ALOHA or VR data fail
๐ Mobile skills: very high success on a six-stage long-horizon mobile manipulation task
Let's check out our demos ๐
๐ ๐ฎ๐ธ๐ถ๐ป๐ด ๐ฑ๐ฒ๐บ๐ผ๐ ๐น๐ผ๐ผ๐ธ ๐น๐ถ๐ธ๐ฒ ๐ฑ๐ฒ๐ฝ๐น๐ผ๐๐บ๐ฒ๐ป๐ ๐ช
Walking with the robot is natural, but the data still has to match what the robot sees and does at test time.
๐น Base motion: two encoders track the platform, and we map their readings to the base's linear and angular velocity, then shift them in time to account for the base's response delay
๐น Vision: we detect the operator's hands (CaRE-Ego), track them with SAM2 and inpaint them away with DiffuEraser
Removing the hands alone lifts Push Chair success from 55% to 85%.
๐๐ฎ๐๐ฎ ๐ฐ๐ผ๐น๐น๐ฒ๐ฐ๐๐ถ๐ผ๐ป ๐ฒ๐ณ๐ณ๐ถ๐ฐ๐ถ๐ฒ๐ป๐ฐ๐ ๐
We compared HALO with Mobile ALOHA and VR teleoperation on Push Chair and Disassemble Lego: same operator, same tasks, same time budget, and the operator had never used any of the three before.
๐น vs Mobile ALOHA: 2.8โ2.9ร faster per successful demo, with equal or higher success
๐น vs VR: 1.6โ2.7ร faster, with 4โ5ร higher success
VR makes precise gripper control hard, while Mobile ALOHA's powered base makes the operator pause before grasping the chair.
๐ช๐ต๐ฎ๐ ๐๐ต๐ฒ ๐ฝ๐ผ๐น๐ถ๐ฐ๐ถ๐ฒ๐ ๐น๐ฒ๐ฎ๐ฟ๐ป ๐ค
We fine-tune ฯ0.5 per task on HALO data. It takes head and wrist cameras and outputs arm joints plus base velocity, using asynchronous real-time action chunking.
๐น Push Chair 85%: handles 2โ3 m changes in start pose and rooms it has never seen
๐น Disassemble Lego 75% (30/40), adapting its strategy to randomized structures
๐น Zip-Tie Fastening 30%, where Mobile ALOHA and VR data give 0%
๐น Long-horizon (shelf โ load โ lift โ drive โ conveyor โ return): 73.3% (11/15)
With the same collection budget, policies trained on Mobile ALOHA or VR data fail on Lego and the long-horizon task.
๐ฃ๐ฎ๐ฝ๐ฒ๐ฟ & ๐ฝ๐ฟ๐ผ๐ท๐ฒ๐ฐ๐ ๐ฝ๐ฎ๐ด๐ฒ ๐
๐ Paper: https://t.co/feFUXVKiGC
๐ Project page: https://t.co/25j2zECxAM
๐ป Code: https://t.co/NlIlNaQcGR
Huge thanks to the team from HKUST and Robbyant, led by Shuai Yang and Fangyi Xu!
๐ Exciting news! @intern_lm 7/20B models are now live on the @huggingface Open LLM Leaderboard!
๐ Highlights:
- 200K context length for base/chat models.
- 20B model is on par with the performance of Yi-34B.
- 7B model is the best in the <= 13B range.
https://t.co/AzpQhlOfhy