Brandon Yifan Yang

Robotics at @Penn GRASP | Honda Research Institute | Robot + AI Research at @AfterQuery | Prev at @UVA

[Brandon Yifan Yang @brandonyyang02]

@Taodav Seems like it’s working quite well? People are voicing their opinions publicly about this, and everyone gets to see them.

[Brandon Yifan Yang @brandonyyang02]

@reggitales @amish_sethi04 Thank you @reggitales

[Brandon Yifan Yang @brandonyyang02]

@paigeinsf @amish_sethi04 If you mean gripper opening and closing, we record this by tracking the distance between the Aruco tags placed on the left and right jaw.

If you mean tactile force, we do not record any tactile force sensor data at the moment.

[Chris Paxton @chris_j_paxton]

Open source UMI solution

[Brandon Yifan Yang @brandonyyang02]

@amish_sethi04 Congrats team @gvelamoor @spotatkd @amish_sethi04 !

[Brandon Yifan Yang @brandonyyang02]

And it definitely wasn’t a one shot…

[Brandon Yifan Yang @brandonyyang02]

Look what we’ve built!

Sexy 😍😍😍

[Amish Sethi @amish_sethi04]

Can collecting robot data be as easy as 𝗽𝗿𝗲𝘀𝘀𝗶𝗻𝗴 𝗿𝗲𝗰𝗼𝗿𝗱?

Introducing UMI-Live, a 𝗰𝗼𝗺𝗽𝗹𝗲𝘁𝗲𝗹𝘆 𝗼𝗽𝗲𝗻-𝘀𝗼𝘂𝗿𝗰𝗲 bimanual UMI we made that takes you from handheld demonstrations all the way to a trained robot policy.

It's built to be easy to use. Each gripper has its own camera, LiDAR, compute, and battery, so there are no base stations to mount and nothing to calibrate in the scene. You press one button and both hands start recording.

While you keep collecting, our pipeline processes finished episodes in parallel and turns them into LeRobot datasets, automatically uploaded on Huggingface. VLAs fine-tuned mostly on this data succeed in 𝟭𝟬/𝟭𝟬 (π0.5) and 𝟴/𝟭𝟬 (MolmoAct2) trials on a real bimanual task.

We're open-sourcing everything, from the hardware designs and build guides to the code and ~40 hours of data 🧵

[Amish Sethi @amish_sethi04]

Can collecting robot data be as easy as 𝗽𝗿𝗲𝘀𝘀𝗶𝗻𝗴 𝗿𝗲𝗰𝗼𝗿𝗱?

Introducing UMI-Live, a 𝗰𝗼𝗺𝗽𝗹𝗲𝘁𝗲𝗹𝘆 𝗼𝗽𝗲𝗻-𝘀𝗼𝘂𝗿𝗰𝗲 bimanual UMI we made that takes you from handheld demonstrations all the way to a trained robot policy.

It's built to be easy to use. Each gripper has its own camera, LiDAR, compute, and battery, so there are no base stations to mount and nothing to calibrate in the scene. You press one button and both hands start recording.

While you keep collecting, our pipeline processes finished episodes in parallel and turns them into LeRobot datasets, automatically uploaded on Huggingface. VLAs fine-tuned mostly on this data succeed in 𝟭𝟬/𝟭𝟬 (π0.5) and 𝟴/𝟭𝟬 (MolmoAct2) trials on a real bimanual task.

We're open-sourcing everything, from the hardware designs and build guides to the code and ~40 hours of data 🧵

[Brandon Yifan Yang @brandonyyang02]

@PrismaXai @castorhat Or if you use openpi's JAX or pytorch compiled version its about 50ms.

[Felix Zheng @felix_zheng_77]

We’ve seen Astra on tabletops. What about around the house?

Can GPT-6 Astra toast a bagel or clean up toys, zero-shot?

We tested 24 household tasks on a Hello Robot Stretch 4. Here’s a glimpse of what worked, what failed, and what surprised us.

[Brandon Yifan Yang @brandonyyang02]

I went through https://t.co/RxKFXaYMpq line by line and created this.

[Brandon Yifan Yang @brandonyyang02]

New unsolved problem in robotics:

"Flip the pot upside down."

[Brandon Yifan Yang @brandonyyang02]

Again, Astra has come out on top, despite not being a Robot Foundational Model. Of course, we are still far from solving reactive manipulation tasks, but I’m very excited to see how far we can go with general foundational models.

[Amish Sethi @amish_sethi04]

Is RoboMME solved?

Robotic memory with Astra can be surprisingly simple: just keep the episode in context, and let it act.

We reached #1 on RoboMME, a large-scale benchmark for memory-dependent robotic manipulation, with Astra. Astra reached 92.1% success , outperforming even the human baseline (90.5%) and every listed method.

We simply kept the full sampled visual history in context and let Astra generate every waypoint and gripper command with no RoboMME-specific training.

We believe the future of robotics is reliable memory which agents have shown promise in. And the next step is to scale it to longer tasks and changing environments for continual learning!

[Amish Sethi @amish_sethi04]

Is RoboMME solved?

Robotic memory with Astra can be surprisingly simple: just keep the episode in context, and let it act.

We reached #1 on RoboMME, a large-scale benchmark for memory-dependent robotic manipulation, with Astra. Astra reached 92.1% success , outperforming even the human baseline (90.5%) and every listed method.

We simply kept the full sampled visual history in context and let Astra generate every waypoint and gripper command with no RoboMME-specific training.

We believe the future of robotics is reliable memory which agents have shown promise in. And the next step is to scale it to longer tasks and changing environments for continual learning!

[Brandon Yifan Yang @brandonyyang02]

@fel1de @DJiafei Well it seems like recently LLMs are moving beyond just the orchestration layer, as we have seen several examples where LLM agents directly output continuous robot actions given a natural language prompt.

[Brandon Yifan Yang @brandonyyang02]

Dense prompts and detailed text instructions weren’t enough to get the agent to tie a knot in our first attempt.

With just one human demonstration video, Astra inferred the actions, mimicked the human movements, and successfully tied the knot.

[Amish Sethi @amish_sethi04]

With the advent of Astra as a policy, we wanted to test it on a difficult dexterous task: tying a shoelace knot.
Given just one video of the human doing the task, Astra figured out how to perform the task on real YAM robots in our custom coding harness. It directly generates end-effector poses and gripper commands, using camera feedback to adjust its actions.
This is an early glimpse into a new era of in-context learning for robotics, where new skills can be acquired at test time.
Code and more demos coming soon!

[Brandon Yifan Yang @brandonyyang02]

@kyiiiifan @sethi_amish The entire rollout took about 90 minutes in real time.

[Brandon Yifan Yang @brandonyyang02]

@TimSong52005757 @sethi_amish @LongLeRobot A good amount👍

[Amish Sethi @amish_sethi04]

With the advent of Astra as a policy, we wanted to test it on a difficult dexterous task: tying a shoelace knot.
Given just one video of the human doing the task, Astra figured out how to perform the task on real YAM robots in our custom coding harness. It directly generates end-effector poses and gripper commands, using camera feedback to adjust its actions.
This is an early glimpse into a new era of in-context learning for robotics, where new skills can be acquired at test time.
Code and more demos coming soon!

[Brandon Yifan Yang @brandonyyang02]

@yuntiandeng I have this command to keep everything running even if my laptop lid is closed.

sudo pmset disablesleep 1

This will completely drain your battery though.