Immanuel Thomas

Recovering PM getting back to building | Prev @Square, @Meta

[Immanuel Thomas @immanuelt]

https://t.co/7NwVDrFbDN

[Immanuel Thomas @immanuelt]

Added simple mobility by calibrating the top camera's (fixed) view of the floor. The roombot can now move to a specified point corresponding to a pixel.

Claude created a calibration marker, and then calibrated both the camera lens and the relative floor position.

Next up, sys2 control with opus. To navigate to a spot and use the gripper.

[Immanuel Thomas @immanuelt]

Surreal! I got @dreamlabs_ai Dream-1 to control my SO-101 (on a Roomba), zero-shot! Followed the prompt 2/3 times.

[Immanuel Thomas @immanuelt]

@jang_yoel @dreamlabs_ai Thanks! and yes, the plan is to add a sys2 and get it to autonomously clean up the room!

[Immanuel Thomas @immanuelt]

Surreal! I got @dreamlabs_ai Dream-1 to control my SO-101 (on a Roomba), zero-shot! Followed the prompt 2/3 times.

[Dream Labs @dreamlabs_ai]

Our mission is to bring "robotics within reach", so today we updated Dream-1 to provide support for the SO-101 with one arm or two, and for single-arm Frankas too, since not all users have a bimanual arm set-up.

Try it out, and let us know what you think :)

[Immanuel Thomas @immanuelt]

@BharatJain @dreamlabs_ai An rpi connected to the arm, and to the roomba's serial port through a FTDI module. lerobot client and various scripts can not control the base for teleop, calibration, etc.

[Immanuel Thomas @immanuelt]

We’re now mobile for the next experiment. Mounted the SO101 and camera on an old Roomba.

What should I run on this? Let’s see if I can get @dreamlabs_ai over API control this one-shot. What else?

[Immanuel Thomas @immanuelt]

Hacking together a new embodiment.

Nov goal: get a new end-to-end functional robotic application up and running

[Immanuel Thomas @immanuelt]

@dreamlabs_ai What timing! Just yesterday I was wondering how to use Dream-1 for my mobile SO101 embodiment.

[Immanuel Thomas @immanuelt]

We’re now mobile for the next experiment. Mounted the SO101 and camera on an old Roomba.

What should I run on this? Let’s see if I can get @dreamlabs_ai over API control this one-shot. What else?

[Immanuel Thomas @immanuelt]

Hacking together a new embodiment.

Nov goal: get a new end-to-end functional robotic application up and running

[Immanuel Thomas @immanuelt]

@walnut_sensei @LeRobotHF RTC?

[Immanuel Thomas @immanuelt]

@rrhoover Even that robot’s face is going like 😬

[Immanuel Thomas @immanuelt]

The gap is sharper looking at first-attempt successes, where the task is completed at first approach and grasp.

SmolVLA: 84%
pi0.5 + LoRA: 60%
pi0.5 frozen VLM: 6%

The frozen pi0.5 aims wide, the scatter (in action expert denoising) is ~5X SmolVLA's. It misses, retries and often carries empty gripper to the bin.

[Immanuel Thomas @immanuelt]

After ~2 hours of rollout across 400 episodes (+ a lot of trial-and-error), the evals show a clearer picture of my experiments with pick-and-place task on the SO101.

At 10K training steps, success rate:
SmolVLA - 88%
pi0.5 - 26%
pi0.5 with lora on the VLM - 76%

Finetuning the VLM with lora closes most of the gap. Action expert was finetuned in all variants

[Immanuel Thomas @immanuelt]

@DJiafei If you squint it already looks like the actuator transfer setup from @DominiqueCAPaul. Maybe @dm_robots will lead the Ketchup Bench eval

[Immanuel Thomas @immanuelt]

me looking at the gripper missing the block by a whisker during eval, after all the data collection, training, parameter sweeping, scene setup, debugging, scripting...

[Immanuel Thomas @immanuelt]

Had a frustrating few sessions with the SO101 and camera setup. The finicky connector on the gripper camera was disconnecting and crashing the rollout. Had to use the good 'ol disconnect-and-connect.

And then couldn't reproduce earlier results from the pi0.5 which I found was caused by the overhead camera which was misaligned while fixing the connector issue!

Always a good idea to run a scene check script to compare the rollout with one of the training runs to see if something is off.

[Immanuel Thomas @immanuelt]

In case you have not already seen, these results from fine-tuning pi05 is very informative. May not address this particular issue you're seeing here, but worth reading and might save you time and money - especially around what not to worry about and how important DAgger is.

https://t.co/QWOWCedxtG

[Dominique Paul | 📍 CoRL @DominiqueCAPaul]

Read the full results in our first blog post: Empirical results fine-tuning π0.5 on a real manufacturing task.
https://t.co/kKQogGVXCm

[Immanuel Thomas @immanuelt]

@ZipMind_AI I didn’t run evals for all the checkpoints within a variant. The scatter does keep reducing and I suspect the success rate would keep increasing. Similar to the 10k vs 20k of the pi05 frozen run.

[Immanuel Thomas @immanuelt]

Ran a more rigorous eval to measure performance of all the variants. Bottom line - you should always train the VLM backbone (at least with LoRA) and not just the action expert.

The setup was an ~in-distribution example, and pi0.5 w/ lora has 2X success rate than the version fine-tuned with the frozen backbone.

And it also saves training compute - the lora fine tuned to 10k steps matches the accuracy of the frozen run at 20k steps! 2X

🧵 1/4

[Immanuel Thomas @immanuelt]

Improving the pi0.5 performance with LoRA finetuning of the VLM backbone. After finetuning just the action expert, predictions had ~5X more scatter than the SmolVLA - measured by sampling from the action head with different input noise.

Finetuning the VLM backbone (lora 16) along with full action head training, reduced the scatter by 63%.

Accuracy also seems much better. Need to measure at scale with a longer eval run.

[Immanuel Thomas @immanuelt]

Checking how much an observation’s action chunks are scattered as the noise is varied, seems like a good proxy for performance.

It’s a great metric because it can be run offline. 4/4

[Immanuel Thomas @immanuelt]

The pi0.5 seems a tad bit faster, finishing the task with a median time of 5.7s while the smolvla takes 6.3s.

This doesn’t count the inference time tho. I was running them on a MacBook - no rtc, no async - and the arm freezes while the pi0.5 model executes the VLM and the 10 denoising steps. 3/4

[Immanuel Thomas @immanuelt]

The smolvla at 10k steps still beats the pi0.5 fine tuned to the same steps. Likely due to pre training with same embodiment as the eval.

Smolvla completed the task at the first attempt more often 96% vs 72% for pi0.5 2/4