Thomas Lummerstorfer | Physical AI

Physical AI with the SO-101 🦾
Sharing experiments, failures & lessons
Stream my robot live ⬇️

[Thomas Lummerstorfer | Physical AI @zerotorobots]

I am teaching LeRobot to pickup a pen, draw a square and put the pen back. It’s difficult to work with the cheap hardware, but its still absolutely amazing what you get for just a couple hundred dollars. This was the first episode

[Thomas Lummerstorfer | Physical AI @zerotorobots]

This is an example for remote controlling the robot via https://t.co/3P2sArhPLm💫

uncut, real-time, no filter. just screen recording myself controlling the robot to pick up a box and placing it on top of another box 🤖

[Thomas Lummerstorfer | Physical AI @zerotorobots]

Next up: Preparing my SO-101 for autonomous tasks. 🤖

Built a simple white workspace to reduce visual distractions and create a more consistent environment for training.

Let’s see how well it works! 📦

[Thomas Lummerstorfer | Physical AI @zerotorobots]

currently the robot is not yet running 24/7. i need a remote control power socket first ⚡️

[Thomas Lummerstorfer | Physical AI @zerotorobots]

robots are cool. i built a website for mine. the video feed is live. for now you can only trigger a simple movement. fully deterministic for now..

➡️ https://t.co/3P2sArhPLm

what would you like to see?

[Thomas Lummerstorfer | Physical AI @zerotorobots]

🤖 Built a web app to livestream and remotely control my SO-101 robot arm.

Two control modes:

- Direct joint control
- Cartesian control using inverse kinematics (IK)

One step closer to making my robot accessible from anywhere. 🌍

[Thomas Lummerstorfer | Physical AI @zerotorobots]

stream the video from my robot in my living room live here -> https://t.co/3P2sArhhVO

active controls are coming soon 👀

[Thomas Lummerstorfer | Physical AI @zerotorobots]

it‘s now live at https://t.co/3P2sArhhVO

[Thomas Lummerstorfer | Physical AI @zerotorobots]

i am working on a website where anyone can control my robot (SO-101) in real time and stream it live over the internet

[Thomas Lummerstorfer | Physical AI @zerotorobots]

@DominiqueCAPaul @aurel_arnold wow really impressive

[Thomas Lummerstorfer | Physical AI @zerotorobots]

coming very soon soon

[Thomas Lummerstorfer | Physical AI @zerotorobots]

i tried controlling the SO101 solely with astra. not a good idea. it takes ages and even then has states where it fails

[Thomas Lummerstorfer | Physical AI @zerotorobots]

i did my first flight from zugerberg yesterday 🪂 wondering if paragliding can help to generate useful robotics data

[Thomas Lummerstorfer | Physical AI @zerotorobots]

i got astra to move boxes inside a bowl under different lightning. it can recover from failures itself. but it takes ages to complete a run. sometimes 90min, i really need to improve on that. the goal is 30seconds. and what nobody tells you its ofc burning through lots of tokens

[Thomas Lummerstorfer | Physical AI @zerotorobots]

its now calibrating the motors and joint movements

[Thomas Lummerstorfer | Physical AI @zerotorobots]

ditching the current imitation learning approach and letting #astra take over. currently calibrating the cameras. could be very interesting to use smart models for error recovery where latency, price and co are less relevant

[thijs @cdngdev]

i gave astra a robot, a paint brush, and a camera then asked it to paint the golden gate bridge in real life!

it figured out how to control the robot, and progressively got better throughout its attempts. the timelapse is sick

[Thomas Lummerstorfer | Physical AI @zerotorobots]

training failure recovery was more difficult than expected. the model learned from the failure recovery episodes and then didnt correctly aim at the boxes anymore

[Thomas Lummerstorfer | Physical AI @zerotorobots]

somehow lerobot was struggling to pick up boxes in some scenarios. to fix this i recorded episodes on failure recovery. this should help him understand and recover from his mistakes. currently the policy is being trained. cant wait to see after its done

[Thomas Lummerstorfer | Physical AI @zerotorobots]

@cdngdev imitation learning is over. if you think about it, humans dont need someone to demonstrate a task 50 times in order to learn it. once or twice is usually enough. i am trying similar experiments with my SO101 now

[Thomas Lummerstorfer | Physical AI @zerotorobots]

here some more cool impressions from the #allaboutautomation that took place around 2 weeks ago in zurich. these machines are really advanced and expensive. quite the contrast to the current state of #physicalai

[Thomas Lummerstorfer | Physical AI @zerotorobots]

had to postpone this setup. it requires a way better gripper. visible in the video too. so i am starting with an experiment to place boxes in a bowl. once this works reliably the next setup is maybe a dinner serving robot fully mounted on the dinning table of the shared flat 🤣

[Thomas Lummerstorfer | Physical AI @zerotorobots]

I am teaching LeRobot to pickup a pen, draw a square and put the pen back. It’s difficult to work with the cheap hardware, but its still absolutely amazing what you get for just a couple hundred dollars. This was the first episode

[Thomas Lummerstorfer | Physical AI @zerotorobots]

training the new more robust policy with error recovery takes so long in google colab. its literally running for a day now on an nvidia tesla t4, because it crashed a few times and progress was lost