Why am I spending all this time training a robot arm to move colorful rubber ducks?
Fair question.
On the surface, this is absolutely a silly project.
I’m literally collecting demos, training AI policies, debugging, and spending way too many hours trying to get a small robot arm to pick up ducks and place them on a target.
And part of the answer is simple: It’s fun.
But that’s not the whole reason.
I’ve noticed a huge gap in education and excitement about AI and robotics in the X tech community vs the frequent doomerism on mainstream social media.
And I want to counter that narrative.
So what is this project?
It is an interactive livestream where the chat can control a robot.
Why a livestream?
Mainstream short-form social media platforms like TikTok and YouTube Shorts have been massively amplifying live content.
And in a world of brainrot and slop, live content is one of the best ways to engage an audience in order to share the message of techno-optimism.
How does it work?
The idea is simple, you send a command like:
“put the green duck on the target”
The system then parses the command, checks the current state of the workspace, picks the right robot policy, and the arm executes it live.
It works by combining:
• An SO-101 3D printed robotic arm
• A top-down camera
• Hugging Face LeRobot imitation learning
• 8 ACT policies: 1 to place or remove each duck color
• An LLM to parse commands
• Multimodal AI to check the workspace state
• Text-to-speech audio feedback so the robot can explain what it’s doing
In the end, colored rubber ducks are the hook.
The bigger goal is to use fun, interactive robotics content as a gateway into tech education and inspiration.
People are not just watching the robot.
They are participating.
They are giving it commands.
They are seeing embodied intelligence interact with the real world.
My hope is that projects like this can make AI and robotics feel less scary and more approachable.
I’m not a robotics researcher, just an at-home builder that asks ChatGPT a lot of questions.
And if this colored duck robot is the trojan horse that gets someone to care about robotics,
That's good enough for me.
I’ve been playing with the number of demo episodes and training steps for ACT and I think I finally found the sweet spot for my SO-101.
I got decent performance with 50 episodes and 100k training steps, however operation was certainly not smooth.
But when I increased to around 80-90 episodes and 200k training steps, it eliminated a lot of the jitters and erratic movements for much smoother operation.
I’ve got each color of duck pick or place running AI models with various training params, so I’m excited to experiment on stream
The tweet-controlled livestream where you can tell my robot what to do should be live in less than a week now!
@ptruiz_dev Oof yeah I wish I had one.
I’m still slumming it without one, but one of these days it’s gonna be faster for me to buy a 3d printer than get a replacement part shipped. And that will be the last straw.
I put Plasti Dip on the gripper fingers of my robot arm a few weeks back.
I was curious how well it would hold up,
and honestly it works great!
I did 3 coats total, but I could have maybe done a 4th
Curious what other modifications people have made to their SO-101 grippers?
@cyrux004 Thanks! I really want to do this actually, but I don't have a good enough GPU locally.
However, I want to try this using @interlatent soon!
They provide infrastructure for remote inference that leverages optimized networking to mitigate issues caused by latency.