Hanming Ye

@theWaddleLabs | math @ harvard | rsi '23

[Hanming Ye @DozenDucc]

Introducing Waddle Labs: Claude Code for robots.

Connect our API to your robot and enter a prompt, then our agents write code to achieve the task in 20 minutes.

@yiding_song @theWaddleLabs

[Hanming Ye @DozenDucc]

@myanvoos Curious what challenges you ran into? We prob faced the same things initially

[Hanming Ye @DozenDucc]

@Selen7005717917 @yiding_song @theWaddleLabs https://t.co/KEmKdT8Bai gives an overview. At the bottom it also links to related papers!

[Hanming Ye @DozenDucc]

Introducing Waddle Labs: Claude Code for robots.

Connect our API to your robot and enter a prompt, then our agents write code to achieve the task in 20 minutes.

@yiding_song @theWaddleLabs

[Hanming Ye @DozenDucc]

Sometimes code-as-policy works best (eg. repetitive tasks in factories), sometimes robot models work best (eg. walking a humanoid, controlling 5-fingered hand).

People over-index on models, but we still need em. What's surprising is that code-as-policy and LLM controlling robots is far more capable than expected. Ths should unlock faster deployments and automated data collection.

[Hanming Ye @DozenDucc]

@artofelection @yiding_song @theWaddleLabs What task do you think we should work on

[Hanming Ye @DozenDucc]

@hatebunnyplzzz @yiding_song @theWaddleLabs Cap-X is a foundational paper for code-as-policy! We didn't know "code-as-policy" was a thing in research when we started. But once we read about Cap-X it helped us design some of our primitives (see https://t.co/KEmKdT8Bai)

[Hanming Ye @DozenDucc]

@DJiafei Also using TopReward and Robometer to judge rollouts :)

[Hanming Ye @DozenDucc]

@DJiafei Yes! Models beget users beget data begets models

[Hanming Ye @DozenDucc]

@tomcocobrico @NickADobos Nah thanks Jeffrey you raise a very valid point (also its X).

We've been using code-as-policy to collect tons of data internally - let's see where this goes

[Hanming Ye @DozenDucc]

It sounds inelegant to burn a million tokens to fold a shirt. But isn't it cool that you can do this at all with a big next-token predictor?

[Hanming Ye @DozenDucc]

@NickADobos What's surprising is that code-as-policy and LLM controlling robots is far more capable than expected.

I expect this to unlock faster deployments and automated data collection.

[Hanming Ye @DozenDucc]

@NickADobos I don't think so.

There’s no one path to robot intelligence. Sometimes code-as-policy works best (eg. repetitive tasks in factories), sometimes robot models work best (eg. walking a humanoid, controlling 5-fingered hand).

People over-index on models, but we still need em.

[Hanming Ye @DozenDucc]

@Sujeetsoni123 @yiding_song @theWaddleLabs There's hacks to get these models to work better but defo much room for improvement

[Hanming Ye @DozenDucc]

@Sujeetsoni123 @yiding_song @theWaddleLabs LLMs definitely suck as physics. Maybe the root cause is that they suck at perception. We tried many models (every gemini, Nvidia Cosmos, you name it) and they all hallucinate an unbearable amount on videos.

[Hanming Ye @DozenDucc]

@BrutalCaeser @yiding_song @theWaddleLabs In all the demos in the video there were no robot models (eg. ACT, VLAs). This goes to show that code-as-policy works in diverse situations.

I believe that robot models will be crucial for dexterous tasks, it's just that they're not the only way towards physical intelligence.

[Hanming Ye @DozenDucc]

@BrainsAndTennis The voyager paper was a substantial inspiration for our skill library design!

[Hanming Ye @DozenDucc]

@Sujeetsoni123 @yiding_song @theWaddleLabs I thought about duplex models as a way to improve accuracy actually. Because turn-based models miss changes in the environment unless they call perception tools intentionally. So full-duplex would make it easier to catch things like wind blowing over a water bottle.

[Hanming Ye @DozenDucc]

@Sujeetsoni123 @yiding_song @theWaddleLabs A big help would be better full duplex vision models (like what Thinking Machines or OAI have but for perception in particular) - lmk if you have ideas on this!

[Hanming Ye @DozenDucc]

@Evose_AI @yiding_song @theWaddleLabs Good question - a lotta our work has gone into this. One thing we are working on: can we use sim / world models to evaluate programs before they run irl?

[Hanming Ye @DozenDucc]

@Ken_Goldberg Many robot companies can make cool demos but we need standardised benchmarks to tell which robots are useful for society vs which just look cool

[Hanming Ye @DozenDucc]

@xennygrimmato_ @Figure_robot To be fair figure’s robots look hella sci fi. If they did a demo with an army of humanoids marching we wouldn’t be able to replicate that