Austin Patel

Robotics PhD student @Stanford advised by Shuran Song

[Austin Patel @austinapatel]

LLMs learn new tasks in-context. It’s time robots do the same 🤖

Introducing Behavior Prompting: human shows one demo, and the robot adapts immediately. Turns out robot demos are great in-context prompts!

And the magic: no paired human2robot data needed

https://t.co/TNikpJT61S (1/8)

[Huy Ha @haqhuy]

Last week, I defended my PhD dissertation, "Robots From Anyone: Shaping Behaviors and Embodiments with Handheld Grippers". You can find the talk here!

https://t.co/mTxhvBY4Lo

[Austin Patel @austinapatel]

Behavior prompting is a new way to control your robot which we broadly hope:
1) provides an interface for humans to specify preferences to the robot via demonstration and
2) simplifies adaptation to new tasks/environments by providing a single demo in the target environment.

Shoutout to my amazing co-authors: @ben_pekarek, Joel Enrique Castro Hernandez and @SongShuran

Check out Behavior Prompting Policy:
📄 Paper: https://t.co/kZXJzZlJF5
🌐 Website: https://t.co/PKH9q6Zkv6
💻 BPP Code: https://t.co/Dt8IQwblMS
💻 iPhUMI Code: https://t.co/HYifek0RmS
(8/8)

[Austin Patel @austinapatel]

LLMs learn new tasks in-context. It’s time robots do the same 🤖

Introducing Behavior Prompting: human shows one demo, and the robot adapts immediately. Turns out robot demos are great in-context prompts!

And the magic: no paired human2robot data needed

https://t.co/TNikpJT61S (1/8)

[Austin Patel @austinapatel]

We are also excited to release iPhUMI ("eye-foo-me”)! It solves the localization challenges with the GoPro UMI, enabling rapid data collection across diverse environments and tasks. During deployment, iPhUMI lets you command your robot via demonstration.

https://t.co/HYifek0RmS (7/8)

[Jaden Clark @jadenvclark]

Can we enable robots to develop a sense of touch without forgetting what they learned from large-scale vision-only pretraining?

Introducing MultiSensory World Model (MuSe) 🌍: A new approach for finetuning visuomotor policies on minimal data from new sensor modalities, such as force/torque (F/T)

With Muse, touch learned later improves skills learned earlier — a small amount of F/T data on new tasks improves zero-shot on diverse pretraining tasks that were never supervised with F/T

We believe MuSe provides a practical pathway towards training multisensory foundation models that leverage both abundant vision data, and smaller multisensory datasets 🧵👇

[Haoyu Xiong @Haoyu_Xiong_]

Your bimanual manipulators might need a Robot Neck 🤖🦒

Introducing Vision in Action: Learning Active Perception from Human Demonstrations

ViA learns task-specific, active perceptual strategies—such as searching, tracking, and focusing—directly from human demos, enabling robust visuomotor policies under visual occlusions. 🧵👇

[Neil Nie @neil_nie_]

Thrilled to share our CoRL 2024 paper on learning from demonstrations for long-horizon manipulation! Check out real-world demos here: https://t.co/7QT5yUNvoi. Weiyu will be presenting BLADE today!

Very grateful to Weiyu for being an amazing mentor and co-lead on this journey!

[Shuran Song @SongShuran]

We recently launched https://t.co/mshIJSnIYu as a community-driven effort to pool UMI-related data together. 🦾

If you are using a UMI-like system, please consider adding your data here. 🤩🤝

No dataset is too small; small data WILL add up!📈

[Austin Patel @austinapatel]

Without any finetuning, GET-Zero can zero-shot control a wide variety of hand designs, even if we remove joints or add link length extensions (shown in orange). None of the hand designs below were seen during training.

Check out our project page: https://t.co/GJanm1X9MP

[Austin Patel @austinapatel]

What if you could control new hand designs without a new policy?

Introducing GET-Zero, an embodiment-aware policy that can zero-shot control a wide range of hand designs with a single set of network weights.

https://t.co/JzIwlKuxC0

[Austin Patel @austinapatel]

How can our model adapt to different embodiments? Graph Embodiment Transformer (GET) is an embodiment-aware transformer encoder that combines joint hardware properties with an attention-based embodiment graph encoding to flexibly represent a wide range of hand designs.