Pierre Sermanet

Co-Founder & Chief Scientist at UMA

[Mustafa Shukor @MustafaShukor1]

We’re growing our team @UMA_Robots and building the infrastructure to take our in-house model to the next level.

If you have experience in large-scale training on massive multimodal data, please apply https://t.co/3B8aFCbpBm.

(hiring RS/REs to work on Training/Data)

[Remi Cadene @RemiCadene]

Our team scaled @Tesla Autopilot and Optimus, VL-JEPA at FAIR, RL post-training on Gemini, RL for locomotion.

Come work with us to push the frontier of AI in robotics!
Open roles in 🧵 -> Infra | Scientist | Engineer

[Pierre Sermanet @psermanet]

“Decommissioning” our @boosterobotics robot with a front flip off a table for fun. Made end-to-end in half a day.

[Louis Le Lay ✈️ CoRL @LouisLeLay4]

At UMA, they flip and live 😁

[Louis Le Lay ✈️ CoRL @LouisLeLay4]

At UMA, they flip and live 😁

[Remi Cadene @RemiCadene]

At @UMA_Robots , we are convinced that latent-space world models will unlock scaling for robotics

1 position left for the roles

🧪 Research Scientist:
https://t.co/9XYHAzKDyt
🚀 Research Engineer:
https://t.co/27BTppPQto

[Pierre Sermanet @psermanet]

We didn't investigate the minimum floor for language, it's still a useful conditioning for high-level control but when it comes to the understanding the physical world, language is a weak signal to shape learned representations of the world, it's good for high-level representations.

[Pierre Sermanet @psermanet]

The SPRIND grant will accelerate UMA’s effort to develop latent world models for humanoid robots.

Throughout my career I’ve kept the objective of scaling up learning using minimal supervision and efficient data collection. This led me to propose self-supervision methods in latent space such as Time-Contrastive Networks, which demonstrated 3rd person imitation of a human entirely without labels: https://t.co/4uTO1b27ST

One key component was co-training different modalities, viewpoints and embodiments into a single latent space, entirely from raw unlabeled videos. The dimensions of the world discovered by the models ended up naturally aligning in latent space. This latent alignment across viewpoints and embodiments is what unlocked imitating humans directly from 3rd-person raw pixels, similarly to the concept of “mirror neurons” firing when observing someone else performing a task.

We later developed self-supervised methods with @coreylynch for generating actions by pulling together in latent space the representations of (start, goal) image pairs and sequence of images [start, …, goal]. We combined this label-free learning method with an efficient data collection method: playing.

Play is an efficient way to discover the world by leveraging existing knowledge and skills. Humans and animals use it to learn about the world and practice in advance. We called this Learning Latent Plans from Play: https://t.co/sg80FkyI7v

We subsequently augmented this approach with language representations so that you could control the latent space via language conditioning. But most of the learning was coming from self-supervising on video, language labels accounted for less than 1% of data: https://t.co/UKki1f9TTg

I still believe that learning the world in latent space by mostly self-supervising on raw unlabeled data is the way to go. We now have stronger unsupervised learning methods, more capable hardware and better data acquisition means, it is an exciting time to keep pushing that vision.

If that vision speaks to you, we are hiring for our Embodied World Models team:

Research Scientist: https://t.co/3Tq2PqMBBq

Research Engineer: https://t.co/PEVNvqaPsN

[Remi Cadene @RemiCadene]

Join our team to work on Embodied World-Models with @MustafaShukor1 (co-author VL-JEPA with @ylecun )

Link to apply in comments 😇

[Pierre Sermanet @psermanet]

At UMA, we build and own the full stack, from hardware to software.
This allows us to bake safety in at all levels, rather than bolt it on as an afterthought.
Because trust is everything when robots share our world.
Reach out if you want to be part of this journey.

[Mustafa Shukor @MustafaShukor1]

Excited by what we’ve achieved in the last 9 months.

We went from a model that could merely pick objects to one that can run autonomously for hours on challenging, high-precision tasks.

[Pierre Sermanet @psermanet]

In house embodied AI designed for robust and continuous operation.

[Remi Cadene @RemiCadene]

AI developed at @UMA_Robots

Picking and Scanning items
Deformable, Slippery, Thin, Fragile
We mess with it at the end :)

[Remi Cadene @RemiCadene]

Starting with the fundamentals

Prototype Version 0
AI, Software, Hardware
A small team, 9 months
Designed and assembled in Paris at @UMA_Robots

[Pierre Sermanet @psermanet]

At UMA we're pushing the boundaries of embodied AI with our own humanoid robot.

[Remi Cadene @RemiCadene]

Unveiling our Northstar design @UMA_Robots

A body that can navigate our space
Hands that can use our tools
Approachable, competent, calm
A robot you feel at ease having at your workplace and home

[Gabriel Dulac-Arnold @gabepsilon]

Tl;dr After 10+ years pushing the boundaries of RL at @GoogleDeepMind , @Google Brain and then Gemini, I’m looping back to startup mode and joining @UMA_Robots to push the frontier of human-centric robotics. (1/4)🧵

[Pierre Sermanet @psermanet]

@gabepsilon @GoogleDeepMind @Google @UMA_Robots Amazing to have you on board @gabepsilon !

[Pierre Sermanet @psermanet]

@AractingiMichel @UMA_Robots Welcome to the team @AractingiMichel !

[Remi Cadene @RemiCadene]

One of the first AgiBot X2 humanoid robots just landed in our lab 🤖
I like its articulated neck, back handle, battery location, anthropomorphic wrist and white soft cover in a sort of Polyurethane foam.
Impressive delivery speed and support from the @AGIBOTofficial team!

[Pierre Sermanet @psermanet]

@WilliamVerst @teslasi @UMA_Robots Amazing to have you onboard @WilliamVerst !

[William Verstraete @WilliamVerst]

I recently moved from @teslasi to @UMA_Robots.

🧵(1/5)

[Pierre Sermanet @psermanet]

@RemiCadene @UMA_Robots @AdilZtn @alibert_s @Mickael_Chen @LeoTronchon @MustafaShukor1 @therobotstudio @marine_caous Lua was a cool language ❤️

[Pierre Sermanet @psermanet]

@r0b0t_sp1der @RemiCadene @alibert_s @therobotstudio @Mickael_Chen @ylecun @AdilZtn @LeoTronchon you can apply at https://t.co/v6hJMAr062 ! :)

[Pierre Sermanet @psermanet]

I’m really happy to share that we’re launching UMA.

Together with @RemiCadene, @alibert_s, @therobotstudio, and an exceptional founding team, we’re building general-purpose mobile and humanoid robots. If you want to be part of this adventure, reach out at https://t.co/X7IMRZtHfB

Throughout my career, I have been obsessed with scalable learning and data acquisition methods that require little to no labels.

Back in 2005 with @ylecun, we were self-supervising our “deep” 2-layer network to do long range vision using short range stereo information, this was running live onboard our robot. However, because our deep model was so slow, the robot would crash constantly, so I designed a decoupled fast & far architecture for robust navigation, allowing fast control to coexist with slow long horizon thinking, much like systems 1 & 2 in modern humanoids.

My PhD was focused on making deep learning work for computer vision, including unsupervised feature learning with @koraykv, writing and open-sourcing a C++ deep learning library with @soumithchintala, and open-sourcing one of the first deep learning vision systems.

I came back towards robotics at @Google Brain and @GoogleDeepMind, where I pushed for entirely label-free methods on real robots. In 2017, @coreylynch and I managed to make our robot imitate human motion by co-training self-supervision across sim and real domains jointly, without any labels. With @imkelvinxu and @svlevine , we showed that unsupervised visual reward learning could be used for RL in the real world.

In 2020, Corey and I developed the first manipulation VLA, which was trained with very few language labels thanks to self-supervision on play data (playing is an efficient way to demonstrate and practice a broad set of skills and is essential for human development).

I was never satisfied with the status quo of top-down data collection, where researchers decide a few tasks to collect data on. Instead, I believed that we should let the data speak: tasks should be automatically discovered bottom-up (scalable and general) from cheap and continuous data collection, with a sprinkle of more expensive data and labels.

In 2022, I explored long-horizon reasoning for robotics using scalable automatic labeling augmentations for VQA tasks and studied the economics of different data collection schemes.

Most recently, I developed approaches to scalably discover laws of robotics from real data (images, hospital reports, sci-fi literature) in a broad and bottom-up fashion, which improved robot behavior over top-down approaches like Asimov’s laws.

All these experiences nourished my vision for UMA as Chief Scientist, I’m incredibly excited to put everything together and so grateful I get to contribute to this incredible moment in human history.

Picture: Yann supporting UMA as an advisor and investor, with the team in Paris a couple weeks ago.