Ruben D'Sa

Product for AI/Simulation frameworks @NVIDIA prev. @openai @1x @talos @kogilabs

[Ruben D'Sa @ruben_dsa]

We just launched Isaac Lab 3.0! I wanted to take a minute to share what this one means to me, because the story goes back a while.

Back in 2015, during my PhD, I happened to stumble across the DQN paper in Nature. I was familiar with the ML space at the time, but it was the first time I really had any visibility on RL. I came from a hardcore electro-mechanical and controls background, and I remember reading it and thinking, this is some of the coolest work I’d ever seen. It completely shifted what my research goals looked like. I still did a lot of hardware design, and I led a team to design some crazy shape-shifting UAVs, but that’s around the time I started obsessing over simulation and RL.

That change in headspace led me to join @OpenAI's robotics team in 2019 and the Rubik’s Cube project, which was all about scaling sim-to-real with RL on dexterous manipulation tasks. We had a huge CPU cluster running MuJoCo-C for rollouts, plus a very large V100 cluster for async rendering. It was a really sophisticated setup, and it was the first time I witnessed the power of scaling RL.

During this time, I vividly remember a backyard barbecue at @woj_zaremba’s house with @npew. He showed me his phone and said, “Hey, our friends at @nvidia have a new simulator, and they said they can train entire Rubik’s Cube policies on a single GPU.” I didn’t know it at the time, but what I was looking at was an early build of Isaac Gym.

Little did I know that years later, through a very serendipitous set of events, I’d end up working with some of the same people who built it. @gavrielstate, @kelly_guo, @milesmacklin and so many of the people behind that OG simulator everyone loved are the team behind Isaac Lab 3.0. And for the last year, I’ve had the privilege of leading product for Isaac Lab alongside them.

Lab 3.0 is a huge, huge achievement for the team and for robotics at @NVIDIARobotics. It unlocks a whole bunch of new capabilities, it’s lightweight, and we really built it to be something people love. We’re already getting strong support from customers, and it’s gaining a bunch of positive traction. I’m incredibly proud of it.

I truly believe simulation can unlock incredible robot capabilities, and I’m so excited to keep building simulation products people love. I hope you all enjoy this release. There’s a lot more coming, and we’re really thankful for your support.

[NVIDIA Robotics @NVIDIARobotics]

We’re excited to announce the official release of NVIDIA Isaac Lab 3.0. 🎉

Isaac Lab 3.0 brings a new modular and lightweight architecture along with major new capabilities across the robot learning workflow: multi-physics and multi-rendering support, training at scale, and streamlined deployment on real hardware.

With Newton integration, the new OVRTX renderer, RL post-training, and LEAPP deployment, there’s a lot to explore.

Here’s a look at what’s new 🧵

(1/7)

[Lucky Iyinbor @Luckyballa]

A lot of dexterous manipulation tasks that require significant force rely on objects being at least slightly deformable

Makes me wonder where the optimal tradeoff between simulation accuracy and policy training speed will land at the end of the day

[Stone Tao @Stone_Tao]

there has never been a better time to learn robotics simulation. Lots of free tools and datasets out there to get started!

[Gavriel State @gavrielstate]

After we released the original Isaac Gym and then the Orbit framework, I took a step back from robotics to focus on 3D reconstruction and rendering - NeRFs and 3D Gaussians.

Then, in October 2024, I got a call from @JensenHuang , asking me to come back and take charge of Isaac Lab.

I was really enjoying being deep in the weeds working on technologies that were defining the future of 3D graphics. I also understood exactly what it would mean to take this on, and the level of commitment it would require.

So, I hesitated.

“It’s very flattering that you think I’m the right person for this, but I’ve been away from robotics for a couple of years, and things move fast. I don’t know our current strategy, or even who’s on the team now.”

Jensen’s response was simple, direct, and effective: “You decide.”

With that level of trust, the opportunity was hard to pass up.

Over the past two years, I’ve had the privilege of working with an extraordinary group of collaborators to define the future of robotics simulation with Newton and Isaac Lab 3.0.

I couldn’t be more proud of our incredible, hardworking team and what we’ve been able to accomplish together. Seeing everything come to fruition at GTC last week was amazing.

Thanks, Jensen!

[Ruben D'Sa @ruben_dsa]

This week was unreal. In addition to the Isaac Lab 3.0 launch, I also led the @NVIDIARobotics side of the Microducks demo in Jenson's Keynote. Isaac, Omniverse, RTX Spark, and Creative teams basically worked around the clock to put together the script, tech, and live action. We made almost 4 complete rewrites in a very short period of time and generated hundreds of RL policies for various stages for the script. Working with so many talented teams at such high velocity was an incredible privilege.

Fun fact, all the Microduck motions in the keynote presentation were built with RL policies in Isaac Lab 3.0 💚

[NVIDIA RTX Spark @NVIDIARTXSpark]

Big day. Little buddy.

[Ruben D'Sa @ruben_dsa]

@Nicolas_Keller Thanks so much Nicolas! btw wonderful catching up at GTCB, your work at @pokeandwiggle is so awesome! 💚

[Ruben D'Sa @ruben_dsa]

We just launched Isaac Lab 3.0! I wanted to take a minute to share what this one means to me, because the story goes back a while.

Back in 2015, during my PhD, I happened to stumble across the DQN paper in Nature. I was familiar with the ML space at the time, but it was the first time I really had any visibility on RL. I came from a hardcore electro-mechanical and controls background, and I remember reading it and thinking, this is some of the coolest work I’d ever seen. It completely shifted what my research goals looked like. I still did a lot of hardware design, and I led a team to design some crazy shape-shifting UAVs, but that’s around the time I started obsessing over simulation and RL.

That change in headspace led me to join @OpenAI's robotics team in 2019 and the Rubik’s Cube project, which was all about scaling sim-to-real with RL on dexterous manipulation tasks. We had a huge CPU cluster running MuJoCo-C for rollouts, plus a very large V100 cluster for async rendering. It was a really sophisticated setup, and it was the first time I witnessed the power of scaling RL.

During this time, I vividly remember a backyard barbecue at @woj_zaremba’s house with @npew. He showed me his phone and said, “Hey, our friends at @nvidia have a new simulator, and they said they can train entire Rubik’s Cube policies on a single GPU.” I didn’t know it at the time, but what I was looking at was an early build of Isaac Gym.

Little did I know that years later, through a very serendipitous set of events, I’d end up working with some of the same people who built it. @gavrielstate, @kelly_guo, @milesmacklin and so many of the people behind that OG simulator everyone loved are the team behind Isaac Lab 3.0. And for the last year, I’ve had the privilege of leading product for Isaac Lab alongside them.

Lab 3.0 is a huge, huge achievement for the team and for robotics at @NVIDIARobotics. It unlocks a whole bunch of new capabilities, it’s lightweight, and we really built it to be something people love. We’re already getting strong support from customers, and it’s gaining a bunch of positive traction. I’m incredibly proud of it.

I truly believe simulation can unlock incredible robot capabilities, and I’m so excited to keep building simulation products people love. I hope you all enjoy this release. There’s a lot more coming, and we’re really thankful for your support.

[NVIDIA Robotics @NVIDIARobotics]

We’re excited to announce the official release of NVIDIA Isaac Lab 3.0. 🎉

Isaac Lab 3.0 brings a new modular and lightweight architecture along with major new capabilities across the robot learning workflow: multi-physics and multi-rendering support, training at scale, and streamlined deployment on real hardware.

With Newton integration, the new OVRTX renderer, RL post-training, and LEAPP deployment, there’s a lot to explore.

Here’s a look at what’s new 🧵

(1/7)

[Ruben D'Sa @ruben_dsa]

@AikaanL @NVIDIARobotics ✍️Track your experiments ✍️

[Ruben D'Sa @ruben_dsa]

@kdpisda @NVIDIARobotics Exactly @kdpisda ! 💚

[Ruben D'Sa @ruben_dsa]

@_itsskofficial_ @NVIDIARobotics All you gotta do is paste and run `uvx --python 3.12 isaaclab demo h1-locomotion --viz newton_gl` in terminal 💚

[Ruben D'Sa @ruben_dsa]

@galaxper @NVIDIARobotics It really is 💚

[Ruben D'Sa @ruben_dsa]

This is a great insight @ChenChaim42853 ! It’s actually one of the main reasons for sim-sim transfer, a fast eval mechanism for testing policy overfitting to physics engine behavior.

With Lab 3, you get this out of the box just by changing a flag. See here: https://t.co/JW6DIMdnhr

[NVIDIA Robotics @NVIDIARobotics]

Isaac Lab 3.0 lets you choose the physics and rendering backends that best fit your robot learning workload, as well as different visualization approaches to see what your policy looks like while training.

For example, the SONIC whole-body controller from NVIDIA runs across different physics engines and transfers to real hardware, demonstrating policy robustness across simulation backends and the real world.

(3/7)

[Ruben D'Sa @ruben_dsa]

@hongfz16 This is really cool @hongfz16 !

[Ruben D'Sa @ruben_dsa]

@isaactevin @NVIDIARobotics I’m gonna start calling the team Isaacs 💚

[Ruben D'Sa @ruben_dsa]

@EternitiesAI @NVIDIARobotics This is the way 💚

[Ruben D'Sa @ruben_dsa]

@frame_aix You can likely handle the embodiment challenges at the policy architecture level (ex diffusion heads). Best part is Lab 3 makes it easy to iterate, and you can get signal from experiments fast.

[Ruben D'Sa @ruben_dsa]

@yangyang_2048 I wanna see a Lab env for this. Oreo’s and Panda bots 💚

[Ruben D'Sa @ruben_dsa]

@ariel__shaulov @gavrielstate We’re so excited, this is a good one 💚

[Ruben D'Sa @ruben_dsa]

@SediBY571 @NVIDIARobotics It really is @SediBY571 💚

[Nick Blauch @nmblauch]

I've been spending the last several months stress testing this software in beta for my research. It's so incredibly feature rich and flexible for all sorts of different needs and applications.

Critically, it is super scalable but also amenable to academic grade hardware; the more compute you have the better, but the different backends can support fast sim training that is ready to convert to high fidelity training whenever more compute becomes available. That means one workflow, different presets.

This is a huge unlock for democratizing sim2real robotics research, but also - and this is extremely underappreciated - for theoretical neuroscience. If we want to understand the brain, we need to simulate how perception, cognition, and action come together in dynamic environments. That's what training robot policies in sim is all about, and no headaches about hardware.

Somehow, we've built superintelligent AI far before we've understood it. Now, we can use AI to help us understand ourselves, from prototyping the task environment, to testing different embodiments, to implementing different theories of neural computation and different forms of experience.

Give it a shot and let us know how we can help. Isaac Lab is developed completely open source on GitHub.

[Ruben D'Sa @ruben_dsa]

@CryptideRL @NVIDIARobotics Lets send it @CryptideRL 🦾