Today may be the ImageNet moment for robotics.
RT-X: the largest open-source robot dataset ever compiled, across 33 institutes, 22 robot hardware, 527 skills, and 1M episodes.
Why is robotics lagging so far behind NLP, vision, and other AI domains? Data scarcity is the main culprit to blame, among other difficulties. Unlike text, images, and videos, you cannot download mass amounts of onboard robot control data from the internet. They simply don't exist in the wild.
11 yrs ago, ImageNet kicked off the deep learning revolution. 3-4 yrs ago, internet-scale data fueled the first GPTs and Diffusions that define this era of foundation models. I think 2023 is finally the year for robotics to scale up.
Robot foundation models like VIMA (https://t.co/Tn3L63uGrv, my team's work at NVIDIA) and RT-1/2 (https://t.co/voAGkpPTWa, Google DeepMind's effort) are extremely data hungry. While massively parallel simulations like NVIDIA IsaacGym & Omniverse can alleviate the problem to some extent, it's still not quite enough to bridge the gap to the messy, physical world.
This new dataset is not just a technical contribution. I also see it as a commendable effort to overcome institutional bureaucracies and unite researchers from around the world to tackle a grand challenge together.
Robotics will be the final holy grail that we capture in AI. We are not there yet, but ascending in the right gradient direction.
RT-X website: https://t.co/KG6B53EgDP
Launch blog: https://t.co/ItVfPr9Fnk
How do I get started with coding after settling on so many ideas in the ideation phase when building @elser_ai ?
Current approach:
We split the work between the frontend and backend. The backend team starts with setting up databases, cloud services, and other foundations.
Meanwhile, the frontend team designs the UI based on Figma sketches. We then bring both parts together. Generally, it works okay, but integrating both ends can be quite a challenge. I spent over a week just diving into the frontend (and that was a slow week for me π ).
Shape Up approach:
We emphasize the UI/UX design phase, extracting one or two core features from the ideas. Both frontend and backend should be able to implement these core features, so when integrated, they're functional.
Following this, we decided to implement an "Elser" chat interface. In this chat, users input messages, and the backend directly sends these to ChatGPT.
Merging both frontend and backend gives everyone a tangible feel of the progress, boosting team morale π and eliminating uncertainties.
Now, weβre not just holding onto an idea, but have a basic prototype to work with! π₯οΈπ€
#Elser