Can GPT-4 teach a robot hand to do pen spinning tricks better than you do?
I'm excited to announce Eureka, an open-ended agent that designs reward functions for robot dexterity at super-human level. It’s like Voyager in the space of a physics simulator API!
Eureka bridges the gap between high-level reasoning (coding) and low-level motor control. It is a “hybrid-gradient architecture”: a black box, inference-only LLM instructs a white box, learnable neural network. The outer loop runs GPT-4 to refine the reward function (gradient-free), while the inner loop runs reinforcement learning to train a robot controller (gradient-based).
We are able to scale up Eureka thanks to IsaacGym, a GPU-accelerated physics simulator that speeds up reality by 1000x. On a benchmark suite of 29 tasks across 10 robots, Eureka rewards outperform expert human-written ones on 83% of the tasks by 52% improvement margin on average. We are surprised that Eureka is able to learn pen spinning tricks, which are very difficult even for CGI artists to animate frame by frame!
Eureka also enables a new form of in-context RLHF, which is able to incorporate a human operator’s feedback in natural language to steer and align the reward functions. It can serve as a powerful co-pilot for robot engineers to design sophisticated motor behaviors.
As usual, we open-source everything! Welcome you all to check out our video gallery and try the codebase today: https://t.co/BHiNmqPoWE
Paper: https://t.co/bdh9TYQtHm
Code: https://t.co/lqKiaM2yYJ
Deep dive with me: 🧵
Unity quietly released a new hands package that introduces a hands subsystem enabling native hand tracking via OpenXR in vanilla Unity. It also fully supports Meta’s aim extension that adds far rays and pinch.
The sample hand mesh is getting fixed soon and xri samples are coming
I've been noodling with ReadyPlayerMe avatars in StereoKit, and sometimes coding is truly delightful :D I'm... guessing I nailed the head position part!
Welcome to how I normally see the world I guess?
Excited to announce the first update to the MRTK3 public preview. We're here to make MRTK better, faster, and cleaner, and chew bubble gum. And we're all out of bubble gum. Come join us! #mrtk#mrtk3
https://t.co/R74FjZQHve
Wait...switching from testing my Unity app on a HoloLens 2 vs. a Quest 2 is now as easy as just plugging in a different headset and hitting play? Awesome! https://t.co/cLdqkrcNAu I used to get lost trying to remember all the config changes...
#mrtk3#openxr#mrdevdays
Welcome to the world MRTK3, & welcome to Twitter MRTK PM Grace Hsu!
Action items:
1. Download MRTK3 (linked here👇)
2. Follow Grace @its_graceh
3. Check out more MR Dev Days 🧵
https://t.co/FGHFb4lFm9
Forth Dimension Displays and VividQ Collaborate on Holographic 3D Mixed Reality Headset that offers true depth perception. The article provides a good overview of why this is relevant. https://t.co/BtJutRCUD3
Edge compute - This Qualcomm blog post is the best description I've seen yet of why and how edge cloud/compute + 5G will revolutionize the miniaturization of XR devices. At first, these edge compute devices will be deployed in bus…https://t.co/RdG39dOXOc https://t.co/H2rxI785TV
My new novel, FALL, OR DODGE IN HELL, goes on sale in June 2019 in the USA. Info on tour & signed copies to follow. In the meantime, here’s the link to pre-order and learn more. https://t.co/GxDkujL68h