Can GPT-4 teach a robot hand to do pen spinning tricks better than you do?
I'm excited to announce Eureka, an open-ended agent that designs reward functions for robot dexterity at super-human level. It’s like Voyager in the space of a physics simulator API!
Eureka bridges the gap between high-level reasoning (coding) and low-level motor control. It is a “hybrid-gradient architecture”: a black box, inference-only LLM instructs a white box, learnable neural network. The outer loop runs GPT-4 to refine the reward function (gradient-free), while the inner loop runs reinforcement learning to train a robot controller (gradient-based).
We are able to scale up Eureka thanks to IsaacGym, a GPU-accelerated physics simulator that speeds up reality by 1000x. On a benchmark suite of 29 tasks across 10 robots, Eureka rewards outperform expert human-written ones on 83% of the tasks by 52% improvement margin on average. We are surprised that Eureka is able to learn pen spinning tricks, which are very difficult even for CGI artists to animate frame by frame!
Eureka also enables a new form of in-context RLHF, which is able to incorporate a human operator’s feedback in natural language to steer and align the reward functions. It can serve as a powerful co-pilot for robot engineers to design sophisticated motor behaviors.
As usual, we open-source everything! Welcome you all to check out our video gallery and try the codebase today: https://t.co/BHiNmqPoWE
Paper: https://t.co/bdh9TYQtHm
Code: https://t.co/lqKiaM2yYJ
Deep dive with me: 🧵
Three findings from a trial use of a GPT-4 tutor for Harvard's intro CS course:
1) Students found it useful (88% found it helpful) & used it a lot
2) The AI made mistakes, even with RAG (81% accurate)
3) It was pretty cheap
Lots of potential but R&D needed https://t.co/3Q13zvkhFf
I had rough sketch of glass design and exchanged conversations with ChatGPT4. I like how it turned out in the end. While it is hard to make with my current skill, the design helped me see more.
#generativeAI#art