Daphne Cornelisse

(Multi-agent) simulation and RL • Ph.D. student NYU • Prev @Waymo @NVIDIAAI.

[Daphne Cornelisse @daphnesolves]

New Paper: Human-like Autonomy Emerges from Self-Play and a Pinch of Human Data.

We trained self-play RL on 60 years of simulation on 1 GPU in ~15 hours. Regularizing with 30 minutes of demonstration data produces much more human-like driving policies!

[Daphne Cornelisse @daphnesolves]

@yieldthought I see, some of them are (unfortunately) paywalled. It is annoying for sure; I just go to the next one.

[Daphne Cornelisse @daphnesolves]

Stop scrolling on X and read something wonderful today https://t.co/slQS5EeP9k 🙂

[Daphne Cornelisse @daphnesolves]

@yieldthought It is not? it should give you a random essay

[Daphne Cornelisse @daphnesolves]

Gotcha. I would be curious to understand how many additional rewards and how much domain randomization is needed to make regularized self-play RL work for a planner in the wild.

The regularizer should nudge the behavior towards conventions that are implicit in good human driving and can be hard to specify. For that reason, it should make the training easier.

[Daphne Cornelisse @daphnesolves]

@yassineyousfi_ @BrantonDeMoss @___Harald___ Why not?

[Daphne Cornelisse @daphnesolves]

@BrantonDeMoss @___Harald___ Simulator is https://t.co/MXJnv2Asx8 built on the pufferlib infrastructure https://t.co/9T7g3QHmND

[Daphne Cornelisse @daphnesolves]

@BrantonDeMoss @___Harald___ Thanks for the tag @BrantonDeMoss!

@___Harald___ We have a series of papers on this topic. Most recently https://t.co/ndoXaga9Rr. All code is open-source

[David Pfau @pfau]

Making it possible to do things that were previously impossible is infinitely more exciting than speeding up things that were already possible.

[Daphne Cornelisse @daphnesolves]

@ArdaGoreci Congrats, Arda and team! Cool to see this out.

[Daphne Cornelisse @daphnesolves]

Working on biology is humbling. It gives me a similar feeling to looking up at the stars at night and imagining how tiny the Earth is in relation to the rest of the universe.

[Daphne Cornelisse @daphnesolves]

@RaymondRChua @ZuckermanBrain Congrats!! We should grab a coffee :)

[Itai Yanai @ItaiYanai]

Using AI to write for you - start to finish - is really just you robbing yourself of one of life’s greatest joys: to think through a problem and figure out what you think about it, and in the process becoming passionate enough to take on the new challenge of solving it.

[Daphne Cornelisse @daphnesolves]

@tw_killian @BYUCS @BYU Cool!

[Kyle🤖🚀🦭 @KyleMorgenstein]

Octi/Nvidia + Abhishek's lab on a generational run. dex manip will be solved with sim.

[Daphne Cornelisse @daphnesolves]

@elliotarledge The network encoder is the bottleneck. This project was done with PufferDrive 2.0. It runs at ~300-400k SPS on a 4090 or 5090. IIRC, using one of the standard PufferLib MLPs gets you 1-1.5 M SPS, but the entity encoder is more performant.

Starting from PufferLib 5.0 + spending a little time optimizing the regularizer implementation should get you to ~800k or higher SPS, but the network is still a bottleneck (overview below).

[Daphne Cornelisse @daphnesolves]

New Paper: Human-like Autonomy Emerges from Self-Play and a Pinch of Human Data.

We trained self-play RL on 60 years of simulation on 1 GPU in ~15 hours. Regularizing with 30 minutes of demonstration data produces much more human-like driving policies!

[Daphne Cornelisse @daphnesolves]

@davidstutz92 @Nature @ImanisMind Congrats! Cool and important work.

[Samuel Sokota @ssokota]

In our Nature paper, we introduce the first superhuman Stratego AI, which we built using general techniques that we developed for RL & test-time compute under imperfect information. 1/N

[Daphne Cornelisse @daphnesolves]

Depends entirely on your interests and goals, of course!

One potentially interesting direction in Craftax could be to iteratively remove feedback and see whether tabula-rasa RL/Pre-trained LLMs can still solve it. E.g., can Astra still beat the final boss without any of the intermediate achievements? Another direction is that the models are typically large and expensive to use. Could you compress their knowledge about the world in some way? Third, I think we can still improve on-policy RL algorithms further.

Dwarf Fortress is an interesting suggestion, though it seems challenging to evaluate models given the open-ended nature of the game.

[Daphne Cornelisse @daphnesolves]

I really like the work you do, but it would be great if you could share more information about the outcome of the Galleri test. Currently, it only reports true/false on the app. This binary outcome is not very telling, especially given the high false negative rates of the Galleri test.