Michel Ma

PhD student @Mila_Quebec

[Alexandre @alexpiche_]

Introducing ReSearch: An iterative self-reflection algorithm that enhances LLM's self-restraint abilities:

• Encouraging abstention when uncertain
• Producing accurate, informative content when confident

Result: Significant accuracy boost for Llama2 7B Chat and Mistral 7B! 🚀

[Pierluca D'Oro @proceduralia]

Is backprop through transformer world models the future of AI agents?

We're going to present this as a contributed talk tomorrow morning at the ICLR workshop on generative models for decision making. Come to hear about it!

[Mo Samsami @M_R_Samsami]

🚀 Thrilled to introduce Recall to Imagine (R2I), the 1st model-based RL approach integrating SSMs to excel in memory-intensive domains. Not just setting new SOTA, but achieving superhuman results in complex memory tasks, while efficiently operating across diverse domains.
1/

[Pierluca D'Oro @proceduralia]

Do transformer world models give better policy gradients?

In our new paper, co-led with @michel_ma_, we answer this question: traditional transformer world models conditioned on the full history do not give better policy gradients, but transformers conditioned only on actions do!

If you want to make the world differentiable (as @SchmidhuberAI suggested in the 90s) and do reinforcement learning by backpropagation through time, then using the right architecture for your world model is what matters!

Transformers propagate gradients efficiently over long horizons due to short gradient paths, but we show (in theory and in practice) that this property doesn’t necessarily translate into better differentiable world models when this translation is done naively.

We propose to use world models working on sequences of actions, constraining transformers to create their internal models of reality, with no autoregressive unrolling. We call these Actions World Models.

Transformer Actions World Models not only outperform history-based and one-step world models in realistic domains, but they can also be better than the real differentiable simulator.

Curious to know why? Learn more in the thread 🧵

[Tianwei Ni @twni2016]

Curious about self-supervised learning in reinforcement learning, but unsure where to start?
Our recent ICLR paper connects various methods through a *self-predictive* condition (called ZP) and sheds light on learning self-predictive representations. 🧵⬇️: #ICLR2024 #RL #SSL

[Pierre-Luc Bacon @pierrelux]

Congrats to Mahan, who is finishing his Master's thesis in beauty with this second paper.

[Mahan Fathi @MahanFathi]

Course Correcting Koopman Representations
Accepted at #ICLR2024!

We identify problems with unrolling in imagination and propose an unconventional, simple, yet effective solution: periodically "𝒓𝒆𝒆𝒏𝒄𝒐𝒅𝒊𝒏𝒈" the latent.

📄 https://t.co/ULNzqAV3bB
@GoogleDeepMind

1/🧵

[Pierluca D'Oro @proceduralia]

Our work showing how reinforcement learning from an LLM's feedback can be used to train AI agents has been accepted at ICLR!

See you in Vienna!

[Tianwei Ni @twni2016]

Greetings DRL enthusiasts! Join us for our oral presentation at 3:20 pm in Ballroom A-C (Level 2), followed by our poster session #1425 from 5 to 7 pm. See you there!

[Tianwei Ni @twni2016]

I'll be at #NeurIPS2023 next week! Let's connect and say hi! Presenting two papers:

"When Do Transformers Shine in RL?" at the main conf (oral; Dec 14 PM) & FMDM workshop (Dec 15).

A new work on understanding self-predictive RL in POMDPs at the SSL workshop (oral; Dec 16).

[Pierluca D'Oro @proceduralia]

I'm launching my new blog with a short post: "On building the modern science of AI agents".

Applying LLMs to decision-making calls for the emergence of a set of new questions, and of a new science.

Let’s build together the modern science of AI agents!

https://t.co/vB3V1XyQz3

[Pierluca D'Oro @proceduralia]

Can reinforcement learning from AI feedback unlock new capabilities in AI agents?

Introducing Motif, an LLM-powered method for intrinsic motivation from AI feedback. Motif extracts reward functions from Llama 2's preferences and uses them to train agents with reinforcement learning.

On the complex NetHack game, Motif solves previously unsolved tasks without needing any expert demonstrations. Surprisingly, Motif's reward leads to better game score than the one obtained by using the score itself as a reward.

Given access to an event captioning mechanism, a few properties make Motif a general method:
• it is entirely based on open models
• the LLM doesn't need direct access to the environment dynamics (e.g., its source code)
• the LLM doesn't need to understand observation and action spaces

The best part? You can start using Motif right now, even on a small compute budget: the whole pipeline can take less than two GPU-days.

Feel free to read our paper and try our code out.
Paper: https://t.co/qHJqpJX6Gl
Code: https://t.co/aqDGr2LsXo
Blog post: https://t.co/ULDRodTcyK

Work co-lead by @MartinKlissarov and myself, with @shagunsodhani @robertarail @pierrelux Pascal Vincent @yayitsamyzhang @HenaffMikael

Learn more in the thread 🧵

[Tianwei Ni @twni2016]

Is Attention All You Need in RL?
Excited to share our #NeurIPS2023 (oral) paper 🚀https://t.co/Ehjoe6eP3w Transformers can recall old memories to solve tasks but struggle to find out which old actions contributed to future success. A 🧵.