Minttu Alakuijala

LLM + RL research at @Meta
// Prev: Postdoc @AaltoUniversity, PhD at @GoogleAI & @Inria
// LLM agents, RL & planning, reward models, robotic manipulation

[TMLR Papers with Videos @TmlrVideos]

Video-Language Critic: Transferable Reward Functions for Language-Conditioned Robotics

Minttu Alakuijala, Reginald McLean, Isaac Woungang et al.

https://t.co/zIqZnbuUKI

#robot #critic #robots

[Accepted papers at TMLR @TmlrPub]

Video-Language Critic: Transferable Reward Functions for Language-Conditioned Robotics

Minttu Alakuijala, Reginald McLean, Isaac Woungang et al..

Action editor: Héctor Palacios.

https://t.co/ZhIyUwZjnw

#robot #critic #robots

[OpenAI @OpenAI]

Detecting misbehavior in frontier reasoning models

Chain-of-thought (CoT) reasoning models “think” in natural language understandable by humans. Monitoring their “thinking” has allowed us to detect misbehavior such as subverting tests in coding tasks, deceiving users, or giving up when a problem is too hard.

We believe that CoT monitoring may be one of few tools we will have to oversee superhuman models of the future.

We have further found that directly optimizing the CoT to adhere to specific criteria (e.g. to not think about reward hacking) may boost performance in the short run; however, it does not eliminate all misbehavior and can cause a model to hide its intent. We hope future research will find ways to directly optimize CoTs without this drawback, but until then:

We recommend against applying strong optimization pressure directly to the CoTs of frontier reasoning models, leaving CoTs unrestricted for monitoring.

We understand that leaving CoTs unrestricted may make them unfit to be shown to end-users, as they might violate some misuse policies. Still, if one wanted to show policy-compliant CoTs directly to users while avoiding putting strong supervision on them, one could use a separate model, such as a CoT summarizer or sanitizer, to accomplish that.

[Minttu Alakuijala @malakuijala]

After training, our Llama-3.1-70B based agent outperforms GPT-4o and DeepSeek-V3 on ToolQA tasks, without regressing on other benchmarks!

Work with Ya Gao, Georgy Ananov, @samikaski, @marttinen_pekka at @AaltoUniversity and Alexander Ilin and Harri Valpola at System 2 AI. (3/3)

[Minttu Alakuijala @malakuijala]

New preprint!
Current LLM agents usually rely on prompting to learn task-specific knowledge. In Memento No More (https://t.co/aErKPUwMaM) we train agents to internalize knowledge and skills for multiple tasks without relying on ever-expanding prompts or prior demonstrations (1/3)

[Minttu Alakuijala @malakuijala]

Prompting alone does not scale to mastering multiple new tasks due to information overload (the Memento effect) and inference cost. Instead, our agent integrates new knowledge into its weights through context distillation and efficient use of corrective feedback from humans (2/3)

[Minttu Alakuijala @malakuijala]

@RaymondRChua @NeurIPSConf @dainese_nicola @MerloMerler @marttinen_pekka Thanks Raymond!

[Minttu Alakuijala @malakuijala]

I'm happy to announce our paper
Generating Code World Models with Large Language Models Guided by Monte Carlo Tree Search, has been accepted @NeurIPSConf'24!

Link: https://t.co/R8Si9oohje
Joint work with @dainese_nicola, @MerloMerler and @marttinen_pekka.

#NeurIPS2024

[Minttu Alakuijala @malakuijala]

Specifically, we employ a novel variant of MCTS with generation, refinement and bug fixing actions, and find it to perform well not only in world model building but in general long-form programming problems, as well!
For more details, check out the paper: https://t.co/R8Si9oohje

[Minttu Alakuijala @malakuijala]

Instead, we build world models in an explicit, interpretable and efficient form: code. We iteratively prompt an LLM to write and refine code to match an environment's text description and collected data. These candidate programs form a search tree informed by prediction accuracy.

[Finnish Center for AI 🦣 @[email protected] @FCAI_fi]

Looking for a fully-funded PhD position in AI or ML? We are opening the next call for applications in the Finnish AI doctoral program soon!
See all the details and how to apply: https://t.co/G4MkmpB152

[Minttu Alakuijala @malakuijala]

Join our summer school this July to learn how generative models are used in human-in-the-loop and collaborative systems!
Registrations are open until April 7th

[Collaborative and Generative AI Summer School @CoGenAISchool]

Join us at the ELLIS Summer School on Collaborative and Generative AI in Helsinki, Finland, from July 1st to 5th, 2024!

For more information, or to register, see https://t.co/2I0d3quIAz

#AI #CoGenAI #ELLIS #SummerSchool #Helsinki #AaltoUniversity

[Minttu Alakuijala @malakuijala]

@marcgbellemare I would love to hear more!

[Minttu Alakuijala @malakuijala]

A single reward model generalizes to new tasks given a goal image. We further show the predicted rewards accelerate training of several unseen robotic manipulation tasks compared to existing label-free reward learning methods.
3/3

[Minttu Alakuijala @malakuijala]

I'm excited to present our paper
"Learning Reward Functions for Robotic Manipulation by Observing Humans" at #ICRA2023 this week!

ArXiv: https://t.co/I7t7q9KrYO
Website: https://t.co/mZ0zCFWf4g
Work with @gabepsilon, @julienmairal, Jean Ponce & @CordeliaSchmid
1/3

[Minttu Alakuijala @malakuijala]

We introduce HOLD (Human Offline Learned Distances), and show that image-conditioned robotic reward functions can be learned exclusively from unlabeled, mixed-task human videos, using time-contrastive or regression objectives.
2/3

[Minttu Alakuijala @malakuijala]

Apply by Dec 24th to join our 2021 #GoogleResearch intern cohort! https://t.co/KkDp9UfXEE

[Csaba Szepesvari @CsabaSzepesvari]

With some glitches, but we are done with the first of the series. Never knew so many people care about RL theory, yay! Great talk Chi Jin! Awesome audience! Next one can only be smoother:) Sign up here if you have not signed up yet: https://t.co/pS5mTVXcir

[Minttu Alakuijala @malakuijala]

A nice summary by Leo Gao of some of the most influential leaps forward deep learning took in the 2010s. https://t.co/gjXJJDLBYm