Hao Sun - RL

RS @GoogleDeepMind. Prev. PhD @CambridgeUni, #MMLab, B.Phys. @PKU1898

[Hao Sun - RL @HolarisSun]

I have been working on Reward Modeling (and Inverse RL) for LLMs for the past 1.5 years. We built reward models (RMs) for prompting, dense RMs to improve credit assignment, and RMs from the SFT data

However, many questions remained unclear to me until this paper was finished.🧵

[Hao Sun - RL @HolarisSun]

I’d be keen to vibe code (w/ Gemini🤓) an AI-native Google scholar system. (AI for pub is ai4s!)

[Sian Gooding @SianGooding]

I strongly agree with the @arxiv ban on hallucinated references but it makes me wonder if papers are even the right format to disseminate knowledge now? If it is LLMs all the way down: writing, coding, reviewing, citing then we've automated a wrapper around the science. A narrative makes sense when you have one, not when it's forced. Maybe it's time to automate the verification instead. Agentic harnesses where you submit code and data, not a story?

[Edward Grefenstette @egrefen]

🚨 Deadline update!

The two Research Scientist jobs for @GoogleDeepMind's Autonomous Agents team now have a deadline of Friday, 6th March 2026 at 10:00am GMT.

We will look to start interview loops shortly thereafter.

See postings below for links to larger threads. [1/3]

[Hao Sun - RL @HolarisSun]

Our team is hiring an RS working on RL, apply and join us!

[Edward Grefenstette @egrefen]

📣 Announcing a second, different job posting from yesterday's.

The @GoogleDeepMind Autonomous Agents team is seeking to hire a Research Scientist to work on established projects on sample efficient learning and robust self-improvement. Details and link below. [1/5]

[Martin Klissarov @MartinKlissarov]

In the limit, what's important is our ability to adapt. 

What is a good recipe for teaching agents to adapt on-the-fly?

We introduce two meta-learning for LLMs papers written with @JonnyCoook at @GoogleDeepMind.

This is research from last year we can finally share 🧵👇

[Martin Klissarov @MartinKlissarov]

💫 If you are interested in improving the meta-learning abilities of SOTA agents, keep in touch with our team at @GoogleDeepmind led by @egrefen! 

We will keep exploring diverse ideas in this space, keep an eye out as we’ll have job postings for RS and RE positions very soon! 🚀

[Hao Sun - RL @HolarisSun]

Dynamics of the embeddings can tell us so much as feedback. Those super exciting discoveries are opening up a new era of reward modeling!

[Sian Gooding @SianGooding]

New work with @egrefen at @GoogleDeepMind:

🚨Interaction Dynamics as a Reward Signal for LLMs🚨

When it comes to interactions, the "how" is just as important as the "what"

There is a signal in how we interact with a model that text analysis misses: hesitation, drift, friction

[Hao Sun - RL @HolarisSun]

Excited to be co-hosting this with Martin!
2026 Student Researcher opportunity at @GoogleDeepMind
continual learning × LLM reasoning × RL, and more to explore!

[Martin Klissarov @MartinKlissarov]

🚨Internship alert 🚨

Together with @HolarisSun, we will be hosting a Student Researcher next year at @GoogleDeepMind.

The research will be at the intersection of continual learning, self-improvement and social learning.

➡️ Please do fill this form: https://t.co/TCTpbG96jT

[Jingling Li @jingling_li]

Can AI reliably judge itself? The success of RL hinges on good verifications. We're hiring an intern to investigate a key AI challenge: the limits of current models as judges for complex, open-ended tasks where systematic verification is difficult. DM if you are interested!

[Hao Sun - RL @HolarisSun]

Truly insightful thread. Having played poker for 4 yrs and thought a lot about GTO, SP&Exploit, this minimax view is illuminating.
We discussed the contrast between SP in 2p0s games and in LLMs at our AAAI’25 tutorial on IRL×LLMs, tho definitely not at this level of depth.

[Noam Brown @polynoamial]

Below is a deep dive into why self play works for two-player zero-sum (2p0s) games like Go/Poker/Starcraft but is so much harder to use in "real world" domains. tl;dr: self play converges to minimax in 2p0s games, and minimax is really useful in those games.

Every finite 2p0s game has a minimax equilibrium, which is essentially an unbeatable strategy in expectation (assuming the players alternate sides). In rock paper scissors, for example, minimax is 1/3rd on each action.

Is minimax what we want? Not necessarily. If you're playing minimax in Rock Paper Scissors when most opponents' strategies are "always throw Rock" then you're clearly suboptimal, even though you're not losing in expectation. This especially matters in a game like poker because playing minimax means you might not make as much money off of weak players as you could if you maximally exploited them.

But the guarantee of "you will not lose in expectation" is really nice to have. And in games like Chess and Go, the difference between a minimax strategy and a strategy that optimally exploits the population of opponents is negligible. For that reason, minimax is typically considered the goal for a two-player zero-sum game. Even in poker, the conventional wisdom among top pros is to play minimax (game theory optimal) and then only deviate if you spot clear weaknesses in the opponent.

Sound self play, even from scratch, is guaranteed to converge to a minimax equilibrium in finite 2p0s games. That's amazing! By simply scaling memory and compute, and with no human data, we can converge to a strategy that's unbeatable in expectation.

What about non-2p0s games? Sadly, pure self play, with no human data, is no longer guaranteed to converge to a useful strategy. This can be clearly seen in the Ultimatum Game. Alice must offer Bob $0-100. Bob then accepts or rejects. If Bob accepts, the money is split according to Alice's proposal. If Bob rejects, both receive $0.

The equilibrium (specifically, subgame perfect equilibrium) strategy is to offer 1 penny and for Bob to accept. But in the real world, people aren't so rational. If Alice were to try that strategy with real humans she would end up with very little money. Self play becomes untethered from what we as humans find useful.

A lot of folks have proposed games like "an LLM teacher proposes hard math problems, and a student LLM tries to solve them" to achieve self-play training, but this runs into similar problems as the Ultimatum game where the equilibrium is untethered from what we as humans find useful.

What should the reward for the teacher be in such a game? If it's 2p0s then the teacher is rewarded if the student couldn't solve the problem, so the teacher will pose impossible problems. Okay, what if we reward it for the student having a 50% success rate? Then the teacher could just flip a coin and ask the student if it landed Heads. Or the teacher could ask the student to decrypt a message via an exhaustive key search. Reward shaping to achieve intended behavior becomes a major challenge. This isn't an issue in 2p0s games.

I do believe in self play. It provides an infinite source of training, and it continuously matches an agent with an equally skilled peer. We've also seen it work in some complex non-2p0s settings like Diplomacy and Hanabi. But applying it outside of 2p0s games is a lot harder than it was for Go, Poker, Dota, and Starcraft.

[Yunyi Shen/申云逸 🐺 @ShenRaphael]

Decide to go on academic job market first (rather than e.g. quant). I'm now actively looking for postdoc and/or AP/fellow in stats/ML! I do mostly probablistic ML for bio and physics. Can also do LLM a bit mostly on post training. DM open!

[Hao Sun - RL @HolarisSun]

@leonardtang_ Thank you, Leonard!

[Hao Sun - RL @HolarisSun]

This is SCIENCE🚀!!!

[Alan Jeffares @Jeffaresalan]

Our new ICML 2025 oral paper proposes a new unified theory of both Double Descent and Grokking, revealing that both of these deep learning phenomena can be understood as being caused by prime numbers in the network parameters 🤯🤯

🧵[1/8]

[Hao Sun - RL @HolarisSun]

🚀 RL is powering breakthroughs in LLM alignment, reasoning, and agentic apps.
Are you ready to dive into the RL x LLM frontier?

Join us at @aclmeeting ACL’25 tutorial:
Inverse RL Meets LLM Alignment
this Sunday at Vienna🇦🇹(Jul 27th, 9am)

📄 Preprint at https://t.co/WMfVgb05E8

[Hao Sun - RL @HolarisSun]

@ariapoy @ShenRaphael @jeanfrancois287 Thank you Poy!

[Hao Sun - RL @HolarisSun]

Unfortunately won't be able to attend #ICML2025 due to a long pending Canadian visa application — submitted in Oct 2023, still pending after 625 days 🙂‍↔️

That said, I'm excited to share our paper on Active Preference Learning & Understanding Reward Models 🧵👇

[Hao Sun - RL @HolarisSun]

If you're interested in RLHF and reward modeling check it out — and feel free to chat with Jef at #ICML2025!

📄 https://t.co/Vp9dm85hmn
🔗 https://t.co/668OvhepBb
🤝 Joint work with @ShenRaphael and @jeanfrancois287

[Hao Sun - RL @HolarisSun]

We revive classical tools from the stats & experimental design literature. It turns out, many modern challenges already have elegant, sample-efficient solutions hidden there.

All experiments were run efficiently on CPU-only machines, using our RM infra (open-sourced)!

[Hao Sun - RL @HolarisSun]

Now with Qwen’s RL-fine-tuning results, are we witnessing a quiet return of prompt optimization/engineering?

Now we have a 2-player game: users become “lazy prompters”, but the system prompts (e.g. thinking patterns) need to be highly optimized.

Next: Bi-level optimization?

[Hao Sun - RL @HolarisSun]

@YouJiacheng 🤣🤣🤣attention is all you need!