Soichiro Nishimori

PhD candidate@UTokyo
Reinforcement Learning
Game AI

[Soichiro Nishimori @nissymori1]

Our paper "Emergence of Exploration in Policy Gradient Reinforcement Learning via Retrying" is accepted at #ICML2026 🇰🇷

Proposed a novel exploration objective called ReMax, evaluating best of multiple trials under uncertainty.

The objective comes from the basic question,

Why do RL agents need to explore?

We argue it is because
♻️ Agents are allowed to retry (otherwise, the rational choice is the current best action).
📈 Return is uncertain (otherwise, no point in trying suboptimal actions.)

ReMax naturally captures these intuitions by modeling the distribution of returns and evaluating the maximum over multiple retries, thereby encouraging agents to select actions that are currently suboptimal but highly uncertain.

The diagram is inspired by the Vector Policy Optimization (VPO) paper.

🧵1/n

[Soichiro Nishimori @nissymori1]

My first ICML was really good!
It was great meeting you all!

[Sky Thanawat @skydddoogg]

We have two caps 🧢 today. CapCode and CapReward. ICML 2026 AgenticUQ workshop. Poster 2804!

[Soichiro Nishimori @nissymori1]

@_______music_ \exist exit

[共立出版 アリがと蟻 @1738310]

「大偏差原理」は、確率論の極限の理論の中でも「大数の法則」や「中心極限定理」に匹敵する重要な理論です。確率論において極めて広範に現れる基本的な研究対象でアリ、その応用も数理物理学・力学系・情報理論など多岐にわたります。

[Soichiro Nishimori @nissymori1]

I’ll be co-presenting our poster with my internship mentor, @PaavoParmas

📍 Thu Jul 9, 14:30 / 2:30 PM, Hall A #111

We’ll be presenting ReMax, a novel exploration objective based on the idea that agents can retry.

I first heard about ReMax from Paavo back in 2022, when I was still a bachelor’s student. At the time, I didn’t fully appreciate how natural the objective was.
I later re-encountered ReMax during my internship with him in 2024 as a PhD student. After working through it in depth, I finally came to appreciate its power.
Now I really believe ReMax is a very natural objective for exploration. There has also been a lot of follow-up work since then, so please check his post for more details!

It has been a great pleasure to work with him.
Please stop by our poster to learn about ReMax, discuss set RL and follow-up work, and hear the story behind it. It should be fun！

[Paavo ✈️ ICML @PaavoParmas]

At ICML, tomorrow (Thu 14:30) we’re presenting ReMax: a simple idea that leads to stochastic exploration when greedily maximizing rewards over retries.
What I find exciting is that this idea did not stay in one corner of RL. It now connects to LLM post-training, bandits, image diversity, continuous control, and distributional RL. I created a new thread about our body of follow-up works🧵
#ICML2026 @icmlconf

[Paavo ✈️ ICML @PaavoParmas]

Finally, OrderGrad asks: why stop at the maximum?
ReMax is one order statistic. By changing rank weights, we can optimize top-m, medians, trimmed means, CVaR-like objectives, lower-tail objectives, and more.
The nice part is that the reward transformation can be one line of code, and can be combined with GRPO, DAPO, GSPO, or other REINFORCE-style policy-gradient methods.
Coauthors: @yongmin97, Kohsei Matsutani, Shota Takashiro, @nissymori1, @kojima_tks, Yusuke Iwasawa, Yutaka Matsuo

[Paavo ✈️ ICML @PaavoParmas]

We extend the ReMax estimator to continuous action spaces with an SAC-style pathwise estimator.
Here, I particularly like the learning dynamics visualizations by @nissymori1. Even in deterministic tasks, the dynamics can be altered so that the policy entropy increases. Note that this never happens with the standard RL objective on this task!
Coauthor: @nissymori1

[Kazuki Ota @kazukiota0]

I’m happy to share that our paper, “Revisiting Regularized Policy Optimization for Stable and Efficient Reinforcement Learning in Two-Player Games,” has been accepted at #ICML2026!

📍Project Page: https://t.co/lZKi5rvtW5

Many thanks to my collaborators!

[Paavo ✈️ ICML @PaavoParmas]

📣 I will co-present this at ICML next week!
It was super fun mentoring Soichiro on this project.

The paper builds on an idea I first explored in 2021 and co-wrote in an early 2022 preprint: optimizing set objectives, such as max@k. It has been exciting to see this direction become much more active since then, with related work including SetRL, VPO, and more.

Come chat with us at the poster! 🇰🇷
#ICML2026 @icmlconf

[Shinnosuke ONO / 小野新之介 ✈️ ICML2026 🇰🇷 @shinnosukeono]

We present our paper "Mitigating Reward Hacking via Adversarial Robustness" at EIML@ICML2026!

We conjecture that reward hacking is often caused by flipped advantage-sign estimations, and propose SignCert-PO, a new algorithm built on the theory of randomized smoothing! 🧵

[Shinnosuke ONO / 小野新之介 ✈️ ICML2026 🇰🇷 @shinnosukeono]

Come chat on Friday, July 10 at EIML@ICML2026, Seoul!🇰🇷

Joint work with @johannesack, @nissymori1, @tksii and Masashi Sugiyama.

https://t.co/k9Yr4teESz

[Stefano V. Albrecht @s_albrecht]

Thrilled to share that my new book "Becoming an AI Researcher" can now be pre-ordered from @CambridgeUP! The final book cover turned out great. Slated to arrive in stores in Nov 2026. 📘

Read the latest Substack post for updates and links:
➡️https://t.co/qLvWgiYIfF

#PhD #AI

[Soichiro Nishimori @nissymori1]

Thanks a lot for your interest!

The key point is that the additional computational overhead of ReMax itself is negligible compared to the base algorithm, PPO in our case, once the Q and policy vectors are available. In our experiments, most of the compute comes from environment steps and network inference, rather than from the ReMax computation.

That said, our current experiments are based on benchmarks such as Atari, where these vectors are relatively accessible. In LLM policy optimization, the main challenge is that the corresponding Q and policy vectors are not readily available due to the large token/action space, so the method is not directly applicable as-is. For this direction, we need sampling-based estimation. Please also check out our related work on max@k policy optimization: https://t.co/MbUlZPEzez

Happy to chat more in person!

[Soichiro Nishimori @nissymori1]

Our paper "Emergence of Exploration in Policy Gradient Reinforcement Learning via Retrying" is accepted at #ICML2026 🇰🇷

Proposed a novel exploration objective called ReMax, evaluating best of multiple trials under uncertainty.

The objective comes from the basic question,

Why do RL agents need to explore?

We argue it is because
♻️ Agents are allowed to retry (otherwise, the rational choice is the current best action).
📈 Return is uncertain (otherwise, no point in trying suboptimal actions.)

ReMax naturally captures these intuitions by modeling the distribution of returns and evaluating the maximum over multiple retries, thereby encouraging agents to select actions that are currently suboptimal but highly uncertain.

The diagram is inspired by the Vector Policy Optimization (VPO) paper.

🧵1/n

[Soichiro Nishimori @nissymori1]

@Matsuo_Lab @PaavoParmas @sotetsuk @Tdash_Koz @t_kitamura14 I’ll be presenting my poster at Hall A #111 on Wed, July 8, 2026, 10:30 PM–12:15 AM PDT.

Please stop by if you’re around — I’d be happy to chat!

Paper: https://t.co/jtlQbXZfON
Code: https://t.co/M0IR1eKWwk
See you in Seoul!! 🇰🇷

[Soichiro Nishimori @nissymori1]

🧵[8/n]
This work was done during my internship at @Matsuo_Lab , mentored by @PaavoParmas.
I am also grateful to the other collaborators: @sotetsuk, @Tdash_Koz, @t_kitamura14, Shin Ishii, and Yutaka Matsuo!

Core contributors: me (implementation, experiments, writing), Paavo (conceptualization, key mathematical derivations and proposals), see paper for full contributions.