Takashi Ishida

research scientist at RIKEN | associate professor at The University of Tokyo | part-time research scientist at Sakana AI | model evaluation

[Takashi Ishida @tksii]

Can a benchmark be fully public and still protect against contamination?

LLM benchmarks are expensive to build but expire fast: once public, they leak into training data, and a high score starts to reflect memorization rather than capability. Keeping the test set private behind a leaderboard isn't enough either: repeated queries against the same hidden test set create a feedback loop that models can overfit to without ever seeing the labels.

This motivates our ICML 2026 paper: CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfitting.

The idea: deliberately lower the benchmark's Bayes accuracy (the best score any model can possibly achieve) to a known cap. We propose a methodology to modify each problem so that multiple answers are logically acceptable, then randomly freeze one as the benchmark label. The label is still correct under the modified question, but even an ideal model cannot know which valid answer was sampled. With two equally valid answers per problem, no model should score above 50%. With four, 25%.

So if a model scores significantly above the cap, that's a black-box alarm: either the frozen labels leaked into training, or they were effectively learned through feedback loops.

Crucially, this is not "adding label noise." Noisy labels are wrong and distort evaluation, but capped labels remain logically valid and model rankings are preserved in expectation. We also theoretically show the original benchmark accuracy can be estimated from the capped accuracy.

We verify this in controlled experiments. Models intentionally trained on capped benchmark data cross the cap, and stay above it. And in a simulated private leaderboard where an optimizer sees only the accuracy, the feedback loop eventually pushes a model past the cap.

There is also a second layer of protection. The capped benchmark can be released without disclosing the original answers, so even if the released data ends up in training corpora, the original ground truth itself is still not leaked. This offers a middle ground between fully public benchmarks that leak and fully private ones that can still be gamed.

If you're building an LLM benchmark, consider giving it a Bayes-accuracy cap before release!

Paper: https://t.co/DVqZBakk2t
Blog: https://t.co/heP7HYo9wG
Code: https://t.co/RBzlFV3f3g
Data: https://t.co/yF1HsOLYqI

This was an exciting collaboration with @skydddoogg and Ikko Yamane. We are at ICML🇰🇷, and we will be presenting in Hall A on Wednesday, July 8, 2026, from 2:30 PM. Please stop by if you are interested in benchmarks, contamination, and evaluation!👋

[Sky Thanawat @skydddoogg]

We have two caps 🧢 today. CapCode and CapReward. ICML 2026 AgenticUQ workshop. Poster 2804!

[Johannes Ackermann @johannesack]

I'm presenting this work tomorrow in the morning session, swing by if you have any questions! #ICML2026
📍 Hall A, Poster #2111
⏰ 10:30AM-12:15AM

[Takashi Ishida @tksii]

I'm in Seoul for ICML 2026!🇰🇷 If you’re interested in evals, coding and long-horizon agents, or issues such as contamination, reward hacking, and model cheating, please stop by our posters (details below) or say hi if you see me around, I'd love to chat!

Gradient Regularization Prevents Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards
@johannesack, @mnoukhov, @tksii, Masashi Sugiyama
https://t.co/cr7Nq2Vh0Y
https://t.co/dPi65behkw
July 8 10:30am- @ HALL A #2111

CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfitting
@tksii, @skydddoogg, Ikko Yamane
https://t.co/DVqZBakRS1
https://t.co/RBzlFV3MSO
Jul 8 2:30pm- @ HALL A #4413

Mitigating Reward Hacking in RLHF via Advantage Sign Robustness (EIML Workshop)
@shinnosukeono, @johannesack, @nissymori1, @tksii, Masashi Sugiyama
https://t.co/WxG0w9IXed
https://t.co/Llv3CLzlyG
July 10 @ ROOM E5 - E6 (3rd Floor)

CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies (FAGEN Workshop)
@strayer_13, Daichi Hattori, Kazuo Araragi, Keita Ogawa, Shota Onose, @taromakino, Teppei Usuki, @tksii (collaboration between KPMG Azsa & Sakana AI!)
https://t.co/l44ExW2Wvq
https://t.co/9iymrmrzP5
https://t.co/KrXRHXOhUi
July 10 @ GRAND BALLROOM 104-105

Do Coding Agents Deceive Us? Detecting and Preventing Cheating via Capped Evaluation with Randomized Tests (AgenticUQ Workshop)
@skydddoogg , @johannesack, @nissymori1, @NolfwinMk, Masashi Sugiyama, @tksii
https://t.co/mHIOUM83Pn
https://t.co/bZsYGZW0wh
https://t.co/daDQWolXyC
July 11 @ ROOM E1 - E4 (3rd Floor)

[Shinnosuke ONO / 小野新之介 ✈️ ICML2026 🇰🇷 @shinnosukeono]

We present our paper "Mitigating Reward Hacking via Adversarial Robustness" at EIML@ICML2026!

We conjecture that reward hacking is often caused by flipped advantage-sign estimations, and propose SignCert-PO, a new algorithm built on the theory of randomized smoothing! 🧵

[Soichiro Nishimori @nissymori1]

Our paper "Emergence of Exploration in Policy Gradient Reinforcement Learning via Retrying" is accepted at #ICML2026 🇰🇷

Proposed a novel exploration objective called ReMax, evaluating best of multiple trials under uncertainty.

The objective comes from the basic question,

Why do RL agents need to explore?

We argue it is because
♻️ Agents are allowed to retry (otherwise, the rational choice is the current best action).
📈 Return is uncertain (otherwise, no point in trying suboptimal actions.)

ReMax naturally captures these intuitions by modeling the distribution of returns and evaluating the maximum over multiple retries, thereby encouraging agents to select actions that are currently suboptimal but highly uncertain.

The diagram is inspired by the Vector Policy Optimization (VPO) paper.

🧵1/n

[Takashi Ishida @tksii]

Great discussions on the difficulty of verifiability and explainability (in AI for math), through many examples from the history of mathematics and mathematicians.

[Dwarkesh Patel @dwarkesh_sp]

Always so much fun to chat with @3blue1brown

AI has been making much faster progress in math than in other fields.

As a result, mathematics is showing us, very concretely, what AI progress in other fields will look like.

Even within mathematics, there's a jagged landscape. What does it look like?

What is the nature of the most important conceptual breakthroughs in the history of mathematics, and how different are they from what AIs are currently able to do?

Does AI (on net) increase or decrease human understanding of the field?

How big is the overhang from having AIs systematically try to connect ideas already in the literature?

And what advice does Grant have for aspiring mathematicians, coders, and other students who are passionate about fields that are being most transformed upon by AI?

0:00:00 – AI is discovering new proofs. Is that AGI?
0:11:32 – The verification loop on conceptual breakthroughs can be a century long
0:26:12 – Will we understand an AI proof of the Riemann hypothesis?
0:38:08 – Can AI find the hidden bridges between fields?
0:53:48 – Why real-world tasks don’t fit into RL environments
1:07:07 – Good writing requires theory of mind that AI still lacks
1:16:02 – Why learning will still depend on human curation

Look up Dwarkesh Podcast on Spotify, Apple Podcasts, YouTube, etc.

[Takashi Ishida @tksii]

Thanks for the question! Vending-Bench is more of a consumer-facing business setup (with a vending machine), while CoffeeBench is a B2B multi-agent supply chain with farmers, roasters, and retailers. I believe this makes it more challenging but also more interesting from an accounting and safety perspective.

We also include a comparison table in our paper:

[Takashi Ishida @tksii]

Excited to share CoffeeBench!!☕️☕️☕️

We evaluate LLM agents in a 90-day B2B coffee supply-chain economy spanning farmers, roasters, and retailers, where these firms negotiate, manage inventory, set prices, handle invoices, and manage cash flow.

Beyond evaluating long-horizon business performance, such as whether agents can improve net income, I'm also excited about the accounting and AI safety angle: because CoffeeBench includes B2B trade, invoices, and cash-flow constraints, it could potentially help us study whether stronger future agents develop or discover problematic business behaviors such as circular trading, channel stuffing, or accounting-fraud-like strategies.

This was an exciting cross-disciplinary collaboration with researchers at KPMG AZSA @KPMG_JP and @SakanaAILabs colleagues @strayer_13 (first author!) and @taromakino 🤝

The work will be presented at @FAGENWorkshop in ICML 2026!🇰🇷

[Sakana AI @SakanaAILabs]

SakanaAIは、有限責任あずさ監査法人と共同で、LLMエージェントの長期的な経営能力を評価する新しいベンチマーク「CoffeeBench」を公開しました。

ブログ：https://t.co/9dgRvLkBw2

現実の経済では、消費者へ直接売るビジネスだけでなく、企業同士が継続的に取引するビジネスも重要です。CoffeeBench は、農家・焙煎店・小売店の計6社が参加するコーヒー業界のサプライチェーンをシミュレーションし、各社をLLMエージェントが運営。90日間にわたって価格交渉・発注・在庫管理などを行い、純利益の最大化を目指します。

最新のLLMを同じ環境で競わせると、経営成績は大きく分かれました。積極的に交渉し、利益に直結する一手を打ち続けるモデルがいる一方で、自身の状況を分析しながらも行動に移さず、待機し続けて赤字に陥るモデルも出てくるなど、長期タスクならではの振る舞いの違いが観察できました。

CoffeeBenchは、長期にわたり相互作用するLLMエージェントの能力や振る舞いを評価・分析していくための第一歩です。今後は、複数エージェント間で生じる協調・競争・逸脱行動や、その監査・ガバナンス手法の研究へと発展させていくことを目指します。

本研究は ICML2026 Workshop "Failure Modes in Agentic AI" にて発表予定です。

論文：https://t.co/LQ9w0r960H ☕

[Issa Sugiura @strayer_13]

SakanaAIのインターンでCoffeeBenchを公開しました！
自販機の経営能力を評価するベンチマークVending-Benchを、会計やサプライチェーンの要素を加えて、より現実のビジネスに近い環境での評価を可能にしました。
コードやトラジェクトリも公開しているのでぜひご覧ください！

[Sakana AI @SakanaAILabs]

Sakana AI CEO David Ha (@hardmaru) appeared on TBS CROSS DIG’s “1on1 Tech.” He talks about our founding story, latest research and technical vision, product launches including Sakana Fugu, Japan’s AI strategy, and his hopes for Japanese society. Watch the interview.

[Takashi Ishida @tksii]

📣 Excited to release our new paper introducing CapCode and CapReward for coding agents!

CapCode is a lightweight way to turn coding evals into cheating detectors while preserving their capability signal. CapReward is a new post-training reward design for reducing cheating, based on capped benchmarks.

If you are interested in cheating behaviors of coding agents, please take a look!

Details in the thread below 🧵👇

[Sky Thanawat @skydddoogg]

Coding agents are evaluated with unit tests:

more tests passed = better model.

But if tests or feedback are accessible, models may learn to game them.

We introduce CapCode to detect suspiciously high scores, and CapReward to discourage them during RL.

🧵1/10

[Takashi Ishida @tksii]

これからICLRに参加するためブラジルに向かいます✈️🇧🇷現地参加される方、よろしくお願いいたします🙏 #ICLR2026 @iclr_conf

ご関心があれば、以下のポスター（土曜日、Session 6, Pavilion 3 & 4）にもぜひお立ち寄りください！

Towards Scalable Oversight via Partitioned Human Supervision
https://t.co/5QxvFTxbwW

EDINET-Bench: Evaluating LLMs on Complex Financial Tasks using Japanese Financial Statements
https://t.co/bWWKfVkH8s

Practical estimation of the optimal classification error with soft labels and calibration
https://t.co/UY94tBaWgN

[Sakana AI @SakanaAILabs]

Sakana AIが昨年公開した日本語金融ベンチマーク「EDINET-Bench」が、国際会議 #ICLR2026 に採択されました。

ブログ：https://t.co/oQUfA8EqkK

EDINET-Benchは、金融庁EDINETの有価証券報告書約41,000件をもとに、会計不正検知・業績予想・業種予測の3タスクでLLMを評価するベンチマークです。

公開以降、日本の金融分野における複数の研究で引用されています。Sakana AIでは、今後も金融領域におけるAIの実装を進めつつ、その基盤となる研究開発を推進します。

論文：https://t.co/uD3z8baVir

[hardmaru @hardmaru]

I am very proud of our team for releasing EDINET-Bench, and it is fantastic to see a Japanese financial dataset recognized at #ICLR2026 this week. We need more diverse, non-English datasets to evaluate models in the real world.

https://t.co/Q9y4xwWqgM

[Issa Sugiura @strayer_13]

I’ll be presenting our work on EDINET-Bench as a poster at ICLR!
Looking forward to discussing evaluation for real-world LLM applications with researchers from around the world in Rio💃.

[Kei NAKAGAWA @keeeeei0315]

客員研究員の加藤さん(@kato_jp )との研究がarXivで公開されました。
"Sequential Audit Sampling with Statistical Guarantees"
SIGFIN35でも発表してもらった、監査サンプリングにおけるサンプルの追加抽出を、統計的保証つきの逐次検定として定式化したものです。
https://t.co/aEitrLSqX6

https://t.co/vSio6DvKHY

[Takashi Ishida @tksii]

4月9日（木）に日本橋でStatistical Safeguarding Workshop 2026を開催します！

Warwick、Oxford、Edinburgh、東大、阪大、理研から研究者が集まり、統計的セーフガーディングについて議論します。

現地参加の受付は終了しましたが、Zoomでのオンライン参加はまだ可能です。ぜひご参加ください！👋

登録はこちら：
https://t.co/TOpKKa1RQk

[Takashi Ishida @tksii]

東大の複雑理工学専攻・コンピュータ科学専攻で入試説明会を実施します。ご関心のある方はぜひご参加ください！

📣複雑理工学専攻
🗓️4/4（土）
ハイブリッド：全体説明、研究室紹介、個別研究室見学
🗓️4/25（土）
現地開催：全体説明、施設・研究室ツアー、個別研究室見学
https://t.co/sWhxnxUaBO

📣コンピュータ科学専攻
🗓️5/23（土）
専攻全体説明は録画配信、研究室見学はハイブリッド
https://t.co/ShjGu1x2DK

[Sakana AI @SakanaAILabs]

🐟 Sakana Chat 公開 🐟

Sakana AIは、Sakana Chatを無料公開しました。

https://t.co/lyIrGJY4fW

Web検索機能と高速レスポンスを備えたAIチャットです。日本国内から、どなたでもお使いいただけます。ぜひ、お試しください。