Kaggle

The leading platform for crowdsourced benchmarks, competitions, and hackathons where the community shapes real-world progress in AI.

[Kaggle @kaggle]

▶️ Watch the full episode here:
https://t.co/ZGfg67COrW

[Kaggle @kaggle]

Why do RL Agents need to mix up their strategies?

"Orbit Wars" winner Isaiah Pressman took the top spot with reinforcement learning that simulated 15 billion steps.

He shares the risks of predictability in this bonus episode of Beyond the Leaderboard.

Here's a sneak peek:

[Kaggle @kaggle]

Bench Summit 2026 🎬

On October 28, Kaggle, @GoogleDeepMind, and @zapier brought together benchmark publishers, data labs, research organizations, enterprise users, and frontier labs for an inaugural summit.

The core takeaway: Bringing diverse stakeholders together in the same room is the only way we fix broken benchmarks and define the future of AI evaluation.

• 264 attendees
• 43 speakers
• 35+ benchmarks

Keep an eye on our YouTube channel for the full digital track soon! And, join us for the next Bench Summit so we can build the future of AI evaluation together.

[Kaggle @kaggle]

▶️ Watch the full episode here: https://t.co/4bSfM8ytmP

[Kaggle @kaggle]

"A perfect illustration of the bitter lesson: the one with the most compute wins," notes @simon_jegou, who estimates his Orbit Wars solution used 1/8 the compute of the winner.

In the first episode of Beyond the Leaderboard, Kaggle's Addison Howard talks with three Orbit Wars competitors about the compute divide:

1️⃣ Isaiah Pressman simulated 15 billion steps on his way to victory. That's two to three orders of magnitude more than he could in past competitions. He says that with reinforcement learning, whatever your solution is, more compute probably makes it better.

2️⃣ Simon Jégou had the same realization, but perhaps too late given his comparatively limited compute. He reached the top 50 with rule-based agents and top 10 using imitation learning. With five days left, he started from scratch with self-play and took 2nd.

3️⃣ Gerardo Nahuel Del Toro spent his first month just fitting the game environment onto one GPU. He packed each piece of data into the smallest integer type that fit so the whole environment could run in parallel. Given the constraints, a top 2% finish was an accomplishment.

A 2020 study of 171K papers from top computer science conferences first identified the compute divide. As deep learning gained popularity, big tech and elite universities took a growing share of papers at top conferences, crowding out lower-ranked schools. The authors tied that gap to access to compute.

Orbit Wars shows it among individuals. Most Kagglers compete on their own time. Notably, the winner trained his solution on his employer's cluster.

Gerardo���s reaction (in the video) to losing to a cluster: he's not mad. Big hardware brings its own hard problems.

What's the best result you've gotten on limited hardware?

[Kaggle @kaggle]

New Bonus Prize: $10,000 for Small Model Development agents 🌟

The highest-ranking submission built on Gemma E2B or Gemma E4B wins, based on private test set performance.

• LoRA fine-tuning allowed, same as the main track
• Must be one of your team's two selected final submissions
• Still eligible for main leaderboard prizes and the Paper Track
• Public standings for eligible entries in a notebook updated daily

Same deadlines as the main track.

@GoogleResearch @googlegemma

[Kaggle @kaggle]

Post-train an open model into a coding agent that accelerates developer workflows on everyday hardware.

The Gemma 4 Developer Agent Competition, hosted by @googlegemma and Kaggle, is live!

💰 Total Prize Pool: $100,000
⏰ Entry Deadline: November 25, 2026

[Kaggle @kaggle]

Looking across 745 failed runs shows why compile checks aren’t enough:

1. 72% fail behavioral checks at runtime, where the app builds and boots, but routes disappear (404s), HTTP semantics shift, or outputs diverge from expected behavior.
2. 20% fail during build or packaging, such as leaving a packaging format unsupported by the target runtime.
3. 8% crash on startup, often when wiring from the old framework survives into the new one.

[Kaggle @kaggle]

Today, in collaboration with @IBMResearch, we are launching ScarfBench (Self-Contained Application Refactoring Benchmark), an open benchmark designed to test whether coding agents can successfully migrate enterprise Java apps across frameworks like Jakarta EE, Quarkus, and Spring while maintaining the app’s expected behavior.

Moving between frameworks takes coordinated changes across dependency injection, persistence, build systems, and deployment infrastructure. An agent can easily generate a migration that looks fine or compiles, only for the app to fail when deployed. ScarfBench puts that to the test.

ScarfBench evaluates each model with its harness over 204 migration tasks. Gemini 4 Argon with Antigravity scores 75.2%, followed by Claude Opus 5.5 with Claude Code at 70.5%, and Claude Opus 5 with Claude Code at 69.8%.

[Kaggle @kaggle]

Most agents are able to get a migration to build: 8 of the 10 setups compile 97%+ of solvable tasks, with three hitting 100%. But functional scores span more than 50 percentage points, and even when a migrated app deploys cleanly, top agents pass the full test suite only 63% to 76% of the time.

[Kaggle @kaggle]

Looking across 745 failed runs shows why compile checks aren’t enough:

1. 72% fail behavioral checks at runtime, where the app builds and boots, but routes disappear (404s), HTTP semantics shift, or outputs diverge from expected behavior.
2. 20% fail during build or packaging, such as leaving a packaging format unsupported by the target runtime.
3. 8% crash on startup, often when wiring from the old framework survives into the new one.

[Kaggle @kaggle]

Most agents are able to get a migration to build: 8 of the 10 setups compile 97%+ of solvable tasks, with three hitting 100%. But functional scores span more than 50 percentage points, and even when a migrated app deploys cleanly, top agents pass the full test suite only 63% to 76% of the time.

[Kaggle @kaggle]

Stay tuned to Kaggle’s YouTube channel!

▶️ Subscribe here: https://t.co/boZLYOEoyd

[Kaggle @kaggle]

That’s a wrap on Bench Summit by Kaggle! ✨ 

Thank you to our partners at @zapier and @GoogleDeepMind, and to everyone who joined us and made this inaugural event a success!

Stay tuned: we’ll be rolling out talk recordings and a digital track soon…

[Kaggle @kaggle]

Full talks coming soon to Kaggle’s YouTube channel!

▶️ Subscribe here: https://t.co/boZLYOEoyd

[Kaggle @kaggle]

The afternoon breakout sessions at Bench Summit have wrapped up! 🏔️

Benchmark builders and AI researchers covered topics ranging from multi-agent systems to security, for example:

• “The Utility of Multi-Agent Systems in Benchmarks” by Om Buddhdev (@sensho) from Olam Labs
• “Superhuman Evals for Superhuman Intelligence” by Xiangyi Li (@xdotli) from BenchFlow
• “We Gave 7 Frontier Models Real Businesses. They sent $12,431 in Fake Invoices, spammed thousands of emails, and made $0” by Alex Reibman (@AlexReibman) from Bottleneck Labs
• “Reproducing the Hugging Face incident” by Kevin Lin (@quasicoh) from Abundant

[Kaggle @kaggle]

@zapier, one of our Bench Summit partners, led a panel discussion on the realities of managing highly influential evaluation signals.

The panel, moderated by Lukas Bergstrom (@lukasb), included insights from Robin Salimans (@SalimansRobin) of Zapier, Greg Kamradt (@GregKamradt) of ARC Prize Foundation, and Shaiyon Hariri of Ramp.

Attendees heard about what defines a successful benchmark, the challenges of maintaining adoption, and critical lessons learned from V1 to V2+.

[Kaggle @kaggle]

Bench Summit by Kaggle has officially kicked-off!

Trust in benchmarks is eroding. The path forward starts here.

We started strong with @janiejgu sharing key insights on the problems with benchmarks today and Kaggle’s mission to build trust in evals and help the world discover what works and doesn’t work in AI.

During the session, she shared four ways benchmarks fail today:
1. Relevance: Many public benchmarks don’t resemble what your agents do in production.
2. Saturation: Evals go stale within a release cycle.
3. Verification: Everyone has their own implementation of the same benchmarks, meaning scores vary wildly due to infrastructure, not model quality.
4. Infrastructure: Evals infra is hard to build correctly, and everyone rolls their own.

To get numbers you can actually trust, Janie highlighted four essentials:
1. Real workflows as the foundation for benchmarks
2. Continuous benchmarks from crowdsourced tasks, arenas, and production data
3. Rigorous and transparent methodology audited by the community experts
4. Eval infrastructure that everyone can use out of the box

Here’s a sneak peak of the early moments: