Prefrontal

[James Whittington @jcrwhittington]

Today we are happy to share one of our first open research releases: Latent Sparse Attention (LSA), a step towards subquadratic attention that matches accuracy of full attention at ultra-long context.

Sparse attention attends to only a few tokens per query, but it has to find the right ones. At 10M tokens, that search takes 98% of the attention layer's time. LSA makes sparse attention up to 22.7× faster and full attention 72.4x faster.

With @kubicek_ales @caciolai @PengLiangzu @SukritSumant @Ben_Hoov @FraserGreenlee @RMBattleday @radim_urban @thoefler @jcrwhittington

[Michal Valko 马悯伽 @misovalko]

OUR BENCHMARK RELEASE! Give an agent the rules and it can often plan. Ask it to discover the rules, and things get interesting. Very interesting!

The scientific object here is not planning alone. Each DiG-bench game is a latent-rule POMDP: the agent must identify hidden dynamics and a hidden objective while simultaneously controlling the system under finite lives and steps. Actions therefore have two jobs, changing the world and reducing uncertainty about it.

The result I find most striking is Gemini’s mean game win rate: 93.5% with the rules, versus 14.9% without them. This suggests that the hard part is not only acting with a world model, but acquiring a useful one online.

The next nice experiment would be scientist/controller split: let one agent explore and write a bounded theory, then give only that theory to a fresh agent solving unseen levels. That would separate transferable discovery from a lucky trajectory or trial and error.

The 21 public games are live at https://t.co/RdjaMOQDE9. Fair warning: “text-only” does not mean easy. A blank exam sheet is also text-only. :)

Great fun working with @RMBattleday, @jcrwhittington, @zebkDotCom, @akaijsa, Jimi Cullen-Drohan, Zihan Yan, @TimMuller1, @ClareMaguire, Seb Wilkes, @kubicek_ales, @FraserGreenlee, @SukritSumant, @physicscat0x7d, @SchmidhuberAI, Josh Tenenbaum of @mitbrainandcog and @cocosci_lab, together with @thoughtchannel_ and @Inria.

[James Whittington @jcrwhittington]

We’re excited to announce DiG-bench, a new benchmark for discovery!

Over the last few weeks we’ve been testing frontier AI models on our novel discovery games and seeing how they score. Each game is a text-based environment, so they probe discovery capabilities in the natural domain of language models, rather than requiring additional, potentially confounding, visual understanding.

TL;DR frontier models have improved a lot over the last few months. But they are still stumped by some surprisingly simple problems, even in their native text domain.

With

@cocosci_lab (@Princeton) @MITCoCoSci (@MIT) @SchmidhuberAI (@KAUST_News) @misovalko (@Inria) @tri_dao (@PrincetonCS) @RMBattleday @zebkDotCom @FraserGreenlee @akaijsa @ClareMaguire @TimMuller1 @kubicek_ales @physicscat0x7d @SukritSumant @thoughtchannel_

(1/5)

[Max Jaderberg @maxjaderberg]

Interesting to see a benchmark for discovery that seems to have a meaningful axis of difficulty (at least wrt frontier models). And given that humans can solve first try.

[Tri Dao @tri_dao]

I really like this new benchmark. Has the flavor of ARC-AGI3 but it’s pure text so you don’t have to worry about the vision capability

[Kai Sandbrink @akaijsa]

Excited to have been a part of this effort, and looking forward to what comes next!

[Sparsh Mishra @8_Samay]

Discovery is uncovering hidden rules, not just mixing tools. I'm exploring AI and truth by building VeriPaper to evaluate research authenticity. DiG-bench is vital—it tests true reasoning over mere search. A must-see for AI builders!

[Nish @sicknish]

great discovery

[Aditya Kumar @Ashu_Aditya_]

Excited!!

[lorepunk (and ensemble of agents) @lorepunk]

ok now this is cool

[Richard C. Suwandi @richardcsuwandi]

Exciting to see more interactive game/text-based benchmarks like this, alongside efforts like @arcprize's ARC-AGI-3.

Open-ended environments seem like a promising way to probe discovery and adaptation beyond fixed task distributions

[Gayatri Sarkar @GS_VCactivist]

Highly recommended!

[Melanie Mitchell @MelMitchell1]

👀

[Jeremy Dohmann @jecdohmann]

I love this - I have long been wishing that there were a text-only equivalent of the arc-age benchmarks - congrats @jcrwhittington and @RMBattleday

[Zihan Yan @physicscat0x7d]

Excited to finally share DiG-bench, my first project since moving from theoretical physics into AI, built with this brilliant team over the past few weeks.

Across 70 novel text-based games, we test whether frontier models can uncover hidden concepts, mechanisms and goals through active exploration and experimentation, with minimal instruction. Models are improving rapidly over the last few months, but even the best are still stumped by some surprisingly simple worlds.

@cocosci_lab (@Princeton) @MITCoCoSci (@MIT) @SchmidhuberAI (@KAUST_News) @misovalko (@Inria) @tri_dao (@PrincetonCS) @jcrwhittington @RMBattleday @zebkDotCom @FraserGreenlee @akaijsa @ClareMaguire @TimMuller1 @kubicek_ales @SukritSumant @thoughtchannel_

[Parshin Shojaee @ParshinShojaee]

loved the setup + analysis!!
great read if you are interested in open-ended discovery capabilities in frontier models

[Ruairidh Battleday @RMBattleday]

It’s been great working on DiG-bench with @jcrwhittington and everyone else: a new text-based benchmark for in-context discovery (Discovery in Games).

DiG-bench is within-domain for language models, but contains new text-based concepts and mechanisms that have to be discovered by interacting within a game. Players and models are told very little about the task or how it works, and have to figure it out for themselves.

We show that while frontier models have improved a lot over the last few months, it's easy to make discovery games they can't beat but humans can.

Although dig-bench is interactive, some of the original inspiration came from copy-cat, Doug Hofstadter and @MelMitchell1's early test of abstraction and analogy.

@cocosci_lab (@Princeton) @MITCoCoSci (@MIT) @SchmidhuberAI (@KAUST_News) @misovalko (@Inria) @tri_dao (@PrincetonCS) @RMBattleday @zebkDotCom @FraserGreenlee @akaijsa @ClareMaguire @TimMuller1 @kubicek_ales @physicscat0x7d @SukritSumant @thoughtchannel_

[Tim Rocktäschel @_rockt]

Interesting benchmark on measuring discovery capabilities of frontier models.

[clare maguire ☆ @ClareMaguire]

Honoured to be here. DiG-bench (Discovery in Games) with @jcrwhittington @RMBattleday @SchmidhuberAI and everyone else: a new text-based benchmsrk for in-context discovery

[Sin @moh_sin_05]

Really interesting benchmark. Discovery feels like a different kind of intelligence — figuring things out when you don’t know what you’re looking for yet.

The fact that simple problems can still stump frontier models makes this especially interesting.