Benjamin Ou

Benchmarking with games at @EpochAIResearch. Opinions are my own. Send me your weird and esoteric LLM benchmarks!

[Jaeho @enthusednotebk]

Excited to have worked on this! AI detectors will continue to be super important in the slop-filled world we're heading towards. Overall super happy about the low FPR, and some takeaways in the thread: (1/n)

[Benjamin Ou @AlephNuul]

@enthusednotebk what happened to your concern about physical labor disenpowerment!

[Benjamin Ou @AlephNuul]

@ChaseBrowe32432 @Lone_Hawke I do recall that some people tried games with denser interfaces like Paradox stuff, and the models struggled with finer details and more finicky interfaces. Personally I'm curious whether there's also a drop in performance if the game isn't as well-known.

[Benjamin Ou @AlephNuul]

@ChaseBrowe32432 @FriesIlover49 Dang, are you using a specific harness or is this out of the box computer use? I had Sol beat slay the spire a0 with its out-of-the-box harness from ChatGPT desktop last week.

[Benjamin Ou @AlephNuul]

@bigseb31213 It is absurd, yes, and the nerds at labs are horrible at politics and comms, but it also reflects that they really do believe in this stuff.

[Benjamin Ou @AlephNuul]

@tmkadamcz @alexolegimas ...intelligence doesn't solve, because the root cause was never a lack of intelligence.

[Benjamin Ou @AlephNuul]

@tmkadamcz @alexolegimas I guess the analogous single payer pathology is doctor strikes, though I hear about them happening in Korea more than the UK.

In any case, I imagine the actual Imas rejoinder is that waste and cruft from institutional idiosyncrasies is exactly the sort of problem more...

[Benjamin Ou @AlephNuul]

@fleetingbytes what are some things that work and things that don't? for me,

thing that works: distinct identity from most of bay area suburbia
thing that doesn't: housing (duh)

[Benjamin Ou @AlephNuul]

Cautiously willing to call this a trend break in gaming skill, and therefore reasonably general strategic decision making. We lack data on older models but I think other gaming benchmarks like Runebench are pointing in this direction, I think.

[Benjamin Ou @AlephNuul]

@emollick seems Sol can beat vanilla a0 StS out of the box as well, with codex desktop + computer use. Its quality of play is definitely significantly higher than what I've seen from 5.5, and this is consistent with some strong results from Sol on EBR-bench.

[Benjamin Ou @AlephNuul]

@ModeledBehavior I mean apparently grok was the happiest to help Boko Haram

[Benjamin Ou @AlephNuul]

@TNOQuoProQuid I'll rec blades in the dark here; the flashback mechanic and heist structure really tries to tamp down on over planning and rewards cool improv.

[Benjamin Ou @AlephNuul]

@emollick Not to downplay this! Huge leap in computer use, vision, long-term planning, all those things. Just trying to get a sense of whether there's a huge trend break here; gonna try running this experiment on my laptop with StS1 now

[Benjamin Ou @AlephNuul]

@emollick Oh hm, I wonder if draft is what did it (relative to LLMs struggling on StS1). Starting with a deck of non-starter cards should make it significantly easier to snowball in theory?

[Ethan Mollick @emollick]

This was one of those impressive AI thresholds for me. I gave GPT-5.6 Sol in Codex control over my computer, and asked it to win the daily challenge for the game Slay the Spire 2 (randomized factors, so can't cheat).

It worked for 5 hours, making complex game choices... and won.

[Benjamin Ou @AlephNuul]

@emollick ...challenge is remotely comparable to StS1, this is either a massive trend break or there's something unusual going on.

[Benjamin Ou @AlephNuul]

@emollick I'm not as familiar with StS2 as StS1; what was the daily modifier here? These decks seem unusually large, how were they functioning consistently?
This is a pretty unusual result that could use some replication; GPT-5.5 and Fable could not beat a0 StS1, so if this StS2...

[Qokedas @Qokedas]

Can today’s frontier models reliably decode data from a real scientific instrument?

We gave four of them a raw recording from a 1,024-microphone acoustic camera and asked for heatmaps showing where the sound comes from.

[Greg Burnham @GregHBurnham]

We’ve run Fable 5 (max) and Sol 5.6 (max) on EBR-bench. Both get new high scores, but they achieve this with better out-of-the-box play, not learning from repeated attempts. Here's some commentary beyond the headline.