Kilian Lieret

Meta Superintelligence, prev. Princeton. SWE-bench multilingual/multimodal, SWE-agent, mini-swe-agent, SWE-smith, CodeClash, ProgramBench

[Kilian Lieret @KLieret]

John is literally the goat of benchmarks, hear him talk about ProgramBench, CodeClash and SWE-Bench

[vincent sunn chen @vincentsunnchen]

Kudos to the ProgramBench team! + @KLieret (co-lead)  @18jeffreyma @parth007_96 @dpedch @sten_sootla @micmylin @pengchengyin @magpie_rayhou @syhw @Diyi_Yang @OfirPress

YouTube here: https://t.co/01Bsd3SxHQ

[Kilian Lieret @KLieret]

ProgramBench mascot IRL. Couldn't find a gray fox plush

[Kilian Lieret @KLieret]

@Cypher_Samurai @SarahLacard @xeophon @OfirPress @jyangballin @18jeffreyma So the agent runs within the docker container so there's also no user in a docker group (there's no docker in docker installed)

[Kilian Lieret @KLieret]

@SarahLacard @xeophon @OfirPress @jyangballin @18jeffreyma the agents were running as a non-root user in the container and I haven't seen any access escalation hacks

[Ofir Press @OfirPress]

Anyone who has spent more than 30 seconds running frontier models on tough benchmarks knows that they like finding ways to cheat. Here's the most creative method we caught an agent using to cheat on ProgramBench.
w/ @jyangballin @KLieret @18jeffreyma

[Ofir Press @OfirPress]

@jyangballin Full ProgramBench Q&A: https://t.co/4oKq27BZf2

Full benchmark at https://t.co/CphwwpKzmf

[Ofir Press @OfirPress]

John (@jyangballin) talking about the wide behavioral differences between GPT and Claude on ProgramBench.

[Ofir Press @OfirPress]

Kilian (@KLieret) on why the initial 0% top scores on ProgramBench also surprised us

[Kilian Lieret @KLieret]

@Jokerinfina @jyangballin will update early next week

[Kilian Lieret @KLieret]

One of my favorite things from the Anthropic system cards are the examples of strange model behavior. This one is on frustration in chain of thought reasoning (and seems to have been largely resolved)

[Kilian Lieret @KLieret]

I like this specific study from the Opus 4.8 model card: How often is the model lazy and (incorrectly) guesses program behavior without actually checking it by tracing through the whole call stack. Definitely have had that happen before

[Kilian Lieret @KLieret]

SWE-bench multimodal is still very hard (numbers from Anthropic system cards)

[Kilian Lieret @KLieret]

@stalkermustang it would still not be very fair, because we don't really know what agent setup they were running. They talk about "episodes" on the left plot, so they probably reran an agent several times. We're planning to update our leaderboard next week

[Kilian Lieret @KLieret]

Very cool to see ProgramBench scaling charts for Opus 4.8! The % hidden tests passed is not the "official" metric, but it makes sense for these studies (though I generally consider it to be misleading for the overall benchmark)

[Jeff Ma @18jeffreyma]

hyped to see our work on ProgramBench make it in the Opus 4.8 model card!

[Kilian Lieret @KLieret]

ProgramBench is a joint effort across Meta FAIR, Meta TBD, Stanford, Harvard

@jyangballin (co-first author) @18jeffreyma @parth007_96 @dpedch @sten_sootla @micmylin @pengchengyin @magpie_rayhou @syhw @Diyi_Yang @OfirPress

[Kilian Lieret @KLieret]

Very interesting study from Opus 4.8 card: Multi-agents do not deliver better results on ProgramBench, but they get to mediocre solutions 2x faster.

[Kilian Lieret @KLieret]

ProgramBench is open source at https://t.co/1Sagj6kK6m

[Kilian Lieret @KLieret]

ProgramBench is open source at https://t.co/1Sagj6kK6m

[Kilian Lieret @KLieret]

More on metrics: https://t.co/p2uQUIbHOE