Ollie Jaffe

{eval, safety}/acc

[Ollie Jaffe @JaffeOliver]

"~10% of ICLR 2025 papers have (significantly) incorrect results" is a fun paper begging to be written:
1. Scrape all ICLR codebases for papers (+check they reproduce paper)
2. Use CC/codex to automatically review+fix+re-run

I researched this in Jan but no capacity to finish

[joe @simulated_land]

https://t.co/rlvMJswbIv

[Ollie Jaffe @JaffeOliver]

Autonomous orgs/sub-teams are right around the corner - shocking diff in perf of multi-agent vs. best-of-N single-agent on >100h tasks. Just put multiple CC in the same slack

Feels like there is massive underelicitation across many evals

I would worry if your moat is SWE

[Ollie Jaffe @JaffeOliver]

@ChowdhuryNeil @cognition Sorry just saw this, can't just DM you a script sadly since I'm using a bunch of internal infra. But it is just a matter of cost

[Ollie Jaffe @JaffeOliver]

GLM-4.6 performs about the same as SWE-1.5 on SWE-Bench Pro. I evaluated GLM-4.6 myself this morning, just giving it simple bash tools.

If the rumours are true and SWE-1.5 is a descendant of GLM-4.6, this is embarrassing for @cognition.

[Cognition @cognition]

Today we’re releasing SWE-1.5, our fast agent model.

It achieves near-SOTA coding performance while setting a new standard for speed. Now available in @windsurf.

[Ollie Jaffe @JaffeOliver]

@ChowdhuryNeil @cognition I'd be happy to eval gpt-5-high but alas it is expensive

[Ollie Jaffe @JaffeOliver]

@cognition Setup for evaluating GLM4.6. Reasoning enabled. 300 randomly selected tasks (verified this is representative of using full set). Tools for executing arbitrary bash commands, viewing files, editing files. Only one seed because that's what SWE-1.5 authors seemingly did :)

[Ollie Jaffe @JaffeOliver]

@cognition The SWE-1.5 release feels suspicious. Why did they only release scores on a single eval? Why didn't they report confidence intervals? Why didn't they disclose the model they finetuned from? And why does GLM-4.6 even outperform gpt-5-high on this benchmark?

[Ollie Jaffe @JaffeOliver]

Did some PaperBench and MLE-Bench runs with C4.5. Seeing mixed results. Seems only slightly better than C3.6 on PaperBench. On MLE-Bench it seems good but sometimes "flew blind" by deciding not to measure/report any evaluation metrics of the models it trained. Doing more runs now

[Ollie Jaffe @JaffeOliver]

However, SWE-Bench style tasks are a pretty bad way of evaluating SE capabilities. We don't have many better alternatives right now, but I'm much more excited by new evaluation methods rather than "SWE-Bench but harder+cleaner" so we actually measure the right thing

[Ollie Jaffe @JaffeOliver]

Generally impressed and on initial take would recommend this over SWE-Bench Verified. Complaints after reading ~15 tasks are 1) high variance in test quality; often tests only evaluate small subset of expected changes, 2) authors poorly documented their verification process

[Bing Liu @vbingliu]

🚀 Introducing SWE-Bench Pro — a new benchmark to evaluate LLM coding agents on real, enterprise-grade software engineering tasks.

This is the next step beyond SWE-Bench: harder, contamination-resistant, and closer to real-world repos.

[Ollie Jaffe @JaffeOliver]

@thelokasiffers This this flew under the radar, you might like
https://t.co/lOe1zDNkao

[Ollie Jaffe @JaffeOliver]

@thelokasiffers Good, recent, 10 pages!
https://t.co/wE3XeqEIKT

[Ollie Jaffe @JaffeOliver]

Short timeline believers point to METRs time horizons results as evidence, but HCAST tasks are not representative enough of the most useful lab labour
- Not messy enough (~3/16)
- Well-specified
- Few steps required to solve many of the tasks (below)

OPQA seems good

[Ollie Jaffe @JaffeOliver]

Short timeline believers are in shambles right now
- o1-preview was a monster and we haven't seen much agentic uplift since then on complex problems that require >100s steps (see MLE-Bench, PaperBench)
- C4.1 was a slight regression on their reported autonomy evals

[Ollie Jaffe @JaffeOliver]

@thelokasiffers At least we finally have actual tests. Vibe testing can only take you so far

[OpenAI @OpenAI]

We’re releasing PaperBench, a benchmark evaluating the ability of AI agents to replicate state-of-the-art AI research, as part of our Preparedness Framework.

Agents must replicate top ICML 2024 papers, including understanding the paper, writing code, and executing experiments.

[Ollie Jaffe @JaffeOliver]

christmas detox week. no caffeine, internet, or ayahuasca

[Ollie Jaffe @JaffeOliver]

@evanon0ping Final boss of waymo racing

[Ollie Jaffe @JaffeOliver]

@matthewclifford Free EF advertising!