Gabe Orlanski

PhD student @WisconsinCS | Intern @SnorkelAI & Prime Research Resident

Former intern @replit / @magicailabs / @Google

[Gabe Orlanski @GOrlanski]

Very excited to announce the v1.0 of SlopCodeBench release:
- Doubling the size of the dataset
- @harborframework support
- scb-check: a CLI that flags slop anti-patterns
- Way more model results

https://t.co/RQkB8wdzAu
https://t.co/36qQR3azeE

🧵

[vincent sunn chen @vincentsunnchen]

will be speaking.. come say hi 👋 (thanks @FrancoisChauba1 for organizing!)

after a decade of data research from @StanfordAILab to @SnorkelAI, I'll share what's changed (individual labels → full-scale environments) and what's stayed fundamental across every paradigm shift

[Henry Kiss Ehrenberg @henryehrenberg]

Just shipped a few tune-ups to the Senior SWE-Bench leaderboard, and somehow there's a 3-way tie for first: Fable 5, Opus 5, and GPT-5.6 Sol.

Yes it's sus. But we've verified it!

New blog dives into differences in coverage (pass@k), reliability (pass^k), perf-per-$, and more.

https://t.co/WVCJgnPwOw

[Gabe Orlanski @GOrlanski]

Never thought I would be refreshing to see new results with a benchmark we made.

Really great work by Dex on putting new models through there paces!

[dex @dexhorthy]

benchmarking opus 5 on slopCodeBench - we are part way into the eval run - will keep posting status updates as we go - so far (no surprise) sonnet 5 slower and more expensive than opus 4.8 and opus 5

Checkpoint 1, problem one:

Opus 4.8 - 7 turns
Opus 5 - 11 turns
Sonnet 5 - 33 turns and counting 😅

more results as they stream in

[Gabe Orlanski @GOrlanski]

@Michael_WCD @dexhorthy All of the instructions themselves are hand written. A lot of the overall problems are even based on real repositories.

We tried to make these specs as close to real prompts as possible but there is definitely room to improve!

[Henry Kiss Ehrenberg @henryehrenberg]

Opus 5 debuts at #2 on Senior SWE-bench, just behind the leader Fable 5.

Opus 5 improves on performance-per-compute, achieving its score at 32% of the output tokens of Fable 5 on average.

It also excels on senior-level bug investigations, taking the #1 spot across all models.

[Alex Shaw @alexgshaw]

While building Terminal-Bench 2.0, we realized releasing one benchmark every 6 months wasn't going to scale.

So we took everything we knew about building great benchmarks and shipped it as @harborframework.

There is a disconnect between benchmark scores and reality, because a handful of benchmarks cannot capture the full range of capabilities and the nuance of great vs. mediocre work within a specific domain.

Instead, we need thousands, or millions of benchmarks, built by people who deeply understand the capability they want to measure. I believe Harbor is the way to make this happen.

ReactBench is a great example of this. Despite SOTA scores on Terminal-Bench, this more targeted benchmark shows how bad the models are at writing correct and performant React code (which is literally one of the most popular use cases of coding agents).

Every company should be building its own benchmarks. If you don't, you're blind and you have to trust brand and pay through the nose for "frontier" tokens that might not even be frontier on your use case.

(P.S. once you have your benchmark (test split) the next step is optimization (train split). We'll have more to say there soon.)

[Henry Kiss Ehrenberg @henryehrenberg]

Grok 4.5 and GPT-5.6 Sol join the Pareto frontier on Senior SWE-bench, and there's a clear trend towards efficiency.

GPT-5.6 Sol: Opus 4.8 perf @ 40% of the cost
Grok 4.5: GPT-5.5 perf @ 25% of the cost

Grok 4.5 climbed to #2 on senior-level bug solving but for just ~$1 / task

[Nicholas Roberts @nick11roberts]

Excited to share that Train-to-test (T^2) scaling was accepted to COLM! 🌉

We show that when you factor test-time scaling into pretraining scaling, extreme overtraining becomes compute optimal.

Check out our paper below! 👇

[John Cooper @jfrcooper2]

Excited to be presenting an oral and poster at #ICML2026 about hybrid models and their capabilities! There have been many empirical results, but far too few theoretical ones explaining their expressivity. We show there are tasks with a separation! Come chat! Oral 6A, poster 4622.

[Henry Kiss Ehrenberg @henryehrenberg]

Claude Fable 5 results are in! It's the new Senior SWE-bench leader at 27.9%, 3 pts above Opus 4.8 (the previous #1).

Fable 5 excels at open-ended feature tasks, improving 35% over Opus 4.8. It's also more expensive: 8x more output tokens than GPT-5.5 on avg. And we observed some capability jumps that we didn't fully expect.

[Avi Trost @atrost3122]

Nested models let you train a whole family of submodels at once. What if you could use them all at once, too?

Block triangular weights enable this structure.

It gives us token-adaptive routing, self-speculative decoding, and more.

Introducing: Fully Nested Transformers (1/9)

[Tzu-Heng (Brian) Huang @zihengh1]

LLM-as-a-judge is now everywhere for automated evaluation. But it can be slow, expensive, and opaque. What if we ask the judge for its rubric once, and execute that logic as a program? Introducing PAJAMA—a new hybrid evaluation system that pushes the LLM-judge Pareto frontier! 🚀

[Gabe Orlanski @GOrlanski]

An agent's poor taste is probably 95% of the issues I run into with agents. Senior SWE-Bench measures exactly this with realistic requests that give the agent the freedom to mess up. Was a lot of fun to be a part of.

Worth reading @henryehrenberg's amazing work here!

[Henry Kiss Ehrenberg @henryehrenberg]

We expect agents to act like senior engineers, but most benchmarks still evaluate them like interns.

Excited to introduce Senior SWE-Bench, an open-source and @harborframework-native benchmark that assesses agents as senior engineers on long-horizon tasks with realistically under-specified instructions.

We expect agents to build real features going on just a quick Slack message, nothing like the super technical instructions most benchmarks provide. Senior SWE-Bench fixes that.

Claude Opus 4.8 is the current leader at 24% high quality solves, but it took 117K tokens on average to get there. Claude Sonnet 5 looked like it was going to swoop in for the top spot, but we found it cheated on 26% of trials.

[Parth Asawa @pgasawa]

I'll be at ICML July 7th-10th, hit me up if you want to chat about continual learning, AI policy, etc!

I’m giving an invited talk on evaluating continual learning at the CATS workshop on July 10th at 8AM KST and @aczhu1326 and I are presenting a poster on Advisor Models on July 8th at 10:30AM KST (Hall A #2107).

[Braden Hancock @bradenjhancock]

T minus one hour until https://t.co/3vL0YST0b2 — one room with the top 100 researchers driving AI progress in the open. The attendee list looks like a joke, because you couldn’t seriously expect that many icons to gather in one place. I’m excited for literally every segment, but the working sessions especially. Join us on the livestream!

[vincent sunn chen @vincentsunnchen]

capability != learning

new benchtalks with @pgasawa on continual learning, where we discuss teaching models to learn from experience, measuring learning ability, the bet on parametric models, and more

01:06 What is continual learning?
04:10 Why capability and learning are different
06:13 Why build a benchmark?
08:07 Continual Learning Bench launch and reception
09:13 Anthropic's Fable release and Continual Learning Bench
11:02 How to design tasks for continual learning
18:41 The gain metric
24:01 What good looks like on the leaderboard
29:13 Failure modes: why models can't update their beliefs
31:12 Parametric systems and future architectures
34:30 Open science and AI safety
45:42 Lightning round
49:08 How to contribute to Continual Learning Bench

[Gabe Orlanski @GOrlanski]

@zebassembly Making a pre commit tool that flags these is the only consistent way I have found. Even then it is hard to flag exactly these statically. There are cases where it makes sense to have a smallish function

[Braden Hancock @bradenjhancock]

Hard agree: we absolutely need more detailed visions of what AI going well for the species looks like and how to get the right funds to the right orgs fast enough to pull it off.

Obviously I'm biased here, but @LaudeInstitute's recently launched Moonshots program (https://t.co/XL2zX9z1pO) feels pretty relevant to this line of thinking. Philanthropically funded, technology focused, executed by former startup founders & operators and a Turing award winner, structured to be competitive/efficient, etc.

[Gabe Orlanski @GOrlanski]

@teortaxesTex It is really, really cost-effective. Very surprising, especially compared to the base Kimi K2.5. K2.6 def gets close in terms of capabilities, but Kimi's harness seems not to be as good https://t.co/gNvFxR3a7H

[Gabe Orlanski @GOrlanski]

Doubling the size and running more models has shown a quite interesting Pareto frontier for cost vs solve rate.

The GPT family is dominant. Opus 4.7 marks a massive efficiency jump for Anthropic. And to my surprise, @cursor_ai's Composer-2 is a very cost-effective model.

[Gabe Orlanski @GOrlanski]

@raw_works @lateinteraction @a1zhang @xeophon @omouamoua @RLanceMartin SCbench could be a good setup for RLMs. https://t.co/mMtC8cbQGi

Or Continual Learning Bench as well: https://t.co/LjttMtwPyr

Especially on the latter as more long horizion/long context tasks get added

[Parth Asawa @pgasawa]

Today, we’re releasing Continual Learning Bench 1.0: the first, realistic benchmark for measuring how AI systems can improve in online settings.

Benchmarks today assume models are stateless. Each example is independent, and once a system finishes a task, it moves on as if nothing happened.

But deployed AI systems should learn from experience. We tested 10+ frontier systems against novel, expert-validated tasks and find there’s still plenty of headroom for learning. (1/n)