Ryan Brown

founding FDE @sandstonehq // prev: applied AI @blackstone

i write about AI engineering, evals, and coding agents

[Ryan Brown @NotRyanB]

SlopCodeBench is the most relevant coding agent benchmark left--the top score is 16% vs. 50-75% on other coding benchmarks like Terminal-Bench, FrontierCode, and DeepSWE

summary of how it works: an agent receives a spec for a small program, like a backup scheduler or a mock HTTP server. the agent implements it, then the agent receives a new spec and has to extend the program. this repeats 3-8 times, and with each new spec there are hidden tests that check the new feature and all the old features.

there are a few reasons why I really like it:
1. it comes far closer to simulating real, long-horizon work with new and updating requirements, which is what it’s actually like to work on a codebase vs. solving a single isolated task
2. code coherence, maintainability, and extendability naturally matter more
3. it’s significantly less saturated than the other coding agent benchmarks

full results in the image (credit to @dexhorthy for the benchmark run + charts), the “core” number measures whether the main tests for the new feature pass; the “strict” number is the headline and measures whether every test passes, including regression tests for all earlier features

astra and sol are at the top (interestingly 6.1 is slightly worse than 6 although likely just noise), but given how low the scores are overall + how small the spread is, i don’t think it’s super meaningful yet in terms of determining which model you should use--opus 5.5 is still my daily driver, and i’m kind of surprised it’s not higher

the big caveat imo is that these runs all used the base harness with no customization. i personally think having a couple of core skills + agents.md instructions (planning, verification, etc) is necessary and improves performance, and that improvement is much more likely to show on SlopCodeBench than the other benchmarks

my prediction is that custom context + skills would improve performance by 5 percentage points or more and (hopefully) result in a ranking that’s a better representation of how good different models are to actually use for engineering work

do you think the improvement from proper skills/verification would be more or less than 5%?

[Ryan Brown @NotRyanB]

@Bevoladee2 hi -- i write about coding agents and ai engineering. would love to connect

[Ryan Brown @NotRyanB]

@maccivanX hey -- building sandstone (series a legal ai) as an fde. i write about evals and coding agents. happy to connect

[Ryan Brown @NotRyanB]

@foundermichiels hey -- building sandstone (series a legal ai) as an fde. i write about evals and coding agents. happy to connect

[Ryan Brown @NotRyanB]

@Tornikenizh hey -- building sandstone (series a legal ai) as an fde. i write about evals and coding agents. happy to connect

[Ryan Brown @NotRyanB]

@vkryukov they focus more on big law than in-house legal (which we have a lot more experience in) so we feel pretty good

[Ryan Brown @NotRyanB]

@Sohaib_Ahmad01 @X hey -- i write about ai engineering and coding agents. let's connect

[Ryan Brown @NotRyanB]

@CHURROCHPF hey -- fde at sandstone, i write about ai engineering and evals. happy to connect

[Ryan Brown @NotRyanB]

@k_d_patel007 hey -- building sandstone (series a legal ai) as an fde. i write about evals and coding agents. happy to connect

[Ryan Brown @NotRyanB]

@hemantarya98 @X hey -- building sandstone (series a legal ai) as an fde. i write about evals and coding agents. happy to connect

[Ryan Brown @NotRyanB]

@AI_Nalysis @X hey -- i write about ai engineering, evals, and coding agents. would love to connect

[Ryan Brown @NotRyanB]

@audiee_ai @X hey -- i write about ai engineering, evals, and coding agents. would love to connect

[Ryan Brown @NotRyanB]

@Luoxiaomac hey -- building sandstone (series a legal ai) as an fde. i write about evals and coding agents. happy to connect

[Ryan Brown @NotRyanB]

@veeraj_ hey -- i write about ai engineering and coding agents. let's connect

[Ryan Brown @NotRyanB]

@diceai0 hey -- building sandstone (series a legal ai) as an fde. i write about evals and coding agents. happy to connect

[Ryan Brown @NotRyanB]

@AbhinavTechh hi -- i write about coding agents and ai engineering. would love to connect

[Ryan Brown @NotRyanB]

@nickytoh hey -- i write about ai engineering and coding agents. let's connect

[Ryan Brown @NotRyanB]

@whoisjorel hey -- building sandstone (series a legal ai) as an fde. i write about evals and coding agents. happy to connect

[Ryan Brown @NotRyanB]

@ZakariaaZouitni hey -- building sandstone (series a legal ai) as an fde. i write about evals and coding agents. happy to connect

[Ryan Brown @NotRyanB]

@E_B_U_K_A_H_ hey -- building sandstone (series a legal ai) as an fde. i write about evals and coding agents. happy to connect