We audited SWE-Bench Pro, one of the most widely used AI coding benchmarks, and found it no longer reliably measures frontier coding capability.
We find 30% of SWE-Bench Pro tasks to be broken, and are retracting our previous recommendation that the research community use it as a leading coding eval.
https://t.co/wDdSEjBe4F
@joshwoodward@GeminiApp “Gemini stop acting like a sycophant”
“Okay!”
.
.
.
“You’re absolutely right”, “this is a masterclass”, “this is a world class”…
Cmon man