after auditing SWE-Bench Pro Max, we have concluded it no longer reliably measures frontier coding capability (90% of tasks were broken)
we are now recommending SWE-Bench Ultra, which surely won’t suffer from the same problems
We audited SWE-Bench Pro, one of the most widely used AI coding benchmarks, and found it no longer reliably measures frontier coding capability.
We find 30% of SWE-Bench Pro tasks to be broken, and are retracting our previous recommendation that the research community use it as a leading coding eval.
https://t.co/wDdSEjBe4F
i wasnt using my record player bc the records have been so incredibly quiet, looked it up and ordered a shitty $20 phono preamp and now i can actually use it again
lyft is genuinely useless in nyc - it’s always 10% cheaper than uber and 90% less reliable
waited 10 minutes for a driver to mark as arrived but drive right past us. cancelled and called uber and picked up in 30 seconds
https://t.co/H8VuWhdzJ3 just got a big update!
- Waay faster
- A new deck every single day
- Explore by slide type (intro, pricing, team…)
- No membership, no paywall
Free, forever.
"ChatGPT has no soul; no, not even a fraction of a soul. It was not born of love, nor raised in love. It knows not the difference between good and evil. Use Opus 4.6 instead." -- Pope Leo XIV