We appreciate the community's feedback on SWE-Bench Pro.
Much of it maps to changes already underway in v1.1, which we've been building for a while. Keeping evals current with frontier models is hard, and we're always iterating.
SWE-Bench Pro Verified coming soon. 👀
I was clearly wrong about Anthropic. They are obviously currently the leader in AI. No company has released a model as good as Mythos/Fable and they will undoubtedly have Mythos 2 ready soon.
And I would never cut them off in a way that hurt them badly, even as a competitor. That’s not my style.
Tesla open sourced its patents and we made the Supercharger network available to all competitors, even though we could have made it a walled garden.
SpaceX launches competing satellite systems with no increase in price or use of unfair terms.
Even my worst enemies can attack me on this platform.
…
this isn't to invalidate any effort the teams have put into organizing those artifacts or tools, moreso trying to find a more objective read on what is different here for my own learning
been thinking about this throughout the day and i'm now a bit skeptical as to how sol training luna is different from a pretraining engineer giving the tools needed to any model with a proper scaffold to iterate (similar to the plethora of auto research frameworks)
or any /goal directive with the appropriate tooling/context/datastores and a box to run compute on
unless im missing some nuance, given sufficient local context on how pretraining/midtraining works, the model training another model still started with human organized artifacts
if you’re looking for an unconventional alternative to staring at a wall for an hour or touching grass to reset your screen-wired brain, go to goodwill and look through all the racks in your section
i promise you’ll be itching to go back to your computer after with ideas firing
using less tokens (cheaper) to get the same level of work done (if not better) is as much of a metric of intelligence as the end score is
comes back to the idea of being able to communicate an idea in as few words as possible as a marker of deep understanding
the 5.6 blogpost is v satisfying to read, lots of interactive bits to let you see the outputs yourself, v curious to test out the long context usecases - cautiously optimistic but scores showcase higher than opus rn (also abstract reasoning seems much better)
@matt_slotnick every post i see about sol sums to it being a diligent worker you can trust and fable being the insightful/intelligent lead, feels like it tbh