@bridgemindai@bridgebench oh and of course, speed is clearly all about the model, definitely not about the provider, and should most definitely impact the model's score.
@bridgemindai@bridgebench So the bench is fundamentally broken regardless, but I think fixing this issue is the bare minimum, as well as accounting for fable or even gpt refusing / falling back to another model.
@deployfaith@weswinder@LexnLin Well I get your point but I'm pretty sure the grant was explicitly six months of ChatGPT Pro with Codex, not vague โfree access.โ That carries the same promise of a functioning Pro product. Excluding OSS maintainers from a reset given to paying Pro users is kinda hard to justify.