@mikeknoop could you change the scoring to allow scores over 100%? that would be very useful now that models can score above it, right? new better models will still manage to do the games with fewer and fewer actions, and I see no reason why that shouldn't be reflected on the leaderboard?
@edliu1105@nvidia@JensenHuang can you do what the OpenAI employees did to bring back their CEO? Basically, go on strike until the board agrees to reinstate Jensen.
@TokenGremlin@flopsy42 6.2 Astra launch today is at only 9% on Polymarket. Very unlikely it launches today without any insiders knowing it and bringing up the % on Polymarket.
@mhmazur@arcprize@Alibaba_Qwen for Qwen 3.8 27B ARC AGI 3 testing, could you maybe also run it once at like Q4 quantization? would be really interesting to know how the best open source LLM performs on ARC AGI 3 in a quantization that people can actually run on mainstream gaming GPUs
@mhmazur Have you tried to simply ask OpenAI if they would sponsor a full GPT-5.6 Sol run with the Responses API? They might also be interested in knowing how much improvement exactly between Astra and Sol comes purely from the model.
@fchollet so that means you think the official ARC AGI 3 score should be measured using the general purpose OpenAI Responses API with compaction enabled? Will you re-run the semi private test using that now?
@googleflowmusic@GoogleDeepMind can it make songs now without beats? that was the biggest annoyance with it before, I ask for something ambient without any beats and it always adds beats that grow.
@thsottiaux My only issues are that I keep repeatedly getting "Selected model is at capacity" during Sol Ultra runs, and that the Codex CLI still feels much less good to use than Claude Code. Like the subagent management during Fable Ultracode is much better UX in Claude Code.
@GregKamradt@arcprize but do you still apply the 10K cap in the future if you know the pricing in advance? so if you had known the price of 5.6 before, would it basically be stuck at a lower score because it couldn't have gone above 10K in price?
@GregKamradt@leothecurious I don't get why you can't just require the frontier labs to give you some free API usage for a benchmark run, and remove any cost cap. if a model could achieve 20% on ARC AGI 3 but cost 50k USD, I would want to know, and not only see the results capped at 10k.
@GregKamradt I still don't understand how you will deal with the price, with the 10k USD cap you have on ARC AGI 3. If Opus 4.8 already hits the 10k USD cap, how can you ever benchmark Fable?
@GregKamradt@arcprize You really need to fix the X axis with the costs. When the Mythos-class model comes out that will probably be 5x Opus cost, how is that supposed to ever show up correctly on the leaderboard? It seems in the near future models will only become more and more expensive.
@thsottiaux When a session reaches a certain size, I always get the "Falling back from WebSockets to HTTPS transport. stream disconnected before completion: websocket closed by server before response.completed" error and that's super annoying, as then it continues much slower.