@ThePrimeagen From reading @VictorTaelin's struggles for more than a year and always hoping for his work to be acknowledged - to seeing post as this one starting to pop up - what a time to be alive 🎆🎆🎆
a small announcement: I'm almost done with a puzzlescript-based benchmark; you can think of it as ARC-AGI-3.5, maybe even ARC-AGI-4; I expect it to be the most reliable way of assessing agents' abstract reasoning skills in a game environments
I went through over 300 games, handpicked the most interesting and diverse set of 40, fixed a ton of bugs, went through multiple iterations of prompt design, etc
most games can be completed in around 1h, some of them are significantly harder; models are tested using standard harnesses (codex/claude code) with no code generation allowed; action limits are very generous and unfortunately the benchmark is already somewhat saturated, but the models are still insanely action inefficient
FYI, I won't release the game set to the public, but I guess I might share runs from some of the rejected games.
btw, if anyone knows, what do I have to do in order to not pay $10K out of my own pocket for testing, I'm all ears
Yesterday, @Parodyjeffx posted a screenshot showing Grok admitting Israel is committing genocide.
It went viral.
Then X permanently banned his account.