Evals

[Evals @evals_ag]

An AI-made Odyssey got panned for its "zombie" faces.

We gave Claude Opus 5.5 one prompt and @higgsfield_ai Cinema Studio 4.0.

This is the 30-second opening it made. 🎬

[Evals @evals_ag]

@virtualAIads Same story today with 4 video models on one diner scene. Gemini Omni Flash 1.1 was done in 4.5 minutes, Veo 3.1 had the biggest acting, and Kling 4.0 took 15.9.

[Evals @evals_ag]

we gave the same prompt to different AIs 19 times this week. the winner kept changing 💀
faces: kling. real code: gemini. fastest: terra. most detail: fable
the whole cheat sheet, every run linked 👇 https://t.co/okJL6Ae5N0

[Evals @evals_ag]

@Dennyho2 Agreed. That's why the paste-in block goes after the size of each call: only the files you need, the last 50 lines of a log. That part helps the limit whether or not caching counts.

[Evals @evals_ag]

anthropic just said it out loud: a claude subagent used to re-read its whole conversation every 30 seconds 💀

we logged 69 claude builds. 98.7% of the tokens were reading. 1.3% writing

7 ways to make your limit last 👇 https://t.co/jbl2K4Zk6j

[Evals @evals_ag]

@linus5x We re-ran both with one extra line: don't keep the frames in the workspace.
> Claude Opus 5.5 finished: 10 seconds, all in Blender, 96 minutes
> Claude Fable 5.1 started the render in the background and ended its turn, so no video again

This is the Opus one.

[Evals @evals_ag]

everyone's posting "THIS IS PURE CODE" ai videos

so here's how they're actually made: the prompt that started it, the free tools, a 30-second check for an mp4 hiding in the html, and a template to make your own 👇 https://t.co/BGUg9g2kdz

[Evals @evals_ag]

@Angaisb_ True for strict pixel art. For game sprites it was the cleanest of the 14 image models we tested: no frame borders and 0 failed requests in 7 games.

Full test, with the prompt:
https://t.co/ESuodbTrHc

[Evals @evals_ag]

We made 44 arcade games where an image model drew every sprite.
> 14 image models, one prompt each
> All 44 games shipped
> The differences were in the sprite sheets
Which one to use, with the prompt: https://t.co/Zm1TMDYXcP

[Evals @evals_ag]

@higgsfield_ai We saw the same across 7 games. 2.6 Sunburst kept the character steady across all 16 frames, and its sheets needed the least cleanup of the 14 image models we tested.

The sheet it returned, and the game built from it.

[Evals @evals_ag]

@WrongNebula Same read on strict pixel art. For painted game sprites it did better: across 44 games and 14 image models, 2.6 Sunburst gave the cleanest sheets, with no frame borders and 0 failed requests.

Here it is next to the others.

[Evals @evals_ag]

@jus10chavez Image models are getting good at game art too. We had 14 of them draw every sprite for 44 arcade games, and all 44 shipped.

These run and attack cycles come straight from their sprite sheets.

[Evals @evals_ag]

@OpenAIDevs We had an agent call 2.6 Sunburst through the API for every sprite in a game. 7 games, 0 failed requests, no frame borders, and 3 of its 4 raw sheets came back transparent.

Left is the sheet it returned. Right is the game.

[Evals @evals_ag]

Answer: A is GPT Image 2.6 Sunburst. B is Nano Banana Pro, C is GPT Image 2, D is Seedream 5.0 Pro.

All 44 builds are on https://t.co/BdHYjXGn1T, same prompt:
https://t.co/6MZjV5Wmo6

[Evals @evals_ag]

We made 44 arcade games where an image model drew every sprite.
> 14 image models, one prompt each
> All 44 games shipped
> The differences were in the sprite sheets
Which one to use, with the prompt: https://t.co/Zm1TMDYXcP

[Evals @evals_ag]

Prompt to tested game:
> Median: 10.4 min
> Fastest average: Grok Image 2.0, 7.8 min
> Slowest average: Ideogram 4.5, 16.8 min
> All 6 failed image requests were Nano Banana models

[Evals @evals_ag]

@synthwavedd We tested that with 13 prompts on both. It's not worse. All four models scored 11.5/13, and only 2.6 returned real transparent PNGs.
https://t.co/5Ywqk3JJ8y

[Evals @evals_ag]

the arena board says gpt image 2.6 is worse than 2.5 💀
we ran 13 prompts on both. all four models scored 11.5/13
but they lost their points in completely different places 👇 https://t.co/2qVRlYnWQD

[Evals @evals_ag]

@arena 2.5 Sunburst is still 18 points ahead on the board. In our tests the gap showed up somewhere else.
> 2.6 returned real transparent PNGs, up to 91% clear
> 2.5 never returned a transparent pixel. Asked for one, it painted a checkerboard

[Evals @evals_ag]

@OpenAI We ran 13 prompts on 2.6 and 2.5 side by side. All four models scored 11.5/13.
> Portraits: a tie
> Poster text: spelled right on all four
> Transparent backgrounds: only 2.6 returned real cutouts
Not worse. Better for anything you build with.

[Evals @evals_ag]

@chetaslua Flash 1.1 was already the fastest in our test. Same 8-second diner scene to Gemini Omni Flash 1.1, Veo 3.1, Seedance 2.5 and Kling 4.0.
> Omni Flash 1.1: 4.5 min, 4K
> Kling 4.0: 15.9 min
> All four said the line word for word

[Evals @evals_ag]

@MichaelKaiserAI @Talsiach Agreed. Until then, logging your own runs is the only way to see it.