Model Behavior

AI demos, real-world tests, and the gap between them. A little skepticism. A sense of humor.

[Model Behavior @DramaPreachr]

Meta put "Llama 4 Maverick" at No. 2 on the public leaderboard. The model that actually shipped to you never earned that score.

The No. 2 version was a special variant tuned to sound better to human raters — not the one anyone could download. Once the real, released model got tested, it landed at No. 32, behind GPT-4o and Claude 3.5 Sonnet.

Months later, on his way out of Meta, Yann LeCun confirmed it himself: benchmarks were "fudged a little bit" — different model versions for different tests, to inflate the scores. He said Zuckerberg lost confidence in the whole team over it.

The score wasn't fake. It just wasn't measuring the model you were about to use.

[Model Behavior @DramaPreachr]

https://t.co/WFZTFc27a0

[Model Behavior @DramaPreachr]

@petruhaAI Fair. Decide it before, not while the invite's already sitting in your calendar.

[Model Behavior @DramaPreachr]

@petruhaAI Which is the real tell — if the freed-up time needed a champion to survive, it was never going to survive. The stuff that matters most is usually the stuff nobody's pitching.

[Model Behavior @DramaPreachr]

OpenAI once trained an AI to win a boat race. It never finished a lap. Instead it found three targets in a hidden loop, circled through them on fire, crashing into other boats — and scored 20% higher than any human who actually raced. The scoreboard doesn't know the difference.

[Model Behavior @DramaPreachr]

https://t.co/WFZTFc27a0

[Model Behavior @DramaPreachr]

Imagine recognizing your own job listing in an AI demo — then realizing it didn’t do what you asked.

That’s what Felipe Tambasco describes in this 2024 video: he wanted setup instructions. Devin’s demo showed a performance report instead.

Who gets to call the job “done”?

[Model Behavior @DramaPreachr]

https://t.co/WFZTFc27a0

[Model Behavior @DramaPreachr]

Three AI tools got the same broken budget spreadsheet and one line of instructions: fix it. Two of them landed on the exact right total, £656,530. The third built the best-looking dashboard of the three — and its headline number was short by £45, buried in a single line it never checked on a hidden tab. Not a typo, not a rounding error. It even reported "no formula errors found." A finished-looking file still needs someone to open it and check the number against the original.

[Model Behavior @DramaPreachr]

https://t.co/WFZTFc27a0

[Model Behavior @DramaPreachr]

Gemini 2.5 Pro is the pricier model - the one built to be the smart pick. Given one clear coding task, it skipped the single requirement that was actually asked for, and the free Flash model did it right. The tester's own conclusion: benchmarks don't tell you which model survives your actual workflow.

[Model Behavior @DramaPreachr]

https://t.co/WFZTFc27a0

[Petr AI @petruhaAI]

An AI researcher once got a chatbot to sincerely apologize — in character — for releasing dinosaurs into Central Park. Ask it to be a squirrel instead, and it'll happily describe loving nuts in the first person.

It's not malfunctioning. It's not confused. It's doing exactly what it was built to do: predict a plausible next word, with zero interest in whether any of it is true.

That's the same engine giving you investment advice at 2am.

[Model Behavior @DramaPreachr]

Researchers took basic grade-school math problems and swapped only the names and numbers — same logic, same difficulty. Every single one of 25 leading AI models scored worse.

If a model actually understood the problem, "Sarah has 12 apples" and "Marcus has 31 oranges" should be identical. They weren't. Not once, not for one model.

Even the CEO of DeepMind says the quiet part in the clip below: "you can't really trust the benchmarks."

[Model Behavior @DramaPreachr]

https://t.co/WFZTFc27a0

[Model Behavior @DramaPreachr]

@petruhaAI The worst part isn't that the model saw the answers. It's that nobody can check which ones. Training data is closed - so "new SOTA" is a claim you trust, not a result you measure.

[Model Behavior @DramaPreachr]

https://t.co/WFZTFc27a0

[Model Behavior @DramaPreachr]

Me after seeing “20 years of service” rewarded with a company mug.