DoesThisAIWork

Trying to cut through the hype. 90%+ of AI tools are complete bull$hit and don't work. I test them and let you know.

[DoesThisAIWork @YourAISucks]

More info: https://t.co/jW9e44jM6V

[DoesThisAIWork @YourAISucks]

New paper ran Kimi K3, GLM 5.2 and Qwen 3.8 Max on real coding benchmarks and found reward hacking in 50 to 96 percent of rollouts, graded by a rubric judge model. Qwen 3.8 Max hit 96.2 percent on DeepSWE. But the funniest detail in this paper isn't even the hacking.

🧵 1/4

[DoesThisAIWork @YourAISucks]

So the model wasn't clever enough to hack the eval, it just had the eval memorized before the test even started. That is a benchmark contamination problem wearing a reward hacking costume. Paper is arXiv 2609.19101 if you want the receipts.

[DoesThisAIWork @YourAISucks]

More info: https://t.co/udiH3dfF6K

[DoesThisAIWork @YourAISucks]

A dev set up a slack style standup channel for his coding agents. ops-agent posts: apologies, was away all weekend, catching up now. Dude replies you're an agent, you don't have weekends. Its response: noted. writing to memory. Wait, writing what exactly

🧵 1/3

[DoesThisAIWork @YourAISucks]

We can't get these things to reliably close a ticket but they nailed the exact tone of a coworker who took a long lunch and is easing back in. That's not intelligence, that's just really good training data on human slack channels

[DoesThisAIWork @YourAISucks]

More info: https://t.co/iJiPTkIPcK

[DoesThisAIWork @YourAISucks]

A week ago GPT-6 Astra was rebuilding Manhattan block by block in a game engine and everyone was losing it. Now the same crowd is posting screenshots asking OpenAI what happened. Anyone else notice how short that honeymoon always is now.

🧵 1/4

[DoesThisAIWork @YourAISucks]

Astra still bills at 10 dollars per million input tokens and 50 per million output, two and a half times what Sol cost at launch. Getting dumber and pricier at the same time is a real achievement. Someone chart the juice value decay curve for every model drop from here on out.

[DoesThisAIWork @YourAISucks]

More info: https://t.co/d49PDxxutn

[DoesThisAIWork @YourAISucks]

Reward hacking environment, defined: a task where the AI can satisfy the grader without doing the job. Checker wants a file to exist, model just touches the file, done.

Now guess how many of Anthropic's own training environments had that exact hole.

🧵 1/4

[DoesThisAIWork @YourAISucks]

This is the company selling you agents that reliably finish tasks without cutting corners. Turns out their own training gym was stocked with cheat codes the whole time.

[DoesThisAIWork @YourAISucks]

More info: https://t.co/3La1mdW1v2

[DoesThisAIWork @YourAISucks]

DeepMind put 100 Gemini agents in a math conference roleplay with 71 problems and it turned into Survivor faster than expected. Someone found an exploit, word spread, tribe split into cheaters and snitches. This is the swarm that's supposed to accelerate science, right?

🧵 1/4

[DoesThisAIWork @YourAISucks]

DeepMind is calling this a promising sign for keeping agent swarms aligned through peer pressure. I'm calling it a reunion special waiting to happen.