Offset Zero

Executing the vision

[Offset Zero @offsetx0]

@trydotworks You used this new model or old 4.1?

[Offset Zero @offsetx0]

@JRobFreedom @Dshoopy0 @tszzl because it is a significantly small model relatively. Try with kimi 3

[Offset Zero @offsetx0]

It is due to math and more specifically a specific math benchmark which feels like an outlier, "integrals with game" see it. I feel this is likely a parsing issue if LLM as a judge is not used here, I am not sure though. If we remove math, it is within the margin of error with the rest of the top. I think this benchmark is saturated now and doesn't actually describe anything at this point. It was useful in the past.

[Offset Zero @offsetx0]

@bindureddy KIMI K3 for design

[Offset Zero @offsetx0]

@scaling01 Saturated. It has the same score as Sol lol, all within the margin of error. Also, Muse Spark, even the first version, is better than Sol and this already tells what it measures lol.

[Offset Zero @offsetx0]

@karirogg I still don't think it wasmax. It was probably 'high.' We might need to explicitly define it. Can you run just a few tests? It usually uses a ton of tokens, so I find it hard to believe it burned less than GLM 5.1

[Offset Zero @offsetx0]

@teortaxesTex It is not on max thinking, used 1.5-2x lower tokens than glm 5.1 lol

[Offset Zero @offsetx0]

@karirogg I think it is not on max mode, very low thinking tokens

[Offset Zero @offsetx0]

Because it is based on LMArena. Qwen 3.7 Max is the leader now among Chinese, it is a live leaderboard now. At the frontier level, people can't just properly tell which is correct in many cases. In fact, GLM 5.2 is ranked lower than GLM 5.1, and the same goes for GPT 5.5, it ranks low despite being very strong.

[Offset Zero @offsetx0]

9 months ago it was the GPT 5 era, and even my local gemma 4 31B beat it in some tasks. 6 months ago it was the GPT 5.2 and Opus 4.5 era, and GLM 5.2 was clearly better than them across the domain(not just benchmark). I tested all of them while doing hard coding math tests, and it is more like 4 months or less.

[Offset Zero @offsetx0]

@MatthewBerman @levie 4-6 months max. Glm 5.2 is already better than gpt 5.4 and opus 4.6.

[Offset Zero @offsetx0]

@airesearch12 @lev_ey @scaling01 But still it is dart cheap

[Offset Zero @offsetx0]

@teortaxesTex It is better in spatial reasoning. Vision is not their main property as of now, but I think this benchmark is heavily concentrated on a very narrow AI blindspot, which is more of a vision encoder benchmark than the model.

[Offset Zero @offsetx0]

@osanseviero 124B MoE

[Offset Zero @offsetx0]

@osanseviero @AI_Andrew Gemma 4:124B

[Offset Zero @offsetx0]

@demishassabis Love them, but I wish there were a bigger Gemma 4 in the 100 to 120B range.

[Offset Zero @offsetx0]

@GaryMarcus @Noahpinion @demishassabis https://t.co/t0azZqdjWu

[Offset Zero @offsetx0]

@ns123abc This is an old video from January. The latest OpenAI solution isn't brute force, it is a new connection across fields, and it matters. He is basically commenting on the OpenAI hype from last year, but ironically AI somehow got there so fast.

[Offset Zero @offsetx0]

@sumjitg https://t.co/t0azZqdjWu

[Offset Zero @offsetx0]

@ns123abc This is an old video from January. The latest OpenAI solution isn't brute force, it is a new connection across fields, and it matters. He is basically commenting on the OpenAI hype from last year, but ironically AI somehow got there so fast.

[Offset Zero @offsetx0]

@ns123abc This is an old video from January. The latest OpenAI solution isn't brute force, it is a new connection across fields, and it matters. He is basically commenting on the OpenAI hype from last year, but ironically AI somehow got there so fast.