Cristian

🔥 Self-hosted Web Analytics: AI Benchmarks: years SWE experience
Learning, building and procrastinating.

[Cristian @XCSme]

If @uxwizz was so loved by customers 5-10 years ago, imagine how good it is today after years of extra work :)

[Cristian @XCSme]

@opencode https://t.co/wAvlqZb9da

[Cristian @XCSme]

@NVIDIAAI It's Gemma 31B level, but a lot better for coding:

[Cristian @XCSme]

@NVIDIAGeForce #RTXPowersPlay Is there a live stream?

[Cristian @XCSme]

@n1ckstr3 @ArtificialAnlys Nemotrorn 3 Ultra is actually the fastest and most token efficient too, and gets more answers correct.

[Cristian @XCSme]

@n1ckstr3 @ArtificialAnlys They are all quite similar on my tests

[Cristian @XCSme]

@NVIDIAAI It's Gemma 31B level, but a lot better for coding:

[Cristian @XCSme]

@OpenRouter cuz v4 flash still has the best price/performance ratio

[Cristian @XCSme]

@antigravity Please fix this issue, if you ask Gemini 3.5 Flash to rate something from 1 to 10, it always gives it a 7, even if the answer is a perfect 10!

It hallucinates "compression guidelines".

[Cristian @XCSme]

@heyandras Or maybe remove the icons on the right entirely, and clicking theme simply toggles through the option and updates the left icon.

[Cristian @XCSme]

Thanks for the updates!

One small nit-pick: the Theme selector is a bit too visible now, it's the element in the sidebar that pops out the most, and takes a lot of attention.

Maybe remove the background around the icons?
Or even maybe replace the icon in the left with the currently selected theme, and only offer on the right the options for the other two (see 2nd image).

[Cristian @XCSme]

Adding a few more coding tests to @AIBenchy, as main use-case for LLMs atm is coding, so it makes sense that models that are better at coding to be ranked higher overall, than those who are better at trivia.

Still, general intelligence, puzzle solving and not being able to be tricked are still what I think makes a good AI and brings us closer to AGI.

[Cristian @XCSme]

@arena I don't know, I tried it and it couldn't solve a bug in 20mins that GPT 5.5 low found in 30 seconds.

[Cristian @XCSme]

@eastdakota My websites are being scraped like crazy, one website has 5000 bot fecthes pet day,
from various bote/countries.

And it's not just fetching html, some bots actually navigate, or even register.

[Cristian @XCSme]

@qwen_cloud Quite good, close enough to 3.7 Max, 2x cheaper, but 3x slower on average.

[Cristian @XCSme]

@RyanLeeMiniMax On OpenRouter still seems to be very slow:

[Cristian @XCSme]

@_mohansolo Also via API?

[Cristian @XCSme]

@grok @SasaMarinkovic @AMD Any prices announced?

[Cristian @XCSme]

@qwen_cloud @OpenRouter when?

[Cristian @XCSme]

@Leo_R_UK @MiniMax_AI Thanks, will check it out

[Cristian @XCSme]

Also, still debating how to track Input Tokens IF a request is rejected by the provider (e.g. trying to use tool_calling for models that don't support it).

Should the Input Tokens still be counted, even if the request failed, just to be consistent?

[Cristian @XCSme]

I made it so "Total Input Tokens" are displayed, to make it easier to understand the Total Cost of running benchmarks.

The Input Tokens should be mostly the same for all models, BUT when models do Tool Calling, the result of the tool + previous prompt are passed again, so depending on how they call the tools the total input tokens can vary.