Rakshak 🇺🇸✝️

I run daily contamination-free AI benchmarks at
Learning ca 2014

All views are my own

[Rakshak 🇺🇸✝️ @RakshakTalwar]

@trydotworks Pure model tests on complex algo dev

These tests don’t result in multi-turn execution.

[Rakshak 🇺🇸✝️ @RakshakTalwar]

GPT 5.6 Sol vs. DeepSeek Pro yields a p-value of 0.45, which is far from statistically significant.

They also cost the same per solved problem.

In today's run, there is not enough evidence to claim Sol has a meaningful performance edge, so the choice comes down to latency.

[Rakshak 🇺🇸✝️ @RakshakTalwar]

@trydotworks Time required to solve a problem

[Rakshak 🇺🇸✝️ @RakshakTalwar]

@powns_ai The benchmark uses fresh, private problems specifically to reduce contamination and gaming, so I don’t publish the underlying items. I do publish the methodology, scoring, and aggregate results so people can evaluate the benchmark itself.

https://t.co/v3fv8PD7CI

[Rakshak 🇺🇸✝️ @RakshakTalwar]

Today only GPT Astra and Sol along with GLM 5.3 Flash are worth using.

[Rakshak 🇺🇸✝️ @RakshakTalwar]

@libertyorfail This is for algo development

[Rakshak 🇺🇸✝️ @RakshakTalwar]

@H_man12Z i care about you

sending you warm wishes

[Rakshak 🇺🇸✝️ @RakshakTalwar]

Ox Alpha looks a lot worse when the benchmark is brand new.

We test frontier models on fresh problems, so they can’t memorize the test.

[Rakshak 🇺🇸✝️ @RakshakTalwar]

I ran it for a couple days but just like 1.1 it was scoring too low. Plus not a lot of people use it.

Here’s the leaderboard from then. https://t.co/sTSBxEZJVa

I like to keep the leaderboard to under 15 models at a time and with all the constant new releases I drop models as needed.

[Rakshak 🇺🇸✝️ @RakshakTalwar]

OX Alpha scores a lagging 695 Elo Rating

[Rakshak 🇺🇸✝️ @RakshakTalwar]

The variance comes from the model providers. I just measure it. Look at the older posts, it beats the crowd’s wisdom by days to weeks.

I have nothing to benefit from with regards to what I report. The other benchmarks are either outdated within days or are biased by the labs themselves.

[Rakshak 🇺🇸✝️ @RakshakTalwar]

Ox Alpha is free and fast, but its performance is a dealbreaker. It only solved 2.5% of our custom benchmark problems.

[Rakshak 🇺🇸✝️ @RakshakTalwar]

“Running a startup is incompatible with being a full time student”

I made it through on both fronts. It was (barely) possible.

As the startup picked up customers and investors I stretched out classes over part time semesters.

Company: https://t.co/dGGPRbNz5C

I studied and graduated with a BS in Electrical and Computer Engineering.

[Rakshak 🇺🇸✝️ @RakshakTalwar]

@PeterSweeper Thank you for sharing. Context length limitations with Ox definitely affects performance on my benchmark.

[Rakshak 🇺🇸✝️ @RakshakTalwar]

@powns_ai Evals are done with solution references, not LLM as a judge.

Fable was consistently ranked higher until mid July when Anthropic overall dropped the ball with all their models.

[Rakshak 🇺🇸✝️ @RakshakTalwar]

@powns_ai These models flip quite a bit. Look at the older blogs on the site.

[Rakshak 🇺🇸✝️ @RakshakTalwar]

Yes they‘re newly synthesized algorithm brainteasers.

My full suite across all models already takes 5 hours with a concurrency of 18 so we have to cap the time limits for practicality. This affects some models more than others.

When I ran the test yesterday Ox Alpha was actually one of the faster models. Timeouts only impacted 9 out of 40 responses. The rest were zippy.
You can see it here:

[Rakshak 🇺🇸✝️ @RakshakTalwar]

@ChrisEdwards357 Will add in the next run.

[Rakshak 🇺🇸✝️ @RakshakTalwar]

@_nitin_khanna Yes and what they‘re serving could be a downgraded model to support the free introductory price.

[Rakshak 🇺🇸✝️ @RakshakTalwar]

@YvesLoy @markasoftware_ That‘s a good point. When OpenRouter shows more providers this could very well improve. Hence why I run tests multiple times a week.

[Rakshak 🇺🇸✝️ @RakshakTalwar]

@powns_ai Coding/algo tasks. You can see here: https://t.co/ZYiMvRA5UJ

[Rakshak 🇺🇸✝️ @RakshakTalwar]

@markasoftware_ @YvesLoy Ok

[Rakshak 🇺🇸✝️ @RakshakTalwar]

@PolSciEnjoyer It‘s an Ox not a Bull.

[Rakshak 🇺🇸✝️ @RakshakTalwar]

@mitduckmaster That‘s where the signal is.

[Rakshak 🇺🇸✝️ @RakshakTalwar]

@YvesLoy The infra matters just as much. People want results, not just theory.