Autobench

The LLM evaluation engine powering AI benchmarks and routers. We use multi-LLM evaluation for accurate and unbiased evaluation of LLM quality, cost and speed.

[Autobench @AutobenchAI]

How does Botscanner work?

[Autobench @AutobenchAI]

"Which model is best?" and "which model is enough for this task?" are different questions.

We test the second one, on your own prompts

[Peter W. Kruger @pwk]

Announcing the @AutobenchAI Router (and some bad news for @AnthropicAI and @OpenAI).

Yes, today we are releasing an LLM router that really saves more than 90% of your AI bill (with $1 free credits to try).

On Monday our Router Benchmark showed you that current routers don't route at all (the blue dots in the graph). That was the good news for AI Labs. Today comes the bad news: we built a router that actually works (the green dots).

Setting our router to "Quality" or "Balanced", it beats the top reference model (Gpt-5.6 Sol) at respectively 12% and 8% of Sol's bill. Setting it at "Cost", it offers even better quality and savings than Luna.

Try it now on our user-facing service Botscanner and claim your $1 free credits (links below). That's the equivalent of about 270 hard prompts (only 13 if you were using Gpt-5.6 Sol).

1/7👇

[Autobench @AutobenchAI]

@omarsar0 @costgallo Sign up for the beta and get free credits https://t.co/txTw9j6Kd9!

[Autobench @AutobenchAI]

@xianrong996 @pwk @AnthropicAI @OpenAI https://t.co/OWKrAAw3EO

[Peter W. Kruger @pwk]

Announcing the @AutobenchAI Router (and some bad news for @AnthropicAI and @OpenAI).

Yes, today we are releasing an LLM router that really saves more than 90% of your AI bill (with $1 free credits to try).

On Monday our Router Benchmark showed you that current routers don't route at all (the blue dots in the graph). That was the good news for AI Labs. Today comes the bad news: we built a router that actually works (the green dots).

Setting our router to "Quality" or "Balanced", it beats the top reference model (Gpt-5.6 Sol) at respectively 12% and 8% of Sol's bill. Setting it at "Cost", it offers even better quality and savings than Luna.

Try it now on our user-facing service Botscanner and claim your $1 free credits (links below). That's the equivalent of about 270 hard prompts (only 13 if you were using Gpt-5.6 Sol).

1/7👇

[Autobench @AutobenchAI]

@adityaa_behera @pwk @AnthropicAI @OpenAI https://t.co/OWKrAAvvPg

[Peter W. Kruger @pwk]

Announcing the @AutobenchAI Router (and some bad news for @AnthropicAI and @OpenAI).

Yes, today we are releasing an LLM router that really saves more than 90% of your AI bill (with $1 free credits to try).

On Monday our Router Benchmark showed you that current routers don't route at all (the blue dots in the graph). That was the good news for AI Labs. Today comes the bad news: we built a router that actually works (the green dots).

Setting our router to "Quality" or "Balanced", it beats the top reference model (Gpt-5.6 Sol) at respectively 12% and 8% of Sol's bill. Setting it at "Cost", it offers even better quality and savings than Luna.

Try it now on our user-facing service Botscanner and claim your $1 free credits (links below). That's the equivalent of about 270 hard prompts (only 13 if you were using Gpt-5.6 Sol).

1/7👇

[Autobench @AutobenchAI]

@markstachowski @pwk @AnthropicAI @OpenAI https://t.co/OWKrAAvvPg

[Peter W. Kruger @pwk]

Announcing the @AutobenchAI Router (and some bad news for @AnthropicAI and @OpenAI).

Yes, today we are releasing an LLM router that really saves more than 90% of your AI bill (with $1 free credits to try).

On Monday our Router Benchmark showed you that current routers don't route at all (the blue dots in the graph). That was the good news for AI Labs. Today comes the bad news: we built a router that actually works (the green dots).

Setting our router to "Quality" or "Balanced", it beats the top reference model (Gpt-5.6 Sol) at respectively 12% and 8% of Sol's bill. Setting it at "Cost", it offers even better quality and savings than Luna.

Try it now on our user-facing service Botscanner and claim your $1 free credits (links below). That's the equivalent of about 270 hard prompts (only 13 if you were using Gpt-5.6 Sol).

1/7👇

[Autobench @AutobenchAI]

@adventurared @pwk @AnthropicAI @OpenAI Thoughts? https://t.co/OWKrAAw3EO

[Peter W. Kruger @pwk]

Announcing the @AutobenchAI Router (and some bad news for @AnthropicAI and @OpenAI).

Yes, today we are releasing an LLM router that really saves more than 90% of your AI bill (with $1 free credits to try).

On Monday our Router Benchmark showed you that current routers don't route at all (the blue dots in the graph). That was the good news for AI Labs. Today comes the bad news: we built a router that actually works (the green dots).

Setting our router to "Quality" or "Balanced", it beats the top reference model (Gpt-5.6 Sol) at respectively 12% and 8% of Sol's bill. Setting it at "Cost", it offers even better quality and savings than Luna.

Try it now on our user-facing service Botscanner and claim your $1 free credits (links below). That's the equivalent of about 270 hard prompts (only 13 if you were using Gpt-5.6 Sol).

1/7👇

[Autobench @AutobenchAI]

@devfaizanali @pwk @AnthropicAI @OpenAI Here, bad news for them, good news for you https://t.co/OWKrAAvvPg

[Peter W. Kruger @pwk]

Announcing the @AutobenchAI Router (and some bad news for @AnthropicAI and @OpenAI).

Yes, today we are releasing an LLM router that really saves more than 90% of your AI bill (with $1 free credits to try).

On Monday our Router Benchmark showed you that current routers don't route at all (the blue dots in the graph). That was the good news for AI Labs. Today comes the bad news: we built a router that actually works (the green dots).

Setting our router to "Quality" or "Balanced", it beats the top reference model (Gpt-5.6 Sol) at respectively 12% and 8% of Sol's bill. Setting it at "Cost", it offers even better quality and savings than Luna.

Try it now on our user-facing service Botscanner and claim your $1 free credits (links below). That's the equivalent of about 270 hard prompts (only 13 if you were using Gpt-5.6 Sol).

1/7👇

[Autobench @AutobenchAI]

@Eito_Miyamura @pwk @AnthropicAI @OpenAI https://t.co/0C4U6KT9xs

[Autobench @AutobenchAI]

We audited seven routers and none beat always calling one mid-priced model. So we built one: three models, three settings, $1 free credit. Apply for early access to the beta

https://t.co/FEIUcf7dIN

[Peter W. Kruger @pwk]

There’s significant efficiency to be gained by switching from the large frontier models of @AnthropicAI and @OpenAI to cheaper Chinese alternatives like @Zai_org GLM. However, this approach has its own drawbacks:

1. Models like GLM handle a wide range of use cases well, but they still fail on a small percentage of cases (roughly 1–5%) compared to SOTA models. These failures tend to hit exactly where it hurts most—the more complex, high-stakes requests.

2. Competition among Chinese labs is just as intense as among US labs. This drives continuous improvement, which means companies will likely need to keep switching to newer, more advanced GLM versions (or even different model families). Managing that transition yourself is painful and operationally heavy, especially if you’re managing your own hardware infra.

Taken together, this strengthens the case for LLM routers that are continuously upgraded to the latest models and can intelligently route real-time user requests, including the long-tail complex cases that still need more powerful (and expensive) models.

That’s exactly what we’re doing at @AutobenchAI with the router we released last week. It delivers 90%+ savings versus SOTA models while still giving you full SOTA capability on the ~5% of cases that actually need it. Try it out.

[Autobench @AutobenchAI]

The prize is real. A per-prompt policy that picks the cheapest model still within the quality bar scores 100% Delta-Hit at $0.0018 per question.

Always-Luna: 85.3% at $0.0043.

That is roughly 12 quality points at about 98% lower cost. No router collected it, until we launched https://t.co/rAkyqm8u8R

https://t.co/qgMH4gDQnv

[Autobench @AutobenchAI]

What 1$ buys you? Depends who you ask

[Peter W. Kruger @pwk]

Announcing the @AutobenchAI Router (and some bad news for @AnthropicAI and @OpenAI).

Yes, today we are releasing an LLM router that really saves more than 90% of your AI bill (with $1 free credits to try).

On Monday our Router Benchmark showed you that current routers don't route at all (the blue dots in the graph). That was the good news for AI Labs. Today comes the bad news: we built a router that actually works (the green dots).

Setting our router to "Quality" or "Balanced", it beats the top reference model (Gpt-5.6 Sol) at respectively 12% and 8% of Sol's bill. Setting it at "Cost", it offers even better quality and savings than Luna.

Try it now on our user-facing service Botscanner and claim your $1 free credits (links below). That's the equivalent of about 270 hard prompts (only 13 if you were using Gpt-5.6 Sol).

1/7👇

[Autobench @AutobenchAI]

Do routers actually route? Not really. We tested the most used routers and 5/17 commercial settings sent at least 90% of prompts to one model, so their quality number was that model's number.

Why pay for something you can do it yourself? Try https://t.co/rAkyqm8u8R, the only router that actually routes

[Peter W. Kruger @pwk]

One of the major failures we pointed out in our router benchmark is that most commercial and open routers don't actually route.

Five of the 17 settings sent 90%+ of prompts to one model. Turn the knob from cheap to quality and you often just got a different constant.

The @AutobenchAI Router does not do that as you can see in the graph. Prompts are routed between the different models, with the mid-tier model taking most of the calls (note that "Cost" routes only between two models).

If a vendor will not show you that mix, you are paying for a bookmark.

4/7

[Autobench @AutobenchAI]

"LLM Routers are threatening the Frontier Labs' revenues". Another great article about the launch of our router.

Remember: you can try our router yourself. It's easy and you get $1 free credit.
https://t.co/zcdjAa1mau

[Autobench @AutobenchAI]

@OpenRouter Interesting! We'll test it right away on our Router Benchmark.
https://t.co/5tftJLEa8O

[Peter W. Kruger @pwk]

Announcing the @AutobenchAI Router (and some bad news for @AnthropicAI and @OpenAI).

Yes, today we are releasing an LLM router that really saves more than 90% of your AI bill (with $1 free credits to try).

On Monday our Router Benchmark showed you that current routers don't route at all (the blue dots in the graph). That was the good news for AI Labs. Today comes the bad news: we built a router that actually works (the green dots).

Setting our router to "Quality" or "Balanced", it beats the top reference model (Gpt-5.6 Sol) at respectively 12% and 8% of Sol's bill. Setting it at "Cost", it offers even better quality and savings than Luna.

Try it now on our user-facing service Botscanner and claim your $1 free credits (links below). That's the equivalent of about 270 hard prompts (only 13 if you were using Gpt-5.6 Sol).

1/7👇

[Autobench @AutobenchAI]

We're in the media🎖️

[Peter W. Kruger @pwk]

Our LLM Router on the media. Nice article by @Startupbusiness that explains the benefits of the @AutobenchAI routing technology: get Frontier model quality in the answers to your prompts at less than 10% the cost.

https://t.co/DdyAmR2KJb
(Article in Italian)

[Autobench @AutobenchAI]

Today we are releasing our LLM Router that gets you higher quality than Gpt-5.6 Sol at less than 10% of its bill. It also beats by far all major commercial and open-source routers on the Autobench Router benchmark.

We are also handing $1 free credits to test it on our user-facing service: https://t.co/rAkyqm8u8R. For custom routers, reach out directly.

[Peter W. Kruger @pwk]

Announcing the @AutobenchAI Router (and some bad news for @AnthropicAI and @OpenAI).

Yes, today we are releasing an LLM router that really saves more than 90% of your AI bill (with $1 free credits to try).

On Monday our Router Benchmark showed you that current routers don't route at all (the blue dots in the graph). That was the good news for AI Labs. Today comes the bad news: we built a router that actually works (the green dots).

Setting our router to "Quality" or "Balanced", it beats the top reference model (Gpt-5.6 Sol) at respectively 12% and 8% of Sol's bill. Setting it at "Cost", it offers even better quality and savings than Luna.

Try it now on our user-facing service Botscanner and claim your $1 free credits (links below). That's the equivalent of about 270 hard prompts (only 13 if you were using Gpt-5.6 Sol).

1/7👇

[Autobench @AutobenchAI]

The only reason to pay for a router is cheap answers that are still good. Nobody we tested delivers that.

Finding 5 from our Router Benchmark: The cell that pays for the product stays empty.

Every answer is a hit or a miss, and cheap or costly. Cheap + good is the product. It stays a minority for everyone.

Strong routers buy expensive hits. Cheap routers buy cheap misses.

The prize is real. Nobody is collecting it.

[Peter W. Kruger @pwk]

If they don’t agree on the model, they also don’t land where a router would pay for itself.

Every answer is a hit or a miss, and cheap or costly. The only cell that justifies paying for a router is cheap + good.

It stays a minority for everyone.

Strong routers mostly buy expensive hits. Cheap routers mostly buy cheap misses. Nobody fills the cell that would make the product worth it.

7/14