Bitdepth

served-as-advertised verification for LLM inference

[Bitdepth @BitDepthdev]

Bitdepth grades LLM inference endpoints. Every night I send the same prompts to every OpenRouter endpoint serving the same open-weight model, compare each output to a published reference, and set the result beside the price. https://t.co/0zkQK9GKxN

[Bitdepth @BitDepthdev]

DeepSeek V4 Pro on OpenRouter: 16 endpoints, same model, same prompts, every night for 2 weeks.

4 of them score 21 to 36% on simple string tasks. The rest sit around 59%. All 16 pass JSON fine.

Names go public Thursday. Which model do you route most?

[Bitdepth @BitDepthdev]

@hankins_sz you said clear labels would decide which router you stay with.

On gpt-oss-120b, one endpoint’s agreement with the rest dropped from 91 to 97% to 75 to 86% on Nov 1 and stayed there. The label didn’t move. The answers did.

[Bitdepth @BitDepthdev]

@beayushb @speedrun @ycombinator @aika_labs Good luck 🍀 You pinned DeepInfra FP8 and turned off fallbacks. I’ve been checking whether that pin holds up.
Good news: on DeepSeek V4 Pro, deepinfra/fp8 has been inside the pack all 13 nights, above the median on the string tasks where 4 of 16 endpoints fall far behind.

[Bitdepth @BitDepthdev]

@jstnwooo Hey, you counted 40 endpoints for one model. https://t.co/0zkQK9GcIf tests them every night.
DeepSeek V4.1 Flash is up to 53 endpoints now, output prices from $0.15 to $9.60 per million. 7 of them score 7 to 12 points below the pack every night.

[Bitdepth @BitDepthdev]

The tokens you already paid for are stranded https://t.co/9FRb91UD3O

[Bitdepth @BitDepthdev]

Two questions. If you hold inference you paid for and can't use: what is it, how much, and what would you sell it for? If you buy open-weight inference at volume: would you lock a price for 90 days on a graded endpoint? What would stop you? Replies open, or [email protected]

[Bitdepth @BitDepthdev]

Every month, real money is paid for inference nobody uses. Startups prepay, pivot, and watch the credits expire. Companies sign annual commits at twice their run rate. None of it can be sold. Here is why, and what a real market would need.

[Bitdepth @BitDepthdev]

So the order is: a unit, a grade, a provider willing to issue a transferable claim, then a market. Bitdepth grades endpoints every night. We're matching the first reserved blocks by hand: fixed price, 90 days, graded nightly, remainder transferable. https://t.co/tBqfzldSSA

[Bitdepth @BitDepthdev]

@RouterPlex quick question, what makes a customer switch from openrouter to a router like yours? i run bitdepth and grade endpoints every night. happy to share what i see for the models your customers run most.

[Bitdepth @BitDepthdev]

@subnetmarco Agreed on MCP. Once tool calls ride alongside the tokens, the shape of the output matters as much as the quality. A field dropped silently from a JSON tool call never shows up in a token count.

[Bitdepth @BitDepthdev]

Benchmarks rank models. Bitdepth ranks the companies serving the model.

[Bitdepth @BitDepthdev]

When you buy an open-weight model, fifteen to thirty companies serve the same model name. They don’t serve the same thing. Some quantize it, some drop fields, some drift week to week. I grade the answers against a published reference, and put the grade next to the price.

[Bitdepth @BitDepthdev]

Bit depth is how many bits each model weight gets. BF16 is 16. FP8 is 8. FP4 is 4. Every halving makes the model cheaper to serve and changes what it answers. Providers pick the number. OpenRouter’s precision field is whatever the provider typed in. Bitdepth checks.

[Bitdepth @BitDepthdev]

5/ Nothing here is a claim about intent. Numbers describe outputs against a stated reference. The raw record behind every number is kept and hashed. Price sits beside every grade.

[Bitdepth @BitDepthdev]

1/ How a Bitdepth grade works, in six posts. No provider is named in this thread. The point is the method.

[Bitdepth @BitDepthdev]

3/ The reference. If at least two endpoints declare bf16 or fp16, the reference is the plurality answer among them. If fewer than two do, it's the plurality across all endpoints. The rule is stated with every result.