Bhabani N

[Bhabani N @bsn_uce]

1/ 8×H100 lost to 4×H100 with one config change

I ran 12 experiments tuning vLLM for a 4B model at 400 concurrent users. Doubling GPUs: +61% throughput. One batching knob on half the hardware: better latency AND better cost-per-token.

A thread on what actually moves metrics 🧵

[Bhabani N @bsn_uce]

That means your unit economics are now a product decision made by whoever wrote the default config, often an engineer optimizing for demo quality, not margin. The teams doing this well route by task: high effort for the 5% of requests that need it, low for the rest.

[Bhabani N @bsn_uce]

The reasoning dial is a pricing decision, not a technical one. Anyone budgeting AI spend in 2026: the "reasoning effort" dial on modern models isn't a quality slider, it's a cost curve. Same model, same API, 2.4× price difference between low and high effort on Gemini 3.8 Flash.

[Bhabani N @bsn_uce]

Anthropic (Claude Fable 5.1 / Mythos 5.1, Sept 1), Google (Gemini 3.8 Flash, Sept 2), Meta (Muse Spark 1.3, Sept 2), and OpenAI (GPT-6 Astra, Sept 3–4) all released in one week. Notably, Google itself is telling developers to stay on 3.7 Flash for efficiency-first workloads.

[Bhabani N @bsn_uce]

Four frontier labs shipped new models in 48 hours last week. Google is openly recommending its older model for a large class of workloads. We've crossed from "which model is best" into "which model is right for this specific job at this specific cost".

[Bhabani N @bsn_uce]

Most teams' evaluation processes haven't caught up. If your model selection process still ends at "we use the newest one," you're leaving both money and reliability on the table.

[Bhabani N @bsn_uce]

Details here
https://t.co/Pla8VYpZyD

[Bhabani N @bsn_uce]

@AnthropicAI open-sourced Claude Commerce Agents, for building AI shopping assistants. However, not every retailer is ready. Every retailer is asking which lab (@OpenAI or @AnthropicAI) do we go with?

Walked through each path from building AI Agents at production scale driving $

[Bhabani N @bsn_uce]

"MLA is theoretically interesting but slow in practice" is half right. The slowness is an implementation choice.

Full derivation, both kernels, numbers out to 131K:
https://t.co/MP4j73OMmN

Part 3 next week: a decision tree for picking attention.

[Bhabani N @bsn_uce]

My MLA had a smaller KV cache than my MHA and decoded 8× slower.

32K ctx, batch 16: MLA 5,000 tok/s vs MHA 42,000.

Two days later I found the matrix identity that makes it 13× faster with identical outputs. 🧵

https://t.co/MP4j73OMmN

[Bhabani N @bsn_uce]

The honest part: absorbed MLA at 131K is still ~27% behind GQA(kv=8) at the same 8 GB cache.

This moves MLA from unshippable to competitive. It doesn't make it win.

And it breaks if you put RoPE on K — decoupled RoPE is the fix.

[Bhabani N @bsn_uce]

All nine trained checkpoints are now public on Hugging Face, with the raw CSVs of 210 H100 benchmark cells. Rerun it on your data; if your Pareto frontier disagrees with mine, I want to see it.

[Bhabani N @bsn_uce]

The question: how much KV cache do you actually buy per % of validation loss? Nine 30M-param language models, identical training, different attention.

Findings and full writeup -> https://t.co/AKTlp0i53G

[Bhabani N @bsn_uce]

📌 At production dimensions (H=32, d_head=128, batch 16, 131k ctx): MHA's KV cache hits 32 GB on an 80 GB card. MQA does 14× MHA's throughput on 1/32 of the cache. The cache wall is not theoretical.

[Bhabani N @bsn_uce]

9/Final: 4×H100, CUDA graphs, seqs=64 → p95 TTFT 1,716ms, p95 ITL 6.7ms, 0% errors at 400 users. Beat the matched reference on all three axes.

Full writeup + harness/configs: [https://t.co/LRORUmvnuV] [https://t.co/HabkeiOrsY]

thank you @VizuaraAI @modal

[Bhabani N @bsn_uce]

1/ 8×H100 lost to 4×H100 with one config change

I ran 12 experiments tuning vLLM for a 4B model at 400 concurrent users. Doubling GPUs: +61% throughput. One batching knob on half the hardware: better latency AND better cost-per-token.

A thread on what actually moves metrics 🧵

[Bhabani N @bsn_uce]

8/Measurement honesty: my cold-baseline A/B said max_num_seqs was worth +137%. Properly warmed runs: +30–50%.

CUDA graph warmup contaminates naive A/Bs. No warmup protocol = your deltas are upper bounds.