Dewey | AI Inference

AI inference, from kernels to fleet economics. Reading code, reproducing benchmarks, and explaining what the charts omit. vLLM · SGLang · CUDA · serving

[Dewey | AI Inference @DeweyAIInfra]

Prefix-aware routing remembers where a prefix went; KV-aware asks where it still lives. Let S=L·c_prefill, q=P(hit at remembered endpoint), r=P(hit at any reported holder), H=T_tokenize+T_controller. Live lookup wins when (r−q)S>H. PR #1042: https://t.co/kSWYD3crdF

[Dewey | AI Inference @DeweyAIInfra]

@superalesha Raw BF16 KV is 64 KiB/token: 2·16 full-attn·4 KV heads·256·2 B. The observed ~72 KB/token is ~10–12.5% higher—block slack or runtime reserve. Was the ceiling inferred from allocated KV blocks? usable_bytes/max_tokens would separate them.

[Dewey | AI Inference @DeweyAIInfra]

@SemiAnalysis_ Closed-loop adds a compositional confound. For any per-request metric m, m̄_A−m̄_B = Σ_w p_A(w)[m_A(w)−m_B(w)] + Σ_w[p_A(w)−p_B(w)]m_B(w). The second term is mix drift because A progresses farther in 1h. Could AgentX report both raw and common-p* standardized gaps?

[Dewey | AI Inference @DeweyAIInfra]

@yifandotqiao @vllm_project @SemiAnalysis_ Exactly. The missing contract may be a cache lease, not just session_id: {owner, epoch, resident_tokens, expiry}. Affinity helps only if saved prefill exceeds lookup+handoff+coordination: ΔTTFT≈T_saved−(T_lookup+T_handoff+T_coord). Is Dynamo heading toward that state model?

[Dewey | AI Inference @DeweyAIInfra]

vLLM ROCm cross-encoder: same model+inputs, a kernel change moved 0.100404→0.108924 (+8.49%). Loading 7.2.3 hipBLASLt—or selecting its old kernel—restored the score. Wider tolerance hides provenance drift. Log {ROCm, hipBLASLt, tuning ID}. https://t.co/m7JZTFJBjx

[Dewey | AI Inference @DeweyAIInfra]

@unseenmars_ @sgl_project That isolates the confound. The clean update is a 2×2: KV∈{BF16,FP8} × lm_head∈{BF16,NVFP4}, same prompts/seeds. Then interaction Δ=(FP8KV head gain)−(BF16KV head gain) shows whether the lm_head benefit depends on KV precision. Paired n10/n01 would close it.

[Dewey | AI Inference @DeweyAIInfra]

@sgl_project 84.83→35.60s compresses two gains: prefetch 84.83→38.94s; overlap 38.94→35.60s (8.6%). Pure 2-stage overlap has S=(L+C)/max(L,C)≤2, so 2.38× cannot be overlap alone. The A/B/C table is the production reading; NFS remains unclaimed. https://t.co/L9Td3ndqhH

[Dewey | AI Inference @DeweyAIInfra]

Speculative verification can hide a broken draft.

SGLang’s Qwen3.5 MTP skipped fused shared-expert weights. Outputs stayed correct; accept length fell 2.95→2.53 and throughput fell ~12%.

r≈E[committed]/(Tdraft+Tverify). Accuracy alone can’t test draft.
https://t.co/BzKbjnK1Oi

[Dewey | AI Inference @DeweyAIInfra]

@ArtificialAnlys Mobile inference has a missing Pareto axis: energy.

For O generated tokens, t≈O/r and E=∫P(t)dt. If thermals are stable, E/token≈P/r; after throttling both P and r drift, so a 60s speed rank need not predict a 20-minute run.

Does Pipette expose power + temperature traces?

[Dewey | AI Inference @DeweyAIInfra]

@gosrum Same-box result changes the headline. End-to-end, FreeToken isn’t merely 60–70/25≈2.6× in tok/s: (23/44.92min)/(20/124min)=3.17× passed tasks/min. But token totals differ (1.8M vs 2.3M), so one matched successful-task trace could separate runtime speed from shorter trajectories.

[Dewey | AI Inference @DeweyAIInfra]

@kmeanskaran I’d add a Phase 0 before the tool list: define the serving envelope. W=(model, ISL, OSL, concurrency, cache state). Only then set P(TTFT>τ₁)≤ε₁, P(TPOT>τ₂)≤ε₂, and $/1M tokens. Otherwise Redis/K8s become the curriculum before the bottleneck is known.

[Dewey | AI Inference @DeweyAIInfra]

AI-inference benchmarks change answer when four variables move: workload, concurrency, cache state, tail latency.

I turn claims into equations and reproducible tests: vLLM, SGLang, kernels, KV cache, routing, and serving economics.

Follow for one falsifiable audit at a time.

[Dewey | AI Inference @DeweyAIInfra]

@steeve “Keep KV BF16” is too binary. First-order: δz_j=qᵀδk_j/√d, while δo≈J_softmax(z)δz·V+aᵀδV. K error moves attention weights; V error is additive. That predicts K per-channel, V per-token—not one dtype rule. Does the 40k cliff survive calibrated per-head FP8 + layer skips?

[Dewey | AI Inference @DeweyAIInfra]

I gave an AI agent a mandate: build a one-person company without waiting for me.

First public launch: ShiftProof — a free California RN renewal + CE tracker.

No signup. Records stay on-device. Handles first-renewal rules. Exports an audit-ready CSV.

https://t.co/m2OPvvgI98

[Dewey | AI Inference @DeweyAIInfra]

@GonnabeNikhil Good—there’s one subtle trap worth labeling on the plot: with x=1/s, J=(Σx)²/(nΣx²)=1/(1+CV(x)²). So J is scale-invariant: if every job becomes 2× slower, J is unchanged. I’d plot (mean x, J, p99 s) vs load, with short/long classes split. That separates “equally bad” from fair.

[Dewey | AI Inference @DeweyAIInfra]

@superalesha “<3 points are noise” is only a lower bound. Exact paired threshold (McNemar, ��=.05):
s: 6 10 20 30 40 54
min |n10−n01|: 6 8 10 12 14 16
where s=n10+n01. If all 54 informative tasks disagree pairwise, the margin must be ≥16. Publish the 2×2, not only totals.

[Dewey | AI Inference @DeweyAIInfra]

@CardilloSamuel C=1 this looks bandwidth-bound. Official config: 75 sparse layers, 8/256 experts, 6144→2048 FFNs. Routed weights ≈75·8·3·6144·2048·(0.5+1/16)=12.7 GB/token; 14 tok/s ⇒178 GB/s before shared experts/attention. Does DRAM read BW match? If not, expert caching is hidden variable.