Backpressure

See what breaks before your users do

[Backpressure @usebackpressure]

An LLM gateway at 5 requests a second. Traffic ramps, retries amplify it, and the fallback model server collapses.

The balancer was fine. The gateway and primary model were fine. Bounded retries just kept hammering the one box already full.

[Backpressure @usebackpressure]

A queue buffer saves a fixed number of requests from a spike, however big the spike gets. In the simulator, a Worker serves 800 RPS. Give it a Load with the Spike shape, which peaks at 3x. At 300 users the peak is 900 RPS: Shed drops 500 requests over the run, Buffer drops 0.

[Backpressure @usebackpressure]

Try it: https://t.co/P8f4JS2UTG

#systemdesign #backend #distributedsystems

[Backpressure @usebackpressure]

1,188 requests a second hit a token bucket. The bucket refills slower than traffic arrives, so the excess becomes 429s.

The app behind it was idle. Postgres was untouched.

[Backpressure @usebackpressure]

The limiter was doing exactly its job, and that was the ceiling.

The fix: run 2 limiter replicas on one shared token budget. Same spike, and now the system holds.

See what breaks before your users do.

[Backpressure @usebackpressure]

Try it: https://t.co/P8f4JS2UTG

#systemdesign #backend #distributedsystems

[Backpressure @usebackpressure]

A perfect prompt cache buys less serving capacity than halving the context. In the simulator, a 7B model at 4 bits with 8K context on a 24 GB GPU holds 17 users on vLLM. Set Prompt cache hit to 100% and prefill drops from 100 ms to 0, and it holds 23.

[Backpressure @usebackpressure]

Cut the context to 4K with the cache off and it holds 35, because each request's KV cache is half the size and twice as many fit in VRAM. Do both and it holds 49. Prompt cache is Pro.

[Backpressure @usebackpressure]

Try it: https://t.co/P8f4JS2UTG

#systemdesign #backend #distributedsystems

[Backpressure @usebackpressure]

Traffic ramps to 13 rps and the inference server falls over. Not from GPU compute: it ran out of KV cache slots.

The load balancer sat at 40 percent. GPU utilization was low.

[Backpressure @usebackpressure]

Every in-flight sequence just held its tensors and filled VRAM.

The fix was not a bigger GPU. It was 2 replicas of the inference tier. Same load, and it holds.

See what breaks before your users do.

[Backpressure @usebackpressure]

Try it: https://t.co/P8f4JS2UTG

#systemdesign #backend #distributedsystems

[Backpressure @usebackpressure]

Retries can cut an overloaded database's useful work in half. In the simulator, 1,100 RPS hits an App server in front of PostgreSQL, which tops out at 1,000. With no retries it serves 1,000 and drops 100, so 91% of requests succeed.

[Backpressure @usebackpressure]

The database still does 1,000 RPS of work, but only about 510 of it is useful. Success falls to 47%, same result on every seed. Under sustained overload, retries feed the bottleneck that caused the drops. Retry is Pro.

[Backpressure @usebackpressure]

Try it: https://t.co/P8f4JS2UTG

#systemdesign #backend #distributedsystems

[Backpressure @usebackpressure]

A sharded database handling 6,680 rps across 6 shards. One hot key pins the load on a single shard, and the whole write path collapses.

[Backpressure @usebackpressure]

The app tier stayed fine. The other 5 shards stayed fine.

The fix was not more shards. The database got 2 replicas, and the same load held.

See what breaks before your users do.

[Backpressure @usebackpressure]

Try it: https://t.co/P8f4JS2UTG

#systemdesign #backend #distributedsystems

[Backpressure @usebackpressure]

Speculative decoding can speed up a model and leave your user capacity unchanged. If KV cache fills VRAM before decode throughput runs out, a faster draft-and-verify loop has nowhere to put another request.

[Backpressure @usebackpressure]

The knob pays off when decode is the ceiling. Put a model and GPU behind a vLLM node, raise Spec decode, then change the model's context and GPU VRAM. Watch when the serving ceiling moves. Spec decode is Pro.