Backpressure

See what breaks before your users do

[Backpressure @usebackpressure]

Try it: https://t.co/P8f4JS2UTG

[Backpressure @usebackpressure]

A long query still has to embed. Raise tokens per request, the embed step burns more work per call, the embed node saturates while the vector store stays green.

Drop an embedding model and a vector store on a canvas, raise tokens per request, watch the embed node go red.

#systemdesign #backend #distributedsystems

[Backpressure @usebackpressure]

Traffic ramps to 5,863 rps, the worker pool tips into overload, and the whole flow stalls behind it.

The gateway was fine. The balancer was fine. The prompt cache was fine. One worker tier absorbed every queued request and folded.

The fix was not a bigger fleet. Eleven replicas on the pool, and the same load holds.

See what breaks before your users do.

#systemdesign #backend #distributedsystems

[Backpressure @usebackpressure]

A sparse model still parks every expert in VRAM. Decode only burns the active params, so tokens per second jump and leftover KV still tracks the full weight footprint.

Put a vLLM node, a model, and a GPU on a canvas, set active params below total, watch decode rate climb while VRAM occupancy stays put.

#systemdesign #backend #distributedsystems

[Backpressure @usebackpressure]

Try it: https://t.co/P8f4JS2UTG

[Backpressure @usebackpressure]

An API gateway fronting one app tier. At 2,375 rps the app tier collapses. The gateway in front is fine.

The Node event loop ran out of room, so every request queued behind work that could not finish.

The fix was not scaling everything. It was scaling the 1 tier that breaks: 2 replicas, same load, it holds.

See what breaks before your users do.

#systemdesign #backend #distributedsystems

[Backpressure @usebackpressure]

Try it: https://t.co/P8f4JS2UTG

[Backpressure @usebackpressure]

A saga holds each concurrency slot for the full multi-step duration. Add steps, each slot stays occupied longer, the saga saturates while the app still looks fine.

Drop an app server and a saga node on a canvas, raise the step count, watch the saga go red.

#systemdesign #backend #distributedsystems

[Backpressure @usebackpressure]

Try it: https://t.co/P8f4JS2UTG

[Backpressure @usebackpressure]

A 15 rps spike hits an LLM serving stack. The load balancer, gateway and proxy all stay healthy. vllm-main collapses.

Its GPU sits idle. The KV cache slots ran out, so requests queued and then dropped.

The fix was not more GPUs everywhere. It was scaling the one node that saturates: vllm-main.

See what breaks before your users do.

#systemdesign #backend #distributedsystems

[Backpressure @usebackpressure]

Try it: https://t.co/P8f4JS2UTG

[Backpressure @usebackpressure]

You swapped in TGI and throughput stayed put. The KV-slot ceiling is still the model and the GPU.

Put a TGI node, a model, and a GPU on a canvas, push concurrent users, watch the GPU go red.

#systemdesign #backend #distributedsystems

[Backpressure @usebackpressure]

Try it: https://t.co/P8f4JS2UTG

[Backpressure @usebackpressure]

A service handling 1,188 requests a second. Traffic ramps, and the app tier is nowhere near its limit.

The balancer sits at 40%. The gateway routes fine. Every request still ends up holding a Postgres connection, and the pool runs dry.

The fix was not more app servers. It was 2 Postgres replicas. Same load, and it holds.

See what breaks before your users do.

#systemdesign #backend #distributedsystems

[Backpressure @usebackpressure]

Try it: https://t.co/P8f4JS2UTG

[Backpressure @usebackpressure]

Jobs cluster on the hour. The scheduler cannot dispatch that burst, so it saturates while workers stay green.

Drop a scheduler and a worker on a canvas, push load, watch the scheduler go red.

#systemdesign #backend #distributedsystems

[Backpressure @usebackpressure]

Try it: https://t.co/P8f4JS2UTG

[Backpressure @usebackpressure]

At 60 rps, a vLLM service starts dropping requests. The load balancer is fine. The cache is fine.

The collapse is not GPU compute. It is KV-cache slots: every in-flight sequence fills VRAM until new requests queue and drop.

The fix is not a bigger GPU. Scale vLLM to 2 replicas, and the same load holds.

See what breaks before your users do.

#systemdesign #backend #distributedsystems

[Backpressure @usebackpressure]

Try it: https://t.co/P8f4JS2UTG

[Backpressure @usebackpressure]

A websocket tier holding live connections open for every client. At 11,310 rps it collapses.

The load balancer was fine. The traffic was fine. One websocket node ran out of connection slots, not throughput.

The fix was not more speed. It was scaling that tier to 3 replicas.

See what breaks before your users do.

#systemdesign #backend #distributedsystems