milo @mi__lo_ - Twitter Profile

mi__lo_ retweeted

5 days ago

🚀 Can Attention-FFN Disaggregation still win on the newest rack-scale GPU systems? We built FastAFD, an open-source AFD runtime for GB200 NVL72: 72 Blackwell GPUs in one NVLink domain. It improves per-GPU decode throughput by 1.35-1.45×. 🧵 Code: https://t.co/FwgLPUhFiS Blog: https://t.co/yQgA72msqd

5

158

33

89

22K

mi__lo_ retweeted

Akshay 🚀

@akshay_pachaar

11 days ago

You're in an ML Engineer interview at Anthropic. The interviewer asks: "Our model generates 100 tokens in 42 seconds. How do you make it 5x faster?" You: "I'll optimize the model architecture and use a better GPU." Interview over. Here's what you missed: The real bottleneck isn't compute. It's redundant computation. Without KV caching, your model recalculates the same attention keys and values for every single token generation. That's why a 9-second inference becomes 42 seconds. You're wasting 80% of your time on repeated calculations. The fundamental issue: (refer image below as you read ahead) LLM token generation is autoregressive: - Generate token 1 from the prompt - Generate token 2 from prompt + token 1 - Generate token 3 from prompt + token 1 + token 2 At each step, you're reprocessing ALL previous tokens through attention. Token 50? You've computed attention for token 1 fifty times. The reality of attention mechanism: For each token, the transformer computes: - Query (Q) from current token - Key (K) from all previous tokens - Value (V) from all previous tokens Then: Attention(Q, K, V) = softmax(QK^T)V Problem: K and V for previous tokens never change. You're recalculating identical matrices every single step. How KV caching solves this: Instead of recomputing K and V matrices: - Cache them after first computation - Reuse cached values for subsequent tokens - Only compute K and V for the new token Without KV caching (token 50): - Compute Q, K, V for all 50 tokens → O(n²) With KV caching (token 50): - Load cached K, V for tokens 1-49 - Compute Q, K, V only for token 50 → O(n) You've eliminated quadratic redundancy. So what's the tradeoff: While KV caching makes the inference faster, it also takes up a lot of memory, so there is always a tradeoff between speed and memory. Why your first token always takes longer: KV caching speeds up inference by computing the prompt's KV cache before generating tokens. This is exactly why ChatGPT takes longer to generate the first token than the rest. First token: Computing KV cache for entire prompt Remaining tokens: Just loading cached KVs + computing new token ---- Everything above is how KV caching works inside a single request. Running it in production is a different problem. Caches break on document reordering, and a single GPU throws away roughly 15 TB of reusable cache per day. I wrote an article that picks up exactly where this post ends: how a new open-source architecture manages KV cache at production scale, with 14x faster time-to-first-token to show for it. The article is quoted below.

akshay_pachaar's tweet photo. You're in an ML Engineer interview at Anthropic.

The interviewer asks:

"Our model generates 100 tokens in 42 seconds. How do you make it 5x faster?"

You: "I'll optimize the model architecture and use a better GPU."

Interview over.

Here's what you missed:

The real bottleneck isn't compute. It's redundant computation.

Without KV caching, your model recalculates the same attention keys and values for every single token generation.

That's why a 9-second inference becomes 42 seconds. You're wasting 80% of your time on repeated calculations.

The fundamental issue:

(refer image below as you read ahead)

LLM token generation is autoregressive:

- Generate token 1 from the prompt
- Generate token 2 from prompt + token 1
- Generate token 3 from prompt + token 1 + token 2

At each step, you're reprocessing ALL previous tokens through attention.

Token 50? You've computed attention for token 1 fifty times.

The reality of attention mechanism:

For each token, the transformer computes:

- Query (Q) from current token
- Key (K) from all previous tokens
- Value (V) from all previous tokens

Then: Attention(Q, K, V) = softmax(QK^T)V

Problem: K and V for previous tokens never change. You're recalculating identical matrices every single step.

How KV caching solves this:

Instead of recomputing K and V matrices:

- Cache them after first computation
- Reuse cached values for subsequent tokens
- Only compute K and V for the new token

Without KV caching (token 50):
- Compute Q, K, V for all 50 tokens → O(n²)

With KV caching (token 50):
- Load cached K, V for tokens 1-49
- Compute Q, K, V only for token 50 → O(n)

You've eliminated quadratic redundancy.

So what's the tradeoff:

While KV caching makes the inference faster, it also takes up a lot of memory, so there is always a tradeoff between speed and memory.

Why your first token always takes longer:

KV caching speeds up inference by computing the prompt's KV cache before generating tokens.

This is exactly why ChatGPT takes longer to generate the first token than the rest.

First token: Computing KV cache for entire prompt

Remaining tokens: Just loading cached KVs + computing new token

----
Everything above is how KV caching works inside a single request.

Running it in production is a different problem. Caches break on document reordering, and a single GPU throws away roughly 15 TB of reusable cache per day.

I wrote an article that picks up exactly where this post ends: how a new open-source architecture manages KV cache at production scale, with 14x faster time-to-first-token to show for it.

The article is quoted below.

21

978

184

1K

77K

mi__lo_ retweeted

vLLM

@vllm_project

about 2 months ago

We've shipped two major upgrades for RL✨! 1. Native weight syncing APIs: Standardizes weight transfer, provides optimized implementations for NCCL and CUDA IPC out of the box, and also lets frameworks easily bring their own. 2. Improved pause/resume for Async RL: Careful coordination between DP ranks so that engines don’t deadlock. Validated at scale in P/D, wide-EP setups! In collaboration with @anyscalecompute, @NovaSkyAI, and @RedHat. More and more RL frameworks are using vLLM as the default for inference, details in the blog 👇 https://t.co/LLmL8zJLtR

vllm_project's tweet photo. We've shipped two major upgrades for RL✨!

1. Native weight syncing APIs: Standardizes weight transfer, provides optimized implementations for NCCL and CUDA IPC out of the box, and also lets frameworks easily bring their own.
2. Improved pause/resume for Async RL: Careful coordination between DP ranks so that engines don’t deadlock. Validated at scale in P/D, wide-EP setups!

In collaboration with @anyscalecompute, @NovaSkyAI, and @RedHat.

More and more RL frameworks are using vLLM as the default for inference, details in the blog 👇

https://t.co/LLmL8zJLtR

2

215

28

73

12K

mi__lo_ retweeted

LMCache Lab

@lmcache

2 months ago

LMCache now supports DeepSeek V4 in multi-process(MP) mode! Huge thanks to the community and everyone who helped make this happen. So what makes DeepSeek V4 different? Most models LMCache supports today use a uniform KV cache layout across layers. For example, Llama, Qwen, and GPT-OSS typically use the same block_size across the model. DeepSeek V4 is different. It uses multiple KV cache formats in the same model(illustrative example: actual numbers may differ): Layers 1–50: block_size = 16 — regular attention, uncompressed Layers 51–58: block_size = 4 — compressor caches Layers 59–60: block_size = 1 — indexer caches You can find more details on this in our technical deep dive blog: https://t.co/W8DJuMhHSq On the LMCache side, MP mode now tracks the KV layout per layer group instead of assuming one packed format across all layers. This lets LMCache read and transfer KV cache from the correct memory locations, while models with uniform layouts continue to use the original fast path. If you are running DSV4 on vLLM and want to use LMCache for KV reuse or offloading, the MP path is ready to try! #AI #LLM #Inference #LMCache #DeepSeek

lmcache's tweet photo. LMCache now supports DeepSeek V4 in multi-process(MP) mode! Huge thanks to the community and everyone who helped make this happen.

So what makes DeepSeek V4 different?

Most models LMCache supports today use a uniform KV cache layout across layers. For example, Llama, Qwen, and GPT-OSS typically use the same block_size across the model.

DeepSeek V4 is different. It uses multiple KV cache formats in the same model(illustrative example: actual numbers may differ):
Layers 1–50: block_size = 16 — regular attention, uncompressed
Layers 51–58: block_size = 4 — compressor caches
Layers 59–60: block_size = 1 — indexer caches

You can find more details on this in our technical deep dive blog:
https://t.co/W8DJuMhHSq

On the LMCache side, MP mode now tracks the KV layout per layer group instead of assuming one packed format across all layers. This lets LMCache read and transfer KV cache from the correct memory locations, while models with uniform layouts continue to use the original fast path.

If you are running DSV4 on vLLM and want to use LMCache for KV reuse or offloading, the MP path is ready to try!

#AI #LLM #Inference #LMCache #DeepSeek

1

10

4

5

471

Who to follow

mi__lo_ retweeted

3 months ago

Btw Deepseek can do this coz they do SO MUCH KV compression with those CSA + HCA layers. CSA (compressed sparse att) groups 4 tokens in the same block. HCA (heavily compressed att) group 128 tokens together! These alternate at each layer! If you a harness dev, you now MUST MUST design harnesses that maximize prefix persistence across requests. Tbf, most harnesses already do this, so this shouldn’t be news to you, but you are rewarded more handsomely now. DeepSeek continues to give us some of the most open innovations in LM training/inferencing. I feel other LLM providers are gonna follow suit watching this and make these architectural changes.