kvtop

[LMCache Lab @lmcache]

When is it better to fetch a KV cache than to recompute it?

New on the LMCache blog: an Internet for the KV cache, by Siddhant Ray, adapted from his arXiv paper with Nick Feamster and Junchen Jiang of the University of Chicago.

One example: DeepSeek-V4-Pro, about 100,000 tokens, about 1 GB of compressed KV cache. Sending it over a 10 Gbps link takes about 0.8 s and $0.09. Recomputing it on AWS 8×B200 takes about 15 s and $0.47. Under the post’s assumptions, transfer is about 19× faster and 5.2× cheaper ⚡ (AWS prices are examples only.)

https://t.co/xkB4EX5NOp