β¨ LMCache v0.5.7.post1 is now available, with safer defaults, more accelerators, and native gRPC.
The official Docker images now run as the non-root user lmcache (UID 1000). The MP HTTP server, the frontend HTTP API, and the coordinator gain optional API key authentication. It stays off until you configure a key with --api-key-file.
Biren SUPA and Cambricon MLU are new platforms, and the gRPC MP transport adds native and asyncio server modes. Packages are built against PyTorch 2.14.0.
This release includes work from 34 new contributors. Thank you, and welcome.
β Before you upgrade: make sure UID 1000 can read and write every path LMCache uses, including mounted volumes.
https://t.co/ku0it3BjII
#LMCache #KVCache
No GPU? You can still develop LMCache.
Our June starter guide shows a new contributor how to run vLLM and LMCache end to end on an Apple Silicon MacBook: a CPU build of vLLM, LMCache installed with NO_GPU_EXT=1, and a two-request test that shows the second request reusing the cache.
It also names four areas you can work on from a laptop: frontend, L1 eviction, L2 storage, and observability.
https://t.co/12NnZeBQNo
#ThrowbackThursday #Evergreen #Knowledge
When a vLLM worker on an Ascend NPU restarts, does it have to lose its warm KV cache?
With LMCache multi-process (MP) mode, now supported on Ascend NPUs, the cache stays. A standalone lmcache server owns the KV cache, so the cache survives engine restarts, and several vLLM instances share it.
The harder question is how KV tensors cross the process boundary on an NPU. The new post explains the two native transfer paths, why DeepSeek-V4 works with no extra configuration, and how to start in three steps.
β Supported on Atlas 800I A2 and A3 and validated with Qwen models and DeepSeek-V4. The Ascend C kernels are open source.
https://t.co/qimv9gKE53
#LMCache #KVCache
Last December we traced every LLM call Claude Code made while it fixed one small Django bug from SWE-bench Verified. The result: 92 calls and a 92 percent prefix reuse rate.
The post shows where that reuse comes from, including subagents and "warm-up" calls that prime the cache before the real work starts. It is a useful read if you serve agent traffic and want to know what your KV cache will see.
https://t.co/wDrXbe48br
#ThrowbackThursday #Evergreen #Knowledge
When an RL job spreads sixteen samples of one prompt over four nodes, why does each node run prefill for that same prompt again?
In LMCache multi-process (MP) mode, each nodeβs KV cache server sees only itself. Node B does not know that node A already holds the KV.
β¨ LMCache contributor Rui Zhang introduces the LMCache coordinator: a control plane that builds a fleet-wide view of every serverβs cache and lets pluggable policies act on it. It is opt-in, in LMCache v0.5.6 and later, and stays out of the request path.
How it stays correct through restarts, how to add your own policy, and where it goes next:
https://t.co/eHmfoyxy8P
#LMCache #KVCache
LMCache v0.5.6 now tells KV-cache-aware routers when a prefix leaves CPU memory. Routers such as NVIDIA Dynamo and llm-d can stop sending requests to a worker that no longer holds the prefix.
Before, routers could learn that LMCache stored a prefix in CPU memory, but nothing told them when LMCache evicted it.
The LMCache multi-process (MP) server now streams CPU store completions and evictions to vLLM. One worker rank per server republishes them as standard vLLM BlockStored and BlockRemoved KV events, through the pipeline routers already consume. If workers A and B both cache prefix P and A evicts it, BlockRemoved withdraws A's placement and P goes to B.
Events travel over a persistent ZMQ or gRPC subscription, with no polling, from a bounded, sequenced log (32,768 records by default). After a restart, a disconnect, or lost events, the worker withdraws every placement it announced.
β Dynamo over ZMQ (TP=2) and llm-d over gRPC: in each, 6 of 6 requests routed correctly, with full prefix hits and zero event errors.
β To enable it, turn on vLLM KV events (--kv-events-config with enable_kv_cache_events: true), keep the server's observability event bus on, and match the worker hash algorithm to the server's --hash-algorithm (default blake3). CPU (L1) tier only for now.
Built by LMCache contributor Dongjoo Seo (DongDongJu on GitHub), reviewed by KuntaiDu on GitHub.
https://t.co/NLGhIY71QX
"Every millisecond spent pulling data back up is a millisecond the GPU spends idle."
LMCache's file backends send each KV block through host memory on its way to the GPU. On four PCIe 4.0 NVMe drives that delivered 25 GB/s, that path reached the GPU at 10.2 GB/s, about 40 percent of the hardware.
Phoenix, an open source GPU direct storage layer from xPU-IO, lets the drive write straight into GPU memory in one hop.
How does it make GPU memory an ordinary I/O buffer for Linux? LMCache contributor Shi Qiu explains:
https://t.co/2EPZZO3AjI
Well then! Calling all inference engines looking to get more cache hits:
@LMCache is the home to the cutting edge researchers and builders defining interfaces and handling transport at this AI-native data layer.
We welcome your integration with our fully decoupled, vendor-neutral, and stack-neutral open source project.
inference engines should be bending over backwards more to get cache hits
really interesting direction:
> An Internet for the KV Cache - sending a KV cache across the Internet can now be faster and cheaper than computing it again.
Is it faster to retrieve a saved KV cache from host memory, or to recompute it?
At KVCache Meetup Korea, Donghun Lee of FuriosaAI shared one test from Furiosa-LLM on RNGD. With a 12,370-token prompt, one request at a time, and three measurements for each condition, retrieving the host cache and generating the first token took about 92 milliseconds. Recomputing the cache took about 900 milliseconds.
The benefit depends on how much data moves and on the available bandwidth. The recap by Junghwan Park of PyTorchKR explains how the engine overlaps cache transfer with computation at each layer.
https://t.co/ftAqQCMA0n
#LMCache #KVCache
Where should the KV cache be stored, and what does it cost to move it?
At KVCache Meetup Korea on September 30, speakers from XCENA, SK hynix, FuriosaAI, and LMCache answered from different layers: CXL memory pools, GPU and HBM bandwidth, request scheduling on RNGD, and LMCache MP Mode.
PyTorchKR and XCENA co-hosted, and Tensormesh sponsored. Junghwan Park of PyTorchKR wrote the recap in English and Korean. It includes a test where retrieving a host cache was faster than recomputing it, and what attendees want at the next meetup.
https://t.co/ceGLHus5UH
#LMCache #KVCache
Sovereign AI shouldn't mean unpredictable costs. @SUSE secures the AI platform. Tensormesh makes inference on it faster and cost-efficient.
See how at SUSE booth #107 at #KubeCon β Tue, Nov 10, 1:30 PM.
#SUSE#SUSEAIFactorywithNVIDIA
Our first LMCache meetup in the San Francisco Bay Area was on 22 October 2026 at Samsung Semiconductor in San Jose, co-hosted by Dongjoo Seo of Samsung and Karsten Wade of Tensormesh.
Samuel Shen of Tensormesh led a hands-on notebook session. Luis Chamberlain of Samsung spoke on asymmetric quantization, dma-buf with io_uring, and Cartridges. Dongjoo Seo closed with LMCache on CXL memory.
If you missed it, the recap has all the slides and Sam's notebook, which runs LMCache on a laptop with no GPU.
https://t.co/yOE0u1TaMe
When is it better to fetch a KV cache than to recompute it?
New on the LMCache blog: an Internet for the KV cache, by Siddhant Ray, adapted from his arXiv paper with Nick Feamster and Junchen Jiang of the University of Chicago.
One example: DeepSeek-V4-Pro, about 100,000 tokens, about 1 GB of compressed KV cache. Sending it over a 10 Gbps link takes about 0.8 s and $0.09. Recomputing it on AWS 8ΓB200 takes about 15 s and $0.47. Under the postβs assumptions, transfer is about 19Γ faster and 5.2Γ cheaper β‘ (AWS prices are examples only.)
https://t.co/xkB4EX5NOp
What should we ask Dongjoo Seo about KV cache system deployment? π¬
Dongjoo Seo of Samsung speaks at 4:30 PM today at the LMCache hands-on workshop: "What we should consider for KV Cache system deployment." Reply with one question, and we will bring a few into the room.
The workshop starts at 1:30 PM Pacific at Samsung, 3655 N First St, San Jose. It is free, and you can still register: β°
https://t.co/9DzKY7y6LX
Today: hands-on LMCache in San Jose. β°
Doors open at 1:30 PM Pacific at Samsung, 3655 N First St. Bring a laptop (recommended). Parking is available, and signs lead you to check-in. π
Four hours: an introduction to LMCache, a 45-minute deep dive with mini-lmcache, two technical sessions, and a reception until 5:30 PM. π οΈ
Registration is free:
https://t.co/N6aNT2ZBBM
Today is the last day of PyTorch Conference. Tomorrow, 22 October, stay in San Jose for the LMCache hands-on workshop: 1:30 to 5:30 PM Pacific, at Samsung, 3655 N First St. π οΈ
Today is the last full day to register. It is free. ποΈ
Know someone at the conference who works on LLM inference? Send this to them before they leave San Jose. π
https://t.co/g43PaP0yKC
Tomorrow at 1:30 PM, doors open at Samsung, 3655 N First St, San Jose. π
Bring a laptop; it is recommended for the hands-on part. Parking is available, and signs lead you to check-in at the event location. Refreshments are provided. Setup help starts at 1:30 PM, so arrive early if you want someone to check your environment. π οΈ
The LMCache hands-on workshop runs until 5:30 PM Pacific. It is free, and registration is required. β°
https://t.co/MyvI4VSg9q
What is the hardest KV cache question in your stack right now? π§
Bring it to the LMCache booth at PyTorch Conference, today or tomorrow, at the San Jose Convention Center. If it needs more time, bring it to our hands-on workshop on Thursday, and test your ideas against a live LMCache environment.
Thursday 22 October, 1:30 to 5:30 PM Pacific, at Samsung, 3655 N First St, San Jose. Free. π
https://t.co/dvgGbiQwhg
Here's a question from our ecosystem of LMCache integrators:
When you run several vLLM processes, does each one keep LMCache inside its own process, or can all of them share one LMCache service? Here's one way:
In multi-process mode, you start lmcache server with CPU memory as L1 and FlashBlade over GPUDirect Storage (GDS) as L2. Each vLLM process connects to the shared server through LMCacheMPConnector.
Josh Lay and Jean-Baptiste Thomas published the full server and vLLM configuration, pinned to lmcache==0.5.3 and vllm==0.24.0, on the Everpure blog:
https://t.co/GSwWV93k1k
#LMCache #KVCache
PyTorch Conference day 1 starts this morning at the San Jose Convention Center, and LMCache has a booth. π
Find us to talk about KV cache management, and register at the booth for our hands-on workshop.
The workshop is in 2 days: Thursday 22 October, 1:30 to 5:30 PM Pacific, at Samsung, 3655 N First St, San Jose. Free. β°
What should we show you at the booth? π¬
https://t.co/QvMZLW9lU2