Karsten Wade [email protected] @quaid.bsky.social

Founder ex-Red Hat, ex-VA LNX
Cares deeply. Parent, survivor (again), & esp your brother.
He/Him/Él/They. All copyright works CC BY SA 4.0

[Karsten Wade [email protected] @quaid.bsky.social @quaid]

Donald Leon Wade, 23 Jan 1942 - 21 Dec 2021

Of his four sons, I had the fewest formative years w/him. But being his one biological son, his effect on me is as profound as for my brothers K*, T*, & A*.

A little 🧵about Dad: scientist, storyteller, home chef, minister's hubby ...

[LMCache Lab @lmcache]

𝗟𝗠𝗖𝗮𝗰𝗵𝗲 𝗵𝗮𝘀 𝘀𝘂𝗽𝗽𝗼𝗿𝘁𝗲𝗱 𝗞𝗶𝗺𝗶 𝗞𝟯 𝗳𝗿𝗼𝗺 𝗗𝗮𝘆 𝟬!

Kimi K3 combines Kimi Delta Attention (KDA) recurrent-state layers with Multi-head Latent Attention (MLA) layers. LMCache can store and reuse both cache types, enabling prefix reuse across repeated context.

K3 builds on LMCache’s existing support for Mamba and linear-attention hybrids. The only additional adaptation aligns K3’s kernel-paged MLA tensors with vLLM’s logical cache blocks, so the same block IDs used by the scheduler map correctly to what LMCache stores and retrieves.

✅ Validated with vLLM on 8× NVIDIA B300
✅ KDA recurrent-state caching
✅ MLA cache reuse
✅ Available through the LMCache MP connector

👉 Deployment recipe: https://t.co/zGcxYkWK7J

#LMCache #LLMInference #OpenSource #KVCache #KimiK3

[Karsten Wade [email protected] @quaid.bsky.social @quaid]

@lmcache That is the very impressive sound of "no sweat"!

[Avi Chawla @_avichawla]

Researchers made LLM inference 14x faster and 90% cheaper.

The video below depicts the speed up in action.

Providers discount cached input tokens by as much as 90% because a cache hit skips prefill compute entirely.

For stable system prompts and tool definitions, hit rates of 60 to 85% are achievable, which makes it the highest-leverage inference optimization.

But the cost saving only works when the cached text is an exact, byte-for-byte prefix of the new request. If you change one character anywhere before it, the entire cached region is missed.

Three common request patterns produce full cache misses:

- A query that needs documents A and B together can't reuse B's standalone cache, because those KV entries were computed without A in front of them.

- The same three documents retrieved in a different order produce a full cache miss, even though nothing about the documents changed.

- In multi-turn conversations, every new turn invalidates whatever was cached beyond the stable prefix.

Alibaba's production data did a study on this and found that just 10% of cached KV blocks serve 77% of all cache hits.

So most of what gets cached sits in storage and is never used a single time. And the root cause is that KV entries are position-dependent.

Each token's KV encodes attention to everything before it, so a cached block is only valid in the exact context it was computed in.

There's a second, less discussed problem as well.

Cache management runs inside the inference engine's process.

Moving KV tensors between GPU, CPU, and disk competes with inference for the same resources.

This is why Google's TurboQuant compresses KV caches to 3 bits with no accuracy loss and still causes a 20%+ slowdown when it runs in-process.

Fixing both problems means restructuring where caching lives. Cache management moves into its own process, the engine only exchanges block IDs over shared GPU memory, and heavy data movement runs across GPU, CPU, disk, and remote storage in parallel.

Non-prefix reuse gets handled by selectively recomputing only the small set of tokens that attend across document boundaries.

LMCache is the open-source project (10k+ stars) that implements this exact architecture, and it plugs into vLLM, SGLang, and TensorRT-LLM.

The selective recomputation part is implemented in its CacheBlend technique, which makes cached docs in any order and combination, with 2-4x faster multi-document processing.

On H200s running Qwen3-235B with 50 concurrent users, LMCache's multiprocess mode delivers 14x faster time-to-first-token and 4x faster decoding compared to in-process caching.

GitHub repo: https://t.co/MiOCiMoRgo

(don't forget to star 🌟)

My co-founder wrote a full breakdown of KV cache management.

It covers the disaggregated architecture behind the 14x speed up, how CacheBlend preserves generation quality while skipping recomputation, and how to turn every document in a knowledge base into a reusable cached asset.

Read it below.

[Akshay 🚀 @akshay_pachaar]

90% of your KV cache never gets reused.

(prompt caching was never meant to fix it)

if your system prompt and tool definitions are stable, prompt caching is the single highest-leverage optimization available today. cached input tokens get up to 90% cheaper, and hit rates of 60 to 85% are realistic.

but it comes with one rigid rule. the cached portion must be an exact, byte-for-byte prefix of the new request. change a single character in that region and you get a full cache miss.

that rule breaks in three situations you hit constantly:

→ RAG with multiple documents. you cached document A alone and document B alone. a query now needs both. document B's cached state was computed without any awareness of A, so it's invalid and gets recomputed from scratch.

→ document order changes. the same three documents appear in a different order across requests. every permutation is a cache miss, even though the content is identical.

→ growing conversation history. each new turn changes everything after the stable prefix, so earlier cached states beyond it become useless.

Alibaba Cloud's production data shows how bad this gets: 10% of KV cache blocks serve 77% of all hits. the rest sits in storage and never gets reused, because prefix matching won't allow it.

CacheBlend, a research paper from the LMCache team (EuroSys 2025 Best Paper Award), attacks exactly this. the insight is that in modern transformers, tokens overwhelmingly attend to their own local context. only a small fraction of tokens carry real connections across document boundaries.

so instead of recomputing everything after the first cached document, CacheBlend reuses every document's cache as-is and selectively recomputes just those few boundary tokens. those are the small orange fixes between documents in the diagram, and they are the entire cost. the result is 2 to 4x faster processing on multi-document queries with no quality loss.

the order problem disappears with it. shuffle the same documents however you like, and every permutation stays cached, where prefix caching recomputes all of them every time. the bottom of the diagram shows that side by side.

that's the real shift: from caching prefixes to caching knowledge. every document in your knowledge base becomes a reusable cached asset, regardless of what order it appears in or what sits next to it.

CacheBlend ships inside LMCache, the open-source cache management layer that runs outside the inference engine and integrates with vLLM, SGLang, and TensorRT-LLM, on both NVIDIA and AMD GPUs.

check it out on GitHub: https://t.co/TXlaLLu04a

(don't forget to star 🌟)

i wrote the full breakdown of the architecture, including why cache management should never live inside your inference engine. the article is quoted below.

stay tuned for more on this!

[Akshay 🚀 @akshay_pachaar]

14x faster and 90% cheaper LLM inference.

(100% open-source, KV cache management)

your LLM does the same expensive work over and over.

every request, it re-reads the same system prompts and the same documents from scratch, even if it processed them one second ago. token prices keep falling, but agent workloads re-send so much repeated context that the bill climbs anyway.

LMCache fixes this. it's an open-source KV cache management layer that plugs into vLLM, SGLang, and TensorRT-LLM.

here's how it works:

LLMs recompute their understanding of the same content on every request. the same system prompts, the same documents, processed from scratch every time, and a single GPU throws away roughly 15 TB of this reusable cache per day.

LMCache stores that cache and serves it back on repeat requests, running as a separate process completely outside the inference engine.

the engine just asks for the cache blocks it needs. LMCache handles all the heavy data movement across GPU, CPU, disk, and remote storage in parallel, so cache work never steals compute from inference.

it also reuses cache beyond exact prefixes. their CacheBlend technique (EuroSys 2025 best paper) keeps RAG documents cached no matter what order they appear in.

on H200s with a 235B model, that adds up to 14x faster time-to-first-token and 4x faster decoding. and since reuse skips the compute entirely (the same reason providers discount cached tokens by 90%), the cost savings follow directly.

GitHub repo: https://t.co/TXlaLLu04a

(don't forget to star 🌟)

i wrote a full breakdown of KV cache management that walks through why 𝗽𝗿𝗲𝗳𝗶𝘅 𝗰𝗮𝗰𝗵𝗶𝗻𝗴 silently breaks in three common cases, the 𝗱𝗶𝘀𝗮𝗴𝗴𝗿𝗲𝗴𝗮𝘁𝗲𝗱 𝗮𝗿𝗰𝗵𝗶𝘁𝗲𝗰𝘁𝘂𝗿𝗲 behind the 14x speedup, 𝗖𝗮𝗰𝗵𝗲𝗕𝗹𝗲𝗻𝗱, and how to turn every document in your knowledge base into a reusable cached asset.

the article is quoted below.

[Matt Hicks @matthicksj]

With Project Lightwell, we are scaling AI-augmented engineering teams and building a trusted security clearinghouse designed to shore up the open source ecosystem from the inside out. Hear more about this investment on @IBM's Security Intelligence Podcast featuring @RedHat expert Brent Holden: https://t.co/z7gYI1ZEqN.

[Karsten Wade [email protected] @quaid.bsky.social @quaid]

Heads up for anyone checking out the new 'quaid-scanner', I just did a bugfix release 0.1.3 for a handful of things I caught while doing a large scale scan Sunday & Monday.

`npm update quaid-scanner`

Details at https://t.co/JTmNCcY5O2

More info coming...

[Karsten Wade [email protected] @quaid.bsky.social @quaid]

Tired of the AI echo chamber? 🤖

Stop prompting & start shipping

Join us in Santa Cruz or online on 4/26 for AI Native Camp: https://t.co/7SkRpZjVw8

No slide decks—it's 8 hours of validation, pair programming w/agents, & building real-world products

#AINative #SantaCruzTech

[Karsten Wade [email protected] @quaid.bsky.social @quaid]

Curious or already deep into AI-native pair programming or vibe coding?

AI Native Camp – Santa Cruz (in-person) & Online

Saturday March 28th from 9 to 5 Pacific:

https://t.co/sArO6dhvJ8

[AINative Studio™ @ainativestudio]

Jensen said AI-native. Good.

Now the bar is higher:
- not demo-native
- not prompt-native
- actually workflow-native

If your agents can’t remember, retrieve, and act across modalities, you’re still in prototype land.

[Senay Yakut @SenayYaku]

https://t.co/833261r5lu
#AI #Fintech #Blockchain #Compliance #SoftwareEngineering #DevCommunity

[Senay Yakut @SenayYaku]

Agent-402 changes this with:

✅ x402 protocol (signed micropayments)
✅ Circle Wallets (agent treasury)
✅ Arc Network (USDC settlement)
✅ ZeroDB (immutable ledgers)
✅ CrewAI (multi-agent orchestration)

Every action: signed, stored, auditable, replayable

[Senay Yakut @SenayYaku]

The problem:

❌ Agents make decisions but can't prove it
❌ No cryptographic verification
❌ Zero audit trail
❌ Can't replay workflows

This blocks AI from finance, healthcare, legal—any regulated domain.

Current agents = compliance nightmare

[Senay Yakut @SenayYaku]

AI agents can't work in regulated finance.

I just built the infrastructure to fix that.

Introducing Agent-402: Cryptographically-signed autonomous agents
for fintech with x402 micropayments on @circle's Arc Network 🧵

#AgenticCommerce #AI #fintech

[Barack Obama @BarackObama]

A few weeks ago, I met with Girl Scouts on their trip to @MontereyAq. We talked about the magical world just beneath the surface of our oceans, and how our fates are inextricably linked. I hope you’ll check out Our Oceans from @HGMedia, which is streaming now on @Netflix.

[Mueller, She Wrote @MuellerSheWrote]

Obama’s 2008 map. Could we see something similar?

[Elie Mystal @ElieNYC]

Please understand that this platform is about to become the very least reliable source of information in the aggregate if Trump starts losing tomorrow.

Coup Two requires people to feel "robbed" and Musk is going to go into overdrive making the MAGAs feel that.

[Karsten Wade [email protected] @quaid.bsky.social @quaid]

"...expects to see..."

My 'Harris mandate' prediction:
- Harris/Walz break 270 Tuesday night
- House +15 seats or more for a new Dem majority
- Senate gain 1 or 2 seats for the Dem majority
- Final electoral college victory matches/beats 2008 Obama/Biden 365 w/85M votes

[BrooklynDad_Defiant!☮️ @mmpadellan]

Who else wants to see trump lose in a crushing landslide defeat?🤚

[Karsten Wade [email protected] @quaid.bsky.social @quaid]

This video is on 🔥 and makes my heart ❤️ light up ...

... because I ❤️‍🔥 urbanism in my backyard (literally)

[Ray Delahanty | CityNerd @Nerd4Cities]

https://t.co/lJh2DWhIjL

[Karsten Wade [email protected] @quaid.bsky.social @quaid]

@jonobacon @realDonaldTrump @KamalaHarris @JDVance @Tim_Walz I could go on, the list is not exhaustive but it also goes back a few years, too.

It seems like @joerogan has forgotten he is negotiating an interview with a candidate who is also the sitting vice-president? His list of conditions, uh ... WTF?

[Karsten Wade [email protected] @quaid.bsky.social @quaid]

@jonobacon @realDonaldTrump @KamalaHarris @JDVance @Tim_Walz https://t.co/zhNKuXE3zW

[Karsten Wade [email protected] @quaid.bsky.social @quaid]

@jonobacon @realDonaldTrump @KamalaHarris @JDVance @Tim_Walz https://t.co/YsAiLkW9wU