KVCache.AI

Hi, this is official account. We build systems for efficient LLM serving, including KTransformers, Mooncake and AgentENV.

[KVCache.AI @KVCache_AI]

🚀 Exciting news! Mooncake is now listed in the CNCF Landscape!

We're excited to be part of the cloud native ecosystem, bringing high-performance KV cache management and data transfer to LLM inference at scale.

[KVCache.AI @KVCache_AI]

Elastic Expert Parallelism scaling in SGLang!

Excited to see Mooncake PG powering elastic membership and synchronization, working alongside @sgl_project and NIXL to enable more elastic and efficient MoE inference without restarting live deployments!

🔗 https://t.co/mmHgwjTTYQ

[KVCache.AI @KVCache_AI]

Great to see Mooncake powering KV cache offloading in @PrimeIntellect's Prime Inference stack! 🚀

This blog dives into building fast, reliable inference for long-context agent workloads, featuring Dynamo, vLLM, Mooncake and FlashInfer.

Read more:

🔗 https://t.co/X28nxlgvGQ

[KVCache.AI @KVCache_AI]

🚀 Excited to share our new paper, KTransformers-FineTune (KT-FT), developed in collaboration with the amazing @llamafactory_ai team!

Fine-tune 671B MoE models with just 16GB GPU VRAM + CPU memory.

⚡ 2.5–21.2× faster than offloading baselines
🧠 Up to 128K-token contexts
🔧 CPU-GPU co-computation, beyond offloading

📄 Paper: https://t.co/1fhCeO8hKm

[KVCache.AI @KVCache_AI]

Looking forward to learning how Mooncake + HiCache are pushing KV cache efficiency further in real-world deployments. 🚀

[Kevin Flansburg @kevin_flansburg]

Excited to be speaking at the first SGLang Summit (@sgl_project), Nov 12–13 at Fort Mason in SF!

My team (@areneau @KastanDay ) and I are looking forward to presenting our work on system one models (Clef and Clairic), and sharing how we are maximizing KV cache hit rates using a multi-deployment Mooncake cluster with HiCache. Come say hi if you're there!

[KVCache.AI @KVCache_AI]

Today is the Mid-Autumn Festival, the day when the moon is said to shine at its fullest and brightest.

Mooncakes, rich pastries filled with sweet bean or lotus seed paste, are shared and enjoyed with family and friends during this festival.

As the poem goes: “The bright moon rises over the sea; though far apart, we share this moment together.”

Wishing everyone a happy Mid-Autumn Festival and a joyful time with your family! 🥮

[KVCache.AI @KVCache_AI]

Huge congrats to the @vllm_project team on pushing DSpark training for Kimi-K3 to this scale! 🚀

As the high-performance data plane behind the disaggregated pipeline, Mooncake moves hidden states between vLLM inference and training workers, fully utilizing RDMA bandwidth with topology-aware routing and multi-NIC aggregation.

A great example of Mooncake powering large-scale AI data movement beyond KV cache transfer.

Read more: https://t.co/RY6iFhf2Rd

[Red Hat AI @RedHat_AI]

Kimi K3 is a 2.8T-parameter model. We trained a DSpark speculator for it, and the speedup holds up.

Single-stream math reasoning goes from ~110 to ~435 tokens/sec per user. Under concurrent load, up to ~3.5x higher output throughput at matched interactivity. The drafter is a 5B model proposing 8 tokens a step, and on math it gets ~6.4 of them accepted per round.

Training a drafter for a model this big meant going multi-node: Speculators plus a new Mooncake connector streaming hidden states between vLLM inference and training over RDMA. Two GB300 nodes to serve the target, one to train.

https://t.co/A5ibPUnSmk

[Red Hat AI @RedHat_AI]

Full session, vLLM Office Hours #55: Mooncake + vLLM/llm-d deep dive: https://t.co/WDxtvI4Jg5

Live every other Thursday: https://t.co/X8hAHYRBgT

[KVCache.AI @KVCache_AI]

Docs & examples:

https://t.co/xz1E7WZYYY

https://t.co/qYjDYlMtYN

[KVCache.AI @KVCache_AI]

KTransformers v0.7.1 is here 🚀

Now supporting:
• Qwen VLM multimodal LoRA fine-tuning in BF16
• Native RAWINT4 LoRA for Kimi K2.5 / K2.6

Fine-tune massive VLM and MoE models with a more memory-efficient training path.

[KVCache.AI @KVCache_AI]

A great read on how GLM-5.3 helped optimize its own inference stack, tripling end-to-end throughput in under two weeks, all driven by a dense feedback loop between the agent and the system.

One particularly interesting case for us: while debugging a KV Transfer bottleneck, the team traced a >20% Prefill + KV Transfer performance gap due to Python GIL contention between DeepEP and the Mooncake Transfer thread. After releasing the GIL in the relevant C++ paths, the gap dropped to below 1%.

[Z.ai @Zai_org]

We’re sharing how GLM-5.3 helped build and optimize the inference infrastructure serving GLM-5.3-Flash.

The system went from its first successful run to production readiness in less than two weeks, with end-to-end throughput tripling relative to the initial baseline.

The key was dense feedback: local correctness tests, execution traces, microbenchmarks, and end-to-end measurements that enabled targeted hypothesis testing rather than reliance on aggregate performance metrics alone.

https://t.co/yUf6OpJD7c

[KVCache.AI @KVCache_AI]

🚀 Over one trillion tokens per day, powered by Mooncake and SGLang.

@ApproachingAI shares the system architecture and engineering lessons behind its token factory, running leading trillion-parameter models with SGLang HiCache + Mooncake Store.

A deep dive into:
🏗️ Production architecture and system design trade-offs
⚡ Performance optimizations for trillion-token-scale workloads
🛡️ Failure isolation, fault tolerance, and recovery

Real production lessons on scaling KV cache reuse while meeting strict inference SLOs.

Read more: https://t.co/5zUwkScRXp

[KVCache.AI @KVCache_AI]

Thanks for the recognition!

Mooncake has always been focused on building high-performance storage and communication infrastructure for large-scale distributed LLM serving. We’re excited to keep pushing this forward with the community.

[zhyncs @zhyncs42]

Gravity matters. I caught up with a friend today, and we talked about Mooncake @KVCache_AI — by 2026, whether for PD disaggregation or KV storage, Mooncake has become a first-class citizen and the default solution across the top 3 LLM inference engines.

[KVCache.AI @KVCache_AI]

Congratulations to the @lightseekorg TokenSpeed team! Great to see Kimi K3 optimizations running stably at production scale on GB300 and H200.

[LightSeek Foundation @lightseekorg]

TokenSpeed × Kimi K3, Part I: TP8 optimization on GB300. Now stable 3+ weeks in production across ~1,000 B300 and ~2,000 H200 GPUs. Thanks @NVIDIAAI for engineering support and GB300 NVL72 access, and community teams for their support and contributions. 👇
https://t.co/yby1Z7YuY8

[KVCache.AI @KVCache_AI]

The mystery model "Ox Alpha" is officially revealed as GLM-5.3-Flash, now with native KTransformers support!

[KVCache.AI @KVCache_AI]

Mooncake v0.3.13 is here! 🚀
This release brings major updates across Mooncake Store, Transfer Engine, Process Group, hardware support, and ecosystem integrations, with 276 commits and 33 new contributors. A huge thank you to all the users and contributors! ❤️

✨ Highlights:
• Mooncake Store: Multi-tenancy and NVMe KV SSD support
• Transfer Engine: Enhanced NVLink, IBGDA, and multi-rail transfer support
• Process Group: Dynamic membership and recovery
• New integrations with Miles and Speculators
• Broader platform support: NCCL, MUSA IPC, SUPA, and FlagCX
• Published wheel packages for ROCm

[KVCache.AI @KVCache_AI]

📖 Cookbook: https://t.co/CPYqIP2SR5

[KVCache.AI @KVCache_AI]

🚀 KTransformers v0.7.0 is here!
This release brings major upgrades for ultra-large MoE fine-tuning:
• Native FP8 Expert weights for LoRA, reducing host memory usage for DeepSeek-V3.1 from ~1.4 TB to ~800 GB
• Support AVX-512 backend for compatible AMD and x86 CPUs
• Support full fine-tuning
• Checkpoint saving and resuming
• A complete KTransformers × LLaMA-Factory fine-tuning Cookbook
🔗 Blog: https://t.co/mH4fV5rYSt
#KTransformers #LLM #FineTuning #DeepSeek

[KVCache.AI @KVCache_AI]

I think both contribute. Disaggregation removes prefill/decode interference. It also lets us optimize and schedule each compute phase independently with more specialized kernels and policies. That separation becomes increasingly important as we scale inference serving to larger deployments.

[KVCache.AI @KVCache_AI]

TileRT is pushing NVIDIA GPUs into a new regime of ultra-high-interactivity inference, with a specialized decode engine delivering hundreds of tokens per second per user.

In its disaggregated prefill/decode architecture, Mooncake Transfer Engine provides the high-performance data path for moving KV cache from vLLM prefill nodes to TileRT decode nodes.

Read the full deep dive from SemiAnalysis: https://t.co/1UI8X0XFh2

#LLMInference #TileRT #vLLM #Mooncake