Zachary Xi

product @inferact

[Woosuk Kwon @woosuk_k]

vLLM now supports Vera Rubin! DeepSeek, Kimi, MiniMax, GLM, and more are ready on Day 0, thanks to Rubin’s compatibility with most Blackwell kernels. We are also shipping a list of new Rubin-specific optimizations too 😄
Huge thanks to @NVIDIAAI for all the support!

[Zachary Xi @zlxi02]

VR support is live!! @inferact team has been working nonstop to make sure on vLLM performance is first class.

[vLLM @vllm_project]

vLLM now supports NVIDIA Vera Rubin. The early results show more than 7.8x the throughput of GB200 on MiniMax M3 on AgentX.

@inferact, @NVIDIAAI, @RedHat_AI, and the vLLM community have been bringing vLLM up on Rubin since it was announced. Here is where things stand.

🧵 1/5

[vLLM @vllm_project]

vLLM now supports NVIDIA Vera Rubin. The early results show more than 7.8x the throughput of GB200 on MiniMax M3 on AgentX.

@inferact, @NVIDIAAI, @RedHat_AI, and the vLLM community have been bringing vLLM up on Rubin since it was announced. Here is where things stand.

🧵 1/5

[vLLM @vllm_project]

The @inferact team open sourced a TPU megakernel for Kimi K3 achieving 709 tokens/s, against 450 tokens/s on GB200.

All 92 of K3's MoE layers run in a single Pallas kernel, with weight prefetching that reaches across layer boundaries so transfers for one layer overlap with computation in the previous one.

Shoutout to the team! Writeup and repo link in the thread.

[Inferact @inferact]

Our first TPU megakernel for Kimi K3 reaches 709 tokens/s on low-concurrency decode, against 450 tokens/s for our GB200 baseline, both with DSpark speculative decoding.

To our knowledge, this is the first TPU inference megakernel. The whole model runs in a single Pallas kernel, and without spec decoding it is roughly 1.4 to 2x the GB200 baseline at batch sizes 1 through 8.

We are open sourcing it today.

1/2

[Inferact @inferact]

@googlecloud and Inferact are announcing today a partnership to make TPU a first-class citizen in @vllm_project.

This partnership puts both teams on one engineering roadmap to bring TPU to the broader open model ecosystem, optimizing vLLM as the agentic production serving engine for TPU:

• Production serving features and optimized kernels
• A native PyTorch path via TorchTPU
• Moving towards day-0 support for frontier model releases

We're also launching a community program: shared TPU capacity for open-source contributors, plus dedicated review and design help from the core vLLM maintainers at Inferact.

Everything this collaboration produces is open source. Read the full announcement: https://t.co/asTJSktyQK

[Google Cloud Tech @GoogleCloudTech]

We’re partnering with @inferact (the team behind @vllm_project) to make TPUs a first-class citizen for open-source AI inference.

Together, we’re delivering:
- Upstreamed open-source TPU kernels
- Native PyTorch integration via TorchTPU
+ so much more ↓

[vLLM @vllm_project]

https://t.co/OhdmKTt57r

[vLLM @vllm_project]

New blog is out: vLLM x AgentX: Optimizing for Real-World Agentic Serving.

Agent traffic stresses every layer of the serving stack at once. This post walks the full-stack work for optimizing vLLM on Agentic workloads, including the architecture, framework, and runtime optimizations, measured on AgentX, @SemiAnalysis_'s public agentic benchmark.

🧵1/6

[Zachary Xi @zlxi02]

@yifandotqiao and the team cooked

[vLLM @vllm_project]

Congratulations to @SemiAnalysis_ on the release of AgentX 1.0 🎊, an open-source multi-turn agentic coding benchmark collected from ~$3M of real traces, running on 1000+ chips and ~2MW of continuously operated compute.

We are excited to see @vllm_project’s competitive performance on frontier open models:
🔷130,093 tok/s/chip for DeepSeek V4 Pro
🔷 77,079 tok/s/chip for Minimax M3
🔷 12,479 tok/s/chip for Kimi K3.

The following thread covers an overview of the work from @vllm_project and @inferact: what we tested, found, and shipped upstream. This work highlights vLLM’s performance on real-world workloads and our committed focus to making vLLM an agentic-first engine.

Optimizing AgentX performance meant tackling three major challenges: prefix reuse, efficient long context parallelism, and scaling performance with PD disaggregation.

First, long agentic sessions stress prefix caching and KV cache offloading. Modern hybrid models have greatly reduced the required KV cache sizes, and caching every block boundary still saturates the KV cache pool which causes prefix cache thrashing across sessions.

The fix was sparse retention: one state per interval-sized segment plus the latest replay boundary (vllm-project/vllm #43447, #45845), then preserving shared-prefix boundaries so the interval can go to 0 for agent sessions (#47782). That gets us >95% hit rate at 14 concurrent requests with contexts to 1M.

The bigger structural change in KV cache offloading was making the shared KV pool distributed. With Mooncake Store as a first-class connector, prefill ranks can hit the prefix cache both within a worker and across workers, so we no longer have to trade cache locality against load balance to keep a cluster busy. Session-aware routing (48048) is what lets the router act on it.

For a single node deployment, SimpleCPUOffloadConnector has been greatly improved to support all hybrid model architectures and across both CUDA and ROCm platforms. For DeepSeek V4 Pro on ROCm, this implementation gave +81.7% output throughput and 46.6% lower mean e2e latency versus recomputing the prefix.

Kimi K3 at 2.8T barely fits on a single node, and squeezing it in leaves almost no headroom for KV cache, which is exactly what a long multi-turn session needs most. Parallelism strategy matters more here than on any other model we tested.

TP8/DCP8 tops the K3 agentic frontier across configs, with B300 vLLM peaking at ~12.5k tok/s/chip at ~8 tok/s/user and GB300 NVL72 on Dynamo + vLLM holding the curve out past 200 tok/s/user. K3 also surfaced a routing bug worth pulling: vllm-project/router#194 fixes the router dropping reasoning_content, which hits any reasoning model served behind it.

On MiniMax M3, B200 vLLM reaches ~44k tok/s/chip and vLLM leads TRT-LLM on throughput vs p90 TTFT. M3 and Qwen3.5 both shipped day-0.

Finally, fully optimized performance requires scaling the deployment with prefill-decode disaggregation and distributed KV cache offloading. Thanks to vLLM’s MultiConnector, this is natively supported with NIXL PD connector + MooncakeStoreConnector. By rate matching to find the ideal prefill-decode ratio, we achieved 4.45x higher throughput at a 60 tok/s interactivity for DeepSeek V4 Pro on GB300 Dynamo compared to B300.

Shoutout to @NVIDIAAI, who co-tuned most of these configs with us. Dynamo's router optimizations took AgentX replay time down 23.7% on the vLLM backend, the AIPerf replay harness is what made the traces runnable at all, and NIXL and their kernels sit under each respective performance point.

Shoutout to @AIatAMD as well: the team’s AITER sparse-MLA decode selection led to +5.22% AgentX output throughput; the hybrid AITER/native CSA selector led to 1.21–1.76x e2e performance boost, and the team made prebuilt lmcache gfx942/gfx950 wheels and a published Mooncake ROCm wheel.

Thank you to @SemiAnalysis_ for building AgentX and for the collaboration throughout. Agentic workloads are what users serve in real production, and benchmarking on real workloads is what improves vLLM and open source inference.

Next up: more upstream work and a blog with a full technical deep dive later this week. Stay tuned. 🚀

[SemiAnalysis @SemiAnalysis_]

Please answer honestly and accurately

[vLLM @vllm_project]

🎉 Congrats to @deepseek_ai on open-sourcing the official DeepSeek-V4-Flash-0731 release!

A sparse MoE with 256 routed experts and six active per token, a 1M-token context window, and three reasoning-effort levels. Built for code agents and tool use. 🤖

Same architecture as the DSpark preview checkpoint, so your serving config carries over. And the DSpark draft module ships inside these weights, so speculative decoding is one flag instead of a second model to load:

--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'

Model: https://t.co/nH83pfMdfw

[Zachary Xi @zlxi02]

@zixi_qi_ the goat

[vLLM @vllm_project]

vLLM hit new peak bs=1 decode on Kimi-K3: 464 tok/s 🚀

Under a low-entropy reasoning workload, Kimi-K3 + DSpark on vLLM reaches 464 tok/s on batch size 1 with 4×4 GB300.

This benchmark is fully reproducible with public image: vllm/vllm-openai:kimi-k3 and @inferact's DSpark draft model linked in the thread.

1/3

[Simon Mo @simon_mo_]

https://t.co/RU2EI6GkyB

[Zachary Xi @zlxi02]

insane work from the team - it's been a crazy past few days

[vLLM @vllm_project]

With Kimi K3 Day-0 on vLLM: Open Frontier Intelligence for Everyone 🚀

At 2.8 trillion parameters, Moonshot AI's Kimi K3 is one of the most powerful open-weight models ever released. Starting today, you can serve it on vLLM the moment the weights are public.

What K3 brings:
🧠 2.8T-parameter Mixture-of-Experts (16 of 896 experts active per token)
📚 1M-token context window
🛠️ Native multimodal understanding, including vision
⚡ Kimi Delta Attention: a hybrid of linear and full attention that makes million-token context affordable

Huge thank you to @Kimi_Moonshot AI for the model release and partnership, @inferact for leading the vLLM optimizations, and to our partners at @nvidia, @AMD, and the broader vLLM community.

1/6

[Zachary Xi @zlxi02]

thank you for having me! onto the next one 🫡

[vLLM @vllm_project]

We celebrated the return of the vLLM meetup series last night! 🎉 🎊

Shoutout to @simon_mo_ , @zlxi02, and @zijingliu__ for representing the community at the vLLM x @CrusoeAI x @inferact meetup. Their presentations covered the vLLM Roadmap, agentic workloads, and production vLLM serving.

It's always great to see the community gather in person to talk vLLM and open source inference.

Huge thank you to @CrusoeAI, @a16z, and @inferact for cohosting this event. Onto the next one! 🚀

[Zachary Xi @zlxi02]

🎊🎊🎊

[vLLM @vllm_project]

Announcing the first-ever vLLM Conference — hosted by @inferact at Ray Summit, Aug 24–26 in San Francisco 🎉🌉

This is where we'll get into the work pushing open, high-performance inference forward, such as:
🗺️ Where the vLLM roadmap is headed
⚡ Getting the most out of accelerators including NVIDIA, AMD, TPU
🔗 Wiring vLLM into training and serving pipelines
🚀 Running inference on production scale

The summit features speakers from Inferact, NVIDIA, AMD, Google TPU, Anyscale, PyTorch, Meta, Red Hat, and more 🎤

Come learn where the future of inference, open source, and AI is heading — and meet the leading builders driving it 👇

https://t.co/B72yDhG4ge

[Zachary Xi @zlxi02]

🎉 Nominations are open for the 2026 PyTorch Foundation Contributor Awards!

vLLM is proud to be part of the PyTorch Foundation hosted projects alongside PyTorch, DeepSpeed, Ray, Helion, and Safetensors.

If someone has made a big impact on vLLM through code, docs, reviews, releases, or community support, consider nominating them!

https://t.co/Rtcozy02KA

[PyTorch @PyTorch]

Nominations are officially open for the 2026 PyTorch Foundation Contributor Awards.

These awards celebrate outstanding impact across PyTorch Foundation hosted projects, including PyTorch, vLLM, DeepSpeed, Ray, Helion, and Safetensors.

Whether it be through code development, documentation improvements, mentorship or community leadership - we want to recognise the individuals who help move our ecosystem forward.

Submit your nomination by July 17. Link in comments

[Zachary Xi @zlxi02]

Good read from @SemiAnalysis_ on DeepSeek V4 deployment from day 0 onward.

Useful context for vLLM users on how real deployment data drives serving-stack iteration after launch. ⚡

https://t.co/SMNyTj7vjw