Jeff Ma

PhD student @Umich | @vllm_project @inferact
@ml_energy |

[SemiAnalysis @SemiAnalysis_]

Shoutout to the cracked team at @vllm_project that implemented recent agentic workload optimizations. (1/5)🧵

[Tan TJian @Rxday000]

Agents just made years of inference optimization matter even more.

At the first vLLM Conference: KV cache, prefix reuse, disaggregation, scheduling → core to keeping agents running efficiently.

Qwen + GLM dropped mid-conference too.

When maintainers meet, we go faster.

[Roger Wang @rogerw0108]

Why benchmarks and metrics for LLM inference are not easy to get done right!

This also shows the efficiency of the system is no longer just about the runtime; the entire stack matters!

[Jeff Ma @jeffjma]

🚀🚀🚀

[vLLM @vllm_project]

Congratulations to @SemiAnalysis_ on the release of AgentX 1.0 🎊, an open-source multi-turn agentic coding benchmark collected from ~$3M of real traces, running on 1000+ chips and ~2MW of continuously operated compute.

We are excited to see @vllm_project’s competitive performance on frontier open models:
🔷130,093 tok/s/chip for DeepSeek V4 Pro
🔷 77,079 tok/s/chip for Minimax M3
🔷 12,479 tok/s/chip for Kimi K3.

The following thread covers an overview of the work from @vllm_project and @inferact: what we tested, found, and shipped upstream. This work highlights vLLM’s performance on real-world workloads and our committed focus to making vLLM an agentic-first engine.

Optimizing AgentX performance meant tackling three major challenges: prefix reuse, efficient long context parallelism, and scaling performance with PD disaggregation.

First, long agentic sessions stress prefix caching and KV cache offloading. Modern hybrid models have greatly reduced the required KV cache sizes, and caching every block boundary still saturates the KV cache pool which causes prefix cache thrashing across sessions.

The fix was sparse retention: one state per interval-sized segment plus the latest replay boundary (vllm-project/vllm #43447, #45845), then preserving shared-prefix boundaries so the interval can go to 0 for agent sessions (#47782). That gets us >95% hit rate at 14 concurrent requests with contexts to 1M.

The bigger structural change in KV cache offloading was making the shared KV pool distributed. With Mooncake Store as a first-class connector, prefill ranks can hit the prefix cache both within a worker and across workers, so we no longer have to trade cache locality against load balance to keep a cluster busy. Session-aware routing (48048) is what lets the router act on it.

For a single node deployment, SimpleCPUOffloadConnector has been greatly improved to support all hybrid model architectures and across both CUDA and ROCm platforms. For DeepSeek V4 Pro on ROCm, this implementation gave +81.7% output throughput and 46.6% lower mean e2e latency versus recomputing the prefix.

Kimi K3 at 2.8T barely fits on a single node, and squeezing it in leaves almost no headroom for KV cache, which is exactly what a long multi-turn session needs most. Parallelism strategy matters more here than on any other model we tested.

TP8/DCP8 tops the K3 agentic frontier across configs, with B300 vLLM peaking at ~12.5k tok/s/chip at ~8 tok/s/user and GB300 NVL72 on Dynamo + vLLM holding the curve out past 200 tok/s/user. K3 also surfaced a routing bug worth pulling: vllm-project/router#194 fixes the router dropping reasoning_content, which hits any reasoning model served behind it.

On MiniMax M3, B200 vLLM reaches ~44k tok/s/chip and vLLM leads TRT-LLM on throughput vs p90 TTFT. M3 and Qwen3.5 both shipped day-0.

Finally, fully optimized performance requires scaling the deployment with prefill-decode disaggregation and distributed KV cache offloading. Thanks to vLLM’s MultiConnector, this is natively supported with NIXL PD connector + MooncakeStoreConnector. By rate matching to find the ideal prefill-decode ratio, we achieved 4.45x higher throughput at a 60 tok/s interactivity for DeepSeek V4 Pro on GB300 Dynamo compared to B300.

Shoutout to @NVIDIAAI, who co-tuned most of these configs with us. Dynamo's router optimizations took AgentX replay time down 23.7% on the vLLM backend, the AIPerf replay harness is what made the traces runnable at all, and NIXL and their kernels sit under each respective performance point.

Shoutout to @AIatAMD as well: the team’s AITER sparse-MLA decode selection led to +5.22% AgentX output throughput; the hybrid AITER/native CSA selector led to 1.21–1.76x e2e performance boost, and the team made prebuilt lmcache gfx942/gfx950 wheels and a published Mooncake ROCm wheel.

Thank you to @SemiAnalysis_ for building AgentX and for the collaboration throughout. Agentic workloads are what users serve in real production, and benchmarking on real workloads is what improves vLLM and open source inference.

Next up: more upstream work and a blog with a full technical deep dive later this week. Stay tuned. 🚀

[Jeff Ma @jeffjma]

It’s finally here! Very proud to have contributed, and kudos to @SemiAnalysis_ and @inferact @vllm_project 🚀🚀

[SemiAnalysis @SemiAnalysis_]

AgentX - InferenceXv3:
Does CUDA Moat Hold up in Agentic Inferencing?
$3 Million USD dataset open sourced,
1 Mil+ Context Length, Multiturn,
Sub Agents 95%+ KVCache HitRate,
GB300 NVL72, MI355, B200

https://t.co/PgtKNYXdFr

[Amber (Jiachen) Liu 🍊 @JIACHENLIU8]

https://t.co/PTM5uKBmrd

[SemiAnalysis @SemiAnalysis_]

The @vllm_project maintainers at @inferact 🚀 are some of the most cracked engineers in the world. They’re building one of the inference engines that powers much of the world’s intelligence—and doing so with remarkable dedication, kindness, and hard work.

[Inferact @inferact]

With @inferact, @vllm_project is now officially verified by @Kimi_Moonshot to serve Kimi K3 with full accuracy, according to the Kimi Vendor Verifier.

As the leading inference engine, vLLM delivers best results across vision, 1M context, and agentic coding; with prod quality.

[Jeff Ma @jeffjma]

@simon_mo_ 🚀

[Yifan Qiao @yifandotqiao]

Kimi K3 x vLLM!
Countless hours building and optimizing together with @Kimi_Moonshot to make vLLM the open frontier infrastructure powering Kimi K3’s open frontier intelligence.
Enjoy! 🫡🚀

[Jeff Ma @jeffjma]

Such an amazing experience being part of the K3 support, witnessing prefix caching support, DSpark training, kernel optimizations, and so much more come together. Great work from a great team!

[vLLM @vllm_project]

With Kimi K3 Day-0 on vLLM: Open Frontier Intelligence for Everyone 🚀

At 2.8 trillion parameters, Moonshot AI's Kimi K3 is one of the most powerful open-weight models ever released. Starting today, you can serve it on vLLM the moment the weights are public.

What K3 brings:
🧠 2.8T-parameter Mixture-of-Experts (16 of 896 experts active per token)
📚 1M-token context window
🛠️ Native multimodal understanding, including vision
⚡ Kimi Delta Attention: a hybrid of linear and full attention that makes million-token context affordable

Huge thank you to @Kimi_Moonshot AI for the model release and partnership, @inferact for leading the vLLM optimizations, and to our partners at @nvidia, @AMD, and the broader vLLM community.

1/6

[ML ENERGY @ml_energy]

Energy and power are first-class resources in scaling AI compute.

https://t.co/uq5rWvY6jB builds open-source infrastructure for measuring, understanding, and optimizing the energy use of ML workloads.

Start here:
- https://t.co/dlMO31ZCLT
- https://t.co/51t3ymSyEX

[Jeff Ma @jeffjma]

@woosuk_k @inferact @vllm_project Congrats!

[Jae-Won Chung @jaewon_chung_cs]

ML Power & Energy Tutorial @NeurIPSConf 2025
Tuesday, December 2nd, 9:30 AM - 12:00 PM

We're excited to present the tutorial "Energy and Power as First‑Class ML Design Metrics" at NeurIPS ’25 together with NVIDIA!

More details in our tutorial homepage: https://t.co/rgZwXY7t70

[Jeff Ma @jeffjma]

@_thomasip @jaewon_chung_cs We already support streaming audio output for Qwen Omni, and we are actively working on supporting streaming input as well!

[Jeff Ma @jeffjma]

@cyodyssey 也是小红书过来的，昨天就私信了。可以回一下吗siyuan哥