Robin Cheng

MTS @Inferact

[Roger Wang @rogerw0108]

Personally I believe open research that benefits the broader OSS community deserves encouragement, not dismissal.

Academia → production is one of the hardest jumps in MLSys and the FastVideo team did it with limited compute. Efforts like this deserve support, not trash talk.

[Robin Cheng @rchpmv]

That's quite accusative. Making the right dashboard is genuinely difficult; the same number means different things to different people in different contexts. SA and the vLLM community generally strives to present numbers with as much transparency as possible so I'm sure they will take this feedback into account and clarify the intent here. I think the intent here is to capture real world performance, as having the right serving infra with the right kv cache management matters a great deal in agentic use cases, sometimes having even more impact than the engine itself.

[vLLM @vllm_project]

The vLLM Conference is coming up in 3 weeks! 🎉

Come learn about the current state and future of AI inference, Aug 24–26 in San Francisco 🌉, hosted by @inferact at @anyscalecompute Ray Summit.

We'll have speakers from Inferact, NVIDIA, AMD, Google TPU, Anyscale, PyTorch, Meta, Red Hat, and key builders around vLLM. The talks on the roadmap deep dive into the latest on accelerators, training and serving pipelines, and production-scale inference 🚀

[Simon Mo @simon_mo_]

https://t.co/RU2EI6GkyB

[Robin Cheng @rchpmv]

@ekzhang1 You're right! I'd add batch size to it among other things (but then it becomes the Pareto curve). A lot of this is also highly dependent on workload: gains from MTP can vary hugely (~2x-4x) based on the topic; KV cache hit rate and average time-to-eviction matter more than size.

[Robin Cheng @rchpmv]

@vllm_project The best part of this is that whenever vLLM posts numbers, they are real, not inflated, and not cherry-picked. That's actually quite rare in the industry. If vLLM gives us a number, we can trust it's reproducible and real.

[Robin Cheng @rchpmv]

Open source is exploding. It's so amazing to be part of this community!

[vLLM @vllm_project]

With Kimi K3 Day-0 on vLLM: Open Frontier Intelligence for Everyone 🚀

At 2.8 trillion parameters, Moonshot AI's Kimi K3 is one of the most powerful open-weight models ever released. Starting today, you can serve it on vLLM the moment the weights are public.

What K3 brings:
🧠 2.8T-parameter Mixture-of-Experts (16 of 896 experts active per token)
📚 1M-token context window
🛠️ Native multimodal understanding, including vision
⚡ Kimi Delta Attention: a hybrid of linear and full attention that makes million-token context affordable

Huge thank you to @Kimi_Moonshot AI for the model release and partnership, @inferact for leading the vLLM optimizations, and to our partners at @nvidia, @AMD, and the broader vLLM community.

1/6

[Inferact @inferact]

Inferact is proud to co-sign. We were founded to grow @vllm_project into the world’s leading open-source inference engine, making open-weight AI faster, cheaper, and accessible across hardware and clouds so intelligence can be on tap for everyone.

[Robin Cheng @rchpmv]

@Justin_Halford_ @mooncat_is This is a good point except that model abliteration gets increasingly hard with modern large models. It's hard to abliterate them without also rendering the model less capable.

[vLLM @vllm_project]

🎉 Meet vLLM-Omni v0.22.0, a major upgrade for omnimodal world models and production-grade multimodal serving.

🌍 Day-0 @NVIDIAAI Cosmos 3 world models: text, image, audio, video, and action, in and out.
🤖 Robot serving: DreamZero + OpenPI realtime API.
🎙️ Production TTS: Qwen3-TTS, Qwen3-Omni, VoxCPM2 and more.
🎨 Faster image/video/diffusion: Wan 2.2, HunyuanVideo 1.5, LTX-2.3.
⚡ Broader quantization (FP8/INT8, MXFP4/MXFP8, W4A16, ModelOpt) and hardware coverage.

339 commits, 124 contributors, 52 of them new. Thank you all. 🙌

🔗 https://t.co/76ttSM6FHs

[Andrew Ng @AndrewYNg]

New course on serving LLMs efficiently -- how do you serve models to many concurrent users at low latency and reasonable cost? This short course is built with @RedHat and taught by @cedricclyburn.

Efficient LLM serving requires efficient memory management. A 70B-parameter model takes ~140 GB just to load the weights. On top of that, every active request needs its own chunk of GPU memory, the KV cache, to store the token context it has built up so far. In this course, you'll learn to reduce a model's memory footprint with quantization and serve it using vLLM, which handles many concurrent requests efficiently through smart memory management.

Skills you'll gain:
- Quantize a model and measure the accuracy tradeoff
- Serve a model with vLLM and watch it handle concurrent requests efficiently
- Benchmark your deployment and make informed tradeoffs between speed, cost, and accuracy

Join and learn to serve LLMs efficiently:
https://t.co/x04xMbFlkO

[Sumanth Hegde @sumanthrh]

Excited to share some of our work on improving vLLM for RL!

A number of RL frameworks, including SkyRL, use vLLM for inference, and we’ve noticed some common problems:
1. Weight syncing between training and inference is implemented in an ad-hoc fashion and duplicated across frameworks.
2. Asynchronous RL is prone to break at scale, especially in P/D and DPEP deployments.

We’ve been working on improving both!

[Matej Sirovatka @m_sirovatka]

prime-rl is vLLM native - we are using vLLM as our engine of choice and the work the team does is legit amazing. It powers our large scale runs and we couldn't be happier with the collaboration we're having with the vLLM team 🙏

[Robin Cheng @rchpmv]

Inference really *is* hard. But really exciting to be solving these problems at the frontier.

[Robin Cheng @rchpmv]

@mobicham --use-vllm-instead ;)

[Robin Cheng @rchpmv]

@bcherny Claude code being so expensive is kind of the bigger problem though... Codex performs better and is way cheaper so I never even *need* to wonder where my token usages went even when I hammer codex with multiple sessions all day...

[Kaichao You @KaichaoYou]

Most people think new model support is just porting a reference implementation to vLLM. It is not.

More often than not, we need to work with model vendors to figure out if anything goes wrong, and correct the reference implementation together, when needed.

That's why people love and trust vLLM. vLLM has the best serving capability to serve open-source models accurately and efficiently excluding the model vendors themselves.

[vLLM @vllm_project]

vLLM v0.20.0 is here! 752 commits from 320 contributors (123 new). 🎉

Highlights: DeepSeek V4, Hunyuan v3 preview support, CUDA 13 / PyTorch 2.11 / Transformers v5 baseline, FA4 as default MLA prefill, TurboQuant 2-bit KV (4× capacity), vLLM IR foundation.

Thread 👇

[NVIDIA AI @NVIDIAAI]

📊 Day 0 performance is here: DeepSeek-V4-Pro running on NVIDIA Blackwell Ultra.

Using @vllm_project's Day 0 recipe, we’ve captured the initial performance Pareto for DeepSeek’s flagship 1M long-context model. This curve highlights the baseline for balancing AI factory throughput with real-time user interactivity.

This is just the starting line. Expect these numbers to climb as we optimize the full co-design stack, including:

• NVFP4 & Dynamo
• Optimized CUDA kernels
• Advanced parallelization techniques and beyond

Read the full technical deep dive: https://t.co/KUT2jJzkTi

[vLLM @vllm_project]

🎉 Day-0 support for @deepseek_ai V4 Pro and Flash on vLLM — a new generation of DeepSeek model, purpose-built for tasks up to 1M tokens. Alongside the release, we're publishing a first-principles walkthrough of the new long-context attention and how we implemented it in vLLM.

The new attention mechanism, in four moves:
• Shared K/V + inverse RoPE → 2× memory savings
• c4a / c128a KV compression → 4×–128× savings
• DeepSeek Sparse Attention over compressed tokens
• Short sliding window for locality across compression boundaries

At 1M context, per-layer KV state is ~8.7× smaller than a DeepSeek V3.2-style 61-layer stack (9.62 GiB vs 83.9 GiB, bf16). fp8 attention cache + fp4 indexer cache shrink it further.

vLLM side:
• Unified hybrid KV cache — single logical block size (256 native positions) across all compression rates; compressor state folded into the SWA KV cache spec so prefix caching, disagg prefill, CUDA graphs and MTP reuse the same abstraction
• Three page-size buckets for the full 5-way cache stack → no cross-kind fragmentation
• Fused kernels: compressor + RMSNorm + RoPE + cache insert (1.4–3×), inverse RoPE + fp8 quant (2–3×), Q-norm + KV RoPE + K insert (10–20×)
• Multi-stream overlap of indexer vs main-KV compression vs SWA insertion

Disaggregated serving is supported out of the box and strongly recommended for best performance.

Follow our recipes site for verified commands for @nvidia Blackwell (B200, B300, GB200, GB300) and Hopper (H100/H200/H20) systems.

Thanks to the @deepseek_ai team for open-sourcing DeepSeek V4, and to @inferact for landing day-0 support 🤝

📝 Blog: https://t.co/Eh7vk6xVJy
📖 Recipes: https://t.co/jlWuzYyZeX
🤗 https://t.co/IA9qAysqJk