Jee Jee Li

@inferact | @vllm_project

[Jee Jee Li @jeejeeleee]

@inferact 能飞 🚀🚀🚀@vllm_project 能飞🚀🚀🚀

[Kaichao You @KaichaoYou]

Inferact is taking off 🛫 vLLM is taking off 🛫
Our team is full of superheroes!!!
After months of hard work, our major project is finally public!

We’re committed to tackling the hardest challenges in AI inference—working closely with model vendors to optimize token quality and with sovereign AI partners to perfect deployment. Even under strict constraints, we can dramatically increase token throughput and deliver substantial economic value. 🤩🤩🤩

If you’re passionate about inference technology, come join us. If you need high-quality tokens, let’s talk. And if you have compute resources, we’d love to collaborate! 😁😁😁

[vLLM @vllm_project]

Thank you to the @SemiAnalysis_ team for the shoutout and for the collaboration on AgentX 🙏 Benchmarks are only useful when they measure the workloads people actually run, and AgentX measures the real thing: multi-turn, long-context agent traffic.

vLLM is the engine for production agentic workloads. For teams serving tokens at scale, revenue depends on optimized inference over long multi-turn contexts.

Our AgentX deep dive blog is coming soon! Stay tuned 📖

[SemiAnalysis @SemiAnalysis_]

Shoutout to the cracked team at @vllm_project that implemented recent agentic workload optimizations. (1/5)🧵

[vLLM @vllm_project]

🎉 Congrats to @deepseek_ai on DeepSeek-V4-Flash-Vision-Exp, the first multimodal model in the V4 family! vLLM serves it now.

A vision encoder and aligner on the 285B/13B V4-Flash MoE backbone, with text-agent scores holding steady. 🚀

🔗 https://t.co/sxl7k5w9rs

[Jee Jee Li @jeejeeleee]

@vllm_project @deepseek_ai We're also actively pushing to land the related PR — see: https://t.co/5Sv1mF7oEJ

[vLLM @vllm_project]

New blog! 🚀 MTP, EAGLE-3, DFlash or DSpark, which speculative decoding method should you actually use?

There’s no universal winner. The best choice changes with the model, workload, and speculation depth.

We break down how 5 methods work, how to enable and tune them in vLLM, and benchmark them across Gemma, Qwen, Kimi and MiniMax on @AMD Instinct MI300X & MI355X.

Deep dive 👇
https://t.co/CsTx2MvoPx

[Jee Jee Li @jeejeeleee]

@khluu000 master K 👍

[vLLM @vllm_project]

vLLM v0.28.0 is out! 584 commits from 270 contributors (76 new). 🎉

Highlights:

🌙 A stack-wide optimization push for Kimi-K3
🐳 DeepSeek-V4 sparse MLA now works end to end for plain decode, MTP and DSpark
⚡ Speculative decoding adds DFlash2 and DSpark confidence-scheduled verification
🛠️ Model Runner V2 picks up E/P/D disaggregation and weight offloading
🗄️ Tiered KV offloading gains a disk tier and out-of-tree secondary tiers
🆕 New models: Muse Glimmer, Ling 3.0 Flash, Dots3 NOTE, Interns2mobius

Thread 👇

[vLLM @vllm_project]

On Monday before the vLLM Conference, we co-hosted the vLLM x NVIDIA Dynamo meetup. We had room for 300 people and got 1,600 signups. Incredible turnout from this community. 🎉

Talks:
🔷 Zachary Xi and Zijing Liu (Inferact) talked about the State of vLLM and agentic serving in production
🔷 Harry Kim, Akshatha Kamath, and Anish Maddipoti (NVIDIA) talked about Dynamo and distributed serving

There were great conversations over food and drinks with people building across the inference stack, and a special focus on the vLLM and Dynamo relationship for production inference.

Thank you to @inferact and @NVIDIAAI for organizing! More meetups to come soon.

[vLLM @vllm_project]

Qwen3.8-Flash-Next from @Alibaba_Qwen has day-0 support in vLLM, verified on NVIDIA and AMD GPUs. 🎉

Ultra-sparse multimodal MoE: 125B params, 6B active, 262K native, 1M via YaRN. On top of those sits a separate 51B N-gram table you can offload.

Most of it will look familiar. The Gated DeltaNet layers reuse the KV path vLLM has had since Qwen3-Next: only a quarter of the layers hold a growing KV cache. Keep the 51B table in host RAM instead of HBM with VLLM_PLE_CPU_OFFLOAD=1. Qwen Sparse Attention is where the new engine work went. For now the model runs from vllm/vllm-openai:qwen38-flash-next.

Thanks to @Alibaba_Qwen for the weights, and for opening them this early! 🙌

🔗 https://t.co/NPnI6LNrGA

[vLLM @vllm_project]

🎉 Congrats to @Zai_org on GLM-5.3-Flash, the first natively multimodal model in the GLM-5 series, and their first hybrid of sparse and linear attention.

320B total, 18B active, 45 layers where GLM-4.5 had 92. Linear attention carries local dependencies as state, sparse attention pulls global context back through a lightweight indexer, and IndexPool weights four indexer key vectors into one to keep that indexer cheap at 1M tokens. mHC on top, for scaling.

vLLM already had each half of that mix. This is the first model to ask for both at once, and it has day-0 support, verified on @NVIDIA and @AMD GPUs.

🔗 https://t.co/RKaHPa7KAG

[Jee Jee Li @jeejeeleee]

Kimi-K3 was served (locally) with @vllm_project by @inferact😎

[Thien Tran @gaunernst]

Recently I tried using K3 for Triton kernels and I was surprised by how good it was. The generated kernels are definitely very different from how Codex would have done it.
Not a proper experiment but I asked Codex (Sol medium) and K3 (high) to work a the same problem. K3 came out on top by a large margin.
@Kimi_Moonshot @KimiDevs
Kimi-K3 was served (locally) with @vllm_project by @inferact 😎

[Jee Jee Li @jeejeeleee]

🚀🚀🚀

[vLLM @vllm_project]

This weight transfer engine is native to vLLM. Any Ray-based trainer can adopt it with a single WeightSource iterator.

Great work here!

🔗 https://t.co/Lpcxfxam3R

[Jee Jee Li @jeejeeleee]

🚀🚀🚀

[vLLM @vllm_project]

One model, four tricks:
🎬 text → video with synced sound
🖼️ first/first+last frame → video with audio
🎤 image + audio → lip-synced clip
🎥 green-screen footage + background video → auto relight & composite

🫶Thanks @MiniMax_AI for the shoutout on vLLM-Omni! Check out our recipe to get started👇
https://t.co/iWylcIkV9u

[vLLM @vllm_project]

Congratulations to @SemiAnalysis_ on the release of AgentX 1.0 🎊, an open-source multi-turn agentic coding benchmark collected from ~$3M of real traces, running on 1000+ chips and ~2MW of continuously operated compute.

We are excited to see @vllm_project’s competitive performance on frontier open models:
🔷130,093 tok/s/chip for DeepSeek V4 Pro
🔷 77,079 tok/s/chip for Minimax M3
🔷 12,479 tok/s/chip for Kimi K3.

The following thread covers an overview of the work from @vllm_project and @inferact: what we tested, found, and shipped upstream. This work highlights vLLM’s performance on real-world workloads and our committed focus to making vLLM an agentic-first engine.

Optimizing AgentX performance meant tackling three major challenges: prefix reuse, efficient long context parallelism, and scaling performance with PD disaggregation.

First, long agentic sessions stress prefix caching and KV cache offloading. Modern hybrid models have greatly reduced the required KV cache sizes, and caching every block boundary still saturates the KV cache pool which causes prefix cache thrashing across sessions.

The fix was sparse retention: one state per interval-sized segment plus the latest replay boundary (vllm-project/vllm #43447, #45845), then preserving shared-prefix boundaries so the interval can go to 0 for agent sessions (#47782). That gets us >95% hit rate at 14 concurrent requests with contexts to 1M.

The bigger structural change in KV cache offloading was making the shared KV pool distributed. With Mooncake Store as a first-class connector, prefill ranks can hit the prefix cache both within a worker and across workers, so we no longer have to trade cache locality against load balance to keep a cluster busy. Session-aware routing (48048) is what lets the router act on it.

For a single node deployment, SimpleCPUOffloadConnector has been greatly improved to support all hybrid model architectures and across both CUDA and ROCm platforms. For DeepSeek V4 Pro on ROCm, this implementation gave +81.7% output throughput and 46.6% lower mean e2e latency versus recomputing the prefix.

Kimi K3 at 2.8T barely fits on a single node, and squeezing it in leaves almost no headroom for KV cache, which is exactly what a long multi-turn session needs most. Parallelism strategy matters more here than on any other model we tested.

TP8/DCP8 tops the K3 agentic frontier across configs, with B300 vLLM peaking at ~12.5k tok/s/chip at ~8 tok/s/user and GB300 NVL72 on Dynamo + vLLM holding the curve out past 200 tok/s/user. K3 also surfaced a routing bug worth pulling: vllm-project/router#194 fixes the router dropping reasoning_content, which hits any reasoning model served behind it.

On MiniMax M3, B200 vLLM reaches ~44k tok/s/chip and vLLM leads TRT-LLM on throughput vs p90 TTFT. M3 and Qwen3.5 both shipped day-0.

Finally, fully optimized performance requires scaling the deployment with prefill-decode disaggregation and distributed KV cache offloading. Thanks to vLLM’s MultiConnector, this is natively supported with NIXL PD connector + MooncakeStoreConnector. By rate matching to find the ideal prefill-decode ratio, we achieved 4.45x higher throughput at a 60 tok/s interactivity for DeepSeek V4 Pro on GB300 Dynamo compared to B300.

Shoutout to @NVIDIAAI, who co-tuned most of these configs with us. Dynamo's router optimizations took AgentX replay time down 23.7% on the vLLM backend, the AIPerf replay harness is what made the traces runnable at all, and NIXL and their kernels sit under each respective performance point.

Shoutout to @AIatAMD as well: the team’s AITER sparse-MLA decode selection led to +5.22% AgentX output throughput; the hybrid AITER/native CSA selector led to 1.21–1.76x e2e performance boost, and the team made prebuilt lmcache gfx942/gfx950 wheels and a published Mooncake ROCm wheel.

Thank you to @SemiAnalysis_ for building AgentX and for the collaboration throughout. Agentic workloads are what users serve in real production, and benchmarking on real workloads is what improves vLLM and open source inference.

Next up: more upstream work and a blog with a full technical deep dive later this week. Stay tuned. 🚀

[vLLM @vllm_project]

vLLM Conference is next week, and we have a packed schedule 🎊 📅. Here's the full list of events you should know:

Mon:
🔷 4–6PM vLLM × Ray × Google Cloud Happy Hour: https://t.co/Exw10SVIrV
🔷 6–9PM vLLM × Dynamo Meetup: https://t.co/id6TaOfIMs

Tue:
🔷 11–11:30AM vLLM Keynote from @simon_mo_
🔷 12–5PM vLLM Track Day 1
🔷 6–9PM vLLM × AMD Happy Hour: https://t.co/RVXTxGq0Lz

Wed:
🔷 12–5PM vLLM Track Day 2
🔷 6–8:30PM vLLM x DigitalOcean × NVIDIA Happy Hour: https://t.co/U1FSK6aYhZ

No ticket needed for the happy hours and meetups, but space is limited. To join the full event, register here: https://t.co/bTDMkOOwIR

[vLLM @vllm_project]

In RL training, a vLLM rollout engine and a Megatron trainer can run the same policy yet disagree on a token's logprob due to floating-point non-associativity.

SkyRL's IsoExec combines an execution contract with a unified model, aligning rounding-sensitive execution choices across rollout and training.

Bitwise parity holds across different TP, EP, and SP layouts. For Gated DeltaNet, the chunkwise-parallel recurrent algorithm makes parallel training and prefill bitwise identical to recurrent decode.

Qwen3.5-35B-A3B, DAPO, 8xH100, 50 steps: logprob diff 1.6e-2 to 6.7e-7, full-step overhead 25.3% ✅ vLLM's scheduler and CUDA graphs still apply.

Thanks to @JiangAlexander1 and the SkyRL team at @NovaSkyAI.

🔗 https://t.co/Y8SIIgHqAr

[vLLM @vllm_project]

We're capping Day 1 of the first vLLM Conference with a happy hour hosted by @AMD and @inferact 🎉!

We'll have engineers from AMD, Inferact and the vLLM community for casual conversations alongside drinks and food.

📅Tue Aug 25, 6-9pm, San Francisco.
No conference ticket needed, RSVP below!

[Jee Jee Li @jeejeeleee]

Feels like @ornith_ brought these models, and you can directly use @vllm_project recent wheel or docker images to deploy these models.
https://t.co/ug4DYHlWYy

[Ornith @ornith_]

Aloha! 🌺Introducing Ornith-1.5, a family of open-source LLMs spanning 9B Dense, 35B MoE, and 397B MoE, trained with self-improving strategies.

It achieves state-of-the-art performance among open-source models of comparable size and delivers performance comparable to Claude Opus 4.8 across reasoning, agentic, and coding tasks:
✅Terminal-Bench 2.1 (86.1)
✅SWE-Bench (86 on verified, 65.1 on pro, 79.6 on Multilingual)
✅DeepSWE (56)
✅HLE (44.6)
✅ClawEval (81.4)
✅Tool Decathlon (71.2)

Ornith-1.5 takes a major step toward training foundation models through end-to-end self-improvement, extending the self-scaffolding strategies introduced in Ornith-1.0 into a more complete self-improvement loop: the model proposes new tasks, generates task-specific scaffolds, and produces solution rollouts for reinforcement learning, continuously creating new learning experiences from which it can improve.

All models, along with their quantized versions (FP8, GGUF, MLX, and NVFP4), have been released under the MIT License, enabling unrestricted commercial and research use.
📘Tech Blog: https://t.co/OZ63scRWLB
🤗Huggingface: https://t.co/mGJLwhrQOM

[Jee Jee Li @jeejeeleee]

🚀🚀🚀

[Tiezhen WANG @Xianbao_QIAN]

DFlash 2 is available on @vllm_project with https://t.co/cUWahsWstG

[Jee Jee Li @jeejeeleee]

@vllm_project builds day-0 model docker images for everyone.🚀🚀🚀

[Omer Shlomovits @OmerShlomovits]

https://t.co/SJ1baCWpr8