ZIJING LIU

Making LLM inference faster and more affordable @inferact
ex @Meta MSL PyTorch

[ZIJING LIU @zijingliu__]

@gaunernst @yifandotqiao @woosuk_k When should we have a TPU tech share🫡

[ZIJING LIU @zijingliu__]

TPU as it is 🫡

[Inferact @inferact]

Our first TPU megakernel for Kimi K3 reaches 709 tokens/s on low-concurrency decode, against 450 tokens/s for our GB200 baseline, both with DSpark speculative decoding.

To our knowledge, this is the first TPU inference megakernel. The whole model runs in a single Pallas kernel, and without spec decoding it is roughly 1.4 to 2x the GB200 baseline at batch sizes 1 through 8.

We are open sourcing it today.

1/2

[ZIJING LIU @zijingliu__]

Critical step for us! It’s important that inference is reliable and efficient on all hardware and platform!

[Inferact @inferact]

@googlecloud and Inferact are announcing today a partnership to make TPU a first-class citizen in @vllm_project.

This partnership puts both teams on one engineering roadmap to bring TPU to the broader open model ecosystem, optimizing vLLM as the agentic production serving engine for TPU:

• Production serving features and optimized kernels
• A native PyTorch path via TorchTPU
• Moving towards day-0 support for frontier model releases

We're also launching a community program: shared TPU capacity for open-source contributors, plus dedicated review and design help from the core vLLM maintainers at Inferact.

Everything this collaboration produces is open source. Read the full announcement: https://t.co/asTJSktyQK

[ZIJING LIU @zijingliu__]

🫡🫡🫡

[Roger Wang @rogerw0108]

Pick wisely.

[ZIJING LIU @zijingliu__]

@gaunernst @Kimi_Moonshot how you compare k3 with gpt5.6 or fable? are they better but with much higher pricing or just pretty much perform the same?

[ZIJING LIU @zijingliu__]

It has been a great night talking to all the people and learn from different perspectives on inference.

[vLLM @vllm_project]

On Monday before the vLLM Conference, we co-hosted the vLLM x NVIDIA Dynamo meetup. We had room for 300 people and got 1,600 signups. Incredible turnout from this community. 🎉

Talks:
🔷 Zachary Xi and Zijing Liu (Inferact) talked about the State of vLLM and agentic serving in production
🔷 Harry Kim, Akshatha Kamath, and Anish Maddipoti (NVIDIA) talked about Dynamo and distributed serving

There were great conversations over food and drinks with people building across the inference stack, and a special focus on the vLLM and Dynamo relationship for production inference.

Thank you to @inferact and @NVIDIAAI for organizing! More meetups to come soon.

[vLLM @vllm_project]

On Monday before the vLLM Conference, we co-hosted the vLLM x NVIDIA Dynamo meetup. We had room for 300 people and got 1,600 signups. Incredible turnout from this community. 🎉

Talks:
🔷 Zachary Xi and Zijing Liu (Inferact) talked about the State of vLLM and agentic serving in production
🔷 Harry Kim, Akshatha Kamath, and Anish Maddipoti (NVIDIA) talked about Dynamo and distributed serving

There were great conversations over food and drinks with people building across the inference stack, and a special focus on the vLLM and Dynamo relationship for production inference.

Thank you to @inferact and @NVIDIAAI for organizing! More meetups to come soon.

[ZIJING LIU @zijingliu__]

Congrats on the launch! We are ready to serve with Applied Compute together!

[Applied Compute @appliedcompute]

Today, we're introducing AC2, the Applied Compute Agent Cloud, to enable every team to train, serve, and improve their own frontier models.

[vLLM @vllm_project]

vLLM Conference is next week, and we have a packed schedule 🎊 📅. Here's the full list of events you should know:

Mon:
🔷 4–6PM vLLM × Ray × Google Cloud Happy Hour: https://t.co/Exw10SVIrV
🔷 6–9PM vLLM × Dynamo Meetup: https://t.co/id6TaOfIMs

Tue:
🔷 11–11:30AM vLLM Keynote from @simon_mo_
🔷 12–5PM vLLM Track Day 1
🔷 6–9PM vLLM × AMD Happy Hour: https://t.co/RVXTxGq0Lz

Wed:
🔷 12–5PM vLLM Track Day 2
🔷 6–8:30PM vLLM x DigitalOcean × NVIDIA Happy Hour: https://t.co/U1FSK6aYhZ

No ticket needed for the happy hours and meetups, but space is limited. To join the full event, register here: https://t.co/bTDMkOOwIR

[ZIJING LIU @zijingliu__]

Facto 👌

[SemiAnalysis @SemiAnalysis_]

The @vllm_project maintainers at @inferact 🚀 are some of the most cracked engineers in the world. They’re building one of the inference engines that powers much of the world’s intelligence—and doing so with remarkable dedication, kindness, and hard work.

[ZIJING LIU @zijingliu__]

vLLM for production serving, stay tuned more to come!

[Inferact @inferact]

With @inferact, @vllm_project is now officially verified by @moonshot_ai to serve Kimi K3 with full accuracy, according to the Kimi Vendor Verifier.

As the leading inference engine, vLLM delivers best results across vision, 1M context, and agentic coding; with prod quality.

[vLLM @vllm_project]

Excited to host the first-ever vLLM Conference in SF in under 3 weeks alongside @anyscalecompute Ray Summit 2026!

We'll cover the State of vLLM, agentic serving, and where inference is heading. Sign up below! 🎟️

[a16z @a16z]

vLLM runs on half a million GPUs at any given moment. Most people have never heard of it.

Simon Mo, co-founder and CEO of @inferact and lead maintainer of vLLM, sits down with a16z’s Matt Bornstein and Elena Burger to discuss what it takes to actually run open models in production, the advantages of open models, Simon’s mission at Inferact, and more.

00:00 Intro
01:46 When open source became critical infrastructure
08:55 Day zero model releases, and the drama behind them
14:59 What Kimi K3 actually buys you
18:56 Why open model licenses are changing
22:24 The pharmaceutical analogy for funding model training
26:16 If GPUs got 99% cheaper
29:08 Why vLLM, OpenRouter, and Ollama all started before ChatGPT
35:42 Building a company on an open source project
39:48 The inventor of RoPE removing RoPE

@simon_mo_ @BornsteinMatt @VirtualElena

[vLLM @vllm_project]

The vLLM Conference is coming up in 3 weeks! 🎉

Come learn about the current state and future of AI inference, Aug 24–26 in San Francisco 🌉, hosted by @inferact at @anyscalecompute Ray Summit.

We'll have speakers from Inferact, NVIDIA, AMD, Google TPU, Anyscale, PyTorch, Meta, Red Hat, and key builders around vLLM. The talks on the roadmap deep dive into the latest on accelerators, training and serving pipelines, and production-scale inference 🚀

[ZIJING LIU @zijingliu__]

Great to see vLLM works for real production serving scale! It’s about not just performance, but also reliability and resilience towards unavoidable hardware failure.

[vLLM @vllm_project]

Kimi K3 has first class support for vLLM on NVIDIA!

vLLM serves K3 across Grace Blackwell, Blackwell, Hopper, and NVL72 systems, also integrating with Dynamo to keep a model this large, fast and efficient at production scale.

Available Day 0, the moment the weights went public.
@vllm_project × @NVIDIAAI × @Kimi_Moonshot

[vLLM @vllm_project]

With Kimi K3 Day-0 on vLLM: Open Frontier Intelligence for Everyone 🚀

At 2.8 trillion parameters, Moonshot AI's Kimi K3 is one of the most powerful open-weight models ever released. Starting today, you can serve it on vLLM the moment the weights are public.

What K3 brings:
🧠 2.8T-parameter Mixture-of-Experts (16 of 896 experts active per token)
📚 1M-token context window
🛠️ Native multimodal understanding, including vision
⚡ Kimi Delta Attention: a hybrid of linear and full attention that makes million-token context affordable

Huge thank you to @Kimi_Moonshot AI for the model release and partnership, @inferact for leading the vLLM optimizations, and to our partners at @nvidia, @AMD, and the broader vLLM community.

1/6

[vLLM @vllm_project]

Kimi K3 is here: a 2.8T-parameter MoE with a 1M-token context and native vision, running on vLLM from day 0.

Here is our canonical deployment guide: architecture, kernels, recipes, and the flags to run it in production. https://t.co/SATHaHHYef

[vLLM @vllm_project]

🦀 The Rust frontend is officially merged into vLLM!

As GPUs get faster, the frontend has become a real share of CPU time. The new Rust frontend is a drop-in alternative to the Python API server — same engine, same ZMQ boundary. Opt in with VLLM_USE_RUST_FRONTEND=1.

Early numbers: on a preprocess-heavy workload, ~837 req/s vs ~162 req/s for default Python — ~5x in a single process.

A few design choices we're excited about:
• Layered crates with clear boundaries
• Stream-native pipeline — non-streaming for free
• Builds on stable Rust

Huge thanks to @BugenZhao from @inferact for introducing the work at @PyTorch Meetup Singapore.

https://t.co/Tw8PoIjbH9

[ZIJING LIU @zijingliu__]

@simon_mo_ @profjoeyg @inferact @vllm_project Congrats!

[vLLM @vllm_project]

🚀 New on the @vllm_project blog: Serving Agentic Workloads at Scale with vLLM x Mooncake.

Agentic traces grow to 80K+ tokens with 94%+ reusable prefixes, but local KV caches evict them and cross-instance routing misses them.

By integrating Mooncake Store as a distributed KV cache pool, vLLM gets:

🚀 3.8x higher throughput
⚡ 46x lower P50 TTFT
⏱️ 8.6x lower E2E latency
📈 Cache hit rate 1.7% -> 92.2%
🌐 Scales near-linearly to 60 GB200 GPUs at >95% hit rate

🔥 Powered by a deep collaboration between @Inferact and @KT_Project_AI

📖 Read more: https://t.co/XIRtQ9pYVQ
🧵👇