Kevin Luu

Building @inferact @vllm_project | prev @raydistributed @anyscalecompute

[Kevin Luu @khluu000]

K3 powers 90% of my work now. Own your intelligence 🧑‍🎨

[Roger Wang @rogerw0108]

A few weeks ago we stood up our own multi-rack K3 deployment and I told everyone @inferact to dogfood it.

Nobody went back since then. It's better and faster at most of what we do daily, not to mention it never blocks us mid-task.

This is the power of open intelligence.

[Kevin Luu @khluu000]

@_seemethere

[Kevin Luu @khluu000]

@istdrc amazing!

[vLLM @vllm_project]

On Tuesday, capping off a full day of speakers at the first vLLM Conference, the vLLM community gathered for a rooftop happy hour at sunset. The event hit capacity, and sunset over the city was a great backdrop.

Thank you to @AMD and @inferact for co-hosting! More meetups to come soon.

[Simon Mo @simon_mo_]

@mgoin_ is the hero at vLLM helping the community always staying on the cutting edge of inference infrastructure!!

[Kevin Luu @khluu000]

🐐🐐🐐🐐🐐🐐@yifandotqiao @jeffjma @LiZhewen71800 @rogerw0108 and team!

[Roger Wang @rogerw0108]

🚀🚀🚀Shoutout to @SemiAnalysis_ for this tremendous effort on benchmarking real-world agentic workloads!

Building an efficient, performant and reliable inference system for real frontier agentic workloads is what we care about and stay laser-focused on at @vllm_project and @inferact. We don't ship features that look good on paper or benchmax on 8k/1k - we optimize for what matters on real workloads - TTFT, interactivity, and cache hit rate!

Check out the released results where vLLM delivers SOTA performance across normalized interactivity and cost efficiency!

[vLLM @vllm_project]

vLLM Conference is next week, and we have a packed schedule 🎊 📅. Here's the full list of events you should know:

Mon:
🔷 4–6PM vLLM × Ray × Google Cloud Happy Hour: https://t.co/Exw10SVIrV
🔷 6–9PM vLLM × Dynamo Meetup: https://t.co/id6TaOfIMs

Tue:
🔷 11–11:30AM vLLM Keynote from @simon_mo_
🔷 12–5PM vLLM Track Day 1
🔷 6–9PM vLLM × AMD Happy Hour: https://t.co/RVXTxGq0Lz

Wed:
🔷 12–5PM vLLM Track Day 2
🔷 6–8:30PM vLLM x DigitalOcean × NVIDIA Happy Hour: https://t.co/U1FSK6aYhZ

No ticket needed for the happy hours and meetups, but space is limited. To join the full event, register here: https://t.co/bTDMkOOwIR

[Kevin Luu @khluu000]

join us if you wanna be a cracked engineer just like Woosuk 🧠

[Woosuk Kwon @woosuk_k]

We are hiring! Come join us and advance the frontier of AI inference!

[SemiAnalysis @SemiAnalysis_]

The @vllm_project maintainers at @inferact 🚀 are some of the most cracked engineers in the world. They’re building one of the inference engines that powers much of the world’s intelligence—and doing so with remarkable dedication, kindness, and hard work.

[Kevin Luu @khluu000]

@__tinygrad__ @satyanadella @github Consider switching to @buildkite
They didn't pay me to say this. I've just been a happy customer for the past 2 years

[Kevin Luu @khluu000]

🚀🌕

[Inferact @inferact]

With @inferact, @vllm_project is now officially verified by @Kimi_Moonshot to serve Kimi K3 with full accuracy, according to the Kimi Vendor Verifier.

As the leading inference engine, vLLM delivers best results across vision, 1M context, and agentic coding; with prod quality.

[Kevin Luu @khluu000]

!!!

[vLLM @vllm_project]

The vLLM Conference is coming up in 3 weeks! 🎉

Come learn about the current state and future of AI inference, Aug 24–26 in San Francisco 🌉, hosted by @inferact at @anyscalecompute Ray Summit.

We'll have speakers from Inferact, NVIDIA, AMD, Google TPU, Anyscale, PyTorch, Meta, Red Hat, and key builders around vLLM. The talks on the roadmap deep dive into the latest on accelerators, training and serving pipelines, and production-scale inference 🚀

[Kevin Luu @khluu000]

Thank you @buildkite for your support over the past 2 years!

[Buildkite @buildkite]

The most widely used open-source model inference engine @vllm_project built their own testing dashboard + an overnight bot to find what broke and draft reverts on top of our platform data

[Kevin Luu @khluu000]

@vllm_project go brrrrrr 🚀

[vLLM @vllm_project]

vLLM hit new peak bs=1 decode on Kimi-K3: 464 tok/s 🚀

Under a low-entropy reasoning workload, Kimi-K3 + DSpark on vLLM reaches 464 tok/s on batch size 1 with 4×4 GB300.

This benchmark is fully reproducible with public image: vllm/vllm-openai:kimi-k3 and @inferact's DSpark draft model linked in the thread.

1/3

[Kevin Luu @khluu000]

🫡🚀

[vLLM @vllm_project]

With Kimi K3 Day-0 on vLLM: Open Frontier Intelligence for Everyone 🚀

At 2.8 trillion parameters, Moonshot AI's Kimi K3 is one of the most powerful open-weight models ever released. Starting today, you can serve it on vLLM the moment the weights are public.

What K3 brings:
🧠 2.8T-parameter Mixture-of-Experts (16 of 896 experts active per token)
📚 1M-token context window
🛠️ Native multimodal understanding, including vision
⚡ Kimi Delta Attention: a hybrid of linear and full attention that makes million-token context affordable

Huge thank you to @Kimi_Moonshot AI for the model release and partnership, @inferact for leading the vLLM optimizations, and to our partners at @nvidia, @AMD, and the broader vLLM community.

1/6

[Robin Cheng @rchpmv]

@vllm_project The best part of this is that whenever vLLM posts numbers, they are real, not inflated, and not cherry-picked. That's actually quite rare in the industry. If vLLM gives us a number, we can trust it's reproducible and real.

[vLLM @vllm_project]

With Kimi K3 Day-0 on vLLM: Open Frontier Intelligence for Everyone 🚀

At 2.8 trillion parameters, Moonshot AI's Kimi K3 is one of the most powerful open-weight models ever released. Starting today, you can serve it on vLLM the moment the weights are public.

What K3 brings:
🧠 2.8T-parameter Mixture-of-Experts (16 of 896 experts active per token)
📚 1M-token context window
🛠️ Native multimodal understanding, including vision
⚡ Kimi Delta Attention: a hybrid of linear and full attention that makes million-token context affordable

Huge thank you to @Kimi_Moonshot AI for the model release and partnership, @inferact for leading the vLLM optimizations, and to our partners at @nvidia, @AMD, and the broader vLLM community.

1/6

[vLLM @vllm_project]

On behalf of the vLLM community, @inferact just signed the open letter. Let's build an open ecosystem together!

[Kevin Luu @khluu000]

@MetheDudee check out https://t.co/MIWJ90ZiKe! If it still doesn't work, you can ask in vLLM Slack #questions

[Kevin Luu @khluu000]

Keeping vLLM stable is a huge collaborative effort from the community. Grateful to build this up together with everyone for the past 2 years!

[vLLM @vllm_project]

Great writeup from @khluu000 👏

How does vLLM stay production-quality while merging ~2,000 commits/month and shipping every 2 weeks?

The team broke down the three layers that make it possible. A huge community effort—thank you to everyone who is helping along the way.

https://t.co/F9FoltezVp