Robert Shaw

@redhat | @neuralmagic | @vllm_project

[Red Hat AI @RedHat_AI]

Flow control is an admission layer for shared GPU inference.

Before a request gets routed to a model server, it's either admitted or held. Under saturation, priority decides which class of traffic advances, fairness spreads dispatch turns across tenants so no one gets starved, and ordering picks the next request. Lower-priority work waits instead of crowding out what matters.

Admission decides when a request runs. Routing decides where. Keeping those separate is the whole idea.

https://t.co/EzZeCKop3Y

[Robert Shaw @robertshaw21]

@AryaTschand Note that DeepSeekv4 also uses multiple subquadratic methods (DSA and SWA - which is similar to mamba in that its kv size does not grow as sequence gets longer)

VLLM blog: https://t.co/qb5eODdV9Q

It’s quite interesting to see all the innovations in kv cache!

[Robert Shaw @robertshaw21]

@rogerw0108 Early days of working on V1 was special

[alexa griffith @alexa_griffith_]

Dedicated GPU pools protect latency-sensitive products. The downside is that those GPUs are most likely underutilized. Shared GPU capacity is ideal, but the trade-off is losing your strict latency/SLO guarantees for high-priority requests.

That's why I'm excited about @_llm_d_ 's flow control feature. It manages request admission and multitenant priority policies for shared GPU capacity under pressure.

If you're interested in learning more, I ran some benchmarks here https://t.co/YDdMGsLEhb @RedHat_AI

[Helen Zhao @hezhao_z]

Check out our new kimi-k3 dspark checkpoint https://t.co/BNM8bbu3Rw 😎

[KVCache.AI @KVCache_AI]

In the upcoming vLLM Office Hour, we’ll take a deep dive into Mooncake and its integration with vLLM and llm-d for large-scale disaggregated LLM serving.

Welcome to join us!

🕒 Thursday at 2:00 PM ET / 11:00 AM PT
🕒 Friday at 2:00 AM UTC+8

[DigitalOcean @digitalocean]

Getting @Kimi_Moonshot K3, all 2.8T parameters of it, production-ready and fast on day zero is complex. We take on that complexity so you don't have to.

Serving recipe tuned with the @vllm_project team for @nvidia HGX™ B300 and @AMD Instinct™ MI350X GPUs, verified against Moonshot’s own suite. 🔗 https://t.co/4odl4R44IL

[Robert Shaw @robertshaw21]

me and claude

[Buildkite @buildkite]

The most widely used open-source model inference engine @vllm_project built their own testing dashboard + an overnight bot to find what broke and draft reverts on top of our platform data

[Matt Hicks @matthicksj]

Decades ago, critics questioned whether open source could meet enterprise security standards. Time and real-world adoption have since proven open source software as the powerhouse behind robust, resilient IT. We are seeing a similar evolution with AI today. Open-weight models offer vital flexibility, transparency, and defensive capabilities, making them an essential part of a secure AI ecosystem that is collaborative, transparent, and open. @RedHat is proud to sign this letter in support of open weights alongside @Microsoft and leaders across the industry. https://t.co/43DOhU4z9a

[Red Hat AI @RedHat_AI]

vLLM Office Hours today at 2pm ET: RL at 1T Scale, a prime-rl performance deep dive with @m_sirovatka (@PrimeIntellect). Training trillion-parameter MoE models like GLM-5.1 on agentic RL, plus what's new in vLLM 0.25 by @mgoin_. Get a recurring cal invite: https://t.co/KkF6DfGs1q

[Red Hat AI @RedHat_AI]

Serving a 700B+ param model (GLM-5.2) on H200s: llm-d's Wide EP + prefix-cache routing hits 90%+ cache reuse and sub-3s TTFT, at ~$2/M output tokens vs $4.40 on hosted APIs. @robertshaw21 breaks it down in last week's vLLM Office Hours.

Watch: https://t.co/Z3i9ZGn7yh

[KVCache.AI @KVCache_AI]

Congrats to the @vLLM_Project and excited to see Mooncake powering the full online DSpark training pipeline on GB300 NVL72 🚀

Mooncake efficiently moves hidden-state from vLLM nodes to Speculators through RDMA, eliminating the need for massive hidden-state storage in offline training.

Excited to keep pushing the limits together!

[Roger Wang @rogerw0108]

Qwen3-Omni is a truly one-of-a-kind model and was the original inspiration behind vLLM-Omni.

We've just published a technical blog post covering the techniques and optimizations we developed along the way. Check it out!

[Misha Goin @mgoin_]

DSpark update: Turns out with a little Speculators+Mooncake, I'm able to scale training on GB300 NVL72!
9 vLLM nodes serve the full GLM 5.2 FP8 verifier -> Mooncake RDMA store -> 6 nodes train the DSpark with FSDP (DP=24). 125k prefill tok/s, 1.5 steps/s, full online training :)

[vLLM @vllm_project]

And it's not locked to DeepSeek's checkpoints. 🧩

The Speculators library (https://t.co/pLpZP4xqfA) lets you train and package DSpark draft models in a standard, HF-compatible format that vLLM loads directly. Already validated on Qwen3-8B and GLM-5.2. Run it on vLLM nightly now:

vllm serve zai-org/GLM-5.2-FP8 -tp 8 --speculative-config '{"method":"dspark","model":"RedHatAI/GLM-5.2-speculator.dspark-preview","num_speculative_tokens":7,"attention_backend":"FLASH_ATTN","draft_sample_method":"greedy"}'

speculators support PR: https://t.co/ZIuOMe7zP1

Thanks to the @RedHat_AI team for the Speculators integration! 🙏

[Robert Shaw @robertshaw21]

@mgoin_ Vagueposting 👑

[SkyPilot @skypilot_org]

Introducing SkyPilot Endpoints - production-ready inference on every cluster you own.

One spec, one endpoint across your GPU fleet. SkyPilot handles scaling, routing and failure recovery - keeping your inference fast and resilient.

• Runs on GPUs you own - any cloud, neocloud, on-prem
• Production-ready - autoscaling, rolling updates, metrics + logs
• Boost GPU utilization - put every idle GPU to work with a unified training/inference pool

Designed for performance, built on the open-source stack you already trust (@vllm_project, @_llm_d_). Already serving frontier models for top AI teams.

Blog: https://t.co/kI2OwpckqN

[samsja @samsja19]

prime-rl can now train 1T parameters MoE blazingly fast, under 5 minutes per step, or 1k steps in ~3 days

To achieve this we shipped in our latest prime-rl 0.6.0:

* inference: wide-ep, fp8 inference, llm-d router, mooncake, kv cache cpu offloading

* training: fsdp2, deep-ep expert parallelism, dsa cp, fp8 training, router replay

* agentic rollout: we rewrote the core of our rollout orchestrator for better scalability

support for glm5, kimi, nemotron, ...,

prime-rl is open source but also end to end optimized to run on our dedicated RL infra and compute layer

[Robert Shaw @robertshaw21]

RT @vllm_project: Congrats to @GoogleDeepMind on DiffusionGemma 🎉 A 26B diffusion language model on the Gemma4 backbone, and the first dLLM…