Nils Matteson

MTS @inferact | @vllm_project | opinions my own

[Nils Matteson @nilsmatteson]

beyond excited to say i’m joining @inferact, working on @vllm_project full-time!

i care a lot about making powerful AI something more people can run and build on themselves. pretty surreal to get to do that with this team.

thank you @simon_mo_ & @rogerw0108 for everything.

can’t wait :D

[vLLM @vllm_project]

High throughput on the grill. Zero latency on the drinks.

vLLM maintainers and friends gathered during the PyTorch Conference for an evening of Brazilian steak, drinks, and great conversation! 🥩🥂

Grateful for this community and everyone who keeps vLLM moving forward. 🚀

[avery @codeshaunted]

never forget what tailscale took from you

[Nils Matteson @nilsmatteson]

@Wbackdown Just startup! Python would compile those packages to bytecode on import anyway. the flag does it during the image build. once loaded it's the same code running so the benefit is a fresh replica serving sooner, not higher tokens/sec.

[Nils Matteson @nilsmatteson]

Fresh vLLM containers can recompile thousands of Python files at startup.

One build flag moves that work to image build time: 14–18% less startup time (~18 s) in my H100 benchmarks.

Already in the CUDA nightly; official release images from v0.32.0.

https://t.co/5dzclSPUun

[Simon Mo @simon_mo_]

One flag to cut 18s cold startup time in vLLM prod

[vLLM @vllm_project]

Cold start, meet build time. 📷

Shipping precompiled Python bytecode in our images cuts vLLM startup by 14–18% (~18s on H100) — one uv flag at build time, zero recompiles at launch.

Already in CUDA nightlies, official release images from v0.32.0. Great work @nilsmatteson 📷

[will brown @willcb]

twitter is the bloomberg of ai research

[Nils Matteson @nilsmatteson]

I built the snapshot support here! Still experimental and TP1-only. The goal is to deploy prepared engines, so compatible replicas start near weight-loading time without giving up throughput. More to come, ordinary vllm serve integration is already in progress :D

[Laurence Moroney 🇺🇸🇮🇪 🏴󠁧󠁢󠁷󠁬󠁳󠁿 @lmoroney]

Restarting a vLLM server and waiting for big quantized weights to load all over again? The new release goes after that wait.

vLLM v0.31.0 landed late Sunday with 717 commits from 307 contributors. The piece I'd try first is the new vllm preload command, which starts a daemon that keeps post-quantized weights resident in GPU memory across engine restarts, with a /health endpoint and a readiness wait. There's also an experimental snapshot feature that uses CRIU to restore a fully initialized engine running on one GPU. The breaking-changes list is real: per-request mm_processor_kwargs and media_io_kwargs are now rejected unless you set --trust-request-mm-kwargs, tokenizer_mode="slow" is gone, and quantization="fp8" for online quantization is replaced by the fp8_per_tensor shorthand.

Before you upgrade a production server, search your launch scripts and request code for every flag on that list. Then time a restart with and without preload so you know what you're actually saving. #vLLM

https://t.co/yoXhUyGyVo