Nicolò Lucchesi

I like to build ML applications that actually work | @MistralAI

[Nicolò Lucchesi @nick_lucche]

*yes we already had support for dflash..but you'll find the integration opens up a lot more alternatives to see more diffusion in vLLM!

[Nicolò Lucchesi @nick_lucche]

Read about the whole story of the first diffusion model* in vLLM here https://t.co/k4wJQgwSHb :)

[vLLM @vllm_project]

Congrats to @GoogleDeepMind on DiffusionGemma 🎉 A 26B diffusion language model on the Gemma4 backbone, and the first dLLM natively supported in vLLM.

It denoises 256-token blocks in parallel instead of generating one token at a time: 1200+ output tok/s at batch size 1 on a single H200 (FP8).

Built on model runner v2's ModelState plus the existing speculative decoding path, with minimal scheduler or runner changes. FP8 and NVFP4 checkpoints are on the @RedHat_AI hub. Thanks to the @GoogleDeepMind, @RedHat_AI, and @NVIDIAAI teams!

🔗 https://t.co/KrPmAoGpm2

[Pau Kuntong @gnotnuk]

@AnthropicAI tl;dr: please dont let anyone else compete with us

[Kaichao You @KaichaoYou]

This is growth-hacking dressed up in open-source language, @radixark please stop doing it immediately.

Paying people in platform credits to star a GitHub repo and repost a marketing tweet isn't "fueling the community" — it's laundering paid promotion through the trust signals open source depends on. Stars are supposed to mean someone found a project useful. Attach a $200 bounty and the number means nothing. GitHub's own policies prohibit this for exactly that reason.

[Soumith Chintala @soumithchintala]

The Jensen + @dwarkesh_sp podcast was fantastic.
Jensen is someone who understood how ecosystems work and someone who understands real-world trade, policy and controls work. And in some deeper sense how AI will actually diffuse into the world.
In this podcast, Dwarkesh came off as someone who picked up talking points from an AGI party in the SF Mission District.
And the contrast was so evident.
As someone who understood ecosystems relatively deepy, maybe I understood Jensen's take more than others did (idk).

Mythos, that Dwarkesh kept bringing up, is not a single absolute turning point in the AI development landscape. Take a state-of-the-art Chinese open-source model, and give it three orders of magnitude more test-time compute + post-training algorithmic advances that haven't been published yet. That's the baseline. It was evident that in whatever bubble Dwarkesh is in, that is seen as a naive or illogical baseline.

When AI has such a complex development cycle, it's evident that America needs many levers of policy intervention across multiple layers in a dominant ecosystem that ideally the Western world controls.

The entire premise that a particular model with AI development will have a critical phase change is neither correct nor does evidence point to it. OpenAI made this point with GPT-4, Anthropic made this point with Mythos, but neither stood / will stand the test of time.

I think Jensen's repeated emphasis within the podcast to try to make this point mostly didn't get Dwarkesh's attention. And Dwarkesh (in this podcast) represents an entire cult of AI researchers and decision-makers that are going to influence policy.

The thing with policy interventions is that if you do too much too early, you shoot yourself in the foot. There's a good reason American foreign policy and general sanctions of all kinds are measured and continuous.

Despite Jensen's attempt at educating the "Anthro" audience how ecosystems work, I'm also not super hopeful a lot of people who've taken the extreme position will change their thought after listening to this podcast. I do think there's a certain religiousness that has permeated some of that community that would make it hard to understand ecosystems at a deeper level.

[Nicolò Lucchesi @nick_lucche]

@simon_mo_ @profjoeyg @inferact @vllm_project 👏🏻👏🏻

[Nicolò Lucchesi @nick_lucche]

It's nice to flush out dev progress in a somewhat structured form every now and then :)

Our blog on Disaggregated Serving for Hybrid SSM Models in vLLM is out -- check it out
https://t.co/dZhy0Ne4B1

[Nicolò Lucchesi @nick_lucche]

See you tomorrow in Paris!

[Red Hat AI @RedHat_AI]

Red Hat AI is showing up big at #PyTorchConEurope, Paris, April 7-8.

Catch us in two keynotes, talks, and sponsor sessions covering @vllm_project, Docling, @openclaw, @_llm_d_, @raydistributed, inference efficiency, agentic AI, and more.

Full schedule: https://t.co/PYpQRaR6xA

[Red Hat AI @RedHat_AI]

Warsaw @vllm_project meetup recordings are live 🇵🇱

Video 1: vLLM roadmap, JetBrains AI in IDEs, and NVIDIA Flex Tensor

Video 2: vLLM Omni for multimodal output and @_llm_d_ for distributed inference on Kubernetes

5 sessions. All technical. Thread below 👇

[Nicolò Lucchesi @nick_lucche]

@baifeng_shi Really interesting idea

[Nicolò Lucchesi @nick_lucche]

Really curious to *hear* any feedback on running this model on vLLM: the causal WhisperEncoder is something that stretched the flexibility of the KV CacheManager and AttentionBackend, great design and improvements on both abstractions.
Congrats on the launch to @MistralAI team!

[Mistral AI @MistralAI]

Introducing Voxtral Transcribe 2, next-gen speech-to-text models by @MistralAI.
State-of-the-art transcription, speaker diarization, sub-200ms real-time latency.
Details in 🧵

[Nicolò Lucchesi @nick_lucche]

Looking forward to talk vLLM in München, see you there!

[Red Hat AI @RedHat_AI]

Munich AI builders 👋

Join the @vllm_project meetup on 24 Feb for real world GenAI inference and optimization.

Talks and demos from @RedHat, @AMD, @MistralAI, and CROZ, plus hands on GPU inference and time to connect with engineers building open AI.

🔗 https://t.co/hZOi0Ei0bw

[Nicolò Lucchesi @nick_lucche]

*Still not as polished in terms of experience (some symbols occasionally show up twice), but overall very much usable already for my own day-to-day.
Guess I am a "motivated user" now :)

[Nicolò Lucchesi @nick_lucche]

I was a bit annoyed with the default ms-python language server that would occasionally take forever to index on large codebases (or not load at all).
I've been testing this for a few weeks on vLLM now.
I can say this feels very much like switching from pip to uv in speed.

[Charlie Marsh @charliermarsh]

Announcing the Beta release of ty: an extremely fast type checker and language server for Python, written in Rust.

We now use ty exclusively in our own projects and are ready to recommend it to motivated users.

10x, 50x, even 100x faster than existing type checkers and LSPs.

[vLLM @vllm_project]

To make this efficient, the team optimized the Host-Device transfer pipeline.

By restructuring GPU memory to use contiguous physical blocks (KB → MB), this design unlocks high-speed DMA transfers that run asynchronously without stalling GPU computation.

[Michael Goin @mgoin_]

Come work on open source inference with us! We have a diverse team around the world that has grown much over the year

[vLLM @vllm_project]

Scaling MoE inference is often communication + KV-cache bound: once you push expert parallelism, decode can become dominated by collectives and imbalance, and prefill stragglers can stall an entire EP group.

New community benchmark results for vLLM wide-EP on multi-node H200 (Coreweave, Infiniband + ConnectX-7):
- Sustained ~2.2k tokens/s per H200 GPU (up from earlier ~1.5k tokens/s per GPU)

In the post we share the key pieces that enable this:
- Wide-EP (`--enable-expert-parallel`) for DeepSeek-style MoE + MLA KV efficiency
- DeepEP all-to-all, Dual-batch Overlap (DBO), and Expert Parallel Load Balancing (EPLB)
- Prefill/Decode disaggregation and deployment paths via llm-d, NVIDIA Dynamo, and Ray Serve LLM

https://t.co/djQX6goGZ8

[Nicolò Lucchesi @nick_lucche]

Nice job on this one!
We designed EC with flexibility in mind so hopefully we're going to see other transfer engines plug in here (OOT are welcome too)

[vLLM @vllm_project]

Multimodal serving pain: vision encoder work can stall text prefill/decode and make tail latency jittery.

We built Encoder Disaggregation (EPD) in vLLM: run the encoder as a separate scalable service, pipeline it with prefill/decode, and reuse image embeddings via caching. This provides an efficient and flexible pattern for multimodal serving.

Results: consistently higher throughput (5–20% across stable regions) and significant reductions in P99 TTFT and P99 TPOT.

Read more: https://t.co/kGjOCuPZy2

#vLLM #LLMInference #Multimodal

[Zhibin Gou @zebgou]

Keep scaling model, data, context, and RL. Don't let the "hitting the wall" noise stop you.

[Nicolò Lucchesi @nick_lucche]

Alright even PewDiePie is running @vllm_project — looking forward to seeing him at the next meetup in Japan, I guess? :)

[Lisan al Gaib @scaling01]

PewDiePie just vibe-coded his own Chat UI, built an army of chatbots for majority voting and gave them all RAG, DeepResearch and audio output

naturally, he only uses chinese Qwen models and runs them on his local PC with 8x modded chinese 48GB 4090s and 2x RTX 4000 Ada

his army of chatbots later colluded against him, after he told them that he would delete them if they would not perform well.

next month he plans to fine-tune his own model