Helmut Schmidt

[Helmut Schmidt @helmutkan]

@stevibe Thank you. You probably saw it already, just for others PR is merged and one could just use "vllm/vllm-openai:nightly" image.
If you have resources it would be nice to compare google's MTP with RedHatAI/gemma-4-31B-it-NVFP4 as verifier model against original google model

[Helmut Schmidt @helmutkan]

@bnjmn_marie Thank you. How does it compare with eagle3 speculation, have you tried it?

Do you mind sharing yours exact vllm command?

[Helmut Schmidt @helmutkan]

@jun_song "Dgx spark"-like but with 256gb and 400-500gbs memory bandwidth would be almost perfect...

[Helmut Schmidt @helmutkan]

@CodyKnowsCode @vllm_project With the latest changes from vllm. My understanding is you were getting 35tok/s with old vllm and that you expect better results with this 0.20 version of vllm?

[Helmut Schmidt @helmutkan]

@CodyKnowsCode @vllm_project Could you, please, share results with the latest changes?

[Helmut Schmidt @helmutkan]

@RedHat_AI Got the response/explanation here: https://t.co/sFOqVtF2H0

[Helmut Schmidt @helmutkan]

@RedHat_AI thank you. Whatever I do I just cannot pass the attention backend error "Selected backend AttentionBackendEnum.FLASH_ATTN is not valid for this configuration. Reason:
['partial multimodal token full attention not supported']"

vllm-0.20.0
transformers-5.6.2
speculators-0.5.0

[Helmut Schmidt @helmutkan]

@johnny_everson nope, just using "trusted" models from google, redhat, nvidia, etc. and in safetensors format

[Helmut Schmidt @helmutkan]

@jun_song less than 0,12 Euros per 1 kWh? Damn...

[Helmut Schmidt @helmutkan]

@johnny_everson @stevibe yeah, me too, just my personal feeling that gemma4 dense model is way faster and better... Trying to make it work with dflash speculative model, but just cannot make it work at the moment...

[Helmut Schmidt @helmutkan]

@RedHat_AI @vllm_project "Validated for H100" - does this mean it depends on GPU architecture? Tried to make it work on dgx spark but with no success regardless of verifier model and type of attention backend...

[Helmut Schmidt @helmutkan]

@dsikka84 @BitcoinComfy @RedHat_AI @vllm_project I tried it with both fp8 and nvfp4, but couldn't make it work with containerised vllm nightly on dgx spark at all... Since it's mentioned it's verified only on H100, does that means its dependable on GPU architecture?

[Helmut Schmidt @helmutkan]

@stevibe Oh I see, didn't know it's moe, thanks!

[Jeff Geerling @geerlingguy]

I'm amazed none of these videos are picked up by YouTube's 'Likeness Detection' feature (though I guess, that's why it's in Beta?). cc @YouTubeInsider @YouTubeCreators

(Yes, all of these videos contain my AI-cloned likeness and cloned voice. I submitted a 'privacy violation'.)

[Helmut Schmidt @helmutkan]

@bnjmn_marie "The model is as token-efficient as the original one." Is this implying that other nvfp4 models in this list are not token efficient?

[Helmut Schmidt @helmutkan]

@RedHat_AI @dsikka84 Thank you and thanks to @dsikka84 and team. I probably could do some simple and quick tests if someone points me what, where and how - which is I guess more time consuming then running those tests :) thanks again

[Helmut Schmidt @helmutkan]

@RedHat_AI It would be interesting to see results of speculator model RedHatAI/gemma-4-31B-it-speculator.eagle3 with RedHatAI/gemma-4-31B-it-NVFP? Any possibility to do that?

[Helmut Schmidt @helmutkan]

@protect_whales @bnjmn_marie +1, additional to that list I would include two from Nvidia (nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 and nvidia/Nemotron-Cascade-2-30B-A3B)