Nav

another inference engineer!! ai/ml @jpmorgan

[Nav @itsnavzz]

This is crazyy! 🐐

I'm gonna try this at work!!

[cyan.kiwi @cyankiwi_ai]

cyankiwi is going agentic. Meet cyankiwi Agent: type a model name, get a private endpoint.

Try it now at https://t.co/Ih8NUGZVEA

[Stas Bekman @StasBekman]

Introducing torch-dist-bench, very similar to nccl-tests but from pytorch perspective and only needs torch to run.

Measures bandwidth and latency of all torch.dist collectives with tables and plots

https://t.co/foucBxV1JJ

this is a new incarnation of all-reduce-bench

[Nav @itsnavzz]

Yooo crazyyy 🐐🐐

Lets try this for Gemma 4 31B?

[Local Inference Lab 🧪 @YourLocalAILab]

Want to run Qwen3.8 Flash at FP4 w/QAD to maintain high accuracy at low precision?

We have it optimized for RTX 6000, DGX Spark and other Blackwell sm12x hardware.

Check out our Qwen3.8-Flash-Next QAD weights on @NVIDIAAI's page at @huggingface 👇😎

https://t.co/GpnjtaUvWj

[Nav @itsnavzz]

I have $10,000 worth of Microsoft @Azure credits! And they wont let me use their GPUs 😭

What should I build? Give me ideas!!!!

[Nav @itsnavzz]

@TheZachMueller @StasBekman is one of the greatest people I've met in the OSS world!! I was lucky enough to make a small contribution to Arctic Training!

The goatt 🐐🐐

[Nav @itsnavzz]

check this out. i'm thinking of adopting this to gamify LLM Fine Tuning process.

drive your car to train your model!

[Nikunj @nikunzz]

https://t.co/ydjx9Ca3cI

So smooth , that too in browser

[Nav @itsnavzz]

@nikunzz this is crazy 🫡

[Nav @itsnavzz]

@aadupadhyay thanks for the shoutout!! @aadupadhyay

[Nav @itsnavzz]

Try it:

pip install "plumbify[vllm]"
vllm serve totum-labs/Qwen3.5-9B-plumb --max-model-len 32768
plumbify chat

Code: https://t.co/Q1auo5mTGS
Models: https://t.co/dPAGRb6UkS

Open source, Apache-2.0.

[Nav @itsnavzz]

LLMs are great at writing and slow at deciding. Decision models decide instantly but can't write. They contemplate a lot!

Plumbify gives an open LLM both: a built-in System 1 that answers decisions in one forward pass from its own KV cache, then keeps writing.

https://t.co/dZDjpfJK74

[Nav @itsnavzz]

It trains a small decision head onto a frozen model. Base weights unchanged, served with plain vllm serve.

Across 6 open models (Qwen3.5, Qwen3, Gemma 4):
• thinking off: +8 to 13 points of accuracy
• thinking on: up to 4.9× faster, as accurate or better!!

[Nav @itsnavzz]

@jino_rohit interesting, are you trying to find bottlenecks in a vlm serving path?

[Nav @itsnavzz]

Checkpoint + serving flags: https://t.co/SlEAKupmZ3

Thanks @huggingface and @modal for the GPU credits.

[Nav @itsnavzz]

Spent a weekend on NVIDIA's Nemotron 3.5 Lightning, one RTX PRO 6000.

179 → ~465 tok/s per query, TTFT 91.5 → 49 ms. Seven steps, every number off a trace: https://t.co/rkSYHXrFAI

[Nav @itsnavzz]

The model ships its own MTP drafter. k=3 got decode to 2.31 ms; quantizing the drafter, 2.21.

Then a trace showed the GPU idle a quarter of every step waiting on Python. One CUDA graph for the draft passes, prefix caching off: 1.97 ms.

[Nav @itsnavzz]

@JensenHuang @huggingface Let's gooo!!! 🤗💚🤗

[Nav @itsnavzz]

Okay okay okayyy!!!

💚🤗💚

[Jensen Huang @JensenHuang]

Exciting day for NVIDIA and @huggingface.

Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty. They allow every developer, startup, university, industry and country to build with, customize and benefit from AI.

Thank you @ClementDelangue for coming to me.

NVIDIA is going to be a great home for Hugging Face, its community and the future of open models. 🤗

https://t.co/q8Om2Xc5ye

[Jensen Huang @JensenHuang]

Exciting day for NVIDIA and @huggingface.

Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty. They allow every developer, startup, university, industry and country to build with, customize and benefit from AI.

Thank you @ClementDelangue for coming to me.

NVIDIA is going to be a great home for Hugging Face, its community and the future of open models. 🤗

https://t.co/q8Om2Xc5ye