mobicham

I like to shrink dem models 🤏
ML/AI Perf @dropbox
Prev. Co-Founder & Principal Scientist @mobius_labs (acquired by @dropbox)
PhD @inria

[mobicham @mobicham]

@zhyncs42 Congrats! Would you guys share the right config to run Qwen3.6 models on B300s? Would be nice to have some recipes, I couldn't find enough info on Github

[mobicham @mobicham]

Can't wait for something like Claude Code for Ableton 👀, the time to manually eq and edit MIDI should be over

[mobicham @mobicham]

@EmrickSini It uses a vLLM plugin actually, that's why it works by just setting an env variable, and it's easy to add it to a docker image: https://t.co/zgU1qlzgJC

[mobicham @mobicham]

https://t.co/orcApi6B7J
You can now easily load various pre-quantized models (block FP8, NVFP4, AWQ/GPTQ/HQQ, certain GGUFs) via a vLLM plugin! You can also run on-the-fly quant as well, easy to use: 1 or 2 flags to enable!

[mobicham @mobicham]

There's a very simply one-shot trick that seems to improve the quality of low-bit weight quantization by quite a bit in some cases: simply reordering the rows.
It doesn't require changing the matmul kernel, only reshuffling the activations.

[mobicham @mobicham]

Skimming through the PolarQuant paper by Google and found HQQ is still alive 👀

[mobicham @mobicham]

@cHHillee 😂

[mobicham @mobicham]

@cHHillee 😂

[mobicham @mobicham]

@GPU_MODE @tbpn @AnushElangovan 🔥

[mobicham @mobicham]

@PatrickToulme Goat 🫡

[mobicham @mobicham]

Babe, wake up, new GemLite update.
Up to 1.7x+ faster FP8 block quantization on the RTX PRO 6000 end-2-end in vLLM!

[mobicham @mobicham]

What kind of FP4 format does the new TPU8 use? MXFP4 quality is pretty poor, and NVFP4 is specific to Nvidia, so I'm guessing it uses a smaller group size (<32) to achieve better quality ? 🤔

[mobicham @mobicham]

@elliotarledge Did you run it end-to-end in the full-stack vLLM/SGLang? You'll only see real perf issues when you do that and not just the kernel in isolation

[mobicham @mobicham]

@BoyuanChen0 Sick results 🫡

[mobicham @mobicham]

@vikhyatk Not supporting tcgen instructions doesn't matter at all. The RTX Pro 6000 is an excellent card for its price.

[mobicham @mobicham]

@fchollet The best of both worlds: https://t.co/RTf0M7vdxK

[mobicham @mobicham]

Some great on-device multi-vector work at Dropbox, check it out!

[Josh Clemm @joshclemm]

Open sourcing something fun from @Dropbox: Witchcraft.

It's a local search engine built in Rust with no API keys or vector DB required.

Think: ColBERT / late interaction style retrieval, but packaged to run locally (perfect for coding agents).

Let's dive in👇

[mobicham @mobicham]

@Teknium Qwen3.5-27B caveman think mode would be fire

[mobicham @mobicham]

@bernhardsson @_gongy RDMA tag 👌

[mobicham @mobicham]

@vikhyatk