Ganesh Nanduru

LLM researcher in SF Bay Area

[Ganesh Nanduru @gnanduru1]

@sameQCU in moe regime i feel like param-maxing attention is not the move, though i agree sparse routing for heads is a good idea. minimax pulled it off and is frontier

[Ganesh Nanduru @gnanduru1]

@Michael_J_Lutz excellent writeup

[Ganesh Nanduru @gnanduru1]

@profcelsofontes flash-msa is just for training, minimax themselves have uploaded inference code to both

vllm: https://t.co/8Lt5hpLLXJ
sglang: https://t.co/riaJv1ODBM

[Ganesh Nanduru @gnanduru1]

Introducing Flash-MSA, the world’s first open source sparse attention training kernels optimized for extreme context lengths. 4 simple kernels, 400%+ speedup over dense flash attention at long context ⚡

[Ganesh Nanduru @gnanduru1]

@allanzhangML @zxytim it currently is faster on blackwell, but not fully optimized, e.g. i didn't use sm100-specific MMAs yet to preserve compatibility with Hopper, but getting to 4x on blackwell is achievable

[Ganesh Nanduru @gnanduru1]

@zxytim ran on h100 for price, fa4 zooms on sm100/blackwell

[Ganesh Nanduru @gnanduru1]

blog: https://t.co/AsyJJTBCvv
code: https://t.co/jDHFJ9p2on

very open to collab on flash-msa. I’m excited to see what people do with it, and I hope it encourages folks to bring their wild architecture ideas into the open, where the community is eager to optimize them 🚀

10/10

[Ganesh Nanduru @gnanduru1]

To verify kernel correctness, I ran sweeps of comparisons to eager MSA, for example this table sweeps over Q/B/S in bf16. I also implement Flash-MSA to be flexible- it supports various main and proxy attention configs, variable top-k, and the proxy dense warmup phase.

9/

[Ganesh Nanduru @gnanduru1]

@rishiiyer01 harsh performance drops as well apparently