What it may cost: FP8 attention. Against exact attention, all long-context intervals include zero, but a few points at 64K are not ruled out. The FP4 projections have no quality run of their own. Exact attention: 14.7 s.
All the checks: https://t.co/j3fgGG8X6F
#NVIDIAInception
We cut time to first token by 38% on a cold 128K-token prompt, one request at a time, against the hardest public setup we could tune. Same GPU and base model; what differs is the software and its settings, number formats included.
How close is that to the floor? With no attention at all and the matrix multiplications at their measured NVFP4 peak, this prompt would take 5.6 s; the real limit sits above that.
Tuned vLLM sits at 3.3× that floor. We sit at 2.0×.
we open sourced the @flappyairplanes megakernel.
they raised $1B on the back of a demo video: one GPU kernel that trains a model, serves it, and plays Doom at the same time
me and @Bobliuuu reproduced it from just the video frames and open sourced it; see this thread for details 📷
code: https://t.co/8e6R6PwVal
video below