every gpu has a “speed of light” for this: memory bandwidth ÷ model size. on our hardware (RTX 4060) the speed of light is 205 tokens per second. no code can go faster. we want to get as close we can to this limit with the hardware we have
follow along 👀 big things are coming!
every time an AI model generates a token, the gpu has to read the entire model from memory. that memory speed limits how fast it can write
me, @e_yang6 and @jeffreyywongg are building an inference engine from scratch to push that to the physical limit
we plan to run Microsoft’s BitNet b1.58, a model whose weights are only −1, 0 or +1 making it ~8× smaller than a normal model. then we will fuse the whole model into one gpu program (a “megakernel”) instead of hundreds of small ones, so the gpu never sits idle between them