I wake up, 5am.
A Taiwan engineer has been profiling KV cache memory across 8 hours of decode benchmarks for me.
He found the leak.
He texts “quantize the KV cache to INT8, halve the HBM footprint”
I text back “wow” and google what that means.
I close my laptop.
I pour a coffee.