Vladimir Vlejd Macko

I like taking things from 0 to 1. From nothing to something. Sprinkled with ML when necessary.

[Vladimir Vlejd Macko @vlejd]

Unstructured weight #sparsity made practical.
50% unstructured weight sparsity was considered too low for real GPU speed up without specific hardware support (like @cerebras).
With @bozavlado we built MACKO-SpMV - a new matrix format + SpMV kernel to change that. 🧵

[Vlado Boza @bozavlado]

Now, accepted to ICML as a spotlight paper.

I am super proud of my student @vlejd.

[Vladimir Vlejd Macko @vlejd]

@0xJayHK @cerebras @bozavlado tldr: optimize memory while keeping the compression compatible with the GPU programming model. The first step ofc is to figure out, what is the GPU programming model :D

[Vladimir Vlejd Macko @vlejd]

Unstructured weight #sparsity made practical.
50% unstructured weight sparsity was considered too low for real GPU speed up without specific hardware support (like @cerebras).
With @bozavlado we built MACKO-SpMV - a new matrix format + SpMV kernel to change that. 🧵

[Vladimir Vlejd Macko @vlejd]

@mmaaz_98 Nice work! if you want to add support for low 20-90% sparsity, we have an implementation at https://t.co/WLHL7X8dt8

[Vladimir Vlejd Macko @vlejd]

@mariyaivasileva I worked at a company that sometimes had spare GPUs.

[James Bradbury @jekbradbury]

opus 4.5 is really good at GPU programming, but somehow it’s even better at GPU programming jokes (h/t @Si_Boehm)

[Vladimir Vlejd Macko @vlejd]

🛠️ Next step: server GPUs.
If you know how to implement a minimal CUDA matvec on H100 that hits ≥95% of cuBLAS
👉 My DMs are open.

[Vladimir Vlejd Macko @vlejd]

And yes, it translates to real LLM inference speed ups.

[Vladimir Vlejd Macko @vlejd]

It is funny how little correct information is there about how to properly benchmark a CUDA kernel. Most papers are wrong, eval libraries are hard to inspect and even this could have a problem because it may include the kernel launch depending on clear_cache implementation

[tender @tenderizzation]

btw, I think BackendBench just uses triton's do_bench function, which uses a very similar timing mechanism to the one exploited here and wouldn't be robust to the same side-stream shenanigans

[Vladimir Vlejd Macko @vlejd]

@miru_why @niklassheth @ronusedh @intology My second personal favorite is to not clean the cache between invocations, and testing only on matrices that fit the cache. You can get some truly unbelievable flops :D

[Vladimir Vlejd Macko @vlejd]

@miru_why @niklassheth @ronusedh @intology https://t.co/9mX22jfGW5

[Vladimir Vlejd Macko @vlejd]

@miru_why @niklassheth @ronusedh @intology Hahaha. I spent months debugging this. Had to fix the official torch documentation that contained the same problem in it's examples.
Unfortunately, pretty common pattern.

[Julian @julianboolean_]

holy shit they found a power series solution to ALL polynomial equations!! (bypassing Galois which says you can’t solve them in radicals)

[Vladimir Vlejd Macko @vlejd]

@ollama local app is coming up! Awesome. Local models are the future.

[Vladimir Vlejd Macko @vlejd]

Happy birthday @ollama !

[Vladimir Vlejd Macko @vlejd]

I do model compression and optimization. It is essential to have access to different GPUs and that would be impossible without @vast_ai . Happy to finally meet you guys at #ICML2025 . And thanks a lot for the Nintendo Switch!