Day 83/365 of GPU Programming
Looking at DeepSeek's Multi-Head Latent Attention today. The last part of the AMD challenge series is to optimize an MLA decode kernel for MI355X where the absorbed Q and compressed KV cache are given and your task is to do the attention computation.
A resource that really helped internalize what MLA does was @rasbt's incredible visual guide to attention variants in LLMs (luckily he posted that last week!), which covers everything from MHA to GQA to MLA to SWA, et cetera. If there's one place to get a visual intuition for recent attention mechanisms, it's this blog post.
@jbhuang0604's video on MQA, GQA,MLA and DSA was the best conceptual intro I found on the topic and progressively builds up the ideas from first principles.
The Welch Labs analysis of MLA is a great watch as well. Beautiful visualization of the changes DeepSeek made for MLA.
Tried out a few kernels once I had a basic understanding of MLA and I think I'm slowly getting more comfortable with at least analyzing kernels.
Day 82/365 of GPU Programming
Taking a closer look at Mixture of Experts today, so I can write better MoE kernels. Specifically, to optimize an MXFP4 MoE fused kernel for the GPU Mode challenge.
I haven't had much prior exposure to MoEs, so lots of new concepts I learned today. Luckily I found the best intro to MoEs thanks to @MaartenGr visual overview of the topic.
I then watched @tatsu_hashimoto's amazing Stanford CS336 lecture on MoEs, which added deeper context around why MoEs are gaining popularity, FLOPs, OLMoE, infra complexity, routing functions (mindblown this works so well...), expert sizes, training objectives, top k routing and DeepSeek variations.
Once I had a basic understanding I started playing around with the some AITER kernels but progress there is tbd.
Also had a nice chat with @juscallmevyom (who was kind enough to reach out!) about the AMD kernels and the challenge of materialization overhead.
I have a counterexample on TPU v7x. In a matmul kernel, issuing matmuls in bursts triggers power-management throttling, which in turn stalls weight delivery. Spacing out the matmuls yields a 25.9% speedup over an issue-when-ready greedy scheduler and 26.4% over vanilla Pallas.
Day 316/365 of GPU Programming
Learning more about Pallas kernels on TPU SparseCore today. As someone not super familiar with the JAX ecosystem and only vaguely familiar with TPU architecture, it's interesting to learn about the differences between ecosystems. Especially now as we're seeing greater acceleration in "time to onboard new silicon" as Anthropic put it, we'll probably see a much larger heterogeneity in chips/programming models.
Day 315/365 of GPU Programming
Onto lecture 18 which is part 2 of the series on finding efficiency gains in diffusion models. This is almost the end of the course (two more lectures on distributed training and some guest lectures before final projects), so my conclusion is this course has been really helpful and enjoyable. One interesting facet of the course is that lectures often center a around the focus of the research group itself (MIT's Han Lab) with regular throwbacks to "older" research that came out of the lab but I honestly really enjoyed that part as well since it feels like you get a pretty opinionated look into efficient ML research.
Day 315/365 of GPU Programming
Onto lecture 18 which is part 2 of the series on finding efficiency gains in diffusion models. This is almost the end of the course (two more lectures on distributed training and some guest lectures before final projects), so my conclusion is this course has been really helpful and enjoyable. One interesting facet of the course is that lectures often center a around the focus of the research group itself (MIT's Han Lab) with regular throwbacks to "older" research that came out of the lab but I honestly really enjoyed that part as well since it feels like you get a pretty opinionated look into efficient ML research.
Day 314/365 of GPU Programming
Spending the weekend catching up on some lectures. Looks like the next section of the https://t.co/aiEnyPO3fU course is about diffusion models and how to make them run efficiently. I don't know anything about the topic, so should be a fun one to explore further.
Day 314/365 of GPU Programming
Spending the weekend catching up on some lectures. Looks like the next section of the https://t.co/aiEnyPO3fU course is about diffusion models and how to make them run efficiently. I don't know anything about the topic, so should be a fun one to explore further.
Day 313/365 of GPU Programming
Reading up on some of the great posts on the LMSys blog. Especially interested in their recent megakernel article which delineates why SGLang has been moving towards larger GPU kernels. In it, they also describe what their kernels will evolve to (with a nod to the flappy folks) with a virtual machine “that takes control of the GPU and schedules different workloads across it”.
Day 313/365 of GPU Programming
Reading up on some of the great posts on the LMSys blog. Especially interested in their recent megakernel article which delineates why SGLang has been moving towards larger GPU kernels. In it, they also describe what their kernels will evolve to (with a nod to the flappy folks) with a virtual machine “that takes control of the GPU and schedules different workloads across it”.
Day 312/365 of GPU Programming
So far really enjoying the zero to sglang course by @YichiZ03 which tries to take you from the basics of SGL by building your own "mini SGL" all the way to making your first contribution to their repo. Think this is a great playbook for OS projects in general to find and train future contributors. Wouldn't be surprised to see more projects come out with similar courses.
Day 312/365 of GPU Programming
So far really enjoying the zero to sglang course by @YichiZ03 which tries to take you from the basics of SGL by building your own "mini SGL" all the way to making your first contribution to their repo. Think this is a great playbook for OS projects in general to find and train future contributors. Wouldn't be surprised to see more projects come out with similar courses.
Day 311/365 of GPU Programming
Reading more upon each of the formal verification talks that were delineated in the talks yesterday. Don't have a strong opinion yet but interesting topic nonetheless. Also my first time learning more about Lean which I've wanted to try out for a while given its current relevance.
Day 311/365 of GPU Programming
Reading more upon each of the formal verification talks that were delineated in the talks yesterday. Don't have a strong opinion yet but interesting topic nonetheless. Also my first time learning more about Lean which I've wanted to try out for a while given its current relevance.
Day 310/365 of GPU Programming
Have been seeing more and more discussions around formal verification of GPU kernels recently, and I saw that lectures 115 and 116 at GPU Mode are on that topic, so taking some time today going through those talks. Seems like they're taking two fairly different approaches which is quite interesting.
Day 310/365 of GPU Programming
Have been seeing more and more discussions around formal verification of GPU kernels recently, and I saw that lectures 115 and 116 at GPU Mode are on that topic, so taking some time today going through those talks. Seems like they're taking two fairly different approaches which is quite interesting.
Day 309/365 of GPU Programming
Still going through the assignment today. Have been really enjoying this one since it's a nice combination of concepts/techniques that were taught throughout the lectures of this course (though lecture 11 on TinyEngine and Parallel Processing is probably the most relevant).
Day 309/365 of GPU Programming
Still going through the assignment today. Have been really enjoying this one since it's a nice combination of concepts/techniques that were taught throughout the lectures of this course (though lecture 11 on TinyEngine and Parallel Processing is probably the most relevant).
Day 308/365 of GPU Programming
Taking a look at assignment/lab 5 of the efficient AI computing course today. This one's about optimizing LLMs on edge devices (specifically, LLaMA2-7B-chat) using loop unrolling, multithreading and SIMD programming. Should be a fun exercise and a nice overview of local model optimization.
Spent half a year of spare time building HuggingArch — a "compiler" that doesn't run tensors, but computes FLOPs, memory traffic & KV cache for any HuggingFace model.
Agent writes the spec → sympy evaluates → one click gives theoretical perf across GPUs & parallel configs.
No GPU. No weights. No runtime.
web: https://t.co/NhTMuwadfT
blog: https://t.co/9VFd9aZHma
github: https://t.co/UVTKOlcCrn
Star ⭐ & come build it with me!