max

learning model perf one epoch at a time…

[max @maxxfuu]

@highperfcache @skydotcs @pangram are your dms open, (@highperfcache)?

would love to probe ur brain on what you know about high performance cache (not a joke)

[max @maxxfuu]

Day 6/90 of Inference Engineering

I wrote a CUDA kernel for 1D Convolution, just getting the reps in for writing unoptimized boring but correct CUDA! I also read PagedAttention (2312.07104), watched 1 short lecture on GPU Memory, and hack squatted 455lb x6 and 495lb x3 as a top set!

Heres what I've learned about PagedAttention:

PagedAttention reduces internal fragmentation and solves external fragmentation.

During the decode phase, the KV cache manager actually hands out one cache block at a time to store output tokens. If the output tokens overflows the cache block, the KV cache manager assigns a new cache block to hold the remaining tokens that didn't fit into the previous cache block. What ends up happening is that the VRAM is used effectively such that each cache block is filled before new cache block is allocated on the VRAM.

In a naive KV cache implementation, memory is reserved up front for the maximum sequence length, which might not ever be used, meaning there is just a massive chunk of VRAM that's not touched.

PagedAttention reduces internal fragmentation by ensuring memory is never wasted between each decode sequence.

To make usage of memory even more efficient, any unique prompts that share the same prefix tokens, the KV cache manager allows them to share the same cache block, however, it writes to a new cache block starting from where the new tokens differ within the two prompts.

While each decode phase effectively writes to random parts of the VRAM, the Block table is able to provide an abstraction that makes KV cache seem like a contiguous memory to the model. Making the allocation process as simple as checking if a cache block is full or not.

This continuous practice of using cache blocks to allocate just enough memory and using a block table to allocate non-contiguous free scattered memory on the VRAM reduces internal fragmentation and solves external fragmentation.

---

Watched a video on GPU Memory: https://t.co/2TySFdj1im

I read this blog that made absolutely cemented everything, written by (@hamzaelshafie): https://t.co/GEzvexo7pl

[max @maxxfuu]

@highperfcache @skydotcs @pangram haha, i was sad at first because i thought that he thought my writing sounded too robotic. but then again, each reflection takes about 1 hour, so it’s high quality human slop. this is the best yall are gonna get 😌

[max @maxxfuu]

@prathamgrv @TensorTonic super excited for you!!!

[max @maxxfuu]

@maharshii yo chill...

[max @maxxfuu]

@SzymonOzog_ oh my god i didn’t know you’re simon oz on yt!

[max @maxxfuu]

@skydotcs @pangram im putting my reputation on the line and documenting everything. theres a higher chance of you seeing human slop than AI slop LOL

[max @maxxfuu]

https://t.co/3Bv4VrN3dV

[max @maxxfuu]

This article talks about the hardware imbalance: https://t.co/UahPU9a8Pw

Heres a video that I watched today: https://t.co/bgGqQRgKnA

Here my repo:
https://t.co/3Bv4VrN3dV

[max @maxxfuu]

Day 5/90 of Inference Engineering

I wrote a matrix transpose and a softmax CUDA kernel today. Following yesterday's post, Nicholas Wilt (@CUDAHandbook ) commented that the intuition for CUDA really comes from working with kernels that benefit from shared memory.

I can't fully comprehend this yet, but here's my current guess: refetching the same bytes from DRAM burns bandwidth, which saturates the memory system, which leaves the GPU cores sitting idle waiting to be fed.

Calling back to what I learned from my first project, the roofline profiler. If an algorithm has an intrinsic arithmetic intensity that makes it memory bound, the thing we actually control is the implementation's intensity. Meaning we can change how many times we refetch the same bytes from DRAM. Cut the redundant work, push the implementation's arithmetic intensity up toward the ridge point, and it should result in a better use of the hardware. And using the hardware better through CUDA is the intuition for CUDA.

Anyways, this gave me a much clearer direction for where to actually point my CUDA skills. The goal now: learn CUDA, write the most optimized CUDA I can within 90 days, and reimplement parts of the PyTorch in my GPT3 reimplementation as CUDA kernels that use shared memory. That should give me the practical reps to build the intuition.

Plan for tomorrow is simple:
1. Write more CUDA
2. Finish reading the 3 research papers
3. Read more on CPU and memory so I can see where and how the GPU really shines

Hit my 10K steps today and did 2x6 on 75lb DB shoulder presses.

See y'all tomorrow!

[max @maxxfuu]

Loving @FaradawnYang explanation! https://t.co/i6OWkpVjhR

[max @maxxfuu]

Day 4/90 of Inference Engineering

I built a CUDA kernel for matrix addition. The objective was to write correct CUDA while instilling the right intuition.

Mainly putting the reps in with allocating memory on the host and device and writing the actual kernel through the lens of writing a thread instead of a sequential program.

I learned that the idea of blocks and grids are just a software abstraction. However at launch time, grids are already partitioned into blocks. This allows the global block scheduler to schedule blocks and pin them to a SM for a lifetime. Without this abstraction we would have to schedule each thread individually for execution.

I also learned that the block abstractions are what allows threads to cooperate cheaply within a block. Since threads within the same block share the same shared memory, this gives threads visibility into each other's data, and __syncthreads__() synchronizes completion for every thread within the block.

Very interesting ideas. Love to hear a Kernel Engineer shed some light on this topic~

On the hardware side, I previously thought that the L1 Cache and Shared Memory were interchangeable terms. Turns out, they are two unique hardware components. I also forgot to include polymorph, TCP, Raster Engine within my old diagram. I attached an image below of the updated diagram that I drew.

I also went over some CUDA documentation to understand the syntax of cudamalloc(). Today was a great day, on to the next!

[max @maxxfuu]

https://t.co/3Bv4VrN3dV

[max @maxxfuu]

@lordvanpepee perhaps going over a vetted textbook and go over each chapter. assume each chapter has a lot of unknowns, pick the first unknown or reason with what you think is the biggest unknown. hope this helps!

[max @maxxfuu]

Day 3/90 of Inference Engineering

I wrote a CUDA Kernel for vector addition today. It felt refreshing to go through the works of managing memory between the GPU and CPU. Feels like those days when I was tinkering with C++s RAII and Move semantics.

Throughout this process I realized I needed to quickly understand the GPU architecture so I read an article on it, watched a video, took some notes.

Main takeaway for today is that CUDA program has two main parts. The host (CPU) and device (GPU). We have to manage the memories within each component and data must be transferred between the two.

Also hit 10K steps today and a working set PR of 2x6 of 455lb on hack squats @ RPE 10.

Github Repo: https://t.co/9r7biniB5M

GPU Architecture: https://t.co/SYdDRibF0a

[max @maxxfuu]

@lordvanpepee i try to figure out what’s most important out of all the things i don’t know. that’s usually what i’d work on for the next day/s!

in short, i don’t follow a course~

[max @maxxfuu]

@Morpheeus_04 it’s a series of videos i found on youtube, let me post it on my next check in

[max @maxxfuu]

Day 2/90 of Inference Engineering

I built a PyTorch roofline profiler because I want to prove and understand intuitively why decode is memory bound and prefill is compute bound.

Also went over GPU Lectures 1-3 as well while getting in my 10K steps in. Finished up on my notes on MHA.

I will polish up the Ridgeline over the coming weeks.

For the next 7 days, I want to:
- Finish the GPU Lectures
- Tinker with CUDA
- Read 3 papers for my next project:
- SGLang: 2309.06180
- Sarathi-Serve: 2403.02310
- PagedAttention: 2312.07104

https://t.co/OuLoMmppF7