Finally done reading about PagedAttention, and this one made the connection between memory management and LLM serving much more obvious to me.
As LLMs generate tokens, they need to store Key and Value (KV) states for all previous tokens. With traditional memory allocation, these KV caches are stored in large contiguous blocks, which leads to memory fragmentation, wasted GPU memory and limits the number of concurrent requests.
PagedAttention takes a different approach. Inspired by virtual memory in operating systems, it divides the KV cache into fixed-size blocks that can be stored anywhere in GPU memory. A block table maps each request's logical token positions to their physical memory blocks.
This allows the KV cache to grow dynamically as new tokens are generated, without requiring large contiguous memory allocations. It also enables efficient memory sharing between requests and reduces fragmentation.
The interesting part is that the attention computation itself does not change. The improvement comes from how the KV cache is allocated, accessed and managed during inference.
The result is better GPU memory utilization, support for more concurrent requests and higher serving throughput, without changing the underlying model.
Here you go with a mental model you can refer to for a better visual understanding.
Day 37 of CS336 Language Modeling from Scratch
- Completed Lecture 9: Scaling Laws
- Scaling laws turn expensive large-scale model-design decisions into cheaper small-scale experiments by exploiting predictable relationships between compute, data, parameters, optimization choices, and loss.
- The predictability is empirical and regime-dependent, so parameter definitions, optimization settings, data regime, measurement variance, and extrapolation range matter enormously.
Day 36 of CS336 Language Modeling from Scratch
- Completed lecture 8: Parallelism by Tatsu.
- Modern LLM training is not "data parallelism with lots of GPUs"; it is a carefully engineered composition of data, tensor, pipeline, expert, sequence/context, and parameter-sharding strategies, chosen according to memory requirements, batch size, computation, and most importantly, the communication topology of the hardware.
- Parallelism is fundamentally a resource-allocation problem: decide what to replicate, what to shard, what to communicate, where to communicate it, and whether that communication can be hidden behind computation.
Day 36 of CS336 Language Modeling from Scratch
- Completed lecture 8: Parallelism by Tatsu.
- Modern LLM training is not "data parallelism with lots of GPUs"; it is a carefully engineered composition of data, tensor, pipeline, expert, sequence/context, and parameter-sharding strategies, chosen according to memory requirements, batch size, computation, and most importantly, the communication topology of the hardware.
- Parallelism is fundamentally a resource-allocation problem: decide what to replicate, what to shard, what to communicate, where to communicate it, and whether that communication can be hidden behind computation.
Day 35 of CS336 Language Modeling from Scratch
- Finished Section 4 - GPU Kernels (FlashAttention-2)
- Benchmarked vanilla PyTorch attention hitting the quadratic O(S^2) HBM wall at long context lengths
- Implemented a custom Triton forward kernel with online softmax and block pointers (tl.make_block_ptr), added causal masking, and built backward pass recomputation via logsumexp (L) to drop memory complexity to O(S).
- Next up is Section 5 - Distributed Data Parallel Training (DDP).
Need to speed uppp
Day 35 of CS336 Language Modeling from Scratch
- Finished Section 4 - GPU Kernels (FlashAttention-2)
- Benchmarked vanilla PyTorch attention hitting the quadratic O(S^2) HBM wall at long context lengths
- Implemented a custom Triton forward kernel with online softmax and block pointers (tl.make_block_ptr), added causal masking, and built backward pass recomputation via logsumexp (L) to drop memory complexity to O(S).
- Next up is Section 5 - Distributed Data Parallel Training (DDP).
Need to speed uppp
Day 34 of CS336 Language Modeling from Scratch
- Continued on Assignment 2
- Finished Section 3 - Single-GPU Memory
- Analyzed autograd activation bloat using tensor hooks, derived asymptotic bounds for recursive checkpointing (O(log N) memory), and benchmarked why per-block checkpointing (m=1) minimizes peak VRAM.
- Next up is 4 - GPU Kernel.
Day 34 of CS336 Language Modeling from Scratch
- Continued on Assignment 2
- Finished Section 3 - Single-GPU Memory
- Analyzed autograd activation bloat using tensor hooks, derived asymptotic bounds for recursive checkpointing (O(log N) memory), and benchmarked why per-block checkpointing (m=1) minimizes peak VRAM.
- Next up is 4 - GPU Kernel.
Day 33 of CS336 Language Modeling from Scratch
- Continued Assignment 2, while doing benchmarking mixed precision part, it kept crashing the GPU due to the OOM error so had drop it to smaller batches.
- Completed till 2.1.6 Memory Profiling.
- Continued researching on Context management for Agents using LAYA/JEV as a decision model.
Here is a breakdown of why warm-up iterations are mandatory for GPU benchmarking, since a lot of people measure latency from iteration 1 and get wildly inflated, high-variance numbers.
Step 1 -> CUDA context init, driver loading, PyTorch allocator setup, lazy cuBLAS/cuDNN loading, & optimizer states (often 2–5x slower)
Later steps -> Pure kernel execution reusing pre-allocated memory pools and cached GEMM kernels (steady-state latency)
The benchmarking workflow:
Move inputs to GPU -> run 5–10 warm-up iterations -> torch.cuda.synchronize() -> timed loop (50–100 iterations) -> torch.cuda.synchronize() -> report mean ± std
Why 1–2 warm-up steps still aren't enough: cuBLAS/cuDNN kernel auto-tuning heuristics, autograd backward graphs, and dynamic workspace allocations often take a few iterations to fully stabilize.
Sharing this as a visual reference for anyone benchmarking PyTorch models, profiling custom kernels, or doing GPU performance engineering.
Day 33 of CS336 Language Modeling from Scratch
- Continued Assignment 2, while doing benchmarking mixed precision part, it kept crashing the GPU due to the OOM error so had drop it to smaller batches.
- Completed till 2.1.6 Memory Profiling.
- Continued researching on Context management for Agents using LAYA/JEV as a decision model.
Here is an architectural comparison between Normal LLMs vs Classification / Decision Models (like JEV and LAYA), since a lot of people treat every transformer as an autoregressive text generator.
Normal LLM (Decoder-only) -> Predict what token comes next (causal masked attention)
Classification Model (JEV / LAYA) -> Filter, score, and route in a single pass (bidirectional attention)
The workflow pattern:
Raw input/chunks -> Classification model (JEV / LAYA) -> triage & route -> pass only relevant context -> Normal LLM -> synthesize final response
Why this matters:
Instead of paying quadratic prefill and autoregressive token generation latency just to extract a JSON label or choose a tool, you use a fast single-pass encoder upstream to keep the generative context window lean.
Sharing this as a visual reference for mental map.
Day 33 of CS336 Language Modeling from Scratch
- Continued Assignment 2, while doing benchmarking mixed precision part, it kept crashing the GPU due to the OOM error so had drop it to smaller batches.
- Completed till 2.1.6 Memory Profiling.
- Continued researching on Context management for Agents using LAYA/JEV as a decision model.
Day 32 of CS336 Language Modeling from Scratch
- Playing with Nvidia nsight systems, seeing how to read the reports and all. Still cant find much learning resources for it, recommendations are appreciated.
- Completed section 2.1.4 Nsight Systems Profiler from Assignment 2.
- I'm also focusing on a side research, that is - "Using Jev/Laya like classification/decision models to manage context for AI agents".
- A great relevant research I found was Jev-Mem by Dongming Jiang, Yi Li and Bingzhe Li.
GPUs process thousands of pixels in parallel, but what happens when we write a single pixel to the framebuffer? How is it done?
Watch a CPU paint each pixel in this amazing RISC-V visual simulator (load the bitmap example):
https://t.co/sgv0sZQVva
Here is a comparison and mind map for using Nsight Systems vs Nsight Compute, since there isn’t much beginner-friendly learning material that clearly explains when to use which.
Nsight Systems (nsys) -> Find what’s slow and where
Nsight Compute (ncu) -> Understand why a kernel is slow
The workflow I’m following
Nsight Systems -> identify bottleneck -> select kernel -> Nsight Compute -> analyze metrics -> optimize -> repeat
I also added a practical Triton GEMM on T4 example to connect the profiling concepts with actual kernel optimization
Sharing this as a visual reference for anyone learning GPU performance engineering
Day 32 of CS336 Language Modeling from Scratch
- Playing with Nvidia nsight systems, seeing how to read the reports and all. Still cant find much learning resources for it, recommendations are appreciated.
- Completed section 2.1.4 Nsight Systems Profiler from Assignment 2.
- I'm also focusing on a side research, that is - "Using Jev/Laya like classification/decision models to manage context for AI agents".
- A great relevant research I found was Jev-Mem by Dongming Jiang, Yi Li and Bingzhe Li.
Day 32 of CS336 Language Modeling from Scratch
- Playing with Nvidia nsight systems, seeing how to read the reports and all. Still cant find much learning resources for it, recommendations are appreciated.
- Completed section 2.1.4 Nsight Systems Profiler from Assignment 2.
- I'm also focusing on a side research, that is - "Using Jev/Laya like classification/decision models to manage context for AI agents".
- A great relevant research I found was Jev-Mem by Dongming Jiang, Yi Li and Bingzhe Li.
Day 31 of CS336 Language Modeling from Scratch
- Continued read chapter 3
- Completed benchmarking script - section 2.1.3 from assignment 2.
- Set up of nsys is done, currently working on section 2.1.4 nsys_profile.
Some insights I found while doing section 2.1.3 - Backward dominates training cost, the optimizer was only ~5–6% of the full step.
The first iteration isn't representative. A single warmup dramatically improves timing stability.
CUDA event timing gives a much better measurement of GPU execution than naive CPU-side timing.
Scaling from the small to medium Transformer increased step latency by ~3.1×
Day 31 of CS336 Language Modeling from Scratch
- Continued read chapter 3
- Completed benchmarking script - section 2.1.3 from assignment 2.
- Set up of nsys is done, currently working on section 2.1.4 nsys_profile.
Some insights I found while doing section 2.1.3 - Backward dominates training cost, the optimizer was only ~5–6% of the full step.
The first iteration isn't representative. A single warmup dramatically improves timing stability.
CUDA event timing gives a much better measurement of GPU execution than naive CPU-side timing.
Scaling from the small to medium Transformer increased step latency by ~3.1×
Day 30 of CS336 Language Modeling from Scratch
- Read chapter 3 of Al Systems Performance Engineering.
- Started the Assignment 2 : Systems and Parallelism
- Currently working through Profiling and Benchmarking. Setting up Nsight Systems Profiler.
If you have good resources for profiling, it would be greatly helpful as I couldn't find much apart from the User guide by nvidia.
Day 30 of CS336 Language Modeling from Scratch
- Read chapter 3 of Al Systems Performance Engineering.
- Started the Assignment 2 : Systems and Parallelism
- Currently working through Profiling and Benchmarking. Setting up Nsight Systems Profiler.
If you have good resources for profiling, it would be greatly helpful as I couldn't find much apart from the User guide by nvidia.
Day 29 of CS336 Language Modeling from Scratch
- Completed Lecture 7: Parallelism
- Multi-GPU training is fundamentally about deciding what each GPU should own and minimizing the amount of data and synchronization required between them. Data parallelism splits the batch, tensor parallelism splits computation within layers, and pipeline parallelism splits the model depth; the right choice depends heavily on model size, computation/communication ratio, and hardware topology.
- Fast GPUs alone don't make distributed training fast. The performance comes from matching the parallelization strategy to the communication hierarchy - HBM, NVLink/NVSwitch, InfiniBand, and Ethernet and overlapping communication with useful computation whenever possible.
NUMA-Friendly Memory Allocation and Memory Pinning for GPU workloads.
CPU pinning alone isn’t enough. You also need to make sure the memory being accessed is local to that CPU and the GPU it is feeding.
If a process runs on NUMA node 0 but its memory lives on node 1, every access has to cross the inter-node link. numactl --cpunodebind keeps CPU execution local, while --membind keeps memory allocation on the same NUMA node.
Then there’s pinned (page-locked) memory. Unlike normal pageable RAM, pinned memory can be accessed directly by DMA, making CPU -> GPU transfers significantly more efficient.
Also looked at how this connects to PyTorch pin_memory=True, non-blocking H2D transfers, bandwidthTest, GPUDirect RDMA, GPUDirect Storage, and the OS/container limits on locked memory.
The mental model is simple:
CPU -> local NUMA memory -> GPU
Keep that path local, avoid unnecessary copies and inter-node traffic, and you get lower transfer latency and more predictable GPU utilisation.
Here’s the visual mental model-
NUMA Awareness and CPU Pinning for GPU workloads.
The interesting part is that the GPU doesn’t necessarily need to be changed to get better performance. A lot of the optimisation comes from where the CPU threads run and where memory is allocated.
When a CPU process and the GPU it is feeding sit on different NUMA nodes, data has to cross the inter-node link, increasing latency. CPU pinning keeps the process on the CPU cores local to that GPU, while memory binding keeps allocations in the same NUMA domain.
Also looked at how this works in practice with numactl, nvidia-smi topo, PyTorch DataLoader workers, pinned memory, and non-blocking H2D transfers.
Same workload. Better locality. Less unnecessary data movement. More predictable GPU utilisation.
Here’s the visual mental model for better understanding
I think Stanford's CS336 is one of the best materials to learn LLMs from scratch. I just finished working through it (tokenizer, transformer, triton kernels, distributed training, scaling laws, alignment) and highly recommend it. 🧵 Some notes:
At smaller sequence lengths like 1024, the FFN dominates, not attention, which is a big part of why MoE goes on the FFN. At 16k ctx, attention usually takes over.
Naive DDP on 2 gpus: 54% of each step was gradient sync. overlapping comms with backward: 3%. ZeRO-1 sharding cut peak memory 68.7 -> 47.7 GB.
GRPO: base model with the r1-zero prompt got 0/1300 correct for gsm8k. it gets 224/1300 correct with the 3-shot prompt. RL is most effective when it amplifies signal that's already there.
🔗 check it out: https://t.co/xXOOqAT0RN
Day 29 of CS336 Language Modeling from Scratch
- Completed Lecture 7: Parallelism
- Multi-GPU training is fundamentally about deciding what each GPU should own and minimizing the amount of data and synchronization required between them. Data parallelism splits the batch, tensor parallelism splits computation within layers, and pipeline parallelism splits the model depth; the right choice depends heavily on model size, computation/communication ratio, and hardware topology.
- Fast GPUs alone don't make distributed training fast. The performance comes from matching the parallelization strategy to the communication hierarchy - HBM, NVLink/NVSwitch, InfiniBand, and Ethernet and overlapping communication with useful computation whenever possible.
Day 28 of CS336 Language Modeling from Scratch
FlashAttention 2 Backward Pass
- Today I finished implementing and analyzing the FlashAttention 2 backward pass in Triton.
- The backward pass turned out to be considerably more involved than the forward pass.
The kernel now -
• Recomputes attention probabilities from Q, K and LogSumExp
• Computes the Delta vector from O . dO
• Accumulates dV and dK across query tiles
• Atomically accumulates dQ across KV tiles
• Avoids materializing the full N×N attention matrix
The key part I understood was the work partitioning: each program owns a KV tile and walks through the relevant query tiles. This makes dQ a reduction across multiple program instances, which is why atomic accumulation is required.
Forward pass -> memory efficient attention
Backward pass -> recomputation + tiled reductions + synchronization
Three ways to persist data in Docker:
Volume: Docker manages the path. Survives restarts. Best for databases.
Bind mount: maps a host path into the container. Good for local dev.
tmpfs: RAM only . Gone when container stops.
Volumes for prod. Bind mounts for dev. tmpfs rarely.
Containers on the same machine can't talk to each other by default.
Put them on a user-defined bridge network ,they can reach each other by container name.
No IPs. No hardcoding. Just name resolution.
This is how docker compose networking works under the hood.