One more interesting finding:
For FP32, estimated memory traffic per element:
Eager: 16 bytes
Compiled: 8 bytes
Potentially 50% less memory traffic!
Numerical comparison:
Max absolute difference: 4.77e-7
Results remained within relaxed FP32 tolerance.
Experimenting with torch.compile in PyTorch: Eager vs Compiled execution.
I benchmarked a simple operation:
y = x * scale + bias
The interesting part? Two element-wise operations can potentially be fused into a single GPU kernel.
My results (NVIDIA T4, FP32):
Why the difference?
Row-major accesses contiguous memory, utilizing CPU cache lines efficiently.
Column-major jumps between memory locations, hurting cache efficiency.
The same principle applies to GPU tiling and optimized GEMM kernels.
Detailed Blog -https://t.co/zhLmeiYbff
Today, I benchmarked row-major vs. column-major traversal in C++ to understand cache locality.
The results:
- Row-major: 9.31 ms
- Column-major: 422.74 ms
That's a 45.41× slowdown just by changing memory access patterns!
But why does this happen?
Having doctor friends around you hits differently. Their dedication to learning and improving constantly motivates you to study harder, push your limits, and build something meaningful in your own field.
Together, accounting for ~89% of self CPU time.
Key takeaway: Linear layers and Attention are heavily dependent on matrix multiplications.
Profile first. Optimize later.
Next, I'm exploring how torch.compile changes execution and performance.
#PyTorch#LLMInference
Learning LLM Inference Engineering
Today I Profiled a Tiny Transformer using PyTorch Profiler.
Two operators dominated self CPU time:
aten::addmm: 57.86%
aten::bmm: 30.88%
Built my first tiny Transformer using PyTorch!
Implemented Q, K, V attention, softmax, residual connections, layer normalization, and a feed-forward network.
Moving from understanding the theory to writing the code myself.
Code -https://t.co/LtqgwGzI1Z
AI Engineer working on LLMs,RAG,MCP,AI agents & Python backend in my day to day life .
Now exploring LLM Inference Engineering — Transformers, PyTorch, CUDA & GPU optimization.
For those in this space, any advice on where to start or resources to follow?
Open to suggestions!