Ramchalam K R

Staff Machine learning Research Engineer, @qualcomm AI research | @mcgillu grad.

Working on GenAI efficiency, inference, foundation models, SpD, dLLMs.

[Ramchalam K R @ramchalamkr]

@rasbt Looking forward to reading it 😃

[Ramchalam K R @ramchalamkr]

@AryaTschand Yea that's a fair point. I was thinking more in terms of newer chips using TMEM, which reduces the need to store the accumulation in registers, which could reduce spillover potentially and thus it becomes more about latency bound (cache movement) than memory or compute bound.

[Ramchalam K R @ramchalamkr]

@CyrusHakha Great read! One question regarding the kV transfer, does it matter if the kV transfer from prefill to decode nodes follow different distributed parallism techniques and whether it complicates the end to end latency? Going from TP to PP for eg.

[Ramchalam K R @ramchalamkr]

@PavloMolchanov @sid_dev1 Would it still not evict the weights of some of the layers from the cache depending on the model and cache size ? So during one denoising step also there could be eviction right ? And this in turn across the time steps.
Thanks

[Ramchalam K R @ramchalamkr]

@eisokant Congratulations 👏

[Ramchalam K R @ramchalamkr]

Provides a good intuition on how turboquant actually works with clear examples. Worth a read for sure :)

[surya @suryasure05]

wrote an article breaking down the math behind TurboQuant by @GoogleResearch.

I walk through a toy example using concrete numbers to show every single operation that goes on under the hood.

link below:

[Ramchalam K R @ramchalamkr]

@suryasure05 @GoogleResearch Yes that's right. I think beyond a specific dimension it does become close to a Gaussian and I think d=128 is possibly sufficiently Gaussian. Overall the concept and intuition is much clearer in your post with the actual numbers. :)

[Ramchalam K R @ramchalamkr]

@ashwingop Wow. A very elegant optimization on top of turboquant. One thing that's still hard to wrap my head around is the value 4 being the signal and the rest being noise which is supposedly model agnostic. So does it seem more like it is a byproduct of how we do the pretraining of LLMs?

[Ramchalam K R @ramchalamkr]

@anirudhbv_ce @GoogleResearch Wow. Very very impressive. Kudos

[Ramchalam K R @ramchalamkr]

@bubbleboi @TheGregYang Excellent summarization and the potential impact of the memory /hw :)

[Ramchalam K R @ramchalamkr]

@Hxlfed14 A very interesting read on the agent harness. Good insights:)

[Ramchalam K R @ramchalamkr]

@o_v_shake I just checked it out. The last few sections are literally what I have been wanting to update my knowledge base on. Thank you very much for the initiative and sharing to the public:)

[Ramchalam K R @ramchalamkr]

Finally took some time and effort this weekend to dive deeper into NCCL primitives and how they actually power distributed training & inference.

I focused on the "Big Three" and their specific roles in LLM scaling
• All-Gather → weight gathering (tensor parallelism)
• Ring All-Reduce → gradient aggregation (data parallelism)
• All-to-All → token routing (MoE architectures)

The big takeaway across all these 3 approaches? All three use ring topologies to achieve near-constant bandwidth scaling. No parameter server bottleneck, even with 100+ GPUs.

Wrote a technical breakdown with examples + pseudo code.
Part of an ML Systems Efficiency series I'm starting: https://t.co/27jUNgeqyV

Huge thanks to these resources once again:
@huggingface Ultra-scale Playbook
ML Systems for Dummies by @o_v_shake

Happy to hear feedback or answer questions - Feel free to DM.

P.S. Diagram generated with Gemini @antigravity - turned out pretty good!

#LLM #DistributedTraining #NCCL #GPU #MLOps #DeepLearning #MachineLearning

[Ramchalam K R @ramchalamkr]

Recently, I have been digging into distributed training and NCCL primitives at a lower level. We usually talk a lot about the various architectural tweaks within #LLMs, but a little less on the intricacies of data movement.

When you’re training an LLM across 128 GPUs, every single GPU calculates its own gradients on its micro-batch of data. To move forward, every GPU needs the sum of all those gradients.

At first glance, it seems straightforward: just send everything to a central server, sum it up, and send it back. But if you try that with a 70B parameter model, your central server immediately becomes the bottleneck!

The Math of the Bottleneck: In a naive setting, if you have 𝐍 GPUs and a model of size 𝐌, the main server must receive 𝐌𝐍 data and send 𝐌𝐍 data. As you add more GPUs, the server's bandwidth requirement scales linearly. The system eventually chokes.

This is where 𝐑𝐢𝐧𝐠 𝐀𝐥𝐥-𝐑𝐞𝐝𝐮𝐜𝐞 comes into play

𝐒𝐭𝐞𝐩 1: 𝐑𝐞𝐝𝐮𝐜𝐞-𝐒𝐜𝐚𝐭𝐭𝐞𝐫 The model is broken into 𝐍 chunks. Each GPU sends a chunk to its neighbour and receives one. After 𝐍-1 steps, each GPU holds the complete sum for exactly 1/𝐍 of the parameters.

𝐒𝐭𝐞𝐩 2: 𝐀𝐥𝐥-𝐆𝐚𝐭𝐡𝐞𝐫 The GPUs pass these completed sums around the ring until everyone has the full picture.

The Key Point: In this setup, each GPU only sends and receives roughly 2𝐌 worth of data—specifically ((2(𝐍-1)/𝐍) 𝐱 𝐌).

The brilliance here is that the data sent per GPU is virtually independent of the number of nodes. Whether you have 8 GPUs or 100, the communication overhead per device stays nearly constant.

If you're looking into Systems for ML or LLM Inference, this is one of the foundation. These are some of the best resources I’ve found that digs deeper into various parallelism approaches like FSDP, TP, PP etc. and NCCL primitives.

@huggingface Ultrascale playbook - https://t.co/T5vawRyCtr.
@o_v_shake ML for Dummies: https://t.co/T9TiYP4Xap

#DeepLearning #DistributedSystems #NCCL #LLMs #MLOps #GPU

[Ramchalam K R @ramchalamkr]

Although even without explicit token dropping, I believe your kV compression work combined with the skip softmax (blaast work) optimization on flash attention should still provide memory io speedup on top of compute speedup on attention. Will double check if the papers already have that linked inside the work.

[Ramchalam K R @ramchalamkr]

@p_nawrot Thanks for linking the works. Seems literally what I was hoping to see. Appreciate it.

[Ramchalam K R @ramchalamkr]

@p_nawrot I do wonder if we combine this with token dropping, then technically it should provide further improvements, albeit at potential accuracy drop, maybe a Pareto can be shown for certain models on certain tasks. Very interesting indeed. Thanks

[Ramchalam K R @ramchalamkr]

@p_nawrot Excellent technique. Really caught my attention on how we can save further memory footprint:)