@thestoicccoder if that's gulab jamun & those on the left are pineapple and watermelon then why aren't they on separate plates! (just my OCD brain thinking xD)
@jino_rohit 🔗https://t.co/3Bpm9nS4JX
I've put a link to blog on online softmax at the beginning that I found quite helpful while learning.
Btw, any feedback is appreciated 🙌
@jino_rohit This is cool.
What resources are you referring to?
I tried to write a blog putting together my understanding of flashattn, and I didn't go super deep into online softmax there (but getting to the code took a lot of notebook scribbling & dry runs to really convince my brain)
With NVLink training is faster compared to without NVLink.
If you're curious about NVLink, give this a read: https://t.co/WnqTMoPkXO
Quick note: I might take a break from this series for a few days, will pick it up again soon!
Day 14 of learning distributed training:
Exploring GPU topology 👇
X means the GPU is referring to itself (so yeah, no communication with itself xD)
When you see PHB, it means GPUs are connected via the PCIe Host Bridge (CPU) so no direct GPU <-> GPU link
(1/n)
Day 13 of learning distributed training:
We've covered collective operations where multiple processes take part in communication.
Now there’s this -> point-to-point communication (one-to-one)
where you pass data from one specific process to another (not all processes).
(1/n)
NVLink = direct GPU interconnect -> much higher bandwidth than PCIe
So communication paths actually matter a lot for distributed training.
I was reading a Hugging Face article where they compared DDP performance with & without NVLink, and the difference is pretty clear:
(2/n)
Hi people!
It's been a while since I started exploring distributed training, so I thought I'll start sharing what I'm learning.
(and will try to stay consistent)
Right now I'm starting with parallelism strategies for model training.
(1/3)
Both should NOT send at the same time they'll wait forever because both are sending and both will keep waiting to receive leads to deadlock.
Don’t modify the tensor before .wait() when using non-blocking functions.
Send the tensor between processes using send() and recv()
There's also isend() and irecv() - non-blocking functions > transfer can happen in the background while some other computation/work runs simultaneously
(2/n)
Things NOT to do!
Be careful while setting src & dst and the operation If process0 sends to process1, then process1 should receive from process0
(3/n)
Day 13 of learning distributed training:
We've covered collective operations where multiple processes take part in communication.
Now there’s this -> point-to-point communication (one-to-one)
where you pass data from one specific process to another (not all processes).
(1/n)
Day 12 of learning distributed training:
We saw a linear relationship between workers and the steps needed for communication, so there was an assumption that latency doesn’t matter much. But in large distributed systems, that assumption breaks as latency is not negligible.
(1/n)
Day 12 of learning distributed training:
We saw a linear relationship between workers and the steps needed for communication, so there was an assumption that latency doesn’t matter much. But in large distributed systems, that assumption breaks as latency is not negligible.
(1/n)
Day 11 of learning distributed training:
Let's keep going with collective ops by zooming into All-Reduce and what's happening behind the scenes.
So I found a couple of ways to do naive all-reduce:
(1/n)
Then, to fix the utilization issue, Double Binary Trees were introduced. It’s a sweet spot between the previous two approaches.
There are two trees: a root in Tree A acts as a leaf in Tree B.
(3/n)