Ramsha Khan

Learning & exploring ML systems & infra, RL

[Ramsha Khan @__ramshaaa__]

@Caffeinix_alche Eid mubarak 🫂✨

[Ramsha Khan @__ramshaaa__]

@thestoicccoder if that's gulab jamun & those on the left are pineapple and watermelon then why aren't they on separate plates! (just my OCD brain thinking xD)

[Khushi🪐 @_khusheyyy]

i feel like all roads lead to

mathematics
physics
philosophy

[Ramsha Khan @__ramshaaa__]

@jino_rohit 🔗https://t.co/3Bpm9nS4JX
I've put a link to blog on online softmax at the beginning that I found quite helpful while learning.
Btw, any feedback is appreciated 🙌

[Ramsha Khan @__ramshaaa__]

@jino_rohit This is cool.
What resources are you referring to?
I tried to write a blog putting together my understanding of flashattn, and I didn't go super deep into online softmax there (but getting to the code took a lot of notebook scribbling & dry runs to really convince my brain)

[Ramsha Khan @__ramshaaa__]

@ashanviii Cooool !

[Ramsha Khan @__ramshaaa__]

@mmaaz_98 @thinkymachines @miramurati Woww congratsss maaz!!

[Ramsha Khan @__ramshaaa__]

@weiiiisuiii Adding one more point: many options out there to create your resume, I prefer: https://t.co/GHKBS17xPC (you can check it out)

[Ramsha Khan @__ramshaaa__]

With NVLink training is faster compared to without NVLink.
If you're curious about NVLink, give this a read: https://t.co/WnqTMoPkXO
Quick note: I might take a break from this series for a few days, will pick it up again soon!

[Ramsha Khan @__ramshaaa__]

Day 14 of learning distributed training:

Exploring GPU topology 👇
X means the GPU is referring to itself (so yeah, no communication with itself xD)
When you see PHB, it means GPUs are connected via the PCIe Host Bridge (CPU) so no direct GPU <-> GPU link
(1/n)

[Ramsha Khan @__ramshaaa__]

Day 13 of learning distributed training:
We've covered collective operations where multiple processes take part in communication.
Now there’s this -> point-to-point communication (one-to-one)
where you pass data from one specific process to another (not all processes).
(1/n)

[Ramsha Khan @__ramshaaa__]

NVLink = direct GPU interconnect -> much higher bandwidth than PCIe
So communication paths actually matter a lot for distributed training.
I was reading a Hugging Face article where they compared DDP performance with & without NVLink, and the difference is pretty clear:
(2/n)

[Ramsha Khan @__ramshaaa__]

https://t.co/jGt9Xd94gI

[Ramsha Khan @__ramshaaa__]

Hi people!
It's been a while since I started exploring distributed training, so I thought I'll start sharing what I'm learning.
(and will try to stay consistent)
Right now I'm starting with parallelism strategies for model training.
(1/3)

[Ramsha Khan @__ramshaaa__]

Both should NOT send at the same time they'll wait forever because both are sending and both will keep waiting to receive leads to deadlock.
Don’t modify the tensor before .wait() when using non-blocking functions.

[Ramsha Khan @__ramshaaa__]

Send the tensor between processes using send() and recv()
There's also isend() and irecv() - non-blocking functions > transfer can happen in the background while some other computation/work runs simultaneously
(2/n)

[Ramsha Khan @__ramshaaa__]

Things NOT to do!
Be careful while setting src & dst and the operation If process0 sends to process1, then process1 should receive from process0
(3/n)

[Ramsha Khan @__ramshaaa__]

Day 13 of learning distributed training:
We've covered collective operations where multiple processes take part in communication.
Now there’s this -> point-to-point communication (one-to-one)
where you pass data from one specific process to another (not all processes).
(1/n)

[Ramsha Khan @__ramshaaa__]

Day 12 of learning distributed training:
We saw a linear relationship between workers and the steps needed for communication, so there was an assumption that latency doesn’t matter much. But in large distributed systems, that assumption breaks as latency is not negligible.
(1/n)

[Ramsha Khan @__ramshaaa__]

@sodakeyEatsMush Glad you liked it, thank youu!

[Ramsha Khan @__ramshaaa__]

A leaf that was idle in Tree A now becomes active in Tree B. So even if some GPUs are idle in one tree, they are active in the other.

[Ramsha Khan @__ramshaaa__]

Day 12 of learning distributed training:
We saw a linear relationship between workers and the steps needed for communication, so there was an assumption that latency doesn’t matter much. But in large distributed systems, that assumption breaks as latency is not negligible.
(1/n)

[Ramsha Khan @__ramshaaa__]

Day 11 of learning distributed training:
Let's keep going with collective ops by zooming into All-Reduce and what's happening behind the scenes.
So I found a couple of ways to do naive all-reduce:
(1/n)

[Ramsha Khan @__ramshaaa__]

Then, to fix the utilization issue, Double Binary Trees were introduced. It’s a sweet spot between the previous two approaches.
There are two trees: a root in Tree A acts as a leaf in Tree B.

(3/n)