In this (extremely long) video, I'll build a distributed training framework from first principles using PyTorch.
We'll derive everything from first principles, explaining the mathematics behind distributed training as well as every auxiliary component we build along the way, including Multi-head Latent Attention (MLA), Rotary Position Embeddings (RoPE) and YaRN.
I'll explain process groups, ranks, collective communication operations, distributed autograd, device meshes and distributed tensors, and then use these foundations to implement the major forms of parallelism used to train modern language models.
We'll derive and code pipeline parallelism, data parallelism, FSDP, HSDP, tensor parallelism and context parallelism, including pipeline schedules, block matrix multiplication, model sharding and ring attention. We'll then combine these techniques and examine how tensor layouts and gradients must be tracked as data moves between devices.
Most importantly, we won't study each parallelism technique in isolation. We'll combine pipeline, data, tensor, context and expert parallelism into a single working framework and follow the tensors, communication operations and gradients as they move across GPUs. By the end, you'll understand not only how these techniques work individually, but how modern distributed training systems compose them to train large dense and mixture-of-experts models.
We'll also build the important components of a modern transformer, including rotary embeddings, long-context extensions, multi-head latent attention and a mixture-of-experts architecture. This leads into expert parallelism, expert tensor parallelism, token dispatching, all-to-all communication and combining different parallel dimensions through dense and sparse device meshes.
We'll also build the supporting training infrastructure, including the training loop, optimizer, learning-rate schedule, checkpointing and metrics.
The goal is to give you the foundations required to understand how distributed training frameworks work internally and how to design and implement new parallelization strategies yourself. No prior knowledge of distributed training is required.
Link to the video: https://t.co/ioSYostMYS
Coaching industry is huge in India minting billions. What PM has announced with online direct coaching by GoI is going to hurt them for sure.
Which means they are going to organize and not just create protests, but may program every student against the PM and his policies in their institutes.
So this is a brave move by PM. "Political suicide", some would say, given the GoI with its MIB is unable to even counter blatant fake news propaganda.
But good luck to the PM fighting the coaching mafia. Regulating them has to be done at some point in time by govt. Let this be the start.
Of course, many of these coaching institutes are genuine and do a great service to the nation. We should not forget this.
Every coaching institute cannot be called "coaching mafia." Hope they will join this effort of the PM.
Warm greetings on the occasion of Independence Day.
We remember with gratitude the countless freedom fighters whose courage, sacrifice and unwavering commitment ensured the end of colonial rule. Their dreams continue to inspire us as we work together to build a Viksit Bharat.
Powered by 140 crore Indians, our nation is scaling new heights of progress across different sectors. May this journey keep progressing at an even greater pace in the times to come.
सभी देशवासियों को स्वतंत्रता दिवस की असीम शुभकामनाएं।
यह राष्ट्रीय पर्व हमें देश के वीर सेनानियों के अद्भुत साहस, अथक संघर्ष और मातृभूमि के प्रति अतुलनीय समर्पण का स्मरण कराता है।
आइए, उनके आदर्शों को अपनाकर विकसित भारत के संकल्प को साकार करें।
THE MISSION SHIFTS. THE COMPASS HOLDS.
This was never the plan. Initially, the plan was to climb just G2. But just before setting off for Pakistan, I ran the numbers on my 8000m summits. That’s when it hit me: if I tick off Broad Peak whilst I’m here , only one remains “Cho Oyu.” Then I become the FIRST person in history to climb all 14 eight-thousanders twice. Without oxygen.
This wasn’t planned. The Hattrick project was the focus because it carried a bigger mission. But opportunities don’t scream: they whisper to those already working in silence.
Here’s the secret I’ve carried through every expedition: I never competed with anyone else. My only battle was against the man I was yesterday. Every single day, pushing my own limits. Because the moment you look sideways: at what others are doing, saying, achieving: you drift off your compass. Your heading is YOURS. Guard it fiercely.
They criticized me when I climbed the 14 without O’s during Project Possible. If you haven’t seen what that took, watch 14 Peaks on Netflix. That film is the unfiltered truth. I was there doing multiple rescues of climbers left behind by other expeditions, raising the funds myself, leading from the front, fixing our own lines, dealing with the Shishapangma permit with China, being at hospitals for my mum’s operation and so much more. And now? Now I’m heading for the double without oxygen: triple in total. The noise hasn’t stopped. I just stopped listening.
To those who still doubt: I already climbed all 5 of Pakistan’s 8000ers in just 26 days. Leading. Guiding. Without oxygen. Breaking the records of those who broke mine with oxygen. The receipts are there.
Broad Peak, I ask for nothing but safe passage up and back down. I take zero mountains for granted. Not one. The moment my foot leaves basecamp, it’s 100%. Always has been. Always will be.
This one is dedicated to everyone who carried me here. Supporters. Critics. All of you. Without both sides, there is no fire. So thank you genuinely.
My purpose has never been about me. It’s about what I represent. It’s about showing YOU that your own mountains—whatever they are—are climbable.
#nimsdai #eliteexped #everest
India's Jadumani Singh after defeating Pakistan's Sumama Rehman in Boxing
"I dedicate my win to the Kargil heroes... I felt great winning against Pakistan. I will perform even better in the quaterfinals and secure a Gold for India."
🇮🇳💪
The 2026 Fields Medals have been awarded at the International Congress of Mathematicians in Philadelphia.
The recipients are:
• Yu Deng (University of Chicago)
• John Pardon (Stony Brook University)
• Hong Wang (New York University & IHES)
• Jacob Tsimerman (University of Toronto)
The Fields Medal recognizes outstanding mathematical achievement by researchers under 40. Each post below summarizes the official citation and key contributions.
MathCast IV
Today, we interviewed four of the six members and the team leader of India's team at the International Mathematical Olympiad 2026, Shanghai, for our fourth math podcast. We could not the remaining two team members as they had to catch an early flight to Delhi.
India won two gold medals and four silver medals at the 2026 IMO, ranking seventh out of 117 countries. ThiswasI our second best performance since 1989.
Team members interviewed (standing, left to right)
Abel George Matthew (Karnataka)
Sanjana Philo Chacko (Keralam)
Bairav Murugan (Karnataka)
Shreya Shantanu Mundhada (Maharashtra)
Team members we missed
Aarav Gupta (Delhi)
Kanav Talwar (Delhi)
Team leader (seated, right)
Anant Mudgal
The podcast will be released in a week or two.
Leandro Paredes should be banned from international football for life viciously assaulting Spanish players after the final whistle.
Indefensible thuggery and criminality.
@LtGenDPPandey Argentina has a really solid team — they’re not just relying on Messi. When you watch their matches, the whole squad steps up and performs at a high level. But Spain yesterday was on another planet. What a perfectly coordinated, united team — they played like one single force.
@rathor7_@rathor7_ No one builds any country's image single-handedly. As a public influencer, have some shame. India is much bigger than anyones need for building its image.
During the war with Pak, wasn't he supporting his friend travelling to Pakistan & meeting officials?"
🇮🇳GOLDEN SWEEP FOR INDIA 🏆
All 5 Indian students win GOLD at the 56 International Physics Olympiad (IPhO) 2026 in Bucaramanga, Colombia -placing India at Rank #1 in the world (jointly with China, Kazakhstan, Russia, South Korea & Taiwan) among 381 students from 87 countries 1/5
Anthropic is hiring a Research Scientist (post PhD) with biology & computational skills in its life sciences division for a $300-$320 K salary (2-3 times what a starting Assistant Professor would make in academia, after a serious postdoctoral fellowship).
Brave new world!