lorenzo

Software engineer
#AI #LLMs #Transformers #PyTorch

[Towards Data Science @TDataScience]

Follow along @l_cesconetto's latest tutorial to learn how the Zero Redundancy Optimizer works, how to implement it from scratch, and how to use it in PyTorch. https://t.co/KxuhugTtuZ

[Towards Data Science @TDataScience]

Curious about how the zero redundancy optimizer works? You're in luck: @l_cesconetto devotes the latest installment of his series on multiple-GPU AI workflows to ZeRO, including a from-scratch implementation in PyTorch. https://t.co/KxuhugTtuZ

[Towards Data Science @TDataScience]

Learn how to implement gradient accumulation and data parallelism from scratch in PyTorch — @l_cesconetto presents a comprehensive guide as part of his "AI in Multiple GPUs" series. https://t.co/42wXn6XVPL

[Towards Data Science @TDataScience]

If you've been following along @l_cesconetto's series on AI in multiple GPUs (and even if you haven't), you don't want to miss his brand-new installment, focusing on gradient accumulation and data parallelism. https://t.co/42wXn6XVPL

[Towards Data Science @TDataScience]

How do GPUs communicate? @l_cesconetto explores the hardware infrastructure behind multi-GPU communication for AI workloads. https://t.co/qmOA70SRXU

[lorenzo @l_cesconetto]

go from zero to hero in GPU for AI in this series.
follow me for more high quality free AI content

[Towards Data Science @TDataScience]

Pushing ahead with his series on AI in multiple GPUs, @l_cesconetto zooms in on the hardware infrastructure that enables multi-GPU communication for AI workloads. https://t.co/qmOA70SRXU

[Towards Data Science @TDataScience]

Pushing ahead with his series on AI in multiple GPUs, @l_cesconetto zooms in on the hardware infrastructure that enables multi-GPU communication for AI workloads. https://t.co/qmOA70SRXU

[lorenzo @l_cesconetto]

https://t.co/mSdRVhZE5K

[lorenzo @l_cesconetto]

New post on @TDataScience! 🎉 I dive into point-to-point and collective operations. I cover must-know concepts if you're working with distributed ML systems. Link in thread! 🧵

[lorenzo @l_cesconetto]

if you’re an engineer excited about AI but never used GPUs, this is the perfect intro for you.

In this @TDataScience series I’ll gently take you from zero to hero on multi-GPU AI workloads.

Follow me to read the next posts!!

https://t.co/Saars4XxN5

#NVIDIA #GPU #CUDA #NCCL

[lorenzo @l_cesconetto]

I worked through the math behind RoPE and turned it into an intuitive visual explanation.

it was just published in @TDataScience!!

https://t.co/O4iVbPrn5C

[lorenzo @l_cesconetto]

I worked through the math behind RoPE and turned it into an intuitive visual explanation.

it was just published in @TDataScience!!

https://t.co/O4iVbPrn5C

[lorenzo @l_cesconetto]

Gradient Accumulation (GA) + Distributed Data Parallel (DDP) implemented from scratch!

In this fourth blog post, we dive into:

- The importance of the batch size in neural network optimization
- Why you might need GA & DDP and what problem it solves
- We developed GA and DDP from scratch to make sure you understand how it works under the hood

https://t.co/34OSpXehTq

#PyTorch #DDP #DeepLearning #MLOps #DistributedTraining