Karthik

AI Platform @Pinterest; Previously Mesos and Aurora @twitter

[Karthik @treadstone90]

Hooked on @bot.

Have a fitness bot that is on Hevy + Whoop. Pretty amazing to tell me what I should I focus on next.

Household manager is the one though. Amazon, DoorDash, Instacart. I just talk to it and stuff shows up..

This is going to help optimize so many things at home

[Karthik @treadstone90]

@mlevchin There is also steelman. Seen it enough times now

[Tyler Griggs @tyler_griggs_]

SkyRL now implements the Tinker API.

Now, training scripts written for Tinker can run on your own GPUs with zero code changes using SkyRL's FSDP2, Megatron, and vLLM backends.

Blog: https://t.co/GAtW81jM38
🧵

[Karthik @treadstone90]

Fire Upon the Deep by Vernor Vinge has so many ideas in one book, that is is mind-boggling.

And TIL that he also coined the term "Singularity"

[Dan Go @CoachDanGo]

One of the greatest cheat codes in life is to never get offended. Train yourself to have a thick skin. Don't take things personally. Let others disagree with you. Being easily offended means you're easily manipulated. Want more peace? Avoid getting offended.

[John Ousterhout @JohnOusterhout]

I have uploaded videos of my "Thoughts For The Weekend" to YouTube: https://t.co/DEbpYKqevS

[kalomaze @kalomaze]

we'd be in a world with much better pools of independent research talent if even just 4x PCIe A100s at home didn't cost you several kidneys

[Kamal Ravikant @kamalravikant]

The hardest thing is taking responsibility for everything in your life. It is also the most liberating thing.

[Prasenjit @prasenx]

The voices in my head

[Karthik @treadstone90]

Exciting work using @raydistributed to accelerate ML dataset iteration.

https://t.co/NYx9C3Uvbx

[Robert Nishihara @robertnishihara]

The AI compute software stack consists of 3 specialized layers:

🔧🔧🔧 Layer 1: Training & Inference Framework (PyTorch + vLLM)
• Runs models efficiently on GPUs
• Handles model optimization and model parallelism strategies
• Manages accelerator memory and automatic differentiation

⚡⚡⚡ Layer 2: Distributed Compute Engine (Ray)
• Schedules tasks within jobs and coordinates processes
• Ingests and moves data
• Provides workload-aware failure handling and autoscaling

🏗️🏗️🏗️ Layer 3: Container Orchestrator (Kubernetes)
• Provisions compute resources
• Schedules entire jobs
• Manages user and workload multitenancy

The same recipe: Kubernetes + Ray + PyTorch + vLLM.

I cover a few case studies in this blog, and the results speak for themselves:
• Pinterest cut dataset iteration time from 90 hours to 15 hours (6x improvement) and reduced batch inference cost by 30x.
• Uber increased LLM training throughput by 2-3x.
• Roblox sped up LLM batch inference 9x.

Each layer handles what it does best. The separation of concerns makes this stack so powerful.

https://t.co/tYQ4VD0IRL

[Karthik @treadstone90]

@rasbt Get well soon! Thanks for all your learning materials , deeply appreciated.

[Robert Nishihara @robertnishihara]

Paper from @BytedanceTalk on training graph neural networks on very large graphs. Leverages @raydistributed's fault tolerance and elasticity.

https://t.co/jP794P0d30

[Pinterest Engineering @PinterestEng]

We're excited to share the third installment of our series on Ray at Pinterest: Ray Batch Inference. Discover how we're harnessing the power of Ray to efficiently scale and optimize our machine learning models for batch processing. 🚀 Learn more: https://t.co/pZrsJLOh2H

[Karthik @treadstone90]

More exciting work on how we are optimizing Batch Inference on top of @raydistributed https://t.co/BPP2dWGeDZ

[Path of Men @PathOfMen_]

Men, get to work

[Reads with Ravi @readswithravi]

“Resilience matters in success.

And greatness does not come from intelligence. Greatness comes from character, and character isn’t formed out of smart people: it's formed out of people who have suffered.”

— Jensen Huang

[Soumith Chintala @soumithchintala]

Why do 16k GPU jobs fail?
The Llama3 paper has many cool details -- but notably, has a huge infrastructure section that covers how we parallelize, keep things reliable, etc.
We hit an overall 90% effective-training-time.
https://t.co/5gngOZJHBO

[Pinterest Engineering @PinterestEng]

Part 2 is here! 🥳 Learn about ‘Ray Infrastructure at Pinterest’ written by Chia-Wei Chen, Raymond Lee, Alex Wang, Saurabh Vishwas Joshi, Karthik Anantha Padmanabhan and Se Won Jang. 📌 https://t.co/2w183KtH57