Jaikumar Ganesh

Engineering Leader @ Baseten

ex Head of Engineering at @anyscalecompute, Uber, Android@Google

[Jaikumar Ganesh @jaikumarganes]

Reconnecting with old colleagues from Uber after 8 years and them recollecting how happy they were on the team #special_teams

The same is happening at @baseten
Ping me

[Jaikumar Ganesh @jaikumarganes]

Excited to share that I have joined Baseten.

A month ago, one of my former colleagues, who works on RL and ML infrastructure, remarked that “something magical is happening at @baseten .”

It's hard to describe, but you know it when you see it.

And I can confirm it is happening here.

The right environment, right people, market timing, conviction and clarity - with everyone rowing in the same direction - creates a different kind of energy, a fun environment, a sense of momentum, and possibility - hard to manufacture but unmistakable when you experience it.

We are hiring across the board -  inference, forward deployed engineers, infra, solution architects, marketing, gtm, sales, recruiting….

Now is the time to experience magic. Join us!
https://t.co/MUlV1DnWQv

@amiruci @tuhinone

[Robert Nishihara @robertnishihara]

The AI compute software stack consists of 3 specialized layers:

🔧🔧🔧 Layer 1: Training & Inference Framework (PyTorch + vLLM)
• Runs models efficiently on GPUs
• Handles model optimization and model parallelism strategies
• Manages accelerator memory and automatic differentiation

⚡⚡⚡ Layer 2: Distributed Compute Engine (Ray)
• Schedules tasks within jobs and coordinates processes
• Ingests and moves data
• Provides workload-aware failure handling and autoscaling

🏗️🏗️🏗️ Layer 3: Container Orchestrator (Kubernetes)
• Provisions compute resources
• Schedules entire jobs
• Manages user and workload multitenancy

The same recipe: Kubernetes + Ray + PyTorch + vLLM.

I cover a few case studies in this blog, and the results speak for themselves:
• Pinterest cut dataset iteration time from 90 hours to 15 hours (6x improvement) and reduced batch inference cost by 30x.
• Uber increased LLM training throughput by 2-3x.
• Roblox sped up LLM batch inference 9x.

Each layer handles what it does best. The separation of concerns makes this stack so powerful.

https://t.co/tYQ4VD0IRL

[Jaikumar Ganesh @jaikumarganes]

@jonsid @ashugarg @FoundationCap Unstructured data processing is going to become more popular esp with video, audio and other modalities becoming mainstream. Check out Ray Data on https://t.co/ZuYklVWqAJ. lots of cool talks on how enterprises are using Ray Data at the #RaySummit

[Jaikumar Ganesh @jaikumarganes]

Would love to chat with @raydistributed users while I am in Bengaluru. DM me. @anyscalecompute

[Jaikumar Ganesh @jaikumarganes]

@JAGGlutenFree Thanks for writing the book and all the posts. Do you have a recommended list of safe restaurants in Bangalore for Celiacs?

[Jaikumar Ganesh @jaikumarganes]

This used to be such a pain 5 years back.

Excellent blog post showing how easy it is to build an image search and retrieval system for e-commerce using Ray and Anyscale.

https://t.co/47kfhihnbZ

@raydistributed
@anyscalecompute

#PowerOfPlattorm.

[Jaikumar Ganesh @jaikumarganes]

@FlyAirNZ When do you plan to improve your airpoints program? The original plan reported was mid 2023

[Waleed Kadous @waleedk]

📊 Case study of Llama-2’s capability in summarization tasks.
📌 TLDR: In summarization; Llama-2-70b is as factual as GPT-4 while being 30x cheaper 🧵 👇

https://t.co/ujso0HDjrZ

[Robert Nishihara @robertnishihara]

We've built a ton of #LLM applications recently. Reasoning about performance & feasibility is painful without reference points.

Here are the reference points we use to anchor our intuition (inspired by @JeffDean's "Numbers every engineer should know").

https://t.co/586bDcAgJ8

[Jaikumar Ganesh @jaikumarganes]

@jasvir Amazon lost that package

[Harrison Chase @hwchase17]

lots of people have asked how to use langchain with self-hosted LLMs

really excited to see this tutorial! @raydistributed are extremely knowledgable, anything they share is worth paying attention to

[Richard Liaw @richliaw]

Everyone is excited about LLMs and generative AI, but existing infra tools struggle for operationalizing these new large models.

Companies are using @raydistributed for solving these problems. Check out our new blog:

https://t.co/jKxuzO6r8P

[Jaikumar Ganesh @jaikumarganes]

@AKL_Airport please fix baggage claim at international terminal. It takes close to 1 hr to get bags

[{Jules Damji}=💻📚📝 @2twitme]

💔 what #GPT-3 thinks of @raydistributed at this prompt. Check it out 👇 #rayrocks

[Jaikumar Ganesh @jaikumarganes]

@benthompson any plans to do a deep dive on Snowflake/Databricks and their business models in your articles and also the AI cloud computing startups?

[Jaikumar Ganesh @jaikumarganes]

@NZ_Customs Can you please fix the long lines at the Auckland International airport at arrivals?