Marwan Sarieddine

AI engineer @anyscale. Ex-founding engineer Infima technologies

[Marwan Sarieddine @Marwan1112]

Excited to talk and be present at this year's Ray Summit #RaySummit2024

Join us (and hundreds of other AI leaders) in San Francisco from September 30-October 2nd.

Save your spot today at https://t.co/ogLLObdbvx

[ray @raydistributed]

🚨The Ray Summit Training Guide is here!🚨 Check it out and make the most out of your experience.

🎓 Explore all the classes available,
🍎 Discover personalized learning paths (LLMs, Ray Core, and more),
🎯 Reserve your spot early to avoid missing out! 🚀 #RaySummit

Training Guide: https://t.co/NFBNbXLF2A
Sign up today: https://t.co/VTdE4cbXB9

[Robert Nishihara @robertnishihara]

Running hands on training sessions is something we started in the very early days of Ray. We got this idea of course from the @ApacheSpark community.

Here are two photos from the early days
1. The first Ray meetup, hosted at @OpenAI.
2. A tutorial we ran at an @OReillyMedia conference.

[kourosh hakhamaneshi @CyrusHakha]

OpenAI just released Structured outputs. We've been working on this problem for a while at @anyscalecompute. It's tricky to get right, especially for production-ready large-scale serving systems. Some technical insights from our work: ↓ 🧵

[Marwan Sarieddine @Marwan1112]

A great way to articulate the behavior of compound AI systems of the future by @robertnishihara

[a16z @a16z]

An upcoming focus on video will change how teams train and run generative models, @robertnishihara says.

On the latest AI + a16z podcast, he shares thoughts on:

🎥 Video data
🧑‍💻 Productionizing AI models
🏫 Spinning a company out of Berkeley, where the Sky Computing Lab (the successor to RISELab and AMPLab) has spawned many successful open source projects and companies.💡

[Robert Nishihara @robertnishihara]

The core Ray API has remained the same since we first built it (with roughly two major API additions), but the library ecosystem on top has pivoted wildly. The vision was always somewhat analogous to the Python ecosystem: (1) build flexible lower level primitives (like functions and classes), and (2) build a rich ecosystem of libraries on top of those primitives (like numpy, pandas, pytorch, fastapi, etc).

The first two libraries we built on top of Ray were Ray Tune for hyperparameter tuning and RLlib for reinforcement learning. Why was that? It was somewhat coincidental.

Hyperparameter tuning is the idea of running many copies of a machine learning experiment (typically a training run) with different “hyperparameters” (e.g., learning rates, layer sizes, dropout rates, etc). There was a ton of research activity around techniques like grid search, random search, Bayesian optimization, and so on. Running hundreds of experiments is compute intensive and requires scaling, and we needed to do it for our own research, so hyperparameter tuning was a natural first use case.

Of course, today as people train larger and larger models, we’ve entered a regime where running many full-scale experiments is prohibitively expensive, and so the problem statement has changed. The question is now more about how to progressively move in a principled way from small scale experiments (smaller models, smaller compute budgets), to larger experiments in a way where you can choose the right parameters at a large scale based on the experiments you ran at a small scale. And much more of the experimentation and iteration is around the sourcing and refinement of the data rather than just the training parameters.

As for reinforcement learning, the whole field was excited about reinforcement learning ever since DeepMind demonstrated some successes with Q-learning on Atari games. There was an explosion of algorithms like DQN, A3C, TRPO, SAC, and many many more. Reinforcement learning was a great fit for Ray because it mixed CPU compute (for simulations) along with GPU compute (for training), which needed to be scaled independently, and because it combined workloads ranging from data handling to simulation to inference. This level of heterogeneity was hard to achieve with other systems, and so when we wrote the 2018 OSDI paper on Ray, we used RL as a motivating example.

Because of that, a lot of people associated Ray with reinforcement learning, and it wasn’t until recently that we shook the assumption that “Ray is just for reinforcement learning.”

Next was an early library for scaling deep learning training (initially called Ray SGD, now Ray Train). This library went through a ton of iteration over the years, and companies like Uber were key design partners in this process. The most important development here was around Ray Train’s data ingest story. In order to do training, it was clear that we needed to do some amount of data processing, if only to load the data and preprocess it first. That insight led to a small utility library termed “parallel iterators.” Six months later, we added a lightweight dataset API for Ray Train to facilitate data loading and various preprocessing transformations (sharding, batching, etc). A year later, this emerged as a standalone library (Ray Data, initially called Ray Datasets). Ray Data has blossomed as a standalone library today for all sorts of data handling and batch inference, but it’s also an essential part of the training story. Training on top of Ray is not really just Ray Train, it is Ray Train plus Ray Data. This combination plays to Ray’s strengths of handling GPU intensive and data intensive workloads.

One of the most important trends in AI over this time period has been the shift (or expansion) in focus from just iterating on model architectures to iterating on datasets. A decade ago, researchers worked with static datasets like ImageNet and tried to invent better model architectures and optimization techniques. While that’s still immensely important, now a ridiculous amount of time, money, and compute go into improving the training data (because the quality of the training data plays such a big role in the quality of the eventual model). This means classifiers to filter out low quality data, a lot of inference to generate additional labels, data augmentation, synthetic data generation, and so on. In some cases, preparing pre-training data can be as GPU intensive as training.

In parallel with the training and data efforts, we began working on a model serving library. The inspiration for this began back at Berkeley with a model serving project called Clipper (not based on Ray). Ray Serve was experimental for quite a while, initially focused on horizontally scaling Ray actors behind an HTTP server. It’s grown far more mature as the requirements around model serving have evolved to include model composition, heterogeneous hardware, model multiplexing, independent autoscaling of different replicas, and so forth.

Model serving is continuing to evolve rapidly. As an example, a personal assistant product that uses agents to complete end-to-end tasks (like booking an Airbnb) may involve 100 model calls. These model calls typically start out as foundation model calls and many get replaced with fine-tuned smaller models as latency becomes an issue. The composition of model calls is extremely dynamic based on the output of previous calls, and there are many models in play.

I skipped a few things, but that more or less brings us to where we are today. Along the way, there was an effort termed Ray AIR (AIR stood for AI Runtime), aimed at improving the interoperability of the Ray libraries, since they had evolved independently, but the interfaces between them were increasingly important (especially data + train).

[Marwan Sarieddine @Marwan1112]

Take a look at our open-source implementation for building a scalable cross-modal search application on Anyscale. Find the repository on GitHub: https://t.co/A20ArH6GN3 🌟

I provided a detailed guide and code walkthrough in this blog post: 📝
https://t.co/nbxytG0VCu

[Anyscale @anyscalecompute]

We built an end to end cross-modal image retrieval system for e-commerce on Anyscale with @deepsense_ai ! The application enables a user to discover items by describing them in text or uploading a similar image.

App: https://t.co/qmok5zSajC
Blog : https://t.co/RsshFxznv7

[Marwan Sarieddine @Marwan1112]

Sharing an optimization for pre-filling a transformer's KV cache.

A KV cache is used to speed up LLM inference by avoiding the recompute of keys and values in attention layer forward passes.

However this introduces overhead negatively impacting decoding if implemented naively

[Anyscale @anyscalecompute]

Recently, we’ve contributed chunked prefill to @vllm_project, leading to up to 2x speedup for higher QPS regimes!

In vLLM, prefilling, which fills the KV cache, and decoding, which outputs new tokens, can interfere with each other, resulting in latency degradation. 1/n

[Marwan Sarieddine @Marwan1112]

@BassemDy lovely to see - red cups put to good use!

[Marwan Sarieddine @Marwan1112]

Check out my detailed guide on utilizing Ray on Anyscale for distributed training of Stable Diffusion. The key points we cover include:

1. Performing online preprocessing on a heterogenous cluster of resources
2. Utilizing Ray Train's fault tolerance to prevent costly failures

[ray @raydistributed]

See how we optimized large-scale ML training in Part 3 of our Stable Diffusion series! We used Ray Train, Ray Data, and PyTorch Lightning to train on 2B images with fault tolerance, data streaming, and advanced strategies like FSDP and DDP.

Read more: https://t.co/9ttI6iLi9u

[Marwan Sarieddine @Marwan1112]

Excited to present an in-depth code walkthrough of our Stable Diffusion implementation using Ray on Anyscale in an hour long webinar format

Register here to learn more: https://t.co/3uROsqJ2jk

[Robert Nishihara @robertnishihara]

Deep dive on large-scale training (stable diffusion models).

https://t.co/Fdveh729yK

[Robert Nishihara @robertnishihara]

Data ingest is a bottleneck for training (as you scale).

@metaai wrote a great blog a while back outlining these challenges and how they solved them (by scaling data ingestion and training independently).

As far as I know, Ray is the only open source framework that enables this. And the reason is that Ray was designed for heterogeneous compute (e.g., mixed CPU + GPU workloads). As you scale training, you end up with workloads that are simultaneously data intensive and GPU intensive.

https://t.co/o1iNoJtFWL

[Marwan Sarieddine @Marwan1112]

When constructing distributed data pipelines, maximizing hardware utilization is crucial.

To achieve this, leverage Anyscale's observability tools (Grafana + Prometheus + Ray dashboard) to fine-tune your data pipelines and optimize throughput.

[Marwan Sarieddine @Marwan1112]

Check out the reference implementation I recently published on creating a scalable data processing pipeline using Ray on Anyscale for preprocessing 2 billion images.

1️⃣ Download LAION 5B datasets to S3 with Ray.
2️⃣ Process and encode data on CPUs and GPUs.

[ray @raydistributed]

Here is part 2, zooming way in on data preparation (along with runnable code).

https://t.co/2XtLAySlPS

[Marwan Sarieddine @Marwan1112]

Maximize cost efficiency in data processing by using Anyscale's spot instance support and Ray's object store to stream data directly from Hugging Face to AWS S3.

This approach allows you to download the LAION 5B dataset at a fraction of the cost compared to using img2dataset

[Robert Nishihara @robertnishihara]

When scaling training on GPUs, it's common to become bottlenecked by data ingest and preprocessing. When training stable diffusion models, that "preprocessing" may include running other models to embed the text & image inputs (a pre-trained VAE and a text encoder (OpenCLIP-ViT/H) in this case).

One way to eliminate this bottleneck is to scale the data ingest and preprocessing on a separate pool of CPUs (or even a different type of GPUs).

We're doing a deep dive webinar here: https://t.co/mAtqbSHZZO

[Marwan Sarieddine @Marwan1112]

You can view the full ablation study here:

[Marwan Sarieddine @Marwan1112]

An insightful post offering valuable pointers that can be applied broadly when optimizing your distributed training pipelines 🧵

[ray @raydistributed]

We pretrained a stable diffusion model on 2 billion images for under $40K. Here's what we learned.

https://t.co/ZfXl5WgXHN

[Marwan Sarieddine @Marwan1112]

Enhance your infrastructure by boosting network bandwidth with an elastic fiber adapter EFA in case IO is a bottleneck 📈

Based on the Stable Diffusion ablation study, EFA provided a 2.6x training throughput speed up