Cody Mazza-Anthony

Machine Learning @shopify. Former quant.

[Shopify Engineering @ShopifyEng]

Starts in 3 hours at COEX Hall D1 (11:30am KST) — great lunch break talk. @Drewch and @cmazzaanthony show how we cut GraphQL agent serving costs 96% while still beating frontier models on quality.

👊 Looking forward to a great onsite discussion!

[Shopify Engineering @ShopifyEng]

Shopify's LLMs beat frontier models on a range of tasks at a fraction of the cost. The reason: we put systems in place that enable them to improve themselves, learning from a range of commerce tasks every day.

We're presenting our Model Optimization Flywheel at @ICMLconf: a continuous pipeline that turns Shopify's product expertise into robust evals, mines low-scoring conversations, critiques them, repairs them, and feeds them back into the model. Then we compress the prompts without losing quality, so we can make it faster and cheaper.

We present an example of the flywheel working at scale: our GraphQL agent. Serving cost dropped from $27M to $1M annualized (−96%). We compressed our system prompt 4× and still beat frontier models on quality.

@Drewch and @cmazzaanthony will share concrete recipes, quality-cost-latency trade-offs, and a blueprint you can actually build from.

📅 Monday, July 6 · 11:30am–12:30pm KST
📍 COEX, Hall D1

Link in thread. 👇

[Cody Mazza-Anthony @cmazzaanthony]

I rebuilt my personal website (https://t.co/tVfipcx9gl) from scratch with no frameworks or build tools. Just HTML, CSS, and the simplest possible markdown rendering. Turns out "as little as possible" is pretty liberating.

[Cody Mazza-Anthony @cmazzaanthony]

Just discovered this amazing repository of machine learning algorithm implementations! Check it out 👉🏼 https://t.co/OTFpjiRRqz. The Gaussian mixture model and linear model implementations were particularly enjoyable. #AI #ML

[Cody Mazza-Anthony @cmazzaanthony]

Game changer for inference of open source models! 🚀

[Jay Scambler @JayScambler]

Groq is serving the fastest responses I've ever seen. We're talking almost 500 T/s!

I did some research on how they're able to do it. Turns out they developed their own hardware that utilize LPUs instead of GPUs. Here's the skinny:

Groq created a novel processing unit known as the Tensor Streaming Processor (TSP) which they categorize as a Linear Processor Unit (LPU). Unlike traditional GPUs that are parallel processors with hundreds of cores designed for graphics rendering, LPUs are architected to deliver deterministic performance for AI computations.

The LPU's architecture is a departure from the SIMD (Single Instruction, Multiple Data) model used by GPUs and favor a more streamlined approach that eliminate the need for complex scheduling hardware. This design allows every clock cycle to be utilized effectively, ensuring consistent latency and throughput.

For developers, this means that performance can be precisely predicted and optimized which is critical in real-time AI applications.

Energy efficiency is another area where LPUs shine. By reducing the overhead of managing multiple threads and avoiding the underutilization of cores, LPUs can deliver more computations per watt.

Groq's innovative chip design allows multiple TSPs to be linked together without the traditional bottlenecks found in GPU clusters making them extremely scalable. This enables linear scaling of performance as more LPUs are added simplifying the hardware requirements for large-scale AI models and making it easier for developers to scale their applications without rearchitecting their systems.

So what does this all mean? LPUs could provide a massive improvement compared to GPUs for serving AI applications in the future! If anything it will be great to have alternative high performing hardware since A100s and H100s are so in demand

[Cody Mazza-Anthony @cmazzaanthony]

Exciting news! The collaboration between @huggingface and @googlecloud makes deploying on Vertex AI hassle-free. Hopefully, I won't have to manually transfer model files anymore. Looking forward to trying it out! #AI #LLM https://t.co/WPznvs8vt5

[Cody Mazza-Anthony @cmazzaanthony]

Check out this awesome Lora finetuning tutorial (https://t.co/KRGz3m29vr) by @LightningAI! Learn how to write a Lora finetune from scratch using Pytorch. #AI #PyTorch #LLM

[Cody Mazza-Anthony @cmazzaanthony]

Love this linear algebra reference material by @zicokolter!

https://t.co/J9MJhuztKw

It's my go-to. I always bookmark it for quick review. Highly recommend! #Math #MachineLearning

[Cody Mazza-Anthony @cmazzaanthony]

"Competence is often less of a problem than confidence.

An underrated aspect of doing anything hard is believing in yourself. Action creates both confidence and momentum ..."

A Tiny Thought found in the @farnamstreet Weekly Newsletter https://t.co/OprUpBzXVt

[Cody Mazza-Anthony @cmazzaanthony]

I struggle to write system prompts. I prefer a more TDD approach to writing system prompts. Prompts Royale makes it easy (https://t.co/REsmHVYdEn). It takes your test cases and generates multiple system prompts, and chooses the best one. Game-changing tool! #promptengineering

[Cody Mazza-Anthony @cmazzaanthony]

Just watched A Hackers' Guide to Language Models (https://t.co/zuGgEEhkgY). I love the practical approach to understanding LLMs, it reminded me of Andrej Karpathy's Zero to Hero series.

[tobi lutke @tobi]

It's amazing that you can start a fresh python notebook, and this code is all you need to run hardware accelerated, streamed, inference on your mac.

It even downloads the model.

[Cody Mazza-Anthony @cmazzaanthony]

Loved this blog post!

[Chip Huyen @chipro]

Open challenges in LLM research

The first two challenges, hallucinations and context learning, are probably the most talked about today.

I’m the most excited about 3 (multimodality), 5 (new architecture), and 6 (GPU alternatives).

Number 5 and number 6, new architectures and new hardware, are very challenging, but are inevitable with time. Because of the symbiosis between architecture and hardware – new architecture will need to be optimized for common hardware, and hardware will need to support common architecture – they might be solved by the same company.

I referenced a lot of papers here, but I have no doubt that I still missed a ton. If there’s something you think I missed, please let me know!

https://t.co/Al2b2Zjqb7

[Cody Mazza-Anthony @cmazzaanthony]

Cool!

[AI at Meta @AIatMeta]

Today we’re releasing Code Llama, a large language model built on top of Llama 2, fine-tuned for coding & state-of-the-art for publicly available coding tools.

Keeping with our open approach, Code Llama is publicly-available now for both research & commercial use.

More ⬇️

[Cody Mazza-Anthony @cmazzaanthony]

Looking to enhance your training data? Instruction backtranslation could be a simple yet effective technique to boost data quality.

[Jason Weston @jaseweston]

🚨New Paper 🚨
Self-Alignment with Instruction Backtranslation

- New method auto-labels web text with instructions & curates high quality ones for FTing

- Our model Humpback 🐋 outperforms LIMA, Claude, Guanaco, davinci-003 & Falcon-Inst

https://t.co/93qi4JDnpb
(1/4)🧵

[Cody Mazza-Anthony @cmazzaanthony]

Very excited about this🚀

[OpenAI @OpenAI]

We've just launched fine-tuning for GPT-3.5 Turbo! Fine-tuning lets you train the model on your company's data and run it at scale. Early tests have shown that fine-tuned GPT-3.5 Turbo can match or exceed GPT-4 on narrow tasks: https://t.co/VaageW9Kaw

[Cody Mazza-Anthony @cmazzaanthony]

So excited to be a part of the sidekick team! Definitely the best thing I have ever worked on!

[tobi lutke @tobi]

I've got something very exciting to announce:

Entrepreneurs are heroes, and every hero needs a Sidekick ✨