Monsoon

[Akshay 🚀 @akshay_pachaar]

LLM engineer's handbook

(30 minutes a day, 10 weeks, 50 lessons)

a roadmap for llm inference serving where everything points at one service instead of scattering across demos. you get the mental model first, then serve a model, instrument it, load test it past 1000 concurrent requests, and tune it. you finish with a stack you configured yourself and a benchmark worth publishing.

here is what it covers:

→ the roofline model, and why decode waits on memory while prefill waits on compute

→ vLLM internals, PagedAttention and the scheduler, read from the code

→ Prometheus and Grafana for TTFT, inter-token latency, and queue depth

→ SGLang and RadixAttention prefix reuse, benchmarked against vLLM

→ load testing past 1000 concurrent requests

→ quantization across FP16, FP8, and INT4, on quality as well as speed

→ speculative decoding and KV eviction, including where the gains disappear

→ disaggregated prefill and decode, deployed on Kubernetes

→ a cost, latency, and quality router with per-request token budgeting

→ publishing a reproducible benchmark

the roadmap on GitHub: https://t.co/pOqkWvDJ2d

(don't forget to star 🌟)

i am also writing an article for each major topic. the first one is out, on how a GPU actually works.

the article is quoted below.

[Monsoon @monsoonic]

Favorited "Dub Transmission w/ Alecsey - 6th June 2023" https://t.co/1DNpQZLKCL by Black Rhino Radio

[Monsoon @monsoonic]

@nicoperez Amazing, favorite London company hosts event at favorite London venue, can't get better than that 🔊🍹💯

[Monsoon @monsoonic]

Just favorited "Headzmusik w/ WRK - 16th February 2022" by @BlkRhinoMusic on Mixcloud https://t.co/4dx2IPVdzb

[Monsoon @monsoonic]

Just loved "Dub Transmission w/ Alecsey - 23rd November 2021" by @BlkRhinoMusic on @mixcloud https://t.co/tr8CRuv056

[Monsoon @monsoonic]

Favorited "BeatCast w/ Misbits - 21st October 2021" https://t.co/cDLqnCoMjG by @BlkRhinoMusic

[Monsoon @monsoonic]

Just favorited "#SmirnoffHouse 2017: Lauren Lo Sung at Parklife" by Mixmag on Mixcloud https://t.co/X5y27Z4WpR

[Monsoon @monsoonic]

Favorited "[simple:tracks] w/ Victor Stancov - 26th August 2021" https://t.co/VQnM8tteHP by @BlkRhinoMusic

[Monsoon @monsoonic]

Favorited "All aboard w/ Issa - 11th August 2021" https://t.co/l3Qo1lCtBl by @BlkRhinoMusic

[Monsoon @monsoonic]

Just loved "Headzmusik w/ WRK - 7th of July" by @BlkRhinoMusic on @mixcloud https://t.co/kX1GQaZrKF

[Monsoon @monsoonic]

Just loved "Night Walk | Deep House Set | DEM Radio Podcast" by Johnny M on @mixcloud https://t.co/CddDyFYV3t

[Monsoon @monsoonic]

Just loved "EXPLORATIONS #003" by CHVL on @mixcloud https://t.co/GurLH8SCa5

[Monsoon @monsoonic]

Favorited "EXPLORATIONS #002" https://t.co/rkvny4Hc8h by CHVL

[Monsoon @monsoonic]

Favorited "Mr Goju - Between Dreams" https://t.co/SGWL39nlVF by @Mr_Goju

[Monsoon @monsoonic]

Just loved "Fold & Loods - 10th May 2018" by NTS Radio on @mixcloud https://t.co/7b7oIdzkok

[Monsoon @monsoonic]

Just favorited "GourmetBeats SubFM Dec 2018" by @joenicedj on Mixcloud https://t.co/3jmLbLjIeV

[Monsoon @monsoonic]

Just loved "Kamaal Williams Live From Flesh & Bone Studios" by NTS Radio on @mixcloud https://t.co/fhfFHqdsOE

[Monsoon @monsoonic]

Just loved "Mr Goju - Last of The Summer Wine 2018" by @Mr_Goju on @mixcloud https://t.co/3Ihx0YssgL

[Monsoon @monsoonic]

Favorited "Dubstep TakeOver 'New Era' - Episode 1 - Waves" https://t.co/9XOVaivS4I by Mouse Dubz

[Monsoon @monsoonic]

Favorited "Social Sundays 011: ZO" https://t.co/WOkoiBQrUu by @BlkRhinoMusic