Marin 535B-A23B is ~7% done training, and so far things look on track. Next Tuesday (Sept 1 @ 10 PT), we will have a Zoom panel/discussion where the Marin team will talk about the design decisions that went into this run, the tradeoffs made, and our learnings. If you're interested, join the Discord for details (https://t.co/xwJSxd2IxH).
Kubernetes meets LLM inference.
Google, NVIDIA, IBM, and Red Hat are all backing the same open-source project to make it work.
the problem is that LLM inference does not scale the way normal web services do, and the usual Kubernetes answer makes it worse.
let me explain:
run one vLLM or SGLang server and the KV cache is a clean win. the server keeps the attention keys and values for tokens it has already processed, so a prompt that shares a prefix with an earlier one skips past that computation and starts generating.
put a standard Kubernetes Service in front of several replicas and that saving mostly evaporates. the Service hands each request to whichever pod is next in rotation, and that pod usually never saw the prefix, so it recomputes the entire context from scratch.
round-robin assumes every replica serves every request equally well. true for stateless web traffic. false the moment prefill caching exists, because replicas now differ by what they remember.
teaching Kubernetes that difference turns into four problems.
→ knowing which replica holds the prefix. each server streams an event every time it creates or evicts a cache block, and the router keeps a live index of who holds what.
→ knowing when to ignore that index. cache affinity pulls traffic onto warm replicas, so past a load threshold the router drops affinity and picks on load alone. otherwise the warm replica turns into the bottleneck.
→ extending where the cache lives. accelerator memory fills fast, so blocks spill to CPU memory and then disk. on four H100s at 250 concurrent users, that hierarchy delivered 13.9x the throughput of keeping everything on the GPU.
→ separating prefill from decode. prefill is compute-bound, decode is memory-bandwidth-bound, and running both on one replica underuses each. AWS measured up to 70% higher tokens per second after splitting them onto dedicated pools, though the KV cache now has to cross the network before the first token appears.
the KV cache stops being something a single server manages. it becomes cluster state, and the routing layer has to track it.
solve that and you get roughly 3x the output throughput and half the time to first token, on the same hardware, running the same model.
llm-d is the project handling all four on Kubernetes. it sits above vLLM and SGLang rather than replacing them, so you keep whichever engine you already run and it takes over the routing, the cache index, the offloading, and the prefill/decode split. Apache 2.0, CNCF sandbox, with Tesla, Snowflake, Cohere and DigitalOcean running it.
check it out on GitHub: https://t.co/IX1vW7phsj
i wrote the full breakdown of how inference works underneath all of this. the article is quoted below.
Over 75 yrs ago, Shannon estimated that English is ~75% redundant. When you consider the long-range context of a phrase or sentence, the entropy is no more than ~1 bit per letter. It wasn't until very relatively recently that text encoders achieving this rate were created.
In fact, modern LLMs predict the next character so accurately that they routinely compress English text to under 1 bit per character.
Important for non-technical folks to understand what’s going on.
Chinese AI labs are publishing and building on each other’s innovations. Especially in compute and memory efficiency.
Their open-source strategy is a bet that the whole is greater than the sum of the parts.
Bro made it, incredible.
GitHub Trending is a black box, and making it to the top is getting harder every day.
There is no luck involved here. This project is pure hardcore engineering.
If you build agents and want to understand how they are structured, check out this repo.
Give the URL to your agent and ask it to generate a beautiful architecture diagram for your project.
Simple, useful, and seriously impressive. Bravo!