making giant models answer faster @SarvamAI 🏎️ · professional KV-cache hoarder · part-time GPU whisperer · I read NCCL hang dumps for fun (mostly not fun)
@praveingk@SarvamAI Our stack builds on vLLM, SGLang, Dynamo, Mooncake, NIXL, DeepEP, DeepGEMM and FlashInfer, with our own changes across the engine, the cache and the control plane.
Sarvam Token Factory is live 🚀
For the past few months at @SarvamAI, I’ve worked across its serving stack end to end: bringing up models like Kimi K3.1 Pro and GLM 5.3 on our B200 clusters, the multi-tier KV cache, and the prefill/decode handoff. 🧵
@SarvamAI Where it landed:
• Up to 2× faster first token than the stock open-source stack, at the same cost per GPU
• Matching or beating leading global providers on speed and cost across the models we serve
• New open-weight models served from day one
Deep dive: https://t.co/I7dJZTb7Yw
@SarvamAI The pattern every time: a new model quietly breaks an assumption somewhere in the stack, whether that’s a deadlock between nodes, a cache lookup that misses, or a draft model losing state on the way to decode. The job is finding it within hours of a release.
Where it landed:
• Kimi K3.1 Pro: up to 1.9× tokens/GPU vs the best published B200 results
• GLM 5.3: 2× faster on real chat with MXFP8
• Shared cache: GLM 5.3 peak prefill fleet 58 → 34 nodes
Deep dive: https://t.co/I7dJZTb7Yw
Congratulations on the launch! We have been using DeepSeek on Sarvam as the main LLM for our voice support agent. Voice needs very low latency, and we get a p90 time to first token under 600 ms. Our insured members get fast, high-quality help on complex insurance calls.
Gpu memory forgets an agent's prompt in about 4 seconds
agents return from tool calls much later, so nearly every turn re-read the whole prompt
a shared cache tier that holds prompts for hours halved the recompute, across 450B+ tokens a day
Sarvam token factory is GA 🚀