anshuman

mostly ml here;

[anshuman @athletic_coder]

fp8 kv cache is pretty normal feat and supported by vllm off the shelf. nothing new here tbh. int4 is slowly getting there and i think there is noticeable quality degradation in this one.

kv cache compression is very interesting line of work. folks at @baseten have gone as far as taking old model like qwen3 and implementing a kv cache compressor (sort of similar to MLA, and CSA from deepseek v4) to cut the memory requirements

[anshuman @athletic_coder]

@dejavucoder banger

[anshuman @athletic_coder]

Recently, we spoke at AI Infra Day by Sarvam , Hugging Face & SGLang.

Our talk was mainly on our in-house distributed training infra we have been building from scratch at Zomato for all of Post Training usecases at Eternal.

Through due research and experiment, we realised over last few months that small models paired with good harness and specialized domain specific post training lead to performance on par with frontier models like GPT, Gemini & Claude at fraction of cost. This eventually contributes to company owning the data, weights and intelligence.

And why build sovereign infra? To make the most out of compute! Because MFU is the king.

By the way, we are hiring a couple cracked people who not only have solid background in Post Training / Harness Engineering but also think from first principles. (If that's you feel free to hmu with a note on what you bring to the table).

That's it. That's the post.

[anshuman @athletic_coder]

Recently, we spoke at AI Infra Day by Sarvam , Hugging Face & SGLang.

Our talk was mainly on our in-house distributed training infra we have been building from scratch at Zomato for all of Post Training usecases at Eternal.

Through due research and experiment, we realised over last few months that small models paired with good harness and specialized domain specific post training lead to performance on par with frontier models like GPT, Gemini & Claude at fraction of cost. This eventually contributes to company owning the data, weights and intelligence.

And why build sovereign infra? To make the most out of compute! Because MFU is the king.

By the way, we are hiring a couple cracked people who not only have solid background in Post Training / Harness Engineering but also think from first principles. (If that's you feel free to hmu with a note on what you bring to the table).

That's it. That's the post.

[anshuman @athletic_coder]

am i the only one who thinks qwen's vibe has changed. (a LOT.)

[Qwen @Alibaba_Qwen]

Since launching Qwen3.8-Max-Preview, we've received valuable feedback from developers!

To help users better explore Qwen3.8's agentic capabilities, we're officially launching the #QwenGrowthPlan today! 🚀

We invite you to:
- Use Qwen3.8 to complete your real-world tasks
- Submit your good or bad cases to us
Every real task is nourishment for Qwen3.8's growth. We've prepared valuable rewards to recognize your active participation.

For more detailed participation info and prizes, please check out the posters below👇. Besides emailing us, feel free to share your awesome cases or feedback directly on X and @Alibaba_Qwen

Join us and let's grow together!

[anshuman @athletic_coder]

@ry4nrodriguez_ what does that even mean?
why do you need to train across tabs?

[anshuman @athletic_coder]

@sama @victormustar do you?

[anshuman @athletic_coder]

@knowShubhangi go for a run

[anshuman @athletic_coder]

@plastixcxx @StalwartCoder @1littlecoder

[anshuman @athletic_coder]

in the next 2 minutes i'll teach you how to pick hardware for local LLMs.

tl;dr
> VRAM tells you what fits. and not how fast model runs.
> decode is limited by how fast memory feeds the chip, not by raw compute. a smaller, better-fed GPU can beat a bigger one.

inference is not one operation. it's two.

"prefill" reads your whole prompt and builds the KV cache. it's heavy on compute.
"decode" produces one token at a time, re-reading the weights and cache every step. it's bottlenecked on memory bandwidth.

so your workload shape changes everything: short prompt, long answer, you live and die by memory bandwidth. long prompt, short answer, attention kernels and the prefill path matter most. many users at once, the scheduler is the whole game.

when picking stack, i've seen people doing this -
a) count the VRAM.
b) find a benchmark screenshot with a big tokens per second number.
c) pick the engine with the biggest number.
d) buy the card with the most memory.

but it's not the appropriate.

let's say benchmark said 180 tok/s figure, it might have come from one user, one prompt shape, probably a 1K in, 128 out toy run. but the coding agent, you are gonna be using local-llm for will drag 80K of context for example

you must not start from the engine.
you must start from: what hardware is actually in the rack. whether the model sits in fast memory or spills into slow shared memory.
whether your problem is prefill or decode.
how long context runs and how many users hit it at once. whether prompts share prefixes you can cache.

answer those and the engine picks itself:
weird, offline, cpu-heavy, edge, GGUF, llama.cpp.
mac with big unified memory, MLX. models fit that never would on a 24gb card, but memory is slower, so you trade speed for capacity. one 4090 or 5090 on low-bit weights, ExLlamaV2.
a few consumer nvidia cards or local MoE, ExLlamaV3.
serving open models in production, vLLM, the safe default.
long context, MoE, routing, disaggregation, SGLang.
all nvidia, squeezing the last drop, TensorRT-LLM. a whole fleet, put Dynamo on top.

and remember - without a fast interconnect like NVLink, spraying tensor parallelism across multiple GPUs can be slower than plain pipeline parallelism. the moment a model crosses a GPU boundary you start paying a communication tax, and cheap PCIe links make you pay a lot of it.

so before anyone argues about which engine wins, answer the fundamental questions first. the hardware and the workload decide. the choice of engine just follows.

[anshuman @athletic_coder]

in the next 2 minutes i'll teach you how to pick hardware for local LLMs.

tl;dr
> VRAM tells you what fits. and not how fast model runs.
> decode is limited by how fast memory feeds the chip, not by raw compute. a smaller, better-fed GPU can beat a bigger one.

inference is not one operation. it's two.

"prefill" reads your whole prompt and builds the KV cache. it's heavy on compute.
"decode" produces one token at a time, re-reading the weights and cache every step. it's bottlenecked on memory bandwidth.

so your workload shape changes everything: short prompt, long answer, you live and die by memory bandwidth. long prompt, short answer, attention kernels and the prefill path matter most. many users at once, the scheduler is the whole game.

when picking stack, i've seen people doing this -
a) count the VRAM.
b) find a benchmark screenshot with a big tokens per second number.
c) pick the engine with the biggest number.
d) buy the card with the most memory.

but it's not the appropriate.

let's say benchmark said 180 tok/s figure, it might have come from one user, one prompt shape, probably a 1K in, 128 out toy run. but the coding agent, you are gonna be using local-llm for will drag 80K of context for example

you must not start from the engine.
you must start from: what hardware is actually in the rack. whether the model sits in fast memory or spills into slow shared memory.
whether your problem is prefill or decode.
how long context runs and how many users hit it at once. whether prompts share prefixes you can cache.

answer those and the engine picks itself:
weird, offline, cpu-heavy, edge, GGUF, llama.cpp.
mac with big unified memory, MLX. models fit that never would on a 24gb card, but memory is slower, so you trade speed for capacity. one 4090 or 5090 on low-bit weights, ExLlamaV2.
a few consumer nvidia cards or local MoE, ExLlamaV3.
serving open models in production, vLLM, the safe default.
long context, MoE, routing, disaggregation, SGLang.
all nvidia, squeezing the last drop, TensorRT-LLM. a whole fleet, put Dynamo on top.

and remember - without a fast interconnect like NVLink, spraying tensor parallelism across multiple GPUs can be slower than plain pipeline parallelism. the moment a model crosses a GPU boundary you start paying a communication tax, and cheap PCIe links make you pay a lot of it.

so before anyone argues about which engine wins, answer the fundamental questions first. the hardware and the workload decide. the choice of engine just follows.

[anshuman @athletic_coder]

@sachaarbonel Banger

[anshuman @athletic_coder]

@dejavucoder let's try this.

[anshuman @athletic_coder]

@HaimantikaM 😭

[anshuman @athletic_coder]

@HaimantikaM on a 2nd thought arogya doesn't have circular tables😂

[anshuman @athletic_coder]

@HaimantikaM arogya ahara/asha tiffins?

[anshuman @athletic_coder]

@vadiamit woah

[anshuman @athletic_coder]

@harshilmathur @kamath_sutra i can go on and on about how true this is based on lived realty

[anshuman @athletic_coder]

@monali_dambre Manthan bhaiya spotted