Khoa

mts @radixark. no longer dilly dallying

[Khoa @kwafam7]

@Oasis_a19 📈📈📈

[Khoa @kwafam7]

@Norapom04 crazy

[Khoa @kwafam7]

@tenderizzation tender rizz indeed

[Khoa @kwafam7]

@MainzOnX Can help connect with the broader team. What’s up

[Khoa @kwafam7]

@jino_rohit exciting 👀

[Khoa @kwafam7]

@tenderizzation my TPOT would agree

[Khoa @kwafam7]

SGLang team moves as fast as the SGLang girl btw

[LMSYS Org @lmsysorg]

SGLang day-0 speed on Kimi K3: 423 tok/s (measured on gsm8k), plus RL support ready in Miles @radixark!

How the largest open-source model runs this fast: we natively implemented and deeply optimized K3’s new architecture with fused KDA decode kernels, DP attention, DSpark, PD disagg, and KDA-aware prefix caching. We've passed Kimi Vendor Verifier and are ready for production!

Thanks to @Kimi_Moonshot, @nvidia, @AMD, @KVCache_AI, @modal, and @baseten for building this with us, and to @googlecloud, @nebiustf, @fal, @digitalocean, @runpod, @DeepInfra and @gmi_cloud for serving K3 on SGLang.

Blog, cookbook, benchmarks in the comments.

P.S. This demo video? Kimi K3 made it itself. Play the game 👇

[Khoa @kwafam7]

@Norapom04 does he know

[LMSYS Org @lmsysorg]

Inkling, @thinkymachines' first open model, dropped today: 975B total / 41B active MoE, up to 1M context, reasoning natively over text, images, and audio.

Serving and RL support are already live: you can run and shape it on an open stack, starting now.

Day 0 support on SGLang @sgl_project and Miles @radixark👇

- Inkling's new architecture (ShortConv, attention with relative positional embedding, shared expert sink MoE) is natively implemented and deeply optimized, with prefill full CUDA graph and MXFP8 KV cache

- Full parameter and LoRA RL in a customized Megatron backend, train inference consistency via customized kernels, routing replay, and cross-runtime parameter synchronization

- DFlash speculative decoding from @modal for low-latency serving

Launching now, blog and cookbook in the comments ⬇️

[Khoa @kwafam7]

Imagine how many features you can vibecode with 500 TPS

[LMSYS Org @lmsysorg]

Serving GLM5.2 NVFP4 Agentic Workload with SGLang: How We Reached 500 TPS on 8xB300 at bs=1

In this deep dive, we explain how SGLang reaches 500+ tok/s/user at bs=1 on 8xB300, with 18 to 34% higher single-user interactivity within two weeks since day-0, and 6 to 11% better peak throughput at high concurrency, benchmarked on a real multi-turn agentic coding workload. Our new TopK-V2 kernel is 2.33x faster at 80K ISL, scaling to 10.17x at 1M ISL, keeping interactivity essentially flat out to 1M tokens.

Part of the story is the architecture itself. GLM-5.2 applies IndexShare to its DSA layers and ships a stronger MTP head reusing IndexShare and KVShare. The rest comes from our serving optimizations.

Special thanks to @NVIDIAAI for the help in day-0 support of GLM-5.2 NVFP4, and to @Zai_org for IndexShare in SGLang!

[Khoa @kwafam7]

@Norapom04 what dense attn does to a mf

[Khoa @kwafam7]

Best spec dec algo 🤝 best spec dec runtime. Only at @sgl_project

[LMSYS Org @lmsysorg]

SGLang now supports DSpark, enabling confidence-driven, variable-length verification for speculative decoding 🎉

DSpark addresses a key bottleneck under load: instead of verifying every draft token, it verifies only where the draft model is confident, so the gains hold even as batch size scales.

We heavily optimized variable-length verification in SGLang. Across batch sizes 1 to 256, DSpark gives the best throughput/latency tradeoff on DeepSeek-V4-Flash, ahead of both MTP and non-spec.

At high concurrency, dynamic scheduling provides up to ~20% higher throughput compared to a fixed budget, while maintaining high verification quality across workloads.

With fused kernels and zero-overhead scheduling, DeepSeek-V4-Pro reaches 383.7 tok/s at B=1 on B300.

DSpark is now available in SGLang with support for Qwen3 and DeepSeek-V4. Thanks @deepseek_ai for open-sourcing!

Blog with full technical details and commands to run below 👇

[Khoa @kwafam7]

@tenderizzation banger

[Khoa @kwafam7]

@Norapom04 @ grok is this true

[Khoa @kwafam7]

@_dcw02 @tenderizzation same

[Khoa @kwafam7]

@Leik0w0 so true

[LMSYS Org @lmsysorg]

🚀New record on GB300 NVL72: SGLang exceeds 12K tok/s per GPU on DeepSeek V4 Pro 1.6T (FP4, 8K/1K), orchestrated with NVIDIA Dynamo (SGLang) and MTP.

Per @SemiAnalysis_ InferenceX benchmarks, performance stays strong across the entire interactivity curve.

More to come with @NVIDIAAIInfra 🤝

[Norman Mu @TheNormanMu]

sglang downloads 7x'ing since april is an interesting signal for open-weight models https://t.co/IYvUr9r6T1

[Charles 🎉 Frye @charles_irl]

The Nemotron series is impressive -- strong capabilities in an efficient form factor, with high-performance implementations (esp on Blackwell) in open source.

Deploy the new Nemotron 3 Ultra (550B-A55B-NVFP4) on @modal starting from this recipe:

https://t.co/JahPuS8zB0

[Khoa @kwafam7]

@eliebakouch @TheZachMueller many such cases