Kai

Making it nice @sgl_project and @radixark

[Kai @unseenmars_]

@alecqfong goat

[Kai @unseenmars_]

@LD_Theron we ordered them from amazon🖖

[Kai @unseenmars_]

iykyk

[Kai @unseenmars_]

@yishan yes sirrrrrr

[Kai @unseenmars_]

@melvindvivas yes yes and yes

DeepSeek v4 flash felt like THE model for local workflow now. Hope to get nvfp4 soon.

[Kai @unseenmars_]

@Boondit @witcheer cooking this now

[Kai @unseenmars_]

Need for speed ❤️‍🔥

[Charles 🎉 Frye @charles_irl]

the "Fast" is silent

[Kai @unseenmars_]

@usr_bin_roygbiv I also found my coworkers love playing with these DGX.

They're hardcore infra engineers and they do this for a living. It's like F1 drivers who still drive for fun on their days off 🏎️

[Kai @unseenmars_]

It's for people who want to own open frontier intelligence like Inkling-small and DeepSeek V4 Flash!

Lots of people are already running these locally in their actual workflow.

[Kai @unseenmars_]

@mbaril010 @sgl_project hummm… let me dm you

[Kai @unseenmars_]

@mbaril010 @sgl_project yess dm me or

our slack is https://t.co/AwI4NnoTQI

[Kai @unseenmars_]

@WescheNex1q They are in good hands!

[Kai @unseenmars_]

@GoodyTTD Haha they went to @realJerryZhou and @lsyincs’s desks 🧑‍🍳👨‍🍳

[Kai @unseenmars_]

@johnny_ver_2 How about inkling-small and DeepSeek v4 flash? And yes sm120 will be our focus too

[Kai @unseenmars_]

@_MacFadden_ will do 🫡🧡

[Kai @unseenmars_]

@mbaril010 We want to make sure people can run models like Qwen, Inkling, GLM, DeepSeek with good optimization with @sgl_project

[Kai @unseenmars_]

@DegenApeDev for the local AI people 🫡🫡

[Byron Hsu @hsu_byron]

Scaling Kimi K3 on H200s with @sgl_project @lmsysorg

While most discussion on X focuses on Blackwell, I thought it would be useful to share what we learned while tuning Kimi K3 for throughput-oriented workloads on H200s.

- TP+EP beat DP+EP for K3. Only about 1/4 of its layers use gated MLA, so DP attention provides less benefit from avoiding KV-cache duplication. Having each DP rank process every attention head also substantially increased latency. This differs from our K2.6 and GLM setups, where we generally prefer DP+EP.

- FP8 KV cache worked well. We measured no quality or throughput regression in our evaluations, while effectively doubling KV-cache capacity.

- TP32+EP32 provided significantly more cache capacity: approximately 4M KV-cache tokens per replica, compared with about 800K using TP16+EP16.

- For low cache-hit workloads, consider disabling radix cache. Prefix caching requires four Mamba-state slots per sequence in our setup; without radix cache, each sequence needs only one.

- Non-speculative decoding won at high concurrency. Above roughly 64 concurrent requests, draft-and-verify overhead outweighed the savings from accepted speculative tokens in our tests.

- A few caveats: symmetric memory caused instability in our multi-node environment. We also observed hangs when combining speculative decoding, HiCache, and high KV pressure. Both issues are still under investigation.

We’ve been running this configuration at full load across roughly ~O(1000) chips for several days. Aside from hardware failures, it has been reliable.

Feedback, discussion, and better configurations are very welcome! Sharing my launch command in the first comment.

h/t to @hanming_lu @liin1211 @0xishand @ChengWan17 @NealBayya for the discussion

[Kai @unseenmars_]

@aryan559552 Hadn't tried Inkling yet but we had recipe for Inkling-small

https://t.co/qlNQxzEmbj

[Kai @unseenmars_]

@TechMDAI @sgl_project @thinkymachines @NVIDIAAI @realJerryZhou @ispobaoke @u1tra_instinct Let's goooooooo

[Kai @unseenmars_]

More optimization and more models 🐳 to come!

[SGLang @sgl_project]

🚀 SGLang now officially supports @thinkymachines Inkling-Small running on 2× @NVIDIAAI DGX Spark systems connected via ConnectX-7.

Huge thanks to @realJerryzhou @ispobaoke for implementing the support, with valuable community contributions from @u1tra_instinct and @TechMDAI.

This enables strong agentic capabilities to run locally on a compact platform. The current setup achieves 24 tok/s (concurrency=1) without MTP.

DSpark support is coming soon.

Check out the cookbook below to get started 👇