Danny Liu

gpu @wafer_ai

[Danny Liu @gpudanny]

@adamheimann @wafer_ai @AMD scale up scale down, amortization across workloads, caching via l2/l3.

also use an moe instead of 27b dense for prefill speed. also dont use vllm for mamba if you want caching to work properly for those long prefills. sglang radix cache sir is better for this sir.

[wafer @wafer_ai]

🚨 BREAKING:

these engineers figured out how to serve Kimi K3 on @AMD MI355X at 952 tok/s/node and 118 tok/s single stream!

this crushes B200 by 3.8x in aggregate throughput/node and 1.3x in single stream decode + beats B300 on performance per dollar (48 vs 33 tok/s/$)

See how in the thread.

[Danny Liu @gpudanny]

@dhruvbhatia0 noooo come back

[Danny Liu @gpudanny]

kimi

[steve @gpusteve]

my team figured out how to run Kimi K3 on @AMD MI355X at 952 tok/s/node and 118 tok/s single stream.

3.8x the aggregate throughput/node and 1.3x the single stream decode of B200 and beat B300 on performance per dollar: 48 vs 33 tok/s/$

details in reply

[Danny Liu @gpudanny]

@hsu_byron @sgl_project @lmsysorg concurrency sir

[Danny Liu @gpudanny]

@hsu_byron @sgl_project @lmsysorg also was 75 with dspark at c=1-4 with ib? we saw perf bottlenecked by roCE on b200s but never tested ib :(

[Danny Liu @gpudanny]

@hsu_byron @sgl_project @lmsysorg would love to see interactivity/throughput on a cluster of h200

[Danny Liu @gpudanny]

@AAAzzam @OpenRouter @modal sure man

[Danny Liu @gpudanny]

@gpusteve what if you just cached the text tho.

[steve @gpusteve]

kimi k3 fast is live on @OpenRouter

fastest serverless kimi k3.

pick @wafer_ai as your provider: https://t.co/0fB588bjrZ

[Danny Liu @gpudanny]

@gpuemi

[Danny Liu @gpudanny]

holy decode

[Danny Liu @gpudanny]

@mert @gpusteve

[Danny Liu @gpudanny]

@gpusteve @ShubhamInTech not the 10 card draw

[steve @gpusteve]

find next hire. impossible edition. 💀😭💀😭💀😭

[Danny Liu @gpudanny]

@gpusteve

[Danny Liu @gpudanny]

@gpusteve where is the smash requirement

[Danny Liu @gpudanny]

@ianye0212 @gpusteve alright

[Danny Liu @gpudanny]

@gpusteve