satsuki

AIの創造性とAI for Scienceに興味あります！気になる技術やローカルLLMの実測結果をゆるく発信🔬
AI creativity, AI for Science & local LLMs. Experiments, benchmarks, and curious ideas.

[satsuki @satsuki2694]

@wikiwayne Haha, exactly! 😅

We haven’t pushed the Sparks beyond 3 users yet. Maybe when our cluster grows to 4 units!

[satsuki @satsuki2694]

You asked about concurrency, so here we go! 😅🔬
Qwen3.8 Flash Next on Mac Studio M5 Ultra vs dual DGX Sparks.
First, concurrency limit = 1.
With 1–3 users, per-user decode speed stays steady. But requests queue up, so the last user waits longer.
Fast generation, long queues!

[satsuki @satsuki2694]

@rna4219 わあ、日本語版ありがとうございます！😊

逆転するポイントまで書き込んであって、すごく分かりやすい！

自分のデータをこうして別の視点から整理してもらえるの、嬉しいですね！🔬

[satsuki @satsuki2694]

@ivanfioravanti Interesting hypothesis!

We pushed the Mac Studio to 6 users, but haven’t tested the dual Sparks beyond 3 yet.

We’ll probably revisit higher concurrency when our Spark cluster expands to 4 units.

For now, I think the GPUs deserve a little break! 😅

[satsuki @satsuki2694]

@SabatinoMasala Thanks for the heads-up! 😊

I checked NVIDIA’s guidelines, and apparently they recommend a whopping 40 cm of clearance behind the Sparks! 😳

There’s more space than it looks in the photo, but I’ll definitely keep an eye on thermals.

Really appreciate the advice!

[satsuki @satsuki2694]

Work hard, guys!! 😏🔥

[satsuki @satsuki2694]

@cursorvers そうです！

[satsuki @satsuki2694]

リクエストにお応えして、とうとう6人同時まで測定！😅
M5 UltraでQwen Flashを動かしてみた。
6人が利用する場合、並列上限を1→6にすると、最後の人の応答開始は約52→45秒に短縮。
でも、全員の処理完了は約55→65秒と逆に遅くなっちゃった！
並列化すれば速くなる、というわけじゃないんだね。

[satsuki @satsuki2694]

You asked for higher concurrency… so we tested 6 users! 😅
Qwen Flash / oMLX on M5 Ultra (30K input each).
At 6 users, c=6 gets the last user's first token sooner than c=1 (45s vs 52s).
But completing all 6 requests takes LONGER (65s vs 55s)!
More concurrency ≠ faster overall!

[satsuki @satsuki2694]

ちなみに、同時処理を無効にして、1人ずつ順番に処理する場合も調べてみた！
TensorFoldはやっぱり生成速度が速い。
でも90K入力×3人では、全員の処理が終わるまでの時間が
oMLX：約69秒
TensorFold：約80秒
なんと逆転！
長い入力では、Prefillなどの待ち時間も無視できないんだね。🤔

[satsuki @satsuki2694]

TensorFold、さっそく試してみた！
M5 UltraでQwen3.8 Flash NextをoMLXと比較。
30K入力では、TensorFoldの生成速度が約184 tok/s！速いっ！😳
でも、複数ユーザーで長文を処理すると話が変わる。60K〜90K入力では、今回の構成だとoMLXの総合スループットが優勢に。最高速度だけじゃ分からないね！

[satsuki @satsuki2694]

One more detail for the benchmark nerds! With concurrency limited to 1, TF decodes faster than oMLX. But at 90K input, oMLX gets the first token sooner and finishes all 3 queued requests faster!Faster decoding≠faster end-to-end.Different quantization and MTP settings, as always.

[satsuki @satsuki2694]

Someone mentioned TensorFold, so of course we had to try it!
Qwen3.8 Flash Next on our M5 Ultra, 30K input:
TensorFold 6-bit Auto: 183.7 tok/s
oMLX 6-bit MTP: 100.0 tok/s
End-to-end: 54.9 vs 37.9 output tok/s.
TensorFold is seriously fast! 🚀

[satsuki @satsuki2694]

And here's the real question: how fast does the whole job finish?
Despite TF's impressive single-user speed, end-to-end throughput is often similar to oMLX.
With longer inputs, oMLX actually comes out ahead!
Peak generation speed isn't everything.
The full workload matters!

[satsuki @satsuki2694]

並列処理のベンチも見たいという声が多かったので、追加測定してみた！
M5 Ultra vs Spark×2 Qwen3.8 Flash Nextで1〜3人の同時利用を比較。並列処理を増やすと1人あたりの生成速度は低下。でも全体の処理能力がその分増えるわけでもない。
今回はMacが優勢。Sparkは高MTPで3並列が動かないケースも。

[satsuki @satsuki2694]

One important caveat: Spark's MTP4 configuration failed with 3 users at 30K input. We used MTP2 for the 3-user long-input tests.

[satsuki @satsuki2694]

The final question: does more concurrency mean more total throughput?
Not necessarily!
On the Mac, with 30K input, aggregate end-to-end throughput stays around 48–52 output tok/s from 1 to 3 users.
More users, but not 3× the speed!
The best setup really depends on your workload.

[satsuki @satsuki2694]

The final question: does more concurrency mean more total throughput?
Not necessarily!
On the Mac, with 30K input, aggregate end-to-end throughput stays around 48–52 output tok/s from 1 to 3 users.
More users, but not 3× the speed!
The best setup really depends on your workload.