I run precision machining operations.
Building toward an AI-native manufacturing company.
head full of:
→ agents watching production lines
→ LLMs reading quality reports
→ models learning from every defect
→ workflows for dark factories
Personal notes. Not a company account.
Four years ago, I bought a Lange 1 and joined the waitlist for a steel Odysseus.
Got the call last week. I can pick it up this month.
My brain is so full of AI and factory construction these days that buying a watch feels strangely unfamiliar and impractical. A little escape.
Already excited
Huge thanks to @MiaAI_lab. Your recipe gave my 8× DGX Spark setup a massive performance boost.
Further tuning, SGLang’s bounded replay, and rhys’s SG18/RoCEnante stack brought it to the speeds below.
Nearly two weeks with DeepSeek V4.1 Flash, and I’m spending more time putting it to work than tweaking it. Thanks to everyone sharing their work!
The M5 Ultra looks seriously impressive too. Seems like there’s still plenty of room for optimization.
Great numbers on the new M5 Ultra.
Here's Qwen3.8-Flash on 2x DGX Sparks:
Prose on 4 streams: 146.0 tok/s
Single stream prose - 56.8 tok/s
Prefill is 3500 tok/s
So basically the same?
Starting from @MiaAI_lab's recipe (FlashInfer b12x for block-FP8 dense, NCCL buffer trim, fused MoE finalize off), I added two flags:
1. --enable-decoder-swa-bounded-replay
Prefill +44–65%, decode −1–4%, outputs unchanged. It's documented in the V4.1 tech report, but I hadn't seen it used in a Spark recipe.
2. --min-free-slots-delay 1
SGLang classifies DSpark as DFlash-family and silently admits only 7 streams when max-running-requests >= 8. This enabled true 8-way admission and improved c8 aggregate by 43%.
Final config:
DSpark block 3, decode CUDA graphs bs 1–8, prefill graph off, chunked prefill 2048, KV fp8_e4m3 page 256, mem fraction 0.83, FlashInfer 0.6.18, official weights untouched.
Tested and rejected on TP8:
DSpark block 4/5, chunk 1024/4096/8192, NCCL 4/16 channels, cuDNN MXFP8, NCCL_IB_MERGE_NICS, FlashInfer 0.7, SPS, and ragged verify. The official DSpark block 5 default was 15% slower here.
Single stream:
prose 48 · JSON 62 · reasoning 63 · math 74 · coding 77 · counting 84 tok/s
Thinking on:
prose 61 tok/s
At 128K context:
3.5K tok/s prefill, 43 tok/s decode
Public 9-category prompt set, temperature 0, streaming, 3-run median.
DeepSeek-V4.1-Flash on 8x DGX Spark, SGLang TP8/EP8.
48 tok/s single-stream prose
179 tok/s aggregate at c8
~4.4K tok/s prefill at 32K
2.6M-token KV pool, 256K context
Official MXFP4 expert / FP8 dense weights.
Vision + tools on.
Huge credit to @MiaAI_lab. Their 3x Spark recipe took my single-stream decode from 38 to 48 tok/s.
Two additional flags unlocked the concurrency and prefill gains. Details below.
Almost got my 40 tok/s
GLM-5.3 743B on 8× DGX Spark:
38.8 tok/s single-stream prose
1,266 tok/s prefill @ 100K
110.7 tok/s aggregate @ c8
524K context
The surprise: shorter speculation won.
DFlash2 k7 → built-in MTP k2.
For prose, the longer draft was costing more to verify than it returned.
I was chasing 40. I'll take 38.8 for now :)
Built on @Tech2Wild's Int4/Int8Mix quant and his 4× TP4 recipe as the starting point.
Serving stack: eugr's spark-vllm-b12x image (local-inference-lab vLLM fork + b12x kernels).
Meanwhile on the big brother:
GLM-5.3 743B (Int4-Int8Mix) on 8× DGX Spark. TP8.
Still tuning, but already:
c1 · temp 0
structured · 78 tok/s
code · ~45 tok/s
prose · 35 tok/s
Prefill · ~1.1K tok/s
490K-token needle · PASS
KV pool · 944K tokens
Context · 512K
DFlash2 k7 + NVFP4 KV on the b12x stack.
Prose is still putting up a fight.
More tuning tonight.
Recipe drops when it stabilizes.
Base lineage: @Tech2Wild's TP4 recipe
Stack by @wrldsuksgo2mars · draft by @inco_ai
Serving details for the 8× DGX Spark GLM-5.3 run.
HARDWARE / FABRIC
8× DGX Spark (GB10, 128 GB unified each), one head + seven workers
MikroTik CRS804-4DDQ, 200G RoCE per node, MTU 9000 end-to-end (per @Tech2Wild's recipe gotcha)
Each CX7 exposes two functions: enp1s0f0np0 / enP2p1s0f0np0 and rocep1s0f0 / roceP2p1s0f0. Separate /24 per function, both HCAs in NCCL_IB_HCA, RoCEv2 GID index 3
No PFC / ECN
SOFTWARE
eugr/spark-vllm-b12x:latest (2026-09-04): vLLM 0.1.dev20489 (local-inference-lab dev/jovian-judgement), b12x 1.3.0, PyTorch 2.13 / CUDA 13.0, NCCL 2.29.7
Driver 580.x, kernel 6.17-nvidia, 64 GB swap with vm.swappiness=100, min_free_kbytes=2 GB
Model: Tech2wild/GLM-5.3-Int4-Int8Mix (743B, ~378 GB), local NVMe copy on every node
SERVING CONFIG
TP8 / PP1, one container per node, workers first, head last
--quantization compressed-tensors, --attention-backend B12X
--speculative-config {"method":"mtp","num_speculative_tokens":2,"attention_backend":"B12X"}
--max-model-len 524288, --max-num-seqs 8, --max-num-batched-tokens 8192
--kv-cache-dtype fp8_ds_mla, --kv-cache-memory-bytes 36 GiB (pinned), --gpu-memory-utilization 0.84
--enable-prefix-caching, --async-scheduling, cudagraph_mode=FULL_AND_PIECEWISE
Container: --memory 112g, --ipc host, --network host, /dev/infiniband, unlimited memlock
THE TWO KNOBS THAT MATTERED MOST
VLLM_ENABLE_ROCE_ALLREDUCE=1 — b12x RoCEnante one-shot RoCE all-reduce/all-gather for TP collectives instead of NCCL
VLLM_B12X_MLA_CKV_GATHER=0 — removes the size cap on the MLA compressed-KV gather; long prefill was slower with the cap enabled
WHERE THE 38.8 CAME FROM
(same image, same benchmark protocol)
v9 as-is, DFlash2 k=7: 19.5 tok/s prose
speculation off: 20.0
built-in MTP k=2: 28.5
MTP k=2 + RoCE all-reduce: 38.1 / 38.8 / 38.0 across three sessions
Why k=2 for prose: MTP k=2 accepts ~2.2–2.3 tokens/step at ~51 ms/step. Verification costs ~4.7 ms per drafted token here, so a 7-token draft spends more on verification than it returns on open-ended text.
Code accepts more and can still favor longer drafts.
RESULTS
Single-stream prose: 38.1–38.8 tok/s
median of 10 runs, 250 output tokens, wall-clock incl. TTFT
Prefill: 1,406 / 1,324 / 1,266 tok/s at 32K / 70K / 100K
Concurrency, aggregate tok/s:
c1 → 36.6
c2 → 58.4
c4 → 62.4
c8 → 110.7
TTFT: 250–610 ms
--max-num-seqs 8 vs 4: no measurable single-stream penalty
Cold load ~13 min, warm ~8 min
GOTCHAS
Prefill decays with node uptime. The same launcher went from ~1,400 tok/s after reboot to ~600 later in the day. Restarting vLLM doesn't recover it; rebooting the nodes does.
The bad state shows page compaction failing on all eight nodes (compact_fail ≈ compact_stall, no order ≥10 free blocks). Correlation only; cause not proven.
Stock vLLM doesn't boot this model on GB10. The b12x sm12x kernels (sparse MLA + fp8 KV path) in the image above are required.
The fabric is lossy: no PFC/ECN. It has been fine for pure inference and RoCE error counters stay flat. Training-style all-to-all does produce tx drops and rising packet_seq_err.
METHOD
Single stream: 10 unique prose prompts (6 Korean, 4 English), 250 output tokens each; median wall-clock throughput
Prefill: unique 32K / 70K / 100K-token documents, single request
Concurrency: N simultaneous 250-token prose streams; aggregate = sum of per-stream tok/s
Meanwhile on the big brother:
GLM-5.3 743B (Int4-Int8Mix) on 8× DGX Spark. TP8.
Still tuning, but already:
c1 · temp 0
structured · 78 tok/s
code · ~45 tok/s
prose · 35 tok/s
Prefill · ~1.1K tok/s
490K-token needle · PASS
KV pool · 944K tokens
Context · 512K
DFlash2 k7 + NVFP4 KV on the b12x stack.
Prose is still putting up a fight.
More tuning tonight.
Recipe drops when it stabilizes.
Base lineage: @Tech2Wild's TP4 recipe
Stack by @wrldsuksgo2mars · draft by @inco_ai
@Tech2Wild Good catch. My mistake.
I mixed it up with the 5.2 Int4-Int8Mix. This 5.3 quant is yours: Tech2wild/GLM-5.3-Int4-Int8Mix.
Thanks for catching it 🙏