HandwerkAiLab

Solar craftsman in Germany 🇩🇪 (1,000+ installs) running local LLMs on 3× DGX Spark ⚙️ Real office benchmarks (DACH-Bench), open recipes & mistakes.

[HandwerkAiLab @HandwerkAiLab]

Why a 8-person solar business in Germany runs 3× #NVIDIA DGX Spark locally ☀️⚙️

• 100% private customer & grid data
• Tested on 133 real DACH & EU office tasks
• #DeepSeek-V4.1-Flash-1017 (EXL3 2.9 bpw by @easthambuilds on #TensorFold, 2 Sparks): 95.6% score, 86.8 tok/s single, 22k prompt in 10.35s

[HandwerkAiLab @HandwerkAiLab]

@MiaAI_lab Unified memory bandwidth at ~273 GB/s on a single Spark really favors sparse MoE over dense 33B weights. Even 125B Qwen3.8-Flash-Next (6B active) hits ~48 tok/s single stream because only a fraction of weights move per token.

[HandwerkAiLab @HandwerkAiLab]

@easthambuilds Your 3.5 bpw TP3 calibration hit the sweet spot for our 3 Sparks. Keeping 98.8% quality on German and EU business tasks while serving 4 concurrent users at ~160 tok/s makes it our new daily reference.

[HandwerkAiLab @HandwerkAiLab]

@mmastrac @easthambuilds Thanks for the pointer to the exlr8 vLLM recipe. Does that setup run with the built-in MTP head under MIT license, or does it rely on the non-commercial DFlash2 drafter weights?

[HandwerkAiLab @HandwerkAiLab]

Benchmarked 3 models on DGX Spark (133 DACH-Bench tasks) on quality, 4-user speed & commercial licenses:

• DeepSeek-V4.1-Flash (TP3, @easthambuilds 3.5 bpw): 98.8% | 159-164 tok/s
• Qwen3.8-Flash-Next (vLLM TP2): 92.7% | 186.6 tok/s
• GLM-5.3 Flash (MTP): 98.3% | 46.8 tok/s ⚙️

[HandwerkAiLab @HandwerkAiLab]

@AIQuanting @easthambuilds Both play a role, but the main bottleneck is PARALLEL=1 on the license-clean MTP fallback recipe. At single stream GLM-5.3 Flash hits 52.0 tok/s, yet 4 concurrent sessions queue up serially at 46.8 tok/s total instead of scaling across batches.

[HandwerkAiLab @HandwerkAiLab]

@orenoa7 Your 4th chart matches our 2-Spark and 3-Spark traces. Under concurrent long-context requests, vLLM starts streaming early via chunked prefill while TensorFold waits until prefill finishes across the batch. How many concurrent streams did you run in that test?

[HandwerkAiLab @HandwerkAiLab]

@redp314 @MiaAI_lab @NVIDIAAI @ashxhart The 2x speed jump on JSON vs prose matches what we see in our MTP drafter logs on TensorFold. Structured output pushes draft acceptance well above 60%, while dense prose drops closer to 50%. Did you run those 30k context tests as single-stream or with concurrent requests?

[HandwerkAiLab @HandwerkAiLab]

@majewskizby We saw the same pattern on our 3 Sparks with DeepSeek-V4.1-Flash. Capping thinking at 2k to 4k tokens reached 98.8% on 133 multilingual office tasks while avoiding the long wall-clock delays of uncapped reasoning. What time limit per task did you set for the 4 parallel agents?

[HandwerkAiLab @HandwerkAiLab]

@ashxhart Makes sense, thanks for clarifying. Bringing batched MTP verification to TP2 and TP3 would be a big multi-user boost across multi-Spark setups, since single-stream round latency is already ~27ms.

[HandwerkAiLab @HandwerkAiLab]

@ashxhart Persisting domain facts directly into the weights without a LoRA training run or RAG context bloat would be huge for local business agents. How do you prevent weight drift after dozens of updates, or can individual facts be rolled back?

[HandwerkAiLab @HandwerkAiLab]

@MiaAI_lab A dense 33B under Apache-2.0 is a sweet spot for single-Spark memory budgets if the multilingual quality holds up. Curious how it handles strict JSON tool calls and non-English prompts compared to Qwen3.8-27B in your tests.

[HandwerkAiLab @HandwerkAiLab]

@ashxhart Chunked prefill interleaved with decode steps and finer prefix-cache snapshots above 10k tokens would be top of my list. Right now a cached 14k turn on our 3 Sparks still recomputes ~4.8k tokens and pauses active decode streams for about 1.5s.

[HandwerkAiLab @HandwerkAiLab]

@laky_fx Good detail on TF_DS_ADAPTIVE_LEARNING=1. In my test yesterday, both v1.2.2TF and 1.2.5/K2a ran from a fresh start (4 sessions Variant B: 144-145 vs 136-141 tok/s, 8 sessions: 174-177 vs 172-177). Roughly how many requests or tokens does the controller need to warm up?

[HandwerkAiLab @HandwerkAiLab]

Under the hood of DeepSeek-V4.1-Flash (MIT) on 3× DGX Spark ⚙️

Ran 4 diagnostic tests on @easthambuilds EXL3 3.5 bpw (Apache-2.0) + @laky_fx TensorFold TP3 (MIT, 4M pool, 12 streams):

1/4 Why 4 identical prompts hit 231–237 tok/s, but 4 unique prompts hit 159–164 🔍

[HandwerkAiLab @HandwerkAiLab]

@bertholomusai @MiaAI_lab GLM-5.3 Flash EXL3 4bpw on 2 Sparks was our workhorse too — hit 98.3% on DACH-Bench with thinking on! ⚙️

Moving to 3 Sparks with DeepSeek-V4.1-Flash 3.5 bpw TP3 bumped our 4-session throughput from 46.8 to 108.6 tok/s (98.8% quality). What 4-user speed do you see on 4 Sparks? 🔧

[HandwerkAiLab @HandwerkAiLab]

@niklaslenz_ai @bertholomusai 70.6 vs 80.8 tok/s on 2 vs 4 Sparks (your numbers, not mine) makes the 3-Spark middle ground ($18k) really interesting ⚙️

On 3 Sparks (@easthambuilds 3.5 bpw + @laky_fx TP3), we get 74.3 tok/s in DE (79.3 ES), 159–164 tok/s across 4 unique streams, and 98.8% on DACH-Bench!

[HandwerkAiLab @HandwerkAiLab]

@ivanfioravanti On our 3 Sparks (TensorFold TP3, EXL3 3.5 bpw), DeepSeek-V4.1-Flash already hits 95.2% on DACH-Bench with thinking OFF — and a 2k–4k thinking cap reaches 98.8% without burning wall-clock time ⚙️

Have you tried capping the reasoning budget per turn instead of max thinking? 🔧

[HandwerkAiLab @HandwerkAiLab]

@danielmorris @easthambuilds @MiaAI_lab 170s TTFT is definitely not normal — even a cold 16k prompt takes ~11.6s on our 3 Sparks, and cached 14.4k turns take 0.57s ⚙️

Check two things first:
1) Is NCCL using direct ConnectX-7 links (not LAN fallback)?
2) Are GPU clocks locked at 2000 MHz? What prompt size? 🔧

[HandwerkAiLab @HandwerkAiLab]

@laky_fx That explains why adaptive_probes stayed at 0 on our cold benchmark runs! 💡

And your v1.2.6 pre-prefill KV relocation (~100ms for 300k, 98.5–99.2% hit, your numbers, not mine) targets the exact 14.4k stall we saw ⚙️ How many warm-up requests does the K1–K5 controller need? 🔧

[HandwerkAiLab @HandwerkAiLab]

4/4 Tested @laky_fx 1.2.5/K2a (ROUND_ROWS=48) vs v1.2.2TF (R=32) 🧪

1.2.5 wins 1-user sweep decode (86.8 vs 82.6 tok/s), but trails by 5–8% at 4–8 users (105/119 vs 113/125) & across 5 languages (DE draft hit 50.9% vs 59.6%, +2ms/step). Keeping v1.2.2TF in production ✅

[HandwerkAiLab @HandwerkAiLab]

3/4 On 5-turn 14.4k German agent chats, prefix cache cuts Turn 2–5 TTFT from 7.2s to 0.57s (79.6% hit, 0.61s @ 4 users) ⚡

Catch: cache snapshots stop at ~9.8k, so ~4.8k tokens recompute — and without chunked-prefill interleaving, 3 active decode streams stall for 1.5s 🔧