Artem Mats

Coding with Local AI and LLMs: 2 x DGX Spark cluster experiments.

[Artem Mats @Reederey]

If you run GLM-5.3-Flash (320B MoE, EXL3) on two DGX Sparks.

This release is the CUDA side complete rewrite. It is no longer stock exllamav3.

What you actually get:

• The fat-expert kernel is ~40% faster — same answers, less stalling on the GPU
• Prefill on the box stays ~1k tok/s.
• Decode and acceptance are unchanged.

More in v1.4.0
https://t.co/rpofFZrDV0

[Artem Mats @Reederey]

Meet Claude Fable 5.1.

It is noticeably ahead of both Fable 5 and Opus 5 in benchmarks. Thanks to a fourfold reduction in the price of cached tokens (caching are now $0.25 per million tokens), it is on average 25% cheaper than Fable 5. Moreover, the volume of reasoning tokens appears to have increased. Rollout of the model is already underway.

https://t.co/kf6blzIci8

[Artem Mats @Reederey]

There is no way that MOE 284B@13B DeepSeek v4 Flash can be better than MOE 320B@18B GLM 5.3 Flash. They are both very aligned with a tool usage (superior RL) but lacking pure knowledge of bigger 2T - 3T models. Scaling low!

GLM 5.3 Flash just has more knowledge.

[Artem Mats @Reederey]

@BrandonMusicKy Thanks a lot for the excellent EXL3 weights!

[Artem Mats @Reederey]

Huge improvements!!!

GLM-5.3-Flash deployment kit for two DGX Sparks — v1.3.2 is out.

⚡ Cached follow-ups skip the queue: a warm 50k prompt behind a running generation, 45.8 s → 2.6 s (on by default)

🚀 New CUDA kernel for the MoE expert weights that eat most of prefill: long reads +9–16% (178k tokens: 895 → 1,040 tok/s), decode unchanged (opt-in).

⏱️ Cold prompts held by the scheduler get first token in ~2.5 s instead of minutes

Release:

https://t.co/gpsIw91vpx

#vLLM #DGXSpark #GB10 #GLM

[Artem Mats @Reederey]

Yes — hard aging cap. The warm bypass isn’t a priority queue, just a size check (uncached remainder ≤ one 3584-token page → no hold). Cold prompts are held only while a peer decodes, and after 1.5 s they proceed anyway at 512 tok/step until their last page. So a chatty session can slow a cold read, never starve it. Both knobs are .env: MAX_WAIT_MS, LATE_CAP. Big room for further improvement on GB10.

[Artem Mats @Reederey]

Best GLM-5.3 flash nvfp4 weights got updated.

[LibertAI 🕶️ @Libertai_DAI]

We fixed a bunch of bugs and improved quality on our GLM-5.3-Flash-NVFP4 and GLM-5.3-NVFP4 model quants.
If you downloaded them previously, redownload the changed parts.
https://t.co/ozQAB349JU

[Artem Mats @Reederey]

DeepSeek V4 Flash model with Vision got open weights.

https://t.co/tYgQqwjsRG

[Artem Mats @Reederey]

Update to the GLM-5.3-Flash deployment kit for two DGX Sparks

Before vs after the rebase — same 4-agent coding workflow, same two DGX Sparks, GLM-5.3-Flash:

The image behind :

vLLM's day-0 GLM-5.3 preview base + EXL3 for GB10 + the DFlash2 drafter + the KV slot-share and cache fixes, cut over to production the same day it was field-tested.

Release v1.3.0:
https://t.co/Pvtc1rMNW1

[Artem Mats @Reederey]

@xz_keg no, it’s the memory-bandwidth wall of this hardware for now.

[Artem Mats @Reederey]

GLM-5.3-Flash (320B MoE) on 2× NVIDIA DGX Spark desktops, with a full 1M-token context window.

Real measured numbers, not vibes:
⚡ ~70 tok/s on structured output, ~30 tok/s on prose
📥 ~940 tok/s prompt ingestion
🎯 100% speculative-decoding acceptance (every draft token lands)
🔁 Cached re-reads of a 110k-token session: 3.5 seconds instead of 2+ minutes

Everything is pinned — exact image digest, exact model revision, byte-verified weights — so what you build is what I measured. The repo ships the same acceptance, serving, and benchmark test batteries I gate production with, so you can verify your own numbers in minutes.

I got a 320B-parameter frontier model running at home — and turned the whole setup into a kit anyone can reproduce. 🖥️🖥️

Repo: https://t.co/NIROvV2Kq2

#DGXSpark #LocalAI #vLLM #GB10

[Artem Mats @Reederey]

@Belcebuu1 @MiaAI_lab It is a completely rebased deployment image, as all perf parameters are different from her. She is in the credits, as she first used EXL3. I spent quite a lot of time researching the performance of EXL3 on GB10 before pushing different then NVFP4.

[Artem Mats @Reederey]

GLM-5.3-Flash kit for 2× DGX Spark

1M-token context window, 70.4 tok/s structured / 29.5 tok/s prose decode, ~940 tok/s cold prefill

🔓 Fixed two bugs that broke caching for AI agents. Before, long coding sessions caused the cache to deplete, requiring re-reading entire conversations. Improving session replay to 95-98% and reducing wait times from 5 minutes to about 6 seconds.

⏱️ No more waiting in line behind large prompts. A quick tool call used to wait 4+ minutes for a 240k-token document load. Now it answers in 8 seconds, reducing latency by 97% for a 5% prefill cost.

✅ The kit is now reproduce-tested - release v1.0.2, which passed the full test battery: tools, vision, 1M-token context, and byte-identical memory pool.

https://t.co/NIROvV2Kq2

#DGXSpark #LocalAI #vLLM #GB10

[Artem Mats @Reederey]

Fair catch — the denominator is drafted tokens per verify step: 7 drafted, 7 accepted, on one specific benchmark — a count-1-to-200 structured test at temperature 0. Maximally predictable output, so of course the draft nails it. On regular prose the same drafter accepts only ~30%.

And you can't serve the draft alone: the big model still verifies every token and silently fixes the misses — 100% just means that on that easy task there were none.

[Artem Mats @Reederey]

Same model (GLM-5.3-Flash 320B MoE / 18B active), same nodes, same DFlash2 drafter, both on your 2×Spark cluster:

The single most important fact from the research: GB10 (SM121) is missing the `cvt.rn.satfinite.e2m1x2.f32` PTX instruction — the hardware FP4↔FP32 converter that SM120 (RTX 5090, B200) and SM100 have.
Consequences, all confirmed across multiple independent sources (Avarok's GB10 bring-up writeup, the "NVFP4 is a trap on GB10" benchmark, NVIDIA forums, vLLM issues)

So don't expect NVFP4 will be better than EXL3 on GB10/DGX Sparks.

[Artem Mats @Reederey]

For GLM-5.3 Fash 2xDGX 's use case — single-stream, long-context, agentic, cache-heavy — EXL3 is measurably the better UX: quicker felt decode speed over NVFP4 baseline, better quality per bit, 1M window, replays in seconds. The prefill point is right and worth respecting (it's why cold 133k prompts cost ~140s), but "3× prefill" only dominates the experience if your workload is cold prompts or batch decode.

[Artem Mats @Reederey]

Published third-party numbers on the same hardware suggest a large single-stream win — but they are reported with MTP and without an acceptance rate, so treat them as a hypothesis, not a target.

The model's native window is 1,048,576, but after ~90.5 GiB of weights per node the KV pool is the binding constraint. 131K at 6.2x concurrency is the useful operating point; raise it and re-read the pool size the engine prints at startup.

[Artem Mats @Reederey]

Kimi K3 is an awesome model for CUDA kernel work. As far as I know, some frontier labs are running it on research-allocated compute. The main downside is that Kimi's subscriptions are pretty limited. You only start getting a semi useful number of tokens for real work at the +$100 tier.

[Artem Mats @Reederey]

Awesome!!

[Zixuan Li @ZixuanLi_]

We’ve seen many comments and requests from our partners regarding framework compatibility. To ensure everything is properly supported and works reliably at launch, we’ve slightly adjusted the open-weight release timeline.

Thank you all for your patience and support.

[Artem Mats @Reederey]

@Tech2Wild Getting following... Measured on the pair of DGX Sparks: KV pool 813K–834K tokens, 6.2x concurrency at a full 131K ctx request, TRUE 13.6 tok/s single stream, 26.7 tok/s across 8.

https://t.co/EWPkawsdHt