ata

inferring

[ata @atasrb]

Same weights, same two DGX Sparks, same precision.

GLM 5.3 Flash, one user:
vLLM: 34.6 tok/s
TensorFold: 53.1 tok/s

@MiaAI_lab's recipe on @ashxhart's TensorFold

[ata @atasrb]

Running it locally on two DGX Sparks tonight.

GLM 5.3 Flash EXL3 4-bit on vLLM, Mia's recipe:
34.6 tok/s

next is the tensorfold version of the same recipe @ashxhart @MiaAI_lab

[ata @atasrb]

recipe for two DGX Sparks, one command from the first Spark:
https://t.co/IL5chlrVpf (a clean run of it gave 98 tok/s on JSON and 95 on code)

[ata @atasrb]

Qwen3.8-Flash-Next on two DGX Sparks with NVIDIA's own vLLM cluster command (FP4-Local, TP=2): 94 tok/s on JSON, 90 on code. The same model on one DGX Spark gives 62 and 58.

Decode, tok/s (one user, temperature 0.7, MTP 4, median of 5):
JSON 94.1
code 89.7
rewrite 71.0
explain 68.1
story 50.6

Engine: vLLM 8244d1c (eugr/spark-vllm-b12x nightly-20261015). The server starts in about 10 minutes, with 262k context per request.

Quality: not measured by me. @NVIDIAAI 's model card gives GPQA Diamond 91.5% for this FP4 build against 92.2% for BF16.

[ata @atasrb]

@TheAhmadOsman Dangerously skip permissions is the only way lol

[ata @atasrb]

Yes just one GPU rack that costs around $4 mil. lol

[MTS @MTSlive]

SITUATION DETECTED: Jack Dorsey's company Block's new text-to-speech model Dessa was trained on just one NVIDIA GB200 NVL72 GPU rack.

[ata @atasrb]

@DanielLockyer Which open weights models specifically ?

[ata @atasrb]

https://t.co/oX7CpSqGZ3

[ata @atasrb]

Qwen4 PRs are popping up.

A merged Intel vLLM kernels PR (PR link below) shows Qwen4-27B is dense, with QSA sparse attention and hyper-connections like Qwen3.8-Flash-Next. FP8 on 2 Intel cards, INT4 on 1 to 2.

I'll run it on the DGX Spark when they land.

[ata @atasrb]

@gitghxst isnt every other company doing the same thing? theo videos were probably the most obvious ones

[ata @atasrb]

@swyx @latentspacepod Nice graph, im in

[ata @atasrb]

@VincentVentalon lol, Claude at work

[ata @atasrb]

@atomic_chat_hq @MiniMax_AI ran your q3_k_m on two dgx sparks, llama.cpp rpc layer split: 10.5 tok/s, 8k prompt in 28 s. use --no-kv-unified with more than one slot, or sparse attention falls back to dense. it thinks even with enable_thinking false. tight fit: one spark locked up on a reload.

[ata @atasrb]

@laidick @MiaAI_lab @ashxhart i have not tested that yet. my runs sent one request at a time, didn't have a long session near 400k

[ata @atasrb]

deepseek-v4.1-flash-1017 (763b moe) on two dgx sparks: 103 tok/s on json, 89 on code.

@MiaAI_lab's 2.9 bpw exl3 build (v1) and two-spark recipe, on @ashxhart's tensorfold. one half of the model on each spark, linked over qsfp.

decode, tok/s (median of 5, temperature 0.7, one user, dspark draft on):
json 102.8
code 88.7
explain 72.8
story 57.3

an 8k-token prompt reads in 3.0 s. the weights load in 27 s.

quality, thinking off: gpqa diamond 76.8%, ifeval 88.5%. the best ornith-1.5-122b scores i measured were 74.2 (q4_k_m) and 82.0 (nvfp4) on the same set. gpqa is within noise of ornith's. i could not run the full model, so no reference for it.

[Mia @MiaAI_lab]

Run DeepSeek V4.1 Flash 1017 on 2x DGX Sparks ✨

- 1M total KV pool, 512K context
- 75 tok/s on prose, single stream
- 130 tok/s on prose, 4 concurrent streams
- Prefill 2000-2175 tok/s
- EXL3-EAQv1, 2.9 bpw

The quant in this recipe got about the same calibration as my GLM 5.3 Flash, with some extras. Feedback would be appreciated!

Special thanks to @bertholomusai and all the contributors for the engine work, and to @ashxhart for TensorFold.

Get it here:
https://t.co/GG9H4IkXlt

[ata @atasrb]

@Da7_Tech Qwen4 and Fable 5.5 as one open weights and one frontier. Especially waiting for Qwen4 27B, hope they release it

[ata @atasrb]

@0xSero which engine is this?

[ata @atasrb]

@ashxhart @MiaAI_lab thanks ash, trying minimax-m3.1-flash on the same two sparks next, with llama.cpp (waiting for tensorfold)

[ata @atasrb]

@niklaslenz_ai @sudoingX same model on two sparks here, with @MiaAI_lab's EXL3 2.9 bpw on tensorfold at temperature 0.7: 103 tok/s on json, 89 on code, 57 on a story prompt. different engine and quant, so not a clean comparison.

[ata @atasrb]

flint 40b (2.5b active) on a 2021 macbook pro (m1 max, 64 gb): 76 tok/s, with @Recursive_SI's q4_1 gguf on llama.cpp b11836 and the flags from the model card.

code 75.9, json 75.7, explain 75.4, story 75.2 tok/s (median of 5, temperature 0.7, one user). flint has no draft model, so the speed is the same on every prompt. an 8k-token prompt reads in 11.6 s, llama-server uses 31.6 gb.

speed only, quality not measured.

[Richard Socher @RichardSocher]

Introducing Flint, @Recursive_SI’s first model: a fast local coding model built with our automated research system, the Eureka Machine.

• Best-in-class fast local coding
• Up to 2x faster than comparable models: 125+ tokens/sec on a MacBook Pro, 150+ on RTX Spark
• Built from scratch in under 4 months

Private, unmetered intelligence on your own machine. Open weights available now on @huggingface, run it locally on your favorite framework or try it for free on @opencode or @pidotdev.

[ata @atasrb]

both ran their defaults: flare with 5 drafts and adaptive verification, vllm with nvidia's flags from the model card. the vllm setup, pinned: https://t.co/NVuGqm4Mdk

[ata @atasrb]

nvidia's new flare vs vllm on the same dgx spark, qwen3.8-flash-next fp4, mtp 5 on both.

on real prompts they're close (vllm vs flare, tok/s):
json 65.2 vs 60.0
code 58.0 vs 56.3
rewrite 44.0 vs 50.9
explain 43.7 vs 42.6

on random text, where the draft rarely hits, flare is 1.7x faster per token. on real text the draft hits often and the gap closes. flare restarts in under a minute, vllm in about 5 once its caches are warm.

temperature 0.7, median of 5, one user, thinking off. flare is a research prototype by @NVIDIAAI and @YourLocalAILab.