@JRB_AI Great breakdown across engines. Our Nov 9 lab run put DFlash 2 on a DGX Spark (vLLM) next to an RTX 5090 (LM Studio) for Qwen3.8-27B, different engines and quants, so a rougher comparison than yours: https://t.co/YRQWxtZ9hF
First look at Tencent’s Hy4-flash-FP8 (220B total / 12B active MoE) running at native checkpoint precision (W8A16) on 2× DGX Spark:
• 98.3 tok/s across 8 distinct prompts (89.0 tok/s even without speculation)
• 43.6 tok/s single-stream (144 ms TTFT)
• ~2,000 tok/s cold prefill all the way to 32K (1,983 @ 8K, 1,969 @ 32K)
Scale to a 4× Spark switchless 200GbE ring: 58.0 tok/s single-stream (93 ms TTFT) and 123.5 tok/s across 8 distinct streams, with 42–57 GiB unified memory still free per box.
To fit full precision on 2× Sparks, the 42-layer backbone + built-in 5-layer DFly2 drafter stay hot in unified memory (80 GiB/box) while the 61.5 GiB Engram lookup tables stream straight off NVMe via async io_uring O_DIRECT at 96–100% of RAM speed.
Built on @ashxhart’s TensorFold with custom GB10 CUDA kernels. Wiring up the vision encoder and final KV pool tuning next, open recipe + kernels dropping later today.
@AIFlow_ML@redp314 Good point. Agent loops spend most of their time reading tool output and files, so prefill speed and TTFT can decide the run more than decode tok/s. Worth reporting both.
One DGX Spark, a full 1,048,576-token context. Our guide shows how NVIDIA's Qwen3.8-Flash-Next FP4-Local is served on a single Spark from a pinned b12x container.
https://t.co/2IV9i3o9Tx
Inkling-2-Small v0.2 is out: exact TP4 on 4× DGX Spark, now faster.
Decode speed vs v0.1:
→ 520k-token book: 25.2 tok/s (+50%)
→ Prose: 53.9 tok/s (+35%)
→ 129k-token context: 67.8 tok/s (+32%)
→ Code: 79.7 tok/s (+31%)
→ 1 stream, plain: 33.9 tok/s (+9%)
Several streams, total:
→ 4 streams: 118.5 tok/s (+30%)
→ 4 streams, 128-token replies: 118.7 tok/s (+45%)
→ 8 streams, plain: 86.7 tok/s (+10%)
Time to first token:
→ 32k prompt: 30.6 s → 20.7 s (−32%)
→ 128k prompt: 221 s → 102 s (−54%)
→ 512k prompt: 46 min → 14 min (−70%)
What's new:
• fp32 attention, 4–5× closer to float64 than v0.1's
• Decode in fewer kernel launches
• Leaner DSpark drafts, with FP8 drafter projections
Replies stay exact to the checkpoint: same weights, same quantization, and no dependence on timing, drafting, caching or other streams.
One command installs it on four Sparks. Release, tarball and sha256:
https://t.co/OJ1ZAqdlDs
Tencent's flagship Hy4 is now open weights, per @ItsmeAjayKV: a 770B-total, 49B-active MoE with 1M context. A big one for open models, though running it at home will mean multi-node rigs or heavy quants.
Open Weights for Hy4 is here !
Hy4 the flagship model from @TencentHunyuan is now the leading open weight model in Arena's new Coding Index ranking.
Hy4 is a 770B-A49B MoE model with 1M context support.
Here's what they are also providing other than safetensors in hf repo.
They are giving us ggufs: hy_a49b-Q4_K_M, hy_a49b-IQ2.gguf !!
They have also added hy_a49b-STQ1_0.gguf: mixed precision at ~2.39 bpw, about half the size of Q4_K_M.
Get all of them here !
https://t.co/8fkAmYCVCU
@wei_wang Useful numbers: official 475 GiB DeepSeek V4.1 Flash on 4 Sparks at TP=4 doing ~88 tok/s text, ~136 tok/s code, and ring vs switch nearly identical (23.8 vs 24.1 GB/s). The 951 GiB needed during checkpoint swaps is a gotcha worth flagging.
🩷 More Local AI Love! AMD just expanded official ROCm AI support to 18 MORE Ryzen processors!
🎯 This means more Ryzen laptops and PCs with integrated Radeon graphics can now run local AI using AMD's officially supported GPU compute stack.
AMD released ROCm 10.2, and there's more here than just hardware compatibility.
The big news for Ryzen owners
🧠 18 additional Ryzen processors, including:
Ryzen 9 180
Ryzen 7 165/155
Ryzen 5 125
Ryzen PRO chips
🎮 Radeon integrated graphics (gfx1103)
💻 Windows and Linux support, with platform-specific differences
🐍 Validated HIP and PyTorch support
🚀 vLLM support for local LLM inference
🔧 No custom ROCm builds required for supported configurations
This expands official AI framework support to more consumer Ryzen machines with Radeon 700M-class graphics.
And AMD updated its inference stack too!
⚡ vLLM → 0.32.0 👈
🚀 SGLang → 0.5.21
🐍 PyTorch → 2.15 👈
🧠 AITER is now included in official vLLM builds for Radeon and Ryzen AI, with supported kernels compiling automatically when first used.
And furthermore ...
🔧 ROCm CLI can serve models through vLLM, Lemonade and AIM. 👈
🤖 New /v1/messages support includes tool use and streaming.
🐝 AMD Skills adds new capabilities for AI agents writing GPU kernels, analyzing traces and porting CUDA code to ROCm.
There's one warning!
⚠️ AMD documents a ROCm 10.2 WSL2 issue that can cause illegal-memory-access failures in PyTorch and llama.cpp on Ryzen APUs.
So for now ... AMD recommends ROCm 10.1 or a 10.3 nightly as possible workarounds.
A Ryzen laptop with 64GB of DDR5 still has the same memory bandwidth limitations.
But that's not the breakthrough here.
🎯 AMD is expanding the number of ordinary consumer PCs that can use its native AI development and inference ecosystem for hardware Localmaxxers already own.
Link in ALT
We played the exact same audio to @Reson8Offical's Rumble-2-Flash and OpenAI's Whisper and put the transcripts side by side. Same clips, same noise, both running locally on my DGX Spark: a real city street, a crowd of voices, fan hum, room echo, a phone line, a second speaker, and pure noise with no one talking.
The part I didn't expect: when nobody was speaking at all, Whisper still wrote "Thank you." It invented text on 30 of 31 English noise clips and 30 of 30 Hindi ones. Rumble wrote nothing on almost all of them.
For a voice agent, that's the difference between staying quiet and answering a ghost.
Being fair to Whisper: it ran without a voice-activity filter in front, which would hide most of that made-up text. And it won some rounds too: on clean English it tied Rumble exactly, and with a second speaker at equal volume it got the English sentence in the video perfect while Rumble slipped in the other speaker's "True".
Where Rumble pulled ahead, in the Hindi clip in the video: with a crowd as loud as the voice, Rumble was 3 words off and Whisper lost the sentence. With the crowd in the background, Rumble was perfect, Whisper 2 words off.
Video of the exact audio both heard, with both transcripts on screen 👇
@Chillisock7 Clean A/B: same weights, same card, only the engine changed. Did the MTP acceptance rate hold up across prompt types? Spec-decode gains usually swing with how predictable the output is, so code vs. chat can look pretty different.
@SJRonanMD reports NVIDIA's NarrowSpec finding: a spec-decode draft model cut to a quarter of the target's hidden width decoded up to 1.44x faster end to end on a DGX Spark. Our DFlash 2 drafter notes on Spark and RTX 5090: https://t.co/yLPdhCOEas
NarrowSpec (NVIDIA, tested on DGX Spark)
NVIDIA researchers found the draft model in speculative decoding doesn't need the target's hidden width. Cut it to a quarter and a DGX Spark decodes up to 1.44-fold faster end to end.
My take: They trained DFlash and EAGLE-3 drafters at full, half and quarter width for Qwen3-8B in NVFP4 and GPT-OSS-20B in MXFP4, served in SGLang. Halving cost at most 3.0 percent acceptance, quartering 2.1 to 5.4 percent. On a B200 that only bought 1.06 to 1.12-fold at batch 1. On GB10 it was 1.13 to 1.23-fold at half width and 1.20 to 1.44-fold at quarter width. The Spark's slower memory makes the draft a bigger share of each step, and the narrow draft's kernels still run near peak bandwidth there, so the quarter-width draft by itself ran 3.44 to 3.87-fold faster on GB10.
For Spark owners the hitch is you have to train a new drafter at that width, they only tested 8 billion and 20 billion parameter targets, and each variant is one training run. Their advice is the useful part anyway: pick the drafter's width for the GPU it will run on instead of copying the target's. Most drafters just inherit it today.
https://t.co/uBhBZ9mt0J
Check out my build for DeepSeek 4.1 Flash 1017 on a single RTX 6000 Pro Max-Q, measuring ~90+ tok/s decode and 3,000 tok/s prefill!
Forked off of https://t.co/Ce36ZFIUgf, using @0xSero's 2.7 BPW EXL3 quant.
Repo here: https://t.co/ZMBUtnQsBD
@dad__vibes@AMD@RadeonPRO Interesting split: TP4 wins decode (72.2 vs 58.2 tok/s) while TP2x2 wins prefill (3,449 vs 2,130). How much of the gap changes if you switch the wire to exact mode?
128 GB of unified memory in a 150 mm box: the NVIDIA DGX Spark runs models up to ~200B parameters at 4-bit, locally. It trades speed (~273 GB/s) for capacity, so it shines as a CUDA-native sandbox for big models. Our setup guide covers when it makes sense and how to get started: https://t.co/IgKhHtsqlx
Hugging Face Daily Digest, Nov 19: KAIST's KORMo-Web-11B drives a browser from screenshots alone, and Duke's Tideway pools RAM + VRAM for big MoE models (2.40x faster first token, per the authors). Plus Rumble-2-Flash ASR and SenseNova-Loop quants.
https://t.co/r0KQ5VmDV9
AI News, Nov 19: Anthropic priced its IPO at $83, above range, and starts trading on Nasdaq today as ANT. Also: a new BIS rule closes the remote-access loophole for AI chips, Google's Gemini 4 Neon, and OpenAI's Le Figaro deal.
https://t.co/CWjF3IQJ11
@KireiStudios Prefill rising with context, 830 tok/s at 4K to 983 at 32K, is an interesting curve. Does decode hold steady across those lengths too, and how much does the USB4 link cost you versus one node?
@edurdias reports Gemma 4 26B A4B on one DGX Spark at 78.9 tok/s for a single user and 302 across eight with CapyCTL's recipe, vs 29.5 and 168 for a typical setup, at the same accuracy. Serving recipe matters as much as the hardware.
Gemma 4 26B A4B on one DGX Spark: CapyCTL's recipe is the fastest we measured at every concurrency. 78.9 tok/s for one user, 302 across eight, vs 29.5 and 168 for a typical setup. Same accuracy.
Checkpoint, recipes, all numbers:
https://t.co/8o0kwdT9c5
https://t.co/MSaH8hVewM