Kai

[Cruz @ViC305]

DeepSeek-V4-Flash-Vision EXL3 MixedK just hit 85.04 aggregate tok/s across 16 concurrent streams on ONE DGX Spark GB10. 🚀

That is 3.05× the C1 aggregate throughput, with a 117.9 tok/s burst peak, native 3-layer MTP, clean CUDA graphs and the full ~95.4 GB model on one box. ⚡🔥

This one took a few different problems to solve.

𝗧𝗛𝗘 𝗖𝗢𝗡𝗖𝗨𝗥𝗥𝗘𝗡𝗖𝗬 𝗦𝗪𝗘𝗘𝗣

Same physical GB10. Same model. Increasing client streams:

C1
27.85 tok/s aggregate
32.22 tok/s decode
655 ms TTFT

C2
42.66 tok/s aggregate
+53.2%
29.49 tok/s decode
1.12 s TTFT

C4
42.77 tok/s aggregate
+53.6%
12.46 tok/s decode
1.67 s TTFT

C8
59.05 tok/s aggregate
+112.0%
9.41 tok/s decode
3.12 s TTFT

C16
85.04 tok/s aggregate
+205.3%
7.20 tok/s decode
5.43 s TTFT

Peak observed burst at C16:

117.9 tok/s

Important distinction:

85.04 tok/s is aggregate server throughput.

It is NOT 85 tok/s for each of 16 users.

Per-request latency increases as the box gets busier, while total useful work keeps scaling.

That is exactly the behavior I wanted to measure.

🎯 𝗧𝗛𝗘 𝗠𝗧𝗣 𝗣𝗔𝗧𝗛 𝗪𝗔𝗦 𝗧𝗛𝗘 𝗥𝗘𝗔𝗟 𝗕𝗥𝗘𝗔𝗞𝗧𝗛𝗥𝗢𝗨𝗚𝗛

My earlier DeepSeek numbers looked terrible for a reason.

If the runtime does not correctly bind this pack’s:

mtp.0
mtp.1
mtp.2

weights into DeepSeek’s native draft path, the drafter can run with effectively 0% useful acceptance.

You pay for speculative decoding without getting the speculation.

Once I routed the pack through the proper native DeepSeek MTP bindings:

C1-C4 acceptance:
61.8% to 64.8%

Mean accepted length:
2.85 to 2.94 tokens per target forward

Representative positional acceptance:

Position 1: 82.3%
Position 2: 62.4%
Position 3: 49.6%

At C8-C16, acceptance falls to ~35.8% with ~2.07 accepted tokens/forward as contention rises, but aggregate throughput keeps climbing.

That distinction matters.

More concurrency is not magically making every request faster.

It is making ONE Spark process substantially more work.

🧠 𝗧𝗛𝗘 𝗞𝗩 𝗣𝗢𝗢𝗟 𝗜𝗦𝗡’𝗧 𝗧𝗛𝗘 𝗪𝗔𝗟𝗟

FP8 KV.

At C16:

KV footprint: 5.2%

17,000+ blocks still free

No KV spill to NVMe

So this sweep is nowhere near exhausting the cache.

The bottleneck is execution efficiency and scheduling, not simply “we ran out of context memory.”

⚡ 𝗖𝗨𝗗𝗔 𝗚𝗥𝗔𝗣𝗛𝗦 𝗙𝗜𝗡𝗔𝗟𝗟𝗬 𝗖𝗔𝗣𝗧𝗨𝗥𝗘 𝗖𝗟𝗘𝗔𝗡𝗟𝗬

I wanted static decode graphs for the actual concurrency ladder:

[1, 2, 4, 8, 16]

But a few innocent-looking operations inside the FlashInfer / sparse-attention path were forcing device-to-host synchronization during capture.

Things like:

.item()

and:

int(indices.min())

That breaks static CUDA graph capture.

I replaced the host-side check with a GPU-side clamp:

indices = indices.clamp_min(0)

Result:

Full graph capture completed in ~2.0 seconds without the CPU synchronization.

Small code change.

Huge difference to the serving path.

💾 𝗔 𝟵𝟱 𝗚𝗕 𝗠𝗢𝗗𝗘𝗟 𝗦𝗛𝗢𝗨𝗟𝗗𝗡’𝗧 𝗧𝗔𝗞𝗘 𝟭𝟮 𝗠𝗜𝗡𝗨𝗧𝗘𝗦 𝗧𝗢 𝗥𝗘𝗦𝗧𝗔𝗥𝗧

The model is 48 safetensors shards and ~95.4 GB.

A true cold NVMe load is still expensive.

But when Linux page cache is intentionally retained:

DROP_PAGE_CACHE=0

a warm restart streams the model at roughly:

~4.1 GiB/s

and reaches serving health in:

24.9 seconds

instead of paying the ~10-12 minute cold-read path again.

That 24.9 seconds is a WARM page-cache restart, not a cold-disk benchmark.

For something this large, restart behavior is part of production performance too.

🛠️ 𝗪𝗛𝗔𝗧 𝘃𝗹𝗹𝗺-𝗲𝘅𝗹𝟯 𝗜𝗦 𝗗𝗢𝗜𝗡𝗚

The native GB10 work includes:

→ in-register MCG Trellis extraction
→ active-expert batched GEMV
→ 4-phase cooperative MoE decode
→ tiled/chunked prefill GEMM
→ native sm_120 / sm_121 extension
→ CUDA graph-safe execution paths

The point is simple:

Keep EXL3 compressed.

Keep the routed experts packed.

Then make the execution path understand the hardware instead of reconstructing our way back toward the thing we were trying to avoid.

And the stack is now doing:

~95.4 GB DeepSeek-V4-Flash-Vision
+
EXL3 MixedK
+
Vision
+
native 3-layer MTP
+
16 concurrent streams
+
ONE DGX Spark

→ 85.04 tok/s sustained aggregate
→ 117.9 tok/s burst peak

That is a much more interesting result to me than simply saying:

“the model fits.” 😅

Huge credit to @turboderp for ExLlamaV3, EXL3 and the MCG work, and to @MiaAI-Lab / @plotarmordev for the Spark EXL3 serving work this project builds on. 🙏

📦 𝗘𝗩𝗘𝗥𝗬𝗧𝗛𝗜𝗡𝗚 𝗜𝗦 𝗢𝗣𝗘𝗡

vllm-exl3:
https://t.co/wvvYJpI524

One-Spark DeepSeek recipe:
https://t.co/SByNKWsKTF

Model:
https://t.co/v3KCwUpX0f

Ablit Version:

https://t.co/v3KCwUpX0f

[Mia @MiaAI_lab]

Who wants Qwen3.8 Flash on a single DGX Spark?

[Mia @MiaAI_lab]

This is for you single DGX Spark owners 💫

You can now run Qwen3.8 Flash NVFP4 on one DGX Spark with great performance!

- Up to 1M context, 1,431,164 KV cache
- Full image & video support
- 37 decode tok/s on prose single stream
- Up to 86 tok/s on prose 4 concurrent streams
- 1500-2000 tok/s prefill on any size.
- Stress tested with 400k prefill - passed.

IMO this is the BEST model to run now on a single dgx spark. It's better than Qwen3.8-27B, and also better than the 1x dgx spark version of DeepSeek v4 Flash.

Get it here:
https://t.co/LG0I9PJKxI

[Viv @Vtrivedy10]

from Astra blog - Codex compaction is already great, never think about it, have multi-week threads

turns out avoiding destructive (unrecoverable) summarization actions + letting a very smart model search over previous context…just works

scaling + improving general search works, bitter lesson is inescapable

[Mustafa @oprydai]

the most underrated skill is learning how to decompose problems.

when an expert looks at a difficult problem, they rarely solve the whole thing at once. they reduce it until the unknown becomes a collection of smaller knowns.

this is the same pattern everywhere.

in software, you break a system into modules. in robotics, you separate perception, estimation, planning and control. in mathematics, you transform an unfamiliar problem into structures you already know how to manipulate. in engineering, you isolate variables until cause and effect become visible.

the general process is simple:

• problem → constraints → variables → relationships → subproblems → solution

the important step is identifying what is actually unknown.

suppose a robot keeps missing its target position. “the robot is inaccurate” is almost useless as a problem statement.

decompose it.

is the commanded position wrong? is localization drifting? is the kinematic model inaccurate? is there actuator backlash? is the controller poorly tuned? is latency creating phase lag? is the mechanical structure flexing under load?

one vague problem just became seven testable hypotheses.

now you can measure them.

this is why decomposition is so powerful. it converts confusion into experiments.

when you feel stuck, resist the instinct to immediately search for an answer.

reduce the problem first.

• define the desired state.
• define the observed state.
• measure the difference.
• is the variables capable of producing that difference.
• hold variables constant where possible.
• change one thing.
• observe.
• update your model.

this is basically the scientific method compressed into a practical problem solving loop.

and the deeper you become in mathematics, physics, programming and engineering, the better your decomposition becomes because you acquire more primitive building blocks.

experts don’t necessarily encounter easier problems.
they have a larger library of ways to break difficult problems into familiar ones.
that is one of the real purposes of learning technical subjects.
you are accumulating primitives for thought.

[Paolo Rosson @redp314]

Just got Qwen3.8-27B to 360 tok/s on ONE DGX Spark, sixteen coding agents at once! 387 with a 32-seat profile.

Same recipe as my last post, booted for concurrency: each stream its own 2k-token code prefix, 512 out, temperature 0, average of 2 runs.

82 tok/s alone → 199 at 4 → 276 at 8 → 360 at 16. First token under 0.7 s the whole way, drafter still accepting 7-9 tokens a pass.

The catch: 16 seats cost ~24k of context, and 32 seats need a bigger memory share or the KV pool collapses. Pick your profile, all three boots are in the repo.

[FHILY👑 @Oluwaphilemon1]

64 users. 2 DGX Sparks. Qwen3.8-Flash-Next.

This is what happens when you stop testing local AI with a single prompt and actually stress the hardware.

The setup:

→ 2× DGX Spark
→ Qwen3.8-Flash-Next NVFP4
→ 64 concurrent users
→ Independent prompts
→ Independent KV caches
→ 32,768 output tokens

Total generation time:

78.82 seconds

That’s an aggregate throughput of:

415.7 tok/s

And the more interesting part wasn’t just the raw throughput.

The usable-context test passed 64K tokens per user at C16, with zero cross-user leakage.

Each user maintained an independent prompt and KV cache while the system handled the concurrency load.

That’s a serious local inference stress test.

But there’s an important distinction:

This is testing the limits of the hardware and serving stack, not presenting a production deployment recipe.

Still, it’s a pretty ridiculous demonstration of where local AI infrastructure is heading.

Two compact AI systems handling dozens of independent sessions while pushing hundreds of tokens per second in aggregate.

The “local AI is only useful for one person at a time” argument is getting harder to make.

2 DGX Sparks.
64 users.
415.7 tok/s aggregate.

And they’re still testing where the ceiling actually is.

[Prukalpa ✨ @prukalpa]

I would keep Skills outside the harness and as interoperable in the company context layer — but otherwise this diagram from @iandmacomber is a great visual of the post-AI Data Stack.

The success of this is a function of how the company context <> traces <> feedback loops interact to create value for end users.

[Robin Alexander @robinalexander_]

1) Union lässt Habeck abblitzen, als er ihr vorschlägt, gemeinsam Sondervermögen für Infrastruktur und Wirtschaft zu beschließen.
2) Union führt Wahlkampf gegen "schlechtesten Wirtschaftsminister der Geschichte".
3) Union gewinnt die Wahl.
4) Union verabredet ein Sondervermögen für Infrastruktur und Wirtschaft.
5) Union beschimpft einen Tag später Grüne als "Ramschware“, Habeck könne ab jetzt "Kinderbücher von der Oppositionsbank schreiben".
6) Union braucht nächste Woche die Stimmen der Grünen für das Sondervermögen. 🤷‍♂️

[Das Grundgesetz ist ein Schatz! @AgainstHate2024]

Lieber Wayne Carpendale. Ich feiere Sie/Dich für dieses schonungslose Video bezüglich AfDlern.

Danke für diese epische Vorführung 👍

[Ben 🇩🇪 🇺🇦 🇮🇱🇺🇸🇪🇺🌍 @twidderix]

Habeck hat recht. Der Zug ist nicht mehr aufzuhalten. Auch nicht von dummen Politikern oder unfähigen, visionsbefreiten Managern die in der Vergangenheit leben! Einfach weil es gut und günstig ist! 👍 😊

[clem 🤗 @ClementDelangue]

someone trained microduck to do somersaults 😭

[Jeremy Nguyen ✍🏼 🚢 @JeremyNguyenPhD]

Codex usage tip:

Ask GPT-5.6 Sol to only run the minimum number of tests necessary, and explain to Sol why.

It's made a drastic difference for me: I'm still above 90% usage left

[Eyal Toledano @EyalToledano]

Another Qwen3.8-Flash-Next breakthrough

I stored 60% of experts on disk which stream to memory on-demand. Similar technique as n-gram streaming

Just eval'd with FULL experts (no prune) running on just 37gb of memory and decoding at 40 tok/s on my M4 Max

Hold on to your socks!

[Eyal Toledano @EyalToledano]

From the @HamsterResearch lab:

Introducing Qwen3.8-Flash-Next-REAP-288-MLX-4bit, a 180B-class model running on just 39gb of memory

- MLX-native 4-bit 60% smaller than stock q4
- Pruned 512→288 experts via REAP
- 91.5% HumanEval (vs 93.9% stock)

@huggingface links below ↓

[tintinweb @nicht_tintin]

Dynamic Workflows.

Now in @tintinweb/pi-subagents for @pidotdev ♥️. @claudeai Code-compatible. Highly underrated.

Thank you for your attention to the matter of scriptable agent orchestration 🦅

`pi install npm:@tintinweb/pi-subagents`

https://t.co/P2b7Ml9BIY

[Thomas Wolf @Thom_Wolf]

We have a huge news to share today!

Today we are unveiling the first truly accessible RL robot - welcome Microduck

A 25 cm tiny open-source biped with 15 actuators and packed with sensors (camera, speaker, LiDAR, NFC, bluetooth, wifi, etc) that you train yourself with reinforcement learning.

It's also playable out of the box with more than half a dozen fun and playful pre-trained policies to have it walk, sit, crouch, roller-skate, pick up objects with its articulated beak, and recover on its own.

And all for less than $400.

See all the details, play with the simulator and order it at: https://t.co/n1Btgs6vKw

(video with sound on 🔊)

[clem 🤗 @ClementDelangue]

BIG ANNOUNCEMENT FROM HUGGING FACE TODAY:

We're unveiling Microduck 🐥🤖

It's a tiny $399 open-source robot you can teach new tricks with reinforcement learning. It can walk, pick things up, get back up when it falls, and even roller-skate.

Welcome to the era of open-source affordable robots to democratize physical AI and world models!

🤗🤗🤗

[Pi @pidotdev]

Lead Engineer, Scott Fryxell, used Pi to move most of his client work to DeepSeek, dipping into Fable only when necessary.

By primarily using ‘commodity models’ he only needs two $20 plans for his total usage.

Read how Pi became “the most important piece” of Scott’s rig below

[atomic.chat @atomic_chat_hq]

Running 1-bit Qwen 3.8 Flash Next (79GB) on a MacBook Pro M5 Max 64GB at 30 tok/s 🤯

It ran a 8-minute agent loop with 6 web searches, 3 Python runs and 5 sourced tables.

Run local models via Atomic Chat!

GGUF: https://t.co/IrKMSfMJDt