Mark Argyelan

AI Engineer - Agentic AI Architect | Builder-Strategist

[Mark Argyelan @markargyelan_]

Woke up. Vera Rubin rack in my living room. No delivery notice. No invoice. NVIDIA, what did you do.

[Mark Argyelan @markargyelan_]

@ayoub_el77927 You are welcome!
keep grindig! 😉👌

[Mark Argyelan @markargyelan_]

@antoine_os @grok translate this please 😂

[Mark Argyelan @markargyelan_]

Woke up. Vera Rubin rack in my living room. No delivery notice. No invoice. NVIDIA, what did you do.

[Mark Argyelan @markargyelan_]

@MichaelGannotti Respect!
Yeah, I learned this things the hard way.
But maybe it’s better.. 😉

[Mark Argyelan @markargyelan_]

honest limits:
- if you are fine with the WHOLE model in 4bit, a single 5090 is faster (110-180 tok/s, see the recent r/LocalLLM thread). different quality class though. here 48 of 65 layers stay full 8bit

- speed depends on your text. mtp speculation loves code and structured stuff, creative prose drops to around 40 percent acceptance and lands near 35-40 tok/s

- community data says nvfp4 requantizing hurts quality. our evals came back clean anyway, humaneval 95.7 and gsm8k 96.5, probably because 74 percent of the weights stay 8bit

[Mark Argyelan @markargyelan_]

experiment: can i get the 5090s speed and the
DGX Sparks memory at the same time, on one model?

the 5090 is fast but only 32gb. the 8bit qwen3.8-27b barely fits and context spills after about 30k tokens. the spark has 128gb but the memory is like 4x slower.

so we split one model over both boxes with llama.cpp rpc, normal 10g cable. the trick that makes it fast: every layer gets the quant format of the chip it lives on. 5090 keeps full 8bit, the spark layers become nvfp4 because thats what blackwell silicon eats natively.

results:
66 tok/s short context, the 5090 alone does 51 on this model
57 at 39k deep, 34 at 95k, 28 at 195k, needle tests clean everywhere
full 256k window, alone the 5090 cant hold a third of that
164 tok/s over 12 parallel chats
humaneval 95.7, gsm8k 96.5

so yes it works. 5090 speed, spark memory, 8bit quality. all three at once.

recipe and honest limits in the replies.

[Mark Argyelan @markargyelan_]

https://t.co/xrIzpnLuB8

[Mark Argyelan @markargyelan_]

honest notes:
- to be clear about quality class: a FULL-NVFP4 model on the 5090 alone is faster than this rig (~110-180 tok/s, see the recent r/LocalLLM thread - link). that's a different game. this rig exists to run the model at 8-bit grade: 48 of 65 layers stay full Q8, only the slow box's 17 get NVFP4

- community data says NVFP4 requantizing usually hurts quality (~15x worse KLD than 6-bit). our evals disagree for this mix: HumanEval 95.7 (157/164), GSM8K 96.5 (193/200) — probably because 74% of the weights stay 8-bit

- our MTP numbers match that thread exactly: n=3 sweet spot, ~60% acceptance, creative prose drops to ~40%. reproducibility is a feature

- prefill 1624 tok/s — the Blackwell FP4 tensor cores, documented upstream

[Mark Argyelan @markargyelan_]

the whole upgrade is one line changed:

llama-quantize --allow-requantize --tensor-type "blk\.([0-9]|1[0-6])\.=nvfp4" q8.gguf splitbrain-nvfp4.gguf q8_0

serve like before (llama.cpp RPC, -ts 17/48, MTP on). one trap we hit so you don't have to: don't combine -np 12 with --kv-unified — decode collapses to 27 tok/s. -np 4 + --kv-unified is the sweet spot.

[Mark Argyelan @markargyelan_]

fair guess, but we measured it:

the wire adds ~2ms per token, total (0.4ms per hop, traced on the worker).

the tractor spends 20-30ms per token reading its own weights. the straw isn’t the bottleneck - the tractor’s RAM is.

which is exactly why compressing only the tractor’s share works: +22% measured, quality checks intact. and the ferrari isn’t waiting idle

it computes 48 of the 65 layers.

[Mark Argyelan @markargyelan_]

I think we found something.

We call it Split-Brain Quantization.

The problem: I run one AI model across two unequal machines - a fast gaming GPU (RTX 5090, 32GB) and the DGX Spark (128GB, but 6.5x slower memory).

Chained together (10 Gbit-Ethernet), the slow box sets the pace.

Every byte it reads costs 6.5x more time.

The idea: don't compress the model equally. Compress each layer based on WHICH CHIP it lives on.The 5090's 48 layers: full 8-bit quality.

The Spark's 17 layers: compressed to 6-bit - that box can't read fast enough for the difference to matter anyway.

Quality stays where quality gets computed. Bytes shrink where bytes are expensive.

Result on Qwen3.8-27B: full-quality answers, 600 pages of context, ~50 tokens/sec at home.

This morning the same rig did 7.7.

One llama.cpp command, works today,
recipe in the replies.

[Mark Argyelan @markargyelan_]

recipe (llama.cpp, all open source):

slow box: ggml-rpc-server -H 0.0.0.0 -p 50052 -c

fast box: llama-server -m qwen38-asym.gguf --rpc <slowbox>:50052 -ngl 99 -ts 17/48 -fa on -ctk q8_0 -ctv q8_0 --spec-type draft-mtp --spec-draft-n-max 3 -np 12 -c 262144 --jinja

the "compress only the slow half" gguf: llama-quantize --allow-requantize --tensor-type "blk.([0-9]|1[0-6]).=q6_k" q8.gguf asym.gguf q8_0

[Mark Argyelan @markargyelan_]

@Midnight_Captl heaven

[Mark Argyelan @markargyelan_]

Nobody’s machine can run this model. Mine can’t either. It runs anyway.

Hy3 295B, 136GB (Q3_K_M) too big for my 5090, too big for my DGX Spark.

So Split across both over a $30 Ethernet cable: 16.9 tok/s. 😂

Bonus: the model speculates its own tokens across the cable (+19%). The $8k twin-Spark rig with fancy 200GbE couldn’t make that work.

[Mark Argyelan @markargyelan_]

@orskyai ConnectX-7 PCI-Cards are currently non existent in switzerland…
The thing is: nobody’s measured MTP speculation over an Ethernet cable before.

[Mark Argyelan @markargyelan_]

@ChrisVolkernick One day..

[Mark Argyelan @markargyelan_]

@LotusDecoder @hkdom If you drop your cc to 1800, you'll probably reach -10 degrees.😆

[Mark Argyelan @markargyelan_]

@deeperflows I installed a MagSafe adapter so no cable is needed.
Or you have a better solution?

[Mark Argyelan @markargyelan_]

@idare You're totally right, my bad

[Mark Argyelan @markargyelan_]

@en3coa &lt;3