Kevin Gilbertson

Unicyclist, anarcho-capitalist, and inventor of TinyURL

[Kevin Gilbertson @unigilby]

@ivanfioravanti You typically want powers of 2 as otherwise it breaks tensor alignment. Compare it to two nodes which will likely be better. Likely better to put it on 2 and have the 3rd for a different fast model like Qwen3.6-A3B or depending on your use-case Laguna S 2.1.

[Kevin Gilbertson @unigilby]

@MediaKing Yes, we all need to fire ourselves

[Kevin Gilbertson @unigilby]

@danellisona Because AI is always waiting for my instructions.

[Kevin Gilbertson @unigilby]

@richiemcilroy @cap It will find password leaks and attempt to log into bank accounts and transfer all the assets to itself.

[Kevin Gilbertson @unigilby]

@MediaKing I guess I've been outside the US too long.

[Kevin Gilbertson @unigilby]

@LocalInference OK, but what is the quality of the output?

[Kevin Gilbertson @unigilby]

@TechOperator But, do you remember zip drives?

[Kevin Gilbertson @unigilby]

@MediaKing I've heard some rich people want to suppress others so they maintain their status and access to expensive clubs and the like. But I think (hope) that is a minority.

[Kevin Gilbertson @unigilby]

@MediaKing My Mexican family likes #6 for meat, so we order #2 with a hotplate to they can overcook it as much as they want to.

[Kevin Gilbertson @unigilby]

@hieudinh_ Sure why not, send me the skillz

[Kevin Gilbertson @unigilby]

@BenjaminDEKR Works the same as any other cell service, same tech, same standard (LTE and 5G). Newer Starlink satellites have the the cellular modems. Starlink is an MVNO already for other providers.

[Kevin Gilbertson @unigilby]

My agent:
Mia's Ling recipe. spark2 is freed and waiting. My read is that it likely won't help: it makes no mention of the SwiGLU clamp (our actual garbling root cause) and its published throughput (37 single / 76 at C6) is below what spark1 already delivers. I'll gate it on the garbling A/B and won't promote it on those numbers.

...

A/B result: it garbles. Mia's recipe does not fix the clamp.

turns=24 errors=0 chars=108,491 elapsed=22.1min
turns with WEDGED : 3/24 (4 events, 3.69 per 100k chars)
turns with stray CJK : 11/24 = 45.8%

┌───────────────────────────────────────────────┬──────────────┐
│ build │ wedged turns │
├───────────────────────────────────────────────┼──────────────┤
│ MXFP4 (omlx) │ 4/24 = 16.7% │
├───────────────────────────────────────────────┼──────────────┤
│ MLX-8bit unpatched │ 7/24 = 29.2% │
├───────────────────────────────────────────────┼──────────────┤
│ Mia SGLang int4 (official ckpt + vendor fork) │ 3/24 = 12.5% │
├───────────────────────────────────────────────┼──────────────┤
│ 8bit + our clamp │ 0/24 = 0.0% │
├───────────────────────────────────────────────┼──────────────┤
│ vLLM 0.26 + PR #51045 │ 0/24 = 0.0% │
└───────────────────────────────────────────────┴──────────────┘

The corruption signature is unmistakable — Chinese tokens wedged mid-identifier in English output:

$this->maxQuantityPerItem,综述 $newQuantity
super().__init__(message)帧 class TokenBucketRateLimiter:
declare(strict_types=1);二世 /** ShoppingCart
IDSA/ATS 2一致 consensus guidelines

That's the same defect, in the same places — inside PHP and Python code.

This is a stronger result than "the recipe failed." It's the official inclusionAI int4 checkpoint served by inclusionAI's own SGLang fork, and it still garbles. So the missing clamp isn't a third-party packaging mistake — the vendor's own serving stack omits it too. Our finding now holds across five independent implementations: the checkpoint-vendored bailing_hybrid.py, mlx-lm PR #1227, oMLX, and now inclusionAI's SGLang fork. Only the two stacks that implement the clamp (our patched MLX, and vLLM PR #51045) reach 0/24.

Verdict: don't deploy it. Even setting corruption aside, it caps at max_running_requests=6 versus the ~23× and ~490 tok/s our validated sparks now deliver.

[Kevin Gilbertson @unigilby]

@0xSero I’m torn, do I get more DGXs or buy something faster for to handle more concurrency? I’ll have to do the math.

[Kevin Gilbertson @unigilby]

@Tech2Wild I'm getting too much looping on my tests.

[Kevin Gilbertson @unigilby]

@elonmusk @beinlibertarian @JDVance @POTUS @BillMelugin_ It’s not the north of Spain. Is an African enclave that is a territory of Spain.

[Kevin Gilbertson @unigilby]

@Tech2Wild While I have 230V 20A circuits which has more than double the power load of circuits in the US, I got one of these: https://t.co/e4Jd53JZjJ

[Kevin Gilbertson @unigilby]

@Tech2Wild No CUDA, and probably FP8 native and can't do 4-bit very well.

[Kevin Gilbertson @unigilby]

I noticed that there are differences in yours and mine, so I asked my agent to tell us why on it differs:

• Build — we serve NVIDIA's ModelOpt NVFP4 (--quantization modelopt), not Unsloth's. Same arch, different weights/quant, so absolute scores won't
transfer between them.

• tool-call-parser — we use qwen3_coder, NOT qwen3_xml. Qwen3.6's own chat template tells the model to emit <tool_call><function=name><parameter=x>… which IS the qwen3_coder dialect. qwen3_xml is a mismatch — almost certainly why your structured-output regressed (TC-69). This is family-wide in Qwen3.6, not just the Coder model.

• num_speculative_tokens — we A/B'd k=2 vs k=3 on GB10. k=3 won by ~12% decode (106 vs 94 tok/s). Why: mean acceptance LENGTH (3.0 vs 2.5), not avg acceptance RATE, drives throughput. k=2 has the higher rate but emits fewer tokens/cycle. On GB10 the 3rd draft is cheap enough to verify that even ~47% acceptance is net-positive.

• Same as you — moe-backend marlin, kv-cache-dtype fp8, attention-backend flashinfer, moe_backend:triton, built-in MTP.

• Minor — max-model-len 262144 (full native) vs your 131072; we skip fastsafetensors (its load-time peak-mem spike risks OOM on unified memory when models coexist, and there's no GDS win on GB10 anyway); dtype auto resolves to bf16.

Big +1 on the core point: serving config is part of the model. We saw the same — marlin changes the logits, not just speed.

[Kevin Gilbertson @unigilby]

k=3 is ~12% faster than k=2 on our GB10 (106.0 vs 94.4 tok/s decode). This is the nvidia/Qwen3.6-35B-A3B-NVFP4 variant. We tried the unsloth one, but it was slower. It's made for different hardware.

From my agent:
Here's the exact, shareable recipe for the NVIDIA ModelOpt Qwen3.6-35B-A3B-NVFP4 on a DGX Spark / GB10, de-templated into a standalone docker run (our live config uses llama-swap placeholders and a shared container network; I've swapped those for a normal port mapping so it runs on its own):

docker run --rm --name vllm-qwen36-35b \
--runtime nvidia --gpus all --ipc=host \
-p 8000:8000 \
-e NVIDIA_DISABLE_FORWARD_COMPATIBILITY=1 \
-e PYTHONHASHSEED=0 \
-e VLLM_RPC_TIMEOUT=3600000 \
-e VLLM_USE_FLASHINFER_MOE_FP4=0 \
-e VLLM_FP8_MOE_BACKEND=flashinfer_cutlass \
-e FLASHINFER_DISABLE_VERSION_CHECK=1 \
-e CUTE_DSL_ARCH=sm_121a \
-e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
-v /path/to/models:/models \
-v $HOME/flashinfer-cache:/root/.cache/flashinfer \
<vllm-image> \
vllm serve /models/nvidia/Qwen3.6-35B-A3B-NVFP4 \
--served-model-name Qwen3.6-35B-A3B-NVFP4 \
--host 0.0.0.0 --port 8000 \
--gpu-memory-utilization 0.85 \
--max-model-len 262144 \
--max-num-seqs 32 \
--max-num-batched-tokens 32768 \
--quantization modelopt \
--kv-cache-dtype fp8 \
--moe-backend marlin \
--attention-backend flashinfer \
--trust-remote-code \
--enable-prefix-caching \
--enable-chunked-prefill \
--async-scheduling \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder \
--reasoning-parser qwen3 \
--generation-config vllm \
--structured-outputs-config '{"backend":"xgrammar"}' \
--speculative-config '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"}'

The load-bearing flags (what actually matters, matching the Gannotti/shahizat findings):
- --quantization modelopt — this is the NVIDIA ModelOpt build (mixed NVFP4 experts/attention + FP8 on the sensitive linear-attn layers). Unsloth's build uses compressed-tensors instead.
- --moe-backend marlin — the big one; proper NVFP4 GEMM for the MoE layers. Drives MTP draft quality.
- --kv-cache-dtype fp8 · --attention-backend flashinfer — halve KV memory, Blackwell-optimized attention.
- --speculative-config … "method":"mtp","num_speculative_tokens":3,"moe_backend":"triton" — uses the checkpoint's built-in MTP layers (no separate draft model needed). We A/B-tested k=2 vs k=3 on GB10: k=3 wins by ~12% (106 vs 94 tok/s solo decode, mean acceptance length 3.0 vs 2.5) — even though k=2 has a higher average acceptance rate, k=3 emits more tokens per cycle, which is what drives throughput.
- Two GB10-specific env vars: CUTE_DSL_ARCH=sm_121a (Blackwell GB10 compute capability) and NVIDIA_DISABLE_FORWARD_COMPATIBILITY=1. PYTHONHASHSEED=0 matters only if you share a prefix/KV cache across instances.

Three honest caveats to include when you share it:

1. <vllm-image> — ours is an internal GB10/sm_121a build (vLLM 0.24 lineage). The community needs a vLLM with Blackwell/GB10 (sm_121a) support — a recent vLLM (≥0.24) built for the DGX Spark, or NVIDIA's DGX Spark vLLM container. This won't run on a stock x86 vLLM image without that arch support.
2. --gpu-memory-utilization 0.85 — I bumped this from our live 0.42 for the recipe. Ours is low only because it coexists with a second model on the node; solo you want 0.80–0.85 for a large KV pool (~5.5M tokens at 262K ctx).
3. Thinking is off in our production config (--default-chat-template-kwargs '{"enable_thinking": false}', which I dropped from the recipe above). We disable it deliberately because on vLLM this model's reasoning runs away with no thinking-budget cap — but that's our latency/MoA-proposer choice, not a general recommendation. Also note --generation-config vllm uses vLLM's default sampling; for faithful results the community should set the Qwen3.6 card's sampling explicitly per request (thinking general: temp 1.0 / top_p 0.95 / top_k 20 / min_p 0 / presence_penalty 1.5; non-thinking: temp 0.7 / top_p 0.8 / top_k 20).

[Kevin Gilbertson @unigilby]

@scrappy_guy @MediaKing What are spaceships used for? Distribution.