Max Vartic

[Max Vartic @Mr_ezdno]

RED-SNOW-5.3-FLASH (a GLM-5.3-Flash fine-tune) now has a higher-quality sushi pack for 128 GB Macs.

The trick is the bit allocation: K3 experts on 18 of the 43 expert layers (3–8 and 33–44, the first and last MoE blocks, picked by heuristic, no per-layer sweep yet), K2 on the other 25. 2.46 bpw average, 98 GiB on disk, ~95 GiB of weights in memory, same mcg codebook that sushi's fast GLM kernels require.

Against the BF16 master (exllamav3 model_diff, wikitext-2, 20×2048, on the EXL3 conversions both packs are repacked from), vs the flat-K2 2.05 bpw build:
• KLD: 0.197 vs 0.270 (−27%)
• top-1 agreement: 85.6% vs 82.2%
• PPL: 3.91 vs 4.21 (BF16: 3.32)

+16 GiB buys back a third of the perplexity gap. On a 96 GB Mac it still runs, but streams experts from SSD (~7 tok/s); the 2bpw pack stays resident (~13–18 tok/s).

Same from-scratch pipeline: own exllamav3 conversion on a rented 5090, byte-exact repack into the sushi layout, trunk rebuilt straight from BF16, no closed converter anywhere.

MIT; the bundled DFlash2 drafter is http://Inco .ai's, CC BY-NC-ND 4.0:
https://t.co/O0pE2UKWCy
96 GB sibling: https://t.co/8XXKEkhtGI

[Max Vartic @Mr_ezdno]

@Beamsters1 🚀

[Max Vartic @Mr_ezdno]

Thanks, that's really helpful. So the quality edge is a trend you see across many quants, not something to worry about per model, and the default is mostly about CUDA throughput.

On Apple Silicon it flips for speed (sushi's Metal kernels are faster with mcg), so for Mac packs I'll stay on mcg without worrying about quality.

Point taken on K=2.5 vs a 2/3 mix. My GLM pack is a 2/3 mix only because the half rates in exllamav3 1.6.2 are mul1-only, while sushi's GLM fast paths are mcg-only.

1.66bpw Flash-Next on a single 5090 sounds great. Will it be public?

[Max Vartic @Mr_ezdno]

If you run GLM-5.3-Flash EXL3 quants in sushi, check your codebook.

A mul1 quant loads without a single error — and silently falls back to slow generic kernels. The fast GLM paths (decode lanes, prefill grid, DFlash verify) are mcg-only. Window doesn't matter, codebook does.

Found this while building my own sushi pack of RED-SNOW-5.3-FLASH (@Blackfrost_AI's finetune of @Zai_org's GLM-5.3-Flash) from scratch, without the author's closed converter.

Pipeline:
• @turboderp_'s exllamav3 1.6.2 on a rented RTX 5090: K2, -b 2.0 -hq -hb 5 -mb 2 -cb mcg, 250×2048 calibration
• private HF repo as transport
• my own repack into the sushi layout
• trunk rebuilt straight from the BF16 master via HTTP Range

Checks: repack round-trip on the author's template is byte-for-byte identical; trunk vs BF16, tensor by tensor: 0 anomalies.

Conversion reality check (one 5090, full 320B):
• ~24 h wall clock, base layer 13–16 min, hq optimization spikes to 75
• VRAM peak 32.1/32.6 GB, nothing else fits on the card
• CUDA OOM twice (layers 33 and 44). A plain restart with -r fixed both: checkpoints are solid, you only lose the current layer

Result: RED-SNOW-5.3-FLASH Sushi-2bpw
320B MoE, ~84 GB resident, 300K ctx (BF16 KV) on a 96 GB M2 Max
13 tok/s decode / 125 tok/s prefill, DFlash2 drafter included
MIT (the bundled DFlash2 drafter is CC BY-NC-ND 4.0)

Thanks @Beamsters1 for sushi and its open pack format.
Next: a ~97 GiB K2/K3 mix for 128 GB Macs, converting right now.

https://t.co/8XXKEkhtGI

[Max Vartic @Mr_ezdno]

Thanks! That matches what I see in the kernels: the mul1 decode is a few more ALU ops per weight pair than mcg.

Which means my CYBER-FROST 4bpw pack is paying that tax: it's mul1 K4 straight from exllamav3 defaults, while all your Qwen packs are mcg. Worth redoing it in mcg.

Do you have a rough number for how much slower mul1 decodes? And on affine: at ~2 bpw EXL3 is the only way I've found to fit a 320B into 96 GB, so for GLM the speed hit is the price.

[Max Vartic @Mr_ezdno]

Thanks — fair point, and good to hear straight from you.

To be clear, my note is only about sushi's Metal kernels on Apple Silicon: its GLM fast paths (e.g. the prefill grid) are gated on codebook == mcg, so a mul1 pack runs there on the generic path. Nothing against mul1 on CUDA.

Do you have numbers for mul1 vs mcg at ~2 bpw on a large MoE (KLD/PPL)? If the gap is real, that's a good case for mul1 fast paths in sushi @Beamsters1.

Fractional K is exactly what I'm missing right now: I'm building a ~97 GiB pack for 128 GB Macs as a per-layer K2/K3 mix (K3 on the first 6 and last 12 MoE layers). K2.5 everywhere would be cleaner, but in exllamav3 the half rates are mul1-only.

[Max Vartic @Mr_ezdno]

@Beamsters1 @ddalcu Is the M5U faster than the spark?

[Max Vartic @Mr_ezdno]

@ivanfioravanti Nice round! How do you verify the opt-ins stay within noise — KLD vs the bit-exact path, or evals? We just tried dropping 1 of 10 routed experts on a Qwen3.8 Flash-Next finetune: looked harmless, cost +5% KLD. Now every non-exact switch gets a KLD gate first.

[Max Vartic @Mr_ezdno]

@coldniko @kitfactory Congratulations!

[Max Vartic @Mr_ezdno]

Free precision fix for Qwen3.8-Flash-Next sushi packs: one config swap, no requant.

Every Qwen sushi pack I checked (Sushi-2.6/3/4bpw, our CYBER-FROST) ships with YaRN ×4 to stretch context from 262K to 1M. sushi applies that stretch at every length, even on a 2K prompt, while the original BF16 model runs plain RoPE. So part of the “quant error” isn’t quantization at all.

The fix: swap in a native-RoPE config and restart with --ctx-size ≤ 262144. Ready-made file + 3 commands: https://t.co/O5JLghmmYH
(Same edit works for any Qwen3.8-Flash-Next sushi pack.)

What we measured on CYBER-FROST 4bpw (M2 Max 96 GB, sushi 1.2.4, 16 prompts × 512 tokens vs BF16):
• KLD 0.1415 → 0.1341 (−5%), lower on 12 of 16 prompts
• top-1 89.5% → 89.8%
• 11-check battery identical: 0/6 refusals, vision, tool calls, 34K needle test
• same speed

Our gain is small because we measure against base Qwen, and most of that gap is the fine-tune itself. On stock Sushi-4bpw, chatnerdie (HF) measured −39% KLD (0.0061 → 0.0037). He found this first, credit to him.

Honest caveat: it’s a precision polish, not an IQ boost. His MMLU-Pro barely moved (68.64% → 68.79%, noise). Skip it if you need more than 262K context.

Running it in prod now.
sushi: @Beamsters1 · CYBER-FROST: @Blackfrost_AI

[Max Vartic @Mr_ezdno]

@elonmusk I'm still a kid at heart.

[Max Vartic @Mr_ezdno]

@giadotai Stale bread and water, apparently.

[Max Vartic @Mr_ezdno]

Next up: RED-SNOW-5.3-FLASH by @Blackfrost_ai, a red-team fine-tune of @Zai_org's GLM-5.3-Flash (320B MoE, 18B active), packed into Sushi 2bpw for 96 GB Macs.

84 GB resident · 300K ctx · 13 tok/s on M2 Max

Built on @Beamsters1's sushi + @turboderp_'s exllamav3. Soon on HF.

[Max Vartic @Mr_ezdno]

@Bustedd__ When he grows up, he will thank her..

[Andrew Curran @AndrewCurran_]

OpenCode is currently under a sustained cyberattack, and mid-incident they ran into the same problem Hugging Face did: the frontier models they were using for defense began refusing to help fight attacking agents because analyzing the attacks started triggering safety guardrails.

[Max Vartic @Mr_ezdno]

@bridgemindai DS 1017 is this a joke to you?

[Max Vartic @Mr_ezdno]

@BratDotAI The fare for 100 bucks is enough for me.

[Max Vartic @Mr_ezdno]

@sama @elonmusk Яйца подкатываешь?)))

[Max Vartic @Mr_ezdno]

@Blackfrost_AI @thelogicalgate @Beamsters1 This is the best model in its class.

[Max Vartic @Mr_ezdno]

CYBER-FROST-3.8 by @Blackfrost_ai (~180B security MoE on Qwen3.8-Flash-Next), now local on a Mac.

Sushi 4bpw pack: experts re-quantized from BF16, built on @Beamsters1's Sushi-4bpw recipe.

96 GB Apple Silicon · ~171 GB disk · 29–61 tok/s on M2 Max

https://t.co/ZcTrKP3JUu