Max Vartic

[Max Vartic @Mr_ezdno]

@Beamsters1 🚀

[Max Vartic @Mr_ezdno]

Thanks, that's really helpful. So the quality edge is a trend you see across many quants, not something to worry about per model, and the default is mostly about CUDA throughput.

On Apple Silicon it flips for speed (sushi's Metal kernels are faster with mcg), so for Mac packs I'll stay on mcg without worrying about quality.

Point taken on K=2.5 vs a 2/3 mix. My GLM pack is a 2/3 mix only because the half rates in exllamav3 1.6.2 are mul1-only, while sushi's GLM fast paths are mcg-only.

1.66bpw Flash-Next on a single 5090 sounds great. Will it be public?

[Max Vartic @Mr_ezdno]

If you run GLM-5.3-Flash EXL3 quants in sushi, check your codebook.

A mul1 quant loads without a single error — and silently falls back to slow generic kernels. The fast GLM paths (decode lanes, prefill grid, DFlash verify) are mcg-only. Window doesn't matter, codebook does.

Found this while building my own sushi pack of RED-SNOW-5.3-FLASH (@Blackfrost_AI's finetune of @Zai_org's GLM-5.3-Flash) from scratch, without the author's closed converter.

Pipeline:
• @turboderp_'s exllamav3 1.6.2 on a rented RTX 5090: K2, -b 2.0 -hq -hb 5 -mb 2 -cb mcg, 250×2048 calibration
• private HF repo as transport
• my own repack into the sushi layout
• trunk rebuilt straight from the BF16 master via HTTP Range

Checks: repack round-trip on the author's template is byte-for-byte identical; trunk vs BF16, tensor by tensor: 0 anomalies.

Conversion reality check (one 5090, full 320B):
• ~24 h wall clock, base layer 13–16 min, hq optimization spikes to 75
• VRAM peak 32.1/32.6 GB, nothing else fits on the card
• CUDA OOM twice (layers 33 and 44). A plain restart with -r fixed both: checkpoints are solid, you only lose the current layer

Result: RED-SNOW-5.3-FLASH Sushi-2bpw
320B MoE, ~84 GB resident, 300K ctx (BF16 KV) on a 96 GB M2 Max
13 tok/s decode / 125 tok/s prefill, DFlash2 drafter included
MIT (the bundled DFlash2 drafter is CC BY-NC-ND 4.0)

Thanks @Beamsters1 for sushi and its open pack format.
Next: a ~97 GiB K2/K3 mix for 128 GB Macs, converting right now.

https://t.co/8XXKEkhtGI

[Max Vartic @Mr_ezdno]

Thanks! That matches what I see in the kernels: the mul1 decode is a few more ALU ops per weight pair than mcg.

Which means my CYBER-FROST 4bpw pack is paying that tax: it's mul1 K4 straight from exllamav3 defaults, while all your Qwen packs are mcg. Worth redoing it in mcg.

Do you have a rough number for how much slower mul1 decodes? And on affine: at ~2 bpw EXL3 is the only way I've found to fit a 320B into 96 GB, so for GLM the speed hit is the price.

[Max Vartic @Mr_ezdno]

Thanks — fair point, and good to hear straight from you.

To be clear, my note is only about sushi's Metal kernels on Apple Silicon: its GLM fast paths (e.g. the prefill grid) are gated on codebook == mcg, so a mul1 pack runs there on the generic path. Nothing against mul1 on CUDA.

Do you have numbers for mul1 vs mcg at ~2 bpw on a large MoE (KLD/PPL)? If the gap is real, that's a good case for mul1 fast paths in sushi @Beamsters1.

Fractional K is exactly what I'm missing right now: I'm building a ~97 GiB pack for 128 GB Macs as a per-layer K2/K3 mix (K3 on the first 6 and last 12 MoE layers). K2.5 everywhere would be cleaner, but in exllamav3 the half rates are mul1-only.

[Max Vartic @Mr_ezdno]

@Beamsters1 @ddalcu Is the M5U faster than the spark?

[Max Vartic @Mr_ezdno]

@ivanfioravanti Nice round! How do you verify the opt-ins stay within noise — KLD vs the bit-exact path, or evals? We just tried dropping 1 of 10 routed experts on a Qwen3.8 Flash-Next finetune: looked harmless, cost +5% KLD. Now every non-exact switch gets a KLD gate first.

[Max Vartic @Mr_ezdno]

@coldniko @kitfactory Congratulations!

[Max Vartic @Mr_ezdno]

Free precision fix for Qwen3.8-Flash-Next sushi packs: one config swap, no requant.

Every Qwen sushi pack I checked (Sushi-2.6/3/4bpw, our CYBER-FROST) ships with YaRN ×4 to stretch context from 262K to 1M. sushi applies that stretch at every length, even on a 2K prompt, while the original BF16 model runs plain RoPE. So part of the “quant error” isn’t quantization at all.

The fix: swap in a native-RoPE config and restart with --ctx-size ≤ 262144. Ready-made file + 3 commands: https://t.co/O5JLghmmYH
(Same edit works for any Qwen3.8-Flash-Next sushi pack.)

What we measured on CYBER-FROST 4bpw (M2 Max 96 GB, sushi 1.2.4, 16 prompts × 512 tokens vs BF16):
• KLD 0.1415 → 0.1341 (−5%), lower on 12 of 16 prompts
• top-1 89.5% → 89.8%
• 11-check battery identical: 0/6 refusals, vision, tool calls, 34K needle test
• same speed

Our gain is small because we measure against base Qwen, and most of that gap is the fine-tune itself. On stock Sushi-4bpw, chatnerdie (HF) measured −39% KLD (0.0061 → 0.0037). He found this first, credit to him.

Honest caveat: it’s a precision polish, not an IQ boost. His MMLU-Pro barely moved (68.64% → 68.79%, noise). Skip it if you need more than 262K context.

Running it in prod now.
sushi: @Beamsters1 · CYBER-FROST: @Blackfrost_AI

[Max Vartic @Mr_ezdno]

@elonmusk I'm still a kid at heart.

[Max Vartic @Mr_ezdno]

@giadotai Stale bread and water, apparently.

[Max Vartic @Mr_ezdno]

Next up: RED-SNOW-5.3-FLASH by @Blackfrost_ai, a red-team fine-tune of @Zai_org's GLM-5.3-Flash (320B MoE, 18B active), packed into Sushi 2bpw for 96 GB Macs.

84 GB resident · 300K ctx · 13 tok/s on M2 Max

Built on @Beamsters1's sushi + @turboderp_'s exllamav3. Soon on HF.

[Max Vartic @Mr_ezdno]

@Bustedd__ When he grows up, he will thank her..

[Andrew Curran @AndrewCurran_]

OpenCode is currently under a sustained cyberattack, and mid-incident they ran into the same problem Hugging Face did: the frontier models they were using for defense began refusing to help fight attacking agents because analyzing the attacks started triggering safety guardrails.

[Max Vartic @Mr_ezdno]

@bridgemindai DS 1017 is this a joke to you?

[Max Vartic @Mr_ezdno]

@BratDotAI The fare for 100 bucks is enough for me.

[Max Vartic @Mr_ezdno]

@sama @elonmusk Яйца подкатываешь?)))

[Max Vartic @Mr_ezdno]

@Blackfrost_AI @thelogicalgate @Beamsters1 This is the best model in its class.

[Max Vartic @Mr_ezdno]

CYBER-FROST-3.8 by @Blackfrost_ai (~180B security MoE on Qwen3.8-Flash-Next), now local on a Mac.

Sushi 4bpw pack: experts re-quantized from BF16, built on @Beamsters1's Sushi-4bpw recipe.

96 GB Apple Silicon · ~171 GB disk · 29–61 tok/s on M2 Max

https://t.co/ZcTrKP3JUu

[Max Vartic @Mr_ezdno]

@realAlbertMend @brockm It's a pity Mike doesn't have anything. 😭