@MiaAI_lab Did you check out what @ViC305 and I put together? Tp 4, 3.38bpw, but KLD is practically lossless. Up to 73 tok/s structure. Fast, smart, 1m context.
Depends on what you value. What happens when Claude doesn't do the things you want it to, and they made other models illegal?
What happens when only 2 AI providers rule the world?
What happens when you realize that everything you send to Anthropic is getting cataloged and stored, and they may later use your chats to create a competing business?
@bertholomusai@OfficialAureleo@ViC305 Yeah you can't. There is no concurrency. It's maxed for 1m context single stream.
Not sure why that doesn't "actually matter," but it's what it's built for.
๐ฅ @cfontes got the full 753B GLMโ5.3 generating code at 73.6 tokens/sec on four DGX Sparks at 1M CTX. ๐ Thatโs the FASTEST single-stream code result I found across the four public recipes I compared.
When I released my 3.38 bpw SAGEโEXL3 quant, we had all 19,200 routed experts in 319 GB. Chris funded the compute himself, then spent four weeks working on what happens after the weights load.
Hereโs what his TensorFold TP4 + DFlash2 recipe delivers on one request at a time, measured at p50:
๐ป 69.2 tok/s on code
โ๏ธ 55.7 tok/s on prose
๐งฉ 73.6 tok/s on structured output
For perspective, the other published single-stream code figures I found were 57.8 tok/s from @u1tra_instinct , 46.6 from @Knapcio and ~43 from @bertholomusai . Each used its own benchmark harness.
The number I care about most for my own agent sessions is 19.4 seconds. By turn six, the conversation had grown to 15,930 tokens. A cold turn took 85.3 seconds. Chrisโs prefix cache reused 12,288 tokens of history and brought that same turn down to 19.4 seconds. Thatโs over a minute saved on one turn, with a 1M-token context window configured for the session.
On my independent fidelity check, the quant picked the same next token as BF16 9,521 times out of 10,240 (93%) with a perplexity increase of 2.1%!
@cfontes Chris, thank you for putting your own time and money into this and publishing the exact setup so the rest of us can run it. ๐
๐ ๏ธ Chrisโs four-Spark recipe
https://t.co/cxLkWNg9AY
๐ค My SAGEโEXL3 weights
https://t.co/L1N626Y4qx
How does a trusted AI agent become a โdouble agentโ?
Walton Comer and @gregnfrank discuss how manipulation over several interactions can push agents beyond their authorized scope, with @JamesCHeckman on TheStreet Roundtable.
๐ FULL GLM-5.3. FOUR DGX SPARKS. A NEW PARETO-FRONTIER CONTENDER.
This is a HUGE personal milestone for my SAGE work! ๐ฅ
My GLM-5.3 SAGE EXL3 3.38 bpw brings the full model down to 319 GB while preserving ALL 19,200 routed experts.
No pruning. No expert merging. Every bit matters.
๐ฏ INDEPENDENT-SET FIDELITY
โข KL divergence: 0.082388 nats
โข BF16 top-1 agreement: 91.1523%
โข 10,240 scored positions on text excluded from calibration
โก FOUR-SPARK PERFORMANCE
Around 41 tok/s average (69 tok/s burst) across six real prompts with DFlash2, with drafted replies matching serial decoding token for token.
256K-context serving demonstrated. Sized toward a 1M-context memory target, with full-length inference validation still ahead.
The goal is the highest-fidelity full GLM-5.3 possible on four Sparks WITHOUT sacrificing the memory needed for 256Kโ1M context. Not just making the weights fit, but leaving room to actually USE the model.
๐ A MASSIVE thank you to @cfontes, who personally funded the compute required to make this model possible. This release would not have happened without his support!
And weโre not done. Heโll be releasing a recipe soon thatโs going to blow everyoneโs minds! ๐คฏ๐ฅ
This is what community-backed local AI looks like. More capability, less memory, and people coming together to push whatโs possible. ๐ช
Model ๐
https://t.co/L1N626Y4qx
@MichaelGannotti@repressionnap@jauneclaude@grok Yeah I wasn't so much saying you were wrong, just that I've got a different approach to it.
It's hyper frustrating, but also not a shock that they did this.
Just sad.
TensorFold 0.3.6 just took CUDA EXL3 from a GLM-only experiment to a reusable mixed-bit backend. ๐คฏ
I wrote the shared EXL3 module that made that possible.๐
49 files. +5,673 / -37. 3 codebooks. 1โ8 bits. Mixed precision per tensor. โ๏ธ
And it SHIPPED. โ
Before this, TensorFoldโs CUDA EXL3 path was built specifically around GLM-5.3-Flashโs 4-bit MCG experts.
Now the same infrastructure serves:
โ Qwen3.8-27B EXL3 + DFlash2
โ Qwen3.8-Flash-Next EXL3 + MTP
โ Mixed-K SAGE EXL3 packs
โ MiMo-V2.6-Flash EXL3 tensors through the same shared module
No separate EXL3 kernel stack for every new family.
๐ง๐๐ ๐ฃ๐๐ฅ๐ง ๐โ๐ ๐ ๐ข๐ฆ๐ง ๐ฃ๐ฅ๐ข๐จ๐ ๐ข๐
The reader understands:
3inst / MCG / mul1
1 โ 8 bits
different widths per tensor
So one expert can be 2-bit while another is 8-bit.
Unsupported checkpoints fail early instead of getting halfway through a serve before exploding.
The CUDA linear is also row-invariant from 1 โ 128 rows, with no cuBLAS dependency. A tensorโs result doesnโt change because more rows happened to arrive beside it.
Grouped routed-expert GEMV handles mixed widths in one launch per MoE projection and can be captured in a CUDA graph.
And I left TensorFoldโs existing GLM EXL3 and MLX paths alone.
๐ง๐๐ ๐๐ข๐ช-๐๐๐ง ๐๐๐ฅ๐ก๐๐ ๐ช๐๐ฆ ๐ง๐๐ ๐ฆ๐จ๐ฅ๐ฃ๐ฅ๐๐ฆ๐
On real MiMo 2-bit tensors:
q_proj: 157 โ 220 GB/s
o_proj: 76 โ 186 GB/s
down_proj: 79 โ 193 GB/s
That is kernel bandwidth, not model tok/s.
And the new path stayed torch.equal across 2,472 outputs.
ExLlamaV3 measured 176โ233 GB/s on those same tensors, so TensorFoldโs 2-bit rows are no longer sitting 25โ60% behind.
๐๐ก๐ ๐๐ง ๐๐๐ง๐จ๐๐๐๐ฌ ๐ฆ๐๐ฅ๐ฉ๐๐ฆ
On ONE DGX Spark, Qwen3.8-27B 3.00 bpw EXL3:
Code sampled:
83.4 tok/s
TensorFold MLX 4-bit:
57.5 tok/s
vLLM MTP=3:
23.4 tok/s
Flash Next 3.05 bpw EXL3:
Code sampled:
80.8 vs 42.4 tok/s vLLM
Code greedy:
77.7 vs 40.9
Chat greedy:
69.2 vs 37.6
The mixed-K SAGE Flash Next pack also runs 1.51โ1.55ร ExLlamaV3 serial speed in the measured cells.
There are tradeoffs. Prefill still needs work, and ExLlamaV3 remains slightly faster serially on the highly tuned uniform 3.05 bpw pack.
Thatโs exactly why Iโm publishing the actual numbers.
๐ง๐๐๐ก ๐ฌ.๐ฏ.๐ฒ.๐ฎ ๐ฆ๐๐๐ฃ๐ฃ๐๐
I went back in with two more contributions:
โ INT8 / INT4 KV cache for Flash Next CUDA
About 1.7ร / 2.6ร more context capacity in the same memory, while indexer state stays BF16.
โ --mtp-confidence
So the CUDA server can control when an MTP draft chain stops.
โ And a nasty EXL3 stop-token bug.
Some Qwen EXL3 packs stored <|im_end|> only in generation_config.json. TensorFold wasnโt reading it, so replies could run past the end of the assistant turn and leak the next turn or think text.
The CUDA loader reads it now.
TensorFold is a real game-changer for the local inference community where my EXL3 experience and code are now part of its released CUDA stack, and seeing something I wrote move from a model-specific problem into infrastructure other model families can reuse is one of my favorite parts of open source.
Huge respect to @ashxhart the creator of the TensorFold project for reviewing, integrating and shipping the work. ๐
TensorFold 0.3.6:
https://t.co/rNfIcGOuju
TensorFold 0.3.6.2:
https://t.co/NpE6NbKiBi
Bro. Glm 5.3 is better than opus 4.6.
It's better than "the absolute best frontier model ever created" only a few months ago.
It IS frontier level.
Not to mention, frontier doesn't mean "the best" anyway. It's overused. Frontier just means they created their own model.
Laguna S2.1 is Frontier despite not being that great compared to Opus, Fabel, Astra, etc