turboderp

[turboderp @turboderp_]

._.

[turboderp @turboderp_]

That's a trace-driven benchmark running directly on ExLlamaV3. Also uses greedy sampling, it's the second run after warming up the sparse head, and many other caveats. But TabbyAPI would be the way to run it as an inference server: https://t.co/cE6f3ORmzq

These would be the most equivalent settings, though there are many more knobs to fiddle with:

warmup: true
draft_mode: mtp
draft_num_tokens: 7
hybrid_draft_tokens: 15
sparse_draft_head: 16384

[turboderp @turboderp_]

The scoring method is standard KL divergence. The test dataset is a set of prompts generated by the model itself from a set of test prompts. Here's what the test data looks like for this model:

https://t.co/MzYgKKYoQ6

The ground truth is not the test data, though, but rather the logits produced by the unquantized model (BF16 reference in HF Transformers) on the test data.

So the point of the self-generated trace is that you want to test the actual domain of the model, tokens that it would sample when deployed. Its vestigial wikitext2 or openwebtext circuits are irrelevant, and in some models completely unstable (quantizated or not).

The benchmarking harness is included in the exllamav3 repo (eval/qbench.py) if you want to inspect or reproduce. It ensures each model is tested on its appropriate backend (BF16 reference in Transformers, llama.cpp for GGUF models, etc.) while maintaining the same tokenization and KL scoring throughout, so the comparison is apples-to-apples across frameworks.

(Wish there was a way to get logits out of vLLM, cause Transformers is a very high-maintenance dependency, but we live to suffer, right?)

[turboderp @turboderp_]

Don't know if I should spend less time coding and more on farming engagement, but either way here's a neat cat simulation.

It's not entirely accurate (still needs to model viscosity and surface tension), but it was more of a stress test anyway, made by a new 1.66 bpw quant of Qwen3.8-Flash-Next running on one RTX 5090; no experts offloaded to the CPU, only vision tower in pinned memory (negligible overhead), and embeddings on disk.

Stable inference throughout, no reasoning loops, just smooth sailing at 200-300 t/s (and a lot of compacting since one 32 GB GPU still doesn't leave a lot of room for context, even with cache quantization).

Get yours today at https://t.co/TohB8b9DBc

[turboderp @turboderp_]

I don't have any numbers because the gap, while real, is too small to easily measure. It's usually below the noise level and only discernible as a trend across many quants. It is definitely in mul1's favor, though, as predicted by the distribution being much more Gaussian (codebook aligning with regularized tensors is a core assumption made by QTIP).

Ultimately the difference is probably too small to worry about, and the main motivation for defaulting to mul1 is the greatly improved performance on Ampere and AVX (about +50% GEMV throughput on an RTX3090), as well as the small boost on Blackwell. Ada is mostly fine with either codebook.

And yes, fractional-K is a very useful addition because K=2.5 will almost always perform better than a 2/3 mix, and even if you're dynamically sizing layers or individual tensors, doing so at finer intervals helps a great deal. But K=1.5 is the most significant upgrade, putting EXL3 roughly on par with methods like Bonsai2 which require finetuning. I'm currently testing a 1.66bpw Qwen3.8-Flash-Next quant that's performing exceedingly well on a single RTX5090 with no CPU offload.

[turboderp @turboderp_]

@IndepBets @grok @0xSero ROCm is supported in the master branch now. TabbyAPI should pull it automatically with its launch script, or you can run it in Docker.

[turboderp @turboderp_]

QTIP already layers trellis coding on top of LDLQ, which is a generalization of/improvement on GPTQ. QTIP just has more of a research-grade PyTorch implementation that makes it difficult and expensive to run.

So EXL3 is a streamlined version of the same thing with a different regularization scheme, a bunch of optimized kernels to make it practical, and extensions like fractional trellises and whatnot. But it's still the same two components doing the heavy lifting.

[turboderp @turboderp_]

Yep, dithering is a great analogy for the error-feedback mechanisms of GPTQ and LDLQ, but it probably undersells trellis quantization.

You don't end up with 2 bits per weight but with a procedural codebook of 65536 floating-point values drawn from a Gaussian distribution. Every weight gets a 16-bit index, and compression is achieved by letting it share 14 of those bits with other weights in the same trellis.

At 2 bpw, the search space is 2^512 possible encodings for every 16x16 tile, but you can search efficiently (and precisely) with the Viterbi algorithm, to find the optimal encoding that minimizes the RMS error across the entire tile.

Here's an illustration:

[turboderp @turboderp_]

@Greg_GL_87 @0xSero Also runs on AMD now. And CPU offloading is supported for sparse models, but you still need a CUDA or ROCm GPU.

[turboderp @turboderp_]

@subkleo @thesupermannx Community has had a year already with this method. There was also LittleBit-2 five months ago. They never released any models, and probably there's a reason for that.

[turboderp @turboderp_]

@NonlocalLlama Jev be like

[turboderp @turboderp_]

Is anyone else confused why everyone's all hyped about using "decision models" to solve deterministic problems, instead of leveraging AR models' ability to implement classical algorithms?

I challenge any Jev or Jev-like harness to compete with A* in 80 lines of JavaScript. I get the selling point of "you don't need to write any task-specific code, the decision model just figures it out." But in this case the cost was all up-front: four minutes with Astra-medium, and a one-line prompt. What am I missing?

[turboderp @turboderp_]

I would even go a little bit further than that. For the Snake demos, the model is given a neatly formatted string explaining which directions are blocked and which direction leads closer to the fruit, and then Jev translates that string into a distribution over three available directions. I've yet to see a demo of Jev showing any foresight when making that choice, or adaptation. It just picks the most obvious, direct route that isn't immediately blocked.

It seems to me, if you can already translate the board state (or whatever you're deciding on) into a well-defined format, Jev isn't just a redundant and inefficient translation layer, it also precludes a whole bunch of classical-AI algorithms that could have made much better use of that same data.

Like A* here, scanning the whole board, choosing when to stall so it doesn't box itself in later, even if it's right next to the objective. And it's doing it all in microseconds on a single CPU core.

Maybe there's a better use case for decision models, idk.

[turboderp @turboderp_]

It's not that there's anything wrong with the model, and it's not calling the benchmark results into question. It's just that whatever the model is doing, it's only using about 83% of its weights for it.

Needs some long-context benchmarks to fully verify, but the implication would be that the model could be reduced from ~1000B to ~830B parameters, making it that much faster and more compact.

[turboderp @turboderp_]

I doubt I'm the first to notice this, but MiMo-V2.6-Pro seems to have a lot of redundant weights.

If you remove the expert blocks from layers 1 through 12 and the attention blocks from layers 1-4, 6 and 8-11, end-to-end KLD is largely unaffected. Their contribution to the residual is on the order of BF16 rounding errors.

Overall, it looks like you can trim away 17% of the model's ~1T parameters and still end up with largely the same model. That's six whole Qwens.

Is this known, or can anyone confirm?

[turboderp @turboderp_]

I came across it while debugging quantization for the Pro model (before the MOPD update.) Measured zero quantization error on the first 13 layers at 1.5 bpw, which is quite unusual, so it led me down this rabbit hole.

Flash maybe doesn't have the issue at all, we'll see, but I'll rerun the tests with Pro-MOPD soon, which is the interesting case. Biggest bottleneck right now is disk space. ):

[turboderp @turboderp_]

@BrandonMusicKy That does look like much the same curve, yes. Though it's interesting that attention ends up with zero output as well on most of those layers.

So if you want to save 13% on storage and latency, just remove layers 1-4, 6 and 8-11 with a little tensor surgery.