the mispeled witch

Cute nerd no one listens to | 30s | I accidentally my gender, is this bad? | she/her

[the mispeled witch @smol_computer]

@Eschalabs I was getting full re-PP (no context caching at all) every request with a simple Pi install. Too slow to be useful. It sounds like it should be caching though? I'll see if I can find the issue.

FP8 was fine though, seemed to have no issues.

[the mispeled witch @smol_computer]

@Eschalabs Very cool, I'm running it to test. How does KV quantization affect your accuracy that you've found? Some of these "magic" low-bit quants suffer if they get a quantized KV cache too.

In testing: I can say that this seems to think SUBSTANTIALLY less than UD_IQ4_XS.

[the mispeled witch @smol_computer]

@Hesamation Can't serve from SSD though, so even with all those expert tricks, still can't run deepseek unless you can fit the whole thing in RAM. That's the only thing that'd make me switch.

[the mispeled witch @smol_computer]

@MiaAI_lab Yeah, Ornith 1.0 looked better _on paper_ than 35B as well and it was dog water compared to base 35B for my normal workload.

[the mispeled witch @smol_computer]

@populartourist Yeah apparently the community is discovering new ways to pack 27B into smaller and smaller spaces, and as a 16GBer I'm here for it.

I've been running it since my post and I can say it's pretty similar, I do think it stumbles a bit less but that's expected with +0.5bpw.

[the mispeled witch @smol_computer]

@mudler_it Oh man this is tight on 16 GB. `-c 81920 -ctk q5_0 -ctv q4_1 -ub 128` is fitting but it's questionable vs the `-c 98304 -ctk q8_0 -ctv q5_1` I get with the Ridge quant (3.68 bpw vs your 4.08). I'll have to test this, thanks for providing it.

[the mispeled witch @smol_computer]

@ArdaTugsat Try running it _without_ MTP. I get ~35-40 T/S w/o MTP on my 4080 super with -c 98304 -ctk q8_0 -ctv q5_1 (Must have compiled Llama.cpp with "-DGGML_CUDA_FA_ALL_QUANTS=ON" - EXTREMELY important or it will use CPP for k/v with split KV quant and tank T/S)

[the mispeled witch @smol_computer]

@frankb737459085 Gated Delta Net - it's the architecture inside Qwen models.

[the mispeled witch @smol_computer]

@EmperoAI As an APEX quant enjoyer for MOE models, I am actually surprised we haven't seen more mixed quants for dense, and I hope that this method becomes as popular for dense models where some layers are sensitive.

I hope to see more of these and a refinement of the methods.

[the mispeled witch @smol_computer]

@EmperoAI After using it for a few hours and having it explore/recreate the path you used to make it, I can safely say it makes connections and corrects its own mistakes, which is honestly the highest praise I can give.

It stumbles sometimes and overthinks but it's extremely coherent.

[the mispeled witch @smol_computer]

@EmperoAI Yeah I saw that, just was asking about KLD. This thing is flying though, seems smart still, good work. Thanks for publishing it!

[the mispeled witch @smol_computer]

@EmperoAI Hey out of curiosity have you done the KLD vs bf16?

[the mispeled witch @smol_computer]

@sudoingX Yeah this one is _too_ tiny. I want a better Qwen 35A3B-class model, not... whatever this is. It's a cool idea though!

[the mispeled witch @smol_computer]

@MiaAI_lab Pi > everything else for me, I can't spare the context space on local models for the random gunk from the others.

[the mispeled witch @smol_computer]

@alexhillman Yeah it's the actor model again. Always love it, it's a big hit. This time its niche is persistence. Good stuff, I'm certainly never getting tired of more actor model stuff.

[the mispeled witch @smol_computer]

@whoahyi @theodorvaryag This plus "Streaming expert weights from SSD" is the other one I'm shocked is not already in there. Yes there are forks and other projects that do that, I'm shocked it's not in Llama specifically already.

[the mispeled witch @smol_computer]

@Alibaba_Qwen Yeah, nah, my 16GB can't fit 27B. The MoE models are where it's at. Legit don't care about 27B with how it runs with CPU offload unfortunately.