Unstable Llama

[turboderp @turboderp_]

Don't know if I should spend less time coding and more on farming engagement, but either way here's a neat cat simulation.

It's not entirely accurate (still needs to model viscosity and surface tension), but it was more of a stress test anyway, made by a new 1.66 bpw quant of Qwen3.8-Flash-Next running on one RTX 5090; no experts offloaded to the CPU, only vision tower in pinned memory (negligible overhead), and embeddings on disk.

Stable inference throughout, no reasoning loops, just smooth sailing at 200-300 t/s (and a lot of compacting since one 32 GB GPU still doesn't leave a lot of room for context, even with cache quantization).

Get yours today at https://t.co/TohB8b9DBc

[Unstable Llama @NonlocalLlama]

@ai_hakase_ Thank you for sharing this! Let me know if you try it out.

[Unstable Llama @NonlocalLlama]

@xMr_Unoriginalx I am! ;) thanks for spreading my repo, are you going to try it?

[Unstable Llama @NonlocalLlama]

It is now possible to train a LoRA over exl3 quants!

https://t.co/TRiBHvlm7m

https://t.co/vfmOHcUZQV

@0xSero
@turboderp_

[Unstable Llama @NonlocalLlama]

@turboderp_ “Old cat yells at cloud”

[turboderp @turboderp_]

I doubt I'm the first to notice this, but MiMo-V2.6-Pro seems to have a lot of redundant weights.

If you remove the expert blocks from layers 1 through 12 and the attention blocks from layers 1-4, 6 and 8-11, end-to-end KLD is largely unaffected. Their contribution to the residual is on the order of BF16 rounding errors.

Overall, it looks like you can trim away 17% of the model's ~1T parameters and still end up with largely the same model. That's six whole Qwens.

Is this known, or can anyone confirm?

[donald @donaldjewkes]

I made this with one prompt using Opus 5.5

I spoke to my computer for 5mins, claude worked for 12 hours, and I woke up to this

full prompt:

[Unstable Llama @NonlocalLlama]

@yume_arasaki Great work, thank you! Come say hi at the exllama discord https://t.co/XWKwOrS09W

[Unstable Llama @NonlocalLlama]

This forces us to either accept artificially low traffic, or split into ugly single-quant repos. Do you have any ideas on how to fix this?

3/3

[Unstable Llama @NonlocalLlama]

Hey @ClementDelangue & @julien_c, the exllamav3 community is running into a limitation with how HF tracks downloads.
exl3 preserves transformer file conventions, and doesn't package everything into a single file like GGUF.

1/3

[Unstable Llama @NonlocalLlama]

The problem is that downloads from sub directories or branches of repos are not tracked.
We still appear to get some downloads, because downloads to the top branch are counted, but upon A/B testing it looks like ~5% of our real numbers.

2/3

[Unstable Llama @NonlocalLlama]

I just published a short essay questioning how we culturally frame entropy, and whether it has been mistaken for ontology.

Come share your thoughts!

https://t.co/40XwgcS2HH