Anbeeld

Running and benchmarking local AI. Building BeeLlama.cpp and other tools.

[Anbeeld @Anbeeld]

Currently my main project is BeeLlama, a fork of llama.cpp with more than 900 stars on GitHub as of now.

It started very personally. I wanted to run Qwen 3.6 27B on my single RTX 3090, natively on Windows, and run it fast. And what can make it fast? Right, speculative decoding. But at that point, llama.cpp supported neither MTP nor DFlash.

So this May I forked it and made DFlash work there, with more and more tok/s gains in every subsequent release. It also shipped TurboQuant/TCQ for extreme KV cache quantization.

Since then, upstream llama.cpp has implemented a full set of speculative decoding options, and TurboQuant proved outdated in my own benchmarks. So BeeLlama moved on from those features and evolved.

Today, as of v0.4.x, its main additions are KVarN (variance-normalized KV cache quantization by Huawei) and KV Cache Precision Tail (storing recent tokens in full precision while quantizing the rest).

The feature set changes, but the idea stays the same. Local LLMs should be available to as many people as possible, not just owners of 4×3090 setups and entire fleets of DGX Sparks.

Quantization is a compromise, but the alternative is not being able to run the model at all. So the answer is to make quantization more efficient and minimize the losses, allowing people to experience local LLMs in the best shape possible on any half-decent hardware.

I'm constantly working on improving BeeLlama further and developing its unique features, while closely following upstream llama.cpp for improvements of their own.

So thank you to everyone using it, and stay tuned for more.

[Anbeeld @Anbeeld]

@PeasantSmith @Alibaba_Qwen I hope you were not hallucinating, because these sizes are CRAZY good for local bros!

[Anbeeld @Anbeeld]

https://t.co/LbA3BDxDpv

[Anbeeld @Anbeeld]

@Alibaba_Qwen moved Christmas to August?

"Qwen3.8-Flash-Next is a multimodal MoE model built on the next-generation Qwen4 architecture. We are releasing these architectural improvements ahead of schedule so that the community can prepare for the upcoming full family of Qwen4 models."

[Anbeeld @Anbeeld]

@PeasantSmith @Alibaba_Qwen I might be missing something?

[Anbeeld @Anbeeld]

@shauseth Cave paintings

[Anbeeld @Anbeeld]

@Tech2Wild Yep, it's Xiaomi.

[Anbeeld @Anbeeld]

@BadWolf_63 @MiaAI_lab Based.

[Anbeeld @Anbeeld]

@jun_song Now fine tune Qwen 3.8 27B to use compressed chain of throught 😎

[Anbeeld @Anbeeld]

@Hikari_07_jp It worked. Thanks, X.

[Anbeeld @Anbeeld]

Dear X.

If I had a bit of fun in @Hikari_07_jp's replies, it doesn't mean half my feed should now be in Japanese.

Sincerely yours.

[Anbeeld @Anbeeld]

@tmuxvim La intelligence artificielle.

[Anbeeld @Anbeeld]

@jun1228909 3070 Ti + 3070 Ti = 6140 TiTi

[Anbeeld @Anbeeld]

@hibakod Everyone were silently abusing it already. Key word: silently.

[Anbeeld @Anbeeld]

@mehulmpt You know what they say: it's the earliest it would ever be.

[Anbeeld @Anbeeld]

At this point we have like 20x more abliterated models than benchmarks of abliterated models. Makes it a tiny bit confusing.

[Anbeeld @Anbeeld]

@ollama I'm so nostalgic about April. That was the time of nearly infinite limits. You can see a large bump there – yeah, that's me buying a second account.

Now you guys all caught up and ruined it...

[Anbeeld @Anbeeld]

@jtregunna All these "I don't have a cat" setups make me jealous in that particular regard.

[Anbeeld @Anbeeld]

@0xSero GLM feels like riding a goat more often than I would like to.

[Anbeeld @Anbeeld]

@thepanta82 There are more models other than Claude, FYI.