llama.cpp merged something big for Localmaxxers whose AI models don't fit in VRAM. 👀🔥 (Strata has this too)
The upgrade is pure software and no expensive or hard-to-find new GPU needed.
PR #30966 merged November 7.
Here's what they've added.
When you run a huge Mixture-of-Experts (MoE) model that doesn't fit in GPU memory, llama.cpp can cache frequently used experts in VRAM.
But what happens when the model needs an expert that ISN'T in the cache?
Previously, the MoE cache path would upload that expert's weights from system RAM into GPU VRAM for execution.
That means transferring potentially large amounts of data across PCIe.
Now llama.cpp has another option.
Run some of those missing experts directly on the CPU! This is important especially when running models on eGPU (low bandwith thuderbolt)
No need to transfer their full weights into VRAM first.
Okay, so what?
The developer tested Qwen3.8-Flash-Next IQ4_NL with approximately 6.3GiB allocated to the GPU expert cache.
⚡ GPU uploads only → 24.2 tps
🚀 CPU handles 95% of cache misses → 47.6 tps
That's 96.7% FASTER decoding!
And an RTX 3090 user tested Qwen3.8-Flash-Next UD-IQ4_XS with a 5.5GiB GPU cache.
🐌 GPU uploads only → 19.5 tps
🚀 CPU handles 95% of misses → 31.8 tps
Approximately 63% faster on the same RTX 3090!
Now here's the clever part.
The developers also added an automatic tuning script.
It measures and evaluates combinations of:
💾 GPU expert-cache size
🧠 CPU execution share
⚡ Expert-cache hit rates
🔄 CPU/GPU transfer performance
And recommends settings for your hardware.
Because more CPU offloading isn't always faster.
With a larger ~19GiB GPU cache, the developer measured these
⚡ GPU uploads only → 65.5 tps
🚀 50% CPU miss share → 66.4 tps
🐌 95% CPU miss share → 54.1 tps
See what happened?
The best strategy changes depending on how much VRAM is available and how fast the CPU, GPU and memory can move and process data.
And to be clear, 95% CPU miss share does not necessarily mean the CPU runs 95% of the model.
It means the CPU handles approximately 95% of eligible GPU cache misses, while cached experts still execute on the GPU.
Your CPU and system RAM aren't just places to park model weights now. More regular RAM is better.
They can actively help execute the model when moving those weights would be slower.
And now this capability is merged into upstream llama.cpp.
Free. Open source. No new GPU required. 🔥
Link in ALT.
@LaurentLhardit@videolan Sinon coupons les câbles de télécommunications et restons entre nous. Cela simplifiera et évitera d'inventer une nouvelle taxe !
🇫🇷⛽️ FLASH | La France a baissé sa consommation de carburant de 30%, en 10 jours, à cause de la hausse des prix. Cela représente également une forte baisse des recettes fiscales pour le gouvernement.
J'ai menti. Ce n'est pas 3 sites, c'est 6.
- https://t.co/W3GifrkpDB, le portail des API de l'État. Le compte, les applications, les souscriptions, les clés.
- https://t.co/4hWo5gMvvB, le portail de test. Domaine séparé et compte séparé.
- https://t.co/EqnDqzOp4L, un site de plus, uniquement pour déclarer le raccordement de test.
- https://t.co/gAxhavvOgK, le vrai portail. La structure, les habilitations, le compte technique, le RIB.
- https://t.co/gRJKqSKVgl, encore un site à part, uniquement pour déclarer le raccordement de production.
- https://t.co/3zEGpViJhM, pour savoir si la collectivité en face exige un code service.
Et la qualification n'est pas optionnelle, c'est un prérequis officiel à la production.
Le meilleur détail : sur le portail de test, tu ne peux pas créer ta propre société.
Tu génères un jeu d'entreprises fictives, livré en CSV (genre 10min de génération de mémoire) avec leurs mots de passe, et tu déclares ton raccordement en étant connecté comme l'une d'elles.
J'ai fait des conneries dans ma vie mais c'est peut-être bien la pire...
🚀 New Jellyfin release: 12.0!
We are pleased to announce the latest stable release of Jellyfin, version 12.0! This major release brings many new features, improvements, and bugfixes to improve your Jellyfin experience. As always, please ensure you take a full backup before!
🚨 llama.cpp support for Qwen4 architecture / Qwen3.8 Flash Next is already in PR!
PR #27739 just dropped with full support for the new Qwen4 Exp architecture:
- Full Multimodal & MTP Support: Handles the vision encoder (--mmproj) and multi token prediction (--mtp).
- RAM Offloading: Uses -ot "ple_ngram_embd=CPU" to keep the massive 51B n-gram embedding table in host memory without choking your VRAM.
-256k context tested out of the box.
Quick run llama.cpp flags:
llama-server -m qwen4exp-Q8_0.gguf \
--mmproj mmproj-qwen4exp-F16.gguf \
-md mtp-qwen4exp-Q8_0.gguf \
-c 262144 -ngl 99 \
-ot "ple_ngram_embd=CPU"
My benchmarks on a single RTX 4090 (24 GB VRAM) drop soon!
Meet Qwen3-Coder-30B-A3B-Instruct, a game changer in code generation! With 30B total parameters but only 3B active, it's a MoE model that's fast, efficient, and perfect for devs. Over 4.6M downloads, it's taking the community by storm. Ready to see what it can do? #AI#CodeGen
llama.cpp now has an official website: https://t.co/vztdUpdBWL
Our goal is to make local AI accessible to everyone, and improving the user experience is a big part of that. On the new landing page you’ll find a single-line cross-platform installer. The installation provides a single unified `llama` entrypoint which you can use to run/serve models and interface with 3rd-party agentic applications.
While oriented towards simplified user experience, the new `llama` application also provides all the advanced functionality of the existing llama.cpp tooling with which experienced users are already familiar. Also note that all GGUF models that you might have already downloaded with llama.cpp in the past will be automatically available to use without downloading again (they are stored in the common HF cache on your machine).
We have many improvements in the pipeline both at the UX and at the engine level and we plan to iteratively ship new things over the coming months. One of the main focuses will be seamless integration with local-friendly 3rd-party agents (such as Pi). In the meantime, we’ll continue to listen for feedback from the community and adjust accordingly, so keep letting us know what you think and need.
@Mthys_mpg Tenant un budget précis de mes dépenses pour le foyer.
Les achats de nourriture pour le petit déj, repas du midi et du soir, en comptant donc les repas préparés à la maison et les repas extérieurs je dirais que pour une personne adulte, mon évaluation est de 300 euros/mois facile
You can find the full changelogs on the GitHub releases for the server repository (https://t.co/5NPVpaZ7KF) and the web repository (https://t.co/TmUnwr13xq).
Discuss further on our forums: https://t.co/e8cZP1LSKD
Every PNG image. Every ZIP file. Every PDF.
Every HTTP response your browser receives runs through his code.
One Frenchman built it in 1992. He never charged a cent. 🤯
Meet Jean-Loup Gailly 🇫🇷
> French computer scientist. Born 1956. Studied at École Polytechnique.
> 1980s ~ part of the Ada language implementation team.
> 1992 ~ built gzip alone. A free file compression tool for Unix.
> Released it to replace paid, patent-encumbered alternatives.
> 1995 ~ teamed up with Mark Adler. Built zlib together.
> zlib is a tiny library that compresses and decompresses data.
> Today zlib runs on billions of devices and core internet protocols worldwide.
> Every PNG image on the internet uses zlib to compress pixels.
> Every HTTP response your browser receives is typically gzip-compressed.
> Every Android app, Java program, Apache server, PDF, and Office file ~ uses zlib.
> Linux kernel uses it. Git uses it. Docker uses it. WhatsApp uses it.
> Made the entire codebase free. Permissive license. No restrictions.
> Made $0 from it. By choice. 🚀
> Won the 2009 USENIX STUG Award for gzip and zlib.
> 1999-2001 ~ CTO of Mandrake Linux. 2006-2014 ~ Tech Lead Manager at Google.
> Lives in France today. Plays Go. Still tinkers with compression code.
Every photo shared. Every page you open.
All running on the free code of one quiet genius.
Absolute Legend. 🐐