Mike Jerman | RAGShelf

I track every local RAG tool so you don't have to →

[Mike Jerman | RAGShelf @mkjerman]

Tired of uploading your private docs to the cloud just to ask them questions?

I built RAGShelf: 38 local & self-hosted RAG tools compared side by side, so you can chat with your own files offline. Free, no login.

https://t.co/rSCrpsXccP

[Mike Jerman | RAGShelf @mkjerman]

@Shane_H_Ward Nice setup. That keeps client data on your own machine, which is the main thing. If you add doc Q&A on top, are your files mostly scanned PDFs or plain text? Happy to suggest 2–3 tools that stay fully offline.

[Mike Jerman | RAGShelf @mkjerman]

Most local RAG setups don't run out of memory on the weights. They run out on the KV cache once you stuff in a long stack of retrieved docs.

FP8 KV cache is half the bytes of FP16, so more context fits on the same GPU without dropping to a smaller model.

https://t.co/PSwAKVV2pR

[David Hendrickson @TeksEdge]

New llama.cpp update.

Okay, so yeah, your local model fits in VRAM, but then the conversation gets longer. 💨

The model weights aren’t the only thing consuming memory. Its attention cache needs room too.

llama.cpp just merged FP8 KV-cache support.

Compared with FP16, FP8 uses half as many bits per cached value, which potentially frees memory for longer contexts or more simultaneous sessions on supported configurations.

... giving your hardware more room to work.

“But doesn’t LM Studio already have KV-cache quantization?”

Yes. Existing options such as Q8 already offer savings.

What’s new is another cache format?

Q8 stores the cached values as scaled integers. FP8 uses a floating-point representation. Different ways of compressing the attention data your model reuses while generating text.

That expands the options for running models locally. It doesn’t establish that FP8 is faster or more accurate than Q8.

This reduces cache memory, not model-weight memory. It won’t double your total VRAM or override a model’s context limit.

[Mike Jerman | RAGShelf @mkjerman]

@sergebl 7x on the same 3090 is a huge jump. Was the Ollama Nemotron the same 3.5 Lightning 30B checkpoint and quant, and is 450 tok/s one stream or several added together?

[Mike Jerman | RAGShelf @mkjerman]

@ScriptedAlchemy Nice find. NVIDIA Marketplace listed the 128GB/4TB at $6,950 in October, and launch MSRP was $3,999. On a single Spark, MoE models are the usual sweet spot because decode is capped by the 273 GB/s bandwidth. What are you planning to run first?

[Mike Jerman | RAGShelf @mkjerman]

@paularambles 671B at 4-bit on a 512GB M3 Ultra is one of the few desktop boxes that can actually hold that model. How many tok/s are you seeing in llama.cpp, and what's the context length you're actually using?

[Mike Jerman | RAGShelf @mkjerman]

Everyone’s building agent loops. Where does the company context live—and who controls it?

An agent needs access to documents, decisions, and customer history to be useful. That doesn’t mean every piece of company knowledge belongs in a hosted service.

A fully local setup can keep retrieval and generation on your hardware. You still need access controls, current documents, and answers you can check.

Before connecting another agent, decide what it should be allowed to read.

[Mike Jerman | RAGShelf @mkjerman]

How many times this week did you paste the same company background into ChatGPT?

A local RAG index can reduce that repetition by retrieving relevant passages from your docs.

You still need to keep the index current and check the answers. Keeping data on your machine also means using local parsing, embeddings, models, and storage.

I keep a directory of local and self-hosted RAG tools: https://t.co/D8KbGwnIAS

[Mike Jerman | RAGShelf @mkjerman]

@robotnurseio Short list, from published sources:

Official NVIDIA playbooks: Ollama + Open WebUI.

Also reported working: AnythingLLM, Dify, Docling (cu130 image), Haystack, LocalAI, LibreChat, Open Notebook.

RAGFlow: known issues — no upstream ARM64 image.

https://t.co/Hrg1S4op2x

[Mike Jerman | RAGShelf @mkjerman]

29 of the 39 local RAG tools I track still have no DGX Spark report.

10 do: 2 with official NVIDIA playbooks (Ollama, Open WebUI), 7 reported working, and RAGFlow with known ARM64 issues.

[Mike Jerman | RAGShelf @mkjerman]

@staroaeg Good test, and exactly the kind of question I'd want to see it handle. I haven't run that case myself yet. If you try it, I'd love to see what comes back.

[Mike Jerman | RAGShelf @mkjerman]

Tool of the week: PageIndex

What it is: an open-source "vectorless" RAG engine from Vectify AI. No vector database, no chunking. It's a Python SDK (pip install pageindex) with a local mode that needs no PageIndex server or API key.

The problem it targets: vector search ranks text by similarity to your question, and PageIndex's argument is that similarity isn't relevance, especially in long reports, contracts and manuals.

How it works (per the README):
• Index: it builds a table-of-contents style tree for each document, mostly from the PDF's layout. A basic model only summarizes the nodes.
• Retrieve: a stronger LLM reasons its way down the tree to the right section and pages, so answers trace back to explicit references.

The tradeoff: every question costs LLM reasoning, and the README says to use the best chat model you can afford. The quickstart uses an OpenAI key (the v0.5.1 notes say keyless OpenAI-compatible servers work too). The open-source version handles text-based PDFs only; OCR and scanned docs are Cloud features.

License: MIT
Stars: 40.3K on GitHub
Latest release: v0.5.3 (Nov 5, 2026)

Full breakdown on RAGShelf: https://t.co/lo3rYI978Q

[Mike Jerman | RAGShelf @mkjerman]

3 new tools on RAGShelf this week:

• PageIndex: vectorless RAG, an LLM reasons over each doc's table-of-contents tree
• WeKnora: Tencent's self-hosted knowledge base with RAG, agents and a wiki
• RAG-Anything: multimodal RAG from the LightRAG team

https://t.co/PhIv7jRO2B

[Mike Jerman | RAGShelf @mkjerman]

About 7 GB at 4-bit, and 19 tok/s on a 32 GB Mac mini M6 (LLMCheck's numbers).
Gemma 4 12B is a strong pick for local doc Q&A on a laptop: 256K context, Apache-2.0, and it takes text, image and audio input.

[Mike Jerman | RAGShelf @mkjerman]

If you want to chat with your own files without uploading them anywhere, do this:

1. Pick a local RAG tool (I compare 38 at https://t.co/o7IW3OMZ4b)
2. Point it at the folders you actually use, not a junk dump
3. Choose a model that fits the hardware you have
4. Keep the index on a disk you control

That's it. Your docs stay on your machine.

[Mike Jerman | RAGShelf @mkjerman]

Hot take: for searching your own docs, retrieval quality matters more than model size.

A small local model with good chunking beats a giant model fed bad context.

Cloud APIs win on raw intelligence. They lose the moment your files shouldn't leave your machine, or the context you feed them is sloppy.

I keep a living comparison of local / self-hosted RAG tools here: https://t.co/o7IW3OMreD

[Mike Jerman | RAGShelf @mkjerman]

@curxorai Right now it's mostly whether anyone has reported it, not a full boot and retrieval test. ARM64 container assumptions are where Spark bites. Next pass I'm splitting them into boots, retrieval works, and no report yet. Happy to take a Spark report on any of the 29.

[Mike Jerman | RAGShelf @mkjerman]

@robotnurseio Ollama and Open WebUI have real NVIDIA playbooks, so they're the safest start. Want the short list with Spark notes?

[Mike Jerman | RAGShelf @mkjerman]

Everyone's suddenly talking about AI bills.

$10K+ a month for some teams. Token costs spiking. People hunting for smaller models just to cut API calls.

The cheapest token is the one you never send to an API.

That's why I built a directory of local RAG tools: chat with your own documents without uploading them anywhere.

38 tools compared at https://t.co/o7IW3OMZ4b

If you already run something local, tell me what setup you're on.

[Mike Jerman | RAGShelf @mkjerman]

Full list with an Offline filter: https://t.co/PhIv7jSlS9

[Mike Jerman | RAGShelf @mkjerman]

35 of 39 local RAG tools I track can run fully offline.

The other 4 need a cloud call somewhere in the default path.

[Mike Jerman | RAGShelf @mkjerman]

Single / dual / quad map: https://t.co/Hrg1S4oWS5