Local RAG, clearly explained!
(bookmark this)
Most people debug RAG by swapping the model. That's usually the wrong place to look.
Here's the full pipeline I ran on my laptop:
→ Load: 3 plain txt files
→ Split: chunk size 500, overlap 50 (6 chunks total)
→ Embed: nomic-embed-text via Ollama
→ Store: Chroma, fully local
→ Retrieve: top 3 chunks per question
→ Generate: Gemma 4 E4B answers from only those chunks
The key insight: retrieval sets the ceiling, not the model.
If the right sentence never lands in those 3 chunks, no model can give you a grounded answer. A bigger LLM just hallucinates more confidently.
So when an answer is wrong, I check the retrieved chunks first. Nine times out of ten, that's where the problem is.
Local makes this easier. No API cost per test, so you can rerun retrieval as many times as you want.