Local RAG in practice: Ollama, a vector store, a model

RAG exists because a model cannot read your files. Retrieval finds the passages that bear on a question and puts them in the prompt, so the model reasons over your material instead of its training memory. Each stage in that chain makes exactly one decision.

Chunking decides the unit of retrieval. Nothing smaller than a chunk can be returned, and nothing larger arrives whole. The default recipe is a recursive character splitter: cut on paragraph breaks, then sentences, then characters, targeting a few hundred tokens with a small overlap. That default is a fallback, not a design. Splitting on structure instead (Markdown headings, function definitions, speaker turns) produces chunks that are about one thing, and that property is what retrieval depends on.

Embedding decides what similar means. An embedding model maps a string to a fixed-length vector so that distance approximates relatedness in meaning. Locally, nomic-embed-text and mxbai-embed-large are the common choices, and both run acceptably on CPU, which matters because embedding is the one stage that runs over the entire corpus. Two constraints are easy to miss: the model has a maximum input length, past which text is silently truncated rather than rejected; and every vector in an index must come from the same model, so changing it means re-embedding everything.

The vector store decides how you search and what you can filter on. Underneath it is an approximate nearest-neighbour index, usually HNSW, that finds close vectors without comparing against all of them. Around it sits the part you use daily: metadata beside each vector, so a query can be restricted to one folder, one author or a date range before similarity is computed at all.

Retrieval decides what the model is allowed to see. Embed the question, take the nearest k chunks, optionally apply a similarity floor, optionally rerank with a cross-encoder that scores each candidate against the question directly. One default is worth stating plainly: a top-k retriever always returns k chunks, whether or not any is relevant. Relevance is imposed, not given.

Generation decides the wording. The passages go into the prompt with the question and an instruction to answer from them, cite the file behind each claim, and say so when they contain no answer. Llama, Qwen, Mistral and Gemma in quantised form all do this competently. It is the least interesting stage and by a wide margin the one people tune most.

At personal scale every option below works, and the choice is about operational shape rather than performance. The differences start to bite in the tens of millions of vectors, well past a folder of documents.

The framework decision is a different question, because the honest first row is that you may not need one.

Vector search is good at paraphrase and bad at exact tokens. Ask it about scheduling policy and it finds the page that says roster rules, without either phrase matching. Ask it for error code E4021, a ticket reference or a function name, and the embedding blurs the thing you were looking for, because rare identifiers carry little semantic signal.

Keyword search has the opposite profile. BM25, the ranking function behind most full-text search, scores on term frequency weighted by rarity, so a rare exact token is its strongest signal and a paraphrase is invisible to it. Hybrid search runs both and fuses the ranked lists, usually with reciprocal rank fusion, which scores a result by its position in each list rather than by incomparable similarity numbers.

Use hybrid when the corpus holds identifiers, product names, jargon, code or proper nouns, and when you cannot predict whether someone will type a concept or a string. That covers most technical document sets, which is why hybrid is a sensible default rather than an optimisation. Pure vector is enough when the material is continuous prose and the questions are conceptual, and it is cheaper to operate.

Plain grep beats both more often than the RAG literature admits, for a specific reason: retrieval ranks, grep enumerates. Given a distinctive string, rg -i postgres notes/ returns every occurrence in every file in milliseconds, with no index, no embedding model, and no chance the answer was ranked sixth when you asked for five. It also answers the negative case honestly. A retriever cannot tell you nothing matched, because it always returns something. Grep returning nothing is proof.

A workable rule: identifiers to grep, exact phrases to keyword search, questions to vector search, and a corpus that gets all three to hybrid retrieval with grep still beside it. Grep answers whether a word was ever written; retrieval answers what was concluded.

The case for local is strong, and stronger stated without exaggeration.

• Privacy that is structural rather than contractual. Documents and questions never leave the machine. With a hosted model, by contrast, the index can sit locally and every retrieved chunk is still transmitted on every query, so the sensitive material travels regardless of where the vectors live.
• Offline operation. The pipeline works on a plane, on a locked-down network, or anywhere outbound calls are not permitted.
• No per-token cost. Generation is free after the hardware, and so is embedding, which is where hosted setups surprise people: a one-off per document, but the document set is large and all of it must be redone when you change embedding models.
• No vendor dependency. No deprecated model version, no rate limit, no pricing change, no terms update. The weights are a file on your disk and will still run in five years.

Against that, four costs, all real.

• You need the hardware. A quantised 7B or 8B model fits on a machine with 16 GB of unified memory or a mid-range GPU; larger models want considerably more. CPU-only is fine for embedding, but generation drops to a speed that changes how you use the system.
• Quality trails hosted frontier models. Local models answer well from supplied passages, and remain behind on synthesis across many documents, on following long instructions, and on noticing that two passages contradict each other.
• Context is limited, often more than you think. Runners frequently default to a context window smaller than the model supports, and overflow is dropped rather than reported. Check the configured value before blaming retrieval.
• You now maintain an index. Every edited document is stale until something re-embeds it, and a changed embedding model invalidates all of it. That is a background job, a failure mode and a small piece of infrastructure you own.

RAG searches raw material at the moment you ask. The main alternative inverts that: let a model compile the sources ahead of time into structured, linked Markdown pages, and query those instead. Andrej Karpathy published that pattern as a gist in September 2024, and it is covered in LLM Wiki. Retrieval is cheap to build, complete (everything is reachable, including the aside nobody thought mattered) and weak at synthesis. Compilation is expensive, lossy by construction, and much better at questions of the form how did our thinking change.

They compose rather than compete: Karpathy's own design runs ordinary retrieval over the compiled pages, on the grounds that a clean corpus retrieves better than a noisy one. If you are starting today, start with retrieval. It needs no schema decisions, it fails visibly, and the index is not wasted if you add a compiled layer above it later. Models embedded directly in note systems are covered in AI-enhanced PKM, the case for keeping the stack on your own machines in local-first AI at work, the pipeline from spoken material to something queryable in from meeting to knowledge base, and the plain-text substrate under all of it in PKM for developers.