Local RAG: querying your meeting history with AI

Nothing here is exotic; the stack has stabilized over the last two years and each piece is swappable.

• Documents — your transcripts. One file per day, timestamped and speaker-labelled if you have it.
• Chunking — split into passages of a few hundred tokens with some overlap. The most consequential and least discussed step; see below.
• Embedding model — turns each chunk into a vector. nomic-embed-text or mxbai-embed-large via Ollama both run acceptably on CPU.
• Vector database — ChromaDB is the simplest to start with and Python-native; Qdrant is faster and more production-shaped; LanceDB is a good middle. At the scale of one person's meetings, any of them is fine.
• Retriever — embeds the question, returns the top-k nearest chunks.
• The model — Llama, Qwen, Mistral or similar through Ollama or llama.cpp, answering from the retrieved chunks.

LangChain or LlamaIndex will wire these together; for a personal corpus, LlamaIndex is the lighter fit. Open WebUI is the shortest path to something usable — it sits on Ollama, has document upload and RAG built in, and gets you to a working chat over your transcripts in an afternoon without writing the pipeline yourself.

Most RAG tutorials assume documents: structured, edited, with paragraphs that are about one thing. Speech is none of those, and four adjustments matter more than the choice of vector store.

• Chunk by turn or topic, not by character count. A fixed 500-character split will cut a decision in half and pair the reasoning with the wrong conclusion. Speaker turns are a natural boundary, and grouping consecutive turns until you hit a size limit works well.
• Put the metadata in the chunk text. Date, meeting title, participants — prepended to each chunk, not just stored beside it. Otherwise a retrieved passage arrives with no way for the model to know when it was said, and time is the axis that matters most here.
• Expect a low signal ratio. An hour of speech might hold three minutes of durable content. Retrieval will surface confident-looking passages of small talk; retrieving more chunks makes this worse, not better.
• Add a date filter, and use it. "What did we decide last quarter" is a metadata query with a semantic component. Pure vector similarity has no idea what "last quarter" means, and this is the single most common disappointment people report.

One more, easy to miss: transcripts contain errors. Whisper misrenders names and technical terms, and a wrong word is embedded as confidently as a right one. Keeping the original file linked from every answer is what lets you notice, which is the same argument as everywhere else on this site — the record stays the authority, the summary does not.

The advantages of running locally are real and the costs are also real. Both deserve stating plainly.

For meeting transcripts the first row usually decides it, and not on ideological grounds. A transcript contains other people who did not consent to a cloud provider, frequently material covered by an NDA, and in some sectors data with a legal location requirement. "We index our meetings into a vector database and query them through an external API" is a sentence that has to survive a security review. The local version does not need one.

Embedding is also the step where cloud costs surprise people: it is a one-off per document but the document set is large, and it must be redone if you change embedding models. Locally that decision costs an evening of compute rather than a bill.

The other pattern for this material compiles the transcripts ahead of time into an agent-maintained knowledge base, and then queries that instead — Karpathy's LLM Wiki, covered in LLM Wiki from meeting transcripts. They solve overlapping problems with opposite cost shapes.

• RAG is cheap to build and complete. Everything ever said is reachable, including the aside nobody thought was important. Nothing is lost to a summarizer's judgment.
• A wiki is expensive to build and better at synthesis. "How has our position on this evolved" is a question retrieval answers badly — it returns chunks — and a compiled page answers well, because assembling the history is what compiling is.
• They compose. Karpathy's own design runs retrieval over the wiki rather than the raw sources. A practical setup: RAG over the transcripts for "did anyone ever mention", a compiled layer for "what do we think about".

If you are starting, start with RAG. It requires no schema decisions, it fails visibly rather than silently, and the index you build is not wasted if you add a compiled layer later.

All of this assumes a folder of transcripts that is complete and machine-readable. Incomplete is the usual failure: an assistant that only recorded the calls someone remembered to start produces a corpus with holes exactly where the informal conversations were, and retrieval over it returns confident answers drawn from a biased sample — the worst possible outcome, because nothing signals the gap.

Earkeep is built to produce that corpus as a byproduct. It records mic and system audio continuously while it runs and transcribes locally on your device with whisper.cpp, so nothing depends on remembering to start it — no bot to invite either. Meetings are defined afterwards by selecting a span on a timeline, or automatically from a connected calendar. Audio is processed only in memory and never written to disk, and each day is written as a plain .jsonl file with timestamps in a folder you choose — which is already the ideal input for a chunker, and is equally readable by grep on a day you do not want to run a model at all. See private by design for how the local pipeline works, and agents and MCP for the agent path that skips the vector store entirely when a coding agent can just read the files.

The broader case for keeping the whole stack on your own hardware is in local-first AI at work; the practical patterns for handing transcripts to an agent are in feeding meeting context to AI agents; and the end-to-end pipeline from a spoken sentence to a queryable knowledge base is in from meeting to knowledge base.

Download Earkeep free for macOS and start accumulating the transcripts your retrieval stack will need.