Retrieval-augmented generation

The pipeline has an offline half and an online half. Offline, source documents are split into chunks, embedded into vectors and indexed. Online, the query is embedded, the nearest chunks are retrieved, and they are placed in the prompt with an instruction to answer from them. Hybrid retrieval, combining vector similarity with keyword search, usually beats either alone because exact terms like product codes are precisely what embeddings handle worst.

Chunking is the decision that most affects quality and it is easy to get wrong. Chunks that are too small lose the context that made a passage meaningful; chunks that are too large dilute the embedding and crowd the prompt. Splitting on document structure — sections and headings — generally outperforms fixed-size windows, and keeping a small overlap prevents an answer from being cut in half at a boundary.

Voice adds constraints on both ends. Retrieval latency lands inside the response gap, so an index that is fast enough for a chat interface may be too slow for a call. And retrieved passages are written to be read: an agent that recites a knowledge-base article verbatim is unlistenable, so the prompt must require a short spoken answer rather than a summary of the document.

RAG reduces hallucination but does not eliminate it. If retrieval returns nothing relevant, the model may still answer from general knowledge unless it is explicitly told not to, and the retrieved passage may itself be out of date. Retrieval quality is worth evaluating separately from answer quality, because a good answer over a bad passage is luck.