A retrieval demo takes an afternoon. Keeping the same system accurate six months later is a different project.
Retrieval fails before the model does
When a RAG answer is wrong, the model is usually not to blame. The right passage never reached it. Chunks were split mid-table, the embedding missed the vocabulary the user typed, or the top matches were five near-identical paragraphs from the same page.
Before touching the prompt, look at what retrieval returned. If a person could not answer the question from those passages, no model will.
Chunking is a product decision
Fixed windows of 500 tokens are a starting point, not an answer. Contracts, tickets, transcripts and product docs each break differently. Splitting on structure, keeping headings with their body and storing the source path alongside every chunk pays for itself quickly.
Hybrid search matters more than most teams expect. Keyword matching catches part numbers, error codes and names that embeddings blur together.
Freshness quietly ruins trust
A confident answer from a policy that changed in March is worse than no answer. Track when each document was indexed, re-index on change rather than on a weekly cron, and remove deleted sources instead of letting them linger in the vector store.
If a source cannot be kept current, say so in the interface. Users forgive a caveat; they do not forgive being misled twice.
Citations are not decoration
Every answer should link to the passages it came from. It lets users verify in one click, gives support a way to debug a complaint, and gives you a signal to measure: how often people open the source, and how often they then disagree with the answer.
Measure with real questions
Collect the questions users actually ask, label the correct sources, and score retrieval separately from generation. When quality drops you then know which half moved.
This set is the most valuable artifact of the project. Models change every few months. A well-maintained evaluation set is what lets you swap one in over a weekend instead of a quarter.
