RAG that works in production — not just in the demo
Citations, permissions, evaluation, and human review: how we scope retrieval-augmented generation so teams keep using it after week two.
RAG demos are easy. RAG in production is a product problem. Retrieval-augmented generation looks magical when you paste three PDFs into a notebook and ask a question. It looks fragile when the corpus is messy, permissions matter, answers must be cited, and a wrong summary can reach a customer or a student record. In 2026 the teams winning with RAG are not chasing the newest model name — they are scoping retrieval, evaluation, and review like any other feature.
Start with the decision the model assists. Who is the user? What are they allowed to see? What happens if the answer is wrong? If you cannot answer those, you are building a chat toy, not a workflow. RAG over private corpora needs role-aware retrieval: the model should only see chunks the user is allowed to access. Skipping permissions is how demos become incidents.
Citations are not optional for trust. Operators need to see where an answer came from — document, section, timestamp — so they can challenge or approve it. Without grounding you can explain, “human-in-the-loop” becomes rubber-stamping. Pair citations with a review screen: edit, reject-with-reason, and an audit trail you can export.
Evaluation beats vibes. Build a small golden set of questions and expected behaviours. Measure retrieval hit rate, answer faithfulness, and how often humans override. Use override taxonomy (wrong context, outdated doc, hallucinated detail, tone, other) so week-twelve you know what to fix — chunking, access filters, or prompts — instead of guessing.
Cost and latency are product constraints. Naive “embed everything and stuff the prompt” dies under real traffic. Chunk deliberately, cache where safe, limit context windows, and put human review before anything is saved to a system of record. Quiet AI — fewer clicks, faster first drafts, clear handoffs — beats a flashy box nobody opens twice.
At Mechabits we scope RAG as a product slice: corpus, permissions, citations, eval set, review UI, and independent logs. If you are adding “chat with our docs” to a real product, start with the workflow and the trust bar — not with a vendor pitch deck. That is how RAG survives production.