Loading
Preparing your experience...
Loading
Preparing your experience...
Retrieval-augmented generation sounds simple in theory. Here's what actually makes it work in production environments with real business constraints.
Retrieval-augmented generation has become the default architecture for building AI systems that need to answer questions grounded in private data. The concept is elegant: instead of fine-tuning a large language model on your documents, you retrieve relevant chunks at query time and feed them to the model as context. In a demo, it looks magical.
In production, it falls apart for reasons that have nothing to do with the language model. After deploying RAG systems across healthcare, financial services, and e-commerce, we have identified the failure modes that separate production-grade systems from impressive prototypes. This article covers the practical lessons — the decisions that actually matter when your RAG system needs to work reliably at scale.
The way you split documents into chunks is the single most impactful decision in a RAG pipeline, and it is the one most teams get wrong. The default approach — splitting by character count or token limit — creates chunks that break mid-sentence, split tables across boundaries, and lose the semantic structure of the original document.
In our healthcare RAG deployment, we moved from fixed-size chunks to semantic chunking based on document structure (headers, sections, paragraphs) and saw retrieval precision improve from 71% to 89% without changing any other component. The key insight is that chunks should represent complete thoughts or concepts, not arbitrary text spans.
For structured documents like clinical protocols or financial reports, we use hierarchical chunking: each chunk retains metadata about its parent section, document, and position. This lets the retrieval layer filter by document type and section before doing semantic search, dramatically reducing false positives.
The MTEB leaderboard is useful for initial orientation, but leaderboard performance on academic benchmarks does not predict performance on your specific domain and query patterns. We have seen cases where a smaller, domain-adapted model outperforms the top-ranked embedding model by 15+ percentage points on real queries.
The evaluation that matters is retrieval precision on your data with your queries. Before committing to an embedding model, build a test set of 200-500 real queries with known relevant documents and measure precision@5 and recall@10. This takes a day of work and saves months of debugging the wrong problem.
For most business applications, we recommend starting with a strong general-purpose model (like BGE-large or Cohere embed-v3) and only investing in domain adaptation if your test set reveals systematic failures on domain-specific terminology. Fine-tuning embeddings on your data is powerful but expensive — make sure you actually need it before committing.
Pure vector similarity search works well for semantic queries ('What is our policy on remote work?') but fails on exact-match queries ('What is the threshold for SOC-2 Type II compliance?'). In every production RAG system we have built, hybrid retrieval — combining dense vector search with sparse keyword search (BM25) — outperforms either approach alone.
The implementation is straightforward: run both searches in parallel, normalise the scores, and use reciprocal rank fusion (RRF) to combine the results. In our experience, a 60/40 weighting of semantic to keyword search is a reasonable starting point, but the optimal ratio depends on your query distribution.
We also implement query classification as a pre-retrieval step. Queries that look like they need exact matches (containing specific codes, numbers, or proper nouns) get higher keyword weight. Conceptual queries get higher semantic weight. This adaptive approach consistently outperforms static weighting.
The retrieval layer determines what information the model sees. The generation layer determines what it does with that information. Both need explicit guardrails, but generation guardrails are where most teams under-invest.
Our standard approach includes three layers of protection. First, prompt engineering that explicitly instructs the model to only answer from provided context and to say 'I don't have enough information' when the retrieved chunks do not contain the answer. Second, a citation requirement: every claim must reference a specific chunk, and we validate that the cited chunk actually supports the claim. Third, a confidence scoring mechanism that routes low-confidence answers to human review rather than presenting them as authoritative.
In our healthcare deployment, these guardrails reduced hallucination rates from 8.2% (with basic prompting) to 0.3% (with the full guardrail stack). For a system answering clinical queries, this difference is the line between a useful tool and a liability.
You cannot improve what you cannot measure, and RAG evaluation is notoriously difficult because there are multiple failure modes at different stages of the pipeline. We evaluate at three levels independently.
Retrieval evaluation measures whether the right chunks are being retrieved. We use precision@k (are the top k results relevant?) and recall@k (are all relevant chunks in the top k?). This requires a labelled test set of queries mapped to their ground-truth relevant documents.
Generation evaluation measures whether the model produces correct, faithful, and complete answers from the retrieved context. We use a combination of automated metrics (faithfulness scoring with a judge LLM) and human evaluation on a random sample.
End-to-end evaluation measures whether the system actually answers user questions correctly. This is the metric that matters most, but it requires ongoing investment: we continuously sample production queries and have domain experts rate the quality of responses.
Deploying a RAG system is not the finish line — it is the starting line. Production RAG systems degrade over time as the underlying knowledge base changes, query patterns shift, and new edge cases emerge. You need monitoring that catches degradation before your users do.
We implement four categories of production monitoring. Performance monitoring tracks latency, throughput, and error rates. Quality monitoring samples production queries and automatically evaluates retrieval and generation quality against baseline metrics. Usage monitoring tracks which topics and document types are queried most frequently. Drift monitoring detects changes in query patterns, retrieval score distributions, and answer length distributions that might indicate systematic issues.
The most valuable operational practice we have adopted is a weekly review of flagged interactions — queries where the system expressed low confidence, where users gave negative feedback, or where automated quality checks detected potential issues. These reviews consistently surface improvement opportunities that automated monitoring alone would miss.
RAG systems have three primary cost drivers: embedding generation (one-time for indexing, ongoing for queries), vector database hosting, and LLM inference for generation. At scale, these costs compound quickly.
The most effective cost optimisation we have implemented is tiered retrieval: use a fast, cheap embedding model for initial candidate retrieval (top 50), then re-rank with a more expensive cross-encoder model to select the final top 5 chunks. This gives you the quality of expensive models at a fraction of the cost because the expensive model only processes a small set of candidates.
For LLM inference costs, consider using smaller models for routine queries and routing complex or high-stakes queries to more capable models. In one deployment, we found that 70% of queries could be answered accurately by a smaller model, reducing average inference cost by 60% with no measurable quality degradation on those query types.
If there is one lesson we would emphasise from building RAG systems across multiple industries and scales, it is this: start with your evaluation framework, not your architecture. Build a test set of real queries with ground-truth answers before you write a single line of retrieval code. This test set becomes the foundation that lets you make every subsequent decision with confidence.
The teams that struggle with RAG are almost always the teams that skipped evaluation and jumped straight to building. They end up debugging symptoms ('the answers seem wrong sometimes') instead of diagnosing root causes ('retrieval precision drops to 45% for queries about regulatory requirements because the chunking strategy breaks compliance documents').
Production RAG is not a weekend project. But with the right evaluation framework, systematic architecture decisions, and proper operational practices, it is one of the highest-ROI AI investments a company can make. We have seen it transform information access from a bottleneck into a competitive advantage — but only when the engineering is treated with the same rigour as any other production system.
Written by the StarkLabsAI engineering team, drawing on over a decade of experience building production AI systems across healthcare, financial services, and e-commerce.
Our team can help you apply these insights to your specific business context. Book a strategy call to discuss your needs.
Book a Strategy Call