How to Build a RAG Agent That Actually Works in Production
Retrieval-augmented generation is easy to prototype and hard to productionize. Here's the architecture, chunking strategy, and eval loop we use.
Retrieval-augmented generation is the single most important pattern for building LLM systems that don't hallucinate on your own data. It's also the one most teams get wrong — usually by underinvesting in the retrieval half and expecting the generation half to compensate.
This is the RAG blueprint we use for production deployments at CapraZone, covering the pieces that matter and the ones you can skip.
The RAG loop, minus the buzzwords
RAG is three steps: turn a query into search terms, retrieve relevant chunks from your knowledge base, and feed those chunks + the original query to an LLM as grounded context. The output is a response the model couldn't have produced from its training data alone.
Every failure mode of RAG is one of these three steps producing garbage. Getting each right is what separates a working system from a very expensive autocomplete.
Chunking: the underrated art
Chunk too big and you dilute the signal; chunk too small and you lose context. The default advice — 500 tokens with 50-token overlap — is a fine starting point but you'll almost always improve on it with a per-document strategy.
For structured docs (PDFs with headings, HTML with semantic tags), chunk at heading boundaries. For code, chunk at function level. For chat transcripts, chunk by turn plus surrounding context. Store the original hierarchy as metadata so you can expand chunks upward when a match warrants it.
Embeddings and hybrid search
Pure dense embedding search (Cohere, OpenAI, Voyage) is great for semantic similarity and bad for keyword-specific queries — product SKUs, dates, proper nouns. Pure BM25 keyword search is the inverse. Every serious RAG system in 2026 uses hybrid: run both, merge with reciprocal rank fusion.
Add a reranker (Cohere Rerank, BGE) on the top 50 hybrid results and you'll typically double retrieval quality over dense-only for a marginal latency cost.
Prompting for grounded generation
Give the model retrieved chunks with clear delimiters and instruct it to cite the chunk ID for every claim. If a claim isn't grounded in a chunk, it doesn't ship. This one instruction eliminates the majority of hallucinations because the model literally cannot invent a citation.
For high-stakes deployments, run a second-pass grounding classifier that verifies every claim in the response maps back to a retrieved chunk. Reject responses that don't pass.
The eval loop nobody wants to build
The reason most RAG systems drift is that teams treat eval as "vibes." Build a labeled set of 100–300 query/expected-answer pairs from real user questions. Run it after every retrieval or prompt change. Track: retrieval recall@k, answer correctness (LLM-as-judge is fine here), and citation accuracy.
Without this loop you're guessing whether your changes helped. With it, you can ship weekly with confidence.
- Retrieval recall@10 — did we surface the right chunk?
- Answer correctness — does the generated answer match ground truth?
- Citation faithfulness — do citations actually support claims?
- Latency P95 — end-to-end response time
- Cost per query — inference + retrieval + storage
When RAG is the wrong tool
RAG is for questions answered from a knowledge base. It's the wrong architecture when the user wants an action performed (that's tool use), when the answer requires reasoning over the entire corpus (that's fine-tuning or long context), or when the corpus is small enough to fit in context (skip retrieval and pass the whole thing).
The strongest agents combine RAG for knowledge, tools for actions, and long context for holistic reasoning — orchestrated by a planner that knows which to reach for.
Failure modes we see repeatedly
The over-scoped version one. A team tries to cover every case in the first release, spends five months building, and ships something that is mediocre everywhere instead of excellent in one place. The counter is a wedge: pick the single highest-volume, lowest-risk category and be genuinely better than the status quo at it before touching anything else.
The missing evaluation set. Without fifty to two hundred golden cases with known-correct handling, every change becomes a vibe check and every regression ships. Build the eval set during discovery from real historical cases, including the ugly ones, and run it on every deployment. It is a day of work that pays back within a fortnight.
The undocumented process. Teams assume the current workflow is written down somewhere. It almost never is — the real rules live in the heads of two or three tenured operators. Interview them before you write a single instruction, and expect to discover legitimate exceptions that no policy document mentions. Those exceptions are usually where the actual customer value is, and encoding them badly is how how to build a rag agent projects lose trust in week one.
- Scope creep in version one
- No golden-case evaluation set
- No kill switch or rollback path
- Policy written without operator input
- Success measured on activity rather than outcomes
- No named owner after launch
How this connects to the rest of your stack
Nothing in this category delivers standalone value. The returns come from the connections: to the CRM that holds the commercial truth, to the ticketing or job-management system where the work lives, to billing, and to the data warehouse where you will eventually want to analyse all of it together. Plan those integrations as first-class scope with their own testing, not as a final-week task.
The most common ordering mistake is automating on top of broken data. If ownership, stage definitions or lifecycle statuses are inconsistent, an automated system will apply that inconsistency faster and at greater volume. Two weeks of data remediation before launch reliably beats two quarters of explaining anomalous outputs. Our revenue operations team usually runs that remediation in parallel with the build.
Think about the second and third use case while designing the first. If the ingestion, context and control layers are genuinely reusable, use case two costs a fraction of use case one — and that ratio is what turns a single project into a platform. Explore how we structure that on our solutions overview or start a scoping conversation through the contact page.
- CRM and system of record integration as first-class scope
- Data remediation before automation, not after
- Reusable ingestion, context and control layers
- A named second use case to validate reusability
Frequently asked questions
Which vector database should I use?
For most teams: Postgres with pgvector — the operational simplicity beats specialized DBs at anything under ~10M chunks. Above that, look at Turbopuffer, Pinecone, or Weaviate depending on your latency and filtering needs.
How large should chunks be?
Start at 500–800 tokens with 10–15% overlap. Then evaluate on your data. Documents with strong structure (headings, sections) benefit from chunking on those boundaries rather than fixed sizes.
Do I still need RAG with long-context models?
Yes, at scale. A 1M-token context is huge, but pushing your whole corpus through the model on every query is expensive, slow, and often less accurate than retrieving the top-k relevant chunks.
How often should I re-embed my knowledge base?
On document change (incremental) plus a full re-embed when you upgrade the embedding model. Never re-embed on a fixed schedule — it's wasted compute if nothing changed.
What is the smallest useful first version of how to build a rag agent?
A single high-volume category, handled in suggest-only mode on live traffic, with every human correction captured as a labelled example. That version is typically live in three to four weeks and already saves drafting time while it earns the data for autonomy.
How do we avoid getting locked into one model or vendor?
Keep policy, retrieval and orchestration in your own code and treat the model as a swappable component behind an interface. Maintain an evaluation set so switching is a measured decision rather than a leap of faith.
What does CapraZone actually deliver at handover?
Source code, infrastructure as code, the evaluation suite, the observability dashboard, runbooks for every failure mode, and a training session for the internal owner. You can operate it without us, and many clients do.
