Building a RAG System That Survives Production: Chunking, Retrieval and Evaluation

A RAG prototype takes an afternoon. Load documents, embed them, search, stuff the results into a prompt. It will look impressive immediately.

Then you point it at ten thousand real documents and accuracy collapses. Nothing broke — the naive version was always fragile, and small test sets hide it. Here is what actually determines whether retrieval holds up.

Decision 1: chunking

How you split documents matters more than which embedding model you choose, and it gets a fraction of the attention.

Fixed-size chunking — every 500 tokens with an overlap — is the default and the reason many systems underperform. It cuts through the middle of tables, separates a heading from the paragraph explaining it, and splits a clause from its exception.

• Structure-aware chunking: split on headings, sections and list boundaries. A chunk should be a complete idea.
• Keep the hierarchy: prepend the document title and section path to each chunk so an isolated paragraph still carries its context.
• Handle tables separately. Flattened into prose they retrieve badly and read worse.
• Match chunk size to question shape. Specific factual questions want small chunks; "how does this process work" wants larger ones.

Decision 2: retrieval

Pure vector search fails on exactly the queries that matter most in business contexts: product codes, error numbers, policy references, people’s names. Embeddings capture meaning, and an invoice number does not have meaning — it has an exact form.

Hybrid retrieval solves it. Run semantic search and keyword search together, then combine the results.

1. Semantic search for conceptual questions — "what is our refund policy for damaged goods".
2. Keyword or BM25 search for identifiers, codes and exact phrases.
3. Fuse the two result sets, then rerank the combined shortlist with a cross-encoder.
4. Return fewer, better chunks. Ten mediocre chunks produce worse answers than three strong ones and cost more.

Reranking is the highest-return addition most systems are missing. Retrieve broadly, then score the candidates properly against the query and keep the top few. It costs one extra step and routinely moves accuracy more than swapping embedding models does.

Decision 3: evaluation

This is the one teams skip, and it is the one that determines whether the system is improvable.

Without an evaluation set you cannot tell whether a change helped. You are making adjustments and forming impressions. Every RAG system we have seen decay in production decayed because nobody could measure it.

• Build a set of 50 to 200 real questions with known correct answers. Take them from actual support tickets, not imagination.
• Measure retrieval separately from generation. If the right chunk was never retrieved, the model was never going to answer correctly, and prompt tweaks are wasted effort.
• Track: was the correct chunk in the results, at what rank, and did the final answer use it.
• Re-run the whole set on every meaningful change. Chunking, embeddings, prompts, model version.

Retrieval accuracy is the ceiling on answer accuracy. No prompt recovers information the retriever never returned.

The failures that only appear at scale

• Near-duplicate documents — three versions of the same policy, and retrieval confidently returns the outdated one.
• No recency signal, so a 2021 procedure outranks its 2026 replacement because it matches the wording better.
• Permissions ignored, so the system cheerfully surfaces content the asker should not see. This is a data breach with a chat interface.
• No refusal path, so when nothing relevant is retrieved the model answers from general knowledge and sounds exactly as confident.

What a solid RAG stack contains

1. An ingestion pipeline that re-indexes when source documents change, rather than a one-off import.
2. Structure-aware chunking with hierarchy preserved.
3. Hybrid retrieval plus reranking.
4. Permission filtering applied at query time, not after.
5. Citations back to the source, so answers are checkable.
6. A refusal threshold.
7. An evaluation set that runs on every change.

None of that is exotic. It is the difference between a demo that impresses in a meeting and a system your team relies on without thinking about it.

---

If you have a RAG system that worked in testing and is quietly getting things wrong in production, the diagnosis is usually retrieval rather than the model. We are happy to look at it.