The model sees only what retrieval hands it
A frontier model cannot reason about a clause it never received. If the retriever returns three chunks and the answer lives in a fourth, the model will either refuse or synthesise something plausible from what it has. Upgrading the model changes the fluency of the wrong answer, not its correctness. Diagnose retrieval before touching generation.
Fixed-size chunking destroys structure
Splitting every document at 512 tokens will cut a table in half, separate a heading from the clause it governs, and strand a cross-reference from its target. We chunk on document structure instead — headings, clauses, list boundaries, table units — with a small overlap and the parent heading path prepended to every chunk so context survives the split.
Metadata is half the retrieval quality
Effective date, jurisdiction, document type, product line, owning department, supersession status. Without these, a query about current policy happily retrieves the 2019 version. We extract metadata at ingestion and filter on it at query time, which removes a whole class of confidently-wrong answers that no amount of prompt engineering fixes.
Hybrid search, then rerank
Pure vector search misses exact identifiers — part numbers, error codes, statute references — because embeddings blur them. Pure keyword search misses paraphrase. Run both, fuse the results, then apply a cross-encoder reranker over the top 40 to produce the final 5. In our engagements this single change typically moves answer accuracy more than any model swap.
Measure retrieval separately from generation
Build two evaluation sets. The first asks: did the correct chunk appear in the top K? That is a retrieval metric, and it isolates the part you can fix with engineering. The second asks: given the correct chunks, was the answer faithful and complete? Teams that only measure end-to-end quality cannot tell which half to work on, and usually work on the wrong one.
What we check first, in order
Recall at K on a golden question set. Chunk boundary sanity on a random sample read by a human. Metadata completeness and freshness. Presence and effect of reranking. Only after those four are healthy do we evaluate whether the model tier is actually the constraint. It rarely is.