Retrieval & grounding evaluation harness

RAG systems are unusually easy to improve by accident and unusually easy to break by accident, and both feel identical from the inside. You change a prompt, swap an embedding model, adjust chunk size, add a reranker — and the way almost everyone checks the result is to run a few favourite questions by hand and see whether the answers look better.

They will look better. Language models produce fluent text under nearly all conditions, and a handful of hand-picked questions is a sample you chose because you already knew the system handled them. That method cannot detect a regression, which is the only thing it needed to be able to do.

The harness exists so that quality claims about these systems are measurements. It is also, not coincidentally, why every number on this site is a specific one.

Each knowledge base gets its own golden question set: a fixed collection of questions with known correct sources, built to cover the ordinary cases, the edge cases, and — importantly — the questions the corpus genuinely cannot answer.

That last category is the one most evaluation sets leave out, and it is where the interesting failures are. A system that answers every in-corpus question well and also confidently answers questions it has no basis for is not a good system, and a test set made only of answerable questions will never say so.

Scoring the final answer alone tells you something is wrong without telling you where, and retrieval failures and generation failures need completely different fixes. So the two stages are measured independently.

Retrieval is scored on hit-rate — whether the right source made it into the candidate set at all — on precision@k, meaning how much of what was handed to the generator actually bears on the question, and on MRR, which cares about how high the right source ranked rather than merely that it appeared. A pipeline can hold a good hit-rate while its precision quietly collapses, and only separating them shows it.

Generation is scored on groundedness, judged per answer: is every claim supported by the retrieved material. On citation coverage: are the sources attached and do they hold up. And on refusal correctness — did the system decline the questions it should have declined. Refusal correctness is the metric that keeps the whole thesis honest, because a system that never refuses is not grounded, it is lucky.

The harness runs in CI as a regression gate. Every change re-runs the set, and regressions block the merge. This is the part that makes it work: an evaluation that produces a report someone reads sometimes is an evaluation that gets skipped in exactly the week it mattered.

LLM-as-judge does the scoring where a judgement call is needed, against fixed criteria and the same question set every time. Python throughout, wired into GitHub Actions.

The before-and-after on the corrective RAG pipeline — precision@5 from 0.72 to 0.89 after adding cross-encoder reranking — is an output of this harness, produced the same way every other change to that system is checked.