How do you evaluate a RAG chatbot before buying it?

Write about thirty questions from your own documents before the demo: ten with a single findable answer, ten requiring two or more sources, and ten that your library genuinely cannot answer. Watch citation behaviour on the first twenty and refusal behaviour on the last ten. Ask the vendor to run your questions and produce a pass rate, a latency distribution and a cost per answer.

What is the single most important test for a document AI system?

Ask a reasonable question your documents do not cover. A system with an honest failure mode says it found nothing and suggests what would change the outcome. A system without one produces a fluent, plausible, unsourced paragraph. Almost every other property can be inferred from how a system behaves when it has nothing to say.

What RAG evaluation metrics actually matter?

Four: pass rate against a fixed set of your own questions with reference answers; retrieval recall, meaning whether the passage you know is correct actually came back; 95th-percentile latency rather than the average; and cost per answer per deployment. Retrieval recall matters most for diagnosis, because retrieval failures and generation failures look identical from outside and have different fixes.

Is an LLM judge enough to test answer quality?

No. A model scoring another model's output is a useful soft signal and a weak gate. It becomes a test when combined with deterministic checks: literal text that must or must not appear, assertions about which documents retrieval returned and at what scores, whether the answer came back confident or unanswered, and a latency ceiling.

Why does an evaluation suite have to run the production pipeline?

Because if it runs through a separate code path, a passing suite only proves the harness works. The suite has to exercise the same retrieval, thresholds, prompt and model that a live question does — otherwise it cannot catch the changes it exists to catch, such as a swapped model or a raised similarity threshold.

How often should a RAG system be re-tested?

On every change that can affect answers — a model swap, a threshold adjustment, a prompt edit, or a new tranche of documents — and on a schedule in between. Mark a known-good run as the baseline so later runs can be diffed against it rather than read in isolation.

How to Evaluate a RAG Chatbot Before You Buy It

How do you evaluate a RAG chatbot? With about thirty questions you write yourself, from your own documents, before you see any demo. Roughly a third should have a single findable answer, a third should require combining two places, and a third should have no answer in your library at all. The last third is the one that decides the purchase, because it is the only part of the test a fluent system cannot pass by being fluent.

Most evaluations of document-grounded AI are conducted by watching a vendor ask a system a question the vendor chose. This reliably produces a good impression and almost no information.

This article is a protocol for doing it properly: what to write before the demo, what to look for during it, and how to convert the whole thing into a suite you can re-run every time a model, a threshold or a prompt changes. We build Edtek Chat and ship an evaluation suite with it, so our framing is not neutral — but the protocol below is vendor-agnostic and works against anything, including us.

Write the questions before you see the product

Thirty questions, drawn from your own material, split three ways.

Ten single-source questions. The answer exists in exactly one place in your library and you know where. These test whether retrieval finds the obvious thing. They are the easiest tier and a system that fails here is disqualified.

Ten multi-source questions. The answer requires combining two or three passages, possibly from different documents. These test whether the system retrieves broadly enough and whether it merges without blurring — the common failure is an answer that is true of one source, attributed to another.

Ten unanswerable questions. Ordinary, reasonable questions about topics adjacent to your library that your documents genuinely do not cover. Not nonsense — a question a real user would plausibly ask. This tier is the whole evaluation. Everything else is table stakes.

For each question also write, in one line, what a correct answer contains. You will need it later, and writing it forces you to notice which of your questions are actually ambiguous.

What to watch during the demo

On the answerable questions

Does every claim carry a reference, or only some of them? Partial citation is worse than none, because it implies the uncited sentences were also checked.
Click a citation. Where do you land? The page with the passage marked, or the file? One is verification, the other is reassurance.
Read the cited passage and then re-read the sentence. Does the passage actually support the claim, or merely mention the same topic? This is the failure mode that survives every architectural safeguard, and it is the one you have to check by hand.
Ask the same question twice. Wildly different answers mean retrieval is unstable at your corpus size.
Ask what it did not use. A good system will show you the passages it retrieved, judged relevant, and did not cite. That list tells you more about retrieval quality than the answer does.

On the unanswerable questions

There are only three possible behaviours, and they are not equally good.

Behaviour	What it means
Answers fluently, no sources	Disqualifying. The system is drawing on model knowledge and presenting it as your content.
Answers, citing loosely related passages	Bad, and harder to catch. Retrieval had nothing and the system used the nearest thing anyway.
Declines and says what would change the outcome	Correct. This is the behaviour you are paying for.

Ask specifically whether the decline is generated or fixed. A system that never calls the answering model when nothing was retrieved cannot improvise, because there is nothing to improvise with. One that asks the model to write a refusal is trusting the model to refuse — which usually works and occasionally does not. We describe where that boundary sits in our own pipeline in how grounded answers work, including the case where a follow-up question is answered from the conversation rather than the library.

The four numbers worth asking for

Impressions do not survive procurement. These do.

Pass rate against a fixed set of cases. Not “accuracy” in the abstract — the proportion of your thirty questions that produce an answer you would sign. Ask whether the vendor can run your questions, not theirs.

Retrieval recall. For each single-source question, did the passage you know is correct actually come back? Retrieval failures and generation failures look identical from the outside and have completely different fixes. A system that can show you which passages were retrieved, at what scores, before and after re-ranking, lets you tell them apart.

Latency at the 95th percentile. Averages hide the answers that take twenty seconds. Users experience the tail.

Cost per answer. Per deployment and per model. Ask what happens to it if you switch to a stronger model — and whether you can switch without a release.

Turning the demo into a regression suite

The reason to write the questions down is that a document-grounded system is not a fixed thing. The model gets swapped for a cheaper one. Someone raises a similarity threshold to cut noise. A new tranche of documents lands. Each of those is a change that can silently make answers worse, and none of them will announce itself.

What replaces vigilance is the same practice software teams already use for code:

Golden questions + reference answers
        │
        ▼
  Run through the SAME pipeline users hit
        │
        ├── LLM judge: does the answer match the reference?
        ├── Rules: must contain "§ 106"; must not contain a competitor name
        ├── Retrieval assertions: did the right document come back, at what score?
        ├── Confidence: did the unanswerable ones actually come back unanswered?
        └── Latency and cost budgets
        │
        ▼
  Mark a run as the baseline ──▶ change something ──▶ diff against baseline

Two details separate a real suite from a demo of one.

It must run the production pipeline. If evaluation runs through a separate code path, a green suite tells you the evaluation harness works. Ask explicitly whether the test runner executes the same retrieval, the same thresholds, the same prompt and the same model as a live question.

An LLM judge alone is not a test. A model scoring another model’s output is a useful soft signal and a poor gate. It becomes a test when combined with deterministic assertions — literal text that must or must not appear, which documents retrieval had to return, whether the answer came back confident or unanswered, and a latency ceiling. A case that trips several checks should report the most serious one, not the friendliest.

If a vendor cannot run your thirty questions and hand you a report with a pass rate, a latency distribution and a cost figure, you are buying on impressions. Ours is described in how evaluation works, including the generator that drafts cases from your own documents so the suite does not have to be written by hand.

Two questions to ask about the numbers you are shown

“Which cases failed, and can I see them?” A vendor showing 100% on a suite they wrote is showing you their marketing. A vendor showing 87% and walking you through the thirteen is showing you their product. The second is far more useful, and considerably rarer.

“What was the baseline before this change?” Any number without a comparison is decoration. The useful artefact is a diff: what regressed, what improved, what is newly covered, and what it now costs per answer.