Does the test suite run the real product?

Yes. A run executes the same answering pipeline that serves your readers — the same retrieval, the same thresholds, the same prompt, the same model. It is not a simulation or a separate code path, which is the only way a green suite means anything.

How do you test that it refuses?

You write a case whose expected outcome is no answer, and assert on it. The test generator will even write these for you: one of its question types is a question the documents deliberately do not answer, verified as unanswerable before the case is saved.

Can we catch a regression before it reaches readers?

Mark a finished run as the baseline, then compare any later run against it. The comparison lists what regressed, what improved, what is newly covered and what disappeared, with the deltas on pass rate, latency, cost and judge score. Runs can also be scheduled on a cron expression in your timezone.

Do we have to write all the test cases by hand?

No. Point the test builder at a set of documents and it drafts cases from their actual content: it selects a passage, writes a question about it, then verifies through the real pipeline that the question is answerable from your library before writing the reference answer. Generated cases arrive switched off and wait for a human to review and enable them.

Does test traffic pollute our analytics?

No. Runs are recorded against the run, not the deployment — test turns are not written into the model-call log that feeds usage and cost reporting.

Can we get the results out?

Every run exports as CSV, JSON or Markdown, including the retrieval evidence columns — which documents were retrieved, at what scores, before and after reranking — so a failure can be diagnosed outside the console.

Evaluation

Answer quality, as a test suite.

Software teams do not ship without tests. A chatbot over your own library should not either. Edtek Chat comes with a regression suite for answers — golden questions, a judge, retrieval assertions, latency and cost budgets — running through the same pipeline that serves your readers.

Book a demo See the whole product

A finished run: pass rate, latency at p50 and p95, cost, average judge score, and every case with its checks.

Why it exists

Every tuning decision is a bet. This is how you settle it.

Raise the similarity floor and fewer wrong passages get through — along with some right ones. Switch models to cut cost and something you never tested gets worse. Without a suite, all of that is opinion.

Before you change it
Run the suite on the current settings and mark it as the baseline. That is the number any later argument has to beat.
While you change it
Override the model or the thresholds for one run only. The deployment your readers use never moves while you are experimenting.
After you change it
Compare against the baseline. What regressed, what improved, and what it now costs per answer — as a list, not an impression.

How a case is judged

Six ways a case can fail, and only one of them is a model opinion.

An LLM judge alone is a soft signal. Combined with deterministic assertions about retrieval, confidence, wording and latency, it becomes a test.

Cases

A question, a reference answer, and what must be true

Cases are grouped so a suite can be run whole or by area, switched on and off individually, and carried between deployments as JSON.

The checks

Judge score: A second model compares the answer against your reference answer and scores it. Below your threshold, the case fails.
Rule expressions: Require or forbid literal text with AND, OR, NOT and regular expressions — for the section number that must appear, or the competitor name that must not.
Retrieval assertions: Bounds on how many passages survived each stage, the top match score, and which documents or collections the final set had to come from.
Confidence: Assert that a question comes back confident, low-confidence, or unanswered — the last of these is how you test that it refuses.
Latency ceiling: Fail a case that takes longer than the budget you set for it.
Repeats: Run each case up to ten times in one go and fold the results to the worst outcome, so an intermittent failure cannot pass by luck.

A case that trips several checks reports the most serious one, so a run summary never flatters itself.

Test builder

Cases written from your documents, verified before they are saved

Writing a hundred golden questions by hand is why most teams never build a suite. The builder drafts them from the content itself, then checks its own work.

Four steps per caseChoose a passage worth asking about, write a self-contained question, run it through the real pipeline to confirm the library answers it, then write the reference answer from everything retrieved.
Seven kinds of questionFactual, definitional, conceptual, descriptive, comparative, procedural — and questions the documents deliberately cannot answer, verified to come back unanswered.
Nothing goes live unreviewedGenerated cases are created switched off. A person enables them, having read them.

A test build report showing seven cases written across four documents, broken down by question type — A build: seven cases across four documents, with coverage by question type and by document.

Running a suite

What you can change for a single run.

A run is an experiment. Everything that shapes an answer can be overridden for the duration of one, and nothing you override touches the live deployment.

Scope: The whole suite, one group, or a hand-picked set of cases.
Model override: Run the same suite on a different answering model without touching the deployment.
Judge and threshold: Which model judges, and how strict a pass is.
Retrieval override: Strictness, passage counts and per-document cap, just for this run.
Check overrides: Force the relevance, verification or moderation passes off to isolate what a change actually did.
Collection subset: Narrow the run to part of the library.
Concurrency: How many cases run at once.
Stored evidence: Optionally keep the retrieved passage text with the result for later inspection.

Keeping it honest

A suite that only runs when someone remembers is not a suite.

Schedule runs on a cron expression in your own timezone, keep a baseline per deployment, and export every result for whoever has to sign off on it.

Scheduled runs
Nightly, weekly, or before each content update — with the run’s parameters saved alongside the schedule so every run is comparable.
Baseline and compare
One baseline per deployment. Any run can be diffed against it: regressed, improved, added, removed, with deltas on pass rate, latency, cost and judge score.
Exports with evidence
CSV, JSON or Markdown, carrying the retrieval evidence — which documents came back, at what scores, before and after reranking.

Frequently asked questions

Does the test suite run the real product?
Yes. A run executes the same answering pipeline that serves your readers — the same retrieval, the same thresholds, the same prompt, the same model. It is not a simulation or a separate code path, which is the only way a green suite means anything.
How do you test that it refuses?
You write a case whose expected outcome is no answer, and assert on it. The test generator will even write these for you: one of its question types is a question the documents deliberately do not answer, verified as unanswerable before the case is saved.
Can we catch a regression before it reaches readers?
Mark a finished run as the baseline, then compare any later run against it. The comparison lists what regressed, what improved, what is newly covered and what disappeared, with the deltas on pass rate, latency, cost and judge score. Runs can also be scheduled on a cron expression in your timezone.
Do we have to write all the test cases by hand?
No. Point the test builder at a set of documents and it drafts cases from their actual content: it selects a passage, writes a question about it, then verifies through the real pipeline that the question is answerable from your library before writing the reference answer. Generated cases arrive switched off and wait for a human to review and enable them.
Does test traffic pollute our analytics?
No. Runs are recorded against the run, not the deployment — test turns are not written into the model-call log that feeds usage and cost reporting.
Can we get the results out?
Every run exports as CSV, JSON or Markdown, including the retrieval evidence columns — which documents were retrieved, at what scores, before and after reranking — so a failure can be diagnosed outside the console.

Ask us to run your questions, not ours.

Send a handful of the questions your readers actually ask. We will build them into a suite and show you the report.

Book a demo Enterprise deployment