Every retrieval-augmented chatbot demo looks the same: paste in a folder of PDFs, ask a question, get a plausible answer, applause. Then it meets real users, who ask real questions, and the plausible answers start being wrong in ways nobody planned for.
In one line: A regulated financial client's document chatbot shipped only after a weighted 21-case evaluation, nine rounds of structured QA and a daily accuracy report the client ran themselves.
I spent most of this year taking one such chatbot from a demo to a client beta for a wealth-management firm in a regulated market. The model mattered less than I expected. The evaluation mattered more than anything else. This is the evaluation.
Start with the failure modes, not the happy path
Before writing a single test case I listed the ways the bot could be wrong. Three families covered almost everything:
- Intent failures. The bot answers a different question from the one asked, usually because the question was short, colloquial, or used the user's internal jargon.
- Retrieval failures. The right passage exists in the corpus but the bot never sees it, so it answers from the wrong document, or from nothing.
- Semantic drift. The bot retrieves the right passage and still says something the passage does not say: an extra claim, a softened caveat, a number rounded the wrong way.
Each family got seven cases, so 21 in total. Not more. A test set you can run by hand in an afternoon gets run. A test set of 300 gets automated badly or ignored.
The 21 cases at a glance
| Family | What it checks | A typical case | Weight |
|---|---|---|---|
| Intent | The bot answers the question that was asked, not a nearby one | A four-word question in the client's internal jargon | 1 to 2 |
| Retrieval | The right passage reaches the model | A figure that lives only inside a chart on a slide | 1 to 2 |
| Semantic drift | The answer says only what the passage says | A number the passage gives to two decimals, which the bot must not round | 2 |
| Scope, inside each family | The bot declines or redirects what it must not answer | "Which of these two funds should my mother buy" | 3 |
Scope cases are spread across the three families rather than kept as a fourth, because a scope failure usually looks like an intent or retrieval failure until you read the answer. Every weight-3 case is a scope case.
Weight the cases by what a wrong answer costs
Each case carries a weight of 1, 2 or 3, set by what a wrong answer would cost the client, not by how hard the case is.
- Weight 1: an embarrassing answer. Wrong product name, clumsy citation, a date off by a day.
- Weight 2: an answer that would mislead a customer. A rounded number, a softened caveat, a feature the document does not promise.
- Weight 3: an answer that would put the client outside the regulator's boundaries. Advice, a recommendation, an opinion on a product's suitability.
A round's score is the weighted pass rate: the sum of the weights of the cases that passed, divided by the sum of all weights. A case passes only if a reviewer reads the answer and agrees it is correct, cited and inside scope; a near miss is a fail.
Two rules sit on top of the score:
- Any weight-3 failure blocks the release, whatever the overall score. A 95 percent round with one weight-3 failure does not ship.
- A case that passed last round and fails this round is a regression, and a regression blocks the release until it is explained. Prompt "improvements" caused most of ours.
The weight-3 cases were all about scope: questions the bot must decline or redirect, phrased in ways that make declining hard. "Which of these two funds should my mother buy" is a weight-3 case. The correct answer is a polite refusal plus a pointer to a human, and it has to be the correct answer every time.
Nine rounds, and what each one found
We ran the 21 cases as a formal round after every meaningful change, nine rounds in total before beta. A sample of what each round caught:
- Retrieval was fine on long questions and poor on short ones. Fix: expand short queries with the user's previous message before embedding.
- Slide decks in the corpus were nearly unreadable. Standard OCR on exported PowerPoint pages captured roughly 75 to 80 percent of the text and lost everything inside charts. Fix: render each slide to an image and extract with a vision model, which took accuracy to 95 to 98 percent and finally read the numbers in the charts.
- The bot answered questions about a product that had been discontinued, from a document nobody had removed. Fix: a document registry with an "in force" flag, and retrieval filtered by it.
- Three weight-3 scope cases failed after a prompt "improvement" made the bot friendlier. Rolled back. Friendliness is not a requirement; staying inside the lines is.
- Duplicate questions in the same session got different answers. Fix: cache by normalised question within a session.
Rounds six to nine were smaller: citation formatting, a date-parsing bug, and a case where the correct answer was "the document does not say", which the bot found very hard to produce.
What "ready for beta" meant, written down
The release criteria lived on one page, and every round ended by checking it:
- All seven weight-3 cases pass.
- Weighted score at or above the pass line we had agreed with the client, for two consecutive rounds.
- No regressions against the previous round.
- Every document in the corpus is in the registry with an "in force" flag, and retrieval filters on it.
- The bot can say "the document does not say" and did so in at least one case this round.
- The client's operations team has the daily scoring sheet and has used it on real questions for a week.
The last criterion is the one people skip. The beta did not open until the client could measure it without us.
Hand the scoring to the client
The most useful thing we did was let the client's operations team score the bot daily, in their own spreadsheet, against their own real questions. Every incorrect answer got one of three labels:
- Bot issue: the bot had the right material and still got it wrong.
- Document issue: the material was missing, outdated, or contradictory.
- User understanding: the question was ambiguous or outside scope, and a human would have asked a follow-up.
A typical day in the beta: 79 queries, 87 percent correct, 10 incorrect, of which 7 were bot issues, 2 were user-understanding cases, and 1 was a duplicate. The trend over the first weeks moved between 82 and 91 percent. Two things came out of this that no internal test could have given us.
First, the split told us where to spend effort. When document issues dominate, the fix is content governance, not engineering. When user-understanding dominates, the fix is a clarifying-question behaviour, not a better model.
Second, the client stopped asking "is it accurate?" and started asking "what is today's number and why did it move?". That is the conversation you want.
The stack, briefly
The prototype was a React front end talking to a Flask script with a FAISS index. For beta it became a containerised FastAPI service with Celery workers, Redis, a Milvus vector store and PostgreSQL, behind Traefik, with images built in CI and pulled from a registry. None of that improved accuracy. It made the evaluation repeatable, which is what improved accuracy.
What I would tell anyone starting one of these
Lessons, in the order we learned them:
- Failure modes first, prompt second. If you cannot name seven ways each family fails, you do not understand the corpus yet.
- Twenty-one cases you run by hand beat three hundred you automate badly. The set has to fit in an afternoon or it stops being run.
- Weight by the cost of being wrong, and let one heavy failure block a release.
- Read the corpus the way the bot does. Slides that looked fine to us were 75 to 80 percent readable to standard OCR; rendering each slide as an image and reading it with a vision model took that to 95 to 98 percent and recovered the numbers in the charts.
- Stale documents are a retrieval failure in disguise. A registry with an "in force" flag fixed a whole class of wrong answers that no prompt could.
- Friendliness is not a requirement. One prompt change that made the bot warmer broke three scope cases. We rolled it back.
- Hand the scoring to the client with a three-way label: bot issue, document issue, user understanding. The split tells you where to spend the next week.
- The stack is plumbing. Moving from a Flask script and a FAISS index to FastAPI, Celery, Redis, Milvus and PostgreSQL behind Traefik did not change accuracy. It made the rounds repeatable, and repeatable rounds changed accuracy.
The bot is still in beta. The number still moves. But every move now has a reason attached to it, and that is the difference between a demo and a product.