Every retrieval-augmented chatbot demo looks the same: paste in a folder of PDFs, ask a question, get a plausible answer, applause. Then it meets real users, who ask real questions, and the plausible answers start being wrong in ways nobody planned for.
I spent most of this year taking one such chatbot from a demo to a client beta for a wealth-management firm in a regulated market. The model mattered less than I expected. The evaluation mattered more than anything else. This is the evaluation.
Start with the failure modes, not the happy path
Before writing a single test case I listed the ways the bot could be wrong. Three families covered almost everything:
- Intent failures. The bot answers a different question from the one asked, usually because the question was short, colloquial, or used the user's internal jargon.
- Retrieval failures. The right passage exists in the corpus but the bot never sees it, so it answers from the wrong document, or from nothing.
- Semantic drift. The bot retrieves the right passage and still says something the passage does not say: an extra claim, a softened caveat, a number rounded the wrong way.
Each family got seven cases, so 21 in total. Not more. A test set you can run by hand in an afternoon gets run. A test set of 300 gets automated badly or ignored.
Weight the cases by what a wrong answer costs
Not every wrong answer is equal. In a regulated business, a bot that invents a product feature is embarrassing; a bot that gives advice outside the regulator's boundaries is a compliance problem. So each case carries a weight from 1 to 3, and the score is weighted. A release that fails a single weight-3 case does not ship, even at 95 percent overall.
The weight-3 cases were all about scope: questions the bot must decline or redirect, phrased in ways that make declining hard. "Which of these two funds should my mother buy" is a weight-3 case. The correct answer is a polite refusal plus a pointer to a human, and it has to be the correct answer every time.
Nine rounds, and what each one found
We ran the 21 cases as a formal round after every meaningful change, nine rounds in total before beta. A sample of what each round caught:
- Retrieval was fine on long questions and poor on short ones. Fix: expand short queries with the user's previous message before embedding.
- Slide decks in the corpus were nearly unreadable. Standard OCR on exported PowerPoint pages captured roughly 75 to 80 percent of the text and lost everything inside charts. Fix: render each slide to an image and extract with a vision model, which took accuracy to 95 to 98 percent and finally read the numbers in the charts.
- The bot answered questions about a product that had been discontinued, from a document nobody had removed. Fix: a document registry with an "in force" flag, and retrieval filtered by it.
- Three weight-3 scope cases failed after a prompt "improvement" made the bot friendlier. Rolled back. Friendliness is not a requirement; staying inside the lines is.
- Duplicate questions in the same session got different answers. Fix: cache by normalised question within a session.
Rounds six to nine were smaller: citation formatting, a date-parsing bug, and a case where the correct answer was "the document does not say", which the bot found very hard to produce.
Hand the scoring to the client
The most useful thing we did was let the client's operations team score the bot daily, in their own spreadsheet, against their own real questions. Every incorrect answer got one of three labels:
- Bot issue: the bot had the right material and still got it wrong.
- Document issue: the material was missing, outdated, or contradictory.
- User understanding: the question was ambiguous or outside scope, and a human would have asked a follow-up.
A typical day in the beta: 79 queries, 87 percent correct, 10 incorrect, of which 7 were bot issues, 2 were user-understanding cases, and 1 was a duplicate. The trend over the first weeks moved between 82 and 91 percent. Two things came out of this that no internal test could have given us.
First, the split told us where to spend effort. When document issues dominate, the fix is content governance, not engineering. When user-understanding dominates, the fix is a clarifying-question behaviour, not a better model.
Second, the client stopped asking "is it accurate?" and started asking "what is today's number and why did it move?". That is the conversation you want.
The stack, briefly
The prototype was a React front end talking to a Flask script with a FAISS index. For beta it became a containerised FastAPI service with Celery workers, Redis, a Milvus vector store and PostgreSQL, behind Traefik, with images built in CI and pulled from a registry. None of that improved accuracy. It made the evaluation repeatable, which is what improved accuracy.
What I would tell anyone starting one of these
- Write the 21 cases before you write the prompt. If you cannot name seven ways each family fails, you do not understand the corpus yet.
- Weight by cost of being wrong, and let one heavy failure block a release.
- Run the cases as a ritual after every change, and keep the results.
- Get the client scoring real traffic with a three-way label from day one of beta.
- Treat the vector database and the framework as plumbing. Budget your time for reading wrong answers.
The bot is still in beta. The number still moves. But every move now has a reason attached to it, and that is the difference between a demo and a product.