The Chatbot Worked Until Customers Started Using It
Imagine a chatbot that passes every internal test.
It answers questions about pricing, returns, account access, and delivery times. The team tries different prompts, reviews the responses, and approves the launch.
Then real customers arrive.
One person types, “My order says delivered but nothing came.” Another writes, “charged twice pls help.” Someone else asks three questions in one message and adds an angry complaint at the end.
The chatbot that appeared intelligent during testing suddenly becomes confused. It repeats policy text, misunderstands short messages, and gives confident answers when it should ask for clarification.
The model has not suddenly become worse. The testing was simply too clean.
Real Customers Do Not Write Test Questions
Internal teams usually test chatbots with complete and clearly written questions:
“What is your refund policy?”
Real customers rarely communicate that way.
They use spelling mistakes, incomplete sentences, screenshots, local phrases, and emotional language. They may change the subject halfway through a conversation or assume the chatbot already knows what happened earlier.
A reliable chatbot must understand more than the words in a message. It needs conversation history, customer context, current business information, and a safe way to ask follow-up questions.
Without these elements, even a capable model may be forced to guess.
The Knowledge Base Was Part of the Problem
The team connected the chatbot to its support documents, so everyone assumed the answers would be accurate.
Get Ravi Mansuriya’s stories in your inbox
Join Medium for free to get updates from this writer.
But the knowledge base contained several versions of the refund policy. Product names were inconsistent. Some help articles had not been updated for months. Important exceptions were hidden inside long PDF documents.
When a customer asked about a refund, the retrieval system sometimes selected an older policy. The model then created a polished answer using the wrong information.
Connecting documents through RAG does not automatically produce a reliable chatbot.
Google Cloud’s guidance on generative AI evaluation recommends evaluating whether a RAG system retrieves the correct information and whether its final response remains faithful and relevant to that information.
A few convincing test answers cannot prove that the complete system is ready.
The Chatbot Was Never Taught to Stop
The most serious problem was not that the chatbot occasionally misunderstood customers. It was that it always tried to answer.
When information was missing, it guessed. When two policies conflicted, it selected one. When a customer needed human support, it continued talking instead of escalating the conversation.
Claude’s official hallucination guidance recommends allowing a model to admit uncertainty, grounding factual answers in evidence, and removing claims that cannot be supported by citations.
These controls may make a chatbot sound less confident, but they make it more trustworthy.
Real Conversations Should Become the Test Set
The solution is not one better prompt.
Teams need to review failed conversations, remove outdated documents, improve retrieval, and define clear escalation rules. Testing should include spelling errors, vague requests, emotional messages, conflicting questions, and situations where no correct answer exists.
Whether the work is handled internally or with an AI chatbot development company, evaluation should reflect how customers actually communicate.
Each failed conversation can become a new test case. Over time, the test set stops reflecting what the team expects customers to ask and starts reflecting what customers truly ask.
The chatbot worked exactly as it had been tested.
The problem was that nobody had tested it for reality.
