The illusion of intelligence: when AI sounds right but feels wrong
Most AI products are technically correct and experientially broken. Here is why that keeps happening, and what it actually takes to fix it.
A user wanted to add IPTV to their existing broadband connection. Simple enough. They opened the app, found the chatbot, and raised a connection request. What happened next had nothing to do with connecting IPTV.
The bot showed them a list of high-priced channel subscription plans. They did not want a new plan. They already had broadband. They just wanted IPTV added to it. So they tried calling the customer center. An automated voice system took over and walked them through the same irrelevant plans, in a loop, with no exit to a human being.
Within seconds, friction began.
“Where do I find the connection request?” “How do I talk to a real person?” “Why does none of this match what I actually need?”
They clicked. Scrolled. Backed out. Tried again. What should have been a two-minute task turned into a quiet, grinding frustration with no resolution in sight.
Now zoom out. This is not a story about one bad telecom chatbot. This exact pattern is playing out every day in enterprise tools, procurement dashboards, AI writing assistants, and so-called intelligent copilots. We are building systems that are functionally complete but experientially broken. The problem is not a lack of features. It is a lack of understanding.
Users do not want answers. They want progress.
Read that again, because it changes how you should think about every AI product you build or use.
The user in our IPTV story did not want information about plans. They wanted their IPTV connected. A user who asks “where is my order?” does not want a definition of order tracking. They want to feel certain their package is on the way. Someone calling their bank about a fraud charge does not want to be educated about fraud policy. They want the charge reversed and their card working again.
The answer is almost never the point. The progress is. And most AI products today are built entirely as answer engines: type a question, receive a response, interaction ends. That shift from answering to facilitating action is the entire design challenge of conversational AI, and most products are not even close to making it.
AI does not fail because it is not intelligent enough. It fails because it is designed around systems, not real human situations.
What is actually happening under the hood
Here is something most AI product teams know but rarely say out loud. The chatbot you built does not understand anything. It predicts. When a user types a message, the model calculates which words are most likely to follow, based on billions of sentences it has read before. That is the whole mechanism. There is no comprehension happening. No awareness that this is a specific person, with a specific history, who has been trying to solve a specific problem for the last twenty minutes.
AI responds to what you say, not what you need. It pattern-matches the surface of a message and retrieves the most statistically likely response. In controlled scenarios and product demos, this looks like intelligence. The moment reality becomes messy, contextual, and human, the illusion breaks.
“We built a GPT-4 powered assistant. Users still said it felt dumb. Turns out, the model was fine. The conversation design was the problem.”
Head of Product, a Series B fintech
And this is not a niche problem. Research consistently shows that user dissatisfaction with AI chatbots comes not from wrong answers but from responses that feel generic, incomplete, and disconnected from the actual situation. The absence of context, continuity, and even basic empathy makes interactions feel mechanical rather than helpful. Many companies that aggressively adopted AI in customer service are now quietly pulling back, reporting that inaccurate and contextually irrelevant responses are eroding user trust at scale.
These numbers tell a consistent story. The investment goes in. The users do not stay. When systems optimize for automation over resolution, they do not just frustrate individual users. They erode trust across the entire product.
A real conversation gone wrong
Before looking at why this happens systematically, it helps to see it in a single conversation. Notice how quickly, and how silently, the experience falls apart.
Four failure patterns in six messages. The AI ignored a clearly stated goal, asked for information it already had, took an irreversible action without confirmation, and then abandoned the user completely when things went outside its script. This is not a broken model. This is broken conversation design.
The gaps that make AI experiences fail
After looking at cases like these across different industries and products, the same gaps appear again and again. None of them are about the underlying model. All of them are about the experience built around it.
Get Amjad Khan’s stories in your inbox
Join Medium for free to get updates from this writer.
01. No real understanding of user intent:
AI listens to the words in a message and maps them to the closest predefined answer. It does not understand what the person is actually trying to accomplish. This is why “I want IPTV on my existing broadband” becomes a list of new subscription plans. The words got processed. The goal got ignored.
02. Designed for happy paths only:
Most AI experiences work perfectly in demos because demos follow the expected script. Real users do not. A payment that gets stuck mid-process, a service that activates for the wrong account, a hybrid issue that spans two departments: these are not edge cases. They are everyday reality. And they are almost never modeled.
AI listens to the words in a message and maps them to the closest predefined answer. It does not understand what the person is actually trying to accomplish. This is why “I want IPTV on my existing broadband” becomes a list of new subscription plans. The words got processed. The goal got ignored.
03. No context across the conversation:
Every time a user moves from the app to a chatbot to a call center, they start from zero. No history. No memory of what was already said. The user repeats themselves. The experience fragments. What feels like a connected journey is actually a series of isolated interactions with no shared memory between them.
04. No way out when things go wrong:
IVR loops. Chatbot dead ends. “Please contact our support team” with no contact method provided. When AI hits a wall, it traps the user inside the wall with it. There is no escalation to a human, no graceful exit, no acknowledgment that the system has reached its limit. The frustration peaks exactly when the stakes are highest.
05. Optimized for business metrics, not user resolution:
Many AI systems are built to reduce call volume, increase upsell opportunities, and deflect support tickets. These are business goals, not user goals. When the IPTV chatbot pushed subscription plans instead of processing a connection request, it was doing exactly what it was designed to do. The problem is the design brief itself.
06. No ownership when something breaks:
No ticket created. No case owner assigned. No way to track whether the issue was resolved. AI systems frequently avoid traceable accountability, because accountability is expensive and deflection is cheaper. The user is left not just without a solution, but without any certainty that anyone is working on one.
The deepest insight from all of these gaps is this: the system did not fail because it was broken. It failed because it only understands perfect problems. Real users bring messy, contextual, in-between situations. The system was simply never designed to meet them there.
The difference between an answer and progress
The same situation. The same model. Completely different experience based on how the conversation is designed.
The difference in each pair is not the model. It is the intent behind the design. One version was built to respond. The other was built to help someone get somewhere.
How to actually fix it: the solution framework
The shift from answer engine to action engine does not require a better AI. It requires better decisions about how AI is designed. Here are six principles that separate conversational experiences that earn trust from ones that destroy it.
01Only promise what you can actually deliver:
If your chatbot handles three scenarios well, say those three things clearly at the start. “Ask me anything” is a promise. Most chatbots cannot keep it. The moment a user arrives with scenario four, you have not just failed their request. You have broken their trust in the entire product.
Replace vague with specific. Always.
02Design for when things go wrong, not just when they go right:
Every “I do not know” moment needs a designed next step. A clarifying question. An escalation to a human. An honest acknowledgment of the limit. There is no acceptable dead end. “I cannot help with that” followed by silence is not a design decision. It is giving up on the user.
Every dead end is a design failure.
03Always ask before doing anything:
If your bot can take an action on someone’s behalf, show them exactly what is about to happen and wait for a yes. Every time. People can forgive a wrong answer. They cannot forgive an action they did not ask for. Show, confirm, then act. Never skip the confirm step.
Show. Confirm. Act. In that order, always.
04Remember what the user already told you:
If someone is logged in, use their data. If they mentioned their order number two messages ago, do not ask again. Context continuity is the single biggest lever for perceived intelligence, more than any model upgrade. Asking for information you already have signals incompetence faster than any wrong answer.
Never ask for what you already know.
05Never go silent while working:
If your bot takes more than a second and a half to respond, show something specific. Not a spinning circle. “Looking into your account…” or “Checking your order now.” A contextual message feels like progress. A blank screen feels like a crash. Users quit when they think something is broken, even when it is not.
Silence feels like failure. Fill it.
06Give your bot a consistent voice, not just a system prompt:
A system prompt tells the bot what to do. A voice system tells it how to sound, every single time. Write three to five clear tone rules: “Acknowledge how the user feels before solving the problem. Be direct, not verbose. Never start with an apology. Use plain language.” Then test real outputs against those rules. Consistency is what makes a chatbot feel like a product rather than a random function.
Consistent tone builds trust over time.
Answer engine vs action engine
The model is not your product. The conversation is your product. And conversations are designed, not generated.
Every team building AI chat experiences has access to roughly the same underlying models. The ones that earn lasting user trust are not winning on intelligence. They are winning because someone on the team asked the harder question: what happens when this does not work? What happens when the user is frustrated, the data is missing, the scenario is not in the script? What happens at the edges?
Most AI products are designed from the center. The happy path. The expected input. The demo scenario. But real users live at the edges. They come in mid-frustration. They have tried three other things already. Their situation is messier than any flow chart can anticipate. Designing for them, not the demo version of them, is what separates an AI experience that feels alive from one that just sounds smart.
The IPTV user in this story did not give up on technology. They gave up on a specific product that treated their very real, very simple problem like it was impossible to understand. That is not an AI limitation. That is a choice that was made somewhere in the design process. And choices can be made differently.
If you are building something with AI right now, try this: stop asking “is it giving correct answers?” and start asking “is it helping real people make progress?” That one shift in question will change everything you
build after it.
