Hallucinations

Why LLMs Sometimes Sound Confidently Wrong

This article builds on concepts covered earlier in the LLM Fundamentals series.

In the previous article, we learned how structured outputs make LLM responses easier for applications to consume.

We saw how schemas can define the expected structure.

We saw how validation can verify that a response follows that structure.

But there is an important limitation.

A response can be perfectly structured and still be incorrect.

For example:

{
  "capital": "Berlin"
}

The JSON is valid.

The structure is correct.

The schema passes validation.

The answer is still wrong.

This introduces one of the most important concepts in modern AI systems.

Hallucinations.

What Is a Hallucination?

A hallucination occurs when a model generates information that appears believable but is not actually correct.

The generated response may look convincing.

It may be well-written.

It may even contain detailed explanations.

But the information itself can be inaccurate, misleading, or completely fabricated.

For example:

Who won the 2032 Cricket World Cup?

A model might confidently generate an answer describing the final match, the winning team, and the player of the tournament.

The response may sound realistic.

The problem is that the 2032 Cricket World Cup has not happened yet.

The model isn’t recalling a fact.

It is generating a continuation that looks plausible.

And that’s an important distinction.

Why Do Hallucinations Happen?

To understand hallucinations, we need to revisit one of the core ideas from earlier in this series.

Large Language Models are trained to predict the next token.

They are not trained to verify facts before responding.

When a user asks a question, the model does not stop and perform a fact-checking process.

Instead, it generates the sequence of tokens that appears most likely based on its training and the information available in the context window.

This means the model is fundamentally operating through pattern prediction.

Not fact verification.

And that difference explains why hallucinations can occur.

Facts vs Patterns

Humans often think of knowledge as stored facts.

For example:

The capital of France is Paris.

We imagine a model retrieving that fact from memory whenever it is needed.

In reality, LLMs do not work like a traditional database.

They learn patterns from enormous amounts of text.

When asked a question, they generate the most probable continuation based on those patterns.

Most of the time, the learned patterns align with reality.

Sometimes they do not.

This is why a response can sound extremely confident while still being incorrect.

The model is generating what looks statistically reasonable.

It is not checking whether the statement is actually true.

There Is No Built-In “I Don’t Know”

Humans often respond with:

I don't know.

or

I'm not sure.

when they lack enough information.

LLMs do not naturally have that reflex.

If a prompt resembles patterns seen during training, the model often continues generating a response.

Even when certainty is low.

This does not mean the model is intentionally misleading anyone.

It is simply doing what it was trained to do:

Predict the next token.

And sometimes the most probable continuation happens to be wrong.

The Three Root Causes of Hallucinations

At this point, it may seem like hallucinations happen randomly.

In reality, they usually originate from a few common causes.

Understanding these causes makes it easier to understand why hallucinations are difficult to eliminate completely.

1. Training Gaps

A model can only learn from the information available during training.

If certain facts were never present in the training data, the model has no direct knowledge of them.

Let’s see the example we discussed earlier

Who won the 2032 Cricket World Cup?

The model cannot know the answer because that event has not happened yet.

However, instead of stopping, it may generate a response that looks plausible based on similar patterns seen during training.

The result is a confident but incorrect answer.

2. Pattern Matching

Even when information exists, the model does not retrieve it like a database.

It predicts what text is likely to come next.

Suppose a model has seen thousands of examples that look like:

Country → Capital

When asked a similar question, it tries to continue the pattern.

Most of the time this works.

Sometimes the generated continuation sounds reasonable but is wrong.

This happens because the model is optimizing for probable text, not verified truth.

3. Optimization Pressure

During training, models are rewarded for generating useful and coherent responses.

They are not rewarded for frequently saying:

I don't know.

As a result, models often prefer producing an answer over producing no answer at all.

This behavior makes conversations feel more natural.

But it also increases the likelihood of hallucinations when information is uncertain or unavailable.

By now, we understand why hallucinations occur.

Models generate responses through pattern prediction.

They do not continuously verify facts before producing an answer.

As a result, hallucinations can appear in many different forms.

Sometimes the model generates an incorrect fact.

Sometimes it invents a source that does not exist.

In other situations, it may misunderstand instructions or even claim that an external tool returned information when it never did.

Although we often use the word “hallucination” as if it describes a single problem, in practice it can appear in several different ways.

Understanding these patterns makes it easier to recognize and design around them.

1. Factual Hallucinations

These are the most common type.

The model generates information that is simply incorrect.

For example:

The Eiffel Tower was built in 1958.

The statement sounds reasonable.

The fact is wrong.

2. Source Hallucinations

Sometimes the model invents sources, citations, books, research papers, or URLs.

For example:

According to a 2019 research paper published in Nature...

The paper may not actually exist.

The citation may be completely fabricated.

This is why generated references should always be verified before being trusted.

3. Instruction Hallucinations

In some situations, the model may partially ignore instructions or invent details that were never requested.

For example:

Summarize this article.

The summary may include assumptions that were not present in the original content.

The model is still trying to produce a helpful response, but it has drifted beyond the provided information.

4. Tool and API Hallucinations

Modern AI applications often connect LLMs to tools, databases, and APIs.

A model may sometimes claim that:

• a tool was executed
• a database was queried
• an API returned data

when none of those actions actually occurred.

This is one of the reasons production systems typically verify tool execution rather than trusting model-generated claims.

At this point, hallucinations may sound like a serious limitation of LLMs.

And they are.

But in practice, engineers rarely expect models to be perfectly accurate all the time.

Instead, systems are designed to reduce hallucinations, detect them when they occur, and verify important information before it reaches users.

The goal is not to eliminate hallucinations entirely.

The goal is to make them manageable.

So the next question becomes:

How do modern AI systems reduce and detect hallucinations in production?

Reducing Hallucinations

By now, one thing should be clear.

Hallucinations are not a bug that can simply be switched off.

They are a natural consequence of how Large Language Models generate text.

As a result, engineers rarely focus on eliminating hallucinations completely.

Instead, they focus on reducing the chances of hallucinations and limiting their impact when they occur.

Over the years, a few approaches have emerged as particularly effective.

1. Retrieval-Augmented Generation (RAG)

One of the most effective ways to reduce hallucinations is to provide the model with relevant information before it generates a response.

Instead of relying entirely on information learned during training, the system first retrieves documents, database records, or knowledge base articles related to the user’s question.

Those documents are then inserted into the context window.

The model can use this information while generating its answer.

Rather than asking the model to rely on memory, we give it access to the facts it needs.

This significantly reduces hallucinations, especially when dealing with company knowledge, documentation, support articles, or other domain-specific information.

We’ll explore Retrieval-Augmented Generation in much more detail later in this series.

2. Using Facts From External Systems

Sometimes the most reliable source of information is not the model itself.

It is an external system.

For example:

• Databases
• APIs
• Search systems
• Internal company services

Imagine asking:

What is my current account balance?

A language model should not attempt to generate that answer.

Instead, the application should retrieve the value from a banking system and provide the result to the user.

Get Kusuma Pindi’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The same idea applies to weather information, stock prices, order status, and many other forms of dynamic data.

Whenever possible, factual information should come from systems designed to store facts.

3. Prompting Techniques

Prompting cannot eliminate hallucinations, but it can often reduce them.

For example, prompts can instruct the model to:

• Answer only from the provided context
• State when information is unavailable
• Avoid making assumptions
• Cite supporting evidence when possible

A prompt like:

If the answer is not present in the provided information, say "I don't know."

can sometimes reduce unnecessary fabrication.

However, it is important to remember that prompts are still instructions.

The model may follow them most of the time, but there is no absolute guarantee.

This is why prompting is usually combined with other techniques rather than relied upon as the only defense.

4. Structured Outputs

Structured outputs help constrain the format of a response.

Instead of generating completely free-form text, the model is guided toward a predefined structure.

For example:

{
  "customer_name": "",
  "issue_type": "",
  "priority": ""
}

This reduces ambiguity and makes responses easier to validate and process.

Structured outputs can also help prevent certain classes of errors by forcing the model to return information in a predictable format.

However, they do not guarantee correctness.

A response can be perfectly structured and still contain incorrect information.

This is an important distinction.

Structured outputs improve reliability.

They do not verify facts.

Even after applying these techniques, hallucinations can still occur.

Retrieval systems can return incomplete information.

External systems may be unavailable.

Prompts may be misunderstood.

And structured outputs can only validate format, not truth.

As a result, modern AI systems often include another layer of protection.

They attempt to detect hallucinations before the response reaches the user.

Let’s look at some of the common approaches used in practice.

Detecting Hallucinations

Unlike traditional software bugs, hallucinations are often difficult to identify.

The response may look professional.

It may be grammatically correct.

It may even sound highly confident.

This makes hallucinations particularly challenging because they do not always look like mistakes.

As a result, modern AI systems often use additional techniques to identify potentially incorrect responses before they reach users.

Let’s look at some of the common approaches used to detect them.

1. LLM-as-a-Judge

One common approach is to use another model to evaluate the generated response.

The first model generates an answer.

A second model reviews that answer and checks for issues such as:

• Factual inconsistencies
• Missing evidence
• Contradictions
• Unsupported claims

This technique is often called LLM-as-a-Judge.

While the second model is not perfect, it can sometimes catch mistakes that the original model missed.

However, it is important to remember that using another LLM does not guarantee correctness.

A model evaluating another model can still make mistakes.

2. Self-Consistency

Another technique is to generate multiple responses for the same question.

If the answers are significantly different from one another, it may indicate uncertainty.

For example:

Question:
What caused Event X?

Response 1:
Answer A
Response 2:
Answer B
Response 3:
Answer A

If most responses converge on the same conclusion, confidence may be higher.

If the responses vary dramatically, additional verification may be needed.

The idea is simple:

When a model cannot consistently arrive at the same answer, there is a greater chance that the response contains errors.

3. Confidence Signals

No matter how convincing a response sounds, confidence and correctness are not the same thing.

However, some systems still use confidence-related signals to estimate how reliable a response might be.

For example, a system may look for:

• Unusual uncertainty in the generated tokens
• Large variations between multiple responses
• Missing supporting evidence
• Low agreement with retrieved information

These signals do not prove that a response is correct or incorrect.

Instead, they act as indicators that additional verification may be needed.

Think of them as warning signs rather than proof.

A low-confidence response is not always wrong.

A high-confidence response is not always right.

This is why confidence signals are usually combined with other detection techniques rather than used on their own.

4. Citations and Verification

Another common approach is to require the model to provide evidence for its claims.

Instead of accepting statements at face value, the system asks:

Where did this information come from?

The model may then provide:

• Retrieved documents
• Knowledge base articles
• Database records
• External references

The application can verify whether those sources actually support the generated answer.

For example, consider the statement:

The company's refund period is 60 days.

Rather than trusting the statement directly, the system can check the referenced policy document and verify that the claim is supported by the source.

This approach shifts the focus from trusting the model to trusting the evidence.

In many production systems, verified citations are often more valuable than the generated answer itself.

Detection Is Never Perfect

Just like hallucination reduction techniques, detection methods are not perfect.

A hallucination may go unnoticed.

A correct answer may sometimes be flagged as suspicious.

This is why modern AI systems rarely rely on a single defense.

Instead, they combine multiple layers of protection.

Retrieval reduces hallucinations.

Validation catches formatting issues.

Detection identifies suspicious responses.

And together, these layers make AI systems significantly more reliable than any individual technique alone.

Putting It Together: Defenses Against Hallucinations

Throughout this article, we’ve looked at several techniques that help make LLM systems more reliable.

Some techniques focus on reducing hallucinations before they occur.

Others focus on detecting hallucinations after a response has been generated.

Together, they form multiple layers of defense.

For example:

• Retrieval provides relevant facts.
• APIs and databases provide trusted information.
• Prompting can reduce unnecessary assumptions.
• Structured outputs improve consistency.
• Detection systems identify suspicious responses.
• Verification systems validate supporting evidence.

No single technique solves the problem entirely.

Instead, modern AI systems combine several of these approaches to achieve higher reliability.

This is one of the reasons production LLM applications look very different from a simple chatbot.

If we have so many defenses available, where should we start?

Effort vs Impact

By now, we’ve looked at several techniques for reducing and detecting hallucinations.

Naturally, the next question becomes:

If you’re building an AI application today, where should you start?

Not all techniques require the same amount of effort.

And not all of them provide the same level of benefit.

In practice, a good approach is to start with the simplest improvements first and gradually move toward more sophisticated solutions as requirements grow.

Low Effort, Moderate Impact

Prompting is usually the easiest place to start.

Clear instructions, grounding the model in provided context, and asking it to acknowledge uncertainty can often reduce obvious hallucinations.

Structured outputs are another relatively low-effort improvement.

They make responses easier to validate and process, even though they cannot guarantee factual correctness.

Medium Effort, Higher Impact

Techniques such as self-consistency and LLM-as-a-judge require additional model calls and evaluation logic.

They can help identify suspicious responses, but they also increase latency and cost.

These approaches are often useful when accuracy is important but full retrieval systems are not yet available.

Higher Effort, Highest Impact

Retrieval systems, databases, APIs, and external knowledge sources usually provide the biggest improvements in reliability.

Instead of asking the model to rely on patterns learned during training, they provide access to actual information at runtime.

This significantly reduces the chances of the model inventing facts.

The tradeoff is increased system complexity.

Additional infrastructure is required to retrieve, validate, and manage external information sources.

So Where Should You Start?

For many applications, a practical progression looks like this:

Prompting
      ↓
Structured Outputs
      ↓
Retrieval
      ↓
Verification

Each layer improves reliability.

Each layer adds complexity.

The right balance depends on the requirements of the system you’re building.

A Different Way to Think About Hallucinations

One of the biggest misconceptions about Large Language Models is that hallucinations are a temporary problem that will eventually disappear.

In reality, hallucinations are closely tied to how these models work.

LLMs generate responses by predicting patterns in language.

They do not inherently distinguish between a statement that is true and a statement that merely sounds plausible.

This is why experienced engineers rarely assume that a model will always be correct.

Instead, they build systems that can provide facts, verify outputs, and handle uncertainty when it appears.

The goal is not perfection.

The goal is reliability.

As we’ve seen throughout this article, one of the most effective ways to improve reliability is to provide the model with relevant information before it generates a response.

But that introduces another interesting question.

How does a system know which information is relevant?

How can two pieces of text be compared based on meaning rather than exact words?

And how does an AI assistant search through thousands of documents and find the few pieces of information that actually matter?

To answer those questions, we first need to understand Embeddings.

And that’s exactly what we’ll explore in the next article.

See you there 👋