Embeddings

How Machines Learn to Understand Meaning

This article builds on concepts covered earlier in the LLM Fundamentals series.

The Search Problem

Imagine searching for:

Imagine searching for:

And somewhere in a document collection, there is a page titled:

Guide to Purchasing an Automobile

Most humans immediately understand that:

car ≈ automobile

Even though the words are different, the meaning is almost identical.

For a long time, computers struggled with problems like this.

Traditional search systems primarily relied on matching keywords.

If the exact words were not present, relevant documents could easily be missed.

Humans search by meaning.

Computers traditionally searched by words.

To solve this problem, we needed a way to represent meaning itself.

This is exactly what embeddings were designed to do.

What Is an Embedding?

At a high level:

An embedding is a numerical representation of meaning.

Instead of working directly with text, an embedding model converts text into numbers.

For example:

"dog"

might become:

[0.27, -0.81, 0.45, ...]

And:

"puppy"

might become:

[0.25, -0.79, 0.47, ...]

The actual numbers are not important.

What matters is that texts with similar meanings produce vectors that are close to one another.

Okay, but how does text become numbers?

How Embeddings Are Created

At this point, a natural question appears.

How does a piece of text become an embedding?

The exact process involves neural networks trained on enormous amounts of text and is far beyond the scope of this article.

Fortunately, we don’t need to understand every mathematical detail to understand the core idea.

An embedding model is trained to recognize relationships between words, phrases, and concepts.

Over time, it learns that certain pieces of text are often used in similar contexts.

For example:

Car
Automobile
Vehicle

are frequently associated with concepts such as:

Drive
Road
Engine
Transportation

Because these words appear in similar contexts, the model gradually learns that they are related.

When text is passed through the embedding model, it is transformed into a vector that captures these learned relationships.

A simplified view looks like this:

Text
   ↓
Embedding Model
   ↓
Vector

The resulting vector becomes a mathematical representation of the text’s mean

Where Do These Vectors Go?

At this point, another question naturally appears.

Once an embedding has been generated, what happens to it?

After all, creating vectors is only useful if we can store and search them later.

This is where vector databases enter the picture.

Vector Databases

A vector database is a specialized database designed to store and search embeddings.

Unlike traditional databases, which are optimized for exact matches and structured queries, vector databases are optimized for finding similar vectors.

A simplified workflow looks like this:

Document
     ↓
Embedding Model
     ↓
Vector
     ↓
Vector Database

When thousands or even millions of documents are converted into embeddings, their vectors can be stored inside a vector database.

Later, when a user submits a query, that query is also converted into an embedding.

The database can then search for vectors that are similar to the query vector.

At this point, we are no longer searching by keywords.

We are searching by meaning.

How Similarity Emerges

This naturally answers:

How does the database know which vectors are similar?

At first glance, embeddings may look like nothing more than lists of numbers.

Get Kusuma Pindi’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

But the interesting part is not the numbers themselves.

It is the relationships between them.

During training, embedding models learn which words, phrases, and concepts tend to appear in similar contexts.

For example:

Car
Automobile
Vehicle

often appear near concepts such as:

Drive
Road
Engine
Transportation

As a result, their embeddings tend to end up close to one another.

Meanwhile:

Pizza
Recipe
Cheese
Oven

appear in completely different contexts.

Their embeddings tend to form a different region of the vector space.

Over time, similar meanings naturally cluster together.

Semantic Search

This ability to search based on meaning rather than exact words is known as semantic search.

Traditional search systems often ask:

Do these words match?

Semantic search asks:

Do these meanings match?

For example:

Query:
How do I buy a car?

Document:

Guide to purchasing an automobile

The words are different.

The meaning is nearly identical.

Keyword search may struggle with this.

Semantic search can often identify the relationship immediately because the embeddings are located close together.

Embeddings make semantic search possible.

They allow computers to compare meaning mathematically rather than relying only on exact word matches.

We shall see more about Semantic search in next article

Applications of Embeddings

At this point, embeddings may sound like a feature built specifically for Large Language Models.

In reality, embeddings are used in many different types of systems.

Whenever a system needs to compare meaning rather than exact words, embeddings often play a role.

Search

Perhaps the most common application is semantic search.

Instead of matching exact keywords, search systems can retrieve documents based on meaning.

This allows users to find relevant information even when their wording differs from the original document.

Recommendation Systems

Many recommendation systems use embeddings behind the scenes.

For example, music platforms can identify songs with similar characteristics.

E-commerce platforms can recommend products that are semantically related to items a user has already viewed or purchased.

Content Clustering

Embeddings can also be used to group similar content together.

Articles discussing related topics often end up close to one another in vector space.

This makes it possible to organize and categorize large collections of information automatically.

Retrieval for LLMs

One of the most important applications of embeddings today is retrieval.

Before an LLM generates a response, a system can search through thousands of documents and retrieve information that is relevant to the user’s question.

Those documents can then be provided to the model as additional context.

As we’ll see later in this series, this idea forms the foundation of Retrieval-Augmented Generation, or RAG.

Embeddings and Hallucinations

In the previous article, we learned that hallucinations often occur when a model lacks access to the information needed to answer a question.

One way to reduce hallucinations is to provide relevant information before the model generates a response.

But before information can be provided, it first needs to be found.

And that is exactly where embeddings become valuable.

By enabling semantic search, embeddings help systems locate information that is relevant to a user’s query, even when the wording differs significantly.

In many modern AI applications, embeddings act as the bridge between a user’s question and the information needed to answer it.

What’s Missing?

At this point, we can convert text into embeddings.

We can store those embeddings inside a vector database.

And we can use them to perform semantic search.

But one important question still remains.

How does a system actually determine whether two embeddings are similar?

How does it know which document is the best match?

And how does it rank one result above another?

To answer those questions, we need to understand Semantic Similarity.

And that’s exactly what we’ll explore in the next article.

See you there 👋