What Is an Embedding, Really?
From Raw Text to Vectors- the Full Journey
Tokenizer • Embedding Matrix • d_model • Chunking Strategies • Vector Space • Semantic Search • RAG
self.token_embedding = nn.Embedding(vocab_size, embedding_dim)
One line. In the Mini-GPT article, this line appeared, the model trained, and it worked. But what is actually living inside it? What is an embedding, really — not as a concept, but mechanically, at the level of numbers? And how does this single idea connect to everything in modern production AI — chunking strategies, vector databases, semantic search, RAG pipelines?
This article opens that one line completely. From raw text through tokenizer through embedding matrix through chunking through vector space — the mathematics and the production patterns at every step. Whether you are just starting to understand embeddings or already building RAG systems in production, this is the complete picture.
1. Why Text Has to Become Numbers
Neural networks are, at their core, mathematical functions. They multiply matrices, compute dot products, apply non-linearities. None of that works on the string ‘hello’. There is no matrix multiplication you can perform on a word.
So the first problem in any language AI system is a translation problem: how do you convert human language — symbolic, discrete, sequential — into continuous numerical space where mathematical operations become possible?
That translation happens in two distinct steps, and understanding the difference between them is the first concept this article builds on:
2. The Tokenizer — Splitting Text Into Units the Model Can Process
A tokenizer is the boundary between human language and model input. It takes raw text, splits it into tokens, and maps each token to an integer ID from a fixed vocabulary. The vocabulary is the complete set of all possible tokens the model knows — determined once during training and fixed forever after.
2.1 Types of Tokenization
Real production LLMs use Byte Pair Encoding (BPE). The reason: even a word the model never saw during training can be broken into subpieces it did see, so the model always has something to work with. Mini-GPT uses simple whitespace splitting — conceptually identical, just at toy scale.
2.2 Encode and Decode
3. The Embedding Matrix — Where Meaning Is Stored
Once text is tokenized into integer IDs, those IDs are used to look up rows in a matrix — the embedding matrix. This matrix is the most important single object in any language model. It is where the model’s learned representation of every word in its vocabulary lives.
3.1 The Matrix — Exact Shape and Contents
Looking up an embedding is just selecting a row. When the model sees token ID 17, it retrieves row 17 from this matrix — a vector of 32 numbers. That is the embedding for that word. Nothing more complex than a table lookup.
3.2 What vocab_size Controls
vocab_size sets the number of rows. More unique tokens the model knows = more rows = more parameters. The trade-off:
• Larger vocab: fewer unknown words, but more memory and more parameters to learn
• Smaller vocab: leaner model, but rare words get broken into subpieces (which is usually fine with BPE)
3.3 What d_model Controls
d_model is the number of columns — the richness of each token’s representation. It is one of the most consequential architectural choices in any Transformer because it determines the width of every downstream matrix multiplication in every attention layer.
3.4 Embeddings Are Trainable — This Is the Key Idea
When nn.Embedding is created, every number is initialised to small random values. The model knows nothing. Training changes this: every time a token appears in a training batch, its entire row gets a gradient — a signal showing how to adjust each of those 32 numbers to reduce the prediction error. Over thousands of steps, words that appear in similar contexts converge toward similar vectors in the 32-dimensional space.
4. Positional Embeddings — Restoring What Attention Loses
Here is a fundamental problem: the embedding lookup is entirely order-blind. Whether ‘tea is hot’ or ‘hot is tea’, the embedding vector for ‘tea’ is identical — the same row, regardless of where in the sentence it appears. Self-attention inherits this blindness by default.
Positional embeddings fix this by adding a second learned vector to each token’s embedding — one that encodes position, not meaning:
The positional embedding is a second matrix of shape [block_size x d_model]. Position 0 has its own row, position 1 has its own row, and so on. These rows are also learned during training. After the addition, every token carries both what it is and where it sits — and that fused vector is what flows into the Transformer blocks.
5. Chunking Strategies — Before Embedding, Documents Must Be Split
In production AI systems — RAG pipelines, knowledge bases, document search — you are rarely embedding a single sentence. You are embedding entire documents, PDFs, knowledge articles, support transcripts. These are too long to embed as a single unit. This is where chunking comes in.
Chunking is the process of splitting a large document into smaller, coherent pieces before embedding. Each chunk becomes one vector in your vector store. The quality of your chunks directly determines the quality of your retrieval — bad chunking means bad RAG, regardless of how good your embedding model is.
5.1 Fixed-Size (Sliding Window) Chunking
The simplest strategy: split the document into chunks of a fixed number of characters or tokens, with an optional overlap between consecutive chunks.
• Best for: homogeneous documents with consistent density — logs, transcripts, unstructured text
• Avoid when: document has clear section/paragraph structure that this method would break mid-sentence
• Overlap tip: 10–15% of chunk_size is a good starting point for overlap
5.2 Recursive Character Chunking
A smarter approach: instead of splitting at a fixed character count, it tries a priority-ordered list of delimiters. It first tries to split at paragraph breaks, then sentences, then spaces, then characters — stopping as soon as chunks are within the target size. This preserves semantic boundaries as much as possible
• Best for: general-purpose RAG over mixed documents — this is the most widely used default
• Why it works: concepts rarely get severed mid-thought; the model prefers natural language boundaries
Get APRAJITA TIWARI’s stories in your inbox
Join Medium for free to get updates from this writer.
• Used by: LangChain default, most RAG tutorials, AWS Bedrock default chunking mode
5.3 Semantic Chunking
Instead of splitting on character counts or delimiters, semantic chunking evaluates the meaning of consecutive sentences. It computes embeddings for each sentence and splits when the cosine similarity between adjacent sentences drops below a threshold — signalling a topic change.
• Best for: long documents with multiple distinct topics — research papers, legal documents, reports
• Trade-off: computationally expensive — requires embedding every sentence before chunking even starts
• Result quality: highest semantic isolation — each chunk is genuinely about one thing
5.4 Document Structure-Aware Chunking
Uses the document’s own formatting — Markdown headers, HTML tags, PDF section boundaries, table boundaries — to define chunk boundaries. The document’s structure is treated as a signal for where topics begin and end.
• Best for: structured documents — technical documentation, wikis, knowledge bases with clear sections
• Key advantage: chunk boundaries align with the author’s intended topic divisions
• Also handles: HTML (BeautifulSoupSplitter), code files (language-aware splitters)
5.5 LLM / Agentic Chunking
The most sophisticated approach: use a language model itself to analyse the document and decide where chunk boundaries should be, based on the specific retrieval task. The LLM understands context, identifies when a new concept begins, and can generate clean, semantically isolated chunks tailored to Q&A use cases.
• Best for: high-value, complex documents where retrieval quality is critical
• Trade-off: slowest and most expensive — LLM inference for every document at ingestion time
• Use when: accuracy matters more than ingestion speed — legal, medical, compliance content
5.6 Chunking Strategy Decision Guide
5.7 How AWS Bedrock Knowledge Bases Handles Chunking
Amazon Bedrock Knowledge Bases provides managed chunking at the infrastructure level — you configure the strategy and Bedrock handles the splitting, embedding, and storage automatically when you ingest documents from S3.
Outside of managed services — when building off-cloud or custom pipelines — the same strategies apply using LangChain, LlamaIndex, or custom splitting logic. The chunking happens before you call any embedding API. The output is identical: a list of text strings, each to be embedded independently into a vector.
6. Vector Space — How Embeddings Are Stored and Searched
After chunking and embedding, you have a collection of vectors — one per chunk — each of length d_model (typically 768, 1536, or 3072 for production embedding models). These vectors need to be stored somewhere that supports fast similarity search. That is the vector database.
6.1 What a Vector Database Actually Is
A vector database is a data store optimised for one specific operation: given a query vector, find the most similar stored vectors quickly. Unlike a traditional SQL database that searches by exact value, a vector database searches by geometric proximity in high-dimensional space.
• Popular options: Pinecone, Weaviate, Qdrant, Chroma (open-source), FAISS (Meta, library), pgvector (Postgres extension), Amazon OpenSearch with k-NN plugin
• AWS native: Amazon OpenSearch Service with k-NN, or Amazon Bedrock’s managed vector store
6.2 How Vectors Are Stored — The Index
When you insert vectors into a vector database, they are stored in an index — a data structure optimised for nearest-neighbour search. Three common index types:
HNSW (Hierarchical Navigable Small World) is the most widely used index in production RAG systems. It trades a small amount of recall accuracy for dramatically faster query times — milliseconds instead of seconds at millions of vectors.
6.3 Cosine Similarity — The Mathematics of Semantic Search
The similarity between two vectors is measured by the angle between them, using cosine similarity. The mathematical intuition: two vectors pointing in the same direction in d_model-dimensional space represent similar meaning, regardless of their length.
6.4 The Full Semantic Search Flow
Putting all the pieces together, here is exactly what happens when a user sends a query to a RAG system:
6.5 The Famous Word Vector Arithmetic
After training on large corpora, embedding vectors encode semantic relationships geometrically. The most celebrated example from Word2Vec (Mikolov et al., 2013):
Mini-GPT’s corpus is too small to produce this effect visibly. But the training process is identical — same backpropagation, same gradient updates, same embedding matrix structure. At scale, these geometric regularities emerge naturally from the statistics of language.
7. The Complete Pipeline — Raw Text to Answer
8. Quick Reference — Every Term in This Article
9. Conclusion
One line of code — nn.Embedding(vocab_size, embedding_dim) — opens into an entire world: tokenization, lookup tables, learnable parameters, positional encoding, cosine similarity, chunking strategies, vector databases, and the full architecture of modern RAG pipelines.
Embeddings are not a black box that ‘converts text to numbers’. They are a learned geometric space where meaning becomes distance, where relationships between concepts become directions, and where a question and its answer converge toward each other because the training process made them do so.
Every production RAG system you build — on AWS Bedrock, on any cloud, or entirely self-hosted — is built on top of exactly these ideas. Understanding them at this level means you are not just using the system. You know what it is doing, why it sometimes fails, and how to make it better.
Next in this series: the attention mechanism itself. Query, Key, Value — what the mathematics is actually computing, why it works, and what it looks like in code.
