Understanding LLMs from Scratch — Part 1: How LLMs Process Human Language (Tokenization)

Have you ever wondered how Large Language Models (LLMs) like ChatGPT, LLaMA, or Mistral actually “read” the text we type? Before these complex AI models can generate poetry or write code, they must first translate human language into something machines can digest: numbers.

This critical bridge is called tokenization. Let’s dive into how it works, the challenges it solves, and why the shift to byte-level processing changed everything.

I. Introduction & The Limits of Classical Approaches

1. The Tokenization Challenge: The Human-Machine Bridge

At its core, tokenization is about breaking text down into manageable pieces called “tokens.” The ultimate challenge in NLP (Natural Language Processing) is finding the perfect balance between vocabulary size and context window size. If your tokens are too big, your vocabulary explodes. If they are too small, your sequences become too long for the model’s memory (context window).

2. Word-Level Tokenization

The most intuitive approach is to split text by words (e.g., “I”, “love”, “AI”).

• The Limit: Human languages have practically infinite vocabularies. If a model encounters a word it hasn’t seen before, it gets mapped to an “Unknown” (<UNK>) token, completely losing its meaning. Handling…