Build LLMs from Scratch · Issue 2 · also on the newsletter

Issue 1 gave you the loop: pieces so far → bet on the next one → append → repeat.

Same job today. New grain: token IDs. And one myth to retire — that an LLM is a polite search box over a giant hard drive.

Recap, then the new idea

Last time the pieces were characters. You printed a count table. You watched ugly text crawl out of random.choices. Honesty, not fluency.

Tonight the pieces become integers. Same loop. We put that loop next to a dumb exact-string lookup so you can feel retrieve versus generate.

An LLM is a next-token machine. Scale makes it look like understanding. It is still not a database.

What an LLM actually is

Three facts, then a lab.

Tokens are pieces, not words. Your eyes want “unbelievable” to be one unit. A real tokenizer may crack it into several IDs. The model never reads like you do. It reads integers, turns each into a vector, and bets on the next ID. Words are a human convenience. Tokens are the machine’s alphabet.

“Large” is about knobs, not manners. A small table finishes your text. A model with millions, then billions, of weights finishes your argument — sometimes. Scale buys coverage and style, not honesty. Fluently wrong is still wrong; it just sounds finished.

Old NLP was a drawer of single tools (sentiment here, translation there). An LLM is closer to a multi-tool — still worth sharpening later. Sharpening is Issue 3’s neighborhood. Tonight stays on identity.

Buzzword nest, one breath: AI ⊃ machine learning ⊃ deep learning ⊃ generative AI ⊃ LLMs (the language corner). Image and audio generators are GenAI too. Not every GenAI thing is an LLM.

What it is not: a search engine. Not a database with manners. Lookup returns a line that already existed. Generate samples a continuation that is plausible given the stats. Sometimes that matches a memorized phrase. Often it is a remix. The mechanism is still “what ID is likely next?”

Diagram: lookup vs generate

Lookup returns stored bytes. Generate samples the next token ID. Only generate is what an LLM does.

Same corpus, two machines

Corpus on disk: "maya opened the notebook. …"

LOOKUP (database move) 1. Query: "maya opened the" 2. Find that exact leftover in the text 3. Return the bytes that follow — or shrug / None on a miss

GENERATE (LLM move) 1. Encode the query into token IDs 2. Look at counts (or weights) after the last ID 3. Sample a next ID → append → repeat 4. Miss on the exact string? It still answers — with a guess

Same corpus. Two machines. Only one of them is what an LLM is doing.

Lab: token IDs are not a database

Upgrade Issue 1’s character table to token IDs, then contrast exact leftover-string search with sampling the next ID.

Save as token_ids_not_a_db.py and run python3 token_ids_not_a_db.py.

"""Issue 2 lab — token IDs, next-ID counts, lookup vs generate. Stdlib only."""
 
import random
import re
from collections import defaultdict
 
CORPUS = """
Ravi packed a lunch. Maya packed a notebook.
Ravi opened the lunch. Maya opened the notebook.
The lunch smelled good. The notebook smelled like ink.
Ravi wrote unbelievable notes. Maya wrote unbelievable jokes.
Notes help. Jokes help. Lunch helps too.
"""
 
START_ID = 0  # reserved id for start-of-text
 
 
def tokenize(text: str) -> list[str]:
    """Split into word-ish pieces: letters/digits kept, punctuation alone."""
    text = text.lower().strip()
    return re.findall(r"[a-z0-9]+|[.,!?;:]", text)
 
 
def build_vocab(pieces: list[str]) -> dict[str, int]:
    """Map each unique piece to an integer ID. 0 is START."""
    vocab = {"<start>": START_ID}
    for p in pieces:
        if p not in vocab:
            vocab[p] = len(vocab)
    return vocab
 
 
def encode(pieces: list[str], vocab: dict[str, int]) -> list[int]:
    return [vocab[p] for p in pieces]
 
 
def invert_vocab(vocab: dict[str, int]) -> dict[int, str]:
    return {i: p for p, i in vocab.items()}
 
 
def build_next_id_table(ids: list[int]) -> dict[int, dict[int, int]]:
    """Like Issue 1's char table, but over token IDs."""
    seq = [START_ID] + ids
    table: dict[int, dict[int, int]] = defaultdict(lambda: defaultdict(int))
    for a, b in zip(seq, seq[1:]):
        table[a][b] += 1
    return table
 
 
def sample_next(table: dict[int, dict[int, int]], current: int) -> int:
    followers = table.get(current)
    if not followers:
        return START_ID
    ids, weights = zip(*followers.items())
    return random.choices(ids, weights=weights, k=1)[0]
 
 
def generate_ids(table: dict[int, dict[int, int]], n: int = 12) -> list[int]:
    cur = START_ID
    out = []
    for _ in range(n):
        cur = sample_next(table, cur)
        if cur == START_ID:
            break
        out.append(cur)
    return out
 
 
def lookup_exact(corpus: str, leftover: str) -> str | None:
    """Dumb 'database': find leftover as an exact substring; return what follows."""
    hay = " ".join(corpus.lower().split())
    needle = " ".join(leftover.lower().split())
    idx = hay.find(needle)
    if idx < 0:
        return None
    after = hay[idx + len(needle) :].strip()
    if not after:
        return ""
    # return the next few words after the match
    return " ".join(after.split()[:6])
 
 
def decode(ids: list[int], id_to_piece: dict[int, str]) -> str:
    parts = [id_to_piece[i] for i in ids]
    out = []
    for p in parts:
        if p in ".,!?;:":
            out.append(p)
        elif not out:
            out.append(p)
        else:
            out.append(" " + p)
    return "".join(out)
 
 
if __name__ == "__main__":
    random.seed(2)
 
    pieces = tokenize(CORPUS)
    vocab = build_vocab(pieces)
    id_to_piece = invert_vocab(vocab)
    ids = encode(pieces, vocab)
    table = build_next_id_table(ids)
 
    demo = "Ravi wrote unbelievable notes."
    demo_pieces = tokenize(demo)
    demo_ids = encode(demo_pieces, vocab)
 
    print("1) Sentence → pieces → IDs")
    print("   text:  ", demo)
    print("   pieces:", demo_pieces)
    print("   ids:   ", demo_ids)
    print("   note:  'unbelievable' is ONE piece here (our toy splitter);")
    print("          real tokenizers often crack it into several IDs.")
    print()
 
    # pick an ID that appears mid-corpus
    probe_piece = "wrote"
    probe_id = vocab[probe_piece]
    followers = {id_to_piece[i]: c for i, c in table[probe_id].items()}
    print(f"2) What follows ID {probe_id} ('{probe_piece}')?")
    print("  ", followers)
    print()
 
    gen_ids = generate_ids(table, n=14)
    print("3) Generate (sample next IDs from the count table)")
    print("   ids: ", gen_ids)
    print("   text:", decode(gen_ids, id_to_piece))
    print()
 
    hit_query = "maya opened the"
    miss_query = "ravi opened the notebook"
    print("4) Lookup vs generate")
    print("   lookup hit for", repr(hit_query), "→", repr(lookup_exact(CORPUS, hit_query)))
    print("   lookup miss for", repr(miss_query), "→", repr(lookup_exact(CORPUS, miss_query)))
    last = tokenize(miss_query)[-1]
    last_id = vocab[last]
    cur = last_id
    cont = []
    for _ in range(5):
        cur = sample_next(table, cur)
        if cur == START_ID:
            break
        cont.append(cur)
    print(
        "   generate after ID",
        last_id,
        f"('{last}') →",
        repr(decode(cont, id_to_piece)),
        "(fluent guess, not a retrieved line)",
    )

With random.seed(2) you should see: the demo sentence as pieces and IDs; followers of wrote → unbelievable; a mashed-up generated line; a lookup hit for maya opened the, a lookup miss for ravi opened the notebook (never written that way), and a generate guess after notebook anyway.

Lookup vs generate (cheat sheet)

• Lookup: needs an exact leftover · returns stored bytes · miss = empty hands
• Generate: needs the last token ID · samples a plausible next ID · miss on the string? still talks

What you built today

• A tiny tokenizer: text → word-ish pieces → integer IDs.
• A next-ID count table: Issue 1’s engine, coarser grain.
• A side-by-side: exact leftover lookup vs sampling the next ID.
• A sharper definition: LLM = scaled next-token machine, not a polite hard drive.

You did not build a real subword tokenizer, a transformer, or a chatbot. You named the job and showed why “it answered” ≠ “it found the row.”

Next issue

Issue 3: pretrain vs finetune — same engine, different stages of teaching it.

Change CORPUS to a paragraph you wrote, keep the miss-query deliberately almost right, and watch lookup shrug while generate keeps talking. That shrug is the point.

—

Further reading (optional): Sebastian Raschka, Build a Large Language Model (From Scratch) — a full book-length path if you want the long form after these labs. This issue is not a summary of that book.

Build LLMs from Scratch · Issue 1 · also on the newsletter

Most people “learn LLMs” the way you’d learn restaurants by ordering takeout.

You call an API. Something fluent comes back. You ship a demo. Then someone asks what is a token? or why is it confidently wrong? and the whole thing turns into a shrug.

This series is the kitchen. One mini-lab per issue. Plain English, one picture, code you can paste into a file and run. No paid model. No framework. No branding tour.

Today you will not train GPT. You will build the job an LLM is actually doing — predict the next piece — so the rest of the series has somewhere to sit.

The trap

The usual path looks like this:

1. Open a chat UI.
2. Wrap it in a script.
3. Tell people you “built an LLM app.”
4. Freeze when the output is weird and you cannot name the moving parts.

That path is fine for product demos. It is a terrible foundation if you want to debug, interview, or train anything yourself.

Two reasons to go slower:

• Debug intuition. You will know what prompting can fix, what finetuning is for, and what only a better base model can buy.
• Builder pride. Open weights and closed APIs are both tools. Understanding the machine is the skill.

The only job (for now)

Strip the costume and a language model is answering one question:

Given the pieces so far, what comes next?

Do that well, at huge scale, and it looks like writing, reasoning, and coding. The look is a side effect. The job is next-piece prediction.

In later issues those “pieces” will be tokens (subword IDs). Tonight they are characters, so you can see the loop with your eyes.

The loop (read top to bottom, then jump back):

1. Start with raw text
2. Break it into pieces
3. Look at the context so far
4. Bet on the next piece
5. Append it
6. Go back to step 3 and repeat

That loop is the engine. Chat UIs are the dining room.

Lab: a next-token engine with zero libraries

We will count how often one character follows another in a tiny story, then sample the next character from those counts. No torch. No API key. If this runs, you have a baby language model.

Save as baby_next_token.py and run python3 baby_next_token.py.

"""Issue 1 lab — character-level next-token engine. Stdlib only."""
 
import random
from collections import defaultdict
 
STORY = """
sam sat. sam saw a cat. the cat sat. sam saw the sun.
the sun sat. a cat saw sam. sam sat in the sun.
"""
 
START = "~"  # fake "start of text" so we can open a sentence
 
 
def build_table(text: str) -> dict[str, dict[str, int]]:
    """For each character, count what comes next."""
    text = START + "".join(text.lower().split())  # squash whitespace
    table: dict[str, dict[str, int]] = defaultdict(lambda: defaultdict(int))
    for a, b in zip(text, text[1:]):
        table[a][b] += 1
    return table
 
 
def sample_next(table: dict[str, dict[str, int]], current: str) -> str:
    followers = table.get(current)
    if not followers:
        return "."
    chars, weights = zip(*followers.items())
    return random.choices(chars, weights=weights, k=1)[0]
 
 
def generate(table: dict[str, dict[str, int]], n: int = 80) -> str:
    ch = START
    out = []
    for _ in range(n):
        ch = sample_next(table, ch)
        if ch == START:
            continue
        out.append(ch)
    return "".join(out)
 
 
if __name__ == "__main__":
    random.seed(7)
    table = build_table(STORY)
    print("What follows 's'? ", dict(table["s"]))
    print("What follows 'a'? ", dict(table["a"]))
    print()
    print("Sample 1:", generate(table))
    print("Sample 2:", generate(table))

You should see something like: after s the model mostly wants a (from sam / sat / saw). After a it has seen m, t, w, c. The “sentences” it writes will be ugly. That is the point. Fluency is what happens when this same loop is trained on a library, not a paragraph.

API takeout vs this lab

• API takeout: hidden weights · hidden tokenizer · “it just works”
• This lab: a dict of counts you can print · characters you can see · you watch it fail in public

What you built today

• A mental model: next piece, then append, then repeat.
• A count table: the cheapest possible “weights.”
• A sampler: the cheapest possible “generate” button.
• A reason to distrust fluency: ugly output is honesty. Pretty output is scale.

You did not build a transformer, a tokenizer, or a chatbot. Do not skip ahead and call a wrapper a lab.

Next issue

Issue 2: what an LLM actually is — tokens vs words, why “large” matters, and why a next-token engine is not a database with manners.

If you want a preview, change STORY to a paragraph you wrote and run the file again. The table changes. The loop does not. That is the whole series in one sentence.

—

Further reading (optional): Sebastian Raschka, Build a Large Language Model (From Scratch) — a full book-length path if you want the long form after these labs. This issue is not a summary of that book.