The Internet as a Textbook: How “Guess the Next Word” Became GPT
How We Got Here: Episode 11 of ~26. We are walking the AI timeline one contribution at a time.
Your phone does this to you a dozen times a day and you have never once thought about it.
You start typing a message (“sounds good, I’ll see you”) and before your thumb reaches the next key, your phone has already offered “tomorrow.” You start a sentence in your email and the gray ghost-text finishes the clause for you, and it’s right, and you press tab. Small, ordinary, forgettable. You’ve done it a thousand times.
But there was probably a moment, once, when the guess was too good. When it finished not just the word but the whole thought. A phrasing you were about to reach for, a turn you hadn’t typed yet. And for half a second something flickered. How did it know I was going to say that?
That flicker is the entire subject of this episode.
Because the thing that made your autocomplete uncannily good is the same idea, just made enormous, that runs the chatbot you probably had open in another tab this morning. The model you type questions into and get essays back from was trained on one task. The task your phone keyboard is doing in miniature. Given some words, predict the word that comes next.
That’s it — the whole training objective behind the most talked-about technology of the last decade. Predict the next word. A child does a version of it when they finish your sentence at the dinner table.
The first time somebody took that childish little task, pointed it at a huge chunk of the internet, and turned the crank hard enough, they got something that could write. And it startled even the people who built it. Enough that, in 2019, they were afraid to hand it over.
Quick recap
If you’re just joining, a quick catch-up.
In Episode 10, we finished building the transformer, the architecture behind basically every large language model running today. The key part was attention: a mechanism that lets a model, when it’s processing one word, look back at every other word in the passage and decide which ones matter for what it’s doing right now. No more shoving a whole sentence through a narrow bottleneck one word at a time, the way the older recurrent networks did. Every word gets to see every other word, directly.
I also told you the transformer came in two halves. An encoder, which reads an input and builds a rich understanding of it, and a decoder, which generates output one word at a time. The original 2017 paper wired both together to translate English into German.
And I left you with a promise. Someone was going to take just the second half of that machine (the decoder, the part that generates), throw away the translation job entirely, make it huge, and feed it the open internet with one instruction, predict the next word. That is where the modern story actually starts, with GPT.
The whole training task, in one sentence
The training loop is short enough to write out, and the shortness is the point.
You take the transformer’s decoder (the generating half from last episode). You take a mountain of ordinary text: books, articles, web pages, forum posts, whatever people have written. And you play a fill-in-the-blank game with it, billions of times.
One round of the game looks like this. You take a real sentence from your pile:
The cat sat on the ___
And you hide what comes next. You show the model those words and ask it one thing. What’s the next word? The model, which starts out knowing nothing, spits out a guess. Early on the guess is garbage (maybe it says “helicopter”). You check against the real sentence, which said “mat.” So the model was wrong, and you can measure exactly how wrong.
Then you do the thing we covered all the way back in Episode 3: you take that error and push it backward through the network, nudging every internal weight a hair in the direction that would have made “mat” a little more likely and “helicopter” a little less. That’s backpropagation, or credit assignment: figuring out which of the millions of internal knobs deserve blame for the mistake and adjusting each one’s share. Nothing new under the hood. It’s the same learning engine from 1986, just running on a much bigger machine against a much bigger pile of text.
Now do that again. And again. Billions of times, over billions of sentences, each time hiding the next word and forcing the model to guess and correcting it. There is nothing else to the training process. There’s no human labeling what’s a cat and what’s a dog, no answer key someone had to write. The text is its own answer key. The next word is always sitting right there in the original, waiting to grade the guess. That’s why you can do it at the scale of the whole internet, and why nobody has to annotate anything.
This has a name: next-token prediction. And it sounds almost too dumb to matter. The first time you really hear it, the reaction is usually a small letdown. That’s what’s inside the thing writing people’s cover letters and debugging their code? Guess-the-next-word?
Yes. Genuinely, that’s it.
Why guessing the next word is secretly enormous
The task looks trivial. Predicting the next word is not trivial — not once you demand the model do it well across everything humans have written. And the reason is a little sneaky.
To guess the next word well, you’re forced to understand what came before. There’s no shortcut. The prediction can’t be good unless the understanding is real.
Take the simplest possible example:
The capital of France is ___
To fill that blank with “Paris,” a model has to have absorbed a fact about the world. Not the grammar of the sentence. The fact. There’s no way to be reliably right about that blank without having, somewhere in your weights, the knowledge that France’s capital is Paris. The fill-in-the-blank game forced the model to learn geography, because geography is what it takes to win the game on that sentence.
Now scale that up. To predict the next word across the whole internet, the model runs into every kind of sentence there is, and each one demands something to get right.
To finish “the opposite of hot is ___,” it has to have learned what words mean. To finish “if x = 3, then x + 2 = ___,” it has to have picked up a little arithmetic. A line of Python demands the syntax of the language. And the last sentence of a mystery novel (“and the murderer was, all along, ___”) can’t be guessed from grammar. You can only guess it by having tracked the entire plot: the clues, who was where, which character the whole book was pointing at. By having modeled the story.
Predicting text well requires modeling the world that produced the text. Grammar, facts, arithmetic, cause and effect, the arc of an argument, the personality of a narrator. All of it helps you guess the next word better, so all of it gets pulled into the network as a side effect of getting good at the game. Nobody sat down and taught the model geography or logic or story structure. Those things were the price of admission for lowering the error on next-word prediction, so the training process reached in and built them.
The whole field was slow to believe this. A task simple enough to explain to a child, run at enough scale, backs its way into something that looks a lot like understanding, because understanding was the cheapest way to win.
GPT-1: the idea, proven small
In June 2018, a team at OpenAI put out a paper with a title that tells you exactly what they did: “Improving Language Understanding by Generative Pre-Training.” The lead author was Alec Radford, working alongside Karthik Narasimhan, Tim Salimans, and a name you’ll recognize, Ilya Sutskever.
Get Cordero Core’s stories in your inbox
Join Medium for free to get updates from this writer.
The model was GPT-1. Generative Pre-trained Transformer, version one. It was a transformer decoder with about 117 million parameters (the internal weights, the knobs backprop tunes). By today’s standards that’s tiny. It was trained on a corpus of around 7,000 unpublished books, playing exactly the next-word game I just described.
But the two-step move underneath it is what made the paper matter.
Step one: pre-train. Take the model and let it play guess-the-next-word on all those books until it’s good at it. At the end of this, you don’t have a model that does anything useful in particular. You have a model that has soaked up an enormous amount about how language works, purely from trying to predict text. Grammar, some facts, some reasoning, all of it latent in the weights with no specific job attached.
Step two: fine-tune. Now take that pre-trained model and give it a small amount of labeled data for a specific task you actually care about, say deciding whether a movie review is positive or negative. Because the model already understands language from step one, it needs only a little nudging to learn the narrow task. The model already speaks English. You’re just pointing an already-fluent system at a chore.
If that pattern sounds familiar, it should. We watched it get born back in Episode 6, in 2006, when the deep belief network people showed you could pre-train a network layer by layer on unlabeled data and then fine-tune it for a task. That was the idea. Learn general structure first, from cheap unlabeled data, then specialize. In 2006 it was a clever trick for getting deep networks to train at all. Twelve years later, GPT-1 took that same pretrain-then-adapt idea, aimed it at language with the transformer as the engine, and made it work. That old approach finally had the architecture and the scale it had been waiting for.
GPT-1 was the proof of concept. It beat the specialized state of the art on a bunch of language tasks, which was the headline at the time.
GPT-2: bigger, and then something new
In 2019, the same group scaled the idea up. Way up. GPT-2 was the same basic machine (a transformer decoder trained on next-word prediction), but grown to 1.5 billion parameters (more than ten times GPT-1) and trained not on books but on WebText, a pile of around 8 million web pages pulled from links people had actually shared. Roughly 40 gigabytes of text. The internet, more or less, as a textbook.
And something happened that wasn’t in the plan.
Remember step two from GPT-1 (fine-tune the model for each specific task)? GPT-2 started doing tasks without the fine-tuning. You could just… ask. Type a few lines of French followed by a colon and the English, and it would continue the pattern and translate. Give it an article and write “TL;DR:” at the end, and it would summarize. It picked both up as a byproduct of getting extremely good at predicting the next word across a huge, messy slice of the web, because translation and summarization were patterns in that web, and modeling them helped it guess better.
The paper’s title says it out loud: “Language Models are Unsupervised Multitask Learners.” In plain terms, a thing we only trained to predict text turned out to have learned a pile of tasks nobody asked it to learn.
The word for this is zero-shot, meaning the model does a task it was given zero training examples for, working from the prompt alone. And it was the first strong hint of a pattern that would come to dominate everything after. You didn’t necessarily need to build a specialized model for each job. You could build one big general model and talk to it. The capability was already in there, waiting, put there by scale and the next-word game.
So GPT-1 proved the recipe, and GPT-2 suggested the recipe had no obvious ceiling. Maybe you didn’t need cleverer ideas, just a bigger version of the same dumb, beautiful task.
The moment they got scared
The story takes a turn nobody expected this early.
GPT-2 could produce paragraphs of text that read like a person wrote them. Not always coherent, but often fluent enough to pass a quick glance. And OpenAI looked at that and got nervous. If a model can generate convincing text on demand, cheaply, at volume, what does that do to fake news, to spam, to someone impersonating a real person across a thousand comment threads at once?
So they did something unusual. In February 2019, when they announced GPT-2, they didn’t release the full model. They put out a small version, held the big one back, and said, roughly, we’re not sure it’s safe to hand this to everyone yet. The press ran with a phrase that stuck: too dangerous to release.
Then they did a staged release, letting the model out in pieces over most of a year, watching each time to see what happened. The 124-million-parameter version in February. The 355-million in May. A 774-million partial in August. And finally, on November 5, 2019, having said they’d seen no strong evidence of misuse, they released the full 1.5-billion-parameter model.
You can argue about whether the fear was warranted. Plenty of people thought it was overblown, even a publicity move, and it’s hard to know from the outside. But that’s almost beside the point now. What matters is that this was one of the first times the field had a loud, public argument about whether a model was too capable to be safely shared, about release norms, about who decides, about the responsibility that comes with building something that works better than you expected.
Sit with the timing. This is 2019. We’re eleven episodes into a series that started with a single artificial neuron, and the technology has only just learned to write a passable paragraph. And already the people closest to it are uneasy enough to hide their own work. The safety questions that dominate every AI conversation now didn’t arrive with ChatGPT or the big frontier models. They showed up here, with a 1.5-billion-parameter text-completion engine that couldn’t hold a thought together for a full page.
What GPT-2 absolutely was not
GPT-2 was not an assistant, and the last few sections have probably made it sound like one. It couldn’t follow instructions, and it had no idea it was supposed to be helpful or truthful, or to answer your question rather than just continue your sentence. A raw text-continuation engine, very different from what you use today.
Give it a question and it might answer, or it might generate three more questions just like yours, because a list of questions is also a plausible way for text to continue. It would state made-up things with total confidence, because it was never optimizing for truth, only for plausible next words. Over a short span it was impressive. Over a long one it wandered, contradicted itself, forgot what it had said, drifted into nonsense. A brilliant mimic of the shape of writing with no reliable commitment to the substance.
Crossing the gap to something useful took two more things we haven’t reached yet. It took even more scale, the jump to a model a hundred times larger, which is next week. And it took a way to teach the model to actually follow instructions and be helpful instead of autocompleting the internet, which is the episode after that. So there was a real gap between “this can generate fluent text” and “this is useful to a normal person trying to get something done.”
What this means for the tab you have open
Bring that back to the model you were using this morning.
Whatever you talk to (whichever assistant, whichever company), the thing underneath it is a direct, scaled-up descendant of what we just walked through. A transformer decoder, trained at its foundation on next-token prediction. Predict the next word, billions of times, over a staggering amount of text, until modeling the world is simply what the arithmetic settles on. Everything layered on top since (the instruction-following, the chat interface, the guardrails) sits on that base.
That matters, because it cuts against how these systems get talked about. It’s tempting to imagine something exotic inside, a hidden reasoning module, a ghost you can’t quite name. And the models are astonishing at what they do. But the training objective is not exotic. It’s almost aggressively mundane. Predict the next word. The magic, to whatever extent there’s magic, is in what that mundane task forces into existence when you run it at scale on enough of what humans have written.
The whole series is walking toward that demystifying idea. Not that the models are simple. They aren’t. But that the principle at the bottom is one you can hold in your head. You already understood it before this episode started. You’ve felt it every time your phone finished your sentence and you paused, for half a second, and wondered how it knew.
It knew for the same reason GPT does. To guess your next word well, it had to learn a little bit about you and your world. Turn that crank hard enough, on enough of the world, and the little bit becomes a lot.
The most powerful software of the decade is trained by the game a kid plays when they blurt out the end of your sentence before you finish it.
We spent years assuming intelligence would have to be engineered in, feature by careful feature. It turned out you could also just make the guessing game big enough that understanding fell out as a side effect. That’s either the most elegant shortcut in the history of the field or a slightly unsettling thing to learn about understanding itself.
Next Tuesday: GPT-2 hinted that bigger meant better. Nobody knew whether that was a rule or a lucky break. In 2020, someone made the model a hundred times larger and did something GPT-2 hadn’t: they wrote down the mathematics of it. GPT-3 and the scaling laws, the moment “just make it bigger” stopped being a hunch and became a strategy with an actual graph behind it. The point where the whole industry decided which direction to run.
If Episode 11 was the discovery that a dumb task hides a deep one, Episode 12 is the discovery that the dumb task got predictably better the more you fed it, and what an entire field did once it believed that.
If this series is useful to someone on your team (the engineer who ships against these models every day but has never been walked through what’s actually inside them), send it their way. The whole point of taking the timeline one Tuesday at a time is that by the time we reach the tools you use now, none of it feels like magic. It feels like the engineering it is.
You can follow along on Medium or subscribe to the Substack.
Cordero is a senior research software engineer at the Scientific Software Engineering Center at the University of Washington’s eScience Institute. If this landed, forward it to whoever taught you to be impressed by these things, and see whether the bottom of it changes their mind.
