Deep Belief Networks: How Deep Learning Got Its Name
How We Got Here — Episode 6 of ~26. We are walking the AI timeline one contribution at a time.
You’ve done this move a hundred times, probably this week.
You need a model to do something: classify support tickets, tag images, pull structure out of a pile of text. You don’t start from nothing. You reach for a checkpoint somebody else already trained. from_pretrained. You download the weights, point the thing at your own data, and fine-tune. A big pretrained model does the heavy lifting of understanding the world, and you spend a little compute teaching it your specific corner of it.
That workflow is so normal now it doesn’t feel like a technique. It feels like the air. Of course you start from a pretrained model. What else would you do? Train a whole network from scratch on your three thousand labeled examples?
Here’s the thing I want to sit with you today. That entire habit (pretrain first, then specialize) has a birthday. It was born in 2006, in Toronto, out of a problem that had nothing to do with saving you compute. It was born because deep networks flatly would not train, and one stubborn researcher found a way to sneak past the wall.
This is the episode where the pattern finally breaks in the believers’ favor. After two winters, after twenty years of “the brain-inspired stuff is a dead end,” the door cracks open. And the word deep, which had been a slur, basically, gets its comeback.
Let me show you how.
Quick recap
If you’re just joining, here’s where we left off.
In Episode 5, we walked through the 1990s, the decade neural networks lost. The elegant math of support vector machines took the prestige and the funding, and neural nets got pushed to the margins as a messy, unfashionable curiosity. That was the second AI winter.
But we ended on a specific, technical reason the winter was so hard to escape. It’s called the vanishing gradient, and it’s the wall this whole episode is built against, so let me make sure it’s solid before we go on.
Remember backprop from Episode 3. You run an input forward through the network, measure how wrong the answer was at the end, and then push that error backward, layer by layer, so every weight learns its share of the blame. That’s how a network learns.
Now stack the network deep: many layers between the input and the answer. As the error signal travels backward through all those layers, it gets multiplied down at each step, again and again, until by the time it reaches the early layers it’s a whisper. Nearly zero. The layers near the input (the ones that are supposed to learn the most basic features) barely get any correction at all. They just sit there, essentially random, while the training grinds on.
So you had this cruel situation. Backprop, in theory, could train a network of any depth. In practice, the deeper you went, the less the front of the network learned. Depth was where the power was supposed to be, and depth was exactly what you couldn’t train.
That’s where Episode 6 opens. Not “how do we train networks better.” The harder question underneath it: how do you get a deep network to work at all?
The wall, stated plainly
Let’s be concrete about why this mattered, because the fix only lands once you feel how stuck the field was.
Everyone agreed, at least in principle, that depth was the point. A shallow network can memorize. A deep network can build — simple features in the early layers, combined into richer ones as you climb, the way we saw the visual cortex do it back in Episode 4. Edges into shapes, shapes into objects. That layered abstraction was the whole promise.
But the vanishing gradient meant you couldn’t reach it. Try to train a deep network from scratch, all layers at once, and the early layers never got a usable signal. You’d end up with a deep network whose bottom floors were noise and whose top floors were desperately trying to make sense of that noise. It performed worse than a shallow network you could actually train.
So the practical wisdom of the early 2000s was blunt: don’t bother going deep. It doesn’t train. Use something shallow, use an SVM, move on.
The question nobody had a good answer to was almost embarrassingly simple. If you can’t train all the layers together, because the signal dies on the way down. What if you didn’t train them all together?
Hinton’s move: build the network one floor at a time
Here’s the idea, and it came from Geoffrey Hinton.
You’ve met him already in this series. Episode 3. He’s one of the three names on the 1986 paper that made backpropagation famous. Great-great-grandson of George Boole, the logician whose algebra every computer runs on. He’d moved to Toronto in part to keep his research clear of military funding, and he’d spent the two winters we’ve been describing doing deeply unfashionable work while most of the field looked elsewhere. If anyone was the keeper of the flame through the cold, it was him.
In 2006, Hinton, along with Simon Osindero and Yee-Whye Teh, published a paper with a modest title: “A Fast Learning Algorithm for Deep Belief Nets.”And the core trick has a name worth remembering: greedy layer-wise pretraining.
Here’s the analogy I keep reaching for. You don’t build a skyscraper by pouring all the floors at once and hoping it stands. You set the ground floor solid. Then you build the next floor on top of that finished floor. Then the next. Each level rests on something already stable underneath it.
Hinton did the same thing to the network.
Instead of training all the layers together (where the deep signal dies), he trained them one layer at a time, from the bottom up. Take the first layer. Train it, by itself, to do one job: learn to represent the raw input well. Freeze it. Now take the second layer, and train it to represent the output of the first layer. Freeze that. Third layer learns to represent the second. And so on, up the stack.
Each layer only ever has to learn a shallow, local problem — represent the thing directly below me. There’s no deep chain for the signal to vanish across, because at each step you’re only training one layer. The vanishing gradient never gets a chance to bite.
And crucially, this pretraining is unsupervised. No labels. Each layer isn’t learning “this is a cat.” It’s just learning the structure of whatever data flows into it: the recurring shapes, the patterns that tend to appear together. The network is teaching itself what its input generally looks like, floor by floor, before anyone tells it what to do with that knowledge.
Then comes the second half of the move. Once all the layers are pretrained and stacked, now you run backprop across the whole thing. This is the fine-tuning step. And here’s why it works this time. Every layer already started somewhere sensible. The weights aren’t random noise anymore; they’re already tuned to represent real structure in the data. So fine-tuning doesn’t have to learn a deep network from scratch. It only has to nudge, gently adjusting weights that are already in a good neighborhood.
Nudging survives a weak gradient. Learning from nothing does not. That’s the whole escape. You dodge the vanishing gradient by making sure the early layers never had to learn from a dying signal in the first place. They learned locally, on the way up, and backprop just polishes the result.
Read that back and notice what changed. The wall wasn’t knocked down. It was walked around.
What’s actually in each layer: a gentle look at the RBM
I’ve been saying “train each layer to represent the layer below it.” Let me make that less abstract, because the building block has a name you’ll run into: the Restricted Boltzmann Machine, or RBM. It’s simpler than it sounds.
Get Cordero Core’s stories in your inbox
Join Medium for free to get updates from this writer.
Picture two rows of nodes. A bottom row that holds the input, and a top row that’s going to learn to describe it. Every node on the bottom connects to every node on the top (but here’s the “restricted” part): nodes in the same row don’t connect to each other. That restriction is what makes it trainable in practice.
Now, what does an RBM actually do? It learns to reconstruct its own input.
You show it a pattern on the bottom row. It uses its connections to light up some nodes on the top row: a compressed impression of what it just saw. Then it runs that impression backward, from the top row down, and tries to regenerate the original pattern on the bottom. At first the regeneration is garbage. But you adjust the weights to make the reconstruction a little closer to the original, and you repeat, thousands of times. Slowly, the top row becomes a layer that has learned the shape of the data. It’s good enough that it can regenerate the input it’s shown.
This is the unit Hinton stacked. Each layer in the deep belief network is an RBM that learned to capture the shape of the layer beneath it. Stack them, and each floor is a richer, more compressed description of the one below — exactly the layered abstraction the field wanted but couldn’t train directly.
You don’t need the math to hold the intuition: a layer that learns by trying to regenerate what it sees is a layer that has understood something real about its input.
Where the RBM came from: a short walk back to the 1980s
The RBM didn’t appear out of nowhere in 2006. It has ancestors, and they’re worth a paragraph or two because they close a loop this series has been leaving open.
Go back to 1982. A physicist named John Hopfield described a kind of network that behaves like an associative memory. The idea borrows straight from physics. Every possible state of the network has an “energy,” and the network naturally rolls downhill toward low-energy states, the way a ball settles into a valley. You train it so that the patterns you want to remember are those low-energy valleys. Then you show it a corrupted or partial version of a pattern — and it settles, on its own, into the nearest stored memory. Give it half a face and it falls into the whole face: memory recovered from a fragment.
The Hopfield network reawakened interest in neural nets in the middle of the first winter. Hinton and a collaborator took that energy-based idea and made it learn richer things. The Boltzmann Machine is named for the physicist whose statistics it borrowed. The Boltzmann Machine was powerful and painfully slow to train. The RBM is its stripped-down, practical descendant. It’s the version you can actually train fast enough to stack. So the thing that broke the winter in 2006 was, underneath, a 1980s physics idea finally made efficient.
There’s a coda here that I love. In 2024, John Hopfield and Geoffrey Hinton shared the Nobel Prize in Physics for exactly this lineage of work: the energy-based memory from 1982 and the man who kept its descendants alive through two winters, standing on the same stage. Backstory in this episode; a Nobel in ours.
The part that mattered more than the math
Now here’s the turn, and it’s the reason this episode isn’t just another technical milestone.
The deep belief network’s accuracy numbers, on their own, were good but not earth-shaking. What the 2006 result actually did was break a belief. For twenty years the field had operated on a shared assumption: deep networks can’t be trained, so stop trying. Hinton’s paper was an existence proof against that assumption. Here is a deep network. Here is it training. Watch the thing you all agreed was impossible.
And Hinton understood something about the field beyond the algorithm. The term “neural networks” was radioactive. Two winters had made it a punchline; put it in a grant proposal and reviewers’ eyes glazed. So he and his collaborators did something deliberate and, honestly, a little cheeky. They rebranded. The unfashionable thing wearing the unfashionable name got a new one: deep learning.
It was a survival move. Same core idea the field had been circling since the perceptron, dressed in a new word: deep. It pointed at the one thing the new method could suddenly do that nothing else could. New name, clean of two decades of baggage, and it worked. Funding trickled back, and attention followed. A small group, much of it kept afloat for years by the Canadian Institute for Advanced Research when almost no one else was paying, had held the flame long enough to hand it forward.
This is what I keep turning over. The most important thing to come out of 2006 wasn’t a number on a benchmark. It was permission. Permission to say the word “deep” out loud in a serious room and not get laughed out of it.
The twist: the scaffold got kicked away
I owe you the honest part, the way I try to every week. And this one is a genuine plot twist, because it undercuts the very technique we just spent the whole episode admiring.
Within a few years, greedy layer-wise pretraining became mostly obsolete.
Not wrong. Obsolete. Researchers figured out that if you were just a bit smarter about how you initialized the weights at the start, and if you swapped the network’s activation function for something called a ReLU (a small change that let the gradient flow backward without shrinking to nothing), and if you had a lot more data and, critically, GPUs to chew through it, then you could train a deep network from scratch. Directly. All layers at once, with plain backprop, no layer-by-layer pretraining ritual at all.
The elaborate scaffolding Hinton built to get around the vanishing gradient wasn’t needed anymore. The wall he’d so carefully walked around, the field eventually just climbed straight over.
So was 2006 a dead end? No. And this is the idea I most want you to carry out of this episode.
Think about how a stone arch gets built. You raise a wooden frame first, and you lay the stones over that frame, and only when the keystone drops in does the arch hold its own weight. Then you take the wooden frame away. The frame is gone. The arch is standing because the frame was there. Nobody looks at the finished arch and calls the scaffolding a failure because it isn’t in the photo.
That’s what layer-wise pretraining was: a scaffold. It got kicked away once the building could stand on its own with better initialization, ReLU, GPUs, and data. But without it, the field never would have believed the building was possible to raise in the first place. It broke the “untrainable” consensus and thawed the funding, and that is what bought the time for the better tools to arrive. The specific algorithm was temporary. The permission it granted was the whole gift.
The idea worked hard, did its job, and stepped aside. Some of the most important ideas do exactly that.
Why you actually reach for from_pretrained
Bring it back to your keyboard, because the descendant of 2006 is in your hands constantly.
That instinct to grab a pretrained model and fine-tune it? That’s the shape Hinton drew. Do the expensive, general learning once (let a big model absorb the structure of the world without task-specific labels), and then cheaply specialize it to your problem. In 2006 the general learning was unsupervised, layer by layer, on small data. Today it’s a foundation model trained on a corpus the size of the internet. The scale is unrecognizable. The shape is identical: pretrain broadly first, adapt narrowly second.
And there’s a thread here I’ll only pull gently, because we’ll earn it properly later in the series. That phrase: unsupervised pretraining, then adapt. It’s not just how you fine-tune a vision model. It’s the seed of how the models you talk to every day were made. A network that teaches itself the structure of its input by trying to predict or regenerate it, with no labels, before anyone fine-tunes it toward a task. Hold that sentence. When we get to GPT, you’ll recognize its grandparent. It’s sitting right here, in a 2006 paper about stacking Restricted Boltzmann Machines to read handwritten digits.
The perceptron was the neuron; backprop made it trainable; CNNs gave it the right shape for seeing. And 2006 was the year the field stopped believing depth was impossible and started, finally, to go deep.
A researcher who’d spent two decades on an idea most of his field had written off found a way to train the thing everyone said couldn’t be trained. The technique itself didn’t last very long. The permission it granted has never gone away.
Sometimes the point of an idea isn’t to be the final answer. It’s to prove the door opens at all — so the people with the better key can walk through.
Next Tuesday: the year the dam broke. 2006 cracked the door; 2012 tears it off the frame. A deep network enters a competition to recognize a thousand kinds of objects in a million photographs, wins by a margin so wide it looks like a typo, and the entire field pivots inside of a single week. GPUs make it possible. And the check that Episode 4 wrote — that convolutional networks would come back and “kick the door off the hinges” — finally clears.
If Episode 6 was the turn, Episode 7 is the avalanche.
If this series is useful to someone on your team — the one who fine-tunes a checkpoint every week and has never once wondered where that habit came from — send it their way. The whole point of walking the timeline one Tuesday at a time is that, by the time we reach the tools you use today, none of it feels like magic. It feels like the long, stubborn, human engineering it actually was.
You can follow along on Medium or subscribe to the Substack. Both carry the full series, and the midweek pieces go deeper on whatever I’m building or watching that week.
Cordero is a senior research software engineer at the Scientific Software Engineering Center at the University of Washington’s eScience Institute. If this landed, forward it to the colleague who keeps a folder of pretrained checkpoints and treats from_pretrained like gravity — show them the 2006 paper where their favorite shortcut was born.
