I don’t remember most equations, I remember images like this.
Every few months, I run into the same problem.
I read a research paper. I understand the high-level idea. I follow the architecture diagram. I can even convince myself that I understand the equations.
Then a few weeks later, someone asks me about it.
And I realize that what I retained was surprisingly little.
Not because the paper was bad.
Not because the math was too hard.
Because information and understanding are not the same thing.
Recently, while implementing a Transformer from scratch, I ended up staring at two visualizations for much longer than I expected: a loss curve and an attention heatmap.
The loss curve wasn’t surprising.
The heatmap was.
And weirdly enough, that image explained more about Transformers than dozens of blog posts I had previously read.
the triangle
If you look at the attention heatmap, the first thing you notice is the shape.
A very deliberate triangle.
Half of the visualization is dark.
Half of it isn’t.
That shape exists because the model is forbidden from looking into the future.
Every token in a Transformer is trying to answer a very simple question:
Given everything I have seen so far, what comes next?
That’s it, no philosophy, no reasoning, no intelligence, just prediction.
If the model were allowed to see future tokens during training, the entire task would become meaningless.
Imagine asking someone to predict the next word in a sentence while secretly allowing them to read the rest of the sentence first.
They’re no longer predicting, they’re copying.
The Transformer has the same problem.
Without constraints, it would simply learn shortcuts.
So we introduce something called a causal mask.
The causal mask blocks every token from seeing tokens that appear later in the sequence.
When visualized, that constraint becomes the triangle.
Everything above the diagonal is inaccessible.
The future literally does not exist from the model’s perspective.
And suddenly, a piece of architecture that felt abstract in the paper becomes obvious.
The model isn’t choosing not to cheat.
The model physically cannot.
transformers are surprisingly repetitive
One thing that stood out while implementing the architecture is how repetitive the entire process really is.
When people hear terms like:
- Multi-Head Attention
- Positional Encoding
- Self-Attention
- Decoder Blocks
it sounds like there are dozens of different things happening simultaneously.
In reality, the core loop is remarkably simple.
A token looks backwards.
It decides which previous tokens matter.
It combines information from those tokens.
It predicts what comes next.
Then the process repeats.
Again.
And again.
And again.
Across billions of parameters and trillions of examples.
The complexity isn’t coming from one brilliant operation.
It’s coming from an absurd amount of repetition.
Get Manas Dange’s stories in your inbox
Join Medium for free to get updates from this writer.
That’s something I think gets lost in a lot of AI discussions.
People often look for the magical component.
The secret ingredient.
The moment where intelligence suddenly appears.
What I keep seeing instead is the power of scale applied to simple mechanisms.
the loss curve tells a different story
The second visualization was the training loss.
It started around 4.3 and gradually dropped to around 1.5.
Nothing dramatic happened.
There wasn’t a moment where the model suddenly figured out Shakespeare.
There wasn’t a point where intelligence switched on.
The graph just kept moving downward.
Slowly.
Consistently.
Thousands of tiny improvements stacked on top of each other.
That’s what training actually looks like.
And honestly, I think that’s one of the most underrated lessons in machine learning.
The public narrative around AI often focuses on capabilities.
What can the model do?
How powerful is it?
How human-like is it?
The implementation perspective focuses on something completely different.
How does it get there?
And the answer is usually much less dramatic than people expect.
A prediction.
An error.
A weight update.
Repeat.
Millions of times.
why visualizations matter
I think Andrew Ng understood something years ago that many technical explanations still miss.
Humans are terrible at remembering abstractions.
We’re much better at remembering visual intuition.
I don’t remember most cost functions from memory.
I don’t remember every equation from every paper I’ve read.
But I still remember visualizations from years ago.
I remember seeing gradient descent moving toward a minimum.
I remember watching decision boundaries shift during training.
I remember activation maps from convolutional neural networks.
And now I’ll probably remember this Transformer heatmap long after I’ve forgotten some implementation details.
Because visualizations transform knowledge into intuition.
Knowledge is knowing what something does.
Intuition is feeling why it behaves that way.
Those are very different things.
what changed for me
The most valuable outcome of implementing the Transformer wasn’t the generated Shakespeare.
It was realizing how much understanding comes from observing behavior instead of reading descriptions.
Research papers tell you what should happen.
Implementations show you what actually happens.
Visualizations make the gap between those two disappear.
And increasingly, that’s becoming my preferred way to learn.
Not just reading.
Not just coding.
Building, breaking, visualizing, and observing.
Because once you can see an idea working, it’s much harder to forget.
And sometimes a triangle can teach you more than twenty pages of equations.
