The History of Transformers (And Why We Needed Them)
Before the math, the story of why Transformers had to be invented
The history of Transformers goes back to 2017 , to one paper by the Google research team that completely redefined what would become the modern idea of LLMs.
Before that, in the older days, we had RNNs (Recurrent Neural Networks). They processed text one word at a time, sequentially, paying attention to only one word at each step.
Then came LSTMs (Long Short-Term Memory networks) , an improvement that introduced gates and memory cells, allowing the model to carry information across longer stretches of text. For a while, this worked well.
But it had a real limitation once context grew too long.
Get serilevanjaline’s stories in your inbox
Join Medium for free to get updates from this writer.
Here’s an easy example. Take the sentence: “Paris is the capital of France.” The model has no trouble here , it easily understands the strong relationship between “France” and “Paris,” and gives them the right attention.
Now consider this instead: “The book is… [several sentences later]… excellent.”
By the time the model reaches the word “excellent,” it struggles to connect it back to “book.” In an LSTM, “book” gets pushed out by the forget gate long before the model ever reaches “excellent.” So when it finally arrives at that word, it no longer remembers what it was even describing.
This was the major bottleneck , memory that faded with distance.
And to solve this problem, our Transformer hero entered the picture.
In Part 2, I’ll walk through exactly how tracing one sentence, step by step, through tokenization, embeddings, attention, and beyond (Understanding Attention Without the Math (Mostly))
