Paper Review: A Beginners Guide to “Attention is all you need”
Before we begin, let’s understand some key concepts:
What is a Neural Network?
A neural network is a type of machine learning model that’s inspired by how the human brain works. It’s made up of layers of “neurons” (also called nodes), and these neurons are connected to each other.
Basic Structure of a Neural Network
Input Layer:
- This is where the data enters the network.
- For example, if you’re giving it a picture, each pixel is part of the input.
Hidden Layers:
- These are the middle layers that do most of the thinking.
- Each neuron takes some input, does a simple calculation, and passes the result to the next layer.
Output Layer:
- This layer gives the final answer — like what object is in a picture, or what word comes next in a sentence.
Key Ideas:
- Each neuron combines input data and assigns it weights (how important each input is).
- It applies a simple rule (called an activation function) to decide what to pass on.
- The network learns by adjusting weights to reduce mistakes, using a method called backpropagation.
How It Works:
Imagine you’re trying to teach a neural network to recognize pictures of cats and dogs: You show it a picture and tell it, “this is a cat.” It makes a guess — maybe it says “dog” at first. You correct it. It adjusts its internal connections a tiny bit. You do this many times with lots of examples. Over time, it learns how to tell the difference on its own.
How is Deep Neural Network different?
The term “deep” refers to the fact that the network has many layers — not that it’s necessarily smarter or more mysterious.
Each extra layer allows the model to learn more complex patterns. The first layer might learn to recognize edges in an image. The next layer might learn shapes. The next one might learn objects like eyes or wheels. Eventually, it learns to recognize a whole face or a car.
What is Recurrent neural networks, long short-term memory and gated recurrent unit?
There are special types of machine learning models called Recurrent Neural Networks (RNNs). These models are really good at working with sequences — things that come in order — like words in a sentence or notes in music.
Two improved versions of RNNs are called:
- Long Short-Term Memory (LSTM) networks
- Gated Recurrent Units (GRUs)
These models are considered some of the best tools (“state of the art”) for tasks where understanding the order of data is important. Examples include:
- Language modeling — figuring out which word is likely to come next in a sentence
- Machine translation — converting a sentence from one language to another
They’re good at this because they can “remember” important things from earlier in a sequence while processing the rest — kind of like how you remember the beginning of a sentence while reading the rest of it.
Key Issue
Older models like Extended Neural GPU, ByteNet, and ConvS2S were trying to speed things up. Instead of processing words one at a time (like older RNNs), they used convolutional neural networks (CNNs) to look at many parts of a sentence at the same time. This is called parallel computation, and it’s faster.
They were doing this because if two words are far apart in the sentence, it takes more steps to relate them.For ConvS2S, it grows linearly (if words are twice as far, it takes twice the steps).For ByteNet, it grows logarithmically (slower growth, but still increases).
This makes it harder for the model to understand long-range relationships in a sentence.
How Transformer Solves This
The Transformer model handles all positions in a sentence at once using self-attention. This means no matter how far apart two words are, it only takes the same number of steps to connect them. That’s a big improvement for speed and accuracy!
The tradeoff: Since it averages the attention over multiple words, it might blur some important details.
Get Animesh S. Basnet’s stories in your inbox
Join Medium for free to get updates from this writer.
To fix that, the Transformer uses a trick called Multi-Head Attention — it looks at the sentence in different ways at once so it doesn’t miss important relationships.
What is Attention?
Example: Imagine you’re reading a sentence like:
“The cat sat on the mat because it was tired.”
When you get to the word “it”, your brain automatically pays attention to “cat” — you know “it” refers to the cat, not the mat. This is what attention does in machine learning models — it helps the model focus on the most relevant parts of the input when making a decision.
To make this work mathematically, models use an attention function, which can be thought of like a smart matching system. Here’s how it works:
- Query: What we’re currently focused on (e.g., the word “it”).
- Keys: The list of possible reference points (like “cat”, “mat”, etc.).
- Values: The information tied to each key.
- Output: A combination of values, based on how well each key matches the query.
So in our example, the model compares the query “it” with all the keys in the sentence (like “cat”, “mat”, etc.), finds the best match (most likely “cat”), and then blends the associated values to make a decision.
The model learns how much attention to give each word by calculating how strongly each key matches the query. Then, it uses those match scores to create a weighted average of the values — producing an output that’s focused and context-aware.
Types of Attention
Self-Attention
- The model compares each word in a sentence to all other words in the same sentence.
- Used in Transformers.
- Example: In “The ball that the dog chased was red,” self-attention helps understand how “ball” and “was red” are connected.
Cross-Attention (or Encoder-Decoder Attention)
- The model uses attention between two sequences — like input and output.
- Used in translation tasks: The decoder focuses on different parts of the input sentence while generating each word of the translation.
Soft Attention
- The model assigns different weights (probabilities) to all words — and blends them softly.
- Think of it as looking at everything, but giving more focus to the important parts.
Hard Attention
- Instead of soft blending, the model selects one item to focus on.
- It’s harder to train because it’s more like making a yes/no choice instead of a weighted average.
Multi-Head Attention
- The model applies multiple attention layers (called “heads”) in parallel.
- Each one looks at different patterns or relationships in the input.
- This gives the model a richer understanding of the context.
Transformer Model Architecture
Encoder and Decoder Stacks
The Transformer has two main parts:
- Encoder: Understands the input sentence.
- Decoder: Generates the output sentence.
Each part is made up of layers (usually 6 or more) that repeat the same structure. These layers help the model learn deeper patterns in the text.
- The Encoder Stack reads the input and creates a rich representation of it.
- The Decoder Stack takes that representation and generates the output, one word at a time.
Attention
There are two main types of attention here:
- Self-Attention (in Encoder and Decoder): Lets each word look at other words in the same sentence to understand the context.
- Encoder-Decoder Attention (in Decoder): Helps the decoder focus on the relevant parts of the input sentence from the encoder.
Embeddings and Softmax
- Embeddings: Words are turned into vectors (numbers) before going into the model. This is like turning words into a form the computer can understand.
- Softmax: At the end of the decoder, the model picks the most likely next word using a function called softmax, which turns numbers into probabilities (e.g., 70% chance the next word is “cat”).
Positional Encoding
Since Transformers don’t read words in order like older models, they need a way to know where each word is in the sentence. That’s where positional encoding comes in — it adds special patterns to the word vectors to show their position (first, second, third, etc.). This way, the model knows the order of the words.
Why Self-Attention?
Self-attention not only works better — it can also be more interpretable. You can see where the model is “looking” when it makes decisions. For example, one attention head might learn to look at subjects, another at verbs, another at question words, etc. This aligns with grammar and sentence meaning!
If you wish to read the full paper, you can read it here : https://arxiv.org/abs/1706.03762v7
