TRANSFORMERS

What are transformers?

Transformers are deep learning models used to process sequential data like text, images, audio or code. They are the backbone of modern AI systems such as ChatGPT, BERT, Google Translate etc.

Unlike RNNs, Transformers do not process data step-by-step. They use a mechanism called attention to look at the entire sequence at once.

Why Transformers exist?

Problems with RNN/LSTM/GRU

1. Sequential Processing: RNNs read input one token at a time, Token 10 cannot be processed before token 9. This makes training slow and not parallelizable.
2. Long-Term Dependency Problem: RNNs struggle to remember information from far back in a sentence, Important early words may be forgotten
3. Vanishing/Exploding Gradients: During backpropagation, gradients become very small or very large, Training becomes unstable.
4. Speed bottleneck: Large datasets take a very long time to train, GPUs are not used efficiently.

Why Attention was Introduced

• Humans do not read word-by-word equally
• We focus on important words
• Attention allows the model to focus on relevant parts of input dynamically.

Basically Attention allows the model to look at all words at once and decide what matters most.

Transformers were created to remove recurrence, speed up training, and handle long context better.

Attention Mechanism

Self Attention: It means each word looks at all other words in the same sentence and decides how important they are.

Ex: The animal didn’t cross the street because it was tired.

The word “it” needs to know whether it refers to animals or streets. Self Attention helps solve this.

[kehne ka mtlb Self-attention ek mechanism h jisme sentence ka har word sentence ke baaki words ko dekhta h aur decide krta h ki kon sa word kitna important h, yahan Self ka mtlb h same sentence ke andar]

Query, Key Value(QKV)

Each token is converted into three vectors:

• Query(Q): What am i looking for? (M kya dhoondh rh?)
• Key(K): What do I represent?(M kya bta rh?)
• Value(V): What information do i contain?(Mere pass actual info kya h?)

Simple Analogy (Search Engine)

• Query: search text
• Key: webpage title
• Value: webpage content

The model compares Query with Keys to decide which Values to focus on.

Attention Formula: Attention(Q,K,V) = softmax(QKt/√dk)V

1. Dot product (Q•K)→similarity score
2. Scaling→divide by √dk to stabilize values
3. Softmax→convert scores to probabilities
4. Weighted Sum with V→final output

Why Scaling?

• Large dimensions produce large dot products
• Softmax becomes unstable
• Scaling fixes this

Multi-Head Attention

Instead of one attention:

• Use multiple attention heads
• Each head learns a different type of relationship

[kehne ka mtlb Multi-head attention multiple Self-attentions ka combination h, jahan har attention “head” sentence ko alag angle se dekhta h

Mean: Self-attentions*multiple times aur har time different projection

basically, Multi-head attention=multiple perspectives on the same sentence]

Technical View

• Input ko multiple parts me split krte h
• har part pe separate Self-attention
• output ko concatenate+ linear layer

• Self-attentions=Who should i listen to?
• Multi-Head Attention=Let me ask the same ques in many different ways

Input Representation

Tokenization: Converts text into tokens

Types:

• Word-level: large vocabulary, OOV problem
• Subword-level: BPE, WordPiece
• ex: unhappiness: un+happy+ness

Embeddings

• Tokens are converted into vectors(e.g, 768 dimensions)
• Similar words have similar vectors.

Embeddings capture semantic meaning.

Positional Encoding

Why needed?

• Attention has no sense of order
• Dog bites man is not equal to Man bites dog l
• That’s why we add positional information to embeddings

Sinusoidal Encoding

Get Shraddha Tiwari’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Use sine and cosine functions to encode position.

• Unique pattern for each position
• Helps models understand relative order

Basically:

• Tokenization breaks text
• Embeddings give meaning
• Positional encoding gives order

Transformer Block

A transformer block is the basic building unit of a transformer model.

It is a structured combination of layers that process input tokens, understand their relationships, and transform them into richer representations.

A Transformer model is created by stacking many identical Transformer blocks.

Why do we need Transformer Blocks?

Traditional neural networks:

• process data independently
• Do not understand relationships between tokens

Transformer blocks:

• Allow tokens to interact with each other
• Capture context and meaning
• Work in parallel, making them fast and scalable

Types of Transformer Blocks

• Encoder
• Decoder

Encoder block

It is used to understand the input data. eg. Text classification, Sentence embeddings etc

Structure of Encoder block

1. Multi-Head Self Attention
2. Add and layer Normalization
3. Feed Forward Network
4. Add and layer Normalization
5. Output

Multi-Head Attention: A mechanism where each token looks at all other tokens in the same input to understand context.

Why is it used:

• To capture relationships between words
• To resolve ambiguity(e.g., pronouns)

Output: Each token becomes context-aware

Residual Connection(Add): The input of a layer is added back to its output.

Output=x+F(x)

It is used to prevent vanishing gradients and enables training of deep Transformer model

Layer Normalization: It normalizes the values of each token’s features to stabilize training.

• It keeps activations well scaled
• Makes training faster and more stable

Feed Forward Network(FNN): A small neural network applied independently to each token

[FNN ka kaam tokens ke beech baat krwana nhi hota, Wo kaam hota h attention ka. FNN ka kaam hota h har token ko individually “sochne” dena]

[Simple si baat ka koi image liya usko zoom kiya, details jyada clearly dikha basically hidden features create kiya, fir wapas se ussi dimension me kr diya qki input aur output ki dimension same honi chahiye]

It is used to add non-linearity

Input vector(512)-> Linear(512->2048)->Activation->Linear(2048->512)->Output vector(512)

Decoder Block

The decoder block is used to generate output tokens, one step at a time. e.g., Text generation, Machine Translation, Chatbots

Structure:

1. Masked Multi-Head Attention
2. Add and layer Normalization
3. Encoder-Decoder Attention
4. Add and layer Normalization
5. Feed Forward Network
6. Add and layer Normalization
7. Output

Masked Self-Attentions

A form of Self-attention where tokens are not allowed to see future tokens.

Why is it used?

• Ensures autoregressive generation
• Prevents cheating during training

Encoder-Decoder Attention(Cross-Attention)

The decoder attends to the encoder’s output reproduction.

Why is it used?

• Connects input understanding (encoder) with output generation
• Helps align input and output tokens

Why are Transformer Blocks Repeated?

One block captures basic relationships

Multiple stacked blocks allow:

• Deeper understanding
• Higher-level abstraction
• Better reasoning

Why Transformers blocks are Powerful

• Parallel processing(fast)
• Long-range dependency
• Highly scalable
• Foundation of modern AI systems