Transformers from the Linear Transformation Perspective

The Transformer architecture, which has revolutionised natural language processing and other fields, can be deeply understood through the lens of linear transformations.

While Transformers are complex neural networks, many of their core operations are fundamentally linear, or can be viewed as sequences of linear transformations followed by non-linear activations.

Theoretical explanation of “Attention is all you need” is given by many authors but I believe we will be able to understand a latent model training and intuition behind the architecture can be understood only after visualising the liner transformations of a training data point/batch. Hence I would like to present the bear bones of Transformers with a simple example of translating from English to Hindi.

Here’s a breakdown of how linear transformations are central to Transformers:

1. Embeddings:

• Input Embeddings: The very first step in a Transformer involves converting discrete input tokens (like words) into continuous vector representations called embeddings. This is often achieved through a lookup table, but conceptually, it’s a linear mapping from a one-hot encoding of the token to a dense vector in a high-dimensional space.
• Positional Encodings: Since Transformers process sequences in parallel without inherent sequential understanding(unlike RNNs which process sequentially), positional encodings are added to the input embeddings. These encodings, which capture the relative or absolute position of tokens, are often generated using sinusoidal functions and then linearly combined (positional encoding is added to embeddings, not multiplied) with the token embeddings.

English sentence: “When will you come home tonight ?”

Translated Hindi Sentence: “आज रात घर कब आओगे ?”

Tokenised: [When, will, you, come, home, tonight, ?]

Sequence length: T = 7 tokens

Assume Input embedding dimension : d_model = 64 (for sake of simplicity)

Query/Key dimension: d_k = 128

Value dimension: d_v = 256

Sinusoidal Positional Encoding:

PE(pos, 2i) = sin(pos / 10000^(2i/d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))
Where:
  pos = position in sequence (0, 1, 2, …)
  - i = dimension index (0, 1, 2, …, d_model/2–1
  - 2i = even dimensions get sine
  - 2i+1 = odd dimensions get cosine

In our example :
  English: "When will you come home tonight ?"
  Positions: [0, 1, 2, 3, 4, 5, 6]
  d_model = 64

For each position pos and dimension i: 
  ω_i = 1 / 10000^(2i/64) = 1 / 10000^(i/32)

Position 0: “When”

PE(0, 0) = sin(0 × 1.0) = sin(0) = 0.0
PE(0, 1) = cos(0 × 1.0) = cos(0) = 1.0
PE(0, 2) = sin(0 × 0.794) = sin(0) = 0.0  
PE(0, 3) = cos(0 × 0.794) = cos(0) = 1.0
PE(0, 4) = sin(0 × 0.631) = sin(0) = 0.0
PE(0, 5) = cos(0 × 0.631) = cos(0) = 1.0
...
PE(0, 62) = sin(0 × 0.001) = sin(0) = 0.0
PE(0, 63) = cos(0 × 0.001) = cos(0) = 1.0

PE₀ = [0.0, 1.0, 0.0, 1.0, 0.0, 1.0, ..., 0.0, 1.0] ∈ ℝ^64

Position 1: “will”

PE(1, 0) = sin(1 × 1.0) = sin(1) ≈ 0.841
PE(1, 1) = cos(1 × 1.0) = cos(1) ≈ 0.540
PE(1, 2) = sin(1 × 0.794) = sin(0.794) ≈ 0.717
PE(1, 3) = cos(1 × 0.794) = cos(0.794) ≈ 0.697
PE(1, 4) = sin(1 × 0.631) = sin(0.631) ≈ 0.589
PE(1, 5) = cos(1 × 0.631) = cos(0.631) ≈ 0.808
...

PE₁ = [0.841, 0.540, 0.717, 0.697, 0.589, 0.808, ...] ∈ ℝ^64

Postion 2: “you”

PE(2, 0) = sin(2 × 1.0) = sin(2) ≈ 0.909
PE(2, 1) = cos(2 × 1.0) = cos(2) ≈ -0.416
PE(2, 2) = sin(2 × 0.794) = sin(1.588) ≈ 0.999
PE(2, 3) = cos(2 × 0.794) = cos(1.588) ≈ -0.041
...

PE₂ = [0.909, -0.416, 0.999, -0.041, ...] ∈ ℝ^64

Positional encoding matrix for english sentence:

PE_english ∈ ℝ^(7 × 64)

PE_english = [
    PE₀  # Position 0 ("When")     [0.0,   1.0,   0.0,   1.0,   ...]
    PE₁  # Position 1 ("will")     [0.841, 0.540, 0.717, 0.697, ...]
    PE₂  # Position 2 ("you")      [0.909, -0.416, 0.999, -0.041, ...]
    PE₃  # Position 3 ("come")     [0.141, -0.990, 0.717, -0.697, ...]
    PE₄  # Position 4 ("home")     [-0.757, -0.653, -0.587, -0.809, ...]
    PE₅  # Position 5 ("tonight")  [-0.959, 0.284, -0.999, 0.041, ...]
    PE₆  # Position 6 ("?")        [-0.279, 0.960, -0.717, 0.697, ...]
]

Positional encoding matrix for hindi sentence:

Hindi: "आज रात घर कब आओगे ?"
Positions: [0, 1, 2, 3, 4, 5]

PE_hindi ∈ ℝ^(6 × 64)

PE_hindi = [
    PE₀  # Position 0 ("आज")       [0.0,   1.0,   0.0,   1.0,   ...]
    PE₁  # Position 1 ("रात")      [0.841, 0.540, 0.717, 0.697, ...]
    PE₂  # Position 2 ("घर")       [0.909, -0.416, 0.999, -0.041, ...]
    PE₃  # Position 3 ("कब")       [0.141, -0.990, 0.717, -0.697, ...]
    PE₄  # Position 4 ("आओगे")     [-0.757, -0.653, -0.587, -0.809, ...]
    PE₅  # Position 5 ("?")        [-0.959, 0.284, -0.999, 0.041, ...]
]

Adding positional encoding to embeddings

English Encoder Input:

Word embeddings: X_embed ∈ ℝ^(7 × 64)
Positional encodings: PE_english ∈ ℝ^(7 × 64)

Final input: X_input = X_embed + PE_english ∈ ℝ^(7 × 64)

For "When" (position 0):
x_input₀ = x_embed₀ + PE₀
x_input₀[j] = x_embed₀[j] + PE₀[j]  for j = 1, 2, ..., 64

Example:
x_embed₀ = [0.1, -0.3, 0.7, 0.2, -0.5, 0.8, ...]
PE₀ = [0.0, 1.0, 0.0, 1.0, 0.0, 1.0, ...]
x_input₀ = [0.1, 0.7, 0.7, 1.2, -0.5, 1.8, ...]

Hindi Decoder Input:

Word embeddings: X_dec_embed ∈ ℝ^(6 × 64)
Positional encodings: PE_hindi ∈ ℝ^(6 × 64)

Final input: X_dec_input = X_dec_embed + PE_hindi ∈ ℝ^(6 × 64)

For "कब" (position 3):
x_dec_input₃ = x_dec_embed₃ + PE₃
x_dec_input₃[j] = x_dec_embed₃[j] + PE₃[j]  for j = 1, 2, ..., 64

Properties of Sinusoidal Encoding:

1. Unique Encoding for Each Position

Each position gets a unique 64-dimensional vector:

PE₀ ≠ PE₁ ≠ PE₂ ≠ ... ≠ PE₆

2. Relative Position Information

The encoding allows the model to learn relative positions, 
For any fixed offset k:

PE(pos+k) can be expressed as a linear function of PE(pos)

3. Frequency Spectrum

- Low dimensions (i=0,1,2,...): High frequency, change rapidly with position

- High dimensions (i=30,31): Low frequency, change slowly with position

Mathematical Flow of Positional Encoding:

Word Embeddings: X_embed ∈ ℝ^(seq_len × 64)
                    ↓
Compute Positional Encoding:
PE[pos, 2i] = sin(pos / 10000^(2i/64))
PE[pos, 2i+1] = cos(pos / 10000^(2i/64))
PE ∈ ℝ^(seq_len × 64)
                    ↓
Add to Embeddings:
X_input = X_embed + PE ∈ ℝ^(seq_len × 64)
                    ↓
Feed to Transformer Layers:
Q = X_input · W^Q  # Contains both content and position info
K = X_input · W^K
V = X_input · W^V

Example: Position-Aware Attention:

When computing attention between "कब" (pos 3) and "When" (pos 0):

Hindi query: q₃ = (x_dec_embed₃ + PE_hindi₃) · W^Q
English key: k₀ = (x_enc_embed₀ + PE_english₀) · W^K

Attention score: s₃₀ = q₃ · k₀

This score now includes:

-> Content similarity: How well "कब" semantically matches "When"
-> Position information: Relative position relationship (3 vs 0)

The model learns to use both types of information for accurate translation alignment!

After Adding positional encoding lets recall the original english input sentence setup:

English sentence: "When will you come home tonight ?"
Tokenized: [When, will, you, come, home, tonight, ?]
Sequence length: T = 7 tokens
Input embedding dimension: d_model = 64
Query/Key dimension: d_k = 128
Value dimension: d_v = 256

Step 2 : Input Embedding

Each token gets embedded into a 64-dimensional vector:

X ∈ ℝ^(T × d_model) = ℝ^(7 × 64)

X = [
    x₁  # "When"   - [0.1, -0.3, 0.7, ..., 0.2]  ∈ ℝ^64
    x₂  # "will"   - [0.4, 0.1, -0.2, ..., 0.8]  ∈ ℝ^64
    x₃  # "you"    - [-0.1, 0.6, 0.3, ..., -0.4] ∈ ℝ^64
    x₄  # "come"   - [0.2, -0.1, 0.9, ..., 0.1]  ∈ ℝ^64
    x₅  # "home"   - [0.8, 0.3, -0.5, ..., 0.6]  ∈ ℝ^64
    x₆  # "tonight"- [-0.2, 0.4, 0.1, ..., -0.3] ∈ ℝ^64
    x₇  # "?"      - [0.0, 0.2, -0.1, ..., 0.5]  ∈ ℝ^64
]

Step 3 : Linear transformation of Query, Key and Value matrices

Latent matrices
These matrices are randomly initialized and learned during training :

W^Q ∈ ℝ^(d_model × d_k) = ℝ^(64 × 128)  # Query projection
W^K ∈ ℝ^(d_model × d_k) = ℝ^(64 × 128)  # Key projection  
W^V ∈ ℝ^(d_model × d_v) = ℝ^(64 × 256)  # Value projection

Step 4 : Computing Queries, Keys, and Values

Query Matrix:

Q = X · W^Q
Q ∈ ℝ^(7 × 128)

For token "When" (position 1):
q₁ = x₁ · W^Q = [x₁₁, x₁₂, ..., x₁₆₄] · W^Q = [q₁₁, q₁₂, ..., q₁₁₂₈]

Mathematically:
q₁ⱼ = Σᵢ₌₁⁶⁴ x₁ᵢ · W^Q_ij  for j = 1, 2, ..., 128


Key Matrix:

K = X · W^K  
K ∈ ℝ^(7 × 128)

For token "will" (position 2):
k₂ = x₂ · W^K = [k₂₁, k₂₂, ..., k₂₁₂₈]

k₂ⱼ = Σᵢ₌₁⁶⁴ x₂ᵢ · W^K_ij  for j = 1, 2, ..., 128

Value Matrix:

V = X · W^V
V ∈ ℝ^(7 × 256)

For token "you" (position 3):
v₃ = x₃ · W^V = [v₃₁, v₃₂, ..., v₃₂₅₆]

v₃ⱼ = Σᵢ₌₁⁶⁴ x₃ᵢ · W^V_ij  for j = 1, 2, ..., 256

Step 5: Attention Score Computation

Raw Attention Scores:

S = Q · K^T
S ∈ ℝ^(7 × 7)

S_ij = qᵢ · kⱼ = Σₖ₌₁¹²⁸ qᵢₖ · kⱼₖ

Example for "When" attending to "home":
S₁₅ = q₁ · k₅ = q₁₁·k₅₁ + q₁₂·k₅₂ + ... + q₁₁₂₈·k₅₁₂₈


The attention matrix S looks like:

       When  will  you  come  home tonight  ?
When   [s₁₁  s₁₂  s₁₃  s₁₄   s₁₅   s₁₆   s₁₇]
will   [s₂₁  s₂₂  s₂₃  s₂₄   s₂₅   s₂₆   s₂₇]  
you    [s₃₁  s₃₂  s₃₃  s₃₄   s₃₅   s₃₆   s₃₇]
come   [s₄₁  s₄₂  s₄₃  s₄₄   s₄₅   s₄₆   s₄₇]
home   [s₅₁  s₅₂  s₅₃  s₅₄   s₅₅   s₅₆   s₅₇]
tonight[s₆₁  s₆₂  s₆₃  s₆₄   s₆₅   s₆₆   s₆₇]
?      [s₇₁  s₇₂  s₇₃  s₇₄   s₇₅   s₇₆   s₇₇]



Scaled Attention Scores:

S_scaled = S / √d_k = S / √128 ≈ S / 11.31

S_scaled_ij = S_ij / √128


Softmax Attention Weights:

A = softmax(S_scaled)
A ∈ ℝ^(7 × 7)

For row i (token i attending to all tokens):
Aᵢⱼ = exp(S_scaled_ij) / Σₖ₌₁⁷ exp(S_scaled_ik)

Example for "When" (row 1):
A₁ⱼ = exp(S_scaled_1j) / [exp(S_scaled_11) + exp(S_scaled_12) + ... + exp(S_scaled_17)]

Each row sums to 1: Σⱼ₌₁⁷ A₁ⱼ = 1

Step 6: Final Output Computation

Z = A · V
Z ∈ ℝ^(7 × 256)

For token "When" (position 1):
z₁ = Σⱼ₌₁⁷ A₁ⱼ · vⱼ

z₁ = A₁₁·v₁ + A₁₂·v₂ + A₁₃·v₃ + A₁₄·v₄ + A₁₅·v₅ + A₁₆·v₆ + A₁₇·v₇

Each element of z₁:
z₁ₖ = Σⱼ₌₁⁷ A₁ⱼ · vⱼₖ  for k = 1, 2, ..., 256

Complete Mathematical Flow:

Input: X ∈ ℝ^(7×64) (added with Positional Encoding)
       ↓
Linear Transformations:
Q = X·W^Q ∈ ℝ^(7×128)
K = X·W^K ∈ ℝ^(7×128)  
V = X·W^V ∈ ℝ^(7×256)
       ↓
Attention Scores:
S = Q·K^T ∈ ℝ^(7×7)
       ↓
Scaled & Softmax:
A = softmax(S/√128) ∈ ℝ^(7×7)
       ↓
Output:
Z = A·V ∈ ℝ^(7×256)



Interpretation for Translation:

When translating to Hindi, each English token learns to attend to relevant parts:

-> "When" might attend strongly to "?" (question marker)
-> "tonight" might attend to "When" (temporal relationship)
-> "come" and "home" might attend to each other (semantic relationship)

The attention weights A₁ⱼ represent how much token 1 focuses on token j, 
and the final representation z₁ is a weighted combination of all value vectors, 
allowing rich contextual understanding for the Hindi translation.

Now lets explore decoder linear transformations:

Get Kushvanth’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Lets refresh the decoder setup:

English (Source): "When will you come home tonight ?"
Hindi (Target): "आज रात घर कब आओगे ?"
Tokenized Hindi: [आज, रात, घर, कब, आओगे, ?]
English encoder outputs: Z_enc ∈ ℝ^(7 × 64) (from previous self-attention)
Hindi decoder states: X_dec ∈ ℝ^(6 × 64)

Cross-Attention Architecture (Step-by-Step Cross-Attention):

Step 1: Weight Matrices for Cross-Attention

In cross-attention:
Queries (Q): Come from Hindi decoder (target language)
Keys (K) & Values (V): Come from English encoder (source language)

Q = X_dec · W^Q_cross  # Hindi queries
K = Z_enc · W^K_cross  # English keys  
V = Z_enc · W^V_cross  # English values

Step 2: English Encoder Outputs (Keys & Values Source)

Z_enc = [
    z_enc₁  # "When"     - encoded representation ∈ ℝ^64
    z_enc₂  # "will"     - encoded representation ∈ ℝ^64  
    z_enc₃  # "you"      - encoded representation ∈ ℝ^64
    z_enc₄  # "come"     - encoded representation ∈ ℝ^64
    z_enc₅  # "home"     - encoded representation ∈ ℝ^64
    z_enc₆  # "tonight"  - encoded representation ∈ ℝ^64
    z_enc₇  # "?"        - encoded representation ∈ ℝ^64
] ∈ ℝ^(7 × 64)

Step 3: Hindi Decoder States (Query Source)

X_dec = [
    x_dec₁  # "आज"      - decoder state ∈ ℝ^64
    x_dec₂  # "रात"     - decoder state ∈ ℝ^64
    x_dec₃  # "घर"      - decoder state ∈ ℝ^64  
    x_dec₄  # "कब"      - decoder state ∈ ℝ^64
    x_dec₅  # "आओगे"    - decoder state ∈ ℝ^64
    x_dec₆  # "?"       - decoder state ∈ ℝ^64
] ∈ ℝ^(6 × 64)

Step 4: Linear Transformations

Queries from Hindi (Decoder):
  Q_cross = X_dec · W^Q_cross ∈ ℝ^(6 × 128)
  
  For Hindi token "कब" (position 4):
    q₄ = x_dec₄ · W^Q_cross = [q₄₁, q₄₂, ..., q₄₁₂₈]
    q₄ⱼ = Σᵢ₌₁⁶⁴ x_dec₄ᵢ · W^Q_cross_ij  for j = 1, 2, ..., 128

Keys from English (Encoder):
  K_cross = Z_enc · W^K_cross ∈ ℝ^(7 × 128)

  For English token "When" (position 1):
    k₁ = z_enc₁ · W^K_cross = [k₁₁, k₁₂, ..., k₁₁₂₈]
    k₁ⱼ = Σᵢ₌₁⁶⁴ z_enc₁ᵢ · W^K_cross_ij  for j = 1, 2, ..., 128

Values from English (Encoder):
  V_cross = Z_enc · W^V_cross ∈ ℝ^(7 × 256)

  For English token "tonight" (position 6):
    v₆ = z_enc₆ · W^V_cross = [v₆₁, v₆₂, ..., v₆₂₅₆]
    v₆ⱼ = Σᵢ₌₁⁶⁴ z_enc₆ᵢ · W^V_cross_ij  for j = 1, 2, ..., 256

Step 5: Cross-Attention Computation

Attention Scores:

S_cross = Q_cross · K_cross^T ∈ ℝ^(6 × 7)

S_cross[i,j] = q_i · k_j = Σₖ₌₁¹²⁸ q_iₖ · k_jₖ

Cross-attention matrix (Hindi tokens × English tokens):

       When will  you  come  home tonight ?
आज    [s₁₁  s₁₂  s₁₃  s₁₄  s₁₅   s₁₆   s₁₇]
रात     [s₂₁  s₂₂  s₂₃  s₂₄  s₂₅   s₂₆   s₂₇]  
घर     [s₃₁  s₃₂  s₃₃  s₃₄  s₃₅   s₃₆   s₃₇]
कब     [s₄₁  s₄₂  s₄₃  s₄₄  s₄₅   s₄₆   s₄₇]
आओगे   [s₅₁  s₅₂  s₅₃  s₅₄  s₅₅   s₅₆   s₅₇]
?      [s₆₁  s₆₂  s₆₃  s₆₄  s₆₅   s₆₆   s₆₇]

Example: "कब" attending to English tokens

  For Hindi "कब" (when/what time) attending to English "When":
  S_cross[4,1] = q₄ · k₁ = Σₖ₌₁¹²⁸ q₄ₖ · k₁ₖ

  This should be HIGH because "कब" semantically corresponds to "When"


Scaled and Softmax:

  S_cross_scaled = S_cross / √128

  A_cross = softmax(S_cross_scaled) ∈ ℝ^(6 × 7)
  
  For Hindi token "कब" (row 4):
  A_cross[4,j] = exp(S_cross_scaled[4,j]) / Σₖ₌₁⁷ exp(S_cross_scaled[4,k])

Step 6: Cross-Attention Output

Z_cross = A_cross · V_cross ∈ ℝ^(6 × 256)

For Hindi "कब":
z_cross₄ = Σⱼ₌₁⁷ A_cross[4,j] · v_j
z_cross₄ = A_cross[4,1]·v₁ + A_cross[4,2]·v₂ + ... + A_cross[4,7]·v₇
           (attention to "When")  (attention to "will")      (attention to "?")


Expected Attention Patterns 
(Based on semantic alignment) :

  "आज" (today) → "tonight":
      A_cross[1,6] should be HIGH
      "आज" attends strongly to "tonight" for temporal information

  "रात" (night) → "tonight":
      A_cross[2,6] should be HIGH
      "रात" directly corresponds to "tonight"

  "कब" (when) → "When":
      A_cross[4,1] should be HIGH
      Question words align: "कब" ↔ "When"

  "आओगे" (will come) → "will", "come":
      A_cross[5,2] and A_cross[5,4] should be HIGH
      "आओगे" is compound: future tense "will" + verb "come"      

Complete Cross-Attention Flow:

Hindi Decoder States: X_dec ∈ ℝ^(6×64)
English Encoder Output: Z_enc ∈ ℝ^(7×64)
                 ↓
Linear Transformations:
Q_cross = X_dec · W^Q_cross ∈ ℝ^(6×128)  # Hindi queries
K_cross = Z_enc · W^K_cross ∈ ℝ^(7×128)  # English keys
V_cross = Z_enc · W^V_cross ∈ ℝ^(7×256)  # English values
                 ↓
Cross-Attention Scores:
S_cross = Q_cross · K_cross^T ∈ ℝ^(6×7)
                 ↓
Scaled & Softmax:
A_cross = softmax(S_cross/√128) ∈ ℝ^(6×7)
                 ↓
Cross-Attention Output:
Z_cross = A_cross · V_cross ∈ ℝ^(6×256)

Key Insight:
Cross-attention allows each Hindi word to look back at all English words and decide which ones are most relevant for translation. The attention weights A_cross[i,j] represent “how much does Hindi word i need to focus on English word j” to generate accurate translation.
This is fundamentally different from self-attention, where tokens attend within the same language. Cross-attention bridges the language gap!

This intuition is with single head below is with multihead attention.

Multi-Head Cross-Attention (4 Heads)

Head Dimension Split
  With 4 heads:
  -> d_k per head = 128 ÷ 4 = 32
  -> d_v per head = 256 ÷ 4 = 64

Weight Matrices for Each Head

For head h ∈ {1,2,3,4}:

  W^Q_h ∈ ℝ^(64 × 32)   # Query projection for head h
  W^K_h ∈ ℝ^(64 × 32)   # Key projection for head h  
  W^V_h ∈ ℝ^(64 × 64)   # Value projection for head h
  
  Output projection:
  W^O ∈ ℝ^(256 × 64)    # Concatenated heads → d_model

Head-wise Computation

Head 1: Focus on word alignment

  Q₁ = X_dec · W^Q_1 ∈ ℝ^(6 × 32)  # Hindi queries
  K₁ = Z_enc · W^K_1 ∈ ℝ^(7 × 32)  # English keys
  V₁ = Z_enc · W^V_1 ∈ ℝ^(7 × 64)  # English values
  
  S₁ = Q₁ · K₁^T / √32 ∈ ℝ^(6 × 7)
  A₁ = softmax(S₁) ∈ ℝ^(6 × 7)
  Z₁ = A₁ · V₁ ∈ ℝ^(6 × 64)

Head 2: Focus on syntactic relationships

  Q₂ = X_dec · W^Q_2 ∈ ℝ^(6 × 32)
  K₂ = Z_enc · W^K_2 ∈ ℝ^(7 × 32)  
  V₂ = Z_enc · W^V_2 ∈ ℝ^(7 × 64)
  
  S₂ = Q₂ · K₂^T / √32 ∈ ℝ^(6 × 7)
  A₂ = softmax(S₂) ∈ ℝ^(6 × 7)
  Z₂ = A₂ · V₂ ∈ ℝ^(6 × 64)

Head 3: Focus on temporal relationships

  Similar computation → Z₃ ∈ ℝ^(6 × 64)

Head 4: Focus on question structure

  Similar computation → Z₃ ∈ ℝ^(6 × 64)



Multi-Head Concatenation & Projection:

  Z_concat = [Z₁ || Z₂ || Z₃ || Z₄] ∈ ℝ^(6 × 256)
  
  Multi-Head Output:
  Z_multi = Z_concat · W^O ∈ ℝ^(6 × 64)
  
  For token "कब" (position 4):
  z_multi₄ = [z₁₄ || z₂₄ || z₃₄ || z₄₄] · W^O

Add & Norm Layer 1 (After Multi-Head Attention):

Residual Connection + Layer Normalisation

Input to attention: X_dec ∈ ℝ^(6 × 64)
Attention output: Z_multi ∈ ℝ^(6 × 64)

Residual connection:
  Z_residual = X_dec + Z_multi ∈ ℝ^(6 × 64)

Layer Normalization:
  Z_norm1 = LayerNorm(Z_residual) ∈ ℝ^(6 × 64)

Layer Normalization Mathematics
For each token position i:

  μᵢ = (1/64) Σⱼ₌₁⁶⁴ Z_residual[i,j]  # Mean
  
  σᵢ² = (1/64) Σⱼ₌₁⁶⁴ (Z_residual[i,j] - μᵢ)²  # Variance
  
  Z_norm1[i,j] = γⱼ · (Z_residual[i,j] - μᵢ)/√(σᵢ² + ε) + βⱼ

Where γ, β ∈ ℝ^64 are learned parameters, ε = 1e-5

Feed Forward Network:

Two Linear Transformations

W₁ ∈ ℝ^(64 × 2048)   # First projection (expand)
W₂ ∈ ℝ^(2048 × 64)   # Second projection (contract)
b₁ ∈ ℝ^2048          # First bias
b₂ ∈ ℝ^64            # Second bias

FFN Computation

FFN(x) = max(0, x·W₁ + b₁)·W₂ + b₂

For each token position i:
  H₁ᵢ = Z_norm1[i] · W₁ + b₁ ∈ ℝ^2048
  H₁ᵢ = max(0, H₁ᵢ)                    # ReLU activation
  Z_ffnᵢ = H₁ᵢ · W₂ + b₂ ∈ ℝ^64
  
  Z_ffn ∈ ℝ^(6 × 64)

Detailed Example for "कब":

  Input: z_norm1₄ ∈ ℝ^64

  First linear layer:
  h₁₄ⱼ = Σₖ₌₁⁶⁴ z_norm1₄ₖ · W₁ₖⱼ + b₁ⱼ  for j = 1,...,2048
  
  ReLU activation:
  h₁₄ⱼ = max(0, h₁₄ⱼ)
  
  Second linear layer:
  z_ffn₄ₖ = Σⱼ₌₁²⁰⁴⁸ h₁₄ⱼ · W₂ⱼₖ + b₂ₖ  for k = 1,...,64

Add & Norm Layer 2 (After FFN):

Z_residual2 = Z_norm1 + Z_ffn ∈ ℝ^(6 × 64)
Z_norm2 = LayerNorm(Z_residual2) ∈ ℝ^(6 × 64)

This is the final output of one decoder layer.

Output Generation (Final Softmax)

Output Projection to Vocabulary

  Hindi vocabulary size: V_hindi = 50,000
  Output projection matrix: W_out ∈ ℝ^(64 × 50000)
  Output bias: b_out ∈ ℝ^50000
  
Logits Computation:
  
  Logits = Z_norm2 · W_out + b_out ∈ ℝ^(6 × 50000)

  For position i, vocabulary word j:
  logit[i,j] = Σₖ₌₁⁶⁴ Z_norm2[i,k] · W_out[k,j] + b_out[j]

Softmax for Token Generation:

  P(word_j | position_i) = exp(logit[i,j]) / Σₖ₌₁⁵⁰⁰⁰⁰ exp(logit[i,k])

Example: Generating “कब” at position 4:

Hindi vocabulary indices:
आज → 1247, रात → 3891, घर → 2156, कब → 4782, आओगे → 8934, ? → 2

For position 4:
P(कब | pos_4) = exp(logit[4,4782]) / Σₖ₌₁⁵⁰⁰⁰⁰ exp(logit[4,k])

Expected high probabilities:
P(कब | pos_4) ≈ 0.85   (target word)
P(कैसे | pos_4) ≈ 0.08  (alternative question word)
P(क्यों | pos_4) ≈ 0.03  (another question word)
...

Complete Architecture Flow

Hindi Decoder Input: X_dec ∈ ℝ^(6×64)
English Encoder Output: Z_enc ∈ ℝ^(7×64)
                    ↓
Multi-Head Cross-Attention (4 heads):
Z_multi = MultiHead(X_dec, Z_enc, Z_enc) ∈ ℝ^(6×64)
                    ↓
Add & Norm 1:
Z_norm1 = LayerNorm(X_dec + Z_multi) ∈ ℝ^(6×64)
                    ↓
Feed Forward Network:
Z_ffn = FFN(Z_norm1) ∈ ℝ^(6×64)
                    ↓
Add & Norm 2:
Z_norm2 = LayerNorm(Z_norm1 + Z_ffn) ∈ ℝ^(6×64)
                    ↓
Output Projection:
Logits = Z_norm2 · W_out + b_out ∈ ℝ^(6×50000)
                    ↓
Softmax:
P(words) = softmax(Logits) ∈ ℝ^(6×50000)
                    ↓
Generated: [आज, रात, घर, कब, आओगे, ?]

Training vs Inference

Training (Teacher Forcing):

• All Hindi positions processed in parallel
• Ground truth used for next token prediction

Inference (Autoregressive):

• Generate one token at a time
• Use previously generated tokens as input

Step 1: Generate "आज" given <START>
Step 2: Generate "रात" given <START> आज  
Step 3: Generate "घर" given <START> आज रात
...
Step 6: Generate "?" given <START> आज रात घर कब आओगे

Each step involves the full computation pipeline above!

Finally I’ll explain in detail how the final linear projection and softmax layers work to generate Hindi vocabulary probabilities.

Starting Point: Final Add & Norm Output

Z_norm2 ∈ ℝ^(6 × 64)  # 6 Hindi tokens, 64-dimensional representations

Z_norm2 = [
    z₁  # Representation for position 1 ("आज")    ∈ ℝ^64
    z₂  # Representation for position 2 ("रात")   ∈ ℝ^64  
    z₃  # Representation for position 3 ("घर")    ∈ ℝ^64
    z₄  # Representation for position 4 ("कब")    ∈ ℝ^64
    z₅  # Representation for position 5 ("आओगे")  ∈ ℝ^64
    z₆  # Representation for position 6 ("?")     ∈ ℝ^64
]

Each zᵢ is a dense 64-dimensional vector containing contextual information about what token should appear at position i.

Linear Projection Layer (Output Layer)

Weight Matrix and Bias

    W_out ∈ ℝ^(64 × 50000)  # Projects 64-D hidden state to vocabulary logits
    b_out ∈ ℝ^50000         # Bias term for each vocabulary word
    
    Hindi vocabulary mapping:
    vocab = {
        0: "<PAD>",     1: "<START>",   2: "<END>",     3: "?",
        4: "।",         5: "क",         6: "का",        7: "की",
        ...
        1247: "आज",     2156: "घर",     3891: "रात",    4782: "कब", 
        8934: "आओगे",   ...            49999: "last_word"
    }

Linear Transformation Computation

    Logits = Z_norm2 · W_out + b_out ∈ ℝ^(6 × 50000)
    
    For position i and vocabulary word j:
    logit[i,j] = Σₖ₌₁⁶⁴ Z_norm2[i,k] · W_out[k,j] + b_out[j]

Detailed Example: Position 4 ("कब")
Let's compute logits for position 4:
    
    z₄ = [z₄₁, z₄₂, z₄₃, ..., z₄₆₄]  # 64-dimensional representation

    For vocabulary word "कब" (index 4782):
    logit[4, 4782] = z₄₁·W_out[1,4782] + z₄₂·W_out[2,4782] + ... + z₄₆₄·W_out[64,4782] + b_out[4782]
    
    For vocabulary word "कैसे" (index 5631):  
    logit[4, 5631] = z₄₁·W_out[1,5631] + z₄₂·W_out[2,5631] + ... + z₄₆₄·W_out[64,5631] + b_out[5631]
    
    ...for all 50,000 vocabulary words

This gives us a vector of raw logits for position 4:
    
    logits₄ = [logit[4,0], logit[4,1], logit[4,2], ..., logit[4,49999]] ∈ ℝ^50000


Complete Logits Matrix:
    
    Logits = [
        [logit[1,0], logit[1,1], ..., logit[1,49999]]  # Position 1 logits
        [logit[2,0], logit[2,1], ..., logit[2,49999]]  # Position 2 logits
        [logit[3,0], logit[3,1], ..., logit[3,49999]]  # Position 3 logits
        [logit[4,0], logit[4,1], ..., logit[4,49999]]  # Position 4 logits
        [logit[5,0], logit[5,1], ..., logit[5,49999]]  # Position 5 logits
        [logit[6,0], logit[6,1], ..., logit[6,49999]]  # Position 6 logits
      ] ∈ ℝ^(6 × 50000)

Softmax Layer:

Softmax converts raw logits to probabilities. Applied independently to each position.

Softmax Formula
  For position i:
    P(word_j | position_i) = exp(logit[i,j]) / Σₖ₌₀⁴⁹⁹⁹⁹ exp(logit[i,k])

Example: Position 4 Softmax Computation

    Raw logits for position 4:
    logit[4, 4782] = 8.2   # "कब" 
    logit[4, 5631] = 3.1   # "कैसे"
    logit[4, 6244] = 2.8   # "क्यों" 
    logit[4, 7891] = 1.5   # "कितना"
    logit[4, 0] = -2.1     # "<PAD>"
    ...
    logit[4, 1247] = -0.5  # "आज"


Step 1: Compute exponentials
    
    exp(logit[4, 4782]) = exp(8.2) ≈ 3641.0
    exp(logit[4, 5631]) = exp(3.1) ≈ 22.2
    exp(logit[4, 6244]) = exp(2.8) ≈ 16.4
    exp(logit[4, 7891]) = exp(1.5) ≈ 4.5
    exp(logit[4, 0]) = exp(-2.1) ≈ 0.12
    ...
    exp(logit[4, 1247]) = exp(-0.5) ≈ 0.61

Step 2: Sum all exponentials

    Z₄ = Σₖ₌₀⁴⁹⁹⁹⁹ exp(logit[4,k]) 
       ≈ 3641.0 + 22.2 + 16.4 + 4.5 + 0.12 + ... + 0.61 + ... 
       ≈ 4000.0  (approximate)

Step 3: Normalize to get probabilities

    P(कब | pos_4) = 3641.0 / 4000.0 ≈ 0.91
    P(कैसे | pos_4) = 22.2 / 4000.0 ≈ 0.0056  
    P(क्यों | pos_4) = 16.4 / 4000.0 ≈ 0.0041
    P(कितना | pos_4) = 4.5 / 4000.0 ≈ 0.0011
    P(<PAD> | pos_4) = 0.12 / 4000.0 ≈ 0.00003
    ...
    P(आज | pos_4) = 0.61 / 4000.0 ≈ 0.00015

Verification:

  Σₖ₌₀⁴⁹⁹⁹⁹ P(word_k | pos_4) = 1.0  ✓



Complete Probability Matrix:

P ∈ ℝ^(6 × 50000)

P = [
    [P(w₀|pos₁), P(w₁|pos₁), ..., P(w₄₉₉₉₉|pos₁)]  # Position 1 probabilities
    [P(w₀|pos₂), P(w₁|pos₂), ..., P(w₄₉₉₉₉|pos₂)]  # Position 2 probabilities  
    [P(w₀|pos₃), P(w₁|pos₃), ..., P(w₄₉₉₉₉|pos₃)]  # Position 3 probabilities
    [P(w₀|pos₄), P(w₁|pos₄), ..., P(w₄₉₉₉₉|pos₄)]  # Position 4 probabilities
    [P(w₀|pos₅), P(w₁|pos₅), ..., P(w₄₉₉₉₉|pos₅)]  # Position 5 probabilities
    [P(w₀|pos₆), P(w₁|pos₆), ..., P(w₄₉₉₉₉|pos₆)]  # Position 6 probabilities
]

Each row sums to 1:

Σⱼ₌₀⁴⁹⁹⁹⁹ P[i,j] = 1.0  for all i ∈ {1,2,3,4,5,6}

Token Selection Strategies:

1. Greedy Decoding
    Select highest probability token:
    
    predicted_token[i] = argmax_j P[i,j]
    
    For position 4:
    predicted_token[4] = argmax_j P[4,j] = 4782 ("कब")

2. Top-k Sampling (k=5)
    Sample from top 5 highest probability tokens:
    
    Top-5 for position 4:
    1. "कब": P = 0.91
    2. "कैसे": P = 0.0056  
    3. "क्यों": P = 0.0041
    4. "कितना": P = 0.0011
    5. "कहाँ": P = 0.0008
    
    Renormalize: P_new = P_old / (0.91 + 0.0056 + 0.0041 + 0.0011 + 0.0008)
    Sample from this distribution

3. Temperature Sampling
Modify logits before softmax:

    logit_temp[i,j] = logit[i,j] / temperature
    
    Temperature = 0.1 → More peaked (conservative)
    Temperature = 2.0 → Flatter (more random)

Mathematical Flow Summary

Final Hidden States: Z_norm2 ∈ ℝ^(6×64)
                           ↓
Linear Projection:
  logit[i,j] = Σₖ₌₁⁶⁴ Z_norm2[i,k] · W_out[k,j] + b_out[j]
  Logits ∈ ℝ^(6×50000)
                           ↓
Softmax (per position):
  P[i,j] = exp(logit[i,j]) / Σₖ₌₀⁴⁹⁹⁹⁹ exp(logit[i,k])
  Probabilities ∈ ℝ^(6×50000)
                           ↓
Token Selection:
  predicted_tokens = [आज, रात, घर, कब, आओगे, ?]

Why This Works

Linear Layer: Maps contextual representations to vocabulary space
Softmax: Ensures valid probability distribution over all possible words
Position Independence: Each position gets its own probability distribution
Learned Weights: W_out and b_out learn which hidden features correspond to which words

This architecture allows the model to generate any valid Hindi sentence by 
learning the mapping from contextual representations to vocabulary probabilities!