Day 4 of 100: Building a Transformer Block From Scratch With NumPy

On Day 3, I implemented scaled dot-product attention and learned how Queries, Keys, and Values help a Transformer decide which tokens are relevant to each other.

That raised the next question:

What happens after attention?

For Day 4 of my 100 Days of Generative AI journey, I extended the previous experiment and built a small Transformer block using NumPy.

The block contains:

Attention
→ Residual Connection
→ Layer Normalization
→ Feed-Forward Network
→ Residual Connection
→ Layer Normalization

This exercise helped me understand how several simple components work together to form one of the core building blocks of a Transformer.

Starting From Day 3

I reused the same small Query, Key, and Value matrices from Day 3.

Q = np.array([
    [1.0, 0.0],
    [0.0, 1.0],
    [1.0, 1.0]
])

K = np.array([
    [1.0, 0.0],
    [0.0, 1.0],
    [1.0, 1.0]
])

V = np.array([
    [10.0, 0.0],
    [0.0, 10.0],
    [5.0, 5.0]
])

There are three tokens, each represented by two features.

I also defined the original token representation:

X = np.array([
    [1.0, 0.0],
    [0.0, 1.0],
    [1.0, 1.0]
])

The attention calculation is the same as Day 3:

scores = Q @ K.T
norm_scores = scores / np.sqrt(K.shape[-1])
attention_weights = softmax(norm_scores)
attention_output = attention_weights @ V

The attention output is approximately:

[[6.017, 3.983],
 [3.983, 6.017],
 [5.000, 5.000]]

Day 4 begins from here.

Step 1: Add a Residual Connection

The first new operation is:

residual1 = X + attention_output

This gives:

[[7.017, 3.983],
 [3.983, 7.017],
 [6.000, 6.000]]

This is called a residual connection, or skip connection.

Instead of replacing the original representation with the attention output, the Transformer adds the two together.

Conceptually:

Original token representation
          +
Context from attention
          ↓
Updated representation

A simple way I think about it is:

Attention adds new information, while the residual connection makes sure the original information is not lost.

Residual connections also help information and gradients move through deep neural networks more effectively.

Step 2: Apply Layer Normalization

After the residual connection, I apply LayerNorm:

normalized1 = layer_norm(residual1)

I implemented it manually:

def layer_norm(x, eps=1e-5):
    mean = np.mean(x, axis=-1, keepdims=True)
    variance = np.var(x, axis=-1, keepdims=True)

    return (x - mean) / np.sqrt(variance + eps)

The result is approximately:

[[ 1.0, -1.0],
 [-1.0,  1.0],
 [ 0.0,  0.0]]

Layer normalization standardizes the features of each token.

In simple terms:

Token features
   ↓
Calculate mean
   ↓
Calculate variance
   ↓
Normalize values

This helps keep representations on a more stable scale as they pass through Transformer layers.

Why Does the Third Token Become [0, 0]?

This was the most interesting part of the experiment.

Before LayerNorm, the third token is:

[6, 6]

Its mean is:

(6 + 6) / 2 = 6

Subtracting the mean:

[6 - 6, 6 - 6]

gives:

[0, 0]

Because both features are identical, there is no variation between them.

So after normalization:

[6, 6] → [0, 0]

Seeing this happen numerically made LayerNorm much easier to understand.

Step 3: Feed-Forward Network

The next component is a small neural network.

I defined two weight matrices:

W1 = np.array([
    [1.0, -1.0, 0.5],
    [0.5, 1.0, -0.5]
])

W2 = np.array([
    [1.0, 0.0],
    [0.0, 1.0],
    [0.5, 0.5]
])

The calculation is:

hidden = relu(normalized1 @ W1)
ff_output = hidden @ W2

The dimensional flow is:

2 features
    ↓
3 hidden features
    ↓
2 output features

So the feed-forward network temporarily expands the representation and then projects it back down.

Get Sunil Veeramachaneni’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Real Transformers use much larger dimensions, but the basic idea is the same.

Why Use an Activation Function?

Between the two linear transformations, I apply ReLU:

def relu(x):
    return np.maximum(0, x)

For example:

[-2.0, 0.5, 3.0]

becomes:

[0.0, 0.5, 3.0]

Without an activation function, multiple linear layers would still behave like one larger linear transformation.

The activation introduces non-linearity, which allows the network to learn more complex patterns.

Modern Transformer architectures may use other activation functions such as GELU or SwiGLU, but ReLU keeps this example easy to follow.

Step 4: Add the Second Residual Connection

After the feed-forward network, I add another residual connection:

residual2 = normalized1 + ff_output

Then I normalize again:

output = layer_norm(residual2)

The final output is approximately:

[[ 1.0, -1.0],
 [-1.0,  1.0],
 [ 0.0,  0.0]]

And with that, the small Transformer block is complete.

Attention and Feed-Forward Networks Do Different Jobs

This distinction became much clearer to me today.

Attention

Attention allows one token to gather information from other tokens.

Token 1 ↔ Token 2 ↔ Token 3

It answers:

Which other tokens are important to me?

Feed-Forward Network

The feed-forward network processes each token independently after attention.

Token representation
      ↓
Linear layer
      ↓
Activation
      ↓
Linear layer

So my simplified mental model is:

Attention
→ gather contextual information

Feed-Forward Network
→ process that information

The Complete Transformer Block

The whole program can be summarized in a few lines:

scores = Q @ K.T
norm_scores = scores / np.sqrt(K.shape[-1])

attention_weights = softmax(norm_scores)
attention_output = attention_weights @ V

residual1 = X + attention_output
normalized1 = layer_norm(residual1)

hidden = relu(normalized1 @ W1)
ff_output = hidden @ W2

residual2 = normalized1 + ff_output
output = layer_norm(residual2)

Conceptually:

Input
  ↓
Attention
  ↓
Add Residual
  ↓
LayerNorm
  ↓
Feed-Forward Network
  ↓
Add Residual
  ↓
LayerNorm
  ↓
Output

What initially looked like a complicated Transformer architecture is starting to look like a sequence of understandable operations.

Connecting Days 1–4

My learning path now looks like this:

Day 1 — Tokenization

Text
→ Tokens
→ Token IDs

Day 2 — Embeddings

Text
→ Vectors
→ Semantic Similarity

Day 3 — Attention

Q, K, V
→ Attention Scores
→ Softmax
→ Contextual Information

Day 4 — Transformer Block

Attention
→ Residual
→ LayerNorm
→ Feed Forward
→ Residual
→ LayerNorm

Putting everything together:

Text
 ↓
Tokens
 ↓
Embeddings
 ↓
Attention
 ↓
Residual + LayerNorm
 ↓
Feed-Forward Network
 ↓
Residual + LayerNorm
 ↓
Transformer Block Output

For the first time, I can clearly see how the lower-level concepts are beginning to form the architecture behind an LLM.

What I Learned Today

My main takeaways from Day 4 are:

1. Attention is only one part of a Transformer block.
2. Residual connections preserve earlier information while adding new information.
3. Layer normalization keeps token representations on a stable scale.
4. The feed-forward network processes each token independently.
5. The hidden layer can expand the representation before projecting it back down.
6. Activation functions introduce non-linearity.
7. A Transformer block is a combination of several simple components, not one mysterious algorithm.

The biggest takeaway for me was seeing how these parts fit together:

Attention
+
Residual Connections
+
Normalization
+
Feed-Forward Network

Understanding them individually makes the overall Transformer architecture much easier to follow.

Day 4 Complete

Four days into this journey, the progression is becoming clearer.

I started with:

Text

and gradually moved through:

Tokens
→ Embeddings
→ Attention
→ Transformer Block

There is still a lot ahead — multi-head attention, positional information, deeper Transformer stacks, and eventually complete language models.

But building each component from scratch is making the architecture feel much less like a black box.

Day 4/100 complete.

Code:

https://github.com/sunilveera/learn_AI_daily/tree/main/Day_004

One concept, one small program, one day at a time.

On to Day 5.