Convolutional Neural Networks: How Machines Learn to See
AI From First Principles — Part 7
Well, It’s been a while. And we are nearing the end of this series.
The last time I wrote in this series we were inside Recurrent Neural Networks — the first architecture that could actually process time. We built and saw a sequence, unfolding step by step, with memory carried forward through a hidden state. We built it in C with a genetic algorithm and watched 168 weights learn a sequence from nothing.
CNNs took longer. Not because they are complicated (once you actually see what is happening they are quite elegant ), but because I kept reading explanations that skipped the part I most wanted to understand. Every article jumps to “slide a filter across the image” on paragraph two and treats the rest as obvious. It is not obvious. Why does sliding a filter work? What is convolution actually doing mathematically? Why does it solve a problem that a standard neural network cannot? And before any of that: what even is an image to a computer?
If you skip those questions you end up with a recipe, not understanding. This series is not about recipes.
So we are going all the way back. To the pixel. Then we will build up from there. It is very long, but worth it.
Get a pen, paper and possibly open up your IDE of choice like Vs code, or an cloud environment like colab.
Code at the end. GitHub link at the end. Right now, we talk.
What Is an Image to a Computer?
When you look at a photo of a dog you see fur, eyes, a tail, maybe a personality. When Python opens that same file, the computer sees a grid of numbers. That is the entire gap we need to bridge before we can talk about any network architecture.
The Pixel Grid
An image is a rectangle of tiny squares called pixels. Each pixel holds a number. That number represents something about the color or brightness at that location.
For a grayscale image the number is simple: brightness. The convention is 0 to 255. Zero is pure black. 255 is pure white. Everything in between is a shade of gray.
A 6×6 grayscale image is literally just a 6×6 matrix of integers:
import numpy as npimg = np.array([
[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 200, 200, 0, 0],
[ 0, 0, 200, 200, 0, 0],
[ 0, 0, 200, 200, 0, 0],
[ 0, 0, 0, 0, 0, 0],
[ 0, 0, 0, 0, 0, 0]
], dtype=np.uint8)That array is a small bright rectangle floating in darkness. There is no picture here. No semantics. Just a matrix. This is what the network receives.
Why 0 to 255? Quantization.
The range 0–255 comes from storage. The standard image format allocates 8 bits per channel per pixel. With 8 bits you can represent 2 ⁸ = 256 distinct values, numbered 0 through 255.
Here is the thing though. Real light is continuous. A surface does not have brightness 73 or 74, rather, it has some continuous physical intensity somewhere between them. When a camera sensor captures a scene, it measures continuous light and converts it to a discrete integer. That process ( snapping a continuous value to the nearest bucket ) is called quantization.
Think of it like this. You have an infinite ruler going from 0.0 to 1.0, and you have 256 buckets. Each bucket covers a range of about 1/256th of the ruler. Anything that falls in that range gets assigned that bucket number.
Continuous brightness → 8-bit bucket
0.000 → 0
0.200 → 51
0.500 → 127
0.784 → 200
1.000 → 255You are throwing away information. But 256 buckets turns out to be enough that the eye usually cannot tell. The human visual system cannot reliably distinguish differences smaller than about 0.4% of the brightness range, and 1/256 ≈ 0.39%. So the quantization sits right at the edge of perception.
Higher bit depths exist (12-bit RAW files, 16-bit medical scans ) but for standard computer vision, 8-bit is the convention.
Color: Enter the Channels
Grayscale has one number per pixel. Color needs three.
The RGB model represents each pixel as a combination of Red, Green, and Blue light. Each component runs 0 to 255 independently. Mix them and you get any color:
[255, 0, 0] → pure red
[ 0, 255, 0] → pure green
[ 0, 0, 255] → pure blue
[255, 255, 0] → yellow (red + green at full, no blue)
[128, 0, 128] → purple (half red, half blue)
[255, 255, 255] → white (all channels full)
[ 0, 0, 0] → black (all channels zero)The reason RGB works is that the human eye has three types of color receptors, each sensitive to roughly red, green, or blue wavelengths. You mix those three and you can trick the eye into seeing any color. Computer displays do the same thing, every pixel on your screen is actually three tiny lights, one red, one green, one blue.
Now instead of a 2D matrix, a color image is a 3D tensor. A 28×28 color image has shape (28, 28, 3). Twenty-eight rows, twenty-eight columns, three channels. Each position in the grid holds three numbers instead of one.
# Grayscale — one number per pixel
gray_img = np.zeros((28, 28), dtype=np.uint8)
print(gray_img.shape) # (28, 28)
# Color - three numbers per pixel
color_img = np.zeros((28, 28, 3), dtype=np.uint8)
print(color_img.shape) # (28, 28, 3)
# Set one pixel to red
color_img[14, 14] = [255, 0, 0]You can also think of a color image as three separate grayscale images stacked on top of each other. One layer is how red each pixel is. One layer is how green. One layer is how blue. Those three layers are the three channels.
This idea of channels becomes really important in a moment. CNNs generalize it heavily.
Normalization
Networks do not like inputs in the range 0–255. Large input values lead to large activations which lead to unstable training. The simple fix is dividing by 255 before the network sees the image:
img_normalized = img.astype(np.float64) / 255.0
# Now every value is between 0.0 and 1.0A slightly stronger version also subtracts the mean and divides by the standard deviation of the training set . This centers the values near zero, which keeps gradients better behaved throughout training, this is called Standardization. It comes from the standard normal distribution, that’s a story for another day (statistics). For our from-scratch CNN on toy data, dividing by 255 is sufficient.
Why a Standard Neural Network Cannot Handle Images
In the neural network chapter we built a fully connected network. Every neuron in layer n connects to every neuron in layer n+1. For small tabular inputs , like ten features, twenty features, this is fine.
For images it is a disaster. Here is why.
The Parameter Explosion
A 28×28 grayscale image has 784 pixel values. To connect those to a first hidden layer of 256 neurons, you need:
784 × 256 = 200,704 weightsThat is just one layer on the smallest standard benchmark. Scale to a real-world image:
224 × 224 × 3 (ImageNet standard) = 150,528 input values
150,528 × 256 = 38,535,168 weights in the first layer aloneThirty-eight million parameters before the network has extracted a single useful feature. You would need enormous amounts of training data just to prevent those weights from overfitting to noise. Most of those weights would learn nothing meaningful.
But the parameter count is actually the smaller problem.
Spatial Blindness
When you flatten a 28×28 image into a vector of 784 numbers, you destroy all spatial structure. Pixel (0, 0) and pixel (27, 27) become two unrelated entries in a list. The network has no way to know they were ever part of a grid, let alone that neighboring pixels tend to be related.
And in images, neighbors are everything. An edge is a region where pixel values change sharply across adjacent pixels. A curve is a connected sequence of edge responses. A face is a specific spatial arrangement of curves and regions with consistent relationships between them. None of that information survives flattening.
The network would theoretically have to reconstruct all of that from scratch through learning. It is possible in principle. In practice it requires far more data than anyone has, and generalization is poor.
CNNs fix this by never flattening in the first place. They process the image while preserving its spatial structure throughout.
The Intuition for Convolution
This is the part that most articles rush. Do not rush it. Understanding why convolution works is the whole thing.
Start With a Question
Here is an image of a handwritten digit. Somewhere in that 28×28 grid there is a horizontal stroke — part of the top of a 7, say. The network needs to detect that stroke. How should it do it?
Option A: Learn a separate weight from every pixel to a “horizontal stroke” neuron. 784 weights, most of which correspond to positions where the stroke never appears, teaching the neuron nothing useful.
Option B: Learn what a horizontal stroke looks like locally, a small template, maybe 3×3 pixels, and then check every region of the image for that pattern.
Option B is what convolution does. And it is clearly smarter.
The Filter
A filter is a small matrix of learned weights. For a 3×3 filter:
┌──────┬──────┬──────┐
│ w₁₁ │ w₁₂ │ w₁₃ │
├──────┼──────┼──────┤
│ w₂₁ │ w₂₂ │ w₂₃ │
├──────┼──────┼──────┤
│ w₃₁ │ w₃₂ │ w₃₃ │
└──────┴──────┴──────┘Nine numbers. These are what the network learns.
To apply the filter to the image, you place it over a 3×3 patch of the image and compute the dot product — multiply each filter weight by the corresponding pixel value and sum them all up. That gives you one number: how strongly this filter responded to this patch.
For a patch P and filter F:
response = Σᵢ Σⱼ F[i,j] × P[i,j]If the filter is designed to detect horizontal edges, it might look like:
[-1, -1, -1]
[ 0, 0, 0]
[ 1, 1, 1]Negative weights on the top row, zero in the middle, positive on the bottom. Apply this to a region where the top is dark and the bottom is bright (the definition of a horizontal edge where brightness increases downward ) and you get a large positive response. Apply it to a flat uniform region and the negatives and positives cancel out to near zero. The filter fires exactly where the pattern it encodes is present.
The Sliding Window
You do not apply the filter to just one patch. You slide it across the entire image, one position at a time, computing a response at every position.
Image (6×6): Filter (3×3):
┌──┬──┬──┬──┬──┬──┐ ┌──┬──┬──┐
│ │ │ │ │ │ │ │-1│-1│-1│
├──┼──┼──┼──┼──┼──┤ ├──┼──┼──┤
│ │ │ │ │ │ │ │ 0│ 0│ 0│
├──┼──┼──┼──┼──┼──┤ ├──┼──┼──┤
│ │██│██│██│ │ │ →apply→ │ 1│ 1│ 1│
├──┼──┼──┼──┼──┼──┤ └──┴──┴──┘
│ │ │ │ │ │ │
├──┼──┼──┼──┼──┼──┤
│ │ │ │ │ │ │
├──┼──┼──┼──┼──┼──┤
│ │ │ │ │ │ │
└──┴──┴──┴──┴──┴──┘The filter can sit at (0,0), (0,1), (0,2), (0,3) across the top row, then shift down and repeat. For a 6×6 image and a 3×3 filter with no padding, it can sit at 4×4 = 16 positions. At each position it produces one number. Those 16 numbers form a 4×4 output called a feature map.
The output size formula:
output_size = input_size - filter_size + 1
4 = 6 - 3 + 1 ✓
The feature map is a spatial map of where the pattern appeared. High values where the filter fired strongly. Low values where it did not.
def convolve2d(image, kernel):
H, W = image.shape
kH, kW = kernel.shape
out_H = H - kH + 1
out_W = W - kW + 1
output = np.zeros((out_H, out_W))
for i in range(out_H):
for j in range(out_W):
patch = image[i:i+kH, j:j+kW]
output[i, j] = np.sum(patch * kernel)
return outputRunning it gives :
image = np.array([
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0],
[1, 1, 1, 1, 1, 1], # bright horizontal line
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0],
[0, 0, 0, 0, 0, 0]
], dtype=float)
h_edge = np.array([
[-1, -1, -1],
[ 0, 0, 0],
[ 1, 1, 1]
], dtype=float)
feature_map = convolve2d(image, h_edge)
print(feature_map)The row in the output that crosses the bright line in the image will light up with large values. Every other row stays near zero. The filter found exactly what it was looking for.
Weight Sharing — Why This Changes Everything
Here is the thing that makes CNNs fundamentally different from fully connected networks.
The same nine filter weights are used at every position in the image.
Not a different set of weights for the top-left corner versus the center versus the bottom-right. The same nine numbers, reused everywhere.
This is called weight sharing, and it has two consequences.
First: efficiency. A 3×3 filter applied across a 224×224 image uses 9 weights to produce a 222×222 feature map. A fully connected equivalent would need 222×222 = 49,284 separate weights just to produce one output value per position. The difference is not small.
Get Ramadhan Zome’s stories in your inbox
Join Medium for free to get updates from this writer.
Second: translation equivariance. If a cat’s ear appears in the top-left of the image, the filter that detects ear-like patterns will fire there. If you move the cat so the ear is in the bottom-right, the same filter fires there instead. The detection travels with the pattern. A fully connected network has no such property, it would need to separately learn “ear in top-left” and “ear in bottom-right” as distinct features.
Multiple Filters — Multiple Feature Maps
One filter detects one type of pattern. But images have many types of patterns. So we use multiple filters simultaneously.
If you use 4 filters on a grayscale image, you get 4 feature maps — one per filter. Each filter learns to respond to something different. One might respond to horizontal edges. Another to vertical edges. Another to a corner pattern. Another to a texture. The network learns which filters are useful for the task entirely on its own.
For our 8×8 grayscale input with 4 filters of size 3×3:
Input: (8, 8)
After 4 filters: (6, 6, 4)The output now has a third dimension — the channel dimension. We started with 1 channel (grayscale) and produced 4 channels (one feature map per filter). This is the key moment where channels stop meaning “color” and start meaning “detected feature.” This is the ‘what’ , it confused me for a while
Convolution Across Channels
When the input has multiple channels, like a color image with 3, the filter also needs to span all channels. A 3×3 filter on a 3-channel image is actually a 3×3×3 volume. At each position, it reaches into all three channels simultaneously and computes one dot product across all of them.
RGB image patch at position (i, j):
Red channel: 3×3 values
Green channel: 3×3 values → filter is also 3×3×3 → one output number
Blue channel: 3×3 valuesThe math: for input with C channels and a filter of size kH×kW×C:
response(i,j) = Σc Σm Σn F[c,m,n] × X[i+m, j+n, c]Sum across all channels c, all filter rows m, all filter columns n. One number out.
This is why filter design in a CNN always has to account for the number of input channels. When you see Conv(in_channels=3, out_channels=16, kernel_size=3), that means: 16 filters, each of size 3×3×3. Each filter produces one feature map. Together they produce 16 feature maps stacked as the new output, the new channel dimension.
ReLU (Rectified Linear Unit) — The Non-Linearity That Keeps Things Honest
After convolution, we apply a non-linearity. Without it, stacking multiple conv layers is mathematically identical to doing one, the compositions of linear functions is still a linear function. All the depth buys you nothing.
The activation function used in CNNs is almost universally ReLU:
ReLU(x) = max(0, x)Negative values become zero. Positive values pass through unchanged.
def relu(x):
return np.maximum(0, x)Why does this work? A few reasons.
Negative activations in a feature map generally mean “this filter did not fire here.” The magnitude of the negative value carries almost no signal. There is no useful distinction between “slightly did not fire” and “strongly did not fire.” ReLU cleans that up.
More importantly, ReLU does not saturate for positive values. Compare this to sigmoid, which squashes everything toward 0 or 1 for large inputs, killing gradients. ReLU lets gradients flow through cleanly wherever the input is positive. This matters enormously in deep networks. We saw what vanishing gradients do to RNNs — same enemy here, same need to prevent it.
During backpropagation, ReLU’s gradient is trivial:
dL/dx = dL/dy if x > 0
dL/dx = 0 if x ≤ 0Where the input was positive, the gradient passes through unchanged. Where it was negative, the gradient stops. Clean and fast.
Pooling — Compressing Without Losing What Matters
After convolution and ReLU, the feature maps are still large. We want to reduce their spatial size, smaller means cheaper computation in later layers. But we also want to keep the information that matters.
Max pooling does this by dividing the feature map into small non-overlapping windows and keeping only the maximum value from each:
Feature map (4×4): After 2×2 max pooling (2×2):┌────┬────┬────┬────┐ ┌────┬────┐
│ 1 │ 3 │ 5 │ 2 │ │ 6 │ 5 │
├────┼────┼────┼────┤ → ├────┼────┤
│ 4 │ 6 │ 1 │ 0 │ │ 9 │ 8 │
├────┼────┼────┼────┤ └────┴────┘
│ 2 │ 9 │ 8 │ 3 │
├────┼────┼────┼────┤
│ 1 │ 3 │ 4 │ 7 │
└────┴────┴────┴────┘Top-left 2×2 window: values 1, 3, 4, 6 → keep 6. Top-right 2×2 window: values 5, 2, 1, 0 → keep 5. And so on.
The spatial dimensions halve. A 6×6 feature map becomes 3×3. The number of channels stays the same.
Why keep the maximum? Because in a feature map produced by a convolution, high values mean “the pattern was strongly present here.” The maximum value in a region is the strongest evidence. Keeping it means keeping the fact that the pattern appeared somewhere in that region, even if you lose the exact position.
This second point is actually a feature, not a loss. An edge that appears at pixel (22, 15) versus pixel (22, 16) should not confuse a classifier. Max pooling introduces a small amount of spatial tolerance , where the response is the same as long as the maximum stays in the same pooling window. This makes the network more robust to small shifts and distortions in the input.
Putting It All Together
Here is the full architecture for our from-scratch CNN. Input is an 8×8 grayscale image. We classify into three categories: vertical line, horizontal line, diagonal.
Input image: 8 × 8 × 1Input image: 8 x 8 x 1 │
▼Convolution: 4 filters, 3×3
Each filter slides across the image
Each produces one 6×6 feature map
4 filters → 4 feature maps stackedOutput: 6 × 6 × 4 │
▼ReLU
Negatives zeroed out
Shape unchangedOutput: 6 × 6 × 4 │
▼Max Pooling: 2×2 windows
Each 6×6 feature map → 3×3Output: 3 × 3 × 4 │
▼Flatten
3 × 3 × 4 = 36 numbers in a vectorOutput: 36 │
▼Dense Layer
36 inputs → 3 class scoresOutput: 3 │
▼Softmax → Probabilities
Cross-Entropy → Loss
Why does flattening work at this point but not at the start? Because by the time we flatten, the spatial structure has already been processed. The 36 numbers are not raw pixels, they are compressed, spatially-pooled summaries of detected features. The meaning is already extracted. Connecting them to a dense classifier is now sensible.
How the Network Learns
Training a CNN uses the same logic as any network we have built in this series: forward pass, compute loss, backpropagate gradients, update weights.
The loss is cross-entropy. Softmax converts the three raw class scores into probabilities, and cross-entropy measures how far the predicted probabilities are from the correct answer:
Loss = -log(probability assigned to the correct class)If the correct class gets probability 0.9, loss is -log(0.9) ≈ 0.1. Small. If the correct class gets probability 0.1, loss is -log(0.1) ≈ 2.3. Large. The network gets penalized proportionally to how wrong it was.
The gradient of softmax cross-entropy combined is elegant:
gradient = predicted_probabilities - one_hot_targetWhere one_hot_target is a vector with 1 in the correct class position and 0 everywhere else. If the network predicted [0.1, 0.7, 0.2] and the correct class was 1, the gradient is [0.1, -0.3, 0.2]. Push the correct class higher, pull the others lower.
Backprop through the dense layer is identical to what we saw in the earlier neural network chapter. Backprop through pooling sends gradients only to the position that held the maximum. Backprop through ReLU sends gradients through where the input was positive, zeros everywhere else.
Backprop through convolution is the interesting one. During the forward pass, the same filter weights touched every spatial position. So during the backward pass, gradients accumulate from every position:
# Simplified: accumulate filter gradient across all positions
for i in range(out_H):
for j in range(out_W):
patch = image[i:i+kH, j:j+kW]
d_filter += patch * d_out[i, j] # accumulate
d_image[i:i+kH, j:j+kW] += filter * d_out[i, j]Weight sharing in the forward pass becomes gradient accumulation in the backward pass. They are two sides of the same operation.
What the Trained Filters Look Like
Here is something worth sitting with. After training, the filters are not random anymore. They have learned structure.
For a network trained on real images, the first-layer filters typically end up looking like oriented edge detectors, color contrast detectors, or frequency patterns. Nobody programmed those shapes. The network learned them because they were useful.
This is not specific to vision. It is a general property of training: the representations that emerge are the ones that were useful for the task. The architecture just sets up the conditions under which learning happens. The specific content of what gets learned is determined by the data.
The Code
Everything described here is implemented from scratch using only NumPy in the buildneural repository.
Every layer — convolution, ReLU, max pooling, flatten, dense, softmax cross-entropy, is written by hand, including the full backward passes.
The network trains on a toy dataset of 8×8 images with three pattern classes, generated in code so there is nothing to download. You can watch it go from random weights to 100% accuracy in about 12 epochs.
https://github.com/RamadhanAdam/buildneural/tree/main/03-convolutional-neural-network
Run in bash:
cd 03-convolutional-neural-network
pip install numpy
python main.pyThen the output looks like this:
Training a CNN from scratch
Training images: 192
Epoch 01 | loss 0.9950 | accuracy 54.17%
Epoch 02 | loss 0.7161 | accuracy 89.58%
Epoch 03 | loss 0.5019 | accuracy 97.92%
Epoch 04 | loss 0.3233 | accuracy 100.00%
...
Epoch 12 | loss 0.0229 | accuracy 100.00%Read layers.py for the individual layer implementations. Read cnn.py for how they wire together. Read main.py for the training loop and dataset generation. Every line is commented.
Where This Sits in the Series
We started with one neuron making a binary decision. Then stacked layers and taught them to learn from error. Then gave them competition. Then memory through Hopfield networks. Then time through RNNs.
This is the first architecture that processes space as a first-class concept. We have an actual spatial structure with local patterns, hierarchical composition, and position-aware processing.
The natural next question is what happens when you go very deep. When you have 50 layers instead of 2. When gradients vanish long before they reach the early layers. When the network needs to learn not just local patterns but long-range dependencies across the entire image.
That question leads to residual connections, attention mechanisms, and eventually transformers, which, as it turns out, abandon the spatial inductive bias of CNNs entirely and replace it with something more general.
But that is the next part.
Written by: 6152 616D 6864 6E61 4120 6164 206D 6F5A 656D
