ResNet Model and Its Implementation Using Python

ResNet Model

Before 2015, conventional deep learning wisdom held that stacking more layers onto a Convolutional Neural Network (CNN) would naturally increase model capacity and yield higher accuracy. However, machine learning researchers encountered a strange paradox when scaling networks beyond 20 layers: deeper models began performing worse than shallower networks on both training and validation sets.

This performance degradation was not caused by overfitting. Instead, it stemmed from the Vanishing Gradient Problem. As backpropagation errors flowed backward through dozens of sequential weight matrices, numerical gradient values shrank exponentially toward zero, leaving early layers virtually untrainable.

He et al. solved this structural barrier with Residual Neural Networks (ResNet). By introducing identity shortcut connections — commonly called skip connections — ResNet reformulated deep learning architectures, allowing neural networks to scale beyond 100 or even 1,000 layers without optimization degradation.

In this article, we will examine the theoretical principles of residual learning, unpack the mathematical framework behind skip connections, explore bottleneck block variations, and construct a complete, runnable ResNet architecture in Python using PyTorch.

The Degradation Problem and Residual Learning Paradigm

In a standard plain convolutional network (such as VGG), stacked layers attempt to directly learn an underlying target mapping function H(x) from an input feature matrix x:

• Direct Target Mapping: H(x)

As networks grow deeper, forcing every sequence of stacked weight layers to fit H(x) from scratch becomes numerically unstable.

ResNet reformulates this learning objective. Instead of asking stacked convolutional layers to approximate H(x) directly, it lets them fit a residual mapping function F(x):

F(x) = H(x) — x

Rearranging this equation yields the true output of a Residual Block:

H(x) = F(x) + x

By adding the original input x directly to the output of the convolutional block via an identity shortcut, the network only needs to learn the residual difference F(x). If a specific set of convolutional layers is unnecessary for performance, the optimization algorithm can simply drive weights in F(x) toward zero, collapsing the block into an identity mapping (H(x) = x) rather than degrading overall network accuracy.

1. Mathematical Framework: Why Skip Connections Fix Vanishing Gradients

The true power of residual blocks becomes clear when analyzing backpropagation mathematically.

During training, the gradient of the loss function L with respect to input x is calculated using the chain rule:

dL / dx = (dL / dH(x)) * (dH(x) / dx)

Since H(x) = F(x) + x, taking the derivative with respect to x gives:

dH(x) / dx = (dF(x) / dx) + 1

Substituting this back into the chain rule gradient formula yields:

dL / dx = (dL / dH(x)) * ( (dF(x) / dx) + 1 )

= (dL / dH(x)) * (dF(x) / dx) + (dL / dH(x))

Notice the standalone +1 term on the right side of the equation! Even if the weight gradients within the convolutional layers dF(x) / dx shrink toward zero due to vanishing gradients, the error term (dL / dH(x)) can still flow backward completely unimpeded through the identity path. This guarantees that early layers continue receiving clean gradient updates regardless of total network depth.

1. Residual Block Variants: BasicBlock vs. Bottleneck

Depending on network depth and computational constraints, ResNet architectures use two primary block configurations:

Get Iqra Mahmood’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

A. BasicBlock (ResNet-18 and ResNet-34) The BasicBlock consists of two consecutive 3x3 Convolutional layers, each followed by Batch Normalization and ReLU non-linear activations. An identity shortcut bypasses both convolutions and adds the original input x to the block output right before the final ReLU activation.

B. Bottleneck Block (ResNet-50, ResNet-101, and ResNet-152) As networks grow deeper, 3x3 convolutions on high-channel feature maps become computationally expensive. The Bottleneck block uses three sequential convolutions:

1. 1x1 Convolution: Reduces channel dimensions (e.g., from 256 down to 64) to save compute.
2. 3x3 Convolution: Performs spatial feature extraction on the reduced channel feature map.
3. 1x1 Convolution: Restores channel dimensions back to the target depth (e.g., from 64 up to 256).

This design allows deeper networks like ResNet-50 to achieve higher non-linear capacity with lower computational training overhead.

1. Complete Python Implementation Using PyTorch

Below is an end-to-end, modular Python script that implements a modular ResNet BasicBlock, handles automatic spatial downsampling and channel matching, constructs a full ResNet-18 classification backbone, and verifies execution tensor shapes using PyTorch.

Python

import torch
import torch.nn as nn
import torch.nn.functional as F

# Ensure reproducible executions
torch.manual_seed(42)
# -------------------------------------------------------------------
# Step 1: Define the ResNet Basic Block Architecture
# -------------------------------------------------------------------
class BasicBlock(nn.Module):
    expansion = 1
    def __init__(self, in_channels, out_channels, stride=1):
        super(BasicBlock, self).__init__()
        
        # Main Residual Path F(x)
        self.conv1 = nn.Conv2d(
            in_channels, out_channels, kernel_size=3, stride=stride, padding=1, bias=False
        )
        self.bn1 = nn.BatchNorm2d(out_channels)
        
        self.conv2 = nn.Conv2d(
            out_channels, out_channels, kernel_size=3, stride=1, padding=1, bias=False
        )
        self.bn2 = nn.BatchNorm2d(out_channels)
        
        # Shortcut Path (x)
        self.shortcut = nn.Sequential()
        
        # If spatial size shrinks (stride > 1) or channels mismatch, project input x
        if stride != 1 or in_channels != self.expansion * out_channels:
            self.shortcut = nn.Sequential(
                nn.Conv2d(
                    in_channels, self.expansion * out_channels, kernel_size=1, stride=stride, bias=False
                ),
                nn.BatchNorm2d(self.expansion * out_channels)
            )
    def forward(self, x):
        # 1. Compute residual function path F(x)
        residual = F.relu(self.bn1(self.conv1(x)))
        residual = self.bn2(self.conv2(residual))
        
        # 2. Add identity shortcut: F(x) + x
        out = residual + self.shortcut(x)
        
        # 3. Apply final non-linear activation
        out = F.relu(out)
        return out

# -------------------------------------------------------------------
# Step 2: Build the Full ResNet Architecture
# -------------------------------------------------------------------
class ResNet(nn.Module):
    def __init__(self, block, num_blocks, num_classes=10):
        super(ResNet, self).__init__()
        self.in_channels = 64
        # Initial stem layers for image inputs
        self.conv1 = nn.Conv2d(3, 64, kernel_size=3, stride=1, padding=1, bias=False)
        self.bn1 = nn.BatchNorm2d(64)
        
        # Residual Stage Layers
        self.layer1 = self._make_layer(block, 64, num_blocks[0], stride=1)
        self.layer2 = self._make_layer(block, 128, num_blocks[1], stride=2)
        self.layer3 = self._make_layer(block, 256, num_blocks[2], stride=2)
        self.layer4 = self._make_layer(block, 512, num_blocks[3], stride=2)
        
        # Global Average Pooling and Dense Classification Head
        self.avg_pool = nn.AdaptiveAvgPool2d((1, 1))
        self.linear = nn.Linear(512 * block.expansion, num_classes)
    def _make_layer(self, block, out_channels, num_blocks, stride):
        strides = [stride] + [1] * (num_blocks - 1)
        layers = []
        for s in strides:
            layers.append(block(self.in_channels, out_channels, s))
            self.in_channels = out_channels * block.expansion
        return nn.Sequential(*layers)
    def forward(self, x):
        out = F.relu(self.bn1(self.conv1(x)))
        
        out = self.layer1(out)
        out = self.layer2(out)
        out = self.layer3(out)
        out = self.layer4(out)
        
        out = self.avg_pool(out)
        out = torch.flatten(out, 1)
        out = self.linear(out)
        return out

# Helper function to instantiate standard ResNet-18 configuration
def ResNet18(num_classes=10):
    return ResNet(BasicBlock, [2, 2, 2, 2], num_classes=num_classes)

# -------------------------------------------------------------------
# Step 3: Model Verification and Input Forward Test
# -------------------------------------------------------------------
if __name__ == "__main__":
    # Simulate a dummy batch of 2 RGB images (32x32)
    dummy_input = torch.randn(2, 3, 32, 32)
    
    # Instantiate ResNet-18 model
    model = ResNet18(num_classes=10)
    
    # Execute forward pass
    output_logits = model(dummy_input)
    
    print("--- ResNet-18 Execution Summary ---")
    print(f"Input Tensor Shape:  {list(dummy_input.shape)} (Batch, Channels, Height, Width)")
    print(f"Output Tensor Shape: {list(output_logits.shape)} (Batch, Num_Classes)")
    print(f"Total Model Parameters: {sum(p.numel() for p in model.parameters() if p.requires_grad):,}")

Best Practices for Training ResNet Models in Production

To maximize classification accuracy and training efficiency when working with ResNet architectures, apply these proven engineering guidelines:

• Always Use Batch Normalization: ResNet relies heavily on Batch Normalization placed immediately after convolutions and before element-wise additions. It stabilizes layer activations, speeds up training convergence, and acts as an implicit regularizer.
• Initialize Weights Properly: Use Kaiming (He) Normal Initialization (nn.init.kaiming_normal_) for convolutional weight matrices to ensure proper signal variance propagation across deep residual networks.
• Leverage Pre-trained Backbones (Transfer Learning): For custom computer vision projects, avoid training ResNet from scratch. Fine-tune pre-trained torchvision models (e.g., torchvision.models.resnet50(pretrained=True)) to benefit from rich feature representations learned on ImageNet.

Summary and Conclusion

Residual Networks transformed deep learning by demonstrating that identity shortcut connections allow gradient signals to flow directly through hundreds of stacked layers without vanishing. By reformulating optimization targets from direct mappings to residual mappings, ResNet paved the way for modern ultra-deep vision and Transformer backbones.

With this modular PyTorch implementation, you can now construct custom residual blocks, scale network depth confidently, and apply deep vision architectures to complex computer vision tasks.