A Feature Pyramid Network (FPN) is a neural network architecture designed to enhance computer vision tasks like object detection and segmentation, especially in scenarios with objects of varying sizes. FPNs solve a fundamental limitation of traditional CNNs: while deeper layers capture more semantic (high-level) features, they lose spatial resolution, making it harder to detect small objects. FPNs are specifically engineered to aggregate both high level semantic and low level detailed spatial information at multiple scales, producing robust multi scale feature maps for downstream tasks.
Why Do We Need FPN?
Objects in images can appear in many different sizes. For example, a cat can look huge in one photo and tiny in another. Traditional convolutional networks often struggle with this, because they use a single scale feature map, missing small or large objects.
FPN solves this by:
- Building a feature pyramid a set of feature maps at multiple scales.
- Combining both high-level/contextual and fine/local information.
Types of Feature Image Pyramid
- Feature Image Pyramid : Builds an image pyramid by resizing input images to multiple scales, then extracts features at each scale separately.
- Single Feature Map : Processes the image at a single scale, extracting features only once, without explicit multi-scale fusion.
- Pyramidal Feature Hierarchy : Leverages the inherent hierarchical outputs of deep CNNs, where features are taken from backbone layers at different depths to provide multi-scale representations.
- Feature Pyramid Network (FPN) : Constructs a multi-level feature pyramid by merging bottom-up features (from the backbone) with top-down pathways and lateral connections, enhancing high-level semantic information at all scales.
How Does FPN Work?
The steps are as follows:
- Backbone Network : A standard CNN (e.g., ResNet) processes the input image, producing feature maps at several resolutions/layers (capturing details from fine to coarse).
- Top-Down Pathway (Upsampling) : The high-level (semantic, low-resolution) features are upsampled (enlarged) step by step.
- Lateral Connections : At each level, the upsampled features are combined (usually via addition) with features from the corresponding level in the backbone. This allows FPN to merge fine details with semantic context.
- Output Pyramid : The result is a series of feature maps (the “pyramid”) each at a different scale, each rich in both context and detail.
Visual Example: Suppose you're detecting a cat in an image
- The FPN’s highest level identifies the presence of a cat (body shape).
- The mid-level hones in on the head and the lowest level finds the ears and whiskers.
- Combining all these, the model can confidently detect the cat, big or small.
Technical Architecture (with Example)
- Bottom-up pathway: Standard deep network produces feature maps (C2,C3,C4,C5).
- Top-down pathway: Starts from high-level feature maps, progressively upsamples (e.g., using nearest-neighbor interpolation).
- Lateral connections: 1x1 convolutions align the channels before addition with upsampled maps.
- Output Pyramid: Final multi-scale feature maps (P2,P3,P4,P5) used for detection or segmentation.
Mathematical representation:
P_i = \text{Upsample}(P_{i+1}) + \text{Conv}_{1 \times 1}(C_i)
Example Application: Object Detection with FPN
In a modern object detection pipeline like Faster R-CNN with FPN:
1. The model uses each level of the FPN output to scan for objects of certain sizes.
- Small anchors for high-resolution feature maps.
- Large anchors for low-resolution, semantic-rich maps.
2. This powers state-of-the-art detection results on major datasets like COCO.
In Practice
FPNs are widely used not only for object detection, but also for:
- Instance segmentation (Mask R-CNN)
- Semantic segmentation
- Scene understanding : They have become a foundational component for modern computer vision architectures due to their effectiveness and simplicity.
Advantages of FPN
- Better Detection of Small Objects: Uses high-resolution features to spot small items.
- Multi-Scale Robustness: Handles objects large and small in the same image.
- Efficiency: Reuses backbone features, adding little computational cost.
- Plug-and-play: Compatible with popular detectors (Faster R-CNN, RetinaNet, Mask R-CNN).