Fast RCNN

Fast R-CNN was introduced by Ross Girshick in 2015 as an improvement over R-CNN. It significantly increased the speed of object detection while maintaining high accuracy.

How Fast R-CNN Works (Step by Step)

Step 1: Image Processing with a CNN Backbone

• The entire input image is fed into a Convolutional Neural Network (CNN) (such as VGG16 or ResNet).
• This CNN extracts feature maps from the whole image instead of processing each region separately.

Step 2: Region Proposal via Selective Search

• Region proposals are generated using Selective Search (same as R-CNN).
• However, unlike R-CNN, the feature extraction is done only once for the whole image, not for each proposal.

So now, we have the feature maps obtained from CNN and region proposals obtained by using region proposal algorithm like Selective Search.

Now we need to map these region proposals on to the feature maps.

But here the size of the feature map is not the same as that of the original image. Also size of each region proposal is going to be different

Question

We have 100 feature maps (let’s say each feature map is of size 14x14, for example). We have 2000 region proposals (boxes) on the original image. How are these 2000 region proposals mapped on the 100 feature maps?

Example:

Original image size: 224x224.

Feature maps size: 14x14 (100 feature maps).

Region proposal on original image: A box at (50, 50, 100, 100).

1. Mapping:

• The region proposal is scaled down to fit the feature maps. Since the feature maps are 16 times smaller, the box becomes (50/16, 50/16, 100/16, 100/16) ≈ (3, 3, 6, 6) on the feature maps.

2. Extracting Features:

• A 3x3 region (from 3,3 to 6,6) is extracted from each of the 100 feature maps. So, we get 100 small 3x3 feature grids.

3. RoI Pooling:

• These 100 feature grids are resized to a fixed size (e.g., 7x7) using RoI Pooling. Now, we have a 7x7x100 tensor for this region proposal.

4. Repeat for All Proposals:

• This process is repeated for all 2000 region proposals. Each proposal gets its own 7x7x100 tensor.

For example:

• Suppose a region proposal (bounding box) in the original image has coordinates (x₁, y₁, x₂, y₂).
• The feature map is downsampled by a factor of 16 (e.g., from an 800×800 image to a 50×50 feature map).
• The new mapped coordinates on the feature map become: x′=x/16, y′=y/16​
• Now, this rescaled bounding box correctly corresponds to a region on the feature map instead of the original image.

ROI Pooling

The ROI pooling layer is a special case of the spatial pyramid pooling (SPP) layer with just one pyramid level

Region of Interest (ROI) Pooling is a technique used to convert feature maps of different sizes into a fixed-size representation. This allows each region proposal to have the same feature dimension before passing it to the fully connected layers for classification and bounding box regression.

Why is ROI Pooling Needed?

• Region proposals vary in size → Feature maps corresponding to different regions will have different dimensions.
• Fully connected layers require fixed-size input → We need to normalize all regions into a fixed shape (e.g., 7×7 or 14×14).
• Prevents computational overhead → It avoids the need to resize feature maps repeatedly.

Case 1: When the input size for the ROI pooling is greater than the required output size

For this case, we will take the input size as (4x4) and the output size as (2x2)

Step-by-Step Process of ROI Pooling

Let’s go through the process with an example.

1. Input: Feature Map and Region Proposal

• Assume the input image is 800×800.
• After passing through a CNN (e.g., VGG16), the feature map is 50×50.
• A region proposal (bounding box) is given as (10, 20, 14, 24) on the feature map (after mapping from the original image). Here (10,20) is the coordinate of the top left of the bbox and (14, 24) is the co-oridnate of the bottom right of the bbox
• We want to resize this region to a fixed 2×2 size (for simplicity; in real models, it’s usually 7×7 or 14×14).

2. Divide the Region into Equal-Sized Sections

• The region (10, 20) to (14, 24) has a height of 4 and a width of 4.
• We divide this 4×4 region into a 2×2 grid (since our target is 2×2).

3. Max-Pooling in Each Section

We apply max-pooling in each section, which means:

• Take the maximum value from all pixels in the 2x2 area.
• This ensures that only the most dominant feature is retained.

4. Output: Fixed-Size Feature Map

After applying max pooling, we obtain a 2×2 matrix

In real cases, we don’t resize to 2×2, but to 7×7 or 14×14, depending on the model. The process remains the same

ROI pooling done on 4x4 image to get a 2x2 pooling.

Step 4: Fully Connected Layers

• The fixed-size feature vectors from ROI pooling are passed through fully connected (FC) layers, which generate final predictions.

Case 2: When the Input Size for ROI Pooling is Smaller than the Required Output Size

In this case, we will consider an input size of 3x3 for the ROI pooling and a required output size of 5x5. This scenario typically involves upsampling the input feature map to match the desired output size.

So here, the first thing which we will do is make the input size same as the output size by either using techniques called nearest neighbor interpolation or bilinear interpolation.

For simplicity, we will use nearest neighbor interpolation in this example

Nearest neighbor interpolation means that each pixel in the output will take the value of the nearest pixel from the input feature map

Get Abhishek Jain’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Note: Once the input size is same as the output size, ROI pooling wont be done, as it will yield the same result (Reason : ROI pooling done on n x n image to get a n x n image as output will produce the same input image as the output image)

Next Step

Flattening layer and Fully Connected (FC) layer

• After ROI pooling, each fixed-size feature map (e.g., 7×7×C) is flattened into a 1D feature vector.
• This vector is passed through fully connected layers (like in standard CNNs). These FC layers reduce the dimensionality and extract high-level features specific to the region proposal.

Compute Classification and Bounding Box Regression Outputs

• The network branches into two heads:
• Classification Head → Outputs class probabilities using Softmax
• Bounding Box Regression Head → Outputs bbox adjustments using Linear Regression

During training

Classification network

Labeling the Proposals

• Positive and Negative Samples: Similar to R-CNN, region proposals are categorized into positive and negative samples based on their overlap with ground truth bounding boxes:
• Positive Samples: Region proposals with an Intersection over Union (IoU) greater than a certain threshold (e.g., IoU > 0.5) are labeled as positive samples.
• Negative Samples: Region proposals with IoU less than the threshold are labeled as negative samples

Training Data Preparation

• Balanced Dataset: A balanced dataset is created by selecting a fixed number of positive and negative samples. For example, Fast R-CNN might use 2000 region proposals per image, with a certain proportion being positive and the rest negative.
• Feature Vectors and Labels: The fixed-size feature vectors obtained from the RoI Pooling layer are paired with their corresponding labels (class labels for positive samples and a background label for negative samples)

Training the Classifier

• Classifier Choice: A softmax classifier is typically used for multi-class classification. The softmax function outputs a probability distribution over the classes for each region proposal.
• Loss Function: The loss function used for training is typically a combination of:
• Softmax Loss: This loss measures the classification error for the predicted class probabilities.
• Bounding Box Regression Loss: If bounding box regression is also being trained simultaneously, a smooth L1 loss (or another suitable loss) is used to refine the bounding box coordinates.
• Optimization: The optimization algorithm (e.g., stochastic gradient descent) adjusts the parameters of the CNN and the classifier to minimize the combined loss function

Softmax Classifier in Fast R-CNN

1. Definition of Softmax Classifier:

• In Fast R-CNN, the softmax classifier is used as the final layer of the network to perform multi-class classification. The softmax function converts the raw output scores (logits) from the preceding layer into probabilities that sum to 1. This is particularly useful for multi-class classification tasks.

2. Output Nodes:

• If there are ( n ) object classes in the dataset, the softmax classifier will have ( n + 1 ) output nodes:
• ( n ) nodes for the object classes (e.g., cat, dog, car, etc.).
• 1 additional node for the background class (indicating that the region proposal does not contain any object).
• Therefore, the softmax function will output ( n + 1 ) probabilities, where each probability corresponds to the likelihood that the region proposal belongs to a specific class or the background.

In Fast RCNN there is only one network that handles both classification and bbox regressionHere’s a breakdown:

1.Shared Convolutional Feature Map:

The entire image is passed through a CNN to generate a convolutional feature map.

2. RoI Pooling:

Region proposals are projected onto the feature map, and RoI pooling extracts fixed-size feature vectors for each proposal.

3. Fully Connected Layers:

These feature vectors are then fed into a series of fully connected layers.

4. Branching at the End:

At the end of the fully connected layers, the network branches into two output layers:

5. Classification Branch:

This branch uses a softmax layer to output a probability distribution over the object classes (plus a background class).

6. Bounding Box Regression Branch:

This branch uses linear regression to output the bounding box refinements (dx, dy, dw, dh) for each class.

Explanation with example

1. During training

1. Input Image and Region Proposals

• Input: An image (e.g., 224x224x3).
• Region Proposals: A set of region proposals (e.g., 2000 proposals) generated by a region proposal method (e.g., Selective Search).

2. Feature Extraction (Backbone CNN)

• The input image is passed through a backbone CNN (e.g., VGG16) to extract feature maps.
• Input: Image (224x224x3).
• Output: Feature maps (e.g., 14x14x512 for VGG16).

3. RoI Pooling

• Each region proposal is mapped onto the feature maps, and RoI Pooling is applied to extract a fixed-size feature map for each proposal.
• Input:
• Feature maps (14x14x512).
• Region proposal coordinates (e.g., (x1, y1, x2, y2)).
• Output: Fixed-size feature map (e.g., 7x7x512) for each region proposal.

4. Flattening

• The fixed-size feature map (7x7x512) is flattened into a 1D vector.
• Input: 7x7x512 feature map.
• Output: Flattened vector (e.g., 25088-dimensional).

5. Fully Connected (FC) Layers

• The flattened vector is passed through one or more FC layers to produce a high-level feature vector.
• Input: Flattened vector (25088-dimensional).
• Output: High-level feature vector (e.g., 4096-dimensional).

6. Classification Head

• The high-level feature vector is passed through the classification head, which consists of an FC layer followed by a softmax activation.
• Input: High-level feature vector (4096-dimensional).
• Output: Class probabilities (e.g., C+1 probabilities, where C is the number of classes and +1 is for the background class).

7. Bounding Box Regression Head

• The same high-level feature vector is passed through the bounding box regression head, which consists of an FC layer.
• Input: High-level feature vector (4096-dimensional).
• Output: Bounding box adjustments (e.g., 4×(C+1) values, where 4 corresponds to Δx,Δy,Δw,Δh for each class).

Training Time:

1. Ground Truth Labels

• Each region proposal is assigned a ground truth label:
• Positive Sample: If the region proposal overlaps significantly with a ground truth box (IoU > 0.5), it is labeled with the object’s class.
• Negative Sample: If the region proposal does not overlap significantly with any ground truth box (IoU < 0.5), it is labeled as background.

2. Multi-Task Loss

• The network is trained using a multi-task loss function that combines:

1. Classification Loss: Measures how well the network predicts the correct class for each region proposal (e.g., cross-entropy loss).
2. Bounding Box Regression Loss: Measures how well the network predicts the adjustments to the region proposal to match the ground truth bounding box (e.g., smooth L1 loss).

• The loss function ensures that the network learns to:
• Correctly classify region proposals that contain objects.
• Adjust the bounding boxes to better localize the objects.

3. Backpropagation

• The network updates its weights based on the combined loss using backpropagation.

Testing Time (Inference):

1. Forward Pass

• The input image is passed through the network to produce:
• Class probabilities for each region proposal.
• Bounding box adjustments for each region proposal.

2. Post-Processing

• Non-Maximum Suppression (NMS): Removes overlapping boxes to keep only the most confident predictions.
• Final Output: The class labels and refined bounding boxes for the detected objects.