Data Science & Machine Learning — Article 65

GoogleNet/Inception: A Comprehensive Guide

Explore detailed GoogleNet/Inception

Data Science & Machine Learning — Article 65

Introduction

Articles 63 through 65 measured the same problem three times. AlexNet put 96.17 percent of its parameters in dense layers, VGG-16 put 89.36 percent, and VGG-19 added three convolutional layers without touching a head that was already 86 percent of the model.

GoogLeNet, published in the same year as VGG, deletes that head entirely.

The result is 6, 998, 552 parameters against VGG-16’s 138, 357, 544: 19.8× smaller, and better on the benchmark.

Two mechanisms account for it. Global average pooling replaces the flatten-then-dense head, and on the same feature map a VGG-style first dense layer would cost 205, 524, 992 against GoogLeNet’s 1, 025, 000, a factor of 201. And 1 × 1 convolutions placed before the expensive branches cut the inception modules by 4.13×.

There is also a detail that is rarely stated: the two auxiliary classifiers used during training are 91.2 percent of the size of the deployed network, and they are thrown away.

Table of Contents

• Introduction
• Table of Contents
• What is GoogLeNet
• Core Components of GoogLeNet
• Mathematical Foundation
• Types of Inception
• Evaluation Metrics
• Selecting Evaluation Metrics
• Assumptions and Practical Considerations

What is GoogLeNet

Definition

GoogLeNet is a 22-layer convolutional network built from nine Inception modules, each applying 1 × 1, 3 × 3, and 5 × 5 convolutions and a pooling operation in parallel and concatenating their outputs.

With 1 × 1 convolutions inserted to reduce channel count before the expensive branches, and with global average pooling in place of a dense head.

Why it mattered

• It broke the assumption that a network is a chain. An Inception module is a small parallel graph, and the network chooses among kernel sizes rather than committing to one.
• It removed the dense head, which Articles 62 through 65 measured as 97.1, 96.2, 89.4, and 86.1 percent of four consecutive architectures.

The module

The four branches see the same input and their outputs are stacked, so the module offers several receptive field sizes at once and lets the training decide how much of each to use.

Against its contemporaries

• GoogLeNet’s trunk is 2.6× larger than AlexNet’s and its total is 8.7× smaller, because the head went from 58.6 million to 1.0 million.

Core Components of GoogLeNet

The Inception module

• Four parallel branches, concatenated. Section 5.1 measures each branch.

The 1 × 1 reduction

• Placed before the 3 × 3 and 5 × 5 convolutions to cut the channel count first.
• Article 60 established that a 1 × 1 convolution is a per-pixel dense layer across channels and is the cheapest way to change channel count; here it is used as a dimension bottleneck.

The pool projection

• The pooling branch is followed by a 1 × 1 convolution, because pooling preserves channel count and the concatenation would otherwise be dominated by unmodified input channels.

Global average pooling

• Reduces the final 7×7×1024 map to a 1024-vector by averaging each channel. No parameters, and it is what removes the dense head.

Auxiliary classifiers

• Two additional classification heads attached partway through the network during training, discarded at inference.

The stem

• A conventional 7 × 7 stride-2 convolution followed by pooling and two more convolutions, before any Inception module. At 124, 416 parameters it is 1.8 percent of the network.

Mathematical Foundation

What the 1 × 1 reductions save

Take Inception (3a): 192 input channels at 28×28, producing 64 + 128 + 32 + 32 = 256 output channels.

With reductions, as built:

Without reductions, applying 3×3 and 5×5 directly to all 192 channels:

A factor of 2.40 on this module. The 5×5 branch is where it bites hardest:

a factor of 9.65. The reduction is worthwhile precisely because k² multiplies the channel product, so shrinking channels before a large kernel saves more than shrinking them before a small one.

Every module

The reductions save 4.13× overall, and the saving grows with depth, from 2.40 at module 3a to 5.15 at 5a.

That follows from the formula above: the ratio approaches cᵢₙ/r, and cᵢₙ rises from 192 to 832 while the reduction targets stay small.

Without the 1 × 1 layers, GoogLeNet’s modules alone would be 24.2 million parameters and the network would lose most of its size advantage.

Assembling the network

The paper describes roughly 6.8 million parameters; this accounting gives 6, 998, 552, within 2.9 percent.

The residual difference is bookkeeping around bias terms and the exact stem definition, and it is small enough to leave alone rather than to force.

Note the stem’s FLOPs: 477.7 MFLOPs for 124,416 parameters, because it operates at 112×112 and 56×56.

It is 1.8 percent of the parameters and 30 percent of the convolutional compute, which is Article 63’s inversion appearing again in miniature.

The head is the whole difference

Global average pooling turns the final 7×7×1024 map into a 1024-vector with no parameters:

Applying a VGG-style head to the same map instead:

Get Arko Sengupta’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

A single VGG-style layer on this feature map would be 29 times larger than the entire GoogLeNet. The flatten multiplies the cost by the spatial area, here 49, and then by the 4096 hidden width.

Writing the two heads side by side makes the source of the ratio explicit:

so the ratio is HW · n(hidden)/n(class) = 49 × 4096/1000 ≈ 201. That is the whole size story. GoogLeNet’s trunk at 5, 973, 552 parameters is 2.6× AlexNet’s, so it is not a small network in the part that does the work.

It is small because it does not carry a 58 or 124 million parameter classifier.

The auxiliary classifiers are nearly the size of the network

Two additional heads are attached at modules 4a and 4d during training. Each consists of a 5 × 5 average pool to 4 × 4, a 1 × 1 convolution to 128 channels, a dense layer to 1024, and a dense layer to 1000:

That is 91.2 percent of the deployed network. The trained model is 13,378,280 parameters and the shipped model is 6,998,552; nearly half of what is optimized is discarded.

The reason they are so large is the same one that made VGG’s head large: each contains a dense layer over a flattened 4×4×128 map, costing 4·4·128·1024 + 1024 = 2, 098, 176 parameters.

The architecture that removed the dense head from its main path attached two of them to its side paths.

Their stated purpose was to inject gradient at intermediate depths, on the reasoning that a 22-layer network’s early layers receive an attenuated signal.

Article 57 quantified the mechanism: with ReLU the per-layer derivative is 1 where active, but the composition of 22 weight matrices can still shrink the signal geometrically, which Article 59 measured across twelve layers as capable of 10⁻⁸ decay.

Residual connections solve the same problem by giving every layer a gradient path of length 1, at zero parameter cost.

The auxiliary classifiers cost 6.4 million parameters and were abandoned. This is a case where two solutions to one problem can be compared directly, and one of them is free.

Where the parameters sit inside a module

Summing across the nine modules by branch:

The 3 × 3 path is 62.66 percent of the Inception modules. VGG reached the conclusion that 3 × 3 is the right kernel by using nothing else.

GoogLeNet reached it by offering alternatives and having the parameter budget land there anyway.

The 5 × 5 path is only 12.64 percent, and it survives at all because the 1 × 1 reduction makes it affordable.

Without the reduction it would be roughly 9.65× larger on module 3a and correspondingly more elsewhere, which would make it the dominant branch and almost certainly not worth keeping.

Later Inception versions replaced it with two stacked 3×3 layers, which is the same substitution VGG made.

Types of Inception

Inception v2

• Adds batch normalization throughout and replaces each 5 × 5 with two stacked 3 × 3 layers, which Article 59 measured as 1.39× cheaper for the same receptive field plus an extra nonlinearity.

Inception v3

• Factorizes n× n convolutions into 1× n followed by n× 1.
• For n=7 this is 14/49 of the parameters, a 3.5× saving, and it works because a separable kernel is a rank-1 approximation of a full one.

Inception-ResNet

• Adds residual connections to Inception modules, which is an admission that the two ideas are complementary rather than competing.

Xception

• Takes the reasoning to its limit: if 1×1 handles channels and k× k handles space, separate them completely.
• This is depthwise separable convolution, which Article 60 measured as saving a factor approaching k².

Evaluation Metrics

Head share of parameters

• 14.65 percent here against VGG-16’s 89.36. The single number that explains the 19.8× size difference.

Reduction ratio per module

• Naive against reduced, 2.40 to 5.15 across the network. It measures how much the 1×1 layers are earning.

Branch share within a module

• Section 5.6’s distribution. It reveals which kernel size the training actually funds.

Training-only parameters

• 6, 379, 728 here, 91.2 percent of the deployed model. Worth reporting separately whenever auxiliary heads or teacher networks are involved.

Parameters against FLOPs for the stem

• 124,416 parameters and 477.7 MFLOPs. Early layers at high resolution are always in this regime.

Concatenated channel count

• The module output width, which sets the next module’s input cost. It is the quantity that grows through the network, 256 → 480 → 832.

Selecting Evaluation Metrics

Selection rules

• Report the head share before anything else when comparing architectures. It accounts for GoogLeNet being 19.8× smaller than VGG-16 despite a larger trunk.
• State deployed and trained parameter counts separately when auxiliary heads exist. The gap here is nearly a factor of two.
• Check the reduction ratio at both the shallowest and deepest module. It nearly doubled from 2.40 to 5.15 across this network.
• Do not assume a parallel branch earns its place. The 5×5 path is 12.64 percent of parameters and was removed entirely in the next version.

Assumptions and Practical Considerations

Branch outputs have matching spatial dimensions

• Concatenation requires it. Every branch uses padding chosen so that 28×28 in gives 28×28 out, including the pooling branch, which uses stride 1.

The pooling branch is projected

• Pooling preserves channel count, so without a 1 × 1 projection the concatenation would carry all cᵢₙ input channels unchanged into the output.

Reduction widths are hyperparameters per module

• The paper’s Table 1 specifies them individually, and they are not a fixed fraction.
• Module 3a reduces 192 → 16 before its 5 × 5 while 4e reduces 528 → 32.

Global average pooling replaces the flatten

• This is the size decision. Substituting a flatten reinstates a 200-million-parameter layer, per Section 5.4.

Auxiliary classifiers are training-only

• They must be removed for inference, and their loss is weighted, typically at 0.3.
• Leaving them in changes both the parameter count and the objective.

The concatenated width grows and must be tracked

• 256, then 480, then 832. Each module’s input cost scales with it, and the 1 × 1 reductions are what keep that growth affordable.

Depth is 22 layers by the paper’s counting

• Only layers with parameters are counted, and parallel branches within a module count once.
• Different counting conventions give different numbers for the same network.