The “Social Butterfly” of AI
How DenseNet Reimagined Convolution Neural Networks
One of the foundations of image processing in the modern world has been the famed Convolutional Neural Networks (CNNs) that were the outcome of the exemplary work done by David Hubel and Torsten Wiesel [2].
These networks then became the building blocks for various other networks like AlexNet, ResNet, VGGNet, etc.
However, in the field of computer vision, one principle dominated the conversation: “deeper means better”. This implies that if we wanted better performance on computer vision tasks (recognition, segmentation, etc.), then stacking more CNN networks is the key.
With computing power matching the pace of this growth, this idea was followed quite well till ResNets and Highway Nets broke the 100-layer barrier. Although these architectural towers grew taller, they started to wobble. Models were now just large, bloated resource-hogs that took up a lot of computing power for training, which in itself was a nightmare.
This is where the idea of Densely Connected Convolutional Networks, a.k.a DenseNets, came in. Instead of building a skyscraper where information only moves from one floor to the floor above, DenseNets built an open-floor office where everyone talks to everyone.
Basically, the larger and “denser” connections allowed for faster and smoother information flow, which not only improved training statistics for the models but also solved some of the critical problems that were plaguing the previous CNN-based models.
NOTE: The work discussed here is based on the work done by Huang. et. al. [1] and the images have also been taken from their work. The authors of this blog claim no novelty in the work presented.
Inside DenseNet: Architecture and Intuition
The basic intuition behind DenseNet can be understood using the example of a classroom. Imagine that every student in this classroom only talks to the person sitting next to them. In this case, information flows, but it is slow and often distorted. Now imagine the same classroom, but this time, every student can directly talk to everyone else. The information and ideas in this case spread faster with minimal distortion. This is the intuition behind DenseNet: a simple, yet effective idea.
Let us define this technically. Suppose the feature map of the initial image is represented as ‘x0’, and each layer l implements a non-linear transformation ‘Hl’.
Then, in traditional CNN frameworks, we observe that the feature map learned at layer l is dependent solely on the output of the previous layer.
DenseNets, on the other hand, make use of every single feature map till the ‘l − 1-th’ layer to get the desired output.
Why is DenseNet a Game-Changer?
The only difference between standard CNN architectures and DenseNet is that DenseNet uses the features that all the previous layers learn as well, which has been highlighted in Figure 1, which shows a 5-layer DenseNet architecture.
Even though this update in architecture seems simple compared to the regular CNN-based architectures we are so used to seeing around us, this change actually has some interesting and elegant effects.
Model compactness & Parameter efficiency
As a direct consequence of the input concatenation, the feature maps learned by any of the DenseNet layers can be accessed by all subsequent layers, which leads to much smaller (compact) models. This also implies that DenseNets can achieve the same functionality as regular CNN architectures with considerably fewer trainable parameters.
One such trend is shown in Figure 2, where DenseNet-BC consistently uses three times fewer parameters to achieve the same test loss (accuracy) when compared to a pre-trained ResNet model (which is state-of-the-art (SOTA) at the time when DenseNets were proposed).
Implicit Deep-Supervision
It was observed that a DenseNet architecture with 190 layers and a growth rate (number of new feature maps produced at each layer) of 40 vastly outperformed most of the popular, SOTA architectures when tested in CIFAR-10, CIFAR-100, and SVHN datasets.
This increase in accuracy was a pleasant surprise and is attributed to a phenomenon known as “deep supervision.” This is when individual layers receive additional supervision from the loss function through the shorter connections — a single classifier on top of the network provides direct supervision to all layers through at most two or three transition layers.
Get Yash Sarang’s stories in your inbox
Join Medium for free to get updates from this writer.
This sort of deep supervision has been observed to improve accuracy in previous models as well, but is more inherent in the DenseNet architecture.
Finding the vanishing gradients
One problem with deeper networks is that the gradients of the initial layers become weaker the deeper we go, and at some point, become very close to zero, making training tougher and longer. This is the classic vanishing gradient problem and often plagues most deep CNN architectures.
However, thanks to the direct connections that DenseNet provides, each layer’s gradient flow is easier across subsequent layers, and the training is more efficient.
Feature Reuse
By design, DenseNets allow layers access to feature maps from all of their preceding layers and effectively “reuse” the features learned previously. Looking at Figure 3 highlights some interesting details in this aspect. All layers spread their weights over many inputs within the same block. This indicates that features extracted by very early layers are, indeed, directly used by deep layers throughout the same dense block.
Additionally, the layers within the second and third dense block consistently assign the least weight to the outputs of the transition layer, indicating that the transition layer outputs many redundant features.
Is DenseNet not bulky?
This was the question most people (including myself) had when we first read about this idea. At each layer, the architecture stores the feature maps of the previous layers as a tensor and pushes them forward. Over 100s of layers, this would make the model very bulky. So why does it not do that?
The simple answer is another basic aspect of CNN architectures, the down-sampling/pooling layers. To facilitate down-sampling, the DenseNet architecture is divided into multiple densely connected dense blocks (see Figure 4) with transition layers in between that do 3 convolution and pooling. This allows for the DenseNet models to be manageable, despite storing feature information of all the layers before the current one.
Are DenseNets still valid, though?
Even though we have seen that DenseNets are simple enough and also have important advantages, there is still a lingering question pertaining to their longevity: are these models still SOTA?. Would someone still use DenseNets in the big 2026? Surprisingly, an “ancient” idea (in AI terms) like DenseNet is still a staple part of a CV researcher’s toolkit, because it is comparable to some of the SOTA architectures today
DenseNet vs. EfficientNet
EfficientNet, which was introduced by Google in 2019, focuses on scaling so as to perfectly balance a network’s depth, width, and resolution. On the other hand, DenseNet focuses more on connectivity. Thus, even though EfficientNet is often faster and more accurate on standard benchmarks like ImageNet, DenseNet is often preferred in medical imaging because its dense connections are better at capturing the tiny, fine-grained details.
DenseNet vs. Vision Transformers (ViT)
Transformers were a revolutionary step in AI history that took the conversation from traditional convolutions to focusing the attention on well, uhm, “Attention” (badum-tishh). The transformer models could now take the whole image at once and not focus on it piece by piece. However, one thing to note is that Transformers require massive amounts of data to beat a CNN. If you don’t have a million images, DenseNet is often the more reliable, “plug-and-play” choice.
DenseNet vs. ConvNeXt
ConvNeXt is essentially a “modernized” version of a standard CNN that borrows ideas from Transformers. Basically, it is designed to be as “global” as a Transformer while keeping the simplicity of a CNN. Although ConvNeXt generally outperforms DenseNet in raw speed and accuracy on massive datasets, for small-to-medium datasets, DenseNet’s “feature reuse” means it can often reach high accuracy with far less training data than a massive ConvNeXt model.
Conclusion
The long and short of this article is that a simple idea can have quite interesting and useful advantages, which is highlighted in the case of DenseNets. By introducing direct connections between any two layers with the same feature map size, the DenseNet architecture scales naturally to hundreds of layers without optimization difficulties.
Additionally, it also resolves vanishing gradient issues, in turn outperforming some of the most popular image processing CNN models. The direct connections also naturally integrate the properties of identity mappings, deep supervision, and diversified depth.
Last but not least, they are still part of the CV toolkit and are often used when working with small-to-medium datasets.
NOTE: For implementation details, please refer to this GitHub repository.
References
[1] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017.
[2] David Hubel and Torsten Wiesel. David hubel and torsten wiesel. Neuron, 75(2):182–184, 2012.
