Vision Transformers Part1: Why Replace Convolution with Attention

Understanding vision transformers foundation, architectural differences and performance comparisons with CNNs. Some of the graphs in this article are AI generated.

Convolutional Neural Networks (CNNs) used to be the dominant model family for different computer vision tasks such as image classification, object detection, segmentation, etc. However, with the advancements of transformers and modern CNNs their usability has changed and is now highly application dependant. The specific application, software and hardware constraints are a few factors that need to be considered when deciding which model to use.

The first models that really challenged CNNs dominance were the Vision Transformers (ViTs) that surpassed the ResNet model on public benchmark datasets. The structural differences along with the large scale pretraining conditions, newer training and augmentation strategies contributed to the superior performance of these models. I want to investigate the changes that follow when we use attention mechanism instead of convolutions.

Understand ViT Architecture

Standard convolution applies filters over spatial neighbourhoods in the image. These filters are reused across the image to produce feature maps. They restrict the feature extraction to…