Sitemap
Ornela Megne

Ornela Megne

Machine Learning Enthusiast

Image Captioning

5 min readJun 6, 2023

--

In the realm of multimodal learning, image captioning stands as a process that involves the generation of meaningful textual descriptions or captions corresponding to given images. Unlike the task of assigning class labels to images (image classification), image captioning combines two distinct modalities, namely images and texts used in computer vision and natural language processing (NLP) respectively. In this blog post, we embark on an exploration of the captivating domain of image captioning, specifically focusing on the utilization of Contrastive Language-Image Pre-training (CLIP)[1].

This blog post can be considered as a synopsis of the article [1] which incorporates some terminologies and concepts outlined in the paper.

As mentioned in the paper [1], the key idea of image captioning is to obtain a wide understanding of both visual and textual data. The whole architecture can be divided into three main modules namely CLIP’s encoding, mapping network, and the pre-trained language model as shown in figure 1.

Press enter or click to view image in full size
Figure 1: Bref overview of the image captioning architecture. Image by the author.

Let’s break down each module for a better understanding.

1. CLIP encoder

Figure 2: CLIP’s architecture. Image by the author.

Developed by OpenAI in 2021 [2], CLIP jointly trains an image encoder and a text encoder to predict the correct image — text pair within a batch. During training, the image and text encoder produce respectively image embeddings (I1 and I2) and text embeddings (T1 and T2) in a way to maximize the cosine similarities between correct image-text pairs (<I1-T1>, and <I2-T2>) while minimizing the cosine similarities for dissimilar pairs (non-diagonal elements), in a contrastive fashion. The image encoder is a computer vision backbone such as ResNet, or Vision Transformer (ViT)[4] and the text encoder is a transformer [5].

Get Ornela Megne’s stories in your inbox

Join Medium for free to get updates from this writer.

The CLIP architecture has been trained on a dataset of 400 million image-text pairs collected from the internet. CLIP has a special ability called “zero-shot image classification”, whereby it can classify images even without specific training on those exact images. This means that CLIP possesses a generalized understanding of different objects or concepts and can apply this knowledge to classify unseen images. To illustrate this, the authors of [2] compare a ResNet-101 model trained specifically on ImageNet dataset to CLIP when both are evaluated on other datasets derived from ImageNet.

Press enter or click to view image in full size
Figure 3: The performance of best zero-shot CLIP model is compared with the performance of ResNet-101. Image from the paper [2].

The authors observed that the performance of ResNet-101, trained specifically on the ImageNet dataset, significantly lags behind CLIP when evaluated on comparable datasets for similar tasks.

2. Language model

To generate the image caption itself, a textual decoder is utilized. In the research study [1], the authors employed a state-of-the-art language model known as Generative Pre-trained Transformer 2 (GPT-2)[3]. GPT-2 is a transformer-based model that has been pre-trained on a large corpus of text data and is capable of generating coherent and contextually relevant textual outputs. By leveraging the power of GPT-2, the researchers were able to generate meaningful captions that describe the content of the given image.

3. Mapping network

The mapping network acts as a bridge between the CLIP encoding and the GPT-2 language model, helping map the visual-textual representation of CLIP to the GPT-2 representation space. The researchers explored two different strategies for this purpose.

  • Firstly, they employed a simple Multi-Layer Perceptron (MLP) approach, where the language model is fine-tuned using the CLIP’s embedding as a guide to orient the text generation process.
  • Alternatively, to mitigate the increase in trainable parameters resulting from fine-tuning the language model, the researchers opted to freeze the language model and utilize the transformer architecture as the mapping network. In this configuration, only the mapping network is trained.

The experiments revealed that the second approach, utilizing the transformer-based mapping network while freezing the language model, achieved superior results compared to the MLP-based approach in certain scenarios.

Simple mathematical description of the task

Problem statement: Given a dataset of paired images and captions/texts {x^i, c^i}, where

is a set of tokens of a fixed length l, the objective is to learn the generation of a meaningful caption for an unseen input image.
The training objective is defined as maximizing the likelihood of generating the target caption c^i given the input image x^i and a model parameterized by θ:

Since GPT-2 is an autoregressive language model, it generates text one token at a time based on the previously generated tokens. Therefore, the training objective can be reformulated as maximizing the conditional probability of generating the current token c_i given the previously generated tokens and the input image x_i:

For more details consult the paper [1].

Some results

Below are several inference cases that were tested within a web-based environment. The code of the paper is available here.

Output: silhouette of a girl on the beach at sunset.
Press enter or click to view image in full size
Output: ice cream in a glass bowl.

Conclusion

We have explained the concept of image captioning using the Contrastive Learning Image Pre-training (CLIP) architecture. By harnessing the capabilities of CLIP and GPT-2, multimodal learning can extract informative insights from images. The field of image captioning holds immense potential for real-life applications. For instance, real-time provision of textual descriptions during video calls and multimedia presentations for individuals with hearing impairments would enable them appreciate and understand the content.

Thank you for taking the time to read this article! Hope that it has been both inspiring and insightful.

References

[1]: ClipCap: CLIP Prefix for Image Captioning

[2]: Learning Transferable Visual Models From Natural Language Supervision

[3]: Improving Language Understanding by Generative Pre-Training

[4]: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

[5]: Attention Is All You Need

BECOME a WRITER at MLearning.ai // new career // Super Cheap AI.

--