In this story, we try to build a simple framework to think about the seen classes, the unseen classes and the unknown classes while dealing with Zero-Shot Learning problems.

On a weekend, pre-covid, you go to a birdwatching park. You are a novice bird watcher and hence you ask a friend to help you recognize the birds that you might encounter.

The Seen

You just know 1 type of bird that you have seen around your house during your childhood: the House Sparrow

House Sparrow is ‘The Seen’ bird type.

How you recognize this bird is:

1. You capture its image when you see it.
2. Process it through the vision model of your brain.
3. The model will then generate a semantic vector after processing the image.
4. Then you will match this generated semantic vector with your semantic vector of a House Sparrow which has been trained by multiple instances of the bird’s image throughout your childhood.
5. If the similarity of these vectors is higher than a threshold, you are confident that the bird you are looking at is a House Sparrow.

The Unseen

Your friend knows your lack of knowledge of birds, hence they give you a bird’s description to help you out during your birdwatching picnic.

It looks like a sparrow but has a yellow beak. It looks like it is wearing a white crown. It’s a White-crowned sparrow.

On your picnic, you witness a lot of birds, a few Home Sparrows too, but you see a couple that looked like sparrows but weren't Home Sparrows (lower than your similarity threshold).

Do you recognise any of these? You recognise the white-crowned sparrow instantly!

White-crowned Sparrow is ‘The Unseen’ bird type.

How you recognise the bird is:

1. You pass the text description of the bird through your multi-modal model in your brain and generate a semantic vector of the description.
2. You take in the new image of the bird, pass it through the same multi-modal model and generate a semantic vector of the image.
3. You will calculate the similarity of these semantic vectors.
4. If the similarity of these vectors is higher than a threshold, you are confident that the bird you are watching is a White-crowned Sparrow.

The Unknown

After impressively recognizing 2 types of sparrows, you are thrilled and can’t wait to go back and gloat about your birdwatching story to your expert friend. But as you are about to leave, you witness a very peculiar type of bird. You have never seen such a bird before. It’s not just different from the house sparrow or the white-crowned sparrow, but it does not resemble a sparrow in any way.

So you go back to your friend and describe the bird that you had seen. In no way can you associate this bird with a name, but you sure can describe it.

Red-headed Woodpecker is ‘The Unknown’ bird type.

How you describe the bird is:

1. You look at the bird, pass it through the multimodal model in your brain and then it generates a semantic vector.
2. Using this semantic vector, you convert your experience to describe it into words.
3. You say, “It is a tall bird with long pointy beak, red-coloured head, black eyes, white belly and half black, half white wing.”

Obviously listening to this description your expert friend can recognize this as a Red-headed Woodpecker with good certainty.

Comparing with ZSL-models

Of course, the other 2 sparrows (Brewer sparrow, Black-throated sparrow) are also The Unknown type, but in that scenario, the semantic vector generated can be used to find the nearest neighbours in your vector space, and that will give you the closest neighbours as the home sparrow and the white-crowned sparrow, as all of them belong to the same domain — sparrow. Hence you can describe them as ‘sparrow’.

But in the case of the red-headed woodpecker, although it is still a bird, you can’t use the word ‘sparrow’ to describe it, as it is further apart from the existing vectors.

We take for granted how amazing our multi-modal brain model works in these day to day mundane scenarios, seamlessly converting text and image to a common semantic space. Just as we saw in this story, the model too will use the same method to classify: seen, unseen and unknown classes.

Although just like us it won’t be perfect at classifying the unseen classes all the time, it will predict an unseen bird to a class that is semantically quite similar to the actual class.

And in the case of an unknown class, it too will run a nearest neighbours similarity and find semantically similar classes.

The seen, the unseen and the unknown classes are differentiated by the type of data that we have.

One-shot and Few-shot Classification

• If we have images of the class in our data then it is a seen class.
• In the case of seen class classification, the problem boils down to a normal classification problem.
• If the number of images for the seen class is one or a few then it becomes a one-shot or few-shot classification problem.

The face detection algorithm in your smartphone uses few-shot learning to learn to identify your face.

Zero-shot Classificaion

• If we just have the textual/semantic description of it and no images, then it is an unseen class.
• Classification on unseen classes only is called zero-shot (ZS)classification.
• Classification on both seen and unseen classes is called generalized zero-shot (GZS) classification.

One classic example for understanding zero-shot classification is: describing a zebra as a horse with black and white stripes to a person who has never seen a zebra.

When you have neither the text nor the image of the class then it is referred to as an unknown class. The way to abstract information from such classes is by finding the nearest neighbours in the semantic vector space to find similar classes, as we did earlier.

References

• The images of the birds in this blog are used from the CUB-200 dataset
• The images from the ‘comparing with ZSL-models’ section have been taken from the paper: A Deep Multi-Modal Explanation Model for Zero-Shot Learning
• I have a video explanation of this paper and many more on my YouTube channel.

About Author

I am a senior CS undergraduate with a keen interest in Deep Learning. I try to write blogs and I make explanation videos on my YouTube channel where I explain niche and trending deep learning papers every week. I am also open to research collaborations, you can find my details on my website.

Solving the FloodNet problem

In this story, Semi-Supervised Classification and Segmentation on High Resolution Aerial Images, by the authors: Sahil Khose, Abhiraj Tiwari and Ankita Ghosh, is presented. In this paper:

• We analyze the FloodNet dataset for the first time as it was released as a part of the EARTHVISION 2021 workshop (hosted by CVPR) in March 2021 and we were one of the teams who participated in it.
• Solve the problem of scarcity of labelled data for the binary classification problem using pseudo label generation. Beating the best model of the FloodNet paper by 3% with less than half the number of parameters.
• Analyze the results of various multi-class semantic segmentation models with varying backbones over 10 classes and compare them when we introduce our semi-supervised approach from classification.

This paper is submitted to CVPR 2021 under the EARTHVISION 2021 workshop.

Outline:

1. What is FloodNet?
2. Flooded vs Non-Flooded — Semi-Supervised Classification
3. 10 class semantic segmentation
4. Conclusion

1. What is FloodNet?

• FloodNet is a high-resolution image dataset acquired by a small UAV platform, DJI Mavic Pro quadcopters, after Hurricane Harvey.
• These high resolution (3000x4000x3) images are accompanied by detailed semantic annotation regarding the damages.
• To advance the damage assessment process for post-disaster scenarios, the dataset provides a unique challenge considering classification, semantic segmentation and visual question answering.

The FloodNet competition under EARTHVISION 2021 has 2 tracks:

1. Track 1: Image Classification and Semantic Segmentation,
2. Track 2: Visual Question Answering.

We worked on Track 1 of the competition and submitted the paper on the same.

Dataset Details

• The whole dataset has 2343 images, divided into training (~60%), validation (~20%) and test (~20%) sets.
• For Track 1 (Semi-Supervised Classification and Semantic Segmentation), in the training set, we have around 400 labeled images (~25% of the training set) and around 1050 unlabeled images (~75% of the training set).
• For Track 2 (Supervised VQA), in the training set, we have around 1450 images with a total of 4511 image-question pairs.

Track 1

In this track, there are two semi-supervised tasks:

1. Semi-Supervised Classification: Classification for FloodNet dataset requires classifying the images into ‘Flooded’ and ‘Non-Flooded’ classes. Only a few of the training images have their labels available, while most of the training images are unlabeled.
2. Semi-Supervised Semantic Segmentation: The semantic segmentation labels include: 1) Background, 2) Building Flooded, 3) Building Non-Flooded, 4) Road Flooded, 5) Road Non-Flooded, 6) Water, 7) Tree, 8) Vehicle, 9) Pool, 10) Grass.

The dataset for Track 1 can be downloaded from this link

Track 2

For the Visual Question Answering (VQA) task there are images associated with multiple questions. The questions are divided into the following categories:

1. Simple Counting: Questions will be designed to count the number of objects regardless of their attribute. For example: “how many buildings are there in the image?”.
2. Complex Counting: Questions will be asked to count the number of objects belonging to a specific attribute. For example: “how many flooded buildings are there in the image?”.
3. Condition Recognition: In this category, all the questions are mainly designed to ask questions regarding the condition of the object and the neighborhood. For example: “What is the condition of the road in the given image?”.
4. Yes/No: For this type of question, the answer will be either a ‘Yes’ or a ‘No’. For example: “Is there any flooded road?”.

The dataset for Track 2 can be downloaded from this link

2. Flooded vs Non-Flooded — Semi-Supervised Classification

Data and Preprocessing

• In the given dataset 398 sample images were labeled out of which 51 samples were flooded and 347 were non-flooded.
• The large class imbalance prevents the model from achieving a good F1 score while training with the labeled dataset.
• To prevent this we used a weighted sampling strategy while loading the data in the model. Both the classes were sampled equally during batch generation.
• The labeled dataset was heavily augmented to get more images for training the model under supervision.
• The image samples were randomly cropped, shifted, resized and flipped along the horizontal and vertical axes.
• We downsized the image from 3000×4000 to 300×400 dimensions to strike a balance between processing efficiency gained by the lower dimensional images and information retrieval of the high-resolution images.

Methodology

• ResNet18 with a binary classification head was used for semi-supervised training on the dataset.
• The model was trained for E epochs out of which only the labeled samples were used for Eα_i epochs after which pseudo labels were used to further train the model. α has an initial value of α_i that increases up to α_f from epoch Eα_i to Eα_f as described in Algorithm 1.
• A modified form of Binary Cross-Entropy (BCE) was used as the loss function as shown in line 10 in Algorithm 1 where l is the label of a sample, l_hat is the predicted class for labeled sample and u_epoch is the predicted class for an unlabeled sample in the current epoch. This loss function was optimized using Stochastic Gradient Descent (SGD)

Experiments

• We used ResNet18 as it is computationally efficient. We experimented with Adam optimizer and SGD. Optimizing using SGD was much more stable and the optimizer was less susceptible to overshooting.
• Different values of α were experimented with and it was found that a slow and gradual increase in alpha was better for training the model.
• Our best performing model uses α_i= 0 and α_f= 1. The value of α increases from epoch Eα_i= 10 to Eα_f= 135. The model was trained on a batch size of 64.

Results

• Our system performed significantly better than all the classification baseline results mentioned in the FloodNet paper while having a considerably smaller architecture (half the number of parameters) as shown in Table 1.
• Our best model achieves 98.10% F1 and 96.70% accuracy on the test set.

3. 10 class semantic segmentation

Data and Preprocessing

• To expedite the process of feature extraction for the deep learning model, we apply a bilateral filter to the image, followed by two iterations of dilation and one iteration of erosion.
• For image augmentation we perform shuffling, rotation, scaling, shifting and brightness contrast.
• The images and masks are resized to 512×512 dimensions while training since high-resolution images preserve useful information.

Methodology

• The dataset contains labeled masks of dimension 3000×4000×3 with pixel values ranging from 0 to 9, each denoting a particular semantic label.
• These are one-hot encoded to generate labels with 10 channels, where i_th channel contains information about i_th class.
• We experiment with various encoder-decoder and pyramid pooling based architectures to train our model, the details of which are mentioned in the Experiments section.
• The loss function used is a weighted combination of Binary Cross-Entropy loss (BCE) and Dice loss as it provides visually cleaner results.
• We apply a semi-supervised approach and generate pseudo masks for the unlabeled images.
• While training the model for E epochs, the labeled samples were used for Eα_i epochs where Adam is used as an optimizer.
• After that pseudo masks were used to further train the model as described in Algorithm 1. α has an initial value of α_i that increases up to α_f from epoch Eα_i to Eα_f.
• SGD optimizer with 0.01 LR is used when pseudo masks are introduced to the model.

Experiments

• We apply three state-of-the-art semantic segmentation models on the FloodNet dataset.
• We adopt one encoder-decoder based network named UNet, one pyramid pooling module-based network PSPNet and the last network model DeepLabV3+ employs both encoder-decoder and pyramid pooling based module.
• We train all of them in a supervised fashion. For UNet, PSPNet and DeepLabV3+ the back-bones used were ResNet34, ResNet101 and EfficientNet-B3 respectively.
• For UNet the learning rate was 0.01 with step LR scheduler set at intervals [10,30,50] and decay factor γ set to 0.1.
• For PSPNet the learning rate was 0.001 without any LR decay. For DeepLabV3+ the learning rate was 0.001 with step LR scheduler set at intervals [7,20] and γ set to 0.1.
• Adam optimizer and batch size of 24 was used for all the models with Mean Intersection over Union (MIoU) as the evaluation metric.
• We observed the best results when we weighed the BCE loss and Dice loss equally.
• Once we recognized the best performing model on the task, we trained a DeepLabV3+ model with EfficientNet-B3 as the backbone and used SGD optimizer instead of Adam optimizer in a semi-supervised fashion.
• Due to computation and memory constraints, we randomly sampled unlabeled data with the ratio of 1:10 for generating the pseudo masks.

Results

• Table 2 showcases the comparison of the best models we achieved for each of the 3 architectures.
• The best test set result was achieved by a DeepLabV3+ architecture with an EfficientNet-B3 backbone.
• A few example images of the predictions of the model against the ground truth are provided in Figure 1.
• It is evident from the table that small objects like vehicles and pools are the most difficult tasks for our models.

To look at all our semantic segmentation maps like the one in Figure 1 checkout our demo

4. Conclusion

• In this work, we have explored methods to approach semi-supervised classification and segmentation along with handling the class imbalance problem on high-resolution images.
• We have conducted a range of experiments to obtain the best possible technique and models to optimize for the tasks.
• Our classification framework achieves laudable results with just 398 labeled images and also utilizes the entirety of the unlabeled data.
• Our segmentation framework shows an increase of 0.19% on using the unlabeled data as pseudo labels. This provides a wide scope of improvement.
• The amount of unlabeled data is approximately three times the amount of labeled data which if employed efficiently can produce superior results. We foresee multiple opportunities for future research.
• Training the model in an unsupervised fashion and fine-tuning it with the labeled data followed by distillation as presented in SimCLRv2 is a very promising method.
• Training with a contrastive loss has been incremental at times. With the emergence of Visual Transformers, self-supervised Vision Transformers could also be explored for this task.

Links

• Semi-Supervised Classification and Segmentation on High Resolution Aerial Images (our paper)
• Semantic Segmentation Outputs Demo
• GitHub repository for our paper’s implementation
• YouTube video of the authors explaining the work

About Author

I am a senior CS undergraduate with a keen interest in Deep Learning. I try to write blogs and I make explanation videos on my YouTube channel where I explain niche and trending deep learning papers every week. I am also open to research collaborations, you can find my details on my website.