[paper review] Zero-shot Referring Image Segmentation with Global-Local Context Features

#referring_image_segmentation #CLIP #zero-shot #CVPR_2023

Get 小連同學’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

[paper link] [code link]

My Key Takeaways

1. The knowledge in linguistics gives rise to the important approach — Global-Local Context Features. The target object noun is extracted by the spaCy tool, helping this model catch the visual relationship between target and scene.
2. The transformer structure has extremely potential flexibility. As you can transfer any data type into sequence, multi-modal features may be fused through transformer mechanism. This is the reason why I want to continue my research on CV transformer.

Global-local CLIP

model architecture

The model is briefly composed of 4 parts:

1. Global-context Visual Features Extraction:
   In this paper, the two different architectures, ResNet and ViT, are employed as the visual encoder, similar to CLIP (the latter performs sightly better). These features have comprehensive information about the scene.
2. Local-context Visual Features Extraction:
   The cropping operation, commonly used in zero-shot semantic segmentation, is applied to remove irrelevant region, enabling these features to focus on the masked parts.
3. Global-local Context Visual Features Extraction:
   The global-local context visual feature is computed using global-context and local-context visual features.
4. Global-local Context Textual Features Extraction:
   To address RIS problem which CLIP cannot handle because of image-level presentation, spaCy is utilized to parse dependency and identify the target noun in Local-context textual features extraction. Eventually, the global-local context textual feature is weighted sum of global and local textual features.

formulas of each part

Below is analysis show the best α, β value on different test dataset:

Analysis on hyperparameters α and β for each visual and textual modality

Why Global-local CLIP?

The shortcoming of the original CLIP should be discussed. Here are the insights after studying this paper:

• CLIP view image as image-level presentation, so the visual feature it extracts cannot understand the global relationship between multiple objects while the textual feature it extracts cannot specify the target noun in the expression.
• As mentioned above, CLIP isn’t completely suitable for dense prediction tasks like object detection and image segmentation because they’re pixel-level.

Why Zero-shot?

• For specific prompts, annotation incurs high costs.
• There are weakly-supervised methods to solve RIS problems, but it is expensive to perform high-level annotation, as mentioned above..
• As of the publication of this paper, there is no zero-shot approach to solving the RIS problem.

Two Different CLIP visual encoder — ResNet and ViT

[ResNet]

• A global average pooling layer in original ResNet is replaced by an attention pooling layer, the same architecture as the multihead attention in a Transformer.
• A query is computed by a global average pooling operation onto the feature maps extracted by the ResNet backbone while a key and a value given by a flattened feature map.

[ViT]

• Just like the original ViT model.
• To capture global-context of images, masking tokens is in only the last k Transformer layers (k = 3 the best). The tokens are reshaped and masked by a given mask proposall, and then flattened and applied to the subsequent Transformer layer.
• The output from the class token (CLS) is as the global-context visual representation.

detailed illustration of our mask-guided global-context visual encoders

Main Results

The global-local context approach is innovative and achieves the ability to distinguish between different instances by referring text. Through the significant indicator, we can find that:

• Compared with other zero-shot RIS baseline, Global-Local CLIP outperform them including weakly-supervised one.

comparison with Zero-shot RIS baseline

• Below left chart shows Global-Local CLIP also performs better than both supervised method and works robustly on the unseen domain while the right chart shows in a few-shot learning setting, it outperforms others with large margins when k is small.

comparisons to supervised methods in zero-shot setting

Ablation Study

The relatively important analysis includes 2 insights:

1. oIoU results of our method and the baselines using COCO instance GT masks

2. . oIoU results with different context-level features on the val split of RefCOCOg

1. If the mask generation part performs better and generates more precise results, similar to the ground truth, this model would definitely improve.
2. The performance of the model, utilizing global-local context features is the best, which indicates the necessity of incorporating the idea of word dependency. Below are the visualization results:

results with different levels of textual features

Personal Point of View

• The Global-Local CLIP model stands on the shoulders of giants, CLIP FreeSOLO, etc., incorporating numerous existing concepts and models to forge its own innovative approach.
• There are many comparison methods of similarity, such as Jaccard similarity, Euclidean distance, and so on. They can be utilized in ablation study part.