[paper review] CRIS: CLIP-Driven Referring Image Segmentation

#CRIS #CLIP #referring_image_segmentation #CVPR_2022

[paper link] [code link]

Personal Key Takeaways

• The main adjustments to the RIS model architecture focus on how and when fusion is handled; currently, I haven’t seen papers specifically discussing the issue of data preprocessing or augmentation before training. I would like to explore this direction further.

comparison table for the recently outstanding models

• As mentioned above, how and when fusion is handled is extremely different in every model, so I organized a comparison table for the outstanding model I’ve seriously studied so far, which helps me gain a clearer understanding to design my own.

CRIS (CLIP-Driven RIS)

an illustration of CRIS’ main idea

Why CRIS?

• Early methods for text-pixel alignment included viewing images as image-level, directly concatenating them, or employing language-vision attention mechanisms, …etc. (There is probably no loss function or other training method specifically designed for this task.)
• Like the problem Referring Image Segmentation (RIS) focusing on, if there were duplicate objects, they would all be segmented, but the models at that time still have limitation.

the overview of the proposed CRIS framework

There are mainly six modules in CRIS:

1. text encoder

2 textual features

CIRS adopts a Transformer (Attention is all you need) modified by “Language models are unsupervised multitask learners’’ to extract text features.
While the Ft is the original output of modified transformer, Fs is the global textual representation further transformed from the activations of the highest layer, [EOS] token there.

2. image encoder

3 visual features

CRIS utilizes multiple visual features from the 2th-4th stages of the ResNet.

3. cross-modal neck

the process of how Fv comes

This stage involves a series of transformations and calculations, which are clearly described in the paper. Especially, the last function leverage the method, An intriguing failing of convolutional neural networks and the coordconv solution, to learn a mapping between coordinates in (x,y) Cartesian space and one-hot pixel space.

4. vision-language decoder

how to generate the multi-modal features Fc

• Why vision-language decoder?
  The decoder obtains fine-grained semantic information and generate multi-modal features for the final output of segmentation masks.
• Architecture:
  1. each layer follows the classic transformer architecture, including Self-Attention (catching global contextual information), Cross-Attention, and Feed-Forward Network.
  2. MLP — two layers with Layer Normalization and residual connections.
• Above function finally generates the multi-modal features, Fc. It is straightforward to understand that this module operates similarly to the classical transformer.

5. text projector

textual projection

The text projector is adopted to transform Fs.

6. image projector

visual projection

The image projector is adopted to transform Fc.

Get 小連同學’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

After transformed, text-to-pixel contrastive loss plays the important role!

text-to-pixel contrastive loss

• This loss function optimizes the relationship between two modalities in same embedding space (after projection).
• The similarity is measured by dot product. Zt is encouraged to be similar with its corresponding Zv and dissimilar with other irrelevant Zv.

Experiments

Main Results

• The whole table shows CRIS effectively transfers the knowledge of the CLIP model from image-level to pixel-level, enhancing the ability of cross-modal matching.
• Especially for G-Ref dataset with longer and more complex expression, the proposed method consistently achieves notable improvement and has the strong ability to model the global contextual information.
• However, its performance is actually a little worse than simultaneously published LAVT.

Ablation Study

• The contrastive loss actually pulls closer linguistic and relevant pixel-level visual representations and pushes away other irrelevances, which is proved by adding this module to the baseline.
• Compared with the baseline, the decoder brings improvement. Particularly, the self-attention operation can help capture long-range dependencies across each pixel to understand complex scenarios. As for propagating fine-grained semantic information from textual features to pixel-level features, the use of encoded words in cross-attention operation also help increase the accuracy.
• The effectiveness of contrastive learning and vision-language decoder boosts the ability of cross-modal matching and generates precise segmentation masks.
• n = 3 is the best setting while n = 4 may cause over-fitting.

Qualitative Analysis

• Vision-language decoder and contrastive learning actually help generate the high-quality segmentation masks, but the model is still confused in some hard regions without any one part.

qualitative examples with different settings

The failure cases show the room for improvement:

• the ambiguity of the input expression is not specific enough to describe the target object
• the wrong label in the dataset
• the inaccurate boundaries (It may be alleviated by introducing other technologies like refine module.)
• occlusion in image

qualitative examples of failure cases

Personal Point of View

Why Neck? Why multi-modal fusion here?

Multi-modal is essential in RIS tasks. Different models adopt different strategies to fuse features in different times and in different ways.
All differences lead to varying outcomes, which, for now, can only be observed through evaluation metrics like IoU. Visualized results and qualitative analysis seem unable to point out the distinctions in the effects produced by fusion in different times or using different methods.

Why Fs, Ft?

Fs is the global textual representation while Ft is extracted by the modified Transformer, the original transformed features. These two enable the model to learn textual information from different perspectives, capturing both local object details and global scene information.