[paper review] Referring Image Segmentation via Cross-Modal Progressive Comprehension

#referring_image_segmentation #referring_expression_comprehension #multi-modal #CVPR_2020

Get 小連同學’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

[paper link] [code link]

My Key Takeaways

• When the learnable parameters come in useful or how they make fusion have been my lessons recently.
• Let the research composed of many modules. It helps design the ablation study and makes the paper more structural (explanation can be clearer and exhibit a stronger cause-and-effect relationship lol).

Model Architecture

Overview of this proposed method

This model can be briefly separated into two modules:

1. Cross-Modal Progressive Comprehension (CMPC)
   step1. Entity Perception: Entity and Attribute words (e.g. white ball) are extracted and fused with visual features.
   step2. Relation-aware reasoning on the multimodal graph: Spatial vertexes that have strong responses to the relational words will have more interaction with others. Therefore, the one actually having some relationship with others (referent in expression) could be identified.
2. Text-Guided Feature Exchange (TGFE)
   It utilizes linguistic features as guidance to select useful feature channels so as to realize information communication. Then, the multi-level features are fused by ConvLSTM for mask prediction.

Method

[CMPC]

CMPC module consists of two stages

Entity Perception

• The words are classified into 4 types: entity, attribute, relation and unnecessary word. The probability of which type the word is calculated by below function, in which W1, W2, b1 and b2 are learnable parameters.

• The global language context feature is calculated as a weighted combination. (l is extracted with a language encoder)

• Then, associate q with the visual feature of each spatial region by bilinear fusion strategy (element wise product). A fully-connected multimodal graph is constructed over M with relational words. (X is visual feature transformed by CNN backbone)

• Finally, fusion makes integration of both visual and linguistic context, which lets all the entities referred to by the expression perceived.

Relation-aware Reasoning

• The multi-modal graph is defined as G = (V; ε; Mg; A), where V and ε are the set of vertexes and edges, Mg the set of vertex features, A the adjacency matrix. Reasoning is done by connecting vertexes (i.e. image region) with edges.

R: features of relational words

• B is the affinity matrix between Mg and R.
• A represents the normalized magnitude of information flow from the spatial region i to the region j, which depends on their affinities with relational words in the expression.

adjacency matrix A formula

• I (identity matrix) serves as a shortcut to ease optimization. (I’m not sure how it useful)

graph convolution applied to the multimodal graph

• After graph convolution applied to the multimodal graph, the model would take reshaping operation. It repeat s for H x W times and concatenate it with X and -Mg along channel dimension.

textual combine features of all necessary words

[TGFE]

Information Communication Among Multi-level Features

• Feature exchange (n rounds) and multi-level semantics are important to make it. And ConvLSTM deals with final mask prediction.
• In feature exchange, Yi updates every time based on the old value.
  Context vector Ci is calculated by fusing old probability vectors s (mentioned in RAR) and old global vector gi (calculated by below function) with a fully connected layer.

weight matrix (s mentioned in RAR)

by weighted global pooling

output feature of TGFE (level i (3, 4, 5) at round k)

Main Results

comparison with SOTA

Experiments

1. Module combination
This actually proved the effectiveness of every module, especially for RAR.
(Table Detail:
- The baseline is implemented as simply concatenating the visual feature extracted with DeepLab-101 and linguistic feature extracted with an LSTM and making prediction on the fusion of them.
- row 1–5 based on single-level features, row 6–11 based on multi-level
- GloVe means using GloVe word embeddings to initialize the embedding layer.)

ablation studies on UNC val set

2. The number of graph convolution layers in CMPC module
There is an awesome score just with CMPC module, and it become little better with TGFE module though the improvement is subtle as # convolution layer increasing.

Overall IoUs on UNC val set

3. The number of graph convolution layer
Not always the deeper layer it has, the better performance it achieves.

overall IoUs on UNC val set and G-Ref val set

4. Visualization of Affinity Maps to check what the model actually look for
The experimental modification of expression proved the flexibility.

Personal Point of View

1. This paper actually focuses on REC and reasoning issue. At its core, the task is referring image segmentation (semantic), but the main contribution is thanks to linguistic features and textual information processing.
2. There are many learnable matrices or parameters, which leads to high complexity and low explainability. However, fusion or anything related to multi-modal seem to be calculated by them.