[paper review] LAVT: Language-Aware Vision Transformer for Referring Image Segmentation

#referring_image_segmentation #language-aware #CVPR_2022

Get 小連同學’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

[paper link] [code link]

My Key Takeaways

1. Complex systems arise from simple beginnings. LAVT appears to be a Unet variant with a pixel-word attention mechanism and a language gate. This finding reminds me that, in addition to studying state-of-the-art (SOTA) models, it’s equally important to revisit the basics and classical models.
2. Feature reuse always plays an important role in Referring Image Segmentation (RIS) tasks. For instance, Fn is utilized not only in fusion but also in the segmentation stage. Similarly, in the CMPC model, the original visual features are reused for reasoning.

LAVT (Language-Aware Vision Transformer)

the overall model architecture of LAVT

Before learning about the detail of LAVT…

Why LAVT?

The LAVT model directly integrates linguistic information into visual features at intermediate levels of a vision transformer network, different from the previous method.

• The most of state-of-the-art (SOTA) models at that time first perform feature encoding separately before handling image-word alignment in the decoder.
• The authors consider the model can further improve its understanding of semantics. In other words, the textual features can be fused through a better approach.

Model Architecture

Okay, so this model can be divided into four parts:

1. Features Extraction

The visual Features are extracted by Swin Transformer while the textual features are from the powerful model, BERT.

2. Pixel-word Attention Mechanism

the pipeline of PWAM

• 2 kinds of textual features: PWAM aggregates L across the word dimension to generate a position-specific, sentence-level feature vector.
• The calculation of attention mechanism leverages the classical work, Attention is all you need. Visual features Vi as the query and linguistic features L as the key and the value.
• Every ω is projection function used to transform feature maps.
• Flatten unrolls the two spatial dimensions into one dimension in row-major for proper calculation.
• Lower computation: PWAM produces a much smaller memory footprint as it avoids computing attention weights between two image-sized spatial feature maps.

3. Language Gate (Pathway)

the schema of the language pathway

• This module is designed to prevent Fi from overwhelming the visual signals in Vi and to allow an adaptive amount of linguistic information flowing to the next stage of Transformer layers.
• By adding Vi, the final multi-modal features avoids disrupting the initialization weights pre-trained on pure vision data.

4. Segmentation

the decoding process of segmentation

• LAVT has been identified as a model that incorporates both bottom-up and top-down structure.
• After every bilinear interpolation-based upsampling, the output Yi is concatenated with the feature Fi, as described by the functions mentioned above.
• The final feature maps, Y1, are projected into two class score maps via a 1x1 convolution. I consider this approach ensures that the original visual features, Fi, retain a certain level of importance.

Main Result

LAVT outperforms the SOTA model at that time by over 7⭡ overall IoU and is 1–2 higher than contemporary CRIS model (introduced in next story!).

Ablation Study

visualized predictions and feature maps on an example from the RefCOCO validation set

Visualized predictions

Above visualization shows the effectiveness of two modules. It is found the removal of LP and the removal of PWAM both lead to false negative predictions on the front window area of the target bus, while the removal of LP additionally results in the false positive identification of the middle bus.

Fair comparison with reference methods

comparison between LTS, VLT and EFN on the RefCOCO validation set

All models use the same backbones to achieve fairness, and the authors further combine their approach with VLT by substituting the original mask predictor with the cross-modal decoder from VLT to prove the proposed encoding scheme is more effective.

ablation studies on the RefCOCO validation set

Activation function in LG and Normalization layer in PWAM

The best final activation function in LG is Tanh, while the best normalization layer in PWAM is instance normalization.

Features used for final classification

• the outputs from PWAMs (Fi, i ∈ {1, 2, 3, 4})
• the outputs from the Transformer layers (Vi, i ∈ {2, 3, 4})
• the inputs to the following Transformer layers (Ei, i ∈ {1, 2, 3})
• G / NG indicated with or without the proposed language gate.

multi-modal attention module

• To show PWAM performs better, the authors directly replace PWAM with other attention modules in their basic framework.
• BCAM: computationally-heavy
  GA: the most top-performing module at that time

Personal Point of View

• This paper’s ablation study ensures fairness, setting all comparisons on the same standard without resorting to parameter tuning. Instead, it seeks the optimal modules combination through the addition, removal, and substitution of different functions.
• The modules of LAVT are clearly defined, allowing for transplantation into different models or experiments. This paves the way for future work to build upon this foundation for further development.