[paper review] PolyFormer: Referring Image Segmentation as Sequential Polygon Generation

#referring_image_segmentation #transformer #regression-based_decoder #CVPR_2023

Get 小連同學’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

[paper link] [project link]

My Key Takeaways

1. Polygon-based segmentation has better performance than mask-based. The description of object contour only requires vertices, which also allows more intuitive representation of an object’s shape and contour visually.
2. In the demo of results, it is evident that the model possesses the ability to recognize the order of objects before and after. This is likely attributed to the effectiveness of text features. Hence, for RIS, I think the ability to understand depth information in images could also be the one of the evaluation indicator in qualitative analysis.
3. Continuous coordinate is definitely more precise than discrete one, and it would be easier to do the interpolation among pixels or geometric transformation, which may be the inspiration of new method or the improvement.

PolyFormer

model architecture

The model is briefly composed of 3 parts: the features extraction is done by the concatenation of the output from visual and text encoder, the transformer encoder is responsible for multi-modal fusion and the transformer decoder is in charge of coordinate embedding and prediction.

Target outcomes

1. Trough concatenation (it seems to be done by fully connected layer), the multi-modal features allow image and text to coexist in the same dimensions.
2. It improves the shortcomings of mask-based segmentation by polygon-based method. Segmentation is no longer determined by classifying each individual pixel; instead, prediction is calculated through regression-based method.
3. Converting pixels into continuous coordinate values reduces quantization errors, thereby enhancing segmentation accuracy.
4. It is the model generating sequence of polygon vertices and bounding box corner points at the same time. And the generalization ability is clearly increased.

Methodology

[encoder]

Target Sequence Construction

• The sequence utilizes specific symbols to distinguish different classes:
  BOS -> the beginning of a sequence
  SEP -> separate different polygons
  EOS -> mark the end of a sequence
• In terms of the representation, polygons are defined with the point closest to the top-left corner as the starting point and are arranged clockwise, which is also proved to have the best performance.

Image and Text Feature Extraction & Fusion

• In encoder, it uses existing models, Swin and BERT, to transform image and text. The resulting features are processed through learnable matrices and bias vectors, followed by concatenation.

concatenation

• For multi-modal fusion, the combined features are sent into a transformer layer, including multi-head self-attention layers, layer normalization, and a feed-forward network.

[decoder]

Regression-based Transformer

decoder architecture

• The model utilizes 2D coordinate embedding and bilinear interpolation techniques to correct quantization errors by the formula |x — [x]|, which isn’t achievable for pixel-based method.
• The vertex prediction process involves N layers of feed-forward networks, multi-head cross-attention, and multi-head self-attention.
• The output layer consists of two heads: the class head, which identifies sequence symbols such as <EOS> and <SEP>, and the coordinate head, responsible for predicting the final output bounding box and polygon.
• Regression loss and label smoothed cross-entropy loss are applied in decoder. The total loss be like:

total loss

Main Results

PolyFormer is multi-task model. For RIS, REC and even zero-shot transfer to referring video object segmentation, it has remarkable performance.

comparison with the state-of-the-art methods on RIS benchmarks

comparison with the state-of-the-art methods on REC benchmarks

Ablation Study & Evaluation

• Coordinate Classification vs. Regression: the regression-based model consistently outperforms the classification-based model on all datasets. But it seems to be without any reason(?), but the finding is important.

ablation study on decoder

• As for the sequence construction, the polygon vertices are ordered clock-wise and the starting point is set as the vertex closest to the image origin. Whether data augmentation is applied and it is multi-task also effects the performance. Most importantly (I think lol), adding <SEP> to handle multi-polygon cases is very ingenious.

ablation study on target sequence construction

Through visualized result:

• When comes to segmentation details from (a), it can be noticed that the dog’s feet are not clearly outlined, and the paws are outlined with triangular shapes(?). I believe increasing the number of vertices might address this issue.
• From (b) and (d), it can be observed that the model’s recognition is not affected by occlusions, and it successfully identifies different segmented areas as the same object.
• Overall, compared to previous outstanding models, PolyFormer’s segmentation performance is significantly better.

Personal Point of View

• Using polygon vertices as the segmentation contour may result in unevenness on the contour if the number of points is limited. If the number of vertices could be specified, it may be improved.
• As for decoder comparison, it seems that there isn’t enough explanability of why regression-based is better than classification-based regardless of the experiment data. (or just I can’t understand math or the how the practical implement is done lol)
• Due to the contour presentation of polygon vertices, interpolation methods can be directly utilized as data augmentation technique, offering new option compared to traditional mask-based method. However, determining how to set parameters for interpolation in polygon-based method would be a potential research issue.