Cross Modal Distillation for Flood Extent Mapping

Sentinel-1 image: Sentinel-1 [39] mission launched by European Space Agency (ESA) consists of 2 polar orbiting satellites to provide free SAR data. However, recently one of the satellite malfunctioned and currently remains out of service. ESA has plans to launch another satellite in 2023. This satellite is an example of active remote sensing satellite and uses radio waves operating at a centre frequency of 5.405 GHz. This allows it to see through cloud cover. It has a spatial resolution of 10m and has a return period of 6 days. We use the Sentinel-1 GRD product and utilize the bands that consist of dual polarized data: Vertical Transmit-Vertical Receive (VV) and Vertical Transmit-Horizontal Receive (VH). These bands representing the backscatter coefficient are converted to logarithmic(dB) scale. The backscatter coefficient is mainly influenced by the physical characteristics such as roughness and the geometry of the terrain and the dielectric constant of the surface. It is discriminative for detecting surface water as water reflects away all the emitted radiation from the satellite in a specular way.

Sentinel-2 image: Sentinel-2 [11] mission was also launched by the European Space Agency (ESA) and has 2 polar orbiting satellites. It provides multispectral data at a resolution of 10m and has a return period of 5 days. We use the L1-C Top of Atmosphere (TOA) product. It is a passive remote sensing satellite operating in visible and infrared wavelength. Its images are affected by atmospheric conditions and often contain significant cloud cover. The multispectral data consists of 13 bands and in this work we use 4 bands: B2 (Blue), B3 (Green), B4 (Red) and B8 (NIR). These bands are used for various tasks like land cover/use monitoring, climate change and disaster monitoring.

Weak label: The weak labels are computed from the Sentinel-2 image. Though cloud free Sentinel-2 data is rarely available during a flooding event, we can still opportunistically sample timestamps from a long time range and get enough cloud-free views. Initially, the cloud mask is estimated using Sentinel-2 quality assurance band (QA60). The cloud mask is then dilated to mask out the nearby cloud shadows as the spectral signature of the cloud shadows is similar to that of water [24]. Weak flood labels are then created by thresholding the Normalized Difference Water Index ($NDWI=(B3-B8)/(B3+B8)$) band. The pixels having value greater than 0 are marked as water and the rest are marked dry.

Water occurrence map: The water occurrence map [31] shows how water is distributed temporally throughout the 1984-2020 period. It provides the probability of each pixel being classified as water averaged over the above time period. It has a 30m resolution data. The data was generated using optical data from Landsat 5, 7, and 8. This map can be used to capture both the intra and inter-annual changes and to differentiate seasonal/ephemeral flooding pixels from permanent water pixels. Each pixel was individually classified into water / non-water using an expert system and the results were collated into a monthly history for the entire time period. Averaging the results of all monthly calculations gives the long-term overall surface water occurrence.

Sen1Floods11 dataset: This is is a publicly available dataset [6], containing 4831 tiles from 11 flooding events across 6 continents. It contains paired Sentinel-1 SAR and Sentinel-2 multi-spectral images. Each tile is $512{\times}512$ pixels at a resolution of 10m per pixel. Due to the high cost of labeling, only 446 tiles out of 4831 are hand labelled by remote sensing experts to provide high quality flood water labels. The authors provide an IID split of these hand labelled tiles, containing 252 training, 89 validation, and 90 test sample chips. The remaining 4,385 tiles have weak labels prepared by thresholding NDVI (Normalized Difference Vegetation Index) and MNDWI (Modified Normalized Difference Water Index) values. The weak labels are only used for training as they are not accurate enough to be used in validation or test. We also augment the dataset with the water occurrence map for every tile.

Floods208 dataset: We curated additional imagery by downloading closely acquired Sentinel-1 and Sentinel-2 images from Earth Engine [14] during flood events provided to us by external partners.

The data is extracted using the following steps:

• •

  For each data point consisting of latitude, longitude and flooding event timestamp, get the Sentinel-1 image of the Area of Interest (AOI).

• •

  Search for overlapping Sentinel-2 images within 12hrs of Sentinel-1 timestamp. Filter out images with $>12\%$ cloud cover.

• •

  Pick the Sentinel-2 image closest to Sentinel-1 timestamp from the filtered images. If none are available, we discard this data point.

The data points were extracted from 208 flooding events across Bangladesh, Brazil, Colombia, India and Peru. These regions are shown in Figure 1 and were chosen according to the regions of interest for final deployment. We also extracted water occurrence map and a weak label for each image using the Normalized Difference Water Index (NDWI) band from the Sentinel-2 image. The extracted images had a resolution of 16m per pixel. Each image from a flooding event was then partitioned into multiple small tiles of size $320{\times}320$. The tiles at the edge of image were padded to fit the $320{\times}320$ tile size. Tiles having cloud percentage greater than $80\%$ were discarded. In total 23,260 valid tiles of size $320{\times}320$ were extracted from the whole process. Figure 2 shows some selected data points from both the datasets.

The provided Sen1Floods11 dataset has Latitude / Longitude Projection and has a resolution of 10m per pixel. To match the projection and resolution of our Floods208 dataset, all the images are scaled to 16m per pixel input resolution and projected to the Universal Transverse Mercator (UTM) coordinate system. For Sentinel-1 image normalization, VV band was clipped to [-20, 0] and VH to [-30, 0] and then linearly scaled these values to the range [0,1]. For Sentinel-2 image, the 4 bands were clipped to [0, 3000] range and then linearly scaled them to [0, 1] range. Table 1 summarises the final attributes of both the datasets.

We train two supervised models for comparison. The first model is trained only on hand labelled data $D_{l}$. The second model is trained only on the larger weak labelled dataset $D_{wl}$. The large size of the weak labeled data helps the network to generalize better (despite there model learning some of those label errors during training)  [32]. We use Deeplab v3+ [8] with an Xception 65 encoder [9] as the model architecture. Common regularization techniques like data augmentations (random crop with distortion, horizontal/vertical flips and colour jitter), dropout, weight decay and batch normalization are used to improve generalization. We also tried to fine tune the second model i.e the model trained on weak labelled data, using hand labelled data $D_{l}$. However, this additional training step did not improve the performance, so we decided not to include it in the baseline model.

Edge weighted loss

The network is trained to minimize the cross entropy loss. For every batch of images ( $B_{wl}$ ), the parameters $f$ of the network are updated by minimizing the cross entropy loss given by:

Here $W_{i}$ represents the pixel wise weights for the $i^{th}$ image - $X_{i}$, applied to the cross entropy loss. We apply an edge based weighting that gives higher weights to the edges of the binary label. As suggested by [43], from all the pixels in an image available for training, the pixels lying in the middle of an object can be easily discriminated by the model and are usually classified correctly. Hence, an edge weighted loss helps the network to focus on the harder to segment regions lying on the boundary during training. We compute two kinds of edges, inner and outer edges. Inner edges are obtained by subtracting the eroded labels from the original labels. Outer edges are obtained by subtracting the original labels from the dilated labels. All the other pixels are given a unit weight. The weights for inner and outer edges are decided by tuning these parameters during training. Figure 3 shows these edges.

Despite using regularization techniques, we notice that the model learns the label mistakes present in the training data. Figure 4 demonstrates this on the training set images. Often such mistakes consist of cases where a complete or a major part of the river is missing. While the network is resilient to learning small label mistakes (such as slightly overflooded pixels surrounding the true boundary or random noise similar to cut out augmentation), it does tend to overfit to cases when the label mistakes are large. This is because from an optimization perspective, pixels in an extremely noisy label will dominate the loss, forcing the network to overfit on them in order to reduce the loss.

We aim to correct the labels having large parts of the river missing by using water occurrence map to get a rough estimate of such missing rivers. The label initially misses the river because they are formed by thresholding Sentinel-2 image and the label is sensitive to the value of the threshold. Since the water occurrence map [31] is made by averaging long-term overall surface water occurrence, the permanent water pixels will have higher probability values and areas where water where sometimes occurs will have lower probability values. The weak label is improved by additionally marking the pixels having their water occurrence probability above a certain threshold, as wet in the label. This method will not be able to capture the seasonal/flooded or ephemeral water pixels, but these pixels mostly constitute a small majority of the label and as seen earlier, the model can learn to overcome such small mistakes in the labels. Figure 5 shows some randomly selected tiles before and after the label improvement from the training set.

Though water occurrence map improved the weak labels in areas which are permanently water, it does not help with label mistakes in seasonal/flooded pixels. As a result, we can still get grossly incorrect weak labels in rare situations where flooded pixels constitute a majority portion of the water. To overcome this, we explore a cross modal distillation framework that uses a network trained on a richer modality with a small set of hand labels, to generate more accurate labels on the training split. A teacher-student setup is used to transfer supervision between the two modalities. The teacher is trained on stacked Sentinel-1 and Sentinel-2 images using accurate labels from $D_{l}$, and is used to supervise a Sentinel-1 only student model on the unlabelled images from $D_{wl}$. The advantage of this method over binary weak labels (used in Section 3.1), is that the soft labels predicted by the teacher capture uncertainty better compared to the binary labels  [3, 18]. The soft targets are also outputs of a network trained on hand labeled data and are more likely to be correct than weak labels. Compared to self distillation, cross modal distillation enables us to provide more accurate supervision by transferring information from a richer knowledge modality. Figure 6 summarizes the training setup used in our work. Both the teacher and the student have identical architecture backbones. Let $f_{t}$ and $f_{s}$ represent the teacher and student network function, respectively. The training is done in two stages as described below:

Stage 1: Training the teacher network Let $X_{S1+S2}^{i}$ denote the $i^{th}$ stacked Sentinel-1 and Sentinel-2 image. $X_{S1+S2}^{i}\in D_{l}$ is used as input to the teacher network. The teacher is trained in the same manner as the supervised baseline described in Section 3.1. The training set is small but contains data from geographic locations spanning 6 different continents. This helps the teacher generalize well to different geographies in the unlabelled data seen during the next stage of training.

Stage 2: Training the student network The teacher weights from Stage 1 are kept frozen in this stage. We use paired Sentinel-1 and Sentinel-2 images from Sen1Floods11 data(i.e. both hand labelled and weak labelled) and Floods208 weak labelled data as the unlabelled data to train the student network. The data in each batch is sampled equally from both the data sources to ensure equal weighting for the datasets. The stacked Sentinel-1 and Sentinel-2 image $X_{S1+S2}^{i}$ is passed through the teacher to obtain the probabilities $p_{t}=\sigma(f_{t}(X_{S1+S2}^{i}))$ and the augmented paired Sentinel-1 image $\tilde{X}_{S1}^{i}=Aug(X_{S1}^{i})$ is passed through the student to get the student probabilities $p_{s}=\sigma(f_{s}(\tilde{X}_{S1}^{i}))$. Here $\sigma$ refers to the sigmoid function used to convert the model’s output into a probability score. KL divergence loss ($L_{KD}$) is then minimized for $K=2$ classes to update the student weights:

For all the models, we use Deeplab v3+ model [8] with Xception 65 [9] as the backbone encoder. The skip connection from the encoder features to the decoder is applied at stride 4 and 2. We use a batch size of 64 with input image shape of $(321,321,C)$ (here $C=2$ for Sentinel-1 images and $C=4$ for Sentinel-2 images). For optimization, Momentum optimizer is used with momentum set to 0.9. The learning rate is decayed with a polynomial schedule from initial value to zero with a power of 0.9. The models are trained for 30k steps. A learning rate and weight decay grid search hyperparameter tuning is done by choosing a learning rate from {0.3, 0.1, 0.003, 0.001} and weight decay from {1e-3, 1e-4, 1e-5, 1e-6}. For edge weighted cross entropy loss, the inner and outer edge weight is set to 10 and 5 respectively. A threshold of value 0.5 is used to create a binary mask of permanent water pixels from the water occurrence map. All the hyperparameter tuning and best model checkpoint selection is then done on the validation split. After the best checkpoint selection, the model is frozen and all the results are reported on the test split.

We use pixel-wise intersection over union (IoU) of the water class to validate our model performance. It is defined as follows:

In the above formula, $i$ is the iterator over all the images and $N$ represents the total number of images. For the $i^{th}$ image, ${TP}_{i}$, ${FP}_{i}$ and ${FN}_{i}$ denotes the number of true positives, false positives and false negatives of water class respectively.

Comparison with our baselines

Table 2 shows the result of the supervised baseline model trained on the Sen1Floods11 hand labelled data. As expected, the model using Sentinel-2 as input image is much higher than the models using Sentinel-1 image because Sentinel-2 image is a richer modality. However, Sentinel-2 image is not suitable for inference because of cloud cover issues mentioned in Section 2.1. Hence, even though we can’t use the model using stacked Sentinel-1 and Sentinel-2 image during inference, we can opportunistically utilize this as a richer set of features to train the teacher model for cross modal distillation as discussed in Section 3.3.

Table 3 shows a quantitative comparison of the supervised baseline method with model trained after improving the weak labels and the cross distillation model. In Table 3 first two rows, it can be seen that a supervised model trained on Sentinel-2 weak label can match the performance of a supervised model trained on small set of hand labelled data. This empirically verifies the claim that a large weak labelled dataset can act as a quick substitute for a small amount of costly hand label annotations. Including Floods208 weak labelled data to Sen1Floods11 weak labelled data, further led to an increase in the model performance by $1.18\%$ from Sen1Floods11 weak label supervised baseline. This shows that there were still more gains to be had by increasing the weak label dataset size. Even though Floods208 weak labelled dataset by itself is also a very large dataset, the model trained on only Floods208 does not perform good. This is because Floods208 data is localised to a few specific locations while Sentinel-2 weak labelled split has data from 6 continents across the world. This emphasizes the importance of sampling training data strategically, such that the locations are more global and includes various topographies. We can also see that improving weak labels using water occurrence map as described in Section 3.2 increases the model IoU by $1.7\%$ compared to its weak supervised counterpart and by $2.88\%$ compared to the Sen1Floods11 weak label baseline. This emphasises the importance of quality of training labels in producing more accurate models. Our cross distillation model further produces improves the quality of training label using a teacher model and performs better than all the other models. It exceeds Sen1Floods11 weak label baseline by $4.1\%$ IoU and improved weak label supervised model by $1.22\%$ IoU. Figure 7 shows a qualitative comparison of the cross distillation model with the baseline model.

Comparison with other methods

Benchmark comparisons of IoU on Sen1Floods11 test split are provided in Table 4. Note that even though our method is trained at input images with resolution of 16m, here we do an evaluation at 10m Sentinel-1 images and hand labels for a fair comparison with other methods. To get an inference output at a resolution of 10m, we first downsample the original image of shape $512\times 512$ to a resolution of 16m i.e. $320\times 320$, feed the downsampled image to the model and then upsample the probabilities to their original resolution.

From Table 4 we can see that our method outperform Sen1Floods11  [6] weak label baseline by a absolute margin of 6.53% IoU.

In the Deeplabv3+ [8] architecture, an input image is first passed to an encoder to generate semantically meaningful features at different scales. The final encoder output feature has an output stride of 16, i.e. the input image is down-sampled by a factor of 16. This feature is then passed through ASPP (Atrous Spatial Pyramid Pooling) layers and the decoder to produce the output logits. The decoder processes the encoder features at a list of multiple strides - a hyper-parameter that can be tuned to refine the output segmentation masks. At each stride, the low level features from the encoder having the same spatial dimensions, are concatenated with the decoder features to process them further. The smallest stride in the list refers to the final ratio of the original input image size with the final logits size. The final logits are then bi-linearly upsampled to the required input size. Table 5 shows the effect of the decoder stride on the validation split results. It can be seen that as we concatenate finer resolutions features from the encoder with the decoder features, the results improve. However, the rate of improvement declines. Considering practical considerations related to the number of model parameters, we decided to go with decoder stride (4, 2).

We also show the comparison of different loss function in Table 6 on the validation split. As can be seen from the table that vanilla cross entropy loss performs poorly. On the other hand, cross entropy with edge weighting improves vanilla cross entropy loss by $9.68\%$ IoU. Also Tversky focal loss outperforms edge weighted Cross Entropy loss by a margin of $1.91\%$ IoU. However, we observed that the water class probability outputs with Tversky focal loss were not calibrated. A neural network is said to be calibrated if its output confidence is equal to the probability of it being correct. Compared to Tversky loss that classifies both correct and incorrect water pixels with high confidence, Cross Entropy loss produces a more calibrated output. Because a calibrated output helps in getting a better estimate of the uncertainty of the model prediction, we decided to go with edge weighted Cross Entropy loss.