Weakly Supervised Segmentation: How Far Can a Few Labelled Pixels Take You?
How spreading sparse point labels yields 98% of full-mask performance on aerial images
Semantic segmentation assigns a class to every pixel in an image. The standard way to train it is with a full mask for every training image, i.e. a ground truth where every pixel already has its class. Producing those masks is slow, because someone has to trace the outline of every building, tree and car.
A much cheaper form of annotation is to click a few points on each thing in the image. This is point supervision, one of several forms of weakly supervised semantic segmentation, and Bearman et al. found it took about a tenth of the annotation time of full masks on PASCAL VOC. I wanted to know how far a model gets when the loss only sees those few labelled pixels, and what makes the most of them.
I ran this on the ISPRS Potsdam aerial dataset, which has six classes: impervious surfaces, building, low vegetation, tree, car and clutter. I varied four things: the number of points, the loss, spreading labels with superpixels, and where the points are placed. One of them clearly helped and one clearly hurt. Most of the rest were hard to separate from the variation between identical runs. To see how much the missing pixels cost, I also trained the same model on full masks.
The code and results are on GitHub.
The setup
The model is DeepLabV3+, a standard encoder-decoder segmentation network, with an ImageNet-pretrained ResNet-50 backbone from segmentation-models-pytorch. It trains on 2,400 patches of 300×300 pixels, split 70/15/15 into train, validation and test, for 50 epochs with Adam at a learning rate of 1e-4. All scores are mean intersection over union (mIoU) on the test split, measured against the full ground-truth masks.
The model still predicts every pixel. The trick is in the loss, which is only computed at the labelled pixels:
p_t = probs.gather(1, targets.unsqueeze(1)).squeeze(1)
focal_weight = (1.0 - p_t) ** self.gamma
pixel_loss = -focal_weight * torch.log(p_t + 1e-8)
loss = (pixel_loss * point_mask).sum() / point_mask.sum().clamp(min=1)p_t is the probability the model gives the correct class. The focal weight reduces the loss on pixels the model is already confident about, and γ controls how strongly. At γ = 0 this is plain cross-entropy. point_mask is 1 at labelled pixels and 0 everywhere else, so unlabelled pixels contribute no gradient.
Points are simulated by sampling from each class in the ground-truth mask. Each training image gets its points once, before augmentation, and keeps them for the whole run, as it would with a human annotator. The densities are nested, i.e. the 10-point labels include the 5-point labels, which include the single point, so density comparisons differ only in the number of points. Not every class appears in every patch, so at 10 points per class an image has about 27 labelled pixels out of 90,000, roughly 0.03%.
To tell real differences from noise, I trained the reference setup (10 points per class, γ = 2, uniform sampling) three times with the same points and different training seeds. The runs scored 0.560, 0.559 and 0.544. So a gap of under about 1.5 points between two single runs tells us nothing, and I treat it that way below. Per-class IoU varies by up to 3 points between those runs.
More points help, with diminishing returns
With 1, 5, 20 and 50 points per class, the model scored 0.491, 0.551, 0.569 and 0.572. The reference, at 10 points, sat between 0.544 and 0.560.
A single point per class is clearly not enough. By 5 points the model is already inside the reference band. Going to 20 or 50 adds only about a point above the top of the band, and 50 is no better than 20, so most of the gain comes from the first few clicks.
Focal weighting: no measurable effect
With 10 points per class, γ values of 0, 0.5 and 1 gave 0.561, 0.564 and 0.556, against 0.544–0.560 at γ = 2.
All four settings are within about half a point of the reference band, so I can’t separate any of them from noise. With so few labelled pixels per image, down-weighting the easy ones doesn’t seem to change much. Plain cross-entropy is a fine default here.
Superpixel label propagation: the one clear win
Each labelled point supervises one pixel. But the pixels around it usually belong to the same class, since a point on a roof is mostly surrounded by roof. So the next experiment spreads each point’s label to its neighbourhood.
It uses SLIC, which splits an image into a few hundred small regions of similar colour, called superpixels. Each point’s label is copied to every pixel in the superpixel it lands in. At 200 superpixels per image, each point then labels about 560 pixels instead of one. The expanded labels act as pseudo-labels for the unlabelled pixels around each point, without a separate stage to generate them.
This only helps if most superpixels contain a single class. Across 50 images, the median superpixel was essentially pure, i.e. all its pixels shared the point’s class, and 74% were at least 80% pure. Mean purity was lower, at 0.84, because a minority of superpixels straddle a class boundary.
With the same 10 points per class, SLIC expansion scored 0.603. That’s 4.3 points above the best reference run, and 3 points above the 50-point model (0.572). The only extra cost is running SLIC over each image.
Get Sijuade Oguntayo’s stories in your inbox
Join Medium for free to get updates from this writer.
Every class improved beyond the per-class spread of the reference runs. The largest gain was on cars, from 0.62–0.65 IoU to 0.71. A car is a compact object of fairly uniform colour, so it fits neatly into one or two superpixels.
Cleaner labels were not better labels
The last experiment asked where points should go: deep inside objects, or near their edges.
A point inside a region has clean surroundings, e.g. a point in the middle of a road is surrounded by road. A point near an edge has mixed surroundings, but it sits where the model has to make its hardest decisions. I tried three sampling strategies: uniform, biased towards boundaries, and biased towards interiors. To check they did what they claimed, I measured local purity, the share of an 11×11 patch around each point that has the point’s class:
- Interior: 0.974, with 11% of patches containing a second class
- Uniform: 0.918, with 31%
- Boundary: 0.848, with 51%
If cleaner labels made better models, interior sampling would win. It came last, at 0.532, about a point below the lowest reference run. Boundary sampling scored 0.553, inside the reference band, so it didn’t beat uniform either.
My reading is that points deep inside large regions are mostly redundant. The model would get the middle of a large roof right anyway, so a label there adds little. Purity measures how trustworthy a label is, which isn’t the same as how much the model learns from it. Uniform sampling is a sensible default, and interior-biased sampling is worth avoiding.
How close this gets to full masks
To put these scores in context, I trained the same model with the same settings on the full masks, so every pixel contributes to the loss. It scored 0.617.
Ten points per class, about 0.03% of the pixels, reach 90% of that score. Fifty points reach 93%. With SLIC, the same 10 points reach 98%, and every class ends within about 3 points of its full-mask IoU. The remaining gap, 1.4 points, is smaller than the spread between the reference runs, so with one run each I can’t tell the SLIC model and the full-mask model apart.
What I’d take away
The results that hold up are:
- Superpixel expansion was the biggest improvement, larger than using five times as many points, and brought 10 points per class to within noise of full masks.
- Interior points, the cleanest labels, gave the worst model at 10 points per class.
- More points help up to about 5 per class. Beyond that, the gains are around a point.
- Focal weighting made no difference I could measure.
Clutter was the weakest class everywhere, never above 0.16 IoU even with full masks. It’s a catch-all for everything that isn’t one of the other five, so there’s no consistent appearance to learn.
Limitations
- One set of points. Every run uses the same sampled points. A different draw would move the scores, and I haven’t measured by how much.
- One run per condition, except the reference setup. This includes the full-mask model, and gaps under about 1.5 points shouldn’t be trusted.
- Simulated annotators. Points come from ground-truth masks. Real annotators click differently, so the placement experiment is a proxy for their behaviour rather than a study of it.
- Fixed learning rate. Validation mIoU was still oscillating in the last epochs, and a scheduler might raise every score.
Notes
- Bearman et al., “What’s the Point: Semantic Segmentation with Point Supervision”, ECCV 2016.
- Code, configs and per-run results: https://github.com/cydal/partial-segmentation
- Lin et al., “Focal Loss for Dense Object Detection”, ICCV 2017.
- Achanta et al., “SLIC Superpixels Compared to State-of-the-Art Superpixel Methods”, IEEE TPAMI, 2012.
