Object Detection Pitfalls: Fixing Data, Labels, Models & Metrics

The detector looked ready.

On the validation set, it performed well. The loss curves were smooth, the demo video was convincing, and most bounding boxes appeared exactly where they should. Then the system was installed in a warehouse, where it began missing forklifts near the loading bay at dusk.

The first reaction was predictable: use a larger model.

That produced a modest benchmark improvement, but the field failures remained. A closer inspection uncovered several less glamorous problems. Frames from the same video sequences had been divided between training and validation, making the validation set easier than it appeared. Partially hidden forklifts had been labeled inconsistently. The deployment service resized images differently from the training pipeline. Non-maximum suppression was also merging two nearby forklifts into a single detection.

The model was not the only problem. It may not even have been the main problem.

That is the frustrating truth about object detection. A detector can fail because of the data, annotation policy, preprocessing, model design, post-processing, evaluation code, or production environment. All of those failures eventually appear on the screen in much the same way: a missing, misplaced, duplicated…