Road Feature Detection & GeoTagging with Deep Learning

How to utilize MMS data to identify and locate road features for automated, efficient asset management tasks.

Authors: David Yu, Hayley Miller

Disclaimer: This article makes use of data released from AZGeo with consent from the Arizona Department of Transportation. The full disclaimer can be found at the bottom of this article.

Overview

In this article we will go over how to:
1. Generate augmented training data for road features from MMS imagery
2. Perform object detection using Faster R-CNN and tie in Microsoft’s Computer Vision API for optical character recognition.
3. Infer an object’s latitude and longitude algorithmically, and then apply machine learning to enhance our prediction.
4. Perform clustering analysis using ArcGIS Pro to refine our outputs.
5. Visualize the results in the Operations Dashboard for ArcGIS
6. Use Collector for ArcGIS to integrate mobile field capability to the workflow.

So sit back and enjoy!

Preamble: Laying the groundwork for an ML approach to road feature detection

One of the fundamental promises of ML has always been the capacity of trained models to generalize to new data. The ease of extensibility of a well-trained model has in turn allowed for automation in both data collection and insight extraction/consumption. In the case of road features, the ability to extract actionable insights from existing repertoires of data such as geo-referenced photos and videos captured from Mobile Mapping Systems (MMS) would provide a low-cost solution to a variety of use cases common to many local government departments or private entities. These use cases include but are not limited to: inventory management, damage assessment and change detection.

We begin our investigation with MMS data because they come in a variety of formats, are easy to collect en masse and, in the case of pure imagery without video or LiDAR outputs, are relatively inexpensive to collect.

Of course when it comes to cost-shaving, an often overlooked avenue of data is crowd-sourced images, such as those available from Open Street Map. The downside to using these images is that they may not include the necessary metadata to perform accurate geolocation, come in a variety of non-standard formats and aren’t always available if a systematic sweep of a select route or region is needed.

Road features are any objects or points of interest that can be captured and identified from MMS data. They can be categorized as asset features and non-asset features. In the case of the former, depending on the owner of these assets, they can constitute items such as road signs, road damage, guard rails, hydrants, trees, roadside vegetation, vehicles, debris, etc. In this case study, Esri worked with the Arizona Department of Transportation (ADOT) to identify, geolocate and classify highway road signs as part of an inventory management demo.

We can break down the aforementioned solution into three distinct pieces in terms of architecture and functionality: detection/classification, optical character recognition (OCR) and geotagging.

Let there be Training Data

The AZGeo dataset consists of a photolog database of ~2.5 million images taken in the year of 2016, which extensively covers all the interstate freeways, state routes and streets of the State of Arizona. These images are taken from the same MMS platform and have a uniform resolution of 640×480 px.

The LabelImg library is a great open-source python library that provides a user-friendly Qt GUI for generating training labels in either Pascal VOC or YOLO formats. I used LabelImg to create 766 training images which took around 3 hours to do. For the purpose of this workflow, because we are only interested in detecting highway signs which don’t tend to involve a lot of variation, it suffices to use a set of weights for the backbone ResNet101 that was pretrained on ImageNet. For other use cases such as detecting street signs in an urban setting, it is highly recommend that you pretrain your network on the LISA dataset which contains a little over 6000 images of common urban and highway street signs at a mixture of resolutions. Unfortunately, the annotations that come with these images do not conform to any standard formats so you would have to write your own parser to handles these annotated files.

The problem with generating such a small set of training images is that most CV models would tend to overfit on these images, regardless of whether your model was pretrained or not. This problem becomes especially severe considering that you would have to set aside a certain portion of the training set for testing and generating evaluation metrics. To get around this problem, it has become standard practice in the CV community to apply various augmentation techniques to your initial set of images. The imgaug library is a great library to use and offers augmentation techniques such as affine transformations, random cropping, dropout, gaussian blurring and many more on bounding box, keypoints or segmentation mask labels alike.

Here’s the list of sequential transforms I used by the way, which are modified from the examples in the official imgaug documentation, which I found to work quite well for street signs. I recommend checking out the documentation on how to apply these transforms.

Detection & Classification

I will not cover too much material re: detection & classification here, suffice it to say that a Faster R-CNN network was used. Please refer to this article on how Faster R-CNN may be applied to perform parking lot occupancy detection. If you are interested, this article explores the architecture in greater detail.

In terms of performance, through experimentation we found that inferencing takes around 0.3 seconds per image, which seems quite fast until one takes into account that to cover just a single route (e.g. Interstate 15 in our example which has ~6600 images), the process could take up to 33 minutes.

A second option to performing detection is to apply the YOLOv3 network in order to perform detection which significantly reduces the inference time (down to 0.05 seconds per image, or a mere 5.5 minutes for Interstate 15).

Factoring in Optical Character Recognition

For the use case of sign detection, it may be advantageous for inventory management to also determine the texts of detected signs. This would allow for easier signage indexing or to perform text-based queries on these indexed signs. Traditionally, OCR tasks have always been built from the ground up using python libraries such as pytesseract. Unfortunately, these libraries were designed to parse scanned documents and therefore the user would typically need to perform various enhancement techniques prior to obtaining a somewhat usable input image, and even then, the results tend to be sub-optimal.

Luckily, Microsoft’s Computer Vision API simplifies this process greatly by being flexible enough to consume a multitude of image formats and styles while performing all the text detection, RoI enhancements and text cleaning processes under the hood.

The Computer Vision API produces the following text detection and bounding box information that can then be visualized over the original image.

However as great as this API is, its performance nonetheless falls short given low-resolution images, and doesn’t quite stack up against human vision especially in cases where text detection is often aided by environmental context (being a domain where human vision excels). This is when we have to go the extra mile in order to squeeze the remaining bits of information from these images.

Super-resolution is the standard technique to tackle situations where high-resolution (which does not automatically correlate with high-information-content) inputs are called for. We have carried out some intriguing applications of super-resolution in the past to tackle upsampling images for building damage classification or vehicle detection. Though super-resolution excels in the domain of photo enhancement, when we consider text images or scanned documents, we must remember to strive for image clarity and sharpness rather than fidelity to the underlying distribution of ground-truth pixels, which might otherwise produce a more realistic but blurrier image. With these constraints in place, it makes more sense to use traditional methods of image upsampling such as 2D nearest-neighbor, bilinear or bicubic interpolation. However, it should be noted that there is an amazing library out there called (I kid you not) waifu2x that performs super-resolution on anime-styled images which happens to work great for road signs. In fact, this website lets you input your own images and see the end result after being waifu2x’d and its a pretty good way to spend a Friday afternoon if you have nothing better to do.

The GIS twist that brings everything together: GeoReferencing

Up until this point, what we have built does not constitute a geo-enabled solution. If not utilizing all your available data to the fullest constitutes a crime in data science then not incorporating GIS elements given MMS data should be considered a cardinal sin, given the vast amount of geo-metadata that is embedded within each image. For the AZGeo dataset, the following information is contained within each image.

The lat/long coordinates are measured by GPS with a degree precision of 12 mm (though not necessarily accurate to within 12 mm of the target). This is all well and fine save for the thorny problem that these coordinates are not representative of the true lat/longs of objects of points of interest (PoI) contained within this image. This is not an easy problem to tackle, nor would a working solution ever be trivial because the most crucial piece of information needed is missing: how far a PoI is from the camera. Knowing this piece of information, one could easily arrive at an algorithmic solution. However even human observers can sometimes find it difficult to ascertain exactly how far away a thing is from the camera.

One thing we humans do have going for us though, is a built-in prior that shines a little context on the problem. We know a sign is roughly x meters away because we have a rough idea of how large road signs are in real life. However without a sense of scale guiding our judgement as in the picture below, even we would be at a loss.

This insight into how we humans judge distance gives us a starting point into tackling the problem of distance inference, if only we could determine how large an object is in physical space. A naive approach to this problem should be immediately obvious: simply perform classification on signage to a reasonable degree of granularity, and then assign a physical size to each sign class. Fortunately, the Federal Highway Administration has made publicly available a Manual on Uniform Traffic Control Devices (MUTCD) which precisely delineates the dimensions of each class of signs. The downside to this is that the regulation on dimensions of certain text signs such as destination signs (D1–3) and jurisdictional boundary signs (I-2) are only accurate to within a certain range or are only semi-bounded with no upper limit restrictions on size.

Nonetheless this gives us something to work with, and we can now quite accurately determine the physical sizes of non-text signs such as warning signs, speed limit signs, state route markers and mile markers. With this information, the distance of the sign from the camera can easily be calculated using similar triangles:

Finally, we must also take into account auxiliary information such as the bearing of the camera, which is made up of the bearing of the vehicle and camera yaw. We will for now, not consider other parameters such as camera pitch and roll and how they would contribute to changes in inferred distance.

More accurate distance inference with Deep Learning

The above approach to distance inference gave us a good starting point to improve upon, but is by no means foolproof. Apart from the issue of non-deterministic physical sizes on select classes of signs, we also have the issue of perspective distortion should a sign get really close to the camera. What we need to figure out then, is whether we can bake an additional prior understanding of size into the model itself. Where there’s a will, there is a way involving deep learning.

The idea of applying deep learning to solve distance inference is itself not very novel, but the approach taken by the monodepth library is a very refreshing one: Rather than train a convolutional neural net to minimize the loss between an inferred depth-map and some kind of ground-truth map informed by LiDAR data, monodepth uses a dual-camera set up to capture data for training. Back propagation in this case involves adjusting the network’s weights so that it is able to reproduce a right-lens image from a single left-lens input (and vice versa). The corollary is that the model builds up an intrinsic understanding of depth over time, and this is all achieved without any expensive data collection equipment.

To figure out which pixel value corresponds to which distance is then a simple regression problem and can be done automatically on the testing set where signs have a known location.

After integrating this ML solution for distance inference to our programmatic method, we are now able to precisely determine an object’s geocoordinates from a geotagged image. Neat!

Metrics, Metrics, Metrics

Of the five design aspects that define a machine learning workflow (data, model architecture, loss function, optimization method and performance measure), it is performance measure that is the most commonly overlooked yet most crucial facet. Throwing together a ML model without designing appropriate metrics would be like building a car and calling it a finished product without ever turning the key to see if it runs. The issue is that most developers are too lazy to test their models against a set of standard benchmarks, or to come up with custom metrics from the outset where standard performance measures are clearly insufficient.

To do this properly, I have created three separate metrics for each of the three outputs of the model: lat/long inference, OCR and detection/classification. But before any of that, we would first need to designate a set of testing data against which future benchmark tests would be carried out. There are two advantages to keeping a fixed test set: First, we can be assured that when performing model comparison that all non-systematic errors are eliminated. Second, depending on where we wish to deploy the model, we could manually incorporate images from that testing space to assess the model’s ability to generalize to foreign environments. The only downside to a fixed testing set is that lat/long of PoIs would have to be manually assigned, which we have painstakingly done over a period of two hours. As it stands, the current testing set consists of 43 images which does not sound like a lot but is plenty to illustrate the current level of model performance.

Lat/Long Inference
Two pieces of data are collected to measure the accuracy of lat/long inference. First, the L-2 standard deviation of inferred locations. This piece of information lets us understand the spread of inferred coordinates, and is something we wish to reduce over time. The other is the probability distribution of bearing of line AB where A is the ground-truth location and B is the inferred location. Ideally, the bearing should have an even distribution, indicating a lack of systematic errors which might otherwise manifest as clusters around certain bearing values.

OCR
Two metrics are used to assess the OCR accuracy: Number of words detected (each associated with the inferred signage distance) and the percentage of correctly identified words for all detections (based on a similarity index produced by Python’s SequenceMatcher). The ground-truth data in this case are manually entered into a python dict. The implicit rule I used when writing these strings is that I must not spend more than a second to interpret the text from a sign, and that I should certainly write out an empty ground-truth string if I deem the text to be indecipherable. Nonetheless, in the case of OCR, the model did not produce very accurate outputs for the few signs that contained any text at all.

Detection/Classification
This is one of the few cases where a standard metric does the job better than any metric you can concoct. Since the detection model consists of Mask R-CNN or YOLOv3, the recommended performance measure is mean average precision (mAP). A detailed explanation of what this entails is given here.

Integrating into the ArcGIS framework

The model in its current iteration is capable of consuming mass amounts of raw data and converting those into useful features. It is however not a full-fledged solution because there are no mechanisms in place for an end-user to easily consume these features and hence cannot derive actionable insights. This is where an ArcGIS end-to-end solution offers the user a decisive advantage through offerings such as real-time status monitoring using Operations Dashboard, post-processing techniques such as pattern analysis, spatial relationship modelling and clustering methods exposed through various spatial statistics tool sets, as well as ground crew management and coordination platforms like Workforce or Collector.

On the topic of post-processing, we realized that a pitfall with using MMS data is that unique objects tend to get identified multiple times from different camera frames as the MMS vehicle approaches the PoI. This would not be an issue if lat/long inference or object classification can be carried out without error, however this is typically not the case.

Spatially constrained multivariate clustering is a great tool available as an ArcGIS Pro geoprocessing tool (alternatively available as an Arcpy function) which solves our problem. This tool allows one to specify which layer attribute to apply a constraint. In our case, we want to constrain clustering to only those points that share the same class label, and to only apply clustering within a Euclidean distance region from each point.

Operations Dashboard for ArcGIS

The Ops Dashboard for ArcGIS allows the user to use charts, gauges, maps and other visual elements to monitor geo-enabled assets. For the purpose of this demo, we partnered with ADOT to make a Dashboard to illustrate the detected road sign inventory along Interstate 15 NB in Arizona.

Collector for ArcGIS

The integration with Collector for ArcGIS brings a mobile field capability to the workflow. We imported the sign detection feature layer into a web map and added relevant attribute fields to the dataset. Crew members can add information to the dataset and verify machine learning processes in the field. In this instance, we wanted to verify the location of the georeferenced points and the validity of the sign classification, so we added those fields. We used coded domains to provide field workers with a pre-determined drop-down list of attributes. The ability to choose from the drop-down list prevents errors introduced by fat-finger typing and ensures better data quality and assurance. Additionally, the fieldworkers can use voice-to-text capabilities to report issues with detection, sign damage, or changes to the dataset. This, too, avoids fat-finger errors as well as saves time. Another benefit of the application is its offline capability that allows offline changes to be synced to the dataset when service is restored. This workflow streamlines the process of updating, editing, deleting, and adding points in the field.

These field updates were monitored in near real-time using an Operations Dashboard, as shown above.

Field editing was enabled as well as the ability to sync data and download map areas for offline data collection and updates.

The Collector for ArcGIS app allows the end-user to view the photos of each sign, as well as the location, and data gathered from the machine learning techniques above.

Future Work

This demo project is very much a work-in-progress, and as such, there are elements of functionality which we wanted to integrate into this solution but have not had the time to do so. Expect an updated blog in short order. In the meantime, we will continue to seek innovative ways to enhance our model and drive for better accuracy.

Full disclaimerTo the maximum extent permitted by law, the liability of ADOT, its employees, officials, and agents shall not include liability for lost profits, indirect, incidental, special, punitive or consequential damages, or claims from third parties to the public, or any loss of business, revenue or data, whether based upon a claim or action of tort, contract, warranty, negligence, strict liability, breach of statutory duty, or any other legal theory or cause of action, even if advised of the possibility of such damages. In additional to the disclaimers above, to the fullest extent permitted by law, any user of the AZGeo data portal agrees to release, and hold harmless the State of Arizona, its employees, officers, officials, and agents from and against any and all claims, actions, liabilities, damages, losses, or expenses (including court costs, attorney’s fees, and costs of claim processing, investigation and litigation) arising out of the performance of this agreement. : Although the Arizona Department of Transportation’s data in AZGeo has been produced from sources believed to be reliable, no warranty expressed or implied is made regarding accuracy, adequacy, completeness, legality, reliability or usefulness of any information. This disclaimer applies to both isolated and aggregate uses of the information. The Arizona Department of Transportation provides this information on an “AS IS” basis. All warranties of any kind, express or implied, including but not limited to the implied warranties of merchantability, fitness for a particular purpose, freedom from contamination by computer viruses and non-infringement of proprietary rights are disclaimed. Changes may be periodically added to the information herein; these changes may or may not be incorporated in any new version of the publication. If the user has obtained information from the AZGeo webpage from a source other than the AZGeo Home Page, the user must be aware that electronic data can be altered subsequent to original distribution. Data can also quickly become out-of-date. It is recommended that the user pay careful attention to the contents of any metadata associated with a file, and that the originator (MPD GIS) of the data information be contacted with any questions regarding appropriate use. If the user finds any errors or omissions, we encourage the user to report them to the Arizona Department of Transportation GIS team at MPDGIS@azdot.gov.

Road Feature Detection & GeoTagging was originally published in GeoAI on Medium, where people are continuing the conversation by highlighting and responding to this story.

A preliminary survey into an object detection workflow using machine learning combined with GIS technologies

The Era of Drones

Although the idea of vehicle detection is not a groundbreaking one and has been around since the emergence of video cameras and embedded sensors, these methods were often marred by high capital and maintenance costs and a high complexity from having to integrate multiple data sources, each with a limited band of inputs. The prevalence of drones in the commercial market in recent years on the other hand, has brought about a new era of state-of-the-art aerial photogrammetry and a drastic reduction in the cost of obtaining aerial data. With this sudden increase in information, and by combining machine learning with GIS technologies, we are now capable of performing new and insightful analyses on issues of interest.

Existent business problems which stand to benefit from this include customer flow analyses and demographic modelling. This is particularly useful for those in the retail sector looking to monitor peak business hours by counting the number of parked vehicles at a given time and also extrapolate useful customer information (such as income, marital status, household size and even political inclination) by classifying the types of vehicles they own.

So, can we solve these problems using AI and GIS? The answer is yes, and a good starting point would be to come up with a workflow that tallies the number of cars per unit time as well as infer the vehicle type for every positive detection. In this article, I aim to give a comprehensive overview of a such a workflow — from data acquisition and processing using Drone2Map to performing data inferencing using TensorFlow and ArcGIS Pro, and finally to creating actionable BI visualizations using The Operations Dashboard in ArcGIS Online (AGOL).

Data Collection & Exploratory Analysis using Drone2Map

To obtain some sample data, we flew a drone over a busy parking lot here at our office in Redlands, California and obtained a series of geo-tagged tiff files with geolocation corroborated by 5 ground control points (GCPs) to ensure the result would be accurate enough to identify the correct parking space for each vehicle. These images were captured along a “lawn mower” flight path with an overlap of 70% along flight lines and at least 60% between flight lines. The reason we do this is to facilitate the generation of true orthomosaics. Traditional orthos (or “frame” orthos) suffer from what is known as the “layover effect”, where tall structures such as trees or buildings seemingly “lean” toward or away from the observer as a consequence of stitching together disparate frames that do not capture objects at a true nadir perspective. This effect worsens for objects that are at the edges of a drone’s field of view.

Using Drone2Map, we can take these individual frames and resolve for an object’s true ortho by finding common views between a frame and its 8 adjacent frames in a point cloud and then keeping the views that have a high degree of overlap. The resultant orthomosaic is not only a true ortho, but one that does not reveal seamlines between images typical of frame orthos.

Of course, all of this is automated, and the actual image processing step is simple: Create a 2D mapping project in Drone2Map, pull in your data, add ground control points as needed and then hit start to generate a true orthomosaic.

From this, we obtain 3 classes of output products: A 2D orthomosaic of our parking lot, a digital surface model (DSM) layer and a digital terrain model (DTM) layer. An initial thought was to simply pass the DSM to a detection network to produce bounding boxes on distinctly “car-like” protrusions. However, on closer inspection of the dataset we identified some potential issues with this approach: in this particular parking lot, the coverage of foliage was so extensive as to affect the detection of certain cars partially or completely hidden by overhanging branches and leaves.

The overhanging vegetation affected both the DSM and orthomosaic, but since the edge of each image includes oblique view angles at the image edges, some images were able to view partially or completely underneath the tree canopy. ArcGIS also enables each image from the drone to be orthorectified. Following photogrammetric processing in Drone2Map, each image could be analyzed in its proper geospatial placement, providing multiples views of each parking space.

Processing of these oblique views was beyond the scope of this initial project, but will be the subject of future testing. In addition, parts of the canopy depicted by the orthomosaic are not fully opaque, and the RGB bands may also provide additional channels that would allow correct subdivision of vehicles into categories such as trucks, sedans and SUVs. From a data collection standpoint, it is also much easier to collect pure ortho imagery than outfitting drones with LIDAR sensors for DSM/DTM detection.

Simple Approach: Building a classification model using InceptionV3

Our first solution for tackling this problem was an obvious one: Simply overlay a polygon layer from a mapped parking lot on top of the orthomosaic raster and clip out cars using the Split Raster geoprocessing tool to get our prediction set. This was very easily done.

Then comes the question of which classification model to apply atop which finetuning set. A simple off-the-shelf model that’s available from both TensorFlow Slim and TensorFlow Hub is InceptionV3. Based off the original InceptionNet (Szegedy et al.), this third revision bears much resemblance in terms of core structure to the original with similar component modules. However, it has the addition of factorization methods to reduce the representational bottleneck as well as label smoothing and batch norm operations on the auxiliary classifiers to increase regularization.

As with most TensorFlow Hub models, there is no need to train from scratch when we can apply transfer learning; Luckily, InceptionV3 was pretrained on ImageNet.

As for the finetuning set, The Cars Overhead with Context (COWC) dataset from LLNL was an easy pick for its richly annotated set of 32,716 vehicles as well as hard negative examples (boats, commercial vehicles etc.). Although the dataset doesn’t plug straight into the classification network, the only preprocessing work here involves reading through the list of the annotated text files and cropping the associated jpegs with either OpenCV or ImageMagick. (N.B. the COWC dataset has 4 class labels: Sedan, Pickup, Others and Unknown. With an additional background class that makes 5 classes. I had to play around with class balancing to ensure each class was sufficiently represented, and also mine some samples for background classes which weren’t provided in COWC).

These images have been extracted from the original COWC format which is comprised of large 22000x22000 images with bounding box information contained in ancillary .txt files. One thing to note is that the original bounding box coordinates tended to crop off large sections of the vehicles in question, which ultimately lead to worse overall performance in identifying the pickup class. Therefore these images have all been cropped using larger 34x34 bounding boxes.

By looking at the COWC dataset we can immediately tell that the resolution of the dataset is not ideal. Early training results have shown that the classification network performed exceedingly well in binary classification tasks of identifying occupied/unoccupied spaces, but performed worse in distinguishing between vehicle classes. Part of the reason for this is also due to the fact that the pickup class was severely underrepresented. Testing proved that undersampling the sedan/other/background classes yielded the best validation accuracy (as opposed to oversampling the pickup class).

Once the model was sufficiently trained (on a good GPU this takes a couple hours — for me it was a Tesla K80 on the GeoAI VM for about 1.5 hours at a training accuracy of 0.85 and a validation accuracy of 0.84), we can proceed to apply our previously extracted prediction set to the model.

Output these results into a .csv file (it might be useful to apply some smart naming conventions here to ensure your data items match the OBJECTIDs of each polygon in the parking space feature layer. From here, simply combine the two layers using Add Join and voilà, you have a polygon layer that associates a class probability with each parking space based on an aerial image you took.

Thus far, we have only created some rich geotagged layers that are not yet informative nor intuitive enough give any kind of analytical insight. This is where ArcGIS Online offers us a path forward: we export both the orthomosaic as well as the feature layer to our ArcGIS Online Portal, then optionally in the MapViewer, modify the symbology of our polygon layer to be attribute-driven.

You can then import individual maps and visualize the results interactively on the Operations Dashboard:

We scheduled two drone flights over the same parking lot at different times of the day. To complement this data, we have also generated some mock input to illustrate occupancy info on an hourly basis that simulates customer flow within a typical work day. The Hour Selector cycles through parking patterns over every time segment. To suit your use case, you may also decide to deploy a drone every few weeks or every few months. The Average Occupancy Heatmap is a visual representation of “hot spots” wherein vehicles aggregate. If you like it is also possible to generate a “Turnover Heatmap” that maps regions in which vehicles are likely to park for longer/shorter periods. Both of these views are potentially useful for understanding demographic behavior when crossed-referenced with vehicle types or to detect favorite stores & customer stay-times at shopping venues.

The two useful vehicle categories from COWC: sedans and pickups, are shown in the Vehicle Types visual element, each color coded and also with a transparency level linked to its classification confidence. Finally, pure vehicle counting/detection is presented in the Current Vehicle Classification Analysis view with a gauge to show the current occupancy.

Shortcomings of a classification-only model

Our blind assumption for a classification-only approach is that all cars fit neatly inside each parking space polygon (failing to take into account bad drivers, double-parkers or your regular F150s so easily cropped off by the sensibly-sized parking spaces). Of course, there are other use cases for vehicle detection for which a predefined polygon layer is simply impossible to draw (think roadside parking or parking lots for which there are no guidelines). These coupled with the fact that a simple classification network is simply not “smart enough” prompted us to think of another approach to this problem.

Better Approach: Building a detection model using Faster-RCNN

The fundamental idea behind a Faster-RCNN network is that it does two things at once: It detects the bounding boxes of objects of interest using a Region Proposal Network (RPN), and performs classification on those detections using a base classifier after region of interest pooling (ROI).

Like our previous attempt, we used COWC for fine-tuning. This time, there is no need to extract individual vehicles for training. To visualize what the dataset looks like, you can test out the following snippet by replacing the (x,y), width and height values in the patches.Rectangle() method with whatever value is shown in an image’s corresponding .txt file.

I also had to manually convert these images into Pascal VOC format to be consumed by the Faster R-CNN model. Unlike the classification model, training this model took roughly 24 hours on the K80. This is because in Faster R-CNN, anchor generation within the RPN forms a bottleneck (potentially generating up to 6000 region proposals per image). Other detection models such as SSD or YOLO (at least the first generation) ameliorate the speed issue at the cost of lower mAP scores. However, seeing as we are not concerned with near real-time detection, it suffices to use Faster R-CNN to produce a good output at reasonable speeds (~7 seconds or so on the CPU for each image tile of size of 600x600 which is rescaled from the original 5000x5000 px). Again, Faster R-CNN is provided in the TensorFlow models library. There you can also find some default configuration files for the base classifier you have chosen (ResNet101 pretrained on the MS COCO dataset in my case). There are in fact a whole slew of pretrained base models from TensorFlow’s detection model zoo you can choose for yourself, each with their own speed/mAP trade-off.

This is part of the configuration file I used, note the height and width values that match the COWC “patch_size”, or size of each input image tile. This can be configured in the CreateDetectionScenes.py script from COWC’s Github page. Additionally you can choose to base your classification model on something other than ResNet101. Depending on the size of your detection it may also be prudent to adjust the scales of your anchor for faster convergence, although I find it much more effective to re-tile your inputs if each object ends up comprising less than 4% of your input size.

You can of course write your own evaluation script to visualize the trained model. There is in fact a very good template on TensorFlow’s Github page. I made some modifications to the following snippet to also allow you to adjust the detection threshold and the number of boxes to draw which I find very useful in visually understanding the performance of your model early on in the training process:

Alternatively, if you wish for a more straightforward approach to performing inferencing, the upcoming ArcGIS Pro 2.3 offers a convenient geoprocessing tool “Detect Objects Using Deep Learning” to perform evaluation on any input raster by passing it a trained model in the form of a frozen inference graph protobuf defined inside a model description JSON file. As well as several evaluation hyperparameters such as padding and detection threshold. Hit run and you get a new feature layer of bounding boxes in return.

Similar to the classification approach, we can visualize these layers much more effectively by exporting them to ArcGIS Online and viewing them in the Operations Dashboard.

All the views from the classification dashboard can also be represented here with the same analyses drawn, but now vehicles parked outside of predesignated spaces (including vehicles in motion) are all detected using Faster R-CNN.

Closing thoughts and future work

This article serves as an exploratory glimpse into machine learning-driven GIS for vehicle detection, and of how important business decisions can be informed from start to finish by leveraging the powerful Esri ecosystem to create a complete BI workflow.

The models described above can of course be taken a couple steps further, by training on an input set with richer annotations or cross-referencing the COWC dataset with datasets that tie together vehicle types with income and other demographic info to produce even more powerful analyses, reveal subtler patterns and yield finer customer segmentations. Likewise, these models apply to aerial detection of all kinds (crops, utility poles, animals, forest fires, ships), as long as you have access to a decent GPU and a fine-tuning dataset.

Another potential expansion to this pipeline would be to introduce oblique imagery into the mix in a manner similar to what was done in Esri’s Oblique Viewer App that allows for multiple oblique views to be attached to the same ortho frame. For vehicle detection this effectively increases the number of images from which we can extract high fidelity data and also gives an unobstructed view of cars hidden under tree canopies.

Hopefully this has been an interesting read, please give us a clap and share this post if you enjoyed it, and let us know in the comments what other insights can be drawn from these data and whether you think there’s a better approach to be considered!

This effort was done as part of the Esri GeoAI team. For other cool GISxML projects, check out the GeoAI Medium page here.

Parking Lot Vehicle Detection Using Deep Learning was originally published in GeoAI on Medium, where people are continuing the conversation by highlighting and responding to this story.