<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Amin Tayyebi on Medium]]></title>
        <description><![CDATA[Stories by Amin Tayyebi on Medium]]></description>
        <link>https://medium.com/@amin.tayyebi?source=rss-7d56687076d8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*mEsiuzTZBleqyhlq.</url>
            <title>Stories by Amin Tayyebi on Medium</title>
            <link>https://medium.com/@amin.tayyebi?source=rss-7d56687076d8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 09:57:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@amin.tayyebi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Automating Railway Asset Detection using Deep Learning]]></title>
            <link>https://medium.com/geoai/automating-railway-asset-detection-using-deep-learning-64d14ad66e39?source=rss-7d56687076d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/64d14ad66e39</guid>
            <category><![CDATA[gis]]></category>
            <category><![CDATA[asset-management]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[lidar]]></category>
            <dc:creator><![CDATA[Amin Tayyebi]]></dc:creator>
            <pubDate>Tue, 17 Sep 2019 13:56:45 GMT</pubDate>
            <atom:updated>2019-10-27T18:34:22.540Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ucOptU0b63QqEXzWCvielw.jpeg" /></figure><h3>Automating Railway Asset Detection using LiDAR and Deep Learning</h3><p>An overview of extracting railway assets from 3D point clouds derived from LiDAR using ArcGIS, the ArcGIS API for Python and deep learning model<em>.</em></p><p><strong>Outline:</strong></p><p>1. Asset Inventory Management in Railway<br>2. Deep Learning from 2D to 3D Domain<br>3. Study Area and Data Preparation<br>4. Train PointNet for Classification<br>5. Inference Workflow<br>6. Conclusion and Future Works<br>7. Acknowledgment and References</p><p><strong>1. Asset Inventory Management in Railway</strong></p><p>Asset inventory management for railway (e.g., railway, milepost, crossing, signal, switch, etc.) requires comprehensive surveying and modern asset management. This task is going to be much harder when it comes to the new railway. GIS plays a critical role to make asset management more effective by providing a platform for storage and management of location data across time as well as serving as a decision support tool for better decision‐making. Besides inventory management of existing assets, large railway companies are also interested in 1) updating the existing inventory with new assets that are placed by US Department of Transportation and 2) dropping the old assets from inventory that are removed from the railway by US Department of Transportation. This is a very difficult task as large railway companies have lots of long railways (~over 10K mile) across the entire US.</p><p>During the last few years, most of the large railway companies started to capture LiDAR data along the railway. LiDAR provides fast, reliable, highly dense and highly accurate data for assets mapping. LiDAR data are usually captured in discrete patches and later registered to get a complete 3D point cloud of the railway. The ability to collect RGB values from 360 Ladybug along with intensity makes this technology most relevant for training deep learning models. Most of the large railway companies currently have a large number of technicians that go through 3D point clouds to detect their desired assets manually and label them as a point feature class using GIS software such as ArcGIS Pro. However, this operation is very tedious and time-consuming at a large scale (e.g., entire the US). GeoAI team in ESRI was asked to automate feature extraction (e.g., signal, switch, crossing, milepost) from 3D point cloud (Figure 1).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f5wQUhMn7biXefLgg2ec5g.jpeg" /><figcaption>Figure 1. Examples of Assets (Signals, Crossings, Mileposts, and Switches) in 3D Point Clouds as well as Imagery</figcaption></figure><p><strong>2. Deep Learning from 2D to 3D Domain</strong></p><p>There are lots of studies that applied deep learning in 2D domain for a variety of applications such as classification, segmentation, change detection, localization, recognition and scene understanding with notable successes. Most of these networks use convolution neural networks (CNNs) that enable them to progressively learn discriminative hierarchical features of the large training data. While most works in deep learning focus on regular input representations like sequences (in speech and language processing), images and volumes (video or 3D data), applying deep learning on the 3D space has not been as effective as 2D even with the latest advances in 3D sensing technologies and the increased availability of affordable 3D data. This can be due to a variety of reasons such as lack of a large amount of 3D labeled data, lack of complex deep learning models in 3D space, the variation of the structure and the geometric properties of 3D data, and complex task of learning from 3D geometry.</p><p>Application of deep learning models in 3D domain can fall in four various groups:</p><ol><li>Multi-View CNNs that have tried to render 3D point cloud into 2D images and then apply 2D CNNs to classify them. Although such methods have achieved good performance on shape classification, it’s difficult to extend them to 3D tasks such as point classification or point segmentation.</li><li>Volumetric CNNs are the pioneers applying 3D CNNs on voxelized shapes. However, volumetric representation is constrained by its resolution due to data sparsity and the computation cost of 3D convolution.</li><li>Spectral CNNs use spectral CNNs on meshes. However, these methods are currently constrained on manifold meshes such as organic objects and it’s not obvious how to extend them to non-isometric shapes such as furniture.</li><li>Feature-based DNNs convert the 3D data into a vector, by extracting traditional shape features and then use a fully connected net to classify the shape. However, they are constrained by the representation power of the features extracted.</li></ol><p>Here, I used PointNet as a deep learning model to detect railway assets from 3D point clouds. PointNet could be considered a feature-based DNN, but the features are learned implicitly by the network through a series of affine transformations and feature-wise fully connected layers.</p><p><strong>3. Study Area and Data Preparation</strong></p><p>I had access to the centerline of the railway in three cities in of the States in the United States (confidential). I also received point clouds along the three railways and the location of the four assets: mileposts, crossings, signals, and switches in a point feature format.</p><p>As the point clouds were not labeled, I could not use PointNet for segmentation. Thus, I had to discover a way to implement PointNet for classification. Finding the best way to process and prepare training data for PointNet as a classification model was a huge obstacle. As each railway contains over billion points, I had to filter the extra points out and keep only critical 3D points for the training run.</p><p>I followed various steps to prepare data for the training. First, I had to clip point clouds both along and perpendicular to the railway (3D boxes). As desired assets fall within 15m buffer of railway center, I had to generate 15m buffer along and perpendicular to the railway (15m×15m box). To achieve this, I used “Generate Points Along Lines” geoprocessing tool with a distance interval of 15m in ArcGIS Pro to create points along the railway [<a href="https://pro.arcgis.com/en/pro-app/tool-reference/data-management/generate-points-along-lines.htm">1</a>]. I then used the “Split Line at Point” function in ArcGIS Pro to split the railway using points generated from the “Generate Points Along Lines” tool [<a href="http://desktop.arcgis.com/en/arcmap/10.3/tools/data-management-toolbox/split-line-at-point.htm">2</a>]. The lines along the railway with 15m length are the output of the previous step. I then applied the buffer tool with the 15m distance to these lines to generate 15m×15m boxes (Figure 2) along the railway [<a href="http://desktop.arcgis.com/en/arcmap/10.3/tools/analysis-toolbox/buffer.htm">3</a>].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/699/1*nIkkBnEMQKkT84uEClMnaA.jpeg" /><figcaption>Figure 2. 15m×15m Boxes Along the Railway</figcaption></figure><p>Second, I prepared positive and negative samples for the training run. Positive samples refer to the boxes that contain at least one or more of the desired assets; however, negative samples refer to the boxes that do not contain any of the desired assets. To generate positive and negative samples, I performed a spatial join between the target feature, which is the layer with 15m×15m boxes (Figure 2), and the joined feature which is the labeled assets layer in point format [<a href="https://pro.arcgis.com/en/pro-app/tool-reference/analysis/spatial-join.htm">4</a>]. As the number of the negative samples were much larger (over 100 times) than positive samples, I sampled 10% of the total negative samples (Figure 3).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/504/1*680vsufw64HPjUhs1afFfA.jpeg" /><figcaption>Figure 3. Positive and Negative Samples Along the Railway</figcaption></figure><p>Third, I used the “Extract LAS” tool to keep point clouds (Figure 4) within the generated 15m×15m boxes (Figure 3) with positive and negative samples [<a href="https://pro.arcgis.com/en/pro-app/tool-reference/3d-analyst/extract-las.htm">5</a>]. As the attribute table of 15m×15m boxes contain the class of each asset falls in each box, I used it (Figure 4) to keep track of positive and negative samples while exporting point clouds to separate LAS file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B4hhiVtyLVO01_ZwH2bGkg.jpeg" /><figcaption>Figure 4. Examples of the Output of the “Extract LAS“ Function for Positive Samples within 15m×15m Boxes</figcaption></figure><p>Fourth, as my desired assets are off the ground, to filter out more unnecessary points within each box, I used “Classify LAS Ground” tool to classify point clouds to the ground and non-ground points [<a href="https://pro.arcgis.com/en/pro-app/tool-reference/3d-analyst/classify-las-ground.htm">6</a>]. I then dropped ground points within each 15m×15m box from further analysis (Figure 5).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*Ha2hTFMjrkynIhJFEkKneQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*T2Ki7ZUKdmd7-BX42EXcpQ.jpeg" /><figcaption>Figure 5. Examples of the Ground and Non-Ground Point Clouds in each 15m×15m Box</figcaption></figure><p>Fifth, I then used the “Classify LAS Noise” tool in ArcGIS Pro to drop anomalous or noises from further analysis [<a href="https://pro.arcgis.com/en/pro-app/tool-reference/3d-analyst/classify-las-noise.htm">7</a>]. To drop further noises, I also applied Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm. DBSCAN finds clusters of high-density point features within surrounding noise based on their spatial distribution [<a href="https://pro.arcgis.com/en/pro-app/tool-reference/spatial-statistics/densitybasedclustering.htm">8</a>]. DBSCAN is marking as outliers’ points that lie alone in low-density regions whose nearest neighbors are too far away (Figure 6).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/489/1*KfykZ8TbI4WkpbWapi3X-g.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/1*cB_VVq1O6-jYgJ_sxHco7Q.jpeg" /><figcaption>Figure 6. Example of the 15m×15m Box Before and After Dropping Noises using “Classify LAS Noise” Function and DBSCAN. The Image on the Left is Color-Coded based on Elevation Values. The Image is Color-Coded based on DBSCAN Classes</figcaption></figure><p>After dropping ground points and noises from further analysis, each 15m×15m box still has points vary from 50K to 100K. It&#39;s recommended using 4096 points in each box for training. Finally, I developed a spatial random sampling method to go over each box and keeps only 4096 point clouds in each box. This method initially converts each box to 3D voxels with 1m length along X, Y, and Z direction. It then randomly samples from each voxel based on the density of the point clouds within each voxel. With keeping the necessary points in each box, its time for the training run.</p><p><strong>4. Train PointNet for Classification</strong></p><p>I used PointNet as a deep learning model for classification. PointNet directly takes point clouds as input and outputs either class labels for the entire input or per point segment/part labels for each point of the input (Figure 7). PointNet processes each point, which is represented by its three coordinates (X, Y, Z) or more attributes such as intensity, RGB and etc. First, PointNet is invariant to permutations of point clouds due to using a single symmetric function called max pooling. Second, PointNet is invariant to transformations of the point clouds due to using affine transformations which are applied to each point independently. PointNet effectively selects informative points from the point clouds and encode the reason for their selection. The fully connected layers of the network aggregate these learned optimal values into the global descriptors for classification or concatenate global descriptors with local descriptors to predict the class of each point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b9KmDxU-ga7ReWOMfg0-dA.jpeg" /><figcaption>Figure 7. The Architecture of PointNet for Classification and Segmentation [<a href="https://arxiv.org/abs/1612.00593">9</a>]</figcaption></figure><p>After data processing, I extracted X, Y, Z coordinates and intensity associated with point cloud in each box for training. I then calculated the range of Z for each point in each box to use for training instead of absolute Z. I also shuffled the points in each box as well as the order of the boxes before training the model. Table 1 summarized the total number of boxes for the training and the testing run. I used two railways for the training run and left out another railway for the testing run.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/497/1*445CrHu-SiKvqUK1mWwfmg.jpeg" /><figcaption>Table 1. Comparing the Number of 15m×15m Boxes for Each Class in the Training and Testing Run</figcaption></figure><p>Deep learning models, in general, require large dataset for training. As the number of samples was not enough for positive classes (e.g., signal, milepost, crossing), I used data augmentation to increase the number of samples for training data. I used two common methods of data augmentation in 3D domain to increase the number of training data for the rare positive classes (e.g., signal, milepost, crossing, switch): 1) rotation: As LiDAR sensor mounts in front of the train, rotation along X and Y axis are not that significant. Thus, each box randomly rotated along X and Y axis for less than 5 degrees. However, for Z-axis situation is different. As the train can travel on the same railway back and fort, a similar object can be viewed from multiple views. Thus, each box randomly rotated along Z-axis between -180 to 180 degrees, and 2) jittering: Jittering can be explained as the process of damaging the point clouds or adding noise to the point data. While the point density of similar object might vary from scene to scene due to the distance of the object from the LiDAR sensor, jittering enables the model to learn only critical points in the training run.</p><p>I then merged the newly generated boxes from data augmentation with existing boxes. I used two railways for the training run and left out another railway for the testing run. The training data are coming from two railways while the testing data are coming from other railways that left out from the training run. I trained PointNet for over 200 cycles and saved the best model that produced the best precision and recall for entire classes as well as each class, separately. Overall, PointNet performed pretty well with producing over 70% precision and recall for most of the classes except switches (Table 2). PointNet failed to detect switches and misclassified them as a negative class and vice versa. This is due to the fact that the width, length, and height of switches are small (less than 20cm across dimensions).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/398/1*1DJpAwEJhSqx45kNwMlZ6w.jpeg" /><figcaption>Table 2. Accuracy Metrics such as Precision and Recall of PointNet Calculated in Testing Run</figcaption></figure><p>Figure 8 shows some of the extracted assets inside the boxes along the railway in the testing run. I have not labeled negative classes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*CyaBDzahx9mqSnQDhFv01g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/1*ue14SG1Ok1JjzKGqO4CGaw.jpeg" /><figcaption>Figure 8. Black Dots Show the Location of Assets in Point Clouds. Yellow Colors Show the Boundary of 15m×15m Boxes. The Image on the Left has Signal and Switch. The Image on the Right has Crossing. Both Images Color-Coded based on Intensity</figcaption></figure><p><strong>5. Inference Workflow</strong></p><p>After training, we need a workflow for the inference (Figure 9). This workflow is responsible to run against any new dataset using the existing trained model to detect assets and export them as a point feature class on ArcGIS Online [<a href="https://www.arcgis.com/index.html">10</a>]. The workflow that I am currently working on has four main steps: 1) web user-interface where user can drop point clouds in LAS format as well as railway centerline in shapefile format. As data flows to an S3 bucket, AWS Lambda which is an event-driven, serverless computing platform runs Data Pipeline in response and automatically manages the computing resources required by that code, 2) Data Pipeline contains most of the data processing steps that I explained through the blog such as clipping, random sampling, noise reduction and etc., 3) Model Pipeline which can be served as a Geoprocessing Tool [<a href="https://pro.arcgis.com/en/pro-app/help/analysis/geoprocessing/basics/what-is-geoprocessing-.htm">11</a>] to run PointNet and detect asset in each box and 4) Output component which converts the outcome of model, point in centroid of box and its associated class, to feature class and push it to Portal on ArcGIS Online for user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bGSX8VUX6y9fyEJpU2GSGA.jpeg" /><figcaption>Figure 9. Inference Workflow</figcaption></figure><p><strong>6. Conclusion and Future Works</strong></p><p>In this article, I walked you through the end to end workflow that developed to extract features from 3d point clouds. The workflow is only using trackline, 3D point clouds along the railway as well as the location of assets in a point feature format to train the deep learning model. GIS played a key role here to map the location of assets and prepare the location data for training. Our future works focus on a diverse range of topics such as comparing PointNet with PointNet++ and PointCNN for both classification and segmentation railway applications.</p><p><strong>7. Acknowledgment and References</strong></p><p>I want to say thanks to <a href="https://medium.com/u/508d054415bc">Daniel Wilson</a>, <a href="https://medium.com/u/de11c2f6925b">Omar Maher</a> and Khalid Duri who reviewed this blog.</p><p>1] <a href="https://pro.arcgis.com/en/pro-app/tool-reference/data-management/generate-points-along-lines.htm">https://pro.arcgis.com/en/pro-app/tool-reference/data-management/generate-points-along-lines.htm</a><br>2] <a href="http://desktop.arcgis.com/en/arcmap/10.3/tools/data-management-toolbox/split-line-at-point.htm">http://desktop.arcgis.com/en/arcmap/10.3/tools/data-management-toolbox/split-line-at-point.htm</a><br>3] <a href="http://desktop.arcgis.com/en/arcmap/10.3/tools/analysis-toolbox/buffer.htm">http://desktop.arcgis.com/en/arcmap/10.3/tools/analysis-toolbox/buffer.htm</a><br>4] <a href="https://pro.arcgis.com/en/pro-app/tool-reference/analysis/spatial-join.htm">https://pro.arcgis.com/en/pro-app/tool-reference/analysis/spatial-join.htm</a><br>5] <a href="https://pro.arcgis.com/en/pro-app/tool-reference/data-management/generate-points-along-lines.htm">https://pro.arcgis.com/en/pro-app/tool-reference/3d-analyst/extract-las.htm</a><br>6] <a href="https://pro.arcgis.com/en/pro-app/tool-reference/3d-analyst/classify-las-ground.htm">https://pro.arcgis.com/en/pro-app/tool-reference/3d-analyst/classify-las-ground.htm</a><br>7] <a href="https://pro.arcgis.com/en/pro-app/tool-reference/3d-analyst/classify-las-noise.htm">https://pro.arcgis.com/en/pro-app/tool-reference/3d-analyst/classify-las-noise.htm</a><br>8] <a href="https://pro.arcgis.com/en/pro-app/tool-reference/spatial-statistics/densitybasedclustering.htm">https://pro.arcgis.com/en/pro-app/tool-reference/spatial-statistics/densitybasedclustering.htm</a><br>9] <a href="https://arxiv.org/abs/1612.00593">https://arxiv.org/abs/1612.00593</a><br>10] <a href="https://www.arcgis.com/index.html">https://www.arcgis.com/index.html</a><br>11] <a href="https://pro.arcgis.com/en/pro-app/help/analysis/geoprocessing/basics/what-is-geoprocessing-.htm">https://pro.arcgis.com/en/pro-app/help/analysis/geoprocessing/basics/what-is-geoprocessing-.htm</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=64d14ad66e39" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geoai/automating-railway-asset-detection-using-deep-learning-64d14ad66e39">Automating Railway Asset Detection using Deep Learning</a> was originally published in <a href="https://medium.com/geoai">GeoAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An End to End Workflow on the Cloud to Monitor Traffic Flow using Deep Learning]]></title>
            <link>https://medium.com/geoai/an-end-to-end-solution-on-the-cloud-to-monitor-traffic-flow-using-deep-learning-9dfdfd00b621?source=rss-7d56687076d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/9dfdfd00b621</guid>
            <category><![CDATA[traffic-management]]></category>
            <category><![CDATA[geoai]]></category>
            <category><![CDATA[live-streaming]]></category>
            <category><![CDATA[gis]]></category>
            <category><![CDATA[yolov3]]></category>
            <dc:creator><![CDATA[Amin Tayyebi]]></dc:creator>
            <pubDate>Wed, 17 Jul 2019 13:47:56 GMT</pubDate>
            <atom:updated>2019-09-15T01:49:53.144Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GXNPZvlsKTDXmpvkQTaNXg.jpeg" /></figure><p>An overview of integrating live feed streaming video technology with ArcGIS, the ArcGIS API for Python and deep learning models in Amazon Web Services (AWS) to monitor traffic flow<em>.</em></p><p>Outline:</p><p>1. Traffic Management and Problem Statement<br>2. Live Feed Streaming Technology and Data Labeling<br>3. Object Detection: Training YOLO3 on AWS<br>4. End to End Workflow Architecture<br>5. Monitor Live Traffic Flow using the Dashboard<br>6. Anomaly Detection using Historical Data<br>7. Conclusion and Future Works<br>8. Acknowledgment and References</p><ol><li><strong>Traffic Management and Problem Statement</strong></li></ol><p>Traffic flow and intensity are important factors to monitor in urban environments. The need to control road traffic flow is not a new idea. In most big cities, busy roads, highways, and intersections are monitored using cameras. People at the department of transportation are usually interested in a variety of things such as accidents, road coverage such as snow, crime, car breakdowns, speed of cars, traffic jams and number of pedestrians. Information from live feed streaming technology can help better to manage traffic and provide public safety for citizens. A study by the National Highway Traffic Safety Administration showed that 36% of crashes happen at intersections. Thus, intersections are a major cause of city congestion leading to delays and a major concern of traffic management centers. To monitor and manage traffic flow, many cameras are installed at intersections. These can be both cameras with a fixed position, as well as remotely controlled (pan/tilt/zoom) cameras (Figure 1).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/1*N9O9Rt6zUMXuC6JuQhlhBQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/1*KREU-Q_IjEXJmZprWRbWcQ.jpeg" /><figcaption>Figure 1. Images’ Examples of Live Feed Streaming Technology with Detected Objects at Two Intersections in Washington D.C. (Day-Time and Night-Time)</figcaption></figure><p>The District Department of Transportation (DDOT) in Washington D.C. asked Esri to build an end-to-end workflow on the cloud that can 1) monitor traffic (mainly cars, buses, trucks, bikes, and people) at the main 111 intersections in Washington D.C. and visualize traffic flow using GIS, 2) detect anomalies in traffic (mainly cars, buses, trucks, bikes, and people) at intersections in terms of volume, and 3) detect pedestrians at unsafe locations at intersections. Such an end-to-end workflow requires not only detecting desired objects from live feed streaming video, but it also needs to integrate geospatial data with a deep learning framework.</p><p>In this blog post, I show how I built an end-to-end workflow by leveraging ArcGIS, the ArcGIS API for Python [<a href="https://developers.arcgis.com/python">1</a>], AWS and a deep learning framework (Keras here). The end to end workflow uses AWS for training and inference using GPU to speed up the process and serve this workflow as a live workflow for the department of transportation [<a href="https://aws.amazon.com/machine-learning/amis">2</a>]. It also uses the ArcGIS API for Python to integrate geospatial information such as the location of objects from live feed streaming video with deep learning framework and store the time series records in using ArcGIS Enterprise[<a href="http://www.arcgis.com/index.html">3</a>].</p><p><strong>2. Live Feed Streaming Technology and Data Labelling</strong></p><p>Deep learning projects require large data to train the model. Thus, I had to collect training samples from live cameras in Washington D.C. TrafficLand provides live traffic video for 111 cameras in Washington D.C. as a RestAPI [<a href="http://www.trafficland.com/">4</a>]. To prepare the training samples, I wrote a Python Script that randomly calls TrafficLand Rest API and collected ~1000 images at both day-time and night-time from 111 cameras. I then gathered images into a folder and have labeled images using LabelMe (Figure 2) by drawing bounding boxes around desired objects [<a href="https://github.com/tzutalin/labelImg">5</a>]. I then exported labels in a text file format that is readable by most of the object detection algorithms.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lXSy2L_kNgkKV3Z_wU6NBg.jpeg" /><figcaption>Figure 2. User-interface of LabelImg Software and Example of Labeled Image in Day-Time</figcaption></figure><p><strong>3. Object Detection: Training YOLO3 on AWS</strong></p><p>To build an end to end workflow, we initially need to detect objects from live feed streaming. You Only Look Once (YOLO) is a popular deep learning object detection algorithm that can achieve high accuracy in real-time applications. The output of YOLO provides the location of objects in the image as well as its associated class. YOLO requires only one forward propagation pass through the network to make predictions. The earlier version of YOLO such as YOLO2 often had missed small objects in the image due to loss of fine-grained features as the layers downsampled the input image. In addition, YOLO2’s architecture was still lacking most of the state-of-the-art algorithms such as residual blocks, skip connections and upsampling. YOLO3 tackle such issues by adding more layers and incorporate residual skip connections and upsampling. YOLO3 performs the detection by applying a 1×1 detection kernels on feature maps of three different sizes at three different places in the network (Figure 3). There are lots of papers and blogs that clearly explain about YOLO and how YOLO progressed during the last few years. Thus, I am going to skip covering YOLO in more details here. You can find more about original YOLO as well as how they are different from each other in these references [<a href="https://arxiv.org/abs/1506.02640">6</a>].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xqgd5K3Smh9XfUUFdUwdEg.png" /><figcaption>Figure 3. YOLO3 Architecture</figcaption></figure><p>Around 1000 images from 111 live feeds cameras at the day-time and the night-time with their associated labels used to train YOLO3 on AWS. I ran an EC2 instance on AWS that had AWS deep learning AMIs [<a href="https://aws.amazon.com/machine-learning/amis">2</a>] which comes with pre-installed popular deep learning frameworks and interfaces such as TensorFlow, PyTorch, and Keras to train sophisticated deep learning models. I used the pre-trained YOLO3 model and did transfer learning. I then tested the trained model by comparing the predicted bounding boxes with the actual labels. The trained model performed ~95% mean intersection of union (IOU). I used existing Github repo to train YOLO3 model as a reference [<a href="https://github.com/qqwweee/keras-yolo3">7</a>].</p><p><strong>4. End to End Workflow Architecture</strong></p><p>To build a live workflow on AWS, we came up with below architecture to facilitate traffic monitoring (Figure 4): 1) We used parallel processing to speed up collecting live feed images from TrafficLand RestAPI, 2) Then images pass to the trained YOLO3 model on AWS EC2 instance. YOLO3 detects objects in each image and their associated bounding boxes. Overall, we were able to collect 111 images and run inference in 10 seconds on AWS using a single NVIDIA Tesla K80 GPU, 3) We then send the outcome of YOLO3 model to GeoEvent [<a href="https://enterprise.arcgis.com/en/geoevent/latest/get-started/a-quick-tour-of-geoevent-server.htm">8</a>] on AWS to be stored in the big data store as well as visualizing traffic flow on the dashboard. Our workflow also stores the associated image with each camera in S3 bucket across times.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xIh8We0kjlXsmxY716xGxg.jpeg" /><figcaption>Figure 4. An End to End Workflow Architecture</figcaption></figure><p>Our IT team set up deep learning EC2 instance and ArcGIS GeoEvent Server [<a href="https://enterprise.arcgis.com/en/geoevent/latest/get-started/a-quick-tour-of-geoevent-server.htm">8</a>] on AWS. GeoEvent Server enables real-time event-based data streams to be integrated as location data in a feature class or big data store [<a href="https://enterprise.arcgis.com/en/geoevent/latest/administer/managing-big-data-stores.htm">9</a>]. To connect GeoEvent Server to YOLO3 model on deep learning EC2 instance, I had to configure GeoEvent Service which includes input connectors, processors and output connectors. GeoEvent Services are created using a simple graphic interface like ModelBuilder [<a href="http://desktop.arcgis.com/en/arcmap/10.3/analyze/modelbuilder/what-is-modelbuilder.htm">10</a>].</p><p>The input connector informs GeoEvent about the structure of event data that coming from YOLO3 model and passes data to the processors. If you send data with other format or structure, GeoEvent won’t be able to consume data. Several connectors are provided for common data formats (text, RSS, ESRI feature JSON, and generic JSON) and data communication channels (system file, HTTP, TCP, UDP, WebSocket, and ESRI feature service). To set up the input connector, we need to define GeoEvent Definition which shows the data structure of event data coming (Figure 5a). Figure 5a shows GeoEvent Definition in GeoJSON format and Figure 5b shows RestAPI as a data communication channel. Thus, The outcome of YOLO3 model for each camera converted to GeoJSON using camera location information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/1*KgFGuCTyZk_UQXaKrhK4vA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*LfaW6euFuYfgziIaBHdfkQ.jpeg" /><figcaption>Figure 5. GeoEvent Definition on the Left Side and Input Connector on the Right Side</figcaption></figure><p>The processors are a configurable element of a GeoEvent Service that performs specific actions on event data as it is received such as identification or enrichment as event data is routed from inputs to outputs. Since we are not processing data before writing them into feature class, we did not use processors in our GeoEvent services.</p><p>The output connectors are responsible for converting the GeoEvents back to a data stream and sending the events through a selected data communication channel. We set up two GeoEvent services: 1) real-time GeoEvent service (called Update-Counts in Figure 6a) is responsible to digest only current or live feed data and visualize them on the dashboard while 2) historical GeoEvent service (called Sum-Counts in Figure 6b) is responsible to store historical data in a feature class or big data store for further analysis such as anomaly detection. The difference between two services is that real-time service only has 111 records that updates based on current or live feed data while historical service keeps appending new records. Let’s do simple math to figure out a number of records per day. If the historical GeoEvent service consumes event data from 111 cameras per second, our system stores ~9.6 million records per day (111 cameras × 24 hours × 60 minutes × 60 seconds).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/765/1*vnpJFSe2P67_fDqwqAOcLA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/739/1*a7rmAVv-H6oZ96PsdGTtEQ.jpeg" /><figcaption>Figure 6. Real-Time GeoEvent Service Output on the Left Side and Historical GeoEvent Service Output on the Right Side</figcaption></figure><p><strong>5. Monitor Live Traffic Flow using the Dashboard</strong></p><p>To automate real-time traffic flow monitoring, we provided a dashboard [<a href="https://www.esri.com/en-us/arcgis/products/operations-dashboard/overview">11</a>] for the department of transportation. The dashboard shows the location of 111 live feeds cameras in D.C. as well as the counts of cars, buses, trucks, bikes, and people at each intersection. The dashboard is connected to feature class that populates by real-time GeoEvent service. Users can visually see where in Washington D.C. is more crowded by cars versus people and vice versa. Users can also visually see the live feed image associated with each camera. Figure 7 illustrates the interface of the dashboard. The dashboard has also updated the statistics on the left panel when the user zooms and pans across the study area.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dFTn5GSWZ56MkZdlsgK4nQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xg1XCl8URGLLLOUzaLA0xg.jpeg" /><figcaption>Figure 7. Top Image Shows the Dashboard Interface for entire Washington D.C. Each Red Dot Refers to One Camera. Numbers on the Left Side Show Number of Objects in Entire D.C. Lower Image Shows the Dashboard Interface for the Zoomed Area in D.C. Zooming Further Enables you to See Count of Objects in Four Corners of each Camera as well as Associated Current Image with Camera.</figcaption></figure><p><strong>6. Anomaly Detection using Historical Data</strong></p><p>The department of transportation was also interested to know how the volume of desired objects (mainly cars, buses, trucks, bikes and people) change across intersections and across times. To solve this problem, we leveraged the outcome of historical GeoEvents Service after one week to calculate the volume of each object for each camera per minute per day. Let’s do simple math to figure out the number of combinations. There are ~1 million possible combinations (111 cameras × 7 days × 24 hours × 60 minutes). You can think of this as a lookup table to detect the anomaly. For any new given count per object, we compare the results with historical counts. If it is higher than 30% of historical counts, we call it an anomaly and visualize it on the map. We also write anomalies in a separate feature class. Figure 8 shows the image with the pedestrian’s and car’s anomalies at one of the intersections.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/1*r29meAiQeL7jq4ILpxU_5g.jpeg" /><figcaption>Figure 8. Anomalies of Pedestrians and Cars at One of the Intersections</figcaption></figure><p>DDOT was also interested to learn more about pedestrian behaviors across intersections. They were mainly interested to know in which intersection pedestrians are passing from one side to another side of the street without using the crosswalk. There are lots of ways to approach this problem. The typical way is to detect crosswalks from the given image and identify pedestrians outside of crosswalks as an anomaly.</p><p>I took another way here. To do this, I ran YOLO3 for ~5 hours across one of the intersections as an example and kept the location of pedestrians (row and column of image) in each time frame. Since YOLO3 provides four coordinates of the bounding box for the given object (top right, top left, lower right, and lower left), I averaged lower row and column coordinates (lower right, and lower left) and converted the bounding box to a point. This is because lower coordinates are closer to the ground and can represent the crosswalks better. Each red point represents the location of the pedestrian across time (Figure 9). Such data can be used to reveal the pattern and behavior of how pedestrians walk across intersections. Since most of the pedestrians use the crosswalk to pass the street, there is a high density of points across the crosswalk (Figure 9). In contrast, there is a low density of the points on the street (Figure 9).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*Ksr26ng5OzUH8s4oDLFNXw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*N3MoCiHf3gNA7fbBKmTBfg.jpeg" /><figcaption>Figure 9. Raw Image on the Left Side and Pedestrian&#39;s Locations across Time on the Right Side</figcaption></figure><p>To separate high- and low-density points, I used DBSCAN, Density-Based Spatial Clustering of Applications with Noise, as an anomaly non-parametric algorithm. DBSCAN finds clusters of high-density point features within surrounding noise based on their spatial distribution [<a href="https://pro.arcgis.com/en/pro-app/tool-reference/spatial-statistics/densitybasedclustering.htm">12</a>]. DBSCAN is also marking as outliers’ points that lie alone in low-density regions whose nearest neighbors are too far away. Figure 10 shows the outcome of DBSCAN where it separated low-density points or anomalies from high-density points.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*oP8MpRFOzJCQNItdorrn0Q.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/430/1*goLcS9fZjauLZpMfd_Fx0A.jpeg" /><figcaption>Figure 10. Example of the Pedestrian that Walked from One Side of the Street to Other Side Without Using the Crosswalk</figcaption></figure><p><strong>7. Conclusion and Future Works</strong></p><p>In this article, I walked you through the end to end workflow that the GeoAI team developed on the cloud to monitor traffic flow. Our workflow is able to 1) access to the live feed streaming technology, 2) detect cars, buses, trucks, bikes, and people using YOLO3 from live feed streaming on AWS, 3) send the outcomes of YOLO3 to GeoEvent on AWS to visualize the traffic flow on the dashboard and store the them in the big data store for further analysis, 4) perform anomaly detection in terms of objects volumes as well as identifying pedestrians in unsafe locations. GIS played a key role here to map the location of cameras and visualize them on the dashboard. Our future works focus on a diverse range of topics such as tracking objects, calculating speed, calculating the volume of objects per lane, etc.</p><p><strong>8. Acknowledgment and References</strong></p><p>I want to say thanks to <a href="https://medium.com/u/508d054415bc">Daniel Wilson</a> for speeding up the workflow 10 times faster on AWS and setting up S3 bucket to save images, Joel McCune who set up the dashboard on AWS, RJ Sunderman who taught me about ArcGIS GeoEvent Server, Alberto Nieto who initially started this project with DDOT where we expanded his works by adding YOLO3 model, ArcGIS GeoEvent Server, AWS to make it a real-time workflow on the cloud and finally Mark Carlson who set up ArcGIS GeoEvent Server and AWS Deep Learning AMIs. We also replicated the entire workflow on Azure. Please let me know if you either have a question or you think we can collaborate on a similar project.</p><p>1] <a href="https://developers.arcgis.com/python">https://developers.arcgis.com/python</a><br>2] <a href="https://aws.amazon.com/machine-learning/amis">https://aws.amazon.com/machine-learning/amis</a><br>3] <a href="http://www.arcgis.com/index.html">http://www.arcgis.com/index.html</a><br>4] <a href="http://www.trafficland.com/">http://www.trafficland.com</a><br>5] <a href="https://github.com/tzutalin/labelImg">https://github.com/tzutalin/labelImg</a><br>6] <a href="https://arxiv.org/abs/1506.02640">https://arxiv.org/abs/1506.02640</a><br><a href="https://arxiv.org/abs/1612.08242">https://arxiv.org/abs/1612.08242</a><br><a href="https://arxiv.org/abs/1804.02767">https://arxiv.org/abs/1804.02767</a><br><a href="https://lilianweng.github.io/lil-log/2018/12/27/object-detection-part-4.html#yolo-you-only-look-once">https://lilianweng.github.io/lil-log/2018/12/27/object-detection-part-4.html#yolo-you-only-look-once</a><br><a href="https://towardsdatascience.com/yolo-v3-object-detection-53fb7d3bfe6b">https://towardsdatascience.com/yolo-v3-object-detection-53fb7d3bfe6b</a><br>7] <a href="https://github.com/qqwweee/keras-yolo3">https://github.com/qqwweee/keras-yolo3</a><br>8] <a href="https://enterprise.arcgis.com/en/geoevent/latest/get-started/a-quick-tour-of-geoevent-server.htm">https://enterprise.arcgis.com/en/geoevent/latest/get-started/a-quick-tour-of-geoevent-server.htm</a><br>9] <a href="https://enterprise.arcgis.com/en/geoevent/latest/administer/managing-big-data-stores.htm">https://enterprise.arcgis.com/en/geoevent/latest/administer/managing-big-data-stores.htm</a><br>10] <a href="http://desktop.arcgis.com/en/arcmap/10.3/analyze/modelbuilder/what-is-modelbuilder.htm">http://desktop.arcgis.com/en/arcmap/10.3/analyze/modelbuilder/what-is-modelbuilder.htm</a><br>11] <a href="https://www.esri.com/en-us/arcgis/products/operations-dashboard/overview">https://www.esri.com/en-us/arcgis/products/operations-dashboard/overview</a><br>12] <a href="https://pro.arcgis.com/en/pro-app/tool-reference/spatial-statistics/densitybasedclustering.htm">https://pro.arcgis.com/en/pro-app/tool-reference/spatial-statistics/densitybasedclustering.htm</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9dfdfd00b621" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geoai/an-end-to-end-solution-on-the-cloud-to-monitor-traffic-flow-using-deep-learning-9dfdfd00b621">An End to End Workflow on the Cloud to Monitor Traffic Flow using Deep Learning</a> was originally published in <a href="https://medium.com/geoai">GeoAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[High Resolution Land Cover Mapping using Deep learning]]></title>
            <link>https://medium.com/geoai/high-resolution-land-cover-mapping-using-deep-learning-7126fee571dd?source=rss-7d56687076d8------2</link>
            <guid isPermaLink="false">https://medium.com/p/7126fee571dd</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[remote-sensing]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[land-governance]]></category>
            <dc:creator><![CDATA[Amin Tayyebi]]></dc:creator>
            <pubDate>Wed, 06 Feb 2019 21:29:48 GMT</pubDate>
            <atom:updated>2019-09-15T01:48:07.858Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0KlOmrZqy9lqE9zWyyZ4jg.jpeg" /></figure><h3><strong>High-Resolution Land Cover Mapping using Deep Learning</strong></h3><p>An overview of applying deep learning models to provide high-resolution land cover in the state of Alabama using <em>Keras</em> and <em>ArcGIS</em></p><p>Outline:</p><p><strong>1. </strong>Land Cover Mapping <br> <strong>2. </strong>Image Segmentation<br> <strong>3. </strong>Data Sources<br> <strong>4. </strong>Data Preparation using ArcGIS Pro and ArcGIS API for Python<br> <strong>5. </strong>Data Augmentation<br> <strong>6. </strong>Modify and Train U-Net Model<br> <strong>7. </strong>Deploy Model in ArcGIS Pro and Run The Trained Model on NAIP Images<br> <strong>8. </strong>Accuracy Assessment<br> <strong>9. </strong>GeoAI Cookiecutter Data Science Template<br> <strong>10. </strong>Conclusion<br> <strong>11. </strong>Acknowledgment and References</p><p><strong>1. Land Cover Mapping</strong></p><p>Global land cover maps have been extensively used for a variety of applications including ecosystem services, climate change, hydrological processes, and policymaking at local and regional scales. While low resolution, spatial (e.g., 30m) and temporal (e.g., every 5 years), land cover maps have been developed by various agencies (e.g., USGS, USDA, NASA) for the entire Europe and USA, creation of real-time high resolution spatial and temporal land cover maps (e.g., 1 meter) at a regional scale is lacking. The land change science community has been pursuing this goal since the early 2000s without broad success.</p><p>In this blog article, I will walk you through the model we developed that can create a classified land cover map with 1-meter resolution from National Agriculture Imagery Program (NAIP) image at the pixel level (Figure 1). The trained model classifies NAIP image to six land cover classes: 1) buildings, 2) roads or parking lots, 3) water, 4) harvested, open land or bare land, 5) forest and 6) planted or dark cropland.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*B597eI5mL84I0HqqEwN0jQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/1*dRDIT-DS2XUEJnXSrY6oag.jpeg" /><figcaption>Figure 1. Raw NAIP Image versus Classified Image</figcaption></figure><p>Beside land cover classification approaches that you might be able to find in other blogs, I am also interested to show here that how you can leverage ArcGIS API for Python and ArcGIS Pro and integrate them with deep learning tools (e.g., Keras here). This enables you to prepare geospatial data (raster or vector data) faster by leveraging Geoprocessing Tools in ArcGIS Pro that can be used in Python and visualize progress in ArcGIS Pro (Figure 2).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/1*sjLwfZ8vR1TpIXpC9-PP2A.jpeg" /><figcaption>Figure 2. Integrating ArcGIS Pro, Python API and Deep Learning</figcaption></figure><p><strong>2. Image Segmentation</strong></p><p>Image segmentation is one of the critical problems in the field of computer vision. Image segmentation is dividing an image into multiple segments. In other word, image segmentation is the process of assigning a label to every pixel in an image such that pixels with the same label share certain characteristics. Image segmentation is an ideal approach to be used for land cover classification since within each land cover class pixels share similar characteristics across multiple bands. The importance of image segmentation has been illustrated in various applications such as self-driving vehicles, human-computer interaction, virtual reality etc. With the popularity of deep learning, many image segmentation problems are being tackled using deep architectures which surpass other traditional machine learning approaches by a large margin in terms of accuracy.</p><p><strong>3. Data Sources</strong></p><p>Our study area was part of Alabama State. we collected 12 NAIP images <strong>[Ref 1]</strong>, 8 NAIP images in the north and 4 NAIP images in the south, using ArcGIS Online <strong>[Ref 2]</strong>. NAIP imagery is acquired at a one-meter ground sample distance with a horizontal accuracy. The spectral resolution of NAIP has four bands including natural color (Red, Green and Blue, or RGB) and Near Infrared.</p><p>First, I used the Create Mosaic Dataset function in ArcGIS Pro <strong>[Ref 3] </strong>to create an empty mosaic dataset in a geodatabase. Second, I used the Add Rasters To Mosaic Dataset function in ArcGIS Pro <strong>[Ref 4] </strong>to add NAIP images to an empty mosaic dataset. As a result, the Add Rasters To Mosaic Dataset function creates two feature classes called 1) boundary layer that shows the extent of the study area, 2) footprint layer that shows the extent of each NAIP image as well as one raster that called Image layer that contains the mosaic of NAIP images (Figure 3).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/1*J9pfR9iczGt8JQu9hYZgYw.jpeg" /><figcaption>Figure 3. Geoprocessing Tools, NAIP Images and Study Area</figcaption></figure><p><strong>4. Data Preparation using ArcGIS Pro and ArcGIS API for Python</strong></p><p>For the training run, I need to have labelled data for each land cover class like other deep learning models. I used Training Samples Manager in ArcGIS Pro <strong>[Ref 5] </strong>to label six land cover classes on top of NAIP images. The labels collected randomly across the study area (Figure 4).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/414/1*s--aRiP7rgwUOmHk-yy_ig.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/1*GgbYBhR8bOzvA9Kpk_QCSQ.jpeg" /><figcaption>Figure 4. Training Samples Manager and Sampled Labels</figcaption></figure><p>The input and output of Image Segmentation model should be in raster format for the training run. Since labelled data are in a feature class or vector format, I used Feature to Raster function in ArcGIS Pro <strong>[Ref 6] </strong>to convert labelled data in feature class format to raster format.</p><p>Since it is very time-consuming to label an entire NAIP image, I had areas across NAIP images that were not labeled. For areas that I did not have labeled data, ArcGIS Pro assigned No-Data. I used Reclassify function in ArcGIS Pro <strong>[Ref 7] </strong>to convert No-Data to zero and keep the values of other land classes same. You can think areas with 0 values as background class that won’t have any influence in the training run. I will explain later in loss function how I minimized that effect of background class.</p><p>Since NAIP image has a large footprint (~7.5km × ~6.5km) and they cannot be fed directly to the model, I converted the NAIP images and corresponding labelled data in a raster format to smaller images. To achieve this, I used Export Training Data For Deep Learning in ArcGIS Pro [Ref 8] to convert NAIP mosaic raster as the input of model and the corresponding raster labelled data to the smaller chip. This tool lets you choose the size of each chip as well as a stride in X and Y axes. I chose the chip size of 256 and stride size 64 along X and Y axes. This tool only exports chips that have both NAIP and labeled data (Figure 5; Python #1). I defined the format of images and labels in TIFF format. The total number of chips for each NAIP image depends on labeled data across each NAIP image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*A-MJm5HNhUlxj6QC664eIA.jpeg" /><figcaption>Figure 5. Export Training Data for Deep Learning</figcaption></figure><p>If more than 50% of the chip had background class (0 value), they are dropped from further analysis. Due to the large volume and number of images in each folder (12 folders correspond to 12 NAIP images), I stacked chips across folders and converted to separate HDF5 format for each land cover class (Python #2). This enabled me to keep track of the number of chips for each land cover class.</p><p><strong>5. Data Augmentation</strong></p><p>Deep learning models require lots of data for training. From the training data, I generated 420, 438, 702, 1008, 837, 891 chips for buildings, roads, water, harvested lands, forest, and planted lands, respectively. The main solution for the lack of training data is to use data augmentation to increase the number of training data. I only used HDF5 files for rare land cover classes in data augmentation and made the number of rare classes two or three times more.</p><p>I used three common methods of data augmentation to increase the number of training data for rare land cover classes (buildings, roads or parking lots, and water): 1) shifting: Export Training Data For Deep Learning in ArcGIS Pro <strong>[Ref 8] </strong>has stride option (distance to move when creating the next image chip) along X and Y direction. The exported chip was 256 along X and Y direction. I set stride 64 along X and Y direction to get more chips. This happened during the Data Processing step and I did not do it again here, 2) rotation: in each run of data augmentation, each chip randomly selects among four values <strong>[-180, -90, 90, 180] </strong>to create new chip, and 3) zoom: in each run of data augmentation, each chip randomly selects a zoom factor across the given range <strong>[0.05, 0.45] </strong>to create new chip (Figure 6; Python #3). I then merged newly generated data with existing often land cover classes (harvested or bare land, forest, and planted or dark cropland). Finally, I normalized each band of NAIP image and shuffled training samples.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/290/1*mobPlepq8mfz_7r-Kv0AIg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/290/1*K6ITnrHF357QI9ooFeaU-g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/290/1*DCaF_O5B-ddC7WDnXrc07w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/290/1*QV5TY3DzKKd5_bNKnHNo3A.png" /><figcaption>Figure 6. Example of Data Augmentation</figcaption></figure><p><strong>6. Modify and Train U-Net Model</strong></p><p>The U-Net architecture is an encoder-decoder architecture. U-Net, a type of fully conventional network, comprises three parts: 1) A contracting path like an encoder, 2) A symmetric expanding path like a decoder and 3) skip connections (e.g., Residual Neural Network) that passes the feature maps in encoder part to decoder part.</p><p>Since I did not have a large dataset, I had to modify U-Net to a new structure with fewer parameters (Figure 7). The new U-Net model has 1,941,351 parameters (Python #3). As discussed, in each chip there are cells with values 0 or background class that it’s not our interest. I had to generate this class because it’s not typically feasible to label all cells in the image. To overcome this issue in the training run, I had to write a custom loss function that ignores zeros while calculating the loss. This customized loss function handles this issue by defining weights for each land cover class. I set weight for background class close to zero. In the training run, I used 90% of data for calibration and left out 10% of data for validation. I defined the mean of intersection over Union (IoU) to calculate the accuracy of the model using validation data in the training run. I ran the model for 30 epochs. Model stopped training in epoch 20 since validation loss did not improve significantly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EakSREsgT2wjSHBK-vU3cg.png" /><figcaption>Figure 7. Modified Structure of U-Net</figcaption></figure><p><strong>7. Deploy Model in ArcGIS Pro and Run The Trained Model on NAIP Images</strong></p><p>The U-Net model was saved in HDF5 format. ArcGIS Pro has a pleasant way to deploy models and run them at scale <strong>[Ref 8]</strong>. I used Python raster functions in ArcGIS Pro to deploy the model. Raster functions in ArcGIS Pro use parallel processing to run models faster. ArcGIS Pro has two geoprocessing tools that can run deep learning models: “Detect Objects Using Deep Learning” and “Classify Pixels Using Deep Learning”. I used the Classify Pixels option since I am running the segmentation model. Integration of external deep learning model frameworks currently works any deep learning framework, provided you can supply a raster function. Out of the box, raster functions are supplied for the TensorFlow Object Detection API, and a few other frameworks. Once the model is trained, you use an Esri Model Definition file (.emd) to run geoprocessing tools to detect or classify features in ArcGIS Pro. You also need to install the proper deep learning framework and supporting Python libraries (TensorFlow, CNTK, PyTorch, or Keras) in the ArcGIS Pro Python environment; otherwise, an error will occur when you add the .emd file to the tool. The .emd file is a JSON file that describes the trained deep learning model. It contains model definition parameters that are required to run the inference tools, and it should be modified by the data scientist that trained the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*ArwZIAuYm0UCrJCGQZvL0w.jpeg" /><figcaption>Figure 8. Structure of EMD file for U-Net Model</figcaption></figure><p>After creating the .emd file (Figure 8), I ran inference on 12 NAIP images (Figure 9).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*B597eI5mL84I0HqqEwN0jQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/1*dRDIT-DS2XUEJnXSrY6oag.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4qzOxIR3dMO8vE18Hq-TxA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XRLiSzkTle4WhRNBkvZjMA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dosrDLK_OxXtuchqIP3Jjw.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qHydHht631cLN6fH7UL-QA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sm-7wU_p5aWkMi3gMGniCA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b0OcLCzIXTxNdt1oHEjXNA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rzoM_9m_MKgi_AgUkZgMNg.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LVrpW98l9B9IdRwNdDmd2Q.jpeg" /><figcaption>Figure 9. Raw NAIP image and Classified Land Cover Map from U-Net</figcaption></figure><p><strong>8. Accuracy Assessment</strong></p><p>I separated one of the NAIP images that had labels and was not used in the training run. I used this NAIP image for the testing run. I ran inference on this image and compared the output of the model with the rasterized version of the labelled data. The outcome of this comparison was a contingency table that is usually used in remote sensing for accuracy assessment. I calculated the precision and recall for each land cover class. The overall accuracy of the U-Net model is around 85% for the given area. It was not surprising that the model performed better for frequently seen classes compared to rare classes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/335/1*pI2E1g3TDGmL2TvCCArWfw.jpeg" /><figcaption>Table 1. Accuracy Assessment of U-Net Model (Precision and Recall in %)</figcaption></figure><p><strong>9. GeoAI Cookiecutter Data Science Template</strong></p><p>Sharing a Data Science project with other data scientists is always challenging since everybody has their own structure to build a template for a Data Science project. The common format for a Data Science project enables data scientists to expect specific format when they share or receive a project from others. Here, I used <a href="https://github.com/drivendata/cookiecutter-data-science/">Cookiecutter Data Science Template</a> which is a logical, reasonably standardized, but flexible project structure for doing and sharing data science work. You can easily set up the template for your project using a few command lines. Our team (the Esri GeoAI team) implemented a new cookiecutter template for Geospatial projects strongly based on the data science cookiecutter template.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/903/1*keHuF3_xgYzoDTDSDrfSGg.jpeg" /><figcaption>Figure 10. Cookiecutter Data Science Template for GeoAI Team</figcaption></figure><p><strong>10. Conclusion</strong></p><p>In this article, I walked you through the U-Net model I used to create a classified land cover map with 1m resolution from NAIP images at the pixel level. The trained model classifies NAIP images into six land cover classes called 1) buildings, 2) roads or parking lots, 3) water, 4) harvested, open land or bare land, 5) forest and 6) planted or dark cropland. I also showed how you can leverage the ArcGIS API for Python with ArcGIS Pro and integrate them with deep learning tools (e.g., Keras) to facilitate data preparation and modeling. The U-Net model performed well using 4 NAIP bands to classify NAIP images.</p><p><strong>11. Acknowledgment and References</strong></p><p>I want to say thanks to <a href="https://medium.com/u/508d054415bc">Daniel Wilson</a> and Mary Wahl (Microsoft) who answered my questions during the project.</p><p>1] <a href="https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery">https://www.fsa.usda.gov/programs-and-services/aerial-photography/imagery-programs/naip-imagery</a><br>2] <a href="https://www.arcgis.com/index.html">https://www.arcgis.com/index.html</a><br>3] <a href="http://pro.arcgis.com/en/pro-app/tool-reference/data-management/create-mosaic-dataset.htm">http://pro.arcgis.com/en/pro-app/tool-reference/data-management/create-mosaic-dataset.htm</a><br>4] <a href="http://pro.arcgis.com/en/pro-app/tool-reference/data-management/add-rasters-to-mosaic-dataset.htm">http://pro.arcgis.com/en/pro-app/tool-reference/data-management/add-rasters-to-mosaic-dataset.htm</a><br>5] <a href="https://pro.arcgis.com/en/pro-app/help/analysis/image-analyst/training-samples-manager.htm">https://pro.arcgis.com/en/pro-app/help/analysis/image-analyst/training-samples-manager.htm</a><br>6] <a href="http://pro.arcgis.com/en/pro-app/tool-reference/conversion/feature-to-raster.htm">http://pro.arcgis.com/en/pro-app/tool-reference/conversion/feature-to-raster.htm</a><br>7] <a href="http://pro.arcgis.com/en/pro-app/tool-reference/3d-analyst/reclassify.htm">http://pro.arcgis.com/en/pro-app/tool-reference/3d-analyst/reclassify.htm</a><br>8] <a href="http://pro.arcgis.com/en/pro-app/tool-reference/image-analyst/export-training-data-for-deep-learning.htm">http://pro.arcgis.com/en/pro-app/tool-reference/image-analyst/export-training-data-for-deep-learning.htm</a><br>9] <a href="http://pro.arcgis.com/en/pro-app/help/analysis/image-analyst/deep-learning-in-arcgis-pro.htm">http://pro.arcgis.com/en/pro-app/help/analysis/image-analyst/deep-learning-in-arcgis-pro.htm</a><br>10] <a href="https://github.com/ArcGIS/geoai-cookiecutter">https://github.com/ArcGIS/geoai-cookiecutter</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7126fee571dd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geoai/high-resolution-land-cover-mapping-using-deep-learning-7126fee571dd">High Resolution Land Cover Mapping using Deep learning</a> was originally published in <a href="https://medium.com/geoai">GeoAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>