<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Chunguang (Wayne) Zhang on Medium]]></title>
        <description><![CDATA[Stories by Chunguang (Wayne) Zhang on Medium]]></description>
        <link>https://medium.com/@ellenebay88?source=rss-222a9011dd53------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*Ay3aoN10hk6OIZXW0I9WNw.png</url>
            <title>Stories by Chunguang (Wayne) Zhang on Medium</title>
            <link>https://medium.com/@ellenebay88?source=rss-222a9011dd53------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 05:18:36 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ellenebay88/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A Spatio-temporal Graph Neural Networks (STGNN) for Traffic Incident Prediction Using Arcpy and…]]></title>
            <link>https://medium.com/@ellenebay88/building-a-spatio-temporal-gcnn-model-for-traffic-incident-forcasting-using-arcpy-and-pytorch-3ae15edb7a6f?source=rss-222a9011dd53------2</link>
            <guid isPermaLink="false">https://medium.com/p/3ae15edb7a6f</guid>
            <category><![CDATA[graph-neural-networks]]></category>
            <category><![CDATA[gcnn]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[traffic-safety]]></category>
            <category><![CDATA[machine-learning-ai]]></category>
            <dc:creator><![CDATA[Chunguang (Wayne) Zhang]]></dc:creator>
            <pubDate>Wed, 27 Dec 2023 02:54:32 GMT</pubDate>
            <atom:updated>2024-10-30T22:19:10.901Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*j9m5cjcnag93jq1TnJPvNA.png" /><figcaption>Cobb Graph nodes -&gt;road center points, edges -&gt; road intersections, image: author</figcaption></figure><h3>A Spatio-temporal Graph Neural Networks (STGNN) for Traffic Incident Prediction Using Arcpy and Pytorch Gemetric</h3><p><strong>A unique graph deep learning approach for traffic incident prediction</strong>.</p><p><strong>Introduction:</strong> Graph Neural Network (GNN) has gained attention for representing complex data structures, such as molecular networks and biological protein networks, chemical structures using nodes and edges, where nodes are embedded as low dimensional vector space to summarize every node’s position and the structure of its local neighborhood [1]. In my last blog of 2021, I explored the experimental study of a ConvLSTM model for next-day car crash prediction using the Cobb county map as a grid image.</p><p>In this blog, I intend to delve further into the exploration of spatial-temporal graph neural networks (STGNNs) for traffic crash forecasting. The term <em>spatio-temporal graph </em>has been often referred to a homogeneous graph of fixed topology, and node features that change over time at discrete time steps corresponding to sampled observations. In September 2020, DeepMind collaborated with Google Maps to unveiled a advanced GNNs model for improved ETA accuracy. Traffic networks inherently exhibit graph-like structures, making this approach particularly intuitive. The image below is sourced from <a href="https://deepmind.google/discover/blog/traffic-prediction-with-advanced-graph-neural-networks/"><em>Google DeepMind blog post</em></a>[2].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*A_KUCaFhsDkC2qox7mmTPQ.gif" /><figcaption>Image by <a href="https://deepmind.google/discover/blog/traffic-prediction-with-advanced-graph-neural-networks/">https://deepmind.google/discover/blog/traffic-prediction-with-advanced-graph-neural-networks/</a></figcaption></figure><p><strong>Problem:</strong> Predicting traffic incidents like crashes within a <em>spatial-temporal</em> context is crucial for public safety and government authorities. However, this task is exceptionally challenging due to various contributing factors multitude of contributing factors and merging the heterogeneous data sources like human behavior, time, road geometry, weather, and further complicated by the sparse datasets. Traditional accident prediction methods, such as Poisson and Negative Binomial (NB) regression, often prove inadequate when faced with the intricacies of spatio-temporal correlations.</p><p><strong>Solution:</strong> The proposed solution involves 1. Building a graph structure using ArcGIS Pro geoprocessing tools and <a href="https://pytorch-geometric.readthedocs.io/en/latest/"><em>Pytorch Geometric</em></a>. 2. Adaptable A3T-GCN attention Graph Convolutional Neural Networks will be implemented for short-term crash prediction along county road segments within specified time frames, addressing questions of when, where, and how many accidents occur. This graph-based architecture can also be extended to other traffic studies, including estimated traffic speed, volume, and travel time.</p><p><strong>Google ETAs:</strong> It’s natural to view road segments as graph edges and intersections as graph nodes. However, intersections lack the detailed feature information discovered in road lines, which is essential for an accurate road network representation. Google’s travel time graph architecture provides an effective approach to modeling the road network and its dynamics. By replacing graph nodes with road segments, we can create graph edges to connect these segments, as illustrated in fig 2. This allows for the aggregation of informative messages between road segments as graph nodes facilitating the information flow throughout the road network. In addition, leveraging the learning bias of convolutional and recurrent neural networks further assist to capture the spatio-temporal relationships among road segment links to predict traffic incidents occurring adjacent and interesting roads.</p><p><strong>Graph Informative Messages:</strong> The graph itself can be denoted as <em>G</em>(<em>V</em>,<em>E</em>), where <em>V</em> is the set of nodes and <em>E</em> is the set of edges. The adjacency matrix <em>A</em> can be employed to represent connections between nodes. If <em>Avu</em>​=1, it signifies an edge between nodes <em>v</em> and <em>u</em>; otherwise, <em>Avu</em>​=0. Graph neural network (GNN) operations can be denoted as <em>H</em>(<em>l</em>+1)=<em>f</em>(<em>H</em>(<em>l</em>),<em>A</em>), where <em>H</em>(<em>l</em>) represents node features at layer <em>l</em>, and <em>f</em> denotes the aggregation function. Graph Convolutional Networks (GCNs) leverage pairwise message passing, allowing graph nodes to iteratively update their representations. In the short-term crash prediction model of deep learning practice, various factors like traffic conditions, weather, road geometry, and driver characteristics are engineered as node features in the graph model.</p><p><strong>Data Sources:</strong> The analysis utilizes Cobb County road crash data from 2018 to 2021. Bidirectional road networks are removed and consolidated into single directions for efficient graph learning representation. Fig 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*wZI9Z56rIU_eC07rZuYwwA.jpeg" /><figcaption>Fig. 1 single direction Cobb road networks. image: author</figcaption></figure><p><strong>Graph Building Process:</strong> Several ESRI ArcGIS Pro geoprocessing tools and Arcpy are employed to construct the nodes and edges of the Cobb County road network. A total of 246 nodes and 564 road edges are constructed.</p><p><strong>1.</strong> <strong>Dissolve roads by street name:</strong></p><ul><li>Use ArcGIS Pro DISSOLVE on the Road Arterial layer with the attribute Street Name and option UNSPLIT_LINES.</li><li>Result layer: ROAD_ARTERIAL_DISSOLVE.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/167/1*ICNrXjeLKxsXo_AUJu6G3Q.png" /><figcaption>image source, <a href="https://pro.arcgis.com/en/pro-app/latest/tool-reference/data-management/dissolve.htm">https://pro.arcgis.com/en/pro-app/latest/tool-reference/data-management/dissolve.htm</a></figcaption></figure><p><strong>2.</strong> <strong>Generate lines at intersections:</strong></p><ul><li>Use FEATURE TO LINE to create lines generated by splitting at intersections.</li><li>Result layer: ROAD_ARTERIAL_FEATURETOLINE.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/1*reXQcfefqwTrR-r_0foduw.png" /><figcaption>image source, <a href="https://pro.arcgis.com/en/pro-app/latest/tool-reference/data-management/feature-to-line.htm">https://pro.arcgis.com/en/pro-app/latest/tool-reference/data-management/feature-to-line.htm</a></figcaption></figure><p><strong>3.</strong> <strong>Create road intersection points:</strong></p><ul><li>Use INTERSECT tool on ROAD_ARTERIAL_FEATURETOLINE to generate points at intersections.</li><li>Result layer: ROAD_ARTERIAL_INTERSECTIONS.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/250/1*lCRz4xM-sPpF1KI-dUf2iA.png" /><figcaption>Intersection points. image: author</figcaption></figure><p><strong>4. Create road center line points as graph nodes:</strong></p><ul><li>Use GENERATE POINTS ALONG LINE on ROAD_ARTERIAL_FEATURETOLINE with a percentage of 50% to create ROAD_ARTERAL_MIDPT</li><li>This feature serves as graph nodes, transferring road attributes with the same road IDs</li></ul><p><strong>5.</strong> <strong>Spatially join intersections and roads:</strong></p><ul><li>Use SPATIAL JOIN with a one-to-many option to join ROAD_ARTERIAl_INTERSECTIONS and ROAD_ARTERIAL_FEATURETOLINE.</li><li>Result layer: NODES_SPATIAL_ARTERIAL.</li></ul><p>This process establishes connections between intersection points and roads polylines,</p><p><strong>6.</strong> <strong>Create edge_index for Graph:</strong></p><ul><li>Create an edgeIndex, a 2D tensor defining source and target nodes of all edges, represented as <em>e</em> = <em>uv</em> in a graph.</li><li>This edgeIndex is crucial for graph representation in <a href="https://pytorch-geometric.readthedocs.io/en/latest/"><em>PyTorch Geometric.</em></a></li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b99983d0466f42a6cb6764e0ffb85495/href">https://medium.com/media/b99983d0466f42a6cb6764e0ffb85495/href</a></iframe><p>result is a 2D tensor:</p><p>tensor=([[ 1, 1, 2, …, 241, 241, 242], [ 2, 3, 3, …, 242, 243, 243]], dtype=torch.int32)</p><p><strong>7.</strong> <strong>Create edgeLink feature lines for Graph:</strong></p><ul><li>This line feature class is based on the index to link all the ROAD_ARTERAL_MIDPT as line feature.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/57011b43597390fedb5deb2bc30fbc2d/href">https://medium.com/media/57011b43597390fedb5deb2bc30fbc2d/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*j6f3e9nUCih7rDDWi9YWiQ.png" /><figcaption>Fig 2. Edge link feature lines (green) connected road center line points. image: author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/703/1*IPBIL-J-51icqmiMBgrjPw.jpeg" /><figcaption>Fig. 3. Cobb County Road Network Graph with Nodes and Edges, image: author</figcaption></figure><p><strong>Apply Graph GCNN Model: </strong>With the county road network treated as a fixed graph built as above processes, each road segment corresponds to a node, and edges connect consecutive segments or intersections . Fig 3. In addition, each node contains X=6 features of road characteristics. We can now experiment the A3T-GCN (Attention Temporal Graph Convolutional Network) model for the training and prediction of next-day or week crashes along road segments. A3T-GCN is an extension of the TGCN model by adding an attention mechanism to re-weight the influence of historical crash states and thus to capture the global variation trends of crashes on the road. TGCN was a temporal GCN model constructed by combining GCN and GRU. The details of the model can be found in the <a href="https://arxiv.org/abs/2006.11583"><em>paper</em></a><em>.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/851/1*aNcmQ6s8n0V2OJYw0jPezA.png" /><figcaption>image source: <a href="https://arxiv.org/abs/2006.11583">arXiv:2006.11583</a></figcaption></figure><p>I used seven day crash sequence data as inputs of the model. The data dimensions as (1096, 245,6, 7) is batched into the model with 80% sequences for training and 20% sequences for testing. Each sequence is shifted one day with total of 1096 sequences. The implementation is credited to the <a href="https://github.com/benedekrozemberczki/pytorch_geometric_temporal"><em>pytorch geometric temporal </em></a>[4] source codes on GitHub. The results and codes are on <a href="https://github.com/gitWayneZhang/graph/blob/main/A3T-GCN_Sptial_temporal_crash_predict.ipynb"><em>my GitHub</em></a> as well.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fe3f0272a8d201421136157da9219d66/href">https://medium.com/media/fe3f0272a8d201421136157da9219d66/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RXryaMLrpoDyvcU5TSaneA.png" /><figcaption>GNN model. image: author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X-67Zf52Df5WVGUV6Gg3gA.jpeg" /><figcaption>One epoch training with loss RMSE:0.3124. image: author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/481/1*ia22i76MFSlc08ZG1uZNRg.png" /><figcaption>A typical one day prediction map. image: author</figcaption></figure><p><strong>Why it matters:</strong> This empirical study shows how we can use graph model such as <strong>GCNN, a </strong>deep learning techniques to find a promising solution for crash prediction. The workflows and data processes can be further enhanced with more engineering features and data, automated, deployed as services. As a result, the same methodology can be applied to spatio-temporal solutions for traffic flow, rainfall, crime prediction and more you can think of benefitting of humanity.</p><p>The world today presents us with many challenges. However, with the power of <strong>AI</strong> and deep learning, we can work together to seek solutions and make the world a much better place to live. Thank you for reading this blog.</p><p><em>References:</em></p><p>[1] Leskovec, <em>Representation learning on graphs: methods and applications</em>, IEEE Data Engineering Bulletin 40 (3) (2017) 52–74.</p><p>[2] <em>ETA Prediction with Graph Neural Networks in Google Maps</em>, <a href="https://arxiv.org/abs/2108.11482">https://arxiv.org/abs/2108.11482</a></p><p>[3] Li, Yaguang and Yu, Rose and Shahabi, Cyrus and Liu, Yan, <em>Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting, </em>International Conference on Learning Representations (ICLR ‘18)</p><p>[4] Benedek Rozemberczki and Paul Scherer and Yixuan He and George Panagopoulos and Alexander Riedel and Maria Astefanoaei and Oliver Kiss and Ferenc Beres and Guzman Lopez and Nicolas Collignon and Rik Sarkar, <em>PyTorch Geometric Temporal: Spatiotemporal Signal Processing with Neural Machine Learning Models, </em>Proceedings of the 30th ACM International Conference on Information and Knowledge Management, 4564–4573, 2021</p><p>[5] <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Zhu,+J">Jiawei Zhu</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Song,+Y">Yujiao Song</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Zhao,+L">Ling Zhao</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Li,+H">Haifeng Li</a>, <em>A3T-GCN: Attention Temporal Graph Convolutional Network for Traffic Forecasting,</em><a href="https://arxiv.org/abs/2006.11583"><br>arXiv:2006.11583</a></p><p>[6] <a href="https://en.wikipedia.org/wiki/Graph_neural_network#cite_note-fey2019-18">Graph neural network — <em>Wikipedia</em></a></p><p>Github code: <a href="https://github.com/gitWayneZhang/graph">https://github.com/gitWayneZhang/graph</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3ae15edb7a6f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spatio-Temporal ConvLSTM for Crash Prediction]]></title>
            <link>https://medium.com/data-science/spatial-temporal-convlstm-for-crash-prediction-411909ed2cfa?source=rss-222a9011dd53------2</link>
            <guid isPermaLink="false">https://medium.com/p/411909ed2cfa</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[lstm]]></category>
            <category><![CDATA[arcgis-pro]]></category>
            <dc:creator><![CDATA[Chunguang (Wayne) Zhang]]></dc:creator>
            <pubDate>Tue, 14 Sep 2021 00:04:50 GMT</pubDate>
            <atom:updated>2023-12-20T03:17:09.926Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*-wpyktE1BcpaT2E1Mq_xUg.png" /><figcaption>3D Scene traffic accidents in city of Atlanta, year 2015–2019. data source GADOT. Image by author</figcaption></figure><h4><strong><em>A unique deep learning approach for accident prediction</em></strong></h4><p>This post talks about a unique empirical study using <a href="https://arxiv.org/abs/1506.04214"><strong>ConvLSTM</strong></a> deep-learning model and <strong>ArcPy</strong> to predict next-day crash risk locations with time sequences of crash feature data. The traffic crashes are formulated as a spatio-temporal sequence forecasting problem in which both the input and the prediction data are spatio-temporal sequences. The convolutional LSTM (<strong>ConvLSTM</strong>) approach is to build an end-to-end trainable model for the crash prediction. The result has shown that the <strong>ConvLSTM</strong> network can capture spatio-temporal correlations of traffic accidents when and where happening.</p><p><strong>Problems: </strong>Traffic accidents lead to severe human injuries and casualties and huge economic losses. The ability to predict the risk of traffic accidents in a spatio-temporal context is important to prevent the occurrence of accidents not only for public citizens but also government officials. However, it is a very challenging task to predict traffic accidents not only the causes of multiple factors e.g. human, time, geometric and environmental but also the rare factors and sparse data sets. Traditional accident prediction commonly applies statistic regression such as <strong>Poisson</strong>, <strong>Negative Binomial</strong> <strong>(NB)</strong> and <strong>multivariate regression</strong>. But they often fail when dealing with complex and highly nonlinear data such as spatio-temporal correlations of the accidents e.g. when, where and why. There are classic machine learning approaches e.g. <strong>XGBoost</strong>, <strong>SVM</strong> and <strong>RandomForest</strong> classifiers which engineer features into the models <a href="https://github.com/gitWayneZhang/my_Spark/blob/master/right_angle%20crash%20prediction.ipynb"><em>to seek feature importance for the probability</em></a>. We are talking about features in machine learning which are the arrays of numbers in multi dimensional space. To know more about feature engineering in Machine Learning, please read one of the best stories from <a href="https://storymaps.arcgis.com/stories/19d90071cee34cf09acce4d00d758357"><em>Daniel Wilson</em></a>.</p><p>Moreover, time and seasonality can play an important role in deciding the probability of accidents of when and where. Using ArcPy in ArcGIS Pro, we can aggregate crash locations in certain time windows. <a href="https://cobbcountyga.maps.arcgis.com/home/webscene/viewer.html?webscene=15780a20c76b4f11b09e5c350052c4be"><em>Click to see examples of crash locations</em></a> in 3D scenes.</p><p><strong>Solution:</strong> So, can we apply the classic machine learning models to search important causal features and then propose a practical deep-learning network to predict when and where the accidents happen? The answer is yes. The <strong>ConvLSTM</strong> model is one of the most interesting deep-learning models that is used to predict next-frame video or image. The original research was done for <a href="https://arxiv.org/abs/1506.04214"><em>Precipitation Nowcasting</em></a>. To better understand <strong>ConvLSTM </strong>model, let’s first look at the ordinary <strong>LSTM</strong> network model. <a href="https://en.wikipedia.org/wiki/Long_short-term_memory"><em>Long Short-Term Memory</em><strong><em> (LSTM</em></strong><em>)</em></a> network is a type of artificial recurrent neural network (<strong>RNN</strong><em>)</em> that has feedback connections. It can not only process single data points (such as images), but also entire sequences of data (such as speech or video). Below is one of the best explanatory videos credited to Michael Phil for illustrated Guide to <strong>LSTM</strong>. You can also read <a href="https://towardsdatascience.com/illustrated-guide-to-lstms-and-gru-s-a-step-by-step-explanation-44e9eb85bf21"><em>his blog</em></a><em>.</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F8HyCNIVRbSU%3Fstart%3D3%26feature%3Doembed%26start%3D3&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D8HyCNIVRbSU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F8HyCNIVRbSU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/2d0e3eaa4a9d2531c99dd8f25dd0dfa8/href">https://medium.com/media/2d0e3eaa4a9d2531c99dd8f25dd0dfa8/href</a></iframe><p><strong>Does it work? </strong>Now, let us see what is <strong>ConvLSTM</strong>. Simply saying, <strong>ConvLSTM</strong> is <strong>CNN</strong> convolution neural network combined with <strong>LSTM</strong> Network. Instead of only the input of sequence of data, its input is a sequence of data of <strong>CNN</strong> convolution neural network which is best suited for images and videos. This combination ensures<strong> ConvLSTM</strong> captures underlying local spatio-temporal correlations. See figure 1 below of the key equations of <strong>ConvLSTM</strong> and diagram of the inner structure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*V1rAzn1vSAv51aNvkrN_KA.png" /><figcaption>figure 1. Image from <a href="http://homepage.divms.uiowa.edu/~zhuoning/papers/p984-yuan.pdf">http://homepage.divms.uiowa.edu/~zhuoning/papers/p984-yuan.pdf</a></figcaption></figure><p><strong>How it works:</strong> The goal of this empirical research is to engineer previous sequence of accident data to predict a fixed length of time period crash locations in Cobb County using <strong>ConvLSTM </strong>network. Each accident has a recorded time and location coordinate. Imagining each time period i.e. every two hours is a picture frame in the spatio -temporal space. When stacking these crash picture frames and feeding into the <strong>ConvLSTM </strong>network, we can predict fixed length i.e next certain hours of traffic accident picture frames. In order to fit the data into the model, the data will be formulated as 3D tensors such as images and stack as sequences of 4D tensors with features as described above as fourth dimension. (sequence, imageH, imageW, features). The entire Cobb County region is partitioned into 45hx41w (1845 total with 0.25 mile each) square bins using <strong>ArcPy</strong>. In this initial project, I use seven day crash sequence data as inputs of the model and predict the next day crash image map because County level crash data is very sparse and has strong weekly pattern. There will be another project to predict next two hour crash using the same model and algorithm. When applying<em> </em><a href="https://github.com/gitWayneZhang/my_Spark/blob/master/right_angle%20crash%20prediction.ipynb"><em>the </em><strong><em>XGBoost and RandomForest</em></strong><em> models</em></a><em>,</em> it concludes 11 important features that causes traffic accidents in Cobb. These feature attributes include time invariant variables, such as road length, road curvature, average slope, population density, etc. and time variant variables such as weather, time of the day and accidents and locations etc. These features are formatted as 4th dimension of 4D tensor (7, 45, 41, 11). Three years’ Cobb county accident data is processed and batched into the <strong>ConvLSTM</strong> model with 90% training sequences and 10% percent sequences for testing. Each sequence is shifted one day with total of 1096 sequences for training and testing. The output prediction is the next day crashes happen in the 1845 bins (see figure 2 explanatory picture to display stacked County crash bin maps. The blue bins are the ones where accidents happen in that time period)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1019/1*wty2Vy9BRLizhuakUpBoBg.png" /><figcaption>figure 2. Image by author</figcaption></figure><p><strong>Build the model, train the data and make the risk prediction</strong></p><p><strong>Behind the scenes</strong>: A typical deep-learning model is to engineer the dataset, build the model, train the data, make the prediction and deploy the model. The model was trained for 50 epochs with <strong><em>Binary-crossentropy</em></strong> and <strong><em>Adam</em></strong> optimizer (in reality, it can be trained several hundred epochs). The result turns out to be very promising. The above image gallery displays some of the steps except the data engineering part. The ground truth along with the prediction sample of Dec 12th, 2020 is randomly selected which is not used in the training. By visually comparing the ground truth and prediction using certain probability threshold, the model can predict the most matched crash locations and patterns on that day. see figure 3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ypfksv33Fdr8GZA6JWEjIQ.png" /><figcaption>figure 3. Image by author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/813/1*-Cf8ScC-ErdnXY8SnjGcTQ.png" /><figcaption>Image by author</figcaption></figure><p><strong>Results:</strong> With a few lines of <strong>ArcPy</strong> code, we can add the model output records to the grid feature class and map in <strong>ArcGIS</strong> Pro. It turns out to be easier to visualize in <strong>ArcGIS Pro </strong>with symbolized roads and other features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/802/1*XvTJvkrhPIOgqfVRn2xntQ.png" /><figcaption>Image by author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3huRMp35b63FoW5-nI3fwA.png" /><figcaption>Image by author</figcaption></figure><p><strong>Why it matters:</strong> This empirical research project shows how we can use <strong>ArcPy</strong> and deep-learning techniques such as <strong>ConvLSTM</strong> to find a promising solution for crash prediction. The entire workflows and data processes can be further enhanced with more engineering features and data, automated, deployed and mapped in<strong> ESRI ArcGIS </strong>Pro desktop or make as an <strong>ArcGIS </strong>enterprise service. As a result, the same methodology can be applied to spatio-temporal solutions for traffic flow, rainfall, crime prediction and more you can think of benefitting of humanity.</p><p>The world today presents us with many challenges. However, with the power of <strong>AI</strong> and deep learning, we can work together to seek solutions and make the world a much better place to live. Thank you for reading my post.</p><p><strong>References</strong></p><p>[1] Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-kin Wong, Wang-chun Woo, <a href="https://arxiv.org/abs/1506.04214"><em>Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting. </em></a>Computer Vision and Pattern Recognition (cs.CV) Cite as:<a href="https://arxiv.org/abs/1506.04214">arXiv:1506.04214</a> [cs.CV]</p><p>[2] Zhuoning Yuan, Xun Zhou, Tianbao Yang, <a href="http://homepage.divms.uiowa.edu/~zhuoning/papers/p984-yuan.pdf"><em>Hetero-ConvLSTM: A Deep Learning Approach to Traffic Accident Prediction on Heterogeneous Spatio-Temporal Data</em></a><em>. </em>KDD 2018, August 19–23, 2018, London, United Kingdom.</p><p>[3] Sobhan Moosavi, Mohammad Hossein Samavatian, Srinivasan Parthasarathy, Radu Teodorescu, Rajiv Ramnath, <a href="https://arxiv.org/abs/1909.09638"><em>Accident Risk Prediction based on Heterogeneous Sparse Data: New Dataset and Insights</em></a><em>. </em>Machine Learning (cs.LG); Databases (cs.DB); as:<a href="https://arxiv.org/abs/1909.09638">arXiv:1909.09638</a> [cs.LG].</p><p>[4] Jeremiah Roland *, Peter D. Way, Connor Firat, Thanh-Nam Doan, Mina Sartipi, <a href="https://www.sciencedirect.com/science/article/pii/S0001457520316808?via%3Dihub">Modeling and predicting vehicle accident occurrence in Chattanooga, Tennessee, </a>Accident Analysis and Prevention 149 (2021) 105860</p><p>[5] <a href="https://pro.arcgis.com/en/pro-app/latest/arcpy/get-started/what-is-arcpy-.htm">ESRI ArcPy</a></p><p>[6] Daniel Wilson, <a href="https://storymaps.arcgis.com/stories/19d90071cee34cf09acce4d00d758357"><em>Feature Engineering for Car Crash Prediction</em></a><em>. </em>Story Map May 19, 2020</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=411909ed2cfa" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/spatial-temporal-convlstm-for-crash-prediction-411909ed2cfa">Spatio-Temporal ConvLSTM for Crash Prediction</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Interesting Animation of Covid-19 cases with Charts and Map using gganimate and tmap in R | Data…]]></title>
            <link>https://medium.com/@ellenebay88/interesting-animation-of-covid-19-cases-with-charts-and-map-using-gganimate-and-tmap-in-r-data-db7e2bb7c28f?source=rss-222a9011dd53------2</link>
            <guid isPermaLink="false">https://medium.com/p/db7e2bb7c28f</guid>
            <category><![CDATA[covid19]]></category>
            <category><![CDATA[gganimate]]></category>
            <category><![CDATA[tmap]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[r]]></category>
            <dc:creator><![CDATA[Chunguang (Wayne) Zhang]]></dc:creator>
            <pubDate>Wed, 22 Jul 2020 16:01:40 GMT</pubDate>
            <atom:updated>2020-08-02T23:59:21.151Z</atom:updated>
            <cc:license>http://creativecommons.org/publicdomain/zero/1.0/</cc:license>
            <content:encoded><![CDATA[<h3>Interesting Animation of Covid-19 cases with Charts and Map using gganimate and tmap in R</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NOeVAparrd6V8h20.jpg" /></figure><p>Covid-19 has drawn many people’s attention to creating charts and graphs for data analysis and visualization. In the last month, I spent some time experiencing <a href="https://www.datadriveninvestor.com/glossary/r/">R</a> studio and am impressed with its powerful statistic analysis package in every way, especially with their charting capability. The nice thing about it that you can also integrate with spatial analysis with the data.</p><p>Animation of data-driven story can naturally capture the interest of the readers and makes an impression. Creating animations with data can engage readers in ways other visualization tools may not be able to. With several easy to use R packages, making beautiful animation chart could be very simple.</p><h3>Covid-19 Data in Cobb County</h3><p>First, let’s bring in data. Cobb Covid-19 daily cases that you can download from Cobb and Douglas public health Covid-19 daily report website <a href="https://reports.mysidewalk.com/17554e75c1#c-452024"><em>Covid-19 daily report</em></a>. The spatial data is in geojson format with only polygons and no data. figure 1</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*lRAR5bqZDSdXe8Yk.png" /><figcaption>figure 1. Cobb and Douglas County Public health web site download graph</figcaption></figure><p>You can click the button on top to download daily report as a CSV file like form shows in figure 2. so you can join to spatial data with CSV.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*yK5p8wvb4SaWmuu9.png" /><figcaption>figure 2. Covid-19 daily case report CSV format.</figcaption></figure><p>Load required Library packages in R. With several lines of R code, you filter out just Cobb county spatial geojson</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*nEbeH2J7CEXD3Yr9.png" /></figure><p>Now let’s format the date field to gather all date cases into one column yearDate with GATHER function and join with spatial data. The yearDate field will be used for animation series to save each frame into GIF. First to convert SF object to a data frame in order to feed into ggplot graph function.</p><h3>Fun animations</h3><p>Before you do animation, you should be carefully thinking which visualization changes you would bring in to engage readers. The most common type is the result of updated data in a series of dates.</p><p>This is the first ZIP code point animation from May 17th- June 29th in Cobb:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*hUnWyWZb2tbU95FH.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*iZRQQnLt5BS1eVVj.gif" /><figcaption>figure 3. Cobb case date animation in ZIP code</figcaption></figure><p>The CSV contains a city column, you can animate city case by line graphs with R script like below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*SnTHd9L-9mIeUwWW.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*XZ711N3iXx5m69Np.gif" /><figcaption>figure 4: Cobb case date animation in City</figcaption></figure><p>This bar chart is credited from <a href="https://www.datadriveninvestor.com/glossary/github/">Github</a> resources that are generated the top 10 Covid-19 cases in Cobb from May 17th to June 29th.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*W6FMN2vU0e1NDRea.gif" /><figcaption>figure 5. Bar chart animation of top 10 ZIP code Covid-19 cases in Cobb</figcaption></figure><p>The last one, I experienced Animate ZIP code cases using map. Here we use (tmap) package. I should say we don’t see a clear pattern from this Tmap animation so it may not be a good usage of map animation in this small geographic region.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*vWsCJ3b9XvnBy0J-.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*tsQpe6yzsLDG4cyy.gif" /><figcaption>figure 6. Animation map by ZIP code cases</figcaption></figure><p><strong>My thinking</strong></p><p>Animation is fun when you apply appropriate charts and graphs. And they engage readers in a way other visualization tools may not be able to. Now you can incorporate these tools into your presentations, websites, and blogs. The key point is to highlight meaningful changes in the data over the attribute which is driving the animation. Thanks so much for reading! Code is <a href="https://github.com/gitWayneZhang/R_Projects"><em>here</em></a></p><p><em>Originally published at </em><a href="https://www.datadriveninvestor.com/2020/07/22/interesting-animation-of-covid-19-cases-with-charts-and-map-using-gganimate-and-tmap-in-r/"><em>https://www.datadriveninvestor.com</em></a><em> on July 22, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=db7e2bb7c28f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Urban water inlet features detection from EagleView aerial imagery using Mask R-CNN/Keras/ArcGIS]]></title>
            <link>https://medium.com/data-science/urban-water-inlet-features-detection-from-eagleview-aerial-imagery-using-mask-r-cnn-keras-arcgis-fd3f2ae062bf?source=rss-222a9011dd53------2</link>
            <guid isPermaLink="false">https://medium.com/p/fd3f2ae062bf</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[mask-rcnn]]></category>
            <dc:creator><![CDATA[Chunguang (Wayne) Zhang]]></dc:creator>
            <pubDate>Sat, 04 Jan 2020 01:07:21 GMT</pubDate>
            <atom:updated>2020-08-01T14:15:52.759Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-nc/4.0/</cc:license>
            <content:encoded><![CDATA[<h3><strong>Urban water inlet features image segmentation from EagleView aerial imagery using Mask-RCNN/</strong>Keras/ArcGIS</h3><p><em>A promising deep learning practice</em></p><p>In my last <a href="https://towardsdatascience.com/eagleview-super-high-resolution-image-segmentation-with-deeplabv3-mask-rcnn-using-keras-arcgis-9be08caac42c"><strong>blog</strong></a>, I talked about image segmentation of building footprint with EagleView super-high-resolution imagery using Mask R-CNN and deepLabV3+ models. Although the outputs from both deep learning models are generally promising, they are still missing a certain threshold of accuracy. Why is that? Because of the GIS data for ground truth labels are not validated by anyone, many ground truth labels are either not accurate or missed the actual true object delineation, which could potentially confuse the model classifier. My curiosity: what if I verify each ground truth label and delineate each feature as accurate as possible, would be the output better and more promising? What would be the statistics of output like Precision and Recall of the trained model prediction? Would the prediction of the model be applied in practical work to save potential time-consuming job by human digitizer?</p><p>This time I chose the water inlet as a target of the training model using Mask R-CNN. Training images to find the water inlet features present several challenges. Water inlet follows a general oval and triangular pattern but with a highly varied color scheme due to the years of environmental weathering, sometimes it is even hard to detect with human eyes on aerial images. The model has to learn not to be fooled by road and house features, which can trace superficially similar patterns and shapes.</p><p>To test the experiment and better understand the model output, I personally identified 178 water inlet features (based on previous vendor digitized features) on top of one tile of 1 x 1 mile EagleView (Pictometry ) 3-in resolution aerial image (fig.1 and fig.2) and digitize each as a polygon feature using ArcGIS software from ESRI for ground truth labels. After training a deep neural network to identify water inlet features from the tile image with Mask R-CNN, another different location of 1 x 1 mile tile image was fed to the trained model to see if the model could spot water inlet features in the image that digitizer had missed. The process turns out a much better result as the model discovered more water inlet features that the human digitizer had been either unnoticed or missed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/1*LjdWhzxgiHQ766ovuHJsiA.jpeg" /><figcaption>fig. 1: Red outline tile image for training data and blue outline tile image for inference.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/1*vWJyIa0yyCJgRwaxzEzQ_A.jpeg" /><figcaption>fig.2: Zoom-in tile image of dimension 20,000 x 20,000 x3 RGB bands used for training.</figcaption></figure><p><strong><em>Training with Mask-RCNN model:</em></strong></p><p>Mask R-CNN is a state of art deep neural network to solve instance segmentation problems in machine learning. Mask-RCNN efficiently detects objects in an image while simultaneously generating a high-quality segmentation mask for each instance. The method extends Faster R-CNN by adding a branch for predicting an object mask in parallel with the existing branch for bounding box recognition. You can read the <a href="https://arxiv.org/abs/1703.06870">research paper</a> to better understand the model (fig. 3).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/447/0*KdaYq_CJMWf4jcAP.jpeg" /><figcaption>fig. 3: Mask R-CNN framework for instance segmentation. Source: <a href="https://arxiv.org/abs/1703.06870">https://arxiv.org/abs/1703.06870</a></figcaption></figure><p>The <a href="http://pro.arcgis.com/en/pro-app/tool-reference/spatial-analyst/export-training-data-for-deep-learning.htm">“Export Training Data for Deep Learning”</a> in ArcGIS Pro was used to export tile images and water inlet polygon features for instance segmentation datasets (fig. 4). The dimension of the output images is 512x512x3 and rotation is set to 90 degrees to generate more images to prevent overfitting and help the model generalize better. There are traditional image augmentation methods that you can apply to generate more simulated images that you can refer to here the <strong>i</strong><a href="https://github.com/aleju/imgaug"><strong>mgaug</strong></a> Python library. For this experiment, I skip the augmentation techniques as the rotation of the cropped images did generate some augmentation training dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CCBAtJoeeTUOu8HOX5U7cQ.jpeg" /><figcaption>fig. 4: Water inlet polygon features (red) overlay the image used for generating training datasets</figcaption></figure><p>The resulting training datasets contain over 1800 images and labels. With further data processing to remove no labeling images, final datasets had over 1764 training images and labels. I used 32 GB CPU Dell notebook to train the model. This is quite a bit small dataset to train a deep neural network to extract an urban feature from aerial images. With applied transfer learning, I start with a weights file that’s been trained on the ImageNet dataset. Although the ImageNet dataset does not include a water inlet class, it includes many other kinds of images, so the trained weights have already learned a lot of the features common in natural images, which helps training the model.</p><p>I used the impressive open-source implementation <a href="https://github.com/matterport/mask_rcnn">Mask-RCNN library</a> that MatterPort built on Github <a href="https://github.com/matterport/Mask_RCNN">here</a> to train the model.</p><p>There are three main functions need to be modified in the class (Utils.dataset) to load your own datasets into the framework. See below for the data loading implementation(fig 5). The anchor’s ratio is setup (16,32,64,128,256) to predict a smaller size of the water inlet features. IMAGES_PER_GPU set to =1 so CPU can be used to train the model. An example of an image and mask(fig. 6).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v8IMPcPY3cAKNVWvTg1-Dg.jpeg" /><figcaption>fig 5 load training images and labels to the model</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aaisApKmucZgXSIXE5Hpxw.jpeg" /><figcaption>fig. 6: A random image and label mask for training</figcaption></figure><p>The transfer learning technique was applied with model backbone ResNet-101. I trained the last fully connected layers first with 8 epochs to adapt the water inlet feature class, then trained the full network for 100 epochs(fig.7)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ickcMQKPHmRHLmhpHemKZQ.jpeg" /><figcaption>fig.7: training head layers to adapt the water inlet features</figcaption></figure><p>At a 32-GB CPU, it took over 5 days to finish the training process (fig.8). The charts indicate the loss of validation dataset plateaued after certain epochs. This normally happens as overfitting when you have a small training dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NCZAvrtLYoYw3519PcAkwg.jpeg" /><figcaption>fig.8: training full network layers 5 days and 18hrs</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7KYwTw5LM0yNpBR1kRvPDw.jpeg" /><figcaption>fig. 9: training loss charts</figcaption></figure><p>Here is a test image that was not in the training set and there is one water inlet feature was not identified in the digitized polygons(fig. 10)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*g6yNe1RXsKKIGSV6XY5u7g.jpeg" /><figcaption>fig. 10: a test image and label with missing detection by the human digitizer</figcaption></figure><p>The water inlet feature inference from the above image for comparison. Interesting to see a missing water inlet feature was detected by inference.(fig.11)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/1*hHFaG09sQXo5H-naInWZbQ.jpeg" /><figcaption>fig.11: the missing water inlet feature was identified with inference from the above image.</figcaption></figure><p>Another example from the test image with inference(fig.12 and fig.13).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/764/1*UY3Q5cIqqv8NZwZqbSPFvw.jpeg" /><figcaption>fig. 12: a test image</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*xycAa30TiXJDQo3P25C4Mg.jpeg" /><figcaption>fig.13: the inference from the above test image.</figcaption></figure><p>Then I run python inference scripts with one complete 2019 tile image of 20,000 x 20,000 x 3 dimension which is 3 miles away from the tile image that was used in the training (fig. 14). The scripts crop and process 1600 (512x512x3 dim) images for the inference. It took approximately one hour to finish the process using 32GB RAM of CPU laptop. There are missed water inlet delineations mostly because of using the very small training dataset and trees’ covering on top of the water inlet features. Choosing several representative tiles as training datasets from different locations of the County could improve the accuracy of the result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j0HVdl_yPatf3gWFiZR74A.jpeg" /><figcaption>fig.14: a complete tile image inference which equals 1600 (512x512x1) cropped rasters</figcaption></figure><p>I would like to use statistic Precision and Recall to measure output performance. For those of you not familiar with the terms, here are the basic:</p><p><strong>Precision:</strong> the fraction of the true positive instances among all positive instances classified by the model</p><p><strong>Recall:</strong> the fraction of the total amount of positive instances that are actually retrieved by the classified model.</p><p>With the threshold of binary cross-entropy confidence set to ≥0.9, the model selected 271 candidates within a 1 -square-mile inference tile range. Then I reviewed the candidates manually and choose the most promising one and confirmed 194 in the aerial imagery. Precision is 71.58%. The recall is 70.78%. When the threshold of confidence is set to ≥0.75, the precision is dropped to 53.98%. but Recall is up to 78.11% which means more water inlets are correctly identified and also many incorrect detected water inlet features. Depending on the project, we would more likely to maximize recall to retrieve more water inlet features and then process quickly to eliminate the false positives manually. Below is the zoom-in of the inference water inlet features that overlay with aerial imagery (fig.15).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/804/1*S1P_ALpR-YYjSB0YOoxkVA.jpeg" /><figcaption>fig.15: from inference: a false positive example and positive water inlet features (red)</figcaption></figure><p><strong><em>Conclusion:</em></strong></p><p>Although it is a relatively very small dataset, Mask-RCNN presents promising results for super-high-resolution image segmentation using the transfer learning techniques. The result of the performance could be further enhanced by increasing high-quality training datasets from different locations of the county and applying data variation augmentation methods. The model can also be used in multi-year imagery to infer feature detection for comparison or even used for low-cost feature delineation with ArcGIS tools ModelBuilder or any open source codes to automate the business tasks. More importantly, the above deep learning training process can be applied to other types of image instances or segmentation cases. To sum up, I should mention an interesting fact: when I compare the results from confidence level ≥0.75 with the polygon features that vendor digitized on the inference tile. The model delineates 67 more water inlet features than the human digitizer accomplished but misses 45 human digitized inlet features. This convinces me that if we apply the right deep learning model and do it right, ML or deep learning can be utilized in practical real-world projects and save users a massive amount of time and creates a brand new experience.</p><p><em>1. </em><a href="https://arxiv.org/search/cs?searchtype=author&amp;query=He%2C+K">Kaiming He</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Gkioxari%2C+G">Georgia Gkioxari</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Doll%C3%A1r%2C+P">Piotr Dollár</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Girshick%2C+R">Ross Girshick</a> (2018). Mask-RCNN<em>, </em><a href="https://arxiv.org/abs/1703.06870v3"><em>https://arxiv.org/abs/1703.06870v3</em></a></p><p><em>2.</em> Mask R-CNN <a href="https://github.com/matterport/Mask_RCNN"><em>https://github.com/matterport/Mask_RCNN</em></a></p><p><em>3. </em>U-Net: Convolutional Networks for Biomedical Image segmentation<a href="https://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/"><em>:https//lmb.informatik.uni-freiburg.de/people/ronneber/u-net/</em></a></p><p><em>4.</em> <a href="http://pro.arcgis.com/en/pro-app/tool-reference/image-analyst/export-training-data-for-deep-learning.htm"><em>http://pro.arcgis.com/en/pro-app/tool-reference/image-analyst/export-training-data-for-deep-learning.htm</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fd3f2ae062bf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/urban-water-inlet-features-detection-from-eagleview-aerial-imagery-using-mask-r-cnn-keras-arcgis-fd3f2ae062bf">Urban water inlet features detection from EagleView aerial imagery using Mask R-CNN/Keras/ArcGIS</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[EagleView Super-High-Resolution Image Segmentation with Deeplabv3+ /Mask-RCNN using Keras/ArcGIS]]></title>
            <link>https://medium.com/data-science/eagleview-super-high-resolution-image-segmentation-with-deeplabv3-mask-rcnn-using-keras-arcgis-9be08caac42c?source=rss-222a9011dd53------2</link>
            <guid isPermaLink="false">https://medium.com/p/9be08caac42c</guid>
            <category><![CDATA[deeplabv3]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[geoai]]></category>
            <category><![CDATA[mask-rcnn]]></category>
            <dc:creator><![CDATA[Chunguang (Wayne) Zhang]]></dc:creator>
            <pubDate>Tue, 10 Dec 2019 17:06:02 GMT</pubDate>
            <atom:updated>2020-08-01T14:14:46.131Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>EagleView high-resolution image semantic segmentation with Mask-RCNN/DeepLabV3+ using Keras and ArcGIS Pro</strong></h3><p>Computer vision in Machine Learning provides enormous opportunities for GIS. Its tasks include methods for acquiring, processing, analyzing and understanding digital images, and extraction of high-dimensional data from the real world in order to produce numerical or symbolic information, e.g. in the forms of decisions.[1][2][3][4] In the last several years, computer vision is increasingly shifting from traditional statistical methods to the state-of-art deep learning neural network techniques.</p><p>In this blog, I will share several empirical practices using Keras and ESRI ArcGIS Pro tools with deep learning and transfer learning techniques to build a building footprint image segmentation network model from a super-high-resolution 3-inch of EagleView (Pictometry) imagery.</p><p>In 2018, <a href="https://blogs.technet.microsoft.com/machinelearning/2018/03/12/pixel-level-land-cover-classification-using-the-geo-ai-data-science-virtual-machine-and-batch-ai/">ESRI and Microsoft collaborated </a>with the Chesapeake Conservancy to train a deep neural network model to predict land cover from 1-meter NAIP resolution aerial imagery data source. The neural network similar in architecture to Ronnenberger et al.’s U-net (<a href="https://arxiv.org/abs/1505.04597">2015</a>), a commonly-used semantic segmentation model was used in that case. Each year, the GIS Core group in Cobb County Georgia receives 3-inch super-high-resolution ortho imagery from EagleView(Pictometry). Could deep learning models be applied to this super-high-resolution ortho imagery to classify land cover or extract building footprints? There are several challenges — Super-high-resolution imagery usually presents varieties of vegetation types and overlaps; buildings and trees creating heavy shadows in the images, which could potentially misclassify the true ground objects.</p><p>In the beginning, I was very conservative as I decided to use a CPU-only laptop to train roughly 3800 images. Considering the complexity of land cover and building footprints, this is quite a small dataset for deep learning because if you read textbooks, often say deep learning requires a huge amount of training data for better performance. But it is also a realistic classification problem: in a real world-cases, even small-scale image data can be extremely hard to collect and expensive or sometimes almost impossible. Being able to use small datasets and train a powerful classifier is a key skill for a competent data scientist. After many tries and runs, the results turn out very promising especially with state-of-the-art Deeplabv3+ and Mask-RCNN models.</p><p><strong><em>Study Area and training image dataset preparation</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/1*ObGiKFHJyOGYoOdlNo4hBQ.jpeg" /><figcaption>fig.1 — Cobb County 2018 3in EagleView imagery covers with 433 1x1 mile tiles.</figcaption></figure><p>The geographical area of Cobb County covers with 433 of 1 x 1 mile Pictometry image tiles at a resolution of 3-inch. The county GIS group has a building footprint polygon layer in certain areas. For training purposes, one image tile close to the center of the County was chosen for the image training dataset(fig. 1). The building footprint polygon feature layer was used to process as ground truth mask labels. The <a href="http://pro.arcgis.com/en/pro-app/tool-reference/spatial-analyst/export-training-data-for-deep-learning.htm">“Export Training Data for Deep Learning”</a> in ArcGIS Pro 2.4 ver. of Geoprocessing tool was used to export images and masks for instance segmentation datasets(fig.2). The dimension of the output images are 512x512x3 and rotation is set to 90 degrees to generate more images to prevent overfitting and help the model generalize better.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*d4XbEUob65UqjdpcMfTmkA.png" /><figcaption>Fig. 2 — ArcGIS <a href="http://pro.arcgis.com/en/pro-app/tool-reference/spatial-analyst/export-training-data-for-deep-learning.htm">“Export Training Data for Deep Learning”</a></figcaption></figure><p><strong><em>1. Training with Mask-RCNN model</em></strong></p><p>The resulting training datasets contain over 18000 images and labels. With further data processing to remove no labeling images, final datasets had over 15000 training images and labels. However, with CPU only 32-GB memory laptop, it is impossible to feed such large datasets into the Mask-RCNN model which requires huge memory for the training.</p><p>The training strategy is to see how the proof of concept will work, so I gradually increased the datasets to feed into CNN with a trial of 3800 datasets.</p><p>I used the impressive open-source implementation <a href="https://github.com/matterport/mask_rcnn">Mask-RCNN library</a> that MatterPort built on Github <a href="https://github.com/matterport/Mask_RCNN">here</a> to train the model.</p><p>Mask-RCNN efficiently detects objects in an image while simultaneously generating a high-quality segmentation mask for each instance. The method extends Faster R-CNN by adding a branch for predicting an object mask in parallel with the existing branch for bounding box recognition[5]. You can read the <a href="https://arxiv.org/abs/1703.06870">research paper</a> to better understand the model. (fig. 3).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/447/1*RN9oNruYzQhWsx8xbwjgtA.jpeg" /><figcaption>fig. 3. — Mask R-CNN framework for instance segmentation. Source: <a href="https://arxiv.org/abs/1703.06870">https://arxiv.org/abs/1703.06870</a></figcaption></figure><p>There are three main functions need to be modified in the class (Utils.dataset) to load your own datasets into the framework. See below for the data loading implementation. The anchor&#39;s ratio is setup (16,32,64,128,256) to predict a smaller size of residential buildings. IMAGES_PER_GPU set to =1 so CPU can be used to train the model (fig. 4). An example of an image and mask (fig. 5).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*miaDnUp214CkZEkEUhzjcA.jpeg" /><figcaption>fig. 4 — Load Cobb Pictometry datasets to Mask-RCNN framework.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kqVhpA7YvQZPOW6HV6nidw.jpeg" /><figcaption>fig. 5 — An example of a random image and mask from datasets.</figcaption></figure><p>Here, the transfer learning technique was applied with model backbone ResNet-101. I trained the last fully connected layers first with epoch =5 to adapt the residential building class, then trained the full network for 35 epochs.</p><p>At a 32-GB CPU, it took nearly 48 hours to finish the training process (fig. 6 and fig.7).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3Xd0-ICGsC8Oe1VSm_Roxg.jpeg" /><figcaption>fig. 6 — Model training result. The loss is reasonable good.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R3PppHklWwg5n_7Q1SYe4Q.jpeg" /><figcaption>fig. 7 — Loss charts.</figcaption></figure><p>Here are two inferences that original images were not used in training (fig.8 and fig.9). Interestingly to see the inference mask is more accurately delineating the building than original mask.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*L8LNrtpQ8uY7mf1kjO7wWQ.png" /><figcaption>fig. 8 — Original image that was not used in training</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*6ZK6FufcBxswIzUmwd07tw.png" /><figcaption>fig. 9 — Inference mask more accurately delineating the building than the original mask.</figcaption></figure><p>Another interesting example of one image that was not used in training and mask inference result (fig.10 and fig.11).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/1*fb1a3ARQIqOCErDtrdSGgg.png" /><figcaption>fig. 10 — An image that was not used in training.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*k2dQCWLX-aF64pU6UTkeMw.png" /><figcaption>fig. 11 — Inference instance masks.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fkudrNlmUZP6N3_O-dp6uA.jpeg" /><figcaption>fig. 12 — This is a cropped image and inference mask not used in the training. The orange line indicates the image cropped position. With 3000 training datasets, the result is very promising.</figcaption></figure><p><strong><em>2. Training with Deeplabv3+ model</em></strong></p><p>Deeplabv3+ is the latest state-of-art semantic image segmentation model developed by <a href="https://ai.googleblog.com/2018/03/semantic-image-segmentation-with.html">google research team</a>. The distinctive of this model is to employ atrous convolution in cascade or in parallel to capture multi-scale context by adopting multiple atrous Rates (fig.13).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jV_o9Pk4xRicykZS.png" /><figcaption>fig.13 — <a href="https://ai.googleblog.com/2018/03/semantic-image-segmentation-with.html">https://ai.googleblog.com/2018/03/semantic-image-segmentation-with.htm</a>l</figcaption></figure><p>With the same training datasets from ArcGIS export training data for a deep learning tool, the images and masks were processed with augmentations and saved to a HDF5 compressed file for conveniently loading to training model (fig.14).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*s115kSMshIyb3doxT-76zw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*mbYSVIOvx1MZcGUt6LqHHw.jpeg" /><figcaption>fig. 14 — A random example of image and mask.</figcaption></figure><p>I used Keras implementation of Deeplabv3+ on Github <a href="https://github.com/bonlime/keras-deeplab-v3-plus">here</a>. Below is the Keras training model with backbone Mobilenetv2 which has fewer parameters than the Xception model (fig.15).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6s-RZdY34LHJQdiQG26NPw.jpeg" /><figcaption>fig.15 — Define DeepLabv3+model.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4naBv1_WTdifmTFL6dfGGA.jpeg" /><figcaption>fig. 16 — Five epochs training result.</figcaption></figure><p>With only 5 epoch training runs, the result turns out very promising(fig.16).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eDyoiyiC8JrO0lyDwISN0w.jpeg" /><figcaption>fig. 17 — Training loss converged plot.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9B416Kk-6NKrrBN5CowEcg.jpeg" /><figcaption>fig. 18 — Image and inference from trained year 2018 imagery of Deeplabv3+ model</figcaption></figure><p>I run Python scripts for inference of an arbitrary cropped 2064x1463 dimension image that crop and process 16 (512x512x3 dim) images to obtain the inference rasters. (fig.19). With further inspection of the images and inference, we can see the effect of the building shadow can lower the accuracy of the edge of the buildings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IgD1Dc5t9selO8vieTsTnA.jpeg" /><figcaption>fig. 19 — A cropped 2018 EagleView image with inference raster (25 rasters of 512x512 dim)</figcaption></figure><p>With the same trained model to predict the 2019 same area cropped the image, the result is very similar with only minor localized differences(fig.20). The model can really help in future year image inference.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a8gAaMNeFM-SNT9VSnDEWQ.jpeg" /><figcaption>fig. 20 — A cropped 2019 EagleView image with inference raster (25 rasters of 512x512 dim)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*HOv51eNcBVU9su0UjbeHrg.jpeg" /><figcaption>fig. 21 — 2018 inference raster overlay the original image.</figcaption></figure><p>After adding the above image and inference to the ArcGIS Pro. (fig.21)</p><p>The above image raster was converted to the polygon feature and then use <a href="https://pro.arcgis.com/en/pro-app/tool-reference/3d-analyst/regularize-building-footprint.htm">Regularize Building Footprint</a> in ArcGIS Pro 3D analysis with appropriate parameters to regularize raw detection. (fig.22)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/1*qHb_Ve7Uvs1t8sSnzKEDBA.jpeg" /><figcaption>fig. 22 — Use ArcGIS Pro Regularize Building Footprint tool to clean up building polygons.</figcaption></figure><p>Then I run python inference scripts with two complete 2018 tile images of 20,000 x 20,000 dimension about 3-mile away from each other. The scripts crop and process 1600 (512x512x3 dim) images for the inference. It took approximately one hour to finish each tile using 32GB RAM of CPU only laptop. see (fig. 23). There are missed classified buildings mostly because of using very small training dataset and trees covering on top of the buildings. Choosing several representative tiles as training dataset from different locations of the County could improve the accuracy of the result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N4tm79l6ZEl1Kjy8SLEU2w.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9NDX0QYJTrIJwWhm6hxI6Q.jpeg" /><figcaption>fig. 23 — Two complete tiles of EagleView 2018 image with inference raster (1600 rasters of 512x512 dim on each)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/1*4otAOO3EOFVJ6hXKUWhDcw.jpeg" /><figcaption>fig.24 inference raster overlay the image tile in ArcGIS Pro.</figcaption></figure><p><strong>Conclusion:</strong></p><p>Although it is a relatively small dataset, Mask-RCNN, and Deeplabv3+ deep learning models both present promising results for super-high-resolution image segmentation using transferred learning techniques. Due to the less accuracy of original building footprints ground truth feature polygons and the laptop CPU and memory limitation, the result of the performance may not surpass human digitizer in some image classifications and instance segmentation. However, the accuracy of this Deep learning training process can be further enhanced by increasing high-quality training datasets from different locations of the county and applying data variation augmentation methods. The model can be used in multi-year imagery to infer feature detection for comparison or even used for low-cost feature delineation with ArcGIS tools ModelBuilder to automate the business tasks. More importantly, the above deep learning training process can be applied to other types of image instances or segmentation cases. (<a href="https://towardsdatascience.com/urban-water-inlet-features-detection-from-eagleview-aerial-imagery-using-mask-r-cnn-keras-arcgis-fd3f2ae062bf"><strong>please see my next blog</strong></a><strong>)</strong></p><p><em>1.Reinhard Klette (2014). Concise Computer Vision. Springer. </em><a href="https://en.wikipedia.org/wiki/International_Standard_Book_Number"><em>ISBN</em></a><em> </em><a href="https://en.wikipedia.org/wiki/Special:BookSources/978-1-4471-6320-6"><em>978–1–4471–6320–6</em></a><em>.</em></p><p><a href="https://en.wikipedia.org/wiki/Linda_Shapiro"><em>2.Linda G. Shapiro</em></a><em>; George C. Stockman (2001). Computer Vision. Prentice Hall. </em><a href="https://en.wikipedia.org/wiki/International_Standard_Book_Number"><em>ISBN</em></a><em> </em><a href="https://en.wikipedia.org/wiki/Special:BookSources/978-0-13-030796-5"><em>978–0–13–030796–5</em></a><em>.</em></p><p><em>3.Tim Morris (2004). Computer Vision and Image Processing. Palgrave Macmillan. </em><a href="https://en.wikipedia.org/wiki/International_Standard_Book_Number"><em>ISBN</em></a><em> </em><a href="https://en.wikipedia.org/wiki/Special:BookSources/978-0-333-99451-1"><em>978–0–333–99451–1</em></a><em>.</em></p><p><em>4.Bernd Jähne; Horst Haußecker (2000). </em>Computer Vision and Applications, A Guide for Students and Practitioners. Academic Press<em>. </em><a href="https://en.wikipedia.org/wiki/International_Standard_Book_Number"><em>ISBN</em></a><em> </em><a href="https://en.wikipedia.org/wiki/Special:BookSources/978-0-13-085198-7"><em>978–013085198–</em></a><em>7.</em></p><p><em>5. </em><a href="https://arxiv.org/search/cs?searchtype=author&amp;query=He%2C+K"><em>Kaiming He</em></a><em>, </em><a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Gkioxari%2C+G"><em>Georgia Gkioxari</em></a><em>, </em><a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Doll%C3%A1r%2C+P"><em>Piotr Dollár</em></a><em>, </em><a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Girshick%2C+R"><em>Ross Girshick</em></a><em>.</em>(2018). Mask-RCNN<em>, </em><a href="https://arxiv.org/abs/1703.06870v3">https://arxiv.org/abs/1703.06870v3</a></p><p>6. <em>Deeplabv3+ model,</em> <a href="https://github.com/tensorflow/models/tree/master/research/deeplab">https://github.com/tensorflow/models/tree/master/research/deeplab</a></p><p><a href="https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html"><em>7. https://blog.keras.io/building-powerful-image-classification-models-using-very-little-data.html</em></a></p><p><a href="http://pro.arcgis.com/en/pro-app/tool-reference/image-analyst/export-training-data-for-deep-learning.htm"><em>8. http://pro.arcgis.com/en/pro-app/tool-reference/image-analyst/export-training-data-for-deep-learning.htm</em></a></p><p><a href="https://pro.arcgis.com/en/pro-app/tool-reference/3d-analyst/regularize-building-footprint.htm"><em>9. https://pro.arcgis.com/en/pro-app/tool-reference/3d-analyst/regularize-building-footprint.htm</em></a></p><p><em>10.</em><a href="https://blogs.technet.microsoft.com/machinelearning/2018/03/12/pixel-level-land-cover-classification-using-the-geo-ai-data-science-virtual-machine-and-batch-ai/"><em>https://blogs.technet.microsoft.com/machinelearning/2018/03/12/pixel-level-land-cover-classification-using-the-geo-ai-data-science-virtual-machine-and-batch-ai/</em></a></p><p><em>11. U-Net: Convolutional Networks for Biomedical Image segmentation</em><a href="https://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/"><em>:https//lmb.informatik.uni-freiburg.de/people/ronneber/u-net/</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9be08caac42c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/eagleview-super-high-resolution-image-segmentation-with-deeplabv3-mask-rcnn-using-keras-arcgis-9be08caac42c">EagleView Super-High-Resolution Image Segmentation with Deeplabv3+ /Mask-RCNN using Keras/ArcGIS</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>