<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Batran on Medium]]></title>
        <description><![CDATA[Stories by Batran on Medium]]></description>
        <link>https://medium.com/@mbatran?source=rss-137c11a56fbc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*KE-pOz0q_qPibXFJANcgjg.jpeg</url>
            <title>Stories by Batran on Medium</title>
            <link>https://medium.com/@mbatran?source=rss-137c11a56fbc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 23:58:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mbatran/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Airport Passenger Journey Mapping Post COVID Era]]></title>
            <link>https://medium.com/data-science/airport-passenger-journey-mapping-post-covid-era-5864a877b47b?source=rss-137c11a56fbc------2</link>
            <guid isPermaLink="false">https://medium.com/p/5864a877b47b</guid>
            <category><![CDATA[gps]]></category>
            <category><![CDATA[mobility]]></category>
            <category><![CDATA[customer-journey]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[airports]]></category>
            <dc:creator><![CDATA[Batran]]></dc:creator>
            <pubDate>Sun, 07 Nov 2021 13:25:08 GMT</pubDate>
            <atom:updated>2021-11-09T14:43:54.602Z</atom:updated>
            <content:encoded><![CDATA[<h3>Airport Passenger Journey Mapping Post Covid Era by GPS and Python</h3><h4>A GPS data-driven time measurement of each process at Narita Airport — Japan</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Qef3UM10PBf9TPfl" /><figcaption>Photo by <a href="https://unsplash.com/@jeshoots?utm_source=medium&amp;utm_medium=referral">JESHOOTS.COM</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><blockquote><strong>Batran:</strong> Hi Omar, my flight is arriving at Narita Airport at 17:30, can you pick me up?</blockquote><blockquote><strong>Omar: </strong>Sure, happy to! When should I pick you up?</blockquote><blockquote><strong>Batran:</strong> Wait! There are lots of COVID-19 measures and I heard we need to get tested upon arrival and that might take time.</blockquote><blockquote><strong>Omar:</strong> That’s right, but you know it is quite challenging to wait at the airport so we want to estimate if you land at 17:30 — when do you expect to reach the luggage claim</blockquote><blockquote><strong>Batran: </strong>Can’t find much on the internet? If only someone has measured that time.</blockquote><p>Covid-19 has introduced many changes to the travel industry. It has also added more complexity to the already existing ones during the airport process from landing to luggage claim. A few of those changes are:</p><ul><li>Showing a negative PCR certificate</li><li>Getting your temperature tested</li><li>Getting a saliva test and waiting for the result</li><li>Filling in many documents and consents</li><li>Installing and configuring a health monitoring app</li></ul><p>With that much complexity, scheduling a pickup service or setting up any time dependant commitment is challenging. <strong>In this article, we show data-driven visual analysis on the operation flow and the time it takes from flight landing to luggage claim in Narita Airport.</strong></p><h3>Data</h3><p>Mobile phone data is one of the best ways to understand and measure the timing of the processes in different places. Most mobile phones collect user location data to provide a better service to users and they often allow users to access their data or request a copy.</p><p>Once we landed at Narita Airport, we were connected to the internet and the GPS locations started being recorded. On average one point was collected every 5 seconds. By looking at the sequence of points over time, it is quite easy to see visually where I was at which time and draw a quick conclusion on the overall distance or the total time traveled.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FZe8_9SAC7A8SfBP2Jor1A.gif" /><figcaption>Movement pattern from flight landing to baggage claim at Narita Airpot. “Image by Author”. Visualization using <a href="https://mobmap.locationmind.com/">https://mobmap.locationmind.com/</a></figcaption></figure><blockquote><em>It took us</em><strong><em> 2 hours and 20 minutes</em></strong><em> from flight landing time to submit the customs declaration form and exit the airport on August, 2021.</em></blockquote><h3>Downsampling the data points</h3><p>To reduce the complexity and see the data trends more clearly such as the stops where points accumulate. Let us reduce the number of points to be one point every 10 seconds instead of one point every 5 seconds. This process is called subsampling and it’s highly used in time-series analytics.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5gUO-DKS5GmoVblf" /><figcaption>downsampling GPS data from 5 seconds to 10 seconds “Image by Author”</figcaption></figure><p>Now that the data has been downsampled, we want to only extract the stopping points. Stopping points indicate an activity, for example when you wait in a line to take the saliva test or when you wait to collect the luggage. When someone stops in the same location, the GPS data points keep accumulating at the same point where it is easy to detect a trend.</p><h3>Detecting the walking points through speed</h3><p>To capture those stops, we first need to remove the points where we were walking or running. To do that we can calculate the speed of each point by comparing it to the point before. The average human walking speed is about 1.4 meters per second. For me I was walking with my 2 years old daughter, so the speed was barely 1 meter per second, this is in the velocity_mps, which refers to meter per second.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*F4d7EWV-xm0l0r7U" /><figcaption>Calculating speed in mile per second from GPS data “Image by Author”</figcaption></figure><p>Next, all we have to do is drop the data points with a speed that equals to or more than the average walking speed. Since my walking speed was quite low in the first place, what has been removed are high-speed points and the points when the plane was landing and moving. However with regular walking speed. Expect better results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_HxiL6SnvxVmXhfb" /><figcaption>Filtering out points with high speed from the moving GPS trajectory “Image by Author”</figcaption></figure><h3>Finding the stops by clustering</h3><p>Clustering is a technique for finding common data points, Spatiotemporal density-based clustering or STDBSCAN is an unsupervised machine learning algorithm to cluster the points that are close in space and time which can represent a potential stop. Once we run the model on the data point and adjust the parameters. We can extract relevant points that represent the stops where the movement was relatively still as below.</p><p>Every stop represents an activity or something that describes why did we stop in the first place. In most cases, we can map a stop to a shop nearby or a restaurant and so on. But in this case, it is specific to the operation so we can label those points manually. After all, they are only a few clusters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Du-CmW3c1uHq38L9" /><figcaption>Using unsupervised machine learning spatiotemporal density-based clustering to detect stops from GPS data “Image by Author”</figcaption></figure><h3>Measuring the time it takes for every stop</h3><blockquote>How long does it take for every process?</blockquote><p>If we look at every GPS cluster point, we can simply find the point with the minimum time and the point with the maximum time in each cluster. This will represent the start of the stop and the end of the stop. The difference between both is the duration of the stop. In the table below the duration of every cluster represent how long It took us to finish each process. We were quite relaxed and had no reason to rush and we had kids too. So the time for a single person in a run might be lower.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BKhoSXH3r7_uYAIU" /><figcaption>stay time measurement using GPS data through STDBSCAN clustering method “Image by Author”</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZH8UY5Qk4VxsAZ5T" /><figcaption>Airport Passenger Journey Mapping using GPS data for every process “Image by Author”</figcaption></figure><h3>Estimating the walking time in the Airport</h3><p>Walking time is the time between the different stops, this is an important part of characterizing mobility patterns. How long does it take to walk from a place to another? To calculate that from the data, the walking time can be the time between the last point of a stop and the first point of the next stop. The table below list the time spent in nonstops, this include time hanging around or going back and forth looking for the next stop or when the kids peak from the glass to look at the airplanes. Which was quite often. The time walking might be exaggerated though when some points falsely are not calculated as being part of a cluster or when a small stop is being neglected. Such as stopping by the restroom.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wmtk71h4fRIStwxu" /><figcaption>Walking time estimation between different processes at the airport</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-SdlMpwrwsXmOXfX" /><figcaption>Walking time and stop time estimation from GPS data points at Narita Airport “Image by Author”</figcaption></figure><h3>Detecting the approximate location of the stop</h3><p>Finding the approximate location of the stop can sometimes be important. When a stop happens GPS locations scatter around one point. This is because the accuracy of the GPS is usually around 4 meters in mobile phones. To find an approximation of the stop point, we can look at each GPS point cluster and estimate the location of the centroid of the cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pgq_a4Wq2UvX2AX7" /><figcaption>Estimating stay location from GPS data based on clustering “Image by Author”</figcaption></figure><h3>Final results</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*h-kHPTbrdOtQHYpJ" /><figcaption>Airport Passenger Journey Mapping Post Covid Era — Time from flight landing to luggage claim at Narita Airport “Image by Author”</figcaption></figure><h3>Conclusions</h3><ul><li>It takes about 2 hours and 30 minutes from flight landing to luggage claim and customs clearance.</li><li>Waiting for saliva test results takes most of the waiting time followed by app download and installation confirmation by the airport staff.</li><li>Walking from the airplane to the saliva test location took most of the time</li><li>Luggage claim was quite fast as there are very few people using Narita Airport at the moment</li></ul><h3>Business opportunities for this / similar analysis</h3><p>Designing successful facility experiences starts by monitoring the existing situation. Measuring processes time and mapping customer journeys can reflect the efficiency of those processes to introduce data-driven enhancements and to keep enhancing the operation in an iterative process (Kaizen). The ubiquity and availability of mobile phone data have introduced an amble of opportunities to understand user journeys in the offline world and the efficiency of the operation provided for those customers. There are multiple sides of introducing data-driven performance monitoring products:</p><h3>Data collection</h3><p>Data collection poses a challenge as it requires user consent in order for their data to be collected. Offering a service such as an app dedicated to the facility or free wifi where a customer will agree to provide their data with a promise to keep it anonymized and secure is one way of collecting such data. Another route is using data aggregator companies or companies that operate mobile phone apps at a large scale and having a business partnership or data-sharing agreements.</p><h3>Data representativeness</h3><p>The collected data will always represent a sample. After all not all people will install the app or use the free wifi. Even partnering with mobile phone app companies will also only offer a fraction of all visitors. This is usually acceptable in the standards of data processing when calculating aggregated insights. In addition, a suitable scaling factor can be calculated to have representative insights.</p><h3>Privacy</h3><p>I’m comfortable showing you where I have been on a map. But most people will not be happy knowing this much information about their mobility is being extracted and tied to their identity. In addition, regulations are getting strict over time. To maintain a good standard for privacy, there are many steps to follow such as:</p><h3>Anonymization</h3><p>The data coming from users should not be traced back to their identity. This needs appropriate private information hashing in a non reversed ways such as SHA-2 or other more complex hashing that can not be reverse-engineered.</p><h3>Aggregation</h3><p>Calculated insights on an individual level should be aggregated to show an overall trend or a business intelligence dashboard from the aggregated product. For example, saying users named kyle, mark, and Omar has stayed in this process for 15 minutes is not a good strategy. we can instead extract the average time staying in this facility is 15 minutes +- 3 minutes after aggregation.</p><h3>Access control</h3><p>Minimizing the number of administrators or data users is best practice when ensuring only relevant people access this information.</p><h3>Final notes</h3><p>Data-driven operation enhancement is no longer just proof of concept work. It has already been introduced at scale in many businesses and for many applications. The ability to start data collection in a way that preserves data privacy standards will be key in what future data-driven technology adoption an entity can embrace.</p><p>Having said that, The applications for understanding service time and customer journey can be applied to many facilities and many industries. For example,</p><p><strong>Shopping malls</strong></p><ul><li>To dynamically price different areas of the mall based on the demand</li><li>To extract insights about the difference in behavior between weekday and weekend and holidays for better allocation of security personnel</li></ul><p><strong>Factories</strong></p><ul><li>Understand the time it takes for each process for operation enhancement</li><li>Find out unused spaces or facilities by drawing a heat map utilization</li><li>Manage personnel safety when an emergency happens by knowing where each person is</li></ul><p><strong>Airports</strong></p><ul><li>Better understand peak hours and the change of service time on different days and weekend</li><li>Allocate more human or less human resources by forecasting the demand on different days and different areas</li></ul><p><strong>Smart city and residential compounds</strong></p><ul><li>Understand the movement pattern of residence for better designing a customized service</li><li>Understand road utilization density and factors that inro ranking roads based on importance to allocate better maintenance strategy</li></ul><p>and more …</p><h3>About me</h3><p>Mohamed Batran is an Artificial Intelligence expert and a data monetization specialist working in Japan. Throughout his career, he worked in data from industries such as finance, e-commerce, logistics, telecom, transportation in designing data utilization to increase revenue, cut down costs or improve customer satisfaction.</p><p>You can read the same article or contact me here, I will be very happy to receive any comment, feedback, or set up a call to discuss collaboration:</p><p><a href="http://mohamedbatran.com/">http://mohamedbatran.com/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5864a877b47b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/airport-passenger-journey-mapping-post-covid-era-5864a877b47b">Airport Passenger Journey Mapping Post COVID Era</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A GIS Pipeline for LIDAR Point Cloud Feature Extraction]]></title>
            <link>https://medium.com/data-science/a-gis-pipeline-for-lidar-point-cloud-feature-extraction-8cd1c686468a?source=rss-137c11a56fbc------2</link>
            <guid isPermaLink="false">https://medium.com/p/8cd1c686468a</guid>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[lidar]]></category>
            <category><![CDATA[gis]]></category>
            <category><![CDATA[gdal]]></category>
            <category><![CDATA[dbscan]]></category>
            <dc:creator><![CDATA[Batran]]></dc:creator>
            <pubDate>Tue, 31 Aug 2021 14:24:26 GMT</pubDate>
            <atom:updated>2021-11-16T06:41:22.612Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to extract features from LIDAR point cloud in an automated manner</h4><figure><img alt="Classified point cloud data using DBSCAN model" src="https://cdn-images-1.medium.com/max/1024/1*iuS-9dR8_rQupTqiCJbG6g.png" /><figcaption>Classified point cloud data points using DBSCAN unsupervised density-based machine learning model “Image by Author”</figcaption></figure><p>In this article, I will walk you through extracting features from LIDAR point cloud in an automated manner.</p><p>LIDAR point cloud data is a collection of points that describe a surface or an object. Each point is described with x, y, z coordinates and in some cases is associated with additional attributes such as a classification of the point.</p><p>LIDAR data is used to extract accurately geo-referenced vector features such as buildings, roads, trees. To realize that there are two main steps as in [1]:</p><ol><li><strong>Classification</strong>: assign a class to each point in the point cloud dataset. One way is using the PointCNN neural network given that there is sufficient training data. The output is a class for each data point.</li><li><strong>GIS pipeline</strong>: extracting relevant features from classified point cloud data in a usable format. In addition to extraction, generating attributes information to the extracted features such as object height, object surface area, centroid latitude/longitude, proximity to other features (e..g how far from the nearest road), and others.</li></ol><p>This article will expand on point no (2) focusing on the <strong>Automated GIS feature extraction</strong> assuming the data is already classified and each data point is assigned a class.</p><h3>Data and Preprocessing</h3><p>This section will dive into data acquisition, preprocessing, and visual inspection. It’s an often important step before starting science or development work.</p><h4><strong>1. Data download and transformation</strong></h4><p>There are many services that offer to download LIDAR data. For sample data acquisition to go along with this tutorial, we will download a LIDAR dataset from [3]. When working with geospatial data, it’s always important to know what is the spatial reference system of the dataset. The spatial reference system for this tile is EPSG:7415 [4]. The definition of spatial reference system according to Wikipedia is</p><blockquote>“A spatial reference system or coordinate reference system is a coordinate-based local, regional or global system used to locate geographical entities.”</blockquote><p>The server to download the data is not very fast, so downloading may take some time.</p><blockquote>It took me a few hours to download the dataset, so at the end of this section, I will share a link for downloading a preprocessed version from my google drive.</blockquote><p>The downloaded file is in <strong>Laz</strong> format</p><blockquote>LAZ format is a lossless compression of LIDAR data.</blockquote><p>It can compress original Las files to only 7 to 25 percent of their size. The speed is one to three million points per second [5].</p><p>The downloaded file size is 2.2 GB. To decompress LAZ files to the original Las one option is to use pylas. The extracted Las file is 13 GB. Impressive compression ratio! Below is how to decompress a LAZ file into LAS file in python using pylas python package.</p><pre>#!pip install pylas[lazrs,laszip]<br>input_laz_path = ‘data/C_37EZ2.laz’<br>output_las_path = ‘data/C_37EZ2.las’<br>laz = pylas.read(input_laz_path)<br>las = pylas.convert(laz)<br>las.write(output_las_path)</pre><h4>2. Point Cloud visual inspection</h4><p>Starting from QGIS 3.18, there is a feature to load and visualize las point cloud data directly in QGIS. It’s pretty straightforward and allows data visualization, overlaying satellite imagery, and rendering high-quality maps.</p><p>We had to manually define the CRS (Coordinate Reference System) of the point cloud file to appear on the viewer window of QGIS. For satellite imagery overlay, There are many QGIS plugins, one of the best is Quick Map Services developed by NEXTGIS which have the most known base maps xyz urls [6]. The dataset we are using already has most points classified and can be mapped with a legend as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*b7PYIoSzcYdCtg6y" /><figcaption>Visual inspection of the downloaded point cloud file in QGIS “Image by Author”</figcaption></figure><p>Looking closely at the classified point cloud with overlaid satellite images, we can see that not all features are exactly classified as in the background satellite image. Only Ground, Building, Water objects are captured and there is no additional information about e.g. Road surface of trees.</p><p>One reason may be that the date of point cloud data acquisition (2014) is different from the date of the overlaid background map (2021 by Google). Green areas are also misclassified as ground, maybe it was the winter season with a lot of snow?</p><p>For a POC (proof of concept), we should reduce the data size to the minimum AOI (Area of Interest). However, we should also consider that the output algorithm on the minimum AOI will guarantee the same results when scaled to a bigger dataset.</p><p>To do that, let’s handpick an AOI that contains most of the classes and draw a polygon with the same CRS on the screen as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Y6a-GctJ09oZxuKA" /><figcaption>A user-defined Area of Interest to reduce the size of data during development “Image by Author”</figcaption></figure><p>To clip LIDAR point cloud data to a specific AOI, there are many tools and libraries such as LAStools [7] clip or other python options such as PDAL. Why do we need to clip the LIDAR dataset into a smaller area of interest:</p><ol><li>In the development phase, we need to try out many solutions, and running the script every time on a big dataset and waiting for the solution to validate is a nightmare</li><li>In most cases, development happens on local machines, where resources are limited. Then the same code can be used on the cloud or a bigger server</li><li>It’s much easier to visually inspect the quality of the algorithm is the data size is small and can fit in memory.</li></ol><p>The output smaller Area of Interest looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/0*__U4Zr3PRG4cU_Wg" /><figcaption>Clipped area of interest from a large LIDAR data file “Image by Author”</figcaption></figure><p>Quick visual investigation shows that data contain classes of three categories:</p><ol><li>Buildings</li><li>Ground</li><li>Water</li><li>Unclassified</li></ol><p>Building features seem very well represented and have easy to identify the geometrical shapes. Water features look incomplete and will probably result in wired polygon pieces if we automate capturing it from this sample.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8mKGaaTlI35epquy" /><figcaption>Each class visualization in the classified LIDAR data point in the area of interest “Image by Author”</figcaption></figure><blockquote>For limiting the scope of this analysis, This work will focus on only Extracting buildings from the classified LIDAR data.</blockquote><h3>Feature extraction</h3><p>According to [1] to extract objects from LIDAR data, There are two techniques to be used, each one of the two can be suitable for a specific use case and depending on the business requirement and acceptance criteria:</p><ol><li><strong>Raster analysis</strong> by rasterizing point cloud layers, then apply raster to polygon</li><li>Use <strong>unsupervised machine learning algorithm DBSCAN</strong> to separate each object as a cluster, then apply a bounding polygon operation or other to approximate the boundary of the object.</li></ol><blockquote>In this report, we will explain and apply both techniques.</blockquote><h4>1. Raster Analysis</h4><p>To capture features from LIDAR data points, the first step is converting the 3d vector points (X, Y, Z) into grid-based raster representation. This process is called rasterization, the output raster will be a Digital Surface Model (DSM). and will also contain the corresponding average points height in each pixel.</p><p>One library that can do that is PDAL[7]. To move forward, we need to specify the resolution of the resulting DSM. In this case, let’s use 4 times the average spacing between points, this is best practice but can vary depending on the use case. The average spacing between points is roughly 30 cm ( manually measuring that on-screen a few times), then we chose pixel size to be 1.2 meters.</p><p>To install PDAL on mac, use the following command:</p><pre>brew install pdal<br>Pip install pdal</pre><p>PDAL has a python API to use in scripting and to create data pipelines from python, the API implementation is as below:</p><pre>pdal_json = {<br> “pipeline”: [<br> INPUT_LAS_PATH,<br> {<br> “type”: “filters.range”,<br> “limits”: “Classification[6:6]”<br> },<br> {<br> “filename”: OUTPUT_TIFF_DEM,<br> “gdaldriver”: “GTiff”,<br> “output_type”: “all”,<br> “resolution”: “1.2”,<br> “type”: “writers.gdal”<br> }<br> ]<br> }<br> pdal_json_str = json.dumps(pdal_json)<br> pipeline = pdal.Pipeline(pdal_json_str)<br> count = pipeline.execute()</pre><p>In the above code block, the JSON object pdal_json stores the required pipeline definition. First, we input the las file path as INPUT_LAS_PATH, Then we specify the limit class we want to extract is class no. 6, which is building. After that, we specify the output file name as OUTPUT_TIFF_DEM and the resolution of the output TIFF image.</p><p>Once we pass that definition document to PDAL Pipeline API, it will start executing based on the input and output the TIFF file from the input list. Pretty straightforward and convenient.</p><p>This will result in a GeoTIFF file as below, it’s a raster file with a pixel size of 1.2 meters as we specified. Also, each pixel contains the height of the pixel from the point cloud data. The visualization below can be generated using QGIS and quickmapservices plugin for background satellite images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8D-SpTLzSvHurEMS" /><figcaption>The output raster from the point cloud using PDAL “Image by Author”</figcaption></figure><p>The next step is to extract the building footprint from the resulting DEM. There is more than one tool to do that. For e.g:</p><ol><li>Using GDAL polygonize implementation</li><li>Using rasterio shapes</li></ol><p>Both tools create vector polygons for all connected regions of pixels in the raster sharing a common pixel value. Each polygon is created with an attribute indicating the pixel value of that polygon. Since the DEM contains the height of the pixel as a pixel value, extracting polygons from the DEM file will result in multiple small polygons with the same pixel value. It’s not what we are looking for.</p><p>To extract building footprint, we should treat the DEM as a binary tiff file where building pixels can be set to 1, and no building data are set to np.nan value. This preprocessing step can then feed into another process of rasterio shape extraction or <em>GDAL Polygonize</em> to extract pixels with the same value as one polygon.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*S6A6Nvn3T9YHkAhw" /><figcaption>Applying rasterization and GDAL polygonize on LIDAR data points “Image by Author”</figcaption></figure><p>Finally, having the polygonized coordinates of each polygon, we can use shapely and geopandas to create a geodata frame with every building as a feature.</p><blockquote>Shapely and geopandas are among of the most famous geospatial data handling libraries in python.</blockquote><p>Please note, In this step, the CRS of the features should be set to 7415 before saving the output vector file.</p><blockquote>The total extracted features are <strong>155 buildings</strong> in the area of interest</blockquote><pre>import rasterio<br>from rasterio.features import shapes<br>import numpy as np<br>from shapely.geometry import Polygon</pre><pre>mask = None<br>with rasterio.Env():<br> with rasterio.open(‘SlopeNew.tif’) as src:<br> image = src.read(1) # first band<br> results = (<br> {‘properties’: {‘raster_val’: v}, ‘geometry’: s}<br> for i, (s, v)<br> in enumerate(<br> shapes(image.astype(np.float32), mask=mask, transform=src.transform)))<br> <br> geoms = list(results)</pre><p>By doing that, we have iterated through each extracted polygon and saved them all to a list of geometries. Those geometries can be read in geopandas and will appear as geodataframe as bellow, with this, 90% of the hard work is completed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/914/0*U8_oqUTdFbl3xmaE" /></figure><p>The resulting vectorized building shapes look like below, later in this article, we will apply a post-processing step to extract meaningful features from those buildings, so keep reading.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kn1uKFtl1_06H7ub" /><figcaption>Vectorized buildings from LIDAR “Image by Author”</figcaption></figure><h4>2. Unsupervised Machine Learning DBSCAN Clustering</h4><p>In the previous section, we extracted the building footprint using raster analysis and polygonization. While this method works ok, it requires data transformation to a completely different type (Raster) which loses some important values associated with the original data points.</p><p>In the following method, we will use an unsupervised machine learning clustering method to cluster all points that are close to each other as one cluster, then extract building boundaries from those points.</p><p>LIDAR data is stored in a las file (or compressed laz file). To apply clustering, let’s first extract the coordinates of each point X, Y, Z and the associated class per point as below:</p><pre>def return_xyzc(point):<br> x = point[0]<br> y = point[1]<br> z = point[2]<br> c = point[5]<br> return [x, y, z , c]</pre><pre>pcloud = pylas.read(input_las_path)<br>points = [return_xyzc(i) for i in pcloud]<br>building_pts = [point for point in points if point[3] == 6]</pre><p>The resulting data format will be an array of points X, Y, Z, C where x is the longitude, y is the latitude, z is the elevation and c is the class of the point. Resulting in total data points are 12,333,019 for the area of interest. To start with building data, let’s filter only the building class which is class no. 6. This result in a total number of 1,992,715 building points</p><pre>[[87972210, 439205417, 699, 6],<br> [87972259, 439205351, 57, 6],<br> [87972267, 439205339, 1716, 6],<br> [87972419, 439205135, -331, 6],<br> [87972319, 439205284, 1030, 6]]</pre><p>The points above can then be visualized on a background map to view how building classes look like in 2d, in this case each data point is stored as a 2d GIS point stored in a shapefile and elevation is an attribute. Which is different from the las file data storage before.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*I-ZrLYO__lzfRvNw" /><figcaption>Extracted vector data points of one class (Building Class) “Image by Author”</figcaption></figure><p>The next step after that is applying <strong>DBSCAN clustering.</strong></p><blockquote>DB scan is Density-Based Spatial Clustering of Applications with Noise. Finds core samples of high density and expands clusters from them. Good for data which contains clusters of similar density.</blockquote><p>In DB scan, there are two core parameters, first one is epsilon which is The maximum distance between two samples for one to be considered as in the neighborhood of the other. The second parameter is min_samples, which is the number of samples (or total weight) in a neighborhood for a point to be considered as a core point. This includes the point itself.</p><p>Using DBSCAN algorithm, the output is as follows where every point is associated with a cluster label that separates that cluster from other data points, for points that are noise and not close to other points within the epsilon parameter, the algorithm will set its value to -1 as it is not associated to any cluster data point. See how clear the output is below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LBvtWJpK7qauLdvc" /><figcaption>Clustered LIDAR data points using DBSCAN and visualized over a background satellite imagery “Image by Author”</figcaption></figure><p>The next step is using data points from each cluster separately to extract the building footprint and the associated attributes of this object. In order to process each cluster individually, the problem statement is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KT9k_MbI15wew7i0" /><figcaption>One cluster of points separated and visualized on a background satellite image map “Image by Author”</figcaption></figure><p>The total number of points in the above building = 35580 points, Just for one single building.</p><p>Given an object 3d coordinates, the objective of the next step is:</p><ol><li>Extract the mean elevation of that object</li><li>Calculate the 2d boundary of the object</li><li>Calculate the associated geometry details such as area and centroid point.</li></ol><p>To extract the 2d boundary of an object from its associated vector data points, there are many techniques such as convex hull and alpha shape. We will use Alpha shape for this purpose</p><blockquote><strong>Alpha Shape algorithm construct a best fit boundary around data points</strong></blockquote><p>The API usage from python is straightforward, the only parameter needed is alpha. In our case, we will set it to 1 since data points are close to each other and we are interested in the precise boundaries.</p><pre><br>building = building_pts_gdf[building_pts_gdf.cluster==76]<br>shape = alphashape(building.geometry, alpha=1)</pre><p>The output of The Alpha Shape Algorithm is a well-identified polygon that can be stored in any geometry format such as WKT</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YuSIIJ9MPS4TJYcv" /><figcaption>The result of running the Alpha Shape Algorithm on one cluster of the DBSCAN classified point cloud “Image by Author”</figcaption></figure><blockquote>Let’s take some time to look how clean the borders are based on the input points. If you use convext hull, the output will look like a circle</blockquote><p>All we have to do now is iterate through all clusters to compute the building boundaries for each cluster separately.</p><p>Next, for every building shape, we are able to compute the following attributes while is very useful for using the output data file:</p><ol><li>Boundary geometry</li><li>Elevation of the building by taking the median altitude of points</li><li>Total Area</li><li>Perimeter</li><li>Centroid x coordinates</li><li>Centroid y coordinates</li></ol><p>This can be done using simple python and shapely functions</p><pre>building = building_pts_gdf[building_pts_gdf.cluster==76]<br>shape = alphashape(building.geometry, alpha=1)<br>height = building.z.median()<br>area = shape.area<br>perimeter = shape.length<br>centroid_x = shape.centroid.x<br>centroid_y = shape.centroid.y<br></pre><p>The resulted extracted building shapes are as below, awesome and clean as if they are drawn manually by an AutoCAD professional.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ne5oNw9bWKrZumKE" /><figcaption>LIDAR point cloud vector feature extraction result including height of the building “Image by Author”</figcaption></figure><p>The hard work is done, now we have a data pipeline that can take a point cloud input and output automatically extracted features and the associated geometry attributes. Next is post-processing</p><h3>Post-processing</h3><p>It’s common to apply post-processing after the main processing pipeline is completed. The purpose of the post-processing work vary from one project and data type to another, broadly speaking those are some post-processing target objectives:</p><ol><li>Quality check and outlier removal: for example, removing features with unrealistic areas or with small areas, detecting overlapping features and merging them, and so on</li><li>Geometrical calculation: Computing meaningful information from the object geometry such as area or the perimeter. For example, calculating features centroid, calculating feature</li><li>Enrichment: Adding more attribute information to features from an external dataset, e.g. attaching distance for all houses from the nearest bus stop by referring to an external data source of bus stops.</li></ol><h4>1. Quality check and outlier removal</h4><p>Automating quality check and outlier removal need a fairly good understanding of the permitted applications of the extracted features and collaboration with end-users or the product manager (the end user’s voice) to put together a final product quality acceptance criteria.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3iBGSI9Rob5YdKVp" /><figcaption>An outlier building in the extracted buildings from the GIS data pipeline created in the previous step “Image by Author”</figcaption></figure><p>For our use case, we will limit the scope of quality check and outlier removal to the area of the extracted building where it should be within a reasonable range. For this purpose, we will assume that the smallest building consists of one room 3*3 square meters (9 square meters). This post preprocessing step will eliminate 40 polygons leaving the total extracted buildings in the area of interest to 115.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/654/0*wuQ1dr0_5OJNLnFU" /><figcaption>Final extracted building layer after Outlier detection and removal “Image by Author”</figcaption></figure><h4>2. Geometrical calculation</h4><p>For practical business use, and to create an easy to query dataset, meaningful calculated attributes for each feature can be extracted in the post-processing step. Some examples of those features are area, centroid x, centroid y to name a few. This step will later help nongeospatial data users to query the data for various business use cases and utilization.</p><p>Geometry calculation is pretty straightforward in geopandas as below</p><pre>building_gdf[‘area_m2’] = building_gdf.area<br>building_gdf[‘centroid_x’] = building_gdf.geometry.centroid.x<br>building_gdf[‘centroid_y’] = building_gdf.geometry.centroid.y</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lToha1jiSPAqK9V3" /></figure><h4>3. Data enrichment</h4><p>Enrichment commonly refers to appending or otherwise enhancing collected data with relevant context obtained from additional sources. Geospatial data enrichment means adding more attributes to the dataset by referring to external data sources. Few common data sources are:</p><ol><li>Road infrastructure networks</li><li>POI database</li><li>Demographics distribution</li><li>Administrative boundaries</li></ol><p>We will skip applying this step in this report since it can expand indefinitely.</p><h3>Engineering</h3><p>Writing in notebooks works well for prototyping, but in order to create a data pipeline and collaborate with other team members, few steps need to be applied to the final logically correct script. Below are three quick things for productization.</p><ol><li><strong>Code formatting</strong></li></ol><p>When someone is prototyping, usually the code is not well styled and indentations and spaces are not a priority at this phase. After finishing the logic, you can use one of the automated styling tools such as YAPF [8 ]by google. Once the library is installed, run it from the command line and let google style your code.</p><pre>yapf building_extractor.py &gt; building_extractor_styled.py</pre><p><strong>2. Dockerization</strong></p><p>It’s a good practice to dockerize a python application to avoid dependencies conflicts, especially in this case we used so many libraries and some of them rely on binary data that are separately installed such as PDAL. Reproducing the same environment without docker will be challenging.</p><p><strong>3. Unit tests</strong></p><p>Logical functions such as alpha shape algorithm, geometry calculation should be tested using unit tests before running the output.</p><h3>Visualization</h3><p>In addition to static maps, it’s always great to have an interactive map and be able to visualize the input and the output. For this purpose, I’ve used deck.gl with some extra HTML / JS coding.</p><h4>1. Input point could interactive</h4><p>To visualize the input point cloud, I’ve extracted few buildings from the output of DBSCAN classified point cloud data, the interactive map is here:</p><p><a href="https://xmap.ai/maps/pcloud/dbscan_pcloud_3d.html">https://xmap.ai/maps/pcloud/dbscan_pcloud_3d.html</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hTwMHYibwRCE62t8" /><figcaption>HTML visualization in 3d to view the point cloud data out of DBSCAN “Image by Author”</figcaption></figure><h4>2. Output extracted polygons with enrichment attributes</h4><p>This is the final product of the GIS data pipeline. The output data is saved as geojson object and visualized using deck.gl and colored by height.</p><p><a href="https://xmap.ai/maps/pcloud/buildings_vector.html">https://xmap.ai/maps/pcloud/buildings_vector.html</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kwhIN6pAlkcxzSZz" /><figcaption>3D visualization of the final output product of the GIS pipeline, building — Building vectors with attributes “Image by Author”</figcaption></figure><h3>Summary</h3><p>LIDAR data processing is essential for many applications such as mapping and autonomous driving. For most applications, automating object detection and extraction is necessary to ensure the business success of a LIDAR product. While a lot of tools now offer out-of-the-box analytics such as Arcgis, programming one’s own tools give more flexibility to add or adjust to a specific use case. In this article we explored how to extract building footprint from LIDAR dense point cloud using two different techniques:</p><ol><li>Rasterization and polygonization</li><li>Unsupervised machine learning density-based clustering and alpha shape boundary detection</li></ol><p>We also explored the standard post-processing techniques such as</p><ol><li>The geometric calculation to extract essential properties about each geometry such as area, perimeter, and centroid</li><li>Enrichment to append more entity attributes from external datasets such as road network of POIs database if needed</li></ol><p>Then, we looked into some engineering steps to ensure code reusability such as:</p><ol><li>Code formatting using automatic formatting tool</li><li>Dockerization</li><li>Unit testing</li></ol><p>Finally, we use interactive geospatial visualization to look at the classified point cloud and the final output building data as in the two links below:</p><ul><li><a href="https://xmap.ai/maps/pcloud/dbscan_pcloud_3d.html">xMao.ai - DBSCAN 3d points</a></li><li><a href="https://xmap.ai/maps/pcloud/buildings_vector.html">xMap.ai - gis pipeline building vectors</a></li></ul><p>The whole analysis in this report can be automated end to end and scheduled as a batch job if needed.</p><h3>Questions or Inquiries</h3><p>I’m here <a href="http://mohamedbatran.com/">https://mohamedbatran.com</a>, please reach out for any questions or If you want us to collaborate!</p><p>LinkedIn <a href="https://www.linkedin.com/in/mbatran/">https://www.linkedin.com/in/mbatran/</a></p><h3>References</h3><p>[1] <a href="https://medium.com/geoai/object-extraction-from-mobile-lidar-point-clouds-with-machine-learning-cb15fcbb5597">https://medium.com/geoai/object-extraction-from-mobile-lidar-point-clouds-with-machine-learning-cb15fcbb5597</a></p><p>[2] <a href="https://github.com/beedotkiran/Lidar_For_AD_references">https://github.com/beedotkiran/Lidar_For_AD_references</a></p><p>[3] <a href="https://download.pdok.nl/rws/ahn3/v1_0/laz/C_37EZ1.LAZ">https://download.pdok.nl/rws/ahn3/v1_0/laz/C_37EZ1.LAZ</a></p><p>[4] <a href="https://epsg.io/7415">https://epsg.io/7415</a></p><p>[5] <a href="https://www.cs.unc.edu/~isenburg/lastools/download/laszip.pdf">https://www.cs.unc.edu/~isenburg/lastools/download/laszip.pdf</a></p><p>[6] <a href="https://plugins.qgis.org/plugins/quick_map_services/">https://plugins.qgis.org/plugins/quick_map_services/</a></p><p>[7] <a href="https://github.com/LAStools/LAStools">https://github.com/LAStools/LAStools</a></p><p>[8] <a href="https://pdal.io/workshop/exercises/analysis/dtm/dtm.html">https://pdal.io/workshop/exercises/analysis/dtm/dtm.html</a></p><p>[8] <a href="https://github.com/google/yapf">https://github.com/google/yapf</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8cd1c686468a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/a-gis-pipeline-for-lidar-point-cloud-feature-extraction-8cd1c686468a">A GIS Pipeline for LIDAR Point Cloud Feature Extraction</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>