<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ahmad Zaenun Faiz on Medium]]></title>
        <description><![CDATA[Stories by Ahmad Zaenun Faiz on Medium]]></description>
        <link>https://medium.com/@zaenun.faiz?source=rss-93280e763a34------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*hVjTdbTUxLwrA8QuN3CLOg.jpeg</url>
            <title>Stories by Ahmad Zaenun Faiz on Medium</title>
            <link>https://medium.com/@zaenun.faiz?source=rss-93280e763a34------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 21:34:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@zaenun.faiz/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why your GIS polygons might be lying to you: The Zero-Area Paradox]]></title>
            <link>https://medium.com/@zaenun.faiz/why-your-gis-polygons-might-be-lying-to-you-the-zero-area-paradox-febae3e6fb04?source=rss-93280e763a34------2</link>
            <guid isPermaLink="false">https://medium.com/p/febae3e6fb04</guid>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[spatial-computing]]></category>
            <category><![CDATA[gis]]></category>
            <category><![CDATA[geometry]]></category>
            <dc:creator><![CDATA[Ahmad Zaenun Faiz]]></dc:creator>
            <pubDate>Tue, 24 Feb 2026 02:59:10 GMT</pubDate>
            <atom:updated>2026-02-24T04:14:12.548Z</atom:updated>
            <content:encoded><![CDATA[<p>Do you know when you are using GIS Software, sometime the area calculation of a polygon might yield wrong result?</p><p>Some people might advice you to “fix geometry” and then recalculate the area to get it correct. While that usually works, simply clicking a button won’t help you understand <em>why</em> the calculation went wrong in the first place. So in this article, I’ll explain how GIS Software such as QGIS or ArcGIS Pro does their magic to calculate the area.</p><p>When you attend a math subject in the high school, you may learn how to calculate an area of square, triangle circle etc. using some specific math formula. However, the way computer calculate an area is not as simple calculation as you studied in the high school.</p><p>Let me introduce <strong>Shoelace formula</strong>, also known as <strong>Gauss’s area formula</strong>. This is the secret sauce on how GIS softwares to calculate area of polygon from your spatial data. The formula of Shoelace is defined as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/345/0*dHDGCXxqQLpWuVnO" /></figure><p>It sounds terrifying for you if you hate Maths 😀. But I’ll explain it to make it more understandable.</p><p>So basically, GIS software will calculate the polygon triangle-per-triangle. Let me give an example if you have a polygon like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BYxNML7o9JCSHRyawQgGvg.png" /></figure><p>It’s pretty a simple polygon, and how GIS softwares will calculate the area using Shoelace formula is like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/218/0*sbAPgBl6K-_VgVBY" /></figure><p>And this is how GIS software calculate the area, triangle-per triangle. The software draws lines from O to every vertex of your polygon:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1J7qf5OKiYZ6ioKMqlbnIw.png" /></figure><p>Great, now all the triangles is identified. We can calculate it with triangle are calculation formula and combine the result. But wait, this will create additional area calculations if we just add them up. So how Shoelace formula handles the area calculation, to calculate only the area inside A-B-C vertices and excluding area that comes from O point? This is the concept that very important for you to know.</p><p>You need to understand how Shoelace formula use winding order to get the area result. The formula is essentially the sum of many small triangles created between each line segment of your polygon and the x-axis. To make it simple, the winding order is like this:</p><ul><li>If the order is clockwise, it yields <strong><em>Negative</em></strong> area calculation</li><li>If the order is counter-clockwise, it yields <strong><em>Positive</em></strong> area calculation</li></ul><p>Why does it behaves like that? the answer is because the Horizontal “Sweep”. When you calculate the area of a single segment from point x₁y₁ to x₂y₂, the calculation looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/180/0*s9GRW-2dZZ4bCF3T" /></figure><p>This is actually calculating the area of the triangle sitting under that specific line.</p><ul><li><strong>Counter-Clockwise (CCW):</strong> As you move CCW, the “top” segments of your polygon usually move from right to left, and the “bottom” segments move from left to right. The math is designed so that the “forward” segments add area, and the “return” segments subtract the empty space below the shape. You are left with a <strong>positive</strong> remainder, the interior area.</li><li><strong>Clockwise (CW):</strong> If you go the other way, the subtractions happen where the additions should be. You are essentially “subtracting” the shape from the empty space, which results in the same number but with a <strong>negative</strong> sign.</li></ul><p>Now let’s check again this triangle directions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wIUb11oAvt9fYLnrufXhPQ.png" /></figure><p>Green polygon is CCW from A to B. Blue polygon also CCW from B to C. But the red polygon is CW from C to A. By using this logic, the area of polygon is <strong><em>(</em></strong><em>green + blue - red</em><strong><em>)</em></strong>. Now we get the actual area of the polygon.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*bCCthVZpmwEx_Utu-wdF6Q.png" /></figure><p>Great, now you know how the geometry area calculation is done in GIS software. But wait, how if you have a polygon like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xkS2ozuiUSDW9T2uzeVtug.png" /></figure><p>This is what we called a self-intersection polygon, and in GIS this will be considered as invalid geometry. Why it’s invalid? because it creates a bowtie shape that the right and left area have different direction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OCbJdhl0Fql470uGo8tsGw.png" /></figure><p>By using the winding order logic, the right area will get negative area calculation since it’s CW and the left area will get positive area calculation since it’s CCW. Since both area have equal area, then the area calculation is 0 in this case. This is where the problem comes. Your GIS software don’t produce the correct area as we see in that polygon, and produce area calculation that totally wrong.</p><blockquote>Computers are literal, not intuitive. If you tell a computer to follow a path that cancels itself out, it will happily tell you that a forest has 0 area. Always validate your geometry before trusting your statistics.</blockquote><p>By understanding this, it’s important to clean and validate your spatial data before using it for analysis. In GIS term, this is what we called <strong>Topology Check</strong>. Using GIS software without deep knowledge about how geometry works just create another <em>garbage-in garbage-out</em>. Always validate your data before performing spatial analysis on it.</p><p>Also some GIS Software isn’t immune to basic geometry logic. For example:</p><ol><li><strong>ArcGIS Pro</strong> sometimes tries to “hide” the error by repairing it or blocking the calculation. Without proper topology check, you will never know if your data is problematic or not.</li><li><strong>QGIS</strong> is a bit more transparent, which means it shows you the “math error” more clearly. QGIS will shows you the “raw math”, which might be 0, unless you explicitly tell it to fix the geometry.</li><li><strong>Both</strong> will fail during “Spatial Overlays” (clipping, intersecting) if the geometry isn’t “Simple.”</li></ol><p>If you prefer doing some scripting with SQL using PostGIS, or with Python using Shapely, they will often give you a number anyway. And yes, that is exactly how unnoticed bugs enter production systems.</p><h4>How Shapely Handles It</h4><ul><li>If you create a self-intersecting Polygon and call .area, Shapely will apply the Shoelace formula to the vertices in the order you provided.</li><li>If you have a “bowtie,” Shapely will return 0.0 (or a very small number). It <strong>does not</strong> automatically fix it because it assumes you gave it a &quot;valid&quot; geometry.</li><li>If you try to do a more complex operation like union or intersection on that invalid polygon, Shapely might finally throw a TopologicalError. But the .area call itself often remains silent.</li></ul><h4>How PostGIS Handles It</h4><p>PostGIS is a bit stricter but still has “blind spots” depending on which function you use:</p><ul><li>In <strong>ST_Area(geom)</strong>, this function is fast and “dumb”. It follows the vertex order. If the polygon is a bowtie, it will subtract the areas and give you the wrong total without warning.</li><li>Use <strong>ST_IsValid(geom)</strong>. PostGIS has this specific function to check for these issues. High-level pipelines usually run SELECT * FROM table WHERE NOT ST_IsValid(geom); to find &quot;bugs&quot; before doing math.</li></ul><h3>The GIS Data Integrity Checklist</h3><p>Before you run an .area, $area or ST_Area calculation, ensure your data passes these four checks to avoid &quot;silent failures.&quot;</p><h4>1. The Validity Test</h4><p>Never assume a geometry is valid just because it renders on a map.</p><ul><li>In PostGIS, run SELECT COUNT(*) FROM table WHERE ST_IsValid(geom) = false;.</li><li>In Python, use if not shape.is_valid.</li><li>Always search for the <strong>“Check Validity”</strong> tool in QGIS or the <strong>“Check Geometry”</strong> tool in ArcGIS before they ever hit the “Calculate Area” button</li><li>The Goal is to identify “Bowties” and “Spikes” before they enter your statistics.</li></ul><h4>2. Standardize Winding Order</h4><p>Ensure your “Handedness” is consistent. Most modern systems expect the <strong>Right-Hand Rule</strong>.</p><ul><li>Exterior Rings should be Counter-Clockwise (CCW).</li><li>Interior Rings (Holes), should be Clockwise (CW).</li><li>Why? This ensures that holes subtract from the area and the outer boundary adds to it.</li></ul><h4>3. Use “MakeValid” Routines</h4><p>If you find invalid geometries, don’t just delete them. Repair them.</p><ul><li>Use ST_MakeValid (PostGIS) or shapely.validation.make_valid.</li><li>Use fix geometry in QGIS to make sure that the data is valid.</li><li>It performs <strong>Planarization</strong>. It finds the intersection point, adds a node, and splits the shape into multiple valid polygons. This turns a 0 area bowtie into a +20 area MultiPolygon.</li></ul><h4><strong>4. Snap-to-Grid (The “Sliver” Killer)</strong></h4><p>Often, self-intersections are caused by tiny floating-point errors (e.g., 0.00000000001).</p><ul><li>Use a “Snap to Grid” function.</li><li>It rounds coordinates to a sensible precision (like 1mm). This often “merges” points that are nearly touching, cleaning up tiny overlaps and spikes that cause math errors.</li></ul><blockquote><em>“In geometry, </em><strong><em>Area</em></strong><em> is a property of a surface. But in GIS, </em><strong><em>Area</em></strong><em> is a property of a </em>path<em>. If your path is broken, your area is a lie. Don’t just calculate, validate.”</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=febae3e6fb04" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Processing Large Geospatial Dataset using GeoHash Spatial Index]]></title>
            <link>https://medium.com/@zaenun.faiz/processing-large-geospatial-dataset-using-geohash-spatial-index-6f78079951d3?source=rss-93280e763a34------2</link>
            <guid isPermaLink="false">https://medium.com/p/6f78079951d3</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[geohash]]></category>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[geospatial-intelligence]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Ahmad Zaenun Faiz]]></dc:creator>
            <pubDate>Fri, 19 Sep 2025 09:48:54 GMT</pubDate>
            <atom:updated>2025-09-19T09:48:54.761Z</atom:updated>
            <content:encoded><![CDATA[<p>Do you ever wonder how mapping apps like Google Maps, Uber, or Gojek can compute nearest locations or drivers in a matter of seconds? The secret lies in highly optimized data processing techniques that overcome the significant performance challenges often faced by professionals in GIS and data science. Working with extensive vector datasets containing millions of geographic features — from global transportation networks to building footprints and GPS coordinates — can bring traditional GIS applications like QGIS or ArcGIS Pro to a crawl. These programs often struggle to handle such massive data volumes, leading to operations that can take hours.</p><p>This performance bottleneck stems from the architectural limitations of traditional GIS platforms, which were not designed for today’s large-scale, complex datasets. These applications often try to process entire datasets in memory or use inefficient spatial indexing methods, severely hampering performance.</p><p>This article presents a powerful alternative, using geohash indexing techniques in Python. Geohashing is a hierarchical spatial indexing system that divides geographic space into discrete grid cells identified by alphanumeric codes. By leveraging this method, we can achieve significantly accelerated vector operations like proximity analysis and spatial filtering. This approach combines Python’s flexibility with optimized spatial data structures to deliver substantial performance improvements, potentially reducing processing times from hours to just minutes or even seconds on the same large-scale datasets that challenge traditional GIS tools.</p><h3>What is Geohash?</h3><p>Geohash is a hierarchical spatial indexing system that encodes geographic coordinates (latitude and longitude) into short alphanumeric strings. Developed by Gustavo Niemeyer in 2008, geohashing transforms continuous two-dimensional geographic space into a discrete grid representation, providing an elegant solution for efficiently indexing, storing, and querying geographic data across multiple scales and precision levels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/1*IkLDrXLRj0JwpVoiBzMFtQ.png" /><figcaption>An example of how Geohash works. Source: <a href="https://www.researchgate.net/publication/361076955_Hall_of_Mirrors_A_Novel_Strategy_to_Address_Locality_in_Geocoded-Based_PoI_Private_Queries">https://www.researchgate.net/publication/361076955_Hall_of_Mirrors_A_Novel_Strategy_to_Address_Locality_in_Geocoded-Based_PoI_Private_Queries</a></figcaption></figure><h4>Technical Foundation of Geohash</h4><p>At its core, geohash implements a variant of binary space partitioning through a process known as Z-order curve mapping (or Morton coding). This mathematical approach interleaves the bits of latitude and longitude values to create a single dimensional value that preserves spatial locality. The resulting binary string is then encoded using a base-32 character set for human readability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5h7lFbTR23hh0dzvf1IXdg.jpeg" /><figcaption>Representation of the whole world in terms of rectangles. Source: <a href="https://vivekbansal.substack.com/p/how-to-find-nearby-entities">https://vivekbansal.substack.com/p/how-to-find-nearby-entities</a></figcaption></figure><p>The geohash algorithm systematically subdivides the world through the following precise process:</p><ol><li>The initial bounding box encompasses the entire world: latitude range [-90°, 90°] and longitude range [-180°, 180°].</li><li>This bounding box undergoes binary subdivision: for even-indexed bits, the longitude range is bisected, and for odd-indexed bits, the latitude range is bisected.</li><li>Each subdivision produces a binary digit: 0 if the point falls in the lower/left half, 1 if it falls in the upper/right half.</li><li>The resulting bit sequence is grouped into 5-bit chunks and mapped to the geohash character set (0–9 and b-z, excluding a, i, l, o).</li><li>The recursive subdivision continues until the desired precision is achieved, with each additional character increasing precision exponentially.</li></ol><p>This process creates a hierarchical grid where each cell is uniquely identified by its geohash code. The precision relationship follows a consistent pattern: each additional character in a geohash increases precision by approximately 5 bits, reducing the cell size by a factor of 32 (²⁵).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/884/1*3WK3--_Ct8oG1lzbKG-2PA.png" /><figcaption>Estimation of geohash precision cell table</figcaption></figure><h4>Case Study: Calculating nearest amenities with Euclidean distance</h4><p>This case study explores the application of geohashing techniques to efficiently calculate the nearest amenities using Euclidean distance measurements. The implementation utilizes the OpenStreetMap global dataset, specifically targeting elements with the amenity tag, to demonstrate the effectiveness of spatial indexing for large-scale geospatial data processing.</p><p>The key components of this implementation include:</p><ul><li><strong>Dataset:</strong> Global OpenStreetMap data containing millions of amenity points worldwide</li><li><strong>Indexing method:</strong> Geohashing with pygeohash library to create efficient spatial indices</li><li><strong>Storage format:</strong> Partitioned Parquet files organized by geohash prefixes</li><li><strong>Processing framework:</strong> Dask-GeoPandas for distributed computation</li><li><strong>Query optimization:</strong> Bounding box (bbox) filtering to reduce the search space before precise distance calculations</li></ul><p>The implementation addresses the computational challenge of finding nearest amenities from any global location with significantly improved performance compared to traditional methods. By combining geohashing with optimized storage formats and distributed processing, the system achieves near real-time results for queries that would otherwise require prohibitive computational resources.</p><h3>Analysis</h3><h4>Data Preparation</h4><p>I utilized Google BigQuery Public Dataset to retrieve the OpenStreetMap dataset, specifically extracting records that contain the amenity tag and name tag. This approach allowed me to focus on points of interest such as restaurants, schools, hospitals, and other public facilities across the globe while ensuring each location was properly identified. To perform this data retrieval, I executed the following SQL query in Google BigQuery:</p><pre>SELECT<br>  t1.osm_id,<br>  amenity.value AS amenity_type,<br>  name_tag.value AS amenity_name,<br>  t1.geometry<br>FROM<br>  `bigquery-public-data.geo_openstreetmap.planet_features` AS t1<br>JOIN<br>  UNNEST(t1.all_tags) AS amenity<br>ON amenity.key = &#39;amenity&#39;<br>LEFT JOIN<br>  UNNEST(t1.all_tags) AS name_tag<br>ON name_tag.key = &#39;name&#39;</pre><p>I extracted this dataset from BigQuery and saved it in two formats: GeoJSON for visualization and Apache Parquet for efficient processing. The GeoJSON format, while standard for geospatial data exchange, resulted in a much larger file size (2.22 GB) compared to the columnar Parquet format. Loading this massive dataset in traditional GIS software proved challenging — QGIS took approximately 20 minutes to fully render the GeoJSON file in my machine, highlighting the performance limitations we aim to address with geohashing techniques. Calculating distances from specific reference locations would be extremely resource-intensive in QGIS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SjrIC7a_-SGVQdb-KxX8mA.png" /><figcaption>Loading GeoJSON data format with QGIS to render 11 million points</figcaption></figure><p>Based on the visualization in the screenshot, our dataset contains 11,139,527 amenity features globally mapped by OpenStreetMap contributors. This extensive point dataset presents significant processing challenges when using traditional GIS applications. The sheer volume of data makes spatial operations like nearest neighbor searches and distance calculations computationally expensive and time-consuming. In the next phase of our implementation, we will leverage Dask, a parallel computing library for Python, to partition this massive dataset using geohash indexing. This approach will distribute the computational workload across multiple cores, dramatically reducing processing time for spatial queries. By combining geohash-based spatial partitioning with Dask’s distributed computing capabilities, we can transform what would typically be hour-long operations into processes that complete in minutes or even seconds.</p><h4>Read Parquet files with Dask-Geopandas</h4><p>The first step is to read the large dataset. The ddg.read_parquet() function is incredibly smart, it doesn&#39;t try to load everything at once. Instead, it reads the data in a &quot;lazy&quot; fashion, creating a Dask-Geopandas DataFrame with spatial partitions that you can inspect before any heavy lifting begins. This allows me to get a high-level view of your dataset, like the geographic extents of each partition, without consuming a single megabyte more than necessary.</p><pre>import dask_geopandas as ddg<br><br>parquet_path = &quot;~/Desktop/content/osm_amenity/data/&quot;<br>ddf = ddg.read_parquet(parquet_path)<br><br>print(ddf.spatial_partitions)<br><br>--------------------------------------------------------<br>0      POLYGON ((-139.2728 -89.998, -139.2728 -89.998...<br>1      POLYGON ((-147.6845 -23.86955, -147.6845 -22.6...<br>2      POLYGON ((-144.33215 -27.61732, -144.33215 -27...<br>3      POLYGON ((-169.90091 -21.36256, -169.90091 -18...<br>4      POLYGON ((-169.42273 -16.8734, -169.42273 -13....<br>                             ...                        <br>457    POLYGON ((177.74593 64.71328, 177.74593 66.375...<br>458    POLYGON ((139.94372 67.52425, 139.94372 71.439...<br>459    POLYGON ((139.18373 75.49466, 139.18373 75.494...<br>460    POLYGON ((167.99891 68.05605, 167.99891 69.937...<br>461    POLYGON ((175.75317 68.89638, 175.75317 69.877...<br>Length: 462, dtype: geometry</pre><p>Once ready to dive deeper, I can perform a computation to get a full picture of the dataset’s characteristics. The ddf.compute().info() command will reveal the sheer scale of the data — in this case, over 11 million entries, requiring more than 552 megabytes of memory. This demonstrates the challenge, attempting to run complex spatial computations on a dataset of this size could easily exhaust your system’s resources, especially when calculating things like distances, which can be computationally intensive.</p><pre>ddf.compute().info()<br><br>--------------------------------------------------------<br>&lt;class &#39;geopandas.geodataframe.GeoDataFrame&#39;&gt;<br>Index: 11139527 entries, 0 to 24<br>Data columns (total 6 columns):<br> #   Column        Dtype   <br>---  ------        -----   <br> 0   fid           int32   <br> 1   osm_id        int64   <br> 2   amenity_type  object  <br> 3   amenity_name  object  <br> 4   geometry      geometry<br> 5   geohash       object  <br>dtypes: geometry(1), int32(1), int64(1), object(3)<br>memory usage: 552.4+ MB</pre><p>To find the nearest amenities, first I need to define the points of interest. Here, we’ve prepared a small GeoDataFrame containing key location in Yogyakarta, the historic Tugu Pal Putih Jogja. Despite its tiny size, this simple GeoDataFrame is the starting point for our query, providing the crucial reference point for finding amenities nearby.</p><pre>from shapely import Point<br>import geopandas as gpd<br><br>locations = [<br>    {<br>        &quot;geometry&quot;: Point(110.36706, -7.78291), <br>        &quot;name&quot;: &quot;Tugu Pal Putih Jogja&quot;<br>    },<br>]<br><br>gdf = gpd.GeoDataFrame(locations, crs=&quot;EPSG:4326&quot;)<br><br>gdf.info()<br><br>--------------------------------------------------------<br>&lt;class &#39;geopandas.geodataframe.GeoDataFrame&#39;&gt;<br>RangeIndex: 1 entries, 0 to 0<br>Data columns (total 2 columns):<br> #   Column    Non-Null Count  Dtype   <br>---  ------    --------------  -----   <br> 0   geometry  1 non-null      geometry<br> 1   name      1 non-null      object  <br>dtypes: geometry(1), object(1)<br>memory usage: 148.0+ bytes</pre><p>Now for the main challenge is finding the nearest restaurant to each of it within the massive 11-million-entry dataset. A straightforward, brute-force search would be extremely inefficient, forcing us to calculate the distance from each of our reference location to every single one of the millions of amenities in the dataset. A much smarter and more scalable approach is to first filter our dataset using a <strong>geohash</strong>, which acts like a specialized index for geographic data. By using a 6-precision geohash, we can efficiently narrow down our search space to only those amenities within a small, specific area — a radius of approximately 1 kilometer — before we run any final distance calculations. This significantly reduces the computational load and makes the problem manageable.</p><h4>Calculate geohash for data discovery and filtering</h4><p>To make this possible, we first define two helper functions. The get_geohash function will take a point geometry and a desired precision and return the corresponding geohash string. The get_geohash_bbox function is the key to our optimization. It takes a geohash string and returns the exact bounding box (a tuple of longitude and latitude bounds) that the geohash represents. This bounding box is what we&#39;ll use to quickly filter our massive dataset.</p><pre>import pygeohash as pgh<br><br>def get_geohash(geom, precision=5):<br>    if geom.geom_type == &quot;Point&quot;:<br>        return pgh.encode(geom.y, geom.x, precision=precision)<br>    return None<br><br>def get_geohash_bbox(geohash_input):<br>    lat, lon, lat_err, lon_err = pgh.decode_exactly(geohash_input)<br>    return (<br>        lon - lon_err,<br>        lon + lon_err,<br>        lat - lat_err,<br>        lat + lat_err,<br>    )</pre><p>With these functions in hand, we can now apply them to our small gdf GeoDataFrame containing our two points of interest. By using the apply method, we create a new geohash column and a corresponding geohash_bounds column for each location. This step is incredibly fast since we’re only operating on two data points. The output shows exactly what we’ve accomplished: for each location, we now have a unique geohash and a tuple of coordinates that define its precise bounding box.</p><pre># Compute geohash<br>gdf[&#39;geohash&#39;] = gdf.geometry.apply(<br>    lambda geom: get_geohash(geom, 6)<br>)<br><br># Compute geohash bbox<br>gdf[&quot;geohash_bounds&quot;] = gdf[&quot;geohash&quot;].apply(get_geohash_bbox)<br><br>print(gdf)<br><br>-------------------------------------------------------- <br>                     geometry                  name geohash  \<br>0  POINT (110.36706 -7.78291)  Tugu Pal Putih Jogja  qqw7xr   <br><br>                                      geohash_bounds  <br>0  (110.357666015625, 110.36865234375, -7.7838134... </pre><p>Now that we have the geohash bounds for our location, we can use them to efficiently filter the massive Dask-Geopandas DataFrame. Instead of performing a slow, brute-force spatial join against the entire 11-million-entry dataset, we can use a bbox query to select only the data that falls within the small, targeted bounding boxes. This approach is extremely powerful because Dask-Geopandas can read only the necessary partitions from the Parquet files, avoiding the need to load the entire dataset into memory.</p><pre>minx, maxx, miny, maxy = gdf.iloc[0][&quot;geohash_bounds&quot;]<br>tugu_ddf_bbox = ddf.cx[minx:maxx, miny:maxy]</pre><h4>Compute Euclidean distance on filtered dataframe by geohash</h4><p>The code below first filters the dataset to only include amenities of type &#39;restaurant&#39;. Then, it uses a cx filter to perform a targeted bbox_query on the Dask DataFrame. The final output demonstrates just how effective this is, reducing the data from over 11 million rows down to a manageable handful of locations that are actually relevant to our query. This is a crucial step for making large-scale geospatial analysis feasible on a standard machine.</p><pre>tugu_restaurant = tugu_ddf_bbox[&quot;amenity_type&quot;] == &quot;restaurant&quot;<br>tugu_result = tugu_ddf_bbox[tugu_restaurant].compute()<br><br>print(tugu_result.info())<br>print(tugu_result[[&quot;osm_id&quot;, &quot;amenity_type&quot;]])<br><br>-------------------------------------------------------- <br>&lt;class &#39;geopandas.geodataframe.GeoDataFrame&#39;&gt;<br>Index: 6 entries, 20798 to 21288<br>Data columns (total 6 columns):<br> #   Column        Non-Null Count  Dtype   <br>---  ------        --------------  -----   <br> 0   fid           6 non-null      int32   <br> 1   osm_id        6 non-null      int64   <br> 2   amenity_type  6 non-null      object  <br> 3   amenity_name  1 non-null      object  <br> 4   geometry      6 non-null      geometry<br> 5   geohash       6 non-null      object  <br>dtypes: geometry(1), int32(1), int64(1), object(3)<br>memory usage: 312.0+ bytes<br>None<br>           osm_id amenity_type<br>20798  3557258295   restaurant<br>21246  4792695825   restaurant<br>21249  6780921541   restaurant<br>21250  4424600989   restaurant<br>21256  6237524886   restaurant<br>21288  4853355021   restaurant</pre><p>The efficiency of this approach is remarkable, taking only a fraction of a second to filter this massive dataset. The next logical step is to calculate the precise distance between the found restaurants and our reference point. This is computed using a geodesic ellipsoid calculation from the pyproj framework, which provides highly accurate results.</p><pre>from pyproj import Geod<br><br>geod = Geod(ellps=&#39;WGS84&#39;)<br>tugu_location = locations[1][&quot;geometry&quot;]<br><br>tugu_result[&quot;distance&quot;] = tugu_result.geometry.apply(<br>    lambda geom: geod.inv(<br>        tugu_location.x,<br>        tugu_location.y,<br>        geom.x,<br>        geom.y<br>    )[2]<br>)<br><br>print(tugu_result[[&quot;osm_id&quot;, &quot;amenity_type&quot;, &quot;distance&quot;]])<br><br>-------------------------------------------------------- <br>           osm_id amenity_type    distance<br>20798  3557258295   restaurant  369.819136<br>21246  4792695825   restaurant   69.483241<br>21249  6780921541   restaurant  384.999354<br>21250  4424600989   restaurant  703.775058<br>21256  6237524886   restaurant   50.351825<br>21288  4853355021   restaurant  521.704300</pre><p>The distance calculation unit here is in meters, as we’re using the geodesic ellipsoid calculation with the Geod object from the pyproj library, which is ideal for accurate measurements over the curved surface of the Earth.</p><h3>Result</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jPxXdRPnPXL4FrKxl1EL3Q.png" /><figcaption>Data result visuals in QGIS, calculate nearest restaurants using geohash approach with Dask-GeoPandas</figcaption></figure><p>The methodology outlined in this analysis successfully demonstrates an efficient and scalable approach to performing spatial queries on large-scale geospatial datasets. By strategically combining Dask’s parallel computing capabilities with geohash-based spatial indexing, we transformed a traditionally time-consuming operation into a highly performant one. The initial ddf.compute().info() command confirmed the challenge, a dataset of over 11 million entries that would be nearly impossible to process with a brute-force approach. However, the subsequent steps proved the effectiveness of our strategy.</p><p>The geohash filter served as a powerful preliminary step, drastically reducing the search space from millions of features to a manageable handful. For the Tugu Pal Putih Jogja location, the bbox_query successfully filtered the dataset down to just <strong>6 relevant restaurants</strong> within the targeted area. This represents a significant reduction in computational load, enabling the subsequent, more precise distance calculations to execute with speed and accuracy.</p><p>The final step, utilizing the pyproj library, provided a precise, real-world distance measurement for each of the filtered locations, confirming that the nearest restaurant to Tugu Pal Putih Jogja is located only <strong>50.35 meters</strong> away. This comprehensive approach proves that by leveraging modern libraries and smart indexing techniques, we can tackle large-scale geospatial problems efficiently, accurately, and without overwhelming system resources, emulating the performance seen in modern mapping and delivery applications.</p><h3>Conclusion</h3><p>Geohashing represents a transformative approach to processing large geospatial datasets, addressing the fundamental limitations of traditional GIS platforms through mathematical elegance and computational efficiency. By converting complex two-dimensional spatial relationships into manageable one-dimensional strings or integers, geohashing enables orders-of-magnitude performance improvements for many common geospatial operations.</p><p>While not a universal solution for all geospatial challenges, geohashing provides an exceptionally powerful tool for the initial filtering and organization of large vector datasets, dramatically reducing the computational burden of subsequent precise geometric operations. For data scientists and GIS professionals working with extensive geographic datasets, implementing geohash-based processing pipelines can unlock new analytical possibilities and transform previously intractable spatial problems into manageable workflows.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6f78079951d3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Processing Large Geospatial Dataset using Quadtree Partitioning]]></title>
            <link>https://medium.com/@zaenun.faiz/processing-large-geospatial-dataset-using-quadtree-partitioning-4feded11ca93?source=rss-93280e763a34------2</link>
            <guid isPermaLink="false">https://medium.com/p/4feded11ca93</guid>
            <category><![CDATA[geospatial-intelligence]]></category>
            <category><![CDATA[quadtree]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Ahmad Zaenun Faiz]]></dc:creator>
            <pubDate>Sun, 31 Aug 2025 10:51:16 GMT</pubDate>
            <atom:updated>2025-08-31T14:41:22.353Z</atom:updated>
            <content:encoded><![CDATA[<p>The rapid growth in both volume and complexity of geospatial data — particularly from satellite imagery and remote sensing — creates significant computational challenges. Traditional GIS software like QGIS or ArcGIS Pro often cannot handle these massive datasets, necessitating more efficient processing approaches. Conventional methods lead to inefficiencies, excessive memory usage, and long processing times. Additionally, many geospatial datasets require periodic updates, such as urban development data or natural disaster monitoring. These applications demand precise and reliable processing methods to maintain data accuracy and quality. Standard approaches typically load entire files into memory, which becomes impractical for large raster data and causes unnecessary disk I/O operations and computational overhead. Efficient and scalable data processing techniques are therefore essential to address these challenges.</p><h4>Recursive Partitioning and Quadtree Method</h4><p>To overcome the challenges of processing large geospatial datasets, I researched advanced data handling methods implementable in Python. A key discovery was using recursive methods to process data in smaller, manageable chunks. This approach effectively divides large datasets into smaller parts, reducing memory requirements and minimizing unnecessary data reads from storage.</p><p>For this recursive partitioning, I adopted the quadtree method. A quadtree is a tree data structure where each internal node has exactly four children. It partitions two-dimensional space by recursively subdividing it into four quadrants or regions. This method works exceptionally well for geospatial data because it efficiently handles spatial hierarchies and varying data densities. Using a quadtree, large raster files can be broken down into smaller, manageable tiles that can be processed independently.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eda_ToiSkoS1vq_SOCCWRA.png" /><figcaption>Recursive partitioning using Quadtree method</figcaption></figure><h4>Case Studies: Evenly and Unevenly Distributed Data</h4><p>To validate the effectiveness of this recursive splitting method, I developed two case studies. The first used a raster dataset representing evenly distributed geospatial data, where features spread uniformly across the area. In this scenario, the quadtree divided the space into equal-sized chunks, enabling parallel processing of each quadrant. I defined the splitting based on a maximum degree of 5, with the recursive method repeating until each chunk reached a suitable size for efficient analysis.</p><p>The second case study examined the Open Buildings V3 Polygons dataset, which presents a more realistic scenario with clustered data in urban centers and sparse data in rural areas. Here, the adaptive nature of the quadtree method proved crucial. The algorithm efficiently detected data density variations, creating smaller, more detailed quadrants in high-density areas while maintaining larger quadrants in low-density regions. I set the splitting threshold at a maximum of 10,000 building features per chunk. This targeted approach significantly reduced unnecessary data reads and optimized processing by allocating computational resources where most needed.</p><h3>Case 1: Recursive Spatial Data Partitioning with Maximum Longitude-Latitude Degree Approach</h3><p>This first case study tests the effectiveness of recursive partitioning for processing large-scale raster data. The approach aims to improve computational efficiency by breaking down the dataset into smaller, manageable segments, thereby minimizing unnecessary data reads.</p><h4>Approach and Data</h4><p>The partitioning method is applied to geospatial data covering Indonesian territory, using two key datasets:</p><ol><li><strong>Administrative Boundary Data</strong>: Indonesian level 2 administrative boundary polygon data from <a href="https://gadm.org/data.html">GADM</a>, which serves as the spatial reference for defining the relevant area. Total size: 4.8MB.</li><li><strong>Tree Cover Dataset</strong>: High-resolution raster data from Hansen Global Forest Change (<a href="https://storage.googleapis.com/earthenginepartners-hansen/GFC-2022-v1.10/download.html">Hansen Tree Cover</a>), representing evenly distributed spatial data. This dataset’s immense size makes it an ideal candidate for testing recursive partitioning. Total size: 2.12 GB.</li></ol><h4>Step 1: Calculating the Bounding Box Index with Quadtree</h4><p>The first step is creating a spatial index for the entire study area. I selected the <strong>quadtree</strong> method to recursively partition Indonesia’s bounding box. This technique divides each spatial box (quadrant) into four smaller sub-quadrants. The recursion continues until each quadrant is no more than <strong>5 degrees longitude-latitude</strong> in either dimension. This threshold ensures each resulting segment is small enough for efficient processing without overwhelming memory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t3mgomaES7OZT8MrbPAXdA.png" /><figcaption>Indonesia’s bounding boxes recursive process with Quadtree using lon-lat maximum value of 5-degree</figcaption></figure><p>Using this approach, the initial bounding box covering all of Indonesia is broken down into smaller, manageable sections. The process generated <strong>164 valid bounding boxes</strong> with unique indices from B000 to B163. Each bounding box has a uniform size of approximately 3.05 square degrees, well within the established <strong>5-degree maximum</strong> boundary. This systematic recursive quadtree subdivision only retains bounding boxes that intersect with Indonesian territory and eliminates duplicates. The result is a set of optimally sized data chunks ready for efficient raster processing.</p><p>The entire recursive process can be visualized in five progressive depths. At <strong>Depth 0</strong>, the initial bounding box for Indonesia measured 46.04° × 16.94°. This split at <strong>Depth 1</strong> into 4 quadrants, each measuring 23.02° × 8.47°. The recursion continued through <strong>Depth 2</strong> with 16 quadrants (11.51° × 4.24° each) and <strong>Depth 3</strong> with 64 quadrants (5.75° × 2.12° each). The splitting stopped at <strong>Depth 4</strong>, where quadrants reached 2.88° × 1.06° — well below the 5-degree threshold. After filtering out quadrants that didn’t intersect with Indonesia’s administrative boundaries, 164 valid bounding boxes remained.</p><h4>Step 2: Split Raster Data by Bounding Boxes</h4><p>The second step uses the quadtree-generated bounding boxes to split the large raster dataset. This workflow ensures efficient and accurate data extraction. First, the system loads all 164 valid bounding boxes. Then, it identifies which Hansen Global Forest Change raster files intersect with each individual box.</p><p>To prevent dimension mismatch errors and ensure seamless data integration, the process follows a “merge first, then clip” approach. For each bounding box, all intersecting rasters are first merged using the rasterio.merge function. The merged raster is then clipped to match the exact dimensions of the bounding box.</p><p>This process repeats for all 164 bounding boxes, with each clipped raster saved as a single TIF file named by its unique bounding box index (e.g., B001.tif, B002.tif). The workflow successfully produced <strong>130 unified raster files</strong>, demonstrating proper merging and complete coverage across all bounding boxes. The smart intersection detection and efficient temporary file management create a highly automated data preparation pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BOZE0_LBG3mAqs7CvWAelw.png" /><figcaption>Clipping raster result after Quadtree recursive partition process</figcaption></figure><h4>Step 3: Classify Tree Cover by Administrative Areas</h4><p>This step involves implementing the recursive splitting method to analyze the processed tree cover data at the administrative level. The script loads the clipped raster files from the previous step along with the administrative boundaries from GADM. It establishes five tree cover classification categories: <strong>Very Sparse</strong> (&lt;10% tree cover), <strong>Sparse</strong> (10–30%), <strong>Medium</strong> (30–50%), <strong>Dense</strong> (50–75%), and <strong>Very Dense</strong> (&gt;75%). For each of the 502 administrative level 2 areas, the script identifies which processed raster files intersect with the area, clips the data to the administrative boundary, and classifies each pixel according to these predefined categories. This approach enables precise calculation of the percentage of each category within every administrative area, producing a comprehensive CSV output with detailed tree cover statistics for each district and province. The entire process leverages the efficient quadtree partitioning for targeted data access, ensuring thorough analysis across all Indonesian administrative areas.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_saFhT2XFslJ8JlX4vRpXg.png" /><figcaption>Tree cover data processing using Admin 2 from GADM dataset</figcaption></figure><h4>Step 4: Efficiency Analysis for Jakarta Pusat</h4><p>To demonstrate the efficiency gains of the recursive partitioning method, I conducted a side-by-side comparison of processing the Jakarta Pusat administrative area. This analysis shows how the targeted quadtree approach significantly reduces data volume and processing time compared to loading the full original raster files.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QRfz68krZ5Y_KuJ6M4eagA.png" /><figcaption>Data processing comparison with and without partition to process raster data around Jakarta Pusat</figcaption></figure><p><strong>With Partitioning (Targeted)</strong>: The process only needs to load a single bounding box file (B017) containing Jakarta Pusat. This bounding box measures approximately 3.05° × 1.06° with <strong>48.8 million pixels</strong>. This targeted approach enables fast and efficient data access by processing only the relevant data chunk.</p><p><strong>Without Partitioning (Full Scope)</strong>: The traditional approach would require processing the entire original Hansen raster file (00N_110E) covering the Jakarta Pusat area. This massive file spans ~10° × 10° with <strong>1.6 billion pixels</strong>. Processing this immense dataset is significantly slower and less efficient since most of the data is irrelevant to the Jakarta Pusat analysis.</p><p>The results demonstrate remarkable efficiency gains: the partitioned approach is <strong>33 times faster</strong> while processing only a fraction of the data. This represents a data reduction of approximately <strong>97%</strong>, confirming the effectiveness of recursive spatial partitioning for large-scale geospatial analysis.</p><h3>Case 2: Recursive Spatial Data Partitioning with Maximum Feature Numbers Approach</h3><p>To showcase the versatility of recursive partitioning beyond raster data, this second case study examines <strong>polygon data</strong> from the Google Open Buildings dataset. The goal is to efficiently process a massive number of features by ensuring each partitioned segment contains a manageable number of buildings, preventing memory overload and accelerating feature-level analysis.</p><h4>Approach and Data</h4><p>This case study applies partitioning to geospatial data covering Java and Bali’s administrative boundaries. Two primary datasets are used:</p><ol><li><strong>Administrative Boundary Data</strong>: Indonesian level 1 and 2 administrative boundary data from <a href="https://gadm.org/data.html">GADM</a> for Java and Bali provinces, serving as the spatial reference. Total size: 1MB.</li><li><strong>Building Polygon Dataset</strong>: <a href="https://sites.research.google/open-buildings">Google Open Buildings</a> dataset in CSV format. Due to its immense size, the analysis focuses on specific quadkey tiles: 2dd, 2e5, and 2e7. Total size: 22.06GB.</li></ol><h4>Step 1: Convert CSV to GeoPackage</h4><p>The first step in this workflow converts the large tabular data from Google Open Buildings CSV files into a suitable geospatial format. To handle this massive dataset, the process begins by setting up a <strong>Dask distributed computing cluster</strong> for parallel processing. This optimized approach processes all three raw CSV files — 2dd_buildings.csv, 2e5_buildings.csv, and 2e7_buildings.csv—which together contain over <strong>85.8 million rows</strong> of data. The workflow transforms each building&#39;s WKT polygon data into a shapely geometry while preserving all original attributes such as latitude, longitude, and confidence. Finally, the processed data for each file is saved as a separate, highly efficient <strong>GeoPackage (.gpkg)</strong> file, creating 2dd_buildings.gpkg, 2e5_buildings.gpkg, and 2e7_buildings.gpkg.</p><h4>Step 2: Processing and Partitioning Geometries</h4><p>This step implements a targeted partitioning strategy to efficiently manage and process millions of building features. Using the quadtree approach, I divided the massive dataset into <strong>20,433 segments</strong>, with each segment containing between <strong>46 and 8,325 features</strong> — a manageable range for processing. The results are stored in highly efficient <strong>Parquet files</strong> organized in a hierarchical directory structure that facilitates easy data access and scalability. This method ensures optimal distribution of data across files, creating an efficient storage solution for large-scale geospatial analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mDQD3yJ01AridB5zz2hrcw.png" /><figcaption>Quadtree partitioning result with maximum features of 10,000</figcaption></figure><p>The distribution of partitions across each quadtree depth level is as follows:</p><ul><li><strong>Depth 1</strong>: 0 partitions</li><li><strong>Depth 2</strong>: 2 partitions</li><li><strong>Depth 3</strong>: 9 partitions</li><li><strong>Depth 4</strong>: 25 partitions</li><li><strong>Depth 5</strong>: 105 partitions</li><li><strong>Depth 6</strong>: 445 partitions</li><li><strong>Depth 7</strong>: 2,448 partitions</li><li><strong>Depth 8</strong>: 7,093 partitions</li><li><strong>Depth 9</strong>: 7,327 partitions</li><li><strong>Depth 10</strong>: 2,979 partitions</li></ul><p>The Google OpenBuildings Dataset is partitioned using a Quadtree with detail result as mentioned in the figure. This approach follows a simple rule: each partition (leaf node) can contain no more than <strong>maximum of 10,000 building features</strong>. When a quadrant exceeds this limit, it’s further subdivided until all resulting quadrants meet the requirement. Each quadrant receives a unique identifier: 1 for Southwest (SW), 2 for Southeast (SE), 3 for Northwest (NW), and 4 for Northeast (NE). A path like <strong>1/3/2</strong> represents a specific location through a sequence of subdivisions: first into the Southwest quadrant, then to its Northwest sub-quadrant, and finally into the Southeast sub-quadrant of that region. This hierarchical structure optimizes both data storage and retrieval, allowing complex queries about building locations in densely populated areas to be processed with significantly improved speed and efficiency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n9oHiiYUs2DbBuJFFnX8Qw.png" /><figcaption>Simulation of Quadtree Partitioning depths around Java-Bali</figcaption></figure><h4>Step 3: Processing Buildings Data around Jakarta Pusat</h4><p>Processing Google Open Building data for Jakarta Pusat provides a compelling example of the <strong>Quadtree partitioning</strong> method’s effectiveness. Without this approach, analysts would need to load the entire 2e7 grid dataset—5.9GB containing over <strong>60 million buildings</strong> (60,037,155 entities). Such massive data loads significantly hinder computational performance, causing high latency and inefficient memory usage.</p><p>The <strong>Quadtree partitioning</strong> method transforms this process by hierarchically dividing geographic space to identify only relevant data sections. For Jakarta Pusat, this approach isolated the necessary data into just <strong>73 unique partitions</strong>. Instead of processing the full 5.9GB dataset, these targeted partitions total only <strong>80.6MB</strong> — a dramatic reduction in data volume.</p><p>This approach yields remarkable efficiency gains. From these compact partitions, I extracted and processed <strong>312,074 buildings</strong> that accurately represent Jakarta Pusat. Though comprising less than 1% of the complete dataset, this subset provides all necessary information for thorough analysis without computational burden. Quadtree partitioning thus proves essential for optimizing large-scale geospatial data processing, delivering faster, more efficient, and resource-conscious operations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lKOGJlgibgydubfgYMQNPg.png" /><figcaption>Comparison of buildings data call with and without partitioning around Jakarta Pusat</figcaption></figure><h3>Recommendations</h3><p>When optimizing geospatial data processing, selecting the appropriate Quadtree partitioning method depends directly on the spatial distribution pattern of your data. Based on data characteristics, two main approaches can maximize efficiency.</p><h4>1. Partitioning Based on Maximum Degree (Longitude-Latitude)</h4><p>This method works best for data with <strong>uniform spatial distribution</strong>, such as raster data (satellite imagery, elevation maps, or temperature data). Since each quadrant maintains relatively consistent data density, Quadtree partitions can be created using a predetermined maximum longitude and latitude degree. This creates balanced partitions where each section contains roughly equal data volume, optimizing computational processes across the entire area and preventing density imbalances between quadrants.</p><h4>2. Partitioning Based on Maximum Features</h4><p>This method is ideal for <strong>unevenly distributed data</strong>, such as vector data representing physical objects like buildings, trees, or population points. For example, in the Google Open Building dataset, urban areas have significantly higher building density than rural regions. By setting a maximum feature limit (e.g., 10,000 buildings per partition), the Quadtree dynamically adjusts its divisions — creating smaller, more detailed sections in dense areas while maintaining larger quadrants in sparse regions. This approach efficiently prevents unnecessary partitioning in empty areas while ensuring each partition maintains an optimal processing load.</p><h4>3. The Importance of Data Statistics</h4><p>Regardless of your chosen method, partitioning must align with your data’s statistical properties to avoid <strong>over-partitioning</strong> or <strong>under-partitioning</strong>. <strong>Over-partitioning</strong> splits data into too many small partitions with very few features each, increasing computational overhead and slowing data access by requiring excessive file operations. <strong>Under-partitioning</strong> creates too few, overly large partitions that burden memory and slow processing — similar to using no partitioning at all. Analyzing key data statistics, particularly average feature density, is therefore crucial for determining optimal partitioning parameters that balance partition count with efficient data size per partition.</p><h3>Conclusion</h3><p>In this discussion, I’ve demonstrated how <strong>Quadtree partitioning</strong> effectively manages and processes large, unevenly distributed geospatial data, such as the Google Open Building dataset. The Jakarta Pusat case clearly shows how this approach can reduce data loads from gigabytes to megabytes, dramatically improving computational efficiency.</p><p>Additionally, I applied a similar method to datasets with more uniform distribution, specifically tree cover data. Unlike buildings that are concentrated in urban areas, tree cover has a relatively uniform distribution pattern across the entire region. In this case, the partitioning approach based on maximum longitude-latitude degrees proved more optimal, producing quadrants with consistent sizes. This demonstrates the flexibility of the Quadtree method, which can be adapted to the spatial distribution characteristics of various types of datasets.</p><p>However, Quadtree is just one of several spatial partitioning models available. While effective for localized areas, Quadtree’s limitation lies in its use of planar projection, which distorts data when applied globally. For worldwide datasets, the geospatial world offers other powerful methods with distinct advantages, including <strong>S2Sphere</strong> which uses spherical geometry and <strong>H3Hexagon</strong> which employs a global hexagonal grid system to maintain consistent spatial relationships across the planet.</p><p>If you’re interested in how these models work and compare to Quadtree, follow my next article. I’ll explore the spherical-based S2Sphere and the hexagonal grid system of H3Hexagon to see how they serve as compelling alternative solutions.</p><p>Wondering which approach best fits your project? Stay tuned for the complete review!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4feded11ca93" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pengelolaan Data Spasial dengan Spatial Data Infrastructure]]></title>
            <link>https://medium.com/@zaenun.faiz/pengelolaan-data-spasial-dengan-spatial-data-infrastructure-e036878af617?source=rss-93280e763a34------2</link>
            <guid isPermaLink="false">https://medium.com/p/e036878af617</guid>
            <category><![CDATA[web-gis-services]]></category>
            <category><![CDATA[geography]]></category>
            <category><![CDATA[spatialdatainfrastructure]]></category>
            <category><![CDATA[gis]]></category>
            <category><![CDATA[geospatial]]></category>
            <dc:creator><![CDATA[Ahmad Zaenun Faiz]]></dc:creator>
            <pubDate>Sat, 17 Aug 2024 10:09:42 GMT</pubDate>
            <atom:updated>2024-08-17T10:10:28.640Z</atom:updated>
            <content:encoded><![CDATA[<p>Kebutuhan akan data spasial di era saat ini sangatlah vital. Hampir di setiap aspek dapat dianalisis dengan menggunakan pendekatan lokasi. Sayangnya, informasi-informasi yang bersifat spasial ini terkadang tidak terkelola dengan baik. Pengelola yang satu dengan yang lain seringkali memiliki standar yang berbeda-beda, sehingga perlu banyak tenaga guna mempersatukan informasi-informasi spasial yang memiliki format yang sangat beragam.</p><p>Spatial Data Infrastructure (SDI) adalah salah satu pengelolaan data spasial yang terintegrasi, dengan menggunakan kerangka berbasis web (WebGIS) serta dapat melakukan interoperasi dengan perangkat lunak berbasis desktop dan penyimpanan di basis data spasial. Posisi SDI yang berperan sebagai jembatan antara ketiga aspek tersebut menjadikan pengelolaan data spasial menjadi lebih mudah guna menghasilkan konsistensi data yang bagus.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QYn2xE0T-AkshGdylBoplw.png" /><figcaption>Spatial Data Infrastructure sebagai jembatan antara WebGIS, GIS Desktop dan Basis Data Spasial</figcaption></figure><blockquote><strong>Apa itu GeoNode?</strong></blockquote><p>GeoNode adalah aplikasi dan perangkat lunak berbasis web untuk pengembangan sistem informasi geografis (GIS) dan menyalurkan infrastruktur data spasial (SDI). Manajemen data pada GeoNode didesain agar pengguna dapat mengintegrasikan pembuatan data, metadata dan visualisasi peta. Setiap dataset dapat dibagikan secara publik<strong> </strong>atau dibatasi aksesnya pada pengguna-pengguna tertentu. Terdapat pula fitur sosial seperti profil pengguna, kolom komentar<strong> </strong>dan sistem rating<strong> </strong>untuk memberikan tanggapan dan kontrol kualitas<strong> </strong>terhadap suatu dataset.</p><p>GeoNode dikembangkan dengan menggunakan Django, sebuah web framework yang dikembangkan dengan menggunakan bahasa pemrograman Python. Dalam menyajikan data spasial, GeoNode di belakang layar menggunakan protokol OGC seperti WFS, WMS, WMS-T, WCS dan sebagainya melalui GeoServer. Nantinya, tampilan data spasial ini dapat divisualkan dengan menggunakan OpenLayers pada sebuah mapview, dashboard maupun geostory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*7bpk9oNZg8K39HYbFaKYVg.png" /><figcaption>Arsitektur GeoNode dalam menyajikan data spasial</figcaption></figure><p>GeoNode didesain untuk menghasilkan kolaborasi antar perangkat dan berbagai macam tipe data dalam satu kesatuan infrastruktur. Sehingga data yang dihasilkan akan berada dalam satu standar penyimpanan data, meski diproduksi dari berbagai macam teknik dan metode pengambilan dan pembuatan data.</p><p>Berbagai permasalahan<strong> </strong>yang selalu dihadapi oleh praktisi GIS seperti format data yang berbeda-beda, sistem koordinat beragam, maupun permasalahan lainnya dapat diatasi dengan integrasi<strong> </strong>antara GIS<strong> </strong>dengan SDI. Tanpa adanya integrasi ini, para praktisi GIS menghabiskan banyak waktunya untuk mengolah data spasial dari berbagai macam format dan sumber data.</p><blockquote><strong>GeoNode adalah proyek terbuka, open source</strong></blockquote><p>Berbagai macam layanan SDI yang bersifat komersial seperti ArcGIS Online dan ArcGIS Enterprise membanderol harga layanan mereka dengan harga yang tergolong mahal. GeoNode ini adalah sebuah SDI alternatif yang dapat dikembangkan tanpa harus membeli lisensi perangkat lunak, dikarenakan proyek ini bersifat terbuka atau open source.</p><p>GeoNode adalah software open source dengan lisensi GNU General Public License (GPL 3.0). Lisensi ini membolehkan penggunanya untuk menggunakan secara bebas, serta diperbolehkan untuk memodifikasi maupun mendistribusikannya secara umum dengan ketentuan hasil modifikasi harus memiliki lisensi serupa.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8u2WVo9-wRn__3aANW3gFQ.png" /></figure><blockquote><strong>Apa saja yang harus dipelajari untuk mengoperasikan GeoNode?</strong></blockquote><p>Secara umum, GeoNode memperkenalkan 3 (tiga) jenis peran yakni pengguna (<em>users</em>), pengelola (<em>admins</em>) dan pengembang (<em>developers</em>). Setiap peran ini memiliki spesifikasi dan keahlian yang berbeda satu sama lain.</p><h4><em>GeoNode untuk pengguna (users)</em></h4><p>Peran dari seorang pengguna adalah membuat, membagikan dan kolaborasi dalam pengelolaan dataset spasial. Pengguna lebih bertanggung jawab pada pengelolaan data seperti menambahkan data, pembaruan data, kurasi data pengguna lain serta memberikan informasi-informasi tambahan seperti abstraksi, lisensi dan sebagainya. Pengguna juga dapat mempublikasi data dalam bentuk mapviews, dashboards maupun geostories.</p><h4>GeoNode untuk pengelola (<em>admins</em>)</h4><p>Jika pengguna lebih bertanggung jawab terhadap data, admin lebih bertanggungjawab terhadap penggunaan perangkat lunak GeoNode itu sendiri. Peran ini antara lain seperti membuat batasan-batasan akses, pengumuman bagi para pengguna, mengubah tata letak dan tampilan fungsionalitas web, membuat grup berdasarkan topik dan sebagainya. Admin berperan penting agar para pengguna GeoNode ini bisa berjalan sesuai dengan kebijakan penggunaan perangkat lunak ini.</p><h4>GeoNode untuk pengembang (<em>developers</em>)</h4><p>Peran ini memerlukan keahlian khusus dalam bahasa pemrograman seperti Python dan JavaScript. Secara umum, peran ini bertugas untuk menambahkan fitur-fitur baru seperti template dashboard, template mapview, menambah kapasitas mesin data dan sebagainya. Mengenai peran ini, telah saya jabarkan lebih lanjut pada <a href="https://medium.com/@zaenun.faiz/open-source-webgis-development-from-a-to-z-496fe3ca455f">artikel berikut</a> mengenai pengembangan WebGIS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7MkuqYQTSGF_K7MNMplmtQ.png" /></figure><p>Jadi, apakah kamu tertarik untuk menyelami data spasial lebih jauh dengan menggunakan Spatial Data Infrastructure?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e036878af617" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Open Source WebGIS Development from A to Z]]></title>
            <link>https://medium.com/@zaenun.faiz/open-source-webgis-development-from-a-to-z-496fe3ca455f?source=rss-93280e763a34------2</link>
            <guid isPermaLink="false">https://medium.com/p/496fe3ca455f</guid>
            <category><![CDATA[geographic-information]]></category>
            <category><![CDATA[geography]]></category>
            <category><![CDATA[gis]]></category>
            <category><![CDATA[spatial]]></category>
            <category><![CDATA[web-gis-services]]></category>
            <dc:creator><![CDATA[Ahmad Zaenun Faiz]]></dc:creator>
            <pubDate>Mon, 26 Jun 2023 16:28:15 GMT</pubDate>
            <atom:updated>2023-06-26T17:24:06.112Z</atom:updated>
            <content:encoded><![CDATA[<p>Tak bisa dipungkiri, perkembangan teknologi yang semakin pesat mendorong aspek-aspek kehidupan yang berubah menuju ke era digital, tak terkecuali ilmu mengenai data spasial.</p><p>Peta adalah bentuk awal dari penyajian data spasial untuk memberikan informasi mengenai ruang. Peta sebagai representasi data spasial mulai bergeser dari dalam wujud peta cetak menuju ke peta digital. Hingga saat ini, kita mengenal istilah peta digital berbasis web yang dikenal dengan nama webgis.</p><p>Praktisi geospasial sekarang tidak hanya dituntut mahir membuat peta cetak, tetapi juga peta digital. Jadi, apa yang harus dipelajari agar bisa membuat webgis?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z8auPJLVcwxjF1I9-xw4Zg.png" /><figcaption>Road map belajar membuat full-stack webgis untuk pemula</figcaption></figure><blockquote><strong>Mulai dari QGIS</strong></blockquote><p>Hal pertama yang saya pelajari untuk membuat webgis adalah QGIS. Kenapa QGIS?</p><ul><li><em>Opensource</em>, tidak memerlukan lisensi premium apapun</li><li>Memiliki kemampuan <em>cross-platform</em> yang baik, mendukung lebih banyak format data ketimbang aplikasi GIS Desktop lainnya</li><li>Tidak didesain untuk berjalan di <em>framework</em> tertentu, atau dikenal sebagai <em>framework agnostic</em></li></ul><p>QGIS mendukung berbagai teknologi geospasial modern seperti integrasi dengan database spasial serta <em>open geospatial consortium</em> (OGC) framework seperti WFS, WMS, WMS-T, WCS dsb. QGIS juga mendukung untuk koneksi ke database seperti PostGIS, Oracle Spatial, Geopackage, SpatiaLite dsb. serta dapat melakukan kueri SQL secara langsung menggunakan toolbar database.</p><blockquote><strong>Mempelajari dasar-dasar tampilan antar muka web dengan HTML, CSS dan JS</strong></blockquote><p>Pengembangan tampilan antar muka dengan menggunakan HTML, CSS dan JavaScript agar pengguna dapat berinteraksi dengan <em>website</em> tersebut. Ketiga komponen ini menjadi tulang punggung pengembangan tampak muka sebuah web. Fungsi masing-masing komponen ini dapat dijabarkan sebagai berikut:</p><ul><li><strong>Hypertext Markup Language</strong> <strong>(HTML) </strong>adalah sebuah kerangka web yang berfungsi agar halaman web bisa diakses pengguna melalui web browser.</li><li><strong>Cascading Stylesheet (CSS) </strong>adalah kerangka pada web untuk menghasilkan karakter dari kerangka yang dijabarkan pada HTML seperti warna, ukuran font, bentuk margin dsb.</li><li><strong>JavaScript (JS)</strong> adalah kerangka agak membuat tampilan web menjadi lebih dinamis.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*Isi5uf1b_fxf3bDcEYj-9A.gif" /><figcaption>Kerangka web dengan HTML, CSS dan JS. Source: <a href="https://legacycap.hostgator.com/">https://legacycap.hostgator.com/</a></figcaption></figure><blockquote><strong>Menggunakan JavaScript Mapping Library</strong></blockquote><p>Setelah kita memahami bagaimana HTML, CSS dan JavaScript bekerja, langkah berikutnya adalah mendalami library JS yang dapat digunakan untuk membuat peta interaktif. Di dunia open source, library JavaScript yang paling populer digunakan untuk webgis adalah Leaflet dan OpenLayers. Ada beberapa pertimbangan dalam memilih JS mapping library untuk sebuah aplikasi peta berbasis web, terutama terkait spesifikasinya.</p><p>Leaflet didesain untuk menjadi library peta yang ringan dan fitur dasar lyang lengkap untuk hanya menampilkan peta di dalam web. Sebaliknya, OpenLayers menawarkan fitur-fitur GIS yang lebih mumpuni ketimbang Leaflet seperti sistem proyeksi, penyajian geometri dengan SRID yang berbeda-beda dll.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9EduwjMVogbIOmuZl_w6rA.png" /><figcaption>Pertimbangan yang perlu dilakukan sebelum memilih antara Leaflet atau OpenLayers</figcaption></figure><blockquote><strong>Membuat web-service dan GeoAPI dengan menggunakan OGC-standard web service melalui GeoServer</strong></blockquote><p>Web service atau layanan web adalah sebuah layanan pada perangkat elektronik agar dapat terkoneksi dengan perangkat elektronik lainnya. Web service ini terdiri atas API atau <em>Application Programming Interface</em> yang berfungsi sebagai jembatan antar aplikasi dalam melakukan pertukaran atau transaksi data. OGC-standard web service sendiri merupakan framework baku untuk pertukaran data spasial berbasis web. OGC-standard yang didukung di QGIS antara lain:</p><ul><li><strong>WMS/WMTS</strong> — Web service spasial untuk pertukaran data raster. Contoh: GeoTIFF, Geospatial JPG/PNG</li><li><strong>WFS </strong>— Web service spasial untuk pertukaran data vektor. Contoh: KML, GeoJSON</li><li><strong>WCS</strong> — Web service untuk menghasilkan cakupan ruang secara spasial, representasi fenomena dengan dimensi ruang dan waktu</li></ul><p>GeoServer adalah sebuah layanan web untuk menghasilkan pertukaran data spasial di berbagai macam perangkat elektronik yang terkoneksi dengan internet, baik pada PC ataupun Mobile. GeoServer mendukung format-format data spasial dengan menggunakan standar API data spasial OGC yakni WFS, WMS, WMTS, WCS dan lain sebagainya.</p><p>Penggunaan GeoServer sebagai layanan web bertujuan agar data spasial saling terintegrasi satu sama lain, antar platform dan antar aplikasi. Sehingga semua data spasial akan memiliki nilai dan variabel yang sama, di setiap perangkat yang terkoneksi dengan GeoServer.</p><blockquote><strong>Spasial Database dengan PostgreSQL dan PostGIS</strong></blockquote><p>Kita pasti sudah tidak asing menyimpan data spasial dengan menggunakan ekstensi file spasial milik ESRI, yakni Shapefile. Penyimpanan data dalam bentuk file dilihat sebagai sebuah sistem penyimpanan yang praktis, tanpa harus melakukan konfigurasi yang bermacam-macam.</p><p>Namun jika jumlah datanya sudah terlampau banyak, penyimpanan data dalam bentuk file akan menjadi sebuah bencana. Terkadang file yang berbeda akan menghasilkan data yang berbeda pula. Jika kita memiliki ratusan bahkan ribuan data, file akan berat untuk dibuka dan menghasilkan konsistensi data yang buruk seiring dengan bertambahnya informasi. Di kondisi inilah kita memerlukan sebuah database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/240/1*rUg6kvl1p1dktU65PyLrfg.png" /><figcaption>PostGIS, ekstensi database spasial yang mendukung berbagai macam operasi spasial</figcaption></figure><p>PostgreSQL/PostGIS tergolong dalam database relasional (RDBMS). Transaksi data pada database relasional / SQL dilakukan dengan melakukan kueri data. Secara umum ada 4 tipe komando untuk mengelola data pada database relasional yakni:</p><ul><li>DDL — Data Definition Language</li><li>DQL — Data Query Language</li><li>DML — Data Manipulation Language</li><li>DCL — Data Control Language</li></ul><p>Pada database spasial seperti PostgreSQL/PostGIS, terdapat berbagai macam kueri-kueri spasial seperti ST_Buffer, ST_Intersect, ST_Area, ST_Transform dan lain sebagainya. Kueri-kueri inilah yang digunakan untuk menghasilkan fungsi manipulasi data spasial di dalam website.</p><blockquote><strong>Full-stack WebGIS Development dengan Django dan GeoDjango</strong></blockquote><p>Produk akhir dari sebuah webgis pastinya menginginkan fitur-fitur yang premium. Bukan hanya jendela peta interaktif saya yang termuat dalam webgis, namun juga berbagai macam sistem penunjangnya seperti Sistem Login, Sistem Input Data, Administrasi Website, User Management, dan lain sebagainya. Sehingga pengguna bisa melakukan navigasi data spasial dengan mudah.</p><p>Webgis dengan spesifikasi semacam ini memerlukan sebuah framework backend yang mumpuni agar menghasilkan website yang rapi dan sistematis. Bahasa dan framework backend yang bisa digunakan untuk webgis antara lain Java, Ruby, Python, Go, NodeJS dan PHP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*8Gr2pMDdhQVf89D6TLuVag.jpeg" /><figcaption>Source: <a href="https://www.kanhasoft.com/blog/why-django-is-the-best-web-framework-for-your-project/">kanhasoft.com</a></figcaption></figure><p>Kenapa memilih Django? Jawabannya adalah fitur <em>built-in geopastial </em>pada Django yang sudah sangat matang, yang dikenal sebagai GeoDjango. Framework ini sangatlah praktis, berbeda jika kita menggunakan framework lainnya yang harus membuat operasi spasial dari nol.</p><p>Django memiliki interkoneksi yang baik dengan PostgreSQL. Selain itu, Django juga mendukung database lainnya seperti MariaDB, MySQL, Oracle dan SQLite. Selain fitur autentikasi dan koneksi database, Django juga mendukung berbagai macam fitur lainnya seperti admin page, view layer, template layer, forms, security settings, dan geographic framework dengan GeoDjango.</p><p>GeoDjango memiliki koneksi yang baik dengan PostGIS serta menggunakan OpenLayers sebagai framework front-end spasial bawaannya. Sehingga mampu menghasilkan fitur-fitur spasial yang lengkap pada web. GeoDjango juga menggunakan GDAL sebagai tulang punggung operasi data spasial. Framework ini memungkinkan untuk menghasilkan kapabilitas pengolahan data geospasial tingkat tinggi seperti transformasi sistem koordinat, kalkulasi geometris dan geografis serta pengolahan data raster.</p><blockquote>Learn from experience is the best learning method</blockquote><p>Dengan menggunakan rincian stack di atas, kita sudah siap menapaki jalan sebagai seorang webgis developer dengan kemampuan membuat <em>full-stack </em>webgis secara lengkap dan sistematis. Kita bisa menggunakan Django/GeoDjango untuk membuat kerangka web keseluruhan, PostgreSQL/PostGIS sebagai database data spasial dan non spasial aplikasi webgis serta menggunakan OpenLayers dan fungsi-fungsi geospasial pada GDAL untuk menghasilkan interaksi data spasial yang baik.</p><p>GeoServer juga bisa ditambahkan jika menginginkan interkoneksi dengan berbagai macam apps dan platform lainnya sehingga menghasilkan data spasial yang terintegrasi.</p><p>Jadi, sudah sejauh manakah kemampuan webgis-mu?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=496fe3ca455f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>