<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mo Sarwat on Medium]]></title>
        <description><![CDATA[Stories by Mo Sarwat on Medium]]></description>
        <link>https://medium.com/@msarwat?source=rss-77dc6e0006e4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*iOMYVAKrr761GWG0Qyfd3g.jpeg</url>
            <title>Stories by Mo Sarwat on Medium</title>
            <link>https://medium.com/@msarwat?source=rss-77dc6e0006e4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 02:29:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@msarwat/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[H3 Spatial Grid Support in Apache Sedona]]></title>
            <link>https://medium.com/@msarwat/h3-spatial-grid-support-in-apache-sedona-4fcb924127e5?source=rss-77dc6e0006e4------2</link>
            <guid isPermaLink="false">https://medium.com/p/4fcb924127e5</guid>
            <category><![CDATA[spatial]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Mo Sarwat]]></dc:creator>
            <pubDate>Mon, 16 Oct 2023 15:20:43 GMT</pubDate>
            <atom:updated>2023-10-16T15:20:43.862Z</atom:updated>
            <content:encoded><![CDATA[<p>Uber H3 is a technique to model and index and geospatial data. H3 grids the earth surface by casting it on a icosahedron, and tiles it by hexagons + pentagons. <a href="https://github.com/apache/sedona">Apache Sedona</a> is a spatial computing engine that enables developers to easily process spatial data at any scale within modern cluster computing systems such as Apache Spark and Apache Flink. Sedona developers can express their spatial data processing tasks in Spatial SQL, Spatial Python or Spatial R. Internally, Sedona provides spatial data loading, indexing, partitioning, and query processing/optimization functionality that enable users to efficiently analyze spatial data at any scale. In this article, we will explain how developers can use H3 in Apache Sedona.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kzHzqCzwdNaU839Os4iWAQ.png" /></figure><h3>First, Why H3 for Spatial Data?</h3><p>H3 is one of many ways to model and index spatial data. However, H3 has many advantages compared to other grid systems, but the following are very important:<br>1. Neighbor Traversal: H3 implemented many high efficient algorithms like find neighbors within a specific distance<br>2. Find the shortest path from cell A to cell b: All this is useful for approximate statistics like hotspot calculation, finding nearest neighbors.</p><p>Actually, companies like Uber uses H3 mainly for ride planning. Even though bare metal H3 does not reflect accurate geometric relationship between shapes, but the approximation is good enough for the Uber ride planning use case. That is because hexagons can reflect the distance between cells since the distance between their centroids are equal.</p><p>Another benefit is that the spatial distortion is largely minimized. The geometry shape of H3 Hexagonal cells are similar while the shape of rectangles change drastically with the coordinates due to the nature of the Mercator projection. This makes H3 more suitable for visualization. This is the other use case for H3 in Uber.</p><h3>Comprehensive support for Uber H3 in Apache Sedona</h3><p>Apache Sedona 1.5.0 now provides the following functions related to Uber H3:</p><ul><li>ST_H3CellIDs(geom: geometry, level: Int, fullCover: true)</li><li>ST_H3CellDistance(cell1: Long, cell2: Long)</li><li>ST_H3KRing(cell: Long, k: Int, exactRing: Boolean)</li><li>ST_H3ToGeom(cells: Array[Long])</li></ul><p>Let’s use the Seattle road network dataset (from OSM) as an example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*hfEJI1EXektttP0L4fQcEg.png" /><figcaption>Seattle Road Network</figcaption></figure><h4>Create H3 cell IDs for geometries</h4><p>You can create H3 cell ids using ST_H3CellIds as follows.</p><pre>roadDf = sedona.read.format(&quot;csv&quot;).option(&quot;header&quot;, &quot;true&quot;).load(PATH_PREFIX + &quot;data/OSM2015_roads.csv&quot;)<br>roadDf = roadDf.selectExpr(<br>    &quot;monotonically_increasing_id() as id&quot;,<br>    &quot;ST_GeomFromWkt(geometry) as road&quot;, &quot;`attr#1` as attr&quot;,<br>    &quot;ST_H3ToGeom(ST_H3CellIDs(ST_GeomFromWkt(geometry), 10, false)) as roadH3Geom&quot;,<br>    &quot;ST_H3CellIDs(ST_GeomFromWkt(geometry), 10, false) as roadH3&quot;<br>)<br>roadDf.show(10)<br><br>+---+--------------------+--------------------+--------------------+--------------------+<br>| id|                road|                attr|          roadH3Geom|              roadH3|<br>+---+--------------------+--------------------+--------------------+--------------------+<br>|  0|LINESTRING (-122....|[tiger:county#Kin...|MULTIPOLYGON (((-...|[6222150966219243...|<br>|  1|LINESTRING (-122....|      [service#alley|MULTIPOLYGON (((-...|[6222150966281830...|<br>|  2|LINESTRING (-122....|    [surface#asphalt|MULTIPOLYGON (((-...|[6222150966517759...|<br>|  3|LINESTRING (-122....|[tiger:county#Kin...|MULTIPOLYGON (((-...|[6222150966243164...|<br>|  4|LINESTRING (-122....|[tiger:county#Kin...|MULTIPOLYGON (((-...|[6222150966286090...|<br>|  5|LINESTRING (-122....|[tiger:county#Kin...|MULTIPOLYGON (((-...|[6222150966439772...|<br>|  6|LINESTRING (-122....|[tiger:county#Kin...|MULTIPOLYGON (((-...|[6222150966722559...|<br>|  7|LINESTRING (-122....|   [highway#service]|MULTIPOLYGON (((-...|[622215096642338815]|<br>|  8|LINESTRING (-122....|[tiger:county#Kin...|MULTIPOLYGON (((-...|[6222150966830039...|<br>|  9|LINESTRING (-122....|      [service#alley|MULTIPOLYGON (((-...|[6222150959224913...|<br>+---+--------------------+--------------------+--------------------+--------------------+</pre><h4>Visualize H3 cells using GeoPandas Plot</h4><p>You can use ST_H3ToGeom to generate the boundary of an H3 cell given its ID. To demonstrate, we will plot a subset of these cells using GeoPandas. The resulting plot is shown below:</p><pre>pandasDf = roadDf.limit(100).toPandas()<br>roadGpd = gpd.GeoDataFrame(pandasDf, geometry=&quot;road&quot;)<br>h3Gpd = gpd.GeoDataFrame(pandasDf, geometry=&quot;roadH3Geom&quot;)<br>ax = h3Gpd.plot(edgecolor=&#39;gray&#39;, linewidth=1.0)<br>ax.set_xlabel(&#39;Longitude (degrees)&#39;)<br>ax.set_ylabel(&#39;Latitude (degrees)&#39;)<br>roadGpd.plot(ax=ax, edgecolor=&#39;black&#39;, linewidth=1.0)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/327/1*kMBN14DcJuKVizIQ5VD5eg.png" /></figure><h4>Join geometries by H3</h4><p>The example above demonstrates how to join two datasets together using their H3 cell IDs. Even more impressively, you can create a ring buffer around the original geometry using ST_H3KRing and find matches using the rings. The following query returns roads located within 10 cells of ST_POINT(-122.390, 47.54717658413222):</p><pre>ringCells = pointDf.selectExpr(f&quot;ST_H3KRing(ST_H3CellIDs(ST_POINT(-122.390, 47.54717658413222), 10, false)[0], 10, true) as cells&quot;)<br># do join<br>ringCells.select(explode(&quot;cells&quot;).alias(&quot;cell&quot;)).alias(&quot;ring&quot;).join(<br>        roadDf.select(&quot;id&quot;, &quot;road&quot;, explode(&quot;roadH3&quot;).alias(&quot;cell&quot;), &quot;roadH3Geom&quot;).alias(&quot;roads&quot;),<br>        expr(&quot;ring.cell = roads.cell&quot;)<br>    ).dropDuplicates([&quot;id&quot;])</pre><p>We can visualize the ring of the cell created by ST_H3KRing(ST_H3CellIDs(ST_POINT(-122.390, 47.54717658413222), 10, false)[0], 10, false) as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/346/1*v4MtPJA9vsdMXOVIR8a0ow.png" /></figure><p>Alternatively, we can fill in all cells in the ring like this ST_H3KRing(ST_H3CellIDs(ST_POINT(-122.390, 47.54717658413222), 10, false)[0], 10, false) :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/346/1*Mb1Hs7fswuTMBqIA232BXQ.png" /></figure><h3>Conclusion:</h3><p>H3 can be used to index spatial data by partitioning the earth surface into hexagons. Apache Sedona is large-scale spatial data processing engine. Sedona 1.5.0 now supports H3 — Developers can create H3 grids, attach H3 grids to geometries, and perfrom spatial operations on geometries using H3 in Apache Sedona. For more details, visit:</p><p>Apache Sedona Github repo: <a href="https://github.com/apache/sedona">https://github.com/apache/sedona</a></p><p>Apache Sedona Website: <a href="https://sedona.apache.org/1.5.0/">https://sedona.apache.org/1.5.0/</a></p><p>H3 docs in Apache Sedona: <a href="https://sedona.apache.org/1.5.0/api/sql/Function/#st_h3celldistance">https://sedona.apache.org/1.5.0/api/sql/Function/#st_h3celldistance</a></p><p>Apache Sedona 1.5.0 Release Notes: <a href="https://sedona.apache.org/1.5.0/setup/release-notes/">https://sedona.apache.org/1.5.0/setup/release-notes/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4fcb924127e5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Processing Spatial Raster Data in Apache Sedona]]></title>
            <link>https://medium.com/@msarwat/processing-spatial-raster-data-in-apache-sedona-3ce3703e49d8?source=rss-77dc6e0006e4------2</link>
            <guid isPermaLink="false">https://medium.com/p/3ce3703e49d8</guid>
            <category><![CDATA[geospatial-data]]></category>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[spatial-computing]]></category>
            <category><![CDATA[spatial-analysis]]></category>
            <category><![CDATA[spatial]]></category>
            <dc:creator><![CDATA[Mo Sarwat]]></dc:creator>
            <pubDate>Thu, 03 Aug 2023 15:14:52 GMT</pubDate>
            <atom:updated>2023-08-03T15:14:52.118Z</atom:updated>
            <content:encoded><![CDATA[<p>Raster data is a type of geospatial data representation of the Earth’s surface and atmosphere from a distance, typically using satellite or aerial sensors. It is a grid-based data structure where each cell, or pixel, in the grid represents a specific location on the Earth’s surface. These pixels can store various types of information, such as elevation, temperature, land cover, precipitation, or any other continuous or categorical attribute associated with that location.</p><h3>What is raster data used for?</h3><p>Some of the key applications of raster data processing include:</p><ol><li>Agriculture: Monitoring crop health, estimating crop yield, detecting disease or pest infestations, and assessing soil moisture and nutrient levels.</li><li>Environmental Monitoring: Tracking deforestation, urban sprawl, land use changes, and assessing the impact of natural disasters like wildfires, floods, and hurricanes.</li><li>Climate Studies: Monitoring climate patterns, studying changes in glaciers and ice caps, and measuring greenhouse gas emissions.</li><li>Natural Resource Management: Monitoring forests, water bodies, and wildlife habitats for conservation and management purposes.</li><li>Disaster Management: Assessing the extent of damage after disasters, aiding in search and rescue operations, and assisting in emergency response planning.</li><li>Urban Planning: Analyzing urban growth, land use patterns, and infrastructure development in cities.</li><li>Geology and Geomorphology: Identifying geological features, mapping geological formations, and studying landforms and terrain.</li><li>Oceanography: Monitoring sea surface temperature, ocean currents, and marine ecosystems.</li><li>Archaeology: Identifying ancient ruins, buried structures, and archaeological sites from aerial and satellite images.</li><li>Transportation and Infrastructure: Monitoring transportation networks, assessing road conditions, and planning new infrastructure development.</li><li>Water Resource Management: Monitoring water quality, detecting pollution, and assessing water availability in reservoirs and rivers.</li><li>Health and Epidemiology: Tracking disease outbreaks, monitoring environmental factors affecting public health, and analyzing the spread of infectious diseases.</li><li>Forestry: Managing and monitoring forest resources, detecting illegal logging activities, and assessing tree health and biomass.</li><li>Land Use and Land Cover Mapping: Classifying and monitoring different land cover types and land use changes over time.</li></ol><p>These are just a few examples of the wide range of applications of remote sensing. As technology advances and data becomes more accessible, remote sensing continues to play an increasingly important role in understanding and managing the Earth’s natural and human-made environments.</p><h3>Raster Support in Sedona</h3><p>There are more than 7000 satellites orbiting the earth, collecting tons of raster data. The scale and complexity of such data makes it difficult to process. <a href="https://github.com/apache/sedona">Apache Sedona</a> has been implementing scalable spatial vector data functions for several years, and the support has become mostly mature. In recent releases, the Sedona community has invested more in scalable raster data support, including raster data reader, writer, and raster manipulation. To enable comprehensive raster data support, Sedona has made the following efforts:</p><h4><strong>Raster data type</strong></h4><p>Sedona introduces a native raster data type. Similar to the geometry type, the raster type indicates that the column in a schema contains raster data. Each record stored in this type contains two parts: metadata and raster band information. With this in place, we can create a table that has both a geometry column and a raster column.</p><h4>Raster reader</h4><p>To begin processing raster data, the first step is to load the raster data. Additionally, we must be able to load raster data in parallel given the vast amount of available raster data. SEDONA-251 introduces a scalable raster reader and native raster data constructors. For example, we can read raster data files in Sedona as follows:</p><pre>sedona.read.format(&quot;binaryFile&quot;)<br>.load(&quot;raster/*.tiff&quot;).createOrReplaceTempView(&quot;binaryTable&quot;)</pre><p>To construct a raster column, use the following steps. We have added additional constructors, such as RS_FromGeoTiff and RS_FromAscGrid, which allow you to convert various raster binary formats to a unified raster format.</p><pre>CREATE OR REPLACE TEMP VIEW rasterTable AS<br>SELECT RS_FromGeoTiff(binaryTable.content) AS raster<br>FROM binaryTable</pre><p>We can print the schema of this table. As we can see, the type of this column now is raster.</p><pre>sedone.table(&quot;binaryTable&quot;).printSchema()</pre><pre>root<br> |-- raster: raster (nullable = true)</pre><h4>Raster writer</h4><p>After processing raster data, it is necessary to store the data in an external storage. With SEDONA-269, this can be achieved in Sedona Spark as follows:</p><p>First, convert the raster data to a specific binary format, such as GeoTiff. Note that the raster can be imported from one image format, but exported to another image format.</p><pre>SELECT RS_AsGeoTiff(raster) AS image<br>FROM rasterTable</pre><p>After processing the data, we can save it to an external location.</p><pre>sedona.table(&quot;rasterTable&quot;).write.format(&quot;raster&quot;).mode(SaveMode.Overwrite).save(&quot;my_raster_file&quot;)</pre><h4>Raster functions</h4><p>SEDONA-251, SEDONA-269, and SEDONA-292 together introduce several functions to transform and analyze raster data. Let’s examine a few examples.</p><p>First, let’s assume that we have a GeoTiff image as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V9UNybyRDc7KVMeAAngC3w.jpeg" /></figure><p><strong>Extract bounding box, SRID and number of bands</strong></p><p>We can extract the bounding box, srid, and number of bands of this raster.</p><pre>SELECT raster, RS_Envelope(raster) as bbox, RS_Metadata(raster), RS_NumBands(raster)<br>FROM rasterTable</pre><pre>+--------------------+--------------------+-----+--------+<br>|              raster|                geom| srid|numBands|<br>+--------------------+--------------------+-----+--------+<br>|GridCoverage2D[&quot;g...|POLYGON ((590520 ...|32631|       3|<br>+--------------------+--------------------+-----+--------+</pre><p>Now this resulting table has a schema that contains both raster and geometry type (see below)</p><pre>root<br> |-- raster: raster (nullable = true)<br> |-- geom: geometry (nullable = true)<br> |-- srid: integer (nullable = true)<br> |-- numBands: integer (nullable = true)</pre><p>This allows for more complex geometric operations on the table. For instance, we can save the table as a GeoParquet format and perform filter pushdown. When executing a Sedona spatial range query on this GeoParquet table, Sedona will retrieve only the data that intersects the spatial range query window.</p><pre>sedona.table(&quot;rasterTable&quot;).write.format(&quot;geoparquet&quot;).save(&quot;rasterTable.parquet&quot;)</pre><p><strong>Extract individual band</strong></p><p>We can extract any band in a raster.</p><pre>SELECT RS_BandAsArray(raster, 1) as band1, RS_BandAsArray(raster, 2) as band2,<br> RS_BandAsArray(raster, 3) as band3<br>FROM rasterTable</pre><pre>+--------------------+--------------------+--------------------+<br>|               band1|               band2|               band3|<br>+--------------------+--------------------+--------------------+<br>|[0.0, 799.0, 788....|[0.0, 555.0, 546....|[0.0, 330.0, 322....|<br>+--------------------+--------------------+--------------------+</pre><p><strong>Modify a band value</strong></p><p>We can modify a band value and save it to the original raster</p><pre>SELECT RS_AddBandFromArray(raster, RS_GreaterThan(band1, 0), 1) AS raster)<br>FROM rasterTable</pre><p>The resulting GeoTiff looks like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VyDUOjpqQT9S_swV66poaQ.jpeg" /></figure><h3>Conclusion</h3><p>Raster data processing is quite essential for data and AI teams working with massive-scale satellite imagery and earth observation datasets. Apache Sedona now provides extensive support for scalable raster data reading, writing, and processing functionality. That will democratize the process of using such data in applications. To use Apahche Sedona Today, visit the <a href="https://github.com/apache/sedona">Github repo</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3ce3703e49d8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Sedona Frequently Asked Questions (FAQ)]]></title>
            <link>https://medium.com/@msarwat/apache-sedona-frequently-asked-questions-faq-b4139f142c97?source=rss-77dc6e0006e4------2</link>
            <guid isPermaLink="false">https://medium.com/p/b4139f142c97</guid>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[gis]]></category>
            <dc:creator><![CDATA[Mo Sarwat]]></dc:creator>
            <pubDate>Thu, 08 Dec 2022 19:07:27 GMT</pubDate>
            <atom:updated>2022-12-08T19:15:47.188Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LGpR-YYsLzOmfzgexuc5EA.png" /></figure><p><a href="https://github.com/apache/incubator-sedona">Apache Sedona</a> is a distributed spatial analytics platform that provides tools for working with geospatial data in a distributed computing environment. It is an open-source project of the Apache Software Foundation. Apache Sedona provides APIs and libraries for working with geospatial data in the Scala, Java, Python and SQL programming languages, and it offers support for a wide range of geospatial data formats. It also provides tools for spatial indexing, querying, and spatial join operations, as well as support for common spatial analytics tasks such as clustering and classification. Apache Sedona is designed to enable scalable and efficient analysis of large datasets, and it can be deployed in standalone, local, or cluster modes. This answers a few frequently asked questions about Apache Sedona.</p><p><strong>1. What are the top features of Apache Sedona?</strong></p><p>Apache Sedona is a distributed spatial analytics platform that provides APIs and libraries for working with geospatial data in a distributed computing environment. Some of the key features of Apache Sedona include:</p><ul><li>Support for a wide range of geospatial data formats, including GeoJSON, WKT, and ESRI Shapefile.</li><li>Scalable distributed processing of large datasets.</li><li>Tools for spatial indexing, spatial querying, and spatial join operations.</li><li>Support for common spatial analytics tasks, such as clustering, classification, and regression analysis.</li><li>Integration with popular big data tools, such as Apache Spark, Apache Hadopp, Apache Hive, and Apache Flink for data storage and querying.</li><li>A user-friendly API for working with geospatial data in the Scala and Java programming languages.</li><li>Flexible deployment options, including standalone, local, and cluster modes.</li></ul><p>These are some of the key features of Apache Sedona, but it may offer additional capabilities depending on the specific version and configuration.</p><p><strong>2. How is Apache Sedona Compared to PostGIS</strong><br>Apache Sedona and PostGIS are both tools for working with geospatial data, but they have some key differences. Apache Sedona is a distributed spatial analytics platform that provides APIs and libraries for working with geospatial data in a distributed computing environment. It is designed to enable scalable and efficient analysis of large datasets. PostGIS, on the other hand, is a spatial database extension for the popular open-source database PostgreSQL. It adds spatial data types, functions, and indexing capabilities to PostgreSQL, allowing users to store, query, and analyze geospatial data.</p><p><strong>3. Does Apache Sedona support Spatial SQL?</strong> <br>Yes, Apache Sedona provides support for spatial SQL, which is a specialized variant of the SQL language that is designed specifically for working with spatial data. Spatial SQL includes a range of spatial data types, functions, and operators that can be used to perform spatial queries, analysis, and other operations on spatial data. For example, you can use spatial SQL to create, index, and query spatial data sets, or to perform spatial joins and other operations on spatial data. Apache Sedona also provides APIs for working with spatial SQL in both Scala and Python, making it easy to use with the Spark ecosystem.</p><p><strong>4. Does Sedona Support Spatial Python?<br></strong>Yes, Apache Sedona provides support for spatial Python, which is a set of Python libraries and tools for working with spatial data. These libraries and tools provide a range of functions and APIs for performing spatial queries, analysis, and visualization on spatial data sets. For example, you can use spatial Python to load, manipulate, and analyze spatial data, or to create interactive visualizations of spatial data. Apache Sedona also provides support for integrating spatial Python with the Spark ecosystem, making it easy to use with distributed computing platforms such as Apache Spark, Hive, and Flink. In addition, Apache Sedona provides a number of tutorials and examples that demonstrate how to use spatial Python in a variety of different scenarios.</p><p><strong>5. Does Apache Sedona Support Spatial Join?</strong><br>​​Yes, Apache Sedona provides support for spatial join, which is a type of operation that is commonly used in spatial data analysis. A spatial join is a process of combining two or more spatial data sets based on their spatial relationship, such as their location or proximity to one another. For example, you might use a spatial join to combine a data set of customer locations with a data set of stores, in order to find the store nearest to each customer. Apache Sedona provides a number of functions and APIs for performing spatial join operations, including support for different types of spatial joins, such as nearest neighbor, intersection, and within distance. You can use these functions and APIs to perform spatial join operations on spatial data sets in a distributed and scalable way, using Apache Spark.</p><p><strong>6. Does Apache Sedona support GeoJSON ?<br></strong>Yes, Apache Sedona provides support for the GeoJSON format, which is a widely used open standard for encoding geographic data as JSON objects. GeoJSON is commonly used to represent spatial data in web applications and other contexts, and is supported by many different tools and libraries for working with spatial data. Apache Sedona includes a number of functions and APIs for working with GeoJSON data, including support for reading and writing GeoJSON data, converting between GeoJSON and other spatial data formats, and performing spatial queries and analysis on GeoJSON data sets. You can use these functions and APIs to integrate GeoJSON support into your Apache Sedona applications and work with GeoJSON data in a distributed and scalable way using Apache Spark.</p><p><strong>7. Can you create spatial indexes in Apache Sedona?</strong><br>Yes, Apache Sedona provides support for creating spatial indexes on spatial data sets. A spatial index is a data structure that is used to optimize the performance of spatial queries and other operations on spatial data. Spatial indexes use spatial data structures, such as quadtrees, R-trees, or H3 grids to organize the data in a way that allows for efficient spatial queries and analysis. Apache Sedona includes a number of functions and APIs for creating spatial indexes on spatial data sets, including support for different types of spatial indexes and indexing strategies. You can use these functions and APIs to create spatial indexes on your spatial data sets in Apache Sedona, and improve the performance of your spatial applications.</p><p><strong>8. Does Sedona Support ShapeFiles?</strong><br>Yes, Apache Sedona provides support for ShapeFiles, which is a widely used spatial data format for storing geographic data. ShapeFiles are commonly used to represent spatial data in GIS applications, and are supported by many different tools and libraries for working with spatial data. Apache Sedona includes a number of functions and APIs for working with ShapeFiles, including support for reading and writing ShapeFiles, converting between ShapeFiles and other spatial data formats, and performing spatial queries and analysis on ShapeFiles. You can use these functions and APIs to integrate ShapeFile support into your Apache Sedona applications and work with ShapeFiles in a distributed and scalable way.</p><p><strong>9. Can Sedona process Raster Data?</strong><br>Yes, Apache Sedona is well-suited for processing raster data, which is a type of spatial data that is commonly used in many different applications and industries. Raster data is a grid of cells that are organized into rows and columns, and each cell in the grid contains a value or set of values that represent some aspect of the spatial information being represented. Apache Sedona provides a range of functions and APIs for working with raster data, including support for reading and writing raster data, performing spatial queries and analysis on raster data sets. You can use these functions and APIs to process raster data in Apache Sedona, and perform a wide range of spatial analysis tasks on raster data sets. Apache Sedona’s distributed and scalable architecture makes it particularly well-suited for processing large and complex raster data sets, and can help you to extract insights and information from raster data in a efficient and effective way.</p><p><strong>10. What are the use cases of Sedona?</strong><br>Apache Sedona is a widely used framework for working with spatial data, and it has many different use cases and applications. Some of the main use cases for Apache Sedona include:</p><ul><li><em>Urban planning and development:</em> Apache Sedona is commonly used in urban planning and development applications to analyze and visualize spatial data sets related to urban environments, such as land use, transportation networks, and population density.</li><li><em>Geospatial analytics:</em> Apache Sedona is widely used in geospatial analytics applications, where it is used to perform spatial analysis and data mining on large and complex spatial data sets.</li><li><em>Location-based services:</em> Apache Sedona is often used in location-based services, such as mapping and navigation applications, where it is used to process and analyze spatial data to provide location-based information and services to users.</li><li><em>Environmental modeling and analysis:</em> Apache Sedona is used in many different environmental modeling and analysis applications, where it is used to process and analyze spatial data related to environmental factors, such as air quality, water quality, and weather patterns.</li><li><em>Disaster response and management:</em> Apache Sedona is used in disaster response and management applications to process and analyze spatial data related to disasters, such as floods, earthquakes, and other natural disasters, in order to support emergency response and recovery efforts.</li></ul><p>These are just a few examples of the many different use cases for Apache Sedona. It is a versatile and powerful framework for working with spatial data, and is used in many different industries and applications.</p><p><strong>11. Can Sedona be used to solve problems in the Automotive industry</strong></p><p>Yes, automotive is one of the many potential use cases for Apache Sedona. Apache Sedona is a widely used framework for working with spatial data, and it can be used in many different automotive applications, such as:</p><ul><li><em>Automotive mapping and navigation:</em> Apache Sedona can be used to process and analyze spatial data related to automotive mapping and navigation, such as road networks, traffic patterns, and points of interest. This can be used to provide location-based services to drivers, such as real-time traffic information and route planning.</li><li><em>Automotive safety and autonomous vehicles:</em> Apache Sedona can be used to process and analyze spatial data related to automotive safety and autonomous vehicles, such as sensor data from cameras, lidar, and other sensors. This can be used to support the development of advanced driver assistance systems (ADAS) and autonomous vehicle technologies.</li><li><em>Automotive logistics and supply chain management:</em> Apache Sedona can be used to process and analyze spatial data related to automotive logistics and supply chain management, such as data on vehicle routes, delivery schedules, and vehicle tracking. This can be used to optimize and improve the efficiency of automotive supply chain operations.</li></ul><p>These are just a few examples of how Apache Sedona can be used in the automotive industry. It is a powerful and versatile framework for working with spatial data, and can be used in many different ways to support automotive applications and technologies.</p><p><strong>12. Does Apache Sedona Support GeoFencing?</strong><br>Yes, Apache Sedona provides support for geofencing, which is a technique for defining virtual boundaries or areas around a geographic location. Geofencing is commonly used in a variety of applications, such as location-based services, security and surveillance, and asset tracking. Apache Sedona includes a number of functions and APIs for working with geofences, including support for creating and managing geofences, performing spatial queries and analysis on geofences, and visualizing geofences. You can use these functions and APIs to integrate geofencing support into your Apache Sedona applications and work with geofences in a distributed and scalable way. For example, you could use Apache Sedona to create a geofence around a particular geographic location, and then use spatial queries to find all of the objects or events that occurred within that geofence.</p><p><strong>13. What data sources is Apache Sedona able to connect to?</strong><br>Apache Sedona does not directly connect to any specific data sources, but instead provides a range of functions and APIs for working with spatial data that can be used with data from a variety of different sources. Apache Sedona can be used to work with spatial data from many different sources, such as spatial data files, databases, web services, sensors, and other sources. Apache Sedona provides support for a number of different spatial data formats, including GeoJSON, ShapeFiles, and raster data, and can be used to read, write, and process spatial data from these and other sources. You can use Apache Sedona to work with spatial data from any source that is compatible with the Sedona API, and to perform a wide range of spatial analysis and visualization tasks on that data.</p><p><strong>14. Does Apache Sedona support spatial stream processing?</strong><br>Yes, Apache Sedona provides support for spatial stream processing, which is a technique for analyzing and processing spatial data in real-time as it is being generated or streamed from a data source. Spatial stream processing is commonly used in applications where it is important to analyze and act on spatial data in real-time, such as location-based services, sensor networks, and real-time tracking and monitoring systems. Apache Sedona includes a number of functions and APIs for working with spatial data streams, including support for creating, managing, and analyzing spatial data streams, and for integrating spatial stream processing with other components of the modern data stack. You can use these functions and APIs to build real-time spatial stream processing applications with Apache Sedona, and to analyze and act on spatial data as it is being generated in real-time.</p><p><strong>15. How is Apache Sedona compared to ArcGIS</strong><br>Apache Sedona and ArcGIS are both tools for working with spatial data, but they have some important differences. Apache Sedona is a distributed spatial data processing framework that runs on top of highly scalable data infrastructure. It provides a range of functions and APIs for working with spatial data, and is designed to support large-scale and complex spatial data processing tasks in a distributed and scalable way. In contrast, ArcGIS is a commercial geographic information system (GIS) software platform that is developed and marketed by the company Esri. It provides a range of tools and capabilities for working with spatial data, including support for data management, analysis, visualization, and mapping.</p><p><strong>16. How is Apache Sedona compared to Spark</strong><br>Apache Sedona and Apache Spark are related but distinct technologies. Apache Sedona is a distributed spatial data processing framework that seamlessly integrates with Apache Spark, a popular open-source distributed computing platform. Apache Sedona provides a range of functions and APIs for working with spatial data, and is designed to support large-scale and complex spatial data processing tasks in a distributed and scalable way. Apache Spark, on the other hand, is a general-purpose distributed computing platform that provides a range of APIs and libraries for working with large-scale data sets in a distributed and scalable way. Apache Spark provides the underlying cluster computing infrastructure for Apache Sedona, and enables Apache Sedona to perform spatial data processing tasks in a distributed and scalable way.</p><p><strong>17. How Scalable is Apache Sedona?</strong><br>Apache Sedona is a distributed spatial data processing framework, which means that its performance is determined by the underlying distributed computing infrastructure. The performance of Apache Sedona will depend on a number of factors, such as the size and complexity of the spatial data sets being processed, the specific operations being performed, and the hardware and infrastructure on which it is running. In general, Apache Sedona is capable of performing spatial data processing tasks very quickly, and is well-suited for large-scale and complex spatial data processing tasks. You can use Apache Sedona’s distributed and scalable architecture to process spatial data sets quickly and efficiently, and to extract valuable insights and information from spatial data in a timely manner.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b4139f142c97" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GeoSpark stands out for processing geospatial data at Scale]]></title>
            <link>https://medium.com/data-science/geospark-stands-out-for-processing-geospatial-data-at-scale-548077270ec0?source=rss-77dc6e0006e4------2</link>
            <guid isPermaLink="false">https://medium.com/p/548077270ec0</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[spatial-analysis]]></category>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[gis]]></category>
            <dc:creator><![CDATA[Mo Sarwat]]></dc:creator>
            <pubDate>Tue, 12 May 2020 21:01:17 GMT</pubDate>
            <atom:updated>2022-12-09T05:56:40.002Z</atom:updated>
            <content:encoded><![CDATA[<h3>Apache Sedona stands out for processing geospatial data at Scale</h3><h4>The spatial data deluge</h4><p>In the past decade, the volume of available geospatial data increased tremendously. Such data includes but not limited to: weather maps, socio-economic data, and geo-tagged social media. For example, spacecrafts from NASA keep monitoring the status of the earth, including land temperature, atmosphere humidity. As of today, NASA has released over 22PB satellite data. Today, we have close 5 billion mobile devices all around the world. In consequence, Mobile Apps generate tons of gesoaptial data. For instance, Lyft, Uber, and Mobike collect terabytes of GPS data from millions of riders every day. In fact, everything we do on our mobile devices leaves digital traces on the surface of the Earth. Moreover, the unprecedented popularity of GPS-equipped mobile devices and Internet of Things (IoT) sensors has led to continuously generating large-scale location information combined with the status of surrounding environments. For example, several cities have started installing sensors across the road intersections to monitor the environment, traffic and air quality.</p><p>Making sense of the rich geospatial properties hidden in the data may greatly transform our society. This includes many subjects undergoing intense study, such as climate change analysis, study of deforestation, population migration, analyzing pandemic spread, urban planning, transportation, commerce and advertisement. These data-intensive geospatial analytics applications highly rely on the underlying data management systems (DBMSs) to efficiently retrieve, process, wrangle and manage data.</p><h3>Apache Sedona (Formerly GeoSpark) Overview</h3><p>Apache Sedona (Formerly GeoSpark) (http://sedona.apache.org) is a cluster computing framework that can process geospatial data at scale. GeoSpark extends the Resilient Distributed Dataset (RDD), the core data structure in Apache Spark, to accommodate big geospatial data in a cluster. A SpatialRDD consists of data partitions that are distributed across the Spark cluster. A Spatial RDD can be created by RDD transformation or be loaded from a file that is stored on permanent storage. This layer provides a number of APIs which allow users to read heterogeneous spatial object from various data formats.</p><p>GeoSpark allows users to issue queries using the out-of-box Spatial SQL API and RDD API. The RDD API provides a set of interfaces written in operational programming languages including Scala, Java, Python and R. The Spatial SQL interfaces offers a declarative language interface to the users so they can enjoy more flexibility when creating their own applications. These SQL API implements the SQL/MM Part 3 standard which is widely used in many existing spatial databases such as PostGIS (on top of PostgreSQL). Next, we show how to use GeoSpark.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*tdyvghDo3tZTO_QBglPiBw.png" /></figure><h3>Supported spatial data sources in Apache Sedona</h3><p>In the past, researchers and practitioners have developed a number of geospatial data formats for different purposes. However, the heterogeneous sources make it extremely difficult to integrate geospatial data together. For example, WKT format is a widely used spatial data format that stores data in a human readable tab-separated-value file. Shapefile is a spatial database file which includes several sub-files such as index file, and non-spatial attribute file. In addition, geospatial data usually possess different shapes such as points, polygons and trajectories.</p><p>Currently, Sedona (GeoSpark) can read WKT, WKB, GeoJSON, Shapefile, and NetCDF / HDF format data from different external storage systems such as local disk, Amazon S3 and Hadoop Distributed File System (HDFS) to Spatial RDDs. Spatial RDDs now can accommodate seven types of spatial data including Point, Multi-Point, Polygon, Multi-Polygon, Line String, Multi-Line String, GeometryCollection, and Circle. Moreover, spatial objects that have different shapes can co-exist in the same Spatial RDD because Sedona adopts a flexible design which generalizes the geometrical computation interfaces of different spatial objects.</p><p><strong>Spatial RDD built-in geometrical library: </strong>It is quite common that spatial data scientists need to exploit some geometrical attributes of spatial objects in Apache Sedona, such as perimeter, area and intersection. Spatial RDD equips a built-in geometrical library to perform geometrical operations at scale so the users will not be involved into sophisticated computational geometry problems. Currently, the system provides over 20 different functions in this library and put them in two separate categories</p><p>Regular geometry functions are applied to every single spatial object in a Spatial RDD. For every object, it generates a corresponding result such as perimeter or area. The output must be either a regular RDD or Spatial RDD.</p><p>Geometry aggregation functions are applied to a Spatial RDD for producing an aggregate value. It only generates a single value or spatial object for the entire Spatial RDD. For example, the system can compute the bounding box or polygonal union of the entire Spatial RDD.</p><h3>Run queries using RDD API</h3><p>Here, we outline the steps to create Spatial RDDs and run spatial queries using GeoSpark RDD APIs. The example code is written in Scala but also works for Java.</p><p><strong>Setup Dependencies:</strong> Before starting to use Apache Sedona (i.e., GeoSpark), users must add the corresponding package to their projects as a dependency. For the ease of managing dependencies, the binary packages of GeoSpark are hosted on the Maven Central Repository which includes all JVM based packages from the entire world. As long as the projects are managed by popular project management tools such as Apache Maven and sbt, users can easily add Apache Sedona by adding the artifact id in the project specification file such as POM.xml and build.sbt.</p><p><strong>Initialize Spark Context:</strong> Any RDD in Spark or Apache Sedona must be created by SparkContext. Therefore, the first task in a GeoSpark application is to initiate a SparkContext. The code snippet below gives an example. In order to use custom spatial object and index serializer, users must enable them in the SparkContext.</p><pre>val conf = new SparkConf()</pre><pre>conf.setAppName(“GeoSparkExample”)</pre><pre>// Enable GeoSpark custom Kryo serializer<br>conf.set(“spark.serializer”, classOf[KryoSerializer].getName)</pre><pre>conf.set(“spark.kryo.registrator”, classOf[GeoSparkKryoRegistrator].getName)</pre><pre>val sc = new SparkContext(conf)</pre><p><strong>Create a Spatial RDD: </strong>Spatial objects in a SpatialRDD is not typed to a certain geometry type and open to more scenarios. It allows an input data file which contains mixed types of geometries. For instance, a WKT file might include three types of spatial objects, such as LineString, Polygon and MultiPolygon. Currently, the system can load data in many different data formats. This is done by a set of file readers such as WktReader and GeoJsonReader. For example, users can call ShapefileReader to read ESRI Shapefiles.</p><pre>val spatialRDD = ShapefileReader.readToGeometryRDD(sc, filePath)</pre><p><strong>Transform the coordinate reference system:</strong> Apache Sedona doesn’t control the coordinate unit (i.e., degree-based or meter-based) of objects in a Spatial RDD. When calculating the distance between two coordinates, GeoSpark simply computes the euclidean distance. In practice, if users want to obtain the accurate geospatial distance, they need to transform coordinates from the degree-based coordinate reference system (CRS), i.e., WGS84, to a planar coordinate reference system (i.e., EPSG: 3857). GeoSpark provides this function to the users such that they can perform this transformation to every object in a Spatial RDD and scale out the workload using a cluster.</p><pre>// epsg:4326: is WGS84, the most common degree-based CRS</pre><pre>val sourceCrsCode = “epsg:4326&quot;</pre><pre> // epsg:3857: The most common meter-based CRS</pre><pre>val targetCrsCode = “epsg:3857&quot;</pre><pre>objectRDD.CRSTransform(sourceCrsCode, targetCrsCode)</pre><p><strong>Build a spatial index:</strong> Users can call APIs to build a distributed spatial index on the Spatial RDD. Currently, the system provides two types of spatial indexes, Quad-Tree and R-Tree, as the local index on each partition. The code of this step is as follows:</p><pre>spatialRDD.buildIndex(IndexType.QUADTREE, false) // Set to true only if the index will be used join query</pre><p><strong>Write a spatial range query:</strong> A spatial range query returns all spatial objects that lie within a geographical region. For example, a range query may find all parks in the Phoenix metropolitan area or return all restaurants within one mile of the user’s current location. In terms of the format, a spatial range query takes a set of spatial objects and a polygonal query window as input and returns all the spatial objects which lie in the query area. A spatial range query takes as input a range query window and a Spatial RDD and returns all geometries that intersect/are fully covered by the query window. Assume the user has a Spatial RDD. He or she can use the following code to issue a spatial range query on this Spatial RDD. The output format of the spatial range query is another Spatial RDD.</p><pre>val rangeQueryWindow = new Envelope(-90.01, -80.01, 30.01, 40.01)</pre><pre> /*If true, return gemeotries intersect or are fully covered by the window; If false, only return the latter. */</pre><pre>val considerIntersect = false</pre><pre>// If true, it will leverage the distributed spatial index to speed up the query execution</pre><pre>val usingIndex = false</pre><pre>var queryResult = RangeQuery.SpatialRangeQuery(spatialRDD, rangeQueryWindow, considerIntersect, usingIndex)</pre><p><strong>Write a spatial K Nearnest Neighbor query:</strong> takes as input a K, a query point and a Spatial RDD and finds the K geometries in the RDD which are the closest to the query point. If the user has a Spatial RDD, he or she then can perform the query as follows. The output format of the spatial KNN query is a list which contains K spatial objects.</p><pre>val geometryFactory = new GeometryFactory()</pre><pre>val pointObject = geometryFactory.createPoint(new Coordinate(-84.01, 34.01)) // query point</pre><pre>val K = 1000 // K Nearest Neighbors</pre><pre>val usingIndex = false</pre><pre>val result = KNNQuery.SpatialKnnQuery(objectRDD, pointObject, K, usingIndex)</pre><p><strong>Write a spatial join query:</strong> Spatial join queries are queries that combine two datasets or more with a spatial predicate, such as distance and containment relations. There are also some real scenarios in life: tell me all the parks which have lakes and tell me all of the gas stations which have grocery stores within 500 feet. Spatial join query needs two sets of spatial objects as inputs. It finds a subset from the cross product of these two datasets such that every record satisfies the given spatial predicate. In Sedona, a spatial join query takes as input two Spatial RDDs A and B. For each object in A, finds the objects (from B) covered/intersected by it. A and B can be any geometry type and are not necessary to have the same geometry type. Spatial RDD spatial partitioning can significantly speed up the join query. Three spatial partitioning methods are available: KDB-Tree, Quad-Tree and R-Tree. Two Spatial RDDs must be partitioned by the same spatial partitioning grid file. In other words, If the user first partitions Spatial RDD A, then he or she must use the data partitioner of A to partition B. The example code is as follows:</p><pre>// Perform the spatial partitioning</pre><pre>objectRDD.spatialPartitioning(joinQueryPartitioningType)</pre><pre>queryWindowRDD.spatialPartitioning(objectRDD.getPartitioner)</pre><pre>// Build the spatial index</pre><pre>val usingIndex = true</pre><pre>queryWindowRDD.buildIndex(IndexType.QUADTREE, true) // Set to true only if the index will be used join query</pre><pre>val result = JoinQuery.SpatialJoinQueryFlat(objectRDD, queryWindowRDD, usingIndex, considerBoundaryIntersection)</pre><h3>Run spatial queries using SQL APIs</h3><p>Here, we outline the steps to manage spatial data using the Spatial SQL interface of GeoSpark. The SQL interface follows SQL/MM Part3 Spatial SQL Standard. In particular, GeoSpark put the available Spatial SQL functions into three categories: (1) Constructors: create a geometry type column (2) Predicates: evaluate whether a spatial condition is true or false. Predicates are usually used in WHERE clauses, HAVING clauses and so on (3) Geometrical functions: perform a specific geometrical operation on the given inputs. These functions can produce geometries or numerical values such as area or perimeter.</p><p>In order to use the system, users need to add GeoSpark as the dependency of their projects, as mentioned in the previous section.</p><p><strong>Initiate SparkSession:</strong> Any SQL query in Spark or Sedona must be issued by SparkSession, which is the central scheduler of a cluster. To initiate a SparkSession, the user should use the code as follows:</p><pre>var sparkSession = SparkSession.builder()</pre><pre>.appName(“GeoSparkExample”)</pre><pre>// Enable GeoSpark custom Kryo serializer</pre><pre>.config(“spark.serializer”, classOf[KryoSerializer].getName)</pre><pre>.config(“spark.kryo.registrator”, classOf[GeoSparkKryoRegistrator].getName)</pre><pre>.getOrCreate()</pre><p><strong>Register SQL functions: </strong>GeoSpark adds new SQL API functions and optimization strategies to the catalyst optimizer of Spark. In order to enable these functionalities, the users need to explicitly register GeoSpark to the Spark Session using the code as follows.</p><pre>GeoSparkSQLRegistrator.registerAll(sparkSession)</pre><p><strong>Create a geometry type column: </strong>Apache Spark offers a couple of format parsers to load data from disk to a Spark DataFrame (a structured RDD). After obtaining a DataFrame, users who want to run Spatial SQL queries will have to first create a geometry type column on this DataFrame because every attribute must have a type in a relational data system. This can be done via some constructors functions such as ST\_GeomFromWKT. After this step, the users will obtain a Spatial DataFrame. The following example shows the usage of this function.</p><pre>SELECT ST_GeomFromWKT(wkt_text) AS geom_col, name, address<br>FROM input</pre><p><strong>Transform the coordinate reference system: </strong>Similar to the RDD APIs, the Spatial SQL APIs also provide a function, namely ST_Transform, to transform the coordinate reference system of spatial objects. It works as follows:</p><pre>SELECT ST_Transform(geom_col, “epsg:4326&quot;, “epsg:3857&quot;) AS geom_col<br>FROM spatial_data_frame</pre><p><strong>Write a spatial range query: </strong>GeoSpark Spatial SQL APIs have a set of predicates which evaluate whether a spatial condition is true or false. ST\_Contains is a classical function that takes as input two objects A and returns true if A contains B. In a given SQL query, if A is a single spatial object and B is a column, this becomes a spatial range query in GeoSpark (see the code below).</p><pre>SELECT *<br>FROM spatial_data_frame<br>WHERE ST_Contains (ST_Envelope(1.0,10.0,100.0,110.0), geom_col)</pre><p><strong>Write a spatial KNN query: </strong>To perform a spatial KNN query using the SQL APIs, the user needs to first compute the distance between the query point and other spatial objects, rank the distances in an ascending order and take the top K objects. The following code finds the 5 nearest neighbors of Point(1, 1).</p><pre>SELECT name, ST_Distance(ST_Point(1.0, 1.0), geom_col) AS distance<br>FROM spatial_data_frame<br>ORDER BY distance ASC<br>LIMIT 5</pre><p><strong>Write a spatial join query: </strong>A spatial join query in Spatial SQL also uses the aforementioned spatial predicates which evaluate spatial conditions. However, to trigger a join query, the inputs of a spatial predicate must involve at least two geometry type columns which can be from two different DataFrames or the same DataFrame. The following query involves two Spatial DataFrames, one polygon column and one point column. It finds every possible pair of $&lt;$polygon, point$&gt;$ such that the polygon contains the point.</p><pre>SELECT *<br>FROM spatial_data_frame1 df1, spatial_data_frame2 df2<br>WHERE ST_Contains(df1.polygon_col, df2.point_col)</pre><p><strong>Perform geometrical operations:</strong> GeoSpark provides over 15 SQL functions. for geometrical computation. Users can easily call these functions in their Spatial SQL query and GeoSpark will run the query in parallel. For instance, a very simple query to get the area of every spatial object is as follows:</p><pre>SELECT ST_Area(geom_col)<br>FROM spatial_data_frame</pre><p>Aggregate functions for spatial objects are also available in the system. They usually take as input all spatial objects in the DataFrame and yield a single value. For example, the code below computes the union of all polygons in the Data Frame.</p><pre>SELECT ST_Union_Aggr(geom_col)<br>FROM spatial_data_frame</pre><h3>Interact with GeoSpark via Zeppelin notebook</h3><p>Although Spark bundles interactive Scala and SQL shells in every release, these shells are not user-friendly and not possible to do complex analysis and charts. Data scientists tend to run programs and draw charts interactively using a graphic interface. Starting from 1.2.0, GeoSpark (Apache Sedona) provides a Helium plugin tailored for Apache Zeppelin web-based notebook. Users can perform spatial analytics on Zeppelin web notebook and Zeppelin will send the tasks to the underlying Spark cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*qYlvsXeeIExSB-Vku0UeGQ.png" /></figure><p>Users can create a new paragraph on a Zeppelin notebook and write code in Scala, Python or SQL to interact with GeoSpark. Moreover, users can click different options available on the interface and ask GeoSpark to render different charts such as bar, line and pie over the query results. For example, Zeppelin can visualize the result of the following query as a bar chart and show that the number of landmarks in every US county.</p><pre>SELECT C.name, count(*)<br>FROM US_county C, US_landmark L<br>WHERE ST_Contains(C.geom_col, L.geom_col)<br>GROUPBY C.name</pre><p>Another example is to find the area of each US county and visualize it on a bar chart. The corresponding query is as follows. This actually leverages the geometrical functions offered in GeoSpark.</p><pre>SELECT C.name, ST_Area(C.geom_col) AS area<br>FROM US_county C</pre><h3>How Apache Sedona can handle the Spatial Data Deluge</h3><h4>Spatial Data Partitioning</h4><p>Moreover, Spatial RDDs equip distributed spatial indices and distributed spatial partitioning to speed up spatial queries. The adopted data partitioning method is tailored to spatial data processing in a cluster. Data in Spatial RDDs are partitioned according to the spatial data distribution and nearby spatial objects are very likely to be put into the same partition. The effect of spatial partitioning is two-fold: (1) when running spatial queries that target at particular spatial regions, GeoSpark can speed up queries by avoiding the unnecessary computation on partitions that are not spatially close. (2) it can chop a Spatial RDD to a number of data partitions which have similar number of records per partition. This way, the system can ensure the load balance and avoid stragglers when performing computation in the cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*wdriTIbaf1b6EME7O0QbKg.png" /></figure><h4>Spatial Indexes</h4><p>Sedona employs a distributed spatial index to index Spatial RDDs in the cluster. This distributed index consists of two parts (1) global index: is stored on the master machine and generated during the spatial partitioning phase. It indexes the bounding box of partitions in Spatial RDDs. The purpose of having such a global index is to prune partitions that are guaranteed to have no qualified spatial objects. (2) local index: is built on each partition of a Spatial RDD. Since each local index only works on the data in its own partition, it can have a small index size. Given a spatial query, the local indices in the Spatial RDD can speed up queries in parallel.</p><h4>Spatial RDD customized serializer</h4><p>Sedona provides a customized serializer for spatial objects and spatial indexes. The proposed serializer can serialize spatial objects and indices into compressed byte arrays. This serializer is faster than the widely used kryo serializer and has a smaller memory footprint when running complex spatial operations, e.g., spatial join query. When converting spatial objects to a byte array, the serializer follows the encoding and decoding specification of Shapefile.</p><p>The serializer can also serialize and deserialize local spatial indices, such as Quad-Tree and R-Tree. For serialization, it uses the Depth-First Search (DFS) to traverse each tree node following the pre-order strategy (first write current node information then write its children nodes). For de-serialization, it will follow the same strategy used in the serialization phase. The de-serialization is also a recursive procedure. When serialize or de-serialize every tree node, the index serializer will call the spatial object serializer to deal with individual spatial objects.</p><h3>Conclusion</h3><p>In Conclusion, Apache Sedona provides an easy to use interface for data scientists to process geospatial data at scale. Currently, the system supports SQL, Python, R, and Scala as well as so many spatial data formats, e.g., ShapeFiles, ESRI, GeoJSON, NASA formats. Here is a link to the GitHub repository:</p><p><a href="http://sedona.apache.org/">Apache Sedona (incubating)</a></p><p>GeoSpark has a small active community of developers from both industry and academia. You can also try more coding examples here:</p><p><a href="https://github.com/apache/incubator-sedona/">apache/incubator-sedona</a></p><p>If you have more questions please feel free to message me on <a href="https://twitter.com/mosarwat">Twitter</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=548077270ec0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/geospark-stands-out-for-processing-geospatial-data-at-scale-548077270ec0">GeoSpark stands out for processing geospatial data at Scale</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>