<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Norman Barker on Medium]]></title>
        <description><![CDATA[Stories by Norman Barker on Medium]]></description>
        <link>https://medium.com/@normanbarker?source=rss-c9614ecd3c1d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*LOSW-jWgum0F91mmHs6UVw.png</url>
            <title>Stories by Norman Barker on Medium</title>
            <link>https://medium.com/@normanbarker?source=rss-c9614ecd3c1d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 02:02:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@normanbarker/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why TileDB for Geometries]]></title>
            <link>https://medium.com/tiledb/why-tiledb-for-geometries-022ba42219bc?source=rss-c9614ecd3c1d------2</link>
            <guid isPermaLink="false">https://medium.com/p/022ba42219bc</guid>
            <category><![CDATA[geopandas]]></category>
            <category><![CDATA[esri]]></category>
            <category><![CDATA[postgis]]></category>
            <category><![CDATA[tiledb]]></category>
            <category><![CDATA[geospatial-data]]></category>
            <dc:creator><![CDATA[Norman Barker]]></dc:creator>
            <pubDate>Wed, 08 Nov 2023 17:44:15 GMT</pubDate>
            <atom:updated>2023-11-08T17:45:59.471Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*urESCsBdocbwUNrM_lF-vQ.jpeg" /></figure><p>Spatial data is dominant in numerous applications, such as earth observation, location-based services, defense, biomedical and many more. Such data is very diverse and comes in various forms, including two important classes: (1) 2D or 3D points (e.g., LiDAR and SONAR) and (2) arbitrary 2D or 3D polygons (e.g., buildings, cell and nucleus boundaries and areas of interest). Querying such data entails submitting a rectangular or other arbitrary multi-polygon regions with a spatial operator such as contains or intersects with the points or polygons. These geospatial data and queries are commonly referred to as “geometries” that represent geographic data objects in the Geospatial world. Providing an efficient mechanism for handling querying and storing geometries at grand scale (hundreds of millions of billions of points and polygons) is a challenging task.</p><p>The Geospatial domain has well known query methods and storage encodings for geometries so we are focusing on geospatial applications in this blog. However, look out for a blog post on our innovative work with spatially resolved omics data where we combine the domain knowledge from the biomedical and geospatial experts within TileDB Inc.</p><p><em>Today we are announcing full geometries support in TileDB!</em> TileDB is a multi-dimensional array database management system and in this blog I explain why it is a natural, super efficient choice for handling geometries. Databases such as PostgreSQL (and specifically its extension, PostGIS) and Oracle have had spatial capabilities for many years, whereas recently new open formats for storing geometries have come up, such as GeoParquet. TileDB brings a lot more to the table as compared to these other approaches. Here are a few takeaways:</p><ul><li>TileDB integrates with <strong>GDAL</strong> and <strong>MariaDB</strong>, so you can plug it into a huge geospatial open-source ecosystem or use it as a replacement for PostGIS if you prefer SQL.</li><li>TileDB is <strong>4x faster than GeoParquet</strong> when slicing a geometry dataset. GeoParquet requires a spatial partitioning scheme whereas TileDB internally implements spatial indexes (such as R-trees).</li><li>TileDB works on any storage backend, including scalable and inexpensive cloud <strong>object stores</strong> (AWS S3, Google Cloud Storage and Azure Blob Storage are all supported).</li><li>TileDB has a <strong>completely</strong> <strong>serverless</strong>, <strong>massively distributed compute</strong> infrastructure and can handle billions of geometries and tens of thousands of queries per second.</li><li>TileDB is a single, <strong>unified solution</strong> that manages the geospatial data objects along with the raw original data (e.g., images, text files, etc), the ML embedding models, and all the other data modalities in your application (tables, rasters, point clouds, etc).</li><li>You can get a lot of value from TileDB as a geospatial database either from our <a href="https://github.com/TileDB-Inc/TileDB">open-source offerings</a> (MIT License), or our enterprise-grade commercial product (<a href="https://cloud.tiledb.com/auth/signup">TileDB Cloud</a>).</li></ul><p>We worked with Even Rouault, the primary maintainer of GDAL, to implement support for geometries in the GDAL OGR driver which we use to ingest geospatial data into TileDB arrays.</p><p>In the remainder of the article, I will first cover the very basics on Geometries and TileDB. I will then explain how TileDB supports geometries and why it is a very powerful solution. Next, I will outline the competitive advantage of TileDB versus other approaches. Finally, I will conclude with our upcoming work on geometries.</p><p>For those who wish to jump directly on some action, read our <a href="https://tiledb.com/blog/tiledb-101-geometries">TileDB 101: Geometries</a> blog post.</p><p>Enjoy!</p><h3>The Basics</h3><h3>What are Geometries?</h3><p>Geospatial geometries are defined in <a href="https://committee.iso.org/sites/tc211/home/projects/projects---complete-list/iso-19125-1.html">ISO-19125</a> parts 1 and 2 and are primarily points, polygons, curves, and surfaces. The figure below shows some examples.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*s07theCC_P46_FSm" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/0*AB31tg8Qa5iCaP5C" /><figcaption>(b) ST_Intersects query over multiple building geometries. <em>Map data by © </em><a href="http://openstreetmap.org/"><em>OpenStreetMap</em></a><em>, under </em><a href="http://www.openstreetmap.org/copyright"><em>ODbL</em></a><em>. Buildings from </em><a href="https://overturemaps.org/"><em>Overture Maps Foundation</em></a><em>, under </em><a href="http://www.openstreetmap.org/copyright"><em>ODbL</em></a><em>.</em></figcaption></figure><p>Geometries can be applied as queries on 2D or 3D point cloud data, or they can be stored as the underlying data and queried by other geometries. In every case, storage compression and computational performance are very important.</p><p>Software that supports geometries includes <a href="https://gdal.org/">GDAL</a>, Fiona, R-Spatial and the broader open-source ecosystem that integrates with it (e.g., such as <a href="https://geopandas.org/">GeoPandas</a> with <a href="https://pyogrio.readthedocs.io/">pyogrio</a> in Python), as well as <a href="https://postgis.net/">PostGIS</a> where you can use SQL statements to ingest and query your data. <a href="https://geoparquet.org/">GeoParquet</a> is another alternative, which can integrate with other database systems like <a href="https://duckdb.org/">DuckDB</a>.</p><h3>What is TileDB?</h3><p>TileDB is a database system architected around dense and sparse multi-dimensional arrays. It consists of (1) an open-source C++ library called <a href="https://github.com/TileDB-Inc/TileDB">TileDB Embedded</a> and a broad open-source ecosystem of APIs and integrations, and (2) a commercial product called <a href="https://tiledb.com/products/tiledb-cloud/">TileDB Cloud</a>. TileDB Embedded is the core array engine responsible for the open-spec array format, whereas TileDB Cloud is a full-fledged, secure database management system, which comes with a powerful serverless distributed computing platform, a holistic catalog, governance capabilities and interactive analysis tools.</p><p>The TileDB array model serves as the basis for many vertical applications, such as in Life Sciences, Geospatial and Machine Learning. We were fortunate to attract and build geospatial expertise within our team and dedicated resources to build vertical solutions for the Geospatial sector. Geometries is one of our geospatial vertical products.</p><p>The next section explains how we implemented geometries at TileDB, and why it is a great fit for such a use case.</p><h3>Geometries in TileDB</h3><p>TileDB has native support for sparse arrays, which can effectively store any multi-dimensional <em>point</em> in space. But how would TileDB store arbitrary shapes? The trick is to calculate and store the <em>center of the envelope</em> of the 2D or 3D geometry. In addition, for each geometry, we calculate the distances in X, Y and Z from the center of the envelope of that geometry to the envelope boundary. We finally store the <em>maximum</em> of all those distances as part of the array metadata. This value will be used during the query phase. The geometry itself is stored as a TileDB attribute in Well Known Binary format, and the geospatial metadata is maintained in the TileDB array or group.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*R-lSg7XFgYkc3rUY" /></figure><p>During any arbitrary spatial query, we first calculate the minimum bounding box of the query geometry, and then expand it by the maximum distance we calculated during ingestion. Any point in the array included in this expanded bounding box is either a result (i.e., an actual data geometry intersected or included in the query geometry) or not included in the actual query geometry. We then filter the returned results from the expanded bounding box query in a fast post-processing step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rR8emMXrmWftd-tJ" /></figure><p>The cool thing about this implementation is that the centers of the geometry bounds, represented as sparse TileDB array cells, are <em>natively indexed in TileDB with R-trees</em>. That allows for super efficient pruning of unnecessary points that are guaranteed not to contribute the result, leading to superb query performance.</p><p>You can use the TileDB geometries in a variety of ways:</p><ul><li>Vanilla TileDB arrays via a large number of APIs (C, C++, Python, Go, Java, C#, R, …)</li><li>Via GDAL, Fiona and using GeoPandas</li><li>Via MariaDB</li></ul><p>Here are some code examples with GDAL / GeoPandas and MariaDB:</p><pre>import fiona<br>fiona.drvsupport.supported_drivers[&quot;TileDB&quot;] = &quot;rw&quot;<br>fiona.vfs.SCHEMES[&quot;tiledb&quot;] = &quot;tiledb&quot;<br>import geopandas as gpd<br>import matplotlib.pyplot as plt<br>import shapely<br><br>x = 172.63<br>y = -43.53<br>circle = shapely.buffer(shapely.Point(x, y), 0.0025)<br>gdf = gpd.read_file(building_array, engine=&quot;pyogrio&quot;, use_arrow=True, bbox=circle.bounds)<br>for r in range(gdf.shape[0]):<br>    if gdf.geometry[r].intersects(circle):<br>        for g in gdf.geometry[r].geoms:<br>            plt.plot(*g.exterior.xy)<br><br>plt.plot(*circle.exterior.xy)<br>plt.show()</pre><p>The same query with MariaDB and SQL:</p><pre>tiledb.cloud.sql.exec(<br>    query=&quot;&quot;&quot;<br>            SELECT names, ST_GeometryFromWkb(geometry) as geom<br>            FROM `tiledb://norman/overture_buildings`<br>            WHERE ST_Intersects(<br>                ST_GeometryFromWkb(geometry),<br>                ST_Buffer(ST_GeometryFromText(&#39;POINT(172.63 -43.53)&#39;), 0.0025)<br>            );    <br>          &quot;&quot;&quot;<br>    )</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/509/0*OOY30CYtAEyrGs4T" /><figcaption>TileDB ST_INTERSECTS radius query over the Overture Map Foundation Building dataset</figcaption></figure><p>Other benefits with open-source TileDB:</p><ul><li>Efficient object store support</li><li>Versioning and time traveling</li><li>Support for multiple different compressors</li><li>Schema evolution (e.g., for adding classification of geometries)</li></ul><p>Extra benefits in TileDB Cloud:</p><ul><li>A secure way to catalog and share your geometry data</li><li>Public datasets and notebooks</li><li>Easy notebook and dashboard environments for day-to-day analysis</li><li>Serverless, large-scale ingestion and analysis via task graphs</li></ul><p>As a nice demonstration of the power of TileDB Cloud’s computational infrastructure, we worked with the buildings dataset from the <a href="https://overturemaps.org/">Overture Maps Foundation</a> that contains buildings from the entire world. Currently this dataset consists of 120 Parquet files that contain 785,524,851 buildings.</p><p>It was possible to ingest the whole dataset in parallel and <em>completely serverless</em> with the task graph shown in the figure below. The ingestion took <em>20 minutes</em>, and the cost was <em>just a few dollars</em>. If you are interested in learning more about task graphs, I recommend reading the blog post <a href="https://tiledb.com/blog/tiledb-cloud-task-graphs">TileDB Cloud: Task Graphs</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SB30NotS4TPepdxq" /><figcaption>TileDB task graph for ingesting the global Overture building dataset</figcaption></figure><p>After the ingestion, all buildings reside as geometries stored in a singleTileDB array on AWS S3. We use the spatial filter from this <a href="https://medium.com/@dr.jiayu/harnessing-overture-maps-data-apache-sedonas-journey-from-parquet-to-geoparquet-d99f7767a499">article</a> to measure performance of the spatial queries against TileDB and others.</p><p>This array can be read through any one of our APIs or integrations. Querying the array through GDAL or our Python API for all buildings <em>WITHIN </em>the polygon boundary below takes about 12 seconds and returns 9,037 buildings (whereas querying using a specialized alternative like Apache Sedona takes about 48 seconds).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/0*vMuuiDUBI36rpUs5" /><figcaption>Spatial query for buildings within a polygon boundary for the City of Bellevue, Washington</figcaption></figure><p>Yet one more benefit for adopting TileDB:</p><ul><li>TileDB stores more than geometries. Its dense and sparse multi-dimensional arrays allow it to morph and capture <em>any data modality</em>, such as point clouds, rasters and tables — even flat files.</li></ul><h3>TileDB Differentiation</h3><p>TileDB’s competitive advantage for geometries can be summarized as follows:</p><ul><li><strong>Performance:</strong> Geometries are represented as TileDB sparse array cells, which are indexed with R-trees. That yields superb performance.</li><li><strong>Serverless:</strong> All deployment modes supported by TileDB are serverless, even when you need to outgrow your local machine and scale in the cloud. This has a direct impact on the operational cost, which TileDB minimizes.</li><li><strong>Cloud-native:</strong> TileDB is uber optimized for object stores. As such, you can scale your vectors to cloud storage, while enjoying superb performance. Again, this leads to significant cost savings.</li><li><strong>Multiple modalities:</strong> TileDB is not just a spatial database. TileDB envisions to store, manage and analyze all your geospatial, tabular, ML data and files.</li><li><strong>Geospatial domain expertise:</strong> The team has industry experts in managing and processing geospatial data who can provide valuable data modeling and optimization insights.</li></ul><p>In addition to the above, here is a brief qualitative and quantitative comparison versus other solutions you might know. In our experiments we used the building dataset from Overture Maps Foundation and tested with storing on S3 as well as local disk on a 8-core machine and 32 GB of RAM (m5a.2xlarge on AWS EC2) with the polygon boundary query above as the actual spatial filter. Note that the system caches were cleared before each test run and with the exception of PostGIS all datasets used ZSTD compression.</p><p>We include the preliminary benchmarks below and summarize the results in turn for each solution immediately after that.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*jP154wcw11yQ1gPN" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*xr80GO1X-A-tyqXC" /><figcaption>GDAL performance metrics</figcaption></figure><p>Notes:</p><ol><li>bench_ogr_c_api iterates over features with calls to GetNextFeature() in the GDAL API</li><li>bench-ogr_batch can be used to measure the raw performance of the Arrow API within GDAL OGR.</li><li>bench_ogr_batch could not be used with a partitioned GeoParquet dataset.</li><li>GDAL_NUM_THREADS was set to be ALL_CPUS.</li><li>PostGIS required using the local file system and was restarted and the disk cache cleared before each test.</li><li>GeoParquet used the <a href="https://github.com/wherobots/OvertureMaps/">Wherobots</a> GeoHash dataset with ZSTD compression so that the geometries could be read by GDAL 3.7.1. The GDAL client does not use Hive Partitioning.</li><li>The “-spat” input argument to the test utility was the 2-D bounding box of the spatial filter (minx, miny, maxx, maxy) of (-122.235128, 47.574044, -122.181229, 47.657588).</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*_OefveBqYnf-6nr1" /><figcaption>Custom client performance metrics</figcaption></figure><p>Notes:</p><ol><li>PostGeese tested against the S3 public release from <a href="https://github.com/wherobots/OvertureMaps/">Wherobots</a>.</li><li>Apache Sedona was tested against the S3 public release from <a href="https://github.com/wherobots/OvertureMaps/">Wherobots</a></li><li>PostGIS required using the local file system and was restarted before each test.</li><li>GeoPandas+TileDB is using pyogrio and Apache Arrow.</li><li>Server-less MyTile is required to be run in the cloud.</li><li>TileDB-Py results will be as fast as geopandas+tiledb when we add support for geometry types in the library.</li></ol><ul><li><strong>GeoParquet:</strong> GeoParquet is an incubating Open Geospatial Consortium standard that adds interoperable geospatial types (e.g. Point, Line, Polygon) to Parquet. It has early adoption despite not having an internal spatial index. The current approach is to index multiple parquet files using Hive partitioning which reintroduces the data management issue when dealing with data at scale. TileDB enables you to store your geospatial data objects in a single array with a performant spatial index. To be used efficiently GeoParquet requires spatial dataset partitioning, which is not supported by GDAL but is by Apache Sedona and Postgeese, without partitioning a full scan needs to be performed. Many tools cannot use Hive partitioning and in these cases TileDB is over 40x faster than GeoParquet. When using a spatial partition, such as when using PostGeese, TileDB is 4x faster.</li><li><strong>ESRI:</strong> ESRI is a commercial vendor for GIS data, in particular analysis of geospatial data objects either as online maps or using desktop tooling along with cloud pipelines. The knowledge required to operate ESRI products requires specialist hires for your team. We did not purchase a license to perform benchmarks at this stage.</li><li><strong>PostGIS:</strong> The elephant in the room! PostGIS extends the capabilities of the PostgreSQL relational database by adding support for storing, indexing and querying geographic data. TileDB goes further and indexes, versions and stores all of your data as TileDB arrays. TileDB is a “column store” allowing faster processing on the attributes of the data that you care about. It is also possible to use the Foreign Data Wrapper in PostgreSQL to migrate your data to TileDB arrays. When used through GDAL, TileDB is nearly 3x faster when performing a spatial query on the buildings dataset when using the new GDAL Arrow API, even when the data is stored on S3 TileDB outperforms PostGIS with local storage by 1.6x. When PostGIS is queried directly through the PostgreSQL CLI TileDB is still faster (1.4x). Additionally TileDB allows storage of your data in your object storage. Serverless SQL performance with MyTile is fast and does not require setting up PostGIS infrastructure. The OGR C API is the older way of accessing geometries and PostGIS outperforms TileDB, but this access approach is not recommended.</li><li><strong>PostGeese:</strong> This is a spatial extension to DuckDB to enable geospatial processing. In our testing of loading the building outlines GeoParquet datasets into a DuckDB table and testing a spatial intersection TileDB outperforms DuckDB by 4x.</li><li><strong>Apache Sedona:</strong> This is a cluster computing system for processing large-scale spatial data. In our experiments with the building dataset and testing a spatial intersection, TileDB is 4x faster.</li></ul><h3>TileDB Roadmap</h3><p>Currently, the primary method of easily accessing geometries that are stored in TileDB arrays is through GDAL and libraries built on top of it such as Fiona/Shapely or R-Spatial. The spatial queries can also be performed using many of our integrations in C++, Java and Python but require decoding the Well Known Binary (WKB) and interpreting the geospatial metadata. In the near future, we plan to do the following:</p><ol><li>Store geometries directly in the R-tree as bounding boxes to prevent calculating centers of mass and expanding the query by the maximum distances from the centers. That will speed up the query, especially for datasets with skewed geometry sizes.</li><li>Adding additional geometry types to our core TileDB library so that users outside of the GDAL ecosystem will know that the data stored in the blob is WKB.</li><li>Support for geospatial queries directly within our own APIs, the spatial filtering of data will happen in the core TileDB library.</li><li>Multi-dimensional spatial indexing — we will extend full (more than 2.5D) spatial filtering over multiple dimensions, e.g. X, Y and Z as used in collision detection between point clouds and CAD models.</li><li>Spatial-temporal indexing such as those found in predictive and historical R-trees.</li><li>More advanced visualizations with our integration into BabylonJS.</li></ol><p>Supporting geometries in TileDB is exciting and enables users to support simpler workflows for processing their geospatial data. Stay tuned for more updates from the TileDB team!</p><h3>What’s Next?</h3><p>In this article I described the support for geometries in TileDB and how you can now store geographic data objects as TileDB arrays. You can get kickstarted with TileDB geometries by reading the blog <a href="https://tiledb.com/blog/tiledb-101-geometries">TileDB 101: Geometries</a>. You can find a plethora of more information about TileDB on our <a href="https://tiledb.com/">official website</a>.</p><p>We’d love to connect with you and hear what you think! Feel free to <a href="https://tiledb.com/contact">contact us</a>, join our <a href="https://join.slack.com/t/tiledb-community/shared_invite/zt-ndq1ipwl-QcithaWG6j1BImtuQGSpag">Slack community</a>, or let us know on <a href="https://twitter.com/tiledb">Twitter</a> and <a href="https://www.linkedin.com/company/tiledb-inc">LinkedIn</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=022ba42219bc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tiledb/why-tiledb-for-geometries-022ba42219bc">Why TileDB for Geometries</a> was originally published in <a href="https://medium.com/tiledb">tiledb</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Open SAR data and scalable analytics]]></title>
            <link>https://medium.com/tiledb/open-sar-data-and-scalable-analytics-bfd1a5257e9?source=rss-c9614ecd3c1d------2</link>
            <guid isPermaLink="false">https://medium.com/p/bfd1a5257e9</guid>
            <category><![CDATA[sharing]]></category>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[tiledb]]></category>
            <category><![CDATA[sars]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Norman Barker]]></dc:creator>
            <pubDate>Tue, 23 Jun 2020 16:52:48 GMT</pubDate>
            <atom:updated>2020-06-23T16:52:48.589Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*clTYrLu0Ecnzx5vfuFiY2Q.png" /></figure><h4>TileDB joins forces with Capella Space on building SAR developer communities</h4><p>SAR stands for Synthetic Aperture Radar. NASA has been using it since the 1970s to detect sub-meter changes to the Earth’s surface, stunningly through clouds, smoke, fog, and darkness. <a href="https://www.capellaspace.com/">Capella Space</a> is now taking SAR a step further, enabling high-cadence global monitoring for government and commerce.</p><p>The velocity and volume of SAR data collections calls for a new data management approach to be adopted by the geospatial application developers and scientists. Specifically, there is a need for a new multi-dimensional, cloud-native data format and storage engine that can be directly queried spatially and temporally, and easily analyzed using familiar tools.</p><p>This blog post will cover how we at TileDB, Inc. innovate in the space of SAR data management and why we partnered with Capella Space as they grow and expand their open data program and SAR developer community.</p><h3>TLDR?</h3><p>We introduce two innovations: (1) <a href="https://github.com/TileDB-Inc/TileDB">TileDB</a>, a powerful open-source storage engine that can efficiently model SAR data collections as multi-dimensional arrays, and (2) <a href="https://tiledb.com/cloud/">TileDB Cloud</a>, an elastic, serverless platform that enables planet-scale sharing and scalable analysis. In a nutshell, these offer the following benefits:</p><ul><li><strong>Ease of use</strong>: Once the data is in the TileDB format, it is analysis-ready and accessible with multiple programming languages and data science tools.</li><li><strong>Planet-scale sharing</strong>: TileDB Cloud allows you to share data and code within or outside your organization without reingestion or extra copying while auditing all activity via detailed logs.</li><li><strong>Scalability</strong>: You can perform from simple slicing queries to custom user-defined functions and task graphs, scaling easily to thousands of parallel tasks.</li><li><strong>Cost Reduction</strong>: No egress costs to download and host copies of files. No overhead maintaining a cloud infrastructure. No idle compute that leads to cumulative losses. The TileDB storage engine and the TileDB Cloud serverless platform comprise the most affordable solution to SAR data management.</li></ul><p>We have partnered with Capella Space to offer their SAR data on TileDB Cloud and address the scalable analytics needs of the SAR developer community. Join today the <a href="https://www.capellaspace.com/community/">Capella Developer Community</a> and<a href="https://console.tiledb.com/auth/signup"> sign up to TileDB Cloud</a> to access the Capella Space SAR datasets and example notebooks.</p><h3>Why TileDB and TileDB Cloud for SAR data?</h3><p>SAR data files often need to be modeled as time-series datasets with complex values (amplitude and phase) for analysis, such as in change detection functions. TileDB is an open-source storage engine that introduces a cloud-native universal format that is ideal for this type of data. It models the collected data as chunked, compressed, versioned, multi-dimensional arrays and offers easy and efficient access. The TileDB storage engine offers extreme interoperability by being integrated with the broader data science ecosystem and geospatial tooling.</p><p>TileDB Cloud is a serverless platform built on the top of the TileDB storage engine that aims at alleviating the engineering hassles for geospatial developers, such as cluster sizing, deployment and monitoring. TileDB Cloud allows these developers to focus on reproducible and shareable analytics while enjoying extreme scalability. Accessing data and performing geospatial analytics on a platform like TileDB Cloud is all about ease-of-use and performance.</p><p>TileDB and TileDB Cloud collectively offer four main capabilities, described in more detail below.</p><h4>Analysis-ready data, not files</h4><p>A SAR dataset has two main characteristics: (1) it is a collection of numerous files, and (2) it has a very large size. To avoid the potentially enormous data hosting costs, it is preferable to store the data on inexpensive cloud object stores (such as AWS S3). Accessing and analyzing such data at petabyte scale becomes very challenging. While each individual SAR file might be defined perfectly, SAR analysis typically involves accessing numerous files, and potentially selective portions of these files, based on different parameters and metadata. Metadata handling and indexing currently falls solely on the user’s shoulders. Moreover, to access the data, you typically need to download the entire SAR files relevant to the analysis and wrangle the data in some other format that your favorite tool understands. This process is cumbersome and costly.</p><p>TileDB allows you to store the numerous SAR files as a coherent multi-dimensional array. All metadata, indexing, updates, slicing, compression, cloud backend optimizations, etc. are all abstracted by TileDB and you do not need to worry again about files. TileDB arrays can be updated with versioning that allows you to time-travel. Most importantly, the TileDB arrays can be queried directly by your favorite language (C, C++, Python, R, Go, Java) and tools (GDAL, Spark, Dask, even databases like MariaDB and PrestoDB). In other words, your data in TileDB is analysis-ready; there is no need for copying, downloading or wrangling.</p><h4>Easier sharing</h4><p>Sharing your SAR files with your peers is a pain, especially if you wish to impose access policies. This is true even on cloud services like AWS. The main reason is that the cloud services understand file semantics, whereas what you need is spatiotemporal semantics (e.g., I want to share this geospatial region in this time interval). The information you wish to share may be located in numerous files, and numerous potentially non-contiguous file byte regions. It is simply impractical to reason around file-based access policies.</p><p>This means that you need to build your own scalable infrastructure to handle the metadata, your customers/collaborators, the efficient querying, auditing, etc. This is extremely expensive in terms of resources for your organization, and also wasteful in terms of human hours.</p><p>TileDB allows you to store your data in arrays (instead of files), and TileDB Cloud provides all the cloud infrastructure you need to share those arrays at planet-scale. This includes scalable slicing, imposing all access policies, and auditing all access, all at the array level (a natively spatiotemporal object) instead of the file level. The way it works is that you store your data as TileDB arrays on your AWS S3 bucket, and you “register” your arrays with TileDB Cloud. Then everything is handled by TileDB Cloud without copying or moving the data. You practically give us the means to govern the access to your data; you continue to own your data and you can access it anytime outside of TileDB Cloud with the open-source TileDB engine. TileDB Cloud allows you to define the access policies with any other TileDB Cloud user, and will dictate all access after that point, providing you with detailed logs. TileDB Cloud charges in a pay-as-you-go manner for data access, not data hosting. Finally, you can always export subsets of TileDB arrays to COG files for your traditional workflows.</p><h4>Serverless data management</h4><p>In addition to easy planet-scale sharing, we strive to help the SAR community make fast discoveries and alleviate all pains related to deploying scalable analysis. With TileDB Cloud, you can perform your geospatial analysis on the SAR data you have access to at extreme scale, via user-defined tasks and workflow graphs, which are handled by TileDB Cloud in a serverless, pay-as-you-go manner. This means that you do not need to deploy a cluster, decide on its size, monitor it, etc. You just define the tasks and TileDB Cloud will execute them in parallel by elastically scaling its compute. TileDB Cloud charges in a pay-as-you-go fashion, eliminating costs from idle compute. Analyzing SAR data has never been simpler than what TileDB Cloud offers.</p><h4>Notebooks for geospatial developers</h4><p>The most recent release of TileDB Cloud offers hosted Jupyter Notebooks, which enable developers to query public and permissioned datasets without having to spin up their own separate compute environment. This effectively allows you to sign up, sign in and code within seconds. Also the Jupyter notebooks are easily shareable and, therefore, it is extremely easy to reproduce a scientific analysis with somebody else’s code, on anyone’s data you have permission to access.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*OXEEenETr1m2inB5pE_QRQ.jpeg" /><figcaption><em>Capella’s open SAR imagery within TileDB Cloud with array-based data access in a Jupyter Notebook</em></figcaption></figure><h3>Partnering with Capella Space: an industry pace setter</h3><p><em>Proving out the value of TileDB Cloud for the SAR community</em></p><p>For the past year, TileDB has been working closely with Capella Space, a US space company founded by Payam Banazadeh that delivers 24-hour all-weather Earth observation imagery of anywhere on the globe. Capella is developing space-based radar earth observation satellites equipped with synthetic aperture radar and delivering innovative SAR imagery data products to the market. Frequent revisit and rapid delivery do not in themselves make a significant impact, unless the SAR imagery is high quality and easily accessible via self-serve online access. Towards this end, Capella is pioneering the next generation of an open developer program and we are excited that TileDB Cloud is a key capability to help <a href="https://www.capellaspace.com/capellas-intuitive-self-serve-online-platform/">scale self-service data access and analytics</a> to thousands of geospatial developers.</p><p>According to Scott Soenen, VP Product Engineering, “The partnership with TileDB gelled perfectly with our desire to deliver a new level of innovation in open data programs aimed at the geospatial community. Open data alone isn’t enough — it’s also about easy access to compute resources and versatility of analytics. TileDB Cloud removes multiple manual steps in data access for the geospatial developer community and offers intuitive self-service and interactive analytics. It enables users of Capella SAR data to dive straight into deriving insights from the data at scale.”</p><h3>Conclusion</h3><p>These are exciting times in the SAR community. We are happy to be able to contribute with TileDB and TileDB Cloud, and we are thankful for Capella’s collaboration and leadership. We are eager to hear your feedback and ideas, you can reach out to us by email at hello@tiledb.com. The best way to understand how to work with SAR data on TileDB Cloud is to <a href="https://tiledb.com/cloud">sign up </a>and explore the public datasets and notebooks (we offer $10 credit upon signing up). Also we encourage you to register to the Capella Space <a href="https://www.capellaspace.com/community/">developer community</a> and ask for access to their data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bfd1a5257e9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/tiledb/open-sar-data-and-scalable-analytics-bfd1a5257e9">Open SAR data and scalable analytics</a> was originally published in <a href="https://medium.com/tiledb">tiledb</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>