<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Anton Biatov on Medium]]></title>
        <description><![CDATA[Stories by Anton Biatov on Medium]]></description>
        <link>https://medium.com/@anton.biatov?source=rss-9beb986612a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*iEL85PadiyMotj5rZ5NrTA.jpeg</url>
            <title>Stories by Anton Biatov on Medium</title>
            <link>https://medium.com/@anton.biatov?source=rss-9beb986612a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 23:33:09 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@anton.biatov/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to Create a Biodiversity Data Portal That Works on Any Hosting Platform]]></title>
            <link>https://medium.com/@anton.biatov/how-to-create-a-biodiversity-data-portal-that-works-on-any-hosting-platform-29154803d178?source=rss-9beb986612a------2</link>
            <guid isPermaLink="false">https://medium.com/p/29154803d178</guid>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[gis]]></category>
            <category><![CDATA[biodiversity]]></category>
            <category><![CDATA[cloud-native-geospatial]]></category>
            <category><![CDATA[gbif]]></category>
            <dc:creator><![CDATA[Anton Biatov]]></dc:creator>
            <pubDate>Mon, 10 Aug 2026 22:38:32 GMT</pubDate>
            <atom:updated>2026-09-24T15:45:34.840Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KlIHtSSA-OvWNHpem-DZjw.jpeg" /></figure><p>In this article, we will look at how to create a thematic geoportal using the example of “Birds of the Fruska Gora National Park”, based on a Darwin Core Archive downloaded from GBIF.</p><p>Any web hosting platform is suitable for running such a Biodiversity Data Portal. This method uses the Serverless GIS paradigm. There is no need to create any databases or run special software on a server or in Docker. You can quickly launch a thematic geoportal without PostGIS, GeoServer, or other complex software.</p><p>We will follow this plan:</p><ol><li>Download the required data from GBIF.org</li><li>Convert a Darwin Core Archive into a minimalist web application with default settings.</li><li>Configure the web application using a settings file (change the title and description, hide some filters, data links, and other interface elements).</li><li>Add new features to the application with AI.</li><li>Upload the finished application to hosting.</li></ol><h3><strong>Getting Data from GBIF</strong></h3><p>Go to the <a href="https://www.gbif.org/">gbif.org</a> portal. On the home page, you will see a search bar where you can search for the required taxon. I entered the taxon “<em>aves</em>” because I have decided to create the “Birds of the Fruška Gora National Park” geoportal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h8loDqXwZT6908l2K86VRQ.jpeg" /></figure><p>After clicking the “Search” button, you will be taken to <a href="https://www.gbif.org/search?q=aves">https://www.gbif.org/search?q=aves</a> . On this page, click the “<strong>Occurrences</strong>” button on the right side of the screen, because we need records of specific observations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QG4MsAye9F06VdhORO8MiA.jpeg" /></figure><p>You will be redirected to <a href="https://www.gbif.org/occurrence/search?taxonKey=V2">https://www.gbif.org/occurrence/search?taxonKey=V2</a> . This page contains all bird records available in the GBIF database.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GygYQLmUnRLq0gE_TvxSSA.jpeg" /></figure><p>Next, we need to configure filters that will leave only the data we need.</p><p>First, select “<strong>Occurrence status</strong>”: “<strong>Present</strong>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mWPXAcq9kkwaWeBw2Z2adQ.jpeg" /></figure><p>Next, select only the points that have coordinates. To do this, apply the filter “<strong>Location</strong>” → “<strong>Has coordinate</strong>” → “<strong>Yes</strong>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2pud-Z4jzNTe0D3GlZuz8Q.jpeg" /></figure><p>You can then filter the remaining points by country, region, extent, a polygon drawn on the map, or upload your own area of interest in GeoJSON or WKT Geometry format.</p><p>I created a 500-metre buffer around the Fruska Gora National Park polygon and converted it to GeoJSON in the WGS84 coordinate system (EPSG:4326).</p><p>I opened the resulting GeoJSON file in a text editor and copied its contents to the clipboard.</p><p>Next, on the Occurrences page, open the “<strong>Location</strong>” filter and, in the pop-up window, switch to the “<strong>Geometry</strong>” tab.</p><p><strong>Paste the text copied from the GeoJSON</strong> file into the text field and click “<strong>Add</strong>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1y2wrirhqZJSPkWIap_BBQ.jpeg" /></figure><p>If the polygon geometry contains too many vertices, you will see the “<em>Geometry too large</em>” warning.</p><p>You can simplify the geometry automatically right here. To do this, click “<strong>Simplify</strong>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UXqCkt0MfUU6jM2msJqloQ.jpeg" /></figure><p>After the geometry has been simplified, the result will be displayed in WKT format. Click “<strong>Apply</strong>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EGXr34X_g9QnZ3A3YOBlOw.jpeg" /></figure><p>You can then add the “<strong>Basis of record</strong>” filter. Click “<strong>More</strong>” → “<strong>Basis of record</strong>” → “<strong>Human observation</strong>” and “<strong>Machine observation</strong>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f1YhcS_pU-cHUWZmrbxvyQ.jpeg" /></figure><p>On the “Map” tab, you can preview the filtering polygon and the filtered data on the web map. The filtered points are grouped into several clusters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bqVpXkjRv8gdymQSHCzZDA.jpeg" /></figure><p>Then go to the “<strong>Download</strong>” tab. Here you need to select the export format: “<strong>Darwin Core Archive</strong>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WTxu3O33r7ptNTSpNA1Lvg.jpeg" /></figure><p>At the next step, go straight to the “<strong>Continue to Terms</strong>” button. You do not need to select any Extensions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zxJOWx9CHt-pfRGuKMUmwQ.jpeg" /></figure><p>At the next step, tick the “<strong>Data Use Agreement</strong>” and “<strong>Data Citation</strong>” checkboxes, then click “<strong>Create download</strong>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XZocy7Db9smYXvfd59nU2Q.jpeg" /></figure><p>After this, you will be redirected to the download page. You will need to wait some time while the data is prepared. Once the Darwin Core Archive with the filtered data is ready, you will receive an email notification. The download page will also display a dataset summary and citation information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QvUU38AyG6mAMyazKkdgOQ.jpeg" /></figure><p>In this example, I use a dataset containing 4,854 bird occurrence points in the area adjacent to Fruška Gora National Park.</p><p><em>GBIF.org (5 August 2026) GBIF Occurrence Download </em><a href="https://doi.org/10.15468/dl.zqcmw5"><em>https://doi.org/10.15468/dl.zqcmw5</em></a></p><p>Next, download the archive and work with it locally.</p><h3><strong>Converting a Darwin Core Archive into a Serverless Web Application</strong></h3><p>For the conversion, use the <a href="https://github.com/ABiatov/dwca-cloud-geospatial"><strong>https://github.com/ABiatov/dwca-cloud-geospatial</strong></a> converter.</p><p>This converter can be used either as an application installed on your computer or online in a Google Colab notebook. The <strong>dwca-cloud-geospatial</strong> converter has a GUI (Graphical User Interface), CLI (Command Line Interface), and Python API. I described the different ways of using this tool in more detail in the article &quot;<a href="https://medium.com/@anton.biatov/how-to-convert-and-stream-biodiversity-data-with-flatgeobuf-geoparquet-and-zero-infrastructure-de303715c160">How to Convert and Stream Biodiversity Data with FlatGeobuf, GeoParquet, and Zero Infrastructure</a>&quot;.</p><p><em>In this article, I will show only the GUI option, because the goal of this article is to customise the viewer produced by the conversion, without going into the details of the conversion itself.</em></p><p>To install this converter, open a terminal and run the following commands step by step:</p><pre>git clone https://github.com/ABiatov/dwca-cloud-geospatial.git<br>cd dwca-cloud-geospatial<br>export REPO=&quot;$(pwd)&quot;<br>python -m venv &quot;${REPO}/.venv&quot;<br>source &quot;${REPO}/.venv/bin/activate&quot;<br>python -m pip install --upgrade pip<br>python -m pip install -e &quot;${REPO}[dev,flatgeobuf,validation]&quot;</pre><p>If you installed the converter earlier, after opening the terminal you only need to go to the installation folder and activate the environment so that the system can find the converter commands.</p><pre>cd dwca-cloud-geospatial<br>export REPO=&quot;$(pwd)&quot;<br>python -m venv &quot;${REPO}/.venv&quot;<br>source &quot;${REPO}/.venv/bin/activate&quot;</pre><p>Then, to launch the graphical interface, run this command in the terminal:</p><pre>dwca-cloud-geospatial-gui</pre><p>The converter has a very simple interface.</p><p>The minimum parameters required to create a portal are:</p><ul><li>input: the path to the ZIP file of the source Darwin Core Archive</li><li>output: the path to the folder where the conversion results will be exported</li><li>FlatGeobuf (checkbox): indicates that we need to save the result in FlatGeobuf format in order to display the data on the map</li><li>GBIF DOI citation lookup (checkbox): additionally retrieves the DOI citation for the specific archive from the GBIF server</li><li>Overwrite existing output (checkbox): I recommend this to avoid problems when writing the result</li></ul><p>If you also need GeoParquet to provide access to the data or use the data programmatically, tick “GeoParquet” and “GeoParquet large-output mode (bbox+grid sort)”.</p><p>To start the conversion, click “Convert”.</p><p>The conversion time depends on the volume of data.</p><p>As a result, data files, metadata files, and viewer files will be created in the folder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/1*kbX21uy_5tww59jL3mFpbw.jpeg" /></figure><p>The conversion creates a web application that can be run in a web browser on <em>localhost</em> or uploaded to web hosting.</p><p>For the sake of simplicity, to run it on <em>localhost</em> and quickly view the customisation results, I use the <strong>Live Server</strong> plugin for <strong>VS Code</strong>. You can do without it by following the instructions provided by the converter.</p><p>The web application has a default title and a default application description. The left panel also has four tabs: “Dataset info,” “Filters,” “List of records,” and “Download”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O1pSjaBVfTYtSZI0Yr0r7A.jpeg" /></figure><h3><strong>Configuring the Web Application</strong></h3><p>The application is configured in the <strong>manifest.json</strong> file.</p><p>Open <strong>manifest.json</strong> in a text editor. I recommend using <strong>VS Code</strong> or any other editor with syntax highlighting.</p><h4><strong>Changing the Application Title</strong></h4><p>Find the <strong>”map_title”</strong> key in the file. Replace the default application title with your own title. In my case, I entered “Birds of the Fruska Gora National Park” instead of “Custom map title, edit it in manifest.json”.</p><pre>&quot;map_title&quot;: &quot;Birds of the Fruška Gora National Park&quot;,</pre><p>Save the changes and then reload the application page in the browser.</p><h4><strong>Customising “App Description”</strong></h4><p>If you click the “App Description” button, a pop-up will open where you can place additional information about the application.</p><p>App Description supports a limited set of HTML tags: `<em>p, b, i, h2, h3, h4, a, img, br, ol, ul, li, table, tr, td, iframe, center, small</em>`.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AZ0qrSGFhQAsjbR0G9U5qw.jpeg" /><figcaption>Default App Description</figcaption></figure><p>To change “App Description,” find the <strong>&quot;appDescription&quot;</strong> key in <strong>manifest.json</strong>.</p><p>Replace its contents with your custom HTML text. Note that the content must be written on a single line, and quotation marks inside the string must be escaped with a backslash.</p><p>Here is an example of my custom “App Description”:</p><pre>&quot;appDescription&quot;: &quot;&lt;center&gt;&lt;h2&gt;Demonstration project \&quot;Birds of the Fruska Gora National Park.\&quot;&lt;/h2&gt;&lt;/center&gt;&lt;/p&gt;This portal was created using the dwca-cloud-geospatial tool and data from &lt;a href=\&quot;https://www.gbif.org\&quot; target=\&quot;_blank\&quot;&gt;GBIF.org&lt;/a&gt;&lt;/p&gt;&lt;p&gt;The demonstration site is &lt;b&gt;Fruska Gora National Park&lt;/b&gt; (&lt;a href=\&quot;https://npfruskagora.co.rs/\&quot; target=\&quot;_blank\&quot;&gt;website&lt;/a&gt;, &lt;a href=\&quot;https://www.facebook.com/fruskagora1960\&quot; target=\&quot;_blank\&quot;&gt;Facebook&lt;/a&gt;, &lt;a href=\&quot;https://www.instagram.com/fruskagoranp/\&quot; target=\&quot;_blank\&quot;&gt;Instagram&lt;/a&gt;).&lt;/p&gt;&lt;center&gt;&lt;img src=\&quot;https://fruskagoranovisajt.rs/wp-content/uploads/2026/01/kiselez.jpg\&quot; alt=\&quot;Fruska Gora National Park Logo\&quot; width=\&quot;400\&quot;&gt;&lt;/center&gt;&lt;p&gt;Fruska Gora National Park is located in the northern part of Serbia, in the province of Vojvodina. It is known for its rich biodiversity, including a variety of bird species, making it an important area for bird conservation and research.&lt;/p&gt;&lt;p&gt;The national park&#39;s polygon was taken from the &lt;a href=\&quot;https://www.protectedplanet.net/country/SRB\&quot; target=\&quot;_blank\&quot;&gt;Protected Planet&lt;/a&gt; portal.&lt;/p&gt;&quot;,</pre><p>Save the changes and reload the page in the browser.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-9kDY8GhqhbguecBE5BryQ.jpeg" /><figcaption>Preview of my custom App Description</figcaption></figure><h4><strong>Hiding Individual Interface Elements, Filters, and Data</strong></h4><p>To hide panels and their contents, <strong>manifest.json</strong> provides a whole set of keys in the “<strong>visibility</strong>” section.</p><p>By changing <strong>”is_visible”: true</strong> to <strong>”is_visible”: false</strong>, you can hide individual elements in the panels and entire panels. You can also disable the opening of a pop-up when a point on the map is clicked.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s8lMHIofH4yiA2xRbY3qZA.jpeg" /><figcaption>Screenshot of the visibility section in the manifest.json file</figcaption></figure><p>Save the changes and reload the page in the browser.</p><p>At this stage, the application is already configured and can be used as a local Biodiversity Data Portal.</p><h3><strong>Extending Application Functionality with AI</strong></h3><p>The resulting application is a convenient starting point for customising and extending the functionality that you need for your local project.</p><p>Today, AI makes it possible to quickly add various features to this application. To do this, you literally only need to write one prompt.</p><p>For example, I want to add the Fruska Gora National Park polygon to the web map as a separate layer.</p><p>First, I copy the national park polygon GeoJSON file, <strong>np-frushka-gora.geojson</strong>, into the <strong>data</strong> folder.</p><p>Then, to edit the application, I use Codex, which I have connected to VS Code using the official “Codex — OpenAI’s coding agent” plugin. You can use any other AI coding agent.</p><p>I open the folder containing the application in <strong>VS Code</strong> and write a prompt for Codex:</p><pre># Add the Fruska Gora National Park polygon to the web map as a separate layer.<br><br>Context:<br>- `index.html`<br>- `app.js`<br><br>Layer to add: `data/np-frushka-gora.geojson`<br>Layer fill opacity 0.2. Layer stroke width 2 and non-transparent.<br>This layer is enabled by default when loading the map.<br>Add a simple menu where I can toggle this layer on and off.<br>The added layer should be located below the layer of points loaded from the FGB file.</pre><p>Codex will modify the application files, add a new layer to the application, and create a checkbox for controlling the layer’s visibility. You can then configure the display of the layer and the layer-control widget.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U3wycBhe5AnCWk6JfbqtrA.jpeg" /></figure><p>In the same way, you can make other changes; for example, you can add satellite imagery to the basemaps.</p><h3><strong>Uploading to Hosting</strong></h3><p>To publish the application on the internet, simply copy the application files and data to hosting. Keep the relative paths between the files intact.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p79FNvFkmOPVhNVAGnIptA.jpeg" /></figure><p>You can use an FTP client or the hosting provider’s web interface for uploading.</p><p>I uploaded the created application to AWS S3 Storage: <a href="https://abspatial.s3.amazonaws.com/birds-fruskagora/index.html">https://abspatial.s3.amazonaws.com/birds-fruskagora/index.html</a></p><h3><strong>Conclusions</strong></h3><p>Darwin Core Archive lets you quickly create a standalone thematic geoportal that doesn’t require a database, PostGIS, GeoServer, or server-side logic. The data is stored in static files, and the map and interface run directly in the browser.</p><p>This approach is well suited for small and medium-sized thematic datasets: national park projects, species monitoring, expeditions, local collections, and demonstration portals. The portal can easily be hosted on standard static hosting, including AWS S3, GitHub Pages, or shared hosting.</p><p>The manifest.json file allows you to customize the finished application without programming, and the coding agent helps you extend it: add thematic layers, switches, new basemaps, and interface elements.</p><p>When publishing, it’s important to preserve the directory structure and relative paths to the data. Large, frequently updated datasets, complex search, authorization, or in-browser editing may require a server-side architecture and a full-fledged geospatial database.</p><p>If you have any suggestions for improving the functionality of the <a href="https://github.com/ABiatov/dwca-cloud-geospatial">dwca-cloud-geospatial</a> converter, please write to me here, on LinkedIn, or on GitHub.</p><p>You can watch the entire process described in this article in the video:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FtZGF7ZwpjZk%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DtZGF7ZwpjZk&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FtZGF7ZwpjZk%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/fd2bbcbb94b4a951a91e71993f843052/href">https://medium.com/media/fd2bbcbb94b4a951a91e71993f843052/href</a></iframe><p>If you find this guide helpful,</p><p>Let’s connect on LinkedIn <a href="https://www.linkedin.com/in/anton-biatov/">https://www.linkedin.com/in/anton-biatov/</a> !</p><p>And follow my blog on Medium <a href="https://medium.com/@anton.biatov">https://medium.com/@anton.biatov</a></p><p>If you need help, message me on <a href="https://www.linkedin.com/in/anton-biatov/">LinkedIn</a>. I am open to collaboration and work opportunities.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=29154803d178" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[7 Common Mistakes in Species Distribution Models or Ecological Niche Models and How to Fix Them]]></title>
            <link>https://medium.com/@anton.biatov/7-common-mistakes-in-species-distribution-models-or-ecological-niche-models-and-how-to-fix-them-315d8654de8c?source=rss-9beb986612a------2</link>
            <guid isPermaLink="false">https://medium.com/p/315d8654de8c</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[scientific-research]]></category>
            <category><![CDATA[spatial-statistics]]></category>
            <category><![CDATA[gis]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Anton Biatov]]></dc:creator>
            <pubDate>Sat, 01 Aug 2026 16:32:44 GMT</pubDate>
            <atom:updated>2026-09-24T15:51:41.247Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0Fm1u69ZXuQdR_lv1r84vA.png" /></figure><p>Easy access to data and AI-generated code can produce convincing maps with flawed science. Learn what to watch for.</p><p>Today, species distribution data (from GBIF, iNaturalist), climate rasters (e.g., WorldClim), or any other spatial variables (terrain, vegetation, humidity, etc.) have become publicly available, and software tools (in particular, the Maxent algorithm) are extremely easy to use. Because of this, researchers often apply them mechanically, as a “black box,” losing sight of the basic principles of spatial statistics and ecology. This problem is especially pressing now, when AI easily generates code that works perfectly from a syntax standpoint, but produces absolutely disastrous results from a scientific standpoint. This post examines seven common pitfalls and their solutions.</p><p>This article is a continuation of the previous post about Species Distribution Modeling (SDM) with Random Forest in R on Colab.</p><h3><strong>1. Confusing “Background” with “Pseudo-Absence”</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*WZxzhJIOB4QXmZ3wrsmR3w.png" /></figure><p><strong>The Mistake:</strong> Algorithms like <strong>Maxent</strong> or <strong>ENFA</strong> (Ecological Niche Factor Analysis) are often mistakenly referred to as presence/pseudo-absence or simply presence-only methods.</p><p><strong>The Core Problem:</strong> Background records (<em>background</em>) are a sample of the <strong>entire spectrum of available conditions</strong> in the study region, including grid cells where the species is present. They are necessary for the model to understand what conditions are generally available in the environment. Pseudo-absences (<em>pseudo-absences</em>) are artificial points of likely species <strong>absence</strong>.</p><p><strong>How to Fix It:</strong> Understand that <em>presence-background</em> methods compare suitable conditions with the average conditions of the region, not with unsuitable ones. The choice of the background area (study area) boundaries affects the model output just as much as the occurrence points themselves: a background area that is too broad or too narrow will radically distort the predictions.</p><h3><strong>2. Confusing Spatial Autocorrelation with Data Clustering</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*4uXwIWhV8hjwHEgytwi5Jw.png" /></figure><p><strong>The Mistake:</strong> When researchers remove closely spaced species occurrence points (a procedure known as <strong>spatial filtering</strong> or <strong>thinning</strong>), they often state that they are “reducing spatial autocorrelation.”</p><p><strong>The Core Problem:</strong> Thinning eliminates <strong>spatial clustering</strong> (clumping of points), which arises from uneven sampling effort (e.g., species are sampled more often near roads, cities, or research institutions). <strong>Spatial autocorrelation</strong> itself (Tobler’s First Law of Geography: “everything is related to everything else, but near things are more related than distant things”) is a fundamental property of nature. It is precisely because of spatial climate autocorrelation that environmental gradients exist. If variables lack autocorrelation, building a niche model is impossible.</p><p><strong>How to Fix It:</strong> Use <em>thinning</em> exclusively to address <strong>sampling bias</strong> and the violation of observation independence, not to combat autocorrelation itself.</p><h3><strong>3. Using Environmental Predictors with Higher Spatial Resolution than Occurrence Data</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*OqlOYta7TkHcbYSAN0lk2g.png" /></figure><p><strong>The Mistake:</strong> Overlaying coarse occurrence data (e.g., old museum records with a coordinate precision of ±5 km or grid-based atlases) onto high-resolution environmental rasters (e.g., climate data at 100 m or 1 km resolution).</p><p><strong>The Core Problem:</strong> The algorithm attempts to link the species’ presence to the microclimate of a specific 100×100 m pixel, whereas the actual observation could have occurred under completely different conditions within a radius of several kilometers.</p><p><strong>How to Fix It:</strong> The spatial resolution of predictors must always match the precision of the initial biological data. If the species coordinates are coarse, the environmental rasters must be coarsened (<em>upscaled</em>) to match the appropriate pixel size.</p><h3><strong>4. Ignoring Multicollinearity Among Environmental Variables</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*iDAFvbyPVRqrgIqW2rmQDA.png" /></figure><p><strong>The Mistake:</strong> Including predictors in the model that are strongly correlated with each other (e.g., simultaneously using mean annual temperature and temperature of the warmest month when they follow the exact same spatial pattern).</p><p><strong>The Core Problem: Multicollinearity</strong> gives excessive weight to redundant factors, leads to model <em>overfitting</em>, distorts estimates of variable importance, and makes it impossible to reliably transfer the model to other regions or future climate scenarios.</p><p><strong>How to Fix It:</strong> Before modeling, perform a correlation analysis (e.g., exclude variables with a Pearson/Spearman coefficient of &quot;<strong>r &gt; 0.7&quot;</strong>) or use the <strong>VIF</strong> (<em>Variance Inflation Factor</em>), retaining only biologically meaningful predictors.</p><h3><strong>5. Lack of Replication in Machine Learning Models</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*_NPfE33rrvV2w6tzSICxgA.png" /></figure><p><strong>The Mistake:</strong> Running a machine learning algorithm (Maxent, Random Forest, Boosted Regression Trees) only once on a single dataset without repetitions.</p><p><strong>The Core Problem:</strong> Machine learning algorithms depend on random data partitioning into training and testing sets, as well as internal stochastic processes. A single run can produce a random or unstable result.</p><p><strong>How to Fix It:</strong> Always use replications — running the model multiple times (e.g., 10–50 replicates) using <em>k-fold cross-validation</em> or <em>bootstrapping</em>. This allows you to average the predictions and evaluate uncertainty (calculate the standard deviation).</p><h3><strong>6. Calculating Topographical Variables in Unprojected Coordinate Systems</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*YcwfSpO4L4OJA49URtMvdg.png" /></figure><p><strong>The Mistake:</strong> Calculating <em>slope</em>, <em>aspect</em>, or distances to rivers directly in geographical coordinates (latitude and longitude in degrees, e.g., in the <strong>WGS84</strong> coordinate reference system).</p><p><strong>The Core Problem:</strong> In geographical coordinates, the linear distance of one degree of longitude decreases from the equator toward the poles. Because of this, grid cells are not square but rectangular (trapezoidal), and the measurement units along the X and Y axes are not equal in meters. Calculating angles and metric distances on such a grid results in severe mathematical errors.</p><p><strong>How to Fix It:</strong> Before calculating any distance-based or morphometric indicators, the Digital Elevation Model (DEM) must be reprojected into a metric equal-area or conformal projection (e.g., <strong>UTM</strong>), where the pixel size is expressed in meters and is identical along both axes.</p><h3><strong>7. Artificial Downscaling of Environmental Variables by Simple Resampling</strong></h3><figure><img alt="Diagram comparing two methods of increasing the apparent resolution of coarse climate data. Topography-informed downscaling incorporates high-resolution elevation data to estimate fine-scale temperature patterns, while bilinear resampling simply interpolates between coarse pixels, creating a smoother but not more informative raster." src="https://cdn-images-1.medium.com/max/880/1*RPCwgT1gUY6Z6WTj8j41nA.png" /></figure><p><strong>The Mistake:</strong> Changing the resolution of a climate raster from coarse to fine (e.g., from 5 km to 1 km) using simple <em>bilinear resampling</em> in GIS packages.</p><p><strong>The Core Problem:</strong> Simple resampling does not add any new physical information about the actual microclimate to the data — it merely smooths the boundaries of coarse pixels, creating an <strong>illusion of high precision</strong> while introducing additional spatial errors.</p><p><strong>How to Fix It:</strong> If high resolution is required, apply statistical or dynamical <em>downscaling</em>, which incorporates additional high-resolution data (e.g., the influence of local topography on temperature via vertical lapse rates), or work at the original, coarser raster resolution.</p><p><strong>The main conclusion: </strong>The technical availability of GIS and machine learning tools does not replace the need to understand the biology of a species and the theoretical foundations of spatial statistics. Every modeling decision (from point filtering to raster projection selection) must be ecologically and mathematically justified, not accepted by default in program settings.</p><p>This post was inspired by the excellent article by Neftalí Sillero: “Sillero, Neftalí, and A. Márcia Barbosa. 2021. “Common Mistakes in Ecological Niche Models.” International Journal of Geographical Information Science 35 (2): 213–26. doi:10.1080/13658816.2020.1798968&quot;</p><p>Read the full article here: <a href="https://doi.org/10.1080/13658816.2020.1798968">https://doi.org/10.1080/13658816.2020.1798968</a></p><p>See also the post about &quot;<a href="https://medium.com/@anton.biatov/species-distribution-modeling-sdm-with-random-forest-in-r-on-colab-bc6f7f1d4a5b"><strong>Species Distribution Modeling (SDM) with Random Forest in R on Colab</strong></a>&quot;.</p><p>If you find this guide helpful, Let’s connect on <a href="https://www.linkedin.com/in/anton-biatov/">LinkedIn</a> and follow my blog on <a href="https://medium.com/@anton.biatov">Medium</a></p><p>If you need help, message me on <a href="https://www.linkedin.com/in/anton-biatov/">LinkedIn</a>. I am open to collaboration and work opportunities.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=315d8654de8c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Species Distribution Modeling (SDM) with Random Forest in R on Colab]]></title>
            <link>https://medium.com/@anton.biatov/species-distribution-modeling-sdm-with-random-forest-in-r-on-colab-bc6f7f1d4a5b?source=rss-9beb986612a------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc6f7f1d4a5b</guid>
            <category><![CDATA[biodiversity]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[google-colab]]></category>
            <category><![CDATA[spatial-analysis]]></category>
            <category><![CDATA[data-modeling]]></category>
            <dc:creator><![CDATA[Anton Biatov]]></dc:creator>
            <pubDate>Mon, 13 Jul 2026 17:56:41 GMT</pubDate>
            <atom:updated>2026-09-24T15:56:32.559Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Em091BkwZIF2EiA-IP9reQ.jpeg" /></figure><p>This article is a step-by-step guide for researchers who are beginning to work with <strong>Species Distribution Modeling (SDM)</strong>.</p><p><strong>Training example:</strong> To make the workflow as clear as possible, we will use the classic dataset from the `dismo` package containing occurrence records for the <strong>brown-throated sloth</strong> (<em>Bradypus variegatus</em>) in South and Central America. We will train the algorithm to associate known occurrence coordinates with climatic factors such as temperature and precipitation, and then predict which additional, unsampled areas may be suitable for the species.</p><p><strong>What this pipeline includes:</strong></p><ul><li><strong>Environment setup:</strong> How to run full R code inside Google Colab’s Python environment using the `<strong>%%R</strong>` magic.</li><li><strong>Data preparation:</strong> Loading sloth occurrence points and extracting climatic predictors at those coordinates.</li><li><strong>Background generation:</strong> Creating pseudo-absence points required to train the algorithm.</li><li><strong>Training and testing:</strong> Splitting the data, configuring Random Forest, and analyzing variable importance.</li><li><strong>Model evaluation:</strong> Testing the model on unseen data, calculating AUC, and plotting the ROC curve.</li><li><strong>Spatial prediction:</strong> Projecting the model across a map and exporting the resulting raster.</li></ul><blockquote><strong>Note:</strong> All code in this example is written in <strong>R</strong>. Special commands at the beginning of cells allow Google Colab to interpret R syntax without requiring RStudio on your computer. Run the cells in order.</blockquote><h3><strong>Step 1: Install the required libraries</strong></h3><pre># Import Colab modules for clearing cell output and downloading files<br>from google.colab import output<br>from google.colab import files<br><br># Colab is a Linux (Ubuntu) virtual machine.<br># Commands beginning with an exclamation mark run directly in the Linux terminal.<br># First, update the system package index; `-qq` suppresses most output.<br>!apt-get -qq update<br><br># This is the MOST IMPORTANT line for geographers and ecologists.<br># R packages such as raster, terra, and rgdal depend on system-level C++ libraries: GDAL, PROJ, and GEOS.<br># Install base R and these spatial libraries.<br>!apt-get -qq install -y r-base libgdal-dev libproj-dev libgeos-dev libudunits2-dev<br><br># rpy2 is the bridge between Python and R.<br># It enables the `%%R` magic used in later cells.<br># Recent Colab versions usually include it, so the installation line is commented out.<br># Uncomment it if an error occurs.<br># !pip install rpy2<br><br># System package installation produces a large amount of technical output.<br># Clear the output to keep the notebook tidy.<br>output.clear()<br>print(&quot;The environment is successfully configured! R and spatial libraries are ready to use.&quot;)</pre><h3><strong>Step 2: Load the rpy2 extension</strong></h3><p>`<strong>%load_ext</strong>` is a special system command, or magic command, used in Jupyter/Colab. It tells the environment to load an additional module. Here we load the `<strong>rpy2.ipython</strong>` extension.</p><p>Why is this necessary?</p><p>This single line creates a bridge between Python and R. After it runs, Google Colab can understand the `<strong>%%R</strong>` command. When `<strong>%%R</strong>` appears on the first line of a cell, Colab treats the rest of that cell as R code rather than Python code and passes it to the built-in R interpreter.</p><pre>%load_ext rpy2.ipython</pre><h3><strong>Step 3: Installing R Packages</strong></h3><p>You can now install and import the required `<strong>R</strong>` packages.</p><p>Base R does not include the required machine-learning and advanced mapping tools.</p><ul><li>`<strong>sp</strong>`, `<strong>terra</strong>`, and `<strong>raster</strong>` provide core spatial vector and raster functionality.</li><li>`<strong>dismo</strong>` is the main package used here for Species Distribution Modeling.</li><li>`<strong>randomForest</strong>` supplies the machine-learning algorithm.</li></ul><p>The `<strong>repos</strong>` argument explicitly selects a download server so Colab does not pause for input.</p><p>This cell may take several minutes to complete and can sometimes take quite a while.</p><pre>%%R<br><br># Record the time before installation begins.<br>start_time &lt;- Sys.time()<br><br># 1. Install packages from CRAN.<br>install.packages(c(&quot;sp&quot;, &quot;terra&quot;, &quot;raster&quot;, &quot;dismo&quot;, &quot;randomForest&quot;), <br>                 repos = &quot;https://cloud.r-project.org&quot;)<br><br># 2. Load the libraries into memory for the current session.<br>library(raster)<br>library(dismo)<br>library(randomForest)<br><br># Record the end time and calculate the elapsed time.<br>end_time &lt;- Sys.time()<br>execution_time &lt;- round(end_time - start_time, 2)<br><br># Print the results.<br>print(&quot;All required R packages have been successfully installed and loaded!&quot;)<br>print(paste(&quot;Installation time:&quot;, execution_time, units(execution_time)))</pre><h3><strong>Step 4: Run the basic Species Distribution Modeling pipeline</strong></h3><p>We will use demonstration data included with the `<strong>dismo</strong>` package.</p><h4><strong>4.1. Load species-presence data — occurrence coordinates</strong></h4><p>At this stage, you can upload your own point data.</p><pre>%%R<br><br># Use the training dataset included with `dismo`.<br># `system.file()` finds the exact path to `bradypus.csv` inside the package.<br># `paste()` combines the path into one character string.<br>file &lt;- paste(system.file(package=&quot;dismo&quot;), &quot;/ex/bradypus.csv&quot;, sep=&quot;&quot;)<br><br># Read the CSV file into a standard data frame.<br># `header=TRUE` means that the first row contains column names;<br># `sep=&#39;,&#39;` means comma-separated columns.<br>bradypus &lt;- read.table(file, header=TRUE, sep=&#39;,&#39;)<br><br># Preview the imported table.<br># 1. species: species name, `Bradypus variegatus`.<br># 2. lon: longitude.<br># 3. lat: latitude.<br>print(head(bradypus))</pre><figure><img alt="Preview the imported table. 1. species: species name, `Bradypus variegatus`. 2. lon: longitude. 3. lat: latitude." src="https://cdn-images-1.medium.com/max/757/1*ZpvlhIElAd5QOSfKmKfU9A.png" /><figcaption>Preview the imported table</figcaption></figure><pre><br>%%R<br># Species names are not needed for the spatial analysis; only X and Y coordinates are required.<br># In R, `[, 2:3]` means all rows and columns two through three.<br>bradypus_coords &lt;- bradypus[, 2:3]<br><br># Print the first six rows to confirm that only coordinates remain.<br>print(head(bradypus_coords))</pre><figure><img alt="first six rows to confirm that only coordinates remain" src="https://cdn-images-1.medium.com/max/824/1*Vlt0ctFmIyDvO1QO2wjUrw.png" /><figcaption>First six rows to confirm that only coordinates remain</figcaption></figure><p>Split presence points into training (75%) and testing (25%) sets.</p><pre>%%R<br># Fix the random seed for reproducibility.<br>set.seed(42)<br><br># Use `kfold` from `dismo` to assign every point randomly to one of four groups.<br># Because there are four groups, each contains about 25% of the points.<br># The resulting `group` object is a vector of group numbers corresponding to table rows (for example: 2, 1, 4, 3, 3, 1, 2...).<br>group &lt;- kfold(bradypus_coords, 4) # Split into four parts.<br>train_points &lt;- bradypus_coords[group != 1, ] # Build the training set (75%).<br>test_points &lt;- bradypus_coords[group == 1, ] # Build the test set (25%).</pre><h4><strong>4.2. Load predictors — environmental and climatic data</strong></h4><p>At this stage, you can load your raster stack.</p><pre>%%R<br># Use the bioclimatic layers included with `dismo`.<br><br># Find the `ex` directory inside the installed package; it contains the South America training maps.<br>path &lt;- file.path(system.file(package=&quot;dismo&quot;), &#39;ex&#39;)<br><br><br># The list.files() function searches this folder for all files ending in &#39;.grd&#39; (pattern=&#39;grd$&#39;).<br># `.grd` is a raster format in which every pixel stores a numeric value.<br># `full.names=TRUE` returns complete file paths.<br>files &lt;- list.files(path, pattern=&#39;grd$&#39;, full.names=TRUE)<br><br># `stack()` combines separate maps, such as precipitation, temperature, and biome layers, into a RasterStack.<br># Important condition: All layers must have the same resolution and geographic extent.<br>predictors &lt;- stack(files)<br><br># Print information about the raster stack.<br>print(predictors)</pre><p>Printout result: information about the raster stack:</p><pre>class      : RasterStack <br>dimensions : 192, 186, 35712, 9  (nrow, ncol, ncell, nlayers)<br>resolution : 0.5, 0.5  (x, y)<br>extent     : -125, -32, -56, 40  (xmin, xmax, ymin, ymax)<br>crs        : +proj=longlat +ellps=WGS84 +towgs84=0,0,0,0,0,0,0 +no_defs <br>names      : bio1, bio12, bio16, bio17, bio5, bio6, bio7, bio8, biome <br>min values :  -23,     0,     0,     0,   61, -212,   60,  -66,     1 <br>max values :  289,  7682,  2458,  1496,  422,  242,  461,  323,    14</pre><p>The `<strong>dismo</strong>` package includes a limited demonstration set of bioclimatic data based on standard <strong>WorldClim</strong> variables:</p><ul><li>`<strong>bio1</strong>` Annual mean temperature</li><li>`<strong>bio5</strong>` Maximum temperature of the warmest month</li><li>`<strong>bio6</strong>` Minimum temperature of the coldest month</li><li>`<strong>bio7</strong>` Annual temperature range `(bio5 — bio6)`</li><li>`<strong>bio8</strong>` Mean temperature of the wettest quarter</li><li>`<strong>bio12</strong>` Total annual precipitation</li><li>`<strong>bio16</strong>` Precipitation of the wettest quarter</li><li>`<strong>bio17</strong>` Precipitation of the driest quarter</li><li>`<strong>biome</strong>` Biomes, represented as a categorical/factor raster</li></ul><p>For a description of all 19 bioclimatic variables and instructions for calculating them from your own temperature and precipitation data with `<strong>biovars()</strong>`, see the <a href="https://rdrr.io/cran/dismo/man/biovars.html">dismo package documentation</a>.</p><p>For real studies, current global climate rasters can be downloaded automatically with the `<strong>worldclim()</strong>` function from the modern `<strong>geodata</strong>` package; see its <a href="https://rdrr.io/cran/geodata/man/worldclim.html">documentation</a>.</p><p>Display the predictor variables in the `<strong>predictors</strong>` raster stack.</p><pre>%%R<br>plot(predictors)</pre><figure><img alt="WorldClim Predictor Variables Map Preview" src="https://cdn-images-1.medium.com/max/480/1*lV5dG0LdPRm_lGxi2Rd3kA.png" /><figcaption>WorldClim Predictor Variables Map Preview</figcaption></figure><h4><strong>4.3. Prepare background / pseudo-absence points</strong></h4><p>Classification algorithms such as Random Forest need examples of both classes: presence (1) and absence (0).</p><p>Field surveys generally record presences only, so artificial absence examples must be generated as background points.</p><pre>%%R<br># `set.seed()` fixes the random-number generator seed.<br># This is essential for reproducible scientific results because the same coordinates are generated on every run.<br>set.seed(0)<br><br># Instead of a hardcoded number, we can generate a number of pseudo-absences<br># exactly equal to the number of our presence points (a 1:1 class balance).<br>n_pres &lt;- nrow(bradypus_coords)<br>print(paste(&quot;Number of presence points:&quot;, n_pres))<br><br># Generate a balanced background:<br># The randomPoints() function from the dismo package scatters random points across the map.<br># 1. The &#39;predictors&#39; argument (our stack of climate maps) acts as a mask.<br>#    This ensures that random points will only land on land (where climate<br>#    data is available) and not in the ocean.<br># 2. &#39;n&#39; is the number of background points we want to generate.<br>bg &lt;- randomPoints(predictors, n = n_pres)<br><br># At this stage, you can set any number for generating pseudo-absence points,<br># but in real-world projects using RF, keep an eye on class balance.</pre><p><strong>⚠️ IMPORTANT METHODOLOGICAL DIGRESSION: Pseudo-absences in Random Forest vs. MaxEnt</strong></p><p>The `<strong>randomPoints()</strong>` command scatters random coordinates across the map. However, depending on the chosen machine learning algorithm, the meaning of these points changes fundamentally:</p><ol><li><strong>What is the problem with Random Forest?<br></strong>Random Forest is a binary classifier. It strictly divides the world into two classes: <strong>1 (presence)</strong> and <strong>0 (absence)</strong>. The algorithm does not understand the concept of a &quot;background environment&quot; and treats our generated random points as <strong>True Absences</strong>.<br>- <strong>Risk of error:</strong> Since points are scattered randomly across the entire study area, there is a high probability that some will fall into ideal environmental conditions for the species or even overlap with actual occurrence coordinates! The model will receive contradictory signals (under identical conditions, the species is simultaneously “present” and “absent”).<br>- <strong>Class Imbalance:</strong> If we have 50 occurrence points and generate 500 pseudo-absences, Random Forest will be heavily biased toward “zero” (absence). The model will tend to predict absence simply because there are 10 times more such examples.</li><li><strong>Why does MaxEnt work differently?<br></strong>The <strong>MaxEnt</strong> (Maximum Entropy) algorithm, on the other hand, is designed specifically for <em>Presence-Only</em> data. It <strong>does not need</strong> absence points. For MaxEnt, random background points are a way to “sample” the available environmental space of the entire study region (Calibration Area). Therefore, for MaxEnt, generating 10,000+ background points is not only normal but necessary.</li></ol><p><strong>💡 Best practices for Random Forest (if no true absences are available):</strong></p><ul><li><strong>1:1 Balancing:</strong> It is recommended to generate a number of pseudo-absences comparable to the number of presence points (a ratio of ~1:1 or 1:2) to minimize class imbalance.</li><li><strong>Ensemble approach (Bootstrap):</strong> In rigorous studies, researchers train dozens of RF models, generating a new random batch of pseudo-absences (equal to the number of occurrence points) each time, and produce the final map by averaging the predictions across all iterations.</li><li><strong>Geographic filtering:</strong> Exclude the generation of background points in the immediate vicinity of known occurrences (for example, by setting a buffer zone around presence points).</li></ul><p><em>For more details on typical mistakes in SDMs, read the article: Sillero, N., &amp; Barbosa, A. M. (2021). Common mistakes in ecological niche models. International Journal of Geographical Information Science, 35(2), 213–226. </em><a href="https://doi.org/10.1080/13658816.2020.1798968"><em>https://doi.org/10.1080/13658816.2020.1798968</em></a></p><h3><strong>Step 5: Modeling with Random Forest</strong></h3><h4><strong>5.1. Prepare the data and train the model</strong></h4><p>Random Forest can sometimes have problems with categorical raster layers. In this example, that layer is `<strong>biome</strong>`, so we remove it for reliability.</p><pre>%%R<br># Prepare the training dataset.<br><br># 1. Remove the categorical `biome` layer to improve Random Forest stability.<br># The R `randomForest` implementation can perform poorly with rasters containing many categorical levels.<br># `dropLayer()` removes this layer.<br>predictors_rf &lt;- dropLayer(predictors, &quot;biome&quot;)<br><br># 2. Extract climatic raster values at the training coordinates.<br># `extract()` retrieves temperature, precipitation, and other values for each point.<br># `data.frame()` converts the result into a table with points as rows and climatic variables as columns.<br>env_train_pres &lt;- data.frame(extract(predictors_rf, train_points)) # Climate at confirmed presence locations.<br>env_train_bg   &lt;- data.frame(extract(predictors_rf, bg)) # Climate at random background locations.<br><br># 3. Combine the tables vertically with `rbind()`.<br># The `rbind()` (row bind) function joins two tables vertically (adds rows to one another).<br>train_data &lt;- rbind(env_train_pres, env_train_bg)<br><br># 4. Create the target response vector.<br># `rep(1, ...)` creates one value for each presence; `rep(0, ...)` does the same for background points.<br># `c(...)` combines them into the presence/absence vector `pa`.<br>pa &lt;- c(rep(1, nrow(env_train_pres)), rep(0, nrow(env_train_bg)))<br><br># IMPORTANT: Convert zeros and ones into categorical factors with `as.factor()`.<br># Otherwise Random Forest will interpret them as continuous values and fit a regression model.<br># A classification model is required here.<br>train_data$presence &lt;- as.factor(pa)<br><br># 5. Remove rows containing missing values with `na.omit()`.<br># This prevents errors if a point falls outside the raster coverage.<br>train_data &lt;- na.omit(train_data)</pre><h4><strong>5.2. Train the model</strong></h4><pre>%%R<br># Fix the random processes again with `set.seed()`.<br># Random Forest builds many random data subsets, so the seed makes the fitted model reproducible.<br>set.seed(42)<br><br># Train the model. Key arguments:<br># 1. Formula `presence ~ .`: predict `presence` from every other column.<br># 2. `data=train_data`: use the prepared presence and background table.<br># 3. `importance=TRUE`: calculate the importance of each climatic predictor. A crucial parameter for ecologists!<br># 4. `ntree=500`: build 500 decision trees, a common balance between accuracy and speed.<br>rf_model &lt;- randomForest(presence ~ ., data=train_data, importance=TRUE, ntree=500)<br><br># Print the model summary.<br># In the output includes the OOB error (Out-of-Bag estimate of error rate) - this is<br># an internal estimate of how often the model makes mistakes, as well as the confusion matrix,<br># which shows how well classes 0 and 1 are separated.<br>print(rf_model)</pre><p>Printed model summary:</p><pre>Call:<br> randomForest(formula = presence ~ ., data = train_data, importance = TRUE,      ntree = 500) <br>               Type of random forest: classification<br>                     Number of trees: 500<br>No. of variables tried at each split: 2<br><br>        OOB estimate of  error rate: 11.58%<br>Confusion matrix:<br>    0  1 class.error<br>0 477 23   0.0460000<br>1  45 42   0.5172414</pre><h4><strong>5.3. Assess predictor importance</strong></h4><p>For ecologists, understanding the biology can be more important than simply producing a map: which factors — heat, cold, or precipitation — most strongly limit the species’ range?</p><p>We obtain variable importance with `<strong>importance()</strong>`. This function extracts importance metrics for every predictor from the trained model.</p><ol><li>`<strong>MeanDecreaseAccuracy</strong>`: Shows how much model accuracy decreases when a climatic factor is removed or disrupted. The HIGHER the value, the MORE IMPORTANT the predictor.</li><li>`<strong>MeanDecreaseGini</strong>`: Shows how effectively a variable separates presence (1) from absence/background (0) within the decision trees. Again, a larger value indicates greater importance for the sloth.</li></ol><pre>%%R<br># This works because `importance=TRUE` was set when the model was trained.<br>rf_importance &lt;- importance(rf_model)<br><br># Print the importance table.<br>print(rf_importance)</pre><p>Printed importance table:</p><pre>              0         1 MeanDecreaseAccuracy MeanDecreaseGini<br>bio1  15.037336  5.957899             17.90901         13.75154<br>bio12  9.565314 16.945758             18.14090         21.85422<br>bio16  8.652195 19.902882             19.00987         22.90766<br>bio17 12.085979 10.654911             17.97669         17.96224<br>bio5  13.556747 15.526494             20.99897         14.54840<br>bio6  14.352640 13.524926             17.94403         17.12503<br>bio7  10.139052 23.230274             21.14614         24.64354<br>bio8  12.671139  7.639760             16.64006         12.42611</pre><p>Visualize variable importance.</p><pre>%%R -w 900 -h 600 -r 100<br># -w 1200 (width) and -h 600 (height) set the resulting image size in pixels.<br># -r 100 (resolution) sets the resolution (DPI).<br># This makes the graphs wide and clear.<br><br># `varImpPlot()` is the standard importance plot for `randomForest` models.<br># It is more informative than a simple percentage bar chart.<br>varImpPlot(rf_model, main=&quot;Predictor Importance (Random Forest)&quot;)</pre><figure><img alt="Predictor Importance (Random Forest)" src="https://cdn-images-1.medium.com/max/900/1*sAUKekIeEkkKHBkFYHoiiw.png" /></figure><h4><strong>5.4. Evaluate model accuracy with AUC — step-by-step</strong></h4><p>The `<strong>evaluate()</strong>` function from `<strong>dismo</strong>` works well with Random Forest. To avoid confusion over data formats, however, we will manually generate predictions at the test points and pass the resulting probabilities to the function.</p><pre>%%R<br># STEP 1: Extract climatic predictor values for the test points.<br># `extract()` samples the rasters at the supplied coordinates.<br># `data.frame()` converts values into a table; `na.omit()` removes points outside raster coverage.<br>env_test_pres &lt;- na.omit(data.frame(extract(predictors_rf, test_points))) # Real species-presence points.<br>env_test_bg &lt;- na.omit(data.frame(extract(predictors_rf, bg)))            # Background or pseudo-absence points.<br><br># STEP 2: Generate probability predictions for the new data.<br># Feed our trained model (rf_model) the new climate tables.<br># Important: `type=&quot;prob&quot;` tells Random Forest to output probabilities, not just a hard class (0 or 1).<br># The predict output is a two-column matrix.<br># Column 1: absence probability, column 2: presence probability.<br># We need the presence probability, so we take the second column using [, 2].<br>rf_pred_pres &lt;- predict(rf_model, env_test_pres, type=&quot;prob&quot;)[, 2]<br>rf_pred_bg &lt;- predict(rf_model, env_test_bg, type=&quot;prob&quot;)[, 2]<br><br># STEP 3: Calculate model-quality metrics.<br># The evaluate() function from the dismo package compares the predicted probabilities<br># at presence points (p) with the probabilities at background points (a).<br># Based on this, statistics are calculated, including the main metric, AUC.<br>rf_eval &lt;- evaluate(p=rf_pred_pres, a=rf_pred_bg)<br><br># Print the results.<br>print(rf_eval)<br>print(paste(&quot;Test AUC (Random Forest):&quot;, rf_eval@auc))<br><br># STEP 4: Visualize the result.<br># Plot the ROC curve. The more the curve bends toward the upper left,<br># the better the model is at distinguishing suitable conditions from unsuitable ones.<br>plot(rf_eval, &#39;ROC&#39;, main=&quot;ROC curve for the test sample&quot;)</pre><figure><img alt="Random Forest model-quality metrics and Plot the ROC curve" src="https://cdn-images-1.medium.com/max/1024/1*0ySjCBVB2v3J96WqZsyUWA.png" /></figure><p><strong>Interpreting the model metrics:</strong></p><ul><li>`<strong>n presences</strong>`: Number of presence points used for evaluation that were not used during training.</li><li>`<strong>n absences</strong>`: Number of background or pseudo-absence points.</li><li>`<strong>AUC</strong>`: Area Under the Curve. This metric summarizes model performance as the area under the ROC curve. AUC ranges from 0 to 1; higher values indicate better performance. An AUC of 0.5 indicates no discriminatory ability, while 1.0 represents perfect class separation.</li><li>`<strong>cor</strong>`: Correlation between model predictions and observed classes, where 1 is presence and 0 is background. For SDMs, values around 0.3–0.5 can be entirely normal.</li><li>`<strong>max TPR+TNR</strong>`: The optimal classification threshold. The model produces continuous suitability values from 0.0 to 1.0, but researchers often need a binary map showing where the species can or cannot occur. This threshold maximizes the balance between sensitivity (TPR, correctly identified presences) and specificity (TNR, correctly identified background points).</li></ul><h4><strong>5.5. Optional: Select a threshold for a binary map</strong></h4><p>The model returns probabilities from 0 to 1. Conservation decisions often require a strict boundary: the species is considered present here (1) and absent there (0). To convert the continuous suitability map into a binary map, we must choose a classification threshold.</p><pre>%%R<br># Calling `threshold()` without specifying a method calculates <br># all available statistical thresholds.<br>all_thresholds &lt;- threshold(rf_eval)<br><br># Print all available threshold criteria.<br>print(&quot;All available criteria for the threshold:&quot;)<br>print(all_thresholds)</pre><figure><img alt="All available criteria for the threshol: kappa, spec_sens, no_omission, prevalence, equal_sens_spec, sensitivity" src="https://cdn-images-1.medium.com/max/1024/1*vD87CgTC6B7HYdKvd3TNoQ.png" /></figure><p><strong>Interpretation of each criterion:</strong></p><ol><li>`<strong>kappa</strong>` — Maximum Cohen’s kappa<br>- Calculation: Finds the threshold that maximizes Cohen’s kappa, which measures how much better the prediction is than chance while accounting for class imbalance.<br>- Use: It was common in the past but is now less frequently used in studies. A high threshold makes the model very strict and retains only the most suitable habitat.</li><li>`<strong>spec_sens</strong>` — Maximum balance, or Max TPR + TNR<br>- Calculation: Finds the point where the sum of sensitivity and specificity is greatest.<br>- Use: This is one of the most popular and broadly applicable approaches in ecology and usually produces a balanced binary map.</li><li>`<strong>no_omission</strong>` — No omissions, or 0% false negatives<br>- Calculation: Selects the highest threshold at which none of the test presence points is excluded. In other words, all known occurrences remain inside the predicted suitable area.<br>- Use: Appropriate when missing suitable habitat is especially costly, such as when planning reserves for an endangered species or predicting a dangerous invasive plant. It is conservative and therefore usually produces a broad range.</li><li>`<strong>prevalence</strong>` — Preserve prevalence<br>- Calculation: Chooses a threshold so that the proportion of suitable map area matches the proportion of presence points relative to background points.<br>- Use: Rarely appropriate for workflows unless true prevalence is known. Because the background points here are generated artificially, this threshold has little biological meaning.</li><li>`<strong>equal_sens_spec</strong>` — Equal sensitivity and specificity<br>- Calculation: Selects the threshold at which the percentage of correctly predicted presences equals the percentage of correctly rejected background points.<br>- Use: An alternative to `<strong>spec_sens</strong>`. It provides strict mathematical parity between the two error types, but may produce a somewhat narrower predicted range.</li><li>`<strong>sensitivity</strong>` — Fixed sensitivity<br>- Calculation: In `<strong>dismo</strong>`, this criterion normally finds the threshold at which 90% of observed presences are predicted correctly, while 10% are treated as possible errors or outliers.<br>- Use: A useful compromise when occurrence data may contain mistakes. Unlike `<strong>no_omission</strong>`, it prevents one erroneous point from greatly inflating the predicted range.</li></ol><h4><strong>5.6. Prediction map</strong></h4><p>The `<strong>raster</strong>` package supports Random Forest models directly. We apply the trained model to the predictor stack with `<strong>predict()</strong>`.</p><p>What happens here:</p><ol><li>`<strong>predictors_rf</strong>` is the stack of climatic rasters. `<strong>predict()</strong>` reads predictor values pixel by pixel and passes them to the model.</li><li>`<strong>rf_model</strong>` is the trained Random Forest model.</li><li>`<strong>type=”prob”</strong>` requests probabilities from 0 to 1 rather than hard presence/absence classes.</li><li>`<strong>index=2</strong>` is CRITICAL. For classification, Random Forest returns two probability columns: the first for class `<strong>0</strong>` (absence) and the second for class `<strong>1</strong>` (presence). The `raster` package would normally use the first output layer. Setting `<strong>index=2</strong>` explicitly stores the probability of species PRESENCE.</li></ol><pre>%%R<br># Create the habitat-suitability raster as a spatial prediction.<br>rf_suitability_map &lt;- predict(predictors_rf, rf_model, type=&quot;prob&quot;, index=2)</pre><p>Create a binary raster using the `max TPR+TNR` threshold.</p><pre>%%R<br># 1. Extract the numeric threshold value by selecting `spec_sens`.<br>rf_threshold &lt;- threshold(rf_eval, &#39;spec_sens&#39;)<br><br># 2. Create a binary raster with raster algebra.<br>rf_binary_map &lt;- rf_suitability_map &gt;= rf_threshold</pre><p><strong>Visualize and compare the results</strong></p><pre>%%R -w 1200 -h 600 -r 100<br># Colab magic settings: `-w` and `-h` set image width and height; <br># `-r` sets resolution in DPI.<br># These settings produce a wide, clear figure containing two maps.<br><br># `par(mfrow=c(1,2))` divides the R graphics device into one row and two columns.<br># The next two plots are drawn side by side.<br>par(mfrow = c(1, 2))<br><br># PLOT 1: Continuous probability map on the left.<br>plot(rf_suitability_map, main=&quot;RF: Probability (Habitat Suitability)&quot;)<br># `points()` overlays occurrence coordinates; `pch=20` selects filled circles <br># and `cex=0.4` sets their size.<br>points(bradypus_coords, pch=20, cex=0.4, col=&quot;black&quot;)<br><br># PLOT 2: Binary range map on the right.<br># A two-color palette is used because values are only 0 and 1.<br># 0 means unsuitable and 1 means suitable.<br># Hide the numeric legend because the classes are self-explanatory.<br>plot(rf_binary_map,<br>     main=&quot;RF: Binary Map (Presence/Absence)&quot;,<br>     col=c(&quot;lightgrey&quot;, &quot;forestgreen&quot;),<br>     legend=FALSE)<br>points(bradypus_coords, pch=20, cex=0.4, col=&quot;black&quot;)<br><br># IMPORTANT: Restore the standard one-plot layout.<br># Otherwise later plots will remain squeezed into one of the two columns.<br>par(mfrow = c(1, 1))</pre><figure><img alt="Visualize and compare the results: Continuous probability map on the left “RF: Probability (Habitat Suitability)”, Binary range map on the right “RF: Binary Map (Presence/Absence)”" src="https://cdn-images-1.medium.com/max/1024/1*V0hD50Jq424q-U3ex6V97w.png" /></figure><h4><strong>5.7. Export the modeling results to GeoTIFF</strong></h4><p><strong>Create GeoTIFF rasters</strong></p><p>The `<strong>writeRaster()</strong>` function from the `<strong>raster</strong>` package saves the maps as files.</p><p>Key arguments:</p><ul><li>`<strong>filename</strong>`: Name of the output file, including the `<strong>.tif</strong>` extension.</li><li>`<strong>format=”GTiff”</strong>`: GeoTIFF is a global GIS standard. It is essentially an image with geographic coordinates embedded for every pixel.</li><li>`<strong>overwrite=TRUE</strong>`: Overwrite the file when the cell is run again, avoiding an error that the file already exists.</li></ul><pre>%%R<br># 1. Export the continuous probability map with values from 0.0 to 1.0.<br>writeRaster(rf_suitability_map,<br>            filename=&quot;rf_probability_map.tif&quot;,<br>            format=&quot;GTiff&quot;,<br>            overwrite=TRUE)<br><br># 2. Export the binary map with values 0 and 1, classified using the `max TPR+TNR` threshold.<br>writeRaster(rf_binary_map,<br>            filename=&quot;rf_binary_map.tif&quot;,<br>            format=&quot;GTiff&quot;,<br>            overwrite=TRUE)<br><br>print(&quot;Random Forest maps successfully saved to GeoTIFF!&quot;)</pre><p><strong>Download the results</strong></p><p>The files are stored in Colab’s temporary storage, in the `/content` directory.</p><p>To download them manually, click the folder icon in the left-hand menu, locate the `.tif` files, click the three dots beside a file, and select <strong>**Download**</strong>.</p><figure><img alt="How to download file from Google Colab" src="https://cdn-images-1.medium.com/max/1024/1*bV4lz9sCCNK_GghU4T5bbg.jpeg" /></figure><p>You can also use `<strong>files.download()</strong>`. It searches for the named file in Colab’s virtual storage and opens the browser download dialog automatically.</p><pre>print(&quot;Preparing to download the probability map...&quot;)<br>files.download(&#39;rf_probability_map.tif&#39;)<br><br>print(&quot;Preparing to download the binary map...&quot;)<br>files.download(&#39;rf_binary_map.tif&#39;)</pre><p>I’ve compiled all the steps listed in the article into a Colab Notebook. Go to it, create a copy, and simply run each cell one by one. You can also edit the notebook copy, insert your own data, and build models using your data.<br><a href="https://colab.research.google.com/drive/1vxFNtyQkvUQ4RQzQeEkAv0puuqWGuI1l?usp=sharing">https://colab.research.google.com/drive/1vxFNtyQkvUQ4RQzQeEkAv0puuqWGuI1l?usp=sharing</a></p><p>If you find this guide helpful, Let’s connect on <a href="https://www.linkedin.com/in/anton-biatov/">LinkedIn</a> and follow my blog on <a href="https://medium.com/@anton.biatov">Medium</a></p><p>If you need help, message me on <a href="https://www.linkedin.com/in/anton-biatov/">LinkedIn</a>. <strong>I am open to collaboration and work opportunities.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc6f7f1d4a5b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Convert and Stream Biodiversity Data with FlatGeobuf, GeoParquet, and Zero Infrastructure]]></title>
            <link>https://medium.com/@anton.biatov/how-to-convert-and-stream-biodiversity-data-with-flatgeobuf-geoparquet-and-zero-infrastructure-de303715c160?source=rss-9beb986612a------2</link>
            <guid isPermaLink="false">https://medium.com/p/de303715c160</guid>
            <category><![CDATA[biodiversity]]></category>
            <category><![CDATA[gis]]></category>
            <category><![CDATA[gbif]]></category>
            <category><![CDATA[geoparquet]]></category>
            <category><![CDATA[cloud-native-geospatial]]></category>
            <dc:creator><![CDATA[Anton Biatov]]></dc:creator>
            <pubDate>Tue, 23 Jun 2026 23:47:40 GMT</pubDate>
            <atom:updated>2026-09-24T15:52:28.995Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OTpo1yYwVFp88VWP_yKEsA.png" /></figure><p>In this article, I’ll guide you through converting biodiversity occurrence data from Darwin Core Archives into cloud-optimized geospatial formats — including FlatGeobuf and GeoParquet — and streaming those datasets directly from static storage. We’ll use the `dwca-cloud-geospatial` Python toolkit to move from raw DwC-A downloads to ready-to-query files and an interactive web map, without deploying PostGIS, GeoServer, or a custom backend. This approach is efficient for geospatial analysis and web mapping, especially when working with large GBIF or OBIS datasets.</p><p>Biodiversity data is gold. Platforms like GBIF (Global Biodiversity Information Facility) and OBIS mediate billions of species occurrence records that are critical for climate models, ecological research, and geospatial analysis.</p><p>But these massive datasets are traditionally shared using a standard called <strong>Darwin Core Archive (DwC-A)</strong>. Developed in the early 2000s, a DwC-A file is essentially a time capsule: a heavy ZIP archive containing giant, flat CSV/TSV text files and restrictive XML metadata schemas.</p><p>If you are a data engineer, GIS analyst, or web developer wanting to build a quick interactive map or perform spatial queries on a fresh GBIF download, you face a daunting infrastructure tax:</p><ol><li>Download and extract a 10GB+ ZIP archive.</li><li>Spin up a PostgreSQL database with the PostGIS extension.</li><li>Write custom ingestion scripts, hoping column data types don’t break mid-way.</li><li>Set up and configure a geospatial server like GeoServer or MapServer to render vector tiles.</li><li>Maintain a dedicated backend API just to filter points.</li></ol><p>You end up spending days managing servers and paying for cloud compute just because you wanted to <em>look at points on a map</em>.</p><h3>The Paradigm Shift: Cloud-Native &amp; Serverless GIS</h3><p>What if you didn’t need a geospatial server at all?</p><p>The modern geospatial ecosystem has fundamentally shifted toward <strong>Cloud-Native and Serverless architectures</strong>. By utilizing smart, indexing file formats hosted on basic static cloud storage (like AWS S3, GitHub Pages, or Cloudflare), client-side applications can fetch exactly what they need using standard HTTP range requests.</p><ul><li><strong>FlatGeobuf:</strong> The ultimate format for web mapping. It includes a built-in spatial index, allowing a browser to stream only the coordinates intersecting the user’s current viewport (BBox).</li><li><strong>GeoParquet:</strong> The king of columnar storage and Big Data analytics. It enables analytical engines like DuckDB to execute lightning-fast SQL queries over millions of remote records without database ingestion.</li></ul><p>To bridge the gap between legacy biodiversity data and modern serverless GIS, I built `<strong>dwca-cloud-geospatial</strong>`. It is an open-source Python tool (equipped with a CLI, GUI, and API) that transforms raw DwC-A archives into optimized, ready-to-share geospatial bundles along with a built-in static viewer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oRNEVC1rMUAaWHzcEiObSw.png" /></figure><h3><strong>Getting Started &amp; Installation</strong></h3><p>The tool is published as a Python package. For local testing, it is highly recommended to use an isolated virtual environment:</p><pre># Clone the repository<br>git clone https://github.com/ABiatov/dwca-cloud-geospatial.git<br># gh repo clone ABiatov/dwca-cloud-geospatial # An alternative way to clone a repository<br>cd dwca-cloud-geospatial<br>export REPO=&quot;$(pwd)&quot;<br><br># Set up local virtual environment<br>python -m venv &quot;${REPO}/.venv&quot;<br>source &quot;${REPO}/.venv/bin/activate&quot;<br>python -m pip install --upgrade pip<br><br># Install with FlatGeobuf and Validation extras<br>python -m pip install -e &quot;${REPO}[dev,flatgeobuf,validation]&quot;</pre><h3><strong>Three Ways to Convert: GUI, CLI, and Python API</strong></h3><p>`<strong>dwca-cloud-geospatial</strong>` calls the exact same underlying pipeline whether you prefer a desktop interface, terminal commands, or pure Python code.</p><h4><strong>1. The Desktop Workflow (GUI)</strong></h4><p>If you prefer a visual interface, run:</p><pre>dwca-cloud-geospatial-gui</pre><p>This boots a clean interface where you can point to a downloaded archive, toggle output options, and initiate conversion.</p><figure><img alt="A simple desktop interface, dwca-cloud-geospatial, allows you to select a Darwin Core Archive (DwC-A), configure metadata enrichment parameters, file export format (FlatGeobuf / GeoParquet), and save the output." src="https://cdn-images-1.medium.com/max/1024/1*CpGCtsJuaE8AHnfCrx36Ag.png" /></figure><h4><strong>2. The Automation Workflow (CLI)</strong></h4><p>For pipeline integration, the command line provides a non-blocking workflow. First, inspect your archive to verify its schema contains the required latitude and longitude definitions:</p><pre>dwca-cloud-geospatial inspect /path/to/gbif_archive.zip</pre><p>To run the conversion, explicitly passing GBIF provenance fields so they are baked directly into the static bundle’s metadata, run:</p><pre>dwca-cloud-geospatial convert \<br>  /path/to/GBIF_DwC_Archive.zip \<br>  ./output_bundle \<br>  --gbif-enrich \<br>  --format flatgeobuf \<br>  --format geoparquet \<br>  --geoparquet-large-output \<br>  --chunk-size 10000 \<br>  --overwrite</pre><h4><strong>3. The Programmatic Workflow (Python API)</strong></h4><p>If you are processing datasets within a larger workflow or inside a notebook, you can import and configure the underlying core library directly:</p><pre>from pathlib import Path<br>from dwca_cloud_geospatial.conversion import ConversionOptions, convert_dwca_archive<br>from dwca_cloud_geospatial.gbif import GbifDownloadOptions<br>from dwca_cloud_geospatial.geoparquet import GeoParquetWriterOptions<br><br>archive_path = Path(&quot;./source/gbif_archive.zip&quot;)<br>output_path = Path(&quot;./output_bundle&quot;)<br><br>result = convert_dwca_archive(<br>    archive_path,<br>    output_path,<br>    options=ConversionOptions(<br>        output_formats=(&quot;flatgeobuf&quot;, &quot;geoparquet&quot;),<br>        overwrite=True,<br>        gbif=GbifDownloadOptions(enrich=True),<br>        geoparquet=GeoParquetWriterOptions(large_output_mode=True),<br>        chunk_size=10000,<br>    ),<br>)<br>print(f&quot;Bundle successfully created at: {result.output_directory}&quot;)</pre><h3><strong>Under the Hood: Memory-Bounded Data Ingestion</strong></h3><p>Processing multi-gigabyte files in vanilla Python memory spaces can quickly result in an out-of-memory crash. To combat this, `<strong>dwca-cloud-geospatial</strong>` implements a strict file-based pipeline:</p><ol><li><strong>Streaming Handoff:</strong> It streams text-heavy occurrence rows from the DwC-A archive dynamically instead of loading everything at once.</li><li><strong>Intermediate Staging:</strong> Records that pass coordinate sanity tests are normalized and dumped into an intermediate, disk-backed GeoPackage staging file (`data/occurrences.gpkg`).</li><li><strong>GDAL Optimization:</strong> The pipeline hands off the persistent SQLite/GeoPackage layer directly to Pyogrio and GDAL’s native drivers to construct the final, spatially indexed FlatGeobuf map file.</li></ol><h3><strong>Deploying a Zero-Infrastructure Map in 60 Seconds</strong></h3><p>Once conversion finishes, the tool leaves you with a compiled static folder (`output_bundle/`) containing metadata registries, data files, and an pre-configured frontend:</p><pre>output_bundle/<br>  index.html                    # Viewer elements<br>  styles.css<br>  app.js<br>  manifest.json<br>  data/<br>    occurrences.fgb       # Your indexed web map file<br>    occurrences.parquet   # Your analytical big data file<br>    occurrences.gpkg      # Staging database</pre><p>All you have to do is upload this folder to a regular hosting service or cloud object storage like AWS S3, and your web map with data is ready!</p><p>You can test a live example of this bundle right now. I’ve deployed an interactive demo viewer on GitHub Pages:</p><p>👉 <strong>Watch the live demo of the viewer: </strong><a href="https://abiatov.github.io/dwca-cloud-geospatial/demo/output/index.html">https://abiatov.github.io/dwca-cloud-geospatial/demo/output/index.html</a></p><p>As you pan and zoom across the globe, MapLibre issues selective HTTP range requests to the underlying `<strong>occurrences.fgb</strong>` file. The browser only downloads the bytes needed to render points on the visible area — without active server backends or specialized map platforms.</p><p>The Viewer includes filters for Scientific Name, Kingdom, IUCN Red List Categories, Event Year, and several others. The viewer also displays the source Darwin Core dataset’s metadata (Description, DOI, Citation, License, GBIF dataset key, GBIF download key, and others). Each selected point displays its attributes, including a link to the Source Record in GBIF.</p><figure><img alt="The lightweight MapLibre viewer dwca-cloud-geospatial renders spatial coordinate points directly from a static FlatGeobuf file on GitHub Pages." src="https://cdn-images-1.medium.com/max/1024/1*fniuhpmlXxc95emiLVTdMw.png" /><figcaption>The lightweight MapLibre viewer renders spatial coordinate points directly from a static FlatGeobuf file on GitHub Pages.</figcaption></figure><p>The bottom of the left sidebar contains links to FlatGeobuf, GeoParquet, GeoPackage and metadata files.</p><figure><img alt="The viewer dwca-cloud-geospatial with links to FlatGeobuf, GeoParquet, GeoPackage and metadata files" src="https://cdn-images-1.medium.com/max/1024/1*34mJXcgz7ztIxD8VQOgxQQ.png" /></figure><h3><strong>Streaming Directly into Desktop GIS (QGIS)</strong></h3><p>The beauty of a cloud-optimized format like FlatGeobuf is that it isn’t locked into web browsers. Modern desktop GIS applications can stream it natively.</p><p>You don’t need to download the full dataset to work with it in <strong>QGIS</strong>. You can add the remote cloud file as a network layer using its direct URL:</p><pre>https://abiatov.github.io/dwca-cloud-geospatial/demo/output/data/occurrences.fgb</pre><p>To load it:</p><ol><li>Open QGIS and go to <strong>Layer -&gt; Add Layer -&gt; Add Vector Layer…</strong></li><li>Select Source Type <strong>Protocol: HTTP(S), cloud, etc.</strong> as the Protocol Type select <strong>HTTP/HTTPS/FTP</strong>.</li><li>Paste the URL into the <strong>URI</strong> field and click <strong>Add</strong>.</li></ol><figure><img alt="How to add a remote FlatGeobuf layer via HTTP(S) to QGIS for instant rendering without downloading the file." src="https://cdn-images-1.medium.com/max/1024/1*ugOCQjkuuSGHygv9QF_EyQ.png" /></figure><p>QGIS will instantly fetch the spatial index data and stream the points directly onto your map canvas, loading details on-demand as you navigate.</p><figure><img alt="points from FlatGeobuf in QGIS over OpenStreetMap" src="https://cdn-images-1.medium.com/max/1024/1*QcF_9_aY0t7tWxEyOb4bzw.png" /></figure><h3><strong>Cloud Queries: Serverless Big Data with DuckDB</strong></h3><p>Having a web map is great, but data engineers and data scientists want to run analytical queries. Thanks to the integrated <strong>GeoParquet</strong> generator, you can execute fast SQL pipelines without setting up a PostGIS database.</p><p>The generated Parquet asset for the demo is available here:</p><pre>https://abiatov.github.io/dwca-cloud-geospatial/demo/output/data/occurrences.parquet</pre><p>Let’s look at a practical scenario: we want to extract highly threatened species (categorized as Critically Endangered (CR), Endangered (EN), Vulnerable (VU), or Near Threatened (NT) on the IUCN Red List) that have been recorded since 1995.</p><p>Here is how you can perform this targeted cloud query directly using <strong>DuckDB</strong>:</p><pre>import duckdb<br><br># Initialize the connection and load the spatial module<br>con = duckdb.connect()<br>con.execute(&quot;INSTALL spatial; LOAD spatial;&quot;)<br><br># We execute an analytical query directly to a cloud file<br>query = &quot;&quot;&quot;<br>SELECT <br>    id, <br>    scientificName, <br>    event_year, <br>    iucn_red_list_category, <br>    decimalLatitude, <br>    decimalLongitude <br>FROM read_parquet(&#39;https://abiatov.github.io/dwca-cloud-geospatial/demo/output/data/occurrences.parquet&#39;)<br>WHERE event_year &gt;= 1995 <br>  AND iucn_red_list_category IN (&#39;CR&#39;, &#39;EN&#39;, &#39;VU&#39;, &#39;NT&#39;)<br>LIMIT 5;<br>&quot;&quot;&quot;<br><br>print(con.execute(query).df())</pre><p>DuckDB parses the structural metadata from the file’s footer, reads only the relevant column byte-ranges over the network, and prints the filtered analytical table in a fraction of a second.</p><h3><strong>Try It Yourself</strong></h3><p>If you want to evaluate the whole flow instantly without configuring anything on your local computer, I have published a fully interactive blueprint.</p><p>You can run the full conversion pipeline using this interactive link:</p><p>👉 <a href="https://colab.research.google.com/drive/1qc0yYgmtGMnOcQNpvY4HevhBzHibf6eb?usp=sharing"><strong>Open in Colab: DWCA-Cloud-Geospatial Demo Notebook</strong></a></p><p>You can create a copy of the notebook, insert the “GBIF_download_key” of your dataset and run the entire pipeline on your data.</p><p>The notebook automatically handles downloading a demo GBIF archive, triggers the `<em>inspect</em>`-`<em>convert</em>`-`<em>validate</em>` pipeline, and lets you extract the generated cloud artifacts directly out of the runtime container.</p><p>The tool is completely open-source under the GNU Affero General Public License v3.0. Check out the codebase, submit an issue, or drop a star on GitHub if this helps simplify your deployment architecture:</p><p><strong>GitHub Repository: </strong><a href="https://github.com/ABiatov/dwca-cloud-geospatial">ABiatov/dwca-cloud-geospatial</a></p><p><em>Next up on our roadmap: implementing native PMTiles slicing to handle web visualizations that stretch well into tens of millions of records. Stay tuned!</em></p><p>If you need help, message me on <a href="https://www.linkedin.com/in/anton-biatov/">LinkedIn</a>. I am open to collaboration and work opportunities.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=de303715c160" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PDAL in Python: A Practical Guide to Point Cloud Pipelines]]></title>
            <link>https://medium.com/@anton.biatov/pdal-in-python-a-practical-guide-to-point-cloud-pipelines-1f7e2bc3e2ea?source=rss-9beb986612a------2</link>
            <guid isPermaLink="false">https://medium.com/p/1f7e2bc3e2ea</guid>
            <category><![CDATA[gis]]></category>
            <category><![CDATA[lidar]]></category>
            <category><![CDATA[pdal]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[point-cloud]]></category>
            <dc:creator><![CDATA[Anton Biatov]]></dc:creator>
            <pubDate>Sat, 28 Mar 2026 22:07:21 GMT</pubDate>
            <atom:updated>2026-09-24T15:44:23.783Z</atom:updated>
            <content:encoded><![CDATA[<p>Learn how to run PDAL pipelines in Python, process LAZ point clouds, clip and merge data, and create a DEM GeoTIFF from ground-classified points.</p><figure><img alt="Preview image for working with point clouds in PDAL and running PDAL pipelines in Python" src="https://cdn-images-1.medium.com/max/1024/1*9u32vPU2C4fBOsGDat042A.jpeg" /></figure><p>PDAL (Point Data Abstraction Library) is an open-source library and toolset for point cloud processing. It is often described as a kind of “GDAL for point clouds” — just as GDAL works with raster and vector geodata, PDAL works with point clouds. I prefer using PDAL for all my point cloud processing tasks. PDAL works equally well with LiDAR data and point clouds generated through photogrammetry.</p><p>In this article, I will walk through several simple but useful PDAL examples that you can use directly in your own work or integrate into your data processing pipelines. I will also show an example of creating a GeoTIFF digital elevation model from a LAZ file. The article includes code blocks that you can copy and adapt to your own tasks.</p><h3>Pipelines</h3><p>The main PDAL concept that can feel intimidating at first is the pipeline, which is described as a JSON structure.</p><p>A JSON pipeline is simply a list where the following elements appear in sequence:</p><pre>Input (Reader) → Filters → Output (Writer)</pre><p><strong>Pipeline components:</strong></p><p>1. Readers — Load data (for example, LAS/LAZ files)</p><p>2. Filters — Transform the data:</p><ul><li>noise filtering</li><li>classification</li><li>merging</li><li>clipping</li><li>clustering</li><li>and many other <a href="https://pdal.io/en/2.9.0/stages/filters.html">filters</a></li></ul><p>3. Writers — Save the result (to a file or a database)</p><p>Each step is an independent block:</p><ul><li>you can easily change the order</li><li>you can combine them as needed</li></ul><h3>Basic Example of Running a PDAL Pipeline</h3><p>The simplest approach is to write the pipeline into a separate JSON file and then run it.</p><p>Here is our pipeline saved in a separate JSON file, <em>`</em><strong><em>my_pipeline.json</em></strong><em>`</em>:</p><pre>{<br>  &quot;pipeline&quot;: [<br>    &quot;input.laz&quot;,<br>    {<br>      &quot;type&quot;: &quot;filters.outlier&quot;,<br>      &quot;method&quot;: &quot;statistical&quot;,<br>      &quot;mean_k&quot;: 8,<br>      &quot;multiplier&quot;: 2.0<br>    },<br>    {<br>      &quot;type&quot;: &quot;filters.smrf&quot;<br>    },<br>    {<br>        &quot;type&quot;:&quot;filters.expression&quot;,<br>        &quot;expression&quot;:&quot;Classification == 2&quot;<br>    },<br>    &quot;output.laz&quot;<br>  ]<br>}</pre><p>Then run this command in the terminal:</p><pre>pdal pipeline my_pipeline.json</pre><p>What happens here:</p><ul><li>read the <em>`</em><strong><em>input.laz</em></strong><em>`</em> file</li><li>remove outliers (noise)</li><li>classify ground with the Simple Morphological Filter (SMRF)</li><li>keep only ground points (<em>`</em><strong><em>Classification == 2</em></strong><em>`</em>)</li><li>save the result to <em>`</em><strong><em>output.laz</em></strong><em>`</em></li></ul><p>The result looks like this:</p><figure><img alt="Comparison of the PDAL pipeline result: on the left the original point cloud, in the center the cloud after outlier filtering and coloring by height, on the right only ground points after SMRF and the Classification == 2 filter" src="https://cdn-images-1.medium.com/max/1024/1*JRxTL61P_kQYLmtL4mgfuw.jpeg" /></figure><h3>Running a PDAL Pipeline in Python</h3><p>Now let us take the same pipeline, run it in Python, and pass the file paths through variables.</p><pre>import json # Needed to turn a Python data structure into a JSON string<br>import pdal # Python binding for PDAL, used to create and run a pipeline.<br><br># File paths<br>input_laz_file = &quot;input.laz&quot;<br>output_laz_file = &quot;output.laz&quot;<br><br># Pipeline definition<br>my_pipeline = [<br>    # Stage 1 - read the input file<br>    input_laz_file,<br>    # Stage 2 - remove outliers<br>    {<br>      &quot;type&quot;: &quot;filters.outlier&quot;, # Filter for finding outliers - suspicious points that are too far from their neighbors.<br>      &quot;method&quot;: &quot;statistical&quot;, # Use a statistical method<br>      &quot;mean_k&quot;: 8, # For each point, consider 8 neighbors<br>      &quot;multiplier&quot;: 2.0 # Threshold for how far from normal a point can be before it is treated as an outlier.<br>    },<br>    # Stage 3 - classify ground with SMRF<br>    {<br>      &quot;type&quot;: &quot;filters.smrf&quot;<br>    },<br>    # Stage 4 - keep only ground points<br>    {<br>        &quot;type&quot;:&quot;filters.expression&quot;,<br>        &quot;expression&quot;:&quot;Classification == 2&quot;<br>    },<br>    # Stage 5 - write the result<br>    {<br>        &quot;type&quot;: &quot;writers.las&quot;,<br>        &quot;filename&quot;: output_laz_file<br>    }<br>  ]<br><br># Convert the pipeline to JSON<br>pipeline_json = json.dumps(my_pipeline)<br><br># Just debug output. Print it if you want to check that everything looks correct.<br># Useful for:<br># - inspecting the final pipeline structure<br># - making sure the JSON was built correctly<br># - copying it and running it separately from the CLI if needed<br>print(f&quot;pipeline_json: {pipeline_json}&quot;)<br><br># A Python PDAL pipeline object is created here.<br># At this stage, the pipeline has not been executed yet.<br>pipeline = pdal.Pipeline(pipeline_json)<br><br># Execute the pipeline<br># Returns the number of processed points<br>count = pipeline.execute()<br><br># Pipeline execution metadata<br>metadata = pipeline.metadata<br><br># Text execution log. If the pipeline completed successfully, the log is empty.<br>log = pipeline.log<br><br><br>print(f&quot;metadata: {metadata}&quot;)<br>print(f&quot;count: {count}&quot;)<br>print(f&quot;log: {log}&quot;)</pre><h3>Reading a Point Cloud into a NumPy Array</h3><p>You can also set things up so that the pipeline result is not written to a file, but returned as a NumPy array instead. This is useful if you want to pass the data into other tools in your workflow or visualize it in Jupyter.</p><p>To do that, remove the file-writing stage from the pipeline and add:</p><pre>arrays = pipeline.arrays</pre><p>Here is a minimal example:</p><pre>import json <br>import pdal<br><br>input_laz_file = &quot;input.laz&quot;<br><br>my_pipeline = [<br>    input_laz_file,<br>    {<br>        &quot;type&quot;:&quot;filters.expression&quot;,<br>        &quot;expression&quot;:&quot;Classification == 2&quot;<br>    }<br>]<br><br>pipeline_json = json.dumps(my_pipeline)<br>pipeline = pdal.Pipeline(pipeline_json)<br>count = pipeline.execute()<br>arrays = pipeline.arrays<br>print(f&quot;count: {count}&quot;)</pre><p>Then you can simply preview NumPy array as a table:</p><pre>import pandas as pd<br><br>arr = pipeline.arrays[0]<br>df = pd.DataFrame(arr)<br><br># df.head()<br><br># If there are many fields, you can select only the ones you need for preview:<br>df[[&quot;X&quot;, &quot;Y&quot;, &quot;Z&quot;, &quot;Classification&quot;, &quot;Red&quot;, &quot;Green&quot;, &quot;Blue&quot;, &quot;Infrared&quot; ]].head()</pre><figure><img alt="Preview of the first rows of a DataFrame created from pipeline.arrays[0], with fields X, Y, Z, Classification, Red, Green, Blue, and Infrared; the shown points have Classification equal to 2" src="https://cdn-images-1.medium.com/max/1024/1*YzGeEnbkJQB-0JgKcwgazA.png" /></figure><p>You can also visualize this NumPy array directly with matplotlib:</p><pre>import numpy as np<br>import matplotlib.pyplot as plt<br><br>arr = pipeline.arrays[0]<br><br># To speed up visualization, randomly select point indices from arr<br>idx = np.random.choice(len(arr), size=min(50000, len(arr)), replace=False)<br><br>x = arr[&quot;X&quot;][idx]<br>y = arr[&quot;Y&quot;][idx]<br>z = arr[&quot;Z&quot;][idx]<br><br>fig = plt.figure(figsize=(12, 7))<br>ax = fig.add_subplot(111, projection=&quot;3d&quot;)<br><br>sc = ax.scatter(<br>    x, y, z,<br>    c=z,          # Color by Z<br>    cmap=&quot;turbo&quot;, # Turbo palette<br>    s=0.2,<br>    vmin=z.min(),<br>    vmax=z.max()<br>)<br><br>ax.set_xlabel(&quot;X&quot;)<br>ax.set_ylabel(&quot;Y&quot;)<br>ax.set_zlabel(&quot;Z&quot;)<br><br>cbar = plt.colorbar(sc, ax=ax, pad=0.1)<br>cbar.set_label(&quot;Z&quot;)<br><br>plt.show()</pre><figure><img alt="3D scatter plot of a point cloud from pipeline.arrays[0] in matplotlib: X, Y, Z axes, points colored by height Z with the turbo palette, and a Z color scale shown on the right" src="https://cdn-images-1.medium.com/max/734/1*jxDNz2S2QoqodG4K3UJWgQ.png" /></figure><h3>Simple but Useful PDAL Pipelines</h3><p>Below are a few simple but useful pipelines.</p><h4>Clipping a Point Cloud by Extent</h4><p>The variables <em>`</em><strong><em>minx</em></strong><em>`</em>, <em>`</em><strong><em>maxx</em></strong><em>`</em>, <em>`</em><strong><em>miny</em></strong><em>`</em>, and <em>`</em><strong><em>maxy</em></strong><em>`</em> must be in the same coordinate system as the source LAZ file.</p><pre>pipeline_clip_by_extent = [<br>    {<br>      &quot;type&quot;: &quot;readers.las&quot;,<br>      &quot;filename&quot;: input_laz_file<br>    },<br>    {<br>        &quot;type&quot;: &quot;filters.crop&quot;,<br>        &quot;bounds&quot;: f&quot;([{minx},{maxx}],[{miny},{maxy}])&quot;<br>    },<br>    {<br>        &quot;type&quot;: &quot;writers.las&quot;,<br>        &quot;filename&quot;: clipped_by_bbox_file<br>    }<br>]</pre><h4>Clipping a Point Cloud by Polygon</h4><p>PDAL accepts a polygon in WKT format. So first you need to get the geometry in WKT format from your source polygon vector file. For this, we will use <em>`</em><strong><em>geopandas</em></strong><em>`</em>.</p><pre>import json<br>import pdal<br>import geopandas as gpd<br><br>def get_wkt_from_gpkg(gpkg_path):<br>    &quot;&quot;&quot;Reads the first polygon from a gpkg and returns it in WKT format&quot;&quot;&quot;<br>    gdf = gpd.read_file(gpkg_path)<br>    # Take the geometry of the first feature and merge it if there are several polygons<br>    geom = gdf.geometry.union_all()<br>    return geom.wkt<br><br>mask_wkt_geometry = get_wkt_from_gpkg(mask_file)<br><br>pipeline_clip_by_polygon = [<br>    input_laz_file,<br>    {<br>        &quot;type&quot;: &quot;filters.crop&quot;,<br>        &quot;polygon&quot;: f&quot;{mask_wkt_geometry}&quot;<br>    },<br>    {<br>        &quot;type&quot;: &quot;writers.las&quot;,<br>        &quot;filename&quot;: clipped_by_polygon_file<br>    }<br>]<br><br>pipeline_json = json.dumps(pipeline_clip_by_polygon)<br>pipeline = pdal.Pipeline(pipeline_json)<br>count = pipeline.execute()</pre><h4>Merging Several Point Clouds</h4><p>To merge several point clouds, use the PDAL tool <em>`</em><strong><em>filters.merge</em></strong><em>`</em>. First, pass the paths to several LAZ files into the pipeline list, then apply the <em>`</em><strong><em>filters.merge</em></strong><em>`</em> filter. The result is a single merged point cloud.</p><pre>pipeline_merge = [<br>    input_laz_file_1,<br>    input_laz_file_2,<br>    {<br>      &quot;type&quot;: &quot;filters.merge&quot;,<br>    },<br>    {<br>      &quot;type&quot;: &quot;writers.las&quot;,<br>      &quot;filename&quot;: output_merged_laz_file<br>    }<br>  ]</pre><p><strong>Creating a GeoTIFF DEM from a Point Cloud</strong></p><p>The DEM usually comes out with plenty of holes. Those holes appear where trees, buildings, and other objects were removed. You can fill them later with other tools if needed.</p><pre>target_resolution = 0.25 # Raster pixel size in coordinate units. In this case, meters.<br><br>work_srs = &quot;EPSG:3763&quot; # Coordinate system of the source point cloud.<br><br>pipeline_dem = [<br>    {<br>      &quot;type&quot;: &quot;readers.las&quot;,<br>      &quot;filename&quot;: input_laz_2_dem_file<br>    },<br>    {<br>        &quot;type&quot;:&quot;filters.expression&quot;,<br>        &quot;expression&quot;:&quot;Classification == 2&quot;<br>    },<br>    {<br>        &quot;type&quot;: &quot;writers.gdal&quot;,<br>        &quot;filename&quot;: output_dem_geotiff_file,<br>        &quot;override_srs&quot;: work_srs, # Optional parameter. Explicitly writes the coordinate system into the GeoTIFF<br>        &quot;resolution&quot;: target_resolution, <br>        &quot;radius&quot;: target_resolution * 1.5, # Search radius for points around the pixel center.<br>        # A radius slightly larger than the pixel size helps pick neighboring points<br>        # even where points are not perfectly centered on the pixel grid.<br>        # Radius too small -&gt; many empty cells<br>        # Radius too large -&gt; the surface becomes overly smoothed<br>        &quot;output_type&quot;: &quot;idw&quot;, # Method used to compute the pixel value from neighboring points.<br>        # &quot;output_type&quot; can be: min, max, mean, idw, count, stdev<br>        &quot;bounds&quot;: f&quot;([{minx},{maxx}],[{miny},{maxy}])&quot;, # Bounds of the output raster. Optional, but useful when you want precise control over the extent of the final GeoTIFF.<br>        &quot;gdaldriver&quot;: &quot;GTiff&quot;,<br>        &quot;nodata&quot;: -9999 # Value written into cells with no data<br>    }<br>]</pre><p>You can fill nodata cells using QGIS, GDAL, Rasterio, scipy and other tools.</p><figure><img alt="Visualization of a digital elevation model in QGIS. The digital elevation model (DEM) was obtained using PDAL from the LAZ point cloud." src="https://cdn-images-1.medium.com/max/1024/1*pD35lWgbXz7KkaxQyCoNlA.jpeg" /><figcaption>Visualizing the resulting digital elevation model in QGIS</figcaption></figure><h3>Wrapping up</h3><p>There are <a href="https://pdal.io/en/2.9.0/stages/filters.html">a huge number</a> of tools you can use inside PDAL pipelines. Pipelines are very flexible and can be configured and optimized in many different ways.</p><p>I hope this article makes it easier to navigate the official <a href="https://pdal.io/">PDAL documentation</a>.</p><p>You can install PDAL by following the instructions on the official website: <a href="https://pdal.org/en/2.10.0/quickstart.html">PDAL quickstart</a>. If you want me to write a separate guide on how I installed PDAL, let me know in the comments.</p><p>If you need help, message me on <a href="https://www.linkedin.com/in/anton-biatov/">LinkedIn</a>. I am open to collaboration and work opportunities.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1f7e2bc3e2ea" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Zonal statistics in python using Earth Engine and Google Colab]]></title>
            <link>https://medium.com/@anton.biatov/zonal-statistics-in-python-using-earth-engine-and-google-colab-7101d1d42ffd?source=rss-9beb986612a------2</link>
            <guid isPermaLink="false">https://medium.com/p/7101d1d42ffd</guid>
            <category><![CDATA[geospatial-analytics]]></category>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[ndvi]]></category>
            <category><![CDATA[google-earth-engine]]></category>
            <category><![CDATA[gis]]></category>
            <dc:creator><![CDATA[Anton Biatov]]></dc:creator>
            <pubDate>Sat, 04 Oct 2025 12:59:45 GMT</pubDate>
            <atom:updated>2026-09-24T15:53:55.889Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Zonal statistics with the Google Earth Engine Python API right inside Google Colab" src="https://cdn-images-1.medium.com/max/1024/1*ecED3a2_k7SVR9m50tLUdw.jpeg" /></figure><p>In this article, I’ll guide you through calculating zonal statistics for multiple parameters — like mean, median, standard deviation, variance, minimum, and maximum — across multiple raster layers simultaneously. We’ll use the Google Earth Engine Python API right inside Google Colab. This approach is efficient for geospatial analysis, especially when dealing with global datasets.</p><p><strong>What you’ll get</strong></p><ul><li>A universal template for zonal statistics across multiple rasters.</li><li>A single composite reducer to minimize the number of data passes.</li><li>Export to GeoPackage and CSV, as well as upload to Google Drive.</li><li>A quick preview rendering of the resulting metrics on the map.</li><li>Bonus: Ready to use Colab notebook</li></ul><p><strong>The data we’ll use includes:</strong></p><p>Polygons:</p><ul><li><a href="https://www.naturalearthdata.com/downloads/110m-cultural-vectors/">Natural Earth (1:110m)</a> Admin 0 — Countries without large lakes.</li></ul><p>Rasters:</p><ul><li>Relief — <a href="https://developers.google.com/earth-engine/datasets/catalog/NOAA_NGDC_ETOPO1">ETOPO1: Global 1 Arc-Minute Elevation</a></li><li>Soil temperature — <a href="https://developers.google.com/earth-engine/datasets/catalog/ECMWF_ERA5_LAND_MONTHLY_BY_HOUR">ERA5-Land Monthly Averaged by Hour of Day — ECMWF Climate Reanalysis</a></li><li>Vegetation — <a href="https://developers.google.com/earth-engine/datasets/catalog/MODIS_061_MOD13A3">MOD13A3.061 Vegetation Indices Monthly L3 Global 1 km SIN Grid</a></li></ul><p>Open the <a href="https://www.naturalearthdata.com/downloads/110m-cultural-vectors/">Natural Earth (1:110m) website</a> in a new tab and download the file &quot;Admin 0 — Countries without boundary lakes&quot;.</p><p>Next, execute the code cells in your Сolab notebook one by one.</p><h3><strong>Initializing the Environment (Colab)</strong></h3><h4><strong>Install Required Packages</strong></h4><pre># Import a helper module from Google Colab for managing output.<br># Used, for example, to clean up a cell after installing packages.<br>from google.colab import output # Only needed in Colab. This module is not available in regular Jupyter.<br><br># Installing Required Libraries.<br><br># Shell commands are supported in Colab, using !. Here, we install GeoPandas and geemap.<br>!pip install -q geopandas geemap<br><br># Import standard and external libraries that we will use below.<br>import os<br>from google.colab import files # Convenient for uploading/downloading files to Colab.<br><br>import ee # Google Earth Engine Python API<br>import geopandas as gpd<br>import geemap<br><br># Clear cell output after installation/import to make the notebook look cleaner.<br>output.clear()</pre><h4><strong>GEE Authentication and Initialization</strong></h4><p><strong>ee.Authenticate()</strong> starts the interactive authorization flow:</p><ul><li>In Colab, it will redirect to the Google account login page.</li><li>You need to log in to your account and authorize Colab notebook. This is done once per session.</li></ul><p>NOTE: If you are already logged in in this session, you do not need to call it again.</p><p><strong>ee.Initialize(…)</strong> establishes a connection to Earth Engine.</p><p>The <strong>project=</strong> parameter specifies the GCP (Google Cloud Platform) project under which requests (quotas/billing) will be made. Must match your GCP project linked to Earth Engine. Replace ‘<strong>MY-GEE-PROJECT</strong>’ with your actual GEE/GCP project.</p><pre>ee.Authenticate()<br><br>ee.Initialize(project=&#39;MY-GEE-PROJECT&#39;)</pre><h3><strong>Analysis</strong></h3><h4><strong>Creating a composite reducer for summary statistics</strong></h4><p>Idea: combine several reducers into one to pass over the data only once.</p><p><strong>sharedInputs=True</strong> is an important flag: the same inputs are fed to all reducers, which saves compute time and quotas.</p><p>NOTE: Each base reducer has its own output names. When combined, they are concatenated. For multiband images, GEE will add suffixes to band names, e.g., NDVI_mean, NDVI_median, …</p><pre>reducers = (<br>    ee.Reducer.mean()      # mean value (output &#39;mean&#39;)<br>      .combine(ee.Reducer.median(),   sharedInputs=True)   # median (&#39;median&#39;)<br>      .combine(ee.Reducer.stdDev(),   sharedInputs=True)   # standard deviation (&#39;stdDev&#39;)<br>      .combine(ee.Reducer.variance(), sharedInputs=True)   # variance (&#39;variance&#39;)<br>      .combine(ee.Reducer.min(),      sharedInputs=True)   # minimum (&#39;min&#39;)<br>      .combine(ee.Reducer.max(),      sharedInputs=True)   # maximum (&#39;max&#39;)<br>)</pre><h4><strong>Importing images</strong></h4><p>Raster layers: elevation (ETOPO1), NDVI (MODIS), soil temperature (ERA5-Land)</p><pre># Relief: ETOPO1 bedrock (meters)<br>elev_dataset = ee.Image(&#39;NOAA/NGDC/ETOPO1&#39;)<br>elevation = elev_dataset.select(&#39;bedrock&#39;)  # &quot;bedrock&quot; layer<br># Land mask: keep only &gt;= 0 m (seas/oceans with negative elevation are masked out)<br>elevation = elevation.updateMask(elevation.gte(0)).rename(&#39;elevation&#39;)<br><br># Vegetation: MODIS MOD13A3 for 2024, annual mean<br># NDVI has a scale factor of 0.0001 and a valid range of approximately [-2000..10000] before scaling.<br>vi_dataset = (<br>    ee.ImageCollection(&#39;MODIS/061/MOD13A3&#39;)<br>    .filter(ee.Filter.date(&#39;2024-01-01&#39;, &#39;2024-12-31&#39;))<br>    .mean()<br>)<br><br>ndvi_raw = vi_dataset.select(&#39;NDVI&#39;)<br>ndvi = (<br>    ndvi_raw<br>    .multiply(0.0001)                    # convert to the [-1..1] range<br>    .updateMask(ndvi_raw.neq(-3000))     # remove fill values (typical for MODIS)<br>    .rename(&#39;ndvi&#39;)<br>)<br><br># Climate: ERA5-Land MONTHLY_BY_HOUR for 2024, annual mean<br># Units of &#39;soil_temperature_level_1&#39; are Kelvin. Convert to °C.<br>ECMWF_dataset = (<br>    ee.ImageCollection(&#39;ECMWF/ERA5_LAND/MONTHLY_BY_HOUR&#39;)<br>    .filter(ee.Filter.date(&#39;2024-01-01&#39;, &#39;2024-12-31&#39;))<br>    .mean()<br>)<br><br>temperature_k = ECMWF_dataset.select(&#39;soil_temperature_level_1&#39;)<br>temperature_c = temperature_k.subtract(273.15).rename(&#39;temperature_c&#39;)<br><br># ---------------------------------------------<br># Merge layers into a single image<br># ---------------------------------------------<br># Band order: elevation (m), NDVI (unitless), soil temperature (°C).<br># Important: the layers have different native projections/resolutions; when using reduceRegion/export<br># specify an explicit scale (or crs/crsTransform) to avoid unwanted resampling.<br>image_to_process = elevation.addBands([ndvi, temperature_c])</pre><h4><strong>Uploading a vector file to Colab</strong></h4><p>In this article I will show 2 methods for uploading files into a Colab session.</p><p><strong>1. Uploading a file through the Colab interface</strong></p><p>You can upload a vector file to session storage using the corresponding button in the interface.</p><p>Then pass the path to the uploaded file to the `<strong>polygons_filename</strong>` variable.</p><figure><img alt="upload a file to Colab session storage using the button in the interface" src="https://cdn-images-1.medium.com/max/1024/1*Hy1p340H159MKeR9NXNa6w.png" /></figure><p><strong>2. Uploading a file through the files.upload() widget</strong></p><p>I recommend this method of uploading files.</p><pre># After running, a &quot;Choose files&quot; dialog will appear.<br># Important: a shapefile consists of multiple files (.shp, .shx, .dbf, .prj, etc.).<br># It&#39;s more convenient to upload a single ZIP containing the set, or use GeoJSON/GPKG/KML formats.<br>uploaded = files.upload()  # dict: {filename: bytes}<br><br># Basic (current) approach: assume a single file and just take its name.<br># If multiple are uploaded, the first in the dictionary order is taken.<br>polygons_filename = list(uploaded.keys())[0]<br><br>print(f&quot;File &#39;{polygons_filename}&#39; loaded successfully.&quot;)<br><br># Get the base filename without the extension.<br># os.path.splitext will cut off only the last extension (e.g., &#39;.zip&#39; or &#39;.shp&#39;).<br># Example: &#39;my.layer.v1.geojson&#39; -&gt; (&#39;my.layer.v1&#39;, &#39;.geojson&#39;)<br>polygons_basename = os.path.splitext(polygons_filename)[0]<br>print(f&quot;polygons_basename: {polygons_basename}&quot;)<br><br># Reading a vector file with geopandas<br>polygons_gdf = gpd.read_file(polygons_filename)<br><br># Checking the coordinate reference system (CRS) of the vector layer<br>if polygons_gdf.crs is None:<br>    print(&quot;⚠️ The layer is missing a CRS. If you know the original projection, set it manually, otherwise the results may be incorrect.&quot;)<br>else:<br>    if polygons_gdf.crs.to_string().upper() != &quot;EPSG:4326&quot;:<br>        polygons_gdf = polygons_gdf.to_crs(epsg=4326)<br>        print(&quot;Projection redefined to WGS84 (EPSG:4326).&quot;)</pre><p><strong>Converting GeoDataFrame to ee.FeatureCollection</strong></p><p>! This is a key step to pass the data into GEE !</p><pre>polygons_ee = geemap.geopandas_to_ee(polygons_gdf)</pre><h4><strong>Zonal statistics over polygons</strong></h4><p><strong>reduceRegions</strong> applies a composite reducer to each feature in a FeatureCollection.</p><ul><li>Preserves the original polygon attributes.</li><li>Adds fields with statistics for each image band. Example names: elevation_mean, ndvi_median, temperature_c_stdDev, …</li></ul><p>WARNING: different layers have different native resolutions; with the `scale` parameter we set a unified sampling resolution (here 5000 m, which is coarser than MODIS (~1 km) and ETOPO1 (~1.8 km), but closer to ERA5-Land (~10 km)). Choose `scale` deliberately for your task.</p><pre>stats_results_ee = image_to_process.reduceRegions(<br>    collection=polygons_ee,  # polygons (ee.FeatureCollection)<br>    reducer=reducers,        # previously assembled composite reducer<br>    scale=5000               # pixel size for sampling (m)<br>)</pre><h4><strong>Converting ee.FeatureCollection to GeoDataFrame</strong></h4><p><strong>eemap.ee_to_gdf</strong> pulls ALL features to the client (into the notebook’s memory).</p><p>This is convenient for small/medium datasets, but can be heavy for large collections.</p><p>You can use the command `<strong>%%time</strong>` to display the execution time of calculations.</p><pre>%%time<br><br>stats_results_gdf = geemap.ee_to_gdf(stats_results_ee)</pre><p>Quick overview of columns and types</p><pre>stats_results_gdf.info()</pre><p>View the first four lines of the result.</p><pre>stats_results_gdf.head(4)</pre><figure><img alt="gdf.head(4)" src="https://cdn-images-1.medium.com/max/1024/1*xFWG-NP6-OkIOGXz_C9B1Q.png" /></figure><p>Preview of the first feature</p><pre>stats_results_ee.first().getInfo()</pre><figure><img alt="stats_results_ee.first().getInfo() — displaying the first ee.FeatureCollection object" src="https://cdn-images-1.medium.com/max/1024/1*K9cmB8-Zq50L93AjbdFcng.jpeg" /></figure><h4><strong>Reordering and cleaning columns (optional)</strong></h4><p>If you use your own vector file, you’ll have a different set of fields. Adapt this code to your dataset.</p><pre># Define a new column order. Remove extras and keep the necessary fields, placing them in the desired order.<br><br>new_order = [<br>    &#39;geometry&#39;, &#39;ABBREV&#39;, &#39;NAME&#39;, &#39;CONTINENT&#39;, &#39;ECONOMY&#39;, &#39;ISO_A2&#39;, &#39;ISO_A3&#39;, &#39;ISO_N3&#39;,<br>    &#39;elevation_mean&#39;, &#39;elevation_median&#39;, &#39;elevation_min&#39;, &#39;elevation_max&#39;, &#39;elevation_stdDev&#39;, &#39;elevation_variance&#39;,<br>    &#39;ndvi_mean&#39;, &#39;ndvi_median&#39;, &#39;ndvi_min&#39;, &#39;ndvi_max&#39;, &#39;ndvi_stdDev&#39;, &#39;ndvi_variance&#39;,<br>    &#39;temperature_c_mean&#39;, &#39;temperature_c_median&#39;, &#39;temperature_c_min&#39;, &#39;temperature_c_max&#39;, &#39;temperature_c_stdDev&#39;, &#39;temperature_c_variance&#39;<br>]<br><br># Apply the new order<br>stats_results_gdf = stats_results_gdf[new_order]</pre><p>View the first few rows of the result</p><pre>stats_results_gdf.head()</pre><figure><img alt="gdf.head()" src="https://cdn-images-1.medium.com/max/1024/1*0qMpBwa2Kw39-4Yw418vyA.png" /></figure><h3><strong>Saving results to GPKG and CSV</strong></h3><p>The stats_results_gdf object is a GeoDataFrame and you can convert it to any vector format supported by the GeoPandas library.</p><pre>layer_name = &quot;zonal_statistics&quot;  # layer name in GPKG<br>output_gpkg_filename = f&#39;{polygons_basename}_statistics.gpkg&#39;<br>output_csv_filename = f&#39;{polygons_basename}_statistics.csv&#39;<br><br># Save GeoPackage (geometry + attributes)<br># driver=&#39;GPKG&#39; is a required parameter. layer= sets the layer name inside the file, optional parameter.<br>stats_results_gdf.to_file(output_gpkg_filename, driver=&#39;GPKG&#39;, layer=layer_name)<br># Save CSV (attributes only)<br># errors=&#39;ignore&#39; prevents errors if the geometry column is missing.<br>stats_results_gdf.drop(columns=&quot;geometry&quot;, errors=&quot;ignore&quot;).to_csv(<br>    output_csv_filename,<br>    index=False,<br>    encoding=&quot;utf-8&quot;  # for Cyrillic<br>    # float_format=&quot;%.6f&quot;  # enable if you want a fixed number of decimal places<br>    )<br><br>print(f&quot;Saved:\n- {output_gpkg_filename} (layer: {layer_name})\n- {output_csv_filename}&quot;)</pre><p>You can use the code for automatic file download.</p><pre>files.download(output_gpkg_filename)<br>files.download(output_csv_filename)</pre><p>You can also download the finished files by selecting them in your browser’s file system.</p><figure><img alt="download data from Doogle Colab" src="https://cdn-images-1.medium.com/max/1024/1*WAFzO7Jk4_tdPN-ZyCCZWg.png" /></figure><h4><strong>Exporting the resulting table to Google Drive</strong></h4><p>You can also export the stats_results_ee object, which is an ee.FeatureCollection, to Google Drive directly without transforming it into a GeoDataFrame.</p><p>Choose a format: ‘<strong>GeoJSON</strong>’ (with geometry), ‘<strong>CSV</strong>’ (attributes only), ‘<strong>SHP</strong>’ (has attribute name length limits), ‘<strong>KML</strong>’, ‘<strong>KMZ</strong>’</p><p>For large and complex geometries, <strong>GeoJSON</strong> is usually more convenient</p><pre>result_filename = f&#39;{polygons_basename}_zonal_statistics&#39;<br>export_folder = &#39;GEE_Exports&#39; # Folder on your Google Drive<br>file_format = &#39;GeoJSON&#39;<br><br>task = ee.batch.Export.table.toDrive(<br>    collection=stats_results_ee,           # ee.FeatureCollection<br>    description=result_filename,           # task name and filename without extension<br>    folder=export_folder,                  # folder in Google Drive<br>    fileFormat=file_format                 # file format<br>)<br><br># Start the export task<br>task.start()<br><br>print(&#39;Export task started. Check the Earth Engine Tasks tab for progress:&#39;)<br>print(task.status())</pre><p>Exported GeoJSON, GeoPackage, and CSV files can be used in various analytical and GIS tools.</p><ul><li>Python (GeoPandas, Matplotlib, etc.)</li><li>R(sf, ggplot2)</li><li>QGIS, ArcGIS, and other GIS systems</li><li>Web mapping software</li></ul><p>You can also integrate this analysis method into your Python workflow.</p><h3><strong>Limitations of the method</strong></h3><p>The described method has a serious limitation.</p><p>When processing large vector files, you’ll quickly encounter the error:</p><blockquote>Request payload size exceeds the limit: 10485760 bytes</blockquote><p>This means the data transferred in the Earth Engine API request is larger than 10 MB.</p><p>In the next article, we’ll look at several ways to circumvent this limitation.</p><p>See also the article about <a href="https://medium.com/@anton.biatov/how-to-combine-topography-and-vegetation-a-practical-guide-to-zonal-statistics-in-google-earth-d8167615937b">Zonal Statistics in the Google Earth Engine web interface &gt;&gt;&gt;</a></p><h3><strong>Preview Data on Maps (optional)</strong></h3><p>The next step is optional, since you already have the data.</p><p>In this step, we’ll simply look at the resulting statistics on the map.</p><h4><strong>Rasterize polygons with calculation results for quick preview on the map</strong></h4><pre>elev_mean_img = (<br>    stats_results_ee<br>    .reduceToImage(properties=[&#39;elevation_mean&#39;], reducer=ee.Reducer.first())<br>    .rename(&#39;elevation&#39;)<br>)<br><br>elev_stdDev_img = (<br>    stats_results_ee<br>    .reduceToImage(properties=[&#39;elevation_stdDev&#39;], reducer=ee.Reducer.first())<br>    .rename(&#39;elevation&#39;)<br>)<br><br>elev_variance_img = (<br>    stats_results_ee<br>    .reduceToImage(properties=[&#39;elevation_variance&#39;], reducer=ee.Reducer.first())<br>    .rename(&#39;elevation&#39;)<br>)<br><br>ndvi_mean_img = (<br>    stats_results_ee<br>    .reduceToImage(properties=[&#39;ndvi_mean&#39;], reducer=ee.Reducer.first())<br>    .rename(&#39;ndvi&#39;)<br>)<br><br>ndvi_stdDev_img = (<br>    stats_results_ee<br>    .reduceToImage(properties=[&#39;ndvi_stdDev&#39;], reducer=ee.Reducer.first())<br>    .rename(&#39;ndvi&#39;)<br>)<br><br>ndvi_variance_img = (<br>    stats_results_ee<br>    .reduceToImage(properties=[&#39;ndvi_variance&#39;], reducer=ee.Reducer.first())<br>    .rename(&#39;ndvi&#39;)<br>)<br><br>temperature_mean_img = (<br>    stats_results_ee<br>    .reduceToImage(properties=[&#39;temperature_c_mean&#39;], reducer=ee.Reducer.first())<br>    .rename(&#39;temperature&#39;)<br>)<br><br>temperature_stdDev_img = (<br>    stats_results_ee<br>    .reduceToImage(properties=[&#39;temperature_c_stdDev&#39;], reducer=ee.Reducer.first())<br>    .rename(&#39;temperature&#39;)<br>)<br><br>temperature_variance_img = (<br>    stats_results_ee<br>    .reduceToImage(properties=[&#39;temperature_c_variance&#39;], reducer=ee.Reducer.first())<br>    .rename(&#39;temperature&#39;)<br>)</pre><h4><strong>Let’s set up palettes for displaying layers</strong></h4><pre>palette_elev = [<br>    &quot;006837&quot;,&quot;1a9850&quot;,&quot;66bd63&quot;,&quot;a6d96a&quot;,&quot;d9ef8b&quot;,<br>    &quot;ffffbf&quot;,&quot;fee08b&quot;,&quot;fdae61&quot;,&quot;f46d43&quot;,&quot;d73027&quot;,&quot;a50026&quot;<br>]<br><br>palette_ndvi = [<br>    &#39;ffffff&#39;,&#39;fcd163&#39;,&#39;99b718&#39;,&#39;66a000&#39;, &#39;3e8601&#39;,<br>    &#39;207401&#39;,&#39;056201&#39;,&#39;004c00&#39;,&#39;011301&#39;<br>    ]<br><br>palette_temperature = [<br>    &#39;040274&#39;, &#39;040281&#39;, &#39;0502a3&#39;, &#39;0502b8&#39;, &#39;0502ce&#39;, &#39;0502e6&#39;,<br>    &#39;0602ff&#39;, &#39;235cb1&#39;, &#39;307ef3&#39;, &#39;269db1&#39;, &#39;30c8e2&#39;, &#39;32d3ef&#39;,<br>    &#39;3be285&#39;, &#39;3ff38f&#39;, &#39;86e26f&#39;, &#39;3ae237&#39;, &#39;b5e22e&#39;, &#39;d6e21f&#39;,<br>    &#39;fff705&#39;, &#39;ffd611&#39;, &#39;ffb613&#39;, &#39;ff8b13&#39;, &#39;ff6e08&#39;, &#39;ff500d&#39;,<br>    &#39;ff0000&#39;, &#39;de0101&#39;, &#39;c21301&#39;, &#39;a71001&#39;, &#39;911003&#39;<br>  ]<br><br>palette_std = [<br>    &quot;5e4fa2&quot;,&quot;3288bd&quot;,&quot;66c2a5&quot;,&quot;abdda4&quot;,&quot;e6f598&quot;,<br>    &quot;fee08b&quot;,&quot;fdae61&quot;,&quot;f46d43&quot;,&quot;d53e4f&quot;,&quot;9e0142&quot;,&quot;5e4fa2&quot;<br>]</pre><h4><strong>Displaying the obtained statistics on the map</strong></h4><p><strong>Display terrain statistics</strong></p><pre># Visualization parameters for different layers<br>vis_elev = dict(min=50, max=4000, palette=palette_elev)<br>vis_elev_mean = dict(min=50, max=2000, palette=palette_elev)<br>vis_elev_std = dict(min=0, max=1500, palette=palette_std)<br>vis_elev_variance = dict(min=0, max=3000000, palette=palette_std)<br><br># List of images to display (order corresponds to vis_params and labels)<br>images = [<br>    elevation,<br>    elev_mean_img,<br>    elev_stdDev_img,<br>    elev_variance_img<br>]<br><br># Corresponding visualization parameters for each image in the list above<br>vis_params = [<br>    vis_elev,<br>    vis_elev_mean,<br>    vis_elev_std,<br>    vis_elev_variance<br>]<br><br># Layer labels (shown in each linked map&#39;s title)<br>labels = [<br>    &quot;Elevation&quot;,<br>    &quot;Elev. Mean&quot;,<br>    &quot;Elev. Standard deviation&quot;,<br>    &quot;Elev. Variance&quot;<br>]<br><br># Create 4 linked maps (panning/zoom are synchronized)<br>geemap.linked_maps(<br>    rows=1,                     # number of rows (1 row)<br>    cols=4,                     # number of columns (4 maps side by side)<br>    height=&quot;600px&quot;,             # widget height<br>    center=[30, -85],           # map center [lat, lon]<br>    zoom=2,                     # initial zoom level<br>    ee_objects=images,          # list of ee.Image objects to display<br>    vis_params=vis_params,      # list of visualization parameters (must match length of images)<br>    labels=labels,              # map labels (length must also match)<br>    label_position=&quot;topright&quot;,  # label position on each map<br>)</pre><figure><img alt="Result of Zonal statistis of Elevation layer on Earth Engine (4 interactive maps: source raster, Mean, Standard deviation, Variance)" src="https://cdn-images-1.medium.com/max/1024/1*_oz3P1nOaOMc6lHawo43zw.jpeg" /><figcaption>Zonal statistis of Elevation layer (source raster, Mean, Standard deviation, Variance)</figcaption></figure><p><strong>Display vegetation statistics</strong></p><pre>vis_ndvi = dict(min=0, max=0.8, palette=palette_ndvi)<br>vis_ndvi_mean = dict(min=0, max=0.8, palette=palette_ndvi)<br>vis_ndvi_std = dict(min=0, max=0.25, palette=palette_std)<br>vis_ndvi_variance = dict(min=0, max=0.05, palette=palette_std)<br><br><br>images = [<br>    ndvi,<br>    ndvi_mean_img,<br>    ndvi_stdDev_img,<br>    ndvi_variance_img,<br>]<br><br>vis_params = [<br>    vis_ndvi,<br>    vis_ndvi_mean,<br>    vis_ndvi_std,<br>    vis_ndvi_variance,<br>]<br><br>labels = [<br>    &quot;NDVI&quot;,<br>    &quot;NDVI Mean&quot;,<br>    &quot;NDVI Standard deviation&quot;,<br>    &quot;NDVI Variance&quot;,<br>]<br><br>geemap.linked_maps(<br>    rows=1,<br>    cols=4,<br>    height=&quot;600px&quot;,<br>    center=[30, -85],<br>    zoom=2,<br>    ee_objects=images,<br>    vis_params=vis_params,<br>    labels=labels,<br>    label_position=&quot;topright&quot;,<br>)</pre><figure><img alt="Result of Zonal statistis of NDVI layer on Earth Engine (4 interactive maps: source raster, Mean, Standard deviation, Variance)" src="https://cdn-images-1.medium.com/max/1024/1*D-G-lgdqwrXAN491d3g1nQ.jpeg" /><figcaption>Zonal statistis of NDVI layer (source raster, Mean, Standard deviation, Variance)</figcaption></figure><p><strong>Display temperature statistics</strong></p><pre>vis_temperature = dict(min=0, max=35, palette=palette_temperature)<br>vis_temperature_mean = dict(min=0, max=35, palette=palette_temperature)<br>vis_temperature_std = dict(min=0, max=9, palette=palette_std)<br>vis_temperature_variance = dict(min=0, max=70, palette=palette_std)<br><br>images = [<br>    temperature_c,<br>    temperature_mean_img,<br>    temperature_stdDev_img,<br>    temperature_variance_img<br>]<br><br>vis_params = [<br>    vis_temperature,<br>    vis_temperature_mean,<br>    vis_temperature_std,<br>    vis_temperature_variance<br>]<br><br>labels = [<br>    &quot;Temperature&quot;,<br>    &quot;Temperature Mean&quot;,<br>    &quot;Temperature Standard deviation&quot;,<br>    &quot;Temperature Variance&quot;<br>]<br><br>geemap.linked_maps(<br>    rows=1,<br>    cols=4,<br>    height=&quot;700px&quot;,<br>    center=[30, -85],<br>    zoom=2,<br>    ee_objects=images,<br>    vis_params=vis_params,<br>    labels=labels,<br>    label_position=&quot;topright&quot;,<br>)</pre><figure><img alt="Result of Zonal statistis of Temperature layer on Earth Engine (4 interactive maps: source raster, Mean, Standard deviation, Variance)" src="https://cdn-images-1.medium.com/max/1024/1*Hii4IwXeTxT978wD2b1XGQ.jpeg" /><figcaption>Zonal statistis of Temperature layer (source raster, Mean, Standard deviation, Variance)</figcaption></figure><p><strong>Ready to try?</strong> Copy the code into the <a href="https://colab.research.google.com/">Google Colab</a> or make a copy of this <a href="https://colab.research.google.com/drive/1AnjocWMR7B67CgjomNeIq5NSGSMv4bS8?usp=sharing">Colab notebook</a> and adapt it to your needs.</p><p>If you need help, message me on <a href="https://www.linkedin.com/in/anton-biatov/">LinkedIn</a>. <strong>I am open to collaboration and work opportunities.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7101d1d42ffd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Find Oil & Gas in America? An Overview of a GIS Portal for Identifying Prospective Areas]]></title>
            <link>https://medium.com/@anton.biatov/how-to-find-oil-gas-in-america-an-overview-of-a-gis-portal-for-identifying-prospective-areas-1622b49ac38d?source=rss-9beb986612a------2</link>
            <guid isPermaLink="false">https://medium.com/p/1622b49ac38d</guid>
            <category><![CDATA[mineral-rights]]></category>
            <category><![CDATA[geology]]></category>
            <category><![CDATA[drilling]]></category>
            <category><![CDATA[mineral-exploration]]></category>
            <category><![CDATA[oil-and-gas]]></category>
            <dc:creator><![CDATA[Anton Biatov]]></dc:creator>
            <pubDate>Tue, 30 Sep 2025 08:59:09 GMT</pubDate>
            <atom:updated>2025-09-30T16:14:19.006Z</atom:updated>
            <content:encoded><![CDATA[<p>It’s increasingly important to locate new oil and gas deposits across various U.S. states (Texas, Oklahoma, Arkansas, Michigan, Indiana, etc.). This matters for identifying new exploration zones, evaluating investment potential, planning natural resource use, and understanding the potential value of land parcels. This can be useful for oil and gas companies, venture investors, and private landowners alike.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bd8hLXGf5FMHN1vv-H9rdA.jpeg" /></figure><p>This service provides a report for a specific area of interest. The PDF report includes: an AI-driven forecast of oil and gas deposits with an accuracy assessment, topographic data (relief, slopes, vegetation), and maps &amp; geodata in JPG, GeoTIFF, and KMZ formats.</p><p>To train the neural networks and generate the forecast, the following data were used:</p><ul><li>Data on more than 2 million wells</li><li>Gravity and magnetometry data</li><li>Digital Elevation Models (DEMs)</li><li>Lithology and stratigraphy data</li><li>Landsat satellite imagery</li><li>Soil Moisture Active Passive (SMAP) data</li></ul><p>Below is a description of the steps I took to obtain the hydrocarbon occurrence forecast. I’ll do a separate post later with a breakdown of the delivered report.</p><h4><strong>How to Order a Report</strong></h4><p>First, I went to the website: <a href="https://www.mineralroyalty.ai/">https://www.mineralroyalty.ai</a></p><p>There is also a page where you can read more about this service <a href="https://mineralroyalty.ai/main/">https://mineralroyalty.ai/main/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6D3gbmfBmBtr84ntvatWvA.jpeg" /></figure><p>I zoomed into the area where I needed the forecast. To quickly find a location, you can use the Geosearch widget.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oclD8KML4uN1Nm7rTxOvIA.jpeg" /></figure><p>Next, I traced the area of interest. The portal will display a warning if the area is too large or if you’re trying to select a location where data isn’t available. I’ll describe possible warnings at the end of the article.</p><p>To start digitizing a land parcel, I clicked the corresponding button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_i_zXQuFD-g5WLHNnpd-Vw.jpeg" /></figure><p>While digitizing, the system showed the boundary outline and the perimeter length.</p><p>To finish digitizing, click the first point again or press the “Finish” button.</p><p>You can also delete the last point added or cancel digitizing using the respective buttons.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-8eJbgzX5wlUhhZ4xvY_ew.jpeg" /></figure><p>Once you draw your polygon of interest, the system automatically calculates the area and shows the price for the report covering that polygon.</p><p>To continue, click the “Proceed to Checkout” button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nQYfimMAYQB1S8YFLv6iUA.jpeg" /></figure><p>A “Terms and Conditions” pop-up will appear. Read through it and scroll to the bottom.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W_Nyk_QIAtRyiSWxaWXDSA.jpeg" /></figure><p>After that, the “Continue to Checkout” button becomes active. Click it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eVJRa3oQVJQc-8_e_OyZOQ.jpeg" /></figure><p>You’ll be redirected to Stripe to pay for your order.</p><p>Important: at this step, enter a valid email address — the report will be sent there after it’s generated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lVVPO_WCekVCcKmv8Ldy6w.jpeg" /></figure><p>After payment, you’ll be redirected to a page with your order code. It also states the report will be ready within 72 hours, but I actually received mine within 30 minutes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7dRD--kotzlY1cESo6xgkg.png" /></figure><p>I received an email once the report was ready, with a download link.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wy9rgIZr-HYQ3kTVmBjL9g.png" /></figure><p>The oil and gas prospect forecast report is a pretty interesting document. The PDF included:</p><ul><li>A probability forecast for oil and gas deposits</li><li>A standard deviation map to assess model confidence</li><li>A topographic map (DEM)</li><li>A slope map</li><li>An NDVI map</li><li>Links to download the above maps and geodata in JPG, GeoTIFF, and KMZ formats</li><li>A summary table of oil and gas probabilities with area breakdowns for different probability ranges</li></ul><p>And best of all! <strong>The price is only $50 per report for up to 1,000 acres</strong>.</p><p>I’ll record a detailed breakdown of the report in a separate article in one of the upcoming posts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iuPC_73kYnhNVqvplYR_vA.jpeg" /></figure><h4><strong>Service Limitations</strong></h4><p>The oil and gas forecast currently covers only the contiguous United States. There’s no data for Alaska or for a ~20 km strip along the ocean coastline.</p><p>If you try to draw an area of interest where data isn’t available, the system will show a notice and suggest emailing the service creators.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AcWFMI0XKlgpbcZftV8JzQ.png" /></figure><p>There’s also a size limit. The maximum order area is 1000 acres. If you need a larger area, you can email the service creators — they can manually prepare a report for a larger territory. Potentially, they can generate a forecast for an entire state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rauIa3-C_KmapuyIrs0uMA.jpeg" /></figure><p>The service cannot handle complex polygons. When attempting to draw a highly detailed area, you will see the warning “Selected area has incorrect shape.” This means you’ve drawn too many vertices in the polygon. I recommend drawing simple shapes with up to 12 nodes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aIBdpyRymp7UHQbv22guZQ.jpeg" /></figure><p>Overall, I find the service very useful for the following user groups:</p><ol><li><strong>Oil &amp; gas companies</strong> — to identify new exploration zones</li><li><strong>Geological exploration contractors / consulting firms</strong> — as part of comprehensive studies</li><li><strong>Investment funds / venture investors </strong>— to assess investment attractiveness</li><li><strong>Government and municipal agencies </strong>— for natural-resource planning and assessment</li><li><strong>Landowners / private owners</strong> — to understand the potential value of their parcels</li><li><strong>Legal and brokerage firms</strong> — to support land-asset transactions</li></ol><p>I’ll publish the detailed report walkthrough in a separate article in one of the next posts.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1622b49ac38d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Combine Topography and Vegetation: A Practical Guide to Zonal Statistics in Google Earth…]]></title>
            <link>https://medium.com/@anton.biatov/how-to-combine-topography-and-vegetation-a-practical-guide-to-zonal-statistics-in-google-earth-d8167615937b?source=rss-9beb986612a------2</link>
            <guid isPermaLink="false">https://medium.com/p/d8167615937b</guid>
            <category><![CDATA[google-earth-engine]]></category>
            <category><![CDATA[gis]]></category>
            <category><![CDATA[remote-sensing]]></category>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Anton Biatov]]></dc:creator>
            <pubDate>Sun, 28 Sep 2025 13:54:21 GMT</pubDate>
            <atom:updated>2026-09-24T15:55:59.705Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>How to Combine Topography and Vegetation: A Practical Guide to Zonal Statistics in Google Earth Engine</strong></h3><p><em>A step-by-step guide to building an efficient tool for spatial data analysis</em></p><figure><img alt="Comparing the “NDVI” raster and the computed “ndvi_mean” value — zonal staristics in Google Earth Engine" src="https://cdn-images-1.medium.com/max/1024/1*xIlSHXUBqBMLBV5jnl-WbQ.jpeg" /><figcaption>Comparing the “NDVI” raster and the computed “ndvi_mean” value on Earth Engine viewer</figcaption></figure><p>In the era of big data and climate change, the ability to quickly analyze massive volumes of geospatial information is becoming critically important. In this post, I’ll show how to use Google Earth Engine to build a powerful tool for analyzing the relationship between topography and vegetation cover at the level of administrative units. Computing zonal statistics across multiple parameters (Mean, Median, Standard Deviation, Variance, Minimum, and Maximum) for several raster layers at once is exactly such a tool.</p><h3><strong>What will we do?</strong></h3><p>We’ll create a script that:</p><ul><li>Loads terrain data from <a href="https://developers.google.com/earth-engine/datasets/catalog/NOAA_NGDC_ETOPO1">NOAA ETOPO1</a> and vegetation index data <a href="https://developers.google.com/earth-engine/datasets/catalog/MODIS_061_MOD13A3">NDVI (MODIS)</a> for 2024</li><li>Computes zonal statistics for each U.S. county (GADM level 2)</li><li>Visualizes the results on an interactive map</li><li>Exports the data for further analysis</li></ul><p>To start, download the vector layer of administrative boundaries from <a href="https://gadm.org/download_country.html">gadm.org</a> and upload it to Earth Engine Assets.</p><h3><strong>Key Solution Architecture</strong></h3><h4><strong>1. Optimized computation</strong></h4><p>A key feature of our approach is the use of a composite reducer:</p><pre>var reducers = ee.Reducer.mean()<br>  .combine({reducer2: ee.Reducer.median(),   sharedInputs: true})<br>  .combine({reducer2: ee.Reducer.stdDev(),   sharedInputs: true})<br>  .combine({reducer2: ee.Reducer.variance(), sharedInputs: true})<br>  .combine({reducer2: ee.Reducer.min(),      sharedInputs: true})<br>  .combine({reducer2: ee.Reducer.max(),      sharedInputs: true});</pre><p>The `<em>sharedInputs: true</em>` parameter lets all reducers use the same inputs, which significantly speeds up computation when working with thousands of polygons.</p><h4><strong>2. Smart data stacking</strong></h4><p>Instead of processing each input raster layer separately, we create a single stack:</p><pre>var imageToProcess = elevation<br>  .addBands(ndvi)<br>  .rename([&#39;elevation&#39;, &#39;ndvi&#39;]);</pre><p>This allows us to apply all statistical operations to both layers at once, ensuring result consistency.</p><h3><strong>Full script example</strong></h3><p>The code is fully documented and can be easily adapted for different tasks:</p><pre>/**********************************************************<br>* Composite reducer: compute mean, median, stdDev, variance,<br>* min, max in a single pass.<br>* combine(sharedInputs: true) — all reducers receive the<br>* same input (speeds up processing).<br>**********************************************************/<br>var reducers = ee.Reducer.mean()<br>  .combine({reducer2: ee.Reducer.median(),   sharedInputs: true})<br>  .combine({reducer2: ee.Reducer.stdDev(),   sharedInputs: true})<br>  .combine({reducer2: ee.Reducer.variance(), sharedInputs: true})<br>  .combine({reducer2: ee.Reducer.min(),      sharedInputs: true})<br>  .combine({reducer2: ee.Reducer.max(),      sharedInputs: true});<br><br>/**********************************************************<br> * INPUT DATA<br> **********************************************************/<br><br> // Terrain elevation: ETOPO1 (NOAA/NGDC).<br> // ee.Image loads a raster; select picks the desired band.<br> var elevDataset = ee.Image(&#39;NOAA/NGDC/ETOPO1&#39;);<br> var elevation   = elevDataset.select(&#39;bedrock&#39;);<br><br> // MODIS NDVI (monthly composites, collection MOD13A3, v061).<br> // filterDate limits the period; mean() averages over 2024.<br> var viDataset = ee.ImageCollection(&#39;MODIS/061/MOD13A3&#39;)<br>   .filterDate(&#39;2024-01-01&#39;, &#39;2024-12-31&#39;)<br>   .mean();<br> var ndvi = viDataset.select(&#39;NDVI&#39;);<br><br> /**********************************************************<br>  * BUILD A SINGLE IMAGE<br>  * addBands merges bands from different images.<br>  * rename gives clear band names in the final stack.<br>  **********************************************************/<br> var imageToProcess = elevation<br>   .addBands(ndvi)<br>   .rename([&#39;elevation&#39;, &#39;ndvi&#39;]);<br><br> /**********************************************************<br>  * VECTOR POLYGONS (YOUR ASSETS)<br>  * Specify the path to your polygon vector layer in Assets.<br>  * FeatureCollection — a set of geometries with attributes.<br>  **********************************************************/<br> var remotePolygons = ee.FeatureCollection(&#39;projects/gee-antonbiatov/assets/gadm41_USA_2&#39;);<br><br> /**********************************************************<br>  * ZONAL STATISTICS<br>  * reduceRegions applies the reducer to all features<br>  * in the collection.<br>  * - collection: which polygons to analyze<br>  * - reducer: what to compute (our composite reducer)<br>  * - scale: nominal spatial resolution, m/pixel<br>  * Output: a FeatureCollection where attributes of features<br>  * are augmented with computed fields, e.g., elevation_mean,<br>  * ndvi_max, etc.<br>  **********************************************************/<br> var statsResults = imageToProcess.reduceRegions({<br>   collection: remotePolygons,<br>   reducer: reducers,<br>   scale: 1000<br> });<br><br> // Quick preview of the first features in the Console<br> // (helps verify the fields).<br> print(&#39;Sample result:&#39;, statsResults.limit(5));<br><br> /*****************<br>  * VISUALIZATION:<br>  *****************/<br><br> /**********************************************************<br>  * RASTER STYLES<br>  * min/max — stretch range; palette — color ramp.<br>  **********************************************************/<br> var elevVis = {<br>   min: 0,<br>   max: 4000,<br>   palette: [<br>    &#39;3ae237&#39;, &#39;b5e22e&#39;, &#39;d6e21f&#39;, &#39;fff705&#39;, &#39;ffd611&#39;, &#39;ffb613&#39;, &#39;ff8b13&#39;,<br>    &#39;ff6e08&#39;, &#39;ff500d&#39;, &#39;ff0000&#39;, &#39;de0101&#39;, &#39;c21301&#39;, &#39;0602ff&#39;, &#39;235cb1&#39;,<br>    &#39;307ef3&#39;, &#39;269db1&#39;, &#39;30c8e2&#39;, &#39;32d3ef&#39;, &#39;3be285&#39;, &#39;3ff38f&#39;, &#39;86e26f&#39;<br>  ]<br> };<br><br> var ndviVis = {<br>   min: 0,<br>   max: 9000, // MOD13A3 stores NDVI scaled by 1e4<br>   palette: [<br>     &#39;ffffff&#39;, &#39;ce7e45&#39;, &#39;df923d&#39;, &#39;f1b555&#39;, &#39;fcd163&#39;, &#39;99b718&#39;, &#39;74a901&#39;,<br>     &#39;66a000&#39;, &#39;529400&#39;, &#39;3e8601&#39;, &#39;207401&#39;, &#39;056201&#39;, &#39;004c00&#39;, &#39;023b01&#39;,<br>     &#39;012e01&#39;, &#39;011d01&#39;, &#39;011301&#39;<br>   ],<br> };<br><br> /**********************************************************<br>  * RASTERIZING MEAN FIELDS FROM THE VECTOR — for web-map<br>  * visualization. reduceToImage converts a feature attribute<br>  * (field) to an image, “painting” each polygon with the<br>  * corresponding value. Reducer.first() takes the value from<br>  * the feature.<br>  **********************************************************/<br><br> // Rasterize elevation_mean, rename the band for convenience.<br> var elevMeanImg = statsResults.reduceToImage({<br>   properties: [&#39;elevation_mean&#39;],<br>   reducer: ee.Reducer.first()<br> }).rename(&#39;elevation&#39;);<br><br> // Rasterize ndvi_mean, rename the band.<br> var ndviMeanImg = statsResults.reduceToImage({<br>   properties: [&#39;ndvi_mean&#39;],<br>   reducer: ee.Reducer.first()<br> }).rename(&#39;ndvi&#39;);<br><br> /**********************************************************<br>  * MAP CAMERA &amp; LAYERS<br>  **********************************************************/<br><br> // Manually center the map over the USA (longitude, latitude, zoom).<br> Map.setCenter(-104.0, 40.0, 4);<br><br> // Add raw 2024 NDVI (from MODIS) — for context.<br> Map.addLayer(ndvi, ndviVis, &#39;NDVI (scaled)&#39;);<br><br> // Add the rasterized ndvi_mean derived from vector stats.<br> Map.addLayer(ndviMeanImg.select(&#39;ndvi&#39;), ndviVis, &#39;ndvi_mean (rasterized)&#39;);<br><br> // Add raw ETOPO1 elevation — for context.<br> Map.addLayer(elevation, elevVis, &#39;Elevation&#39;);<br><br> // Add the rasterized elevation_mean by polygons (zonal stats).<br> Map.addLayer(elevMeanImg.select(&#39;elevation&#39;), elevVis, &#39;elevation_mean (rasterized)&#39;);<br><br> // (Optional) polygon outlines on top — convenient for visual comparison.<br> // style with fillColor: &#39;00000000&#39; makes the fill transparent;<br> // color/width control outline color and line width.<br> var outlines = statsResults.style({color: &#39;000000&#39;, width: 1, fillColor: &#39;00000000&#39;});<br> Map.addLayer(outlines, {}, &#39;Polygons outline&#39;);<br><br> /**********************************************************<br>  * EXPORT RESULTS TO GOOGLE DRIVE<br>  * Export.table.toDrive saves the FeatureCollection (vector)<br>  * to the specified format and folder on Google Drive.<br>  * Launch the export manually from the Tasks panel in the UI.<br>  **********************************************************/<br> Export.table.toDrive({<br>   collection: statsResults,                       // what to export<br>   description: &#39;GADM_United_States_Level_2_statistics_JS&#39;, // output file name<br>   folder: &#39;GEE_Exports&#39;,                          // Google Drive folder<br>   fileFormat: &#39;GeoJSON&#39;                           // CSV, SHP, GeoJSON, KML, KMZ<br> });</pre><p>Before exporting, we can preview the results in the Console using:</p><pre>print(&#39;Sample result:&#39;, statsResults.limit(5));</pre><figure><img alt="Sample of zonal statistics results in Google Earth Engine" src="https://cdn-images-1.medium.com/max/1024/1*DLkFZF4ggqsoXadkizKXmA.png" /><figcaption>Sample of zonal statistics results</figcaption></figure><p>We can also see part of the zonal statistics on the web map.</p><figure><img alt="result of zonal statistics on the web map in google earth engine" src="https://cdn-images-1.medium.com/max/1024/1*G4KT6-d3yWXTn4UrXFURtQ.jpeg" /><figcaption>Result of zonal statistics on the web map in Google Earth Engine</figcaption></figure><h3><strong>Technical advantages of the approach</strong></h3><h4><strong>Scalability</strong></h4><p>Google Earth Engine processes petabytes of data in the cloud. My script is easily adapted for:</p><ul><li>Other countries and regions</li><li>Different time periods</li><li>Additional parameters (temperature, precipitation, soils etc.)</li></ul><h4><strong>Reproducibility</strong></h4><p>The code is fully documented and can be easily modified for various tasks:</p><pre>// Specify the path to your polygon vector layer in Assets<br>var remotePolygons = ee.FeatureCollection(&#39;projects/your-project/assets/your-boundaries&#39;);<br><br>// Change the time period<br>var viDataset = ee.ImageCollection(&#39;MODIS/061/MOD13A3&#39;)<br>  .filterDate(&#39;2023-01-01&#39;, &#39;2023-12-31&#39;);</pre><p>You can also add other raster data sources, for example <a href="https://developers.google.com/earth-engine/datasets/catalog/ECMWF_ERA5_LAND_MONTHLY_BY_HOUR">temperature</a>.</p><h4>Integration with analytics pipelines</h4><p>Exporting to GeoJSON ensures compatibility with:</p><ul><li>Python (GeoPandas, Matplotlib)</li><li>R (sf, ggplot2)</li><li>QGIS and other GIS systems</li><li>Web-mapping solutions</li></ul><h3><strong>Practical applications</strong></h3><p>This specific script can be useful in the following sectors.</p><h4><strong>Environmental monitoring</strong></h4><p>Combining elevation and vegetation data allows you to:</p><ul><li>Identify anomalies in vegetation cover</li><li>Monitor ecosystem degradation</li><li>Plan restoration activities</li></ul><p><strong>Use case:</strong> Detecting regions where expected (by elevation) and actual vegetation diverge can indicate anthropogenic impact or early stages of desertification.</p><h4><strong>Government planning</strong></h4><p>Administrative units receive quantitative characterizations of their territories:</p><ul><li>Average elevation</li><li>Terrain variability (standard deviation of elevations)</li><li>“Greenness” index of the area</li></ul><p><strong>Practical case:</strong> Regions with high elevation variability (mountainous) may require special approaches to infrastructure planning.</p><h4><strong>Agro-ecological zoning</strong></h4><p>The data helps identify:</p><ul><li>Potential land productivity</li><li>Risks to agriculture</li><li>Optimal zones for different crops</li></ul><h3><strong>Conclusion</strong></h3><p>This approach demonstrates the power of modern cloud platforms for geospatial analysis. Just a few dozen lines of code let you:</p><ul><li>Process terabytes of satellite data</li><li>Obtain statistically meaningful results for thousands of regions</li><li>Create a ready-to-use dataset for further research</li></ul><p>Google Earth Engine democratizes access to complex geospatial analyses, making them available not only to large research centers but also to small teams and individual researchers.</p><p><strong>Ready to try?</strong> Copy the code into the <a href="https://code.earthengine.google.com/">Google Earth Engine Code Editor</a> and adapt it to your needs.</p><p>If you find this guide helpful, Let’s connect on <a href="https://www.linkedin.com/in/anton-biatov/">LinkedIn</a> and follow my blog on <a href="https://medium.com/@anton.biatov">Medium</a></p><p>If you need help, message me on <a href="https://www.linkedin.com/in/anton-biatov/">LinkedIn</a>. <strong>I am open to collaboration and work opportunities.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d8167615937b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>