<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Chinmay Deval on Medium]]></title>
        <description><![CDATA[Stories by Chinmay Deval on Medium]]></description>
        <link>https://medium.com/@chinmaydeval?source=rss-35d240104029------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*DOlYeYr9w5sb7kVkqb6Ajg.png</url>
            <title>Stories by Chinmay Deval on Medium</title>
            <link>https://medium.com/@chinmaydeval?source=rss-35d240104029------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 06:22:51 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@chinmaydeval/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[15x Faster Geospatial Pipelines: Why I Swapped Pandas for DuckDB]]></title>
            <link>https://medium.com/@chinmaydeval/15x-faster-geospatial-pipelines-why-i-swapped-pandas-for-duckdb-ff6e7cc814f4?source=rss-35d240104029------2</link>
            <guid isPermaLink="false">https://medium.com/p/ff6e7cc814f4</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[geospatial]]></category>
            <dc:creator><![CDATA[Chinmay Deval]]></dc:creator>
            <pubDate>Thu, 02 Apr 2026 05:49:05 GMT</pubDate>
            <atom:updated>2026-04-02T05:49:05.405Z</atom:updated>
            <content:encoded><![CDATA[<h4>Accelerating geospatial data pipelines with DuckDB’s native SQL spatial joins</h4><h3>⚡ TL;DR</h3><ul><li><strong>The Task:</strong> Map 73M+ FEMA NFIP policy records and 3M+ claims (since 1978) to U.S. county boundaries</li><li><strong>The Bottleneck:</strong> A Pandas/GeoPandas pipeline that took 10.5s and consumed significant memory</li><li><strong>The Fix:</strong> A single DuckDB SQL query using the spatial extension</li><li><strong>The Result:</strong> <strong>0.69 seconds (15.2x faster)</strong></li></ul><p>This post walks through how I optimized a real-world FEMA geospatial pipeline using DuckDB and why it significantly outperforms Pandas for large-scale spatial joins.</p><figure><img alt="Infographic: Benchmarking the 15.2x Speedup from Pandas to DuckDB. (Generated using Gemini.)" src="https://cdn-images-1.medium.com/max/1024/1*rLgzPRFbn_eR_nZd3q5KTQ.png" /><figcaption>Infographic: Benchmarking the 15.2x Speedup from Pandas to DuckDB. (Generated using Gemini.)</figcaption></figure><p>In geospatial data workflows, <em>waiting</em> often feels like part of the job.</p><p>Kick off a heavy join.<br>Watch the kernel spin.<br>Maybe grab a coffee — and hope nothing crashes.</p><p>While analyzing FEMA’s National Flood Insurance Program (NFIP) dataset— spanning nearly 50 years of flood insurance data across the United States — I hit exactly that wall.</p><p>I needed to:</p><ul><li>Aggregate <strong>2.7 million claims</strong></li><li>Combine <strong>72.5 million policies</strong></li><li>Map everything to <strong>U.S. county shapefiles (Census TIGER/Line)</strong></li></ul><p>Normally, I’d reach for Pandas + GeoPandas without thinking twice.</p><p>This time, I didn’t.</p><p>Instead, I pushed everything into <strong>DuckDB</strong> — and it changed everything.</p><h3>📊 The Benchmark</h3><p>I ran a strict head-to-head comparison:</p><ul><li>Timer starts before reading data</li><li>Timer stops after the final GeoDataFrame is ready</li><li>Pandas version fully optimized (column pruning, memory cleanup)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0NiN92tbuZgpz5X9RexEUg.png" /></figure><p>Even with careful tuning, Pandas just couldn’t keep up.</p><h3>The “Before”: Pandas Bottleneck</h3><p>Here’s what the typical workflow looked like:</p><pre>import duckdb<br>import pandas as pd<br>import geopandas as gpd<br><br># 1. Read files into RAM<br>con = duckdb.connect()<br>claims_df = con.execute(&quot;&quot;&quot;<br>    SELECT countyCode, netBuildingPaymentAmount <br>    FROM read_parquet(&#39;../data/FimaNfipClaimsV2.parquet&#39;)<br>&quot;&quot;&quot;).df()<br><br>policies_df = con.execute(&quot;&quot;&quot;<br>    SELECT countyCode, policyCount, totalBuildingInsuranceCoverage <br>    FROM read_parquet(&#39;../data/FimaNfipPoliciesV2.parquet&#39;)<br>&quot;&quot;&quot;).df()<br>con.close()<br><br># 2. Aggregate millions of rows (Single-threaded)<br>claims_agg = (claims_df.dropna(subset=[&#39;countyCode&#39;])<br>              .groupby(&#39;countyCode&#39;)<br>              .agg(avg_claim_amt=(&#39;netBuildingPaymentAmount&#39;, &#39;sum&#39;))<br>              .reset_index())<br><br>policies_agg = (policies_df.dropna(subset=[&#39;countyCode&#39;])<br>                .groupby(&#39;countyCode&#39;)<br>                .agg(total_policies=(&#39;policyCount&#39;, &#39;sum&#39;), <br>                     avg_coverage=(&#39;totalBuildingInsuranceCoverage&#39;, &#39;sum&#39;))<br>                .reset_index())<br><br># 3. Clean and Join<br>agg_df = pd.merge(claims_agg, policies_agg, on=&#39;countyCode&#39;, how=&#39;outer&#39;)<br>agg_df = agg_df.rename(columns={&#39;countyCode&#39;: &#39;GEOID&#39;})<br>agg_df[&#39;GEOID&#39;] = agg_df[&#39;GEOID&#39;].astype(str).str.zfill(5)<br><br># 4. Spatial Merge<br>counties = gpd.read_file(&#39;../data/tl_2025_us_county/tl_2025_us_county.shp&#39;)<br>counties = counties[[&#39;GEOID&#39;, &#39;STATEFP&#39;, &#39;geometry&#39;]]<br>merged_gdf = counties.merge(agg_df, on=&#39;GEOID&#39;, how=&#39;left&#39;)</pre><h3>What’s going wrong?</h3><ul><li>Millions of rows pulled into memory</li><li>Single-threaded groupby operations</li><li>Manual column cleanup</li><li>GeoPandas doing joins in Python</li></ul><p>It works — but it doesn’t scale.</p><h3>The “After”: All-in-DuckDB</h3><p>Instead of bouncing between tools, I let DuckDB handle everything:</p><pre>import duckdb<br>import pandas as pd<br>import geopandas as gpd<br><br>def load_and_join_all_in_duckdb():<br>    con = duckdb.connect()<br>    con.execute(&quot;INSTALL spatial; LOAD spatial;&quot;)<br>    <br>    # Unified SQL query for aggregation and spatial join<br>    query = &quot;&quot;&quot;<br>    WITH aggregated_claims AS (<br>        SELECT <br>            lpad(cast(countyCode AS VARCHAR), 5, &#39;0&#39;) AS GEOID,<br>            sum(netBuildingPaymentAmount)             AS avg_claim_amt<br>        FROM read_parquet(&#39;../data/FimaNfipClaimsV2.parquet&#39;)<br>        WHERE countyCode IS NOT NULL<br>        GROUP BY 1<br>    ),<br>    aggregated_policies AS (<br>        SELECT <br>            lpad(cast(countyCode AS VARCHAR), 5, &#39;0&#39;) AS GEOID,<br>            sum(policyCount)                          AS total_policies,<br>            sum(totalBuildingInsuranceCoverage)       AS avg_coverage<br>        FROM read_parquet(&#39;../data/FimaNfipPoliciesV2.parquet&#39;)<br>        WHERE countyCode IS NOT NULL<br>        GROUP BY 1<br>    ),<br>    counties_shapefile AS (<br>        SELECT <br>            GEOID, <br>            STATEFP,<br>            ST_AsWKB(geom) AS geometry <br>        FROM ST_Read(&#39;../data/tl_2025_us_county/tl_2025_us_county.shp&#39;)<br>    )<br>    SELECT <br>        s.GEOID, s.STATEFP, p.total_policies, <br>        p.avg_coverage, c.avg_claim_amt, s.geometry<br>    FROM counties_shapefile s<br>    LEFT JOIN aggregated_policies p ON s.GEOID = p.GEOID<br>    LEFT JOIN aggregated_claims c   ON s.GEOID = c.GEOID<br>    &quot;&quot;&quot;<br>    <br>    df = con.execute(query).df()<br>    con.close()<br>    <br>    # Cast bytearray to strict bytes for Shapely compatibility<br>    df[&#39;geometry&#39;] = df[&#39;geometry&#39;].apply(bytes)<br>    <br>    # Final Handoff to GeoPandas<br>    df[&#39;geometry&#39;] = gpd.GeoSeries.from_wkb(df[&#39;geometry&#39;])<br>    return gpd.GeoDataFrame(df, geometry=&#39;geometry&#39;, crs=&quot;EPSG:4269&quot;)<br><br>if __name__ == &quot;__main__&quot;:<br>    gdf = load_and_join_all_in_duckdb()</pre><h3>What Changed (And Why It Matters)</h3><p>The speedup wasn’t just about switching tools — it came from changing <em>where</em> the work happens.</p><h3>1. No Python Overhead</h3><p>The biggest win? The raw datasets never enter Python memory.</p><p>Instead of pulling tens of millions of rows into Pandas, DuckDB does all the heavy lifting internally — and only returns the final, compact result.</p><h3>2. Native Spatial + SQL Execution</h3><p>Everything runs in one place:</p><ul><li>Aggregations happen in SQL</li><li>Joins happen in SQL</li><li>Even shapefiles are read directly with ST_Read()</li></ul><p>No handoffs. No intermediate DataFrames. No extra overhead.</p><h3>3. A Cleaner Data Model</h3><p>Because everything is defined in a single SQL query:</p><ul><li>No column duplication</li><li>No more left_on / right_on gymnastics during joins</li><li>No post-merge cleanup</li></ul><p>What you select is exactly what you get.</p><h3>4. Fast Geometry Transfer (The WKB Trick)</h3><p>Geospatial data is usually expensive to move around.</p><p>Instead of passing full geometry objects, DuckDB serializes them as <strong>WKB (Well-Known Binary)</strong>:</p><ul><li>DuckDB → sends compact binary</li><li>GeoPandas → reconstructs geometry instantly</li></ul><p>This makes the Python handoff surprisingly fast.</p><h3>⚠️ Small Gotcha</h3><p>DuckDB returns geometry as a bytearray, but GeoPandas expects bytes. Easy fix—but easy to miss. A simple fix:</p><pre>df[&#39;geometry&#39;] = df[&#39;geometry&#39;].apply(bytes)</pre><h3>💡 The Takeaway</h3><p>Moving your join logic out of Pandas and into native SQL is one of the highest-ROI changes you can make for large geospatial datasets. It ensures clean data filtering at the source and provides a massive performance leap. You’ll get:</p><ul><li>⚡ Faster execution</li><li>🧠 Lower memory usage</li><li>🧹 Cleaner, more maintainable code</li><li>📈 Better scalability as your data grows</li></ul><p>And honestly… fewer coffee breaks waiting for your code to finish ☕</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ff6e7cc814f4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Turning Data into Action: Three Ways SERVIR is Strengthening Water Security]]></title>
            <link>https://medium.com/@chinmaydeval/turning-data-into-action-three-ways-servir-is-strengthening-water-security-a357e6ee1c04?source=rss-35d240104029------2</link>
            <guid isPermaLink="false">https://medium.com/p/a357e6ee1c04</guid>
            <category><![CDATA[drought]]></category>
            <category><![CDATA[water-quality]]></category>
            <category><![CDATA[earth-observation]]></category>
            <category><![CDATA[floods]]></category>
            <category><![CDATA[water]]></category>
            <dc:creator><![CDATA[Chinmay Deval]]></dc:creator>
            <pubDate>Fri, 22 Mar 2024 12:00:24 GMT</pubDate>
            <atom:updated>2024-05-31T01:59:40.712Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QqCBd7mTab_d8fGYQpFJvw.jpeg" /><figcaption>Photo credit: Government of South Africa/DIRCO</figcaption></figure><p>As the world grapples with water scarcity, deteriorating water quality, and increasingly frequent extreme weather events, the importance of timely and accurate Earth observation data in decision-making cannot be overstated.</p><p>Imagine this: there’s a huge storm brewing out at sea and advancing towards land. But there are no satellites that are observing the Earth and collecting the data that is needed to inform communities of the impending disaster. How would at-risk communities prepare or respond to impending floods? Without timely early warning data, vulnerable people would be caught off guard when the storm strikes.</p><p>Beyond disasters, we still increasingly rely on water data for our day-to-day lives. Water data can help farmers know how much groundwater is available to them, it can help pastoralists guide their herds to water bodies, and it can help communities know if their water is safe to drink.</p><p>Disaster preparedness and the water resource management require reliable and timely information. In many regions of the world, ground observation data is scarce. For example, a country might have limited access to local weather stations or gauges to measure the water level of a river. Even if this data is available, there may be gaps in coverage, both in terms of geography and historical record. This also reduces the ability of local communities to make proactive, informed water-related decisions. Earth observation data, coupled with sophisticated models, can bridge this gap as a crucial source of information, especially in data-scarce regions.</p><p>Here are three ways that SERVIR’s innovative services and tools enable decision-makers and authorities to address water challenges:</p><h3>When water is scarce: locating water during the dry season</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XpJSOK56atNJDWF0.jpg" /><figcaption>Photo credit: CSE</figcaption></figure><p>In the vast Ferlo region of north-central Senegal, dry savannah dominates and pastoralism is central to local livelihoods. The dry season in this region can last up to nine months, which means that finding water is critical for sustaining livestock. Traditional reliance on historical knowledge to locate water sources during the dry season has become less dependable due to shifting weather patterns. To address this challenge, SERVIR West Africa and Senegal’s Center for Ecological Monitoring (CSE) developed a service that uses satellite imagery from NASA and the European Space Agency to detect seasonal watering holes. This information is broadcasted via text and radio, which herders can use to direct their livestock to water.</p><h3>When there is too much water: flood early warning systems</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9p0QsOpQsYZCRGL6Os9hHQ.png" /><figcaption>Photo credit: Arjan van de Merwe/UNDP</figcaption></figure><p>The devastating impact of Cyclone Idai that hit East Africa in 2019 underscored the critical need for robust early warning systems in Malawi. The absence of adequate weather forecasting and alert systems exacerbated the disaster, resulting in 1,500 deaths and 2,300 missing persons. In response, Malawi’s government collaborated with SERVIR to boost its flood warning systems, creating the Community-Based Flood Early Warning System (CBFEWS).</p><p>Led by SERVIR-supported scientists in the region, CBFEWS extended local flood warning time from only a few hours to days. When two large cyclones struck Malawi in 2022, these flood warnings saved the country an estimated $40 million in losses. With the support of various international organizations, including the United Nations Development Programme (UNDP) and the World Bank, CBFEWS is now set to expand to cover all of Malawi. Through collaborative climate adaptation efforts, water Earth observation data can save lives and protect livelihoods in flood-prone areas.</p><h3>When water quality is unsafe: monitoring algal blooms</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/1*kWL8EuzFrxufU9y7DbJ1TQ.png" /><figcaption>Photo Credit: SERVIR/Africa Flores</figcaption></figure><p>Lake Atitlán in Guatemala is celebrated as one of the world’s beautiful lakes. It also faces significant threats, particularly from massive algal blooms. An algal bloom can kill off the plants and animals that live in water and lead to the growth of toxic bacteria, threatening local drinking water supplies. With the support of SERVIR, lake managers now employ the Lake Atitlán Forecasting System, a web tool that uses NASA Earth Science data to predict the necessary conditions for algal blooms, such as weather and water runoff forecasts. These predictions are used to trigger on-the-ground water quality testing for timely interventions. This forecasting that combines space-based data with local water sampling is crucial for safeguarding the lake’s ecology and the communities that rely on it.</p><p><a href="https://servirglobal.net/what-we-do/water-security">Click here</a> to learn more about SERVIR’s water security efforts.</p><p><em>Originally published at </em><a href="https://servirglobal.net/news/turning-data-action-three-ways-servir-strengthening-water-security"><em>https://servirglobal.net</em></a><em> on March 22, 2024.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a357e6ee1c04" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Extracting Time Series using Google Earth Engine and JavaScript]]></title>
            <link>https://medium.com/@chinmaydeval/extracting-time-series-using-google-earth-engine-and-javascript-931b7f23340a?source=rss-35d240104029------2</link>
            <guid isPermaLink="false">https://medium.com/p/931b7f23340a</guid>
            <category><![CDATA[google-earth-engine]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[javascript]]></category>
            <category><![CDATA[geospatial]]></category>
            <dc:creator><![CDATA[Chinmay Deval]]></dc:creator>
            <pubDate>Fri, 23 Dec 2022 01:27:41 GMT</pubDate>
            <atom:updated>2022-12-23T22:31:36.841Z</atom:updated>
            <content:encoded><![CDATA[<h4>-Sentinel NDVI data as examples</h4><h3>TL;DR</h3><ul><li>Add your Shapefile to Assets</li><li>Use Google Earth Engine’s JavaScript API</li><li>Create a Normalized Difference Vegetation Index (NDVI) from Sentinel data</li><li>Extract and visualize time series in the Earth Engine code editor console</li><li>Extract and export a large time-series data to a CSV</li></ul><p>When working with geospatial data we are often interested in extracting time series for a set of locations. These, for instance, could be several points or polygons representing different locations. In this short blog, we will use JavaScript in Google Earth Engine’s code editor and extract time series over a set of polygons. We will use Sentinel-2 data to extract the Normalized Difference Vegetation Index (NDVI) for a set of irrigated fields. Let’s begin by bringing our polygon data to the google earth engine.</p><h3>Adding polygon shapefile to assets</h3><p>To upload a shapefile as an asset on Google Earth Engine, navigate to Assets in the left sidebar, click New, and select Shapefiles in the drop-down. A window to upload the shapefile will pop up as shown in the screenshot below.</p><figure><img alt="Workflow for uploading shapefile to google earth engine." src="https://cdn-images-1.medium.com/max/1024/0*8HXOieUlsnO_iq9X.PNG" /></figure><p>Select the file from your computer to be uploaded and click upload. In this case, the file is called mgck_shp.zip. You will be able to see the upload progress in the Tasks sidebar on the right (see screenshot below) and when the upload is complete the shapefile will appear under the CLOUD ASSETS on the left sidebar (highlighted in yellow in this case).</p><figure><img alt="Uploaded shapefile appears in the assets." src="https://cdn-images-1.medium.com/max/1024/0*j3dvTqdUx-UiIDvt.PNG" /></figure><h3>Import and visualize FeatureCollection</h3><p>The shapefile that was just added to the Assets can be loaded and displayed on the map using the following code chunk:</p><pre>//Import GEE Feature Collection (field geometry) <br>var geometry =ee.FeatureCollection(&quot;projects/ee-chinmaydeval91/assets/mgck_shp&quot;); <br><br>//Display polygons on a map <br>Map.addLayer(geometry); Map.centerObject(geometry,14);</pre><p>This displays on the map eight irrigated agriculture fields in the Magic Valley, Idaho.</p><figure><img alt="Display the imported feature collection on the Map using the javascript API for google earth engine." src="https://cdn-images-1.medium.com/max/1024/0*oYaymfUXh75FBQDw.PNG" /></figure><h3>Create and extract Sentinel NDVI</h3><p>Unlike MODIS NDVI composite products, to my knowledge, there is no such pre-computed NDVI product for Sentinel on GEE. So we will have to create the NDVI first.</p><h4>Import image collection</h4><p>Let’s start by importing the Sentinel-2 surface reflectance image collection by typing the following code:</p><pre>// Import image collection of Sentinel-2 imagery, and <br>// filter by start and end date, and <br>// filter by boundary extent <br><br>var S2 = ee.ImageCollection(&#39;COPERNICUS/S2&#39;) <br>.filterDate(&#39;2019-03-28&#39;, &#39;2021-12-31&#39;) <br>.filterBounds(geometry);</pre><p>This imports the Sentinel-2 collections and filters them for the given date range as well as the bounds of the feature collection. Next, we need to mask out the cloudy and snowy images.</p><h4>Mask Clouds and Snow</h4><p>I came across this maskS2sr function on <a href="https://un-spider.org/sites/default/files/Code_GEE_burn_sev.txt">UN Spider’s Burn Severity mapping tutorial</a> that is written to mask out clouds and snow. I will be using it here:</p><pre>// Function to mask clouds from the pixel quality band of Sentinel-2 SR data. <br><br>function maskS2sr(image) {<br>  // Bits 10 and 11 are clouds and cirrus, respectively.<br>  var cloudBitMask = ee.Number(2).pow(10).int();<br>  var cirrusBitMask = ee.Number(2).pow(11).int();<br>  // Get the pixel QA band.<br>  var qa = image.select(&#39;QA60&#39;);<br>  // All flags should be set to zero, indicating clear conditions.<br>  var mask = qa.bitwiseAnd(cloudBitMask).eq(0)<br>      .and(qa.bitwiseAnd(cirrusBitMask).eq(0));<br>  // Return the masked image, scaled to TOA reflectance, without the QA bands.<br>  return image.updateMask(mask)<br>      .copyProperties(image, [&quot;system:time_start&quot;]);<br>}<br><br>// mask clouds and snow across the image collection<br>var S2 = S2.map(maskS2sr)</pre><p>Now we have the Sentinel-2 image collection that is cloud and snow-free and is filtered for the given spatial and temporal bounds. We can even clip the image collection to these irrigated fields.</p><h4>Clip to irrigated field polygons</h4><p>To clip the image by polygon we can use the following clipImgCollect function and then map it across the entire image collection:</p><pre>////// function to clip the entire ImageCollection to the field boundary //////<br>function clipImgCollect(img) {<br>  return img.clip(geometry);<br>}<br><br>// clip S2 ImageCollection to poly bounds<br>var clippedS2 = S2.map(clipImgCollect);</pre><h4>Calculate NDVI</h4><p>Now, to the final step before visualization — calculating NDVI. This can be done using the NDVI_from_S2 function below that perform the band math calculations. We can then map this function to the entire image collection:</p><pre>// Function to calculate and create NDVI<br>var NDVI_from_S2 = function(img) {<br>return img.addBands(img.normalizedDifference([&#39;B8&#39;, &#39;B4&#39;]));<br>};<br><br>// map function across the S2 image collection and<br>// create NDVI<br>var clippedS2 = clippedS2.map(NDVI_from_S2);</pre><p>This step will add a band called nd containing the NDVI to the clippedS2 collection.</p><h3>Visualize time-series</h3><p>To create a time series plot use the seriesByRegion functionality and pass the statistic to be calculated to the ee.Reducer method that helps aggregate data. In this case, we will aggregate the mean values of the data over the geometry using the ee.Reducer.mean() function.</p><pre><br>// Create a time series chart.<br>var plotNDVI_TS = ui.Chart.image.seriesByRegion(<br>  clippedS2, <br>  geometry,ee.Reducer.mean(),<br>&#39;nd&#39;,<br>500,<br>&#39;system:time_start&#39;, &#39;system:index&#39;)<br>              .setChartType(&#39;LineChart&#39;)<br>              .setOptions({ // Plot customization options<br>      interpolateNulls: true,<br>      lineWidth: 1,<br>      pointSize: 2,<br>      title: &#39;Field mean NDVI&#39;,<br>      hAxis: {title: &#39;Date&#39;},<br>      vAxis: {title: &#39;NDVI&#39;}<br>});<br><br>print(plotNDVI_TS)</pre><p>This will generate a graph of field mean NDVI time series for all the irrigated fields as shown below. This chart can be downloaded as is, or one can even download the data plotted in the chart as a table (CSV).</p><figure><img alt="Field average NDVI timeseries" src="https://cdn-images-1.medium.com/max/1024/0*_hZn15xvZNz1Yu7j.PNG" /></figure><p>However, the creation of plotNDVI_TS as shown above will certainly fail if your query request exceeds 5000 elements. For example in this example, I just increased my image collection size by selecting a much older start date (2017-03-28) in filterDate function. When I executed plotNDVI_TS function, I got an error in your console that looks like this:</p><figure><img alt="Error on google earth engine when the query exceeds the limits." src="https://cdn-images-1.medium.com/max/720/0*dYoeR1d7LGE7COpi.PNG" /></figure><p>The data being queried is too large to be plotted in the console and needs to be exported from the google earth engine.</p><h3>Exporting time series in a tabulated format</h3><p>To be able to export large time series, first select the NDVI band from the clippedS2 image collection. Then map the reduceRegions function to this NDVI image collection to calculate the mean NDVI value for each feature (irrigated field polygon) for all the images in the collection. We will also store the date of the image in the date property and format it as YYYY-MM-DD. This dat object will give us the polygon ID, Image date, and the mean NDVI for that polygon.</p><pre>// Select ndvi band<br>var clippedS2_ndvi = clippedS2.select([&#39;nd&#39;])<br><br><br>// Collect image date and ndvi mean value for all features<br>var dat = clippedS2_ndvi.map(function(image) {<br>  return image.reduceRegions({<br>     collection:geometry ,<br>    reducer: ee.Reducer.mean().setOutputs([&#39;ndvi&#39;]), <br>    scale: 10<br>  }).filter(ee.Filter.neq(&#39;ndvi&#39;, null))<br>    .map(function(f) { <br>      return f.set(&#39;date&#39;, image.date().format(&#39;YYYY-MM-dd&#39;));<br>    });<br>}).flatten();</pre><p>Now to prepare the table for export, use the format function shown below and apply it to dat object to format it such that each row in the table contains a time series of mean NDVI value for each irrigated field polygon. The column headers for each of these values are the image dates for which the mean NDVI was calculated. The formatted table is stored in the sentinelNDVI_Results object. This format function was adapted from the source <a href="https://stackoverflow.com/questions/47697793/export-large-time-series-in-google-earth-engine">linked here.</a></p><pre>var format = function(table, rowId, colId) {<br>  var rows = table.distinct(rowId); <br>  var joined = ee.Join.saveAll(&#39;matches&#39;).apply({<br>    primary: rows, <br>    secondary: table, <br>    condition: ee.Filter.equals({<br>      leftField: rowId, <br>      rightField: rowId<br>    })<br>  });<br>  return joined.map(function(row) {<br>      var values = ee.List(row.get(&#39;matches&#39;))<br>        .map(function(feature) {<br>          feature = ee.Feature(feature);<br>          var ndvi_val = ee.List([feature.get(&#39;ndvi&#39;), -9999]).reduce(ee.Reducer.firstNonNull());<br>          return [feature.get(colId), ee.Number(ndvi_val).format(&quot;%.3f&quot;)];<br>        });<br>      return row.select([rowId]).set(ee.Dictionary(values.flatten()));<br>    });<br>};<br><br>// Apply format function to the dat object<br>var sentinelNDVI_Results = format(dat, &#39;ID&#39;, &#39;date&#39;);</pre><p>The export of this formatted table to Google Drive is fairly straightforward. Just pass the sentinelNDVI_Results object to the Export.table.toDrive function. Other arguments that you need to specify include the folder on the drive where you want to export the file, fileNamePrefix - prefix to be used in the file name, and fileFormat:</p><pre>// Export data frame to Google Drive as a CSV<br>Export.table.toDrive({<br>  collection: sentinelNDVI_Results,<br>  description: &#39;Sentinel_NDVI_TS&#39;,<br>  folder: &#39;earthengine&#39;,<br>  fileNamePrefix: &#39;sentinel_ndvi_ts_demo_&#39;,<br>  fileFormat: &#39;CSV&#39;<br>})</pre><p>Here’s a sneak peek into the exported time series that I quickly plotted in excel for demonstration:</p><figure><img alt="Quick plot in excel of the longer time series extrated using google earth engine." src="https://cdn-images-1.medium.com/max/1024/0*vPPHXUerJopwb4Ja.PNG" /></figure><p>You can find the complete code used in this blog at <a href="https://code.earthengine.google.com/70290727293395d938b073362ef842a4">this GEE link</a>.</p><p><em>Originally published at </em><a href="https://chinmaydeval.com/blog/extracting-time-series-using-google-earth-engine-and-javascript/"><em>https://chinmaydeval.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=931b7f23340a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating 2D/3D visuals with R has never been this easy. Thanks to the Rayshader package!]]></title>
            <link>https://medium.com/@chinmaydeval/creating-2d-3d-visuals-with-r-has-never-been-this-easy-thanks-to-the-rayshader-package-13082defc79c?source=rss-35d240104029------2</link>
            <guid isPermaLink="false">https://medium.com/p/13082defc79c</guid>
            <category><![CDATA[r]]></category>
            <category><![CDATA[rstats]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[3d]]></category>
            <dc:creator><![CDATA[Chinmay Deval]]></dc:creator>
            <pubDate>Thu, 22 Dec 2022 01:33:45 GMT</pubDate>
            <atom:updated>2022-12-22T01:33:45.680Z</atom:updated>
            <content:encoded><![CDATA[<h3>Impressive 2D/3D plots, using just R? Wait, What?!</h3><p>Do you love creating 2D/3D visuals? Is #rstats a part of your workflow? Have you always wanted to create amazing 3D visuals like this 👇?</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FHwF-ScxzeEs%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DHwF-ScxzeEs&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FHwF-ScxzeEs%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/0b146020406ba62ac0f1ed06f9702c34/href">https://medium.com/media/0b146020406ba62ac0f1ed06f9702c34/href</a></iframe><p>Then the <a href="https://www.rayshader.com/">Rayshader</a> package is your friend. Thanks to this package, creating such visuals with R is very easy and straightforward.</p><h4>Rayshader</h4><p>Rayshader is a powerful open-source R package developed by <a href="https://twitter.com/tylermorganwall">Tyler Morgan-Wall</a>. You can learn more about the package <a href="https://www.rayshader.com/">here</a> and also find a comprehensive documentation of different functions <a href="https://github.com/tylermorganwall/rayshader">here</a>. Rayshader simplifies visualizing data in 2D and 3D with amazing levels of detail and using just R. Not just that! What amazes me is the ease at which one can customize almost anything in the visual.</p><h4>Exploring Rayshader</h4><p>Let’s start exploring some of the functionalities Rayshader has on offer. Here I will use the digital elevation model (DEM) data of my home town in India. Let’s jump right in and begin by loading the packages and the DEM layer. This is a gridded raster file. I will also crop it to the bounding box that I am interested in.</p><pre># pacakges we will need <br>library(raster)<br>library(rayshader)<br>library(tidyverse)<br>library(leaflet) <br><br># import data <br>data = raster::raster(&quot;D:/GitHub/MyContributions_to_30DayMapChallenge/11_3d/data/srtm_51_09.tif&quot;) <br><br># define bounding box <br>e = extent(c(xmax = 73.4, xmin = 74, ymin = 18.3, ymax = 18.72))<br><br># clip raster to the bounding box <br>data = crop(data,e)</pre><p>Here’s a quick look at the data:</p><pre># define color palette <br>pal = leaflet::colorNumeric(palette =&quot;inferno&quot;,<br>                            values(data), na.color = &quot;transparent&quot;) <br># plot raster layer <br>leaflet() %&gt;% <br>addTiles() %&gt;%<br>addRasterImage(data, colors = pal, opacity = 0.6) %&gt;%<br>addLegend(pal = pal, values = values(data), title = &quot;elevation (m)&quot;)</pre><figure><img alt="The image displays the elevation data being used in the blog." src="https://cdn-images-1.medium.com/max/838/1*V7ArqLucCwe2ST44_jBSRg.png" /></figure><h4>2D mapping</h4><p>Now, let us convert this data into a 2D matrix using the raster_to_matrix function from the Rayshader package.</p><pre>data_matrix = rayshader::raster_to_matrix(data)</pre><p>Generate a 2D map using the code below:</p><pre>data_matrix%&gt;% <br>rayshader::sphere_shade(texture = &quot;desert&quot;,<br> sunangle = 90) %&gt;%<br> rayshader::add_water(rayshader::detect_water(data_matrix),<br> color = &quot;imhof4&quot;) %&gt;% <br>rayshader::add_shadow(rayshader::ray_shade(data_matrix), 0.5) %&gt;%<br> rayshader::add_shadow(rayshader::ambient_shade(data_matrix), 0) %&gt;%<br> rayshader::plot_map()</pre><figure><img alt="2-D visualization of the eleveation data" src="https://cdn-images-1.medium.com/max/480/0*SwMhf7LpZ90X7pIq.png" /></figure><p>The detect_water and add_water functions detect and add the water layers to the plot. We can even shift around the sun&#39;s direction using the sunangle argument in the sphere_shade function. The ray_shade function creates a ray-traced lighting layer from a given sun angle and the ambient_shade function models and adds the lighting layer as a result of atmospheric scattering.</p><h4>3D mapping</h4><p>Generate a 3D map using the code below:</p><pre>data_matrix%&gt;% <br>rayshader::sphere_shade(texture = &quot;desert&quot;) %&gt;%<br> rayshader::add_water(rayshader::detect_water(data_matrix),<br> color = &quot;imhof4&quot;) %&gt;%<br> rayshader::add_shadow(rayshader::ray_shade(data_matrix, zscale = 3), 0.5) %&gt;%<br> rayshader::add_shadow(rayshader::ambient_shade(data_matrix), 0) %&gt;%<br> rayshader::plot_3d(data_matrix, zscale = 10, fov = 10, theta = 45,<br> zoom = .65, phi = 45, windowsize = c(1000, 800)) Sys.sleep(0.2) <br><br># save snapshot to a file <br>render_snapshot(&quot;3d_snap.png&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*Hlr2p4t-0sdsuPvN.png" /></figure><h4>Adding labels</h4><p>What if you wanted to add labels on this map? That is easy too. you can use the render_label function to add labels as shown below:</p><pre>data_matrix%&gt;%<br> rayshader::sphere_shade(texture = &quot;desert&quot;) %&gt;%<br> rayshader::add_water(rayshader::detect_water(data_matrix), color = &quot;imhof4&quot;) %&gt;%<br> rayshader::add_shadow(rayshader::ray_shade(data_matrix, zscale = 3), 0.5) %&gt;%<br> rayshader::add_shadow(rayshader::ambient_shade(data_matrix), 0) %&gt;%<br> rayshader::plot_3d(data_matrix, zscale = 10, fov = 10, theta = 65,<br> zoom = .65, phi = 45, windowsize = c(1000, 800)) <br><br>render_label(data_matrix, x = 450, y = 340, z = 18000, zscale = 50,<br> text = &quot;Khadakwasla&quot;, textsize = 2, linewidth = 5, freetype = FALSE,<br> linecolor = &quot;#df7c4d&quot;, textcolor = &quot;#df7c4d&quot;, dashed = TRUE) <br><br>render_label(data_matrix, x = 250, y = 390, z = 15000, zscale = 50,<br> text = &quot;Panshet&quot;, textsize = 2, linewidth = 5, freetype = FALSE, <br> linecolor = &quot;#df7c4d&quot;, textcolor = &quot;#df7c4d&quot;, dashed = TRUE) <br><br>render_label(data_matrix, x = 100, y = 220, z = 10000, zscale = 50,<br> text = &quot;Mulshi&quot;, textsize = 2, linewidth = 5, freetype = FALSE,<br> linecolor = &quot;#df7c4d&quot;, textcolor = &quot;#df7c4d&quot;, dashed = TRUE) <br><br>render_label(data_matrix, x = 100, y = 100, z = 9000, zscale = 50,<br> text = &quot;Pawna&quot;, textsize = 2, linewidth = 5, freetype = FALSE,<br> linecolor = &quot;#df7c4d&quot;, textcolor = &quot;#df7c4d&quot;, dashed = TRUE) <br><br>Sys.sleep(0.2) <br><br># save snapshot to a file render_snapshot(&quot;3d_snap_with_labels_featured.png&quot;)</pre><figure><img alt="3-D visualization of the eleveation data along with labels" src="https://cdn-images-1.medium.com/max/1000/0*LY9-sNwsnVFiiqqU.png" /></figure><h4>Exporting a movie</h4><p>To export a movie/video like the one at the begining of this piece you can use the following lines of code. This will save the rendered 3D frames as a movie.</p><pre>data_matrix%&gt;%<br>  rayshader::sphere_shade(texture = &quot;desert&quot;) %&gt;%<br>  rayshader::add_water(rayshader::detect_water(data_matrix), color = &quot;imhof4&quot;) %&gt;%<br>  rayshader::add_shadow(rayshader::ray_shade(data_matrix, zscale = 3), 0.5) %&gt;%<br>  rayshader::add_shadow(rayshader::ambient_shade(data_matrix), 0) %&gt;%<br>  rayshader::plot_3d(data_matrix, zscale = 10, fov = 10, theta = 65, zoom = .65, phi = 45,<br>          windowsize = c(1000, 800))<br> render_label(data_matrix, x = 450, y = 340, z = 18000, zscale = 50,<br>             text = &quot;Khadakwasla&quot;, textsize = 2, linewidth = 5, freetype = FALSE,<br>             linecolor = &quot;#df7c4d&quot;,<br>             textcolor = &quot;#df7c4d&quot;,<br>             dashed = TRUE)<br> render_label(data_matrix, x = 250, y = 390, z = 15000, zscale = 50,<br>              text = &quot;Panshet&quot;, textsize = 2, linewidth = 5, freetype = FALSE,<br>              linecolor = &quot;#df7c4d&quot;,<br>              textcolor = &quot;#df7c4d&quot;,<br>              dashed = TRUE)<br> render_label(data_matrix, x = 100, y = 220, z = 10000, zscale = 50,<br>              text = &quot;Mulshi&quot;, textsize = 2, linewidth = 5, freetype = FALSE,<br>              linecolor = &quot;#df7c4d&quot;,<br>              textcolor = &quot;#df7c4d&quot;,<br>              dashed = TRUE)<br> render_label(data_matrix, x = 100, y = 100, z = 9000, zscale = 50,<br>              text = &quot;Pawna&quot;, textsize = 2, linewidth = 5, freetype = FALSE,<br>              linecolor = &quot;#df7c4d&quot;,<br>              textcolor = &quot;#df7c4d&quot;,<br>              dashed = TRUE)<br>Sys.sleep(0.2)</pre><p><em>Originally published at </em><a href="https://chinmaydeval.com/blog/rayshader/"><em>https://chinmaydeval.com</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13082defc79c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>