<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Samapriya Roy on Medium]]></title>
        <description><![CDATA[Stories by Samapriya Roy on Medium]]></description>
        <link>https://medium.com/@samapriyaroy?source=rss-3a3f6dbb93ce------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ZzkjE9Ka06B6T3oJoIrZOA.jpeg</url>
            <title>Stories by Samapriya Roy on Medium</title>
            <link>https://medium.com/@samapriyaroy?source=rss-3a3f6dbb93ce------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 20:35:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@samapriyaroy/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How ClimateEngine.org and Awesome GEE Community Catalog are Expanding Open Geospatial Commons]]></title>
            <link>https://samapriyaroy.medium.com/how-climateengine-org-and-awesome-gee-community-catalog-are-expanding-open-geospatial-commons-30120b1bfbaf?source=rss-3a3f6dbb93ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/30120b1bfbaf</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[climate-change]]></category>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[google-earth-engine]]></category>
            <category><![CDATA[geospatial]]></category>
            <dc:creator><![CDATA[Samapriya Roy]]></dc:creator>
            <pubDate>Thu, 05 Oct 2023 19:46:32 GMT</pubDate>
            <atom:updated>2023-10-05T20:03:47.007Z</atom:updated>
            <content:encoded><![CDATA[<p>In conversation with <a href="https://medium.com/u/3c769fc79e7b">Justin Huntington</a> and <a href="https://medium.com/u/d8df88d9621d">Eric Jensen</a></p><blockquote><strong><em>What are we if not storytellers with data? There are more narratives hidden in a time series than collective memories, more than just raw cold data.</em></strong></blockquote><p>No one knows how to tell stories better with data than the <a href="https://www.dri.edu/">Desert Research Institute (DRI)</a>. As Justin Huntington, a Research Professor at DRI and Climate Engine project lead, once explained, “<strong>If you’re trying to study climate and environmental change, nothing beats the combination of maps and time series.</strong>”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SVBAqbwzwNgfksdof_L8wA.gif" /><figcaption>Climate Engine Banner from <a href="https://www.climateengine.org/">ClimateEngine.org</a></figcaption></figure><p>ClimateEngine.org was founded in 2014 on the belief that open access to Earth observations data can empower discoveries that improve lives. Its public platform now makes decades of satellite imagery and climate data available for use without the need to code. Now, the ClimateEngine.org partnership between Desert Research Institute (DRI) and the University of California, Merced (UC-Merced) is expanding its contribution to the open data commons by making its internal Google Earth Engine datasets publicly available through the Awesome GEE community catalog.</p><p>When asked about Climate Engine’s origin story, Justin recalls, “<strong><em>In 2014 we got a Google Faculty grant to bring drought and evapotranspiration data into Earth Engine. This allowed agencies to run algorithms at scale instead of moving data locally.</em></strong>” Justin adds, “<strong><em>Everyone was looking to scale what they were doing. It drove rapid adoption beyond just drought and ET.</em></strong>”</p><p>The Awesome GEE Community Catalog was started in 2020 by Samapriya Roy to promote the openness and accessibility of geospatial data. This unfunded open-source project has grown rapidly to over 320TB of data and 1350+ datasets across diverse themes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*albDyJ6tnSrK_5876b6blA.png" /><figcaption><a href="https://gee-community-catalog.org/"><strong>Awesome GEE Community Catalog</strong></a></figcaption></figure><p>Just as shared norms create value in offline commons, principles of openness reduce friction for downstream users of digital commons. Awesome GEE community catalog is a manifestation of the belief that, as I have always said,</p><blockquote>“Communities are what communities build together.”</blockquote><p>Recently, DRI researchers Justin Huntington and Eric Jensen reached out to collaborate with me on getting many of the datasets at ClimateEngine.org over to the Awesome Google Earth Engine (GEE) Community Catalog, resulting in the addition of 29 new datasets. By contributing national and global datasets on precipitation, fire, evapotranspiration, subseasonal to seasonal climate and drought forecasts, and more, we are all hoping to unlock new potential uses of these resources. The next couple of sections are some questions and thoughts I got to ask Eric and Justin during our conversations. To find all the datasets that were contributed <a href="https://gee-community-catalog.org/changelog/"><strong>check out the changelog</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CGl8vh7PMuedFTuM7_0NDw.gif" /><figcaption>Dataset subsets from ClimateEngine.org contributions to the Awesome GEE Community Catalog</figcaption></figure><blockquote><strong>Sam</strong>: <em>Earth Engine is typically effective at aggregating datasets. However, agencies often face challenges in recruiting and training staff to utilize it. Climate Engine.org seems to address this issue by serving as a bridge. What is your perspective on this</em>?</blockquote><p><strong>Eric</strong>: We receive funding from Federal agencies, each with unique requirements and objectives for their datasets. Many of these agencies have sponsored various projects that have produced datasets, but the current approach of providing multiple tools and websites for agency staff to access these datasets isn’t the most efficient. This is where ClimateEngine.org plays a crucial role by providing a unified platform for performing analysis. This platform not only offers a centralized location to access common analysis tools and curated datasets but also simplifies the work involved in applying these resources, including creating derivative products such as automated reports.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mutIgR6a0cnAfo1d" /><figcaption><em>By adding operational vegetation cover datasets like the Rangeland Analysis (RAP) and the Rangeland Condition Monitoring Assessment and Projection (RCMAP) to Climate Engine, it reduces friction for resource managers and researchers to use these datasets to inform decisions and make findings. This map shows tree cover reduction after a sagebrush restoration project to support sage grouse habitat in Idaho, US.</em></figcaption></figure><p>According to Eric, a key motivation was allowing users worldwide the ability to access publically-funded data easily through a common platform. This reduced barriers for researchers and decision-makers alike. Climate Engine’s tools and datasets are now relied upon globally by thousands each month.</p><blockquote><strong>Sam</strong>: <em>What do you envision as the next steps? It’s evident that there’s a drive to incorporate more data and foster more collaborations. What developments are you observing</em>?</blockquote><p><strong>Justin</strong> : “As we incorporated additional datasets, we recognized the potential to perform numerous calculations within the app, resulting in variables or datasets that could stand on their own. One example was producing maps of anomalies of the normalized difference vegetation index (NDVI) from Landsat or MODIS, which blew our minds at the time — we realized we could make those calculations available easily within the app. Our aim is to modularize and provide code and variables that users can easily access. The next phase in this gradual evolution is linking data to methodologies and providing code and notebook tutorials. As a hydrologist, I never thought I’d see papers citing precipitation and NDVI data created with the Climate Engine app for urban poverty and human health studies, but it’s happening more and more. There are just so many needs, and now with all these multidisciplinary data and calculation options at our fingertips, the important stories to tell are endless”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pn2JzzAAcajJizoS" /><figcaption>The Climate Engine team discovered while developing the app that they could make mapping NDVI anomalies easy for users. When paired with real-time drought information, also in the app, it unlocks the ability to perform powerful analysis of vegetation sensitivity to drought stress.</figcaption></figure><p><strong>Eric </strong>: “We were also surprised to find 70% of our website traffic comes from outside the US, with significant traffic coming from some of the most biologically rich nations in the world, including India, Colombia, Indonesia, Mexico, Brazil, and Argentina. It validates the tremendous value of global open data.”</p><blockquote><strong>Sam</strong>: Let’s delve into the data aspect. I see various themes like Evapotranspiration, Drought, Climate Grids, and more. To kick things off, could you tell me about the Drought outlook dataset, especially considering my role in maintaining the United States Drought Monitor weekly data?</blockquote><p>Justin: “Certainly, the Seasonal Drought Outlook dataset extends large-scale drought trends guided by statistical and dynamical forecasts. It provides a forward-looking view for the next 3-months, offering an outlook map that predicts whether drought conditions are expected to improve, worsen, or remain the same. Using this outlook in combination with more detailed current condition maps and downscaled forecast indices available in Climate Engine, such as gridMET-CFS evaporative demand forecast anomalies, is super powerful for drought early warning.”</p><blockquote><strong>Sam</strong>: With your extensive list of approximately 29 datasets, can you highlight some of your personal favorites?</blockquote><p><strong>Justin</strong>: “One dataset I find particularly interesting is the US Drought Monitor and advanced calculations that you can now perform easily with a few clicks, like median, trends, and probabilities over the period of record. Other noteworthy datasets are those that include potential or reference evapotranspiration, such as MERRA2, AgERA5, and the Forecast Reference Crop Evapotranspiration (FRET) dataset. It’s fascinating because it combines multiple input variables like solar radiation, air temperature, windspeed, and humidity. The real power of these datasets shines when we can create visual narratives within seconds using maps and time series that combine land surface observations like MODIS land surface temperature, with climate anomalies, and more, to hypothesize and describe complex land surface and atmospheric feedback.”</p><p><strong>Eric</strong>: “We’ve been operationally ingesting and working with various climate and hydrology datasets, some of which were requested by NOAA, such as the NOAA U.S. Climate Gridded Dataset (NClimGrid), Applied Climate Information System (ACIS), and SNOw Data Assimilation System (SNODAS). Many of these datasets involve automated processes through Google Cloud Functions jobs. The Google Cloud Functions retrieve data from public data repositories and websites and write them to ClimateEngine.org buckets, which we then ingest as Earth Engine datasets for use in the ClimateEngine.org app and, now, as publicly available endpoints available through the catalog.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uHTAk4A0x_KnR1a1" /><figcaption><strong>SNODAS provides 1 km resolution daily estimates of snow depth and snow water equivalent (SWE), which can be applied to use cases such as near real-time snowpack monitoring or long-term mapping and analysis of trends in the annual maximum snowpack (</strong><a href="https://code.earthengine.google.com/d450590be6fa75bc1d4b75d865f52836"><strong>see EE script for producing trend map here</strong></a><strong>)</strong></figcaption></figure><p><strong>Justin</strong>: “Lastly, having land surface model fluxes in Climate Engine has truly been eye opening. Currently, we are thinking of how we can ingest and add Western Land Data Assimilation System (WLDAS). It provides information on soil moisture at various depths, snowpack data, and components of the entire water cycle at a 1km resolution. These models offer a comprehensive view of Earth’s surface processes.”</p><p>Thank you both for contributing to the community catalog and for taking the time to chat with me today</p><p>The collaborations between organizations like DRI, UC-Merced, and ClimateEngine.org and grassroots efforts like the Awesome GEE Community Catalog demonstrate the power of an open data community coming together for the greater good. When researchers and advocates join forces to make scientific datasets freely accessible, it unlocks new potential for discovery and innovation.</p><p>Open-source ecosystems thrive through these kinds of mutually beneficial partnerships. Contributors of open data benefit from seeing how others build on their work to advance science. Open access also garners more citations and recognition for the researchers who produced the datasets. For those using open data, the ability to leverage existing research accelerates innovation and discovery.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=30120b1bfbaf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Behind the Scenes: The Role of NSF’s Jetstream2 in Building the Awesome GEE Community Catalog]]></title>
            <link>https://samapriyaroy.medium.com/behind-the-scenes-the-role-of-nsfs-jetstream2-in-building-the-awesome-gee-community-catalog-3f563b8cb9f0?source=rss-3a3f6dbb93ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/3f563b8cb9f0</guid>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[google-earth-engine]]></category>
            <category><![CDATA[geospatial]]></category>
            <dc:creator><![CDATA[Samapriya Roy]]></dc:creator>
            <pubDate>Tue, 12 Sep 2023 15:11:36 GMT</pubDate>
            <atom:updated>2023-09-12T15:25:32.146Z</atom:updated>
            <content:encoded><![CDATA[<p>Delve into the world of data preprocessing, where we demystify what happens behind the scenes at the <a href="https://gee-community-catalog.org/"><strong>Awesome GEE community catalog</strong></a> and how <a href="https://jetstream-cloud.org/index.html"><strong>Jetstream2 a National Science Foundation computing resource</strong></a> plays a vital role in making geospatial data more accessible and revolutionizing research workflows via the community catalog.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q6ZiJT9L4vrBD9Qp6SSJqw.png" /><figcaption>Jetstream 2 logo from <a href="https://jetstream-cloud.org/index.html">Jetstream2 mainpage visit here</a></figcaption></figure><p>We’ll trace the evolution of the community Catalog, revealing how it has grown over time into a valuable resource for geospatial research.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*albDyJ6tnSrK_5876b6blA.png" /><figcaption><a href="https://gee-community-catalog.org/"><strong>Visit the Awesome GEE Community Catalog to get started</strong></a></figcaption></figure><p>Community and collaboration are at the core of this ecosystem. Explore the collaborative efforts, partnerships, and contributions that drive the Awesome GEE Community Catalog. We’ll also discuss NSF’s role in promoting open science through transformative projects like Jetstream2.</p><blockquote><strong>Evolution of the </strong><a href="https://gee-community-catalog.org/"><strong>Awesome GEE Community Catalog</strong></a></blockquote><p>It all started approximately three years ago with a single dataset sourced from the High-Resolution Settlement Layers provided by Facebook. This humble beginning marked the inception of what would soon become an open data catalog, a valuable resource for the geospatial community. You can <a href="https://medium.com/geospatial-processing-at-scale/community-datasets-data-commons-in-google-earth-engine-8585d8baef1f?source=your_stories_page-------------------------------------"><strong>read about our approach to community commons and our early start here</strong></a>. Fast forward to today, and this experiment has flourished into an impressive project, boasting over <strong>320 terabytes of datasets</strong> and encompassing more than <strong>1300 unique datasets</strong> with over a <strong>million images</strong> and over a <strong>billion features</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*3wvrMObVEXCoEI-6LOo3Uw.png" /><figcaption><a href="https://gee-community-catalog.org/stats/"><strong>Catalog stats 2023–09–01</strong></a></figcaption></figure><p>Behind this transformation lies a commitment to fostering a geospatial data commons. The concept was elegantly simple: create a platform for datasets, and let the community nurture its growth. Over time, what began as the development of a single tool to fetch a lone dataset has evolved into a sophisticated system, encompassing over <strong>120 thousand lines of code</strong>. This codebase facilitates seamless interactions with datasets, automates processes, handles downloads, and conducts preprocessing, all to ensure that the community catalog remains a valuable resource for geospatial researchers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*lSMCGlG-S27-diTyYZ1J8w.png" /><figcaption>Catalog features for community catalog</figcaption></figure><p>Throughout this journey, collaboration has been key. Individual data contributors have played a vital role in enriching the catalog, while tools have been crafted to assist organizations in sharing their datasets with ease. The process of building this community has been an ongoing effort, reflecting a shared commitment to advancing geospatial research.</p><blockquote><strong>Behind the Scenes of Data Preprocessing</strong></blockquote><p>Behind the scenes, the intriguing journey of data preprocessing unfolds, commencing with data requests or proactive updates and additions to the catalog. This process transcends the conventional approach of archiving data on personal websites or within data centers, aligning with the mandates of many funding agencies. For instance, anyone with geospatial data can initiate the journey by submitting their dataset for inclusion in the community catalog.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f923Irt-rNrWVjQPdk6ObA.gif" /><figcaption><strong>Getting started with</strong> <a href="https://gee-community-catalog.org/startup/submit/"><strong>submitting a dataset request</strong></a></figcaption></figure><p>What sets this process apart is its inclusivity. You need not be the dataset author to contribute; if you’re utilizing specific products within your projects, you can suggest their inclusion. The fundamental idea is to pool these datasets together, beginning by selecting those that a few contributors have uploaded or preprocessed to some degree. This involves a meticulous verification process that encompasses license types, data sources, and more. The overarching goals are twofold: to enhance data accessibility and to significantly boost the reproducibility of scientific research.</p><p>To facilitate this intricate journey, an extensive arsenal of custom scripts has been developed, comprising over <strong>120,000 lines of code</strong>. These scripts serve as the backbone, capable of handling datasets with varying levels of complexity. They excel in tasks such as fetching data from diverse providers like Zenodo, Dryad, Figshare, and others, preprocessing, and optimizing datasets to ensure seamless integration into the Community catalog. <strong>All of this runs in NSF-funded</strong> <a href="https://docs.jetstream-cloud.org/ui/exo/exo/"><strong>Jetstream2’s exosphere VMs</strong></a> and is backed with an impressive amount of always-on-compute resources which we will explore soon.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y_smP68yc077jlKdLXF83w.gif" /><figcaption><a href="https://jetstream2.exosphere.app/exosphere/"><strong>Jetstream 2 exosphere app</strong></a></figcaption></figure><p>Tasks range from reprojecting data to conducting batch LZW compressions and format conversions, all seamlessly executed by these scripts. Moreover, they play a crucial role in keeping the community catalog continuously updated with the latest data. Finally, the datasets are ingested into Google Earth Engine for use via transitioning them to an intermediate GCS bucket.</p><p>For those seeking deeper dive into this <a href="https://medium.com/@samapriyaroy/awesome-google-earth-engine-community-catalog-bd86d0ba63b8"><strong>intricate journey, a Medium article</strong></a> and a presentation from the <a href="https://earthoutreachonair.withgoogle.com/events/geoforgood22"><strong>Geo for Good 2022</strong></a> event provide detailed information to complement this narrative.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FQLzDet5-CvY%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DQLzDet5-CvY&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FQLzDet5-CvY%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/37ebed945bf03ece4fa9232019630d0f/href">https://medium.com/media/37ebed945bf03ece4fa9232019630d0f/href</a></iframe><p>These steps ensure that datasets are not only accessible but also seamlessly integrated into the geospatial research ecosystem. Over the next few paragraphs, I am going to dive into Jetstream2’s role and setup and how this functions as the powerhouse behind a lot of this work.</p><blockquote><strong>Jetstream2’s Role in Geospatial Data Accessibility</strong></blockquote><p>Jetstream2 is a cloud-based on-demand computing and data analysis resource. Its significance extends beyond the boundaries of traditional high-performance computing, offering an accessible and user-friendly environment tailored to meet the evolving needs of researchers. This NSF-funded initiative plays a pivotal role in helping build the GEE community catalog and for other researchers towards transformative research endeavors. This accessibility empowers a diverse array of institutions, including small colleges, historically black colleges and universities, minority-serving institutions, tribal colleges, and higher education institutions in EPSCoR states.</p><p>You can <a href="https://docs.jetstream-cloud.org/alloc/general-allocations/"><strong>apply for access here</strong></a><strong>. I</strong>t’s allocated via <a href="https://access-ci.org/"><strong>ACCESS</strong></a> and all allocations are free to US-based researchers doing <a href="https://docs.jetstream-cloud.org/general/policies/#jetstream2-and-general-research-policies"><strong>open, publishable science</strong></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_a-IMlvaQPaV0A3ELqmFNw.jpeg" /><figcaption>Panel artwork on <a href="https://www.nsf.gov/news/mmg/mmg_disp.jsp?med_id=188612&amp;from="><strong>Jetstream2 from NSF gallery</strong></a></figcaption></figure><p>Jetstream2 extends its capabilities to encompass a broader range of hardware and services, accommodating diverse research needs. Jetstream2 serves as a valuable resource not only for individual researchers but also for gateway projects and other “always on” services, enhancing the collaborative nature of geospatial research. By providing a user-friendly interface and a suite of features, including interactive virtual machines, secure data movement, and virtual desktops.</p><p>The National Science Foundation (NSF) has made significant investments toward making high-performance computing more accessible through projects like STAMPEDE-2, FRONTERA, &amp; Jetstream2 to name a few. These resources have not only reduced the cost barriers associated with cutting-edge research applications but have also played a critical role in supporting large-scale research and development.</p><blockquote><strong>Jetstream2 Behind the Scenes</strong></blockquote><p>At the heart of Jetstream2’s accessibility is Exosphere, a user-friendly interface that simplifies resource management. Researchers can easily create and manage instances, volumes, and persistent IP addresses through Exosphere. It also provides convenient tools such as a one-click web shell (terminal) in your browser, a one-click desktop environment for running graphical software, and a browser-based file upload/download tool. For newcomers to Jetstream2 or those seeking a straightforward resource management solution, Exosphere is the ideal starting point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hNsdD4vf2ypo5aDa85ESuQ.gif" /><figcaption>Launching an VM on Exosphere on Jetstream2</figcaption></figure><p>Jetstream2 goes beyond simplicity by offering custom application-specific virtual machine (VM) configurations. It supports CPU and GPU nodes, providing researchers with tailored options to suit their specific computational needs. Researchers can submit allocation proposals, obtain approvals, and convert allocations into desired configurations, whether storage volumes or GPU hours. This flexibility ensures that Jetstream2 caters to a wide range of research requirements.</p><p>Jetstream2’s capabilities extend to its impressive connectivity. With a 100 Gbps network connection from compute hosts to the cloud’s internal network infrastructure and 2x100 Gbps uplinks from the cloud infrastructure to the data center infrastructure, it ensures rapid data transfer and computational performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YtajX3BwVXaiZnpTCjwpXA.gif" /><figcaption><strong>Logging into VM via ssh and running neofetch and speedtest</strong></figcaption></figure><p>Additionally, it boasts 100 Gbps connectivity to the Internet2 backbone and 100 Gbps connectivity to the XSEDE research network via virtualized links. These robust network connections empower researchers to seamlessly transfer large datasets and run computationally intensive applications without encountering performance bottlenecks.</p><p>Efficient storage management is critical in geospatial computation, and Jetstream2 excels in this aspect. For the community catalog and under the project allocation we get approximately 30 TB of transient storage, distributed across multiple drives that can be attached or detached from instances as needed. What sets Jetstream2 apart is its unique hot swap capability, allowing users to detach a volume from an active instance and attach it to another, enhancing operational efficiency and data accessibility.</p><blockquote><strong>Geospatial Accessibility and Open Science</strong></blockquote><p>In the ever-evolving landscape of geospatial research, the fusion of open data, cutting-edge supercomputing, and high-performance clusters has propelled us into an era of unprecedented possibilities. At the forefront of this transformative journey stand programs like the NSF-funded Jetstream2 are part of the National Science Foundation’s commitment to making research data not just accessible but also highly usable. You can cite Jetstream2 <a href="https://jetstream-cloud.org/research/index.html"><strong>using the citation information here</strong></a><strong>.</strong></p><p>In the year of Open Source data and science, 2023, the principles of <a href="https://www.go-fair.org/fair-principles/"><strong>FAIR (Findability, Accessibility, Interoperability, and Reuse)</strong></a> resonate profoundly. These principles underpin our efforts, guiding us in our mission to build a geospatial data ecosystem that empowers researchers and advances the frontiers of knowledge.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XO5Aj6lmLqrPdwhwZZr5yg.png" /><figcaption><strong>Community Commons on Open Compute</strong></figcaption></figure><p>The <a href="https://gee-community-catalog.org/"><strong>Awesome GEE Community Catalog</strong></a>, is a testament to collaborative effort and innovation, embodies these principles in its very essence. You can also cite the community catalog using <a href="https://doi.org/10.5281/zenodo.7144933"><strong>citations available here</strong></a></p><p>Reflecting on the complexity of this endeavor keeps us humble. The scale of data, the intricacies of preprocessing, and the seamless usability of the catalog are a testament to the dedication of the geospatial community. As I often say, “<strong><em>Communities are what communities build together.</em></strong>” This sentiment embodies the spirit of collaboration, echoing the harmonious synergy of researchers, contributors, and organizations who have collectively constructed a resource that continues to drive geospatial research forward.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3f563b8cb9f0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Paper to Pixels- Rediscovering Historical USGS Topo Maps in the Google Earth Engine Community…]]></title>
            <link>https://samapriyaroy.medium.com/from-paper-to-pixels-rediscovering-historical-usgs-topo-maps-in-the-google-earth-engine-community-f514c97c46a?source=rss-3a3f6dbb93ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/f514c97c46a</guid>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[google-earth-engine]]></category>
            <category><![CDATA[maps]]></category>
            <category><![CDATA[geography]]></category>
            <dc:creator><![CDATA[Samapriya Roy]]></dc:creator>
            <pubDate>Tue, 08 Aug 2023 06:01:11 GMT</pubDate>
            <atom:updated>2023-08-08T06:21:03.191Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>From Paper to Pixels- Rediscovering Historical USGS Topo Maps in the Google Earth Engine Community Catalog</strong></h3><p>Maps have long been essential tools for understanding and navigating the world around us. One prominent series of maps that have played a crucial role in the United States is the <a href="https://www.usgs.gov/programs/national-geospatial-program/topographic-maps"><strong>USGS (United States Geological Survey) Topo Maps</strong></a>. These maps have a rich history, dating back to the late 19th century, and have evolved into a powerful resource for various applications. Among the various scales used, the 1:24,000 scale stands out for its significance in striking a balance between detail and coverage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QUT_oFqikl_giWwfJv55rg.gif" /><figcaption>USGS historical topo maps rendered in Google Earth Engine</figcaption></figure><p>The history of USGS Topo Maps traces back to the late 19th century when the USGS embarked on a mission to map the entire United States in intricate detail. The 1:24,000 scale, also known as 7.5-minute quadrangle maps, emerged as one of the most widely used scales. Each map sheet covers an area of 7.5 minutes of longitude and latitude, resulting in a detailed representation of approximately 64 square miles (166 square kilometers). These maps became indispensable for a myriad of activities, including scientific research, land use planning, recreational exploration, and more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/813/1*Xq-h-XKMRcgtBp5HLAgbZA.jpeg" /><figcaption><a href="https://www.flickr.com/photos/uconnlibrariesmagic/4733412473">Southbridge Quadrangle 1952 — USGS Topographic Map 1:24,000</a></figcaption></figure><blockquote><strong>Getting to the Data</strong></blockquote><p>The USGS aimed at preserving and utilizing the historical USGS Topo Maps had started to scan and georeference thousands of these maps over varying scales across the US. Knowing the significance of the 1:24000 scale, I started to <a href="https://ngmdb.usgs.gov/topoview/viewer/#4/40.00/-100.00"><strong>download over 81,000 maps </strong></a>spanning from the 1900s to the present. The historical USGS Topo Maps at the 1:24,000 scale offers more than just topographic details.</p><p>Occasionally, they include ortho photomosaics, providing an additional layer of information. These photomosaics are aerial photographs that have been orthorectified, meaning they have been corrected for distortions caused by the Earth’s curvature and terrain variations. Incorporating such imagery further enhances the understanding of historical landscapes. Alongside the map scans, a 1:24,000 map index or grid was obtained, guiding users to the precise location of each map.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iu8BZS5aq8LoaZ6b_qzItA.gif" /><figcaption>topoView page to download USGS historical quads</figcaption></figure><blockquote><strong>Preprocessing and removing map collars</strong></blockquote><p>Ingesting these thousands of maps, totaling over 6.86 terabytes of data, into the Google Earth Engine collection was challenging. However, the effort was well worth it, as this vast archive of historical maps now offers researchers, historians, and enthusiasts an unprecedented glimpse into the past. From understanding how the landscape has evolved over time to comparing historical and contemporary features, the possibilities for exploration and visualization are boundless.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qg1e8M_92TIRCrHD48HrAA.gif" /><figcaption>USGS topo map in Earth Engine with collar</figcaption></figure><p>A key challenge arose when stitching together the individual map sheets due to the presence of collars. These map collars contain important metadata, such as map legends and other details, but can obstruct the seamless joining of adjacent maps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6yVl5r4fZDRAPlHhRUsu4A.gif" /><figcaption>USGS map index 1:24000</figcaption></figure><p>To address this, a function was developed to utilize the 1:24,000 map index layer. By clipping out the collars, the maps could be precisely aligned, creating a unified and comprehensive representation of the region. I further added metadata so there is region and location information available in these map collection metadata apart from year and state which could be used for aggregation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K-A8TTmGVzSJGyUZLO0yGQ.gif" /><figcaption>USGS topo map in Earth Engine without collar</figcaption></figure><blockquote><strong>Dataset availability</strong></blockquote><p>The collection of historical USGS Topo Maps with a 1:24,000 scale was part of the insiders’ list and is now generally available in the upcoming release 1.0.9 and will have continued tweaks and improvements. You can find <a href="https://gee-community-catalog.org/projects/usgs_topo/"><strong>details and the links here</strong></a>. This endeavor was not only an ambitious experiment in unearthing and preserving historical data but also an opportunity to learn about traditional map-making techniques and the art of cartography. Some states are not included yet like California and Texas among a few other states owing to the number of assets and the size limitations.</p><p>Try an <a href="https://code.earthengine.google.com/?scriptPath=users/sat-io/awesome-gee-catalog-examples:analysis-ready-data/USGS-TOPO-RENDER"><strong>example script here</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7JZZ01HV4Iv9L3VzIboh2Q.gif" /><figcaption>USGS Collection example at varying scales in GEE</figcaption></figure><p>The USGS Topo Maps with a 1:24,000 scale provide a fascinating window into the past, offering valuable insights into the changing landscapes of the United States. The effort to digitize, ingest, and stitch together this treasure trove of over 81,000 maps was challenging, but the result is a comprehensive and invaluable resource for researchers, educators, and map enthusiasts alike. Through this project, I can extend my love for historical maps to others and also celebrate the evolution of map-making methods and the timeless importance of cartography in comprehending our world.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f514c97c46a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Community-Enhanced: Google Earth Engine Community Catalog Upgrades]]></title>
            <link>https://samapriyaroy.medium.com/community-enhanced-google-earth-engine-community-catalog-upgrades-66d4b7076826?source=rss-3a3f6dbb93ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/66d4b7076826</guid>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[google-earth-engine]]></category>
            <category><![CDATA[code]]></category>
            <category><![CDATA[community-engagement]]></category>
            <dc:creator><![CDATA[Samapriya Roy]]></dc:creator>
            <pubDate>Tue, 25 Jul 2023 11:01:37 GMT</pubDate>
            <atom:updated>2023-07-25T11:01:37.405Z</atom:updated>
            <content:encoded><![CDATA[<p>When I created the <a href="https://gee-community-catalog.org/"><strong>Google Earth Engine Community catalog</strong></a> my primary focus was always on fostering collaboration and addressing any issues in our datasets with the help of valuable feedback from the geospatial community. What better way to learn from the community than from feedback? So this one is about a lesson learned and an adventure in fixing or updating the <a href="https://docs.digitalearthafrica.org/en/latest/data_specs/Cropland_extent_specs.html"><strong>Digital Earth Africa Cropland mask</strong></a>. One of the emails I received captured what seemed to be this edge effect missing tiles around the corners of regions when looking at the datasets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7OPdBbqyr-vpDNz_IsbuCA.png" /><figcaption>Left: Complete tiles from Digital Earth Africa collection from Source Right: Missing Tiles in Digital Earth Africa Cropmask Earth Engine Collection from Awesome GEE community catalog</figcaption></figure><p>While the datasets were downloaded from the <a href="https://registry.opendata.aws/deafrica-crop-extent/">Amazon Open Data registry on AWS</a> and while STAC objects are unique within their collection a multi-collection STAC has recurrent objects with the name id. This was not as obvious because bringing all images to the same folder even after using an AWS sync operation or using STAC download tools would simply overwrite or skip over existing files.</p><blockquote>Note: This would happen if you were to bring all filtered, mask, and prob tif files to a single folder trying to combine them into a single collection. This is tied to the fact that edges between regions have recurrent tiles across collections and that is what created the missing tiles effect.</blockquote><blockquote><strong>Finding the problem</strong></blockquote><p>One of my favorite parts of digging for a problem or debugging is looking at it from every single aspect. So I went digging into STAC. <a href="https://stacspec.org/en"><strong>Spatio Temporal Asset Catalogs</strong></a> for the uninitiated serve out metadata and arrange metadata in hierarchies like collections of images or image collections and items or images themselves 🗂️🌄. The benefit of STAC is standardization and interoperability with other STAC manifests and each collection tree is unique in some way. While that is true multiple collections create an interesting issue where unique objects may be repeated just nested under the different collections. This was an assumption I made and now I had to check if the STAC ids across multiple collections could have the same ID. Turns out a few of them overlapped. So I made sure I made a mental note 📝. You can visualize the STAC collections using the <a href="https://radiantearth.github.io/stac-browser/#/external/explorer.digitalearth.africa/stac/?.language=en">STAC browser easily just search for crop</a> 🔍📊.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VUEfLAbkuBW2GICKSYXm6Q.gif" /><figcaption>STAC Browser with Digital Earth Africa Crop Mask collections</figcaption></figure><blockquote><strong>All in the overlays</strong></blockquote><p>💡 Next on the agenda was the most apparent step: examining the actual datasets and pinpointing the missing tiles. 🧩 To ensure the utmost accuracy in tiles and versions, we re-downloaded all mask, probability, and filtered collections 🔄📥. The user’s sample coverage also revealed complete overlap, prompting us to diligently isolate each collection with their image objects separately 🗂️🔄🧩. This meticulous approach guarantees a more refined and comprehensive geospatial data experience 🛠️🌍🚀.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ga346EA6RUUqZSu8rjQV5A.png" /><figcaption>AWS CLI command to check on data structure</figcaption></figure><blockquote><strong>Avoiding Overwrites in Earth Engine Collections 🛰️</strong></blockquote><p>When I first created Earth Engine collections, my goal was to merge 8 different image collections into one comprehensive dataset. However, I encountered a challenge with the system index, which required uniqueness. To swiftly address this, I decided to conduct a quick file name check. The process involved gathering filenames for each collection and appending them to a master list. By running a “set” function to count unique objects only once, I was able to create 8 separate collections, each with its distinct length. It was a significant breakthrough as it shed light on the root of the problem — the edge tiles causing regions to overwrite each other 🚧💡. By implementing this file name check, I could ensure the integrity of my Earth Engine collections and foster a more streamlined geospatial data experience 🛠️🗺️🚀.</p><p><strong>Fix and deploy</strong><br>In my pursuit to optimize STAC metadata for Google Earth Engine, I crafted custom code to facilitate the conversion process. The remedy was simple yet effective — I incorporated the region name at the beginning of each file name. This minor tweak enabled me to merge all the images seamlessly into a single Earth Engine image collection. Moreover, I added an “id_no” field that retained the original item id, presenting a unique opportunity. Leveraging this, I harnessed the power of the aggregate histogram tool on the “id_no” field. The results were promising, with most image ids showing a count of 1. However, a few exceptions emerged, with a count of 2, providing additional validation for our solution from both ends 📊🔍💡.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0OzFxIGyNz-mu94ErY9JXA.gif" /><figcaption>Histogram on id_no (non unique) vs system index(unique) highlighting the issue and the fix</figcaption></figure><p>The good news I was able to fix this. While there are straightforward recommendations, dataset, and catalog curations can significantly benefit from this type of additional information. Moving forward, our goal is to capture and highlight such insights, fostering a collaborative environment where users and dataset managers can learn together and propel our collective knowledge forward 📚🤝🚀. The updated and merged collections are now accessible to all, ready to empower your geospatial endeavors</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=66d4b7076826" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Google Cloud Projects and Earth Engine Integration in Universities: A Practical Guide]]></title>
            <link>https://samapriyaroy.medium.com/google-cloud-projects-and-earth-engine-integration-in-universities-a-practical-guide-fb16d8dad14a?source=rss-3a3f6dbb93ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/fb16d8dad14a</guid>
            <category><![CDATA[google-earth-engine]]></category>
            <category><![CDATA[university]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Samapriya Roy]]></dc:creator>
            <pubDate>Tue, 31 Jan 2023 05:21:22 GMT</pubDate>
            <atom:updated>2023-01-31T05:21:22.442Z</atom:updated>
            <content:encoded><![CDATA[<h3>Google Cloud Projects and Earth Engine Integration at Universities: A Practical Guide</h3><p>If you’ve followed the developments in Google’s Earth Engine, you may have noticed a shift from legacy collections to Cloud Project in your assets tab and accounts. The shift has brought about some exciting new features. One of the most notable is the improved flexibility in billing for commercial projects. Creating multiple projects within Earth Engine also allows for greater organization and collaboration.</p><p>By creating different projects, you can now invite other users with varying levels of access and permissions, as well as keep project environments separate. These updates make using Google Projects with Earth Engine more powerful and user-friendly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TLQSpGWdyECqHnhfk_Z8Lw.gif" /><figcaption>Google Cloud Project add/change project.</figcaption></figure><p>Google Cloud Projects offer many benefits, but it’s essential to keep in mind that there are also some limitations.</p><ul><li><strong>Google Cloud projects differ from Google Cloud Storage buckets (which usually require a billing account &amp; credit card).</strong> Google Cloud Storage buckets are frequently utilized with GEE accounts for storing data for ingestion or as Cloud Optimized Geotiffs for registering into an image collection. This is a crucial difference, allowing everyone to access Google projects regardless of whether they have a billable account.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lSfaBucVqy-MvUxm_mKJnQ.gif" /><figcaption>Creating a new Google Cloud Project</figcaption></figure><ul><li><strong>While Google Cloud Projects are technically free, the number of free projects you can have depends on your account and can vary greatly. </strong>You can <a href="https://support.google.com/code/contact/project_quota_increase">submit a form to get your number of projects increased </a>as needed, but the rate of success on that varies. On a personal Google account, you can expect to have around 10–20 free projects by default.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZVMTsySAXKx-gCn6NK08fw.gif" /><figcaption>Increasing the number of projects available to you</figcaption></figure><ul><li>Another thing to remember is that <strong>Google Earth Engine has changed its authentication process for use in notebooks, the python command line tool, and the SDK</strong>. Now, all GEE computing is linked to a single project at any time.</li></ul><p>In summary, while Google Projects are free for personal use, commercial entities like companies and universities may have different restrictions on creating and managing Google Project accounts.</p><blockquote>In this blog post, we will dive deeper into the process of creating a Google Project account, linking it to a separate Google account, such as a university account, and discussing limitations to keep in mind as you go through the steps. It’s important to carefully consider these limitations and plan accordingly when using Google Cloud Projects and Earth Engine, especially in a commercial or institutional setting.</blockquote><p><strong>Note: Cloud projects refer to Google Cloud Projects</strong></p><blockquote><strong><em>Balancing University Policies and Cloud Project Needs</em></strong></blockquote><p>For most university accounts, students and researchers rarely have access to the process of creating a new cloud project. While the methodology to create a cloud project in a personal and official work setup is exactly the same, it is the policies that make this process nearly impossible. Though I recommend working with your university teams if they are open to opening this feature up for you. It can range from a complete inability to access Google Cloud consoles to getting to the project creation and management page; visit <a href="http://console.cloud.google.com/">console.cloud.google.com</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JV0uoNgQEkNmgxIvlesucg.gif" /><figcaption><strong>No permissions page when you cannot go to the </strong><a href="http://console.cloud.google.com/"><strong>Google Cloud Console</strong></a></figcaption></figure><p>Another common issue is the inability for you to select an organization setup that permits you to create a Google project. For example, the two setups explored below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5afBrOfoQ-Bei1dVEnWDTA.gif" /><figcaption>Starting with the GCS console</figcaption></figure><p>In the case above, while I am able to log into the Google Cloud console, creating ,new projects may still be a problem</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vPJrSBVg0MGOVVgx2kmN-A.gif" /><figcaption>Unable to create google projects in the cloud console</figcaption></figure><p>It’s important to note that while I will be using examples of university accounts with varying degrees of permissions and policy restrictions, similar limitations and challenges can apply to many agencies and companies.</p><blockquote><strong><em>1. Creating a Google Cloud Project on your Personal account</em></strong></blockquote><p>This is relatively easier and requires you to log into the GCS console click on the projects dropdown and click on “<strong>New Project</strong>”. The project is created in a few seconds</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lSfaBucVqy-MvUxm_mKJnQ.gif" /><figcaption>Creating your Google Cloud Project</figcaption></figure><blockquote><strong><em>2. Enabling Google Earth Engine API on your Personal account</em></strong></blockquote><p>To ensure that your newly created Google Cloud Project is compatible with GEE, you’ll need to enable the Earth Engine API. To do this, navigate to the API and Services section in your Google Cloud account and click on “Enabled API &amp; Services.” From there, click on “Enable API” and search for “Earth Engine.” Once you’ve located it, enable the API. Keep in mind that it may take a few minutes for the changes to propagate. In this example, I am performing these steps on my personal account, as I have full control over the policies here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ra8w3ut2_xQjkxR4_cNgoA.gif" /><figcaption>Enabling Earth Engine API</figcaption></figure><blockquote><strong><em>3. Assigning Earth Engine Roles to Personal Cloud Projects</em></strong></blockquote><p>Earth Engine comes with several preconfigured roles and permissions, which can be found on the <a href="https://developers.google.com/earth-engine/cloud/roles_permissions"><strong>Earth Engine roles definition page</strong></a>. Although you can configure fine-grained permissions, the Earth Engine resource writer role should be sufficient for sharing the project with your university account. Keep in mind, however, that permissions may take a few minutes to propagate, so be sure to wait before attempting to add the shared Google Cloud project to your university Earth Engine account.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jmCQkgdLujwaOXUzLFlo6w.gif" /><figcaption>Adding the right Google Earth Engine role</figcaption></figure><blockquote><strong><em>4. Adding Personal Google Cloud Project to University GEE account</em></strong></blockquote><p>Once you have set up the roles, simply add the cloud project you created with the proper APIs enabled and roles defined to your university GEE account.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RPo6u_zCxF59gf2JTRID6Q.gif" /><figcaption>Adding personal Google Cloud project to university GEE account</figcaption></figure><p>This should be it; it will now grant you access to the folder for collecting your assets. In the next section, we see how asset-level and project-level configurations differ and how they can be configured properly if needed.</p><blockquote><strong><em>Optional: Configuring Asset and Project-Level Permissions for Your Personal Account</em></strong></blockquote><p>While it is possible to use your Google Earth Engine account without a Cloud project, doing so may limit your access to certain features and resources. For example, the Earth Engine resource writer role should be sufficient for basic usage; however, if you decide to use a Cloud project with your Earth Engine account or use GEE in notebook settings or while calling the REST API using the python library. You may need to grant additional permissions, such as the <strong>Service Usage Consumer </strong>role.</p><blockquote>This is not typically required for university account setups and while using the code editor. But it’s something to keep in mind if you’re planning on migrating your computation and linking it to an individual project.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nz2AaRU3N95NUO_Gry2dDQ.gif" /><figcaption>Setting Service Usage Consumer</figcaption></figure><p>If you are adding the extra role and would like to migrate your computation to a cloud project, you can now choose the project again in the user dropdown</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CG5AoDz1M7PeooxvoGnJUg.gif" /><figcaption>Adding personal project to your university GEE account</figcaption></figure><blockquote><strong><em>Wrap-Up and Key Takeaways</em></strong></blockquote><p>By linking a Google Cloud Project from your personal account to your university Google Earth Engine account, you can now easily share project spaces and resources with multiple users. One handy tip is to share the entire Google project and make the other account a co-owner, giving you full permission to manage roles and definitions from either account. This does send an email to the co-owner and you must accept the Google project. But that trick and exploring its benefit are for a different day.</p><p>If you’re using Google Earth Engine, don’t forget to check out the <a href="https://gee-community-catalog.org/"><strong>Awesome GEE Community Catalog</strong></a> for additional datasets apart from the GEE Master Catalog</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*albDyJ6tnSrK_5876b6blA.png" /><figcaption><a href="https://gee-community-catalog.org/">Awesome GEE community catalog</a></figcaption></figure><p>Explore other <a href="https://github.com/samapriya?tab=repositories"><strong>projects on my Github page</strong></a>. Follow me on <a href="https://www.linkedin.com/in/samapriya/">Linkedin</a>, <a href="https://twitter.com/samapriyaroy">Twitter</a> or <a href="https://mapstodon.space/@samapriya">Mastodon</a> to stay updated with more.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fb16d8dad14a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What’s new with the Google Earth Engine Community Catalog]]></title>
            <link>https://samapriyaroy.medium.com/whats-new-with-the-google-earth-engine-community-catalog-76e09a76a1ff?source=rss-3a3f6dbb93ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/76e09a76a1ff</guid>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[google-earth-engine]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[earth-observation]]></category>
            <category><![CDATA[geospatial]]></category>
            <dc:creator><![CDATA[Samapriya Roy]]></dc:creator>
            <pubDate>Wed, 11 Jan 2023 15:54:06 GMT</pubDate>
            <atom:updated>2023-01-11T16:57:51.699Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://medium.com/u/3a3f6dbb93ce">Samapriya Roy</a>, <a href="https://medium.com/u/fb558926cc24">Erin Dawn Trochim</a> and <a href="https://medium.com/u/aab25fff52df">Valerie Pasquarella</a></p><p>The GEE community catalog made its debut on May 8, 2020, with a commitment to accessibility and collaboration. Our earlier posts (<a href="https://samapriyaroy.medium.com/awesome-google-earth-engine-community-catalog-bd86d0ba63b8"><strong>Community Catalog</strong></a> and <a href="https://medium.com/geospatial-processing-at-scale/community-datasets-data-commons-in-google-earth-engine-8585d8baef1f"><strong>Community Datasets &amp; Data Commons</strong></a>) provide more information about the motivations behind the project. This round-up aims to share updates and changes and give the community more opportunities to work together. This article summarizes critical facets of what has changed over the last couple of years and introduces you to new features. Visit the <a href="https://gee-community-catalog.org/"><strong>catalog here</strong></a><strong> </strong>and find, share and sponsor the <a href="https://github.com/samapriya/awesome-gee-community-datasets"><strong>GitHub project</strong></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*albDyJ6tnSrK_5876b6blA.png" /><figcaption>Awesome GEE community catalog logo</figcaption></figure><p>The Awesome GEE Community Catalog provides various ways to contribute data and datasets, as well as opportunities to explore the catalog’s features and enhance core capabilities. These include contributing to example repositories, ensuring compliance with licenses and proper tagging, and utilizing keyword and tag search capabilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*Qay3CWdBXB1lHX29dB2aLw.png" /><figcaption>Awesome GEE Community Catalog Features and contribution workflows</figcaption></figure><blockquote><strong><em>Getting to a 1000 datasets </em></strong><em>🤘️</em></blockquote><p>We are excited to share that the Google Earth Engine Community Catalog now has over 1,000 datasets, with 1,112 currently hosted. This milestone was achieved towards the end of last year, and the catalog continues to grow with contributions of open datasets. As of today, the catalog includes 112 TB of data and billions of features. The community catalog is steadily expanding, and we are thrilled to see so many valuable resources and datasets being shared. We are also at 432 ⭐️ and 76 forks and about 40,000 views per month and growing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JT5Kqds-4v9mWA-_wcnnZQ.png" /><figcaption>Catalog stats and key features</figcaption></figure><h3>Datasets Highlights</h3><p>Over the last year, we have added some impressive ones, including</p><blockquote><a href="https://samapriya.github.io/awesome-gee-community-datasets/projects/usa_structures"><strong>USA structures</strong></a></blockquote><p>The U.S. Department of Homeland Security (DHS), Federal Emergency Management Agency (FEMA), Oak Ridge National Laboratory (ORNL), and the U.S. Geological Survey worked together to establish the first comprehensive inventory of all structures larger than 450 square feet. This inventory is intended to support Flood Insurance Mitigation, Emergency Preparedness, and Response efforts. The building outline inventory was created through a partnership between FEMA and DHS Science and Technology, with ORNL leading the charge in extracting building outlines using commercially available satellite imagery. This valuable resource is available for <a href="https://gis-fema.hub.arcgis.com/pages/usa-structures"><strong>download on this link</strong></a> or can be explored <a href="https://fema.maps.arcgis.com/home/item.html?id=0ec8512ad21e4bb987d7e848d14e7e24#overview"><strong>through this link</strong></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/895/1*6vFp00BP9thrnVhwSDec3A.gif" /><figcaption>USA structures layer</figcaption></figure><blockquote><a href="https://samapriya.github.io/awesome-gee-community-datasets/projects/hydro90"><strong>Hydrography 90m Layers</strong></a></blockquote><p>The Hydrography 90 layers utilize the MERIT Hydro digital elevation model, which has a resolution of 3 arc seconds (approximately 90 meters at the equator), to create a seamless and standardized global hydrographic network, known as “<strong>Hydrography90m</strong>”. This includes topographic and topological information on stream networks. A unique aspect of this network is the use of a minimal upstream contributing area of 0.05 square kilometers (or 5 hectares) as the threshold for identifying a stream channel. This approach allows for the detailed extraction of even the smallest headwater streams. Find additional details <a href="https://essd.copernicus.org/articles/14/4525/2022/essd-14-4525-2022.pdf">in the paper here</a>. The datasets can <a href="https://hydrography.org/hydrography90m/hydrography90m_layers/">be downloaded here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xcwNvGtnfPEd8JPgO4JNWg.gif" /><figcaption>Hydrography 90m layers</figcaption></figure><blockquote><a href="https://samapriya.github.io/awesome-gee-community-datasets/projects/maxar_opendata"><strong>MAXAR Open Data Events: Multispectral</strong></a></blockquote><p>High-resolution satellite imagery, both pre and post-event, is provided to assist with emergency planning, risk assessment, monitoring of emergency response and recovery efforts, and damage assessment. These images are generated through the use of the Maxar ARD pipeline and are organized and optimized for cloud usage. As events occur, new data is released, and older data may be updated to the ARD format as necessary. Additional information and available datasets can be found by <a href="https://www.maxar.com/open-data"><strong>visiting this link</strong></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vv7NROUdG3tZAM4glATZWQ.gif" /><figcaption>MAXAR Open Data Events</figcaption></figure><h3>Insiders Program</h3><p>The <strong>GEE Community Catalog Insiders program</strong> is dedicated to those who support the sustainability of open-source projects and contribute to the growth and curation of the catalog. The program is open to sponsors and data contributors to the project. If you fall under any of these categories, <a href="https://forms.gle/VPmETGKyvMAd37MV6"><strong>please fill out this form </strong></a>to gain Insider access. By signing up for the Insiders program, you will gain access to the following benefits:</p><ul><li>Joining a Google Group called “<strong>catalog-contributors,</strong>” which grants you early access to specific datasets as they are being processed, reviewed, and released.</li><li>Receive periodic changelog and update emails.</li><li>Opportunities to post questions, concerns, and thoughts on the Google group</li></ul><p>Your support for the project is highly appreciated, and you can show your support by sponsoring the project or contributing to data curation for the catalog. The Insiders program has just added two new datasets as part of its offerings starting this new year.</p><p><em>All datasets that are part of the insiders’ program are released within a month to the general catalog, along with the monthly release cycle for the catalog.</em></p><blockquote><a href="https://gee-community-catalog.org/projects/hrdem/"><strong>Canada High-Resolution Digital Elevation Model (HRDEM)</strong></a></blockquote><p>The High-Resolution Digital Elevation Model (HRDEM) is a product created from a combination of airborne LiDAR data and satellite imagery, with a focus on LiDAR data primarily in the southern regions and satellite imagery in the northern regions. The coverage of the Canadian territory is being gradually expanded. The HRDEM includes a Digital Terrain Model (DTM), a Digital Surface Model (DSM), and other derived data. Derived data for the DTM datasets include slope, aspect, shaded relief, color relief, and color-shaded relief maps. For the DSM datasets, derived data include shaded relief, color relief, and color-shaded relief maps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WoqI2k0L5M8GE1AQTck9DA.gif" /><figcaption>Canada High Resolution DEM (HRDEM)</figcaption></figure><blockquote><a href="https://gee-community-catalog.org/projects/msroads/"><strong>Microsoft Bing Global Mined Roads</strong></a></blockquote><p>Bing Maps is unveiling a global collection of mined roads. Through our detection methods, we have identified a total of 47.8 million kilometers of roads, with an additional 1165 thousand kilometers of roads that were previously missing from OpenStreetMap (OSM). Our mining process utilizes Bing Maps imagery collected between 2020 and 2022, including imagery from Maxar and Airbus. The datasets were provided in TSV formats, and we have also applied additional steps to convert them into a format that is compatible with Google Earth Engine (GEE).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E5nA0OxXOvBPpt-TJpV8bg.gif" /><figcaption>Microsoft Bing Global Mined Roads</figcaption></figure><p>We have put a lot of effort and care into creating this community catalog, and we plan to keep expanding and enhancing it. Our Insiders program and upcoming datasets will continue to evolve with the help of user input and enhancements to make them more automated and accessible. Please help spread the word about the catalog and consider supporting it by sharing your own datasets or contributing to the ones that are already available. Here’s to an amazing 2023 and working together to build a stronger community resource.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=76e09a76a1ff" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Awesome Google Earth Engine Community Catalog]]></title>
            <link>https://samapriyaroy.medium.com/awesome-google-earth-engine-community-catalog-bd86d0ba63b8?source=rss-3a3f6dbb93ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd86d0ba63b8</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[earth-observation]]></category>
            <category><![CDATA[google-earth-engine]]></category>
            <dc:creator><![CDATA[Samapriya Roy]]></dc:creator>
            <pubDate>Fri, 07 Oct 2022 14:08:46 GMT</pubDate>
            <atom:updated>2022-10-07T14:08:46.840Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://medium.com/u/3a3f6dbb93ce">Samapriya Roy</a>, <a href="https://medium.com/u/aab25fff52df">Valerie Pasquarella</a> and <a href="https://medium.com/u/fb558926cc24">Erin Dawn Trochim</a></p><p>About two years ago a simple movement toward data commons and the need for accessibility to publishing open datasets was born over a data request. The idea was a simple one, what if Google Earth Engine users, community actors, researchers, and domain experts could help enrich the ever-growing need for more data? All this while building a searchable data catalog for their own needs. The <a href="https://gee-community-catalog.org/"><strong>awesome GEE community catalog</strong></a> lives and serves alongside the <a href="https://developers.google.com/earth-engine/datasets/catalog">Google Earth Engine data catalog</a> and also houses datasets that are often requested by the community and under a variety of open licenses. Still drawing inspiration from the core ideas of <strong><em>how commons across different societies are managed by those who build and are a part of them</em></strong>. You can read more about <a href="https://medium.com/geospatial-processing-at-scale/community-datasets-data-commons-in-google-earth-engine-8585d8baef1f">how we started here</a>.</p><p>I like to believe that “<em>Communities are what communities build together</em>”. Thank you for all the support we hope if you find this work useful you will ⭐ <a href="https://github.com/samapriya/awesome-gee-community-datasets"><strong>the GitHub repository</strong> </a>it really supports us and lets others know that this is a well-used resource in the community.</p><blockquote><strong>Getting down to Numbers</strong></blockquote><p>For the last two years, the community catalog has grown with over 952 datasets and over 100+ TB of community-contributed and available datasets. We have made improvements and redesigned how to make this catalog more searchable and usable by most users. This includes a lot of feedback and learning from the community. This article focuses on the current structure of the community catalog, improvements, navigation, and getting to where you need to be. We also dive deeper into how you can contributeto the catalog in more than one way and help this grow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1021/1*LOu8HuxNtRSIUeBanfkkRw.jpeg" /><figcaption>Community Catalog Stats from 2022–10–03</figcaption></figure><blockquote><strong>Finding a home</strong></blockquote><p>We have a <a href="https://gee-community-catalog.org/"><strong>new website</strong></a>. Behind the scenes, most of the data crunching, cleanup, and analysis to get the data ready for the community catalog happens using <a href="https://medium.com/u/fa022bd81c97">National Science Foundation</a>, funded <a href="https://jetstream-cloud.org/">Jetstream 2</a> project. More details to follow behind the scenes in a later article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1t1yLmP5J2QKCNgpllgasw.png" /><figcaption><a href="https://gee-community-catalog.org/">Awesome GEE Community Catalog logo</a></figcaption></figure><p>For now, you can head over to the community catalog at</p><p><a href="https://gee-community-catalog.org/"><strong>https://gee-community-catalog.org/</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ONJ0b1EpQsIHjmjk4NpF2A.gif" /></figure><blockquote><strong>Getting Started</strong></blockquote><p>For months we have focused on streamlining how users can contribute to and navigate the catalog. We have made improvements based on these, and we are hoping the getting started section allows you to focus on navigation and provides direct links to templates we use for interaction. We have new templates for everything from <strong>adding new datasets, reporting bugs, and helping us build a series of examples </strong>fitting these datasets. Find the templates and more in the <a href="https://gee-community-catalog.org/">Getting started section</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xBX5YIuH7bM9RsiDCIG60A.gif" /><figcaption><a href="https://github.com/samapriya/awesome-gee-community-datasets/issues/new?assignees=samapriya%2C+valpasq%2C+edtrochim&amp;labels=data+request&amp;template=suggest-new-community-data-for-awesome--gee-catalog.md&amp;title=Dataset+%26+Curator+Name">Sample Template to add new datasets to the community catalog</a></figcaption></figure><blockquote><strong>The Catalog Examples Repo in GEE</strong></blockquote><p>An obvious thing about learning from great examples is knowing how the world expects you to build what works. We learned from the Examples folder in GEE and wanted to push ourselves to collect and make available all the datasets through examples. Now you can add the GEE repository and have it updated automatically as we add new datasets.</p><p>The repository should appear in your reader’s repo list, and you might have to refresh once if you don’t see it. Use this <a href="https://code.earthengine.google.com/?accept_repo=users/sat-io/awesome-gee-catalog-examples"><strong>link to add the examples repo from the community catalog to your GEE repos</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hS-3kJg-nRnkT8HEireScA.gif" /><figcaption>Adding and exploring the GEE community catalog repo</figcaption></figure><blockquote><strong>Finding the right fit: Searching the Catalog</strong></blockquote><p>The datasets are grouped under domains; you can expand them to get to an individual dataset of interest.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cuhOCNhdT-qBE8HqXOmUhg.gif" /><figcaption>Using domains and groups to look for data</figcaption></figure><p>You can also search using the search bar with keywords, tags, and even author names in the paper and sources of these datasets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w8hE3B6AFliwv8UPoiHX_w.gif" /><figcaption>Using the search tool to look for keywords, tags, authors &amp; more</figcaption></figure><blockquote><strong>Anatomy of a dataset page</strong></blockquote><p>The dataset pages in the catalog have a general outline. Usually, a page is split into</p><ul><li><strong>Description</strong>: Includes description of the dataset(s) and links to the paper and the source files for the datasets</li><li><strong>Dataset citation</strong>: We love what community members produce, so citations are crucial and critical. This can be both a paper citation as well a dataset citation that is provided</li><li><strong>Earth Engine snippets</strong>: This is designed to introduce you to all assets that are part of the dataset. This could be all collections, tables, images, and so on.</li><li><strong>Sample Script</strong>: It was essential to add a way for users to get started with these datasets, so the sample scripts are designed to help with just that.</li><li><strong>License</strong>: There are hundreds of grants available across different datasets, and it is important to highlight which datasets are covered are which specific license before use.</li><li><strong>Keywords &amp; Curation</strong>: While some users have ingested their datasets into GEE, curation information and who curates the datasets are essential in case their access is modified or things change.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-wxhutAu6Mgs_GrI_p5YJg.gif" /><figcaption>Anatomy of a single dataset (ASTER GDEM v3)</figcaption></figure><blockquote><strong>Release &amp; Citation</strong></blockquote><p>While we keep adding data continuously, we hope to create releases once every month, which would tie to our catalog citations. We have had many questions on how to cite our catalog, and we hope this allows users like you a quick way to do just that apart from using the data citations listed and highlighted in the dataset pages.</p><pre><strong>Samapriya Roy, Erin Trochim, Alec L. Robitaille, &amp; Valerie Pasquarella. (2022). samapriya/awesome-gee-community-datasets: Community Catalog(1.0.0). Zenodo. </strong><a href="https://doi.org/10.5281/zenodo.7144934"><strong>https://doi.org/10.5281/zenodo.7144934</strong></a></pre><p>This has been a labor of love, and we hope to continue building and enriching the community catalog. There will be lessons we learn, improvements we make, and, more importantly, people we meet and friends we make, just like at the Geo for Good conference this year, 2022. Look out for the Geo for Good session video on this and <a href="https://earthoutreachonair.withgoogle.com/events/geoforgood22">more to be posted here</a>. Also, look for the medium series on how to get your data into Google Earth Engine lessons from the community catalog series to follow.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd86d0ba63b8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Microsoft Building Footprints in GEE: Revisiting Scale & accessibility]]></title>
            <link>https://samapriyaroy.medium.com/microsoft-building-footprints-in-gee-revisiting-scale-accessibility-eee5e97c17a3?source=rss-3a3f6dbb93ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/eee5e97c17a3</guid>
            <category><![CDATA[google-earth-engine]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[geospatial]]></category>
            <category><![CDATA[microsoft]]></category>
            <dc:creator><![CDATA[Samapriya Roy]]></dc:creator>
            <pubDate>Wed, 01 Jun 2022 06:32:42 GMT</pubDate>
            <atom:updated>2022-10-07T14:05:37.034Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dwbxoqDRUI-74YyFrCgCAA.gif" /></figure><p>Building footprints are probably one of the most visible modifications of the natural landscape. Building types vary, but the built class and the overall patterns of how these spaces evolve are tied to human action, growth, and decay. Mapping human settlements are not new and everything from Night Lights to open street map generated massive feature maps that have helped expand this understanding. These are inherently tied to census features, roads, and distribution networks and serve as an effective means of understanding migration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T5FWg1e426NaXdlGvXpNsQ.png" /><figcaption><a href="https://gena.users.earthengine.app/view/urban-lights">Urban Night Lights App</a></figcaption></figure><p>So why an entire story for a dataset, you might ask? After all, we have been running <a href="https://samapriya.github.io/awesome-gee-community-datasets/"><strong>the community catalog</strong></a> for over a year now with over 100 TB+ and 850+ dataset types. The story goes back to accessibility and what it took to navigate from in-memory reads to streaming GeoJSONs to understanding what really fits the user experience.</p><p>The <strong>MSBuildings dataset </strong>that I have ingested into Google Earth Engine includes earlier releases apart from Microsoft’s <a href="https://github.com/microsoft/GlobalMLBuildingFootprints"><strong>777 Million Global building footprints</strong></a> and, in its final state, stands at <strong>1 Billion+ footprints (1,069,059,359)</strong>. This is perhaps the largest vector ingest that I have ingested from my end, and the three subfolders include the United States, Indonesia, and Nigeria. In addition, datasets such as Canada and Australia were merged into single vector composites.</p><blockquote><strong><em>High-Resolution Building Footprint</em></strong></blockquote><p>Large-scale building footprint mapping was an exercise in diving deep into finer grain understanding of building shape, size, and patterns. With Open Street Map, users were able to digitize the buildings of choice and improve the richness of existing databases on building information. You can extract OSM extract using sites like <a href="https://extract.bbbike.org/">bbbike extractor</a> or <a href="https://download.geofabrik.de/">geofabrik</a>. Unfortunately, while many neighborhoods, municipalities, and local governments further have this dataset, there are no cohesive and easy approaches to gathering them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UGSCNtTix8Z-JOSVwUeQIg.png" /><figcaption>Open Street Map Export</figcaption></figure><p>So in 2018, Microsoft started experimenting with releasing Building Footprint under an open license for consumption covered by the likes of the <a href="https://www.nytimes.com/interactive/2018/10/12/us/map-of-every-building-in-the-united-states.html">New York Times</a>. This is one of the most significant single approach releases for a building dataset, followed by Microsoft releasing this across multiple countries and regions. Further updates and <a href="https://blogs.bing.com/maps/2022-01/New-and-updated-Building-Footprints">release notes followed</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i1YDe29C7Ofx8zwzaSftug.jpeg" /><figcaption>Source <a href="https://www.nytimes.com/interactive/2018/10/12/us/map-of-every-building-in-the-united-states.html">NYT A Map of Every Building in America Oct 12, 2018</a></figcaption></figure><p>Google also released the <a href="https://ai.googleblog.com/2021/07/mapping-africas-buildings-with.html"><strong>continent of Africa dataset in 2021</strong></a> marking one of the most significant releases under a <a href="https://creativecommons.org/licenses/by/4.0/">CC-BY 4.0</a> license and as part of Google’s Open Buildings initiative.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BA3i6AxUvGv9JPyr5eGBkQ.jpeg" /><figcaption>Example of segmenting buildings in satellite imagery from <a href="https://ai.googleblog.com/2021/07/mapping-africas-buildings-with.html">Google AI Blog</a></figcaption></figure><p>Performance across different approaches and models varies as expected and the type of data you need or want to use depends on the application in mind.</p><p><strong><em>Microsoft Global Building Footprint Dataset</em></strong></p><p>The MS global datasets contain 777M buildings from Bing Maps imagery between 2014 and 2021 including Maxar and Airbus imagery. The dataset does not include earlier releases like Canada, the US, Australia, and so on and is massive for direct use and application. Turns out you can search for all their releases simply using <a href="https://github.com/orgs/microsoft/repositories?q=Building+Footprint&amp;type=all&amp;language=&amp;sort=">Github keywords like this.</a> Not only that, you can get to the download links pretty easily.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Df44x3m-s512MattF_c4CA.gif" /><figcaption>Global ML extract links</figcaption></figure><blockquote><strong><em>Understanding the Source data: Sort and Split</em></strong></blockquote><p>For those who have worked with a few of these datasets, here are a few quick observations</p><ul><li>The size of data subsets can range from a few KB to multiple GB. While we have come a long way to reduce client-side dependency, this could still challenge those with limited hardware and bandwidth.</li><li>The global ML release links to earlier releases that can be aggregated to ensure the collection is truly global and complete.</li><li>While some datasets are released as GeoJSON some are released in Large GeoJSON format (GeoJSONL). While the zipped sizes are large enough to limit the type of hardware, the unzipped extracts are massive vector files. Hence the need to sort and split datasets.</li></ul><blockquote><strong>Google Earth Engine &amp; Vector Spaces</strong></blockquote><p>Google Earth Engine is powerful for raster analysis and has periodically improved vector capabilities, including releases such as FeatureView for quick tile rendering. However, table or vector ingests are limited to specific file types, including CSVs and Shapefiles. This means you have to get the files into either of these formats.</p><p>There are some interesting performance behaviors across Ingest.</p><ul><li>Ingest times are not necessarily linear across file sizes; it seems a complex geometry can take longer to ingest though this is not a consistent enough generalization.</li><li>Extensive datasets were split into smaller subsets and ingested.</li><li>Once the ingestion was completed, sub-parts in a folder could be merged, flattened, and exported with varying success.</li></ul><blockquote><strong>Access to Accessibility</strong></blockquote><p>Accessibility goes beyond just making the data available; it can be limited by what you can and cannot do with any data. The attempt here was to make it at least available primarily to some users and capture the steps needed to create your own pipeline. For now, you can <a href="https://samapriya.github.io/awesome-gee-community-datasets/projects/msbuildings/"><strong>read about this dataset and access it here.</strong></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iAgZfymb1SNG0w9QdV7k-A.gif" /><figcaption>Chile Buildings Global Buildings Footprint</figcaption></figure><p>At 201 objects, the folder’s total size, including all datasets, is 62.03 GB, which might be owing to compression and optimization within Google Earth Engine. I am eager to see if users create their own subset <a href="https://medium.com/google-earth/map-tiles-comin-in-hot-61019f48c105"><strong>feature view assets</strong></a>, which came out a few weeks ago. The entire process was a fascinating deep dive into challenges in data subsetting, flattening, task failures, and reattempts. In the end, you have a <a href="https://samapriya.github.io/awesome-gee-community-datasets/projects/msbuildings"><strong>complete Global ML Building Footprint</strong></a> dataset from Microsoft within Google Earth Engine, and there is more to come in the<a href="https://samapriya.github.io/awesome-gee-community-datasets/"><strong> awesome GEE community datasets catalog</strong></a>.</p><p>If you like the effort, <a href="https://github.com/samapriya/awesome-gee-community-datasets/"><strong>Star the project to get more updates</strong></a> and support the community. You can also <a href="https://twitter.com/intent/follow?screen_name=samapriyaroy"><strong>follow me on Twitter</strong></a> for more frequent updates.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eee5e97c17a3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Open Ocean Data: with Argofloats Tool]]></title>
            <link>https://samapriyaroy.medium.com/open-ocean-data-with-argofloats-tool-50efef6c4f05?source=rss-3a3f6dbb93ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/50efef6c4f05</guid>
            <category><![CDATA[oceans]]></category>
            <category><![CDATA[sensors]]></category>
            <category><![CDATA[ocean-conservation]]></category>
            <category><![CDATA[open-data]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Samapriya Roy]]></dc:creator>
            <pubDate>Tue, 18 Jan 2022 14:47:45 GMT</pubDate>
            <atom:updated>2022-01-18T14:47:45.624Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IHqBfqAbqgt3Tc2yyhk3qQ.jpeg" /><figcaption>One of many Argo floats collecting data from open oceans (courtesy: <a href="https://www.climate.gov/news-features/features/argo-revolution">https://www.climate.gov/news-features/features/argo-revolution</a>)</figcaption></figure><p>Imagine some of the harshest operational conditions in the world, with swings in temperature and pressure. A world that is profound and yet vastly less explored than most missions undertaken by humanity. and yet something we rely on every single day. With the need for more ocean data and continuous monitoring, the Argo floats and Argo program deployed the first Argo float in the year 2000. Since then these drifting sentinels of ocean data, these record keepers have numbered in the thousands and with every cycle have created a story that measures the oceans in millions of space and time points.</p><blockquote><strong>The Argo Network</strong></blockquote><p>The Argo network is a global array of autonomous floats or profilers, deployed across the world’s oceans, and with some 4000 total floats they measure temperature, &amp; salinity and relay that via satellite transmission links to data centers. The International Argo program is further integrated into the <a href="https://www.copernicus.eu/en">Copernicus</a> program and the <a href="http://www.earthobservations.org/gci.php">Global Earth Observation System of Systems (GEOSS)</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dZxAxP1MOBI7FsfpN78uhQ.png" /><figcaption>Argovis generated 3 day snapshot of Argo floats</figcaption></figure><p>Each instrument (float) spends almost all its life below the surface. The name Argo was chosen because the array of floats works in partnership with the Jason earth-observing satellites that measure the shape of the ocean surface. (In Greek mythology Jason sailed on his ship the Argo in search of the golden fleece). To learn more about Argo, how it works, <a href="https://argo.ucsd.edu/">its data and technology, and its scientific and environmental impact, click here</a>. The best part all of this data is open source and available for use.</p><blockquote><strong>Argovis and accessing data</strong></blockquote><p>Argo floats networks have received tremendous support from uses and across different groups who have created such wonderful libraries like <a href="https://argopy.readthedocs.io/en/latest/"><strong>argopy </strong></a>and <a href="https://cran.r-project.org/web/packages/argoFloats/index.html"><strong>ArgoFloats for R</strong></a> to explore the datasets from these thousands of floats and millions of profiles. Argovis was created and group at the University of Boulder, Colorado and was designed to “Access Argo profiles via API, or visualize temperature, salinity, and BGC data by location”. You can check out the interface at <a href="https://argovis.colorado.edu/">https://argovis.colorado.edu/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gErUQOzZexWQWlrN7685WA.png" /><figcaption>Argovis interface</figcaption></figure><p>Before we go any further, you will quite often hear those two terms so easy to get that cleared, each float is known as a platform (a platform is usually anything that carries a sensor or an array of sensors). Each time a float submerges and resurfaces it measure a profile across time and across a depth profile in the ocean, these are referred to as profiles or platform profiles.</p><blockquote><a href="https://samapriya.github.io/argofloats/"><strong>Argofloats Python tool</strong></a></blockquote><p>The<a href="https://github.com/samapriya/argofloats/"><strong> Argofloats python tool</strong></a> was generally influenced by making these datasets easily searchable and exportable to be used alongside other open ocean datasets. I created this from the <a href="https://argovis.colorado.edu/docs/Argovis_API.html">Argovis API</a> and is a work in development to support the efforts to make these more searchable and quickly usable as a command-line tool. You can find the <a href="https://samapriya.github.io/argofloats/"><strong>latest docs here</strong></a> and the <a href="https://github.com/samapriya/argofloats/"><strong>GitHub page here</strong></a><strong>.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VlD0eSFETMofMQ7DlY34iw.gif" /><figcaption>argofloats pypi main and readme redirect to docs</figcaption></figure><blockquote><strong>Getting stated with argofloats python package: Installation</strong></blockquote><p>This assumes that you have native python &amp; pip installed in your system, you can test this by going to the terminal (or windows command prompt) and trying</p><p><strong>python</strong> and then <strong>pip list</strong></p><p>To install <strong>argofloats: Simple CLI for ArgoVis and Argofloats</strong> you can install using two methods.</p><p><strong>pip install argofloats</strong></p><p>or you can also try</p><pre><strong>git clone https://github.com/samapriya/argofloats.git<br>cd argofloats<br>python setup.py install</strong></pre><p>For Linux use sudo or try <strong>pip install argofloats --user</strong>.</p><p>I recommend installation within a virtual environment. Find more information on <a href="https://docs.python.org/3/library/venv.html">creating virtual environments here</a>. For more detailed instructions visit the <a href="https://samapriya.github.io/argofloats/installation/"><strong>installation page here</strong></a></p><blockquote><strong>Quick overview tool</strong></blockquote><p>The overview tool fetches the latest information about the platform and profile count, last updated as well as the distribution of datasets across the different data centers or DACs. There are no arguments for this tool and it simply serves up existing database information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-zqWdz0oV0DsOGc3MOUr3Q.gif" /><figcaption>argofloats overview tool</figcaption></figure><blockquote><strong>Platform and Profile Metadata</strong></blockquote><p>The platform and profile metadata tools are designed around providing the user with specific information about a given platform or a profile associated with a platform. For platforms, the tool fetches metadata about a platform as pretty prints the information as a JSON object. This does require a Platform ID also called WMO number for the Argo float.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LEwrUWaqA11oH8cwj-7gpw.gif" /><figcaption>platform metadata tool</figcaption></figure><p>Each platform consists of profiles where each profile is attached to a platform and is a single cycle of data collection. So platforms can have multiple profiles and are generally represented as PlatformID_ProfileNo. This is the argument used by the tool and it pulls the metadata for that specific profile for that platform/Argo float.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BVkQqIVK0VLFzAWSeat2xA.gif" /><figcaption>profile metadata tool</figcaption></figure><blockquote><strong>Argovis Platform and Profile exploration</strong></blockquote><p>Argovis makes it really easy for a user to follow a platform and export both the platform level and/or profile level data from the interface. The user can either click on a single platform or draw an area to choose multiple profiles. The user can also change the date ranges to expand the search</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pMIebWtLyVaujcBr_m_mqQ.gif" /><figcaption>argovis platform and profile exploration</figcaption></figure><p>The argofloats tool was built on the same principle of being able to select using a single platform or a profile id or even AOI. It is also possible to do an all-time export of all profiles associated with a single platform. This also includes date selection similar to argovis as shown below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xplDbUgP_PpLTAg1X7ZC0g.gif" /><figcaption>argovis area and date select and export</figcaption></figure><blockquote><strong>Argofloats tool Platform Profiles and Profiles export</strong></blockquote><p>The <strong>Platform profile tool</strong>: Each platform consists of profiles where each profile is attached to a platform and is a single cycle of data collection. So platforms can have multiple profiles and are generally represented as PlatformID_ProfileNo. This tool fetches all platforms linked to a specific platform and exports it as a single CSV including platform profile id and measurements. If no path is provided the profile CSV is saved onto the home folder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a5BH0QDnjDEhcQrIUjURdg.gif" /><figcaption>platform profiles tool to export all profiles associated with a platform</figcaption></figure><p>The profiles export tool is more general-purpose and allows you to search the Argo floats database using either lat long and a buffer area, or a geometry.geojson file or a given profile id. This tool is also capable of running long-time searches overcoming the 3-month limit constrained by the argovis API. All use cases are shown below. The outputs are written as CSV file with a prefix <strong><em>argoprofile_</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fF7bIDyI1N8vbiXtp02CkQ.gif" /><figcaption>argofloats tool profile export tool using platform profile id</figcaption></figure><p>Using the point geometry you can supply a lat lon and a buffer to create a search and export results</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rP9F3N3O1FCcU-Q4_LOuVg.gif" /><figcaption>argofloats tool profile exporter: point search only</figcaption></figure><p>The area search allows you to parse using a geojson file instead of a point and buffer setup</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DuHR8MU5xDwrJMEfaUJKsA.gif" /><figcaption>argofloats tool profile exporter: area search using GeoJSON</figcaption></figure><blockquote><a href="https://open-oceans.github.io/"><strong>Open Oceans project</strong></a><strong> and Google Earth Engine (GEE) compatability</strong></blockquote><p>This tool is part of the <a href="https://open-oceans.github.io/"><strong>open oceans project</strong></a> I started a few weeks ago to help curate open oceans tools and datasets. As I was building the data sandbox in Google Earth Engine, I decided to make all exports from this tool GEE compatible, so you can import these CSVs directly into Earth Engine for analysis as well as locate and find other datasets that might be collocated in time and space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vuIC3CdyM8uI1xaAa1hPjg.gif" /><figcaption>argofloats tool exported data in Google Earth Engine for quick analysis</figcaption></figure><p>I hope to continue developing this tool with support from and suggestions from the user community. As I said in my earlier blog on <a href="https://samapriyaroy.medium.com/open-ocean-data-with-aqualink-pyaqua-32fb4d99c837"><strong>aqualink &amp; pyaqua</strong></a>, building a better understanding of oceans is critical, it is tied to measuring challenging amounts of data from harsh environments.</p><blockquote><em>Access isn’t the same as accessibility</em></blockquote><p>There is always more than one way for users to access, read and then follow up on the data. Asking the right questions, tinkering on solutions, and building as a community are what allow growth in different domains. You can also <a href="https://twitter.com/samapriyaroy"><strong>follow me on Twitter to get more updates </strong></a>as I build more for the open oceans.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=50efef6c4f05" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Open Ocean Data: with Aqualink & pyaqua]]></title>
            <link>https://samapriyaroy.medium.com/open-ocean-data-with-aqualink-pyaqua-32fb4d99c837?source=rss-3a3f6dbb93ce------2</link>
            <guid isPermaLink="false">https://medium.com/p/32fb4d99c837</guid>
            <category><![CDATA[ocean-conservation]]></category>
            <category><![CDATA[oceans]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[coral-reefs]]></category>
            <category><![CDATA[open-data]]></category>
            <dc:creator><![CDATA[Samapriya Roy]]></dc:creator>
            <pubDate>Mon, 27 Dec 2021 08:28:28 GMT</pubDate>
            <atom:updated>2022-01-18T09:05:46.904Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b0P5Cvf6kpYBzJWRI6IzQA.jpeg" /><figcaption>Shantar Islands and part of Uda Bay in the western Sea of Okhotsk (Landsat 8 Courtesy NASA)</figcaption></figure><p>Measuring near-real-time data over open oceans over the last couple of decades has been mostly challenging. Over 80% of our oceans are unexplored and as the impetus to expand and understand these cyclically connected systems grow, getting to data is going to be critical. While overland systems are more easily observable ocean dynamics have proven to be complex and required a complex array of investment into both hardware and software architecture.</p><p>These include modes of measurement that are expected to survive some of the harshest environments on our planet. By measuring parameters like wind height and ocean temperature open ocean systems provide a much-needed understanding and capability to predict massive shifts and changes to marine ecosystems.</p><blockquote><strong>Introducing Aqualink</strong></blockquote><p>Over the last summer, I was introduced to <a href="https://aqualink.org/"><strong>aqualink.org</strong> </a>a philanthropic engineering organization working on building ocean conservation technology and working with partners like <a href="https://www.sofarocean.com/"><strong>Sofar Ocean</strong></a> to build an open ocean data network. You can read their own medium post: <a href="https://medium.com/aqualink/introducing-aqualink-dd1023393b8"><strong>Introducing Aqualink</strong> here</a>. The best part, they decided to keep the data completely open and available for everyone to use.</p><blockquote><strong>Getting started with Aqualink Map</strong></blockquote><p>Aqualink provides direct access to a dashboard map hybrid. You can sign up for a free account right now but it can be used completely without any signups required.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5KIUnKbAUqkJcJp9QaZq9g.gif" /><figcaption>Aqualink Signup</figcaption></figure><p>With or without the signup process, click on the view map to show you a quick representation of the overall sites of interest for now. Note how there is a slider button that allows you to only show sites with actively deployed buoys.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dex6IIowb5t2GBmrBgP0PQ.gif" /><figcaption>Aqualink map and deployed buoys only filter</figcaption></figure><p>Deployed buoys have near real-time temperatures relayed back from these devices which generate a sea surface as well as the temperature at depth readings.</p><blockquote><a href="https://samapriya.github.io/pyaqua/"><strong><em>pyaqua: Simple Command Line Interface (CLI) for A</em>qualink org</strong></a></blockquote><p>The <strong>pyaqua</strong> command-line tool written in python is not an official tool but an unofficial one that allows users to interact with the aqualink.org map and is designed to work with sites and locations with buoys. <em>The motivation to build this tool was to allow users to programmatically interact with the aqualink data dashboard and map as well as allow users to explore and filter data as needed for further analysis.</em></p><p>The blog performs a side by side example application of both the map and the command-line tool for the user to follow</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6zm-Od_Lh5dbnOpQ46FCoA.png" /><figcaption>pyaqua command-line tool</figcaption></figure><p>This tool is useful if you have more than one site of interest and includes some useful tools for interacting with and exporting data. Since it is not built on any available or standard API, some things might break over time and I will try my best to maintain and improve on this over time as well. <strong>You can find the </strong><a href="https://github.com/samapriya/pyaqua"><strong>GitHub page for the project here</strong></a><strong> and the most updated version of the </strong><a href="https://samapriya.github.io/pyaqua/"><strong>readme on the readme page here</strong></a><strong>. </strong>You can star and follow the project for more updates as updates and releases are made.</p><blockquote><strong>Installing pyaqua</strong></blockquote><p>This assumes that you have native python &amp; pip installed in your system, you can test this by going to the terminal (or windows command prompt) and trying</p><p><strong>python</strong> and then <strong>pip list</strong></p><p><em>pyaqua only supports Python v3.7 or higher</em></p><p>You can install using two methods.</p><p><strong>pip install pyaqua</strong></p><p>or you can also try</p><pre><strong>git clone </strong><a href="https://github.com/samapriya/pyaqua.git"><strong>https://github.com/samapriya/pyaqua.git</strong></a><strong><br>cd pyaqua<br>python setup.py install</strong></pre><p>For Linux use sudo or try <strong>pip install pyaqua --user</strong>.</p><p>I recommend installation within a virtual environment. Find more information on <a href="https://docs.python.org/3/library/venv.html">creating virtual environments here</a>.</p><blockquote><strong>Accessing site level data on aqualink map</strong></blockquote><p>Site-level data are available for multiple sites and again here we are selecting from the list of deployed buoys to make sure we have both remote sensing estimates as well as in situ measurements of ocean conditions at locations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NIZ1sLpyx4TlVOZ9i-q2jw.gif" /><figcaption>site level information from the aqualink map</figcaption></figure><blockquote><strong>Accessing site level data using pyaqua (Site list ,Info &amp; Live tools)</strong></blockquote><p>The <strong>site-list</strong> and <strong>site-info</strong> tools are designed to filter for and find sites with buoys as well as provide information about a specific site.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ye8ia47M_0iugYHZm0i8hg.gif" /><figcaption>pyaqua site list</figcaption></figure><p>The <strong>site-list</strong> tool returns a site name and site id pair and the rest of the tools rely on this site id to perform operations, so keep that one handy. <strong>Site-info tool </strong>can now use the site id to get information about a specific site including data like historical information as well as site admin data for use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VC4hDK6lUQTLxcDi395MMg.gif" /><figcaption>pyaqua site info for a given site</figcaption></figure><p><strong>Site live tool </strong>simply generates a quick snapshot for your site and returns a JSON output for the site</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pdSXp_f-caYSwdjK0-uYsA.gif" /><figcaption>pyaqua site-live tool</figcaption></figure><blockquote><strong>Exporting site level time-series data using aqualink</strong></blockquote><p>One of the best features of the overall setup is the capability for the user to choose a time range of choice and then be able to export out the dataset from these sites as a CSV file. For now, the export is limited to Sea Surface Temperature (SST), and for sites with spotters or buoys with temperature measurement at two depths, you get both BottomTemp as well as Top Temp.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-fXjQ4lSWO3V5NQiCwk4Ww.gif" /><figcaption>Exporting time series data for a given site with a year range of data</figcaption></figure><p>You can modify the date range to get longer queried results depending on when the spotter was deployed and or removals for maintenance or any other similar purpose.</p><blockquote><strong>Exporting site level time-series data using pyaqua</strong></blockquote><p>A <strong>site-timeseries </strong>tool<strong> </strong>is<strong> </strong>included as part of pyaqua and designed to allow the users to export time-series datasets for varying time periods. The results are written as CSV files to a given folder across different parameter types. <strong><em>The tool additionally allows you to export not just temperature but also wind/wave and other data type information are available for the given site</em></strong>. You can also choose to only export specific data type subsets like wind/wave/temp/sat_temp and so on</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EK8OzJJy8F9YFmSDlgM1Wg.gif" /><figcaption>pyaqua time series tool</figcaption></figure><blockquote><strong>Accessing Heat Stress &amp; site alerts in aqualink map</strong></blockquote><p>One of the best features on aqualink is the Heat Stress alert dashboard that gives you a glimpse of all sites categorized by heat stress and alert levels. You can go to the <a href="https://aqualink.org/collections/heat-stress"><strong>heat stress dashboard here</strong></a><strong>. </strong>If you are signed in you can create your own collections and <a href="https://aqualink.org/tracker">instructions can be found here</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tjbedeIcZImU2tR4kVOBxw.gif" /><figcaption>Heat stress tracker @ <a href="https://aqualink.org/collections/heat-stress">https://aqualink.org/collections/heat-stress</a></figcaption></figure><p>The sites include both sites with and without buoys based on satellite estimation of heat stress for all sites and can be sorted and additional information per site is available for users to dig into the specific location.</p><blockquote><strong>Accessing Heat Stress &amp; site alerts using pyaqua</strong></blockquote><p>The pyaqua tool connects to the heat stress dashboard and filters out and selects only those tools with buoys. The tool can be extended to include all sites in the future but for now, this is to maintain a consistency that all of pyaqua is designed for only those sites with buoys.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fyn9em-_bjdeZmlowKO7_w.gif" /><figcaption>pyaqua site alert tool</figcaption></figure><p>This can be used as a helper tool along with the site list tool to list only those sites which have an alert level greater than a specific value for example on a given day.</p><p>Over the last couple of months, I have tried to take deeper dives in understanding open oceans better, these massive and complex systems that shape climate patterns to formations and degradations of landmasses and ecosystems. This follows my first article on another tool called pycoral to gather data from Allen Coral Atlas, you can read about <a href="https://samapriyaroy.medium.com/chasing-coral-data-fd872809c271"><strong>Chasing Coral data here</strong></a><strong> and </strong><a href="https://samapriya.github.io/pyspotter/"><strong>pyspotter for SofarOcean spotters</strong></a>. You can read about my <a href="https://open-oceans.github.io/"><strong>open oceans project here</strong></a>. Building a better understanding of oceans is critical, it is tied to measuring challenging amounts of data from harsh environments.</p><blockquote>Access isn’t the same as accessibility</blockquote><p>There is always more than one way for users to access, read and then follow up on the data. Asking the right questions, tinkering on solutions, and building as a community are what allow growth in different domains. As a bonus the time series export of these datasets can be brought directly into platforms like Google Earth Engine to mix and match with other datasets</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o2etB7zslZUuV0F3nhe2tA.gif" /><figcaption>pyaqua exported Temperature data from a single buoy in Google Earth Engine (GEE)</figcaption></figure><p>Efforts like the aqualink project are creating an opportunity for users to be part of the network. It is helping bridge the gap between data-rich and data-poor geographies and allowing for hopefully a more sustainable model. I can only hope that many others will follow the same approach. For now, my open ocean toolkit grows as I contribute to the pyaqua tool and many more.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=32fb4d99c837" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>