The Challenge: Data Inequity in Rural Infrastructure

Eighty percent of people living in extreme poverty are in rural areas without reliable transportation access to hospitals, schools, and markets. While governments invest in rural infrastructure, the most basic data needed to identify transportation barriers often doesn’t exist. Rural footpaths, the lifelines connecting isolated communities, are largely invisible in standard mapping datasets. Without this data, critical transportation corridors remain unmapped, unplanned for, and underserved.

A First-of-Its-Kind Dataset

Today, we’re releasing the Walking Path and Waterway Crossings Dataset on Hugging Face — the world’s first open-source machine learning dataset for footpath identification focused on areas where these footpaths cross waterways. This dataset includes labeled data across multiple countries and is purpose-built to help AI systems distinguish rural footpaths from roads, trails, and natural features in high-resolution satellite imagery. By open-sourcing this training data, we’re enabling researchers, NGOs, and governments worldwide to develop their own path detection models and accelerate infrastructure planning where it’s needed most.

From Detection to Impact

With PJMF’s support, we’ve advanced from initial data labeling to designing our own algorithm trained on this data. The model we’re developing detects walking paths in high-resolution satellite imagery and produces a landscape-aware dataset. With this, we aim to identify precise crossing locations where footbridges or other transportation infrastructure could address access gaps. We’re advancing toward this milestone through ongoing collaboration with Better Planet Laboratory at the University of Colorado Boulder.

What This Means

The Patrick J. McGovern Foundation grant has accelerated our research timeline and expanded our impact mapping capabilities. This work demonstrates how AI can serve a public purpose — by giving communities and governments greater visibility into their own needs and supporting informed infrastructure decisions. When rural communities participate in mapping the paths they walk every day, technology becomes a tool for equity.

We’re grateful to PJMF for supporting this work and for their commitment to applying AI to real-world needs. As PJMF President Vilas Dhar, has noted: “When people decide how AI is applied to real needs, technology can strengthen democracy, protect human rights, and widen opportunity.” For millions of isolated communities, that opportunity begins with access — and with better data to guide where and how infrastructure investments are made.

Learn more about the Walking Path and Waterway Crossings Dataset on Hugging Face, and explore our impact mapping work at fika.org.

Mapping Rural Access: How We’re Finding Footpaths That Connect Isolated Communities was originally published in Fika on Medium, where people are continuing the conversation by highlighting and responding to this story.

Imagine drought conditions take out 40% of the grain that is grown in West Bengal; where in the world can we expect to see food insecurity? What about if the workers at the ports of Rotterdam go on strike; where should global leaders direct food resources to reduce the impacts to downstream consumers?

Answering these types of questions just got easier. Earth Genome and Better Planet Laboratory are proud to announce the launch of Global Food Twin, an ambitious expansion of our food system digital twin that now models supply chains across the entire world’s food system. Building on the success of our US-focused proof of concept, we created a model of food production, consumption, transportation infrastructure, and trade of food resources across the globe. Today we are open sourcing the data, model and proof of concept application for all to access

This model and the proof of concept Food Twin app reveals major risks in our food system. Roughly 9% (~350) of sub-national administrative units account for 80% of global caloric flows. Small impacts in the wrong areas will cause downstream impacts that ripple around the globe. When analyzing transportation pathways, similar concentrations are seen in major maritime, road, and rail routes. Beyond risks, analyzing food systems end to end through these new data can help us prepare and proactively build more resilient food systems for the future.

This project was supported by a grant from Climate Works Foundation. Credit to Development Seed who built the majority of the web application; Zia Mehrabi with the vision behind this concept; Ginni Braich who developed the supply and demand data; Kushank Bajaj who developed the transportation and flow data; and Jasper Verschuur assistant professor at Delft University whose work on supply chain mapping we built on to develop this model.

Key Resources

• Proof of concept Global Food Twin web application
• All data produced by the model
• Methodology and code for the model
• Application exposing impacts of regional production deficits

From National to Global: The Evolution of Food Twin

When we launched the US Food Twin last year, we always intended to expand our vision beyond national borders. Food systems don’t exist in isolation — what happens in Brazil’s soybean fields affects dinner tables in Beijing, and drought in Australia ripples through global wheat markets. Our new Global Food Twin captures these complex relationships, giving us unprecedented visibility into how food moves around our planet.

Most of us have no idea where our breakfast comes from or who grows it. This disconnect becomes even more pronounced when we consider international food systems. The coffee in your cup might come from Ethiopia, the bananas in your cereal from Ecuador, and the cereal itself is made from wheat grown across multiple continents. Global Food Twin illuminates these connections, tracking calories from farm to fork across oceans and borders.

The Architecture of Global Food Twin

The Global Food Twin represents a significant advancement in modeling complexity. While our US model focused primarily on plant-based foods within national borders, our global model integrates:

• Production across ecosystems: Crops from farmlands, animal products from pasturelands, and aquatic food sources from oceans and freshwater environments
• Multi-modal transportation: Maritime shipping routes, railway networks, road transportation, and inland waterways
• International trade flows: Bilateral trade relationships between countries and subnational regions
• Cultural and economic factors: Different dietary patterns and economic access to food across regions

Production Model

Our production dataset integrates multiple sources organized into 10km x 10km grid cells across the entire planet. We track total mass (kg), calories (kcal), and macro-nutritional content (fats, carbohydrates, proteins) of all crop, aquatic, and livestock commodities produced normalized to 2020 levels.

Key data sources include:

• CropGRID’s global yields and production for 160 crops
• Global crop calendars showing planting and harvesting dates
• FAO livestock density data for 8 major species
• FAO global catch data for aquatic food sources

Demand Model

Our demand model is a simple population driven estimate (that we found works well for the scale of our supply chain allocations, see our repo for more information), and matches production to consumption needs based on:

• Culturally relevant diets and caloric requirements in each region
• Population density at a fine-grained resolution
• Nutritional transitions occurring in different regions

Trade Model

One key advancement in our global model is the trade component. In collaboration with Dr. Jasper Verschuur, we simulated maritime and hinterland routes connecting 3787 subnational regions across 240 countries. Our data now represents food flows alongside all major transportation infrastructure in the world — with a level of fidelity and detail, never before achieved.

This has allowed us to:

• Calculate least-cost flows between production and consumption regions
• Model multi-modal transportation of 82 commodities including transshipments
• Identify port-to-port connections for three unique vessel types (bulk, container, general)
• Visualize dependencies on critical maritime, rail, road, inland waterways infrastructure

Application Design

The design of the application was a feat within itself. Working with our partner Development Seed we took on an extensive design sprint. We detailed our learnings and vision in this blog post.

The basemap is one of the more interesting parts of the application. We’re big fans of dot maps here at Earth Genome. This particular basemap was designed by taking the production data for each 1km x 1km grid cell and comparing all 82 food groups in the model. We calculated the variance of each production value and displayed the value with the highest variance as a colored dot. The color of the dot represents the most “unique” type of production in that region; the size of the dot represents how much total production occurs in that region for that particular food type.

Key Insights from Global Food Twin

Similar to our US Food Twin, the global model reveals insights about a brittle global food system. The centralized production and bottlenecks in transportation are a visualization pointing to how we can prepare for shocks and disruptions and make a more resilient food system globally.

Here are a few of the insights from the model:

• Our exports analysis capture 13.2M unique flows between 3787 admin1 units and for 82 food groups
• Just ~17,500 bilateral trades are responsible for half of the global export volume, this is only 0.13% of all export flows
• 100 largest bilateral flows are 6.7% of all export volumes, largely soyabean (2/3s of these trades), then maize, wheat and sugar
• Exports from Brazil to China is the largest volume of trade, represents almost 4% of all global export flows, USA to China is second, representing 3.3% of all global export flows by volume
• Exports to China represent 15% of all global flows
• The 5 largest exporters by volume are USA, Brazil, Canada, Argentina and Ukraine
• The 5 largest importers by volume are China, Germany, Netherlands, USA and Japan
• Egypt is the largest importer of wheat in the world, also a prominent re-exporter of wheat to Africa (Tunisia, Chad, UAE, Kuwait)
• 64.5% of all exports are by sea/hinterland, 24.8% by rail, 9.7% by road and 0.9% by Inland Waterway

Concentrated Production Meets Global Demand

While the US model showed that just 5.5% of counties produce half the nation’s food, the global picture is even more concentrated. Globally, just 1.2% of regions are responsible for 50% of the world’s grain supply; this concentration creates critical vulnerabilities in our food supply. For example, let’s take Uttar Pradesh’s exports. Our model predicts that this region alone provides the yearly calories for 165 million people around the world. This can be seen clearly in the impact view of the application. All highlighted regions receive food exports from Uttar Pradesh.

Chokepoints in Global Food Transportation

The global model highlights critical chokepoints where disruptions could have cascading effects. Key maritime passages like the Suez Canal, the Panama Canal, and the Strait of Malacca handle significant portions of the world’s food trade. This can be seen in the app when looking at a region like Castile and León in northern Spain. This region exports 8 Gigatons of food worldwide, but it can be seen that many of the meat exports going to Asia travel through the Suez Canal. Even small disruptions to individual shipping lanes can cause major impacts to those expecting these shipments.

Further visualization can be done using the full data set to explore which ports, rail lines, roads, and maritime routes are the most critical and most at risk of disrupting food systems.

Vulnerability to production shocks

Climate impacts combined with conflicts and global trade shocks cause major impacts to food systems that reverberate around the globe. Heat stress and drought events that impact single regions trigger worldwide price spikes and food insecurity.

Global conflicts provide a clear example of these dynamics. The map below shows a major grain-producing region in Ukraine that exports over 4 gigatons of food annually, primarily grain. This visualization reveals why Russia’s invasion of Ukraine had such effects on global grain markets, disrupting both production and export channels from this critical agricultural hub.

Further analysis on the impacts of eliminating individual regions can be done in this tool.

Applications of Global Food Twin

The Global Food Twin represents not just an academic exercise but a practical tool for addressing food security challenges.

Climate Resilience Planning

By modeling how climate shocks propagate through the global food system, we can identify vulnerable communities and develop targeted interventions before crises occur. Policymakers can use these insights to diversify food sources and strengthen local production where appropriate.

Supply Chain Optimization

Food companies and distributors can use our model to evaluate supply chain risks and identify more resilient sourcing strategies. This could lead to more stable food prices and fewer disruptions in availability.

Food Security Policy

International organizations focused on hunger and food security can better target their efforts by understanding the complex interdependencies of global food systems. Our model helps identify which regions are most vulnerable to various types of shocks.

Join the Effort

The Global Food Twin, like the US Food Twin, is built on principles of open data and community collaboration. We invite researchers, policymakers, industry leaders, and concerned citizens to engage with this model and contribute to our understanding of global food systems.

Here’s how you can get involved:

• Explore the System: Dive into our interactive visualization and explore global food flows.
• Use the Data: We’ve open sourced the data we use in this application. Use this in your projects and be sure to let us know how you use it.
• Contribute expertise: If you’re a researcher, we welcome collaborations to refine and expand the model.
• Share insights: Help spread awareness about food system vulnerabilities in your community and globally.
• Support local resilience: Use these global insights to better understand how food systems work. Local food networks are the secret to a resilient food system; support your local food network and be part of the solution
• Get in touch: If you have ideas or want to collaborate, we’d love to hear from you. Reach out to us here.

The Road Ahead

While this global model represents a significant advancement, our work is far from complete. Future development will focus on:

• Higher temporal resolution: Move to monthly, weekly, or even daily timesteps to capture rapid changes
• Production shock modeling: Integrating climate projections, global conflict, or other impacts to anticipate future disruptions
• Policy simulation: Building tools to test how policy interventions might impact food security
• Nutritional quality: Moving beyond calories to better model micronutrient flows through the system

We also invite the community to join forces to develop an application that allows users to simulate shocks to the system.

In a world where climate change and geopolitical tensions increasingly threaten food security, tools like the Global Food Twin are essential for building resilience and ensuring that nutritious food remains accessible to all. By making the invisible visible, we hope to inspire action toward a more sustainable and equitable global food system.

Global Food Twin: Modeling the World’s Food System was originally published in Earth Genome on Medium, where people are continuing the conversation by highlighting and responding to this story.

If you’ve tried making map tiles and you’re anything like me, you probably feel a lot like Ron. Thanks to the incredible work from people like Erica Fischer at Mapbox and Felt, there is great open source tooling that magically takes geospatial data files and turns them into small files for highly performant beautiful maps. But as your needs get more specific the magic starts to get more complicated and before you know it, you’re feeling like a muggle wrestling with your terminal hoping that you get the right combination of spells that create a dataset that turns out just right.

Today, I’m sharing a new tool that makes this process a little bit easier for us muggles as well as some tricks of the trade that I’ve picked up along the way. Check out the tool and let me know what you think!

A young wizard’s introduction to Tippecanoe

Tippecanoe is the powerful spell book for transforming geospatial data into vector tiles. Tippecanoe can process enormous datasets — millions of features — and intelligently simplify them at different zoom levels while preserving their essential characteristics.

Unlike other methods that might simply discard features when things get crowded, Tippecanoe offers a remarkable array of enchantments — options like feature dropping, attribute filtering, and clever clustering — that maintain the visual integrity of your maps even at low zoom levels. Knowing what to use when may feel daunting at first (with commands like --drop-densest-as-needed and--extend-zooms-if-still-dropping), but once mastered, you can create tilesets that render quickly and beautifully across platforms, making you the cartographic equivalent of a Defense Against the Dark Arts professor, but hopefully with a longer career.

If you aren’t familiar with Tippecanoe, go install it following the instructions here.

Introducing Tippecanoe GUI

Tippecanoe has great documentation in its ReadMe with all the info you need to get started, but I’ve often found that half the battle is finding the commands, deciding which ones to use and stringing them together into a coherent command you can actually run in your terminal. I’ve made this a little easier by taking many of the Tippecanoe features and converting them into an interface you can use to generate your commands. The premise of this tool is you select all the options you want from the dashboard and a command is generated below you can copy and paste in your terminal. Check it out and let me know what you think!

Tips and Tricks I’ve learned along the way

I’ve got to say here, the whole reason I’m writing this is because I am not an expert. There are people and companies that specialize in just making map tiles. That said, I’ve wrestled through making enough map tiles that I’ve learned strategies to make this as painless as possible.

Off the bat know that there are currently two major types of map tiles

• mbtiles: these are meant to interface nicely with Mapbox maps and are the OG map tiles format. The downside is they need a server to serve them to your app. You can upload these into you Mapbox account to serve them for you, but if your file is bigger than 300mb you will likely need to go through the Mapbox Tile Service.
• pmtiles: this is a newer map tiles filetype built for open source maps through MapLibre. They can also be added to Mapbox maps pretty easily with this cool node package.

Get a Map Tile Viewer

Do yourself a favor and get a local map tiles viewer. If you’re planning to make mbtiles I really like this no nonsense viewer developed by GitHub user Akylas. If you’re working with pmtiles this is a great online viewer. If you try to view your map tiles using something like QGIS you can view your data, but it doesn’t display correctly based on zoom level

Lower zoom levels are the hardest

Based on how map tiling works, lower zoom levels (the map is more zoomed out) will have tiles with larger geographic areas than the higher zoom levels (the map is more zoomed in). The first thing you should do is run some tests to figure out what the lowest zoom level is where none of your data is dropped. The brute force way to do this is to run a command like the one below that attempts to generate a tileset with one zoom level without dropping anything.

tippecanoe -o output.mbtiles -z12 -Z12 -r1 input.geojson

• -z12: Sets the maximum zoom level to 12
• -Z12: Sets the minimum zoom level to 12 (together with -z12, this means only level 12 will be generated)
• -r1: Prevents any automatic dropping of points at zoom levels below the base zoom

There are much more elegant ways to do this, but this, but something like the above is a great foolproof way to learn about your data; if this command fails, it’s too big for that zoom level. Keep trying this until you get a minimum zoom that works. Once you know this, ask yourself how you want your data to display between zoom 0 and whatever that zoom level is. This is where you will use all the magic commands like--drop-smallest-as-needed or --drop-smallest-as-needed to control what the data looks like at these zoom levels. Note that just not writing these zoom levels is a totally acceptable solution if you don’t plan on making this data visible at these zoom levels.

Use -zg for max zoom

Unless precision is extremely important for your data, just use the -zg flag for max zoom. This flag guesses the maximum needed zoom to show your data with decent precision. The lower zoom tiles will be what most inflate your map tiles file size. -zg does a good job of making sure that you don’t create tiles that are overly zoomed.

Drop the details

Before you even start trying to make map tiles, try to reduce your file size. I am almost always working in geopandas manipulating data before I write a geojson file that I convert to map tiles. When creating your data set:

• Get rid of columns you don’t need
• Reduce precision of your floats as much as possible or even better use integers
• Can you convert any string columns to integers

One trick I’ve used is to write a tiny geojson file with just geometry and a feature id. I write this at lower zoom levels and then have a fully detailed file that I write to map tiles visible at lower zoom levels. This makes data visible at all zoom levels and you can design a UI that work with your data constraints.

Make a directory and convert this to map tiles

This is a somewhat advanced method, but one output option for tippecanoe is a “directory.” What this will do is write each zoom level to a directory. This isn’t necessarily best practice, but you can granularly write each zoom level you want with different options. You’ll end up with folders for each zoom level that you can move into a single folder with subfolders showing each zoom. Once you’ve done this, take one of the metadata files that was written and change the metadata to reflect the zoom levels you’ve used. Once you have this directory, you can create a map tiles file with tile-join.

Big tile energy

If all else fails and you just want to show your data on a map forget about tile size limits using the -pf and -pk flags. This will decrease the performance of your map and you will not be able to upload these tiles to your Mapbox account, but you can write to a pmtiles file and host this in an s3 bucket on AWS. I used this trick for our vector waterways data in this web app. It’s not ideal, and the performance lags at low zoom levels, but it shows all the data and sometimes that’s all you care about.

Run your scripts in Jupyter notebooks

This is coming from a guy that just recently learned that option click will move your cursor anywhere on your terminal (you’re welcome if you didn’t know that already). I’m usually manipulating data in Jupyter notebooks. You still need to write a geojson file first, but you can run terminal commands in Jupyter notebooks by putting ! before the command. This is cool as you can also use curly brackets to reference python {variables} in your scripts. It’s a nicer user interface and sometimes easier to just run this in the notebook where you are working with your data.

Wrapping up

Tippecanoe may seem like advanced magic at first, but with practice and the right tools, you’ll soon be conjuring beautiful, performant maps. Give the Tippecanoe GUI a try; hopefully this tool will help make your mapping journey less of a Chamber of Secrets. Now go forth and create maps so magical even the Marauders would be impressed!

The Dark Art of Vector Map Tiling was originally published in Fika on Medium, where people are continuing the conversation by highlighting and responding to this story.

Background

Our research team at Better Planet Labs had successfully completed an ambitious project to map the world’s waterways. After architecting and deploying an algorithm that processed terabytes of cloud imagery, we tripled the number of mapped waterways around the world. (If that sounds interesting, check out our detailed post about it) The challenge then became making this valuable dataset accessible to researchers and practitioners worldwide.

Rather than simply storing files in a cloud bucket, I wanted to optimize for accessibility and usability. After evaluating various options, I chose Source Coop, a non-profit platform dedicated to making geospatial data more accessible. Their strong reputation in the geospatial community and my positive experiences using their hosted data made this an easy choice.

This guide is split into sections covering both vector and raster data processing. For context: vector data consists of lines (think roads, boundaries), while raster data is made up of pixels (think satellite imagery, land use maps). All tools and processes described here use open-source software, primarily Python and command-line tools.

Vector Data Files

If you’re working with geospatial data, you probably have your vector data in ESRI shapefiles, GeoJSON, or Parquet formats. The current best practice in cloud-native communities is to use GeoParquet files for large vector data (500MB+) and GeoJSON for smaller datasets. Here’s how to convert between formats:

Using ogr2ogr (command line):

ogr2ogr -f "Parquet" output_filename.parquet input_filename.geojson -progress

Or using Python with GeoPandas:

import geopandas as gpd
gdf = gpd.read_file('input_file.geojson')
gdf.to_parquet('output_file.parquet')

For larger datasets, you may need to split them into multiple parquet files. When doing this, follow existing community best practices for data splitting. In our case, we organized parquet files by Level 2 HydroBasins, using the naming format: {hydrobasins_level_2_id}_{part}.parquet.

To verify your parquet files are correctly configured as GeoParquet: ogrinfo -so FILE_NAME.parquet

You should see a defined geometry column (typically “geometry” or “geom”) and the specific geometry type (POINT, POLYGON, etc.). For visual verification, you can use Lonboard to check file integrity.

Finally, create a GeoJSON file showing the extents of your parquet files:

import geopandas as gpd
from shapely.geometry import box

# List of parquet files to process
files = [
    'file1.parquet',
    'file2.parquet',
    'file3.parquet'
]

# Create GeoDataFrame containing the extent (bounding box) of each parquet file
extents = gpd.pd.concat([
    gpd.GeoDataFrame(
        {'file': f},
        geometry=[box(*gpd.read_parquet(f).total_bounds)],
        crs=gpd.read_parquet(f).crs
    ) for f in files
])

# Export the extents to a GeoJSON file
extents.to_file('extents.geojson', driver='GeoJSON')

Tiled Vector Data

Vector tiles enable direct visualization of your data on web maps. PMTiles are the current standard, offering advantages over MBTiles by allowing direct map references without running a separate server. The creation process requires careful attention to detail:

1. Convert parquet files to GeoJSON (if needed): ogr2ogr -f GeoJSON output_filename.geojson input_filename.parquet -progress -lco COORDINATE_PRECISION=5

2. Create PMTiles using Tippecanoe: tippecanoe -z11 -Z1 -o output_filename.pmtiles -l waterways --no-tile-size-limit --drop-smallest-as-needed input_filename.geojson

Pro tip: Process files locally or in the cloud rather than on external drives to avoid potential errors with large files.

3. Join multiple PMTiles files: tile-join -o joined_output.pmtiles --no-tile-size-limit file_1.pmtiles file_2.pmtiles file_3.pmtiles

Raster Data Files

Most raster data comes in GeoTIFF format, but cloud optimization requires Cloud Optimized GeoTIFFs (COGs). To check if your file is already a COG: gdalinfo SOURCE_PATH/FILE_NAME.tif | grep -i "layout"

If the output doesn’t show LAYOUT=COG, convert your files:

for file in SOURCE_PATH/*.tif; do
    gdal_translate "$file" \
        "DESTINATION_PATH/${file%.}.tif" \
        -of COG \
        -co COMPRESS=deflate
done

For enhanced accessibility, consider creating a STAC catalog for your data.

Raster Tiles

Raster tiles enable efficient web visualization by creating multi-level pyramids that load only the data needed for the current view. To create them: gdal2tiles.py -z 1–12 — processes=4 — webviewer=none input.tif output_folder

Key parameters:

• -z 1-12 Specifies zoom levels (adjust based on your needs)
• --processes=4Enables parallel processing
• --webviewer=none Skips demo HTML viewer creation

While modern approaches might favor creating overview tilesets and generating raster tiles on-demand, pre-generated tiles offer extremely easy integration.

Cloud Upload Process

With all assets prepared (parquet vector files, PMTiles vector tiles, COG imagery, and raster tiles), you’re ready to:

1. Create a repository on Source Coop (check out this guide)

2. Upload data to the connected S3 bucket

3. Create a comprehensive README including:

• Dataset description
• Data sources
• Access instructions
• Usage examples
• Technical specifications

Thank you!

This work wouldn’t have been possible without guidance from the Cloud Native Geospatial community, particularly Tyler Erickson, Matt Travis, and Michelle Roby. You can access our Source Coop waterways data repository here. Check out this map of our being rendered directly from Source Cooop in an exploratory app.

Share your thoughts and what you would have done differently in the comments!

Cloudy with a Chance of Data: A Cloud-Native Geospatial Tutorial was originally published in Radiant Earth Insights on Medium, where people are continuing the conversation by highlighting and responding to this story.

Background

Our research team at Better Planet Labs had successfully completed an ambitious project to map the world's waterways. After architecting and deploying an algorithm that processed terabytes of cloud imagery, we tripled the number of mapped waterways around the world. (If that sounds interesting, check out our detailed post about it) The challenge then became making this valuable dataset accessible to researchers and practitioners worldwide.

Rather than simply storing files in a cloud bucket, I wanted to optimize for accessibility and usability. After evaluating various options, I chose Source Coop, a non-profit platform dedicated to making geospatial data more accessible. Their strong reputation in the geospatial community and my positive experiences using their hosted data made this an easy choice.

This guide is split into sections covering both vector and raster data processing. For context: vector data consists of lines (think roads, boundaries), while raster data is made up of pixels (think satellite imagery, land use maps). All tools and processes described here use open-source software, primarily Python and command-line tools.

Vector Data Files

If you're working with geospatial data, you likely have your vector data in ESRI shapefiles, GeoJSON, or Parquet formats. The current best practice in cloud-native communities is to use GeoParquet files for large vector data (500MB+) and GeoJSON for smaller datasets. Here's how to convert between formats:

Using ogr2ogr (command line):

ogr2ogr -f "Parquet" output_filename.parquet input_filename.geojson -progress

Or using Python with GeoPandas:

import geopandas as gpd
gdf = gpd.read_file('input_file.geojson')
gdf.to_parquet('output_file.parquet')

For larger datasets, you may need to split them into multiple parquet files. When doing this, follow existing community best practices for data splitting. In our case, we organized parquet files by Level 2 HydroBasins, using the naming format: {hydrobasins_level_2_id}_{part}.parquet.

To verify your parquet files are correctly configured as GeoParquet: ogrinfo -so FILE_NAME.parquet

You should see a defined geometry column (typically "geometry" or "geom") and the specific geometry type (POINT, POLYGON, etc.). For visual verification, you can use Lonboard to check file integrity.

Finally, create a GeoJSON file showing the extents of your parquet files:

import geopandas as gpd
from shapely.geometry import box

# List of parquet files to process
files = [
    'file1.parquet',
    'file2.parquet',
    'file3.parquet'
]

# Create GeoDataFrame containing the extent (bounding box) of each parquet file
extents = gpd.pd.concat([
    gpd.GeoDataFrame(
        {'file': f},
        geometry=[box(*gpd.read_parquet(f).total_bounds)],
        crs=gpd.read_parquet(f).crs
    ) for f in files
])

# Export the extents to a GeoJSON file
extents.to_file('extents.geojson', driver='GeoJSON')

Tiled Vector Data

Vector tiles enable direct visualization of your data on web maps. PMTiles are the current standard, offering advantages over MBTiles by allowing direct map references without running a separate server. The creation process requires careful attention to detail:

1. Convert parquet files to GeoJSON (if needed): ogr2ogr -f GeoJSON output_filename.geojson input_filename.parquet -progress -lco COORDINATE_PRECISION=5

2. Create PMTiles using Tippecanoe: tippecanoe -z11 -Z1 -o output_filename.pmtiles -l waterways --no-tile-size-limit --drop-smallest-as-needed input_filename.geojson

Pro tip: Process files locally or in the cloud rather than on external drives to avoid potential errors with large files.

3. Join multiple PMTiles files: tile-join -o joined_output.pmtiles --no-tile-size-limit file_1.pmtiles file_2.pmtiles file_3.pmtiles

Raster Data Files

Most raster data comes in GeoTIFF format, but cloud optimization requires Cloud Optimized GeoTIFFs (COGs). To check if your file is already a COG: gdalinfo SOURCE_PATH/FILE_NAME.tif | grep -i "layout"

If the output doesn't show LAYOUT=COG, convert your files:

for file in SOURCE_PATH/*.tif; do
    gdal_translate "$file" \
        "DESTINATION_PATH/${file%.}.tif" \
        -of COG \
        -co COMPRESS=deflate
done

For enhanced accessibility, consider creating a STAC catalog for your data.

Raster Tiles

Raster tiles enable efficient web visualization by creating multi-level pyramids that load only the data needed for the current view. To create them: gdal2tiles.py -z 1–12 — processes=4 — webviewer=none input.tif output_folder

Key parameters:

• -z 1-12 Specifies zoom levels (adjust based on your needs)
• --processes=4Enables parallel processing
• --webviewer=none Skips demo HTML viewer creation

While modern approaches might favor creating overview tilesets and generating raster tiles on-demand, pre-generated tiles offer extremely easy integration.

Cloud Upload Process

With all assets prepared (parquet vector files, PMTiles vector tiles, COG imagery, and raster tiles), you're ready to:

1. Create a repository on Source Coop (check out this guide)

2. Upload data to the connected S3 bucket

3. Create a comprehensive README including:

• Dataset description
• Data sources
• Access instructions
• Usage examples
• Technical specifications

Thank you!

This work wouldn't have been possible without guidance from the Cloud Native Geospatial community, particularly Tyler Erickson, Matt Travis, and Michelle Roby. You can access our Source Coop waterways data repository here. Check out this map of our being rendered directly from Source Cooop in an exploratory app.

Share your thoughts and what you would have done differently in the comments!

Cloudy with a Chance of Data: A Cloud-Native Geospatial Tutorial was originally published in Fika on Medium, where people are continuing the conversation by highlighting and responding to this story.

Today, we’re announcing the launch of a new dataset developed as a collaboration between Bridges to Prosperity (B2P) and the Better Planet Laboratory. This new dataset triples the known extent of mapped waterways globally, adding 124 million kilometers to the previously mapped 54 million kilometers. The data was generated by a new AI model called WaterNet, also developed by the Better Planet Laboratory, which has also been released along with the research papers on the methodology. Explore a map of the new data or dive straight into the research.

The Unseen Challenge: Bridging the Data Gap

For years, the world has faced a critical blind spot: an incomplete map of its waterways. This is especially true in low and middle-income regions, where isolation caused by uncharted rivers, streams, and seasonal waterways traps communities in cycles of poverty and vulnerability.

At B2P, we’ve seen this challenge firsthand. Out of nearly 5000 crossings assessed, an average of 38% are on unmapped waterways. In some countries with more significant data gaps, over 60% of verified crossings are on unmapped waterways. Furthermore, we’ve struggled to scale our efforts to identify where bridges are needed using remote assessments, as the data we fed into our models about where rivers are was so incomplete. Looking beyond just planning for transportation, how can you plan for climate resilience if you don’t know where the water is?

Our digital team at B2P, in partnership with a team of data scientists at the Better Planet Laboratory, is developing a suite of remote analysis tools that will assess transportation barriers to critical destinations in rural low-income communities to locate where bridges are needed. Collectively called Fika Map (“fika” means “arrive” in Swahili), the objective of this tool suite is to identify the scope and distribution of transportation barriers through entire regions or nations using publicly available data and machine learning technology and estimate the cost and social impact of addressing those barriers, once identified. While the past five years have seen incredible progress in the ability of remote sensing data and AI to address major challenges, those solutions often rely on data that is expensive or impossible to obtain on a meaningful scale.

The Technology Behind WaterNet

WaterNet represents a significant leap forward in waterway mapping technology. The model combines two powerful approaches using training data from 10m per pixel satellite imagery (Sentinel-2 Level-2A NRGB bands) and 30m resolution digital elevation model (Digital GLO-30 Copernicus). The model itself is generally based on ideas from U-Net and ResNet.

Unlike previous mapping efforts that relied primarily on elevation data, WaterNet leverages the wealth of information available in visual and near-infrared satellite imagery. The model can identify subtle signatures of water flow, from clear river paths to vegetation patterns that indicate seasonal waterways. Through the training process the model, in essence, learns to draw where waterways exist.

The second technical output produced by this research was the vectorization process. The raster outputs of the WaterNet model were vectorized to a TDX-Hydro backbone, in order to increase compatibility of the data. This data was then put through a connecting process, which was then thinned, and properly vectorized into line strings. Strahler stream order was also added to each river segment.

Access the research paper on the model here for more information on the model and methodology.

Global Impact and Accessibility

The implications of this breakthrough extend far beyond mapping. As Dr. Zia Mehrabi, Director of the Better Planet Laboratory and WaterNet’s principal investigator, notes: “This is an excellent example of the use of AI for Good. We are glad to release this new data in the hope that it may help close that important gap in our current understanding of where rivers and streams are on the planet and aid humanitarian organizations in ways not possible with existing data.”

The dataset is particularly beneficial for:

• Climate resilience planning in vulnerable communities
• Transportation infrastructure development
• Humanitarian aid routing
• Environmental protection efforts
• Scientific research on Earth’s water systems

About the Data

All data uses the EPSG:4326 projection system for global compatibility. The WaterNet dataset is available in multiple formats to suit different use cases. Below we detail the technical specifications behind the data

Raster Outputs

The raster data is available as cloud optimized GeoTiff (COG) format.

• Resolution: 20-meter global coverage
• Data type: 8-bit unsigned integer
• Probability scaling: 0 (low) to 255 (high) probability of waterway

Vector Outputs

Available as geoparquet and pmtiles file types. The vector data includes rich attributes for each stream segment:

• Unique stream IDs and downstream connectivity
• Strahler stream order classification
• Length measurements in meters
• Lake intersection indicators
• TDX-Hydro correspondence markers

Open Access

In the spirit of advancing equity and global collaboration, the entire dataset is being released under a CC 4.0 license, making it freely available to governments, researchers, and organizations worldwide. The data, model and code can be accessed through multiple platforms.

Interactive Map:

An interactive map of the data has been made available at apps.fikamap.com/waternet.

Data Access:

• Cloud-optimized formats through Source Coop
• Harvard Dataverse houses the data set with a DOI: https://doi.org/10.7910/DVN/YY2XMG

Research Papers:

• Deep learning waterways for rural infrastructure development
• Mapping waterways worldwide with deep learning

Code:

Code for model creation, training, and vectorization are linked in the GitHub Repositories below. Global inference code has not been included as the large amount of data required makes such an effort difficult to generalize across compute hardware.

• WaterNet
• WaterNet Training and Evaluation
• WaterNet Vectorization

The Path Forward

This release is just the beginning. The dataset will serve as a foundation for B2P’s Fika Map initiative — a tool suite designed to identify transportation barriers in rural, low-income communities. The name “Fika,” meaning “arrive” in Swahili, embodies the project’s goal: ensuring everyone can reach essential services, regardless of where they live.

B2P and the Better Planet Laboratory continue their collaboration, seeking resources to expand their work in remote site identification and route development. With 80% of people experiencing extreme poverty living in rural areas, the need to leverage new technology for climate-smart infrastructure solutions has never been more urgent.

WaterNet: AI-Powered Global Water Mapping Triples Known Waterways was originally published in Fika on Medium, where people are continuing the conversation by highlighting and responding to this story.

We recently had the opportunity to work with Conservation International investigating how to use AI to make their environmental research more accessible. Through a series of experiments combining large language models with geospatial data, we developed new ways to allow people to engage with complex environmental research, particularly focusing on Conservation International’s Not Just Carbon portfolio of work to advance the non-carbon benefits of forests highlighted in WRI’s report. In our work we found that AI can be an incredible tool to effectively translate complex research for different audiences, from policymakers to teachers, multilingual capabilities can dramatically expand research reach, and interactive visualizations powered by AI can make data more intuitive and engaging. Below we share more about our process and findings to help others that are working on similar projects.

Project Structure

Scientists know that forests are even more important for stabilizing the climate than usually understood by local and regional stakeholders and policymakers. By influencing rainfall patterns and reducing local temperatures, forests also affect agriculture production, local health, and water security. Conservation International came to us with this body of research, and the challenge of making it relevant to hundreds of local and regional policy makers and stakeholders, each with different interests.

Working with large language models is very much an experimental endeavor. As such we structured the project in a series of short 2-week sprints that would allow the team to pivot into many different areas. Between each sprint we’d meet with the Conservation International team to discuss the prototypes we developed and sketch up what we’d continue building in the next sprint and which new ideas we may want to pursue.

Before the project and throughout the various experiments, the Conservation International team shared with us relevant reports, research papers, and other content. They also put us in touch with the scientists conducting research to hear more about the parts of their work they thought was most interesting and to get feedback on the prototypes that our team was building out.

The prototypes were deployed on a live demo site and refined throughout the project

The Anatomy of a Generative AI Product

Most of the prototypes we built were using the Assistants API from OpenAI. The ways these work is you script a prompt that controls how the agent responds to an input. Sometimes we’d use one of these assistants, but we’d often chain them together to produce multiple responses or process inputs in different ways. We developed a common vernacular to help us explain our architectures which we detail below.

Overview of Demos built

We built several demos, but the two detailed below were the two that showed the most promise at the end of the project.

CI AI Research Paper Chat

Imagine having a conversation with an expert who has read over 100 research papers about climate change and conservation. That’s what we’ve created with the CI AI Research Paper Chat. Users can interact with different AI personas — from researchers to policy experts to teachers — each offering unique perspectives on Conservation International’s research.

The system goes beyond simple question-and-answer interactions. Users can customize responses based on their needs, adjusting for creativity, length, and formatting. For educators, there’s even an option to specify grade level for responses. Most importantly, every response is grounded in actual research papers, with direct citations that users can click through to access the original sources. Check it out here.

Interactive AI-Powered Map Visualization

Our second major experiment combines multiple environmental datasets into an interactive map that responds to natural language queries. When users ask questions about heat, deforestation, or climate change impacts on health, the AI automatically styles the map to highlight relevant data in meaningful ways. The system not only visualizes the data but also provides context about what you’re seeing and suggests related datasets to explore. You can find the prototype here.

Key Learnings

Through this project, we learned a lot about how AI can make environmental research more accessible:

AI can greatly broaden the reach of research findings: Our experiments showed that AI can help people engage with scientific content in more natural and intuitive ways. For instance, most research is published in English, but using AI someone can ask questions about a corpus of research and the AI model will respond in the language of the user. Of course it can also be tailored to explain content in more engaging and digestible ways.

The Power of Multiple Perspectives: Different users need different approaches. A policymaker might want technical details and methodologies, while a teacher might need grade-appropriate explanations of the same concepts. Our multi-persona approach helps serve these diverse needs.

The Importance of Data Infrastructure: One of our key findings was the need for better standardization and accessibility of research data. Making environmental data more structured and accessible would greatly enhance our ability to build tools with AI that access and use that data to produce compelling summaries and visualizations.

Looking Forward

This project represents just the beginning of what’s possible when we combine AI with environmental research. We see enormous potential for:

• Building AI assistants that launch alongside major research publications
• Leveraging AI for multilingual access to research
• Creating more interactive, data-driven visualizations of environmental changes
• Developing tools that help researchers synthesize findings across multiple studies

Try It Yourself

We invite you to explore our experiments. Whether you’re a researcher, journalist, educator, or simply someone interested in environmental conservation, these tools offer new ways to engage with important environmental research.

The code for this project is open source and available on GitHub. We welcome contributions and feedback from the community as we continue to develop these tools.

If you have ideas for how AI can increase accessibility of climate research drop a note in the comments below!

Not Just AI was originally published in Earth Genome on Medium, where people are continuing the conversation by highlighting and responding to this story.

In our episode, viewers are introduced to individuals like Karabiyo Safuroza, a cocoa farmer whose life has been transformed by the improved access provided by a bridge built in partnership with the Government of Uganda. “The bridge has contributed positively towards the community because people can move freely now,” Karabiyo shared, underscoring the life-changing impact of safe, reliable transportation infrastructure.

To commemorate the release of our episode, B2P staff returned to the Bundimulinga community in Bundibugyo District, where we hosted a special documentary screening. The event, held on August 6th, 2024, was a true celebration of the community’s achievements and the power of connection. The preparations for this event began with coordination and communication with local authorities, including the LC5 Chairman, RDC, DPC, UPDF Commander, and others, to ensure smooth operations. Invitations were distributed to community members in the form of B2P stickers, which also served as tickets to the screening.

On the day of the event, B2P staff arrived early at J and R Gardens, near the Bundimulinga cable trail bridge, to oversee the setup of tents, chairs, and the large screen. Local service providers from Nyahuka Town Council were engaged for catering, equipment, and security, ensuring that the community fully supported the event.

The celebration began with a vibrant motorbike ride across the trail bridge, where B2P staff and local motorcyclists wearing reflector vests participated in a festive parade. The sense of pride and joy was palpable as the community celebrated their accomplishments.

A particularly touching moment was when we presented Mrs. Karabiyo with a photo book, capturing the memories of her experience during the filming. Her grandchildren were delighted with B2P-branded school bags, and her youngest son proudly wore his new B2P cap. The smiles on their faces reflected the deep connection and gratitude shared between B2P and the community.

As the community gathered in anticipation, the event commenced with speeches from local leaders, landowners, and elders, all highlighting the transformative impact of the bridge on their daily lives. At 3:30 PM, the screening of the BBC documentary began, with the film shown three times to ensure everyone could watch. The atmosphere was electric, with cheers erupting as the community members saw their stories brought to life on the big screen. All shared pride in their achievements as they reflected on how the bridge has improved access to schools, health centers, and markets. The event concluded with a joyful feast, where food and soft drinks were served to the 200 attendees. It was a day filled with celebration, connection, and a renewed community spirit.

We are incredibly proud of the work done through Fika Map and the continued support from the Cisco Foundation. This episode of The Human Component showcases the innovative solutions being developed and highlights the resilience and strength of communities like Bundimulinga. As we continue to build literal and metaphoric bridges, we are committed to connecting more communities to essential services, creating opportunities, and improving lives worldwide.

Stay tuned for more stories of impact and innovation as we work together to bridge gaps and build futures. Thank you to everyone who has supported us on this journey.

Bridges to Prosperity’s Groundbreaking AI Tool Featured in BBC’s The Human Component was originally published in Fika on Medium, where people are continuing the conversation by highlighting and responding to this story.

At COP28 U.S. Secretary of State Blinken announced the commitment of $150 million to develop food crops well-suited to the changing climate in Africa. To support this substantial announcement for the Vision for Adapted Crops and Soils (VACS) the Plotline built and has launched the VACS Explorer to bring to life the research driving investments into key crops for African food security in a changing climate.

VACS Explorer is built on data produced by AgMIP projecting expected crop yields in a low or high emissions future. Earth Genome worked with our friends at Stamen Design to illustrate this data in a fresh and compelling visualization that clearly surfaces important information. Future crop yields are visualized in comparison to their historical levels, highlighting where crops will be harder or easier to grow in our changing climate. Crop yields can be compared against each other and assessed geographically across the continent to assist decisions on investments. VACS Explorer also profiles details on each crop including nutritional, genetic, and biophysical attributes. Today, the application is live to the general public at vacs.theplotline.org.

The VACS program is addressing the food future in Africa

Africa has been and will continue to be the region in the world most adversely affected by climate change. Vision for Adapted Crops and Soils (VACS) addresses food security in Africa by developing and mobilizing investment in resilient, nutritious indigenous crops suited to the changing climate.

The importance of a program like this cannot be understated. In the words of Secretary Blinken:

“If parents can’t feed their children, nothing else matters…. We’re working with partners to rethink what, where, and how we produce food, in the context of a changing planet. Our goal is for farmers, for ranchers, to be able to sustainably achieve bigger yields of more nutritious crops, at lower costs, using less land, producing lower emissions.”

At The Plotline, we’ve worked closely with the Global Food Security group at the US Department of State researching and exploring topics that they’ve brought to our attention. We built a tool to explore the impacts of the Russian invasion of Ukraine on grain production, developed a map based story on the 2022 flooding in Pakistan, and aggregated food insecurity data into an interactive application. These projects have all been focused on the past or current state of food systems. VACS is different — it’s a future focused vision crafted by a community most familiar with the shortcomings of our current food systems, designed to solve many of these challenges in a world where there will be more people to feed and a much harsher climate.

The Global Food Security group introduced us to the climate and agricultural modeling group, AgMIP, who had the task to make predictions on how climate will impact crops in the future.

We worked with AgMIP to take their preliminary results and develop the VACS Explorer application to bring to life the key insights. You can read more about how we worked with their data and prepared it for the application here. Below, we share a few insights and learnings we’ve found from using this tool.

Crops respond to climate change differently

These maps show the predicted yields for 20 crops across Africa in the year 2050, in a world that does not significantly cut back on greenhouse gas emissions. Specifically, this represents the climate scenario SSP370. The green above indicates areas where crop yield is predicted to increase, yellow where the predicted yield would decrease. As this shows, certain crops like maize are projected to see more major impacts than crops indigenous to Africa like tef or fonio.

When comparing the scale of breeding programs supporting these three crops it’s evident that Maize is getting more support than the indigenous crops. If we want to seriously mitigate the harms of climate change on future food security, we need to invest more in resilient, nutritious indigenous crops suited to the changing climate.

Climate will influence nutrition not just calories

In the comparison tool of the VACS Explorer, crops are sorted by different nutritional parameters. Here are the top 5 crops as ranked by protein content.

Clicking into the explore page, it’s revealed that a high protein crop like soy bean is predicted to be greatly impacted by climate change in West Africa.

Food security projections often look at calories alone. Many high calorie crops like grains aren’t projected to decrease as significantly as high protein crops and high iron crops. Surfacing nutritional benefits of crops introduces a more nuanced lens for answering questions about how to both provide enough calories for a growing population and healthy plant based proteins and vitamin rich diets across the continent.

Greenhouse gas emissions will define the extremes

One of the more prominent insights of VACS Explorer is how higher levels of greenhouse gasses in the future will create more extreme outcomes. For any of the crops, the most obvious change between the high and low emissions scenarios is everything looking brighter. For example, take the legume lablab.

Where crops change in yield, these changes are more amplified by differences in projected emissions. This is even more evident when examining the charts on the map explorer page.

The white line on the strip plot illustrates crop yield under a low emission and high emission scenario, and shows a significant decrease of lablab yields in West Africa under a high emissions future. Even more interesting, the general trend is that many locations skew towards the extremes with the more regions seeing yields with higher decreases or sometimes increases. In the strip plot, this trend appears as more brightly colored lines towards the edges of the plot.

Feeding the future is a shared vision

Spending significant time analyzing the data in this application has offered a unique perspective into climate change and food security. We’re often presented with the narrative that climate change will negatively impact food systems in the future; this is true, but not a binary. Certain foods will do much better in a warmer climate and these are often crops that are historically grown in these regions. To feed a growing population and create a more food secure world amidst a changing climate we must not only learn from indigenous communities, but give these communities a voice and platform to share their knowledge. VACS is a good example of a program that is doing this and supporting this vision with significant investments.

Explore the application and learn for yourself the types of crops that will do better in a changing climate. Take note of the types of agriculture that are providing diverse nutrients and plant based proteins. Do you know organizations or programs driving investment, research, or distribution of locally adapted crops to historically food insecure communities? Please share in the comments or get in touch.

Building the vision for a more sustainable food future in Africa was originally published in Earth Genome on Medium, where people are continuing the conversation by highlighting and responding to this story.

Today the Plotline, the food climate and data community organized by Earth Genome, is launching Food Twin, a proof of concept digital twin of the United States food system. Food Twin is based on a custom model, built in collaboration with food system expert Zia Mehrabi, that maps production of grown foods in the United States to where these foods are consumed. This model reveals a brittle and centralized food system that is at a high risk to be impacted by climate change economic shocks.

Why Build a Food Twin?

Did you grow the food you ate for breakfast this morning? Do you know who did? I certainly didn’t and don’t know where it even came from. In one view, our society’s ability to abstract away activities like hunting and gathering is what defines progress; no longer are we spending our days worrying about where our next meal is coming from or storing food so we don’t go hungry over the winter. However, this abstraction also comes with real risks. The food system isn’t without cracks, but most of us don’t have a clear line of sight into where these are, who they will impact, or what will make them worse. As the impacts of climate change become more evident, it’s concerning that such a fundamental resource is so opaque.

The Food Twin developed by Zia Mehrabi and The Plotline in collaboration with Development Seed is a proof of concept digital twin of the United States food system designed to model and illustrate our food system from production through consumption. Recent events such as the COVID-19 pandemic and Russia’s invasion of Ukraine have revealed inherent gaps in the way food systems are modeled. Particularly, these events have exposed how shocks cascade through food systems and impact food security at an individual and community level. Digital twins of food systems hold massive potential to support decision making during and prior to food crises, including those caused or exacerbated by climate change.

The application visualizes a custom built model designed to predict the connection between where food is grown and where it is consumed. Each colored point moving on the map correlates to an amount of calories from that food group. The consumer view on the map display how the food consumed in a county travels to that region; in the producer view it shows where food that grows in that county is consumed and how it gets to that area.

Climate change is destabilizing our food systems across the world, and the US is not immune to this stressor. A missing piece of the puzzle is how extreme weather shocks, which are becoming more common, experienced in one location, are felt by consumers in other locations. In The Food Twin we model climate impacts for all major grown crops in the US. We couple these predictions with our food system model, to identify communities at risk at the other end of the supply chain when drought and heat wave events hit. The scenarios we represent are illustrative in nature, but highlight how digital twins can be used to evaluate how risks spread through food systems, affecting commodities and consumers across the nation.

Our Takeaways

The Food Twin project, although still just a proof of concept, offers several insights into the current state and future of the U.S. food system.

Our current system is brittle and centralized

One of the key takeaways from the Food Twin model is the inherent brittleness of the current food system. In our model, just 5.5% of U.S. counties are responsible for half of the country’s food production. Additionally, this system relies on a limited number of distribution centers and transport routes, further exacerbating its fragility. This centralization, and the just in time contracting environment on which so many consumers rely, creates vulnerabilities, particularly in the face of unexpected events like pandemics or geopolitical conflicts.

Vulnerability to climate change

The model vividly illustrates how weather is already impacting food production. For instance, Kern county, one of the counties producing the highest diversity of crops stands to lose almost 8% of its production in a heat stress event like the one we’ve modeled. With the predicted increases in extreme weather events and shifting climate patterns, these impacts are only expected to grow. This underscores the urgency of integrating climate resilience into food system planning and management. And critically highlights how shocks felt in one distant location propagate through the supply chain to consumers.

Need for diversification and localized solutions

The project points towards the need for more diversified food systems, and proactive policies to ensure consumers relying on brittle or fragile supply chains are buffered from shocks in the future By reducing reliance on a small number of food sources and distribution channels, the system can become more resilient to climate & economic shocks, and the impact felt by consumers buffered from shocks that happen further up in the supply chain

Model Description

For those interested in the technical details, here’s a short summary of how the model is built.

Production

The model that powers this tool uses the USDA’s Cropland Data Layer to generate an estimate of the types of crops that are grown in different regions across the United States. This data is converted from satellite data observations (pixels) to production using USDA Census of Agriculture state outputs from this study on crop stability and diversity. Production to calorie conversions, accounting for waste, as well as losses in conversion to non-plant commodities (such as crops used for animal feed or biofuels) are made using the balance sheets from the Food Availability (Per Capita) Data System. Imported foods are aggregated from the US Census Bureau data; the app shows imported foods as being sourced from counties where the port of entry is located.

Consumption

Consumption per county was calculated using food availability surveys from the USDA and paired with demographic data from the census. Census data used in the model include population, racial demographics, and poverty rates. We used this data to determine the demand for food groups in counties. Population, wealth, and racial demographics influence the types of food groups and amount of calories consumed.

Connecting supply and demand

Consumption and production data were connected via a matrix that took into account possible shipment routes between all counties in the United States. Each crop’s supply and demand was solved in the most efficient way possible based on minimizing travel time along roads, given the central role of trucking in moving food in the nation (trucks move 84% of agricultural freight and 88% of volume and nearly every trip from farm to fork includes at least one trucking step).The routes were calculated using a version of the OSRM algorithm.

Climate impacts

Reduced yield due to climate stress was generated using statistical crop yield models. Specifically, we estimated the impacts of weather (main growing season temperature, precipitation, standardized precipitation index, and days >30C) on 25+ crops and crop groups within the USA over the time series 1983–2016. Statistical crop yield models were fit to state level observations and the yearly impact of weather predicted at the county level. Predicted yields are compared for a given crop and county for a given year and weather observation to the predicted yield under the mean climate observed in that county, to identify the variation in yield due to weather. Illustrative drought and heat wave impacts were identified by drawing on the spatial distribution of drought and heat waves from 2012 and 2016, respectively, and enhancing the magnitude by 25%, to simulate what might be expected for consumers under a more extreme climate in the future.

More information

• This repository lists all data inputs that are used in the supply chain model
• This notebook contains the scripts used to develop data for the supply and demand models
• This folder contains the scripts used to solve the routing connecting supply to demand
• This repository contains information and code used to generate the statistical crop yield models and climate impact data

The Future of Food Twin

Although this model and application offer insights into the U.S food system, our next step at the Plotline is to build a global model. Food systems are interconnected; analyzing only a single country misses out on many of the rippling effects shocks can have. We’re actively convening experts, researchers, and enthusiasts to join this effort. Whether you’re a data scientist, a climate expert, or just someone passionate about food security, we’d be interested to hear how you would use a digital twin model like this.

Animal food products are the other major piece to this model that we want to explore in the future. A large percentage of land area and grown foods go towards meat and dairy products. Animal based proteins are not only some of the highest contributing factors to climate change, but also vulnerable food sources. Integrating non-plant commodities explicitly in our representation is certainly a next step, as is increased resolution on different modes of transport and representation of the supply chain surrounding particular products.

Beyond the model itself, we at Earth Genome believe in the power of open data and community involvement. By making our data and findings accessible, we hope to foster a broader conversation about food security and sustainability. We encourage you to share this information within your networks and start dialogues about food sourcing and food system solutions. If you want to get involved here are a few ways we’d love to collaborate:

1. Engage with the data: Explore the data we’ve compiled. Use it to inform your understanding and start conversations about food systems in your community.
2. Spread the word: Share information about Food Twin and its insights. The more people are aware of how food systems function, the more we can be insightful as a society as a whole to build resilience.
3. Support your local community: Where does your food come from? How can you build more resilient food systems for your friends and family? Start asking these questions and exploring ways you can strengthen your local food system against shocks in future.

The Food Twin project is more than just a digital model; it’s a step towards a more resilient, sustainable, and equitable food future. Join us in this crucial effort to reshape how we understand and interact with our food systems. Here’s to a future where we know where our breakfast came from and can have confidence it will be there tomorrow.

Food Twin: Stress Testing the U.S. Food System was originally published in Earth Genome on Medium, where people are continuing the conversation by highlighting and responding to this story.