Interactive web viewers are an important part of my work as a data scientist at Stanford University, supporting decision-makers to make targeted spatial planning choices for people and nature. When teaching, I get asked a lot about underlying tools, so here’s a quick take on the matter.

Interactive data visualisation is fantastically useful

In the data era, where we suffer infobesity (information overload), making sense of lots of data efficiently has become an important skill; as a consequence, data visualisation has boomed. Indeed, data viz has become essential to explore large multidimensional datasets. Interactive data viz is particularly helpful to link different views of the data, filter through big datasets, or visualize fine resolution maps across scales…

In my work on ecosystem management, I’ve seen how communicating complex quantitative results to decision-makers in a user-friendly format is key to the success of a project. Interactive viewers facilitate discussions and tend to engage users easily, letting them investigate the datasets intuitively. In addition to their attractivity potential, they are really helpful to gain insights from complex datasets, research shows.

Famous global viewers monitor deforestation, coral reefs health, climate resilience, food systems, resources, sea level rise (…), and most recently coronavirus cases. These often even serve as databases for their topic, gathering up-to-date datasets. At a smaller scale, project-specific viewers serve a specific goal and intended audience, allowing collaborators to share and visualize data, fostering discussion across groups as teams iterate. For example, in the context of spatial development plans, viewers help partners investigate scenarios of developments, such as impact of the construction of a new road or of new protected areas…

My favorite data visualisation tools & libraries

So, what’s the best data visualization tool? There’s no one size-fits-all magic solution; like any good visual, the right tool depends on your audience, data formats and the key message to convey. Below, I’ll list the tools that I enjoy using, and link some examples.

For web-viewers, leverage powerful JavaScript libraries

Let’s start with the most powerful interactive plotting library: D3.js is the beast of interactive data visualisation. From the very simple interactive charts easily made with C3.js, to the most badass psychedelic, artistic, and insightful visuals. It cover almost all types of plots, from chord charts to word trees. D3.js is not easy, but there are plenty of tutorials and examples to support.

When it come to spatial data, Leaflet is the queen library for interactive mapping. Its power relies on its plug-ins, contributed by the community, which adapts to the trendiest data types. I use these for, in particular the one for shapefiles, geoJSONs, and most recently, found this one for (relatively small) rasters. For bigger rasters (and fastest display), I recommend using tiles. Commercial solutions (like Mapbox, ArcGIS Online or Carto) let you make and host tiles easily, but have data hosting limitations, and are never 100% customizable. I prefer to create my own tilesets, and it’s not as complicated as it seems (3 GDAL commands and then serve them smoothly with GitHub pages) > see my fantastic hack to building and serving home-made tilesets!

Now, for satellite imagery, Google Earth Engine (GEE) is probably your best bet. Pros: if you’re displaying datasets hosted by GEE (and they host many!), you’ll save considerable time. You also don’t need to worry about hosting or serving your viewer, as it’s hosted and served through GEE Apps. Cons: While the coding is simplified (thanks Google), it’s sandboxed so you don’t have full control to create the pretty interface you’re dreaming of. Also, app deployment is still in beta version. And it’s not open-source of course.

Deploying your viewer

GitHub Pages makes it fantastically easy to deploy your own viewer (Settings > Options > GitHub Pages > Source > choose your branch and viewer folder). Your viewer’s URL looks like {your-username}.github.io/{repository-name}, but is customizable if you own a cooler domain.

Only need a fast prototype? If you don’t need to deploy a whole web-viewer, you can save a considerable amount of time (basically skip all the web development: no HTML nor CSS, jump straight to the JavaScript you need for your viz), by using Observable’s interactive JavaScript notebooks, usable online by loading your data.

What if you’d rather use Python?

Allergic to JavaScript? Worry no more. Some great Python librairies (Bokeh, Altair) let you to play around interactively (in Jupyter notebooks). Though Javascript remains the reference for anything that will be shared on the web, most of the js librairies mentioned above also have a python version (e.g Folium integrates Leaflet.js for Python data).

What if I don’t code?

It’s always a trade-off between customizability and ease of implementation.

Commercial solutions exist (ArcGIS Storymaps allows easy creation of interactive stories combining medias, text, maps and charts; Tableau lets you create dashboards easily…), though they are inherently limited in terms of functionalities and customizability. Most function on a Freemium model. I honestly don’t know them well enough to provide good recommendations.

If you aren’t afraid of looking at code, and you know enough github to get your way around, try this:

Build your custom web-viewer in 2 hrs!

Getting a hold of these visualization libraries, and getting everything to work together smoothly — from data formatting to deploying a viewer online — can be time consuming. To simplify this process, the wonderful Anna Häägg and I made a template for our colleagues to build web viewers for free, really quickly. It’s all open source, so it’s yours too!

https://charlottegiseleweil.github.io/viewerTemplate/

It’s still quite new, so your feedback would be very welcome — take a look the demo, and clone the repo to get started.

Bottomline: as illustrated in this article, there are a plenty new libraries and tools to make ever fancier, prettier, colorful webmaps… I recommend to stay focused on the key message(s) to convey and the audience when picking a tool and designing a solution — sometimes the simplest is the most effective!

Keep in mind graphical integrity principles, and have fun!

My favorite (free) interactive data visualisation tools was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Last week, the world turned their eyes to the Amazon, watching the rainforest burning. Over ~20 000 km2 (7200 square miles) have burnt this year in the Brazilian Amazon (Jan-July, according to Brazil’s National Institute of Space Research, INPE). That’s almost the size of the whole country of Belize, and this means 1 or 2 soccer fields of rainforest burnt every minute.

Loss of the lungs of the world has disastrous consequences. Forests are carbon sinks - when they burn, they release the carbon to the atmosphere, thus intensifying climate change. Not only that, but trees cooled down and humidified their surroundings, the newly cleared land leaves a drier, hotter environment prone to more fires (ugh, that’s a nasty feedback loop 🤦‍♀️)

AmazonFires

Humans are to blame

The vast majority of these fires aren’t natural. Farmers set fires to clear land for agriculture, or mostly cattle ranching. Indeed, Brazil is the #1 beef exporter — this same beef you find in your fast food cheeseburger. So yes, deforestation and our diets are tightly linked. Greenpeace estimates cattle ranching to be responsible for 80% of Amazon forest destruction.

Demand for beef drives need for cattle land which leads to deforestation. As simple as that.

A. Nature and its vital contributions to people are deteriorating worldwide.

Nature underpins all dimensions of human existence, including (physical and mental) health — just an example: 4 billion people (estimate) rely primarily on natural medicines for their health care (and 70% of cancer drugs are natural or nature-inspired).

Contributions of nature (like land productivity) are declining. Humans have now altered 75% of lands and 66% of oceans, while 85% of wetlands are lost.

B. Change has accelerated during the past 50 years

1 million species already face extinction, but it’s not just biodiversity loss: land degradation, pollution, pollinator loss ( → food insecurity), forest loss, fires, floods, hurricanes, droughts… enhanced by climate change. Meanwhile the pressure increases, with ever more people consuming ever more.

Nature deterioration is mostly due to land and sea use changes, exploitation of organisms, climate change, pollution and invasion of alien species.

C. Sustainability goals will NOT be met with current trajectories.

Many climate change actions are successful, but there are way too few. Paris Agreement won’t be met unless highly transformative societal changes (including in production & consumption of energy, food, water, and more).

Regions projected to be the most impacted are home to many of the world’s poorest & indigenous communities.

D. Through transformative changes, including nature-based solutions, it IS possible to both restore environment & meet societal goals (such as feeding humanity)

This implies steering away from the current limited paradigm of economic growth (reducing inequalities, waste, overconsumption)

Recipe for success: Good incentives & Education, International cooperation, Precautionary action, Resilient choices for uncertain future, Strong environmental law.

Key leverage points:

• Redefine our vision of a good life (not linked to material consumption)
• Lowering consumption and waste
• Give responsibility & account for externalities
• Reduce inequalities (income, gender). Justice & inclusion in conservation
• Environmentally-friendly technology, innovation and investment
• Education, from a plurality of knowledge systems.

More: IPBES’ 2019 Global Assessment Report on Biodiversity and Ecosystem Services.

Day 1 — Explore: What are the deep or neglected leverage points for sustainability transformations?

In a building where no room is square, trees and pouffes are installed in the main foyer (“the forest”), colorful games await players in the corners of the hallways alongside participatory maps while music plays in the background. Unlike any other academical conference I’ve attended, this one feels particularly human and coherent, serving (delicious) vegan meals, offering kids’ care and even a quiet meditation room. Clever minds sharing kindly and humanly — “This is the Academia I love” I thought, and I felt comfortable, comfortable to be vulnerable, where creative magic happens.

« This is the end of the world as we know it ».

I felt inspired by the opening keynotes by Ioan Fazey sparkling with energy and magic tales and by Elena Benett, who leads a marvellous initiative gathering Seeds of a Good Anthropocene.

Reconciling Science & Spirituality

I discovered about Inner Sustainability, and felt whole, conciliating a lot of what makes me feel at home in my intentional community house (where 25 wonderful beings share their journeys of personal healing, spiritual growth & social transformation), and my professional life as a researcher data scientist, which brought me here.

A lot of this wasn’t new to me: Eco-psychology, Deep Ecology, Eco-feminism, the incredible Work That Reconnects by Joanna Macy…, but it was really novel to see these topics approached and formalised from a researcher perspective. Shifting the paradigm by truth and love?

Questioning paradigms: Humans as part of Nature

I work for the Natural Capital Project, where our mission is to shine a light on the benefits people get from nature. There’s an ongoing thorny debate on whether to call those Ecosystem Services (ES) or Nature’s Contributions to People (NCP). Both options are somewhat problematic though, as they emphasize the distinction between people and nature. Aren’t people part of nature?

I’ve had challenges with this phrasing for a while (but don’t have a satisfying alternative either). Although modeling ES (or NCP!) is absolutely not “putting a dollar tag on nature” (and Anne Guerry explains it very eloquently: Nature is priceless, so let’s value it!), after all, we do work on quantifying “Natural Capital”, to “correct capitalism myopia”. Meanwhile, we live the stories that we tell, but how could we deeply change our story — if we tell it in capitalist words?

« We cannot solve the problems with the same thinking we have created it »

said Albert Einstein. Could we combine new desirable narratives with the capitalist-friendly terminology to anchor the pathways to better futures in the present?

Day 2 — Navigate: How do we navigate complexity?

By the end of the first day, it seemed like the assembly was ready to “pull the levers”, but this second day is ambiguously positioned between still exploring and not really acting. I heard loads of buzz words today, one of them being self-organization, I’ll let them self-organize:

Despite lots of insights, I deplore that no clear reasoning or conclusion emerged (for me at least). Universities aren’t fit to solve today’s challenges, suggests Ray Ison from The Open University, but in a quite academical-text-dense-powerpoint presentation. In another talk (otherwise fascinating) on wicked problems, a slide titled Best practices recommends “Better communication” (avoid academical jargon), and in the next bullet point “Epistemological Pluralism”. Walk your talk?

So, I felt concerned about our (academics) capacity to transform, and transform beyond academia… On the other hand, it was wonderful that not just academia was represented. Charlotte Niekamp, market gardener, invited us to care for, recognize and honor farmers: “Peasantry is dying, but peasantry is not dead!”. Plus, you know you’re in the right place when spirituals healers are part of the conversation! Gogo Dineo Ndlanzi made my soul smile and reminded us that disease (when you’re not at ease) is a spiritual awakening. “We’re spiritual beings, having a human experience”. As we were invited to ask the really hard questions repetitively this week, she reminded us in a rhythmic voice about all the indigenous knowledge to learn from and to get out of comfort zone:

Discomfort is where change comes from.

Wicked problems, complex interdependencies, ever-evolving parameters, non-linear and not solvable questions… Plus, complexity of the system increases as one start analysing it (is academia making complexity more complex?). As we discuss how to be good Earth Stewards, and navigate complexity, we seem to agree that radical listening, letting go, self-organization, humility, empathy and diversity are keys. Embracing complexity, relaxing into it but not letting it become an excuse for not acting!

Graphic facilitation for Knowledge Harvesting

I was particularly excited about the graphic facilitation very present at this conference. Using post-its (renamed golden nuggets for the occasion), drawing walls and an army of students, graphic facilitators take visual notes of the talks — extremely useful as reminder and visual synthesis exhibited in the main foyer. They even take it to the next level when projecting the visuals as they are being drawn, which I would argue is when it becomes too much — as participants lose focus observing the drawings, and interprete the talk through the artist’s lens before their own.

Talks are 12 minutes, which is fantastic to get a large plurality of opinions, but we mostly all did a terrible job at it: trying to squeeze as much information as possible, and speaking way too fast about slowing down. By 5pm, when we re-group in plenary, the energy and attendance are particularly low despite the almond-milked coffee we’re all drinking. Everyone is feeling tired.

Day 3 — Act (or Explore, explore, explore?)

I learned about Edible Cities, Horticulture-based healing for prisoners, biomimicry and nested hierarchies as a mean to change worldviews, collective Earth Stewardship full of aliveness, and also the wizard-giant-fairy version of paper-rock-scissors (highly recommended).

Throwing microphones randomly to participants, just like at Google, effectively led to democratic participation. A student calls out researchers for pretending to be objective. Researchers are traditionally supposedly neutral external observers of the system. But one can’t abstract themselves from a system they are part of. Plus, being a both a citizen and a researcher of the sustainability transition entails its share of challenges. It raises interesting questions around when to honor subjectivity, and the place of intuition is the knowledge-creation process.

« It’s not just about matter, it’s about mattering, and you matter more than you think »

I was precisely drawing fractals when Karen O’Brien, IPCC contributor, walked on stage. In an analogy to physics, where we get to weird quantum theory at the sub-atomic level, she shared the theory of quantum social change — which I think I understand it as such: We’re all waves and particles, and we’re connected, beyond just the bits and bytes of information shared through optical fiber. Individuals aren’t 100% separable entities. This “non-local entanglement” implies that social science is essentially based on a mistake (assuming humans obeyed the laws of classical physics). Rather than being deterministic, humans are “walking wave functions”…

So, what if we were underestimating our capacity for social change? From crisis to opportunity, let us choose the stories we tell and the ones we live.

I am because you are and since we are, therefore I am

Thank you Dave Abson & team, feeling grateful for these inspiring 3 days. In fact, they felt more like Explore-Explore-Explore than Explore > Navigate > Act, but I very much enjoyed exploring, and am left with deep questioning and allowing myself to take the time to meditate on them. As Elena Benett pertinently cited one of my favorite quote from Antoine de St Exupéry :

“When you want to build a ship, don’t begin by gathering wood, cutting planks and dividing the work, but rather by awaking in hearts the longing for the endless immensity of the sea”. (freely translated)

Notes

The need for datasets of georeferenced dams

Millions of dams worldwide have been constructed to manage water resources for human benefit. They also have immense environmental impacts: on freshwater biodiversity (e.g dams modify habitats, block fish migration…), hydrological ecosystem services and human health. Despite this, only the largest dams have been mapped, but they are far outnumbered by smaller dams that have tremendous impacts.

Efforts by researchers, NGOs and governments to understand and mitigate the impacts of dams and manage hydrological ecosystems smartly are restricted by the available datasets of georeferenced dams.

Indeed, Lisa Mandle, Senior Scientist at the Natural Capital Project, Stanford University studies how Myanmar’s forests contribute to the country’s water supplies by keeping dams free from sediments. “We needed to know where these dams were located. This data turned out to be hard to get. It was spread across different government offices, and not publicly available in any comprehensive, digital form”, she reports. “The best information we could find included only 15 dams in Myanmar. By only considering the largest handful of dams, we know our analysis underestimates the importance of nature.”

Halfway around the world, Marcia Macedo, Assistant Scientist at the Woods Hole Research Center, ran into a similar challenge while studying the impacts of small agricultural dams in the Brazilian Amazon. “Our field studies clearly show that small reservoirs have had a big impact on freshwater ecosystems,” she explains. “A single reservoir can increase stream water temperatures by up to 4°C. It also fundamentally changes aquatic habitats, nutrient cycles, and food webs.” Macedo has already mapped over 10,000 small dams in Brazil. “We know there are millions more of these dams out there, and that they have a massive cumulative impact. But we can’t manage them if we can’t map them — inexpensively, at high resolution, and over very large areas” she adds.

The solution: satellite imagery + deep learning magic

With satellite imagery increasingly accessible (thanks to open data from policies, e.g from Nasa or ESA and tools such as Google Earth Engine), plus the magic of machine learning, it sounds like you can do anything! But seriously, deep learning algorithms distinguish cats from dogs, detect obstacles in front of cars and sort cucumbers among many other applications of image classifications. So it seems reasonable to ask:

Can we detect dams from satellite imagery?

In the span of a 4-hours hackathon at GeoForGood 2018, we approached this question — and prototyped a pipeline using Google Earth Engine to create the training data and Tensorflow for the deep learning dam classification model.

Sounds like yes, we can locate dams from satellite imagery. So now, the question is:

What would it take to locate every dam on the planet with satellite imagery?

That all sounds wonderful but really, this crazy 94% accuracy mostly demonstrated the pipeline worked. Doesn’t represent a reliable result. So let’s dive in, and discuss what we’ve done, some challenges, and how we could do it better.

Workflow

1. Training data

For dams images, we used the GRanD dataset to export raster images of areas surrounding dams outlets, as TFRecords. These include 5 bands: RGB and NDWI from Sentinel 2 (15m), as well as elevation (DEM data: ALOS DSM 30m).

For non-dams images, we prioritized areas at the edge of water bodies that are easily confused with dam reservoirs. Otherwise the algorithm would classify between edge-of-water and everything else, instead of locating the dams outlet (hence the 94.4%!). Using the JRC Global Surface Water dataset in GEE, we randomly sampled points on the edge of water bodies along with non-water points (so the algorithm wouldn’t get confused by, for example, the straight line of a road…) and followed the same procedure to export rasters of 300m*300m as TFRecords.

2. Deep learning classification

You were wondering what TFRecords were, now it’s all going to start to make sense: a TFRecord is a (binary storage) data format optimized for use with Tensorflow, an open-source library particularly handy for deep learning applications such as … image classification!

We were very lucky to come across a notebook developed by Chris Brown which trained a fully convolutional neural network (FCNN) to detect cars in images of parking lots. A FCNN allows to make predictions on images of any dimensions, so we can train the model on our 300m*300m images, and make predicitions later on images of any dimensions (yay!)

3. Results

The latest iteration was trained on a fusion of about a thousand training data points and resulted in an accuracy of 92% on a 600-points test set, with 50,000 steps.

The results are very promising, yet the approach still needs to be tuned. Nathan Pavlovic, who played the Machine Learning expert with virtue for this Hackathon, raises his eyes from the computer, seemingly satisfied. But he still notes a couple issues, for example that the algorithm still seems to sometimes confuse dense forest with water, as seen in the lower center of the image. None of the issues are insurmountable. In fact, this one is probably due to the fact that dense forest was an under-represented class in the non-dam training set we quickly put together. With more time, a number of improvements could be made on the training data to increase performance.

So it does seem very possible to locate every dam on the planet, with satellite imagery!

Now what?

What about commercial solutions for object recognition from satellite images? They’re not helpful on this question, so far (and we love open-source anyway). Descartes Lab has an impressive geovisual search tool, however limited (to a fixed search area and a single global satellite dataset), its featured applications seem quite promising. But it does a pretty terrible job at locating dams (likely because it’s just using RGB bands, where elevation and NDWI are key here). Let’s see what Planet Queryable Earth will offer in the future, with its mission to “index and make accessible what’s on the Earth, just as Google indexed and made accessible what’s on the internet”… For the moment, these algorithms remain proprietary anyway.

A couple challenges identified so far include the extreme variability in dam size and reservoir water storage. To address them, inputs data temporal scale and spatial resolution would need to be adapted, e.g we may need images from several points in time, from different seasons or years, to capture dams that dry out seasonally or during droughts. And what about the very small dams hidden in a forest?

If we were to do it better, the training data could be augmented and improved, finely tuned to avoid the artefacts we identified: more training points, but also higher resolution imagery, especially for smaller dams - e.g Planetscope captures 3m images, however these aren’t freely available at scale nor easily integrated (yet?) in Google Earth Engine.

Acknowledging wonderful people

Thank you Nathan Pavlovic, Lisa Mandle, Nick Clinton, Marcia Macedo, Chris Anderson, Chris Brown, Xavier Giroux-Bougard, Melissa Hanham, Heena Tabassum, Pedro Ramos.