Lake Mendota in Madison Wisconsin is arguably one of the most-studied lakes in the world and boasts an ice phenology record dating back to the 1850’s. The Climatology Office at the University of Wisconsin, which currently maintains the record, has made amazing strides to make the data downloadable. However, a simple plot of the ice-on date over time still requires some work by the user to make it legible. This is the reason I created mendotapy, an open source (MIT-licensed) Python package that allows retrieval of this valuable data in an analysis-ready format. It has some built-in utilities to make common transformations a breeze.

I find this dataset interesting from a climate perspective, but I think anybody can find this dataset relevant. For instance, it could be used as a toy dataset to practice regression analysis, statistical forecasting, or spectral analysis.

In this post, I will describe the package, present some plots of the data, and conclude with the challenges I faced making this package. All the code to generate the figures is available in this post and the package code is available on GitHub (contributions are welcome)

Installation and using the package

First, you will need to install the package via pip.

pip install mendotapy

Once installed, it only takes two lines to retrieve an analysis-ready copy of Lake Mendota’s ice phenology returned as a Pandas DataFrame.

import mendotapy
df = mendotapy.retrieve_data()

The beauty of this is all the pre-processing has been done for you so you can you jump right into analysis! Figure 1 shows a screenshot of the dataframe.

You will notice that some seasons are duplicated (e.g. 2018–19). These indicate years where the lake thawed twice in a single season. However, the climatology office only provides the duration for the most recent freeze/thaw cycle in the season.

I have also added latitude , longitude , lakename , and lakecode to the dataset for consistency with the National Snow and Ice Data Center’s (NSIDC) Global Lake and Rive Ice Phenology. The latter is a great resource for ice phenology data from over 500 lakes and rivers. Although, at the time of writing, the database contains a temporal coverage up to February 2nd, 2018. mendotapy gives you access to the most recent data for Lake Mendota.

Dataframe Accessor

What if you just want the start of the season? For instance, the season is 2023-24 but you just want 2023? Or what if you want the day of the year (DOY) for the ice on date? To facilitate this sort of functionality, a dataframe accessor was added to the package.

This additional functionality is within an accessor named utils

df.utils.season_start() # returns list of the start of the season

df.utils.iceon_doy()    # returns list of ice on day of year
df.utils.iceoff_doy()   # returns list of ice off day of year

This makes plotting the ice on DOY like this:

import matplotlib.pyplot as plt

season_start = df.utils.season_start()
iceon_doy = df.utils.iceon_doy()

plt.plot(season_start, iceon_doy)

Which produces a the graph shown in Figure 2.

Why does this look so spiky? In most seasons, the ice-on date happens in December, but sometimes it occurs in January if the summer is unusually warm that season. One way to improve this plot is to “wrap” the DOY values into the following year. In other words, add 365 to DOYs occurring in January. to do this in mendotapy , simply use the df.utils.iceon_doy_wrapped() method. So the new code looks like

import matplotlib.pyplot as plt

season_start = df.utils.season_start()
iceon_doy_wrapped = df.utils.iceon_doy_wrapped()

plt.plot(season_start, iceon_doy_wrapped)

Applying this trick, we can clearly see the upward trend in the data

How to make an accessor

Making an accessor like this was new for me and opens up a whole world of possibilities with Pandas (and Xarray too!). Let’s dive into how these accessors work.

My approach was to start with a function I know works on a DataFrame. For instance, the function below will return a list with the start of the season.

def season_start(df):
        """start of the season

        Returns
        -------
            list: start of the season.
                  e.g., the 2023-2024 winter season return 2023
        """
        season_list = []
        for index, row in df.iterrows():
            season = int(row["season"].split("-")[0])
            season_list.append(season)
        return season_list

You can then apply the function to a DataFrame.

import mendotapy

# load raw data
df = mendotapy.load()

# apply season_start function
start_of_season = season_start(df)

To turn this into an accessor and use the function like this df.utils.seasons_start() we need to to take the following steps:

1. Make the function into a class method (class name is irrelevant)
2. Initialize the class with a DataFrame
3. Decorate the function to register an accessor

Here is the code that makes this work in mendotapy

@pd.api.extensions.register_dataframe_accessor("utils")
class MendotapyAccessor:
    def __init__(self, df):
        self._df = df

    def season_start(self):
        """start of the season

        Returns
        -------
            list: start of the season.
                  e.g., the 2023-2024 winter season return 2023
        """
        season_list = []
        for index, row in self._df.iterrows():
            season = int(row["season"].split("-")[0])
            season_list.append(season)
        return season_list

The secret sauce that makes all this work is the decorator. The name you pass as an argument will be the name of the accessor.

@pd.api.extensions.register_dataframe_accessor("utils")

Accessors are great for grouping related methods or properties together. However, for mendotapy I took the naive route and dumped all the methods into a single accessor named utils .

Plotting and Analyzing the Data

This is a fun dataset to play around with and one of my go-to-datasets when explaining climate change and global warming. It was even featured in a 2018 data visualization battle (r/dataisbeautiful DataViz battle).

Although I did not participate in the battle, I took a stab a making my own visualization. Whether it’s “beautiful” or not is still up for debate. My goal was to make a compact, and informative, graphic conveying as much relevant information as possible. Here is what I came up with. Code used to generate this figure is at the bottom of this post.

You will notice a distinct downward trend in the duration of ice cover, due to the ice-on date occurring later in the season and the ice-off date occurring earlier. This trend is mainly the result of human-caused warming, which has a more dramatic impact on the winter months. See the blog post below for a more detailed explanation of this.

Climate Change in Madison, WI

And for another wintery climate related story, check out this post discussing snowfall trends in Minnesota.

Does record snowfall disprove global warming?

Code used to generate the summary diagram

The code below was used to generate the summary diagram above. I am providing it here in the hope that it serves as a useful reference to help you enhance your plotting skills and create your own insightful visualizations.

from collections import defaultdict as defaultdict
from datetime import datetime

import pandas as pd
import numpy as np
from sklearn import linear_model as linear_model
import matplotlib as matplotlib
import matplotlib.pyplot as plt
from matplotlib.ticker import AutoMinorLocator

import mendotapy


#========================
# load the data
#========================
df = mendotapy.load()

#========================
# create figure and axes
#========================
fig = plt.figure(figsize=(12, 6))
gs = matplotlib.gridspec.GridSpec(nrows=2, ncols=2,  width_ratios=[1, 1])
ax0 = fig.add_subplot(gs[:, 0])
ax1 = fig.add_subplot(gs[0, 1])
ax2 = fig.add_subplot(gs[1, 1])

#========================
# duration plot
#========================
ax = ax0

x = df.utils.season_start() 
y = df['duration']

ax.scatter(x, y,
           marker='o',
           s=100,
           linewidth=3,
           color=(1, 1, 1, 1),
           edgecolor=[0.7, 0.7, 0.7],
           clip_on=False,
           zorder=2,
           )

# darker edgecolor if thawed during season
duplicates = df['season'].value_counts()[df['season'].value_counts() > 1].index
filt = df['season'].isin(duplicates)
x_thaw = df[filt].utils.season_start() 
y_thaw = df.loc[filt, 'duration']

ax.scatter(x_thaw, y_thaw,
           marker='o',
           s=100,
           linewidth=3,
           color=(1, 1, 1, 1),
           edgecolor=[0.3, 0.3, 0.3],
           clip_on=False,
           zorder=2,
           )


# Hide the right and top spines
ax.spines['right'].set_visible(False)
ax.spines['left'].set_visible(False)
ax.spines['top'].set_visible(False)
ax.spines['bottom'].set_visible(False)

# set axis limits
ax.set_ylim([0, 170])
ax.set_xlim([1855, 2022])

# tick marks
ax.set_xticks(np.arange(1860, 2021, 40))
ax.set_yticks(np.arange(14, 180, 14))

# major/minor tick lines
ax.grid(axis='y', which='major', color=[0.8, 0.8, 0.8], linestyle='-')

# make labels
ax.tick_params(axis='x', labelsize=16)
ax.tick_params(axis='y', labelsize=16)

# Turn off the display of all ticks.
ax.tick_params(axis='both',
               which='major',  # Options for both major and minor ticks
               top='off',  # turn off top ticks
               left='off',  # turn off left ticks
               right='off',  # turn off right ticks
               bottom='off',  # turn off bottom ticks
               length=0,
               )

# annotations
title_str = """Duration of Ice Cover (Days)"""
ax.set_title(title_str, fontsize=18, fontweight='bold', ha='center')

x_new = []
y_new = []
for xi, yi in zip(x,y):
    if yi is not None:
        x_new.append(xi)
        y_new.append(yi)

# linear regression
regr_out = linear_model.LinearRegression()
regr_out.fit(np.array(x_new).reshape(-1,1), np.array(y_new).reshape(-1,1))

# plot trend line
ax.plot(x_new, regr_out.predict(np.array(x_new).reshape(-1,1)), color='k', linewidth=3)


# Annotate the plot with the equation of the trend line
trend_per_decade = f"trend: {(regr_out.coef_).item() * 10:.0f} days / decade"
plt.annotate(trend_per_decade, xy=(-0.9, 2.05), xycoords='axes fraction', fontsize=12, color='k')
    
#========================
# iceoff plot
#========================

ax = ax1

ax.plot(df.utils.season_start(),
        [result if result is None else result + 365 for result in df.utils.iceoff_doy_wrapped()], color=[0.5, 0.5, 0.5], linewidth=2)

# Range ov axes
ax.set_ylim([340, 492])
ax.set_xlim([1855, 2022])

yticks_start_end_step = (340+7, 500, 14)
xticks_start_end_step = (1860, 2021, 40)

# Turn off the display of all ticks.
ax.tick_params(which='both',  # Options for both major and minor ticks
               top='off',  # turn off top ticks
               left='off',  # turn off left ticks
               right='off',  # turn off right ticks
               bottom='off')  # turn off bottom ticks

# Remove x tick marks
plt.setp(ax.get_xticklabels(), rotation=0)

# Hide the right and top spines
ax.spines['right'].set_visible(False)
ax.spines['left'].set_visible(False)
ax.spines['top'].set_visible(False)
ax.spines['bottom'].set_visible(False)

# major/minor tick lines
ax.grid(axis='y', which='major', color=[0.8, 0.8, 0.8], linestyle='-')

# Turn off the display of all ticks.
ax.tick_params(axis='both',
               which='major',  # Options for both major and minor ticks
               top='off',  # turn off top ticks
               left='off',  # turn off left ticks
               right='off',  # turn off right ticks
               bottom='off',  # turn off bottom ticks
               length=0,
               )

# Only show ticks on the left and bottom spines
ax.yaxis.set_ticks_position('left')
ax.xaxis.set_ticks_position('bottom')

# Don't allow the axis to be on top of your data
ax.set_axisbelow(True)
ax.set_xticks(np.arange(*xticks_start_end_step))
ax.set_yticks(np.arange(*yticks_start_end_step))

ax.set_yticklabels([datetime.fromordinal(doy).strftime('%b-%d')
                    for doy in range(*yticks_start_end_step)])

# Labels
ax.tick_params(axis='x', labelsize=16)
ax.tick_params(axis='y', labelsize=16, labelleft=False, labelright=True)

ax.xaxis.set_ticklabels([])

# Annotations
title_str = """Ice-off date"""
ax.set_title(title_str, fontsize=18, fontweight='bold', ha='center')


#========================
# iceon plot
#========================
ax = ax2
ax.plot(df.utils.season_start(),
        df.utils.iceon_doy_wrapped(), color='k', linewidth=2)

# Range ov axes
ax.set_ylim([320, 440])
ax.set_xlim([1855, 2022])

yticks_start_end_step = (320+7, 440, 14)
xticks_start_end_step = (1860, 2021, 40)

# Turn off the display of all ticks.
ax.tick_params(which='both',  # Options for both major and minor ticks
               top='off',  # turn off top ticks
               left='off',  # turn off left ticks
               right='off',  # turn off right ticks
               bottom='off')  # turn off bottom ticks

# Remove x tick marks
plt.setp(ax.get_xticklabels(), rotation=0)

# Hide the right and top spines
ax.spines['right'].set_visible(False)
ax.spines['left'].set_visible(False)
ax.spines['top'].set_visible(False)
ax.spines['bottom'].set_visible(False)

# major/minor tick lines
ax.grid(axis='y', which='major', color=[0.8, 0.8, 0.8], linestyle='-')

# Turn off the display of all ticks.
ax.tick_params(axis='both',
               which='major',  # Options for both major and minor ticks
               top='off',      # turn off top ticks
               left='off',     # turn off left ticks
               right='off',    # turn off right ticks
               bottom='off',   # turn off bottom ticks
               length=0,
               )

# Only show ticks on the left and bottom spines
ax.yaxis.set_ticks_position('left')
ax.xaxis.set_ticks_position('bottom')

# Don't allow the axis to be on top of your data
ax.set_axisbelow(True)

ax.set_xticks(np.arange(*xticks_start_end_step))
ax.set_yticks(np.arange(*yticks_start_end_step))

ax.set_yticklabels([datetime.fromordinal(doy).strftime('%b-%d')
                    for doy in range(*yticks_start_end_step)])

# Labels
ax.tick_params(axis='x', labelsize=16)
ax.tick_params(axis='y', labelsize=16, labelleft=False, labelright=True)

# Annotations
title_str = """Ice-on date"""
ax.set_title(title_str, fontsize=18, fontweight='bold', ha='center')

# save figure
fig.savefig("./lake-mendota.png", bbox_inches='tight', dpi=300)

Challenges making the plot

The trickiest part about making this figure was displaying the date as tick marks. I am sure there is a more elegant solution, but my approach was to format ordinal dates:

yticks_start_end_step = (340+7, 500, 14)
yticks = range(*yticks_start_end_step)

list_of_dates = [datetime.fromordinal(doy).strftime('%b-%d') for doy in yticks]

ax.set_yticklabels(list_of_dates)

datetime(year, month, day).timetuple().tm_yday

The datetime.fromordinal() function uses the proleptic Gregorian ordinal, where January 1 of year 1 has ordinal 1. I then format these as a date. Although I may be off by a day in leap years, I think this approach should suffice since it is for visualization purposes only.

Conclusions

Creating the mendotapy package was a fun and educational experience, allowing me to deepen my understanding of the datetimepackage and enhance my plotting skills. This post only scratches the surface of what can be done with this rich dataset. Now that mendotapy makes it easy to retrieve and work with the data, I encourage you to dive deeper, use it as a toy dataset for statistical analyses, and share your insights. Contributions and feedback are always welcome on GitHub, and I look forward to seeing how this data sparks your curiosity and innovation.

Analyzing Lake Mendota Ice Phenology with Python was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Quickly send someone a small file download link with transfer

File sharing typically involves signing into a storage provider, uploading the file, and then remembering how to share it with someone else.

Although not a tedious process, this can be accomplished with a lot less brain power thanks to tools such as transfer.sh.

With a single terminal command, you can generate a temporary download link you can share with anybody. The link becomes inactive after two weeks.

This free service lets you upload unlimited data for short-term file sharing.

You can use this service right now via Client URL (cURL). The code below will generate a shareable download link.

curl --upload-file ./hello.txt https://transfer.sh/hello.txt

Transfer Function: Oh-My-Zsh

If you use Oh-My-Zsh, simply add the transfer plugin to your .zshrc.

plugins=(... transfer)

Now transferring a file or directory is as simple as:

transfer filename
transfer directory

Transfer Function: Other shells

You do not need Oh-My-Zsh in order to have a simple transfer.sh function.

The function below is an example from the “Sample use cases” on the transfer.sh webpage.

I have found this service handy to quickly transfer a file to colleagues or to a different computer I am working on — Sure, I could use scp, but sometimes I find a transfer command faster to type.

Be mindful of what you are transferring; I would not suggest using this service to send any sensitive information. However, you can encrypt your content first using GPG or keybase.io.

Emerging technologies, such as the InterPlanetary FileSystem (IPFS), can contribute to an ecosystem that is more verifiable and open. Since IPFS relies on content identifiers (CIDs) which are a hash of the content, you can be confident that the returned data is correct. In addition, IPFS is an open and public network. So anybody can access content on the network if they have the correct CID.

One project I am particularly excited about is web3.storage, which is a free service that reduces the friction of using decentralized storage.

In this post, I will …

• Provide an introduction to web3.storage and how to set it up
• Provide examples of basic CRUD operations with IPFS in Python
• Describe my current solution to create a decentralized data repository

Introduction to web3.storage

Our aim today is to provide a user-friendly experience that massively reduces the burden for onboarding new use cases into the web3 ecosystem today — while providing an upgrade path for the future. — Web3.Storage

Web3.Storage allows users and developers to use decentralized storage provided by IPFS and the Filecoin network. Anything uploaded is duplicated across geographically distributed storage providers, ensuring the resiliency of the network. The service also handles the work of pinning your content across multiple servers.

Be mindful of what you upload, since anybody can access the content on the network. However, you can encrypt content before uploading. My rule of thumb is, only upload content you are comfortable with being permanently public.

The permanent part is important. Unlike location-based addressing (e.g. URLs), content-based addressing (e.g. CIDs) makes it challenging to remove data from the network once it has been uploaded since it can be pinned on multiple servers.

Your quota is initially capped at 1TiB but can be increased for free by submitting a request. The overhead cost is currently subsidized by Protocol Labs, this will likely switch to some form of crypto-native payment model in the near future (e.g. staking Filecoin to increase storage limits).

The backbone that holds everything together is IPFS, a hypermedia protocol designed to make the web more resilient by addressing data by its content instead of by its location. To do this, IPFS uses CIDs instead of URLs — which point to the server the data is hosted on.

There is a lot more to web3.storage and I encourage you to explore the docs if you are also excited by this project — especially if you are a developer.

Setting up web3.storage

Go to https://web3.storage to create an account

See the documentation for detailed instructions.

Create API Token

An API token is necessary to use web3.storage from the command line

1. Log into your web3.storage account
2. Click on Account at the top and then Create API token
3. Enter a descriptive name for your token and click Create
4. You can click Copy to copy your new API token to your clipboard.

Do not share your API key with anybody, it is specific to your account. You should also make a note of the Token field somewhere and securely store it.

Install the w3 Command-Line Interface

The w3 command-line interface (CLI) is node-based tool to use web3.storage from the terminal

On a Mac, you can easily install node via homebrew, this is also install the node package manager (npm).

brew install node

Use npm to install the w3 command-line interface.

npm install -g @web3-storage/w3

Run the following command to connect w3 to web3.storage.

w3 token

You will be prompted for the API token you previously created.

The following displays information about each of the available commands.

w3 --help

Upload and Download Commands

• w3 put /path/to/file(this is how we upload content to web3.storage)
• w3 get CID(this is how we download content from a specific CID)

List your files on web3.storage

w3 list

Example using put

First, create a text file with a message.

echo "Hello web3.storage" > hello.txt

Let’s now use the put command to push the file to IPFS.

w3 put hello.txt --name hello

hello will be the name that appears in web3.storage, use w3 list to verify

The CID and a public gateway link are output.

If you followed the steps above exactly, then your CID should be identical to mine. The CID is a hash that uniquely identifies the content.

Use the link below to view the message through a public gateway.

https://dweb.link/ipfs/bafybeifzoby6p5khannw2wmdn3sbi3626zxatkzjhvqvp26bg7uo6gxxc4

Note: the link will be different if your message is different

Retrieve Content with Python

In the future, web3.storage will hopefully be S3 compliant, meaning we can access data stored there similar to how we access data in S3 buckets.

For now, we can use HTTP requests to read the data into Python. However, libraries like pandas allow us to directly read CSV files from a gateway URL. Also, ipfsspec allows us to read zarr data stores from IPFS with xarray.

I will demonstrate reading each of these in the following sections

Reading JSON

Here is an example of reading a .json file stored on IPFS.

Reading CSV files

If you have a CSV file, you can read it directly into a pandas DataFrame.

Zarr and IPFS

The Zarr format is new storage format which makes large datasets easily accessible to distributed computing, making it an improvement over the commonly used NetCDF — a format for storing multidimensional data.

If you starting to switch to zarr format, I encourage you check out Pangeo’s “Guide to preparing cloud-optimized data”.

NetCDF files are still very common and xarray’s functions make it easy to convert these NetCDF files to zarr data stores.

import xarray as xr
ds = xr.open_dataset('/path/to/file.nc')
ds.to_zarr("./file.nc", consolidated=True)

consolidated=True creates a “hidden” dotfile along with the data store that makes reading from zarr faster with xarray. In a simple test, I found reading consolidated data stores to be 5 times faster than unconsolidated.

zarr (consolidated): 635 ms

zarr (unconsolidated): 3.19 s

If you would like to test out the code above, I uploaded NOAA Optimum Interpolation SST V2 dataset in consolidated and unconsoldiated zarr format to IPFS. This dataset provides weekly means of ocean sea surface temperature (SST) from 1990 to the present with a 1 degree spatial resolution.

The gateway URL for this data is shown below

https://dweb.link/ipfs/bafybeicocwmfbct57lt62klus2adkoq5rlpb6dfpjt7en66vo6s2lf3qmq

Upload Zarr data stores to IPFS

When uploading zarr files to IPFS you have to make sure to upload the “hidden” dotfiles. With w3 this entails adding the --hidden flag:

w3 put ./* --hidden --name filename.zarr

Read Zarr data stores from IPFS

In order to read zarr data stores from IPFS with xarray you will need the ipfsspec package (as well as xarray and zarr)

conda install -c conda-forge xarray zarr
pip install ipfsspec

ipfsspec ensures xarray can interpret the IPFS protocol.

Notice in the example below I am using the IPFS protocol instead of the gateway URL with HTTPS. However, behind the scenes the code is actually reading from a gateway.

Decentralized Data Repository

IPFS can be used as a data repository. However, you need to link the CID to something that is human-readable in order for this to be a viable solution. I am not aware of any best practices surrounding CID management. However, taking some cues from best practices for NFT data, my current approach is to store CIDs and their associated file name as a key:value pair in JSON and then use a NoSQL database, such as MongoDB, for queries.

After you upload the content to web3.storage you add a new record to the file that identifies what the dataset is and the CID of the content. This is the minimum amount of information needed.

Here is an example of CIDs stored as a JSON array.

The pymongo package makes it easy to work with MongoDB in Python.

• Connecting to MongoDB Atlas with Python-PyMongo
• Introduction to MongoDB and Python in Real Python
• pymongo documentation

Here is an example of inserting records into a collection and then querying the database for a specific CID.

Alternative approach

An alternative approach I have been considering, but have not implemented yet, is taking advantage of the w3 list output.

This displays the CID and name you supplied when you uploaded the content.

The idea is to write an awkscript to generate a JSON file from the output.

Some downsides or pitfalls to this include:

• Handling CIDs that point to directories instead of pure files
• Ignoring CIDs that are irrelevant to your database

The biggest hurdle I can see is dealing with directories. For that reason alone I am sticking with manually updating my database, especially since it is small and easy to manage — for now at least.

Final thoughts

IPFS could be a place for long-term storage with CIDs functioning like built-in version control.

The w3 CLI makes it easy to push data up to IPFS and any common format (JSON, CSV, Zarr) can be read from IPFS into Python.

However, one the challenges with using IPFS is CID management. My current solution to create a decentralized data repository is to store CIDs and their associated file names as key-value pairs in MongoDB. The solution is a little clunky and I am open to suggestions on how to improve this setup.

Using IPFS for web hosting

Another great application of IPFS is hosting a website. My personal webpage is hosted on IPFS. I think I came up with an elegant solution to manage a decentralized website, you can read about my solution in the article below.

Modern Way to Host a Site on Ipfs

Storj: another decentralized storage option

Finally, IPFS is not your only option. If you want an S3 compatible object storage, then I encourage you to checkout Storj. The Storj solution relies on storing shards of encrypted data across a decentralized network as opposed to duplicating your data. This is a paid service but is very affordable, especially if you are currently using AWS. I personally use Storj for my day-to-day work. Check out the article below if you are interested in learning how I use Storj.

Storj Decentralized Cloud Storage: My New Favorite Cloud Object Storage

Store Unlimited Data for Free on IPFS and Retrieve it with Python was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

This article is an updated version of my “Guide to Hosting Websites on IPFS.”

A Guide to Hosting Websites on IPFS

In the time since that article was published, I have streamlined my approach and distilled the process of updating my website down to three commands:

1. git push
2. gh run view -v
3. w3 name publish <KEY_ID> <CID>

The magic of this setup comes from:

1. Using a GitHub action to push the site to IPFS via web3.storage
2. Using IPNS to avoid manually changing the DNS link.

There are a lot of steps to set up this workflow, but it is worth the effort to be able to update your decentralized site with only a few terminal commands.

Setup web3.storage

To make pushing content to IPFS easier, we are going to use web3.storage, which was created by the same team behind IPFS and Filecoin.

Web3.storage allows you to upload an unlimited amount of data to IPFS and it takes care of pinning the data to multiple servers for you, which is awesome!

Go to https://web3.storage to create an account:

Create web3.storage API Token

In order to push content to web3.storage from the command line and from GitHub Actions, we are going to need an API token.

1. Log into your web3.storage account
2. Click on Account at the top and then Create API token
3. Enter a descriptive name for your token and click Create
4. You can click Copy to copy your new API token to your clipboard.

Do not share your API key with anybody, it is specific to your account. You should also make a note of the Token field somewhere and securely store it.

Install the w3 Command-Line Interface

The w3 command-line interface (CLI) allows us to interact with web3.storage from the terminal. This is a node-based CLI, so you will first have to install node on your machine

If you are using a Mac, you can easily install node via homebrew

brew install node

This will include the node package manager (npm), which we will use to install the w3 CLI.

npm install -g @web3-storage/w3

Now, let’s test to make sure w3 installed with the following command:

w3 --help

This will display all the commands available.

Now that we know the CLI is installed, run the token command to connect the CLI to web3.storage.

w3 token

You will then be prompted to paste your API token.

This is the web3.storage API token created in the previous section.

⁂ API token saved signifies the CLI was successfully linked

Put Files on web3.storage

If you look back at the command available in w3 you will see one called put.

As you probably guessed, this is how we “put” content on IPFS.

w3 put <path>

As a simple example, create a text file with a message as follows:

echo "Hello web3.storage" > hello.txt

Let’s now use the put command to push the file to IPFS:

w3 put hello.txt --name hello

--name hello tells the command to name the file hello in web3.storage.

The content identifier (CID), as well as a public gateway link, are output.

https://dweb.link/ipfs/bafybeifzoby6p5khannw2wmdn3sbi3626zxatkzjhvqvp26bg7uo6gxxc4

If you followed the steps above exactly, then your CID should be identical to mine. Remember, the CID is a hash that uniquely identifies the content.

We can upload folders in the exact same way. Simply provide the put command with the path to the directory you want to upload.

Later in this post, we will use a GitHub action to automate this put step.

Create an IPNS address

CIDs are immutable and can not be changed. The InterPlanetary Name System (IPNS) solves this issue with mutable names that are a hash of a public key. You can think of an IPNS address as a pointer to a CID.

You can create an IPNS address using w3 using the following command:

w3 name create

Your address will be displayed, but you can always use w3 name list to show the name. However, I would still write down the address and store it somewhere.

To link a CID to the IPNS address we use the publish command, as shown below:

w3 name publish <key_id> <cid>

<key_id> is your IPNS address (what create name list displays) and <cid> is what we are linking to the address

For now, don’t worry about linking any CID to your IPNS address. We will do that later.

Setup GitHub Repo / GitHub Actions

Create a GitHub repo to store your website code. If this is a person's webpage, I suggest naming the repo <account_name>.github.io, which will allow you to use GitHub Pages too.

With your repo setup, let’s add an action that pushes the repo to web3.storage each time we use git push

Place the following code snippet in .github/workflows/web3_storage.yaml in your repo:

The line ${{ secrets.WEB3_STORAGE_TOKEN }} reads your API token, this is how the action knows to push to your account.

Store API token in GitHub with the following steps:

1. Log into your GitHub account
2. Go to Settings
3. Then Secrets
4. Click on Actions and then New Repository Secret
5. Your token must be namedWEB3_STORAGE_TOKEN
6. Copy the web3.storage API token we created earlier into the field

Now each time you run git push the action will be triggered and your repo will be uploaded to IPFS. If you try this, you should have a green check ✅ signifying there were no errors.

Click on the check to view the details of the action. This will display the CID and gateway URL (that is what the last two lines in the action are doing).

Using the GitHub CLI

You can use the GitHub CLI to view the CID in the terminal. This requires installing another command-line tool and linking it to your GitHub account.

The GitHub CLI is not integral to the setup but makes it so you can view the output from your actions in the terminal.

On Mac, we can install the gh CLI with homebrew, as follows:

brew install gh

See the documentation for installing on a different OS.

Go to https://github.com/settings/tokens to create an access token. You can find details steps in the documentation. Note that this token is different from the web3.storage token we created earlier. This one is so the GitHub CLI can communicate with your GitHub account.

Once you have an access token, run the following command:

gh auth login

You will be prompted to enter your GitHub Authentication token

We can use the gh command to view information about the actions we run:

gh run view -v

You have to run this command from a directory with a git remote setup, or you will receive the error: “failed to determine base repo: no git remotes found”

The -v flag here enables verbose output. Otherwise, you will be greeted with a message saying “For more information about the job, try: gh run view …”

Select the workflow you want to view and the output will open in a pagination tool.

This displays the output from the GitHub action, specifically the CID.

You can open the repo with gh repo view username/repo --web if you prefer to copy the CID from the webpage.

Now that we know our CID, let’s link to the IPNS address we created earlier.

Link CID to IPNS

To link the CID to the IPNS address, use the following command:

w3 name publish <key_id> <cid>

Note that <key_id> is the IPNS address and <cid> is the CID from the last section

The benefit of this is the IPNS address is mutable. What this means is after we update the website we can have the address point to the updated CID.

Finally, let’s set up a DNSLink with our IPNS address.

Setup DNS link

You have to have a custom domain purchased through a DNS provider. I used Cloudflare, but any provider should work.

The idea here is to have our custom domain retrieve the content from a gateway URL, but we have to tell it the IPNS address to retrieve.

DNS linking instructions using Cloudflare

1. Log into Cloudflare
2. Click on your active website and then on DNS in the left-hand column.
3. In the DNS management section, add a TXT and CNAME record.
4. Set the TXT record name to _dnslink.yourwebsite.com (note: make sure to replace yourwebsite.com with your own domain)
5. Set the TXT record content to dnslink=/ipns/<key_id> (note: <key_id> is your IPNS address
6. Set the CNAME record to any IPFS gateway (e.g. gateway.ipfs.io )

As an example, here is a screenshot of my TXT and CNAME records:

DNS linking only needs to be done once with a changeable IPNS address.

What Is Happening Here

What we just did is linked our domain name (which is hosted on a central server) to a gateway server. That gateway server (which is also a centralized server) fetches our requested content from the IPFS network (which is decentralized).

In this way, your site is pseudo-hosted on IPFS. The gateway server is centralized and your website will be inactive if that server goes down.

However, the content of your website is still on the IPFS network. This gateway server approach is one of the ways we can have our decentralized website communicate with an IPFS-ignorant browser. Gateways are a stop-gap solution to this problem.

Final Thoughts

I think I have distilled the process down as much as I can. This approach removes the need to update CIDs in the DNS link and everything can be accomplished from the command line with only three commands

The three commands to update the website

1. git push : this updates repo and pushes the website to IPFS
2. gh run view -h : this is to view our CID
3. w3 name publish <key_id> <cid> : this updates the IPNS link

The approach takes some time to set up, but I think it is worth the effort.

I am open to suggestions if anybody has ways to improve this workflow.

Acronyms

• InterPlanetary FileSystem (IPFS)
• InterPlanetary Naming System(IPNS)
• Content IDentifier (CID)
• Domain Name System (DNS)
• Command-Line Interface (CLI)

Modern Way to Host a Site on IPFS was originally published in Better Programming on Medium, where people are continuing the conversation by highlighting and responding to this story.

TL;DR: you need to include the suppelementary material’s output.aux file in the project, then you can reference figures as if they are in the main file

Overleaf makes working collaboratively on a scientific manuscript a breeze and popular journals, such as AGU, even have an official template.

If you have used overleaf to write a manuscript, then you have probably struggled with referencing your supplementary material in your main text. It is non-trivial how to do this in overleaf, which is why I am outlining the steps here.

Steps to reference supplementary material

1. After you compile your supplementary material, click on the logs and output files between recompile and download PDF

2. After that, click on other logs and files in the bottom right

3. This will bring up a box with different files. Click on output.aux and save this file somewhere on your computer

4. Rename output.aux to the same name as your supplementary file. For example, if your supplementary material is in supplementary.tex then name the aux file supplementary.aux

5. Copy this aux file into your overleaf project

6. Now you can start referencing supplementary figures in the main file.

Keep in mind that if you change the supplementary file, such as adding a new figure, then you will have to replace the aux file.

How to start analyzing/visualizing a 150-year ice record

In this quick pos I will present mendotapy, a Python package to read analysis-ready Lake Mendota ice phenology data into memory. This ice phenology dataset is maintained by the Wisconsin State Climatology office.

Get the data

mendotapy source code is available on GitHub.

Install the package via pip

pip install mendotapy

Import the package and load the data as an analysis-ready Pandas dataFrame

import mendotapy
df = mendotapy.load()

Data Columns

• season: the winter season
• iceon_date: first date majority of the lake surface is estimated to be frozen
• iceoff_date: latest date of ice breakup
• duration: number of days the lake remained frozen during the season
• n_freezes: number of freeze/breakup cycles during the season
• n_iceon_dates: list of ice-on dates during the season
• n_iceff_dates: list of ice-off dates during the season
• iceon_doy: day-of-year the lake first froze
• iceoff_doy: day-of-year the lake last broke up during the season

Final Thoughts

This is a great dataset See the r/dataisbeautiful DataViz battle from 2018 for creative visualizations. to create the title figure for this post simply run the following:

import mendotapy
mendotapy.plot()

Check out the following story for a discussion on why ice cover is decreasing.

Climate Change in Madison, WI

Thank you for reading and supporting Medium writers

Code snippets to help you analyze your data

Earth science data typically comes packaged in NetCDF files with labeled dimensions, making xarray the perfect analysis tool. Xarray has some powerful, yet versatile, built-in methods, such as resample(), groupby(), and concat(). This package is integral to geospatial analysis, which is why it is the backbone of the Pangeo software stack. Xarray is available under the open source Apache License.

This article is a collection of snippets geared towards Earth scientists.

Table of Contents

0. Installation
  0.1 Tutorial dataset
1. Climatology and Anomalies
2. Downsampling: Monthly Average
  2.1 Monthly Max, Min, Median, etc.
  2.2 N-month Average
3. Upsampling: Daily Interpolation
4. Weighted Average
5. Moving Average
6. Ensemble Average
7. Assign New Variables or Coordinate
  7.1 Assign new variable
  7.1 Changing time coordinate
  7.2 Changing longitude coordinate
8. Select a Specific Location
9. Fill in Missing Values
  9.1 Fill NaN with value
  9.2 Replace with climatology
  9.3 Interpolate between points
  9.4 Forward/backward filling
10. Filter Data
11. Mask Data
12. Final Thoughts
  12.1 Split-apply-combine

0. Installation

The developers recommend using the conda package manager and the community-maintained conda-forge channel for installation.

0.1 Tutorial dataset

Once installed, try loading this tutorial dataset, which is two years of air temperature sampled 4 times daily.

This dataset can be used to explore the recipes in this article

1. Climatology and Anomalies

• Climatology: A monthly climatology entails averaging all the Januaries in a time series, then all the Februaries, etc.
• Anomalies: These are deviations from climatology or the difference between the original time series and climatology. For example, if the time series is monthly, then the January 2013 anomaly is the January 2013 value minus the January climatology.

groupby()will collect all the like coordinate values, in this case, each group contains each month— regardless of the year. We then calculate the mean across each group and combines them back together

The argument supplied to groupby() specifes what we want to group. You can get the months via the .dt accessor, which is what we are doing in this example.

When calculating monthly anomalies, we first group by month and then subtract the climatology. For example, the January climatology is subtracted from each member in the January group, this is then repeated for every other month.

2. Downsampling: Monthly Average

• Downsampling: Decreasing the frequency of the samples. For example, going from a daily time series to monthly

To achieve this with xarray we use .resample(). The argument supplied specifies the temporal dimension (e.g. time) and resample frequency (e.g. monthly). In the example above, the sampling frequency string '1MS’ means sample monthly with the new time vector centered on the start of the month (2000-Jan.-01, 2000-Feb.-01, etc.).

The sampling frequency (e.g. ‘1MS’) specifies how to resample the data. See offset aliases in Pandas documentation for other options.

To calculate the monthly average you have to attach .mean(). Removing this method will return a DatasetResample object. Always specify what you want to do with the samples.

2.1 Monthly Max, Min, Median, etc.

There is a suite of other methods you can use with resample() including: max ,min ,median,std,var,quantile,sum

Check out the resample documentation for more details

2.2 N-month Average

Instead of averaging monthly, what if you wanted to average every 2 months? In this case, you would simply supply the number of months to sample by in the sampling frequency string:

• Monthly average: ‘1MS’ (the 1 is optional)
• 2-month average: ‘2MS’
• …
• 15-month average: ‘15MS’ (I don’t know why you would want to do this, but you can)

3. Upsampling: Daily Interpolation

• Upsampling: Increasing the sampling frequency. For example, going from a monthly time series to daily

Here is an example of upscaling data from a monthly resolution to daily using linear interpolation. In other words, converting a “low” temporal resolution (e.g. monthly) to a “high” resolution (e.g. daily). The argument supplied specifies the temporal dimension (e.g. time) and resample frequency (e.g. daily). In the example above the sampling frequency string '1D’ means sample daily.

We then attach .interpolate(“linear”) to linearly interpolate between the points.

4. Weighted Average

• Weighted average: Average of values which are scaled by their importance. Calculated as the sum of weights multiplied by the values divided by the sum of the weights.

When averaging geospatial spatial data, it is often important to weight the data so small regions do not skew the result. For instance, if your data is on a uniform latitude-longitude grid, then data near the poles occupy less area than the low latitudes. If you average temperature over the entire globe then you should weight the arctic less than the tropics since it occupies less area.

In this example, I am doing a common approach and weighting by the cosine of the latitude. This example is from the xarray webpage.

When calculating a weighted average with xarray note that ds.weighted(weights) is an instance of the “weighted class.” In addition to a weighted mean, we can calculate a weighted standard deviation, weighted sum, etc. See options here.

Another approach is weight by grid cell area, see the post below

The correct way to average the globe

5. Moving Average

• Moving average: Technique to smooth data by averaging within specific intervals. The interval is defined by a specific number of data points, termed the window length.

Rolling mean/running mean/moving average is a technique to smooth short-term fluctuations to enhance the signal-to-noise ratio. You use the .rolling() method and supply it the dimension to apply the mean over and the window length.

In the example above I am applying a 30 day running mean over the air-temperature tutorial dataset. Since data is sampled 4 times each day there are 120 samples every 30 days.

An optional argument you can supply to rolling() is center=True , this will set the labels at the center of the window instead of the beginning.

6. Ensemble Average

• Ensemble Average: Averaging multiple estimates of the same quantify. For example, the collection of CMIP models is an example of an ensemble.

If you have a dataset with multiple related variables you can concatenate them across a new dimension and then perform statistics.

I previously wrote about this topic earlier describing a different technique to achieve this with xarray. I now prefer the code above.

Pythonic Way to Perform Statistics Across Multiple Variables with Xarray

7. Assign New Variables or Coordinate

Xarray provides three “assign” methods:

• .assign() → Assign new data variables to a Dataset
• .assign_coords() → Assign new coordinates to a Dataset
• .assign_attrs() → Assign new attributes to a Dataset

7.1 Assign new variable

To add new variables to a dataset you can use .assign(). Keep in mind there is not an ‘inplace’ option with xarray, you will always have to explicitly assign the object to a variable.

In this example, I am changing the units from degrees Celsius to Kelvin and assigning it to a variable named temp_k. Units are very important, remember the Mars Climate Orbiter incident?

The second line is optional and simply adds attributes to the new variable.

7.1 Changing time coordinate

Here is an example of changing the time coordinate. This is useful when you want the time coordinate to start on the 15th of the month instead of the 1st.

7.2 Changing longitude coordinate

There are two conventions for longitude

• “360 convention”: 0 to 360, with 0 at the prime meridian and values increasing Eastward
• “180 convention”: -180W to 180E, centered on zero at the prime meridian

I made up the convention names. Please let me know if proper names exist

Change from 180 to 360

Change from 360 to 180

Alternatively, you could use list comprehension to make this more intuitive.

8. Select a Specific Location

Use sel() to select a specific latitude/longitude location. By default, sel() looks for exact matches in the dataset, but supplying method=’nearest’tells xarray to find the closest match to your selection.

9. Fill in Missing Values

Often times your data will have missing values you need to fill.

Unfortunately, there is not a “one size fits all” solution to this problem

Common approaches:

• Fill missing values with some value
• Fill missing values with climatology
• Interpolate between points
• Propagate values

9.1 Fill NaN with value

Using the fillna() method is a straightforward way to replace missing values some value. In the example above I am using 0 as the fill value, but this could be anything, the choice depends on the situation.

9.2 Replace with climatology

To replace with climatological values you first have to group the data via grouby() and then use fillna() supplied with the climatological values.

9.3 Interpolate between points

Another approach is to interpolate across missing values using interpolate_na(). Here I am linear interpolating across time.

See the interpolate_na() documentation for a complete list of interp methods.

9.4 Forward/backward filling

ds.ffill('time')
ds.bfill('time')

• Forward fill → ffill() propagates values forward
• Backward fill → bfill() propagates values backward

These methods fill NaN values with the first non-NaN value, with ffill() filling forward and bfill() filling backward along the supplied dimension.

10. Filter Data

Select data based on a conditional expression. Only the data where the expression evaluates to True will be retained, everything else will be NaN.

11. Mask Data

Mask data based on a conditional expression. Mask values will be True if the supplied condition is True and False otherwise. This can be turned into a mask of 1s and 0s simply by multiplying by 1.

12. Final Thoughts

I hope this post helped you with your work and illustrated the power of xarray. Keep in mind this post merely provides one person’s perspective. There may be more efficient or intuitive solutions to some of these problems. For that reason, I strongly encourage feedback and comments

A common task I did not include in this post is regridding data. I encourage you to check out the awesome xESMF package.

Finally, the examples in this post can be adapted for other purposes. For example, calculating climatology is really an application of a split-apply-combine strategy.

12.1 Split-apply-combine

1. Split the data into groups,
2. Apply a function to each group
3. Combine all the groups back together.

When calculating climatologies the “apply” step is mean. However, this can be swapped out for: max ,min ,median,std,var,quantile, or sum . A custom method can even be supplied by using map() .

Thank you for reading and supporting Medium writers

Xarray Recipes for Earth Scientists was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

Storj DCS (formerly Tardigrade) has matured to a point where it is easy to use and is compatible with S3 tools (such as s3fs), which makes migrating from AWS painless.

In this post, I will briefly outline how Storj Decentralized Cloud Storage (DCS) works, how to create buckets using the command line, and how to read from buckets into Python.

Table of Contents
1. Storj Decentralized Cloud Storage
  1.1 How it works
  1.2 Free tier 
2. Creating an account
3. The Uplink CLI
  3.1 Uplink commands
4. Read Data into Python
  4.1 Create an Access Grant
  4.2 Read a JSON file
  4.3 Read CSV file with Pandas
  4.4 Read Multidimensional Data with Xarray
5. Conclusions

1. Storj Decentralized Cloud Storage

Storj offers a decentralized alternative to cloud storage providers such as Amazon Web Services (AWS). Data uploaded to Storj is encrypted, divided into chunks, and dispersed over uncorrelated nodes around the world, i.e. storage providers with different geographic locations, internet service providers, power supplies, etc. The network nodes are independent and communicate via a peer-to-peer communication protocol.

Anybody can operate a node to help secure the network. Running a node means hosting encrypted partial segments of someone else’s data and getting compensated for it. Compensation comes in the form of the Storj token, an ERC-20 token on the Ethereum blockchain.

1.1 How it works

Files uploaded to Storj DCS are encrypted, split into pieces, and stored on storage devices all over the world. Each file is split into 80 or more pieces and distributed across the network. Since there is no central point of failure, file loss is statistically unlikely thanks to redundancy that’s built into the system. Even if pieces are lost due to storage failure or nodes leaving the network, the file can still be reconstructed since only 29 pieces are needed. This is because Storj uses Reed-Solomon encoding, the same encoding that powers QR codes, CDs, and DVDs. So far, the network has never lost a file. If you want to learn more, Cryptopedia has a nice overview of Storj, check out the webpage at storj.io/how-it-works, or check out the whitepaper.

1.2 Free tier

Storj DCS has one of the best free tiers I have seen — 150 GB of free storage. On top of that, the paid storage plan is $4 per TB per month and more cost-effective than AWS S3. Both integrate well with Python but, depending on your needs, Storj DCS has more value.

2. Creating an account

Begin by creating an account with Storj if you do not have one already

Storj | Fast, secure cloud storage at a fraction of the cost.

Upon creation, you will see a page that looks like this

I suggest clicking on Continue in CLI, which will help you install the command-line interface (CLI). You will first see your satellite and API keys. Make sure to save these as you only see them once. Then you will see command prompts to install the uplink CLI. The next few pages will walk you through setting up the uplink CLI to interact with the Storj network and it provides an example of how to make a bucket and upload data to it.

The CLI is not required to use Storj, but it definitely comes in handy. If you prefer, you can always use the web interface, which is clean and intuitive.

3. The Uplink CLI

If you already have an account and do not have the uplink CLI installed, sign in and then click Quick Start and Upload using CLI. This will walk you through setting up the CLI along with an example of how to use it.

3.1 Uplink commands

The Uplink CLI is simple to use with a short list of commands

The commands are intuitive. Here is a list of some basic operations.

Basic uplink commands

• uplink mb bucket_name — Makes a bucket
• uplink cp /file/path s3://bucket_name — copy to a bucket
• uplink cp s3://bucket_name /file/path — copy from a bucket
• uplink ls s3://bucket_name — list contents of a bucket
• uplink share --url s3://bucket_name/file.txt — generate a sharable URL of a file in a bucket

4. Read Data into Python

Once you have an account setup and some data stored on the network, let’s learn how to read it from the cloud into Python.

If you have used S3 buckets with Python in the past, then you have likely used the s3fs package. This same package can interact with Storj buckets.

conda install s3fs -c conda-forge

or

pip install s3fs

4.1 Create an Access Grant

In order for s3fs to interact with Storj you will need to create an access grant. To do this, log into Storj and click on Access in the left then Create Access Grant. Follow the prompted steps. When you get to the permissions step, be sure to click Continue in Browser and when you get to the last step, click Generate S3 Gateway Credentials at the bottom.

Your credentials will be prompted

Save these fields to file or keep this page open for a moment.

Open Python and copy these fields into their appropriate place in the code snippet below. This snippet is giving you access to your Storj DCS account from Python.

Best solution is to store the secret keys in a dot file so it is not exposed in a script.

Now try listing the contents of one of your buckets

s3.ls(bucket_name)

4.2 Read a JSON file

Here is one way to read a JSON file stored in a bucket

4.3 Read CSV file with Pandas

Pandas makes it easy to read a CSV directly from cloud storage.

Don’t forget the storage options (see section 4.1)

4.4 Read Multidimensional Data with Xarray

Zarr files are cloud-optimized storage of multidimensional data. Reading data from zarr looks and feels identical to reading traditional netCDF files.

Xarray makes it easy to convert NetCDF output to zarr, which are easier to read from object storage than NetCDF.

Once you have a zarr file, you can upload it to Stroj using the following:

Don’t forget the storage options (see section 4.1)

Reading a zarr file can be accomplished simply by

Don’t forget the storage options (see section 4.1)

It is possible to read a NetCDF file from cloud storage, but it feels a little awkward.

I do not recommend uploading and reading NetCDF files to cloud object storage. Zarr is a better solution.

5. Conclusions

Storj has matured over the years into a dead-simple object storage solution that offers a competitive free tier including 150 GB. Contrast this with the 5 GB AWS S3 offers. If you need more storage, Storj will only charge $4 per TB each month, while the AWS S3 price would be around $23.

In addition to the competitive pricing, Storj is working to reduce the environmental impact of the Storj token and make its network more sustainable. The network will become more sustainable as Ethereum fully migrates to proof-of-stake, which is expected in Q2 of 2022.

In addition, Storj cloud storage does not rely on building new data centers and did not cut down any trees or construct any new buildings to make its product. Instead, it relies on existing infrastructure.

Storj is not the only decentralized cloud storage provider. Filecoin, which is built on IPFS and developed by Protocol Labs, is a buzzier product in the space. Filecoin is blockchain-based and relies on duplicating or “pinning” files to different nodes to secure the network. Duplication makes the network more durable but at the expense of copying the file many times. This differs from Storj which achieves durability through erasure encoding. Having said that, Filecoin and IPFS have their place in the web3 stack and I encourage you to look into them. One Filecoin-based product I am intrigued by is web3.storage.

Finally, As much as I like the Storj DCS, I am aware it may not be the best solution for everybody. The pricing structure is simple and the storage cost is cheap, but if you are deeply embedded in the AWS ecosystem, then transitioning may not be the best solution.

Storj Decentralized Cloud Storage: My New Favorite Cloud Object Storage was originally published in Better Programming on Medium, where people are continuing the conversation by highlighting and responding to this story.

Have you ever wondered why many lakes turn unsightly green when it starts getting warm, especially those with heavy agricultural activity surrounding them?

Scientists set out to answer this question by conducting whole-lake experiments at the Experimental Lake Area (ELA); a collection of 58 formerly pristine lakes set aside for scientific research. Located in Ontario about 150 miles North of International Falls, Minnesota.

The Natural Aging Process

All lakes have a natural aging process. Material carried into the lake via tributaries or wind will gradually fill the lake as it settles to the bottom, slowly turning the lake into a wetland; this natural process is called “eutrophication”. However, naturally going from young and pristine to old and marshy can take hundreds to thousands of years. Human intervention can speed up the natural process and it took a whole-lake experiment to get at the dominant factor.

The Experiment at Lake 226

One of the most famous experiments at ELA began in 1973 when Lake 226 was physically partitioned with a vinyl-nylon sea curtain (see cover image). Over the course of eight years, the South end of the lake was fertilized with carbon and nitrogen while the North end was fertilized with carbon, nitrogen, and phosphorus.

The phosphorus side turned green due to a massive algal bloom, which demonstrated that phosphorus is a primary nutrient responsible for algal blooms. Excess phosphorus speeds up the natural aging process of a lake, termed “cultural eutrophication.”

Impact on Legislation

At a time when nitrogen cleanup was underway in many communities, this landmark experiment convinced the public and policymakers that phosphorus was the nutrient that needed to be controlled and it paved the way for major legislative change.

Programs were implemented to control phosphorus in sewage and remove phosphates from detergents, and a multi-million dollar phosphate control program was put in place in the Great Lakes. At the time, nitrogen cleanup was underway in many communities which this study showed was unnecessary. Communities were willing to change since phosphorous cleanup is cheaper than nitrogen.

Cultural Eutrophication

Cultural eutrophication is still a major problem in lakes with heavy agriculture activity surrounding them. For example, the western basin of Lake Erie is noticeably green in satellite images due to algal blooms. Lake Erie is commonly plagued with harmful algal blooms, which are forecasted annually by NOAA.

This is not only unsightly but in 2014 caused the Toledo Water Crisis, where residents were urged not to drink or use water from their tap because it was contaminated with highly toxic cyanobacteria.

The event prompted Ohio to pass laws prohibiting farmers from applying fertilizer when the ground is solid or waterlogged. However, the impact of this legislation on water quality is unclear.

Where do we go from here?

The overuse of fertilizer is one of the major contributors to cultural eutrophication. Solutions including cover crops, no-till systems, and buffers between crops and waterways have been proposed.

Farmers obviously don’t want to overuse fertilizer since it translates into higher costs. However, it is generally assumed that a certain amount will be lost so farmers typically apply a little extra to make sure that the net amount after loss is adequate for their crop.

Researchers from Imperial College in London sought out a solution to this issue. They have devised a sensor that measures ammonia in soil and then uses an algorithm to determine how much fertilizer to apply and makes a prediction how the amount of fertilizer will change within a two-week time frame. While it is a few years away from coming into mainstream use, this sensor could help farmers efficiently apply fertilizer without the need to overuse, reducing their costs and simultaneously helping mitigate eutrophication of nearby waters.

Conclusions

The landmark study at Lake 226 demonstrated why we need to control phosphorus but now we have to figure out how to control it. Removal from wastewater and detergents has solved the direct part of the problem, but we still have the indirect agricultural runoff problem to address. Thus, the development of more efficient fertilizer application techniques and being cognizant of how much we are using will help us attain our goals.

In addition to improved farming practices, lake management policies need to adapt. The typical mantra “dilution is the solution to pollution” does not necessarily hold water in large lakes. For instance, depending on the time of year, pollutants entering a lake via a river can get coastally trapped, which can be detrimental to coastal ecosystem services.

Why Lakes Turn Green was originally published in Climate Conscious on Medium, where people are continuing the conversation by highlighting and responding to this story.

In this post, I will describe how to host a website on the Inter-Planetary File System (IPFS), a peer-to-peer network for storing and accessing files, webpages, data, etc across a network of computers. Decentralized storage protocols, such as IPFS, will play an integral in the web3 technology stack, helping to make the internet decentralized and immutable. I believe this nascent protocol will be an important component of the internet in the coming years.

There are many uses for this protocol, but in this post, I will focus solely on hosting a website. Keep in mind that the IPFS protocol is currently not natively supported in all browsers yet, which can limit the reach of your content. For those IPFS-ignorant browsers, we can simply route our webpage through a “gateway.” This is what I will be showcasing in this post. As an example, here is my webpage hosted on IPFS routed through a gateway.

Table of contents

0. A brief overview of IPFS
  0.1 Content Identifiers (CID)
  0.2 Gateway Servers
  0.3 Where to go from here
1. Install IPFS CLI
  1.1 Brief tutorial of IPFS
2. Upload your website to IPFS
3. Connecting IPFS to your own domain name
  3.1 DNS link domain to IPFS
  3.2 What actually happens?
4. Conclusions

0. A brief overview of IPFS

One of the main differences between IPFS and the ubiquitous hypertext transfer protocol (HTTP) is the way data is referenced. HTTP references by location while IPFS references by content. When you enter a uniform resource locator (URL) into your browser (e.g. www.medium.com) you are specifying the location of the content you want to access.

However, maybe the person next to you at the coffee shop has a copy of the content you are asking for. So why not get it from them? This is possible with IPFS where multiple “peers” can host the same content allowing you to retrieve it from any of them instead of a central server. Files on IPFS may not be stored in just one place but in many places around the world, and someday maybe even on other planets (hence the name inter-planetary file system).

When asking for content on IPFS, we specify what we want with a content identifier (CID), regardless of where it is located. This CID is a cryptographically unique hash of the content. Thus, if the content changes then the CID will change. This is important to ensure the content was not tamped with.

IPFS makes the internet immutable and resilient. The CID is unique to the content and thus if the content is changed then a new CID is generated. The data is resilient since it is hosted on multiple computers, making it difficult to be completely removed from the network.

0.1 Content Identifiers (CID)

A CID is analogous to a URL. While a URL describes the location of a server, a CID describes the content itself, regardless of where it is located. Put simply, IPFS asks computers connected to you “do you know where the content associated with this CID is located?” To which a computer in the network will respond, “yes, you can find the content at this server.”

As an example, here is the CID to the IPFS Wikipedia mirror:

ipfs://QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1mXWo6uco

Just like a file system, we can navigate to the content we really want.

ipfs://bafybeiemxf5abjwjbikoz4mc3a3dla6ual3jsgpdr4cjr3oz3evfyavhwq/wiki/Beatles.html

The challenge with this system is that most browsers do not know how to resolve an IPFS address. Brave and Opera are two browsers with native IPFS support. For the IPFS-ignorant browsers, we have to either install the IPFS companion or rely on a “gateway”.

0.2 Gateway servers

A gateway is a middleman between your browser and IPFS. You make an HTTP request to a centralized gateway server that then fetches IPFS content.

Gateways allow anybody to access IPFS content (regardless of your browser), the downside is that the server is centralized and thus prone to being off-line. This is one problem with gateway servers.

Although a gateway is centralized and could permanently go offline, the content itself is still preserved on IPFS. The gateway may be gone, but the content itself still persists— assuming it is pinned to a node in the network.

As an example, here is the same Beatles page as before routed through the https://dweb.link gateway:

https://dweb.link/ipfs/QmXoypizjW3WknFiJnKLwHCnL72vedxjQkDDP1mXWo6uco/wiki/Beatles.html

0.3 Where to go from here

Now that we understand a little bit about IPFS and how it works, let’s install IPFS and host some of our own content!

1. Install IPFS CLI

On macOS, you can use brew to install IPFS

brew install ipfs

This will install the IPFS command-line interface (CLI).

Or you can download it directly from ipfs.io

Command-line | IPFS Docs

Ensure the CLI is installed by checking the version.

ipfs --version

1.1 Brief tutorial of IPFS

Before we can start using IPFS, we need to initialize the repository

ipfs init

This creates files in ~/.ipfs to store settings and internal data.

We can connect to the network by starting the daemon in a new session.

ipfs daemon

The daeman can be started in the background as well. ipfs daemon & , just don’t forget to stop it. On macOS, killall ipfs will stop the daemon.

Keep the daemon running so you can view content in your browser.

let’s now add some content to the network and save the CID to a variable.

CID=`echo "I <3 IPFS -$(whoami)" | ipfs add -q`

This will always generate the same CID, even if I change $(whoami) to my actual username. This is because IPFS is hashing the content. Therefore, if the content doesn’t change, then the hash (i.e. CID) wont change.

Use the following to view the CID of the content you just added

echo $CID

On my machine, the CID is: QmTUoo1hN87Vq8rjK9XbJgE6W2ZpHQLshTq31ZsrSLnGWT

You can view the content you just added with ipfs cat

ipfs cat QmTUoo1hN87Vq8rjK9XbJgE6W2ZpHQLshTq31ZsrSLnGWT

Try resolving the CID via a gateway (e.g. https://dweb.link/ipfs/CID). For example, here is how I could resolve my CID.

https://dweb.link/ipfs/QmTUoo1hN87Vq8rjK9XbJgE6W2ZpHQLshTq31ZsrSLnGWT

You should see your message displayed in the browser.

This is great! We can successfully upload content to IPFS and view it in a browser. Now let’s add a full webpage.

2. Upload your website to IPFS

You will need a website to host. Here, I am not going to do anything fancy and just use vanilla HTML, CSS, and JavaScript with the following structure.

.
├── index.html
├── css
│   └── styles.css
├── img
│   └── image.png
└── javascript
    └── scripts.js

Just like before, we can add the directory with ipfs add -r <site_dir>

ipfs add -r .

The -r flag adds recurively and the “.” indicates we are running the command from the webpage directory.

You will notice that each item in the directory tree has a CID. All we care about is the top-level CID pointing to the entire website.

Just like before, we can view the webpage via a gateway

ipfs.io/ipfs/QmdJcNS1M42rjBT6mnfEuE1oTqP5nGWt3msgFA9KH36a9e/

but you probably don’t want to be telling people you have a cool webpage at ipfs.io/ipfs/QmdJcNS1M42rjBT6mnfEuE1oTqP5nGWt3msgFA9KH36a9e/

One solution is to purchase a domain name and link to content on IPFS.

3. Connecting IPFS to your own domain name

This step requires a domain name that you own. I purchased my domain name through Cloudflare, but any provider should work.

We will use DNSLink to map the IPFS CID to our domain address.

3.1 DNS link domain to IPFS

You will need a DNS provider. I am using Cloudflare, but the basic instructions should hold for any DNS provider.

1. Log into Cloudflare
2. Click on your active website and then on DNS in the left-hand column.
3. DNS management section, add a TXT and CNAME record.
4. Set the TXT record name to _dnslink.yourwebsite.com (make sure to replace yourwebsite.com with your own domain)
5. Set the TXT record content to dnslink=/ipfs/CID (make sure replace CID with your own)
6. Set the CNAME record to any IPFS gateway (e.g. gateway.ipfs.io )

As an example, here is a screenshot of my TXT and CNAME records.

Ensure there isn’t a leading or trailing blank space when copying a CID

Once you enter all the information, click save and you will have successfully hosted your website on IPFS.

3.2 What actually happens?

Your browser performs a DNS lookup to get the IP address of the gateway server. The gateway notices the request is coming from your webpage and it will look for the DNS entry with the IPFS address to the content. The gateway will retrieve the content from IPFS and return it to your browser.

4. Conclusions

In this post, I provided one solution to host a webpage on IPFS and have it be viewable view anybody. Hosting a website on IPFS is easy, but does require a little bit of work (and money) to have a human-readable domain name.

The solution works, but it is cumbersome to keep changing the IPFS address in the TXT record each time you update the webpage. As a solution to this, I encourage you to check out the Inter-Planetary Name System (IPNS). I also suggest checking out the IPFS GitHub Action and Unstoppable Domains, if you want a completely decentralized website with a human-readable domain name (at the time of writing, browser support is limited with this approach.)

Another issue with the solution is accessibility. To ensure your content is always available it is best to “pin” your content to multiple nodes. I encourage you to check out different pinning services which make this easy. Piñata is one such service where you can pin a lightweight website using a free account.

Hosting a website is one of many use cases that exist for this emerging technology. An obvious application is a decentralized storage solution.

I for one am extremely excited by this emerging technology and look forward to seeing what is in store for the future. If you want to learn more I encourage you to check out the IPFS docs and browser the tutorials in this gitbook.

Want to Connect?

L. Gloege

A Guide to Hosting Websites on IPFS was originally published in Better Programming on Medium, where people are continuing the conversation by highlighting and responding to this story.