<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Olivier Lejeune on Medium]]></title>
        <description><![CDATA[Stories by Olivier Lejeune on Medium]]></description>
        <link>https://medium.com/@olivierlejeune?source=rss-a3b019b339dc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*QO6DjlkAsLUh4AKE.</url>
            <title>Stories by Olivier Lejeune on Medium</title>
            <link>https://medium.com/@olivierlejeune?source=rss-a3b019b339dc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 08:45:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@olivierlejeune/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Using machine learning to predict Rhine water levels]]></title>
            <link>https://medium.com/data-science/using-machine-learning-to-predict-rhine-water-levels-44afce697074?source=rss-a3b019b339dc------2</link>
            <guid isPermaLink="false">https://medium.com/p/44afce697074</guid>
            <dc:creator><![CDATA[Olivier Lejeune]]></dc:creator>
            <pubDate>Fri, 18 Sep 2020 14:36:23 GMT</pubDate>
            <atom:updated>2020-09-19T06:24:12.993Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>How an LSTM model vastly improved my forecast</strong></h4><p>Long Short Term Memory (LSTM) models are a powerful type of neural network ideally suited to predict time-dependent data. Rhine water levels fit right into this category: they vary over time, depending on a range of variables such as rain, temperatures and snow cover in the Alps.</p><p>The Rhine is Europe’s lifeblood. For centuries it has been used as a major artery for shipping goods into Germany, France, Switzerland and Central Europe. However, with climate change, water levels on the river are likely to become more variable. Forecasting the river’s level accurately is therefore a primary concern for a whole range of actors, from shipping companies to commodity traders and industrial conglomerates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IkA1LZfGJy8I9hrxrxh6JQ.jpeg" /><figcaption>A barge laden with coal navigating the Rhine (source: <a href="https://commons.wikimedia.org/wiki/File:Coal_barge_Chilandia_on_Rhine_-_looking_south.jpg">https://commons.wikimedia.org/wiki/File:Coal_barge_Chilandia_on_Rhine_-_looking_south.jpg</a>)</figcaption></figure><p>Unlike classical regression-based models, LSTMs are able to capture non-linear relationships between different variables; more precisely, the sequence dependence among these variables. This blog focuses on the problem of Rhine river forecasting using LSTMs, rather than the theory behind these models.</p><h4>Problem at hand</h4><p>The problem we are looking to solve here is the following: we would like to forecast next-day water levels at Kaub, a key chokepoint in western Germany, with the highest possible accuracy.</p><p>We have historical daily data from 2 January 2000 to 27 July 2020, equivalent to 7513 observations. The dataset includes 15 different categories, displayed as columns:</p><ul><li>‘date’: the date of the observation</li><li>‘Kaub’: the day-on-day difference in Kaub water level, in centimetres — this is the ‘y’ value we are trying to forecast (source: WSV)</li><li>‘Rheinfelden’: the absolute value of the water flow at Rheinfelden, in Switzerland, in cubic metres per second (source: BAFU)</li><li>‘Domat’: the absolute value of the water flow at Domat, near the source of the Rhine, in cubic metres per second (source: BAFU)</li><li>‘precip_middle’: the average daily amount of rain recorded at 20 weather stations along the Rhine, in millimetres (source: DWD)</li><li>‘avgtemp_middle’: the average temperature recorded at the same stations, in degrees Celsius</li><li>‘maxtemp_middle’: the maximum temperature recorded at the same stations</li><li>‘mintemp_middle’: the minimum temperature recorded at the same stations</li><li>‘precip_main’: the average daily amount of rain recorded at 8 weather stations along the Main, a major tributary of the Rhine, in millimetres (source: DWD)</li><li>‘avgtemp_main’: the average temperature recorded at the same stations, in degrees Celsius</li><li>‘maxtemp_main’: the maximum temperature recorded at the same stations</li><li>‘mintemp_main’: the minimum temperature recorded at the same stations</li><li>‘precip_neckar’: the average daily amount of rain recorded at 7 weather stations along the Neckar, also a major tributary of the Rhine, in millimetres (source: DWD)</li><li>‘avgtemp_neckar’: the average temperature recorded at the same stations, in degrees Celsius</li><li>‘maxtemp_neckar’: the maximum temperature recorded at the same stations</li><li>‘mintemp_neckar’: the minimum temperature recorded at the same stations</li></ul><p>Note that the choice of variables is entirely mine and is based on my experience dealing with Rhine analysis. Selecting the right inputs is one of the most important steps in time series analysis, whether you are using classical regression models or neural networks. If you select too few variables, the model may not capture the full complexity of the data (this is called underfitting). By contrast, if you choose too many inputs, the model is likely to overfit the training set. This is bad as well as it could mean the model struggles to generalise to a new dataset, which is essential for predictions.</p><p>First, let’s load all the libraries we will need for this exercice:</p><pre>import datetime<br>import matplotlib as mpl<br>import matplotlib.pyplot as plt<br>import numpy as np<br>import pandas as pd<br>import seaborn as sns<br>import tensorflow as tf<br>from sklearn.preprocessing import LabelEncoder<br>from sklearn.preprocessing import StandardScaler<br>from sklearn.metrics import mean_squared_error<br>import joblib</pre><p>Below is a sample of the first few lines of the dataset. We can load it easily with the Pandas library:</p><pre># first, we import data from excel using the read_excel function<br>df = pd.read_excel(&#39;RhineLSTM.xlsx&#39;)<br># then, we set the date of the observation as the index<br>df.set_index(&#39;date&#39;, inplace=True)<br>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D-cb13kMnV9bobEexgYwYg.png" /></figure><p>Once loaded, we can plot the dataset using the Matplotlib library:</p><pre># specify columns to plot<br>columns = [0, 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14]<br>i = 1<br>values = df.values</pre><pre># define figure object and size<br>plt.figure(figsize=(9,40))<br># plot each column with a for loop<br>for variable in columns:<br>     plt.subplot(len(columns), 1, i)<br>     plt.plot(values[:, variable])<br>     plt.title(df.columns[variable], y=0.5, loc=&#39;right&#39;)<br>     i += 1<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/1*aYgq2-zWLvjXnZKnMBWmaA.png" /></figure><p>It’s also generally a good idea to plot histograms of the variables:</p><pre># histograms of the variables<br>df.hist(figsize=(9,18))<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/553/1*HU4ShFadbm6l2GQMWnmQMA.png" /></figure><p>Using the Seaborn library, you can create a violin plot to understand the distribution of each variable:</p><pre># calculate dataset mean and standard deviation<br>mean = df.mean()<br>std = df.std()<br># normalise dataset with previously calculated values<br>df_std = (df - mean) / std<br># create violin plot<br>df_std = df_std.melt(var_name=&#39;Column&#39;, value_name=&#39;Normalised&#39;)<br>plt.figure(figsize=(12, 6))<br>ax = sns.violinplot(x=&#39;Column&#39;, y=&#39;Normalised&#39;, data=df_std)<br>_ = ax.set_xticklabels(df.keys(), rotation=90)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/1*Myo1rrkwK0N0voOIYcNsHg.png" /></figure><h4>Evaluating different models</h4><p>I have trained different types of models on the Rhine dataset to establish which one fits best:</p><ul><li>The baseline model, also known as persistence model, returns the current rate of change in Kaub water level as the prediction (essentially predicting “no change”). This is a reasonable baseline as Rhine water levels typically change over a number of days due to wider weather phenomena (eg, slow melt in the Alps progressively making its way downstream).</li><li>The simplest model you can train assumes a linear relationship between the input variables and the predicted output. Its main advantage over more complex models is that it is easy to interpret, however it performs only marginally better than the baseline network.</li><li>A dense network is more powerful, but cannot see how the input variables are changing over time. This shortcoming is addressed by multi-step dense and convolution neural networks, which take multiple time steps as input for each prediction.</li><li>The LSTM model comes out on top, with a lower mean absolute error rate on the validation and test sets.</li></ul><p>The Python code required to create this chart is too long for this blog, but you can access it <a href="https://www.tensorflow.org/tutorials/structured_data/time_series">here</a>, applied to a different dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/1*HERgemdj375OuV9avrUXtw.png" /></figure><h4>LSTM model for regression</h4><p>LSTM networks are a form of recurrent neural network that can learn long sequences of data. Instead of neurons, they are made of memory blocks connected to each other via layers. A memory block contains gates (input, forget, output) that manage its state and output, and enable it to be smarter than a typical neuron.</p><p>They have been used extensively in academic circles to forecast river height and have been proven to <a href="https://arxiv.org/abs/1907.08456">outperform</a> classical hydrological models in certain situations.</p><p>Let’s start all over again with the code from the beginning of this article to load the Rhine database:</p><pre>import datetime<br>import matplotlib as mpl<br>import matplotlib.pyplot as plt<br>import numpy as np<br>import pandas as pd<br>import seaborn as sns<br>import tensorflow as tf<br>from sklearn.preprocessing import LabelEncoder<br>from sklearn.preprocessing import StandardScaler<br>from sklearn.metrics import mean_squared_error<br>import joblib</pre><pre># first, we import data from excel using the read_excel function<br>df = pd.read_excel(&#39;RhineLSTM.xlsx&#39;, sheet_name=&#39;Detailed4_MAIN’)<br># then, we set the date of the observation as the index<br>df.set_index(&#39;date&#39;, inplace=True)</pre><p><strong><em>Data preparation</em></strong></p><p>To build a functioning LSTM network, the first (and most difficult) step is to prepare the data.</p><p>We will frame the problem as predicting today’s rate of change in Kaub water level (t) given the weather and Swiss upstream flows of today and the previous 6 days (backward_steps = 7).</p><p>The dataset is standardised using the StandardScaler() function in the Scikit-Learn library. For each column of the dataframe, each value in the column has the mean value subtracted, and then divided by the standard deviation of the whole column. This is a pretty ordinary step for most machine learning models and allows the whole network to learn faster (more on this below).</p><p>Then, the dataframe is passed through a transformation function. For each column, we create a copy of each of the previous 7 days’ values (15 * 7 = 120 columns). The shape of the resulting dataframe is 7506 rows x 120 columns.</p><p>A lot of the code was inspired by this fantastic <a href="https://machinelearningmastery.com/multivariate-time-series-forecasting-lstms-keras/">blog post</a>.</p><pre># load dataset<br>values = df.values<br># ensure all data is float<br>values = values.astype(&#39;float32&#39;)<br># normalise each feature variable using Scikit-Learn<br>scaler = StandardScaler()<br>scaled = scaler.fit_transform(values)<br># save scaler for later use<br>joblib.dump(scaler, &#39;scaler.gz&#39;)</pre><pre># specify the number of lagged steps and features<br>backward_steps = 7<br>n_features = df.shape[1]</pre><pre># convert series to supervised learning<br>def series_to_supervised(data, n_in=1, n_out=1, dropnan=True):<br>     n_vars = 1 if type(data) is list else data.shape[1]<br>     df = pd.DataFrame(data)<br>     cols, names = list(), list()<br>     # input sequence (t-n, ... t-1)<br>     for i in range(n_in, 0, -1):<br>          cols.append(df.shift(i))<br>          names += [(&#39;var%d(t-%d)&#39; % (j+1, i)) for j in range(n_vars)]<br>     # forecast sequence (t, t+1, ... t+n)<br>     for i in range(0, n_out):<br>          cols.append(df.shift(-i))<br>          if i == 0:<br>                names += [(&#39;var%d(t)&#39; % (j+1)) for j in range(n_vars)]<br>          else:<br>               names += [(&#39;var%d(t+%d)&#39; % (j+1, i)) for j in range(n_vars)]<br>      # put it all together<br>      agg = pd.concat(cols, axis=1)<br>      agg.columns = names<br>      # drop rows with NaN values<br>      if dropnan:<br>          agg.dropna(inplace=True)<br>      return agg</pre><pre># frame as supervised learning<br>reframed = series_to_supervised(scaled, backward_steps, 1)</pre><p><strong><em>Define training and test datasets</em></strong></p><p>We must split the prepared dataframe into training and test datasets to allow a fair evaluation of our results. The training dataset represents 80% of our values and we will use the remaining 20% for evaluation. As we are dealing with data ordered through time, it is a very bad idea to shuffle the dataset, so we keep it as is. Next, we reshape our training and test datasets into three dimensions for later use.</p><pre># split into train and test sets<br>values = reframed.values<br>threshold = int(0.8 * len(reframed))<br>train = values[:threshold, :]<br>test = values[threshold:, :]<br># split into input and outputs<br>n_obs = backward_steps * n_features<br>train_X, train_y = train[:, :n_obs], train[:, -n_features]<br>test_X, test_y = test[:, :n_obs], test[:, -n_features]<br>print(train_X.shape, len(train_X), train_y.shape)<br># reshape input to be 3D [samples, timesteps, features]<br>train_X = train_X.reshape((train_X.shape[0], backward_steps, n_features))<br>test_X = test_X.reshape((test_X.shape[0], backward_steps, n_features))<br>print(train_X.shape, train_y.shape, test_X.shape, test_y.shape)</pre><p><strong><em>Fit model</em></strong></p><p>Finally, we are able to fit our LSTM network. Thanks to the TensorFlow/Keras library, this only requires a few lines of code. I have chosen to fit 64 memory blocks in batch sizes of 72. I use the Adam optimisation algorithm, which is more efficient than the classical gradient descent procedure.</p><pre># design network<br>model = tf.keras.models.Sequential()<br>model.add(tf.keras.layers.LSTM(64, input_shape=(train_X.shape[1], train_X.shape[2])))<br>model.add(tf.keras.layers.Dense(1))<br>model.compile(loss=&#39;mae&#39;, optimizer=&#39;adam&#39;)<br># define early stopping parameter<br>callback = tf.keras.callbacks.EarlyStopping(monitor=&#39;loss&#39;, patience=3)<br># fit network<br>history = model.fit(train_X, train_y, epochs=25, callbacks=[callback], batch_size=72, validation_data=(test_X, test_y), verbose=2, shuffle=False)<br># plot history<br>plt.figure(figsize=(12, 6))<br>plt.plot(history.history[&#39;loss&#39;], label=&#39;train&#39;)<br>plt.plot(history.history[&#39;val_loss&#39;], label=&#39;test&#39;)<br>plt.ylabel(&#39;mean absolute error [Kaub, normalised]&#39;)<br>plt.legend()<br>plt.show()</pre><h4>Model results</h4><p>After the model is fit, we can launch the forecast and invert the scaling to obtain our final results. We can then calculate an error score for the model. Here, the model achieved a root mean squared error (RMSE) of 6.2 centimetres, which is good but can probably be improved on.</p><pre># make a prediction<br>yhat = model.predict(test_X)<br>test_X = test_X.reshape((test_X.shape[0], backward_steps*n_features))<br># invert scaling for forecast<br>inv_yhat = np.concatenate((yhat, test_X[:, -(n_features - 1):]), axis=1)<br>inv_yhat = scaler.inverse_transform(inv_yhat)<br>inv_yhat = inv_yhat[:,0]<br># invert scaling for actual<br>test_y = test_y.reshape((len(test_y), 1))<br>inv_y = np.concatenate((test_y, test_X[:, -(n_features - 1):]), axis=1)<br>inv_y = scaler.inverse_transform(inv_y)<br>inv_y = inv_y[:,0]<br># calculate RMSE<br>rmse = np.sqrt(mean_squared_error(inv_y, inv_yhat))<br>print(&#39;Test RMSE: %.3f&#39; % rmse)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*hNqkugfwQwT-68o-R2QK8Q.png" /></figure><p>I have also checked the model’s performance during extreme weather events, such as the extensive floods recorded in Europe in May-June 2016 which caused more than Eur1 billion of damage in Bavaria alone. The model was generally able to track the rise in water levels, however during two particular peaks (one in mid-April and the other in early June) it gave low figures.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*2wQ6j21wI7FwPWp4Tz3BXg.png" /></figure><p>The trained LSTM network also performed well during storm Axel, in May 2019, which caused a very rapid rise in water height of more than 1 metre on two consecutive days. Once again, however, it gave slightly lower estimates than actual figures when floods peaked.</p><p>In both cases, I suspect the low figures are due to one major Rhine tributary, the Moselle, which was left out of the analysis for lack of reliable meteorological data. The Moselle joins the Rhine at Koblenz, just north of Kaub. This is a possible area of improvement in future.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*cjYe7gmefpboOePCCnIrVQ.png" /></figure><h4>Model performance</h4><p>Finding the best hyperparameters is one of the most important (and time-consuming) tasks in machine learning. I ran many different versions of the network in order to find the best possible setup:</p><ul><li><em>Pre-processing:</em> For those familiar with Scikit-Learn, I found the StandardScaler() to be more suitable for the Rhine dataset than the MinMaxScaler(), which normalises all values to between 0 and 1. Normalising values with the MinMaxScaler() gave erratic performance. A quick look at the histograms plotted at the beginning of this blog post indicates that most of the Rhine input variables follow a Gaussian distribution, so this is logical.</li><li><em>Backward steps:</em> I found 7 days to be a reasonable time frame for weather analysis and given the time required for water to flow down the Alps. There is no meaningful improvement in the model’s performance by increasing the steps beyond this figure.</li><li><em>Neurons:</em> Small networks of 16 memory cells perform less well, but larger networks of 32 neurons and beyond are fairly similar. I picked 64 memory cells for my model.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/829/1*1lgQgoSR4sNfdG5b368-GQ.png" /></figure><ul><li><em>Epochs:</em> I have chosen to stop training once the algorithm has gone through the dataset 25 times. Beyond this threshold, performance begins to plateau and the model starts to overfit the training set.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*Bm9PbL7FjeSOpNKWGYO4_w.png" /></figure><p>I have deployed a version of this model online. Its performance can be monitored <a href="http://www.rhineforecast.com/">here</a>.</p><p>I’ll try to improve this model in future by adding/removing variables and tweaking hyperparameters further. If you have any thoughts, don’t hesitate to <a href="/service/mailto:%20oliv.lejeune@gmail.com">get in touch</a> with me.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*T5AiXqHi0zA0svO2yU1j9A.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=44afce697074" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/using-machine-learning-to-predict-rhine-water-levels-44afce697074">Using machine learning to predict Rhine water levels</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>