<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Andres Lamont on Medium]]></title>
        <description><![CDATA[Stories by Andres Lamont on Medium]]></description>
        <link>https://medium.com/@alamont?source=rss-8e694925bcf5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Ttp22_26wlndLLBj.jpg</url>
            <title>Stories by Andres Lamont on Medium</title>
            <link>https://medium.com/@alamont?source=rss-8e694925bcf5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 21:34:34 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@alamont/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Predicting water quality with Neural Nets and Boosted Decision Trees]]></title>
            <link>https://medium.com/orikami-blog/predicting-water-quality-with-neural-nets-and-boosted-decision-trees-fc4c2c3523ae?source=rss-8e694925bcf5------2</link>
            <guid isPermaLink="false">https://medium.com/p/fc4c2c3523ae</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[challenge]]></category>
            <dc:creator><![CDATA[Andres Lamont]]></dc:creator>
            <pubDate>Tue, 08 Mar 2016 13:37:12 GMT</pubDate>
            <atom:updated>2016-03-08T13:44:38.880Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OTXI0sHurJ8kXRtONK2cAA.jpeg" /></figure><p>Recently Vitens, a Dutch water company, organized an interesting data science challenge concerning the water quality in the province of Friesland.</p><p>Vitens’ question was how does the water quality correlate with other measurement such as pressure, flow and acidity among others.</p><h4>Exploratory Data Science</h4><p>Part of the challenge was trying to make sense of the data which were given to us. Unfortunately Vitens purposefully stripped all sensor data from any location data and we also had no knowledge of the structure of the water network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SQb2aev8vRxpiFSU9zDFOg.png" /><figcaption>Sensor locations in Leeuwarden. Green is water quality at some end points. Red are all other sensor such as water pressure, flow, salinity etc.</figcaption></figure><p>Fortunately, after some exploratory data analysis we managed to estimate the location of some of the sensors. It turns out in the end we completely ignored the locations of all sensors but it’s still was useful to get a better feeling for the data you are working with.</p><p>We checked the data of the water quality sensors for any missing values and found a lot of gaps in the data. Some sensors seem to have never worked and all sensor have missing data over a couple of months were Vitens failed to record the data. In the figure below show which part of the data for each month was missing (time is on the x-axis). Dark blue indicates 100% missing values. Light yellow is no missing data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/1*0I8JDpZOw48kYd9TM_PP6w.png" /><figcaption>The ratio of missing values for each water quality sensor. Time is on the x-axis (the labels are missing). You can clearly see that there some sensors have little to no data. Also there is a period where all data for all sensors is missing. Apparently Vitens forgot to pay for the internet connection to the sensors.</figcaption></figure><p>Instead of determining correlations between de sensors and water quality we chose to see if we could predict the water quality based on solely the other sensor such as water pressure, flow, salinity, pH (acidity) and temperature.</p><h4>Prediction of water quality</h4><p>The relation between the water quality and other measurements (on different locations) is inherently time dependent. While there are good machine learning techniques to handle these problems (such as <a href="https://en.wikipedia.org/wiki/Long_short-term_memory">LSTMs)</a> they are more difficult to train and greatly increases the complexity of the analysis. Instead we took a hourly aggregate of the sensor data, assuming that most time dependent behavior is resolved within the hour, making it possible to treat every hour independently.</p><p>We use two different machine learning methods to predict the water quality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/237/1*4eUw-sEVrVtfYvX8usbXaw.png" /><figcaption>A simple fully connected neural network wit 3 input neurons, 1 hidden layer with 4 neurons and 2 output neurons.</figcaption></figure><p>First we tried an artificial neural network (ANN), specifically a multi layer perceptron (MLP). Basically an ANN is a model which is inspired by the way neurons in your brain work. A neural network has a set of input ‘neurons’ and a set of ‘output’ neurons. In between there are hidden layers of neurons which act as a black box. During training of the model the weights between the neurons are set so that we get the desired output. A Multi layer perceptron is a type of neural network with multiple hidden layers where each neuron is connected with all neurons of neighboring layers (fully connected).</p><p>The following video demonstrates what happens in the neural network during training. We used a model with close to 300 input neurons (one for each sensor), 4 hidden layers and 3 output neurons corresponding to one set of water quality sensors.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FfYzFjn_pQBA%3Ffeature%3Doembed&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DfYzFjn_pQBA%26feature%3Dyoutu.be&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FfYzFjn_pQBA%2Fhqdefault.jpg&amp;key=d04bfffea46d4aeda930ec88cc64b87c&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/111044ab8ecfd33269cb8f87d471db24/href">https://medium.com/media/111044ab8ecfd33269cb8f87d471db24/href</a></iframe><p>Visualizing the training of a neural network in such a way gives us insight in which features or combination of features (input sensors) are predictive or relevant.</p><p>We trained a separate model for each set of water quality sensors. The following plot is show the true and estimated water quality of one of the sensors sets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/847/1*HxAdnzPItiZWvzG3LBAs2g.png" /><figcaption>The estimated water quality (red) closely follows the real water quality.</figcaption></figure><p>Another method we tried to predict the water quality is Gradient Boosted Regression with Decision Trees. That’s a mouthful but I’ll try to explain.</p><p><a href="https://en.wikipedia.org/wiki/Regression_analysis">Regression </a>means that we are trying to predict a continuous value as opposed to, for example, classifying the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hVz6wIB3GoLRLkN-kA5_rA.png" /><figcaption>A simple decision tree to decide if you should go play outside.</figcaption></figure><p>On the left you can see a simple <a href="https://en.wikipedia.org/wiki/Decision_tree">decision tree</a>. It works just like a flow chart. In this example the goal is to decide to go play outside or stay inside.</p><p>There are <a href="https://en.wikipedia.org/wiki/Decision_tree_learning">many algorithms</a> to learn decision trees. Our implementation uses an algorithm called <a href="http://en.wikipedia.org/wiki/Predictive_analytics#Classification_and_regression_trees">CART.</a> It tries to find features an threshold which yield the largest amount of information at each node.</p><p>While decision trees are very nice, the classification or regression performance usually leaves much to be desired because they can be unstable to small variation in data and are prone to overfitting by building large complex trees which don’t generalize.</p><p>Fortunately, training decision trees is very fast. This makes it possible to train a whole bunch of slightly different decision trees, each which is a weak learner, and combine them all to create a strong learner (even state-of-the-art) estimator. <a href="https://en.wikipedia.org/wiki/Boosting_(machine_learning)">Boosting</a> is one of the ways to combine all those learners. The more models you train, the better the result will be.</p><p>In this case this method proved to be very successful:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/1*3iKxhCZF5q6z1L0Dv2_aQg.png" /></figure><h4>Conclusions</h4><p>We have some pretty nice results in which we showed that we can estimate the water quality based on the other sensor data.</p><p>There are some problems with the prediction however. Because our knowledge of the water network was very limited our models are very naïve; they don’t know anything of the underlying structure of the data or network. This causes some overfitting of the data.</p><p>We did not do any feature selection or dimensionality reduction on the training data. There is a lot of correlation between the sensor so we would expect to be able te greatly reduce the number of features by clustering or using auto-encoders to reduce the number of dimensions.</p><p>Also, it would be very interesting to see if we can predict the water quality some hours in advance. This should be possible with the current setup. To take the prediction a step further we could go fully time dependent and implement an LSTM, which is a kind of recurrent neural network (network with cyclic connections).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fc4c2c3523ae" width="1" height="1" alt=""><hr><p><a href="https://medium.com/orikami-blog/predicting-water-quality-with-neural-nets-and-boosted-decision-trees-fc4c2c3523ae">Predicting water quality with Neural Nets and Boosted Decision Trees</a> was originally published in <a href="https://medium.com/orikami-blog">Orikami blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>