<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by PaddleSoft on Medium]]></title>
        <description><![CDATA[Stories by PaddleSoft on Medium]]></description>
        <link>https://medium.com/@paddlesoft?source=rss-ffe5d69df3f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*7JUhW1t68rmO-5N2QPY3Hw.png</url>
            <title>Stories by PaddleSoft on Medium</title>
            <link>https://medium.com/@paddlesoft?source=rss-ffe5d69df3f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 05:36:59 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@paddlesoft/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[FBLYZE: a full service Facebook scraping engine in an easy to use Docker container.]]></title>
            <link>https://medium.com/@paddlesoft/building-a-full-service-facebook-scraping-and-analysis-engine-e6b5e45e3ea3?source=rss-ffe5d69df3f------2</link>
            <guid isPermaLink="false">https://medium.com/p/e6b5e45e3ea3</guid>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[data-mining]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[PaddleSoft]]></dc:creator>
            <pubDate>Sun, 30 Apr 2017 21:20:05 GMT</pubDate>
            <atom:updated>2017-05-01T01:18:33.427Z</atom:updated>
            <content:encoded><![CDATA[<p>Social media analysis is becoming increasingly important for a vast variety of companies. As social media sources continue to grow, the need for tools which can easily scrap and analyze the social sphere becomes more and more apparent.</p><p>Every social media site is different and some are easier scrap data from than others. Twitter for instance, provides an easily accessible streaming API which connects quite nicely with many applications. Facebook, however, is a bit more difficult. Yet for our purposes extracting meaningful data from Facebook is crucial. There are hundreds of open groups and whitewater dedicated pages across Facebook with frequent discussion of paddling related events. These groups contain data that is necessary for our services, whether it is quickly summarizing what paddlers are discussing, generating flow information for streams without gages, or building river recommendation engines. We played around with a few different ideas including the scraper from <a href="https://github.com/minimaxir">minimaxir</a>, however, this did not fulfill our need entirely.</p><p>Enter <a href="http://github.com/isaacmg/fb_scraper">FBLYZE</a>. FBLYZE is an ongoing open-source project aimed at creating a continuous Facebook scraping and analysis engine. We want to make it really simple to scrape a variety of Facebook groups and pages on a regular basis (e.g. every ten minutes, hour, day…etc) and then save/process the scraped content. We are trying to make FBLYZE as robust as possible in order to accommodate a variety of scraping needs. So far we have a created a Docker container which you can pull with the command docker pull paddlesoft/fb_scraper This Docker file can then also be used in conjunction with Apache Airflow in order to continuously scrape data and save it to your desired location.</p><p>So far we have added several modes of file saving including saving posts directly to ElasticSearch, uploading CSV and JSON data to S3, and shipping posts directly to Kafka. We plan on adding more ‘connectors’ shortly including HDFS, MySQL, and Neo4j as well as improving our current file save formats.</p><h4><strong>How to get started</strong></h4><ol><li>Pull our Docker image docker pull paddlesoft/fb_scraper</li><li>Create a variables.list file and include the necessary environment variables. See here for <a href="https://github.com/isaacmg/fb_scraper/wiki/Docker-Image">configuration instructions</a>.</li><li>Now you can test by running docker run --env-file variables.list paddlesoft/fb_scraper</li></ol><p><strong>Using Apache Airflow to schedule jobs</strong></p><p>4. <a href="https://github.com/isaacmg/fb_scraper/blob/master/airflow/dags/fb_docker.py">Download our template</a> DAG or create your own from scratch.</p><p>5. Modify the DAG based on your requirements.</p><p>6. Then start up and trigger the DAG.</p><p><strong>Analyze your results</strong></p><p>We mainly leave this step up to you! However, in the future we will be publishing some articles on how we employ FBLYZE in our production pipeline. We do have a <a href="https://github.com/isaacmg/fb_scraper/blob/master/data/Examining%20data%20using%20Spark.ipynb">Spark tutorial</a> on analyzing the CSV data that you can check out as well!</p><p><a href="https://github.com/isaacmg/fb_scraper">isaacmg/fb_scraper</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e6b5e45e3ea3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Kafka with NodeJS, SocketIO and D3js to build a real time map.]]></title>
            <link>https://medium.com/hackernoon/using-kafka-with-nodejs-socketio-and-d3js-to-build-a-real-time-map-b6d3c3eae381?source=rss-ffe5d69df3f------2</link>
            <guid isPermaLink="false">https://medium.com/p/b6d3c3eae381</guid>
            <category><![CDATA[d3js]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[realtime]]></category>
            <category><![CDATA[javascript]]></category>
            <dc:creator><![CDATA[PaddleSoft]]></dc:creator>
            <pubDate>Tue, 21 Feb 2017 01:52:30 GMT</pubDate>
            <atom:updated>2017-05-07T22:46:51.115Z</atom:updated>
            <content:encoded><![CDATA[<p>By Isaac Godfried | founder/data scientist</p><p>Hi everyone and welcome back to our series. A few weeks ago we published a story on how we developed a Kafka “backbone” in order to get USGS flow information. This week we follow up by showcasing how you can use that Kafka producer in conjunction with NodeJS and SocketIO to push real-time updates to your clients. In our specific case our updates will not be anywhere near real time due to the USGS API; we nonetheless decided to implement it in a real time “ready” fashion in case the USGS speeds up their API (unlikely) or to possibly port it to other applications and since it’s just a good experience to have.</p><p>To get started we created a simple NodeJS app with Express. We created a index.js that contained the core of our code and an index.html file. In order to use Kafka with Node you need to install a Kafka JavaScript library using NPM. We recommend that you use kafka-node as it seemed to work fairly well for us. For this you can use the command: npm install kafka-node . Warning DO NOT simply do npm install node as this will install a very old Kafka package that has not been updated in 3+ years which will throw errors when it tries to connect to your topic.</p><p>Once we had the library set up we created a very simple function to send a SocketIO message. We then initialized a Kafka consumer with kafka-node library and wrote a consumer.on(message) function to call the SocketIO function. We also added some simple code to save the message as well for persistence purposes. If this seems confusing don’t worry the full code is below and you’ll see how simple it is.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3b336352eb29b2e6aabbb058dec38fd7/href">https://medium.com/media/3b336352eb29b2e6aabbb058dec38fd7/href</a></iframe><p>The next step for us was sending the messages to the client in the web browser. Luckily this is fairly trivial using SocketIO.</p><p>Now we wanted update our map using the JSON message from the USGS. For this we decided to use the very popular data visualization library D3js. Basically on our D3js map there are circles of rivers which display a different color based on whether the flow of the river is high or low (you can see <a href="http://rivermaps.herokuapp.com/">http://rivermaps.herokuapp.com</a> for an example). We wanted to take the information from the message being received from SocketIO and update our map in real time. To do this we created an update function to be called on the socket event. The update event would change the color of the circle (i.e. river) by selecting the relevant circle based on the latitude/longitude which corresponded to the JSON message . If this sounds confusing then don’t worry because like last time the code is actually much simpler.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4a364e1a80d0734a40b3c848ebff21ec/href">https://medium.com/media/4a364e1a80d0734a40b3c848ebff21ec/href</a></iframe><p>Pretty simple all things considered! So in summary for creating a program like this you will need a Kafka Producer (in whatever language suits you best), a Kafka consumer in NodeJS which will call SocketIO, and an update method for your graph which SocketIO will call upon receiving a message.</p><p>Some final thoughts:</p><p>It seems that the inclusion of SocketIO is almost redundant. I wonder if you could send the message just using kafka-node?</p><p>Although this works, for the user to derive any benefit (in our example) they would have to stay on the page for more than an hour without refreshing. It would be much more interesting to see a case with actual real time data updating every second. If only the USGS would get their API running faster!</p><p>It would also be nice to tie an example of real time ML and stream processing with Flink at some as well. Who knows maybe an article on real time Twitter sentiment analysis and geographic mapping is in the future.</p><p>Anyways, that is all for today! Hope that was helpful to anyone who came here looking for information on getting your own real time system up and running.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b6d3c3eae381" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hackernoon/using-kafka-with-nodejs-socketio-and-d3js-to-build-a-real-time-map-b6d3c3eae381">Using Kafka with NodeJS, SocketIO and D3js to build a real time map.</a> was originally published in <a href="https://medium.com/hackernoon">HackerNoon.com</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting hourly USGS flow information with Airflow and Kafka]]></title>
            <link>https://medium.com/@paddlesoft/getting-hourly-usgs-flow-information-with-airflow-and-kafka-d88baf2a0950?source=rss-ffe5d69df3f------2</link>
            <guid isPermaLink="false">https://medium.com/p/d88baf2a0950</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[maven]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[PaddleSoft]]></dc:creator>
            <pubDate>Tue, 31 Jan 2017 01:03:58 GMT</pubDate>
            <atom:updated>2017-05-08T18:50:44.241Z</atom:updated>
            <content:encoded><![CDATA[<p>By Isaac Godfried | founder/data scientist</p><p>Last year when we deployed our Kenduskeag flow prediction model we faced major challenges in updating our model with the newest data in a timely fashion. We subsequently realized that our entire data infrastructure had serious flaws and over the past six months we have been working on fixing them.</p><p>One of the major challenges we wanted to address was getting USGS flow data as soon as it becomes available and immediately shipping it to our models/databases for further transformation and analysis. Ideally we would fetch the data in real-time and combine it with real-time weather data to generate our predictions. To do this we would ideally construct some sort of sink for Kafka to hook up to the USGS API. However, the USGS API only updates their flow data on an hourly basis(always at hour:15) so this solution was/is not practical (hopefully a real-time API will be in their future). After researching various methods we came up with the following solution: we would schedule an hourly job to get the data from the USGS API and as soon as the data was captured it would be sent down the pipeline.</p><p>For the job scheduler we decided to use <a href="https://airflow.incubator.apache.org/">Apache Airflow</a>. Airflow is an open source job scheduler, designed by Airbnb and written in Python, that allows you to schedule many complex jobs on a recurring basis. Airflow is built on DAG objects or directed acyclical graphs. Our goal was to create a DAG which would then issue a bash command to run a fat JAR which contained our compiled USGS and Kafka code. Before creating our core JAR we decided to create a test JAR and run it through Airflow.</p><p>We created a simple Java program and used maven to create the JAR. The program would simply print “hello this jar is working.” We then attempted to schedule this jar to run on an hourly basis in Airflow.</p><p>Airflow was a little difficult to initially get up and running (note you need to put your DAGs in the airflow/dags folder), however we eventually got it working. We ended up creating a very simple DAG to run our JAR file.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3191042f15445a045e0d6ba60e7f20dc/href">https://medium.com/media/3191042f15445a045e0d6ba60e7f20dc/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EsTEG7Kt7tfgLERY6wlT6A.png" /><figcaption>Airflow seems to be working on our test JAR.</figcaption></figure><p>Airflow has a very nice UI with which you can view and manage your DAGs and also view their logs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lYVrV2KFXs5jqR4VMZeQkw.png" /><figcaption>Yeah it ran successfully!</figcaption></figure><p>Now that our job was running in Airflow it was time to move into the trickier stuff. We wanted to get the data from the USGS API and send it directly to a Kafka producer which would then publish it to relevant consumers. Here is an excerpt from our code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3e6746bbd390bc70202c4ff003e9923f/href">https://medium.com/media/3e6746bbd390bc70202c4ff003e9923f/href</a></iframe><p>Obviously in the full production version we would change the key to match that of the river instead of just having it say “a river” but for testing purposes this works. Additionally, for testing purposes we created a simple Kafka consumer in a windows command prompt.</p><p>Time to run this thing and see if it works</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JtQfbOL1ZeOiHSqgzVkSKA.png" /><figcaption>It works!</figcaption></figure><p>To create our final fat JAR we had to do some playing around with Maven. Specifically, we had to add the maven-assembly-plugin to our POM file and then mvn clean compile assembly:single however, after that it was finally up and running.</p><p>This is a good start to solving some of our data flow problems. In future articles we will show how we constructed the Consumer clients and how this factors into our overall data backbone at PaddleSoft. Additionally, in the future we hope to expand our Java code to make use of multi-threading to get the data from the USGS API even faster and load the river data directly from our database into Java instead of manually initializing an app.</p><p>Full code will be made available on GitHub shortly.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d88baf2a0950" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Forecasting the flow of the Kenduskeag]]></title>
            <link>https://medium.com/@paddlesoft/forecasting-the-flow-of-the-kenduskeag-a731c5d35aa6?source=rss-ffe5d69df3f------2</link>
            <guid isPermaLink="false">https://medium.com/p/a731c5d35aa6</guid>
            <category><![CDATA[maine]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[PaddleSoft]]></dc:creator>
            <pubDate>Tue, 05 Apr 2016 00:39:55 GMT</pubDate>
            <atom:updated>2017-05-07T22:46:09.207Z</atom:updated>
            <content:encoded><![CDATA[<p>By Isaac Godfried | founder/data scientist</p><p>Every year in mid-to late April hundreds of paddlers gather for the annual Kenduskeag stream canoe race. In May another group of paddlers gather for the Kenduskeag Slalom. In addition, a variety of canoers, kayakers, and other whitewater enthusiasts frequently paddle the Kenduskeag throughout the spring, summer, and fall. A common question of paddlers regardless of whether they will be racing downriver, navigating slalom gates, or are just looking for a fun day of whitewater is what will the flow be when I’m on it?</p><p>Frequently paddlers look at forecasts in the paper or on the Internet and try to make rough anecdotal predictions. It is common to hear things like “hopefully it will hold at 5ft we are supposed to have a hot stretch with no rain,” or “an inch of rain should make it runnable for the weekend.” These thoughts of course are not isolated to the Kenduskeag. Around the country paddlers religiously check weather forecasts and watch online USGS gages. Intuitively, on any free flowing river there should be a positive correlation between rainfall and CFS and a negative correlation between CFS and temperature.</p><p>But how can we estimate the flow empirically and create an algorithm that effectively predicts flow based on weather forecasts? At first look this might seem like an “easy” problem that any high school student who has taken Algebra could solve. You have a function Flow(rainfall,temp) = coefficient(rainfall)-coefficient(temp); however, in reality it is nowhere near this simple. The river is a complex system that changes with respect to time, rainfall, temperature, and its current height. However, rainfall and temperature also vary with respect to time. This quickly becomes a very complicated problem. There are a number of ways to attempt to solve this problem some requiring more specialized knowledge than others. In this article we look at finding a method to predict the Kenduskeag’s flows that can also be applied to any other free flowing river/stream with a USGS gage.</p><p>In a broader sense this article discusses how PaddleSoft makes it easier for paddlers and organizers to <strong>Plan </strong>their whitewater adventures/events. This fits in with our larger idea of making it easy for you to find (new) rivers and paddling partners, plan an outing with your new knowledge, get out and actually paddle, and finally share your newest adventure with all of your paddling friends.</p><h4>Finding a “generalizable” solution</h4><p>Now that we have gotten in our heavy handed marketing digression let’s get back to finding a solution.</p><ol><li>Hydrology: You can solve this problem with a specialized knowledge of the watershed and some mathematical modeling. With this method you would look at the bodies of water that feed the Kenduskeag or more specifically, how/where rain water enters the stream. However, I am not a hydrologist and the larger goal is to generalize our solution in order to be able to forecast flows of streams across the country. This method requires specific knowledge of the watershed. So this would require years of research on each stream’s watershed.</li><li>Data science: This method involves creating a giant time series dataset, feeding it to some machine learning algorithm, tuning the parameters, and hoping the resulting trained network will spitout accurate predictions (based on weather forecasts). This method is the one I chose to use, since I happen to specialize in data science, and you can easily generalize this to just about any stream in the country.ma</li></ol><h4>Constructing a dataset</h4><p>In order to construct a dataset we needed the hourly flow data of the Kenduskeag along with the hourly rainfall, and temperature data for a significant period of time. Unfortunately, NOAA’s historical weather was far from complete. As a result we built an engine back in June to record the flow, height, rainfall, and temperature into a SQL database. Our system checked, once per hour, the flow of the river, the temperature, and the precipitation that fell in that hour. Unfortunately, during the winter our database exceeded its row limit and temporarily stopped recording. This stop lasted all the way until early March when we finally got our database back up and running. Altogether we ended with about 3500 rows (or hours) of data. While not a huge training set we hoped that this would due for now.</p><h4>Basic Analysis</h4><p>Before throwing our data directly into our machine learning algorithms we wanted to get a sense of how hourly rainfall correlated to river height. In the graphs below we looked at significant rainfall events that occurred during the summer and their impact on the CFS and Height of the Kenduskeag.</p><p>In the graphs below we looked at the flow over a five to six day (158 hour) period in September. Prior to the rainfall, the stream was flowing at 10 CFS and 2.74ft on the gage. The rainfall began approximately around hour 12 and lasted until hour 34. Over this 22 hour period approximately 5.7 inches of rain fell with the peak rainfall happening at hour 24 this amount was .97in. The peak stream flow and height occurred around hour 53 with a peak flow of 3120cfs and a height of 9.18ft. In general we noticed a 20–25hr delay between the peak rainfall and the peak flow on the gage. You can also notice around a four hour delay between the largest rainfall amount and the peak change in height/CFS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PGiJ75CSdLd4Se_bcqoA5Q.png" /><figcaption>Graph of a rainfall event during the summer. Time (x axis) is measured in hours while flow is measured in cfs (left y-axis). Rainfall is measured in inches (right y-axis). Notice the ten to twenty hour gap between rainfall and increase in cfs.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*owyfL7-4tKz02NEpq-bQ4A.png" /><figcaption>Graph of the same rainfall event except this time with river height instead of cfs.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QEeGJMJ1-7afIlJ_Jd6qQQ.png" /><figcaption>Orange represents the change in height(ft) and blue represents the rainfall in inches. Notice the peaks are much closer in this one (4 hours apart).</figcaption></figure><h4>Modeling the Data</h4><p>Okay enough fooling around. Making pretty graphs of rainfall, CFS, and height vs. time was fun and all but we need actual numerical predictions. This is fundamentally a regression problem in that we want the function to predict the flow of the Kenduskeag. From the graphs it should be quite obvious that this is NOT a linear regression problem.</p><p>Unfortunately, we cannot just throw the data into a neural network or decision tree until we have a better understanding of the problem. In this problem we are looking at the relation of height (or cubic feet per second) at a specific point in time, h(t), with respect to the rainfall at a specific period in time r(t), and the temperature at a specific point in time e(t). So basically h(t)=h(t-1)+ r(t)+e(t). But wait! This actually is still not entirely correct: remember the delays we were talking about? In reality the height at time t probably will not be most directly affected by the rainfall at time t. In all probability the rainfall at time t will not affect the flow at time t at all. So really the function would look more like h(t)=h(t-1)+f(r(t-1),r(t-2),r(t-3),…r(t-d))+e(t). Where f is some random weighting function that will weight each rainfall at time t-x all the way to t-d accordingly. If you don’t understand this that’s okay ( in all honesty it took me awhile to figure it out too).</p><h4>Nonlinear regression techniques</h4><p>Moving on, we now have to decide what model to use. Now I personally have never been very much into regression trees for modeling time series data. I understand that sometimes they can produce good results but I have never liked them as much as Neural Networks (NN). As such I only tried one and it produced unbelievably wacky results such as predicting negative flows. (If someone is a big fan of regression trees feel free to illuminate me!). Anyways I decided to use a NN called a NARX to model the data. The reason its called a NARX is because no one wants to say it’s full name: <strong>N</strong>onlinear <strong>A</strong>uto<strong>r</strong>egressive with E<strong>x</strong>ogenous input network.</p><p><strong>Neural Network</strong>:</p><p>In MATLAB lingo this is the two input functions x(t) and the target y(t).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/659/1*E70shywg8XR-g06TUSd4og.png" /><figcaption>The structure of one our NARXs. It has two exogenous inputs (i.e. temp and rainfall), one nonlinear target, a max delay of eight hours, 10 neurons in the hidden layer.</figcaption></figure><p>We trained the network on the dataset collected online and then converted the open-loop network to a closed-loop in order to make predictions. We worked with variety of networks ranging in from ten to over thirty neurons.</p><h4>Testing/Results</h4><p>We tested our NARX(s or i? How would you pluralize NARX???) on Kenduskeag stream flows during the month of March. In doing this we took the recorded rainfall and temperature, stripped it of the CFS and height data, and fed it to our neural network. We then plotted the predicted flow or along with the actual flow (or height) vs. time. Initially the predictions were rather bad with the networks either greatly under estimating or over estimating the flow. Below are some graphs of our NARX(s) and their predictions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VpXaIibQxobhfR4HcdSAFg.png" /><figcaption>Some of our early (very poor) NARX projections when we were playing with various parameters. The actual is in light blue.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z2YsWeca-f5juglGWSf5oQ.png" /><figcaption>Some of our later NARX projections. Even though the orange series underestimates the flow from the rainfall event, I still in general thought it was good as it modeled the shape of graph. I found this rainfall event somewhat peculiar as only 1.18 inches of rain produced a change of over 1400 cfs at its peak.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*k0o2G0XRyJW2DHcjfyUO-A.png" /></figure><p>As you can see we slowly began to achieve better and better results with our NN performance. However, some questions still remain…</p><h3>Unanswered Questions/ Conclusion</h3><h4>But what about snow melt and its impact on flow???</h4><p>Short answer: We are currently looking into ways of predicting how snow melt will influence the Kenduskeag; however this is a lot more challenging for a variety of reasons.</p><p>Long answers: Obviously in the Spring months the melting of snow greatly influences the level of the Kenduskeag. For instance, in 2014 the late and rapid thaw sent the Kenduskeag soaring to over 10ft on the gage; because of this and the tides, the race course had to be shortened. Therefore, it is very useful to be able to predict such events ahead of time. However, unfortunately unlike rain it is very difficult to predict the influence of snow melt on the stream. One of the reasons this is so much more difficult is not having an accurate dataset on snow depth. Both NOAA and other weather sites have very spotty average snow depth records. Assuming you could, actually get a good average snow depth dataset, you would then have to estimate the rate of melt as a function of temperature and time. Finally you would have to correlate this back to a rise in CFS. Even this would not do it entirely because when it rains the rate of melt would also increase. Suffice to say this is a very complicated problem, but don’t worry as I said previously, we at PaddleSoft are still dedicated to solving it (although it might be awhile).</p><h4>How likely is the model to accurately predict the flow of the river?</h4><p>Well first remember that our model is based on the weather forecast. So if the weather forecast is off so will our model be. As we often say in data science, garbage in = garbage out. So if our inputs are garbage the model will produce garbage. Assuming the weather forecast is actually correct, I’m fairly confident in our model. Numerically, on the days tested in March our final model was off an average of .5 feet, however, this was when we predicted about two weeks into the future. The further out the prediction is, the less likely it is to be accurate. This is due to each subsequent prediction using the previous y(t) flow to predict y(t+1). Remember, also that most of the training dataset is of the summer months. In general in summer everything is a lot dryer, so it takes a lot more rain to make stuff happen. Therefore, I noticed that in some instances it was under-predicting rainfall events in March. As we collect more data during the Spring hopefully we can eliminate these under predictions for the wetter months.</p><p>Finally, I think the model is most likely to be incorrect if we get bizarre weather that we haven’t seen before. For instance, while testing performance I accidentally inverted the temperature and rainfall columns. This meant it was reading in the temp as rainfall so it was thinking we got around 40 inches of rain and it was .01 degrees outside. This caused it to initially shoot way up but then also come way down (into negative flows).</p><h4>So when/where can I see actual predictions of this model?</h4><p>We will release our real-time prediction engine on Wednesday April 6th (just in time for the Kenduskeag stream race!). I’m currently working on deploying our algorithm for production on our web server. Once up and running the tool will show a ten day projection of the flow of the Kenduskeag based on the current weather forecast. The projection will update once per hour with the latest flow and latest predictions. We are also investigating the possibility of creating a tool where you can define inputs so that you can simulate what would happen if, for instance were to get five inches of rain.</p><p>We will be posting a link on our Facebook page first and only posting links on other pages several hours later. So make sure to like our Facebook Page at <a href="https://www.facebook.com/paddlesoft">https://www.facebook.com/paddlesof</a>t !! Also, remember to follow us on Medium for more articles like this one!</p><p>:</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a731c5d35aa6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Recommendation Engine with Neo4j: Part 1 Simple graph based approaches]]></title>
            <link>https://medium.com/@paddlesoft/building-a-recommendation-engine-with-neo4j-part-1-simple-graph-based-approaches-16c3cea32ff?source=rss-ffe5d69df3f------2</link>
            <guid isPermaLink="false">https://medium.com/p/16c3cea32ff</guid>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[paddling]]></category>
            <category><![CDATA[whitewater]]></category>
            <dc:creator><![CDATA[PaddleSoft]]></dc:creator>
            <pubDate>Fri, 25 Mar 2016 22:52:05 GMT</pubDate>
            <atom:updated>2016-03-25T22:52:05.278Z</atom:updated>
            <content:encoded><![CDATA[<p>This is part one in our five part series on building a recommendation engine. In this issue we will discuss some simple out of the box techniques that utilize basic graph algorithms. This article will rely heavily on Cypher Query Language or CQL so if you are not familiar with it you can read up on it here: <a href="http://neo4j.com/docs/stable/preface.html">http://neo4j.com/docs/stable/preface.html </a>.</p><p>So we will now jump right into building our graph recommendation engine. For the purposes of this project I created a small test data set containing about 12 nodes (we’ll expand this as the project goes on). The nodes in green are rivers and the nodes in blue are people. People are connected by the friendship relationship (i.e. a directed edge between two members of the set of people) and people are connected to rivers based whether they have paddled the river or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xnh_inWqyrODcrMisPuojA.png" /><figcaption>Our current data set that contains both river nodes (green) and people nodes (light blue)</figcaption></figure><p>A simple way to find rivers that users might be interested in paddling (or rivers they have already paddled but have not saved on PaddleSoft) is by creating a query that finds all the user’s friends and then finds the rivers they have paddled. For instance let’s find rivers that Jeff may want to paddle:</p><p>*Please note, the formatting of Medium unfortunately, does not preserve format of the “” and ‘’ in queries. So if you try to copy them directly into Neo4j it will raise syntax errors.</p><p>MATCH (n {full_name:“Jeff”})-[:friend]-&gt;(m)-[:has_paddled]-&gt;(s) RETURN s</p><p>This will return:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/1*rI4Zj3tqPAQpnIbiIkG4nw.png" /><figcaption>Rivers that friends of Jeff have paddled</figcaption></figure><p>However, (as you probably noticed) this returns rivers that Jeff has already paddled like the Penobscot (or Penobs in the graphic). So we need a query that weeds these out. To do this we have to check to see if a relationship exists between the two using the WHERE not condition.</p><p>MATCH (n {full_name:“Jeff”})-[:friend]-&gt;(m)-[:has_paddled]-&gt;(s) WHERE not (n)-[:has_paddled]-(s) RETURN s</p><p>This will remove the Penobscot from the results and give us only rivers his friends have paddled but he has not.</p><p>Lets now move on to recommending friends based on similar rivers paddled. Since most people in the dataset were already “friends” I decided to add an additional person node (i.e. Dave) onto the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g6FjyFMPr2o3BN6Uy1XvXQ.png" /><figcaption>Dataset with Dave added. Dave has paddled both the Ottawa and the Charles.</figcaption></figure><p>Okay so both Natalia and Dave have several paddled rivers but they have no friends. So we will now recommend some potential paddling partners to them. Same logic as before:</p><p>MATCH (n {full_name:”Dave”})-[:has_paddled]-&gt;(r)-[:has_paddled]-(friend_rec) RETURN friend_rec</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*3FFIXWOjiBMLcvGZ5iPj3A.png" /><figcaption>Here we see the results (or recommendations) from the query. Natalia is recommended since they have both paddled the Ottawa river and Leila because they have both paddled the Charles.</figcaption></figure><p>MATCH (n {full_name:”Natalia”})-[:has_paddled]-&gt;(r)-[:has_paddled]-(friend_rec) RETURN r</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7UhHE8skkdp9IVVoyo-_GA.png" /><figcaption>Possible friends for Natalia. Neo4j in the UI also shows the friendship connections between Jeff, Charles, and Kailey.</figcaption></figure><p>Since Natalia and Dave have no existing friends we do not need the WHERE statement (and by not having it we actually save time), but for someone like Jeff we will need to add the WHERE not (n)-[:friend]-(friend_rec) statement.</p><h4>More advanced queries</h4><p>So let’s recommend friends based both on users they are friends with and rivers they have paddled. Unfortunately, we will have to expand our data set further to test this fully. So I added a new person node “Ken” and made him a friend of Jeff’s and I also added Charles as paddling the Rouge. In case you find this confusing I added</p><p>one new person node “Ken”</p><p>a friend relationship between Ken and Jeff</p><p>added a has_paddled relationship between Charles and the Rouge river.</p><p>Okay now that we have a larger data set to test this on let’s tryout our new idea:</p><p>MATCH (n {full_name: “Charles”})-[:has_paddled]-&gt;(r)-[:has_paddled]-(friend_rec) RETURN friend_rec UNION MATCH (n {full_name: “Charles”})-[:friend]-(t)-[:friend]-(friend_rec) WHERE not (n)-[:friend]-(friend_rec) RETURN friend_rec</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/1*R-HPsywZjOVizb9PpxYyHQ.png" /></figure><p>Exactly what we wanted. Ken is returned because he is a mutual friend of Jeff but not of Charles. Natalia and Dave are returned because of similar rivers that they have paddled.</p><p>This is good but what if we could return ranked results based on the number of connections. Specifically, we want a system that gives the results with more rivers in common and more friends in common a higher rank.</p><p>MATCH (n {full_name: “Charles”})-[:has_paddled]-&gt;(r)-[c:has_paddled]-(friend_rec) RETURN count(c) as count, friend_rec UNION MATCH (n{full_name: “Charles”})-[:friend]-(t)-[s:friend]-(friend_rec) WHERE not (n)-[:friend]-(friend_rec) RETURN count(s) as count, friend_rec ORDER BY count</p><p>Now we cannot see the rankings in the normal graphical UI we’ve been using above, however, the query does indeed return the correct results.</p><p>Node count</p><p>Ken 1</p><p>Dave 1</p><p>Natalia 2</p><p>So that about sums up Part 1 of our five part series. We are off to a good start however, our system is still very much incomplete. Some problems with this system are that it rates all relationships equally and does not factor in location. For instance, someone could have paddled a river once and the relationship will be weighted the same as a river they paddle everyday. Or someone could be friends with both someone they met once at a paddling festival and someone they paddle with all the time, yet they will have the same weight in the recommendation algorithm. In addition, someone could live in the North Carolina and have a lot friends with someone in Oregon and as a result, the majority of the recommendations will be for western rivers.</p><p>In Part 4 we will discuss some more sophisticated graph algorithms such as page rank, forests, and weighted edges. However, many of these require setting up Hadoop with Spark (which is a long task in and of itself). Our goal in this part was to show some simple (non-math intensive) out of the box techniques for recommending potential paddling partners and rivers. In Part 2 which will be published in a few weeks (plus or minus) we will discuss collaborative filtering algorithms and how use them with Neo4j.</p><p>We will have the code to recreate our test database at the following link shortly:</p><p><a href="https://github.com/isaacmg/rec-engine">isaacmg/rec-engine</a></p><p>Remember to follow us on Medium and like us on Facebook. We hope you have a great weekend on the water!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=16c3cea32ff" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a River and Paddling Partner Recommendation Engine]]></title>
            <link>https://medium.com/@paddlesoft/building-a-river-and-paddling-partner-recommendation-engine-1fda8c8bbb82?source=rss-ffe5d69df3f------2</link>
            <guid isPermaLink="false">https://medium.com/p/1fda8c8bbb82</guid>
            <category><![CDATA[paddling]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <category><![CDATA[algorithms]]></category>
            <dc:creator><![CDATA[PaddleSoft]]></dc:creator>
            <pubDate>Tue, 08 Mar 2016 20:03:39 GMT</pubDate>
            <atom:updated>2016-03-08T20:03:39.096Z</atom:updated>
            <content:encoded><![CDATA[<p>At PaddleSoft we are always trying to help paddlers make new connections. As a result, from the very beginning we wanted to develop a system that would make it easy for us to give paddlers customized recommendations, regarding, both new paddling partners and rivers that they might enjoy. Online recommendation engines have become very common in a variety of industries. Some examples of recommendation systems are Amazon product recommendations, Netflix film recommendations, and Facebook friend recommendations. Of course one of the most notable examples of recommendation systems are online dating websites almost all of which claim their advanced heuristics will help you find your “soulmate.” While recommending a date and recommending a movie to watch may seem like completely different things, both employ essentially the same fundamental techniques and algorithms from data science. Moreover, these techniques can easily be applied to matching any two groups of items.</p><p>With this in mind we will be writing a five part mini-series describing the construction of a Paddling Partner and River Recommendation engine. In our series we will mostly use the graph database Neo4j. So if you do not know too much about it you may want to check it out at: <a href="http://neo4j.com/">http://neo4j.com/</a></p><p>In the first part, we will go over simple techniques that can be used “out of the box” with Neo4j. In part two we will discuss collaborative filtering based approaches to recommendations and in part three we will discuss content based methods. In part four, we will investigate more complex graph algorithms/techniques such as forests, weighted edges, cycles, and page rank. This will require installing Hadoop and Spark, which we will detail in this part as well. Finally, in part five we will put the techniques together to create a hybrid engine.</p><p><strong>Part 1 Simple Graph Algorithms</strong></p><p><strong>Part 2 Collaborative Filtering</strong></p><p><strong>Part 3 Content Based Filtering</strong></p><p><strong>Part 4 Complex Graph Algorithms/Analysis using Hadoop and Spark</strong></p><p><strong>Part 5 Hybrid Recommendation</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1fda8c8bbb82" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Teaching “Paddling Lingo” Part 1]]></title>
            <link>https://medium.com/@paddlesoft/teaching-paddling-lingo-part-1-4dbf639bfa6c?source=rss-ffe5d69df3f------2</link>
            <guid isPermaLink="false">https://medium.com/p/4dbf639bfa6c</guid>
            <category><![CDATA[whitewater]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[PaddleSoft]]></dc:creator>
            <pubDate>Fri, 08 Jan 2016 04:58:25 GMT</pubDate>
            <atom:updated>2016-01-08T16:04:13.509Z</atom:updated>
            <content:encoded><![CDATA[<p>At PaddleSoft we constantly seek to incorporate new technologies that we believe paddlers will find useful. One of our central goals is to provide paddlers with information quickly and easily. In order to do this however, computers need to “understand” the language that paddlers use. The field of Natural Language Processing (NLP) has made significant strides over the past few years with many advanced systems such as IBM Watson answering Jeopardy questions, online chat bots, and advanced Q&amp;A modules. One of the techniques used by computers is to understand the meaning of words based on their context. Boaters like other groups, use words in certain contexts that differ from the dictionary definition, or what people commonly associate with the word. Therefore, we thought it would be interesting to see if using this technique we could get the computer to “recognize” or “understand” certain words that paddlers use frequently and what they mean by them.</p><p>Some of these words we will look at are:</p><p>(definitions from <a href="http://www.keelhauler.org/khcc/Paddling_Dictionary.htm">http://www.keelhauler.org/khcc/Paddling_Dictionary.htm)</a></p><p>eddy — “A place in the river, often behind an obstruction or inside a sharp turn, where the water reverses and flows upstream. Eddies are a good place to pause, rest, or boat scout. They are also the place where your gear is likely to collect after your bowman misses the draw stroke, your boat broaches and you forget to lean downstream. See yard sale.”</p><p>hole- “is a river feature where water drops over a obstruction (rock ledge or a rock) into deeper water on the downstream side. This causes water on the surface to be drawn back toward the rock or ledge. This can be a potentially hazardous feature but it could also be a feature used for playboating. Low head dam’s are the most dangerous example of a hydraulic.”</p><p>ferry- “Angling the boat to move sideways or upstream against a current, a properly executed ferry uses the current to help move the boat sideways. A Hairy Ferry is a ferry with dire consequences if you screw up.”</p><p>creek- “Paddling (or simply bouncing down) small, high gradient streams. Also known as steep creeking.”</p><p>Let’s see how well a computer can pickup these words based on context.</p><h3><strong>Word2Vec</strong></h3><p>“Given enough data, usage and contexts, Word2Vec can make highly accurate guesses about a word’s meaning based on past appearances. Those guesses can be used to establish a word’s association with other words (e.g. “man” is to “boy” what “woman” is to “girl”), or cluster documents and classify them by topic. Those clusters can form the basis of search, sentiment analysis and recommendations in such diverse fields as scientific research, legal discovery, e-commerce and customer relationship management.” -<a href="http://deeplearning4j.org/word2vec.html">http://deeplearning4j.org/word2vec.html</a></p><p>Word2Vec is an implementation (using neural networks) of the Skip-Gram Model which converts words into vectors based on their context in sentences (if you want more of the technical mathematics behind it you can find that here <a href="http://deeplearning4j.org/word2vec.html">http://deeplearning4j.org/word2vec.html</a>). There are many Word2Vec versions in multiple languages: we chose to use the Java implementation Deeplearning4j or Dl4j version. Once trained you can use the model to determine the closeness i.e, the cosine similarity between words. The Dl4j website for instance, shows the cosine similarity between Sweden and Norway is .76. For Word2Vec to properly function, it requires a large, and by large I mean massive corpus of plain text for use as training data. For example, people frequently use the entire Wikipedia corpus (that is every English language article) as training data. However, we decided against this because paddling terminology can often mean different things in a paddling context than a general context (for example hole, eddy, ferry…). We needed a large corpus of paddling related text in order to get Word2Vec working.</p><h3>Initial Data Mining</h3><p>Anyone in the field of data science will tell you that getting the data is often the most difficult part of the process. In order to build a plain text file to run Word2Vec we initially went to Facebook. There are hundreds of paddling related Facebook groups with thousands of posts on whitewater paddling. However, extracting the plain text posts to form a text corpus would obviously be way too time consuming to do manually. So we used Facebook’s Graph API explorer to issue bulk requests of whitewater related group and page feeds. These requests resulted in JSON data. We however, still needed just the text extracted from the posts so we built a JSON parser in Java (using the JSON library) to extract the message attribute. This produced plaintext which we then compiled into a single file. Facebook unfortunately, limits page feed requests to 250 posts and group requests to 2000 posts. While this may seem like a lot, remember the majority of posts are only one or two line messages. After five hours of searching for and extracting data we still had only a 410kb text file. Nevertheless, we decided to at least attempt to run Word2Vec on the dataset and see what it produced.</p><h3>Initial Results</h3><p>When we first ran Word2Vec on the dataset it produced almost complete garbage. This was mainly due to the fact that we forgot to remove the “stop words” from the text. Stop words are words like “a,” “I,” “you,” “of,” that contribute very little to the text but are very common in English. This caused words to become disproportionately related to these stop words. We then removed the stop words and reran Word2Vec. Once removed our dataset contained approximately 70,635 words (much smaller than the 1.9 billion of the Wikipedia corpus).</p><p>Closest words on FB Dataset (we removed about hundred excess ):</p><p>eddy:</p><p>[@, est, dryway, pm, begin, mens, team, west, bulls, bridge]</p><p>creek:<br>[the, this, sunday, -, weekend, saturday, am, release, river, is]</p><p>kayak: <br>[technology, adventure, paddles, oakley, aunz, yakima, equipment, est, iceland, anton]</p><p>wave: <br>[technology, @, est, mens, begin, paddles, iceland, anton, pm, oakley]</p><p>hole:<br>[dryway, saturday, sunday, deerfield, pm, river, west, @, am, release]</p><p>cosine similarities:</p><p>river to creek <br>0.54892897605896</p><p>hole to sticky <br>0.3262569010257721</p><p>Here we can clearly see the limitations of a small dataset. Certain words are over represented and Word2Vec even considered non words like @ and est as valid vectors. Additionally, it might be useful to remove days of the week as they do not add meaning to the text, along with punctuation.</p><h3>More data mining</h3><p>Clearly we needed a bigger dataset to effectively teach paddling terminology. So we decided to look into other data sources like online paddling message boards and other paddling websites. However, without access to these website’s databases we had to build a text extraction engine. Information extraction is a science in and of itself as many webpages differ in formatting. For this reason we generally tried to pick online message boards that generated &lt;divs&gt; with consistent class ids and whose links to posts were easy to extract. We built a text extraction engine using JSOUP, an information extraction API built for Java. We also looked at various paddling clubs’ trip reports. Trip reports provided useful data as they gave several paragraphs of uninterrupted text with few URLs or other odd formatting arrangements. Using this we gathered an additional 18mb of data or 3.2 million additional words.</p><h3>Cleaning Text and fine tuning the training</h3><p>Before retraining the neural network we did extensive cleaning on the text. We removed stop words and other punctuation marks. We set the minimum number of words to five in the training parameters in order to weed out URLs and misspelled words. Our final Word2Vec model ended up looking like this</p><p>Word2Vec</p><p>vec = new Word2Vec.Builder()<br> .minWordFrequency(5).iterations(1)<br> .layerSize(layer).lookupTable(table)<br> .stopWords(new ArrayList&lt;String&gt;())<br> .vocabCache(cache).seed(42)<br> .windowSize(5).iterate(iter).tokenizerFactory(t).build();</p><h3>Results</h3><p>Initial results on the new dataset were still not great despite our best efforts; useless words managed to creep into the results. Abbreviations for the days of the week remained along with odd punctuation marks. This demonstrated problems with our text extraction and cleaning methods as clearly a lot of interference was still getting into the text. We did not remove all the stop words either as some still showed up. However, part of the problem can also stem from the informal “nature” of message boards themselves; whereas the Wikipedia corpus only has minor grammatical mistakes and minimal use of abbreviations, message boards feature tons of impromptu and poorly written posts.</p><p>eddy</p><p>[triple, am-, -, aka, along, haven, video, lifetime, rt, race]</p><p>creek <br>[decided, flipped, times, wait, managed, thru, walked, perfect, deal, realized]</p><p>kayak <br>[learned, teach, look, next, gotten, learning, canoe, taught, possible, loved]</p><p>wave <br>[hull, alot, can, available, tons, comfortable, are, glad, an, &lt;span]</p><p>hole<br>[curious, &lt;span&gt;, have, ledge, tue, &lt;span, or, except, sep, thu]</p><p>cosine similarity</p><p>river and creek <br>0.048014018684625626</p><p>hole and sticky <br>0.32787105441093445</p><h3>Discussion, Code and Further Experiments</h3><p>This experiment provided interesting insight into the use of Natural language techniques with regards to paddling and whitewater. We believe we could achieve much better results by continuing to mine paddling messaging boards and further cleaning the text. Over the next few months we will build and refine our text corpus. While we at PaddleSoft are still unsure of the final role Word2Vec will play in our Information Retrieval systems, we found this test a fun way to show the utility of emerging NLP technologies. Also we plan on testing newer versions of the Skip-Gram Model such as the Multi-Sense Skip Gram and the Non-Parametric Skip-Gram.</p><p>You can find the code for Word2Vec algorithm at</p><p><a href="https://github.com/isaacmg/word2vec-paddlelingo">https://github.com/isaacmg/word2vec-paddlelingo</a></p><p>as well as our Facebook JSON parser at <a href="https://github.com/isaacmg/FBJSONParser">https://github.com/isaacmg/FBJSONParser</a></p><p>Remember we at PaddleSoft are constantly investigating the newest technologies in order to make it easier to find whitewater related information.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4dbf639bfa6c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>