<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Felix Ude on Medium]]></title>
        <description><![CDATA[Stories by Felix Ude on Medium]]></description>
        <link>https://medium.com/@doppelfelix?source=rss-59fcb20cdb33------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*0FLsh09CIaAIfYzUI5zK-A.jpeg</url>
            <title>Stories by Felix Ude on Medium</title>
            <link>https://medium.com/@doppelfelix?source=rss-59fcb20cdb33------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 21:19:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@doppelfelix/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Spatial Analysis of Public Transport Infrastructure]]></title>
            <link>https://doppelfelix.medium.com/spatial-analysis-of-public-transport-infrastructure-8f0972fb8f11?source=rss-59fcb20cdb33------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f0972fb8f11</guid>
            <category><![CDATA[geopandas]]></category>
            <category><![CDATA[mapping]]></category>
            <category><![CDATA[spatial-analysis]]></category>
            <category><![CDATA[gis]]></category>
            <category><![CDATA[infrastructure]]></category>
            <dc:creator><![CDATA[Felix Ude]]></dc:creator>
            <pubDate>Thu, 01 Sep 2022 15:13:42 GMT</pubDate>
            <atom:updated>2022-09-09T07:11:42.817Z</atom:updated>
            <content:encoded><![CDATA[<p>How to make sense out of public data with the help of a spatial analysis and mapping techniques</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*ru5TsYhjwXBDCxYTnSuHog.gif" /><figcaption>Bike availability in Hamburg by hour | Image by the author</figcaption></figure><h4>Introduction</h4><p>Living myself in the rather large city of Hamburg, public transport is generally considered widely available. However, even though I live in a center location, I always felt a little frustrated due to far distances to public bike stations.</p><p>Today, I want to show you how we can use public data to generate insights with the help of a spatial analysis and data visualizations. In detail, we will investigate the following things: 1. The availability of public bikes over the day. 2. The walking distances to bike stations. 3. Whether I am really out of luck or just lazy.</p><p><strong>TL;DR</strong></p><p>Turns out I’m too lazy! You can find an overview of the results on this <em>streamlit</em> application:</p><p><a href="https://doppelfelix-isochron-hamburg-streamlitbikes-over-the-day-awtwm5.streamlitapp.com/">Walking distances to Infrastructure in Hamburg</a></p><h3><strong>Availability of Bikes</strong></h3><h4>The data</h4><p>To answer any kind of question, we need to get our hands on some data first. In this case, we can make use of the public data REST API provided by the city of Hamburg. The data is rather nested and the extraction process not particular interesting (in case you are still interested: you can find the link to the script I wrote <a href="https://github.com/Doppelfelix/isochron_hamburg/blob/main/upload_bike_data.py">here</a>). Essentially, I created a JSON file for each bike station in Hamburg that included the changes of amount of bikes in the last two years and uploaded it to AWS S3. After the cleaning, it had the following structure:</p><pre>{<br>“thingID”: 10,<br>“description”: “StadtRad-Station Mundsburger Brücke / Papenhuder Straße”,<br>“coordinatesX”: 10.019142,<br>“coordinatesY”: 53.565952,<br>“obs”: [<br>{<br>“result”: 1,<br>“resultTime”: “2020–12–23T08:22:02.456Z”,<br>“observationID”: 45812737<br>},<br> {<br>&quot;result&quot;: 18,<br>&quot;resultTime&quot;: &quot;2020-08-24T10:41:24.098Z&quot;,<br>&quot;observationID&quot;: 34813796<br>},<br>...<br>]</pre><h4>Data Preparation</h4><p>In order to work with data, we need to aggregate it in a useful way. After creating a single entry for each hour for each station in the last two years, the dataset already amounted 5.5 million observations. This called for some <strong>PySpark, </strong>which is a framework that allows to run big data operations on a cluster in a very efficient and fast way.</p><p>I’ll skip the details again, as this is not the focus of the article. The script basically loads all the files on S3, flattens the structure and creates one entry per hour for the last two years. After that, it will aggregate each station by hour and weekday and exports everything to a csv file. The file is now much smaller, as we only have 188 data points per station left, leaving us at around 48k rows, which is a much better number to work with.</p><h4>Visualization</h4><p>Using the aggregated data and two python libraries called <strong><em>folium</em></strong><em> </em>and<em> </em><strong><em>geopandas</em></strong><em> </em>we can create this animated plot (day of the week is decoded by the leading integer, starting with Monday). There is an obvious pattern to identify here. In the morning of the weekdays, a lot of bikes move from the outskirts towards the city center and return in the evenings. Saturday is starting later, but then a lot more bikes move to the center, whereas Sunday seems to be a lazy day overall, with general less movement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/594/1*7Z1Ci2KRJPMURvKmm2B9uw.gif" /><figcaption>Bike availability in Hamburg by day and hour | Image by the author</figcaption></figure><p>We can also use a radial plot to analyze a single station, in this case I picked one in the city center and one in the further outskirts. The same pattern as above can be noticed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ajm65g7xOU_kYYh1-IkF1A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bx_3-D9eo2ocX6yycdPJ_g.png" /><figcaption>Bike availability at different distances to city center | Images by the authors</figcaption></figure><p>If we aggregate all stations by city district and compare the available bikes with the number of inhabitants of that district, we can see a huge imbalance between some districts. The next graph shows the districts with the most and the least amount of bikes per inhabitant. The explanation for the high numbers seems to be rather straight forward. The districts on top are mostly districts in the city center, where lots of people commute to in the morning, raising the amount of bikes available, even though fewer people actually tend to live there.</p><p>The districts on the bottom are mostly vast and highly populated districts, that seem to be a little underrepresented by the public bike stations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_wb8BXR3BZPoFB884Mo55Q.png" /><figcaption>Top and Bottom 5 District for average bikes per inhabitant | Image by the author</figcaption></figure><p>How about the difference of deviation of bikes during the day between districts? Again, we can see huge differences here. Centered and industry districts tend to have a large daily deviation of bikes, whereas very remote districts tend to have a consistent amount of bikes during the day. The next two plots gives an example of such a comparison.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kKetmxGgbFlQO3Z1wU2fWw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ajm65g7xOU_kYYh1-IkF1A.png" /><figcaption>Two districts with the least and highest deviation of bikes during the day | Image by the author</figcaption></figure><p>Geospatial imbalances are always a concern for ride-sharing institutions. If this turns out to be too much of a concern, potentially a dynamic pricing system could flatten out some of the huge spikes and valleys. For example, making it cheaper to rent a bike from an under demanded station and vice versa.</p><h3>Walking Distance to the nearest Bike Station</h3><p>Another important aspect of public bike transports is the walking distances to the nearest bike station. For the purpose of the analysis, we can create a graph network of the city and calculate the shortest walking distance to the nearest station for each point (Node) on our map. For a more technical explanation, you can check out the recent article I published on the topic.</p><p><a href="https://doppelfelix.medium.com/creating-a-traveling-distance-map-for-a-whole-city-with-python-f2e063272832">Creating a traveling distance map for a whole city with Python</a></p><p>In the graph below (for the interactive version, follow <a href="https://doppelfelix-isochron-hamburg-streamlitbikes-over-the-day-awtwm5.streamlitapp.com/Walking_Distances_to_Infrastructure">this link</a>) you can see the average walking distance from most points in Hamburg to the nearest bike station. The brighter the color, the fewer minutes you need to walk. This can range from 0 minutes in bright yellow (if you&#39;re standing next to a station) to over 50 minutes in purple.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R4YZmkgk9Gj2rsRE0iyh2w.png" /><figcaption>Walking distance to closes bike station (brighter color implies fewer minutes) | Image by the author</figcaption></figure><p>You can not only identify the bike stations without any explicit markers, but also already identify areas where there seems to be a lack of access to bike stations. In a perfect world, we could ask the city to place a station on any purple node and would therefore reduce the walking distance for most people. But in order to do this more economically, we also need to consider population and population density.</p><p>I would make the case, that as a city planer you want to help as many people as possible with the least resources. For that reason, you wouldn&#39;t place new stations to places that are currently the furthest away from an existing station, but rather placing new stations at places that are far away for many people.</p><p>As an approximation, we can use the population of each district and assume that the number is evenly distributed over its area (which of course it is not, but we got to start somewhere). If we use the average population per node on our map depending on the district it’s located inside and then multiply it by the distance of that node, we create a metric that might be suitable to optimize for.</p><p>Weighted Walking Distance = Population * Walking Distance</p><p>With the population, we can also calculate the average walking distance to the nearest station: <strong>18.42 Minutes. </strong>Turns out, with my 10 minutes, I’m not as bad off as I thought.</p><p>Plotting the nodes again, for the districts within Hamburg using the weighted walking distance, we get the following Map:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XXqu8xgr6eg2g4W-T5Hy_w.png" /><figcaption>Walking distance to closes bike station weighted by population | Image by the author</figcaption></figure><p>This map now allows us to see the real deal. Areas which appear to have decent walking distances are now worse off due to a high population density, whereas more remote districts now perform a little better. We can also see that the bright yellow city center is indicating a high availability of bike stations for a considerable small population.</p><p>Starting with this map, we could now begin to place new hypothetical stations and try to optimize our network for the weighted distance KPI we created. I am using a very straight forward algorithm for this:</p><pre>1. Pick the node with the highest weighted walking distance<br>2. Place a new station there<br>3. Recalculate all the weighted distances<br>4. Repeat step 1 to 3 for each new station</pre><p>Now we let the algorithm run 100 times. You can see the performance of the algorithm in the next plot:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/1*4rnNq2JKm6CgpVlPV45dhQ.png" /><figcaption>Performance of the simple algorithm | Image by the author</figcaption></figure><p>As you can see, the algorithm managed to reduce the average amount of walking distance by over 4 minutes, roughly a reduction of over 20%. But we can also see that the algorithm is slowly losing its effect with each new station placed. Let’s have a look at the results on our map. On the left we see the original distance map and on the right we see the new version with the hundred new stations. We can already tell, that some of the further nodes have been supported by the new stations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jf1Z2BT3dnbiDLJmP1jT8w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q0J8A33AN8zDCvdHvtKhog.png" /><figcaption>Walking distance in minutes without and with the new stations | Images by the author</figcaption></figure><p>It gets more interesting though, if we have a look at the comparison between the weighted walking distance maps. Now, we can really tell that the algorithm did its job. Most of the problematic zones have been successfully eliminated. Even if some of those stations might not be super economical worthwhile, I would argue in general this approach worked out quite well and could be a potential starting point for future city planning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KKh3oIdO4gJegefqW1hP5Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WcmxglvgwLFs58gjl-eODA.png" /><figcaption>Weighted walking distances in minutes without and with the new stations | Images by the author</figcaption></figure><p>This brings us to the end of this small analysis. There is still much more one could dig into, but I hoped you learned something new today and got a sense of how to use public data for insights that could potentially be useful.</p><p>If you have some questions or want more information, feel free to reach out to me: <a href="https://www.linkedin.com/in/felix-ude-201bb8153/">Felix Ude on LinkedIn</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f0972fb8f11" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating a Traveling Distance Map for a Whole City With Python]]></title>
            <link>https://medium.com/better-programming/creating-a-traveling-distance-map-for-a-whole-city-with-python-f2e063272832?source=rss-59fcb20cdb33------2</link>
            <guid isPermaLink="false">https://medium.com/p/f2e063272832</guid>
            <category><![CDATA[folium]]></category>
            <category><![CDATA[graph]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[gis]]></category>
            <category><![CDATA[geopandas]]></category>
            <dc:creator><![CDATA[Felix Ude]]></dc:creator>
            <pubDate>Thu, 25 Aug 2022 16:46:12 GMT</pubDate>
            <atom:updated>2023-03-14T19:12:20.518Z</atom:updated>
            <content:encoded><![CDATA[<h4>Isochron mapping using graph networks and folium maps for multiple points simultaneously</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QXmNSXo4sUnTHnh_TIDUEQ.png" /><figcaption>Availability of public transport in Hamburg | Image by the author</figcaption></figure><p>Calculating the distances to all the places you can reach from your location is called an isochron and is quite useful if you would like to find out what places you can reach within a specific time. But what if you want to do it the other way around and create a map that shows the travelling distances from a list of points of interest to all points in your city? This tutorial will show you how you can create such isochron maps yourself!</p><p>You can find an overview of the results on this streamlit application: <a href="https://doppelfelix-isochron-hamburg-streamlitbikes-over-the-day-awtwm5.streamlitapp.com/Walking_Distances_to_Infrastructure">Walking distances to Infrastructure in Hamburg</a>.</p><h3><strong>Creating the Graph Network</strong></h3><p>Creating the graph network is rather easy. First, install and import the required libraries.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b09370d73c56b061a018821610037404/href">https://medium.com/media/b09370d73c56b061a018821610037404/href</a></iframe><p>Next, we create the graph for our starting address, giving us a list of nodes, which are essentially simplified street intersections. Afterward, we convert it to a GeoPandas dataframe to plot the generated nodes.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4cc40aedfd279b2ae6ee0bd3ba6fbef6/href">https://medium.com/media/4cc40aedfd279b2ae6ee0bd3ba6fbef6/href</a></iframe><p>dist is the distance from your starting location used to create the graph.</p><p>network_type is the type of transport you want to use, other options are: “all_private,” “all,” “bike,” “drive,” “drive_service.”</p><p>We can look at our network on a folium map using this code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/706501364041e039fc7043954457357c/href">https://medium.com/media/706501364041e039fc7043954457357c/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/1*Xh0sRE7LqcYaZwG1RiMV5g.png" /><figcaption>Folium map with the nodes created by us | Image by the author</figcaption></figure><h4>Creating isochrons for each point of interest</h4><p>Now for the interesting part: Creating the isochrons for each point of interest and calculating the distance to each of our freshly created nodes. In this case, we will use the locations of all subway stations in the city of Hamburg. To use the locations, we have to find the nearest node for each coordinate tuple. This is easily achieved with the help of the osmnx function nearest_nodes.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/98fd3203b9675161d9e865fe60298e79/href">https://medium.com/media/98fd3203b9675161d9e865fe60298e79/href</a></iframe><p>To calculate actual distances, we need to project the network on a map. The transformation varies depending on the latitude. (<a href="https://en.wikipedia.org/wiki/Map_projection">Wikipedia: Map projection</a>) We will use the default setting of the project_graph function to do so.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a95230f5b5b0e010b9f1ba5ed5471c21/href">https://medium.com/media/a95230f5b5b0e010b9f1ba5ed5471c21/href</a></iframe><p>Now, we could already create a distance array for a single point of interest (POI). If we use the networkx function shortest_path_length and only provide a source, it will calculate the shortest distance to all other nodes in the provided graph.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/280496580cedbb2e7575d1090e552004/href">https://medium.com/media/280496580cedbb2e7575d1090e552004/href</a></iframe><p>The structure of our dictionary would look like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bfc1549d6579063e66517de1e3255cd5/href">https://medium.com/media/bfc1549d6579063e66517de1e3255cd5/href</a></iframe><p>We could already proceed with this and create a plot from it. This will require some work, though, as we need to create a color map ourselves to present the distance nicely in a visual way on the Folium map.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/005667ca3e61c58422424c174b5763b2/href">https://medium.com/media/005667ca3e61c58422424c174b5763b2/href</a></iframe><p>We create a color map ourselves with steps for each minute from one to 50. With the help of a small mapping function, we rewrite the dictionary function to include the color code instead of the distance. Now, we have a color for each node in our graph and can plot our folium map again.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ca507737eac62f5cb302a0ac3995bba4/href">https://medium.com/media/ca507737eac62f5cb302a0ac3995bba4/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/493/1*Kg5ckUht_nQ6HRtYX-ECVg.png" /><figcaption>Colored nodes depending on the distance to POI (brighter is closer) | Image by the author</figcaption></figure><h4>Using all points of interest</h4><p>Remember that we wanted to use several points of interest to create our maps? We are nearly there. To iterate over our list of POI, we want to make sure to always keep the smallest distance. For that reason, I wrote a small custom dictionary class that only updates values if it is smaller than the existing value or not yet existing.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/037c18795b94affd7657b2ad1fb2ccf7/href">https://medium.com/media/037c18795b94affd7657b2ad1fb2ccf7/href</a></iframe><p>With this, we can finally loop over all POI and create a dictionary with each node and the smallest distance to any POI.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ec858a3ba9d3241694f62563d9d9e2b6/href">https://medium.com/media/ec858a3ba9d3241694f62563d9d9e2b6/href</a></iframe><p>With this new node_distances, we just repeat the steps above, i.e., mapping the colors and creating a folium plot, and voilà, we have our final plot:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zXFD8b-6tKozpAbwJXF9QQ.png" /><figcaption>Walking distance to the nearest subway station in Hamburg (15 km render distance) | Image by the author</figcaption></figure><p>I hope you liked this small tutorial! If you are interested, you can find the whole notebook <a href="https://github.com/Doppelfelix/isochron_hamburg/blob/main/Isochron%20Maps%20Tutorial/Isochron%20Maps.ipynb">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f2e063272832" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/creating-a-traveling-distance-map-for-a-whole-city-with-python-f2e063272832">Creating a Traveling Distance Map for a Whole City With Python</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pipeline in the cloud]]></title>
            <link>https://doppelfelix.medium.com/pipeline-in-the-cloud-6edb007c4d52?source=rss-59fcb20cdb33------2</link>
            <guid isPermaLink="false">https://medium.com/p/6edb007c4d52</guid>
            <category><![CDATA[apache-beam]]></category>
            <category><![CDATA[cloud-scheduler]]></category>
            <category><![CDATA[dataflow]]></category>
            <category><![CDATA[gcp]]></category>
            <dc:creator><![CDATA[Felix Ude]]></dc:creator>
            <pubDate>Mon, 13 Dec 2021 08:52:51 GMT</pubDate>
            <atom:updated>2021-12-13T08:52:51.547Z</atom:updated>
            <content:encoded><![CDATA[<h3>Setting up your first Dataflow on GCP</h3><p>Scheduling an automatic Dataflow Pipeline that extracts and cleans data in the cloud</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DVrZQ4UvhyuwImRDu_pyag.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@lachlangowen?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Lachlan Gowen</a> on <a href="https://unsplash.com/s/photos/river?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p><strong>Introduction</strong></p><p>Writing a functional Data ETL pipeline is already a hard job. Setting up your Kubernetes cluster that handles instances to dynamically scale with the workload is even harder. But don’t you worry, Google and its Dataflow got you covered, taking care of the heavy lifting for you. Just give it your pipeline and Google will take care of the rest.</p><p>In this article, you will learn how to move an existing Apache Beam pipeline to the Google Cloud and its Dataflow and have it run regularly. If you don’t have a working Beam pipeline yet, that runs locally on your machine or don’t even know what an Apache Beam even is, feel free to check out the first part of this series.</p><p><a href="https://doppelfelix.medium.com/using-apache-beam-to-automate-your-preprocessing-in-data-science-144a89392f15">Using Apache Beam to automate your Preprocessing in Data Science</a></p><p><strong>Prerequisites &amp; Costs</strong></p><p>For this project we need several prerequisites on Google Cloud Platform. As of the time of writing all resources used, fell into the free tier. However, the cost structure from Google is complex and depends on the region and also can potentially vary in the future. Please refer to <a href="https://cloud.google.com/free">https://cloud.google.com/free</a> for the current conditions.</p><p>Now let’s begin with the initial setup of GCP:</p><p>If you haven’t already signup and create a new project. You will also need a working Cloud SDK on your local machine (see here for installation and setup <a href="https://cloud.google.com/sdk/docs/install">https://cloud.google.com/sdk/docs/install</a>). Next, we will activate several APIs and products we will need later. You can find each via the global search of GCP.</p><ol><li>Firstly, create a new <strong>Cloud Storage</strong> bucket (you don’t need to specify anything, just pick a name and use all default parameters)</li><li>Go to <strong>Bigquery </strong>and create a new dataset (again just go with the default parameters)</li><li>Search for <strong>App Engine</strong> and activate it (it will help us to start our instances later)</li><li>Search for <strong>Dataflow API</strong> and activate it</li></ol><p><strong>Creating a Dataflow Template</strong></p><p>To have GCP run our dataflow regularly, we will first need to create a template that is stored in the bucket we created. For this, we will run the following command locally on our machine (make sure Cloud SDK is set up and you have a working Apache Beam). Most of these parameters just describe Apache Beam where to save the template as well of how much disk size will be required to run this (values below 50GB can lead to errors when executing later).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0071e836b9d05639f4b1af8772924171/href">https://medium.com/media/0071e836b9d05639f4b1af8772924171/href</a></iframe><p>If everything runs smoothly, we get the following feedback:</p><pre>INFO:apache_beam.runners.dataflow.internal.apiclient:A template was just created at location gs://tutorial-bucket-dataflow/templates/traffic_beam</pre><p>And if you would look in the bucket, you would indeed find the template file called traffic_beam. If we would like to, we can now already trigger a Dataflow job manually from our local machine using the following command.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cef7867a4e910a0a0c049d6eff85a4a6/href">https://medium.com/media/cef7867a4e910a0a0c049d6eff85a4a6/href</a></iframe><p>Here we specify not only the job but also the details of the worker type we want to assign it. In this case, as the job is not super resource-heavy, a single worker on one of the smallest available machines will suffice (Be aware, that increasing some of these parameters could lead to the exceeding of the free tier!). Also, we pass our parameter <em>query_date, </em>which we added to control for which day the data should be uploaded to our Bigquery table. We can now check the Dataflow page and should find our just created job there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hrmbXyLflf7PBjr6SQX2oA.png" /></figure><p>When opening the detail page of the job, we can also see the pipeline steps we defined in our template before.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/241/1*x7l-MYkdAaDIs1CYL8NPFg.png" /></figure><p><strong>Schedule the Dataflow Job regularly</strong></p><p>We now want to set up a schedule, so that each day all the entries of the previous days are cleaned and uploaded to our Bigquery. We could’ve hard-coded yesterday as a parameter in our Apache Beam, but that would’ve given us less flexibility and we wouldn&#39;t learn how to pass runtime value parameters. If our parameter would be the same every time we run our Dataflow we could’ve just created a new schedule job using <strong>Cloud Scheduler </strong>and just don’t have it pass any fixed parameter to the Dataflow. However, as our date parameter changes every day (for demonstration purposes) we will create a <strong>Cloud Function</strong> that executes custom Python code, in which we will create the Dataflow job and pass yesterday’s date as a string to our Apache beam script.</p><p>Access the <strong>Cloud Function</strong> page via the global search and create a new function. As the trigger select <strong>Cloud Pub/Sub</strong> and create a new topic. Pub/Sub is essentially a feed to which certain Google products can subscribe and listen to, while other products publish or post information. This will be our link between the function and the scheduler.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/547/1*ROmDHUW_t1kY7g4TA0tF9A.png" /></figure><p>In the next window, you will be able to select Python as a Runtime language and <em>start_dataflow </em>as the entry point. In the main window create the following function:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e3fa94c0d404625c7f01705569abb302/href">https://medium.com/media/e3fa94c0d404625c7f01705569abb302/href</a></iframe><p>We just create the string of yesterday’s date and create a Dataflow job via Python’s google api client. Lastly, we need to add two packages to the <em>requirements.txt</em> for the virtual environment to be properly set up.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5a631465ef3c2129e913d432bbbd2023/href">https://medium.com/media/5a631465ef3c2129e913d432bbbd2023/href</a></iframe><p>With the function setup, we would have another way to manually trigger the Dataflow job, always using yesterday’s date. As we want to automate this project, we will now set up a scheduler that triggers our new function daily. Search for Cloud Scheduler in the search bar and create a new schedule. This is pretty straightforward as you can see in the screenshot below. Just make sure to select the same Pub/Sub topic as you did in the step above. The field frequency is specified in the standard cron format. In this case, I set it up to run every day at 1 AM CET.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*QsFGhb6hYFqzFVYGFqdD5w.png" /></figure><p>The Message Body field is required, but we don’t need to actually pass anything meaningful here, so just pick any string you like.</p><p>After you press the “Create” button you’re done! Congratulations, you just scheduled your first Dataflow in GCP!</p><p>I hope you liked this small tutorial on how to get started with Dataflow in GCP. Stay tuned for the next part of this series, where I will show you how to extract the first insights from the gathered data and build your first machine learning pipeline.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6edb007c4d52" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Apache Beam to automate your Preprocessing in Data Science]]></title>
            <link>https://doppelfelix.medium.com/using-apache-beam-to-automate-your-preprocessing-in-data-science-144a89392f15?source=rss-59fcb20cdb33------2</link>
            <guid isPermaLink="false">https://medium.com/p/144a89392f15</guid>
            <category><![CDATA[gcp]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pipeline]]></category>
            <category><![CDATA[apache-beam]]></category>
            <category><![CDATA[bigquery]]></category>
            <dc:creator><![CDATA[Felix Ude]]></dc:creator>
            <pubDate>Sat, 27 Nov 2021 14:54:12 GMT</pubDate>
            <atom:updated>2021-12-05T12:53:34.392Z</atom:updated>
            <content:encoded><![CDATA[<p>Extracting, Cleaning and Exporting the data from a public API with the help of Apache Beam and GCP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vUz82dKdBkvLm0ZH9egNQA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@dvnil?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Danil Sorokin</a> on <a href="https://unsplash.com/s/photos/pipelines?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p><strong>Introduction</strong></p><p>During recent years it became apparent that Data Science has surpassed its peak of the hype cycle. Doing exploratory Data Analysis and trying out new algorithms within a Jupyter Notebook are still important steps, however, without any automation, there isn’t any inherent business value. For the work of a Data Scientist, it becomes therefore more and more important to set up Data pipelines that automatically digest new data and create new value out of it. One very promising framework for such a task is Apache Beam. A very potent, yet very easy to learn pipeline Framework, that runs locally just as fine as on a cluster in the cloud. This tutorial will show you the basic steps to create your own Apache Beam pipelines with a real-life example.</p><p><strong>The Problem to tackle</strong></p><p>The city of Hamburg recently launched its first data platform, including traffic flow data for cars and bikes in the city. The structure of the data is quite nested, so to get some insights out of it and use it in the future for Machine Learning, we need to extract the data first, transform (clean) it and then export it to somewhere useful. In other words, we need a good ETL process here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/1*vM-IQyiugJdSTM6ZH3Umcg.png" /><figcaption>Overview of Data Source Structure (Icons: Freepik)</figcaption></figure><p><strong>Apache Beam to the rescue</strong></p><p>As we want this process to run regularly and in the cloud, we are going to write our ETL pipeline in Apache Beam. This framework is optimized to run on clusters such as Google’s Dataflow, which we will use in an upcoming article. But for now, we focus on writing a working pipeline that runs locally on our machine and uploads the data to Google’s Bigquery Data warehouse. Our goal is to write a pipeline, that takes all available traffic stations as an input and then searches for the corresponding data streams, which again have several observations we want to obtain. As the last step, we want to clean and transform our data and export it. Perfect steps for Apache Beam and its parallelized pipeline.</p><p><strong>Let’s begin</strong></p><p>We begin of course with our imports which are mainly apache beam as well as some utility libraries, most noteworthy <em>urllib</em>, which we need to parse the REST API. Furthermore, we set up our logger, which will be especially useful once we eventually set up our pipeline up in the cloud.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d13060f116ae50ff8d1de788872037bb/href">https://medium.com/media/d13060f116ae50ff8d1de788872037bb/href</a></iframe><p>First, we will retrieve our pipeline input, the list of available stations. For this, we will in part make use of the query function of the API we are calling. Essentially, this base URL will return us all stations and their information that match our filter for car traffic stations. We will use this information to create our <strong>PCollection</strong>, the dataset within a beam pipeline.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/93ce4e99bb1bf7540b5976253a6728e8/href">https://medium.com/media/93ce4e99bb1bf7540b5976253a6728e8/href</a></iframe><p>Next, we will create our first beam function, also called <strong>DoFn</strong>. This function will take the stations from above and will filter each station’s data stream URL for the smallest aggregation duration, which is 15 minutes (encoded as “PT15M”). It will also extract the coordinates of the station and return it together with the description.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ba65b5710c7f0ece0597d383f50c712f/href">https://medium.com/media/ba65b5710c7f0ece0597d383f50c712f/href</a></iframe><p>The next <strong>DoFn</strong> will take the <strong>PCollection</strong> from the previous pipeline step as an input and will also take a date string as an input. Note that the parameter here is a special value provider argument, we therefore need to call the <strong>.get()</strong> function in order to receive the value during runtime.</p><p>As the API only returns observations in 5000 steps, we will just iterate until there is no more data being returned. Again, we make use of the query function of the API and only filter for the date which was passed as an input. Further down, we will describe how a parameter can be passed to a pipeline.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f1e2ef79cddd60cc7f9490737886bbc3/href">https://medium.com/media/f1e2ef79cddd60cc7f9490737886bbc3/href</a></iframe><p>This last function is just returning everything in a neat dictionary, we could potentially also add further pre-processing steps here later.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e87aeff0c000f4dd42840f05f2580196/href">https://medium.com/media/e87aeff0c000f4dd42840f05f2580196/href</a></iframe><p>Now it’s time to build our pipeline and put it all together. We first create our pipeline with <strong>beam.Pipeline()</strong>. Each step of the pipeline is added using the pipe operator ( “|” ) followed by a string describing the step as well as the actual beam function.<strong> Create()</strong> will create a new <strong>Pcollection</strong>, while <strong>ParDo()</strong> will apply our defined function to each of those elements within that <strong>Pcollection</strong>. We don’t have any fancy branching going on, just a linear pipeline of steps our beam will follow. Also for now, we will pass a hard-coded date to our <strong>get_obs()</strong> function.</p><p>Lastly, we will export our cleaned data using the I/O connector. For now, we will just export it to a local CSV file using the <strong>io.WritetoText </strong>function. And that’s it! We just call the <strong>run()</strong> function and the <strong>wait_until_finished() </strong>function that both do exactly as their names suggest.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/96d6e2aed3475390e53116aa0be9ebae/href">https://medium.com/media/96d6e2aed3475390e53116aa0be9ebae/href</a></iframe><p>We can now just run the beam with</p><pre>python traffic_beam.py</pre><p>Here is some example output of the log with the last two lines telling us that the output of our pipeline has been successfully sunk into a file:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/78d2c57e799cbf73467df7609eb87bb5/href">https://medium.com/media/78d2c57e799cbf73467df7609eb87bb5/href</a></iframe><p>If we now open at our output file, we can see that it is a list of JSON entries. Looking good! Here are some examples:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/173d711d4ccca8c146464650cfa46790/href">https://medium.com/media/173d711d4ccca8c146464650cfa46790/href</a></iframe><p><strong>Adding parameterization</strong></p><p>As promised we will now add the parameterization, because how else could we tell our beam easily to extract the data for a specific day. We first define a new class called <strong>UserOptions</strong>, that inherits the beam <strong>PipelineOptions. </strong>Inside the class, we define the new function <strong>_add_argparse_args </strong>and inside that again we call the <strong>add_value_provider_argument </strong>function, which works very similar to a regular python parsing argument. We could set a default value, but for this case, it makes more sense to set the new parameter “<em>query_dat</em>e” to required so that it needs to be specified every time we run it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/945b56edd404129c95248d5e6500f5d3/href">https://medium.com/media/945b56edd404129c95248d5e6500f5d3/href</a></iframe><p>Now we just need to add it to our main <strong>run() </strong>script and our<strong> get_obs() </strong>function, and we now have our first runtime parameter!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c9c36dde0e435f94371cdd8a309e3bc1/href">https://medium.com/media/c9c36dde0e435f94371cdd8a309e3bc1/href</a></iframe><p>As with normal python parameters, we can now just call our script with the inline parameter in the console:</p><pre>python traffic_beam.py --query_date 2021-11-17</pre><p><strong>Uploading to GBQ</strong></p><p>Extracting and cleaning the data is nice and all, but in most real-life scenarios, a CSV file on a local machine is not the desired result. Luckily, one of the advantages of apache beam is its modularity. We can just switch out one of our pipeline steps with another. In our case, we just replace the <strong>WriteToText() </strong>with <strong>WriteToBigquery() </strong>inside the <strong>run() </strong>function and add some more parameters to it. For that to work you need of course a GCP project setup, with a storage bucket and a Bigquery dataset.</p><p>There are 3 required parameters here:</p><ul><li><strong>schema</strong>: This needs to describe how each column is defined</li><li><strong>table</strong>: This tells GCP where to write our data to</li><li><strong>custom_gcs_temp_location</strong>: This will upload the results first to a temporary GCP storage bucket, before actually writing it to your Bigquery table</li></ul><p>The other 2 parameters are optional but helpful for us.</p><ul><li><strong>create_disposition</strong>: to create a new table if the table name could not be found.</li><li><strong>write_disposition: </strong>to append new data (optionally overwrites everything)</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1e42e345ec61d7db1923f33abd0c75ec/href">https://medium.com/media/1e42e345ec61d7db1923f33abd0c75ec/href</a></iframe><p>And that’s it! Just like before, we call our script and pass a query date, but now instead of a local CSV file our retrieved data will be uploaded to our Bigquery dataset.</p><p><strong>Summary</strong></p><p>In this article, we learned how to construct a simple Apache Beam pipeline, that pulls data from an API, cleans it, and then uploads it to Bigquery. While it is a great way to get your hands dirty and do your first steps, it won’t benefit you on its own. Beam shows its real power when used in a cloud system such as Google’s Dataflow. In the next article, I’ll show you how to set up one up and schedule it to run automatically.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=144a89392f15" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>