<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tim Walles on Medium]]></title>
        <description><![CDATA[Stories by Tim Walles on Medium]]></description>
        <link>https://medium.com/@tim.walles?source=rss-a32f3561c9df------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*D5SPxfCepCVXvmjYeIwFuw.jpeg</url>
            <title>Stories by Tim Walles on Medium</title>
            <link>https://medium.com/@tim.walles?source=rss-a32f3561c9df------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 21:29:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tim.walles/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How I developed an automated data-pipeline for a doubtful business model on AWS lambda using docker]]></title>
            <link>https://medium.com/@tim.walles/how-i-developed-an-automated-data-pipeline-for-a-doubtful-business-model-on-aws-lambda-using-docker-652bfaa7154f?source=rss-a32f3561c9df------2</link>
            <guid isPermaLink="false">https://medium.com/p/652bfaa7154f</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[education]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <dc:creator><![CDATA[Tim Walles]]></dc:creator>
            <pubDate>Fri, 10 Nov 2023 11:12:35 GMT</pubDate>
            <atom:updated>2023-11-10T12:52:22.472Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Gans is a startup developing an e-scooter-sharing system. It aspires to operate in the most populous cities all around the world. In each city, the company will have hundreds of e-scooters parked in the streets and allow users to rent them by the minute. One big potential group of users are young tourists travelling with cheap flights and use e-scooters as transport to the city center when weather allows it…</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hivxSVGSR1vZl4ly" /></figure><p>No, Gans isn’t a real company that believes that e-scooters with top speeds of 25km/h are viable means of transport to travel from, for example, Berlin Brandenburg airport to Brandenburger Tor (~25km). Gans is a fictional company inside a case study of the <a href="https://www.wbscodingschool.com/data-science-bootcamp/">Data Science bootcamp of WBS coding school</a> that I’m currently enrolled into. The goal of this case study in short: “develop a data pipeline that collects weather and flight data and deploy this pipeline on the cloud”.</p><blockquote>“Great story, just give me the code: <a href="https://github.com/TimWalles/Data_pipeline_on_cloud">GitHub repo</a>”</blockquote><h4>What I’ll be focusing on in this article</h4><p>Creating a data-pipeline is nice and all, but for what data, where was this data collected and where is it all stored? Each an important topic and will be briefly mentioned but, in this article, I’ll focus on the data pipeline with at its core a <a href="https://aws.amazon.com/pm/lambda/?trk=87411c70-c006-45e9-9121-9032c2dff285&amp;sc_channel=ps&amp;s_kwcid=AL!4422!10!71468491682362!71469018663901&amp;ef_id=f1a9bab354a51dba4b02dffea0ac4985:G:s">Amason Web Service (AWS) Lambda function</a>. I will show the logic I implemented into the AWS lambda function, how to deploy such a function using docker onto the AWS lambda cloud service and automate the function by setting triggers on AWS.</p><h3>The data</h3><p>My data pipeline automatically obtained weather forecast and flight data for the next 24 hours and stored onto a MySQL cloud database on AWS. The MySQL Database structure is shown below and the collected data from the pipeline would populate the weathers and flights table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/849/1*S6VXoraxER7x9VFXYC7AeA.png" /></figure><p>The tables cities, populations and cities_location were static tables containing city data obtained via web scrapping <a href="https://en.wikipedia.org/wiki/Main_Page">Wikipedia</a> (more detail see <a href="https://github.com/TimWalles/Data_pipeline_on_cloud/blob/6ca86f78850c4219c9922c69ab65599b68b078d4/city_data.ipynb">city data notebook</a>).</p><p>The cities_airports, airports, airports_locations tables, also static, contained airport data from airports near each city and was obtained via the <a href="https://rapidapi.com/aedbx-aedbx/api/aerodatabox/">AeroDataBox API</a> (check out <a href="https://github.com/TimWalles/Data_pipeline_on_cloud/blob/6ca86f78850c4219c9922c69ab65599b68b078d4/flight_data.ipynb">flight data notebook</a>).</p><h3>Setting up the automated data pipeline</h3><p>I developed a lambda function on AWS that triggers an update of weather and flight data each day, but how? It all starts with developing a handler function.</p><blockquote>Check out the official <a href="https://docs.aws.amazon.com/lambda/latest/dg/python-image.html">AWS developers guide</a></blockquote><h4>The handler function</h4><p>A handler function “handles” an incoming request, is responsible for firing the requested event and returns a response and creates logs for error monitoring and debugging purposes.</p><blockquote>Normally one would create two separate lambda functions, one for weather update and another for flights update as both access different sources, I decided to combine them for the sake of learning</blockquote><p>Below is my developed handler function. At first it might seem a lot is happening within but let&#39;s break it down into its individual parts.</p><pre>def handler(event, context):<br>    sql_con = MySQL()<br><br>    if &#39;event&#39; in event:<br>        match event[&#39;event&#39;]:<br>            case &#39;weather&#39;:<br>                # read cities_df from the sql server<br>                cities_df = pd.read_sql_table(&#39;cities&#39;, con=sql_con.con())<br><br>                # read cities_df from the sql server<br>                cities_df = cities_df.merge(pd.read_sql_table(&#39;cities_location&#39;, con=sql_con.con()), how=&#39;left&#39;)<br><br>                # update weather data on database:<br>                weather_df = get_weather(cities_df)<br>                if weather_df:<br>                    # truncate database<br>                    sql_con.truncate_df(&#39;weathers&#39;)<br>                    pd.DataFrame(weather_df).to_sql(&#39;weathers&#39;, if_exists=&#39;append&#39;, con=sql_con.con(), index=False)<br>                    logger.info({&#39;statusCode&#39;: 200, &#39;body&#39;: &#39;updated weather data successfully&#39;})<br>                    return {&#39;statusCode&#39;: 200, &#39;body&#39;: &#39;updated weather data successfully&#39;}<br>            case &#39;flights&#39;:<br>                # read cities_df from the sql server<br>                airports_df = pd.read_sql_table(&#39;airports&#39;, con=sql_con.con())<br><br>                # update flights data on database:<br>                flights_df = get_flight_info(airports_df)<br>                if flights_df:<br>                    sql_con.truncate_df(&#39;flights&#39;)<br>                    pd.DataFrame(flights_df).to_sql(&#39;flights&#39;, if_exists=&#39;append&#39;, con=sql_con.con(), index=False)<br>                    logger.info({&#39;statusCode&#39;: 200, &#39;body&#39;: &#39;updated flight data successfully&#39;})<br>                    return {&#39;statusCode&#39;: 200, &#39;body&#39;: &#39;updated flight data successfully&#39;}<br>            case _:<br>                logger.info({&#39;statusCode&#39;: 404, &#39;body&#39;: &#39;event trigger not found&#39;})<br>                return {&#39;statusCode&#39;: 404, &#39;body&#39;: &#39;event trigger not found&#39;}<br>    else:<br>        logger.info({&#39;statusCode&#39;: 404, &#39;body&#39;: &#39;event not found&#39;})<br>        return {&#39;statusCode&#39;: 404, &#39;body&#39;: &#39;event not found&#39;}</pre><p>When we call the handler function we need to pass in a <a href="https://www.w3schools.com/Js/js_json_objects.asp">JSON object</a> that populates the event variable. In the first step the handler will look inside this JSON object for a key called “event” and will return a status code 404 error if it doesn’t find this key. When a “event” key is present the handler will proceed with matching the value under the event-key against two cases, weather or flights; and determine which of these events should get triggered.</p><p>For example:</p><pre>{&quot;event&quot;:&quot;weather&quot;}&quot; # this will trigger the get_weather event</pre><p>The JSON object in the above example instructs the handler that we want to update the weathers table inside our MySQL database with new data.</p><blockquote>By changing the value under the event key, we either update weather or flights data making it possible for a single handler to handle multiple API endpoints.</blockquote><p>At this point you might think: <em>“Okay, you developed a handler function, so what? And why didn’t you do this directly inside the AWS lambda function creator?”</em></p><p>For the handler file and API endpoints to work means importing dependencies and functions, and we would like to do so using a folder structure that we designed.</p><pre>import logging<br><br>import pandas as pd<br>from utils.get_times import get_times<br><br>from normalizers.aero_databox_normalizer import normalize_flights_infos<br>from normalizers.open_weather_normalizer import normalize_open_weather_response<br>from service.aero_databox_api import AeroDataBox<br>from service.mysql_db import MySQL<br>from service.open_weather_api import OpenWeather</pre><p>Yes, none of this is impossible to do by manually creating <a href="https://docs.aws.amazon.com/lambda/latest/dg/chapter-layers.html">layers</a> inside the AWS lambda function creator containing all dependencies, but easier (and faster) is using docker…</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FYQ5bm1pbOFnSE6jhwI%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FYQ5bm1pbOFnSE6jhwI%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FYQ5bm1pbOFnSE6jhwI%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="435" frameborder="0" scrolling="no"><a href="https://medium.com/media/ecafce4796349092adfa477f56017a5c/href">https://medium.com/media/ecafce4796349092adfa477f56017a5c/href</a></iframe><h4>Docker</h4><p>Docker uses Dockerfiles that contain instructions on how to build the application. These building instructions, together with building blocks (scripts etc.) are stored into a docker container that can used by AWS Lambda to create a lambda function out of it without having to create layers. Below is the Dockerfile with building instructions I used.</p><pre>FROM public.ecr.aws/lambda/python:3.11<br><br># Copy requirements.txt<br>COPY requirements.txt ${LAMBDA_TASK_ROOT}<br><br># Install the specified packages<br>RUN pip install -r requirements.txt<br><br># Copy src folder<br>COPY src/ ${LAMBDA_TASK_ROOT}<br><br># Set the CMD to your handler (could also be done as a parameter override outside of the Dockerfile)<br>CMD [ &quot;main.handler&quot; ]</pre><p>My application runs in python3.11 and the first “building” instruction is to pull python3.11 from the public image link provided by AWS. After pulling python3.11 all dependencies listed in the requirements.txt needed for my lambda function to work are installed. In the third step the src folder is copied over to wherever the lambda function is running on AWS. In the final step the command for calling the handler function is set (filename.function_name)</p><p>Before I actually proceeded with pushing my handler function onto AWS lambda, I tested it locally. I built the docker image and ran it by running these commands from inside the folder containing the Dockerfile:</p><pre>docker build --platform linux/amd64 -t docker-image:test .<br>docker run -p 9000:8080 docker-image:test</pre><p>From a second terminal I could then trigger an event and (hopefully) got the expected response:</p><pre>curl &quot;http://localhost:9000/2015-03-31/functions/function/invocations&quot; -d &#39;{&quot;event&quot;:&quot;weather&quot;}&#39; # Change the payload and see how the response changes<br># response<br>{<br>  &quot;statusCode&quot;: 200, <br>  &quot;body&quot;: &quot;updated weather data successfully&quot;<br>}<br># success<br></pre><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2F3ohzdIuqJoo8QdKlnW%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2F3ohzdIuqJoo8QdKlnW%2Fgiphy.gif&amp;image=https%3A%2F%2Fmedia4.giphy.com%2Fmedia%2Fv1.Y2lkPTc5MGI3NjExOXB3dDd4a2pva2JnNW45dnJlZW05anRwbGFycGxkNDg2MXRoOWZrbCZlcD12MV9naWZzX2dpZklkJmN0PWc%2F3ohzdIuqJoo8QdKlnW%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="200" frameborder="0" scrolling="no"><a href="https://medium.com/media/2b6be93d955e20819e24de81eebb003f/href">https://medium.com/media/2b6be93d955e20819e24de81eebb003f/href</a></iframe><h4>Deploying the image</h4><blockquote>I deployed my docker image setting environment variables for lambda. This isn’t recommended but done by me as the AWS secret manager charges per kept secret and this being a learning project</blockquote><p>I compiled all commands used in the <a href="https://docs.aws.amazon.com/lambda/latest/dg/python-image.html">AWS developer guide</a> together into a shellscript.</p><pre>#!/bin/bash<br>REGION=<br>AWS_ID=<br>REPO_NAME=<br>LAMBDA_FUNCTION_NAME=<br>ARN_ROLE=<br><br># environment variables<br>OPEN_WEATHER_API=<br>OPEN_WEATHER_KEY=<br>AERO_DATA_KEY=<br>RD_HOST=<br>RD_SCHEMA=<br>RD_USER=<br>RD_KEY=<br>RD_PORT=<br><br># authorization<br>aws ecr get-login-password --region ${REGION} | docker login --username AWS --password-stdin ${AWS_ID}.dkr.ecr.${REGION}.amazonaws.com<br><br># create repository if not exist already<br>aws ecr describe-repositories \<br>    --repository-names ${REPO_NAME}  || aws ecr create-repository \<br>    --repository-name ${REPO_NAME} \<br>    --region ${REGION} \<br>    --image-scanning-configuration scanOnPush=true \<br>    --image-tag-mutability MUTABLE<br><br># build image<br>docker build --platform linux/amd64 -t docker-image:test .<br><br># add tag to aws repo<br>docker tag docker-image:test ${AWS_ID}.dkr.ecr.${REGION}.amazonaws.com/${REPO_NAME}:latest<br><br># push image to repo<br>docker push ${AWS_ID}.dkr.ecr.${REGION}.amazonaws.com/${REPO_NAME}:latest<br><br># create lambda function<br>aws lambda create-function \<br>    --function-name ${LAMBDA_FUNCTION_NAME} \<br>    --package-type Image \<br>    --code ImageUri=${AWS_ID}.dkr.ecr.${REGION}.amazonaws.com/gans:latest \<br>    --role ${ARN_ROLE} \<br>    --environment Variables=&quot;{OPEN_WEATHER_API=${OPEN_WEATHER_API},OPEN_WEATHER_KEY=${OPEN_WEATHER_KEY},AERO_DATA_KEY=${AERO_DATA_KEY},RD_HOST=${RD_HOST},RD_SCHEMA=${RD_SCHEMA},RD_USER=${RD_USER},RD_KEY=${RD_KEY},RD_PORT=${RD_PORT}}&quot;<br><br># invoke function for check<br>aws lambda invoke --function-name ${LAMBDA_FUNCTION_NAME} response.json</pre><p>To deploy my lambda function I had to fill in all required information at top of the shellscript and run it:</p><pre>bash deploy_image.sh</pre><p>or, in case of updating the lambda function:</p><pre>aws lambda update-function-code \<br>    --function-name  my-function \ # replace my-function with your lambda function name<br>    --image-uri # URI of a container image in the Amazon ECR registry</pre><p>When I then went to lambda inside my AWS console, I could see my deployed lamda function:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mvEEJnl1-797-bqCH65Apg.png" /></figure><p>But because it’s built from container image I couldn’t see the lambda handler content:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wwrpJPhk5-jhilwMs5zg9Q.png" /></figure><p>Celebration time! I deployed my lambda function on the AWS cloud service using a docker-image.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2Fo75ajIFH0QnQC3nCeD%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2Fo75ajIFH0QnQC3nCeD%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2Fo75ajIFH0QnQC3nCeD%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="362" frameborder="0" scrolling="no"><a href="https://medium.com/media/dabce226b8278078396e695c6d3ec384/href">https://medium.com/media/dabce226b8278078396e695c6d3ec384/href</a></iframe><h3>Automating the data-pipeline</h3><p>With the lambda function deployed it was time to automate the event triggers and start collecting data at a daily rate.</p><h4>Setting up the trigger event</h4><p>In the AWS <a href="https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-what-is.html">EventBridge</a> console I created two events and with logic naming:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wFiGdrlNnUZUHw7wc0edFA.png" /></figure><p>To trigger the right event, I had to define the payload correctly. For example, to trigger the weather update the payload had to look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eBAOlG8AGYmGvrROMWK5og.png" /></figure><p>Inside the <a href="https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/WhatIsCloudWatch.html">cloudwatch</a> events log I could see that the event was triggered and no errors occurred, meaning all executed as designed!</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FS1YVlFY74PCs0CLGfK%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FS1YVlFY74PCs0CLGfK%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FS1YVlFY74PCs0CLGfK%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="362" frameborder="0" scrolling="no"><a href="https://medium.com/media/c58113a4c91fefe469646f02b1ceee7d/href">https://medium.com/media/c58113a4c91fefe469646f02b1ceee7d/href</a></iframe><h3>Was all this helpful?</h3><p>For the gans company? You can decide whether e-scooters as transport to and from airports is the next big thing. For myself? Developing the data pipeline, building a cloud database, using docker and deploying docker images on the cloud was new, a lot of fun and did what I like most, learning a lot of things. For you? I don’t know but would love to hear!</p><p>Ciao!</p><p>LinkedIn: <a href="http://www.linkedin.com/in/timwalles">Tim Walles</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=652bfaa7154f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>