<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Wairagu Wilberforce on Medium]]></title>
        <description><![CDATA[Stories by Wairagu Wilberforce on Medium]]></description>
        <link>https://medium.com/@willy-wairagu?source=rss-ee28a250fa00------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*L87nfZ5PTAhpG-iqaEeBQg.jpeg</url>
            <title>Stories by Wairagu Wilberforce on Medium</title>
            <link>https://medium.com/@willy-wairagu?source=rss-ee28a250fa00------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 02:07:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@willy-wairagu/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Apache Airflow — Part 2]]></title>
            <link>https://medium.com/data-pulse-for-ai/apache-airflow-part-2-6c4092f1fe99?source=rss-ee28a250fa00------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c4092f1fe99</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[aws-ec2]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Wairagu Wilberforce]]></dc:creator>
            <pubDate>Tue, 21 Feb 2023 11:24:52 GMT</pubDate>
            <atom:updated>2023-11-10T07:17:50.908Z</atom:updated>
            <content:encoded><![CDATA[<h3>Apache Airflow — Part 2</h3><h4><strong>Setting up Airflow using an AWS EC2 Instance</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/441/1*i5z6hqG90xBh-udh8xG2tA.png" /></figure><p>For the second part of this series, we are going to look at one of the simplest way to set up airflow to get you up to speed in building scalable data pipelines. The overall objective of this project will be to pull data from the World bank API. Before getting into the nitty gritties of the project, lets first look at setting up Apache Airflow.</p><p>some of the most common methods of setting up airflow include:</p><ol><li>Using Python’s official third party software repository <a href="https://pypi.org/"><strong>PyPi</strong></a><strong>: </strong>the most effective way to install airflow using PyPi is by running the following command from your preferred local terminal:</li></ol><pre>pip install apache-airflow</pre><p>once installed, you can run the airflow server using the following code snippet</p><pre>airflow webserver -p 8080<br>airflow scheduler</pre><p>2. Using <a href="https://docs.docker.com/get-started/"><strong>Docker</strong></a>: You can also use Docker to set up Apache Airflow. This method involves creating a Docker image and running it as a container. To do this, you need to have Docker installed on your system. You can use the following command to create an Airflow Docker image:</p><pre>docker build -t my-airflow .</pre><p>Once the image is created, you can start Airflow by running the following command:</p><pre>docker run -d -p 8080:8080 my-airflow webserver<br>docker run -d my-airflow scheduler</pre><p>3. Using Apache Airflow <a href="https://airflow.apache.org/docs/helm-chart/stable/index.html"><strong>Helm Chart</strong></a>: Helm is a package manager for <a href="https://kubernetes.io/">Kubernetes</a> that allows you to install and manage applications on Kubernetes clusters. This chart will bootstrap an airflow deployment on a Kubernetes cluster using the <a href="https://helm.sh/">helm</a> package manager. To install Airflow using Helm, you need to have a Kubernetes cluster and Helm installed. You can install the Airflow Helm chart by running the following command:</p><pre>helm install apache-airflow apache-airflow/airflow</pre><p>4. Using cloud providers such as AWS: AWS provides a variety of options for deploying Airflow that can be categorized under IaaS, PaaS, as well as SaaS. The three main options of deploying airflow on AWS include:</p><p>i. Using AWS EKS: Stands for Elastic Kubernetes service, .</p><p>ii. Using an EC2 instance: stands for Elastic Compute Cloud</p><p>iii. Using AWS MWAA service: stands for AWS Managed Workflows for Apache Airflow service</p><p>In this article, we’ll look extensively on option 4 (ii) where we’ll set up an Ubuntu instance and install Airflow on it. We’ll also look at some of the perks that comes with using this approach as well as some of its limitations.</p><h4>Setting up an AWS EC2 Instance</h4><p>For the readers new to the AWS cloud, its a comprehensive cloud computing platform that includes infrastructure as a service (Iaas) and platform as a service (PaaS) offerings. In simple terms is that we can access computing services over the internet from anywhere. this services majorly include, computing, storage, databases, analytics, Application integration and many more. Here’s a detailed <a href="https://docs.aws.amazon.com/accounts/latest/reference/manage-acct-creating.html">documentation</a> on how to create your AWS account and get you started with the cloud</p><p>The AWS service majorly responsible for offering compute resources is AWS EC2 which stands for Elastic Compute Cloud. Its a web based service that provides resizable compute capacity. literally, you get to build and host your applications on servers at Amazon’s data center. Let’s get started with spinning up our ubuntu instance that we’ll use to install and run airflow.</p><ol><li>from you AWS Management console, search for EC2 and once on it click on the launch instance button as shown:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*CvBSmVDoiAUGtbfHo8ER0A.png" /></figure><p>2. Next give your instance a name, for this case we’ll call it “airflow_instance”. Then select a 64-bit Ubuntu AMI Image. to learn more on AMIs visit this <a href="https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AMIs.html"><strong>link</strong></a><strong>.</strong></p><p>3. For the instance size, select <strong>t2.medium</strong> from the drop down list which offers <strong>2CPU cores</strong> and <strong>4 GB RAM</strong> which is sufficient to run Airflow. This type is past the free tier limits and might charge <strong>USD 0.15 per hour</strong>.</p><p>4. Additionally, you should create a <strong>Key Pair</strong> that would aid you in accessing your instance. Once created, this key pair file will automatically get downloaded to your machine with a <strong>.pem</strong> extension. make sure to save it in your working directory for you will need it later on while connecting to your instance via SSH.</p><p>5. Under the networking section, create a new <strong>security group </strong>and be sure to check all three boxes to allow both SSH and HTTPs access. This will permit access to your instance from the specified IP range that you’ll choose on the adjacent drop down list. for starters its best you leave it at Anywhere IPv4. Here is more on <a href="https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html"><strong>security groups</strong></a><strong>.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XZTx_iuAKJZGw7lGfvq6SQ.png" /></figure><p>6. Finally, you can click on the <strong>Launch instance</strong> button and your EC2 instance will be created in a minute or two.</p><p>7. Once created, you can click on your instance and a brief summary of it will show up. Click on the <strong>connect</strong> button and navigate to the SSH client tab, copy paste the ssh command shown and open your preferred terminal. I&#39;m going to use git bash to connect to our instance.</p><p><strong>NB: </strong>make sure you run the command from the same working directory where your key pair file exists, otherwise an error will appear:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/575/1*k7SQKGdfC19UJQpHlDCGQQ.png" /></figure><h4><strong>Installing and starting the airflow server</strong></h4><p>Before starting the installation process, we first need to update as well as install python’s <strong>pip</strong> library to our ubuntu filesystem using the following commands</p><pre>sudo apt-get update<br>sudo apt install python3-pip</pre><p>Once this is in order, next step involves installing airflow, we first check the python version we are currently running using the following command:</p><pre>python --version</pre><p>Next is for us to set the environmental variable for our constraint url that we will use to install airflow based on the python version we are running:</p><p>Execute the following command replacing where we have the first {} with the airflow version you want to install and the second {} with the python version you are currently running. In my case i’m installing airflow 2.5.1 an d im running python 3.10</p><pre>CONSTRAINT_URL=&quot;https://raw.githubusercontent.com/apache/airflow/constraints-{airflow_version}/constraints-{python_version}.txt&quot;</pre><p>Once that is done, we can now install airflow using the following command;</p><pre>pip install apache-airflow==2.5.1 --constraint &quot;${CONSTRAINT_URL}&quot;</pre><p>Finally once airflow is installed we can spin up a default standalone user using the following commands</p><pre>airflow db init <br><br>airflow standalone</pre><p>The above command sets up all airflow components and creates a default user credentials for you to Log in. By default, the server runs at a dedicated port <strong>8080 </strong>that you can access from any web browser.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*xNYY9WQvlC4PrXXRT9UGPA.png" /></figure><p>to access your airflow UI from your browser, head to your instance details page from the management console. copy paste the <strong>Public IPv4 address</strong> to a new tab of your browser and add <strong>“:8080”</strong> at the end of that URL. Enter the log in credentials given from your terminal and the following page appears with demo DAGs to get you started.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/1*ZC5R6BjSEJC6DY5bdfS4IQ.png" /></figure><p>NB: Always remember to shutdown all running instances to minimize on costs when not in use.</p><p><strong>Limitations of using this approach</strong></p><p>For production level workflows, data pipelines need to be triggered either on demand or on a schedule. For scheduled based workflows can vary from minutes, hours to daily based triggers. For instance, if one’s project needs demands a daily based trigger, this means that your instance will need to run through out which is a very bad idea.</p><p>This method is therefore suitable for Proof of concept applications to check if your Airflow DAGs runs successfully and is not to be used for production level workflows.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c4092f1fe99" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-pulse-for-ai/apache-airflow-part-2-6c4092f1fe99">Apache Airflow — Part 2</a> was originally published in <a href="https://medium.com/data-pulse-for-ai">Data Pulse for AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A deep dive into Apache Airflow — Part 1]]></title>
            <link>https://medium.com/data-pulse-for-ai/a-deep-dive-into-apache-airflow-part-1-278357d17699?source=rss-ee28a250fa00------2</link>
            <guid isPermaLink="false">https://medium.com/p/278357d17699</guid>
            <category><![CDATA[etl-pipeline]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[data-engineer]]></category>
            <category><![CDATA[apache-airflow]]></category>
            <dc:creator><![CDATA[Wairagu Wilberforce]]></dc:creator>
            <pubDate>Tue, 01 Nov 2022 09:54:05 GMT</pubDate>
            <atom:updated>2023-11-10T07:17:02.100Z</atom:updated>
            <content:encoded><![CDATA[<h3>A deep dive into Apache Airflow — Part 1</h3><h4><strong>What is Apache Airflow?</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/345/1*rGAJOeEAba5nO6ge-o2MyQ.png" /><figcaption>Apache Airflow</figcaption></figure><p><a href="https://airflow.apache.org/">Apache Airflow</a> is an open-source workflow engine that can easily schedule and run complicated data pipelines, represented as DAGs, in a robust manner and at a massive scale. The term data pipeline is used to describe a set of sequential processing steps applied to data to get it from one initial state to a desired end state. some of this steps are independent of one another allowing them to be executed in parallel. DAGS in the other hand in full stands for Directed Acyclic Graphs. These are directed graphs(with directional dependence) representing a data pipeline process without loops or cycles. that is, no node is dependent on nodes further down the chain of execution of the pipeline. A node in a DAG represents tasks to be executed in a data pipeline process, task dependencies are represented by directed connections(known as edges) between the task nodes.</p><p>Apache Airflow uses the above described acyclic property of DAGS to build out complex, robust and scalable data pipelines. The general algorithm used is as described,</p><ol><li>For each open node in the DAG,</li></ol><ul><li>identify the upstream task(s) upon which the current task is dependent.</li><li>Determine if all upstream tasks identified have been successfully executed.</li><li>If all upstream tasks identified have not been successfully executed, move to the previous task and repeat the first two steps.</li><li>If all dependent tasks have been successfully executed, add the current task to a queue of tasks to be executed.</li></ul><p>2. Sequentially execute all tasks in Queue by assigning a “completed” status to the tasks that have been successfully executed</p><p>3. Repeat steps 1 and 2 for the remaining tasks in the DAG.</p><p>The below simulation shows an example DAG execution algorithm that would, fetch students exam marks, perform some transformations such as filling missing marks with 0’s then load the final processed data to a dashboard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*os1yoD4nFGCG_3bwPaJ6BQ.gif" /></figure><h4>Applications and pros of Apache Airflow</h4><p>Airflow has in the recent years grown to having a widespread community support with over 22K GitHub stars of active users. some of its applications include.</p><ol><li>Coordinating <a href="https://www.databricks.com/glossary/extract-transform-load">ETL</a> and <a href="https://www.techtarget.com/searchdatamanagement/definition/Extract-Load-Transform-ELT">ELT</a> pipelines that extract data from multiple sources, run spark jobs or other transformations</li><li>Ingesting data from multiple <a href="https://aws.amazon.com/what-is/restful-api/">REST APIs</a></li><li>Automating Machine learning workflows, i.e training, testing and deploying machine learning models.</li></ol><p>Advantages of using Airflow as compared to other alternative orchestration tools;</p><ol><li>Has a vast open source community</li><li>Very easy to use</li><li>Allows integrations with cloud platforms such as GCP, AWS, Azure etc</li><li>Easy to use with standard Python</li><li>Has a flexible easy to use User interface.</li></ol><h4>Scheduling and Executing Airflow Pipelines</h4><p>The core components used by airflow while constructing and executing data pipelines are as listed;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UfBJcPzKfy6xlS4IScf__g.jpeg" /><figcaption>A typical Airflow Session</figcaption></figure><p>An outline of this components is as follows;</p><ol><li><strong>User</strong> — The example Airflow session starts with the <em>user</em> creating various DAG files defining data pipelines to be executed.</li><li><strong>The DAGs folder</strong> — The various DAG files are placed in a central <em>DAGs folder</em> with their location known to the Airflow scheduler.</li><li><strong>Airflow scheduler</strong> — The <em>Airflow scheduler</em> reads the DAG files to extract the corresponding tasks, dependencies, and schedule intervals of each DAG. Once completed, the <em>Airflow scheduler</em> checks if the schedule intervals for each specific DAG have passed, and if so, the DAG’s tasks are scheduled for execution. The scheduler’s next action is to check if every task dependency has been met. When this is the case, the task is added to the execution queue.</li><li><strong>Execution queue</strong> — The purpose of the <em>execution queue</em> is two-fold. First, it ensures that the sequential dependence of task execution is preserved. Second, it provides an <em>executor</em> process that is responsible for managing the resources of workers that will run the tasks.</li><li><strong>Airflow workers </strong>— Following tasks being placed in the execution queue, <em>Airflow workers</em> are each assigned a respective task from the executor process that they are responsible for completing. The workers are responsible for running jobs.</li><li><strong>Airflow Metastore</strong> — Metadata, such as if a pipeline ran successfully, when it was queued and executed, and conditions surrounding how it was run, are then written to the Airflow metastore. It’s important to note that metadata are captured from both the workers and scheduler to this storage repository.</li><li><strong>Airflow webserver</strong> — The relevant metadata are then displayed visually in the Airflow User Interface (UI) via the <em>Airflow webserver</em> service. A user can then access the Airflow UI and investigate details surrounding each task and its execution in a specific DAG.</li></ol><p>Here, we’ve looked at a brief basic theory behind Apache Airflow and how it’s fundamental block components interact with each other. Next we’ll look at a sample project that extract data from World Bank’s open API and orchestrate the process using Airflow. With this data we can try building an analysis around it or report some key indicators using a dashboard. Stay tuned.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=278357d17699" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-pulse-for-ai/a-deep-dive-into-apache-airflow-part-1-278357d17699">A deep dive into Apache Airflow — Part 1</a> was originally published in <a href="https://medium.com/data-pulse-for-ai">Data Pulse for AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>