<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Oscar García on Medium]]></title>
        <description><![CDATA[Stories by Oscar García on Medium]]></description>
        <link>https://medium.com/@garc1a0scar?source=rss-8b92eced7bba------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*082eiZjF1S4g1VF2</url>
            <title>Stories by Oscar García on Medium</title>
            <link>https://medium.com/@garc1a0scar?source=rss-8b92eced7bba------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 05:41:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@garc1a0scar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Build your own AWS Glue Databrew operator for Apache Airflow]]></title>
            <link>https://medium.com/@garc1a0scar/build-your-own-aws-glue-databrew-operator-for-apache-airflow-b77662035d36?source=rss-8b92eced7bba------2</link>
            <guid isPermaLink="false">https://medium.com/p/b77662035d36</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[glue-databrew]]></category>
            <dc:creator><![CDATA[Oscar García]]></dc:creator>
            <pubDate>Tue, 22 Mar 2022 20:44:26 GMT</pubDate>
            <atom:updated>2022-03-22T20:44:26.801Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*hxNRP7alVn8Yl4R2wfy_5A.png" /><figcaption>Apache Airflow logo</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/210/1*l6PNZ0BpMFQ3epMefcPsXg.png" /><figcaption>AWS Glue DataBrew logo</figcaption></figure><p>In our article<a href="https://medium.com/@garc1a0scar/understanding-better-the-dags-and-operators-concepts-99187d397857"> Understanding better the DAGs (And Operators) concepts</a>, we could see in detail the purpose and difference of <strong>operators</strong> and <strong>hooks</strong>. Also, we could see that an <strong>operator</strong> is the set of instructions that internally call a <strong>hook</strong> in order to provide integration with external platforms.</p><p>The purpose of this article is to see in detail how <strong>operators</strong> and <strong>hooks</strong> interact and in addition, we will build an AWS Glue Databrew operator that will allow us to execute a job from Apache Airflow, analyzing in detail and understanding each of the steps followed to do it.</p><h3>Hook construction</h3><p>We will start with the code of the <strong>Hook </strong>that Apache Airflow uses for all AWS Services (Glue, S3, etc.) and we will analyze it. The full code is available at <a href="https://airflow.apache.org/docs/apache-airflow-providers-amazon/2.2.0/_modules/airflow/providers/amazon/aws/hooks/base_aws.html#AwsBaseHook">https://airflow.apache.org/docs/apache-airflow-providers-amazon/2.2.0/_modules/airflow/providers/amazon/aws/hooks/base_aws.html#AwsBaseHook</a>, but we will focus on the class AwsBaseHook, from which we can summarize the following things:</p><ol><li>The <strong>Hook</strong> can retrieve the parameters to create a connection and, in consequence, a boto3 low-level client with the defined <strong>client type</strong> parameter. This boto client is able to create a session with many of the AWS Services, including of course. Glue Databrew.</li><li>Among the functions that the boto client of Databrew has, we can find two main functions: <strong>start_job_run()</strong>, which, as its name implies, allows us to execute a job within the tool and<strong> describe_job_run() </strong>that allows us to retrieve metadata of our databrew jobs, including the <strong>run state</strong>. To know more about these functions or others, we can always refer to the official documentation in the <a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/databrew.html">boto official documentation</a>.</li></ol><p>With everything mentioned above, we can continue with the construction of the <strong>Hook</strong> that our operator will use to execute our jobs. Let’s see the following code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/23c15128d5af6930d588a0b0e0d60f43/href">https://medium.com/media/23c15128d5af6930d588a0b0e0d60f43/href</a></iframe><p>So, as we can see in the code above, a <strong>Hook</strong> code has a <strong>library/modules import section</strong> and the definition of the main class, with an<strong> init function</strong> and one or other optional functions, depending in the needed requirements.</p><p>In this example, our <strong>GlueDBJobHook </strong>is a derived class of the AWS Base Hook, that will inherit the functions within it.</p><p>In general terms, we have the following:</p><ol><li>An <strong>init</strong> function that retrieves the global parameters of the class and define one more parameter: client_type. This parameter will be used by the AWS Base Hook in order to create the boto3 client.</li><li>An <strong>initialize_job</strong> function that calls the <strong>start_job_run</strong> function of the client.</li><li>A <strong>job_completion</strong> function that calls the <strong>get_job_state</strong> function to receive the jun rob state every x seconds until completed.</li><li>A <strong>get_job_state</strong> function that calls the <strong>describe_job_run</strong> of boto3 client that reads the metadata in order to retrieve the job run state and send it to the function in 3.</li></ol><h3>Operator construction</h3><p>With all the above analyzed and understood, now we can finally create our <strong>operator</strong>. Let’s see the code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/220ff177bf3bb08424f888e9f247c882/href">https://medium.com/media/220ff177bf3bb08424f888e9f247c882/href</a></iframe><p>As we can see, an <strong>operator</strong> is also a class that has an <strong>init </strong>function and one or many optional functions. Generally, an operator is a derived class of the <strong>BaseOperator </strong>class. Now, let’s see in detail our operator code.</p><ol><li>First, we have our traditional <strong>libraries/modules import </strong>section.</li></ol><blockquote>import os<br>import warnings<br>from typing import TYPE_CHECKING, Optional, Sequence<br>from airflow.models.baseoperator import BaseOperator<br>from glue_db_hook import GlueDBJobHookimport time<br>from airflow.hooks.base import BaseHook</blockquote><p>2. After that, we need to define our <strong>init function</strong> with all the parameters that the function will be able to receive. In our case: job_name, aws_conn_id, region_name, **kwargs.</p><p>3. Finally, we have our <strong>execute function </strong>that, as we can see, calls the GlueDBJobHook that we reviewed above with the following parameters: job_name, aws_conn_id and region_name. This parameters will be used by the boto3 client in order to create the session.</p><p>4. After the session/client is created, we can finally call the <strong>initialize_job</strong> and <strong>job_completion</strong> functions, that will allow us to run the defined job and wait until completed to retrieve its status (failed/succeeded) and will return it to Apache Airflow.</p><p>Good news: We have finished in the construction of our <strong>Operator</strong>. Here is an example of how we can use it:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ac9282b156ed65aeb16a0a348c98398d/href">https://medium.com/media/ac9282b156ed65aeb16a0a348c98398d/href</a></iframe><p>Our DAG will have one simple <strong>task</strong>, called <strong>execute_job</strong>, that will run the AWS Glue Databrew job, will wait until completed (in any state) and will return the state to our execution in Apache Airflow.</p><h3>Conclusion</h3><p>In this article, we could see an example of how we can construct our own operators (and hooks) in order to interact with many platforms outside Apache Airflow.</p><p>Remember that for this to work, we need to upload our codes (hooks and operators) in our plugins folder. This is an easy task to do, but will depend in where and how is implemented our environment. But, in general terms, what you need to do is create a dags-level folder called “<strong>plugins</strong>” and configure your environment to read it.</p><p>I hope this article has given you a broader overview of how operators and hooks work, and with that, you can create your own operators to do much more.</p><p>Note: this article foused on running a Databrew job, but, as we could see, the boto3 client, has a lot of other functions that we can use within this code or other custom operators. In addition, you can review the boto3 documentation for glue, s3, kms, etc.</p><h3>References</h3><p>[1] Boto3. <em>Glue DataBrew . </em><a href="https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/databrew.html">https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/databrew.html</a></p><p>[2] AWS. <em>Orchestrate AWS Glue DataBrew jobs using Amazon Managed Workflows for Apache Airflow. </em><a href="https://aws.amazon.com/blogs/big-data/orchestrate-aws-glue-databrew-jobs-using-amazon-managed-workflows-for-apache-airflow/">https://aws.amazon.com/blogs/big-data/orchestrate-aws-glue-databrew-jobs-using-amazon-managed-workflows-for-apache-airflow/</a></p><h3>Until next code!</h3><h3>Os😉</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b77662035d36" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to make a Dynamic DAG and how to use a Parameters file]]></title>
            <link>https://medium.com/@garc1a0scar/how-to-make-a-dynamic-dag-and-how-to-use-a-parameters-file-685a8193703c?source=rss-8b92eced7bba------2</link>
            <guid isPermaLink="false">https://medium.com/p/685a8193703c</guid>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[parameter]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[dynamic-programming]]></category>
            <dc:creator><![CDATA[Oscar García]]></dc:creator>
            <pubDate>Tue, 08 Mar 2022 19:18:32 GMT</pubDate>
            <atom:updated>2022-03-08T19:18:32.392Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/350/0*czQF8OlVUMggKBdf.png" /><figcaption>Apache Airflow Logo</figcaption></figure><p>I know that after you read <a href="https://medium.com/@garc1a0scar/getting-started-with-apache-airflow-b8a8bba82428?source=your_stories_page----------------------------------------">Getting started with Apache Airflow</a> and <a href="https://medium.com/@garc1a0scar/understanding-better-the-dags-and-operators-concepts-99187d397857?source=your_stories_page----------------------------------------">Understanding better the DAGs (And Operators) concepts</a> you thought that it would be awesome if you could simplify your code to only construct “one” operator that can receive different parameters in every run to do different things. Well… good news! You can do it.</p><p>In this article we will learn the logic behind constructing a dynamic DAG and the possible use cases of them.</p><h3>Use case</h3><p>Imagine that you to need to send different commands to the operative system with a Bash Operator; in a normal case, we would create one Bash Operator with every command we need. We would end up with something like this:</p><blockquote>start = DummyOperator(<br> task_id=’start_wf’<br> )</blockquote><blockquote>end= DummyOperator(<br> task_id=’end_wf’<br> )</blockquote><blockquote>task1 = BashOperator(<br> task_id=’task1’,<br> bash_command=’&lt;command 1&gt;’</blockquote><blockquote>task2 = BashOperator(<br> task_id=’task2’,<br> bash_command=’&lt;command 2&gt;’</blockquote><blockquote>task3 = BashOperator(<br> task_id=’task3’,<br> bash_command=’&lt;command 3&gt;’</blockquote><blockquote>start&gt;&gt;task1&gt;&gt;task2&gt;&gt;task3&gt;&gt;end</blockquote><p>As we can see, we are “repeating” the construction of the<em> Bash Operators</em> and the DAG could become huge if we have a lot of commands to execute. Also, if we need to change one or more commands or if we need to add others, we would have to go directly to our code, modify it, add tasks, change the flow segment code and upload the DAG again.</p><h3>Solution</h3><p>Look at the following DAG code, which we will analyze later:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e9ae4d45932238b44e8d8b338200f87d/href">https://medium.com/media/e9ae4d45932238b44e8d8b338200f87d/href</a></iframe><p>First, we have our classic modules/libraries import section. We have an additional library called pandas that will allow us to handle the file content (this is an optional library; if you are more comfortable with another library or function, you can use it).</p><p>Then, in line 7, we read our file called “dynamic_dag_tutorial.conf” that has a column header: <em>command </em>and 4 records of respective commands.</p><blockquote>command<br>echo 1<br>echo 2<br>echo 3<br>echo 4</blockquote><p>Inside of the DAG construction, we have two dummy operators that will act as start and end of the workflow, respectively.</p><p>And below them is where magic happens. As we can see, we are now using a for loop to create as many operators (and not creating one for every command) as commands we have in our configuration (parameters) file. The command will be the value in the <em>command </em>column, for every iteration (index) in our file.</p><p>Our DAG will look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/969/1*h7eUnS_lZHjTjaLALubTkg.png" /></figure><p>Finally, we also need to construct our DAG flow dynamically. The logic in last section is:</p><ol><li>If we only have one <em>task </em>(command) to execute, we can only link the start with the <em>command_0</em> task and then with the end.</li><li>If we have two <em>tasks </em>or more (commands) to execute, for the first iteration we need to link the start with the <em>first task </em>and the <em>first task</em> with the <em>second task</em>.</li><li>Then, for the following indexes (except for the last one) we need to link the actual <em>task </em>to the next <em>task </em>in line. If we only have two tasks, this condition is never fulfilled.</li><li>Finally, for the last index, we need to link the last <em>task </em>to the end (else condition).</li></ol><p>If we execute our DAG, we will see that we got the expected output for every task.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MKk58O5EBN-YX9hFnwi_NA.png" /><figcaption>command_0 task output.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A-X_FDqwHI0T3cOLFZ4DPw.png" /><figcaption>command_3 task output</figcaption></figure><p>With all this, if we need to change the command of one of the tasks or add more commands, we only need to change the configuration file content and the DAG will be built dynamically with all the required tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/155/1*4ZHhbKrLgkMh1tDih_W81g.png" /><figcaption>Change for dynamic_dag_tutorial.conf file</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wsRqyPt0dSsY_4V_YHoirg.png" /><figcaption>DAG graphical view with changes in configuration file</figcaption></figure><h3>Conclusion</h3><p>As we could see throughout the article, it’s possible to construct dynamically <em>tasks</em> for our DAGs.</p><p>With this, we can simplify our code by reducing the written lines, and also we make our code being capable of handling possible future changes. As we could see, we only need to construct our DAG once and then, we only need to make changes in our configuration file.</p><p>We used a very simple example with a <em>Bash Operator</em> but you can come up new solutions, using other operators: <em>Python Operator, Snowflake Operator, Big Query Operators, MySQL Operators, S3 Operators, etc.</em></p><p>This is the solution I came to with my experience, but feel free to find improvements in your own solutions. I will be happy to read your comments and come up with ever simpler solutions.</p><h3>Until next code!</h3><h3>Os😉</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=685a8193703c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding better the DAGs (And Operators) concepts.]]></title>
            <link>https://medium.com/@garc1a0scar/understanding-better-the-dags-and-operators-concepts-99187d397857?source=rss-8b92eced7bba------2</link>
            <guid isPermaLink="false">https://medium.com/p/99187d397857</guid>
            <category><![CDATA[data-parallelism]]></category>
            <category><![CDATA[xcom]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[trigger]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Oscar García]]></dc:creator>
            <pubDate>Wed, 23 Feb 2022 23:06:13 GMT</pubDate>
            <atom:updated>2022-02-28T20:17:58.604Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/350/0*NSzMdAXF9-KFW3jp.png" /><figcaption>Apache Airflow logo</figcaption></figure><p>After understanding in a general way what is Apache Airflow, what are the DAGs and what are the operators, described in our last article “<a href="https://medium.com/@garc1a0scar/getting-started-with-apache-airflow-b8a8bba82428">Getting Started with apache Airflow</a>”, we will understand in a deeper way what a DAG and also the operators (tasks) that make up the DAG, can do.</p><h3>Understanding the DAGs</h3><p>Imagine that you have the following code, and let’s break down it:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b09b6e75629d96cc54e0653e7900f6dc/href">https://medium.com/media/b09b6e75629d96cc54e0653e7900f6dc/href</a></iframe><p>So, first, we have our library import section:</p><blockquote>from airflow import DAG<br>from airflow.operators.bash_operator import BashOperator<br>from airflow.utils.dates import days_ago<br>import pendulum<br>from datetime import datetime</blockquote><p>These are the libraries/operators that we will need for the execution of our DAG. Take in account that we need always to import the DAG library.</p><p>Then we have our next section:</p><blockquote>local_tz=pendulum.timezone(“America/Mexico_City”)<br>default_args = { “owner”: “airflow”, <br>“email”: [“airflow@airflow.com”], <br>“email_on_failure”: False, <br>“email_on_retry”: False, <br>“retries”: 1}</blockquote><p>This section is about our DAG’s default arguments/configuration. As you can see, we can define a dictionary with the default arguments that we can pass it to our DAG construction.</p><p><strong>Local_tz </strong>is a variable that obtain the timezone of Mexico City.</p><p><strong>Owner: </strong>it is the owner of the DAG. The owner is shown in the User Interface.</p><p><strong>Start date </strong>shows us that DAG will start on February 22, 2022 at 4a.m. of Mexico City time.</p><p><strong>Email </strong>is the address that will use to email on failure or email on retry.</p><p><strong>Email_on_failure </strong>with value False, means that it will not send any email if DAG fails. Change it to True if you want to send failure emails. Take in account that you will need to configure SMTP service to achieve this.</p><p><strong>Email_on_retry </strong>with value False, means that it will not send any email if DAG do a retry. Change it to True if you want to send failure emails. Take in account that you will need to configure SMTP service to achieve this.</p><p><strong>Retries </strong>is the number of times that the DAG will attempt to retry the execution if DAG fails.</p><p>After that, we have our DAG construction and operators definition:</p><blockquote>with DAG(“tutorial”, default_args=default_args,<br> start_date= datetime(2022, 2, 23, 4,00, tzinfo=local_tz),<br> schedule_interval=’0 4 * * *’) as dag:</blockquote><blockquote>t1 = BashOperator(task_id=”print_date”, bash_command=”date”)</blockquote><blockquote>t2 = BashOperator(task_id=”sleep”, bash_command=”sleep 5&quot;)</blockquote><p>In this section we define our dag in the first lines with the <strong>default_args </strong>built in previous section and with the schedule interval defined at 4 a.m. daily. For more information about scheduling and start dates, view <a href="https://airflow.apache.org/docs/apache-airflow/stable/dag-run.html">DAG Runs documentation</a>¹.</p><p>As we can see, we have two tasks called <em>t1 and </em>t2; so finally we have our flow construction (relationships and direction):</p><blockquote>t1&gt;&gt;t2</blockquote><p>There are two ways to set this relationships:</p><ul><li><strong>Bitshift operator:</strong> t1&gt;&gt; t2 (This means t1 will run first and t2 will run after t1)</li><li><strong>Set downstream function:</strong> t1.set_downstream(t2) (This means the same as above; this way is considered deprecated and the first one is preferred)</li></ul><p>Also you can do the opposite with both ways, i.e.:</p><ul><li><strong>Bitshift operator:</strong> t1&lt;&lt; t2 (This means t2 will run first and t1 will run after t2)</li><li><strong>Set upstream function:</strong> t1.set_upstream(t2) (This means the same as above; this way is considered deprecated and the first one is preferred)</li></ul><p>With all that configured, our DAG will look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/314/1*W0h30eqiFno3N0gcSNtRaA.png" /><figcaption>DAG graphical representation in UI</figcaption></figure><p>Note: The flow construction can be in several lines if we have more tasks, i.e.:</p><blockquote>t1&gt;&gt;t2<br>t2&gt;&gt;t3<br>t3&gt;&gt;t4</blockquote><h4>Running tasks in parallel</h4><p>There is also a way to execute tasks in parallel. Let’s suppose that we have a third task called t3 that also make a sleep and that we need to run t1 first and then t2 and t3 in parallel. The way to achieve this is to enclose the tasks that will run in parallel in brackets [].</p><blockquote>t1&gt;&gt;[t2,t3]</blockquote><p>And our DAG will look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/401/1*bUlxc8ZoMX4Yr-9bm3CFMA.png" /><figcaption>DAG graphical representation of tasks in parallel</figcaption></figure><p>After that, we can continue our DAG as we want:</p><blockquote>t1&gt;&gt;[t2,t3]&gt;&gt;t4</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/459/1*SZWKN62arrTcdiygKLJuPA.png" /></figure><p>Note: it is not possible to directly link a group of tasks running in parallel to another group running in parallel. You need to link each of the tasks in the first group to the other group.</p><p>I.e. You can’t do this:</p><blockquote>t1&gt;&gt;[t2,t3]&gt;&gt;[t4,t5]</blockquote><p>In order, you have to do this:</p><blockquote>t1&gt;&gt;[t2,t3]<br>t2&gt;&gt;[t4,t5]<br>t3&gt;&gt;[t4,t5]<br>[t4,t5]&gt;&gt;t6</blockquote><p>And our DAG will look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*Pydk1iq1GinmI_LTibRvnQ.png" /><figcaption>DAG graphical representation of two continuous groups of tasks in parallel</figcaption></figure><h3>Understanding the Operators</h3><p>As we saw in Getting Started with Apache Airflow, each one of the nodes (tasks) of a DAG is represented by an operator. An operator provides integration with some other service like MySQLOperator, BigQueryOperator, SnowflakeOperator, GlueJobOperator (AWS), etc.</p><p>Each operator have their own parameters, but there are two that they all share: <strong>task_id </strong>that is the name that the UI will show for the task associated and <strong>trigger_rule </strong>which we will see in more detail below.</p><h4>Understanding the Trigger rules</h4><p>The trigger rule, as its name suggests, is an argument that defines under which circumstances the task should be executed. The default for <strong>trigger_rule</strong> argument is <strong>all_success </strong>that means that the task should be executed only when all upstream tasks have been successfully executed. The possible values are listed below:</p><ul><li><strong>all_success</strong>: (default) all parents have succeeded</li><li><strong>all_failed</strong>: all parents are in a failed or upstream_failed state</li><li><strong>all_done</strong>: all parents are done with their execution</li><li><strong>one_failed</strong>: fires as soon as at least one parent has failed, it does not wait for all parents to be done.</li><li><strong>one_success</strong>: fires as soon as at least one parent succeeds, it does not wait for all parents to be done</li><li><strong>none_failed</strong>: all parents have not failed (failed or upstream_failed) i.e. all parents have succeeded or been skipped</li><li><strong>dummy</strong>: dependencies are just for show, trigger at will.</li></ul><p>With this, we could have more control between our tasks and also we can “simulate” a conditional branch, based on status for any of them. See the example below:</p><blockquote>with DAG(“test”, default_args=default_args,<br> start_date= days_ago(1)) as dag:<br> start = DummyOperator (task_id=”start”)<br> t1 = BashOperator(task_id=”print_date”, bash_command=”exit 123”)<br> t2 = BashOperator(task_id=”task_2&quot;, bash_command=”sleep 5&quot;, trigger_rule=”all_failed”)<br> t3 = BashOperator(task_id=”task_3&quot;, bash_command=”sleep 10&quot;, trigger_rule=”all_success”)<br> end = DummyOperator (task_id=”end”, trigger_rule=”one_success”)</blockquote><blockquote>start&gt;&gt;t1&gt;&gt;t2<br>t1&gt;&gt;t3<br>[t2,t3]&gt;&gt;end</blockquote><p>As we can see, in task t1, a bash command “exit 123” is executed that will lead to an error. Of course, this is a non-sense example but it will demonstrate how we can use trigger rules. Our dag will look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/1*CWuuUXVwo-u-MTpoA9jTBg.png" /></figure><p>After executing the DAG, the result is the following.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/569/1*HV4njNfI_iKe7RbmgbUjOQ.png" /></figure><p>As you can see, “print_date” leads to an error and task_3 is not executed, since it has defined a trigger_rule of “all_success”; task 2 is executed successfully, since it has defined a trigger_rule of “all_failed”; and finally, end task is sucessfully executed, since it has defined a trigger_rule of “one_success” (task_2). Let’s see now what happens if print_date doesn’t fail.</p><blockquote>with DAG(“test”, default_args=default_args,<br> start_date= days_ago(1)) as dag:<br>start = DummyOperator (task_id=”start”)<br>t1 = BashOperator(task_id=”print_date”, bash_command=”echo x”)<br>t2 = BashOperator(task_id=”task_2&quot;, bash_command=”sleep 5&quot;, trigger_rule=”all_failed”)<br>t3 = BashOperator(task_id=”task_3&quot;, bash_command=”sleep 10&quot;, trigger_rule=”all_success”)<br>end = DummyOperator (task_id=”end”, trigger_rule=”one_success”)</blockquote><blockquote>start&gt;&gt;t1&gt;&gt;t2<br>t1&gt;&gt;t3<br>[t2,t3]&gt;&gt;end</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/1*s9Mi7WDIB_jqksR9XUMY2A.png" /></figure><p>We can see that now task_2 is not executed and task_3 is executed after print_date.</p><p>For additional information, check <a href="https://airflow.apache.org/docs/apache-airflow/stable/dag-run.html">Airflow documentation</a>.</p><h3>Understanding the Sensors</h3><p>Sensors are special types of operators whose purpose is to wait on some external or internal trigger. These are commonly used to trigger some or all of the DAG, based on the occurrence of some external event.</p><p>There are also many operators that you can use within Apache Airflow to wait for http, s3, sql, tasks in other DAGs, etc.</p><p>Sensors have two different modes of running:</p><ol><li>poke (this is the default): sensor takes up a worker for its entire runtime.</li><li>reschedule: sensor takes up a worker slot only when it is checking, and sleeps for a set duration between checks.</li></ol><p>There is an additional mode called <strong>smart sensor,</strong> but its a deprecated feature in Airflow&gt;2.4.0.</p><p>For additional information about sensors, check <a href="https://airflow.apache.org/docs/apache-airflow/stable/concepts/sensors.html">Sensors documentation</a>.</p><h3>Understanding Variables</h3><p>It is possible in Airflow to define variables and retrieve them to execute some code in your DAGs. We can define any variable, with any name, inside the Web User Interface.</p><p>Follow the next steps to define a variable:</p><ol><li>Go to the web server interface.</li><li>Go to Admin Section.</li><li>Click on variables.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*EMNRp7WR5NjxJrOHFO7l2w.png" /></figure><p>4. Click on +symbol to add a new variable.</p><p>5. On the next screen, we can define the variable name and value. For example purposes a variable called “TEST” with value “OSCAR” was defined.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/703/1*6gNBMwBVXBcDvCBFsD7CEw.png" /><figcaption>Example of variable definition.</figcaption></figure><p>Once we have define the “TEST” variable, check out the following code that we are also going to break down.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ca23526bc1260eb820b629662c606654/href">https://medium.com/media/ca23526bc1260eb820b629662c606654/href</a></iframe><p>As always, we have first our libraries/modules importing section:</p><blockquote>from airflow import DAG<br>from airflow.operators.bash_operator import BashOperator<br>from airflow.utils.dates import days_ago<br>from airflow.models import Variable</blockquote><p>As you can see, there is a new import called <em>Variable</em> from module airflow.models. This library allows us to import any variable previously defined in the web server interface.</p><p>Then, we have our variable definition section:</p><blockquote>CLIENT_NAME = Variable.get(‘TEST’)<br>echo_cmd = “echo “+ CLIENT_NAME</blockquote><p>In this section we retrieve the TEST variable value with the command <em>Variable.get</em> and then we build the echo_cmd that we will need in our operator.</p><p>Finally, we have our DAG and flow construction sections:</p><blockquote>default_args = {<br> “owner”: “airflow”,<br> “depends_on_past”: False,<br> “email”: [“<a href="/service/mailto:airflow@airflow.com">airflow@airflow.com</a>”],<br> “email_on_failure”: False,<br> “email_on_retry”: False,<br> “retries”: 1<br>}</blockquote><blockquote>with DAG(“tutorial_variable”, default_args=default_args,<br> start_date = days_ago(1)) as dag:</blockquote><blockquote>t1 = BashOperator(task_id=”echo_name”, bash_command=echo_cmd)</blockquote><blockquote>t1</blockquote><p>As you can see, t1 is a <em>BashOperator </em>task that send the command “echo_cmd” previously defined. If we check the execution log, we can see that the command was perfectly exectuted and with the desired output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rao6tpP6S76vaKybp0w8tg.png" /><figcaption>Execution log for DAG using variables.</figcaption></figure><p>If then we need to change the CLIENT NAME, we don’t have to change directly the code, but only the variable value.</p><h3>Understanding Xcom</h3><p>It is very common in programming that you define a function and need to pass its result to another function to do another thing. In Apache Airflow is also probable that you need to pass a value from one task (operator) to another. So, I have good news for you: Apache Airflow is totally capable to do this with XCom (cross-communications).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fd9c185cd6da7fa9af0f107717947742/href">https://medium.com/media/fd9c185cd6da7fa9af0f107717947742/href</a></iframe><p>Let’s break it down one more time. First, we have our libraries/modules import section:</p><blockquote>from airflow import DAG<br>from airflow.utils.dates import days_ago<br>from airflow.models import Variable<br>from airflow.operators.python_operator import PythonOperator</blockquote><p>Then, we have our variables definition section:</p><blockquote>NAME_CLIENT = Variable.get(“TEST”)</blockquote><p>After that, we have a functions definition section that looks like this:</p><blockquote>def build_greeting(ti):<br> greeting = ‘Hello, ‘+NAME_CLIENT+”. Nice to meet you”<br> ti.xcom_push(key=’build_greeting’, value=greeting)</blockquote><blockquote>def build_email(ti):<br> greeting = ti.xcom_pull(key=’build_greeting’)<br> email = greeting + ‘. I would like to invite you to my party.’<br> return email</blockquote><p>As we can see, in this section we defined two functions: 1) Function that build a generic greeting, using the value of NAME_CLIENT variable; 2) Function that do a pull of previous function and builds an email body.</p><p>Finally, we have our DAG and flow construction, as in previous examples:</p><blockquote>with DAG(dag_id=”tutorial_xcom1&quot;<br> , schedule_interval=None<br> , catchup=False<br> , start_date=days_ago(1)) as dag:</blockquote><blockquote>build_greeting = PythonOperator(task_id=”build_greeting”, <br> python_callable=build_greeting)</blockquote><blockquote>build_email = PythonOperator(task_id=”build_email”, <br> python_callable=build_email)</blockquote><blockquote>build_greeting&gt;&gt;build_email</blockquote><p>In this DAG example, we defined two <em>PythonOperator </em>tasks that call the defined functions. Our DAG will look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/398/1*wNucliTR3bsm6DeV7TgvJw.png" /><figcaption>Example of DAG with XCom</figcaption></figure><p>After executing the DAG, we can go to Admin section in Airflow menu, and then to XCom.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/291/1*2lMscM4hZZZ4MBwTbRWpLA.png" /></figure><p>Here, we will see that 2 records were generated:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yJsEl2sv00Ur582h2f5grw.png" /><figcaption>Records generated with XCom</figcaption></figure><p>The first one, is generated by the <strong>build_greeting </strong>function, which uses the variable to build the greeting; the second one, is generated by the <strong>build_email </strong>function, which uses the result of the first function to build and email body.</p><p><strong>Pro tip: </strong>If you were curious, you might have noticed that the second function still generated an XCom record, without using the xcom_push function. This is because any function that has a “return” instruction, generates an XCom register that can be consumed by another function and, although it is an easier way to handle it, the advantage of using xcom_push is that you can define the name of the key that allows a better control.</p><h3>Understanding Connections and Hooks</h3><p>There are two final concepts that I would like to mention:</p><ol><li><strong>Connections</strong>: are the objects that store credentials that are used to communicate to external systems. They have fields/parameters like hostname, username, password, port to connect to, etc. It also has a unique name, called <strong>conn_id, </strong>that is used to reference the set of parameters previously described (you can access them directly from your code or from a Hook; e.g. BaseHook). You can define a connection via UI, within the Admin section or via CLI.</li><li><strong>Hooks</strong>: Are high-level interfaces that allow us to communicate with external platforms without having to write low-level code or to use special libraries. They integrate with <strong>connections </strong>to retrieve all parameters needed to initiate a communication channel.</li></ol><p>In conclusion, the flow is something like this: <strong>DAGs </strong>have <strong>operators </strong>that, in turn, call <strong>hooks</strong>, which create an interface between our code and <strong>external platforms</strong>, using parameters of the defined <strong>connections</strong>.</p><p>With all the mentioned above, we can confirm that Apache Airflow is capable of a lot of things and that can connect to many systems in different ways. In this article we use many simple examples (nonsense examples) but I hope it helps to understand how all Apache Airflow components work and how to use them.</p><p>I’m sure that this is an excellent way to start and that you will create more complex things that I would love to see in a future.</p><h4>References</h4><p>[1] Airflow. <em>DAG Runs. </em><a href="https://airflow.apache.org/docs/apache-airflow/stable/dag-run.html">https://airflow.apache.org/docs/apache-airflow/stable/dag-run.html</a></p><p>[2] Airflow. <em>Concepts</em>. <a href="https://airflow.apache.org/docs/apache-airflow/1.10.5/concepts.html?highlight=trigger%20rule#:~:text=a%3E-,Trigger%20Rules,generated%20task%20get%20triggered">https://airflow.apache.org/docs/apache-airflow/1.10.5/concepts.html?highlight=trigger%20rule#:~:text=a%3E-,Trigger%20Rules,generated%20task%20get%20triggered</a>.</p><p>[3] Airflow. <em>Xcom. </em><a href="https://airflow.apache.org/docs/apache-airflow/stable/concepts/xcoms.html">https://airflow.apache.org/docs/apache-airflow/stable/concepts/xcoms.html</a></p><p>[4] Airflow. <em>Sensors. </em><a href="https://airflow.apache.org/docs/apache-airflow/stable/concepts/sensors.html">https://airflow.apache.org/docs/apache-airflow/stable/concepts/sensors.html</a></p><p>[5] Airflow. <em>Connections and Hooks.</em><a href="https://airflow.apache.org/docs/apache-airflow/stable/concepts/connections.html">https://airflow.apache.org/docs/apache-airflow/stable/concepts/connections.html</a></p><h3>Until next code!</h3><h3>Os😉</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=99187d397857" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to start with Apache Airflow in a MWAA look-alike docker (Mac OS)]]></title>
            <link>https://medium.com/@garc1a0scar/how-to-start-with-apache-airflow-in-a-mwaa-look-alike-docker-mac-os-d908fa0276d?source=rss-8b92eced7bba------2</link>
            <guid isPermaLink="false">https://medium.com/p/d908fa0276d</guid>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[mac]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[mwaa]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Oscar García]]></dc:creator>
            <pubDate>Tue, 22 Feb 2022 22:07:48 GMT</pubDate>
            <atom:updated>2022-05-16T19:06:39.211Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gpFb4F-yaHWVWCuPhiCeHg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/225/1*g6Y88dIQ99yG2o48hddh_Q.png" /><figcaption>Docker and MWAA logos</figcaption></figure><p>If you’ve read the article “<a href="https://medium.com/@garc1a0scar/getting-started-with-apache-airflow-b8a8bba82428">Getting Started with Apache Airflow</a>” you can see that Apache Airflow is a powerful tool that allows us to manage, monitor, plan<em> </em>and schedule our workflows.</p><p>Also, in the article <a href="https://medium.com/@garc1a0scar/how-to-start-with-apache-airflow-in-docker-windows-902674ad1bbe">“How to start with Apache Airflow in Docker (Windows)”</a> we could see that is very easy to start an Apache Airflow local environment in our computers to navigate through the UI, develop our first dags or test another ones.</p><p>Nowadays, there are several cloud providers (like Azure, Google or Amazon) that have these services (and others like storage, virtual machines, etc.) within their platforms and offer them to clients as managed services (SaaS).</p><p>With these services, providers try to eliminate the need of maintenance and management on the client’s side, so that the latter can focus on “what is truly important” which is to develop their solutions.</p><p>In this article we will focus on how to start an Apache Airflow instance in our Mac that has a configuration very similar to what we will face within Amazon Web Services service, called MWAA (Managed Workflows for Apache Airflow).</p><h4>Prerequisites</h4><ol><li>Install Docker Desktop.</li></ol><h4>Get started</h4><p>Follow this steps in order to start with your environment in docker:</p><ol><li>Create a folder in your computer, called airflow in the path of your election. Personally, I like to create all my folders in /Users/my_user/.</li><li>Open a terminal and go to the created folder.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*bbL5AsZmFc8cKH4xsuSxTQ.png" /><figcaption>Terminal in airflow folder</figcaption></figure><p>3. When you are inside the above path, you will need to execute the next command:</p><blockquote>git clone <a href="https://github.com/aws/aws-mwaa-local-runner.git">https://github.com/aws/aws-mwaa-local-runner.git</a></blockquote><p>4. The terminal will start to download all the repository that you can find in <a href="https://github.com/aws/aws-mwaa-local-runner">https://github.com/aws/aws-mwaa-local-runner</a></p><p>5. Once finished, you will see that a folder was created inside the airflow folder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/603/1*le0Sj-h4OLFheJMfjvmabA.png" /><figcaption>Folder created after cloning the repository</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*qWK6YBG9ESR3IRJPlh3KPA.png" /><figcaption>Folder content after cloning the repository.</figcaption></figure><p>6. At this time, you could build the docker image and start the instance, but we will make some adjustments before.</p><p>7. Create a plugins folder that will hold your posible custom plugins at root folder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/1*EYILiKmKTPQYiV4wZqi7Bw.png" /><figcaption>Folder content after creating plugins folder</figcaption></figure><p>8. Go to dags folder, and you will find a tutorial dag and a requirements.txt file. Open the requirements file and include the additional packages that you will need to install. Take into account that the environment is a 2.0.2 version of airflow and therefore, there are some restrictions regarding package versions. You can check the constraints <a href="https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-3.6.txt">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*tmw3J3_MX8-jqOykPdTgPg.png" /><figcaption>Example of requriments.txt file to install additional packages.</figcaption></figure><p>9. Optional: Go to docker folder and open the Dockerfile. Here, you will want to include environment variables or the installation of certain initial libraries. Note, that with a docker environment, you may not be able to define environment variables later. This shouldn’t happen like this, within the managed service, which we’ll cover in another article.</p><p>E.g. I installed library libaio and define my LD_LIBRARY_PATH variable in order tu use my Oracle instant client.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/664/1*B813YbP-9iTVSHoAuoM7fQ.png" /></figure><p>10. Last but not least, inside docker folder, go to script folder, open entrypoint.sh and do the following:</p><p>a) Remove line 7 that corresponds to global default of Fernet Key:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f7qNLqbQN3Y705kO_zrIuw.png" /></figure><p>b) In line 14, below Load DAGs examples section, add the following code:</p><blockquote>if grep -q &#39;$FERNET_KEY&#39; /usr/local/airflow/airflow.cfg; then<br> echo &quot;fernet_key not set in airflow.cfg!&quot;<br> sed -i &#39;s/$FERNET_KEY/TEMP_KEY/&#39; /usr/local/airflow/airflow.cfg<br> : &quot;${AIRFLOW__CORE__FERNET_KEY:=${FERNET_KEY_GEN:=$(python -c &quot;from cryptography.fernet import Fernet; FERNET_KEY = Fernet.generate_key().decode(); print(FERNET_KEY)&quot;)}}&quot;<br> sed -i &quot;s/TEMP_KEY/$AIRFLOW__CORE__FERNET_KEY/&quot; /usr/local/airflow/airflow.cfg<br> echo &quot;fernet_key is now set in airflow.cfg&quot;<br> else<br> echo &quot;fernet_key is already set in airflow.cfg&quot;<br>fi</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FeC9ctTIUEPPFAnrL4GzCg.png" /></figure><p>This is a known bug where shutting down your instance or computer will cause that previously configured connections no longer work correctly.²</p><p>11. With all these steps followed correctly, we can proceed to build our image and start our instance. Go again to your terminal and go to path /airflow/aws-mwaa-local-runner.</p><p>12. Run the following command to build the docker image</p><blockquote>./mwaa-local-env build-image</blockquote><p>13. After completion, run the following command to start the instance.</p><blockquote>./mwaa-local-env start</blockquote><p>14. Last step will last around 5 minutes but you will see how every container and service is started. When you see a text like the following in your terminal (with the Listening at legend), you can continue with the process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*yp7G0HSy9wBdlVxAxeJXTg.png" /></figure><p>15. Go to your web browser and go to localhost:8080. It will open the login screen. By default, a username and password is created:</p><blockquote>Username: admin</blockquote><blockquote>Password: test</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*HvR88PsFBDtPekoIQxz1rQ.png" /><figcaption>Airflow UI login screen</figcaption></figure><p>16. Enter the credentials and click on sign in. Your UI is ready to start your dags development. It will look like something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JS6MbMmDnLof9CHWpEZu9g.png" /><figcaption>Airflow UI</figcaption></figure><p>17: After that, you can put your DAGs in the dags folder inside /airflow/aws-mwaa-local-runner/, your custom plugins in your plugins folder and the instance will load them automatically to your UI. Take in consideration that DAGs upload takes a while so do not panic if you don’t see them immediately.</p><p>If it’s been a long time and you still don’t see them, try refreshing the page or starting the instance again with step 13 command.</p><h4>Additional information</h4><ol><li><strong>Never stop your instance from Docker Desktop app</strong>. It will cause paths (dags, plugins, etc.) to be recalculated/re-built and no longer work correctly.</li><li>If you need to install additional packages, you can always include them in requirements file of step 8, and they will install automatically when you restart your instance. You can also install them locally, as mentioned in <a href="https://github.com/aws/aws-mwaa-local-runner">https://github.com/aws/aws-mwaa-local-runner</a>¹ and they will work correctly.</li><li>The installation should be very similar in windows, but using Windows Subsystem for Linux (WSL). Check <a href="https://github.com/aws/aws-mwaa-local-runner">https://github.com/aws/aws-mwaa-local-runner</a>¹</li></ol><h4>References</h4><p>[1] GitHub (May 26,2021). <em>AWS Local Runner</em>: <a href="https://github.com/aws/aws-mwaa-local-runner">https://github.com/aws/aws-mwaa-local-runner</a></p><p>[2] Github (June 28, 2021).<em> Add logic for persisting fernet key to airflow.cfg </em>: <a href="https://github.com/aws/aws-mwaa-local-runner/pull/28">https://github.com/aws/aws-mwaa-local-runner/pull/28</a></p><h3>Until next code!</h3><h4>Os😉</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d908fa0276d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to start with Apache Airflow in Docker (Windows)]]></title>
            <link>https://medium.com/@garc1a0scar/how-to-start-with-apache-airflow-in-docker-windows-902674ad1bbe?source=rss-8b92eced7bba------2</link>
            <guid isPermaLink="false">https://medium.com/p/902674ad1bbe</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[airflow-2]]></category>
            <category><![CDATA[airflow]]></category>
            <dc:creator><![CDATA[Oscar García]]></dc:creator>
            <pubDate>Tue, 22 Feb 2022 22:05:38 GMT</pubDate>
            <atom:updated>2022-02-23T23:18:21.261Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gpFb4F-yaHWVWCuPhiCeHg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*nYwEMPPkz-vUxNtDfQ8VJg.png" /><figcaption>Docker and Apache Airflow logos</figcaption></figure><p>As we mentioned in the article <a href="https://medium.com/@garc1a0scar/getting-started-with-apache-airflow-b8a8bba82428">“Getting Started with Apache Airflow”</a>, in general terms, Apache Airflow is an open-source tool that allow us to manage, monitor, plan<em> </em>and schedule workflows that is normally used as a workflow (services) orchestrator.</p><p>In this article, we will learn how to start an instance of Apache Airflow 2.0.2 (this version is the most recent in managed services nowadays)in Docker, which if you are already familiar with it, you know that is excellent for developing and testing our solutions without affecting a managed service or a production environment.</p><h4>Prerequisistes:</h4><ol><li>Install Docker Desktop in your computer.</li></ol><h4><strong>Get Started</strong></h4><p>Follow this steps in order to start with your testing Airflow environment in docker:</p><ol><li>The first step is to download Docker Desktop from the <a href="https://www.docker.com/products/docker-desktop">official website</a>. For this article, I installed 4.4.4 version.</li><li>After installing Docker Desktop, we need to download a docker-compose.yaml file that you can also find <a href="https://airflow.apache.org/docs/apache-airflow/2.0.2/docker-compose.yaml">here</a>.</li><li>Now that we have both files, we need to create our airflow directory. Go to the following path: C:/Users/&lt;your_user&gt;/. Inside of that directory, create a folder called docker and inside of docker create another folder called airflow.</li><li>Now that we have our airflow folder, we must do the following: a)Create three folders called dags, plugins and logs respectively; b) Move our YAML to that directory.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/1*d6tpmvJYmhc4ARhPvaLVcg.png" /><figcaption>Your directory will look like this</figcaption></figure><p>5. Now we are ready to start our instance of Airflow in docker. We need to open a PowerShell window and go to the directory above.</p><p>6. Then, we need to run the following commands:</p><blockquote>docker-compose up airflow-init</blockquote><blockquote>docker-compose up</blockquote><p>Note: After second command, a window will keep running some code. We can close this window without any problem.</p><p>7. After that, we can go to our Docker Desktop app and we will see that a container named airflow was created inside section Containers/Apps. This container will have 7 sub-containers inside.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*857Z6SjYdWyr-oPY8JPKeQ.png" /><figcaption>Your container will look like this</figcaption></figure><p>8. With this, we have our Apache Airflow instance completely ready to start developing our DAGS.</p><p>9. Go to localhost:8080, login with user “airflow” and password “airflow” and start coding.</p><p>Note: You can put all your DAGs in the dags folder and all your plugins in plugins folder both created in step 4.</p><h4>References</h4><p>[1] Airflow. <em>Running Airflow in Docker</em>. <a href="https://airflow.apache.org/docs/apache-airflow/2.0.2/start/docker.html">https://airflow.apache.org/docs/apache-airflow/2.0.2/start/docker.html</a></p><h3>Until next code!</h3><h4>Os😉</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=902674ad1bbe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting started with Apache Airflow]]></title>
            <link>https://medium.com/@garc1a0scar/getting-started-with-apache-airflow-b8a8bba82428?source=rss-8b92eced7bba------2</link>
            <guid isPermaLink="false">https://medium.com/p/b8a8bba82428</guid>
            <category><![CDATA[airflow-2]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Oscar García]]></dc:creator>
            <pubDate>Tue, 22 Feb 2022 22:03:32 GMT</pubDate>
            <atom:updated>2022-02-28T19:16:57.163Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*nYwEMPPkz-vUxNtDfQ8VJg.png" /><figcaption>Apache Airflow</figcaption></figure><h3>Have you ever wanted a tool that allows you to run, manage and monitor your processes from different platforms in one place? Well… you came to the right place.</h3><h4>What is Apache Airflow?</h4><p>Apache Airflow is an open-source project that was created in 2014 in Airbnb by Maxime Beauchemin, and published in June 2015.</p><p>Apache Airflow is a workflow manager tool that allows us to manage, monitor and schedule workflows, used as a service orchestrator. It is used to programmatically automate jobs by dividing them into subtasks.</p><p>The most common use cases are the automation of data ingestions, periodic maintenance actions and administration tasks, but the sky is the limit.</p><p>The main principles of Apache Airflow are:</p><ul><li>Scalable: It means that it is ready to scale to infinity if needed, since it uses a message queue to orchestrate an arbitrary number of workers.</li><li>Dynamic: Since pipelines are defined in Python, it allows dynamic pipeline generation (we will cover this topic in another article).</li><li>Extensible: It allows to define our own operators and extend libraries.</li><li>Elegant: Pipelines are lean and explicit. Parametrization is built into its core.</li></ul><p>The main object of Apache Airflow is called a DAG (Directed Acyclic Graph) that we will cover in detail in the next section.</p><h4>Apache Airflow Architecture²</h4><p>Apache Airflow has four main components:</p><ul><li><strong>Web server</strong> that hosts the API and the <strong>user interface</strong> and also manage requests.</li><li><strong>Scheduler </strong>in charge of interpreting, launching workers and distributing tasks among them with an executor to run and monitoring the tasks defined in the DAGs.</li><li><strong>Workers </strong>in charge of executing the tasks sent by the scheduler.</li><li>A <strong>database </strong>that acts as a backend in charge of storing metadata, users and executions.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/0*R5TB8_EhBQxylDCe.png" /><figcaption>Apache Airflow architecture</figcaption></figure><h4>What is a DAG?</h4><p>As we previously mentioned, a DAG is an acronym for Directed Acyclic Graph, but what does that really mean?</p><p>A DAG is a collection of tasks or jobs nodes (graphs) programmed through Python code that are connected by relationships and dependencies that should fulfill the following:</p><ul><li>Directed: It means that they should be one-way relationships. From start to end.</li><li>Acyclic: Means that they cannot loop, that is, the execution cannot return to a node that has already been executed.</li></ul><p><strong>Each of the DAG tasks represented as a node, is described with an operator</strong>. There are hundreds of predefined operators, but, as mentioned before, it is also possible to extend and create new operators if necessary. Airflow also provides communication interfaces called Hooks to connect to other platforms and external databases such as Oracle, MySQl, SQL Server, Snowflake, BigQuery, HDFS, Apache Hive, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*hSW52QvIQ0hvON3AZZc_nA.png" /><figcaption>A simple DAG looks like this</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T7yPXyk-ddaSH1KTKIxujA.png" /><figcaption>Example of a more complex DAG representation</figcaption></figure><h4>Apache Airflow DAGs — Code main structure</h4><p>A simple DAG is made up of three main sections:</p><ul><li>Libraries/operators import: in this section, all needed libraries and operators should be explicitly imported. Lines from 1–6 in image below.</li><li>DAG and Operators definition: in this section, the operators (tasks) are defined. Take in consideration that they can be defined in any order. Lines from 14–22 in image below.</li><li>Workflow construction: in this section is defined the order in which operators (tasks) will be executed. Line 24 in image below.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JeJEzTiC39-olA_NhlSPYQ.png" /></figure><p>This code will result in a DAG that starts with “start_dag” task that corresponds to a Dummy Operator, followed by “task2” that corresponds to a Bash Operator that executes a “echo 1” command.</p><h4>Personal experience</h4><p>As you can see, Apache Airflow is a very powerful tool that has hundreds of operators (and developing more) to execute tasks in all the cutting-edge platforms. It has operators for AWS S3, BigQuery, Snowflake, SFTP, FTP, Oracle, MySql, SQL Server, Google Cloud Data Fusion, Tableau, Bash (OS), File, Email, Python (functions) and more.</p><p>In my experience, I have used Apache Airflow to:</p><ul><li>Execute data pipelines in Google Cloud Data Fusion to extract data from on-prem sources like MySql, Sql Server, Flat files, etc.</li><li>Load data warehouse tables in BigQuery with custom Data Vault 2.0 patterns, creating dynamic DAGs.</li><li>Load data warehouse tables in Snowflake with custom Data Vault 2.0 patterns, creating dynamic DAGs.</li><li>Upload query results from Oracle to AWS S3 buckets in CSV and Parquet format.</li><li>Upload files and creating “folder” structures in AWS S3 buckets from CSV files located on a SFTP server.</li><li>Retrieve marketing data from Facebook Ads, Google Ads and LinkedIn to load our data lake.</li><li>Create data delivery tables in BigQuery from custom SQL queries.</li><li>Refresh tableau workbooks/extracts.</li><li>Schedule all my workflows at specific times.</li><li>Execute user-defined python functions.</li><li>And more.</li></ul><p>With everything listed before, I’m sure that Airflow is an excellent tool that allows data engineers like me to have better control over the processes throughout the data processing flow and also to reduce additional costs that could be generated by acquiring specialized tools.</p><p>I do not regret for a moment that I started working with this great tool 1 year ago and I will certainly be happy to share my work with you in future articles. And I know that if you have different platforms that you need to consume or access for your data processing, you won’t regret it either, once you try it.</p><h4>References</h4><p>[1] Apache Airflow. <em>Apache Airflow. </em><a href="https://airflow.apache.org/">https://airflow.apache.org/</a></p><p>[2] Apache Airflow. <em>Architecture Overview. </em><a href="https://airflow.apache.org/docs/apache-airflow/stable/concepts/overview.html">https://airflow.apache.org/docs/apache-airflow/stable/concepts/overview.html</a></p><p>[3] Aprender BIG DATA (February 16, 2022). <em>¿Qué es Apache Airflow? Introducción</em>. <a href="https://aprenderbigdata.com/apache-airflow/">https://aprenderbigdata.com/apache-airflow/</a></p><h3>Until next code!</h3><h3>Os😉</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b8a8bba82428" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>