<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ishan Ojha on Medium]]></title>
        <description><![CDATA[Stories by Ishan Ojha on Medium]]></description>
        <link>https://medium.com/@CalvinTheExplorer?source=rss-f59b74919855------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*BRIj9Uq7c6EepNc6.</url>
            <title>Stories by Ishan Ojha on Medium</title>
            <link>https://medium.com/@CalvinTheExplorer?source=rss-f59b74919855------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 18:10:21 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@CalvinTheExplorer/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Exploring Airflow: Clarifying its Capabilities and Limitations]]></title>
            <link>https://medium.com/@CalvinTheExplorer/exploring-airflow-clarifying-its-capabilities-and-limitations-2410dc6b1a4b?source=rss-f59b74919855------2</link>
            <guid isPermaLink="false">https://medium.com/p/2410dc6b1a4b</guid>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Ishan Ojha]]></dc:creator>
            <pubDate>Sun, 22 Jan 2023 12:22:58 GMT</pubDate>
            <atom:updated>2023-01-22T13:08:45.304Z</atom:updated>
            <content:encoded><![CDATA[<p>Airflow has gained a lot of popularity in recent years in the field of data engineering and data science because of its Scalability,Monitoring and Alerting features and Rich User Interface.</p><h3><strong>What is Airflow ? When to use it?</strong></h3><p>Airflow is a workflow management/Orchestration system that allows to schedule ,monitor and execute complex workflow.</p><p>Consider a data warehouse solution, not for a single team but for across team where data is coming from different sources in different formats ,which is going to make it complex so frequency is going to increase as the job is periodic. Lets consider data coming into the warehouse is critical ,so in order to monitor the entire process we need a robust tool. Something that only scheduler cannot do.</p><p>So we need Error handling and monitoring and a framework where a entire process takes place in a particular manner and we can see our previous tasks.And once the complexity increases we need to scale more jobs in the future when it is required and traffic on the UI servers.That when we dont go for something like scheduler and go for an Orchestrator like Airflow.</p><h4>What is a Workflow management/Orchestration system?</h4><p>Imagine you’re in charge of managing a data pipeline, where you need to extract, transform and load data from various sources into a data warehouse. Workflow management/Orchestration system would <em>help you keep track of all the tasks and dependencies </em>that need to be done, like running data quality checks, scheduling data imports, and sending out notifications if something goes wrong. It can also be integrated with other systems like databases and messaging queues, making it a comprehensive solution for data pipeline management.</p><p>In a Nutshell ,Workflow management/Orchestration system is a system that helps you plan, execute, and monitor the steps of a process, ensuring that everything is done correctly and on schedule. It is particularly useful for managing and automating data pipelines and other complex workflows.</p><p><strong>How does Airflow <em>help you keep track of all the tasks and dependencies?</em></strong></p><p>Airflow uses DAGs,a directed acyclic graph (DAG) is a collection of all the tasks you want to run, organized in a way that reflects their relationships and dependencies. DAGs are defined using Python code and are used to represent the workflow of tasks that need to be executed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*PSjbnGkKO4BBkzEu.png" /><figcaption>Directed Acyclic Graph</figcaption></figure><p>In the picture above every node is task and one specialty about DAGs is <em>once a node is executed it cannot be run again unless the entire process executes .</em></p><h4>What Airflow is’nt?</h4><p>It is not a data stream solution,tasks can communicate but cannot share data between them.</p><p>It is not an ETL framework.Though with airflow we can manage ETL Orchestration.</p><h4>What are other alternatives of Airflow?</h4><p>There are other open-source and commercial workflow management tools available, like Apache Nifi, Luigi, and Jenkins etc.</p><p>For code/DAG related introduction wait for the next write-up……</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2410dc6b1a4b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>