<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tamara Janina Fingerlin on Medium]]></title>
        <description><![CDATA[Stories by Tamara Janina Fingerlin on Medium]]></description>
        <link>https://medium.com/@tjanif?source=rss-bf3ba0b8be12------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*X0TStFNcMf1YbkmOfz_GbA.jpeg</url>
            <title>Stories by Tamara Janina Fingerlin on Medium</title>
            <link>https://medium.com/@tjanif?source=rss-bf3ba0b8be12------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 15:07:04 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tjanif/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How Apache Airflow® is really used in 2026: What I learned analyzing data from 5818 engineers]]></title>
            <link>https://medium.com/@tjanif/how-apache-airflow-is-really-used-in-2026-what-i-learned-analyzing-data-from-5818-engineers-da0b313421f9?source=rss-bf3ba0b8be12------2</link>
            <guid isPermaLink="false">https://medium.com/p/da0b313421f9</guid>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[apache-airflow]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[airflow]]></category>
            <dc:creator><![CDATA[Tamara Janina Fingerlin]]></dc:creator>
            <pubDate>Wed, 18 Feb 2026 22:09:31 GMT</pubDate>
            <atom:updated>2026-02-18T22:09:31.154Z</atom:updated>
            <content:encoded><![CDATA[<p>Being the standard for data workflows, we all either interact with <a href="https://airflow.apache.org/">Apache Airflow®</a> ourselves, or know someone who does. From <a href="https://www.astronomer.io/blog/airflow-in-action-openai?utm_medium=organicsocial?utm_source=medium?utm_content=tamara">OpenAI</a>, to <a href="https://airflowsummit.org/sessions/2025/enabling-sql-testing-in-airflow-workflows-using-pydantic-types/">Anthropic</a>, to <a href="https://www.astronomer.io/blog/airflow-in-action-github?utm_medium=organicsocial?utm_source=medium?utm_content=tamara">GitHub</a>, <strong>Airflow is just everywhere (really! check for yourself using this </strong><a href="https://tjanif.github.io/airflow-survey-cities-map/"><strong>interactive map</strong></a><strong>)</strong>.</p><p>But how many Airflow users are already on <strong>Airflow 3</strong>? Are people really using it for <strong>AI pipelines</strong>? Can you make <strong>money</strong> with Airflow? Which is the most popular Airflow feature, and how many of you put over 2500 tasks into one single Dag? (I hope you are using <a href="https://www.astronomer.io/docs/learn/task-groups">task groups</a>!)</p><p>Once a year I am one of the lucky people who gets to answer these and many more questions by analyzing the responses to the yearly Apache Airflow® survey. Here are some of my favorite results! :)</p><h3>You are everywhere ❤</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PWLbi9a6uqudcxv2KFKzQA.png" /><figcaption>World map showing the global scale of Airflow. The 5818 responses came from 122 distinct countries.</figcaption></figure><p>The survey runs at the end of the year and for 2025 we had 5818 responses from <strong>122 individual countries</strong> (!). An amazing view into how engineers use Airflow today and plan to use it tomorrow and the biggest data engineering survey ever (to my knowledge).</p><p>The survey is also an opportunity for some friendly competition: Which city has the most Airflow users (that answered the Airflow survey)? This year it is <strong>Bangalore</strong>, which won before São Paulo and Mumbai. Congrats!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/813/1*MlZqTYpMA4RH8TW19d-5XQ.png" /><figcaption>Top 5 cities of Airflow users responding to the Airflow survey 2025: 1. Bangalore, 2. São Paulo, 3. Mumbai, 4. Colombo, 5. Chennai.</figcaption></figure><p>Want to see how many people from your city filled out the survey? Check out <a href="https://tjanif.github.io/airflow-survey-cities-map/">this interactive map</a>.</p><h3>Airflow 3 is 🚀</h3><p>The 2025 Airflow survey ran from Sep 15th to Nov 20th 2025. This was just half a year after the release of Airflow 3.0, so naturally I was very curious to see how many people had already upgraded to Airflow 3. I was expecting around 15% and amazed to see that <strong>26.1% of respondents said that they were already on Airflow 3.0</strong> (or even 3.1).</p><p>Another 42.1% of users were on Airflow 2.8–2.11 and of all users on the past major version, 84.2% said they are gearing up to upgrade.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C-GGGcbEOAycnwTMGuSE3A.png" /><figcaption>Share of Airflow versions half a year after the release of Airflow 3. 26.1% are on Airflow 3.0–3.1!</figcaption></figure><p>In a closely related question, we asked how often Airflow users upgrade their environments and got a surprisingly even spread: from 22.4% that upgrade as soon as there is a new version available to the 18.3% that take over a year to upgrade.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*z3_L86Yfh_FltbBv6Ag2vA.png" /><figcaption>Upgrading frequencies among Airflow users.</figcaption></figure><p>Using these numbers we can extrapolate that right now in February 2026 about 36% of Airflow users have yet to upgrade to Airflow 3 with only 2 months left until the <a href="https://www.astronomer.io/airflow-2-eol/">EoL of OSS Airflow 2</a>.</p><p>If that is you, don’t worry, I recently wrote a <a href="https://www.astronomer.io/ebooks/upgrade-from-airflow-2-to-3?utm_campaign=ebook-upgrading-11-25?utm_medium=organicsocial?utm_source=medium?utm_content=tamara"><strong>book on how to upgrade from Airflow 2 to Airflow 3 which you can get for free here</strong></a>. I hope this helps 😊.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_Xn2TyZArC54MylxeWDG1w.png" /><figcaption>Real life footage of what it feels like to upgrade Airflow from 2 to 3.</figcaption></figure><h3>Popularity of Airflow 3 features</h3><p>Of course people don’t just upgrade for the sake of being on a new version, or because their security team wants them to. Those 26.1% of early Airflow 3 adopters were eager to upgrade to use some long-requested Airflow features such as Dag versioning, event-driven scheduling or remote execution.</p><p>Looking at the most popular Airflow features we see <strong>Data-driven scheduling with Assets (Datasets in Airflow 2) winning</strong> a close race against the Taskflow API and dynamic task mapping coming in third. What pleasently surprised me when looking at the responses to this question was that several Airflow 3 exclusive features have already been used by many Airflow users, especially event-driven scheduling, which showed almost 25% adoption.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1G2GKJjPUBgkmtMtJYMZ5A.png" /><figcaption>The most popular Airflow features are: Data-driven scheduling with Assets/Datasets, Taskflow API and dynamic task mapping.</figcaption></figure><h4>Tutorials for the most popular Airflow features</h4><p>I know sometimes the speed at which new features are added to Airflow can feel a bit overwhelming. If you missed any of the above features and would like to read up on it to add it to your pipelines check out the following guides:</p><ul><li><a href="https://www.astronomer.io/docs/learn/airflow-datasets">Assets and @asset syntax</a></li><li><a href="https://www.astronomer.io/docs/learn/airflow-decorators">Taskflow API</a> (Airflow decorators like @task, @task.bash, @task.branch)</li><li><a href="https://www.astronomer.io/docs/learn/dynamic-tasks">Dynamic Task Mapping</a></li><li><a href="https://www.astronomer.io/docs/learn/airflow-event-driven-scheduling">Event-driven scheduling</a></li><li><a href="https://www.astronomer.io/docs/learn/airflow-dag-versioning">Dag versioning and Dag bundles</a></li><li><a href="https://www.astronomer.io/docs/learn/custom-xcom-backend-strategies">Custom XCom backends</a></li><li><a href="https://www.astronomer.io/docs/learn/deferrable-operators">Async/Deferrable Operators</a></li><li><a href="https://www.astronomer.io/docs/astro/remote-execution-overview">Remote execution on Astro</a></li><li><a href="https://www.astronomer.io/docs/learn/airflow-setup-teardown">Setup/Teardown tasks</a></li></ul><p>And even if (you think) your Dags don’t need any new features: just one look at the 3.1 UI would make me want to upgrade immediately 😎.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Neso0XSnsL1Hccm6.png" /><figcaption>Screenshot of the Airflow 3.1 UI in light and dark mode.</figcaption></figure><h3>Orchestrating AI with Airflow</h3><p>Airflow is already the standard for data workflows… and in the age of 🦞s quickly becoming the standard for orchestrating AI and ML as well! Comparing the statistics from the 2024 and 2025 survey I found a significant increase in both, MLOps and GenAI use cases, overall <strong>almost a third of Airflow users now use it to orchestrate AI pipelines</strong>.</p><p>Particularly exciting: beginner users were outpacing users with 1–3 years of Airflow experience in GenAI use cases, indicating that some people are picking up Airflow with GenAI already in mind.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jgRpDsnsoMbQxj7T-hix5g.png" /><figcaption>Comparison of AI/ML use cases in the Airflow survey 2024 vs 2025.</figcaption></figure><h4>Features+ you need to know for AI</h4><ul><li><a href="https://github.com/astronomer/airflow-ai-sdk">Airflow AI SDK</a>: Not a core Airflow feature but a provider package making it easy to interact with AI agents and LLM (literally just <strong>@task.llm or @task.agent</strong>).</li><li><a href="https://www.astronomer.io/docs/learn/airflow-event-driven-scheduling">Event-driven scheduling</a>: The ability to schedule Dags based on messages in a message queue for inference execution workflows.</li><li><a href="https://www.astronomer.io/docs/learn/airflow-human-in-the-loop">Human(or agent)-in-the-loop</a>: This feature introduced in Airflow 3.1 is one of the most crucial ones to orchestrate AI in my opinion. It adds tasks that pause (defer themselves) and <strong>wait for input, either from a human or from an AI agent.</strong></li></ul><pre>from airflow.providers.standard.operators.hitl import HITLOperator<br><br>_hitl_task = HITLOperator(<br>    task_id=&quot;hitl_task&quot;,<br>    subject=&quot;Expense Approval Required&quot;,  # templatable<br>    body=&quot;{{ ti.xcom_pull(task_ids=&#39;upstream_task&#39;) }}&quot;, # templatable<br>    options=[&quot;ACH Transfer&quot;, &quot;Wire Transfer&quot;, &quot;Corporate Check&quot;],  # cannot be empty!<br>    defaults=[&quot;ACH Transfer&quot;],<br>    multiple=False,  # default: False<br>    params={<br>        &quot;expense_amount&quot;: Param(<br>            10000,<br>            type=&quot;number&quot;,<br>        )<br>    },<br>    execution_timeout=timedelta(minutes=5),  # default: None<br>)<br><br>@task<br>def print_result(hitl_output):<br>    print(f&quot;Expense amount: ${hitl_output[&#39;params_input&#39;][&#39;expense_amount&#39;]}&quot;)<br>    print(f&quot;Payment method: {hitl_output[&#39;chosen_options&#39;]}&quot;)<br>_print_result = print_result(_hitl_task.output)</pre><p>The body text is rendered as markdown in the Airflow UI and stakeholders can enter information and make decisions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tFBeZ8OaH0ZMnjtt.png" /><figcaption>Screenshot of the Airflow 3.1 UI showing a Required Action interface created by the HITLOperator code above.</figcaption></figure><p>Of course this feature is also accessible via the Airflow REST API, meaning you can integrate it into any frontend! Recently created a proof-of -concept <a href="https://github.com/TJaniF/airflow-hitl-slack-plugin">Slack HITL plugin for Airflow</a>.</p><h3>ChatGPT write me a Dag — no, not like this!</h3><p>Of course engineers are not just using Airflow to orchestrate AI, they use AI to write Airflow Dags! <strong>84.2% of Airflow users use AI to assist them in writing Airflow code</strong>, with 34.6% saying they use it often or always.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AyjaonmpM-PQGh8C50GjjQ.png" /><figcaption>Almost everyone uses AI to write Airflow code…</figcaption></figure><p>That’s a lot of AI generated code! Unfortunately generic AI tools are not up to the task: <strong>only 9% of Airflow users say that they are happy with the generated code</strong>, with many citing hallucinations, outdated syntax, bad practices and <strong>missing business context</strong> as reasons for having to fix the code that the 🤖 wrote.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/631/1*aq3B3XieynxyAx-pcODgQA.png" /><figcaption>… and almost no one is fully happy with that code (yet).</figcaption></figure><p>These statistics were a key factor for <strong>Astronomer to open-source our own </strong><a href="https://github.com/astronomer/agents"><strong>Airflow-related agent skills</strong></a>. That is right, you can make your AI better at writing Airflow Dags today with just one command!</p><pre>npx skills add astronomer/agents --skill &#39;*&#39;</pre><h3>Airflow is used at huge scales (and to make money!)</h3><p>Some organizations use Airflow at dizzying scales. Like <a href="https://www.astronomer.io/blog/airflow-in-action-qualcomm/">Qualcomm</a> who run millions of tasks to orchestrate chip design workflows (how many CPUs do you need to develop next generation CPUs? turns out <em>a lot</em>).</p><p>22% of Airflow users work in an organization running 7 or more Airflow instances in production, with some reporting that they have <strong>over 1000!</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/927/1*navD5u1_AwR9TpZ3_O4M-g.png" /><figcaption>The scale of Airflow: 7.2% of users said they have 20+ production instances running at their company.</figcaption></figure><p>Zooming in to individual Dags: I often get asked how many tasks should be in a Dag. The answer depends a lot on what you are doing with Airflow and how you are structuring your Dags, from dynamic task mapping to task groups. Looking at the spread, a lot of Airflow users like to write smaller modular Dags with &lt;10 (32%) or 11–50 (36.4%) tasks. <strong>2.7% of you like to create monster Dags with over 1000 tasks!</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CjygYS4Ie33dzsuvyElToQ.png" /><figcaption>Some of you might want to break down some monolithic Dags with Assets…</figcaption></figure><p>Lastly, we asked if people are making money with Airflow by using it for external or revenue-generating use cases (like, for example an app serving personalized suggestions orchestrated by Airflow). <strong>An amazing 89% of users said they are expecting to use Airflow for more such use cases next year! 💰</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uDwy2guMVFseSSY1WemC4A.png" /><figcaption>Many businesses are built on top of Airflow pipelines!</figcaption></figure><h3>Conclusion</h3><p>Does anyone else feel like time itself has accelerated? It certainly feels that way as part of the Airflow and data engineering community! We are moving fast, from maintaining and extending foundational ETL/ELT to LLM pipeliens to orchestrating agents to using Airflow to <a href="https://www.astronomer.io/blog/how-to-build-a-decision-tracing-context-graph-with-apache-airflow/">create context graphs</a>… <strong>exciting times ahead!</strong></p><h3>I want more!</h3><ul><li>A comprehensive write-up you can share with your manager to explain why Airflow (and your Airflow knowledge) is important (💵) exists in the Astronomer <a href="https://www.astronomer.io/airflow/state-of-airflow?utm_campaign=whitepaper-state-of-airflow-2-26?utm_medium=organicsocial?utm_source=medium?utm_content=tamara"><strong>State of Apache Airflow® report</strong></a><strong>.</strong></li><li>You can watch a recording of Jarek Potiuk, Vikram Koka (two Airflow PMC members) and me talk about the survey results and what is coming next in Airflow in <a href="https://www.astronomer.io/events/webinars/the-state-of-airflow-2026-video?utm_campaign=webinar-state-of-airflow-2-26?utm_medium=organicsocial?utm_source=medium?utm_content=tamara"><strong>The State of Apache Airflow® 2026 webinar</strong></a><strong>.</strong></li><li>More graphs can be found in the <a href="https://airflow.apache.org/blog/airflow-survey-2025/">Airflow survey infographic</a>.</li><li>You can now get in on the fun of crunching this data yourself! The full Airflow survey 2025 results are available <a href="https://astronomer.typeform.com/reports/01KESPS8SJ2Y80THJAEYCECE5B">here</a>.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da0b313421f9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Kafka ❤️ Apache Airflow: A no-install click-and-play demo of the Kafka Airflow provider]]></title>
            <link>https://medium.com/apache-airflow/apache-kafka-%EF%B8%8F-apache-airflow-a-no-install-click-and-play-demo-of-the-kafka-airflow-provider-6a1df9427c01?source=rss-bf3ba0b8be12------2</link>
            <guid isPermaLink="false">https://medium.com/p/6a1df9427c01</guid>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[apache-airflow]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Tamara Janina Fingerlin]]></dc:creator>
            <pubDate>Thu, 18 May 2023 16:02:04 GMT</pubDate>
            <atom:updated>2023-05-18T17:00:13.220Z</atom:updated>
            <content:encoded><![CDATA[<p><em>I want to explore on my own, where is the code? There you go: </em><a href="https://github.com/TJaniF/airflow-kafka-quickstart/">https://github.com/TJaniF/airflow-kafka-quickstart/</a></p><p>Important note: All credit for the <a href="https://registry.astronomer.io/providers/apache-airflow-providers-apache-kafka/versions/latest">Kafka Airflow provider</a> goes to Dylan Storey (<a href="https://www.linkedin.com/in/dylanbstorey/">LinkedIn</a>, <a href="https://github.com/dylanbstorey">GitHub</a>)! Thank you for introducing me to the streaming world! 😊</p><p>Are you a data engineer using <a href="https://kafka.apache.org/">Apache Kafka</a> and <a href="https://airflow.apache.org/">Apache Airflow</a> and want the two to seamlessly connect? Or are you an avid user of one of these tools curious to see how one integrates with the other? Or do you just want to play and build upon a streaming/batch data project?</p><p>If you thought yes to any of these questions, this project is for you!</p><figure><img alt="Airflow UI showing the DAGs present in the Airflow Kafka Quickstart repository." src="https://cdn-images-1.medium.com/max/1024/1*BJxAcIr6xbWW48Y2zCkuKA.png" /><figcaption>Airflow UI showing the DAGs present in the Airflow Kafka Quickstart repository.</figcaption></figure><p>The Airflow Kafka Quickstart repository has been created to start both an Airflow environment, as well as a local Kafka cluster in their respective Docker containers and connect them for you. No setup needed!</p><p>This ready-to-run project uses three DAGs to directly produce to and consume from a Kafka cluster as well as orchestrate downstream dependencies based on a specific message being detected in the stream, using Airflow operators from the <a href="https://registry.astronomer.io/providers/apache-airflow-providers-apache-kafka/versions/latest">Kafka Airflow provider</a>.</p><p>Follow these 3 simple steps to run and explore the project by yourself:</p><h4>Step 1: Create a Codespace</h4><p><a href="https://www.linkedin.com/in/faisalhoda/">Faisal Hoda</a> created a devcontainer image which enables you to run Airflow with the <a href="https://docs.astronomer.io/astro/cli/overview">Astro CLI</a> in <a href="https://docs.github.com/en/codespaces">GitHub codespaces</a>, without having to install or download anything locally. After forking <a href="https://github.com/TJaniF/airflow-kafka-quickstart">the repository</a>, just create a codespace with at least 4 cores and both, the Airflow environment and the local Kafka cluster will automatically start and connect to each other.</p><h4>Step 2: Enter your and your (imaginary) pet’s name</h4><p>At the start of the dags/produce_consume_treats.py file 3 important variables are defined: YOUR_NAME , YOUR_PET_NAME and NUMBER_OF_TREATS . You can adjust them to your liking. (Note that NUMBER_OF_TREATS will determine how many messages are produced to the Kafka topic, so if your pet is very hungry, you might want to adjust the max_messages parameter of the consume_treats task.)</p><h4>Step 3: Run the Pipeline</h4><p>Once Airflow is running, you can access the Airflow UI at the Local Address of the forwarded port 8080.</p><figure><img alt="Screenshot of a codespace running the Airflow Kafka quickstart repository, showing the local address for the 8080 port." src="https://cdn-images-1.medium.com/max/1024/0*Mdi0HPQ_fmV4MDIr.png" /><figcaption>Screenshot of a codespace running the Airflow Kafka quickstart repository, showing the local address for the 8080 port.</figcaption></figure><p>In the Airflow UI unpause all DAGs (Directed Acyclic Graphs, Airflows workflow unit) by clicking the toggle on the left side of the DAG name. The listen_to_the_stream DAG will immediately start running and start listening to messages in the Kafka stream that indicate your pet is in a bouncy or zoomy mood after the last treat of a series of treats.</p><p>To give treats to your pet, manually run the produce_consume_treats DAG. The listen_to_the_stream DAG will automatically cause a run of the walking_my_pet DAG whenever your pet ends up bouncy or zoomy at the end of a series of treats 🐶.</p><h4>How it works</h4><p>The produce_consume_treats DAG contains a task that uses the <br><a href="https://registry.astronomer.io/providers/apache-airflow-providers-apache-kafka/versions/latest/modules/ProduceToTopicOperator">ProduceToTopicOperator</a> to produce messages to your Kafka topic. For each treat you give to your pet, a message is produced containing your pet’s name, their mood after the treat and whether it was the last treat in a series. For example:</p><p>{“pet_name”: “Nevermore”, “pet_mood_post_treat”: “bouncy”, “final_treat”: false}</p><p>You can see the messages via the logs of the consume_treats task in the same DAG using the <a href="https://registry.astronomer.io/providers/apache-airflow-providers-apache-kafka/versions/latest/modules/ConsumeFromTopicOperator">ConsumeFromTopicOperator</a>.</p><figure><img alt="Airflow task logs of the `consume_treats` task in the `produce_consume_treats` DAG containing the lines like: [2023–05–17, 15:52:08 UTC] {logging_mixin.py:149} INFO — Message #0: Hello Jani, your pet Nevermore has consumed another treat and is now content!" src="https://cdn-images-1.medium.com/max/1024/1*P2wPJBm8EqUE3jOSQp0ZSg.png" /><figcaption>Airflow task logs of the `consume_treats` task in the `produce_consume_treats` DAG showing print statements containing information from the messages consumed from the Kafka topic.</figcaption></figure><p>While the manual runs of the produce_consume_treats DAG create new messages in the Kafka topic, the listen_to_the_stream DAG uses the <a href="https://registry.astronomer.io/providers/apache-airflow-providers-apache-kafka/versions/latest/modules/AwaitMessageTriggerFunctionSensor">AwaitMessageTriggerFunctionSensor</a> to listen to these messages in a <a href="https://docs.astronomer.io/learn/deferrable-operators">deferrable state</a> for one that fits the criteria defined in the function (listen_function) provided to its apply_function parameter.</p><p>If your pet ended up bouncy or zoomy at the end of the of a series of treats, the listen_function will return a value, causing the event_triggered_function to run and kick off a run of the walking_my_pet DAG via the execute method of the <a href="https://registry.astronomer.io/providers/apache-airflow/versions/latest/modules/TriggerDagRunOperator">TriggerDagRunOperator</a>. Afterwards the listen_for_mood task defers itself again, creating a permanent listener!</p><figure><img alt="Airflow task logs showing the run of the TriggerDagRunOperator caused in by the `listen_to_mood` task in the `listen_to_the_stream` DAG." src="https://cdn-images-1.medium.com/max/1024/1*ZHsUd0745yRz8aEBXJZChw.png" /><figcaption>Airflow task logs showing the run of the TriggerDagRunOperator caused in by the `listen_to_mood` task in the `listen_to_the_stream` DAG.</figcaption></figure><p>Finally, the walking_my_pet DAG takes your pet on a well-deserved walk 😊.</p><h4>Resources</h4><p>You can learn more about the components shown in this repository at the following links:</p><ul><li><a href="https://kafka.apache.org/documentation/">Apache Kafka documentation</a></li><li><a href="https://airflow.apache.org/docs/apache-airflow/stable/index.html">Apache Airflow documentation</a></li><li><a href="https://airflow.apache.org/docs/apache-airflow-providers-apache-kafka/stable/index.html">Kafka Airflow provider documentation</a></li><li><a href="https://docs.astronomer.io/learn/airflow-kafka">Use Apache Kafka with Apache Airflow tutorial</a></li></ul><p><em>Disclaimer: This project and blog post was created with </em>❤️<em> by the DevRel team at </em><a href="http://astronomer.io/"><em>Astronomer</em></a><em>, the company offering hosted managed Apache Airflow and Airflow expertise. All tools and features shown in this blog post are fully open-source and free for you to use. If you want to try Astro, Astronomer’s cloud-based Airflow solution, you can sign up for a </em><a href="https://www.astronomer.io/try-astro/"><em>14-day free trial</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a1df9427c01" width="1" height="1" alt=""><hr><p><a href="https://medium.com/apache-airflow/apache-kafka-%EF%B8%8F-apache-airflow-a-no-install-click-and-play-demo-of-the-kafka-airflow-provider-6a1df9427c01">Apache Kafka ❤️ Apache Airflow: A no-install click-and-play demo of the Kafka Airflow provider</a> was originally published in <a href="https://medium.com/apache-airflow">Apache Airflow</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ELT with Airflow: a no-install click-and-play blueprint project running in codespaces]]></title>
            <link>https://medium.com/apache-airflow/elt-with-airflow-a-no-install-click-and-play-blueprint-project-running-in-codespaces-1b92c1713569?source=rss-bf3ba0b8be12------2</link>
            <guid isPermaLink="false">https://medium.com/p/1b92c1713569</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[github-codespaces]]></category>
            <dc:creator><![CDATA[Tamara Janina Fingerlin]]></dc:creator>
            <pubDate>Tue, 28 Mar 2023 16:39:04 GMT</pubDate>
            <atom:updated>2023-03-30T12:30:31.622Z</atom:updated>
            <content:encoded><![CDATA[<p><em>I’m in a hurry, where is the code? Here: </em><a href="https://github.com/TJaniF/airflow-elt-blueprint"><em>https://github.com/TJaniF/airflow-elt-blueprint</em></a></p><p>Are you a data professional who knows their way around Python code, and wants to get started with <a href="https://airflow.apache.org/docs/apache-airflow/stable/index.html">Airflow</a> without reading a tutorial or having to install any packages? Or maybe you’ve used Airflow before and want to explore new features in a pre-built sandbox environment? Or are you considering learning Airflow but want to see it do something cool first without any previous knowledge?</p><p>This project was made for you! 😊</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*prEz70MxNwNTtkWaoEd1kQ.png" /><figcaption>Airflow UI showing the DAGs present in the blueprint repository.</figcaption></figure><p>The Airflow ELT Blueprint repository contains a fully functional extract-load-transform (ELT) pipeline which ingests climate- and weather-related data from two different sources into <a href="https://min.io/">MinIO</a>, loads and transforms the data in a <a href="https://duckdb.org/">DuckDB</a> instance using the <a href="https://astro-sdk-python.readthedocs.io/en/stable/index.html">Astro Python SDK</a>, and powers a <a href="https://streamlit.io/">Streamlit</a> App.</p><p>In order to run and explore the project yourself, you only need to follow 4 simple steps:</p><h4>Step 1: Create a codespace</h4><p>Thanks to the work of <a href="https://www.linkedin.com/in/faisalhoda/">Faisal Hoda</a>, you can use GitHub codespaces to run Airflow with the <a href="https://docs.astronomer.io/astro/cli/overview">Astro CLI</a>, without having to install or download anything locally. Just <a href="https://github.com/TJaniF/airflow-elt-blueprint">fork the repo</a> and create a codespace with at least four cores; the Astro CLI will automatically start an Airflow environment.</p><h4><strong>Step 2: Enter your name and city</strong></h4><p>Navigate to include/global_variables/global_variables.py and provide your values for MY_NAME and MY_CITY.</p><h4>Step 3: Run the pipeline</h4><p>Once Airflow is running, you can access the Airflow UI at the Local Address of the forwarded port 8080.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XrXvyPxhIpvq5JlI.png" /><figcaption>Screenshot of a codespace running the blueprint repo, showing the Local Address for the 8080 port.</figcaption></figure><p>In the UI, unpause the DAGs (Directed Acyclic Graphs, Airflows workflow unit) starting from the first DAG and continuing to the DAG named start by clicking on the toggle to the left of the DAG names.</p><p>As soon as the start DAG runs, the other DAGs will run based on <a href="https://docs.astronomer.io/learn/airflow-datasets">Data-driven scheduling using Datasets</a>. In the Datasets tab, you can view the interdependence between DAGs and Datasets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pl_h1TkreWnekw3p.png" /><figcaption>Screenshot of the Datasets tab of the blueprint repository.</figcaption></figure><h4>Step 4: View the Streamlit app</h4><p>As you can see in the image above, the last DAG to run is the run_streamlit_app DAG, it will stay in a running state creating a Streamlit app based upon the data that the pipeline ingested and transformed.</p><p>Open the Local Address of the forwarded port 8501 to view the interactive Streamlit App.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e4cm4T3-_NHaAQSSEk7yOQ.png" /><figcaption>Screenshot of the upper half of the Global Climate and Local Weather Streamlit App.</figcaption></figure><h4>(Some of the) Airflow features shown</h4><p>The blueprint repository uses a variety of Airflow features:</p><ul><li><a href="https://docs.astronomer.io/learn/airflow-datasets">Datasets</a>: as explained above, Datasets are used to create data-driven dependencies between DAGs.</li><li><a href="https://docs.astronomer.io/learn/dynamic-tasks">Dynamic task mapping</a>: Several tasks in this repository are dynamically mapped to adjust the number of needed mapped task instances depending on inputs at runtime.</li><li><a href="https://astro-sdk-python.readthedocs.io/en/stable/index.html">Astro Python SDK</a>: The create_reporting_table DAG uses the transform operator of the Astro Python SDK to make creating a reporting table from a SQL Select query simple.</li><li><a href="https://docs.astronomer.io/learn/airflow-importing-custom-hooks-operators">Custom operators and hooks</a>: When interacting with MinIO the blueprint repository uses custom MinIO operators, which are stored locally in include/custom_operators/minio.py the operators use a custom Airflow hook to interact with MinIO using credentials saved in an Airflow connection.</li><li><a href="https://www.astronomer.io/events/live/reusable-dag-patterns-with-taskgroups/">Custom re-useable task group</a>: The pattern of checking if a MinIO bucket of a specific name already exists and, if not, creating the bucket occurs several times in this repository. A great use case too, instead of redefining the tasks every time, use a reusable task group. Explore the task group code at include/custom_task_groups/create_bucket.py .</li><li><a href="https://docs.astronomer.io/learn/airflow-passing-data-between-tasks">Airflow XCom</a> and <a href="https://airflow.apache.org/docs/apache-airflow/stable/howto/variable.html">Airflow Variables</a>: Small amounts of data like the current weather in the user-defined city are passed from one task to another using XCom. The coordinates of the city are saved as an Airflow Variable.</li></ul><h4>Have fun!</h4><p>Use this project and the patterns and features shown within it as a blueprint for your own work or to try out new things (bonus points for sharing what you create). For data professionals new to Airflow, I hope this repo has sparked a desire to <a href="https://docs.astronomer.io/learn">learn more</a> and join the great community of Airflow users!</p><p><em>Disclaimer: This project and blog post was created with </em>❤️<em> by the DevRel team at </em><a href="http://astronomer.io/"><em>Astronomer</em></a><em>, the commercial developer behind Apache Airflow. All tools and features shown are fully open-source and free for you to use.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b92c1713569" width="1" height="1" alt=""><hr><p><a href="https://medium.com/apache-airflow/elt-with-airflow-a-no-install-click-and-play-blueprint-project-running-in-codespaces-1b92c1713569">ELT with Airflow: a no-install click-and-play blueprint project running in codespaces</a> was originally published in <a href="https://medium.com/apache-airflow">Apache Airflow</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DataTalksClub: Data Engineering Zoomcamp — Week 1]]></title>
            <link>https://medium.com/@tjanif/datatalksclub-data-engineering-zoomcamp-week-1-a56d0f5a8760?source=rss-bf3ba0b8be12------2</link>
            <guid isPermaLink="false">https://medium.com/p/a56d0f5a8760</guid>
            <category><![CDATA[datatalksclub]]></category>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[zoomcamp]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Tamara Janina Fingerlin]]></dc:creator>
            <pubDate>Wed, 19 Jan 2022 12:35:52 GMT</pubDate>
            <atom:updated>2023-03-28T16:50:40.266Z</atom:updated>
            <content:encoded><![CDATA[<h3>DataTalksClub: Data Engineering Zoomcamp — Week 1 (Docker)</h3><p>(sharing my notes and learning process during the first week of the free <a href="https://github.com/DataTalksClub/data-engineering-zoomcamp">Data Engineering Zoomcamp</a> from the <a href="https://datatalks.club/">DataTalksClub</a>)</p><p>After meeting the instructors in the kick-off session on Monday and hearing the reassuring “yes”-answer to the question “am I in the right place if I am a Data Scientist seeking to learn more about Data Engineering”, I decided to block some time today and go through the material of the first week.</p><p>Since I’ve always learned best by summarising material in my own words I decided to make a little blog post out of this first session. If you find a mistake in my notes, whether technical or not, please tell me! (English is not my first language) :)</p><p>The GitHub repo is linked above, the instructional videos can be found in <a href="https://www.youtube.com/watch?v=EYNwNlOrpr0&amp;list=PL3MmuxUbc_hJed7dXYoJw8DoCuVHhGEQb&amp;index=3&amp;ab_channel=DataTalksClub">this playlist on YouTube</a>.</p><p>If you are a Data Scientist why do you want to learn about Data Engineering?</p><p>First of all because I find it interesting! Secondly, in my experience it is very useful to know at least the basics of the tools and processes that adjacent professionals use; it helps me to know how to structure information and how to best ask questions about what is possible. For example knowing about the workflow of residents of other specialities and nurses in different wards, I knew what information they needed form me and how to get it to them in the most efficient way.</p><h4>Prerequisites</h4><ul><li>Install <a href="https://docs.docker.com/get-docker/">Docker</a></li><li>Install a code editor, for example <a href="https://code.visualstudio.com/">VSCode</a></li></ul><h3>Docker</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*fa3zBrjxQXEZMNt1.png" /><figcaption>Logo of docker (very adorable)</figcaption></figure><p>Docker is a software that allows you to have multiple containers that use different versions of OS, tools and packages in neat little containers, you don’t need to install anything on your own computer, just in the Docker container you need it in. This avoids conflicts between databases and pipelines, you can run experiments without messing up other projects.</p><p>A Docker-image is a snapshot of a container with all the specifications, which can be used on any computer, including Cloud Storages (AWS, Google Cloud etc), 100% compatibility and reproducibility out of the box!</p><p><em>First little comment: to start VSCode from the command-line like Alexey does you need to add VSCode to the PATH. Having VSCode open use Cmd+Shift+P (on Mac) and type ‘shell command’, then select the ‘install code to PATH’ command.</em></p><blockquote>docker run -it ubuntu bash</blockquote><p>Will start a ubuntu image with bash executed in interactive mode (-it). We can do silly things here, without affecting anything outside the container, just exit and restart again for a clean slate.</p><p><em>Alright, I’ll be right back, hopping in my time machine to tell 16 y/o me about this and prevent some horrible things she did to her first linux machine in the name of exploration.</em></p><blockquote>docker run -it --entrypoint=bash python:3.9</blockquote><p>use :version_number to specify a version. By using the bash entrypoint it is possible to pip install python packages in the container. But keep in mind: the packages will vanish after closing the image.</p><p>We can use a Dockerfile to specify instructions for a new image:</p><blockquote>FROM python:3.9.1</blockquote><blockquote>RUN pip install pandas</blockquote><blockquote>WORKDIR /app</blockquote><blockquote>COPY some_file.py some_file.py</blockquote><blockquote>ENTRYPOINT [ “python”, “some_file.py” ]</blockquote><p>FROM specifies the base image we want. RUN tells the container what commands to run. WORKDIR is the location in the image where files we are copying will be put. COPY takes a file from the host machine and copies it to the container. ENTRYPOINT is where we want to end up opening the container, in this case we want docker to run some_file.py using the python command.</p><p>The new container can be built using:</p><blockquote>docker build -t image_name:version [directory where the dockerfile is]</blockquote><p>it can be run using</p><blockquote>docker run -it image_name:version [optional arguments for sys.argv]</blockquote><p>If the python script contains sys.argv one can pass arguments (starting at index 1) to the python file when running the docker image.</p><blockquote>docker ps</blockquote><p>Shows all the docker containers currently running.</p><h4>Postgres in Docker</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JwaLcke4K547Rnpg.png" /><figcaption>Logo of PostgreSQL.</figcaption></figure><p><a href="https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/week_1_basics_n_setup/2_docker_sql/README.md">This README</a> contains all necessary commands for this section, also for Windows.</p><blockquote>docker run -it \</blockquote><blockquote>-e POSTGRES_USER=”root” \</blockquote><blockquote>-e POSTGRES_PASSWORD=”root” \</blockquote><blockquote>-e POSTGRES_DB=”ny_taxi” \</blockquote><blockquote>-v $(pwd)/ny_taxi_postgres_data:/var/lib/postgresql/data \</blockquote><blockquote>-p 5432:5432 \</blockquote><blockquote>postgres:13</blockquote><p><em>Note: I am probably the only person for whom this is news but the \ in the command line lets you type a multi-line command. Also, if you copy and paste the above make sure to replace the quotation marks, the pasted marks wont be recognised.</em></p><p>the -e flag is used to set environmental variables. -v stands for volumes used to map folders on the host machine (fist path) to the container (path after the colon). This step is called mounting and prevents our Postgres database from being lost after the container is closed. -p specifies the port ‘port on hostmachine’: ‘port in the container’, 5432 is the standard port for Postgres.</p><p>After the container is up and running next we need to pip install the ‘pgcli’ client for python.</p><blockquote>pgcli -h localhost -p 5432 -u root -d ny_taxi</blockquote><p>-h being the host, -p the port, -u the user and -d the database.</p><p><em>Note: at this point I got an error that the user “root” does not exist. I eventually was able to fix it by running the docker with my own username (tjanif) and then running the command with -u tjanif. Also if you, like me, manage to mess up in the process and need to get rid of a container blocking a port ‘docker container ls’ followed by ‘docker stop [container id]’ are your new friends.</em></p><p>Finally in the (empty) database! ‘\dt’ lists tables. ‘\d table_name’ describes a table.</p><h4><strong>Reading in Data</strong></h4><p>It is possible to use a <a href="https://github.com/DataTalksClub/data-engineering-zoomcamp/blob/main/week_1_basics_n_setup/2_docker_sql/upload-data.ipynb">Jupyter Notebook</a> to read in a .csv and create a schema from it (preferably with a smaller sample) using the ‘pd.io.sql.get_schema()’ function. The create_engine() function from the sqlalchemy library will tell pandas that we want to put this into a Postgres database.</p><p>Next we create an iterator, simply by setting iterator=True in the pd.read_csv() function to iterate through batches of the data.</p><blockquote>df.head(n=0).to_sql(name=’yellow_taxi_data’, con=engine, if_exists=’replace’)</blockquote><p>writes the header into the database and creates the table.</p><blockquote>df.to_sql(name=’yellow_taxi_data’, con=engine, if_exists=’append’)</blockquote><p>populates the database with the first batch of data. With a while loop (see the notbook) we can insert the rest of the data.</p><h4>pgAdmin</h4><p>is a <a href="https://www.pgadmin.org/download/pgadmin-4-container/">web-based GUI for Postgres</a>.</p><blockquote>docker run -it \</blockquote><blockquote>-e PGADMIN_DEFAULT_EMAIL=”admin@admin.com” \</blockquote><blockquote>-e PGADMIN_DEFAULT_PASSWORD=”root” \</blockquote><blockquote>-p 8080:80 \</blockquote><blockquote>dpage/pgadmin4</blockquote><p><em>Again, remember to retype the quotation marks. Don’t be me, don’t spend half a hour on debugging this because of wonky quotation marks.</em></p><p>The above command will create a docker container using pgadmin. Again -e are the environment variables. -p is the portmapping, meaning pgAdmin will listen for requests at port 80. ‘<a href="http://localhost:8080/">http://localhost:8080/</a>’ is the URL to find the pgAdmin interface.</p><p>Now we are running 2 different containers: one with the PostgresDB and the other one with the pgAdmin interface.</p><p>Let’s introduce them to each other using a network.</p><ol><li>Stop both containers.</li><li>Create a network</li></ol><blockquote>docker network create pg-network</blockquote><p>3. restart the Postgres and the pgAdmin container again, adding two new flags to each one</p><blockquote>--network=pg-network \</blockquote><blockquote>--name [pg-database for the Postgres container, pgadmin for the pgAdmin container]\</blockquote><p><em>Note: if things error try to put all on one line, extra spaces confuse the command line.</em></p><p>4. set the Connection Host name to pg-database</p><p>The containers are now talking to each other :)</p><p>That is it for the first 3 videos. Definitely a steep learning curve for someone like me without a CS background but doable and enjoyable!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a56d0f5a8760" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>