<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kenten Danas on Medium]]></title>
        <description><![CDATA[Stories by Kenten Danas on Medium]]></description>
        <link>https://medium.com/@kentdanas?source=rss-1884f45f4b61------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*t-AUwf2xpKy7tAX72ka2hw.jpeg</url>
            <title>Stories by Kenten Danas on Medium</title>
            <link>https://medium.com/@kentdanas?source=rss-1884f45f4b61------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 15:20:01 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kentdanas/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Avoiding the pitfalls of top-level DAG code]]></title>
            <link>https://medium.com/apache-airflow/avoiding-the-pitfalls-of-top-level-dag-code-fa480d9e75c6?source=rss-1884f45f4b61------2</link>
            <guid isPermaLink="false">https://medium.com/p/fa480d9e75c6</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[airflow]]></category>
            <dc:creator><![CDATA[Kenten Danas]]></dc:creator>
            <pubDate>Mon, 08 May 2023 19:42:17 GMT</pubDate>
            <atom:updated>2023-05-08T19:54:44.274Z</atom:updated>
            <content:encoded><![CDATA[<p>One of Airflow’s great strengths is its flexibility. Because everything is written in Python code, you can structure your Airflow environment and write your DAGs to do nearly anything. But with this great power comes great responsibility. You need your production orchestration system to be scalable and stable, and sometimes this means considering not “can I do this?”, but “should I?”.</p><p>Following some basic Airflow best practices can help you avoid headaches when relying on Airflow for your critical workflows. One very common one we’ve seen trip people up is using top-level code in DAGs. In this post, we’ll detail the issues with top-level code and some ways to get around it.</p><p>This post was developed in collaboration with <a href="https://www.linkedin.com/in/fritz-davenport-03b73b209/">Fritz Davenport</a>.</p><h3>What is “top-level” code and why is it problematic?</h3><p>In the context of Airflow, top-level code refers to any code that isn’t part of your DAG or operator instantiations. In other words, code that is “outside” of your operators, particularly code making requests to external systems.</p><p>Typically in Airflow, top-level code is used to dynamically generate tasks or DAGs, or otherwise change what your DAG does on each DAG run. For example, you might have a script that queries a database for a list of customers and creates one copy of a DAG for each customer. Since everything in Airflow is Python-based, this type of use case is straightforward to implement.</p><p>However, generating DAGs or tasks in this way starts to create problems if that customer list is large, or the system you’re querying from is sensitive to a large volume of requests or is ever unreachable. Airflow executes all code in your dags_folder every 30 seconds by default. Because of this, top-level code that makes requests to external systems, like an API or a database, or makes function calls outside of your tasks, can cause performance issues since these requests and connections are being made at least every 30 seconds rather than only when the DAG is scheduled to run. This can stress both your Airflow infrastructure and the system you’re querying from.</p><h3>How do I avoid the pitfalls?</h3><p>There are multiple ways to avoid the pitfalls of top-level code and still have dynamic tasks or DAGs in Airflow.</p><h3>Avoid top-level code when possible</h3><p>The easiest way to avoid issues with top-level code is to not have any at all. Sometimes it may be unavoidable (we’ll get to those situations later), but other times it might be as simple as designing your DAGs in a better way.</p><p>For example, you might have a use case where you need to use an Airflow variable as a parameter for a task in your DAG. If you design the DAG with top-level code, you would make the request to get the variable outside of an operator instantiation.</p><pre>...<br>foo = Variable.get(&quot;foo&quot;)<br><br>BashOperator(<br>  task_id=&quot;bash_use_variable_good&quot;,<br>  bash_command=&quot;echo variable foo=${foo_env}&quot;,<br>  env={&quot;foo_env&quot;: foo}<br>)<br>...</pre><p>The `Variable.get(“foo”)` function will be run every 30 seconds and make a query to your Airflow metadata database to get the variable. This can put strain on your database, and if you did this with many variables, could cause performance issues.</p><p>But it’s not necessary to write the DAG this way. Instead, you can use a templated variable, which will only make the request to the database at runtime.</p><pre>…<br>BashOperator(<br>  task_id=&quot;bash_use_variable_good&quot;,<br>  bash_command=&quot;echo variable foo=${foo_env}&quot;,<br>  env={&quot;foo_env&quot;: &quot;{{ var.value.foo }}&quot;}<br>)<br>…</pre><p><a href="https://docs.astronomer.io/learn/templating">Templating</a> can be a great way to pass dynamic information to tasks at runtime without relying on top-level code.</p><h3>Use a first class feature like dynamic task mapping</h3><p>A great alternative to using top-level code to dynamically create tasks or DAGs is to use <a href="https://docs.astronomer.io/learn/dynamic-tasks">dynamic task mapping</a>, which allows you to create parallel copies of tasks at runtime based on external criteria. Dynamic task mapping was released in Airflow 2.3 and is a first-class Airflow feature, meaning it is fully supported, more stable, and more observable than writing custom code to implement this type of use case.</p><p>For example, maybe you have a use case where you need to create a task for every item in a list, and that list is generated by a query to a database. A common implementation of this type of use case is to get the list in top-level code and loop through the results to create your tasks.</p><pre>&quot;&quot;&quot;WARNING: This DAG is used as an example for _bad_ Airflow practices. <br>Do not use this DAG.&quot;&quot;&quot;<br>from airflow.decorators import dag<br>from airflow.providers.postgres.operators.postgres import PostgresOperator<br>from airflow.providers.postgres.hooks.postgres import PostgresHook<br>from pendulum import datetime<br><br># Bad practice: top-level code in a DAG file<br>hook = PostgresHook(&quot;database_conn&quot;)<br>results = hook.get_records(&quot;SELECT * FROM grocery_list;&quot;)<br><br>sql_queries = []<br>for result in results:<br>  grocery = result[0]<br>  amount = result[1]<br>  sql_query = f&quot;INSERT INTO purchase_order VALUES (&#39;{grocery}&#39;, {amount});&quot;<br>  sql_queries.append(sql_query)<br><br>@dag(<br>  start_date=datetime(2023, 1, 1), <br>  max_active_runs=3, <br>  schedule=&quot;@daily&quot;, <br>  catchup=False<br>)<br>  def bad_practices_dag_1():<br>    insert_into_purchase_order_postgres = PostgresOperator.partial(<br>        task_id=&quot;insert_into_purchase_order_postgres&quot;,<br>        postgres_conn_id=&quot;postgres_default&quot;,<br>      ).expand(sql=sql_queries)<br>  bad_practices_dag_1()</pre><p>But this DAG could easily cause performance issues, because the connection to the database and the “SELECT * FROM grocery_list;” query will be run every 30 seconds.</p><p>Instead, you can use a Python task to get the list of values, and use dynamic task mapping to create a task for each item in the list.</p><pre>from airflow.decorators import dag, task<br>from airflow.providers.postgres.operators.postgres import PostgresOperator<br>from airflow.providers.postgres.hooks.postgres import PostgresHook<br>from pendulum import datetime<br><br><br>@dag(<br>    start_date=datetime(2023, 1, 1), max_active_runs=3, schedule=&quot;@daily&quot;, catchup=False<br>)<br>def good_practices_dag_1():<br>    @task<br>    def get_list_of_results():<br>        # good practice: wrap database connections into a task<br>        hook = PostgresHook(&quot;database_conn&quot;)<br>        results = hook.get_records(&quot;SELECT * FROM grocery_list;&quot;)<br>        return results<br><br>    @task<br>    def create_sql_query(result):<br>        grocery = result[0]<br>        amount = result[1]<br>        sql = f&quot;INSERT INTO purchase_order VALUES (&#39;{grocery}&#39;, {amount});&quot;<br>        return sql<br><br>    sql_queries = create_sql_query.expand(result=get_list_of_results())<br><br>    insert_into_purchase_order_postgres = PostgresOperator.partial(<br>        task_id=&quot;insert_into_purchase_order_postgres&quot;,<br>        postgres_conn_id=&quot;postgres_default&quot;,<br>    ).expand(sql=sql_queries)<br><br><br>good_practices_dag_1()</pre><p>This method alleviates the performance issues associated with top-level calls to your database, and ensures you maintain visibility into all your historical task instances even as they change during each DAG run.</p><h3>Use tools to alleviate the pressure</h3><p>If the first two options are not available to you and you absolutely must have top-level code to dynamically generate DAGs, you can consider leveraging Python-based tools to alleviate the pressure on your Airflow infrastructure and external systems.</p><p>One option is to use the `get_with_cache` function created by Astronomer data engineers, which when used as top-level code in your dynamic DAG generation script, caches the results of your request to an external system for a specified period of time. This allows you to limit the amount of requests your top-level code makes for a single DAG, essentially an alternative to modifying the `min_file_process_interval` parameter for your entire Airflow environment.</p><p>The function looks like this:</p><pre>def get_with_cache(fn: Callable = None, cache_path: str = None, expire_at_sec: int = 3600, *args, **kwargs) -&gt; Any:<br>    &quot;&quot;&quot;A function, intended to be used at top-level in a dag, to return database/networked results<br>    with the intent of building dynamic dags / tasks, while keeping scheduler degradation to a minimum.<br>   <br>    :param fn: A function, with optional arguments, receives *args<br>    :param expire_at_sec: Number of seconds after which cached results should be considered stale=<br>    default 1 hour (3600 seconds), cache will be refreshed eventually after, but before this number<br>    :param cache_path: a string which is a file path to store the pickled object as a disk-based cache<br>    :return: the results of whichever function is given as &quot;fn&quot;<br>    &quot;&quot;&quot;<br>    if cache_path is None or fn is None:<br>        raise RuntimeError(&quot;Function and cache path required, cache failed to load!&quot;)<br><br>    if os.path.isfile(cache_path) and int(time.time() - os.path.getmtime(cache_path)) &lt; expire_at_sec:<br>        return pickle.load(open(cache_path, &#39;rb&#39;))<br>    else:<br>        results = fn(*args, **kwargs)<br>        pickle.dump(results, open(cache_path, &#39;wb&#39;))<br>        return results</pre><p>And can be used in your DAG generation script like this:</p><pre>from datetime import datetime<br>from airflow import DAG<br>from dags.caching_util import get_with_cache<br><br>def get_sql_tables():<br>…<br>return …<br><br>with DAG(dag_id=&quot;my_caching_dag&quot;, schedule_interval=None, start_time=datetime(1970, 1, 1)):<br><br>  my_sql_tables = get_with_cache(get_sql_tables, f&quot;{dag_id}.cache&quot;)<br>  <br>  for table in my_sql_tables:<br>  SnowflakeOperator(…)<br></pre><p>Other Python alternatives to this function could include using the <a href="https://pypi.org/project/cachier/">`cachier` library</a> to improve caching or using a library like <a href="https://tenacity.readthedocs.io/en/latest/">tenacity</a> to manage retries so your DAG will still parse even if there is an issue in your top-level code query.</p><p>All of that said, even when using tools like these remember that dynamic DAGs can always fail or degrade performance in unpredictable ways. Be very cautious when relying on features like these for production systems, and make sure that you have backups and trust the source you are building off.</p><h3>Conclusion</h3><p>In summary, top-level code causes frequent issues for Airflow users, and if possible is best avoided in favor of better DAG design or first-class Airflow features. If you can’t avoid top-level code in your DAGs, consider getting creative with one of the tools mentioned here to minimize performance issues in production as you scale.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fa480d9e75c6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/apache-airflow/avoiding-the-pitfalls-of-top-level-dag-code-fa480d9e75c6">Avoiding the pitfalls of top-level DAG code</a> was originally published in <a href="https://medium.com/apache-airflow">Apache Airflow</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>