<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pedram Navid on Medium]]></title>
        <description><![CDATA[Stories by Pedram Navid on Medium]]></description>
        <link>https://medium.com/@pedram.navid?source=rss-5f81439507d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*2CRc8002AmzLuDhQ7f2Wgg.png</url>
            <title>Stories by Pedram Navid on Medium</title>
            <link>https://medium.com/@pedram.navid?source=rss-5f81439507d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 18:45:31 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pedram.navid/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Writing Your First dbt Package]]></title>
            <link>https://medium.com/hex-blog/writing-your-first-dbt-package-4d1013696cf5?source=rss-5f81439507d------2</link>
            <guid isPermaLink="false">https://medium.com/p/4d1013696cf5</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[package]]></category>
            <dc:creator><![CDATA[Pedram Navid]]></dc:creator>
            <pubDate>Fri, 11 Nov 2022 20:26:08 GMT</pubDate>
            <atom:updated>2022-11-11T20:26:08.629Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A comprehensive guide on how to create dbt packages with working code examples, and an open-source repo that you can build from.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s8FzftHI5qaDIzyW8YvnfA.png" /></figure><p>If you’ve spent enough time working with dbt or in the dbt community, you’ve likely come across dbt packages. Your first interaction with one of these packages might have been the <a href="https://hub.getdbt.com/dbt-labs/dbt_utils/latest/">dbt utils</a> package or one of the many packages <a href="https://www.fivetran.com/blog/fivetran-dbt-packages">Fivetran</a> provides for their data models. But if you’re like me, your curiosity is never satisfied, and you soon find yourself wondering how to create your own dbt package. That’s what we’ll be exploring today.</p><p>We’ll cover everything from the different types of packages and what you need to build one to more subtle details like how a package works under the hood and the best development workflow.</p><h3>Types of dbt packages</h3><p>Two types of dbt packages are generally available:</p><ul><li>Internal packages built for use cases within your organization</li><li>External packages that are more generalized for broader users across companies.</li></ul><p>Internal packages can be beneficial for helping spread the adoption of best practices and for making the lives of your peers better.</p><p>For example, some ideas for internal packages might be bespoke tests or custom macros for repetitive tasks such as parsing and cleaning website URLs and emails. Internal packages can take advantage of your business domain context without the overhead of making a solution overly generalized and customizable.</p><p>External packages take what you’ve learned internally and go one step further. They can take many forms, and all you need to do is browse <a href="https://hub.getdbt.com/">dbt’s package hub</a> to get a sense of what’s possible; from code generation to artifact creation to modeling, there are many ways to build valuable packages for the community. One caveat is that building an external package often means building in a more generalized and abstract way, which can be tricky at times, but hopefully, this guide will help you get started.</p><h3>When should you write a dbt package, and why?</h3><p>Not every idea needs a dbt package, and abstractions can be costly. For example, if you’re a small team working out of one dbt repo, you might not need a dbt package since you can add macros and tests directly to your project repository.</p><p>However, a package might be a good idea if you have multiple projects within your company or you are building something unique that you think is broadly helpful and doesn’t involve business-specific logic. When creating a package, you open up your work to the broader community, which can help others facing a similar problem and even encourage contributions from the <a href="https://www.getdbt.com/community/">dbt community</a>!</p><h3>Basic dbt package concepts</h3><p>At its core, a dbt package is a dbt project that can be installed as a module within another dbt project and referenced. To install a package, you define it in your packages.yml file. You can install packages from dbt hub, git, or a local folder on your computer (great for local development).</p><pre># packages.yml<br>packages:<br> - package: dbt-labs/snowplow<br> version: 0.7.0</pre><pre> - git: &quot;https://github.com/dbt-labs/dbt-utils.git&quot;<br> revision: 0.1.21</pre><pre> - local: /opt/dbt/redshift</pre><p>Once you run dbt deps, dbt will install the packages specified in your packages.yml file to your dbt_packages directory.</p><p>Packages are namespaced, similar to your main dbt project. In your dbt_project.yml file, you declare a name for your package. You also use that package name to configure your models. The same principle applies to packages you install.</p><pre># dbt_project.yml<br>name: jaffle_shop<br>models:<br>   jaffle_shop:<br>   	 sales:<br> 		+schema: sales<br>    my_installed_package:<br>	+schema: my_package</pre><p>You can extend this logic to variables as well. Variables can be defined globally or tied to a particular package. We’ll touch on variables and how they can enhance your packages shortly.</p><pre>vars:<br>my_global_var: &#39;i_love_cookies&#39;<br>  jaffle_shop:<br>    start_date: &#39;2022-01-01&#39;<br>  my_installed_package:<br>	platforms: [&#39;web&#39;, &#39;mobile&#39;]</pre><h3>Writing a package</h3><p>When learning to write packages, there are a couple of approaches. First, it’s helpful to learn from other packages. Fortunately, there are many open-source dbt packages that you can use as a starting point to understand how they’re built — take a spin through the <a href="https://hub.getdbt.com/">dbt hub</a> and crack open the hood on a few interesting looking packages!</p><p>The second approach is to just dive right in and write a package, which we’ll do now. In this example, we’ll pretend that we are a software company called Penta that writes data in a standard format to a data warehouse. We want to build a dbt package for our customers so they can take advantage of modeled views based on the raw data our application feeds in.</p><h3>Creating a dbt Package Project</h3><p>We’ll start by creating a new dbt project.</p><pre>dbt init dbt-penta</pre><p>This command will create a project skeleton. First, we’ll update the dbt_project.yml file with a barebones schema. Next, we&#39;ll also define the schema where we want to write our models by default.</p><pre># dbt_project.yml<br>name: &quot;dbt_penta&quot;<br>	version: &quot;0.1.0&quot;<br>	config-version: 2<br>	require-dbt-version: [&quot;&gt;=1.0.0&quot;, &quot;&lt;2.0.0&quot;]<br>models:<br>	dbt_penta:<br>		+schema: Penta</pre><h3>Defining the Raw Data</h3><p>Like any good project, we first need to define where our raw data is. Since users may have custom locations for data ingestion, it’s best to use a variable to allow users to set their schema and table names. We will be good data citizens and also add some documentation to our tables.</p><pre>version: 2<br>	sources:<br>	 - name: penta_raw<br>	 schema: &quot;{{ var(&#39;penta_raw_schema&#39;, &#39;Penta&#39;) }}&quot;<br>	 database: &quot;{{ var(&#39;penta_raw_database&#39;, target.database) }}&quot;<br>	 tables:<br>	 - name: changelog<br>	 description: |<br>	 This table contains a row for every operation performed by Penta<br>	 It includes the result of the operation, as well as any error messages.<br>	 columns:<br>	 - name: id<br>	 description: The id of the change<br>	 - name: event<br>	 description: The event that caused the change<br>	 - name: meta<br>	 description: Additional metadata about the change<br>	 - name: op_type<br>	 description: |<br>	 Whether the row was added, changed, or removed relative to the last run.</pre><p>You can imagine repeating this process for every raw table and dataset’s columns.</p><p>From here, you would continue to build your package as if it was a typical dbt project. Use the same dbt best practices, such as creating staging tables that reference source tables and making sure to only reference staging tables in your downstream models.</p><pre># stg_penta__changelog.yml</pre><pre>with base as (<br>select * from {{ source(&#39;penta_raw&#39;, &#39;changelog&#39;) }}<br>),</pre><h3>Consider compatibility issues</h3><p>Recall that your users may use different warehouses, so using the dbt_utils package for cross-database compatible functions is essential. For example, datediff has different arguments depending on the underlying warehouse, so a more compatible approach might look like this:</p><pre>select<br>{{ dbt_utils.datediff(&#39;started_at&#39;, &#39;finished_at&#39;, &#39;second&#39;) }} as sync_elapsed_seconds<br>...</pre><h3>Using variables in your models</h3><p>You can also use variables to control the behavior of your models. For example, you may wish to allow your users to disable specific models:</p><p>In your model, set a model as enabled based on the presence of a variable like this:</p><pre>{{ config(<br>	 enabled = not var(&#39;penta_complex_models_disabled&#39;, False)<br>	 )<br>	 }}<br>...</pre><h3>Local testing</h3><p>Once you’ve built and documented your models, you can test them locally. The easiest way is to use your existing dbt project as a test bed. First, add a local definition in your dbt project (not in your new package).</p><pre># packages.yml<br>packages:<br>	- local: /Users/pedram/my_cool_packages/packages/penta</pre><p>Then run dbt deps to ensure the local version installs and works as expected. Next, test out your variables to make sure they function too. Finally, if everything looks okay, you&#39;re ready for the scariest part, deploying it in the wild! (You&#39;ll do great, I promise.)</p><h3>Publishing your package</h3><p>The final step is to make your package available to the public. You’ll want to create a helpful README that describes what your package does. Here’s a great example of what a README looks like by the folks at Fivetran.</p><p>If you really want to go the extra mile, you can create a Hex project that showcases your dbt package in action for bonus points! Here’s an example of just that, from the Hightouch team. Get inspired!</p><p>Next, create a GitHub repo and add the contents of your package directory. Once you’ve pushed your repo to Github, people can now install it directly from git, similar to how you specified a local folder above:</p><pre>packages:<br> - git: &quot;https://github.com/penta-labs/dbt-penta.git&quot; # git URL<br> 	revision: main # tag or branch name</pre><p>Once pushed, you’ll want to test this again to ensure your package still installs. Then, if everything looks good, you’re ready to <a href="https://docs.github.com/en/repositories/releasing-projects-on-github/managing-releases-in-a-repository">cut a release</a>. Github makes it straightforward to create a named release, and I’d encourage you to read up on <a href="https://semver.org/">Semantic Versioning</a> to help your users understand what changes might break compatibility with previous versions.</p><p>If everything looks good, the last step is to release your package on <a href="https://github.com/dbt-labs/hubcap">dbt hub by creating a Pull Request</a>. Take a look at <a href="https://github.com/dbt-labs/hubcap/pull/120/files">this PR</a> if you need some inspiration; only a few lines of code are needed.</p><h3>Wrapping up</h3><p>Once you’ve published your package, it’s time to celebrate. If your package helped solve a problem you face, odds are it’ll help someone else too, so let the community know on <a href="https://www.getdbt.com/community/join-the-community/">dbt Slack</a> to encourage adoption and feedback. If you get stuck anywhere, the community is very helpful as well. I hope this was a useful tour of the dbt package ecosystem. You’re well on your way to being a dbt expert.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d1013696cf5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/hex-blog/writing-your-first-dbt-package-4d1013696cf5">Writing Your First dbt Package</a> was originally published in <a href="https://medium.com/hex-blog">The Hex Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dagster, Airflow and Prefect: A Deep Dive]]></title>
            <link>https://medium.com/data-science/airflow-prefect-and-dagster-an-inside-look-6074781c9b77?source=rss-5f81439507d------2</link>
            <guid isPermaLink="false">https://medium.com/p/6074781c9b77</guid>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[dagster]]></category>
            <category><![CDATA[prefect]]></category>
            <category><![CDATA[etl]]></category>
            <dc:creator><![CDATA[Pedram Navid]]></dc:creator>
            <pubDate>Mon, 10 Jan 2022 21:59:42 GMT</pubDate>
            <atom:updated>2024-05-17T22:57:15.215Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JUnRi0Yl2YQqZCiV" /><figcaption>Photo by <a href="https://unsplash.com/@sortino?utm_source=medium&amp;utm_medium=referral">Joshua Sortino</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><em>editor’s note: this post was updated on May 17 2024</em></p><p>One of the great things about the Modern Data Stack is the interoperability with all the different components that make up the stack, but the unfortunate truth is that orchestration is still a tricky problem. Data pipelines still involve custom scripts and logic that don’t fit perfectly into the ETL workflow. Whether it’s custom internal services, or something as simple as downloading a file, unzipping it, and reading its contents, there’s still a need for orchestration tools.</p><p>While the Modern Data Stack provides a lot of flexibility with its various components that play well together, orchestrating workflows across the stack still isn’t a solved problem.</p><p>Enter Orchestration tools like Dagster, Apache Airflow, and Prefect. These tools are the bread and butter of data engineering teams. Apache Airflow, the oldest of the three, is a battle-tested and reliable solution that was born out of Airbnb and created by Maxime Beauchemin. Data engineering was a different world back then, largely focused on regularly scheduled batch jobs that often involved ugly systems with words like Hive and Druid. You still see a lot of this heritage in Airflow today.</p><p>Prefect and Dagster are newer products supported by their cloud offerings, Prefect Cloud and Dagster Plus. Prefect Cloud is free to start and hosts a scheduler, while the hybrid architecture allows you to run your tasks locally or on your infrastructure. Dagster Plus offers both hybrid and serverless options, which allows you to have the flexibility of either running workers on your internal clusters or offloading that work to Dagster Plus.</p><h3>What is Airflow and what are its top alternatives?</h3><p>Airflow is a workflow orchestration tool used for orchestrating distributed applications. It works by scheduling jobs across different servers or nodes using DAGs (Directed Acyclic Graphs). Apache Airflow provides a rich user interface that makes it easy to visualize the flow of data through the pipeline. You can also monitor the progress of each task and view logs.</p><p>If you’ve ever been confused by how dates work in Airflow, you’ll have seen some of that heritage. It’s rare that someone new to Airflow doesn’t get confused by what an execution date is and why their DAG isn’t running when they expect it to. All of this bred from the days of running daily jobs sometime after midnight.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*shXSY68nEp6eiDNv.gif" /><figcaption>A look at Airflow 1.4 from many years ago, source:<a href="https://github.com/apache/airflow/tree/efd9e4c4f5dc413308fa958a8038240a38840f67"> Airflow on Github</a></figcaption></figure><p>Airflow is now an Apache project, and its adoption by the Apache Foundation has cemented the project&#39;s status as the status-quo open-source orchestration and scheduling tool. Today, thousands of companies use Airflow to manage their data pipelines, and you’d be hard-pressed to find a major company that doesn’t have a little Airflow in their stack somewhere. Companies like Astronomer and AWS even provide managed Airflow as a Service, so that the infrastructure around deploying and maintaining an instance is no longer a concern for engineering teams.</p><p>That said, with the changing data landscape, Airflow often encounters hurdles when it comes to testing, non-scheduled workflows, data modeling, parameterization, data transfer, and storage abstraction.</p><h3>The Benefits of Airflow</h3><p>Before we dig into some of those pitfalls, it’s important to mention some of the benefits of Airflow: Without a doubt, a project that has been around for over a decade, has the support of the Apache Foundation, is entirely open-source, and used by thousands of companies is a project worth considering.</p><p>For example, there are two orders of magnitude more questions on Stack Overflow for Airflow than any of the other competitors. Odds are if you’re having a problem, you’re not alone, and someone else has hopefully found a solution. There are also Airflow Providers for nearly any tool you can think of, making it easy to create pipelines with your existing data tools.</p><h3>The Problems with Airflow</h3><p>As the data landscape continued to evolve, data teams were doing much more with their tools. They were building out complex pipelines to support data science and machine learning use cases, ingesting data from various systems and endpoints to collect them into warehouses and data lakes, and orchestrating workflows for end-user tools across multiple data systems. Airflow was the only real orchestration tool available for a while, so many teams tried to squeeze their increasingly complex demands into Airflow, often hitting a brick wall.</p><p>The main issues we’ve seen with Airflow deployments fall into one of several categories:</p><ul><li>local development, testing, and storage abstractions</li><li>one-off and irregularly scheduled tasks</li><li>the movement of data between tasks</li><li>dynamic and parameterized workflows</li></ul><p>We’ll dive into each one of these issues by exploring how two alternative tools, Dagster and Prefect address these.</p><h3>A Look at Dagster</h3><p>Dagster is a relatively young project, started in April 2018 by Nick Schrock, who previously co-created GraphQL at Facebook. Similarly, Prefect was founded in 2018 by Jeremiah Lowin, who used his learnings as a PMC member of Apache Airflow to design Prefect.</p><p>Both projects are approaching a common problem but with different driving philosophies. Dagster takes a first-principles approach to data engineering. It is built with the full development lifecycle in mind, from development to deployment to monitoring and observability.</p><p>Prefect, on the other hand, adheres to a philosophy of negative engineering, built on the assumption that the user knows how to code. It makes it as simple as possible to build that code into a distributed pipeline, backed by its scheduling and orchestration engine. However, that simplicity can sometimes come at a cost when trying to address more complex data pipelines.</p><p>Both projects have gained considerable momentum and are rapidly improving. Let’s examine how these two projects handle some of the challenges Airflow struggles with.</p><h3>Local Development and Testing</h3><p>With Airflow, local development and testing can be a nightmare. If your production Airflow instance uses Kubernetes as an execution engine, then your local development will need Kubernetes locally. A task written with the S3Operator requires a connection to AWS S3 to run, which is not ideal for local development.</p><pre>import pandas as pd<br>from io import StringIO<br>from dagster import Definitions, asset<br>from dagster_aws.s3 import S3Resource</pre><pre><a href="http://twitter.com/asset">@asset</a><br>def upload_to_s3(s3: S3Resource):<br>    path_file = &quot;test.csv&quot;<br>    df = pd.read_csv(path_file)<br>    csv_buffer = StringIO()<br>    df.to_csv(csv_buffer)</pre><pre>bucket = s3.Bucket(&quot;my-bucket&quot;)<br>    bucket.put_object(Body=csv_buffer.getvalue(), Key=&quot;test.csv&quot;)</pre><pre>defs = Definitions(assets=[upload_to_s3], resources={&quot;s3&quot;: S3Resource()})</pre><p>With Dagster, you define your pipelines using an asset-centric approach rather than as tasks. Dagster’s modular architecture lets you provide custom resources as inputs into your pipelines that can be swapped based on environments. You may want to use S3 in production but minio in development for faster iteration.</p><p>Prefect also supports a level of abstraction on storage although through RunConfigs.</p><pre>from prefect import Flow<br>from prefect.run_configs import KubernetesRun</pre><pre># Set run_config as part of the constructor<br>with Flow(&quot;example&quot;, run_config=KubernetesRun()) as flow:<br>    ...</pre><p>However, this doesn’t provide the same level of abstraction as Dagster, which can make local development more tricky. For Prefect, parametrization is the focus of local development. By being able to parametrize your Flows, you can provide smaller datasets for local development and larger ones for production use.</p><h3>Scheduling Tasks</h3><p>In Airflow, off-schedule tasks can cause a lot of unexpected issues and all DAGs need some type of schedule. Running multiple runs of a DAG with the same execution time is not possible.</p><p>With Prefect, a Flow can be run at any time, as workflows are standalone objects. While we often wait 5–10 seconds for an Airflow DAG to run from the scheduled time due to the way its scheduler works, Prefect allows incredibly fast scheduling of DAGs and tasks by taking advantage of tools like Dask.</p><p>Similarly, Dagster allows a lot of flexibility for both manual runs and scheduled DAGs. You can even modify the behavior of a particular job based on the schedule itself, which can be incredibly powerful. For example, if you want to provide different run-time configurations on weekends vs weekdays.</p><pre>@schedule(job=configurable_job, cron_schedule=&quot;0 0 * * *&quot;)<br>def configurable_job_schedule(context: ScheduleEvaluationContext):<br>    scheduled_date = context.scheduled_execution_time.strftime(&quot;%Y-%m-%d&quot;)<br>    return RunRequest(<br>        run_key=None,<br>        run_config={&quot;ops&quot;: {&quot;configurable_op&quot;: {&quot;config&quot;: {&quot;scheduled_date&quot;: scheduled_date}}}},<br>        tags={&quot;date&quot;: scheduled_date},<br>    )</pre><p>And running a job in Dagster is as simple as:</p><pre>dagster job execute -f hello_world.py</pre><h3>Data Flow in Airflow, Prefect, and Dagster</h3><p>One of the biggest pain points with Airflow is the movement of data between related tasks. Traditionally, each task would have to store data in some external storage device, pass information about where it is stored using XComs (let’s not talk about life before XComs), and the following task would have to parse that information to retrieve the data and process it.</p><p>In Dagster, the inputs and outputs of jobs can be made much more explicit.</p><pre>import csv</pre><pre>import requests<br>from dagster import get_dagster_logger, job, op</pre><pre>@op<br>def download_cereals():<br>    response = requests.get(&quot;&lt;https://docs.dagster.io/assets/cereal.csv&gt;&quot;)<br>    lines = response.text.split(&quot;\\n&quot;)<br>    return [row for row in csv.DictReader(lines)]</pre><pre>@op<br>def find_sugariest(cereals):<br>    sorted_by_sugar = sorted(cereals, key=lambda cereal: cereal[&quot;sugars&quot;])<br>    get_dagster_logger().info(<br>        f&#39;{sorted_by_sugar[-1][&quot;name&quot;]} is the sugariest cereal&#39;<br>    )</pre><pre>@job<br>def serial():<br>    find_sugariest(download_cereals())</pre><p>The above example clearly shows that the download_cereals op returns an output and the find_sugariest op accepts an input. Dagster also provides an optional type hinting system to enhance the testing experience, something not possible in Airflow tasks and DAGs.</p><pre>@op(out=Out(SimpleDataFrame))<br>def download_csv():<br>    response = requests.get(&quot;&lt;https://docs.dagster.io/assets/cereal.csv&gt;&quot;)<br>    lines = response.text.split(&quot;\\n&quot;)<br>    get_dagster_logger().info(f&quot;Read {len(lines)} lines&quot;)<br>    return [row for row in csv.DictReader(lines)]</pre><pre>@op(ins={&quot;cereals&quot;: In(SimpleDataFrame)})<br>def sort_by_calories(cereals):<br>    sorted_cereals = sorted(cereals, key=lambda cereal: cereal[&quot;calories&quot;])<br>    get_dagster_logger().info(<br>        f&#39;Most caloric cereal: {sorted_cereals[-1][&quot;name&quot;]}&#39;<br>    )</pre><p>In Prefect, inputs and outputs are also clear and easy to wire together.</p><pre>with Flow(&quot;Aircraft-ETL&quot;) as flow:<br>    reference_data = extract_reference_data()<br>    live_data = extract_live_data()</pre><pre>    transformed_live_data = transform(live_data, reference_data)</pre><pre>    load_reference_data(reference_data)<br>    load_live_data(transformed_live_data)</pre><p>The transform function accepts the outputs from both reference_data and live_data. For large files and expensive operations, Prefect even offers the ability to cache and persist inputs and outputs, improving development time when debugging.</p><h3>Dynamic Workflows</h3><p>Another great feature of both Dagster and Prefect that is missing in Airflow is an easy interface for creating dynamic workflows.</p><p>In Prefect, parameters can be explicitly specified in the Cloud Interface or provided to the Flow runner. This makes scaling out to large complex computations easy allowing for sane initial development as you work on your pipelines.</p><p>In Dagster, you can define a graph and then parametrize it to allow for dynamic configurations, which allows you to fully customize resources, configurations, hooks, and executors.</p><pre>from dagster import graph, op<br>from dagster import ResourceDefinition</pre><pre>@op(required_resource_keys={&quot;server&quot;})<br>def interact_with_server(context):<br>    context.resources.server.ping_server()</pre><pre>@graph<br>def do_stuff():<br>    interact_with_server()</pre><pre>prod_server = ResourceDefinition.mock_resource()<br>local_server = ResourceDefinition.mock_resource()</pre><pre>prod_job = do_stuff.to_job(resource_defs={&quot;server&quot;: prod_server})<br>local_job = do_stuff.to_job(resource_defs={&quot;local&quot;: local_server})</pre><h3>Wrapping Up</h3><p>I hope this was a helpful exploration of some of the new orchestration tools that have started to gain traction in the data landscape. Despite the shortcomings of Airflow, it is still a solid and well-architected platform that serves many people well. However, competition in this space will only help improve all the tools as they learn and improve from each other. I’m excited to see how this space develops and would love to know what you think.</p><p>Which scheduler are you using? Do you have plans on migrating off Airflow? Let me know by commenting below or reaching out on <a href="https://twitter.com/@pedram_navid">Twitter</a> or by <a href="/service/mailto:pedram@hightouch.io">email!</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6074781c9b77" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/airflow-prefect-and-dagster-an-inside-look-6074781c9b77">Dagster, Airflow and Prefect: A Deep Dive</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>