Airflow 101 course from Astronomer Quiz Solutions Part 1
Introduction to Orchestration and Airflow
Why Orchestration?
Which of the following describes the early-computing era of Data Orchestration? (select all that apply)
Manual processes
Lack of standards around data formats, processes, and processing techniques
Rise of proprietary scheduling and workload management tools (e.g., AutoSys)
Time-based scheduling tools (e.g., Cron)
Which of the following describes the pre-unix era of Data Orchestration? (select all that apply)
Manual and error-prone processes
Automated Processes
Lack of standards around data formats, processes, and processing techniques
Dedicated ETL tooling
Which of the following describes the data and open-source era of Data Orchestration? (select all that apply)
Manual batch processing
An increase in data size and complexity of scheduling and workloads
Tools that were designed for specific ecosystems (e.g., Hadoop)
Adoption of proprietary data formats and protocols to restrict access to data ecosystems
Which of the following is the definition of modern data orchestration?
The process of manually handling data flow across different platforms to ensure data security and compliance.
The coordination and automation of data flow across various tools and systems to deliver quality data products and analytics.
The random distribution and storage of data across various databases without any automated processes.
The exclusive use of a single tool to manage all data-related tasks within an organization.
Introduction to Airflow
Why is Airflow considered the central nervous system of a data stack?
It functions as a data visualization platform.
It acts as a standalone tool for data analysis
It connects, coordinates, and manages different tools and systems across a organization’s data ecosystem
It provides data storage solutions for large datasets.
Which of the following are characteristics of Airflow? (select all that apply)
Extensibility to connect to external systems
The ability to write pipelines-as-code with Python
Built-in observability features
Proprietary and closed-source
Who uses Airflow and for what?
Which of the following data practitioner personas would most likely use Airflow to automate processes related to retraining and validating machine learning models?
Data Analysts
Data Scientists
Machine Learning Engineers
Data Engineers
Which of the following data practitioner personas would most likely use Airflow to automate the process of pre-processing and analyzing new data sets?
Data Analysts
Data Scientists
Data Engineers
Machine Learning Engineers
Which of the following data practitioner personas would most likely use Airflow to productionize SQL queries to create dashboards and reports?
Data Engineers
Data Scientists
Data Analysts
Machine Learning Engineers
Which of the following data practitioner personas would most likely be primarily responsible for using Airflow to build, maintain, and manage most or all data pipelines for an organization?
Machine Learning Engineers
Data Scientists
Data Analysts
Data Engineers
Imagine that you are working for a startup that wants to streamline customer support by creating an AI chatbot that would give tips and solutions based on customer errors and actions. Is this a use case for Airflow?
Yes
No
Considerations of using Airflow
Which of the following would most likely be the root cause of errors when using Airflow for data processing? (select all that apply)
Resource allocation
A dedicated streaming solution (e.g., Apache Kafka) integrated with Airflow
Infrastructure
Number of data pipelines created
Which of the following statements are true about Airflow? (select all that apply)
Airflow was designed to be used as a dedicated streaming system
Airflow can preform data processing
Airflow is a scalable solution for data orchestration
Fill in the blank:
Airflow was designed to be used for _______ and not streaming.
batch processing
Running Airflow
Which of the following are most likely the main requirements for running Airflow for production in a multi-team environment? (select all that apply)
Low cost
High availability
Scalability
Multi-tenancy
Which of the following are most likely the main requirements for running Airflow for production as a single developer? (select all that apply)
Multi-tenancy
High levels of scalability
Low infrastructure overhead
Low cost
Which of the following is the most likely way a developer or team is going to run Airflow if they have to adhere to strict security or regulatory guidelines?
On a single developers machine
On-premise
On a cloud software
Which of the following are the most likely ways Airflow will be run in production by a single team? (select all that apply)
On a container cloud offering (e.g., EKS)
Get Shalini’s stories in your inbox
Join Medium for free to get updates from this writer.
On a single developers machine
On a fully-managed Airflow service (e.g., Astro)
On cloud virtual machines (e.g., EC2)
Which of the following is the most likely way a solo developer will run Airflow for the purposes of experimentation and learning?
On a multi-deployment cloud solution (e.g., EKS)
On a paid cloud virtual machine (VM) service (e.g., EC2)
On a highly secure on-premises architecture
On a local machine or a free virtual machine (VM)
How does Airflow Work?
What is a task in Airflow?
A single unit of work in a DAG
The entire data pipeline
A configuration file for how Airflow is setup on a machine
The direction the data pipeline executes
Which of the following are characteristics of a Airflow DAG? (select all that apply)
Represents multiple data pipelines
Graph based
Cyclical
Composed of Tasks
Directed
Assume you are building a DAG and want to run a Python script. What type of Operator would you use to accomplish this goal?
There are no Operators that allow a DAG to run a Python script
Sensor Operator
Action Operator
Transfer Operator
Assume you are building a DAG and want to move data from one system or data source to another. What type of Operator would you use to accomplish this goal?
Sensor Operator
Action Operator
There are no Operators that allow a DAG to move data from one system or data source to another
Transfer Operator
Fill in the blank:
In Airflow, an Operator is the specific work a _____ does.
task
Assume you are building a DAG and want to wait for a file to load before executing the next task. What type of Operator would you use to accomplish this goal?
Action Operator
There are no Operators that allow a DAG to wait for something to load
Sensor Operator
Transfer Operator
Review
Data Orchestration Fundamentals
- Data Orchestration is the coordination and automation of data flow across various tools and systems to deliver quality data products and analytics
- The path to modern data orchestration solutions like Airflow had various evolutions starting with basic time-based scheduling tools (e.g., Cron, WTS), followed by proprietary software (e.g., AutoSys, Informatica), and finally older open-source solutions (e.g., Oozie, Luigi).
Airflow for Data Orchestration
- Airflow is an open-source tool for programmatically authoring, scheduling, and monitoring data pipelines.
- Airflow is defined by key characteristics such as pipelines-as-code with Python, built-in observability, a large open-source community, and much more!
- Airflow is used across a variety of industries, data practitioners, and use-cases.
- Airflow has key considerations to account for when using it for streaming, data processing, and when it is scaled.
Demystifying Airflow
- Airflow can be used by single developers, single teams, and multi-teams. In each case, the way it is run is different.
- Airflow can be run using a variety of architecture types (e.g., on-premise) and systems (e.g., cloud services)
- Airflow key terms to remember:
- DAG: A directed acyclical graph that represents a single data pipeline
- Task: An individual unit of work in a DAG
- Operator: The specific work that a Task performs
- There are three main types of operators:
- Action: Perform a specific action such as running code or a bash command
- Transfer: Perform transfer operations that move data between two systems
- Sensor: Wait for a specific condition to be met (e.g., waiting for a file to be present) before running the next task
Final Quiz
Examine the diagram of a data pipeline in Airflow below:
Each letter label represents a key term to describe the associated component of the data pipeline. Which of the following is the correct labeling of each component?
A) DAG B) Task C) Operator
A) Task B) DAG C) Operator
A) Task B) Operator C) DAG
A) Operator B) Task C) DAG
Examine the data orchestration diagram below:
The diagram has the following labeled processes:
- A) Data ingestion of data sources
- B) Creating analytics and reporting
- C) Creating new data sets and data products
- D) Using MLFlow to productionize and manage predictive models
- Which of the following best matches each data practitioner persona to each process in the diagram?
A) ML Engineer B) Data Engineer C) Data Scientist D) Data Analyst
A) Data Engineer B) Data Scientist C) Data Analyst D) ML Engineer
A) Data Engineer B) Data Analyst C) Data Scientist D) ML Engineer
A) Data Scientist B) Data Engineer C) ML Engineer D) Data Analyst
Which of the following scenarios would be a valid use-case for Airflow? (select all that apply)
Periodic analysis on sensor data from manufacturing equipment to forecast potential failures and schedule maintenance automatically
As a dedicated system for processing real-time, high-volume data streams for a stock trading application
A health care application that schedules email reminders for patients for their appointments
An AI chatbot that gives tips and solutions based on customer errors and actions
The benefits of using a modern data orchestration tool like Airflow is because it offers the ability to…(select all that apply)
Integrate with hundreds of external systems/software
Schedule data based on events in addition to time
Author pipelines as configuration files
Use a rich feature set with features like built-in observability
Store, secure and preserve large quantities of data permanently
