Apache Airflow Components: Explained

A practical breakdown of the components that power workflow orchestration in Airflow.

Apache Airflow is an open-source workflow orchestration framework used to develop, schedule, and monitor data pipelines. There are several components work together to parse workflows, schedule tasks, execute them, store their state, and communicate with external systems.

1. Metadata:

Metadata is the data about data. Airflow needs to store all the metadata about the DAG, such as DAG runs, schedules, status, task instances, etc.

2. DAG file processor:

DAG file processing refers to the system mechanism in Airflow that reads, parses, and converts workflow definition code into active, scheduled task objects.

3. API server:

An Airflow API server in Airflow is a web service that lets external programs talk to it. It accepts safe commands, like starting a data task or checking its status. You can use it to run pipelines from other apps without opening the main Airflow screen

Example:

• Imagine an online store app, and you want to run an Airflow data task called update_sales_report the moment a customer buys an item:

A customer purchases from a store website/app → the app/website triggers code to send a web command to your Airflow API server: POST /api/v1/dags/update_sales_report/dagRuns → The Airflow API server gets the message, checks if your website is allowed, and starts the update_sales_report task right away.

4. Scheduler:

The scheduler is responsible for WHAT & WHEN the tasks are to be executed. A scheduler continuously reads the workflow files, checks task dependencies, and triggers execution at the correct time while managing when the job runs but handing off the actual work to an executor.

Get Mufeedkcp’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Example:
→ The scheduler reads the Python scripts.
→ If it sees schedule="@daily", or schedule="@hourly", or schedule="@weekly", or schedule="@monthly".
→ It waits until the time criteria are met, then triggers the task automatically.

5. Executor:

• As the name suggests, Executor doesn't execute the DAG at all; rather, the Scheduler determines WHEN a task is ready; the Executor determines HOW that task is executed. While the scheduler figures out when a task is ready based on time and dependencies, the Executor determines HOW tasks are executed by determining the execution mechanism and how task instances are launched.
• Example: Think of the Executor as the logistics manager of Airflow. It figures out exactly how to run your tasks, choosing the right machinery and routing each job to the right place.

6. Workers:

• Workers are the components that actually execute the tasks. The executor is just a strategy layer that defines how tasks are submitted for execution.
• Example:

→ Imagine a simple workflow (DAG):
	→ The Scheduler decides it is time to run the task and sends a message to the queue.
	→ The Worker picks up that message from the queue.
	→ The Worker runs the actual Python command on its machine.

7. Queue:

• A queue acts like a waiting line for tasks that are ready to run. When tasks are ready to execute, they are placed into a queue so that workers can pick them up according to the execution architecture
• Example: Think of it like a restaurant: the scheduler is the waitress taking orders, the queue is the ticket spike holding them, and the workers are the kitchen staff cooking the food.

8. Triggerer:

• A triggerer is a background helper worker. It allows waiting tasks to release their heavy worker slots, saving compute power and cost, while waiting for external events to complete.
• Example:
  → Imagine a normal worker: it starts a task and stays busy until it ends. If the task waits for 5 hours, that worker is locked and does nothing else. A Triggerer handles tasks that need to pause and wait. It checks if the wait is over. When the wait ends, it wakes up the main system to finish the job. This results in the system saving computer power and money because it needs fewer active workers.