PHASE 2: Local Machine set up

PHASE 2.1 — Install Required Software

We use 4 main tools (what you called APKs):

1. Python🐍
2. Docker Desktop🐳
3. Google Cloud SDK☁️
4. Astronomer CLI🚀

Each one has a clear role in the data pipeline.

1️⃣ Python — The Language of the Pipeline

Why we use Python

• Airflow DAGs are written in Python
• PySpark code uses Python
• Most data engineering tools support Python

Interview one-liner

“Python is used to write Airflow DAGs and Spark transformation logic.”

2️⃣ Docker Desktop — Run Everything Safely

Why we use Docker

• Airflow has many dependencies
• Avoids “works on my machine” problems
• Runs Airflow in isolated containers

Interview one-liner

“Docker provides a consistent runtime environment for Airflow.”

3️⃣ Google Cloud SDK — Connect Local Code to GCP

Why we use gcloud

• Authenticate your laptop with GCP
• Allow Airflow/Spark to access:
• GCS
• BigQuery

Simple explanation

It allows your local machine to talk to Google Cloud securely.

Analogy

🔐 gcloud = ID card to enter Google Cloud

4️⃣ Astronomer CLI — Airflow Made Easy

Why we use Astronomer CLI

• Simplifies Airflow setup
• Runs Airflow using Docker
• Industry-standard Airflow tooling

Simple explanation

Astronomer CLI is a helper tool to create and run Airflow easily.

Analogy

🚗 Astronomer CLI = Automatic gear for Airflow

🧱 How These Tools Work Together (IMPORTANT)

Full Flow

Python code

↓

Astronomer CLI

↓

Docker containers

↓

Airflow scheduler

↓

GCP (GCS + BigQuery)

📌 This diagram is INTERVIEW GOLD.

🧠 PERFECT INTERVIEW ANSWER (2–3 Lines)

“We use Python to write DAGs and Spark logic, Docker to run Airflow in isolated containers, Google Cloud SDK for secure GCP authentication, and Astronomer CLI to simplify Airflow setup and execution. Together, they enable local development and cloud-ready data pipelines.”

PHASE 2.2 — Local Project Folder Structure

🎯 Goal of this phase

Create a clean project structure that:

• Airflow understands
• Astronomer expects
• Matches real industry projects

📌 Interview line:

“Before writing DAGs or Spark code, we set up a proper project structure.”

🪜 STEP-BY-STEP (DO EXACTLY THIS)

✅ Step 1: Create a main project folder

On your system (any drive):

Create a folder named:

retail-data-pipeline

📌 This is your root project folder

✅ Step 2: Open this folder in Command Prompt

1. Go inside retail-data-pipeline
2. Click on the address bar
3. Type cmd
4. Press Enter

Your command prompt path should look like:

C:\…\retail-data-pipeline>

✅ Step 3: Initialize Astronomer Project (VERY IMPORTANT)

Run this command:

astro dev init

⏳ Wait for it to finish (1–2 minutes)

✅ What Just Happened? (Beginner Explanation)

Astronomer CLI automatically created Airflow-ready files for you.

This saves hours of manual setup.

📁 Expected Folder Structure (VERY IMPORTANT)

After astro dev init, you should see:

retail-data-pipeline/
├── dags/
 └── example_dag.py
├── plugins/
├── Dockerfile
├── requirements.txt
├── airflow_settings.yaml
└── .env

📌 If you see this → you are 100% correct

🧠 Why This Step Is Important (Interview Ready)

“Astronomer CLI initializes a standardized Airflow project structure, making it easy to run and deploy DAGs consistently.” — It will make airflow ready files for you.

Why each is for?

PHASE 2.3 — Prepare code structure & configs

🪜 STEP 1: Understand the Current Folder Structure

Inside your retail-data-pipeline folder, you now have:

retail-data-pipeline/
├── dags/
├── plugins/
├── Dockerfile
├── requirements.txt
├── airflow_settings.yaml
└── .env

🪜 STEP 2: Clean the Example DAG (IMPORTANT)

Astronomer creates a sample DAG by default.

Do this:

1. Go to dags/
2. You’ll see something like:
3. example_dag.py
4. Delete it ❌

📌 Why?

We don’t want demo DAGs interfering with our project.

Interview line:

“I remove default example DAGs to avoid unnecessary scheduling.”

WHAT IS A DAG?

DAG stands for Directed Acyclic Graph.

1️⃣ Graph (simple meaning)

A graph is just:

• Nodes → things/tasks/steps
• Edges (arrows) → connections between them

Think of it like boxes connected with arrows.

2️⃣ Directed

Get Stephy Thomas’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Directed means:

• The arrows have a direction
• You go from one step to the next

➡️ Example:
Task A → Task B
(A must finish before B starts)

3️⃣ Acyclic

Acyclic means:

• No cycles
• You can never come back to a previous step

❌ Not allowed:
A → B → C → A (this is a cycle)

✅ Allowed:
A → B → C

🔹 Putting it all together

A DAG (Directed Acyclic Graph) is:

A set of tasks connected by arrows,where each task has a direction, and there are no loops or repeats.

📌 Why DAG is important (real-world use)

🔸 In Data Engineering (Airflow, Spark, etc.)

A DAG defines:

• What task runs first
• What depends on what
• The execution order

Example (ETL pipeline):

Start
  ↓
Extract Data
  ↓
Transform Data
  ↓
Load Data

This is a DAG because:

• Direction exists ✅
• No task loops back ✅

🛠 DAG in Apache Airflow

In Airflow:

• A DAG represents a workflow
• Each node = a task
• Arrows = dependencies

👉 Airflow does not process data
👉 It only manages and schedules DAGs

🪜 STEP 3: Create Our Project DAG File

Inside the dags/ folder:

Create a new file named:

retail_pipeline_dag.py

⚠️ Don’t write code yet — just create the file.

📌 This file will:

• Orchestrate ingestion
• Trigger Spark transformation
• Load data to BigQuery

🪜 STEP 4 — Create the scripts/ Folder (DO THIS NOW)

What to do

Inside your project root folder:

1. Create a folder named:

scripts

2. Inside scripts/, create a file:

retail_transform.py

📌 Don’t write code yet.

Why this step exists

This is where PySpark transformation logic will live.

🪜 STEP 5 — Create data/ Folder with Sample Files

What to do

In the project root:

1. Create a folder:

data

2. Inside data/, place:

• customers.json
• products.json
• orders.json

(You already downloaded these earlier.)

Why this step exists

Allows local testing without touching cloud resources.

Interview line:

“Local sample data enables faster development and debugging.”

🪜 STEP 6 — Update requirements.txt (VERY IMPORTANT)

Open requirements.txt and add these lines:

apache-airflow-providers-google
pyspark
google-cloud-storage
google-cloud-bigquery

📌 Do not remove existing content
📌 Just append these lines

Why this step exists

Docker installs dependencies when Airflow starts.

Its like a shopping list of your python libraries which is needed for your project, before the code is being executed it first reads requirements.txt then installs everything listed.

📌 If a library is not in requirements.txt → Airflow cannot use it

Why each library is using?

1️⃣ apache-airflow-providers-google

Why do we need this?

Airflow by default:

• Cannot talk to GCS
• Cannot talk to BigQuery

This package provides:

• GCS operators
• BigQuery operators
• GCP hooks

📌 Without this → Airflow cannot access GCP

“Airflow providers enable integration with external systems like GCP.”

2️⃣ pyspark

Why do we need this?

Your transformation logic:

• Uses Spark
• Uses Spark DataFrames
• Uses Spark functions

📌 Without this → Spark code will crash

“PySpark is required to perform distributed data transformations.”

3️⃣ google-cloud-storage

Why do we need this?

Your pipeline will:

• Read raw data from GCS
• Write processed data to GCS

This library allows Python to:

• Access buckets
• Read/write objects

📌 Without this → Python cannot talk to GCS

“The GCS client library enables programmatic access to Cloud Storage.”

4️⃣ google-cloud-bigquery

Why do we need this?

Final step of pipeline:

• Load curated data into BigQuery
• Run queries / inserts

📌 Without this → Cannot load data to BigQuery

“BigQuery client library is required for loading and querying data.”

🪜 STEP 7 — DO NOT Start Airflow Yet ❌

Even if you’re tempted — don’t.

Why?

• No DAG logic yet
• No Spark logic yet

We finish preparation first.

Now comes the exciting part — PHASE 3: PySpark Processing, where we clean 🧹, transform 🔄, and shape 🛠️ raw data into something useful 📊.Will see in the next chapter.

Consistency is the key 🔑 Little by little, day by day, you grow 📈. Happy learning!