<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jason O. on Medium]]></title>
        <description><![CDATA[Stories by Jason O. on Medium]]></description>
        <link>https://medium.com/@thisisjasonowens?source=rss-ecbef8df7ad4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*rq4BgDpAhLYrr30LLQjU2g.jpeg</url>
            <title>Stories by Jason O. on Medium</title>
            <link>https://medium.com/@thisisjasonowens?source=rss-ecbef8df7ad4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 04:41:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@thisisjasonowens/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Web Scraping: What It Is, Why It Matters, and How to Use It Responsibly]]></title>
            <link>https://medium.com/@thisisjasonowens/web-scraping-what-it-is-why-it-matters-and-how-to-use-it-responsibly-13867af05e79?source=rss-ecbef8df7ad4------2</link>
            <guid isPermaLink="false">https://medium.com/p/13867af05e79</guid>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[automation]]></category>
            <dc:creator><![CDATA[Jason O.]]></dc:creator>
            <pubDate>Tue, 30 Jun 2026 18:21:20 GMT</pubDate>
            <atom:updated>2026-07-11T02:57:31.645Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>In this post, I will explain what web scraping is, how it works, why it matters, and where its limitations lie. Over the past 30 years, the web has become one of the main places where commercial, public, and operational information is published. This includes product catalogues, pricing pages, business directories, marketplaces, job boards, customer reviews, public filings, technical documentation, government datasets, company reports, and open-source repositories. Access to this information can help businesses make better-informed decisions.</p><p>Web scraping can be defined as the process of extracting information from sources/websites and converting it into structured data that can be stored, analysed, searched, or used inside applications. The core idea is simple: web pages are designed primarily for people to read, not for computers to analyse at scale. Scraping bridges that gap by turning messy pages into usable records. In practice, this means taking relevant information from web pages and reshaping it into formats that can support databases, dashboards, alerts, or downstream applications.</p><p>A key thing to remember is that data is not valuable just because it exists online. It only becomes valuable when it is collected, cleaned, structured, used to answer a clear question, and maintained in a form that supports action. For a retailer, that question might be: “How are our competitors pricing equivalent items this week?” For a data team, it might be: “What external signals improve our forecasts?” For an individual, it might be: “When do these trainers drop below a target price?”</p><h3>What Web Scraping Is</h3><p>Web scraping is often confused with several adjacent concepts, so it helps to separate them clearly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xpSMMwrFNagbC3PcX5bN9w.jpeg" /><figcaption>Table 1: Web scraping compared with related data collection methods</figcaption></figure><p>In practical terms, if a script downloads a page and extracts a product title, that is scraping. If it also discovers hundreds of related pages by following links, that becomes crawling. If the site exposes a clean, documented endpoint, that is API ingestion, which should usually be preferred. If JavaScript must execute before the data appears, then browser automation becomes relevant.</p><h3>How Web Scraping Works</h3><p>At a high level, scraping usually follows six stages:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*zbW9B2kDyqQTltrgWG38FQ.jpeg" /><figcaption>Table 2: A typical web scraping workflow from request to downstream use</figcaption></figure><p>The first two stages are usually handled by an HTTP client and a parser. The Python requests library can handle the fetch, including sessions, cookies, SSL verification, proxies, streaming, and timeouts. Extraction, cleaning, storage, and analysis are then handled using the programming language, database, or automation logic of your choice.</p><p>A useful mental model is to think of scraping as an ingestion pipeline. Once extraction becomes repeatable, the output can feed dashboards, alerts, enrichment jobs, and business workflows.</p><h3>Origins and Evolution</h3><p>Web scraping grew out of an earlier web that was far more static than today’s. In the late 1990s and early 2000s, much of the work was straightforward: fetch HTML, parse tags, extract text, and move on.</p><p>As the web shifted toward forms, sessions, cookies, client-side rendering, and API-driven front ends, those basic fetch-and-parse methods started to break down. The data was no longer always sitting inside the original HTML response. Sometimes it depended on a user session. Sometimes it loaded after JavaScript ran. Sometimes it appeared only after clicking through pages, submitting forms, or making background API requests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tNjjCfEd_h2l733TVwolCQ.jpeg" /><figcaption>Figure 1: Scraping has moved from simple static HTML parsing toward stateful, JavaScript-aware, API-driven, and monitored collection pipelines.</figcaption></figure><p>As a result, scraping evolved from simple page parsing into a broader data collection process. Modern scrapers may need to preserve state, move through pages, retry failed requests, control request rates, store historical outputs, and sometimes run a real browser. That shift is why the scraping ecosystem now includes lightweight HTML parsers, crawling frameworks, browser automation tools, and production pipeline components.</p><p>In that sense, scraping is one of the few areas of software where the target keeps moving. Websites, platform owners, and anti-bot systems are constantly changing how content is delivered and protected. Scrapers, data providers, and automation engineers then adapt their collection methods in response. The result is a constant back-and-forth: websites change how data is exposed, and scraping systems evolve to keep public web data accessible, structured, and useful.</p><h3>Levels of Difficulty</h3><p>In practice, scraping projects tend to fall into four broad levels of difficulty.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JY__4bCTdw2IPDUCJ3lv3Q.jpeg" /><figcaption>Table 3: Scraping complexity increases from static-page extraction to production pipelines and browser infrastructure at scale.</figcaption></figure><h4>Level 1</h4><p>Level 1 is the static-page case. The useful data is already present in the HTML returned by the server. A typical stack here is Requests plus Beautiful Soup, optionally with pandas or CSV output. Requests handles the fetch, and Beautiful Soup handles the parse tree and selectors.</p><p>Typical tasks at this level include pulling titles, links, dates, prices, headings, and simple table fields from one page or a small batch of pages. The main challenges are selector accuracy, inconsistent markup, and basic error handling.</p><h4>Level 2</h4><p>At Level 2, web scraping operations start to span multiple pages or deal with pagination, search results, filters, cookies, repeated requests, or JavaScript-rendered content. This is where frameworks such as Scrapy, Playwright, or Selenium become useful, depending on the type of site being scraped.</p><p>Typical tasks at this level include category pages, job boards, property listings, multi-step discovery, and pages where the content only appears after rendering or interaction.</p><h4>Level 3</h4><p>Level 3 is where scraping becomes a data pipeline rather than a script. The focus moves from “Can I grab the information once?” to “Can this system run reliably over time?”</p><p>This is where logging, telemetry, monitoring, and orchestration become important. At this level, you start thinking about failed runs, retries, alerts, quality checks (think CI/CD), and whether the system can be scheduled and maintained without constant manual intervention. Tools such as Airflow, Prefect, and Dagster can be useful here because they help manage recurring jobs, dependencies, and pipeline visibility.</p><p>Reproducibility also becomes more important. Docker can be used to package the scraper and its dependencies so the same job can run consistently across different machines, environments, or deployment targets.</p><h4>Level 4</h4><p>The final level is where scraping becomes more focused on browser infrastructure rather than simply using existing automation tools. At this point, you are typically designing custom scraping tools instead of relying on existing frameworks. For example, you may fork a version of Playwright and adjust it for your own needs, or create your own unique/bespoke solution.</p><p>This is where the idea of custom browser environments comes in. In this context, a custom browser environment means a controlled browser setup built around an open-source browser such as Chromium or Firefox. This may involve patching a copy of a Chromium browser at the binary level, or using specific profiles, settings, dependencies, containers, and execution rules chosen so the browser behaves consistently across runs. The goal is to make large-scale collection more stable, reproducible, and observable.</p><p>Concepts such as remote browser execution, isolated profiles, browser pools, containerisation, queues, orchestration, and lower-level browser engineering become more relevant at this stage.</p><p>A stronger understanding of browser behaviour, infrastructure, and lower-level languages such as C++ can also become useful, but that is something that would require a deeper dive.</p><h3>Why It Matters</h3><p>The strongest reason to care about scraping is business value. Relevant public information, collected at scale, can be converted into competitive intelligence. A company that can track competitor pricing, assortment changes, review volume, hiring patterns, availability shifts, or public-record updates more consistently than its peers can often make faster and better decisions.</p><p>These decisions can be made by feeding said to data science and analytics teams within an organization. Public web data can provide external signals: competitor movement, market sentiment proxies, labor-market patterns, product launches, geographical expansion, or customer-experience signals. This allows scraping to become part of an enrichment or monitoring pipeline rather than a standalone project. At the enterprise level, the real output is usually a time series, a reference dataset, or a dashboard input, not a pile of scraped pages.</p><p>Smaller organisations and individuals can use the same basic pattern at a different scale. A small retailer may monitor a handful of competitor items rather than thousands. A freelancer may track niche job listings or public directories. An individual might write a script to check whether a favourite artist has announced a show in their city, whether a coat is back in stock, whether a particular trainer has dropped in price, or whether a new property listing fits a location filter.</p><blockquote><strong><em>A Real Business Case: Competitor Scraping and Legal Risk</em></strong></blockquote><blockquote><em>A useful case study is Compulife Software, Inc. v. Newman, a </em><strong><em>2024 U.S. federal court case</em></strong><em> involving life-insurance quote data. Compulife generated quotes online through its Term4Sale website, while Newman and the other defendants operated a competing quote service.</em></blockquote><blockquote><a href="https://www.reuters.com/legal/legalindustry/acquiring-trade-secrets-through-web-scraping-is-it-misappropriation-2024-09-23/"><em>Reuters reported</em></a><em> that the defendants scraped life-insurance quote data from Compulife’s public website and used that data in their competing service. The district court found that between 3 million and 43.5 million insurance quotes had been taken. Compulife also showed that its revenue declined after the scraping, that it lost business it expected to receive, and that fewer free trials converted into paid subscriptions. The case eventually resulted in </em><a href="https://law.justia.com/cases/federal/appellate-courts/ca11/21-14071/21-14071-2024-08-01.html"><em>judgment for Compulife</em></a><em>: $184,225.87 in compensatory damages and $368,451.74 in punitive damages.</em></blockquote><blockquote><em>The lesson is not that scraping is automatically illegal. The lesson is that the right data can have real competitive value. Once that value matters, the way the data is collected, what type of data it is, how it is used, and the legal context all matter too.</em></blockquote><h3>Responsible Use</h3><p>A clear thing to understand is that just because data is publicly visible does not automatically mean it is freely reusable whenever you want. Many websites and organisations explicitly forbid scraping in their terms and conditions, and continuing anyway can lead to legal or operational problems, as seen in the case above.</p><p>Before deciding to run a scraper, a few questions should be asked.</p><ol><li><strong>Has the website explicitly forbidden scraping in its terms of service? </strong>If so proceed at your own risk, but ideally stop. Continuing to do so may result in fines or even legal penalties.</li><li><strong>Does the website have a </strong><strong>robots.txt file? <br></strong>This file gives guidance to crawlers about which parts of a site they should or should not access. It is not the same as legal permission, but it is still something you should check and respect. If a website clearly says not to access a certain path, ignoring that guidance can get your scraper blocked, your IP banned, or create bigger issues if the scraping is aggressive.</li><li><strong>Does the website provide an official API? <br></strong>If it does, it is usually better to use that instead. APIs are designed for structured access, are less brittle than scraping HTML, and usually make the rules of access clearer.</li><li><strong>Are you collecting personal, copyrighted, proprietary, or commercially sensitive data? <br></strong>If so, you need to be more careful. Different countries and regions have their own privacy, data protection, copyright, and database-rights rules. In the UK and EU, for example, GDPR-style obligations may apply when personal data is collected, stored, or processed. Failure to follow the relevant rules can lead to fines, legal claims, reputational damage, or enforcement action.</li><li><strong>Can the scraper run politely, with rate limits, retries, caching, and minimal load? <br></strong>Even if scraping is technically possible, every request still consumes resources on the other side. It can create bandwidth costs, server load, database queries, CDN usage, logging overhead, and extra traffic for the website owner to manage. Polite scraping is much better for long-term use than aggressively grabbing as much data as possible in the shortest time.</li></ol><p>The main point is simple: be respectful, understand the rules of the website, avoid unnecessary load, and do not assume public means free to reuse. That is how you reduce the chances of turning a useful data collection project into a legal or operational problem.</p><h3>AI-Era Changes</h3><p>AI has changed scraping in two directions at once. On one side, LLM-based tools and coding assistants lower the barrier to building simple collectors. Today, someone can use tools such as ChatGPT, Gemini, Claude, or open-source LLMs to generate a basic Level 1 or Level 2 scraper much faster than before.</p><p>On the other side, AI browsing agents and automated tools have increased the volume and sophistication of machine-driven web traffic. This has encouraged more websites to invest in anti-bot systems, rate limits, behavioural checks, and browser-level detection. Many open-source scraping approaches that worked a few years ago are now less reliable against modern sites.</p><p>As a result, high-end scraping has become more valuable. Basic scraping is easier to generate, but reliable custom solutions are harder to create and maintain. Running a compliant, recurring collection process against modern websites now requires stronger engineering, better monitoring, clearer governance, and a better understanding of how websites deliver and protect their data. Which in a positive way, means you can charge more.</p><h3><strong>Conclusion</strong></h3><p>Web scraping still matters because the web remains one of the largest stores of public and commercial information. Used well, it supports business intelligence, research, automation, and analytics. Used carelessly, it produces brittle pipelines, questionable data, and avoidable risk.</p><p>The real value comes from turning the web information you collect into structured, reliable, and responsibly collected data.</p><p>If you have a web scraping, data collection, automation, or competitor-monitoring project you would like to discuss, feel free to reach out.</p><p><strong>Email:</strong> <a href="/service/mailto:thisisjasonowens@gmail.com">thisisjasonowens@gmail.com</a><br><strong>LinkedIn:</strong> <a href="https://www.linkedin.com/in/thejasonowens/">linkedin.com/in/thejasonowens</a><br><strong>GitHub:</strong> <a href="https://github.com/thejasono">github.com/thejasono</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13867af05e79" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Transactional + Analytics Data Platform with FastAPI, PostgreSQL, and dbt (with Full…]]></title>
            <link>https://medium.com/@thisisjasonowens/building-a-transactional-analytics-data-platform-with-fastapi-postgresql-and-dbt-with-full-7b5b0f86d1a0?source=rss-ecbef8df7ad4------2</link>
            <guid isPermaLink="false">https://medium.com/p/7b5b0f86d1a0</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[backend-development]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[fastapi]]></category>
            <dc:creator><![CDATA[Jason O.]]></dc:creator>
            <pubDate>Wed, 08 Apr 2026 02:28:25 GMT</pubDate>
            <atom:updated>2026-04-18T23:49:22.016Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Building a Transactional + Analytics Data Platform with FastAPI, PostgreSQL, and dbt (with Full Telemetry)</strong></h3><p>This post shows how I turned northwind.sql (a static SQL dataset) into a data platform with two clear paths: an API for operational reads and writes, and a dbt-powered analytics layer for reporting queries exposed through API endpoints, backed by full telemetry and alerting for failure detection and response.</p><p>This walkthrough is intentionally detailed and structured like internal technical documentation. I wrote it as a reusable reference, so feel free to work through it section by section at your own pace. It explains how the system is implemented and how to use the code in practice.</p><h4><strong>Key features</strong></h4><ul><li><strong>FastAPI application layer:</strong> Exposes endpoints for transactional operations and analytics reads, with clear separation between write and read paths.</li><li><strong>Transactional safety controls:</strong> Uses idempotency keys and optimistic concurrency (ETag / If-Match) to keep order workflows safe under retries and concurrent updates.</li><li><strong>Living dataset simulation:</strong> Includes an activity simulator that generates day-by-day operational changes, so pipelines are tested against evolving data instead of static snapshots.</li><li><strong>Incremental ingestion pipeline:</strong> Captures changes into a raw schema using high-watermark and primary-key cursor patterns to preserve lineage and support rerun safety.</li><li><strong>dbt analytics layer:</strong> Transforms raw captured data into analytics-ready models in an analytics schema, exposed through read-only endpoints.</li><li><strong>Operational hardening:</strong> Supports separate local and production runtimes (.env.local vs .env.prod), with stricter ingress and network controls in production mode.</li><li><strong>Telemetry and alerting:</strong> Provides end-to-end metrics and traces, dashboards, and Slack alerting for faster detection and response.</li></ul><p>By the end of this walkthrough, you will be able to:</p><ul><li>Run the platform locally and understand the runtime topology.</li><li>Execute core workflows: seed, nightly ingestion, and dbt runs.</li><li>Trace lineage from operational writes to analytics endpoints.</li><li>Validate behavior with CI checks across local and production-oriented paths.</li></ul><h3>0. View the Code on Github</h3><p>This post is part of my Data Engineering and Backend Engineering guides, focused on production-shaped architecture, implementation patterns, and operational discipline.</p><p>👉 <a href="https://github.com/thejasono/northwind_data_platform">View the full project on GitHub</a></p><p>👉 <a href="https://medium.com/@thisisjasonowens/list/dataengineereing101-171b7a060c94">Access Data Engineering Playlist</a></p><p>👉 <a href="https://medium.com/@thisisjasonowens/list/backend-fundamentals-b8a04b40c744">Access Back-end Engineering Playlist</a></p><h3>1. Setup and Testing</h3><p>To run the system correctly, follow the documented workflows instead of invoking services manually.</p><h4><strong>1. Read the right docs first</strong></h4><ul><li>General runtime and architecture: README.md, docs/assets/architecture/platform_system_map.png</li><li>End-to-end validation for both local and production paths: docs/testing_plan.md, docs/production_testing_plan.md</li><li>External connectivity (<strong>Production</strong>): docs/domains/operations/external_device_access_production.md</li><li>Command source of truth: Makefile</li></ul><h4>2. Environment prerequisites</h4><ul><li><strong>OS/Shell:</strong> POSIX shell environments (Linux, Bash recommended). Windows users should use <strong>WSL</strong> or <strong>Git Bash</strong>.</li><li><strong>Tooling:</strong> make installed and available in PATH.</li><li><strong>Container runtime:</strong> Docker/Compose required for local and production-mode flows.</li><li><strong>Access:</strong> permissions to run containers, create networks, and bind required ports.</li></ul><p>The repository supports two runtime modes:</p><ul><li><strong>Local mode</strong> (recommended for development).</li><li><strong>Production mode</strong> (stricter controls and team-oriented runtime behavior).</li></ul><h4>3. Local setup and test flow (recommended for development)</h4><p>Before running commands, populate .env.local from .env.example</p><p>Global note: command examples in this post assume required services are already running and authentication values (for example, API keys/tokens) are configured.</p><p>Use Makefile commands as authoritative; illustrative snippets show shape only.</p><p><strong>Quick local Python environment setup:</strong></p><pre>make venv-bootstrap</pre><p><strong>Run full local validation:</strong></p><pre>make testing_plan_run_all</pre><p><strong>Start the local runtime with one of these options:</strong></p><pre>make up<br>make up-full-local</pre><p><strong>Seed and run ingestion:</strong></p><pre>make seed<br>make nightly</pre><p><strong>Optional day simulation before nightly:</strong></p><pre>make simulate-day RUN_DATE=2026–03–31<br>make nightly</pre><h4>4. Production mode setup and verification flow</h4><p>Before running commands, populate .env.prod using .env.prod.example.</p><p><strong>Run Full Production Validation:</strong></p><pre>make testing_plan_run_all_production</pre><p><strong>Persistent production-mode runtime (no cleanup):</strong></p><pre>make testing_plan_run_all_production TESTING_PLAN_PRODUCTION_ARGS=\&quot;--skip-cleanup\&quot;</pre><p><strong>Production mode step-by-step:</strong></p><pre>make up-full-production<br>make production-seed<br>make production-ingest<br>make production-dbt-run<br>make production-dbt-smoke</pre><p><strong>Note:</strong> For connecting external devices (e.g., mobile, second PC) to the production API, follow the guidance in docs/domains/operations/external_device_access_production.md.</p><h4>5. Key quality gates</h4><p>Run baseline checks before or alongside integration tests:</p><pre>make lint<br>make typecheck<br>make ci-gates-all</pre><ul><li>make lint - runs static linting (Ruff) to catch style issues, invalid patterns, and basic correctness problems early.</li><li>make typecheck - runs MyPy type checking to catch type/interface mismatches before runtime.</li><li>make ci-gates-all - runs the full CI governance gate suite (policy/contract checks) to verify local parity with repository CI rules.</li></ul><h4>6. IP Firewall Script Disclaimer</h4><p>Firewall policy changes are operator-invoked and are not applied automatically at service startup.</p><pre>make firewall-apply<br>make firewall-rollback</pre><blockquote><strong>Disclaimer:</strong></blockquote><blockquote>Explicit firewall operations (make firewall-apply, make firewall-rollback) modify host/network firewall rules, including allow/deny behavior for specific IP ranges or ports.</blockquote><blockquote>Full testing-plan runs do not modify host/network firewall rules by default.</blockquote><blockquote>For local testing-plan scripts, firewall changes only occur when the optional firewall phase is explicitly enabled.</blockquote><blockquote>Running these on a shared host, production node, VPN-connected laptop, or remote VM can interrupt access to services, including your current SSH/session.</blockquote><blockquote>Review scripts/security/apply_firewall_policy.sh, validate with docs/domains/operations/firewall_lifecycle.md and docs/domains/operations/external_device_access_production.md, run first in an isolated/test environment, and keep a rollback path ready.</blockquote><h4>Minimum safe-use guidance</h4><ul><li>Inspect the firewall script before running.</li><li>Confirm whether policy affects host-level firewall or container-only networking.</li><li>Prefer local console access over SSH-only access for first run.</li><li>Capture current firewall state before apply and define rollback commands.</li><li>Validate post-apply connectivity for app, database, telemetry, and management access.</li></ul><h3>2. System Overview</h3><p>The following is an overview of the system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y2KLkRkD8A0DwEorMi1HXg.png" /><figcaption><em>End-to-end platform architecture showing the core data plane (northwind.sql -&gt; PostgreSQL app/raw/analytics -&gt; FastAPI), reliability and control planes, and observability/telemetry pipeline (OpenTelemetry, Prometheus, Tempo, Grafana, Alertmanager, Slack) in a containerized runtime.</em></figcaption></figure><p>This platform starts from northwind.sql, normalizes operational data into app, incrementally captures changes into raw, transforms analytics with <strong>dbt</strong>, and serves transactional and analytics APIs through <strong>FastAPI</strong>.</p><p><strong>FastAPI</strong> request protection includes authentication/authorization, <strong>CIDR</strong> policy checks, and rate limiting (<strong>Redis</strong>-backed in production mode).</p><p>Metrics are emitted from API and pipeline instrumentation through <strong>OpenTelemetry</strong>, scraped and evaluated by <strong>Prometheus</strong>, routed through <strong>Alertmanager</strong>, and sent to <strong>Slack</strong>.</p><p>Traces are emitted through <strong>OpenTelemetry</strong> and stored in <strong>Tempo</strong> for <strong>Grafana</strong> exploration.</p><p>The system runs in two modes:</p><ul><li><strong>Local/dev:</strong> docker-compose.yaml</li><li><strong>Production mode:</strong> docker-compose.prod.yaml</li></ul><h3><strong>3. Northwind Static Database</strong></h3><p>Northwind is the bootstrap dataset for this platform. It is stored as northwind.sql and provides initial entities (customers, orders, products, suppliers) for Northwind Traders, a fictional wholesale specialty-food company.</p><p>The first key stage is database bootstrap: loading northwind.sql into PostgreSQL (running in Docker) to establish baseline state.</p><p><strong>Reference:</strong></p><ul><li>PostgreSQL variant: <a href="https://github.com/pthom/northwind_psql">https://github.com/pthom/northwind_psql</a></li></ul><p>Northwind is starting data only. Runtime contracts are enforced later through schema design, ingestion, and transformation layers.</p><p>Bootstrap can happen through two Docker-driven paths:</p><ul><li>Compose loader path (init + northwind_dataset_loader), with idempotency tracked in public.bootstrap_seed_state.</li><li>Seed command path (make seed), which applies northwind.sql, runs Alembic migrations, and executes python -m scripts.pipeline.seed_app --force.</li></ul><p>For day-to-day local setup, use make up + make seed, and treat the compose loader path as an alternate bootstrap mechanism.</p><p><strong>SEED_MODE behavior:</strong></p><ul><li>SEED_MODE=northwind: northwind_dataset_loader applies /app/northwind.sql.</li><li>SEED_MODE=sample: northwind_dataset_loader is skipped, and sample fixtures are seeded instead (sample_dataset_loader, sample-data profile, LOAD_SAMPLE_DATASET=true).</li></ul><p>Data persistence is controlled by Docker volumes, not by the loader itself. Unless volumes are removed, bootstrap state remains across restarts.</p><p>To fully reset bootstrap state:</p><pre>docker compose down -v --remove-orphans</pre><p>Northwind data is intentionally treated as baseline state. Operational and analytics contracts are enforced afterward through app/raw schema boundaries and pipeline + dbt layers, rather than querying baseline public.* tables directly.</p><h3><strong>4. App Schema</strong></h3><p><strong>app schema </strong>is the set of <strong>business-facing tables</strong> that:</p><ol><li>Extract only the required data from the northwind/public schema</li><li>Remove unnecessary fields</li><li>Restructure data into a transactional contract the API can enforce</li></ol><p>To illustrate this, let’s look at a simple example.</p><h4>Source vs Target Shape (Orders Example)</h4><p>Let’s compare the source schema in northwind.sql (public.orders) with the target contract in migrations/versions/0001_baseline_core_schema.py (app.orders).</p><p><strong>public.orders source shape from </strong><strong>northwind.sql:</strong></p><pre>-- illustrative example (not exact runtime output)<br>CREATE TABLE orders (<br>order_id smallint NOT NULL,<br>customer_id character varying(5),<br>employee_id smallint,<br>order_date date,<br>required_date date,<br>shipped_date date,<br>ship_via smallint,<br>freight real,<br>ship_name character varying(40),<br>ship_address character varying(60),<br>ship_city character varying(15),<br>ship_region character varying(15),<br>ship_postal_code character varying(10),<br>ship_country character varying(15)<br>);</pre><p><strong>app.orders target shape from migration contract:</strong></p><pre>-- source-of-truth shape mirrors migrations/versions/0001_baseline_core_schema.py<br>CREATE TABLE IF NOT EXISTS app.orders (<br>order_id BIGSERIAL PRIMARY KEY,<br>customer_id BIGINT NOT NULL REFERENCES app.customers(customer_id),<br>status TEXT NOT NULL DEFAULT &#39;PENDING&#39;,<br>order_ts TIMESTAMPTZ NOT NULL DEFAULT now(),<br>created_at TIMESTAMPTZ NOT NULL DEFAULT now(),<br>updated_at TIMESTAMPTZ NOT NULL DEFAULT now(),<br>version INTEGER NOT NULL DEFAULT 1, <br>CONSTRAINT order_status_valid CHECK (status IN (&#39;PENDING&#39;, &#39;PAID&#39;, &#39;SHIPPED&#39;, &#39;CANCELLED&#39;))<br>);</pre><p>public.orders carries fields that are not required in the core transactional model, including:</p><ul><li>Legacy fulfillment/shipping detail (ship_name, ship_address, ship_city, ship_region, ship_postal_code, ship_country)</li><li>Legacy process-specific attributes (employee_id, required_date, shipped_date, ship_via, freight)</li></ul><p>To address this, the app.orders contract:</p><ul><li>Keeps only what the app needs for the core order lifecycle (order_id, customer_id, status, order_ts, created_at, updated_at, version)</li><li>Enforces integrity rules (foreign key + status check constraint)</li><li>Adds targeted indexes for app query patterns (customer_id, order_ts, status, composite status/order_ts)</li></ul><p>This normalization pattern is applied across multiple northwind.sql entities</p><h4>How Is This Implemented Across the System</h4><p>The process of moving from northwind.sql/public source shape to app contracts is implemented through schema migrations managed by Alembic.</p><blockquote><strong>What Alembic is and its purpose in this system**</strong></blockquote><blockquote>- Alembic is a migration tool that versions and applies database schema changes over time.</blockquote><blockquote>- Files in migrations/versions/ define and evolve app and raw schema contracts, constraints, indexes, and permissions.</blockquote><blockquote>- Example: migrations/versions/0001_baseline_core_schema.py defines the app.orders contract used by the transactional layer.</blockquote><blockquote>- Data population is separate from migration DDL and is handled by scripts/pipeline/seed_app.py (for example, _seed_customers, _seed_products, _seed_orders).</blockquote><p>So at this point, the bootstrap/data flow is:</p><ol><li>Load bootstrap SQL payload (northwind.sql) via the dataset loader path.</li><li>Apply migrations (migrations/versions/*) to create/evolve versioned schema structures.</li><li>Seed/transform app data (scripts/pipeline/seed_app.py).</li><li>Run nightly capture (scripts/pipeline/run_nightly.py) to append incremental changes into raw.</li></ol><h4>Migration Sequence in Alembic</h4><p>The migration process in this system applies these files in order:</p><ul><li>0001_baseline_core_schema.py - creates baseline app and raw schemas, core transactional tables, and pipeline control tables/indexes.</li><li>0002_app_updated_at_triggers.py - keeps generic updated_at triggers for app tables and uses an order-specific trigger on app.orders that sets updated_at and increments version.</li><li>0003_pipeline_state_last_run_fk.py - adds the FK from raw.pipeline_state.last_run_id to raw.pipeline_runs.run_id to enforce run-reference integrity.</li><li>0004_permissions.py - applies schema ownership, grants, and default privileges for database roles.</li><li>0005_retention_archive.py - introduces retention/archive structures, including history partitioning and archive cutover support.</li><li>0006_idempotency_keys.py - creates app.idempotency_keys and supporting indexes for idempotent write handling.</li><li>0007_business_mutation_audit_events.py - adds the business mutation audit-events table and timestamp indexing.</li><li>0008_business_mutation_audit_events_immutable.py - enforces append-only audit behavior by blocking updates/deletes on audit rows.</li><li>0009_api_key_table.py - creates the database-backed API key table and lookup indexes.</li><li>0010_api_key_lifecycle_columns.py - adds API key lifecycle/governance fields and supporting indexes.</li></ul><p><strong>Migration and data flow sequence</strong></p><p><em>northwind.sql load -&gt; Alembic migrations -&gt; scripts/pipeline/seed_app.py -&gt; scripts/pipeline/run_nightly.py</em></p><h3>5. Raw Schema</h3><p>raw is the second schema created alongside app during Alembic migrations. Its purpose is to track incremental changes in app over time, run by run.</p><p>At runtime, business activity is simulated by writing new rows into app.orders using scripts/pipeline/simulate_app_day.py.</p><p>That simulation does three key things:</p><ol><li>Inserts new rows into app.orders with statuses such as PAID, SHIPPED, and PENDING.</li><li>Sets order_ts, created_at, and updated_at to the generated event timestamp.</li><li>Uses run-date gating (order_ts::date = run_date) so each simulated day is isolated.</li></ol><p>After that, scripts/pipeline/run_nightly.py captures only new/changed rows into raw. dbt then reads from raw, types payload fields, and builds analytics models.</p><p>The two key tracking tables in raw are:</p><p>I. raw.orders: run-scoped captured change records (created at runtime by schema/nightly_pipeline.py in _ensure_raw_table).</p><pre>CREATE TABLE IF NOT EXISTS raw.orders (<br>    run_id TEXT NOT NULL,<br>    run_date DATE NOT NULL,<br>    source_pk_hash TEXT NOT NULL,<br>    payload JSONB NOT NULL,<br>    updated_at TIMESTAMPTZ NOT NULL,<br>    loaded_at TIMESTAMPTZ NOT NULL DEFAULT now(),<br>    PRIMARY KEY (run_id, source_pk_hash, run_date)<br>) PARTITION BY RANGE (run_date);</pre><p>II. raw.pipeline_state: checkpoint table that tells the pipeline where to resume next time (created in migrations/versions/0001_baseline_core_schema.py).</p><pre>CREATE TABLE IF NOT EXISTS raw.pipeline_state (<br>source_table TEXT PRIMARY KEY,<br>watermark TIMESTAMPTZ NOT NULL,<br>last_pk TEXT,<br>last_run_id TEXT,<br>updated_at TIMESTAMPTZ NOT NULL DEFAULT now()<br>);</pre><p>Over time, both app and raw volumes grow as activity is generated and captured. That mirrors normal production behavior.</p><p>These newly created records are then made available for downstream dbt analysis.</p><h4>Simple Walk through: Two Simulated Days</h4><p>The easiest way to understand this is to walk through a concrete example.</p><p>Assume we simulate two days: 2026-03-27 and 2026-03-28.</p><p>That inserts new rows into app.orders. You can inspect them directly by using the SQL query below:</p><pre>SELECT order_id, customer_id, status, order_ts, created_at, updated_at<br>FROM app.orders<br>WHERE order_ts::date IN (&#39;2026-03-27&#39;, &#39;2026-03-28&#39;)<br>ORDER BY order_ts, order_id;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zCj_qbi0Deswfl6klnfYGQ.png" /><figcaption><strong>Example app.orders ouput for SQL query above</strong></figcaption></figure><p>run_nightly.py (the script that simulates daily business activity every 24 hours) then reads the last successful checkpoint from raw.pipeline_state and continues incremental capture from app.orders without starting over.</p><p>That checkpoint has two parts:</p><ul><li>watermark: latest updated_at timestamp fully processed.</li><li>last_pk: last primary key processed at that exact timestamp.</li></ul><p>Using both values allows safe boundary resumes. The pipeline captures rows where:</p><ul><li>updated_at is greater than the stored watermark, or</li><li>updated_at equals the watermark and the primary key is greater than last_pk (tie-break rule).</li></ul><p>This prevents duplicates and missed rows when multiple records share the same timestamp.</p><p>Captured rows are appended to raw.orders with run metadata (run_id, run_date), and raw.pipeline_state is advanced to the newest processed position.</p><p>Using the earlier app.orders example, those changes now appear in raw.orders. For each captured row, payload stores the original app.orders row as JSON, while the other columns store ingestion/run metadata.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cEK6lVVPXS9mNhMJzfyN0g.png" /><figcaption><strong>Example of what the raw.orders would look like if we ran a nightly run for the app.orders for the simulated date_range</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/927/1*xeyl4wN5cYdjri7QzwTFnw.png" /><figcaption><strong>Example of what the raw.pipeline_state would look like if we ran a nightly run for the app.orders for the simulated date_range</strong></figcaption></figure><p>dbt then reads those captured rows from raw.orders and parses fields such as customer_id from payload.</p><p>For example, a row for customer_id = 1012 dbt effectively reads a raw payload like this:</p><pre>{<br>  &quot;order_id&quot;: 2001,<br>  &quot;customer_id&quot;: 1012,<br>  &quot;status&quot;: &quot;PAID&quot;,<br>  &quot;order_ts&quot;: &quot;2026-03-27T10:15:00Z&quot;,<br>  &quot;created_at&quot;: &quot;2026-03-27T10:15:00Z&quot;,<br>  &quot;updated_at&quot;: &quot;2026-03-27T10:15:00Z&quot;<br>}<br></pre><p>From there, dbt reads raw.orders.payload, casts fields into correct SQL types in staging models, and builds marts such as fact_orders and customer-level aggregates.</p><p>Example: in stg_orders, (payload-&gt;&gt;&#39;customer_id&#39;)::bigint means:</p><ul><li>extract customer_id from JSON as text,</li><li>cast to BIGINT for reliable joins and aggregations downstream.</li></ul><h4>Why create a separate ‘raw’ schema (Complexity + Traceability)</h4><p>In theory, deltas could be detected directly from app each run. In practice, that often rescans large tables to find a small number of changed rows.</p><p>If nightly runs recompute deltas via full-table scans, processing tends toward O(N) per table per run (N = table size), even when only a few rows changed.</p><p>With watermark + PK-cursor tracking in raw.pipeline_state, extraction targets rows changed since the last checkpoint, making practical work closer to O(Δ) (Δ = changed rows), assuming supporting indexes exist.</p><p>Example: if app.orders grows from 54 to 59 rows after two runs, incremental capture should process 5 changed rows (Δ=5) instead of rescanning all 59 (N=59).</p><p>As long as Δ &lt;&lt; N, O(Δ) work is substantially cheaper than O(N) work.</p><h4>How does ingestion differ between the two startup modes</h4><p>Ingestion semantics do not differ between local and production modes.</p><p>Core logic is the same in both modes: incremental capture from app.* to raw.* with watermark + PK-cursor checkpoints, and ingestion runs with --skip-dbt so transformation remains a separate stage.</p><p>What changes is the runtime entrypoint:</p><ul><li>Local mode: make nightly (invokes run_nightly with --skip-dbt).</li><li>Production mode: make production-ingest (runs the production ingest one-shot job from docker-compose.prod.yaml, also with --skip-dbt).</li></ul><p>So the entrypoint changes, not the ingestion semantics.</p><h3>6. dbt</h3><h4>Ingest -&gt; Transform Handoff (Local + Production)</h4><p><strong>dbt </strong>runs only after ingestion has already completed and raw data is present.</p><p>Entrypoints differ by runtime mode:</p><ul><li>local: make dbt-run (optional validation: make dbt-test)</li><li>production: make production-dbt-run (optional validation: make production-dbt-smoke)</li></ul><p>In production mode, run context is passed as <strong>dbt</strong> vars; local make dbt-run/make dbt-test use macro/env-based resolution unless vars are explicitly supplied.</p><h3><strong>How dbt run works (Local path)</strong></h3><p>In local mode, make dbt-run is a wrapper command that runs the dbt CLI from the shell.</p><p>It is comprised of two stages:</p><ul><li>run dbt-precheck-raw first (fail fast if required raw relations are not ready)</li><li>then run dbt via the containerized wrapper script</li></ul><p>So the local command that actually runs is:</p><pre>scripts/dbt/dbt_stream.sh run --project-dir . --profiles-dir . (via make dbt-run)</pre><p><strong>With the flag intent being:</strong></p><ul><li>--project-dir: points dbt to dbt_project.yml (models, materializations, schema behavior)</li><li>--profiles-dir: points dbt to profiles.yml (target/profile connection config)</li></ul><h4>dbt Selection Modes</h4><p>dbt supports two selection modes:</p><ul><li><strong>single</strong>: builds models for one resolved ingest run</li><li><strong>range</strong>: builds models for an inclusive business-date window</li></ul><p><strong>Range mode is configured with:</strong></p><ul><li>DBT_RUN_SELECTION_MODE=range</li><li>DBT_RANGE_START_DATE=YYYY-MM-DD</li><li>DBT_RANGE_END_DATE=YYYY-MM-DD</li></ul><p><strong>Example (</strong><strong>.env.local):</strong></p><pre>DBT_RUN_SELECTION_MODE=range<br>DBT_RANGE_START_DATE=2026-07-01<br>DBT_RANGE_END_DATE=2026-07-31</pre><h4>Run Resolution Logic</h4><p><strong>In single mode, run resolution is:</strong></p><ul><li>Use run_id directly if provided</li><li>Otherwise use latest eligible SUCCESS run from raw.pipeline_runs</li><li>If run_date is provided in fallback mode, constrain to that date first</li></ul><p>raw_schema controls source resolution through var(&#39;raw_schema&#39;, &#39;raw&#39;), so source(&#39;raw&#39;, &#39;orders&#39;) can target raw, raw_custom, etc.</p><p>In range mode, filtering is based on business-event timestamps in raw payloads (for orders: order_ts, fallback created_at), then dbt builds the usual stack:</p><p>staging -&gt; dimensions/facts -&gt; marts/aggregates</p><h4>How dbt-test works</h4><p>dbt-test is the validation stage for transformed models. It runs dbt-native tests (schema tests + singular SQL tests) defined in the dbt project.</p><p>It uses the same runtime context pattern as dbt-run, but executes dbt test instead of dbt run.</p><p><strong>Local wrapper flow (</strong><strong>make dbt-test) is:</strong></p><ol><li>validate dbt selection inputs (single or range; range dates required in range mode)</li><li>Run dbt-precheck-raw</li><li>Execute dbt tests</li></ol><p><strong>Execution behavior:</strong></p><ul><li>single mode:tests resolve one selected run context</li><li>range mode:tests resolve the configured date window and assert against that window</li></ul><p><strong>Examples of tests in this project:</strong></p><ul><li>dbt/tests/raw_snapshot_contract.sql:validates that staged row counts align with the selected raw snapshot/window across key raw tables</li><li>dbt/tests/prod_smoke_latest_transforms.sql:used for smoke coverage and checks transformed outputs are consistent with expected latest transformed dates</li></ul><h4>How production-dbt-run works</h4><p>production-dbt-run is the production-mode equivalent of local dbt-run.</p><p><strong>Entry:</strong></p><ul><li>make production-dbt-run</li></ul><p><strong>Wrapper path:</strong></p><ol><li>Run production preflight</li><li>Validate selection inputs (including required range dates in range mode)</li><li>Run one-shot compose job: dbt_run</li></ol><p><strong>Compose job execution (</strong><strong>docker-compose.prod.yaml):</strong></p><ul><li>Runs python -m scripts.pipeline.run_dbt_job --dsn ... --run-id ... --run-date ... --raw-schema ...</li><li>The script creates a LocalDbtRunner and executes dbt run with vars payload</li><li>Vars include run_id, run_date, raw_schema, dbt_run_selection_mode, and optional range dates</li><li>After dbt completes, it persists dbt_status=SUCCESS|FAILED back into raw.pipeline_runs</li><li>Then it publishes views from analytics_analytics into analytics for API-facing reads</li></ul><p>Production transform semantics are the same as local, but with explicit production wrappers and durable status writeback.</p><h4>How production-dbt-smoke works</h4><p>production-dbt-smoke is the production-mode equivalent of local dbt-test, but intentionally scoped to smoke coverage.</p><p><strong>Entry:</strong></p><pre>make production-dbt-smoke</pre><p><strong>Wrapper path:</strong></p><ol><li>Run production preflight</li><li>Validate selection inputs</li><li>Run one-shot compose job: dbt_smoke</li></ol><p>Compose dbt_smoke job executes:</p><pre>scripts/dbt/dbt_stream.sh test --project-dir . --profiles-dir . --select tag:prod_smoke --vars &#39;{...}&#39;</pre><p>So it is not the full local dbt-test suite.</p><p>It runs only tests tagged prod_smoke, with the same run/range context vars (run_id, run_date, raw_schema, selection mode, optional range dates), to provide a fast production safety check for both single-run and range-run execution paths.</p><h4>Quick model-by-model breakdown</h4><p><strong>Staging</strong></p><ul><li>stg_customers: parses raw customer JSON into typed customer fields (customer_id, email, timestamps) for one selected pipeline run.</li><li>stg_products: normalizes product payloads into typed catalog fields (product_id, sku, unit_price, is_active, timestamps).</li><li>stg_orders: normalizes order payloads, standardizes status casing, and derives order_date from order_ts.</li><li>stg_order_items: parses line-item payloads and derives extended_price (quantity * unit_price).</li><li>stg_inventory_movements: parses movement events and derives movement_date from movement_ts.</li></ul><p><strong>Dimensions</strong></p><ul><li>dim_customers: customer dimension enriched with first_activity_at and cohort_month (first-order month fallback to customer creation time).</li><li>dim_products: canonical product attribute dimension for joins from facts/aggregates.</li><li>dim_dates: generated date spine across activity bounds for consistent daily/monthly joins and time bucketing.</li></ul><p><strong>Facts</strong></p><ul><li>fact_orders: one row per order with status/timestamps plus order-level order_amount rolled up from line items.</li><li>fact_order_items: one row per order item with product, quantity, price, extended value, and joined order context.</li><li>fact_inventory_movements: one row per inventory movement event with product/time/delta/reason.</li></ul><p><strong>Aggregates</strong></p><ul><li>agg_daily_revenue: daily rollup of order count, gross revenue, and quantity from order/item facts.</li><li>agg_customer_cohorts: cohort-month activity metrics (active customers, orders, revenue, retention rate).</li><li>agg_inventory_risk: product-day running balance with a derived risk score tier.</li></ul><p><strong>How they interlink (dependency view)</strong></p><ul><li>raw.* -&gt; stg_*</li><li>stg_customers + stg_orders -&gt; dim_customers</li><li>stg_products -&gt; dim_products</li><li>stg_orders + stg_inventory_movements -&gt; dim_dates (date bounds)</li><li>stg_orders + stg_order_items -&gt; fact_orders</li><li>stg_order_items + stg_orders -&gt; fact_order_items</li><li>stg_inventory_movements -&gt; fact_inventory_movements</li><li>fact_order_items + fact_orders -&gt; agg_daily_revenue</li><li>dim_customers + fact_orders -&gt; agg_customer_cohorts</li><li>dim_products + dim_dates + fact_inventory_movements -&gt; agg_inventory_risk</li></ul><h3>7. API Endpoints</h3><h4>Overview</h4><p>With pipeline and modeling context established, the API layer is best understood as three endpoint groups:</p><ol><li><strong>OLTP endpoints</strong> for live transactional reads/writes on app.*</li><li><strong>Analytics endpoints</strong> for read-only metric outputs from analytics.*</li><li><strong>Operational metadata endpoints</strong> for pipeline/runtime status visibility</li></ol><p>Transactional write paths and analytics read paths stay separate so they can evolve and scale independently.</p><p>Routes are defined under app/routes/, composed in app/routes/__init__.py, and mounted in app/runtime/main.py.</p><h4>Pathing and access control model</h4><p>Access to the endpoint is enforced per request, with each request required to pass multiple security checks.</p><ul><li><strong>Credential checks</strong>: API key and/or bearer-token validation</li><li><strong>Authorization checks</strong>: route-level permission enforcement after authentication</li><li><strong>CIDR checks</strong>: network-source allow/deny policy (host firewall and/or ingress policy by mode)</li><li><strong>Rate-limit checks</strong>: runtime policy enforcement, including Redis-backed behavior in production mode</li></ul><h4><strong>CIDR and Nginx Primer (What This Means in Practice)</strong></h4><ul><li><strong>CIDR</strong> (<strong>Classless Inter-Domain Routing</strong>) is a compact way to define an IP range (subnet) for network access policy.</li><li><strong>Nginx</strong> is a high-performance web server and reverse proxy. In API systems, it sits in front of the app to handle TLS, routing, header normalization, and edge policy.</li><li>A <strong>reverse proxy</strong> is an intermediary between clients and backend services. Clients connect to the proxy, not the API process directly. The proxy forwards requests to the upstream service (for example, api:8000) and returns responses. This centralizes controls such as TLS termination, request filtering, header normalization, request-size limits, and timeouts.</li><li><strong>Ingress</strong> is the external entry layer for service traffic. It defines how requests are admitted and forwarded internally (host/path routing, TLS/mTLS requirements, and perimeter policy). In this production setup, ingress is implemented by Nginx before traffic reaches FastAPI.</li></ul><h4>Layered request security flow</h4><p>In production, these controls work together as layered request security:</p><p>client IP -&gt; ingress (Nginx) -&gt; API transport middleware -&gt; FastAPI route</p><ul><li>Nginx applies edge checks first (TLS/mTLS, routing, forwarded-header handling)</li><li>The API then enforces source-subnet policy using the API_ALLOWED_CIDRS environment variable.</li><li>If the resolved client IP does not match any configured CIDR, the request is rejected with 403 api_source_cidr_denied</li></ul><h4>CIDR notation</h4><p>A CIDR block is written as address/prefix, where <strong>prefix</strong> is the number of fixed network bits.</p><ul><li><strong>Example:</strong></li><li>10.20.30.0/24 allows 10.20.30.0 to 10.20.30.255 (<strong>256 addresses</strong>)</li><li>10.20.30.14/32 allows exactly <strong>one host IP</strong></li></ul><p>In practical terms, CIDR defines which source IP ranges are allowed to access the API.</p><h4>Runtime-Dependent Enforcement</h4><p>Enforcement differs by runtime mode:</p><ul><li>Local/dev mode (docker-compose.yaml): CIDR can be enforced at host and app layers. Host firewall changes are operator-invoked (make firewall-apply, make firewall-rollback, or optional firewall phase in testing-plan scripts).</li><li>Production mode (docker-compose.prod.yaml): traffic is routed through ingress (client -&gt; ingress -&gt; API), and ingress centralizes TLS termination, routing, header normalization, and perimeter policy. CIDR policy is enforced automatically in the production path through configured ingress/app policy controls (API_ALLOWED_CIDRS with CIDR enforcement enabled).</li></ul><h4>Probe behavior</h4><p>Probes are implemented to confirm both service availability and traffic readiness before requests are routed to protected business endpoints.</p><ul><li>/health (<strong>liveness</strong>) checks whether the API process is up. In practice, it is a lightweight heartbeat endpoint: if the service is running and reachable, it returns a simple 200 status payload with a timestamp. Because it is process-level, it is not intended to prove full dependency health; instead, it confirms the application is alive and responding.</li><li>/ready (<strong>readiness</strong>) checks whether required dependencies are healthy enough to serve requests. It goes beyond liveness by evaluating readiness conditions (database first, then configured downstream checks, plus Redis rate-limit backend health when applicable). It then aggregates those results into component-level status output and returns 200 only when the overall state is ready; otherwise it returns 503 with dependency reasons. To reduce repeated probe load, readiness responses are also short-TTL cached.</li></ul><p>Probe routes are unauthenticated by default so orchestrators/load balancers/monitoring can call them directly. They can still be restricted by transport/network policy (<strong>CIDR</strong> allowlists, strict <strong>HTTPS</strong>/<strong>ingress</strong> policy).</p><p>This is intentionally different from versioned business routes (/api/v1/*), which are policy-protected.</p><p>For protected routes, checks typically occur in this order:</p><ol><li>approved source range</li><li>ingress policy (production mode)</li><li>valid credentials</li><li>route-level authorization</li></ol><h4>Versioned Routes Overview</h4><p>Here is a brief overview of the main versioned routes and what each route does.</p><p><strong>OLTP API (transactional path)</strong></p><p>These routes operate on live app.* tables and support real-time business reads/writes.</p><ul><li><strong>POST</strong>/api/v1/orders — create an order<br>Order items are persisted through a bulk insert path (single batch write), reducing round-trips compared with per-item inserts and improving create-order throughput.<strong><br>Example:</strong><br>Send a JSON request body with X-API-Key and Idempotency-Key headers to create an order. This writes immediately to app.orders, and the record is then captured into raw.orders on the next nightly run.</li></ul><pre>curl -X POST &quot;http://10.20.30.14:8000/api/v1/orders&quot; \<br>  -H &quot;X-API-Key: &lt;your-api-key&gt;&quot; \<br>  -H &quot;Idempotency-Key: order-create-2026-04-05-001&quot; \<br>  -H &quot;Content-Type: application/json&quot; \<br>  -d &#39;{&quot;customer_id&quot;:1,&quot;status&quot;:&quot;PENDING&quot;,&quot;items&quot;:[{&quot;product_id&quot;:1,&quot;quantity&quot;:2,&quot;unit_price&quot;:&quot;12.50&quot;}]}&#39;</pre><ul><li><strong>GET </strong>/api/v1/orders — list orders (with filters/pagination)<br> <strong>Example:</strong> GET /api/v1/orders?<br>limit=50&amp;&amp;cursor=&lt;opaque_token_from_previous_response_.next_cursor&gt;</li><li><strong>GET </strong>/api/v1/orders/{order_id} — fetch a single order<br> <strong>Example:</strong> GET /api/v1/orders/1024</li><li><strong>PATCH </strong>/api/v1/orders/{order_id} — update/cancel order status<br>Updates require If-Match, and ETag validation is based on order_id + version. <strong><br>Example:</strong> PATCH /api/v1/orders/1024</li><li><strong>POST </strong>/api/v1/inventory/movements — record inventory movement events<br>This writes immediately to app.inventory_movements, and the record is then captured into raw.inventory_movements on the next nightly run.<strong><br>Example:</strong> POST /api/v1/inventory/movements</li><li><strong>GET </strong>/api/v1/inventory/levels — return current inventory levels<br> <strong>Example:</strong> GET /api/v1/inventory/levels?limit=100&amp;offset=0</li></ul><p>Transactional changes made through these routes are later captured by the nightly pipeline and reflected in analytics models.</p><p><strong>Analytics API (read path)</strong></p><p>These routes read precomputed analytics outputs from analytics.* (derived via dbt).</p><ul><li><strong>GET </strong><strong>/api/v1/analytics/daily-revenue</strong> — daily revenue metrics over a date range<br> <strong>Example:</strong> GET /api/v1/analytics/daily-revenue?from=2026-03-01&amp;to=2026-03-31&amp;limit=100&amp;offset=0&amp;sort=date</li><li><strong>GET </strong><strong>/api/v1/analytics/cohort-retention</strong> — cohort retention metrics by activity month<br> <strong>Example:</strong> GET /api/v1/analytics/cohort-retention?cohort_month=2026-01&amp;limit=100&amp;offset=0&amp;sort=activity_month</li><li><strong>GET </strong><strong>/api/v1/analytics/inventory-risk</strong> — inventory risk metrics as of a selected date<br> <strong>Example:</strong> GET /api/v1/analytics/inventory-risk?as_of=2026-03-31&amp;limit=100&amp;offset=0&amp;sort=product_id</li></ul><p><strong>Operational Metadata API (pipeline/run status)</strong></p><ul><li><strong>GET </strong><strong>/api/v1/pipeline/runs</strong> — pipeline/dbt run history and runtime status metadata<br> <strong>Example:</strong> GET /api/v1/pipeline/runs?limit=50&amp;offset=0&amp;sort=run_ts</li></ul><p><strong>Other Protected Versioned Endpoints</strong></p><ul><li><strong>GET </strong><strong>/api/v1/protected/ping</strong> — authenticated connectivity/policy check<br> <strong>Example:</strong> GET /api/v1/protected/ping</li><li><strong>GET </strong><strong>/api/v1/system/contract</strong> — runtime route-policy contract and key invariants<br> <strong>Example:</strong> GET /api/v1/system/contract</li></ul><h4>Offset Pagination vs Cursor/Keyset Pagination</h4><p>To prevent oversized responses and reduce database/API load, list endpoints use standardized pagination behavior.</p><blockquote><strong><em>Pagination primer</em></strong></blockquote><blockquote><em>Pagination means returning data in fixed-size chunks instead of one large response. In APIs, this matters for endpoints that can return large record sets from a database; without pagination, payloads can grow too large, increase memory pressure, and raise latency or timeout risk. Pagination divides results into pages (a fixed-size subset of rows), improving response-size control, reducing server/client memory usage, and making high-volume routes easier to scale.</em></blockquote><p>The system uses two pagination styles: <strong>offset-based</strong> and <strong>cursor/keyset-based</strong>.</p><p><strong>Offset Pagination</strong></p><ul><li>Returns a page (fixed-size subset of rows) by numeric position using limit + offset.<br>- <strong>General syntax:</strong> GET /api/v1/&lt;resource&gt;?limit=&lt;page_size&gt;&amp;offset=&lt;start_index&gt;<br><strong>- Simple example:</strong> if a table has <strong>10,000 records</strong>, GET /api/v1/orders?limit=100&amp;offset=200 returns records <strong>201–300</strong> in the current sort order.<br>- For offset-based pagination, clients must provide:<br> limit: number of records to return<br> offset: starting index in the result set</li></ul><p><strong>Cursor/keyset pagination</strong></p><ul><li>Returns the next page using an opaque cursor tied to the last seen row.<br>- <strong>General syntax (first page):</strong> GET /api/v1/&lt;resource&gt;?limit=&lt;page_size&gt;<br><strong>- General syntax (next page):</strong> GET /api/v1/&lt;resource&gt;?limit=&lt;page_size&gt;&amp;cursor=&lt;opaque_token&gt;<br>- <strong>Simple example:</strong> <br>with <strong>10,000 records</strong> <br>first call GET /api/v1/orders?limit=100 returns first 100 rows + next_cursor=&quot;xyz...&quot;<br>next call GET /api/v1/orders?limit=100&amp;cursor=xyz... returns the following 100 rows.</li></ul><p><strong>How pagination is used in this system</strong></p><p>In this system, both approaches are used:</p><ul><li>Most list routes use <strong>offset pagination</strong> by default</li><li>/orders supports <strong>cursor/keyset pagination</strong> for stable traversal of ordered data.</li></ul><h4><strong>Pydantic boundary contracts</strong></h4><p>Pydantic models are used to enforce API correctness at service boundaries.</p><p><strong>Pydantic model primer</strong><br><strong>Pydantic</strong> is a class type focused on data validation and parsing. It enforces field types and constraints, and raises validation errors on bad input.</p><p>A typical Python class is a general-purpose object that contains attributes and methods. It can model behavior and internal state, but it does not automatically validate incoming data unless you implement that logic yourself.</p><p><strong>Pydantic</strong> differs from a typical class by making structured data contracts a first-class concern:</p><ul><li>Field definitions are declarative.</li><li>Parsing/validation happens automatically at model creation.</li><li>Invalid inputs fail fast with explicit error details.</li></ul><p><strong>Simple Pydantic example</strong></p><pre>from pydantic import BaseModel, Field, ValidationError<br><br>class UserCreate(BaseModel):<br>    user_id: int = Field(..., ge=1)<br>    email: str<br>    is_active: bool = True<br><br>ok = UserCreate(user_id=1, email=&quot;a@b.com&quot;)<br># UserCreate(user_id=-2, email=&quot;a@b.com&quot;) -&gt; ValidationError (ge=1 fails)</pre><h4>How is Pydantic Used Within the System</h4><ul><li><strong>Request models </strong>validate input structure and types before business logic executes</li><li><strong>Response models </strong>enforce consistent output schemas and improve OpenAPI generation</li><li><strong>Error models </strong>normalize failure payloads (for example: detail, request_id, timestamp) so clients can implement uniform error handling</li></ul><p>In this system, Pydantic route schemas are defined in the API schema layer (for example, under app/routes/schemas/) and referenced by route handlers in app/routes/*.</p><p><strong>Example route:</strong> POST /api/v1/orders</p><ul><li><strong>Request model:</strong> OrderCreate (with nested OrderItemCreate)</li><li><strong>Response model:</strong> Order (with nested OrderItem)</li></ul><p><em>Illustrative excerpt (shape only, not live data): source-of-truth is </em><em>app/routes/schemas/orders.py.</em></p><pre>from pydantic import BaseModel, Field, condecimal, conint<br>from typing import List, Optional<br>import datetime as dt<br>from decimal import Decimal<br><br>class OrderItemCreate(BaseModel):<br>    product_id: int = Field(..., ge=1)<br>    quantity: conint(gt=0)<br>    unit_price: condecimal(gt=0)<br>class OrderCreate(BaseModel):<br>    customer_id: int = Field(..., ge=1)<br>    status: str = Field(default=&quot;PENDING&quot;)<br>    order_ts: Optional[dt.datetime] = None<br>    items: List[OrderItemCreate]<br>class OrderItem(BaseModel):<br>    order_item_id: int<br>    product_id: int<br>    quantity: int<br>    unit_price: Decimal<br>    extended_price: Decimal<br>class Order(BaseModel):<br>    order_id: int<br>    customer_id: int<br>    status: str<br>    order_ts: dt.datetime<br>    created_at: dt.datetime<br>    updated_at: dt.datetime<br>    items: List[OrderItem]</pre><p>This pattern ensures typed, predictable request/response behavior and stable API contracts as the system evolves.</p><blockquote><strong>Note:</strong> version is internal concurrency state used for optimistic locking (ETag/If-Match) and is intentionally not exposed in the public Order API response model.</blockquote><h3>8. CI/CD Primer (Concept + Workflow Model)</h3><h4>Overview (CI/CD Primer)</h4><p>At a high level, CI/CD is a set of automated gates that run before and after integration to ensure code changes meet quality standards and do not introduce regressions.</p><h4>CI (Continuous Integration)</h4><p><strong>CI (Continuous Integration)</strong> runs validation checks when changes are pushed or proposed (for example, via pull requests).</p><p><strong>In plain English:</strong></p><ul><li>CI checks every proposed change before merge to make sure it is safe, valid, and consistent with project standards.</li><li>Typically, CI runs after commit/push and acts as a gate to verify that a change meets required standards (for example: code quality, typing, tests, contract checks, and runtime/data checks).</li><li>If required CI checks fail, the change is blocked from merging to main, and the baseline codebase remains unchanged.</li></ul><p><strong>Common CI gates</strong></p><ul><li><strong>linting</strong><br>static code-style and quality checks that scan source files for invalid patterns, formatting violations, and maintainability issues before runtime</li><li><strong>type checks</strong><br> static type analysis that validates type hints across functions, variables, and interfaces to catch mismatches before execution</li><li><strong>unit/integration tests</strong><br> automated tests that verify isolated components (<strong>unit</strong>) and cross-component behavior with real dependencies or boundaries (<strong>integration</strong>)</li><li><strong>API/data contract checks</strong><br> schema/interface validation that confirms request/response shapes, endpoint behavior, and data model contracts remain compatible and consistent</li><li><strong>selected runtime validations</strong><br> targeted execution checks (for example startup/readiness/config/path validations) that confirm critical runtime invariants still hold in CI conditions</li></ul><p><strong>How CI is triggered in this repo</strong></p><p>In this repo, ci.yml is triggered by:</p><ul><li>push (all branches + v* tags)</li><li>pull_request</li><li>schedule</li><li>workflow_dispatch</li></ul><h4>CD (Continuous Delivery/Deployment)</h4><p>CD handles packaging and promotion of already-validated changes through environment-specific workflows. So once CI proves a change is safe to ship, CD prepares and releases that validated change into real environments.</p><p>After CI passes, CD typically:</p><ul><li>builds a deployable artifact/container image</li><li>promotes it through release stages (for example: dev -&gt; staging -&gt; production)</li></ul><h4><strong>Continuous Delivery vs Continuous Deployment</strong></h4><p>These are two release styles under CD:</p><ul><li><strong>Continuous Delivery</strong>: the artifact is deployable at any time, but production release usually requires manual approval.</li><li><strong>Continuous Deployment</strong>: deployment to the target environment is automated once release criteria are satisfied.</li></ul><p>In the scenario where there is a simple bug-fix example after CI passes:</p><ul><li><strong>Continuous Delivery</strong>: image is built and pushed, staging deploys, then a human approves production.</li><li><strong>Continuous Deployment</strong>: image is built and pushed, and production deploy runs automatically if policy checks pass.</li></ul><p><strong>Objectives of Continuous Delivery/Deployment</strong></p><ul><li><strong>Reproducible</strong>: same inputs and pipeline steps produce the same release outcome each time.</li><li><strong>Testable</strong>: each promotion stage can be verified with automated checks before moving forward.</li><li><strong>Auditable</strong>: deployment history is traceable (what changed, what ran, and when/where it deployed).</li><li><strong>Low-risk change delivery</strong>: controlled rollouts and validation gates reduce incidents and speed up recovery when issues occur.<strong>How CD is triggered in this repo</strong></li></ul><h4>End-to-end flow</h4><p>code change -&gt; commit/push -&gt; CI validation gate -&gt; CD artifact/container build -&gt; release/deploy under policy</p><h4>How CI/CD files work</h4><p>CI/CD is defined in YAML workflow files (configuration) executed by GitHub Actions.<br> GitHub Actions is GitHub’s built-in automation system for running workflows in response to repository events (for example: push, pull_request, schedule, or manual trigger).</p><p>At pipeline level, the entire workflow file (for example, .github/workflows/ci.yml) is the automation unit being executed end to end.</p><p>These pipelines are composed of <strong>workflows</strong>, <strong>jobs</strong>, and <strong>steps</strong>.</p><p><strong>1) Workflow (top level)</strong></p><ul><li>Entire pipeline definition</li><li>Defined in .github/workflows/*.yml</li><li>Triggered by events such as:<br>- push <br>- pull_request <br>- schedule <br>- workflow_dispatch (manual)<br>- workflow_run (used in CD chaining)</li><li>Mental model:<br><strong>“When event X occurs, run pipeline Y.”</strong></li></ul><p><strong>2) Jobs (middle level)</strong></p><ul><li>A workflow contains one or more jobs</li><li>Each job runs on a runner</li><li>Jobs can:<br>- run in parallel<br>- depend on other jobs</li><li>Typical job names:<br>- build<br>- test<br>- deploy</li><li>Mental model: <strong>“A stage in the pipeline.”</strong></li></ul><p><strong>3) Steps (lowest level)</strong></p><ul><li>A job contains ordered steps</li><li>Each step is either:<br>- a shell command (run)<br>- a reusable action (uses)</li><li>Examples:<br>- checkout code<br>- install dependencies<br>- run tests<br>- build image</li><li>Mental model:<br><strong>“An individual instruction.”</strong></li></ul><h4>Hierarchy</h4><p>This is the hierarchy you can visualize whenever you see a CI/CD/Workflow YAML file.</p><pre>Workflow<br>├── Job A<br>│ ├── Step 1<br>│ ├── Step 2<br>│ └── Step 3<br>└── Job B<br>  ├── Step 1<br>  └── Step 2</pre><h4>Minimal GitHub Actions example</h4><p>This is a simple CI Pipeline workflow that runs on every push to main</p><pre>name: CI Pipeline<br><br>on:<br>  push:<br>    branches: [main]<br><br>jobs:<br>  build:<br>    runs-on: ubuntu-latest<br>    steps:<br>      - name: Checkout code<br>        uses: actions/checkout@v4<br><br>      - name: Install deps<br>        run: pip install -r requirements.txt<br><br>      - name: Run tests<br>        run: pytest</pre><p><strong>How this works:</strong></p><ul><li>GitHub sees a push to main</li><li>GitHub triggers this workflow file</li><li>A Linux runner is provisioned (ubuntu-latest)</li><li>The build job starts</li><li>actions/checkout@v4 clones your repository into the runner so later steps can access the code</li><li>Dependencies are installed</li><li>Tests are executed with pytest</li><li>If tests pass, the workflow passes; if tests fail, the workflow fails (and branch protection can block merge)</li></ul><h4>Job dependencies and parallelism</h4><ul><li>Jobs are independent unless linked explicitly.</li><li>Without dependencies → jobs may run in parallel</li><li>With needs → jobs form a DAG</li></ul><p><strong>Example:</strong></p><pre>jobs:<br>  build:<br>    # ...<br><br>  test:<br>    needs: build<br>    # ...</pre><p>This enforces:</p><ul><li>build -&gt; test</li></ul><h3>9. CI/CD Implementation in This System (Repository-Specific)</h3><h4>How CI is triggered in this repo</h4><p>ci.yml is triggered by:</p><ul><li>push (all branches + v* tags)</li><li>pull_request</li><li>schedule</li><li>workflow_dispatch</li></ul><h4>How CD is triggered in this repo</h4><p>cd.yml is triggered by:</p><ul><li>tag push (v*)</li><li>workflow_run after CI completes</li><li>workflow_dispatch (manual)</li></ul><p>It also supports manual promotion inputs for staging/production.</p><p><strong>Routing nuance:</strong></p><ul><li><strong>workflow_run-triggered CD runs</strong> only proceed when the upstream CI workflow has succeeded <strong>and</strong> that CI run came from a direct push to main</li><li><strong>Tag-based releases</strong> (for example v1.2.3) do <strong>not</strong> depend on workflow_run they start from the tag push event itself via the direct tag-trigger path</li></ul><h4>How CI/CD works within the system</h4><ul><li>CI validates API code, data pipeline logic, analytics model quality, and runtime/ops invariants</li><li>CD packages and promotes the validated application/runtime stack</li></ul><p>To inspect workflow definitions, go to .github/workflows/.</p><h4>Primary pipelines</h4><ul><li>ci.yml — integration and quality gates</li><li>cd.yml — build, promotion, and deployment</li></ul><h4>Operational workflows</h4><p>This repo also includes operational workflows:</p><ul><li>nightly-elt.yml</li><li>monthly-maintenance.yml</li><li>compliance-cadence.yml</li><li>dr-restore-drill.yml</li></ul><p>Because there are multiple workflows, it is clearer to explain them by responsibility rather than listing every individual step.</p><h4>CI workflow (ci.yml)</h4><p>Pull-request and push validation gates.</p><p>Covers:</p><ul><li>quality checks</li><li>tests</li><li>integration/dbt checks</li><li>security checks</li><li>readiness gates</li><li>required status checks</li></ul><h4>CD workflow (cd.yml)</h4><p>Handles:</p><ul><li>trigger resolution</li><li>image build and push</li><li>release governance gates</li><li>deployment</li></ul><p>Key jobs:</p><ul><li>resolve-trigger — determines which CD path should run based on trigger source (for example tag push vs workflow_run) and prepares shared context values</li><li>build-and-push-image — builds the deployable container image for the validated revision and pushes it to the configured registry</li><li>release-governance-gates — enforces required release policy checks and approvals before deployment is allowed to proceed</li><li>deploy-staging — deploys the approved image to staging for environment-level validation and pre-production checks</li><li>deploy-prod — deploys the approved release to production once all required gates and promotion conditions are satisfied</li></ul><h4>Nightly ELT (nightly-elt.yml)</h4><p>Scheduled and manual nightly ELT validation.</p><p>Runs:</p><ul><li>schema contract tests</li><li>smoke data preparation</li><li>nightly pipeline execution</li><li>quality gate verification</li><li>artifact upload</li></ul><p>Uses:</p><ul><li>RUN_DATE_MODE (fixed / dynamic)</li><li>effective run-date computation and validation</li></ul><p><strong>Purpose:</strong> detect freshness, drift, and regression issues on a recurring cadence</p><h4>Monthly maintenance (monthly-maintenance.yml)</h4><p>Scheduled and manual maintenance routine.</p><p>Runs:</p><ul><li>dependency installation</li><li>vulnerability scan (pip-audit)</li><li>regression tests</li><li>maintenance evidence upload</li></ul><p><strong>Purpose:</strong> dependency/security hygiene and periodic baseline validation</p><h4>Compliance cadence (compliance-cadence.yml)</h4><p>Scheduled and manual compliance-evidence freshness checks.</p><p>Runs governance and drill evidence validation scripts.</p><p><strong>Purpose:</strong> keep operational and compliance evidence current</p><h4>DR restore drill (dr-restore-drill.yml)</h4><p>Scheduled and manual disaster-recovery restore drill.</p><p><strong>Runs:</strong></p><ul><li>PostgreSQL in workflow context</li><li>restore drill script</li><li>artifact upload</li></ul><p><strong>Purpose:</strong><br> continuously validate restore readiness and capture DR evidence (RTO/RPO posture)</p><h4>Additional implementation notes</h4><p>Some extra implementation details worth calling out:</p><ul><li>CI and CD both define concurrency controls to reduce duplicate or overlapping runs.<br>In the system, runs are grouped by workflow/ref so only one active run per group is kept; when a newer run starts on the same branch, older in-flight runs can be canceled to avoid duplicate validation/deploy work.</li><li>Workflows define explicit token permissions so each job gets only the access it needs (least privilege).<br>CI jobs mostly need repository read + status/check permissions, while CD jobs need additional write-level scopes for release actions (for example image publishing and deployment-related updates).</li></ul><p>CD requires broader scopes than CI as CD performs release-side operations CI usually does not:</p><ul><li>package publishing: builds and publishes deployable container images used by production-path workflows (for example the image consumed by production ingest/deploy jobs).</li><li>security reporting: uploads release/security evidence and metadata used by governance and audit workflows.</li><li>OIDC/attestation: uses short-lived federated identity (OIDC) to authenticate to cloud/registry services without long-lived static secrets, and emits artifact attestation/provenance metadata for supply-chain verification.</li><li>release operations: promotes approved builds across environments (for example staging -&gt; production), applies governance gates, and records release/deployment state used by runtime and operational tracking.</li></ul><h3>10. Telemetry</h3><h4><strong>What is telemetry and how does it apply here?</strong></h4><p>Telemetry is the raw operational data your system emits and is used to understand system behavior and diagnose issues. In this system, telemetry is primarily composed of metrics, logs, and traces. Health/readiness endpoints and alerting are operational controls built on top of those signals. The application emits telemetry through instrumentation and probe endpoints, and the monitoring stack evaluates that data and turns threshold breaches into alerts.</p><h4>Overview</h4><ul><li><strong>Metrics</strong> — numeric measurements collected over time.<br> In this system, that includes request latency, response status patterns, health/readiness state, rate-limit backend behavior, and other operational counters emitted through the application telemetry layer, including app/observability/metrics.py.</li><li><strong>Logs</strong> — timestamped event records produced during application and runtime execution.<br> Here, logs capture structured lifecycle, operational, and security events such as startup/shutdown, readiness changes, authentication failures, policy denials, idempotency conflicts, and workflow transitions.</li><li><strong>Traces — </strong>end-to-end timelines of a single request across components (for example: inbound API request -&gt; route handler -&gt; DB query -&gt; response). <br><br>A trace is made of spans, where each span is one timed step (for example, GET /api/v1/orders handler or SELECT from app.orders).<br><br>Together they capture operation names, per-step latency, and error status so you can pinpoint exactly where failures or slowdowns occur.<br>In this system, spans are recorded across API and database work via app/observability/tracing.py, then exported by the configured tracing exporter when enabled.</li><li><strong>Health and readiness endpoints</strong> — probe endpoints that show whether the service is alive and whether it is ready to accept traffic.<br> /health is used for liveness and /ready is used for traffic readiness by both operators and runtime orchestration.</li></ul><p><strong>Alerting</strong> — rule-based incident signaling generated from observed system behavior.<br> In this system, Prometheus evaluates telemetry against alert rules, and Alertmanager groups and routes firing alerts to Slack.</p><h4>How telemetry moves through the stack</h4><p>The telemetry layer in this system is <strong>OpenTelemetry (OTel)</strong>, an open standard and tooling ecosystem for collecting and exporting telemetry data.</p><p><strong>OTel</strong> provides:</p><ul><li>an instrumentation layer: hooks/wrappers around app and DB operations that automatically generate spans/metrics</li><li>an SDK: the application-side library that creates and emits telemetry from code</li></ul><p>This allows the application to generate telemetry in-process and export it to the collector.</p><p>The application sends telemetry using <strong>OTLP (OpenTelemetry Protocol)</strong>, which is the transport format for moving telemetry from app services to the <strong>OTel Collector</strong> (a vendor-neutral telemetry gateway service that receives, processes, and routes telemetry to one or more backends) in continuous batches.</p><p>A great way to think about the relationship between <strong>OpenTelemetry (OTel)</strong>, <strong>OTel Collector</strong>, and <strong>OpenTelemetry Protocol</strong> is:</p><p>Application (OTel SDK/instrumentation) -&gt; OTLP -&gt; OTel Collector -&gt; Prometheus/Tempo/other backends</p><p><strong>Operationally in this project</strong></p><ul><li><strong>Metrics</strong> are produced in app/observability/metrics.py and exported via <strong>OTLP</strong> to the collector.</li><li><strong>Traces</strong> are produced in app/observability/tracing.py and exported via the configured tracing exporter (<strong>OTLP</strong> by enabled, with fallback-to-console behavior in code).</li><li><strong>Logs</strong> follow the application/runtime logging path and are separate from the collector metrics pipeline.</li></ul><blockquote><strong><em>Why I didn’t add an OTEL logs pipeline:</em></strong><em><br>I could have pushed logs through OTLP too (to Tempo/Prometheus-adjacent stack / other backend), but that adds complexity I didn’t need for this project stage. The current flow is: use metrics + traces first to find and narrow issues, then use JSON logs for deeper context when needed.</em></blockquote><p>The <strong>OTel Collector</strong> acts as the translator/router in the middle:</p><ul><li>receives <strong>OTLP telemetry</strong> from the app</li><li>batches/processes telemetry</li><li>re-exposes metrics in <strong>Prometheus format</strong> for scraping; tracing can be converted into metrics through the collector spanmetrics connector, which is useful for <strong>SLO-style latency/error alerting in Prometheus</strong></li></ul><p><em>Illustrative collector config (source-of-truth: </em><em>otel-collector-config.yaml):</em></p><pre>receivers:<br>  otlp:<br>    protocols:<br>      http:<br>        endpoint: 0.0.0.0:4318<br>exporters:<br>  prometheus:<br>    endpoint: 0.0.0.0:9464</pre><p><em>This is from </em><em>otel-collector-config.yaml. </em><strong><em>OTEL Collector</em></strong><em> defines a metrics pipeline where telemetry comes in on </em><em>:4318 using </em><strong><em>OTLP</em></strong><em> and metrics are re-exposed on </em><em>:9464 in </em><strong><em>Prometheus</em></strong><em> format.</em></p><p><strong>Prometheus</strong> then scrapes the collector endpoint, stores time series, evaluates rules from prometheus/rules/*.yml, and forwards firing alerts to <strong>Alertmanager</strong>(alertmanager:9093, configured in prometheus.yml)</p><p>Rule families are split (API SLO, readiness/database, rate-limit backend) so each alert maps to a specific failure type, which improves triage and reduces noisy paging. For example, if /ready starts failing because the database is unhealthy, that should trigger a readiness or database alert, not an API latency alert. Keeping rules separate makes triage faster and avoids noisy, ambiguous paging.</p><p><strong>Alertmanager</strong> then handles notification policy such as grouping, routing, and deduplication, and sends notifications to Slack:</p><p><em>Illustrative routing config (source-of-truth: </em><em>alertmanager/alertmanager.yml):</em></p><pre>route:<br>  group_by: [alertname, service, team]<br>  receiver: sre-slack<br>receivers:<br>  - name: sre-slack<br>    slack_configs:<br>      - api_url: ${ALERTMANAGER_SLACK_WEBHOOK_URL}</pre><p><em>This is from </em><em>alertmanager/alertmanager.yml. It is configured so </em><strong><em>Alertmanager</em></strong><em> sends notifications to </em><strong><em>Slack</em></strong><em> through the webhook receiver.</em></p><p>Responsibility split:<br>- Prometheus decides whether an alert is firing.<br>- Alertmanager decides how/where that alert is delivered.</p><p>Prometheus and Alertmanager run as separate services in compose (`docker-compose.observability.yaml` and production compose wiring).</p><h4>Grafana + Tempo in this system</h4><p>Span-derived metrics are not a full replacement for a trace backend as they summarize behavior instead of preserving full per-trace drill-down context.</p><p>So the system uses <strong>Tempo</strong> as the trace backend (it stores full distributed trace records and span relationships, not just aggregated counters/latency buckets) and <strong>Grafana</strong> as the operator UI:</p><ul><li><strong>Prometheus</strong> is used for metrics and alerting.</li><li><strong>Tempo</strong> is used for trace search and trace drill-down.</li><li><strong>Grafana</strong> is used to query both datasources from one place (grafana/provisioning/datasources/datasources.yml).</li></ul><p>Wiring is validated by scripts/ops/verify_grafana_dashboards_smoke.py (<strong>Prometheus query path + Tempo trace search path</strong>).</p><p><em>Illustrative collector -&gt; Tempo trace export (source-of-truth: </em><em>otel-collector-config.yaml):</em></p><pre>exporters:<br>  otlp/tempo:<br>    endpoint: tempo:4317<br>    tls:<br>      insecure: true<br><br>service:<br>  pipelines:<br>    traces:<br>      receivers: [otlp]<br>      processors: [memory_limiter, probabilistic_sampler, batch]<br>      exporters: [spanmetrics, otlp/tempo]</pre><p><em>Illustrative Grafana datasource wiring (source-of-truth: </em><em>grafana/provisioning/datasources/datasources.yml):</em></p><pre>datasources:<br>  - name: Prometheus<br>    uid: prometheus<br>    type: prometheus<br>    url: http://prometheus:9090<br>  - name: Tempo<br>    uid: tempo<br>    type: tempo<br>    url: http://tempo:3200</pre><h4>One concrete alert example (single path)</h4><p>To make the telemetry flow concrete, let’s walk through a failure scenario:</p><p><strong>Scenario <em>-&gt;</em></strong>GET /api/v1/orders shows a sustained 5xx error ratio of 1.8%.</p><p>A 5xx error ratio is calculated via:</p><p>5xx ratio = HTTP 5xx responses / all responses</p><p>An error ratio is the share of total requests that failed within a time window. It is used to measure user-visible reliability</p><p>In the system,<strong> Prometheus</strong> computes that from request-rate series over a 5-minute window and requires the condition to remain true for 10 minutes (for: 10m). rate(...[5m]) computes per-second change over a rolling 5-minute window.</p><p>If the service handled 10,000 requests and 180 were 5xx, the ratio is 1.8%. That is sustained user-visible failure and would need to be debugged.</p><p>Configured expectation:</p><ul><li><strong>alert threshold:</strong> greater than 1.0% (&gt; 0.01)</li><li><strong>sustained 1.8%:</strong> above tolerance and should page</li></ul><p>The path from detecting the failure to sending the alert goes through several steps:</p><ol><li><strong>Application records failing requests as metrics</strong></li></ol><p>Each completed API request records one observation into the http_server_duration_milliseconds histogram.</p><p>Observation = one measured request-latency sample for one completed request (for example, 187.4 ms for one GET /api/v1/orders response).<br>Unit in this project: milliseconds (ms), as recorded in http_server_duration_milliseconds.</p><p>One completed request contributes to:</p><ul><li>_count: total number of requests.</li><li>_sum: total accumulated request duration (ms).</li><li>_bucket: number of requests at or below each latency bucket boundary.</li></ul><p>So if one GET /api/v1/orders request records 1874 ms, it:</p><ul><li>adds 1874 to _sum.</li><li>increments _count by 1.</li><li>increments each _bucket{le=&quot;...&quot;} whose boundary is &gt;= 1874 (and does not increment lower buckets).</li></ul><p>Each series is labeled with dimensions such as http_status_code, http_route, and http_request_method. Each unique label set creates a separate time series. The 5xx alert uses these labels to isolate failed requests (for example, http_status_code=~&quot;5..&quot;) in prometheus/rules/api_slo_alerts.yml.</p><p>Example series (source-of-truth alert usage: prometheus/rules/api_slo_alerts.yml):</p><pre>http_server_duration_milliseconds_count{job=&quot;otel_collector&quot;,service_name=&quot;backend-data-platform-api&quot;,http_route=&quot;/api/v1/orders&quot;,http_request_method=&quot;GET&quot;,http_status_code=&quot;500&quot;}<br>http_server_duration_milliseconds_sum{job=&quot;otel_collector&quot;,service_name=&quot;backend-data-platform-api&quot;,http_route=&quot;/api/v1/orders&quot;,http_request_method=&quot;GET&quot;,http_status_code=&quot;500&quot;}<br>http_server_duration_milliseconds_bucket{job=&quot;otel_collector&quot;,service_name=&quot;backend-data-platform-api&quot;,http_route=&quot;/api/v1/orders&quot;,http_request_method=&quot;GET&quot;,http_status_code=&quot;500&quot;,le=&quot;250&quot;}</pre><p>These metrics are emitted by the application’s <strong>OTel</strong> instrumentation and sent to the <strong>OTel Collector</strong> via <strong>OTLP</strong>.</p><p><strong>2. OTel Collector receives and exposes metrics to Prometheus</strong></p><p>The application sends <strong>OTLP</strong> telemetry to the <strong>OTel Collector</strong> on :4318.<br>The collector republishes metrics on :9464 in <strong>Prometheus</strong> format for scraping.</p><p><strong>3. Prometheus then evaluates the API rule.</strong></p><p>Prometheus continuously scrapes collector metrics and recalculates the rule in prometheus/rules/api_slo_alerts.yml:</p><p><em>Illustrative excerpt (source-of-truth: </em><em>prometheus/rules/api_slo_alerts.yml):</em></p><pre>- alert: ApiHighErrorRate<br>  expr: |<br>    (<br>      sum(rate(http_server_duration_milliseconds_count{job=&quot;otel_collector&quot;,service_name=&quot;backend-data-platform-api&quot;,http_status_code=~&quot;5..&quot;}[5m]))<br>      /<br>      sum(rate(http_server_duration_milliseconds_count{job=&quot;otel_collector&quot;,service_name=&quot;backend-data-platform-api&quot;}[5m]))<br>    ) &gt; 0.01<br>  for: 10m</pre><p>Once 5xx exceeds 1% for 10 continuous minutes, the alert state for ApiHighErrorRate changes to firing.</p><p><strong>4. Prometheus sends the firing alert to Alertmanager</strong></p><p>Prometheus posts the firing event to alertmanager:9093.</p><p><em>Representative payload shape:</em></p><pre>{<br>  &quot;status&quot;: &quot;firing&quot;,<br>  &quot;labels&quot;: {<br>    &quot;alertname&quot;: &quot;ApiHighErrorRate&quot;,<br>    &quot;service&quot;: &quot;backend-data-platform-api&quot;,<br>    &quot;team&quot;: &quot;platform-observability&quot;,<br>    &quot;severity&quot;: &quot;page&quot;<br>  },<br>  &quot;annotations&quot;: {<br>    &quot;summary&quot;: &quot;API 5xx error rate via collector export is above SLO&quot;,<br>    &quot;description&quot;: &quot;5xx error rate exceeded 1% for 10m (observed ~1.8%).&quot;<br>  }<br>}</pre><p><strong>5. Alertmanager groups and routes the event</strong></p><p><strong>Alertmanager</strong> then applies a notification policy: it groups alerts by alertname, service, and team, deduplicates repeated firing updates for the same alert instance, and routes according to route policy (with a default receiver configured).</p><p>In plain terms, <strong>Prometheus</strong> may keep sending the same firing alert state, but <strong>Alertmanager</strong> does not forward every update to <strong>Slack</strong>. It batches related alerts, suppresses duplicates, and sends notifications on a controlled schedule.</p><p>The timing policy in alertmanager.yml controls that schedule:</p><ul><li>group_wait: 30s — wait 30 seconds before the first notification so related alerts can be included together.</li><li>group_interval: 5m — after sending, wait at least 5 minutes before sending another notification for changes in that same group.</li><li>repeat_interval: 3h — if the alert is still firing, send reminder notifications every 3 hours.</li></ul><p><em>Illustrative sequence (source-of-truth behavior is Alertmanager route/group timing config):</em></p><pre>[<br>  {&quot;status&quot;:&quot;firing&quot;,&quot;labels&quot;:{&quot;alertname&quot;:&quot;ApiHighErrorRate&quot;,&quot;service&quot;:&quot;backend-data-platform-api&quot;,&quot;team&quot;:&quot;platform-observability&quot;,&quot;severity&quot;:&quot;page&quot;}},<br>  {&quot;status&quot;:&quot;firing&quot;,&quot;labels&quot;:{&quot;alertname&quot;:&quot;ApiHighErrorRate&quot;,&quot;service&quot;:&quot;backend-data-platform-api&quot;,&quot;team&quot;:&quot;platform-observability&quot;,&quot;severity&quot;:&quot;page&quot;}},<br>  {&quot;status&quot;:&quot;firing&quot;,&quot;labels&quot;:{&quot;alertname&quot;:&quot;ApiHighErrorRate&quot;,&quot;service&quot;:&quot;backend-data-platform-api&quot;,&quot;team&quot;:&quot;platform-observability&quot;,&quot;severity&quot;:&quot;page&quot;}}<br>]</pre><p>Output behavior: repeated firing updates become one grouped notification per policy window, not one Slack message per update.</p><p><em>Illustrative payload shape:</em></p><pre>{<br>  &quot;groupLabels&quot;: {<br>    &quot;alertname&quot;: &quot;ApiHighErrorRate&quot;,<br>    &quot;service&quot;: &quot;backend-data-platform-api&quot;,<br>    &quot;team&quot;: &quot;platform-observability&quot;<br>  },<br>  &quot;status&quot;: &quot;firing&quot;,<br>  &quot;alerts_in_group&quot;: 1,<br>  &quot;receiver&quot;: &quot;sre-slack&quot;<br>}</pre><p><strong>6. Slack receives the notification</strong></p><p>Slack message content is rendered from Alertmanager template fields (status, alertname, service, severity, summary, description).</p><p><em>Example </em><strong><em>FIRING</em></strong><em> message:</em></p><pre>[FIRING] ApiHighErrorRate<br>Service: backend-data-platform-api<br>Severity: page<br>Summary: API 5xx error rate via collector export is above SLO<br>Description: 5xx error rate exceeded 1% for 10m (observed ~1.8%).</pre><p>At this point, incident response starts with enough context to triage quickly.</p><p>Once the bug is resolved, ApiHighErrorRate stops firing and Prometheus changes the rule state from firing to resolved. Prometheus sends that state change to Alertmanager, and Alertmanager generates a resolved notification so the team can confirm recovery and close the incident.</p><p><em>Example </em><strong><em>RESOLVED</em></strong><em> message:</em></p><pre>[RESOLVED] ApiHighErrorRate<br>Service: backend-data-platform-api<br>Severity: page<br>Summary: API 5xx error rate recovered<br>Description: 5xx error rate is back below 1% threshold.</pre><h3>11. Design Choices</h3><p>Rather than restating each component’s purpose, here are the main trade-offs I encountered while building and operating the system, and how I mitigated them in this implementation.</p><h4>Trade-offs</h4><ul><li>More infrastructure components to run, patch, and secure.</li><li>Higher cross-service integration and configuration complexity.</li><li>Strong dependency on consistent telemetry and schema governance.</li><li>Debugging can still be slower when instrumentation coverage is incomplete.</li><li>More CI/CD and runbook discipline required to keep behavior predictable across environments.</li></ul><h4>Mitigations Used</h4><ul><li><strong>Explicit schema separation (</strong><strong>app, </strong><strong>raw, </strong><strong>analytics)</strong><br>Prevents contract sprawl and forces a clear boundary between transactional, ingestion, and analytical concerns.</li><li><strong>Incremental ingestion with rerun-safe watermark/cursor control</strong><br> Ensures idempotency and avoids duplicate or inconsistent state during retries or backfills.</li><li><strong>Layer-specific tests across API, repository, pipeline, dbt, and ops</strong><br> Detects failures at the correct abstraction level instead of relying on downstream symptoms.</li><li><strong>Controlled migrations and backward-compatibility discipline</strong><br> Prevents breaking changes from propagating across services and environments.</li><li><strong>Standardized telemetry conventions and alert severity/routing policy</strong><br> Keeps telemetry usable rather than noisy; ensures alerts map to actionable failure modes.</li><li><strong>Fail-closed readiness and policy gating for production runtime invariants</strong><br> Stops traffic or deployments when invariants are violated instead of allowing degraded behavior.</li><li><strong>Environment separation (local vs production compose) with stricter production invariants</strong><br> Avoids false confidence from permissive local setups and enforces production realism where it matters.</li></ul><p>The main idea was to introduce complexity, then constrain it aggressively.</p><h3>12. Next Steps</h3><p>If anything is unclear, open Codex/Claude/DeepSearch (or any AI coding assistant), inspect the repo interactively, ask targeted questions, and take notes. That is usually the fastest way to understand how the system behaves in practice and it should help you improve as a programmer.</p><h4>Suggested exercises</h4><ol><li><strong>Trace one request end-to-end</strong><br> transactional API route → app table change → raw capture → dbt model in analytics → analytics API response</li><li><strong>Compare offset vs cursor pagination behavior</strong><br> Use two list endpoints and document trade-offs (performance, consistency, edge cases under mutation).</li><li><strong>Inspect schema enforcement</strong><br> Locate one request model and one response model and explain how they enforce API contracts (validation, typing, failure modes).</li><li><strong>Map CI/CD structure</strong><br> Read ci.yml and cd.yml and map each major job to its purpose (validation, packaging, promotion, deploy).<br> Focus on trigger conditions and gating logic.</li><li><strong>Follow one telemetry signal path</strong><br> metric/log/trace generation → collection/export → alert rule → Slack notification<br> Identify where signal loss or misconfiguration could occur.</li><li><strong>Review readiness/health behavior</strong><br> Inspect /health and /ready and explain how they gate traffic and protect runtime invariants during deploys.</li></ol><h3>13. Closing Notes</h3><p>Thanks for reading. I hope you learned something useful from this walkthrough, or at least picked up ideas that make you a better software engineer.<br>If you’d like to follow for future work or connect, you can find me here:</p><p>GitHub: <a href="https://github.com/thejasono">https://github.com/thejasono</a><br>LinkedIn: <a href="https://www.linkedin.com/in/thejasonowens/">https://www.linkedin.com/in/thejasonowens/</a><br>Email (business &amp; contract enquiries): <a href="/service/mailto:thisisjasonowens@gmail.com">thisisjasonowens@gmail.com</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b5b0f86d1a0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learn Databricks Fundamentals Through a Bronze–Silver–Gold Mini Lakehouse]]></title>
            <link>https://medium.com/@thisisjasonowens/build-a-databricks-lakehouse-end-to-end-bronze-silver-gold-with-dlt-and-unity-catalog-24c8f7236046?source=rss-ecbef8df7ad4------2</link>
            <guid isPermaLink="false">https://medium.com/p/24c8f7236046</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[Jason O.]]></dc:creator>
            <pubDate>Sun, 04 Jan 2026 21:03:01 GMT</pubDate>
            <atom:updated>2026-01-09T11:51:20.872Z</atom:updated>
            <content:encoded><![CDATA[<p>This walkthrough teaches Databricks fundamentals by building a practical Bronze → Silver → Gold pipeline on the NYC Taxi dataset.</p><p>You will use the databricks-nyctaxi-lakehouse repo to ingest data, enforce quality rules (e.g., required fields and nonnegative fares), and generate simple KPI tables for analytics. Along the way, I explain what Databricks is doing at each step: how compute runs your transformations, how Delta tables store results reliably, how Unity Catalog organizes and governs data, and how “DLT-style” definitions map to production pipeline behavior.</p><p>By the end, you will be able to:</p><ul><li>Configure Unity Catalog (catalogs, schemas, and permissions) for the specific repo.</li><li>Ingest Files into Bronze Delta Tables Using Either Copy Into or Auto Loader.</li><li>Run the Bronze → Silver → Gold transformations using the repo’s DLT-style SQL definitions (executed as notebooks in Free Edition).</li><li>Paid workspaces: run the same SQL as a managed Lakeflow/DLT pipeline via pipeline_settings.json.</li><li>Query Gold tables in Databricks SQL.</li><li>Optionally log basic pipeline metrics to MLflow for observability.</li></ul><p>This post is part of my <strong>Data Engineering Fundamentals Series</strong>, where we go through all the major projects and tools you need to know to be a practitioner in this space.</p><p>👉 <a href="https://medium.com/@thisisjasonowens/list/data-engineereing-fundamentals-171b7a060c94"><strong>View the playlist here</strong></a></p><h3>0. View the Code on GitHub</h3><p>All code, notebooks, and configurations for this lakehouse project are available in the public repository:</p><p>👉 <a href="https://github.com/thejasono/databricks-nyctaxi-lakehouse"><strong>View the full project on GitHub</strong></a></p><p>The repo contains Unity Catalog setup, ingestion notebooks, <strong>DLT-style SQL table definitions</strong>, validation SQL, and a monitoring script (plus optional pipeline_settings.json for deploying as a managed DLT pipeline in paid workspaces).</p><p>You will also need to download the NYC Taxi Trip Records dataset and upload it to Databricks. Use the official NYC Taxi &amp; Limousine Commission dataset and select any <strong>Yellow Taxi Trip Records (Parquet)</strong> files from <strong>2020–2025</strong>. The specific month you choose does not matter.</p><p><strong>👉</strong><a href="https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page"><strong>Link Here</strong></a></p><p>You will also need to create a Databricks Free Account in order to run the repo</p><p><strong>👉</strong><a href="https://login.databricks.com/signup?provider=DB&amp;scid=7018Y000001Fi0oQAC&amp;utm_medium=paid+search&amp;utm_source=google&amp;utm_campaign=14925725269&amp;utm_adgroup=133613194532&amp;utm_content=trial&amp;utm_offer=trial&amp;utm_ad=724888407498&amp;utm_term=is+databricks+free&amp;dbx_source=paid&amp;gad_source=1&amp;gad_campaignid=14925725269&amp;gbraid=0AAAAABYBeAgt_Mw1ujXellP8BD4ZMvHmL&amp;gclid=Cj0KCQiAvOjKBhC9ARIsAFvz5ljphmzMKptBgPwF259ckuBIY5U-wuSbaj2ckHgqSRanDY8aHHvLjtsaAv5gEALw_wcB&amp;tuuid=f045a97f-2918-49f2-87af-87a929f7dff0&amp;intent=SIGN_UP&amp;rl_aid=d2208060-b680-4b25-ae58-7c96f5346376&amp;sisu_state=eyJsZWdhbFRleHRTZWVuIjp7Ii9zaWdudXAiOnsidG9zIjp0cnVlLCJwcml2YWN5Ijp0cnVlLCJjb3Jwb3JhdGVFbWFpbFNoYXJpbmciOnRydWV9fX0%3D"><strong>Link Here</strong></a></p><h3><strong>1. Typical Company Data Environment</strong></h3><p>Before we run the project, it helps to map Databricks to the “typical company data environment” most data engineers inherit. A typical company data stack has several moving parts:</p><p>1. <strong>Data Sources</strong><br>Operational databases (e.g., Postgres/MySQL), SaaS systems (e.g., Salesforce, Stripe), logs, and sometimes event streams (e.g., Kafka/Kinesis).<br>These systems generate data in multiple shapes: structured tables, semi-structured JSON, files, and events.</p><p>2. <strong>A landing / raw zone (usually on object storage)</strong><br>Raw data is commonly landed in cloud object storage (S3 / ADLS / GCS). Object storage is a cloud storage system designed to store large amounts of data as files (“objects”), rather than as rows in database tables. Teams use it because it is cheap, highly scalable, and can store any format (CSV, JSON, Parquet, images, logs, etc.).</p><p>3. <strong>Table Layer:</strong><br>To make data in object storage usable for analytics and downstream processing, a table layer is added on top using open table formats such as Delta Lake, Iceberg, or Hudi. Its purpose is to turn raw files in storage into reliable, queryable tables. Instead of treating object storage as “just a folder of files” (for example, months of JSON transaction data), the table layer adds transactional metadata and schema rules so the data behaves like a database table: consistent queries, enforced and evolvable schemas, and reliable reads and writes.</p><blockquote><em>Interlude: </em><strong><em>The Medallion Architecture (Bronze → Silver → Gold)</em></strong></blockquote><blockquote><em>Once you have a table layer, you still need a consistent way to organize tables. In Databricks, many pipelines follow the Medallion Architecture, which improves data quality and usability in stages:</em></blockquote><blockquote><strong><em>i. Bronze (Raw):</em></strong><em> A direct copy of your raw data (CSVs, JSON, logs) loaded into a Delta table. Minimal transformation; you mainly capture and preserve the input. If downstream transformations or business logic change, you can replay from Bronze without going back to the source system.</em></blockquote><blockquote><strong><em>ii. Silver (Cleansed &amp; Conformed):</em></strong><em> This is where data is cleaned and standardized: types are fixed (e.g., timestamps are actual timestamps), records are deduplicated, and quality rules are enforced (e.g., “fare_amount ≥ 0”). Silver is commonly treated as the main clean layer that downstream analytics and ML build on.</em></blockquote><blockquote><strong><em>iii. Gold (Curated/Aggregated):</em></strong><em> Business-facing tables for reporting and consumption. Silver data is shaped into metrics (e.g., “daily revenue by cab type”) that are ready for dashboards and common queries. Gold is typically more read-optimized and more opinionated (aligned to business definitions) than Silver.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pC4HmkYxmvGRJpYFKwRL8g.png" /><figcaption><strong>Medallion Architecture.</strong> Data quality improves as it flows from Raw Integration (Bronze) to Filtered/Cleaned (Silver) and finally to Business-Level Aggregates (Gold).</figcaption></figure><p>4. <strong>Metadata + Governance (Catalog/Metastore):</strong><br>Metadata tracks what tables in the table layer exist, their schemas, where they physically live in storage, and who is allowed to access them. The full repository of metadata is known as the metastore or catalog. In Databricks, Unity Catalog provides this function: the metastore is the top-level container that registers metadata for data and AI assets and enforces the permissions that govern access to them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cr2VAJWVwcnNs5F3JVbDbg.png" /><figcaption>Unity Catalog Object Hierarchy. Metastore → Catalog → Schema → Table/Volume</figcaption></figure><p>5. <strong>Compute to transform data (batch and/or streaming):</strong><br>Compute is the execution layer of a data system: the CPU, memory, and runtime processes that actually execute code to read data, apply transformations, and write results. In Databricks, this compute is primarily provided by Apache Spark running on clusters. Spark is used to clean, standardize, join, and aggregate raw data from object storage into curated table-layer datasets that can be queried by downstream consumers.</p><p>This compute runs on clusters (multiple machines working in parallel) rather than a single virtual machine, which allows Databricks to process large datasets efficiently in both batch and streaming workloads.</p><p>6. <strong>Orchestration + monitoring</strong><br>Pipelines need scheduling, dependency management, retries, alerting, and audit logs (what ran, when, and what happened). Common external tools include Airflow and Dagster; Databricks also provides native surfaces for this (Jobs/Workflows and DLT/Lakeflow pipelines).</p><p>7. <strong>A serving/analytics layer</strong><br>This is where analysts and BI tools query curated datasets (often Silver/Gold). Many teams call this “the warehouse,” even if the data still lives in object storage (for example, queried through Databricks SQL).</p><p>In practice, teams assemble these pieces either as managed cloud services or self-hosted services.</p><h3><strong>2. How Does This All Run and Function Together?</strong></h3><p>The surrounding services (catalog, orchestration, monitoring) can run in a few different ways: as managed cloud services, on virtual machines, or as containerized workloads. Kubernetes is one common way to run containerized services at scale, but it is not required for a data environment to be distributed — distribution comes from the compute engine and storage architecture, not from Kubernetes itself.</p><h4><strong>Example (a simple Kubernetes-based setup)</strong></h4><p>Imagine a small UK bike retailer that runs most internal systems on Kubernetes. Their website and inventory services are containerized, but if we zoom in specifically on the data environment, it would look something like this:</p><ul><li><strong>Sources:</strong> orders/customers in Postgres; inventory events; web analytics; ad spend exports; optional event stream.</li><li><strong>Storage:</strong> raw extracts land in object storage.</li><li><strong>Tables:</strong> raw becomes Bronze; cleaned/conformed becomes Silver; business aggregates become Gold (implemented via a table format like Delta/Iceberg/Hudi).</li><li><strong>Catalog:</strong> tracks tables, schemas, locations, permissions.</li><li><strong>Compute:</strong> Spark/SQL transforms and updates tables (batch or streaming).</li><li><strong>Orchestration:</strong> schedules and monitors ingestion + transformations.</li><li><strong>Serving:</strong> curated Silver/Gold tables are queried by analysts/Power BI via a SQL endpoint.</li></ul><h3><strong>3. Where Databricks Comes in</strong></h3><p>Databricks bundles and manages much of the stack you would otherwise assemble yourself — table/storage format, distributed compute, pipeline execution, governance, and the SQL serving surface — into one platform, built around Apache Spark, Delta Lake, Delta Live Tables, and Unity Catalog.</p><ul><li><strong>Delta Lake (storage + tables)</strong><br>Delta Lake is an open table format commonly used by Databricks for Bronze/Silver/Gold tables. The table data lives as files in cloud object storage (S3/ADLS/GCS), and Delta Lake’s transaction log records changes so you get database-like reliability (ACID transactions, versioning/time travel).</li><li><strong>Ingestion tooling (getting data into Bronze)</strong><br>Databricks provides ingestion components that standardize common ingestion patterns. Auto Loader is a common example: incremental file ingestion from cloud storage into a Bronze table (new files are picked up automatically, schema evolution can be handled, and ingestion state is tracked so you do not reprocess the same files repeatedly). Commonly, ingestion is either from files (e.g., Auto Loader) or from an existing upstream table/source (e.g., CDC output or a shared dataset).</li><li><strong>Apache Spark (distributed compute)</strong><br>Apache Spark is the distributed compute engine that performs ingestion and transformations. This is where clusters come in: Spark distributes work across multiple machines so you can process large datasets in parallel (batch and streaming). Depending on workspace configuration, compute can run on user-managed clusters or Databricks-managed/serverless compute.</li><li><strong>Delta Live Tables (DLT)<br></strong>Delta Live Tables (DLT)<strong> </strong>is Databricks’ managed pipeline framework. Instead of hard-coding execution order (“run step 1, then step 2”), you define the target tables and their derivations. The platform infers and executes the Bronze → Silver → Gold dependency graph. <br>- In paid workspaces (managed DLT pipelines), Databricks builds and runs this graph and can manage incremental processing and state (where applicable), with progress tracked via checkpoints. DLT expectations can be configured to:<br>-<strong>Fail update </strong>(stop the pipeline)<br><strong>-Drop row</strong> (exclude invalid records)<br>-<strong>Log only </strong>(record violations while continuing processing)<br>In this tutorial’s Free Edition workflow, we execute the same table definitions as standard notebook runs (for example, using CREATE OR REPLACE TABLE), which rebuild outputs according to the SQL you run.</li><li><strong>Unity Catalog (governance + metadata)</strong><br>Unity Catalog is the governance and metadata layer: it defines catalogs and schemas, stores table metadata, and enforces permissions (who can read/write what). Conceptually, it replaces a standalone metastore/catalog and centralizes access control across workspaces and compute.</li><li><strong>Databricks SQL (serving/analytics)</strong><br>Databricks SQL (via SQL Warehouses) is the serving surface for fast SQL queries over governed Delta tables. Unity Catalog governs what you can see and do; Databricks SQL is the query endpoint analysts and dashboards use to access curated Silver/Gold datasets. Key point: in Databricks, “warehouse” usually refers to the SQL warehouse (a compute endpoint), while the data itself lives in governed Delta tables.</li></ul><p><strong>Putting it together (one end-to-end flow)</strong><br>Ingest into Bronze Delta tables → transform into Silver/Gold Delta tables using Spark → serve curated tables through Databricks SQL, with Unity Catalog enforcing governance and providing metadata across the environment.</p><h4>Same Kubernetes Example With Databricks Instead</h4><p>If we take the same UK bike retailer, the external sources do not change (e-commerce DB, inventory DB, web/ad/support systems, optional Kafka/Kinesis streams). What changes is that Databricks becomes the central platform that standardizes how those pieces are operationalized:</p><ul><li><strong>Landing (object storage)+ Bronze:</strong> raw data still lands in S3/ADLS/GCS, but ingestion writes into Bronze Delta tables (often using Auto Loader for incremental file pickup).</li><li><strong>Table layer: </strong>data is modeled as Delta Lake tables across Bronze → Silver → Gold (raw → cleaned/conformed → business aggregates).</li><li><strong>Governance: </strong>Unity Catalog stores table metadata and enforces permissions (e.g., analysts query Gold but not raw PII in Bronze).</li><li><strong>Compute + orchestration:</strong> transformations run as Spark jobs on Databricks, operationalized via Jobs/Workflows and/or Delta Live Tables (part of Lakeflow pipelines).</li><li><strong>Serving: </strong>analysts and BI tools query curated tables through Databricks SQL using a SQL warehouse (compute endpoint), with Unity Catalog controlling access.</li></ul><p>Net effect: the retailer still has external sources and object storage, but Databricks consolidates the operational pieces into one governed workflow aligned to the same Bronze/Silver/Gold lifecycle.</p><h3><strong>4. Is Databricks Low Code?</strong></h3><p>Strictly speaking: yes and no.</p><p>It is not “low code” in the sense that you can avoid programming. For non-trivial pipelines you still write SQL, and often Python/Scala for transformations, validation, testing, and integration. What Databricks reduces is operational friction: you spend less time wiring infrastructure, and more time on data logic. It standardizes and automates common concerns like compute provisioning, autoscaling, orchestration/monitoring, and repeatable execution of data pipelines.</p><h4><strong>Example pipeline (very typical)</strong></h4><ul><li><strong>Source A: </strong>clickstream events (streaming)</li><li><strong>Source B: </strong>orders/customers in Postgres (batch or CDC)</li><li><strong>Goal:</strong> daily metrics (conversion, revenue by category, drop-off pages) with basic quality rules (required IDs, valid timestamps, no negative revenue)</li></ul><h4>Traditional approach (many components)</h4><p>In a traditional stack, this pipeline is usually built by assembling and operating a toolchain such as:</p><ul><li><em>Kafka or Kinesis</em> for clickstream ingestion</li><li><em>Debezium </em>or custom jobs for Postgres CDC</li><li><em>Object storage </em>(S3 / ADLS / GCS) for raw data</li><li><em>Spark</em> jobs for transformations</li><li>A table format (Delta / Iceberg / Hudi) configured and managed manually</li><li><em>Airflow or Dagster</em> for orchestration and dependencies</li><li>A<em> metastore or catalog</em> for table metadata</li><li><em>Separate data-quality checks</em> (custom code or a framework)</li><li>A <em>warehouse or query engine</em> for BI access</li></ul><p>Each tool does its job, but all integration points are owned by the team: IAM and networking, schema and state management, retries and backfills, upgrades, observability, and failure handling across systems. This typically results in slower delivery and significantly higher operational overhead, in exchange for greater flexibility and direct ownership of the stack.</p><h4>Databricks approach (same outcome, fewer moving parts)</h4><p>Databricks consolidates most of that into platform features: ingestion (Auto Loader/streaming), transformations (Spark SQL + Python), managed pipelines (Delta Live Tables as part of Lakeflow pipelines + Jobs), storage (Delta Bronze/Silver/Gold), governance (Unity Catalog), and serving (Databricks SQL).</p><p>So you get simpler operations, but you have to accept Databricks’ conventions and managed abstractions.</p><h3><strong>5. Why do I need to know this?</strong></h3><p>Databricks is widely deployed in industry, and familiarity with it maps directly onto real work: ingestion patterns, medallion layering, governance, incremental processing, and serving curated datasets to analytics. <a href="https://www.databricks.com/company/newsroom/press-kit?utm_source=chatgpt.com">Databricks themselves have stated that over 60% of the Fortune 500 use the platform in some capacity, and it reports over 20,000 organizations as customers.</a></p><p>Even when you are not working in Databricks, the core ideas it packages (Spark-based distributed compute, table formats like Delta, centralized catalogs/governance, declarative pipeline graphs) are the same ideas you will encounter across modern data platforms. Knowing how the platform abstracts a “typical company data environment” makes it easier to drop into new environments, diagnose problems, and deliver work without spending weeks rebuilding basic infrastructure understanding.</p><h3><strong>6. How Does Apache Spark Work With Databricks?</strong></h3><p>As mentioned in prior posts, Apache Spark is a distributed compute engine designed to process large datasets in parallel across a cluster. In Databricks, most workloads ultimately execute as Spark jobs on Databricks-managed compute.</p><p>Databricks does not run “vanilla Spark” by default. It runs Databricks Runtime, which includes Apache Spark plus platform-level optimizations for usability, performance, and security. On the performance side, some workloads can also use Photon, a Databricks-native vectorized engine that accelerates many SQL and DataFrame operations while remaining compatible with Spark APIs.</p><p>Those Spark jobs run on compute you choose in a few common modes:<br>- All-purpose clusters: interactive exploration (notebooks, ad-hoc analysis).<br>- Job clusters: production runs (created for a job, then terminated).<br>- Serverless compute (if enabled): Databricks abstracts cluster management further.</p><p>Why is this important? C<strong>ost, runtime, and failure behavior are determined by the compute mode and how long that compute stays alive</strong>.</p><p>For example, consider a daily batch pipeline that prepares revenue and customer metrics for finance and leadership reporting. If this pipeline runs on an all-purpose cluster, compute can remain active outside the actual processing window, increasing cost without adding value. Running the same pipeline on a job cluster or serverless compute ensures resources exist only while the Spark job is executing, reducing waste and improving cost predictability.</p><h3><strong>7. Now Let’s Ground This in a Real Project</strong></h3><p>Now that you understand Databricks components (Delta Lake for storage, Spark for compute, Unity Catalog for governance, and DLT for managed pipelines), the next step is a hands-on exercise. We will run a complete lakehouse pipeline using a real dataset.</p><p>The project repository is organized to separate core responsibilities into dedicated folders.</p><h4><strong>Repo map (folder responsibilities)</strong></h4><ul><li><strong>/unity/ — Unity Catalog bootstrap and access control.</strong> 00_main_nyctaxi_catalogue_creator.ipynb (and its .py export) creates the main_nyctaxi catalog, defines the raw, ref, and mart schemas, and applies grants. The folder also includes templates for external locations and volumes.</li><li><strong>/notebooks/ — Bronze ingestion entrypoint.</strong> 01_copy_into_bronze.sql.ipynb uses batch ingestion using COPY INTO in order to load raw taxi files into the Bronze Delta table (and can optionally seeds from samples.nyctaxi.trips).</li><li><strong>/dlt/ — Batch Silver/Gold transformations.</strong> 02_dlt_pipeline.sql.ipynb defines the Bronze → Silver → Gold logic and data quality expectations, rebuilding raw.taxi_bronze, ref.trips_clean, ref.trips_valid, and mart.daily_kpis.</li><li><strong>/sql/ — Gold-layer validation and analytics queries.</strong> <br>03_gold_queries.sql.ipynb (and its .sql export) contains validation checks and example analytical queries executed from a SQL Warehouse against mart.daily_kpis and related tables.</li><li><strong>/monitoring/ — lightweight observability. </strong>mlflow_demo.py logs row counts, freshness, and simple quality signals from the Silver layer to MLflow after each batch run.</li><li><strong>README.md —</strong> Runbook and execution checklist tying the workflow together.</li></ul><h4><strong>How to Run It</strong></h4><ol><li><strong>Import the repo into Databricks Repos.</strong> In your Databricks workspace, go to <strong>Workspace → Repos → Add Repo (Git)</strong> and clone databricks-nyctaxi-lakehouse. You should see the project under Workspace/Repos/….</li><li><strong>Run the Unity Catalog bootstrap (required).</strong> Open /unity/00_main_nyctaxi_catalogue_creator.ipynb and run all cells. This step:<br>- Creates the main_nyctaxi catalog<br>- Creates the raw, ref, and mart schemas<br>- Creates the <strong>storage volume used as the landing zone</strong> for raw NYC Taxi files<br>- Applies the required permissions<br>This establishes the governed namespace and the physical landing location for raw data.</li><li><strong>Upload the NYC Taxi dataset to the landing volume <br></strong>Go to the <strong>Catalog</strong> tab in Databricks and select the main_nyctaxi catalog.<br>You should now see the raw, ref, and mart schemas, along with the storage volume created by the bootstrap notebook.<br>Upload the NYC Taxi <strong>Parquet</strong> files into the volume:<br>main_nyctaxi.raw.nyctaxi_landing<br>This makes the raw files available to Spark under Unity Catalog governance and allows them to be ingested into the Bronze layer.</li><li><strong>Ingest raw data (batch ingestion with COPY INTO):</strong><br>Run /notebooks/01_copy_into_bronze.sql.ipynb<br>This notebook performs <strong>batch ingestion</strong> using COPY INTO, loading the raw Parquet files from the landing volume into:<br>main_nyctaxi.raw.taxi_raw<br>Batch ingestion here means:<br>- Files are loaded explicitly in one operation<br>- COPY INTO is idempotent: it skips files that were already loaded (even if modified), unless you set COPY_OPTIONS (&#39;force&#39;=&#39;true&#39;).<br>This raw landing table (taxi_raw) is then standardized into the Bronze table (main_nyctaxi.raw.taxi_bronze) by /dlt/02_dlt_pipeline.sql.ipynb, and that Bronze table feeds the downstream Silver/Gold transformations in the medallion workflow.</li><li><strong>Execute the Silver/Gold pipeline logic (DLT-style SQL executed as notebooks):<br></strong>Run /dlt/02_dlt_pipeline.sql.ipynb as a normal notebook. In Free Edition, this executes the Bronze → Silver → Gold logic as a standard Spark SQL workload and materializes the outputs as regular Delta tables.<br>- This step materializes:<br>- ref.trips_clean<br>- ref.trips_valid<br>-mart.daily_kpis<br>- In a paid workspace, the same logic can be deployed as a managed Lakeflow/DLT pipeline by adapting the SQL to use LIVE tables (for example, replacing CREATE OR REPLACE TABLE with CREATE OR REFRESH LIVE TABLE) and applying the pipeline configuration.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/415/1*eCKVunYcAuy3WzQUIebQ_g.png" /><figcaption>What a managed DLT pipeline (DAG) looks like. In Free Edition, we run the transformations as notebooks; in paid workspaces, DLT can generate and operate this dependency graph as a managed pipeline.</figcaption></figure><ol><li><strong>Validate and query Gold Outputs:<br></strong>Run /sql/03_gold_queries.sql.ipynb<br>This notebook runs validation checks and example analytical queries <strong>against the Gold tables/views created by the pipeline</strong> (for example, main_nyctaxi.mart.daily_kpis). These queries represent the kind of downstream access BI and analytics teams use: they <strong>verify correctness</strong> and <strong>demonstrate consumption</strong>, but they do not create the Gold layer.</li><li><strong>Log metrics to MLflow.</strong> <br>For basic validation and observability, run /monitoring/mlflow_demo.py<br>This script:<br>- Logs silver_rows (row count of trips_valid)<br>- Logs basic quality/health signals from Silver data:<br> i. avg_fare_amount (average fare_amount)<br>ii. null_pickup_zip (count of rows where pickup_zip is NULL)<br>- Logs simple run context (pipeline_name, run_date)<br>- Writes a small sample (trips_valid_sample.csv) as an MLflow artifact for inspection in the MLflow UI.</li></ol><blockquote><strong><em>Note:</em></strong><em> this notebook does not configure access grants by default. Update the following line in </em><em>02_dlt_pipeline.sql.ipynb with your own account email:</em></blockquote><pre>GRANT ALL PRIVILEGES ON SCHEMA main_nyctaxi.raw TO `enter_your_email_here`</pre><blockquote><strong><em>Note: </em></strong>0<em>2_dlt_pipeline.sql.ipynb uses fully qualified table names under </em><em>main_nyctaxi.raw, </em><em>main_nyctaxi.ref, and </em><em>main_nyctaxi.mart. If you want to use a different catalog or schema, you must update the table references in the notebook as well as </em><em>dlt/pipeline_settings.json—changing the JSON target alone will not redirect outputs when running the notebook manually.</em></blockquote><blockquote><strong><em>Note: </em></strong><em>in Free Edition, this notebook run does not provide managed DLT features such as pipeline-driven incremental processing/state management, the DLT UI, or automatically tracked lineage. Those are available when the same definitions are deployed as a managed Lakeflow/DLT pipeline in paid workspaces (where applicable).</em></blockquote><h3>8. Project Artifacts &amp; Unity Catalog Objects</h3><p>Now that the pipeline has run, we can do a quick tour of Unity Catalog to see the artifacts. If any of these are missing, use the notes below to pinpoint where the failure occurred.</p><h4>1. Governance Objects</h4><ul><li><strong>Catalog:</strong> main_nyctaxi</li><li><strong>Schemas:</strong> main_nyctaxi.raw, main_nyctaxi.ref, main_nyctaxi.mart</li><li><em>Verification:</em> If these are missing, the <strong>Unity Catalog bootstrap notebook</strong> (00_main_nyctaxi_catalogue_creator) did not run or complete successfully.</li></ul><h4>2. Bronze Layer (Raw Landing)</h4><ul><li><strong>Table:</strong> main_nyctaxi.raw.taxi_bronze</li><li><strong>What it is</strong><br> The Bronze Delta table created by executing the DLT SQL logic in<br> 02_dlt_pipeline.sql.ipynb. In the Free Edition, this table is created by running the notebook manually, not by a managed pipeline.</li><li><strong>Verification</strong><br> If this table is missing or empty:<br>- Check that raw data was successfully ingested into main_nyctaxi.raw.taxi_raw<br>- Re-run the ingestion notebook (01_copy_into_bronze.sql.ipynb)<br>- Re-run 02_dlt_pipeline.sql.ipynb</li><li><strong>Table (ingestion only): </strong>main_nyctaxi.raw.taxi_raw</li><li><strong>What it is</strong><br> The raw landing Delta table populated from Parquet files via batch ingestion (COPY INTO). This table is the input to the Bronze transformation logic.</li></ul><h4>3. Silver Layer (Cleansed &amp; Validated)</h4><ul><li><strong>Tables<br></strong>- main_nyctaxi.ref.trips_clean (typed and normalized)<br>- main_nyctaxi.ref.trips_valid (contract-enforced)</li><li><strong>What they are</strong><br> The Silver layer is the <strong>trust boundary</strong>. Data here has been cleaned, typed, and validated against explicit quality rules (for example, fare_amount &gt;= 0).</li><li><strong>Verification</strong><br> If these tables are missing<br>- 02_dlt_pipeline.sql.ipynb did not execute successfully<br> - Review notebook execution output and errors (there is no DLT UI in Free Edition)</li></ul><p>If the tables exist but are empty:<br>- A data-quality rule may have dropped all rows<br>- There may be an upstream ingestion issue</p><h4>4. Gold Layer (Business Outputs)</h4><ul><li><strong>Gold output: </strong>main_nyctaxi.mart.daily_kpis</li><li><strong>What it is</strong><br> A curated analytical output containing daily business metrics (e.g. total_revenue, revenue_per_mile).<br> In the Free Edition, this is created as a standard Delta table or view by running the SQL notebook manually. Automatic refresh via managed materialized views requires a paid Databricks tier.</li><li><strong>Verification: </strong>Run the queries in 03_gold_queries.sql.ipynb using the SQL Editor or a SQL Warehouse</li><li><strong>Troubleshooting<br>- </strong>Empty or nonsensical results usually indicate:<br>- All rows were dropped at the Silver layer due to contract violations, or<br>- An upstream ingestion issue</li></ul><h4>5. Observability</h4><ul><li>If you run /monitoring/mlflow_demo.py, an MLflow run will be created containing basic pipeline health metrics such as silver_rows and counts of rows dropped due to data-quality rules. These metrics are derived from the tables and execution logs produced during the run and provide a minimal but useful signal for validating correctness and detecting regressions across executions.</li></ul><p>That is the end-to-end loop in one view: cataloged data, medallion tables, notebook-executed transformations, queryable outputs, and a place to hang basic operational metrics.</p><h3><strong>9. Validation Tests/Checks (and Why They Matter)</strong></h3><p>A pipeline can “succeed” and still be wrong. Most real failures are silent: instead of hard execution errors, you get missing rows, duplicated data, schema drift, or downstream metrics that quietly become incorrect. Because of this, validation should be applied throughout the lakehouse workflow so issues are detected as early as possible. In this repo, checks are aligned to each stage of the medallion architecture.</p><h4>Bronze checks (did ingestion behave?)</h4><p>In Catalog Explorer, start at:</p><ul><li>main_nyctaxi.raw.taxi_raw</li><li>main_nyctaxi.raw.taxi_bronze</li></ul><p>Minimum checks at this stage focus on volume and shape:</p><ul><li>Row counts are non-zero and broadly sensible</li><li>Expected columns exist with correct types</li><li>Timestamps or partitions advance consistently between runs</li></ul><p>These checks confirm that ingestion is working correctly and producing usable Bronze data.</p><h4>Silver checks (did the contract boundary hold?)</h4><p>Inspect:</p><ul><li>main_nyctaxi.ref.trips_clean</li><li>main_nyctaxi.ref.trips_valid</li></ul><p>Confirm that:</p><ul><li>Data-quality rules (for example, fare_amount &gt;= 0) are enforced</li><li>Invalid rows are excluded from trips_valid</li><li>The resulting tables are non-empty</li></ul><p>In the Free Edition, this validation is done by inspecting table contents and notebook execution output rather than a managed DLT UI.</p><h4>Gold checks (did we compute stable business outputs?)</h4><p>Run:</p><ul><li>/sql/03_gold_queries.sql.ipynb against main_nyctaxi.mart.daily_kpis</li></ul><p>Focus on invariants rather than complex tests:</p><ul><li>Results are non-empty</li><li>Dates are continuous</li><li>KPI values fall within plausible bounds</li><li>No duplicate rows are generated</li></ul><h4>Cross-run monitoring (do we detect regressions over time?)</h4><p>If /monitoring/mlflow_demo.py is run:</p><ul><li>Silver row counts are logged to MLflow</li><li>Counts of data-quality drops are logged per run</li></ul><p>This turns correctness into a time series, making it easier to detect volume drops, increased rejection rates, or unexpected KPI shifts after changes. For accurate dropped-row totals, aggregate drop-related metrics from the underlying event data rather than relying on event counts alone.</p><h3>10. Data Contracts (What They Are and How This Repo Enforces Them)</h3><p>A data contract is a defined set of rules specifying what values downstream tables are allowed to contain.</p><p>Data contracts are an increasingly common concept you should know for Databricks and data engineering in general, so it’s best we cover it here with a short primer. In a lakehouse pipeline, enforcing a contract at the Silver stage ensures the “clean” data layer is deterministic for all downstream consumers (Gold aggregations, dashboards, or ML models). Without explicit contracts, invalid records can slip through unnoticed and distort outputs.</p><p>Databricks provides a native mechanism for data contracts in Delta Live Tables called <strong>expectations</strong>. Expectations let you define rules and specify actions on violations: you can log failures, drop invalid records, or fail the pipeline update when a rule is broken.</p><ul><li><strong>Contract in this repo:</strong> Defined in /dlt/02_dlt_pipeline.sql (expectations in the Silver logic)</li><li><strong>Enforced rules (examples):<br>- </strong>vendor_id must be non-null.<br>- fare_amount must be &gt;= 0.</li><li><strong>Enforced at:</strong> The Silver validated table main_nyctaxi.ref.trips_valid.</li></ul><p><strong>Note (what happens to invalid rows)</strong></p><ul><li>With ON VIOLATION DROP ROW, violating rows are not retained in trips_valid and are not directly queryable as “failed rows.”</li><li>If you want to retain invalid records, implement a quarantine table explicitly (for example, create main_nyctaxi.ref.trips_invalid from trips_clean using the inverse of your contract predicates), and keep trips_valid as the contract-enforced dataset.</li><li>For auditing and metrics, read counts from logs/metrics rather than expecting rejected rows to be available automatically.</li></ul><h4>Demonstrating a contract break</h4><p><em>(Use batch ingestion so you control the input files.)</em></p><ol><li>Inject a bad record</li></ol><ul><li>Insert (or modify) a record so fare_amount = -5 or vendor_id IS NULL, then rerun /dlt/02_dlt_pipeline.sql.ipynb</li></ul><p>2. Re-run the Silver/Gold logic</p><ul><li>Re-run /dlt/02_dlt_pipeline.sql.ipynb to recompute trips_valid and downstream outputs.</li></ul><p>3. Verify enforcement</p><ul><li>Confirm the bad record does not appear in main_nyctaxi.ref.trips_valid.</li><li>Compare counts between trips_clean and trips_valid to quantify what was rejected.</li></ul><h4>Why this matters</h4><p>This is the <strong>trust boundary</strong> between layers. Bronze can be messy (raw landing), but Silver is where we enforce correctness (typed, cleaned, contract-valid). Gold can then assume Silver is reliable; no negative fares or missing vendor IDs, for example.</p><p>Contracts also create an <strong>audit trail</strong>: expectation metrics quantify how much data was rejected and why. This provides operational guardrails , if the drop rate suddenly spikes, it is treated as a data incident to investigate, rather than a mysterious dashboard error.</p><h4>(Optional) Trending in MLflow</h4><p>The script /monitoring/mlflow_demo.py provides a minimal bridge from “pipeline ran” to “health is measurable”.</p><p>It logs silver_rows each run, and logs data-quality drop signals derived from execution logs. Over time, you can compare runs in the MLflow UI to detect regressions early, and you can extend the query above to aggregate drop metrics if you need exact dropped-row totals.</p><h3>11. Cost and Compute Control</h3><p>Using Databricks is not free, and as the scale increases, cost can become a primary constraint for an organization. A large part of using Databricks is figuring out how to lower total spend.</p><p>Databricks usage is measured in <strong>Databricks Units (DBUs)</strong>. DBUs accumulate while compute is running; how usage is metered and billed depends on the product and cloud. In addition to DBUs, your cloud provider charges for the underlying VMs, storage, and network.</p><p>In practice, the main cost drivers are <strong>how long compute runs</strong> and <strong>which compute SKU</strong> you use.</p><h4>Typical compute types in a Databricks environment</h4><p>In a full Databricks setup, you might use:</p><ul><li>An <strong>all-purpose cluster</strong> running for 20 minutes at C DBU/hour (for example, 15 minutes for ingestion and 5 minutes for MLflow logging)</li><li>A <strong>DLT pipeline compute</strong> running for 25 minutes at P DBU/hour</li><li>A <strong>SQL warehouse</strong> running for 10 minutes at W DBU/hour</li></ul><p>The DBU consumption would be estimated as:</p><pre>Total DBUs = C * (20/60) + P * (25/60) + W * (10/60)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*-6Qmo3FO6kGZd8dYovleyA.png" /><figcaption>Generic Example of Compute Costs</figcaption></figure><h4><strong>Compute used in this tutorial (Free Edition)</strong></h4><p>In this walkthrough, we use:</p><ul><li>An <strong>all-purpose cluster</strong> to run ingestion, transformation, and validation notebooks</li><li>A <strong>Serverless Starter SQL Warehouse (2X-Small)</strong> to query Gold outputs</li></ul><p>As noted earlier, there is <strong>no managed DLT pipeline compute</strong> in the Free Edition. The DLT SQL logic is executed manually as standard Spark SQL workloads on a cluster, so there is no separate P term in practice.</p><p>A Free Edition cost estimate therefore simplifies to:</p><pre>Total DBUs ≈ C × (runtime / 60) + W × (warehouse_runtime / 60)</pre><blockquote><strong><em>Note:</em></strong><em> </em>C<em>, </em>P<em>, and </em>W<em> are placeholders for the DBU rates of the all-purpose cluster, pipeline compute, and warehouse compute. Use your workspace’s actual DBU rates/SKUs to compute the real cost in your currency.</em></blockquote><h3>12. Next Steps / Possible Improvements</h3><p>This repo is intentionally minimal. To make it production-ready, you could extend it in the following areas:</p><ul><li><strong>Stronger data contracts</strong><br> Add additional expectations (value ranges, cross-field checks). Quarantine rejected rows for analysis instead of dropping them outright, and optionally configure critical rules to fail execution on violation.</li><li><strong>Automated quality checks and observability</strong><br> Enforce row-count thresholds and maximum drop rates (for example, fail a run if Gold KPIs fall outside expected bounds). Log expectation failures and drop rates to MLflow or another metrics store for trend monitoring.</li><li><strong>Idempotency and backfills</strong><br> Support parameterized reprocessing (for example, by date range) and ensure reruns do not duplicate data or introduce inconsistencies.</li><li><strong>Environment separation and deployment</strong><br> Use separate development and production catalogs or schemas. Deploy pipelines via configuration and CI/CD (for example, Databricks Asset Bundles) instead of manual notebook execution.</li><li><strong>Cost and performance optimization</strong><br> Partition and optimize Silver and Gold tables for query performance. Enforce aggressive auto-termination on clusters and SQL warehouses to avoid idle compute costs.</li></ul><h3>13. Conclusion</h3><p>In this walkthrough, we built and ran an end-to-end lakehouse pipeline on Databricks: Delta Lake for storage, Apache Spark for compute, and Unity Catalog for governance. The Silver/Gold layer is authored in DLT-style SQL (tables + expectations) and executed as a notebook run. Unity Catalog provides the governed namespace, Spark runs the transformations on Databricks Runtime, and the expectations enforce basic data contracts as data moves from Bronze → Silver → Gold. MLflow is included as a minimal monitoring hook to track correctness signals (for example, row counts or drop rates) over time.</p><p>While this project runs without managed pipelines in the Free Edition, the structure and logic mirror a production Databricks deployment. The SQL definitions can be easily promoted to a managed Lakeflow (DLT) pipeline in a paid workspace with minor syntax adjustments (swapping TABLE for LIVE TABLE), unlocking platform-managed orchestration, incremental execution, and operational monitoring.</p><p>At this point, you should have a concrete mental model of how Databricks components fit together in practice and how a real lakehouse pipeline is structured from raw ingestion through curated business outputs.</p><p>If you want to explore the code, follow future work, or discuss building similar pipelines, you can find me here:</p><ul><li><strong>GitHub: </strong><a href="https://github.com/thejasono">https://github.com/thejasono</a></li><li><strong>LinkedIn:</strong> <a href="https://www.linkedin.com/in/thejasonowens/">https://www.linkedin.com/in/thejasonowens/</a></li><li><strong>Email: </strong>thisisjasonowens@gmail.com<br>(available for freelance, contract, full-time roles, and consultation)</li></ul><p>#DataEngineering #Databricks #Spark #Lakehouse #UnityCatalog</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=24c8f7236046" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Real-Time Data Pipeline with Kafka, Spark, Airflow, and S3]]></title>
            <link>https://medium.com/@thisisjasonowens/building-a-real-time-data-pipeline-with-kafka-spark-airflow-and-s3-c51950324d92?source=rss-ecbef8df7ad4------2</link>
            <guid isPermaLink="false">https://medium.com/p/c51950324d92</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[airflow]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[Jason O.]]></dc:creator>
            <pubDate>Wed, 24 Dec 2025 18:17:06 GMT</pubDate>
            <atom:updated>2026-01-06T22:09:18.058Z</atom:updated>
            <content:encoded><![CDATA[<p>To those just joining now: this blog is part of my <a href="https://medium.com/@thisisjasonowens/list/dataengineereing101-171b7a060c94"><strong>Data Engineering Fundamentals Series</strong></a>, where we break down essential tooling and workflows used in modern data pipelines. In the <a href="https://medium.com/@thisisjasonowens/an-end-to-end-data-pipeline-using-docker-airflow-dbt-and-postgres-34fbdef69cfd">previous article</a>, we built a basic data pipeline using Docker, Airflow, dbt, and Postgres — a batch-based pipeline that ran on a schedule and delivered data daily.</p><p>Now, we’re moving beyond periodic batch jobs into <strong>real-time, event-driven processing</strong>. In this tutorial, we’ll build a <strong>streaming data pipeline</strong> using APIs, Apache Kafka, Apache Spark, and AWS. If this sounds overwhelming, don’t worry, we’ll walk through the architecture one component at a time.</p><p>By the end, you’ll understand how data can flow continuously from a source API, into Kafka, through Spark for processing, and into Amazon S3 for long-term storage,all while using Docker to simulate a realistic production-like environment locally. We’ll also show how to use Airflow to orchestrate this system, making sure each component runs in the correct sequence.</p><p><em>This is a longer read (about an hour). Use the section headings to jump to what you care about, and take your time to digest it.</em></p><h3>0. View the Code on GitHub</h3><p>All code, configs, and examples for this pipeline live in the public repo:</p><p>👉 <a href="https://github.com/thejasono/data_pipeline-kakfa-spark-airflow-s3-athena">View the full project on GitHub</a></p><p>The repo includes:</p><ul><li>docker-compose.yaml to spin up Kafka, Spark, Airflow, and supporting services</li><li>Spark driver code in spark/app/spark_processing.py</li><li>Airflow DAGs in dags/ (including name_stream_dag)</li><li>Kafka producer helper in producer/kafka_streaming_service.py</li><li>Supporting scripts (e.g., airflow.sh, run_spark.ps1) and requirements</li></ul><p>Clone it, then prep your environment before starting:</p><ol><li>Create/update .env and .env.aws (or export environment variables) with:</li></ol><ul><li>S3_BUCKET, S3_OUTPUT_PREFIX, S3_CHECKPOINT_PREFIX, S3_REGION</li><li>AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, optional AWS_SESSION_TOKEN</li><li>If using MinIO, also set S3_ENDPOINT and S3_PATH_STYLE_ACCESS=true</li></ul><p>2. Make sure the Airflow DAG uses the same bucket, prefix, and region<br> (update dags/kafka_stream_dag.py or set matching environment variables).</p><p>Then run:</p><pre>docker compose up -d -- build</pre><p>You’ll have the full pipeline running locally in minutes.</p><h3>1. Pipeline Layers and Tools</h3><p>Before diving into the actual pipeline, let’s clarify what the pipeline is doing and introduce the main tools it consists of.</p><h4><strong>Pipeline Layers</strong></h4><p>Every modern data pipeline follows a rough pattern consisting of five core layers:</p><ul><li><strong>Ingestion</strong> — Pulling data from external sources (e.g. APIs, logs, databases)</li><li><strong>Transformation</strong> — Cleaning, typing, and restructuring data into usable formats.</li><li><strong>Storage</strong> — Persisting raw or transformed data (e.g. Postgres, S3, Data Lake).</li><li><strong>Orchestration</strong> — Managing execution order, schedules, retries, and dependencies.</li><li><strong>Visualization</strong> — Delivering results to end-users via dashboards or reports.</li></ul><p>In this project, we’re skipping visualization. The final output is structured JSON stored in an S3 bucket. So we’ll focus on:</p><ul><li>Ingestion</li><li>Storage</li><li>Transformation</li><li>Orchestration</li></ul><p>To make these layers concrete, the next section walks through the specific technologies that implement them. Once you see how each tool fits into a particular layer, the deeper code explanations later on will make far more sense. We’ll start with the ingestion/streaming layer, implemented with Kafka.</p><h3>2. The Tech Stack</h3><h3>2.1 Apache Kafka — Event Ingestion and Streaming</h3><p>Kafka is a <strong>distributed event streaming platform</strong>. “Distributed” means it can run across multiple machines to scale horizontally. “Event streaming” refers to its ability to handle continuous flows of data in real time.</p><p>At its core, Kafka facilitates communication between two roles:</p><ul><li><strong>Producers</strong> — applications that send data into Kafka.</li><li><strong>Consumers</strong> — applications that read data from Kafka.</li></ul><p>Kafka’s job is to move data from producers to consumers <strong>efficiently and durably</strong>, with minimal transformation along the way.</p><p>Each piece of data sent into Kafka is called an <strong>event</strong> — a discrete record of something that happened (e.g., a new user signing up, a sensor reading, etc.). Kafka stores these events as <strong>byte arrays</strong> in <strong>topics</strong>. A <em>topic</em> is essentially a named stream, or channel, for a specific kind of event. For example, a topic called “user_signups” might hold one event per new user registration.</p><h4><strong>Writing Events to Kafka (Producer Side)</strong></h4><p>To send data to Kafka, we must first serialize it — i.e., convert it into a byte array format that Kafka can handle. If we take user signup example from earlier, a simplified example using Python would look like:</p><pre>from kafka import KafkaProducer<br>import json<br><br>producer = KafkaProducer(bootstrap_servers=&#39;localhost:9092&#39;,<br>value_serializer=lambda v: json.dumps(v).encode(&#39;utf-8&#39;))<br>event_data = {&quot;username&quot;: &quot;jdoe&quot;, &quot;timestamp&quot;: &quot;2025–11–29T12:00:00Z&quot;}<br>producer.send(&quot;user_signups&quot;, value=event_data)<br>producer.flush()</pre><p>This snippet:</p><ul><li>Connects to a Kafka broker at localhost:9092.</li><li>Serializes a dictionary (event_data) into a JSON byte string.</li><li>Sends it to the topic “user_signups”.</li><li>Flushes the buffer to ensure the message is delivered.</li></ul><p>In this example we connect to localhost:9092 from the host machine. When the producer runs inside Docker, it instead uses the internal listener kafka:19092, which we inject via the KAFKA_BOOTSTRAP environment variable in docker-compose.yaml.</p><p>In this toy example we call flush() once to block until all messages are sent. In the real producer service (build_producer + publish_once) we instead rely on producer.poll() in the loop and a final flush() on shutdown, which is the recommended pattern in confluent_kafka.</p><h4><strong>Reading Events from Kafka (Consumer Side)</strong></h4><p>On the consumer side, the message needs to be <strong>deserialized</strong> — converted back from a byte string into a usable data structure (e.g., a Python dictionary or Spark DataFrame). A basic Python consumer would look like this:</p><pre>from kafka import KafkaConsumer<br>import json<br><br>consumer = KafkaConsumer(<br>    &quot;user_signups&quot;,<br>    bootstrap_servers=&quot;localhost:9092&quot;,<br>    group_id=&quot;demo_group&quot;,<br>    auto_offset_reset=&quot;earliest&quot;,<br>    value_deserializer=lambda x: json.loads(x.decode(&quot;utf-8&quot;)),<br>    )<br><br>for message in consumer:<br>    print(&quot;Consumed event:&quot;, message.value)</pre><ul><li>group_id=”demo_group” tells Kafka which consumer group this client belongs to. Kafka stores offsets per group, so using a named group means this consumer can stop and restart without losing its place.</li><li>In Spark, deserialization happens inside the read logic: the Kafka value column is cast to a string and then parsed with from_json into structured fields.</li></ul><h4><strong>Topics, Partitions, and Parallelism</strong></h4><p>So far, we’ve established that Kafka transmits data between <strong>producers</strong> and <strong>consumers</strong> via discrete units called <strong>events</strong>, and these events are grouped into <strong>topics</strong>. But a topic is not just a label or a mailbox. Internally, a topic behaves like a <strong>log</strong> and can be <strong>partitioned</strong> to achieve parallelism and scalability.</p><p>Let’s break that down.</p><p><strong>Topics as Append-Only Logs</strong></p><p>A Kafka topic is implemented as an <strong>append-only log</strong>.</p><p>Each event is assigned a monotonically increasing <strong>offset</strong>, which represents its position in the log. Offset 0 is simply the first event that was ever written to that partition. As new events arrive, they receive offsets 1, 2, 3, and so on.</p><p>For example, the log grows like this:</p><p><strong>Append #1</strong></p><pre>[ 0 ] user_id=42, action=login</pre><p><strong>Append #2</strong></p><pre>[ 0 ] user_id=42, action=login<br>[ 1 ] user_id=17, action=logout</pre><p><strong>Append #3</strong></p><pre>[ 0 ] user_id=42, action=login<br>[ 1 ] user_id=17, action=logout<br>[ 2 ] user_id=42, action=purchase</pre><p>After additional events are appended, the log as stored on disk might look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/411/1*-YpGTM8Qi5e_G6bnVDlc_w.jpeg" /><figcaption>final log table</figcaption></figure><p>Each row represents a fixed position in the log. Offsets are assigned once, in order of arrival, and never change.</p><p>Once an event is written:</p><ul><li>It cannot be changed,</li><li>It cannot be reordered,</li><li>It can only be read again.</li></ul><p>This log-based structure is critical. It allows Kafka to:</p><ul><li>Maintain event order (within a partition),</li><li>Let multiple consumers read the same data independently,</li><li>Replay historical data by re-reading from an earlier offset,</li><li>Recover from failures by resuming from a known offset.</li></ul><p>Kafka does not push messages to consumers. Consumers pull from the log at their own pace.</p><p><strong>Partitions: Scaling the Log</strong></p><p>A single log does not scale indefinitely. To achieve parallelism, Kafka allows a topic to be split into <strong>partitions</strong>.</p><p>A <strong>partition</strong> is a smaller, independently ordered append-only log that belongs to a topic. Each partition maintains its own log, its own ordering, and its own offset tracking.</p><p>For example:</p><pre><br>Partition 0:  [0][1][2][3]...<br>Partition 1:  [0][1][2]...<br>Partition 2:  [0][1][2][3][4]...</pre><p>Each partition:</p><ul><li>Preserves ordering internally,</li><li>Has its own offsets,</li><li>Can be written to and read from independently.</li></ul><p>Kafka guarantees ordering <strong>within a partition</strong>, not across the entire topic.</p><p>This is the key trade-off:</p><ul><li>More partitions → more parallelism and higher throughput,</li><li>But ordering is only preserved per partition.</li></ul><p>Partitioning is what allows Kafka to scale. It enables:</p><ul><li><strong>Parallelism</strong> — multiple producers can write to different partitions simultaneously, and multiple consumers in the same consumer group can read in parallel.</li><li><strong>Load distribution</strong> — partitions are spread across Kafka brokers, preventing any single broker from becoming a bottleneck.</li><li><strong>Horizontal scalability</strong> — increasing throughput by adding partitions and consumers rather than replacing hardware.</li></ul><p><strong>Mental Model</strong></p><p>The core idea is simple: <strong>topics are logs, and partitions are parallel logs</strong>.</p><p>You can think of it like this:</p><ul><li>An <strong>event</strong> is a single data record (e.g. {&quot;user_id&quot;: 123, &quot;action&quot;: &quot;click&quot;}).</li><li>A <strong>topic</strong> is a named stream of related events (e.g. &quot;user_activity&quot;).</li><li>A <strong>partition</strong> is a distinct slice of that topic (e.g. &quot;user_activity&quot;, partition 3).</li></ul><p>Events are written to a topic, which is composed of one or more partitions. Each partition is its own ordered log, enabling distributed and parallel processing.</p><p>A simplified analogy:</p><ul><li>Ingredients are events,</li><li>A cake is a topic,</li><li>Each slice of the cake is a partition.</li></ul><p>The slices can be handled independently, but the order of ingredients within a single slice is preserved.</p><p><strong>Our Pipeline Choice</strong></p><p>In this pipeline, the main topic is configured with a <strong>single partition</strong>, so all events are processed in strict order by one consumer task. This prioritizes correctness and simplicity.</p><p>Scaling to multiple partitions would allow multiple consumers to process events in parallel, but at the cost of losing global ordering across the topic. That trade-off — throughput versus ordering — is a recurring theme in Kafka, and one you will consciously revisit as your system grows.</p><h4><strong>Why does all of this matter?</strong></h4><p>At this point we understand how Kafka works internally. What matters in practice is how these mechanics translate into design decisions.</p><p>Kafka forces you to make explicit trade-offs between throughput, ordering, and fault tolerance — and those trade-offs shape how your entire pipeline behaves.</p><p>In practice, partitions are the lever you use to control parallelism. The number of partitions in a topic determines how much work can be done concurrently by consumers in the same consumer group. More partitions increase throughput and reduce processing latency. Fewer partitions simplify reasoning and preserve stronger ordering guarantees.</p><p>The cost of that parallelism is ordering. Kafka only guarantees order within a partition. If related events must be processed sequentially — such as all events for a single user — those events must be routed to the same partition using a partition key (for example, user_id). Kafka will not enforce this for you; it is an application-level responsibility.</p><p>Offsets complete the picture. Kafka tracks offsets per consumer group, not per consumer instance. This allows consumers to crash and restart without losing progress, lets multiple consumer groups read the same topic independently, and enables replay by resetting offsets to an earlier position.</p><p>Together, these properties explain Kafka’s core behavior:</p><ul><li>High throughput comes from partition-level parallelism,</li><li>Fault tolerance comes from durable logs and offset tracking,</li><li>Flexibility comes from decoupling producers and consumers entirely.</li></ul><p>Kafka is not just moving messages — it is giving you control over how data flows, how fast it is processed, and what guarantees you are willing to enforce.</p><p><strong>So to relate this all back into our pipeline:</strong></p><ul><li>The <strong>producer</strong> is a Python service that fetches data from a public API.</li><li>The <strong>consumer</strong> is Apache Spark, which continuously reads from Kafka and transforms the incoming data.</li><li>Kafka acts as the <strong>buffer</strong> between them, decoupling the ingestion and processing layers.</li></ul><p><strong>Kafka Deployment Notes (KRaft and Brokers)</strong></p><p>We’re running Kafka in <strong>KRaft mode</strong>, a newer configuration that removes the need for ZooKeeper. Previously, ZooKeeper was used to manage metadata and broker coordination. KRaft (Kafka Raft Metadata mode) simplifies deployment by consolidating those responsibilities directly into the Kafka brokers.</p><p>And just to cover as much as possible — A broker is a Kafka server instance. In older Kafka versions, brokers depended on ZooKeeper for metadata coordination. KRaft (Kafka Raft Metadata mode) consolidates metadata management into the brokers (eliminating ZooKeeper), which we discuss further in the Kafka Docker setup section.</p><h3>2.2 Apache Spark — Data Transformation</h3><p>Apache Spark is a distributed analytics engine designed to process large-scale data across clusters of machines. While it’s traditionally associated with batch workloads (processing large chunks of data at scheduled intervals), it also supports <strong>Structured Streaming</strong>, which allows us to process real-time data streams using the same DataFrame-based APIs familiar from batch processing.</p><p>In contrast to Kafka — which stores events as byte arrays optimized for transport — Spark reads data into <strong>DataFrames</strong>, and you then parse/cast those raw columns into structured fields. This enables transformation, filtering, aggregation, and more using a rich, SQL-like API.</p><h4><strong>Setting Up Spark: Cluster Model</strong></h4><p>To run Spark in our pipeline, we deploy a <strong>Spark Standalone cluster</strong>, which consists of:</p><ul><li><strong>One Master node</strong> — responsible for <strong>coordinating the cluster</strong>, assigning work, and managing the scheduling of Spark applications (called jobs).</li><li><strong>Multiple Worker nodes</strong> — these execute the actual tasks and transformations assigned by the master.</li><li><strong>Driver </strong>—your application process (created by spark-submit / your container) that builds the execution plan and schedules work.</li></ul><p>When you run a Spark job, the <em>driver</em> builds an execution plan (DAG) and schedules tasks onto executors running on workers; the <em>master</em> allocates resources for those executors.</p><h4><strong>Code Integration: Using PySpark</strong></h4><p>To interface with the Spark cluster in code, we use <strong>PySpark</strong> — the official Python API for Apache Spark. PySpark acts as the bridge between our Python application and the underlying distributed Spark engine.</p><p>In our case, PySpark is installed inside a Docker container that connects to the Spark master. When we launch a Spark job (e.g., from Airflow or the CLI), the Python script will communicate with the Spark cluster through PySpark.</p><h4><strong>What Is a Spark Job?</strong></h4><p>A <strong>Spark job</strong> is a unit of work submitted to the Spark cluster. It defines:</p><ul><li>The source of the data (e.g., Kafka),</li><li>How the data should be interpreted and parsed (e.g., casting byte arrays into strings, parsing JSON),</li><li>The transformations to apply (e.g., select, filter, groupBy, aggregate),</li><li>The destination (e.g., writing out to S3).</li></ul><p>Under the hood, Spark leverages Hadoop-compatible file systems (such as Amazon S3 or HDFS) for reading/writing data and uses its DAG scheduler to build an execution plan for your job. Spark uses Hadoop’s filesystem layer (via the s3a:// connector) to write to our S3 bucket. We’ll configure this later when setting up Spark’s S3 access (in practice this includes things like AWS/Hadoop JARs, credentials, endpoints—especially for MinIO—and fs.s3a.* settings).</p><p>To simplify:</p><ol><li>You define your input (Kafka topic).</li><li>You convert that input into a <strong>DataFrame</strong> using PySpark.</li><li>You define a series of transformations.</li><li>Spark breaks the transformations into stages and tasks.</li><li>Those tasks are distributed across workers.</li><li>Output is written to your destination (in our case, S3 as JSON files).</li></ol><h4><strong>Example: Reading from Kafka, Transforming, and Writing to S3</strong></h4><p>Here’s a simplified PySpark job that reads a stream of JSON events from a Kafka topic, parses them, and writes them to Amazon S3:</p><pre>from pyspark.sql import SparkSession<br>from pyspark.sql.functions import from_json, col<br>from pyspark.sql.types import StructType, StringType, TimestampType<br><br># Define schema of incoming data<br>schema = (<br>    StructType()<br>    .add(&quot;username&quot;, StringType())<br>    .add(&quot;timestamp&quot;, TimestampType())<br>)<br><br># Create Spark session (ensure we target the Standalone cluster)<br>spark = (<br>    SparkSession.builder<br>    .appName(&quot;KafkaToS3Stream&quot;)<br>    .master(&quot;spark://spark-master:7077&quot;)<br>    .getOrCreate()<br>)<br><br># Read stream from Kafka<br>raw_df = (<br>    spark.readStream<br>    .format(&quot;kafka&quot;)<br>    .option(&quot;kafka.bootstrap.servers&quot;, &quot;kafka:19092&quot;)<br>    .option(&quot;subscribe&quot;, &quot;user_signups&quot;)<br>    .load()<br>)<br><br># Extract and parse Kafka value<br>parsed_df = (<br>    raw_df.selectExpr(&quot;CAST(value AS STRING)&quot;)<br>    .select(from_json(col(&quot;value&quot;), schema).alias(&quot;data&quot;))<br>    .select(&quot;data.*&quot;)<br>)<br><br># Write to S3 in JSON format<br>(<br>    parsed_df.writeStream<br>    .format(&quot;json&quot;)<br>    .option(&quot;path&quot;, &quot;s3a://my-bucket/output/&quot;)<br>    .option(&quot;checkpointLocation&quot;, &quot;s3a://my-bucket/checkpoints/&quot;)<br>    .outputMode(&quot;append&quot;)<br>    .start()<br>    .awaitTermination()<br>)</pre><blockquote><strong>Schema note: </strong>The snippet below is illustrative. The real streaming job in spark/app/spark_processing.py uses fields name, gender, address, city, nation, zip, latitude, longitude, and email. Those are the columns that land in S3/Athena—refer to that file for the authoritative schema.</blockquote><p>What’s happening here:</p><ul><li>Spark reads the Kafka byte stream.</li><li>It casts the Kafka value to a string, then parses it into structured columns using a schema.</li><li>The resulting DataFrame is written continuously to an S3 bucket in append mode.</li><li>A checkpoint location is used to track progress and ensure fault tolerance.</li></ul><p>Timestamp parsing note: TimestampType() will only parse timestamps correctly if the incoming JSON is in a format Spark recognizes (often ISO-8601). If parsing fails, the timestamp field will become null, and you may need to parse it explicitly with to_timestamp(...).</p><p>One practical warning: writing with .format(&quot;json&quot;) is a <strong>file sink</strong>. With micro-batches, it will typically produce many small files. That is fine for a learning pipeline, but production pipelines commonly write <strong>Parquet</strong> and/or use a table format (Iceberg/Delta) to avoid small-file problems.</p><p>This job runs continuously, processing micro-batches under the hood — meaning Spark collects new events every few seconds and processes them in small groups. This model offers the scalability of batch with near-real-time behavior. Spark also supports a Continuous Processing mode (Trigger.Continuous), though micro-batch is more common and easier to orchestrate with Airflow.</p><h3>2.3 S3</h3><p>This is where the processed data from the Spark job lands. Amazon S3 (Simple Storage Service) is an object-storage system designed for durability, scalability, and cost-efficiency.</p><p>In concrete terms:</p><ul><li>S3 stores <strong>objects</strong>, not traditional filesystem files.</li><li>Each object is identified by a <strong>key</strong>. Keys look like file paths, but the underlying storage is flat.</li><li>Every object lives inside a <strong>bucket</strong>, and Spark accesses objects using URIs like:</li></ul><pre>s3a://my-bucket/path/to/object.json</pre><p>Here, my-bucket is the bucket name and path/to/object.json is the object key. The slashes are not folders; they are characters in the key string (the S3 console treats prefixes as “folders” for convenience).</p><p>In this pipeline, S3 acts as our <strong>data lake</strong> — the destination for transformed, structured data.</p><p>A data lake is a centralized storage layer that keeps raw and processed data in native formats (JSON, Parquet, CSV, etc.) and applies schema at query time (<strong>schema-on-read</strong>) rather than enforcing rigid schemas on write. Tools like Athena or Redshift Spectrum can query data directly from S3 using SQL once tables/metadata are defined.</p><h4>Why Use S3?</h4><p>We use S3 because:</p><ul><li>It decouples <strong>storage from compute</strong> (Spark can finish; data persists).</li><li>It integrates cleanly with downstream tools (Athena, Redshift Spectrum, Snowflake external tables, dbt, and engines that read Parquet/JSON from S3).</li><li>It scales automatically with data volume and supports high-throughput writes.</li></ul><h4>AWS vs MinIO</h4><p>To use AWS S3, you need:</p><ul><li>A bucket name, and</li><li>Valid credentials with write access.</li></ul><p>For local development, you do not want to use AWS directly, you can use <strong>MinIO</strong>, an S3-compatible local server. From Spark’s perspective, MinIO still appears as an S3 endpoint via the same s3a:// paths; only the endpoint and credentials differ.</p><h4><strong>AWS Credentials and Configuration</strong></h4><p>Spark needs credentials to authenticate with S3. Common options:</p><ul><li><strong>Environment variables</strong> in the Spark container:<br> AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, optionally AWS_SESSION_TOKEN</li><li><strong>The host machine’s standard AWS config files</strong> (~/.aws/credentials, ~/.aws/config) if mounted into the container.</li><li><strong>Explicit Spark Hadoop configuration</strong> (what this project uses)</li></ul><p>In this setup, values from .env.aws are injected into the Spark container as environment variables, and the Spark session is initialized using those values. For production systems, credentials should come from IAM roles, instance profiles, or a secrets manager rather than embedding keys.</p><h4>Connecting Spark to S3</h4><p>The key step is configuring Spark’s Hadoop layer so that s3a://… URLs resolve correctly. The configuration includes:</p><ul><li>Region (e.g. eu-west-1)</li><li>Access key / secret key / optional session token</li><li>Endpoint (AWS S3 or MinIO, e.g. <a href="http://minio:9000)">http://minio:9000)</a></li><li>SSL and path-style access settings</li></ul><p>Example (conceptual) initialization:</p><pre>spark = initialize_spark_session(<br>    app_name,<br>    region=s3_region,<br>    access_key=access_key,<br>    secret_key=secret_key,<br>    session_token=session_token,<br>    endpoint=endpoint_host,     # e.g. s3.amazonaws.com or http://minio:9000<br>    path_style=path_style,      # True for MinIO, usually False for AWS<br>    ssl_enabled=ssl_pref,<br>)</pre><p>Once this session is created, any s3a://bucket/… read or write targets the configured backend (AWS or MinIO).</p><p><strong>Example Write Logic</strong></p><p>Once the data is transformed in Spark, writing to S3 is straightforward:</p><pre>(parsed_df.writeStream<br>    .format(&quot;json&quot;)<br>    .option(&quot;path&quot;, &quot;s3a://my-bucket/output/&quot;)<br>    .option(&quot;checkpointLocation&quot;, &quot;s3a://my-bucket/checkpoints/&quot;)<br>    .outputMode(&quot;append&quot;)<br>    .start()<br>    .awaitTermination()<br>)</pre><h4><strong>Key points</strong></h4><ul><li><strong>The path is an S3A URI</strong>:<br> s3a://my-bucket/output/</li><li>Under the hood, Spark will create objects with keys like:<br> output/part-0000-….json</li><li><strong>Data is written incrementally</strong> as JSON files in append mode.<br> Each micro-batch from the Kafka stream produces one or more new objects under output/.</li><li><strong>The checkpointLocation is not optional for Structured Streaming.</strong><br> Spark’s .writeStream requires a checkpoint directory so it can store:<br>- Offsets already processed<br>- Batch IDs<br>- State needed for exactly-once/at-least-once semantics (depending on the sink)</li></ul><p>On restart, Sparkb reads the checkpoint data to reconstruct its state and continue from where it left off instead of reprocessing the entire stream.</p><ul><li>In this project, both the <strong>output data</strong> and the <strong>checkpoint metadata</strong> are stored in S3 (the output/ and checkpoints/ prefixes within the same bucket).</li><li>Later, in the Athena section, we will query the output/ prefix using SQL to confirm that the stream has landed correctly.</li></ul><h3>2.4 Apache Airflow</h3><p>Apache Airflow is a <strong>workflow scheduler</strong> widely used in data engineering to define, monitor, and manage pipelines. In Airflow, each workflow is expressed as a <strong>Directed Acyclic Graph (DAG)</strong>: a collection of tasks connected by dependencies, with no cycles. The DAG defines <em>what depends on what</em>; Airflow then executes that graph repeatedly as <strong>DAG runs</strong> on a schedule or trigger.</p><p>Although our overall system processes streaming data, we use Airflow to orchestrate the pipeline in <strong>finite, repeatable execution windows</strong>. Instead of running forever inside Airflow, each component is started, verified, and managed in controlled bursts (for example, every few minutes). This gives us operational control and observability without abandoning the benefits of a streaming architecture.</p><p>Using Airflow allows us to:</p><ul><li><strong>Schedule regular ingestion and processing windows</strong> (e.g., run every few minutes).</li><li>Automatically retry tasks when transient failures occur, with configurable alerting.</li><li><strong>Track execution history</strong> through logs and Airflow’s built-in web interface.</li></ul><p>To orchestrate containerized components like Spark, we use the <strong>DockerOperator</strong>, which runs specific commands inside Docker containers. For lightweight checks and logic, such as verifying that Kafka is reachable or confirming S3 credentials are valid, we use the <strong>PythonOperator</strong>.</p><p>Because this is a <strong>streaming pipeline</strong>, there aren’t many complex DAG tasks, the DAG itself is intentionally simple. We are not chaining long sequences of transformations inside Airflow. Instead, we rely on <strong>preflight and health-check tasks</strong> to:</p><ul><li>To validate that each core service (Kafka, Spark, S3) is up and reachable.</li><li>To ensure the pipeline is wired correctly end to end before any processing begins.</li></ul><p>This hybrid approach — using Airflow for orchestration, even in a streaming context — keeps things modular, transparent, and recoverable without compromising the nature of continuous data flow.</p><h3>2.5 How This Would Differ in Actual Production vs Local Setup</h3><p>In our local environment, we’re running <strong>Kafka</strong>, <strong>Spark</strong>, <strong>Airflow</strong>, and other components using <strong>Docker Compose</strong> on a single machine. This setup <em>resembles</em> a distributed system because each component runs in its own container, but in reality they all share the same underlying CPU, memory, and disk.</p><p>In a real production environment, these services are rarely self-hosted with Docker Compose. Instead, each major layer of the pipeline is typically replaced with a <strong>managed cloud service</strong>. Managed services handle scaling, high availability, upgrades, and monitoring for you, reducing operational overhead and failure risk.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/1*ne-TEpqI8S0wYJGzPqatRA.png" /><figcaption>Local vs. Production Architecture Components</figcaption></figure><p>In AWS-based production setups, common replacements include:</p><ul><li><strong>Amazon MSK</strong>, AWS’s managed Kafka service</li><li><strong>Amazon Kinesis Data Streams</strong>, a managed alternative for real-time event ingestion</li><li><strong>AWS Glue Streaming</strong>, a serverless Spark-based streaming environment</li><li><strong>Amazon MWAA</strong>, AWS’s managed Apache Airflow service</li><li><strong>EMR Serverless</strong>, which supports Spark job execution, including streaming workloads</li></ul><p><strong>Why This Still Matters</strong></p><p>Although the <em>tools</em> change in production, the <strong>architecture and data flow remain the same</strong>. What matters most is understanding the role of each layer in the system: where data is ingested, how it is processed, and where it is ultimately stored.</p><p>That understanding allows you to replace components without changing the overall logic of the pipeline. For example, moving to production may involve:</p><ul><li>changing endpoints (local Kafka → MSK, MinIO → S3)</li><li>changing execution backends (Spark Standalone → EMR or Glue)</li><li>changing where orchestration runs (local Airflow → MWAA or Step Functions)</li></ul><p>Rather than redesigning the pipeline’s logic.</p><p><strong>With that context established, we can now zoom back in to the local architecture we are actually running in this project.</strong> The next section revisits the end-to-end flow so that, when we start walking through Docker Compose and service configuration, you always know where each component fits.</p><h3>3. Pipeline Overview</h3><p><strong>Revisiting the Pipeline Architecture</strong></p><p>Now that we’ve covered the core technologies and what they would map to in production, let’s re-anchor on the <strong>end-to-end pipeline flow we are implementing locally</strong>.</p><p>At a high level, the pipeline consists of a small number of clearly separated layers, each with a single responsibility. The data flows through the system from ingestion to storage, while orchestration sits above the data path and controls <em>when</em> each step runs.</p><p><strong>Pipeline Layers and Responsibilities</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*fLR_WZYsL7xju8in0IA4zg.jpeg" /><figcaption>Mapping Each Pipeline Component To Their Respective Role</figcaption></figure><p>You can visualize the pipeline like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EX6HLGMcGzQYKO_6BnDhiQ.gif" /><figcaption>Conceptual pipeline flow. Airflow orchestrates execution by running preflight checks (for example, Kafka topic availability and object-store write access) before triggering the producer and the Spark job. Airflow manages <em>when</em> components run and fails fast if dependencies are not ready, but it does not participate in the data path itself.</figcaption></figure><p><strong>Note:</strong> This diagram is a conceptual illustration of the architecture, not a 1:1 representation of the local Docker Compose setup. The Kafka cluster and S3 bucket shown represent logical components; locally, Kafka may run as a single broker and object storage is provided by MinIO, which is S3-compatible.</p><p>This separation of concerns — Kafka for transport, Spark for processing, object storage for persistence, and Airflow for orchestration — results in a modular, testable pipeline that mirrors real production architectures while remaining simple enough to run locally.</p><h3>4. Docker Compose</h3><p>All components in this pipeline run inside Docker containers. Containers provide consistent, isolated environments for Kafka, Spark, Airflow, and S3-compatible storage, ensuring the system behaves the same way across machines. <strong>Docker Compose</strong> is used to define how these containers are built, networked, and run together on a local system.</p><p>Rather than managing each service manually, Docker Compose allows the entire stack to be described declaratively in a single docker-compose.yaml file. That file is fairly large and not practical to inline here, but it is worth opening and reading directly in the repository. While it may look complex at first glance, it is structured around a small number of clear service groups.</p><p>At a high level, the Compose file is organized into three sections:</p><ul><li><strong>Airflow services</strong>: the Airflow webserver, scheduler, and a PostgreSQL database used for Airflow metadata.</li><li><strong>Kafka services</strong>: a Kafka broker running in KRaft mode, along with Kafka UI, Kafka Connect, and the Schema Registry.</li><li><strong>Spark services</strong>: a Spark master, one or more Spark workers, and a short-lived container responsible for submitting the streaming job using spark-submit.</li></ul><blockquote>Be aware that the spark_streaming container (the “short-lived” container we’re referring to) runs availableNow, so each run is finite. But because it’s set to restart: unless-stopped, Docker will relaunch it—drain the backlog, exit, then start again. For a true one-shot run, drop the restart policy; for an always-on stream, swap availableNow for a continuous trigger.</blockquote><p>Each service runs in its own container and is configured using environment variables and mounted volumes. This keeps Kafka, Spark, and Airflow cleanly separated while still allowing them to communicate over well-defined interfaces.</p><p>Some services use standard, off-the-shelf images — for example, postgres:16 for the Airflow metadata database. Others use <strong>custom images</strong> built from Dockerfiles included in this project, such as custom-airflow and custom-spark. These custom images ensure that the correct Python dependencies, Spark packages, and Kafka connectors are available inside the runtime environment.</p><p>Docker Compose also defines two pieces of shared infrastructure that tie the stack together:</p><ul><li><strong>Networks</strong>, which allow containers to discover and communicate with each other by service name (for example, kafka-broker, spark-master, or airflow-webserver).</li><li><strong>Volumes</strong>, which persist state across container restarts, such as PostgreSQL data and Airflow logs.</li></ul><p>We will not inline the entire docker-compose.yaml file here. Instead, the rest of this post walks through it section by section:</p><ul><li>In the <strong>Kafka</strong> section, we examine the broker, controller quorum, UI, Kafka Connect, and Schema Registry services.</li><li>In the <strong>Spark</strong> section, we cover the Spark master, workers, and the streaming driver container that runs spark-submit.</li><li>In the <strong>Airflow</strong> section, we explain how the webserver, scheduler, and metadata database are wired together.</li></ul><p>Once the docker-compose.yaml file and the accompanying Dockerfiles are in place, the entire stack can be started from the project root with a single command:</p><pre>docker compose up -d --build</pre><blockquote>This is of course assuming you have the .env/.env.aws populated with your own version of S3_BUCKET, S3_OUTPUT_PREFIX, S3_CHECKPOINT_PREFIX, S3_REGION, AWS creds (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, optional AWS_SESSION_TOKEN), and S3_ENDPOINT + S3_PATH_STYLE_ACCESS=true when using MinIO</blockquote><blockquote>Also make sure you set KAFKA_CLUSTER_ID (e.g., KAFKA_CLUSTER_ID=$(kafka-storage random-uuid)) in your .env so the KRaft broker can start.</blockquote><p>Docker Compose will build any required custom images, start all containers, attach them to the appropriate networks, and leave the stack running in the background. At that point, Kafka, Spark, and Airflow are all available and ready to support the rest of the pipeline.</p><p>If you are looking for a more detailed walkthrough of how to read and understand Docker Compose files, you can refer to a previous project of mine:<br> <a href="https://medium.com/@thisisjasonowens/an-end-to-end-data-pipeline-using-docker-airflow-dbt-and-postgres-34fbdef69cfd">https://medium.com/@thisisjasonowens/an-end-to-end-data-pipeline-using-docker-airflow-dbt-and-postgres-34fbdef69cfd</a></p><p>In that post, I break down the basic syntax of a Docker Compose service and explain how each section of the YAML file works line by line. In this project, we will follow a similar approach, but with less emphasis on Docker fundamentals and more focus on how the services are wired together to support the pipeline.</p><h3>5. Kafka</h3><p>We’ll start with the <strong>Kafka</strong> part of the stack, since everything else builds on it.</p><p>Kafka is the event backbone of this stack: everything else (Spark, Airflow, S3) is built on top of a stream of messages moving through Kafka topics.</p><p>Earlier we covered the basic data model:</p><ul><li><strong>Topics</strong> — named streams of events.</li><li><strong>Partitions</strong> — shards of a topic for parallelism.</li><li><strong>Producers</strong> — write events to topics.</li><li><strong>Consumers</strong> — read events from topics.</li></ul><p>To understand how Kafka actually runs in our Docker setup, we need to add a few more ideas:</p><ul><li><strong>Brokers</strong></li><li><strong>Controllers</strong></li><li><strong>Quorums (KRaft)</strong></li><li><strong>Consumer groups</strong></li><li><strong>Serialization</strong></li></ul><p>Once those are clear, we can map them directly onto the Docker services in our Compose file.</p><h4><strong>Brokers, Controllers, Quorums, and Consumer Groups</strong></h4><p><strong>Broker</strong></p><p>A <strong>Kafka broker</strong> is just a single Kafka server process. It:</p><ul><li>Stores topic partitions on disk,</li><li>Accepts messages from producers,</li><li>Serves data to consumers,</li><li>Replicates data to other brokers (in multi-broker clusters).</li></ul><p>In this project we run <strong>one broker</strong>, so all topic partitions live on that single node.</p><p><strong>Controller and quorum (KRaft mode)</strong></p><p>Kafka also needs a component to coordinate cluster metadata:</p><ul><li>Which brokers are alive,</li><li>Which broker is leader for each partition,</li><li>Where topics and partitions are placed.</li></ul><p>In modern Kafka, this is handled by a <strong>controller</strong> backed by a <strong>metadata quorum</strong>. Kafka calls this <strong>KRaft</strong> (Kafka + Raft). The quorum is a small set of controller nodes that run the Raft consensus algorithm to agree on metadata changes, so the cluster has a single source of truth.</p><p>In older deployments ZooKeeper did this job. In our setup we use a <strong>single-node KRaft cluster</strong> for simplicity: one process acts as both <strong>broker</strong> and <strong>controller</strong>.</p><p>That is why the environment variables look like this:</p><pre>KAFKA_PROCESS_ROLES=broker,controller<br>KAFKA_NODE_ID=1<br>KAFKA_CONTROLLER_QUORUM_VOTERS=1@kafka:29093</pre><p>This means:</p><ul><li>The kafka container is <strong>both</strong> broker and controller.</li><li>Its node ID is 1.</li><li>The controller quorum consists of just this node at kafka:29093.</li></ul><p>In production you would run multiple brokers and multiple controllers, but the logical structure is the same.</p><p>KRaft clusters also require a <strong>cluster ID</strong> (UUID) to initialize/format storage. In Confluent’s Docker configuration, this is supplied via CLUSTER_ID.</p><p><strong>Consumer groups</strong></p><p>The last core concept is the <strong>consumer group</strong>. A consumer group is a set of consumers that share the work of reading from a topic:</p><ul><li>Each partition of a topic is assigned to <strong>one</strong> consumer in the group.</li><li>Different consumers can read different partitions <strong>in parallel</strong>.</li><li>From Kafka’s point of view, a consumer group represents one logical “application”.</li></ul><p>In this stack, Spark Structured Streaming acts as a <strong>single consumer group</strong>:</p><ul><li>The streaming job subscribes to a topic as one group,</li><li>Spark spins up multiple tasks to read from different partitions concurrently,</li><li>Offsets are tracked per group so the job can restart without losing its place.</li></ul><p>In this stack, Spark Structured Streaming reads Kafka topics as a streaming source. Spark tracks progress using its <strong>checkpoint directory</strong> (so it can restart without losing its place), rather than relying on “classic” Kafka offset commits as your primary source of truth.</p><h4><strong>Serialization: turning data into bytes</strong></h4><p>Kafka does not understand JSON, dicts, or objects. It only transports <strong>byte arrays</strong>.</p><p>That means every producer must:</p><ol><li>Take an in-memory object (e.g. a Python dict),</li><li><strong>Serialize</strong> it into bytes (commonly JSON-encode + UTF-8),</li><li>Send those bytes to Kafka.</li></ol><p>Consumers do the reverse:</p><ol><li>Read bytes from Kafka,</li><li><strong>Deserialize</strong> them (UTF-8 decode + json.loads),</li><li>Work with the resulting Python object.</li></ol><p>So whenever you see JSON in the producer/consumer code, remember: Kafka itself is blind to JSON. Serialization is what makes the bridge between “Python objects” and “Kafka bytes”.</p><h4><strong>Topic creation from code</strong></h4><p>We don’t pre-create topics via CLI. Instead, the producer calls Kafka’s Admin API at startup (ensure_topic) to check whether names_topic exists and creates it if missing. In this demo we create the topic with num_partitions=1 and replication_factor=1, matching a single-broker local stack. After the topic exists, the producer enters its loop: fetch user JSON → normalize/flatten → serialize to UTF-8 JSON bytes → publish to Kafka. producer, and have the required topic(s) created automatically.</p><h4><strong>Kafka Services in Our Docker Compose</strong></h4><p>In the Docker Compose file, the Kafka layer is split into four services:</p><ol><li>kafka — Kafka broker (in KRaft mode)</li><li>kafka_connect — Kafka Connect</li><li>kafka_schema_registry — Schema Registry</li><li>kafka_ui — Kafka UI</li></ol><p>Each has a distinct role in a realistic Kafka ecosystem.</p><h4><strong>1. kafka — The Broker (KRaft)</strong></h4><pre><br>  kafka:<br>    image: confluentinc/cp-kafka:7.6.1<br>    container_name: kafka<br>    volumes:<br>      - kafka_data:/var/lib/kafka/data<br>    environment:<br>      - KAFKA_PROCESS_ROLES=broker,controller<br>      - KAFKA_NODE_ID=1<br>      - KAFKA_CONTROLLER_QUORUM_VOTERS=1@kafka:29093<br>      - KAFKA_CONTROLLER_LISTENER_NAMES=CONTROLLER<br>      - KAFKA_LISTENERS=INTERNAL://0.0.0.0:19092,EXTERNAL://0.0.0.0:9092,CONTROLLER://0.0.0.0:29093<br>      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT<br>      - KAFKA_INTER_BROKER_LISTENER_NAME=INTERNAL<br>      - KAFKA_ADVERTISED_LISTENERS=INTERNAL://kafka:19092,EXTERNAL://${DOCKER_HOST_IP:-localhost}:9092<br>      - KAFKA_CLUSTER_ID=${KAFKA_CLUSTER_ID}<br>      - CLUSTER_ID=${KAFKA_CLUSTER_ID} <br>      - KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1<br>      - KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR=1<br>      - KAFKA_TRANSACTION_STATE_LOG_MIN_ISR=1<br>      - KAFKA_LOG_DIRS=/var/lib/kafka/data            # explicit<br>      - KAFKA_LOG4J_LOGGERS=kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO<br>    ports:<br>      - &quot;9092:9092&quot;<br>    healthcheck:<br>      test: [&quot;CMD-SHELL&quot;, &quot;kafka-broker-api-versions --bootstrap-server localhost:9092 &gt;/dev/null 2&gt;&amp;1&quot;]<br>      interval: 30s<br>      timeout: 10s<br>      retries: 5<br>    networks:<br>      - kafka_network<br>      - docker_streamingKey points:</pre><p><strong>Persistent storage</strong></p><ul><li>kafka_data:/var/lib/kafka/data is where Kafka stores log segments (topic partitions).</li><li>Using a named volume means data survives container restarts.</li></ul><p><strong>Roles and identity (KRaft)</strong></p><ul><li>KAFKA_PROCESS_ROLES=broker,controller makes this node both broker and controller</li><li>KAFKA_NODE_ID=1 sets the node identity</li><li>KAFKA_CONTROLLER_QUORUM_VOTERS=1@kafka:29093 defines the quorum (single node here)</li><li>CLUSTER_ID=${KAFKA_CLUSTER_ID} supplies the KRaft cluster UUID required to initialize storage</li></ul><p><strong>Listeners (internal vs external)</strong></p><ul><li>INTERNAL://kafka:19092 is for other containers on the Docker network (Spark, Airflow, Connect, UI)</li><li>EXTERNAL://localhost:9092 (or ${DOCKER_HOST_IP}) is for clients on your host machine</li><li>CONTROLLER://kafka:29093 is used for controller/quorum traffic</li></ul><p><strong>Important:</strong> KAFKA_ADVERTISED_LISTENERS is what clients are told to connect to, so it must match how the client reaches Kafka (container-to-container vs host-to-container).</p><p><strong>Replication factors</strong></p><ul><li>The internal Kafka topics for offsets/transactions are set to replication factor 1 because we have one broker. Confluent explicitly calls out replication factor 1 as required in single-node setups for certain internal topics.</li></ul><h4><strong>2. kafka_connect — Kafka Connect</strong></h4><p>Kafka Connect is a framework for running managed <strong>source</strong> and <strong>sink</strong> connectors (e.g., DB → Kafka, Kafka → S3). In this project, Spark handles the Kafka → S3 write path, but wiring Connect into the stack makes it easy to add connectors later.</p><pre>kafka_connect:<br>    image: confluentinc/cp-kafka-connect:7.6.1<br>    depends_on:<br>      - kafka<br>    ports:<br>      - &quot;8083:8083&quot;<br>    environment:<br>      - CONNECT_BOOTSTRAP_SERVERS=kafka:19092<br>      - CONNECT_REST_PORT=8083<br>      - CONNECT_GROUP_ID=connect-cluster<br>      - CONNECT_CONFIG_STORAGE_TOPIC=_connect-configs<br>      - CONNECT_OFFSET_STORAGE_TOPIC=_connect-offsets<br>      - CONNECT_STATUS_STORAGE_TOPIC=_connect-status<br>      - CONNECT_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter<br>      - CONNECT_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter<br>      - CONNECT_INTERNAL_KEY_CONVERTER=org.apache.kafka.connect.json.JsonConverter<br>      - CONNECT_INTERNAL_VALUE_CONVERTER=org.apache.kafka.connect.json.JsonConverter<br>      - CONNECT_KEY_CONVERTER_SCHEMAS_ENABLE=false<br>      - CONNECT_VALUE_CONVERTER_SCHEMAS_ENABLE=false<br>      - CONNECT_PLUGIN_PATH=/usr/share/java,/etc/kafka-connect/jars<br>      - CONNECT_LOG4J_ROOT_LOGLEVEL=INFO<br>      - CONNECT_LOG4J_LOGGERS=org.reflections=ERROR<br>      - CONNECT_OFFSET_FLUSH_INTERVAL_MS=5000<br>      - CONNECT_REST_ADVERTISED_HOST_NAME=kafka_connect<br>      - CONNECT_CONFIG_STORAGE_REPLICATION_FACTOR=1<br>      - CONNECT_OFFSET_STORAGE_REPLICATION_FACTOR=1<br>      - CONNECT_STATUS_STORAGE_REPLICATION_FACTOR=1<br>    networks:<br>      - kafka_network<br>      - docker_streaming<br>    healthcheck:<br>      test: [&quot;CMD-SHELL&quot;, &quot;curl -fsS http://localhost:8083/connectors &gt;/dev/null&quot;]<br>      interval: 30s<br>      timeout: 10s<br>      retries: 5</pre><p><strong>Startup ordering</strong></p><ul><li>depends_on ensures <strong>container start order</strong>, but it does <strong>not</strong> guarantee Kafka is ready to accept connections at the moment Connect starts. Readiness is handled by retries/backoff in clients and by the healthcheck for observability. <a href="https://docs.docker.com/compose/how-tos/startup-order/?utm_source=chatgpt.com">Docker Documentation+1</a></li></ul><p><strong>Connect internal state</strong></p><ul><li>_connect-configs, _connect-offsets, _connect-status are Kafka topics used by Connect to store its own config, offsets, and status</li><li>Replication factors are set to 1 because this is a single-broker setup.</li></ul><p><strong>JSON converter settings</strong></p><ul><li>JsonConverter with *_SCHEMAS_ENABLE=false is the setting you use when you want <strong>plain JSON payloads</strong> (instead of the {schema, payload} envelope). <a href="https://www.confluent.io/blog/kafka-connect-deep-dive-converters-serialization-explained/?utm_source=chatgpt.com">Confluent</a></li></ul><p><strong>Plugins</strong></p><ul><li>CONNECT_PLUGIN_PATH is where additional connector JARs can be added (commonly via a volume mount into /etc/kafka-connect/jars).</li></ul><h4><strong>3. kafka_schema_registry — Schema Registry</strong></h4><p>Schema Registry manages schemas for Kafka messages (Avro, Protobuf, JSON Schema) and allows safe schema evolution and comptibility rules.</p><pre>kafka_schema_registry:<br>    image: confluentinc/cp-schema-registry:7.6.1<br>    depends_on:<br>      - kafka<br>    ports:<br>      - &quot;8081:8081&quot;<br>    environment:<br>      - SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS=${SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS:-PLAINTEXT://kafka:19092}<br>      - SCHEMA_REGISTRY_KAFKASTORE_SECURITY_PROTOCOL=${SCHEMA_REGISTRY_KAFKASTORE_SECURITY_PROTOCOL:-PLAINTEXT}<br>      - SCHEMA_REGISTRY_HOST_NAME=${SCHEMA_REGISTRY_HOST_NAME:-kafka_schema_registry}<br>      - SCHEMA_REGISTRY_LISTENERS=${SCHEMA_REGISTRY_LISTENERS:-http://0.0.0.0:8081}<br>      - SCHEMA_REGISTRY_KAFKASTORE_TOPIC_REPLICATION_FACTOR=1<br>    networks:<br>      - kafka_network<br>      - docker_streaming</pre><p><strong>Important limitation (by design in this stack)</strong></p><ul><li>With the current Connect configuration (JSON converter) and with a Python producer sending raw JSON bytes, Schema Registry is <strong>not enforcing anything automatically</strong>.</li><li>Schema Registry becomes “active” when producers/consumers (or Connect) use Schema Registry-aware converters/serializers such as Avro/Protobuf/JSON Schema converters. <a href="https://docs.confluent.io/platform/current/schema-registry/connect.html?utm_source=chatgpt.com">docs.confluent.io</a></li></ul><p>So: Schema Registry is included to mirror a realistic Kafka ecosystem and to support future evolution, but the core pipeline can run without it.</p><p><strong>4. kafka_ui — Kafka UI</strong></p><p>Kafka UI provides a web interface to inspect topics, browse messages, view consumer groups/offsets, and see connector and schema registry integrations.</p><pre>  kafka_ui:<br>    image: provectuslabs/kafka-ui:v0.7.2            # pin to a fixed tag; avoid :latest<br>    container_name: kafka-ui-1<br>    depends_on:<br>      - kafka<br>      - kafka_schema_registry<br>      - kafka_connect<br>    ports:<br>      - &quot;8888:8080&quot;                                  # http://localhost:8888<br>    environment:<br>      - KAFKA_CLUSTERS_0_NAME=dev-cluster<br>      - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=kafka:19092<br>      - KAFKA_CLUSTERS_0_SCHEMAREGISTRY=http://kafka_schema_registry:8081<br>      - KAFKA_CLUSTERS_0_KAFKACONNECT_0_NAME=connect<br>      - KAFKA_CLUSTERS_0_KAFKACONNECT_0_ADDRESS=http://kafka_connect:8083<br>      - DYNAMIC_CONFIG_ENABLED=true<br>    networks:<br>      - kafka_network<br>      - docker_streaming</pre><ul><li>Access it at http://localhost:8888 (host port 8888 → container port 8080).</li><li>It’s wired to:<br>- Kafka broker via kafka:19092<br>- Schema Registry via <a href="http://kafka_schema_registry:8081/">http://kafka_schema_registry:8081</a><br>- Kafka Connect via <a href="http://kafka_connect:8083/">http://kafka_connect:8083</a><br>- Schema Registry via <a href="http://kafka_schema_registry:8081/">http://kafka_schema_registry:8081</a></li></ul><h4><strong>Networks: kafka_network vs docker_streaming</strong></h4><p>All Kafka services attach to <strong>two</strong> Docker bridge networks in this project:</p><pre>networks:<br>  kafka_network:<br>    driver: bridge<br>  docker_streaming:<br>    driver: bridge</pre><p>A <strong>Docker bridge network</strong> is Docker’s default container network type. Docker creates a Linux bridge on the host, assigns each container an IP on that bridge, and provides built-in DNS so containers can reach each other by service name (e.g., kafka, kafka_connect). Containers on the same bridge network can communicate directly over TCP/UDP; containers not on the same network cannot, unless they share another network or you explicitly connect them.</p><p><strong>Why two networks here?</strong><br> The practical effect is <strong>reachability grouping</strong>:</p><ul><li>Anything attached to <strong>kafka_network</strong> can reach Kafka services over that network (for example, Spark services, if they join kafka_network)</li><li>Anything attached to <strong>docker_streaming</strong> can also reach Kafka services (for example, Airflow services, if they join docker_streaming).</li><li>Because the Kafka services are attached to <strong>both</strong> networks, Kafka is reachable from both “sides” of the stack without forcing every container into one single network.</li></ul><p>At this point you have a direct mapping from Kafka concepts (broker/controller/quorum, consumer groups, serialization, topics) to the concrete Docker services and environment variables that run Kafka in this project.</p><p>Now that Kafka is wired up at the Docker level, we can look at the actual producer service: kafka_streaming_service.py.</p><h3>5.1 Kafka Code</h3><p>Now that we’ve wired up Kafka at the Docker level, we can look at the actual producer service: kafka_streaming_service.py.</p><p>The core idea is simple:</p><p><strong><em>Take data from an external API → normalize it → serialize it to JSON bytes → send it into a Kafka topic for Spark to consume.</em></strong></p><p>To do that cleanly, the script is structured into four main stages:</p><ol><li><strong>Setup </strong>(Kafka client + topic creation)</li><li><strong>Ingest</strong> (HTTP call to the API)</li><li><strong>Transform </strong>(flatten and clean the payload)</li><li><strong>Publish </strong>(serialize and send events to Kafka in a loop)</li></ol><h4>Imports, Type Hints, and Dynamic Kafka Import</h4><p>At the top of the file, the code separates <strong>static type checking</strong> from <strong>runtime imports</strong>:</p><pre>if TYPE_CHECKING:  # Only for type-checkers.<br>    from confluent_kafka import Producer<br>    from confluent_kafka.admin import AdminClient, NewTopic<br>else:<br>    Producer = Any</pre><ul><li>TYPE_CHECKING is only true for static analyzers (e.g., mypy). That lets editors/type-checkers “see” the real confluent_kafka types.</li><li>At runtime, it avoids importing confluent_kafka at module import time and instead uses Producer = Any.</li></ul><p>That matters because <strong>Airflow repeatedly imports/parses DAG code</strong>; minimizing heavy/optional imports at import-time reduces failures and startup friction. (This is why the real import is deferred into a helper.)</p><p>The helper _require_confluent_kafka() does the runtime import and throws a clear error if the package is missing:</p><pre>def _require_confluent_kafka() -&gt; tuple[Any, Any, Any]:<br>    &quot;&quot;&quot;Import confluent_kafka and raise a clear error if missing.&quot;&quot;&quot;<br>    try:<br>        from confluent_kafka import Producer as _Producer<br>        from confluent_kafka.admin import AdminClient as _AdminClient, NewTopic as _NewTopic<br>    except ModuleNotFoundError as exc:<br>        raise ModuleNotFoundError(<br>            &quot;confluent_kafka is required. Install it where this script runs (e.g., `pip install confluent-kafka`).&quot;<br>        ) from exc<br>    return _Producer, _AdminClient, _NewTopic</pre><p>Just mentally read this function as:</p><p>“Give me the real Producer, AdminClient, NewTopic now, or fail loudly with a useful error.”</p><p>We then define a few constants:</p><pre>KAFKA_BOOTSTRAP = &quot;kafka:19092&quot;<br>API_ENDPOINT = &quot;https://randomuser.me/api/?results=1&quot;<br>KAFKA_TOPIC = &quot;names_topic&quot;<br>PAUSE_INTERVAL = 10<br>STREAMING_DURATION = 120</pre><ul><li>KAFKA_BOOTSTRAP is the address of the Kafka broker <strong>inside</strong> the Docker network (kafka:19092).</li><li>API_ENDPOINT is the external API.</li><li>KAFKA_TOPIC is the topic we’ll publish user events to.</li><li>PAUSE_INTERVAL and STREAMING_DURATION control how often we send events and for how long.</li></ul><h4>Stage 1 — Setup (Ensure Kafka Topic Exists)</h4><pre>def ensure_topic(<br>    admin_bootstrap: str,<br>    topic: str,<br>    num_partitions: int = 1,<br>    replication_factor: int = 1,<br>) -&gt; None:<br>    &quot;&quot;&quot;Create the topic if missing; no-op if it already exists.&quot;&quot;&quot;<br>    _, AdminClient, NewTopic = _require_confluent_kafka()<br>    admin = AdminClient({&quot;bootstrap.servers&quot;: admin_bootstrap})<br><br>    md = admin.list_topics(timeout=5)<br>    if topic in md.topics:<br>        return<br><br>    futures = admin.create_topics(<br>        [NewTopic(topic, num_partitions=num_partitions, replication_factor=replication_factor)]<br>    )<br>    for t, f in futures.items():<br>        try:<br>            f.result()<br>            print(f&quot;Created topic: {t}&quot;)<br>        except Exception as e:<br>            print(f&quot;Topic create warning for {t}: {e}&quot;)</pre><p>What this does, in plain terms:</p><ul><li>Connects to Kafka’s <strong>Admin API</strong> using AdminClient.</li><li>Pulls topic metadata and checks whether topic already exists.</li><li>If it does not exist, it creates the topic with the requested partition count and replication factor.</li></ul><p>One detail worth repeating because it matters: replication_factor=1 here is <strong>not a design recommendation</strong>, it is a constraint of the dev setup. A single-broker cluster cannot replicate partitions across multiple brokers, so there is <strong>no redundancy</strong> in this environment. In production, you would normally run multiple brokers and set a higher replication factor.</p><p>This function is called from the main streaming loop. You can mentally read it as:</p><blockquote><em>“Before streaming starts, create the Kafka topic if it doesn’t already exist.”</em></blockquote><p>That guarantees the producer can start cleanly without any manual topic-creation step.</p><h4>Stage 2 — Ingest (HTTP Call to API)</h4><pre>def retrieve_user_data(<br>    url: str = API_ENDPOINT,<br>    retries: int = 3,<br>    timeout: int = 10,<br>) -&gt; Dict[str, Any]:<br>    &quot;&quot;&quot;GET one random user; retry a few times; fall back to synthetic on repeated failure.&quot;&quot;&quot;<br>    last_exception: Exception | None = None<br>    for attempt in range(1, retries + 1):<br>        try:<br>            resp = requests.get(url, timeout=timeout)<br>            resp.raise_for_status()<br>            data = resp.json()<br>            return data[&quot;results&quot;][0]<br>        except Exception as exc:<br>            last_exception = exc<br>            if attempt == retries:<br>                LOGGER.warning(<br>                    &quot;Failed to reach %s after %s attempts; using fallback.&quot;,<br>                    url,<br>                    retries,<br>                    exc_info=exc,<br>                )<br>                break<br>            time.sleep(1)<br><br>    if last_exception is not None:<br>        return _fallback_user_data()<br>    raise RuntimeError(&quot;retrieve_user_data exited without data or exception&quot;)</pre><p>This function:</p><ul><li>Calls the randomuser.me API</li><li>Returns a single user object from data[“results”][0].</li><li>Retries up to retries times on failure.</li><li>Logs a warning and falls back to _fallback_user_data() if it cannot reach the API (so the stream keeps running even if the API is flaky).</li></ul><p>This is your raw ingestion step: <strong>HTTP → JSON → Python dict</strong>.</p><h4>Stage 3 — Transform (Normalize API Payload)</h4><p>We define a two helpers:</p><pre>def safe_float(x: Any) -&gt; Optional[float]:<br>    &quot;&quot;&quot;float(x) or None on failure.&quot;&quot;&quot;<br>    try:<br>        return float(x)<br>    except Exception:<br>        return None<br><br><br>def encrypt_zip(zip_code: Any) -&gt; int:<br>    &quot;&quot;&quot;Hash postcode → int (deterministic pseudonym).&quot;&quot;&quot;<br>    md5_hex = hashlib.md5(str(zip_code).encode(&quot;utf-8&quot;)).hexdigest()<br>    return int(md5_hex, 16)</pre><p>For stronger privacy, prefer HMAC-SHA256 with a secret.</p><ul><li>safe_float converts latitude/longitude to floats (or None if the cast fails).</li><li>encrypt_zip hashes the postcode into a large integer, so we do not expose raw postcodes directly (pseudo-anonymisation). It is not encryption; at best it can be described as deterministic pseudonymisation. This is not production standard and in a real settings you should use a HMAC-SHA256 with a secret.</li></ul><p>Then the main transformation:</p><pre>def transform_user_data(d: Dict[str, Any]) -&gt; Dict[str, Any]:<br>    &quot;&quot;&quot;Flatten/normalize API payload; cast coords; hash postcode.&quot;&quot;&quot;<br>    title = d[&quot;name&quot;][&quot;title&quot;]<br>    first = d[&quot;name&quot;][&quot;first&quot;]<br>    last = d[&quot;name&quot;][&quot;last&quot;]<br>    name = f&quot;{title}. {first} {last}&quot;<br><br>    address = f&quot;{d[&#39;location&#39;][&#39;street&#39;][&#39;number&#39;]}, {d[&#39;location&#39;][&#39;street&#39;][&#39;name&#39;]}&quot;<br>    city = d[&quot;location&quot;][&quot;city&quot;]<br>    nation = d[&quot;location&quot;][&quot;country&quot;]<br>    postcode_hashed = str(encrypt_zip(d[&quot;location&quot;][&quot;postcode&quot;]))<br>    lat = safe_float(d[&quot;location&quot;][&quot;coordinates&quot;][&quot;latitude&quot;])<br>    lon = safe_float(d[&quot;location&quot;][&quot;coordinates&quot;][&quot;longitude&quot;])<br>    email = d.get(&quot;email&quot;)<br><br>    return {<br>        &quot;name&quot;: name,<br>        &quot;gender&quot;: d.get(&quot;gender&quot;),<br>        &quot;address&quot;: address,<br>        &quot;city&quot;: city,<br>        &quot;nation&quot;: nation,<br>        &quot;zip&quot;: postcode_hashed,<br>        &quot;latitude&quot;: lat,<br>        &quot;longitude&quot;: lon,<br>        &quot;email&quot;: email,<br>    }</pre><p>This step:</p><ul><li>Builds a clean name field from title, first, and last.</li><li>Flattens nested address fields into address, city, nation, zip.</li><li>Hashes the postcode for privacy.</li><li>Casts coordinates to floats</li><li>Returns a normalised dictionary with a stable schema — the exact schema Spark will later parse from Kafka.</li></ul><p>At this point we have clean, structured events: a consistent JSON shape that is safe to store and process. And now we can publish them into Kafka.</p><h4>Stage 4 — Publish (Build Producer and Send Events)</h4><p>At this point we have a clean Python dict coming out of transform_user_data(...). Kafka does not store Python objects, so the final step is:</p><p><strong>dict → JSON string → UTF-8 bytes → Kafka message</strong></p><p><strong>1) Build a Kafka producer</strong></p><pre>def build_producer(bootstrap: str) -&gt; Producer:<br>    &quot;&quot;&quot;Build a Producer with idempotence, acks=all, light batching, and compression.&quot;&quot;&quot;<br>    conf = {<br>        &quot;bootstrap.servers&quot;: bootstrap,<br>        &quot;client.id&quot;: &quot;producer_instance&quot;,<br>        &quot;acks&quot;: &quot;all&quot;,<br>        &quot;enable.idempotence&quot;: True,<br>        &quot;compression.type&quot;: &quot;gzip&quot;,<br>        &quot;linger.ms&quot;: 20,<br>        &quot;batch.size&quot;: 64_000,<br>    }<br>    ProducerCls, _, _ = _require_confluent_kafka()<br>    return ProducerCls(conf)</pre><p>What’s happening here:</p><ul><li>_require_confluent_kafka() returns the real confluent_kafka.Producer class at runtime (stored as ProducerCls).</li><li>We instantiate it with a configuration that is reasonable for a small demo stream.</li></ul><p>Config highlights (what each setting actually means):</p><ul><li>bootstrap.servers: where the producer connects (inside Docker this is typically something like kafka:19092).</li><li>acks=&quot;all&quot;: the broker will only acknowledge the write once all in-sync replicas have confirmed it. In a <strong>single-broker dev cluster</strong>, this still works, but there is no replica redundancy.</li><li>enable.idempotence=True: makes producer retries safer by preventing duplicates caused by retrying the same send (producer-side deduplication).</li><li>compression.type=&quot;gzip&quot;: compresses message batches on the wire.</li><li>linger.ms and batch.size: allow small batching to improve throughput instead of sending every message immediately.</li></ul><p>Important wording fix for the blog:<br> acks=&quot;all&quot; + enable.idempotence=True reduces duplicates caused by <strong>producer retries</strong> and improves durability semantics, but it is not a blanket “no duplicates anywhere in the pipeline” guarantee. It’s a producer-side reliability feature.</p><h4>2) Delivery callback (optional but useful)</h4><pre>def delivery_status(err, msg):<br>    &quot;&quot;&quot;Per-message delivery report.&quot;&quot;&quot;<br>    if err is not None:<br>        print(&quot;Delivery failed:&quot;, err)<br>    else:<br>        print(f&quot;Delivered to {msg.topic()} [{msg.partition()}] @ offset {msg.offset()}&quot;)</pre><p>This callback runs when Kafka confirms delivery (or returns an error). In this demo, it’s purely observability: “did the broker accept the message?”</p><h4>3) Publish a single record</h4><pre>def publish_once(producer: Producer, topic: str, data: Dict[str, Any]) -&gt; None:<br>    &quot;&quot;&quot;Serialize dict → JSON bytes and enqueue to Kafka.&quot;&quot;&quot;<br>    value = json.dumps(data).encode(&quot;utf-8&quot;)<br>    LOGGER.info(&quot;Producing record to topic=%s payload=%s&quot;, topic, data)<br>    producer.produce(<br>        topic,<br>        value=value,<br>        on_delivery=delivery_status,<br>    )<br>    producer.poll(0)  # Drive I/O and callbacks.</pre><p>This is the moment the event actually becomes a Kafka message:</p><ul><li>json.dumps(data) converts the dict into a JSON string.</li><li>.encode(&quot;utf-8&quot;) converts that string into raw bytes.</li><li>producer.produce(...) queues the message to be sent to Kafka.</li><li>producer.poll(0) allows the client to perform I/O and trigger delivery callbacks.</li></ul><p>So the crucial mental model is:</p><blockquote><em>Kafka only receives bytes. In this pipeline, the bytes are UTF-8 encoded JSON payloads.</em></blockquote><p>Production note (keep it short):<br> In a production pipeline you would typically add explicit handling for common Kafka errors (buffer full, broker unavailable, etc.) and rely on flush() during shutdown to ensure all queued messages are delivered before exiting.</p><h4>Orchestrator — Putting It All Together</h4><p>Finally, the orchestrator ties all four stages into a streaming loop:</p><p><strong>ensure topic → build producer → loop(fetch → transform → publish) → flush</strong></p><pre>def initiate_stream() -&gt; None:<br>    &quot;&quot;&quot;Run: ensure topic → build producer → loop(fetch→transform→produce) → flush.&quot;&quot;&quot;<br>    LOGGER.info(<br>        &quot;Starting Kafka stream: bootstrap=%s topic=%s interval=%s duration=%s&quot;,<br>        KAFKA_BOOTSTRAP,<br>        KAFKA_TOPIC,<br>        PAUSE_INTERVAL,<br>        STREAMING_DURATION,<br>    )<br>    ensure_topic(KAFKA_BOOTSTRAP, KAFKA_TOPIC, num_partitions=1, replication_factor=1)<br>    producer = build_producer(KAFKA_BOOTSTRAP)<br><br>    iterations = STREAMING_DURATION // PAUSE_INTERVAL<br>    try:<br>        for i in range(iterations):<br>            raw = retrieve_user_data()<br>            payload = transform_user_data(raw)<br>            LOGGER.info(&quot;Iteration %s/%s – sending payload %s&quot;, i + 1, iterations, payload)<br>            publish_once(producer, KAFKA_TOPIC, payload)<br>            time.sleep(PAUSE_INTERVAL)<br>    finally:<br>        remaining = producer.flush(10)<br>        if remaining:<br>            LOGGER.warning(&quot;Flush timed out with %s message(s) still in queue.&quot;, remaining)<br>        else:<br>            LOGGER.info(&quot;Kafka producer flushed all pending messages.&quot;)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    initiate_stream()</pre><h4>What the loop is doing (in order)</h4><ul><li><strong>Ensure topic exists</strong> (ensure_topic): creates the topic if missing using Kafka’s Admin API.</li><li><strong>Build a producer</strong> (build_producer): instantiates a confluent_kafka.Producer configured for reliable sending.</li><li><strong>Compute iterations</strong>: STREAMING_DURATION // PAUSE_INTERVAL controls how many events are emitted in total.</li><li><strong>For each iteration<br>- </strong>Call the API (retrieve_user_data)<br>- Normalize into a stable schema (transform_user_data)<br>- Serialize + enqueue to Kafka (publish_once)<br>- Sleep for PAUSE_INTERVAL seconds to control pacing</li><li><strong>Shutdown safety</strong>: flush() attempts to deliver any buffered messages before exiting.</li></ul><p>From Kafka’s perspective, this is a small, predictable stream: a steady trickle of UTF-8 encoded JSON events written into one topic over time. Spark Structured Streaming then attaches to that same topic and processes the events downstream.</p><h3>6. Spark</h3><p>We now shift to the Spark layer of the stack.</p><p>Kafka handles event transport; Spark handles distributed compute and streaming. Kafka is the <em>pipe</em> that moves events around. Spark is the <em>engine</em> that reads those events, transforms them, and writes the result somewhere useful (in our case, S3-compatible storage).</p><p>Spark consumes data from Kafka using <strong>Structured Streaming</strong>, which provides a continuous execution model built on Spark’s DataFrame APIs. This allows us to process Kafka events incrementally using the same declarative transformations as batch jobs, while tracking progress and recovery via checkpoints.</p><p>To understand how Spark runs inside our Docker setup, we need to introduce three components:</p><ul><li><strong>Master</strong></li><li><strong>Workers</strong></li><li><strong>Streaming Driver </strong>(spark-submit container)</li></ul><h4>Standalone Architecture Overview</h4><p>The standalone architecture is simple:</p><ul><li>one <strong>master</strong> coordinates the cluster</li><li><strong>workers</strong> provide compute resources</li><li><strong>jobs</strong> (submitted by a <strong>driver</strong>) request <strong>executors</strong> from the master to run distributed tasks</li></ul><p><strong>Structured Streaming in One Sentence</strong><br> Spark’s Structured Streaming lets you treat live data as an unbounded table (a continuously growing DataFrame). You define transformations once (as if you were writing batch or SQL code), and Spark incrementally applies them to new data as it arrives, maintaining state and fault tolerance via checkpoints.</p><h4>Master, Workers, and the Driver</h4><p><strong>Master</strong><br> The Spark master is the cluster manager. It:</p><ul><li>Accepts registrations from workers,</li><li>Tracks available cores and memory across the cluster,</li><li>Receives driver submissions,</li><li>Allocates executors to run tasks.</li></ul><p>It exposes two important endpoints:</p><ul><li><strong>RPC endpoint </strong>used by workers and drivers: spark://spark-master:7077</li><li><strong>Web UI</strong> at port 8080 (mapped to 8085 on the host)</li></ul><p><strong>Workers</strong><br> Workers are the execution layer. Each worker:</p><ul><li>Connects to the master,</li><li>Offers a configurable number of cores and memory,</li><li>Runs executor JVMs for user jobs,</li><li>Reports status to the master UI.</li></ul><p>Workers do not coordinate among themselves; all scheduling decisions come from the master.</p><p><strong>Streaming Driver</strong><br> The streaming driver is the entry point for our Structured Streaming job.</p><p>In Spark terminology:</p><ul><li>The <strong>driver program</strong> is the process that runs your application code (our PySpark script).</li><li>A <strong>job</strong> is the unit of work the driver submits to the cluster (e.g. “read this Kafka stream, transform it, write it to S3”).</li><li><strong>Executors</strong> (on the workers) execute tasks for those jobs in parallel.</li></ul><p>In this setup:</p><ul><li>The spark_streaming container executes spark-submit on startup</li><li>It connects to the master</li><li>It requests executors (which the master places on the workers</li><li>It begins consuming from Kafka using Structured Streaming</li></ul><p>A useful mental model is:</p><ul><li>the driver is the “brain” (it plans and coordinates)</li><li>the executors on workers are the “muscle” (they do the parallel work)</li><li>the master is the “dispatcher” (it decides where the muscle gets allocated)</li></ul><p>The driver holds the streaming query, checkpoint configuration, and control logic; the workers do the heavy lifting on each micro-batch.</p><h4>Spark Services in Our Docker Compose</h4><p>The Spark group in docker-compose consists of:</p><ol><li>spark-master — The standalone master</li><li>spark_worker_1 — Worker node</li><li>spark_worker_2 — Worker node</li><li>spark_streaming — Driver container running our Structured Streaming job</li></ol><p>All of these use the same custom Spark image, built from a dedicated Dockerfile.</p><h4><strong>Why a Custom Spark Image?</strong></h4><p>Out of the box, many Spark images:</p><ul><li>Do not ship with Python configured the way you want,</li><li>Do not include the Kafka Structured Streaming connector jars,</li><li>Do not contain your application code.</li></ul><p>If you try to read from Kafka without the right jars, you get errors such as:</p><pre>Failed to find data source: kafka<br>ClassNotFoundException: org.apache.spark.sql.kafka010.KafkaSourceProvider</pre><p>To avoid that, we build a custom Spark image that:</p><ul><li>Starts from a known base (bitnamilegacy/spark:3.5.1),</li><li>Installs python3 so we can run PySpark jobs,</li><li>Copies our streaming application into the image,</li><li>Downloads and pins the exact Kafka + Spark connector jars into Spark’s jars directory.</li></ul><p>That way, every container (spark-master, workers, and spark_streaming) is consistent and self-contained.</p><h4>Spark Dockerfile: Line-by-Line Breakdown</h4><pre>FROM bitnamilegacy/spark:3.5.1</pre><ul><li>Base image: Bitnami’s Spark 3.5.1 distribution.</li><li>Includes Spark binaries and a standard layout under /opt/bitnami/spark.</li></ul><pre>USER root</pre><ul><li>Switch to root so we can install system packages and modify the filesystem. Bitnami images typically run as non-root by default; this is an explicit escalation for the build phase only.</li></ul><pre>RUN set -eux; \<br>    apt-get update; \<br>    apt-get install -y --no-install-recommends \<br>        curl \<br>        python3 \<br>        python3-pip \<br>        python3-venv; \<br>    ln -s /usr/bin/python3 /usr/bin/python || true; \<br>    rm -rf /var/lib/apt/lists/*<br></pre><p>This block does all the system-level setup:</p><ul><li>set -eux<br>- -e: stop on error<br>- -u: error on unset variables<br>- -x: print each command (useful for debugging builds).</li><li>apt-get update<br>- Refresh package metadata.</li><li>apt-get install …<br>- Install curl (for jar downloads).<br>- Install python3, python3-pip, python3-venv so the container has a working Python runtime.</li><li>ln -s /usr/bin/python3 /usr/bin/python || true<br>- Create a python symlink pointing at python3.<br>- Many tools assume python exists; this avoids “command not found”.</li><li>rm -rf /var/lib/apt/lists/*<br>- Clean up package lists to keep the image smaller.</li></ul><pre>RUN mkdir -p /opt/spark/app<br>COPY app/ /opt/spark/app/</pre><ul><li>Creates a directory for your application code.</li><li>Copies everything from the local app/ directory into the image.</li><li>This is where spark_processing.py (your streaming job) lives.</li><li>Later, spark-submit uses /opt/spark/app/spark_processing.py as the entry point.</li></ul><pre>USER 1001</pre><p>We then drop privileges and run as user 1001. This matches Bitnami’s security model and is safer than running Spark as root. Everything after this runs as a non-root user.</p><pre>ARG KAFKA_CLIENT_VERSION=3.5.1<br>ARG SPARK_KAFKA_VERSION=3.5.1<br>ARG SCALA_BINARY_VERSION=2.12<br>ARG COMMONS_POOL_VERSION=2.12.0</pre><p>Build-time arguments to control dependency versions:</p><ul><li>KAFKA_CLIENT_VERSION — version of kafka-clients.</li><li>SPARK_KAFKA_VERSION — version of Spark’s Kafka connector modules.</li><li>SCALA_BINARY_VERSION — Scala ABI (must match Spark’s build, here 2.12).</li><li>COMMONS_POOL_VERSION — version of Apache Commons Pool used by the connector.</li></ul><p>This keeps versioning explicit and easy to change in one place.</p><p>The final part we add to the Spark Dockerfile is the <strong>Kafka + Spark connector JARs</strong>.</p><p>A <strong>JAR</strong> (Java ARchive) is a packaged bundle of Java/Scala classes and metadata. Spark is a JVM application, and features like <strong>Structured Streaming from Kafka</strong> are implemented as optional connector modules. If those connector JARs are not on Spark’s classpath, Spark cannot resolve the Kafka source, and you will see errors such as Failed to find data source: kafka or ClassNotFoundException: org.apache.spark.sql.kafka010.KafkaSourceProvider.</p><p>The JARs we are downloading are the Kafka–Spark connector libraries required for Structured Streaming, and they are added directly to Spark’s classpath.</p><pre>RUN set -eux; \<br>    cd /opt/bitnami/spark/jars; \<br>    curl -fSLo kafka-clients-${KAFKA_CLIENT_VERSION}.jar \<br>      https://repo1.maven.org/maven2/org/apache/kafka/kafka-clients/${KAFKA_CLIENT_VERSION}/kafka-clients-${KAFKA_CLIENT_VERSION}.jar; \<br>    curl -fSLo spark-sql-kafka-0-10_${SCALA_BINARY_VERSION}-${SPARK_KAFKA_VERSION}.jar \<br>      https://repo1.maven.org/maven2/org/apache/spark/spark-sql-kafka-0-10_${SCALA_BINARY_VERSION}/${SPARK_KAFKA_VERSION}/spark-sql-kafka-0-10_${SCALA_BINARY_VERSION}-${SPARK_KAFKA_VERSION}.jar; \<br>    curl -fSLo spark-token-provider-kafka-0-10_${SCALA_BINARY_VERSION}-${SPARK_KAFKA_VERSION}.jar \<br>      https://repo1.maven.org/maven2/org/apache/spark/spark-token-provider-kafka-0-10_${SCALA_BINARY_VERSION}/${SPARK_KAFKA_VERSION}/spark-token-provider-kafka-0-10_${SCALA_BINARY_VERSION}-${SPARK_KAFKA_VERSION}.jar; \<br>    curl -fSLo commons-pool2-${COMMONS_POOL_VERSION}.jar \<br>      https://repo1.maven.org/maven2/org/apache/commons/commons-pool2/${COMMONS_POOL_VERSION}/commons-pool2-${COMMONS_POOL_VERSION}.jar</pre><ul><li><strong>kafka-clients</strong> — the core Kafka client library used by the connector.</li><li><strong>spark-sql-kafka-0-10</strong> — the Spark SQL / Structured Streaming Kafka source implementation (this powers .format(&quot;kafka&quot;)).</li><li><strong>spark-token-provider-kafka-0-10</strong> — token provider support used in secured Kafka configurations (and prevents missing-class issues as configs evolve).</li><li><strong>commons-pool2</strong> — a common dependency used by the connector stack (missing it can cause runtime ClassNotFoundException).</li></ul><p>All of these JARs are placed in /opt/bitnami/spark/jars, which Spark loads automatically on startup. Because every Spark container (master, workers, and driver) uses the same custom-spark image, <strong>every Spark process can see the same connector JARs</strong>, making the image reusable across all Spark roles in the cluster.</p><p>Finally, Docker Compose builds this single image from context: ./spark (your Spark Dockerfile directory), and each Spark service references that same image.</p><h4>spark-master — The Cluster Manager</h4><p>We deploy Spark as a standalone cluster (one master, two workers). This is enough to demonstrate real cluster behaviour — tasks being scheduled across multiple executors — without introducing YARN, Kubernetes, or EMR. In larger deployments you would run the same streaming code against those managers; the core concepts stay the same.</p><pre>spark-master:<br>    image: &amp;spark-image custom-spark<br>    build: &amp;spark-build<br>      context: ./spark                 # directory containing Dockerfile above<br>      dockerfile: Dockerfile           # or whatever the filename actually is<br>    container_name: spark-master<br>    env_file:<br>      - .env.aws<br>    environment:<br>      - SPARK_MODE=master<br>      - SPARK_MASTER_HOST=spark-master<br>      - SPARK_MASTER_PORT=7077<br>      - SPARK_MASTER_WEBUI_PORT=8080<br>      # Make S3A use env credentials (ACCESS_KEY/SECRET/SESSION_TOKEN)<br>      - SPARK_SUBMIT_OPTS=-Dspark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.EnvironmentVariableCredentialsProvider<br>    ports:<br>      - &quot;8085:8080&quot;<br>    volumes:<br>      - ./:/home<br>      - spark_data:/opt/bitnami/spark/data<br>    networks:<br>      - docker_streaming<br>      - kafka_network<br>    healthcheck:<br>      test: [&quot;CMD-SHELL&quot;, &quot;curl -fsS http://localhost:8080 | grep -q &#39;Spark Master&#39;&quot;]<br>      interval: 10s<br>      timeout: 5s<br>      retries: 10</pre><p><strong>Key points:</strong></p><ul><li><strong>image: custom-spark</strong> — uses the custom Spark image built from the Dockerfile above, ensuring the master has the same Spark, Kafka, and Python dependencies as workers and the driver.</li><li><strong>SPARK_MODE=master</strong> — tells the Bitnami entrypoint script to launch the Spark <em>master</em> daemon instead of a worker.</li><li><strong>SPARK_MASTER_HOST=spark-master / </strong><strong>SPARK_MASTER_PORT=7077</strong> — defines the RPC endpoint (spark://spark-master:7077) that workers register with and drivers submit applications to.</li><li><strong>SPARK_MASTER_WEBUI_PORT=8080 (mapped to host </strong><strong>8085)</strong> — exposes the Spark Master UI so you can inspect registered workers, running applications, and executor allocation.</li><li><strong>SPARK_SUBMIT_OPTS=...EnvironmentVariableCredentialsProvider</strong> — configures Hadoop S3A to read AWS credentials from environment variables, keeping S3 access consistent across all Spark roles.</li><li><strong>spark_data:/opt/bitnami/spark/data</strong> — persists Spark state and logs across container restarts</li><li><strong>Healthcheck</strong> — waits for the Master UI to respond before dependent services start, reducing early startup race conditions.</li></ul><h4>spark_worker_1 &amp; spark_worker_2 — Worker Nodes</h4><p>Each worker uses an almost identical configuration. Remember each of these services define the actual compute capacity of the Spark cluster and advertise bounded reosurces that the master can allocate to executors.</p><pre>  spark_worker_1:<br>    image: *spark-image<br>    depends_on: [spark-master]<br>    env_file:<br>      - .env.aws<br>    environment:<br>      - SPARK_MODE=worker<br>      - SPARK_MASTER_URL=spark://spark-master:7077<br>      - SPARK_WORKER_CORES=2<br>      - SPARK_WORKER_MEMORY=4G<br>      - SPARK_WORKER_WEBUI_PORT=8081<br>      - SPARK_SUBMIT_OPTS=-Dspark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.EnvironmentVariableCredentialsProvider<br>    ports:<br>      - &quot;8086:8081&quot;<br>    networks:<br>      - docker_streaming<br>      - kafka_network</pre><p>Highlights:</p><ul><li><strong>image: custom-spark</strong> — uses the same image as the master so classpaths, Spark version, and Kafka/S3 dependencies are identical.</li><li><strong>SPARK_MODE=worker</strong> — tells the entrypoint script to start a Spark <em>worker</em> process.</li><li><strong>SPARK_MASTER_URL=spark://spark-master:7077</strong> — tells the worker where to register so it can receive executor assignments.</li><li><strong>SPARK_WORKER_CORES=2 / </strong><strong>SPARK_WORKER_MEMORY=4G</strong> — defines the maximum CPU and memory this worker can offer to executors; this effectively caps how much parallel work can run on the node.</li><li><strong>SPARK_WORKER_WEBUI_PORT=8081 (mapped to </strong><strong>8086 / </strong><strong>8087)</strong> — exposes the worker UI so you can see executor placement and task execution at the node level.</li><li><strong>SPARK_SUBMIT_OPTS=...EnvironmentVariableCredentialsProvider</strong> — ensures executors running on this worker can write to s3a:// using environment-provided AWS credentials.</li></ul><p>Two workers are enough here to demonstrate real multi-node scheduling and confirm that work is distributed across executors rather than collapsing onto a single node.</p><h4>spark_streaming — The Structured Streaming Driver</h4><p>This container runs the actual pipeline logic/ the application entrypoint. On startup it loads AWS/Kafka variables from .env.aws and executes the application entrypoint. It runs spark-submit, owns the Structured Streaming query and coordiantes work across the cluster.</p><pre>  spark_streaming:<br>    image: *spark-image<br>    depends_on:<br>      spark-master:<br>        condition: service_healthy<br>      spark_worker_1:<br>        condition: service_started<br>      spark_worker_2:<br>        condition: service_started<br>      kafka:<br>        condition: service_started<br>    env_file:<br>      - .env.aws<br>    environment:<br>      - S3_BUCKET=${S3_BUCKET}<br>      - S3_OUTPUT_PREFIX=${S3_OUTPUT_PREFIX:-names}<br>      - S3_CHECKPOINT_PREFIX=${S3_CHECKPOINT_PREFIX:-checkpoints/names}<br>      - S3_REGION=${S3_REGION:-eu-west-2}<br>      - S3_ENDPOINT=${S3_ENDPOINT:-}<br>      - S3_PATH_STYLE_ACCESS=${S3_PATH_STYLE_ACCESS:-}<br>      # Uncomment the next line when you want the container to reuse a named<br>      # AWS profile from the mounted ~/.aws directory (for example, AWS SSO).<br>      # - AWS_PROFILE=${AWS_PROFILE:-default}<br>      - KAFKA_BOOTSTRAP_SERVERS=kafka:19092<br>      - KAFKA_TOPIC=names_topic<br>      # Ensure S3A reads env credentials<br>      - SPARK_SUBMIT_OPTS=-Dspark.hadoop.fs.s3a.aws.credentials.provider=com.amazonaws.auth.EnvironmentVariableCredentialsProvider<br>    command:<br>      - bash<br>      - -lc<br>      - |<br>        /opt/bitnami/spark/bin/spark-submit \<br>          --master spark://spark-master:7077 \<br>          /opt/spark/app/spark_processing.py<br>    volumes:<br>      - ./spark/app:/opt/spark/app<br>      - spark_data:/opt/bitnami/spark/data<br>      # Bind-mount your host AWS configuration directory when you depend on<br>      # short-lived credentials generated by the AWS CLI (for example, SSO or<br>      # AssumeRole). Leave it commented when credentials are provided solely via<br>      # environment variables.<br>      # - ~/.aws:/root/.aws:ro<br>    restart: unless-stopped<br>    networks:<br>      - docker_streaming<br>      - kafka_network</pre><p><strong>Key Points:</strong></p><ul><li><strong>image: custom-spark</strong> — matches the master and workers so the driver runs with the same Spark and connector versions.</li><li><strong>depends_on with </strong><strong>service_healthy</strong> — ensures the Spark master is ready before submitting the streaming application.</li><li><strong>spark-submit --master spark://spark-master:7077</strong> — starts the driver process and submits the application to the standalone cluster.</li><li><strong>KAFKA_BOOTSTRAP_SERVERS=kafka:19092</strong> — points the streaming job at Kafka’s internal listener on the Docker network.</li><li><strong>KAFKA_TOPIC=names_topic</strong> — defines the Kafka topic consumed by the streaming query.</li><li><strong>S3_OUTPUT_PREFIX / </strong><strong>S3_CHECKPOINT_PREFIX</strong> — define where results and Structured Streaming checkpoints are written in s3a://, enabling recovery on restart.</li><li><strong>SPARK_SUBMIT_OPTS=...EnvironmentVariableCredentialsProvider</strong> — ensures both the driver and executors can authenticate to S3 consistently.</li><li><strong>./spark/app:/opt/spark/app</strong> — bind-mounts the application code so local edits are picked up without rebuilding the image.</li><li><strong>restart: unless-stopped</strong> — restarts the driver container if it exits; with a stable checkpoint path, the streaming query can resume from the last committed offsets.</li></ul><p>The container uses a restart policy; if the Spark job exits unexpectedly, the driver is relaunched. Because Spark uses the checkpoint directory, the streaming job resumes from the last committed offsets.</p><h4>Networking Strategy: Why Spark Joins Two Networks</h4><p>The Spark containers attach both docker_streaming and kafka_network. This is for differing purposes.</p><pre>networks:<br>      - docker_streaming<br>      - kafka_network</pre><ul><li><em>kafka_network </em>→ so Spark can consume from Kafka using the internal listener kafka:19092.</li><li><em>docker_streaming</em>→ so Airflow and other non-Kafka services can interact with Spark (UI, job submission, health checks).</li></ul><p>This fits the broader networking pattern in the stack:</p><ul><li>Kafka internal communication stays isolated on kafka_network.</li><li>Cross-service orchestration flows through docker_streaming.</li></ul><p>In practice:</p><ul><li>Spark Streaming can reliably reach Kafka.</li><li>Airflow can reliably reach Spark.</li><li>Services do not fight over conflicting advertised listeners or DNS names.</li></ul><h4>Spark, Hadoop, and s3a://</h4><p>Spark does not talk to S3 directly; it goes through the Hadoop filesystem layer.</p><p>In a simple Structured Streaming job, writing to S3 often looks like a direct filesystem write:</p><pre>(<br>    df<br>    .writeStream<br>    .format(&quot;parquet&quot;)<br>    .option(&quot;path&quot;, &quot;s3a://my-bucket/names/output/&quot;)<br>    .option(&quot;checkpointLocation&quot;, &quot;s3a://my-bucket/names/checkpoints/&quot;)<br>    .outputMode(&quot;append&quot;)<br>    .start()<br>)</pre><p>From Spark’s perspective, this is just a write to a filesystem path. Spark uses the same file APIs that Hadoop provides for HDFS and object stores. The s3a:// scheme is implemented by Hadoop’s AWS connector (bundled into the image). When the job writes to s3a://my-bucket/..., Spark delegates the operation to Hadoop’s S3A client, which handles authentication, multipart uploads, retries, and object creation using AWS credentials from the environment.</p><p>However, <strong>our pipeline deliberately abstracts this pattern using </strong><strong>foreachBatch</strong>. Instead of letting Spark write files directly at the writeStream level, we intercept each micro-batch and handle the output logic ourselves:</p><pre>query = (<br>    df.writeStream<br>    .outputMode(&quot;append&quot;)<br>    .foreachBatch(_write_batch)<br>    .option(&quot;checkpointLocation&quot;, checkpoint_location)<br>    .trigger(availableNow=True)   # Spark 3.3+ — drain backlog then stop<br>    .start()<br>)</pre><p>At first glance, this looks different because there is no explicit s3a:// path in the streaming definition. That difference is intentional.</p><p>What’s happening under the hood is:</p><ul><li>Spark still manages streaming state, offsets, and progress tracking via Hadoop using the checkpointLocation.</li><li>Each micro-batch is passed to _write_batch(df, batch_id)</li><li>Inside _write_batch, the batch DataFrame is written to s3a://... using standard Spark write APIs.</li></ul><p>The same file’s main() function wires the S3 paths from environment variables and passes them into the streaming sink, for example:</p><ul><li>path = f&quot;s3a://{bucket}/{output_prefix}&quot;</li><li>checkpoint_location = f&quot;s3a://{bucket}/{checkpoint_prefix}&quot;</li></ul><p>In other words, <strong>the S3 interaction is pushed one level down</strong>, but the mechanics are unchanged. Spark still delegates all S3 IO to Hadoop’s S3A filesystem, which performs the actual interaction with S3 using environment-provided AWS credentials.</p><p>This pattern is commonly used when you want:</p><ul><li>Custom partitioning or file layout logic</li><li>Idempotent writes or overwrite semantics per batch</li><li>Additional validation, logging, or side effects on each micro-batch</li></ul><p>Even though the S3 path is no longer visible at the writeStream level, the same underlying flow applies:</p><ul><li>Spark issues logical write requests</li><li>Hadoop S3A performs the actual S3 operations</li></ul><p>This is why the Docker image must include both:</p><ul><li><strong>Spark + Kafka JARs</strong> (for streaming ingestion from Kafka), and</li><li><strong>Hadoop AWS / S3A libraries</strong> (for writing results and checkpoints to S3-compatible storage).</li></ul><p>The result is a Spark job that can stream data from Kafka and write directly to an S3 bucket, while keeping storage concerns cleanly abstracted behind Hadoop’s filesystem layer.</p><p>The result is a Spark job that can stream from Kafka and write directly to an S3 bucket without any additional glue code.</p><h3>6.1 Spark Code</h3><p>Now that Spark is wired into the Docker cluster, we can look at the PySpark <strong>driver script</strong> we submit with spark-submit. This module defines a <strong>Structured Streaming</strong> query that reads from <strong>Kafka</strong> and writes output (and checkpoints) to <strong>S3-compatible object storage</strong>.</p><p>Before diving into the functions, it is worth stating the execution model explicitly once again:</p><p>Spark Structured Streaming treats each input source (like Kafka) as an <strong>unbounded table</strong>. Spark continuously appends new data to this table and processes it in <strong>micro-batches</strong>. Triggers (like availableNow) control how long the query runs and when it stops.</p><p>This project uses Spark 3.5.1, which supports trigger(availableNow=True) for “drain then stop” behaviour. On older Spark versions (≤ 3.2) you would use trigger(once=True) for a similar effect.</p><p>At a high level, this job does:</p><ol><li>Configure a SparkSession with S3A (AWS/MinIO) support.</li><li>Build an <strong>unbounded streaming DataFrame</strong> from Kafka.</li><li>Parse each Kafka JSON payload into typed columns.</li><li>Run an S3 health check by writing a small marker file.</li><li>Stream the transformed data into s3a://bucket/prefix using foreachBatch and availableNow.</li></ol><p>The code is organised as <strong>small, focused functions</strong> (Stage 1–5), plus a main() orchestrator that wires them together.</p><h4>Imports and Logging</h4><pre>import logging<br>import os<br>from typing import Optional, Tuple<br>from urllib.parse import urlparse<br>from datetime import datetime, timezone<br><br>from pyspark.sql import DataFrame, SparkSession<br>from pyspark.sql.functions import from_json, col<br>from pyspark.sql.types import StructType, StructField, StringType, DoubleType</pre><p>We import:</p><ul><li>Standard library tools: os, datetime, logging, urlparse, and typing helpers.</li><li>Spark core types: SparkSession, DataFrame.</li><li>Spark SQL pieces: from_json, col, and data types (StructType, StructField, etc.).</li></ul><p>Logging is configured up-front:</p><pre>logging.basicConfig(<br>    level=logging.INFO,<br>    format=&quot;%(asctime)s:%(name)s:%(funcName)s:%(levelname)s:%(message)s&quot;,<br>)<br>logger = logging.getLogger(&quot;spark_structured_streaming&quot;)</pre><p>This gives structured log lines with timestamp, logger name, function name, level, and message. In Docker/Airflow logs, that makes it substantially easier to trace what happened when.</p><h4><strong>Stage 1 — Spark + S3 Configuration</strong></h4><p>First order of business: build the SparkSession correctly so S3 works. That means we need to:</p><ul><li>Handle auth: feed AWS-style creds into S3A so Spark can actually sign requests.</li><li>Handle the endpoint: normalize whatever S3/MinIO URL you give it, plus SSL and path-style choices.</li><li>Build the SparkSession with all of that baked in.</li></ul><p>By the end of Stage 1, you’ve got the plumbing so everything downstream treats s3a://… paths like a normal file system—so the streaming sink writes straight to object storage, not some local disk.</p><p><strong>_configure_s3_credentials</strong></p><pre>def _configure_s3_credentials(<br>    builder: SparkSession.Builder,<br>    access_key: str,<br>    secret_key: str,<br>    session_token: Optional[str] = None,<br>) -&gt; SparkSession.Builder:<br>    ...</pre><p>This function wires S3 auth into the Spark builder so the SparkSession it creates can actually reach S3A. We start with a SparkSession.Builder—the factory that wires up all Spark parts (SQL engine, catalog, I/O connectors). Every read/write later goes through the SparkSession built here, so this is where we attach creds once and keep auth logic in one place.</p><p>It drops AWS-style creds onto the builder so Hadoop’s S3A client can sign requests to the bucket/endpoint you configure. Access key, secret key, and optional session token all flow in, giving Spark enough to authenticate no matter which S3-compatible host you point it at.</p><p>Key points:</p><ul><li>Provider choice depends on whether there’s a session token:</li><li>No token: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider</li><li>With token: org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider<br>That distinction matters because temporary creds must include the session token; picking the right provider avoids “worked locally, failed in prod” surprises.</li><li>Then it injects:<br>- spark.hadoop.fs.s3a.aws.credentials.provider<br>- spark.hadoop.fs.s3a.access.key <br>- spark.hadoop.fs.s3a.secret.key <br>- spark.hadoop.fs.s3a.session.token (only when used)</li></ul><p>The end result: a Spark builder that already knows how to authenticate to S3A, so the SparkSession it produces can immediately read/write s3a://… without extra configuration.</p><p><strong>_normalize_s3_endpoint</strong></p><p>Now that the builder is wired for auth, the next step is to prep the S3A endpoint we’ll hand to it.</p><pre>def _normalize_s3_endpoint(raw: Optional[str]) -&gt; Tuple[str, Optional[bool]]:<br>    ...</pre><p>This function cleans up whatever S3_ENDPOINT you pass in and turns it into two things: a host[:port] string and an ssl_enabled flag (True/False/None). Spark’s Hadoop layer (and Docker networking) want a bare host:port plus an explicit SSL toggle—not a full URL—so we normalize early to avoid confusion when the S3A client gets built.</p><p>Examples:</p><ul><li>s3.eu-west-2.amazonaws.com → (&quot;s3.eu-west-2.amazonaws.com&quot;, None)</li><li>http://minio:9000 → (&quot;minio:9000&quot;, False)</li><li>https://s3.eu-west-2.amazonaws.com → (&quot;s3.eu-west-2.amazonaws.com&quot;, True)</li></ul><p>It’s intentionally strict:</p><ul><li>No paths, queries, or fragments.</li><li>Only http and https schemes.</li><li>Empty hostnames are rejected.</li></ul><p>That strictness prevents endpoints like https://minio:9000/bucket from “kind of working” and then failing mid-job. It also makes SSL handling explicit: the function tells you whether to turn SSL on, so you’re not guessing later.</p><p><strong>initialize_spark_session (Stage 1 driver)<br></strong>Now that we’ve wired auth on the builder and cleaned the S3A endpoint, we wrap both steps in initialize_spark_session so the SparkSession comes out ready to read/write s3a:// with zero extra tweaks.</p><pre>def initialize_spark_session(<br>    app_name: str,<br>    *,<br>    region: str,<br>    access_key: Optional[str] = None,<br>    secret_key: Optional[str] = None,<br>    session_token: Optional[str] = None,<br>    endpoint: Optional[str] = None,<br>    path_style: Optional[bool] = None,<br>    ssl_enabled: Optional[bool] = None,<br>) -&gt; SparkSession:<br>    ...</pre><p>Configuration flow:</p><p><strong>Base builder</strong></p><pre>builder = SparkSession.builder.appName(app_name)</pre><p>What it does:</p><ol><li>Start with a base builder: SparkSession.builder.appName(app_name).</li><li>Credentials:<br>- If access_key and secret_key are provided → call _configure_s3_credentials(...).<br>- Otherwise use com.amazonaws.auth.DefaultAWSCredentialsProviderChain (IAM roles/env vars on “real” AWS or Docker env).</li></ol><p>3. Region: spark.hadoop.fs.s3a.region = region.</p><p>4. Endpoint: if provided, set spark.hadoop.fs.s3a.endpoint = endpoint.<br>- If path_style isn’t set, default it based on the endpoint (true for non-amazonaws.com like MinIO)</p><p>5. Path-style access: if path_style is set, configure spark.hadoop.fs.s3a.path.style.access accordingly (required for MinIO/other S3-compatible stores).</p><p>6. SSL: if ssl_enabled is set, configure spark.hadoop.fs.s3a.connection.ssl.enabled to &quot;true&quot;/&quot;false&quot;.</p><p>7. Create session: builder.getOrCreate() and drop the Spark log level to ERROR. If anything blows up, it logs the stack trace and re-raises so Airflow/Docker sees a clear failure</p><p>The net effect: one call returns a SparkSession that already knows its credentials, region, endpoint, path-style, and SSL settings — so downstream stages can just read/write s3a://... without extra setup.</p><h4><strong>Stage 2 — Kafka Streaming Source</strong></h4><p>We’ve set up the Spark builder and S3 credentials; now, assume the API has already pushed data into Kafka. Before we ship anything to S3, we need to make sure Spark can actually read and process that Kafka stream — so we circle back to wire up the Kafka source.</p><p><strong>get_streaming_dataframe</strong></p><pre>def get_streaming_dataframe(spark: SparkSession, brokers: str, topic: str) -&gt; DataFrame:<br>    &quot;&quot;&quot;Define a Kafka streaming DataFrame (unbounded table).&quot;&quot;&quot;<br>    return (<br>        spark.readStream<br>        .format(&quot;kafka&quot;)<br>        .option(&quot;kafka.bootstrap.servers&quot;, brokers)<br>        .option(&quot;subscribe&quot;, topic)<br>        .option(&quot;startingOffsets&quot;, &quot;earliest&quot;)<br>        .option(&quot;failOnDataLoss&quot;, &quot;false&quot;)<br>        .load()<br>    )</pre><p>Key points:</p><ul><li>.format(&quot;kafka&quot;) uses the Kafka connector jars baked into the image.</li><li>kafka.bootstrap.servers is the cluster address (e.g., kafka:19092 in Docker).</li><li>subscribe binds this stream to a single topic (here names_topic)</li><li>startingOffsets=&quot;earliest&quot; makes sure we process the backlog from the start.</li><li>failOnDataLoss=&quot;false&quot; keeps the job running if some offsets are missing (for example after topic retention, reset, or compaction). This is acceptable for a controlled demo pipeline, but in production it can hide real data-loss situations and should be handled explicitly. (So use with caution in production).</li></ul><p>The returned DataFrame is an <strong>unbounded streaming table</strong> with Kafka metadata columns (key, value, topic, partition, offset, timestamp, etc.), where value is raw bytes. Later stages will stop caring about Kafka internals and just work with structured rows.</p><h4><strong>Stage 3 — Transforming Kafka JSON Payloads</strong></h4><p>We’ve got the stream coming in from Kafka; now we need to turn those raw bytes into real columns before writing them anywhere. Stage 3 is where the raw Kafka events become strongly typed columns.<strong>transform_streaming_data</strong></p><pre>def transform_streaming_data(df: DataFrame) -&gt; DataFrame:<br>    &quot;&quot;&quot;Cast Kafka value→STRING, parse JSON to typed columns.&quot;&quot;&quot;<br>    schema = StructType(<br>        [<br>            StructField(&quot;name&quot;, StringType(), True),<br>            StructField(&quot;gender&quot;, StringType(), True),<br>            StructField(&quot;address&quot;, StringType(), True),<br>            StructField(&quot;city&quot;, StringType(), True),<br>            StructField(&quot;nation&quot;, StringType(), True),<br>            StructField(&quot;zip&quot;, StringType(), True),<br>            StructField(&quot;latitude&quot;, DoubleType(), True),<br>            StructField(&quot;longitude&quot;, DoubleType(), True),<br>            StructField(&quot;email&quot;, StringType(), True),<br>        ]<br>    )<br>    return (<br>        df.selectExpr(&quot;CAST(value AS STRING) AS json_str&quot;)<br>        .select(from_json(col(&quot;json_str&quot;), schema).alias(&quot;data&quot;))<br>        .select(&quot;data.*&quot;)<br>    )</pre><p>The producer sends JSON-encoded user records as bytes. This function:</p><ol><li>Defines the expected schema for each event (matching the producer).</li><li>Performs three transformations:</li></ol><ul><li>CAST(value AS STRING) → convert the Kafka value bytes into a UTF-8 string column json_str.</li><li>from_json(col(“json_str”), schema).alias(“data”) → parse JSON into a struct with the given schema.</li><li>.select(“data.*”) → flatten that struct so the DataFrame has top-level columns: name, gender, address, etc.</li></ul><p>Result: a <strong>typed streaming DataFrame</strong> that represents people as rows. At this point, the rest of the pipeline can be reasoned about like any other tabular Spark job.</p><h4><strong>Stage 4 — S3 Health Check</strong></h4><p>We’ve parsed the Kafka payloads and are almost ready to write, but first we need to prove S3 is reachable and writable. Stage 4 validates the S3A destination before the streaming sink starts, so we fail fast if creds, endpoint, or bucket setup are wrong. If the bucket doesn’t exist or auth is bad, this health check bails immediately instead of letting the streaming job crash later.</p><p><strong>s3_healthcheck_write</strong></p><pre>def s3_healthcheck_write(spark: SparkSession, path: str) -&gt; None:<br>    base = path.rstrip(&quot;/&quot;)                 # s3a://bucket/names<br>    health_dir = f&quot;{base}/healthcheck&quot;      # s3a://bucket/names/healthcheck<br>    marker = f&quot;{health_dir}/_SUCCESS&quot;       # s3a://bucket/names/healthcheck/_SUCCESS&quot;<br><br>    ts = datetime.now(timezone.utc).isoformat()<br>    logger.info(&quot;Running S3 healthcheck, writing marker to %s&quot;, marker)<br><br>    try:<br>        jvm = spark._jvm<br>        hadoop_conf = spark.sparkContext._jsc.hadoopConfiguration()<br>        Path = jvm.org.apache.hadoop.fs.Path<br><br>        health_dir_path = Path(health_dir)<br>        marker_path = Path(marker)<br><br>        # Get the correct FS implementation for the s3a:// path, not the default FS<br>        fs = marker_path.getFileSystem(hadoop_conf)<br><br>        # Ensure the &quot;directory&quot; prefix exists (harmless for S3A)<br>        fs.mkdirs(health_dir_path)<br><br>        # Create / overwrite the marker file and write a simple payload<br>        out_stream = fs.create(marker_path, True)  # True = overwrite if exists<br>        payload = f&quot;OK {ts}\n&quot;.encode(&quot;utf-8&quot;)<br>        out_stream.write(payload)<br>        out_stream.close()<br><br>        logger.info(&quot;S3 healthcheck marker successfully written at %s&quot;, marker)<br>    except Exception:<br>        logger.exception(&quot;S3 healthcheck FAILED while writing marker to %s&quot;, marker)<br>        raise</pre><p>marker_path.getFileSystem(hadoop_conf) makes sure Spark uses the S3A filesystem, not the local FS. If anything goes wrong (bad keys, bad endpoint, missing bucket), we log and re-raise—better to fail up front than discover it on the first micro-batch write.</p><h4><strong>Stage 5 — Streaming Sink to S3 with availableNow</strong></h4><p>We’ve got typed rows and a verified S3 target; now we need to drain Kafka once and write the backlog to S3. Stage 5 wraps the streaming write with a per-batch logger, checkpointing, and Spark’s availableNow trigger to make the run finite.</p><p><strong>initiate_streaming_to_bucket</strong></p><pre>def initiate_streaming_to_bucket(df: DataFrame, path: str, checkpoint_location: str) -&gt; None:<br>    &quot;&quot;&quot;Start JSON streaming sink with checkpointing and per-batch logging (finite run).&quot;&quot;&quot;<br>    ...</pre><p>Core pieces:</p><p><strong>Per-batch callback</strong></p><pre>def _write_batch(batch_df: DataFrame, batch_id: int) -&gt; None:<br>    count = batch_df.count()<br>    ts = datetime.now(timezone.utc).isoformat()<br><br>    logger.info(<br>        &quot;Batch %s: about to write %s record(s) to %s at %s&quot;,<br>        batch_id, count, path, ts,<br>     )<br><br>    (<br>        batch_df.write<br>        .mode(&quot;append&quot;)<br>        .format(&quot;json&quot;)<br>        .save(path)<br>    )<br><br>    logger.info(<br>        &quot;Batch %s: successfully wrote %s record(s) to %s at %s&quot;,<br>        batch_id, count, path, ts,<br>    )</pre><p>Each micro-batch gets counted, logged, and appended as JSON. You end up with very clear traces like “Batch 0 wrote N records to s3a://bucket/prefix at time T.”</p><p>2. <strong>Streaming query definition</strong></p><pre>query = (<br>    df.writeStream<br>    .outputMode(&quot;append&quot;)<br>    .foreachBatch(_write_batch)<br>    .option(&quot;checkpointLocation&quot;, checkpoint_location)<br>    .trigger(availableNow=True)   # Spark 3.3+ — drain backlog then stop<br>    .start()<br>)</pre><ul><li>outputMode(&quot;append&quot;) — matches the “new events only” pattern typical for Kafka.</li><li>foreachBatch— lets you treat each micro-batch like a normal batch DataFrame write.</li><li>checkpointLocation — stores offsets and progress in S3A. Because a checkpoint directory is set, Spark can recover the latest committed offsets if the container restarts. This prevents reprocessing already-committed batches.</li><li>trigger(availableNow=True) — for Spark ≥ 3.3 tells Spark to process all data currently available, commit everything, then stop the query. This makes the streaming job finite and easy to orchestrate from Airflow. A true continuous streaming service would omit availableNow so the query stays alive indefinitely.</li></ul><p>3. <strong>Termination</strong></p><pre>    logger.info(&quot;Streaming query (availableNow) started; waiting for backlog to be processed.&quot;)<br>    query.awaitTermination()<br>    logger.info(&quot;Streaming query (availableNow) completed and terminated.&quot;)</pre><p>So Stage 5 is a <strong>bounded</strong> Structured Streaming job: it drains the current Kafka backlog once, writes everything, and then exits. That behaviour is much easier to integrate with Airflow than an infinite streaming service.</p><h4><strong>Orchestrator — main()</strong></h4><p>main() stitches everything together: Stage 1 → 4 → 2 → 3 → 5 in a straight line.</p><pre>def main() -&gt; None:<br>    &quot;&quot;&quot;End-to-end: Spark init → Kafka read → JSON parse → JSON sink (AWS S3).&quot;&quot;&quot;<br>    app_name = &quot;spark_streaming&quot;<br><br>    brokers = os.environ.get(&quot;KAFKA_BOOTSTRAP_SERVERS&quot;, &quot;kafka:19092&quot;)<br>    topic = os.environ.get(&quot;KAFKA_TOPIC&quot;, &quot;names_topic&quot;)<br><br>    bucket = os.environ.get(&quot;S3_BUCKET&quot;)<br>    if not bucket:<br>        raise RuntimeError(&quot;S3_BUCKET is required&quot;)<br><br>    output_prefix = os.environ.get(&quot;S3_OUTPUT_PREFIX&quot;, &quot;names&quot;).strip(&quot;/&quot;)<br>    checkpoint_prefix = (<br>        os.environ.get(&quot;S3_CHECKPOINT_PREFIX&quot;) or f&quot;checkpoints/{output_prefix}&quot;<br>    ).strip(&quot;/&quot;)<br><br>    path = f&quot;s3a://{bucket}/{output_prefix}&quot;<br>    checkpoint_location = f&quot;s3a://{bucket}/{checkpoint_prefix}&quot;</pre><p>Block 1 — destinations: pick Kafka source, require S3_BUCKET, set output/checkpoint prefixes, build full S3A URIs.</p><pre>s3_region = os.environ.get(&quot;S3_REGION&quot;) or os.environ.get(&quot;AWS_REGION&quot;) or &quot;eu-west-2&quot;<br>access_key = os.environ.get(&quot;AWS_ACCESS_KEY_ID&quot;)<br>secret_key = os.environ.get(&quot;AWS_SECRET_ACCESS_KEY&quot;)<br>session_token = os.environ.get(&quot;AWS_SESSION_TOKEN&quot;)</pre><p><strong>Block 2 — S3 auth</strong></p><p>Pull region and credentials from the environment. In our local Docker usage these are usually static env vars we have already predefined; on real AWS this could be IAM roles.</p><pre>endpoint_host = None<br>ssl_pref = None<br>endpoint_raw = os.environ.get(&quot;S3_ENDPOINT&quot;)<br>if endpoint_raw:<br>    endpoint_host, ssl_pref = _normalize_s3_endpoint(endpoint_raw)<br><br>path_style_env = os.environ.get(&quot;S3_PATH_STYLE_ACCESS&quot;)<br>path_style = None<br>if path_style_env is not None:<br>    path_style = path_style_env.lower() in {&quot;1&quot;, &quot;true&quot;, &quot;yes&quot;, &quot;on&quot;}</pre><p>Block 3 — endpoint and path-style: normalize S3_ENDPOINT → endpoint_host + ssl_pref; optionally override path-style.</p><p>Then log the whole config in one line for easy debugging:</p><pre>logger.info(<br>    &quot;Spark streaming config: app_name=%s brokers=%s topic=%s bucket=%s &quot;<br>    &quot;output_prefix=%s checkpoint_prefix=%s region=%s endpoint=%s &quot;<br>    &quot;path_style=%s ssl_enabled=%s&quot;,<br>    app_name,<br>    brokers,<br>    topic,<br>    bucket,<br>    output_prefix,<br>    checkpoint_prefix,<br>    s3_region,<br>    endpoint_host,<br>    path_style,<br>    ssl_pref,<br>)</pre><p>Kick off Stage 1, then run Stage 4 → 2 → 3 → 5:</p><pre>spark = initialize_spark_session(<br>    app_name,<br>    region=s3_region,<br>    access_key=access_key,<br>    secret_key=secret_key,<br>    session_token=session_token,<br>    endpoint=endpoint_host,<br>    path_style=path_style,<br>    ssl_enabled=ssl_pref,<br>)<br><br>try:<br>    # Stage 4 – explicit S3 connectivity check<br>    s3_healthcheck_write(spark, path)<br><br>    # Stage 2 – Kafka streaming source<br>    df = get_streaming_dataframe(spark, brokers, topic)<br><br>    # Stage 3 – JSON parsing / struct→columns<br>    transformed_df = transform_streaming_data(df)<br><br>    # Stage 5 – drain backlog to S3 with checkpointing<br>    initiate_streaming_to_bucket(transformed_df, path, checkpoint_location)<br>finally:<br>    logger.info(&quot;Stopping Spark session.&quot;)<br>    spark.stop()<br># Entry point:<br></pre><p>Entry Point:</p><pre>if __name__ == &quot;__main__&quot;:<br>    main()</pre><p>spark-submit can run this file directly, and tests can import main() with mocked env vars and endpoints.</p><h4><strong>Summary: How This Code Fits into the Stack</strong></h4><p>Putting the Spark layer back into the broader pipeline:</p><ul><li>The Kafka producer keeps dropping clean JSON records into names_topic.</li><li>The Spark driver (this script) runs in the spark_streaming container, connects to spark-master, and fans work out to the workers as micro-batches.</li><li>Spark treats the Kafka topic as an unbounded table, parses each JSON record into typed columns, and — thanks to availableNow—drains the current backlog once and stops. Because we use a checkpoint directory, each run only processes data that arrived since the last run; re-running with the same checkpoint resumes where it left off instead of replaying everything.</li><li>The output lands as JSON in an S3-compatible bucket, with explicit S3A configuration, a pre-flight health check, checkpointing, and per-batch logging to show exactly what was written and when.</li></ul><h3>7. Airflow Architecture</h3><p>We now move to the orchestration layer: <strong>Apache Airflow</strong>. In this stack, Kafka moves messages, Spark does the computation, and Airflow is the control layer that decides <strong>when</strong> each piece runs, <strong>in what order</strong>, and <strong>what to do if something fails</strong>.</p><p>Airflow works by reading <strong>DAG (Directed Acyclic Graph)</strong> definitions written in Python. It scans the dags/ directory, imports those files, and registers each DAG and its tasks</p><p>Airflow in this Compose stack maps to four roles:</p><ul><li><strong>Metadata DB (PostgreSQL)</strong> stores state (runs, tasks, schedules, users, connections, etc.).</li><li><strong>Scheduler + Executor (LocalExecutor)</strong> decides what is runnable and actually runs tasks.</li><li><strong>Webserver</strong> is visibility + control (UI/API). It does <strong>not</strong> run tasks.</li><li><strong>Init job</strong> prepares the metadata DB (migrations + admin user) once, then exits.</li></ul><p>All of this runs on a single machine via Docker Compose, using <strong>LocalExecutor</strong>.</p><h4><strong>Core Concepts (what each component does)</strong></h4><p><strong>Scheduler</strong></p><p>The scheduler is the main loop. It:</p><ul><li>scans and parses DAG files from dags/</li><li>decides when tasks become runnable (schedule + dependencies + state)</li><li>records decisions in the metadata DB</li><li>hands runnable tasks to the executor</li></ul><p><strong>LocalExecutor in this setup:</strong> tasks run as subprocesses <strong>inside the scheduler container</strong>, not on a separate worker fleet. That means concurrency is bounded by the scheduler container’s CPU/memory and Airflow concurrency settings.</p><p><strong>Webserver (UI/API only)</strong></p><p>The webserver:</p><ul><li>serves the Airflow UI and REST API</li><li>reads all state from the metadata DB</li><li>shows DAGs, task status, logs, and configuration</li><li>lets you trigger DAGs, pause/unpause, clear tasks, etc.</li></ul><p>It <strong>does not execute tasks</strong>. Treat it as a “control panel” backed by Postgres.</p><p><strong>Metadata Database (PostgreSQL)</strong></p><p>Airflow is state-heavy. The metadata DB holds (among other things):</p><ul><li>DAG run history</li><li>task instance state (queued/running/success/failed)</li><li>schedules and next-run calculations</li><li>variables, connections, pools</li><li>users/roles</li><li>XComs (small cross-task messages)</li></ul><p>Everything else (webserver, scheduler, init job) is basically an app that reads/writes this state.</p><h4><strong>How Airflow Orchestrates “Streaming” Here</strong></h4><p>Airflow is fundamentally a <strong>batch scheduler:</strong> it runs DAGs at discrete times (cron-style), not as infinite daemons. Spark Structured Streaming, by contrast, is typically long-running.</p><p>To reconcile the two, this demo treats <strong>each DAG run as a short, finite “streaming slice”</strong>:</p><ul><li>Each Airflow run triggers a short-lived Spark job (via DockerOperator).</li><li>The Spark driver runs with trigger(availableNow=True), which drains the current Kafka backlog and then stops.</li><li>Because the job uses a checkpoint directory, subsequent runs resume from the last committed offsets and only process data that arrived after the previous run.</li></ul><p>If you schedule the DAG frequently (e.g., every minute), the system behaves like a near-real-time pipeline:</p><ul><li>Kafka continuously receives events.</li><li>Airflow wakes up on schedule, performs any health checks, then starts a finite Spark job that consumes the latest data and writes it into S3.</li></ul><p>This is how the pipeline approximates streaming behavior while still using Airflow’s batch-oriented execution model.</p><h4>Airflow Dockerfile: Why We Build a Custom Image</h4><p>All Airflow services run from the same custom Docker image, built on top of the official base image, because your DAG/task code imports extra libraries not included by default (e.g., Kafka clients, HTTP clients, etc.).</p><pre>FROM apache/airflow:2.9.3-python3.11<br><br>ARG AIRFLOW_VERSION=2.9.3<br>ARG PYTHON_MAJOR_MINOR=3.11<br>ARG AIRFLOW_CONSTRAINTS_URL=&quot;https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_MAJOR_MINOR}.txt&quot;<br><br>USER airflow<br>COPY --chown=airflow:airflow requirements.txt /tmp/requirements.txt<br>RUN pip install --no-cache-dir \<br>        --constraint &quot;${AIRFLOW_CONSTRAINTS_URL}&quot; \<br>        -r /tmp/requirements.txt \<br>    &amp;&amp; rm -f /tmp/requirements.txt</pre><h4>Line-by-Line Breakdown</h4><ul><li>FROM apache/airflow:2.9.3-python3.11 Start from an image that already contains Airflow + its default dependencies for a specific Python version.</li><li>ARG ... AIRFLOW_CONSTRAINTS_URL=... Airflow publishes constraints files that pin dependency versions known to work together for a given Airflow/Python version. Installing with constraints prevents pip from “helpfully” upgrading a core dependency into an incompatible state.</li><li>USER airflow + COPY --chown=airflow:airflow ...Install as the non-root airflow user, matching the official image’s security model.</li><li>pip install --constraint ... -r requirements.txt<br> Adds your project deps while staying inside the validated Airflow dependency set.</li></ul><p><strong>What problem this solves:</strong> without this image, a task that imports those libs will crash at runtime with ModuleNotFoundError. Also, a single shared image across webserver/scheduler/init prevents “works in one container but not another” drift.</p><h4>Service Breakdown</h4><p>From here, we go service-by-service and explain what each container is responsible for and why its Compose settings matter.</p><p>An important to note is all airflow services have common patterns:</p><ul><li>same custom Airflow image (webserver/scheduler/init)</li><li>shared mounts for ./dags, ./logs, ./plugins so code and logs are visible across containers</li><li>same Docker network so services can resolve each other by name (and reach Kafka/Spark/MinIO)</li><li>explicit depends_on with health conditions to avoid startup race conditions</li></ul><h4><strong>1. airflow_db — Metadata Database</strong></h4><pre>airflow_db:<br>  image: postgres:16.0<br>  environment:<br>    - POSTGRES_USER=${POSTGRES_USER}          # e.g. airflow<br>    - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}  # e.g. airflow<br>    - POSTGRES_DB=${POSTGRES_DB}              # e.g. airflow_db<br>  volumes:<br>    - ./airflow_pgdata:/var/lib/postgresql/data<br>  logging:<br>    options:<br>      max-size: 10m<br>      max-file: &quot;3&quot;<br>  healthcheck:<br>    test: [&quot;CMD-SHELL&quot;, &quot;pg_isready -U ${POSTGRES_USER} -d ${POSTGRES_DB}&quot;]<br>    interval: 10s<br>    timeout: 5s<br>    start_period: 90s<br>    retries: 10<br>  networks:<br>    - docker_streaming</pre><p><strong>Key points</strong></p><ul><li>POSTGRES_USER/POSTGRES_PASSWORD/POSTGRES_DB define DB credentials and the database name; Airflow uses these in its SQLAlchemy connection string.</li><li>Volume ./airflow_pgdata:/var/lib/postgresql/data persists Airflow state across restarts (history, runs, users, etc.).</li><li>Healthcheck uses pg_isready to confirm Postgres is accepting connections, preventing startup race conditions.</li><li>Network docker_streaming lets other containers reach it at airflow_db:5432.</li></ul><p><strong>2. airflow_init — One-Time Initialization</strong></p><p>Airflow requires DB migrations before the scheduler/webserver can safely run. airflow_init exists to do that once and exit:</p><ul><li>waits for Postgres</li><li>runs migrations (airflow db migrate)</li><li>creates an admin user (idempotently)</li><li>exits successfully</li></ul><pre> airflow-init:<br>    image: *airflow-image<br>    build: *airflow-build             # &lt;-- ONLY HERE<br>    depends_on:<br>      airflow_db:<br>        condition: service_healthy<br>    environment:<br>      - AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2://${POSTGRES_USER}:${POSTGRES_PASSWORD}@airflow_db:5432/${POSTGRES_DB}<br>      - AIRFLOW__CORE__FERNET_KEY=${AIRFLOW_FERNET_KEY}<br>      - AIRFLOW__CORE__DEFAULT_TIMEZONE=Europe/Berlin<br>      - AIRFLOW__CORE__LOAD_DEFAULT_CONNECTIONS=False<br>      - AIRFLOW__WEBSERVER__SECRET_KEY=${AIRFLOW_SECRET_KEY}<br>      - TZ=${TZ:-Europe/Berlin}<br>    user: &quot;${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-0}&quot;<br>    env_file:<br>    - .env.aws<br>    command:<br>      - bash<br>      - -lc<br>      - |<br>        set -euo pipefail<br>        airflow db migrate<br>        airflow users create \<br>          --role Admin \<br>          --username &quot;${AIRFLOW_ADMIN_USER:-admin}&quot; \<br>          --email &quot;${AIRFLOW_ADMIN_EMAIL:-admin@example.com}&quot; \<br>          --firstname &quot;${AIRFLOW_ADMIN_FN:-admin}&quot; \<br>          --lastname &quot;${AIRFLOW_ADMIN_LN:-admin}&quot; \<br>          --password &quot;${AIRFLOW_ADMIN_PASSWORD:-admin}&quot; || true<br>    restart: &quot;no&quot;<br>    volumes:<br>      - ./dags:/opt/airflow/dags<br>      - ./dags/producer:/opt/airflow/dags/producer<br>      - ./plugins:/opt/airflow/plugins<br>    networks:<br>      - docker_streaming</pre><p><strong>Key points</strong></p><ul><li>Runs exactly once; after successful completion it should not restart.</li><li>depends_on: service_healthy ensures migrations don’t run against an unready DB.</li><li>The build: appears here so Compose builds the custom image at least once; other Airflow services can reuse it.</li></ul><p><strong>3. airflow_webserver — Airflow UI/API</strong></p><p>The webserver is the HTTP front-end:</p><ul><li>serves the</li><li>provides the API</li><li>reads state from the metadata DB</li><li>triggers DAG runs (via UI/API)</li><li>does <strong>not</strong> run tasks</li></ul><pre>airflow_webserver:<br>    image: *airflow-image              # &lt;-- NO build HERE<br>    depends_on:<br>      airflow_db:<br>        condition: service_healthy<br>      airflow-init:<br>        condition: service_completed_successfully<br>    environment:<br>      - AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2://${POSTGRES_USER}:${POSTGRES_PASSWORD}@airflow_db:5432/${POSTGRES_DB}<br>      - AIRFLOW__CORE__EXECUTOR=LocalExecutor<br>      - AIRFLOW__CORE__LOAD_EXAMPLES=False<br>      - AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION=False<br>      - AIRFLOW__CORE__DEFAULT_TIMEZONE=Europe/Berlin<br>      - AIRFLOW__CORE__PARALLELISM=8<br>      - AIRFLOW__CORE__DAG_CONCURRENCY=8<br>      - AIRFLOW__CORE__MAX_ACTIVE_RUNS_PER_DAG=1<br>      - AIRFLOW__CORE__FERNET_KEY=${AIRFLOW_FERNET_KEY}<br>      - AIRFLOW__CORE__LOAD_DEFAULT_CONNECTIONS=False<br>      - AIRFLOW__WEBSERVER__SECRET_KEY=${AIRFLOW_SECRET_KEY}<br>      - AIRFLOW__WEBSERVER__WEB_SERVER_HOST=0.0.0.0<br>      - AIRFLOW__WEBSERVER__WEB_SERVER_PORT=8080<br>      - AIRFLOW_UID=${AIRFLOW_UID:-50000}<br>      - TZ=${TZ:-Europe/Berlin}<br>    env_file:<br>    - .env.aws<br>    user: &quot;${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-0}&quot;<br>    command: [&quot;bash&quot;,&quot;-lc&quot;,&quot;exec airflow webserver&quot;]<br>    healthcheck:<br>      test: [&quot;CMD-SHELL&quot;, &quot;curl -fsS http://localhost:8080/health | grep -q &#39;\&quot;status\&quot;:.*\&quot;healthy\&quot;&#39;&quot;]<br>      interval: 30s<br>      timeout: 10s<br>      retries: 5<br>    ports: [&quot;8080:8080&quot;]<br>    volumes:<br>      - ./dags:/opt/airflow/dags<br>      - ./dags/producer:/opt/airflow/dags/producer<br>      - ./logs:/opt/airflow/logs<br>      - ./plugins:/opt/airflow/plugins<br>      - /var/run/docker.sock:/var/run/docker.sock<br>    restart: unless-stopped<br>    networks:<br>      - docker_streaming</pre><p><strong>Key points</strong></p><ul><li>command: airflow webserver starts the UI + API process.</li><li>ports: 8080:8080 exposes the UI on the host.</li><li>AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG is the current config name for per-DAG task concurrency limits. <a href="https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html?utm_source=chatgpt.com">Apache Airflow+1</a></li><li>The /health endpoint returns a JSON object with component statuses (metadatabase, scheduler), so the healthcheck should validate those fields explicitly. <a href="https://airflow.apache.org/docs/apache-airflow/1.10.3/howto/check-health.html?utm_source=chatgpt.com">Apache Airflow+</a></li><li>The Docker socket mount is required if you use DockerOperator (the Airflow container must talk to the host Docker daemon to start other containers).</li></ul><p><strong>4. airflow_scheduler — Scheduler + LocalExecutor</strong></p><p>This is the execution engine:</p><ul><li>parses DAG files from ./dags</li><li>decides which tasks are runnabl</li><li>queues them in metadat</li><li>runs them via the configured executor</li></ul><p>With <strong>LocalExecutor</strong>, tasks run as subprocesses <em>in the scheduler container</em>. Consequences:</p><ul><li>concurrency is limited by scheduler container resources + Airflow settings</li><li>tasks run “where the scheduler is,” not on separate worker containers</li></ul><pre>airflow_scheduler:<br>  image: *airflow-image              # &lt;-- NO build HERE<br>  depends_on:<br>    airflow_db:<br>      condition: service_healthy<br>    airflow-init:<br>      condition: service_completed_successfully<br>  env_file:<br>  - .env.aws<br>  environment:<br>    - AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2://${POSTGRES_USER}:${POSTGRES_PASSWORD}@airflow_db:5432/${POSTGRES_DB}<br>    - AIRFLOW__CORE__EXECUTOR=LocalExecutor<br>    - AIRFLOW__CORE__LOAD_EXAMPLES=False<br>    - AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION=False<br>    - AIRFLOW__CORE__DEFAULT_TIMEZONE=Europe/Berlin<br>    - AIRFLOW__CORE__PARALLELISM=8<br>    - AIRFLOW__CORE__DAG_CONCURRENCY=8<br>    - AIRFLOW__CORE__MAX_ACTIVE_RUNS_PER_DAG=1<br>    - AIRFLOW__CORE__FERNET_KEY=${AIRFLOW_FERNET_KEY}<br>    - AIRFLOW__CORE__LOAD_DEFAULT_CONNECTIONS=False<br>    - AIRFLOW__WEBSERVER__SECRET_KEY=${AIRFLOW_SECRET_KEY}<br>    - AIRFLOW_UID=${AIRFLOW_UID:-50000}<br>    - TZ=${TZ:-Europe/Berlin}<br>  user: &quot;${AIRFLOW_UID:-50000}:${AIRFLOW_GID:-0}&quot;<br>  command: [&quot;bash&quot;,&quot;-lc&quot;,&quot;exec airflow scheduler&quot;]<br>  healthcheck:<br>    test: [&quot;CMD-SHELL&quot;, &quot;airflow jobs check --job-type SchedulerJob --hostname $(hostname) &gt;/dev/null 2&gt;&amp;1&quot;]<br>    interval: 30s<br>    timeout: 10s<br>    retries: 5<br>  volumes:<br>    - ./dags:/opt/airflow/dags<br>    - ./dags/producer:/opt/airflow/dags/producer<br>    - ./logs:/opt/airflow/logs<br>    - ./plugins:/opt/airflow/plugins<br>    - /var/run/docker.sock:/var/run/docker.sock<br>  restart: unless-stopped<br>  networks:<br>    - docker_streaming</pre><p><strong>Key points</strong></p><ul><li>command: airflow scheduler starts the scheduler daemon</li><li>LocalExecutor runs tasks as subprocesses inside this container (no separate worker containers).</li><li>No exposed ports: it doesn’t serve HTTP; it talks to the metadata DB and external systems (Kafka, Spark, S3, etc.).</li><li>Healthcheck uses airflow jobs check --job-type SchedulerJob --hostname &quot;$(hostname)&quot;, which is the documented pattern for verifying the scheduler is alive.</li></ul><p>The scheduler + LocalExecutor is what actually runs your Kafka producers, Kafka/S3 health checks, Spark triggers, and any other operators you define.</p><h4><strong>Integration with Kafka and Spark</strong></h4><p>This Airflow instance is not isolated; it orchestrates the rest of the stack.</p><p><strong>Kafka</strong></p><p>Because Airflow services live on the docker_streaming network, they can:</p><ul><li>Reach the Kafka broker via kafka:9092 (or kafka:19092 depending on listeners),</li><li>Run tasks that produce test messages, check topic existence, and verify readiness before starting Spark</li></ul><p>The custom Airflow image includes Kafka client libraries (e.g. confluent-kafka), so PythonOperator tasks can talk to Kafka directly from inside the Airflow containers.</p><p><strong>Spark</strong></p><p>Airflow also controls Spark:</p><ul><li>Triggers batch or short-lived streaming jobs,</li><li>Uses DockerOperator to start a driver container (for example, spark_stream_to_s3),</li><li>ptionally verifies outputs and triggers downstream tasks</li></ul><p>Because everything shares the docker_streaming network, Airflow can resolve Spark services by name and sequence the pipeline:</p><h4><strong>Network: docker_streaming</strong></h4><p>All Airflow, Kafka, and Spark services join the same user-defined network:</p><pre>networks:<br>  docker_streaming:<br>    driver: bridge</pre><p>This gives you:</p><ul><li>stable service-name DNS (airflow_webserver, airflow_scheduler, airflow_db, kafka, spark-master, etc.</li><li>isolation from unrelated containers on the hos</li><li>a single logical “data platform” network where orchestration, messaging, and compute can talk to each other without extra host-level port wrangling</li></ul><h3>7.1 Airflow — The DAG</h3><p>With Kafka, Spark, and Airflow wired up, the last piece is the DAG that drives the pipeline every 5 minutes (*/5 * * * *). The DAG itself stays thin: all the real work (producing names, reading from Kafka, writing to S3) lives in the producer (producer.kafka_streaming_service) and the Spark job (spark_processing.py).</p><blockquote>⚠️ S3 config note: In `dags/kafka_stream_dag.py`, the S3 settings are hard-coded to:<br>- Bucket: namegeneratorbucket<br>- Output prefix: names<br>- Checkpoint prefix: checkpoints/names<br>- Region: eu-west-2</blockquote><blockquote>You have two options:<br>(a) Keep your `.env`/`.env.aws` values identical to those literals so Airflow and Spark point to the same bucket/prefix/region, or<br>(b) Update the DAG to pull these from the environment (e.g., `bucket = os.getenv(“S3_BUCKET”, “namegeneratorbucket”)`, same for `S3_OUTPUT_PREFIX`, `S3_CHECKPOINT_PREFIX`, and `AWS_REGION`). This keeps Airflow aligned with whatever you set in `.env`.</blockquote><blockquote>If you skip this alignment, the S3 health check can fail or Spark may write to a different path than the DAG expects.</blockquote><p>Airflow’s job is just to orchestrate and guard the pipeline with four control tasks:</p><ol><li>Check Kafka and the target topic.</li><li>Check the S3 bucket (create it if necessary).</li><li>Run the Kafka producer.</li><li>Launch the Spark streaming consumer as a separate container.</li></ol><p>So the control plane is simply: Kafka health check → S3 health check → Producer → Spark consumer.</p><pre><br><br>from datetime import timedelta<br>import pendulum<br><br>from airflow import DAG<br>from airflow.operators.python import PythonOperator  # for Option A (spark-submit inside Airflow)<br>from airflow.providers.docker.operators.docker import DockerOperator  # for Option B (Spark container)<br><br># Reuse the same Kafka configuration (bootstrap servers, topic name)<br># that we defined in the producer service in Stage 1.<br># These constants come from the Kafka service configuration defined earlier,<br># so the DAG reuses the same bootstrap/server settings.<br>from producer.kafka_streaming_service import ensure_topic, KAFKA_BOOTSTRAP, KAFKA_TOPIC<br>import os<br>import boto3<br>from botocore.exceptions import ClientError<br>from airflow.exceptions import AirflowException<br><br># -------------------------------------------------------------------------<br># Callables used by tasks<br># -------------------------------------------------------------------------<br><br>def _trigger_stream() -&gt; None:<br>    &quot;&quot;&quot;Import and run the Kafka streaming helper at task runtime.<br><br>    Importing inside the callable keeps the DAG importable even if optional<br>    runtime dependencies (e.g., ``confluent_kafka``) are missing when the<br>    scheduler parses the file.<br>    &quot;&quot;&quot;<br>    from producer.kafka_streaming_service import initiate_stream<br>    initiate_stream()<br><br>def _check_kafka() -&gt; None:<br>    &quot;&quot;&quot;Ensure Kafka broker is reachable and the topic exists.<br><br>    Uses the same KAFKA_BOOTSTRAP/KAFKA_TOPIC constants as the producer<br>    so configuration stays consistent end-to-end.<br>    &quot;&quot;&quot;<br>    # This will no-op if the topic already exists<br>    ensure_topic(KAFKA_BOOTSTRAP, KAFKA_TOPIC, num_partitions=1, replication_factor=1)<br><br>def _check_bucket() -&gt; None:<br>    &quot;&quot;&quot;Ensure AWS S3 bucket exists; create it if missing.<br><br>    - 404 / NoSuchBucket: create the bucket in our region.<br>    - 403 / AccessDenied: fail fast with clear message (name taken or no perms).<br>    &quot;&quot;&quot;<br>    bucket = &quot;namegeneratorbucket&quot;<br>    region = os.getenv(&quot;AWS_REGION&quot;, &quot;eu-west-2&quot;)<br><br>    s3 = boto3.client(&quot;s3&quot;, region_name=region)<br><br>    try:<br>        s3.head_bucket(Bucket=bucket)<br>        # If we get here, bucket is accessible and exists; health check passes.<br>        return<br>    except ClientError as e:<br>        code = e.response[&quot;Error&quot;][&quot;Code&quot;]<br><br>        # Bucket not found → we try to create it in our account.<br>        if code in (&quot;404&quot;, &quot;NoSuchBucket&quot;, &quot;NotFound&quot;):<br>            if region == &quot;us-east-1&quot;:<br>                s3.create_bucket(Bucket=bucket)<br>            else:<br>                s3.create_bucket(<br>                    Bucket=bucket,<br>                    CreateBucketConfiguration={&quot;LocationConstraint&quot;: region},<br>                )<br>            return<br><br>        # Access denied / forbidden: likely bucket name already taken or IAM blocked<br>        if code in (&quot;403&quot;, &quot;AccessDenied&quot;):<br>            raise AirflowException(<br>                f&quot;S3 bucket health check failed: cannot access bucket &#39;{bucket}&#39; &quot;<br>                f&quot;(HTTP {code}). The name may be taken by another account or your &quot;<br>                &quot;IAM principal lacks s3:HeadBucket/ListBucket permissions.&quot;<br>            ) from e<br><br>        # Any other error: re-raise<br>        raise<br><br># -------------------------------------------------------------------------<br># DAG default args and config<br># -------------------------------------------------------------------------<br><br>LOCAL_TZ = pendulum.timezone(&quot;Europe/London&quot;)<br>DAG_START_DATE = pendulum.datetime(2024, 1, 1, 0, 0, tz=LOCAL_TZ)<br><br>DEFAULT_ARGS = {<br>    &quot;owner&quot;: &quot;airflow&quot;,<br>    &quot;start_date&quot;: DAG_START_DATE,<br>    &quot;retries&quot;: 1,<br>    &quot;retry_delay&quot;: timedelta(seconds=5),<br>}<br><br># -------------------------------------------------------------------------<br># DAG DEFINITION<br># -------------------------------------------------------------------------<br><br>with DAG(<br>    dag_id=&quot;name_stream_dag&quot;,<br>    description=&quot;Stream random names to Kafka and land them in S3 via Spark&quot;,<br>    default_args=DEFAULT_ARGS,<br>    schedule=&quot;*/5 * * * *&quot;,<br>    catchup=False,<br>    max_active_runs=1,<br>    tags=[&quot;kafka&quot;, &quot;streaming&quot;, &quot;demo&quot;],<br>) as dag:<br><br>    # 1) Kafka health check<br>    kafka_health_check = PythonOperator(<br>        task_id=&quot;kafka_health_check&quot;,<br>        python_callable=_check_kafka,<br>    )<br><br>    # 2) S3 / MinIO bucket health check<br>    bucket_health_check = PythonOperator(<br>        task_id=&quot;s3_bucket_health_check&quot;,<br>        python_callable=_check_bucket,<br>    )<br><br>    # 3) Kafka producer (existing logic)<br>    kafka_stream_task = PythonOperator(<br>        task_id=&quot;stream_to_kafka_task&quot;,<br>        python_callable=_trigger_stream,<br>    )<br><br>    # 4) Spark consumer: choose ONE of the following implementations<br><br>    # -----------------------------------------------------------------<br>    #  DockerOperator (Spark runs in its own container)<br>    # -----------------------------------------------------------------<br>    spark_stream_task = DockerOperator(<br>        task_id=&quot;spark_stream_to_s3&quot;,<br>        image=&quot;custom-spark&quot;,<br>        command=(<br>            &quot;/opt/bitnami/spark/bin/spark-submit &quot;<br>            &quot;--master spark://spark-master:7077 &quot;<br>            &quot;/opt/spark/app/spark_processing.py&quot;<br>        ),<br>        network_mode=&quot;docker_streaming&quot;,<br>        mount_tmp_dir=False,   # &lt;-- important for docker-in-docker / remote engine<br>        environment={<br>            # Kafka<br>            &quot;KAFKA_BOOTSTRAP_SERVERS&quot;: &quot;kafka:19092&quot;,<br>            &quot;KAFKA_TOPIC&quot;: &quot;names_topic&quot;,<br><br>            # S3 / AWS – forward from Airflow’s own environment<br>            &quot;AWS_ACCESS_KEY_ID&quot;: os.environ.get(&quot;AWS_ACCESS_KEY_ID&quot;, &quot;&quot;),<br>            &quot;AWS_SECRET_ACCESS_KEY&quot;: os.environ.get(&quot;AWS_SECRET_ACCESS_KEY&quot;, &quot;&quot;),<br>            &quot;AWS_SESSION_TOKEN&quot;: os.environ.get(&quot;AWS_SESSION_TOKEN&quot;, &quot;&quot;),<br>            &quot;AWS_REGION&quot;: os.environ.get(&quot;AWS_REGION&quot;, &quot;eu-west-2&quot;),<br><br>            # App-specific config<br>            &quot;S3_BUCKET&quot;: &quot;namegeneratorbucket&quot;,<br>            &quot;S3_OUTPUT_PREFIX&quot;: &quot;names&quot;,<br>            &quot;S3_CHECKPOINT_PREFIX&quot;: &quot;checkpoints/names&quot;,<br>            &quot;S3_REGION&quot;: &quot;eu-west-2&quot;,<br>        },<br>        auto_remove=True,<br>        execution_timeout=timedelta(minutes=15),<br>    )<br><br>    # 5) Wiring: health checks → producer → Spark consumer<br>    kafka_health_check &gt;&gt; bucket_health_check &gt;&gt; kafka_stream_task &gt;&gt; spark_stream_task</pre><h4><strong>Four control tasks, not one monolith</strong></h4><p>The DAG is just guardrails. It forces a healthy pathway before Spark ever runs: check Kafka, check storage, produce, then consume.</p><ol><li>kafka_health_check — verify messaging first</li></ol><ul><li>Calls ensure_topic(...) to reach the broker and ensure the topic exists (creates if missing, no-op if present).</li><li>Uses the same Kafka constants as the producer, keeping config in one place.</li><li>If Kafka is down, miswired, or auth-blocked, this task fails fast with a clear error. No healthy topic, no pipeline.</li></ul><p>2. s3_bucket_health_check — assert storage is available</p><ul><li>boto3.head_bucket on namegeneratorbucket in AWS_REGION (default eu-west-2).</li><li>If missing: create it in the correct region. If forbidden (403/AccessDenied): raise an explicit AirflowException (likely name taken or IAM blocked). Any other S3 error is re-raised.</li><li>Ensures a writable bucket before Spark runs; bootstraps on first run, fails early on naming/IAM conflicts.</li></ul><p>3. stream_to_kafka_task — run the producer</p><ul><li>Imports and calls initiate_stream at task runtime.</li><li>Late import keeps DAG parsing safe even if Kafka libs aren’t on the scheduler; missing deps show up as task failures (visible in UI), not parse crashes.</li><li>Sends a burst of events only after Kafka/S3 guardrails have passed.</li></ul><p>4. spark_stream_to_s3 — Spark consumer in a container</p><ul><li>DockerOperator launches custom-spark and runs spark-submit --master spark://spark-master:7077 /opt/spark/app/spark_processing.py.</li><li>On the docker_streaming network to reach kafka and spark-master.</li><li>Env injection for Kafka bootstrap/topic, AWS creds (from Airflow env), and S3 bucket/output/checkpoint/region.</li><li>mount_tmp_dir=False avoids temp-volume quirks, auto_remove=True cleans up the container, execution_timeout=15m bounds the run.</li><li>Delegates consumption to a disposable Spark driver that drains Kafka to S3 using the configs established upstream.</li></ul><h4><strong>Wiring: enforcing order in an automated pipeline</strong></h4><p>The dependency line:</p><pre>kafka_health_check &gt;&gt; bucket_health_check &gt;&gt; kafka_stream_task &gt;&gt; spark_stream_task</pre><ul><li><strong>Step 1: Validate Kafka —</strong> fail immediately if the broker/topic is unreachable.</li><li><strong>Step 2: Validate/prepare S3 —</strong> ensure a writable bucket exists before any data is produced.</li><li><strong>Step 3: Run producer — </strong>emit events only after messaging and storage are confirmed healthy.</li><li><strong>Step 4: Run Spark consumer —</strong> drain Kafka to S3 with checkpoints and logging, trusting that upstream guardrails have cleared the path.</li></ul><p>This linear dependency keeps the contract simple: don’t produce if Kafka is broken; don’t consume if storage is unavailable; don’t launch Spark until the checks have passed.</p><h3>8. Putting It Together/Actually Running the Pipeline</h3><p>Airflow drives the sequence: check Kafka, check S3, produce events, then hand off to Spark to drain Kafka and land results in S3.</p><ul><li>The name_stream_dag first checks Kafka and the target S3 bucket, then runs the Python producer to push random-name JSON into names_topic.</li><li>A Spark Structured Streaming job (launched via the spark_streaming helper container with the DockerOperator) reads names_topic from Kafka starting at the earliest offsets, tolerating missing offsets (failOnDataLoss=false).</li><li>It casts value to STRING, parses JSON into typed columns (name, gender, address, city, nation, zip, latitude, longitude, email).</li><li>Before streaming, it writes a small healthcheck marker to the target S3 prefix to verify connectivity</li><li>Each micro-batch is delivered to S3 via foreachBatch, using the standard batch writer (mode=append, format=json) to s3a://{S3_BUCKET}/{S3_OUTPUT_PREFIX}; checkpoints go to s3a://{S3_BUCKET}/{S3_CHECKPOINT_PREFIX} for recovery.</li><li>The query runs with availableNow=True, so it drains the backlog once and exits cleanly.</li><li>The custom Spark image already bundles the Kafka source and AWS/S3A connector JARs — no extra JAR provisioning needed.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4AqhfBLxOTcEEcC3A1jIsg.jpeg" /><figcaption>A successful end-to-end run pipeline run in Airflow would look like this</figcaption></figure><p><strong>Step-by-step (aligned to the code):</strong></p><ol><li><strong>Read from Kafka (streaming source)</strong></li></ol><pre>raw_df = (<br>    spark.readStream.format(&quot;kafka&quot;)<br>         .option(&quot;kafka.bootstrap.servers&quot;, &quot;kafka:19092&quot;)<br>         .option(&quot;subscribe&quot;, &quot;names_topic&quot;)<br>         .option(&quot;startingOffsets&quot;, &quot;earliest&quot;)<br>         .option(&quot;failOnDataLoss&quot;, &quot;false&quot;)<br>         .load()<br>)</pre><p><strong>2. Parse JSON into structured columns</strong></p><pre>parsed_df = (<br>    raw_df.selectExpr(&quot;CAST(value AS STRING) AS json_str&quot;)<br>          .select(from_json(&quot;json_str&quot;, schema).alias(&quot;data&quot;))<br>          .select(&quot;data.*&quot;)<br>)</pre><p><strong>3. Write to S3 via </strong><strong>foreachBatch (JSON sink + checkpoints)</strong></p><pre>query = (<br>    parsed_df.writeStream<br>        .outputMode(&quot;append&quot;)<br>        .foreachBatch(_write_batch)        # uses .write.format(&quot;json&quot;).mode(&quot;append&quot;).save(path)<br>        .option(&quot;checkpointLocation&quot;, checkpoint_location)<br>        .trigger(availableNow=True)        # drain backlog, then stop<br>        .start()<br>)</pre><ul><li>_write_batch logs batch size and writes to s3a://{bucket}/{output_prefix}.</li><li>checkpointLocation persists progress to s3a://{bucket}/{checkpoint_prefix} for fault tolerance.</li><li>A separate S3 healthcheck writes healthcheck/_SUCCESS under the target prefix before streaming starts.</li></ul><h3>9. End Results: What You Should See in S3</h3><p>To avoid turning this into an AWS account/IAM walkthrough, I will skip over how to set up AWS itself, IAM Identity Center, and so on. The assumption is:</p><ul><li>You already have AWS credentials configured for the Airflow stack.</li><li>AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION are set.</li><li>The DAG has been run at least once successfully.</li></ul><p>Given that, the end result of running name_stream_dag is:</p><ul><li><strong>JSON files</strong> written into your S3 bucket<br> namegeneratorbucket</li><li>Under the prefix:<br> names/ (as configured via S3_OUTPUT_PREFIX)</li></ul><p>A typical layout in the bucket will look roughly like:</p><ul><li>s3://namegeneratorbucket/names/… — data files written by the Spark job</li><li>s3://namegeneratorbucket/checkpoints/names/… — streaming checkpoints</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HiNQ0j78oukFO98Etc8xcA.jpeg" /><figcaption>JSON records landing in S3 under the names/ prefix after moving end-to-end through Kafka and Spark.</figcaption></figure><p>The exact filenames will depend on the Spark configuration (partitioning, number of files, etc.), but you should see a set of .json files arriving under names/ every time the DAG runs and the Spark step succeeds.</p><p>Each JSON record corresponds to one message that moved all the way through the pipeline:</p><ol><li>Produced into Kafka by stream_to_kafka_task.</li><li>Consumed by Spark from the Kafka topic.</li><li>Written out to S3 as JSON.</li></ol><p>At this point, the streaming loop is working end-to-end. The next step is to <strong>query</strong> those files without building a full data warehouse. That is where Athena comes in.</p><h3>10. Querying the Output with Athena</h3><h4><strong>What Athena is</strong></h4><p>Amazon Athena is a serverless, interactive query service that lets you run SQL directly against data in S3. You don’t manage servers; you just point Athena to S3, describe the schema, and run SQL. In this pipeline, Athena is the quick way to verify the JSON files Spark wrote and to run lightweight exploratory queries (counts, filters, samples) without moving data elsewhere.</p><h4><strong>High-level Athena setup</strong></h4><p>Assuming you are already in the Athena console and have configured a query result location in S3, the minimal steps are:</p><ol><li><strong>Create a database</strong> for this demo (once):</li></ol><pre>CREATE DATABASE IF NOT EXISTS demo_streaming;</pre><p><strong>2. Create an external table</strong> over the JSON files in namegeneratorbucket/names/.</p><p>Remember: the Spark job writes JSON lines into s3://namegeneratorbucket/names/ with this shape:</p><pre>{<br>  &quot;name&quot;: &quot;Dr. Ava Patel&quot;,<br>  &quot;gender&quot;: &quot;female&quot;,<br>  &quot;address&quot;: &quot;123 Main Street&quot;,<br>  &quot;city&quot;: &quot;London&quot;,<br>  &quot;nation&quot;: &quot;United Kingdom&quot;,<br>  &quot;zip&quot;: &quot;ab12cd34&quot;,<br>  &quot;latitude&quot;: 51.5074,<br>  &quot;longitude&quot;: -0.1278,<br>  &quot;email&quot;: &quot;ava.patel@example.com&quot;<br>}</pre><p>Create the table to match that schema:</p><pre>CREATE EXTERNAL TABLE IF NOT EXISTS demo_streaming.names_stream (<br>  name       string,<br>  gender     string,<br>  address    string,<br>  city       string,<br>  nation     string,<br>  zip        string,<br>  latitude   double,<br>  longitude  double,<br>  email      string<br>)<br>ROW FORMAT SERDE &#39;org.openx.data.jsonserde.JsonSerDe&#39;<br>LOCATION &#39;s3://namegeneratorbucket/names/&#39;;</pre><p>You can store Athena query results in the same bucket (or another of your choice) as configured in the console.</p><h4>Walking Through a Sample Athena Query</h4><p>Once the table exists, you can run simple queries to validate the end-to-end path.</p><ol><li><strong>Check that data arrived at all:</strong></li></ol><pre>SELECT COUNT(*) AS row_count<br>FROM demo_streaming.names_stream;</pre><p>A non-zero and growing count means the DAG is producing to Kafka, Spark is consuming, and data is landing in S3.</p><p><strong>2. Inspect a small sample of events:</strong></p><pre>SELECT<br>  name,<br>  gender,<br>  address,<br>  city,<br>  nation,<br>  zip,<br>  latitude,<br>  longitude,<br>  email<br>FROM demo_streaming.names_stream<br>LIMIT 10;</pre><p>Should show records matching the JSON schema above.</p><p><strong>3. Sanity-check by nulls/malformed fields (optional):</strong></p><pre>SELECT<br>  COUNT(*) AS total_rows,<br>  SUM(CASE WHEN name IS NULL THEN 1 ELSE 0 END) AS null_name_rows,<br>  SUM(CASE WHEN email IS NULL THEN 1 ELSE 0 END) AS null_email_rows<br>FROM demo_streaming.names_stream;</pre><p>Example Athena usage</p><p>To check that the first 10 records exist we could run the query below in Athena:</p><pre>SELECT * FROM streaming_demo.names_stream ORDER BY name LIMIT 10;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xRP402xOg75yrQ9HQLm69Q.jpeg" /><figcaption>SQL query executed in Athena to sample streamed records</figcaption></figure><p>Which would then produce this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ROPnGsNC2ONcvV24Hv4vrg.jpeg" /><figcaption>Sample results from SELECT * FROM demo_streaming.names_stream LIMIT 10 executed in Athena.</figcaption></figure><p>At this point you have:</p><ul><li>Airflow orchestrating the pipeline and acting as a control layer</li><li>Kafka and Spark handling streaming ingestion and processing.</li><li>S3 storing the results as JSON.</li><li>Athena providing a quick SQL surface to inspect and validate the output.</li></ul><p>That combination is enough to go from “local Docker stack” to something that looks and behaves like a small, realistic streaming data platform.</p><h3>11. Conclusion</h3><p>In this walkthrough, we built an end-to-end streaming pipeline with Docker, Airflow, Kafka, Spark Structured Streaming, and S3 (with Athena for ad‑hoc SQL). Docker Compose gave us a production-like stack locally; Airflow enforced the guardrails and scheduling; Kafka carried events; Spark drained and transformed them with checkpoints and health checks; and S3/Athena made the results easy to inspect. The pattern scales: keep components decoupled, use the right tool for each stage, and let orchestration glue it together.</p><p>If you’ve read this far, thanks for sticking with it and I hope you learnt something. This took far too long to make.</p><p>If you want to follow along, explore the code, or chat about building similar pipelines, you can reach me here:</p><ul><li>GitHub: <a href="https://github.com/thejasono">https://github.com/thejasono</a></li><li>LinkedIn: <a href="https://www.linkedin.com/in/thejasonowens/">https://www.linkedin.com/in/thejasonowens/</a></li><li>Email: thisisjasonowens@gmail.com<br>(available for freelance, contract, full-time roles, and consultation)</li></ul><p>#DataEngineering #Airflow #Kafka #Spark #S3 #Athena #Streaming #OpenSource</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c51950324d92" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An End-to-End Data Pipeline Using Docker, Airflow, dbt, and Postgres]]></title>
            <link>https://medium.com/@thisisjasonowens/an-end-to-end-data-pipeline-using-docker-airflow-dbt-and-postgres-34fbdef69cfd?source=rss-ecbef8df7ad4------2</link>
            <guid isPermaLink="false">https://medium.com/p/34fbdef69cfd</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[dbt]]></category>
            <dc:creator><![CDATA[Jason O.]]></dc:creator>
            <pubDate>Sat, 18 Oct 2025 04:55:46 GMT</pubDate>
            <atom:updated>2026-01-06T22:10:36.094Z</atom:updated>
            <content:encoded><![CDATA[<p>Welcome back! For those reading for the first time, this post is part of my Data Engineering Fundamentals series — where we walk through the key fundamentals every data practitioner should know. In the last article, we covered the five stages of the data journey, defined ETL, and walked through a simple example. Now that we’ve built that foundation, the next step is to actually build a fully functional data pipeline — the kind you’d find in production environments.</p><p><em>This is a longer read, so I’ve added clear section headings. Feel free to jump around to the sections you’re most curious about.</em></p><h4>View the Code on GitHub</h4><p>All the code, configs, and example files used in this walkthrough are available in the public GitHub repository:</p><p>👉<a href="https://github.com/thejasono/data-pipeline-airflow-api-postgres-dbt-powerbi"> <strong>View the full project on GitHub</strong></a></p><p>The repo includes:</p><ul><li>The full docker-compose.yml</li><li>All Dockerfiles for Airflow and the mock API</li><li>Airflow DAG code</li><li>dbt models, configs, and tests</li><li>Postgres initialization SQL</li><li>A sample Power BI file</li></ul><p>Clone it, run docker-compose up -d --build, and you’ll have the whole system running locally in minutes.</p><h4>The Workflow</h4><p>Here’s what our pipeline will do:</p><ul><li><strong>Extract:</strong> Pull live data from a mock API (pre-built for this example).</li><li><strong>Load:</strong> Store the raw data in a PostgreSQL database.</li><li><strong>Transform:</strong> Use dbt to clean and aggregate the data in the database.</li><li><strong>Visualize:</strong> Connect a BI tool (like Power BI) to the transformed data for analysis.</li><li><strong>Orchestrate:</strong> Automate everything on a schedule with Apache Airflow.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/911/1*Lmt04LYPzLFk--OsHiN0yA.jpeg" /><figcaption>API → Postgres → dbt → Power BI (under Airflow)</figcaption></figure><p>At first glance, this might look complex — five distinct tools, each with their own moving parts. But the real trick is how they connect: the glue that ties everything together is <strong>Docker</strong>.</p><h4>Why We Use Docker</h4><p>Running all these tools natively on your laptop is messy. Each one needs its own dependencies, versions, and setup steps. In production, they would live on separate servers or cloud instances, not all on your computer.</p><p>Docker solves this problem through <strong>containerization</strong>: packaging each tool (Postgres, Airflow, dbt, etc.) into its own self-contained unit called a <strong>container</strong>. A container is created from an image, which is basically a blueprint describing the software, version, and configuration needed to run.</p><p>This gives a few key advantages:</p><ul><li><strong>Isolation</strong> — Containers don’t interfere with each other’s dependencies.</li><li><strong>Reproducibility</strong> — You can rebuild the entire stack anywhere with one command. No more “works on my machine” issues.</li><li><strong>Portability</strong> — The setup will run the same on Windows, macOS, or Linux.</li></ul><p>In short, Docker lets you work like a production engineer without needing a data center in your living room.</p><h3>1. Docker Compose — The Master Config</h3><p>To coordinate multiple containers, we use <strong>Docker Compose</strong> with a configuration file called docker-compose.yaml. Think of this file as the <strong>blueprint</strong> for our entire pipeline. It tells Docker which services (containers) to spin up, what images to use, what ports to expose, which environment variables to inject, where to mount volumes, and how each service depends on the others.</p><p>Before jumping into the full config, it helps to look at a simplified pattern first. Try to think of a Docker service like this:</p><pre>services:<br>  my_service:<br>    image: some-image:latest<br>    container_name: my-container<br>    environment:<br>      KEY: value<br>    ports:<br>      - &quot;host_port:container_port&quot;<br>    volumes:<br>      - ./local_path:/container_path<br>    depends_on:<br>      another_service:<br>        condition: service_healthy</pre><p>It’s like a mini spec sheet. Each service defines how to build or run a container, how it connects to others, and what it needs to function. Once this pattern clicks, the full docker-compose.yml will feel a lot less intimidating.</p><h4><strong>The Actual Compose Config</strong></h4><p>Here’s the full Compose file we use for this project:</p><pre>services:<br>  postgres:<br>    image: postgres:16<br>    container_name: postgres<br>    environment:<br>      POSTGRES_USER: ${POSTGRES_USER}<br>      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}<br>      POSTGRES_DB: ${POSTGRES_DB}<br>    ports:<br>      - &quot;5432:5432&quot;<br>    volumes:<br>      - ./postgres/init:/docker-entrypoint-initdb.d<br>      - pgdata:/var/lib/postgresql/data<br>    healthcheck:<br>      test: [&quot;CMD-SHELL&quot;, &quot;pg_isready -U ${POSTGRES_USER} -d ${POSTGRES_DB}&quot;]<br>      interval: 5s<br>      timeout: 3s<br>      retries: 30<br><br>  mock-api:<br>    build: ./mock_api<br>    container_name: mock-api<br>    environment:<br>      API_KEY: ${API_KEY:?API_KEY must be set}<br>      API_RATE_LIMIT_PER_MIN: ${API_RATE_LIMIT_PER_MIN:-300}<br>      TZ: ${TZ:-UTC}<br>    ports:<br>      - &quot;8000:8000&quot;<br>    healthcheck:<br>      test: [&quot;CMD-SHELL&quot;, &quot;curl -fsS http://localhost:8000/health || exit 1&quot;]<br>      interval: 10s<br>      timeout: 3s<br>      retries: 5<br>    depends_on:<br>      postgres:<br>        condition: service_healthy<br><br>  airflow:<br>    build: ./airflow<br>    image: airflow-local<br>    container_name: airflow<br>    working_dir: /workspace<br>    environment:<br>      AIRFLOW__CORE__LOAD_EXAMPLES: &quot;false&quot;<br>      AIRFLOW__CORE__EXECUTOR: &quot;LocalExecutor&quot;<br>      AIRFLOW__WEBSERVER__EXPOSE_CONFIG: &quot;True&quot;<br>      AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: &gt;-<br>        postgresql+psycopg2://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}<br>      DBT_PROFILES_DIR: /opt/airflow/.dbt<br>      TZ: ${TZ:-UTC}<br>      AIRFLOW_CONN_POSTGRES_DEFAULT: &gt;-<br>        postgresql+psycopg2://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}<br>      API_BASE_URL: http://mock-api:8000<br>      API_KEY: ${API_KEY:?API_KEY must be set}<br>      POSTGRES_USER: ${POSTGRES_USER}<br>      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}<br>      POSTGRES_DB: ${POSTGRES_DB}<br>    entrypoint: [&quot;/bin/bash&quot;]<br>    command:<br>      - -lc<br>      - |<br>          set -euo pipefail<br>          airflow db init<br>          airflow users create --role Admin --username admin --password admin --firstname Admin --lastname User --email admin@example.com || true<br>          airflow connections delete postgres_default || true<br>          airflow connections add postgres_default --conn-uri &quot;postgresql+psycopg2://${POSTGRES_USER}:${POSTGRES_PASSWORD}@postgres:5432/${POSTGRES_DB}&quot;<br>          airflow webserver &amp;<br>          exec airflow scheduler<br>    ports:<br>      - &quot;8080:8080&quot;<br>    volumes:<br>      - ./airflow/dags:/opt/airflow/dags<br>      - ./airflow/logs:/opt/airflow/logs<br>      - ./airflow/plugins:/opt/airflow/plugins<br>      - ./dbt:/usr/app<br>      - ./dbt/profiles:/opt/airflow/.dbt<br>      - ./:/workspace<br>      - /var/run/docker.sock:/var/run/docker.sock<br>    depends_on:<br>      postgres:<br>        condition: service_healthy<br>      mock-api:<br>        condition: service_healthy<br><br>  dbt:<br>    image: ghcr.io/dbt-labs/dbt-postgres:1.8.latest<br>    container_name: dbt<br>    working_dir: /usr/app<br>    environment:<br>      DBT_PROFILES_DIR: /root/.dbt<br>      TZ: ${TZ:-UTC}<br>      POSTGRES_USER: ${POSTGRES_USER}<br>      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}<br>      POSTGRES_DB: ${POSTGRES_DB}<br>    volumes:<br>      - ./dbt:/usr/app<br>      - ./dbt/profiles:/root/.dbt<br>    depends_on:<br>      postgres:<br>        condition: service_healthy<br>    entrypoint: [&quot;/bin/bash&quot;, &quot;-lc&quot;, &quot;dbt deps &amp;&amp; dbt run &amp;&amp; tail -f /dev/null&quot;]<br><br>volumes:<br>  pgdata: {}</pre><p>Looks intimidating? Let’s simplify it.</p><h4>How to Read a Service Definition</h4><p>Each section under services: defines one containerized component. Here’s the breakdown of the pattern:</p><ul><li><strong>Service Name</strong> — The name of the service, used as the hostname on Docker’s internal network. Other containers refer to this name (e.g. postgres) when connecting. It’s also used under depends_on to manage startup order.</li><li><strong>container_name</strong> — An explicit name for the container (what you’ll see with docker ps). If omitted, Docker Compose will generate a name like myproject_postgres_1. This is mostly for convenience.</li><li><strong>image</strong> vs <strong>build</strong> — You can either pull a pre-built image (postgres:16) or build from a Dockerfile. Here we use image for the Postgres and dbt containers, and build (pointing to a folder) for custom ones like airflow and mock_api.</li><li><strong>environment</strong> — Key-value environment variables passed into the container. We reference variables like ${POSTGRES_USER} which are set in our .env file. This keeps secrets and config out of the Compose file. For example:</li></ul><pre> POSTGRES_USER: myuser<br> POSTGRES_PASSWORD: mypassword<br> POSTGRES_DB: mydb<br> API_KEY=&lt;&lt;your_api_key_here&gt;&gt;</pre><ul><li><strong>ports</strong> — Maps container ports to your local machine. For instance, “5432:5432” means Postgres is reachable at localhost:5432 on the host.</li><li><strong>volumes</strong> — Mounts directories or persistent volumes. For example, ./postgres/init:/docker-entrypoint-initdb.d means any SQL files in our local postgres/init folder will run when the Postgres container starts. pgdata:/var/lib/postgresql/data is a named volume that stores the database data, so it persists between container restarts.</li><li><strong>healthcheck</strong> — A command to test when a service is ready. In Postgres, we use pg_isready to wait until the database is accepting connections.</li><li><strong>depends_on</strong> — This defines the startup order and health requirements. For example, airflow depends on postgres and mock-api being healthy before it starts, ensuring that the database and API are ready to go when Airflow begins running tasks.</li></ul><h4>Example: The Postgres Service</h4><p>Let’s apply this to the Postgres service in our config:</p><pre>postgres:<br>  image: postgres:16                     # Use the official Postgres 16 image<br>  container_name: postgres               # Name the container &quot;postgres&quot;<br>  environment:<br>    POSTGRES_USER: ${POSTGRES_USER}      # DB username (from .env)<br>    POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}  # DB password (from .env)<br>    POSTGRES_DB: ${POSTGRES_DB}          # Initial database name (from .env)<br>  ports:<br>    - &quot;5432:5432&quot;                        # Expose DB port 5432 to host<br>  volumes:<br>    - ./postgres/init:/docker-entrypoint-initdb.d  # Initialization scripts<br>    - pgdata:/var/lib/postgresql/data    # Named volume for database storage<br>  healthcheck:<br>    test: [&quot;CMD-SHELL&quot;, &quot;pg_isready -U ${POSTGRES_USER} -d ${POSTGRES_DB}&quot;]  <br>    interval: 5s  <br>    timeout: 3s  <br>    retries: 30</pre><p>This sets up a fully initialized Postgres instance with persistent storage and a readiness check. By the time you run docker-compose up, you’ll have a database ready to connect to from dbt, Airflow, or Power BI.</p><p>With the infrastructure in place, we can move on to the data pipeline itself.</p><p>Once you’ve seen one service, you’ve basically seen them all — each one just tweaks the settings for its own needs. With this infrastructure running, we’re ready to build the data pipeline itself.</p><h3>2. Understanding the Dockerfiles</h3><p>If you look at our docker-compose.yml, you’ll notice that for both the <strong>Airflow</strong> and <strong>mock API</strong> services, we use the build key instead of an image. That tells Docker Compose to build each service using a local Dockerfile.</p><p>Why do we need custom images? Because sometimes the standard image isn’t enough — maybe we need extra Python packages, OS-level tools, or we want to bundle in our own application code. A <strong>Dockerfile</strong> is the recipe for building such custom images.</p><h4>Airflow Dockerfile: What’s Inside</h4><p>Let’s take a closer look at the Airflow container’s Dockerfile (airflow/Dockerfile):</p><pre>FROM apache/airflow:2.9.3-python3.11<br><br>USER root<br>RUN apt-get update \<br> &amp;&amp; apt-get install -y --no-install-recommends docker.io docker-compose-plugin \<br> &amp;&amp; rm -rf /var/lib/apt/lists/*<br><br>COPY requirements.txt /requirements.txt<br><br>USER airflow<br>RUN export AIRFLOW_CONSTRAINTS_URL=&quot;https://raw.githubusercontent.com/apache/airflow/constraints-2.9.3/constraints-3.11.txt&quot; \<br>    &amp;&amp; python -m pip install --no-cache-dir psycopg2-binary -c &quot;$AIRFLOW_CONSTRAINTS_URL&quot; \<br>    &amp;&amp; pip install --no-cache-dir -r /requirements.txt -c &quot;$AIRFLOW_CONSTRAINTS_URL&quot;</pre><p><strong>Breaking it down:</strong></p><ul><li>FROM: We start from the official Apache Airflow image with Python 3.11, which already has Airflow pre-installed.</li><li>USER root: Temporarily switch to the root user so we can install system-level packages.</li><li>RUN: Install the Docker CLI and Compose plugin inside the container. This is necessary if Airflow needs to run Docker commands (like launching dbt via docker exec).</li><li>COPY: Add our Python requirements.txt into the image.</li><li>USER airflow: Switch back to the default (non-root) Airflow user for safety.</li><li>RUN: Install Python dependencies using pip — including psycopg2-binary(the Postgres driver) for Postgres access and anything else listed in our requirements. We pin versions using Airflow’s constraints file to ensure compatibility.</li></ul><p>This results in a <strong>custom Airflow image</strong> that includes:</p><ul><li>Airflow itself,</li><li>Docker tooling (for running containers from inside Airflow tasks),</li><li>Python packages specific to our project.</li></ul><p>In the Compose file, we told Docker to build this image via build: ./airflow and gave it the tag airflow-local.</p><h4>Mock API Dockerfile: A Minimal Custom Server</h4><p>Our mock API service follows the same idea. Instead of using a generic image, we build our own from mock_api/Dockerfile. It likely looks something like this:</p><pre>FROM python:3.11-slim<br><br>WORKDIR /app<br>COPY requirements.txt ./<br>RUN pip install --no-cache-dir -r requirements.txt<br>COPY . .<br><br>CMD [&quot;uvicorn&quot;, &quot;main:app&quot;, &quot;--host&quot;, &quot;0.0.0.0&quot;, &quot;--port&quot;, &quot;8000&quot;]</pre><p>This builds a lightweight Python container that:</p><ul><li>Installs our mock API’s Python dependencies (e.g., FastAPI).</li><li>Copies in our API source code.</li><li>Launches the server using Uvicorn.</li></ul><p>Again, this image is defined locally and built at runtime via build: ./mock_api in the Compose file.</p><h4>Why This Matters</h4><p>Using custom Dockerfiles like this gives you:</p><ul><li>Full control over what’s installed in each container</li><li>A consistent, reproducible development environment.</li><li>The ability to version your entire infrastructure alongside your code.</li></ul><p>Because each Dockerfile lives right next to its service’s code, the whole stack is <strong>self-contained</strong>. Anyone can clone your repo, run docker-compose up, and spin up an identical environment — with all customizations already baked in.</p><h3>3. Extract &amp; Load: Ingesting Data from the API into Postgres</h3><p>Our first pipeline step is to automatically pull data from the API and load it into PostgreSQL. We’ll use Airflow for this automation. To keep things clean and DRY (Don’t Repeat Yourself), we’ll write one reusable Python function that handles <em>both</em> extracting data from the API and loading it into Postgres for any given dataset. Then, for each type of data (customers, payments, sessions, etc.), we’ll call this function with different parameters.</p><p>This function — extract_table—does the heavy lifting. It connects to Postgres, figures out where the last run left off, pulls only new or updated records from the API, and upserts them into the appropriate table. Each Airflow task will simply call this function with its own configuration (for example, one task for customers, another for payments, another for sessions). In other words, rather than writing three separate scripts for three data types, we let one smart extractor do all the work.</p><p>Here’s what the extract_table function looks like (simplified for clarity):</p><pre>from datetime import datetime, timedelta<br>from dateutil import parser<br>from airflow.providers.postgres.hooks.postgres import PostgresHook<br><br>def extract_table(table, endpoint, ts_field, wm_name):<br>    pg = PostgresHook(postgres_conn_id=&quot;postgres_default&quot;)<br>    iso_default = (datetime.utcnow() - timedelta(days=365)).isoformat() + &quot;Z&quot;<br>    watermark = _get_watermark(wm_name, iso_default)<br>    max_seen = parser.isoparse(watermark)<br>    rows = 0<br><br>    for page in _fetch_paged(endpoint, {&quot;updated_since&quot;: watermark}):<br>        data = page[&quot;data&quot;]<br>        if not data:<br>            continue<br>        with pg.get_conn() as conn:<br>            with conn.cursor() as cur:<br>                if table == &quot;raw.customers&quot;:<br>                    cur.executemany(<br>                        &quot;&quot;&quot;<br>                        INSERT INTO raw.customers(customer_id, company_name, country, industry, company_size, signup_date, updated_at, is_churned)<br>                        VALUES (%(customer_id)s, %(company_name)s, %(country)s, %(industry)s, %(company_size)s, %(signup_date)s, %(updated_at)s, %(is_churned)s)<br>                        ON CONFLICT (customer_id) DO UPDATE <br>                        SET company_name = EXCLUDED.company_name,<br>                            country = EXCLUDED.country,<br>                            industry = EXCLUDED.industry,<br>                            company_size = EXCLUDED.company_size,<br>                            signup_date = EXCLUDED.signup_date,<br>                            updated_at = EXCLUDED.updated_at,<br>                            is_churned = EXCLUDED.is_churned<br>                        &quot;&quot;&quot;,<br>                        data<br>                    )<br>                elif table == &quot;raw.payments&quot;:<br>                    cur.executemany(<br>                        &quot;&quot;&quot;<br>                        INSERT INTO raw.payments(payment_id, customer_id, product, amount, currency, status, refunded_amount, fee, payment_method, country, created_at, updated_at)<br>                        VALUES (%(payment_id)s, %(customer_id)s, %(product)s, %(amount)s, %(currency)s, %(status)s, %(refunded_amount)s, %(fee)s, %(payment_method)s, %(country)s, %(created_at)s, %(updated_at)s)<br>                        ON CONFLICT (payment_id) DO UPDATE <br>                        SET customer_id    = EXCLUDED.customer_id,<br>                            product        = EXCLUDED.product,<br>                            amount         = EXCLUDED.amount,<br>                            currency       = EXCLUDED.currency,<br>                            status         = EXCLUDED.status,<br>                            refunded_amount= EXCLUDED.refunded_amount,<br>                            fee            = EXCLUDED.fee,<br>                            payment_method = EXCLUDED.payment_method,<br>                            country        = EXCLUDED.country,<br>                            created_at     = EXCLUDED.created_at,<br>                            updated_at     = EXCLUDED.updated_at<br>                        &quot;&quot;&quot;,<br>                        data<br>                    )<br>                else:<br>                    # Assuming this is raw.sessions or any other table<br>                    for d in data:<br>                        # API returns booleans; cast to ints for storage<br>                        d[&quot;bounced&quot;] = int(d.get(&quot;bounced&quot;, False))<br>                        d[&quot;converted&quot;] = int(d.get(&quot;converted&quot;, False))<br>                    cur.executemany(<br>                        &quot;&quot;&quot;<br>                        INSERT INTO raw.sessions(session_id, customer_id, source, medium, campaign, device, country, pageviews, session_duration_s, bounced, converted, session_start, updated_at)<br>                        VALUES (%(session_id)s, %(customer_id)s, %(source)s, %(medium)s, %(campaign)s, %(device)s, %(country)s, %(pageviews)s, %(session_duration_s)s, %(bounced)s, %(converted)s, %(session_start)s, %(updated_at)s)<br>                        ON CONFLICT (session_id) DO NOTHING<br>                        &quot;&quot;&quot;,<br>                        data<br>                    )<br>        rows += len(data)<br>        # Update max_seen timestamp for watermark<br>        for d in data:<br>            ts = parser.isoparse(d.get(ts_field))<br>            if ts &gt; max_seen:<br>                max_seen = ts<br><br>    _set_watermark(wm_name, max_seen.isoformat())<br>    return rows</pre><p>Let’s break down what this function is doing:</p><ol><li><strong>Connect to Postgres:</strong> It uses Airflow’s PostgresHook (from the Airflow Postgres provider) to get a connection to our Postgres database (using the connection ID postgres_default, which we set up in Airflow’s config).</li><li><strong>Determine the start timestamp (watermark):</strong> We want to load data incrementally. The function looks up a stored timestamp (a “watermark”) for where it left off last time. We store these watermarks in Airflow Variables. _get_watermark(wm_name, iso_default) fetches the last processed timestamp for this dataset (identified by wm_name). If no watermark exists yet, it uses a default of one year ago (iso_default) to start. This means on the first run, it will backfill up to a year of data (you can adjust this window as needed).</li><li><strong>Fetch data from the API in pages:</strong> It calls an internal helper _fetch_paged(endpoint, {&quot;updated_since&quot;: watermark}) to retrieve data from the API endpoint, using updated_since to get only records that are new or updated since the last watermark. The API returns data in pages; we loop through each page.</li></ol><ul><li>For each page of results, we upsert the records into the appropriate <strong>raw table</strong> in Postgres:</li><li><strong>Customers &amp; Payments:</strong> We use SQL INSERT ... ON CONFLICT ... DO UPDATE to upsert by primary key (customer_id or payment_id). That means if a record with that primary key already exists, we update it with the new data; if not, we insert a new row. This handles both new records and updates to existing ones.</li><li><strong>Sessions:</strong> For sessions, we first convert boolean flags (bounced, converted) from True/False to integers 0/1 because our raw table stores them as integers. Then we insert the session records. We use ON CONFLICT ... DO NOTHING for sessions, assuming that if the same session appears again (perhaps the API could resend it), we skip it since sessions are not expected to update once recorded (or we’re okay ignoring duplicates).</li></ul><p><strong>4. Track the number of rows and max timestamp:</strong> We keep a running count of how many rows were inserted/updated (rows). We also track the maximum updated_at timestamp seen in the batch (max_seen).</p><p><strong>5. Update the watermark:</strong> After processing all pages, we call _set_watermark(wm_name, max_seen.isoformat()) to save the new latest timestamp back into an Airflow Variable. This updates our watermark so that the next run will start from this point forward, avoiding reprocessing the same data</p><p><strong>6. Return the row count:</strong> The function returns the number of rows loaded, which is useful for logging or debugging (though in production you might not always use the return value).</p><p>With this extractor function ready, the extraction + load portion of our pipeline can be implemented. Next, we’ll integrate this function into Airflow as tasks.</p><h3>4. Airflow and DAGs: How the Pipeline is Scheduled</h3><p>We use <strong>Apache Airflow</strong> to orchestrate and schedule the entire pipeline. Airflow allows us to define workflows as code and handles running tasks in the right order, retries, scheduling, and monitoring — all via a nice UI.</p><p>At the heart of Airflow is the <strong>DAG</strong> (Directed Acyclic Graph), which defines the structure of your workflow. Think of it like a flowchart that answers three key questions for your pipeline:</p><ul><li><strong>What tasks exist?</strong> (e.g., “extract customers”, “run dbt transformations”, etc.)</li><li><strong>When should it run?</strong> (the schedule, e.g., every 15 minutes, daily at midnight, etc.)</li><li><strong>In what order should tasks run?</strong> (the dependencies between tasks, e.g., “run transformations only after all extracts finish”).</li></ul><p>Each node in the graph is a task, and each arrow between nodes defines a dependency (for example, tasks A → B means “A must finish before B can start”). In Airflow, tasks are represented by <em>operators</em> — these are basically templates for work. Two common ones we use here are:</p><ul><li><strong>PythonOperator: </strong>runs a Python function.</li><li><strong>BashOperator: </strong>runs a shell command.</li></ul><p>Airflow lets us visualize the DAG in its UI. For our pipeline DAG (let’s call it <strong>etl_api_to_bi</strong>), once you’ve started up the docker (via docker compose up — build and log into airflow) you should see something like this: three extract tasks branching out in parallel and then converging into two sequential dbt tasks, all within one workflow. Each box is a task and arrows enforce the order (extract tasks first, then dbt steps).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j7Xg7aJz_kxJK1C1faet9w.jpeg" /><figcaption><em>Airflow DAG visualizing the ETL pipeline (</em><em>etl_api_to_bi).</em></figcaption></figure><h4><strong>What Goes Into a DAG?</strong></h4><p>A minimal Airflow DAG definition usually includes a few key settings:</p><ul><li><strong>dag_id:</strong> A unique name for the DAG.</li><li><strong>start_date:</strong> When to begin scheduling (usually in the past, and Airflow will not run anything before this date).</li><li><strong>schedule_interval:</strong> How often to run (e.g., &quot;@daily&quot; for daily at midnight, &quot;*/15 * * * *&quot; for every 15 minutes, or None for manual runs, etc.).</li><li><strong>catchup:</strong> Whether to catch up missed intervals between the start_date and now. If set to False, Airflow won’t try to backfill old runs.</li><li><strong>default_args:</strong> A dictionary of default parameters for tasks (like retry counts, email on failure, etc.).</li><li><strong>tags:</strong> (Optional) Labels to help organize or search for the DAG in the Airflow UI.</li></ul><p>In plain English: you give your DAG a name, tell it how often to run, when to start, and whether to retroactively fill in missed runs. You also wire up the tasks inside it and define how they depend on each other.</p><p><strong>Example 1: A Minimal DAG — </strong>just to illustrate the structure, here’s the smallest possible Airflow DAG:</p><pre>from airflow import DAG<br>from pendulum import datetime<br>from airflow.operators.empty import EmptyOperator<br><br>DEFAULT_ARGS = {<br>    &quot;owner&quot;: &quot;data&quot;,<br>    &quot;retries&quot;: 1,<br>}<br><br>with DAG(<br>    dag_id=&quot;minimal_example&quot;,<br>    start_date=datetime(2024, 1, 1),<br>    schedule_interval=&quot;@daily&quot;,<br>    catchup=False,<br>    default_args=DEFAULT_ARGS,<br>    tags=[&quot;example&quot;],<br>) as dag:<br>    start = EmptyOperator(task_id=&quot;start&quot;)<br>    end = EmptyOperator(task_id=&quot;end&quot;)<br><br>    start &gt;&gt; end</pre><p>This simple DAG runs once per day (starting Jan 1, 2024), has two dummy tasks (“start” and “end”), and does not backfill any runs prior to being turned on. The EmptyOperator tasks don&#39;t actually do anything; they&#39;re just placeholders here. The start &gt;&gt; end line defines the order: start runs first, then end.</p><p><strong>Example 2: The Project DAG — </strong>now let’s look at the actual DAG for our project (simplified for clarity). This DAG runs every 15 minutes, extracts data from multiple API endpoints (customers, payments, sessions), and then triggers dbt to install dependencies and build models. We’ll use our extract_table function for each extract task:</p><pre>from airflow import DAG<br>from pendulum import datetime<br>from datetime import timedelta<br>from airflow.operators.python import PythonOperator<br>from airflow.operators.bash import BashOperator<br><br>DEFAULT_ARGS = {<br>    &quot;owner&quot;: &quot;data&quot;,<br>    &quot;retries&quot;: 2,<br>    &quot;retry_delay&quot;: timedelta(minutes=2),<br>}<br><br>with DAG(<br>    dag_id=&quot;etl_api_to_bi&quot;,<br>    start_date=datetime(2024, 1, 1),<br>    schedule_interval=&quot;*/15 * * * *&quot;,  # every 15 minutes<br>    catchup=False,<br>    default_args=DEFAULT_ARGS,<br>    tags=[&quot;api&quot;, &quot;postgres&quot;, &quot;dbt&quot;, &quot;bi&quot;],<br>) as dag:<br><br>    # Extraction tasks (one per API endpoint)<br>    extract_customers = PythonOperator(<br>        task_id=&quot;extract_customers&quot;,<br>        python_callable=extract_table,<br>        op_kwargs={<br>            &quot;table&quot;: &quot;raw.customers&quot;,<br>            &quot;endpoint&quot;: &quot;customers&quot;,<br>            &quot;ts_field&quot;: &quot;updated_at&quot;,<br>            &quot;wm_name&quot;: &quot;wm_customers&quot;,<br>        },<br>    )<br><br>    extract_payments = PythonOperator(<br>        task_id=&quot;extract_payments&quot;,<br>        python_callable=extract_table,<br>        op_kwargs={<br>            &quot;table&quot;: &quot;raw.payments&quot;,<br>            &quot;endpoint&quot;: &quot;payments&quot;,<br>            &quot;ts_field&quot;: &quot;updated_at&quot;,<br>            &quot;wm_name&quot;: &quot;wm_payments&quot;,<br>        },<br>    )<br><br>    extract_sessions = PythonOperator(<br>        task_id=&quot;extract_sessions&quot;,<br>        python_callable=extract_table,<br>        op_kwargs={<br>            &quot;table&quot;: &quot;raw.sessions&quot;,<br>            &quot;endpoint&quot;: &quot;sessions&quot;,<br>            &quot;ts_field&quot;: &quot;updated_at&quot;,<br>            &quot;wm_name&quot;: &quot;wm_sessions&quot;,<br>        },<br>    )<br><br>    # dbt tasks (install dependencies, then run models/tests)<br>    dbt_deps = BashOperator(<br>        task_id=&quot;dbt_deps&quot;,<br>        bash_command=&quot;cd /opt/airflow/dags/dbt &amp;&amp; dbt deps&quot;,<br>    )<br><br>    dbt_build = BashOperator(<br>        task_id=&quot;dbt_build&quot;,<br>        bash_command=&quot;cd /opt/airflow/dags/dbt &amp;&amp; dbt build&quot;,<br>    )<br><br>    # Define task dependencies:<br>    [extract_customers, extract_payments, extract_sessions] &gt;&gt; dbt_deps &gt;&gt; dbt_build</pre><p>A few things to note in this DAG code:</p><ul><li>We schedule it with schedule_interval=&quot;*/15 * * * *&quot;, which is a cron expression for &quot;every 15 minutes&quot;. This means the pipeline stays up-to-date throughout the day, capturing new data frequently instead of doing one big daily batch.</li><li>We set catchup=False to avoid a backlog of runs if the scheduler was off for a while.</li><li>We define <strong>three PythonOperator tasks</strong>: one for each dataset (customers, payments, sessions). Each one calls the same function extract_table but with different op_kwargs (keyword arguments) specifying which table/endpoint to target and which timestamp field and watermark variable to use. This approach keeps our DAG code concise and avoids duplicating extraction logic; the only difference between these tasks is the parameters.</li><li>We define <strong>two BashOperator tasks</strong> for dbt: dbt_deps to install any dbt packages (if our project depends on external dbt packages) and dbt_build to run the dbt project (which will compile models, run them, and run tests).</li><li>Finally, we set the order of execution: all three extract tasks can run in parallel (they’re independent of each other), but <strong>all of them must complete successfully before</strong> moving on. The &gt;&gt; syntax chains dependencies: the list [extract_customers, extract_payments, extract_sessions] feeds into dbt_deps (meaning dbt_deps waits for all three extracts), and then dbt_deps feeds into dbt_build.</li></ul><p>With this DAG in place, Airflow will orchestrate the entire process: every 15 minutes it will trigger extracts from the API into Postgres, then kick off dbt to transform the data, and so on. We can monitor the runs and any issues in the Airflow web UI.</p><p>Now that the pipeline orchestration is defined, let’s focus on the transformation layer and how <strong>dbt</strong> fits into this stack.</p><h3>5. Wiring the Extractor into Airflow (how the api_dags work)</h3><p>(If you’re already familiar with how we passed arguments to extract_table, you can skip ahead to the next section on dbt.)</p><p>The three extraction tasks in the DAG deserve a closer look to see how we parameterize our single function for multiple uses. Each PythonOperator is given:</p><ul><li>a <strong>task_id</strong> (just a unique name like &quot;extract_customers&quot;),</li><li>the <strong>python_callable</strong> which is our extract_table function,</li><li>and <strong>op_kwargs</strong> — a dictionary of keyword arguments that will be passed into extract_table when it runs.</li></ul><p>For example, the extract_customers task above does:</p><pre>python_callable=extract_table,<br>op_kwargs={<br>    &quot;table&quot;: &quot;raw.customers&quot;,<br>    &quot;endpoint&quot;: &quot;customers&quot;,<br>    &quot;ts_field&quot;: &quot;updated_at&quot;,<br>    &quot;wm_name&quot;: &quot;wm_customers&quot;,<br>},</pre><p>This means when Airflow runs the task, it will execute extract_table(table=&quot;raw.customers&quot;, endpoint=&quot;customers&quot;, ts_field=&quot;updated_at&quot;, wm_name=&quot;wm_customers&quot;). Similarly, extract_payments and extract_sessions provide their respective endpoints and table names. In each case, the function knows which API endpoint to call, which database table to insert into, which field to use for incremental updates, and what name to use for the watermark variable.</p><p>Using one generic function for all these extractions keeps the code <strong>DRY and scalable</strong>. If tomorrow we had a new dataset to ingest, we could simply add another task with a new set of parameters, rather than writing a whole new function. Each extraction task runs independently (Airflow will even run them in parallel up to the scheduler’s capability), and once they’re all done, they flow into the transformation tasks (dbt_deps and dbt_build).</p><p>At this point, we have the <strong>E</strong> (Extract) and <strong>L</strong> (Load) of our ETL pipeline implemented: data is being pulled from the API and stored raw in Postgres, on an automated schedule. Next, we tackle the <strong>T</strong> (Transform) part using dbt.</p><h3>6. Using dbt for Transformations in Our Stack</h3><p>By now, our Airflow DAG is successfully landing three raw tables in Postgres (one each for customers, payments, sessions). However, that raw data isn’t yet in a form that’s easy for analysis or BI tools. This is where <strong>dbt (data build tool)</strong> comes in — to transform the raw data into cleaned, joined, and aggregated tables suitable for analysis.</p><p>Before we dive into how to set up and run dbt in our pipeline, let’s briefly cover <em>what dbt is</em> and <em>why we’re using it</em> in this context.</p><h4><strong>What is dbt</strong></h4><p><strong>dbt (data build tool)</strong> is an open-source framework for managing data transformations. It allows you to build your transformations with <strong>SQL</strong> (and a bit of Jinja templating if needed) while following software engineering best practices like version control, modularity, and testing. With dbt, you define <strong>models</strong> (which are essentially SQL select statements representing new tables or views), and dbt takes care of running them in the right order (based on dependencies), materializing them as tables/views, and even running tests and generating documentation.</p><p>Key features of dbt include:</p><ul><li>Writing transformations as SQL select statements in .sql files, which dbt then turns into tables or views in your database.</li><li>Using Jinja for parametric SQL (for example, you can reuse logic or dynamically reference other models).</li><li>Dependency management: you can reference one model in another (using ref(&#39;other_model&#39;)), and dbt figures out the order to run them.</li><li><strong>Materializations:</strong> You can configure each model to be a table, view, incremental load, etc., depending on how you want it built.</li><li>Built-in testing and documentation: You can define tests to ensure data quality (like “no nulls in this column” or “values must be one of X, Y, Z”) and dbt can generate an HTML documentation site with model schemas and lineage.</li><li><strong>Packages:</strong> You can include pre-built dbt packages (think of them like libraries/macros that others have written) to extend functionality.</li></ul><p>In summary, dbt brings software engineering rigor (modularity, testing, version control) to the SQL transformation layer of your pipeline.</p><h4>Why use dbt in our pipeline?</h4><p>You might wonder, couldn’t we just transform the data with Python or SQL scripts inside Airflow? Yes, that’s possible, but dbt provides a lot of benefits:</p><ul><li><strong>Clear separation of concerns:</strong> Airflow handles scheduling and orchestration, while dbt is purpose-built for data transformation logic and data quality checks. Airflow calls dbt, and dbt does the transformations.</li><li><strong>Reusability and Maintainability:</strong> Instead of one giant SQL script or a mix of Python and SQL logic, dbt encourages modular SQL models. Each model usually corresponds to one logical transformation or table.</li><li><strong>Testing and Documentation:</strong> With dbt, we can easily add tests to ensure our transformed data is correct (e.g., no duplicate IDs, categories are within a known list, etc.) and auto-generate documentation. Doing this from scratch in Airflow or custom scripts would be a lot of work.</li><li><strong>Version control:</strong> The entire transformation codebase (SQL files and config) lives in a repository, so changes are tracked, and you can code review them just like application code.</li><li><strong>Community and packages:</strong> dbt has a strong community and many existing packages/macros that can be leveraged for common transformations or utilities.</li></ul><p>In our stack, you can think of Airflow as the <strong>conductor</strong> and dbt as the <strong>band</strong>. Airflow pulls fresh data into the raw tables on a schedule, then hands off to dbt. dbt takes that raw data and performs the cleaning, joining, and aggregating to produce the final data sets for analysis. The result: reproducible, tested, and production-ready SQL transformations — rather than ad-hoc notebooks or one-off scripts.</p><h4>How dbt Organizes Models (Raw, Staging, Marts)</h4><p>A typical dbt project organizes transformations in <strong>layers</strong>. In this project, we use a simple three-layer approach:</p><ul><li><strong>Raw layer:</strong> These are the raw tables in the database that Airflow populated (in our case, the raw schema in Postgres). They reflect the data exactly as we received it from the API, with minimal tweaks. (For example, we might store booleans as integers, but we haven’t merged or transformed anything yet.) In dbt, we will declare these as <strong>sources</strong>.</li><li><strong>Staging layer:</strong> Staging models are usually one step cleaned-up versions of the raw data. In our project, these are in the public_staging schema. Each staging model typically does trivial transformations: select from the raw data, cast types, standardize field names, maybe filter some rows, etc. Staging models often correspond 1:1 to a single source table (one staging model per raw table) and are often materialized as <strong>views</strong> (since they are just straightforward selects).</li><li><strong>Marts (Analytics) layer:</strong> These are the models that deliver business value — the facts and dimensions that a BI tool or analyst will actually use. We build these in the public_analytics schema. Marts can be materialized as tables (for performance) and usually combine data from multiple staging models, or aggregate data to a higher level (e.g., daily revenue, user profiles, conversion rates). This is where you implement business logic: e.g., calculating churn rates, revenue per day, joining sessions with payments, etc.</li></ul><p>To make it concrete:</p><ul><li>A <strong>staging model</strong> might take raw.payments and filter to only successful payments, casting data types and maybe adding a computed column for net amount.</li><li>A <strong>mart model</strong> might join stg_payments with stg_customers to get a revenue fact table that includes customer info, or sum stg_payments by day for a daily revenue table, etc.</li></ul><p>The layers ensure that each step is manageable and testable, and you can change one part (like how you define net revenue) without breaking an unrelated part (like how sessions are processed).</p><p>Now that we understand dbt’s role, let’s set it up for our project.</p><h3>7. Setting Up the dbt/ Directory (Configs, Profiles, Tests)</h3><p>Our project’s code repository is organized roughly like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/1*GJ499MCKdazC1Cadg3xTHw.jpeg" /></figure><p>Let’s focus on the dbt/ folder – it contains the configuration and model files for our dbt project. Here’s a quick breakdown of the key files and what they do:</p><blockquote><strong><em>Note:</em></strong><em> The </em><strong><em>dashboard/</em></strong><em> folder includes a working Power BI file built on the analytics tables. A full BI walkthrough is out of scope, but once your model exists in Postgres, connecting Power BI/Tableau is straightforward.</em></blockquote><ol><li><strong>packages.yml: </strong>Lists any external dbt packages our project uses. (For example, we might include the dbt_utils package, which provides helpful macros.) Running dbt deps will install these packages into a dbt_packages/ directory. For our project, this might include community packages or could be empty if we don&#39;t need extras.</li><li><strong>package-lock.yml</strong> —An auto-generated lock file that pins the exact versions of the packages in use. This ensures reproducibility (so that dbt deps always grabs the same versions, even if new versions are released).</li></ol><p><strong>3. dbt_project.yml</strong> — The main configuration file for the dbt project. It declares the project name and tells dbt where to find models, seeds, snapshots, etc., and how to build them. For example, it can specify that models in the staging folder should be materialized as views in the public_staging schema, and models in the marts folder as tables in the public_analytics schema. Let&#39;s look at key parts of ours:</p><pre># dbt/dbt_project.yml<br>name: api_bi_project<br>version: 1.0.0<br>config-version: 2<br><br>profile: api_bi_profile  # This tells dbt which profile (in profiles.yml) to use for connection<br><br># Specify where the models are:<br>model-paths: [&quot;models&quot;]<br><br>models:<br>  api_bi_project:      # This must match the project name above.<br>    staging:<br>      +materialized: view       # models in models/staging/ -&gt; create as views<br>      +schema: staging          # and build in schema &quot;public_staging&quot; (see profile base schema)<br>    marts:<br>      +materialized: table      # models in models/marts/ -&gt; create as tables<br>      +schema: analytics        # and build in schema &quot;public_analytics&quot;</pre><p>In our setup, we’ve chosen to prepend the schema names with public_. If you look at the <strong>profile</strong> next, you’ll see we set the default schema to public. By doing that, and then using schema: staging and schema: analytics in the project config, dbt will actually create objects in <strong>public_staging</strong> and <strong>public_analytics</strong>. This naming convention keeps all our schemas together (under the public prefix) and avoids having too many top-level schema names. It also aligns with how we set up the database (we will create public_staging and public_analytics schemas ahead of time).</p><p><strong>4. profiles.yml </strong>—This file defines how dbt connects to your database. It can contain multiple profiles (for example, one for local dev, one for prod), but here we just need one. In our profiles.yml, we have something like:</p><pre># dbt/profiles/profiles.yml<br>api_bi_profile:<br>  target: dev<br>  outputs:<br>    dev:<br>      type: postgres         # We&#39;re using Postgres<br>      host: postgres         # Hostname of the DB (the Docker service name)<br>      user: &quot;{{ env_var(&#39;POSTGRES_USER&#39;) }}&quot;<br>      password: &quot;{{ env_var(&#39;POSTGRES_PASSWORD&#39;) }}&quot;<br>      port: 5432<br>      dbname: &quot;{{ env_var(&#39;POSTGRES_DB&#39;) }}&quot;<br>      schema: &quot;public&quot;       # Base schema; dbt will create schemas like public_staging, public_analytics<br>      threads: 4             # Run up to 4 models in parallel</pre><p>We named our profile api_bi_profile (as referenced in dbt_project.yml). We have a single target called dev. The connection credentials (user, password, database name) are all pulled from environment variables – which are the same ones we passed into the containers via Docker Compose. Both the Airflow container and the dedicated dbt container know these environment variables, and we’ve set DBT_PROFILES_DIR in both to point to the dbt/profiles/ directory. This way, whether we run dbt via Airflow or manually in the container, it uses the same connection info.</p><blockquote><em>A note on the schema: We set </em><em>schema: &quot;public&quot; here so that the actual schemas used will be </em><em>public + whatever suffix we specify in the project config (staging, analytics). This is a design choice:</em></blockquote><blockquote><em>1. It keeps all our schemas grouped (all dbt-created schemas start with public_*).</em></blockquote><blockquote><em>2. It avoids cluttering the database with too many unrelated schema names.</em></blockquote><blockquote><em>3. It works nicely with our initialization SQL, which will create public_staging and public_analytics schemas in advance.</em></blockquote><p><strong>5. models.yml </strong>—This is a configuration file (usually placed in the models/ directory) where we centralize documentation and data tests for our models. In models.yml, we list our models and specify tests or descriptions for each field. For example, we might declare that customer_id in the stg_customers model should be unique and not null, or that the status field in stg_payments should only contain the values &#39;succeeded&#39;, &#39;failed&#39;, or &#39;refunded&#39;. By defining these tests, we let dbt validate our data after transformation. If a test fails (say, a null sneaks into customer_id), dbt will flag it.</p><pre>version: 2<br>models:<br>  - name: stg_customers<br>    columns:<br>      - name: customer_id<br>        tests: [not_null, unique]<br>      - name: country<br>        tests: [not_null]<br><br>  - name: stg_payments<br>    columns:<br>      - name: payment_id<br>        tests: [not_null, unique]<br>      - name: status<br>        tests:<br>          - accepted_values:<br>              values: [succeeded, failed, refunded]<br><br>  - name: stg_sessions<br>    columns:<br>      - name: session_id<br>        tests: [not_null, unique]<br>      - name: source<br>        tests:<br>          - accepted_values:<br>              values: [google, direct, facebook, linkedin, newsletter, referral, bing]<br>              <br>  - name: fct_revenue_daily<br>    columns:<br>      - name: order_day<br>        tests: [not_null]<br>  - name: fct_marketing_attribution<br>    columns:<br>      - name: session_day<br>        tests: [not_null]</pre><p>This configuration is saying, for instance, <strong>stg_customers</strong> should have no nulls and no duplicates in customer_id. Or for <strong>stg_sessions</strong>, the source should only be one of a known list of values (google, direct, etc.). These tests help us catch any unexpected issues early. When we run dbt build later, it will run these tests after building the models.</p><blockquote><strong><em>Tip:</em></strong><em> Installed packages include their own </em><em>dbt_project.yml inside </em><em>dbt_packages/. These are internal to the package and won’t override your main project settings.</em></blockquote><p>Now that dbt knows <strong>how</strong> and <strong>where</strong> to build our models, we have to ensure those target schemas (and proper permissions) exist in Postgres. This brings us to preparing the database.</p><h3>8. Preparing PostgreSQL with Schemas and Roles (for dbt and BI)</h3><p>Before running dbt, we need to set up our Postgres database with the right <strong>schemas</strong> and <strong>user roles</strong>. In particular:</p><ul><li>Create the schemas that we referenced in dbt (e.g., raw, public_staging, public_analytics).</li><li>Create the raw tables that Airflow will load into (if not already created).</li><li>Create a read-only user/role (often for the BI tool to query data) and grant it appropriate permissions (read access to analytics schema, and maybe raw data if needed, but no access to staging or the ability to modify anything).</li></ul><p>We automate this setup by using the Postgres Docker image’s initialization feature. Recall in our Docker Compose file for the Postgres service, we had:<strong>Mount the init folder (Docker Compose)</strong></p><pre>volumes:<br>  - ./postgres/init:/docker-entrypoint-initdb.d</pre><p>This means we have an init folder with SQL scripts that will run the first time the Postgres container is started (they run in alphabetical order). We placed two SQL files in there:</p><ul><li><strong>01_schemas.sql:</strong> Sets up the schemas (raw, public_staging, public_analytics) and creates the raw tables.</li><li><strong>02_bi_read.sql:</strong> Sets up a read-only role/user (bi_read) and grants it select permissions on the appropriate schemas.</li></ul><p>We prefixed them with 01_ and 02_ to ensure the order (the Postgres entrypoint runs files in alphabetical order). It’s important the schemas and tables are created before we try to grant permissions on them, hence the naming. Another important practice is making these scripts <strong>idempotent</strong> – meaning they can run multiple times without causing errors. We do this by using IF NOT EXISTS in CREATE statements and careful GRANT/REVOKE logic. Idempotency ensures that if you ever need to rebuild the database or run the scripts again, you won&#39;t end up with conflicts or duplicate objects.</p><p>Let’s look at <strong>01_schemas.sql</strong> (excerpted for brevity):</p><pre><br>-- Create schemas (if they don&#39;t already exist)<br>CREATE SCHEMA IF NOT EXISTS raw;<br>-- &quot;raw&quot;: landing zone for source data as ingested.<br><br>CREATE SCHEMA IF NOT EXISTS public_staging;<br>-- &quot;public_staging&quot;: intermediate zone for cleaning and standardization.<br><br>CREATE SCHEMA IF NOT EXISTS public_analytics;<br>-- &quot;public_analytics&quot;: presentation layer for analytics/BI consumers.<br><br>-- Create raw tables (if they don&#39;t already exist)<br>CREATE TABLE IF NOT EXISTS raw.customers (<br>  customer_id UUID PRIMARY KEY,    -- Unique ID from source<br>  company_name TEXT,<br>  country TEXT,<br>  industry TEXT,<br>  company_size TEXT,<br>  signup_date TIMESTAMPTZ,         -- timestamps stored in UTC<br>  updated_at TIMESTAMPTZ,<br>  is_churned BOOLEAN<br>);<br><br>CREATE TABLE IF NOT EXISTS raw.payments (<br>  payment_id UUID PRIMARY KEY,<br>  customer_id UUID,       -- (FK to raw.customers.customer_id, not enforced here)<br>  product TEXT,<br>  amount NUMERIC,<br>  currency TEXT,<br>  status TEXT,<br>  refunded_amount NUMERIC,<br>  fee NUMERIC,<br>  payment_method TEXT,<br>  country TEXT,<br>  created_at TIMESTAMPTZ,<br>  updated_at TIMESTAMPTZ<br>);<br>CREATE INDEX IF NOT EXISTS idx_raw_payments_created ON raw.payments(created_at);<br><br>CREATE TABLE IF NOT EXISTS raw.sessions (<br>  session_id UUID PRIMARY KEY,<br>  customer_id UUID,<br>  source TEXT,              -- e.g., &quot;google&quot;, &quot;newsletter&quot;<br>  medium TEXT,              -- e.g., &quot;cpc&quot;, &quot;email&quot;<br>  campaign TEXT,<br>  device TEXT,<br>  country TEXT,<br>  pageviews INT,<br>  session_duration_s INT,<br>  bounced INT,              -- stored as 0/1 (came as boolean from API)<br>  converted INT,            -- stored as 0/1<br>  session_start TIMESTAMPTZ,<br>  updated_at TIMESTAMPTZ<br>);<br>CREATE INDEX IF NOT EXISTS idx_raw_sessions_start ON raw.sessions(session_start);<br><br>-- Note: We avoid defining foreign keys in the raw schema to prevent ingestion failures if referential integrity is temporarily broken or data arrives out of order.</pre><p>A few notes on the above:</p><ul><li>We created the schemas for each stage of our pipeline. These correspond to where data will live: raw for incoming data, public_staging for intermediate cleaned data, and public_analytics for final data.</li><li>We created the raw tables with columns matching the JSON fields from the API. We chose appropriate data types (UUID for IDs, TIMESTAMPTZ for timestamps, etc.). We included primary keys and some basic indexes to help with typical queries (like an index on created_at in payments, maybe useful for time-based loading).</li><li>The IF NOT EXISTS in each CREATE ensures that if the table already exists (say we restarted the container without wiping the volume, or we run the script again), it won&#39;t error out or override the table. It will just keep the existing table.</li><li>We did not create any tables in public_staging or public_analytics here – those will be created by dbt when it runs the models. We just made sure the schemas exist for dbt to create tables/views in them.</li></ul><p>Why is this setup important? Because if any of these schemas are missing or if dbt doesn’t have permission to create tables in them, our pipeline would fail at the transformation step. By pre-creating the schemas and raw tables, we ensure the landing zone is ready. By setting up permissions (coming next), we ensure our data teams and tools can access the final data safely.</p><p>Now, let’s set up the <strong>bi_read</strong> role for read-only access, via <strong>02_bi_read.sql</strong>:</p><p>What this script does:</p><pre>-- 1) Create or update the read-only role (user) for BI.<br>DO<br>$$<br>BEGIN<br>    IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = &#39;bi_read&#39;) THEN<br>        CREATE ROLE bi_read LOGIN PASSWORD &#39;bi_read_password&#39;;<br>    ELSE<br>        ALTER ROLE bi_read WITH LOGIN PASSWORD &#39;bi_read_password&#39;;<br>    END IF;<br>END<br>$$;<br><br>-- 2) Basic connectivity: allow the role to connect to the database.<br>GRANT CONNECT ON DATABASE db TO bi_read;<br><br>-- 2a) (Security hardening) Revoke access to schemas it shouldn&#39;t see.<br>REVOKE ALL ON SCHEMA public_staging FROM bi_read;<br>REVOKE ALL ON SCHEMA public FROM bi_read;<br>-- (We revoke any usage/select on those schemas, in case some default grants exist.)<br>-- We don&#39;t want the BI role reading intermediate or default schema data.<br><br>-- 2b) Grant read-only access on the target schemas (raw and analytics).<br>GRANT USAGE ON SCHEMA raw TO bi_read;<br>GRANT SELECT ON ALL TABLES IN SCHEMA raw TO bi_read;<br>GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA raw TO bi_read;<br><br>GRANT USAGE ON SCHEMA public_analytics TO bi_read;<br>GRANT SELECT ON ALL TABLES IN SCHEMA public_analytics TO bi_read;<br>GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public_analytics TO bi_read;<br><br>-- 3) Set default privileges for future objects created in those schemas:<br>-- (These must be run by the role that will create the objects, which is our dbt user (here likely the same as POSTGRES_USER for simplicity))<br>ALTER DEFAULT PRIVILEGES IN SCHEMA raw GRANT SELECT ON TABLES TO bi_read;<br>ALTER DEFAULT PRIVILEGES IN SCHEMA raw GRANT USAGE, SELECT ON SEQUENCES TO bi_read;<br><br>ALTER DEFAULT PRIVILEGES IN SCHEMA public_analytics GRANT SELECT ON TABLES TO bi_read;<br>ALTER DEFAULT PRIVILEGES IN SCHEMA public_analytics GRANT USAGE, SELECT ON SEQUENCES TO bi_read;<br><br>-- (We also explicitly *revoke* default privileges in staging and public schemas to ensure bi_read doesn&#39;t accidentally get access if someone creates objects there.)<br>ALTER DEFAULT PRIVILEGES IN SCHEMA public_staging REVOKE ALL ON TABLES FROM bi_read;<br>ALTER DEFAULT PRIVILEGES IN SCHEMA public_staging REVOKE ALL ON SEQUENCES FROM bi_read;<br>ALTER DEFAULT PRIVILEGES IN SCHEMA public REVOKE ALL ON TABLES FROM bi_read;<br>ALTER DEFAULT PRIVILEGES IN SCHEMA public REVOKE ALL ON SEQUENCES FROM bi_read;<br><br>-- 4) Set search_path for convenience (so queries by bi_read will search these schemas by default).<br>ALTER ROLE bi_read SET search_path TO public_analytics, raw;</pre><ul><li>It creates a role/user bi_read (or updates its password if it exists). In a real setup, you&#39;d manage the password securely (not plain text in a script) but for our demo it&#39;s hardcoded.</li><li>Grants CONNECT on the database, which is needed for the user to log in.</li><li><strong>Revoke</strong> any access on schemas we want to keep off-limits (the staging and default public schema). This is defense-in-depth in case, for example, the default public schema has some tables or someone grants wide access by mistake; we explicitly remove bi_read from those.</li><li><strong>Grant</strong> usage (ability to access schema) and select on all current tables/sequences in raw and public_analytics schemas. This means the bi_read role can read everything in those schemas.</li><li>Set up <strong>default privileges</strong> so that any new table created in raw or public_analytics in the future will automatically get a SELECT grant to bi_read. This is crucial because when dbt creates new models in public_analytics, we want bi_read to be able to query them without manual intervention. We run the ALTER DEFAULT PRIVILEGES commands as the user who will create the objects (in our case, the main Postgres user that dbt will use).</li><li>We also set the search_path for bi_read to public_analytics, raw so that if bi_read runs a query without schema qualifiers, it will look in the analytics schema first, then raw. This is a convenience so that a BI user can, for example, run SELECT * FROM fct_revenue_daily without specifying the schema, and it will find it in public_analytics.</li></ul><blockquote><strong><em>Security Note:</em></strong><em> The bi_read user has no password complexity or rotation in this demo. In a real production environment, you would want to manage this credential securely (e.g., generate a strong password or use IAM roles if in cloud, etc.). Also, granting raw data access to a BI role depends on your use case — you might or might not want that. Here we did, under the assumption that analysts might need to troubleshoot or dig into raw data occasionally. You can adjust the grants as needed.</em></blockquote><p>At this point, our database is prepared:</p><ul><li>The <strong>raw</strong> schema and tables are ready for data ingestion.</li><li>The <strong>public_staging</strong> and <strong>public_analytics</strong> schemas exist (though empty).</li><li>The permissions are set so that when dbt (using the main user) creates tables in public_analytics, the bi_read role will automatically have access to query them.</li><li>We have a dedicated read-only user for our BI tool to connect with.</li></ul><p>Now it’s time to actually run dbt as part of our pipeline and see the transformations happen.</p><p>Before moving on, let’s quickly illustrate what some of the dbt models look like in this project to solidify understanding:</p><h4><strong>Example Staging models:</strong></h4><ul><li><strong>stg_customers.sql: </strong>A staging model that selects from raw.customers and maybe casts or adds fields.</li></ul><pre>WITH src AS (<br>    SELECT * <br>    FROM raw.customers<br>)<br>SELECT<br>    customer_id,<br>    company_name,<br>    country,<br>    industry,<br>    company_size,<br>    signup_date::timestamp    AS signup_date,   -- cast from timestamptz to timestamp (if needed)<br>    updated_at::timestamp     AS updated_at,<br>    is_churned::boolean       AS is_churned,    -- cast to true boolean type<br>    DATE_TRUNC(&#39;month&#39;, signup_date)::date AS signup_month  -- example derived field (first day of signup month)<br>FROM src;</pre><p>This might materialize as a view in public_staging. It doesn’t reduce or join data, just prepares it.</p><ul><li><strong>stg_payments.sql </strong>—A staging model on payments:</li></ul><pre>WITH src AS (<br>    SELECT * FROM raw.payments<br>),<br>success AS (<br>    SELECT <br>        *, <br>        (amount - COALESCE(fee,0) - COALESCE(refunded_amount,0))::numeric AS net_revenue<br>    FROM src<br>    WHERE status = &#39;succeeded&#39;<br>)<br>SELECT<br>    payment_id,<br>    customer_id,<br>    product,<br>    amount,<br>    currency,<br>    status,<br>    refunded_amount,<br>    fee,<br>    payment_method,<br>    country,<br>    created_at::timestamp AS created_at,<br>    updated_at::timestamp AS updated_at,<br>    net_revenue,<br>    DATE_TRUNC(&#39;day&#39;, created_at)::date AS order_day<br>FROM success;</pre><p>Here we filtered to only successful payments and calculated a new field net_revenue. This is still at the transaction level (one row per payment), but it&#39;s cleaned up.</p><ul><li><strong>stg_sessions.sql </strong>— A staging model for sessions:</li></ul><pre>WITH src AS (<br>    SELECT * FROM raw.sessions<br>)<br>SELECT<br>    session_id,<br>    NULLIF(customer_id::text, &#39;&#39;)::uuid AS customer_id,  -- ensure empty strings are treated as nulls, and cast to UUID<br>    LOWER(source)  AS source,<br>    LOWER(medium)  AS medium,<br>    campaign,<br>    LOWER(device)  AS device,<br>    country,<br>    pageviews,<br>    session_duration_s,<br>    (bounced = 1)  AS bounced,    -- convert 0/1 to true/false<br>    (converted = 1) AS converted,<br>    session_start::timestamp AS session_start,<br>    updated_at::timestamp    AS updated_at,<br>    DATE_TRUNC(&#39;day&#39;, session_start)::date AS session_day<br>FROM src;</pre><p>We normalized some text to lowercase, converted the numeric flags back to boolean, and added a session_day for convenience (date without time).</p><h4><strong>Example Marts (analytics) models:</strong></h4><ul><li><strong>dim_customer.sql </strong>— A dimension table for customers (mostly a cleaned copy from staging, but could include aggregates like total spend, etc., if we joined with payments).</li></ul><pre>SELECT<br>    customer_id,<br>    company_name,<br>    country,<br>    industry,<br>    company_size,<br>    signup_date,<br>    is_churned,<br>    signup_month<br>FROM {{ ref(&#39;stg_customers&#39;) }};</pre><p>Here it’s just passing through the staging data, but in a real scenario you might enhance it with more info.</p><ul><li><strong>fct_revenue_daily.sql:</strong> A fact table rolling up revenue by day (and maybe by product or country).</li></ul><pre>SELECT<br>    order_day,<br>    product,<br>    country,<br>    SUM(net_revenue) AS net_revenue,<br>    COUNT(*) AS orders<br>FROM {{ ref(&#39;stg_payments&#39;) }}<br>GROUP BY order_day, product, country;</pre><p>This would produce one row per day-product-country with total revenue and count of orders.</p><ul><li><strong>fct_marketing_attribution.sql</strong> —A fact table focusing on sessions and conversions:</li></ul><pre>with s as (<br>  select * from {{ ref(&#39;stg_sessions&#39;) }}<br>)<br>select<br>  session_day,<br>  source,<br>  medium,<br>  coalesce(campaign,&#39;&#39;) as campaign,<br>  sum(converted::int) as conversions,<br>  avg((not bounced)::int)::float as engagement_rate<br>from s<br>group by 1,2,3,4</pre><ul><li>engagement rate (where engagement might be defined as “not bounced”).</li></ul><p>These examples show how raw data flows through staging (one-to-one, cleaning) into marts (joined or aggregated business tables).</p><p>Now with our dbt project configured and the database prepped, we can automate running dbt via Airflow.</p><h3>9. Automating dbt with Airflow</h3><p>We already added the tasks dbt_deps and dbt_build in our Airflow DAG. Let&#39;s explain how they work and how Airflow runs dbt in our Docker setup.</p><p>In our Docker Compose, we have a separate service/container for dbt (using the official dbt-postgres image). That container, when started, runs dbt deps &amp;&amp; dbt run once (as per the entrypoint we set) and then waits (tail -f /dev/null keeps it running idle). However, we want to run dbt on a schedule (every 15 minutes, after data is loaded). We achieve this by executing commands inside the running dbt container from Airflow.</p><p>The Airflow <strong>BashOperator</strong> can run shell commands. We use it to run dbt deps and dbt build by <strong>docker exec</strong>-ing into the dbt container:</p><pre>from airflow.operators.bash import BashOperator<br><br>dbt_deps = BashOperator(<br>    task_id=&quot;dbt_deps&quot;,<br>    bash_command=(<br>        &quot;docker exec dbt bash -lc &quot;<br>        &quot;&#39;cd /usr/app &amp;&amp; dbt deps --profiles-dir /root/.dbt&#39;&quot;<br>    ),<br>    dag=dag,<br>)<br><br>dbt_build = BashOperator(<br>    task_id=&quot;dbt_build&quot;,<br>    bash_command=(<br>        &quot;docker exec dbt bash -lc &quot;<br>        &quot;&#39;cd /usr/app &amp;&amp; dbt build --fail-fast --profiles-dir /root/.dbt&#39;&quot;<br>    ),<br>    dag=dag,<br>)</pre><p>What’s happening here?</p><ul><li>docker exec dbt ... means we run a command inside the container named &quot;dbt&quot;.</li><li>bash -lc &quot;...&quot; opens a bash shell, loads the environment (-l for login shell so that things like $DBT_PROFILES_DIR environment variable and any PATH settings are loaded), and -c executes the following string.</li><li>Inside the quotes, we navigate to /usr/app (which is the working directory where our dbt project is mounted in that container) and run dbt deps (for the first task) or dbt build (for the second task). We explicitly specify --profiles-dir /root/.dbt just to be sure it picks up our profiles file (though it should by default because we set the environment variable).</li></ul><p>The reason we split it into two tasks is to install packages (dbt deps) separately from running the models (dbt build). This way, if package installation fails or if we had multiple steps, we can isolate them. In many cases, you might run dbt deps just once on container start rather than every pipeline run (since packages don’t usually change often), but doing it in the DAG ensures that if you update packages.yml, the next run will pick up the change.</p><p>The dbt build command is a convenient all-in-one: it will <strong>run all models</strong> (building tables/views), and then run all tests (as defined in our models.yml or in any packages). We also pass --fail-fast which tells dbt to stop running further models if any model or test fails (this is a preference to catch issues early).</p><p>With these tasks in place, Airflow takes care of orchestrating them. Recall the dependency chain: extract_* &gt;&gt; dbt_deps &gt;&gt; dbt_build. So Airflow will:</p><ol><li>Run all the extract tasks (possibly in parallel).</li><li>Once <em>all three</em> are done (successful), run dbt_deps (which is quick, just installing packages).</li><li>Then run dbt_build.</li><li>If all goes well, the DAG is complete and we have fresh data ready for BI consumption. If something fails (e.g., the API is down, or a dbt test fails), Airflow will mark the run as failed and we can investigate using Airflow’s logs and UI. We set retries=2 on tasks, so Airflow will retry a failing task up to 2 times (with 2-minute delay in between) before giving up, which can handle transient issues.</li></ol><blockquote><em>(For those not as familiar with Bash or Docker: We could also have installed dbt directly in the Airflow container and run it there without </em><em>docker exec, but separating it into its own container aligns with the one-tool-per-container philosophy. The BashOperator simply allows us to trigger the dbt CLI inside that container.)</em></blockquote><h3>10. End-to-end flow (Airflow → dbt → BI)</h3><p>Now, let’s recap how all the pieces work together when you run the pipeline, and what you’d do with the data afterward:</p><ol><li><strong>Airflow (Orchestration &amp; EL)</strong> — Airflow kicks off the pipeline on schedule (every 15 minutes). The three extract tasks use extract_table to fetch new data from the API and load it into the <strong>raw schema</strong> in Postgres (raw.customers, raw.payments, raw.sessions). Each task updates its watermark so next time it only pulls fresh data. Once the raw data is loaded, Airflow moves on to run the dbt tasks.</li><li><strong>dbt (Transformation)</strong> — The dbt_deps task ensures any dependency packages are installed. Then dbt_build runs the entire dbt project:</li></ol><ul><li>dbt reads from the raw tables (defined as sources) and builds the <strong>staging models</strong> in the public_staging schema (materialized as views in our config).</li><li>Then it runs the <strong>mart models</strong> (the fact and dimension tables) in the public_analytics schema. These models might join or aggregate the staging data to produce the final tables.</li><li>As part of dbt build, it also executes our tests (from models.yml). If a test fails (e.g., we got a null where we expected none), dbt_build will error out, and Airflow will mark the task (and DAG run) as failed, alerting us to the issue.</li><li>Because we set up default privileges, the new tables in public_analytics are automatically readable by the bi_read role.</li></ul><p><strong>3. Business Intelligence (Consumption)</strong> — With the data pipeline automated and delivering fresh data, the last step is analysis. We can connect a BI tool like <strong>Power BI, Tableau, or even a Jupyter notebook</strong> to the Postgres database using the bi_read read-only user. The BI tool will primarily query the cleaned public_analytics tables (e.g., the fct_revenue_daily or fct_marketing_attribution table, and the dim_customer table). These are optimized for analysis: they have the metrics and dimensions readily available without the need for the BI tool to do heavy transformations. If needed, the analysts can also peek at the raw data (we did grant read access to raw), but generally the day-to-day work should happen on the curated data.</p><p>In the provided project repository, there might be a Power BI file (dashboard/PowerBI_dashboard.pbix or similar) that is pre-configured to use the bi_read user and is pulling from those analytics tables to create dashboards for, say, revenue trends and marketing conversion rates.</p><h3>11. Conclusion</h3><p>In this tutorial, we built an end-to-end data pipeline using Docker, Airflow, dbt, and Postgres. Docker Compose allowed us to emulate a production-like environment with isolated services for our database, ETL orchestrator, and transformation tool. Airflow handled the scheduling and sequencing of tasks to extract API data into Postgres, while dbt handled the SQL transformations to turn raw data into useful tables. By setting up automated tests and role-based access, we also ensured data quality and secure consumption of the data. The result is a robust pipeline that takes data from an API all the way to a BI dashboard, automatically and reliably.</p><p>This setup can be extended or modified for real-world scenarios — for instance, pointing to a real API, adding more models and tests, or hooking into a cloud data warehouse. But the principles remain the same: decouple the components, use the right tool for each stage, and let orchestration glue it all together. Happy data engineering!</p><p>If you’ve gotten this far — thanks for reading through the entire walkthrough.</p><p>If you’d like to follow along, explore the code, or discuss how to implement similar pipelines for your organization’s workflows or backend systems, you can connect here:</p><p><strong>GitHub:</strong> <a href="https://github.com/thejasono">https://github.com/thejasono</a><br> <strong>LinkedIn:</strong> <a href="https://www.linkedin.com/in/thejasonowens/">https://www.linkedin.com/in/thejasonowens/</a><br> <strong>Email:</strong> thisisjasonowens@gmail.com<br> <em>(available for freelance, contract, full-time roles, and consultation)</em></p><p>#DataEngineering #Airflow #dbt #Docker #Postgres #ETL #AnalyticsEngineering #OpenSource #PortfolioProjects</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=34fbdef69cfd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding ETL]]></title>
            <link>https://medium.com/@thisisjasonowens/understanding-etl-b7e640155299?source=rss-ecbef8df7ad4------2</link>
            <guid isPermaLink="false">https://medium.com/p/b7e640155299</guid>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[Jason O.]]></dc:creator>
            <pubDate>Sun, 28 Sep 2025 00:04:39 GMT</pubDate>
            <atom:updated>2025-12-14T02:53:03.063Z</atom:updated>
            <content:encoded><![CDATA[<p>In order to challenge myself I will be releasing a series of articles over the next couple of months. The goal is to provide some documentation on what data engineering actually is, for both students and professionals, while also educating myself further.</p><p>This first post is about ETL.<br>ETL is the foundation of almost everything else in data engineering. Tools and frameworks evolve constantly, and countless vendors offer variations, but the principle never changes: data engineering is about moving data from one place to another and making it usable on the other side.</p><h3><strong>The five layers of the data journey</strong></h3><p>From experience, this simple process of moving and shaping data can be broken into five main layers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xes2zp8sljKMgt5T64jZeQ.png" /><figcaption>The five layers of the data journey — every pipeline touches some or all of these steps.</figcaption></figure><ol><li>Ingestion — pulling data in from sources like APIs, logs, or databases.</li><li>Storage — placing that data into a system where it can be managed and queried (e.g. Postgres, S3, data lakes).</li><li>Transformation — cleaning, typing, and reshaping it so it is structured and consistent.</li><li>Orchestration — automating the steps, handling schedules, dependencies, and failures.</li><li>Visualization — exposing the data to end-users through dashboards or reports.</li></ol><p>Most pipelines touch some or all of these layers. Some projects are ingestion-heavy, some focus on transformation, others on orchestration. But these five steps form the skeleton of the data journey.<br> ETL lives mainly in the first three layers — extract, transform, load — but it connects directly to orchestration and visualization to create end-to-end value.</p><h3><strong>What is ETL?</strong></h3><p>ETL stands for Extract, Transform, Load. It is the standard process for preparing data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*IN2Emtjmw6s-wlEwIYIonw.jpeg" /><figcaption>ETL turns raw data into structured, usable information.</figcaption></figure><ul><li>Extract — get the raw data from its source. This might be an API, a transactional database, or flat files. Typical issues include authentication, schema changes, or messy formats.</li><li>Transform — clean it, enforce types, remove duplicates, and join it with other datasets. This is where tools like SQL, Python, or dbt come in. The aim is to turn raw data into something trustworthy and ready for analysis.</li><li>Load — put it into the destination system. This might be a warehouse like Postgres, BigQuery, or Snowflake, where analysts or downstream applications can access it.</li></ul><p>Some modern teams flip the order into ELT (Extract, Load, Transform), because powerful warehouses can store first and process later. But whether ETL or ELT, the concept is the same: a repeatable path from raw data to structured data.<br> ETL is the plumbing. Without it, there are no dashboards, no reports, and no insights.</p><h3><strong>A Simple Example</strong></h3><p>To make this concrete, here’s a minimal example of ETL in Python:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/918/1*fcz9bMP7HesKW1kQHQuGnQ.jpeg" /><figcaption>A minimal Python ETL script — JSON API → transform → Postgres.</figcaption></figure><p>This example takes JSON data from a free API, trims it down to three fields, and loads it into a Postgres table called posts.<br> This isn’t production-ready — there’s no error handling or schema enforcement — but the three stages are clear: Extract → Transform → Load.</p><p>For full instructions on how to run this locally, please read the README.md at:<br> <a href="https://github.com/thejasono/data-engineering-portfolio/tree/main/1.%20basic_etl_pipeline">https://github.com/thejasono/data-engineering portfolio/tree/main/1.%20basic_etl_pipeline</a></p><h3><strong>Final Lesson — Setting up an example pipeline</strong></h3><p>Let’s sketch a simple end-to-end pipeline that touches all five layers of the data journey:</p><ul><li>Source (API): a public API provides the raw input data.</li><li>Storage (Postgres): the raw data is landed in Postgres, giving a central, queryable store.</li><li>Transformation (dbt): dbt applies SQL transformations, building structured staging and marts layers.</li><li>Visualization (Power BI): analysts explore the cleaned data through interactive dashboards.</li><li>Orchestration (Airflow): Airflow ties it all together, scheduling runs, managing dependencies, and monitoring for errors.</li></ul><p>Diagrammatically, this looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*JAYqdKxY3TZZuyGborCo7w.jpeg" /><figcaption>API → Postgres → dbt → Power BI (under Airflow)</figcaption></figure><p>In the next post I’ll show a fully fleshed-out example of how this pipeline works in practice, breaking it down step by step and showing how each tool connects to form a complete workflow.</p><h3><strong>Closing</strong></h3><p>ETL is the backbone of data engineering. If you’d like to follow along, explore the code, or discuss how to implement similar pipelines for your organization&#39;s workflows and backend systems, you can connect here:</p><ul><li>GitHub — <a href="https://github.com/thejasono">https://github.com/thejasono</a></li><li>LinkedIn — <a href="https://www.linkedin.com/in/thejasonowens/">https://www.linkedin.com/in/thejasonowens/</a></li><li>Email — thisisjasonowens@gmail.com (available for freelance, contract, full-time roles, and consultation)</li></ul><p>Link to the example project/code snippet used — <a href="https://github.com/thejasono/data-engineering-portfolio/tree/main/1.%20basic_etl_pipeline">https://github.com/thejasono/data-engineering-portfolio/tree/main/1.%20basic_etl_pipeline</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b7e640155299" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>