<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by George Zefkilis on Medium]]></title>
        <description><![CDATA[Stories by George Zefkilis on Medium]]></description>
        <link>https://medium.com/@georgioszefkilis?source=rss-8c7ffca79196------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*fF-jH8breY1SKJs3GGNocg.jpeg</url>
            <title>Stories by George Zefkilis on Medium</title>
            <link>https://medium.com/@georgioszefkilis?source=rss-8c7ffca79196------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 08:58:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@georgioszefkilis/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Query the Stream in Practice: A Local Streamhouse with Fluss, Flink and Iceberg]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://blog.dataengineerthings.org/query-the-stream-in-practice-a-local-streamhouse-with-fluss-flink-and-iceberg-95763e4580c8?source=rss-8c7ffca79196------2"><img src="https://cdn-images-1.medium.com/max/1600/1*-KDluqf_l4UaQXXQ3pK9Ow.png" width="1600"></a></p><p class="medium-feed-snippet">In my previous article, An Introduction to the Streamhouse Pattern, I covered what the streamhouse pattern is and how Apache Fluss makes&#x2026;</p><p class="medium-feed-link"><a href="https://blog.dataengineerthings.org/query-the-stream-in-practice-a-local-streamhouse-with-fluss-flink-and-iceberg-95763e4580c8?source=rss-8c7ffca79196------2">Continue reading on Data Engineer Things »</a></p></div>]]></description>
            <link>https://blog.dataengineerthings.org/query-the-stream-in-practice-a-local-streamhouse-with-fluss-flink-and-iceberg-95763e4580c8?source=rss-8c7ffca79196------2</link>
            <guid isPermaLink="false">https://medium.com/p/95763e4580c8</guid>
            <category><![CDATA[apache-flink]]></category>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[apache-fluss]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <dc:creator><![CDATA[George Zefkilis]]></dc:creator>
            <pubDate>Tue, 06 Oct 2026 23:12:09 GMT</pubDate>
            <atom:updated>2026-10-06T23:12:09.269Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Query the Stream: An Introduction to the Streamhouse Pattern]]></title>
            <link>https://blog.dataengineerthings.org/query-the-stream-an-introduction-to-the-streamhouse-pattern-25f1afc1e961?source=rss-8c7ffca79196------2</link>
            <guid isPermaLink="false">https://medium.com/p/25f1afc1e961</guid>
            <category><![CDATA[streamhouse]]></category>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[apache-flink]]></category>
            <category><![CDATA[apache-fluss]]></category>
            <dc:creator><![CDATA[George Zefkilis]]></dc:creator>
            <pubDate>Mon, 31 Aug 2026 20:48:47 GMT</pubDate>
            <atom:updated>2026-08-31T20:48:47.714Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/236/1*vnEpMiGjAe8GhyNE0WeeHQ.jpeg" /><figcaption>Image taken from <a href="https://www.pinterest.com/tharngalhcr/house-near-water-stream/">here</a></figcaption></figure><p>Over the years, a few patterns have become the default answers for building data pipelines most notably Lambda and Kappa. In a nutshell, Lambda runs a streaming and a batch layer side by side while Kappa keeps only the stream. Both work, both have sharp edges (I’ve written end-to-end tutorials for each <a href="https://blog.dataengineerthings.org/kappa-architecture-in-action-from-sensors-to-dashboards-with-kafka-spark-streaming-starrocks-80492a509e1a">here</a> and <a href="https://blog.dataengineerthings.org/lambda-architecture-in-action-from-sensors-to-dashboards-with-kafka-iceberg-nessie-starrocks-3ae9a4c1c605">here</a>).</p><p>Recently, I ran into a newer idea:</p><blockquote><strong><em>…………………….The Streamhouse………………………..</em></strong></blockquote><p>Today, most of us store the same data twice (once as a stream for machines to react to, once as tables for people to analyze) and spend real effort keeping the two in sync. The streamhouse’s bet is that this split is artificial one storage layer where every table is subscribable like a stream and every stream is queryable like a table. Point SQL at it and you’re reading the last three seconds and the last three years in the same query.</p><p>That’s not a feature of one product it’s an architectural pattern. This post covers the pattern, what a stream house actually is, how it improves on the Kappa architecture it descends from, how it works underneath and where it sits among the systems competing for this space.</p><h3><strong>What is Streamhouse</strong></h3><p>I first heard the term at a <a href="https://www.ververica.com/"><strong>Ververica</strong></a> event (the company founded by the original creators of <a href="https://flink.apache.org/">Apache Flink</a>). <em>Streamhouse</em> is short for <em>streaming lakehouse</em>, and the name is the architecture. Take the lakehouse (open table formats on cheap object storage, one copy of the data, any engine can query it) and extend it downward in time, so the same tables also cover the data that arrived milliseconds ago.</p><p>Two principles make that possible.</p><p><strong>First, the stream/table duality becomes a storage guarantee, not a processing trick.</strong></p><p>Stream processors have long treated a stream as a table’s changelog and a table as a stream’s snapshot but we’ve always stored them separately, the changelog in a streaming platform, the snapshot in the lake. A streamhouse makes the duality physical. Writing to a table produces a subscribable changelog, and subscribing to a stream is just reading a table from now onward.</p><p><strong>Second, tiered storage keeps that layer honest about cost.</strong></p><p>Fresh data lives in a low-latency tier built for sub-second reads and writes. As it ages, it’s continuously compacted into open lakehouse tables. Crucially, this isn’t an ETL job you build it’s a property of the storage layer, and both tiers present as <em>one logical table</em>. Queries that span fresh and historical data get answered across both tiers transparently.</p><p>If Kappa’s thesis was <em>“the stream is the source of truth,”</em> the streamhouse is Kappa with the storage it always needed (Kappa on steroids). The log is no longer an opaque pipe you replay through a processor to get answers it’s a table you query directly, at any point in its history.</p><h3><strong>Why Not Just Kappa?</strong></h3><p>“<em>The storage it always needed</em>” is a strong claim, so let’s be specific. Kappa’s thesis was right. Its weakness was never the idea it was that the only durable stream storage we had was an integration bus, and an integration bus makes a poor analytical store in three specific ways.</p><p><strong>The log is opaque.</strong> A Kafka topic is row-oriented bytes. You can’t project columns, can’t filter pushdown, can’t point a query engine at it. Every analytical question means replaying the log through a processor and materializing the answer <em>somewhere else (</em>and that somewhere else is where the trouble starts). In a streamhouse, the fresh tier is queryable storage the stream itself answers scans, no materialization detour.</p><p><strong>Retention forces the split.</strong> Keeping full history in the log is unaffordable, so history migrates to a warehouse and now the same data lives in two systems with two schemas, and you’re paying the sync tax from the top of this post. In a streamhouse, aging out isn’t an export to a different system data tiers down <em>within the same logical table</em>.</p><p><strong>Reprocessing means replay.</strong> Change your aggregation logic and Kappa’s answer to recomputing last year is streaming a year of log back through your pipeline at stream speed. In a streamhouse, last year is already open-format lake tables. Batch engines read it directly at lake economics, and a query needing the fresh tail gets both tiers stitched into one result.</p><p>None of this is a knock on Kafka, which is still excellent at moving events between systems. Kappa’s mistake was drafting the integration bus into a second job being the storage analytics runs on. What a layer actually shaped for that job looks like is next.</p><h3><strong>How It Works</strong></h3><p>The architecture is four boxes. A <strong>hot tier</strong>, a <strong>tiering service</strong> moving data out of it, a <strong>lake</strong> tier underneath, and a <strong>catalog</strong> that presents the two as one table. In <a href="https://fluss.apache.org/docs/">Fluss</a> (the reference implementation of the pattern) they look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/1*MZg9spHBj74Ozy-baM8t9A.png" /><figcaption>Each box is a role the architecture needs the subtitle underneath is how Fluss fills it.</figcaption></figure><p><strong>The write path.</strong> Events land in the <strong>hot</strong> tier. Low-latency streaming storage built for sub-second reads and writes. In Fluss this is a cluster of tablet servers holding two table types <strong>Log Tables</strong> (append-only) and <strong>PrimaryKey Tables</strong> (upsertable). The table type decides which stores are used underneath.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/1*u9E_4l5laz9_pVBlTlG2qg.png" /><figcaption>The write path. One row, from acknowledgment to lake file. No pipeline in between</figcaption></figure><blockquote>A Fluss cluster runs two kinds of process. The tablet servers shown here, and a CoordinatorServer that tracks cluster metadata and decides which server holds which piece of a table. It sets the write path up but doesn’t sit on it, so it’s left out. The <a href="https://fluss.apache.org/docs/concepts/architecture/">Fluss architecture docs</a> have the complete picture.</blockquote><p>Writes hit the LogStore first ①: an append-only WAL, replicated across an in-sync replica set, Kafka-style. For a PrimaryKey Table the row is then applied to a KvStore ② that serves upserts and point lookups. That KvStore is a derived structure, it isn’t replicated itself, but it can be rebuilt from the log after a failure.</p><p>From the moment the write is acknowledged, the row is both queryable and subscribable ③. The tier stores rows in a streaming Arrow format tuned for latency rather than compression, and holds days, not years.</p><p>Behind it, the <strong>tiering service</strong> ④ continuously compacts aging data into Parquet inside an <strong>open table forma</strong>t (Iceberg, Paimon or Lance) on object storage, where retention stops being a cost problem⑤. This is the crucial non-event. No ETL job you wrote, no pipeline to monitor for drift, tiering is a property of the storage layer.</p><p><strong>The read path.</strong> The catalog is what makes the two tiers one table it holds the metadata for both ①, so an engine asking for orders sees a single logical table, not a hot half and a cold half. Fluss keeps that metadata in sync with the lake catalog as it compacts, which is why Spark, Trino or StarRocks can point at the cold tier and read it directly, with no Fluss in the path.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/743/1*RD_agZhUeviukyCOcJGgIw.png" /><figcaption>One query, two tiers, no seam</figcaption></figure><blockquote>Reading across both tiers is Flink-only today. A Spark union read for Log Tables is in review for 1.0, and native union reads for Spark, Trino and StarRocks are on the project’s 2026 roadmap.</blockquote><p>When a query spans the boundary, the planner splits the scan ② the historical range to the lake tables ④, the fresh tail to the hot tier ③ and the union read merges the two into one answer ⑤. Ask for last year, you’re reading Parquet at lake economics. Ask for the last ten seconds, you’re reading Arrow at millisecond latency. Ask for both, and you never notice the seam.</p><p>That’s the whole trick, two physical tiers, one logical table, and a catalog honest enough to hide the boundary.</p><h3><strong>The Landscape</strong></h3><p>Fluss is new enough that my first question was what else can fill the four boxes above, and how those options differ. The table below is how I summarised it.</p><p>One thing sits outside the comparison, how data arrives. Producers can write to Fluss directly, but in practice most teams land data through Flink from Kafka or CDC, because that’s where it already is. Ingestion looks broadly similar across these rows the table starts once the data is in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R0dreVmokL-VKpdYLU65CA.png" /><figcaption>Summary of technologies</figcaption></figure><p>Row one is where most of us are today, and that second copy is the whole argument of this post. Fluss’s contribution is the fast-tier column the streaming layer becomes something you can run SQL against, so the analytical copy stops being a separate system to keep in step. Kafka doesn’t disappear from the architecture it stops being what your analytics read from. <a href="https://risingwave.com/use-cases/use-case-continuous-analytics/">RisingWave</a> arrives at the same two tiers as one database that owns the compute and the serving interface too less to assemble, correspondingly less to swap. <a href="https://www.databricks.com/">Databricks</a> is the outlier no fast tier at all, reading the change feed of the Delta tables it already serves, which is what puts its floor at seconds rather than milliseconds.</p><p>Worth separating out that materialising a Kafka topic into an Iceberg table looks like tiering, but it produces a second copy with its own refresh cadence. The log stays opaque, and you’re back in row one.</p><p>Which row fits depends on the workload.</p><h3><strong>When It Makes Sense</strong></h3><p>Personally, I’m sceptical when a stakeholder asks for real-time data. These systems are hard to build and harder to keep running, and most requirements I’ve met are served perfectly well by a scheduled batch or micro-batch job. But some genuinely aren’t.</p><p>The ones that aren’t share a shape. The decision has to be made in under a second, and making it well requires history. Fraud scoring, personalisation, anomaly detection fast path and slow path on the same data, which is the pair you’re currently paying to keep in step.</p><p>The other half of the choice is context. Whether the streamhouse is the whole system or a part of one. If it’s the whole system, an integrated engine is less to run (one thing instead of two). But most of us are adding this to something a lake other teams already query, a catalog someone else chose, engines we don’t control. Keeping storage and compute separate is what lets it slot into that, because each piece can match what’s already there.</p><p>And where none of this fits. If minutes of freshness is enough, a lakehouse alone meets it and the hot tier is something you’d pay to run and not use. If your fast and analytical workloads touch genuinely different data, two systems is the honest architecture. At low volume, a single Postgres does both jobs.</p><h3>Conclusion</h3><p>None of this is free. Two physical tiers is two things to operate, and Fluss is young (incubating), with Flink the only engine that can query across both tiers today.</p><p>Still, being able to query the stream is the part that excites me. I stand by the scepticism above that plenty of real-time requirements turn out to be fine on a schedule. But the demand keeps coming, and an architecture that stops making me keep two copies of the same data in step is one I would consider. Though, after enough experimentation to know what it costs to run.</p><p>I hope you found this guide useful and that it inspires you to experiment further with data pipeline architectures.</p><p>Your feedback is always appreciated and helps improve these posts.<br>📦 Check out the code in the <a href="https://github.com/georgezefko">GitHub repository</a><br>🤝 Feel free to connect with me on <a href="http://www.linkedin.com/in/georgioszefkilis">LinkedIn</a></p><h3>References</h3><ol><li><a href="https://fluss.apache.org/docs/">What is Fluss</a></li><li><a href="https://fluss.apache.org/docs/concepts/architecture/">Fluss Architecture</a></li><li><a href="https://substack.com/home/post/p-184309369">The seven sins of streaming</a></li><li>Th<a href="https://www.ververica.com/product/streamhouse">e streamhouse — Ververica</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=25f1afc1e961" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/query-the-stream-an-introduction-to-the-streamhouse-pattern-25f1afc1e961">Query the Stream: An Introduction to the Streamhouse Pattern</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a CDC pipeline, part 3 : From Kafka events to an analytical event log]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://blog.dataengineerthings.org/building-a-cdc-pipeline-part-3-from-kafka-events-to-an-analytical-event-log-f548f3b5dd66?source=rss-8c7ffca79196------2"><img src="https://cdn-images-1.medium.com/max/1395/1*W1oEfz4eafYk_ziNG-yxuA.png" width="1395"></a></p><p class="medium-feed-snippet">In the first 2 parts of this series (part 1, part 2) I dived into the technical details of PostgreSQL WAL and Debezium. Postgres is a&#x2026;</p><p class="medium-feed-link"><a href="https://blog.dataengineerthings.org/building-a-cdc-pipeline-part-3-from-kafka-events-to-an-analytical-event-log-f548f3b5dd66?source=rss-8c7ffca79196------2">Continue reading on Data Engineer Things »</a></p></div>]]></description>
            <link>https://blog.dataengineerthings.org/building-a-cdc-pipeline-part-3-from-kafka-events-to-an-analytical-event-log-f548f3b5dd66?source=rss-8c7ffca79196------2</link>
            <guid isPermaLink="false">https://medium.com/p/f548f3b5dd66</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[George Zefkilis]]></dc:creator>
            <pubDate>Thu, 04 Jun 2026 03:52:54 GMT</pubDate>
            <atom:updated>2026-06-04T03:52:54.892Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Building a CDC pipeline, part 2: Debezium Internals]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://blog.dataengineerthings.org/building-a-cdc-pipeline-part-2-debezium-internals-5f44b6d4e663?source=rss-8c7ffca79196------2"><img src="https://cdn-images-1.medium.com/max/1105/1*rScJNRXsb77g9QoeQpnA9A.png" width="1105"></a></p><p class="medium-feed-snippet">Introduction</p><p class="medium-feed-link"><a href="https://blog.dataengineerthings.org/building-a-cdc-pipeline-part-2-debezium-internals-5f44b6d4e663?source=rss-8c7ffca79196------2">Continue reading on Data Engineer Things »</a></p></div>]]></description>
            <link>https://blog.dataengineerthings.org/building-a-cdc-pipeline-part-2-debezium-internals-5f44b6d4e663?source=rss-8c7ffca79196------2</link>
            <guid isPermaLink="false">https://medium.com/p/5f44b6d4e663</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[debezium]]></category>
            <category><![CDATA[kafka]]></category>
            <dc:creator><![CDATA[George Zefkilis]]></dc:creator>
            <pubDate>Thu, 07 May 2026 19:14:43 GMT</pubDate>
            <atom:updated>2026-05-07T19:14:43.917Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Building a CDC Pipeline, Part 1: PostgreSQL WAL Internals]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://blog.dataengineerthings.org/building-a-cdc-pipeline-part-1-postgresql-wal-internals-b5d225031dc5?source=rss-8c7ffca79196------2"><img src="https://cdn-images-1.medium.com/max/691/1*lx2Sh-aMkytZpWxpq5wmTw.png" width="691"></a></p><p class="medium-feed-snippet">Introduction</p><p class="medium-feed-link"><a href="https://blog.dataengineerthings.org/building-a-cdc-pipeline-part-1-postgresql-wal-internals-b5d225031dc5?source=rss-8c7ffca79196------2">Continue reading on Data Engineer Things »</a></p></div>]]></description>
            <link>https://blog.dataengineerthings.org/building-a-cdc-pipeline-part-1-postgresql-wal-internals-b5d225031dc5?source=rss-8c7ffca79196------2</link>
            <guid isPermaLink="false">https://medium.com/p/b5d225031dc5</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[change-data-capture]]></category>
            <category><![CDATA[postgres]]></category>
            <dc:creator><![CDATA[George Zefkilis]]></dc:creator>
            <pubDate>Sat, 25 Apr 2026 23:02:28 GMT</pubDate>
            <atom:updated>2026-04-25T23:02:28.548Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[From MapReduce to the Modern Data Lakehouse: A Complete Technical Journey]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://blog.dataengineerthings.org/from-mapreduce-to-the-modern-data-lakehouse-a-complete-technical-journey-b6471b6c00d4?source=rss-8c7ffca79196------2"><img src="https://cdn-images-1.medium.com/max/691/1*5M894iBQ3XIHz6OHHBrV1g.png" width="691"></a></p><p class="medium-feed-snippet">Hello again, fellow tech enthusiasts.</p><p class="medium-feed-link"><a href="https://blog.dataengineerthings.org/from-mapreduce-to-the-modern-data-lakehouse-a-complete-technical-journey-b6471b6c00d4?source=rss-8c7ffca79196------2">Continue reading on Data Engineer Things »</a></p></div>]]></description>
            <link>https://blog.dataengineerthings.org/from-mapreduce-to-the-modern-data-lakehouse-a-complete-technical-journey-b6471b6c00d4?source=rss-8c7ffca79196------2</link>
            <guid isPermaLink="false">https://medium.com/p/b6471b6c00d4</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[delta-lake]]></category>
            <category><![CDATA[databricks]]></category>
            <dc:creator><![CDATA[George Zefkilis]]></dc:creator>
            <pubDate>Fri, 27 Mar 2026 20:28:37 GMT</pubDate>
            <atom:updated>2026-03-27T20:28:37.343Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Building a Personal Finance Management App: Deploying to Production with Vercel, Render, and Neon —…]]></title>
            <link>https://levelup.gitconnected.com/building-a-personal-finance-management-app-deploying-to-production-with-vercel-render-and-neon-9ddeabca9aa2?source=rss-8c7ffca79196------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ddeabca9aa2</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[George Zefkilis]]></dc:creator>
            <pubDate>Wed, 18 Feb 2026 18:20:40 GMT</pubDate>
            <atom:updated>2026-02-18T18:20:40.754Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Building a Personal Finance Management App: Deploying to Production with Vercel, Render, and Neon — A Practical Guide</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eohQGwuwBxeQ7cI0kKJeGw.png" /><figcaption>Image taken from <a href="https://www.wissen.com/blog/lets-understand-the-difference-between-continuous-deployment-and-continuous-delivery">here</a></figcaption></figure><p>Hello again fellow tech enthusiasts.</p><p>A couple of years ago, I started building a personal financial tracker to manage my expenses and track my networth. It began as a small side project, but I’ve used it as a learning opportunity to design, build, and operate an application end to end from database design to production deployment.</p><p>So far, I’ve published three posts about the journey. They cover setting up the <a href="https://medium.com/towards-data-engineering/building-a-personal-finance-management-app-database-setup-with-postgresql-and-docker-5075e283303e">database and migrations</a>, <a href="https://medium.com/towards-data-engineering/building-a-personal-finance-management-app-integrating-rest-api-node-js-7a0f0f27bd4e">building a RESTful API</a>, <a href="https://medium.com/gitconnected/building-a-personal-finance-management-app-secure-api-authentication-a-practical-guide-e9d936b6982b">applying basic security with JSON Web Tokens (JWT)</a>, and running everything locally using Docker and Docker Compose.</p><p>In this post, I will move beyond local development and I’ll walk through a <strong>practical CI/CD setup</strong>, showing how to deploy the app using managed services: <a href="https://neon.com/">Neon</a> for the database, <a href="https://render.com/">Render</a> for the backend, and <a href="https://vercel.com/">Vercel</a> for the frontend.</p><blockquote>By “managed services” I mean platforms that abstract away infrastructure management, no VM provisioning, no manual patching, and no load balancer configuration.</blockquote><p>The goal is to achieve a fully working production setup at zero cost using free tiers. While this comes with trade-offs like cold starts and usage limits, it’s more than sufficient for a side project and the architectural patterns are realistic and reusable beyond this application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o-Z67HmTdUOn3fD68u8MUA.png" /><figcaption>This is how Cashflow tab of the app looks like.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VyVmEH4toLTheX_zj4rL-g.png" /><figcaption>This is how Networth tab of the app looks like</figcaption></figure><h3>Repository Structure</h3><p>Before talking about CI/CD, it helps to understand how the project is organized.</p><p>The entire application lives in a single monorepo. It consists of three main parts:</p><ul><li><strong>Backend</strong> — Node.js + Express API</li><li><strong>Frontend</strong> — React application</li><li><strong>Database</strong> — SQL schema and migrations</li></ul><p>The structure looks like this:</p><pre>finance-tracker/<br>│<br>├── backend/          # Node.js + Express API<br>│   └── Dockerfile<br>│<br>├── frontend/         # React application<br>│   └── Dockerfile<br>│<br>├── database/ <br>│   └── Dockerfile    # SQL migrations &amp; schema<br>│<br>├── docker-compose.yml<br>├── .github/workflows/<br>└── package.json      # semantic-release config</pre><p>The database folder does not contain a running database in production. Instead, it holds schema definitions and migration scripts. The Dockerfile is used locally to apply migrations through Docker Compose. In production, the database itself is hosted on Neon.</p><p>Keeping everything in one <a href="https://github.com/georgezefko/finance-tracker">repository</a> makes cross-stack changes atomic and enables unified versioning, but it also shapes the CI/CD approach especially how releases and versioning are handled.</p><h3>Versioning a Monorepo with Semantic Release</h3><p>Because the backend, frontend, and migrations are released together from a single monorepo, versioning is handled at the repository level: <strong>one repo, one version</strong>.</p><p>I use <strong>Semantic Versioning</strong>, automated through semantic-release and driven by Conventional Commits (covered in detail in this <a href="https://medium.com/gitconnected/devops-for-data-engineers-part-1-e70f7bcbabb2">post</a>).</p><p>Commit messages follow a structured format:</p><pre>feat(backend): add recurring transactions support<br>fix(frontend): correct date parsing in reports<br>chore(dependencies): bump dependencies</pre><p>When changes are merged into master, a GitHub Actions workflow runs semantic-release. It analyzes the commit history and determines the next version automatically:</p><ul><li>feat → <strong>minor</strong> bump (1.3.0 → 1.4.0)</li><li>fix → <strong>patch</strong> bump (1.3.0 → 1.3.1)</li><li>BREAKING CHANGE or feat! → <strong>major</strong> bump (1.x.x → 2.0.0)</li></ul><p>It then creates a Git tag like:</p><pre>v1.0.11</pre><p>That tag represents a snapshot of the <em>entire system</em> at that point in time. Along with each version, a CHANGELOG.md file is generated automatically, containing the changes included in every release:</p><pre>#### [1.0.11] (...) (2026-01-29)<br><br>### Bug Fixes<br>* **migrations:** corrected drop duplicates (...)</pre><p>Since backend, frontend, and database changes are tightly coupled in this project, versioning them separately would add unnecessary complexity.</p><p>With a single version:</p><ul><li>Backend, db migrations and frontend Docker images are tagged with the same version (e.g., backend:v1.0.11, frontend:v1.0.11)</li><li>Database migrations correspond to that exact release</li><li>Each Git tag maps to a deterministic system state</li><li>Debugging and rollbacks become straightforward</li></ul><p>If something breaks in production on v1.0.11, I immediately know which backend code, frontend build, and migrations were deployed together.</p><blockquote>Note : For larger systems where services evolve independently, separate versioning (with tools like Lerna, Nx, or Turborepo) makes more sense. But for this project, a shared version keeps the release process simple and predictable.</blockquote><h3>CI/CD: From Validation to Production</h3><p>With automated versioning in place, the next step is defining how changes move safely from a developer laptop to production.</p><p>The pipeline is split into three stages:</p><ol><li><strong>Continuous Integration (CI)</strong> — validate everything</li><li><strong>Release</strong> — version and build artifacts</li><li><strong>Production Migrations</strong> — apply schema changes safely</li></ol><p>Each stage has a clear responsibility.</p><h3>CI Workflow: Build and Validate Everything</h3><p>The <em>CI.yaml </em>workflow runs on every push and pull request. It ensures the backend and frontend build successfully and that migrations apply cleanly before anything is merged into master</p><p>It consists of three jobs:</p><ul><li><strong>Backend job</strong>: installs dependencies and builds the backend.</li><li><strong>Frontend job</strong>: installs dependencies and builds the frontend.</li><li><strong>Migrations job</strong>: spins up a temporary Postgres container and runs migrations against a clean database.</li></ul><p>This provides fast feedback and prevents unhealthy changes from reaching master. In larger systems, you’d typically also add tests, linting/type checks, and security scanning (you can check the code <a href="https://github.com/georgezefko/finance-tracker/blob/master/.github/workflows/ci.yaml">here</a>).</p><h3>Release Workflow: Version and Build Docker Images</h3><p>After changes are merged into master, <em>Release.yaml</em> handles packaging and versioning ( CI already validated the system).</p><p>It performs three main steps:</p><ul><li>runs <em>semantic-release</em> to calculate and tag the next version,</li><li>builds Docker images for backend, frontend, and migrations,</li><li>pushes each image to Docker Hub with both latest and the version tag (for example v1.0.11).</li></ul><p>All images share the same version, keeping environments aligned. Each Git tag now corresponds to immutable Docker images in Docker Hub, so production can deploy an explicit version tag instead of relying on latest.</p><p>At this point, we have reproducible, versioned artifacts ready to deploy (check the code <a href="https://github.com/georgezefko/finance-tracker/blob/master/.github/workflows/release.yaml">here</a>)</p><h3>Running Production Migrations</h3><p>Production migrations are handled separately from the release step.</p><p>Instead of running automatically, they’re executed via a <strong>manually triggered</strong> GitHub Action <a href="https://github.com/georgezefko/finance-tracker/blob/master/.github/workflows/prod-migrations.yaml">workflow</a> that:</p><ul><li>Pulls a specific migrations Docker image (either latest or a version tag)</li><li>Connects to the production Neon database</li><li>Runs the migrations using secrets stored in GitHub Actions</li></ul><p>This makes schema changes <strong>explicit</strong>, <strong>repeatable</strong>, and <strong>decoupled</strong> from application deployment. Since not every release includes schema changes, running migrations automatically on every deployment would add unnecessary risk.</p><p>Migrations follow a backward-compatible approach to minimize disruption. There can be a short window where new backend code is deployed before a migration runs, but changes are small and controlled to keep impact minimal.</p><h3>Configuring the Managed Services</h3><p>With reproducible, versioned artifacts ready, the final step is connecting them to managed infrastructure so the system can run in production.</p><p>The production setup uses free tiers of:</p><ul><li><strong>Neon</strong> for PostgreSQL</li><li><strong>Render</strong> for the backend API</li><li><strong>Vercel</strong> for the frontend</li></ul><p>Render pulls the backend Docker image produced by the GitHub Actions release workflow, and the migrations image is used to apply schema changes against Neon. Vercel deploys the frontend directly from the master branch instead of consuming a Docker image.</p><h3>Neon: Managed Postgres</h3><p>Neon provides serverless PostgreSQL with a generous free tier, built-in branching, and zero infrastructure management.</p><p>Since both the backend container and the migrations container need to talk to the same database, Neon becomes the central piece of the production setup.</p><h4>Creating the Project</h4><p>To get started, I logged into the <a href="https://console.neon.tech/">Neon console</a> using GitHub and created a new project called finance-tracker.</p><p>When creating the project, you select:</p><ul><li>Project name</li><li>Database name (e.g., tracker or the default neondb)</li><li>Region</li></ul><p>Choosing a region close to your backend is important to minimize latency. In my case, that meant selecting an EU region to match where Render runs.</p><p>After clicking <strong>Create Project</strong>, Neon automatically provisions:</p><ul><li>A project</li><li>A primary branch</li><li>A PostgreSQL database</li><li>A database role with credentials</li></ul><p>Within seconds, you have a production-ready PostgreSQL instance running in the cloud without provisioning VMs, configuring storage, or setting up replication manually.</p><blockquote>Under the hood, Neon uses a proxy-based architecture with strict separation of compute and storage. Instead of connecting to a fixed database server, clients connect to a Neon endpoint (proxy), which routes connections to stateless compute nodes backed by a distributed, S3-based storage layer. This design enables serverless autoscaling, scale-to-zero, connection pooling, and branching while remaining fully compatible with standard PostgreSQL clients.</blockquote><h4>Connection Details and Environment Variables</h4><p>Once the project is created, Neon provides several ways to get the credentials, I chose the environment variable format which exposes values as:</p><pre>PGHOST=...<br>PGDATABASE=...<br>PGUSER=...<br>PGPASSWORD=...<br>ENDPOINT_ID=...</pre><p>Neon requires SSL connections, so the backend database client is configured accordingly. This is an important detail without enabling SSL, connection attempts will fail.</p><p>Production credentials are never committed to the repository. Instead, they are stored securely as secrets. In GitHub Actions, the Neon credentials are saved as repository secrets and mapped to environment variables used by the migrations workflow:</p><pre>PROD_POSTGRES_HOST<br>PROD_POSTGRES_PORT<br>PROD_POSTGRES_USER<br>PROD_POSTGRES_PASSWORD<br>PROD_POSTGRES_DB</pre><p>Those same values are used by the backend running on Render. This keeps schema changes and runtime traffic pointed at the same database, avoids configuration drift, and keeps secrets out of Git.</p><p>At this point, the database layer is fully managed, securely configured, and integrated with both the application runtime and the deployment pipeline.</p><h3>Render: Backend API from Docker Image</h3><p>With the database layer fully managed and reachable in production, the next step is exposing the application logic through a publicly accessible backend service.</p><p>For that, I selected <a href="https://render.com/">Render</a> because it can run containerized web services with minimal operational overhead no VM provisioning or manual networking while still letting me deploy the exact Docker image I use locally.</p><h4>Deploying the Backend Service</h4><p>After logging into the Render dashboard (via GitHub), I created a new <strong>Web Service</strong> and chose <strong>Deploy an existing Docker image</strong> (instead of building from a repo). The image is already built and versioned by the GitHub Actions release workflow Render’s job is simply to run it.</p><p>For the image, I entered:</p><pre>DOCKER_HUB_USERNAME/finance-backend:latest<br># eg zefko/finance-backend:latest</pre><p>Render pulls the image directly from Docker Hub. If the repository is public, no additional credentials are required. For private repositories, Docker Hub authentication can be configured in the service settings.</p><p>This setup keeps responsibilities clean:</p><ul><li>CI builds and tags the image</li><li>Docker Hub stores it</li><li>Render runs it</li></ul><p>There is no duplication of build logic in production.</p><blockquote>Note: While using latest is convenient, production deployments should ideally be pinned to a specific version tag (for example, v1.0.11). This guarantees deterministic releases and simplifies rollbacks. Render allows you to configure a specific image tag in the service settings.</blockquote><h4>Service Configuration</h4><p>During setup, I named the service <em>finance-backend</em>, selected a region close to Neon to reduce latency, and used the free instance type.</p><p>Because the container already defines its startup command (CMD) and exposed port, I left the start command empty and let Render use the image defaults. The key requirement is that the app listens on the port provided at runtime Render injects a PORT environment variable, and the Express app reads process.env.PORT.</p><p>Render also restarts the container automatically if it crashes, which provides basic resilience even on the free tier.</p><h4>Configuring Environment Variables</h4><p>At this stage, the container is ready to run but it still needs configuration.</p><p>The Docker image does not contain any production credentials. Instead, these are injected through Render’s <strong>Environment</strong> settings.</p><p>Inside the service configuration, I defined:</p><pre>NODE_ENV=production<br>PORT=8000<br>POSTGRES_HOST<br>POSTGRES_PORT<br>POSTGRES_USER<br>POSTGRES_PASSWORD<br>POSTGRES_DB<br>JWT_SECRET</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WlB2RZ6GC2DGuuBdl9-7Lw.png" /></figure><p>These mirror the same production database credentials used by the migrations workflow in GitHub Actions. The result is <strong>build once, configure per environment</strong>: the same image can run in dev/staging/prod by swapping environment variables.</p><h4>First Deployment and Logs</h4><p>After creating the service, Render pulls the Docker image and starts the container automatically.</p><p>Render assigns a public URL to the service, typically in the form:</p><pre>https://finance-backend.onrender.com</pre><p>This URL can be used directly by the frontend or later mapped to a custom domain.</p><blockquote>On the free tier, services automatically spin down after a period of inactivity. When accessing the API after being idle, it may take 1–2 minutes for the container to start again. This “cold start” behavior is expected and is one of the trade-offs of running at zero cost. Paid plans eliminate this delay.</blockquote><p>During initial rollout, the <strong>Logs</strong> tab is the fastest way to confirm:</p><ul><li>the server booted successfully</li><li>the database connection works (including SSL requirements)</li><li>auth configuration is correct</li></ul><p>Since the image is built and validated in CI, failures here are usually environment/config issues (wrong variables, missing SSL, etc.) rather than build problems.</p><p>At this point the backend API is live and connected to the production Neon database. Next, the frontend is deployed on Vercel and configured to call this public API endpoint.</p><h3>Vercel: Frontend on Free Tier</h3><p>With the backend API live and publicly accessible, the final step is deploying the user interface and wiring it to the backend endpoint.</p><p>For the UI layer, I chose <a href="https://vercel.com/">Vercel</a> because it’s optimized for frontend deployments. It connects to GitHub, runs your build, and serves the result globally with HTTPS. For React-based applications, it works almost effortlessly.</p><h4>Connecting the Repository</h4><p>Inside the Vercel dashboard, I imported the GitHub repository and configured the project.</p><p>Since this is a monorepo, I set the project root to the <strong>frontend/ </strong>directory. That ensures Vercel runs the build process only for the React application and not the entire repository.</p><blockquote>NOTE: Even though the release workflow builds a frontend Docker image, I don’t use it on Vercel. Instead, Vercel pulls the frontend/ code directly and runs npm install + npm run build, which keeps the frontend deployment aligned with Vercel’s model.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qCdVx-U6DH7fip95MNIQTw.png" /><figcaption>Setting the root to frontend directory</figcaption></figure><p>Under <strong>Settings -&gt; Build and Deployment</strong>, I configured:</p><ul><li>Install command: npm install</li><li>Build command: npm run build</li><li>Output directory: build (or dist, depending on your tooling)</li></ul><p>Once configured, Vercel automatically builds and deploys every time changes are pushed to the master branch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k-l8hfdMWnztJzPwlh_Yqw.png" /><figcaption>Setting configuration in Vercel platform</figcaption></figure><h4>Configuring the API Base URL</h4><p>The frontend needs to know where the backend API lives. In code, I default to localhost (eg. <a href="http://localhost:8000.">http://localhost:8000</a>) for development:</p><pre>const RAW_BASE_URL =<br>  process.env.REACT_APP_BASE_URL || &#39;http://localhost:8000&#39;;</pre><p>In production, I override it via a Vercel environment variable:</p><pre>// YOU GET THIS URL FROM RENDER<br>REACT_APP_BASE_URL=https://your-backend-service.onrender.com <br></pre><p>For Create React App (and similar setups), variables prefixed with REACT_APP_ are injected <strong>at build time</strong>, so changing the value requires a new deployment to rebuild the static assets.</p><p>Because the frontend (Vercel) and backend (Render) are on different domains, the backend must allow cross-origin requests. I enabled CORS in Express to allow requests from the Vercel origin.</p><p>After deployment, Vercel provides a public URL (e.g. <a href="https://finance-tracker-green-six.vercel.app/">https://finance-tracker.vercel.app</a>) that serves the static frontend globally over HTTPS.</p><h3>How the Full Flow Comes Together</h3><p>With everything deployed and connected, here’s the end-to-end path a change takes from commit to production:</p><ul><li><strong>Push / pull request</strong> -&gt; triggers <strong>CI</strong>, which builds the backend and frontend and validates migrations</li><li><strong>Merge to master</strong> -&gt; triggers <strong>Release</strong>, which calculates the next semantic version, creates a Git tag, and builds/pushes versioned Docker images to Docker Hub</li><li><strong>Render</strong> -&gt; deploys the selected backend image (either latest or a pinned version tag)</li><li><strong>Vercel</strong> -&gt; rebuilds and deploys the frontend directly from the frontend/ code in the repo</li><li><strong>Frontend -&gt; Backend</strong> -&gt; calls the API using REACT_APP_BASE_URL configured in Vercel</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*wyWQbJP7SuAF9b4n7XNGKw.png" /><figcaption>End-to-end flow from commit to production runtime.</figcaption></figure><p>The result is a production setup built entirely on managed services: PostgreSQL on <strong>Neon</strong>, a containerized backend on <strong>Render</strong>, a Git-driven frontend on <strong>Vercel</strong>, and automation via <strong>GitHub Actions</strong> all tied together by a single semantic version.</p><p>Each Git tag represents a single, traceable system state (backend image, frontend build, and migrations). If something breaks in v1.0.11, you can identify exactly what shipped and roll back to a known-good release.</p><h3>Conclusion</h3><p>This post wasn’t about adding features. It was about operationalizing the system.</p><p>Starting from a working local project, we turned it into something that can be versioned, reproduced, and trusted in production. The infrastructure is managed. Releases are automated. Changes are traceable.</p><p>That shift from “it works on my machine” to “it runs reliably in the cloud” is what separates a side project from a production-ready application.</p><p><strong>Thanks for reading!</strong></p><p>I hope this gives you a practical baseline for experimenting with the operational side of building and deploying apps</p><p>Your feedback is always appreciated and helps improve these posts.<br>📦 Check out the code in the <a href="https://github.com/georgezefko/finance-tracker">GitHub repository</a><br>🤝 Feel free to connect with me on <a href="http://www.linkedin.com/in/georgioszefkilis">LinkedIn</a></p><h3>References</h3><ol><li><a href="https://neon.com/">Neon databases</a></li><li><a href="https://render.com/">Render</a></li><li><a href="https://vercel.com/">Vercel</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ddeabca9aa2" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/building-a-personal-finance-management-app-deploying-to-production-with-vercel-render-and-neon-9ddeabca9aa2">Building a Personal Finance Management App: Deploying to Production with Vercel, Render, and Neon —…</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Local Data Platform with Kubernetes and Terraform]]></title>
            <link>https://blog.dataengineerthings.org/building-a-local-data-platform-with-kubernetes-and-terraform-9547a4256a7f?source=rss-8c7ffca79196------2</link>
            <guid isPermaLink="false">https://medium.com/p/9547a4256a7f</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[terraform]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[platform-engineering]]></category>
            <dc:creator><![CDATA[George Zefkilis]]></dc:creator>
            <pubDate>Thu, 29 Jan 2026 15:42:46 GMT</pubDate>
            <atom:updated>2026-01-29T15:42:46.864Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*yPuRZkUH-CfvBzMijDpP9Q.jpeg" /><figcaption>Image taken from <a href="https://www.michaelpage.com.au/advice/job-description/technology/platform-engineer">here</a></figcaption></figure><p>The more time I spend working as a Data Engineer, the more I realize how much the role overlaps with that of a Data Platform Engineer. Beyond pipelines and transformations, we increasingly take ownership of infrastructure, tooling, and developer experience. With that in mind, this blog aims to provide a <strong>practical example</strong> of how core tools like Kubernetes, Terraform, and DevContainers can be combined to build a local data platform in a structured and maintainable way.</p><p>This post is a continuation of a previous <a href="https://medium.com/gitconnected/from-zero-to-gitops-deploying-mage-ai-on-local-kubernetes-with-argo-cd-25865a035571">article</a> , where I walked through setting up a local Kubernetes cluster and deploying applications manually using Argo CD. If you haven’t read that one yet, I recommend starting there, as some of the concepts and tools will carry over.</p><p>This time, however, the focus shifts toward automation and reproducibility. By leveraging Terraform and DevContainers, the same setup is expressed as code and wrapped in a consistent development environment that closely resembles production workflows. While an actual production deployment would rely on cloud-managed resources, the Kubernetes configurations and operational patterns remain largely the same.</p><blockquote><strong>Note: </strong>I won’t go deep into tools already covered in the previous <a href="https://medium.com/gitconnected/from-zero-to-gitops-deploying-mage-ai-on-local-kubernetes-with-argo-cd-25865a035571">post</a>. Instead, I’ll focus on the new additions specifically <strong>DevContainers</strong> and <strong>Terraform</strong> and, more importantly, why they were chosen and what problems they help solve.</blockquote><h3>The project</h3><p>The main goal of this project is to build a local data platform that allows data tools and applications to be deployed in a structured, reproducible, and maintainable way. Although everything runs on a developer machine, the setup is intentionally designed with a production-like mindset. Infrastructure, platform components, and applications are clearly separated, and the entire environment can be recreated from scratch using code.</p><p>To keep the platform realistic without unnecessary complexity, we deploy a small but representative set of data services on top of it (for full code check <a href="https://github.com/georgezefko/data-platform-engineering">here</a>):</p><ul><li><strong>Mage.ai</strong> for orchestrating data pipelines</li><li><strong>PostgreSQL</strong> as our primary database</li><li><strong>Kafka</strong> for event streaming</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*9jKXaA5DaL8bQX_5vkEiRg.png" /><figcaption>Local data platform with Kubernetes in Docker. Image created by the Author.</figcaption></figure><p>At the foundation of the setup is Kubernetes, which provides a consistent abstraction for running and managing services across local, staging, and production environments. In this case, the Kubernetes cluster is created locally using KinD, while Terraform is used to provision and manage the platform components that run on top of it in a declarative way. Treating infrastructure as code makes the setup reproducible, version-controlled, and easier to evolve as complexity grows.</p><p>Terraform is also responsible for deploying Argo CD, which acts as the GitOps engine for the platform. While it is technically possible to deploy everything using Terraform, I prefer using tools for what they do best:</p><ul><li><strong>Terraform </strong>manages platform-level components and bootstrap configuration</li><li><strong>Argo CD</strong> ensures that applications are deployed and kept in sync with Git</li></ul><p>This clear separation of responsibilities avoids confusion and keeps the system easier to reason about as it scales.</p><p>Finally, <strong>DevContainers</strong> are introduced to standardize the development environment. By defining the tooling and dependencies upfront, we eliminate the <em>“works on my machine”</em> problem and make onboarding straightforward for anyone working on the project.</p><p>You can think of this setup as a pyramid:</p><ul><li><strong>Base layer: </strong>Kubernetes cluster (KinD) and platform bootstrap</li><li><strong>Middle layer:</strong> Application orchestration via Argo CD (GitOps)</li><li><strong>Top layer:</strong> Data applications and services (Mage, PostgreSQL, Kafka, pipelines)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/291/1*MdxTsegO5257t6tc2N2y_w.png" /><figcaption>Image created by the author</figcaption></figure><p>This structure mirrors how real-world data platforms are typically designed, while remaining lightweight enough to run entirely on a local machine.</p><h3>Assembling the local platform</h3><p>With the project goals and design principles in place, we can now look at how they translate into <a href="https://github.com/georgezefko/data-platform-engineering">code</a> and how the platform is assembled end-to-end. Rather than treating structure and execution as separate concerns, this section shows how the repository layout directly supports the bootstrap flow of the platform.</p><p>The project is organized into a small number of top-level directories, each with a clearly defined responsibility. This makes it easy to reason about the system and understand where changes should live as the platform evolves.</p><pre># <br>.<br>├── .devcontainer/<br>├── scripts/<br>├── infra/<br>│   ├── terraform/<br>│   └── kind/<br>├── apps/<br>├── argo/<br>├── Makefile<br>└── README.md</pre><h4>1. Standardizing the Development Environment (.devcontainer/)</h4><p>Everything starts with a consistent development environment. The .<em>devcontainer/ directory</em> defines a fully reproducible setup using DevContainers, ensuring that all contributors work with the same tooling, versions, and dependencies.</p><p>The DevContainer includes all tools required to operate the platform locally Terraform, Kubernetes tooling, and KinD with versions pinned to avoid drift across machines. Kubernetes is run locally using Docker-in-Docker, allowing the cluster to be created and managed entirely from inside the container.</p><p>This approach eliminates <em>“works on my machine”</em> issues and ensures that the platform can be bootstrapped from a clean checkout without any manual setup on the host.</p><blockquote>Note: Running Kubernetes using devcontainer and docker-in-docker comes with limitations. Ingress based access is not practical in this setup, so services are exposed using port-forwarding instead. For ingress set up locally check <a href="https://medium.com/gitconnected/from-zero-to-gitops-deploying-mage-ai-on-local-kubernetes-with-argo-cd-25865a035571">here</a>.</blockquote><h4>2. Creating the Kubernetes Cluster (infra/kind/)</h4><p>With the development environment in place, the next step is creating the Kubernetes cluster itself. This is handled using KinD, with cluster configuration defined under <em>infra/kind/kind-config.yaml.</em></p><pre>kind: Cluster<br>apiVersion: kind.x-k8s.io/v1alpha4<br>name: development<br>nodes:<br>- role: control-plane<br>  kubeadmConfigPatches:<br>  - |<br>    kind: InitConfiguration<br>    nodeRegistration:<br>      kubeletExtraArgs:<br>        node-labels: &quot;ingress-ready=true,kubernetes.io/os=linux&quot;<br>  extraPortMappings:<br>  - containerPort: 80<br>    hostPort: 80<br>    protocol: TCP<br>  - containerPort: 443<br>    hostPort: 443<br>    protocol: TCP</pre><p>The configuration specifies how the local cluster should be created, including node roles, labels, and port mappings. Once created, the cluster is available via a dedicated kubeconfig context, which becomes the foundation for all subsequent steps.</p><p>Importantly, cluster creation is kept separate from infrastructure provisioning. <strong>KinD is responsible only for creating the cluster, not for installing anything into it.</strong></p><h4>3. Bootstrapping the Platform Control Plane (infra/terraform/)</h4><p>Once the cluster exists, Terraform is used to bootstrap the platform-level components that run inside it. All Terraform configuration lives under <em>infra/terraform/</em> and is scoped to platform concerns rather than applications.</p><p>In the local development environment, Terraform installs Argo CD into the cluster using the Helm provider. Provider configuration connects Terraform to the Kubernetes cluster created by KinD, allowing it to manage resources inside the cluster in a declarative way.</p><p>Terraform’s responsibility ends here. It does not manage application deployments. Instead, it prepares the cluster so that GitOps can take over.</p><p>This deliberate narrowing of scope keeps the system easier to reason about and avoids overlapping responsibilities between tools.</p><p><strong>3a. How Terraform Is Wired to the Cluster</strong></p><blockquote>Terraform configuration follows standard conventions and is split across a small set of files, each with a clear responsibility.</blockquote><blockquote><strong>versions.tf</strong> defines the required Terraform version and provider constraints (Helm and Kubernetes), ensuring consistent tooling across environments.</blockquote><blockquote><strong>variables.tf</strong> exposes environment-specific inputs, such as:</blockquote><blockquote>- the namespace where Argo CD is installed</blockquote><blockquote>- the kubeconfig path used inside the DevContainer</blockquote><blockquote>- the Kubernetes context created by KinD</blockquote><blockquote><strong>providers.tf</strong> connects Terraform to the Kubernetes cluster by configuring the Helm and Kubernetes providers to use the correct kubeconfig and context.</blockquote><blockquote><strong>main.tf</strong> contains the actual platform resource definition. In this case, Terraform installs Argo CD using the official Helm chart, applying a small set of value overrides while keeping everything version-controlled.</blockquote><blockquote>With these pieces in place, Terraform can treat the local KinD cluster as a target system and bootstrap the platform control plane declaratively.</blockquote><p><strong>3b. How the Pieces Fit Together</strong></p><blockquote>At a high level, the flow looks like this:</blockquote><blockquote>1. KinD creates the Kubernetes cluster using the configuration under infra/kind/</blockquote><blockquote>2 .Terraform reads environment inputs and provider configuration</blockquote><blockquote>3. Terraform connects to the cluster via the kind-development context</blockquote><blockquote>4. Terraform installs Argo CD into the cluster</blockquote><blockquote>5. Argo CD takes over and deploys everything defined under argo/ and apps/</blockquote><blockquote>This is the key handoff point in the platform: <strong>Terraform bootstraps the control plane, and GitOps takes full ownership from that moment on.</strong></blockquote><h4>4. Defining Applications with GitOps (argo/ and apps/)</h4><p>After Terraform completes, control shifts to Argo CD. The <em>argo/</em> directory contains the Argo CD manifests that define how applications are deployed using GitOps, following the <em>app-of-apps</em> pattern.</p><p>A single root application points Argo CD to a directory containing one Application manifest per data service. Each application is defined independently, with its own configuration and lifecycle.</p><p>The actual deployment configuration for those services lives under <em>apps/</em>. Depending on the tool, different strategies are used:</p><ul><li><strong>Helm</strong> values for tools such as Mage</li><li><strong>Kustomize</strong> bases and overlays for tools such as Kafka</li></ul><p>This separation allows applications to evolve independently while remaining fully declarative and version-controlled.</p><p>From this point onward, Argo CD continuously reconciles the desired state from Git with what runs in the cluster.</p><blockquote>Note: For services like kafka you can use more updated and advanced ways of deployments by utilizing operators like <a href="https://strimzi.io/">strimzi</a>.</blockquote><h4>5. Orchrestrating the Bootstrap Process (scripts/)</h4><p>All of the above steps are tied together by a small set of helper scripts under the <em>scripts/</em> directory. The main entry point is a bootstrap script that orchestrates the full lifecycle:</p><pre>#!/usr/bin/env bash<br>set -euo pipefail<br><br># Resolve repo root (directory where this script lives)<br>SCRIPT_DIR=&quot;$(cd &quot;$(dirname &quot;${BASH_SOURCE[0]}&quot;)&quot; &amp;&amp; pwd)&quot;<br>REPO_ROOT=&quot;$(cd &quot;${SCRIPT_DIR}/..&quot; &amp;&amp; pwd)&quot;<br><br>mkdir -p &quot;${REPO_ROOT}/.kube&quot;<br><br>KIND_UP=&quot;${REPO_ROOT}/scripts/kind-up.sh&quot;<br>TF_DIR=&quot;${REPO_ROOT}/infra/terraform/environments/dev&quot;<br>ARGO_BOOTSTRAP=&quot;${REPO_ROOT}/argo/app-of-apps.yaml&quot;<br><br>if [[ ! -x &quot;${KIND_UP}&quot; ]]; then<br>  echo &quot;ERROR: kind-up script not found or not executable: ${KIND_UP}&quot; &gt;&amp;2<br>  echo &quot;Repo root: ${REPO_ROOT}&quot; &gt;&amp;2<br>  echo &quot;Contents of scripts/:&quot; &gt;&amp;2<br>  ls -la &quot;${REPO_ROOT}/scripts&quot; || true<br>  exit 1<br>fi<br><br>bash &quot;${KIND_UP}&quot; development<br><br>pushd &quot;${TF_DIR}&quot; &gt;/dev/null<br>terraform init<br>terraform apply -auto-approve<br>popd &gt;/dev/null<br><br>kubectl apply -f &quot;${ARGO_BOOTSTRAP}&quot;</pre><ol><li>Create (or recreate) the local Kubernetes cluster</li><li>Apply Terraform to install platform components</li><li>Bootstrap Argo CD with the root application</li></ol><p>This provides a single, repeatable command to bring the entire platform up from scratch. It also establishes a clear execution order that can later be reused in CI/CD pipelines for validation, testing, or environment provisioning.</p><p>By structuring the repository around execution order and clearly separating responsibilities, the platform remains easy to understand despite involving multiple tools. Each directory plays a specific role at a specific stage of the bootstrap process.</p><p>The result is a local data platform that mirrors real-world architectures, is fully reproducible, and can be brought up from a clean checkout with minimal manual effort.</p><h3>Run the show</h3><p>With all components in place, we can now bring the platform up and see it in action. At this stage, the entire setup is designed to be driven from a single entry point, closely mirroring how such a platform would eventually be operated from a CI/CD pipeline.</p><h4>Start the development environment</h4><p>First, open the repository in the <strong>DevContainer</strong>. This ensures all required tooling (Terraform, Kubernetes CLIs, Helm, etc.) is available and correctly configured.</p><p>Once the DevContainer is running, all commands are executed from inside the container.</p><h4>One command to bring everything up</h4><p>With the development environment in place, the entire platform can be bootstrapped with a single command:</p><pre>make up</pre><p>Internally, this is a thin wrapper around <em>scripts/bootstrap.sh</em>, and orchestrates the full bootstrap lifecycle mentioned above.</p><p>After a minute or so, the platform and all applications should be deployed.</p><p>It also mirrors how a real data platform might be bootstrapped in CI/CD, making it easy to evolve this local setup into a more production-like environment later on.</p><h4>Accessing the platform components</h4><p>Because the platform runs inside a DevContainer using Docker-in-Docker, applications are accessed via <strong>port forwarding</strong>. The Makefile already includes helpers for the most common services.</p><p>To access the Argo CD UI, run:</p><pre>make argo-ui</pre><p>This command sets up port forwarding and prints (see image below):</p><ul><li>The local URL</li><li>The username and password</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*Pmd6-ZFVVXFMrEoui_dAyg.png" /></figure><p>Once opened, you should see all platform applications deployed and managed by Argo CD. You’ll also notice that some applications (such as Mage) are deployed using Helm, as defined in their respective Argo CD Application manifests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OWlHPhzkYapBHi3MWKS04g.png" /></figure><p>Mage can be accessed using its corresponding Makefile target:</p><pre>make mage-ui</pre><p>From the Mage UI, you can:</p><ul><li>Create pipelines interactively</li><li>Explore existing workflows</li><li>Experiment with deployment patterns (e.g. Docker-based pipelines). Check this <a href="https://medium.com/gitconnected/devops-for-data-engineers-part-1-e70f7bcbabb2">tutorial</a></li></ul><p>PostgreSQL can also be port-forwarded using the provided Makefile command. Once forwarded, you can connect to DB using a database client such as DBeaver.</p><p><strong>Inspecting Kafka Topics and Messages</strong></p><p>Kafka is slightly different. While it is technically possible to port-forward Kafka, doing so reliably requires careful configuration of advertised listeners and is often more trouble than it’s worth for local debugging.</p><p>Instead, a simpler and more robust approach is to run a temporary debug pod using the same Kafka image and interact with the cluster from inside Kubernetes.</p><pre>kubectl run kafka-debug -n develop \<br>  --image=confluentinc/cp-kafka:7.6.1 \<br>  --rm -it -- bash</pre><p>Once inside the container, you can list available topics:</p><pre>kafka-topics \<br>  --bootstrap-server kafka-svc.develop.svc.cluster.local:9092 \<br>  --list</pre><p>To inspect messages flowing through a topic:</p><pre>kafka-console-consumer \<br>  --bootstrap-server kafka-svc.develop.svc.cluster.local:9092 \<br>  --topic iot-telemetry \ # or whatever name you have<br>  --from-beginning</pre><p>This approach allows you to verify that Kafka is working correctly without exposing brokers outside the cluster.</p><blockquote><strong>Note:</strong> In this project, credentials are stored in plain text for simplicity. In a real production environment, secrets would be managed using a secure secrets manager (e.g. cloud key vaults) and never committed to a repository.</blockquote><h3>Pipeline-Demo</h3><p>With the platform up and running, we can now test how the individual tools work together by running a simple, prebuilt data pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/465/1*dSZBwypNEXyPFUwru_EZSQ.png" /><figcaption>Demo pipeline (image created by the author)</figcaption></figure><p>Start by opening the Mage UI. The demo pipeline is already defined and configured to run without any additional setup. All required environment variables are parameterized, so the pipeline should work out of the box.</p><blockquote>Note: This the <a href="https://github.com/georgezefko/mage_projects">repository</a> for the pipelines</blockquote><blockquote>For a deeper look at how this pipeline would be integrated into a full CI/CD workflow, see the following posts:</blockquote><blockquote>1: <a href="https://medium.com/gitconnected/devops-for-data-engineers-part-1-e70f7bcbabb2">DevOps for Data Engineers: Setting Up CI/CD Pipelines with Github Actions, Semantic Release and Conventional Commit</a><br>2: <a href="https://medium.com/gitconnected/from-zero-to-gitops-deploying-mage-ai-on-local-kubernetes-with-argo-cd-25865a035571">From Zero to GitOps: Deploying Mage AI on Local Kubernetes with Argo CD</a></blockquote><p>For the purpose of this demo, we’ll manually create a table in PostgreSQL to store incoming telemetry data.</p><p>First, port-forward the PostgreSQL service using the provided Makefile target. Once forwarded, connect to the database using a client such as DBeaver.</p><p>Then create the table by running:</p><pre> CREATE TABLE IF NOT EXISTS telemetry (<br>    device_id        TEXT NOT NULL,<br>    ts               TIMESTAMPTZ NOT NULL,<br>    energy_usage     NUMERIC(5,2),<br>    temperature      NUMERIC(4,1),<br>    vibration        NUMERIC(4,1),<br>    signal_strength  INTEGER,<br>    PRIMARY KEY (device_id, ts )<br>    );</pre><p>With the table in place, return to the Mage UI and manually trigger the demo pipeline (it consists of <em>streaming_tutorial and lambda_iot_data_generator </em>you need to trigger both). For the sake of simplicity, the pipeline is run interactively rather than scheduled.</p><p>While the pipeline is running, its status and individual steps can be monitored directly from the Mage UI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wn1mc3dn1DrqpoeHpiD12w.png" /><figcaption>Pipelines running on our local K8s via Mage</figcaption></figure><p>While the pipeline(s) running, you can query the telemetry table in PostgreSQL to verify that data has been written successfully. If everything worked as expected, you should see records corresponding to the pipeline output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*od-IYCjktoxwRDN4v_Llyw.png" /><figcaption>Output copied from Postgres deployed on our local k8s</figcaption></figure><p>This simple demo ties together all major components of the platform Kubernetes networking, application deployment, orchestration, and persistence without introducing unnecessary complexity.</p><p>At this point, you have a fully functional local data platform that can be used to experiment with Kubernetes operations, networking, deployments, and GitOps workflows in a realistic but lightweight environment.</p><h3>Conclusion</h3><p>In this post, we built a local data platform that mirrors many of the patterns used in real-world production environments. By using Kubernetes (via KinD) as the execution layer, Terraform for infrastructure provisioning, Argo CD for GitOps-driven deployments, and DevContainers for a consistent development experience, we created a setup that is reproducible, automation-friendly, and easy to reason about.</p><p>Rather than focusing on individual tools in isolation, the goal was to show how they fit together to form a cohesive platform. The result is a local environment that can be brought up from scratch, evolves through code, and clearly separates responsibilities between infrastructure, platform components, and applications.</p><p>For data engineers, this kind of setup helps bridge the gap between writing pipelines and owning the platforms they run on. It provides a practical way to experiment with Kubernetes, Terraform, and GitOps workflows without the overhead of managing real cloud infrastructure, while still working with patterns that translate directly to production.</p><p>From here, there are several natural next steps: integrating CI/CD to automate the bootstrap process, adding observability to better understand system behaviour, and building more realistic pipelines on top of the platform. Those are the topics I plan to explore next.</p><p><strong>Thanks for reading!</strong></p><p>I hope you found this guide useful and that it inspires you to experiment further with data pipeline architectures.</p><p>Your feedback is always appreciated and helps improve these posts.<br>📦 Check out the code in the <a href="https://github.com/georgezefko/data-platform-engineering">GitHub repository</a><br>🤝 Feel free to connect with me on <a href="http://www.linkedin.com/in/georgioszefkilis">LinkedIn</a></p><h3>References</h3><ol><li><a href="https://medium.com/containers-101/how-to-structure-your-argo-cd-repositories-using-application-sets-1150e75d05b3">How to structure Argo CD repositories</a></li><li><a href="https://medium.com/@mehmetodabashi/installing-argocd-on-minikube-and-deploying-a-test-application-caa68ec55fbf">Installing Argo CD</a></li><li><a href="https://kind.sigs.k8s.io/">Kubernetes in docker</a></li><li><a href="https://developer.hashicorp.com/terraform">Terraform official docs</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9547a4256a7f" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/building-a-local-data-platform-with-kubernetes-and-terraform-9547a4256a7f">Building a Local Data Platform with Kubernetes and Terraform</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kappa Architecture in Action — From Sensors to Dashboards with Kafka, Spark Streaming, StarRocks…]]></title>
            <link>https://blog.dataengineerthings.org/kappa-architecture-in-action-from-sensors-to-dashboards-with-kafka-spark-streaming-starrocks-80492a509e1a?source=rss-8c7ffca79196------2</link>
            <guid isPermaLink="false">https://medium.com/p/80492a509e1a</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[streaming]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[spark-streaming]]></category>
            <category><![CDATA[kappa-architecture]]></category>
            <dc:creator><![CDATA[George Zefkilis]]></dc:creator>
            <pubDate>Fri, 17 Oct 2025 17:52:14 GMT</pubDate>
            <atom:updated>2025-10-17T17:52:14.083Z</atom:updated>
            <content:encoded><![CDATA[<h3>Kappa Architecture in Action — From Sensors to Dashboards with Kafka, Spark Streaming, StarRocks, Minio, and Docker</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*cyQOQ-qHsReR6sZ7lfXstw.jpeg" /><figcaption>Image taken from <a href="https://www.datasciencecentral.com/benefits-of-using-kafka-to-handle-real-time-data-streams/">data science central</a></figcaption></figure><p>Hello again fellow technology enthusiasts.</p><p>In a previous <a href="https://medium.com/data-engineer-things/lambda-architecture-in-action-from-sensors-to-dashboards-with-kafka-iceberg-nessie-starrocks-3ae9a4c1c605">post</a>, we explored the Lambda architecture through a practical end-to-end implementation, breaking down how batch and real-time layers work together to meet different data processing needs. While Lambda remains a powerful approach, its dual-pipeline design can introduce complexity and operational overhead.</p><p>In this post, we’ll tackle the same business scenario using the <strong>Kappa architecture</strong>. Instead of combining batch and streaming, Kappa relies on a single stream processing pipeline to handle both real-time and historical data. We’ll walk through its components, discuss the trade-offs, and see how it simplifies certain aspects of data engineering compared to Lambda.</p><p>Let’s get started.</p><h3>Introduction to Kappa Architecture</h3><p>Kappa Architecture is a data processing design introduced by Jay Kreps (co-creator of Apache Kafka) as a simpler alternative to Lambda Architecture. Instead of maintaining separate batch and speed layers, Kappa uses a single, unified streaming pipeline to handle both real-time and historical data.</p><p>All incoming data is written to an <strong>append-only event log</strong> (e.g., Kafka), which acts as the system’s source of truth. A <strong>stream processing engine</strong> (e.g., Apache Flink, Spark Streaming) continuously processes the log to update views or power downstream systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*YdCvwpKRK2HOYlF_or1DCQ.png" /><figcaption>This is how the general kappa architecture looks like as described by Jay Kreps <a href="https://www.oreilly.com/radar/questioning-the-lambda-architecture/?ref=chaosgenius.io">here</a></figcaption></figure><p>If reprocessing is needed (e.g., fixing logic or recomputing results), the architecture simply <strong>replays the log</strong> with the new code, removing the need for a separate batch pipeline. In Kappa, batch is essentially just a special case of stream processing.</p><p>A few of the key <strong>benefits</strong> of Kappa architecture over the Lambda one are the following:</p><p><strong>Simplicity: </strong><em>There’s only one processing pipeline to develop, test, and maintain, which removes the duplication found in Lambda’s batch and speed layers.</em><br><strong>Real-Time by Default: </strong><em>Since everything flows through a streaming system, results are updated with low latency.</em><br><strong>Consistency: </strong><em>With a single source of truth (like Kafka) and exactly-once processing, the risk of mismatched outputs across layers is eliminated.</em><br><strong>Scalable Reprocessing: </strong><em>Replaying the event log allows you to recompute results with new logic without building a separate batch system.</em></p><p>On the other hand, like any architectural approach, Kappa comes with a few <strong>drawbacks</strong>:</p><p><strong>Log Retention Overhead: </strong><em>Keeping historical events in the streaming platform can increase storage and infrastructure costs.</em><br><strong>Reprocessing Time: </strong><em>Replaying large logs for backfills may be time-consuming.</em><br><strong>Streaming Expertise Required: </strong><em>Teams need to be comfortable with stateful stream processing, fault tolerance, and event-driven design.</em><br> <strong>Resource Competition: </strong><em>Without separate layers, heavy historical queries may contend with real-time workloads if not carefully managed.</em></p><h3>Business Scenario: Industrial IoT Monitoring — ReCap</h3><p>The business scenario is as follows:</p><p>A manufacturing company uses IoT sensors to monitor the health of industrial machines. These sensors collect</p><ul><li><strong>Telemetry data</strong>: Energy usage, temperature, vibration.</li><li><strong>Event logs</strong>: Machine failures, maintenance records.</li></ul><blockquote>In the previous post, I introduced that scenario to demonstrate the Lambda architecture. If you haven’t seen it yet, you can check it out <a href="https://medium.com/data-engineer-things/lambda-architecture-in-action-from-sensors-to-dashboards-with-kafka-iceberg-nessie-starrocks-3ae9a4c1c605">here</a>. For this post, we’ll reuse the same Industrial IoT monitoring use case, but with a slightly simplified version. The original setup turned out to be a bit too complex and may have made the concepts harder to follow, so this time the focus will be clearer.</blockquote><p>The core requirement is to process temperature telemetry in real time while enriching it with contextual operational data. Specifically, the system should:</p><p><strong>Aggregate temperature readings over a 5-minute (</strong><em>for the sake of simplicity, normally it is longer</em><strong>) window</strong> (using minute-level granularity).</p><p><strong>Detect anomalies</strong> by comparing aggregated values against a defined threshold.</p><p><strong>Join telemetry with event logs</strong> to add relevant metadata and operational insights.</p><p>For each 5-minute window, we want to answer:</p><ul><li>Has enough data arrived to finalize the window?</li><li>Is the behavior anomalous based on the temperature threshold?</li><li>What operational events occurred in the same time frame (e.g., failures, maintenance, inspections, severity)?</li></ul><p>This setup gives us a practical foundation to demonstrate how Kappa architecture handles streaming, enrichment, and reprocessing in a unified pipeline.</p><p>The data pipeline architecture, along with the technologies used at each step, is illustrated in the image below (<em>The whole pipeline architecture and technologies used can be found in this </em><a href="https://github.com/georgezefko/learning-spark"><em>repository</em></a><em>)</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mUk5dFZqtZcvbdu7Tf1C2A.png" /><figcaption>Image created by the Author</figcaption></figure><blockquote><strong>Disclaimer:</strong> This is a hypothetical scenario and further development needed to optimize data models, logic and dashboards but I believe it gives a good overview of an end-to-end business case.</blockquote><h3>Core Components of Kappa Architecture in Action</h3><h4>Ingestion</h4><p>The ingestion layer is the entry point of any data architecture, and its primary responsibility is to collect data from various sources. In Kappa, this layer is <strong>stream-first by design</strong>. Rather than separating data into batch and real-time pipelines, all data flows through a streaming backbone.</p><p>This is why <strong>publish/subscribe systems</strong> (e.g., Apache Kafka, Pulsar, or cloud-native equivalents) are the most common choice. They provide durability, scalability, and replay capabilities — making them a perfect fit for Kappa’s “single source of truth” model.</p><p>In our tutorial, we’ll use <strong>Kafka</strong> as the ingestion layer to capture both telemetry streams and operational event logs before passing them downstream for processing. Since we don’t have access to a real IoT source, we’ll simulate one with a small Python script (data_generator.py) .</p><p>This generator continuously produces two kinds of JSON messages:</p><ul><li><strong>Telemetry</strong> (temperatures from multiple devices):</li></ul><pre>{<br>  &quot;device_id&quot;: &quot;device_1&quot;,<br>  &quot;timestamp&quot;: &quot;2025-10-11T12:34:56.123Z&quot;,<br>  &quot;temperature&quot;: 53.2<br>}</pre><ul><li><strong>Events</strong> (failures, inspections, maintenance logs):</li></ul><pre>{<br>  &quot;event_id&quot;: &quot;f3e2…&quot;,<br>  &quot;device_id&quot;: &quot;device_1&quot;,<br>  &quot;event_timestamp&quot;: &quot;2025-10-11T12:35:00.000Z&quot;,<br>  &quot;event_type&quot;: &quot;failure&quot;,<br>  &quot;severity&quot;: &quot;high&quot;<br>}</pre><p>What makes it useful is that it doesn’t just emit random values — it also introduces real-world challenges like <strong>hot periods</strong>(spikes in readings), <strong>outages</strong> (devices going silent), and <strong>late data</strong> (backdated timestamps). These patterns give us a realistic stream to work with and let us exercise key features such as <strong>watermarking, windowing, and event-time joins </strong>in downstream processing.</p><p>For this tutorial, I’ve slightly modified the generator compared to the Lambda version to better control data frequency and lateness, which makes watermark behavior easier to demonstrate in the processing layer.</p><p>Now that we have clarified what the data source and ingestion do time to jump into the most challenging part is the streaming processing.</p><h4>Streaming layer</h4><p>The streaming layer is where the actual <strong>processing of incoming data</strong> happens. In Kappa, this layer continuously consumes events from the ingestion system and applies the required logic in near real-time.</p><p>Depending on the use case, processing can range from <strong>simple operations</strong> such as filtering, transformations, and aggregations to <strong>more advanced analytics</strong> like windowed computations, anomaly detection, or even <strong>stream-to-stream joins</strong> (e.g., combining telemetry with event logs).</p><p>Common technologies in this layer include <strong>Apache Flink </strong>and <strong>Spark Structured Streaming</strong>, each designed to handle high-throughput, low-latency data streams at scale.</p><p>In our tutorial, we’ll use this layer to implement the 5-minute temperature aggregation, anomaly detection, and the enrichment of telemetry with event log data. We’ll use <strong>Spark Structured Streaming</strong>, which provides event-time semantics, watermarks for handling late data, and checkpointing for fault tolerance.</p><p><strong>Spark Structured Streaming</strong> lets us express streaming pipelines with the same DataFrame API you’d use for batch, but with event-time semantics.In practice, that means we can aggregate, join, and enrich streams as they arrive — then push results into an analytics store.</p><p>The whole streaming process takes place in a single script (stream_stream.pyin the <a href="https://github.com/georgezefko/learning-spark">repository</a>). As illustrated in the pipeline sketch above, the streaming job does the following:</p><ul><li>Reads the two Kafka topics: telemetry (temperatures) and events (failures, inspections, maintenance).</li><li>Parses JSON and extracts event-time columns.</li><li>Aggregates both streams in <strong>5-minute windows</strong>.</li><li>Joins telemetry + events + a static device dimension.</li><li>Derives <strong>completeness flags</strong> (coverage &amp; volume) and <strong>anomaly flags</strong> (temperature &gt; threshold).</li><li>Writes results to <strong>StarRocks</strong> with upserts using foreachBatch.</li></ul><p>There are a few key concepts included in the process which I would like to briefly exaplain further as they are very important in streaming processing.</p><p><strong>Watermarks</strong><br>Define how long to wait for late events. With a 3-minute watermark, events that arrive up to 3 minutes late can still update their window; later ones are ignored. This makes lateness from our generator a visible and tunable factor.</p><p><strong>Checkpoints</strong><br>Store Kafka offsets and window state. They ensure Spark can recover from failures without reprocessing data, enabling exactly-once guarantees when paired with an idempotent sink. In our tutorial, we use Minio to store the state.</p><p><strong>foreachBatch</strong><br>Runs custom logic on each micro-batch. Here it’s used to perform <strong>JDBC upserts</strong> into StarRocks, giving us control and reliability.</p><p><strong>Joins</strong><br>Telemetry and events are <strong>windowed and watermarked</strong> before being joined on device_id and the same 5-minute window. A static dimension table enriches results with thresholds, location, and model data.</p><p>Each row represents one device in one 5-minute window. Here’s an example (JSON-style) record:</p><pre>{<br>  &quot;window_start&quot;: &quot;2025-10-11T12:30:00Z&quot;,<br>  &quot;window_end&quot;: &quot;2025-10-11T12:35:00Z&quot;,<br>  &quot;device_id&quot;: &quot;device_1&quot;,<br>  &quot;minutes_covered&quot;: 5,<br>  &quot;cnt_events&quot;: 29,<br>  &quot;avg_temperature&quot;: 51.7,<br>  &quot;min_temperature&quot;: 47.2,<br>  &quot;max_temperature&quot;: 62.9,<br>  &quot;events_total&quot;: 2,<br>  &quot;events_failure&quot;: 1,<br>  &quot;events_maintenance&quot;: 0,<br>  &quot;events_inspection&quot;: 1,<br>  &quot;events_sev_high&quot;: 1,<br>  &quot;incomplete_flag&quot;: false,<br>  &quot;anomaly_flag&quot;: true,<br>  &quot;anomaly_reason&quot;: &quot;MAX_TEMP&gt;THRESHOLD&quot;,<br>  &quot;threshold_used&quot;: 55.0,<br>  &quot;location_id&quot;: &quot;plant-a&quot;,<br>  &quot;model&quot;: &quot;Model-C&quot;,<br>  &quot;updated_at&quot;: &quot;2025-10-11T12:35:02Z&quot;<br>}</pre><blockquote>How the streaming process connects back to ingestion:</blockquote><blockquote><strong>Hot periods</strong> in the generator trigger anomaly flags. <strong>Outages</strong> reduce coverage and volume, raising completeness flags. <strong>Late data</strong> is accepted or dropped depending on the watermark setting.</blockquote><p>Lastly, the most painful (in my opinion) part in streaming systems is <strong>how to re-calculate historical inputs</strong> or “backfill” without disrupting the current process.</p><p>In our current scenario, backfilling isn’t automatic. The simplest way is to restart the job withone startingOffsets=earliest on both topics, while also pointing to a <strong>new checkpoint path</strong>. This forces Spark to replay all retained Kafka messages and rebuild the outputs.</p><p>There could be many strategies depending on the requirements. Below are 2 examples:</p><ul><li><a href="https://www.uber.com/en-DK/blog/kappa-architecture-data-stream-processing/"><strong>Uber-style</strong></a><strong> selective recompute</strong>: periodically pull data from a table inside the streaming job (e.g., on a 1-hour trigger), reprocess it, and push results back to the stream.</li><li><strong>Bronze table approach</strong>: store all raw messages in a multiplex table (Delta, Iceberg, etc.) and consume from there. For backfill, query historical data directly and re-run the pipeline.</li></ul><p>Backfilling in Kappa is a powerful capability but requires careful planning, particularly around watermarks, checkpoints, and sink semantics to avoid duplication or data loss.</p><p>With this, we’ve covered the streaming layer. Next, we’ll move on to the <strong>serving layer</strong>, where processed results become queryable for downstream applications.</p><h4>Serving layer</h4><p>Similar to Lambda, the serving layer is the final component in the Kappa architecture. Its purpose is to <strong>store the processed data and make it queryable</strong> for downstream applications, dashboards, and further analysis.</p><p>Depending on requirements, this layer can be built on top of <strong>databases, data warehouses, or specialized storage systems</strong> optimized for fast reads and analytical queries.</p><p>In our case, we’ll use <strong>StarRocks</strong> as the serving database and <strong>Apache Superset</strong> for dashboards (the same tools used in the Lambda tutorial).</p><p>In this tutorial, I don’t use StarRocks <strong>Routine Load</strong> from <strong>Kafka</strong>; instead, I write directly from <strong>Spark Structured Streaming</strong> via <strong>JDBC</strong>. This path is more straightforward to set up, though inserting Kafka in between would provide a durable buffer in case either end goes down.</p><p>One important table-design note for StarRocks: for streaming writes, use <strong>PRIMARY KEY</strong> tables to avoid duplicate records.</p><p>Once the data lands in the target tables, we can query it and start generating insights. I won’t cover connecting or querying in Superset here, as the steps are identical to the Lambda tutorial (please refer to that step-by-step <a href="https://medium.com/data-engineer-things/lambda-architecture-in-action-from-sensors-to-dashboards-with-kafka-iceberg-nessie-starrocks-3ae9a4c1c605">guide</a>).</p><p>Below is a snippet of our dashboard that answers the core questions from the business scenario: window completeness, anomaly detection, and the operational context in the same time window (you can find the queries in the <a href="https://github.com/georgezefko/learning-spark">repo</a> under db_analytics.sql).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1aVAHXz6bPshwa0gbAQlsg.png" /><figcaption>Sample of Analytics Dashboard</figcaption></figure><blockquote>Note: The more data we generate the more insightful the dashboard will look like. The above is just a demonstration and you can derive any insights you would like by creating your own queries on top.</blockquote><h3>Conclusion</h3><p>In this post, we explored how the <strong>Kappa architecture</strong> can be applied to a practical Industrial IoT monitoring scenario. Unlike Lambda, Kappa simplifies the overall design by relying on a <strong>single streaming pipeline</strong> for both real-time and historical data, removing the need for separate batch and speed layers.</p><p>We walked through its <strong>core components, </strong>from ingestion with Kafka, to stream processing with Spark Structured Streaming, and finally to serving insights through StarRocks and Superset. Along the way, we saw how Kappa addresses requirements like <strong>windowed aggregation, anomaly detection, and contextual enrichment</strong> with operational event logs, all within a unified processing model.</p><p>Of course, Kappa comes with its own trade-offs. Storing and replaying full event logs can add overhead, and streaming-first systems demand a stronger learning curve. But in use cases where <strong>real-time insights are essential</strong> and maintaining dual pipelines is too costly, Kappa offers a powerful and elegant alternative.</p><p>Having now implemented both <strong>Lambda and Kappa</strong> for the same scenario, we can clearly see the strengths and limitations of each. While Lambda provides flexibility with its dedicated batch layer, Kappa shines through <strong>simplicity and real-time responsiveness</strong>.</p><p>Ultimately, the right choice depends on the nature of your workloads. But by understanding and experimenting with both approaches, data engineers can better equip themselves to design pipelines that truly meet business needs.</p><p><strong>Thanks for reading!</strong></p><p>I hope you found this guide useful and that it inspires you to experiment further with data pipeline architectures.</p><p>Your feedback is always appreciated and helps improve these posts.<br>📦 Check out the code in the <a href="https://github.com/georgezefko/learning-spark">GitHub repository</a><br>🤝 Feel free to connect with me on <a href="http://www.linkedin.com/in/georgioszefkilis">LinkedIn</a></p><h3>References</h3><ol><li>Jay Kreps — <a href="https://www.oreilly.com/radar/questioning-the-lambda-architecture/?ref=chaosgenius.io">Questioning the Lambda architecture</a></li><li>Lenon Rodrigues — <a href="https://medium.com/@lenonrodrigues/kappa-architecture-an-efficient-model-for-real-time-processing-767c623d04ad">Kappa Architecture: An Efficient Model for Real-Time Processing</a></li><li>Lenon Rodrigues — <a href="https://medium.com/@lenonrodrigues/a-detailed-comparison-between-spark-structured-streaming-and-apache-flink-comparison-of-features-10ae6ce8aaaa">A Detailed Comparison between Spark Structured Streaming and Apache Flink: Comparison of Features, Usability, and Ecosystem.</a></li><li>Amey Chaugule — <a href="https://www.uber.com/en-DK/blog/kappa-architecture-data-stream-processing/">Designing a Production-Ready Kappa Architecture for Timely Data Stream Processing</a></li><li>Pramit Marattha — <a href="https://www.chaosgenius.io/blog/kappa-architecture/">Kappa Architecture 101 — Deep Dive into Stream-First Design</a></li><li>Michal Milosz — <a href="https://www.dsstream.com/post/designing-scalable-data-pipelines-batch-streaming-and-layered-architectures">Designing Scalable Data Pipelines: Batch, Streaming, and Layered Architectures</a></li><li>Jules S. Damji, Brooke Wenig, Tathagata Das, Denny Lee: <a href="https://www.oreilly.com/library/view/learning-spark-2nd/9781492050032/">Learning Spark, 2nd Edition</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=80492a509e1a" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/kappa-architecture-in-action-from-sensors-to-dashboards-with-kafka-spark-streaming-starrocks-80492a509e1a">Kappa Architecture in Action — From Sensors to Dashboards with Kafka, Spark Streaming, StarRocks…</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Spark Dev Environment with DevContainers and Spark Connect]]></title>
            <link>https://blog.dataengineerthings.org/spark-dev-environment-with-devcontainers-and-spark-connect-647da8b6f0f8?source=rss-8c7ffca79196------2</link>
            <guid isPermaLink="false">https://medium.com/p/647da8b6f0f8</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[docker-compose]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[George Zefkilis]]></dc:creator>
            <pubDate>Mon, 29 Sep 2025 19:23:24 GMT</pubDate>
            <atom:updated>2025-09-29T19:23:24.781Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*n_VQGzjXOJGdhQgQ6r8KeQ.jpeg" /></figure><p>In my previous<a href="https://medium.com/data-engineer-things/modernizing-databricks-setting-up-a-local-dev-environment-with-dev-containers-and-asset-bundles-8385b945be8c"> blog</a>, I explained how to set up a development environment for your Databricks workspace, leveraging Dev Containers and Asset Bundles for deploying code. The only catch with that approach is that it requires a paid Databricks account to run and test your code — which can get quite expensive.</p><p>To make Spark experimentation more accessible, I decided to create a self-contained Spark development environment that you can run locally using <strong>Dev Containers</strong> (my new favorite tool) and <a href="https://spark.apache.org/docs/latest/spark-connect-overview.html"><strong>Spark Connect</strong>.</a></p><p>You can check out the <a href="https://github.com/georgezefko/learning-spark">repo</a>.</p><blockquote>Note: Some of the parts here are repeated from the Databricks post</blockquote><h3>Setting Things Up</h3><p>To keep everything reproducible and portable, the project is structured around <strong>Dev Containers</strong> and <strong>Docker Compose</strong>. Dev Containers provide an isolated VS Code development environment, while Docker Compose orchestrates Spark Connect and any supporting services (like Kafka).</p><pre>├─ .devcontainer/         # VS Code Dev Container setup (devcontainer.json, Dockerfile)<br>├─ docker-compose.yml     # Orchestrates the services (dev + Spark Connect + kafka etc.)<br>├─ spark-connect/         # Spark Connect service image + config<br>│  ├─ Dockerfile<br>│  └─ conf/<br>│     ├─ core-site.xml<br>│     └─ spark-defaults.conf<br>├─ src/     # Example app code<br>│  ├─ main.py<br>├─ data/mm_dataset.csv    # Sample dataset<br>└─ Makefile               # Handy commands to build/run</pre><p>DevContainers has become my go-to solution for creating reproducible development environments. According to the official Dev Containers <a href="https://containers.dev/">documentation</a>:</p><blockquote><em>A </em><a href="https://containers.dev/"><em>development container</em></a><em> (or dev container for short) allows you to use a container as a full-featured development environment. It can be used to run an application, to separate tools, libraries, or runtimes needed for working with a codebase, and to aid in continuous integration and testing.</em></blockquote><p>Before we start, make sure you have <strong>Docker</strong> and the <strong>Dev Containers extension</strong> installed.</p><h4>devcontainer.json — configuring VS Code</h4><p>The .devcontainer folder should have the following structure (the Dockerfile is optional, depending on how you want to build your environment):</p><pre>.devcontainer/<br> ├─ Dockerfile          # builds the image<br> ├─ devcontainer.json   # configures VS Code to run the container</pre><p>devcontainer.json defines how VS Code runs and interacts with your containerized environment</p><p>I use <strong>Docker Compose</strong> instead of a single-container setup because it makes it easy to manage multiple services alongside Spark. For example, we can spin up <strong>Spark Connect</strong>, <strong>Kafka</strong>, <strong>Minio</strong>, and even <strong>StarRocks</strong> or <strong>Superset </strong>in one place. That way, the environment not only supports Spark development but also lets you test integrations with other systems.</p><p>Here’s the devcontainer.json I’m using:</p><pre>{<br>  &quot;name&quot;: &quot;spark-pyspark-dev&quot;,<br>  &quot;dockerComposeFile&quot;: [&quot;../docker-compose.yml&quot;],<br>  &quot;service&quot;: &quot;dev&quot;,<br>  &quot;runServices&quot;: [<br>    &quot;zookeeper&quot;, &quot;kafka&quot;, &quot;schema-registry&quot;,<br>    &quot;minio&quot;, &quot;starrocks-fe-0&quot;, &quot;starrocks-be-0&quot;, &quot;superset&quot;,<br>    &quot;spark-connect&quot;<br>  ],<br>  &quot;features&quot;: {<br><br> },<br>  &quot;workspaceFolder&quot;: &quot;/workspace&quot;,<br>  &quot;overrideCommand&quot;: false,<br>  &quot;updateRemoteUserUID&quot;: true,<br>  &quot;envFile&quot;: &quot;../.env&quot;,<br><br>  &quot;customizations&quot;: {<br>    &quot;vscode&quot;: {<br>      &quot;extensions&quot;: [<br>        &quot;ms-python.python&quot;,<br>        &quot;ms-toolsai.jupyter&quot;,<br>        &quot;ms-python.vscode-pylance&quot;,<br>        &quot;charliermarsh.ruff&quot;,<br>        &quot;GitHub.copilot&quot;,<br>        &quot;ms-azuretools.vscode-docker&quot;,<br>        &quot;tamasfe.even-better-toml&quot;<br>      ],<br>      &quot;settings&quot;: {<br>        &quot;python.defaultInterpreterPath&quot;: &quot;/workspace/.venv/bin/python&quot;,<br>        &quot;python.analysis.typeCheckingMode&quot;: &quot;basic&quot;,<br>        &quot;ruff.enable&quot;: true,<br>        &quot;python.testing.pytestEnabled&quot;: true,<br>        &quot;python.terminal.activateEnvironment&quot;: true<br>      }<br>    }<br>  },<br><br>  &quot;postCreateCommand&quot;: &quot;uv sync --locked  &amp;&amp; uv run pre-commit install&quot;,<br>  &quot;mounts&quot;: [<br>    &quot;source=uv-cache,target=/root/.cache/uv,type=volume&quot;<br>  ]<br>}</pre><ul><li><strong>dockerComposeFile</strong>: points to the docker-compose.yml where all services (Spark, Kafka, Minio, etc.) are defined.</li><li><strong>runServices</strong>: ensures that supporting services like Zookeeper, Schema Registry, or Superset start automatically with the dev container.</li><li><strong>customizations.vscode</strong>: pre-installs useful extensions (Python, Jupyter, Ruff, Docker, etc.) and sets sane defaults for linting, testing, and type checking.</li><li><strong>postCreateCommand</strong>: installs dependencies and pre-commit hooks right after the container is created.</li><li><strong>mounts</strong>: caches Python packages (via uv) between runs so you don’t waste time reinstalling.</li></ul><h4>The Dockerfile</h4><p>The Dockerfile isn’t strictly required (based on your project you could define everything directly in devcontainer.json), but I prefer it for keeping setup explicit, reproducible, and easier to maintain.</p><p>Here’s the Dockerfile I’m using:</p><pre>FROM python:3.12-slim<br><br>ENV DEBIAN_FRONTEND=noninteractive \<br>    PATH=&quot;/root/.local/bin:/workspace/.venv/bin:${PATH}&quot; \<br>    UV_LINK_MODE=copy<br><br># check uv container<br><br>RUN --mount=type=cache,target=/var/cache/apt \<br>    set -eux; \<br>    apt-get update; \<br>    apt-get install -y --no-install-recommends \<br>      curl bash git ca-certificates procps \<br>    &amp;&amp; rm -rf /var/lib/apt/lists/*<br><br># uv package manager<br>RUN curl -LsSf https://astral.sh/uv/install.sh | sh<br><br>WORKDIR /workspace<br><br>COPY pyproject.toml uv.lock ./<br><br># 1) Create venv and install ONLY dependencies<br>RUN --mount=type=cache,target=/root/.cache/uv \<br>    uv venv &amp;&amp; uv sync --locked --no-install-project --no-dev<br><br>COPY . .<br><br>RUN --mount=type=cache,target=/root/.cache/uv \<br>    uv sync --locked --no-dev<br><br><br>ENV PYSPARK_PYTHON=/workspace/.venv/bin/python \<br>    PYTHONUNBUFFERED=1 \<br>    PYTHONDONTWRITEBYTECODE=1<br><br># [OPTIONAL] This line is for auto-activation in interactive shells<br>RUN echo &quot;source /workspace/.venv/bin/activate&quot; &gt;&gt; /root/.bashrc<br><br><br># DevContainers and VS Code can control how the container is launched.<br>ENTRYPOINT []</pre><p>Below is step by step breakdown of Dockerfile’s key components which can look intimidating at first glance</p><ul><li><strong>Base image</strong>: python:3.12-slim keeps things lightweight.</li><li><strong>Utilities</strong>: only minimal tools (curl, git, etc.) are installed.</li><li><strong>uv</strong>: we use <a href="https://github.com/astral-sh/uv">uv</a> instead of pip/poetry for fast, reproducible dependency management. It creates and manages the .venv inside /workspace.</li><li><strong>Dependency caching</strong>: copying pyproject.toml and uv.lock first allows Docker to reuse the install layer if only code changes.</li><li><strong>Two-stage sync</strong>: first install dependencies only, then bring in the project itself, which speeds up rebuilds.</li><li><strong>Environment vars</strong>: ensure Spark uses the virtualenv’s Python, and disable buffering/.pyc files for a cleaner dev setup.</li><li><strong>No Spark installed here</strong>: we don’t include JVM/Spark installation because Spark Connect will provide the runtime driver (covered later).</li><li><strong>Auto-activating the venv</strong>:</li></ul><pre>RUN echo &quot;source /workspace/.venv/bin/activate&quot; &gt;&gt; /root/.bashrc</pre><p>This line makes new shells automatically activate the .venv, so the prompt shows (.venv) and python/pip default to the right interpreter. This is a convenience feature: even without it, everything works correctly because VS Code is already configured to use /workspace/.venv/bin/python, and uv can run commands directly inside the environment (uv run script.py, uv run pytest, etc.)</p><blockquote>Note: A couple of clarification on the uv set up. Firstuv is a new Python package/dependency manager (from the authors of Ruff) designed to be faster and more reliable than pip or Poetry.</blockquote><blockquote><strong>pyproject.toml</strong> → the project metadata and dependency list (similar to requirements.txt, but modern and standardized).</blockquote><blockquote><strong>uv.lock</strong> → a lock file generated by uv that pins exact versions for reproducibility.</blockquote><blockquote>If you’re starting a brand-new project and don’t already have a <em>pyproject.toml</em> and <em>uv.lock</em>, you’ll need to create them <strong>before</strong> you can build the dev container.</blockquote><blockquote>Run the following from your project root:</blockquote><blockquote>uv init<br>uv add pyspark pandas pyarrow etc.<br>uv lock</blockquote><blockquote>This generates the required <em>pyproject.toml</em> and <em>uv.lock</em> files. After that, you can reopen the project in DevContainers, and the environment will build correctly.</blockquote><blockquote>If you’re cloning my repo, don’t worry — the files are already included.</blockquote><h4>Docker Compose — Dev Service</h4><p>Docker Compose is a tool for defining and running multi-container applications. Instead of manually starting each service, you declare everything in a single docker-compose.yml file, and Compose brings them up together on a shared network.</p><p>In our setup, the <strong>dev service</strong> is the VS Code development container that runs alongside Spark Connect, Kafka, Minio, and others whenever you open the project in Dev Containers.</p><p>Here’s the relevant snippet (the full file is in the repo):</p><pre>  dev:<br>    build:<br>      context: .<br>      dockerfile: .devcontainer/Dockerfile<br>    container_name: dev<br>    command: sleep infinity<br>    working_dir: /workspace<br>    volumes:<br>      - .:/workspace:cached<br>      - uv-cache:/root/.cache/uv<br>      - spark-warehouse:/opt/spark/warehouse<br>    environment:<br>      PYSPARK_PYTHON: /workspace/.venv/bin/python<br>      KAFKA_BOOTSTRAP_SERVERS: ${KAFKA_BROKERS}<br>      SCHEMA_REGISTRY_URL: ${KAFKA_SCHEMA_REGISTRY}<br>      MINIO_ENDPOINT: http://minio:9000<br>      SPARK_CONNECT_URL: sc://spark-connect:15002<br>      MINIO_ACCESS_KEY: ${MINIO_ACCESS_KEY}<br>      MINIO_SECRET_KEY: ${MINIO_SECRET_KEY}<br>    depends_on: [kafka, schema-registry, minio, spark-connect]<br>    networks: [spark_network]</pre><p>In general in each service can be given build instructions, volumes, environment variables, networking, and dependencies — everything needed to run side by side on a shared network. Below are the key parts of our dev service.</p><ul><li><strong>build</strong>: uses our .devcontainer/Dockerfile to build the image.</li><li><strong>command: sleep infinity</strong>: keeps the container idle until VS Code attaches.</li><li><strong>volumes</strong>: mounts project code, a persistent uv cache, and Spark’s warehouse directory.</li><li><strong>environment</strong>: passes variables so Spark uses the correct Python, and services like Kafka and Minio are reachable.</li><li><strong>depends_on</strong>: makes sure supporting services are up before the dev container starts.</li><li><strong>networks</strong>: connects everything on spark_network for service-to-service communication.</li></ul><h3>Spark-Connect</h3><p>Now that we have the basics of our development environment in place, the next step is configuring the main component that will actually let us run Spark: <strong>Spark Connect</strong>.</p><p>Spark Connect introduces a <strong>client–server architecture</strong> that decouples the Spark driver from the client code. Instead of running a full Spark cluster inside Docker Compose, we run a single Spark server container and connect to it from our dev container (or even directly from VS Code/Jupyter).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*3c2tD966C4lOH_dFmbf2mg.png" /><figcaption>Image taken from <a href="https://spark.apache.org/docs/latest/spark-connect-overview.html">here</a></figcaption></figure><p>This brings clear benefits for local development:</p><ul><li><strong>Developer Experience</strong> — Run Spark code interactively from IDEs and notebooks.</li><li><strong>Simplified Setup</strong> — Only one Spark service to configure, no full cluster emulation.</li><li><strong>Stability &amp; Isolation</strong> — Client crashes won’t bring Spark down.</li><li><strong>Easier Upgrades</strong> — Clients remain compatible across Spark upgrades.</li></ul><h4>Spark-Connect Dockerfile</h4><p>Inside spark-connect/, I use a dedicated <strong>Dockerfile</strong> for the Spark Connect server. You <em>could</em> configure everything inline in docker-compose.yml, but as you have may noticed I like to keep things modular. The Dockerfile keeps OS-level packages, Python libs, and Spark config changes <strong>explicit, reproducible, and cacheable</strong>.</p><pre>FROM apache/spark:4.0.1<br><br>USER root<br>RUN apt-get update &amp;&amp; apt-get install -y --no-install-recommends \<br>      python3 python3-pip &amp;&amp; \<br>    pip3 install --no-cache-dir \<br>      pandas&gt;=2.0.0 \<br>      pyarrow \<br>      grpcio&gt;=1.74.0 \<br>      grpcio-status&gt;=1.74.0 \<br>      requests &amp;&amp; \<br>    ln -sf /usr/bin/python3 /usr/bin/python &amp;&amp; \<br>    rm -rf /var/lib/apt/lists/*<br><br>USER spark #optional for dev purposes<br>ENV PYSPARK_PYTHON=/usr/bin/python3<br>ENV PYSPARK_DRIVER_PYTHON=/usr/bin/python3<br><br>COPY ./conf/spark-defaults.conf &quot;$SPARK_HOME/conf/spark-defaults.conf&quot;<br>COPY ./conf/core-site.xml &quot;$SPARK_HOME/conf/core-site.xml&quot;</pre><p>The base image apache/spark:4.0.1 provides a prebuilt Spark runtime, saving us from building Spark ourselves. On top of that, I install Python 3 along with a handful of useful libraries such as pandas, pyarrow, and the grpcio packages required by Spark Connect’s gRPC protocol. The environment variables PYSPARK_PYTHON and PYSPARK_DRIVER_PYTHONmake sure Spark uses this interpreter consistently. Finally, I copy in two config files: spark-defaults.conf to define Spark defaults, and core-site.xml for Hadoop/S3 connectivity — in my case this was needed to fix timeout issues when connecting to MinIO.</p><p>Since Spark Connect already enforces a client–server split, the dev container and the Spark server image are naturally separate. This makes the setup lighter and clearer: the dev container doesn’t need Spark installed, it simply connects to the server over gRPC.</p><p>There are two small details worth calling out. First, the image defaults to a non-root user, so if you try to run apt-get directly you’ll hit a <em>permission denied</em> error. To fix this, the Dockerfile briefly switches to root to install system packages, then you can also switch back to the non-root spark user that comes with the base image, which is a sensible practice if you adapt this setup for anything beyond local development.</p><p>Second, while Spark Connect is designed for cross-version compatibility, it’s still best to align the <strong>PySpark version in your dev container</strong> with the <strong>Spark server image version</strong> (e.g. pyspark==4.0.1 with apache/spark:4.0.1) to avoid subtle differences in APIs or SQL behaviour.</p><h4>Docker Compose — Spark Connect Service</h4><p>Now that we have the image in place, we need to define the Spark Connect service in docker-compose.yml:</p><pre>  spark-connect:<br>    build: ./spark-connect<br>    container_name: spark-connect<br>    command:<br>      - /opt/spark/sbin/start-connect-server.sh<br>      - --packages<br>      - org.apache.spark:spark-sql-kafka-0-10_2.13:4.0.1,org.apache.hadoop:hadoop-aws:3.3.6,com.amazonaws:aws-java-sdk-bundle:1.12.262,com.mysql:mysql-connector-j:8.0.33<br>      - --conf<br>      - spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider<br>      - --conf<br>      - spark.hadoop.fs.s3a.access.key=${MINIO_ACCESS_KEY}<br>      - --conf<br>      - spark.hadoop.fs.s3a.secret.key=${MINIO_SECRET_KEY}<br>    environment:<br>      - SPARK_NO_DAEMONIZE=true<br>    ports:<br>      - &quot;15002:15002&quot;<br>      - &quot;4040-4045:4040-4045&quot;<br>    volumes:<br>      - ./data:/opt/spark/work-dir/data<br>      - spark-warehouse:/opt/spark/warehouse<br>      - spark-checkpoints:/tmp/spark-checkpoints<br>    depends_on:<br>      - kafka<br>      - schema-registry<br>      - minio<br>    networks:<br>      - spark_network</pre><p>This service builds from the spark-connect/ Dockerfile and runs the Spark Connect server. The command launches Spark with extra packages for common integrations: spark-sql-kafka for Kafka, hadoop-aws plus the AWS SDK for MinIO/S3, and mysql-connector-j for JDBC access. The --conf entries configure the S3A connector to authenticate with the MinIO credentials defined in our .env file.</p><p>The environment variable SPARK_NO_DAEMONIZE=true ensures Spark stays in the foreground, so Docker can manage its lifecycle. Ports are mapped so the Spark Connect endpoint (15002) and the Spark UI (4040–4045) are available locally. Volumes persist the warehouse directory and streaming checkpoints, while the data mount provides access to demo datasets — this is necessary because Spark Connect cannot directly read files from the client container (we will use that in the example shortly). Finally, depends_on guarantees that Kafka, Schema Registry, and MinIO are running before Spark Connect starts, and networksplaces the service on the shared spark_network for communication with other containers.</p><p>With this setup, Spark Connect runs as a dedicated service inside Docker Compose. The dev container connects to it at sc://spark-connect:15002, giving us a lightweight but fully functional local Spark environment. Compared to running a full Spark cluster (driver and workers) inside Docker Compose, this approach is far simpler to configure and still provides everything we need for development.</p><p>Below is a graphical representation of how all the services linked together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ncl_hrSc2BYXJxDCcLClzw.png" /><figcaption>Image created by the author</figcaption></figure><h3>Run a spark application</h3><p>With everything in place, let’s run a simple Spark job to make sure the environment works.</p><p>First open the project in VS Code and select <strong>“Reopen in Container”</strong>. If everything is configured correctly, VS Code will build the dev environment and attach to the dev container. You should see something like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ID9GOFgNUvxR_a3DmHOKJQ.png" /></figure><p>On the bottom-right corner of VS Code, you should see the <strong>Dev Container status bar</strong> indicating that you are working inside the container.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/290/1*PJvFlKT3mTfUple2tiGsFg.png" /></figure><p>Under src/, I’ve added a main.py based on the classic <a href="https://github.com/databricks/LearningSparkV2/blob/master/chapter2/py/src/mnmcount.py"><em>Learning Spark</em> </a>example: <em>Counting M&amp;Ms for the Cookie Monster</em>. The only difference is how we create the Spark session:</p><pre>from pyspark.sql import SparkSession<br><br># Connect to Spark Connect server<br>spark = (<br>    SparkSession.builder.remote(&quot;sc://spark-connect:15002&quot;)<br>    .appName(&quot;learning-spark&quot;)<br>    .getOrCreate()<br>)<br></pre><p>The key detail here is that we connect to the Spark Connect service on port 15002. Unlike a local setup, we don’t call .master(&quot;local[*]&quot;) — the client delegates execution to the Spark Connect server.</p><p>To run the script inside the dev container, use on of the following options:</p><pre>uv run src/main.py data/mm_dataset.csv <br><br>python src/main.py data/mm_dataset.csv<br><br>make run_spark</pre><p>You should see the following as part of the output in terminal. At the same time, you can open <a href="http://localhost:4040/">http://localhost:4040</a> in your browser to view the Spark UI. This gives you visibility into the job stages, execution plan, and resources, exactly as you would on a remote Spark cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/320/1*J3uqEq7VtUCCP8vfZX87eg.png" /></figure><p>And that’s it! You now have a <strong>fully containerized Spark development environment</strong> with Spark Connect, Kafka, Minio, and supporting services — all running locally and fully reproducible. In the next tutorial, I’ll show you how to build an <strong>end-to-end pipeline</strong> using this setup.</p><h3>Conclusion</h3><p>By combining <strong>Dev Containers</strong>, <strong>Docker Compose</strong>, and <strong>Spark Connect</strong>, we’ve built a fully containerized Spark development environment that runs entirely on your local machine. The setup provides several benefits.</p><p>First, it offers a reproducible VS Code workspace with Python, Spark Connect, and supporting services like Kafka and MinIO. Second, it features a modular design where the development container remains lightweight while Spark operates as a dedicated service. Finally, it delivers the same developer experience you’d expect from a remote Spark cluster — interactive coding, job monitoring through the Spark UI, and integration with external systems — all without the cost of a cloud workspace.</p><p>This approach makes it easy to experiment with Spark, prototype integrations without fighting with local installs or the expense of a paid Databricks environment.</p><p><strong>Thanks for reading!</strong></p><p>I hope you found this guide useful and that it inspires you to build and extend your own local Spark setups.</p><p>Your feedback is always appreciated and helps improve these posts.<br>📦 Check out the code in the <a href="https://github.com/georgezefko/learning-spark">GitHub repository</a><br>🤝 Feel free to connect with me on <a href="http://www.linkedin.com/in/georgioszefkilis">LinkedIn</a></p><h3>References</h3><ol><li>Spark-Connect official docs: <a href="https://spark.apache.org/docs/latest/spark-connect-overview.html">Spark Connect overview</a></li><li>DevContainer official docs: <a href="https://containers.dev/">Containers</a></li><li><a href="https://medium.com/u/b802b89c899e">Vu Trinh</a>: <a href="https://medium.com/data-engineer-things/is-this-feature-a-revolution-in-spark-29636b690ea7">Is this feature a revolution in Spark?</a></li><li><a href="https://medium.com/u/11440077572f">Sergey Kotlov</a>: <a href="https://medium.com/data-science/adopting-spark-connect-cdd6de69fa98">Adopting Spark Connect</a></li><li>UV Docker docs: <a href="https://docs.astral.sh/uv/guides/integration/docker/">Docker Integration</a></li><li>Jules S. Damji, Brooke Wenig, Tathagata Das, Denny Lee: <a href="https://www.oreilly.com/library/view/learning-spark-2nd/9781492050032/">Learning Spark, 2nd Edition</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=647da8b6f0f8" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/spark-dev-environment-with-devcontainers-and-spark-connect-647da8b6f0f8">Spark Dev Environment with DevContainers and Spark Connect</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>