<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Evan Rosa on Medium]]></title>
        <description><![CDATA[Stories by Evan Rosa on Medium]]></description>
        <link>https://medium.com/@evan-rosa?source=rss-443f6a356fab------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*5ynTY3AmPsEiKnmZBt-iFg.jpeg</url>
            <title>Stories by Evan Rosa on Medium</title>
            <link>https://medium.com/@evan-rosa?source=rss-443f6a356fab------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 17:41:34 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@evan-rosa/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Apache Spark 4.0: New Features for Data Engineers | The Inner Join Blog By Evan Rosa]]></title>
            <link>https://evan-rosa.medium.com/apache-spark-4-0-new-features-for-data-engineers-the-inner-join-blog-by-evan-rosa-76f7af16facc?source=rss-443f6a356fab------2</link>
            <guid isPermaLink="false">https://medium.com/p/76f7af16facc</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Evan Rosa]]></dc:creator>
            <pubDate>Sun, 08 Jun 2025 00:00:51 GMT</pubDate>
            <atom:updated>2025-06-09T14:03:40.928Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_mm9gZt264DxXU5x.png" /></figure><blockquote><em>Apache Spark has long been the powerhouse behind batch processing, large-scale analytics, and streaming jobs. With Spark 4.0, it’s not just faster; it’s smarter, cleaner, and more developer-friendly.</em></blockquote><p>If you’ve ever built an ETL pipeline that runs through terabytes of data, you know how crucial performance and flexibility are. Apache Spark 4.0 doubles down on both fronts, delivering new architecture paradigms, deeper SQL capabilities, and better tooling for developers. This isn’t just a version bump, it’s a foundational shift in how we think about distributed computing for data engineering.</p><p>Let’s explore what Spark 4.0 brings to the table and why it’s worth your attention.</p><h3>What is Apache Spark?</h3><p>Apache Spark is a distributed computing framework for large-scale data processing. It allows data engineers to process data in-memory across a cluster of machines using high-level APIs in Python, Scala, SQL, and Java.</p><p>Spark powers everything from batch ETL pipelines and data lakes to streaming analytics and ML model training. If you’ve used tools like Databricks or AWS EMR, chances are Spark was running under the hood.</p><h3>Why Spark 4.0 Matters</h3><p>Spark is the beating heart of modern big data infrastructure. Whether it’s powering batch jobs in enterprise ETL pipelines, real-time analytics for certain scenarios, or iterative machine learning at scale, Spark has earned its place. The open-source engine combines speed (thanks to in-memory execution), scale (via distributed processing), and expressiveness (with APIs in Scala, Java, Python, SQL, and R). But with Spark 4.0, it’s clear the project is pivoting to make the developer experience just as powerful as the engine itself.</p><h3>Spark Connect: Decoupling the Client from the Cluster</h3><p>One of the most revolutionary updates in Spark 4 is Spark Connect, a client-server protocol that allows Spark code to be executed remotely.</p><p>This means your Spark application no longer needs to run in the same process or JVM as the Spark driver. Developers can now build lightweight clients in Python or other languages that connect to a central Spark server.</p><p><strong>Why it matters:</strong></p><ul><li>Enables remote execution from notebooks, IDEs, or microservices</li><li>Supports multi-tenant environments and secure access patterns</li><li>Lowers setup complexity for development and experimentation</li></ul><p>Think of Spark Connect as the REST API era for Spark; decoupled, scalable, and programmable.</p><h3>SQL Language Enhancements: Writing SQL Just Got Nicer</h3><p>Spark 4.0 expands the SQL dialect in meaningful ways, especially for data engineers who rely on SQL-first development.</p><p><strong>New capabilities include:</strong></p><ul><li>SQL Pipe (|&gt; operator): Chain transformations in a readable way</li><li>Session Variables: Set and reuse values across SQL sessions</li><li>Multi-Statement Scripts: Run multiple SQL commands in a batch (great for notebooks and deployments)</li><li>Scripting Support: Embed logic and control flow inside SQL scripts</li><li>ANSI Mode by Default: Enforces consistent behavior across SQL engines (goodbye, silent type coercions)</li></ul><p>This closes the gap between Spark SQL and traditional database SQL engines, making Spark more portable and standards-compliant.</p><h3>Python API Improvements: PySpark Gets a Boost</h3><p>The Python ecosystem has always been a first-class citizen in Spark. With Spark 4.0, PySpark gets both performance upgrades and new interfaces:</p><p><strong>Key features:</strong></p><ul><li>Polymorphic UDTFs (User-Defined Table Functions): Write functions that return variable-length tables</li><li>Data Source V2 Python API: Build custom connectors in pure Python</li><li>Plotting API: Quick visualizations inside PySpark notebooks</li></ul><p>Together, these upgrades let Python users do more with fewer dependencies and contribute more easily to the broader Spark dataflow.</p><h3>Observability and Debugging: Easier to Understand What’s Going On</h3><p>Spark 4.0 improves the developer experience with enhanced debugging tools:</p><ul><li>Structured Logging: Consistent logs across stages and components</li><li>Query Plan Diffing: See how your query plans change between runs</li><li>Improved Error Messages: More context and actionable suggestions</li></ul><p>These might not be glamorous, but they make a huge difference when you’re knee-deep in production issues at 3 AM.</p><h3>Streaming Enhancements: Better State Management</h3><p>While batch processing is still Spark’s bread and butter, streaming use cases continue to grow.</p><p>In Spark 4.0:</p><ul><li>State Store Checkpoint v2 improves fault-tolerance and replay</li><li>State Store as a Source allows reusing intermediate states in downstream jobs</li></ul><p>This is particularly useful for applications like fraud detection or anomaly monitoring, where stateful stream processing is the backbone.</p><h3>Variant Data Types: Handle Semi-Structured Data Gracefully</h3><p>A long-standing challenge in Spark has been handling data with irregular or mixed schemas (e.g., JSON logs, nested arrays).</p><p>Spark 4.0 introduces variant data types, which:</p><ul><li>Allow fields of mixed types in a single column</li><li>Provide schema-on-read flexibility</li><li>Improve compatibility with cloud data lakes</li></ul><p>This makes Spark more resilient when processing unpredictable or user-generated data.</p><h3>Compatibility and Migration</h3><p>Spark 4.0 is not a drop-in replacement. There are API changes and deprecations you need to watch for. But the core abstractions like RDDs, DataFrames, Datasets remain intact.</p><p><strong>Tips for migrating:</strong></p><ul><li>Test your DAGs and pipelines in a staging environment</li><li>Use SQL ANSI mode for future-proof SQL</li><li>Adopt Spark Connect where appropriate (e.g., notebooks or stateless apps)</li></ul><p>For most workloads, the migration will be worth it: lower latency, better resource management, and cleaner code.</p><h3>Final Thoughts: The Future of Big Data Is Modular</h3><p>Apache Spark 4.0 isn’t just about performance, it’s about developer velocity. With Spark Connect, SQL upgrades, and stronger Python support, it’s easier to build, debug, and deploy complex data applications.</p><p>The decoupled architecture signals a shift: Spark is no longer a monolith. It’s evolving into a modular platform that integrates smoothly with data lakes, warehouses, notebooks, and services.</p><p>If you’re already using Spark 3.x, now’s the time to start experimenting with Spark 4. If you’re new to Spark, there’s never been a better time to dive in.</p><p>Want help migrating your pipelines to Spark 4 or modernizing your batch architecture?</p><p><em>Originally published at </em><a href="https://www.evro.dev/blog/apache-spark-4-new-features-for-data-engineers"><em>https://www.evro.dev</em></a><em> on June 8, 2025.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=76f7af16facc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Getting Started with SQLMesh: Versioned, Incremental Data Transformations Made Simple | The Inner…]]></title>
            <link>https://evan-rosa.medium.com/getting-started-with-sqlmesh-versioned-incremental-data-transformations-made-simple-the-inner-bd86f66df63d?source=rss-443f6a356fab------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd86f66df63d</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-tools]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[sqlmesh]]></category>
            <dc:creator><![CDATA[Evan Rosa]]></dc:creator>
            <pubDate>Mon, 12 May 2025 00:00:22 GMT</pubDate>
            <atom:updated>2025-05-12T16:31:10.180Z</atom:updated>
            <content:encoded><![CDATA[<h3>Getting Started with SQLMesh: Versioned, Incremental Data Transformations Made Simple | The Inner Join Blog By Evan Rosa</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3rXzRvTmPmNc9gru.png" /></figure><blockquote><em>SQLMesh isn’t just another SQL framework. It’s a rethink of how we build, deploy, and evolve data pipelines, without breaking things.</em></blockquote><p>If you’ve ever dealt with brittle DAGs, painful backfills, or unexpected schema changes in production, SQLMesh might be the tool you’ve been waiting for.</p><p>In this guide, I’ll walk you through:</p><ul><li>What SQLMesh is</li><li>Why it matters</li><li>Key concepts that make it powerful</li><li>How to implement it</li><li>When to use it (and when not to)</li></ul><h3>What is SQLMesh?</h3><p>SQLMesh is an open-source framework for building SQL pipelines with:</p><ul><li><strong>Version control</strong> (models as code)</li><li><strong>Incremental execution</strong> (smart reprocessing)</li><li><strong>Environment isolation</strong> (dev, staging, prod)</li><li><strong>Schema evolution</strong> tracking</li></ul><p>It helps you iterate fast, test safely, and deploy with confidence-without reinventing your pipeline every time your logic changes.</p><h3>Why Use SQLMesh?</h3><p>Most pipelines break because of one of these:</p><ul><li>Manual dependency management</li><li>No isolation between dev and prod</li><li>Inefficient full refreshes</li><li>Lack of change awareness</li></ul><p>SQLMesh solves this by:</p><ul><li>Tracking dependencies automatically</li><li>Supporting Git-based environments</li><li>Letting you reprocess <strong>only what changed</strong></li><li>Integrating with modern orchestration (Airflow, Dagster, etc.)</li></ul><p>Think of it as <strong>dbt meets Git</strong>, with real CI/CD principles.</p><h3>Key Concepts</h3><h4>1. Models as Files</h4><p>Every model is a .sql file + optional Python metadata.</p><pre>-- models/orders.sql SELECT * FROM raw.orders WHERE order_date &gt;= &#39;2023-01-01&#39;</pre><h4>2. Environment Support</h4><p>Develop locally without impacting production:</p><pre>sqlmesh plan --environment dev sqlmesh apply</pre><h4>3. Incremental Execution</h4><p>SQLMesh knows what needs to be rebuilt:</p><ul><li>It checks for logic changes</li><li>It analyzes dependencies</li><li>It only runs what’s affected</li></ul><h4>4. Versioned Planning</h4><p>Before applying changes, SQLMesh shows you the full plan:</p><ul><li>Models added, changed, removed</li><li>Rows to be reprocessed</li><li>Upstream impacts</li></ul><pre>sqlmesh plan --start &#39;2024-01-01&#39; --end &#39;2024-01-15&#39;</pre><h4>5. Time-Aware Backfills</h4><p>You can backfill incrementally:</p><pre>sqlmesh backfill --start &#39;2024-04-01&#39; --end &#39;2024-04-30&#39;</pre><h3>Getting Started with SQLMesh</h3><h4>Step 1: Install</h4><pre>pip install sqlmesh</pre><h4>Step 2: Scaffold a Project</h4><pre>sqlmesh init</pre><p>This will create:</p><ul><li>models/ - where your SQL models live</li><li>sqlmesh.yaml - main config for model paths, connections, default dialect, and environments</li><li>tests/ - where you can define model tests (e.g., row count or column checks)</li></ul><p>Example sqlmesh.yaml:</p><pre>model_defaults: dialect: snowflake start: 2024-01-01 gateways: prod: connection: type: snowflake account: my-account.region user: my-user password: ${SNOWFLAKE_PASSWORD}</pre><h4>Step 3: Define a Model</h4><p>Create a file in models/:</p><pre>-- models/daily_users.sql SELECT user_id, COUNT(*) as login_count, DATE_TRUNC(&#39;day&#39;, login_time) as login_day FROM raw.user_logins GROUP BY user_id, login_day</pre><h4>Step 4: Plan &amp; Apply</h4><pre>sqlmesh plan --environment dev sqlmesh apply</pre><p>This builds the model in your staging schema.</p><h4>Step 5: Promote to Prod</h4><p>Once validated, switch to prod:</p><pre>sqlmesh plan --environment prod sqlmesh apply</pre><h3>How SQLMesh Fits Into Your Workflow</h3><p>SQLMesh supports modern CI/CD and orchestration patterns:</p><ul><li>Develop and test changes in a feature branch using dev or staging environments</li><li>Integrate sqlmesh plan and sqlmesh apply into GitHub Actions or GitLab CI</li><li>Schedule regular runs via Airflow, Dagster, or cron</li><li>Use backfill commands to rerun specific time windows</li></ul><p>This enables versioned, testable, and repeatable data deployments across teams.</p><h3>When to Use SQLMesh</h3><p><strong>Use SQLMesh if you:</strong></p><ul><li>Want safer deploys with environments</li><li>Need dependency tracking across models</li><li>Prefer Git-based, testable pipelines</li><li>Are building production-grade data systems</li></ul><p><strong>Don’t use SQLMesh if you:</strong></p><ul><li>Only run a few ad-hoc queries</li><li>Don’t use Git or CI/CD for data</li><li>Need only light transformation logic (just a BI tool)</li></ul><h3>Final Thoughts</h3><p>SQLMesh brings the discipline of software engineering to your SQL models:</p><ul><li>Version control</li><li>Dependency awareness</li><li>Incremental execution</li><li>Dev → staging → prod promotion</li></ul><p>It’s not just about writing SQL, it’s about managing change safely.</p><p>If you’re building serious pipelines and want fewer surprises in production, it’s worth adding SQLMesh to your stack.</p><p>🔧 Want help implementing SQLMesh in your workflow? <a href="https://www.evro.dev/consultant">Let’s connect</a></p><p><em>Originally published at </em><a href="https://www.evro.dev/blog/getting-started-with-sqlmesh"><em>https://www.evro.dev</em></a><em> on May 12, 2025.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd86f66df63d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Choosing the Right Tools for Your Data Stack: A Practical Guide for Modern Teams | The Inner Join…]]></title>
            <link>https://evan-rosa.medium.com/choosing-the-right-tools-for-your-data-stack-a-practical-guide-for-modern-teams-the-inner-join-d00a55eabd7c?source=rss-443f6a356fab------2</link>
            <guid isPermaLink="false">https://medium.com/p/d00a55eabd7c</guid>
            <category><![CDATA[data-strategy]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[modern-data-stack]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Evan Rosa]]></dc:creator>
            <pubDate>Tue, 06 May 2025 00:00:43 GMT</pubDate>
            <atom:updated>2025-05-07T12:11:08.903Z</atom:updated>
            <content:encoded><![CDATA[<h3>Choosing the Right Tools for Your Data Stack: A Practical Guide for Modern Teams | The Inner Join Blog By Evan Rosa</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iLvC6EVbQ3aDBz3J.png" /></figure><blockquote><em>Picking tools for your data stack isn’t about what’s popular, it’s about what’s right for your team.</em></blockquote><p>When building scalable data systems, the tools you choose significantly impact your speed, reliability, and overall success. In this guide, I’ll walk through the critical categories of data engineering tools, highlight their pros and cons, and explain when and why you’d choose each one.</p><h3>1. Data Ingestion Tools</h3><p>Ingestion tools move data into your ecosystem from various sources.</p><ul><li><strong>Kafka / Pulsar</strong>: For real-time, high-throughput streaming.</li><li><strong>Fivetran / Stitch</strong>: Managed, reliable ingestion from SaaS applications.</li><li><strong>Debezium</strong>: Real-time CDC from relational databases.</li></ul><p><strong>When to Use</strong>:</p><ul><li>Kafka for real-time event streams.</li><li>Fivetran/Stitch if you prefer hands-off management and simplicity.</li><li>Debezium for capturing database changes in real-time.</li></ul><h3>2. Data Storage &amp; Lakehouse</h3><p>Data lakes and warehouses handle your storage and querying needs.</p><ul><li><strong>Snowflake / BigQuery / Redshift</strong>: Fully managed cloud data warehouses for fast analytics.</li><li><strong>Apache Iceberg / Delta Lake</strong>: For scalable, open lakehouse architectures with schema evolution.</li></ul><p><strong>When to Use</strong>:</p><ul><li>Cloud data warehouses for straightforward analytics and rapid SQL-based querying.</li><li>Iceberg or Delta Lake when you need scalability, open formats, and multi-engine querying.</li></ul><h3>3. Workflow Orchestration Tools</h3><p>Orchestration tools handle the scheduling and monitoring of your workflows.</p><ul><li><strong>Airflow</strong>: Mature, widely-adopted, Python-based DAG scheduling.</li><li><strong>Dagster / Prefect</strong>: Modern workflow orchestration emphasizing simplicity, observability, and flexibility.</li></ul><p><strong>When to Use</strong>:</p><ul><li>Airflow for robust, complex DAG management with extensive community support.</li><li>Dagster/Prefect if your team prefers simpler codebases, rapid iteration, and built-in monitoring.</li></ul><h3>4. Data Transformation &amp; Modeling</h3><p>Transform data into useful, structured assets.</p><ul><li><strong>dbt</strong>: Popular SQL-based transformation framework for analytics engineers.</li><li><strong>SQLMesh</strong>: Powerful for complex transformations, incremental updates, and schema versioning.</li><li><strong>Apache Spark</strong>: Robust, scalable framework for big data batch and real-time stream transformations.</li><li><strong>Apache Flink</strong>: Specialized for real-time stream processing and complex event-driven transformations.</li></ul><p><strong>When to Use</strong>:</p><ul><li>dbt for straightforward analytics engineering and SQL-centric workflows.</li><li>SQLMesh for incremental pipelines, complex transformations, and sophisticated model management.</li><li>Apache Spark for heavy data transformations, analytics at scale, batch processing, and unified batch-streaming workloads.</li><li>Apache Flink for low-latency, stateful real-time stream processing and event-driven use cases.</li></ul><h3>5. Serving Layer &amp; Analytics Tools</h3><p>Delivering data to end-users in an accessible, performant manner.</p><ul><li><strong>Looker / Tableau / Power BI</strong>: Dashboards and business intelligence.</li><li><strong>Apache Superset</strong>: Open-source BI for SQL-savvy teams.</li><li><strong>Redis / Pinot / ClickHouse</strong>: Low-latency analytical queries for interactive apps.</li></ul><p><strong>When to Use</strong>:</p><ul><li>BI tools (Looker/Tableau/Power BI) for executive dashboards and visual storytelling.</li><li>Superset for flexible, open-source SQL-first dashboards.</li><li>Redis/Pinot/ClickHouse for high-performance analytical APIs or real-time analytics.</li></ul><h3>Decision Matrix: A Quick Reference</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*trCeRMuPwC6JDLK4LGyHVA.jpeg" /></figure><h3>Conclusion</h3><p>Choosing your tools wisely isn’t just about current needs-it’s about future growth. Align your choices with team skills, scale expectations, and operational complexity to build a powerful, sustainable data platform.</p><p>Ready to refine your stack? Let’s connect and build together.</p><p><em>Originally published at </em><a href="https://www.evro.dev/blog/data-stack-tool-selection-guide"><em>https://www.evro.dev</em></a><em> on May 6, 2025.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d00a55eabd7c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deep Dive: Enterprise-Grade Architecture for Fresh, Fast, Scalable Data | The Inner Join Blog By…]]></title>
            <link>https://evan-rosa.medium.com/deep-dive-enterprise-grade-architecture-for-fresh-fast-scalable-data-the-inner-join-blog-by-098bc4ae500d?source=rss-443f6a356fab------2</link>
            <guid isPermaLink="false">https://medium.com/p/098bc4ae500d</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-infrastructure]]></category>
            <dc:creator><![CDATA[Evan Rosa]]></dc:creator>
            <pubDate>Tue, 29 Apr 2025 00:00:36 GMT</pubDate>
            <atom:updated>2025-05-05T15:33:56.352Z</atom:updated>
            <content:encoded><![CDATA[<h3>Deep Dive: Enterprise-Grade Architecture for Fresh, Fast, Scalable Data | The Inner Join Blog By Evan Rosa</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/0*R-qS4KpAwkC8dEN2.png" /></figure><blockquote><em>In enterprise, “it works” isn’t enough… You need “it works at scale, fast, and without breaking.”</em></blockquote><p>Today’s article we discuss <strong>design patterns</strong>, <strong>infrastructure components</strong>, and <strong>operational best practices</strong> that make a modern data platform <strong>enterprise-grade</strong>. We will cover:</p><ol><li>Key challenges and patterns</li><li>High-level architecture blueprint</li><li>In-the-weeds patterns for ingestion, processing, and serving</li><li>Governance, testing, and observability</li></ol><h3>1. Enterprise Challenges &amp; Proven Patterns</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/1*s7YMrfYwxxdK9Q4W4cSLsQ.jpeg" /></figure><h3>2. Architecture Blueprint</h3><p><strong>Ingestion</strong></p><ul><li><strong>CDC</strong>: Debezium/Fivetran capture DML from OLTP and stream to Kafka.</li><li><strong>API/webhook events</strong>: Producers push directly into Kafka.</li></ul><p><strong>StreamCompute (Kappa)</strong></p><ul><li><strong>Flink</strong> consumes raw topics, applies watermarks, windowed dedupe, enriches with lookup state (e.g., reference tables).</li><li>Writes out an <strong>append-only Bronze layer</strong> in Iceberg for audit and replay.</li></ul><p><strong>BatchCompute</strong></p><ul><li><strong>Silver tables</strong>: cleaned, typed, deduped.</li><li><strong>Gold tables/views</strong>: dimensional models, aggregated down to business granularity.</li></ul><p><strong>Serving</strong></p><ul><li>BI tools hit Gold materialized views for sub-second dashboards.</li><li>ML pipelines read Silver for feature engineering.</li></ul><h3>3. In-the-Weeds Design Patterns</h3><h4>A. Append-Only Bronze + Periodic Merge</h4><ul><li><strong>Write</strong>: every event → new Iceberg file, partitioned by date/tenant.</li><li><strong>Compact &amp; Merge</strong> (Silver process):</li></ul><pre>MERGE INTO silver.table AS S<br>USING (SELECT * FROM bronze.table WHERE _partition = &#39;2025-04-29&#39;) AS B<br>  ON S.event_id = B.event_id<br>WHEN MATCHED THEN UPDATE SET *<br>WHEN NOT MATCHED THEN INSERT *</pre><ul><li><strong>Benefits</strong>: high-throughput writes, backfill late data, audit trail.</li></ul><h4>B. Stateful Streaming with Watermarks</h4><ul><li><strong>Watermark</strong> at event-time minus allowed lateness (e.g., 5m).</li><li><strong>Deduplicate</strong> in Flink:</li></ul><pre>keyedStream<br>  .assignTimestampsAndWatermarks(watermarkStrategy)<br>  .keyBy(Event::getId)<br>  .process(new DeduplicateFunction())</pre><ul><li><strong>Outcome</strong>: exactly-once, order-corrected streams.</li></ul><h4>C. Kappa Architecture</h4><ul><li><strong>Single code path</strong>: all transformations expressed in Flink + SQLMesh, no separate batch pipeline.</li><li><strong>Replay</strong>: To rebuild Silver/Gold, replay Kafka from offset 0 into SQLMesh jobs.</li></ul><h4>D. Partitioning &amp; Clustering</h4><ul><li><strong>Partition by</strong>:</li><li>ingestion_date (daily/hourly folders)</li><li>tenant_id (multi-tenant isolation)</li><li><strong>Cluster / Z-order</strong> on high-cardinality columns (user_id, product_id) to accelerate selective queries.</li></ul><h4>E. CQRS for Serving Layer</h4><ul><li><strong>Command</strong> (writes): flow through Bronze/Silver → main tables.</li><li><strong>Query</strong> (reads): pre-computed Gold views or caches (Redis, Pinot) for ultra-low latency.</li></ul><h3>4. Governance, Testing &amp; Observability</h3><h4>Data Contracts</h4><ul><li>Schema Registry enforces field types and evolution policies.</li><li>Kafka Schema Registry + Iceberg schema validation.</li></ul><h4>Automated Testing</h4><ul><li><strong>Unit tests</strong> for SQLMesh models (schema diff, row count).</li><li><strong>Integration tests</strong>: run mini-pipelines in CI (using Docker-Compose).</li></ul><h4>Monitoring &amp; Alerting</h4><ul><li><strong>Metrics</strong>: Kafka consumer lag, Flink checkpoint durations, Iceberg file counts.</li><li><strong>Dashboards</strong>: Grafana for infra, custom “data health” dashboards in BI.</li><li><strong>Alerts</strong>: SLA misses (job runtime), data quality (null rate spikes).</li></ul><h3>Conclusion</h3><p>Building an enterprise-grade data platform means more than picking tools, it’s about <strong>patterns</strong> and <strong>processes</strong>:</p><ul><li><strong>Append-only Bronze</strong> for speed &amp; audit</li><li><strong>Stateful streaming</strong> for freshness &amp; correctness</li><li><strong>Incremental batch</strong> for reliability &amp; governance</li><li><strong>CQRS &amp; materialized views</strong> for performance</li></ul><p>With these building blocks, your pipelines will stay <strong>fast</strong>, <strong>fresh</strong>, and <strong>fault-tolerant</strong> -even at millions of events per minute.</p><p>📩 <strong>Need expert guidance?</strong> <a href="https://www.evro.dev/consultant">Let’s talk</a></p><p><em>Originally published at </em><a href="https://www.evro.dev/blog/scalable-real-time-data-architecture-efficiency"><em>https://www.evro.dev</em></a><em> on April 29, 2025.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=098bc4ae500d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Top-Tier Guide to Designing Enterprise-Grade Data Architecture with Bronze, Silver, and Gold Layers…]]></title>
            <link>https://evan-rosa.medium.com/top-tier-guide-to-designing-enterprise-grade-data-architecture-with-bronze-silver-and-gold-layers-5d6c5f91857e?source=rss-443f6a356fab------2</link>
            <guid isPermaLink="false">https://medium.com/p/5d6c5f91857e</guid>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Evan Rosa]]></dc:creator>
            <pubDate>Fri, 11 Apr 2025 00:00:24 GMT</pubDate>
            <atom:updated>2025-04-15T18:14:51.374Z</atom:updated>
            <content:encoded><![CDATA[<h3>Top-Tier Guide to Designing Enterprise-Grade Data Architecture with Bronze, Silver, and Gold Layers | The Inner Join Blog By Evan Rosa</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5YRdJEetop3uV60_.png" /></figure><h3>Why Bronze Silver and Gold Data Layers Matter in Modern Data Architecture</h3><p>In fast-moving startups and enterprise settings alike, one of the biggest challenges is balancing speed with structure. Data teams often rush into dashboards, but fail to build the foundation needed for trust, performance, and scalability.</p><p>This is where the <strong>Bronze Silver and Gold data layers</strong> come into play. These layers form the backbone of a <strong>modular, reliable, and scalable</strong> data platform. They create clarity, enable governance, and prevent tech debt down the line.</p><h3>What Are the Bronze, Silver, and Gold Layers?</h3><figure><img alt="Table explaining differences between bronze, silver, and gold layers." src="https://cdn-images-1.medium.com/max/1024/1*FdvDxJU-dmVva7bWuUipOw.jpeg" /></figure><p>Each layer serves a distinct role and addresses a specific audience-from backend engineers to decision-makers.</p><h3>Bronze Layer: Raw and Untouched</h3><h4>Definition:</h4><p>The Bronze layer holds <strong>raw data straight from the source</strong>, such as Kafka streams, S3 logs, CSV files, or third-party APIs.</p><h4>Core Characteristics:</h4><ul><li><strong>Immutable</strong>: Once written, never changed.</li><li><strong>Timestamped</strong>: Clearly marked with ingestion time.</li><li><strong>Schema-flexible</strong>: Handles semi-structured and evolving formats.</li></ul><h4>Common Tools:</h4><ul><li>Apache Kafka / Apache Flink for ingestion</li><li>Delta Lake / Apache Iceberg for storage</li><li>Cloud buckets like S3 or GCS</li></ul><h4>Why It’s Crucial:</h4><p>This layer gives your team a <strong>replayable, auditable source of truth</strong>, which is vital for debugging and compliance.</p><h3>Silver Layer: Clean and Curated</h3><h4>Definition:</h4><p>The Silver layer is your <strong>standardized, enriched dataset</strong>, prepared for modeling and analysis.</p><h4>Typical Transformations:</h4><ul><li>Null handling and deduplication</li><li>Correct data typing</li><li>Reference data joins</li><li>Filtering and quality checks</li></ul><h4>Tech Stack:</h4><ul><li>SQLMesh or dbt for transformation logic</li><li>Airflow, Dagster, or Prefect for orchestration</li><li>Iceberg or Delta Lake for persistent storage</li></ul><h4>Why It’s Vital:</h4><p>Silver layers provide <strong>queryable, trustworthy datasets</strong> that analysts and ML teams can rely on.</p><h3>Gold Layer: Business-Ready Intelligence</h3><h4>Definition:</h4><p>The Gold layer contains <strong>aggregated, modeled, and optimized datasets</strong> used by business stakeholders.</p><h4>Use Cases:</h4><ul><li>Monthly Recurring Revenue (MRR)</li><li>Customer churn rates</li><li>Sales funnel and conversions</li></ul><h4>Design Focus:</h4><ul><li>Star schemas for dimensional modeling</li><li>Consistent naming conventions</li><li>Near real-time performance for BI tools</li></ul><h4>Why It Matters:</h4><p>Gold tables ensure <strong>clarity, reliability, and speed</strong>, essential for decision-making and operational dashboards.</p><h3>From Bronze to Gold: A Real-World Example</h3><p>Imagine you’re capturing product clickstream data:</p><ol><li><strong>Bronze</strong>: Raw JSON events from front-end apps, stored in Kafka and Iceberg.</li><li><strong>Silver</strong>: Join with user metadata, filter bots, normalize page views.</li><li><strong>Gold</strong>: Cohort-level daily retention, funnel drop-offs, engagement KPIs.</li></ol><p>Every step is <strong>versioned</strong>, testable, and independently queryable.</p><h3>Why Large Enterprises Prefer This Architecture</h3><p>Unlike startups, <strong>enterprise organizations demand governance, scalability, and compliance</strong>. Here’s how layered data architecture meets those needs:</p><ul><li><strong>Data Contracts</strong>: Formal agreements between producers and consumers</li><li><strong>Lineage Tracking</strong>: Understand downstream impact of changes</li><li><strong>Modularity</strong>: Scale across teams and domains</li><li><strong>Compliance Support</strong>: Meets GDPR, SOC2, HIPAA standards</li></ul><p>The layered approach brings <strong>reliability and traceability</strong> to the forefront of analytics.</p><h3>Frequently Asked Questions (FAQs)</h3><h4>1. What is the purpose of Bronze Silver and Gold data layers?</h4><p>These layers help organize and process data in stages-from raw ingestion to business intelligence-ensuring clarity, governance, and scalability.</p><h4>2. Can small companies benefit from this architecture?</h4><p>Yes! Even lean startups benefit from modular pipelines and clean separation of logic-especially as they grow.</p><h4>3. What tools support layered data architecture?</h4><p>Popular tools include Apache Kafka, Iceberg, Delta Lake, Airflow, dbt, and BI platforms like Looker or Power BI.</p><h4>4. How do you maintain performance across layers?</h4><p>Optimize transformations in the Silver layer and use materialized views or star schemas in the Gold layer to improve query speed.</p><h4>5. How often should each layer update?</h4><ul><li>Bronze: real-time or near real-time</li><li>Silver: batch (daily/hourly) depending on needs</li><li>Gold: daily or synced with business reporting cadence</li></ul><h4>6. What are common pitfalls to avoid?</h4><ul><li>Overloading the Gold layer with logic</li><li>Skipping data quality in Silver</li><li>Treating Bronze as “just logs” instead of a critical audit layer</li></ul><h3>Final Thoughts: Architecture First, Dashboards Later</h3><p>Scaling analytics isn’t about flashy dashboards. It’s about strong foundations.</p><p>The <strong>Bronze Silver and Gold data layers</strong> pattern is a proven way to build scalable, reliable, and enterprise-ready data systems. When you separate concerns, enforce governance, and support traceability-you empower your teams to deliver <strong>insight at scale</strong>.</p><p><strong>Need a Consultant?</strong> <a href="https://www.evro.dev/consultant">Let’s talk</a></p><p><em>Originally published at </em><a href="https://www.evro.dev/blog/bronze-silver-gold-data-layers-architecture"><em>https://www.evro.dev</em></a><em> on April 11, 2025.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5d6c5f91857e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Streaming ETL Is the Future — and How to Get Started]]></title>
            <link>https://evan-rosa.medium.com/why-streaming-etl-is-the-future-and-how-to-get-started-503467e78f60?source=rss-443f6a356fab------2</link>
            <guid isPermaLink="false">https://medium.com/p/503467e78f60</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[streaming]]></category>
            <category><![CDATA[flink]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Evan Rosa]]></dc:creator>
            <pubDate>Thu, 03 Apr 2025 00:00:59 GMT</pubDate>
            <atom:updated>2025-04-03T13:42:20.904Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why Streaming ETL Is the Future — and How to Get Started</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XRYkWdb57erL_cLS.png" /></figure><blockquote><em>Data doesn’t like to wait in line. If your pipeline still clocks in at hourly refreshes, you’re already behind. Streaming ETL isn’t a buzzword-it’s how modern systems think, move, and respond.</em></blockquote><h3>Why Real-Time Data Matters</h3><p>Today’s systems run in seconds-not cycles.</p><p>When your product relies on fast feedback loops, <strong>batch just can’t keep up</strong>. Businesses now expect:</p><ul><li>Instant dashboards</li><li>Continuous monitoring</li><li>On-the-fly personalization</li><li>Real-time alerts</li></ul><p>That’s where <strong>streaming ETL</strong> steps in-processing data <em>as it arrives</em> and delivering insight in motion.</p><h3>What Is Streaming ETL?</h3><p>Streaming ETL is a real-time pipeline that:</p><ol><li><strong>Extracts</strong> events continuously from sources like APIs, sensors, or Kafka topics</li><li><strong>Transforms</strong> them on the fly-cleaning, enriching, aggregating</li><li><strong>Loads</strong> them into a live analytics store or data lakehouse</li></ol><p>Instead of waiting for a job to run every hour, streaming pipelines operate in milliseconds. Think of it as ETL without the waiting room.</p><blockquote><em>In practice: Kafka → Flink → Iceberg → BigQuery or Superset.</em></blockquote><h3>When Streaming Makes Sense</h3><p>You don’t need to stream everything-but you <em>do</em> need streaming when:</p><ul><li>You monitor <strong>live events</strong> (clicks, transactions, sensors)</li><li>You need <strong>low-latency alerts</strong> (fraud, outages, churn)</li><li>You support <strong>real-time user interfaces</strong> or dashboards</li><li>You’re building <strong>event-driven microservices</strong> or a data mesh</li></ul><h3>Common Use Cases</h3><ul><li>Real-time churn prediction in SaaS</li><li>Live product recommendations</li><li>Streaming IoT telemetry from devices</li><li>Sports analytics during live events</li></ul><p>These are pipelines that don’t blink-and can’t afford to.</p><h3>Kafka + Flink: The Dynamic Duo</h3><p>When it comes to streaming ETL, <strong>Kafka and Flink</strong> are the peanut butter and jelly of the modern stack.</p><figure><img alt="Kafka and Flink table describing each service" src="https://cdn-images-1.medium.com/max/672/1*rK6Xb8-1d4_Y-nUEDwq3iw.jpeg" /></figure><h3>Example: Kafka → Flink → Iceberg</h3><p>Here’s a simplified flow I’ve used in production:</p><ol><li><strong>Kafka</strong> captures JSON events from user signups</li><li><strong>Flink</strong> job: Parses and validates schema, Enriches with metadata, Deduplicates by event ID, Aggregates by region or channel</li><li><strong>Iceberg</strong> stores cleaned and enriched tables</li><li><strong>Airflow</strong> schedules downstream rollups into <strong>BigQuery</strong></li><li><strong>Superset</strong> powers near real-time dashboards</li></ol><blockquote><em>One of my side projects simulated synthetic subscription events. With Flink powering the stream and Superset rendering the dashboards, I had a real-time analytics engine humming locally in under a day.</em></blockquote><h3>Streaming vs Batch: Choose with Intent</h3><figure><img alt="When to choose batch and stream" src="https://cdn-images-1.medium.com/max/672/1*i9eMb5zGyGF3QiDl5wwJjA.jpeg" /></figure><p>My rule of thumb? Start with streaming where freshness matters most-then layer in batch for history, rollups, and cost efficiency.</p><h3>My Streaming Stack: The Tools I Use</h3><p>When building streaming-first architectures, these are my go-to tools:</p><ul><li><strong>Kafka</strong> — for event ingestion and scalable queues</li><li><strong>Flink</strong> — for rich, low-latency stream processing</li><li><strong>Iceberg</strong> — for schema evolution and time travel</li><li><strong>Airflow</strong> — for hybrid orchestration</li><li><strong>BigQuery</strong> — for SQL-based reporting at scale</li><li><strong>Superset</strong> — for fast, clean dashboards that refresh in real time</li></ul><p>I containerize everything with <strong>Docker</strong>, and I test it all locally before deploying to cloud. Streaming doesn’t need to be scary-but it does need to be thoughtful.</p><h3>Batch + Streaming = Hybrid Done Right</h3><p>Batch and streaming aren’t rivals-they’re teammates.</p><p>Most of my real-world pipelines look like this:</p><ol><li>Ingest <strong>real-time events via Kafka</strong></li><li>Process with <strong>Flink</strong></li><li>Persist clean and raw data to <strong>Iceberg</strong></li><li>Schedule batch rollups with <strong>SQLMesh or Airflow</strong></li><li>Serve real-time insights via <strong>BigQuery + Superset</strong></li></ol><p>The result? Systems that can respond in real time <em>and</em> provide historical depth. That’s the kind of flexibility modern teams need.</p><p><strong>Let’s Connect</strong><br> Building something streaming-first? Wrestling with real-time complexity? I’d love to learn what you’re working on.</p><p><em>Originally published at </em><a href="https://www.evro.dev/blog/streaming-etl-kafka-flink-real-time-pipelines"><em>https://www.evro.dev</em></a><em> on April 3, 2025.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=503467e78f60" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is ETL in 2025? Moving Beyond Extract, Transform, Load]]></title>
            <link>https://evan-rosa.medium.com/what-is-etl-in-2025-moving-beyond-extract-transform-load-da0a3cfe6e97?source=rss-443f6a356fab------2</link>
            <guid isPermaLink="false">https://medium.com/p/da0a3cfe6e97</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[real-time-analytics]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[etl]]></category>
            <dc:creator><![CDATA[Evan Rosa]]></dc:creator>
            <pubDate>Wed, 02 Apr 2025 00:00:57 GMT</pubDate>
            <atom:updated>2025-04-03T13:35:27.422Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2sdgwJMWhB6Z_CMj.png" /></figure><blockquote><em>ETL isn’t dead-it just changed its wardrobe. In 2025, data pipelines are faster, smarter, and more flexible than ever. But to build them right, you need to understand the difference between ETL and ELT-and when to reach for each.</em></blockquote><h3>The Classic ETL Playbook</h3><p>Let’s start at the beginning: <strong>Extract, Transform, Load</strong>. Once upon a time, this was the standard play in every data engineer’s book of spells:</p><ol><li><strong>Extract</strong> data from source systems-databases, APIs, logs, you name it.</li><li><strong>Transform</strong> the data in a staging area-cleaning, joining, shaping.</li><li><strong>Load</strong> the finished product into a warehouse for querying and dashboards.</li></ol><p>This approach worked well in the days of scheduled batch jobs and tightly controlled schemas. But the modern data stack doesn’t sit still long enough for that old-school rhythm.</p><h3>ELT: Flipping the Flow</h3><p><strong>ELT (Extract, Load, Transform)</strong> turns the process inside-out. Now we load data <em>before</em> transformation-letting your warehouse handle the heavy lifting. It’s faster, cheaper, and more scalable for today’s cloud-native environments.</p><figure><img alt="ELT vs. ETL Comparison" src="https://cdn-images-1.medium.com/max/672/1*AwgHBiZ-JgJUfthrBaiGSg.jpeg" /></figure><h3>Why the Shift?</h3><p>Modern businesses don’t want yesterday’s reports tomorrow-they want <strong>insight in motion</strong>:</p><ul><li>Real-time updates</li><li>Scalable pipelines</li><li>Change-tolerant schemas</li><li>Observability and lineage</li><li>Self-serve tooling for data consumers</li></ul><p>Put simply, the old ETL model can’t keep up with today’s speed of business. Enter streaming-first, modular architectures designed to evolve-not break.</p><h3>Batch vs Streaming: Know When to Use What</h3><p>Streaming gives you speed. Batch gives you depth. Smart data systems use both-and know when to lean on each.</p><figure><img alt="Batching vs Streaming Data and When to use it" src="https://cdn-images-1.medium.com/max/671/1*5MCudZTToiklDOkAmaJ6eQ.jpeg" /></figure><h3>My Recommended 2025 Data Stack</h3><p>When I build pipelines today, here’s what’s in my belt:</p><h3>Data Ingestion</h3><ul><li><strong>Kafka</strong> — battle-tested pub/sub for high-volume event streams</li><li><strong>Apache Pulsar</strong> — flexible alternative with multi-tenancy and tiered storage</li><li><strong>Fivetran / Airbyte (open-core)</strong> — connectors galore for quick batch loads</li><li><strong>Custom Python loaders</strong> — when you need more control</li></ul><h3>Data Processing</h3><ul><li><strong>Apache Flink</strong> — real-time stream transformations at scale</li><li><strong>Apache Spark</strong> — batch processing powerhouse with rich APIs</li><li><strong>SQLMesh</strong> — modular, version-controlled SQL transformations with CI/CD baked in</li><li><strong>dbt</strong> — excellent for transformation logic inside warehouses</li></ul><h3>Data Storage</h3><ul><li><strong>Apache Iceberg</strong> — versioned, schema-evolving tables for lakes and lakehouses</li><li><strong>BigQuery</strong> — serverless warehousing with blazing performance</li><li><strong>Snowflake</strong> — cloud-native warehouse with secure data sharing</li><li><strong>PostgreSQL</strong> — trusty relational DB for operational storage</li></ul><h3>Orchestration</h3><ul><li><strong>Apache Airflow</strong> — robust DAG-based scheduling and orchestration</li><li><strong>Dagster</strong> — opinionated, type-safe workflows with stronger developer ergonomics</li><li><strong>Prefect</strong> — more dynamic orchestration with less boilerplate</li></ul><h3>Monitoring &amp; Observability</h3><ul><li><strong>Great Expectations</strong> — test and validate your data with assertions and alerts</li><li><strong>Datafold</strong> — regression testing and column-level lineage for transformations</li><li><strong>DataHub</strong> — open metadata platform for lineage, ownership, and discovery</li><li><strong>Prometheus + Grafana</strong> — metrics and dashboards for pipeline health</li></ul><h3>Visualization</h3><ul><li><strong>Apache Superset</strong> — open-source dashboards with SQL-friendly flexibility</li><li><strong>Metabase</strong> — fast, self-service dashboards for cross-functional teams</li><li><strong>Looker</strong> — semantic layer and governance with clean UX (if budget allows)</li></ul><h3>Choosing the Right Tool: It’s All About Fit</h3><p>Think of this stack like a well-stocked toolbox. You don’t use a sledgehammer to tighten a bolt-and you don’t need Flink to parse a daily CSV.</p><p>Each tool listed here has its strengths, but I don’t believe in hammering every problem with the same wrench. Instead, I match tools to the problem space:</p><ul><li>Need low-latency insights from a high-volume event stream? I’ll reach for <strong>Kafka + Flink</strong>.</li><li>Need quick batch transforms for a reporting layer? <strong>Airflow + SQLMesh</strong> does the trick.</li><li>Need stakeholder-friendly dashboards? Sometimes <strong>Metabase</strong> is the fastest win.</li></ul><p>The point is: great data engineering isn’t about committing to one stack-it’s about knowing your tools and selecting the right one for the job.</p><p>Because in the end, a good data engineer doesn’t just build pipelines-they build <strong>systems that solve problems efficiently</strong>.</p><h3>Best Practices for Modern ETL &amp; ELT</h3><p>Whether you’re ETLing, ELTing, or somewhere in between-here’s what I recommend:</p><ol><li><strong>Design for failure</strong><br> Pipelines break. Make sure yours can recover gracefully. Think retries, timeouts, and dead-letter queues.</li><li><strong>Test early and often</strong><br> Treat data like code. Use tools like Great Expectations or dbt tests to catch bad data before it spreads.</li><li><strong>Make everything modular</strong><br> Pipelines are living systems. Structure them in a way that’s easy to extend and refactor.</li><li><strong>Track lineage and versions</strong><br> Use Iceberg and Nessie to track changes to your data just like Git tracks your code. Debugging becomes a breeze.</li><li><strong>Empower end users</strong><br> Data should be self-serve. Build pipelines and dashboards that analysts love using-not just ones engineers can tolerate.</li></ol><h3>Real-World Impact: From Legacy to Lift-Off</h3><p>At <strong>Digital Turbine</strong>, I refactored legacy batch pipelines that were brittle, slow, and expensive-migrating them to Spark on Airflow. The result?</p><p>On the side, I’ve built hybrid streaming-first platforms combining <strong>Kafka, Flink, Iceberg</strong>, and <strong>SQLMesh</strong> to power systems with millions of daily events-and they’re still humming along.</p><p><strong>Let’s Connect</strong><br> If you’re building modern data systems-or dreaming of pipelines that don’t fall apart when change happens-let’s talk.</p><p><em>Originally published at </em><a href="https://www.evro.dev/blog/modern-etl-2025-etl-vs-elt"><em>https://www.evro.dev</em></a><em> on April 2, 2025.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da0a3cfe6e97" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Pipelines to Purpose: Why I’m Sharing My Journey in Data Engineering]]></title>
            <link>https://evan-rosa.medium.com/from-pipelines-to-purpose-why-im-sharing-my-journey-in-data-engineering-c716fa844771?source=rss-443f6a356fab------2</link>
            <guid isPermaLink="false">https://medium.com/p/c716fa844771</guid>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[real-time-streaming-data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[careers-in-tech]]></category>
            <dc:creator><![CDATA[Evan Rosa]]></dc:creator>
            <pubDate>Tue, 01 Apr 2025 00:00:00 GMT</pubDate>
            <atom:updated>2025-04-03T13:25:32.665Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xiaPjL-QyIBwStU5.jpg" /></figure><blockquote><em>Even after a full week of work and family life, I spend my weekends tinkering with data pipelines-not because I have to, but because this is how I level up. I’m not just refining skills-I’m writing the next chapter of my career, in public.</em></blockquote><h3>Who I Am</h3><p>Hey there-I’m <strong>Evan Rosa</strong>, a data engineer with nearly 15 years of experience in the data space designing, building, and scaling data systems across the public, private, and nonprofit sectors.</p><p>I’ve architected cloud-native solutions for:</p><ul><li><strong>Digital Turbine</strong>, where I helped drive over <strong>3 billion monthly ad impressions</strong></li><li><strong>HHS/NIH (via Booz Allen Hamilton)</strong>, supporting data integrity in critical health research</li><li><strong>The American Chemical Society</strong>, building systems to track engagement to their 80+ peer reviewed journals.</li></ul><p>My toolkit includes <strong>Python, SQL, Airflow, Spark, Kafka, Flink</strong>, and lately I’ve been getting my hands dirty with <strong>Apache Iceberg</strong> and <strong>Project Nessie</strong> to bring version control into the modern data stack. I focus on building flexible, connector-driven ETL frameworks that scale-because brittle pipelines simply don’t work.</p><h3>Why I’m Writing This Blog</h3><p>I’m not just sharing to show off what I’ve built-I’m sharing to <strong>connect</strong>.</p><p>This blog is my digital workbench: part journal, part lab notebook, part blueprint. I’ll walk you through:</p><ul><li>Projects I’ve built-with architecture diagrams, code, and real-world lessons</li><li>Tools I’ve adopted (and abandoned), and why</li><li>Thoughts on what makes great data engineering</li><li>My personal roadmap toward an <strong>advanced data engineering role at a mission-driven company</strong></li></ul><p>I believe in <strong>building in public</strong> -because the best ideas get sharper when shared. If something I’ve wrestled with helps you debug your own pipeline, that’s a win.</p><h3>What I’ve Built (So Far)</h3><p>I’ve led and contributed to systems like:</p><ul><li><strong>Cost-efficient ETL frameworks</strong> with Airflow, Spark, and BigQuery, slashing cloud bills by over <strong>$100K</strong></li><li><strong>Streaming analytics pipelines</strong> using Kafka and Flink to support real-time dashboards in sports tech and subscription platforms</li><li><strong>Composable, connector-based ingestion frameworks</strong> pulling data from APIs, GCS, S3, and live event streams</li><li><strong>Self-serve analytics layers</strong> with Looker and Superset to help non-technical teams explore data confidently</li></ul><p>These aren’t portfolio projects-they’re solutions to business-critical challenges, delivered in production environments with real stakes.</p><h3>How I Think About Modern Data Workflows</h3><p>To me, the best pipelines aren’t fragile contraptions-they’re <strong>well-oiled machines</strong> built for change. They should be:</p><ul><li><strong>Modular</strong>: Swap in new tools without rewriting your stack</li><li><strong>Version-controlled</strong>: Git-like rollbacks for data models using Nessie and Iceberg</li><li><strong>Hybrid</strong>: Real-time and batch playing nicely together</li><li><strong>Self-serve</strong>: So analysts aren’t left waiting on pull requests</li><li><strong>Observable and testable</strong>: Because trust in data is earned, not assumed</li></ul><p>In short: <strong>Data engineers should be user experience designers</strong> -not just of dashboards, but of the pipelines themselves.</p><h3>What You’ll Find Here</h3><p>This isn’t another “hello world” blog.</p><p>Instead, I’ll offer:</p><ul><li>Hard-won insights from real-world projects</li><li>Architectural patterns that stand up to scale</li><li>Thoughts on building resilient, elegant data systems</li><li>Ideas on career growth, tooling, and staying sharp in a fast-moving field</li></ul><p>If you’re looking for practical, thoughtful takes on modern data engineering-you’re in the right place.</p><h3>Let’s Build Together</h3><p>Whether you’re a fellow builder, a hiring manager looking for a senior engineer, or someone just starting out-welcome.</p><p>Let’s share what we know, learn what we don’t, and build something better-together.</p><p><strong>Follow along</strong> if you’re curious about how great pipelines are built-and what it means to build them with purpose.</p><p><em>Originally published at </em><a href="https://www.evro.dev/blog/data-engineering-journey-evan-rosa"><em>https://www.evro.dev</em></a><em> on April 1, 2025.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c716fa844771" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>