<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Wassim Taoueli on Medium]]></title>
        <description><![CDATA[Stories by Wassim Taoueli on Medium]]></description>
        <link>https://medium.com/@wassimtaoueli?source=rss-badcd22bf55e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*GVa7uUn_rtI_ji_x.jpg</url>
            <title>Stories by Wassim Taoueli on Medium</title>
            <link>https://medium.com/@wassimtaoueli?source=rss-badcd22bf55e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 04:41:36 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@wassimtaoueli/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Architecting the Modern Lakehouse: A Blueprint for Complete ELT with Trino, dbt, and Apache Iceberg]]></title>
            <link>https://medium.com/@wassimtaoueli/architecting-the-modern-lakehouse-a-blueprint-for-complete-elt-with-trino-dbt-and-apache-iceberg-92fee2b95e3c?source=rss-badcd22bf55e------2</link>
            <guid isPermaLink="false">https://medium.com/p/92fee2b95e3c</guid>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[dbt]]></category>
            <dc:creator><![CDATA[Wassim Taoueli]]></dc:creator>
            <pubDate>Sun, 21 Sep 2025 17:09:57 GMT</pubDate>
            <atom:updated>2025-09-21T17:09:57.178Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>For years, data engineers have been burdened by the constraints of classic ETL (<em>Extract, Transform, Load</em>) processes. The paradigm is fraught with inherent complexities:</p><p>First, the <strong><em>extraction </em></strong>phase requires pulling data from a sprawling ecosystem of sources ; databases, object storage, APIs, and more. This becomes a significant operational challenge when dealing with massive data volumes and a wide variety of sources, consuming immense time and resources.</p><p>Then comes the most cumbersome stage: <strong><em>transformation</em></strong>. Data must be processed (cleansed, augmented, and aggregated ) before it can be loaded into its final destination. This requires provisioning and maintaining a separate processing cluster (like Spark), adding yet another system to the stack.</p><p>Finally, the processed data is <strong><em>loaded </em></strong>into its final home: <strong>the data warehouse</strong>. This final step introduces a fundamental architectural constraint: the tight coupling of storage and compute. Traditional data warehouses manage their own proprietary storage, locking you into their ecosystem and making it difficult, expensive, and sometimes impossible to use other tools for analysis or processing. This model creates a bottleneck where business agility is held hostage by the warehouse’s scaling costs and performance characteristics.</p><p>The core issue is the proliferation of technologies. Each stage often necessitates a different tool, forcing teams to manage, maintain, and integrate multiple systems. This not only adds layers of operational complexity but also introduces significant latency, meaning data is never truly fresh.</p><p>But what if we could flip this model on its head? What if we could extract and load data first and transform it later, all within the same open, scalable environment ?</p><p>This shift from <strong>ETL </strong>to <strong>ELT </strong>(<em>Extract, Load, Transform</em>) is the cornerstone of the modern data stack. However, a true ELT workflow has been historically difficult to achieve on data lakes due to limitations in performance, reliability, and the lack of transactional guarantees.</p><p>This is where the lakehouse architecture emerges as a solution. It promises the open, scalable storage of a data lake with the performance and manageability of a data warehouse, all while decoupling storage from compute to avoid vendor lock-in and control costs.</p><p>In this article, we will architect a lakehouse that enables a complete and robust <strong>ELT </strong>process. We’ll explore how using <strong>Amazon S3</strong> for open storage, <strong>Apache Iceberg</strong> as our table format, <strong>dbt </strong>as our transformation layer, and <strong>Trino </strong>as our query engine creates a cohesive, powerful, and simplified data platform that breaks free from the old constraints.</p><h3>Introducing the Blueprint: Our Modern Lakehouse Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sfB4NhqXeHxRXCCHKGK2rQ.png" /><figcaption>open Lake house architecture</figcaption></figure><p>A lakehouse architecture combines the best of data lakes and data warehouses through three core components:</p><ol><li><strong>Storage Layer: </strong>Object storage for raw data</li><li><strong>Compute Layer</strong> : Processing engines for transformations</li><li><strong>Table Format: </strong>Transactional capabilities and metadata management</li></ol><h4><strong>Our Technology Stack:</strong></h4><p><strong>Storage: Amazon S3;</strong> We selected S3 as our storage foundation because it offers cost-effective storage for all data types (structured, semi-structured, and unstructured). This flexibility allows us to ingest data in its native format without upfront transformation costs.</p><p><strong>Table Format: Apache Iceberg;</strong> Apache Iceberg serves as our open table format, bringing enterprise-grade features to our data lake:</p><ul><li>ACID transactions for data consistency</li><li>Time travel capabilities for historical analysis</li><li>Schema evolution without breaking downstream processes</li><li>Advanced partitioning strategies that optimize query performance</li></ul><p>Iceberg’s strong open-source community support ensures long-term viability and continuous innovation.</p><p><strong>Compute Engine: Apache Trino ;</strong> Trino acts as our distributed query engine, offering several key advantages:</p><ul><li>Massive parallel processing for handling large datasets</li><li>Extensive connector ecosystem supporting 40+ data sources</li><li>SQL abstraction layer that unifies access across different systems</li><li>Federated query capabilities for cross-platform analytics</li></ul><p><strong>Data Transformation: dbt Core;</strong> dbt handles our data modeling and transformation workflows:</p><ul><li>Native integration with Trino for seamless execution</li><li>Version-controlled, testable data transformations</li><li>Simplified dependency management between data models</li><li>Built-in documentation and lineage tracking</li></ul><h3>ELT complete workflow</h3><p>Now for the exciting part — let’s explore how our modern lakehouse architecture handles the complete ELT (Extract, Load, Transform) workflow. This section details how data extraction, loading, and transformation work seamlessly together.</p><h4>Extract data : Leveraging Trino’s Connector Ecosystem</h4><p>Before extracting data, we configure Trino catalogs that act as connection points to different data sources:</p><p><strong>Iceberg Catalog</strong> — Our primary storage destination</p><ul><li>Configured using the Iceberg connector</li><li>Enables Trino to read from and write to Iceberg tables in S3</li><li>Supports all Iceberg features including time travel and schema evolution</li></ul><p><strong>OLTP Database Catalogs</strong> — For transactional systems</p><ul><li>Example: SQL Server catalog for accessing production databases</li><li>PostgreSQL, MySQL, Oracle catalogs for other relational sources</li><li>Provides real-time access to operational data</li></ul><p><strong>Streaming Data Catalog</strong> — For real-time ingestion</p><ul><li>Kafka connector enables reading from Kafka topics</li><li>Supports various data formats (JSON, Avro, etc.)</li><li>Allows streaming data integration alongside batch processing</li></ul><p><strong>File-based Data Catalog</strong> — For existing data files</p><ul><li>Hive connector accesses files in object storage (S3, GCS, Azure Data Lake)</li><li>Registers CSV, JSON, Parquet, and other file formats as queryable tables</li></ul><p>With these catalogs configured, data extraction becomes remarkably simple. Trino abstracts all complexity behind standard SQL queries : We can query data in the source systems with simple <strong>SELECT </strong>queries and extract it.</p><p>Suppose we configured 3 catalogs (sqlserver catalog, kafka catalog and a hive catalog that points to an S3 object storage containing csv data)</p><pre>-- reading data from a sql server database<br>SELECT * FROM sqlserver_catalog.production.customers;<br><br>-- reading data from a kafka topic<br>SELECT * FROM kafka_catalog.payments.transaction_stream;<br><br>-- reading data from an object storage:<br>-- first we need to register file data as an external hive table<br>CREATE TABLE hive_catalog.default.csv_data (name VARCHAR, address VARCHAR)<br>WITH<br>(<br>format = &#39;CSV&#39;,<br>external_location = &#39;s3://my_folder/csv_data_folder/&#39;,<br>skip_header_line_count = 1<br>); <br><br>-- now we query the data<br>SELECT * FROM hive_catalog.default.csv_data;</pre><h4>Load data</h4><p>With our catalogs configured, loading data into our lakehouse becomes straightforward. Using the Iceberg catalog, Trino can seamlessly write extracted data directly into Iceberg tables stored in our destination S3 bucket.</p><pre>-- Load data from source systems into bronze layer<br>CREATE TABLE iceberg_catalog.bronze.customer_data AS<br>SELECT * FROM sqlserver.production.customers;<br><br>-- Append new records to existing tables<br>INSERT INTO iceberg.bronze.daily_transactions<br>SELECT * FROM kafka_catalog.payments.transaction_stream<br>WHERE event_time &gt;= current_date;<br><br>-- Merging records to existing tables<br>MERGE INTO iceberg_catalog.bronze.accounts t <br>    USING hive_catalog.accounts_data.monthly_accounts_update s<br>    ON (t.customer = s.customer)<br>    WHEN MATCHED<br>        THEN UPDATE SET purchases = s.purchases<br>    WHEN NOT MATCHED<br>        THEN INSERT (customer, purchases, address)<br>              VALUES(s.customer, s.purchases, s.address)</pre><p><strong>Key Advantages</strong></p><ul><li><strong>Direct writes</strong> — No intermediate staging areas required</li><li><strong>ACID compliance</strong> — All writes are transactional and consistent</li><li><strong>Immediate availability</strong> — Data becomes queryable as soon as the transaction completes</li></ul><p>This approach eliminates traditional loading bottlenecks by removing the need for separate loading tools or complex staging processes. Data flows directly from source to destination in a single, atomic operation.</p><h4>Transform data</h4><p>With data now residing in our bronze layer Iceberg tables, transformation becomes a powerful yet straightforward process using standard SQL queries. Data transformations are executed directly within the lakehouse using Trino’s SQL engine:</p><pre>-- Transformations examples:<br>-- Transform bronze data into silver layer<br>CREATE TABLE iceberg.silver.customer_profiles AS<br>SELECT <br>    customer_id,<br>    CONCAT(first_name, &#39; &#39;, last_name) as full_name,<br>    email,<br>    DATE_DIFF(&#39;year&#39;, birth_date, CURRENT_DATE) as age,<br>    registration_date<br>FROM iceberg.bronze.customer_data<br>WHERE status = &#39;active&#39;;<br><br>-- Create aggregated gold layer tables<br>CREATE TABLE iceberg.gold.monthly_sales_summary AS<br>SELECT <br>    DATE_TRUNC(&#39;month&#39;, transaction_date) as month,<br>    product_category,<br>    SUM(amount) as total_sales,<br>    COUNT(*) as transaction_count<br>FROM iceberg.silver.transactions<br>GROUP BY 1, 2;</pre><p>Transformed data can be exposed in multiple formats to suit different use cases:</p><ul><li><strong>Iceberg Tables: </strong>For persistent, queryable datasets with full ACID properties</li><li><strong>Iceberg Views: </strong>For virtual transformations that execute on-demand</li><li><strong>Materialized Views</strong> : For pre-computed results with refresh capabilities</li></ul><h4>Orchestrating the Complete ELT Pipeline with dbt Core</h4><p>Here’s where our architecture truly shines , dbt Core can orchestrate the entire ELT workflow, not just the transformation layer. When coupled with Trino, dbt extends beyond its traditional role to manage extraction, loading, and transformation in a unified framework.</p><p>While dbt is commonly known for transformations (the “T” in ELT), our Trino integration unlocks its full potential:</p><p>Let’s examine a real-world scenario where we need to create a bronze table called daily_transactions. If the table doesn&#39;t exist, dbt creates it; if it does exist, dbt appends only new data. The model definition below demonstrates this incremental loading pattern:</p><pre>-- models/bronze/daily_transactions.sql<br>{{ config(<br>    materialized=&#39;incremental&#39;,<br>    unique_key=&#39;transaction_id&#39;,<br>    on_schema_change=&#39;fail&#39;<br>) }}<br><br>SELECT *<br>FROM kafka_catalog.payments.transaction_stream<br><br>{% if is_incremental() %}<br>    -- Only process new records since last run<br>    WHERE transaction_date &gt; (SELECT MAX(transaction_date) FROM {{ this }})<br>{% endif %}</pre><p>After loading raw data into our bronze layer, we build subsequent transformation models that progressively refine and structure the data. Each transformation stage creates purpose-built datasets tailored to specific analytical requirements.</p><pre>-- models/silver/processed_transactions.sql<br>{{ config(materialized=&#39;table&#39;) }}<br><br>SELECT <br>    transaction_id,<br>    customer_id,<br>    amount,<br>    transaction_date,<br>    payment_method,<br>    CASE <br>        WHEN amount &gt; 1000 THEN &#39;high_value&#39;<br>        WHEN amount &gt; 100 THEN &#39;medium_value&#39;<br>        ELSE &#39;low_value&#39;<br>    END as transaction_tier<br>FROM {{ ref(&#39;daily_transactions&#39;) }}<br>WHERE status = &#39;completed&#39;<br><br>-- models/gold/customer_metrics.sql  <br>{{ config(materialized=&#39;view&#39;) }}<br><br>SELECT <br>    customer_id,<br>    COUNT(*) as total_transactions,<br>    SUM(amount) as lifetime_value,<br>    AVG(amount) as avg_transaction_value<br>FROM {{ ref(&#39;processed_transactions&#39;) }}<br>GROUP BY customer_id</pre><p>This example demonstrates the transformative capability of combining Trino with dbt Core, enabling complete ELT orchestration through a single, cohesive framework.</p><h3>Conclusion</h3><p>The modern lakehouse architecture demonstrates how the right combination of technologies can dramatically simplify data engineering. By unifying S3, Apache Iceberg, Trino, and dbt Core, we’ve eliminated the complexity of traditional ETL pipelines while delivering enterprise-grade reliability and performance.</p><p>The real breakthrough is using dbt Core beyond transformations, orchestrating complete ELT workflows through familiar SQL. This approach reduces operational overhead, improves developer productivity, and creates a foundation that scales with your data needs.</p><p>For data teams looking to move past the constraints of legacy systems, this lakehouse architecture offers a clear path: fewer tools, simpler operations, and faster time to business value. The future of data engineering isn’t about managing complexity — it’s about architecting it away.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=92fee2b95e3c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>