<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kimanthi on Medium]]></title>
        <description><![CDATA[Stories by Kimanthi on Medium]]></description>
        <link>https://medium.com/@kimanthi_de?source=rss-a697849c1f63------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*EKX857mZhaRRxVJLq_0Cpw.png</url>
            <title>Stories by Kimanthi on Medium</title>
            <link>https://medium.com/@kimanthi_de?source=rss-a697849c1f63------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 13:13:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kimanthi_de/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Most AI Data Agents Fail — Here’s How to Build One That Doesn’t]]></title>
            <link>https://medium.com/@kimanthi_de/most-ai-data-agents-fail-heres-how-to-build-one-that-doesn-t-d049a79a434e?source=rss-a697849c1f63------2</link>
            <guid isPermaLink="false">https://medium.com/p/d049a79a434e</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Kimanthi]]></dc:creator>
            <pubDate>Tue, 28 Apr 2026 16:16:37 GMT</pubDate>
            <atom:updated>2026-04-28T16:16:37.317Z</atom:updated>
            <content:encoded><![CDATA[<p>The “data request” cycle is a familiar grind. A stakeholder asks a complex question, a data analyst/engineer writes a custom SQL query, the result is pasted into a slide deck, and a week later, the logic is forgotten or duplicated elsewhere.</p><p>We addressed this with self-service BI. But dashboards are static — they tell you what happened. They rarely explain why it happened or what to do next.</p><p>The next evolution isn’t another dashboard. It’s an autonomous data agent.</p><p>In this article, I’ll walk through how I’d implement a production-minded agentic workflow using an open stack — one that doesn’t just chat, but actually analyzes data, correlates structured and unstructured context, and triggers actions.</p><h3>From Chatbot to Agent</h3><p>An agent is not just an LLM with database access. In fact, giving an LLM raw SQL access is one of the fastest ways to get hallucinated joins, runaway queries, and an angry finance team.</p><p>A production-grade data agent needs structure, tools, and guardrails.</p><p>I like to think of the architecture in four layers:</p><ul><li>Foundation: A Semantic Layer — I will use the dbt Semantic Layer (MetricFlow) in this article</li><li>Context: Data Warehouse + Vector DB for Unstructured Data. The article uses Postgres + pgvector</li><li>Brain (Orchestration): LangGraph or whatever your team uses</li><li>Action Layer(Last Mile): Slack, APIs, PDF report, and downstream systems</li></ul><p>Each layer removes a specific failure mode that a naive “AI + SQL” system might run into.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NRFuqmEP7GFkN6jcr0UsdA.png" /><figcaption>Generic architecture for a data agent: orchestration, governed metrics, contextual retrieval, and controlled action.</figcaption></figure><h3>The Foundation: Semantic Layer as a Contract</h3><p>The biggest mistake in AI-to-SQL systems is asking the model to generate SQL.</p><p>Instead, we ask the model to query a semantic layer.</p><p>For instance, using dbt’s Semantic Layer (via MetricFlow), we expose:</p><ul><li>Standardized metrics (e.g., revenue, repeat_purchase_rate)</li><li>Governed dimensions (region, product_category, fulfillment_center)</li></ul><p>For example:</p><p><em>repeat_purchase_rate = returning_customers / total_customers</em></p><p>The agent doesn’t need to know how orders join to payments or customers. It simply calls a metric API with parameters like:</p><ul><li>metric: repeat_purchase_rate</li><li>dimensions: [product_category, region]</li><li>time_range: last_30_days vs previous_30_days</li></ul><p>This eliminates most hallucination risk and enforces consistency across every query.</p><blockquote>This assumes something critical: strong upstream data contracts and dbt tests (freshness, null checks, relationships). Without that, your agent is just confidently wrong at scale.</blockquote><h3>The Context: Structured Data Meets the “Vibe”</h3><p>A large part of business context lives in what I call the “vibe” — support tickets, Slack threads, call transcripts, PDFs, internal notes, etc.</p><p>For this data to be accessible to our agent, we store it in a vector database.</p><blockquote>pgvector is an open-source PostgreSQL extension that enables storing, indexing, and searching high-dimensional vector embeddings directly within the PostgreSQL database.</blockquote><p>At a high level, the process of storing and incorporating unstructured data into our workflow involves the following:</p><ul><li>Documents are chunked into token windows</li><li>Each chunk is embedded (e.g., OpenAI or similar embedding models)</li><li>Metadata is attached: region, product, timestamp, source</li></ul><p>When the agent sees a signal in structured data, it can query the vector store with targeted.</p><p>This is what allows the system to answer “why,” not just “what.”</p><h3>The Brain: LangGraph for Stateful Reasoning</h3><p>Most “LLM apps” are linear chains. Real data analysis is not.</p><p>It’s iterative, conditional, and often requires backtracking.</p><p>LangGraph lets us model the agent as a state machine with explicit control over flow and memory.</p><p>A simplified version of such a graph would contain the following:</p><ul><li>Node A: Parse user intent into structured query plan</li><li>Node B: Query the semantic layer for metrics</li><li>Node C: Evaluate results (detect anomalies or drops)</li><li>Node D: Retrieve unstructured context (vector search)</li><li>Node E: Synthesize explanation</li><li>Node F (Adaptive Loop): If confidence is low or variance exceeds threshold, pivot dimensions and re-query. The node can also include a break condition that requires human intervention.</li></ul><p>That last step is what makes this an <em>agent</em>.</p><p>For example, if a metric drops at the category level, the agent can automatically drill into:</p><p>category → region → fulfillment center → SKU</p><p>This mimics how an analyst actually thinks.</p><h3>Case Study: Revenue Recovery Agent</h3><p>Let’s make this concrete. Let’s imagine we are working for an e-commerce shop that operates across the United States.</p><p>A stakeholder asks:</p><p>“Our repeat purchase rate in Electronics dropped 15% last month. What happened, and who should we talk to?”</p><h4>Step 1: Define Tools</h4><p>The agent operates through constrained tools:</p><ul><li>get_metrics(): calls dbt Semantic Layer</li><li>search_feedback(): queries pgvector</li><li>trigger_action(): sends outputs to Slack, Jira, or marketing tools</li></ul><p>Ensure each tool is typed, logged, and observable.</p><h4>Step 2: Agent Execution</h4><ol><li><strong>Metric Retrieval </strong>— The agent queries: repeat_purchase_rate(metric), product_category + region(Dimensions), last 30 days vs previous 30 days(Time window)</li></ol><p>It finds:</p><ul><li>Electronics down 15% overall</li><li>High-end headphones down 28% in the Midwest</li></ul><p>2<strong>. Contextual Search — </strong>It then queries: “headphone complaints Midwest last 30 days.”</p><p><strong>3. Discovery — </strong>It retrieves ~40 support tickets mentioning:</p><ul><li>Shipping delays</li><li>Midwest fulfillment center bottleneck</li><li>Delivery times exceeding 7 days</li></ul><p><strong>4. Synthesis — </strong>Instead of a generic answer, the agent produces:</p><p>“Repeat purchase rate for high-end headphones dropped 28% in the Midwest between March 3–18. This correlates with shipping delays from the Chicago fulfillment center. Approximately 1,200 customers were impacted.”</p><p>That’s a real, actionable insight.</p><h3>The Action Layer: Closing the Loop</h3><p>Insights are only useful if they lead to action. It’s part of the system design process to identify the most realistic flow to close this loop.</p><p>In our hypothetical use case, the agent can:</p><ul><li>Send a Slack alert to #growth-leads</li><li>Create a Jira ticket for logistics</li><li>Push affected users to a marketing tool (via reverse ETL)</li><li>Generate a report in Streamlit for human review</li></ul><blockquote>For high-impact decisions, the agent should not act autonomously; it should pause at a human review gate where a reviewer can approve, reject, or modify the recommendation before any downstream action is taken.</blockquote><h3>Making This Production-Ready</h3><p>A real system needs guardrails:</p><ul><li><strong>Query safety</strong>: only allow access through semantic layer APIs (no raw SQL)</li><li><strong>Cost control</strong>: caching (Redis), rate limiting, and query budgets</li><li><strong>Observability</strong>: log every tool call, decision path, and output</li><li><strong>Evaluation</strong>: benchmark against curated question sets, track accuracy, and tool correctness</li><li><strong>Human-in-the-loop</strong>: requires approval for high-impact actions</li><li><strong>Retry logic</strong>: handle failed tool calls and partial results gracefully</li></ul><p>These additional layers transform a demo chatbot into a reliable, production-grade agent — one your users can trust, and engineers can continuously improve.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d049a79a434e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Modern Data Lakehouse Locally: Real-Time E-Commerce Analytics (Part 2/2)]]></title>
            <link>https://medium.com/@kimanthi_de/building-a-modern-data-lakehouse-locally-real-time-e-commerce-analytics-part-2-2-235203bc1e4e?source=rss-a697849c1f63------2</link>
            <guid isPermaLink="false">https://medium.com/p/235203bc1e4e</guid>
            <dc:creator><![CDATA[Kimanthi]]></dc:creator>
            <pubDate>Fri, 21 Nov 2025 00:09:45 GMT</pubDate>
            <atom:updated>2025-11-25T16:51:28.817Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>From Raw Data to Business Insights: Orchestration, Querying, and Visualization</blockquote><p>In <a href="https://medium.com/@kimanthi_de/building-a-modern-data-lakehouse-locally-real-time-e-commerce-analytics-part-1-5a84252586da">Part 1</a>, we laid the foundation for our modern data lakehouse — streaming events through Kafka, processing them with Apache Spark, and storing them in Iceberg tables with Nessie version control. We successfully streamed data from Bronze (raw) to Silver (cleaned) layer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ThE-eGKNf4GdD56e9MWWiw.png" /><figcaption>Image created by author</figcaption></figure><p>But raw and cleaned data alone don’t drive business decisions. We need:</p><ul><li><strong>Aggregated data</strong> that can easily answer specific questions</li><li><strong>Automated orchestration</strong> to keep pipelines running and data up to date</li><li><strong>Fast SQL queries</strong> to explore and analyze data</li><li><strong>Interactive dashboards</strong> to visualize insights</li></ul><p>In Part 2, we’ll complete our lakehouse by integrating:</p><p><strong>Dagster</strong> — Modern orchestration for batch jobs</p><p><strong>Trino</strong> — High-performance distributed SQL engine</p><p><strong>Apache Superset</strong> — Open-source data visualization platform</p><p>By the end, we’ll have a fully functional analytics platform running locally.</p><h3>Dagster: Modern Data Orchestration</h3><p><a href="https://dagster.io/">Dagster</a> is a modern orchestration platform that treats data pipelines as software-defined assets. It centers the workflow around the data itself, providing first-class support for asset lineage, built-in testing, rich observability, and clear mapping of upstream and downstream dependencies.</p><p>Key concepts:</p><ul><li><strong>Assets</strong> — The data artifacts your pipeline creates.</li><li><strong>Jobs</strong> — Collections of assets that run together as a unit.</li><li><strong>Schedules</strong> — Definitions that specify when jobs should execute.</li><li><strong>Resources</strong> — Connections to external systems such as Spark, databases, and APIs.</li><li><strong>Dagster Pipes</strong> — a unified protocol for sending context to external jobs and collecting structured outputs from them — for example, submitting a Spark job to a cluster and streaming back its logs and results.</li></ul><p><strong>Dagster Setup (Docker Compose)</strong></p><pre>services:<br>  postgres:<br>    image: postgres:14<br>    environment:<br>      POSTGRES_DB: dagster<br>      POSTGRES_USER: dagster<br>      POSTGRES_PASSWORD: dagster<br><br>  dagster-daemon:<br>    build:<br>      context: .<br>      dockerfile: Dockerfile.dagster.pyspark<br>    container_name: ecommerce-dagster-daemon<br>    environment:<br>      DAGSTER_POSTGRES_URL: postgresql://dagster:dagster@postgres:5432/dagster<br>    command: dagster-daemon run<br><br>  dagster-webserver:<br>    build:<br>      context: .<br>      dockerfile: Dockerfile.dagster.pyspark<br>    container_name: ecommerce-dagster-webserver<br>    ports:<br>      - &quot;3000:3000&quot;<br>    command: dagster-webserver -h 0.0.0.0 -p 3000</pre><p>In our architecture, Dagster orchestrates batch Spark jobs that transform data from the <strong>Silver layer</strong> into refined<strong> Gold-layer </strong>assets. In a real business environment, the refresh cadence for the Gold layer is driven by business requirements for <strong>data freshness</strong> and <strong>downstream analytical needs</strong>.</p><p><strong>Defining Dagster Assets with Pipes</strong></p><p>The code below defines a daily_customer_analytics Dagster asset that’s created from running a Spark job and uses <strong>Dagster Pipes</strong> to stream real-time logs, capture structured metadata, and report lineage back to Dagster while the external computation runs.</p><pre>from dagster import asset, AssetExecutionContext, PipesSubprocessClient<br><br>@asset(<br>    key_prefix=[&quot;gold&quot;, &quot;analytics&quot;],<br>    description=&quot;Daily customer analytics aggregations using Dagster Pipes&quot;,<br>    group_name=&quot;customer_analytics&quot;,<br>)<br>def daily_customer_analytics(context: AssetExecutionContext, pipes_subprocess_client: PipesSubprocessClient):<br><br>    target_date = &quot;2025-09-20&quot;  <br>    context.log.info(f&quot;Starting customer analytics for {target_date}&quot;)<br><br>    result = pipes_subprocess_client.run(<br>        command=[<br>            &quot;/opt/spark/bin/spark-submit&quot;,<br>            &quot;--master&quot;, &quot;spark://spark-master:7077&quot;,<br>            &quot;--packages&quot;, &quot;org.apache.iceberg:iceberg-spark-runtime-3.5_2.12:1.4.2&quot;,<br>            &quot;/opt/dagster/streaming/spark-streaming/batch_customer_analytics.py&quot;,<br>            &quot;--target-date&quot;, target_date,<br>        ],<br>        context=context,<br>    )<br><br>    return result</pre><blockquote>For the full code of the batch jobs, please refer to the <a href="https://github.com/d-kimanthi/data_lakehouse_pipeline/blob/main/dagster/dagster_project/assets.py">GitHub</a> repository. We have three batch jobs<strong>:</strong><em>daily_customer_analytics</em> <em>product_performance_analytics daily_sales_summary</em></blockquote><p>The <a href="https://github.com/d-kimanthi/data_lakehouse_pipeline/blob/main/streaming/spark-streaming/batch_customer_analytics.py">Spark Job</a>s use the Dagster context to report logs to the Dagster UI.</p><pre>def process_daily_customer_analytics(target_date: str):<br>    &quot;&quot;&quot;Process daily customer analytics for the given date with Dagster Pipes&quot;&quot;&quot;<br>    try:<br>        with dagster_pipes_context() as context:<br>            context.log.info(&quot;Starting Spark customer analytics job&quot;)<br>            context.log.info(f&quot;Processing date: {target_date}&quot;)<br><br>            # Create Spark session using common configuration<br>            spark = create_spark_session(<br>                app_name=f&quot;CustomerAnalytics-{target_date}&quot;, include_kafka=False<br>            )<br><br>            context.log.info(&quot;Spark session initialized successfully&quot;)<br><br>            try:<br>                # Check available silver layer data<br>                check_silver_tables(context, spark)<br><br>                # Calculate daily customer metrics using Iceberg tables<br>                customer_analytics = spark.sql(<br>                    f&quot;&quot;&quot;<br>                    WITH page_view_metrics AS (<br>                        SELECT <br>                            user_id,<br>                            date(timestamp) as date,<br>                            count(*) as page_views,<br>                            count(distinct session_id) as unique_sessions,<br>                            count(distinct CASE WHEN product_id IS NOT NULL THEN product_id END) as unique_products_viewed,<br>                            min(timestamp) as first_visit_time,<br>                            max(timestamp) as last_activity_time<br>                        FROM iceberg.silver.page_views<br>                        WHERE date(timestamp) &gt;= date(&#39;{target_date}&#39;)<br>                        GROUP BY 1,2<br>                    ),</pre><p>Dagster UI:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I1uvhQaEyX-8GVepfDi4Hg.png" /><figcaption>Dagster logs from Spark Cluster</figcaption></figure><p><strong>Jobs and Schedules</strong></p><p>In Dagster, <strong>Jobs</strong> define the execution of one or more assets, while <strong>Schedules</strong> specify when those jobs should run automatically.</p><p>In this code, we create a job that runs the daily_customer_analytics asset and set a schedule to trigger it every day at 3 AM (initially disabled by default).</p><pre>from dagster import define_asset_job, ScheduleDefinition, DefaultScheduleStatus, AssetSelection<br><br>customer_analytics_job = define_asset_job(<br>    name=&quot;customer_analytics_job&quot;,<br>    selection=AssetSelection.assets(daily_customer_analytics),<br>)<br><br>customer_analytics_schedule = ScheduleDefinition(<br>    job=customer_analytics_job,<br>    cron_schedule=&quot;0 3 * * *&quot;,<br>    default_status=DefaultScheduleStatus.STOPPED,<br>)</pre><p><strong>Register definitions</strong></p><pre>from dagster import Definitions, PipesSubprocessClient<br><br>defs = Definitions(<br>    assets=[daily_customer_analytics],<br>    jobs=[customer_analytics_job],<br>    schedules=[customer_analytics_schedule],<br>    resources={&quot;pipes_subprocess_client&quot;: PipesSubprocessClient()},<br>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3hlMLHHu4oLDGJNh005v0Q.png" /><figcaption>Dagster assets overview</figcaption></figure><p>To run the pipeline manually on the Dagster UI, navigate to the daily_customer_analytics asset and click <strong>Materialize — </strong>this submits the Spark job to the cluster and writes the resulting dataset to the gold layer.</p><h3>Trino: High-Performance SQL Engine</h3><p><a href="https://trino.io/"><strong>Trino</strong></a><strong> </strong>is a high-performance, distributed SQL engine that enables fast, interactive querying across diverse data sources without moving the data. It uses a massively parallel processing architecture, supports ANSI SQL, and integrates easily with BI tools.</p><blockquote>Think of Trino as the “query layer” of your lakehouse — connecting business intelligence tools, notebooks, and applications directly to your data.</blockquote><p>In our setup, we’ve configured Trino to query Iceberg tables through the Nessie catalog. This setup enables ad-hoc queries for interactive analytics — ideal for quick data validation checks — and allows seamless integration with Superset for building dashboards and visualizing insights.</p><p><strong>Configuring Trino for Iceberg</strong></p><p>Our Docker Compose includes Trino with an Iceberg catalog.</p><pre>trino:<br>  image: trinodb/trino:435<br>  container_name: ecommerce-trino<br>  ports:<br>    - &quot;8081:8080&quot;<br>  volumes:<br>    - ./trino/catalog:/etc/trino/catalog<br>    - ./trino/config.properties:/etc/trino/config.properties</pre><p>The Iceberg catalog <a href="https://github.com/d-kimanthi/data_lakehouse_pipeline/blob/main/local/trino/catalog/iceberg.properties">configuration</a>:</p><pre>connector.name=iceberg<br>iceberg.catalog.type=nessie<br>iceberg.nessie-catalog.uri=http://nessie:19120/api/v1<br>iceberg.nessie-catalog.default-warehouse-dir=s3a://data-lake/warehouse<br>iceberg.nessie-catalog.ref=main<br><br># MinIO/S3 configuration<br>fs.native-s3.enabled=true<br>s3.endpoint=http://minio:9000<br>s3.path-style-access=true<br>s3.aws-access-key=MINIOUSER<br>s3.aws-secret-key=MINIOPASSWORD<br>s3.region=us-east-1<br><br># File format<br>iceberg.file-format=PARQUET</pre><p><strong>Querying with Trino</strong></p><p>To access the Trino CLI : docker exec -it ecommerce-trino trino</p><pre><br>-- Show all schemas<br>SHOW SCHEMAS FROM iceberg;<br><br>--Show tables in Gold layer<br>SHOW TABLES FROM iceberg.gold;<br><br>-- Top 10 high-value customers<br>SELECT<br>    user_id,<br>    total_revenue,<br>    total_purchases,<br>    avg_order_value,<br>    customer_segment<br>FROM iceberg.gold.daily_customer_analytics<br>WHERE customer_segment = &#39;high_value&#39;<br>ORDER BY total_revenue DESC<br>LIMIT 10;</pre><p><strong>How Trino Interacts with Iceberg</strong></p><p>Trino accesses Iceberg tables via the Iceberg connector, with the data catalog (Nessie in our case) functioning as the central metadata service for tracking schemas, snapshots, and partition information. The query execution flow works as follows:</p><ol><li><strong>Metadata Discovery</strong>: Trino first retrieves comprehensive table metadata from the Iceberg catalog, including schema structure, partitioning strategy, snapshot information, and file paths.</li><li><strong>Intelligent Planning</strong>: Using this metadata, Trino’s optimizer determines the minimal set of data files needed to satisfy the query.</li><li><strong>Parallel Data Access</strong>: Trino workers concurrently read the selected data files from storage (MinIO, S3, GCS, or HDFS) while applying relevant filters and column selections.</li><li><strong>Transaction Isolation</strong>: Iceberg’s snapshot mechanism ensures each query operates on a stable, consistent view of the data, regardless of ongoing modifications.</li><li><strong>Time-Travel Capabilities</strong>: Through catalog integration, Trino can retrieve and query historical table versions, enabling both point-in-time analysis and data recovery.</li></ol><h3><strong>Apache Superset: Interactive Dashboards</strong></h3><p>With clean, aggregated data and a high-performance query engine in place, we can now create dashboards for business users. While Trino excels at ad-hoc queries, a dedicated visualization tool is needed for dashboarding, trend analysis, and KPI monitoring.</p><p><a href="https://superset.apache.org/">Apache Superset</a> is an open-source platform for data exploration and visualization, supporting connections to over 40 data sources and enabling complex SQL queries as well as custom visualizations.</p><p><strong>Setting Up Superset</strong></p><p>Our Docker Compose setup includes Superset with PostgreSQL as its metadata database, which stores Superset’s internal state, configuration, and operational metadata.</p><pre>superset-db:<br>  image: postgres:14<br>  environment:<br>    POSTGRES_DB: superset<br>    POSTGRES_USER: superset_user<br>    POSTGRES_PASSWORD: superset_password<br><br>superset:<br>  image: apache/superset:3.0.0<br>  container_name: ecommerce-superset<br>  ports:<br>    - &quot;8088:8088&quot;<br>  environment:<br>    SUPERSET_SECRET_KEY: your_secret_key_here</pre><p><strong>Connecting Superset to Trino</strong></p><p>To connect Superset to Trino, follow <a href="https://superset.apache.org/docs/configuration/databases#trino">this</a> official documentation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wg2rftshEnXf2z__NOmHEA.png" /><figcaption>Superset Trino Connection</figcaption></figure><p>Before creating dashboards, we need to add datasets to Superset — these correspond to dimension tables or any final aggregated tables intended for reporting. In our setup, we have three datasets representing the data assets produced by our scheduled batch jobs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VUDXg-TY-Wd5gH7ZCUhlpg.png" /></figure><p><strong>Creating A Chart</strong></p><p>To build a dashboard in Superset, you create individual charts using the datasets added in the previous step. You can also write custom SQL queries to shape the data. The image below shows a “Purchases by Category” chart created from the product_performance_analytics dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P4riw2NILFpsem8Q299YYA.png" /></figure><p><strong>Building A Dashboard</strong></p><p>Finally, individual charts are combined and added to dashboards, and a single chart can be included in multiple dashboards. Dashboards in Superset support features such as scheduled data refreshes, interactive filters, and rich visualizations, enabling dynamic and up-to-date reporting for business users.</p><p>Below is a snapshot of a sample dashboard created from our simulated data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S3Ota2lBRyD_dKraoAqizg.png" /><figcaption>Superset Dashboard</figcaption></figure><h3><strong>Conclusion</strong></h3><p>Congratulations! You’ve successfully built a complete modern data lakehouse from scratch. Here’s a recap of what we accomplished:</p><p><strong>Part 1: Foundation</strong></p><ul><li>Simulated realistic e-commerce events using Python</li><li>Streamed events through Kafka topics</li><li>Processed data with Spark (both streaming and batch)</li><li>Stored data in Iceberg tables following the Bronze → Silver → Gold architecture</li><li>Version-controlled data with Nessie</li></ul><p><strong>Part 2: Analytics Layer</strong></p><ul><li>Orchestrated batch jobs with Dagster</li><li>Queried data interactively using Trino</li><li>Created dashboards and visualizations with Superset</li></ul><p>While our local setup demonstrates the core concepts, moving to production requires careful consideration of several key areas to ensure a robust and efficient architecture. Some important aspects to keep in mind include (this is not exhaustive):</p><ul><li><strong>Technology and Tool Selection</strong>: Choose technologies and tools based on business requirements, team size, expertise, and long-term maintainability.</li><li><strong>Scalability</strong>: Apply proper partitioning strategies (e.g., by date, region, or product), perform regular Iceberg table compactions, and scale compute resources horizontally.</li><li><strong>Data Quality</strong>: Implement schema validation using tools like Great Expectations, add data profiling to detect anomalies early, and build backfill processes to handle late-arriving data.</li><li><strong>Security</strong>: Ensure encryption, authentication, and authorization are properly configured to control data access.</li></ul><p>If you found this series helpful, give it a clap 👏 and share it with others who are learning data engineering!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=235203bc1e4e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Modern Data Lakehouse Locally: Real-Time E-Commerce Analytics (Part 1/2)]]></title>
            <link>https://medium.com/@kimanthi_de/building-a-modern-data-lakehouse-locally-real-time-e-commerce-analytics-part-1-5a84252586da?source=rss-a697849c1f63------2</link>
            <guid isPermaLink="false">https://medium.com/p/5a84252586da</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[modern-data-stack]]></category>
            <dc:creator><![CDATA[Kimanthi]]></dc:creator>
            <pubDate>Mon, 10 Nov 2025 02:49:50 GMT</pubDate>
            <atom:updated>2025-11-25T16:50:06.441Z</atom:updated>
            <content:encoded><![CDATA[<blockquote><strong><em>How Kafka, Spark, and Iceberg come together to power modern data engineering</em></strong></blockquote><p>If you’re getting into data engineering, you’ve probably heard the buzzwords — <em>streaming, lakehouse, medallion architecture, ACID tables</em> — but what do they actually mean in practice?</p><p>In this two-part series, we’ll build a working <strong>modern data lakehouse</strong> on your local machine using <strong>Docker</strong>, <strong>Kafka</strong>, <strong>Spark</strong>, and <strong>Iceberg</strong>.<br>You’ll see exactly how raw events flow through each layer — from data generation to structured analytics — and understand <em>why</em> these technologies exist.</p><h3>Our Goal</h3><p>We’re building a <strong>real-time e-commerce analytics system</strong> that can answer questions like:</p><ul><li>How many users added items to their cart but never checked out?</li><li>Which products get the most page views?</li><li>Who are our most valuable customers over time?</li></ul><p>We’ll simulate user events, stream them through Kafka, process them with Spark, and store them as Iceberg tables.</p><h3>The Medallion Architecture</h3><p>At the heart of modern data systems is the <a href="https://www.databricks.com/glossary/medallion-architecture"><strong>medallion architecture</strong></a> — a simple but powerful design that organizes data by layers of refinement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/1*3NRmXk3WscBlSrsqz4yKtQ.png" /><figcaption>Three layers of the medallion archtecture</figcaption></figure><p>In this setup, each data layer corresponds to an S3-style directory structure in MinIO. The <strong>Bronze</strong> layer serves as the raw data repository, capturing raw user events directly from Kafka streams. The <strong>Silver</strong> layer stores cleaned and standardized datasets derived from the Bronze layer, ensuring data quality and consistency. Finally, the <strong>Gold</strong> layer contains aggregated and curated datasets designed to support analytics and answer the specific business questions outlined above.</p><h3>The Tech Stack</h3><p>Here’s what we’ll use to make it happen:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3CG8OxTnzY9vMttp-vqMKg.png" /><figcaption>Modern data lakehouse archtetcure</figcaption></figure><ul><li><strong>Python Event Generator</strong> — Custom Python scripts designed to simulate realistic user interaction events and publish them to Kafka topics.</li><li><strong>Apache Kafka</strong> — Serves as the event streaming backbone, enabling high-throughput, real-time ingestion of user interaction data.</li><li><strong>Apache Spark</strong> — Processes streaming and batch data, performing transformations, cleaning, and aggregations across layers.</li><li><strong>MinIO</strong> — Acts as an S3-compatible object storage system for persisting raw, intermediate, and processed datasets.</li><li><strong>Apache Iceberg</strong> — Provides an ACID-compliant table format for reliable storage, schema evolution, and time travel.</li><li><strong>Project Nessie</strong> — Integrates Git-like version control for Iceberg tables, enabling reproducible data states and branching workflows.</li><li><strong>Dagster</strong> — Manages orchestration and scheduling of batch and ETL pipelines across the data layers.</li><li><strong>Trino</strong> — Serves as a distributed SQL query engine optimized for fast analytics over Iceberg datasets.</li><li><strong>Apache Superset</strong> — Delivers interactive dashboards and visualizations powered by queries from the Gold layer.</li></ul><h3>Local Development Setup</h3><p>You can run the entire stack locally using <strong>Docker Compose</strong> — no cloud setup or external dependencies required. This environment is designed as a <strong>hands-on learning and demo setup</strong> to illustrate how each component of a modern data lakehouse architecture fits together. All the code is in this <a href="https://github.com/d-kimanthi/data_lakehouse_pipeline">git</a> repository.</p><p><strong><em>Note:</em></strong><em> This configuration is tailored for </em><strong><em>educational and exploration purposes</em></strong><em>. It replicates the architecture and workflows used in production-grade systems but uses lightweight settings optimized for running seamlessly on a single machine.</em></p><pre>git clone https://github.com/d-kimanthi/data_lakehouse_pipeline #clone repository<br>cd data_lakehouse_pipeline/local<br>cp .env.example .env #create and update .env file from the provided template<br>docker compose up -d</pre><p>Below is a sample of the <a href="https://github.com/d-kimanthi/data_lakehouse_pipeline/blob/main/local/docker-compose.yml">Docker Compose</a> file:</p><pre><br>services:<br>  # Zookeeper - Required for Kafka<br>  zookeeper:<br>    image: confluentinc/cp-zookeeper:7.4.0<br>    container_name: ecommerce-zookeeper<br>    environment:<br>      ZOOKEEPER_CLIENT_PORT: 2181<br>      ZOOKEEPER_TICK_TIME: 2000<br>      ZOOKEEPER_DATA_DIR: /var/lib/zookeeper/data<br>      ZOOKEEPER_DATA_LOG_DIR: /var/lib/zookeeper/log<br>    ports:<br>      - &quot;2181:2181&quot;<br>    volumes:<br>      - zookeeper_data:/var/lib/zookeeper/data<br>      - zookeeper_logs:/var/lib/zookeeper/log<br>    user: &quot;0:0&quot;<br>    networks:<br>      - ecommerce-network<br><br>  # Kafka - Message streaming platform<br>  kafka:<br>    image: confluentinc/cp-kafka:7.4.0<br>    container_name: ecommerce-kafka<br>    depends_on:<br>      - zookeeper<br>    ports:<br>      - &quot;9092:9092&quot;<br>      - &quot;9101:9101&quot;<br>    environment:<br>      KAFKA_BROKER_ID: 1<br>      KAFKA_ZOOKEEPER_CONNECT: &#39;zookeeper:2181&#39;<br>      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT<br>      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092<br>      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1<br>      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1<br>      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1<br>      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0<br>      KAFKA_JMX_PORT: 9101<br>      KAFKA_JMX_HOSTNAME: localhost<br>      KAFKA_AUTO_CREATE_TOPICS_ENABLE: &#39;true</pre><p>If all containers start successfully, we’re ready to begin developing and testing our data pipeline end-to-end. Additionally, Docker Compose initializes the <strong>MinIO</strong> folder structure, laying the foundation for our Bronze, Silver, and Gold data layers in the lakehouse.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/1*ppALKJ_6InKZr8bN1XHRjQ.png" /><figcaption>MinIO Datalake folder setup</figcaption></figure><h3>1. Event Generation with Python</h3><p>Before we can build data pipelines, we first need data. Our Python event generator, built using dataclasses, simulates realistic e-commerce user behavior, including page views, cart actions, and purchases — just like a real website would produce.</p><pre><br>class KafkaEventProducer:<br>    &quot;&quot;&quot;<br>    Kafka producer for streaming e-commerce events<br>    Routes events to topic-specific queues based on event type<br>    &quot;&quot;&quot;<br><br>    def __init__(self, bootstrap_servers: str, base_topic: str = &quot;raw-events&quot;):<br>        self.base_topic = base_topic<br>        self.bootstrap_servers = bootstrap_servers<br><br>        # Define topic mapping for different event types<br>        self.topic_mapping = {<br>            &quot;page_view&quot;: &quot;page-views&quot;,<br>            &quot;add_to_cart&quot;: &quot;order-events&quot;,<br>            &quot;purchase&quot;: &quot;purchases&quot;,<br>            &quot;user_session&quot;: &quot;user-sessions&quot;,<br>            &quot;product_update&quot;: &quot;product-updates&quot;,<br>        }<br><br>        self.producer = KafkaProducer(<br>            bootstrap_servers=bootstrap_servers,<br>            value_serializer=lambda x: json.dumps(x).encode(&quot;utf-8&quot;),<br>            key_serializer=lambda x: x.encode(&quot;utf-8&quot;) if x else None,<br>            acks=&quot;all&quot;,<br>            retries=3,<br>            batch_size=16384,<br>            linger_ms=10,<br>            buffer_memory=33554432,<br>        )<br>        self.event_generator = EcommerceEventGenerator()</pre><pre>#running the shell script below will start the data generation to the Kafka topics<br>cd data_lakehouse_pipeline/data-generation<br>./generate_sample_data.sh</pre><p>Instead of saving to a file, we <strong>write these events directly to Kafka topics</strong>. In production, this would be replaced by a real web and/or mobile app generating user activity events.</p><h3>2. Apache Kafka: The Streaming Backbone</h3><p><strong>Apache Kafka</strong> is the central nervous system of modern data architectures.<br> It’s a distributed log system where producers (like our event generator) send messages, and consumers (like Spark) read them at their own pace.</p><p><strong>Key Kafka concepts:</strong></p><ul><li><strong>Topics</strong> — Named data streams (e.g., page_views, purchases)</li><li><strong>Producers — A</strong>pplications that publish (write) events to Kafka topics.</li><li><strong>Consumers</strong> — Applications that subscribe to (read) events from topics.</li><li><strong>Partitions</strong> — Each topic is divided into partitions; They enable horizontal scaling</li><li><strong>Consumer Groups</strong> — Multiple consumers can share the load</li><li><strong>Offset</strong> — Position of each consumer in the topic</li></ul><p>In our use case, the event generator class is the sole producer for all the data we will be using. The code sample is shown below.</p><pre>def send_event(self, event: Dict[str, Any], target_topic: str):<br>        &quot;&quot;&quot;Send a single event to the appropriate Kafka topic&quot;&quot;&quot;<br>        try:<br><br>            # Use user_id as key for proper partitioning<br>            key = event.get(&quot;user_id&quot;) or event.get(&quot;product_id&quot;, &quot;unknown&quot;)<br><br>            # Send to the specific topic<br>            self.producer.send(topic=target_topic, key=key, value=event)<br><br>            # Log the routing decision<br>            logger.debug(<br>                f&quot;Sent {event.get(&#39;event_type&#39;)} event to topic: {target_topic} with key: {key}&quot;<br>            )<br><br>        except Exception as e:<br>            logger.error(f&quot;Error sending event: {e}&quot;)</pre><p>To inspect the events being produced and published to Kafka topics, we can open the <strong>Kafka UI</strong> available at <a href="http://localhost:8080/">http://localhost:8080</a>, as defined in our Docker Compose setup. This interface lets us monitor topics, browse messages, and verify that the event generator is streaming data correctly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P-MWzG4emAGf9KjSEOPzbA.png" /><figcaption>Kafka UI</figcaption></figure><p>Kafka ensures reliability through <strong>durability</strong>, <strong>ordering</strong>, and <strong>replayability</strong> — crucial for real-time data systems.</p><h3>3. Apache Spark: Unified Processing Engine</h3><p>Spark is where the heavy lifting happens — transforming, cleaning, and aggregating the data from Kafka. In our setup, we use <strong>Apache Spark</strong> to process data across all three layers of the lakehouse.</p><p>Apache Spark’s unified API simplifies our data pipeline by handling both streaming and batch processing seamlessly. We run three Spark jobs across our lakehouse layers:</p><ol><li><strong>Streaming Job 1 (Bronze ingestion):</strong> Subscribes to Kafka topics, deserializes events, and writes raw data to the Bronze layer in Iceberg format.</li><li><strong>Streaming Job 2 (Silver transformation):</strong> Processes data from Bronze to Silver, applying cleaning and enrichment in real-time.</li><li><strong>Batch Job (Gold aggregation):</strong> Aggregates and curates data from Silver to Gold for analytics and dashboards.</li></ol><blockquote>Please note: before running the streaming jobs, we need to <strong>create the Iceberg tables</strong> with the expected schemas, as defined in the accompanying <a href="https://github.com/d-kimanthi/data_lakehouse_pipeline/blob/main/streaming/spark-streaming/real_time_streaming.py">script</a>.</blockquote><p>To start the <a href="https://github.com/d-kimanthi/data_lakehouse_pipeline/blob/main/streaming/spark-streaming/real_time_streaming.py">Spark streaming</a> jobs, run the following shell script, which accepts --job-type flag for either ingestion ,processing or both</p><pre>cd cd data_lakehouse_pipeline/streaming<br>#Job types: ingestion, processing, both (default)<br># ingestion -&gt; kafka to bronze, processing -&gt; bronze to silver<br>./spark-streaming/run_streaming.sh [--job-type TYPE]</pre><pre># Bronze processing - minimal transformation, preserve raw data<br>      bronze_df = kafka_df.select(<br>          # Extract event data from JSON<br>          coalesce(<br>              col(&quot;key&quot;).cast(&quot;string&quot;),<br>              get_json_object(col(&quot;value&quot;).cast(&quot;string&quot;), &quot;$.event_id&quot;),<br>              concat(lit(&quot;kafka_&quot;), col(&quot;offset&quot;).cast(&quot;string&quot;)),<br>          ).alias(&quot;event_id&quot;),<br>          coalesce(<br>              get_json_object(col(&quot;value&quot;).cast(&quot;string&quot;), &quot;$.event_type&quot;),<br>              lit(&quot;unknown&quot;),<br>          ).alias(&quot;event_type&quot;),<br>          coalesce(<br>              to_timestamp(<br>                  get_json_object(col(&quot;value&quot;).cast(&quot;string&quot;), &quot;$.timestamp&quot;)<br>              ),<br>              col(&quot;timestamp&quot;),<br>          ).alias(&quot;timestamp&quot;),<br>          # Kafka metadata for lineage<br>          col(&quot;topic&quot;),<br>          col(&quot;partition&quot;),<br>          col(&quot;offset&quot;),<br>          # Keep Raw data - complete JSON payload<br>          col(&quot;value&quot;).cast(&quot;string&quot;).alias(&quot;raw_data&quot;),<br>          # Add processing metadata<br>          current_timestamp().alias(&quot;processing_time&quot;),<br>      )<br><br>      # Write to Bronze Iceberg table<br>      bronze_query = (<br>          bronze_df.writeStream.format(&quot;iceberg&quot;)<br>          .outputMode(&quot;append&quot;)<br>          .option(&quot;checkpointLocation&quot;, &quot;s3a://data-lake/checkpoints/bronze&quot;)<br>          .option(&quot;path&quot;, &quot;iceberg.bronze.raw_events&quot;)<br>          .trigger(processingTime=&quot;30 seconds&quot;)<br>          .start()<br>      )</pre><h4>Optimizing Spark</h4><p>Optimizing Spark requires tuning a few key configurations that directly impact performance and reliability — for example:</p><ul><li>spark.executor.memory, spark.driver.memory – manage memory</li><li>spark.sql.shuffle.partitions – control parallelism</li><li>checkpointLocation – ensures fault-tolerance for streaming</li></ul><p>In our setup, we primarily use Spark’s defaults, but we also add custom settings for <strong>streaming</strong> and <strong>Iceberg</strong> to ensure exactly-once processing and efficient writes.</p><h3>4. Apache Iceberg: The Table Format</h3><p><a href="https://iceberg.apache.org/">Apache Iceberg</a> is an <strong>open table format</strong> that brings <strong>ACID transactions</strong>, <strong>schema evolution</strong>, and <strong>time travel</strong> to data lakes. It enables data engineers to manage petabyte-scale datasets on cloud object storage (such as S3, GCS, or Azure Blob) with the same reliability as traditional data warehouses.</p><blockquote>In short, Iceberg gives structure and reliability to what was once just a chaotic collection of Parquet files.</blockquote><h4>How Iceberg Tables Are Structured</h4><p>Iceberg introduces a clear <strong>two-layer architecture</strong> to make a data lake more reliable.</p><p><strong>Data Layer — </strong>Stores the actual records as Parquet (or ORC/Avro) files, typically organized by partitions like date or region.</p><p><strong>Metadata Layer </strong>— Stores JSON and Avro files that describe which data files belong to which version (snapshot) of the table.</p><p>Example table layout:</p><pre>bronze.page_views/<br>├── metadata/<br>│   ├── v1.metadata.json<br>│   ├── v2.metadata.json<br>│   ├── v3.metadata.json      # Current snapshot<br>│   └── manifest-123.avro<br>├── data/<br>│   ├── event_date=2025-09-20/<br>│   │   ├── data-001.parquet<br>│   │   └── data-002.parquet<br>│   ├── event_date=2025-09-21/<br>│   │   ├── data-003.parquet<br>│   │   └── data-004.parquet</pre><p>Each metadata version points to a specific collection of data files — forming a consistent, queryable snapshot.</p><h3>5. Project Nessie: Git for Data</h3><p><a href="https://projectnessie.org/"><strong>Project Nessie</strong></a> is a <strong>data catalog and version control system</strong> that brings <strong>Git-like operations</strong> — such as branching, tagging, and merging — to your data lake. A <strong>data catalog</strong> serves as a centralized metadata repository that tracks datasets, their schemas, versions, and lineage across the lakehouse.</p><p>The Git-like workflow provided by Nessie allows us to <strong>create isolated branches</strong> for development or testing, run new <strong>data pipelines safely</strong> without impacting production, and then <strong>merge validated changes</strong> into the main branch. This approach brings software engineering best practices — such as version control, reproducibility, and CI/CD — to data cataloging.</p><p><strong>How It Works</strong></p><p>By default, Iceberg needs a <strong>catalog</strong> to manage metadata and table locations. Nessie acts as a <strong>catalog implementation</strong> that tracks Iceberg tables just like Git tracks files. So instead of using a HadoopCatalog or HiveCataloyou use a NessieCatalog. To have your Iceberg tables automatically tracked<strong> </strong>by Nessie, you need to configure <strong>Spark</strong> to use <strong>Nessie </strong>as its Iceberg catalog.</p><p><strong>Spark Configuration</strong></p><p>You can define these settings either in your <strong>Spark configuration </strong>file (e.g. spark-defaults.conf), or dynamically when starting a Spark session.</p><pre>from pyspark.sql import SparkSession<br><br>spark = (<br>    SparkSession.builder<br>    .appName(&quot;ecommerce-streaming&quot;)<br>    .config(&quot;spark.sql.catalog.nessie&quot;, &quot;org.apache.iceberg.spark.SparkCatalog&quot;)<br>    .config(&quot;spark.sql.catalog.nessie.catalog-impl&quot;, &quot;org.projectnessie.iceberg.NessieCatalog&quot;)<br>    .config(&quot;spark.sql.catalog.nessie.uri&quot;, &quot;http://nessie:19120/api/v1&quot;)<br>    .config(&quot;spark.sql.catalog.nessie.ref&quot;, &quot;main&quot;)<br>    .config(&quot;spark.sql.catalog.nessie.authentication.type&quot;, &quot;NONE&quot;)<br>    .config(&quot;spark.sql.catalog.nessie.warehouse&quot;, &quot;s3a://datalake/warehouse&quot;)<br>    .getOrCreate()<br>)</pre><p>Now, when you run commands like:</p><pre>spark.sql(&quot;CREATE TABLE iceberg.bronze.raw_events (event_id STRING, event_type STRING, raw_data STRING) USING iceberg&quot;)</pre><p>The table will be:</p><ul><li>Stored in <strong>MinIO</strong> under the specified warehouse path.</li><li><strong>Version-controlled by Nessie</strong>, appearing as part of the active branch (main by default).</li></ul><h3>Wrapping Up</h3><p>In this first part, we’ve:</p><ul><li>Generated streaming events using Python</li><li>Streamed them through Kafka</li><li>Processed them with Spark</li><li>Stored them in Iceberg tables</li><li>Added cataloging and version control with Nessie</li></ul><p><strong>In </strong><a href="https://medium.com/@kimanthi_de/building-a-modern-data-lakehouse-locally-real-time-e-commerce-analytics-part-2-2-235203bc1e4e"><strong>Part</strong></a><strong> 2</strong>, we integrate <strong>Dagster</strong> for orchestration, <strong>Trino</strong> for querying, and <strong>Superset</strong> for visualization — completing our end-to-end local lakehouse.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5a84252586da" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>