<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ismail Simsek on Medium]]></title>
        <description><![CDATA[Stories by Ismail Simsek on Medium]]></description>
        <link>https://medium.com/@ismail-simsek?source=rss-2215c1d16895------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*etjbfH9XK8SlakMGt0DKyA.jpeg</url>
            <title>Stories by Ismail Simsek on Medium</title>
            <link>https://medium.com/@ismail-simsek?source=rss-2215c1d16895------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 00:21:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ismail-simsek/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[MCP and Opportunities: A Practical Approach to AI Integration in Business]]></title>
            <link>https://medium.com/@ismail-simsek/mcp-and-opportunities-a-practical-approach-to-ai-integration-in-business-765bf0677895?source=rss-2215c1d16895------2</link>
            <guid isPermaLink="false">https://medium.com/p/765bf0677895</guid>
            <category><![CDATA[mcp-server]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[agentic-business]]></category>
            <dc:creator><![CDATA[Ismail Simsek]]></dc:creator>
            <pubDate>Tue, 03 Nov 2026 08:16:01 GMT</pubDate>
            <atom:updated>2026-11-03T08:16:01.445Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Why LLMs need tools to be useful, and how to use MCP to unify context without building a bloated middleware trap.</em></p><p>LLM models by themselves are diluted, average, condensed knowledge bases. On their own, they don’t offer much business value beyond giving you global statistics or general knowledge summaries.</p><p>Their real value increases when they are integrated with tools — when they are given access to different internal systems and allowed to take action within them. Today, this is done with <strong>MCP</strong> (Model Context Protocol).</p><p>There is a broad understanding that MCP is similar to an API layer, which is correct. Looking at an <strong>API + MCP</strong> setup in practice, I see two clear opportunities for businesses.</p><h4>1. The Fast Bootstrap: Bulk API-to-MCP Setup</h4><p>First, as a systematic technical setup, you can turn your existing internal APIs into MCP tools in bulk.<br>Almost overnight, you can make your business “LLM-ready” by exposing your API endpoints to an LLM through an MCP server. As a first step to get your systems connected and talking to AI quickly, this is the <strong><em>fastest</em></strong> technical bridge. However, your runway with this approach will be short — eventually, it will need more.</p><h4>2. The Real Value: A Lean Aggregator and Context Layer</h4><p>Second — and more importantly — having MCP on top of your APIs adds a new layer to your IT system. Just as a Backend-for-Frontend (BFF) shapes data for a mobile UI, an MCP server should act as a “Backend-for-Agents,” shaping high-signal, token-efficient context for an LLM. Since an MCP tool is simply code sitting on top of your APIs, we can add lightweight logic on top of those endpoints. This opens up a few valuable opportunities:</p><ul><li><strong>Unifying scattered APIs into one cohesive MCP service</strong>: Instead of forcing the LLM to guess and call multiple fragmented endpoints, the MCP code behind the scenes can connect all relevant sources and build a single, unified representation of a business entity — such as a customer.</li><li><strong>Enhancing APIs with broader context:</strong> We can enrich the data on top of standard API responses by including pre-aggregated metrics, top 10 lists, or the last N orders. With this setup, the MCP layer takes the initiative to provide better, more valuable data to the LLM in a single call.</li></ul><p>To see the difference in practice, compare a naive 1:1 API wrapper to a domain-shaped MCP tool:</p><ul><li>The Naive 1:1 Approach: You expose get_user_by_email, get_orders_by_user_id, and get_support_tickets as separate tools. To answer a simple question about a customer, the LLM has to make three sequential round-trips, parsing bloated raw JSON at every step—burning tokens, adding latency, and increasing the chance of a hallucinated parameter.</li><li>The Context-Shaped MCP Approach: You expose a single get_customer_360_context tool. Behind the scenes, the MCP server orchestrates those calls and returns one clean, token-efficient summary (identifying details, lifetime value, last 5 orders, and open tickets) in a single round-trip.</li></ul><h4>The Hidden Dependency: Classic Data Engineering and MDM</h4><p>When you try to unify scattered APIs inside an MCP service, one foundational question becomes very important: <strong>how well is your data integrated across the business? </strong>(Unfortunately, in real word, usually the answer is: not much! Iknow the pain first hand!)</p><p>If your MCP tool pulls customer data from three different internal systems, can you actually link those records together easily? Suddenly, building an effective AI integration comes right back to a classic Data Engineering and Master Data Management (MDM) problem. If your underlying records don’t link cleanly across systems, your MCP layer cannot magically unify them.</p><h4>What to Watch Out For: Keep It Lean</h4><p>While adding aggregation and context logic inside MCP is powerful, there is one critical architectural trap to watch out for: <strong>do not turn this MCP layer into a business application itself.</strong></p><p>Heavy data integration, complex business rules, and heavy analytical processing still belong in your backend and data platforms. Your MCP layer is there to unify calls and shape high-value context for the LLM — to work well, it should stay lean and simple.</p><p>Two other critical areas to keep on your production watch list — which needs separate post — include:</p><ul><li>Keeping strict security and restrictions around edit and delete endpoints.</li><li>Using SSO login methods and auth integration to map MCP actions directly to the end user.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=765bf0677895" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AI Readiness Isn’t New — It’s Just Classic Data & Software Engineering Paying Off]]></title>
            <link>https://medium.com/@ismail-simsek/ai-readiness-isnt-new-it-s-just-classic-data-software-engineering-paying-off-3e64bda7f7cf?source=rss-2215c1d16895------2</link>
            <guid isPermaLink="false">https://medium.com/p/3e64bda7f7cf</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Ismail Simsek]]></dc:creator>
            <pubDate>Tue, 03 Nov 2026 07:41:01 GMT</pubDate>
            <atom:updated>2026-11-03T07:41:01.526Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Why models have no moat, and why your existing system design decides if AI becomes a business multiplier or a frustrating feature!</em></p><p>Recent benchmarks and pricing from models like Kimi K1 prove one thing clearly: there is no moat in the models themselves.<br>When every business has access to the same cheap, frontier-level intelligence, the model stops being a differentiator. The real moat is how well a business integrates AI into its processes to deliver valuable features to the end user. Because everyone can plug in an API, the actual differentiation comes down to <strong>1:</strong> how fast and cleanly you manage this transition — <strong>2:</strong> how well you connect your data and products to extract real value.</p><p>That second point raises a critical question: <strong>how ready and elegant are your existing IT, data, and software systems?</strong></p><p>That foundation defines your AI success. It determines whether you can realize valuable AI in a short time, or whether the effort spent yields low value and a frustrating experience. Ultimately, AI is going to be the sharpest differentiator between well-managed IT organizations and bad ones.</p><h4>“Business Context” is Just Classic Engineering</h4><p>AI shines when the business, software, or IT system around it is designed meaningfully with real business representation. Today, the industry calls this “<strong>business context.</strong>”</p><p>Whatever technology you integrate with AI, it needs context to work. But is this actually a new concept or requirement? Quite the opposite. It has always been there as part of engineering best practices and elegant design.</p><p>A good engineer always knew how to reflect the real world inside the software and IT systems they build. The core principle behind Domain-Driven Design (DDD). When a system represents the actual business through cohesive domain components, well-documented design, and rich metadata, AI thrives right out of the box. On the other hand, if your domain is scattered across multiple services and does not uniformly represent business entities, events, and processes, it is hard to get any valuable integration with AI..</p><h4>The Semantic Model Reality Check</h4><p>Lets take data and reporting — my own field of experience — as a concrete example.</p><p>If you don’t have proper dimensional reporting in place — where business processes are represented meaningfully with a dimensional model, for example order process is backed by well-defined, high-quality customer, order, and product dimensions— it will be nearly impossible to add a semantic model and self-service AI reporting on top of it.</p><p>If you didn’t build the foundation right to begin with, layering AI on top will just give you a faster mess.</p><p><strong>AI is a moat and a business multiplier only when the foundation is right and the integration is done right. </strong>Otherwise, it produces a “meh!” effect — leaving teams looking at the bill and asking: <em>Was that really worth it? Did we actually get any value out of this? And most importantly it might frustrate the end user and could cause AI slop burnout!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3e64bda7f7cf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Make DBT Great Again Part 2]]></title>
            <link>https://medium.com/@ismail-simsek/make-dbt-great-again-part-2-758b02d9bcdb?source=rss-2215c1d16895------2</link>
            <guid isPermaLink="false">https://medium.com/p/758b02d9bcdb</guid>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[data-lineage]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-catalog]]></category>
            <dc:creator><![CDATA[Ismail Simsek]]></dc:creator>
            <pubDate>Tue, 27 May 2025 11:41:51 GMT</pubDate>
            <atom:updated>2025-05-27T11:42:42.032Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Catalog with <strong>column level lineage</strong></h3><p><strong>Introducing the New opendbt Data Catalog: Simple, Intuitive, and Customizable. Including column level lineage.</strong></p><p>Great news, <a href="https://github.com/memiiso/opendbt">opendbt</a> now includes significant feature: a brand new, simple, and intuitive <a href="https://memiiso.github.io/opendbt/catalog/">data catalog</a>. Designed with clarity and ease of use in mind, this catalog aims to make understanding your data assets more accessible than ever before.</p><p>This initial version includes essential features like:</p><ul><li><strong>Column-level lineage:</strong> Understand the column level dependency and transformation.</li><li><strong>Last execution time and result:</strong> Quickly see the status of your data processes.</li></ul><p>One of the key benefits of this new catalog is its underlying technology: simple HTML and Vue.js. This makes it incredibly easy to customize, add your company logo etc. Want to tailor it further? You can leverage the power of any coding assistant (like Gemini) to enhance and personalize it to your specific needs.</p><p>Most importantly, this catalog is built with the community in mind — it’s open for everyone to explore, improve, and contribute!</p><p>See the catalog in action with our <a href="https://memiiso.github.io/opendbt/opendbtdocs/">live demo here</a></p><p>Let’s dive deeper into the Key Features below.</p><h3>Up to date Run information</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RO6Z9_HeHGwKYAjL.png" /></figure><h3>Run information with error messages</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ISRHqbuWs3JVRgdr.png" /></figure><h3>Model dependencies including tests</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GDepNKxwESoJHGMQ.png" /></figure><h3>Column level dependency lineage, transformation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*k8aNfdOsUcfA86j2.png" /></figure><h3>Dependency lineage</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SZMmrxRMrhpj5cX7.png" /></figure><p>See the <a href="https://medium.com/@ismail-simsek/make-dbt-great-again-ec34f3b661f5">part 1 post here</a></p><p>If you’re struggling with <strong>data pipelines</strong>, <strong>data analytics</strong>, or need expert guidance in the wider <strong>data &amp; analytics</strong> space, we’re here to help. Please feel free to contact office@hi1.at</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=758b02d9bcdb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unleash Your Language Learning Potential: AI-Powered Classic Storybooks]]></title>
            <link>https://medium.com/@ismail-simsek/unleash-your-language-learning-potential-ai-powered-classic-storybooks-be1e0b0b848e?source=rss-2215c1d16895------2</link>
            <guid isPermaLink="false">https://medium.com/p/be1e0b0b848e</guid>
            <category><![CDATA[generative-ai-solution]]></category>
            <category><![CDATA[llm-agent]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[agents]]></category>
            <dc:creator><![CDATA[Ismail Simsek]]></dc:creator>
            <pubDate>Tue, 08 Apr 2025 05:11:38 GMT</pubDate>
            <atom:updated>2025-04-08T13:06:57.711Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>The Challenge of Language Learning: Finding the Right Material</strong></h3><p>Learning a new language is a journey, and like any journey, it requires the right tools and resources. One of the most effective methods is reading stories — reading short tales tailored to your level. However, finding these stories can be a frustrating experience. Traditional resources are often limited, expensive, or simply don’t match your proficiency.</p><p>Imagine wanting to explore the richness of classic literature in a new language. You want to read “Pride and Prejudice” or “Les Misérables,” but the original texts are too difficult.</p><h3><strong>Enter AI: Revolutionizing Language Learning</strong></h3><p>Today, advancements in AI, particularly with Large Language Models (LLMs), are opening up unique opportunities for language learners. We can now leverage AI to create personalized learning experiences, making language acquisition more accessible and engaging.</p><p>This is where my project, “<a href="https://github.com/ismailsimsek/aistorybooks">AI Story Books</a>,” comes in. I’ve developed a free, user-friendly tool that transforms classic novels into simplified, engaging storybooks, customized to your specific language learning needs.</p><p>App available and you could use it on <a href="https://huggingface.co/spaces/ismailsimsek/aistorybooks">Hugging Face Spaces</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ou4ikidSut2kqRVXffOxUg.gif" /></figure><h3><strong>AI Story Books: Your Personalized Language Learning Companion</strong></h3><p>This project explores the use of AI agents and various agentic frameworks to generate effective learning materials. It’s designed to assist language learners by creating storybooks tailored to their proficiency level.</p><p>The primary focus is to assist language learners by generating storybooks tailored to their specific needs. For example, users can transform classic novels into storybooks adapted for language learning, customized to their desired proficiency level.</p><p>Here’s how it works:</p><ul><li><strong>Classic Novel Transformation:</strong> You select a classic novel (available in the public domain) and upload it as a PDF.</li><li><strong>Customize:</strong> You specify your target language, desired reading level (e.g., A1, B2), summary length, and even the writing style (e.g., narrative, philosophical).</li><li><strong>AI-Powered Simplification:</strong> The tool processes the text in chunks, intelligently simplifying the language and adapting the story to your chosen parameters.</li><li><strong>Markdown Output:</strong> The final product is delivered as a clean, easily accessible Markdown file, ready for reading and study.</li></ul><h3><strong>Looking Ahead</strong></h3><p>The possibilities of AI in language learning are enormous. As LLMs continue to evolve, we can expect even more sophisticated tools that optimizes to individual learning styles and needs.</p><p>Try the app on <a href="https://huggingface.co/spaces/ismailsimsek/aistorybooks">Hugginng Face Spaces</a></p><p><em>If you are interested in to leverage AI for your business, improve your data and analytics setup or strategy? Feel free to contact me on </em><a href="https://www.linkedin.com/in/ismailsimsek/"><em>linkedin</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=be1e0b0b848e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Debezium CDC with Python: Real-time Data Replication]]></title>
            <link>https://medium.com/@ismail-simsek/debezium-cdc-with-python-real-time-data-replication-b248aef683f8?source=rss-2215c1d16895------2</link>
            <guid isPermaLink="false">https://medium.com/p/b248aef683f8</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[cdc]]></category>
            <category><![CDATA[real-time-analytics]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Ismail Simsek]]></dc:creator>
            <pubDate>Sat, 01 Feb 2025 00:00:16 GMT</pubDate>
            <atom:updated>2025-02-07T11:42:32.018Z</atom:updated>
            <content:encoded><![CDATA[<h3>Real-time Data Replication with Debezium and <strong>Python</strong></h3><p>When it comes to replicating operational data for analytics, Change Data Capture (CDC) is the gold standard. It offers scalability, near real-time performance, and captures all data modifications, ensuring your analytical datasets are always up-to-date. Debezium is a leading tool in this space, connecting to a wide range of databases and exporting CDC events in various formats like JSON and Avro, making integration with diverse systems a breeze.</p><p>While Debezium itself is a Java-based project, the data engineering world increasingly relies on Python. This blog post demonstrates how to leverage Debezium within a Python environment, using <a href="https://github.com/memiiso/pydbzengine">pydbzengine</a>. We’ll explore how to use these technologies to build a robust and scalable CDC solution.</p><h3>Python-Powered CDC Pipeline</h3><h3>Capturing Change Data with Debezium and Loading into DuckDB using DLT</h3><p>This blog post demonstrates how to use Debezium to capture change data from a PostgreSQL database and load it into a DuckDB database using the Data Load Tool (DLT). <strong>All of this is achieved through a Python-based data processing pipeline using </strong><a href="https://github.com/memiiso/pydbzengine"><strong>pydbzengine</strong></a>.</p><p>We’ll walk through the code, explaining the key components and how they work together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*BE2mfywyf2q3CozHsgHryQ.png" /><figcaption>CDC data consuming using Python, pydbzengine</figcaption></figure><h3>Understanding the Components</h3><ol><li><a href="https://debezium.io/"><strong>Debezium:</strong></a> A powerful open-source platform for change data capture. It monitors database transaction logs and produces change event streams, indicating inserts, updates, and deletes.</li><li><a href="https://github.com/memiiso/pydbzengine"><strong>pydbzengine:</strong></a> A Python library that provides a convenient way to interact with the Debezium embedded engine. It simplifies the process of configuring and running Debezium within a Python application.</li><li><a href="https://dlthub.com/"><strong>DLT:</strong></a> A versatile data loading tool that simplifies the process of extracting, and loading data into various destinations. In this example, we use DLT to load the change events from Debezium into DuckDB.</li><li><a href="https://duckdb.org/"><strong>DuckDB:</strong></a> An embeddable analytical database that provides efficient data processing capabilities. It’s an excellent choice for local development and testing.</li><li><a href="https://testcontainers.com/"><strong>Testcontainers:</strong></a> A library for spinning up throwaway instances of services, such as databases, for testing. We use it to manage the PostgreSQL database during the example.</li></ol><h3>Code Breakdown</h3><p>The provided code demonstrates a complete CDC pipeline in python, from capturing changes with Debezium to loading them into DuckDB with DLT. Let’s break down the key parts:</p><h4>1. Setting up the Environment</h4><p>Let’s begin with setting up the necessary environment, including defining file paths, cleaning up previous runs, and defining a helper class DbPostgresql that uses Testcontainers to manage the PostgreSQL database, which acts as our operational data source.</p><pre>import os<br>from pathlib import Path<br>import dlt<br>import duckdb<br>from testcontainers.core.config import testcontainers_config<br>from testcontainers.core.waiting_utils import wait_for_logs<br>from testcontainers.postgres import PostgresContainer<br>from pydbzengine import DebeziumJsonEngine, Properties<br>from pydbzengine.debeziumdlt import DltChangeHandler<br>from pydbzengine.helper import Utils<br><br># set global variables<br>CURRENT_DIR = Path(__file__).parent<br>DUCKDB_FILE = CURRENT_DIR.joinpath(&quot;dbz_cdc_events_example.duckdb&quot;)<br>OFFSET_FILE = CURRENT_DIR.joinpath(&#39;postgresql-offsets.dat&#39;)<br><br># cleanup<br>if OFFSET_FILE.exists():<br>    os.remove(OFFSET_FILE)<br>if DUCKDB_FILE.exists():<br>    os.remove(DUCKDB_FILE)<br><br>def wait_for_postgresql_to_start(self) -&gt; None:<br>    wait_for_logs(self, &quot;.*database system is ready to accept connections.*&quot;)<br>    wait_for_logs(self, &quot;.*PostgreSQL init process complete.*&quot;)<br><br>class DbPostgresql:<br>    POSTGRES_USER = &quot;postgres&quot;<br>    POSTGRES_PASSWORD = &quot;postgres&quot;<br>    POSTGRES_DBNAME = &quot;postgres&quot;<br>    POSTGRES_IMAGE = &quot;debezium/example-postgres:3.0.0.Final&quot;<br>    POSTGRES_HOST = &quot;localhost&quot;<br>    POSTGRES_PORT_DEFAULT = 5432<br>    CONTAINER: PostgresContainer = (PostgresContainer(image=POSTGRES_IMAGE,<br>                                                      port=POSTGRES_PORT_DEFAULT,<br>                                                      username=POSTGRES_USER,<br>                                                      password=POSTGRES_PASSWORD,<br>                                                      dbname=POSTGRES_DBNAME,<br>                                                      )<br>                                    .with_exposed_ports(POSTGRES_PORT_DEFAULT)<br>                                    )<br>    PostgresContainer._connect = wait_for_postgresql_to_start<br><br>    def start(self):<br>        testcontainers_config.ryuk_disabled = True<br>        print(&quot;Starting Postgresql Db...&quot;)<br>        self.CONTAINER.start()<br><br>    def stop(self):<br>        print(&quot;Stopping Postgresql Db...&quot;)<br>        self.CONTAINER.stop()<br><br>    def __exit__(self, exc_type, exc_value, traceback):<br>        self.stop()py</pre><h4>2. Configuring Debezium</h4><p>We configure Debezium by creating a Java Properties object. This object holds the <a href="https://debezium.io/documentation/reference/stable/connectors/postgresql.html">configuration settings</a> for the Debezium engine. It includes database connection details, the connector class (PostgresConnector), offset storage, and transformations. The transforms property is used to unwrap the Debezium message, simplifying downstream processing.</p><pre><br>def debezium_engine_props(sourcedb: DbPostgresql):<br>    props = Properties()<br>    props.setProperty(&quot;name&quot;, &quot;engine&quot;)<br>    props.setProperty(&quot;snapshot.mode&quot;, &quot;initial_only&quot;)<br>    props.setProperty(&quot;database.hostname&quot;, sourcedb.CONTAINER.get_container_host_ip())<br>    props.setProperty(&quot;database.port&quot;,<br>                      sourcedb.CONTAINER.get_exposed_port(sourcedb.POSTGRES_PORT_DEFAULT))<br>    props.setProperty(&quot;database.user&quot;, sourcedb.POSTGRES_USER)<br>    props.setProperty(&quot;database.password&quot;, sourcedb.POSTGRES_PASSWORD)<br>    props.setProperty(&quot;database.dbname&quot;, sourcedb.POSTGRES_DBNAME)<br>    props.setProperty(&quot;connector.class&quot;, &quot;io.debezium.connector.postgresql.PostgresConnector&quot;)<br>    props.setProperty(&quot;offset.storage&quot;, &quot;org.apache.kafka.connect.storage.FileOffsetBackingStore&quot;)<br>    props.setProperty(&quot;offset.storage.file.filename&quot;, OFFSET_FILE.as_posix())<br>    props.setProperty(&quot;max.batch.size&quot;, &quot;5&quot;)<br>    props.setProperty(&quot;poll.interval.ms&quot;, &quot;10000&quot;)<br>    props.setProperty(&quot;converter.schemas.enable&quot;, &quot;false&quot;)<br>    props.setProperty(&quot;offset.flush.interval.ms&quot;, &quot;1000&quot;)<br>    props.setProperty(&quot;database.server.name&quot;, &quot;testc&quot;)<br>    props.setProperty(&quot;database.server.id&quot;, &quot;1234&quot;)<br>    props.setProperty(&quot;topic.prefix&quot;, &quot;testc&quot;)<br>    props.setProperty(&quot;schema.whitelist&quot;, &quot;inventory&quot;)<br>    props.setProperty(&quot;database.whitelist&quot;, &quot;inventory&quot;)<br>    props.setProperty(&quot;table.whitelist&quot;, &quot;inventory.*&quot;)<br>    props.setProperty(&quot;replica.identity.autoset.values&quot;, &quot;inventory.*:FULL&quot;)<br>    # // debezium unwrap message<br>    props.setProperty(&quot;transforms&quot;, &quot;unwrap&quot;)<br>    props.setProperty(&quot;transforms.unwrap.type&quot;, &quot;io.debezium.transforms.ExtractNewRecordState&quot;)<br>    props.setProperty(&quot;transforms.unwrap.add.fields&quot;, &quot;op,table,source.ts_ms,sourcedb,ts_ms&quot;)<br>    props.setProperty(&quot;transforms.unwrap.delete.handling.mode&quot;, &quot;rewrite&quot;)<br>    # props.setProperty(&quot;debezium.transforms.unwrap.drop.tombstones&quot;, &quot;true&quot;)<br>    return propspy</pre><h4>3. Change Handler Implementation</h4><p>The DltChangeHandler class, provided by the pydbzengine library, acts as the bridge between Debezium and DLT. It receives change data capture events from Debezium and then leverages the DLT pipeline to efficiently load this data into your chosen destination, such as DuckDB. Essentially, it&#39;s the component that connects the real-time change data stream from Debezium to the data loading capabilities of DLT. You can find the <a href="https://github.com/memiiso/pydbzengine/blob/main/pydbzengine/debeziumdlt.py">full implementation</a> in the pydbzengine repository.</p><p><strong>The power of this system lies in its flexibility. You can define your own event consumption logic by simply implementing the </strong><strong>handleJsonBatch method. This allows you to tailor the processing to your specific needs, sending data to a variety of destinations or services.</strong></p><pre>from pydbzengine import BasePythonChangeHandler, ChangeEvent<br><br>class MyXYZChangeHandler(BasePythonChangeHandler):<br>    def handleJsonBatch(self, records: List[ChangeEvent]):<br>        # Process your data here!<br>        for record in records:<br>            # ... your processing logic ...<br>            # Example: send data to another service, database, etc.</pre><h4>4. Running the Debezium Engine and DLT Pipeline</h4><p>The main function orchestrates the entire process. It starts the PostgreSQL container, creates the Debezium engine with the configured properties and the DltChangeHandler, and starts the engine.</p><pre>def main():<br>    # Start the PostgreSQL container that will serve as the replication source.<br>    sourcedb = DbPostgresql()<br>    sourcedb.start()<br><br>    # Get Debezium engine configuration properties<br>    props = debezium_engine_props(sourcedb=sourcedb)<br><br>    # Create a dlt pipeline to consume the change events into DuckDB.<br>    dlt_pipeline = dlt.pipeline(<br>        pipeline_name=&quot;dbz_cdc_events_example&quot;,<br>        destination=&quot;duckdb&quot;,<br>        dataset_name=&quot;dbz_data&quot;<br>    )<br><br>    handler = DltChangeHandler(dlt_pipeline=dlt_pipeline)<br>    engine = DebeziumJsonEngine(properties=props, handler=handler)<br><br>    # Run the Debezium engine asynchronously with a timeout.  This allows the example<br>    # to run for a limited time and then terminate automatically.<br>    Utils.run_engine_async(engine=engine, timeout_sec=60)<br>    # engine.run()  # This would be used for synchronous execution (without timeout)<br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><h4>5. Querying the DuckDB Database, The result</h4><p>After the Debezium engine has run for a specified time (60 seconds in the example), we can connect to destination(DuckDB) database and display the loaded data.</p><pre>    con = duckdb.connect(DUCKDB_FILE.as_posix())<br>    result = con.sql(&quot;SHOW ALL TABLES&quot;).fetchall()<br>    for r in result:<br>        database, schema, table = r[:3]  # Extract database, schema, and table names.<br>        if schema == &quot;dbz_data&quot;:  # Only show data from the schema where Debezium loaded the data.<br>            print(f&quot;Data in table {table}:&quot;)<br>            con.sql(f&quot;select * from {database}.{schema}.{table} limit 5&quot;).show() # Display table data</pre><pre><br>┌────────────────────┬────────────────────────┬────────┬───────────────────────────────┬──────────────────────────────────────────────┐<br>│      load_id       │      schema_name       │ status │          inserted_at          │             schema_version_hash              │<br>│      varchar       │        varchar         │ int64  │   timestamp with time zone    │                   varchar                    │<br>├────────────────────┼────────────────────────┼────────┼───────────────────────────────┼──────────────────────────────────────────────┤<br>│ 1738405897.413279  │ debezium_source_events │      0 │ 2025-02-01 11:31:38.086127+01 │ Q5UNIOd7gJ6ljH5qfKKcO7yWwPvNESKW+mVXJmx9geg= │<br>│ 1738405898.176148  │ debezium_source_events │      0 │ 2025-02-01 11:31:39.381275+01 │ OyUXGP6PvFQuUTPnPdvESnsEqpFAxivJoP+l0G6l4+M= │<br>│ 1738405899.4865642 │ debezium_source_events │      0 │ 2025-02-01 11:31:39.704015+01 │ jqZNcnJXF/33Va2kRWgKOZF4RnZSVgYxMDhFep8+Jg8= │<br>│ 1738405899.775917  │ debezium_source_events │      0 │ 2025-02-01 11:31:39.952311+01 │ jqZNcnJXF/33Va2kRWgKOZF4RnZSVgYxMDhFep8+Jg8= │<br>│ 1738405900.0213661 │ debezium_source_events │      0 │ 2025-02-01 11:31:40.223125+01 │ uMZY5n2NGPecXvVQIePLEg2nZQcAlkoWAXDLALKjWuQ= │<br>└────────────────────┴────────────────────────┴────────┴───────────────────────────────┴──────────────────────────────────────────────┘<br><br>Data in table _dlt_pipeline_state:<br>┌─────────┬────────────────┬──────────────────────┬──────────────────────┬──────────────────────┬────────────────────────────────────────────┬───────────────────┬────────────────┐<br>│ version │ engine_version │    pipeline_name     │        state         │      created_at      │                version_hash                │   _dlt_load_id    │    _dlt_id     │<br>│  int64  │     int64      │       varchar        │       varchar        │ timestamp with tim…  │                  varchar                   │      varchar      │    varchar     │<br>├─────────┼────────────────┼──────────────────────┼──────────────────────┼──────────────────────┼────────────────────────────────────────────┼───────────────────┼────────────────┤<br>│       1 │              4 │ dbz_cdc_events_exa…  │ eNp1j0FLw0AQhf/LXg…  │ 2025-02-01 11:31:3…  │ ZvlGi9hyfXjD2b0imkL9ZA7x3S1/YkmQK4QbA+Jw…  │ 1738405897.413279 │ hNbs3TIc3vRHvA │<br>└─────────┴────────────────┴──────────────────────┴──────────────────────┴──────────────────────┴────────────────────────────────────────────┴───────────────────┴────────────────┘<br><br>Data in table _dlt_version:<br>┌─────────┬────────────────┬──────────────────────┬──────────────────────┬──────────────────────┬─────────────────────────────────────────────────────────────────────────────────┐<br>│ version │ engine_version │     inserted_at      │     schema_name      │     version_hash     │                                     schema                                      │<br>│  int64  │     int64      │ timestamp with tim…  │       varchar        │       varchar        │                                     varchar                                     │<br>├─────────┼────────────────┼──────────────────────┼──────────────────────┼──────────────────────┼─────────────────────────────────────────────────────────────────────────────────┤<br>│       2 │             11 │ 2025-02-01 11:31:3…  │ debezium_source_ev…  │ Q5UNIOd7gJ6ljH5qfK…  │ {&quot;version&quot;:2,&quot;version_hash&quot;:&quot;Q5UNIOd7gJ6ljH5qfKKcO7yWwPvNESKW+mVXJmx9geg=&quot;,&quot;e…  │<br>│       4 │             11 │ 2025-02-01 11:31:3…  │ debezium_source_ev…  │ OyUXGP6PvFQuUTPnPd…  │ {&quot;version&quot;:4,&quot;version_hash&quot;:&quot;OyUXGP6PvFQuUTPnPdvESnsEqpFAxivJoP+l0G6l4+M=&quot;,&quot;e…  │<br>│       6 │             11 │ 2025-02-01 11:31:3…  │ debezium_source_ev…  │ jqZNcnJXF/33Va2kRW…  │ {&quot;version&quot;:6,&quot;version_hash&quot;:&quot;jqZNcnJXF/33Va2kRWgKOZF4RnZSVgYxMDhFep8+Jg8=&quot;,&quot;e…  │<br>│       8 │             11 │ 2025-02-01 11:31:4…  │ debezium_source_ev…  │ uMZY5n2NGPecXvVQIe…  │ {&quot;version&quot;:8,&quot;version_hash&quot;:&quot;uMZY5n2NGPecXvVQIePLEg2nZQcAlkoWAXDLALKjWuQ=&quot;,&quot;e…  │<br>└─────────┴────────────────┴──────────────────────┴──────────────────────┴──────────────────────┴─────────────────────────────────────────────────────────────────────────────────┘<br><br>Data in table testc_inventory_customers:<br>┌───────┬────────────┬───────────┬───────────────────────┬─────────┬─────────┬───────────┬───────────────┬───────────────┬───────────────────┬────────────────┐<br>│  id   │ first_name │ last_name │         email         │ deleted │   op    │   table   │ source_ts_ms  │     ts_ms     │   _dlt_load_id    │    _dlt_id     │<br>│ int64 │  varchar   │  varchar  │        varchar        │ varchar │ varchar │  varchar  │     int64     │     int64     │      varchar      │    varchar     │<br>├───────┼────────────┼───────────┼───────────────────────┼─────────┼─────────┼───────────┼───────────────┼───────────────┼───────────────────┼────────────────┤<br>│  1001 │ Sally      │ Thomas    │ sally.thomas@acme.com │ false   │ r       │ customers │ 1738405883186 │ 1738405896858 │ 1738405897.413279 │ KcWKrYODYJ859w │<br>│  1002 │ George     │ Bailey    │ gbailey@foobar.com    │ false   │ r       │ customers │ 1738405883186 │ 1738405896862 │ 1738405897.413279 │ JU6dR1S27Xt3QA │<br>│  1003 │ Edward     │ Walker    │ ed@walker.com         │ false   │ r       │ customers │ 1738405883186 │ 1738405896862 │ 1738405897.413279 │ 02kMVvIX2/aGGg │<br>│  1004 │ Anne       │ Kretchmar │ annek@noanswer.org    │ false   │ r       │ customers │ 1738405883186 │ 1738405896862 │ 1738405897.413279 │ TI7jpxl9FD2kRQ │<br>└───────┴────────────┴───────────┴───────────────────────┴─────────┴─────────┴───────────┴───────────────┴───────────────┴───────────────────┴────────────────┘<br><br>Data in table testc_inventory_geom:<br>┌───────┬──────────────────────────────────────────────────────────────────────┬─────────┬─────────┬─────────┬───────────────┬───────────────┬───────────────────┬────────────────┐<br>│  id   │                                g__wkb                                │ deleted │   op    │  table  │ source_ts_ms  │     ts_ms     │   _dlt_load_id    │    _dlt_id     │<br>│ int64 │                               varchar                                │ varchar │ varchar │ varchar │     int64     │     int64     │      varchar      │    varchar     │<br>├───────┼──────────────────────────────────────────────────────────────────────┼─────────┼─────────┼─────────┼───────────────┼───────────────┼───────────────────┼────────────────┤<br>│     1 │ AQEAAAAAAAAAAADwPwAAAAAAAPA/                                         │ false   │ r       │ geom    │ 1738405883186 │ 1738405896872 │ 1738405897.413279 │ 17snqevSVWL0xA │<br>│     2 │ AQIAAAACAAAAAAAAAAAAAEAAAAAAAADwPwAAAAAAABhAAAAAAAAAGEA=             │ false   │ r       │ geom    │ 1738405883186 │ 1738405896872 │ 1738405898.176148 │ W4kfG5n5jYhy3w │<br>│     3 │ AQMAAAABAAAABQAAAAAAAAAAAAAAAAAAAAAAFEAAAAAAAAAAQAAAAAAAABRAAAAAAA…  │ false   │ r       │ geom    │ 1738405883186 │ 1738405896872 │ 1738405898.176148 │ 40HrbnruXZaB/g │<br>└───────┴──────────────────────────────────────────────────────────────────────┴─────────┴─────────┴─────────┴───────────────┴───────────────┴───────────────────┴────────────────┘<br><br>Data in table testc_inventory_orders:<br>┌───────┬────────────┬───────────┬──────────┬────────────┬─────────┬─────────┬─────────┬───────────────┬───────────────┬────────────────────┬────────────────┐<br>│  id   │ order_date │ purchaser │ quantity │ product_id │ deleted │   op    │  table  │ source_ts_ms  │     ts_ms     │    _dlt_load_id    │    _dlt_id     │<br>│ int64 │   int64    │   int64   │  int64   │   int64    │ varchar │ varchar │ varchar │     int64     │     int64     │      varchar       │    varchar     │<br>├───────┼────────────┼───────────┼──────────┼────────────┼─────────┼─────────┼─────────┼───────────────┼───────────────┼────────────────────┼────────────────┤<br>│ 10001 │      16816 │      1001 │        1 │        102 │ false   │ r       │ orders  │ 1738405883186 │ 1738405896876 │ 1738405898.176148  │ X7ejebZDxmm+hw │<br>│ 10002 │      16817 │      1002 │        2 │        105 │ false   │ r       │ orders  │ 1738405883186 │ 1738405896876 │ 1738405898.176148  │ 6LU0Fe9UVE3XFQ │<br>│ 10003 │      16850 │      1002 │        2 │        106 │ false   │ r       │ orders  │ 1738405883186 │ 1738405896876 │ 1738405898.176148  │ 0OIBPdMzqjLh0w │<br>│ 10004 │      16852 │      1003 │        1 │        107 │ false   │ r       │ orders  │ 1738405883186 │ 1738405896876 │ 1738405899.4865642 │ CcY6FKlHLQ6mPg │<br>└───────┴────────────┴───────────┴──────────┴────────────┴─────────┴─────────┴─────────┴───────────────┴───────────────┴────────────────────┴────────────────┘<br><br>Data in table testc_inventory_products:<br>┌───────┬────────────────────┬──────────────────────────────────────┬────────┬─────────┬─────────┬──────────┬───────────────┬───────────────┬────────────────────┬────────────────┐<br>│  id   │        name        │             description              │ weight │ deleted │   op    │  table   │ source_ts_ms  │     ts_ms     │    _dlt_load_id    │    _dlt_id     │<br>│ int64 │      varchar       │               varchar                │ double │ varchar │ varchar │ varchar  │     int64     │     int64     │      varchar       │    varchar     │<br>├───────┼────────────────────┼──────────────────────────────────────┼────────┼─────────┼─────────┼──────────┼───────────────┼───────────────┼────────────────────┼────────────────┤<br>│   101 │ scooter            │ Small 2-wheel scooter                │   3.14 │ false   │ r       │ products │ 1738405883186 │ 1738405896879 │ 1738405899.4865642 │ aOf6efrtt48+1Q │<br>│   102 │ car battery        │ 12V car battery                      │    8.1 │ false   │ r       │ products │ 1738405883186 │ 1738405896880 │ 1738405899.4865642 │ kUuPhtKUAsTUaA │<br>│   103 │ 12-pack drill bits │ 12-pack of drill bits with sizes r…  │    0.8 │ false   │ r       │ products │ 1738405883186 │ 1738405896880 │ 1738405899.4865642 │ evSpPy68nldtbg │<br>│   104 │ hammer             │ 12oz carpenter&#39;s hammer              │   0.75 │ false   │ r       │ products │ 1738405883186 │ 1738405896880 │ 1738405899.4865642 │ lCpa9yyHSm8xqA │<br>│   105 │ hammer             │ 14oz carpenter&#39;s hammer              │  0.875 │ false   │ r       │ products │ 1738405883186 │ 1738405896880 │ 1738405899.775917  │ VXcDU/tw/zT2fw │<br>└───────┴────────────────────┴──────────────────────────────────────┴────────┴─────────┴─────────┴──────────┴───────────────┴───────────────┴────────────────────┴────────────────┘<br><br>Data in table testc_inventory_products_on_hand:<br>┌────────────┬──────────┬─────────┬─────────┬──────────────────┬───────────────┬───────────────┬────────────────────┬────────────────┐<br>│ product_id │ quantity │ deleted │   op    │      table       │ source_ts_ms  │     ts_ms     │    _dlt_load_id    │    _dlt_id     │<br>│   int64    │  int64   │ varchar │ varchar │     varchar      │     int64     │     int64     │      varchar       │    varchar     │<br>├────────────┼──────────┼─────────┼─────────┼──────────────────┼───────────────┼───────────────┼────────────────────┼────────────────┤<br>│        101 │        3 │ false   │ r       │ products_on_hand │ 1738405883186 │ 1738405896883 │ 1738405900.0213661 │ 90r3+XR7PH7y6g │<br>│        102 │        8 │ false   │ r       │ products_on_hand │ 1738405883186 │ 1738405896883 │ 1738405900.0213661 │ 5F+LUMVYO3I2wQ │<br>│        103 │       18 │ false   │ r       │ products_on_hand │ 1738405883186 │ 1738405896883 │ 1738405900.0213661 │ SguX65iX7ffyJg │<br>│        104 │        4 │ false   │ r       │ products_on_hand │ 1738405883186 │ 1738405896883 │ 1738405900.0213661 │ Vj/N2j0bN3ipzw │<br>│        105 │        5 │ false   │ r       │ products_on_hand │ 1738405883186 │ 1738405896883 │ 1738405900.0213661 │ z31M4RIQPpq3BA │<br>└────────────┴──────────┴─────────┴─────────┴──────────────────┴───────────────┴───────────────┴────────────────────┴────────────────┘</pre><h4>Testing it yourself</h4><p>You can see complete example and start experimenting using <a href="https://github.com/memiiso/pydbzengine/blob/main/pydbzengine/examples/dlt_consuming.py">dlt_consuming</a>.</p><p>To run this example, you’ll need to have Docker Desktop installed and the required Python libraries. You can install the dependencies using:</p><pre>pip install pydbzengine[dev] <br>python dlt_consuming.py</pre><h3>Key Takeaways</h3><p>This example demonstrates a simple way to capture change data from a database and load it into a data warehouse using Debezium and DLT.</p><p>The use of Python (<a href="https://github.com/memiiso/pydbzengine">pydbzengine</a>) makes it easy to integrate Debezium into existing Python workflows.</p><p>If you enjoyed this article, you may also be interested in “<a href="https://medium.com/@ismail-simsek/building-a-data-lake-with-debezium-and-apache-iceberg-part-1-25124daf2a95">Building a Data Lake with Debezium and Apache Iceberg</a>”</p><h3>Wrap-Up and Contributions</h3><p>Based on <a href="https://debezium.io/">Debezium</a>, <a href="https://github.com/memiiso/pydbzengine">pydbzengine</a> makes it very simple to set up a low-latency data ingestion pipeline using python. The project completely open-source, using the Apache 2.0 license. <a href="https://github.com/memiiso/pydbzengine">pydbzengine</a> still is a young project and there are things to improve. Please feel free to test it, give feedback, open feature requests or send pull requests.</p><p>Happy data engineering…​</p><p><em>Originally published at </em><a href="https://debezium.io/blog/2025/02/01/real-time-data-replication-with-debezium-and-python/"><em>https://debezium.io</em></a><em> on February 1, 2025.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b248aef683f8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[opendbt: Effortlessly Streamline Your Data Pipelines with dbt an dlt, Reduce complexity]]></title>
            <link>https://medium.com/@ismail-simsek/opendbt-effortlessly-streamline-your-data-pipelines-with-dbt-an-dlt-reduce-complexity-1ef065b03d5b?source=rss-2215c1d16895------2</link>
            <guid isPermaLink="false">https://medium.com/p/1ef065b03d5b</guid>
            <category><![CDATA[transformation]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[dlt]]></category>
            <dc:creator><![CDATA[Ismail Simsek]]></dc:creator>
            <pubDate>Tue, 10 Dec 2024 13:02:44 GMT</pubDate>
            <atom:updated>2024-12-10T13:02:44.188Z</atom:updated>
            <content:encoded><![CDATA[<h3>Streamline Your Data Pipelines with opendbt: Leverage dbt and dlt for Effortless End-to-End ELT Workflows</h3><p>In this post, we’ll delve into how <a href="https://github.com/memiiso/opendbt">opendbt</a> leverages the power of <a href="https://dlthub.com/">dlt</a> and dbt-core to deliver end-to-end ELT data flow. Yes! including Extract and Load steps too. This innovative approach streamlines data pipelines, making data engineering more accessible and efficient.</p><p>For more details on opendbt, check out our <a href="https://medium.com/@ismail-simsek/make-dbt-great-again-ec34f3b661f5">previous blog post</a>.</p><h3>dlt: A Lightweight Python Framework for Efficient Data Ingestion</h3><p><a href="https://dlthub.com/">dlt</a> is a lightweight Python framework designed to simplify data ingestion pipelines. It offers a flexible and efficient way to extract and load data from various sources, including databases, APIs, and cloud storage, directly into data warehouses and data lakes.</p><p>Key Features:</p><ul><li><strong>Out-of-the-Box Connectors</strong>: Easily integrate with popular data sources and destinations.</li><li><strong>Python-Native:</strong> Seamlessly integrates with Python environments, especially in Airflow+dbt-core setups.</li><li><strong>Simplified ELT:</strong> Streamline the Extract and Load steps of your ELT process.</li></ul><p>By leveraging dlt, you can accelerate your data pipelines, reduce development time and internal python code.</p><h3>Integrating dlt and dbt-core: A Powerful Combination for Modern Data Pipelines</h3><p><strong>Leveraging the Best of Both Worlds</strong></p><p>dlt and dbt-core are two powerful tools that, when combined, can create robust and efficient data pipelines. While dlt excels at data ingestion (Extract, Load), dbt-core is a specialized tool for data Transformation and modeling.</p><p>By integrating these two frameworks, you can streamline your data ingestion and transformation processes.</p><h3><strong>Step by Step Implementation with opendbt</strong></h3><p>Below we are creating a new class that inherits from <em>DuckDBAdapter</em> and adding a custom method to it to execute the compiled Python model.</p><p>Before executing the model, a dlt pipeline defined with the appropriate destination connection and dataset name, based on the dbt model’s specifications.</p><p>The model method is then executed, passing dbt and dlt pipeline arguments.</p><pre>class DuckDBAdapterV2Custom(DuckDBAdapter):<br>    @available<br>    def submit_local_dlt_job(self, parsed_model: Dict, compiled_code: str):<br># Get connection from dbt<br>        connection = self.connections.get_if_exists()<br>        if not connection:<br>            connection = self.connections.get_thread_connection()<br><br>        import dlt<br># IMPORTANT: here we are pre-configuring and preparing dlt.pipeline <br># this pipeline then passed to the python model!<br>        _pipeline = dlt.pipeline(<br>            pipeline_name=str(parsed_model[&#39;unique_id&#39;]).replace(&quot;.&quot;, &quot;-&quot;),<br>            destination=dlt.destinations.duckdb(connection.handle._env.conn),<br>            dataset_name=parsed_model[&#39;schema&#39;],<br>            dev_mode=False,<br>        )<br># Save compiled model code the import it to pyhon and execute it<br>        with tempfile.NamedTemporaryFile(suffix=f&#39;.py&#39;, delete=True) as model_file:<br>            model_file.write(compiled_code.lstrip().encode(&#39;utf-8&#39;))<br>            model_file.flush()<br>            print(f&quot;Created temp py file {model_file.name}&quot;)<br># load and execute the compiled code!<br>            model_name = parsed_model[&#39;name&#39;]<br>            spec = importlib.util.spec_from_file_location(model_name, model_file.name)<br>            module = importlib.util.module_from_spec(spec)<br>            sys.modules[model_name] = module<br>            spec.loader.exec_module(module)<br>            dbt = module.dbtObj(None)<br># IMPORTANT: passing down duckdb session from the adapter to the model<br># IMPORTANT: passing down pre configured pipeline down to the model<br>            module.model(dbt=dbt, pipeline=_pipeline)<br>            model_file.close()</pre><p>Register the custom adapter with dbt by adding the following code to dbt_project.yml</p><pre>vars:<br>  dbt_custom_adapter: opendbt.examples.DuckDBAdapterV2Custom</pre><p>We create a custom materialization named executedlt that calls the custom adapter method we defined earlier.</p><pre>{% materialization executedlt, supported_languages=[&#39;python&#39;]%}<br>  {%- set identifier = model[&#39;alias&#39;] -%}<br>  {%- set target_relation = api.Relation.create(identifier=identifier, schema=schema, database=database, type=&#39;table&#39;) -%}<br>  <br>  {% call noop_statement(name=&#39;main&#39;, message=&#39;Executed DLT pipeline&#39;, code=compiled_code, rows_affected=-1, res=None) %}<br>      {# IMPORTANT! here we call the dapter method we defined above #}<br>      {%- set res = adapter.submit_local_dlt_job(model, compiled_code) -%}<br>  {% endcall %}<br>  <br>  {{ return({&#39;relations&#39;: [target_relation]}) }}<br>{% endmaterialization %}</pre><p>Now, let’s build our DLT model in dbt. lets name it <em>my_dlt_model.py</em></p><pre>import dlt<br>from dlt.pipeline import TPipeline<br><br><br>@dlt.resource(<br>    columns={&quot;event_tstamp&quot;: {&quot;data_type&quot;: &quot;timestamp&quot;, &quot;precision&quot;: 3}},<br>    primary_key=&quot;event_id&quot;,<br>)<br>def events():<br>    yield [{&quot;event_id&quot;: 1, &quot;event_tstamp&quot;: &quot;2024-07-30T10:00:00.123&quot;},<br>           {&quot;event_id&quot;: 2, &quot;event_tstamp&quot;: &quot;2025-02-30T10:00:00.321&quot;}]<br><br><br>def model(dbt, pipeline: TPipeline):<br>    &quot;&quot;&quot; NOTE: Pre-configured dlt pipeline is provided by the dapter. <br>    dlt target connection and dataset is pre-set using the model config!<br>    &quot;&quot;&quot;<br>    dbt.config(materialized=&quot;executedlt&quot;)<br>    print(&quot;========================================================&quot;)<br>    print(f&quot;INFO: DLT Pipeline pipeline_name:{pipeline.pipeline_name}&quot;)<br>    print(f&quot;INFO: DLT Pipeline dataset_name:{pipeline.dataset_name}&quot;)<br>    print(f&quot;INFO: DLT Pipeline staging:{pipeline.staging}&quot;)<br>    print(f&quot;INFO: DLT Pipeline destination:{pipeline.destination}&quot;)<br>    print(f&quot;INFO: DLT Pipeline _pipeline_storage:{pipeline._pipeline_storage}&quot;)<br>    print(f&quot;INFO: DLT Pipeline _schema_storage:{pipeline._schema_storage}&quot;)<br>    print(f&quot;INFO: DLT Pipeline state:{pipeline.state}&quot;)<br>    print(&quot;========================================================&quot;)<br>    load_info = pipeline.run(events())<br>    print(load_info)<br>    row_counts = pipeline.last_trace.last_normalize_info<br>    print(row_counts)<br>    print(&quot;========================================================&quot;)<br>    return None</pre><p>Run the code</p><pre>dp = OpenDbtProject(project_dir=&#39;/patg/to/dbtproject&#39;, <br>                    profiles_dir=&#39;/patg/to/dbtproject&#39;,<br># Following argument is optional <br># since we configured `dbt_custom_adapter` globally in dbt_project.yml<br>                    args=[&#39;--vars&#39;, &#39;dbt_custom_adapter: opendbt.examples.DuckDBAdapterV2Custom&#39;]<br>)<br>dp.run(command=&quot;run&quot;, args=[&#39;--select&#39;, &#39;my_dlt_model&#39;])</pre><p>The dbt execution logs will appear as follows:</p><pre>16:12:59  Running with dbt=1.7.9<br>16:12:59  Registered adapter: duckdb=1.7.5<br>16:13:00  Found 6 models, 0 sources, 0 exposures, 0 metrics, 409 macros, 0 groups, 0 semantic models<br>16:13:00  <br>16:13:00  Concurrency: 1 threads (target=&#39;dev&#39;)<br>16:13:00  <br>16:13:00  1 of 1 START python executedlt model main.my_executedlt_model.py ............... [RUN]<br>Created temp py file /var/folders/3l/n5dbz15s68592fk76c31hth8ffmnng/T/tmp2knawpam.py<br>========================================================<br>INFO: DLT Pipeline pipeline_name:model-dbttest-my_executedlt_model-py<br>INFO: DLT Pipeline dataset_name:main<br>INFO: DLT Pipeline staging:None<br>INFO: DLT Pipeline destination:&lt;dlt.destinations.impl.duckdb.factory.duckdb object at 0x111a72d90&gt;<br>INFO: DLT Pipeline _pipeline_storage:&lt;dlt.common.storages.file_storage.FileStorage object at 0x1274a4e80&gt;<br>INFO: DLT Pipeline _schema_storage:&lt;dlt.common.storages.live_schema_storage.LiveSchemaStorage object at 0x1274a47c0&gt;<br>INFO: DLT Pipeline state:{&#39;_state_version&#39;: 1, &#39;_state_engine_version&#39;: 4, &#39;_local&#39;: {&#39;first_run&#39;: False, &#39;_last_extracted_at&#39;: DateTime(2024, 12, 7, 16, 32, 58, 793193, tzinfo=datetime.timezone.utc), &#39;_last_extracted_hash&#39;: &#39;36SCJ7CeiMZLzn33uLQzfjnfTcmYk6hhGHXQ6qO9i0A=&#39;}, &#39;schema_names&#39;: [&#39;model_dbttest_my_executedlt_model_py&#39;], &#39;pipeline_name&#39;: &#39;model-dbttest-my_executedlt_model-py&#39;, &#39;dataset_name&#39;: &#39;main&#39;, &#39;default_schema_name&#39;: &#39;model_dbttest_my_executedlt_model_py&#39;, &#39;destination_type&#39;: &#39;dlt.destinations.duckdb&#39;, &#39;destination_name&#39;: &#39;duckdb&#39;, &#39;_version_hash&#39;: &#39;36SCJ7CeiMZLzn33uLQzfjnfTcmYk6hhGHXQ6qO9i0A=&#39;}<br>========================================================<br>Pipeline model-dbttest-my_executedlt_model-py load step completed in 1.06 seconds<br>1 load package(s) were loaded to destination duckdb and into dataset main<br>The duckdb destination used duckdb:///:external: location to store data<br>Load package 1733760789.025732 is LOADED and contains no failed jobs<br>Normalized data for the following tables:<br>- events: 2 row(s)<br><br>Load package 1733760789.025732 is NORMALIZED and NOT YET LOADED to the destination and contains no failed jobs<br>========================================================<br>16:13:10  1 of 1 OK created python executedlt model main.my_executedlt_model.py .......... [Executed DLT pipeline in 10.78s]<br>16:13:10  1 of 1 OK created {&#39;_message&#39;: &#39;Executed DLT pipeline&#39;, &#39;code&#39;: &#39;import dlt\nfrom dlt.pipeline import TPipeline\n\n\n@dlt.resource(\n    columns={&quot;event_tstamp&quot;: {&quot;data_type&quot;: &quot;timestamp&quot;, &quot;precision&quot;: 3}},\n    primary_key=&quot;event_id&quot;,\n)\ndef events():\n    yield [{&quot;event_id&quot;: 1, &quot;event_tstamp&quot;: &quot;2024-07-30T10:00:00.123&quot;},\n           {&quot;event_id&quot;: 2, &quot;event_tstamp&quot;: &quot;2025-02-30T10:00:00.321&quot;}]\n\n\ndef model(dbt, pipeline: TPipeline):\n    &quot;&quot;&quot;\n\n    :param dbt:\n    :param pipeline: Pre-configured dlt pipeline. dlt target connection and dataset is pre-set using the model config!\n    :return:\n    &quot;&quot;&quot;\n    dbt.config(materialized=&quot;executedlt&quot;)\n    print(&quot;========================================================&quot;)\n    print(f&quot;INFO: DLT Pipeline pipeline_name:{pipeline.pipeline_name}&quot;)\n    print(f&quot;INFO: DLT Pipeline dataset_name:{pipeline.dataset_name}&quot;)\n    print(f&quot;INFO: DLT Pipeline staging:{pipeline.staging}&quot;)\n    print(f&quot;INFO: DLT Pipeline destination:{pipeline.destination}&quot;)\n    print(f&quot;INFO: DLT Pipeline _pipeline_storage:{pipeline._pipeline_storage}&quot;)\n    print(f&quot;INFO: DLT Pipeline _schema_storage:{pipeline._schema_storage}&quot;)\n    print(f&quot;INFO: DLT Pipeline state:{pipeline.state}&quot;)\n    print(&quot;========================================================&quot;)\n    load_info = pipeline.run(events())\n    print(load_info)\n    row_counts = pipeline.last_trace.last_normalize_info\n    print(row_counts)\n    print(&quot;========================================================&quot;)\n    return None\n\n\n# This part is user provided model code\n# you will need to copy the next section to run the code\n# COMMAND ----------\n# this part is dbt logic for get ref work, do not modify\n\ndef ref(*args, **kwargs):\n    refs = {}\n    key = \&#39;.\&#39;.join(args)\n    version = kwargs.get(&quot;v&quot;) or kwargs.get(&quot;version&quot;)\n    if version:\n        key += f&quot;.v{version}&quot;\n    dbt_load_df_function = kwargs.get(&quot;dbt_load_df_function&quot;)\n    return dbt_load_df_function(refs[key])\n\n\ndef source(*args, dbt_load_df_function):\n    sources = {}\n    key = \&#39;.\&#39;.join(args)\n    return dbt_load_df_function(sources[key])\n\n\nconfig_dict = {}\n\n\nclass config:\n    def __init__(self, *args, **kwargs):\n        pass\n\n    @staticmethod\n    def get(key, default=None):\n        return config_dict.get(key, default)\n\nclass this:\n    &quot;&quot;&quot;dbt.this() or dbt.this.identifier&quot;&quot;&quot;\n    database = &quot;dev&quot;\n    schema = &quot;main&quot;\n    identifier = &quot;my_executedlt_model.py&quot;\n    \n    def __repr__(self):\n        return \&#39;&quot;dev&quot;.&quot;main&quot;.&quot;my_executedlt_model.py&quot;\&#39;\n\n\nclass dbtObj:\n    def __init__(self, load_df_function) -&gt; None:\n        self.source = lambda *args: source(*args, dbt_load_df_function=load_df_function)\n        self.ref = lambda *args, **kwargs: ref(*args, **kwargs, dbt_load_df_function=load_df_function)\n        self.config = config\n        self.this = this()\n        self.is_incremental = False\n\n# COMMAND ----------\n\n\n&#39;, &#39;rows_affected&#39;: -1}  [Executed DLT pipeline in 10.78s]<br>16:13:10  <br>16:13:10  Finished running 1 executedlt model in 0 hours 0 minutes and 10.88 seconds (10.88s).<br>16:13:11  <br>16:13:11  Completed successfully<br>16:13:11  <br>16:13:11  Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1</pre><p>With this setup, dbt can execute dlt pipelines and import external data. This enables you to develop end-to-end ETL pipelines entirely within the dbt framework.</p><p><a href="https://github.com/memiiso/opendbt">opendbt</a> empowers you to streamline ETL processes with minimal complexity and a straightforward toolset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KjoWcTatHlEd8Tjv.png" /><figcaption>an example airflow workflow with opendbt</figcaption></figure><h3>Contributions</h3><p>The project completely open-source, using the Apache 2.0 license. <a href="https://github.com/memiiso/opendbt">opendbt</a> still is a young project and there are things to improve. Please feel free to test it, give feedback or contribute.</p><p>Please share it if you like it to reach more people.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1ef065b03d5b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Data Lake with Debezium and Apache Iceberg: Part 3]]></title>
            <link>https://medium.com/@ismail-simsek/building-a-data-lake-with-debezium-and-apache-iceberg-part-3-86042974be15?source=rss-2215c1d16895------2</link>
            <guid isPermaLink="false">https://medium.com/p/86042974be15</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[snowflake]]></category>
            <dc:creator><![CDATA[Ismail Simsek]]></dc:creator>
            <pubDate>Fri, 15 Nov 2024 11:27:01 GMT</pubDate>
            <atom:updated>2024-11-15T11:28:35.317Z</atom:updated>
            <content:encoded><![CDATA[<p>In <a href="https://medium.com/@ismail-simsek/building-a-data-lake-with-debezium-and-apache-iceberg-part-1-25124daf2a95">Part 1</a> And <a href="https://medium.com/@ismail-simsek/building-a-data-lake-with-debezium-and-apache-iceberg-part-2-bd13ddb19bbf">Part 2</a> we have seen how to use Debezium Server Iceberg application to create scalable data lake data feed. In this post we will dive into few key features of the application.</p><h3>Upsert Mode</h3><p>By default, the Iceberg consumer is running in upsert mode (debezium.sink.iceberg.upsert set to true). This means that when a row is updated in the source table, the destination is row replaced with the new updated version. And when a row is deleted from the source, it also is deleted from the destination. When using upsert mode, data at the destination is kept identical to the source data. The upsert mode uses the Iceberg equality delete feature and creates delete files using the key of the Debezium change data events (derived from the primary key of the source table). To avoid duplicate data, deduplication is done on each batch and only the last version of the record kept. For example in a single batch of events, the same record could appear twice: once when it is inserted, and another time when it gets updated. With upsert mode, always the last extracted version of the record is stored in Iceberg.</p><p>Note that when a source table doesn’t define a primary key and there is also no key information available by other means (e.g. a unique key or a custom message key defined in Debezium), the consumer uses the append mode for this table (see below).</p><h3>Keeping Deleted Records With Upsert Mode</h3><p>For some use cases it is useful to keep deleted records as a soft delete. This is possible by setting the debezium.sink.iceberg.upsert-keep-deletes option to true. This setting will keep the latest version of deleted records in the destination Iceberg table. Setting it to false will remove deleted records from the destination table.</p><h3>Append Mode</h3><p>This is the most straightforward operation mode, enabled by setting debezium.sink.iceberg.upsert to false. When using Debezium Server Iceberg with append mode, all received records are appended to the destination table. No data deduplication or deletion is done. With append mode it is possible to analyze entire change history of a record.</p><h3>Optimizing Batch Size</h3><p>Debezium extracts and delivers database events in real time, and this could cause too frequent commits to the tables in Iceberg, generating too many small files. This is not optimal for batch processing, especially when a near-realtime data feed is sufficient. To avoid this problem, it is possible to increase the batch size per commit.</p><p>When enabling the MaxBatchSizeWait mode, the Iceberg consumer uses Debezium metrics to optimize the batch size. It periodically retrieves the current size of Debezium’s internal event queue and waits until it has reached max.batch.size. During the wait time, Debezium events are collected in memory (in Debezium’s internal queue). That way, each commit (set of events processed) processes more records and consistent batch size. The maximum wait and check interval are controlled via the debezium.sink.batch.batch-size-wait.max-wait-ms and debezium.sink.batch.batch-size-wait.wait-interval-ms properties. These settings should be configured together with Debezium’s debezium.source.max.queue.size and debezium.source.max.batch.size properties.</p><p>Here’s an example for all the related settings:</p><pre>debezium.sink.batch.batch-size-wait=MaxBatchSizeWait<br>debezium.sink.batch.batch-size-wait.max-wait-ms=60000<br>debezium.sink.batch.batch-size-wait.wait-interval-ms=10000<br># increase max.batch.size to receive large number of events per batch<br>debezium.source.max.batch.size=50000<br>debezium.source.max.queue.size=400000<br>quarkus.log.category.&quot;io.debezium.server.iceberg.batchsizewait&quot;.level=DEBUG</pre><p>For the full story please see: <a href="https://medium.com/@ismail-simsek/building-a-data-lake-with-debezium-and-apache-iceberg-part-1-25124daf2a95">Part 1</a>, <a href="https://medium.com/@ismail-simsek/building-a-data-lake-with-debezium-and-apache-iceberg-part-2-bd13ddb19bbf">Part 2</a>, <a href="https://medium.com/@ismail-simsek/building-a-data-lake-with-debezium-and-apache-iceberg-part-3-86042974be15">Part 3</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=86042974be15" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Data Lake with Debezium and Apache Iceberg: Part 2]]></title>
            <link>https://medium.com/@ismail-simsek/building-a-data-lake-with-debezium-and-apache-iceberg-part-2-bd13ddb19bbf?source=rss-2215c1d16895------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd13ddb19bbf</guid>
            <category><![CDATA[debezium]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Ismail Simsek]]></dc:creator>
            <pubDate>Fri, 15 Nov 2024 11:24:32 GMT</pubDate>
            <atom:updated>2024-11-15T11:28:02.177Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://medium.com/@ismail-simsek/building-a-data-lake-with-debezium-and-apache-iceberg-part-1-25124daf2a95">In Part 1, we introduced Debezium Server Iceberg</a> and its potential for creating efficient data lake data feeds. In this post, we’ll dive into a practical example, demonstrating how to configure the application and leverage its advanced features to optimize data ingestion and transformation.</p><h3>Setting up data replication</h3><p>Debezium Server Iceberg is a Dockerized Quarkus application. You can configure it by providing an application.properties file. We&#39;ll explore some key configuration elements and an example configuration below.</p><p>We’ll start by configuring the Debezium Server Consumer to use an Iceberg sink. We’ll add more specific configurations later.</p><pre># Use iceberg sink. This is the main consumer used by application<br>debezium.sink.type=iceberg</pre><p>As the second step, we’ll configure Debezium to replicate data from an operational (source) database. In this specific example, we’ll replicate a <a href="https://debezium.io/documentation/reference/stable/connectors/postgresql.html">PostgreSQL database</a> using the following configuration. Here, we can leverage the full range of Debezium configurations to fine-tune the CDC replication behavior.</p><pre># Postgres source database<br>debezium.source.connector.class=io.debezium.connector.postgresql.PostgresConnector<br>debezium.source.offset.flush.interval.ms=0<br>debezium.source.database.hostname=postgresqlsourcedb<br>debezium.source.database.port=5432<br>debezium.source.database.user=postgres<br>debezium.source.database.password=postgres<br>debezium.source.database.dbname=postgres<br>debezium.source.database.server.name=tutorial<br>debezium.source.database.server.id=1234<br>debezium.source.schema.include.list=inventory<br>debezium.source.topic.prefix=dbz_<br><br># enable event schemas - used by the consumer to create iceberg tables.<br># And used by the consumer to do automatic schema evoluation, forexample field additions<br>debezium.format.value.schemas.enable=true<br>debezium.format.key.schemas.enable=true<br>debezium.format.value=json<br>debezium.format.key=json<br><br># do event flattening. unwrap complex debezium event type to flat format<br>debezium.transforms=unwrap<br>debezium.transforms.unwrap.type=io.debezium.transforms.ExtractNewRecordState<br>debezium.transforms.unwrap.add.fields=op,table,source.ts_ms,db<br>debezium.transforms.unwrap.delete.handling.mode=rewrite<br>debezium.transforms.unwrap.drop.tombstones=true</pre><p>Next, we’ll configure the Iceberg sink, defining the cloud destination, Iceberg Catalog, and replication mode (<em>append</em> or <em>upsert</em>).</p><pre># Iceberg sink config<br>debezium.sink.iceberg.table-prefix=debeziumcdc_<br>debezium.sink.iceberg.upsert=true<br>debezium.sink.iceberg.upsert-keep-deletes=true<br>debezium.sink.iceberg.write.format.default=parquet<br><br># Using JdbcCatalog as Iceberg catalog<br>debezium.sink.iceberg.type=jdbc<br>debezium.sink.iceberg.catalog-name=iceberg<br>debezium.sink.iceberg.table-namespace=icebergdata<br>debezium.sink.iceberg.uri=jdbc:postgresql://jdbccatalog:5432/test<br>debezium.sink.iceberg.jdbc.user=testuser<br>debezium.sink.iceberg.jdbc.password=testpass<br><br># Use S3FileIO, S3 Cloud Destionation<br>debezium.sink.iceberg.io-impl=org.apache.iceberg.aws.s3.S3FileIO<br>debezium.sink.iceberg.warehouse=s3://warehouse<br>debezium.sink.iceberg.s3.endpoint=http://minio:9000<br>debezium.sink.iceberg.s3.path-style-access=true<br>debezium.sink.iceberg.s3.access-key-id=admin<br>debezium.sink.iceberg.s3.secret-access-key=password<br># debezium.sink.iceberg.{other.debezum-config}=value</pre><p>For fault tolerance and efficient restarts, Debezium application requires persistent storage of state information. We’ll utilize two Iceberg tables to store replication offsets and schema information, ensuring seamless recovery in case of failures.</p><pre># saving debezium state data to destination, to iceberg tables<br># see https://debezium.io/documentation/reference/stable/development/engine.html#advanced-consuming<br>debezium.source.offset.storage=io.debezium.server.iceberg.offset.IcebergOffsetBackingStore<br>debezium.source.offset.storage.iceberg.table-name=debezium_offset_storage_table<br># see https://debezium.io/documentation/reference/stable/development/engine.html#database-history-properties<br>debezium.source.schema.history.internal=io.debezium.server.iceberg.history.IcebergSchemaHistory<br>debezium.source.schema.history.internal.iceberg.table-name=debezium_database_history_storage_table</pre><p>Lastly, configuring the desired log level for the application.</p><pre># ############ SET LOG LEVELS ############<br>quarkus.log.level=INFO<br>quarkus.log.console.json=false<br># set custom log level for certain libs<br>quarkus.log.category.&quot;org.apache.hadoop&quot;.level=WARN<br>quarkus.log.category.&quot;org.apache.parquet&quot;.level=WARN</pre><p>Now, with simple configuration, we can start replicating operational data to a raw data lake layer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bl9fEcLbe_hrOpQ2e5IG1g.png" /><figcaption>Debezium Server Iceberg replication Data Lake</figcaption></figure><p>A complete example setup is provided as a <a href="https://github.com/memiiso/debezium-server-iceberg/tree/master/examples">Docker Compose file here</a>.</p><p>For the full story please see: <a href="https://medium.com/@ismail-simsek/building-a-data-lake-with-debezium-and-apache-iceberg-part-1-25124daf2a95">Part 1</a>, <a href="https://medium.com/@ismail-simsek/building-a-data-lake-with-debezium-and-apache-iceberg-part-2-bd13ddb19bbf">Part 2</a>, <a href="https://medium.com/@ismail-simsek/building-a-data-lake-with-debezium-and-apache-iceberg-part-3-86042974be15">Part 3</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd13ddb19bbf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Data Lake with Debezium and Apache Iceberg: Part 1]]></title>
            <link>https://medium.com/@ismail-simsek/building-a-data-lake-with-debezium-and-apache-iceberg-part-1-25124daf2a95?source=rss-2215c1d16895------2</link>
            <guid isPermaLink="false">https://medium.com/p/25124daf2a95</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-lake]]></category>
            <category><![CDATA[data-replication]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[real-time-data]]></category>
            <dc:creator><![CDATA[Ismail Simsek]]></dc:creator>
            <pubDate>Fri, 15 Nov 2024 11:17:02 GMT</pubDate>
            <atom:updated>2024-11-15T11:28:24.740Z</atom:updated>
            <content:encoded><![CDATA[<p>In the world of Data Analytics, data lakes are a popular choice for storing and managing large amounts of data for data analytics, reporting or machine learning needs.</p><p>In this blog post we will describe a simple way to build a data lake. The solution is using a real-time data pipeline based on <a href="https://debezium.io/">Debezium</a>, supporting ACID transactions, SQL updates and Using highly scalable storage <a href="https://iceberg.apache.org/">Apache Iceberg</a>. And it’s not even requiring to have Apache Kafka or Apache Spark applications, which reduces complexity of the overall solution.</p><h3>Debezium Server Iceberg</h3><p>As operational data typically resides in a relational database or a NoSQL data store, the question is how the data can be propagated into the data lake. This is where the <a href="https://github.com/memiiso/debezium-server-iceberg">Debezium Server Iceberg</a> project comes in: Based on <strong>Debezium</strong> and <strong>Apache Iceberg. </strong>It lets you process real-time data change events from a source database and upload them to any object storage supported by Iceberg. So First let’s take a closer look at these two projects.</p><p><a href="https://debezium.io/"><strong>Debezium</strong></a> is a distributed streaming platform that captures changes to data in real-time. It can be used to capture changes from a <a href="https://debezium.io/documentation/reference/3.0/connectors/index.html">variety of databases</a>, including MySQL, PostgreSQL, and Oracle. The captured changes are then streamed to downstream applications, such as a data lake, event streaming platforms.</p><p><a href="https://iceberg.apache.org/"><strong>Apache Iceberg</strong></a> Iceberg is a high-performance format for huge analytic tables. Iceberg brings the reliability and simplicity of SQL tables to big data, while making it possible for engines like Spark, Trino, Flink, Snowflake, Bigquery, Databricks and <a href="https://iceberg.apache.org/docs/latest/">many more</a> engine to safely work with the same tables, at the same time.</p><h3>The “Debezium Server Iceberg” Consumer</h3><p>Leveraging the power of Debezium and Apache Iceberg, <a href="https://github.com/memiiso/debezium-server-iceberg">Debezium Server Iceber</a> project provides ready to use data pipeline solution. This application efficiently replicates data from operational databases to cloud-based Iceberg tables, enabling timely data analysis and insights. It is containerized application using <a href="https://quarkus.io/">Quarkus framework</a>.</p><p>Here are few advantages this application provides:</p><ul><li>Near real-time ingestion layer</li><li>Efficient data format Parquet or ORC</li><li>Correct data type reflected from source system</li><li>ACID compliant tables</li><li>De-duplicated raw layer (when the feature is enabled. We will see this feature in next article)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HTXUY3o-gz8ofVwqT_VEOg.png" /><figcaption>Data integration possibilities using <a href="https://github.com/memiiso/debezium-server-iceberg">Debezium Server Iceber</a>g</figcaption></figure><p>With <a href="https://github.com/memiiso/debezium-server-iceberg">Debezium Server Iceber</a>g start your data analytics journey ahead of everyone! For further details on how to setup please see Part 2 of the blog post.</p><p><a href="https://github.com/memiiso/debezium-server-iceberg">Debezium Server Iceber</a>g project completely open-source, using the Apache 2.0 license. It is still a young project and there are things to improve. Please feel free to test it, give feedback, open feature requests or send pull requests.</p><p>For the full story please see: <a href="https://medium.com/@ismail-simsek/building-a-data-lake-with-debezium-and-apache-iceberg-part-1-25124daf2a95">Part 1</a>, <a href="https://medium.com/@ismail-simsek/building-a-data-lake-with-debezium-and-apache-iceberg-part-2-bd13ddb19bbf">Part 2</a>, <a href="https://medium.com/@ismail-simsek/building-a-data-lake-with-debezium-and-apache-iceberg-part-3-86042974be15">Part 3</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=25124daf2a95" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Make DBT Great Again!]]></title>
            <link>https://medium.com/@ismail-simsek/make-dbt-great-again-ec34f3b661f5?source=rss-2215c1d16895------2</link>
            <guid isPermaLink="false">https://medium.com/p/ec34f3b661f5</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[dbt]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Ismail Simsek]]></dc:creator>
            <pubDate>Mon, 04 Nov 2024 11:28:03 GMT</pubDate>
            <atom:updated>2024-12-10T13:05:27.779Z</atom:updated>
            <content:encoded><![CDATA[<h3>Extend the Power of dbt with opendbt</h3><p>dbt is a popular solution for batch data processing in data analytics. While it operates on an <a href="https://opencoreventures.com/blog/2023-07-open-core-is-misunderstood-business-model/">open-core model</a>, which can sometimes limit the inclusion of community features in the open-source version. no worries <a href="https://github.com/memiiso/opendbt"><strong>opendbt</strong></a> is here to solve it. <a href="https://github.com/memiiso/opendbt">opendbt</a> offers a fully open-source package to address these concerns. OpenDBT builds upon dbt-core, adding valuable features without altering its core code.</p><p>How? Here’s a step-by-step example. We’ll demonstrate how to use OpenDBT to activate custom adapters and execute Python models locally within dbt. The key feature we’ll use is OpenDBT’s ability to “registering custom adapter classes”.</p><h3>The problem</h3><p>Using dbt core, one could do data Transform step inside the data part-form using dbt SQL, or Python (spark, snowpark). pretty much doing Doing T of ELT however its not possible to do ET Extract and load. for eaxmple extracting data from web api and Loading it to Data platform. the very fist step of data pipeline. This task usually done with python coding outside of dbt, which hides data lineage and dataflow dependencies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*2EQZeooJkfE4LlTmvwU7Jg.png" /><figcaption>External data imports with opendbt</figcaption></figure><p>In this post we will see how to do Extract and Load with dbt using <a href="https://github.com/memiiso/opendbt">opendbt</a>. with the solution entire data flow is done with dbt. That means full data dependencies end to end defined by dbt and documented.</p><p>While dbt core is powerful for data transformation (the ‘T’ in ELT), it’s limited to E and L step. It cannot handle the initial Extraction and Loading (the ‘E’ and ‘L’) of data, such as fetching data from web APIs and loading it into a data platform.</p><p>Typically, these early steps are handled outside of dbt, often with Python scripts. This approach can obscure data lineage and dependencies.</p><p>In this post, we’ll explore how to use OpenDBT to perform both extraction and loading within dbt. By doing so, we can maintain end-to-end data flow visibility and dependency tracking, ensuring a more robust and transparent data pipeline.</p><h3>Using your own customized Adapter</h3><p><strong>Step 1: Create a Custom Adapter</strong></p><p>We’ll start by extending an existing adapter (in this case, DuckDBAdapter) and customizing it to meet our specific needs. We&#39;ll add a new method to the adapter, making it accessible to dbt Jinja templates using the @available decorator. This method will be used to execute dbt Python models later on. see <a href="https://github.com/memiiso/opendbt/blob/main/opendbt/examples.py">full adapter code</a></p><pre>class DuckDBAdapterV2Custom(DuckDBAdapter): <br><br>     @available <br>     def submit_local_python_job(self, parsed_model: Dict, compiled_code: str): <br># python metdond to run local pyhon code, dbt pyhn models<br>         model_unique_id = parsed_model.get(&#39;unique_id&#39;) <br>         __py_code = f&quot;&quot;&quot; <br> {compiled_code} <br>  <br> # NOTE this is local python execution so session is None <br> model(dbt=dbtObj(None), session=None) <br>         &quot;&quot;&quot; <br>         with tempfile.NamedTemporaryFile(suffix=f&#39;__{model_unique_id}.py&#39;, delete=False) as fp: <br>             fp.write(__py_code.encode(&#39;utf-8&#39;)) <br>             fp.close() <br>             print(f&quot;Created temp py file {fp.name}&quot;) <br>             Utils.runcommand(command=[&#39;python&#39;, fp.name])<br>Step 2: Registering the Custom Adapter</pre><p>OpenDBT allows you to define custom adapters using the dbt_custom_adapter variable. To activate our custom adapter, add the following configuration to your dbt_project.yml file:</p><pre>vars:<br>  dbt_custom_adapter: opendbt.examples.DuckDBAdapterV2Custom</pre><p><strong>Step 3: Creating a dbt Macro to Execute Python Code</strong></p><p>Next, we’ll create a dbt macro named execute_python.sql. This macro will pass the compiled Python code to the new method we added to our custom adapter. see <a href="https://github.com/memiiso/opendbt/blob/main/opendbt/macros/executepython.sql">full macro code</a></p><pre>{% materialization executepython, supported_languages=[&#39;python&#39;]%}<br>  {%- set identifier = model[&#39;alias&#39;] -%}<br>  {%- set language = model[&#39;language&#39;] -%}<br>  {% set grant_config = config.get(&#39;grants&#39;) %}<br><br>  {%- set old_relation = adapter.get_relation(database=database, schema=schema, identifier=identifier) -%}<br>  {%- set target_relation = api.Relation.create(identifier=identifier,<br>                                                schema=schema,<br>                                                database=database, type=&#39;table&#39;) -%}<br>  {{ run_hooks(pre_hooks) }}<br>  {% call noop_statement(name=&#39;main&#39;, message=&#39;Executed Python&#39;, code=compiled_code, rows_affected=-1, res=None) %}<br>      {%- set res = adapter.submit_local_python_job(model, compiled_code) -%}<br>  {% endcall %}<br>  {{ run_hooks(post_hooks) }}<br>  {% set should_revoke = should_revoke(old_relation, full_refresh_mode=True) %}<br>  {% do apply_grants(target_relation, grant_config, should_revoke=should_revoke) %}<br>  {% do persist_docs(target_relation, model) %}<br>  {{ return({&#39;relations&#39;: [target_relation]}) }}<br><br>{% endmaterialization %}</pre><p><strong>Step 4: Executing Local Python Models</strong></p><p><strong>Ready to Run</strong>! With everything configured, let’s create and run Python a model locally, eliminating the need for Spark or Snowpark.</p><p><strong>Create Your Python Model:</strong></p><p>Save your model code in a file named models/my_local_python_model.py. In this model, we’ll simply print the Python version and platform information to demonstrate local execution.</p><pre>import os<br>import platform<br><br>def print_info():<br>    _str = f&quot;name:{os.name}, system:{platform.system()} release:{platform.release()}&quot;<br>    _str += f&quot;\npython version:{platform.python_version()}, dbt:{version.__version__}&quot;<br>    print(_str)<br><br>def model(dbt, session):<br>    dbt.config(materialized=&quot;executepython&quot;)<br>    print(&quot;==================================================&quot;)<br>    print(&quot;========IM LOCALLY EXECUTED PYTHON MODEL==========&quot;)<br>    print(&quot;==================================================&quot;)<br>    print_info()<br>    print(&quot;==================================================&quot;)<br>    print(&quot;===============MAKE DBT GREAT AGAIN===============&quot;)<br>    print(&quot;==================================================&quot;)<br>    return None</pre><p><strong>Step 5: Execute the Python Model:</strong></p><pre>dp = OpenDbtProject(project_dir=&#39;/dbt/project&#39;, profiles_dir=&#39;/dbt/project&#39;)<br>dp.run(command=&quot;run&quot;, args=[&#39;--select&#39;, &#39;my_local_python_model&#39;])</pre><p>The output should look something like this:</p><pre>09:57:11  Running with dbt=1.8.7<br>09:57:11  Registered adapter: duckdb=1.8.4<br>09:57:12  Unable to do partial parsing because config vars, config profile, or config target have changed<br>09:57:13  Found 4 models, 4 data tests, 418 macros<br>09:57:13  <br>09:57:15  Concurrency: 1 threads (target=&#39;dev&#39;)<br>09:57:15  <br>09:57:15  1 of 1 START python executepython model main.my_executepython_dbt_model ........ [RUN]<br>Created temp py file /var/folders/3l/n5dbz15s68592fk76c31hth8ffmnng/T/tmp4l6hufwl__model.dbttest.my_executepython_dbt_model.py<br>==================================================<br>========IM LOCALLY EXECUTED PYTHON MODEL==========<br>==================================================<br>name:posix, system:Darwin release:23.5.0<br>python version:3.9.19, dbt:1.8.7<br>==================================================<br>===============MAKE DBT GREAT AGAIN===============<br>==================================================<br>09:57:15  1 of 1 OK created python executepython model main.my_executepython_dbt_model ... [Executed Python in 0.51s]<br>09:57:15  <br>09:57:15  Finished running 1 executepython model in 0 hours 0 minutes and 2.53 seconds (2.53s).<br>09:57:15  <br>09:57:15  Completed successfully<br>09:57:15  <br>09:57:15  Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1</pre><p>Here’s a breakdown of the execution flow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*zQyLFEx45U7V3dSTFmY1sg.png" /></figure><p>The <a href="https://github.com/memiiso/opendbt">full code</a>, test cases, <a href="https://github.com/memiiso/opendbt/blob/main/docs/EXAMPLES.md#example-use-cases">documentation</a> and additional features are available on GitHub.</p><h3>Wrap-Up and Contributions</h3><p>The project completely open-source, using the Apache 2.0 license. <a href="https://github.com/memiiso/opendbt">opendbt</a> still is a young project and there are things to improve. Please feel free to test it, give feedback, open feature requests or send pull requests. You can see more examples and start <a href="https://github.com/memiiso/opendbt">experimenting with opendbt using</a> github project</p><p>For Part 2 please see: <a href="https://medium.com/@ismail-simsek/opendbt-effortlessly-streamline-your-data-pipelines-with-dbt-an-dlt-reduce-complexity-1ef065b03d5b">Streamline Your Data Pipelines with opendbt: Leverage dbt and dlt for Effortless End-to-End ELT Workflows</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ec34f3b661f5" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>