<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Leonardo Araújo on Medium]]></title>
        <description><![CDATA[Stories by Leonardo Araújo on Medium]]></description>
        <link>https://medium.com/@leonardobruno.araujo?source=rss-7a3476500233------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*cT4e0djHIBoRVVx5.jpg</url>
            <title>Stories by Leonardo Araújo on Medium</title>
            <link>https://medium.com/@leonardobruno.araujo?source=rss-7a3476500233------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 06:48:50 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@leonardobruno.araujo/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[When Kafka Meets Delta Lake: A Confluent Tableflow Integration]]></title>
            <link>https://medium.com/@leonardobruno.araujo/when-kafka-meets-delta-lake-a-confluent-tableflow-integration-8d08ef2e7370?source=rss-7a3476500233------2</link>
            <guid isPermaLink="false">https://medium.com/p/8d08ef2e7370</guid>
            <category><![CDATA[databricks]]></category>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[streaming]]></category>
            <category><![CDATA[confluent-cloud]]></category>
            <dc:creator><![CDATA[Leonardo Araújo]]></dc:creator>
            <pubDate>Sat, 08 Nov 2025 21:38:47 GMT</pubDate>
            <atom:updated>2025-11-09T10:51:36.113Z</atom:updated>
            <content:encoded><![CDATA[<p>This article aims to provide a detailed and comprehensive walkthrough to help you achieve this integration. In the following sections, I will share a lot of screenshots and references that I believe will be helpful throughout the process.</p><p>In this article, you will find:</p><p>Learn how Confluent Tableflow automatically materializes Kafka topics into Delta tables and registers them with Unity Catalog.</p><p>Walkthrough how the integration reduces ingestion complexity, enforce data governance, and make real-time data immediately usable for analytics, and AI.</p><p>Happy streaming!</p><h3>What is Tableflow in Confluent Cloud?</h3><p>Tableflow in Confluent Cloud is a feature that enables you to stream data from Kafka topics directly into data lakes and warehouses in table formats like Apache Iceberg and Delta Lake.</p><p>Tableflow simplifies the process of building real-time data pipelines from streaming data (Kafka) to analytical workloads, It acts as a bridge between the operational plan and the analytical plan. Such a idea is visualized in figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*itBOlaL0NaW6RgAT.png" /><figcaption>Source: Confluent Cloud</figcaption></figure><p>Transferring data from the operational estate to the analytical estate is essential for leveraging operational data in analytics. Typically, streaming operational data from Kafka to data lakes is accomplished through a series of complex, expensive, non-reusable, and fragile data-processing tasks.</p><h3>Tableflow and Delta Lake tables</h3><p>Confluent Tableflow enables you to expose Kafka topics as Delta Lake tables. Tableflow automatically materializes Kafka topics into Delta Lake tables in your cloud storage. These tables can be consumed as external read-only Delta tables from Databricks. The figure below shows an example of what this data architecture look like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fxrzhC4E5JlNCnWT.png" /><figcaption>Source: Confluent Cloud</figcaption></figure><p>In this experiment, I will demonstrate how to enable and set up Tableflow and materialize a Kafka topic (transactions) into a Delta Lake table in cloud storage. In this case, I will use a S3 bucket. The high-level design is visualized in the figure below, along with the key components and resources used in this demonstration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xK23GdkfjOOr678n5-MQaw.png" /></figure><p>Tableflow Unity Catalog integration uses Databricks Unity Catalog Open Preview for creating external tables by using Unity Catalog open APIs. For more information, see <a href="https://docs.databricks.com/aws/en/external-access/create-external-tables?session_ref=direct&amp;url_ref=https%3A%2F%2Fdocs.confluent.io%2Fcloud%2Fcurrent%2Ftopics%2Ftableflow%2Fhow-to-guides%2Fcatalog-integration%2Fintegrate-with-unity-catalog.html">Create external Delta tables from external clients</a>.</p><h3>Prerequisites</h3><p>The prerequisites before running the experiment.</p><ul><li>Access to a Databricks Free Edition account</li><li>Access to an AWS account</li><li>Access to a Confluent Cloud account</li><li>Set up Kafka Connect Sample Data to generate the synthetic data used in this experiment. See the <a href="https://developer.confluent.io/courses/kafka-connect/how-to-generate-data-hands-on/?utm_medium=sem&amp;utm_source=google&amp;utm_campaign=ch.sem_br.nonbrand_tp.prs_tgt.dsa_mt.dsa_rgn.latam_lng.eng_dv.all_con.confluent-developer&amp;utm_term=&amp;creative=&amp;device=c&amp;placement=&amp;gad_source=1&amp;gad_campaignid=19560855033&amp;gclid=Cj0KCQjw64jDBhDXARIsABkk8J4w6o3_gwRfdC0XlOo3h1a0N5421In0NW6VRtaqh1YHqc_8FccrKpkaAidgEALw_wcB">documentation</a> for more information.</li><li>DeveloperWrite access on all schema subjects</li><li>CloudClusterAdmin access on your Kafka cluster</li><li>Assigner access on all provider integrations</li></ul><h3>Data sources</h3><p>For this demonstration, I will use a synthetic data from Kafka Connect Sample Data (Datagen) in Confluent Cloud as our data source. This dataset provides information about transactions.</p><h3>Key configurations</h3><p>Here are the key configurations required for running this demonstration.</p><h3><strong>Create a S3 Bucket</strong></h3><p>In this demo, named as poc-cc-tableflow-databricks</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1DtinZGTn2t-OkkcLo0B6g.png" /></figure><h3><strong>Configure Databricks Unity Catalog</strong></h3><p>Follow these steps on Databricks</p><ol><li>Navigate to the Databricks Free Edition workspace settings and click <strong>Identity and Access</strong>.</li><li>Create a new Service Principal named <em>tableflow-quick-start-sp</em> and generate a secret. Store the client ID and secret securely, because you will need them later to configure the Unity Catalog integration in Tableflow.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tRRpphcAmxortvRzmTWQdA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b3_aj8jXGG5fvkXBdzLkZw.png" /></figure><p>3. Create a new External Location</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rszf50LL8jGAKoHnhbFzwA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JOAhyOzOvDGdtsC-see7aQ.png" /></figure><p>Supply the S3 Bucket name and follow the next steps.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C1mDpnShntU56rpSYMre_Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9zsCZUhUnRzncg4O4T-CRA.png" /></figure><p>Test connection</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ossHOf_BqbGAuqcpAfNaZA.png" /></figure><p>If everything was configured correctly, the following screen below will appear.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*JHn-1Vv792R3GAbL1h64dg.png" /></figure><p>4. Create a New Catalog, for this guide, the catalog was named as <em>tableflow-quick-start-catalog</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/423/1*X9aOk8yRK9BYYa15oLVXUg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/644/1*0zsEue23Gr-o4NfwXUQmFQ.png" /></figure><p>Navigate to the <strong>tableflow-quick-start-catalog</strong> and grant the service principal created previously (<em>tableflow-quick-start-sp</em>) Data Editor access to the catalog.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*2xiyGu9SftHDntOfKMMWdg.png" /></figure><p>5. Navigate to the <strong>External Location</strong> created previously and grant the following permissions to <strong>tableflow-quick-start-sp</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/801/1*5FM_fbbiJy_Kpi6Z-3aStQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sWkNYBCfWWkwhHGtDGdgmw.png" /></figure><p>6. Enable External Data Access</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/410/1*TwU-LnuISIsw5C91EwBfQQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/249/1*IHHUdaVWTbwiUf3gEbjvgQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/1*iyljPP4kBsLcOrCYBU_TlQ.png" /></figure><h3>Configure Unity Catalog integration with Tableflow</h3><p>Follow these steps to enable Databricks Unity Catalog sync for Delta Lake tables in Tableflow.</p><ol><li>Log in to Confluent Cloud Console.</li><li>Create a Standard cluster in the AWS Ohio (us-east-2)</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XF_AplF6bPuQFv6-rjL1gQ.png" /></figure><p>3. Navigate to the Kafka cluster (cluster_0) that contains the topic will be synchronized (transactions) and click on <strong>Tableflow</strong>. In the <strong>Storage Connections</strong> section, click <strong>Go to Provider Integrations</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a9qNxns9jel-MPCZfmt_0w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ITH3JJchtE0f-dMhsj8F-g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6kK7kbSMCGjaBx4z2rLRfA.png" /></figure><p>And follow the next steps (configure policy and role on AWS IAM)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YODQiYHXZrRWD-_AXiKlbQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/1*V7-eppxiKCpnY8qr1jQstA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D-6dZUTBMWghVX87__S02w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r_CEbZrn7gwO7Mb99lwBBA.png" /></figure><p>Create role in AWS and map to Confluent</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7MHrVshrsjFGVtFHbvfl5A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f0Lf6GtFZL2TYIZfyTMJUw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VEDooyFR9iQqUVkSBzGtGQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QK-egjtJR1Kz4i2B4OvAYw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7N0f8HYfz36uI8H4HEGTWQ.png" /></figure><p>4. Create and configure Kafka Connect Sample Data to generate synthetic data, using the following datagen: transactions. This topic will be created with default configurations during this process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*grK0SqHpP0Ue0Y0rXDIt_Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/651/1*l7LBXiBWgIH3GRR5bp18AA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nmnuZe1V_HvdcwiZ_wU4PA.png" /></figure><p>Review the topic and confirm that events are being generated as expected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*odL18LSATVB84r6MMrI6rA.png" /></figure><p>5. Enable Tableflow button</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mfiiTZYqv5bQTfV-uGK0tQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/405/1*9dANKEvfUNELteOchP7qvw.png" /></figure><p>Important: <strong>S3 buckets must be in the same region (us-east-2) as the cluster. To maintain data integrity and table structure, avoid changing or removing objects in this bucket.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fHHb0_8JdXAJCR-aC6E3nQ.png" /></figure><p>Review and launch it</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6iC00TwQJtAc6vRe0paZPg.png" /></figure><p>This synchronization takes a few minutes. When properly configured, the Tableflow syncing status will show green as depicted below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wbx0TiVc-FgARr3KG7l_UA.png" /></figure><p>Check if the transactions topic has been represented as a delta table and if it is being synchronized to the S3 bucket. This synchronization takes a few minutes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EADwlxLYShQA9pqwHajh2w.png" /></figure><h3>Tableflow integration with Databricks Unity Catalog (External Catalog)</h3><ol><li>Navigate to the Kafka cluster (cluster_0) that contains the topic will be synchronized (transactions) and click on <strong>Tableflow</strong>. In the <strong>External Catalog Integrations</strong> section, click on<strong> Data Integration</strong>.</li><li>In the <strong>Select integration type</strong> section, select <strong>Databricks Unity</strong> and<strong> </strong>provide a name to identify your catalog integration and click <strong>Continue</strong>.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8EWFgK97ydkXXmQ2xvlmEg.png" /></figure><p>3. On the add connection details form, supply the following inputs and click <strong>Continue</strong>.</p><ul><li>Databricks Workspace URL</li><li>Client ID and secret of the service principal that you obtained in Step 2 (<strong>Configure Databricks Unity Catalog</strong> session above).</li><li>Name of the Unity Catalog to integrate with.</li><li>Review it and click <strong>Launch</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/561/1*qJuvZo2FkO7Gxg0QSe17oA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qy3jFFe58Wm0f6f0cDN3nw.jpeg" /></figure><p>This process takes a few seconds. When properly configured, the Databricks Unity Catalog syncing status will show green as depicted below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XTlpY_PtNt8CHksznbhPnA.png" /></figure><p>Notice that lkc-6vj5o8 below will be mapped as a new schema in Databricks Unity Catalog.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w3-Fc_YP3isbjD-tB08Qnw.png" /></figure><p>And voilà… the connection was established.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g-78-887X55dbf0n3_mecQ.png" /></figure><p>When you click the “Sample Data” button or query the table, the error below appears. It is necessary to grant the Data Reader permission to all account users on the new schema (lkc-6vj5oB).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*s33VUugceWnH1_uxANPBKA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*rYiqzkdoDVY32J7HqyUdjQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j6p16VQ8UwpQyvyy97eYSw.png" /></figure><p>Transactions table details</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/1*AzBLfk3QJmUW7wJzLgszAg.png" /></figure><p>Execute a simple query on the transactions table</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YlReursNxX2Fn0xKIQKLBg.png" /></figure><h3>Key takeaways</h3><ul><li><strong>Tableflow in Confluent Cloud</strong> enables seamless streaming of data from Kafka topics directly into data lakes and warehouses in table formats like Apache Iceberg and Delta Lake, simplifying real-time data pipeline creation.</li><li><strong>Integration with Delta Lake</strong>: Tableflow can automatically materialize Kafka topics as Delta Lake tables in cloud storage, making them accessible as external, read-only tables in Databricks.</li><li><strong>Unity Catalog Integration</strong>: The process leverages Databricks Unity Catalog for managing external tables, requiring configuration of service principals, external locations, and appropriate permissions.</li><li><strong>Step-by-step Setup</strong>: The article provides a detailed walkthrough, including prerequisites (Databricks, AWS, Confluent Cloud accounts), S3 bucket setup, Unity Catalog configuration, and Tableflow activation.</li><li><strong>Automation and Simplification</strong>: This integration reduces the complexity, cost, and fragility of traditional streaming-to-analytics pipelines, making operational data readily available for analytics.</li><li><strong>Permissions and Security</strong>: Proper IAM roles and permissions are essential for successful integration and data synchronization.</li><li><strong>Validation</strong>: The process includes steps to verify that data is being synchronized and accessible as Delta tables in Databricks.</li></ul><p>Special thanks to <a href="https://www.linkedin.com/in/rosaneprado/">Rosane Prado</a> (Confluent Cloud) for making this possible.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8d08ef2e7370" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>