<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Onur Dincol on Medium]]></title>
        <description><![CDATA[Stories by Onur Dincol on Medium]]></description>
        <link>https://medium.com/@onurd80?source=rss-2e3271ecb33c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*OevIy7JpPZyz2zoQo24U6Q.jpeg</url>
            <title>Stories by Onur Dincol on Medium</title>
            <link>https://medium.com/@onurd80?source=rss-2e3271ecb33c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 15:48:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@onurd80/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Your Streaming Pipeline Is Running. Is the Data Still Good?]]></title>
            <link>https://medium.com/@onurd80/your-streaming-pipeline-is-running-is-the-data-still-good-fbc416850d83?source=rss-2e3271ecb33c------2</link>
            <guid isPermaLink="false">https://medium.com/p/fbc416850d83</guid>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Onur Dincol]]></dc:creator>
            <pubDate>Mon, 05 Oct 2026 02:38:31 GMT</pubDate>
            <atom:updated>2026-10-05T02:38:31.792Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Bad data is quieter than bad code. Here’s how to make it loud.</em></p><h3><em>Building a data quality and drift monitor with Kafka, Spark Structured Streaming, and RFF-MMD</em></h3><p>Say an upstream service starts sending purchase amounts in cents instead of dollars. The field is still numeric. The values are still positive. Kafka accepts the events, Spark processes them, and your pipeline keeps running.</p><p>From the infrastructure dashboard, everything looks fine.</p><p>The data is wrong, though, and anything consuming it now has a problem. A model may produce different scores. A revenue dashboard may show a sudden jump. Neither needs a deployment to go wrong.</p><p>That gap is what I wanted to explore with <a href="https://github.com/dincolonur/ai-data-quality-monitor">AI Data Quality Monitor</a>: a runnable project that watches the contents of a stream, detects rule violations and distribution changes, and makes the results visible in a live dashboard.</p><h3>Start with the failures you can describe</h3><p>Some data problems are straightforward to define. An age should not be 200. A purchase amount should not be negative. A required feature should not disappear from most incoming events.</p><p>For those cases, rules are the right starting point.</p><p>The project generates simulated user events and publishes them to a Kafka topic called ml-features. Spark Structured Streaming consumes those events with a 30-second trigger and processes each micro-batch through foreachBatch.</p><p>The validation layer checks missing values, ranges, and allowed categories. For example, age must fall between 0 and 120, purchase_amount must be nonnegative, and device_type should belong to the expected set of categories.</p><p>The result is a batch validity rate: how many rows passed the checks?</p><p>That metric is easy to understand and useful when debugging. If it drops after a client release, you have somewhere concrete to look. You can inspect the failing fields instead of starting with an unexplained change in downstream behavior.</p><p>But those rules would accept the cents-versus-dollars bug. The amounts are valid according to the contract. Their distribution has changed.</p><h3>Valid rows can still make an unfamiliar population</h3><p>Imagine that most users in your baseline traffic are between 18 and 75. Later, incoming ages concentrate between 65 and 105. Every age remains inside the allowed range, but the population no longer resembles the baseline.</p><p>This is a different question from validation. Validation asks whether an event satisfies a rule. Drift detection asks whether a group of events looks different from a reference group.</p><p>I used Maximum Mean Discrepancy, or MMD, for that comparison. It measures the difference between two distributions through their kernel mean embeddings. For this project, one group is baseline traffic and the other is the current batch.</p><p>A higher score indicates a larger difference under the chosen representation. It does not tell you whether the change is harmful. A successful marketing campaign can change your user population, too. The score gives you a reason to investigate; measuring model performance still requires predictions, outcomes, and the right evaluation metrics.</p><h3>Making the comparison practical for micro-batches</h3><p>A direct kernel MMD calculation involves pairwise comparisons. For similarly sized samples, that can mean quadratic work in the number of rows.</p><p>The project uses Random Fourier Features to approximate the kernel. Instead of building the full pairwise kernel matrix, it maps each row into a fixed feature space and compares the average transformed vectors.</p><p>The scoring logic looks roughly like this:</p><pre># Simplified illustration, not a drop-in implementation.<br># Fit preprocessing, weights, and bandwidth on the baseline.<br>def transform(X):<br>    return np.sqrt(2 / D) * np.cos(X @ weights + bias)<br><br>baseline_mean = transform(X_baseline).mean(axis=0)<br><br>def drift_score(X_batch):<br>    batch_mean = transform(X_batch).mean(axis=0)<br>    return np.sum((baseline_mean - batch_mean) ** 2)</pre><p>The important detail is that both groups use the same transformation. Resampling the random weights for each batch would change the representation you are comparing.</p><p>With fixed input dimensionality, the projection work scales roughly with the number of rows times the number of random features. The default configuration uses 200 random features. That provides a practical way to produce one numeric drift score per batch without calculating every pairwise similarity.</p><p>It also leaves choices to evaluate: feature scaling, kernel bandwidth, baseline selection, and the number of random features all affect what the detector sees.</p><h3>The threshold needs a baseline, too</h3><p>A drift score alone is not enough. You need some idea of how much it varies during ordinary traffic.</p><p>The monitor moves through three phases:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TS7WOAJirm4xCWH6y7Zvrw.png" /><figcaption>Project Monitoring Phases</figcaption></figure><p>The configured threshold is the 99th percentile of the calibration scores. At a 30-second trigger interval, those first ten batches represent roughly five minutes, excluding startup and processing overhead.</p><p>This makes the demo easy to run, but five scores are a small calibration sample. Their 99th percentile is close to the largest observed score; it is not evidence of a 1% false-alarm rate.</p><p>For a real deployment, I would collect a longer period of representative traffic and evaluate the threshold against normal variation. I would also check what entered the baseline. If an incident is already underway during warm-up, the monitor can learn that incident as normal.</p><p>Automatic calibration removes one manual step. It still depends on the quality of the reference data.</p><h3>A detector also needs an alert policy</h3><p>There is another decision after computing a score: when should someone hear about it?</p><p>The default policy requires two consecutive bad batches before firing an alert. Afterward, a 120-second cooldown suppresses repeated notifications for the same alert key.</p><p>That introduces a deliberate tradeoff. A brief spike may pass without a notification, while a persistent problem gets attention. The cooldown keeps one continuing incident from generating a message every 30 seconds.</p><p>With those defaults, detection should be understood in terms of batch cadence. Two qualifying batches can mean roughly a minute, with timing affected by when an incident begins and how long processing takes. This is a micro-batch monitor, so the trigger interval and alert policy both matter when setting a latency expectation.</p><p>The project also includes per-feature checks such as KS, PSI, chi-squared, and null-rate diagnostics. The aggregate MMD score helps identify a distribution change; feature-level results help explain where to look.</p><p>An alert becomes much more useful when it gives you a field and a metric to investigate.</p><h3>Keeping the whole path visible</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IQe6rEg8vy4fW22ECkaAlA.png" /><figcaption><em>Spark validates and scores each batch. FastAPI serves dashboard metrics and process controls; live logs reach the browser through SSE.</em></figcaption></figure><p>The streaming path is fairly small: the producer sends events to <strong>Kafka</strong>, <strong>Spark</strong> validates and scores each batch, and the alert dispatcher routes qualifying results to configured destinations such as console output, a JSONL file, and optional Slack or email.</p><p>I added a FastAPI control panel so the project could be explored without switching between several terminals. It starts and stops the producer and Spark job, exposes metrics, and streams process logs to the browser through Server-Sent Events.</p><p>Metrics use a separate path. Spark writes dashboard_state.json after each batch. FastAPI serves that state, and the browser polls it every five seconds to update the charts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_T5kRjSnVAMQU-43YgMOTQ.png" /><figcaption><em>The control panel brings incident controls, batch metrics, drift scores, null rates, and alert history into one view. This screenshot shows retained results; Kafka is unreachable and the Spark job is stopped at capture time.</em></figcaption></figure><p>The layout puts the experiment and its results next to each other. On the left, I can configure the producer or inject a null spike, range violation, schema problem, or distribution shift. In the center, I can compare the MMD score with its threshold and inspect feature null rates. The alert panel on the right adds the metric values and affected features. That makes it easier to follow a simulated incident from the input change to the signal and then to the notification.</p><p>That file-based handoff keeps the local demo simple. A shared deployment would need more work around process supervision, concurrent state access, authentication, and durable operational state. The demo makes the detection behavior inspectable; it is not a substitute for those production concerns.</p><h3>Testing with failures you can reproduce</h3><p>The simulator is useful because it can generate both normal traffic and deliberate incidents:</p><ul><li>Missing values in a large share of events.</li><li>Out-of-range values, including impossible ages and negative amounts.</li><li>Unexpected categories and malformed feature values.</li><li>Distribution changes in ages and purchase amounts.</li></ul><p>Those scenarios exercise different parts of the monitor. A null spike should affect validity and null-rate metrics. A population shift should affect the drift score even when individual values remain valid.</p><p>To try it locally with Docker available:</p><pre>git clone https://github.com/dincolonur/ai-data-quality-monitor<br>cd ai-data-quality-monitor<br>make up</pre><p>Open http://localhost:7070, start the producer and Spark job, and let calibration finish before injecting an incident. Then watch the score, validity rate, and alert feed together. The relationship between those signals is more informative than any one chart.</p><h3>What I would carry into a real pipeline</h3><p>The useful part of this project is the separation of three questions: does the data satisfy its contract, does it resemble the baseline, and does the change warrant a notification?</p><p>Each needs its own decisions. Rules need a feature contract. Drift needs representative reference data. Alerts need a policy that matches the cost of missing an incident and the cost of interrupting someone.</p><p>For a larger stream, I would also examine where the statistical computation runs, how much data it retains, and whether the per-batch work fits the available resources. Using Spark for ingestion does not by itself make every downstream calculation distributed.</p><p>The project gives me a way to test those decisions end to end. I can inject a failure, see which signals move, and inspect the alert that follows.</p><p>A running pipeline tells you that events are moving. This monitor adds another check: whether the events arriving now still look like the data you expected.</p><p><em>The source and local demo are available on GitHub: </em><a href="https://github.com/dincolonur/ai-data-quality-monitor"><em>AI Data Quality Monitor</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fbc416850d83" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stop Using MERGE INTO for High-Volume CDC Lakehouses — Try This Instead]]></title>
            <link>https://medium.com/@onurd80/stop-using-merge-into-for-high-volume-cdc-lakehouses-try-this-instead-0f77f595e7f6?source=rss-2e3271ecb33c------2</link>
            <guid isPermaLink="false">https://medium.com/p/0f77f595e7f6</guid>
            <category><![CDATA[delta-lake]]></category>
            <category><![CDATA[change-data-capture]]></category>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Onur Dincol]]></dc:creator>
            <pubDate>Mon, 14 Sep 2026 02:24:44 GMT</pubDate>
            <atom:updated>2026-09-14T04:53:08.180Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why I moved current-state resolution out of the ingestion path and built a working Postgres → Debezium → Kafka → Spark → Delta Lake demo to test the idea</h3><p>Imagine a Postgres table with billions of rows.</p><p>It receives millions of changes every day, and those changes need to appear in your lakehouse within seconds.</p><p>The architecture most of us would reach for is straightforward:</p><p><strong>Postgres → CDC → Kafka → Spark → </strong><strong>MERGE INTO → Delta Lake</strong></p><p>Capture every insert, update, and delete. Stream the events. Then continuously merge them into the target table.</p><p>It’s intuitive.</p><p>And for many workloads, it’s exactly the right design.</p><p>But as mutation volume grows, I think there’s a more interesting question to ask:</p><blockquote><strong><em>What if we stopped trying to maintain the current state during ingestion?</em></strong></blockquote><p>Instead of resolving every CDC event as it arrives, what if the ingestion path did one thing extremely well:</p><p><strong>Append the event.</strong></p><p>Then let the read layer decide what the current state actually is.</p><p>I built a working version of this architecture to explore that idea.</p><h3>The architecture most of us start with</h3><p>A typical CDC lakehouse pipeline looks something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MgRnnFMPu_KxCbHFujebZQ.png" /></figure><p>The CDC stream tells us what changed.</p><p>Spark processes those events in micro-batches.</p><p>Then MERGE INTO applies those changes against the target table.</p><p>Something conceptually like:</p><pre>MERGE INTO accounts target<br>USING updates source<br>ON target.account_id = source.account_id</pre><pre>WHEN MATCHED THEN UPDATE SET *<br>WHEN NOT MATCHED THEN INSERT *;</pre><p>Simple.</p><p>But that simplicity hides some interesting problems.</p><h3>Problem 1: updates aren’t really tiny updates</h3><p>A lakehouse isn’t a traditional row-oriented OLTP database.</p><p>The underlying data is generally stored in immutable columnar files such as Parquet.</p><p>So logically changing one small row doesn’t necessarily mean physically changing a few bytes.</p><p>Depending on the table format, configuration, file layout, and update strategy, mutations can require file rewrites or additional metadata structures.</p><p>Now multiply that by millions of CDC updates.</p><p>And then run it continuously.</p><p>The problem isn’t that MERGE doesn&#39;t work.</p><p>The problem is that we’re putting increasingly expensive current-state maintenance logic directly into the hottest part of the pipeline:</p><p><strong>the ingestion path.</strong></p><h3>Problem 2: CDC delivery isn’t perfectly clean</h3><p>Real CDC streams aren’t neat sequences of:</p><pre>event 1<br>event 2<br>event 3<br>event 4</pre><p>Jobs restart.</p><p>Kafka consumers rebalance.</p><p>Network failures happen.</p><p>Events are retried.</p><p>Events can arrive late.</p><p>Events can be delivered more than once.</p><p>If determining the latest state depends on processing time or naive “last write wins” semantics, correctness can get surprisingly complicated.</p><p>The ordering information I actually care about comes from the source system.</p><p>For Postgres, that means information such as the commit LSN(Log Sequence Number).</p><p>Kafka offsets can then provide an additional deterministic ordering signal where necessary.</p><h3>Problem 3: deduplication quietly becomes another stateful system</h3><p>One solution is to remember every event we’ve processed.</p><p>Put the IDs in Redis.</p><p>Or RocksDB.</p><p>Or maintain streaming state.</p><p>That works.</p><p>But now we’ve introduced another stateful system whose job is essentially:</p><blockquote><em>“Remember what happened so we don’t accidentally process it again.”</em></blockquote><p>That state has to survive restarts.</p><p>It has to scale with key cardinality.</p><p>It has to remain consistent with the data we’re writing.</p><p>And in some environments, adding another piece of infrastructure isn’t even an option.</p><p>I wanted to see whether I could avoid that entirely.</p><h3>So I reversed the problem</h3><p>Instead of asking:</p><blockquote><strong><em>How can I make the MERGE safer and faster?</em></strong></blockquote><p>I asked:</p><blockquote><strong><em>Do I really need to determine the current state during ingestion?</em></strong></blockquote><p>That led me to this architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yjclZOB-JxWUrtfspA-u2A.png" /></figure><p>There’s an important shift here.</p><p><strong>Silver answers:</strong></p><blockquote><em>What happened?</em></blockquote><p><strong>Gold answers:</strong></p><blockquote><em>What is true now?</em></blockquote><p>Those don’t have to be the same physical table.</p><h3>Every change becomes an immutable event</h3><p>Suppose account 123 changes several times.</p><p>Instead of continuously modifying one physical row, Silver might contain:</p><pre>account_id   operation   balance   source_lsn<br>------------------------------------------------<br>123          INSERT      100       501<br>123          UPDATE      150       620<br>123          UPDATE      175       701<br>123          DELETE      NULL      820</pre><p>We don’t overwrite the 100.</p><p>We don’t overwrite the 150.</p><p>We don’t physically delete the account when the delete event arrives.</p><p>We append everything.</p><p>The ingestion job becomes intentionally boring:</p><blockquote><strong><em>Read CDC event → normalize → append.</em></strong></blockquote><p>That’s a feature.</p><h3>Deterministic IDs matter more than sequential IDs</h3><p>There’s another issue that becomes important in replayable pipelines.</p><p>Consider generating surrogate IDs with:</p><pre>monotonically_increasing_id()</pre><p>That ID describes a particular execution.</p><p>It doesn’t necessarily describe the underlying business event.</p><p>Restart the job.</p><p>Repartition.</p><p>Backfill historical data.</p><p>The same logical event may receive a different identifier.</p><p>For CDC, I want the opposite property.</p><p>I want:</p><blockquote><strong><em>same source event → same identity</em></strong></blockquote><p>So an event key can instead be derived from stable source information:</p><pre>xxhash64(account_id, source_commit_lsn)</pre><p>Now if Kafka redelivers the event, its identity doesn’t change.</p><p>The pipeline can replay data without inventing a new identity for something that already happened.</p><p>That gives us a foundation for idempotent processing without requiring a globally coordinated sequence generator.</p><h3>But users don’t want event history</h3><p>This is the obvious objection.</p><p>An analyst doesn’t want to see:</p><pre>Account 123 = $100<br>Account 123 = $150<br>Account 123 = $175</pre><p>They want:</p><pre>Account 123 = $175</pre><p>So we expose current state as a view.</p><p>Here’s the simplified version:</p><pre>SELECT *<br>FROM (<br>    SELECT *,<br>           ROW_NUMBER() OVER (<br>               PARTITION BY account_id<br>               ORDER BY source_commit_lsn DESC,<br>                        kafka_event_offset DESC<br>           ) AS rn<br>    FROM silver_cdc_staging<br>) ranked<br>WHERE rn = 1<br>  AND operation_type != &#39;DELETE&#39;;</pre><p>For every business key:</p><p><strong>partition → order → rank → take latest</strong></p><p>That’s how we resolve the current state.</p><p>It still happens.</p><p>We simply moved it.</p><h3>And that’s the key idea</h3><p>The storage layer contains immutable history.</p><p>The consumer sees mutable state.</p><p>Those two things aren’t contradictory.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7G7qkL6GUqTnw2F0qiUgag.png" /></figure><p>Silver preserves what happened.</p><p>Gold resolves what should currently be visible.</p><h3>Deletes become surprisingly simple</h3><p>Deletes follow exactly the same rule.</p><p>A source delete produces:</p><pre>account_id = 123<br>operation_type = DELETE<br>source_lsn = 820</pre><p>Silver keeps it.</p><p>The current-state view finds the newest event for account 123, sees that it&#39;s a delete, and excludes the account.</p><p>So:</p><p><strong>DELETE doesn’t mean destroy history.</strong></p><p>It means:</p><p><strong>the latest version says this entity is no longer visible.</strong></p><p>As a side effect, we retain a useful audit trail.</p><p>We can still answer:</p><ul><li>When was this account created?</li><li>What values changed?</li><li>What did it look like yesterday?</li><li>When was it deleted?</li></ul><p>That history is already part of the data model.</p><h3>Didn’t we just make reads expensive?</h3><p>Yes.</p><p>This architecture doesn’t magically remove computational work.</p><p>It <strong>moves it</strong>.</p><p>If an account has 1,000 historical versions, a naive query shouldn’t repeatedly scan and rank all 1,000 versions forever.</p><p>That’s where asynchronous optimization enters the picture.</p><p>And this separation is important:</p><blockquote><strong><em>Correctness and physical optimization don’t have to be the same operation.</em></strong></blockquote><p>The append-only history gives us the correctness model.</p><p>Background maintenance can optimize the physical representation.</p><p>In the Delta Lake version I built, I experimented with Deletion Vectors as part of that optimization process.</p><p>Rather than making the ingestion pipeline continuously resolve existing data, a background process can identify superseded rows and make future reads more efficient.</p><p>The exact mechanism depends on your table format and workload.</p><p>The architectural principle is the interesting part:</p><blockquote><strong><em>Keep the hot write path simple. Optimize asynchronously.</em></strong></blockquote><h3>I wanted to see this actually run</h3><p>Architecture diagrams are easy.</p><p>I wanted to see what happened when the system received real CDC events.</p><p>So I built the entire pipeline locally.</p><p>The demo uses:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8JbUN0Sw7bUUcowRbStDsA.png" /></figure><p>The Spark job continuously consumes CDC events from Kafka and appends them into the Silver Delta table.</p><p>Then the Gold layer reconstructs current state using the ordering rules described above.</p><p>I also built a small dashboard so I could mutate the source database and watch the change move through the system.</p><p>Insert an account.</p><p>Watch the insert propagate.</p><p>Update it.</p><p>Watch Silver retain both versions while Gold exposes only the newest one.</p><p>Delete it.</p><p>Watch the delete remain in history while the account disappears from current state.</p><p>That made the architecture much easier to reason about than another diagram ever could.</p><h3>You can run the whole thing locally</h3><p>The project is available on GitHub:</p><p><a href="https://github.com/dincolonur/Lakehouse_CDC"><strong>Lakehouse_CDC</strong></a></p><p>It includes the Postgres source, Debezium CDC configuration, Kafka, Spark Structured Streaming ingestion, Delta Lake tables, current-state logic, compaction work, and the demo environment.</p><p>The goal wasn’t to create a production platform.</p><p>It was to create a small environment where the architectural tradeoff could actually be tested.</p><p>Clone it, start the environment, generate some mutations, break things, replay events, and watch what happens.</p><p>That’s much more useful than taking my word for it.</p><h3>So should you stop using MERGE INTO?</h3><p>No.</p><p>That’s intentionally more nuanced than the title.</p><p>MERGE INTO is an extremely useful operation, and for many workloads it&#39;s the simplest and best solution.</p><p>If your mutation volume is manageable, your latency requirements are reasonable, and your merge performance is predictable, there may be absolutely no reason to replace it.</p><p>But when you’re designing a high-volume CDC pipeline, I think there’s a question worth asking <strong>before</strong> optimizing your merge:</p><blockquote><strong><em>Does maintaining the current state actually belong in my ingestion path?</em></strong></blockquote><p>If the answer is no, a different architecture opens up:</p><ul><li><strong>append first</strong></li><li><strong>preserve source ordering</strong></li><li><strong>make event identity deterministic</strong></li><li><strong>resolve current state at read time</strong></li><li><strong>optimize asynchronously</strong></li></ul><p>The bigger lesson for me wasn’t really about Delta Lake or CDC.</p><p>It was about where we choose to put complexity.</p><p>Sometimes the best way to optimize an expensive operation isn’t to make it faster.</p><p><strong>It’s to ask whether it needs to happen there at all.</strong></p><h3>Try the demo</h3><p>The complete runnable implementation is available on GitHub: <a href="https://github.com/dincolonur/Lakehouse_CDC"><strong>Lakehouse_CDC</strong></a></p><p><strong>Lakehouse_CDC — Postgres → Debezium → Kafka → Spark → Delta Lake</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LoNlwg6AxGPZpgtWIuCT8w.png" /></figure><p>Clone it, run the pipeline, and try the insert/update/delete flow yourself.</p><p>If you work with CDC or large lakehouse tables, I’d also be interested in hearing where you’ve found the boundary between continuous MERGE and append-first architectures.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0f77f595e7f6" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>