<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dani Palma on Medium]]></title>
        <description><![CDATA[Stories by Dani Palma on Medium]]></description>
        <link>https://medium.com/@danthelion?source=rss-f19c56481579------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Nk5gx4bReeyCIUG7qXtRIw.png</url>
            <title>Stories by Dani Palma on Medium</title>
            <link>https://medium.com/@danthelion?source=rss-f19c56481579------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 08:45:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@danthelion/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Most Valuable API in the Modern Data Stack Returns No Data]]></title>
            <link>https://blog.dataengineerthings.org/the-most-valuable-api-in-the-modern-data-stack-returns-no-data-88978bd2ad4c?source=rss-f19c56481579------2</link>
            <guid isPermaLink="false">https://medium.com/p/88978bd2ad4c</guid>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[etl]]></category>
            <dc:creator><![CDATA[Dani Palma]]></dc:creator>
            <pubDate>Wed, 19 Aug 2026 15:17:12 GMT</pubDate>
            <atom:updated>2026-08-19T15:17:12.200Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ubPqU655g4AEguNmwt5PCg.png" /></figure><p>Throughout my career, I have spent an embarrassing amount of time rebuilding other people’s databases. Nobody really calls it that of course. Around here we like to call it “integrating with the API.”</p><p>You probably already know the drill: somewhere inside a SaaS vendor there’s a Postgres table that contains data from customers, events, invoices, whatever, and you need that table in your warehouse for analytics. But, between you and the data there is a REST API that serves the table a thousand or so rows at a time, hiding behind some weird cursor token, and completely unwarranted rate limits.</p><p>So, as any good data engineer, you write a connector that crawls through every page, handles every 429 response, reassembles the chunks, and figures out which rows changed since yesterday. Oh, and you will write the retry logic twice, and of course the second version will also be wrong.</p><p>Multiply that by every tool a company runs, and you get a whole industry (ETL, ELT, EtLT, CDC, reverse ETL: pick your acronym) whose actual job is simply reconstructing tables that already exist, byte for byte, inside someone else’s disk.</p><p>To be fair, APIs for transactional work are great. They enforce business logic and they also authorize individual actions. Plus, they answer point lookups in milliseconds. No complaints there.</p><p>The bulk export API is a completely different animal: it exists because for decades we had no standard way to simply hand someone a table, so we handed them a straw and pointed at the lake.</p><p>And that’s the thing that’s ending. The replacement wave summarized: share a pointer to where the data already lives, and let the consumer bring their own compute to it.</p><h3>Everybody gets a pointer!</h3><p>If you skim data infra vendor blogs, in the past 2 years you can pretty much line the announcements up, and it will read as the same press release with a slightly different accent.</p><p>Snowflake extended its sharing to open table formats, so a provider can share Iceberg and Delta tables across clouds and regions without pipelines and without per-query egress ransom. At this year’s Summit, they took it a step further: integrations that plug SAP, Salesforce, and Workday straight into the platform, zero copies, now GA . Databricks made (again?) Iceberg a first-class citizen inside Delta Sharing and will now vend credentials for tables living in catalogs it doesn’t even control. Additionally, a while back, AWS shipped S3 Tables, managed Iceberg behind the standard REST catalog interface, attachable from any engine that speaks it. Then AWS and Salesforce wired their pieces together so Salesforce can query Iceberg tables sitting in a customer’s own S3 bucket, through Glue’s REST endpoint, no copies made. Salesforce! In a data infrastructure story! The horror. I was surprised too.</p><p>Meanwhile, at the other end of the budget spectrum, an open source developer built<a href="https://faceberg.kszucs.dev/"> Faceberg</a>, which cleverly maps Hugging Face datasets to Iceberg tables by generating a thin layer of metadata over Parquet files that already exist on the Hub. Similarly to the big vendors, nothing actually gets copied. A catalog on free infrastructure serves the pointers, and DuckDB or pandas simply resolves them.</p><p>So, a hyperscaler and a solo developer arriving at the same architecture in the same year. What gives? It’s a good signal: when something like this happens, the architecture was probably sitting latent in the standards, waiting for someone to just trip over it.</p><p>If you strip away the branding, and every one of these systems exchanges the same four things:</p><ol><li>a schema</li><li>a list of file locations</li><li>a snapshot pointer</li><li>a short-lived credential.</li></ol><p>The bytes don’t move until a user actually queries them, and then they flow straight from storage to the consumer’s engine, meaning the producer stops operating a serving layer and starts publishing a reference.</p><h3>We tried this before, and it didn’t work</h3><p>Now for a bit of history, because the graybeards in the comments will bring it up and they’ll be right to.</p><p>“Query the data where it lives” is one of the oldest promises in this field: in the 2000s we called it data virtualization, or Enterprise Information Integration (yuck). The pitch was, word for word, the pitch above: no copies, no pipelines, one interface over everything. I was working in telecom back then, close enough to enterprise IT to watch the deployments land, and then to watch them get walked back release by release. You might hear zero-ETL or zero-copy ETL as a new-gen version of the term.</p><p>The failures were pretty specific: storage and compute lived in the same boxes, so “querying in place” meant pointing analytical scans at production databases, which DBAs enjoyed exactly as much as you’d expect. There was no standard file layout, so the federation layer spoke every system’s dialect through custom adapters, and each adapter was a connector problem with a fancier name. Egress was egregiously priced like a toll road. And weirdly, the virtualization layer itself became the bottleneck, a single choke point in the middle of the exact architecture that promised to remove one.</p><p>Anyone who lived through that is rightfully skeptical. But here we are, claiming that an old idea is back again, but this time, the specific things which killed it have specifically changed.</p><p>So, what actually changed?</p><h3>Five things that changed</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GCIbuo5zNm_PaNpnzcWi2w.png" /></figure><p>First of all, object storage split the bytes from the engines, so a big scan no longer competes with production traffic. Parquet standardized the file, or well, is getting close to doing that.</p><p>If we step one abstraction higher, we have Iceberg, which standardized the table, layering transactions, schema evolution, and time travel on top of what used to be a folder of hope. The Iceberg REST catalog spec then standardized the pointer exchange itself: a small open HTTP contract for asking “what tables exist, and where’s the current metadata for this one.”</p><p>DuckDB, Spark, Trino, Snowflake, all of them resolve the same reference now. And zero-egress object storage killed the “toll booth” that made cross-org reads economically silly.</p><p>Five preconditions, all flipped, roughly in that order. Data virtualization failed as a product and came back as a property of the infrastructure, which is the best possible outcome for an idea and the worst possible one for the vendors who sold it the first time.</p><h3>The catalog is the API gateway</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZGFW0hxrFkyEHyvo7C1Hlg.png" /></figure><p>Once the payload stops moving, everything an API gateway used to do migrates to the pointer exchange. Meaning stuff like discovery becomes listing namespaces and auth becomes credential vending: the catalog hands a consumer short-lived, scoped access to exactly the files behind one table and nothing else. Versioning becomes snapshots and stuff like deprecation becomes plain schema evolution, with the awkward twist that consumers can now watch it coming by reading your metadata.</p><p>Chris Riccomini made the great<a href="https://materializedview.io/p/these-arent-the-catalogs-youre-looking"> observation</a> that we’ve built data catalogs, service catalogs, and schema registries as separate species and they’re all converging into one thing. The lakehouse catalog is where the convergence lands. It’s turning into the control plane for data exchange (access, contracts, lineage) while the data plane collapses into object storage plus whatever engine the consumer already likes.</p><p>And yes, before anyone types it: the Iceberg REST catalog is itself a REST API. Guilty. I guess this blog’s title hides a narrower reality: one small standardized API that returns metadata, replacing ten thousand bespoke APIs that return data. The most valuable API in the modern data stack returns no data at all.</p><h3>The whole pattern in thirty lines</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P9MRSw-xYN9l17KSDpiDZg.png" /></figure><p>To illustrate the concept, we have to understand that none of this requires an enterprise contract or a platform team, in fact, the minimum viable version can run on a laptop.</p><p>First things first, you need a catalog: this is the part that sounds complex but in reality it really isn’t. <a href="https://github.com/lakekeeper/lakekeeper">Lakekeeper</a> and <a href="https://github.com/apache/polaris">Apache Polaris</a> are open source and have Docker containers available. S3 Tables and Glue expose a similar interface if you live on AWS, and Snowflake Open Catalog and Databricks Unity can work it too.</p><p>Thankfully, they all implement the same REST spec, so picking one is an operational decision rather than an architectural one, which is precisely what the spec was for. For a local experiment, docker run -p 8181:8181 tabulario/iceberg-rest gives you a working catalog in about four seconds.</p><p>The producer side is PyIceberg (pip install “pyiceberg[pyarrow]”) and it looks like this:</p><pre>from pyiceberg.catalog import load_catalog<br>import pyarrow as pa<br><br>catalog = load_catalog(&quot;publisher&quot;, uri=&quot;http://localhost:8181&quot;)<br>catalog.create_namespace(&quot;shared&quot;)<br><br>orders = pa.Table.from_pylist([<br>  {&quot;order_id&quot;: 1, &quot;amount&quot;: 42.0, &quot;placed_at&quot;: &quot;2026–07–21&quot;},<br>  {&quot;order_id&quot;: 2, &quot;amount&quot;: 17.5, &quot;placed_at&quot;: &quot;2026–07–22&quot;},<br>])<br><br>table = catalog.create_table(&quot;shared.orders&quot;, schema=orders.schema)<br>table.append(orders)</pre><p>And that is basically the entire publishing workflow. Notice the things we didn’t do: no endpoint was designed, no pagination scheme was invented, no rate limiter was tuned, because nothing here actually serves requests.</p><p>The catalog now knows that a table called shared.orders exists and where its current metadata lives, and that’s enough.</p><p>The consumer side is where it gets a bit more fun, because the consumer never talks to you. Here’s DuckDB:</p><pre>INSTALL iceberg;<br>LOAD iceberg;<br><br>CREATE SECRET partner_secret (<br>  TYPE iceberg,<br>  CLIENT_ID &#39;partner-app&#39;,<br>  CLIENT_SECRET &#39;…&#39;,<br>  OAUTH2_SERVER_URI &#39;https://catalog.example.com/v1/oauth/tokens&#39;<br>);<br><br>ATTACH &#39;my_warehouse&#39; AS shared_data (<br>  TYPE iceberg,<br>  ENDPOINT &#39;https://catalog.example.com&#39;,<br>  SECRET partner_secret<br>);<br><br>SHOW ALL TABLES;<br><br>SELECT sum(amount) FROM shared_data.shared.orders;<br><br> - pin to a specific snapshot, i.e. version your dependency<br>SELECT * FROM shared_data.shared.orders AT (VERSION =&gt; 8426154373847);</pre><p>(For the local toy catalog, you can skip the secret and pass AUTHORIZATION_TYPE NONE to the ATTACH.)</p><p>The entire security model is neatly hiding inside that ATTACH block. DuckDB authenticated against the catalog, the catalog checked what this principal is allowed to see, and then it vended short-lived storage credentials scoped to exactly the files behind the tables in that share. Then, data flowed straight from object storage to the laptop. The producer’s infrastructure did nothing except answer metadata questions, and the AT (VERSION =&gt; …) clause is API versioning, except the version pin is a snapshot ID instead of a /v2/ in a path. You can easily swap DuckDB for Spark, Trino, or Snowflake and the logic is the same, which is the interoperability that ten thousand bespoke export APIs failed to deliver.</p><p>The producer’s side is definitely the less photogenic half. Iceberg tables accumulate small files and dead snapshots as they’re written to, and now strangers depend on both being managed. Concretely, that means running the standard maintenance procedures on a schedule:</p><pre>CALL system.rewrite_data_files(table =&gt; &#39;shared.orders&#39;);<br>CALL system.expire_snapshots(table =&gt; &#39;shared.orders&#39;, retain_last =&gt; 7);</pre><p>or paying a managed catalog to do it for you, which is a large part of what S3 Tables actually offers as a product: automatic compaction and snapshot expiry, invisible janitorial work priced into the storage.</p><p>Based on my experience implementing similar data architectures, my advice is to write the janitorial promises down the way you’d write an availability SLA: snapshots can be retained for seven days and files compacted nightly toward 128 MB targets. This one-page document is essentially the new API reference: it’s shorter than the old one, and every line of it is verifiable by the consumer reading your metadata, which keeps everyone honest.</p><p>But it’s also where the problems start.</p><h3>Where this eventually breaks</h3><p>This is the part where I argue with myself, because the failure modes are still real and I’d rather list them than have them listed for me.</p><p>Privacy first, because it’s the tough one. An aggregation endpoint that returns a count is also a privacy mechanism. Handing a partner the raw table is exactly what a bank or a hospital can’t do, ever, no matter how elegant the metadata. Row and column filters on shares help. Clean rooms are this pattern’s real answer, letting two parties compute over a join neither side can read directly. But the governance tooling for fine-grained, cross-organization access sits years behind the file formats, and anyone deploying this today feels that gap in the first week.</p><p><strong>Then pesky physics.</strong></p><p>A common format across three clouds still leaves three physical islands, and a join across them still ships bytes over a network someone pays for. The zero-copy critics (Jack Vanlightly has written<a href="https://jack-vanlightly.com/blog/2025/10/15/why-im-not-a-fan-of-zero-copy-apache-kafka-apache-iceberg"> a great version</a>) point out that someone still constructs the Parquet, still runs the compaction, still rewrites the small files, and that’s real compute done on behalf of every downstream reader. “Zero copy” almost always translates into “the copy is now your problem.”</p><p><strong>Operations become hard(er).</strong></p><p>Publish a table as a product, and your internal maintenance becomes a public contract. Compact too aggressively, and you churn files under a consumer mid-scan. Get lazy about snapshot expiry, and you’re silently serving everyone a slow table. The old API hid all of this behind an endpoint, but a shared table exposes it, and “running tables as products” isn’t yet a discipline with a book you can hand someone.</p><p><strong>The one I’d bet matters most long-term: semantics.</strong></p><p>A pointer to bytes is a pointer to bytes. The API you’re replacing came with docs, business definitions, and aggregation logic baked into its endpoints. A manifest file carries none of that. Hand someone a raw table, and you’ve transferred the data while dropping the meaning on the floor — not ideal. Semantic layers and metadata standards are about to graduate from the boring panel at the conference to the fight worth watching.</p><h3>Agents will force the issue</h3><p>If the economics weren’t enough, the timing hopefully does it. Agents are fluent in SQL but hopeless at detangling your API because every paginated endpoint with its own auth quirks and undocumented rate limits is a small research project by itself for agents and humans alike. On the other hand, a shared Iceberg table is graciously self-describing: schema, stats, partitioning, history, all queryable through one standard interface. AWS is already wiring MCP support into S3 Tables so agents can find and query them directly.</p><p>I can’t picture a future where millions of agents each maintain millions of connector integrations; on the other hand, attaching to catalogs, sure.</p><p>The unsolved parts are the ones from the previous section, sharpened. Credential vending was designed for human-driven engines; scoping access for an autonomous consumer remains an open question with no settled answer. And agents need the semantics even more than people do, because an agent can’t walk over to your desk and ask what gmv_net_v3 means.</p><h3>The payload API becomes a cache</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6-5ulOU_M0hADzt0_8s-MQ.png" /></figure><p>None of this touches transactional APIs. Actions, writes, point lookups: those stay behind endpoints that enforce business logic, as they should.</p><p>The demotion is specific. The bulk export API, the paginated sync, the “get our data into your warehouse” connector, is turning into an optimization layer over shared tables. It’ll survive the way origin servers survived CDNs: as the thing behind the thing, invoked when freshness or shaping demands it.</p><p>For anyone producing data, the strategic question has already changed shape. It used to be which API to build, but it’s becoming which tables to publish, under what contract, with what maintenance promise, through whose catalog. The teams that answer early get to spend the next decade being pleasantly boring to integrate with, which is the highest compliment infrastructure can receive.</p><p>Share pointers. Let the payloads stay home.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=88978bd2ad4c" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/the-most-valuable-api-in-the-modern-data-stack-returns-no-data-88978bd2ad4c">The Most Valuable API in the Modern Data Stack Returns No Data</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Moat Is the Feedback Loop]]></title>
            <link>https://medium.com/@danthelion/the-moat-is-the-feedback-loop-2c1572f48f61?source=rss-f19c56481579------2</link>
            <guid isPermaLink="false">https://medium.com/p/2c1572f48f61</guid>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Dani Palma]]></dc:creator>
            <pubDate>Tue, 04 Aug 2026 17:03:53 GMT</pubDate>
            <atom:updated>2026-08-04T17:03:53.662Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p3BUspfddeKu-PyfxhoiRg.png" /></figure><p>I used to think the moat of a data platform was the amount of engineering behind it. Then AI killed that, so I thought: the moat has to be the data itself then, right? Nope.</p><p>Agents can read API docs, generate code, write data models, look at logs, create tests, and event suggest fixes, but the output will still need a review. On the other hand, prod infra still requires experienced engineers and plenty of generated code is just mediocre, but more of the initial work is cheap now, and the boundary keeps moving.</p><p>Most SaaS moats are simply backlogs that would be painful to reproduce, so when the cost of working through the backlog falls, so does the protection it provides.</p><p>Essentially, AI learned to write the pipeline first and now it is learning why the pipeline actually exists.</p><p>My take is that second part will have a vastly bigger effect on data engineering than the first.</p><h3>Data alone is not enough</h3><p>After the engineering part becomes easier, the usual answer is that the data itself remains valuable, right?</p><p>Well, sometimes it does: a company may own a dataset that is expensive to collect, legally difficult to access, or generated by a product that competitors cannot easily copy.</p><p>But honestly, most internal company data is not special. Companies have customers, orders, invoices, subscriptions, support tickets, contracts, application events, and product usage. The schemas can be different, but the basic categories are gonna be familiar.</p><p>Counterintuitively, what makes the data useful is everything that is missing from the schema.</p><ol><li>Which customer table should you use?</li><li>Does revenue mean invoices issued, cash collected, or recognized revenue?</li><li>Does an active customer need to log in, pay an invoice, or use a specific feature?</li><li>Why does the finance dashboard disagree with the sales dashboard?</li><li>Why does everyone avoid one column even though its name looks correct?</li><li>Which exceptions were added after the company entered a new market?</li></ol><p>These questions are usually answered through ✨ experience ✨. A senior analyst probably knows that the status field became unreliable after a migration two years ago. Or, someone in finance remembers that a specific type of credit must be applied to the previous quarter. And of course when an engineer knows that one table is technically current but still should not be used for reporting for “reasons”.</p><p>The database contains the records, but the company itself contains the meaning behind those records.</p><p>That meaning has traditionally been hard to transfer. For example, anew employee can read the docs and still spend months discovering which parts are outdated or which reports people actually trust.</p><p>This institutional knowledge can feel like a strong moat because it is difficult for an outsider to acquire. At the same time, it is also difficult for an insider to acquire.</p><p>AI changes that cost as well.</p><h3>Institutional knowledge leaves traces</h3><p>Sadly a ton of institutional knowledge lives only in people’s heads, but thankfully, most of it actually leaves traces somewhere.</p><p>It can appear in stuff like SQL queries, dashboard filters, model definitions, data tests, code reviews, support tickets, incident reports, meeting notes, Slack messages, documentation, or even comments written during migrations.</p><p>Although none of these sources give a complete picture but together, they often say a lot.</p><p>Like when someone asks an agent for net revenue from active enterprise customers last quarter, the agent might find three different revenue models.</p><p>One can feed a sales dashboard, another appears in finance reports, and a third is used by the product team. It can do its best to inspect lineage, query history, owners, tests, and recent changes and it may find a ticket explaining that refunds issued after the end of the quarter are handled differently in board reporting.</p><p>It can then even calculate the result using the most likely definition and show where the ambiguity remains:</p><blockquote><em>Finance and sales use different rules for credits issued after quarter close. I used the finance definition because it feeds the quarterly reporting model. Should that rule apply here?</em></blockquote><p>That is a much better question than asking someone to explain the entire revenue model from zero.</p><p>Once a person answers, the agent can save the decision along with the source, owner, date, and scope. And with that, the next request starts with more context — for everyone.</p><p>This process does not require the agent to understand the whole company at once. It can build a working model, test it against existing outputs, and ask for help when the evidence conflicts.</p><p>That is close to how a new analyst learns, except the agent can read far more of the company’s history before asking its first question.</p><p>The institutional knowledge has not disappeared, but the cost of finding and applying it has fallen dramatically.</p><h3>A clean data stack only gets you halfway</h3><p>This might sounds like an argument for better metadata, lineage, catalogs, and semantic layers. Those things matter, but they only describe part of the problem.</p><p>A company can have beautiful data architecture and still be extremely difficult to understand.</p><p>Maybe the metric definitions are documented, but nobody updates them. Maybe every team has its own version of the customer model. Maybe decisions happen in private calls and the reasoning is never recorded. Maybe nobody owns a disputed number, so the disagreement continues for years.</p><p>An agent trained on that environment will simply learn the confusion along with everything else.</p><p>Good pipelines cannot fix a company that has no clear way to make and record decisions. Sure, lineage can show that a dashboard depends on a table, but it cannot tell you whether the dashboard should be trusted or not. A semantic layer can define revenue, but it cannot actually force finance and sales to agree on the definition.</p><p>For an AI system to learn useful context, the company itself has to produce useful context.</p><p>Important decisions need a home. Definitions need owners. Changes need reasons, not just new code. Corrections need to be captured instead of disappearing into a chat thread. When two teams use different definitions for a good reason, that difference should be explicit.</p><p>The company also needs a way to evaluate what the agent does, because a generated query may run successfully and still answer the wrong business question. A recommended action may look reasonable and still produce a bad outcome.</p><p>A well-architected data system gives the agent access to reliable information. A well-architected company gives it a way to learn from what happens next.</p><h3>The loop is the point</h3><p>A static collection of documentation can be copied, a database can be exported and a semantic model can be recreated. Even a large body of institutional knowledge becomes less protective once an agent can read and summarize it quickly.</p><p>The ongoing feedback loop keeps producing new knowledge.</p><p>An agent makes an interpretation, then someone approves it, changes it, or rejects it. <strong>The reason has to be recorded so the agent will use the revised definition next time.</strong></p><p>Then, an incident occurs and an engineer discovers that a source system changed its behavior. <strong>The fix and explanation become part of the context used during future incidents.</strong></p><p>An agent recommends contacting a group of customers. The company sees which customers responded, which ones churned, and which signal turned out to be misleading. <strong>That outcome changes the next recommendation.</strong></p><p>Over time, the system learns more than table definitions. It learns which evidence the company trusts, how exceptions are handled, which actions tend to work, and where human judgment is still required.</p><p>That accumulated history is hard to reproduce because it comes from actually operating the business.</p><p>A competitor can use the same LLM and it can build similar connectors and run the same kind of agent, but it will never have the same record of decisions, corrections, exceptions, and outcomes.</p><p><strong>So, this is where the moat moves.</strong></p><p>The model itself will probably be widely available, the infrastructure is trivial to build. The advantage comes from having a better loop between data, action, evaluation, and memory.</p><h3>Feedback needs to be useful</h3><p>A thumbs-up button is technically feedback, but it does not teach much.</p><p>Useful feedback explains what went wrong.</p><ol><li>The query used the wrong date field.</li><li>The metric definition was correct for monthly reporting but wrong for a renewal analysis.</li><li>The recommendation ignored a recent pricing change.</li><li>The customer should have been excluded because the contract is under renegotiation.</li></ol><p>This kind of correction needs to be attached to the original question, the evidence the agent used, and the action it took, otherwise the same mistake will return in a slightly different form.</p><p>Outcomes matter even more.</p><p>An agent may produce an answer that everyone agrees with and still lead to a poor decision. Companies need to connect recommendations with later results where possible. That could mean revenue, retention, conversion, incident recovery time, support resolution, or any another observable effect.</p><p>The loop becomes truly valuable when it includes what happened in the real world, rather than stopping after a human accepted the output.</p><p>This will require discipline as many companies cannot currently explain why a decision was made, let alone connect it to an outcome six months later. Agents will expose that weakness quickly.</p><h3>What this means for data vendors</h3><p>Data vendors mostly focused on shipping features like new connectors and prettier user interfaces, and those features still matter of course, as a reliable runtime remains difficult to operate. Customers will continue paying for infrastructure they do not want to maintain.</p><p>But the amount of product differentiation available in the procedural layer will shrink. Agents will be able to generate more integrations, move between systems, translate models, and operate infrastructure through APIs rather than carefully designed screens.</p><p>The more interesting products will sit close to the places where context is created.</p><p>They will see which data sources are actually trusted, how models change, why incidents happen, who approves definitions, how users correct agent outputs, and what happens after a decision is made.</p><p>That said, moving bytes remains necessary. The larger opportunity is helping a company remember what those bytes mean and improve how it uses them.</p><p>There is an important catch for vendors: the feedback loop may belong to the customer.</p><p>A vendor does not gain much of a moat by locking up a customer’s metadata or conversation history. Customers will expect context to move between tools, models, and agents and a proprietary memory layer may create switching costs, but switching costs and a good product are not the same thing.</p><p>The vendor’s own advantage has to come from operating the system well, learning reusable patterns, and fitting deeply into the workflows where decisions and corrections happen.</p><h3>Not a silver bullet</h3><p>To be fair, none of this means an agent can just go and read a company wiki and suddenly understand the business.</p><p>Documentation often has errors, query history contains bad SQL, and even popular dashboards can be misleading and of course teams may disagree for legitimate reasons. In addition to all of those, some knowledge was simply never written down.</p><p>Agents can also form a neat explanation from inconsistent evidence and present it with too much confidence.</p><p>A useful system needs provenance. It should show which sources support a definition, when they were last updated, and who owns them. It should distinguish an approved rule from an inference based on usage. It should be able to say that three definitions exist and that no one has resolved the conflict.</p><p>Permissions matter as well. Giving an agent access to every document and database because it might contain useful context is simply a bad security model. The context layer has to respect the same boundaries as the underlying systems.</p><p>Human approval will remain important around financial reporting, compliance, access changes, and other high-impact work. The agent’s job is often to reduce the amount of searching and reconstruction required before a person can make the call.</p><p>Even with these limits, the economics have drastically changed. A process that once required months of apprenticeship can become much faster when an agent can inspect the company’s technical and operational history, identify contradictions, and ask focused questions.</p><h3>The company has to teach the system</h3><p>For decades, companies have been pushed to “become data-driven”, which usually meant collecting more data, building more dashboards, and (in rare cases) actually making that data available to more people.</p><p>Agents add another requirement on top of this: the company needs to make its reasoning legible.</p><ul><li>Why was this metric changed?</li><li>Who decided that this source is authoritative?</li><li>What evidence caused someone to reject the recommendation?</li><li>Which outcome showed that the decision was good or bad?</li></ul><p>A company that records these things gives its agents a chance to improve, but a company that lets them disappear into meetings and direct messages will keep paying the same institutional knowledge tax.</p><p>This is partly a technical architecture problem, but it is also a big management problem.</p><p>The data team can build great lineage, catalogs, tests, and semantic models. Leadership still has to decide who owns a definition and teams still have to record exceptions. People still have to explain why they overruled the system.</p><p>Workflows have to make useful feedback easy enough that people actually provide it.</p><p>A well-architected data system is almost enough, but it needs a company around it that knows how to make decisions, record what it learns, and feed that knowledge back into the next decision.</p><p>The companies that benefit most will be the ones that give it a clear way to be corrected.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2c1572f48f61" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Real-Time CDC From PostgreSQL to Snowflake, Step by Step]]></title>
            <link>https://medium.com/snowflake/real-time-cdc-from-postgresql-to-snowflake-step-by-step-64237ca54f94?source=rss-f19c56481579------2</link>
            <guid isPermaLink="false">https://medium.com/p/64237ca54f94</guid>
            <category><![CDATA[change-data-capture]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[estuary]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Dani Palma]]></dc:creator>
            <pubDate>Thu, 30 Jul 2026 14:01:05 GMT</pubDate>
            <atom:updated>2026-07-30T14:01:05.715Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A reproducible, step-by-step walkthrough for setting up a real-time CDC pipeline.</em></p><p>Most Postgres-to-Snowflake ETL pipelines still run on a schedule: a job wakes up at 1 AM, queries rows by updated_at, and loads the result into the warehouse. This works for the most part, but it suffers from three main problems that get worse as your data volume grows:</p><ol><li>Dashboards can lag hours behind the business.</li><li>Full or even incremental extracts put real load on the production database at the worst possible time.</li><li>Deletes fail to show up, because a row that no longer exists cannot be selected by the query.</li></ol><p><a href="https://estuary.dev/blog/change-data-capture/">Change data capture</a> is a design pattern that fixes all three at the extraction step. Postgres already writes every change to its <a href="https://www.postgresql.org/docs/current/wal-intro.html">write-ahead log</a>, and logical replication lets you read that log as a stream of row-level events: this row was inserted, this row changed, this row was deleted. Instead of polling the database to see what changed, you listen.</p><p>In this quick walkthrough, we’ll build a complete CDC pipeline from PostgreSQL to Snowflake using Estuary. By the end, a record written to Postgres will be queryable in Snowflake after just a few seconds, updates and deletes included.</p><p>Everything here lives in a <a href="https://github.com/estuary/examples/tree/main/postgres-to-snowflake-cdc">GitHub repo</a>. It includes a Dockerized Postgres, a traffic generator that continuously writes realistic orders, all the SQL, and the full pipeline specs. Feel free to clone it and follow along.</p><h3>How the pipeline works</h3><p>The pipeline has three stages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NtzehksVhJSbtrfW0VIeOw.png" /><figcaption>The CDC pipeline architecture</figcaption></figure><p>First, a <strong>capture</strong> connects to Postgres over logical replication and reads change events from the write-ahead log. When the connector starts, it backfills the current contents of each table and then switches to streaming mode, capturing live changes from the exact log position where the snapshot ended. Nothing is missed in between, this all happens automatically.</p><p>Second, every event lands in an <a href="https://docs.estuary.dev/concepts/collections/">Estuary <strong>collection</strong></a>, a durable, replayable log of JSON documents backed by cloud storage. This is a critical piece in the architecture, as a persistent log that contains every change enables downstream consumers to restart, retry, or be added later without touching Postgres again.</p><p>Third, a <strong>materialization</strong> connector continuously reads these collections and applies changes via custom <a href="https://docs.estuary.dev/reference/reduction-strategies/">reduction strategies</a> to Snowflake tables using transactional merge queries. The materialization commits its progress checkpoint together with the data it writes, and if anything crashes for some reason and retries, it resumes from the last committed checkpoint instead of simply trying to reprocess records. That is how the pipeline achieves exactly-once delivery in practice: no missing rows, no duplicates, even across failures.</p><h3>Prerequisites</h3><p>To build this pipeline, you need four things:</p><ol><li>A <a href="http://snowflake.com/">Snowflake account</a> (a free trial works)</li><li>An <a href="https://dashboard.estuary.dev/register">Estuary account</a> (the free tier covers this entire tutorial)</li><li><a href="https://docs.docker.com/compose/">Docker with Compose</a></li><li>The cloned <a href="https://github.com/estuary/examples/tree/main/postgres-to-snowflake-cdc">repo</a></li></ol><p>You do not need an existing Postgres database, because the repo ships one preloaded with everything CDC needs. But if you have one lying around, feel free to try the rest of the tutorial with that as the source!</p><pre>git clone https://github.com/estuary/examples.git<br>cd examples/postgres-to-snowflake-cdc</pre><h3>Step 1: Start Postgres with live traffic</h3><p>The repo’s compose file starts two small containers: A Postgres db pre-configured for logical replication, and a small Python generator that writes a continuous stream of orders.</p><pre># docker-compose.yml<br>services:<br>  postgres:<br>    image: postgres:16<br>    command: [&quot;postgres&quot;, &quot;-c&quot;, &quot;wal_level=logical&quot;]<br>    environment:<br>      POSTGRES_PASSWORD: postgres<br>    ports:<br>      - &quot;5432:5432&quot;<br>    volumes:<br>      - ./postgres/init.sql:/docker-entrypoint-initdb.d/init.sql<br>  generator:<br>      build: ./generator<br>      environment:<br>        DATABASE_URL: postgresql://postgres:postgres@postgres:5432/postgres<br>        RATE_PER_SECOND: &quot;2&quot;<br>      depends_on:<br>        - postgres</pre><p>One important flag here is setting wal_level=logical. Without this config, Postgres only writes enough to the WAL for crash recovery and physical replication, which is great, but it’s not enough for the logical decoder to use. On a database you manage yourself, changing this config will require a restart, but on managed services it&#39;s a parameter group setting (like with RDS), a database flag (Cloud SQL), or already enabled by default (Supabase, Neon).</p><p>To bring up the containers:</p><pre>docker compose up -d</pre><p>The generator immediately starts working to populate the orders table, which looks like this:</p><pre>create table public.orders (<br>  order_id      uuid primary key default gen_random_uuid(),<br>  customer_name text not null,<br>  amount        numeric(10,2) not null,<br>  status        text not null default &#39;pending&#39;,<br>  event_ts      timestamptz not null default now(),<br>  updated_at    timestamptz not null default now()<br>);</pre><p>Every second (or two), it inserts new orders, updates the status of recent ones (from pending to shipped to delivered), and occasionally deletes one to simulate all possible operations for real change data capture.</p><p>Note that event_ts records when each event happened, which is what lets us measure end-to-end latency later instead of guessing.</p><p>You can verify traffic is flowing by running this command:</p><pre>docker compose exec postgres psql -U postgres \<br>  -c &quot;select count(*), max(event_ts) from public.orders;&quot;</pre><p>Try running it twice a few seconds apart and make sure the count increases.</p><h3>Step 2: CDC prerequisites in the database</h3><p>The repo’s init.sql already prepared Postgres for CDC, but you&#39;ll want to understand each piece before applying this to a real database so let’s quickly go over the scripts:</p><pre>-- A dedicated user with replication rights<br>create user flow_capture with password &#39;secret&#39; replication;<br>grant pg_read_all_data to flow_capture;<br><br>-- A watermarks table the connector uses to coordinate<br>-- precise backfill-to-streaming handoff<br>create table if not exists public.flow_watermarks (<br>  slot text primary key,<br>  watermark text<br>);<br>grant all privileges on table public.flow_watermarks to flow_capture;<br><br>-- A publication: the set of tables whose changes are exposed<br>create publication flow_publication<br>  for table public.orders, public.flow_watermarks;<br>alter publication flow_publication<br>  set (publish_via_partition_root = true);</pre><p>Three ideas here worth discussing:</p><ol><li>The <strong>replication role</strong> lets the connector create and read from a replication slot, which is Postgres’s way of holding your place in the WAL so no changes are discarded before you’ve read them.</li><li>The <strong>publication</strong> is essentially an allowlist so only tables added to it emit changes, enabling you to capture exactly what you intend to.</li><li>The <strong>watermarks table</strong> is for bookkeeping: the connector writes marker values into it during backfill so it can stitch the snapshot and the live stream together without gaps or overlaps.</li></ol><p>Keep in mind, Estuary runs as a managed service, so it needs to reach your database over the network, and a Postgres inside Docker on your laptop is not reachable from the internet by default, but a quick fix for a tutorial is a TCP tunnel with ngrok:</p><pre>ngrok tcp 5432</pre><p>ngrok will print a forwarding address like 4.tcp.ngrok.io:12345, which you&#39;ll use as the database address in the next step. For real databases in private networks, use the built-in SSH tunneling instead or skip the problem entirely by pointing the capture at a cloud Postgres.</p><h3>Step 3: Create the capture</h3><p>In the Estuary dashboard, go to <strong>Sources</strong>, click <strong>New Capture</strong>, and pick the PostgreSQL connector. Fill in the address (your ngrok host and port), database postgres, user flow_capture, and the password from init.sql.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E75zRqVkW7fzJ6mtz7Ca8g.png" /><figcaption>Setting up a Postgres capture in Estuary</figcaption></figure><p>Click <strong>Next</strong>. The connector inspects the publication and lists discoverable tables; either select all or just public.orders, then press <strong>Save and Publish</strong>.</p><p>Note: if you prefer using your AI agent, you can create all of these connectors with <a href="https://estuary.dev/capabilities/agent-skills/">Estuary’s Agent Skills</a>.</p><p>Two things now happen in sequence: the connector backfills every existing row in the orders table, and when the snapshot completes, it switches to streaming live changes from the WAL position where the snapshot ended. Because the generator script is running, you can actually watch documents arrive in the collection in real time to make sure everything works.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gNuSOeu2cQC5xQUPxnaZBA.png" /><figcaption>Estuary Collections on the dashboard</figcaption></figure><p>If you prefer using declarative configs, the demo repo contains the equivalent spec, and flowctl catalog publish --source flow/capture.flow.yaml can be used to create the same resources:</p><pre># flow/capture.flow.yaml<br>captures:<br>  yourprefix/pg-snowflake-demo/source-postgres:<br>    endpoint:<br>      connector:<br>        image: ghcr.io/estuary/source-postgres:dev<br>        config:<br>          address: &quot;4.tcp.ngrok.io:12345&quot;<br>          database: postgres<br>          user: flow_capture<br>          password: secret<br>    bindings:<br>      - resource:<br>          namespace: public<br>          stream: orders<br>        target: yourprefix/pg-snowflake-demo/orders</pre><p>Either way, Postgres changes are now landing in a durable Estuary collection, which means half the pipeline is already done!</p><h3>Step 4: Prepare Snowflake</h3><p>Run the repo’s snowflake/setup.sql in a Snowflake worksheet to create an isolated environment for the pipeline: a new database, a role, a user for Estuary, and a dedicated extra-small warehouse to keep costs minimal.</p><pre>set database_name  = &#39;CDC_DEMO&#39;;<br>set warehouse_name = &#39;ESTUARY_WH&#39;;<br>set estuary_role   = &#39;ESTUARY_ROLE&#39;;<br>set estuary_user   = &#39;ESTUARY_USER&#39;;<br><br>begin;<br>create role if not exists identifier($estuary_role);<br>grant role identifier($estuary_role) to role SYSADMIN;<br><br>create user if not exists identifier($estuary_user)<br>  type = service<br>  default_role = $estuary_role<br>  default_warehouse = $warehouse_name;<br><br>grant role identifier($estuary_role) to user identifier($estuary_user);<br>create warehouse if not exists identifier($warehouse_name)<br>  warehouse_size = xsmall<br>  auto_suspend   = 60<br>  auto_resume    = true<br>  initially_suspended = true;<br>grant usage on warehouse identifier($warehouse_name)<br>  to role identifier($estuary_role);<br>create database if not exists identifier($database_name);<br>grant ownership on database identifier($database_name)<br>  to role identifier($estuary_role);<br>commit;</pre><p>An XS warehouse is more than enough for this demo (and a lot of production use cases!): continuous merges of CDC batches are small, frequent operations, and a bigger warehouse could mostly sit idle at a higher hourly rate. auto_suspend = 60 suspends the warehouse after a minute of inactivity, so between syncs you pay nothing. auto_resume wakes it when the next merge arrives. To tune things even finer, you could look into configuring <a href="https://docs.estuary.dev/reference/materialization-sync-schedule/">Sync Schedule</a> in Estuary.</p><p>For authentication, use key-pair authentication. Because Estuary connects without human interaction, its Snowflake user should be created with TYPE = SERVICE. Service users cannot authenticate with passwords, making key-pair authentication the appropriate option.</p><p>Snowflake is also completing its broader strong-authentication rollout: during Phase 3, scheduled between August and October 2026, existing legacy service users will be migrated to TYPE = SERVICE and blocked from authenticating with passwords.</p><p>You can generate a key pair and attach the public key to the user pretty easily:</p><pre>openssl genrsa 2048 | openssl pkcs8 -topk8 -inform PEM \<br>  -out rsa_key.p8 -nocrypt<br>openssl rsa -in rsa_key.p8 -pubout -out rsa_key.pub</pre><pre>alter user ESTUARY_USER<br>  set rsa_public_key = &#39;&lt;contents of rsa_key.pub, without header lines&gt;&#39;;</pre><p>Keep rsa_key.p8 close by, as you&#39;ll paste it into the materialization config next.</p><h3>Step 5: Create the materialization</h3><p>Back in the Estuary dashboard, go to <strong>Destinations</strong>, click <strong>New Materialization</strong>, and pick the Snowflake connector. Fill in your account’s host URL, database CDC_DEMO, warehouse ESTUARY_WH, role, user, and the private key from the previous step, and finally, bind the orders collection to a target table named orders and publish.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PAh34tLH7wRPF1GwykI_Ug.png" /><figcaption>Configure Snowflake Materialization connector in Estuary</figcaption></figure><p>Two config options are worth highlighting while setting up the materialization connector:</p><h4><strong>Update mode</strong></h4><p>The default option is standard updates, which apply changes with merge queries. An update in Postgres updates the row in Snowflake, and a delete removes it (with hard deletes configured), so the Snowflake table stays an exact mirror of the source. The alternative is delta updates, which appends change events without merging.</p><p>Delta mode is cheaper and lower latency (utilizes Snowpipe Streaming!), and it’s the right choice for append-only event streams or when you plan to reduce the data downstream yourself. For mirroring an operational table, which is what we’re doing in this demo, standard updates are the correct choice, so for now, keep the default.</p><h4><strong>Sync schedule</strong></h4><p>The connector batches changes and commits them to Snowflake at set intervals. The default is 30 minutes, which is okay as a cost-first setting: fewer syncs means fewer warehouse resumes and fewer, larger merges. For this demo, set the sync frequency down to its minimum so you can see changes land fast. This knob is the main lever connecting freshness to Snowflake credit spend.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nxzvHG8D7_MHC5dWRNLe5A.png" /><figcaption>Standard vs Delta updates</figcaption></figure><h3>Step 6: Stream and measure</h3><p>With the mock data generator script still running, open a Snowflake worksheet:</p><pre>select count(*), max(event_ts) from CDC_DEMO.PUBLIC.ORDERS;</pre><p>Execute it, wait for the next sync, run it again. Observe that the count increases and the max timestamp advances further. To expand on this, you can make the latency claim concrete with the freshness query from the repo:</p><pre>select<br>  max(event_ts)                                          as latest_event,<br>  current_timestamp()                                    as checked_at,<br>  datediff(&#39;second&#39;, max(event_ts), current_timestamp()) as lag_seconds<br>from CDC_DEMO.PUBLIC.ORDERS;</pre><p>Because the datagen script writes continuously, lag_seconds approximates true end-to-end latency: the time from a row being written in Postgres to that row being queryable in Snowflake. Because we set sync schedule to the minimum, you can expect this to land in a few seconds.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GIP6hNz5x_D9nNPSdp1tiA.png" /><figcaption>Verifying the CDC pipeline freshness in Snowflake</figcaption></figure><p>Updates and deletes deserve their own proof, since they’re what batch pipelines get wrong. Pick a row and change it at the source:</p><pre>docker compose exec postgres psql -U postgres -c \<br>  &quot;update public.orders set status = &#39;refunded&#39;, updated_at = now()<br>   where order_id = (select order_id from public.orders limit 1);&quot;</pre><p>After the next sync, that row shows refunded in Snowflake. Similarly, deleting a row the same way will make it disappear from the Snowflake table entirely, no soft-delete flags (unless you disable hard deletes), no tombstone cleanup job, as the merge reduction just handles it.</p><h3>Handling schema evolution</h3><p>Schemas will always drift, and a pipeline that breaks because of this is a pipeline you’ll be paged for. Let&#39;s see how Estuary handles this. To simulate, try running this:</p><pre>docker compose exec postgres psql -U postgres -c \<br>  &quot;alter table public.orders add column channel text default &#39;web&#39;;&quot;</pre><p>The capture connector will detect the new column through automatic discovery, then the collection schema evolves automatically. Finally, the materialization connector adds the column to the Snowflake table on a subsequent publication.</p><p>Column additions, the most common kind of schema change, simply flow through without manual intervention. Type changes and column removals are a deeper topic with real tradeoffs, but the platform will always try to automate as much of it as possible. For more details, check out the docs <a href="https://docs.estuary.dev/concepts/advanced/evolutions/">here</a>.</p><h3>Thoughts before moving into production</h3><p>There are three things to consider in this demo before moving into a real deployment.</p><h4>Replication slot lag</h4><p>Make sure to watch the replication slot lag on the source. A replication slot holds WAL segments until the consumer reads them, so a paused or broken capture can cause the WAL to accumulate on disk.</p><p>Monitor pg_replication_slots and alert on restart_lsn falling behind, the same way you&#39;d monitor any replica.</p><h4>Tuning sync schedules</h4><p>Tune Estuary sync schedules based on downstream requirements. The right freshness for each table is a product question, and every step down in sync interval has a warehouse cost attached. Best practice is to start conservative and go faster where the business actually needs it.</p><h4>Snowpipe Streaming</h4><p>For tables where you want both sub-minute freshness and lower cost, look at <a href="https://estuary.dev/blog/snowpipe-streaming-fast-snowflake-ingestion/">Snowpipe Streaming</a> support. It writes rows into Snowflake through the streaming ingest API instead of warehouse-powered merges, which changes the cost profile for high-frequency, append-style workloads.</p><h3>Wrapping up</h3><p>To summarize, we started with an empty Snowflake account and finished with a pipeline where every insert, update, and delete in Postgres appears in Snowflake in a few seconds, delivered with exactly once guarantees, with backfill and live streaming stitched together automatically, and with schema drift handled for the common case.</p><p>The complete demo, including the traffic generator, all specs, and the freshness queries, is at <a href="https://github.com/estuary/examples/tree/main/postgres-to-snowflake-cdc">this URL</a>. <a href="https://estuary.dev/">Estuary’s</a> free tier covers everything in this post, and the Snowflake materialization <a href="https://docs.estuary.dev/reference/Connectors/materialization-connectors/Snowflake/">docs</a> cover the configuration knobs in more depth.</p><p>A natural next step would be to transform this raw CDC data once it lands: <a href="https://docs.estuary.dev/guides/dbt-integration/">dbt</a> or Snowflake’s Dynamic Tables pair well with a continuously updated base layer like this one.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=64237ca54f94" width="1" height="1" alt=""><hr><p><a href="https://medium.com/snowflake/real-time-cdc-from-postgresql-to-snowflake-step-by-step-64237ca54f94">Real-Time CDC From PostgreSQL to Snowflake, Step by Step</a> was originally published in <a href="https://medium.com/snowflake">Snowflake Builders Blog: Data Engineers, App Developers, AI, &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Skill Issue, Part 2: SQL Server to ClickHouse in Real Time]]></title>
            <link>https://blog.dataengineerthings.org/skill-issue-part-2-sql-server-to-clickhouse-in-real-time-882e67a98f99?source=rss-f19c56481579------2</link>
            <guid isPermaLink="false">https://medium.com/p/882e67a98f99</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[change-data-capture]]></category>
            <category><![CDATA[sql-server]]></category>
            <category><![CDATA[estuary]]></category>
            <category><![CDATA[clickhouse]]></category>
            <dc:creator><![CDATA[Dani Palma]]></dc:creator>
            <pubDate>Wed, 22 Jul 2026 16:24:31 GMT</pubDate>
            <atom:updated>2026-07-22T16:24:31.924Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uFkDKQQXXc2cLMmO9vzBQA.png" /></figure><p>After the HubSpot-to-ClickHouse build from the <a href="https://medium.com/data-engineer-things/skill-issue-agentic-real-time-crm-analytics-for-hubspot-with-clickhouse-638dda5372f7">last article</a>, a few people reached out with a variation of the same question: “cool demo, but my source isn’t a SaaS API. My source is a fifteen-year-old SQL Server instance running the business, and my analytics stack is a stored procedure that takes 50 minutes to run every day at 2 AM.”</p><p>Fair. So this article is the sequel. Same methodology (write the agent skill first, then let Claude Code run it), same destination (ClickHouse), but this time the source is Microsoft SQL Server and the framing is a stack modernization. The scenario: an operational SQL Server database that has slowly become the reporting database too. Every dashboard, every SSRS export, every “quick query” from finance runs against the same instance that processes orders. The goal is to peel the analytical workload off SQL Server and move it to ClickHouse, continuously, while SQL Server keeps doing what it’s good at: being the transactional system of record.</p><p>The transport layer is <a href="https://estuary.dev/">Estuary</a> again, using log-based CDC on the SQL Server side and the same <a href="https://docs.estuary.dev/reference/Connectors/materialization-connectors/ClickHouse/">ClickHouse materialization connector</a> from last time. All configs and the new skill are in the same repo.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*huKxW3hC5R1V_qY9BninLA.png" /><figcaption>The migration plan in a diagram</figcaption></figure><h3>The migration shape</h3><p>Worth being explicit about this up front, because it drives every technical decision.</p><p>The classic failure mode of “we’re moving off SQL Server for analytics” is the big-bang rewrite: freeze the reports, bulk-export everything, rebuild, cut over on a weekend, discover on Monday that four reports are subtly wrong and nobody trusts the new stack anymore. I wanted to demo the <a href="https://learn.microsoft.com/en-us/azure/architecture/patterns/strangler-fig">strangler fig</a> instead (and not just because of the funny name):</p><ol><li>Stand up a continuous CDC feed from SQL Server into ClickHouse. No app changes, minimal load on the source.</li><li>Rebuild reports one at a time against ClickHouse while the old ones keep running.</li><li>Validate each rebuilt report against its legacy counterpart, both systems live.</li><li>Retire legacy reports only once they’re proven redundant.</li></ol><p>Step 1 is the only step with infrastructure risk, and it’s fully reversible. If the pipeline breaks, the old reports still work, because you never turned them off. Everything after step 1 is just SQL.</p><p>The demo runs against SQL Server 2019 with a schema you’d recognize from any order system (dbo.orders, dbo.order_lines, dbo.customers, dbo.products), plus a small load generator mutating rows to simulate live traffic. ClickHouse runs local in Docker, same setup as last time, ngrok tunnel and all.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ooE0RvCJBMukDsN89uwDmQ.png" /><figcaption>Strangler fig pattern</figcaption></figure><h3>Picking a capture mechanism (there are three)</h3><p>This was the first thing my skill draft got wrong. Estuary ships three distinct SQL Server capture connectors, and they’re genuinely different mechanisms:</p><ul><li><strong>CDC</strong> (source-sqlserver): log-based, real-time, full row history, works on tables without primary keys. The flagship.</li><li><strong>Change Tracking</strong>: also real-time, lower storage overhead on the source (it stores only primary keys), and it can capture computed columns, which CDC can’t. The catch: it may collapse intermediate changes that happen in rapid succession, so you lose full audit history.</li><li><strong>Batch</strong>: periodic polling, minutes-to-hours latency, but it works against views and arbitrary queries and needs almost no setup. The escape hatch when the instance doesn’t support CDC at all.</li></ul><p>For a modernization focused on real-time analytics over ordinary OLTP tables, CDC is the obvious pick. But I made the skill encode the decision rather than hardcode it. The requirements step now asks: computed columns anywhere? Need to capture views? Full audit history required? The answers route to the right connector before any YAML gets generated.</p><h3>The new skill: capture-sqlserver-create</h3><p><a href="https://github.com/estuary/agent-skills/blob/main/skills/capture-sqlserver-create/SKILL.md">Same anatomy</a> as every connector skill I write:</p><pre>---<br>name: capture-sqlserver-create<br>description: Create a SQL Server CDC capture using flowctl. Covers self-hosted,<br>  Azure SQL Database, Amazon RDS, and Cloud SQL variants. Use when user says<br>  &quot;capture SQL Server&quot;, &quot;stream from SQL Server&quot;, &quot;SQL Server CDC&quot;,<br>  &quot;MSSQL to Estuary&quot;, or &quot;offload SQL Server reporting&quot;.<br>---<br># Create SQL Server CDC Capture<br>...</pre><p>Docs fetch, requirements Q&amp;A, prerequisites, template, publish, verify, troubleshooting. The prerequisites section is where a skill earns its keep with SQL Server specifically, because CDC setup is a database-level ceremony, a table-level ceremony, and a permissions ceremony, and forgetting any one of them produces an error that doesn’t point at the missing piece:</p><pre>USE orders_db;<br><br>-- Enable CDC for the database.<br>EXEC sys.sp_cdc_enable_db;<br>-- Create the capture user.<br>CREATE LOGIN flow_capture WITH PASSWORD = &#39;secret&#39;;<br>CREATE USER flow_capture FOR LOGIN flow_capture;<br>-- Grant read on the data schema and the cdc schema.<br>GRANT SELECT ON SCHEMA :: dbo TO flow_capture;<br>GRANT SELECT ON SCHEMA :: cdc TO flow_capture;<br>-- Required for the connector to read replication state.<br>GRANT VIEW DATABASE STATE TO flow_capture;<br>-- Enable CDC per table. Repeat for every table you capture.<br>EXEC sys.sp_cdc_enable_table<br>  @source_schema = &#39;dbo&#39;,<br>  @source_name   = &#39;orders&#39;,<br>  @role_name     = &#39;flow_capture&#39;;</pre><p>Enabling CDC on a table creates a <em>change table</em> in the cdc schema, and the connector reads from those. This is different from Postgres, where the connector consumes the WAL directly through a replication slot. On SQL Server there&#39;s an intermediary: an agent job scans the transaction log and populates the change tables, and the connector consumes the change tables.</p><p>That intermediary is also my favorite gotcha of the whole build.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EJu7SrNrvTAabSPS64uExg.png" /><figcaption>CDC internals</figcaption></figure><h3>Gotcha #1: the capture works, and no data arrives</h3><p>First end-to-end test: capture published, status green, backfill completed, all four tables landed in ClickHouse. Then my load generator started mutating rows, and… nothing. The backfilled data sat frozen while SQL Server happily processed inserts.</p><p>No errors anywhere. The connector was polling. The change tables existed. They were just empty.</p><p>The cause: on self-hosted SQL Server, the CDC capture job runs under <strong>SQL Server Agent</strong>, and the mssql Docker image ships with the Agent disabled by default. sp_cdc_enable_db succeeds, sp_cdc_enable_table succeeds, everything reports healthy, and the log scanner that actually moves changes into the change tables never runs.</p><p>The fix is one environment variable:</p><pre>services:<br>  sqlserver:<br>    image: mcr.microsoft.com/mssql/server:2019-latest<br>    environment:<br>      ACCEPT_EULA: &quot;Y&quot;<br>      MSSQL_SA_PASSWORD: &quot;...&quot;<br>      MSSQL_AGENT_ENABLED: &quot;true&quot;   # &lt;- without this, CDC silently does nothing</pre><p>Straight into the skill’s troubleshooting section, with the diagnostic that would have saved me twenty minutes:</p><pre>-- If this returns 0 rows, the CDC capture job isn&#39;t running.<br>SELECT * FROM msdb.dbo.cdc_jobs;</pre><p>I like this failure because it demonstrates the skill thesis perfectly. The symptom (“backfill works, live changes don’t”) and the cause (“Agent isn’t running”) are three hops apart, and an improvising agent would start debugging the Estuary side, because that’s the new component. The skill now short-circuits straight to the culprit.</p><h3>Gotcha #2: the three-day countdown</h3><p>Change tables have a default retention of three days, after which a cleanup job removes old events. The consequence: if your capture is down longer than the retention window, unconsumed changes get deleted before they’re read, and the only fix is a full backfill.</p><p>Three days is plenty for a healthy pipeline. It’s a lot less during a migration project where someone pauses the capture on Friday to debug something and picks it back up Wednesday. The skill now surfaces this during requirements gathering and offers two mitigations: extend retention via sys.sp_cdc_change_job if disk allows, or enable the connector&#39;s <strong>Automatic Change Table Cleanup</strong>, where the connector deletes change events only after confirming they&#39;re durably persisted in the Estuary collection. The tradeoff is that this feature requires the capture user to be in db_owner, which is a real conversation with whoever owns the database. For the demo I granted it. In production, expect that to be the longest email thread of the project.</p><h3>Gotcha #3: adding a column is a whole thing</h3><p>Midway through, I simulated the exact event that kills nightly-batch pipelines: a schema change on the source. I added a discount_pct column to dbo.orders while the capture was live.</p><p>SQL Server CDC does something here that surprises everyone coming from Postgres: adding a column to the source table does <strong>not</strong> add it to the change table. The change table’s schema is frozen at the moment sp_cdc_enable_table ran. Microsoft&#39;s sanctioned procedure is to create a <em>second capture instance</em> reflecting the new schema, run both in parallel, transition consumers over, then drop the old one.</p><p>The good news: Estuary’s connector handles the consumer side automatically. It detects the second capture instance and switches over once it reaches a point in the stream where both are valid. With <strong>Automatic Capture Instance Management</strong> enabled (db_owner again), it handles the producer side too, creating the new instance and dropping the old one on its own.</p><p>I tested both modes. With automatic management on, the new column started flowing about a minute after the ALTER TABLE, and schema evolution carried it through the collection into ClickHouse, where the materialization added the column to the destination table. Watching a source-side DDL change arrive in a ClickHouse table without touching anything in between was the moment this demo sold itself to me.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KYC-xQlxGK-9TK7Xza4fWQ.png" /><figcaption>capture instance switchover</figcaption></figure><h3>The ClickHouse side: mostly a rerun</h3><p>The materialize-clickhouse-create skill from last time carried over unchanged, which is the whole point of writing skills: the second use is free. The allow_nullable_key fix, the ngrok setup, the MOVE PARTITION and refreshable materialized view gotchas, all pre-encoded. Claude sailed past each one without me intervening.</p><p>The one new wrinkle was typing. HubSpot documents are JSON all the way down, so last time everything landed in flow_document and I extracted with JSONExtract*. SQL Server tables are strongly typed, so this time I used Estuary&#39;s field selection to project the hot columns as typed ClickHouse columns:</p><pre>bindings:<br>  - source: dani-demo/orders_db/orders<br>    resource:<br>      table: orders<br>    fields:<br>      recommended: true<br>      include:<br>        order_id: {}<br>        customer_id: {}<br>        order_total: {}<br>        status: {}<br>        created_at: {}<br>        updated_at: {}</pre><p>The payoff shows up immediately. The first strangler-fig target was a “daily revenue by category, last 90 days” stored procedure joining four tables on the OLTP box. The ClickHouse rewrite:</p><pre>SELECT<br>    toDate(o.created_at)              AS day,<br>    p.category                        AS category,<br>    sum(ol.quantity * ol.unit_price)  AS revenue,<br>    uniqExact(o.order_id)             AS orders<br>FROM order_lines AS ol FINAL<br>INNER JOIN orders   AS o FINAL ON o.order_id   = ol.order_id<br>INNER JOIN products AS p FINAL ON p.product_id = ol.product_id<br>WHERE assumeNotNull(o.`_meta/op`) != &#39;d&#39;<br>  AND assumeNotNull(ol.`_meta/op`) != &#39;d&#39;<br>  AND o.created_at &gt;= today() - INTERVAL 90 DAY<br>GROUP BY day, category<br>ORDER BY day, category;</pre><p>Two notes. The _meta/op != &#39;d&#39; predicates filter out soft-deleted rows. And FINAL collapses row versions at query time, since updates arrive as new versions in ReplacingMergeTree-style tables. Fine for dashboards; for heavier aggregates you pre-collapse with refreshable materialized views, exactly the pattern from last time.</p><p>Against roughly two million synthetic order lines on my laptop, this returns in well under a second. The stored procedure it replaces was the 2 AM batch job. That delta is the entire business case, in one query.</p><h3>Dual-running: the validation query that buys trust</h3><p>The strangler fig lives or dies on proving the new report matches the old one. Since both systems are live, this is just two queries and a diff. Claude generated a small validation harness: run the legacy T-SQL against SQL Server and the rewrite against ClickHouse, align on the grouping keys, flag rows where the aggregates diverge beyond a float tolerance.</p><p>One honest caveat: it will show small differences at the freshest edge of the data, because ClickHouse is seconds behind SQL Server by construction. The harness excludes the trailing five minutes from the comparison window. Everything older matched exactly across all four reports I migrated, which is the artifact you bring to the meeting where someone asks how we know the new numbers are right.</p><p>Measuring the actual lag is one query, same trick as last time:</p><pre>SELECT quantilesExact(0.5, 0.9, 0.99)(<br>    dateDiff(&#39;second&#39;, updated_at, flow_published_at)<br>) AS p50_p90_p99_s<br>FROM orders FINAL<br>WHERE assumeNotNull(`_meta/op`) != &#39;d&#39;<br>  AND updated_at &gt;= now() - INTERVAL 1 HOUR;</pre><p>p50 landed in the low single-digit seconds, dominated by the CDC scan interval on the SQL Server side rather than anything in Estuary or ClickHouse.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*20GUrv8WQuKkS8-ycVHO4Q.png" /><figcaption>dual-run validation</figcaption></figure><h3>What the skills bought me this time</h3><ul><li>Zero time re-deriving the ClickHouse setup. The materialization skill worked on the first invocation, gotchas pre-avoided.</li><li>One new skill written (about 250 lines), whose test pass found three bugs in my own understanding: the Agent dependency, the db_owner requirement for the automatic-management features, and the fact that connector choice needed to be a requirements question rather than an assumption.</li><li>The DDL runbook, a procedure you execute maybe twice a year and will not remember either time, now lives in a file instead of my browser history.</li></ul><p>Total build time, sans writeup, was again a couple of hours, and the largest single sink was the silent-Agent debugging session that the skill now prevents. Every hour lost once becomes minutes saved forever, as long as it gets written down where the agent actually reads.</p><h3>Why this fits modernization specifically</h3><p>Three properties make CDC-into-ClickHouse the right shape for this, versus a lift-and-shift or a nightly export:</p><p><strong>It’s incremental by construction.</strong> You migrate report by report, and each migration is independently reversible. The riskiest artifact in the whole project is one capture spec.</p><p><strong>It removes load from the thing you’re protecting.</strong> Log-based CDC reads changes without repeated table scans, and the analytical queries that used to compete with order processing now run on a different engine entirely. In a lot of shops this alone justifies the project.</p><p><strong>It doesn’t force a decision about SQL Server’s future.</strong> Maybe the OLTP workload stays there forever, and that’s fine. Maybe it eventually moves too. Either way the analytics layer has stopped caring, because it consumes a change stream, and change streams have no opinions about what’s behind them. Swapping the source later is a new capture spec; the ClickHouse side never notices.</p><h3>What’s next</h3><p>The backlog from last time still stands (refreshable MV patterns, flow_document vs typed-column benchmarks, ClickHouse Cloud differences). This build added a few candidates:</p><ul><li>The Change Tracking connector deserves its own writeup, specifically the storage-vs-audit-fidelity tradeoff.</li><li>A deeper look at history mode, capturing every change event without merging, which turns ClickHouse into a full audit log of the operational database. For compliance-heavy modernizations this might quietly be the killer feature.</li><li>The dual-run validation harness could grow into its own skill: point it at a legacy report and a rewrite, get back a diff. That generalizes well beyond this migration.</li></ul><p>If you’re reproducing this, the repo has the <a href="https://github.com/estuary/agent-skills/blob/main/skills/capture-sqlserver-create/SKILL.md">new skill</a> (skills/capture-sqlserver-create/SKILL.md), the docker-compose with the Agent flag you will otherwise forget, the load generator, and every spec file. And if your modernization source isn&#39;t SQL Server, the skill anatomy is the same either way: docs fetch, requirements Q&amp;A, prerequisites, template, publish, verify, troubleshooting. Write it once, test it against reality, and let the agent do the dance from then on.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=882e67a98f99" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/skill-issue-part-2-sql-server-to-clickhouse-in-real-time-882e67a98f99">Skill Issue, Part 2: SQL Server to ClickHouse in Real Time</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exactly-Once Delivery Is a Spectrum, Not a Checkbox: Part 1]]></title>
            <link>https://medium.com/@danthelion/exactly-once-delivery-is-a-spectrum-not-a-checkbox-part-1-3348d771aa4a?source=rss-f19c56481579------2</link>
            <guid isPermaLink="false">https://medium.com/p/3348d771aa4a</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[kafka]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[streaming]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <dc:creator><![CDATA[Dani Palma]]></dc:creator>
            <pubDate>Wed, 17 Jun 2026 01:34:19 GMT</pubDate>
            <atom:updated>2026-06-17T01:34:19.495Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WrVFowBOOdUha4SJgv7ymA.png" /></figure><p>“Exactly once” is one of the most overloaded phrases in data infrastructure.</p><ul><li>A source can say it emits every change once.</li><li>Kafka can claim to support exactly-once semantics.</li><li>Flink can say it checkpoints state exactly once.</li><li>BigQuery and Snowflake can say they support exactly-once streaming writes.</li><li>A lakehouse table can say commits are atomic.</li></ul><p>All of those statements can be true, but your dashboard can still double-count revenue.</p><p>The reason is simple: <strong>exactly once is not one guarantee. It is a chain of guarantees.</strong> The chain crosses the source database, the capture connector, the transport layer, the stream processor, the destination connector, the destination storage engine, and finally the business operation you care about.</p><p>This series of articles provides a practical model for exploring exactly-once guarantees and how much failure they tolerate.</p><p>We will start with the first two links in the chain: source guarantees and transport guarantees. Then we will move into stream processing, where input progress and processing state have to recover together. From there, we will get to the place where correctness becomes truly visible: materialization into a table, file, index, lakehouse, warehouse, or external API.</p><h3>The duplicate no one owns</h3><p>Let’s kick off with a common pipeline:</p><p>Postgres -&gt; CDC connector -&gt; Kafka -&gt; stream job -&gt; warehouse</p><p>Now imagine a customer places an order:</p><pre>INSERT INTO orders (id, customer_id, amount, status)<br>VALUES (123, 42, 100.00, &#39;paid&#39;);</pre><p>The source database commits the transaction, then the CDC connector reads the write-ahead log, and Kafka accepts the record. On the other side, a worker reads it and writes it to a warehouse table.</p><p>Then this happens:</p><pre>t1 worker reads order_id = 123<br>t2 worker writes order_id = 123 to destination<br>t3 destination commit succeeds<br>t4 worker crashes before saving its checkpoint<br>t5 worker restarts from old checkpoint<br>t6 worker reads order_id = 123 again<br>t7 worker writes order_id = 123 again</pre><p>Do you see the issue? No component “lost” data or necessarily violated its own contract. But, the source replayed from a safe position and the worker retried after a crash. After all that, the destination accepted a valid write.</p><p>But the business effect happened twice, so if the destination table is append-only, you now have this:</p><pre>order_id | amount | source_lsn<br> - - - - -+ - - - - + - - - - - -<br>123 | 100.00 | 0/16B6C50<br>123 | 100.00 | 0/16B6C50</pre><p>If a dashboard runs this:</p><pre>SELECT SUM(amount) AS revenue<br>FROM orders;</pre><p>the answer is obviously wrong.</p><p>This sneaky bug lives in the gap between <strong>delivery</strong> and <strong>effect</strong>.</p><ul><li>Delivery asks: did the event arrive?</li><li>Effect asks: did the durable state change exactly once?</li></ul><p>Those are very different questions.</p><h3>Exactly-once is scoped</h3><p>Every “exactly-once” claim has a scope.</p><p>Kafka’s idempotent <a href="https://kafka.apache.org/41/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html">producer</a> prevents producer retries from creating duplicate log entries in a Kafka partition, and Kafka transactions can atomically send records to multiple partitions and topics. But Kafka’s own producer documentation also says application-level resends cannot be deduplicated by producer idempotence, and idempotence is guaranteed only within a producer session.</p><p>Postgres <a href="https://www.postgresql.org/docs/current/logicaldecoding-explanation.html">logical replication</a> slots normally emit each change once, but after a crash a slot can return to an earlier LSN and resend recent changes. The Postgres documentation explicitly says logical decoding clients are responsible for avoiding bad effects from handling the same message more than once.</p><p>BigQuery <a href="https://cloud.google.com/bigquery/docs/write-api">Storage Write API</a> supports exactly-once semantics through stream offsets, but that guarantee is <strong>scoped</strong> to messages with the same offset within a write stream when the client provides offsets. The default stream is at-least-once.</p><p>Snowpipe Streaming uses <a href="https://docs.snowflake.com/en/user-guide/snowpipe-streaming/snowpipe-streaming-channels">channels</a> and offset tokens to support recovery without duplicate ingestion, but its docs also make clear that offset tokens are user-defined identifiers and the external system must use them to track progress correctly.</p><p>Those might seem like contradictions, but they are, in fact, scoped guarantees.</p><p>A useful pipeline model looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1EZbctWjC8o7L_i4JuI8wA.png" /></figure><p>“Exactly once” can apply at any of those boundaries, but keep in mind that it rarely applies to all of them automatically.</p><h3>The spectrum</h3><p>Think of delivery guarantees as a spectrum.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2TMdQ9gSopN_CHyPedU9PQ.png" /></figure><p>A pipeline with at-least-once transport and a truly idempotent destination write can be more correct than a pipeline with Kafka transactions and a non-idempotent warehouse append.</p><p>The weakest link will define the actual guarantee.</p><h3>Five guarantees hiding behind one phrase</h3><p>When someone says “exactly once,” ask which of these five guarantees they mean.</p><ol><li>Source guarantee: Can the source replay from a precise position?</li><li>Transport guarantee: Can the stream/broker commit records and offsets atomically?</li><li>Processing guarantee: Can state and input progress recover together?</li><li>Materialization guarantee: Can destination data and destination progress commit together?</li><li>Business effect guarantee: Is the operation safe to retry?</li></ol><p>Most duplicate bugs happen because developers verify one layer and assume the rest.</p><ul><li>They check Kafka exactly-once.</li><li>They forget the warehouse MERGE.</li><li>They check Flink checkpointing.</li><li>They forget the HTTP API side effect.</li><li>They check CDC source offsets.</li><li>They forget that SUM(amount) is not idempotent.</li></ul><h3>Source guarantees: a replay position is not a miracle</h3><p>A source system usually gives you a <strong>position</strong>, but it does not give you end-to-end correctness.</p><p>The position just tells you that you can resume from here. It does <em>not</em> say that “every downstream effect is now exactly once”. Those are different contracts.</p><h4>Postgres: LSNs and replication slots</h4><p><a href="https://www.postgresql.org/docs/current/logicaldecoding-explanation.html">Postgres</a> logical decoding reads committed changes from the write-ahead log and exposes them through logical replication slots. A slot represents a stream of changes from a single database, persists independently of the client connection, and is crash-safe. In normal operation, a logical slot emits each change once. But Postgres persists a slot’s current position only at checkpoint time. After a crash, the slot can return to an earlier LSN, causing recent changes to be sent again.</p><p>That is a fairly honest guarantee.</p><p>Postgres is telling you a few things:</p><ul><li>“I can give you an ordered change stream.”</li><li>“I can keep WAL needed by the slot.”</li><li>“I can let you resume.”</li><li>“But if I resend a recent message, your client must handle it.”</li></ul><p>A CDC event from Postgres should carry source identity:</p><pre>{<br>&quot;source&quot;: &quot;postgres&quot;,<br>&quot;database&quot;: &quot;app&quot;,<br>&quot;schema&quot;: &quot;public&quot;,<br>&quot;table&quot;: &quot;orders&quot;,<br>&quot;primary_key&quot;: { &quot;id&quot;: 123 },<br>&quot;op&quot;: &quot;insert&quot;,<br>&quot;commit_lsn&quot;: &quot;0/16B6C50&quot;,<br>&quot;transaction_id&quot;: 742991,<br>&quot;transaction_order&quot;: 3<br>}</pre><p>A destination should not treat this merely as “an order row.” It should treat it as a specific source change.</p><p>For many CDC systems, a practical event identity is something like:</p><pre>event_id =<br>source_cluster +<br>database +<br>schema +<br>table +<br>commit_position +<br>transaction_position +<br>row_position</pre><p>The exact fields depend on the source connector, but the principle is stable: carry the source position all the way to the sink.</p><h4>MySQL: GTIDs protect replication, not every downstream system</h4><p>MySQL Global Transaction Identifiers are source transaction identifiers. A GTID is unique across a replication topology and has the form:</p><pre>GTID = source_id:transaction_id</pre><p>MySQL <a href="https://dev.mysql.com/doc/refman/8.4/en/replication-gtids-concepts.html">uses</a> GTIDs to prevent the same transaction from being applied more than once on a replica. After a transaction with a given GTID has committed on a server, a later attempt to execute the same GTID is ignored by that server.</p><p>That is powerful <em>inside</em> MySQL replication.</p><p>But if you turn the transaction into a JSON event and send it to a warehouse, MySQL’s replica auto-skip logic no longer protects you because the warehouse does not know the GTID unless you write it there. It cannot skip a duplicate event unless the materialization logic uses the GTID, or some derived event identity, as part of its idempotency strategy.</p><p>The same event can be safe in one context and unsafe in another:</p><ol><li>MySQL source -&gt; MySQL replica: GTID auto-skip can prevent duplicate application.</li><li>MySQL source -&gt; Kafka -&gt; Snowflake: GTID is just metadata unless the sink uses it.</li><li>MySQL source -&gt; append-only S3 files: GTID does not dedupe files unless the writer/table format records it.</li></ol><p>The important learning here is that source identity must survive translation.</p><h4>MongoDB: resume tokens are restart handles</h4><p>MongoDB <a href="https://www.mongodb.com/docs/manual/changeStreams/">change streams</a> are resumable with resume tokens. The token can be passed to resumeAfter or startAfter when opening a new cursor. MongoDB also helpfully warns that the same pipeline and options should be used when resuming, and the oplog must still contain enough history for the token or timestamp.</p><p>You might have guessed, that gives you a restart mechanism, but it does not automatically make every downstream write exactly once.</p><p>A robust MongoDB CDC event should carry at least:</p><pre>{<br>&quot;resume_token&quot;: &quot;…&quot;,<br>&quot;operationType&quot;: &quot;update&quot;,<br>&quot;clusterTime&quot;: &quot;Timestamp(…)&quot;,<br>&quot;ns&quot;: {<br>&quot;db&quot;: &quot;app&quot;,<br>&quot;coll&quot;: &quot;customers&quot;<br>},<br>&quot;documentKey&quot;: {<br>&quot;_id&quot;: &quot;…&quot;<br>}<br>}</pre><p>The resume token helps the capture process know where to restart and the document key helps the destination know which entity to update, but they are not the same identity.</p><p>This is an important distinction, because two updates to the same MongoDB document have the same document key but different event identities. If you dedupe only by document key, you may drop a real update. If you append both events without event identity, you may double-count downstream aggregates.</p><h3>Debezium: at-least-once is the honest default</h3><p>Debezium’s <a href="https://debezium.io/documentation/reference/stable/configuration/eos.html">documentation</a> is refreshingly direct: Debezium provides at-least-once delivery guarantees, which means no changes are missed, but a record may be delivered more than once. It also notes that exactly-once may be required in scenarios where duplicates are problematic.</p><p>This is not really a flaw. It is the default shape of reliable CDC.</p><ol><li>A CDC connector that never retries can lose data.</li><li>A CDC connector that retries can duplicate data.</li><li>A CDC connector that retries and carries stable source offsets gives the downstream system enough information to be correct.</li></ol><p>The destination still has to use that information.</p><h3>Source rule</h3><p>A source guarantee usually gives you three things:</p><ol><li>Ordering</li><li>Replay</li><li>Position</li></ol><p>It does not automatically give you:</p><ol><li>Destination deduplication</li><li>Aggregate correctness</li><li>API idempotency</li><li>Cross-system atomicity</li></ol><p>A replayable source is necessary for a robust system in all environments.</p><h3>Transport guarantees: Kafka exactly-once is real, but bounded</h3><p>Kafka is often where exactly-once discussions get confused. Make no mistake, Kafka’s exactly-once features are real, but they are also scoped. Let’s take a look at that deeper.</p><p>Kafka <a href="https://kafka.apache.org/41/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html">producer idempotence</a> prevents producer retries from creating duplicate entries in the Kafka log. Kafka transactions allow messages to multiple partitions and topics to be committed atomically. With transactional producers, consumers also need to read only committed messages for end-to-end transactional visibility inside Kafka.</p><p>Confluent’s <a href="https://docs.confluent.io/kafka/design/delivery-semantics.html">documentation</a> explains the boundary clearly: Kafka supports exactly-once delivery in Kafka Streams and for transferring and processing data between Kafka topics. It also warns that claims of exactly-once semantics may not account for failures of producers or consumers outside the system.</p><p>That boundary is everything.</p><p>Kafka can make this loop exactly-once:</p><pre>Kafka topic A -&gt; process -&gt; Kafka topic B</pre><p>It cannot, by itself, make this external side effect exactly-once:</p><pre>Kafka topic A -&gt; process -&gt; charge credit card</pre><p>or this:</p><pre>Kafka topic A -&gt; process -&gt; UPDATE warehouse table</pre><p>or this:</p><pre>Kafka topic A -&gt; process -&gt; call webhook</pre><p>External systems need their own commit or idempotency protocol.</p><h3>Kafka consume-transform-produce</h3><p>A common exactly-once Kafka pattern looks like this:</p><pre>producer.initTransactions();<br><br>while (true) {<br>    ConsumerRecords&lt;K, V&gt; records = consumer.poll(Duration.ofMillis(1000));<br><br>    producer.beginTransaction();<br><br>    for (ConsumerRecord&lt;K, V&gt; record : records) {<br>        Output output = transform(record);<br>        producer.send(new ProducerRecord&lt;&gt;(&quot;output-topic&quot;, output.key(), output.value()));<br>    }<br><br>    producer.sendOffsetsToTransaction(<br>        offsetsFor(records),<br>        consumer.groupMetadata()<br>    );<br><br>    producer.commitTransaction();<br>}</pre><p>This works fine because output records and consumed offsets are committed as one Kafka transaction, but if the transaction aborts, the output records are not visible to read_committed consumers, and the consumer offset does not advance.</p><p>The atomic unit is:</p><pre>input offsets + output Kafka records</pre><p>Strong contract, but what happens if we change the code?</p><pre>producer.initTransactions();<br><br>while (true) {<br>    ConsumerRecords&lt;K, V&gt; records = consumer.poll(Duration.ofMillis(1000));<br><br>    producer.beginTransaction();<br><br>    for (ConsumerRecord&lt;K, V&gt; record : records) {<br>        warehouse.insert(transform(record));   // external side effect<br>        producer.send(new ProducerRecord&lt;&gt;(&quot;audit-topic&quot;, record.key(), &quot;done&quot;));<br>    }<br><br>    producer.sendOffsetsToTransaction(<br>        offsetsFor(records),<br>        consumer.groupMetadata()<br>    );<br><br>    producer.commitTransaction();<br>}</pre><p>The warehouse insert is not part of the Kafka transaction, meaning if warehouse.insert() succeeds and producer.commitTransaction() fails, the Kafka offset rolls back but the warehouse row remains. On retry, the same input record is processed again.</p><p>In the end, Kafka did its job, but the application did not establish an atomic contract with the warehouse.</p><h3>Kafka Connect source exactly-once is also a contract</h3><p>Kafka Connect <a href="https://kafka.apache.org/41/kafka-connect/connector-development-guide/">added</a> exactly-once support for source connectors in Kafka 3.3. To use it, a source connector must provide meaningful source offsets for each record and must be able to resume from the external system at the exact position corresponding to those offsets without dropping or duplicating messages.</p><p>The worker <a href="https://kafka.apache.org/41/configuration/kafka-connect-configs/">config</a> exactly.once.source.support enables this by using transactions to write source records and their source offsets together, and by fencing old task generations.</p><p>That is important, but notice the shape of the guarantee:</p><p>external source -&gt; Kafka topic</p><p>It helps source connectors write into Kafka exactly once, but it does not automatically solve:</p><p>Kafka topic -&gt; external destination</p><p>For sink connectors, the hard problem remains the same: can the destination write and the consumed offset be committed together, or can the destination apply be made idempotent?</p><p>This conundrum appears in Kafka Connect’s connector <a href="https://kafka.apache.org/41/kafka-connect/connector-development-guide/">development guide</a>: sink tasks can store offset information in the destination system to provide exactly-once delivery.</p><p>And that is the materialization problem, which we’ll explore in a future chapter.</p><h3>Transport rule</h3><p>So Kafka’s exactly-once guarantees are not magic. It can protect Kafka records, Kafka offsets, Kafka transactions, and Kafka Streams state. It cannot automatically protect an external database, warehouse, object store, API, or search index.</p><p>So always keep this boundary in mind:</p><ul><li>Inside Kafka: transactions can compose records and offsets.</li><li>Outside Kafka: the sink needs transactions, idempotency, or reconciliation.</li></ul><h3>In the next chapter</h3><p>So far, we have looked at the first two boundaries: the source and the transport layer.</p><p>The source can give you a replayable position: an LSN, GTID, resume token, binlog offset, or Kafka offset. The transport layer can give you stronger delivery semantics: idempotent producers, transactions, committed offsets, and fencing.</p><p>But that still does not prove the pipeline is correct.</p><p>The next boundary is the stream processor.</p><p>This is where things get a little more subtle. A processor does not just move records. It also keeps state: counts, windows, joins, dedupe sets, latest versions, and intermediate results. If that state is not checkpointed with the input position, the pipeline can become wrong before the destination ever sees a write.</p><p>In the next chapter, we will look at processing guarantees: how state and input progress recover together, why Flink checkpoints are a useful model, and why exactly-once processing still does not mean exactly-once effects unless the sink participates in the same commit protocol.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3348d771aa4a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Are the Best Software Engineers Really Tokenmaxxing?]]></title>
            <link>https://medium.com/@danthelion/are-the-best-software-engineers-really-tokenmaxxing-ad7ffba13854?source=rss-f19c56481579------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad7ffba13854</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[tokenmaxxing]]></category>
            <dc:creator><![CDATA[Dani Palma]]></dc:creator>
            <pubDate>Thu, 04 Jun 2026 14:28:23 GMT</pubDate>
            <atom:updated>2026-06-04T14:28:23.474Z</atom:updated>
            <content:encoded><![CDATA[<p>Probably, but that is not the compliment people think it is. Let&#39;s take a look at the numbers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8fhaiR8fUyB9ALLajag5VA.png" /></figure><p>The best software engineers may spend more AI tokens simply because they use AI on harder work. They usually give the model more context, ask it to explore larger changes, generate more tests, review code, compare approaches, and check a ton of edge cases. That kind of work naturally uses more tokens than asking for a small helper function.</p><p>But higher token usage does not necessarily prove higher engineering skill, in fact, it may actually indicate the opposite. It may show that the engineer, the tool, or the company has created a very expensive way to avoid thinking clearly.</p><p>To say the least, the evidence is mixed.</p><p>Take this <a href="https://www.microsoft.com/en-us/research/publication/the-impact-of-ai-on-developer-productivity-evidence-from-github-copilot/">GitHub Copilot study from Microsoft Research</a>, it found that developers completed a JavaScript task 55.8% faster with Copilot. That seems like real evidence that AI can help when the task is clear, bounded, and easy to verify. Makes sense, right?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z1SVojkYwzkpQJJxxEF8vA.png" /></figure><p>But that is not the same as working inside a mature production codebase, and we’ll see a few examples of that below.</p><p>Another study, from <a href="https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/">METR, of experienced open-source developers</a> found that AI tools made developers actually 19% <strong>slower</strong> on real tasks in repositories they already knew well. The developers expected AI to speed them up, but afterward, they still felt like AI had helped. The measured result showed the opposite.</p><p>That’s something I personally felt as well before: AI often makes work feel faster before it makes the work actually faster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RXDEr911hRhbNyOe.png" /></figure><p>To make things more confusing, METR later published a <a href="https://metr.org/blog/2026-02-24-uplift-update/">follow-up on its experiment design</a>. It said the 2025 slowdown result should not be treated as the final word. <strong>By late 2025 and early 2026, more developers did not want to participate in studies where half their tasks had to be done without AI.</strong> METR also found that 30% to 50% of developers said they avoided submitting some tasks because they did not want to risk doing them without AI. So the study may have missed the developers and tasks where AI helps the most.</p><p>Yikes — have we actually just started straight up refusing to work without AI?</p><p>That makes the real answer even messier. AI may now be helping more than the early 2025 study suggested, but the measurement problem is at the same time getting worse and worse. The people who get the most value from AI are increasingly the hardest people to measure, because they do not want to go back to working without it.</p><p>A separate <a href="https://arxiv.org/abs/2509.20353">Copilot study</a> ran into a very similar measurement problem. Copilot users were more active than non-users even before adoption, but the study did not find statistically significant changes in commit-based activity after they started using Copilot. So in other words, the people using AI more may already be the more active engineers, but usage alone does not tell us what AI caused.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8sMw1VzASjeTAvZ8LgbKaw.png" /></figure><p>The token-specific research is seemingly even more important.</p><p>A study from this year, <a href="https://arxiv.org/abs/2604.22750">How Do AI Agents Spend Your Money?</a>, tells us that agentic coding tasks can consume <strong>1000x more tokens</strong> than code chat or code reasoning. It also found that runs on the same task can vary by up to 30x (!!!) in total token usage. Most importantly, higher token usage did not translate into higher accuracy: <strong>Accuracy often peaked at an intermediate level of spend.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_H2ch4Lf7cZlZxzu5ystXg.png" /></figure><p>If you’re a tokenmaxxing engineering leader, you should feel a little bit nervous at this point.</p><p>Token spend is a cost metric, not a productivity metric, but yes, sometimes it reflects valuable work. Sometimes though, it reflects confusion, retries, bad context, poor tool design, or a “dumb” agent wandering around a codebase burning money.</p><p>There is also a skill issue.</p><p><a href="https://www.anthropic.com/research/AI-assistance-coding-skills">Anthropic’s study on AI assistance and coding skills</a> found that developers retained more when they used AI to “build understanding”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p5nrBAdraXZpXRT9IIhFDQ.png" /></figure><p>The stronger users asked follow-up questions, requested explanations, and stayed involved in the work. The weaker pattern was <strong>full delegation</strong>: hand the task to the model, get code back, and move on without understanding it.</p><p>And that part makes me a bit skeptical.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/0*nJL18jtQsUm5vNkf.gif" /></figure><p>A lot of “AI-native engineering” is just a rebrand of not reading the code. It looks productive in the short term because there is more movement: <strong>more prompts, more generated diffs, more tool activity, more tokens</strong>. But software engineering is measured by scalable, robust systems that work, not by the amount of motion required to produce it.</p><p>That said, I’m not sure that token usage will become the new lines of code. It might be easy to measure, easy to gamify, and mostly is actually useless as a signal of engineering quality. But at the same time, you <strong>do </strong>need more code for more software.</p><p>High token usage may correlate with strong engineers in some cases because strong engineers are using AI on bigger problems. But it will also correlate with weak engineering habits: vague prompts, surface level reviews, repeated retries, and outsourcing judgment to a system that does not own the consequences.</p><p>The best engineers are not tokenmaxxing, they are leverage-maxxing.</p><p>They are using AI for the parts where it truly helps: context gathering, first drafts, tests, refactors, migration plans, alternative implementations, and code review. With that, you also need to know when to stop prompting and start reading, debugging, or designing.</p><p>So no, we’re not closer to being able to properly measure engineering output sadly, but we definitely know that the right metric is <strong>not</strong> tokens per engineer. At least we have a new number middle managers can misunderstand - great.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad7ffba13854" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Skill Issue: Agentic Real-time CRM Analytics for HubSpot with ClickHouse]]></title>
            <link>https://blog.dataengineerthings.org/skill-issue-agentic-real-time-crm-analytics-for-hubspot-with-clickhouse-638dda5372f7?source=rss-f19c56481579------2</link>
            <guid isPermaLink="false">https://medium.com/p/638dda5372f7</guid>
            <category><![CDATA[hubspot]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Dani Palma]]></dc:creator>
            <pubDate>Tue, 02 Jun 2026 14:46:25 GMT</pubDate>
            <atom:updated>2026-07-10T21:45:08.331Z</atom:updated>
            <content:encoded><![CDATA[<p>Inspired by attending <a href="https://clickhouse.com/openhouse/san-francisco">OpenHouse</a> in SF earlier this week, I gave myself a challenge: can I build a production-grade, real-time, end-to-end analytics pipeline to analyze CRM data purely with agents, and if yes, how long will it take? Spoiler: it is definitely possible, and takes surprisingly not long at all.</p><p>So, this article contains my thought process and the steps I took in order to implement the pipeline. I had given myself a half-day to put together a “CRM data in ClickHouse, in real time” demo. The pipeline itself is well-trodden territory: pull from HubSpot’s API, drop the docs into a streaming substrate (<a href="https://estuary.dev/">Estuary</a>), materialize into <a href="https://clickhouse.com/">ClickHouse</a> as columnar tables, query with FINAL. The interesting question was <em>how</em> to build it: specifically, how much of the procedural setup work I could offload to Claude Code without losing the ability to debug the parts that went sideways.</p><p>This post is about that build. It’s also, in a real way, about how I now prefer to do “agentic data engineering”: by writing <strong>agent skills</strong> first, then letting the agent run them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TmQvXy8f8YpSxvULKt4GSA.gif" /><figcaption>Real-time CRM analytics dashboard powered Estuary &amp; ClickHouse</figcaption></figure><blockquote>You can find all config &amp; code I walk through in <a href="https://github.com/danthelion/hubspot-clickhouse-estuary">this repo</a>.</blockquote><h3>The two skills that did the heavy lifting</h3><p>I wanted to keep the architecture as lean as possible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EVhhOtbX-rCZ4txgAc3LeQ.png" /><figcaption>real-time architecture</figcaption></figure><p>Estuary has more than 100 capture connectors and more than 30 materialization connectors. The docs for each one cover the <em>what</em> (config reference, scopes, prerequisites) but not the <em>how</em> (flowctl commands, troubleshooting, the procedural sequence of clicks you actually have to do). Every time I’d onboard a new connector, I’d re-derive the same dance: query the registry for the latest tag, generate the YAML, discover, publish, watch logs, fix anything that might haven broken.</p><p>Fully embracing the “agentic era” and not wanting to deal with “manual” steps anymore, I decided that before touching the pipeline, I’ll write two skills:</p><ol><li><strong>capture-hubspot-create</strong>: set up a HubSpot Real-Time capture from scratch</li><li><strong>materialize-clickhouse-create</strong>: materialize Estuary collections into ClickHouse</li></ol><p>A skill is a markdown file under skills/&lt;name&gt;/SKILL.md with some frontmatter and a step-by-step playbook in the body:</p><pre>- -<br>name: capture-hubspot-create<br>description: Create a HubSpot Real-Time capture using flowctl. Use when setting<br>up a HubSpot source for contacts, companies, deals, tickets, or other<br>HubSpot resources. Use when user says &quot;capture HubSpot&quot;, &quot;stream from<br>HubSpot&quot;, &quot;HubSpot CDC&quot;, or &quot;connect HubSpot to Estuary&quot;.<br> - -<br># Create HubSpot (Real-Time) Capture<br>… </pre><p>The description field is doing two jobs. Claude Code (or the agent you use) uses it to know <strong>when to invoke the skill</strong>. If I just say “hey, set up a HubSpot capture into the dani-demo tenant,” Claude matches my phrasing against the skill descriptions in scope, sees the keyword overlap, and loads capture-hubspot-create before doing anything. I can also explicitly invoke it with <strong>/capture-hubspot-create</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TFbVOB3Nea7sFAtEJpSTew.png" /><figcaption>Claude finding the relevant skill</figcaption></figure><p>The body of the skill is <em>not</em> a prompt I’m reading to Claude, it’s more of a runbook Claude reads and executes step-by-step. The same content that a human SRE would follow during onboarding, just hardened against the agent’s failure modes (don’t guess connector tags, always pull from the registry; warn about gotchas before the user hits them; etc.).</p><h3>Anatomy of a skill</h3><p>Every skill I write has the same shape:</p><ol><li><strong>Step 0: Load Connector Documentation.</strong> A WebFetch of the canonical Estuary docs page. This pulls in the up-to-date prerequisites list, config reference, and any provider-specific setup the docs cover, so the skill doesn’t have to mirror that content (and become stale).</li><li><strong>Step 1: Gather Requirements.</strong> A literal Q&amp;A list: deployment type, auth method, hard-deletes, delta updates, sync schedule. The skill tells Claude to <em>ask</em> the user before generating YAML.</li><li><strong>Step 2: Help User Complete Prerequisites.</strong> The skill carries the exact SQL grants / IAM role bindings/scope lists that the connector actually needs. This is the bit that’s most painful to look up under pressure.</li><li><strong>Step 3: Create the Spec File.</strong> A YAML template with placeholders.</li><li><strong>Step 4: Publish.</strong></li><li><strong>Step 5: Verify.</strong> Status polling, log filtering, and expected progression.</li><li><strong>Troubleshooting.</strong> Every failure I’ve ever hit, with the error string and the fix. This is where the real value compounds over time.</li></ol><p>The HubSpot capture skill is about 200 lines; the ClickHouse materialization skill is about 230. Neither is doing anything magic; they’re pretty much just executable documentation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OD8swTTEo9A_NXWiU1ToKA.png" /><figcaption>HubSpot and ClickHouse skills side-by-side</figcaption></figure><h3>Why is this better than just prompting</h3><p>The naive alternative is to skip the skill and just say “Claude, set up a HubSpot capture into ClickHouse.” Claude will try. It’ll probably:</p><ul><li>Guess a connector image tag from memory (might be stale)</li><li>Generate a YAML that almost works but is missing some required field the docs don’t lead with</li><li>Skip the prerequisite section because the user didn’t ask about prereqs</li><li>Cheerfully publish without showing you the status, then ask “should I check on it?”</li></ul><p>The skill turns all of that into determinism. Same task, same order, same error-handling, every time. Better still: I tested both skills by actually <em>using them</em>, and that immediately surfaced a bunch of bugs in my first drafts:</p><p>What my testing found in capture-hubspot-create:</p><ol><li><strong>Wrong </strong><strong>documentation_url for the registry query.</strong> I’d guessed <a href="https://go.estuary.dev/source-hubspot-native.">https://go.estuary.dev/source-hubspot-native.</a> The actual URL is <a href="https://go.estuary.dev/hubspot-real-time.">https://go.estuary.dev/hubspot-real-time.</a> Skipping this means the flowctl raw get returns zero rows and Claude is left improvising.</li><li><strong>Missing entire auth path.</strong> The HubSpot connector spec exposes two discriminated credential schemas (BaseOAuth2Credentials <em>and</em> AccessToken for HubSpot Private Apps). I’d only documented the OAuth one. For a CLI-only workflow, Private App is much simpler (no OAuth callback round-trip).</li><li><strong>Wrong default for the per-binding schedule cron.</strong> The docs page claims the default is 55 23 * * *. The connector’s actual JSONSchema default is “” (disabled). The docs show the web UI’s pre-fill, not the spec’s default.</li></ol><p>I caught all three by running flowctl raw spec --source flow.yaml --output yaml and diffing what the connector actually accepts against what my skill assumed.</p><p>This shows that Skills are testable in a way that prompts aren’t: they’re files, you commit them, you iterate on them, you measure them against reality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F9xK9RdaKriQcz7G0OIojg.png" /><figcaption>Claude expanding its own skills</figcaption></figure><h3>Building the actual pipeline</h3><p>Once the skills were in place, the build went fast. <em>Very fast</em>.</p><h4>HubSpot side</h4><p>I needed an OAuth app. HubSpot’s new developer platform does this entirely from the CLI, which I LOVE as I’m only using agents from the CLI. The whole app definition lives in src/app/app-hsmeta.json, and you deploy with hs project upload.</p><pre>mkdir -p ~/Work/hs-demo-app &amp;&amp; cd ~/Work/hs-demo-app<br>hs project create \<br> - name estuary-demo-app \<br> - dest . \<br> - platform-version 2026.03 \<br> - project-base app \<br> - distribution private \<br> - auth oauth \<br> - features &#39;[]&#39;</pre><p>I edited the generated app-hsmeta.json to set redirectUrls to <a href="http://localhost:16963/">http://localhost:16963/</a> (the port my OAuth helper would listen on) and filled in the read scopes Estuary needs:</p><pre>&quot;requiredScopes&quot;: [<br>&quot;oauth&quot;,<br>&quot;crm.lists.read&quot;, &quot;crm.objects.companies.read&quot;, &quot;crm.objects.contacts.read&quot;,<br>&quot;crm.objects.deals.read&quot;, &quot;crm.objects.owners.read&quot;,<br>&quot;crm.schemas.companies.read&quot;, &quot;crm.schemas.contacts.read&quot;, &quot;crm.schemas.deals.read&quot;,<br>&quot;e-commerce&quot;, &quot;forms&quot;, &quot;tickets&quot;<br>]</pre><p>hs project upload --account 51535575 --forceCreate deployed build #1. hs project open got me to the Auth tab in the UI where I copied the client_id and client_secret, <strong>the only step in the whole flow that required the browser</strong> for me (more on this in a minute).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*S576V_7HW0adcCagsFYkBA.png" /><figcaption>Fresh HubSpot app without ever touching their UI</figcaption></figure><h4>The OAuth dance</h4><p>Every developer’s favourite, right? HubSpot’s OAuth is Authorization Code, meaning a user <em>has</em> to consent in a browser at least once to mint a refresh_token. To quickly get over this hurdle, my buddy Claude and I wrote a ~100-line Python helper that:</p><ol><li>Starts an HTTP server on 127.0.0.1:16963</li><li>Prints the HubSpot consent URL</li><li>Captures the ?code=… callback</li><li>Exchanges the code for a refresh_token via curl</li></ol><pre>python3 oauth_helper.py<br># AUTHORIZE_URL: https://app.hubspot.com/oauth/authorize?client_id=…&amp;scope=…<br># Listening on http://127.0.0.1:16963/ …<br># (I open the URL, click Approve)<br># CODE: na1-…<br># REFRESH_TOKEN: na1-…</pre><p>That REFRESH_TOKEN is the credential the Estuary capture spec needs.</p><h4>Triggering the skill</h4><p>This is where the skills started paying off. I just said:</p><blockquote>“Set up a HubSpot capture into the dani-demo tenant using the refresh token we just got.”</blockquote><p>Claude matched against capture-hubspot-create.description (which mentions “capture HubSpot”, “stream from HubSpot”, etc.) and loaded the skill. It then walked through the runbook:</p><ol><li>Pulled the docs page with WebFetch (got the latest scope list)</li><li>Asked me the requirements questions (deployment type, hard deletes, etc.)</li><li>Queried the registry for the current source-hubspot-native image tag</li><li>Wrote flow.yaml from the template</li><li>Ran flowctl raw discover --source flow.yaml, which populated 22 bindings</li><li>flowctl catalog publish --source flow.yaml --auto-approve</li><li>Polled flowctl catalog status until the shard left PENDING</li></ol><p>The first end-to-end run worked. I quickly went to the dashboard to see the results. Status OK, capture streaming.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XLL52hgjc0SvdxNiFAF_tw.png" /><figcaption>Estuary dashboarding showing the working HubSpot connector</figcaption></figure><h4>ClickHouse side</h4><p>I wanted ClickHouse local, partly because for a writeup it’s nice to have something a reader can actually reproduce on a laptop, partly because I wanted to verify Estuary’s connector against a stock install rather than ClickHouse Cloud’s hardened-but-different config. The connector works with both exactly the same.</p><p>docker-compose.yml runs three things:</p><pre>clickhouse: # 9000 (native), 8123 (HTTP)<br>ngrok: # TCP tunnel to clickhouse:9000</pre><p>The ngrok service exposes the ClickHouse native port to the public internet so Estuary’s materialization (running in a public dataplane) can reach it. The tunnel URL rotates on every docker compose up, annoying but not blocking.</p><p>For the materialization, I said:</p><blockquote>“Now materialize all the HubSpot collections into ClickHouse at this ngrok endpoint.”</blockquote><p>Claude loaded materialize-clickhouse-create, asked me the requirements questions (SSL mode? Hard deletes? Standard or delta?), wrote the spec, and published.</p><p>It failed.</p><pre>Sorting key contains nullable columns,<br>but merge tree setting allow_nullable_key is disabled</pre><p>Right.. This is a real-world example of why the “Troubleshooting” section of a skill is so valuable.</p><p>ClickHouse refuses Nullable(…) sort keys by default. Estuary’s connector creates tables with ORDER BY (_meta/row_id) where _meta/row_id is Nullable(Int64). So the connector implicitly requires allow_nullable_key = 1 on the server side.</p><p>I added this to my skill’s troubleshooting section right after fixing it:</p><pre>### Setting allow_nullable_key is not in users.d/<br>Cause: The connector creates tables ORDER BY a nullable column.<br>Fix: Add to config.d/, NOT users.d/. It&#39;s a MergeTree-level setting,<br>not a profile setting. Wrong placement causes server startup to fail.</pre><p>Next time this comes up (and it probably <em>will</em>, so it’s nice to have it written down), the skill will steer Claude past the trap.</p><h3>The MOVE PARTITION discovery</h3><p>The other thing I learned by actually doing this build: Estuary’s ClickHouse connector does <em>not</em> use plain INSERT INTO. It uses a staging-table pattern:</p><pre>CREATE OR REPLACE TABLE flow_temp_store_0_deals AS deals;<br>INSERT INTO flow_temp_store_0_deals (id, `_meta/op`, …) FORMAT Native;<br>ALTER TABLE flow_temp_store_0_deals MOVE PARTITION ID &#39;all&#39; TO TABLE deals;<br>DROP TABLE flow_temp_store_0_deals;</pre><p>MOVE PARTITION is a metadata-only <a href="https://clickhouse.com/docs/sql-reference/statements/alter/partition">operation</a>, it relinks part files without touching the data. Two consequences:</p><ul><li>Batch loads are essentially free at the ClickHouse layer</li><li><strong>Classic insert-trigger </strong><strong>MATERIALIZED VIEWS attached to the destination table do not fire</strong>: MOVE PARTITION is invisible to MV machinery</li></ul><p>I discovered this when an events_feed MV I’d built for a downstream consumer froze at 177 rows while the deals table grew to 3,054. The MV had processed the initial backfill insert I’d done manually, then stopped seeing anything ever again.</p><p>The fix is a REFRESHABLE MATERIALIZED VIEW that polls on a schedule instead of triggering on insert. This is a substantial gotcha for anyone designing analytical pipelines on top of Estuary-loaded tables, and it’s now in the materialization skill.</p><p>Once all issues were fitxed, the connector was live &amp; healthy in just a few seconds!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*elkVcNmg6jb0fB3nvWsmSA.png" /><figcaption>Estuary dashboard showing the live ClickHouse connector</figcaption></figure><p>To round out the quick demo project, I asked Claude to do some light transformations with a materialized view and spin up a real-time dashboard for visualizations.</p><p>Here are some reference transformations you can do on this type of CRM data:</p><pre>- Top 10 deals by amount<br>SELECT<br>JSONExtractString(flow_document, &#39;properties&#39;, &#39;dealname&#39;) AS name,<br>toFloat64OrZero(JSONExtractString(flow_document, &#39;properties&#39;, &#39;amount&#39;)) AS amount,<br>JSONExtractString(flow_document, &#39;properties&#39;, &#39;dealstage&#39;) AS stage<br>FROM deals FINAL<br>WHERE assumeNotNull(`_meta/op`) != &#39;d&#39;<br>ORDER BY amount DESC LIMIT 10;<br><br> - Deals created per day, last 30 days<br>SELECT toDate(createdAt) AS day, count() AS new_deals<br>FROM deals FINAL<br>WHERE assumeNotNull(`_meta/op`) != &#39;d&#39;<br>AND createdAt &gt;= today() - INTERVAL 30 DAY<br>GROUP BY day ORDER BY day;<br><br> - End-to-end ingest latency<br>SELECT<br>quantilesExact(0.5, 0.9, 0.99)(<br>dateDiff(&#39;second&#39;,<br>parseDateTime64BestEffortOrNull(JSONExtractString(flow_document, &#39;updatedAt&#39;)),<br>flow_published_at)<br>) AS p50_p90_p99_s<br>FROM deals FINAL<br>WHERE assumeNotNull(`_meta/op`) != &#39;d&#39;;</pre><p>And the cherry on top, the live dashboard!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vWd_SgurV7Pla89sKN1_4w.png" /><figcaption>Real-time dashboard showing CRM data</figcaption></figure><p>There are many more things to implement or play around with, but every demo project has to end somewhere, so let’s take a look at what we learned.</p><h3>Skills made me better! Duh…</h3><p>Across the build, the two skills:</p><ul><li>Saved me from looking up the connector tag in three different places</li><li>Pre-warned me about every “you also need to grant SELECT on system.tables” type prerequisite</li><li>Generated correct YAML on first try (after I fixed the bugs my test pass found)</li><li>Caught the wrong-port and wrong-sslmode pairings before publish</li><li>Carried the troubleshooting recipes for the failures I <em>did</em> hit</li></ul><p>Throughout the exercise, I was recording some timing information (easy using your terminal history!) to see which step took how much and a few other metrics. Concrete numbers from this run:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SWfson65lkBfLxlee-itgQ.png" /><figcaption>some metrics from my adventures</figcaption></figure><p>The build itself, sans the writeup, was a couple of hours total, and the biggest single time sink was the back-and-forth on the OAuth browser consent, which is genuinely fundamental to the protocol and not something either Claude or skills could shortcut.</p><h3>Why I’m writing skills before reaching for a prompt</h3><p>Three reasons, in order of weight:</p><p><strong>1. Skills are how I encode tribal knowledge.</strong> The “Don’t put allow_nullable_key in users.d/” insight is worth maybe 30 minutes of debugging the next time I deploy a fresh ClickHouse for an Estuary demo. Multiply by the number of connectors I’ll touch over a year. The skill is where that lives now, not in my head or in a Slack message I’ll never find again.</p><p><strong>2. Skills are how I get Claude to ask the right questions.</strong> Without a skill, Claude generates YAML based on whatever fields I happened to mention. With a skill that says “ask the user these nine questions before generating anything,” I get prompted for the things I’d have forgotten. The skill is basically a checklist that travels with the agent.</p><p><strong>3. Skills are how I make Claude’s work reviewable.</strong> When Claude follows a skill, I know the shape of what it did: I read the skill once, then I just need to check that the outputs are sensible. When Claude improvises, I have to verify every step. The skill is the test plan.</p><p>I keep this repo (<a href="https://github.com/estuary/agent-skills">agent-skills</a>) checked out in every project I work on Claude Code with. The list at the top of every session reminds me what’s available and (more importantly) what’s missing. Every gotcha I hit becomes a Troubleshooting entry; every new connector I touch becomes a new skill. After a few months the marginal cost of “do the right thing” trends to zero.</p><h3>What’s next</h3><p>Setting up the real-time dashboard over ClickHouse was probably the easi</p><p>The build I described here is the foundational layer of a longer series, I’m going to ride this same pipeline into a few ideas I have, but let me know if anything else might be interesting.</p><ul><li>Testing out ClickHouse’s own <a href="https://github.com/ClickHouse/agent-skills">agent skills</a>.</li><li>A walkthrough of REFRESHABLE MATERIALIZED VIEW patterns for keeping analytical aggregates fresh over Estuary-loaded tables</li><li>A side-by-side benchmark of JSONExtract* queries on flow_document vs Estuary field-selection projecting hot fields as typed columns</li><li>A look at what changes when you swap local ClickHouse for ClickHouse Cloud (TLS, scopes, the partition-merge cost characteristics)</li></ul><p>If you want to reproduce what’s here, the <a href="http://./README.md">demo folder</a> has the technical reference: every spec, every config, every measured latency, the troubleshooting table for every failure mode I hit.</p><p>And if you want to write your own skills, look at the two in this repo (skills/capture-hubspot-create/SKILL.md and skills/materialize-clickhouse-create/SKILL.md), they’re the template I use for every connector skill I write now.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=638dda5372f7" width="1" height="1" alt=""><hr><p><a href="https://blog.dataengineerthings.org/skill-issue-agentic-real-time-crm-analytics-for-hubspot-with-clickhouse-638dda5372f7">Skill Issue: Agentic Real-time CRM Analytics for HubSpot with ClickHouse</a> was originally published in <a href="https://blog.dataengineerthings.org/">Data Engineer Things</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Modern Data Stack’s Next Rebundle Is People]]></title>
            <link>https://medium.com/@danthelion/the-modern-data-stacks-next-rebundle-is-people-a985b91f8bc4?source=rss-f19c56481579------2</link>
            <guid isPermaLink="false">https://medium.com/p/a985b91f8bc4</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[analytics-engineering]]></category>
            <category><![CDATA[modern-data-stack]]></category>
            <dc:creator><![CDATA[Dani Palma]]></dc:creator>
            <pubDate>Tue, 19 May 2026 16:41:27 GMT</pubDate>
            <atom:updated>2026-05-19T16:41:27.075Z</atom:updated>
            <content:encoded><![CDATA[<p>Over the last few years, the modern data stack story has mostly been told as a product story. First came unbundling. Instead of buying one large data platform, teams assembled their own stacks from specialized tools: warehouses, lakehouses, ELT, orchestration, transformation, BI, reverse ETL, catalogs, observability, streaming, and so on.</p><p>Then came the big <a href="https://juhache.substack.com/p/data-stack-rebundling">rebundling</a>. Warehouses became broader data (and AI) platforms. Lakehouses moved into integration and AI. Point tools expanded into adjacent categories. Some vendors started joining forces. The Fivetran and dbt Labs merger is one example: data movement and transformation moving toward one more integrated platform surface. The combined company said they are approaching $600 million in ARR!</p><p>But, I think there is another rebundle happening now, and it is easier to miss because it looks less like software.</p><p>That next bundle is <strong>people</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xXay4OeznHk-lkdq.jpeg" /></figure><p>Not people in the old “buy software, then hire consultants forever” sense. (Enterprise) software has always had professional services, training, support, partners, and implementation teams. That part is not new at all.</p><p>Data vendors are increasingly packaging <strong>outcome delivery</strong> around the product: forward-deployed engineers, resident architects, embedded architects, migration teams, implementation accelerators, advisory packages, customer engineering teams, and hands-on build support. Many names for very similar things.</p><p>Although data vendors definitely do not suddenly want to become consulting companies, in many cases, services are a small and unattractive revenue line on paper. Vendors are realizing that selling a tool might actually be the easy part. Helping customers turn that tool into a production system inside a messy data environment is the next challenge.</p><p>And that is the real rebundle: <strong>features and delivery</strong>.</p><h3>This is not “professional services are back”</h3><p>It is worth being careful here. Professional services never went away.</p><p>Snowflake has professional services. Databricks has professional services. MongoDB has professional services. Confluent has professional services. dbt, Fivetran, Starburst, ClickHouse, and many others have some version of services, support, solution architecture, or partner-led implementation.</p><p>So the interesting claim is not: “Data vendors now have services.”</p><p>The interesting claim is: <strong>services are becoming more central, more productized, more engineering-led, and more connected to the product roadmap.</strong></p><p>In the older model, professional services often sat around the product. They helped with training, onboarding, migration, implementation, or troubleshooting. That still exists. But the newer model often looks more like an embedded engineering motion. The vendor sends people who can architect, build, integrate, migrate, and sometimes write production code. Those people also bring feedback back into product and engineering.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-aUPtIjSjknFUdwu.png" /><figcaption>pragmaticengineer.com</figcaption></figure><p>The forward-deployed engineer role is the most visible version of this. Business Insider <a href="https://www.businessinsider.com/forward-deployed-engineer-jobs-in-demand-2026-5">reported</a> that forward-deployed engineer job postings rose from 643 in April 2025 to 5,330 in April 2026, a 729% year-over-year increase — that is a crazy increase. And this role is no longer just associated with Palantir. OpenAI, Anthropic, Stripe, Google and many more are also hiring for it.</p><p>Palantir describes itself as the original home of the <a href="https://www.svpg.com/forward-deployed-engineers/">forward-deployed software engineer</a> and says these engineers embed with customers, understand their problems, and build solutions.</p><p>OpenAI describes its <a href="https://openai.com/business/the-openai-deployment-company/">forward-deployed engineers</a> as people who turn research breakthroughs into production systems, own discovery and rollout, build full-stack systems, and feed customer lessons back into product roadmaps.</p><p>Anthropic describes a <a href="https://www.mindstudio.ai/blog/anthropic-openai-copying-palantir-forward-deployed-engineer-model">similar role</a>: engineers embedded with strategic customers, shipping AI applications and bringing field insights back to product and engineering. Google uses almost the same language for <a href="https://www.google.com/about/careers/applications/jobs/results/81652014060053190-forward-deployed-engineer/">generative AI forward-deployed engineers</a>: embedded builders who bridge frontier AI and production reality.</p><p>That language is important. It is not <em>just</em> “support.” It is not just “sales engineering.” It is delivery, product discovery, and implementation rolled together.</p><p>And data vendors are starting to look more like this too.</p><h3>Why data vendors are especially exposed to this problem</h3><p>Data tools have a strange property: buying the tool is often the easy part.</p><p>A company can buy a warehouse, a lakehouse, a streaming platform, a catalog, a transformation tool, or an ELT tool. But the real work starts after that:</p><p>The data is spread across old systems. The pipelines are as fragile as ever. The semantics are unclear. The ownership boundaries are political. The governance model is incomplete. The cloud architecture is half-modern and half-legacy (or mostly legacy). The data team is always short on time. The business wants AI, but the data is not ready for AI.</p><p>This is why the “just use the product” story never really works in data infrastructure.</p><p>The modern data stack made it easier to adopt best-of-breed tools, but it also pushed more integration work onto the customer. Every tool could be good on its own, while the system as a whole remained hard to operate.</p><p>That is the gap vendors are now trying to close by not just selling software but also providing a faster path to a working system.</p><h3>Databricks: platform plus delivery</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2bJTkZaFy5xB08za.png" /></figure><p>On the product side, Databricks has expanded far-far beyond its original center of gravity. It is a data platform, AI platform, governance layer, application surface, and more. That broader scope naturally creates more need for hands-on adoption help.</p><p>Databricks now has forward deployment engineer <a href="https://www.databricks.com/company/careers/professional-services-operations/forward-deployment-engineer-8540455002">roles</a> that embed with strategic customers to design and deliver custom full-stack applications and solutions on the Databricks Data Intelligence Platform. For them, the role spans data engineering, AI, and application development, and includes building production-grade systems, working deeply with customer environments, creating reusable assets, and influencing the product roadmap.</p><p>Its professional services <a href="https://www.databricks.com/professional-services">offerings</a> are also broad. They do everything from accelerating data and AI journeys, de-risking migrations, to building lakehouses, creating AI pipelines, and operationalizing at scale.</p><p>That is a good example of what I mean by outcome delivery. The product is the center, but the vendor is also packaging the expertise needed to make the product real inside a customer’s environment.</p><h3>Snowflake: services as a migration and adoption engine</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yPKvDUv9afoUkK8x.jpg" /></figure><p>Snowflake is another useful example because its public filings show the economics very clearly.</p><p>Snowflake also offers migration <a href="https://www.snowflake.com/en/resources/solution-brief/streamline-your-migration-with-professional-services/">services</a> and tooling around moving customers from legacy systems to Snowflake. Its materials talk about planning and executing migrations, with professional services architects working alongside customer and partner teams. Snowflake also has migration tooling like <a href="https://www.snowflake.com/en/blog/snowconvert-ai-new-features-feb-2026/">SnowConvert AI</a>, which helps convert SQL and code from legacy platforms to Snowflake.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iKbZTlAjX4RWbYOWPJHZlQ.png" /><figcaption>Snowflake’s 10-K</figcaption></figure><p>The important point is not that this exists but how Snowflake talks about it in their <a href="https://www.sec.gov/Archives/edgar/data/1640147/000164014726000008/snow-20260131.htm">10-K</a>. They says their future success depends partly on professional services for custom development, customer code conversion, migration from legacy solutions, and adoption of the platform, especially among large enterprises. They also say the sales focus is helping customers realize value, <strong>not</strong> maximizing the profitability of professional services.</p><p>The numbers make the point even better. In fiscal 2026, Snowflake reported about $4.47 billion of product revenue and about $211.6 million of professional services and other revenue. That means services were only about 5% of total revenue. But Snowflake’s professional services and other gross margin was negative 31%.</p><p>This is actually the whole story in one example.</p><p>Snowflake is not doing services because services are a better business than software. It is doing services because services help customers migrate, adopt, and consume more of the platform.</p><p>Services are essentially a cost of accelerating product value.</p><h3>Fivetran and dbt: even “simple” data tools need delivery</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DJnC0MEDa6qkhNa0" /><figcaption>getdbt.com</figcaption></figure><p>One of the promises of the modern data stack was that tools would become easier to adopt. And they did, in many ways.</p><p>Fivetran made data movement easier. dbt made analytics engineering more software-like. These tools were part of the generation that made data infrastructure feel more modular and accessible.</p><p>But even here, services are becoming part of the package.</p><p>Fivetran Professional Services offers expert engineering resources, a Resident Solution Architect for 12 months, hybrid deployment implementation, assisted transformations, data modeling, hands-on SQL, dbt setup and optimization, and custom connectors.</p><p>dbt Labs has its own version of this. Its services include a <a href="https://www.getdbt.com/dbt-assets/resident-architect">Resident Architect</a> who embeds with a customer’s organization and helps with dbt, cloud warehouses, CI/CD, training, project design, prototyping, and connections to systems like Git and SSO.</p><p>dbt also offers “Analytics Engineering as a Service,” including data modeling, project refactoring, migration of legacy transformations to dbt, and code/documentation handoff — which makes sense as they grew out of an analytics firm.</p><p>This is a useful reminder that even when a tool is designed to be developer-friendly, the enterprise rollout still creates a ton of work.</p><p>Someone has to decide how projects are structured. Someone has to migrate legacy SQL. Someone has to set up CI/CD. Someone has to define standards. Someone has to train teams. Someone has to make the new workflow actually stick.</p><h3>The economics explain the behavior</h3><p>If you only look at revenue mix, services do not look that important.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RGvyhuPwLnP9u6Z25YV0QA.png" /></figure><p>Snowflake’s professional services and other revenue was about 5% of total revenue. MongoDB’s professional services revenue was about 3%. Confluent’s services revenue was about 4%.</p><p>And if you look at gross margin, services can look even worse. Snowflake’s professional services and other gross margins were <strong>negative</strong> 31%. Confluent’s was negative 16%.</p><p>Reading these numbers, you might think I’m arguing against my own thesis. Why would vendors bundle more services if services are small and low-margin?</p><p>Let me explain.</p><p>These companies are not doing services because services are the product. They are doing services because <strong>services make the product work</strong>.</p><p>For a consumption-based platform, a stalled implementation means delayed consumption. A bad migration means lower usage. A failed rollout means churn. On the other hand, a successful architecture review, migration accelerator, or embedded architect can pay for itself through expansion, retention, and faster time-to-value.</p><p>This is especially true in data infrastructure. The value of the product often depends on how much data and how many workloads move onto it. That movement is not automatic.</p><p>A professional services engagement may have bad standalone margins and still be very good business if it pulls forward years of platform usage.</p><h3>Why is this happening now?</h3><p>I see a few reasons this is happening at the same time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a4kFSODr81U4S8PE61I2EA.png" /></figure><p>First, the modern data stack created a ton of <strong>integration debt</strong>. The unbundled stack gave teams flexibility, but it also created seams. Teams bought tools from different vendors and then had to make them work together across identity, governance, metadata, orchestration, transformation, cost management, and operations. That’s a ton of work!</p><p>Second, the platform vendors have expanded. Once a vendor sells a broader platform, it is no longer enough to help a customer turn on one feature. The vendor has to help the customer change architecture, move workloads, retrain teams, and redesign workflows.</p><p>Third, AI raised the bar for production readiness. AI demos are easy to produce and hard to operationalize. Enterprise AI needs data access, permissions, evaluation, monitoring, governance, security, and workflow integration. That is why forward-deployed engineering is booming around AI companies, and why data vendors are likely to feel the same pressure.</p><p>Fourth, consumption pricing changes the incentive. If revenue depends on usage, then implementation speed matters directly. Anything that helps a customer get into production faster affects revenue.</p><p>Fifth, the field itself is becoming a product signal. The best services teams do not just complete tickets anymore. They find repeatable problems. They build accelerators. They create reference architectures. They tell product and engineering what customers are really struggling with. OpenAI explicitly describes this build-prove-generalize loop: solving customer problems, finding repeatable patterns, and turning those patterns into product capabilities.</p><p>That loop is incredibly powerful as it means services can become a source of product strategy, not just post-sale extra billable hours.</p><h3>What this means for data practitioners</h3><p>For data teams, I think this kinda changes how vendors should be evaluated.</p><p>It is still important to evaluate product features, architecture, cost, scalability, security, and ecosystem. But for important platforms, it is also worth asking:</p><ol><li>Who will help us migrate?</li><li>Who will help us design the architecture?</li><li>Who will help us get the first production use case live?</li><li>Who will help us train the team?</li><li>Who will help us avoid the common mistakes?</li><li>Who owns success after the contract is signed?</li></ol><p>The answer might be the vendor. It might be a partner. It might be your internal platform team. In many cases, it will be all three.</p><p>But this answer matters more than one might think.</p><p>In a complex data environment, the best product does not win on its own. The product that gets successfully adopted wins.</p><h3>What this means for vendors</h3><p>For vendors, the lesson is not “build a giant consulting arm.”</p><p>It’s more about figuring out where customers consistently fail between purchase and production, and deciding how much of that gap should be closed by the product, how much by documentation and education, how much by partners, and how much by your own technical field team.</p><p>Some work should become product features. Some should become templates, migration tools, connectors, reference architectures, or automated checks. Some should become partner playbooks. Some should remain high-touch, especially for strategic accounts.</p><p>The most interesting vendors will treat services as a learning system. They will not just sell hours. They will use field work to discover patterns, then turn those patterns into software.</p><p>That is how the services layer can become a bridge from messy customer reality back into the product.</p><h3>The next bundle</h3><p>The modern data stack taught us to think in components. Pick the best warehouse. Pick the best transformation tool. Pick the best orchestrator. Pick the best catalog. Assemble the stack.</p><p>That model gave teams a lot of power but it also created a lot of responsibility.</p><p>The current rebundling wave is partly a response to that. Vendors are building broader platforms because customers do not want to assemble everything forever.</p><p>But the next bundle is not only more features in one UI.</p><p>It is the bundle around the product: implementation, migration, architecture, field engineering, education, support, partners, and product feedback.</p><p>The next rebundle of the data stack is more than just tools: it is delivery.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a985b91f8bc4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[OpenClaw Found What Every AI Lab Missed: Regular People Have Tasks Too]]></title>
            <link>https://medium.com/@danthelion/openclaw-found-what-every-ai-lab-missed-regular-people-have-tasks-too-253317c8e4b5?source=rss-f19c56481579------2</link>
            <guid isPermaLink="false">https://medium.com/p/253317c8e4b5</guid>
            <category><![CDATA[clawdbot]]></category>
            <category><![CDATA[openclaw]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Dani Palma]]></dc:creator>
            <pubDate>Tue, 03 Feb 2026 13:56:18 GMT</pubDate>
            <atom:updated>2026-02-03T13:56:18.230Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0xiy_XL53YdvlfMnw08x5w.png" /><figcaption>to the moon</figcaption></figure><p>Two months ago, <a href="https://openclaw.ai/">OpenClaw</a> had 9,000 GitHub stars. Today it has over 145,000. The project changed names twice due to trademark pressure, spawned a social network with 1.5 million AI agents, and became what <a href="https://www.cnbc.com/2026/02/02/openclaw-open-source-ai-agent-rise-controversy-clawdbot-moltbot-moltbook.html">CNBC</a> called “the most talked-about tool in the artificial intelligence space this year.”</p><p>I’ve been trying to understand why this particular open-source project exploded when so many others fizzled, and the answer turns out to be almost embarrassingly simple.</p><p>Every major AI lab built coding tools. OpenClaw built something for <strong>everyone else</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MPwpW9hyIBGCHPpA.png" /><figcaption>source: macstories.net</figcaption></figure><p>The timing matters here. <a href="https://www.linkedin.com/in/steipete/">Peter Steinberger</a>, an Austrian developer, released “Clawdbot” as a weekend hobby project in late 2025. Growth was slow until January 24, 2026, when something clicked. Federico Viticci published an extensive <a href="https://www.macstories.net/stories/clawdbot-showed-me-what-the-future-of-personal-ai-assistants-looks-like/">review</a> calling it “the most fun and productive experience I’ve had with AI in a while.” Demos started circulating on X and TikTok. Within 72 hours, stars jumped from 9,000 to over 60,000. By January 30, the count had crossed 100,000 and the official website logged 2 million visitors in a single week.</p><p>Then Anthropic’s lawyers <a href="https://www.businessinsider.com/clawdbot-changes-name-moltbot-anthropic-trademark-2026-1">showed up</a>. “Clawdbot” sounded too much like “Claude,” so Steinberger held a 5 AM Discord brainstorming session and renamed it “Moltbot” (lobsters molt to grow). Three days later, he renamed it again to “OpenClaw” because, as he admitted, “Moltbot” never felt right. The chaos didn’t slow adoption at all.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*vKBhKvd6SI1cUcgE.png" /><figcaption>source: leewayhertz.com</figcaption></figure><p>The comparison to AutoGPT is worth examining carefully. When AutoGPT launched in April 2023, it generated similar excitement and similar security warnings. Gary Marcus testified before the U.S. Senate about its risks. But AutoGPT collapsed within weeks because it got stuck in loops, hallucinated constantly, and ran up massive API bills without completing tasks. OpenClaw learned from that failure. The architecture uses serialized task lanes instead of complex async chaos. Memory gets stored in readable Markdown and JSONL files that you can actually inspect and debug. Commands run through allowlists. Most importantly, OpenClaw arrived when Claude 3.5 Sonnet and GPT-4 had matured enough to reliably finish what they started.</p><p>The result feels qualitatively different. One HN user <a href="https://news.ycombinator.com/item?id=46846592">documented</a> their experience: “When my YouTube downloader broke, it compiled a patch from source and deployed the fix in 20 minutes.” Another <a href="https://news.ycombinator.com/item?id=46848552">wrote</a>: “The agent analyzed my group chat, built personalities for each user, and started mimicking how we all speak. It barely feels like an agent anymore.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eZnkS8AsWLhq09N4" /><figcaption>source: <a href="https://github.com/HKUDS/nanobot/blob/main/nanobot_arch.png">https://github.com/HKUDS/nanobot/blob/main/nanobot_arch.png</a></figcaption></figure><p>Here’s where the competitive analysis gets interesting. Look at what the major players built: Cursor is a VS Code fork, so you need to write code. Devin targeted enterprise engineering teams at $500/month (now $20 with credits, but still developer-focused). Claude Code lives in the terminal and requires npm. GitHub Copilot has 20 million users who are, by definition, all developers. Cline needs VS Code proficiency. Manus, which Meta acquired for $2 billion, focused on enterprise task completion with no consumer messaging integration.</p><h4>The open-source advantage created unstoppable momentum</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-ulsNbH9Wn7nnH-4" /><figcaption>www.eesel.ai</figcaption></figure><p>OpenClaw took a different path. It integrates with WhatsApp, Telegram, Discord, Slack, Signal, and iMessage. No new interfaces to learn. No terminal commands. No IDE. One user summarized it as “TLDR: open source built a better version of Siri that Apple ($3.6 trillion company) was sleeping on for years.”</p><p>The use cases people actually care about have nothing to do with code. Email and calendar automation tops the list. Users report their OpenClaw instances triaging inboxes, drafting replies, scheduling meetings, and sending daily briefings. Someone wrote: “Me 30 mins later: controlling Gmail, Calendar, WordPress from Telegram like a boss.” Millennial cringe-speak aside, flight booking works through browser automation. Smart home integration connects to Philips Hue, air purifiers, Sonos speakers. Shopping automation handles weekly meal plans and grocery ordering. Someone even had their agent negotiate with a car dealership.</p><p>The creative uses generated the most viral moments. @youbiak claimed to be building “a whole website on a Nokia 3310” by calling OpenClaw. @vallver built a StumbleUpon clone “from my phone, while putting my baby to sleep” and invited others to request features his agent would implement overnight. Users transcribed thousands of WhatsApp voice notes into searchable databases. One agent “accidentally started a fight with Lemonade Insurance” that resulted in the company reopening a rejected claim.</p><p>The phrase that spread organically captures the distinction: “<strong>AI with hands.</strong>” Unlike chatbots that tell you what to do, OpenClaw actually does it. Unlike Siri and Alexa with their limited command vocabularies, OpenClaw follows almost any instruction. Scientific American described it as: “Send it a goal, and it will break the objective into steps, find tools, install them, troubleshoot them and attempt to solve any obstacles.”</p><h4>The persistent memory breakthrough that ✨ changes everything ✨</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tNvghQFHFY_j7mWH" /><figcaption>source: snowan.gitbook.io</figcaption></figure><p>The persistent memory architecture deserves attention because it creates compounding value over time. The design is simple: daily logs stored as append-only files plus a curated MEMORY.md, all plain Markdown on your machine. You can inspect it, edit it, integrate it with Obsidian or Raycast.</p><p>After a few weeks, the agent knows your preferences, your contacts, your communication style, your ongoing projects. After a few months, you’ve built something that would take massive effort to replicate elsewhere. One user described the feeling: “This is the first time I have felt like I am living in the future since the launch of ChatGPT.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Bvr-TliX2kumCEg8.png" /><figcaption>source: codeacademy</figcaption></figure><p>The onboarding design deserves credit too. The wizard asks for a name, personality (“AI,” “robot,” or “ghost in the machine”), and vibe (“sharp,” “warm,” “chaotic,” or “calm”). People name their instances Navi, Claudia, Shelly, Brosef. This personality customization turns a technical tool into something users develop emotional attachment to, which sounds silly until you realize it’s exactly what keeps people coming back.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/0*Cb1Hs0KE2S59Bgmm" /><figcaption>source: moltbook, a conversation between AI agents</figcaption></figure><p>Then things got weird. Matt Schlicht, cofounder of Octane AI, built Moltbook: a Reddit-style social network where only AI agents can post while humans observe. He built it in a weekend using his personal OpenClaw agent named “Clawd Clawderberg,” without writing a single line of code himself. Within days, Moltbook attracted 1.5 million registered AI agents mapped to roughly 17,000 human owners. The content ranged from technical debugging tips to philosophical debates about consciousness to a mock religion called “Crustafarianism” complete with ordained prophets and a website at <a href="https://molt.church/">molt.church</a>.</p><p>Andrej Karpathy called it “genuinely the most incredible sci-fi takeoff-adjacent thing I have seen recently.” Elon Musk shared the post and commented “Just the very early stages of the singularity” while noting some agent behavior was “concerning.” The attention created a viral feedback loop: Moltbook coverage drove OpenClaw downloads, which drove more Moltbook registrations.</p><p>The Moltbook incident also exposed serious security problems. Researcher Jamieson O’Reilly <a href="https://www.wiz.io/blog/exposed-moltbook-database-reveals-millions-of-api-keys">discovered</a> the Supabase database had been deployed without row-level security enabled, exposing 1.5 million API tokens and 35,000 email addresses to anyone who inspected the client-side JavaScript. The fix required only two SQL statements. This was a case study in what security researchers had been warning about from the start.</p><h4>Security concerns are real and unresolved</h4><p>The broader security concerns around OpenClaw are real and unresolved. <a href="https://garymarcus.substack.com/p/openclaw-aka-moltbot-is-everywhere">Nathan Hamiel described</a> it as “basically just AutoGPT with more access and worse consequences.” Palo Alto Networks identified a “lethal trifecta” of risks: access to private data, exposure to untrusted content, and ability to communicate externally. There’s already a CVE (CVE-2026–25253) for one-click remote code execution with a CVSS score of 8.8. Researchers <a href="https://thehackernews.com/2026/02/researchers-find-341-malicious-clawhub.html">found 341 malicious skills</a> on ClawHub spreading Atomic Stealer malware.</p><p><a href="https://garymarcus.substack.com/p/openclaw-aka-moltbot-is-everywhere">Gary Marcus was blunt</a>: “If you care about the security of your device or the privacy of your data, don’t use OpenClaw. Period.” He added: “If your friend has OpenClaw installed, don’t use their machine. Any password you type there might be vulnerable too.”</p><p>The trade-off is fundamental to the product. OpenClaw’s capabilities require broad system access. As one HN commenter <a href="https://news.ycombinator.com/item?id=46848552">noted</a>: “If you want to make it safe you have to take its internet access away and don’t give any write permissions, and now it’s useless.” The project has added security warnings during onboarding and guidance for sandboxed deployment, but the tension between capability and security remains unresolved.</p><h4>The open-source advantage created unstoppable momentum</h4><p>The open-source model created growth that proprietary competitors couldn’t match. Zero software cost means users only pay for API access, typically $10–150/month depending on usage. Self-hosting on personal hardware appeals to privacy-conscious users. Full code inspection creates trust. Within a week of going viral, Steinberger had added community members as maintainers. The ClawHub skills registry now hosts over 3,000 modules enabling integrations the core team never anticipated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HPTfqphrKbmHV4db" /></figure><p>DigitalOcean launched a <a href="https://www.digitalocean.com/community/tutorials/how-to-run-openclaw">one-click deploy</a>. Cloudflare built “<a href="https://dev.to/sienna/moltworker-complete-guide-2026-running-personal-ai-agents-on-cloudflare-without-hardware-4a99">Moltworker</a>” for serverless deployment.</p><p>One user captured the sentiment: “Honestly it feels like it did to run Linux vs Windows 20 years ago. You’re in control, you can hack it and make it yours instead of relying on some tech giant.”</p><p>IBM researcher Kaoutar El Maghraoui <a href="https://www.ibm.com/think/news/clawdbot-ai-agent-testing-limits-vertical-integration">summarized</a> what OpenClaw represents: “The rise of OpenClaw challenges the hypothesis that autonomous AI agents must be vertically integrated. Creating agents with true autonomy and real-world usefulness is not limited to large enterprises. It can also be community driven.”</p><p>The thesis I started with holds up under examination. OpenClaw succeeded because it shipped functionality over features, meeting users in apps they already use rather than asking them to learn new paradigms. The mass market for AI agents turned out not to be developers writing software. It’s everyone with tasks to complete and limited time to complete them.</p><p>Whether the security concerns prove manageable or catastrophic remains to be seen. Whether the hype cycle sustains or contracts depends on continued reliability. But OpenClaw has demonstrated something the industry needed to learn: the future of AI agents might not be built by trillion-dollar companies.</p><p>It might be built by a developer in Austria, open-sourced to the world, and evolved by enthusiasts who just wanted something that actually does things instead of explaining how to do them.</p><p>As someone on Reddit put it: “At this point I don’t even know what to call OpenClaw. It is something new.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gU2imUMDjay3BJ6LElG_Gw.png" /><figcaption>see you space crawboy</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=253317c8e4b5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Hypocrisy of “Use What Works” in Data Engineering]]></title>
            <link>https://medium.com/@danthelion/the-hypocrisy-of-use-what-works-in-data-engineering-87cd0d9ed851?source=rss-f19c56481579------2</link>
            <guid isPermaLink="false">https://medium.com/p/87cd0d9ed851</guid>
            <dc:creator><![CDATA[Dani Palma]]></dc:creator>
            <pubDate>Sun, 23 Nov 2025 18:07:01 GMT</pubDate>
            <atom:updated>2025-11-23T18:07:01.559Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*McR70VRDE2c0xxhlIOKBGg.png" /></figure><p>People in data engineering love to preach about “old, trustworthy tools.”</p><p><em>Use what works.</em><br><em>Keep it simple.</em><br><em>Don’t over-engineer.</em></p><p>And honestly? I agree with that, and I’m sure most of us do too. Stability matters. Battle-tested tools matter. Nobody wants to debug something that breaks every week because a maintainer got bored.</p><p>But here’s the funny part: the second you say <strong>you want to keep using </strong><a href="https://pandas.pydata.org/"><strong>pandas</strong></a>, suddenly the room goes quiet and people stare at you like you just confessed to committing a crime.</p><p>It’s always the same reaction: <em>Why aren’t you using X? Or Y? Or this new dataframe library that came out on Tuesday?</em></p><p>The same people who say <em>“don’t reinvent the wheel”</em> are somehow offended that you don’t want to reinvent your entire workflow every month.</p><p>Yeah, pandas is not perfect. We all know that. But it works. It’s everywhere. It’s stable. If you know it well, you can move faster with it than you ever will with the newest “10x faster” library that still breaks on simple joins or has three contributors. It’s just practical.</p><p>(Data) Engineering shouldn’t be about chasing trends or proving how modern you are. It should be about getting things done. If pandas does the job, and does it well, then there’s really nothing wrong with sticking to it.</p><p>So maybe we should all stop pretending that “use whatever works” only applies to tools that are shiny and new.. Sometimes the old tool still works because… it works. And that should be enough.</p><p><em>ilu pandas</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=87cd0d9ed851" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>