<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Streamkap on Medium]]></title>
        <description><![CDATA[Stories by Streamkap on Medium]]></description>
        <link>https://medium.com/@streamkap?source=rss-53e191a4c46a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*AzHfFGuW9ywnKBRm2VxDPQ.png</url>
            <title>Stories by Streamkap on Medium</title>
            <link>https://medium.com/@streamkap?source=rss-53e191a4c46a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 17:18:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@streamkap/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[10 catalogs, 3 ETLs, 2 Postgres’ and a partridge in a pear tree]]></title>
            <link>https://medium.com/@streamkap/10-catalogs-3-etls-2-postgres-and-a-partridge-in-a-pear-tree-23a888805993?source=rss-53e191a4c46a------2</link>
            <guid isPermaLink="false">https://medium.com/p/23a888805993</guid>
            <category><![CDATA[fivetran]]></category>
            <category><![CDATA[dbt-labs]]></category>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[apache-kafka]]></category>
            <category><![CDATA[confluent]]></category>
            <dc:creator><![CDATA[Streamkap]]></dc:creator>
            <pubDate>Fri, 19 Dec 2025 23:17:06 GMT</pubDate>
            <atom:updated>2025-12-19T23:17:06.065Z</atom:updated>
            <content:encoded><![CDATA[<p>Are you familiar with geomorphology? It’s a field within geology, the study of how landscapes evolve. It occurred to me this morning that this newsletter has been a bit of geomorphology of the changing data landscape this year, a modern data stack (MDS) morphology, if you will.</p><p>The problem with geomorphology is that you rarely get to see change happen; you can see the current state and infer from signals in that state the processes that took thousands or millions of years to create it.</p><p>We had no such challenge in MDS morphology in 2025. A LOT of stuff happened…</p><p>AI continued drive headline numbers for funding ($200B+), acquisitions, news headlines/controversy (though for a heady 2 weeks in July, MDS staple <a href="https://trends.google.com/trends/explore?geo=US&amp;q=openai,astronomer&amp;hl=en">Astromer took the crown</a>)…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*O9AyDNoxmpNF7whX.png" /></figure><p>In data, there were So. Many. Acquisitions. Thus the title of this post, lots of activity in the catalogue/metadata/semantic layer/ontology space (<a href="https://streamkap.substack.com/p/just-semantics">as I wrote about earlier this year</a>). A bit of ETL. Some Postgres (wrote about <a href="https://streamkap.substack.com/p/htap-is-dead-long-live-htap">that too</a>).</p><p>The biggest news came in the past 6 months.</p><p>I started this newsletter writing about <a href="https://streamkap.substack.com/p/the-modern-data-stack-grows-up">Fivetran and dbt</a>, so perhaps it’s fitting that they announced they are merging. A not-too-surprising development that has dbt devotees worried about the future of the well-loved open source project and community.</p><p>Confluent is being acquired by IBM for $11B, an earthquake for the streaming world as the leading commercial and open source projects will be subsumed into tech’s looming dinosaur, causing yet more questions and consternation about the future of another critical open source project, Kafka.</p><p>Databricks and Snowflake both bought a Postgres…</p><p>From a Streamkap product and company perspective, we’ve also had a huge year, record growth, launching on the Snowflake Marketplace (for private offers for now), Streaming Transformations in product, and helping customers with <a href="https://streamkap.com/blog/streaming-400-000-tables-with-streamkap">CDC from hundreds of thousands of tables into Snowflake</a>.</p><p>We’ll dig into that and more below in the MDS morphology section. But first…</p><h3><strong>Streamkap Customer Story — InHire</strong></h3><p>Rockset was one of our early partners in the Streamkap journey. We helped customers load data into the real-time database. Cut to June 2024, Rockset announced that they were being acquired by OpenAI (hey, yet another acquisition!) and that the service would be completely shut down. Customers on month-to-month plans had just 3 months to migrate🫠.</p><p>That takes us to a fateful Saturday in August, <a href="https://www.linkedin.com/in/julianotebinka/">Juliano Tebinka</a>, Co-Founder and Chief Data Officer of <a href="https://www.inhire.com.br/">InHire</a>, a leading AI applicant tracking system in Brazil, was looking for solutions to replace Rockset. InHire had built key real-time customer-facing functionality, AI agents, reporting, applicant workflows, and more around the native integration between their production database, DynamoDB, and Rockset. Both this integration and Rockset itself would cease to be available in just a few short weeks. As Juliano recently put it to me in a LinkedIn comment:</p><blockquote>“You nailed the story, but left out the part where I almost had a full-on meltdown when that shutdown email hit 😂”</blockquote><p>Juliano was working on a Saturday, trying to find a solution as the clock ticked down. He had looked at options like Fivetran, Airbyte, Confluent, and others but they either couldn’t deliver. Either failing to meet real-time requirements, were too rigid, too expensive, or too slow to engage with.</p><p>Juliano found Streamkap via a Google search, and when he landed on the site he jumped into the chat to ask about his specific use case:</p><blockquote>“I sent a chat message on a Saturday evening, not expecting a reply,” Juliano recalls. “Someone responded almost immediately. That was instrumental. We realized we would get the support we needed.”</blockquote><p>It turned out we were well set up to help Juliano build a replacement.</p><p>The plan was to replace Rockset with ClickHouse, but while Rockset’s native integration with DynamoDB and document-based structure worked seamlessly with InHire’s data structure, ClickHouse did not. The data would need to be cleaned up and structured in-flight and in real-time before loading into ClickHouse.</p><p>Luckily, Streamkap was perfect for the job. Our <a href="https://docs.streamkap.com/transforms-1">Flink-powered stream processing handled</a> the transformations to get the data in the right shape while maintaining sub-second latency. Best yet, we were able to deliver in just 3 weeks with no dedicated resources on the InHire side. Again, in Juliano’s words:</p><blockquote>“3 weeks. THREE. We ripped everything over to Streamkap and now we’re faster, more reliable, and I still don’t have to pretend I’m a data engineer haha.</blockquote><blockquote>Shoutout to you for actually answering the website chat on a Saturday night and turning this from “oh sh*t” into “we got this.” Streamkap isn’t a vendor — you’re the crew we call when the house is on fire, and somehow everything still ships on time.”</blockquote><p>Since that initial deployment, we’ve helped Juliano and team expand their use cases and are now replicating to both <a href="https://docs.streamkap.com/clickhouse#clickhouse-cloud">ClickHouse</a> and <a href="https://docs.streamkap.com/databricks-delta-lake">Databricks</a> for their fast and slow workload,s respectively. Check out the full <a href="https://streamkap.com/blog/case-study-inhire-powers-real-time-ai-recruitment-with-streamkap">InHire + Streamkap case study on the website</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KFUt8NUecVJ7x4kP.png" /></figure><h3><strong>Streamkap Product Update</strong></h3><p>It’s been a big year for us on the product front; some highlights below.</p><h4><strong>1. Flink Stream Processing (“Shift Left”)</strong></h4><p>Tools to clean, mask, and transform data before it reaches the destination.</p><ul><li>Multi-Language Transforms: Support for writing lightweight snippets in<a href="https://docs.streamkap.com/transforms-1#streaming-transforms"> JavaScript and Python</a>.</li><li>TypeScript Development Kit (TDK)<strong>:</strong> A local workflow for building, testing, and bundling<a href="https://docs.streamkap.com/transform-typescript-development-kit"> TypeScript transforms with npm dependencies</a>.</li></ul><h4>2. Expanded Connectors and Kafka Access</h4><ul><li>New Destinations: Iceberg is hot, we now power real-time streams to<a href="https://streamkap.com/connectors/apache-iceberg"> Apache Iceberg</a>, we also released native connectivity for<a href="https://streamkap.com/connectors/motherduck"> Motherduck</a> and <a href="https://streamkap.com/connectors/duckdb">DuckDB</a></li><li>New Sources: Real-time change data capture for<a href="https://docs.streamkap.com/planetscale#planetscale-vitess"> PlanetScale</a> and<a href="https://streamkap.com/connectors/supabase"> Supabase</a>.</li><li>Direct Kafka Access<strong>:</strong> New visibility and management tools for <a href="https://docs.streamkap.com/consumer-groups#consumer-groups">Consumer Groups</a> &amp; <a href="https://docs.streamkap.com/schema-registry#schema-registry">Schema Registry</a>.</li></ul><h4>3. Snapshot Improvements</h4><ul><li>Filtered Snapshots: Allows for partial snapshots to capture specific data rather than whole tables! <a href="https://docs.streamkap.com/snapshots">Filtered Snapshots</a>.</li><li>Snapshot Progress: Real-time<a href="https://docs.streamkap.com/snapshots"> visibility into progress and ETA</a> for active snapshots.</li></ul><h3><strong>MDS Morphology 2025</strong></h3><p>$20B+ dollars of data landscape acquisitions were announced this year, the biggest, Confluent, at $11B, capping off the year earlier this month.</p><p>I will focus a bit more on that in the future. Let’s look at the Fivetran/dbt merger first.</p><p>I’ll start by saying, I get it. From both a dbt perspective and a Fivetran perspective, this was very rational.</p><p>As I <a href="https://www.linkedin.com/posts/pauldudley_dbt-moderndatastack-snowflake-activity-7295158669183172608-CBxW/?utm_source=social_share_send&amp;utm_medium=member_desktop_web&amp;rcm=ACoAAAB55foBkgE2hMeWEA271SFlv96b2aL9QQU">wrote previously</a>, dbt has scrapped their way to within striking distance of meeting their $4.2B valuation from 2021. What I didn’t cover in that post was how they could actually get liquidity at that valuation.</p><p>The IPO market these days is closed to companies below $500M ARR, a milestone likely 2–3 years away, at minimum. dbt has 2 obvious acquirers, Snowflake and Databricks, and I’m sure both have thought long and hard about an acquisition, but clearly neither felt compelled to pull the trigger. My sense is that they are already getting the milk, why buy the cow? dbt is perennially a top partner in terms of credit consumption for Snowflake, and I presume Databricks. What advantage would buying dbt confer? Mostly a competitive one, e.g. if Databricks bought dbt, they could use it to build a migration path from Snowflake to Databricks and/or stop supporting Snowflake. Clearly, neither Snowflake nor Databricks felt this was compelling enough to offer an acquisition more compelling than a Fivetran merger…</p><p>Speaking of, with a Fivetran merger, the combined ARR of both companies is $600M. That’s good enough to go public today ✅. With closing the merger and the actual IPO process, we’re probably talking mid to late 2026 for the IPO but a ~1 year very clear path to liquidity is a lot more attractive than years of fighting to <em>maybe</em> get there in 2028/2029, and probably having to raise money again in the meantime.</p><p>All of this was the carrot for dbt. Fivetran also had a stick. They acquired Tobiko Data, the creators of SQLMesh (dbt’s strongest open source competitor). I’m sure part of the conversation was, “that’s a nice transformation business you have there, it would be a shame if something happened to it”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vpBWOSswpUb0WbFU.png" /></figure><p>What about Fivetran? Well, I suspect they were already going to go public in 2026. But they were under different pressure. Their biggest partner, Snowflake, announced a native solution for ETL this year, Openflow. While Openflow is not much of a threat in the short term, it doesn’t offer the ease, simplicity, and polish that Fivetran (and Streamkap) offers out of the box today, if you’re a public market investor looking at the long term strategic implications. That is going to be a big question.</p><p>The combination? Fivetran and dbt are marketing it as the open data stack. Part of this was, I’m sure, focused on using the word “open” strategically, to ease concerns about the implications for the dbt open source product and community under the boot of Fivetran’s very, ehem, <a href="https://streamkap.substack.com/p/fivetran-pricing-poor-execution-or">commercial</a> approach to customer relationships. But another part was positioning against the increasingly all-in-one solutions offered by Databricks, Snowflake, and MSFT Fabric. The giants are pitching a one-stop shop, whether or not they deliver on that promise is a different question…</p><p>So, enter dbtran or fivebt or fishtran, the savior of the modern data stack. The OPEN data stack. They will provide all of the everything, EXCEPT the compute. That part will be your choice, giving you freedom and flexibility. I think this positioning is fine, makes sense, buuuut if you look at the diagram from the announcement below. What is the first thing you think?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zG9J5A6Sz8AD1NWZ.png" /></figure><p>For me? I think, boy it seems likely they are going to find a compute later to slot in there in the next couple of years…</p><p>Fivetran has been making some noise around iceberg for a while so it could be something like Dremio or Starburst, the startups that were early to the iceberg compute play. Motherduck also comes to mind, built on the back of MDS darling duckDB.</p><p>Speculation on the compute layer aside, what does this merger mean for beloved dbt? The message in the announcement, from George Fraser, Fivetran CEO, was that the open source project would remain a key focus, as would the community, and that he would be a fool to disrupt either. I believe that was a truthful statement. However, what is the main reason that mergers and acquisitions fail? Culture fit, or lack thereof. dbt and Fivetran are unquestionably different cultures, which will dominate?</p><p>On top of that, with the IPO coming, a quote from Mike Tyson comes to mind, “everybody has a plan until they get punched in the mouth”…</p><p>2026 will be interesting! I’m excited:)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=23a888805993" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Evolution from Lambda to Kappa Architecture: A Comprehensive GuideWhat is Kappa Architecture?]]></title>
            <link>https://medium.com/@streamkap/the-evolution-from-lambda-to-kappa-architecture-a-comprehensive-guidewhat-is-kappa-architecture-89de1051847b?source=rss-53e191a4c46a------2</link>
            <guid isPermaLink="false">https://medium.com/p/89de1051847b</guid>
            <dc:creator><![CDATA[Streamkap]]></dc:creator>
            <pubDate>Mon, 01 Sep 2025 12:39:47 GMT</pubDate>
            <atom:updated>2025-09-01T12:39:47.093Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LgSdLcSjzqQwfB82v-07fw.png" /></figure><p>Kappa Architecture is a streamlined data processing framework designed for real-time data handling, offering simplicity and efficiency. As organizations search for “Kappa Architecture vs Lambda Architecture” solutions, Kappa stands out by eliminating the dual-layer complexity of Lambda. Instead of separate batch and streaming systems, Kappa unifies everything into a single stream processing model.</p><p>In Kappa Architecture, all data is treated as an ongoing stream, processed incrementally without needing a dedicated batch layer. This makes it ideal for scenarios requiring:</p><ul><li>Frequent data updates</li><li>Low-latency processing</li><li>Real-time analytics</li><li>Simplified infrastructure compared to multi-layered setups</li></ul><p>By focusing on streaming, Kappa Architecture provides flexible, scalable solutions for modern data needs. It supports batch outputs by storing processed streams in systems like data lakes, allowing versatile consumption. As searches for “benefits of Kappa Architecture” rise, its popularity grows among businesses seeking real-time analytics and scalable systems.</p><h3>What is Lambda Architecture?</h3><p>Lambda Architecture, often compared in “Kappa vs Lambda Architecture” discussions, requires separate batch and streaming layers, with an optional unified serving layer. This dual approach can lead to increased complexity, making Kappa a preferred alternative for many.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*URbnOptJGeFd9sw6.png" /></figure><h3>Understanding How Kappa Architecture Works</h3><ul><li><strong>Data Ingestion</strong>: Data streams from sources into a message broker like Apache Kafka, ensuring durable, unified, and immutable storage for real-time processing.</li><li><strong>Stream Processing</strong>: Engines such as Apache Flink handle transformations, enrichments, and aggregations to deliver insights.</li><li><strong>Data Storage</strong>: Outputs are stored in data lakes, NoSQL databases, relational systems, or formats like Apache Iceberg for long-term access and querying.</li><li><strong>Replayability and Reprocessing</strong>: Historical data can be replayed for logic updates or error fixes, enhancing adaptability.</li><li><strong>Scalability</strong>: Distributed systems enable horizontal scaling for high-volume data.</li></ul><p>These components make Kappa Architecture a top choice for “real-time data processing architectures.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IRYUA-vXAxLONviu.png" /></figure><h3>Key Advantages of Kappa Architecture</h3><p>When exploring “advantages of Kappa Architecture,” key benefits include:</p><ul><li><strong>Simplicity</strong>: A single layer reduces management overhead compared to Lambda.</li><li><strong>Real-Time Insights</strong>: Immediate processing enables quick decision-making.</li><li><strong>Adaptability</strong>: Stream reprocessing supports evolving needs.</li><li><strong>Scalability</strong>: Handles growing data volumes efficiently.</li><li><strong>Cost Efficiency</strong>: No batch layer means lower infrastructure and maintenance costs.</li></ul><p>These perks position Kappa as a leader in “scalable data architectures.”</p><h3>Integrating Shift Left with Kappa Architecture</h3><p>“Shift Left in data processing” refers to addressing issues early in the pipeline, from software development origins. In Kappa Architecture, it involves validating, transforming, and enriching data near ingestion to ensure quality and minimize errors.</p><p>Benefits of Shift Left in Kappa include:</p><ul><li><strong>Early Data Quality</strong>: Identify issues at ingestion for cleaner processing.</li><li><strong>Faster Insights</strong>: Enriched data speeds up delivery.</li><li><strong>Resource Savings</strong>: Reduces reprocessing, optimizing efficiency.</li></ul><p>Combining Shift Left enhances Kappa’s real-time capabilities, creating robust workflows.</p><h3>‍Real-World Examples of Kappa Architecture Implementation</h3><p>Companies adopting Kappa demonstrate its power:</p><ul><li><strong>Shopify</strong>: Uses Apache Kafka for real-time events like transactions and inventory, enabling instant merchant insights.</li><li><strong>Netflix</strong>: Manages billions of events with Kafka and Flink for recommendations and analytics, ensuring smooth experiences.</li><li><strong>Uber</strong>: Processes ride data dynamically with Kafka and Flink for routing and pricing optimization. Explore “Uber’s real-time architecture” for details.</li><li><strong>LinkedIn</strong>: Handles streams for notifications and recommendations via Kafka.</li><li><strong>Stripe</strong>: Employs Kappa-like systems for fraud detection and low-latency payments.</li></ul><p>These cases highlight Kappa’s role in real-time data pipelines.</p><h3>Conclusion: Why Choose Kappa Architecture Today</h3><p>Kappa Architecture, paired with Shift Left, revolutionizes real-time systems, outperforming Lambda in speed, simplicity, scalability, and edge. For implementing Kappa Architecture, tools like Streamkap can ease your transition</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=89de1051847b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[IPO is a filter, not a threshold]]></title>
            <link>https://medium.com/@streamkap/ipo-is-a-filter-not-a-threshold-da4e487adcd1?source=rss-53e191a4c46a------2</link>
            <guid isPermaLink="false">https://medium.com/p/da4e487adcd1</guid>
            <dc:creator><![CDATA[Streamkap]]></dc:creator>
            <pubDate>Fri, 29 Aug 2025 14:29:14 GMT</pubDate>
            <atom:updated>2025-08-29T14:29:14.190Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lDa0ylZmwB_fmNJd.png" /></figure><p>Last week, Databricks announced their Series K(!) funding round at a $100B valuation🤯</p><p>It used to be that once a company hit $100M in revenue, they could go public (as long as they were growing enough and not too unprofitable), and they were MOTIVATED to do so because private market COULDN’T provide enough capital for growth and WOULDN’T enough liquidity for VCs, Employees, and Founders.</p><p>This has all changed. Now, you CAN’T go public until you hit $500M ARR, but you can raise plenty of money, including providing some liquidity, so there is less pressure for an IPO.</p><p>On the other side of the coin, you have the hottest private venture-backed giants,<a href="https://www.linkedin.com/company/stripe/"> Stripe</a>,<a href="https://www.linkedin.com/company/openai/"> OpenAI</a>,<a href="https://www.linkedin.com/company/databricks/"> Databricks</a>, SpaceX. These are all in the billions of revenue and growing incredibly fast.</p><p>They don’t seem to need to go public at all? Databricks raised a new round at a $100B valuation. OpenAI is selling secondaries at a $500B valuation!!</p><p>For Databricks, there has been a clear strategic advantage to staying private. They have been able to acquire companies like Tabular and Neon for prices that<a href="https://www.linkedin.com/company/snowflake-computing/"> Snowflake</a> would have had a hard time justifying to their public shareholders.</p><p>What does this mean for regular americans who want to participate in the crazy growth of the best companies? Nothing good, I’m afraid. Even if the public were able to access private markets (outside of accredited investors), they simply wouldn’t get into the best deals, and it would be very unlikely to benefit in the way they might have in an IPO 10 or 20 years ago.</p><p>I don’t know what the path forward is, but the<a href="https://www.linkedin.com/company/figma/"> Figma</a> IPO pop a couple of weeks back shows that the public is hungry to get their hands on the hottest companies. We need to figure out how to make it happen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/0*jWvKxc4e3vz6IO1f.png" /></figure><p><strong>Warmly,<br>Paul Dudley</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da4e487adcd1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is AI doomerism galaxy brain marketing?]]></title>
            <link>https://medium.com/@streamkap/is-ai-doomerism-galaxy-brain-marketing-c4952d4e68c5?source=rss-53e191a4c46a------2</link>
            <guid isPermaLink="false">https://medium.com/p/c4952d4e68c5</guid>
            <dc:creator><![CDATA[Streamkap]]></dc:creator>
            <pubDate>Wed, 06 Aug 2025 23:23:34 GMT</pubDate>
            <atom:updated>2025-08-06T23:23:34.446Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>PR based growth demands blood 🩸</strong></p><p>We’ve all seen the headlines, AI is going to take our jobs and then, potentially, our lives.</p><p>There are a number of AI researchers, folks who are very credible in the field, whose estimate of the probability that AI will kill us, p(doom), is quite high.</p><p>Take, for example, Eliezer Yudkowsky, a longtime AI researcher and founder of the <a href="https://www.linkedin.com/company/machine-intelligence-research-institute/">Machine Intelligence Research Institute</a>, who did a tour of tech and business podcasts in 2023 sharing the gospel of AI risk. He claimed we were already too far past the point of no return, and within just a few years, there was a 95% chance of an unaligned artificial superintelligence. In other words, an AI that would have no qualms about killing humans, either because it desired to do so or because it simply didn’t care, and whatever goal it might have would kill us off by chance.</p><p>Eliezer moonlights as a science fiction writer, so he has no shortage of creative stories about how this might manifest, and he comes off as relatively credible, if alarmist, in the scenarios he paints. Others share his beliefs, if not on quite as short a timeline. I’ve asked my own helpful AI assistant, Claude, to create a plot of guests on the <a href="https://www.dwarkesh.com/">Dwarkesh podcast</a> by their predictions for p(doom) and timeline for AGI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*0pizpP6G2mNoslOf" /></figure><p>I’m not going to get into too deep an analysis of these claims. On the surface, if you believe we will have human-level intelligence soon and then you could apply that human-level intelligence at scale. Imagine a million automated phd AI researchers with perfect ability to share information, applying themselves to improving AI, you could imagine that accelerating things. Then if you imagine that we do create these super-intelligences, why would they care about humans? Maybe they would view us as their creators and treat us nicely. Maybe they would treat us like humans treated our evolutionary ancestors, have you run into any Neanderthals lately?</p><p>Anyway, I guess I have a hard time believing this will happen so quickly. That’s not to say I don’t think AI will be useful; there is clearly a great deal of value in even the tools we have today. Maybe I’m just a <a href="https://www.forbes.com/sites/hillennevins/2021/11/24/the-parable-of-the-turkey/">turkey 3 weeks before thanksgiving</a> but I can’t imagine we’ll actually have AGI in a couple years.</p><p>Do you want to hear my little pet conspiracy theory? What if the doomerism is actually just brilliant marketing?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/410/0*9Hejlt7Y48u2od0e" /></figure><p>Hear me out. If you’ve got Sam Altman saying GPT 5 scares him and <a href="https://www.linkedin.com/in/dario-amodei-3934934/">Dario Amodei</a> from <a href="https://www.linkedin.com/company/anthropicresearch/">Anthropic</a> claiming 50% of entry level white collar jobs will be lost to AI, those are bold claims.</p><p>BUT, what if some other very credible people are making MUCH more bold claims. All the sudden Sam and Mario’s claims seem much less wild, maybe I’ll cut them each a check for a couple billi 💰💰💰…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/443/0*bSAaYWe3Z_D57JYS" /></figure><p>Another dimension of this is that PR is BACK as a growth strategy, perhaps to an extent we’ve not seen since the dot-com era, if not ever. And for tech companies to get headlines in the mainstream media, you know what gets the clicks? Fear.</p><p>So is it all just a marketing psyop? Maybe a little bit.</p><p>That said, and just to be safe, I, for one, welcome our new robot overlords.</p><p>Yours,</p><p>Paul Dudley</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c4952d4e68c5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Context is King]]></title>
            <link>https://medium.com/@streamkap/context-is-king-728067ce2cfe?source=rss-53e191a4c46a------2</link>
            <guid isPermaLink="false">https://medium.com/p/728067ce2cfe</guid>
            <dc:creator><![CDATA[Streamkap]]></dc:creator>
            <pubDate>Wed, 30 Jul 2025 15:06:35 GMT</pubDate>
            <atom:updated>2025-07-30T15:06:35.869Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qXBNwwVeu34sKJ1K.png" /></figure><p><em>Originally published on July 11th, 2025, on </em><a href="https://streamkap.substack.com/p/context-is-king"><em>Streamkap’s Substack.</em></a></p><p>Last month at <a href="https://www.snowflake.com/en/summit/save-the-date/">Snowflake Summit</a>, Sam Altman anchored the opening keynote with a fireside chat. AGI timelines were discussed, and he, as you would expect, emphasized just how quickly things are moving (as is our bar for what AGI actually is).</p><p>One thing that stood out to me was how <a href="https://youtu.be/gJf39VG87O8?feature=shared&amp;t=1744">context is beginning to become more and more important</a>, even with today’s models, longer context length, memory, and improved retrieval mechanisms will greatly expand the impact that they will have.</p><p>Shopify CEO Tobi Lutke was also <a href="https://x.com/tobi/status/1935533422589399127">singing the song of context</a> later in the month, highlighting that “context engineering” was a better framing than prompt engineering. We’re moving beyond trying merely to coax the correct answer from the underlying model and instead ensuring we are providing it with the information with which to provide the correct answer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uk6sd1XjjZ9Qvin1.png" /></figure><p>This tracks with a bit of what we’ve seen in some of the acquisitions we’ve seen this year. Companies that hold core enterprise data, e.g. Salesforce, ServiceNow, have bought companies that help manage metadata, <a href="https://www.salesforce.com/news/press-releases/2025/05/27/salesforce-signs-definitive-agreement-to-acquire-informatica/">Informatica</a>, and <a href="https://techcrunch.com/2025/05/07/servicenow-acquires-data-world-two-months-after-acquiring-moveworks/">data.world</a> respectively. The metadata provides context on the underlying data, it helps provide LLMs a guide to where to find things, which data to trust, and so on.</p><p>Our previous discussion of <a href="https://streamkap.substack.com/p/just-semantics">semantic layers</a>, inspired in large part by <a href="https://www.linkedin.com/in/david-jayatillake/">David Jayatillake</a>’s writing on this, is related.</p><p>Salesforce and ServiceNow want to give AI a map.</p><p>Interestingly, we’re also starting to see another trend, the hoarding of data. Does it benefit Salesforce to make its CRM or Slack chat data easily accessible by 3rd party services? It does not. They announced recently a <a href="https://www.computerworld.com/article/4005509/salesforce-changes-slack-api-terms-to-block-bulk-data-access-for-llms.html.">change to Slack API terms</a> to block bulk data access for LLMs.</p><p>Salesforce wants Agentforce to be your go-to for understanding the data they manage for you.</p><p>I understand the motivation, but it may hurt them in the long run. Enterprises do not want their data locked away in walled gardens. We’ve seen this with the trend to <a href="https://streamkap.substack.com/p/iceberg-dead-ahead">open table formats recently</a>.</p><p>Version one of this challenge was the analytics stack, where companies go to great efforts to integrate disparate data sources into a single central data model in their data warehouse/data lake to power analytics.</p><p>Most agree that AI will be a much more significant unlock of the value of that data than internal analytics. The most powerful AI will be those with access to ALL of the enterprise’s data, not just the data in one service provider. I can’t help but wonder if by trying to keep all their data proprietary, these platforms will lose out to new contenders built around open data platforms.</p><p>On a related note, one of the questions we’re getting from customers is how they can start to integrate real-time context in AI services for their customers.</p><p>If you’re building an AI agent, do you want to reference static docs/faq or do you want to take into account the recent behavior of a shopper on your website or a the error that just affected someone on your software?</p><p>We believe that this real-time context will be critical to agentic workflows and have already seen significant impact for customers.</p><p>One thing we can be sure of, data and metadata are going to be a critical battleground in an AI, because context is king.</p><p><strong>Warmly,<br>Paul Dudley</strong></p><p>— — — -</p><p>P.S., we previously talked about the shift towards forward-deployed engineers, services is now <a href="https://www.linkedin.com/feed/update/urn:li:activity:7336046835096014848/">hot in silicon valley</a>. This also ties back to context, services allow AI to be tailored based on the right underlying data sources to ensure quality output.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=728067ce2cfe" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[HTAP is dead, long live HTAP]]></title>
            <link>https://medium.com/@streamkap/htap-is-dead-long-live-htap-8972f6dc93ca?source=rss-53e191a4c46a------2</link>
            <guid isPermaLink="false">https://medium.com/p/8972f6dc93ca</guid>
            <dc:creator><![CDATA[Streamkap]]></dc:creator>
            <pubDate>Wed, 23 Jul 2025 15:32:35 GMT</pubDate>
            <atom:updated>2025-07-23T15:32:35.026Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mSar7TJJcA1SvQogm9m_cQ.png" /></figure><p><em>Originally published on June 10th, 2025, on </em><a href="https://streamkap.substack.com/p/htap-is-dead-long-live-htap"><em>Streamkap’s Substack</em></a><em>.</em></p><p>Reporting live in the midst of 2 weeks of the major data conferences, Snowflake Summit and Data and AI Summit (Databricks).</p><p>Today, we’re talking PostgreSQL. The lingua franca of OLTP (online transaction processing, or the core databases that applications and services we use are generally built on).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/0*U35UYjYNIATfqaC_.png" /></figure><p>Databricks announced its <a href="https://www.databricks.com/company/newsroom/press-releases/databricks-agrees-acquire-neon-help-developers-deliver-ai-systems">$1B acquisition of Neon</a>, a 4-year-old serverless Postgres database company, back in May, stating that it would provide “An Open, Serverless Foundation for Developers and AI Agents.” Interestingly, this is not Databricks’ first serverless Postgres acquisition, they bought <a href="http://bit.io/">bit.io</a> <a href="https://www.databricks.com/blog/welcoming-bit-io-databricks-investing-developer-experience">back in 2023</a>. As far as I know, that didn’t lead to any products in Databricks.</p><p>What’s the difference with Neon? AI. Earlier this year, Neon was posting hockey stick graphs of the number of databases created, not by humans, but by AI agents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/0*DTZrl9SGsKqkqeaJ.png" /></figure><p>Databricks is betting that as AI creates apps and services, the underlying databases will be created in serverless offerings like Neon, Supabase, and Crunchy Data.</p><p>Speaking of Crunchy Data, Snowflake <a href="https://www.linkedin.com/feed/update/urn:li:activity:7335403778226995201/">announced on June 2</a> that they were acquiring Crunchy Data for $250M. Crunchy Data has been around longer and isn’t quite as hot as Neon, but I have heard they have more actual revenue.</p><p>Snowflake was probably in the Neon discussions as well, so it’s likely that this Crunchy Data announcement is v2 of the disappointment Snowflake had last year with their attempted <a href="https://streamkap.substack.com/p/iceberg-dead-ahead">acquisition of Tabular</a>. In both cases, the advantages of highly valued private stock won out.</p><p>Databrick’s latest valuation was $62B, they can issue shares for the acquisition, take some dilution, but not affect their cash position. Snowflake is subject to scrutiny of the public markets, so it’s much harder for them to make an acquisition that might be viewed as overvalued. While few private market companies can play this game (OpenAI comes to mind), it’s interesting to see being private as an advantage in acquisitions.</p><p>Now, both Databricks and Snowflake have an OLTP offering and are both obviously giants in analytics workloads or OLAP (online analytical processing). Does this mean a resurgence of HTAP (Hybrid Transactional and Analytical Processing or get you a database that can do both). HTAP offers some great promises, freedom from the latency and complexity of traditional ETL processes, chief among them, but has generally failed to deliver.</p><p>If we look a bit closer at Neon, there are clues to the potential here. Neon’s founder, <a href="https://www.linkedin.com/in/nikitashamgunov/">Nikita Shamgunov</a>, previously founded Singlestore (fka MemSQL), which is the most broadly adopted HTAP database (or at least the most adopted independent one). So he knows HTAP.</p><p>So why hasn’t HTAP worked? Snowflake promised Unistore a few years back, which has been renamed Hybrid Tables, but I know of very few companies using it. Singlestore has been reasonably successful but hasn’t achieved broad adoption.</p><p>The big reason may be more organizational than technical. OLTP is the center of the technical platform, so OLTP performance is the first choice that companies make when building a platform. OLAP is important, but it’s a decision that is made later and works around the core business rather than as the center.</p><p>Engineering teams are generally getting what they need from their OLTP databases, it’s the data teams that are not getting what they need from OLTP, thus OLAP. So HTAP is really mostly solving problems for data teams. That’s all well and good but if we refer back to who decides about OLTP, it’s the engineering teams building the core product, these folks are a.) likely not thinking about the OLAP concerns (for better or worse) and b.) not going to accept performance trade-offs to solve problems that are not their own.</p><p>Coming back around to today, will companies move their OLTP from Amazon RDS or Oracle to Neonbricks or Crunchflake? I don’t think so. Databricks and Snowflake are betting that apps that are spun up by AI coding tools like Lovable and Bolt.new will run on their serverless Postgres offerings, getting them in early. As long as the OLTP performs well, then there will be little motivation to migrate (and significant difficulty to do so even if the desire is there).</p><p>Phase two of this strategy is likely to ensure that the Postgres storage for these services is in Iceberg format so it can be read directly by the analytical services of Snowflake and Databricks.</p><p>Will we see the AI and analytics platforms become the center of gravity for OLTP workloads? I guess we’ll find out!</p><p>Oh yeah, and in Cursor news, they announced this week that they hit $500M ARR and raised a $900M Series C less than 2 months after hitting $300M ARR in April 🤯.</p><p><strong>Warmly,<br>Paul Dudley</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8972f6dc93ca" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Migrating Data from Batch Ingestion to Streamkap: A Technical Deep Dive]]></title>
            <link>https://medium.com/@streamkap/migrating-data-from-batch-ingestion-to-streamkap-a-technical-deep-dive-17063b6d177b?source=rss-53e191a4c46a------2</link>
            <guid isPermaLink="false">https://medium.com/p/17063b6d177b</guid>
            <dc:creator><![CDATA[Streamkap]]></dc:creator>
            <pubDate>Fri, 18 Jul 2025 18:56:56 GMT</pubDate>
            <atom:updated>2025-07-18T18:56:56.811Z</atom:updated>
            <content:encoded><![CDATA[<p><em>We thank </em><a href="https://www.linkedin.com/in/dcorle/"><em>Daniel Corley</em></a><em>, Data Engineer at SpotOn, for writing this blog post.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6K9JX2M2GU01j7rlP518bQ.png" /></figure><p>At <a href="https://www.spoton.com/">SpotOn</a>, we had an existing pipeline that synchronized data from our production MongoDB to Snowflake. During peak demand, our customers require data to be as close to real-time as possible. To improve our latency (and reduce the cost of our maintenance and infrastructure), we started to look for a different solution.</p><p>After researching our options, we decided to migrate our highest volume connectors from a <a href="https://streamkap.com/blog/spoton-case-study">batch ingestion process to Streamkap</a>. Our dbt project has over 2,000 models and the entire process took about a month and a half, which involved transferring the data, refactoring all previous dbt data models to accommodate the new data source, ensuring data validation, and efficiently handling Change Data Capture (CDC). One of the most complex aspects of this migration was refactoring the dbt snapshot models to maintain the integrity of historical data while integrating the new data source.</p><p>The benefits of moving from batch ingestion to Streamkap were substantial. Streamkap provided us ultra-low latency via Snowflake streaming, which means data is available for analysis almost in real-time. Additionally, due to Streamkap’s efficient data handling and integration capabilities, we experienced lower ingestion costs and reduced Snowflake compute costs.</p><h3>Refactoring DBT Data Models</h3><p>After migrating the data to Streamkap, the first task was refactoring all existing DBT data models to point to the new source data. This involved several key steps:</p><ol><li><strong>Updating Source References:</strong> All DBT models previously pointed to data ingested from batch ingestion as the data source. We had to update these references to point to Streamkap instead. This was a straightforward but necessary step to ensure that our models pulled data from the correct source, and did not disrupt our business user’s workflows.</li><li><strong>Validating Data Types:</strong> Given the differences in how batch ingestion and Streamkap handle certain data types, we meticulously validated all data types to ensure consistency. This involved checking the schema and data types in Snowflake and updating our DBT models accordingly.</li><li><strong>Handling CDC Data:</strong> With CDC, every time a record is modified in the database, said record is inserted to Snowflake. While this is efficient to load as we do not have to spin up a Snowflake warehouse to upsert, we had to take great care to ensure we always worked with the most recent state of a record. We utilized subqueries to efficiently scan tables using Snowflake column metadata compared to using qualify with window functions which became costly with high velocity datasets.</li></ol><pre>{{ config(materialized=view) }}<br><br>select<br>  _id<br>  , _streamkap_source_ts_ms<br>  -- all other columns<br>from {{ source(&#39;source_schema&#39;, &#39;source_table&#39;) }}<br>where<br>  -- get most recently inserted record<br>  and (_id, _streamkap_offset) in (<br>    select<br>      _id<br>      , max_by(_streamkap_offset, _streamkap_source_ts_ms) as max_streamkap_offset_by_source_ts_ms<br>    from<br>      {{ source(&#39;source_schema&#39;, &#39;source_table&#39;) }}<br>    group by 1<br>  )</pre><h3>Engineering DBT Snapshot Models</h3><p>With CDC streams, snapshots are no longer needed since all changes to a record are sent directly to the data warehouse, giving us the ability to view a full changelog. However, we still needed to ensure users could view historical data as it was at any point in time. One of the biggest challenges during our migration was integrating the historical data from our existing DBT snapshot models with the new CDC data. The main goal was to maintain the integrity of the historical snapshot data while adding new data from Streamkap. Here’s how we did it, step by step:</p><h3>Steps to Refactor Snapshot Models</h3><ol><li><strong>Common Table Expressions (CTE):</strong></li></ol><p>The CTE was used to preprocess Streamkap data, adding necessary fields and ensuring data was in the correct format for DBT snapshots. This step was crucial for standardizing the data and making it compatible with our existing snapshot models.</p><ul><li>We started by creating a CTE to handle data from the new Streamkap source.</li><li>This CTE included all relevant fields from the Streamkap data, along with additional derived fields required for unioning with DBT snapshots, such as a unique identifier (dbt_scd_id) and timestamps (dbt_updated_at and dbt_valid_from).</li></ul><ol><li><strong>Generating Unique Identifiers and Timestamps:</strong></li></ol><p>Creating a unique identifier involves concatenating key fields and hashing them. This ensured that each record could be uniquely tracked across different sources. Timestamps were standardized using conversion functions, allowing us to handle dates and times consistently.</p><ul><li>We concatenated relevant fields and applied an MD5 hash to derive a dbt_scd_id for each record, ensuring uniqueness.</li><li>We calculated the dbt_updated_at ( _streamkap_ts_ms) timestamp by converting the timestamp from Streamkap to a standard format.</li><li>The dbt_valid_from (_streamkap_source_ts_ms) timestamp was determined using the earliest occurrence of the record, ensuring accurate tracking of when each record was first seen.</li></ul><ol><li><strong>Integrating Existing Snapshot Data:</strong></li></ol><p>The UNION ALL operation combined historical data from batch ingestion with new data from Streamkap. This approach preserved the continuity of our snapshots and ensured that no data was lost during the transition.</p><ul><li>The next step was to combine the existing snapshot data (originally from batch ingestion) with the new data from Streamkap.</li><li>We used a UNION ALL operation to merge these datasets. This allowed us to preserve historical data from the existing snapshots while integrating new records from Streamkap.</li></ul><ol><li><strong>Handling Historical Data:</strong></li></ol><p>Using a window function, we derived the activity windows based on timestamps from each record. This function looked ahead to the next record’s activity timestamp, to derive dbt_valid_from and dbt_valid_to windows.</p><ul><li>We ensured that historical records retained their original dbt_valid_to timestamps for the existing snapshot data.</li><li>For the new Streamkap data, we calculated the dbt_valid_to timestamp using a window function. This function looked at the next available dbt_valid_from timestamp for each record, ensuring that the timeline for each record was continuous and accurate.</li></ul><p><strong>Sample of Snapshot Union:</strong>‍</p><pre>{{ config(materialized=&#39;view&#39;) }}<br><br>--migration took place on 2024-05-28, which is represented in hardcoded values below<br><br>with streamkap_cte as (<br><br>  select<br>    _id<br>    , &#39;streamkap&#39; as source<br>    , md5(concat(_id, _streamkap_source_ts_ms)) as dbt_scd_id<br>    , dateadd(millisecond, _streamkap_ts_ms, &#39;1970-01-01&#39;::timestamp) as dbt_updated_at<br>    --required to set dbt_valid_from/to in cases when the streamkap data is the first<br>    --time we see a record for the entity, otherwise the dbt_valid_from/to will always<br>    --be set to &#39;2024-05-28&#39; when it might not have been seen for the first time till<br>    --much later<br>    , iff(<br>      row_number() over (partition by _id order by _streamkap_source_ts_ms) = 1<br>      , greatest(<br>        &#39;2024-05-28&#39;::timestamp<br>        , dateadd(millisecond, _streamkap_source_ts_ms, &#39;1970-01-01&#39;::timestamp)<br>      )<br>      , dateadd(millisecond, _streamkap_source_ts_ms, &#39;1970-01-01&#39;::timestamp)<br>    ) as dbt_valid_from<br>  from {{ source(&#39;source_schema&#39;, &#39;source_table&#39;) }}<br>  where<br>    not coalesce(__deleted, false)<br>    and dateadd(millisecond, _streamkap_source_ts_ms, &#39;1970-01-01&#39;::timestamp) &gt;= &#39;2024-05-29&#39;<br><br>)<br>select<br>  _id<br>  , &#39;snapshot&#39; as source<br>  , dbt_scd_id<br>  , dbt_updated_at<br>  , dbt_valid_from<br>  , coalesce(dbt_valid_to, &#39;2024-05-28&#39;) as dbt_valid_to<br>from {{ ref(&#39;existing_batch_snapshot&#39;) }}<br>where<br>  not coalesce(_deleted, false)<br>  and dbt_valid_from &lt;= &#39;2024-05-28&#39;<br><br>union all<br><br>select<br>  *<br>  , lead(dbt_valid_from) over (partition by _id order by dbt_valid_from) as dbt_valid_to<br>from<br>  streamkap_cte</pre><h3>Overall Results</h3><p>The consequences of our migration were substantial. After the transition to Streamkap, we had:‍</p><ul><li><strong>Simplified Data Pipelines</strong>: This project focused on a core dataset that needed to be ingested into our data warehouse and other databases used by customer-facing applications. Streamkap allows one source to send data to multiple destinations easily.</li><li><strong>Ease of Streaming</strong>: As the organization saw how easy it was to set up and maintain streaming pipelines, more use cases emerged. This opened up new opportunities for both internal analytics and customer-facing reporting.</li><li><strong>Lower Cost</strong>: Switching to Streamkap cut our data infrastructure costs by 3x.</li><li><strong>Improved Data Capabilities</strong>: Previously, the dbt snapshot schedule limited how well we could track changes in records. With streaming data, we can now see all changes to the state of a record at any point in time.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=17063b6d177b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why to stream (s)now]]></title>
            <link>https://medium.com/@streamkap/why-to-stream-s-now-579566a7059c?source=rss-53e191a4c46a------2</link>
            <guid isPermaLink="false">https://medium.com/p/579566a7059c</guid>
            <dc:creator><![CDATA[Streamkap]]></dc:creator>
            <pubDate>Wed, 16 Jul 2025 10:58:33 GMT</pubDate>
            <atom:updated>2025-07-16T10:58:33.715Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lgiMBiA3vWz0gRSXFXvoAg.png" /></figure><p><em>Originally published on May 30th, 2025, on </em><a href="https://streamkap.substack.com/p/why-to-stream-snow"><em>Streamkap’s Substack.</em></a></p><p>With <a href="https://www.snowflake.com/en/summit/">Snowflake Summit</a> coming up, what better time to look at Snowflake’s place in the streaming landscape.</p><p>But wait, you say, Snowflake is a batch data warehouse.</p><blockquote><em>What does it have to do with Streaming?</em></blockquote><p>The <a href="https://streamkap.substack.com/p/the-first-ever-streamkap-newsletter">rumoured acquisition</a> of RedPanda seems to have fizzled so what gives?</p><p>I wrote <a href="https://streamkap.substack.com/p/streaming-tam-expansion">previously</a> about how streaming would become the dominant data processing approach, vs batch, with a combination of it getting easier, cost-competitive, and more market forces pushing towards real-time. All of which are happening.</p><p>How does Snowflake fit in? Well, to start, they are one of the most popular data warehouses (erm, excuse me, Data and AI Clouds), this means that if Snowflake makes streaming easier, it impacts a huge number of existing data teams who are already Snowflake customers.</p><p>When we started Streamkap in 2022, your best option for streaming data to Snowflake was Snowpipe. Launched in 2018, Snowpipe continuously loads data into Snowflake by automatically detecting new files in cloud storage and using predefined COPY commands to ingest them in near real-time.</p><p>This worked fine, and still does for files in particular, but for event data or database change data capture, it was more complicated and expensive than other options. This meant it was reserved for cases where you really needed to use it. It was also only good for latency in minutes.</p><p>In late 2022, a new feature went into Private Preview, Snowpipe Streaming. Streamkap started testing very early on and working with the product team developing Snowpipe Streaming, led by <a href="https://www.linkedin.com/in/xinyhuang/">Xin Huang</a>.</p><p>This was a game changer (though I wish they had come up with a more unique name, it’s easy to confuse with Snowpipe even if you write out the full name every time). Snowpipe Streaming introduced true streaming, there was no staging in S3 and batching in, effectively a Kafka consumer inside Snowflake that streams messages directly into Snowflake as a serverless process that does not require a warehouse.</p><p>This meant, for raw data, end-to-end latency of a few seconds from the source database, through Streamkap, into Snowflake. The only caveat was that it’s append-only, we’ll come back to that in a bit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/0*suun9_3mKIWGsb02.png" /></figure><p>Credit <a href="https://www.upsolver.com/blog/snowpipe-streaming">Upsolver</a></p><p>So Snowpipe Streaming is fast, what about costs? Well, I’ve got good news for you there as well. Compared to Snowpipe, it’s <a href="https://www.snowflake.com/en/blog/data-ingestion-best-practices-part-three/">up to 92% cheaper</a>. Compared to batch loading processes, it depends on frequency, but with an ETL tool like Fivetran which uses COPY INTO, we’ve seen Snowpipe Streaming be 100x cheaper. There are some dependencies but generally, it works out to around <a href="https://streamkap.com/blog/snowflake-snowpipe-streaming-with-change-data-capture-cdc">$30 per Terabyte</a>!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FRuOu4Nh4Zk_9PrY.png" /></figure><p>This is the kind of shift that has data teams looking at streaming in a new light. In terms of Snowflake credit consumption, it’s less expensive to have your data available in Snowflake in seconds via streaming than every 15 minutes with a batch ETL tool. Add savings on the ETL tool as well, and you’re really cooking.</p><p>I did mention a caveat above, Snowpipe Streaming is append only. That means that if you’re doing CDC on a database and getting update events, you’ll need to do some processing on that raw table to create the current state view of the source. That’s where the next innovation comes in… Dynamic Tables.</p><p>Dynamic Tables came out a few months after Snowpipe Streaming as a wonderful complement to streaming ingestion. Dynamic Tables automatically refresh and materialize the results of SQL queries into a target table. The magic is that you simply set a target latency, currently as fast as 1 min*, and Snowflake figures out the most efficient path to get there. Dynamic Tables simplify data pipeline logic by letting you define transformations as SELECT statements, with Snowflake handling dependencies, freshness, and incremental updates.</p><p>If you combine Snowpipe Streaming and Dynamic Tables, the impact on latency and cost is huge.</p><p>Our customer, SpotOn, switched from batch ETL data loading and incremental debt modeling to Snowpipe Streaming and Dynamic Tables to achieve 4x faster end-to-end latency AND 3x lower total cost of ownership.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/0*hngxugrt7k-xLJPP.png" /></figure><p><strong>Step-by-step guide:</strong></p><p><strong>Ingestion:</strong> A PostgreSQL database generates Change Data Capture (CDC) records from its transaction log. Streamkap captures these changes and then <strong>uses</strong> Snowpipe Streaming to send these CDC records directly into Snowflake, populating staging tables.</p><p><strong>Transformation:</strong> Snowflake Dynamic Tables are then defined with SQL queries that read from the staging tables (which are continuously updated by Snowpipe Streaming). These Dynamic Tables perform transformations — which can include joins between the staging tables, aggregations, or filtering — to derive a Target Table. (A more effective approach, however, is often to perform as many transformations as possible earlier in the pipeline, for instance, using Streamkap hehe, leaving only necessary transformations for this stage.)</p><p><strong>Automatic Refresh:</strong> Snowflake automatically detects new data in the staging tables and triggers an incremental refresh of the dependent Dynamic Tables based on their defined target lag.</p><p>If you’d like to learn more about this, we did a <a href="https://www.linkedin.com/events/streamkaplivedemoandq-a-streami7211199547971137536/theater/">webinar with Marcin Migda from SpotOn</a> and have written a summary <a href="https://streamkap.com/blog/spoton-case-study">case study of their results</a>.</p><p><strong>Warmly,</strong></p><p><strong>Paul Dudley</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=579566a7059c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The RJMetrics Mafia?]]></title>
            <link>https://medium.com/@streamkap/the-rjmetrics-mafia-6be8e1408132?source=rss-53e191a4c46a------2</link>
            <guid isPermaLink="false">https://medium.com/p/6be8e1408132</guid>
            <dc:creator><![CDATA[Streamkap]]></dc:creator>
            <pubDate>Wed, 09 Jul 2025 12:06:54 GMT</pubDate>
            <atom:updated>2025-07-09T12:06:54.063Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UZNo3RoGBs9iKlXA.png" /></figure><p><em>Originally published on May 22d, 2025, on </em><a href="https://streamkap.substack.com/p/the-rjmetrics-mafia"><em>Streamkap’s Substack.</em></a></p><p>Everyone in the startup world has heard of the PayPal mafia, the founders and early employees of PayPal and associated companies. The dons of this mafia include Elon Musk, Peter Thiel, Reid Hoffman, and others. They have collectively created companies valued at over $2 Trillion including Tesla, SpaceX, LinkedIn, Palantir, YouTube, and Yelp.</p><p>More recently Palantir itself has gained quite a reputation as a <a href="https://apoorv03.com/p/palantir-the-founder-foundry">founder factory</a>, spawning over 100 companies that have raised over $11B and include household names like Affirm, and Partiful and data startups like Hex, which we talked about <a href="https://streamkap.substack.com/p/the-rebundling">a couple weeks ago</a>.</p><p>Speaking of the data world. Do we have our own version of a ‘mafia’? Perhaps…</p><p>If we look back to 2008, <a href="https://www.linkedin.com/in/jakestein/">Jake Stein</a> and <a href="https://www.linkedin.com/in/robertjmoore/">Bob Moore</a> had founded a little analytics company called RJMetrics in Bob’s attic in Philadelphia.</p><p>As an aside, we like to think of Silicon Valley as the center of innovation in America but Philly’s innovation roots run deep, Benjamin Franklin was tinkering with kites and keys and inventing bifocals, swim fins, and modern democracy in the City of Brotherly Love in the 1700s. I digress.</p><p>Bob and Jake set out to “inspire and empower data-driven people” by integrating data spread across a wide variety of disparate systems (sadly still a challenge). This was well before cloud data warehouses, in fact, AWS itself had only existed for 2 years at that point. GCP launched that year, Azure was still 3 years away.</p><p>Bob and Jake had to build a full-stack analytics platform, from data ingestion all the way to dashboards, in a SaaS tool. I’m not an expert in the landscape at the time but it was pretty revolutionary, the alternatives were primarily on-prem enterprise software which were non-starters for smaller businesses.</p><p>RJMetrics was a pioneer, a proto-modern data stack, and it would come to have a big impact on the market. Bob and Jake bootstrapped for several years and built a small team, including <a href="https://www.linkedin.com/in/merrickchristopher/">Chris Merrick</a> who joined as an engineer but would later be promoted to VP engineering.</p><p>In 2012, they raised external capital, a $1.2M seed round. Then, they quickly raised a $6.2M A round in 2013 and a $16M B round in 2014. With this funding, the product advanced into modules we recognize today: an Analytical Warehouse, Transformation Cluster, and Visualization Interface — all in one platform.</p><p>It was also during this period that they hired <a href="https://www.linkedin.com/in/tristanhandy/">Tristan Handy</a>, a name you might recognize from <a href="https://streamkap.substack.com/p/the-modern-data-stack-grows-up">past posts</a>, to lead marketing.</p><p>Unfortunately for RJMetrics, another trend was emerging at the same time, Redshift was launched in 2013, Snowflake in 2014, and it became more attractive to run your data workloads on your own dedicated data warehouse. Moore referred to increasing customer acquisition costs leading to a 25-person layoff in early 2016 <a href="https://technical.ly/startups/rjmetrics-ceo-bob-moore-explains-layoffs-failfest/">announced</a> alongside an increased focus on the “Pipeline” product (foreshadowing things to come).</p><p>In parallel with those rocky times in early 2016 at RJMetrics. Tristan Handy <a href="https://www.getdbt.com/blog/goodbye-rjmetrics-hello-fishtown-analytics">began bootstrapping</a> Fishtown Analytics, a data consultancy, alongside other RJMertrics folks, Connor McArthur and Drew Banin, (and occasionally Yev Meyer.)</p><p>The RJ/Fishtown crew also started working on dbt, short for data build tool, an open-source project for data transformation on cloud warehouses. Chris Merrick, then VP of engineering at RJ <a href="https://github.com/dbt-labs/dbt-core/commit/fd7567fb4dd60530a512f3e6218050cb66f0dcea">wrote the first lines of code</a> to <a href="https://www.getdbt.com/blog/on-two-years-of-dbt">Drew Banin’s joking frustration</a>.</p><p>The combination of struggles with the core RJMetrics business and the success of the “Pipelines” product led to a few key events that summer. Magento (now Adobe Commerce) acquired the RJMetrics BI/Analytics product and business, which was rebranded to Magento Analytics. Robert Moore would join Magento to lead that product.</p><p>A key part of the Magento deal was that the Pipeline product would be spun out as a new standalone company called Stitch. Jake Stein would lead Stitch as CEO and Chris Merrick would join as CTO while Bob remained involved in a chairman role.</p><p>We now had all the core elements of the early data stack available as independent solutions. Stitch for ETL (competing with Fivetran), Cloud data warehouses in Redshift, Snowflake, and BigQuery, dbt for transforming data in the warehouse, and standalone cloud BI solutions like Looker (founded in 2012).</p><p>The next act in this play was Stitch’s <a href="https://medium.com/@jakestein/introducing-singer-simple-composable-open-source-etl-a4a6da7eac19">release of Singer</a>, an open-source project for ETL, in 2017. They argued, as Airbyte later would, that open source is the best way to manage support for the huge number of connectors required to support all the SaaS apps in the market. Singer offered a standardized framework that a developer could build on to connect to any arbitrary API, database, etc.</p><p>Stitch would continue to support Singer as a managed service.</p><p>Around this time, the cloud data warehouses really started cooking, and ETL tools were hot. The sales teams at Snowflake, AWS/Redshift, and BigQuery were eager to send business to Fivetran, Stitch, and Matillion (the major players in those early days) because a warehouse wasn’t going to add much value unless you could get your data in. See Google trends for those three players below. Pre-2018, they were all neck and neck. This analysis likely underrates Stitch, as I had to add the qualifier “data” to the keyword to avoid other uses of its name. Side note, it would be interesting to figure out what allowed Fivetran to jump ahead in awareness above Matillion in 2022, but we’ll leave that for another day.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uHki0q2KNZtYap0L.png" /></figure><p>In addition to the partnership-driven growth, Stitch also did a great job with search engine optimization (SEO). ETL in particular lends itself to SEO, the specific problems that data have are very easily articulated with keywords, e.g. connect Salesforce to Redshift. Stitch did a good job of creating this content and ranked well. See metrics from SEMRush below, Stitch was absolutely crushing the competition in search until quite recently.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*e7yugcgIbbx7t7Xt.png" /></figure><p>From what I can tell, Stitch was doing pretty well at this time. They caught the attention of Talend, the leading on-prem open-source ETL company at the time, a public company. In November 2018, Talend announced the acquisition of Stitch for $60M cash. That doesn’t sound like a bad exit, but I do wonder if there was more to the story than simply a good exit. It might be that a complicated cap table due to Stitch having been a spin-out from RJMetrics made it difficult to raise money. At that point, Matillion had recently raised a $20M Series B. Fivetran was still pretty early and would announce a $15M Series A a month after the acquisition. So, whether it was just a good exit, cap table/funding challenges, competitive funding pressures, or some combination of the above. The Stitch team decided to sell.</p><p>For the first couple of years inside Talend, Stitch continued to grow. When I met with Data teams in 2020/2021, my sense was that Stitch had a greater market share than Fivetran, at least in terms of customer count.</p><p>That said, even then, the product and support had started to fall off. From the outside, Stitch didn’t seem to be a priority for Talend.</p><p>In 2021, Talend was acquired by PE firm Thoma Bravo, which really spelled the end for Stitch. The founding team had left by early 2022. In 2023, another Thoma Bravo-owned company acquired Talend, and it was really over for Stitch.</p><p>It seems that Stitch could have been Fivetran, but perhaps it was not to be. Regardless, the end was a new beginning, leading to the further expansion of the mafia.</p><p>Jake Stein left Talend in 2020 and started <a href="https://www.linkedin.com/company/common-paper/">Common Paper</a>, a startup focused on streamlining contracting process. Something I actually considered pursuing myself earlier in my career.</p><p>Chris Merrick left Talend in 2022 to start Omni with co-founders <a href="https://www.linkedin.com/in/colinzima/">Colin Zima </a>and <a href="https://www.linkedin.com/in/jamescdavidson/">Jamie Davidson</a> who where both early members of the Looker team. Google acquired Looker in 2019, and in a somewhat similar process to what happened to Stitch at Talend, allowed a well-loved product to lose at least some of it’s luster. So Chris, Colin, and Jamie, set out to create a better Looker and pick up the slack. Every indication from the outside is that it’s going well, they have raised a total of $97M, most recently in a $69M Series B, and are above $10M ARR run rate.</p><p>So that’s the Stitch side of the story. Almost…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/537/0*DRcgjsxP5phkefSI.png" /></figure><p>Back to Singer, the open-source project. It lives on. With Stitch disappearing into Talend/Qlik the data team at GitLab decided to spin out a project called Meltano which was built around Singer as a standalone company in 2021.</p><p>Around the same time, Airbyte came out of YC W20 cohort, they sold a compelling vision of open-source as the solution to data integration (sound familiar?) and my understanding is that they borrowed heavily from Singer, at the very least the connectors/taps, and perhaps more. Airbyte would quickly raise a $25M series A in May 2021 and, in what was perhaps the best example of Covid/ZIRP era VC/startup excess, raise a $150M series B at a $1.5B valuation. At the time Airbyte was reported to be making less than $1M in revenue, they top ticked the market with a 1500x revenue multiple.</p><p>Alas, Airbyte has yet to live up to that in the way that others in the modern data stack companies <a href="https://streamkap.substack.com/publish/posts/detail/157563045?referrer=%2Fpublish%2Fposts">have</a>. And with that, we can close out the Stitch chapter.</p><p>dbt is the big breakout star of the RJMetrics mafia. Tristan, Drew, and Connor continued on as Fishtown Analytics for the first few years, but dbt was gaining steam. In 2020, it took off.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/0*VLI4b4hrQLu8U5br.png" /></figure><p>What had been a consultancy was increasingly a tech company focused on managing an incredibly popular open-source project. Cut to 2021, and the Fishtown Analytics name was retired in favor of dbt Labs. dbt has since raised a total of $416M to date and recently announced that they <a href="https://www.linkedin.com/posts/pauldudley_dbt-moderndatastack-snowflake-activity-7295158669183172608-CBxW/">passed $100M revenue</a>.</p><p>Those are the heavy hitters of companies founded by RJMetrics employees or derived from associated projects.</p><p>It’s a bit outside the data world proper, but Bob Moore left Magento in 2018 and founded <a href="https://www.linkedin.com/company/getcrossbeam/">Crossbeam</a>, a partnership platform. Data is certainly the underpinning of the platform, as it’s effectively a data clean room for B2B partnerships, but it’s not in the modern data stack, other than facilitating partnerships between data vendors. We’re users of the platform at <a href="http://streamkap.com/">Streamkap</a>.</p><p>Some other former RJMetrics employees who have started companies include <a href="https://www.linkedin.com/in/michael-drogalis/">Michael Drogalis</a> of Shadowtraffic, a Kafka Traffic Simulation, and <a href="https://www.linkedin.com/in/andres-s-recalde/">Andres Recalde</a> of Muffin Data, a CPG Analytics platform.</p><p>If we total it up, RJMetrics alumni founders have raised over $800M!</p><p>Beyond founded companies, former RJMetrics folks are everywhere you look in data, at Rudderstack, Dagster, Snap, Google, Motherduck, Datadog, and others.</p><p>That’s my outside look at the RJMetrics story. Let me know if others are in contention. I have my eye on it.</p><p><strong>Warmly,<br>Paul Dudley</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6be8e1408132" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fivetran Pricing — Poor Execution or Smart Value Extraction?]]></title>
            <link>https://medium.com/@streamkap/fivetran-pricing-poor-execution-or-smart-value-extraction-b2edb7743862?source=rss-53e191a4c46a------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2edb7743862</guid>
            <dc:creator><![CDATA[Streamkap]]></dc:creator>
            <pubDate>Wed, 02 Jul 2025 16:13:57 GMT</pubDate>
            <atom:updated>2025-07-02T16:13:57.000Z</atom:updated>
            <content:encoded><![CDATA[<h3>Fivetran Pricing — Poor Execution or Smart Value Extraction?</h3><p><em>Originally published on May 15th, 2025, on </em><a href="https://streamkap.substack.com/p/fivetran-pricing-poor-execution-or"><em>Streamkap’s Substack.</em></a></p><p>If you follow me on LinkedIn, you know I am often critical of Fivetran’s pricing. They have conveniently made this criticism easy by making frequent and unpopular changes to the model.</p><p>As a founder, I am somewhat sympathetic. You have to run a business, and it’s important to try to align your pricing with the value that your customers get from your product.</p><p>As a customer advocate, I’m happy to criticise as their pricing hasn’t been great for customers, particularly in their traditional sweet spot of mid-market companies.</p><p>As a competitor, I love it because they’ve made some particular missteps which have made criticism particularly easy.</p><p>I was reminded of all of this last week when I saw a surprising <a href="https://www.linkedin.com/posts/ivanstegic_has-a-saas-every-lowered-your-monthly-price-activity-7326229967279706112-Vnog?utm_source=share&amp;utm_medium=member_desktop&amp;rcm=ACoAAAB55foBkgE2hMeWEA271SFlv96b2aL9QQU">post</a> from Ivan Stegic praising Fivetran for their pricing changes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/0*eMm3nBUfk13UfAFX.png" /></figure><p>This was surprising to me, I’ve spoken to a lot of Fivetran customers in the past few months and most had their <strong>bills go up 50 %-60 %</strong> and some as <a href="https://www.reddit.com/r/BusinessIntelligence/comments/1jvcmws/fivetrans_recent_price_changes_are_a_joke/"><strong>much as 2–4X</strong></a>. What is going on🤔? I think I have an answer, read below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/0*-EWIVYGCh3-AKyEE.png" /></figure><p>Before we dig into the latest change, let’s look at the history of Fivetran’s pricing. I’ll try to assess each model based on the customer’s and Fivetran’s perspectives.</p><p><strong>2012–2020 — Startup Era</strong></p><p>Fivetran was founded in 2012 when the dominant model for SaaS was subscription based. Naturally, they aligned to this and initially offered a subscription based on the number of connectors being used with no limitations on actual consumption within those models.</p><p><strong>Customer Perspective:</strong></p><blockquote><strong><em>Pros: </em></strong><em>Very predictable pricing, very attractive for high volume connectors</em></blockquote><blockquote><strong><em>Cons:</em></strong><em> Relatively expensive for low volume connectors, incentivises not bringing those low volume connectors onto the system. Not a good fit in general if a customer has a large number of low-volume connectors</em></blockquote><p><strong>Fivetran Perspective:</strong></p><blockquote><strong><em>Pros:</em></strong><em> Predictable revenue from customers, potentially easier justification of R&amp;D on new connector builds given fixed revenue commits</em></blockquote><blockquote><strong><em>Cons:</em></strong><em> This pricing model violated two very basic principles of pricing, you shouldn’t sell things for less than they cost and pricing should be aligned to the value your customers get from the product. On the first point, you could imagine a customer could have a single database connector with multiple TB of data driving real operational cost for Fivetran and the customer is only paying $150/month for that connector. That same customer is clearly deriving much greater value from those TBs of data being moved so they would presumably pay more than $150/month. On the other hand, as discussed earlier, that customer might have a hard time justifying adding their HR platform to Fivetran because it is low volume and not worth paying $150/month for.</em></blockquote><p>So, if I’m George Fraser, Fivetran’s CEO, in 2020. What do I see? I have a fast-growing business. My partnership with Snowflake is going well, and we’re riding along with their explosive growth. But my pricing is broken.</p><p>At the same time, usage-based pricing is growing in popularity. Twilio and others established in the mid-2010s that usage-based pricing worked well, and Snowflake had clearly established that it would work in the data world.</p><p>Customers liked it, at least initially, and it solved the two key problems above. Assuming you chose the correct metric for usage, when customers used more, they would get more value and be happy to pay more. Relatedly, you could ensure you were always charging enough to maintain good margins.</p><p>At this point, I think Fivetran made the correct decision to move to usage-based pricing. It was HOW they did it that caused issues for them.</p><p><strong>2020–2021 — Enter MAR (monthly active rows)</strong></p><p>The now infamous MAR enters the chat. Monthly Active rows, or MAR, measures the number of unique primary keys (rows) that are added, updated, or deleted from a customer’s source systems and synced to their destination by Fivetran connectors within a given month. On the surface this seems like a customer friendly approach to usage-based billing but it was not well received.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/0*P0RmRSrLL9x3keoG.png" /></figure><p><strong>Customer Perspective:</strong></p><blockquote><strong><em>Pros: </em></strong><em>Incentive to bring all data sources/connectors to Fivetran even if tiny volume.</em></blockquote><blockquote><strong><em>Cons:</em></strong><em> No more free rides for high volume connectors. Lack of predictability in costs. Unexpectedly high costs. MAR itself is confusing and expensive relative to the other popular SaaS ETL tool at the time, Stitch. In particular due to the following:</em></blockquote><ul><li><strong>Data Normalization:</strong> Fivetran’s process of normalizing data, especially from sources with nested or semi-structured data (like JSON), could significantly inflate the MAR count. A single record at the source might be transformed into multiple rows in the destination warehouse, each potentially counting towards MAR.</li><li><strong>Frequent Updates:</strong> Any update to an existing row, no matter how minor, would typically count that row as active for the month. For data sources with high update frequencies, this could lead to substantial MAR counts.</li><li><strong>Historical Syncs:</strong> While initial historical syncs for new connectors were often free, any subsequent large-scale re-syncs or historical backfills could generate significant MAR.</li></ul><p><strong>Fivetran Perspective:</strong></p><blockquote><strong><em>Pros: </em></strong><em>Better alignment to operational costs and customer value</em></blockquote><blockquote><strong><em>Cons:</em></strong><em> Negative customer reaction while new competitors emerging, unique branding (MAR) of an unpopular change, potentially poor alignment to R&amp;D effort, e.g. it’s harder to justify building and maintaining a new connector that will be low volume</em></blockquote><p>Fivetran likely made the right choice moving to usage-based pricing, but the specific execution turned them from a well-loved tool to one which customers begrudgingly used because alternatives were lacking.</p><p>With MAR, Fivetran tried to create its own proprietary pricing that was similar to Stitch but not directly equatable. Rather than tracking total rows ingested, MAR tracked net new rows. Their argument was that these net new rows were the ones that actually added value.</p><p>The problem for customers was it was not as transparent and straightforward to measure or compare for the reasons highlighted above. Stitch had its own problems but most of the folks I spoke to back in those days tended to prefer Stitch’s pricing model, which was to simply count all rows processed (new or otherwise) and to charge a lower rate for them.</p><p><strong>2021–2022</strong></p><p>There was no fundamental change in this period, but execution, how discounts are handled, and the lack of transparency of MAR continued.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/0*4oy8b_Ho384_n_JW.png" /></figure><p><strong>2023</strong></p><p>New tiers introduced, MAR is still the backbone. More historical discounts are being removed and upsetting customers:</p><p>Competitors like <a href="http://streamkap.com/">streamkap.com</a> are maturing and allowing for <a href="https://streamkap.com/blog/spoton-case-study">faster and cheaper data replication</a>.</p><p><strong>2025</strong></p><p>Now we come to this year. Fivetran is now over <a href="https://streamkap.substack.com/p/the-modern-data-stack-grows-up">$300M in revenue</a>, and as I wrote about earlier this year, <a href="https://www.linkedin.com/posts/pauldudley_fivetran-etl-mds-activity-7295873659422793728-ZAg1/?utm_source=social_share_send&amp;utm_medium=member_desktop_web&amp;rcm=ACoAAAB55foBkgE2hMeWEA271SFlv96b2aL9QQU">likely pursuing an IPO soon</a>. This is important context.</p><p>Late last year, I heard from some larger customers that Fivetran was offering an all-you-can-eat package. That sounds compelling; tell me more. The trick was that this was typically 50–100% more expensive than the customer was currently paying. How do you get them to accept that increase? You make them feel the pain of not taking it with increasing complexity in a sort of pincer move…</p><p><strong>Connection-based pricing:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vNgTDsRcoc-3m1-o.png" /></figure><p>Fivetran moves from an aggregate MAR measure across all of your connectors to counting MAR and discounting MAR at a connection level. It’s not immediately clear what this means for individual customers. Things just got even more complicated.</p><p>Fivetran was quick to point out that most of their customers save money, and I take them at face value that this is true. However, there is a VERY important caveat here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9I5GxLNWOKR7qXE6.png" /></figure><p>Fivetran has several thousand customers, how many of those customers do you think look like Ivan, who I quoted above? I would imagine there is a long tail of thousands of customers paying a few hundred dollars a month. Fivetran will happily save those customers 30 %-40 % on average because those customers don’t matter to their overall revenue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fMUA5k-ENb3VdGMR.png" /></figure><p>Fivetran is trying to go public. They are not going to introduce a pricing change that is going to produce a smaller total amount of revenue across their customer base. Fivetran needs to drive more large accounts. So, they are moving up market. They are charging more money to their mid-market and enterprise customers. This is where the pincer move comes in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/0*ldr2P7HGQh3HghLo.png" /></figure><p>You make the pricing more complicated and expensive for your customers paying $36k+ per year, but you offer them a way out of the complexity. Oh, and you won’t have to worry about those pesky MAR on your $100k bill if you just pay us $200k for an unlimited plan.</p><p>So if you’re a mid-market or enterprise company. Expect the squeeze to come. You are the cash cow paying for those long-tail customers like Ivan.</p><p>If you’re looking for an alternative where you can get faster, more reliable, more transparently and simply priced platform, check us out at <a href="http://streamkap.com/">streamkap.com</a> or reach out to me directly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/0*Om5ex5j7d5NXM3JJ.png" /></figure><p><strong>Warmly,<br>Paul Dudley</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2edb7743862" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>