<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[RudderStack - Medium]]></title>
        <description><![CDATA[With RudderStack, your whole business can trust and use customer data. Data teams ship faster and focus on higher leverage work, marketing teams get easy, on-demand access to the data they need to drive growth, and agents can deliver powerful, privacy-safe experiences. - Medium]]></description>
        <link>https://medium.com/rudderstack?source=rss----1840a84a9341---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>RudderStack - Medium</title>
            <link>https://medium.com/rudderstack?source=rss----1840a84a9341---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 18:08:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/rudderstack" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why You Should Care about Dimensional Data Modeling]]></title>
            <link>https://medium.com/rudderstack/why-you-should-care-about-dimensional-data-11ea860a95cd?source=rss----1840a84a9341---4</link>
            <guid isPermaLink="false">https://medium.com/p/11ea860a95cd</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[data-modeling]]></category>
            <dc:creator><![CDATA[Brian Lu]]></dc:creator>
            <pubDate>Thu, 03 Nov 2022 20:08:44 GMT</pubDate>
            <atom:updated>2022-11-03T20:08:44.032Z</atom:updated>
            <content:encoded><![CDATA[<p>It’s easy to overlook all of the magic that happens inside the data warehouse. At RudderStack, our conversations tend to focus on the challenges of getting data into the warehouse and on all of the products we build on top of our warehouse data. When we create architectural diagrams for RudderStack, we tend to represent the warehouse with a single icon. But this doesn’t do justice to all of the work that happens inside of the data warehouse.</p><p>In this post we’ll dive into dimensional data modeling to get a better appreciation of all the activity going on beneath the surface. We’ll also look at how dimensional data modeling shows up within the RudderStack application.</p><h3>What is dimensional data modeling?</h3><p>Dimensional data modeling is a way to structure data in a warehouse that <strong>mitigates the loss of data granularity </strong>and allows people to do data work in a <strong>performant</strong> way. It’s structured around two major principles:</p><ul><li><strong>Things that ARE</strong> are called <em>Dimension Tables</em> (e.g name, address, email, subscription status). These things may change slowly, but they mostly stay the same.</li><li><strong>Things that HAPPENED</strong> are called <em>Fact Tables</em> (e.g. clicks, purchases, etc.). This data is immutable (once it happened it happened!) but the tables can get very big as product usage increases.</li></ul><p>There can be cross-pollination between Dimension Tables and Fact Tables. For example, updating your address is an action that <em>happens</em> (fact) and changes the value for your profile (dimension). This is okay because dimension tables are efficient in aggregating or filtering data. This enables us to answer questions like “<em>how many subscribers live in Chicago?” </em>without having to duplicate the data everywhere.</p><h3>Getting data faster with denormalized tables and cubes</h3><p>You can do additional processing on top of dimensional data modeling to increase speed of access. Optimizing for speed does require sacrificing granularity, but as technology continues to improve, these tradeoffs become less consequential. Denormalized tables and OLAP cubes are the two ways to increase speed of access.</p><p>Building <strong>denormalized tables,</strong> or summary tables, on top of your dimensional data models enables faster performance, but it does require some sacrifice of granularity. For example, you can save the “<em>last 7 day purchases”</em> on a per-user basis in a single data table for fast access, but you’ll lose the ability to get “<em>last 8 day purchases”</em>. For many usage patterns, the speed is worth the tradeoff because users would rather see pre-baked data in 3 seconds than wait 30 seconds for customization.</p><p><strong>OLAP cubes</strong> are a more intensive option to increase speed of access. An OLAP cube pre-aggregates the data so much that lookup queries are near-instant. However, they require much more prep, and they sacrifice more granularity. Denormalized tables are a better way to satisfy performance for most use cases.</p><p>It’s also worth noting that newer technologies such as <a href="https://druid.apache.org/">Druid</a> and <a href="https://pinot.apache.org/">Pinot</a> can have extremely fast querying using a single table. This makes denormalized tables an appealing option if you choose to use these technologies since you don’t have to pre-aggregate. Check out <a href="https://datastackshow.com/podcast/building-pinot-for-real-time-interactive-user-analytics-with-kishore-gopalakrishna-of-startree/">this episode</a> of The Data Stack Show to learn more about how Pinot was purpose built for speed.</p><p>In addition to this, there are also in-memory implementations of data models that allow for fast data access. Tableau has <a href="https://www.tableau.com/about/blog/2014/7/understanding-tableau-data-extracts-part1">Tableau Data Extract</a> — first with the tde format and more recently <a href="https://www.tableau.com/products/technology">hyper</a> formats to enable fast access of large datasets. <a href="https://www.dremio.com/resources/guides/apache-arrow/">Arrow</a> is another in-memory approach that allows for data systems to be built with interactive performance.</p><p>This graph provides a rough representation of the speed vs. granularity tradeoff that’s central to dimensional data modeling. As speed increases, granularity decreases, and vise-versa. But what’s exciting to note is that as technology improves, the graph shifts further to the right. That is, we’re able to maintain more granularity at higher speeds. As each step gets faster, all of a sudden you can use the more granular technique for certain workloads and still meet user time expectations. This means a simpler data pipeline and less loss of granularity to achieve the same goals.</p><p>To drive the point home, consider the OLAP cube. OLAP cubes have largely fallen out of favor because recent advancements make denormalized tables a pretty good balance of performance and flexibility for most teams today.</p><h3>Showing data history with slowly changing dimensions and snapshots</h3><p>Sometimes it’s useful to see how data has changed over time. You may want to answer questions like:</p><ul><li>Have people moved in and out of audiences?</li><li>Who are the people who were subscribers last year but are no longer subscribers today?</li></ul><p>If you remember the <em>dimension</em> tables above, things change slowly, but they do change, and there’s often a great deal of value in understanding these changes. There are two techniques primarily used to show data history:</p><ul><li><a href="https://www.sqlshack.com/implementing-slowly-changing-dimensions-scds-in-data-warehouses/"><strong>Slowly Changing Dimensions</strong></a> — with this technique, you mark old rows as “stale” and add new rows. The most common technique is type-2 SCD, which works well for modern data pipelines.</li><li><a href="https://maximebeauchemin.medium.com/functional-data-engineering-a-modern-paradigm-for-batch-data-processing-2327ec32c42a"><strong>Snapshot</strong></a><strong> </strong>— with this technique, you snapshot your data every day. It’s a good option in a world where data storage is cheap.</li></ul><p>Depending on the questions you’re asking, the optimal technique for seeing your data history may change.</p><h3>The benefits of immutability in fact tables</h3><p>Fact tables are immutable logs, so they will never change. This immutability unlocks a few key benefits:</p><ul><li>You can add up behavior across each time period to get cumulative behavior</li><li>You can tack on the latest behavior without having to recalculate everything</li><li>You can trust that data won’t change under your feet</li><li>You can quickly dive to any point in history</li></ul><p>Immutability is why technologies like <a href="https://grafana.com/">Grafana</a> on InfluxDB (a time series database) are so fast. With immutability guarantees on monitoring, you can build systems that cache and pull data from billions of rows in milliseconds.</p><h3>When immutability isn’t fully reliable</h3><p>While immutability does come with its benefits, you can get into trouble if you try to take advantage of the benefits without safeguards.</p><p>A common situation is when event data is wrong or arrives late. The benefits of immutable data would suggest that you can process the latest data once without having to revisit the raw data. However, if your data within the processing window arrives too late, you run the risk of undercounting that information.</p><p>There are different ways to deal with these risks, and specific technologies have been created to address them. <a href="https://eng.uber.com/hoodie/">Apache Hudi</a>, an incremental processing framework that came out of Uber, was initially based on the assumption that data can always be late, so the system must be able to neatly handle incremental changes. For our purposes today, it’s sufficient to flag that you must be thoughtful when considering immutability.</p><h3>Dimensional data modeling is just the beginning</h3><p>There are now companies working to build a better caching layer on top of dimensional data modeling that provides flexibility, performance, and self-service. Let’s look at a few examples.</p><p><a href="https://transform.co/">Transform</a>, <a href="https://metriql.com/">MetriQL</a>, and even<a href="https://docs.getdbt.com/docs/dbt-cloud/using-dbt-cloud/cloud-metrics-layer"> dbt</a> believe that a <a href="https://datastackshow.com/podcast/how-the-metrics-layer-bridges-the-gap-between-data-business-with-nick-handel-of-transform/">metrics layer</a> is the best way for an organization to interface with increasing data volumes. Metrics are critically important for things like experimentation platforms which require users to select which metrics they expect to go up or down. You simply can’t do this with a dimensional data model alone, you need metrics too.</p><p><a href="https://www.looker.com/">Looker</a> abstracts data in the<a href="https://docs.looker.com/data-modeling/learning-lookml/what-is-lookml"> </a>Looker Markup Language (<a href="https://docs.looker.com/data-modeling/learning-lookml/what-is-lookml">LookML</a>) to make it easier for users to access data without having to understand the underlying data models.</p><h3>How does dimensional data modeling apply to RudderStack?</h3><p>We’ve covered a lot about data modeling so far, and we’re only scratching the surface. It’s easy to see how entire careers are dedicated to data modeling and business intelligence. There really is a lot happening in the warehouse. Now we’ll shift gears a bit to explain how all of this applies to our three core customer data pipelines here at RudderStack: Event Streams, ETL, and Reverse ETL. Plus, we’ll introduce a forthcoming product, RS360, which is based on dimensional data models.</p><h3>Event Streams</h3><p><a href="https://www.rudderstack.com/docs/event-spec/standard-events/track/">Track calls</a> in RudderStack let you record customer events and the properties associated with them. When we write data to the warehouse, we write each event to its own track table as well as one large track table. These track tables emulate the immutability of <strong>fact tables</strong> and keep a running log of all user activity.</p><p>The <a href="https://www.rudderstack.com/docs/event-spec/standard-events/identify/">Identify call</a> lets you identify a visiting user, associate them to their actions, and record <a href="https://www.rudderstack.com/docs/event-spec/standard-events/identify/#identify-traits">traits</a> about them like their name and email address. Identify is a bit more complicated since it’s implemented via logs, but you can easily rebuild a <strong>dimension table</strong> for users by taking the most recent identify information and combining it with other data from your system.</p><h3>Extracts</h3><p>With<strong> </strong><a href="https://www.rudderstack.com/product/etl">ETL</a>/ELT, you can unify and enrich your event data with data that lives in the SaaS systems your sales, support, and marketing teams use. Often these systems contain their own view of the user, which can be invaluable information in building out a 360 view of your customer.</p><p>Sometimes these systems create hybrids that are not obvious in the context of a user because they are not user-centric. For example, A Zendesk ticket is an object (<strong>dimension</strong>) because its properties can be updated, but creating or engaging with a ticket is an event (<strong>fact</strong>) that can be useful to aggregate at both the ticket level and user level. As a <a href="https://www.rudderstack.com/blog/what-is-a-customer-data-platform">customer data platform</a>, we primarily focus on data at the user level, but we also need to be very careful about how different objects interact with the customer data models.</p><h3>Reverse ETL</h3><p><a href="https://www.rudderstack.com/product/reverse-etl">Reverse ETL</a> pipelines enable you to move processed data from your data warehouse into your business applications, effectively <em>operationalizing</em> the valuable analytics data. Reverse ETL use cases often benefit from a mix of static properties and aggregates of actual behavior. For example, in Salesforce you might want to send a mix of:</p><ul><li>Dimensions (company size, user, region, vertical)</li><li>Aggregated facts (total visits, cumulative purchases, last 30 day purchases, feature usage)</li><li>Derived data (ML model results, cohort membership)</li></ul><p>Given the requirements above, the type of table you’d probably send via Reverse ETL is a <strong>denormalized table</strong>: one row, one user, and a bunch of properties. This transitions into how we’re thinking about our RudderStack 360 below.</p><h3>RudderStack 360 (RS360)</h3><p>RS360 will enable you to create a usable 360 table in your warehouse. This will give you a holistic view of all of the users, or entities, that exist within your RudderStack-connected data stack. Most of this data can be fed into RS360 from the other RudderStack-provided tools that we mentioned above (Event Streams and Extracts). RS360 will provide some really useful functionality out of the box:</p><ul><li><strong>ID Stitching</strong>: This allows you to connect user activity through your first-party-app and other data from SaaS tools.</li><li><strong>Feature/ Traits table generation</strong>: This is the denormalized table that we mentioned above (i.e. one row per user with a bunch of features/traits as columns).</li><li><strong>Easy to configure:</strong> You have the ability within RudderStack to customize your id mapping tables and feature tables to meet your needs without writing complex sql.</li><li><strong>Current data</strong>: Because you’re already using the Identify Call you will have up-to-date data for your user’s most recent activities within your app. You can schedule how often and when this data syncs according to your requirements.</li></ul><p>Additionally, all of this data will sit in your warehouse. This means that you have complete control over who has access to the data. Using the 360 table, you’ll be able to perform analytics, ML models, and audience targeting. You’ll also be able to send this data from your warehouse to other applications for activation. RS360 is currently in beta and will launch in Q1 2023.</p><h3>A recap of dimensional data modeling</h3><p>If you want to go deeper, one place to read more is Ralph Kimball’s book <a href="https://www.amazon.com/Data-Warehouse-Toolkit-Definitive-Dimensional-dp-1118530802/dp/1118530802/ref=dp_ob_title_bk">The Data Warehouse Toolkit</a>. However, one of the most important things to remember is that organization is different, and these principles are best used as heuristics rather than strict guidelines.</p><p>In under ten minutes, we’ve covered facts and dimensions–the building blocks of dimensional data modeling. We’ve worked through the speed vs. granularity tradeoff highlighting the denormalized table as today’s technique of choice. We touched on data history. We covered the benefits and the risks associated with immutability. Finally, we looked at a few of the innovations being built on top of dimensional data modeling and examined where it shows up at RudderStack.</p><p>Even if you don’t spend all of your time in the warehouse, It’s helpful to understand the core concepts of dimensional data modeling because, as we detailed here with RudderStack, they show up in the wild more often than you might realize.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=11ea860a95cd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/rudderstack/why-you-should-care-about-dimensional-data-11ea860a95cd">Why You Should Care about Dimensional Data Modeling</a> was originally published in <a href="https://medium.com/rudderstack">RudderStack</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Does The Data Lakehouse Enhance The Customer Data Stack?]]></title>
            <link>https://medium.com/rudderstack/how-does-the-data-lakehouse-enhance-the-customer-data-stack-19ca2b4d3363?source=rss----1840a84a9341---4</link>
            <guid isPermaLink="false">https://medium.com/p/19ca2b4d3363</guid>
            <category><![CDATA[modern-data-stack]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[rudderstack]]></category>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[customer-data-platform]]></category>
            <dc:creator><![CDATA[RudderStack]]></dc:creator>
            <pubDate>Tue, 25 Oct 2022 19:01:28 GMT</pubDate>
            <atom:updated>2022-02-02T09:49:49.445Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*t6OiZUlrKnQNSNGVeIdxqg.png" /></figure><p>Reading the title of this post will probably make you wonder how many buzzwords can fit in one sentence. A fair point, but it’s worth exploring how a customer data stack benefits from the data lakehouse. First, let’s clarify exactly what we mean by customer data stack and data lakehouse.</p><h3>What is a customer data stack?</h3><p>We talk a lot about <a href="https://www.rudderstack.com/video-library/the-modern-data-stack-is-warehouse-first">the modern data stack</a>, but it’s important to make a distinction here because customer data is special. It provides a unique value to the organization, and it comes with a unique set of technical challenges.</p><p>Customer data provides unique value because it’s the main source of behavioral information a business has about its customers. Anyone who has ever built a business will tell you that if you don’t know your customer, you’ll be out of business fast. This is especially true for modern online businesses where direct customer interactions are rare.</p><p>But, like all valuable things, customer data comes with a cost. Remember those unique technical challenges I mentioned? Let’s take at the characteristics of customer data that make it difficult to work with.</p><ul><li><strong>Customer data comes in big quantities.</strong> Modern commerce, driven by online interactions, generates a massive amount of data every day. Just ask anyone who’s worked at a large B2C company like Uber or even a medium-sized e-commerce company. The sheer volume of data generated by even simple interactions is huge.</li><li><strong>Customer data is extremely noisy.</strong> The customer journey always involves many actions, but not every action holds value. The issue here is that there’s no way to know what’s valuable and what isn’t before analysis. Your best bet is to record everything and let your brilliant data analysts and data scientists shine.</li><li><strong>Customer data changes a lot.</strong> No one has the same behavior forever, right? I mean, only dead people have a behavior that remains the same across time. This means you need to keep accumulating big amounts of data, but only some of them will be relevant at a specific point in time.</li><li><strong>Customer data is a multidimensional time series.</strong> This sounds very scientific, but all it means is that time ordering is important, and there’s no single value for each data point. This adds to the complexity of the data and how you interact with it. You can read about <a href="https://www.rudderstack.com/blog/kafka-vs-postgresql-implementing-our-queueing-system-using-postgresql/">how we implemented our queueing system using PostgreSQL</a> if you’d like to go down a rabbit hole with this one.</li><li><strong>Finally, customer data can be pretty much anything</strong>, from very structured to completely unstructured. For example, an invoice issued at a specific time for a specific customer, that’s customer data. So is an interaction that the same customer has on your website. Even the picture of that customer, taken as part of a verification process, is customer data.</li></ul><p>I bring all of this up to convince you that working with customer data is important, and it’s challenging enough to require some unique choices to be made when you <a href="https://www.rudderstack.com/blog/the-data-stack-journey-lessons-from-architecting-stacks-at-heroku-and-mattermost">build your data stack</a>. Thus, we have the term <em>Customer Data Stack</em>.</p><p>TL;DR, <em>A Customer Data Stack is a complete Data Stack (modern or not, it doesn’t matter) that allows you to capture, store and process customer data at scale.</em></p><h3>What is a Data Lakehouse?</h3><p>If you’ve been paying attention lately, you’ve heard a lot of buzz around the terms <em>data lake</em> and <em>data lakehouse</em>. The data lakehouse is even younger than the data lake which, to be honest, is pretty old! Humans started building data lakes with the inception of <a href="https://hadoop.apache.org/docs/r1.2.1/hdfs_design.html">HDFS</a> and Hadoop. Yes, that old. A distributed file system and Map Reduce is all it takes to build a data lake.</p><p>Obviously, many things have changed since the early 00’s when it comes to data technology. The term data lake has now been formalized, and the lakehouse is the new kid on the block.</p><p>But, before we get into any more details, let’s make something clear. When we talk about data lakes and lakehouses, we mainly refer to <em>an architecture and not to a specific technology</em>. It’s important to keep this in mind because a lot of the confusion around these two terms comes from this misconception.</p><h3>Data Lake</h3><p>Let’s start with the definition of a data lake. Getting clear on this first will help us understand the data lakehouse.</p><p>The main concept behind a data lake is the separation of storage and compute. Sounds familiar right? Snowflake talks about this a lot, and so does Databricks. But, if you noticed, at the beginning of this section I mentioned HDFS and Hadoop. The first data lakes, using HDFS as a distributed file system and Map-Reduce as a processing framework, separated storage (HDFS) from processing (Hadoop — MR). This is the fundamental concept of a data lake.</p><p>Today, when we talk about a data lake, the first thing we think about is S3 which replaces HDFS as the storage layer and brings storage on the cloud. Of course, instead of S3 we can have the equivalent products from GCP and Azure, but the idea remains the same: an extremely scalable object storage system that is sitting behind an API and lives on the Cloud.</p><p>Processing has also evolved since Hadoop. First, we had the introduction of <a href="https://spark.apache.org/">Spark</a> that offered an API for Map-Reduce that was more user-friendly, and then we got distributed query engines like <a href="https://trino.io/">Trino</a>. These two processing frameworks co-exist most of the time, addressing different needs. Trino is mainly used for analytical online queries where latency is important while Spark is heavily used for bigger workloads (think ETL) where the volume of data is much bigger and latency is not so important.</p><p>So, having S3, using something like <a href="https://parquet.apache.org/">Parquet</a> for storing the data, and using Trino or Spark for processing the data gives us a lean but capable <em>Data Lake</em>.</p><p>This architecture is good on paper and scales amazingly well, but there are a number of functionalities commonly found in data warehouses and transactional databases that are not present. For example, we haven’t mentioned anything about things like transactions. This is the reason why <em>Lakehouse</em> exists.</p><h3>Data lakehouse</h3><p>A <em>Lakehouse</em> is an architecture that builds on top of the data lake concept and enhances it with functionality commonly found in database systems. The limitations of the data lake led to the emergence of a number of technologies including <a href="https://iceberg.apache.org/">Apache Iceberg</a> and <a href="https://hudi.apache.org/">Apache Hudi</a>. These technologies define a <em>Table Format</em> on top of storage formats like ORC and Parquet on which additional functionality like transactions can be built.</p><p>So, what is a data lakehouse? It’s an <em>architecture</em> that combines the critical data management features of a data warehouse with the openness and scalability of the data lake.</p><h3>Data Lakehouse as the foundation of a Customer Data Stack</h3><p>Now that we’ve covered the definitions and utility of the customer data stack and data lakehouse, I’ll make a case for leveraging the data lakehouse in the customer data stack.</p><p>When building a data stack, one of the most important and impactful decisions you’ll make is with the storage and processing layer. In most cases, this is a data warehouse like Redshift or Snowflake. Here are the main benefits of using a lakehouse when dealing with customer data instead:</p><ul><li><strong>Cheap, scalable storage</strong>. Because of the characteristics of customer data we covered above, to efficiently work with it, you’ll need a technology that can store it at scale and keep costs as low as possible. The only architecture that offers this efficiently is a Lakehouse (or data lake), which allows you to build on top of the cheap infrastructure offered by a cloud object storage service. None of the cloud warehouse solutions can offer that, especially if you don’t want to maintain any pruning policy for older data.</li><li><strong>Support for every format. </strong>The nature of customer data means you might have to both store and process completely heterogeneous data, ranging from structured tabular data to binary data. Cloud data warehouses, though they’ve made progress towards supporting structured and semi-structured data, still have a hard time supporting every possible data format. Data Lakes and Lakehouses don’t suffer from this limitation. They offer a futureproof option capable of supporting every format we currently use.</li><li><strong>Hybrid workloads.</strong> Another big consideration is that customer data usually results in hybrid workloads. What I mean by this is that you will want to do operational analytics on your customer data, and soon you’ll need to incorporate more sophisticated ML techniques to do things like <a href="https://www.rudderstack.com/blog/churn-prediction-with-bigqueryml/">churn prediction</a> and attribution (if you aren’t already). Data lakes and lakehouses are the best fit for covering both workloads, and they do it pretty natively. Cloud Data warehouses do now offer functionalities for ML, but they’re limited compared to what you can do on a data lake or lakehouse. So, a lakehouse is pretty much the de facto storage-processing solution when DS and ML projects are involved. This single solution that can serve all the use cases relieves companies from having to implement both a CDW and a data lake, saving cost and operational complexity.</li><li><strong>Lakehouses are open</strong>. Compared to the cloud data warehouses on the market right now that are proprietary and closed systems, lakehouses are open. That means there’s an ever-evolving ecosystem of technologies and products that can supplement the data lake/lakehouse, allowing the operating team to be extremely agile when it comes to maintaining and extending the customer data stack.</li></ul><h3>Final thoughts</h3><p>If you are still with us, you hopefully have a bit of a better understanding of why we need and use all these buzzwords and, most importantly, how they fit together.</p><p>Data lakes and lakehouses are quickly becoming fully-featured data warehouses with infinite scale and low cost. Customer data is a natural fit for such a storage and processing layer. These architectures will allow you to support any possible use case you might have in the future, even if all you care about today is operational analytics.</p><p>Here at RudderStack, we are strong supporters of the data lake and lakehouse architectures. We have invested our resources to build best-in-class <a href="https://www.rudderstack.com/docs/data-warehouse-integrations/">integrations</a> with both data lakes and lakehouses like Delta Lake. By using RudderStack together with a Lakehouse or data lake, you can have a complete<em> </em>customer data stack that scales amazingly well and allows you to do anything you want with your data.</p><p><em>This blog was originally published at:<br></em><a href="https://www.rudderstack.com/blog/how-does-the-data-lakehouse-enhance-the-customer-data-stack"><em>https://www.rudderstack.com/blog/how-does-the-data-lakehouse-enhance-the-customer-data-stack</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=19ca2b4d3363" width="1" height="1" alt=""><hr><p><a href="https://medium.com/rudderstack/how-does-the-data-lakehouse-enhance-the-customer-data-stack-19ca2b4d3363">How Does The Data Lakehouse Enhance The Customer Data Stack?</a> was originally published in <a href="https://medium.com/rudderstack">RudderStack</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What Is a Customer Data Platform?]]></title>
            <link>https://medium.com/rudderstack/what-is-a-customer-data-platform-b2862c43f7f3?source=rss----1840a84a9341---4</link>
            <guid isPermaLink="false">https://medium.com/p/b2862c43f7f3</guid>
            <category><![CDATA[etl]]></category>
            <category><![CDATA[customer-data-platform]]></category>
            <category><![CDATA[data-privacy]]></category>
            <category><![CDATA[rudderstack]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[RudderStack]]></dc:creator>
            <pubDate>Tue, 25 Oct 2022 19:00:04 GMT</pubDate>
            <atom:updated>2022-02-10T13:25:52.186Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/751/1*AoSe-r0c88Qd3v-TJdB8UA.png" /></figure><p>The goal of a customer data platform (CDP) is to (1) build a unified view of your customers and (2) utilize these profiles to enrich the quality of customer interactions across multiple touchpoints. With a “build once, use everywhere” mentality, organizations can use a CDP to make sure their customer interactions are personalized, consistent across touchpoints, and valuable for all parties involved.</p><h3>What does a customer data platform do?</h3><p>Customers interact with an organization through marketing, sales, support, and the product itself. CDPs help collect and aggregate this information, build customer profiles, and propagate this information to downstream systems. Operational systems take advantage of these profiles to deliver personalized messaging, contextualized interactions, and custom experiences. To achieve this, there are several major things a CDP does:</p><ol><li><strong>Bring in the data.</strong> Customer Data Platforms have integrations with a variety of first-party data sources that capture customer information in relation to an organization. This includes marketing, sales, support, and the product itself.</li><li><strong>Build the profiles.</strong> Customer Data Platforms build unified customer profiles (oftentimes called “Customer 360”) that are a combination of static traits as well as aggregate behavior information.</li><li><strong>Make the profiles useful.</strong> Once the profiles are built, they are utilized by internal apps, sales/marketing/support vendors, and other systems to take advantage of the data to personalize communications or contextualize interactions.</li></ol><h3>Bringing in the data</h3><p>Since customer interactions are captured in a variety of tools and software, there are many</p><p>This includes information such as data used within the product, metadata (e.g. geography), and behavioral data (product usage, marketing responsiveness, sales/support communications). This data is often referred to as “first party data” because it is the customer’s interaction with the organization itself rather than information bought from outside parties.</p><p>There are several ways of bringing in data, many of which are understood in the context of the modern data stack. These are the most common examples.</p><ul><li><strong>Application Event Tracking.</strong> Organizations can leverage SDKs across a variety of languages that enable them to track and understand user behavior and flows throughout a software application. This can be done either on the client (Mobile, Web, Smart Device) or on the server (Python, Java, Go, etc.) with trade-offs for each.</li><li><strong>Vendor Webhooks.</strong> Some software vendors expose webhooks on activity that happens on their platform. These data streams are private to the organization using the vendor and provide information on how the vendors are used.</li><li><strong>Production Databases.</strong> Production databases are a good source of truth for application transactions or production data models. These data streams are often brought in via ELT/ETL (extract, transform, load) processes that leverage modern techniques like change data capture (CDC).</li><li><strong>Vendor Data Extracts.</strong> Vendors that directly engage with customers on behalf of an organization often hold the data within their systems. This data can be extracted in batch through vendor provided APIs.</li></ul><h3>Building the profiles</h3><p>Bringing in customer data enables you to build unified, comprehensive customer profiles based on every touchpoint with your organization. Given the data that is collected through upstream integrations, a customer data platform builds a single view of the user that can be understood, filtered, aggregated, or propagated, depending on the use case.</p><p>Advanced features in building customer profiles include:</p><ul><li><strong>Personalization.</strong> Machine learning models or simple heuristics can help predict what customers are interested in based on existing interaction. This includes personalized recommendations, personalized messaging, or cohort-specific customer journeys. Here’s how one of our customers leverages <a href="https://www.rudderstack.com/blog/real-time-personalization-with-redis-and-rudderstack">RudderStack and Redis for real-time personalization</a>.</li><li><strong>Customer Health.</strong> Customer health is based on a combination of engagement (or lack thereof), quality of interaction, direct feedback, and more. This information can help guide messaging that is encouraging, conciliatory, determined or exploratory depending on how the customer feels about the company/product.</li><li><strong>Identity resolution.</strong> Customer interactions across many tools often share an identifier, but pseudo-anonymous interactions (e.g. pre-login) benefit from being stitched back together to customers once you determine who they were. Learn more about <a href="https://www.rudderstack.com/blog/the-tale-of-identity-graph-and-identity-resolution/">identity resolution</a>.</li></ul><h3>Making the profiles useful</h3><p>Activating data on a customer data platform involves making it available to downstream communication platforms, whether it be sales, marketing, support, or the product itself. These are often the same tools used to capture these interactions from the “Bring in the Data” section above.</p><p>This can involve either pushing the downstream systems or exposing an interface (e.g. API endpoint) that enables vendors to access and aggregate customer data. If the data lives in a cloud data warehouse, there may be technological overlap with products in the <a href="https://www.rudderstack.com/product/reverse-etl">Reverse ETL</a> category which help push data from warehouses to downstream integrations.</p><p>In addition to using data for individual customers, CDPs enable organizations to build audiences lists or properties based on features of audiences. For example, if you want to target all customers from Europe who joined in the past 5 days, a CDP would enable you to leverage the central customer profile to build audience lists that can be used in communication platforms.</p><h3>What should you look for when evaluating a CDP solution?</h3><p>The best customer data platform for you depends on a number of factors you may value for your business. Data storage, privacy, control, cost, completeness, extensibility, and more. Here are a few considerations that should inform your CDP evaluation:</p><ul><li><strong>How complete is the data coming in?</strong> With the new wave of privacy-conscious browser restrictions designed to improve user privacy and to weaken the marketplace of data brokers, first-party event collection systems are often affected even though they are not the intended target. Effective CDPs have a number of approaches to tackling external challenges and verifying data quality.</li><li><strong>How good is the data coming in?</strong> Do you need tight controls over the data quality of the data coming in from your application, your vendors, and your databases? Some CDPs support a concept of Tracking Plans to verify that data coming in has a consistent format that can be used to accurately update a user profile.</li><li><strong>How much control do you have over your data?</strong> When working with a customer data platform, do you need to interact with the data directly, or are you ok working with the data through interfaces provided by the CDP? If a CDP shares its data with you or lives on a cloud data warehouse, then you have more visibility into your data and can re-use your customer data profiles for custom processing and analytics. Read more on <a href="https://www.rudderstack.com/blog/what-is-data-control/">data control</a>.</li><li><strong>What channels do you use to communicate with customers?</strong> B2B organizations have high-value investments in a smaller number of customers through sales processes, while B2C (such as ecommerce or marketplaces) organizations traditionally invest more in digital marketing and advertisements. Different CDPs have product optimizations or cost models that may be more advantageous for one or the other.</li><li><strong>How comprehensive is your data privacy program?</strong> Data privacy laws such as GDPR and CCPA put stricter controls on the collection, access, storage, and facilitation of customer data. A CDP should be able to provide compliance for both data within its system as well as forward deletion requests to downstream systems. In addition to this, some CDPs may go one step further with compliance for medical data (HIPAA). Read more on <a href="https://www.rudderstack.com/blog/customer-data-pipelines-play-a-key-role-in-data-privacy">the role your CDP can play in data privacy</a>.</li></ul><p>Ultimately, the customer data platform is an evolving concept that brings together multiple elements of the modern data stack into a very clear set of goals that enable organizations to make the most out of first-party customer data.</p><p><em>This blog was originally published at:<br></em><a href="https://www.rudderstack.com/blog/what-is-a-customer-data-platform/"><em>https://www.rudderstack.com/blog/what-is-a-customer-data-platform/</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2862c43f7f3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/rudderstack/what-is-a-customer-data-platform-b2862c43f7f3">What Is a Customer Data Platform?</a> was originally published in <a href="https://medium.com/rudderstack">RudderStack</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analytics Engineering vs. Data Engineering]]></title>
            <link>https://medium.com/rudderstack/analytics-engineering-vs-data-engineering-ac748d32a8c4?source=rss----1840a84a9341---4</link>
            <guid isPermaLink="false">https://medium.com/p/ac748d32a8c4</guid>
            <dc:creator><![CDATA[RudderStack]]></dc:creator>
            <pubDate>Fri, 27 May 2022 13:56:07 GMT</pubDate>
            <atom:updated>2022-05-27T13:56:07.026Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/751/1*4o04KHo-W_D0v0PvOoDzmw.jpeg" /></figure><p>With the explosion of new tools and technologies that make collecting, transforming, and analyzing data much easier, we have now entered a new era for the capabilities of a modern data team. With that new roles have emerged and others have shifted focus. In this article, we’ll highlight the Modern Data Stack that has enabled these new roles for data professionals, we’ll dive into the newest role, the Analytics Engineer, and explain where that leaves the Data Engineer.</p><h3>The Modern Data Stack</h3><p>Compared to other fields like software engineering, for example, the data space is relatively nascent. So, it should come as no surprise that it’s changing fast. The data tooling landscape is evolving at what feels like breakneck speed, and this has a trickle down effect on the organization of data teams.</p><p>New technologies are automating, or at least streamlining, much of the data engineering work that used to bog us all down. These innovations catalyzed the creation of new roles and overhauled old ones.</p><p>The tools driving the organizational changes are the usual suspects that make up what we’ve come to call the modern data stack, and they fit into a few major categories:</p><ul><li>Data warehouse/data lake</li><li>ETL/ELT</li><li>Event streaming</li><li>Data transformation</li><li>Data visualization and BI Tools</li></ul><p>Most of these tools are pretty straightforward to set up, and they often don’t require deep technical knowledge. They’re also really reliable and pretty much just work. Gone are the days requiring a full time person just to manage and operate one tool.</p><p>These foundational technologies have also given rise to new tools that are coming out and leverage the Modern Data Stack like:</p><ul><li>Reverse ETL</li><li>Data Quality Monitoring and Observability</li><li>Headless BI</li></ul><h3>What is Analytics Engineering?</h3><p>Analytics Engineering refers to the role that builds clean and flexible data models for use downstream. These models are used primarily for four use cases which I’ll list below along with an example of the associated tool they might end up in:</p><ul><li>Business intelligence — <a href="https://www.looker.com/">Looker</a></li><li>Reverse ETL — <a href="https://www.rudderstack.com/product/reverse-etl">RudderStack</a></li><li>Machine learning models — <a href="http://continual.ai/?utm_campaign=Rudderstack%20comarketing&amp;utm_source=Rudderstack%20blog">Continual</a></li><li>Exploratory notebooks — <a href="https://hex.tech/">Hex</a></li></ul><p>Analytics engineers are also responsible for ensuring software engineering best practices are followed, like testing, continuous integration, and version control.</p><p>Examples of what Analytics Engineering should take on:</p><ul><li>Building and testing dbt models for downstream use</li><li>Refactoring existing dbt models due to changing business requirements</li><li>Working with Data Engineering and other upstream team members to ensure that data is delivered to them in a format that allows them to build the cleanest and most insightful data models</li><li>Working with downstream stakeholders and end users, including business users and data science, to understand their requirements to ensure the models being created service their needs</li></ul><p>If you want to learn more about analytics engineering, <a href="https://www.getdbt.com/what-is-analytics-engineering/">What is Analytics Engineering</a> is a must read from the team at dbt.</p><h3>Why Analytics Engineering?</h3><p>The key value proposition of the Analytics Engineering role is in the creation of clean, scalable, and reusable data models. These models can be used to drive business decisions with the highest value, but building these models requires a unique skillset. The analytics engineer must understand all aspects of the business and possess necessary technical skills. They must understand the underlying raw data and processes on that data that makes the business run.</p><p>Before dbt changed everything for data transformation, Data Engineering was focused on ingesting data and transforming it in transit. This left the Data Analyst to further transform the data for their needs, leading to inefficient and non-scalable solutions like Looker PDTs or huge amounts of SQL underlying dashboards in tools like Tableau or Chart.io. With datasets that are built and tested by an Analytics Engineer, the SQL that must be run by your BI tool becomes much less complicated and also means less chance for consumers of the data to get the wrong answers.</p><h3>What is Data Engineering?</h3><p>The rise of Analytics Engineering changed the role of Data Engineering from what it used to be.</p><p>Before the modern data stack, data engineers had to spend most of their time just ensuring data ingestion pipelines were running, dealing with database administration, and chasing down errors with queries. Now that the majority of these problems have been taken care of with off-the-shelf tools, data engineering has entered a new phase. Data Engineers are now free from the shackles of just keeping a system running, so they can add more value in other areas.Some examples of the new responsibilities now under the purview of data engineering are:</p><p><strong>Building custom data pipelines from internal services — </strong>Due to the custom nature and usually high volume of internal data pipelines that are used to power a company’s product(s) it is the role of the data engineer to build a data pipeline that can pipe this data to the data warehouse.</p><p><strong>Managing and optimizing data infrastructure — </strong>While there are lots of tools that make our lives in the data world easier, they still need to be managed, maintained, and optimized.</p><ul><li>ETL tools need to be set up and maintained.</li><li>CI/CD pipelines need to be created to ensure that any transformation or other pipelines are operating correctly and changes are vetted before being deployed</li><li>Data pipeline failures need to be handled — even in the best of cases, there can always be issues that can cause outages for any of these services. Data engineers are best equipped to understand and fix these issues.</li></ul><p><strong>Optimizing queries — </strong>Data engineers should have a deep understanding of the technologies they’re using, especially the data warehouse, that gives them insight into optimizing queries to make most efficient use of them.</p><h3>A bright future in harmony</h3><p>By harnessing the power of both Analytics Engineering and Data Engineering roles at your organization, you get a whole that is more than the sum of its parts. When the team is working optimally, they can support each other where Data Engineering is ensuring the quality and timeliness of raw data coming into the Data Warehouse and Analytics Engineering takes that and builds the models that will power the business insights.</p><p>With the combined powers of Analytics Engineering and Data Engineering, you find yourself in data nirvana with a full understanding of the data coming and going from the data warehouse. Together, Data Engineers and Analytics Engineers can not only leverage that data for maximum business value, but also suggest what new data should be implemented or extracted to create even more value for the business. The Analytics Engineering, Data Engineering combo enables companies to develop, iterate, analyze, and take action faster than ever before.</p><p>We all stand on the shoulders of giants, and the modern data stack has delivered so much power to all of us who work with data. Let’s continue innovating and building to show how much farther we can go 🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ac748d32a8c4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/rudderstack/analytics-engineering-vs-data-engineering-ac748d32a8c4">Analytics Engineering vs. Data Engineering</a> was originally published in <a href="https://medium.com/rudderstack">RudderStack</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Future of Customer Data Platforms: To Bundle or Not to Bundle?]]></title>
            <link>https://medium.com/rudderstack/the-future-of-customer-data-platforms-to-bundle-or-not-to-bundle-4ba115df8832?source=rss----1840a84a9341---4</link>
            <guid isPermaLink="false">https://medium.com/p/4ba115df8832</guid>
            <category><![CDATA[cdp]]></category>
            <category><![CDATA[customer-data-platform]]></category>
            <dc:creator><![CDATA[Soumyadeb Mitra]]></dc:creator>
            <pubDate>Thu, 26 May 2022 19:08:15 GMT</pubDate>
            <atom:updated>2022-05-26T19:08:15.372Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/751/1*hJRXLyA7hFHylDzc1mQ3ig.jpeg" /></figure><p>Bundling vs. unbundling set the data world <a href="https://blog.fal.ai/the-unbundling-of-airflow-2/">ablaze</a> in February, and the debate quickly made its way to Customer Data Platforms. These recent conversations are the latest logs on a fire that’s been burning for a long time. The cable industry is a classic example, and that debate is <a href="https://stratechery.com/2022/cables-last-laugh/">far from over</a>. 10 years ago, the same discussion unfolded in martech. New martech vendors were pushing to bundle all channels into one vertically integrated solution while the legacy vendors fought to protect their territory. We all know how that ended.</p><p>Interestingly, the CDP world now finds itself in a similar place– legacy vendors are trying to protect their bundling whereas newer vendors (mostly on the data infra side) are pushing their own agenda with unbundling.</p><p>But let’s not make this an ideological debate. Let’s look at it on a more fundamental level. We’ll use two questions to cut straight to the point: What are the limitations of the traditional bundled CDP, and what problems are we trying to solve with unbundling? First, let’s establish a basic understanding of what constitutes a CDP.</p><h3>The building blocks of a customer data platform</h3><figure><img alt="Diagram" src="https://cdn-images-1.medium.com/max/1024/0*NGvp3qX-R7rGcON8" /></figure><p>The diagram above represents the major building blocks of any CDP. Let’s break each of these down:</p><p><strong>Streaming: </strong>Streaming tools collect first party data, like clickstream events, from websites, mobile apps, and PoS devices.</p><p><strong>ETL: </strong>ETL tools collect customer data from different SaaS tools like CRMs, ad tools (campaign data), and marketing tools (audience/cohort data) and send the data to the warehouse.</p><p><strong>Transformations</strong>: At a high level, this represents any custom logic applied to the customer data to change it in some way. Transformations can further be classified into:</p><ul><li>Real-time transformations: these are typically used to update the source data in-stream and in “real-time” before sending to different destinations. Typical use cases include PII removal, fixing events, and enhancing events with additional meta-data.</li><li>Simple batch transformations: these are basic computations done to user attributes while the data is “at rest”, i.e., in the warehouse. These are commonly known as computed attributes. Examples include computing total_revenue, total_num_of_logins, etc.</li><li>Complex batch transformations: these are warehouse transformations that require more computation. The most common example for customer data is user identity stitching which requires computing transitive closure on the identity graph and while can be done in SQL, is quite complex.</li><li>ML transformations: for more advanced use cases, one may want to compute predictive features like churn scores, product recommendations, etc. on a per user basis.</li></ul><p><strong>Activation</strong>: Activation involves sending the transformed data to different marketing and product SaaS destinations. Activation integrations can be of different types — some may stream data in real-time to destinations from different first party and ETL sources, other more batch-oriented integrations may send audiences and attribute data from the warehouse to destinations. This latter category is commonly known as Reverse ETL.</p><p><strong>UI / Segmentation</strong>: Most marketing oriented CDPs offer a graphical user interface where marketing and product teams who don’t want to write code can create workflows around audience segmentation, journey, orchestration, etc. Conceptually, they provide a visual abstraction over the transformation box shown above. Whether this component is essential to the CDP or should be part of a downstream marketing automation system is up for debate.</p><p>To support the non-real time transformation use cases highlighted above, a CDP needs to keep a copy of all the data including user attribute data, event data, and computed traits. Two kinds of stores are typically used to keep this state:</p><ul><li><strong>Offline store</strong>: Non-real time batch transformations are run on top of the offline store, a store such as a data-warehouse or a data-lake .</li><li><strong>Online / Real-time store</strong>: Real-time stores, like Redis, keep the user profile and serve them over an API to be used for in-app personalization.</li></ul><p>One non-CDP component that’s important to note is the loop-back system to send data from SaaS tools back into the CDP. While not traditionally a part of the CDP itself, loop-back of data is critical for end to end attribution of marketing actions. e.g. measuring the success of an email campaign after the email is sent out. CDPs should be able to do more than just activate the data. They should be able to feed post activation data and analysis back into the system.</p><p>Every CDP, bundled or unbundled, should offer all of the above functionality. Traditionally, CDPs have included everything in a single vertically integrated solution. This approach made sense when use cases were limited and marketing teams were the primary owner of the CDP. But this approach doesn’t cut it for today’s use cases.</p><h3>Limitations of the traditional bundled customer data platform</h3><p>The biggest limitation of the vertically integrated approach is that it leads to an inflexible data architecture that is difficult to adapt to as a company grows in its <a href="https://www.rudderstack.com/blog/a-practical-guide-to-the-modern-data-stack-the-data-maturity-journey">Data Maturity Journey</a>. As companies become digital first and marketing stacks get more complex, so does the need for more sophisticated use cases on top of customer data. Think of things like user journeys, attribution, ML models for churn prediction, and product recommendations. These use cases require the expertise of the data analysts and data scientists from engineering or data teams. But the traditional black-box CDP was created for marketing teams, not engineers, so they don’t expose the customer data in a manner conducive to build applications for these more sophisticated use cases.</p><p>Another limitation of the traditional CDP is its inability to bring data from other systems into its profiles. This data is typically collected via ETL pipeline into the data-warehouse, which the traditional CDP cannot access.</p><p>Data privacy and retention concerns make matters worse for the traditional CDP which typically stores PII and other sensitive information inside of its black box.</p><p>The limitations of these bundled systems led many data teams to build another, redundant, customer data stack in parallel to Marketing’s CDP. This parallel stack typically had all the pieces of a CDP covered above but implemented in a brittle, if more capable system. The stack includes the data collection layer to collect first party sources in a data warehouse or data lake, a transformation layer written in SQL/python/spark to process the data, and an activation layer to push the data out to different marketing tools <strong>including</strong> the marketing CDP. Engineering often would build the collection and activation layers using home grown scripts or open-source solutions.</p><p>Believe it or not, running two parallel customer data stacks is never a good idea. It causes massive duplication of data and effort, resulting in burgeoning costs. Moreover, it proliferates data silos, the problem that CDPs were built to solve in the first place, because the systems don’t share data. For example, the cohorts developed by the marketing team in their CDP are not visible in the data-team’s customer data stack while the ML outputs like churn score are not visible in marketing’s CDP. The only thing worse than not having a CDP is having two CDPs! This “two CDP” environment is fueling today’s bundling vs. unbundling debate.</p><p>To overcome the limitations of the traditional, bundled CDP, unbundling is the right move, but there’s more than one way to unbundle the CDP. In part two, we’ll delve into detail on two approaches to unbundling.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4ba115df8832" width="1" height="1" alt=""><hr><p><a href="https://medium.com/rudderstack/the-future-of-customer-data-platforms-to-bundle-or-not-to-bundle-4ba115df8832">The Future of Customer Data Platforms: To Bundle or Not to Bundle?</a> was originally published in <a href="https://medium.com/rudderstack">RudderStack</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How RudderStack Core Enabled Us To Build Reverse ETL]]></title>
            <link>https://medium.com/rudderstack/how-rudderstack-core-enabled-us-to-build-reverse-etl-4dc734df9fdb?source=rss----1840a84a9341---4</link>
            <guid isPermaLink="false">https://medium.com/p/4dc734df9fdb</guid>
            <dc:creator><![CDATA[RudderStack]]></dc:creator>
            <pubDate>Mon, 16 May 2022 20:27:39 GMT</pubDate>
            <atom:updated>2022-04-07T01:27:20.052Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*Oxjey_-K9upTv_6VP9WqBA.jpeg" /></figure><p>One of the goals of a <a href="https://www.rudderstack.com/blog/what-is-a-customer-data-platform/">customer data platform</a> is to make the movement of data from any source to any destination easy while guaranteeing correctness, reliability, efficiency, and observability. In that sense, reverse ETL is no different, it’s <a href="https://www.rudderstack.com/blog/reverse-etl-is-just-another-data-pipeline">just another data pipeline</a>.</p><p>In 2019, RudderStack started as a data infrastructure tool supporting <a href="https://www.rudderstack.com/product/event-stream">event streaming</a> to multiple destinations, including the data warehouse. From the outset, we made the data warehouse (or data lake/lakehouse 🙂) a first class citizen, supplying automated pipelines that allow companies to centralize all of their customer data in the warehouse. It’s important not to overlook the impact of this decision, because placing the storage layer at the center and making all the data accessible is key to unlocking a plethora of use cases. But getting the data into the warehouse is basically only useful for analytics. It’s getting it back out that enables brand new use cases, and that’s where Reverse ETL comes in.</p><h3>What is Reverse ETL?</h3><p>Reverse ETL is a new category enabling the automation of brand new business use cases on top of warehouse data by routing said data to cloud SaaS solutions, or operational systems, where sales, marketing, and customer success teams can activate it.</p><p>Building pipelines for Reverse ETL comes with a unique set of technical challenges, and that is what this blog is about. I’ll detail our engineering journey, how we built <a href="https://www.rudderstack.com/product/reverse-etl">RudderStack Reverse ETL</a>, and how Rudderstack Core helped us solve more than half of the challenges we faced. In a way, building this felt like a natural progression for us to bring the modern data stack full circle.</p><h3>What is RudderStack Core?</h3><p>RudderStack Core is the engine that ingests, processes, and delivers data to downstream destinations. Main features:</p><ul><li>Ingest events at scale</li><li>Handle back pressure when destinations are not reachable</li><li>Run user defined JavaScript functions to modify events on the fly</li><li>Generating reports on deliveries and failures</li><li>Guarantees the ordering of events delivered is same as the order in which they are ingested</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jLCGOEAUO7HzftR6" /></figure><h3>The technical challenges we faced building Reverse ETL</h3><p>First, I’ll give an eagle eye view of the different stages to building Reverse ETL and the challenges associated with them. Along this stroll, I’ll explain how RudderStack Core helped us launch it incrementally, making several big hurdles a piece of cake. I must give major kudos to our founding engineers who built this core in a “think big” way. Their foresight drastically reduced the amount of effort we had to put into designing and building engineering solutions for Reverse ETL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UeGpJuYlc_mg6k4P" /></figure><h3>1. Creating a Reverse ETL pipeline</h3><p>Out of all the steps, this was the easiest one, though it was still a bit tricky.</p><h4>1.1 Creating a source</h4><p>Warehouse source creation gets complicated because of credentials and because of the read and write permissions one needs to maintain transient tables for snapshots and evaluating diffs. It’s important to ensure the user can easily provide only the necessary permissions for reverse ETL, so the pipeline tool does not end up with access to more tables in the customer’s production than needed or with any unnecessary write access.</p><p>This is a tricky problem made harder by the differences between warehouses. We asked ourselves a few key questions when building this:</p><ul><li>How can we simplify and streamline the commands and accesses for different warehouses?</li><li>How can we help one validate these credentials when creating a source?</li></ul><p>In this instance, our control plane enabled us to reuse and build on existing components. This was crucial because we wanted to make validations in a generic way, so they would be reusable as we continue adding more data warehouse and data lake sources. Our team iterated a lot on how to educate users on which permissions are required and why. Check out our documentation on <a href="https://www.rudderstack.com/docs/reverse-etl/snowflake/#step-1-creating-a-new-role-and-user-in-snowflake">creating a new role and user in Snowflake</a> for an example. We had to work to ensure only relevant validations and errors would show when setting up a source, and we came up with faster ways to run some validations.</p><p>As an example, in our first iteration we used Snowflake queries to verify whether the provided credential allowed us to validate the needed schema for RudderStack, so we could read, write, and manage transient tables to it. These queries were scheduled in the normal queue manner by Snowflake, but for some customers it took minutes for these queries to run. So, we found a<a href="https://docs.snowflake.com/en/sql-reference/sql/show.html#general-usage-notes"> better</a> solution from Snowflake where SHOW commands do not require a running warehouse to execute. With this new solution, validations complete within a minute or less for all customers. As we built out the reverse ETL source creation flow, the big wins that we adopted from the existing RudderStack Core platform were:</p><ul><li>Our WebApp React components’ modular designs were re-usable in the UI</li><li>We were able to re-use code for managing credentials securely and propagate it to the Reverse ETL system in the data plane</li><li>We were able to deliver faster because RudderStack Core allowed us to focus on the user experience and features vs. building infrastructure from the ground up</li></ul><h4>1.2 Creating a destination</h4><p>Every data pipeline needs a source and a destination. When it came to creating destinations for Reverse ETL, RudderStack Core really shined. Enabling existing destination integrations from our Event Stream pipelines was straightforward. We built a simple JSON Mapper for translating table rows into payloads and were able to launch our Reverse ETL pipeline with over 100 destinations out of the box. Today the count is over 150 and growing! We’re also incrementally adding these destinations to our <a href="https://www.rudderstack.com/docs/reverse-etl/features/visual-data-mapper/">Visual Data Mapper</a>. For further reading, <a href="https://www.rudderstack.com/blog/using-rudderstack-to-backfill-data-in-mixpanel/">here’s a blog</a> on how we backfilled data into an analytics tool with Reverse ETL and some User Transformations magic.</p><h3>2. Managing orchestration</h3><p>The Orchestrator was critical and one of the more challenging systems to build, especially at the scale RudderStack is running. Reverse ETL works like any batch framework similar to ETL. If you’re familiar with tools like <a href="https://airflow.apache.org/">Apache Airflow</a>, <a href="https://www.prefect.io/">Prefect</a>, <a href="https://dagster.io/">Dagster</a>, or <a href="https://temporal.io/">Temporal</a>, you know what I’m talking about — the ability to schedule complex jobs across different servers or nodes using DAGs as a foundation.</p><p>Of course, you’re probably wondering which framework we used to build out this orchestration layer. We did explore these options, but ultimately decided to build our own orchestrator from scratch for a few key reasons:</p><ul><li>We wanted a solution that would be easily deployed along with a <a href="https://github.com/rudderlabs/rudder-server">rudder-server</a> instance, in the same sense that rudder-server is easily deployed by open source customers.</li><li>We wanted an orchestrator that could potentially depend on the same Postgres of a rudder-server instance for minimal installation and would be easy to deploy as a standalone service or as separate workers.</li><li>We love <a href="https://go.dev/">Go</a>! And we had fun tackling the challenge of building an orchestrator that suits us. In the long run, this will enable us to modify and iterate based on requirements.</li><li>Building our own orchestrator makes local development, debuggability and testing much easier than using complex tools like Airflow.</li><li>We love open source and would like to contribute a simplified version of RudderStack Orchestrator in the future.</li></ul><h3>3. Managing snapshots and diffing</h3><p>Let’s consider one simple mode of syncing data: upsert. This means running only updates or new inserts in every scheduled sync. There are two ways to do this:</p><ul><li><strong>Marker column:</strong> In this method, you define a marker column like <strong>updated_at</strong> and use this in a query to find updates/inserts since the previous sync ran. There are multiple issues with this approach. First, you have to educate the user to build that column into every table. Second, many times it’s difficult to maintain these marker columns in warehouses (for application databases, this is natural, and many times DBs provide this without any extra developer work).</li><li><strong>Primary key and diffing:</strong> In this method, you define a primary key column and have complex logic for diffing.</li></ul><p>We went with the second option. One major reason was that we could run the solution on top of the customer’s warehouse to avoid introducing another storage component into the system. Also, the compute power and fast query support in modern warehouses were perfect for solving this with queries and maintaining snapshots and diffs to create transient sync tables.</p><p><strong>Hubspot table after incremental sync of new rows:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lDktc1_0d8zuyayr" /></figure><p><strong>Sync screen in RudderStack:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*G-7TdytCRn1BIOFz" /></figure><p><strong>Snapshot table view:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*baRA_7MpM6ppnNsl" /></figure><p>Now, you might be thinking: “What’s the big deal? It’s just creating some queries, running them and syncing data?“ I wish, but it’s not as simple as it looks. Also, this was one of the challenges RudderStack core couldn’t help with. Here are a few of the challenges that emerge when you dig deeper into the problem:</p><ul><li>Diffing needs to be very extensible, not only for the multiple warehouse sources we already support, but also for integrating with future warehouse and data lake sources.</li><li>You have to implement state machine based tasks to handle software or system crashes and any errors that occur across a multitude of dependencies.</li><li>You have to maintain record ordering checkpoints during sync to ensure a higher guarantee of delivering exactly once to destinations.</li><li>You have to support functionality for pausing and resuming syncs.</li><li>You have to handle delivery of records that failed to deliver on the previous sync.</li></ul><p>On top of those considerations, there were a number of other interesting problems we found related to memory, choice of CTE vs temporary table, columns data types, structs in BigQuery, and more, but that’s another post for another day.</p><h3>4. Managing syncing, transformations, and delivery to destinations</h3><p>RudderStack Core significantly shortened the development cycle for syncing, running transformations in the data pipeline, and final delivery to destinations.</p><p>In large part, this is because our Reverse ETL and Event Stream pipelines have a lot in common relative to these use cases. In fact, from a source perspective, Reverse ETL pulling from warehouse tables is much simpler than SDK sources, so we were able to have more precise control over ingestion and leverage rudder-server for everything else. Here’s what rudder-server took care of:</p><ul><li>Destination transformations (mapping payloads to destination API specs)</li><li>Calling the right APIs for add, update, delete, and batch APIs if supported</li><li>User transformations (custom JavaScript code written by users to modify payloads)</li><li>Managing the rate limits of destination APIs (which vary significantly) and providing a back pressure mechanism for Reverse ETL</li><li>Handling failed events with retries and providing finally failed events back to Reverse ETL</li><li>A mechanism to identify completion of sync tasks</li><li>New integrations and feature enhancements (automatically usable by our Reverse ETL pipeline when deployed to RudderStack Core)</li></ul><p>Even though the items above were huge wins from RudderStack Core, there were some other interesting problems we had to solve because we use rudder-server as our engine to deliver events. I won’t dive into those now, but here’s a sample:</p><ul><li>It’s challenging to deliver events to our multi-node rudder-server in a multi-tenant setup</li><li>It’s complicated to guarantee event ordering for destinations that require it</li><li>We have to respect the rate limits of different destinations and use back pressure mechanisms, so we don’t overwhelm rudder-server, all while maintaining fast sync times</li><li>Acknowledging completion of a sync run with successful delivery of all records to destination</li></ul><h3>5. Maintaining pipelines with observability, debuggability, and alerting</h3><p>Any automated data pipeline needs some level of observability, debugging, and alerting, so that data engineers can take action when there are problems and align with business users who are dependent on the data.</p><p>This is particularly challenging with systems like Reverse ETL. Here are the main challenges we had to solve:</p><ul><li>Long running processes must account for software crashes, deployments, upgrades, and resource throttling</li><li>The system has dependencies on hundreds of destinations, and those destinations have API upgrades, downtime, configuration changes, etc.</li><li>Because RudderStack doesn’t store data, we have to create innovative ways to accomplish things like observability through things like live debuggers, in-process counts (like sending/succeeded/failures), and reasoning for any errors that are critical</li></ul><p>Accounting for software crashes, deployments, upgrades, and resource throttling required a thoughtful design for Reverse ETL, here’s how we did it:</p><ul><li><strong>State machine</strong>: State based systems look simple but are incredibly powerful if designed well. Specifically, if an application crashes, it can resume correctly. Even failed states like failed snapshots can be handled properly by, say, ignoring it for the next snapshot run.</li><li><strong>Granular checkpoint: </strong>This helps make sure no duplicate events will be sent to destinations. For example, say we send events in a batch of 500 and then checkpoint. The only possibility would be that one entire batch might get sent again if the system restarted or if it happened during deployment as it was sent to rudder-server, but could not checkpoint. On top of this, rudder-server only has to maintain a minimal batch of data to add dedupe logic on top because it doesn’t need to save an identifier for all records for a full sync task.</li><li><strong>Support for handling shutdown and resuming</strong>: Graceful shutdown handling is critical for any application, especially for long running stateful tasks. My colleague Leo <a href="https://www.rudderstack.com/blog/implementing-graceful-shutdown-in-go/">wrote an amazing blog post about how we designed graceful shutdown in Go</a>, which you should definitely read.</li><li><strong>Auto scale systems:</strong> Automatically scaling systems handle tasks that are running in a distributed system, which is necessary for handling scale, both for Reverse ETL side as well as the consumer (rudder-server). At any given time a Reverse ETL task might be running on a single node, but might have to be picked up by another node if the original node crashes for some reason. On the consumer side (rudder-server), data points might be sent to consumers running on multiple nodes. Guaranteeing lesser duplicates, in-progress successfully sent records, and acknowledging completion of sync tasks are really interesting problems at scale.</li><li><strong>Proper metrics and alerts</strong>: We added extensive metrics and various alerts, like time taken for each task, number of records processing from extraction to transformation to destination API calls, sync latencies for batches of records, and more.</li><li><strong>Central reporting on top of metrics</strong>: Beyond just metrics for Reverse ETL, there is a need for a central reporting system as multiple systems are involved in running the pipeline, from extraction to final destination. We wanted to capture details for all stages to ensure we had full auditability for every pipeline run.</li></ul><p>Again, RudderStack Core was a huge help in shipping several of the above components of the system:</p><ul><li><a href="https://www.rudderstack.com/docs/destinations/"><strong>Destinations</strong></a>: when it comes to integrations, maintenance is critical because things must be kept up to date. Many times things fail because of destination API upgrades or different rate limits, not to mention upkeep like adding additional support for new API versions, batch APIs, etc. Because destinations are a part of RudderStack Core, the Reverse ETL team doesn’t have to maintain any destination functionality.</li><li><a href="https://www.rudderstack.com/docs/user-guides/administrators-guide/monitoring-and-metrics/"><strong>Metrics</strong></a>: rudder-server already included metrics for things like successfully sent counts, failed counts with errors, and more, all of which we were able to use for our Reverse ETL pipelines.</li><li><a href="https://www.rudderstack.com/docs/get-started/live-events/"><strong>Live Debugger</strong></a>: Seeing events flow live is incredibly useful for debugging while sync is running, especially because we don’t store data in RudderStack. We were able to use the existing Live Debugger infrastructure for Reverse ETL.</li></ul><h3>Concluding thoughts</h3><p>Building out our Reverse ETL product was an amazing experience. While there were many fun challenges to solve, I have to reiterate my appreciation for the foresight of our founding engineers. As you can see, without RudderStack Core this would have been a much more challenging and time consuming project.</p><p>If you made it this far, thanks for reading, and if you love solving problems like the ones I covered here, come join our team! Check out our open positions <a href="https://boards.greenhouse.io/rudderstack">here</a>.</p><p>This blog as originally published at: <a href="https://www.rudderstack.com/blog/how-rudderstack-core-enabled-us-to-build-reverse-etl/">https://www.rudderstack.com/blog/how-rudderstack-core-enabled-us-to-build-reverse-etl/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4dc734df9fdb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/rudderstack/how-rudderstack-core-enabled-us-to-build-reverse-etl-4dc734df9fdb">How RudderStack Core Enabled Us To Build Reverse ETL</a> was originally published in <a href="https://medium.com/rudderstack">RudderStack</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>