<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tristan Baker on Medium]]></title>
        <description><![CDATA[Stories by Tristan Baker on Medium]]></description>
        <link>https://medium.com/@tcbakes?source=rss-28b642ad293a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*oCDSPONbPSqfewzw</url>
            <title>Stories by Tristan Baker on Medium</title>
            <link>https://medium.com/@tcbakes?source=rss-28b642ad293a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 14:11:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tcbakes/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Data Mesh Strategy Behind Intuit’s Global Financial Technology Platform]]></title>
            <link>https://medium.com/intuit-engineering/the-data-mesh-strategy-behind-intuits-global-financial-technology-platform-db862fd45e0b?source=rss-28b642ad293a------2</link>
            <guid isPermaLink="false">https://medium.com/p/db862fd45e0b</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-mesh]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Tristan Baker]]></dc:creator>
            <pubDate>Mon, 22 Apr 2024 16:41:24 GMT</pubDate>
            <atom:updated>2024-04-22T16:41:24.758Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Over the past several years, Intuit has built foundational capabilities for collecting, processing and transforming raw data into a connected mesh of high quality data. Those capabilities are enabling our technologists to build personalized experiences, with speed and at scale, to deliver on our mission to power prosperity around the world for approximately 100 million consumer and small business customers on our global financial technology platform with TurboTax, Credit Karma, QuickBooks and Mailchimp.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G7dnLZhi6RwISb3XGy-4Fw.jpeg" /></figure><p>In previous posts, I’ve shared the motivation behind Intuit’s <a href="https://medium.com/intuit-engineering/intuits-data-mesh-strategy-778e3edaa017">Data Mesh Strategy</a> and provided an in-depth look at the <a href="https://tcbakes.medium.com/intuits-data-mesh-concepts-214268257dd2">Data Mesh Concepts</a> behind the experiences, services and processes we’ve implemented in a small corner of our our vast data estate, which has given us the confidence to scale more broadly across Intuit. For an enterprise company of our sheer size, scope and history (40+ years!), this means managing hundreds of thousands of tables with petabytes of data and harnessing decades of information across various systems, all while continuing to expand our portfolio of products and services.</p><p>Given reader interest in this hot industry topic over the past few years, I thought it might be helpful to provide a high level overview of our data mesh journey as an introduction for anyone setting out on their own journey — and as a precursor to deeper dives I’ve written about on this platform.</p><h4><strong>The challenge — and opportunity</strong></h4><p>Back in 2021, we were facing an exponential increase in internal demand for access to high quality, real-time data across the enterprise from a broad spectrum of data workers (service engineers, UX developers, business analysts, data analysts, data scientists, machine learning engineers, etc.).</p><p>This led to a variety of challenges that will be familiar to anyone who’s spent any time in a data lake trying to make sense of structured and unstructured data to generate business value:</p><ul><li><strong>Discovering data.</strong> Teams didn’t always understand where they could find data related to the specific problem they were solving, as well as where to find data sourced from a particular product or service.</li><li><strong>Understanding data.</strong> Teams didn’t necessarily know who could give them access to data, how existing data was structured or used in the business, or how data interacted with other concepts or data sets within Intuit.</li><li><strong>Trusting data</strong>. It wasn’t always clear what system produced or used data and how quickly data could be delivered. Data quality wasn’t immediately clear, nor could teams always tell who would help if something broke.</li><li><strong>Consuming data</strong>. It wasn’t always easy to find out who could approve data access for production systems or whether teams would be notified when data structures changed.</li><li><strong>Publishing data.</strong> There was no uniform guidance for describing, hosting and maintaining data systems, so teams were unclear on how they should meet operational and compliance requirements, or whether they were duplicating data or data processes that already existed.</li></ul><p><strong>The solution — a distributed enterprise data mesh architecture</strong></p><p>A survey of solutions for a company of our size and scope led us to Zhamak Dehghani’s <a href="https://martinfowler.com/articles/data-monolith-to-mesh.html">description of a “data mesh.</a>” The idea of shifting from a centralized data lake to a distributed enterprise data mesh architecture immediately caught our attention.</p><p>We recognized the potential to “adapt and apply the learnings of the past decade in building distributed architectures at scale, to the domain of data” with ownership and accountability given to individuals and teams. Having seen this approach work well at the “front of the house” with our <a href="https://quickbooks-engineering.intuit.com/quickbooks-a-platform-transformation-1bd98cc70aa2">microservices architecture</a> and core transactional services, we were confident it could be successfully applied to “back of the house” data systems.</p><p>With data mesh fundamentals in mind, we set out to empower data “producers” to design, develop, fully describe and actively support their own data-driven systems, and enable data “consumers” to easily discover, understand, trust and consume the data. Altogether, our intention was to power a network effect of data production and consumption by:</p><ol><li><strong>Systematically organizing people, code and data</strong>. We define internal processes and initial data sets that produce externally consumable data as a data product. To support discoverability, we link those products to the business problem they help solve.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/0*lCPDAZdzycabHi64" /></figure><ol><li><strong>Clearly defining ownership of each data product.</strong> We ensure teams understand and are accountable for a defined set of responsibilities and best practices for producing high-quality data. Teams are typically made up of one or more data workers, a product manager, and a leader responsible for the timely and efficient execution and delivery of a business solution.</li><li><strong>Providing tools for designing, authoring, deploying and operating data products. </strong>We integrate these capabilities into a central, self-serve platform service with built-in adherence to security, authentication, authorization, access control policy management, change control procedures, schema registration and documentation standards. We also integrate quality, cost and performance measurement and reporting into the platform so that individual teams — and the organization at large — can understand how well a particular data product is performing.</li></ol><p>To date, we’ve seen a <strong>26 percent productivity improvement</strong> from our deployment in a small corner of Intuit’s large data estate, as measured by the time it takes for Intuit teams to discover, access and explore data for a new project when compared to an environment where data mesh has not yet been deployed.</p><p><strong>Expanding the data mesh at scale across the enterprise</strong></p><p>The results of our initial deployment has given us the confidence to scale our deployment to the entirety of Intuit to deliver massive benefits to our teams, our operations and, ultimately, our customers.</p><p>To learn more about how you could apply our learnings to your own data mesh journey, check out my <a href="https://medium.com/intuit-engineering/intuits-data-mesh-strategy-778e3edaa017">Data Mesh Strategy</a> and <a href="https://tcbakes.medium.com/intuits-data-mesh-concepts-214268257dd2">Data Mesh Concepts</a> blogs. To see some of the data mesh tools and experiences in action, see this <a href="https://youtu.be/bGWe0TzkQEE?si=VB7OWQZkmKbdfT6O&amp;t=1218">talk</a> by my Intuit colleague, <a href="https://www.linkedin.com/in/ramansuresh/">Suresh Raman</a>.</p><p>My hope is that you’ll leverage this content in the same way my teams have — as a guide for implementing the experiences, services, and processes necessary to realize the benefits of a data mesh strategy.</p><p>I’m proud of how far we’ve come, and excited about what the future holds. Since my <a href="https://tcbakes.medium.com/intuits-data-mesh-concepts-214268257dd2">October 2023 Data Mesh Concepts post</a>. There’s even a possibility that we’ll open source Intuit’s internal API definitions and service implementations to develop a standard approach for implementing a data mesh across the industry.</p><p><em>Special thanks to my partners at Intuit who have contributed to the insights and ideas that have shaped Intuit’s data mesh strategy: Shachar Bar David, Stephen Molloy, Yaniv Levi, Daniel Sharvit, Krithika Swaminathan, Achal Kumar, Arun Ragothaman, Narayanan Singaram, Juhi Dhingra, Rui Dai, JD Rosensweig, Adi Ohana, Allison Bellah, Samuel Knapp, Robert Mei, Timur Fayruzov, Larry Raab, Sreekanth Martha, Barry Nisly, Ron Sher, Karen Maciolek, Dunja Panic, Ashish Page, Omar Abdelmagid, Robin Oliva-Kraft, Akbar Rangara, Kaushal Sheth, Guru Prakash Narayana, Ryan Quigley, Raj Chandramohan, Denise McInerney, Aveek Misra, Prashanth Sashadri, Elmer KimNii, Jainik Vora and Saikiran Thunuguntla.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=db862fd45e0b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/intuit-engineering/the-data-mesh-strategy-behind-intuits-global-financial-technology-platform-db862fd45e0b">The Data Mesh Strategy Behind Intuit’s Global Financial Technology Platform</a> was originally published in <a href="https://medium.com/intuit-engineering">Intuit Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Intuit’s Data Mesh Concepts]]></title>
            <link>https://tcbakes.medium.com/intuits-data-mesh-concepts-214268257dd2?source=rss-28b642ad293a------2</link>
            <guid isPermaLink="false">https://medium.com/p/214268257dd2</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[architecture]]></category>
            <category><![CDATA[data-mesh]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Tristan Baker]]></dc:creator>
            <pubDate>Wed, 25 Oct 2023 03:03:20 GMT</pubDate>
            <atom:updated>2023-10-25T14:46:36.574Z</atom:updated>
            <content:encoded><![CDATA[<p>In <a href="https://medium.com/intuit-engineering/intuits-data-mesh-strategy-778e3edaa017">a prior article</a>, I described the motivation for Intuit’s data mesh strategy. Intuit needs data-driven systems to enable more intelligent product experiences and the development tools and processes to enable more Intuit teams to create them more easily. The data mesh we built at Intuit has helped us to do that.</p><p>Since publishing the first article, we’ve deployed a data mesh in a small corner of Intuit’s large data estate. (To better understand some of the data mesh tools and experiences, see them in action in <a href="https://youtu.be/bGWe0TzkQEE?si=VB7OWQZkmKbdfT6O&amp;t=1218">a talk</a> by my colleague, <a href="https://www.linkedin.com/in/ramansuresh/">Suresh Raman</a>).</p><p>The solution we’ve created has delivered the following benefits:</p><ol><li>A 26% productivity improvement as measured by the time it takes for Intuit teams to discover, access, and explore data for a new project.</li><li>A significant security improvement, as measured by the principle-of-least-access-privilege (PLAP) score (apologies for the lack of specifics, but Intuit’s security metrics are generally not publically disclosed).</li><li>A 44% decrease in LLM hallucinations in our internal developer-facing chatbots.</li></ol><p><em>NOTE </em>— These improvements were determined by measuring outcome metrics (productivity, security, LLM hallucinations) in an environment where data, systems, and people leveraged the data mesh platform and comparing them to the same measures in an environment where data mesh was not used.</p><p>These numbers give us the confidence to scale from our “small corner” to the entirety of Intuit and see the same benefits, delivering a much more significant total impact for the company.</p><p>This article will detail the concepts that provide the foundation for Intuit’s data mesh implementation. The content of this article is derived from similar internal papers, presentations, and diagrams that document the target state architecture of Intuit’s data mesh implementation and are used as the primary artifacts for generating requirements documents, user flows, experience designs, and service APIs that collectively implement Intuit’s data mesh.</p><p>I hope you might use this in the same way my teams have — as a guide for implementing the experiences, services, and processes necessary to realize the benefits of a data mesh.</p><p>In future work, and depending partly on what kind of feedback I receive from this article, I may open-source Intuit’s internal API definitions and service implementations to develop a standard approach for implementing a data mesh across the industry.</p><p>Special thanks to my partners at Intuit who have contributed to the insights and ideas that have shaped this approach: <a href="https://www.linkedin.com/in/shachar-bar-david-844a4750/">Shachar Bar David</a>, <a href="https://www.linkedin.com/in/stephen-molloy-b0a1aa/">Stephen Molloy</a>, <a href="https://www.linkedin.com/in/yaniv-levi-7155532/">Yaniv Levi</a>, <a href="https://www.linkedin.com/in/dsharvit/">Daniel Sharvit</a>, <a href="https://www.linkedin.com/in/engleader/">Krithika Swaminathan</a>, <a href="https://www.linkedin.com/in/kumarachal/">Achal Kumar</a>, <a href="https://www.linkedin.com/in/arunragothaman/">Arun Ragothaman</a>, <a href="https://www.linkedin.com/in/narayanansingaram/">Narayanan Singaram</a>, <a href="https://www.linkedin.com/in/juhi-dhingra-91280922/">Juhi Dhingra</a>, <a href="https://www.linkedin.com/in/ruidai/">Rui Dai</a>, <a href="https://www.linkedin.com/in/jdrosensweig/">JD Rosensweig</a>, <a href="https://www.linkedin.com/in/adiohana/">Adi Ohana</a>, <a href="https://www.linkedin.com/in/allisonbellah/">Allison Bellah</a>, <a href="https://www.linkedin.com/in/knappsam/">Samuel Knapp</a>, <a href="https://www.linkedin.com/in/robert-mei-a79168a/">Robert Mei</a>, <a href="https://www.linkedin.com/in/timurfayruzov/">Timur Fayruzov</a>, <a href="https://www.linkedin.com/in/lraab/">Larry Raab</a>, <a href="https://www.linkedin.com/in/sreekanthmartha/">Sreekanth Martha</a>, <a href="https://www.linkedin.com/in/barrynisly/">Barry Nisly</a>, <a href="https://www.linkedin.com/in/ronshersoftware/">Ron Sher</a>, <a href="https://www.linkedin.com/in/karen-maciolek-376b183/">Karen Maciolek</a>, <a href="https://www.linkedin.com/in/dunja-panic/">Dunja Panic</a>, <a href="https://www.linkedin.com/in/ashish-page-03429315/">Ashish Page</a>, <a href="https://www.linkedin.com/in/omar-abdelmagid-453b0813/">Omar Abdelmagid</a>, <a href="https://www.linkedin.com/in/robinkraft/">Robin Oliva-Kraft</a>, Elmer KimNii, <a href="https://www.linkedin.com/in/jainikvora/">Jainik Vora</a>, <a href="https://www.linkedin.com/in/saikiranthunuguntla/">Saikiran Thunuguntla</a></p><h3>Concepts</h3><p>This section enumerates all the critical concepts and their relationships to each other. Where helpful, entity relationship diagrams are provided to more clearly illustrate what a term means and how it is used.</p><h4>1 — Business Unit</h4><p>A distinct part of the organization is responsible for specific functions or objectives and comprises one or more teams. Business units are also responsible for paying fees for utilizing resources or services. This financial responsibility ensures that each business unit is accountable for its consumption of resources and encourages efficient usage of the data platform and its services.</p><h4>2 — Team</h4><p>A group of individuals backed by a business unit, working together to achieve shared goals. This group is realized as the owning Team of an Intuit Dev Portal.</p><h4>3 — Intuit Dev Portal Project</h4><p>Intuit Dev Portal is an internal tool for development teams to help them create products and services.<br> <br>An Intuit Dev Portal Project is the primary organizing unit in the Intuit Dev Portal. It is a collaborative workspace where team members can manage, develop, and share resources related to their data work. Typically, a project represents a specific initiative owned by a team within a business unit. The Intuit Dev Portal Project is the central point of reference to connect deployed resources to the responsible owners.</p><h4>3.1 — Human Role</h4><p>A role represents a group of individuals within an Intuit Dev Portal Project Team who fulfill specific job functions within the project and the owning team. Each role is associated with credentials that grant authorized access to consume data, produce data, or both.</p><h4>4 — Data Stewards</h4><p>Data Stewards are accountable for producing, operating, and managing their data per Intuit’s Data Stewardship guidelines. There are two types of data stewards:</p><h4>4.1 — Business Data Steward</h4><p>Responsible for the definition and quality expectations of data that a system produces. The “Data Steward” human role is used by the Business Data Stewards on the team.</p><h4>4.2 — Operational Data Steward</h4><p>Responsible for creating and deploying systems that produce data according to stated data definitions and quality expectations. Operational Data Stewards use the “Developer” human role.</p><h4>5 — Domain</h4><p>A logical grouping of related logical boundaries. Domains provide a standardized approach to organizing data in the Data Map.</p><h4>6 — Bounded Context</h4><p>A logical boundary within a specific domain or subdomain that isolates its data from other contexts. Bounded contexts also serve as the primary way to browse and discover data in the Data Map, making it easier for users to navigate and understand the relationships between different data elements.</p><h4>7 — Data Model</h4><p>A data model is a formal description of a related set of concepts and definitions and their relationships to each other.</p><h4>7.1 — Semantic Model</h4><p>A specific kind of data model that conveys the semantic meaning of data and relationships within a bounded context. It is decoupled from but related to a Data Schema that describes this data model’s representation in a specific hosting location.</p><p>These models follow domain-driven design methodologies and typically define aggregate roots, entities, value objects, etc.</p><h4>7.2 — Data Schema</h4><p>A formal representation of the structure of a data model when stored in a particular hosting location. The data schemas are informed by and should be consistent with the semantic model but may deviate to meet the format requirements and access patterns of the hosting location where the data is stored. The data schema may have semantic and compliance annotations that provide additional context and meaning. The data schema helps ensure that the data is appropriately managed and processed, minimizing the risk of regulatory or policy violations. Examples of data schemas include JSON-Schema, Hive DDL, etc.</p><h4>8 — Data Resource</h4><p>A Data Resource refers to data managed by a data persistence system. This includes but is not limited to a table in a database, a file in a file system, an s3 object in an s3 bucket, a topic in a messaging system, etc. A data resource becomes a data asset when registered and provided an IRN.</p><h4>9 — Hosting Location</h4><p>A system that facilitates the persistence and serving of data assets by adhering to the hosting location requirements, including enforcing a centralized access change control process. This ensures a controlled and secure data exchange with other teams, promoting efficient collaboration and data management across the company.</p><p>Hosting locations include systems like Intuit’s Customer Data Lake, Operational Data Lake, Event Bus, and Customer Data Profile Store.</p><h4>9.1 — Data Protocols</h4><p>The supported ways a hosting location can be interacted with to mutate, store, and retrieve data. This includes but is not limited to, Kafka protocol, GraphQL RPC protocol, HTTP+REST protocol, HTTP+S3 protocol, etc.</p><h4>9.2 — Catalog</h4><p>A hosting location’s catalog maintains a registry of schemas used to describe the data assets stored within it. The catalog enables users and systems to quickly understand and work with the data stored in the Hosting Location. For example, an instance of a Hive Metastore is Intuit’s data lake catalog.</p><h4>9.3 — Tenant</h4><p>A hosting location tenant refers to a team granted access to manage its data resources within a shared hosting location environment. Each tenant has dedicated data resources, ports, and access control policies, ensuring that their data exchange processes are isolated from those of other tenants.</p><p>The following diagram illustrates a hosting location, the team, the hosted data resource, and the data port serving the data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*hUuTBPgM7V6_-eV58CZx7Q.png" /></figure><h4>10 — Data Access</h4><p>Data access refers to the process and resources for granting or restricting the right to interact with specific data resources within hosting locations. It considers human and machine access, ensuring a clear and consistent approach to managing access privileges to hosting locations.</p><h4>10.1 — Data Access List</h4><p>A data access list is a collection of data access entries defining access control privileges for a particular data asset. It specifies the allowed human and machine credentials and their respective access privileges to the data asset.</p><h4>10.2 — Data Access Entry</h4><p>A data access entry is an individual record within a data access list. It enumerates a set of specific credentials and corresponding access privileges granted to actors bearing those credentials. Each entry contains information such as a credential identifier, type (e.g., intuit authentication ticket, AWS role), and level of access granted (e.g., read, write ).</p><h4>10.3 — Credential</h4><p>A credential is a unique identifier assigned to a data processing system or a human role within an Intuit dev portal project team. These credentials facilitate the secure and controlled data exchange between data producers and consumers.</p><h4>10.4 — Project-Scoped Credential</h4><p>A project-scoped credential adheres to the requirement that the credential is only used to write to data assets owned by the same project as the credential. A credential that fails to meet this requirement is misused and contributes to problematic behavior, including creating data in a complex way to govern or attribute ownership.</p><h4>11 — Data Asset</h4><p>A data asset formally represents a data resource that aggregates essential information such as ownership, access control list, reference to the end-to-end data lineage, and data port information such as the connection to a hive table, the connection to an event bus topic, etc.</p><h4>11.1 — IRN</h4><p>An Intuit Resource Name (IRN) is a unique identifier used within Intuit’s data ecosystem to identify and reference data assets. The IRN is designed to provide a standardized naming convention, making it easier to locate, manage, and track data assets across various systems and services.</p><h4>11.2 — Data Port</h4><p>The description of the interface provided by hosting locations enables controlled data exchange of a particular data asset owned by a specific data steward team. The data port facilitates this exchange while adhering to access policies administered and governed by the central data platform, ensuring secure and compliant data exchange across the company. This interface can serve data from data ports (such as a GraphQL endpoint or Kafka topic) or receive data from data ports (such as commands or mutation via an HTTP API).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5GWrD8EYVvRE9QFPzQDz_g.jpeg" /></figure><p>Fig: An illustration of the data asset entity</p><h4>12 — End-to-End Data Lineage</h4><p>End-to-end data lineage is a directed graph representing the relationships and dependencies between data assets and development assets. It provides visibility into the flow, transformation, and origin of data throughout its lifecycle, potentially going down to the attribute level. This holistic view enables better traceability, understanding, and data management across systems and teams. The end-to-end data lineage is a graph of nodes and declared or observed edges.</p><h4>12.1 — Backward Data Lineage</h4><p>Backward data lineage refers to the ability to trace data’s origin, movement, and transformation across its lifecycle, from its creation to its final use in data products. This includes tracking how the data was collected, transformed, and integrated with other sources to create the final data product.</p><h4>12.2 — Forward Data Lineage</h4><p>Forward data lineage refers to tracing the downstream flow of data from its origin through various processes, transformations, and integrations that lead to its consumption in data-consuming systems. It allows data stewards to understand the impact of changes to a specific data port on the dependent data-consuming systems, enabling better management of data quality, governance, and regulatory compliance.</p><h4>12.3 — Declared Data Lineage</h4><p>Declared data lineage refers to tracing the declarations of data usage as described in data access control lists.</p><h4>12.4 — Observed Data Lineage</h4><p>Observed data lineage refers to tracing actual data read and write activity, as inferred from processing systems and hosting location access logs, which are the source of truth for data lineage.</p><p>The following diagram depicts a prototypical data lineage graph:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PAnCTrcPGWN9k2G2KOIDjA.jpeg" /></figure><h4>13 — Data Recipe</h4><p>A data recipe formally represents deployment operations needed to set up a data processing system to produce a data product. Data recipes reference data modules. Data modules describe how to provision and connect hosting locations and processing infrastructure.</p><h4>13.1 — Data Transformation Module</h4><p>Modules that define how data is processed, transformed, and produced to hosting locations to meet specific business requirements. This includes but is not limited to training processing tasks or inference tasks.</p><h4>13.2 — Data Movement Module</h4><p>Modules that specify how data is transferred, but not transformed by processing systems, and ingested between hosting locations.</p><h4>13.3 — Data Observability Module</h4><p>Modules that establish monitoring, logging, and alerting mechanisms for tracking the performance, quality, and health of data processes and data assets. This includes but is not limited to data quality rules, data parity, security policy violations, etc.</p><p>These data modules work together within a data recipe to streamline and automate data processing tasks, making it easier for teams to develop a processing system declaratively without getting bogged down in infrastructure complexities.</p><p>The following diagram depicts the Data Recipe entity and its three types of data modules:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*QH1sbWv9OXIBa_JEXkR5rw.png" /></figure><h4>14 — Data Product</h4><p>Data Products are the foundational data unit of the Data Mesh and are organized by bounded context and data domains. A data product consists of one or more data assets. It consolidates essential information such as ownership, semantic model, data schema, service level agreements (SLAs), star rating, tags, and data recipe. It is important to note that data products are only discoverable once they meet the minimum 3-star threshold for clean data as determined in the <a href="#26d6">clean data maturity model</a>. This rating ensures that the data product is sufficiently described and managed such that it will facilitate cross-team collaboration and reliable data exchange. A data product definition is required for every consumable data asset. An example of a data product could be the “Customer Account Data Product”, describing a semantic model of a “Customer”, realized as a real-time stream of customer-account-created events in an event bus topic and as rows of customer-account-created events in a data lake table.</p><h4>14.1 — Name</h4><p>The name attribute of a data product is a human-readable display name, enabling easy understanding and communication among data stewards and consumers about its content and purpose.</p><h4>14.2 — Tag</h4><p>Tags are descriptive labels or keywords associated with a data product to enhance its discoverability and facilitate easier search and identification within Data Map discovery experiences.</p><h4>14.3 — Service Level Objective</h4><p>A service level objective (SLO) is a measurable target that defines the expected level of performance or reliability for a specific aspect of a data product. Common SLOs include data quality, data freshness, and data availability. The SLO enables stakeholders to set clear and quantifiable targets, monitor performance against those targets, and make data-driven decisions to improve performance and reliability.</p><h4>14.4 — Service Level Agreement</h4><p>A service level agreement (SLA) is a contract between a data steward and its consumers that specifies the set of SLOs for the data product. It is precise enough that measuring adherence to the SLA is automatable. The SLA enables the consumer to monitor the data product’s performance against agreed-upon targets and make data-driven decisions to manage their business operations.</p><h4>14.5 — Data Recipe Reference</h4><p>A reference to a data recipe that represents the producing task involved in moving, transforming, and observing the data of a data product.</p><p>The following diagram illustrates the various components of the data product entity:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AmMHgta5KLQvL8UkPnqU0g.jpeg" /><figcaption>An illustration of data product and its relationship to other concepts</figcaption></figure><h4>15 — Data Asset Kinds</h4><p>Data asset kinds categorize data based on its intended use, access level, and potential consumers. The data asset kinds include:</p><h4>15.1 — Internal Data Assets</h4><p>Internal data assets refer to data generated, managed, and consumed within a specific team. This type of data is often used for operational activities. Access to internal data is restricted to authorized team members and is not intended for external consumption. Internal data must meet two stars on the <a href="#26d6">clean data maturity model</a>, which requires the appropriate metadata to address compliance and security concerns to which all data is subject.</p><h4>15.1.3 — Temporary Data</h4><p>Temporary data refers to generated data that is not meant to be long-living. It is typically generated during data exploration for internal decision-making and analysis. It is often data that does not have a regularly scheduled pipeline associated, is not monitored, and is not consumed (or should not be). Examples include data created by one-off “CREATE TABLE” queries during data exploration or one-off model experiments that generate data but are not promoted to a production model. Temporary data must meet two stars on the <a href="#26d6">clean data maturity model</a>, which requires the appropriate metadata to address compliance and security concerns to which all data is subject.</p><h4>15.2 — Consumable Data</h4><p>Consumable data asset is data made available for cross-team data exchange. It should be understandable, durable, and usable by various consumers, both internal and external to the team. It is measured according to the <a href="#26d6">clean data maturity model</a> and subject to the discoverability and consumption constraints described therein. Consumable data facilitates cross-team collaboration, data exchange, and integration with other systems or applications.</p><p>The following diagram depicts the data kinds and their sub-classifications:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PvvAyn_QAZIAx7-ywvEO6w.png" /></figure><h4>16 — Data Registration</h4><p>The workflows and processes for data producers to register data assets and promote them into data products. This includes workflows for internal and consumable data.</p><h4>16.1 — Data Product Lifecycle</h4><p>The lifecycle of data products includes:</p><ul><li>in-development</li><li>released</li><li>deprecated</li></ul><h4>16.2 — Data Product Intent</h4><p>Data Products can be intended for broad consumption or internal use.</p><ul><li>Internal</li><li>Consumable</li></ul><h4>16.3 — Clean Data Governance</h4><p>Data Registration uses the <a href="#26d6">clean data maturity model</a> to govern and rate data. Data governance depends on the data kind and will be the gatekeeper for what is discoverable and accessible to users.</p><h4>16.3.1 — Consumable Data Threshold</h4><p>Data products with the consumable intent are considered released and have a clean data maturity star rating of 3 or higher.</p><h4>16.4 — Clean Data Maturity Star Rating</h4><p>A quantitative metric, calculated based on the <a href="#26d6">clean data maturity model</a>, evaluates a data product’s overall quality, reliability, and usability. This star rating considers various aspects, such as data cleanliness, schema, semantic context, and adherence to service level agreements (SLAs). The clean data maturity star rating creates a clear baseline of expectations for data stewards to meet while also providing consumers with transparency about a data product’s strengths and weaknesses so that they can make informed decisions when selecting data products for consumption.</p><h4>16.5 — Clean Data</h4><p>Data that meets the 3-star threshold defined in Intuit’s <a href="#26d6">clean data maturity model</a> for consumable data.</p><h4>17 — Data Map</h4><p>User experiences for discovering consumable data products are organized by domain, subdomain, and bounded context. Note that internal data is generally not discoverable or accessible except by the team that owns it. Thus, any Data Map experiences that assist in discovering internal data must consider the profile of the person doing the searching and the set of data assets they own.</p><h4>17.1 — Data Map Registry</h4><p>A registry of internal and consumable data assets includes a graph of declared &amp; inferred data lineage, internal and consumable data assets, data products, domains, subdomains, bounded contexts, and data models.</p><h4>17.2 — Data Map Taxonomy</h4><p>A data domain, subdomain, and bounded context define a taxonomy that organizes all the data products in the data map.</p><h4>18 — Data Processing Runtime</h4><p>A data processing runtime refers to any runtime that enables the execution of data processing tasks, such as data ingestion, transformation, analysis, and storage. These runtimes (e.g., spark batch processing, Kubernetes microservice, and flink runtimes) provide the tools, libraries, and resources to efficiently process and manipulate large data assets in real-time or batch mode.</p><h4>19 — Customer Environment</h4><p>A customer environment comprises production-grade components for teams to use to develop and test their data products. This includes the data processing runtimes and hosting locations that enable customers to deploy, test, and publish data products. The processing runtimes within the customer environment are stable and reliable, providing a standardized and scalable approach to managing the deployment and operation of data products.</p><h4>20 — Experimentation Session</h4><p>An experimentation session enables teams to test new ideas using data assets. This testing is generally an iterative process of analysis and refinement that produces insights into optimizing business processes, products, services, and features. It involves collecting and analyzing data, executing experiments, and drawing conclusions to inform decision-making. This session is controlled by the central data platform, ensuring compliance and proper management of data assets and processes.</p><h4>21 — Data Processing System</h4><p>A deployable unit of code and infrastructure that can consume and produce data assets.</p><h4>21.1 — Data-Producing System</h4><p>A system that generates internal or consumable data assets. Examples include, but are not limited to, microservices and ETL pipelines.</p><h4>21.2 — Data-Consuming System</h4><p>A data-consuming system is a component that accesses data assets. These systems can consume internal data assets produced by the same system and data products generated by other systems.</p><h3>Use Cases</h3><p>This section provides a better understanding of the interrelations between the various concepts as they are typically arranged for various use cases.</p><h4>Use Case 1— A microservice publishes events to a hosting location.</h4><p>This data-producing system generates event data, typically describing specific business operations that have recently occurred. The microservice then publishes these events to designated hosting locations, making them available for other systems or teams to consume. The diagram refers to an “outbox service”, which is a recipe module that implements the <a href="https://microservices.io/patterns/data/transactional-outbox.html">transactional outbox pattern</a> for reliably sending events in a distributed, eventually consistent microservices architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FXC89DgDygySIDXFo1WP8g.png" /></figure><h4>Use Case 2— A frontend application publishes clickstream data to hosting locations.</h4><p>This data-producing system captures user intent on a web application, such as clicks, scrolls, and navigation events. The application then publishes the collected clickstream data to hosting locations, providing valuable insights into user behavior and preferences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*dK_eGGxk7wuSb4iNwkrUqQ.png" /></figure><h4>Use Case 3— An ETL pipeline transforms a data product to create a new data product.</h4><p>This data-producing system transforms existing data products into derivations tailored to specific consumption use cases. The pipeline then moves the transformed data to hosting locations, making it available for further analysis and consumption by other teams or systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*89_k-eWqvgGt-ZeheX0a0Q.png" /></figure><h3>Architecture</h3><h4>System Context</h4><p>The following diagram illustrates the <a href="https://c4model.com/">C4 model</a> system context of the data mesh domain (at Intuit, we refer to this officially as the “L0 Data” domain, hence the references to “L0 Data” in some of the following diagrams):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lM82_lC_TWCOCZmsU_I8mA.jpeg" /></figure><h4>Entity Relationship Diagram</h4><p>The following diagram depicts the representation of the key domain concepts and their relationships. This diagram helps in understanding how the different components of the domain interact with each other. This diagram serves as a guide for further exploration and refinement of the underlying structure and interdependencies within the data mesh domain:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5c9dIhyqgzt7s2LlHDsEag.png" /></figure><h4>Implementation Guidelines</h4><p><strong>Lockdown hosting locations</strong> — The platform must retain administrative control of hosting locations so that data access (or lack thereof) can be used to enforce compliance with standards. As we like to say at Intuit, “if you don’t follow the rules, you can’t touch the data.”</p><p><strong>Develop data maturity standards</strong> — Develop and document a comprehensive data maturity framework that includes schema management, data models, access policies, data quality, and observability controls. This framework will serve as a guideline for data products to adhere to various levels of standardization based on the data maturity decision.</p><p><strong>Facilitate data registration</strong> — Design and implement the architecture to facilitate the registration of domains, bounded contexts, data models, data assets, and data products, enabling data stewards to define and document their data. A system of checks should be in place to ensure that new domains and new data products do not conflict with or duplicate existing domains and data products. We’ve found that duplication of data products is a leading driver of confusion among the general data-consuming population, which is why it has been added as a criterion in the data maturity standard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ymloHnGE3cPrbS_oYKmgeQ.jpeg" /><figcaption>Bounded contexts and entities are incorporated to provide semantic meaning to data products.</figcaption></figure><p><strong>Facilitate the transition of existing data to data assets and data products</strong> — Streamline the process for existing data producers to adopt the data maturity framework by providing clear guidance and interfaces for creating data products from their existing data assets. Simplifying this transition will encourage adherence to data maturity standards that support durable data integration and collaboration across teams.</p><p><strong>Facilitate data lineage registration</strong> — Streamline the process of registration data lineage for each data product so that data consumers can trace the data’s origin and stewards can understand data consumption and usage.</p><p><strong>Facilitate data recipe interfaces to accelerate data development</strong> — To support adopting and using data recipes, provide intuitive and easy-to-use interfaces for creating, managing, and deploying data recipes. These interfaces should allow data stewards, service developers, data engineers, and analysts to define and configure data transformations, movements, and observability modules with minimal effort, embracing low-code or no-code principles.</p><p><strong>Support internal data exchange and persistence — </strong>Empower data producers to create and store secure, high-quality internal data within hosting locations while promoting efficient data exchange within the same team.</p><p>The following diagram depicts the prototypical architecture of an internal data data-producing system:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FFMFdNdVNb3K_B0NKQ6JMQ.jpeg" /><figcaption>An illustration of an internal data data-producing system exchanging data exclusively with systems owned by the same team.</figcaption></figure><p><strong>Apply the principle of least access privileges</strong> — Ensure that credentials, for either machine or human access, are granted the minimum set of permissions necessary to perform their consumption tasks. Continuously review and update user roles and permissions to maintain adherence to the PLAP. Providing data stewards with easy-to-use interfaces and processes that enable them to revoke and grant access to data consumers easily is crucial to ensuring efficient and secure data access management.</p><h3>Key Performance Metrics</h3><p>The following set of metrics has been developed to measure the value delivered by the data mesh and its collection of capabilities.</p><h4>Clean Data Adherence Rate (CDAR)</h4><p>This metric measures the extent to which data producers within the company adhere to the <a href="#26d6">clean data maturity model</a>. It is calculated as the ratio of the total number of data product connections to the total number of data connections between different teams within the company during a given period. A higher CDAR indicates better adherence to the <a href="#26d6">clean data maturity model</a>, thereby improving the overall data quality and reducing the risk of internal data consumption by humans or machines of different teams. CDAR can help identify potential areas for improvement and inform the development teams of the data platform policies and training programs for data producers.</p><h4>Schema Management Efficiency (SME)</h4><p>This metric describes the effectiveness of the data modeling capability in reducing the time it takes for customers to manage their schemas while complementing their existing user journeys. This metric can be measured by monitoring the time spent on schema management tasks, the number of schema modifications, and the user satisfaction rate related to schema management.</p><h4>Principle of Least Privileges Score (PLAP)</h4><p>This metric assesses how closely an organization’s data access policies comply with the principle of least access privilege. This principle requires that humans and machines are granted only the minimum necessary permissions to perform their required tasks, thus reducing the risk of unauthorized access and enhancing data security. PLAP evaluates an organization’s adherence to this principle by measuring the total number of privileges granted to credentials against the actual access usage for these credentials.</p><h4>Hosting Location Lockdown Score ( HLLS )</h4><p>This metric quantifies the extent to which data access control policies are enforced. It is calculated by comparing the number of credentials that strictly adhere to the rule of allowing data-producing systems to write data only to the data assets associated with the same project as the credential and the data-producing system against the total number of credentials that can write data to the same data assets. This score provides insights into the effectiveness of the data access policy enforcement and helps identify areas where improvements may be needed to ensure proper data read and write activities.</p><p>The diagrams below show an example of illegal data reading and writing that would reduce the hosting location&#39;s lockdown score.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6JmgQ_aptbAkG41811D8aQ.jpeg" /><figcaption>Data-producing systems generate and persist data assets in hosting locations. By default, data is considered internal, and any unauthorized attempts to consume data from these ports may result in access removal by the central platform. However, consumption of a data product is always permitted because data products adhere to established access control policies and promote secure data exchange within and across teams.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*89dtVWJLfRqOB--xEy7xVw.jpeg" /><figcaption>Data-producing systems only write data to the project’s associated assets. In contrast, data-consuming systems can read from consumable data assets regardless of their owning project as long as the respective owners have approved a formal request for read access. The red line indicates write access that is never permitted.</figcaption></figure><h4>Developer Velocity ( DPV )</h4><p>This metric measures the speed and efficiency at which developers can register data definitions, manage access requests, and release data-producing systems. This metric is essential for tracking the overall productivity of the development team and ensuring that the architecture supports agile development processes, enabling teams to quickly adapt to changing requirements or new challenges.</p><h4>Data Access Request Time (DART)</h4><p>A performance metric that measures the time it takes for humans to access requested data. It is calculated as the time taken to process a data access request, which starts when the request is submitted and ends when the user receives an access decision. A lower DART value indicates faster data access and a more efficient data management process.</p><h3>Appendix</h3><h4>Clean Data Maturity Model</h4><p>The clean data maturity model describes a rating system for data assets and an accompanying policy about how the data can or cannot be used.</p><p><strong>0-star</strong><br>Criteria — Data exists, but there’s no information available to describe it.<br>Usage policy — Not discoverable or consumable.</p><p><strong>1-star (Compliant)</strong><br>Criteria — An owner is identified, and information exists to understand its security and compliance requirements.<br>Usage policy — Not discoverable or consumable.</p><p><strong>2-star (Controlled)</strong><br>Criteria — Same as above, plus documentation exists to describe the meaning of the data. Data is stored in a system compliant with the hosting location standard, which tightly controls access. <br>Usage policy — Discoverable, but only consumable with an exception.</p><p><strong>3-star (Clean)</strong><br>Criteria — Same as above, plus the owner is more active and accountable (i.e., a ‘data steward’); modeling language standards are followed; data quality monitoring ensures operational reliability; lineage describes upstream and downstream dependencies; change control process ensures no breaking changes; data is broadly is ready for broad use across a large audience of consumers.<br> Usage policy — Consumable</p><p><strong>4-star (Clean)</strong><br>Criteria — Same as above, plus the data product follows modeling design standards, making it more easily processed by any number of systems across the company.<br>Usage policy — Consumable</p><p><strong>5-star (Clean)</strong><br>Criteria — Same as above, plus relationships to other data products are precisely defined; the data product is unique and non-conflicting among all data products at the company.<br>Usage policy — Consumable</p><h4>Example Data Model</h4><p>The following diagram describes how a data model is used to model the “Invoicing Workflows” domain.</p><p>The “Invoicing Workflows” bounded context consists of a domain entity schema and two events: an Invoice notification event, which is used to notify on Invoice state changes, and an event-carried state transfer event, which is used to carry the entire state that changed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*41jVRaflh7Zb-GkC8L1WSA.jpeg" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=214268257dd2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Veza sheds some light on a dark corner of your business]]></title>
            <link>https://tcbakes.medium.com/veza-sheds-some-light-on-a-dark-corner-of-your-business-30d6f934bd7?source=rss-28b642ad293a------2</link>
            <guid isPermaLink="false">https://medium.com/p/30d6f934bd7</guid>
            <category><![CDATA[developer-tools]]></category>
            <category><![CDATA[data-mesh]]></category>
            <category><![CDATA[data-security]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-lake]]></category>
            <dc:creator><![CDATA[Tristan Baker]]></dc:creator>
            <pubDate>Tue, 03 May 2022 20:30:12 GMT</pubDate>
            <atom:updated>2022-05-03T20:31:50.216Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t8ecpKe8Myn1n70LHq8gWg.png" /></figure><p>A good part of my career, and all of my <a href="https://medium.com/intuit-engineering/intuits-data-mesh-strategy-778e3edaa017">most recent work</a>, has been focused on the conundrum of improving the productivity of the engineers building data driven systems while also improving the security and compliance of the environments in which they operate.</p><p>That nature of data driven system is such that they require a variety of data points from which to draw correlations and inferences that help guide system behavior. This can take the form of AI/ML (e.g., a regression model that predicts the likelihood of user abandonment from their click behavior and other signals) but can also be simple rules on a rich set of data points (define an audience for a marketing campaign consisting of users from a specific part of the country who we haven’t returned in 30 days and have an abandoned shopping cart with a pair of blue Nike’s in it).</p><p>Though their implementations might be vastly different, both of these systems share a common set of requirements:</p><ul><li>The teams that build them must be able to quickly find and explore available data sets, with no promise that what they explore today will prove to be relevant to the problem or needed over a long period of time.</li><li>The company must ensure the security and compliance of customer data, which includes adhering to the principles of least access privilege and a process of request, justification and approval before new access is granted.</li></ul><p>The former requirement is traditionally solved by data lakes where customer data is available for teams to explore and use as input to their systems. The latter is traditionally solved by a process called user access reviews; it’s a required control of every major security certification in the industry. Periodically, user access is reviewed to ensure that policies remain appropriate and that people and systems that have had access in the past continue to need it in the future.</p><p>What happens, though, when the data lake contains 10’s of thousands of tables and hundreds of teams? The wrong kind of pressure starts to stress different parts of this system, specifically:</p><ul><li>Administrators feel pressure to deny access requests, since approval implies a commitment to manage the access through all subsequent access reviews. More approvals means more to manage.</li><li>Users feel the pressure to make broader requests for data than is strictly needed because every access request is a time consuming process that they would prefer to minimize. In the exploration phases of a team’s work, they might require access to several new data sets a day over a period of several weeks. One big request for an entire section of the data means less time lost to small and frequent requests.</li></ul><p>What if you don’t want to sacrifice productivity to increase security? What if you don’t want to sacrifice security to increase productivity?</p><p>Enter <a href="http://www.veza.com/">Veza</a>.</p><p><a href="https://medium.com/intuit-engineering/intuits-data-mesh-strategy-778e3edaa017">My experience with these use cases</a>, and many others like them, is the reason I joined Tarun and his team as an advisor when it was an idea on a napkin. Solving these kinds of problems are immensely important to my work and I know they are important for any organization looking to deliver best-in-class environments for their workforce while also providing best-in-class security and compliance for the tools and data that workforce requires.</p><p><a href="http://www.veza.com/">Veza</a> answers a deceptively simple question, “who has access to what?”. A less deceptively simple version of the question might be, “who, through what maze of policy statements across a vast array of technologies, has access to what?”.</p><p>When organizations use <a href="http://www.veza.com/">Veza</a> to provide a clear answer to this question and to manage access control processes, productivity and security start to look like much more tractable problems. Policy administrators have insights into who has access to their resources and whether or not access is actively used, thus reviewing access becomes a straight forward exercise of accepting system provided recommendations. Relieved of this time burden, they are more comfortable taking on the responsibility associated with approvals, and respond to requests appropriately and in a timely fashion. Knowing that the request process is fast and smooth, users feel comfortable submitting requests with smaller scope — after all, if they discover tomorrow that they need more, they can always come back and ask without losing any time to an inefficient process.</p><p>In this era of an explosion of data and tools and the ever present push for efficiency, an organization with a productive workforce and secure and compliant systems will have clear advantages over those that that move at 1/10th the speed with 10x the risk. I’m bullish on <a href="http://www.veza.com/">Veza</a> and look I forward to our continuing collaboration.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=30d6f934bd7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Intuit’s Data Mesh Strategy]]></title>
            <link>https://medium.com/intuit-engineering/intuits-data-mesh-strategy-778e3edaa017?source=rss-28b642ad293a------2</link>
            <guid isPermaLink="false">https://medium.com/p/778e3edaa017</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[data-mesh]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Tristan Baker]]></dc:creator>
            <pubDate>Wed, 17 Feb 2021 19:56:39 GMT</pubDate>
            <atom:updated>2021-02-22T21:20:39.161Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xjtx8AwqdMQjc78KOeOdTQ.png" /></figure><p>Intuit’s mission is to ‘<a href="https://www.intuit.com/company/"><em>Power Prosperity Around the World</em></a>’. And Intuit’s strategy for delivering on that mission is to be an ‘<a href="https://www.intuit.com/strategy/"><em>AI-Driven Expert Platform</em></a>’. I work on Intuit’s Data Platform team and we play a critical role in delivering the solutions required to make that happen. My team’s internal customers include Intuit teams that build the products and offerings responsible for powering prosperity for Intuit’s consumer, small business and self-employed customers.</p><p>My colleague, Mammad Zadeh, Intuit Vice President, Data Platform, recently published <a href="https://medium.com/intuit-engineering/the-intuit-data-journey-d50e644ed279">an article </a>describing what we’ve been up to over the past couple of years. We’ve migrated from an on-premise architecture of centrally-managed analytics data sets and data infrastructure tools to a fully cloud-native and much more expansive set of data and tools that enable Intuit teams to build a wide range of streaming, machine learning, and analytic processing workloads. Intuit’s customers are reaping the benefits of this new architecture and I’m proud of what we’ve accomplished.</p><p>However, as far as we’ve come, the work is unfinished and opportunities abound.</p><p>We surveyed the landscape of problems within our solution and researched what solutions existed for companies with a similar size and scope as ours. Zhamak Dehghani’s <a href="https://martinfowler.com/articles/data-monolith-to-mesh.html">description of a “Data Mesh”</a>, which favors pushing data ownership and responsibility to local domain teams at the edges of the business, immediately caught our interest. The opening “Architectural Failure Modes” section is a frighteningly accurate summary of the problems that plague our current state. Zhamak’s advocacy for Domain Driven Design, with an emphasis on ownership and accountability of singularly focused, problem domain-oriented teams, as the backbone of a strategy to address these issues, resonates loudly. We’ve seen this approach work well at the “front of the house” with our microservices architecture and development of Intuit’s core transactional services. So we were confident that applying that strategy to our “back of the house” data systems could be equally successful.</p><p>The rest of this article will take you through a full articulation of our vision, inherent challenges, and strategy for building better data-driven systems at Intuit. You can think of it as data mesh, interpreted through an Intuit lens. If you’ve spent any time in a typical data lake and caught yourself wondering, “How the heck can we decrease pandemonium and increase productivity to get back to the business of making customers happy?!”, read on.</p><h3>Vision</h3><p>Intuit needs data-driven systems to enable smarter product experiences, and to enable more Intuit teams to more easily create them. This includes a variety of data workers:</p><ul><li>A service engineer building a service that publishes an event when an invoice is paid.</li><li>A UX developer building a landing page that publishes an event when a user visits.</li><li>A business analyst building a report that shows QuickBooks hour-by-hour new subscriptions counts.</li><li>A data scientist building a model that predicts fraud before approving payment.</li><li>A data engineer or analyst building a pipeline to identify new users and returning users by synthesizing data from our customer identity and Intuit back-office systems.</li></ul><h3>Challenges</h3><p>To understand how to drive simpler, rapid development of these systems, we first had to discover what was inhibiting the data workers trying to build such systems. We surveyed 245 current users of our data systems to find out what kinds of questions they were struggling to answer in the course of their daily activities. Across the roles of business analysts, data engineers, data scientist, and machine learning engineers, these questions bubbled to the top:</p><p><strong>Data Discoverability</strong></p><ul><li>Where can I find data about a particular thing (customer, company, etc)?</li><li>Where can I find the data sourced from a particular product or service?</li></ul><p><strong>Data Understandability</strong></p><ul><li>Who can approve my access so that I can see samples of the data?</li><li>What is the schema of the data?</li><li>What is the business meaning and context of the data?</li><li>Is this data related to other concepts? Is it joinable to other data? What is the meaning of the relationship?</li></ul><p><strong>Data Trust</strong></p><ul><li>What system produces this data and at what latency?</li><li>What other systems use this data?</li><li>What is the quality of this data?</li><li>Which team supports this data if it breaks?</li></ul><p><strong>Data Consumption</strong></p><ul><li>How is this table/topic partitioned?</li><li>Who can approve my production system to access it?</li><li>Will I be alerted if the schema changes?</li></ul><p><strong>Data Publication</strong></p><ul><li>How do I describe my data so that others understand what it means and how to use it?</li><li>Where do I host my data so that other systems can access it?</li><li>Data systems are complicated; how can I build and operate one?</li><li>What are my operational responsibilities once my process/data is in production?</li><li>How do I meet my compliance requirements for processing, storing, and publishing data?</li><li>Am I duplicating processing/data that already exists?</li></ul><h3>Strategy</h3><p>With new insight into our users’ experience, we defined a strategy that will enable a greater number of data workers to easily create and own high-quality data-driven systems; resulting in smarter product offerings to Intuit customers. The strategy empowers data workers to design, develop, fully describe and actively support their own data driven systems, allowing the next person to easily discover, understand, trust, and consume data. This powers the network effect of data production and consumption, which increases the productivity of all data workers, resulting in the large variety of high quality data driven systems Intuit requires.</p><p>Discovery, understanding, trust, consumption, and publishing of data systems is supported in three specific ways:</p><ol><li>A systematic approach to organizing the people, code and data which in turn map a solution to a business problem and its owners.</li><li>Ownership that ensures all teams understand and are accountable for a set of defined responsibilities in building and managing their solutions; including adherence to a set of defined best practices to produce only high quality data.</li><li>A rich suite of products<strong> </strong>that enable teams to more easily author, deploy, and support their own solutions, and to do so according to best practices.</li></ol><h3>Organizing People, Code and Data</h3><p>To more precisely describe the set of related code and data that collectively solve a business problem, we introduce a systematic method of organizing them: <strong>Data Product</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/1*wHgPzcRpeN9MLUmaeA9RwA.png" /></figure><p>A Data Product is a set of internal processes and data that produce a set of externally consumable data, all aligned around the same problem in the business domain. It is created and managed by a team collectively capable of understanding both the business problem and the implementation details of the internal processes and data that solve that problem.</p><p>This is in contrast to the current state in which teams are aligned around technology, forcing the same logical business problem to be arbitrarily divided among several different teams. This creates inefficiencies in communication and coordination that cause many of the issues teams experience today. A team is likely composed of one or more Data Workers versed in the technologies used in the solution, a product manager versed in the business problem, and a leader responsible for the timely and efficient execution of the project that delivers the solution to the business problem.</p><p>It is critically important to develop a system for organizing these assets so that we can layer the concept of ownership on top. Without the ability to pinpoint exactly which humans are responsible for what code and data, there is no ability to hold anyone accountable for making sure that it is of value to the company, and thus no reasonable path towards making things better.</p><p>To zoom in on just Intuit’s data lake environment for a moment, we haven’t had a full understanding of the purpose of an overwhelming majority of hundreds of thousands of fields across the thousands of tables it contains. This cruft has meant that nearly half of data workers’ time in the data lake has been wasted due to an inability to find data or an owner of that data to explain what it means.</p><h3>Ownership</h3><p>Building on this system of organization, ownership brings together three important concepts:</p><ol><li>An owning team,</li><li>A scope over which that ownership applies, and</li><li>A set of responsibilities to fulfill within that scope.</li></ol><p>Everywhere there is a Data Product, we require a clear statement to the following effect: “Team Awesome (i.e., owning team) is responsible for the authorship, rich description, governance, quality and operational health (i.e., responsibilities) of the Fantastic Data Product, which is composed of these processes and data (i.e., scope)”.</p><p>Team and scope are already covered in the definition of Data Product earlier in this article. Responsibilities are defined below:</p><p><strong>Authorship — </strong>The owning team designs the solution, authors the code, and arranges the systems that implement it</p><p><strong>Description — </strong>The owning team describes the system in enough detail such that other teams are able to discover, understand and use the Data API.</p><p><strong>Governance — </strong>The owning team is responsible for the solution adhering to all applicable compliance and security requirements.</p><p><strong>Quality — </strong>The owning team is responsible for the quality of the solution and delivering its data API with the promised correctness and latency.</p><p><strong>Operational Health — </strong>The owning team is responsible for the Data Product’s uptime, performance and cost.</p><p>Implicit in the listing of the above responsibilities is that we require teams to be excellent in fulfilling them. A bad Data Product (e.g., one that always breaks, isn’t well described and for which no one can find the owner), does more harm than good. Therefore, we require <em>well</em> designed, <em>richly</em> described, <em>strongly</em> governed, <em>high </em>quality and operationally <em>reliable</em> Data Products.</p><p>To help teams understand these expectations, we’ve established a framework to enumerate specific criteria that must be met in order to be considered high enough value to the business to be published. The framework describes two levels, and the various criteria required to meet those levels:</p><p><strong>BASIC</strong> describes the minimum set of criteria that must be met to comply with our system of organization and ownership responsibilities.</p><p><strong>BEST</strong> goes beyond BASIC to introduce the notion of rationalized data that ensures a Data Product is producing content that is of net-new value to the ecosystem and does not duplicate or contradict important concepts that have already been established and relied upon across the company. This is getting into the world of master data management, which fundamentally pits the goals of a system of federated ownership responsibility against the goals of centralized data definition (how we plan to do this elegantly and at scale at Intuit will be the subject of another post).</p><p>A BASIC Data Product has —</p><ul><li>Direct upstream dependencies identified</li><li>Compliance regulations identified</li><li>Change control and access control processes in place</li><li>Data API decoupled from internal implementation</li><li>Data API schema defined and described</li><li>Data API monitored against stated SLA</li><li>Owning team is assigned, accountable for and measured against their responsibilities</li></ul><p>A BEST Data Product has —</p><ul><li>All of the BASIC stuff, plus</li><li>Data rationalized against and related to all other Data Products</li></ul><h3>A Diverse set of Composable Capabilities Enabling Best Practices</h3><p>A diverse set of composable capabilities gives teams the tools to design, author, deploy and operate Data Products for themselves. The diversity ensures that the equally diverse skills and perspectives of data workers are accommodated and that everyone can be productive in consuming and producing Data Products. The built-in adherence to best practices ensures Data Products are constructed and data is produced in a way that advances, rather than erodes, Intuit’s ability to leverage all of the data at our collective disposal. Each offering in the set enforces that its users produce Data Products that meet the BASIC (or better) requirements for publication.</p><p>Each of these capabilities is integrated with central platform services and standards that ensure that security, authentication, authorization, access control policy management, change control procedures, schema registration and documentation are implemented in a self serve manner so that individual teams are able to efficiently fulfill their ownership responsibilities. Additionally, quality, cost, and performance metrics are measured and reported in the same way, so that the individual teams — as well as the organization at large — have insight into how well (or not) a Data Product is operating.</p><p>The listing and brief description of all product offerings that will be available to Data Workers is as follows. Since many of these products are big and complex and well established in their own rights, I’ve provided only brief summaries. Where possible, I’ve linked out to other Intuit published articles with more information.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/108/0*efek3efrHHSznfEF" /></figure><h3>Capabilities Used by Teams to Implement Their Data Product</h3><p><strong>Instrumentation Services — </strong>A solution for instrumenting user facing experiences with behavior tracking events in compliance with Intuit’s event standards so that a UX Team can track user interactions and so that other Analyst and ML Teams can consume these events from well defined locations for further analysis and transformation.</p><p><strong>DomainEvent Services — </strong><a href="https://microservices.io/patterns/data/domain-event.html">DomainEvents</a> describe the changing state of the sending system. The DomainEvent Services are leveraged by Service Engineers in their transactional microservices to provide DomainEvents asynchronously to downstream Data Products. This is intended to be used in place of where raw CDC data is used today.</p><p><strong>Universal Ingestion Platform (UIP) — </strong>Leverages the CDC streams and/or jdbc connections supported widely by transactional database systems (Oracle, MySql, etc) to pull data into the Data Lake so that it can be accessed by big data processing technologies. UIP is leveraged by Service Engineers to make raw data available to the Data Lake only when DomainEvents are not feasible. A platform constructed from Oracle GoldenGate, <a href="https://debezium.io/">Debezium</a> and <a href="https://delta.io/">DeltaLake</a>.</p><p><strong>Intuit Persistence Services (IPS) — </strong>Enables Service Teams to deploy and manage databases in support of stateful business logic. The deployments encapsulate all Intuit best practices and standards to enable highly available and secure systems.</p><p><strong>Stream Processing Platform (SPP) — </strong>Enables Service Teams or Data Engineering Teams to create systems that consume, process and produce streams of real time asynchronous events. Detailed articles <a href="https://www.ververica.com/blog/how-intuit-built-a-self-serve-stream-processing-platform-with-flink">here</a> and <a href="https://aws.amazon.com/blogs/big-data/real-time-stream-processing-using-apache-spark-streaming-and-apache-kafka-on-aws/">here</a>.</p><p><strong>Pipelines</strong> — Enables Data Engineering<strong> </strong>and Analysts teams to create spark and hive batch processing jobs and schedule them for execution at periodic intervals or upon the availability of results from other upstream Data Products. Detailed in an article <a href="https://quickbooks-engineering.intuit.com/demystifying-complex-data-pipeline-lineage-with-superglue-d5b4014b1482">here</a> (the article is a bit dated now and some things have changed, but it should give you a general idea)</p><p><strong>Schema Manager — </strong>Enables UX, Service, Data Engineer,<strong> </strong>and<strong> </strong>Analytics<strong> </strong>Teams to create the schemas that their DataProducts produce. The Schema Manager is able to enforce schema standards and recommend best practices as well as detect and prevent duplicate and conflicting definitions among data from different teams.</p><p><strong>Entity Builder and Entity Resolver — </strong>Enables Analysts<strong> </strong>and<strong> </strong>Data Stewards to define logic that curates and masters entities so that cross cutting concepts (like Customer, Product, etc) have consistent definitions available for use across the company. <em>We’ve started defining and building this; so it will likely be the subject of further blog posts.</em></p><p><strong>ML Platform Feature Engineering — </strong>Enables Data Scientists<strong> </strong>and<strong> </strong>ML Engineers to define features to be used in model training. More information about the feature, training and execution capabilities of MLPlatform can be found <a href="https://www.usenix.org/conference/opml20/presentation/wenzel">here</a>.</p><p><strong>ML Platform Training Service — </strong>Enables Data Scientists<strong> </strong>and<strong> </strong>ML Engineers to train models.</p><p><strong>ML Platform Model Execution Service — </strong>Enables Data Scientists<strong> </strong>and<strong> </strong>ML Engineers to deploy trained models to be executed in real time or batch.</p><p><strong>Data Quality Services — </strong>A capability available wherever a Data Product produces data. It assists all users by providing context specific quality checks on their data. It supports quality judgement during the build and testing phases as well as in-production quality observation and monitoring to ensure data quality targets and <a href="https://www.montecarlodata.com/how-to-make-your-data-pipelines-more-reliable-with-slas/">data SLAs</a> are met throughout the data lifecycle. We’re using <a href="https://www.montecarlodata.com/">Monte Carlo</a> for this, and will be supplementing it with some of our own innovations, too.</p><p><strong>Data Discovery and Exploration (DDE) — </strong>A tool that allows a user to browse available data sets and fire up a query browser or a notebook to start accessing and manipulating data sets. Assists all users in the R&amp;D work that usually precedes any effort to build and release a Data Product into production.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/108/0*xtvdpytAr3dDvWoD" /></figure><h3>Capabilities Used by Teams to Host their Data Product’s DataAPI</h3><p><strong>Data Lake <em>— </em></strong>The Data Lake hosts large scale data for analytics, machine learning and other scan-and-process-in-parallel-a-lot-of-data-as-fast-as-possible workloads. Leverages HMS, parquet and S3 blob storage. It establishes the authorization model and workflows that enables consumers to request and producers to approve access to data sets for both exploration and production access purposes. More detail <a href="https://aws.amazon.com/blogs/big-data/provisioning-the-intuit-data-lake-with-amazon-emr-amazon-sagemaker-and-aws-service-catalog/">here</a> and <a href="https://medium.com/intuit-engineering/is-your-data-lake-more-like-a-used-book-store-or-a-public-library-f444ef6a1798">here</a>.</p><p><strong>EventBus — </strong>Location for all streaming event data. Directly leveraged by Service engineers and Data Engineers<strong> </strong>implementing asynchronous eventing solutions or creating streaming processing applications. Indirectly leveraged by Analysts, Data Scientists<strong> </strong>and<strong> </strong>ML Engineers when they leverage the real time processing capabilities of ML Platform Feature Management and the Entity Build and Resolver. More detail <a href="https://aws.amazon.com/blogs/big-data/best-practices-for-running-apache-kafka-on-aws/">here</a>.</p><p><strong>Data Mart — </strong>Location for data used to drive business reporting. Primarily used by Data Engineers<strong> </strong>and<strong> </strong>Analysts. Data Marts are purpose built schemas optimized for particular areas of the business (e.g., “tax expert case management and efficiency analytics”). They provide both the data sets and the query engines required to drive Dashboards.</p><p><strong>FeatureStore — </strong>Location for all features required to drive ML training and prediction use cases. It allows the same feature to be hosted in an online and offline location for real time and batch access while maintaining equivalence between the two — this equivalence guarantees stability between the data the model was trained on and the data the trained model uses to generate predictions. Leveraged by Data Scientists<strong> </strong>and<strong> </strong>ML Engineers.</p><p><strong>API Gateway — </strong>A location for all services to host their REST APIs for exposing service behavior (ie, “POST /TaxService/FileTaxReturn/{id}”) and data (ie, “GET /TaxService/TaxReturn/{id}”). The gateway handles authn/authz in a standard way across the entire enterprise, thereby ensuring consistently applied security across all services in the organization.</p><h3>For Example</h3><p>The collection of all meta-data that describes all Data Products is used to create a universal metadata registry (ours is built on <a href="https://atlas.apache.org/#/">Apache Atlas</a>) of all the teams, systems, data, entities, and the dependency relationships between them. It is the foundation for answering all questions about Data Product ownership, scope, responsibility, technical dependency and data meaning.</p><p>A collection of five imagined Data Products, their technical dependencies (grey lines) and their rationalized data definitions and relationships (blue lines) are drawn below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9cPKGJ5A_RUjhWnNrYaB4w.png" /></figure><p>Note that questions that we led with at the beginning of this article and drive our strategy are answerable:</p><ul><li>Where is Customer data available for consumption? What is the schema of the data?</li><li>Which other Data Product(s) did Customer Analytics Data Product get its data from?</li><li>Which team is responsible for Product Entitlements Service Data Product? Who is on that team and how can I get in touch with them?</li><li>How are Intuit Customers related to Intuit Products?</li><li>Which team can approve my request to access the Customer Analytics Data Product’s Data Lake table?</li></ul><h3>Up Next</h3><p>Intuit is at the very beginning of a data mesh journey. Given that it’s a relatively recent concept, no one has developed anything resembling a data mesh standard yet. While many of the capabilities described throughout this article exist in one form or another, the systems required to organize them are just getting underway.</p><p>This article is a lot about the why and the what, I expect my next post to get into more detail about the how. My hope is that this article, and any that follow, might provide other like-minded state-of-data malcontents with a running start in their own endeavors. As this concept gets more mature and gains momentum, I’m interested in hearing your ideas for how we might standardize on some of these approaches. A collective aligned effort might speed things up a bit for all of us. If you are one of those people, please <a href="https://www.linkedin.com/in/tristanbaker/">get in touch</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=778e3edaa017" width="1" height="1" alt=""><hr><p><a href="https://medium.com/intuit-engineering/intuits-data-mesh-strategy-778e3edaa017">Intuit’s Data Mesh Strategy</a> was originally published in <a href="https://medium.com/intuit-engineering">Intuit Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>