<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Arne Roßmann on Medium]]></title>
        <description><![CDATA[Stories by Arne Roßmann on Medium]]></description>
        <link>https://medium.com/@arne.rossmann?source=rss-a4692a2d91f2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*3BAmyLA4P1Ktc9MLKAQP0w.jpeg</url>
            <title>Stories by Arne Roßmann on Medium</title>
            <link>https://medium.com/@arne.rossmann?source=rss-a4692a2d91f2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 05:10:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@arne.rossmann/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Unlocking innovation with platform engineering]]></title>
            <link>https://medium.com/@arne.rossmann/unlocking-innovation-with-platform-engineering-9e7321fffc65?source=rss-a4692a2d91f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/9e7321fffc65</guid>
            <category><![CDATA[data-mesh]]></category>
            <category><![CDATA[innersource]]></category>
            <category><![CDATA[acceleration]]></category>
            <category><![CDATA[idp]]></category>
            <category><![CDATA[innovation]]></category>
            <dc:creator><![CDATA[Arne Roßmann]]></dc:creator>
            <pubDate>Wed, 07 May 2025 00:00:32 GMT</pubDate>
            <atom:updated>2025-05-07T09:35:56.523Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Golden Path to data mesh and AI excellence</h3><p>Data, AI, and innovation are essential to staying competitive. Organizations need to accelerate development while maintaining quality to remain ahead. Platform engineering, traditionally tied to cloud-native environments and microservices, is now crucial for enabling data mesh architectures, transforming how data products and AI applications are built.</p><p>Data mesh allows teams to efficiently develop AI models, AI agents, and analytical tools, such as dashboards. Platform engineering provides the infrastructure and standardized processes — known as the Golden Path — that streamline development, reduce time to market, and improve product quality</p><h3>Internal developer platforms and data mesh</h3><p>Internal developer platforms (IDPs), such as <a href="https://backstage.io/">Backstage</a>, <a href="http://getport.io/">Port</a>, and <a href="https://humanitec.com/">Humanitec</a>, are vital enablers of platform engineering and offer self-service capabilities for managing infrastructure, services, and data pipelines. They abstract infrastructure complexity, allowing teams to focus on developing high-quality data and AI products. In a data mesh environment, IDPs provide essential tools and templates that help teams build and manage data products efficiently. They accelerate the creation of data-driven applications by providing standardized workflows and automation tools, enabling teams to follow the Golden Path for faster and more reliable development.</p><h3>Automated data and AI governance</h3><p>One of the most significant advantages of platform engineering is automated data and AI governance. Data products are automatically published into data catalogs, while AI models are added to model catalogs with relevant metadata. This ensures consistent governance across the organization and simplifies compliance. When semantic integration is established between data sources, data products, AI/ML models, and analytical use cases, the benefits are even greater. By linking these elements, companies can better track the value of their data and AI assets, improving transparency and decision-making.</p><h3>Bridging the gap: From data to actionable insights</h3><p>But the benefits of platform engineering extend beyond the realm of data management. As organizations strive to derive meaningful insights from their data, the need for robust, scalable, and user-friendly analytical tools has never been greater. Platform engineering provides a framework for the development and deployment of custom data and AI-powered applications, enabling organizations to transform raw data into actionable insights. By streamlining the process of building and deploying these analytical tools, platform engineering accelerates the time-to-market and ensures a consistent, high-quality user experience.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/0*YVbSEfleNfNw3Geo.png" /></figure><h3>Developing a predictive maintenance data product</h3><p>Consider a manufacturing company aiming to implement predictive maintenance to reduce factory downtime. The goal is to develop a data product that analyzes machine performance data, detects anomalies, and predicts when maintenance is needed.</p><ol><li><strong>Data ownership and product development:</strong> The manufacturing team owns the machine performance data and uses the data mesh model to turn this into a product for preventing equipment failures.</li><li><strong>Internal developer platform:</strong> With an IDP, the team follows a standardized pipeline for data ingestion, cleaning, and processing. The platform handles infrastructure, allowing the team to focus on developing the data product.</li><li><strong>Model development:</strong> The team builds a machine-learning model to predict equipment failures. The IDP supports training and deployment, allowing data scientists to refine the model’s accuracy.</li><li><strong>Deployment and governance:</strong> The model is deployed using the Golden Path, and the data product is automatically cataloged with metadata for transparency. The AI model is similarly added to the model catalog, ensuring proper governance and future discoverability.</li></ol><p>In this example, platform engineering streamlines the entire process while ensuring that data and AI governance are automated and integrated, improving both development speed and quality.</p><h3>The Golden Path: Empowering developers</h3><p>The Golden Path in platform engineering is used to build and deploy applications efficiently. By removing infrastructure complexity, the Golden Path allows developers to focus on innovation and product development. In the predictive maintenance example, the Golden Path enabled the team to quickly build and deploy its data product. The IDP handled data ingestion, model training, and deployment, allowing the team to concentrate on refining the AI model’s accuracy. By ensuring consistency across teams, the Golden Path helps organizations build AI models, dashboards, and data products faster and with higher quality. It minimizes errors, speeds development, and ensures that best practices are followed throughout the product lifecycle.</p><h3>AI and data products as beneficiaries</h3><p>AI applications and data products benefit immensely from platform engineering and the Golden Path. AI models, such as those for predictive maintenance, require large amounts of data, computation, and standardized processes. Platform engineering provides the necessary infrastructure and automation to manage these resources, freeing data scientists to focus on model development. Automated data and AI governance further strengthens this approach by ensuring that data products and AI models are properly cataloged, discoverable, and compliant. This improves the overall value of data and AI assets within the organization, making them easier to reuse and scale.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/869/0*7PVK91wGJmFUyMlh.png" /></figure><h3>Faster time to market and increased quality</h3><p>The combination of platform engineering, data mesh, and IDPs leads to faster time to market for new data products and AI applications. Standardized processes and self-service environments allow teams to move quickly from concept to deployment. Automated governance ensures that data and AI products meet organizational standards for quality and compliance. Moreover, platform engineering improves product quality by enforcing best practices through the Golden Path. This leads to more robust, scalable data products and AI models that align with business objectives and deliver better outcomes. Platform engineering as a catalyst for innovation Platform engineering, combined with data mesh and IDPs, is a powerful innovation driver. By providing standardized, self-service environments, platform engineering accelerates the development of high-quality data products and AI applications. Automated governance enhances the value of these assets, ensuring transparency and compliance. Embrace platform engineering today to accelerate innovation, deliver better products, and gain a competitive advantage in the data-driven world.</p><p>#InternalDeveloperPortal #DataMesh #PlatformEngineering #AccelerateInnovation #AutomatedGovernance #EmpowerTeams</p><p><em>Originally published at </em><a href="https://dataworldliving.com/post/2025-05-07_unlocking_innovation_with_platform_engineering/"><em>https://dataworldliving.com</em></a><em> on May 7, 2025.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9e7321fffc65" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Mesh — not such a new concept after all?]]></title>
            <link>https://medium.com/@arne.rossmann/data-mesh-not-such-a-new-concept-after-all-5c0f53c77e57?source=rss-a4692a2d91f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c0f53c77e57</guid>
            <dc:creator><![CDATA[Arne Roßmann]]></dc:creator>
            <pubDate>Wed, 13 Apr 2022 09:50:10 GMT</pubDate>
            <atom:updated>2025-05-07T09:35:20.911Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Mesh — not such a new concept after all?</h3><p>I don’t think that I have to tell much about the recent developments on Data Mesh (see <a href="https://martinfowler.com/articles/data-monolith-to-mesh.html">this</a> and <a href="https://martinfowler.com/articles/data-mesh-principles.html">that</a>), the world doesn’t need another “Data Mesh introduction” article just to tell these things again.</p><p>But when we dig deeper into that topic and look on the Data Product there might be some similarities ringing a bell. We’ll come to this later.</p><p>But how is a Data Product defined (by <a href="https://martinfowler.com/articles/data-mesh-principles.html">Zhamak</a>)?</p><blockquote>Data product is the node on the mesh that encapsulates three structural components required for its function, providing access to domain’s analytical data as a product.<em> </em>: it includes (a) code for data pipelines responsible for consuming, transforming and serving upstream data — data received from domain’s operational system or an upstream data product; (b) code for APIs that provide access to data, semantic and syntax schema, observability metrics and other metadata; © code for enforcing traits such as access control policies, compliance, provenance, etc.<em> </em><strong>Data and MetadataInfrastructure</strong>: The infrastructure component enables building, deploying and running the data product’s code, as well as storage and access to big data and metadata.<em> </em>: well that’s what we are all here for, the underlying analytical and historical data in a polyglot form. Depending on the nature of the domain data and its consumption models, data can be served as events, batch files, relational tables, graphs, etc., while maintaining the same semantic. For data to be usable there is an associated set of metadata including data computational documentation, semantic and syntax declaration, quality metrics, etc; metadata that is intrinsic to the data e.g. its semantic definition, and metadeta that communicates the traits used by computational governance to implement the expected behavior e.g. access control policies.</blockquote><p>Additionally, we have the three types of ports.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*q-gLsx_qLqEtvtSb.png" /></figure><p>This looks really interesting and quite new, if you remember how the usual work in the data &amp; analytics domain was done previously. But times have changed, principles like Infrastructure as Code, serverles, DevOps &amp; DataOps, SRE etc. have changed the game towards more code focussed and software engineering thinking.</p><p>This morning I stumpled across a nice article on <a href="https://thecillu.medium.com/cloud-serverless-lock-in-doesnt-exist-if-your-team-knows-hexagoxal-architecture-4f64e93463aa">cloud serverless lock-in</a>. One approach here is the hexagonal architecture, in which Ports and Adapters are present.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/0*gmjR_dD6aJMwGzOV.png" /></figure><p>The goal of the hexagonal architecture is to encapsulate the logic within the core from the outer world. With this one, you can leverage some benefits:</p><ul><li>easier testing in isolation</li><li>development can be done independently from external services</li><li>ports and adapters are replaceable / extandable</li></ul><p>Now let’s try to map it to the Data Product from Data Mesh. The Data Product is connected via the <strong>Input Port(s)</strong> to the data sources, this can be a database (e.g. via jdbc), a Message Queue / Message Broker like Kafka, Web endpoints etc.</p><p>Then within the Data Product, the “standard” procedures of Load, Process and Provide are happening. They load the data from the source via the input port (or receive events via subscription e.g. to a Kafka topic), do some data transformation steps ( = process) and provide the outcome via the Output Port(s). This can be again a database or a S3 endpoint or again a topic in Kafka. The advantage of that encapsulation is that the Data Product can be developed without interfering other data pipelines from other Data Products, I can start with a simple pass-through and add further transformations steps as soon as I need them. And it can be easily extended on new output ports (e.g. starting with a S3 endpoint, later a REST endpoint) Sounds quite similar to the hexagonal architecture? Yes, and reflects the movement to more Microservices in the non-analytical world. And on the Data &amp; Analytics domain we see similar patterns, Data Mesh opens the door towards Domain Driven Development, Serverless, Data Products (within Data Mesh), Event-Driven are pushing towards Microservices / more granular processes.</p><p>So, back to the initial question: Is Data Mesh not so new? Well, from my view it is and it is not.</p><ul><li>It is not: Data Mesh uses approaches and concepts from the Software Engineering domain like domain driven design, hexagonal architecture, Microservices etc.</li><li>It is: these concepts were not the standard way of working in Data &amp; Analytics Space in the last years and therefore a quite new approach. But it is good and helps to bring the data-driven thinking and data-diven practise forward.</li></ul><p><em>Originally published at </em><a href="https://medium.com/@arne.rossmann/data-mesh-not-such-a-new-concept-after-all-d0d57f1ec6cc"><em>https://medium.com</em></a><em> on April 13, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c0f53c77e57" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Shaking the Kinder egg — or: metadata of Data Products?]]></title>
            <link>https://medium.com/@arne.rossmann/shaking-the-kinder-egg-or-metadata-of-data-products-2909ee60beea?source=rss-a4692a2d91f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/2909ee60beea</guid>
            <category><![CDATA[data-mesh]]></category>
            <category><![CDATA[knowledge-graph]]></category>
            <dc:creator><![CDATA[Arne Roßmann]]></dc:creator>
            <pubDate>Mon, 14 Feb 2022 00:00:57 GMT</pubDate>
            <atom:updated>2025-05-07T09:37:07.584Z</atom:updated>
            <content:encoded><![CDATA[<h3>Shaking the Kinder egg — or: metadata of Data Products?</h3><p>Who hasn’t done it? shaking the Kinder eggs to “guess” what’s inside and raise the chance of getting one of the figures. We even put them on the vegetable scale to increase the chance (many many years back the figures had higher weight than the assemble stuff). But in the end, it was all guessing.</p><p>When we deal with data, we don’t want to guess. We want to get the clearest view on “what’s inside” as possible. And with the <em>Data Mesh</em> approach, in with we create the <em>Data Products</em> to reuse existing data, this is more important than ever. But how do we do that?</p><p>First, let’s have a look on ‘What is a data product’ again. in <a href="https://medium.com/@arne.rossmann/data-mesh-not-such-a-new-concept-after-all-d0d57f1ec6cc">the last post</a> I sketched the Data Product already. It has these main elements in it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/221/0*Ccu7Eao-JQqQF8ud.png" /></figure><p>Ideally, everything what is done is <em>code</em> based, including setting up the infrastructure (infrastructure as code), data transformation steps etc.</p><p>The required infrastructure for that data product is connected to it. Depending on the type and usage of the product, it can vary. Ususally there’re two main types:</p><p>The input and output ports are the connectors to the data sources (input) and to the data consumers (output). Depending on the type of data product and the data sources and uses cases, these can be regular API endpoints like jdbc, s3, kafka, mqtt, https, &amp;mldr; .</p><p>But the more important (at least for this post) port is the Control Port. Why? Well, with this specific endpoint we get information from and control the defined data product. So let’s take an example:</p><p><em>As a business user In order to reuse existing data, I want to get the following information from a Data Product within my organization:</em></p><ul><li>In what domain is that data product?</li><li>Who has created it?</li><li>What information / fields are available?</li><li>when was it updated last time?</li><li>what is the data quality of it?</li><li>what data sources is this Data Product connected to?</li><li>what connection options I have to work with the Data Product?</li></ul><p>And here the Control Port comes in.</p><p>This port provides a consistent view on the Data Product. Endpoints are e.g.:</p><ul><li>metadata {name, owner, source systems}</li><li>lineage information</li><li>address of ports {input, output}</li><li>schema {input, output}. link to company ontology</li><li>audit</li><li>metrics, i.e.</li><li>last updated timestamp</li><li>loading frequency</li><li>loading time (how long did it take to load data)</li><li>data volume</li><li>data quality metrics {e.g. # of unmapped records}</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9sHlZeAsHPVDL_aP.png" /></figure><p>As you can see in the image above, the Control Port collects these information from a wide variety of sources like the Data Catalog, Data Quality Tool, the Data Lineage Tool or the DataOps tool of the Platform Foundation. It acts like an endpoint federation, as i.e. usually the Data Catalog tool of choise provides:</p><ol><li>a nice integration to publish the data product into the catalog</li><li>an API to get information out of the catalog</li></ol><p>Same applies (or should be applied) to the other components of the Data Trust layer.</p><p>So let’s take an example on the metrics, as these are quite common to be asked by the business in order evaluate further usage of the existing Data Product. When the business has easy access to the metrics of a Data Product, it can easily evaluate how much value this Data Product brings it for their use case(s). The exemplary metrics mentioned above give a good overview on typical demands and have their basis in the <a href="https://sre.google/workbook/data-processing/">SRE Workbook</a>.</p><p>But how does the user get these information? Do they have to query the API? Well, these information have to be available as API endpoints to allow automatic processing. But for Business Users there has to be a proper frontend incorporting them and allowing searching, visual exploration and interaction with the underlying information. This can be a custom frontend communicating with underlying API or off the shelf tools. (due to the broad landscape of these tools on the market, a separate post is required to capture that topic)</p><p>So, back to the initial question: Yes, the majority of us has found ways to increase the chance to get a rare mini figure as the surprise in the Kinder eggs. And we were creative with it. But nowadays, when we talk about finding the right Data Product(s) for the use cases, it doesn’t need shaking or a scale. In a proper Data Mesh based architecture, all the relevant information are available at the control port and consumable via an API <strong>AND</strong> a proper frontend.</p><p><em>Originally published at </em><a href="https://arnerossmann.github.io/post/2022-02-09_metadata-dataproduct/"><em>https://arnerossmann.github.io</em></a><em> on February 14, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2909ee60beea" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Shaking the Kinder egg — or: metadata of Data Products?]]></title>
            <link>https://medium.com/@arne.rossmann/shaking-the-kinder-egg-or-metadata-of-data-products-211c42925910?source=rss-a4692a2d91f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/211c42925910</guid>
            <category><![CDATA[kinder-eggs]]></category>
            <category><![CDATA[data-mesh]]></category>
            <dc:creator><![CDATA[Arne Roßmann]]></dc:creator>
            <pubDate>Mon, 14 Feb 2022 00:00:00 GMT</pubDate>
            <atom:updated>2022-04-14T11:02:55.339Z</atom:updated>
            <content:encoded><![CDATA[<h3>Shaking the Kinder egg — or: metadata of Data Products?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h1dT-XpyqPn1TdfPU5g-Bg.jpeg" /><figcaption>Kinder Egg (<a href="https://www.pexels.com/photo/broken-chocolate-egg-4108597/">Photo</a> by <a href="https://www.pexels.com/@polina-zimmerman/">Polina Zimmerman</a>)</figcaption></figure><p>Who hasn’t done it? shaking the Kinder eggs to “guess” what’s inside and raise the chance of getting one of the figures. We even put them on the vegetable scale to increase the chance (many many years back the figures had higher weight than the assemble stuff). But in the end, it was all guessing.</p><p>When we deal with data, we don’t want to guess. We want to get the clearest view on “what’s inside” as possible. And with the <em>Data Mesh</em> approach, in with we create the <em>Data Products</em> to reuse existing data, this is more important than ever. But how do we do that?</p><p>First, let’s have a look on ‘What is a data product’ again. in <a href="https://medium.com/@arne.rossmann/data-mesh-not-such-a-new-concept-after-all-d0d57f1ec6cc">the last post</a> I sketched the Data Product already. It has these main elements in it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/221/0*0abVfGH5ZiMSPg1t.png" /></figure><p>Ideally, everything what is done is <em>code</em> based, including setting up the infrastructure (infrastructure as code), data transformation steps etc.</p><p>The required infrastructure for that data product is connected to it. Depending on the type and usage of the product, it can vary. Ususally there’re two main types:</p><p>The input and output ports are the connectors to the data sources (input) and to the data consumers (output). Depending on the type of data product and the data sources and uses cases, these can be regular API endpoints like jdbc, s3, kafka, mqtt, https, &amp;mldr; .</p><p>But the more important (at least for this post) port is the Control Port. Why? Well, with this specific endpoint we get information from and control the defined data product. So let’s take an example:</p><p><em>As a business user In order to reuse existing data, I want to get the following information from a Data Product within my organization:</em></p><ul><li>In what domain is that data product?</li><li>Who has created it?</li><li>What information / fields are available?</li><li>when was it updated last time?</li><li>what is the data quality of it?</li><li>what data sources is this Data Product connected to?</li><li>what connection options I have to work with the Data Product?</li></ul><p>And here the Control Port comes in.</p><p>This port provides a consistent view on the Data Product. Endpoints are e.g.:</p><ul><li>metadata {name, owner, source systems}</li><li>lineage information</li><li>address of ports {input, output}</li><li>schema {input, output}. link to company ontology</li><li>audit</li><li>metrics, i.e.</li><li>last updated timestamp</li><li>loading frequency</li><li>loading time (how long did it take to load data)</li><li>data volume</li><li>data quality metrics {e.g. # of unmapped records}</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cjJxRL8l1_00Gqd9.png" /></figure><p>As you can see in the image above, the Control Port collects these information from a wide variety of sources like the Data Catalog, Data Quality Tool, the Data Lineage Tool or the DataOps tool of the Platform Foundation. It acts like an endpoint federation, as i.e. usually the Data Catalog tool of choise provides:</p><ol><li>a nice integration to publish the data product into the catalog</li><li>an API to get information out of the catalog</li></ol><p>Same applies (or should be applied) to the other components of the Data Trust layer.</p><p>So let’s take an example on the metrics, as these are quite common to be asked by the business in order evaluate further usage of the existing Data Product. When the business has easy access to the metrics of a Data Product, it can easily evaluate how much value this Data Product brings it for their use case(s). The exemplary metrics mentioned above give a good overview on typical demands and have their basis in the <a href="https://sre.google/workbook/data-processing/">SRE Workbook</a>.</p><p>But how does the user get these information? Do they have to query the API? Well, these information have to be available as API endpoints to allow automatic processing. But for Business Users there has to be a proper frontend incorporting them and allowing searching, visual exploration and interaction with the underlying information. This can be a custom frontend communicating with underlying API or off the shelf tools. (due to the broad landscape of these tools on the market, a separate post is required to capture that topic)</p><p>So, back to the initial question: Yes, the majority of us has found ways to increase the chance to get a rare mini figure as the surprise in the Kinder eggs. And we were creative with it. But nowadays, when we talk about finding the right Data Product(s) for the use cases, it doesn’t need shaking or a scale. In a proper Data Mesh based architecture, all the relevant information are available at the control port and consumable via an API <strong>AND</strong> a proper frontend.</p><p><em>Originally published at </em><a href="https://arnerossmann.github.io/post/2022-02-09_metadata-dataproduct/"><em>https://arnerossmann.github.io</em></a><em> on February 14, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=211c42925910" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Mesh — not such a new concept after all?]]></title>
            <link>https://medium.com/@arne.rossmann/data-mesh-not-such-a-new-concept-after-all-d0d57f1ec6cc?source=rss-a4692a2d91f2------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0d57f1ec6cc</guid>
            <category><![CDATA[data-mesh]]></category>
            <category><![CDATA[hexagonal-architecture]]></category>
            <dc:creator><![CDATA[Arne Roßmann]]></dc:creator>
            <pubDate>Fri, 28 Jan 2022 00:00:00 GMT</pubDate>
            <atom:updated>2022-04-13T09:49:21.622Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Mesh — not such a new concept after all?</h3><p>I don’t think that I have to tell much about the recent developments on Data Mesh (see <a href="https://martinfowler.com/articles/data-monolith-to-mesh.html">this</a> and <a href="https://martinfowler.com/articles/data-mesh-principles.html">that</a>), the world doesn’t need another “Data Mesh introduction” article just to tell these things again.</p><p>But when we dig deeper into that topic and look on the Data Product there might be some similarities ringing a bell. We’ll come to this later.</p><p>But how is a Data Product defined (by <a href="https://martinfowler.com/articles/data-mesh-principles.html">Zhamak</a>)?</p><blockquote><em>Data product is the node on the mesh that encapsulates three structural components required for its function, providing access to domain’s analytical data as a product.</em></blockquote><blockquote><strong><em>Code</em></strong><em>: it includes (a) code for data pipelines responsible for consuming, transforming and serving upstream data — data received from domain’s operational system or an upstream data product; (b) code for APIs that provide access to data, semantic and syntax schema, observability metrics and other metadata; © code for enforcing traits such as access control policies, compliance, provenance, etc.</em></blockquote><blockquote><strong><em>Data and Metadata</em></strong><em>: well that’s what we are all here for, the underlying analytical and historical data in a polyglot form. Depending on the nature of the domain data and its consumption models, data can be served as events, batch files, relational tables, graphs, etc., while maintaining the same semantic. For data to be usable there is an associated set of metadata including data computational documentation, semantic and syntax declaration, quality metrics, etc; metadata that is intrinsic to the data e.g. its semantic definition, and metadeta that communicates the traits used by computational governance to implement the expected behavior e.g. access control policies.</em></blockquote><blockquote><strong><em>Infrastructure</em></strong><em>: The infrastructure component enables building, deploying and running the data product’s code, as well as storage and access to big data and metadata.</em></blockquote><p>Additionally, we have the three types of ports.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*_OX1h4mlM8-xaNHt.png" /></figure><p>This looks really interesting and quite new, if you remember how the usual work in the data &amp; analytics domain was done previously. But times have changed, principles like Infrastructure as Code, serverles, DevOps &amp; DataOps, SRE etc. have changed the game towards more code focussed and software engineering thinking.</p><p>This morning I stumpled across a nice article on <a href="https://thecillu.medium.com/cloud-serverless-lock-in-doesnt-exist-if-your-team-knows-hexagoxal-architecture-4f64e93463aa">cloud serverless lock-in</a>. One approach here is the hexagonal architecture, in which Ports and Adapters are present.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/0*TFt6gM6i8ikxMZDk.png" /></figure><p>The goal of the hexagonal architecture is to encapsulate the logic within the core from the outer world. With this one, you can leverage some benefits:</p><ul><li>easier testing in isolation</li><li>development can be done independently from external services</li><li>ports and adapters are replaceable / extandable</li></ul><p>Now let’s try to map it to the Data Product from Data Mesh. The Data Product is connected via the <strong>Input Port(s)</strong> to the data sources, this can be a database (e.g. via jdbc), a Message Queue / Message Broker like Kafka, Web endpoints etc.</p><p>Then within the Data Product, the “standard” procedures of Load, Process and Provide are happening. They load the data from the source via the input port (or receive events via subscription e.g. to a Kafka topic), do some data transformation steps ( = process) and provide the outcome via the Output Port(s). This can be again a database or a S3 endpoint or again a topic in Kafka. The advantage of that encapsulation is that the Data Product can be developed without interfering other data pipelines from other Data Products, I can start with a simple pass-through and add further transformations steps as soon as I need them. And it can be easily extended on new output ports (e.g. starting with a S3 endpoint, later a REST endpoint) Sounds quite similar to the hexagonal architecture? Yes, and reflects the movement to more Microservices in the non-analytical world. And on the Data &amp; Analytics domain we see similar patterns, Data Mesh opens the door towards Domain Driven Development, Serverless, Data Products (within Data Mesh), Event-Driven are pushing towards Microservices / more granular processes.</p><p>So, back to the initial question: Is Data Mesh not so new? Well, from my view it is and it is not.</p><ul><li>It is not: Data Mesh uses approaches and concepts from the Software Engineering domain like domain driven design, hexagonal architecture, Microservices etc.</li><li>It is: these concepts were not the standard way of working in Data &amp; Analytics Space in the last years and therefore a quite new approach. But it is good and helps to bring the data-driven thinking and data-diven practise forward.</li></ul><p><em>Originally published at </em><a href="https://arnerossmann.github.io/post/2022-01-28_datamesh-not-so-new/"><em>https://arnerossmann.github.io</em></a><em> on January 28, 2022.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0d57f1ec6cc" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>