<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Scalable Data Management - Medium]]></title>
        <description><![CDATA[Data Architecture, Governance, Storage, Maintenance, Security - Medium]]></description>
        <link>https://medium.com/data-management-at-scale?source=rss----c003a8dffc41---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Scalable Data Management - Medium</title>
            <link>https://medium.com/data-management-at-scale?source=rss----c003a8dffc41---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 21:49:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/data-management-at-scale" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[6 Things I want to tell you about Data Mesh]]></title>
            <link>https://medium.com/data-management-at-scale/6-things-i-want-to-tell-you-about-data-mesh-4d8f461d8a4e?source=rss----c003a8dffc41---4</link>
            <guid isPermaLink="false">https://medium.com/p/4d8f461d8a4e</guid>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[data-management]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-mesh]]></category>
            <category><![CDATA[architecture]]></category>
            <dc:creator><![CDATA[Wojtek Brycki]]></dc:creator>
            <pubDate>Fri, 28 Jul 2023 08:27:13 GMT</pubDate>
            <atom:updated>2026-08-11T19:40:09.468Z</atom:updated>
            <content:encoded><![CDATA[<p>We are living in the world of fragile and complex pipelines, transferring data from A to B. Data Lake teams are doing everything to meet the expectations of business, for the platform, supporting new integrations, resolving bugs.</p><p>We have many data sources, transformations, different representations of the same data, which at some point gets further “prepared” for the reporting layers.</p><p>If we have luck and very good software engineers, we will get to reporting layer and the reports (where we can finally get some valuable information)…</p><p><strong>…but it doesn’t always work, at least not in the way we want it to.​​</strong></p><p>Imagine for a while a world without all the complicated data pipelines, without transformations, without putting the data somewhere else, but still getting the value from the data. ​​This is what Data Mesh is about, but there is of course something more that.​​</p><p>Here are my 6 findings, after studying the Data Mesh topic:</p><h3><strong>1. Data Mesh is not something we can buy, it a set of principles we follow.​​</strong></h3><p>Data Mesh represents a decentralized approach in accessing data at scale, it takes a lot from Domain-Driven-Design​​.</p><p>It’s about getting <strong>value right from the source</strong>, not having flows of data and then to get a value from that (also by having an ability of linking the different models of those domains).​​</p><p>This domain orientation, is a little bit like microservices for the System architecture, but it’s applied to the data architecture​​.</p><p>Many current data architectures evolve around ETL processes​. The current situation with data lakes and specialized teams can look like that:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a5hvS8FP9gdIp2wVyHK5nA.png" /><figcaption>ETL processes with fragile pipelines within the data lake</figcaption></figure><p>The domains today are generating the data <strong>for their own usage</strong>, they own way of running the business, they don’t care so much about analysts wanted to extract something from that data later.</p><p>The Data Lake teams are responsible for every part starting from data import, cleaning, transform to another representation, data pipelines in data lake, how the data looks like for analytics and reporting.</p><p>Those teams are often a bottleneck of the whole data lake working properly. We cannot freely enlarge them because it just doesn’t work.</p><p><em>Do we even know what we are taking about at the dailies where there are so many topics and many devs working? </em>OK, let’s divide a team.<em> But then… we have even more complicated structure, that needs to talk to each other.</em></p><p>This is one of the problems that Data Mesh can address.</p><p>Domain orientation and decentralized approach.</p><h3>2. Data Mesh involves company transition and technology.</h3><p>There is no specific technology that must be used​​. But we still need some technology — it means whatever fulfills our needs.​​</p><p>You can say you have Data Mesh — or better applying principles of Data Mesh, by having appropriate technology, <strong>and using it correctly</strong>.​​</p><p>We don’t solve these data management concerns with just buying a technical solution,​​ it’s more about how we are gonna use that, how do people and processes interact with the data, because this way we can solve data quality and ownership issues.​​</p><ul><li><em>When following the Data Mesh principles, the highly specialized bottleneck teams are gone​</em></li></ul><p>​The <strong>domain owners have to take the real ownership of the data</strong>, and how this data looks for anyone else who want’s to use it.</p><p><em>Is the whole domain data accessible for everyone?</em></p><p><em>Are only some parts of it accessible (to limited amount of users)?</em></p><p>The technology should allow to have some unified (connectable) models with other domain owners​.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*ZeekH8z1PqDG6L2MGGhgjw.png" /><figcaption>a model can connect just some of the data products from within the domains</figcaption></figure><h3>3. Data Mesh is not for everyone.</h3><p>It resolves problems of larger organizations and bottlenecks of Centralized Data​​. It is rather not for organizations with small amount of domains and/or data.</p><p>Designed for complex environments​​.</p><p><em>If we have a good functioning data lake, with 2 products relatively small amount of data, we may be completely fine without adopting Data Mesh principles.​</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*2BNP3_xv7jykr8qSQaqXdw.png" /><figcaption>based on the diagram used by Zhamak in a Webinar [Dehghani, Z., 2022], References [1]</figcaption></figure><p>When the complexity grows, there are more data sources, bigger scale, the potential of Data Mesh gets bigger.</p><h3>4. Data Mesh is not adopted all at once</h3><p>Start small and improve​​. Build iteratively. Be use-case driven​​.</p><p><strong>Build Data Products</strong>. Relatable and linkable to each other. Later on, you can pass them to domain owners. Even if those are not findable and accessible by other domains right now, they could be in the future, after the platform and other domains will be ready for it.</p><p>Avoid large big bang​​, prevent mistrust of the user (“<em>I will still use the current solution because it works, and the new one is too much to handle”</em>)​​</p><p>There is not a magic button when we say “apply the Data Mesh”… click and it’s done. Because of technological challenge and also the company culture change challenge​​.</p><h3>5. Data Mesh principles interact with each other.</h3><p>​</p><p>The first and most important principle is the <strong>Domain Ownership</strong>.</p><p>All the other ones are actually implication of the first one. The domains alone will not allow to register data products, access them, define the access policies. This is why we have the following 3 principles:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*A5RgiiZsGinjxx7dhrSBDg.png" /><figcaption>own interpretation of Data Mesh principles working together, Brycki, 2023</figcaption></figure><p><strong>Data as a Product</strong></p><p>If we are creating domains, we have to somehow make it accessible to the outside world​​, to treat is as a product, which can be used by someone else.</p><p>Products can be registered on the platform, are findable afterwords and can be addressed, put together with another products in order to extract some meaningful data.</p><p>…and those products have to be registered, maintained on a:</p><p><strong>Self-serve Platform</strong></p><p>which is basically a technical execution of all of what Data Mesh is about.</p><p><strong>Federated Computational Governance</strong></p><ul><li><strong>Federated</strong> means we have a federated group of experts taken from every single domain, they discuss and decide within each other: how the data is accessible, what is exactly accessible and for whom, for which data mesh users/groups.​​</li><li><strong>Computational</strong> means we can compute some results based on the different domain models. ​​</li><li><strong>Governance</strong> means how the policies look like, do we have to mask some column because it just shouldn’t be possible for some data to get out of the domain scope, maybe the data can not get out of some region.</li></ul><h3>6. The biggest technical challenge is to create a truly self-service platform</h3><p>A self-serve platform would allow not only registering and finding the products. Linking of the model (products) between domains​​ is important as well. Those model connections can be realized by distributed query engines (like Trino or Spark SQL)</p><p>When implementing a platform, some questions and considerations arise:</p><ul><li><em>How to justify the increased cost of creating those domains?</em></li><li><em>Do we need new databases?</em></li></ul><p>Perhaps not, it’s the same object storage as we have right now (we just connect to it directly at the source)</p><ul><li><em>Do we have to introduce new roles in the organization? What with the cost of that?</em></li></ul><p>Not necessarily new roles, the right person needs to be empowered to be able to use the self-serve platform (it’s about taking the responsibility of taking care of the domain).​​</p><ul><li><em>Big investment on the development side</em></li></ul><p>This may be true, but also, we are using mostly the same technologies as before, just in a different manner. We also shouldn’t forget about the cost of missed opportunities (when not following a Data Mesh architecture).</p><ul><li><em>Production data directly accessible for anyone else? That’s not gonna work…</em></li></ul><p>Doesn’t always have to be like that. The domain can provide the most relevant copy of that data, but that still should happen within the domain itself, same with the pipelines, if they are needed anyway, they stay within the domain.</p><h3>Conclusion</h3><p>A big takeaway for me is embracing the culture of transformation that needs to happen. As a domain owner, the people need to be responsible for the data, for it to have a good quality, be findable and addressable.​​</p><p>You can have a great tool, but without having the transformation in the company, the changing of the mindsets, it will be pretty much useless.</p><p>Extra point for <em>till the end readers</em>:</p><h3>2x Data Mesh?</h3><p><em>Is there another Data Mesh term as the one that i described here?</em></p><p>Yes, it is. Comes from Netflix (medium article from Aug 1, 2022; Jul 26, 2021). Describes Data Movement platform. And was supposedly started around the same time when Zhamak Dehghani first defined the term <em>Data Mesh</em> in 2018.</p><p>In this article I was talking only about the Data Mesh definition defined by Zhamak.</p><p>My blog: <a href="https://brycki.pro/">www.brycki.pro</a></p><p>References:</p><p>[1] “Data mesh and domain ownership” — Webinar</p><p><a href="https://www.thoughtworks.com/about-us/events/webinars/core-principles-of-data-mesh/data-mesh-and-domain-ownership#recording%E2%80%8B">Data mesh and domain ownership</a></p><p>[2] “Core principles of Data Mesh” — Thoughtworks website</p><p><a href="https://www.thoughtworks.com/about-us/events/webinars/core-principles-of-data-mesh">https://www.thoughtworks.com/about-us/events/webinars/core-principles-of-data-mesh</a></p><p>Used content:</p><p><a href="https://pixabay.com/vectors/care-fragile-packaging-handle-43938/">https://pixabay.com/vectors/care-fragile-packaging-handle-43938/</a></p><p><a href="https://pixabay.com/vectors/cyber-connection-network-6998811/">https://pixabay.com/vectors/cyber-connection-network-6998811/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4d8f461d8a4e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-management-at-scale/6-things-i-want-to-tell-you-about-data-mesh-4d8f461d8a4e">6 Things I want to tell you about Data Mesh</a> was originally published in <a href="https://medium.com/data-management-at-scale">Scalable Data Management</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>