Data Mesh
The current article is based on the ideas presented in the Data Mesh book.
I strongly recommend reading this article even for non-data specialists, as it highlights foundations that every one of us, developers, should be familiar with.
After reading, you will gain a solid understanding of the fundamental concepts outlined by Zhamak Dehghani. Towards the end, I will also share my thoughts on the feasibility of implementing these ideas in current realities.
Ok, Data Mesh — what is that? Framework? Approach? Technology? Or simply a new buzzword? Let’s figure it out.
But first, it’s essential to take several steps back to shortly discuss current data analysis solutions state with all its benefits and accompanying challenges.
Brief overview of data analysis approaches
There has always been a need for efficient data analysis. Over time, solutions created to serve it have evolved to some well-known approaches. The most popular of them I covered below, of course, in an oversimplified way.
Data warehouse
Centralizes data from various sources, serving as a single source of truth for historical organizational data. Predefined structure aids in generating BI reports. Pros include ready-to-use data for business value extraction, while cons involve fragile pipelines and inflexibility for schema changes.
Data lake
Similar to a data warehouse but with minimal data transformations. Allows storing raw data without strict design, suitable for research and development. Pros include openness for future transformations, but cons involve potential messiness and unreliability, impacting data integrity and correlation.
Data lakehouse
Combines advantages of data warehouse and data lake, prioritizing query performance. Aims to offer the best of both worlds, addressing limitations of rigid schema and potential messiness in data lakes. But downside of such approach would be an overall complexity. Implementing and managing a data lakehouse would require careful design and governance to ensure optimal performance.
Data mesh
All the above-mentioned approaches share one common trait — they are centralized, which generally becomes a bottleneck because such centralized solutions usually require one data team for maintenance, constantly bombarded by requests for changing data pipelines. This team is very good, almost excellent in technology stack knowledge but lacks understanding of necessary business aspects of data domains, knowledge possessed only by requester teams. Such a situation leads to wrong priorities and often results in delayed data-related implementations.
It is clear that only the domain team has the best knowledge of how data should look, to which standards it should adhere, and in which timeframe it should be implemented. With an overloaded centralized team, domain teams become kind of blocked. And here, the Data Mesh paradigm comes into play, treating data as a product, allowing teams to focus only on what they do best, and introducing key principles to optimize usability across your organization.
Very briefly, the Data Mesh concept advocates for:
- Domain teams composed of individuals with expertise in the business domain, technical skills, and data knowledge.
- Shifting-left approach for business-specific implementations, avoiding centralization of all complex data pipelines. General reusable practices on a platform layer can be managed centrally, but domain teams should support business-specific implementations.
- Treating data as a product, emphasizing reliability, discoverability, and adherence to shared company standards.
- Ensuring all data is secured and easily searchable for all teams by leveraging platform capabilities.
Data mesh — how to?
Let’s now discuss the necessary steps to bridge the gap between your current situation and the desired state. Generally, all your decisions should lead to the following three things:
1. Empower your platform team to design and develop all necessary infrastructure-level software, removing unnecessary cognitive load from domain teams and allowing them to fully focus on their business goals.
2. Empower domain teams to make all necessary local decisions to extract the most value from their business domain and leverage it through data.
5. Establish necessary governance practices and procedures to ensure that crucial global decisions can be cascaded to teams by leveraging platform capabilities.
Data is a product
Okay, data is a product, but what does it mean?
First, let’s answer the more fundamental question: what is a product?
Typically, we consider something a product if it’s a final outcome for which our clients pay. But what if we go deeper? Does company has it’s internal products? Yes, it does. It may be both data and software.
Let’s first talk about the internal software products as they form the foundation for data products.
Depending on the number of teams and their responsibilities, each company usually requires its own internal software, evolved by one team and used by all others. This could include internal plugins, libraries, customized open-source solutions, UI tools, and more — essentially, anything that can be reused by everyone, encapsulating infrastructure complexities and reducing the cognitive load required to set up infrastructure resources or build something.
Considering this, we have internal software products with their own release cycles, quality assurance, and a dedicated team responsible for their evolution and delivery to internal customers: from developers to developers.
All these software products belong to a so-called platform layer.
Next fundamental question: what is data?
Data is raw knowledge of something represented in an understandable form.
And now let’s merge our understanding of what an internal software product is with what data is, to get a Data Product. Owned by a domain team, a Data Product consists of all data-related code and configurations, treated as a product with the imposition of the quality assurance initiatives. It is used to create and deliver business data to a commonly used Data Mesh.
Data products can be created from scratch or composed from existing ones to generate higher-order business value. The process of data creation, delivery, and evolution should be established and supported from both a technological (platform) and governance perspective.
Platform
Let’s talk more about the platform and the possibilities it can offer.
Get Pavlo Lozovikov’s stories in your inbox
Join Medium for free to get updates from this writer.
All technical support for a Data Mesh is a platform concern. To enable domain teams to focus solely on delivering business value, the platform layer should conceal all internal infrastructure complexities, exposing only APIs, declarative configurations (such as CI/CD for data testing and delivery), and a user-friendly UI for data search, access rights management, policies, and more.
While the specifics of what should be included in the platform layer can vary from one company to another, certain general features would be common for everyone.
CI/CD
To foster trust in each other’s data quality, the process of testing configurations and building data before delivery should happen. Platform level would be the best place to provide such possibilities. For every change in configurations or code directly impacting data quality, an automatic testing CI/CD process can be executed before the change is deployed to production. Let’s walk through a hypothetical example.
Testing
Let’s delve deeper into testing capabilities, a crucial aspect often overlooked in our current realities :)
Data must be in a format understandable by a consumer, be it human or system. The structure of the data (field names, types, formats, etc.) should be stable and versioned. Therefore, the testing stage, or a set of multi-purpose tests, becomes a valuable candidate(s) for this purpose. The domain team determines what aspects should be tested, while platform implementations dictate how and where tests should be executed.
Taking testing to a more comprehensive level involves establishing Service Level Objectives (SLOs) for data and conducting tests against these objectives. SLOs act as a bridge between the known and unknown, enhancing even more trust in the data for its users.
Governance
The Data Mesh paradigm aligns closely with the overall governance structure in the company, particularly with how teams are organized. As one of the core principles of a data product is to avoid blocking centralizations, it emphasizes domain-oriented teams comprising individuals with expertise in business, technology, and data aspects. This results in diverse specialization within each domain team, and the ultimate objective of such team structures is to independently manage the lifecycle of their own data products.
Domain teams should possess all the essential resources and autonomy required to deliver their data products at their desired pace and make all necessary data-related decisions autonomously.
What we can do already?
Despite the current situation you find yourself in, there are still things which everyone can take care of.
Leverage the data you already have
Understand the type of data your systems generate and make it usable. Even if not every piece of software produces business-related data, but each one exposes operational data, essentially functioning as a data product. By leveraging and establishing common observability practices, we can enhance the quality of our internal software.
Apply the same practices to your operational data as discussed earlier: devise a common way to expose observability signals from your systems and implement a unified approach to interpret and analyze it through a platform layer.
Initiate the process with the less risky and impactful parts of your systems, proceed in iterations, and then scale to onboard more complex and critical components.
Stable APIs
Prioritize the semantics and versioning of your APIs and data schemas; it enhances trust and fosters better comprehension, helping delineate what should be grouped together and what should remain separate.
And, NEVER cut quality corners on APIs (contracts) since they are vis significantly more costly to fix after. It’s far easier to fix implementation details within the contract’s “container,” given that even minor changes to a contract impact teams and internal software relying on it.
Everything is a product
The idea of considering things we creating as a products is rewarding — it fosters trust within your organization and will let you be more proud of things you’re creating. A product mindset equals established testing, building, and delivering practices, comprehensive documentation, enabling observability, and measuring key performance indicators (KPIs).
Make right prioroties right way
Consider sources of fundings. While time-bound business initiatives are a great drivers, they should never be the sole budget source because the platform layer is something which is never done but an ongoing evolution, continuously increasing stability and covering more and more use cases. It should be an always-living internal product marketplace.
Start with complementary use cases; this approach allows for the creation of solutions to a problem from different angles. This, in turn, facilitates the identification of both use-case-specific elements to be supported by a domain team and general reusable blocks to be ingested by a platform.
Additionally, start with use cases that have minimal dependencies on the platform’s missing features.
This approach has several advantages:
- It allows leveraging already existing capabilities.
- It avoids complete blocking of anyone.
- It fosters healthy contributions to platform capabilities, enabling users to develop solutions on their end that can be later integrated into the platform if necessary.
Understand the focus groups and the order in which they will join the mesh, as each stage of mesh development will reach its own state of maturity. The later the stage, the fewer changes it will undergo, but it will be utilized on a larger scale. Refer to the s-curve explanation for further insights.
Conclusion
From my honest perception, the idea of a data product and data mesh is nice and beautiful, but implementing it in you organization may require a quite lengthy journey (years?), considering the current technology stack, the efforts for reorganization, and associated complexities. HOWEVER, there are numerous ideas and concepts that can be extracted and repurposed for other valuable initiatives, providing immediate benefits.
The concepts presented in the book shape the understanding of how teams and their focuses can be organized to squeeze maximum business value.
Of course, there are a lot of more detailed examples in the book of how things might be implemented which I did not covered in this article since I decided to focus only on stuff which would be of broader audience interests but also motivating enough to delve deeper into the world of data. After all, our systems, scripts, and the software we create operate on data, often serving as the final output. Taking more care about data can lead to more adequate solutions and much reliable daily decisions taken.
If I’ve motivated you to explore one of the most inspirational and useful books I’ve read recently, I wish you an enjoyable reading experience. Feel free to return to this article later after reading to share your own thoughts, or, even better, do it right now! :)
