Bringing two powerful data tools together for one winning solution.

Table of contents

· Data mesh defined
· Key principles of data mesh
· Data as a product
· Autonomous domains
· Data infrastructure as a platform
· Distributed governance
· The role of a data catalog in data mesh
· Conclusion

Data mesh, a modern architectural and organizational concept that decentralizes data management, aims to overcome the limitations of traditional, monolithic data architectures. Keep reading to discover the benefits of utilizing a data mesh architecture with a data catalog to create a true data-driven strategy for all your teams.

Data mesh defined

Data mesh is an architectural and organizational data management concept that decentralizes an enterprise's design, creation, and management of data. It was inspired by the principles of microservice architecture for software development and aims to treat data as a product with responsibility distributed across multidisciplinary teams.

This approach seeks to overcome the challenges associated with monolithic, centralized data architectures by promoting a culture of collaboration and autonomy

Key principles of data mesh

Data mesh is based on four fundamental principles:

Data as a product

• Concept: Each business area treats its data as a product, with a dedicated team responsible for the quality, accessibility, and usefulness of this data for consumers.
• Dedicated ownership: Each business domain or area assigns a dedicated team responsible for the lifecycle of its data from creation to delivery, quality control, maintenance, and enhancements. This team ensures that data is reliable, up-to-date, well-documented, and easy to consume.
• Consumer focus: Teams prioritize the needs of data consumers (which could be other teams or external clients), ensuring the data they provide is useful, understandable, and easily accessible. This product mindset promotes continual improvement based on feedback, just like traditional products.

Autonomous domains

• Decentralization of data management: Teams are organized around specific business domains, enabling decentralized, independent data management. This autonomy allows teams to manage their data according to their specific needs and expertise.
• Domain-driven design: Drawing from domain-driven design principles, data mesh advocates for organizing teams around business domains. This structure aligns data ownership and operations with the natural boundaries of the organization, ensuring domain experts are responsible for their data and can independently manage it.
• Independence: Each domain operates independently, meaning teams can make decisions regarding their data models, pipelines, and tooling without being reliant on central data teams. This independence accelerates innovation and flexibility, allowing teams to respond faster to their own requirements.

Data infrastructure as a platform

• Self-service data infrastructure: To support the decentralized management of data, data mesh provides a centralized data infrastructure as a platform. This common platform enables domain teams to easily build, deploy, and operate their data products without needing deep technical expertise in data engineering.
• Reusable tools & services: The infrastructure typically includes services such as data storage, access control, data lineage, quality monitoring, security, and privacy features, so domain teams don’t have to reinvent the wheel. These tools ensure consistent data practices across the organization while removing friction for domain teams.
• Automation & scalability: The platform provides automation to handle routine tasks like data ingestion, transformation, and scaling, freeing up domain teams to focus on their core business functions rather than the intricacies of data management.

Distributed governance

• Federated data governance model: In contrast to traditional centralized governance approaches, data governance policies are applied in a distributed way, enabling flexibility and adaptation to the specific needs of each domain while maintaining enterprise-wide consistency and compliance.
• Collaboration & shared responsibility: Governance is a shared responsibility, where domain teams collaborate with governance bodies to implement policies in a way that aligns with their specific business needs. This approach balances enterprise-wide consistency with domain-specific flexibility.
• Global standards with local adaptation: While governance is distributed, there are still universal standards (e.g., security protocols and data privacy regulations like GDPR) that all domains must follow.

The role of a data catalog in data mesh

Data catalogs play a crucial role in data mesh implementation, providing the tools needed to facilitate discovery, understanding, and collaboration around data.

• Data discovery: Data catalogs provide a search engine and navigation mechanisms to easily find available data across the organization — An essential for teams working in a decentralized environment.
• Metadata and documentation: Automatic or manual cataloging of data with rich metadata helps users understand the context, provenance, and quality of data, aligning with the principle of “data as product.”
• Collaboration & sharing: Data catalogs facilitate knowledge sharing and collaboration between teams by enabling users to comment on, evaluate, and discuss datasets.
• Governance & compliance: By integrating governance rules and offering views on access and usage, a data catalog helps maintain data compliance and security in a distributed model.
• Interoperability: Data catalogs ensure seamless integration with other tools and platforms used by different teams, supporting the data infrastructure as a platform.

Conclusion

In conclusion, the data mesh environment is a promising approach to managing the complexities of modern data environments, promoting agility, collaboration, and autonomy. A data catalog is essential to realizing this vision, serving as a link between the various domains and providing the tools needed for effective data management as a product.

As you can see, the data mesh approach and the data catalog are intimately linked. However, it’s easy to confuse the two — Your company can live without data mesh, but the data catalog is a fundamental building block for understanding, managing, and adding value to data, regardless of whether you adopt the data mesh model.

Discover the benefits of creating an intuitive data catalog to fit your needs! Request a demo of DataGalaxy’s Data Knowledge Catalog, an all-in-one data catalog that offers out-of-the-box actionability with fully customizable attributes, powerful visualization tools, standardized business glossaries, and AI integration to help organizations easily document, link, and track all their metadata assets on one dynamic platform.

Creating a winning data strategy with data mesh architecture & a data catalog was originally published in The DataGalaxy Digest on Medium, where people are continuing the conversation by highlighting and responding to this story.

(Not the other way around!)

Table of contents

∘ Introduction
∘ What does “the need generates the model” mean?
∘ Do you recognize yourself?
∘ Ask yourself the right questions
∘ How do you ensure that the need generates the model?

Introduction

“It’s the need that generates the model — Not the other way around!”

This is the phrase I uttered spontaneously more than 15 years ago during a pre-sales meeting for a BI solution, in response to a prospect who asked me if we offered ready-made reporting models.

Hah! How many prospects have asked me that question?

Apart from the fact that I told them it was easier to knit than to unknit, I’ve adopted this answer to many of the questions that prospects ask themselves when it comes to implementing a data tool, as well as customers bogged down in the realization of their data project, or simply despairing over the non-adoption of their data solution.

What does ‘The need generates the model’ mean?

Over the years, I’ve changed the formula to: “It’s the need that guides the modeling,” “It’s the need that determines the content,” “It’s the need that determines the project,” and so on.

It’s clear that the need is at the heart of all data projects.

Do you recognize yourself?

• “I’ve implemented a data cataloging solution, but users make very little use of the application.”
• “We have implemented a master data management tool, in which we have put a maximum of reference data, but users continue to use and exchange Excel files with each other.”
• “We have set up a complete reporting system with all the KPIs we feel are essential for monitoring our organization’s activity, but usage statistics show that users have gradually stopped using it.”
• “We have created a data lake with a maximum of data, with the aim of satisfying as many of the organization’s potential data needs as possible. However, we regularly receive requests from users who can’t find the data they need.”

etc.

I could write a novel just on the symptoms described by my prospects and heard from my customers about not using their data application.

Ask yourself the right questions

• Have you met your users’ needs?
• Have you solved one or more of their problems?
• Are they interested in using the solution?
• Have you listened to their needs?
• Did you put the need at the heart of your project?
• Did the need generate your model?
• (Bonus: Was the project driven solely by IT?)

How do you ensure that the need generates the model?

1. Get people involved
   A good data project is one in which the future users of the solution are involved from the outset and throughout the project:

• Appoint a functional/user project manager
• Appoint key users: Identify and select the users who will drive the project.
• Involve users in key phases of the project: Needs analysis, design, acceptance, deployment, change management, adoption, etc.

Don’t hesitate to work in small iterations, to enable ongoing validation of the future data solution by key users.

This is not a question of forced validation but of genuine collaboration between the producers and future consumers of data.

2. Know how to listen
Taking user needs into account is the key to successful data projects.

And we’re not just talking about “functional” users here, but all the people who use data, from those in charge of maintaining data flows to reporting users, Data Analysts, and Data Scientists.

It is therefore important to identify and listen to the needs of all these users by carrying out interviews and prototypes to involve them in strategic choices.

3. Understand the changing needs
Data projects evolve the most over time and therefore require constant adaptability and attentiveness to user demands.

• “I need a new indicator!”, “I need a new report!” — What are the consequences of these evolutions on my decision-making chain? What impact will they have?
• “We’re integrating a new entity into our organization, and this represents a new source of data in our customer repository.” — What impact will this have on the supply of data to our MDM customer repository? Do we need to review our matching rules?
• “We need to document the data products made available to data analysts so that they can independently choose the right data source for their work.” — What level of detail is required by table or column? How do we involve users in the documentation of these data products?

4. Let’s be agile
Needs that evolve over time call for agility.

Listening to needs and taking them into account in modeling and in what we deliver to users is essential, but so is relying on agile tools to be responsive and maintain user involvement.

Conclusion

Your choice of data tools should focus on agile solutions that are easy to deploy and implement so that realization is not a hindrance but the simple application of the need you’ve taken the time to analyze with your favorite users!

It’s the need that generates the model was originally published in The DataGalaxy Digest on Medium, where people are continuing the conversation by highlighting and responding to this story.