Over the last 2–3 years, I’ve seen a notable resurgence in demand for Master Data Management (MDM) solutions, especially in the mid-market and nonprofit sectors. In addition, at least anecdotally, the success rate for these initiatives seems to be higher than in years past.

So what gives? Two words: Customer data.

Organizations have never had richer, more actionable data about their customers, but that data is increasingly spread across disparate enterprise systems and cloud applications. In the customer space, having actionable intelligence helps lean organizations allocate scarce resources to attract loyal customers, prevent churn, and drive down customer acquisition and retention costs by focusing on the highest-value or most at-risk customers. This means there is real value in having reliable, consistent customer data, and helps explain why MDM is on the radar for more organizations. Many organizations are also focusing on related efforts around Customer Experience, Customer Journey Mapping, and 360-Degree View of the Customer, which are further helping drive MDM demand. And lastly, MDM solutions have continued to evolve and consolidate and have gotten cheaper and easier to implement as a result.

But, more importantly, why might these more recent MDM initiatives be perceived as more successful? Today’s MDM initiatives tend to be focused on resolving specific issues with managing customer data across disparate on-prem and SaaS platforms. This is in contrast to MDM initiatives from years past, where the focus often seemed to be on attaining ambiguously defined “enterprise data/information management” and “data governance” maturity. While Herculean MDM and data governance efforts — complete with technology platforms, full-time data management professionals, and formal data governance organizations and committees — may have made sense for large multinationals and companies with complex data ecosystems, many small and medium-sized organizations who attempted to implement these complex MDM solutions didn’t ultimately realize the value needed to sustain the effort. It’s no surprise that more recent efforts to wrangle customer data that have been smaller, more agile, and more focused on an agreed-upon set of achievable outcomes, might have a higher success rate.

With this in mind, here are a few tips for a successful MDM initiative:

1. Right-size the effort. MDM, like most investments, is subject to diminishing returns. Focus MDM on the highest value data that needs mastering. In most organizations, this is customer and product data. The most successful initiatives I’ve seen stay focused on enabling a few core use cases and then grow organically from there. Remember the lesson given to us by the US Navy: “Keep It Simple, Stupid.”
2. Stay agile and deliver results often. MDM will almost inevitably require a multi-year effort to be fully integrated into an organizations’ technology ecosystem. Teams that choose to go off into a back room for 12+ months to develop a “big bang” MDM solution will almost assuredly run into major setbacks, perhaps even failure. It is imperative that MDM work be tied to business value, and the best way to ensure this is to deliver results to the business early and often, get user feedback, and adapt accordingly. Focusing on achievable results and building a Minimum Viable Product first are great ways to accomplish this.
3. Build a great team. A strong MDM team needs a mix of skills that include MDM strategy and solution architecture, MDM platform/technology expertise, knowledge of current and future business applications and IT infrastructure, MDM and application integration engineers and architects, IT and business SMEs, and data governance/stewardship process SMEs. Last but not least, a great Solution Owner and can make all the difference.
4. Executive sponsorship and organizational buy-in are key. Strong business and IT support is critical for a successful MDM initiative. Few efforts will touch as broad an array of systems and business processes as MDM, so having buy-in and support across the board is a must-have. The MDM implementation team will absolutely need to rely on help from other teams and process owners, so take the time to build relationships with other teams and provide them and their managers with as much visibility as possible into upcoming asks for help and their role in ongoing participation in MDM and governance processes.

4 Tips for a Successful MDM Project was originally published in Harlan’s Data and Analytics Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

After years of innovation in big data, there is now an overwhelming landscape of modern data platforms that companies can choose from to solve their big data needs. In addition, there continues to be aggressively optimistic marketing selling a bright future if only companies adopt this or that platform. The result is that, despite Gartner dropping big data off its annual Hype Cycle report back in 2015, misinformation and misconceptions about modern data architecture are still common. The following are 3 myths about modern data architecture.

1. Modern Data Architecture Will Solve All of Your Problems — Talking about cybersecurity, noted cryptographer Bruce Schneier once said “If you think technology can solve your security problems, then you don’t understand the problems and you don’t understand the technology.” The same is true for modern data architecture. Extracting value from data isn’t about any one particular technology platform or computing paradigm. It’s about piecing together a well-architected ecosystem of technologies and business processes that work together seamlessly. If you think technology will solve all of your problems, chances are you need to dig deeper to understand the real issue.
2. Everyone Else Is Doing It — Back in 2013, Dan Ariely, author of Predictably Irrational, made the brilliant observation that “Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it.” While it’s been several years since Ariely made this comparison, many companies still have a feeling that they need to be moving to modern data architecture platforms just to keep up with the Joneses. The truth is, many companies are just now starting to explore modern data architecture use cases. Sure, many companies have moved some of their internal applications and products to the cloud, and many Silicon Valley companies are built on big data, but there is still relatively plenty of room for growth in modern data architecture for most companies. The important thing to keep in mind is that business is not a race to have the newest, sexiest technology. If you’re implementing technology you don’t need, just to keep up with the Joneses, you might want to rethink your strategy and ask how what you’re doing makes you money, reduces costs, or improves customer experience.
3. Data Modelling is Dead — I’ve touched on this before, but understanding the structure, lineage, and usage of your data is what makes it an asset. With the power of modern data architecture technologies, we can often sweep issues caused by poorly designed data models under the rug with overwhelming computing power. But keep in mind, the technology isn’t worth anything without the data in it.

3 Myths About Modern Data Architecture was originally published in Harlan’s Data and Analytics Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Building a team of great data engineers requires analyzing not only a candidate’s technical knowledge, but also a host of other characteristics, including analytical/critical thinking, communication, problem-solving and perseverance, attention to detail, and more.

In today’s world of rapid technological change and overwhelming choice in solutions and services, companies need to recruit data engineers who, above all, can help the company’s data platforms be more adaptable, integrated, scalable, and secure. But what are the characteristics of a data engineer who can help you do these things?

1. Be Adaptable and Curious

Due to the rapid pace of innovation and low barriers to entry, the data engineering space comprises a staggering range of competitors, from niche players to enterprise platform vendors. As a result, many technology platforms simply haven’t existed long enough or seen the widespread adoption required to form a critical mass of skilled experts. So if you can’t hire an expert, what’s the next best thing? Hiring people who are good at becoming experts!

Great engineers love to learn. This applies to learning new technologies as well as learning new areas of the company’s business. The opportunity to explore areas of passion and interest through continuing education and self-study helps data engineers keep a healthy academic interest in the profession and helps to keep up with changes in the industry as they develop. I would go so far as to argue that great data engineers need to be curious and enjoy learning new things, usually under pressure. They take almost nothing for granted and enjoy questioning long-held assumptions.

2. Have Depth AND Breadth

It is often said that software engineers require “T-shaped” skills: depth of skills in one or two areas, coupled with a breadth of knowledge across the field. With agile development and DevOps catching on within IT and product teams at most large enterprises, developers are increasingly asked to take on a broader range of technologies and responsibilities that might have typically fallen to DBAs or system admins. Developers who are deep in one area and unwilling to expand their horizons are sure to face challenges in today’s job market.

3. Take Ownership

Data engineers are not in the business of finding problems and complaining about them (though they are quite good at it), they are in the business of solving them. They take ownership and drive issues to resolution, collaborating with whomever they need to along the way. In the new DevOps world, there might not be a DBA or a system admin to help you. Empowered teams taking ownership of their own future is one of the reasons agile software development works so well.

4. Have Strong Opinions, Weakly Held

Having opinions requires conviction and sticking your neck out. Admitting we are wrong is hard on our egos. But great data engineers can do both. They can argue convincingly for the opinions they hold, but are always willing for that opinion to be replaced by a better one. The human brain is imperfect and subject to cognitive dissonance and bias, so we must constantly look for these behaviors in ourselves and remind ourselves to be open to new ideas that might break our current modes of thinking.

5. Embrace Modern Data Architecture

Moving data workloads to the cloud and to modern data platforms like Hadoop and Spark really does require a shift in thinking for many experienced data engineers. A fundamental understanding of distributed computing is a prerequisite in this space. Volume, variety, and velocity can all be tamed with the right set of tools, but it takes engineers who ultimately understand which of those they are trying to optimize for and how the technology platforms they are using fundamentally solve those problems. Scalability is paramount in a world where value can be extracted from systematically tracking down needles in haystacks, so developers who can expand their horizons and incorporate modern data architecture into their enterprise ecosystems will be ahead of the game.

5 Characteristics of a Great Data Engineer was originally published in Harlan’s Data and Analytics Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.

Over the years, I’ve worked with many companies to apply agile software development to enterprise data warehouse (EDW) development projects. Data warehousing isn’t that different from other types of software engineering, so it’s no surprise that the basic principles of agile project management, as well as its various flavors like Scrum and Kanban, apply very well to database and ETL development projects.

But I do still continue to see a few common questions that come up on project after project, especially for teams who are new to agile data warehousing. Here are the four most common questions I get from agile data warehouse teams.

1. How do we handle solution architecture with an agile team?

Agile is all about breaking things into small chunks that can be completed in days or weeks. Solution architecture is all about making decisions and predictions about what a system will look like months or sometimes years into the future. Agile helps us deliver fast today, but solution architecture helps us avoid refactoring and technical debt later on.

On many traditional EDW development teams, one or more Solution Architects is put in charge of the overall design of the system. He or she is typically responsible for the high-level system architecture, as well as data models, coding standards, naming conventions, reusable frameworks, and patterns that should be followed by developers, as well as anti-patterns to be avoided. An architect might create architecture diagrams, logical data models, and other design artifacts that can help serve as blueprints for the development team.

Things are a little different in the agile world. In Scrum, for instance, we have roles for a Product Owner, a ScrumMaster, and Developers, but no Architect. So does this mean solution architecture itself is a relic of the past? Absolutely not.

In agile, solution architecture is not the responsibility of just one person; it’s the responsibility of the whole team. Developers in an agile team are responsible for communicating with one another not only what they are working on, but also how they are approaching the work. Critical thinking, design, feedback, consistency, peer review, and open dialog are not only encouraged, they’re expected. Senior engineers (as well as ScrumMasters and Product Owners, where appropriate) may sometimes play a bigger role in major design decisions, but generally speaking, everyone on the team needs to be committed to making a great product.

In agile, we tend to shy away from architects who don’t code. Instead, we encourage our developers to grow their solution architecture skills. This means getting ETL developers up to speed on EDW solution architecture, data modeling, coding practices, and how to deal with (or, preferably, avoid) technical debt and refactoring.

But there is definitely still a place for Solution Architects on agile teams. Larger teams may still find it useful to have a solution architect who oversees the many agile teams working on different parts of the same solution or ecosystem. In this capacity, the architect is responsible for driving a shared understanding of the system architecture, development techniques, and data model. Some teams may also find it useful to think of a solution architect as a technical product manager, responsible for negotiating with the product manager to prioritize technical foundation tasks that will help build a scalable system that won’t require costly rework or refactoring later on.

Solution architects on an agile teams will often:

• Actively participate in “whiteboard” design discussions with product owners and developers to discuss implementation of product features
• Perform early stage design reviews as well as hands-on code reviews with developers
• Meet one-on-one with developers when they have questions about approach or architecture
• Lead periodic training and documentation for solution architecture, data model, and development tools and techniques
• Participate in backlog grooming, sprint planning, and sprint review

2. How are detailed design tasks like data modeling and source-to-target mapping handled? Who does them and when?

This is one I see come up a lot. In ETL development, we often start with a simple spreadsheet that maps how each field in a data warehouse is populated, including source fields and transformation logic. Developers will reference the document while coding their ETL, and will (hopefully) keep it up-to-date if they find any discrepancies or if they take a different approach during implementation.

So, is a source-to-target mapping document included in the user story as part of “Definition of Ready” before a developer pulls the story from the backlog? Or does the developer complete source-to-target mapping during the sprint as part of the ETL development story?

The answer is (wait for it…), it depends.

In large teams, a Product Owner may have one or more analysts working in a product ownership team to help gather and document detailed requirements like this. In these cases, the up-front data discovery and detailed, field-level mappings can be performed by one of these analysts and attached to the user story while it is still in the product backlog.

On smaller teams, a product owner may only be able to give a vague idea of where the data should come from, and a developer will need to analyze and query the source data to perform discovery and mapping on their own.

Both are absolutely valid approaches. What you don’t want, though, is ambiguity on who is doing what. If source-to-target mapping is part of Definition of Ready, make sure that this is agreed upon by the whole team and is consistently enforced. If a developer is doing mapping during the sprint, this also needs to be explicitly called out.

Just keep in mind, if mapping is done during the sprint, you’ll need to be ready for the occasional delay as data discovery can uncover unexpected data anomalies, or we may find out that the data as it is captured or stored doesn’t allow us to meet the requirement of the story.

3. How do we handle dependencies between user stories or tasks?

Data warehouses are chock-full of dependencies. We have ETL packages that are dependent on upstream source data, stored procedures that are dependent on tables, referential integrity constraints that are dependent on not only tables but also the data in them, and don’t get me started on views (especially views that reference other views).

We need a way to manage and track these dependencies so we can plan for them during backlog grooming and sprint planning, and so developers know what order to work on tasks during the sprint.

Many agile project management tools (like Jira and TFS, for instance) allow teams to explicitly list dependencies between stories, and even visualize a dependency map. This is something many teams don’t take the time to do, but believe me; it can be well worth the effort. If you know that you need to build your Customer dimension before you build your Sales Order fact, it will save you some refactoring work if you do them in the right order.

If you are using a low-tech approach like sticky note cards to track your user stories, this can be done by physically stacking the cards on top of one another, with the tasks that must be done first sitting on top.

Part of the difficulty here is that — going back to Definition of Ready — a user story in a sprint backlog is not supposed to have any blockers and should be ready for any developer to work on. To use our example above, one developer might pick up the task to develop the Customer dimension on the same day that another developer picks up the task to work on the Sales Order fact. What do we do in this case?

As with most things in Agile, it comes down to team members communicating. The developer working on the Sales Order fact can start by building everything he or she can that isn’t dependent on the Customer dimension. If he or she gets to a point where work is blocked, perhaps they can switch gears to give the other developer a hand to get the dimension completed.

Many agile EDW teams struggle with dependencies at first, but once developers are communicating with one another and keeping track of the dependencies during planning, this tends to become like second nature.

4. How do we do Data Warehouse DevOps?

OK, this one really is difficult. Unlike most software projects, databases are (by definition) stateful. The fact that we have to be mindful of all the data that’s already in the warehouse when we deploy solutions greatly impacts how we develop and deploy our code. It is relatively easy to diff your Java or .Net code with the production application build, it is much harder to diff your DDL code with a live database schema (this is why I’m so in love with declarative database development tools like SQL Server Data Tools), and harder still to diff the actual data in the EDW.

To be honest, data warehouse teams have a lot of maturing to do when it comes to DevOps. I have seen some great progress and there are many tried and true tools and techniques that have been used in object-oriented programming and web development for many years, which we have been slower to adopt. Again, not because we’re lazy, but because some of the concepts don’t apply as cleanly to multi-terabyte data warehouse environments where historical data and queries must be kept intact.

First of all, monitoring must be built in from the start. If you can’t monitor your system, you can’t manage it either.

Secondly, we need to shift our mindset from separate “Dev” and “Ops” teams and move toward a model where the Dev team is the Ops team.

Third, we need to start adopting the tools and techniques (test-driven development, build automation, continuous integration, etc.) that have been pioneered in OOP and web development. True, this is easier said than done, and it’s difficult to do justice to the topic in just a few short paragraphs. But investment in DevOps is one of the most important thing today’s agile EDW teams can focus on to improve quality and reliability of the systems we build.

The Agile Data Warehouse was originally published in Harlan’s Data and Analytics Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.