Centralised Approach for Data Platforms Architecture
About the Centralised Data Systems: Navigating the Complexities, Seizing the Advantages
Introduction
There are many reasons why companies decide to accommodate the centralised paradigm while designing and building their data platforms. It might include lessons learned from decentralisation, urge to maximise delivered value while minimising the complexity and cost, and many more…
Regardless of the initial motivation, the companies which decide to pursue centralised approach, gather and own data in a monolithic platforms. Such platforms tame the data from different domains in a single, unitary place. Throughout the concentration of not only data, but also policies, processes, security practices, or ownership, centralised platforms bring the promise of making data easy to understand, easy to access and more qualitative.
Centralised platforms are typically built and maintained by team of engineers with the deep technical expertise in big data tooling and frameworks. These people, having strong IT background, often lack business and domain knowledge.
When centralised platforms serve a few domains, and only several use-cases needs to be supported, then the team in charge of data might meet the demands, of providing valuable insight at low cost, relatively well. However as the number of demands from the business grow, and thus the amount of potential use cases to be supported also grow, the significant challenges appear.
What are the challenges, then?
The Challenges:
- Ever-growing backlog. The amount of different requests (i.e to build, modify, prototype the data asset) from the business grows with new data use cases. This, being combined with necessity to maintain already built data solutions make the backlog bigger and bigger.
- Never ending firefighting. Most data use cases involves processing data on daily basis. These data, being a by-product*[1] provided either by teams working in operational data plane*[2] or external data providers do not have product-grade*[3] quality. As a consequence, the errors and failures*[4] in processing occurs. This leads to either low quality data provided to its customer or no data provided at all.
- Engineer’s wide scope of responsibility. In order to build and maintain the data solution*[5] throughout the entire lifecycle the team who owns the data platform need to have deep expertise in many IT areas (i.e infrastructure, security, networking, big data tooling).
- Lack of business knowledge. The owners of centralised platform — teams composed of IT experts — have strong technical background. In order to acquire these skills, they had to obtain STEM degree or spend many years of their career in tech departments. By doing so, they didn’t had a chance to develop business competences. As a result, the data assets, which are being processes for further by its customers, are either poor quality (could be both technical*[6] or business), delayed or not delivered at all.
- Big Data Tools, Infrastructure & Domain Transformations Coupling. Necessity to work in rush, poor business knowledge, wide scope of responsibilities, all of these forces the engineers to trade offs. It results in lack or lowering the code quality standards and coupling between tools and business transformations.
- Data Siloing. The modern big data tools, frameworks and formats require specialised knowledge to be used. Each of these things, combined with harsh security practices makes the data not accessible for business stakeholders who do not have technical expertise.
The factors that have been mentioned above lead to lack of agility, increased sensitivity to change and finally high cost ofmaintenance and development. These might have consequences in not only limited ability to experiment and deliver innovative solutions but also inability to deliver data driven value for the enterprise.
So far, The challenges of centralised platforms have been analysed. These are significant!
What are the benefits of this approach, then?
The Benefits:
- Unified view of Data — aka. single source of truth. As it brings the promise of having good quality data with well designed structure, documented, easy to find and use, etc.
- Centre of excellence — the keynote in such situation is: “Let the experts do their work”, therefore as a result, marketing people are gathered in marketing departments, layers in legal departments, and IT people (of all kind Software Engineers, DevOps, Infrastructure, Security, etc) work in centralised IT departments. Additionally, such organisation of data team guarantees unification of security, governance and tooling across across the enterprise.
- Easy to execute Governance. Data Governance has different shades*[7]. Overlooking the details, the centralised platforms are easier to govern. This might affect the data quality, its security and accessibility. Therefore, many companies give in the temptation to build governance at low cost.
- Experience and expertise available on the market. Centralised paradigm has been followed by many companies in last decades. Hence, almost every experienced data professional has high competences in this area. People working on position like Head of Data or Chief Architect have years of experience working in centralised platform. They know what kind of problems they can expect and how to make them less impactful*[8]
- Low cost — compared to alternatives — decentralised approach or federated one. The centralised paradigm significantly cheaper to build and maintain.
Summary
Centralised approach might be a good way for businesses with single and simple business line. The one without need for innovation and agile experimentation. Once the complexity of business grows, the convolutedness of data solutions increases exponentially. Accordingly, the benefits fade away and challenges become more and more visible.
Get Wojciech Nowak’s stories in your inbox
Join Medium for free to get updates from this writer.
How to choose the proper data architecture, then?
First of all, we shouldn’t reject the centralised or decentralised approach. There are companies, typically small ones, for which decentralised approach might be an only option — it might be caused by lower costs. Being aware of the challenges which comes with decentralisation, such companies might address the incoming issues in advance. For example, the tools proliferation thread might be solved by selection of one tool for entire enterprise and the proper training program for employees.
Similarly, the bottlenecks of centralised platforms might be overcome earlier. For example, the data siloing issue can be dealt with adding data virtualisation*[9] layer between data assets stored within the platform (regardless if data warehouse or data lake) and its consumers.
Finally, the federated solutions like Data Mesh or Data Fabric should have been considered. The latter ones brings the promise to break down the monolithic platforms into distributed, interconnected and easy to work with data assets.
The promise of untangling the issues of both the centralised and decentralised data platforms is huge. Some experienced professionals claims it’s impossible.
But.. what if the world which lacks data driven value, the value that is crafted in pain throughout time consuming process would disappear. Wouldn’t it be perfect?
Do u want to know how to make that happen?
Follow me on Linkedin, Medium or even drop a PM if you want to know more ;)
Keep moving!
WN.
*[1] Data as a By-Product — typically, data pushed away by IT team which provides core capabilities for the business. Teams, enhancing core business capabilities, often do not care about the quality of data, its structure and correctness. Nobody never told them to take care the data for analytics purposes, so they don’t do it. It’s commonly assumed that its the role of data team to handle data for analytical purposes. The problem with such approach is that often data quality can be improved only directly at the source.
*[2] Operational data plane — Data which describe everyday events in the business. Typically transactional data. Handled by the systems supporting the core business and everyday operations.
*[3] Product-grade quality — it’s about treating a data or a software as a living entity that evolves over time. It puts strong emphasis on user needs, continuous improvement, and alignment with long-term business objectives. — Read more in: INSPIRED: How to create tech products customers love
*[4] Errors & Failures — 2 Scenarios can be differentiated. First one — when data didn’t arrive to customer, or he received completely wrong data. Second one — when customer received the data, but not entirely correct (i.e only 20% of usual data feed, data with a lot of null records or data with changed schema)
*[5] Data Solution — both Data Asset and Data Pipeline. The output data ready to be consumed and transformation logic, configuration and pieces of infrastructure needed to create an asset.
*[6] Technical Quality Checks — It’s set of procedures that are performed to ensure technical integrity and correctness of data. These kind of checks is focused on format, structure and consistency of data, rather than business relevance. It might include: format validity, duplicates detection, data type and range checks, data completeness, etc.
*[7] Data Governance (and its shades) — the set of processes, standards and policies that refers to the management of the data usability, but also its security, integrity and availability within the enterprise. — Detailed article about this topic will appear on this blog in the future. If you are interested — follow an author or drop a message :)
*[8] Make the centralisation challenges less impactful — Some issues are integral part of centralised approach. It means that these will always occur while adapting such approach. Being aware of that, certain actions can be taken to suppress their impact. For example alerting mechanism can be implemented to notify the development team about an error while processing data. Then, the IT team, instead waiting for business to report an issue, would be able to identify an error and fix it faster.
*[9] Data Virtualisation — approach for data management. It provides data assets to its consumers in a unified and abstracted way, without requiring technical details about the data, such as how it is formatted or where it is physically located.
