<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Agile Lab on Medium]]></title>
        <description><![CDATA[Stories by Agile Lab on Medium]]></description>
        <link>https://medium.com/@Agile__Lab?source=rss-b8881b365117------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*34OiqQSrf6duGUye2-i-qg.jpeg</url>
            <title>Stories by Agile Lab on Medium</title>
            <link>https://medium.com/@Agile__Lab?source=rss-b8881b365117------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 10:15:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@Agile__Lab/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[My name is Data Mesh. I solve problems.]]></title>
            <link>https://medium.com/codex/my-name-is-data-mesh-i-solve-problems-264b08362bf?source=rss-b8881b365117------2</link>
            <guid isPermaLink="false">https://medium.com/p/264b08362bf</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-management-platform]]></category>
            <category><![CDATA[data-management]]></category>
            <category><![CDATA[data-mesh]]></category>
            <dc:creator><![CDATA[Agile Lab]]></dc:creator>
            <pubDate>Tue, 14 Sep 2021 14:03:15 GMT</pubDate>
            <atom:updated>2021-11-10T15:15:27.496Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2WcZmcJG8gwL82l0w-mnqQ.png" /></figure><p><em>In spite of the innumerable advantages it provides, we often feel that technology has made our work harder and more cumbersome instead of helping our productivity. This happens when technology is badly implemented, too rigid in its rules or with a logic far removed from real-world needs or even trying to help us when we don’t need it (yes, autocorrect, I’m talking to you: it is never “duck”!). But we would never go back to having to drive with a paper map unfolded on our lap in the “pre-GPS” days or to browse the yellow pages looking for a hotel, a restaurant, or a business phone number. We need technology that works and that we are happy to work with. Data Mesh, like Mr. Wolf in Quentin Tarantino’s “Pulp Fiction” solves problems, fast, effectively and without fuss. Let’s take a look to see what it means in practice.</em></p><h3><strong>Imagine that….</strong></h3><p>What’s more frustrating than needing something you know you have, but you don’t remember where it is, or if it still works? Imagine that this weekend you want to go to your lake house, where you haven’t been in a long while. There is no doubt you have the key to the house: you remember locking the door on your way out and then thinking long and hard about a safe place where to store it. It may be at home, or even more likely in the bank’s safe deposit. Checking there first, while it wields a high probability of success, means having to drive to the bank, during business hours, and the risk of wasting half a working day for nothing. You can look for the key at home in the evening and it should be faster, but could also take forever if you don’t find it and keep searching, and then if it’s not at home you’d have to go to the bank anyway. You even had a backup plan: you gave your uncle Bob a copy of the key. The problem is that you can’t remember if you also gave him a new one after you had to change the bolt in 2018…. If you did, Bob’s your uncle, also metaphorically, but are you willing to take the chance, drive all the way to the lake and maybe find out that the old key no longer works? You wanted to go to relax, but it looks like you are only getting more and more stressed out with the preparation, and now you are thinking if it’s really worth it…</p><h3>That’s a real problem</h3><p>The above scenario seems to only relate to our private lives.<br>In reality, this is one of the biggest hurdles in a corporate environment, as well, with the only difference being that at home we look for physical, “tangible” objects, while the challenge in a modern company is finding reliable data. It is not just a matter of knowing where the data is, but also if we can really trust it. In our spare time this is annoying, but we can afford this kind of uncertainty. More frequently than we care to admit, we either spend a disproportionate amount of time looking for what we have misplaced, making sure that it still works or fixing it, or we just plainly give up doing something pleasant and rewarding because we can’t be bothered to search for what we need, not knowing how long it will take. In the ever more competitive business world (whichever your business is, the competition is always tough!), avoiding extra expenses and never passing an opportunity are an absolute must. And you can’t afford these missteps “just” because you can’t find or trust your data.</p><p>Today it is highly unlikely that a company doesn’t have the data it needs for a report, a different KPI, a new Business Intelligence or Business Analytics initiative, for analysis to validate a new business proposition and so on. <strong>We seem, on the contrary, to be submerged by data and to always be struggling to manage it</strong>.<br>New questions arise. The real questions we face are therefore about the <strong>quality of data</strong>. Is it reliable? Who created that data? Does it come from within the company or has it been bought, merged after an acquisition, derived from a different origin dataset? Has it been kept up to date? Does it contain all the information I need in a single place? Is it in a format compatible with my requirements? What is the complexity (and hence the cost) of extracting the information I need from that data? Does that piece of information mean what I think it means? Is there anybody else in the company who already leveraged that dataset and, if so, how was their experience?</p><p>Ok, so there might be problems that need solving before you can use a certain data asset. That, by itself, is par for the course. You know that <strong>every new business analysis, every new BI initiative comes with its own set of hurdles</strong>. The real challenge lies in the capability of estimating these challenges beforehand, so as not to incur in budget overruns or other costly delays. It seems that every time you try to analyze the situation you can never get a definitive answer from data engineers regarding data quality or even the time needed to determine if the data is adequate. There is no way you can set a budget, both in terms of time and resources, to resolve the issues when you can’t know in advance which issues the data might or might not have, and you can’t even get an answer regarding how long it will take and how expensive it will be to find out. It is downright impossible to estimate a Time-To-Market if you don’t know either what challenges you will face, or when you will become aware of those challenges. Therefore, how can you determine if a product or service will be relevant in the marketplace, when there is no way to know how long it will take to launch it? Should you take the risk, or should the whole project be canned? <strong>This is the kind of conundrum that poor data observability and reusability, combined with ineffective data governance policies put you into</strong>. And that is a place where you really don’t want to find yourself.</p><p>What companies have just recently come to realize is that the <strong>“data integration” problem is, in general, more likely to be tackled as a social, rather than a technical one.</strong> Business units are (or at least should be) responsible for data assets, taking ownership both in the technical and functional sense. In the last decade, on the contrary, Data Warehouse and Data Lake architectures (with all their declinations) took the technical burden away from the data owners, while keeping the knowledge over such data in the hands of the originating Business Units. Unfortunately, a direct consequence has been that the <strong>central IT (or data engineering team), once they put in place the first ingestion process, “gained” ownership of such data, thus forcing a Centralized Ownership</strong>. Here the integration breaks up: potential consumers who could create value out of data must now go through the <strong>data engineering team, who doesn’t have any real business knowledge of the data they provide as ETL outcomes</strong>. This eventually ends up in the potential consumers not trusting or not being able to actually leverage the data assets and thus being unable to produce value in the chain.</p><h3>A brand new solution</h3><p>All the above implies that the time has come for not just as another architecture or technology, but rather a <strong>completely new paradigm in the data world to address and solve all these data integration (and organization) issues</strong>. That’s exactly what <strong>Data Mesh is</strong> and where it comes into play. It is, first and foremost, <strong>a new organizational and architectural pattern based on the principle of domain-driven design</strong>, a principle proven to be very successful in the field of micro-services, that now gets <strong>applied to data assets in order to manage them with business-oriented strategies and domains in mind, rather than being just application/technology-oriented.</strong></p><p>In layman’s terms it means data working for you, instead of you working to solve technical complexities (or around them). Moreover, <strong>it is both revolutionary, for the results it provides, and evolutionary, as it leverages existing technologies and is not bound to a specific underlying one.</strong> Let’s try to understand how it works, from a problem-solving perspective.</p><p>Data Mesh is now defined by 4 principles:</p><ul><li>Domain-oriented decentralized data ownership and architecture</li><li>Data as a product</li><li>Self-serve data infrastructure as a platform</li><li>Federated computational governance.</li></ul><p>We already mentioned that one of the most frequent reasons of data strategies failures is the centralized ownership model, because of its intrinsic “bottleneck shape” and inability to scale up. <strong>The adoption of Data Mesh, first of all, breaks down this model transforming it into a decentralized (domain-driven) one. Domains must be the owner of the data they know and that they provide to the company.</strong> This ownership must be both from a business-functional and a technical/technological point of view, so to allow domains to move at their own speed, with the technology they are more comfortable with, yet providing valuable and accessible outcomes to any potential data consumer.</p><h3>Trust and budget</h3><p><strong>“Data as a product” is apparently a simple, almost trivial, concept.</strong> Data is presented as if it were a product, <strong>easily discoverable, described in detail</strong> (what it is, what it contains, and so on), <strong>with public quality metrics and availability guarantee</strong> (in lieu of a product warranty). Products are more likely to be sold (reused, if we talk about data) if a trust relationship can be built with the potential consumers, for instance allowing users to write reviews/FAQs in order to help the community share their experience with that asset. <strong>The success of a data asset is driven by its accessibility</strong>, so data products must provide many and varied access opportunities to meet consumer needs (technical and functional), since the more flexibility consumers find the more likely they are to leverage such data. In the current era, data must have time travel capabilities, and must be accessible via different dynamics such as streams of events (if reasonable considering data characteristics), not just as tables of a database. But let’s not focus too much on the technical side. The truly revolutionary aspect is that now <strong>data, as a product, has a pre-determined and well-determined price.</strong> This has huge implications on the ability of budgeting and reporting a project. In a traditional system, even a modern one like a Data Lake, all data operations (from ingestion, if the data is not yet available, to retrieval and preparation so you can then have it in the format you need) are demanded to the data engineers. If they have time on their hands and can get to your request right away, that’s great for you, but not so much for the company, as it implies that there is overcapacity. This is not efficient and very, very expensive. If they are at capacity dealing with everyday operational needs and the projects already in the pipeline, any new activity has to wait indefinitely (also because of the uncertainties we discussed before) in a scenario where IT is a fixed cost for the company and therefore capacity can not be expanded on demand. Even if it can be expanded, it is nightmarish to determine how much of the added capacity is directly linked to your single new project, how much of the work can or could be reused in the future, how much would have eventually had to be done and so on. When multiple new projects start at the same time, the entire IT overhead can easily become an indirect cost, and those have a tendency to run out of control in each and every company. <strong>With Data Mesh</strong>, on the other hand, <strong>you have immediate visibility of what data is available, how much (and how well) it’s being used and how much it costs.</strong> Time and money are no longer unknown (or worse, unknowable in advance).</p><h3>A game-changer</h3><p>To understand the meaning of the next two points, “Self-serve data infrastructure as a platform” and “Federated computational governance”, let’s draw a parallel with a platform type that “changed the game” in the past. It is an over-simplification, but bear with us. Imagine you work in Logistics and you need to book a hotel for your company’s sales meeting. You have a few requirements (enough rooms available, price within budget, conference room on-premise, easy parking) and a list of “nice-to-have” (half board or restaurant so you don’t have to book a catering for lunch, shuttle to and from the airport for those who fly in, not too far from the airport of from the city center). Your company Data Lake contains all the data you need: it contains all the hotels in the city where the meeting will take place. But so did the “Yellow Pages” of yesteryear. How long does it take to find a suitable hotel using such a directory? It’s basically unknowable: if the Abbey Hotel satisfies all requirements, not too long, but good luck getting to the Zephir Hotel near the end of the list. Not only they are sorted in a pre-determined way (alphabetically), but you have to check each one in sequence by calling on the phone if you want to know availability, price and so on. Moreover, the quantity of information available for each hotel directly in the Yellow Pages is wildly inconsistent. It would be nice to be able to rule out a number of them, so you don’t have to call, but some hotels bought big spaces where they write if they have a conference room or a restaurant, other just list a phone number. If you also need to check the quality of an establishment, to avoid sending the Director of Sales to a squalid dump, the complexity grows exponentially, like the uncertainty of how long it will take to figure it out. Maybe you are lucky and find a colleague who’s already been there and can vouch for it, or you’d have to drive to the place yourself. When you budget the company sale’s meeting how do you figure out how much it will cost, in terms of time and money, just finding the right hotel? And if not having an answer wasn’t bad enough, the Sales Director, who’s paying for the convention, is fuming mad because this problem repeats itself every single year and you can never give an estimate, because the time it took you to find a hotel last time is not at all indicative of how long it will take next time.</p><p>If you now change the word “hotel” with “data” in the above example, you’ll see that it might be a tad extreme, but it is not so far-fetched. A Data Mesh is, in this sense, like a booking platform (think hotels.com or booking.com) where <strong>every data producer becomes a data owner</strong> <strong>and</strong>, just like a hotel owner, <strong>wants to be found by those who use the platform</strong>. In order to be listed, the federated governance imposes some rules. The hotel owner must list a price (cost) and availability (uptime), as well as a structured description of the hotel (dataset) that includes the address, category and so forth, and all the required metadata (is there parking? a restaurant? a pool? breakfast is included? and so on). Each of these features becomes easily visible (it is always present and shown in the same place), searchable and can act as a filter. People who stay at the hotel (use the dataset) also leave reviews, which help both other customers in choosing and the hotel owner in improving the quality of its offer or at least the accuracy of the description.</p><p>The “self-serve” aspect is two-fold. From a user perspective, it means that with such a platform the Sales department can choose and book the hotel directly, without needing (and paying for) the help of Logistics (Data Lake engineers). From an owner perspective (hotel or data owner) it means that they can independently choose and advertise what services to offer (rooms with air conditioning, Jacuzzis, butler service and so on) in order to meet and even exceed the customers’ wishes and demands. In the data world, this second aspect relates to the <strong>freedom of data producers to autonomously choose their technology path, in accordance with the federated governance approved standards</strong>.</p><p>Last, but definitely not least, <strong>the Data Mesh architecture brings to the table the ease of scalability</strong> (once you have all the Hotels/datasets in one city, the system can grow to accommodate those of other cities, as well) <strong>and reuse</strong>. Reuse means that the effort you spent in creating a solution can, at least in part, be employed (reused) to create another. Let’s stick to the hotel analogy. If you created it last year and now want to do something similar for B&amp;Bs, there is a lot you don’t have to redo from scratch. Of course, the “metadata” will be different (Bed and Breakfast don’t have conference rooms), but you can still use the same system of user feedback, the same technology to gather information on prices and availability, that once again will be up to the B&amp;B’s owner to keep up to date, and so on.</p><h3>A no brainer?</h3><p>Put like that, it seems that “going with the Data Mesh” is a no-brainer. And that can be true for large corporations, but do keep in mind that building a Data Mesh is a mammoth task. If you only have three or four hotels, it goes without saying, it doesn’t make sense to build a booking platform. What’s important to keep in mind, though, is that <strong>a Data Mesh architecture, to express its full potential, requires a deep organizational change in the company</strong>. To cite the most obvious aspect, the data engineers need to “migrate” from the center (the Data Lake) to the data producers, to guide them in the process of properly preparing the data, conforming to the federated governance rules and exposing it correctly so that it can be found and utilized (thus also generating revenue through internal sales for the data owner).<strong> It also requires a change of mentality, so that the whole company can start to visualize data as a product, data producers as data owners and break free from the limitations and bottlenecks of a Data Lake, reaping the benefits of a truly distributed architecture and the new paradigm.</strong></p><p>Luca Maestri, Chief Financial Officer of Apple, famously said that people tend to attribute the success of huge companies, the likes of Apple, Amazon, Google, or Facebook, to their being creative laboratories, where a great number of innovative ideas can emerge, but his experience taught him that this is not the case. These companies succeed because they are “execution machines”. In other words, a great idea has no value if you can not execute upon it effectively and quickly: on time and on budget. But, first, you need the right tools to be able to determine time and budget constraints. <strong>Creating a Data Mesh is a huge undertaking, but it means building the solid foundations that will support the evolution of your data-driven business.</strong> You can have all the data in the world into your Data Lake, but if you can’t leverage it, effectively and sustainably, you won’t move forward. Because in today’s world standing still means going backwards, the only way to stay competitive is to create new products, services, and solutions for your customers. In order to be an “execution machine”, you need to be able to spend your time looking for opportunities, instead of, searching for your data, analyzing the marketplace and chasing new clients and upset propositions, instead of rummaging through your Data Lake.<br>If you can do all of that, you definitely deserve a relaxing, rewarding weekend, to look at the placid lake from your house and remember the time when your Data Lake was equally still, unmoving and hard to see under the surface. Come Monday, it will be time for a new week of effective, efficient, data-driven new business.</p><p>If you are interested and would like to know more, get in touch with us and we’ll be happy to answer all your questions.</p><p><a href="http://www.agilelab.it/">www.agilelab.it</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=264b08362bf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codex/my-name-is-data-mesh-i-solve-problems-264b08362bf">My name is Data Mesh. I solve problems.</a> was originally published in <a href="https://medium.com/codex">CodeX</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Mesh explanation]]></title>
            <link>https://medium.com/agile-lab-engineering/data-mesh-explanation-a207fac61341?source=rss-b8881b365117------2</link>
            <guid isPermaLink="false">https://medium.com/p/a207fac61341</guid>
            <category><![CDATA[data-platforms]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-mesh]]></category>
            <category><![CDATA[data-driven]]></category>
            <dc:creator><![CDATA[Agile Lab]]></dc:creator>
            <pubDate>Thu, 04 Mar 2021 14:03:16 GMT</pubDate>
            <atom:updated>2021-09-03T09:03:32.406Z</atom:updated>
            <content:encoded><![CDATA[<h3>How and why successful data-driven companies are adopting Data Mesh</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SgI5bKng0afHkjDTUJUI3g.png" /></figure><p><strong>Paradigm shift</strong></p><p>Every once in a while, a new way of doing things comes along and changes everything. Sometimes this takes the form of new technologies, infrastructures, services. Some other times, the slack arises as an urgent need from the market itself. While the former needs engineering teams to push the change, the latter is very likely an “ask for help” straight from the business, and that’s the most powerful engine industry can have.</p><p>As the data-* (management, processing, governance, …) ecosystem gained its momentum in the last ten years, more and more companies invested in becoming “data-driven” after the early adopters demonstrated how much value data could generate. This wave enabled the development of all the Big Data and Cloud technologies/standards/services we very well know and use today.</p><p>This “gold rush for data” focused on opening silos and centralizing data into platforms, lakes, pushing faster and faster towards distributed architectures, hybrid clouds, even transitioning from IaaS to PaaS to SaaS. But it lacked attention to aspects like data ownership, data quality assurance, scalable governance, usability, trust, availability, and discoverability of data, which are the key factors that allow consumers to find, understand, and safely consume data to provide business value.</p><p>Data Mesh is a paradigm shift that arose as a need from “the fields” from the actual world of monolithic data lakes/platforms. It can be considered revolutionary for the results it promises and evolutionary, as it leverages existing technologies and is not bound to a specific underlying technology.</p><p>It is an organizational and architectural pattern leveraging domain-driven design, that is the capability of designing data domains that are very much business-oriented instead of being technology-oriented. We can see this paradigm shift on data as analogous to when monolithic web-services transitioned to domain-driven-designed micro-services.</p><p><strong>When Data Lake becomes Data Mess</strong></p><p>To better understand the main advantages of Data Mesh and its architectural principles, we need to take a step back and look at what was (and in most cases still is) state-of-the-art for data management before this new paradigm.</p><p>In the last years, data management’s main trend has been to create a single, centralized Data Lake (often built on-premise) to achieve both centralized data governance and a centralized processing platform. While the former proved successful, despite significant technology investments, the latter became counterproductive, both from organizational and technical points of view, for several reasons.</p><p>When creating Data Lakes, the first mantra was to open the silos, which meant setting up as soon as possible ingestion pipelines to bring data from the external systems to the data lake. The data lake’s internal data engineers team usually had the accountability to design these processes. The integration effort was undertaken from a systems point of view, i.e., let’s understand how we can take data in external systems and bring it into the data lake. This happened to occur via the broadest variety of special-purpose or generalized ETL (Extract, Transform, Load) jobs or CDC (Change Data Capture) tools. Once the integration was set up, the data ownership fell automatically in the hands of the data engineering team, who usually did not pay so much effort in first agreeing with the source systems on data documentation, data quality, etc., thus resulting in extra effort to implement checks, metrics, data quality measurements on “not-so-well-known” data.</p><p>This integration-based approach leads to even worse scenarios when something about the source system mutates: schema changes, source domain specifications evolving, GDPR introduction, you name it … It is a model that cannot scale up, especially for multinational corporations centralizing data from different branches/countries and related laws/regulations, because source systems are not aware of the process of data warehousing, they don’t know about data consumer needs, they are not focused on providing data quality on their data because it’s not their business purpose. This usually sets the scene for disengagement in creating added value for the overall organization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-9ioz3qph1Jdqr-0.png" /></figure><p>Another classical problem of Data Lake is the layered structure, where layers are typically technical (cleansing, standardization, harmonization). You can look at these layers as a fixed amount of overhead between data and the business needs that are continuously slowing down the process of value creation.</p><p><strong>Data Mesh overview</strong></p><p>Data Mesh is now defined by 4 principles (according to Zhamak Dehghani):</p><ul><li>Domain-oriented decentralized data ownership and architecture</li><li>Data as a product</li><li>Self-serve data infrastructure as a platform</li><li>Federated computational governance.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/0*DlEhINw4jzUedrfW.png" /></figure><p>To understand what is changing compared with the past, it is useful to start by changing the vocabulary. In Data Mesh, we talk more about serving than ingesting, as it is more important to discover and use data rather than extract and load it.</p><p>Every movement or copy of the data has an intrinsic cost:</p><ul><li>Development: the ETL must be developed, tested, and deployed</li><li>Maintenance: this is the worst one. You need to monitor such processes, adapt them when the sources are changing, take care of data deletion, dispersion of data ownership.</li></ul><p>Often the data movement or copy is needed for the following reasons:</p><ul><li>Technical layers</li><li>Technology needs: you have your data on S3, but SAP requires having the data in an internal table to process it. Or you have a massive dataset on Redshift, and your ML training tool is requiring data on S3.</li><li>No time travel and history capabilities: need to snapshot a data source</li></ul><p>Keep in mind that data movement/copy is not data denormalization. Denormalization is quite normal when you have multiple consumers with different needs, but this does not imply a transfer of ownership.</p><p>When you move data from a system/team to another, you transfer the ownership, and you are creating dependencies with no added value from a business perspective. Data Mesh transfers data ownership only when data is assuming a new functional/business meaning.</p><p>Data Mesh paradigm is also instrumental in “future-proofing” the company when new technologies emerge. Each source system can adopt them and create new connectors to the scaffolding template (we will go deeper on this in the next articles), thus maintaining coherence in providing access to their data for the rest of the company through Mesh Services.</p><p>Data Mesh adoption requires a very high level of automation concerning infrastructure provisioning, realizing the “so-called” self-service infrastructure. Every Data Product team should be able to autonomously provision what it needs. Even if teams are to be autonomous in technology choices and provisioning, they cannot develop their product with access to the full range of technologies that the landscape offers. A key point that makes a data mesh platform successful is the federated computational governance, which allows interoperability through global standardization. The “federated computational governance” is a federation of data product owners with the challenging task of creating rules and automating (or at least simplifying) the adherence to such regulations. What is agreed upon by the “federated computational governance” should, as much as possible, follow DevOps and Infrastructure as Code practices.</p><p>Each Data Product exposes its capabilities through a catalog by defining its input and output ports. A Data Mesh platform should nonetheless provide scaffolding to implement such input and output ports, choosing technology-agnostic standards wherever possible; this includes setting standards for analytical, as well as event-based access to data. Keep in mind that it should ease and push the internally agreed standards, but never lock product teams in technologies cages. The federated computational governance should also be very open to the change, letting the platform evolve with its users (product teams).</p><p>Data Product standardization is the foundation to allow effortless integration between data consumers and data producers. When you buy something on Amazon, you don’t need to interact with the seller to know how to purchase the product or know which characteristics the product has. Product standardization and centralized governance are what a marketplace is doing to smooth and protect the consumer experience.</p><p>If you are interested and would like to know more, get in touch with us and we’ll be happy to answer all your questions.</p><p>www.agilelab.it</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a207fac61341" width="1" height="1" alt=""><hr><p><a href="https://medium.com/agile-lab-engineering/data-mesh-explanation-a207fac61341">Data Mesh explanation</a> was originally published in <a href="https://medium.com/agile-lab-engineering">Agile Lab Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NewSQL….the new era of Relational Databases?]]></title>
            <link>https://medium.com/@Agile__Lab/newsql-the-new-era-of-relational-databases-c99abad6806a?source=rss-b8881b365117------2</link>
            <guid isPermaLink="false">https://medium.com/p/c99abad6806a</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Agile Lab]]></dc:creator>
            <pubDate>Tue, 25 Jun 2019 13:28:56 GMT</pubDate>
            <atom:updated>2019-06-26T10:14:27.529Z</atom:updated>
            <content:encoded><![CDATA[<h3>NewSQL...the new era of Relational Databases?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/425/1*z-n-PVD9RKI9u5Y6YaPN7w.png" /></figure><p>The term <strong>NewSQL</strong> represents a new generation of Relational Database Management Systems whose goal is reaching the same scalability guarantees of NoSQL databases without giving up <strong>ACID</strong> properties and the relational model.</p><h3>ACID</h3><ul><li><strong>Atomicity:</strong> A transaction should be visible only if all the updates to the datastore are applied successfully, no intermediate state should be visible to clients.</li><li><strong>Consistency:</strong> No referential constraints should be violated by a transaction, if any of those constraints is violated the whole transaction must fail, the database as a whole can evolve only from one consistent state to another.</li><li><strong>Isolation:</strong> Each transaction should be independent from other transactions running on the database, operations done inside a transaction should not have side effects on other transactions.</li><li><strong>Durability:</strong> Each successful transaction should be persisted on stable storage by the database system, data committed by a trasaction should never be lost.</li></ul><p><strong>ACID</strong> properties are easy to obtain when the database scales vertically[1] but vertical scaling is not pratical because price of a single machine does not scale linearly with performance.[2]</p><p><em>NoSql</em> databases prefer to give up <strong>ACID</strong> properties to horizontally scale, just add more machines and your performance should scale linearly in their number.</p><p>If your application needs <strong>ACID</strong> properties[3] you are stuck reimplementing those properties in your application, you could have better spent your money on a good <strong>RDBMS</strong> engine.</p><p><strong>NewSQL</strong> engines can give you the best of both worlds.</p><h3>Popular NewSQL Database Management System</h3><h3>CockroachDB</h3><p>Inspired by Google Spanner[4] <strong>CockroachDB</strong> is a <strong>NewSQL</strong> database built on a transactional and strongly-consistent key-value store (<strong>RocksDB</strong>).</p><p><strong>CockroachDB</strong> enables your use case to scale horizontally, survives disk, machine, rack, and even datacenter failures.</p><p><strong>CockroachDB</strong> supports strongly-consistent ACID transactions and provides a familiar SQL API for structuring, manipulating, and querying data.</p><p><strong>CockroachDB</strong> shards your data in ranges and, as more data comes in, splits those ranges. They are continuosly rebalanced between cluster nodes.</p><p>Horizontal scaling is achieved by assigning ranges to different cluster nodes, surviving failures is achieved by making use of replicas of the ranges, cluster split and rebalance is mediated by the raft consensus algorithm.</p><h3>NuoDB</h3><p><strong>NuoDB</strong> like CoackroachDB is a <strong>NewSQL</strong> database that achieves horizontal scale by employing a two level architecture.</p><p><strong>The transaction processing layer</strong> consists of in-memory process nodes called Transaction Engines (TE). TEs allow <strong>NuoDB</strong> to maintain high, in-memory performance. As an application makes requests of <strong>NuoDB</strong>, TEs naturally build up in-memory caches with affinity for that application’s workload. Requests for data not currently in cache (cache misses) can be fulfilled from the memory caches of other TEs or from the storage management layer.</p><p><strong>The storage management layer</strong> consists of process nodes called Storage Managers (SM), which have both in-memory as well as on-disk storage components. The SM also caches data in-memory to speed up data retrieval from disk. SMs also provide on-disk data durability guarantees. Multiple SMs can be used to increase data redundancy.</p><h3>Benchmark</h3><p>All workloads were performed using synthetic data 3 aimed at simulating those often found in the IoT world. More precisely, three tables have been created: devices, measurements and periods of validity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/0*ZU0ANt6O-Gg0qTNA.png" /></figure><p>All the benchmarks have been carried out in the same conditions:</p><ul><li>The <strong>NewSQL</strong> database has been installed on three nodes.</li><li>The queries were carried out by ten worker processes, instantiated on the same physical machine, using as many connections to the database.</li><li>For both databases, a replication factor of three was set. This means that each data point is replicated on each node.</li><li>The tables used have the following dimensions:</li><li>devices : 200 rows and 4 columns</li><li>measurements: 300,000 rows and 5 columns. With 1500 measurements for each device</li><li>validity periods: 10,000 rows and 6 columns. With 50 validity periods for each device.</li></ul><h3>Single row get</h3><p>The first workload under analysis consists of 200,000 reading operations by primary key. As a result, each query selects a single line of the table involved. The transactions were equally distributed among the three tables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wZEJ6M2ZPcFQI0ox.png" /></figure><h3>Write operations</h3><p>15,000 write operations were carried out. The number of rows inserted by each operation varies from one to five. Here again, the writes were equally distributed between the three tables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*E9JcqU3PprC1ybcI.png" /></figure><h3>Joins</h3><p>Workload composed of 75,000 read operations, which require joins among three relationships. Queries select only two lines of the table resulting from the two inner joins.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*luMbQRUx4Ng1pIzI.png" /></figure><h3>Joins 2</h3><p>This workload is similar to the one just analyzed, but it highlights the changes in the performance of the two databases as the number of selected rows increases. In this case, the operations carried out were 15,000 reads with joins involving the three tables, only 300 rows are transferred to the clients.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MQjLZyQrio7944Pe.png" /></figure><h3>Mixed workload</h3><p>The last benchmark in analysis was performed with a workload consisting of mixed operations. In particular, 100,000 queries were carried out of which 80% are reads and 20% are writes. 1% of the reads are joins on the three tables, producing two rows. The remaining read requests were all made by primary key, returning a single record as the output. Finally, the writes inserted a number of rows that varies from one to five.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NJ8MI915yUgqXCkb.png" /></figure><h3>Comparison</h3><h3>CockroachDB</h3><h4>Pros</h4><ul><li>Surely the first big advantage of <strong>CockroachDB</strong> is to be open source. On the other hand, <strong>NuoDB</strong> only provides a free Community Edition, with different limitations.</li><li>Another point in favour of <strong>CockroachDB</strong> is the presence of good documentation and a support forum, which simplify setup and product configuration.</li><li><strong>CockroachDB</strong> also offers a graphical interface presenting rich data and usage statistics. This interface has been very useful during the benchmark to monitor the database performance.</li><li>In terms of performance, <strong>CockroachDB</strong> performed better than <strong>NuoDB</strong> for read operations involving several records in the primary key scans.</li></ul><h4>Cons</h4><ul><li>In general, the performance of <strong>CockroachDB</strong> was lower than that of the of <strong>NuoDB</strong>, especially for join and write operations.</li><li><strong>CockroachDB</strong> does not offer an integrated load balancer, which is why it was necessary to use the one provided by AWS for a fee. Without load balancer, it would not have been possible to distribute the load equally between the available nodes.</li></ul><h3>NuoDB</h3><h4>Pros</h4><ul><li><strong>NuoDB</strong> provides on average better performance than <strong>CockroachDB</strong>, especially for join and write operations.</li><li>The presence of an integrated load balancer, configurable with different policies for load distribution, is another great advantage offered by <strong>NuoDB</strong>.</li></ul><h4>Cons</h4><ul><li>One of the great flaws of <strong>NuoDB</strong> is that it is not open source. The Community Edition, provided free of charge, also has too many limitations and is therefore not pratical to use in production. In particular, this version allows you to have only three transaction engines and a single storage manager. The second limitation results in the impossibility to have the replication of the data resulting in no fault tolerance. All the benchmarks reported above were carried out with the Enterprise Edition of <strong>NuoDB.</strong></li><li>The documentation provided by <strong>NuoDB</strong> was often confusing and sometimes incomplete. This has influenced the setup times in a negative way, no public support forum is available.</li><li><strong>NuoDB</strong>, unlike <strong>CockroachDB</strong>, does not provide a graphical interface for monitoring the database. This lack, although not serious, has made observing the performance during benchmarks harder.</li><li><strong>NuoDB</strong> performance decreases when the number of selected lines in the scanning process grows.</li></ul><p>By Andrea Fonti (Big Data Engineer, Agile Lab) — www.agilelab.it</p><p>— — — — — — — — — — — — — — -</p><p>[1]To scale you have to buy one machine with better performance.</p><p>[2]Machines capable of the needed performance are not available yet.</p><p>[3]Applications do not often really need ACID properties.</p><p>[4]But not requiring hardware-assisted clock synchronization using GPS clocks and atomic clocks to ensure global consistency.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c99abad6806a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>