<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tom Baeyens on Medium]]></title>
        <description><![CDATA[Stories by Tom Baeyens on Medium]]></description>
        <link>https://medium.com/@tombaeyens?source=rss-91de644a221a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*xK7VMJiTkR--m9nZDVMtSA.jpeg</url>
            <title>Stories by Tom Baeyens on Medium</title>
            <link>https://medium.com/@tombaeyens?source=rss-91de644a221a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 11:26:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tombaeyens/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Data contracts as the API for data]]></title>
            <link>https://medium.com/@tombaeyens/data-contracts-as-the-api-for-data-6f2859da10c2?source=rss-91de644a221a------2</link>
            <guid isPermaLink="false">https://medium.com/p/6f2859da10c2</guid>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-contract]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[Tom Baeyens]]></dc:creator>
            <pubDate>Thu, 01 Feb 2024 12:54:38 GMT</pubDate>
            <atom:updated>2024-02-01T14:43:58.478Z</atom:updated>
            <content:encoded><![CDATA[<p>In the previous blog episode <a href="https://medium.com/@tombaeyens/5-reasons-to-start-with-data-contracts-35344a051df6">5 reasons to start with data contracts</a>, we covered why starting with data contracts is a good idea. We already touched on the subject of data contracts being the API for data. Here we go deeper and explore what that means. By comparing practices in data with software engineering, we’ll see how contracts lead to a solid preventative approach.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w1ockiHZK9flKXBxop-TuA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@polarmermaid?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Anne Nygård</a> on <a href="https://unsplash.com/photos/claw-hammer-and-nails-yYRAvcPrWms?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><h3>Data contracts enable encapsulation</h3><p>Encapsulation is a well known software engineering technique that is around for a long time, but not yet commonly applied in the data space. However, encapsulation can solve a number of common problems that we encounter when building data pipelines.</p><p>Everyone knows the pain of spaghetti pipelines: No one wants to take ownership. Problems make it to the end-users of data. Pipelines are hard to debug. Crucial knowledge is stored in the heads of crucial people so when they leave or change teams it’s a big problem. This is not scalable.</p><p>Encapsulation is used to break down a larger software system into smaller components. Each component has an interface and it hides the internal complexities. This way, larger systems can be built more efficiently. Also, in a system with components &amp; interfaces, it’s easier to find bugs and the maintenance of the system is more predictable.</p><p>If you have a hard time finding the components and interfaces in a data pipeline, focus on the datasets that are a handover between 2 software components. Ingestion components are components that extract information from a source system and produce persisted datasets. That persisted dataset is then used by the next software component like perhaps a dbt transformation.</p><p>That dataset that is the output of your ingestion component, and that is exposed with SQL to the dbt transformation is an interface. Those are the ones that deserve to be treated as an interface between software systems. With documentation, monitoring and stability guarantees.</p><p>Just like REST, GraphQL and protobuf are styles of interfaces, making a dataset available over SQL is also an actual style of software interface. Data is accessed over a connection, even though that protocol is database dependent. Then there is a SQL interface on top, which enables users to specify the data that is wanted. In this context, a data contract is actually a full specification of a software API.</p><h3>Contracts as API documentation</h3><p>In software services, publishing documentation on a service API is a very common expectation. In data teams we do not see a lot of documentation of the data exposed in source systems as well as intermediate results.</p><p>The main source of documentation for data can typically be found in a data discovery tool or a catalog, but that often does not qualify as API documentation. Those tools start from the metadata fetched in bulk from the warehouse. Discovery tools can fetch metadata from a warehouse, but in that case you don’t know if someone takes on all the responsibilities for the dataset and provides for instance stability and other guarantees you expect as a consumer of that data. You may be finding an internal, intermediate dataset that will be changed next week when the engineer updates their pipeline implementation.</p><p>Also, what data discovery tools cannot scrape from the database’s metadata are things like semantics and guarantees on freshness. Those are important however, when using data.</p><p>Data contracts provide 2 things to improve documentation for data made available to others:</p><ul><li>Ownership: a contract is maintained and promised by the producer. So there is a team that takes ownership of the data.</li><li>A contract should include all technical details necessary to start consuming the data.</li></ul><h3>An API implies stability</h3><p>In software services, stability is a very common expectation. There is a major, minor and micro version system that expresses what kind of stability is expected with new versions. Consumers (or API-clients) expect that stuff doesn’t just disappear and that they get notifications ahead of time like deprecation warnings that enable consumers to update their systems.</p><p>In a typical data environment, there are many datasets, some act as software interfaces between components in the pipeline and some are mere internal implementation artifacts. That happens without anyone being aware. In that environment, it’s hard to predict everything and when a change is made, then inevitably something will introduce a breaking change. This is very similar to the situation where we built monoliths without unit testing back in the days. The result was that teams got more and more afraid to release.</p><p>Identifying interface datasets is the first step. As you identify a dataset that is used by another component, then you have to take into account stability guarantees. First, you will need to consider what your stability guarantees are. And you need a way for the downstream consumers to get notified when incompatible changes are made. Like considering a time period for deprecation of columns, for instance. Or alternatively, creating a new version of the dataset in case of incompatible changes.</p><p>These are very similar stability considerations as software engineers have to make when evolving their REST- and other APIs.</p><h3>Pushing the hot potato to the source</h3><p>As you start to adopt contracts, we at Soda recommend that you start creating and verifying contracts at consumption all the way at the end of the pipeline. Consider that an integration test.</p><p>However, this poses a problem. When asking the producers of the refined datasets to take ownership of those datasets and contracts, they may see it as a hot potato. This is a critical moment. Don’t stop right there. They cannot provide all these stability and documentation guarantees if they don’t have the same guarantees on their input data.</p><p>Recognize that in a transition period, it will be tough and that not all those guarantees will be expected from the producer of the consumable data. But instead they should be pushing for contracts on their input datasets and the central data team should help here. When all inputs are covered with contracts as well, then the producer should be able to take full ownership seriously with all the guarantees on stability, documentation and monitoring.</p><p>If you push this all the way upstream, you may arrive at what is often a gaping hole at source: ‘the production back door’. When analytical data is needed from production systems, the quickest way to get it is going straight to the (internal!) database tables. The team responsible for the production app is not even aware. Moreover, the team is avoided if possible to prevent any objections 🤯. It’s a quick way to get to the data for sure, but not a stable one.</p><p>What in those situations is not always recognized is that the production app team * should * be involved and whatever mechanism they provide to extract the data, because they * must * take ownership. Lacking that, the rest of the analytical pipelines will be based on quicksand. Contracts will push for those conversations to be happening after the facts. Those are not going to be easy, but very necessary.</p><p>What we see in practice is that the analytical teams first create the contracts for the source data and implement verification and circuit breakers for their own protection. Then the production app people are asked to take ownership of the datasets. It can be negotiated how much stability those teams need to provide. Our recommendation is to be lenient on the guarantees in those situations and make sure they are involved and take on the ownership over the data and start managing the contract. If they proactively start to communicate incompatible changes they make to their database, it will be a big step forward from not knowing at all.</p><h3>Conclusions</h3><p>While the scope of our products is broader, <a href="https://soda.io/">at Soda</a> we believe that contracts are starting a paradigm shift in data engineering. We think it’s comparable to how test driven development has brought back the confidence to release software continuously. Contracts will bring back the trust in using data products.</p><p>Data pipelines consist of a series of extractions and transformations that each can be seen as software components talking to each other over an API. That API is typically querying a table. In that sense, a data contract is nothing more than an API specification. 50 years of software development has learned that apart from an API description, we also need ownership, documentation and stability guarantees as part of that agreement. Only when we treat these data transformation components as software with APIs that depend on each other, can we start applying the lessons learned in software engineering to the data stack.</p><p>Bringing contracts in place between all data that is exchanged between the systems of an analytical data stack will significantly reduce the instability. It therefore will decrease the need for diagnostic tools, but of course, not remove it. Without contracts you will have far more issues to diagnose. Also diagnosing will be a lot harder if you don’t understand how the data should look like half way through in your pipelines.</p><p>Data contracts and ownership are needed to scale data teams more effectively and achieve data that is more trustworthy.</p><p>To learn more about Soda data contracts, check out <a href="https://docs.soda.io/soda/data-contracts.html">https://docs.soda.io/soda/data-contracts.html</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6f2859da10c2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 reasons to start with data contracts]]></title>
            <link>https://medium.com/@tombaeyens/5-reasons-to-start-with-data-contracts-35344a051df6?source=rss-91de644a221a------2</link>
            <guid isPermaLink="false">https://medium.com/p/35344a051df6</guid>
            <category><![CDATA[data-contract]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-quality]]></category>
            <dc:creator><![CDATA[Tom Baeyens]]></dc:creator>
            <pubDate>Wed, 03 Jan 2024 07:21:13 GMT</pubDate>
            <atom:updated>2024-01-03T07:21:13.881Z</atom:updated>
            <content:encoded><![CDATA[<p>Data contracts are starting to make a splash in the data engineering community, and for good reason! They’re long overdue. All too often data teams struggle to scale their work as data products break and data issues regularly occur. After building data quality solutions for more than 5 years, I must share my excitement on the power of data contracts. I truly believe this will drive a major shift in data engineering.</p><p>Contracts are the API for data. It’s the packaging that we’ve been missing in data engineering to achieve encapsulation in data pipelines. Lack of such an interface has put a cap on the scaling and productivity of data teams. It’s more than a technology. It’s an architectural mindset to think in terms of interfaces. That thinking is mostly missing in data engineering and the trend towards data contracts is going to bring more scalable practices into data teams. Let’s look at the most 5 most important benefits that can be achieved by adopting data contracts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0bV3xIUv58qSU3JK" /><figcaption>Source: <a href="https://unsplash.com/photos/two-human-hands-painting-k39RGHmLoV8">Unsplash</a></figcaption></figure><p>I’ll be using the term ‘dataset’ for tabular data like a table. A ‘data component’ refers to a software component in the data pipelines that has one or more datasets as output. Examples of data components are an extraction or a transformation. ‘Data applications’ are applications like reports or machine learning models that consume analytical data. While this deviates a bit from data mesh terminology, I think it helps to better align the vocabulary with general software engineering concepts.</p><h3>Clarify ownership of data</h3><p>Analytical data is often a monolithic web of spaghetti pipelines, a situation that doesn’t scale well. The symptoms of pipeline spaghetti include data quality issues and a lack of trust in data. When a potential data issue occurs and there are no clear owners defined, the first challenge is to dig through the spaghetti and find who is able to diagnose and address the problem.</p><p>Many organizations struggle to establish ownership for their analytical datasets. This is understandable because often, the team responsible for the data is small and both the team and its rules for governing data grew organically, and potentially a little haphazardly. But as teams start to build more data components and team members shift around, it can become hard to find the right people to deal with issues or new requirements when they arise.</p><p>Datasets without an owner are unstable and destined to break downstream consumer assets. As a consumer, you ideally want to use only those datasets which have an owner, an actual human you can contact in case of issues or questions.</p><p>Data contracts create a separation between the implementation of continuously updating a dataset (read: the pipelines) and the resulting data that is made available as a dataset to all consumers. It’s the team that is responsible for building and operating the pipelines that must take ownership of the resulting dataset. Because data contracts help to separate implementation from interface, they help to clarify what ownership of a dataset means.</p><p>The act of creating the contract is the starting point of taking ownership. It’s a declaration saying this is the data we promise to deliver and will keep up to date, whatever happens upstream or however we want to build our implementation. Taking full ownership of a dataset includes</p><ul><li>Create and publish a data contract will enable consumers to start consuming your data as much as possible in a self serve manner without the need for asking questions</li><li>Be transparent on stability guarantees</li><li>Establish a channel for consumers to ask question, change requests or new feature requests</li><li>Build, deploy and monitor the production implementation</li><li>Set up testing for new releases of the implementation</li><li>Monitor data quality</li></ul><p>In short, introducing data contracts will start with the question: Who should create and maintain the contract? In that sense data contracts help drive the notion of ownership and the architectural pattern of encapsulation.</p><h3>Breaking down monolithic data pipelines</h3><p>In the context of monolithic, spaghetti data pipelines, it’s hard to take ownership of a dataset. That’s because the input data for a particular transformation can be undocumented and there are no guarantees of stability. In that case it’s hard to promise anything on the data that you produce.</p><p>But if all the inputs of your transformation all get a data contract, this changes. Adding clarity and guarantees on all the inputs in the form of a data contract, it becomes much easier to oversee the responsibility of taking ownership. An engineer building a single transformation is now able to provide the same clarity and guarantees on the output data. This divide-and-conquer strategy helps to break down the monolithic spaghetti pipelines.</p><p>Each step in extraction and transformation of analytical data can, in fact, be considered an individual system or component. Having clear interfaces between all these components is simply good engineering practice, and it’s a great way to build a robust data architecture. The data team will be more predictable in their output and more scalable in the amount of work they can handle, just like software engineering teams when they transitioned from spaghetti code to object oriented programming.</p><p>The value of breaking down the monolith pipelines in manageable data components increases when considering the diagnosing of potential data issues. Without contracts engineers have to traverse the pipeline upstream without a clear map of what the data should look like. They also can be wasting time on a number of internal datasets that are merely distraction. Data contracts separate internal datasets from the handovers between different data components. And for those datasets the contract is a clear description of how the data should look like. In that sense, contracts help to speed up the diagnostic workflows for engineers significantly.</p><p>However coarse or fine grained your data components are is not really a big factor. If you start from a single, huge monolithic spaghetti pipeline, it will already help to create contracts for the datasets made available to consumers sometimes referred to as ‘refined’ or ‘gold’. That way consumers building data applications can see if their expectations match with the promises made.</p><p>Next, as early as practically possible at the source is also a good place to start introducing contracts. Afterwards you can go finer and identify intermediate datasets for which to define contracts. Contracts allow for a gradual introduction of contracts as interfaces. In that sense, contracts are a lot easier to retrofit compared to API’s on software components. That makes it possible to gradually move from bigger data components to smaller ones.</p><p>The right size for a data component is hard to tell, just like it’s very hard to put the right size on a software component or service with a Rest API.</p><h3>Monitor data quality</h3><p>When analysts build new analytical data applications, it’s not often easy for them to find a clear description of what the data looks like. This leads to the well known approach: ‘Let’s try and if it works we assume it’s all good’. Contracts include a language to express any expectation that a consumer may have on data. That’s data quality.</p><p>The basic data quality expectations are schema, missing data and validity constraints. More advanced data quality assumptions are distribution, average and other aggregations, referential integrity etc.</p><p>Contracts provide a way to describe what the data looks like in more detail than the typical analytical warehouse storage system. Each time new data is produced, the new data can be validated against all the data quality assertions in the contract. This is also referred to as ‘enforcement’, but I consider verification to be a more suitable term. Ideally verification is done with circuit breakers so that no bad data will ever reach data applications. The result is that contracts offer a richer language to express assumptions on data quality that can be verified in production.</p><p>There are many people outside the team producing the data that have domain knowledge of what good data looks like. As part of an overall data quality strategy, they should be allowed to contribute to the picture of what good data looks like. Dataset owners or data producer teams have to set up communication channels so that anyone in the organization can suggest new or data quality checks or changes. The data producer teams have to translate them into the contract and ensure it’s part of the monitoring process.</p><p>Contracts help with monitoring data quality because</p><ul><li>There is a rich language to describe data quality assertions</li><li>Data producer teams (read: owners) are responsible for contract verification, including the data quality assertions</li><li>Consumers can help by proposing data quality expectations as extra data quality checks to be added to the contract</li></ul><h3>Increase data transparency</h3><p>These days, data discovery tools are typically connected to the metadata of a warehouse. This poses a couple of challenges:</p><ol><li><em>All</em> datasets are discovered, even the internal datasets that are not intended for use by other teams.</li><li>The warehouse metadata is not very detailed or unavailable. For example, simple checks like “The average amount should be between 20 and 50.” are not shared nor verified.</li></ol><p>We saw that a data contract includes a more detailed description of what the data looks like and that these details are also verified as new data gets produced.</p><p>As such, data contracts are the perfect source to feed data discovery tools like catalogs. Those tools are the place where people want to find and understand data. Contracts contain more rich metadata of what the data looks like compared to the basic metadata found in analytical warehouses. On top, the presence of a contract also separates the datasets intended for consumption from the internal, intermediate datasets. That is something which is not captured when data discovery tools just scrape the warehouse metadata. We foresee that data discovery tools will migrate away from ingesting the warehouse metadata directly towards ingesting the contracts.</p><p>Plus, data contracts are files that you can manage in git; you can install webhooks so that each change in a production branch of a contract gets pushed to a data discovery tool. That way, data discovery tools contain a more complete, up-to-date, and trustworthy picture of what the data looks like.</p><h3>Build immutable data infrastructure</h3><p>Making available rich metadata to consumers and monitoring the data quality are 2 key ingredients of a data platform. Because data contracts are text files, they can be managed as code and versioned in git. When using data contracts, the rich metadata in contracts and the data quality monitoring become part of the immutable data infrastructure. This makes it easy to duplicate environments for CI/CD and disaster recovery, ensuring that data contracts and data quality are seamlessly integrated as a part of engineering workflows.</p><p>Next to data quality monitoring and storing the metadata for discovery tools, more tools can use the contract as the authoring place for their configurations. Consider PII, access management or retention. Because the structure of data contracts is based on the schema, it’s a natural extension to include those aspects as well.</p><h3>Conclusion</h3><p>Data contracts are an important tool to help organizations deal with ownership and break down monolithic spaghetti pipelines. They form the basis for improving and monitoring data quality. When connected to discovery tools, data contracts provide transparency on which data is intended for sharing, and they include a detailed picture of what good-quality data looks like. Data contracts fit within the engineering workflows because they help establish an immutable data infrastructure and they create better alignment between the analytical data teams and the rest of the business.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=35344a051df6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introducing Soda Data Contracts]]></title>
            <link>https://medium.com/@tombaeyens/introducing-soda-data-contracts-ac752f38d406?source=rss-91de644a221a------2</link>
            <guid isPermaLink="false">https://medium.com/p/ac752f38d406</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-contract]]></category>
            <category><![CDATA[data-quality]]></category>
            <dc:creator><![CDATA[Tom Baeyens]]></dc:creator>
            <pubDate>Fri, 24 Nov 2023 10:17:24 GMT</pubDate>
            <atom:updated>2023-11-24T10:17:24.502Z</atom:updated>
            <content:encoded><![CDATA[<h3>Why Data Contracts?</h3><p>Data contracts are a <a href="https://medium.com/@tombaeyens/data-contracts-the-missing-foundation-3c7a98544d2a">foundation</a> for data producers to take responsibility over the datasets they own. In a data contract, they communicate the “API for data”. Using the contract, data consumers can find out what data is available and how it can be used.</p><p>The focus on this “API for data” was triggered by the Data Mesh movement started by <a href="https://www.amazon.com/Data-Mesh-Delivering-Data-Driven-Value/dp/1492092398">Zhamak Dehghani</a>. Data Mesh is a great strategy for architecting data products at scale. It applies software engineering principles such as Domain-driven Design (DDD) and micro services to data.</p><p>Without the ideas of Data Mesh and a concrete implementation of data contracts, there is a tendency towards building spaghetti pipelines that no one can take ownership of, and no one dares to touch. Introducing data contracts is an implicit enabler to establish ownership. If data producers can rely on their input (because of trustworthy contracts), they can, in turn, product reliable contracts for all of the data products they produce.</p><p>For contracts to be effective and not drift away from reality, they must be enforced. Enforcing a contract means continuously verifying that new data matches the data contract each time new data is produced. That’s where Soda comes in. Consider Soda as the enforcement engine for data contracts. Soda will continuously verify that your contract is up-to-date and correct.</p><p>Using a YAML language as the foundation for contracts fits with the shift-left motion that is going on in data. It means driving the data management processes as much as possible from engineering artefacts that are managed by engineers, to the source (a.k.a. upstream). This enables faster detection of problems and more options to quarantine bad data before it causes harm to data applications.</p><h3>Use Cases for Data Contracts</h3><p>Data contracts serve many important use cases.</p><ul><li>The API for data: Consumers that are searching for data with which to build new products need to understand what data is available and how it can be used. In that sense, data contracts become the system of record for operational metadata that can be surfaced in data catalogs.</li><li>Protecting storage: Databases have schemas to prevent bad data from entering storage. Kafka has schema registries to prevent bad data from entering topics. In the same way, analytical datasets will get contracts to prevent bad data from being stored.</li><li>Protecting usage: Consumers can add (or let others add) data quality checks that are relevant to their usage. Users often have much more intimate knowledge about the data or about specific requirements. Letting them add data quality checks enables them to protect their usage.</li></ul><h3>Soda’s Data Contracts Strategy</h3><p>At Soda, we have a history of building an execution engine for data quality checks. This started with the release of Soda SQL in 2021 and Soda Core in 2022 (Apache 2.0). This OSS engine is the perfect basis for data contract enforcement.</p><p>Today, there are already multiple variations of data contract languages used and in the making. In order to reach as many users as possible, we decided to make it very easy to become the data quality enforcement engine of choice for all those contract language variations.</p><p>We focused on simplicity and ease-of-adoption. New to data contracts? Just get started with this release as a standalone tool. Already using another data contract specification? You can plug-and-play Soda as we ignore any other contract section. Already using Soda Core/Library? Data contracts only adds a new capability and it interoperates nicely with existing Soda deployments.</p><p>Soda remains committed to providing open source software to the data community. We believe that the best way for the community to learn, iterate, and evolve is to get started in OSS.</p><h3>How Does It Work?</h3><p>Let’s start by taking a look at an example Soda contract:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aZs_lUtPWyAtE8PF.png" /><figcaption>Example Soda Data Contract</figcaption></figure><p>Note that the columns section is specifying the full schema. The schema of the dataset must match the specified schema exactly, which ensures that the contract is always a valid and complete representation of the dataset. Also, note how the checks section exposes the full power of <a href="https://docs.soda.io/soda-cl/soda-cl-overview.html">SodaCL</a> enabling users to add any data quality check.</p><p>In order to execute or “enforce” a contract, please refer to the <a href="https://docs.soda.io/soda/data-contracts.html">docs</a> on how contract files can be executed programatically.</p><h3>Deployment Considerations</h3><p>We recommend that the Contract YAML file is colocated in the same git repository as the software producing the new data for that dataset. This implies that after the data is produced, the contract (from the same repo version) is verified. It is easier to manage evolution this way.</p><p>For instance, in CI/CD, you can try out new versions of the contract without any versioning of the contract other than git. This reduces the need for a contract registry and all the hassle that comes with matching contract enforcement with the right contract version. It also reduces the need to have constructs for upcoming and deprecated columns.</p><p>Very similar to software APIs, backwards compatibility is really important. As a data producer, you cannot just release new versions of a dataset that are not compatible with previous versions. Adding columns is okay, but removing columns or changing the data types is very risky and will break your consumer’s data products. Having a contract will force the producer to document the schema and other dataset properties. Storing this in git will create a version history of the contract. The history in git will help make these changes visible. Even tooling can be envisioned in which, upon a commit that has a change, a contract will check if the change is considered backwards compatible or not.</p><p>In this usage of contracts where pipeline code and contracts are in the same repo, publishing the contract has to be done when a new version of the software is being released. Usually in case of CI/CD, that is triggered when a branch is merged to main, and, in case of changes to contract files, ensure that the new version of the contract is pushed to all other systems that depend on it.</p><h3>What’s Next?</h3><p>As our very first, experimental iteration of data contracts, we aim to explore how our early adopters will adopt it in the context of their own data products. We’re extremely eager for feedback and, indeed, would be happy to arrange an online session so we can dig into how data contract management will work in real life. Join us on Soda Community and share your story, question, or request in the <a href="https://community.soda.io/slack">#soda-core-channel</a>!</p><p>In the next few weeks, we will revamp the current implementation of data contracts in Soda Cloud (called Agreements) to be more focused on UI based check creation and its integration into this data contract capability.</p><h3>Where to Learn More</h3><ul><li>To download and play with the release, check out the <a href="https://docs.soda.io/soda/data-contracts.html">docs</a></li><li>Talk to us on our <a href="https://community.soda.io/slack">Soda community on Slack</a> in the #soda-core channel</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ac752f38d406" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data contracts: The missing foundation]]></title>
            <link>https://medium.com/@tombaeyens/data-contracts-the-missing-foundation-3c7a98544d2a?source=rss-91de644a221a------2</link>
            <guid isPermaLink="false">https://medium.com/p/3c7a98544d2a</guid>
            <category><![CDATA[data-engineer]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data-contract]]></category>
            <category><![CDATA[data-observability]]></category>
            <category><![CDATA[data-management]]></category>
            <dc:creator><![CDATA[Tom Baeyens]]></dc:creator>
            <pubDate>Fri, 24 Mar 2023 15:54:33 GMT</pubDate>
            <atom:updated>2023-03-29T18:50:44.472Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AIzK84xK7kzFUMXO6MNTJQ.png" /><figcaption>Photo by <a href="https://unsplash.com/@kellysikkema?utm_source=medium&amp;utm_medium=referral">Kelly Sikkema</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This article will show how data contracts solve some of the crucial problems that data teams struggle with today, how they can be introduced without friction in existing data infrastructures and how this new foundation simplifies integration of the many tools in the data stack.</p><h3>Problem #1: No-one seems to care about encapsulation in data</h3><p>In software engineering, encapsulation is the proven mechanism to divide and conquer larger software systems, and it goes without saying that without encapsulation you get unmaintainable spaghetti code. Now whilst there are some major differences between software engineering and data engineering, they’re similar enough that many of the best practices that originated for software engineering are extremely relevant and beneficial for data engineering. In data engineering, I believe <a href="https://github.com/AltimateAI/awesome-data-contracts">data contracts</a> are the missing link to introduce encapsulation and create more transparency between data producers and data consumers. In this article, I aim to discuss the value of data contracts, address the key reservations that people have expressed, and project how contracts will drive a profound change in how we manage and configure tools in the data stack.</p><p>Data contracts are the missing component in an organization’s data architecture. To create high-quality, reliable data systems, you need to be able to test contracts and observe data pipelines. Data observability is a reactive approach that enables data engineers to detect and diagnose data issues once they have occurred. Data contracts are a pro-active, preventative approach that focus on reducing the number of data issues as early as possible, before they can take place. Because data issues that don’t happen, won’t need to be diagnosed. Now that’s what I would call a big win. Pieces are starting to fit together for contracts to become foundational for the analytical data architecture.</p><p>It’s clear that we quickly and easily accept that preventing data issues is impossible and so we have focused our efforts on solving how to diagnose issues faster. Tristan Handy, Founder and CEO of dbt Labs recently wrote a thought-provoking article on the subject, titled ‘Interfaces and Breaking Stuff’ ( <a href="https://roundup.getdbt.com/p/interfaces-and-breaking-stuff">https://roundup.getdbt.com/p/interfaces-and-breaking-stuff</a> ). I particularly love this statement:</p><blockquote><em>“Don’t make changes that break downstream stuff. Does that feel as painfully obvious to you as it does to me? Rather than building systems that detect and alert on breakages, build systems that don’t break.”</em></blockquote><p>Yes it does, Tristan — it does feel painfully obvious to me and confronted me to ask: “Why is this not yet happening in data engineering?” because we can take so much guidance, best practices, and inspiration from software engineering that we can use design, develop, and maintain better pipelines. We are experiencing too many blockages and leaks and not enough provisioning for a reliable supply of good data.</p><p>And so, is it not feasible then to just not make changes upstream that will inevitably break stuff downstream? I know that sounds flippant but up to now, that has not been the default approach in data pipelines. Instead we see exports of complete databases, scripts that export data without ownership, and purchased (3rd party, external) data without proper documentation nor guarantees of data quality.</p><p>A common source of breaking changes is ingesting the internals of operational systems, as data is often just ingested through a database export. The team managing the operational systems will not realize, nor understand, the type of impact on the analytics team who uses the data for insights and reports, when making a change to their internal database model. At that time, they will break your dependency on the data you consume. That’s why I think taking the basic principle of encapsulation as used in software engineering, will be a major shift to solving this problem. As Petr Janda in his article, <a href="https://petrjanda.substack.com/p/the-art-of-drawing-lines">‘The art of drawing lines’,</a> talks about the “<em>encapsulation of cohesive logic”</em> as the main reason for driving architecture, data design 101 equals encapsulating what changes from what doesn’t.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HSPsifP_K23OqgsmpZ3OOA.png" /><figcaption><em>It’s still common in data products to rely on internal data from other systems. That breaks encapsulation and is a frequent cause of data issues.</em></figcaption></figure><p>Andrew Jones, Principal Engineer at GoCardless, makes the same observation on how changes can break a lot of things and the need for encapsulation, in his article ‘<a href="https://medium.com/gocardless-tech/improving-data-quality-with-data-contracts-238041e35698">Improving Data Quality with Data Contracts’</a>:</p><p><em>“You need to stop using the internal data models and build explicit interfaces between the data generators and the data consumers — much like an API.”</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hpkhhbEskdxobkzZ6r69eg.png" /><figcaption><em>Consuming data should only be done through a clear interface (API) that is provided and guaranteed by the team producing the data.</em></figcaption></figure><p>Here’s an excerpt from another blog written by Tristan, ‘<a href="https://roundup.getdbt.com/p/coalesce-data-contracts-the-semantic">Coalesce. Data Contracts. The Semantic Layer</a>’:</p><blockquote><em>“…never sync data directly from a database. Instead, build an API for any data that you want to sync into your data warehouse and extract all data via this API.”</em></blockquote><p>Every data ingestion and transformation step is a software component and hence it needs to be ‘treated as a product’, which includes proper encapsulation. Whilst it may not <em>always</em> be possible to just prevent analytical data from getting broken, encapsulation does provide a good frame of reference to start addressing the problem and avoiding spaghetti — or clogged — pipelines. Look for the handovers between teams to find the most vulnerable dependencies on internals that should be replaced with dependencies on clear APIs.</p><p>All this aligns with the groundwork that has been laid out by <a href="https://www.oreilly.com/library/view/data-mesh/9781492092384/">Zhamak Dehghani and work on the principles and logical architecture of Data Mesh</a>, which recognized that a paradigm shift was needed in how we manage and decentralize data at scale, whilst following the same principles as APIs in microservices software engineering. One of the four principles of data mesh is “domain-oriented decentralized data ownership” and enabling data practitioners to create, share, discover, and use data. In my opinion, it’s this that has most likely set data contracts into motion with the need — and requirement — to ensure the collaboration and handover between domains can exist and function.</p><p>Every ingestion or transformation step should be considered as a software component. It must be based on other components (software or data) through clear contracts. It must hide its internals from the data consumers. It must produce contracts for its outputs.</p><h3>Problem #2: The broken type system</h3><p>The second problem that I believe data contracts address, is a shortcoming in the storage engines. Storage engines have a type system that’s far too basic for today’s analytical data needs. We are still specifying the schema decimal places in numerics and the max number of chars in a VARCHAR as if we are still programming <a href="https://en.wikipedia.org/wiki/Relational_database">in the seventies</a>.</p><p>To make the point, imagine columns ‘length’ and ‘width’ in a dataset ‘PRODUCTS’. In the storage engine, this will probably be numeric types. But is it expressed in centimeters, millimeters, inches or miles? This you have to communicate to the data consumer completely outside of the storage type system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dirow_6ht0NSjgg3vHlQhA.png" /><figcaption><em>The data typing system of analytical storage engines don’t include proper capabilities for semantics and validity checking</em></figcaption></figure><p>All of the innovation in storage engines has been focused on increasing speed and improving scale. None so far have tackled the problem of a broken type system. In contrast, consider the job of a developer that’s coding in an object-oriented programming language where a significant effort is spent building composite data structures and enforcing data constraints. In programming languages like Java or Python, there is a proper typing system. Developers would be creating a ‘Distance’ type that’s made up of a numeric value and a unit of measure. And this distance would be used composing a larger Product class. The lack of such a proper type system in storage engines prohibits it from becoming the basis for good data management workflows.</p><p>With all the new cloud databases focused on speed and scale, I have no idea why no one ever thought to start a database with a composable type system yet. Please add a comment if I’ve missed this and someone actually has 🙂</p><p>This extra metadata is crucial for consumers that want to evaluate the data or start to use it. It should be exposed in discovery tools just like is the case today. But the system of record where this information is maintained should move from a catalog to a data contract file managed by the producer team in a git repo.</p><h3>The API for data</h3><p>In the last couple of months, a lot has been written on data contracts. But it’s not yet very concrete. This article aims to take a small step to clarify the current fluffy situation around data contracts. Benn Stancil navigates his way through the hype and many conversations, he’s found an interesting way to describes it in <a href="https://benn.substack.com/p/data-contracts">Fine, let’s talk about data contracts</a>:</p><p><em>“My initial reaction to data contracts was the same as my reaction to the data mesh. Both struck me as a kind of Rorschach proposition: Something defined well enough that we can all sense its shape, but abstract enough that we can also project our own opinions on top of it.”</em></p><p>Can a data contract do for data what APIs did for software? Let’s take it back to software engineering. In software engineering, an API contains all of the details a developer needs to start using the functionality of the component. An API is a description maintained by the producer that hides the internals and states how the component can be used by the consumer. Now in data engineering ,for datasets like tables in databases, warehouses, or queues in streaming systems, things are a bit different and up until now, it hasn’t been obvious how to specify an API. But there is a way to specify data APIs and that’s based on the database or warehouse protocol, along with the schema of the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3205S2d45QnNhGNv49RnnQ.png" /><figcaption><em>When consuming software engineering functionality, an API is common practice. In data, there is no common way yet to define an API for consuming datasets.</em></figcaption></figure><p>Before diving in the contents of a contract I want to argue that data contracts must be managed as code. I agree with the notion of shift-left and that the workflows to update contracts should appeal to engineers. Engineers in data producer teams will have to take ownership of maintaining the contracts and ensuring that they are in sync with the actual data. Those engineers want to work as code. It’s natural for them and easy to version control, set up new environments for CI/CD, production and so on. All these aspects imply that operational data infrastructure is best managed by engineers as code. With contracts there is a huge opportunity to become the central place for engineers to configure the aspects they need to manage for different tools and use cases.</p><p>Now let’s go back to defining the data contract as the API for data and let’s approach it from the perspective of a data consumer. To start with, when consumers search for data, what is the minimum that they need to know and understand about the data to determine that it is fit for their purpose and their specific use case?</p><p>I think the right way to look at what should go in or out of a contract, is to start from the absolute minimal contract requirements. And on that basis add a flexible extension mechanism so other configurations and metadata can be layered into the contract.</p><p>The bare minimum that a data contract must include is <strong>access details, dataset name, and schema</strong>, as I think that is the least amount of information required in order to be able to start using the data on your own.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YVJbb3hI8wDgq8tah3PdbA.png" /><figcaption>Minimum, technical content of data contract</figcaption></figure><p><strong>Access details</strong>: This represents the precise location of the dataset. It includes the connection details to the database, warehouse or similar, with variables to be used for credentials. The specification of the storage engine is important as it implies the access protocol. While a reference to the storage engine is often just a logical name to accommodate for staging environments, I think contracts should be primarily targeted at the production environment and that there should be a separate mechanism –inside or outside the contract– to override variations of the contract for staging environments.</p><p><strong>Dataset name</strong>: Logical names may be necessary for discovery, but when using the dataset as for instance in building a report, the consumer needs to know the exact name as it is known in the storage engine.</p><p><strong>Schema</strong>: These are the columns and the structure of the data which must be expressed in the data type system of the storage engine. If you want to use the data from a SQL engine in for instance a query, you need to know the exact SQL types of the columns.</p><p>From a technical perspective, access details, dataset name and schema is sufficient to access the data. I want to highlight the clear line between the technical minimally required information to access the data and other content in the contract file. The technical access details will always be required for any tool to work with the data. All other configurations and information that is added to the contract is there for a use case, workflow or tool. Those extra parts are related to how the data team organized their workflows. For a specific data management workflow in a company, descriptions and tags may be required. But that may be different in other companies.</p><p>From a workflow perspective, the notion of an ‘API for data’ is tightly coupled to the discovery use case. Discovery tools need to present all information for consumers to find and start using the data themselves.</p><p>Currently it’s common for discovery tools to ingest the metadata from the storage engines and present that to its users. To me, after adopting contracts, it makes sense to feed the discovery tools from contracts rather than from the storage engine’s metadata. After all that would make perfect sense: It’s only the ‘APIs for data’ that are carefully crafted and validated by producers that should go in a discovery tool. Instead of connecting discovery tools to a storage engine directly, which typically also includes all the internal datasets that are not intended for consumers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4ojwH7IDtYv5nbq1BdVx6g.png" /><figcaption><em>Ideally discovery tools only publish contracts curated by the data producer</em></figcaption></figure><p><a href="https://dataproducts.substack.com/p/an-engineers-guide-to-data-contracts">Chad Sanderson</a> and <a href="https://davidsj.substack.com/p/yet-another-post-on-data-contracts">David Jayatillake</a> argue that semantics should also be included in contracts. Semantics is information about how to interpret data, for example the descriptions and unit-of-measure. I agree that after the metadata necessary to access the dataset, semantics are the next vital information to start using it so that there are no unexpected surprises (think about when <a href="https://www.simscale.com/blog/nasa-mars-climate-orbiter-metric/">NASA lost a spacecraft</a> due to a metric math mistake!).</p><p>I agree with Chad and David in the sense that from a discovery perspective, it’s essential to require semantics to be included in contracts. Still in transition situations contracts could be used in other scenarios where for instance semantics are stored in the discovery tool’s database not yet linked to the contracts and that those contracts are only used for quality. Of course, in the ideal scenario the same contracts would be used for both discovery and data quality, but the point is that apart from the data access and schema details the other contract information depends on how companies use tools across the data stack to implement their use cases and workflows.</p><p>So rather than focussing on what should go in or out of a contract, I think the clue is to ensure that technical access details are required and that a generic extension mechanism for contracts to allow for extra configurations and information to be added for specific tools and use cases.</p><p>As it stands today, catalogs and discovery tools are already implemented broadly. Chances are small that discovery is going to be the driver for companies to start adopting data contracts. That’s different for data quality. A lot of companies are still looking to get started or improve their data quality strategy. So we think it’s most likely that companies adopt data contracts to implement data quality.</p><h3>Quality as the first use case</h3><p>Data quality is usually the very first use case for data contracts because authoring checks and schema declaratively as code fits with the engineering workflows. In this section we start by looking at how data quality can be layered into contracts and then look ahead to how other data management workflows can be handled as well.</p><p>To look at how data quality can be layered on top of a contract we can consider these conceptual categories of data quality configurations: schema check, extra data quality checks and automated observability monitoring configurations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y7zTsPrtsXI6biHSjjC8sw.png" /><figcaption><em>Content sections in a data contract related to data quality</em></figcaption></figure><p>Schema is already part of the technical essential contract information. That can be leveraged for quality checking to verify if the expected schema is still in place. A new section with quality checks can be added to a contract so that engineers can configure and manage the quality checks as code. But also automated observability monitoring configurations can be added as a separate section. Those configurations can be very minimal as for instance a simple boolean flag to activate automated monitoring. Optionally important columns may get a flag to fine tune the default automated monitoring. The main point I want to make is how for various data quality use cases, some information like schema is already present and other extra configurations like quality checks can be added to the contract.</p><p>Here’s an example of how a data contract could look like with data quality checks and observability configurations:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LHCWrfs9cP4KuAPDl_0-Mg.png" /><figcaption>Example data contract</figcaption></figure><p>For the data quality use case, let’s walk through the steps. First, contract verification needs to be set up to ensure data quality is ok to protect data consumers from changes in the data that break their usage. Contract verification implies the schema is checked and the data quality checks are verified as well. Notifications will be configured in case the data doesn’t match.</p><p>Contract verification should be done if either new data arrives or when the contract changes. Verifying a contract when new data arrives is usually done in the orchestration pipeline itself. After the pipeline has produced new data, an extra step is added to the pipeline to run contract verification.</p><p>The contract will change most likely through git workflows like branching and pull requests (PRs). So to cover the contract update event, a contract verification is added to the CI/CD pipeline of the contract (&amp; transformation pipeline) repo.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aIOoH8cSDURo_Jlzc2E7qg.png" /><figcaption><em>Contract verifications reads the contract and verifies that schema and data quality checks pass on the current data</em></figcaption></figure><p>Ensuring that contracts get verified each time data or a contract potentially changes, ensures that you can trust your contract to be an accurate description of your data. As the business changes, so does the shape of your data. Relying on the discipline of people to keep the contracts in sync with the storage engine is not enough to keep them in sync. Without automated verification, the trust in the data contract will get lost and all the potential value of the contracts are lost. But with contract verification in place, contracts can be a declarative foundation that goes beyond the “API for data” principle.</p><h3>Building data management on contracts</h3><p>Many data management workflows and automations can be implemented using contracts as the foundation. Just like checks were added for data quality to the contract, other configurations and metadata can be added to the contract to control other workflows like discovery, access, privacy, retention etc. A data contract is the perfect place to specify the ownership, retention and access policies and so on.</p><p>When looking at the newer set of data management tools, they adopt the principles of shift-left. Shift-left is the trend that moves interactions of data management from web based UIs to managing data tools as code by the data producer engineers. You’ll see a lot of these tools being configured with configuration or YAML files.</p><p>In this context, we see data contracts will take a central place from which all those tools can be configured. This will lead to massive simplification of managing the large variety of data tools.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rxtmp5uPYSwUZJS29xMavQ.png" /><figcaption><em>Contracts become the central UX through which engineers can configure their aspects for many tools in the data stack</em></figcaption></figure><p>Now you can see that with contracts we’ve created a declarative and extensible way for engineers to manage their parts of the workflow. Contracts for the foundation for many of the data management workflows. This approach therefore has a huge potential to simplify the integration of the different tools in the data stack.</p><h3>Frictionless introduction of contracts</h3><p>Of course we cannot assume that companies are going to redesign their data architectures on top of data contracts. So data contracts must allow for them to be phased in step by step.</p><p>For starters, let’s assume you have an existing data infrastructure that is running. Introducing contracts can be done step by step. Identify a small scope like one dataset, one transformation or one ingestion pipeline.</p><p>A good rule of thumb is to focus on the analytical teams. There, look for the datasets that are shared across teams and that are critical to the business’ day-to-day operations. Those are the ones that require contracts as one team should never rely on the internals of a component from another team. Here’s a good rule of thumb to prioritize the creation of contracts for existing datasets:</p><ol><li>Take a consumer-first approach. Start with datasets on the consumer side and work your way upstream to the source.</li><li>Next, focus on the datasets that are handovers between different teams. That’s important for encapsulation.</li><li>Gradually work your way upstream to cover all datasets including ingestion.</li></ol><p>What about datasets that an analytics team depends on and for which we can’t rely on the producer to create a data contract? This might be the situation when you purchase data from an external vendor or when an upstream data team does not yet have the capacity to produce contracts. In that case it’s the responsibility of the consumer to set up contracts and verification (see below). But it definitely makes sense to push that responsibility as much upstream and towards the producer as possible.</p><p>Next create contracts for those datasets and ensure contract verification. Your pipelines will continue to run as before but now with data quality monitoring enabled.</p><p>Once contract verification is monitoring data quality, something that you may wish to reconsider is how you’ll be connecting the datasets to your discovery tool. That tool will probably be connected directly to the storage engine to extract the metadata from there. But once you have reliable contracts in place, you could switch to ingesting the contract information instead. That will have the extra benefit that much more information can be ingested in one go.</p><h3>Active sync</h3><p>You may be asking yourself: How much work is this to manage these contracts manually in git? Data engineers are proficient to get this done. But there is extra tooling that can help and we’ve called this notion ‘active sync’. In order to fully appreciate what active sync does, let’s review the source of changes to contracts: The foundation is a copy of the storage engine’s schema. You could consider the storage engine’s metadata as the source of truth and hence as the source of contract changes. And separately consider all other information and configurations in contracts for all kinds of data management workflows that are managed by engineers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kD0iXZbGpSYOm1nNHIj6ww.png" /><figcaption><em>An active sync tool can help engineers keep the contract in sync with schema changes while at the same time, engineers manage other configurations in the contract</em></figcaption></figure><p>The maintenance of the schema information is a great candidate for tooling automation. Engineers update transformations that cause the schema of the dataset in the CI/CD to change. Contract verification in the CI/CD flow will fail. Consider schema changes and the fact that we want contracts to be an accurate description of the data. Then we can also see that we want all schema changes to be pushed to the contract in git. There is no manual decision needed. Those schema changes can be pushed to the contract on the PR or on a branch as plain commits.</p><p>When adding new columns, you may have to consider when this contract makes it to the production contract verification. A new column may have to be marked as ‘anticipated’ if publishing of the contract is not tied to the pipeline code in the same repository.</p><p>For removed columns, it always makes sense to mark them as ‘removed’ instead of just removing the whole column. It even makes sense to push this kind of deprecation and removal in discovery tools.</p><p>Another use for the sync tool is simplifying the initial contract. Often, engineers will already have a version of the dataset in staging or production. The active sync tool can extract the metadata from the storage engine and generate the initial contract file. A contract sync tool can remove the burden of crafting these initial contracts. The basics of the contract such as access details, dataset name and schema can just be generated. Once generated, the contract is ready to be pushed and managed in git.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nRNGd1yTVIjtCDLAlzMvIw.png" /><figcaption><em>A contract sync tool can also simplify the getting started by generating the initial version of a contract</em></figcaption></figure><h3>Conclusion</h3><p>We believe that contracts will lead to a paradigm shift in data. I hope to have shown you how data contracts will become the foundation for many of the important data management workflows. They can be introduced step by step without requiring any changes to the existing data architecture and they fit with the data engineers’ way of working (shift-left). Contracts also have the power to prevent a lot of data issues by improving the workflows and communication between producers and consumers. Furthermore the common foundation will make it easier to integrate the large variety of data tools. As data contracts get adopted, analytics teams will be enabled to deal with more data in production and serve the business better with their analytical needs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c7a98544d2a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Event-driven interface design]]></title>
            <link>https://medium.com/@tombaeyens/event-driven-interface-design-209e7e2d3093?source=rss-91de644a221a------2</link>
            <guid isPermaLink="false">https://medium.com/p/209e7e2d3093</guid>
            <category><![CDATA[distributed-systems]]></category>
            <category><![CDATA[ddd]]></category>
            <category><![CDATA[event-driven-systems]]></category>
            <category><![CDATA[cqrs]]></category>
            <category><![CDATA[microservices]]></category>
            <dc:creator><![CDATA[Tom Baeyens]]></dc:creator>
            <pubDate>Thu, 11 Jan 2018 13:03:39 GMT</pubDate>
            <atom:updated>2018-01-12T07:36:12.810Z</atom:updated>
            <content:encoded><![CDATA[<p>The main goal of this article is to put the notion forward that a service API’s should be expressed as queries, commands and events. It is discussed in the context of event-driven, distributed microservices. A practical link is made to implementation technologies like HTTP and messaging systems on a conceptual level.</p><p>Enterprise software development is moving from the 3 tier apps to event-driven, distributed microservices. HTTP interactions don’t support transactions like relations databases do. Dropping distributed transactions has far reaching implications on the architecture. The most common solution is using <a href="https://martinfowler.com/articles/201701-event-driven.html">events</a> to communicate between services and accepting eventual consistency. This article provides guidance when designing a service interface in the context of event-driven, distributed microservices and lambda’s.</p><p>Traditional 3-tier apps with a <a href="https://en.wikipedia.org/wiki/Representational_state_transfer">REST</a> API were often completely based on HTTP and hence on the request-response <a href="https://en.wikipedia.org/wiki/Messaging_pattern">message exchange pattern</a>. Introducing events into an architecture means we need to facilitate output-only and input-only message exchanges as well. For these one way message exchanges, we have more choices for the concrete technologies. Most common are pub/sub message queues, but alternatives include webhooks and Server Sent Events (SSE). See <a href="https://nordicapis.com/5-protocols-for-event-driven-api-architectures/">Kristopher Sandoval’s article</a> for an in-depth comparison.</p><p>While REST heavily builds on the HTTP protocol, I think that <a href="http://cqrs.nu/">CQRS</a> provides a much better basis for structuring a service API. It’s a natural fit to use <a href="https://martinfowler.com/bliki/CQRS.html">CQRS</a> as a way to categorize the interactions of an API and extend it with events. The following figure shows the structure of an event-driven API based on CQRS and that incorporates events as first class citizen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uoB-3zovPWxVeizM." /><figcaption><strong>Figure 1. Event-driven service interactions</strong></figcaption></figure><p>This figure emphasizes that a service interface is composed of 3 things namely <em>queries</em>, <em>commands</em> and <em>events</em>. CQRS and event-driven are often described as related patterns. I think it’s important that each service interaction is qualified as a query, a command or as an event.</p><p>In the figure, I used ‘Q’ as short for message queue to indicate any technology that supports input-only and output-only message exchanges.</p><p>Let’s walk through the different parts of these event-driven service interactions.</p><h4>Queries</h4><p>Queries retrieve data without changing the state of the service. E.g. get the list of customer addresses or get all the details for this order.</p><p>Queries are mostly implemented over HTTP. That’s because request-response nature of queries is a perfect fit with HTTP. While more non-blocking HTTP client technologies are being offered in programming languages, it may be good to keep in mind that many clients will continue to use blocking code because it’s just simpler to read and test and debug.</p><p>A new trend is that moving from the REST approach, which leads toa set resource endpoints to a single endpoint which makes use of a query language like for example <a href="http://graphql.org/">GraphQL</a> and <a href="http://restql.b2w.io/">restQL</a>. The philosophy is to put more emphasis on what the client wants and make the server responsible for fetching that, instead of just serving what the server has to offer.</p><p>In an event-driven architecture, it’s also important to think about the consistency guarantees for each query. In contrast to 3-tier applications using transactions, in event-driven architectures, it’s not guaranteed that the query results will be consistent: Will the changes of all previously performed commands be included in the results? One of the drivers for an event-driven architecture is to increase scalability and reliability in a distributed environment. This may imply that not all commands have been completely processed and that queries may not always contain consistent or up-to-date results.</p><h4>Commands</h4><p>Commands are interactions that perform changes to the service. E.g. update the address of this customer or add a new orderline to an order. Commands should be named with a verb in the imperative mood plus with the aggregate type, for example UpdateAddress or AddOrderline.</p><p>The effect of commands may ripple through the service and it’s dependencies in stages. <a href="http://www.allthingsdistributed.com/2008/12/eventually_consistent.html">Eventual consistency</a> enables scaling at the expense of being temporary inconsistent. These inconsistencies may manifest itself in queries, in the service’s own internal storage or in dependency service queries or storage. This is something to keep in mind when your code performs a command and subsequent a query. The result of the command may not be reflected in the query. Since it is important when writing code, it’s also important to document this aspect when you create a new command.</p><p>Command responses are optional. Only when exposed over HTTP, the client can actually use the result. In the case of sending commands to a service over a message queue, the result is just ignored.</p><p>When invoking commands over HTTP, it’s good practice for command implementations to return the response right after persistence ensures that the command will eventually complete. This implies that persistence id’s have to be generated. It’s also a good idea to include the id’s as part of the command response for testing or correlation.</p><p>Both HTTP and messaging can be used to accept commands. Message queues are good at handling bursts and they usually support transactions. HTTP on the other hand may be required for clients that want to use the optional response.</p><h4>Events</h4><p>An event is a factual description of a change that happened in the service. For example a customer address was changed or an orderline has been added to an order. Events should be used with a past-participle verb like e.g. AddressChanged and OrderlineAdded. Just like commands, events are immutable and a great way to reduce coupling between systems.</p><p>A client typically listens to events as they occur by subscribing to a message queue. This is an output-only message exchange and also referred to as streaming. In order to remain up to date you have to keep track of the continuous stream of events.</p><p>Events should not be tailored for a specific need of one client service. The amount of data to put in an event is a tricky tradeoff for which there are no simple and clear answers. For an in-depth discussion, watch <a href="https://www.youtube.com/watch?v=STKCRSUsyP0">Martin Fowler’s video</a> on event notifications.</p><p>Event consumers can be notified at the moment when the event occurs. But events can also be stored and used afterwards.</p><h4>Events from dependencies</h4><p>An event from a dependency is an event produced by another service that your service needs to be notified of. Subscribing to and relying on events from another service means that other service is a dependency.</p><p>Events from dependencies can be replayed to create a new view, index, cache, <a href="https://martinfowler.com/bliki/MemoryImage.html">memory image</a>, or <a href="https://docs.microsoft.com/en-us/azure/architecture/patterns/materialized-view">materialized view</a> in our service’s domain.</p><p>Clients typically want to process each event exactly once. In a distributed system, that is a tricky problem. A solution can be to ensure that events are applied <a href="https://stackoverflow.com/questions/1077412/what-is-an-idempotent-operation">idempotent</a>. This is a relatively new area with tradeoffs. Up to now, there is no common set of options yet amongst distributed messaging solutions. You should decide on your approach and get your requirements clear for this because it’s an important selection criterion for your message system.</p><p>Events from the dependency service are defined in a different <a href="https://domainlanguage.com/ddd/reference/">domain</a>. For that reason, I explicitly mention that the events are <em>from dependencies</em> and put them in the same gray as the dependencies. Be careful when it comes to subtle language and naming differences.</p><h4>Conclusion</h4><p>While CQRS is known as a pattern, it makes sense to use it as a basis for categorizing service interactions. By adding support for one-way event streaming, it forms a solid structure for building event-driven microservice APIs.</p><h4>References</h4><ul><li>What do you mean by “Event-Driven”? Presentation video by Martin Fowler <br><a href="https://martinfowler.com/articles/201701-event-driven.html">https://martinfowler.com/articles/201701-event-driven.html</a></li><li>Video: The Many Meanings of Event-Driven Architecture by Martin Fowler <br><a href="https://www.youtube.com/watch?v=STKCRSUsyP0">https://www.youtube.com/watch?v=STKCRSUsyP0</a></li><li>CQRS.nu <br><a href="http://cqrs.nu/">http://cqrs.nu/</a></li><li>CQRS by Martin Fowler <br><a href="https://martinfowler.com/bliki/CQRS.html">https://martinfowler.com/bliki/CQRS.html</a></li><li>Domain-Driven Design Reference: Definitions and Pattern Summaries by Eric Evans <br><a href="https://domainlanguage.com/ddd/reference/">https://domainlanguage.com/ddd/reference/</a></li><li>Events As First-Class Citizens by Randy Shoup<br><a href="https://medium.com/@shoup.randy/events-as-first-class-citizens-8633e8479493">https://medium.com/@shoup.randy/events-as-first-class-citizens-8633e8479493</a></li><li>GraphQL: A query language for your API<br><a href="http://graphql.org/">http://graphql.org/</a></li><li>Evolution of business logic from monoliths through microservices, to functions by Adrian Cockcroft <br><a href="https://read.acloud.guru/evolution-of-business-logic-from-monoliths-through-microservices-to-functions-ff464b95a44d">https://read.acloud.guru/evolution-of-business-logic-from-monoliths-through-microservices-to-functions-ff464b95a44d</a></li><li>Pattern: Command Query Responsibility Segregation (CQRS) by Chris Richardson <br><a href="http://microservices.io/patterns/data/cqrs.html">http://microservices.io/patterns/data/cqrs.html</a></li><li>Pattern: Saga by Chris Richardson<br><a href="http://microservices.io/patterns/data/saga.html">http://microservices.io/patterns/data/saga.html</a></li><li>The Seven Deadly Sins of Microservices by Daniel Bryant<br><a href="https://www.infoq.com/presentations/7-sins-microservices">https://www.infoq.com/presentations/7-sins-microservices</a></li><li>Daniel Bryant’s twitter feed <br><a href="https://twitter.com/danielbryantuk">https://twitter.com/danielbryantuk</a></li><li>5 Protocols For Event-Driven API Architectures<br>by Kristopher Sandoval <br><a href="https://nordicapis.com/5-protocols-for-event-driven-api-architectures/">https://nordicapis.com/5-protocols-for-event-driven-api-architectures/</a></li><li><a href="https://en.wikipedia.org/wiki/Messaging_pattern">https://en.wikipedia.org/wiki/Messaging_pattern</a></li><li><a href="https://en.wikipedia.org/wiki/Representational_state_transfer">https://en.wikipedia.org/wiki/Representational_state_transfer</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=209e7e2d3093" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>