<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Great Expectations on Medium]]></title>
        <description><![CDATA[Stories by Great Expectations on Medium]]></description>
        <link>https://medium.com/@expectgreatdata?source=rss-6048af210e26------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*4psN4MNFb4JrnuMhHMuAHg.png</url>
            <title>Stories by Great Expectations on Medium</title>
            <link>https://medium.com/@expectgreatdata?source=rss-6048af210e26------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 03:54:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@expectgreatdata/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Exploring data quality: volume]]></title>
            <link>https://medium.com/@expectgreatdata/exploring-data-quality-volume-5150b689c25d?source=rss-6048af210e26------2</link>
            <guid isPermaLink="false">https://medium.com/p/5150b689c25d</guid>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data-quality-management]]></category>
            <category><![CDATA[data-quality-dimensions]]></category>
            <dc:creator><![CDATA[Great Expectations]]></dc:creator>
            <pubDate>Mon, 28 Oct 2024 15:24:01 GMT</pubDate>
            <atom:updated>2024-10-28T15:24:01.536Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*57jZP-C-EfQQ3HvEyxss-g.png" /></figure><p><em>This post is the third in a seven-part series that uses a regulatory lens to explore how different dimensions of data quality can impact organizations — and how GX can help mitigate them. Read</em><a href="https://greatexpectations.io/blog/exploring-data-quality-schema-validation"><em> part 1 (schema) </em></a><em>and </em><a href="https://greatexpectations.io/blog/exploring-data-quality-missingness"><em>part 2 (missingness)</em></a><em>.</em></p><p>Data volume isn’t just a statistic about the size of your datasets — it’s a critical dimension of data quality that can make or break your business’ operations.</p><p>Managed effectively, data volume can be a goldmine of insights and opportunities. It can help you make better decisions, optimize processes, and stay ahead of the competition.</p><p>On the flip side, when your data volume spirals out of control it can lead to performance issues, inaccuracies, and missed chances to innovate.</p><p>In this post, we’ll explore the world of data volume management and how it can transform your organization’s operational excellence.</p><p>We’ll share real-world examples and strategies for tackling data volume challenges head-on. And we’ll show how the Great Expectations platform can help you monitor and validate your data volume seamlessly, ensuring that your data’s quality remains top-notch.</p><h3>A double-edged sword</h3><p>Data volume is a pervasive challenge facing organizations across industries. While the explosive growth in available data presents immense opportunities, it also poses significant challenges:</p><ol><li>System strain: Unexpected spikes in data volume can overwhelm processing systems, leading to performance issues and even system failures.</li><li>Inaccurate insights: Anomalies in data volume, if undetected, can skew analyses and lead to flawed decision-making.</li><li>Compliance risks: Inconsistencies in data volume across different stages of a pipeline can raise red flags during audits and compliance checks.</li><li>Operational inefficiencies: Managing fluctuations in data volume often requires manual intervention, diverting resources from more strategic initiatives.</li></ol><p>To show how volume-based data quality checks can improve your data pipelines, we’ll return to the example of FinTrust Bank: a fictional financial institution that’s serving as our lens for this series.</p><h3>Leveraging GX for effective volume management</h3><p>Continuing our journey with FinTrust Bank, let’s see how Samantha — the data engineer leading the quality improvement effort — and her team tackle data volume challenges.</p><p>Having chosen GX as their data quality solution, Samantha already has what she needs to start tracking and checking data volumes. This proactive approach can help Samantha and her team prevent problems down the line.</p><p>Using GX Cloud, Samantha’s team collaboratively developed and refined their volume Expectations.</p><p>GX Cloud allows team members to work on developing their data tests (called Expectations) simultaneously, share insights, and track changes in real time. Its support for collaboration significantly accelerated the team’s data quality work.</p><p>For detailed information on volume Expectations and their implementation, see our <a href="https://docs.greatexpectations.io/docs/reference/learn/data_quality_use_cases/volume">technical guide on volume management</a>.</p><h3>Key Expectations for data volume</h3><p>Samantha and her team implemented key <a href="https://docs.greatexpectations.io/docs/reference/learn/data_quality_use_cases/volume">Expectations for data volume</a>. She used “<a href="https://greatexpectations.io/expectations/expect_table_row_count_to_be_between">Expect table row count to be between</a>” to ensure daily transaction volumes remained within expected bounds, setting thresholds based on historical patterns. For reconciliation tasks, she employed “<a href="https://greatexpectations.io/expectations/expect_table_row_count_to_equal">Expect table row count to equal</a>” to verify that specific batch operations processed the exact expected number of records.</p><p>As these Expectations ran, Samantha’s team gained valuable insights into their data volume. They identified unusual spikes in transaction volume during peak periods, inconsistencies between data pipeline stages, and discrepancies in batch processing records.</p><h3>Strategic approaches to manage data volume</h3><p>Armed with insights from GX, Samantha’s team developed a multifaceted strategy to manage data volume effectively:</p><ol><li>Implemented scalable data processing architecture to handle volume spikes gracefully.</li><li>Automated data volume checks using GX Cloud, enabling proactive issue detection.</li><li>Established data volume SLAs across departments, ensuring consistent expectations.</li><li>Integrated volume checks into their CI/CD pipeline, catching issues early in development.</li></ol><p>They also set up automated email alerts using GX Cloud, enabling rapid responses to critical volume deviations.</p><h3>Outcomes and impacts</h3><p>FinTrust Bank’s system performance improved significantly as volume spikes were proactively managed, leading to a more stable and reliable data infrastructure.</p><p>In particular, downtime was reduced for critical business processes. The accuracy of financial forecasting models also improved, thanks to more consistent and reliable data inputs.</p><p>With more reliable output from existing models, FinTrust Bank’s data scientists could now build on those to develop more precise models, enabling better decision-making across the organization.</p><p>Automated volume checks made reconciliation efforts more efficient, allowing time and resources previously spent on manual data reconciliation to be redirected to higher-value tasks.</p><p>When FinTrust Bank ran a promotional campaign, its systems seamlessly accommodated a surge in transactions. The marketing department made real-time optimizations based on reliable data, resulting in a more successful campaign outcome.</p><p>FinTrust Bank’s robust data volume validation processes were even recognized by auditors as industry-leading practices, boosting the bank’s reputation and demonstrating their commitment to maintaining the highest standards of data quality.</p><p>While FinTrust Bank may be a fictional entity, the benefits it reaped from effective data volume management can be realized by organizations in the real world.</p><p>Prioritizing data quality and implementing smart volume management strategies means that you’re taking steps to improve your operational excellence and successful outcomes for your data-driven initiatives.</p><h3>Mastering data volume: A catalyst for operational excellence</h3><p>Managing data volume is not just about keeping systems running smoothly — it’s a strategic lever for driving operational excellence. FinTrust Bank’s journey showcases the transformative power of effective volume management:</p><ul><li>Improved system performance and reliability</li><li>More accurate and timely insights for decision-making</li><li>Increased operational efficiency and agility</li><li>Enhanced regulatory compliance and audit readiness</li></ul><p>By getting a handle on your data volumes <a href="https://docs.greatexpectations.io/docs/reference/learn/data_quality_use_cases/volume">(leveraging techniques detailed in our technical guide)</a>, you’re not just optimizing operations — you’re fueling your organization’s growth and innovation.</p><p><a href="https://docs.greatexpectations.io/docs/reference/learn/data_quality_use_cases/volume">See how GX can help you get a handle on your data volumes.</a></p><h3>Join the conversation</h3><p>How has tackling volume challenges transformed your data operations? <a href="https://discourse.greatexpectations.io/c/share/18">Join the discussion in our community forum</a> to exchange ideas and best practices with other data practitioners. Together we can continue to build a new vision of what it means to excel in data quality.</p><p>Stay tuned for the next installment of our series, where we’ll explore how monitoring your data distribution can elevate your data quality capabilities and your data outcomes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5150b689c25d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why data validation is critical to your pipelines]]></title>
            <link>https://medium.com/@expectgreatdata/why-data-validation-is-critical-to-your-pipelines-08e13a4ea97a?source=rss-6048af210e26------2</link>
            <guid isPermaLink="false">https://medium.com/p/08e13a4ea97a</guid>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data-validation]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Great Expectations]]></dc:creator>
            <pubDate>Mon, 21 Oct 2024 14:02:18 GMT</pubDate>
            <atom:updated>2024-10-21T14:02:18.079Z</atom:updated>
            <content:encoded><![CDATA[<h3>Problems that start small don’t stay that way</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WNHYv7xRKDFv9TBqkBoYlw.jpeg" /><figcaption>📸: <a href="https://stock.adobe.com/images/yellow-triangular-sign-of-electrical-hazards/117319417">Ilia Shcherbakov via Adobe Stock</a></figcaption></figure><p>If your data pipeline serves up bad data to a business stakeholder, one of three things will happen:</p><ul><li>They realize it’s wrong right away, and lose confidence in the data and data team.</li><li>They don’t realize it’s wrong, take action based on it, find out it was wrong later… and <em>then</em> lose confidence in the data and data team.</li><li>They don’t realize it’s wrong, take action, and no one ever realizes. Maybe the outcome is the same as if the data had been correct, maybe not. But no one knows why what happened, happened — their confidence in their decision is misplaced.</li></ul><p>With a few tweaks, all of the above go for your customers, too. And when your customers lose confidence in you, they start being your competitors’ customers instead.</p><p>Your data pipeline might be delivering the data, but testing your pipeline’s functionality can only do so much. Because a lot of data quality issues have absolutely nothing to do with your pipeline; they originate at the source of your data, before it reaches you.</p><p>But regardless of their source, data quality troubles are inevitable, and you need to handle them before stakeholders and customers ever see them.</p><p>The best way to do that is with data validation.</p><p>Data validation in the pipeline gives data teams peace of mind that no matter where errors originate, problematic data won’t reach end users.</p><p>This article belongs to a series exploring how you can use the Great Expectations data platform to integrate data validation into your pipelines. We’ll cover what data to validate, where validation should go, and the features of GX that are most useful at every step.</p><p>In this article, we’re looking at where data validation should go.</p><h3>Why validate throughout the data pipeline</h3><p>We’ve already established that you need to resolve data quality issues before a business stakeholder encounters them.</p><p>Stakeholders are most likely to encounter that’s at rest: that is, it’s at the end of its particular journey through your data pipeline. This is the <em>Target</em> data store in the illustration below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*gsnu_l05JXnxqY7WBc8hHw.png" /><figcaption>Image by GX</figcaption></figure><p>If we want to prevent stakeholders from encountering data quality issues, we need to resolve them before the data comes to rest — ie, when the data is moving through the pipeline.</p><p>Even if we take stakeholders out of the equation, though, logic still points to testing in the pipeline.</p><p>By definition, data at rest in a target store isn’t changing anymore. Any data quality issues are extremely likely to have originated upstream — with a few exceptions, like misalignment on definitions.</p><p><strong>When you catch an issue after the data has come to rest</strong>, you need to backtrace its entire journey and reverse engineer all its transformations until you’ve identified the issue’s source. Then you have to trace forward all of the places that the data went, and implement a backfill process to fix it or do manual repairs. That’s a <em>lot</em> of time and effort.</p><p><strong>When you catch an issue while data is in motion</strong>, you’re closer to the issue’s source. You might still need to do some backtracking and reverse engineering, but you know the scope from the outset based on the last set of checks that it passed. You can also stop the problematic data from moving downstream until you’ve fixed it.</p><p>You’re going to have to spend some resources on resolving data quality issues no matter what. But it’s clear that you’ll have to spend fewer of them when you’re doing data quality monitoring throughout your pipeline, on data in motion.</p><p>Often, the “throughout the pipeline” approach to data validation is called <em>shift-left testing</em>, or just <em>shift left</em>. <a href="https://www.ibm.com/topics/shift-left-testing">The language is borrowed from software testing</a>, but the concept is the same: the resource burden of fixing errors is reduced when you test early and often.</p><p>This is by no means a novel concept — consider “an ounce of prevention is worth a pound of cure,” written by Benjamin Franklin in 1735.</p><p>Even in the more modern sense of the quality of data, <a href="https://books.google.com/books/about/Making_Quality_Work.html?id=RKUaCosIIiEC">the 1–10–100 rule for quantifying the cost of bad data isn’t new</a>. And if a data quality problem cost $1 to prevent, $10 to fix, and lost you $100 if you didn’t do anything about it in 1992… you can imagine how steep that cost increase might look now.</p><h3>Where to validate data in your pipeline</h3><p>Data quality issues can be introduced anywhere in a pipeline: at data input, during ingestion, and post-ingestion during transformations and other processing. Identifying hotspots for data quality issue origination lets you deploy your data validation strategically, so that you can get insight into the most critical spots immediately.</p><p>Following the shift-left principle, the first hotspot to check is data ingestion.</p><p>Unexpected schema changes are one of the most common sources of data quality issues, which is one reason the idea of the <a href="https://datacontract.com/">data contract</a> gained traction so quickly. Schema validation alone will let you detect a significant source of quality errors, even before you consider other types of data quality checks.</p><p>The next series of hotspots is after each transformation step.</p><p>This approach is similar to “black box testing” of software engineering. With a post-transformation validation, you confirm that the transformation’s output conforms to what downstream consumers are expecting.</p><p>Typical checks for post-transformation include schema validation, distribution of numerical values, and relationships between columns.</p><p>It can also be useful to add data validation <em>before</em> transformation steps. This allows you, firstly, to ensure that your transformation will be getting the data you expect it to get.</p><p>In combination with post-transformation checks, pre-transformation data validation allows you to monitor the behavior of the transformation itself. Rather than checking only that the transformation produced reasonable data, you can confirm that it produced the <em>correct</em> data for the exact set of given inputs.</p><p>Many organizations apply <a href="https://www.databricks.com/glossary/medallion-architecture">the medallion architecture</a>, as popularized by Databricks. Setting data quality checks before data passes into the gold (or consumption) layer allows you to verify that the data conforms to the needs and business rule expectations of that specific layer’s consumers.</p><p>There are multiple architectural implementations that allow you to perform data validation, including the popular <a href="https://youtu.be/fXHdeBnpXrg?feature=shared&amp;t=994">Write-Audit-Publish (WAP) approach</a>.</p><h3>What to validate in the data pipeline</h3><h4>Types of validations</h4><p>Data quality is a multifaceted problem. To communicate meaningfully about a dataset’s current state and health, you need a framework against which state and health can be defined.</p><p>This assessment framework is typically defined using characteristics of the data, also known as data quality dimensions.</p><p>Common data quality dimensions used for assessment include:</p><ul><li><strong>Accuracy</strong>. How well data represents entities or events from reality.</li><li><strong>Completeness</strong>. All required data is present with no gaps.</li><li><strong>Consistency</strong>. Uniformity of data representation and measurements within a dataset or across different datasets.</li><li><strong>Timeliness</strong>. Data is present in the expected place, in the proper format, at the right time.</li><li><strong>Uniqueness</strong>. Each data point within a dataset represents a different entity or event.</li><li><strong>Validity</strong>. Data conforms to the expected format, type, or range of acceptable values.</li></ul><p>Different frameworks may use slightly different sets of dimensions and/or define them in slightly different ways, but these six — accuracy, completeness, consistency, timeliness, uniqueness, and validity — are a solid and widely-recognized foundation to use as you explore options for data validation.</p><h4>How to start</h4><p>To implement effective data quality monitoring, you need to understand the dataset and its intended uses, not just blindly implement statistical monitoring.</p><p>The first step in understanding a new dataset is to do <a href="https://greatexpectations.io/blog/what-is-data-profiling">data profiling</a>.</p><p>A simple profile can give you key information about the data’s state, and allows you to begin assessing whether existing validations and assumptions about the data’s state are effective or accurate.</p><p>Once you have an initial idea of the data’s state through profiling, you need to prioritize what to test. In an ideal world, you would be able to test every dimension of the data at every step — but if this was an ideal world, there wouldn’t be data quality issues at all.</p><p>Here are some factors to consider as you prioritize your data quality work:</p><ul><li><strong>Downstream usage.</strong> What data does your company use for its most critical decision-making? What data has a lot of external visibility? What data does day-to-day operations depend on?</li><li><strong>Business rules.</strong> Each column of your data exists in a business context, which means data that’s acceptable from a technological standpoint is still invalid operationally. Columns with complex business requirements could be high priority for data quality testing — especially if regulatory concerns are involved.</li><li><strong>Stakeholders.</strong> The more stakeholders a data has, the more possibilities there are for data quality issues that originate from different understandings of the data. Involving stakeholders to understand what expectations each has for the data — and, crucially, why they have that expectation — is a process better started sooner than later.</li></ul><p>For more considerations about setting priorities ahead of embarking on a data quality process, see our earlier blog post about <a href="https://greatexpectations.io/blog/where-to-start-with-data-quality">where to start with data quality.</a></p><h3>Summary</h3><p>In this article, we explored:</p><ul><li>Why data validation is critical</li><li>Why it should take place throughout your data pipeline</li><li>What aspects of data to validate</li><li>How to get started prioritizing your data validation work.</li></ul><p>In other entries of this series, we’ll explore how the GX platform can be used to implement data validation according to the principles we established here, including key features from GX Core and GX Cloud.</p><p><em>Thanks to Bruno Gonzalez for his contributions to this article.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=08e13a4ea97a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[4 steps to take when your data quality project gets pushed to the back burner]]></title>
            <link>https://medium.com/@expectgreatdata/4-steps-to-take-when-your-data-quality-project-gets-pushed-to-the-back-burner-585c5d357a19?source=rss-6048af210e26------2</link>
            <guid isPermaLink="false">https://medium.com/p/585c5d357a19</guid>
            <category><![CDATA[data-quality-monitoring]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-quality]]></category>
            <dc:creator><![CDATA[Great Expectations]]></dc:creator>
            <pubDate>Tue, 15 Oct 2024 19:19:31 GMT</pubDate>
            <atom:updated>2024-10-16T13:00:38.057Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AOx59WEjN4XhjXLqMvGRdg.jpeg" /><figcaption>GX Cloud can help when your data quality work is deprioritized but the need for it doesn’t go away</figcaption></figure><p>Just as you’re finally making some progress on getting a data quality solution in place, a new directive comes down and back-burners your work. Data quality gets put on hold for next month. Or next quarter. Or next year.</p><p>Sound familiar?</p><p>If it does, you’re in good company. Data quality projects get deprioritized in favor of other work all the time, and it almost doesn’t matter whether that other work is legitimately urgent or not. The effect on you (and on your data) is the same… and your data quality issues do <em>not</em> go away.</p><p>But even with your effort mandated elsewhere and your budget down to peanuts, you can still make incremental progress on your data quality monitoring. And it’s worth doing that because something is always better than nothing — and hopefully, you’ll be able to leverage the results to get focus and resources back onto establishing a data quality process sooner rather than later.</p><p>Here are four key steps to take when you need to deploy bare-bones data quality that puts you in a good position for later, and how GX Cloud can help.</p><h3>Step 1: Get SaaS-y and minimize overhead</h3><p>When you need to minimize your effort, a self-service SaaS is the way to go. Because every moment counts: you can’t be spending time obtaining permissions, installing software, or provisioning infrastructure.</p><p>At the same time, don’t over-focus on the right now: you need room to breathe (i.e., expand your data quality process) in the future. If that means switching platforms, you’re setting yourself up for extra work. Choose a SaaS option that will accommodate incremental growth.</p><p><strong>How GX Cloud helps:</strong> You can create a <a href="https://hubs.li/Q02TwS8J0">GX Cloud account</a> and start validating your data in just minutes using only your email address. That’s it — GX Cloud is fully hosted and will take care of everything else.</p><p>You can <a href="https://hubs.li/Q02TwQZK0">start connecting your data to GX Cloud</a> for testing as soon as you create your account. <a href="https://hubs.li/Q02TwPJp0">All you need to connect</a> your data to GX Cloud is a set of appropriate credentials.</p><p>And for the ‘room to breathe’ aspect, GX Cloud has <a href="https://hubs.li/Q02TwMRQ0">options</a> for teams at every level, from individual developers to enterprises.</p><h3>Step 2: Express yourself to maximize impact</h3><p>When you’re in a deprioritized-data-quality scenario, you need to make every minute of effort yield maximum effects. Meaning that right now, informed and expressive precision is more valuable than undirected surveillance.</p><p>Precision is key because it allows you to focus on what you know is a problem. And <a href="https://hubs.li/Q02TwMkL0">expressivity is the factor</a> that lets you embed your domain knowledge in the test.</p><p>In other words, you <em>know</em> what the data should look like and what kinds of quality problems you’ve had (or fear having). Test for those things to maximize the impact of your efforts. You need to be able to identify specific and relevant issues.</p><p>Once you can point to cold hard numbers about downtime prevented and crises averted, building buy-in to expand your data quality is going to get a lot easier.</p><p><strong>How GX Cloud helps:</strong> GX Cloud’s data quality tests, called Expectations, are extremely expressive by design. They’re simple to build in GX Cloud’s interface and don’t require any coding. That makes them ideal for when you’re crunched for time or can’t give sustained attention to creating tests.</p><h3>Step 3: Give context to your future self</h3><p>Part two of “every minute of effort yielding maximum effects” is thinking forward. When a test fails, you’re going to need to do something about it.</p><p>There are times when you have the leisure to investigate at your own pace, but resolving data quality issues under pressure is not one of them. You need to accelerate your future work, and that means your data quality alerts need to be as informative as possible.</p><p>Time spent deciphering the ‘what’ and ‘why’ of an alert is time you can’t spend resolving it. And you absolutely can’t afford the time to be looking at the results of generic tests and trying to figure out if they indicate a real issue or not.</p><p>If you’ve implemented precise and expressive tests, you’re most of the way there! Just make sure the test information gets embedded in the alert.</p><p><strong>How GX Cloud helps: </strong>GX Cloud can deliver the alerts generated by your precise and expressive Expectations to your email as soon as each testing run completes.</p><p>That means each alert puts the context about what failed, how it failed, and when it failed right in front of you. You have the context you need to begin remediating data quality issues right away.</p><h3>Step 4: Have documentation</h3><p>If creating documentation isn’t your idea of a good time, it’s understandable. But unfortunately, that doesn’t make it less critical.</p><p>Good documentation — documentation that’s complete and up to date — is essential for a project to be effective long-term.</p><p>For your data quality process, that means documenting your tests. Each test’s documentation should include what aspect of the data is being tested, what the test is checking for, and the test’s parameters, and it should be updated every time you update the test.</p><p>If that’s overhead you don’t have time for, the answer is not to go without docs: it’s to either make the time or find a way to automate their production.</p><p><strong>How GX Cloud helps:</strong> Expectations are self-documenting. In GX Cloud, every Expectation has an edit history that automatically records its evolution. And every time an Expectation is run, the results include the Expectation’s complete configuration in plain, readable language.</p><p>So not only is GX Cloud’s documentation automatic and detailed, but it’s readable for even someone who’s never seen an Expectation before — making it easy to share your results or bring in others when you get a chance to grow your data quality process.</p><p>Automatic, detailed documentation that’s readable to anyone is one of the key ways that GX Cloud helps you do more, faster.</p><p>If your data quality work is getting deprioritized, take these four steps to deploy an effective set of ground-floor protections for your data.</p><p>GX Cloud provides everything you need for fast and simple implementation of a process to protect yourself against critical data quality issues. And because GX Cloud supports the entire data quality lifecycle, it has plenty of room for you to grow your data quality process as and when you’re able.</p><p>Don’t let shifting priorities from above keep you from starting to build a data quality process. GX Cloud gives you the tools you need to succeed now and in the future.</p><p><a href="https://hubs.li/Q02TwCsW0">Sign up for GX Cloud</a></p><p><a href="https://hubs.li/Q02TwD8r0">Request a GX Cloud demo</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=585c5d357a19" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Where GX Cloud fits in]]></title>
            <link>https://medium.com/@expectgreatdata/where-gx-cloud-fits-in-e1b2b7d1211f?source=rss-6048af210e26------2</link>
            <guid isPermaLink="false">https://medium.com/p/e1b2b7d1211f</guid>
            <dc:creator><![CDATA[Great Expectations]]></dc:creator>
            <pubDate>Wed, 02 Oct 2024 18:18:30 GMT</pubDate>
            <atom:updated>2024-10-02T18:20:39.130Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="silver puzzle pieces partially assembled and partially piled, with a single orange piece that has the GX logo on it in focus" src="https://cdn-images-1.medium.com/max/1024/1*YwgqnK-0ZDQZAnAIPoXT5g.jpeg" /><figcaption>In the landscape of data technology, where does GX Cloud go?</figcaption></figure><p>Do we need to reiterate the importance of data to a modern organization? No. We’re all on the same page by now.</p><p>But making the data happen and putting it to work effectively can be a real Wild West scenario. And data and analytics tools have proliferated exponentially.</p><p>The result: you’re just trying to solve a problem or fill a gap in your processes… and instead you end up adrift in a sea of revolutionary ML-powered platforms promising to enable your AI-ready data-driven future. In the cloud, obviously.</p><p>So plain and simple, here’s what GX Cloud does for you and how it relates to other kinds of data technology.</p><h3>GX Cloud + data sources</h3><p><em>Data sources: how any of this is even possible.</em></p><p><a href="https://hubs.li/Q02RV8fP0">GX Cloud</a> has a watchdog role for data sources. Use GX Cloud to make sure that the data coming to rest in your storage has been verified as high quality and suitable for its intended uses.</p><p>What are its intended uses? What does ‘high quality’ mean? That’s something your business and technical teams collaborate to specify, together, in GX Cloud.</p><p>How will ‘high quality’ be verified? Using the tests that your data’s stakeholders — the relevant business and data teams — build in GX Cloud.</p><p>How is the verification carried out? GX Cloud executes the tests (Expectations) that your data and business teams built.</p><p>Where is the verification carried out? GX Cloud orchestrates the execution of your Expectations in the data source where it already is.</p><p>GX Cloud’s tests are <a href="https://hubs.li/Q02RV9m70">extremely expressive</a>, so you can build in some serious nuance while Expectations remain perfectly comprehensible to a newcomer.</p><p>In other words: GX Cloud’s tests — aka Expectations — aren’t just the operational mechanism you use to verify a data source’s quality. They’re also the educational knowledge center that <em>documents</em> a data source’s quality standards.</p><h3>GX Cloud + ingestion technologies</h3><p><em>Ingestion technologies: getting data from there to here.</em></p><p>GX Cloud augments your ingestion technology. Use GX Cloud to make sure that only the good stuff actually gets ingested.</p><p>What’s ‘the good stuff’? That’s defined by your data and business teams as they collaborate to build quality tests (Expectations) in GX Cloud.</p><p>How is only the good stuff ingested? GX Cloud executes the tests that identifies the good stuff vs the bad stuff. With the bad stuff identified, GX Cloud can trigger actions that prevent low-quality data from being ingested and/or divert it to a separate location for remediation.</p><h3>GX Cloud + orchestration</h3><p><em>Orchestrators: pulling all the strings.</em></p><p>GX Cloud calls on orchestrators to react to its test results, and can be called upon in turn. Use GX Cloud to ensure high-quality vs low-quality data is treated differently and appropriately by your orchestrators.</p><p>How does GX Cloud identify high-quality vs low-quality data? With its tests, aka Expectations, which your data and business teams collaborated to build.</p><p>How can orchestrators react differently to high- vs low-quality data? GX Cloud indicate whether the data is high-quality at the dataset level and/or at the row level. An orchestrator can then take action to propagate or ingest high-quality data, reject or isolate low-quality data, or other appropriate action.</p><p>Does GX Cloud require a separate orchestrator in order to do anything useful with its monitoring results? No. GX Cloud has built-in orchestration capabilities that allow you to execute core orchestration functions natively. You can choose to use an external orchestrator if you prefer to do so.</p><h3>GX Cloud + observability</h3><p><em>Observability technologies: knowing the state of your data.</em></p><p>GX Cloud is observability technology. It focuses on proactive testing that checks for specific, known needs and requirements of the data.</p><p>How do the tests check for specific, known needs and requirements? Your technical and business teams work together in GX Cloud to define them.</p><p>How do the needs and requirements remain ‘known’ as the individuals who created them come and go? GX Cloud implements tests as highly-expressive Expectations, which document themselves to create a durable knowledge repository.</p><h3>GX Cloud + governance and data catalogs</h3><p><em>Governance and data catalogs: keeping track of what’s happening here.</em></p><p>GX Cloud is complementary to governance and data catalog technologies. Use GX Cloud to be confident that people can actually use the data that your catalogs and governance tools point them to.</p><p>How do you get confidence in your data? GX Cloud’s tests (Expectations) monitor your data against the specific, known needs and requirements that your data and business teams have collaborated to define.</p><h3>GX Cloud + analytics and consumption</h3><p><em>Analytics and consumption: the point of all this.</em></p><p>GX Cloud is a crucial prerequisite for analytics and consumption technologies. Use GX Cloud to ensure that you can trust the results that your analytics and consumption tools are producing.</p><p>Why can you trust your results? Because GX Cloud has verified that your data is suitable for its intended use — which includes its uses in these tools.</p><p>How does GX Cloud know what makes the data suitable? Your data and business teams have collaborated to determine that, using their collective experience from their respective areas of expertise.</p><p>How were your data and business teams able to collaborate effectively? Because GX Cloud provided an accessible, user-friendly SaaS interface where people can work together to create tests (Expectations) without needing any code.</p><p>How can you create effective data quality tests without code? GX Cloud’s tests, called Expectations, use plain language to define precisely what you expect your data to be.</p><p>How are the tests run? GX Cloud executes Expectations on your data in the location where your data already is.</p><p>What if an Expectation finds a problem? GX Cloud’s alerts can notify the relevant people immediately, and identify the problematic data to your orchestrators for isolation or rejection.</p><p>Because Expectations are expressive and context-rich, GX Cloud tells you which data is affected down to the individual columns and rows (if relevant), as well as the nature of the problem. Your data teams have the information they need to begin working on a fix right away.</p><p>That’s the rundown of how GX Cloud relates to other data technologies. Here’s the summary:</p><p><strong>For data sources</strong>, GX Cloud makes sure that the data coming to rest in them has been verified as high quality and suitable for its intended uses.</p><p><strong>For ingestion tools</strong>, GX Cloud ensures that only data you actually want gets ingested.</p><p><strong>For orchestrators</strong>, GX Cloud makes it easy to disseminate high-quality data while isolating low-quality data, and provides meaningful and information-rich test results for propagation to other systems and dashboards.</p><p><strong>For your observability</strong>, GX Cloud implements proactive testing that checks for specific, known needs and requirements of the data.</p><p><strong>For your governance and data catalog tools</strong>, GX Cloud provides monitoring and test results that allows people to make informed decisions about using the data that your governance tools direct them to.</p><p><strong>For your analytics and data consumption</strong>, GX Cloud is a critical tool that ensures you can trust the results of your analyses and have confidence in the correctness of your operational data use.</p><p>Data teams across the globe are discovering how to trust their data with GX Cloud. To join them, <a href="https://youtu.be/1RrC9z4XKQQ?feature=shared">watch a conversation with one of our customers</a> or get started right away:</p><p><a href="https://hubs.li/Q02RV7J90">Try GX Cloud free</a></p><p>or</p><p><a href="https://hubs.li/Q02RV7Pj0">Request a demo</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e1b2b7d1211f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring data quality: missingness]]></title>
            <link>https://medium.com/@expectgreatdata/exploring-data-quality-missingness-ad9c07c790d2?source=rss-6048af210e26------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad9c07c790d2</guid>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data-quality-dimensions]]></category>
            <dc:creator><![CDATA[Great Expectations]]></dc:creator>
            <pubDate>Wed, 18 Sep 2024 15:47:17 GMT</pubDate>
            <atom:updated>2024-09-18T21:48:17.745Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bv5lSr1B7yTdeDzlrhVmtA.png" /></figure><p><em>This post is the second in a seven-part series that uses a regulatory lens to explore how different dimensions of data quality can impact organizations — and how GX can help mitigate them. Read </em><a href="https://hubs.ly/Q02Q38QK0"><em>part 1 here</em></a><em>.</em></p><p>Belonging to a data team isn’t just about managing information. It’s about making sure everyone in your company can make fully informed decisions. Ultimately, your work shapes the strategic direction of your organization.</p><p>In this blog post, we’ll explore how missing data — an obvious issue, but one with more subtleties than you might expect — can negatively impact your data… and what you can do about it.</p><p>As we delve into the world of missing data, we’ll once again use the lens of FinTrust Bank, a fictional financial institution. We’ll explore how a high degree of data missingness can impact risk assessment and regulatory compliance, and how you can effectively use GX to manage and mitigate the challenges of missing data.</p><h3>A pervasive challenge</h3><p>Perhaps unsurprisingly, data missingness is a widespread issue affecting organizations across essentially every sector.</p><p>In some ways it might seem to be a relatively mundane data quality issue — it doesn’t take a deep technical background to spot an empty field — but that doesn’t mean it’s low-stakes. In healthcare, missing patient data can impact treatment decisions; in manufacturing, gaps in production data can lead to inefficiencies, health and safety issues, or low-quality products.</p><p>And in the financial sector, missing financial data can lead to off-base risk assessments, operational inefficiency, and more.</p><p>For an example, we return to FinTrust Bank, where their recent data audit has discovered that key data was often missing from datasets. Specifically, critical fields in loan application data are routinely incomplete.</p><h3>The ripple effects of missing data</h3><p>The implications of missing data extend throughout FinTrust Bank:</p><ol><li><strong>Skewed analytics</strong> due to incomplete data can lead to biased analyses and flawed decision making.</li><li><strong>Regulatory misreporting</strong> because of omitted data can result in penalties and increased scrutiny.</li><li><strong>Operational inefficiencies</strong> happen when remediation is dependent on ad-hoc manual intervention: employees spend time on repairing data instead of their main job, slowing down the bank’s processes and creating an opportunity for human error to introduce a new quality issue.</li><li><strong>Missed opportunities</strong> from incorrect risk analysis and misclassified opportunities.</li></ol><h3>Leveraging GX to tackle missing data</h3><p>Samantha, the data engineering leading the quality improvement effort at FinTrust Bank, chose GX as their data quality solution. That means she has access to its robust tools for detecting and quantifying missing data, so she can quickly build a comprehensive understanding of their data’s completeness.</p><p>With GX Cloud, Samantha’s team collaborated with each other and with subject experts outside of the team to develop and refine Expectations that built a meaningful understanding of their data’s expected completeness.</p><p>That included identifying where missing data is actually a problem, identifying the degree of missingness that warrants remediation, and setting up alerts to notify the appropriate stakeholders immediately when there’s a missing data incident.</p><p>With GX Cloud’s easy-to-use SaaS interface, FinTrust Bank team members could work simultaneously, examine results, and track changes in real time. This significantly accelerated their data quality work.</p><h3>Key Expectations for missing data</h3><p>Samantha’s team quickly implemented the key Expectations for missing data. They used ‘Expect column values to be null’ to check for the presence of null values in critical fields like loan amounts and application dates, where any amount of missing data is unacceptable.</p><p>Other fields, like applicant credit score, required more nuance. On occasion, an applicant is applying for credit for the first time, and has no credit score. This is unusual but not unheard of, so Samantha’s team set a threshold allowing up to 1% of credit score values to be null.</p><p>By allowing this small degree of flexibility, FinTrust Bank stakeholders won’t be unnecessarily alerted every time a first-time credit-seeker applies — but they <em>will</em> be notified if the amount of missing data reaches unexpected levels, indicating a true missing-data problem.</p><p>After applying missingness data quality checks across FinTrust Bank’s data, Samantha’s team uncovered some crucial insights about their data completeness. They found that 3% of loan amounts were missing, 5% of credit scores unavailable, and 7% of employment history records incomplete.</p><h3>Strategies for addressing missing data</h3><p>With insights from GX, Samantha’s team developed a comprehensive strategy to tackle their data gaps:</p><ul><li>They redesigned the loan application form, which dramatically reduced missing employment histories.</li><li>They applied statistical techniques to fill loan amount gaps while maintaining data integrity, flagging all imputed values for transparency.</li><li>They standardized the definition of ‘missing’ data across departments, ensuring consistent identification and handling of the issue.</li><li>They automated their data quality checks using GX Cloud, ensuring that future errors could be identified quickly using a transparent set of criteria.</li><li>They conducted root cause analyses to improve their other data collection strategies and prevent future data gaps.</li></ul><p>Finally, they integrated the automated alerts that they had set up in GX with their team communication tools. When critical data goes missing in the future, the right people will now find out immediately.</p><h3>Outcomes and impacts</h3><p>FinTrust Bank saw immediate results from their improved data missingness checks. The overall rate of missing data dropped by 60%, while the time spent on manual data cleaning decreased by 40%. Perhaps most importantly, the accuracy of their risk assessment models improved by 25%.</p><p>As an example of improved outcomes, the higher-quality data was used to assess a high-value commercial loan application. The data indicated that FinTrust Bank could offer competitive terms while maintaining appropriate risk management, allowing them to secure a $10m deal.</p><p>Later, the original, pre-GX data was used to reproduce the assessment process on the loan application. Samantha’s team found that the old data’s conclusion would have led FinTrust Bank to pass on the opportunity.</p><h3>Elevating data quality: a strategic imperative for competitive advantage</h3><p>Tackling missing data isn’t just about compliance. It’s a strategic imperative that can transform your organization’s decision-making capabilities. FinTrust Bank’s example shows that with the right tools and approach, significant improvements are within reach:</p><ul><li>More accurate risk assessment</li><li>Better decision making</li><li>Increased operational efficiency</li><li>Enhanced regulatory compliance</li></ul><p><a href="https://hubs.ly/Q02Q383f0">As your data quality work continues</a>, you’ll continue to evolve your understanding of how and when missing data most affects your organization. GX Cloud makes it easy to reflect those changes in your tests, so your Expectations stay a reliable central resource about how your organization handles missing data.</p><p><a href="https://hubs.ly/Q02Q383f0">Click here to read more about how GX Cloud can help you manage your missing data.</a></p><p><em>Stay tuned for part 3 of our series, where we’ll explore how unexpected data volumes can cause issues.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad9c07c790d2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring data quality: schema validation]]></title>
            <link>https://medium.com/@expectgreatdata/exploring-data-quality-schema-validation-ad1e796c641f?source=rss-6048af210e26------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad1e796c641f</guid>
            <category><![CDATA[data-quality-dimensions]]></category>
            <category><![CDATA[schema-validation]]></category>
            <category><![CDATA[data-quality]]></category>
            <dc:creator><![CDATA[Great Expectations]]></dc:creator>
            <pubDate>Mon, 16 Sep 2024 21:55:10 GMT</pubDate>
            <atom:updated>2024-09-16T22:04:20.286Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*H6LfAmwv8L6BBMOybc1qEA.png" /><figcaption>Image created by GX</figcaption></figure><p><em>This post is the first in a seven-part series that uses a regulatory lens to explore how different dimensions of data quality can impact organizations — and how GX can help mitigate them.</em></p><p>In this post, we’ll begin to explore how FinTrust Bank, a fictional financial institution, tackles its data quality challenges. And we’ll see how Great Expectations, via its GX data quality platform, mitigates those challenges and ensures the integrity of regulatory reporting.</p><p>Like most modern organizations, FinTrust Bank is inundated with huge amounts of data that are critical for their decision-making. And as a financial institution, they’re in one of the industries that needs to be particularly concerned with regulatory compliance.</p><p>Inaccurate or incomplete data can lead to flawed financial statements, misguided business decisions, and severe regulatory penalties. It erodes trust with stakeholders and damages the institution’s reputation.</p><p>Data teams at financial institutions don’t start out in an ideal place. Vital data often arrives in a less-than-refined state, riddled with inconsistencies and errors. Data teams have the challenge of building a data quality process that can transform this chaos into accurate and reliable data — with extra-high stakes from the regulatory implications.</p><p>So at organizations like FinTrust bank and its real-life counterparts, ensuring data integrity isn’t just a task: it’s a strategic imperative.</p><p>Leading the charge at Fintrust Bank is Samantha, a data engineer with over a decade of experience in handling financial data. Samantha stands in for the data teams at financial institutions who are committed to upholding high standards of data quality, despite the obstacles they face.</p><h3>A data audit revelation</h3><p>Samantha and her team embarked on a comprehensive data audit to assess the current state of their data assets. And their findings were alarming:</p><p>1. <strong>Missing fields</strong>: Key financial data — such as transaction amounts, account numbers, and timestamps — were often missing from the datasets. This omission could lead to incomplete regulatory reports, which might result in substantial fines.</p><p>2. <strong>Incorrect column types</strong>: Several columns contained data of unexpected types. In one case, a ‘transaction_amount’ column, which should contain numerical values, included string entries. This mismatch could lead to calculation errors and inaccurate financial reports.</p><p>3. <strong>Logical inconsistencies</strong>: Certain numerical fields, such as transaction amounts, contained values that defied logic (negative deposits and positive withdrawals are not real things). These anomalies could severely distort financial metrics and forecasts.</p><p>FinTrust’s quarterly compliance report is a cornerstone of its regulatory submissions. After auditing it, Samantha’s team found that 15% of the required fields were either missing or contained erroneous data. These discrepancies put FinTrust Bank in direct danger of substantial penalties from regulatory bodies, not to mention reputational damage.</p><p>Given the audit findings, Samantha knew the team had to act fast. FinTrust needed a robust and systematic way to improve the integrity of its data and ensure that every piece of information was accurate, complete, and compliant.</p><h3>Schema validation to the rescue</h3><p>Following the alarming results of their audit, Samantha and her team decided to tackle schema inconsistencies first. Their reasoning: without a solid foundation for the data structure, solutions for more complex data quality issues wouldn’t hold up. A robust schema validation process was now essential for FinTrust.</p><p>Schema validation defines a clear blueprint for data structure and ensures that the data conforms to predefined structures. When working in the financial industry with its sensitive data and stringent regulatory requirements, schema validation provides crucial risk mitigation, support for compliance, and streamlined operations.</p><h4>What is a data schema?</h4><p>The schema is the structural blueprint of a dataset. It defines the overall organization of information, including elements like column names and data types. Ensuring that data adheres to its expected schema is fundamental to maintaining its quality and reliability.</p><p>The GX platform equipped Samantha’s team with the schema validation tools they needed. With GX, the data quality team could easily <a href="https://hubs.ly/Q02Q37yl0">define their expected data schema and automatically validate whether their data met these standards</a>.</p><h4>Key schema Expectations in GX</h4><p>Let’s look at the key schema Expectations that Samantha’s team leveraged in GX.</p><ul><li><strong>Expect column to exist</strong>: Ensures that all essential columns are present and accounted for with the expected name, detecting missing data fields or renamed columns.</li><li><strong>Expect table column count to equal</strong>: Verifies that only the expected number of columns are present, a safeguard against unexpected data piggybacking on legitimate data.</li><li><strong>Expect column values to be of type</strong>: Confirms that each column contains the correct type of data — dates, numbers, text, etc. This drastically reduces the need for manual oversight of the data.</li><li><strong>Expect column values to be in type list</strong>: Ensures that the values in a specified column are within a specified type list. This is especially useful for columns with varied permissible types, such as mixed-type fields often found in legacy databases.</li></ul><p>These Expectations can be applied directly from the GX Cloud UI or using the GX Core Python framework, streamlining the validation process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7wNGbYQGXWvUk5k_g1pTyg.gif" /><figcaption>Screen capture of creating Expectations in GX Cloud</figcaption></figure><h3>Expect the unexpected (and prevent it)</h3><p>Using GX for schema validation, Samantha and her team were able to transition from firefighting schema-based data quality issues to proactive assurance work.</p><p>Implementing these Expectations created an immediate impact for FinTrust bank. FinTrust Bank’s error rates in its reporting dropped from 5% to 0.25%, significantly reducing their risk of regulatory penalties.</p><h3>Beyond schema validation</h3><p>While establishing your initial schema validation is crucial, it’s not the end of your schema validation work. <a href="https://hubs.ly/Q02Q37yl0">Your schema will continue to evolve</a>, so you need to have a way to manage the corresponding changes and keep your validation up to date. By implementing structured processes for schema versioning and maintaining comprehensive validation practices, you can anticipate and mitigate potential disruptions.</p><p><a href="https://hubs.ly/Q02Q37yl0">See how GX can help you implement schema validation!</a></p><h3>Join the discussion</h3><p>How has schema validation transformed your data processes? <a href="https://discourse.greatexpectations.io/c/share/18">Join the discussion in our community forum</a> to exchange ideas and experiences with other data practitioners. Together, we can improve best practices for data quality in every industry.</p><p><em>Stay tuned for Part 2 of our series, where we explore data missingness and related strategies for improving your data quality.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad1e796c641f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The GX platform: a short introduction]]></title>
            <link>https://medium.com/@expectgreatdata/the-gx-platform-a-short-introduction-8eb5f15df082?source=rss-6048af210e26------2</link>
            <guid isPermaLink="false">https://medium.com/p/8eb5f15df082</guid>
            <category><![CDATA[data-quality-management]]></category>
            <category><![CDATA[data-quality-tool]]></category>
            <category><![CDATA[data-quality]]></category>
            <dc:creator><![CDATA[Great Expectations]]></dc:creator>
            <pubDate>Thu, 22 Aug 2024 14:43:00 GMT</pubDate>
            <atom:updated>2024-08-27T17:53:55.614Z</atom:updated>
            <content:encoded><![CDATA[<p>At Great Expectations, we’re building an end-to-end solution for your data quality process. And that solution — the GX platform — has two components: GX Cloud and GX Core.</p><p><a href="https://hubs.li/Q02MM7Gx0">GX Core</a>, our open source library, is the world’s <a href="https://seattledataguy.substack.com/p/the-state-of-data-engineering-part-b61">most popular data quality</a> framework. It has hundreds of contributors, thousands of customer deployments across many industries and geographies, and millions of downloads every month.</p><p>GX Core provides the platform’s technical data quality capabilities. The biggest of these is, of course, <a href="https://hubs.li/Q02McfP70">Expectations</a> — the clear, intuitive way in which GX implements data quality tests. And as an open source Python framework, GX Core gives data teams a lot of flexibility and control, which is one reason that it’s so popular.</p><p>Obviously, the technical implementation of Expectations alone is a critical part of our platform, not to mention all of the other capabilities that GX Core provides. This criticality illustrates exactly why we’re committed to developing, advancing, and maintaining GX Core in the future. Without GX Core, there’s no GX platform at all.</p><p>But a complete, end-to-end data quality solution needs more than just the technical capabilities of connecting to data, running tests, and displaying results — no matter how effective or expressive those capabilities might be!</p><p>To effectively implement a data quality process, you need infrastructure to host it. You need a way for nontechnical stakeholders to interact with it. You need a place for data teams and stakeholders to record their joint decisions about the appropriate tests and expected qualities of the data.</p><p>You need a way to automate your tests, and a way to be notified when your tests find something significant. You need tools to view your results and store your historical data. In many scenarios, you need access controls to ensure that only approved people can edit or even access the tests.</p><p>None of those things are part of GX Core.</p><p>In the past, GX Core users had to create those capabilities for themselves, using additional tools, custom development, or both — and <a href="https://www.youtube.com/watch?v=9fZi-0qlL60">they</a> <a href="https://www.youtube.com/watch?v=WLXffNScH8U">did</a>! That option is still on the table for teams who want it.</p><p>But our goal with the GX platform is to provide a complete solution. Something that a data team can pick up and start using immediately, with everything they need ready to go. Data teams are busy, and we want to empower them to focus all their energies on understanding their data, implementing tests, and resolving issues.</p><p><a href="https://hubs.li/Q02MchRl0">GX Cloud</a> is the component of the GX platform that makes it easy to harness the powers of GX Core.</p><p>GX Cloud contributes by providing a fully hosted SaaS infrastructure, so startup is just a matter of creating an online account. It’s quick and easy.</p><p>Once you’re logged in, its intuitive user interface is accessible to everyone: data teams to nontechnical stakeholders. This not only centralizes collaborative work — work which is crucial to an effective data quality process — but makes it significantly more straightforward and easier for busy stakeholders to meaningfully engage with data quality efforts.</p><p>Plus that interface enables a whole suite of additional features that support your data quality process: in-app scheduling, configurable notifications, historical results charts, in-context notes and changelog, and role-based access controls, among other things.</p><p>By using GX Cloud, data teams can get the power of Expectations and GX Core, plus start-to-finish support for the rest of their data quality process.</p><p>If you’re an analogies person, think of the GX platform as a car. GX Core is the engine: a powerful machine, but it needs additional components to help you get anywhere. GX Cloud is the rest of the car: everything you need to actually go places, and harnessing the engine’s power to make it happen.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mh9mAmcgC45OHT2ExVyjSA.jpeg" /><figcaption>GX Core is the engine of the GX platform; GX Cloud is the rest of the car. Together, they create the complete solution of the GX platform.</figcaption></figure><p><a href="https://hubs.li/Q02Mc2z60">Try GX Cloud, now in public preview.</a></p><p>Try <a href="https://github.com/great-expectations/great_expectations">GX Core on GitHub</a> or <a href="https://pypi.org/project/great-expectations/">Pypi</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8eb5f15df082" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Down the rabbit hole: Why data quality is actually really difficult]]></title>
            <link>https://medium.com/@expectgreatdata/down-the-rabbit-hole-why-data-quality-is-actually-really-difficult-f08ccff3770e?source=rss-6048af210e26------2</link>
            <guid isPermaLink="false">https://medium.com/p/f08ccff3770e</guid>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Great Expectations]]></dc:creator>
            <pubDate>Tue, 11 Jun 2024 19:39:08 GMT</pubDate>
            <atom:updated>2024-06-11T19:39:08.362Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*TSPOzS_1_0vwYbAB7OvY6A.jpeg" /></figure><p>The world is full of things that feel like they <em>should</em> be solved problems but aren’t. Like not getting reminders to schedule appointments you already scheduled or stopping mail addressed to the person who lived in your house ten years ago.</p><p>Consistently having high-quality data that you can confidently use for its intended purpose is one of those things.</p><p>Why are dashboards constantly breaking, why are reports missing information, why are the analytics always late? We can generate all this data automatically, but trying to <em>do</em> anything with it takes so much manual effort.</p><p><strong>Why don’t we have the tools to solve this universal problem?</strong></p><p><strong>That’s not the right question.</strong></p><p>Let’s back up a minute. You’re trying to do something with your data. You want to see what users are doing in your app or whether your machine learning model acts as you expected. What ads are most effective with which customer segments? Is your application meeting your uptime SLAs? How much money did your company make last quarter? Do you have everything you need for your regulatory reporting?</p><p><strong>Whatever you’re using your data for, the basic reason you’re using it is that you want to learn something.</strong></p><p>If you already had the knowledge you were looking for, you wouldn’t need to learn it. So, by definition, you are going to be surprised by what the data says.</p><p>(Is your level of surprise going to be more “aha” than “shocked and astounded”? Yes, maybe. But the situation is the same regardless.)</p><p>The pivotal question is: <em>why</em> are you being surprised? There are two possibilities:</p><ol><li>You’re being misinformed about the state of the world because the data has an issue.</li><li>There is new information about the state of the world, which the data has revealed to you.</li></ol><p><strong>You can see the informational asymmetry: to learn, you have to be surprised. But you can be surprised and not learn.</strong></p><p>To know that you’re learning about your app, your ML model, your ads, your uptime, your income… you have to be confident that nothing went wrong while collecting or processing your data.</p><p>And maintaining confidence in your data collection and processing hinges on how you deal with those surprises. The key is:</p><p><strong>Understanding the quality of your data isn’t the process of finding the <em>right</em> answer; it’s the process of ruling out all the <em>wrong</em> answers. Which takes much longer because there’s no possibility you can stop early.</strong></p><p>And you probably aren’t going to be able to stop after one round of it, either. You need confidence in the data that you’re being surprised by right now, yes, but underlying it is more data that <em>also</em> came from somewhere, and under that is yet more data…</p><p>The higher the stakes, the deeper the rabbit hole. Following it all the way down is a lot of work.</p><p>It would be bad enough if it were a lot of work that you could do yourself. But to get full context and understanding of your data, where it comes from, how it’s used, and why, you need the collective knowledge of <em>everyone</em> who works with or uses the data. For high-stakes, high-usage data, that could be dozens of people (or even more) for a single dataset, let alone the entire pipeline behind it.</p><p><strong>That’s not just a lot of work; it also requires a lot of people.</strong></p><p>And a lot of people means infinitely more opportunities for miscommunication, or for completely conflicting perspectives on exactly the same data (neither of which is wrong, exactly), or for someone to leave and take crucial knowledge with them.</p><p>At the risk of pushing the metaphor too far, the rabbit hole is actually a full-on rabbit warren.</p><p><strong>So, is it hopeless? Absolutely not.</strong></p><p>We didn’t say all that to make you feel depressed or overwhelmed. We said it to put the right lens on data quality.</p><p><strong>The issue isn’t: <em>we don’t have the tools</em>. It’s: <em>we haven’t been putting the tools together in the right ways yet.</em></strong></p><p>Because the technological problem in data quality isn’t pure computation. It’s just as much, if not more, about communication. Evolving, collaborative, co-co-co-co-…authored communication that evolves over time without losing history.</p><p>Communication that’s fully, totally centralized, so everyone sees the same thing no matter where they are in the rabbit tunnels.</p><p>Knowing that, it’s easy to see why data quality isn’t ‘solved’ — high-quality data will always be a moving target. Ensuring high-quality data means building a durable <em>process</em>.</p><p>Nothing turnkey can create an effective data quality assurance process with a button press (or even a series of button presses). Because having confidence in the quality of your data needs more than the right software tools. It requires insight and investigation and communication and cooperation, from the specific people at your specific organization who work with your specific data.</p><p><strong>And now that we understand the real challenge, we’re empowered to make real progress with developing data confidence.</strong></p><p>We can pick the right tools, bring in the right people, and build the right processes for handling all the surprises waiting in our data.</p><p>GX Cloud from Great Expectations helps you build confidence in your data with a friendly SaaS interface and plain-language tests and results, so it’s easy for data teams and stakeholders to communicate and collaborate: <a href="https://greatexpectations.io/cloud">try it today</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f08ccff3770e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your data tests failed! Now what?]]></title>
            <link>https://medium.com/@expectgreatdata/your-data-tests-failed-now-what-80b3c0b959f8?source=rss-6048af210e26------2</link>
            <guid isPermaLink="false">https://medium.com/p/80b3c0b959f8</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-pipeline]]></category>
            <category><![CDATA[dataops]]></category>
            <category><![CDATA[data-validation]]></category>
            <dc:creator><![CDATA[Great Expectations]]></dc:creator>
            <pubDate>Tue, 17 Nov 2020 20:04:51 GMT</pubDate>
            <atom:updated>2024-03-25T17:40:58.208Z</atom:updated>
            <content:encoded><![CDATA[<h3>Your data tests failed! … now what? [updated 2024]</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kpyv0Lp-wQfSSeH0HwIp_g.jpeg" /><figcaption>Do you have a plan for when your data quality tests start catching errors?</figcaption></figure><p>Imagine this scenario: you know that pipeline tests on your data are absolutely crucial to your overall data quality. You’ve taken the steps to put those tests into place, and whether you’ve used an off-the-shelf tool, an open source option, or homegrown code, you’re ready to sit back, relax, and watch your pipelines deliver beautiful, high-quality data to your stakeholders.</p><p>You imagine your test foiling a dastardly batch of unsuitable data before it can infiltrate your dataset. And that’s when you realize you missed a step: what’s going to happen when your tests actually fail?</p><p>How are alerts going to come in? <em>Are</em> alerts going to come in? Is anyone monitoring the alerts? Who’s in charge of responding to alerts? How will you find out what went wrong? <em>How do you fix the data issues that your pipeline tests find?</em></p><p>As exciting as it is to implement a suite of solid pipeline tests, a significant part of the art of data quality testing is figuring out how to effectively respond when you detect a data problem.</p><p>In this post, we’ll walk through some of the key stages of responding to data tests and outline the major considerations for developing your team’s data quality strategy.</p><p>The stages we’ll discuss are:</p><ul><li>System response to failure</li><li>Logging and alerting</li><li>Alert response</li><li>Root cause identification</li><li>Issue resolution</li></ul><p>Plus, we’ll talk about the importance of maintaining stakeholder communication throughout these stages.</p><h3>System response</h3><p>The first line of response to a failed data test happens before any humans are notified, when the system’s automated responses kick in. These responses decide whether or not to continue any pipeline runs and are typically some variation on the following forms:</p><ul><li><strong>Do (almost) nothing</strong>: the system logs the failure or sends an alert, but the pipeline continues to run.</li><li><strong>Isolate the problem</strong>: the system does something to quarantine the “bad” data, such as moving rows that fail tests to a separate table or file. The rest of the data continues normally through the pipeline.</li><li><strong>Stop the pipeline</strong>.</li></ul><p>It’s usually advisable to vary the system’s response depending on the severity of the detected issue and the downstream use case. Some types of issues might prompt a warning to stakeholders, but the pipeline can keep running; other critical errors could merit stopping the pipeline altogether.</p><h3>Logging and alerting</h3><p>While it’s absolutely possible to simply write all your data validation results to a log and review them at your leisure, in our experience, you’ll almost always have tests that are critical enough for failures to require proactive alerting.</p><p>Some factors to consider when determining your approach to alerting are:</p><ul><li><strong>Which errors absolutely <em>need</em> alerting, and which can simply be logged?</strong> If stakeholders receive too many alerts for issues that turn out to be ignorable, they’ll begin ignoring all alerts. To avoid this alert fatigue, be conscientious about choosing which errors to alert on.</li><li><strong>What medium are you using? </strong>After you’ve carefully chosen only the most critical errors for alerts, don’t undermine all your work by sending them to a busy Slack channel or someone’s inbox where they’re easy to overlook. Make sure your alerts are distinguished from regular status reports and other routine information so the recipients know what to look at. Using a tool like PagerDuty can help you fine-tune your alert management to the level of severity and responsiveness that each one needs.</li><li><strong>How timely are they?</strong> Do they get sent out at a certain time, or will they just show up? If your alert mechanism fails, will anyone notice?</li></ul><h3>Alert response</h3><p>Alongside your plan for producing alerts, you need to figure out who’s going to see and respond to them. Some elements to take into account are:</p><ul><li><strong>Who gets notified and when?</strong> Of everyone that touches your data — upstream producers, downstream consumers, pipeline owners — who’s relevant to a particular alert?</li><li><strong>Who’s responsible for acknowledging and investigating?</strong> Knowing something is up is different from doing something about it, and there needs to be one point of ownership for each alert. Different alerts might have different owners, but without a clear plan, issues can be overlooked or ignored, causing frustration to everyone.</li><li>Consider whether an on-call rotation can help with clearly assigning responsibility — often, it will. Having a designated point person responding to alerts creates a clear chain of responsibility. Importantly, don’t let the phrase “on-call” give you a false sense of urgency: in this scenario, it’s entirely possible that <em>being on call</em> takes place entirely within regular business hours. Don’t set anyone up to get 2 am phone calls unless you genuinely need that level of responsiveness.</li><li><strong>Are your notifications informative?</strong> Do the recipients of your alerts know not just what’s happened but also what implications it has? It’s especially important to ensure that your data consumers understand the ramifications of an alert and know what steps to take to get more information about the problem or potential solutions. Hint: having a clear point of contact, like an on-call engineer, often helps with this!</li></ul><h3>Stakeholder communication</h3><p>Now that you’ve found out about a test failure, it’s tempting to jump right into working on it and figuring out what’s going on. But it’s important to make sure that all the data’s stakeholders know something’s going on as soon as possible — ideally before they have a chance to see the effects in their own work.</p><p>This is pretty standard incident response stuff. But data quality problems often manifest in subtler ways than, for example, a web app bug causing a 404.</p><p>You’ll want to tune your communication to the severity of the issue and the stakeholders involved. Sometimes an automated alert might be good enough; other times, you might need a playbook where the team makes personal contact with affected groups.</p><p>It’s important to keep an open line of communication with your stakeholders so you can give them updates on the issue resolution process, answer any questions, and, if <em>absolutely</em> necessary, make ad-hoc fixes to address urgent data needs. (Since ad-hoc fixes add another opportunity for complexity in an area where you’re already having a problem, think twice — or even three or four times — before you decide to go there.)</p><h3>Root cause identification</h3><p>At a high level, data test failures fall into one of these buckets:</p><ul><li><strong>The data is actually fine</strong>: Sometimes, it’s the test that needs to be adjusted. This can happen when, for example, an unusual but correct outlier occurs for the first time since implementation.</li><li><strong>The data is broken but fixable</strong>: A straightforward example of this is an incorrectly-formatted date or phone number. The key to ‘fixability’ in this sense is that the data can be repaired within the scope of the current pipeline.</li><li><strong>The data is broken and not fixable</strong>: An example of this situation is when the data is missing values. This category isn’t meant to express that the data is unfixable in a universal sense but that you’d have to go outside of the current pipeline to do it: for instance, by using a data validation service to fill in missing address fields.</li></ul><p>Data issues that arise at the data ingestion/loading stages often stem from changes that the data team mostly doesn’t control. Some common examples include:</p><ul><li>Delayed data deliveries leading to out-of-date data.</li><li>Table properties like column names or types changing unexpectedly.</li><li>Values and ranges diverging from what’s expected because of changes in how the data is generated.</li></ul><p>Another major cause of data ingestion problems is issues with the actual ingestion runs or orchestration: when processes hang, crash, or get backed up due to long runtimes. These problems often manifest themselves as stale data.</p><p>So how do you identify the root cause of data ingestion issues? The key is to be methodical: identify the exact issue that’s actually happening and then what’s causing that issue.</p><p>To be successful at this, you can’t take problems and test failures at face value. For example, a test for NULL values in a column could be failing because some rows actually do have NULL values … or because that column no longer exists. Those are two very different problems.</p><p>Once the problem is clear, <em>then</em> you can start investigating.</p><p>Of course, we can’t list every possible cause of a data quality problem in this post, but here are some good initial avenues to investigate:</p><ul><li><strong>Recent changes to ingestion code</strong>: Ask your team (or the team in charge of ingestion) if there have been any recent changes or updates, or go through your version control log.</li><li><strong>Crashed processes or interrupted connections</strong>: Log files are usually helpful and the best place to start.</li><li><strong>Delays in data delivery</strong>: AKA, is the problem the system you’re ingesting from or upstream from that? Check whether the system you’re ingesting from is getting the source data in time.</li><li><strong>Upstream data changes</strong>: Check the state of the source data, and get the data producers to confirm whether it’s as they expect (or not).</li></ul><p>Finally, while data ingestion failures are often outside of your control, test failures on the transformed data are usually caused by changes to the transformation code. One way to head off these unexpected side effects is to enable data pipeline testing as part of your development and CI/CD processes.</p><p>By enabling engineers and data scientists to automatically test their code against, for example, a golden data set, you can decrease the likelihood that data-quality-problem-producing code will get into production.</p><h3>Issue resolution</h3><p>So you’ve identified the problem … how do you fix it?</p><p>It won’t surprise you that there’s no single approach to fixes that can work with every root cause.</p><p>But consider the three buckets of test failures we discussed earlier. Each of these buckets has an associated type of fix that can help your test go green again:</p><ul><li><strong>The data is actually fine</strong>: Update your tests to account for your new knowledge.</li><li><strong>The data is broken but fixable</strong>: You could try rerunning your pipelines, potentially with increased robustness toward disruptions like connection timeouts or resource constraints. You can also change (fix) your pipeline code; ideally, you can also add a mechanism that allows engineers to test their code so the same issue won’t happen again.</li><li><strong>The data is broken and not fixable</strong>: You might need to connect with data producers and have them reissue the data if possible. There might also be situations where you need to isolate the “broken” records, datasets, or partitions until the issue is resolved. And, especially when you’re dealing with third-party data, you might have to stop using the data for good: the producer might delete, modify, or stop updating the data to the point where it’s simply no longer suitable for your use case.</li></ul><h3>Now your tests pass, but …</h3><p>Unfortunately, even when your tests are passing with flying colors, you need to consider that you might not be testing for everything. And it’s nearly impossible to write data tests for every single possible data problem before you encounter it for the first time.</p><p>That’s true for big, obvious factors too, not just very rare edge cases … as Abe Gong, Great Expectations’ CEO, can attest:</p><blockquote><em>I once managed a daily ingestion pipeline that would alert if record counts </em>dropped<em> significantly from one day to the next, since that was usually our biggest concern. Little did I know that a bug in our pipeline would accidentally </em>double<em> the record counts in size, which besides some “hmm, those pipelines are running very slow today” comments aroused shockingly little suspicion — until a human actually looked at the resulting dashboards and noticed that our user count had skyrocketed that day.</em></blockquote><p>So what can you do to make your test more robust against these “unknown unknowns”? We’ve been working on this problem, and while we can’t say we’ve solved it yet, here’s what we recommend starting with.</p><ul><li><strong>Automated profilers</strong>: Use an automated profiler to generate data tests, specifically to increase test coverage in areas that might not be totally obvious to you. For example, you might not consider testing for the mean of a numeric column if you usually define it in terms of min and max — but additional tests could help you detect distribution shifts <em>within</em> that range. Use separate test suites and thoughtful alerting levels to make sure you don’t overwhelm yourself with notifications.</li><li><strong>Socialization</strong>: Don’t have engineers work on data tests solo. Socialize data tests within the team and do code reviews of tests whenever they’re added or modified, just like you would with actual pipeline code. This makes it easier to surface the pipeline team’s assumptions about the data and highlight any shortcomings in the tests.</li><li><strong>Manual spot checks</strong>: Automation is great, but someone who’s operationally familiar with the data can often see when something’s “off,” even if there’s no test in place. A final step of your data quality strategy can be implementing periodic audits of your data assets, which includes manual spot-checking alongside profiler-assisted checks and, importantly, making sure that your tests are still complete, accurate, and actually running as expected.</li></ul><h3>Summary</h3><p>In this post, we reviewed some key areas to consider when you’re implementing data validation in your pipelines. Developing and running tests is only one aspect of your data quality strategy, though admittedly an important one.</p><p>But just as important is what happens after those tests run: who gets alerted and how, assigning ownership of the alert response, communicating with stakeholders, doing effective root cause analysis, and actually fixing the issue. If you do these things well, they’ll take real time and effort: you need to give them just as much consideration as you do the tests themselves.</p><p>For some concrete examples, check out how companies like <a href="https://greatexpectations.io/case-studies/how-rent-the-runway-uses-gx-to-make-data-more-accessible-and-get-more-time">Rent the Runway</a>, <a href="https://greatexpectations.io/case-studies/provectus">Provectus</a>, and <a href="https://greatexpectations.io/case-studies/how-komodo-health-uses-gx-to-safeguard-their-data-pipelines-with-quality">Komodo Health</a> have integrated GX into their data workflows.</p><p><em>This article was updated March 25, 2024.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=80b3c0b959f8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Down with pipeline debt: introducing Great Expectations [updated 2024]]]></title>
            <link>https://medium.com/@expectgreatdata/down-with-pipeline-debt-introducing-great-expectations-862ddc46782a?source=rss-6048af210e26------2</link>
            <guid isPermaLink="false">https://medium.com/p/862ddc46782a</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Great Expectations]]></dc:creator>
            <pubDate>Wed, 21 Feb 2018 00:38:27 GMT</pubDate>
            <atom:updated>2024-03-25T17:41:18.066Z</atom:updated>
            <content:encoded><![CDATA[<p>Every data engineer has had experiences like these:</p><p><strong>A critical dashboard is on the fritz. </strong>Several metrics are broken … well, <em>sometimes</em> they’re broken. It started happening sometime in the last month, but no one noticed until yesterday. Your CMO and the head of product don’t know what to believe and are flipping out. They need these numbers to be reliable by Monday’s board meeting.</p><p>You start looking into it, but the underlying data architecture is reminiscent of a Jenga tower, and no one who worked on this dashboard wrote anything down. You cancel your weekend plans.</p><p><strong>You’re pretty sure your model is broken in production. </strong>Everything seems fine in the development notebooks, but <em>something</em> happened during handoff.</p><p>You can speculate — maybe a feature isn’t being computed correctly? The input data has drifted? Somebody dropped a minus sign? — but all you can really be sure of is that something about the results is off. Oh, and that digging in and fixing it is going to take way more time than you have.</p><p><strong>Everyone seems to have their own way of calculating (what should be) the same metrics.</strong> You didn’t have this problem two years ago when your system was brand new and you only had one data team. But since then, you’ve added two teams, eight engineers, and a mycelial network of interdependent tables.</p><p>The system needs a refactor, but the job seems thankless, and the potential scope is terrifying.</p><p>All of these problems are flavors of pipeline debt. And every flavor of pipeline debt tastes awful.</p><h3><strong>What is pipeline debt?</strong></h3><p>Pipeline debt is a species of <a href="https://enterprisersproject.com/article/2020/6/technical-debt-explained-plain-english">technical debt</a>: think Michael Feather’s concept of legacy code, just one step to the left.</p><p>Pipeline debt accumulates when the assumptions that justified the movement of data throughout an organization get lost. The data moves, but no one knows exactly why anything is set up like it is. So when the data <em>stops</em> moving correctly or the results start looking weird, it’s a Problem.</p><h3>Why is there pipeline debt?</h3><p>It’s easy for organizations to accumulate pipeline debt, even when their engineering teams are very competent. There are two big factors that cause this.</p><p><strong>One: data systems naturally evolve to become more interconnected over time</strong>. It’s generally a given that “breaking down data silos” is a good thing — and it is — but in a DAG context, it’s easy to skip over thinking through what the inverse of a silo actually is.</p><p>There’s a reason that actual agricultural silos exist, after all, and it’s not because leaving piles of grain around was too easy.</p><p>Without a silo, your pipelines <em>want</em> to be hairballs. And since the point of breaking down the silo was for everyone to get data from everywhere, your PMs, executives, and customers are going to help make that hairball happen. And so are you.</p><p><strong>Two: data pipelines cross team borders</strong>. Any time a new model, dashboard, or metric is handed off between roles or teams, there’s an opportunity for pipeline debt to creep in.</p><p>No two roles or teams have exactly the same perspectives, tools, and skill sets. Those differences introduce little continuity gaps. And those gaps are where the misunderstandings and misinterpretations that feed pipeline debt sprout.</p><p>Inside a team, pipeline debt can be introduced by something as simple as an analyst-to-engineer handoff. On a larger scale, pipeline debt can become especially acute when upstream and downstream systems are owned by different teams, as in the common case where the team that owns logging is different from the teams that own analysis and model building.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/959/1*ydCVVmiiF5ejpWeeQbNxBg.gif" /><figcaption>Your pipelines *want* to be a hairball.</figcaption></figure><p>What makes pipeline debt different from other technical debt?</p><p>Debt in data pipelines is different from the kind of debt that accumulates in other software systems, especially when machine learning is in the picture — which is the case with increasing frequency.</p><p>Here are three issues that make data pipelines different from other systems:</p><ul><li><strong>Most pipeline complexity lives in the data the pipeline carries, not in the structure of the pipeline itself</strong>. That ‘most’ is because the structure of a DAG <em>can</em> introduce some complexity. But often, a pipeline consists of relatively simple code processing huge amounts of highly-variable data that’s full of edge cases. That’s just as true for basic ELT pipelines as it is for deep neural nets.</li><li><strong>Data pipelines are full of bugs with soft edges</strong>. Data insight is all about statistics, and statistics always have a margin of error, which makes statistical models notoriously hard to test. And practically speaking, it’s often not worth debugging <em>every single</em> possible Unicode error, so we soften up even supposedly-deterministic pipelines to ignore a small number of edge cases. And then we don’t document what ‘small’ is.</li><li><strong>Insights get left on the cutting room floor</strong>. A data analyst or scientist who’s exploring new data develops a mental model of the shape of the dataset and the context in which it was created. Those mental models are semantically rich and provide a lot of nuance into the data’s real-world relevance and true meaning. Unfortunately, that semantic richness is almost always lost in translation as the data work moves farther from its originator. Highly-personalized understandings of the data end up automated as brittle pipelines that are divorced from the contextual reality of the data they carry.</li></ul><p>These factors, plus the wealth of opportunities to introduce technical debt into a pipeline, mean that pipeline debt is absolutely rampant. Like pollution in a river, it accumulates slowly but surely until the entire thing stinks.</p><h3>What does pipeline debt cause?</h3><p>Pipeline debt isn’t just an annoyance for technical teams. It introduces both productivity costs and operational risks.</p><p>On the <strong>productivity</strong> front, pipeline debt manifests as a proliferation of unintended consequences that means anything involving data has a slow, unpredictable development cadence. This is a huge logistical drag on agile (and Agile) teams scrambling to respond to increasing pressure for high-quality insights and demand for new AI/ML tools. And just as importantly, it’s an emotional drag on those same teams: uncertain and frustrated, data engineers lose speed and the work becomes not fun.</p><p>If you’re not a data engineer, ‘data work not being fun’ might not seem like a real problem. But effective innovation with data, particularly in the fast-evolving AI and ML spaces, requires a lot of creative thinking. Unhappy data engineers just aren’t going to be as effective as happy ones would be.</p><p>(Data engineers are, of course, by no means unique in producing better work when happy.)</p><p><strong>Operationally</strong>, debt-heavy data systems emit a long-tailed stream of glitches. Severity ranges from ‘annoying’ to ‘catastrophic,’ but all of them erode trust, sometimes to the point of putting the system’s core usefulness in doubt. If you can’t trust your dashboard, report, or prediction, why do you even have it?</p><h3>The solution to pipeline debt</h3><p>The solution to general technical debt isn’t a secret: it’s automated testing. Testing builds self-awareness of the system by systematically surfacing errors. Virtually all modern software teams rely heavily on automated testing to manage complexity.</p><p>Except for the teams that build data pipelines.</p><p>They <em>are</em> testing their code. But as we described before, the complexity in data pipelines is mostly not from the code. So the same testing practices don’t do as much for pipelines as they do for other software systems.</p><h3>Test the pipeline’s data, not just its structure</h3><p>For data pipelines, we need to test the place the complexity lives, which is in the data.</p><p>And we need to test at the point when new complexity might be introduced. That mostly isn’t when new code is deployed; it’s when new data arrives.</p><p>A single piece of new data doesn’t fuel the fire of pipeline debt. It’s the data that arrives en masse — the batch — that starts causing problems.</p><p>Our term for a test deployed on a batch of arrived data is the <em>pipeline test</em>.</p><p>We didn’t wholesale invent this concept. Most data teams naturally evolve some defensive data practices to manage the chronic pain of pipeline debt.</p><p>These are all examples of informal pipeline test practices:</p><ul><li>“I always flush new data drops through `DataFrame.describe` before running it through our ETL.”</li><li>“We’ve started to get much more disciplined about enforcing typed data, especially on feeds that get shared across teams.”</li><li>“Every time I process a new batch, I run a series of row counts on the new tables. It’s the simplest possible test, but you’d be surprised how many bugs I’ve caught.”</li><li>“We maintain a team of data quality analysts and impose strict SLAs on internal data products.”</li></ul><p>These ad hoc practices are good ideas, but they aren’t standardized. This means they aren’t easily transferable between teams or organizations, and communicating about them requires a lot of effort and documentation.</p><h3>Why GX does it best</h3><p>We built Great Expectations to create a standard for pipeline testing — one that’s shared, effective, open, and portable:</p><p><strong>Shared</strong>: Using GX creates a shared frame of reference and a shared toolbox for implementation, so any team using GX can easily understand what another team using GX is doing.</p><p><strong>Effective</strong>: By focusing on the data <em>in</em> the pipeline rather than the structure <em>of</em> the pipeline, GX tests the real source of the complexity that’s at the root of the pipeline’s debt.</p><p><strong>Open</strong>: As an open source tool, GX’s transparency in its operations and results means that it doesn’t add another layer of mystery on top of your already-troublesome pipelines.</p><p><strong>Portable</strong>: GX’s low barrier to entry means that the majority of organizations have the technical requirements to use it right now without adding any new technology. Data engineers can bring GX with them between jobs, allowing them to build long-term expertise that benefits every organization they work with.</p><h3>Collaboration is the key</h3><p>One thing that’s been made very clear to us over the past six years of GX is that data quality isn’t a purely technical problem. If it were, it would have been solved by now.</p><p>Data quality’s real challenge is that it’s a collaboration problem. Data professionals face challenges in communicating with other data teams and nontechnical teams alike, though not the <em>same</em> challenges.</p><p>At GX, we’re focused on solving data quality in a way that fosters better communication — and therefore collaboration — for data teams. Improving the speed and integrity of data collaboration is what will allow teams to truly solve data quality problems.</p><p><a href="https://greatexpectations.io/blog/down-with-pipeline-debt-2018?utm_source=article&amp;utm_medium=medium&amp;utm_content=pipeline-updated">Since we first wrote about pipeline debt five years ago</a>, the GX platform has grown by leaps and bounds in pursuit of better data collaboration. GX OSS has hundreds of contributors and millions of PyPI downloads, and we’re making its capabilities available as a SaaS solution with GX Cloud.</p><p>On the people side, our Slack community has more than 11,000 members, with more joining every day. GX has grown from a side project into a company in its own right, with more than 50 employees.</p><p>In other words: we’ve proven that GX is on the right path, though we’re by no means done. <a href="https://greatexpectations.io/?utm_source=article&amp;utm_medium=medium&amp;utm_content=pipeline-updated">Check it out</a>, <a href="https://greatexpectations.io/slack?utm_source=article&amp;utm_medium=medium&amp;utm_content=pipeline-updated">drop in on our Slack</a> or <a href="https://discourse.greatexpectations.io/?utm_source=article&amp;utm_medium=medium&amp;utm_content=pipeline-updated">Discourse forum</a>, or <a href="https://greatexpectations.io/meetup?utm_source=article&amp;utm_medium=medium&amp;utm_content=pipeline-updated">join one of our community meetups</a> to see how we’re working to revolutionize data quality.</p><p><em>This article was updated March 25, 2024.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=862ddc46782a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>