<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Richard Glew on Medium]]></title>
        <description><![CDATA[Stories by Richard Glew on Medium]]></description>
        <link>https://medium.com/@richardglew?source=rss-22a9d2389b70------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*TQde0eDUUciNnUQN8dUqtw.jpeg</url>
            <title>Stories by Richard Glew on Medium</title>
            <link>https://medium.com/@richardglew?source=rss-22a9d2389b70------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 11:16:41 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@richardglew/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Data is Data (Part 1)]]></title>
            <link>https://medium.com/@richardglew/data-is-data-part-1-6e3d3026eb62?source=rss-22a9d2389b70------2</link>
            <guid isPermaLink="false">https://medium.com/p/6e3d3026eb62</guid>
            <dc:creator><![CDATA[Richard Glew]]></dc:creator>
            <pubDate>Tue, 17 Nov 2026 04:22:06 GMT</pubDate>
            <atom:updated>2026-11-17T04:22:06.070Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Treating it like it isn’t is why your data program doesn’t scale.</em></p><p>Until your management processes accept that, every new thing you do with data will cost more than the last. The use case is independent of the data itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R39hZT4fSmOIxJQiVrFAOQ.png" /></figure><p><em>This is part 1 of a 3-part series. Part 2 covers the operating model; part 3 the architecture inversion that makes it all pay.</em></p><h3>The short version</h3><p>Here’s an idea I’ve been working on for a while now, and implemented in pieces — with varying but real success — at several organisations: <strong>data is data</strong>. It doesn’t matter whether it sits in an operational system, an analytics platform, an AI pipeline, a third-party tool or on someones laptop. If it’s data you need to run your business, its existence is independent of the use case you have for it.</p><p>That sounds obvious. Virtually nobody manages data that way.</p><p>The consequence of pretending otherwise is severe: duplicated pipelines and integrations, inconsistent controls, governance processes that get slower every year, and an often hidden management cost that grows <em>exponentially</em> with your data estate.</p><p>The fix is equally clear: one unified, automated management path for all data — designed for your most sensitive data, run like a product — with governance emerging as a side effect of doing management properly.</p><p><strong>That’s the entire thesis across this series.</strong></p><p>This part is the evidence: what the current state actually looks like, why it costs what it costs, and why twenty years of consolidation projects and everything from Data Warehouses to Data Mesh and everything in between haven’t yet fixed it.</p><h3>A concrete example</h3><p>A financial services client I worked with for over a year is the clearest example I’ve seen — though I’ve watched versions of it at more organisations than I want to think about. I probably have PTSD by now.</p><p>Their data was fragmented across so many systems that nobody could reliably or easily get at it. The painful part wasn’t that the data didn’t exist — people could <em>see</em> it, sitting there in one system or another. They just couldn’t get it together. To do routine operational work, employees either went through bureaucratic access procedures measured in months, or they worked around the processes entirely — extracts on share drives, personal copies, the so-called “Shadow IT” that every large organisation pretends not to have.</p><p>Meanwhile the organisation kept trying to fix it the conventional way:</p><p>Consolidate the data into one place…</p><p>And they kept failing.</p><p>Multiple consolidation projects stalled or were abandoned outright, each blocked by manual, lengthy governance and security reviews. All told they spent tens to hundreds of millions of dollars trying to bring their data together — and because they’d tried several times over many years, they ended up with several <em>unfinished versions of the same plan in different technologies</em>, each one making the estate more complicated and harder to escape than before. In this very real large bank, the company actually lost a full architectural view of all their data and systems almost a decade ago due to sheer complexity and constant reorgs. Meanwhile, the business operations that needed this to be done grew more and more angry and disillusioned.</p><p>Sit with that for a second: the data existed, the money was spent, the intent was right — and the outcome was worse than doing nothing, because every attempt added a new silo and a new version of governance theatre without retiring an old one. People trying to run the business never got what they needed. Long live Excel.</p><p>This is not an unusual story. It is <em>the</em> story of enterprise data over the last twenty years. So before we talk about the fix, let’s be honest about what the current state actually looks like — because that bank is not a badly-run outlier. It is the norm.</p><h3>The current state: unmanaged by default</h3><p>Let’s be real about what a typical enterprise data estate actually looks like.</p><p>Data is created inside operational systems that were never designed to share it. It gets extracted into analytics platforms, marketing stacks, CRMs and AI environments — each with its own pipelines, its own handling rules and its own governance conversation, if it has one at all. Vast amounts of it are simply unmanaged: uncatalogued, unclassified, sitting in places nobody has inventoried, with access granted informally and forgotten. Data flows in from places no one currently working there has any detailed knowledge about. It is transformed in ways that are equally mysterious and undocumented.</p><p>For the data that is managed, it’s governed by heavyweight and usually manual processes. Want access to a dataset? Submit a ticket, wait for a review, wait for an approval, wait for someone to build a bespoke pipeline — then repeat the entire cycle for the next use case, even if it’s the same data. And it’s also probably not a <em>ticket. </em>It’s more likely an entire <em>project </em>with all the issues that incurs<em>.</em></p><p>The first complaint about enterprise data is always “I can’t get it” — and when you finally do jump through the hoops, it takes forever and costs a fortune. That’s before we even get to whether it’s fit for purpose.</p><p>The deeper problem is that we’ve built <em>treatment silos</em>, not just storage silos. The same class of data is managed differently depending on where it sits , what it is and who wants it. Some data is locked inside high-security platforms while equally sensitive data lies around unprotected in spreadsheets and share drives. This is why we’ve ended up with entire product categories — CRMs, marketing data platforms and more— that live apart from the rest of the business as if their data were a different species.</p><p>Worse still, cost doesn’t scale linearly with data volume. It scales with the number of ways you insist on treating it. Every distinct pipeline, bespoke handling pattern and special-case governance rule multiplies the work. The pace of new data always outstrips any large organisation’s ability to govern it with the processes — and the headcount — it has today.</p><h3>Read it through the lean wastes</h3><p>“Lean Thinking” has been around since the 80s and is one of the few management process frameworks that has the track record to prove it works when done well. Lean has a lot of good things in it, but its concept of the “Seven Wastes” is directly relevant to enterprise data management. Let’s take a quick look.</p><ul><li><strong>Overproduction</strong> — in the manufacturing sense this means making more than you need to. For data, it’s the same data extracted, transformed and stored five different times for five different consumers, each pipeline a product nobody ordered twice.</li><li><strong>Waiting</strong> — approval queues and budget allocations measured in months and years instead of minutes and hours; access requests aging in ticket systems; analysts blocked on data that exists, just not where they need it. It also applies to the latency of data getting from where it originated to where it is actually needed. Data that would be valuable closer to real time often has to wait for overnight processing to become available.</li><li><strong>Transport</strong> — data copied and re-copied between systems because nothing shares a common path, generating stale copies and untraceable lineage. Unnecessary staging also falls into this bucket.</li><li><strong>Overprocessing</strong> — bespoke handling rules and re-litigated architecture reviews for every use case, even ones identical to the last. The same metrics calculated in multiple places. The same data and processing in duplicate pipelines because no one knew it already existed somewhere else.</li><li><strong>Inventory</strong> — a growing estate of uncatalogued, unmanaged data. Inventory with no ledger isn’t an asset; it’s a liability accruing interest. Data kept in places where it is no longer needed.</li><li><strong>Motion</strong> — skilled people hunting for data, chasing approvals, hand-plumbing integrations instead of doing work that matters. Also data moving from system to system when if things were designed better, would never needed to be moved at all.</li><li><strong>Defects</strong> — quality problems discovered downstream, reworked upstream, rediscovered downstream again.</li></ul><p>Lean’s insight applies directly: you don’t fix waste by working harder inside the same system. You fix it by redesigning the system so the waste can’t occur. For enterprise data, the cost of waste is astonishingly high. Business executives are extremely wary of vendors making claims they have heard many times before but never seen materialise. They gave good reason to be. We can do better.</p><h3>Where this goes next</h3><p>So here’s the diagnosis in one line: data is data, but our architectures, governance and transformation processes and product categories were all built on the pretence that it isn’t — and we pay exponential management costs for this.</p><p>Part 2 is the fix: what a single unified management path actually looks like, why approving one pattern once beats re-litigating every use case, how to run a platform as a product, and why the hardest part is organisational rather than technical.</p><p>Part 3 is where it gets properly interesting: an inversion of how we think about “sources” that turns the data platform into the logical source of truth — and pays a dividend upstream that most architectures leave on the table.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6e3d3026eb62" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Testing like it’s not 1997 — Part 1]]></title>
            <link>https://medium.com/@richardglew/data-testing-like-its-not-1997-part-1-c5e5b49a3a97?source=rss-22a9d2389b70------2</link>
            <guid isPermaLink="false">https://medium.com/p/c5e5b49a3a97</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-governance]]></category>
            <category><![CDATA[data-testing]]></category>
            <dc:creator><![CDATA[Richard Glew]]></dc:creator>
            <pubDate>Mon, 08 Dec 2025 04:33:32 GMT</pubDate>
            <atom:updated>2025-12-08T04:33:32.196Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Testing like it’s not 1997 — Part 1</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*Wy18J6k599kVVSk7RPUbvA.png" /></figure><p>We’ve had a lot of progress in improving how we test software from twenty years ago, but data has always lagged behind. <a href="https://hbr.org/2017/09/only-3-of-companies-data-meets-basic-quality-standards">According to Harvard Business Review</a>, <strong>only 3% of the average companies data meets basic data quality standards</strong>. Based on my long experience working with hundreds of companies big and small, this feels depressingly accurate. The real world impact to companies of this is grossly underestimated and not appreciated. We have to do better, and we can.</p><p>I’m going to take you through how we can apply the lessons learned from software testing to data and how to approach data quality from the ground up.</p><h4><strong>Foundations of Quality</strong></h4><p>The first thing we need to be thinking about is <em>what is quality? </em>A common misperception is that testing and quality are the same thing — they are not.</p><p><em>Quality </em>is some property of the system or thing we care about, <em>Testing</em> is how we assert that whatever we have, provides an acceptable degree of the quality we are looking for. In other words, quality is the thing we want and testing is how we see if we have it.</p><p>A structured approach to quality involves talking to a lot of people and finding out what qualities mean to them and what they need from, in our case the data, in order to for it to be useful to them.</p><p>A great tool I have used for a long time to capture this is something called a Quality Strategy. A Quality Strategy (QS) can be created for anything where you want to define and assure quality is as expected, but for Data Engineering specifically we are talking about the typical ETL (Extraction, Transform, Load) or much better, ELT workflow most companies employ through data processing pipelines.</p><h4>What is a Quality Strategy?</h4><p>A QS is a document that defines a broad framework for:</p><ul><li>What qualities or properties we want to verify</li><li>Why we care about the particular quality dimension</li><li>How and where we are going to verify it</li><li>When we are going to verify it</li><li>Who is responsible for doing it</li></ul><p>It can and should be, lightweight and easy to understand. This is the foundation for defining what quality means in general for data engineering workloads.</p><p>The QS itself should only contain the skeleton of what will become individual <em>test strategies. </em>This is to prevent the QS from becoming too large and cumbersome. It merely provides the framework that individual test strategies that are pipeline specific are derived from. As a side note, you don&#39;t <em>have</em> to have a higher level QS. You might get away just doing a project or pipeline level one but I have found that as this is a new concept for many, having a templated version makes getting the specific one you need at the lower level easier and more consistent across teams.</p><p>It is important to understand that while many qualities are fairly universal, each individual data pipeline is likely to have differences and specific requirements. One example could be for data needed for real time analytics. This kind of data often has a <em>latency</em> requirement where data coming from a source is needed in near real time. This would be very different from the latency needed from a system that only gets updates once a week. Similarly, <em>correctness</em> which is a universal need in data, may have different requirements. The data for your bank account balance or salaries in a payroll system generally needs to be 100% correct. Other data, while we would love to have 100% correctness, may not be as critical. An example of this could be data from surveys — we know not everyone is going to be completely accurate in their responses but if it is only 80% reliable, it is still better than no data and is probably good enough.</p><p><strong>Testing Strategy (derived from QS)</strong></p><p>The <em>Test Strategy</em> (TS) for a given workload/pipeline will have a similar content structure to the QS, but with a lot more context and detail. It will have the specific requirements for a given workload and all the details of what tools, what teams and key contacts and generally how testing for that workload will be done in detail. When structuring our tests overall, we can start to borrow a lot of concepts from software engineering testing.</p><h4>The Test Pyramid</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0E5A_RjiOqOWkLzC1ePjRg.png" /><figcaption>Classical Software Engineering Test Pyramid</figcaption></figure><p>The classical test pyramid used in software engineering is meant to depict the concept that we have different types of tests that differ in volume and time taken. Typically unit tests are very small and fast to run and there are a lot of them. These tests tend to get run a lot, and in order to not impact productivity for engineers, they need to complete quickly, in seconds ideally. Other tests like Integration tests (between components for example) might take a bit longer, or have environmental dependencies that are not always quick to achieve. End to End tests, check across the whole system and typically involve simulating full workflows that may be quite slow to run. The relevance here to our test strategies is that this is something that needs to be taken into account when designing our test strategy so that right tests are run at the same time. Generally speaking in software engineering, most of these tests occur before production deploy but some tests may run in production too.</p><p>This can be applied to data engineering workflows where some of the tests we run, for example, on the correct implementation of a metric are essentially a unit test and will run very quickly, but other tests that we may need, especially when considering large data volumes, may take a long time to run, and in data unlike software we generally don’t have direct control over data validation, so production testing aka observability is a critical necessity. We will get into the details further on about specific test types we commonly need in data engineering later on.</p><p>Another benefit of this approach is that you can expose the higher order tests to end consumers without exposing them to all the lower level, intermediate level testing that is useful for data engineers building the pipelines. It goes beyond the scope of this article, but a <em>domain driven design </em>approach which this idea supports, hides complexity and has many benefits one of which is reducing the amount of noise and cognitive overload by only exposing what is needed. This just makes it easier for people focus on what is important to them.</p><p><strong>Implement automated tests and monitor</strong></p><p>Once we have defined our overarching quality strategy and created a test strategy for a given workload, we get to the fun stuff — actually implementing the tests themselves.</p><p>So far, we have talked at a high level about defining quality and testing, but we obviously need to consider the actual implementation approach as well.</p><p>From here on in, we are assuming an agile, DevOps style of implementation using CI/CD pipelines that focuses on as much automation as possible with a high degree of <strong>actively managed</strong> test coverage.</p><h4><strong>Similarities between software and data engineering</strong></h4><p>There are a lot of similarities between testing software and testing in data engineering pipelines.</p><ul><li>If you can think of a test you need, it can almost always be automated.</li><li>The same techniques and a lot of the common testing tools apply. QS and TS concepts are the same and a lot of open source tooling for software can be applied easily to data problems.</li><li>The same kind of benefits in creating fast feedback loops. Increasing the <em>new work </em>to <em>fixing bugs</em> ratio can be drastically improved. Just these two benefits (and there are more) create happier end users <em>and </em>happier engineers, improve quality drastically, reduce risk and create a significant net increase in productivity and efficiency for engineering teams. In a lot of cases I have seen teams go from spending 80% of their capacity fixing bugs and operational issues to less than 5%. It takes discipline and effort to get there, but it makes a huge difference.</li></ul><h4><strong>Differences between software and data engineering relevant for testing</strong></h4><ul><li>Direct input validation of data does not exist like in a typical software application. Surprises happen at mostly at run time (in production) not at build/design time</li><li>Usage context of data is NOT implied unlike in an application that has its own controls on who can see and do what with its data. This leads to complexities in the willingness of people to share data when they don’t know or trust what will be done with it.</li><li>Comprehensive metadata is critically important in order to create controls that an application may enforce. Most software engineering neglects this as it is implied by the application itself.</li><li>Data can be big — really big — testing can be slow and expensive if not applied mindfully. This actually can be a problem in software engineering in some cases, but it is far less common than in data engineeering.</li><li>Data coming in from the outside world in production <em>will</em> <em>inevitably</em> have issues. In software engineering where we have typically have control of data validation we break the build on a failing test. When a pipeline runs, what to do about data issues is more complex. If correctness of data is paramount, we may indeed abort the pipeline and create an incident but in other cases we may just want to quarantine the questionable data or allow the pipeline to run, but notify someone that it needs remediation or at least investigation and update our catalog or data quality tool accordingly.</li><li>The general categories of things we will test for are not identical. They are generally simpler in data engineering but there are some things as well that are fairly specific to data. Metadata validation is one example.</li></ul><h4><strong>Challenges with data testing and traps to avoid</strong></h4><p>In 2025, a software engineering style testing of data is relatively rare in most organisations. As a result, it is easy to make mistakes in attempting it for the first time that make it harder than it needs to be, and can even lead to total failure.</p><p><strong>Social challenges within the Data team itself</strong></p><p>Many enterprise data teams essentially do <em>no testing</em> at all today. The time cost of including it into a teams workflow, plus a learning curve can feel like it is adding a lot of time and cost to engineering work that impacts committed plans and timeframes. While the learning curve may end up adding extra time, it is false economy to not do it.</p><p>The cost of repairing and reloading data down the line <strong>far</strong> outweighs the initial cost by orders of magnitude. This is self evident to any team who has implemented a good and disciplined automated testing approach. Although this is now generally accepted wisdom in software (though still not universally applied). I was around when CI/CD style test automation was new in software development and the same arguments were made back then as now. The big advantage we have this time is that data testing compared to software testing is a relatively simple affair and the tooling and test frameworks are quite good compared to the mid 2000s when CI tooling was brand new and very basic compared to what is available now as well and automated testing was a hotly debated topic.</p><p><strong>Organisational and Mindset Inertia</strong></p><p>Beyond the data engineering team, most enterprises have a lot of policies and procedures for how data should be managed and controlled. Data governance, security, development and release processes have evolved over time to handle the existing data landscape. Despite the fact that in practice, such policies tend to be <em>at best</em> partially effective, getting these changed to enable a true DevOps model can be extremely difficult in practice.</p><p>In an application world, it can be hard enough where there is tight coupling to legacy applications that make automated integration and acceptance testing difficult. In the world of data, security is a constant concern and we are increasingly all subject to some form of regulation in how we handle data like PII. This is an order of magnitude more scrutinised in regulated industries where the list of stakeholders who take an interest in how data is handled extends broadly.</p><p>The way most enterprises go about managing their data hasn’t evolved much since the 1990s. Some of the tools may be more modern, and may support new ways of working but I see a lot of organisations applying their current way of working and thinking to these new products.</p><p>Getting an organisation&#39;s mindset and <em>then </em>changing<em> </em>processes and policies across a wide group of departments is objectively hard. It’s almost always the single hardest task of moving to an automated approach that is executed as DevOps was intended to be done. It’s part of why recognising and including the social aspect of not just the data team but the wider business is a core element in solving the bigger challenge of data quality management. In Part 2 I will talk about some ways I have been able to do this successfully in various companies.</p><p><strong>Speed Matters</strong></p><p>I have seen teams without a good test strategy lump all testing together and complain that it slows down their development and generates excessive costs due to expensive long running tests being run more than is needed. Certain commercial <em>data quality</em> tools exacerbate this problem by purporting to be the only thing you need (more on this in Part 2). Test the right thing in the right step of a pipeline and optimise for engineer workflow efficiency and it’s a lot easier to manage for ROI on test runs. There is also a high degree of correlation between speed and cost, assuming your code is at least reasonably efficient, faster is cheaper.</p><p><strong>Language Limitations for Testing</strong></p><p>SQL and Python are the main languages used by data and analytics engineers, ML engineers and Data Scientists. SQL has a fairly limited set open source tooling for testing (though some exist). Python on the other hand has great support for testing and has good integration with many data environments. It’s important to note that any test framework in <em>any language </em>can be used provided that it has the ability to access data in your platform.</p><p><strong>“The Business” is involved</strong></p><p>Non-technical business users and analysts are drawn deeper into the solution than with typical software engineering testing. Semantic understanding of the data given to them is <em>required</em> by people interacting with data directly. A good catalog with accurate and up to date quality metrics are mandatory for usability and understanding.</p><p><strong><em>Silver bullet</em> thinking</strong></p><p>There isn’t really a way to put this delicately. The reality of data engineering over the years is that most <em>engineers</em> aren’t really <em>engineers</em> in the sense that they are creating modular pipelines, writing code and tests, applying proper version control to both data and code etc. Generally just following a workflow that any competent software engineering team would follow. Before I start getting hate mail for saying this, there is a long running argument that a lot of software engineers aren’t “real”” engineers either. From the perspective of other engineering disciplines (mechanical, electrical etc) there is often the perception that software folks appropriating the “engineer” title is akin to homeopathy practitioners calling themselves “scientists”. So take this with a grain of salt.</p><p>Here goes. Before “Data Engineer” was a title, most people who did this line of work exclusively were generally known as “ETL developers” and almost exclusively used no/low code ETL tools such as Microsoft’s SSIS (basically the same thing as Azure Data Factory without a subscription model) or Informatica’s ETL offerings. Until the “Big Data” revolution happened that forced a few software engineers into data, almost everything was “click ops” and traditional vendor offerings in the data space center around tools that “do everything”. Successful solutions don’t work like this in practice and no one vendor, no matter how good their product is, does everything you need. In the <em>Data Quality</em> domain, we need a more nuanced approach and though there are commercial tools that are excellent, without a scalable and affordable licensing or cost model, the ability to make effective use of them broadly becomes an issue. A lot of common commercial tools available and in widespread use are ineffective and promote bad practice — as well as being very expensive. The good ones generally focus on solving one type of problem well.</p><p><strong>Capability and Capacity</strong></p><p>A software engineering centric approach to testing necessarily involves a new skillset that most enterprises are very short of in their data teams. Before embarking on a wholesale shift to a new way of working, you need to have people in place who know what good looks like. Fortunately people do exist who know what a proper devops implementation looks like and have experience implementing them. If no one knows what good looks like before you begin, expect the learning curve to be longer and more mistakes to be made.</p><p>I’m going to stop here as we have covered most of the theory parts of a better way to approach data testing, Part 2 will dive into practically how you can go about doing this with a few worked scenario examples.</p><h4>Wrapping Up</h4><p>As a quick recap:</p><ul><li>We need to start by thinking about quality and defining it so we can design the right approach to testing</li><li>Testing data in data engineering pipelines has many similarities to software engineering and some important differences</li><li>If you’re looking to change up the way of testing data it’s a bigger problem than just a technical one. Arguably out of the usual <em>people, process and technology</em> buckets we can usually group things into, the <em>people</em> one is the hardest to solve and the key to success.</li><li>As an industry we have proven approaches to leverage and the knowledge exists as to what good looks like. It’s not super common yet, but it’s there.</li><li>In Part 2 I will talk more about <em>how</em> to apply this sort of approach in a typical larger environment where most of this is not in place and what has worked for me and others to move in a better direction.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c5e5b49a3a97" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Modern Data Operations: Part 4]]></title>
            <link>https://medium.com/@richardglew/modern-data-operations-part-4-08176c956eeb?source=rss-22a9d2389b70------2</link>
            <guid isPermaLink="false">https://medium.com/p/08176c956eeb</guid>
            <category><![CDATA[data-management]]></category>
            <dc:creator><![CDATA[Richard Glew]]></dc:creator>
            <pubDate>Wed, 18 Dec 2024 06:45:06 GMT</pubDate>
            <atom:updated>2024-12-19T00:12:47.044Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*Egct8gMOrjNDXkCx0A_A3Q.jpeg" /><figcaption>Cargo Cult Environments?</figcaption></figure><p>Here is the final piece of the story that my colleague <a href="https://medium.com/u/20a7548465e5">henry lindsay smith</a> have been working on. At the end there is a fairly simple message we want to share that we think isn’t actually controversial at all.</p><p>Before we dive in to the endgame, here is a very quick recap of the story so far:</p><ul><li>In <a href="https://medium.com/@henrylindsaysmith/modern-data-operations-part-1-33f1ae129132">part 1</a> we talked about the origins of environments and what their purpose was as well as how they have evolved</li><li><a href="https://medium.com/@richardglew/modern-data-operations-part-2-c75e24a66096">Part 2</a> was about how all the evolution of environments has happened with a software or application mindset and that the data world is still operating with an approach not much different that the 1990s but has recently started to show some innovation</li><li><a href="https://medium.com/@henrylindsaysmith/modern-data-operations-part-3-7c1515d72ca0">Part 3 </a>was getting to the core reasons, real reasons, why we need environments and touched on some aspects of the data and analytics world that differ significantly from the application world.</li></ul><p>In this final section, we will explore more fully what we see as a better pattern that we’ve seen applied at a number of organizations. It enables a far more streamlined and simple approach and leverages the powerful capabilities of modern cloud and data platforms.</p><p>Before we dive in, let’s take a look at our design goals:</p><h3>What should we be able to do, in order to service the growing need for secure and accessible data?</h3><p>Based on our daily experience working with hundreds of companies in a very diverse range of industries, there are some common needs that we constantly see and that many organizations struggle to do easily and efficiently.<br><em>1. Non analyst/technical business people</em> need easy access to trusted data. I refer to these as operational use cases in the sense that the pipelines that run and keep this data up to date runs every day and is relied upon to run the operations of the business. Operational data sets are also a common foundation for analysts to do exploratory work.<br><em>2. Analysts</em> need to be able to quickly access real data and do ad-hoc processing in a safe way for exploratory use cases. Often, this requires a sandbox environment that uses real data but we need some isolation from our “production” environment so that we don’t break or interfere with our operational needs. By exploratory I mean, “Something has happened and we’d like to understand why.” It is often a one off request that we will never look to make repeatable. Sometimes this can be done with existing data sets but often it requires bringing in new temporary data sets, creating new transformations etc. <br><em>3. The business </em>needs to know what data it has, who has/had access to it and that it is secure. If regulations are in play, then all those concerns need to be covered.<br><em>4. Data engineers </em>working on operational pipelines need to be able to build and test their changes and updates in a way that is least likely to cause issues or breaking changes to operational systems. Due to the often large scale and importance for reliability these pipelines have, we need them to be maintainable and efficient.</p><p>So, that begs the question — what is getting in the way — and what do environments have to do with any of this?</p><h3>Data Security and Access Challenges</h3><p>The advent of the internet, cloud environments, and the proliferation of data storage locations has dramatically altered the security landscape for organizations. The traditional perimeter defense approach imported from the on-premise data center world is not enough. Today’s businesses face a complex challenge; balancing the need for robust data protection with the imperative to leverage data holistically for business insights and operations. Increasingly we are seeing machine learning use cases and now the new trend of agentic systems adding more pressure for universal data accessibility.</p><p>Data breaches are something that keeps CIOs up at night. The potential for data compromise extends across every part of the tech landscape at an organization and much of the data is not well understood. This heightened risk landscape has led many organizations to adopt security measures that attempt to prioritize absolute security over accessibility. What is often not understood or measured, are the very real and large costs this approach imposes. It makes change extremely expensive and slow, as well destroying the organizations ability to make data based decisions quickly. The great irony is that this approach often leads to <em>even more</em> <em>unmanaged</em> data being spread around, increasing the burden of protection for already stretched cyber teams and actually <em>increases</em> the risk of data misuse or a breach.</p><p>Some of this is down simply to the OKRs or targets given to cyber teams. eg: no data breaches or loss. They are not generally incentivized to create usable systems or any have any performance metrics around business enablement. We see this disconnect drive a lot of architectural anti-patterns.</p><p>As technology practitioners, the word “model” gets thrown around a lot and I am of the strong opinion that at a macro level, the model that matters the most is making sure our systems are aligned to the operational areas of the business. Not by technology; and definitely not by data sensitivity classification.</p><h3>The Siloed Approach to Data</h3><p>However it comes about, through security concerns, general IT approaches or whether it just occurs naturally through unplanned organic architecture; it’s blindingly obvious that when we have a patchwork of approaches and conflicting performance metrics across an organization, that we are going to find it hard to bring it all together and make it join up.</p><h4>The Need for a Unified Approach</h4><p>What is required is deceptively simple. <strong>We need a single and consistent method for managing all data</strong>, regardless of its sensitivity level or what technology it lives in. The advantages are many, but here are a few of the biggest:</p><p><em>1. Consistent and simplified security:</em> Implementing a standardized data security approach across all data stores and environments, helps ensure uniform protection and reduce the risk of oversights or vulnerabilities. It is easier to understand and maintain. The cost:benefit of any given security measure should be evaluated on its impact to wider business productivity.</p><p><em>2. Simplified Compliance:</em> A unified data management system makes it easier to monitor, audit and protect data and thus prove compliance with data protection regulations. If you do data management well, you get all your data governance needs for free.</p><p><em>3. More value from your data:</em> With appropriate access controls and masking methods in place, organizations can more easily leverage their data for analytics and decision-making without compromising security and privacy. A good approach to environments for data analytics can play a large part in helping with this.</p><p><em>4. Improved Efficiency:</em> A single approach for managing data reduces operational complexity and greatly reduces data proliferation. Moving data around is one of the most expensive aspects of data management.</p><h3>Safe change management and operational stability</h3><p>The status quo of environment management (usually described as Dev, Test, Prod) presents unique challenges for data teams, primarily due to some fundamental differences between data analytics and engineering and traditional software development. Advancements in software development have championed frequent, iterative changes and continuous deployment, which <strong>are</strong> broadly applicable and valuable to data engineering and analytics but these improvements are insufficient and inadvertently create problems for people working in data and analytics.</p><h4>Contrasting Data and Application Development</h4><p>Application development typically involves complex code, user interfaces, and extensive verification processes before release. The DevOps approach which is mainstream in most organizations looks a bit like:</p><ul><li>Frequent changes, released in short cycles</li><li>Small, easily identifiable change sets that make debugging faster and easier</li><li>Automated testing that is consistent and reliable, and avoids lengthy manual testing</li><li>Reduced cost of marginal change because we focus on making hard things easy and repeatable</li></ul><p>All of this applies to data and analytics, but it’s not the whole story.</p><ul><li><em>devOPS vs DEVops:</em> In comparison to software engineering, the <em>development part</em> of data engineering is relatively simple. We move data from point A to point B and do some transformations along the way. This is usually done in SQL and in some cases another language like Python. Asserting that our transformations are done correctly and that we haven’t lost any data along the way is kind of it. On the <em>ops</em> side the story changes. Unlike an application, data pipelines generally do not control the data they get from their sources. The quality of the actual data coming at run time in production is something that is a real and ongoing problem. No amount of pre-production testing is going to catch a bad value from an upstream system. We have to test in “production” to do that. This is slightly different from general observability where we are concerned with the uptime and performance of our platforms and pipelines.</li><li><em>Scale:</em> With modern cloud platforms, for the average enterprise application it’s unlikely that scaling the application presents a major challenge. In the data world our data sets can be huge even in relatively small organizations and if we need to make copies (sanitized obviously) then this can quickly become an expensive and time consuming exercise.</li><li><em>Access to real data outside the main prod environment:</em> Mentioned earlier, analysts of all kinds need to be able to do exploratory work that is necessarily done with real data. That hopefully sounds kind of obvious, but many of our colleagues have to jump through a lot of hoops to get access to a data set that they need. In some cases I have seen it take so long that the opportunity to gain an insight and act on it has long passed by the time the data is made available and access is granted.</li></ul><h3>The Production Data Paradox</h3><p>A significant challenge arises from the common policy of forbidding “production data” in “non-production” environments. Historically it was a valid argument in the sense that these “non-prod” environments as we explored in parts 1–3 of this series did not have the security or governance needed to safely keep sensitive data.</p><p>This policy creates a dilemma for data professionals that need production data in an environment separate from their operational systems, but are prohibited from accessing it in their development environments.</p><p>Fortunately, we are now at a place with modern cloud and data technology that the constraints of the past no longer need apply. We can have our cake and eat it too. This is not a case of data anarchy, but one where a measured and planned approach to data architecture can <em>improve</em> data security and governance while at the same time drastically increase accessibility and usefulness.</p><p>To achieve this we need two main areas of capability. The first is around technology and architecture and the second is the policy, process and governance side of things.</p><p>From the technology/architecture side we need to be able to:</p><ul><li>Put data to one place and leave it there. We need to be able to bring compute to our data rather than the other way around so that we know where that data is, what it is and who can access it</li><li>The ability to “zero copy clone” production data with appropriate masking and privacy capabilities into a different environment. This means we can (almost) instantly create a safe, isolated copy with the appropriate governance controls, all without actually making a physically different copy of the data. Being able to do this represents enormous cost savings in terms of compute and storage but also in the labor and time it takes to move data, especially if it is to be operationalized.</li><li>A unified data management tool. I am deliberately not naming specific products here because these are fundamental capabilities and the tooling landscape continues to evolve rapidly, but such products exist in market today. If we have all of our data “under management” then feeling confident about its safety is much easier for everyone. Attribute based access controls and policies can really help make this a simpler effort as well.</li></ul><p>On the policy/process/governance side we need:</p><ul><li>To redefine what <em>production</em> and <em>non-production</em> mean and what those terms imply. For our analytics teams to be effective, there should be no difference in governance and security between <em>any</em> environment, production or otherwise.</li><li>Updates to policies and education of a wider set of stakeholders (enterprise and security architects for example)</li><li>Automation of everything helps ensure uniformity and makes updates to standards around security and governance easy to manage and enforce</li><li>Policies and processes that can be audited automatically to build confidence that our governance reality matches what we believe the case to be. Many organizations have very strict policies that are worked around on a daily basis because they make getting work done impossible. We want real security not security theater.</li></ul><p><strong>One side note around testing</strong> which relates to environments. Comprehensive data testing is its own topic altogether but it has been pointed out to me that in many cases we <em>don’t</em> want real data for testing in non prod environments. That is entirely correct. Real data may not test every edge case that could exist and for the purposes of testing a simple transformation it is not cost effective or necessary to do it on a massive production dataset. There are other testing scenarios where real data is not just unnecessary but would be wasteful. We are not saying that <em>all</em> testing should be done with production data, but we are saying that <em>all analytical life cycles should involve testing on production data at some point (even if it’s just on the source values coming in through operational pipelines)</em></p><p>The point we are hoping to make is simple to understand, and less simple to achieve:</p><p><strong>Every environment used by data professionals for work with data should have identical governance and security controls. We might not need or want real data in our other environments, but equally it should not be an issue if we do.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m0IGxNMUIxb8HA9_nJCoTA.png" /><figcaption>As far as the platform team is concerned, every “client” environment is a prod one with all controls applied equally</figcaption></figure><p>My colleague Henry repurposed this meme to get his point across.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/0*LKqAg4QAD0iMPTjp" /></figure><h3>Conclusion</h3><p>The concept of environments has evolved significantly from its origins in software development to its current role in data and analytics. This evolution reflects changes in technology, security needs, and organizational practices. However, the traditional view of environments often holds back what is possible and slows down innovation. We have the technology now that can satisfy the requirements for isolation, security and compliance and at the same time, enable access to real data sets for all types of development. Not only is it faster, better and cheaper; it’s actually safer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/771/1*khoZWIQ9tuOreoqASuFwLw.jpeg" /></figure><p>This wraps up this series and if you have read this far thanks so much for taking the time. As always feel free to throw rocks at us if you think we have missed something or let us know if you think it’s useful.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=08176c956eeb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Modern data operations: part 2]]></title>
            <link>https://medium.com/@richardglew/modern-data-operations-part-2-c75e24a66096?source=rss-22a9d2389b70------2</link>
            <guid isPermaLink="false">https://medium.com/p/c75e24a66096</guid>
            <category><![CDATA[data-architecture]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Richard Glew]]></dc:creator>
            <pubDate>Mon, 09 Dec 2024 21:00:05 GMT</pubDate>
            <atom:updated>2024-12-10T00:22:29.736Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fqZgqk6oPX4xGECztcYCmg.jpeg" /><figcaption>Partially evolved</figcaption></figure><p>This is the second part of a series of articles exploring some of the ideas of modern data operations with my colleague <a href="https://medium.com/u/20a7548465e5">henry lindsay smith</a>. It started off with us having a lot of conversations with various customers around data architecture, environments and lots of related topics and we thought we should probably just write some of it down. It turns out that is harder than you think when you start to unpack it all and for all its awesomeness, GenAI is not very good for long form writing just yet.</p><p>In <a href="https://medium.com/@henrylindsaysmith/modern-data-operations-part-1-33f1ae129132">Part 1</a> we introduced the environments conundrum, talked about the reasons for having environments and ran through a quick history lesson of how infrastructure and software environments have evolved. In part 2 we’ll explore how the environments data teams have worked in have often been structured and how the data team’s requirements differ from that of software development which is so often the default model applied to data teams. It’s a constant source of pain for many of our clients and us as well as data practitioners.</p><p>One of the key distinctions behind the differing requirements lies in the nature of the work itself. While software developers can often work effectively with mock data or limited datasets during much of their development cycle. Data engineers, scientists and analysts require access to “real” data to derive meaningful insights, improve analytical models, validate data or observe production.</p><p>This fundamental difference challenges the many of the conventional wisdom of environment segregation and raises questions about how we can balance the need for data access with security and compliance requirements.</p><p>With this in mind we return to the environment history lesson…</p><p>If the software engineering world has reached the point of microservice environments proliferation, how is data doing?…</p><h3>The Data World: Lagging Behind</h3><p>While application development has experienced rapid evolution, the world of data and analytics has not kept pace. Despite the innovations brought by cloud computing, many data teams have remained entrenched in outdated practices and methodologies, reminiscent of how things were done in the 1990s in data centres. The orthodoxy of <strong>Dev, Test, Prod </strong>is repeated by almost all data teams and IT operations teams. From our observations data teams have not made it to the microservices proliferation stage. Phrases such as <strong>‘no prod data in test and no testing in prod</strong>’ continue to be repeated with apparently no contemplation of the oxymoron this phrase represents. Several factors have contributed to the lag in data practices:</p><p><strong>1. Legacy data landscapes</strong>: Many organisations continue to rely on traditional data warehouses and traditional daily batch extract, transform and load (ETL) processes that are not easily adapted to modern, agile methodologies. The large amount of effort and inertia to migrate from these leaves many teams working with traditional processes and environments. In particular legacy low code data environments are very labour intensive to modernise due to the business logic being tightly coupled in the tool</p><p><strong>2. Tool Limitations:</strong> Many popular low code data tools, while powerful and easy to use, are not designed with modern change management and deployment practices in mind (for instance something as fundamental as a diff is not possible with a low code pipeline). They also often do not support useful abstractions leading to many pieces of functionality being duplicated. The overall result of these two drawback leads to higher maintenance costs as well as providing fertile ground for bugs and issues.</p><p><strong>3. Skill Gap:</strong> Data teams often lack exposure to or skills in modern software engineering practices, which can be traced back to having spent their career working within the paradigms of points 1 and 2. This has hindered the adoption of agile and DevOps principles. Legacy data tools have sought to be low code in part because of the lack of talent in the data sector and in a way the problem they were created to solve has been worsened by the solution. That is now beginning to change with an industry side acceptance that automation needs to be pervasive in the analytical development lifecycle.</p><p><strong>4. Complexity of Data:</strong> The nature of data work means it is inherently more complicated than software engineering. The models and transformations must work on a set of data, but this data is not static. All the edge cases for the data being delivered and what is contained within this data must also be handled prior to experiencing them (e.g. stream fails, database goes down, file does not arrive, file is blank, missing values, duplicates, schema changes, changes in statistical distribution, values that are out of bounds from a business lens). In this sense the traditional software engineer practices of mocking and testing are not fit for purpose, or are at least only part of the solution. Building software solutions that work with data with its emphasis on data quality, governance, and complex transformations, presents unique challenges that are not easily addressed by traditional software development practices.</p><p>The early days of data engineering we can characterise the “small” data era, a time of servers (database and ETL servers). ETL scripts were perhaps written by database administrators and manually deployed and run. Or ETL was performed by UI driven low-code tools, with the same manual approach for deployment and testing. Practices followed a ‘deploy and then don’t touch it’ philosophy. Things were about to change with the Big Data revolution.</p><h3>The Big Data Revolution, Cloud engineering and Its Impact</h3><p>The advent of Hadoop and the “Big Data” movement (multiple parallel processing databases, distributed processing frameworks and the general ability to scale data workloads horizontally), combined with the explosion of interest in AI in recent years, has drawn a large number of software engineers to the data world. These newcomers, accustomed to modern development practices, often found themselves in a state of culture shock when confronted with legacy data practices.</p><p>The influx of software engineering talent has started to catalyse change in the data world:</p><p><strong>1. Improving Data Engineering Practices:</strong> The application of software engineering principles to data pipelines and transformations. The industry now accepts that transformations logic should be version controlled and essentially managed as code.</p><p><strong>2. Decoupling of business logic from data processing technologies:</strong> The lessons of tight coupling of business logic into low code tools has been largely been learned, and indeed even the usage of stored procedure in MPP databases is challenged (as a form of tight coupling). Business logic should sit externally, visible, and accessible.</p><p><strong>2. DataOps/DataDevOps:</strong> The emerging practice that applies DevOps principles to data and analytics, aiming to improve the quality and reduce the cycle time of data analytics.</p><p><strong>3. Data Mesh:</strong> A emerging decentralized approach to data management that applies domain-driven design and self-serve platform design to data architecture. This is a gross oversimplification of quite a deep topic and I will direct you to the many excellent resources out there, including those by Zhamak Dehgahni who coined the term.</p><h3><strong>Recap</strong></h3><p>Environment design thinking is <em>fundamentally</em> shaped by past constraints that don’t have to apply anymore. Whilst a lot of progress has been made in the application development world, the data world, in part due to the sheer scale of the problem, is mostly rooted in the past and has not generally evolved except in a handful of web-scale tech companies. What should the average enterprise data focused person do about this? To answer this, we are going to examine the fundamentals of what environments <em>actually</em> <em>are</em> and how a different approach and newer technologies can satisfy these needs.</p><p>Join us in part 3 as we expand on these ideas</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c75e24a66096" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>