<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Marian Nodine on Medium]]></title>
        <description><![CDATA[Stories by Marian Nodine on Medium]]></description>
        <link>https://medium.com/@mariannodine?source=rss-929a3612398a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ot485yLoj25T578SZ6OQAg.jpeg</url>
            <title>Stories by Marian Nodine on Medium</title>
            <link>https://medium.com/@mariannodine?source=rss-929a3612398a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 01:18:56 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mariannodine/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Sssneaky Data Problems that Creep in Over Time]]></title>
            <link>https://medium.com/data-science/sssneaky-data-problems-that-creep-in-over-time-9394da10b9c9?source=rss-929a3612398a------2</link>
            <guid isPermaLink="false">https://medium.com/p/9394da10b9c9</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-dives]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[notes-from-industry]]></category>
            <category><![CDATA[productivity]]></category>
            <dc:creator><![CDATA[Marian Nodine]]></dc:creator>
            <pubDate>Thu, 01 Dec 2022 07:54:39 GMT</pubDate>
            <atom:updated>2022-12-01T07:54:39.643Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Three notes containing images. The first is an image of a data table, and is labeled ‘schema’. The second has a bar chart, and is labeled ‘shape’. The third is a line graph and is labeled ‘scale’." src="https://cdn-images-1.medium.com/max/903/1*dNRIseSyXZM2M7_4NUHipQ.png" /><figcaption>Image by author</figcaption></figure><h4>Mitigating the unavoidable impacts of data drift and bit rot in a long-lived data product</h4><p>Have you ever had a data product operating smoothly for a long period of time … then suddenly break? Have you spent time wondering what changed, only to discover that someone renamed a column in a table somewhere? Or perhaps, you discover that suddenly your data is one more day stale than it used to be? <strong>When you move from a development mindset to a data product mindset, your focus turns from the specification and development of new functionality to the problem of maintaining continued, reliable and robust execution.</strong> Yet, with a long-lived data product also come the unavoidable and often unanticipated problems related to data drift and bit rot.</p><p>Data drift (also referred to as concept drift in the Data Science Community) is the <strong>change in the statistical properties of data over time</strong>. For Data Scientists, data drift can affect the accuracy and timeliness of your models.</p><p>Bit rot (also known as software rot), is a <strong>slow degradation in software quality and responsiveness that leads to a point where the software is faulty, no longer supports the functions you need, or becomes unusable</strong> in some other way. Data pipelines are implemented using software, either directly or indirectly, and thus are susceptible to bit rot.</p><p>Data drift and bit rot are inevitable in a data product that is based on data that is changing over time. Many of these changes are desired in that they are a part of product development, improvement, and evolution in the systems that produce the data your product uses. That is, <strong>while change causes disruption, often the underlying change is a result of doing actions that are expected to be beneficial</strong> either to your product or to the products that your data product interacts with, or to some aspect of your company’s business.</p><p><strong>The cost of not noticing when a change is impacting your data product is paid in a loss of trust.</strong> When you are not actively looking at the data product on a regular basis, it is very easy not to notice when such changes occur. Surprisingly, <strong>often the cost of fixing a data issue also can be a lack of trust.</strong> The user only sees that the report is changing. Thus, any mistake may impact the trust relationship multiple times, and <strong>trust is hard to regain</strong>.</p><p>So, change is inevitable. One of the ways you as a consumer of changing data and systems can stop that change from impacting a user’s trust in your product is to develop a data culture that supports accurate communication of anticipated changes in data sets being made available. Managers and developers of data products in turn communicate which data sets they are using to the developers of those data sets. <strong>When a data culture supports conversation between data set producers and data set consumers about a data change, the managers and developers of affected data products can prepare for a smooth transition when the change occurs. </strong>In the best of cases, the transition has very little ensuing impact on the data product — the user does not really see it. Hence, <strong>the change also has very little impact on the trust relationship.</strong></p><h3><strong>Takeaways:</strong></h3><p><strong>For the data product manager: </strong>Developing a product that is trustworthy over the course of time takes considerably more care and thought than is apparent from the outside. Your data product developers have little or no control over how the raw data they start with will change over time. <strong>It is your responsibility to ensure that your developers can pay attention to longevity and maintainability concerns as they pertain to the changing nature of the input data.</strong> As a part of that, it is also your responsibility to manage the interactions with the producers of that raw data. Putting in place the the proactive and reactive functions discussed here will be very beneficial to the long-term trustworthiness of your data products.</p><p><strong>For the data product developer:</strong> When you are making a robust data product, <strong>you need to pay attention to the things that are out of your control</strong>, including changes in the input (raw) data your product uses. Many of these changes involve the schema, shape and scale of the input data. Developing your product includes putting in proactive and/or reactive processes to catch relevant changes and adapt your product accordingly.</p><h3>Three Important Types of Sssneaky Problems: Schema, Shape, Scale</h3><p>Every data product is based on some set of raw data. Your initial design of the data pipelines and processes places certain expectations on that raw data. The expectations may be baked into the code or the queries for your data product, or in how you run your pipeline, or in your assumptions about the meanings of specific data sets. You may not actively be thinking about what those expectations are; nevertheless your pipeline will function well only as long as those expectations are met.</p><p>In this article, we discuss three different sets of expectations/requirements about a data product that can impact its responsiveness and trustworthiness — expectations about schema, expectations about shape and expectations about scale. These expectations/requirements emerge from the way the product uses the data.</p><p><strong>Schema: </strong>Generally, we consider a schema to be a description of what the data looks like. A schema includes the tables or data sets that are accessible to the user. For each table or data set, schema information defines the columns or attributes associated with each item in the set — including name, location, type, and representation. Schema-related information includes information about what a column means — for example, if it is an ID of a customer, whether a customer means the person bought something or they just visited the store.</p><p><strong>Shape: </strong>Shape refers to the allowed values and statistical distribution of your data. For example, you may have a shape expectation around whether or not the column contains a key or has an expectation that all values are unique. Another important shape expectation concerns whether or not the column allows nulls, and if so, how null is represented. When building a training set, some other key sets of expectations include whether or not a numeric field has a normalized Gaussian distribution, and what the enumeration values are in an enumeration field.</p><p><strong>Scale: </strong>Scale has to do with the size of the data set and its expected growth (in terms of how many rows or elements in the data set). Usually, a data process or pipeline is optimized based on the scale of the data at the time it is developed, allowing for expected near-term data growth. When the data set grows in size past those expectations, processing time and other process-related issues will cause increasing lag between when your product first sees a particular data item and when that item is reflected in your data product.</p><h3>How Data Issues are reflected in the end product</h3><p>Sometimes when an unanticipated or unaddressed change occurs, the data pipeline just breaks — for example, if someone changes the name of some column in a table that you are using. However, frequently the pipeline continues to function in a degraded way. This degraded quality eventually leads to inaccuracies in the data product; then, the user of the product loses trust in it.</p><p>The trouble is, that just looking at your data product may not give a direct indication that something is wrong. If you actually are looking at the data directly or doing data exploration, often you will see it. However, if you are looking at aggregated data, perhaps in a dashboard, the aggregation may obfuscate the change. Changes in schema or shape may be masked out within the aggregation process. Lag caused by scale issues may not be observable as your product may not retain any features that give a window into data freshness. Generally, the more sophisticated the data processing is, the harder it is to figure out that there are issues with the input data.</p><p>Similarly, if you are creating features from the data and using those features in a pipeline that ends with model-training or testing, any problems with the data are hard to see. For classification or regression models, you may see a drop in your accuracy metrics. In a recommendation systems, the recommendations may become inaccurate, old or outdated — a situation which is hard to test for and generally makes the users annoyed.</p><p>As I mentioned earlier, issues with schema, shape and scale are inevitable in any long-lived data product. They are unavoidable results of changing and improving products. When your data product degrades because you fail to take into account an impactful change in your raw data, that degradation often leads to distrust in the product itself.</p><p>However, it is also important to note that there are situations where delivering a product based on corrected data can be as impactful as suddenly delivering incorrect data. First, any sudden change in how the data product works — tangible and visible sudden changes — can be disruptive and concerning to the user even when the change is beneficial. Second, if the broken version of the data product was providing results that the user preferred to the results your corrected version is providing, then fixing the data product can be a hard sell.</p><h3>Aspects of a protective data culture</h3><p>Producing a trustworthy data product starts by putting structures and processes in place to keep you aware of the trustworthiness of your input data. These processes not only include data monitoring, but also where possible include practices that facilitate communication with the producers of that data about impactful changes they are making or seeing. In a non-production or more informal data environment, often the people who maintain a set of data do not know who actually is using it and how important that usage is. This same concern holds for input data that is collected and produced externally to your company.</p><p>In a more mature data pipeline, the input data you are using also may be used in several other different data products with competing requirements. This means that if the producers of that data ‘fix’ the data for one product, that fix may sabotage another. Communicating your expectations about your input data to the team that produces it, as a part of the productization process, gives the producing team an insight into your requirements on their data. It enables them to be intelligent and careful about how they make changes to their product as they juggle all the requirements from all the different data products that use their data. It also enables them to be proactive about addressing the impact planned changes may have on each of the products that use their data.</p><p><strong>Reactive defenses: </strong>Your first line of defense against unexpected and unaddressed change in your input data is to monitor the characteristics of that data that affect the aspects of your data that are key to your product. In the next section, I will provide some guidelines for how to determine exactly what to monitor.</p><p>Note that monitoring is a component of a reactive solution because you use the monitoring to detect changes that have already taken place. Then you react by adjusting your data product to compensate for the changes.</p><p><strong>Proactive defenses: </strong>Proactive defenses all revolve around maintaining good lines of communication specifically with those responsible for producing the data. As a user, you are not in control of what they do with their product, but you should be a part of managing relevant changes that they make.</p><p>The first step is to keep tabs on what exactly does your data product assume about the incoming data, based on how it is implemented. We discuss how to determine these expectations/requirements in the next section. In the initial stages, it may be sufficient to document your expectations in a way that is accessible to the data producers, that they can use as a reference point. You also can include in the documentation a person to contact when they think they are making a change that will affect you.</p><p>A second approach you can take if the producers of your input data are within your company is to harness your company’s project tracking processes to enable you to get a heads up whenever some change that may impact you is planned. That is, you communicate your expectations/requirements to the project managers responsible for the input data. They can then be sure that you are listed as a stakeholder in any ticket that may affect the ability to meet those specific requirements. You can then be developing updates to your product in synch with their updates — because you have a line of communication and accountability.</p><p>A third approach is including some or all of your data requirements into a lightweight data contract. A data contract is a formal agreement between your group and the group producing your input data. It abstractly describes the constraints that the data needs to maintain. This, coupled with a process for when those constraints need to change, facilitates a smooth transition. However, data contracts can get cumbersome so this approach is better limited to more mature and stable data products.</p><h3>Setting up your data requirements for your specific data product</h3><p>In this section we briefly describe how to determine your requirements from your data product — specifically, given an already implemented data process and product, what requirements must hold for that product to continue to execute as designed?</p><p>We describe the attributes as constraints over a data set (table), its records (rows) and attributes (columns).</p><p><strong>Schema: </strong>For a given attribute, we look at its</p><ul><li><strong>Name:</strong> The string used to name that attribute in the queries and processes.</li><li><strong>Location: </strong>What data set the attribute is found in.</li><li><strong>Type:</strong> The semantic type, e.g., positive integer, identifier, location.</li><li><strong>Representation:</strong> How it is represented, e.g., long, formatted string, uuid, specific enumeration.</li></ul><p>For all attributes named and used in your data product, you will want to ensure that the name, location, type and representation remain constant.</p><p><strong>Shape: </strong>We can describe the attributes by function in the input data and use in the data process:</p><ul><li><strong>Primary identifier (PI):</strong> Identifier that uniquely identifies a record in your data set. For example, a customer ID may be the primary identifier in your customers data. For Primary Identifiers (PI), you want to ensure that all values are unique, no values are null, and no values are invalid.</li><li><strong>Secondary identifier (SI):</strong> A reference to a primary identifier in a different data set. For example, an orders table may use the customer ID to reference the customer that placed the order. For Secondary Identifiers (SI), you want to ensure that no values are null and no values are invalid. In this case, valid means that the value is represented in the Primary Identifier that it connects to.</li><li><strong>Join attribute (JA):</strong> An attribute in your data set that you use to connect your records with records in some other data set, for example, using a join. Note that it is best to avoid doing this. However, if you are doing this, you need to pay attention to problems related to joining on null values and duplicated values. Either situation can cause extra, unexpected connections to be made during the joining process.</li><li><strong>Filter attribute (FA):</strong> An attribute you use to filter out some of the records in your data set. For example, you may use a date attribute to filter out records that are too old. If there is a filter on an attribute, either just filtering out rows or within the context of a join, the expectations/requirements only apply for the records that actually pass the filter.</li><li><strong>Other product attribute (PA):</strong> Any other attribute used in your product, e.g. in your feature set or for your visualizations. For example, you may have a chart that breaks down orders by product category, or use a product category in orders to forecast supply chain needs. In either case, the product category is a PA. For other product attributes (PA), you need to pay attention to how the shape is reflected in any visualizations or in any shape assumptions made related to features in your ML models.</li></ul><p><strong>Scale: </strong>For all data sets, you want there to be no significant deviations from expected row count. Also, you want the new data to be available in the expected time frame.</p><h4>Summary</h4><p>In a <a href="https://towardsdatascience.com/how-to-build-a-data-product-that-wont-come-back-to-haunt-you-1a220f4c75fb">previous post</a>, I discussed four quality criteria to uphold at different points in your data pipeline. This article explicitly discusses maintaining the criterion of trustworthiness, specifically as it relates to the first mile.</p><p>Change is inevitable. Many times, change is a good thing. Your well-designed data product depends on an incoming stream of raw data that is subject over time to data drift, bit rot, and a general degradation due to changes related to the schema, shape and scale of the data in the incoming data stream.</p><p>The implementation of your data product — which parts of the raw data you use and how you use that data — inherently place assumptions or expectations about the schema, shape and scale of the raw data itself. However, that data is produced by others and you have little control over the changes that they are making. Monitoring and adapting to relevant changes in your input data can be done reactively, by testing your input data. Change can also be addressed proactively, by identifying your minimal expectations and maintaining an open line of communication with the producers of the data about those expectations. The use of process components such as data contracts or appropriate stakeholder management can help in supporting the ability to proactively adapt your code to anticipated near-future changes that the producers of your input data are putting into place.</p><p>This article outlines specific data features that affect the long-term function of your data product. Approaches such as monitoring and data contracts need to look specifically at the outlined features. <strong>The more stable and central the data product is, and the longer it is expected to run, the more it is advisable to have formalized and proactive management of change in the relevant aspects of the input data.</strong></p><p><a href="https://www.linkedin.com/in/mistynodine">Marian Nodine</a> is Chief Data Scientist at Mighty Canary.</p><p><a href="http://www.mightycanary.com/">Mighty Canary</a> helps analytics teams find their zen again and do what they love most – discovering and driving insights. Did you know most analytics teams spend 30% of their working hours answering user questions? Mighty Canary has the tools you need to proactively communicate changes in the data and educate users on what all of those reports mean.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9394da10b9c9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/sssneaky-data-problems-that-creep-in-over-time-9394da10b9c9">Sssneaky Data Problems that Creep in Over Time</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Build a Data Product that Won’t Come Back to Haunt You]]></title>
            <link>https://medium.com/data-science/how-to-build-a-data-product-that-wont-come-back-to-haunt-you-1a220f4c75fb?source=rss-929a3612398a------2</link>
            <guid isPermaLink="false">https://medium.com/p/1a220f4c75fb</guid>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[notes-from-industry]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Marian Nodine]]></dc:creator>
            <pubDate>Fri, 30 Sep 2022 04:26:43 GMT</pubDate>
            <atom:updated>2022-09-30T04:26:43.570Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="A Kanban Board with lots of questions like “Why is this graph going down? It used to go up?” and “Where is the Germany data?”. The questions and interruptions are on red sticky notes. The new product the person is working on is “Explore howto recommendation automation”. It is on the lone green sticky note. It is in the Parked state on the Kanban board." src="https://cdn-images-1.medium.com/max/1024/0*T7bXJEQHE6cwTJVL" /><figcaption>Image by author</figcaption></figure><h4>When data science meets production engineering</h4><p>Think back to a time when you were asked to turn your data science analysis into a repeatable, supported report or dashboard that fits in with your company’s production data pipelines and data architecture. Perhaps you previously conducted prototype analysis and the solution was informative and working well, so <strong>your boss asked you to put it into production</strong>. Perhaps you found yourself constantly re-extracting data for the same type of analysis, training, and testing a new model, and you wanted to <strong>automate the whole process to free up precious time</strong>.</p><p>Alternatively, you may be experiencing issues with your production data products. Perhaps you have data science-supported dashboards that you have developed and put in production, but <strong>users are getting frustrated</strong> with them and losing trust. Perhaps you find yourself <strong>wasting time</strong> supporting your existing production dashboards and reports impeding your ability to work on newer, more interesting projects.</p><p><strong>Moving data products into production is an art in and of itself</strong>. Done well, automation can increase the accuracy, credibility, and velocity of the data science team’s work. Done poorly, automation can lead to the compounding of errors, which can spread frustration and distrust.</p><p>Many initial data science analyses start from scratch and process all of the data through a hand-crafted pipeline. Because they start with a bespoke data process, the data scientist later can become bogged down personally in the responsibility for managing the pipeline. They spend extra time examining their pipelines to detect emerging problems, fix them in a timely manner, and do what’s necessary to recover lost trust when the consumers of the product lose trust.</p><p>When putting a dashboard into production, you are looking not only at what the data, reporting, and users are like, but also how the data and user base will change in the future. A report may be clear and accurate to current users now, especially since you are around to answer any questions they may have. However, data drift may cause inaccuracies in the report in the future. A manager you do not normally interact with may come across your report, not clearly understand it, and not know where to go to get help to understand it. Thus, both the quality of the data product and the support for using and interpreting it degrade over time.</p><p>Over the years, I have worked both as a lead data architect, designing the pipelines which feed into the data processes, and as a lead data scientist, leveraging the engineering data to develop models used in production processes and visualizations.</p><p>This article codifies some of the lessons I have learned along the way. I describe four desirable properties for a production data pipeline — self-explanatory, trustworthy, adaptable and resilient. For the data pipeline to function well, you need to provide scaffolding at three different data processing phases — with the data you are using as input (the first mile), the way you present results to the user (the last mile), and the in the functioning and execution of the data processing and model development (everything in between). Considered together, these two axes provide 12 different touch points each of which has one or more important considerations to address. Here are some of the considerations and the important concerns I address at each of these touch points, before I release my production reports, dashboards and/or models into the wild.</p><h3>Three Pipeline Phases</h3><p>We say that a chain is only as strong as its weakest link. This is true also in data pipelines. The strength of your data product is only as strong as the weakest part of the data, pipeline, and processes that are used to form the product. Yet, in the case of a data product, many of these ‘links’ are not under your control.</p><p>How do you build the strongest product possible? You pay attention to the most important things to strengthen at each of the three phases of your particular part of the pipeline — input, output and execution.</p><p><strong>Your first mile: Are you starting with the right stuff?</strong> The concerns in this phase focus on the input data that you are putting into your pipeline or model. Your ability to support your users depends on the quality of the input data you are starting with. Failures and degradation in the input data are often out of your control, but have a serious impact on how well your data product is received and accepted.</p><p><strong>Your last mile: Are you delivering understanding, trust and confidence along with your data product?</strong> You may have an awesome data product that may really meet a need, yet all of this is moot unless the user understands the report and has confidence in the freshness and accuracy of its underlying data. The concerns in the last mile focus around building and maintaining this trust not only from your current report viewers, but also for anyone who may view the report in the future.</p><p><strong>The journey between: Are you making things easy for you, future you, and your future replacement? </strong>A good data product will live on over a long period of time. Longevity and maintainability considerations focus on your own specific extracts, transformations, and load processes. Data pipelines go down, can be degraded due to scheduling inefficiencies and can develop scale issues. Data inputs change or go away. Over time, you may see data drift, or model drift in the models produced using your data.</p><p>Supporting a production process can be time-consuming, and you want to facilitate the maintenance process for yourself, others working with you who may share that load, and anyone who may replace you.</p><h3>Four Important Criteria</h3><p>The second axis comprises the different quality criteria — the ones that make your product stronger. I look at four criteria to evaluate how resilient my data product is to future data issues and user challenges. Is the product <strong>self-explanatory</strong>? Does it <strong>inspire trust</strong> and confidence? Is it <strong>easy to adapt</strong> as data inputs change? Is it <strong>reliable, accurate, and maintained</strong>? Careful attention to these when developing your production pipeline helps reduce the amount of time necessary for future maintenance tasks.</p><p><strong>Self-Explanatory</strong>: As soon as you put your data product into production, people who you don’t interact with directly may be looking at it. The more self-explanatory the data product is, the easier it will be to support those future users. You want to be out of the loop as much as possible while still providing good support.</p><p>Over time, either you or someone else will need to do a certain amount of monitoring and maintenance to ensure that the data product stays updated and accurate. Keeping a low required mental load for picking the report back up and learning, or reminding yourself of how the report works, facilitates this kind of maintenance.</p><p><strong>Trustworthy</strong>: Trust is hard to gain and easy to lose. Usually, when you first deliver a data product, there is a lot of hope and some trust. For a report or dashboard, trust really develops as the user gains a longer-term sense that the report or dashboard is:</p><ul><li>answering their questions</li><li>using accurate data</li><li>being kept up-to-date</li><li>provides a reliable way to resolve issues or potential problems they have with what they see</li></ul><p>If the user starts to doubt any of these things, trust starts to erode and the report no longer is as useful as it was at the beginning. Repeated or overlapping concerns mean that the product eventually drops off their radar, and the user moves to trying to answer their questions using a different method.</p><p><strong>Adaptable</strong>: Data inputs and shapes change over time. Schemas change. The distribution of key values changes. Places stop reporting data. Data sets can grow and scale beyond what your data processing and queries can effectively handle. All of these impact the ability of an automated process to deliver an accurate and timely product.</p><p>As a data scientist, you do not want to spend your time looking through each of your products frequently and in detail to ensure that you notice all glitches in the data and all places where things may not look right. Rather, it is better to delegate the monitoring task to an automated system, and do a detailed deep dive into each report on a slower cadence. Things you can monitor for automatically include:</p><ul><li>relevant changes to the schema for one or more input data streams</li><li>relevant changes to the shapes of the data in key fields that you are using</li><li>significant increases or decreases in data volume</li><li>significant lag between the most recent record used and the current time</li><li>concerning trends in data volume or lag</li></ul><p>In addition to monitoring relevant issues, when someone notices a problem, you need to be set up to react and formulate a solution without undue complications. Knowing exactly where to look is the first step, but setting up your data processes in a way that is easy to adapt and maintain is also key to success.</p><p>When you are designing and building your pipeline, build it with your future self (or possibly your replacement) in mind. Automate the process of periodically retraining your models to take into account changes in the data. Have a cadence for looking over your data and models to ensure they still conform to the assumptions you made when you set up the initial process.</p><p><strong>Reliable</strong>: Reliability is all about how well the execution of your pipelines and models is working. In this context, pipeline reliability relates to process execution and timing in the pipeline. This contrasts with adaptability (above), which is related to the actual data being processed.</p><p>To ensure reliability, you need to ensure that the different parts that make up your pipeline are reliable and the processes that orchestrate those parts are robust.</p><p>When the data and model-building pipelines are automated, you have no immediate and centralized feedback into what is executing, whether any issues were encountered, or even what the resulting data product looks like. To this end, to understand the reliability of the different parts or stages, it helps to have a monitoring system that centralizes this information and alerts you to things you need to pay attention to.</p><p>To monitor your entire process for each processing stage, monitor each stage’s reliability. Monitoring a stage includes checking such things as:</p><ul><li>its input data was updated in a timely manner</li><li>the stage started to execute at the expected time</li><li>the stage completed successfully</li><li>the stage completed before any stages downstream needed to read its results</li></ul><h3>Putting Things Together</h3><p>When shaping my initial system for production quality, I ask specific questions at each phase of the pipeline, focused on the aforementioned concerns. Working through these questions systematically, I am able to address quality concerns at all 12 of the touch points.</p><p><strong>Self-Explanatory / First Mile:</strong></p><ol><li>Is the data I am using well-documented?</li><li>Am I using the data in line with its intended use?</li><li>Am I reusing as much of the existing engineering pipeline as possible to lower overall maintenance effort and ensure that my use is consistent with other uses of the same data items?</li></ol><p><strong>Self-Explanatory / Last Mile:</strong></p><ol><li>Is the report or dashboard presented in a way that is easily accessible and understandable even to people who will be viewing it, and without explanation from me?</li><li>Am I using vocabulary and visualizations that the end user understands, and that they understand in exactly the same way I do?</li><li>Am I providing good supporting documentation and information in a way that is intuitive and easily accessible from the data product itself?</li></ol><p><strong>Self-Explanatory / In Between:</strong></p><ol><li>Are the requirements, constraints and implementation of the data process documented well enough that someone else who may be taking over maintenance from me can understand it?</li></ol><p><strong>Trustworthy / First Mile:</strong></p><ol><li>Am I connected into the input data in a way that is well-supported in the production pipeline?</li><li>Do I have explicit buy-in from those maintaining the data sets I am using?</li><li>Is this input data likely to be maintained for a significant time in the future?</li><li>When might I need to check back to be sure the data is still being maintained?</li><li>How do I report any problems that I see in the input data?</li><li>Who is responsible for notifying me of issues with the data?</li><li>Who is responsible for fixing those issues?</li></ol><p><strong>Trustworthy / Last Mile:</strong></p><ol><li>How does the user know when they can trust the report is accurate and up to date?</li><li>Is there an efficient and/or automated way of communicating possible problems to the end user?</li><li>Is there a clear and accessible process in place for the user to report concerns with the data or report, and for the user to be notified of any remediation processes in place?</li></ol><p><strong>Trustworthy / In Between:</strong></p><ol><li>Have I set up a regular schedule to review the data and report to ensure that the data pipeline is still functioning well and the report is conforming to the requirements?</li><li>What are the conditions under which this report should be marked as deprecated?</li><li>How can I ensure that the user is informed should the report become deprecated?</li></ol><p><strong>Adaptable / First Mile:</strong></p><ol><li>What features in the input data am I depending on for my analysis?</li><li>How will I know if those features stop being supported, are affected by a schema change, or change shape in a way that may affect my analysis?</li><li>How will I know if the scale of the data grows to a point where I need to refactor my process in order to keep up with my product’s requirements for freshness?</li></ol><p><strong>Adaptable / Last Mile:</strong></p><ol><li>Is the product or report set up in a way that it is easy to request a change and/or a new feature?</li></ol><p><strong>Adaptable / In Between:</strong></p><ol><li>Have I set up a regular schedule for re-examining the requirements to ensure that I am still producing what the user needs and expects?</li><li>What is the process for users to indicate changes in requirements, and for those changes to be addressed?</li><li>What is the process for refactoring and retesting the data pipeline when the inputs change in some relevant way?</li></ol><p><strong>Reliable / First Mile:</strong></p><ol><li>Does my process fit well into the data practices and engineering production system in my organization?</li><li>Do I have an automatic notification system in place to monitor the availability, freshness and reliability of my input data?</li></ol><p>Reliable / Last Mile:</p><ol><li>Who is responsible for the ongoing monitoring, reviewing, troubleshooting, and maintenance of the dashboard itself?</li><li>Are responsibilities and procedures clearly in place for reporting and resolving issues internally?</li></ol><p><strong>Reliable / In Between:</strong></p><ol><li>Is each stage in my pipeline executing and completing in a timely manner?</li><li>Is there drift in the processing time and/or amount of data being processed at any stage that may indicate a degradation in pipeline function?</li></ol><h3>Conclusion</h3><p>Putting a data product into production presents challenges beyond those of making something perform well as a one-off or as a prototype. Proper attention to production-related concerns from the start accelerates your overall productivity by minimizing the amount of time you spend on routine maintenance tasks for your in-production products.</p><p>When you first build a data product, you wrestle with issues of what data is available to use and how to best use that data in your product to satisfy the requirements of the intended user. This is often somewhat of a solo effort for the analyst or data scientist.</p><p>Putting something into production is more of a team effort. There are usually others — perhaps a data engineering team — responsible for making sure that the data sources you rely on are kept accurate and up to date. You want to connect with their data in a way that works best for them. You also have your users — now a more open-ended set of people — who will be using your product. You want them to trust your product. You want to make using and understanding your product as straightforward as possible. Finally, you have your own data pipeline. You want to know as soon as possible when something happens that impacts its function.</p><p>The 12 touch points in this article and the questions associated with each touch point provide a structure for working through your productizing process. Paying attention to these questions will help prevent you from getting bogged down in support and will give you more time to be creative and build new things.</p><p><a href="https://www.linkedin.com/in/mistynodine">Marian Nodine</a> is Chief Data Scientist at Mighty Canary.</p><p><a href="http://www.mightycanary.com/">Mighty Canary</a> is a company focused on building tools to help data and analytics professionals increase trust in their reports and reduce the time they spend supporting old reports.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1a220f4c75fb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-build-a-data-product-that-wont-come-back-to-haunt-you-1a220f4c75fb">How to Build a Data Product that Won’t Come Back to Haunt You</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>