<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kyle Kirwan on Medium]]></title>
        <description><![CDATA[Stories by Kyle Kirwan on Medium]]></description>
        <link>https://medium.com/@kylejameskirwan?source=rss-e953d7ecf989------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*7BRfiFucJDX7MnU0tE2UTg.png</url>
            <title>Stories by Kyle Kirwan on Medium</title>
            <link>https://medium.com/@kylejameskirwan?source=rss-e953d7ecf989------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 09:55:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kylejameskirwan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[What is data discovery?]]></title>
            <link>https://medium.com/@kylejameskirwan/what-is-data-discovery-f11ec59d8ce2?source=rss-e953d7ecf989------2</link>
            <guid isPermaLink="false">https://medium.com/p/f11ec59d8ce2</guid>
            <category><![CDATA[data-reliability]]></category>
            <category><![CDATA[data-observability]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-discovery]]></category>
            <category><![CDATA[data-discovery-tools]]></category>
            <dc:creator><![CDATA[Kyle Kirwan]]></dc:creator>
            <pubDate>Fri, 25 Aug 2023 17:04:15 GMT</pubDate>
            <atom:updated>2023-08-25T17:04:15.610Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s_GizUDBc8sTkED2z2cWYw.png" /></figure><p>With data volumes growing exponentially each year, many companies are drowning in their data. New business leaders don’t know which tables answer basic questions around revenue and usage. Often, due to duplicated tables, they can’t even figure out a single trustworthy answer. How do you break this cycle and tap into vast data assets to drive real business value?</p><p>That’s where data discovery comes in. In this post, we explore what data discovery entails, its key capabilities, tooling, and best practices.</p><h3>Data discovery: Objectives and benefits</h3><p>Is your data searchable and accessible? “Data discovery” encompasses the solutions, tools, and practices that allow users to easily search, access, understand and analyze an organization’s data assets.</p><p>One of the primary objectives of data discovery is to provide users with more efficient data access. Organizations deal with vast volumes of data that spreads across multiple sources. It’s challenging to understand all of that data’s overall structure and quality. Data discovery cuts down the time spent searching for and prepping data.</p><p>Data discovery also plays a pivotal role in improving <a href="https://www.bigeye.com/blog/the-complete-guide-to-understanding-data-reliability">data reliability</a>. Through data discovery, teams profile data to assess its accuracy, consistency, and completeness. If duplicates or discrepancies occur, the process of data discovery roots them out.</p><p>Finally, data discovery supports data governance strategies by providing a clear view of data’s location, lineage, and usage. This transparency makes it easier to establish accountability for data quality and protection, enforce data standards, and manage data access rights. Better data governance leads to higher trust in data and its derived insights.</p><h3>The five concepts of data discovery</h3><p>Make sure you understand these foundational concepts associated with data discovery:</p><h3>1. Data source identification</h3><p>The process of finding and recognizing where data is located within an organization’s systems and infrastructure. These data sources can be databases, data warehouses, data lakes, cloud storage, external data feeds, and more. Understanding these sources is critical to performing data discovery.</p><h3>2. Data profiling</h3><p>Not all data matters, and some data is more critical than others. Data profiling is the process of examining and summarizing a dataset to understand its structure. It includes statistical analysis, data type recognition, and detection of anomalies like missing values, outliers, or duplicates. This process provides a comprehensive understanding of the data, to determine its reliability and fitness for various analytical purposes.</p><p>For example, <a href="https://www.bigeye.com/blog/data-in-practice-systematizing-data-quality-at-uber-scale">Uber introduced a tiering system for its data assets</a>, classifying them from Tier 1 (highly critical) to Tier 5 (individually-owned data in staging with no guarantees). Out of over 130k datasets, only 2,500 were identified as Tier 1 and Tier 2. This significantly cut down on the amount of high-priority maintenance work that needed to be done.</p><h3>3. Data classification</h3><p>In data discovery, classification is used to categorize data based on predefined criteria. This process helps understand data types, like whether it is sensitive (like PII or confidential business information) or non-sensitive, structured or unstructured. Classification can also help identify data subject to regulatory compliance like GDPR, CCPA, or HIPAA.</p><h3>4. Data cataloging</h3><p>A data catalog is a structured and enriched inventory of data assets. Cataloging involves the creation of metadata that describes various aspects of the data, like source, owner, format, and usage. This not only improves data accessibility but also supports data governance initiatives.</p><p>Maintaining a comprehensive data catalog makes it easier to search and discover relevant data for specific analytical tasks. Then, analysts and data scientists can access the right data easily and make informed decisions.</p><p>The implementation details of a data catalog service might differ. But, looking at examples from Shopify, Meta, etc., they typically involve an (1) ingestion pipeline, (2) an indexing service and (3) a front end component for end users. The ingestion pipeline fetches information from a variety of data stores, then processes and stores them into a search index such as Elasticsearch. This data is then further exposed to end users, for example, by GraphQL APIs via an Apollo client.</p><h3>5. Data lineage</h3><p>Understanding data lineage — data’s origins, transformations, and where it moves over time — is key in data discovery. It assists in root cause analysis, data quality tracking, compliance, and impact analysis. Data lineage is also key for compliance and auditing purposes. How does a change in data impact the rest of the organization? Data lineage can tell you.</p><p>At Shopify, <a href="https://shopify.engineering/solving-data-discovery-challenges-shopify">data lineage is built on top of a graph database</a>. This allows users to search and filter the dependencies by source, direction (upstream vs. downstream), and lineage distance (direct vs. indirect). The lineage data offers users insight into how other teams use a data asset and informs owners about potential downstream impacts due to modifications.</p><h3>The relationship between data discovery, data governance and data quality</h3><p>A lot of the concepts in data observability can sound and feel similar. So let’s explore some key questions surrounding the intricacies of data discovery and its related practices and frameworks.</p><h3>How are data discovery and data governance related?</h3><p>Data discovery and data governance are like two sides of the same coin in managing a company’s data.</p><p>Data discovery is about finding and understanding your data. It involves figuring out where your data comes from, what it’s about, and how it can be sorted and understood. It’s like making a detailed map of all your data and using tools to better understand what’s in it.</p><p>Data governance is different in that it serves as a rulebook for your data. It outlines the rules for who can use what data, when, and how they can use it. It also involves setting up procedures to ensure everyone follows these rules.</p><p>These processes work closely together. Once you’ve found and understood your data with data discovery, you need to make sure it’s managed properly according to the rules set out by data governance. This partnership ensures the data is used in the right way and that the company stays within any legal or regulatory rules. The process of finding and understanding data can also highlight areas where the rules can improve.</p><h3>How are data discovery and data quality related?</h3><p>Data quality involves checking whether your data is reliable, complete, and accurate. The connection between Data Discovery and Data Quality is crucial. If your data is high-quality, the insights you get from the discovery process will be more accurate and useful.</p><p>The discovery process can also improve data quality, as it generally involves things like assigning ownership and generating metadata. It also surfaces and exposes more data to business users. Because these users typically have business context and expectations into what data <em>should</em> look like, Data Discovery allows them to ask questions or flag potential anomalies or inaccuracies with the data quality team.</p><h3>Data discovery tools</h3><p>Several commercial and open source tools exist to enable data discovery capabilities. The following tools will help you organize data, <a href="https://www.bigeye.com/blog/complete-guide-to-understanding-data-observability">enable data observability</a>, manage details about the data, create visual data representations, and perform flexible data analyses.</p><ul><li>Cloud services like AWS Glue Data Catalog and Azure Purview play a vital role in cataloging, classifying, and organizing metadata. They provide a centralized repository to store information about various data assets, including data sources, schemas, and data quality metrics. These catalogs streamline data discovery by allowing users to search and discover relevant data assets easily.</li><li>Data governance tools like Collibra and Alation provide comprehensive solutions for data lineage, quality, security, and lifecycle management. They facilitate collaboration among data stakeholders, ensuring a consistent understanding of data assets and promoting data-driven decision-making.</li><li>ETL/ELT platforms like <a href="https://www.bigeye.com/blog/getting-started-with-dbt-tests">dbt</a> and Airflow play a crucial role in data profiling, preparation, and catalog integration. They automate the extraction, transformation, and loading of data from various sources, ensuring data consistency and quality. They often also come with <a href="https://www.getdbt.com/blog/using-dbt-docs/">auto-documentation features</a> that make it easier to keep track of data job metadata.</li></ul><h3>Data discovery best practices</h3><ul><li>Data Profiling and catalog maintenance: Utilize data profiling tools to understand data characteristics and maintain an up-to-date data catalog for efficient search and discovery.</li><li>Data lineage tracking: Implement data lineage to trace data transformations and maintain data accuracy across processes.</li><li>Metadata management and governance: Establish robust metadata management and data governance practices to promote consistency and compliance.</li><li>Data observability and quality: Implement data pipeline and quality monitoring and a remediation process with the help of a tool like Bigeye. This will help teams quickly identify and rectify any issues that come up during the discovery process and ensure data is reliable moving forward.</li></ul><p>In addition to these, adopt an <em>iterative</em> approach to data analysis that allows for step-by-step exploration, helping uncover hidden trends and relationships. Don’t ignore data <em>security and privacy safeguards.</em> Finally, make sure to create <em>documentation</em> around data discovery processes and cross-functional training support.</p><h3>Final thoughts</h3><p>To begin your data discovery journey, start by asking and addressing these questions:</p><ol><li>Where does your data reside? Identify all the sources of your data. This includes databases, cloud storage, data lakes, external APIs, and more.</li><li>What is the quality of your current data? Sort the data into different tiers that need to be maintained at different levels of quality.</li><li>What tools and resources do you currently have? Take inventory of the existing tools, platforms, and resources at your disposal. Determine if you will need to purchase an off-the-shelf data discovery tool.</li><li>How will the results and insights be communicated and implemented? Decide on the channels and methods for disseminating the insights derived from data discovery to relevant stakeholders, whether it will be a UI, a data catalog, or simply deleting a bunch of tables.</li></ol><p>Bigeye’s data observability tool can play a critical role in enacting your data discovery framework. Want to learn more? <a href="https://www.bigeye.com/request-demo">Try Bigeye today</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f11ec59d8ce2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Snowflake Summit 2023 and data observability: 5 key takeaways]]></title>
            <link>https://medium.com/@kylejameskirwan/snowflake-summit-2023-and-data-observability-5-key-takeaways-b82f5e8e08a0?source=rss-e953d7ecf989------2</link>
            <guid isPermaLink="false">https://medium.com/p/b82f5e8e08a0</guid>
            <category><![CDATA[data-reliability]]></category>
            <category><![CDATA[data-observability]]></category>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data-quality-management]]></category>
            <dc:creator><![CDATA[Kyle Kirwan]]></dc:creator>
            <pubDate>Mon, 10 Jul 2023 04:00:52 GMT</pubDate>
            <atom:updated>2023-07-11T17:17:53.976Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BV1S2-iVACvktCuh.png" /></figure><p>The desert city of Las Vegas might not be the first place you think of when the word “Snowflake” hits your ears, but it was indeed the epicenter of all things data last week. The annual Snowflake Summit 2023, a confluence of data professionals, industry leaders, and technophiles alike, was the biggest annual gathering yet: thought-provoking discussions, illuminating keynotes, and countless networking opportunities.</p><p>From the unveiling of groundbreaking technologies to visions for the future of data-driven businesses, the summit made waves that are sure to ripple through the industry for years to come. Whether you were able to experience the magic firsthand or, like many, were keeping up from a distance, there’s a lot to unpack. We’ve done the heavy lifting to distill some key takeaways from this incredible event.</p><h3>1. Data observability: What a difference a year makes</h3><p>Even from our niche vantage point, we’ve seen incredible growth in the recognition and mainstreaming of <a href="https://www.bigeye.com/blog/complete-guide-to-understanding-data-observability">data observability</a>. Last our conversations at Snowflake centered around “What is data observability?” This year, it was a different story.</p><p>At our booth, most visitors already knew all about data observability. Their questions were not about education on the space, but more so how Bigeye fits into the data quality monitoring universe.</p><p>For the most part, the majority of folks we spoke to were still doing manual data quality monitoring.</p><h3>2. AI: Not just the buzzword of the hour</h3><p>In Snowflake CEO’s keynote speech, Frank Slootman underscored the importance of a data strategy to undergird an AI strategy. In the era of “generative AI,” organizations need to fundamentally prioritize their data strategy through robust, reliable data that works.</p><p><a href="https://www.businesswire.com/news/home/20230627644318/en/Snowflake-Unveils-New-Large-Language-Model-to-Extract-Deeper-Insights-from-Documents-While-Continuing-to-Advance-Platform-Speed-and-Performance">Snowflake announced “Document AI</a>”, which will leverage Snowflake’s LLM to unlock deeper insights from documents faster, more simply, and with more accuracy. Snowflake has expanded support for Iceberg tables, which will extend Snowflake’s performance and governance to data stored in open formats.</p><p>So while AI and generative AI could be seen as buzzwords (a fact that Slootman even acknowledged during his keynote), there is real innovation to be seen when AI strategy is powered by solid, reliable data.</p><h3>3. An industry-wide focus on simplifying data management</h3><p>Snowflake’s new <a href="https://www.snowflake.com/snowpark-container-services/">Snowpark Container Services (previewed)</a> hopes to bring fully managed Kubernetes to customers. Even with the most complex data environments, there’s an industry-wide drive to bring apps closer to the data layer and simplify data management. Automation and ML will help, and data observability will likely be a critical factor in creating safeguards around this streamlining work. 4.</p><h3>4. The Open Data Quality Framework from Alation</h3><p>At the conference, announced the launch of the Open Data Quality Framework to bring best-of-breed data observability capabilities into the Snowflake Data Cloud. This move formalizes the importance of data observability, further entrenching it as an important piece of the data quality puzzle ( <em>editor’s note:</em> Bigeye is one of the launch partners for this initiative).</p><p>The Open Data Quality Framework aims to strengthen data governance for Snowflake by elevating data quality information. Its existence will catalyze data teams into enforcing data quality policies and rules, as well as assigning data quality stewards and helping to manage the entire data quality lifecycle from one platform.</p><h3>5. The Snowflake Model Registry</h3><p>Currently, most clients of Snowflake develop their models on Snowflake Data by utilizing external tools. Snowflake is working towards a comprehensive machine learning workflow, incorporating smooth integration of model administration and observability. Currently, many clients source LLM models from a variety of sources and modify them to fit their specific needs by optimizing them with data from Snowflake. These clients now have the ability to launch these models in Snowflake for inference via the <a href="https://docs.snowflake.com/en/developer-guide/snowpark-ml/snowpark-ml-modeling">Snowflake Model Registry (Preview)</a>. They have the capacity to add or remove model versions, as well as tagging them with the metadata model. Snowflake also intends to offer a Feature Store in the future to streamline feature management.</p><h3>Final thoughts</h3><p>Snowflake Summit 2023 has not just set the bar high, it has helped redefine the parameters of data observability. New data observability developments point towards a future where data isn’t just passively managed but actively observed and controlled. It’s clear that the data landscape is set to experience a seismic shift. As innovation spreads, we stand on the brink of a new era in data observability, and it’s an exciting place to be. The future of data is not just about accumulation, but about smart, proactive observation, and the tools to do so are no longer a distant dream but a tangible reality. We’re eager to see how these developments will shape the world in the coming years. Until then, let’s keep the spirit alive by pushing the boundaries of what’s possible with data.</p><p><em>Originally published at </em><a href="https://www.bigeye.com/blog/snowflake-summit-2023-and-data-observability-5-key-takeaways"><em>https://www.bigeye.com</em></a><em> on July 10, 2023.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b82f5e8e08a0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Product management skills for data scientists]]></title>
            <link>https://medium.com/@kylejameskirwan/product-management-skills-for-data-scientists-4e423c2b8123?source=rss-e953d7ecf989------2</link>
            <guid isPermaLink="false">https://medium.com/p/4e423c2b8123</guid>
            <category><![CDATA[data-science-careers]]></category>
            <category><![CDATA[career-advice]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[product-manager]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Kyle Kirwan]]></dc:creator>
            <pubDate>Fri, 19 May 2023 18:00:39 GMT</pubDate>
            <atom:updated>2023-05-22T17:14:33.214Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nh1kO-7lXqLh7sTdgBUm_A.jpeg" /><figcaption>Image credit: Jason Goodman</figcaption></figure><p>At a pivotal point in my career, I moved from being a Data Scientist to being a Product Manager. I had worked on Uber’s Growth team, then on the A/B-testing platform. When the opportunity arose to work on internal data engineering and data science tooling, I jumped on it. My mandate: to help solve the day-to-day problems that slowed down Uber’s data people.</p><p>As a PM, I learned several common product management techniques to ship multiple internal tools. These tools served 3000+ weekly active users in Uber. I increased our CSAT (customer satisfaction) scores quarter over quarter.</p><p>In retrospect, I wish I had known those techniques during my work on data science projects. Several of them are useful beyond product management, and would have helped me deliver more impact as a data scientist.</p><p>There’s much more to product management in general than I can cover here, and so much that has informed our work at <a href="http://bigeye.com/">Bigeye</a>. But here are a few specific techniques that data scientists could leverage.</p><h3>Switching from a “Maker” Mindset to a “Solver” Mindset</h3><p>As a data scientist, I loved experimenting with new technologies and frameworks. I used a “Maker’s Mindset” and focused on exploration. A comprehensive understanding of a new technology would often lead to big step jumps in surprising areas.</p><p>One example: I spent several months learning about applying a Bayesian framework in A/B-testing. It was exciting to think about how we could achieve more intuitive experiment outcomes than frequentist approaches can provide.</p><p>My prized idea was shut down by teams who were only familiar with the frequentist approach, and never made it into the final product.</p><p>At the time, my question was: what exciting options exist for designing and analyzing experiments? My question <em>should have </em>been: What decisions will be guided from the results we generate, will a Bayesian framework help guide a different and better decision, and is that worth the cost of implementing something unfamiliar?</p><p>In a “Solver’s Mindset,” you don’t focus on the applications of an exciting technology or framework. You focus on producing an outcome against a target problem /set of problems facing a stakeholder. To put it in the terms of a <a href="https://en.wikipedia.org/wiki/Multi-armed_bandit">multi-armed bandit</a>, you need to switch from exploration to exploitation.</p><p>Had I used this mindset, I would have saved a lot of time and effort on the way to building something beneficial to my users. Periodically, ask: who am I building for, what problems am I trying to solve, and what actions will lead to the delivery of that solution?</p><p>Granted, sometimes it’s a data scientist’s job to understand the cutting edge, just in case it can totally change the product. All of the large language models are a good example of that (chatGPT, for example!). If a data scientist only focuses on solving problems instead of understanding the potential of future problems, the team might be caught in a rut of old tech.</p><p>This technique is listed first because it’s a frame of mind you need to deliberately channel while working to define and deliver a <em>product </em>rather than exploring a <em>technology </em>or a <em>technique</em>.</p><h3>Using the “Five Whys” to root cause your stakeholders</h3><p>Once you’ve adopted the “Solver’s Mindset”, you need to be certain you’re solving the root causes of your stakeholders’ problems, not just the symptoms. Solving root causes gets you more leverage from your time. Each root cause often produces multiple symptoms, some of which your stakeholders may have duct-taped over and forgotten about. Solve the root cause, and you could be solving multiple problems for them at once.</p><p>The <a href="https://en.wikipedia.org/wiki/Five_whys">Five Why’s</a> start from a pain-finding question, e.g. “What’s the thing you feel like you waste the most time on right now?” or “What part of your job sucks the most?” or “If you had a magic wand, what would you change about your current workflow?”. Then you ask, “Why?” five times iteratively.</p><p>Ask your users the first “why,” take careful notes, and then ask again why <em>that </em>thing is a problem. You’ll move closer to uncovering the problem’s true source.</p><p>Doing this exercise with multiple stakeholders gives you insight into the underlying problems holding them back. Find a common theme in the underlying root causes, and massively increase your odds of producing a real impact.</p><h3>Scoping with an ROI Square</h3><p>Once you’ve identified a high-value, root-cause level problem to solve, you need to scope out the solution.</p><p>Assuming you can identify root-cause level solutions, you face a new problem: you’re likely left with more work than you can accomplish in a short timeframe. Scoring a concrete win quickly is the key to getting additional investment to move on to the harder components of your solution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*8yjOeZEtCNXmLIvXAN3Otg.png" /><figcaption><em>Image courtesy of the author</em></figcaption></figure><p>The ROI Square is designed to help you scope work into the parts that deliver the most bang for your buck, time and effort-wise.</p><p>First, draw a two-by-two matrix and label your x-axis “INVESTMENT” and your y-axis “RETURN.” Your x-axis should be in reverse, so higher investment levels are at the bottom of the axis.</p><p>Next, locate each potential solution on the matrix.</p><p>I find it helpful to think purely about INVESTMENT, and then RETURN for all potential solutions, rather than considering each one holistically. Detach from your biases toward any one solution, and treat them more like cattle than pets.</p><p>Rate each attribute on a “1–10” scale, giving the endpoints of each scale a description like “Six months+ of dedicated effort” for 10 on INVESTMENT or “Unlikely to be noticed by stakeholder” for a 1 on RETURN.</p><p>Once you have these ratings, place each solution idea onto the matrix. Look for the solutions in the top-right corner. This “High ROI” corner should contain the highest possible RETURN for the lowest possible INVESTMENT. As you plan out the execution of your project, build and deliver these ideas first, before moving on to the “low-hanging fruit” corner which contains low-INVESTMENT and low-RETURN ideas. Save the last two sub-optimal corners for later (or never, realistically).</p><h3>Mapping out your project with a workback plan</h3><p>At this point, you have high confidence you’re solving a root-cause level problem for your stakeholders, and you’ve identified high-ROI work to get you the most results with the least effort. Awesome.</p><p>Next you want clarity about what can be delivered to stakeholders. That clarity will help you communicate and ensure you’re delivering the RETURN you identified within a reasonable time frame. A project with great potential that remains unfinished because it gets scrapped after running 2–3x over the timeline doesn’t actually produce any result. A project that delivers 80% improvement against the target problem, but finishes roughly on time, is likely to result in follow-up investment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*MG_FjQbySPVTzivCXq_QvA.png" /><figcaption>Image courtesy of the author</figcaption></figure><p>This is an imperfect process. Often, I didn’t want to make an estimate about something I couldn’t predict or didn’t yet understand. But you need a plan that builds in flexibility as you progress.</p><p>Start from the high-ROI ideas you plan to deliver. Then, work backwards to your current state. For the first idea, what must exist or be completed? Those conditions are your first dependencies — document them within your chosen task-tracking system or Google Doc.</p><p>Next, ask what those dependencies require in order to be true. Iterate until you work backwards to the current state of the world. For example, the dependency for a regression model is “Have a data warehouse and the ability to do data transformations in it?” while as it happens, you already have Snowflake and dbt running.</p><p>You should have a fully fleshed out, tree-like structure of tasks be fulfilled. Completing these tasks should bring you from your current state to the high-ROI outcome state you think will resolve your stakeholder’s problem.</p><h3>Keeping stakeholders up to speed with a roadmap</h3><p>Your roadmap will help your stakeholders understand your timelines and deliverables.. It should be high-level enough that they can quickly learn how the project is being “chunked”, which chunks are arriving at what times, and give you feedback about that timeline. It may even guide you away from a task altogether because the timeline isn’t realistic (see “Reinforcement learning” below). A clear roadmap creates clarity, alignment, and confidence in the project from all sides.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*xnT8glemenCl1RTC-WTbNg.png" /><figcaption>Image courtesy of the author</figcaption></figure><p>Periodically review the roadmap with stakeholders to explicate what’s being delivered, any timeline adjustments, and upcoming milestones. Stakeholders are often receptive to timeline adjustments, as long as they’re kept in the loop and invited to provide input.</p><p>The roadmap might be expressed within your chosen task-tracking tool, a Google Sheet, or even photos of a whiteboard. The important part is that it:</p><ul><li>Is high-level enough that stakeholders can consume and understand it</li><li>Helps depict changes since the last update (good and bad)</li><li>Accurately represents the state of the project</li></ul><h3>Conclusion</h3><p>I felt in my element as a data scientist. I spent hours thinking through experiment design approaches, or reading about why the current null-hypothesis-testing paradigm is deeply flawed. But when it came to designing a large scale project with lots of stakeholders, I was a fish out of water.</p><p>I I picked up the above techniques over the course of several years leading a multidisciplinary product team to research, design, scope, and ship multiple products with thousands of users. They’re not the end-all-be-all of product management (a deep and often fuzzy discipline). But hopefully, they’re a helpful addition to your toolkit for embarking on and managing a complex project.</p><p>Have questions or thoughts about these techniques? Is there something I missed that’s worked for you? Let me know in the comments or shoot me an email: kyle at bigeye.com 🙂</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4e423c2b8123" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Reliability Engineering versus Site Reliability Engineering]]></title>
            <link>https://medium.com/bigeye/data-reliability-engineering-versus-site-reliability-engineering-4b5a8318f3fe?source=rss-e953d7ecf989------2</link>
            <guid isPermaLink="false">https://medium.com/p/4b5a8318f3fe</guid>
            <category><![CDATA[data-reliability]]></category>
            <category><![CDATA[data-monitoring]]></category>
            <category><![CDATA[data-observability]]></category>
            <category><![CDATA[site-reliability-engineer]]></category>
            <category><![CDATA[data-quality]]></category>
            <dc:creator><![CDATA[Kyle Kirwan]]></dc:creator>
            <pubDate>Tue, 21 Mar 2023 16:23:36 GMT</pubDate>
            <atom:updated>2023-03-22T15:21:54.661Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JIgGSDwTTI24I1YpIj4ueQ.png" /></figure><p><strong><em>Shameless plug: On April 19, 2023, we’re gathering the largest collection of data reliability engineers and experts in the world. Join us for the third annual </em></strong><a href="https://drecon.org/"><strong><em>Data Reliability Engineering Conference (DRE Con)</em></strong></a><strong><em>!</em></strong></p><p>Data and websites are two of your organization’s most valuable and visible assets. It’s only rational to have established, robust processes for ensuring that they both continue to function and stay available at scale.</p><p>Data Reliability Engineering (DRE) was created a few years ago, and based on Google’s <a href="https://www.bigeye.com/blog/seven-principle-reliable-data-pipelines">Site Reliability Engineering (SRE) principles</a>. DRE and SRE are both frameworks that modern teams use to solve technical problems in a scalable way. They share similarities in their approach to managing data and websites.</p><p>What can the nascent <a href="https://drecon.org/7-principles">Data Reliability Engineering</a> framework take from the more established SRE? Are there any key differences that keep them apart? Let’s explore.</p><h3>Data Reliability Engineering and Site Reliability Engineering: Similarities</h3><p>Whether they’re being applied to data warehouses and pipelines, or applications and infrastructure, the DRE and SRE frameworks exist to keep systems working reliably. DRE borrows the engineering principles and best practices of SRE to ensure reliability and resilience of data systems. Here are some key similarities between both frameworks.</p><h3>1. Standard-setting</h3><p>The way that both SRE and DRE work to monitor and manage incidents is through standard-setting. Whether through <a href="https://www.bigeye.com/blog/defining-data-quality-with-slas">SLAs</a>, data contracts, or less formal agreements, both DRE and SRE set standards to clarify responsibilities and deliverables through clear definitions, deadlines, hard numbers, specific metrics, and cross-team consensus.</p><h3>2. More automation</h3><p>Both DRE and SRE place heavy emphasis on <a href="https://www.bigeye.com/blog/automating-data-observability-at-scale">automation</a>. What does that look like in DRE? Regularly automating data backups, using observability tools to monitor for anomalies, and routinely automating manual processes to remove the possibility of duplicate data, null fields, or human error. In SRE, automation applies across infrastructure management, load balancing, and resource allocation. Additionally, both DRE and SRE build in automated monitoring for errors like pipeline anomalies or website downtime.</p><h3>3. Scalability as a priority</h3><p>Both SRE and DRE aim to ensure reliability, resiliency, and availability as data pipelines and technical infrastructure scales. After all, these frameworks were created due to teams buckling under the pressure as both data and software engineering life cycles grew in volume and complexity. In DRE, scalability means ensuring that data is accurate, up-to-date, and available to the users who need it. In SRE, it ensures the same for websites and web applications.</p><h3>4. Reliability for stakeholders as the end goal</h3><p>Both DRE and SRE have one overarching end goal: deliver a reliable product to all end users. If applied correctly, stakeholders can count on always-available, up-to-date information and reliable architecture.</p><h3>Data Reliability Engineering and Site Reliability Engineering: Differences</h3><p>The field of SRE certainly has a head start on DRE, but that’s not the only difference between these two frameworks. While both share some common goals, there are a few key differences. They are:</p><h3>1. Age and adoption</h3><p>The field of site reliability engineering originated in 2003 at Google. SRE has been widely adopted across the field of engineering. Teams regularly hire Site Reliability Engineers as part of their scaling engineering engine. By contrast, DRE is only a couple of years old, and data teams have only recently started to hire official Data Reliability Engineers as stewards of reliable data at scale. As of now, Data Reliability Engineers tend to be found on very forward-thinking teams that want to adopt the latest practices in data.</p><h3>2. The tools in question</h3><p>SRE focuses on infrastructure and software. The main tools at an SRE team’s disposal are: <a href="https://helm.sh/">Helm</a> (the package manager for Kubernetes), <a href="https://www.datadoghq.com/">Datadog</a> (monitoring and security), and <a href="https://www.pagerduty.com/">PagerDuty</a> (operations and incident management). On the DRE side of things, you’ll find data reliability engineers working with <a href="https://airflow.apache.org/">Airflow</a> (workflow monitoring), <a href="https://www.snowflake.com/en/">Snowflake</a> (powering the data cloud), and <a href="https://www.bigeye.com/">Bigeye</a> (data observability).</p><h3>3. Preparation for the role</h3><p>SREs often come from traditional technical backgrounds in software engineering, systems administration, and technical project management. DREs don’t necessarily come from one specific career path. They often start out in analytics, business intelligence, or data science roles, and might have experience in a variety of fields like product management or business operations. While not necessarily engineers by trade, they do tend to be technically-minded, as they should have a deep understanding of data technologies like Hadoop, Spark, and Kafka.</p><h3>4. The day-to-day</h3><p>The top daily concerns of an SRE probably center around CPU, memory, and API latency. They measure and optimize for system-level metrics like uptime and error rates. They work with load balancers, container orchestration systems, and monitoring and alerting systems. For DREs, the top daily concerns center around data freshness, pipeline volume, and data quality. They measure data-specific metrics around data quality and data availability.</p><h3>Final thoughts</h3><p>Will DRE reach the ubiquity and acceptance of SRE within the next couple of decades? Time will tell. While SRE and DRE teams may not look completely alike, they both work to create a culture of continuous improvement. Change is inevitable. DRE and SRE frameworks help teams build through change, ensuring more favorable (and reliable!) outcomes for the future. Using the principles of DRE / SRE, teams can adopt new technologies, optimize existing systems, and learn from their past failures.</p><p><em>If you’d like to learn more about DRE and the tools that help them work, </em><a href="https://www.bigeye.com/request-demo"><em>get in touch with Bigeye</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b5a8318f3fe" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bigeye/data-reliability-engineering-versus-site-reliability-engineering-4b5a8318f3fe">Data Reliability Engineering versus Site Reliability Engineering</a> was originally published in <a href="https://medium.com/bigeye">Bigeye</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[5 reasons to attend DRE Con 2023]]></title>
            <link>https://medium.com/@kylejameskirwan/5-reasons-to-attend-dre-con-2023-863d154dc1cf?source=rss-e953d7ecf989------2</link>
            <guid isPermaLink="false">https://medium.com/p/863d154dc1cf</guid>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data-management]]></category>
            <category><![CDATA[data-reliability]]></category>
            <category><![CDATA[data-conference]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Kyle Kirwan]]></dc:creator>
            <pubDate>Fri, 03 Mar 2023 22:09:09 GMT</pubDate>
            <atom:updated>2023-03-03T22:12:18.100Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nOb_bWmgf9WFYX8yenW_Aw.png" /></figure><p>On April 19, 2023, we’re gathering the largest collection of data reliability engineers and experts in the world. The third annual <a href="https://drecon.org/">Data Reliability Engineering Conference (DRE Con)</a> is going to be bigger and more spectacular than ever before. Here are 5 reasons you should attend this year’s virtual DRE Con:</p><h3>1. You’ll join a community</h3><p>By attending DRE Con, you’re joining a community of trailblazers, industry experts, and technical practitioners pushing the bounds of a new field. DRE Con is the one place you can access research-backed sessions from SMEs, get expert advice in real-time on your individual challenges, and interact with peers, all under the data reliability umbrella.</p><h3>2. You’ll progress on your data reliability engineering journey</h3><p>How do you become a data reliability expert? How do you avoid some of the pitfalls common to data teams? How can you understand data quality problems from new, interesting angles? Expand your knowledge, not only from our speakers, but also from those in the community who are facing similar challenges and sharing knowledge.</p><h3>3. You’ll be in good company</h3><p>Executives and engineers from the biggest names in tech attend DRE Con. In years past we’ve seen attendees from Google, Netflix, Uber, Meta, Doordash, Microsoft, Shopify, McKinsey, BCG, Stripe, and more. Get in front of hiring managers and partners from the biggest names in tech and data engineering.</p><h3>4. You’ll discover new data reliability tools</h3><p>What technology and tools do leading DREs use in their day-to-day? What does their data stack look like? Throughout the day, you’ll hear about innovative technologies and the specific use cases to which they apply. Analyze and compare how leading engineers gain an edge through technology, so you can walk away with some new ideas about how to add value to your organization.</p><h3>5. You’ll hear from an unparalleled speaking lineup</h3><p>With <a href="https://twitter.com/kelseyhightower">Kelsey Hightower</a> of Google and Kubernetes leading the day, you’ll have access to our expert invite-only speaker lineup. Each speaker is a master of their niche in the industry, and they’ll each have an entire 30 minutes to dive deep into their area of expertise. Many of our speakers are published authors, and all have held leadership roles in some of the world’s biggest, most influential organizations, from Stack Overflow to the FBI.</p><p>Not registered yet? <a href="https://drecon.org/">Secure your free spot at DRE Con today</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=863d154dc1cf" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Netflix maintains data quality]]></title>
            <link>https://medium.com/bigeye/how-netflix-maintains-data-quality-3a77a95f7aad?source=rss-e953d7ecf989------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a77a95f7aad</guid>
            <category><![CDATA[data-observability]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data-quality-management]]></category>
            <category><![CDATA[data-management]]></category>
            <dc:creator><![CDATA[Kyle Kirwan]]></dc:creator>
            <pubDate>Thu, 01 Dec 2022 17:07:55 GMT</pubDate>
            <atom:updated>2022-12-01T17:07:55.349Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/904/1*ryW9nc5YsI1KdRKm7IuQDQ.jpeg" /></figure><p>Netflix has 223 million subscribers in countries all around the world watching over 200 million hours of Netflix each day. Assuming an hour of Netflix HD content is three GB of data, then Netflix delivers more than 6 petabytes of data to customers daily.</p><p>So how do they maintain the quality of this massive core dataset? We talked to Laura Pruitt, Director of Streaming, Platform, and Security Data Science and Engineering. Here’s what she told us.</p><h3>The nuts and bolts of Netflix streaming</h3><p>Netflix has custom-built servers to hold video, audio, and subtitle files. These servers are distributed around the world, wherever customers reside. Netflix wants to localize so that customers don’t have to stream data from far away, no matter where they are.</p><p>The lifecycle of streaming a show on Netflix looks something like this: you find something to watch, and your device sends a request to one of the servers asking for that piece of content. The server sends the first chunk of that video back to you, which your device then decodes and renders in real-time. As your device decodes and renders, it requests more data from the server, which sends it back. This is all done in real-time.</p><p>In this process, Netflix collects a lot of data from both device and server. From the device:</p><ul><li>Who are you as a customer</li><li>What device are you streaming on?</li><li>How quickly did it take for that video to load?</li><li>Did you experience any errors or interruptions during the course of this playback?</li></ul><p>From the server:</p><ul><li>What ISP was the server connected to deliver the content?</li><li>How many bytes did the server transfer?</li><li>How long did it take for those bytes to arrive at their destination?</li></ul><p>These raw logs land in Amazon S3, Netflix’s central data hub. From S3, the data is directed into additional services like Redshift, Kinesis, etc.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Tq_NE3hmR8AiwOGq" /><figcaption>Photo by <a href="https://unsplash.com/@molliesivaram?utm_source=medium&amp;utm_medium=referral">Mollie Sivaram</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>What Pruitt’s team does</h3><p>Pruitt’s team runs ETL pipelines that use business logic and windowing, to process these raw logs into a dataset that is a unified view into both the customer experience and the network experience. This dataset sees several billion new records every day, and is a core dataset at Netflix.</p><p>In putting anomaly detection and data integrity checks on this dataset, Pruitt’s team had the following considerations.</p><h3>Impact</h3><p>This dataset is a very important dataset for Netflix. It is used to answer questions like and make decisions about:</p><ul><li>Which partnerships to invest in</li><li>Which ISPs or devices can bring valuable partnerships to Netflix</li><li>Where to invest internal engineering resources</li><li>Where the service is seeing the most performance issues</li></ul><p>“Any dataset should have a bare minimum of checks in place, but this is one that is being used by many different people and we are making pretty important decisions with it, so it makes sense to make additional investments in making sure the data is of high quality,” Pruitt said.</p><h3>Data Integrity</h3><p>In addition to the devices and the servers, there are several more data sources in this pipeline. Each of these data sources is a place where things can go wrong. Examples of data integrity issues that might pop up include:</p><ul><li>Missing data</li><li>Unexpected datatypes</li><li>Unexpected NULLS</li><li>Malformed records which means you can’t parse out key-value pairs</li></ul><p>Pruitt’s team found that it’s best to detect these sorts of data integrity issues before the ETL process (Netflix, it seems, chooses to monitor their data at the source. See our blog post about whether to monitor at source or destination). They do via a metadata service that gives them high-level metadata metrics on their tables, including:</p><ul><li>Is the partition loaded?</li><li>How many rows are there?</li><li>What’s the min and max value that exists within that column</li><li>What’s the cardinality of that column?</li><li>If a certain amount of data is using thrown away during ETL processing, what is that percentage number?</li></ul><p>Netflix has built reusable frameworks that are shared between data engineering teams and data platform teams to make sure that these basic, generic data quality issues are addressed on source table. For example, every time a service writes out data, the producer can audit it before it’s published to confirm that the main metadata metrics are looking good, before the data is made available for downstream consumption.</p><h3>Business metrics</h3><p>This data pipeline produces dozens of metrics that the company cares about, including things like:</p><ul><li>Error rates</li><li>Customers’ consumption of Netflix</li></ul><p>Additionally, these metrics often have extremely high dimensionality, due to the fact that Netflix operates in hundreds of countries and thousands of ISPs. This makes it challenging to figure out where things are when there are so many permutations.</p><p>For example, consider a business metric like the global playback error rate — the percentage of sessions that end in a fatal error for customers. Let’s say that the spike is actually caused only by Android phones in Brazil — Pruitt’s team needs to identify and annotate this before the CEO comes knocking on the door.</p><p>To deal with the high cardinality, Netflix relies on <strong>anomaly detection</strong>. Netflix pre-aggregates data to grains that they believe are meaningful (devices, countries) and sends that data to an anomaly detection service, which sends back data points they think are anomalous. This pre-aggregation is an effort to reduce the dimensionality of their metrics.</p><p>In terms of alerting, Pruitt’s team started conservatively. It picked the top metrics that it cared about, and only alerted on those to the right people (over email).</p><h3>In summary…</h3><p>Data quality at Netflix directly translates into informed decisions that impact viewing experience and their business bottom line. The company has made a wise decision to invest in it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a77a95f7aad" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bigeye/how-netflix-maintains-data-quality-3a77a95f7aad">How Netflix maintains data quality</a> was originally published in <a href="https://medium.com/bigeye">Bigeye</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Complete guide to understanding data observability]]></title>
            <link>https://medium.com/bigeye/complete-guide-to-understanding-data-observability-6361073e2020?source=rss-e953d7ecf989------2</link>
            <guid isPermaLink="false">https://medium.com/p/6361073e2020</guid>
            <category><![CDATA[data-monitoring]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[data-reliability]]></category>
            <category><![CDATA[data-observability]]></category>
            <dc:creator><![CDATA[Kyle Kirwan]]></dc:creator>
            <pubDate>Mon, 21 Nov 2022 17:38:38 GMT</pubDate>
            <atom:updated>2023-05-18T13:54:16.728Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dil5al1h1yJPNxZaT_LyZw.png" /></figure><p>This is your one-stop shop for all things data observability. Learn what data observability is, who it’s for and who uses it, how it benefits organizations, how it works, key terms, use cases, and best practices.</p><h3>What is data observability?</h3><h3>Definition</h3><p>Data observability refers to the ability of an organization to see and understand the state of their data at all times. By “state” we mean things like: where is it coming from and going within our pipelines, is it moving on time and with the volume we expect, is the quality high enough for our use cases, and is it behaving normally or did it change recently?</p><p>Here are some questions you could answer with data observability:</p><ul><li>Is the customer’s table getting fresh data on time or is it delayed?</li><li>Do we have any duplicated shopping cart transactions and how many?</li><li>Was the huge decrease in average purchase size just a data problem or a real thing?</li><li>Will I be impacting anyone if I delete this table from our data warehouse?</li></ul><p>Observability platforms aim to give a continuous and comprehensive view into the state of data moving through data pipelines, so questions like these can be easily answered.</p><p>Common data observability activities include: monitoring the operational health of the data to ensure it’s fresh and complete, detecting and surfacing anomalies that could indicate data accuracy issues, mapping data lineage to upstream tables to quickly identify the root causes of problems, and mapping lineage downstream to analytics and machine learning applications to understand the impacts of problems.</p><p>Once data teams unlock these activities, they can systematically understand when, where, and why data quality problems occur in their pipelines. They can then prevent those problems from impacting the business, and work to prevent them occurring in the future!</p><p>Data observability unlocks these basic activities, so it’s the first stepping stone toward every organization’s ultimate data wishlist: healthier pipelines, data teams with more free time, more accurate information, and happier customers.</p><h3>Why is data observability important?</h3><p>Organizations push relentlessly to better use their data for strategic decision making, user experience, and efficient operations. All of those use cases assume that the data they run on is reliable.</p><p>The reality is that all data pipelines will experience failures. It’s not a question of if, but when, and how often. What the data team <em>can</em> control is how often issues tend to occur, how big the impact, and how stressed out they are when resolving these failures.</p><p>A data team that lacks this control will lose the trust of their organization, therefore limiting organizational willingness to invest in things like analytics, machine learning, and automation. On the other hand, a data team who consistently delivers reliable data can win the trust of their organization, and fully leverage data to drive the business forward.</p><p>Data observability is important because it is the first step toward having the level of control needed to ensure reliable data pipelines that win the trust of the organization and ultimately unlock more value from the data.</p><h3>What are the benefits of data observability?</h3><p>What do you get once you have total observability over your data pipelines? The bottom line is that the data team can ensure that data reaching the business is fresh, high quality, and reliable — which unlocks trust in the data.</p><p>Let’s break down the tangible benefits of data observability a little further:</p><ol><li><strong>Decreased impacts from data issues</strong> — when problems do occur, they’ll be understood and resolved faster; ideally before they reach a single stakeholder. Data outages will always be a risk, but with observability, their impacts are greatly reduced.</li><li><strong>Less firefighting for the data team</strong> — you’ll spend less time firefighting data outages, and being reactive. That means more time building things, creating automation, and the other fun parts of data engineering and data science.</li><li><strong>Increased trust in the data by stakeholders</strong> — once they stop seeing questionable data in their analytics, and stop hearing about ML model issues, they’ll start trusting the data and assuming it’s good for making decisions with or integrating into their products and services.</li><li><strong>Increased investment in data from the business</strong> — once stakeholders can trust the data, they can feel comfortable using data in more places across the business, which means allowing a bigger budget on data and the data team.</li></ol><h3>Who uses data observability?</h3><p>Data observability can touch many departments within an organization. But if you’re wondering who it’s for and who works with it, let’s explore some context. Following, we’ll explore the relationship between data observability and some common data roles within teams:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4jKjIKUvNllnwSqJJKDSvw.png" /></figure><h3>The history of data observability</h3><p>The concept of “data observability” emerged in the late 2010s. It was initially inspired by internal efforts at companies like Uber, Netflix, Airbnb, and Lyft to improve data quality and monitor data pipelines and tables.</p><p>Most of these data teams developed some sort of pipeline testing system first, before moving on to developing true data observability tools.</p><p>Eventually, smaller companies with lighter technical teams also sounded the alarm for observability capabilities. However, they didn’t have the horsepower to build these solutions in-house. And thus, data observability SaaS solutions were created to fill the gap. Take a look at the data observability origin stories from some companies you’ve heard of:</p><ol><li><a href="https://www.uber.com/blog/monitoring-data-quality-at-scale/">Uber’s data quality monitoring at scale</a></li><li><a href="https://www.databricks.com/session_na20/an-approach-to-data-quality-for-netflix-personalization-systems">Data quality efforts for Netflix personalization</a></li><li><a href="https://medium.com/airbnb-engineering/data-quality-at-airbnb-e582465f3ef7">Airbnb’s data quality initiative</a></li><li><a href="https://www.youtube.com/watch?v=TFcbwOjDpIQ">Lyft’s data quality as a service</a></li></ol><h3>What’s the difference between data observability and data testing?</h3><p>Testing is a manual expression of a condition that we know to be true. For example:</p><ul><li>An engineer expects a data table to be populated with entries. They can test that there is a non-zero number of rows in a data table</li><li>A data scientist creates complex queries to stress test a database and check its responsiveness</li><li>A data reliability engineer uses a sample of input data, runs it through a development version of a data pipeline, and compares the actual output with the expected output</li></ul><p>Testing is generally the first line of defense for data quality issues. Testing is manual and opinionated; something either works or it doesn’t. In comparison with data observability, testing is less all-encompassing. Testing can be a component of a larger data observability program; it feeds into observability. In fact, it’s best practice for both facets to be hard at work in a healthy, high-quality data pipeline.</p><p><a href="https://www.bigeye.com/blog/testing-vs-data-observability">Data Observability vs. Testing</a></p><h3>What’s the difference between data observability and data monitoring?</h3><p>Data monitoring is not opinionated. It simply reports back the conditions of the system on a regular time cadence. A good metaphor is the dashboard of a car. Monitoring tools allow teams to watch and understand the state of their systems, based on predefined sets of metrics.</p><p>In contrast to data monitoring, data observability is more like a vehicle diagnostics test: it scans your data system components to check for issues, reports them, <em>and </em>tells you where they’re coming from and why they’re happening.</p><p>Rather than simply reporting stats, data observability tools provide analytical insights that allow teams to actively address data problems, based on the exploration of properties and patterns that weren’t defined in advance.</p><p>For example, data monitoring might issue an alert if a set of data rows suddenly explodes in quantity, and pre-defined rules explicitly state that there should only be ten rows of data. Data observability, on the other hand, can pinpoint exactly where and why that explosion in volume occurred. Data observability can layer onto data monitoring in order to detect not only <em>that </em>there is an issue, but <em>why </em>and <em>where </em>there is an issue, as well as other details about the issue, like what changes were made, by whom and when, and who is potentially impacted downstream.</p><h3>What’s the difference between data observability and data quality?</h3><p>Data quality is a traditional, all-encompassing term that is focused on fixing data issues in a reactive manner. Data quality refers to the general state of your data: how healthy is it?</p><p>In contrast to data quality, data observability constantly surveys the state of the data pipeline and proactively diagnoses issues. Data observability platforms can help to ensure data quality.</p><h3>What’s the difference between data observability and data reliability?</h3><p>Data observability is a method in which you achieve data reliability. Data reliability is the end state and the goal, and data observability is how you get there. You can deliver reliability through an observability platform.</p><h3>What’s the difference between data observability and machine learning observability?</h3><p>Data engineering, science, and analytics teams use data observability tools like Bigeye to understand the current state of all their <em>data-related systems</em> — data tables, databases, data pipelines — at all times. Data observability deals with metrics like data freshness (how up to date the data is), data completeness (how much missing data there is), and data accuracy (how much of the data is mis-formatted, or contains typos of some sort).</p><p>Machine learning observability involves knowing the current state of your <em>machine learning model</em>’s data and performance across its lifecycle. It deals with issues like model drift, data quality issues, and anomalous performance degradations using baselines.</p><h3>What’s the difference between data observability and observability (the regular kind)?</h3><p>IT operations and site reliability engineering (SRE) teams use infrastructure observability tools like Datadog and SolarWinds to understand the current state of their <em>infrastructure,</em> and the applications that run on them, at all times. For most companies, this means servers, micro-services, and applications. Infrastructure observability typically deals with metrics like uptime, error rate, and number of requests.</p><p>Metrics in infrastructure observability generally remain fairly constant over time. Data observability, on the other hand, can have metrics that fluctuate wildly over time. Data observability as a concept is inspired by infra observability, but differs from it considerably in use case and user.</p><p>For more definitions, check out our <a href="https://www.bigeye.com/blog/data-observability-dictionary">data observability dictionary</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iBDZ5-yjYvQdsfFHnb3Z5Q.png" /></figure><h3>How does data observability work?</h3><p>In general, a good data observability platform should monitor overall warehouse health (freshness, volume, accurate formatting), data quality issues in the tables themselves (anomalies, outliers, potentially erroneous entries), and where problems and improvements will have an impact on the operation (lineage, root cause analysis).</p><p>There are two types of data observability: <em>warehouse metadata monitoring</em>, and<em> deep statistical monitoring</em>.</p><p>General warehouse health visibility is achieved through warehouse metadata monitoring. It’s fairly straightforward and low impact to your data environment, so it’s easy to apply across the entire warehouse.</p><p>Deep statistical monitoring goes deeper. It is achieved by connecting to a read-only account and applying metrics to your data. This approach is very similar to connecting a BI tool. But a good data observability tool won’t actually copy any of your data. It will simply monitor the source and store observability information — limiting operational strain on the database and minimizing security risks.</p><p>Companies generally only apply deep data quality checks on the most critical tables. For most companies, that’s about 20% or less of all the data in their warehouse!</p><p>Common deep monitoring attributes to track include completeness, duplication, format errors, outliers, and distributional statistics. These attributes can’t be collected from metadata. They require direct interrogation of the data itself. This is necessarily more impactful on the warehouse, but also enables the detection of issues inside the datasets themselves.</p><p>Finally, a data observability tool will parse logs and use AI to map out the data model and flow of your environment. This process helps you understand the potential impacts of changes upstream and down.</p><p>NOTE: In both cases, only aggregates need to be returned and stored in the data observability platform. Raw data isn’t needed at this stage, and doesn’t need to leave the data source.</p><h3>Basic data observability concepts</h3><p>If you’re completely new to data observability, there are some foundational concepts that you can digest. The following summaries are meant as bite-sized primers to help you familiarize yourself with the core aspects of a data observability system.</p><h3>1. Freshness</h3><p>Data freshness refers to how up-to-date data is, e.g. the amount of time since a data table was last refreshed. Your data’s freshness is one of the main things that an observability platform will track; ensuring that no data goes stale and out-of-date.</p><h3>2. Schema</h3><p>A database schema is an abstract design that represents the storage of your data in a database. It describes both the organization of data, <em>and</em> the relationships between tables in a given database.</p><h3>3. Metadata</h3><p>Metadata is data about data. In the data observability context, metadata tells you about what’s being done to the data by the infrastructure, e.g. an INSERT to append new rows to a table, or an Airflow job that failed and didn’t write anything to its intended destination.</p><p>Metadata can give you key information about, for example:</p><ol><li>Data freshness — e.g. time since a data table was last refreshed</li><li>Data volume — e.g. number of rows inserted per day</li><li>Read query volume — e.g. number of queries run per day</li></ol><p>Metadata is important for observing the state of your data system. Out-of-date data (for instance in dashboards) is often extremely visible to non-technical leadership, and impedes their decision-making ability. Meanwhile, a massive drop in data volume or data query volume almost certainly indicates that an upstream or downstream dependency is down.</p><h3>4. Lineage</h3><p>A data breakage or outage happens somewhere along the pipeline. The first question is, where? Data lineage helps you find out, by feeding information about which upstream and downstream sources were impacted. It also ties that information back to the teams that are generating and accessing the data. Lineage is a key piece of any data observability program. When a data issue occurs, your lineage serves as a map for how governance, business, and technical teams are impacted.</p><p>To be more specific, data lineage is the path that data takes through your data system, from creation, through any databases and transformation jobs, all the way down to final destinations like analytics dashboards and feature stores. Data lineage is an important tool for data observability because it provides context — it tells you:</p><ul><li>For each data pipeline job, which dataset it’s reading from and the dataset to which it is writing</li><li>Whether problems are localized to just one dataset, or cascade down your pipeline</li></ul><p>This helps to answer questions about where data problems originate, and how widespread their impact is:</p><ul><li>If I change the schema of this table, what other tables will start having problems?</li><li>If I see a problem in this table, how can I identify where that problem originated, and how far it propagates?</li></ul><p>Lineage is most often collected by parsing the logs of queries that write into each table. By inspecting what’s happening inside the query, you can see which tables are being read from, and which tables are being written into.</p><p>But lineage can (and should) go further than just table-to-table or column-to-column relationships within a single source. Companies like AirBnB and Uber have been modeling lineage all the way upstream to the source database or Kafka topic, and all the way downstream to the user level, so they can communicate data problems or changes all the way up to the relevant humans.</p><h3>5. Anomaly detection</h3><p>Detecting data points, events, and/or information that falls outside of a dataset’s normal behavior. Anomaly detection helps companies flag areas that might have issues in their data pipelines.</p><p>The anomaly detection system will need to learn the historical patterns present in each data quality attribute, learn what abnormal behavior looks like, and ultimately fire alerts that indicate real issues while ignoring behavior that’s slightly off but not indicative of a real problem. This can be especially difficult when there are hundreds or thousands of data quality attributes being tracked simultaneously.</p><p>Naive techniques like gaussian models — that simply look at a number of standard deviations above or below the historical mean — fall apart in many commonly occurring time series patterns. A good model will need to adapt to various patterns that can regularly occur in these metadata attributes over time.</p><p>For a <a href="https://www.bigeye.com/blog/anomaly-detection-for-data-observability-part1">deeper dive into anomaly detection, read here</a>.</p><h3>6. SLAs</h3><p>Service Level Agreements are a foundational part of data observability. SLAs were originally designed to clarify and document expectations between a service provider and its users. In a data observability context, they allow for clear communication about what “good” and “not good” data looks like!</p><p>They’re built up in three stages:</p><ul><li>SLIs: Service Level Indicators — what attributes will be tracked and what are acceptable levels</li><li>SLOs: Service Level Objectives — how often will the SLIs be within their expected range</li><li>SLAs: Service Level Agreements — who does what if the SLOs are not met</li></ul><p>For example:</p><ul><li>SLI: Percent of null user_uuids per partition &lt;= 0.5% in last 2 partitions</li><li>SLO: 99% as tracked daily over a trailing 30 day window</li><li>SLA: Data engineering will halt pipeline changes until resolved and SLO back within 99%</li></ul><p>The development of the SLIs, SLOs, and SLA creates a clear framework for data consumers and data teams to align on exactly what “high quality” data means, and what will be done if that definition isn’t being met. It prevents ambiguity and eliminates arguments during high pressure situations when something does happen to go wrong.</p><p>For a <a href="https://www.bigeye.com/blog/defining-data-quality-SLA">deeper dive into SLAs, read here</a>.</p><h3>Architecture and location within the data stack</h3><p>Data observability systems usually consist of the following elements:</p><ul><li>The control plane — the interface where everything is managed from</li><li>The collection system — how the metrics and metadata get collected from data sources</li><li>The anomaly detection system — takes in metrics, builds models, and fires alerts</li><li>The notification connectors — outputs to Slack, PagerDuty, data catalogs, etc.</li></ul><p>The collection system usually operates by running queries on the underlying data sources. This can be done via an agent, or it can be agentless with a direct connection to the data source. In either case a service account will exist on the data source, from which the metric and metadata collection queries can be run.</p><p>Queries are run on the data source, the metrics or metadata are summarized, and sent back to the control plane where they’re stored. This means raw data stays on the data source, and only aggregates are transmitted back to the control plane.</p><p>Once stored in the control plane, the metrics and metadata results are fed to the anomaly detection system so anomaly detection models can be trained on them and alerts can be fired if needed.</p><p>Finally, if an alert is needed, the control plane needs to be connected to the desired alert destination, like a Slack workspace, a PagerDuty workspace, a data catalog like Alation, or an analytics tool like Tableau.</p><h3>Use cases to solve with data observability</h3><p>When should you use data observability? When do teams start to explore data observability platforms, and are there any universal indicators that data observability should deploy? Let’s walk through the various use cases for which we prescribe data observability as the cure.</p><h4>1. Keeping analytics accurate and trustworthy</h4><p>Executives at large organizations regularly complain that “they don’t trust the data”. This is often because the data is internally inconsistent, or because there are discrepancies between the expected numbers and what the dashboard is showing. Ensuring that analytics dashboards and reports display accurate data is probably the top use case for data observability.</p><h4>2. Protecting machine learning performance</h4><p>Machine learning models are “garbage in, garbage out.” In order for recommendation systems, fraud detection systems and computer vision systems, to generate accurate recommendations and predictions, the data inputs must be accurate and up-to-date. Data observability helps ensure these standards.</p><h4>3. Accelerating ETL/ELT development velocity</h4><p>Data model blue-green testing is a framework for deploying changes to data pipeline jobs. The framework expects two schemas in your database, one staging (blue), and one production (green). When you make a change to your job in your data pipeline, you initially run the changed job only in the staging schema. You then compare the outputted data to what is in the production schema. The only differences should be the expected ones from your change.</p><h4>4. Accelerating data warehouse migrations</h4><p>When you invest in a cloud data warehouse, you want to get up and running and start seeing value quickly. Unfortunately, migration complexity and manual validation processes can often kill momentum and force data teams to burn time and resources chasing migration issues instead of adding value to the business. Data observability can reduce the friction and produce faster, smoother migrations by replacing manual checks with automated migration validation and comprehensive reporting — helping ensure every row of data is delivered safe and sound.</p><h3>Common signs that it’s the right time for data observability</h3><h4>You just experienced a high-severity data outage</h4><p>This is the most obvious time to invest in data observability is right after an outage has been resolved! All organizations are busy, and getting buy-in to take preventative measures against a future outage can be difficult. The moments following an outage are the absolute best time to invest in data observability, because all stakeholders are aligned in wanting to prevent future problems from occurring.</p><h4>Your pipelines have gotten complex</h4><p>A lot of your teams are all about data. Where does it come from, where is it going, what is it telling you about how to run your business better, and reach more customers? Data environments are constantly growing, shifting, and evolving. They truly take on a life of their own.</p><p>The “modern data stack” involves more tools now than ever before. That means more pipelines, more tables, and more opportunities for failure or disruption along with them. At all stages of the data pipeline, there are complex transactions occurring. Data is flowing between storage, processing, transformation, modeling, and business intelligence.</p><p>Teams can’t wait to be blindsided by inaccurate, broken, or stale data. One schema change can cause a furious uproar and catastrophic consequences. Change means growth, but it also means unpredictability. Data observability is technology’s answer to that unpredictability; data observability platforms introduce predictability and reliability back into your complex data pipelines. You can’t manually keep data catalogs up-to-date with spreadsheets and the occasional debriefing meeting. You need sharper visibility into your data pipelines and anomalies as soon as they occur.</p><h4>You’ve moved to a hub-and-spoke data team structure</h4><p>Data touches a lot of hands on its journey through your organization. As teams add data scientists, analysts, data reliability engineers, and business analysts, the ownership of data functions might shift or change hands, even within specific teams. And what about multiple teams that partially own some of the same data?</p><p>Data observability can help teams understand how work fits into the larger puzzle of data in your organization. Schema changes, new data sources, and pipeline additions are tracked and communicated with data observability. That way, teams can understand the impact of changes that feel minor, but might cause major ripple effects. Data observability is an effective communication tool; as your data writes a story, data observability serves as the transcript.</p><h3>Building vs. Buying</h3><p>There’s no one-size-fits-all answer to the age-old question, “Do we build it or do we buy it?” You can look at some of the following factors as you consider the best path forward for your data observability.</p><h4>1. Is the observability technology core to your business?</h4><p>Is the technology core to your business, or if a data observability build is going to impact your core product? For example, Uber built their own mapping system. The bespoke “build” approach made sense because maps are a chokepoint to their core business. However, another business for whom a map is a “nice-to-have” would not want to build an in-house mapping infrastructure. They’re a “buy” candidate.</p><h4>2. Does the tool require huge engineering changes?</h4><p>When buying a data tool off-the-shelf, it often comes with a certain framework. For example, a tool might assume all your data is going into a cloud data warehouse.</p><p>If your data isn’t going into a cloud data warehouse, the convenience of the third-party tool is rendered irrelevant. You’ll have to put engineering resources on the case to fully integrate the tool, and that’s not what you want.</p><h4>3. What is the cost and time-to-value?</h4><p>Manual checks take time. We estimate that building a custom solution in-house can cost hundreds of thousands of dollars, and take months to build. Not to mention the strain on your data and engineering teams. Building in-house observability means redirecting engineering time from customer problems to internal work. It’s likely you’d prefer to keep engineers focused on issues that plague paying customers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cHc9yWaIF8Z-KYEXKuyxnQ.png" /></figure><h4>4. Does the vendor have an experienced, communicative, and flexible team?</h4><p>A good third-party vendor can provide some benefits of customization, without your having to build them in-house.A data observability vendor with a track record of supporting scaling, complex companies is ideal. Ask about their support, training, customer success, and integrations. Do they have a roster of enterprise customers?</p><p>What does their product roadmap look like in the near future? You might benefit from feature requests that the vendor gets from their entire customer base, even when it’s not something you would have requested.</p><h3>Signs you should reevaluate your current solution</h3><p>Your data is likely changing frequently: so should your data observability. On a periodic basis (maybe once a year), take stock of your data observability stack and tooling and re-evaluate whether it still suits your organizational needs.</p><p>This might mean conducting an audit of all the tables in your data warehouse, figuring out which of them might be decommissioned or modified, and removing or editing the associated metrics and alerts.</p><h3>Best practices for data observability</h3><p>Whether you have a data observability solution in place or not, there are steps you can take to make your organization more observability-friendly. These best practices can be applied at any point in your data observability journey.</p><h4>1. Start defining the business-critical data</h4><p>Before jumping into data observability, it’s important to first go through your data and understand which data matters most, and which data can be ignored. After all, if you monitored <em>all</em> of your data at all times, you’d quickly reach your maximum capacity to deal with the onslaught of that information.</p><p>Instead, look at what data is being consumed in the most business-critical ways. For example, the “Orders” table consumed by an analytics dashboard that is being regularly reviewed by executives is a high priority.</p><p><strong>Data lineage</strong> products that help visualize the path data takes through your data system, from creation, through any databases and transformation jobs, all the way down to final destinations like analytics dashboards and feature stores, can also be helpful.</p><h4>2. Rollout data observability in a T-shape</h4><p>Once you understand which data matters most, you can apply T-shaped monitoring. Everything in your organization doesn’t need the same level of monitoring. Some areas of your data pipeline need special attention and care, and should be prioritized.</p><p><strong>T-shaped Monitoring</strong> is an approach to data observability that tracks fundamentals across all your data, while applying deeper monitoring on the most critical datasets such as those used for financial planning, machine learning models, and <a href="https://www.bigeye.com/product/dashboard">executive-level dashboards</a>.</p><p>Here’s how it works:</p><ol><li>Track data freshness first</li><li>Then, select some business-critical datasets on which you can apply deep monitoring. Feel free to use a blend of metrics that Bigeye suggests for each table from its library of <a href="https://docs.bigeye.com/docs/available-metrics">70+ pre-built data quality metrics</a>.</li><li>Add custom metrics with <a href="https://docs.bigeye.com/docs/templates">templates</a> and <a href="https://docs.bigeye.com/docs/virtual-tables">virtual tables</a> to ensure custom business logic is being monitored for defects.</li></ol><p>For more information about T-shaped monitoring read on <a href="https://www.bigeye.com/blog/metadata-metrics-instant-data-observability">here</a>.</p><h4>3. Assign ownership over the stewardship the data pipeline</h4><p>Each step in each data pipeline should be assigned to an owner. For example, if data moves from an online service to Kafka to Snowflake, undergoes transformations and lands in tables that are input data for fraud detection algorithm, the online-to-Kafka segment might be “owned” by the Kafka team, while the transformations for the fraud tables might be owned by the fraud team. The owner assigned is responsible for the data pertaining to that segment.</p><h3>Getting started with data observability</h3><p>How do you roll out your data observability solution? Ask your team a few questions first. Namely:</p><ul><li>What can break?</li><li>What is impacted?</li><li>Where do breaks occur?</li><li>Why do breaks occur?</li><li>What are the root causes of breaks?</li></ul><p>Data observability will help you get better answers to these questions. It works in a continuous feedback loop. A successful data observability practice looks like this:</p><ul><li>You understand data quality issues as they occur in production</li><li>You get immediate insight into the impact of each issue</li><li>You can pinpoint the spots along the data pipeline where something broke</li><li>You can take action to fix the issue</li><li>You can learn from the issue so it (and similar issues) don’t reoccur</li></ul><p>With data observability, you’ll be equipped to understand the impact of data issues, so you can take action, fix them, and learn from them so they don’t reoccur.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6361073e2020" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bigeye/complete-guide-to-understanding-data-observability-6361073e2020">Complete guide to understanding data observability</a> was originally published in <a href="https://medium.com/bigeye">Bigeye</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Real “Oh, damn!” moments from data engineers]]></title>
            <link>https://medium.com/bigeye/real-oh-damn-moments-from-data-engineers-d900f1961c14?source=rss-e953d7ecf989------2</link>
            <guid isPermaLink="false">https://medium.com/p/d900f1961c14</guid>
            <category><![CDATA[data-observability]]></category>
            <category><![CDATA[data-engineer]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-quality-management]]></category>
            <category><![CDATA[data-quality]]></category>
            <dc:creator><![CDATA[Kyle Kirwan]]></dc:creator>
            <pubDate>Tue, 11 Oct 2022 14:59:46 GMT</pubDate>
            <atom:updated>2022-10-11T14:59:46.180Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*l_cbyMnDhul5oDJLWCpqUw.png" /></figure><p>The term “data quality” can be vague. It’s abstract to the point where it might seem less urgent. Programming a new feature or putting out a sudden customer success fire? Probably WAY more important than “data quality”, right?</p><p>But that’s not the experience of engineers, data scientists, executives, and product managers who deal with data quality issues.</p><p>That’s why we collected six specific stories from engineers and product managers at companies you’ve heard of. They divulge their horror stories of the one panicky moment when data quality when awry, aka their “Oh, damn!” moments.</p><h3>Shopify</h3><blockquote><em>In 2015 or 2016, Shopify’s data collection system was completely un-schematized. We had a library called Trekkie where you could literally say ‘Trekkie.track()’ and send any data to it. We found that there was a separation between engineering and data where the engineering team thought, ‘I am collecting this data for the data, and data scientists will tell me when things break and whether it’s useful.’ We found the iteration cycle and the data quality cycle here was in the order of months. We sometimes had outages where we discovered something was lost, three months out. It became obvious that this was just not sustainable.”</em></blockquote><p>– <a href="https://www.linkedin.com/in/zeeshanq/">Zeeshan Qureshi</a>, formerly tech lead for the data platform at Shopify</p><h3>Lyft</h3><blockquote><em>We brought on Mode, which is a SQL editor, we brought on Looker, we brought on Tableau, we started incubating Superset — these were all basically BI tools looking at different views of the same data. The challenge was that we started realizing that all these teams were looking at the same metrics in different ways. For example, we never got to a single source of truth on revenue or basic metrics like conversion, because at any given time there were several different versions of it floating around. And where this story ends up going is that you’re at a meeting with a CFO, they’re looking at two different numbers, and they don’t know which number to trust.</em></blockquote><p>– <a href="https://www.linkedin.com/in/georgexing/">George Xing</a>, former analytics lead at Lyft</p><h3>Dessa</h3><blockquote><em>We worked with a </em>Fortune 100<em> bank that had built a really solid fraud detection model — highly accurate, very sophisticated, etc. But over the span of eight months, the model degraded in production. The team was busy dealing with other things, and they didn’t have the tools necessary to actually automatically monitor the model. Ultimately, that one oversight led directly to massive revenue loss for the company.</em></blockquote><p>– <a href="https://www.datacouncil.ai/speakers/mohammed-ridwanul">Mohammed Ridwanul</a>, former product manager at Dessa</p><h3>ZipRecruiter</h3><blockquote><em>It was data that was duplicated in multiple places — when I joined a year and a half ago, we had six different tables in RedShift that were purportedly about applications. Just counting them and getting basic attribution was impossible. You would do queries and you would get weird results back; it was hard just to figure out which database you needed to go to, and very quickly, this became the constraining resource in product development — you just could not get the data you needed to make good decisions.</em></blockquote><blockquote><em>For example, I wanted to ask the question of what percentage of our revenue comes from search traffic visitors. This is something that everyone who wants to make decisions about what products to build or not, should be able to figure out.</em></blockquote><blockquote><em>I opened up Periscope, which had a lot of complicated problems, but was one of our main workhorses, and started putting together a reasonable looking query from reasonable tables. It took me ten minutes to get my first answer, but it was obviously wrong. I pulled in another PM, who had been there longer, and we worked together for an hour, to figure out all the filtering conditions and joins we had to do, we had to reach out to another Slack channel. We finally got to an answer in the end, but when you added all that up, it was probably $1000 in time cost to get that answer, which is completely insane.</em></blockquote><p>– <a href="https://www.linkedin.com/in/paulajgriffin/">Paula Griffin</a>, Director of product at ZipRecruiter</p><h3>LinkedIn</h3><blockquote><em>Back in October 2018, we had an instance at LinkedIn when data quality problems affected the job recommendations platform. Client job views and usage declined by 40 to 60% for a short period of time. Once this decline in views was detected, it took a total of 5 engineers 8 days to identify the root cause and 11 days to resolve the issue.</em></blockquote><p>– <a href="https://www.linkedin.com/in/arunswami/">Arun Swami</a>, Principal Software Engineer, LinkedIn</p><h3>AT&amp;T</h3><blockquote><em>Some years ago I worked in AT&amp;T’s financial services business. At one point I served as an Analyst for the equipment leasing division. (We didn’t call ourselves data scientists back then.)</em></blockquote><blockquote><em>Leadership wanted stats on conversion rates by vertical market. In the leasing business, conversion rate is the ratio of deals done to the deals you approve. Buyers shop around and get multiple approvals before they decide, so you want that rate to be close to 100%.</em></blockquote><blockquote><em>Naturally, we needed the numbers </em>yesterday.</blockquote><blockquote><em>We didn’t have a data warehouse, data mart, or data lake, because the CIO thought such things are foolish luxuries. I pulled data directly from the source system; we’re talking EBCDIC files on an AS/400 LOL.</em></blockquote><blockquote><em>The conversion rates looked pretty good for all but one vertical market, where they were very low. Leadership looked at the numbers, spoke to sales reps, and they were all baffled.</em></blockquote><blockquote><em>The segment manager decided that the problem must be that our pricing was too high, so he quickly implemented price cuts. Meanwhile, I started to drill down into the data.</em></blockquote><blockquote><em>When I pulled individual applications, I discovered the source of the problem. Some genius in IT had run a test of the lease approval system by planting fake applications in the production data. He assigned all of these applications to a single fake merchant, and for reasons known only to him, coded that merchant in the offending segment. The fake applications had no other distinguishing characteristics.</em></blockquote><blockquote><em>The lease approval system processed the applications and dutifully wrote them to the “Approvals” file. There were no real customers to match the fake applications. Thus, the conversion stats computed for that vertical market were wrong.</em></blockquote><p>– <a href="https://www.linkedin.com/in/thomaswdinsmore/">Thomas Dinsmore</a>, head of competitor intelligence, Domino Data Lab</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d900f1961c14" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bigeye/real-oh-damn-moments-from-data-engineers-d900f1961c14">Real “Oh, damn!” moments from data engineers</a> was originally published in <a href="https://medium.com/bigeye">Bigeye</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The data observability dictionary]]></title>
            <link>https://medium.com/bigeye/while-theres-widespread-consensus-that-data-quality-is-very-important-it-s-still-a-relatively-a46da4991426?source=rss-e953d7ecf989------2</link>
            <guid isPermaLink="false">https://medium.com/p/a46da4991426</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-observability]]></category>
            <category><![CDATA[data-reliability]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data-quality-management]]></category>
            <dc:creator><![CDATA[Kyle Kirwan]]></dc:creator>
            <pubDate>Fri, 23 Sep 2022 14:08:59 GMT</pubDate>
            <atom:updated>2022-10-10T14:27:06.286Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W_u0LbAJnxKzB5bSa2bImA.png" /><figcaption>Image credit: Bigeye design team</figcaption></figure><p>While there’s widespread consensus that data quality is very important, it’s still a relatively new field. Even when organizations prioritize their data pipelines, they are likely newcomers to a lot of the terms and context surrounding data quality. That’s why we’ve compiled this dictionary based on common terms we encounter in our day-to-day work. Hopefully it can help you understand some key definitions, differentiations, and components of monitoring, observing, and managing your data.</p><p><strong>Analyst</strong></p><p>An analyst or data analyst examines data to help make business decisions. The meaningful results they pull from the raw data help their company and customers make important decisions by identifying important facts and trends.</p><p><strong>Anomaly detection</strong></p><p>Detecting data points, events, and/or information that falls outside of a dataset’s normal behavior. <a href="https://www.bigeye.com/blog/anomaly-detection-for-data-observability-part1">Anomaly detection</a> helps companies flag areas that might have issues in their data pipelines.</p><p><strong>Business intelligence</strong></p><p>Business intelligence is the practice of using data and analytics to understand what has happened, what is happening now, and why it is happening. It focuses on data from current and past events in order to help you predict the future.</p><p><strong>CI/CD</strong></p><p>Continuous integration (CI): Continuous Integration is a software development practice where members of a team integrate their work frequently, with each integration verified by an automated build (including test) to detect integration errors as quickly as possible.</p><p>Continuous Deployment (CD): Continuous deployment is a software practice where after each code commit, the project is automatically deployed.</p><p><strong>Circuit breaker</strong></p><p>Circuit breaker is a generic term that refers to a mechanism that stops a system when it receives a certain signal. In the data context, it’s been used to refer to stopping data pipelines when certain data quality tests fail. This prevents low-quality data from percolating into downstream processes.</p><p><strong>Dashboard</strong></p><p>A dashboard is an information management tool that visually tracks, analyzes, and displays metrics and key data points to monitor the health of a business, department, or specific process.</p><p><strong>Database</strong></p><p>A database is an organized form of data for easy access, storage, retrieval and management of data.</p><p>Popular databases include MySQL, Postgres (Relational) and MongoDB (NoSQL).</p><p><strong>Data freshness</strong></p><p>Data freshness refers to how up-to-date data is, e.g. the amount of time since a data table was last refreshed.</p><p><strong>Data governance</strong></p><p>Data governance is a collection of processes, roles, policies, standards, and metrics that ensure the quality and security of the data used across a business or organization. Data governance defines who can take what action, upon what data, in what situations, and using what methods.</p><p><strong>Data lake</strong></p><p>A data lake is a central location that holds a large amount of data in its native, raw format. Compared to a hierarchical data warehouse, which stores data in files or folders, a data lake uses a flat architecture and object storage to store the data.‍</p><p>Popular data lakes include Databricks Data Lake and Azure Data Lake.</p><p><strong>Data mart</strong></p><p>A highly structured repository where data is stored and managed until it’s needed. It differs from a data warehouse in scope: data warehouses are capable of serving as the central store of data for an entire business. A data mart fulfills the request of a specific division or business function.</p><p><strong>Data mesh</strong></p><p>Data mesh is a strategic approach for managing your data. Data mesh is a response to traditional centralized data management methods, through data warehouses and data lakes. Instead, data mesh emphasizes decentralization, allocating data ownership to domain-specific groups that can serve, own, and manage data.</p><p><strong>Data migration</strong></p><p>Data migration refers to moving your data from one location or application to another — for example, if a company is moving from Firestore as a database to Postgres. This process generally involves a lot of preparation and post-migration activities including planning, creating backups, quality testing, and validation of results.</p><p><strong>Data monitoring</strong></p><p>Data monitoring refers to periodically querying data sources (i.e. tables in data warehouses) to determine their state (their freshness, volume, quality, etc), and then surfacing this state information in the form of dashboards or alerts.</p><p><strong>Data observability</strong></p><p><a href="https://www.bigeye.com/blog/what-is-data-observability">Data observability</a> is a method for achieving data reliability. It is the ability of an organization to truly <em>see</em> the health of all of their data. Observability platforms give a comprehensive view into the state of data and data pipelines in real-time.</p><p>Data observability specifically encompasses:</p><ul><li><strong>monitoring</strong> the operational health of the data to ensure it’s fresh and complete</li><li><strong>detecting and surfacing</strong> anomalies that could indicate data accuracy issues</li><li><strong>mapping data lineage</strong> from the source all the way through to downstream tables or applications to more quickly identify root cause issues and better understand their impacts</li></ul><p><strong>Data pipeline</strong></p><p>A set of actions to ingest raw data from disparate sources and move the data to a destination for storage and analysis. A pipeline also may include filtering and features that provide resiliency against failure.</p><p><strong>Data quality</strong></p><p>The standard for whether data is fit for use in analytics, whether that is a dashboard, a scheduled export, or an ML model.</p><ul><li>Data quality is not a binary state. Good and poor data quality can vary depending on the intended use for the data.</li><li>Data quality has three phases: operational quality, logical quality, and application quality.</li></ul><p><strong>Data reliability</strong></p><p>Data reliability, a term inspired by <a href="https://sre.google/">Google’s Site Reliability Engineering</a>, refers to the work of creating standards, process, alignment, and tooling to keep data applications — like dashboards and ML models — reliable. This includes data pipeline test automation, manual task automation, data pipeline SLIs/SLOs/SLAs, and data incident management.</p><p><strong>Data Reliability Engineer</strong></p><p>A professional who acts as a steward over the quality of data and reliability of the data process. Data reliability engineers are typically responsible for building data pipelines to bring together information from different source systems. They integrate, consolidate and <a href="https://searchdatamanagement.techtarget.com/definition/data-scrubbing">cleanse data</a> and structure it for use in analytics applications. They aim to make data easily accessible and to optimize their organization’s <a href="https://searchdatamanagement.techtarget.com/definition/big-data">big data</a> ecosystem. Data Engineers contribute, generally, to Data Observability by testing all the known edges and capturing bugs before release of code. By doing so, Data Reliability Engineers can focus on true anomalies.</p><p><strong>Data replication</strong></p><p>Data replication refers to keeping multiple copies of your data and keeping all of them updated.</p><p><strong>Data source</strong></p><p>A data source is where the data being used is located. A data source is commonly referred to as a data source name (DSN), which is defined in the application so that it can find the location of the data.</p><p><strong>Data testing</strong></p><p>Data testing refers to a manual process by which someone who has knowledge of the data, expresses a specific condition about the data, and checks that it’s true. For example, that there are no nulls in a table in a database.</p><p><strong>Data warehouse</strong></p><p>A data warehouse is a system used for storing and reporting on data. The data typically originates in multiple systems (e.g. a production database), then it is moved into the data warehouse for long-term storage and analysis.</p><p>Popular data warehouses include Snowflake, Redshift, and BigQuery.</p><p><strong>DevOps and DataOps</strong></p><p>DevOps is the practice of versioning, reviewing, and automating the release of code. It provides clear and consistent pathways from the engineer’s machine to a production environment. Continuous Integration/Continuous Deployment is a pattern in the DevOps practice. Infrastructure as code is another DevOps pattern. The goal of DevOps is to increase the speed at which code can be developed and released; scale to meet growing teams who are working on the same code; ensure reliability of released code; and ensure the reliability, restartability, resilience, and repeatability of the infrastructure to which that code is deployed.</p><p>As with application products, data products are increasingly being versioned in central repositories and released through CI/CD pipelines — this is the foundation of the DataOps pattern. Distributed and cloud based data platforms also require more complex infrastructure strategies like IAC. DataOps further expands on the DevOps practice by adding other patterns, like Data Observability that are essential to the Data Lifecycle.</p><p><strong>Distribution</strong></p><p>A distribution is a statistical term that refers to a function that shows the possible values for a variable and how often they occur.</p><p><strong>ETL (Extract Transform Load)</strong></p><p>ETL is a type of data integration that refers to the three steps (extract, transform, load) used to blend data from multiple sources. It’s often used to build a data warehouse. ETL pipelines are often set and require an understanding of how the data should look and be transformed before landing in the data warehouse.</p><p><strong>ELT (Extract Load Transform)</strong></p><p>ELT is the process of extracting data from one or multiple sources and loading it into a target data warehouse. Instead of transforming the data before it’s written, ELT takes advantage of the target system to do the data transformation. More and more companies are turning to ELT and investing in tools like Snowflake, Fivetran, DBT, and Airflow. In the world of ELT, traditional data quality processes break down.</p><p><strong>Ingestion</strong></p><p>Data ingestion is the process of absorbing data from a multitude of sources, and then transferring it to a target site where it can be analyzed and deposited. Data ingestion is a broader term than ETL or ELT, which focuses on data warehouses.</p><p><strong>Instrumentation</strong></p><p>Instrumentation refers to the process of collecting summary data by which you’ll observe your data systems. The four core blocks are metadata/logs, metrics, lineage, and deltas. In our case Bigeye is the tool the customer would use to collect data.</p><p><strong>Lineage</strong></p><p>Data lineage is the path that data takes through your data system, from creation, through any databases and transformation jobs, all the way down to final destinations like analytics dashboards and feature stores. Data lineage is an important tool for data observability because it provides context — it tells you:</p><ul><li>For each data pipeline job, which dataset it’s reading from and which dataset it’s writing to.</li><li>Whether problems are localized to just one dataset, or cascade down your pipeline.</li></ul><p><strong>Logs</strong></p><p>Logs are records of what happened when, that are generated by data systems. Logs can be unstructured or structured.</p><p><strong>Machine Learning Engineer</strong></p><p>A machine learning engineer/ML engineer is a role focused on researching, building and designing artificial intelligence (AI) systems to feed and automate predictive models.</p><p><strong>Metrics</strong></p><p>In data observability, metrics are numeric results that come from directly querying a dataset. In the data observability context, they often take the form of a time series.</p><p><strong>Schema</strong></p><p>A database schema is an abstract design that represents the storage of your data in a database. It describes both the organization of data and the relationships between tables in a given database.</p><p><strong>SLA (Service Level Agreement)</strong></p><p>SLA is a feature in the Bigeye platform that enables data teams to create trust with stakeholders and improve data culture. SLA combines multiple data quality metrics into a single group that tracks the health of a key asset like a query, dashboard, or machine learning model to establish transparency and ensure clear communication.</p><ul><li>SLA is based on tried-and-true practices from SRE and DevOps. <a href="https://medium.com/bigeye/defining-data-quality-with-slas-ea79e916ebc7">Read here to learn more about SLAs for data observability.</a></li><li>The data observability metrics collected in SLA are made up of SLIs and SLOs that are put together into an overall agreement on what constitutes data quality for a specific piece of analytics, whether that’s a dashboard or query output.</li><li>For example, an SLA with a 99.9% uptime guarantee (or a 0.1% error budget) allows for <a href="https://www.atlassian.com/incident-management/kpis/error-budget">43 minutes and 50 seconds</a> of downtime each month. By viewing the SLA in Bigeye, the end user has a concrete way to know whether the data meets their expectations, and what level of reliability they should expect week to week or month to month.</li></ul><p><strong>SLI (Service Level Indicator)</strong></p><p>SLIs are an important part of an SLA. SLIs measure specific aspects of performance. SLIs might be “hours since dataset refreshed” or “percentage of values that match a UUID regex.” When building an SLA (see definition) for a specific use case, the SLIs should be chosen based on what data the use case relies on. If an ML model can tolerate some null IDs but not too many, the rate of null IDs is a great SLI to include in the SLA for that model.</p><p><strong>SLO (Service Level Objective)</strong></p><p>SLOs give each SLI a target range. For example, the relevant SLOs could be “less than 6 hours since the dataset was refreshed” or “at least 99.9% of values match a UUID regex.” As long as an SLI is within the range set by its SLO, it’s considered acceptable for the target use case, and that aspect of its parent SLA is being met.</p><p><strong>Streaming data</strong></p><p>Streaming data is the continuous flow of data generated by various sources. By using stream processing technology, data streams can be processed, stored, analyzed, and acted upon as it’s generated in real-time.</p><p>Popular streaming data platforms include Kafka, Confluent, Google Cloud Pub/Sub, and AWS Kinesis.</p><p><strong>Structured, semi-structured, and unstructured data</strong></p><p>Structured data is data that most often follows a tabular convention and is most often stored in conventional databases. Parquet is an example of structured data that is file based and also supports hierarchical data. Structured data enforces schema. Schema changes for structured data require a change to the table in which the data is stored. In the case of Parquet, schema is stored in the metadata of the file and incompatible schema amongst files is handled by the consumer and the read layer. For structured data, the key goals for Data Reliability Engineers, when implementing their data observability strategy, is to monitor values to ensure that those values conform to business rules.</p><p>Semi-structured data does not follow a tabular convention and, while able to be stored in conventional databases, is usually stored in NoSQL databases or in object stores or file systems. Examples of semi-structured data include: CSV, JSON, YAML, AVRO, Semi-structured data conforms, generally, to schema in the sense that the application publishing the data is bound to an object definition. Changes to the schema are code changes and are generally versioned. Different applications and different versions of applications may be writing the same semi-structured data to the same object store. In addition to monitoring values, Data Reliability Engineers should pay strict attention to schema drift and have a schema registry and schema evolution standard in place when implementing their data observability strategy.</p><p>Unstructured data is a loose concept and is generally applied to data that is not readable by a computer. Examples include: word documents, emails, picture files, log dumps, etc. No schema exists for unstructured data and, therefore, unstructured data cannot be observed for anomaly in traditional ways. Often, Data Reliability Engineers target commonality of use case to determine if the unstructured data conforms. Sometimes, machine learning is used to determine fit for use. A good example is a filter of image data based on an image containing certain attributes — like an airplane.</p><p><strong>Toil</strong></p><p>Toil in a data/software context refers to the kind of work tied to running a production data service that tends to be manual, repetitive, automatable and that scales linearly as a service grows.</p><p><strong>Tracing</strong></p><p>Distributed tracing is a method of observing requests as they propagate through distributed cloud environments. Distributed tracing follows an interaction by tagging it with a unique identifier. This identifier stays with the transaction as it interacts with microservices, containers, and infrastructure.</p><p><strong>Volume</strong></p><p>Data volume is the amount of data that you have, or the amount that’s being created within a certain timeframe, e.g. the number of rows being inserted into a table per day.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a46da4991426" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bigeye/while-theres-widespread-consensus-that-data-quality-is-very-important-it-s-still-a-relatively-a46da4991426">The data observability dictionary</a> was originally published in <a href="https://medium.com/bigeye">Bigeye</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to evaluate data observability platforms (RFI)]]></title>
            <link>https://medium.com/bigeye/how-to-evaluate-data-observability-platforms-rfi-26f1704c5528?source=rss-e953d7ecf989------2</link>
            <guid isPermaLink="false">https://medium.com/p/26f1704c5528</guid>
            <category><![CDATA[data-observability]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-quality-management]]></category>
            <category><![CDATA[data-reliability]]></category>
            <dc:creator><![CDATA[Kyle Kirwan]]></dc:creator>
            <pubDate>Tue, 06 Sep 2022 16:25:54 GMT</pubDate>
            <atom:updated>2022-09-06T16:25:54.242Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3liYBtcyMYwmgLkKeTPqPQ.png" /><figcaption>Which platform will you choose? Photo credit: Bigeye design team</figcaption></figure><p>So you’re considering a data observability platform, but want to make sure you’re choosing the best one for your business. We’ve got some tips to help you run a thorough evaluation process.</p><p>We wrote this article after running hundreds of POCs (proofs of concept) — some great, and some painful — to help you run a great evaluation process to decide which data observability solution is right for your team. Read on to learn how to gather the right resources, define your success criteria, decide on a vendor shortlist, and choose the right data observability solution.</p><p><strong>☝️You can also download this handy </strong><a href="https://www.bigeye.com/data-observability-rfi/"><strong>Data Observability RFI (request for information)</strong></a><strong> to help you get up and running quickly.</strong></p><h3>When to invest in data observability</h3><p>The desire to be data driven is nothing new. In the current sense of the word, organizations have been striving for “data-drivenness” for almost 20 years now. And while new data tools and best practices have certainly made us more data driven then we were when Hadoop came hot-stepping onto the scene in ’06, many organizations still find themselves sitting on a big stack of data they either can’t use, or aren’t sure how to. If you asked them “why?” they might say:</p><ul><li>“I don’t know what this data is or how I’m supposed to use it.”</li><li>“We have way too much data and I don’t even know where to start.”</li><li>“I’m not confident the data is reliable.”</li></ul><p>Many innovative companies have popped up in the last few years to try and tackle these challenges but, “I’m not confident the data is reliable” continues to plague even the coolest, most sophisticated DataOps teams. There’s just too much data, too many potential points of failure, and too few skilled humans to test all of it. Even if you could, the tests would still only look for things you anticipated, leaving you open to the, “I’m trying to pull data for our APJ investor meeting and the dashboard is totally #@*!ed!!” type issues you didn’t.</p><h3>Is data observability right for you?</h3><p>Enter data observability, a new category of tools designed to solve the data reliability problem. If you’ve recently asked yourself:</p><ul><li>“How do I find data issues before my executives or customers do?”</li><li>“How can I make sure my replication job completes on time and all the data shows up looking like I meant it to?”</li><li>“How many more failed data merge attempts can I debug before I do something I’ll regret?”</li></ul><p>Then you should be evaluating data observability solutions. But where do you start?</p><h3>How to evaluate data observability solutions</h3><h4>Assemble your team</h4><p>If you’re like most organizations we’ve worked with, you may have lots of different data observability requirements floating around that may not be well documented or defined. As a first step, we recommend creating an evaluation team that can help gather requirements and synthesize them before you begin looking at vendors. Choosing the right mix for your data observability evaluation team is critical to the success of your project.</p><p>Cover the critical roles in the evaluation process:</p><ul><li><strong>Practitioners: </strong>Ensure you get the right balance of representation from across the business by engaging with one or two representatives from each group who might interact with your chosen solution. This could include data engineers, data scientists, ML engineers, analysts, and so on.</li><li><strong>Evaluation leader</strong>: Appointing a project leader who’s close to both the technical work and the business requirements — such as a director of data engineering or a seasoned engineer — will give your initiative the ability to balance requirements and ensure that whatever decision you come to will be correct, technically and strategically.</li><li><strong>Executive sponsor: </strong>Get an executive sponsor such as a CTO or CDO to participate as well. Having this individual’s buy in will go a long way in ensuring your search is aligned with your organization’s strategic goals and you can actually get budget for a solution when the time comes.</li><li><strong>Purchasing: </strong>Once the evaluation is over, you’ll want to move quickly to implement the solution you’ve selected. Procurement and purchasing can take much longer than many people think. Identifying early who will lead any legal reviews, procurement steps, etc. can save a lot of time later on.</li></ul><p>Also consider a cross section of teams who will use the data observability platform:</p><ul><li><strong>Data engineering / data platform:</strong> Data engineering or data platform team engineers will almost certainly be core users of your data observability solution, and should be core decision makers in the selection process.</li><li><strong>Analytics / data science:</strong> Data science and analytics teams are often subject matter experts and heavy users of data observability, once the data platform or data engineering teams have set up basic monitoring for pipeline freshness and volume.</li><li><strong>Machine learning:</strong> Machine learning teams often adopt data observability to protect the pipelines feeding the training and serving data used to drive models that reach production. They’ll want to be familiar with the solution you choose.</li></ul><h4>Example team list</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nuirqEsn57qxzdq6S_tjwg.png" /><figcaption>Photo credit: Bigeye design team</figcaption></figure><h4>Gather your requirements</h4><p>Now that you’ve got your team together and — hopefully — bought them all lunch, it’s time to create a laundry list of wants and needs for your business. Try to tie each list item to a specific business challenge or initiative, this will make it easier to create a business case later. You may also want to give your requirements initial scores or weights to simplify prioritization.</p><p>Here are a list of data observability categories we recommend to help guide you:</p><p><strong>☝️Tip: This handy </strong><a href="https://www.bigeye.com/data-observability-rfi/"><strong>Data Observability RFI (request for information)</strong></a><strong> contains the below criteria and more!</strong></p><p><strong>Deployment:</strong> Consider your deployment needs. Are you a nimble startup in need of a fully-managed Saas solution? A large healthcare provider looking for the flexibility of a single-tenant hosted option? Or a defense contractor with strict air-gap requirements? Also consider how the solution will connect to your existing stack and what performance impacts it may have on your operation.</p><p><strong>Security: </strong>Be sure to engage your IT security team early to understand your organization’s software security requirements. Look for criteria like SOC2 certification, compliance with relevant industry regulations, and if the vendor exports or stores any of your data in their platform.</p><p><strong>Monitoring: </strong>Monitoring capabilities are the foundation of data observability. Use the requirements you outlined to determine which types of monitoring you’ll need and how much effort will be required to deploy and maintain them. Which monitors are available out of the box and which will need to be custom? Can they be deployed automatically or do they require manual setup? Do they monitor both table level and field level? Can you easily add your own monitoring for custom business logic?</p><p><strong>Anomaly detection: </strong>If monitoring is the foundation of data observability, anomaly detection is the crown jewel. Mixed metaphor aside, anomaly detection should help you identify issues you didn’t know existed, a capability very difficult to build yourself. Look at what types of anomaly detection are available and the sophistication of the ML models used. How long does the model need to train before it can start detecting anomalies? Does it adapt to user feedback? Can you easily adjust the sensitivity levels? Is anomaly detection available on custom monitoring? Are simpler methods of detection such as standard deviation available?</p><p><strong>Alerts and notifications: </strong>If APM tools have taught us anything, it’s that proper alerting will make or break a tool. If you don’t get alerted to critical issues then you’ve got a fire drill, if you get constantly overloaded with false positives or non-issues then you’ve got a grumpy ops team and some nice data observability shelfware. Your solution should provide an optimized alerting system and allow you to easily fine-tune notifications to meet your needs.</p><p><strong>Resolution: </strong>knowing about a problem is great, knowing how to resolve it is even better. You should evaluate how the solution allows you to track and manage issues, if it provides root cause analysis and debugging, and if you can show you how issues are impacting downstream tables and systems to help prioritize resolution and manage user expectations.</p><p><strong>Integrations: </strong>Aside from a direct connection to your data warehouse or database, evaluate the current tools in your data stack and consider how a data observability solution will interact with — and add value to — each. These may include analytics tools like Tableau, data catalogs like Alation, and orchestration and transformation tools like Fivetran, dbt and Airflow.</p><p><strong>Reporting: </strong>Another key benefit of data observability is gaining insights into how your data pipelines are performing over time. Ensure the solution you choose can provide insights such as how much of your pipelines are currently in or out of coverage, how many issues have been detected / acknowledged / resolved, and if issues are trending over time. You should also be able to export metadata to a third-party database / warehouse for additional analysis.</p><p><strong>Configuration and management:</strong> While simplicity of configuration and advanced features like REST API access, a CLI, or SDK may not be top of mind in your initial search, they will be critical to your success as data observability usage and requirements grow. Ensuring your solution can provide simple, scalable, Terraform-like “as code” options will pay dividends when you want to deploy monitoring on hundreds of tables or integrate blue-green testing into a version-controlled pipeline.</p><h3>Selecting vendors to evaluate</h3><p>What you’re looking for in a solution will ultimately be driven by the business requirements you defined in the previous step. If your team is primarily made up of data engineers who need to know that data in the warehouse is fresh and complete, then you’ll want to ensure your chosen solution has the ability to query the metadata and expose broad operational health metrics across the data warehouse with little to no setup of management on your part. If you have data science and analytics teams who need to ensure data adheres to business-logic quality rules before it’s fed into an analytics dashboard or ML application, then you’ll want a solution that provides deep business-logic driven metrics such as variance or outlier detection.</p><p><strong>Want to run a great POC?</strong></p><p>Skip the work and download our free Data Observability RFI (Request for Information). It contains a comprehensive checklist of features, and makes it easy to add additional criteria to score vendors on, so your team can make the best decision.</p><p>✔ Pre-built data observability features list</p><p>✔ Grouped capabilities to help you focus on what matters most to you</p><p>✔ Compare multiple vendors during your evaluation</p><p><a href="https://www.bigeye.com/data-observability-rfi/"><strong>Get the guide now.</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=26f1704c5528" width="1" height="1" alt=""><hr><p><a href="https://medium.com/bigeye/how-to-evaluate-data-observability-platforms-rfi-26f1704c5528">How to evaluate data observability platforms (RFI)</a> was originally published in <a href="https://medium.com/bigeye">Bigeye</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>