<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by shane murray on Medium]]></title>
        <description><![CDATA[Stories by shane murray on Medium]]></description>
        <link>https://medium.com/@shane.murray5?source=rss-8aa0d9ae3ebd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*5AiC6qyTiE6MC4vWLxfRYQ.jpeg</url>
            <title>Stories by shane murray on Medium</title>
            <link>https://medium.com/@shane.murray5?source=rss-8aa0d9ae3ebd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 08:18:58 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shane.murray5/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Top 5 AI Reliability Pitfalls]]></title>
            <link>https://medium.com/data-science-collective/the-top-5-ai-reliability-pitfalls-a0a017d29154?source=rss-8aa0d9ae3ebd------2</link>
            <guid isPermaLink="false">https://medium.com/p/a0a017d29154</guid>
            <category><![CDATA[data-reliability]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[shane murray]]></dc:creator>
            <pubDate>Sun, 27 Apr 2025 00:03:10 GMT</pubDate>
            <atom:updated>2025-04-27T00:03:10.394Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ie1Z-6hf6_btsPoW" /><figcaption>Image credit: <a href="http://montecarlodata.com/">Monte Carlo</a></figcaption></figure><p>Hallucination — when an AI confidently generates false or nonsensical outputs — is <strong>the most</strong> notorious failure mode for AI applications.</p><p><em>But is it really the one you need to worry about?</em></p><p>A lot of noise has been made about hallucinations in recent months. As more companies introduce AI into production, the issue of hallucinations is perennially in the news. Even AI code-editing darling <a href="https://www.cursor.com/en">Cursor AI</a> found itself with a spate of <a href="https://fortune.com/article/customer-support-ai-cursor-went-rogue/">public hallucinations</a> from its Customer Support chatbot — reigniting conversations about the necessity for humans in the loop.</p><p>However, I would argue that much of what masquerades as hallucinations in production today is actually rooted in one of several distinct issues.</p><p>I’ve talked with hundreds of data and AI leaders about their companies’ journeys to production-ready AI. While the means and even the destination (chatbots, agents, copilots, etc) might have varied from team to team, there was one thing that remained absolutely consistent — the problems.</p><p>Based on my research, here are the five reliability pitfalls that keep data + AI teams up at night.</p><h3>1. Poor Source Data Quality</h3><p><em>Garbage in… garbage out.</em></p><p>Teams often monitor the model, assuming that hallucinations are derived from an issue within the LLM or the embeddings. But the reality is, if the foundation is shaky to start, garbage output is inevitable — whether the model is functioning properly or not.</p><p>The success of any AI application is fundamentally dependent on the quality of the data that’s feeding it. If your input data — knowledge base articles, internal docs, transcripts and oftentimes even structured data — is outdated, inconsistent, incomplete, or poorly structured, the model’s output will reflect that.</p><p>An example of this would be an AI support bot that hallucinates product details because the knowledge base hasn’t been updated in six months.</p><p>Step 1, 2, and 3 of being “AI-ready” is validating the basic quality of your data. If you can’t trust your source data, you can’t trust anything the model generates.</p><h3>2. Drift in the Embedding Space Over Time</h3><p><em>When “relevant” stops being relevant.</em></p><p>RAG pipelines rely on embeddings to find relevant context — but those embeddings can drift if the source data changes, chunking technique is tweaked or the embedding model updates.</p><p>If drift occurs, those subtle changes can cause silent performance degradation that’s sometimes misdiagnosed as a model issue.</p><h3>3. Confused Context</h3><p><em>A well-tuned model is only as good as the context it’s given.</em></p><p>Even if your embeddings are fresh and your vector database is humming along, retrieval can still go wrong and confuse your model. And a confused model is an unreliable one. Here’s how that can look in a couple different scenarios.</p><p><strong>Ambiguous context</strong></p><p>Almost correct is still wrong. Ambiguous terms, acronyms, or product names can all confuse a retriever. If the retriever pulls the wrong data, it’s going to generate the wrong response. The only question is how wrong it’s going to be.</p><p><strong>Partial or incomplete context</strong></p><p>If I say “Hey! There’s a lion in that cage,” there’s a good chance you’d be excited about it (depending on how you feel about lions). But if I said “Hey! There’s a lion in that cage, and someone left the door open,” your response is likely to change.</p><p>Poor chunking that strips documents of meaning or cuts off key details can mean the difference between a response that’s accurate and one that ends up in the news.</p><p>Each of these failures is bad in its own right. But none of these issues are hallucinations — <strong>they’re context failures masquerading as model issues.</strong></p><h3>4. Output Sensitivity to Code and Prompt Changes</h3><p><em>Small tweaks, big consequences.</em></p><p>Whether it’s swapping an LLM model, adjusting a prompt, or tweaking a temperature setting, tiny changes often lead to large, unexpected output differences.</p><p>For example, I’ve spoken to a number of teams who saw regressions in the accuracy of their model simply by upgrading to the latest LLM version.</p><p>What’s worse, if your output is part of a downstream workflow, these issues can quickly cascade into financial consequences with delayed development cycles that force teams to re-test applications with alpha and beta users before delivering final products to GA.</p><h3>5. Too Many Humans in the Loop — Or Too Few</h3><p><em>Is it truly AI if a human is supervising?</em></p><p>And finally, one of the biggest problems I’ve seen when AI apps go live is teams relying SOLELY on the ability of human evaluators to catch errant responses.</p><p>There are a lot of things you can get away with in pilots that you can’t in production and fully-manual evaluations is one of them. It’s fine with seven customers — its unsustainable with 1,000.</p><p>But just because it can’t be all human, doesn’t mean you need <em>no humans</em>. While some teams use LLMs to evaluate AI outputs (“LLM-as-a-judge”), this approach isn’t foolproof either — AI judges can miss subtle errors or reinforce model biases.</p><p>Finding the right mix of automation, human evaluation, and task-specific evaluation is critical for teams to be able to effectively evaluate application performance.</p><h3>Agents are complex…and so are the problems</h3><p>Agents are complex — but if we want them to be successful, we need to be able to pay attention to every layer of that complexity.</p><p>Like any data product, it’s painfully easy to see when an AI goes bad — its a lot more difficult to determine why or how. Chunking, embedding, retrieval, prompts, models, post-processing.</p><p>Without lineage and tracing across the data + AI stack, “hallucination” can easily become the catch-all response — and misdiagnosed issues have real world consequences.</p><p>After hundreds of hours of conversations, I’m convinced that end-to-end visibility into the data, system, code, and model of an AI application — and agentic resources to resolve issues quickly — is the only way to scalably and reliably develop AI in production.</p><h3>Conclusion: Build for Trust, Not Just Demos</h3><p>If there’s one thing I’ve learned researching AI in production, it’s that what you see isn’t always what you get. In the lab, your AI might look brilliant. But trust breaks down quickly when that system isn’t reliable in the wild.</p><p>If we want to deliver production-capable AI, we need to be looking out for more than the hallucination boogeyman.</p><p>You might be lying on the ground, but it matters whether you tripped over a rock or a bear trap. Building for AI trust means monitoring more than just the model — it means watching the data, the pipeline, the retrieval, and the evaluation loop too.</p><p>Because at the end of the day, your users will never care how cool your model is if they can’t also see how reliable it is.</p><p><a href="https://medium.com/@shane.murray5">Follow me</a> on Medium for more stories on data leadership, applications of data science, and related topics. <a href="https://medium.com/subscribe/@shane.murray5">Subscribe</a> to get my stories delivered to your inbox.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a0a017d29154" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/the-top-5-ai-reliability-pitfalls-a0a017d29154">The Top 5 AI Reliability Pitfalls</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Trust and Quality: The Imperative for AI Systems]]></title>
            <link>https://medium.com/data-science-collective/trust-and-quality-the-imperative-for-ai-systems-d0e8462f7810?source=rss-8aa0d9ae3ebd------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0e8462f7810</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[data-observability]]></category>
            <dc:creator><![CDATA[shane murray]]></dc:creator>
            <pubDate>Mon, 07 Apr 2025 21:34:09 GMT</pubDate>
            <atom:updated>2025-04-07T21:34:09.181Z</atom:updated>
            <content:encoded><![CDATA[<p>Over the past couple of months I’ve spoken to dozens of data teams who are actively building and deploying AI applications. While some of these applications can thrive without perfect accuracy, others demand high reliability as scale, visibility and business impact increase. This post explores the patterns that drive when and why trust becomes an imperative.</p><h3><strong>The Trust Threshold</strong></h3><p>There has long been a concept in reliability engineering that you should design for “as much reliability as you need and no more.” But for any production-ready application of value to the business, the answer is to “how much” is never “nothing at all”. Conversely, if your application requires zero investment in reliability, I could make an argument that it’s worthless.</p><p>So I’ll define the “trust threshold” here as the point in your application’s lifetime where the cost of poor quality exceeds acceptable limits. This threshold is crossed due to four main factors:</p><ul><li>Scale: As the system touches more users or business processes, small inaccuracies multiply into big problems.</li><li>Visibility: Outputs for external customers typically demand the highest accuracy and reliability, followed by internal stakeholders, small groups of experts, then prototypes for personal use.</li><li>Regulatory/Compliance Risk: Certain use cases are inherently high-stakes due to legal or ethical obligations.</li><li>Business Value: As a model’s output drives critical decisions or revenue, the cost of errors increases exponentially.</li></ul><p>Typically, I am seeing teams cross this threshold after a successful proof-of-concept, where they’ve proven they can build something of value to the business, and are now scaling it to enough users where the absolute quality or dips in reliability of the application become noticeable.</p><p>And across the spectrum use cases, when exploring the question of how AI teams are ensuring trust and quality, the most prevalent response is human-in-the-loop evaluation. In some instances this is by design, a feature that allows for the right balance of automation and human judgement.</p><p>But in other cases, especially when a certain scale or visibility is achieved, human-in-the-loop becomes necessary but insufficient to ensure the ongoing reliability and trustworthiness of the application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*X8y-zxqs7NOv7JT0" /><figcaption>Image courtesy of the author.</figcaption></figure><h3><strong>Patterns in Practice</strong></h3><p>As many data leaders are navigating what it means to have “AI-ready data”, the reality is that it depends on the needs of the applications you intend to build.</p><p>We’ve seen public examples of <em>customer chatbots</em> gone awry, making it clear that any data supporting these applications must be secure, accurate and trustworthy as soon as you begin exposing the application to customers.</p><p>I’m seeing leading data teams across a variety of industries take <em>business process automation</em> to the next level — with faster, more efficient processes aimed at image or document processing, as well as regulatory testing; work that now requires human oversight rather than human labor. In these cases the business value is already apparent, so the expectations on reliability are high — downtime can mean $ or non-compliance.</p><p>The next class of use cases are often launched with more of a “test &amp; learn” approach — <em>product personalization, internal chatbots, decision support tools</em>. In many cases, data leaders have told me these are net new capabilities, so the internal expectations are initially low.</p><p>Media &amp; technology companies I spoke to often raised the importance of brand safety in content personalization features, with some talking about features that did not meet user expectations, the most critical trust threshold for them. Ultimately, these creative personalization features are judged based on user response (e.g. thumbs up or down) and adoption. And once they are adopted, they’d better be reliable.</p><p>Internal chatbots or decision support applications may initially skate by as “better than what we had before” but more often than not I’m hearing that failure is correlated with poor quality knowledge bases and metadata. Successful implementations are held to a high standard by their owners and consumers, with a need to ensure those knowledge bases are up-to-date and complete.</p><p>At the lower end of the spectrum on trust expectations are the more opportunistic use cases — creative data practitioners who have generated entirely new datasets that are used for business insight, or identified a place where they could rapidly summarize customer call transcripts to deliver themes or segment their audience. These are typically offline exercises (for now) and their internal customers are invariably stoked for the unprompted support!</p><p>In time, I’m sure they’ll hit a threshold where someone asks them if the latest round of insights can be trusted.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iv6E4dEywivRudJh" /><figcaption>Image courtesy of the author.</figcaption></figure><h3><strong>Recognizing the Trust Threshold</strong></h3><p>To identify whether your AI initiative is approaching the trust threshold, ask yourself, your team and your stakeholders the following:</p><p>Ask the Right Questions:</p><ul><li>What are the consequences of poor-quality outputs?</li><li>Who is the audience, and how sensitive are they to errors?</li><li>How does scale change the risk profile?</li><li>Are there regulatory or ethical implications to consider?</li></ul><p>Early Warning Signs:</p><ul><li>Increased error feedback from key stakeholders.</li><li>Slow adoption despite technical accuracy.</li><li>Hesitation to automate due to trust concerns.</li></ul><h3><strong>How Data Teams Can Stay Ahead</strong></h3><p>Crossing the trust threshold isn’t a matter of it, but when. Leading teams are taking proactive steps to manage this transition effectively.</p><ul><li><strong>Proof-of-concepts &amp; experiments:</strong> The fastest way to AI-ready data is to make it a necessity. Running small-scale experiments helps you identify and resolve quality gaps early, making production readiness a natural next step.</li><li><strong>Prioritize human-in-the-loop systems:</strong> Involving humans at key decision points not only mitigates risk but also aligns with what users expect — a balance of automation and human judgment. Human review becomes both a quality gate and a source of continuous feedback to improve the system.</li><li><strong>Build feedback loops:</strong> Implement systems that capture and respond to user input over time. One of the simplest and most effective methods I’ve seen is a thumbs-up/down mechanism for end users to rate the output’s quality. This data provides valuable insights to fine-tune models and build trust progressively.</li><li><strong>Evaluate the required level of precision:</strong> Not every application requires the same degree of accuracy. Align model performance with business needs — whether it’s 99.9% accuracy for highly regulated use cases or “good enough” for internal brainstorming tools.</li><li><strong>Implement observability:</strong> You can’t improve what you can’t see. Track the data, system, code, and model to identify where failures occur and monitor the system’s ongoing reliability. Observability is foundational for maintaining trust as AI applications evolve.</li></ul><p><a href="https://medium.com/@shane.murray5">Follow me</a> on Medium for more stories on data leadership, applications of data science, and related topics. <a href="https://medium.com/subscribe/@shane.murray5">Subscribe</a> to get my stories delivered to your inbox.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0e8462f7810" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/trust-and-quality-the-imperative-for-ai-systems-d0e8462f7810">Trust and Quality: The Imperative for AI Systems</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Organizing Generative AI: 5 Lessons Learned From Data Science Teams]]></title>
            <link>https://medium.com/data-science/organizing-generative-ai-5-lessons-learned-from-data-science-teams-2f271874ae4a?source=rss-8aa0d9ae3ebd------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f271874ae4a</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[notes-from-industry]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[data-platforms]]></category>
            <dc:creator><![CDATA[shane murray]]></dc:creator>
            <pubDate>Fri, 25 Aug 2023 04:30:15 GMT</pubDate>
            <atom:updated>2023-08-25T04:30:15.997Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>LLMs hold tremendous promise, but generating sustainable value will require more than a tiger team</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/996/0*-x0GcCzXIDF6SkjZ" /><figcaption><em>Image courtesy of the author</em></figcaption></figure><p>You did it!</p><p>After executive leadership vaguely promised stakeholders that new Gen AI features would be incorporated across the organization, your <a href="https://www.lucidchart.com/blog/what-is-a-tiger-team#:~:text=A%20tiger%20team%20is%20a,Apollo%2013%20mission%20in%201970.">tiger team</a> sprinted to produce a MVP that checks the box. Integrating that OpenAI API into your application wasn’t that difficult and it <em>may</em> even turn out to be useful.</p><p>But now what happens? Tiger teams can’t sprint forever. Each member has another role within the organization that will once again require the majority of their time and focus.</p><p>Not to mention, there is a reason for the typical processes and structures that were ignored expedited for this project. It turns out they are pretty critical to ensuring product fit, the transition from development to operations, and cost optimization (among other things).</p><p>Come to think of it, now that the project is complete there really isn’t any platform infrastructure that can help scale the next round of LLM models or other Gen AI product features.</p><p>It looks like it’s time to start thinking about how to structure and support a Gen AI team within your data organization. And as easy as those slick product demos make the process look, there are signs of choppy waters ahead:</p><ul><li>Unless you are one of a half dozen major tech behemoths, data science and Gen AI expertise is a scarce commodity. At this point, no one really has any significant experience either. It’s new to everyone.</li><li>The business knows it wants Gen AI, but it doesn’t really know why just yet. The technology is exciting, but specific use cases are fuzzy. No one has much experience with maintaining a deployment.</li><li>The ecosystem has sprouted up overnight, but supporting technologies and best practices haven’t ripened just yet. Risks are generally unforeseen and uncertainty is high.</li></ul><p>If this sounds familiar, that’s because it is. Data science teams have <a href="https://www.montecarlodata.com/blog-why-production-machine-learning-fails-and-how-to-fix-it/">encountered all of these issues with their machine learning algorithms and applications</a> over the last five years or so.</p><p>It was a painful experience. In 2020, <a href="https://www.gartner.com/en/newsroom/press-releases/2020-10-19-gartner-identifies-the-top-strategic-technology-trends-for-2021#:~:text=Gartner%20research%20shows%20only%2053,a%20production%2Dgrade%20AI%20pipeline.">Gartner reported</a> only 53% of machine learning projects made it from prototype to production — and that’s at organizations with some level of AI experience. For companies still working to develop a data-driven culture, that number is likely far higher, with <a href="https://venturebeat.com/2019/07/19/why-do-87-of-data-science-projects-never-make-it-into-production/">some failure-rate estimates</a> soaring to nearly 90%.</p><p>As someone who led data teams at the New York Times and encountered many of these challenges, I can attest to how important organizational structure, process, and platforms are to the success of these initiatives.</p><p>I’ve also talked with hundreds of data leaders across a wide array of company sizes and industries who have expressed a common set of lessons learned. These best practices–earned through the blood, sweat, and tears of data science teams–should be top of mind for every data leader thinking about their long-term Gen AI strategy and team structure.</p><h3>Lesson 1: Understand structural trade-offs and maturity curve</h3><p>Just like you can’t wake up without any training and go run a marathon, your organization can’t create a Gen AI organizational structure that mirrors leading data teams until it has strengthened its operational muscles.</p><p>One of the most common mistakes I have seen in this regard is stretching your talent pool too thin in a rush to decentralize and embed across the company (perhaps within a <a href="https://www.montecarlodata.com/blog-what-is-a-data-mesh-and-how-not-to-mesh-it-up/">data mesh</a>). While you gain a better understanding and proximity to the business, sustained value creation is difficult.</p><p>There can be pockets of success, but these are often tied to and dependent on the talent of one or two self-starters. Data science talent is scarce, senior data science talent that can independently identify, prioritize, communicate, and execute high value projects is even more rare.</p><p>When those talented individuals depart, the institutional knowledge, code base, and project momentum often leave with them. Their surviving team members are forced to take on the role of an archeologist, attempting to discern the purpose and meaning of the artifacts left behind by these abandoned projects. Most times restarting from near scratch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/489/0*uwvYoi61HfQXD30Y" /><figcaption>Image courtesy of the author.</figcaption></figure><p>The consulting model has often been a more successful approach for smaller data science and machine learning teams. This consolidates a critical mass of talent that can be aimed at the highest priority projects. The potential downside to mitigate is you want to prevent the <a href="https://medium.com/@shane.murray5/organizing-talent-return-of-the-data-center-of-excellence-3c6a2b4a3190">center of excellence</a> from becoming a lab that produces gold plated models that a professor might love, but are not aligned to the business challenge at hand.</p><p>As a team grows bigger and ascends the maturity curve, slightly different organizational structures become more appropriate. The “specialized” model will typically focus data science and machine learning resources around a couple of high-valued problems, with teams deployed within the relevant business domain.</p><p>The most common expression of this occurs when machine learning is a core part of the product (perhaps personalization or fraud detection) and the association with the product or engineering team is more important than the association with the core data team. Often the core data team has its own investment in data science, largely independent from the specialized teams.</p><p>This model can be successful, but it does create inefficiencies and silos. For example, both central and specialized teams will typically have customized platforms with few shared services. The streaming event data within the product domain might benefit from being enriched by the custom data collected by the centralized team, but that connection might never be made.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/489/0*o2F1QqpcBii5XI-V" /><figcaption><em>Image courtesy of the author.</em></figcaption></figure><p>The other later stage organizational structure could be described as the “platform” model. Embedded and specialized models can suffer from a lack of visibility and cohesion across business domains, treating each data science problem with its own full-stack solution, despite inherent similarities in the type of problems being solved across domains.</p><p>The solution is to create some deliberate separation from business domains or verticals, so as to not overfit their operating model, as you would do with other horizontal platform teams.</p><p>A major benefit of treating machine learning as a platform pursuit is the capacity to invest in a shared platform infrastructure once you have demonstrated the value from each ML application, as it lowers the resource and cost to deploying and maintaining new applications. This investment should initially be small relative to the investment in applied teams, allowing them to operate relatively independently and pursue the long-term goals of their business partners.</p><p>In this platform model, a GenAI team could be created as one of the applied teams, with the mandate and engineering resources to tailor their stack as needed to deliver value, while coordinating with other platform teams to reuse infrastructure and standards that will deliver sustained value for the organization. I’d strongly recommend this model over trying to spread Gen AI across many teams. Critical mass matters.</p><h3>Lesson 2: Organize by use case not by business function</h3><p>Recently, I had a conversation with a data leader at a media company that was the inspiration for this post. They told me their data science teams were organized by domain (media properties in this case).</p><p>Data science teams were working on the same types of projects within each domain, namely article recommendation algorithms. No doubt each domain benefits from the dedicated focus on their specific problems and each data science team benefits from the proximity to their respective business and editorial partners. But it helped highlight some of the downsides of this organizational structure; inefficient deployment of talent and a lack of shared infrastructure despite many teams solving the same types of content ranking problems.</p><p>At the New York Times, we found it effective to organize our data science teams around common problems. Once the model had been proven in one domain, it was generally more efficient to tweak and modify it to the unique inputs and constraints of another rather than having two teams create two models in parallel. It makes sense from a logical perspective as well, it always takes more time to build the prototype than the subsequent product.</p><p>Gen AI should be considered in the same vein. Have a team focus on a high value use case appropriate for the technology, perhaps personalized seat suggestions for an event marketplace, or language localization for a media site, then apply that solution to other domains where it makes sense.</p><h3>Lesson 3: Focus on long-term value and hard problems</h3><p>“Long-term” has a unique meaning in the world of tech and data, where the average shelf life of a chief data officer is about the same as <a href="https://www.usatoday.com/story/money/2019/08/11/foods-longest-shelf-life-expiration-date/39924813/">a jar of peanut butter</a>.</p><p>Will the objective still be a problem when it’s time for the project wrap party? Will it still be a need five years from now during which the new model can be iterated and find additional value?</p><p>The reality is, assuming you aren’t leveraging an off-the-shelf model, machine learning and Gen AI initiatives can be expensive (although LLMs <a href="https://www.mosaicml.com/blog/billion-parameter-gpt-training-made-easy">are commoditizing rapidly</a>). Developing a well trained and governed model to fit a use case can take months, or in some cases, even years.</p><p>The lift <em>compared to other alternatives </em>had better be worth it. For example, a machine learning model designed to optimize Facebook ad spending might sound appealing until you realize this is done natively within the ad platform.</p><p>That said, focusing on long-term value does not mean creating a roadmap with the first release scheduled for 2025.</p><h3>Lesson 4: Partner AI teams with a business sponsor</h3><p>So how can you ensure that your data science and generative AI teams are focused on business problems that matter? Pair them up with a business sponsor.</p><p>Finding innovative applications for new technology is unlikely to be a linear journey and detours should be expected. A strong partnership with a business sponsor acts as a compass ensuring the team never wanders too far from business value as they explore the frontier. I’ve found it also broadens the team perspective beyond the horizon to problems that run across teams.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Frtthugg-s-N24ln" /><figcaption>Photo by <a href="https://unsplash.com/@jamie452?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Jamie Street</a> on <a href="https://unsplash.com/photos/_94HLr_QXo8?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>A strong business sponsor will also keep the team well provisioned during their journey, unlocking resources and helping to navigate any difficult terrain involving internal processes or politics. Part of this navigation will likely require aligning roadmaps across teams to deliver a coherent back-end and front-end experience.</p><p>Since these initiatives are likely to span quarters, executive involvement is also critical to ensuring these projects aren’t killed prematurely.</p><h3>Lesson 5: Understand data platform prerequisites</h3><p>Building the machine that builds the machine is always more difficult than producing the end product. That’s true whether it’s a factory producing a car or a data platform used to develop and productionize large language models.</p><p>Business leaders will always have the business objective in mind and will frequently overlook the <a href="https://www.linkedin.com/pulse/youre-all-genai-now-what-barr-moses">data platform investments</a> required to get there. They aren’t being malicious, they are just relying on you, the data expert, to tell them what’s required.</p><p>For example, machine learning teams have invested in building or buying feature stores and MLops solutions. Not to mention the foundational investment in cloud data environments, data quality, and the surrounding bells and whistles.</p><p>For GenAI initiatives, much of the <a href="https://www.montecarlodata.com/blog-what-is-a-data-platform-and-how-to-build-one/">data platform</a> and data pipeline architecture will remain the same (and if you haven’t yet invested in the modern data stack that’s the starting point). You can’t have a Gen AI project without discoverable, high quality data. However, there will be some additional solutions that are specific to <a href="https://huyenchip.com/2023/04/11/llm-engineering.html">LLM engineering</a> like model hosting, caching, AI agent frameworks, and many others that haven’t even been invented yet.</p><h3>Learn from the past or repeat it</h3><p>There is no mistake about it Gen AI is a disruptive technology and learning to leverage it at scale will create a new corpus of painful lessons learned. However, there is no need to start from scratch. Structure your data science and Gen AI teams for success in the long-run.</p><p>This article was co-written with Michael Segner.</p><p><a href="https://medium.com/@shane.murray5">Follow me</a> on Medium for more stories on data leadership, applications of data science, and related topics. <a href="https://medium.com/subscribe/@shane.murray5">Subscribe</a> to get my stories delivered to your inbox.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f271874ae4a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/organizing-generative-ai-5-lessons-learned-from-data-science-teams-2f271874ae4a">Organizing Generative AI: 5 Lessons Learned From Data Science Teams</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Conscious Decoupling: How Far Is Too Far for Storage, Compute, and the Modern Data Stack?]]></title>
            <link>https://medium.com/data-science/conscious-decoupling-how-far-is-too-far-for-storage-compute-and-the-modern-data-stack-ce2d9c61ccd3?source=rss-8aa0d9ae3ebd------2</link>
            <guid isPermaLink="false">https://medium.com/p/ce2d9c61ccd3</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[notes-from-industry]]></category>
            <category><![CDATA[data-platforms]]></category>
            <category><![CDATA[modern-data-stack]]></category>
            <dc:creator><![CDATA[shane murray]]></dc:creator>
            <pubDate>Mon, 24 Jul 2023 17:44:21 GMT</pubDate>
            <atom:updated>2023-07-24T17:44:21.517Z</atom:updated>
            <content:encoded><![CDATA[<h4>While there is no right answer, there is likely a sweet spot for most organizations’ data platforms. Read on to see where that might be.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*gVG_ea-FQ09EhDH7" /><figcaption>Photo by <a href="https://unsplash.com/@kellysikkema?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Kelly Sikkema</a> on <a href="https://unsplash.com/s/photos/breakup?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Data engineers discovered the benefits of <a href="https://goop.com/wellness/relationships/conscious-uncoupling-2/">conscious uncoupling</a> around the same time as Gwyneth Paltrow and Chris Martin in 2014.</p><p>Of course, instead of life partners, engineers were starting to gleefully decouple storage and compute with emerging technologies like Snowflake (2012), Databricks (2013), and BigQuery (2010).</p><p>This had amazing advantages from both a cost and scale perspective compared to on-premises databases. A data engineering manager at a Fortune 500 company expressed the pain of on-prem limitations to me by saying:</p><blockquote>“Our analysts were unable to run the queries they wanted to run when they wanted to run them. Our data warehouse would be down for 12 hours a day because we would regularly need to take it offline for data transformations and loading…The only word I can use to describe this process is painful.”</blockquote><p>A decade later, a considerable amount of innovation in the data management industry is revolving around how different data platforms are coupling or decoupling storage and compute (don’t worry I include examples in the next section). Closely related to this is how those same platforms are bundling or unbundling related data services from data ingestion and transformation to data governance and monitoring.</p><h4><strong>Why are these things related, and more importantly, why should data leaders care?</strong></h4><p>Well, the connective tissue that powers and integrates these services is frequently found in the metadata of table formats (storage) and query/job logs (compute). How these aspects are managed within your data platform will play an outsized role in its performance, cost, ease of use, partner ecosystem, and future viability.</p><p>Asking what type of data platform and what level of decoupling is correct is like asking what is the right way to format your SQL code: it is largely going to depend on personal preferences and professional requirements, however there is a small range of possibilities that will satisfy most.</p><p>I believe–at this current moment–that range for data platforms follows <a href="https://en.wikipedia.org/wiki/Golden_mean_(philosophy)">Aristotle’s golden mean</a>. The majority will be best served by options in the middle of the spectrum, while operating at either extreme will be for the select few with very specialized use cases.</p><p>Let’s dive into the current landscape and recent evolutions before looking closer at why that may be the case.</p><h3>The Storage &amp; Compute Data Platform Spectrum</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*J0pFJAaxqpLOCKFW" /><figcaption>Image courtesy of the author.</figcaption></figure><p>A <a href="https://world.hey.com/dhh/why-we-re-leaving-the-cloud-654b47e0">loud few</a> have made <a href="https://www.theregister.com/2022/10/20/basecamp_decamps_from_cloud/">headlines</a> with a “cloud is expensive let’s go back to our server racks” movement. While that may be a legitimate strategy for some, it is a rapidly dwindling minority.</p><p>Just last week, the <a href="https://open.substack.com/pub/pragmaticengineer/p/twitter-vs-instagram-threads-two?r=18lad5&amp;utm_campaign=post&amp;utm_medium=email">Pragmatic Engineer shone a spotlight on Twitter’s rate throttling</a> and significant user experience issues that likely resulted from their moving machine learning models off of GCP and relying exclusively on their three data centers.</p><p>The ability to scale and consume storage and compute independently is much more cost effective and performative, but having these separate functions within the same data platform has advantages as well.</p><p>The average unoptimized SQL query executed as part of an ad hoc analytics request will typically run a lot faster in these platforms that have been tuned to work well out of the box. A more decoupled architecture that separates compute and storage at the platform level <em>can</em> be very cost effective in running heavy workloads, but that assumes a highly trained staff spends the time optimizing those workloads.</p><p>Data platforms with combined but decoupled storage and compute also provide a more robust, integrated user experience across several key dataops tasks. Take data governance for example. These platforms provide a centralized mechanism to do access control, whereas decoupled architectures require federating roles across multiple query engines — a non-trivial task.</p><p>The decoupled but combined approach is the magic that made one of the most common reviews of Snowflake that, “Everything just works.” It’s no wonder Snowflake recently doubled down with <a href="https://www.snowflake.com/en/data-cloud/workloads/unistore/">Unistore</a> for transactional workloads and opened up Snowpark to support Python and more data science (compute) workloads.</p><p>Databricks saw amazing growth with its focus on its Spark processing framework, but it’s also not a coincidence that it unlocked a new level of growth after adding metadata and ACID-like transactions within Delta tables and governance features within Unity Catalog. They also recently doubled down and made it so when you write to a Delta table (storage) <a href="https://www.databricks.com/company/newsroom/press-releases/announcing-delta-lake-30-new-universal-format-offers-automatic">the metadata within that table is written in formats readable by Delta, Apache, and Hudi</a>.</p><h3>Challengers Platforms</h3><p>This is why it’s interesting to see that many of the latest emerging data engineering technologies are starting to separate storage and compute at the vendor level. For example, Tabular describes itself as a “<a href="https://www.theregister.com/2023/03/08/tabular_launches_data_warehouse/">headless data warehouse</a>” or “everything you need in a data warehouse except compute.”</p><p>Going a step further, some organizations are migrating to Apache Iceberg tables within a data lake with “do it yourself” management of backend infrastructure and using a separate query engine like Trino. This is most commonly due to customer-facing use cases that require highly performant and cost effective, interactive queries.</p><p>DuckDB combines storage and compute, but sacrifices the near infinite compute of the modern data stack, in favor of developer simplicity and reduced cost.</p><p>So the question remains, are these innovations likely to replace incumbent cloud native data platforms?</p><p>Again, that answer will depend on who you are. DuckDB is an incredibly popular tool that many data analysts love, but it’s probably not going to be the rock upon which you build your data platform. Ultimately, we are seeing, and I believe will continue to see a distribution that looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ItBuoWNAR0KmX3kH" /><figcaption>Image courtesy of the author.</figcaption></figure><p>I’ll explain why by looking across the modern data stack and data platform types across a couple of key dimensions.</p><h3>Degree and purpose of consolidation</h3><p>B2B vendors reverently reference the “single pane of glass” all the time. Is there value in having multiple services under a single umbrella? It depends on the quality of each service and how it corresponds to your needs.</p><p>The value in the single pane of glass <em>really</em> comes from unifying what would otherwise be siloed information into a single story, or separate actions into what should be a single workflow. Let’s use Microsoft 365 as an example of this concept.</p><p>It’s valuable having video and email integrated within their Teams collaboration application as those are core aspects of the meeting scheduling and video conferencing process. Is it as valuable to have Sway within their suite of apps? Again, it goes back to your requirements for interactive reporting.</p><p>Coming back to the data universe, compute and storage are vital to that single story (the who, what, when, where, why, how of dataops) and key workflows for aspects like: cost, quality, and access management. For this reason, these platforms will have the most robust partner ecosystems and more seamless integrations. This will likely be a key criterion for you unless you are the type of person who was using Windows and Fire phones rather than iPhones and Androids.</p><p>Adam Woods, CEO of Choozle, <a href="https://www.montecarlodata.com/blog-delivering-on-data-quality-how-choozle-reduced-data-downtime-by-88-with-monte-carlo/">spoke to our team last year</a> about the importance he places on having a robust and tightly integrated partner ecosystem surround his data platform.</p><p>“I love that…my data stack is always up-to-date and I never have to apply a patch. We are able to reinvest the time developers and database analysts would have spent worrying about updates and infrastructure into building exceptional customer experiences,” he said.</p><p>Of course, there are always exceptions to the rule. A true data lake, headless warehouse, or other more complex platform may be ideal if you have edge cases at a large scale.</p><p>Should your semantic layer, data quality, access control, catalog, BI, transformation, ingestion, and other tools all be bundled within the same platform? I think there are valid perspectives across this spectrum, but like every other department, most data teams will have a collection of tools that best fit their requirements.</p><h4>Takeaway:</h4><ul><li>Most data leaders will want to prioritize using a data platform that has both compute and storage services that can facilitate a “single story” and support a diverse partner ecosystem.</li></ul><h3>Performance vs ease of use</h3><p>Generally speaking, the more customizable a platform, the better it can perform across a wide array of use cases…and the harder it is to use. This is a pretty inescapable tradeoff and one you are making when you separate the storage and compute services across vendors.</p><p>When thinking about data platform “ease of use” it’s helpful to consider not only the day-to-day usage of the platform but the simplicity of administration and customization as well.</p><p>In my experience, many teams overfit for platform performance. Our technical backgrounds immediately start comparing platforms like they are cars: “What’s the horsepower for this workload? What about that workload?”</p><p>Don’t get me wrong, an optimized data platform can translate into millions of annual savings. It’s important. But, if you are hiring extra engineers to manage S3 configurations or you need to launch a multi-month project every quarter to onboard new aspects of the business onto your data platform, that has a high cost as well.</p><p>You see this same decision making paradigm playout with open-source solutions. The upfront cost is negligible, but the time cost of maintaining the infrastructure can be substantial.</p><p>Solution costs and engineering salary costs are not the same, and this false equivalency can create issues down the road. There are two reasons why:</p><ul><li>Assuming your usage remains static (an important caveat), your solution costs generally stay the same while efficiency goes up. That’s because SaaS vendors are constantly shipping new features. On the other hand, the efficiency of a more manual implementation will decline over time as key players leave and new team members need to be onboarded.</li><li>When you spend most of your time maintaining infrastructure, your data team starts to lose the plot. The goal slowly drifts from maximizing business value to maintaining infrastructure at peak performance. More meetings become about infrastructure. Niche infrastructure skills take on an outsized importance and these specialists become more prominent within the organization. Organizational culture matters and it is often shaped by the primary tasks and problems the team is solving.</li></ul><p>This second point was a particular emphasis for Michael Sheldon, head of data at Swimply.</p><blockquote>“Because we had this mandate as a data team to support the entire company, we needed a data stack that could solve two central issues,” Michael said. “One, to centralize all of the data from all of the different parts of the company in one stable place that everyone could use and refer to as a source of truth. <strong>And two, to enable us to have enough time to really focus on the insights and not just the data infrastructure itself</strong>.”</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mCBwtG_zAb5EZQLS" /><figcaption>Are they talking about infrastructure or business value? Photo by <a href="https://unsplash.com/@disruptxn?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Desola Lanre-Ologun</a> on <a href="https://unsplash.com/s/photos/team-collaborating?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Of course, there will be times when your business use case calls for premium performance.</p><p>A credit card fraud data product with high latency is just a waste of time. A customer-facing app that gives users the spinning wheel of death will be unacceptable, likely requiring you to deploy a high-performing query engine. However in most cases, your data warehouse or managed data lakehouse will scale just fine. Double check any requirements that say otherwise.</p><h4>Takeaway:</h4><ul><li>While ease of use and performance are interrelated variables that must be balanced, most data leaders will want to have a bias towards ease of use due to relatively hidden maintenance and culture costs. Your competitive advantage is more frequently found in enriching and applying first-party data rather than maintaining complex infrastructure.</li></ul><h3>In defense of the MDS</h3><p>I know it’s fashionable to bash the modern data stack (<a href="https://hightouch.com/blog/you-dont-need-the-mds">and you might not need it to get shit done</a>), but for all its faults, it will be the best choice for the majority of data teams. It’s an ideal blend of quick value generation and future proofing a long-term investment.</p><p>Many of the emerging technologies have significant value albeit with more narrow use cases. It will be exciting to see how they evolve and shape the practice of data engineering.</p><p>However, while compute and storage need to operate and scale separately, having those services and corresponding metadata within the same platform is just too powerful with too many advantages to ignore.</p><p><a href="https://medium.com/@shane.murray5">Follow me</a> on Medium for more stories on data leadership, applications of data science, and related topics. <a href="https://medium.com/subscribe/@shane.murray5">Subscribe</a> to get my stories delivered to your inbox.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ce2d9c61ccd3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/conscious-decoupling-how-far-is-too-far-for-storage-compute-and-the-modern-data-stack-ce2d9c61ccd3">Conscious Decoupling: How Far Is Too Far for Storage, Compute, and the Modern Data Stack?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Which Team Should Own Data Quality?]]></title>
            <link>https://medium.com/data-science/which-team-should-own-data-quality-44f1d6996eb8?source=rss-8aa0d9ae3ebd------2</link>
            <guid isPermaLink="false">https://medium.com/p/44f1d6996eb8</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-reliability]]></category>
            <category><![CDATA[data-governance]]></category>
            <category><![CDATA[data-quality]]></category>
            <category><![CDATA[notes-from-industry]]></category>
            <dc:creator><![CDATA[shane murray]]></dc:creator>
            <pubDate>Fri, 09 Jun 2023 04:38:05 GMT</pubDate>
            <atom:updated>2023-06-09T04:38:05.425Z</atom:updated>
            <content:encoded><![CDATA[<h4>Specialists or generalists? Engineer or analyst? We examine which team structures are the best suited for efficiently improving data quality.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*oR1RlmXRunTVTWwGz35D-g.png" /><figcaption>Image courtesy of Shane Murray.</figcaption></figure><p>Sure, data quality is everyones’ problem. But who owns the solution?</p><p>Given the variations in approach and mixed success, we have a lot of natural experiments from which to learn.</p><p>Some organizations will attempt to diffuse the responsibility widely across data stewards, data owners, data engineering and governance committees, each owning a fraction of the data value chain. Others concentrate the responsibility across only a few specialists who are expected to span across the entire platform. Some teams view data quality mainly as a technical challenge while others look at it as a business or process problem.</p><p>I have spoken to dozens of data leaders in the past year to understand how they approach data quality as part of their overall organizational goals. We also surveyed 200 data professionals to ask, among other things, which of their internal teams is responsible for data quality.</p><p>This post will focus on the most common team ownership models including: data engineering, data reliability engineering, analytics engineering, data quality analysts, and data governance teams.</p><h3>Why is data quality ownership important?</h3><p>But before we dive in, it’s important to answer this frequently asked follow up question. It’s typically posed in a version of, “Why does it matter who owns data quality as long as it gets done?”</p><p>There is a bit of sophistry in the phrasing since “as long as it gets done” is far from certain and frankly the whole purpose of the exercise. There are so many studies on the positive impacts of clear <a href="https://hbr.org/2017/02/how-spotify-balances-employee-autonomy-and-accountability">accountability</a>, <a href="https://www.gallup.com/workplace/236366/right-culture-not-employee-satisfaction.aspx">ownership</a>, and <a href="https://psycnet.apa.org/record/1990-97846-000">goal setting </a>that it is hard to cite just one.</p><p>The reality is that addressing data quality is unlikely to be the initiative that teams <em>want</em> to prioritize ahead of building shiny new products or services, but it’s often the one that they <em>need</em> to prioritize in order to maintain trust or scale their team and platform. And without accountability, the lower visibility tasks that advance data quality, such as unit tests or documentation, simply won’t get done to the degree they should.</p><p>Further, when responsibility is diffuse you typically wind up with fragmented solutions, uncoordinated priorities and communication gaps, ultimately leading to more downtime for your data products.</p><p>While I’ve seen all types of teams successfully implement solutions for data quality, each ownership structure has unique advantages to be leveraged and disadvantages to be mitigated.</p><p>Data leaders need to understand how each team and capability fits together. It doesn’t make sense to have a blazing fast point guard who likes to run in transition if you are also starting a lumbering center. The whole needs to be greater than the sum of its parts.</p><p>Now, let’s take a closer look at the strengths and weaknesses of the most popular data quality team structures.</p><h3>Data engineering</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*SgZmVcS4aoc0fEas44oj6w.png" /><figcaption>Photo by <a href="https://unsplash.com/@lukechesser?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Luke Chesser</a> on <a href="https://unsplash.com/s/photos/data-engineering?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Having the data engineering team lead the response to data quality is by far the most common pattern. It is deployed by about half of all organizations that use a modern data stack.</p><p>Typically, it is accompanied by a “you built the pipeline, you own it” mentality.</p><p>The strength of this approach is accountability rests with highly technical, system thinkers who are well equipped to solve system-wide problems, affecting infrastructure, code or data.</p><p>Data engineers work upstream in the systems that have a large impact on data quality. If an Airflow job, dbt model, or Fivetran sync fails, they will likely be the first to detect the issue and, with features like data lineage, can be empowered to understand the downstream blast radius to triage appropriately.</p><p>There are downsides to this approach however. For one, data engineers are often in short supply and so focused on systems and pipelines that they don’t always have as deep domain knowledge of the data. For example, they may know a dataset originates in Salesforce and the dynamics of the pipeline that land the data in the data warehouse, but they might not know the <em>client_currency_exchange_rate field</em> within that dataset can never be negative.</p><p>And while it is efficient to have the pipeline builder and maintainer be the same person, this can also create silos of tribal knowledge that can be lost when people leave and others attempt to onboard.</p><p>Effective approaches to mitigate these challenges can be to emphasize the importance of documentation to ensure knowledge transfer and to embed teams, or pair engineers with embedded analysts, to better source domain knowledge.</p><p>BlaBlaCar is an example of an organization where data engineering owns data quality. They experienced bottlenecks and challenges around capacity until they transitioned to a data mesh and leveraged data observability to reduce the amount of time required to conduct root cause analysis.</p><h3>Analytics engineering</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*4RFnlXxYPkJsOOB1uJEswA.png" /><figcaption>Used with permission:<strong> </strong><a href="https://resources.montecarlodata.com/impact-2022-on-demand/765881513?lx=a1gE1S">source<strong>.</strong></a></figcaption></figure><p>Analytics engineering teams often boast a blend of technical expertise with deep domain knowledge, making them effective leaders on data quality.</p><p>They are frequently deployed as a way to scale data transformation and access across an organization, often using dbt or similar, while the centralized data engineering team focuses on infrastructure, enterprise data management or shared services.</p><p>The strength of this approach is the strong domain expertise of a typical analytics engineer. They are well-positioned to navigate both pipeline reliability and field-level quality.</p><p>The downside may be limitations in their capacity to solve infrastructure problems or coordinate with upstream data-producing teams and systems. Only a fraction of data quality problems originate in the transformation layer, so the analytics engineers will require strong partnerships with product and platform engineering teams, to effectively triage issues in the source systems and ingestion layer respectively.</p><p>Upside’s analytical engineering team has effectively owned data quality in their organization by positioning themselves as a center of excellence across different teams. When my colleague <a href="https://resources.montecarlodata.com/impact-2022-on-demand/765881513?lx=a1gE1S&amp;_ga=2.258796426.186013647.1685973516-1023222756.1681146267">spoke to</a> senior analytics engineer, Jack Willis, last year he said:</p><blockquote>“Our analytics engineers are supposed to be accelerants, not necessarily domain experts. So we position them in the middle of all of these different specialized teams. This allows them to become a center of excellence and then be able to embed with those teams to acquire the cross-functional expertise.”</blockquote><p>The analytical engineering team found their data quality initiative became much more sustainable when they built custom data pipeline monitors and trained alongside the systems health team. By incorporating the systems health team into the design and creation process, they achieved buy-in and enabled the system health team to set up monitors that have created meaningful insights.</p><h3>Data governance</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*qS4cgmuOWfLYCNYOY_qy7g.png" /><figcaption>How Contentsquare approaches data quality as part of their data governance initiative. Used with permission: <a href="https://www.montecarlodata.com/blog-how-contentsquare-reduced-time-to-data-incident-detection-by-17-percent-with-monte-carlo/">source</a>.</figcaption></figure><p>Data governance teams often take the lead on data quality alongside the larger mandate of data security, privacy and access.</p><p>The strength is in establishing a comprehensive strategy that considers the entire value chain of data, influencing the behavior of data producers, engineers and data consumers. Often we see governance teams owning a suite of solutions for the organization that include data observability, data catalog, and access management.</p><p>Data governance teams enact change through technical standards, policies and business processes for other teams to adopt. But adoption is always more challenging when not backed by the compliance mandate of security or privacy initiatives.</p><p>At scale, it’s critical for governance teams to create the global standards for quality data, such as the minimum requirements for documentation, monitoring and SLAs, then federate accountability across the various data owners, whether they be organized into domains or data teams.</p><p>Contentsquare’s data governance team governs the access and application of data. Quality falls under their remit as well.</p><p>The governance team treats every team output as a data product. Each data product is linked to use cases, which are linked to underlying data. Data quality monitoring underpins all that underlying data, and the data team conducts regular checks to ensure each data product works as designed. My colleague spoke with their former data governance lead, Octávio Bastos, and <a href="https://www.montecarlodata.com/blog-how-contentsquare-reduced-time-to-data-incident-detection-by-17-percent-with-monte-carlo/">he discussed</a> how this was designed to help the team scale.</p><blockquote>“Sometimes when we are going so fast, we tend to focus only on the value creation: new dashboards, new models, new correlations with new data explorations,” said their former global data governance lead. “We forget to put in place good data engineering, good data governance, and an efficient data analytics team. This is very important to make sure in the long term we are scalable, and we can do more with the same team in the future.”</blockquote><h3>Data reliability</h3><p>Data reliability engineering is a specialized subset of data engineering that is focused exclusively on responsive and preventive practices to increase the quality and reliability of data systems. It is not as common of a structure, but rapidly picking up steam (we mention them and the <a href="https://www.montecarlodata.com/blog-13-data-engineering-trends-for-2023/">specialization of the data team</a> as one of our top data engineering trends of 2023).</p><p>In situations where data products are externally-facing and/or strict <a href="https://www.montecarlodata.com/blog-how-to-make-your-data-pipelines-more-reliable-with-slas/">data SLAs</a> need to be met, a dedicated team of data reliability engineers can bring the required focus to both incident response and proactive measures to address reliability.</p><p>We’ve seen within Monte Carlo’s product telemetry that teams that leverage this approach see an improvement in operational metrics for data reliability including much higher incident status updates.</p><p>However, teams and data environments need to be sufficiently large enough to gain the appropriate efficiencies from specialization.</p><p>Mercari uses a data reliability engineering team structure. Key to their success has been setting clear goals and responsibilities such as:</p><ul><li>Onboarding and supporting the most important pipelines</li><li>Modernizing data pipeline infrastructure</li><li>Proliferating data operations and monitoring practices</li><li>Secure access to customer data</li></ul><p>Their focused attention has also enabled them to make smart decisions about when to resolve and make more incremental fixes versus when a larger modernization may be necessary.</p><h3>Data quality analysts</h3><p>Finally, some organizations, particularly larger ones, will leverage data analysts or specialized data quality analysts.</p><p>The strength of this structure is these analysts are usually very close to the business and well positioned to define the required quality standards, and develop tailored tests or monitors to enforce those standards.</p><p>However, these teams often require a strong connection to data engineering in order to effectively be able to triage and troubleshoot issues upstream.</p><p>PayJoy is an example of an organization where data analysts and head of analytics, Trish Pham, have successfully owned data quality. They have over 2,000 tables and leverage data primarily to increase their visibility into business performance and enable data-driven decisions across functions.</p><h3>As long as it gets done</h3><p>Whomever your organization and data team decide to lead the response to data quality, it’s important that ownership and accountability are clear.</p><p>Start by assessing the operational, analytical, and customer facing data use cases and required data reliability levels. Then, within these critical use cases, identify the team with the most leverage over the data value chain — they should be able to own both responsive and preventative solutions, and require influence with both data producers and consumers.</p><p>This isn’t a solo project. The more you can enable the team and facilitate cross department collaboration, the more likely you are to be successful.</p><p><a href="https://medium.com/@shane.murray5">Follow me</a> on Medium for more stories on data leadership, applications of data science, and related topics. <a href="https://medium.com/subscribe/@shane.murray5">Subscribe</a> to get my stories delivered to your inbox.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=44f1d6996eb8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/which-team-should-own-data-quality-44f1d6996eb8">Which Team Should Own Data Quality?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Experimentation: How Data Leaders Can Generate Crystal Clear ROI]]></title>
            <link>https://medium.com/@shane.murray5/experimentation-how-data-leaders-can-generate-crystal-clear-roi-658405022fc8?source=rss-8aa0d9ae3ebd------2</link>
            <guid isPermaLink="false">https://medium.com/p/658405022fc8</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[experimentation]]></category>
            <category><![CDATA[data-platforms]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-quality]]></category>
            <dc:creator><![CDATA[shane murray]]></dc:creator>
            <pubDate>Thu, 13 Apr 2023 18:53:52 GMT</pubDate>
            <atom:updated>2023-04-13T18:53:52.139Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Data teams can drive quantifiable ROI by establishing a strong experimentation program. Here are the lessons we’ve learned at Airbnb, the New York Times, and other stops along the way.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FKHpCWsd_Krb6eKP" /><figcaption>Image courtesy of Shane Murray.</figcaption></figure><p><strong><em>Co-authored with Chetan Sharma</em></strong></p><p>Have you ever stood in the slide projector’s glow and pitched furrowed brow executives a change that could swing millions in revenue? Or flip the mission of hundreds of employees?</p><p><strong>Shane</strong> and his data team navigated journalistic, product and business interests to fine tune the New York Times pay model. This highwire act helped transition the Gray Lady from an ad revenue to subscriber driven publication.</p><p><strong>Chetan</strong> and team dealt an existential blow to Airbnb’s “<a href="https://www.airbnb.com/press/news/airbnb-introduces-neighborhoods">Neighborhoods</a>” travel guide, a feature that had commanded considerable resources to produce and was originally championed by CEO Brian Chesky. The decision to deprioritize its user exposure dramatically boosted incremental bookings.</p><p>What we can tell you from these, and other, experiences is driving large scale change requires a level of conviction that only comes from well run experiments that <strong>prove</strong> a change will lead to a more favorable business outcome. Or nearly psychopathic self-confidence.</p><p>These sweaty palm decisions are among the most visible contributions, but smaller non-controversial experiments can make just as large an impact. The most valuable experiment ever conducted during Chetan’s time at Airbnb was having a listing’s details open in a new window when clicked. No executive’s brows were furrowed during the production of that code.</p><p>Experimentation is a means for data teams to demonstrate strong, easily quantifiable value. As Chetan likes to say, “You’re plugged directly into the outcome of a decision. You have the receipts.”</p><p>Yet despite the increasingly urgent demand for data teams to start producing their receipts, robust experimentation programs are typically concentrated within product led growth (PLG) companies, performance marketing teams, or large tech giants. More ink is spilled on machine learning applications and dashboards than on A/B tests and p-values.</p><p>Drawing on lessons from our journey, this post will explore practical strategies for establishing a data-driven culture of experimentation across an organization such as:</p><ul><li>Building internal capacity;</li><li>Establishing a north star;</li><li>Navigating tradeoffs;</li><li>Avoiding pitfalls; and</li><li>How to get started;</li></ul><h3>Building Internal Capacity: “Fielding A Team”</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*06FEE6xU8V8BpaeO" /><figcaption>Photo by <a href="https://unsplash.com/@billstephan?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Bill Stephan</a> on <a href="https://unsplash.com/images/sports/baseball?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>To quote <em>A League of Their Own’s</em> Jimmy Dugan, “If it wasn’t hard then everyone would do it.” One of the biggest obstacles to creating a data experimentation regime that gets results is the dearth of talent and experience.</p><p>The Googles and LinkedIns of the world may employ software engineering unicorns who also boast a deep knowledge of statistics and causal inference. Most people however, don’t play for the Yankees.</p><p>Few of the people typically involved in experiments, outside of some data scientists and analysts, are likely to have revisited their AP Statistics textbook. They are more than capable, but typically an initial investment needs to be made to empower them either by acquiring expertise for a center of excellence, training, or technology.</p><p>At the New York Times, Shane made a deliberate effort to hire people with quantitative skills that could mentor others. Airbnb’s experimentation program grew from ex-Googlers attempting to increase search rankings to experimentation being core to nearly every product decision as successful experimenters were celebrated and promoted.</p><p>This democratization is crucial because the ROI of experimentation is highly correlated with the volume and diversity of experiments being conducted. Those closest to business operations often have the best hypotheses to test.</p><p>When onboarding and training “low statistical context” users, removing friction is key. It can’t take a week and a half plus a Github review to add a metric. Asking them to understand how to read from Jupyter notebooks one day and Excel spreadsheets the next is not a recipe for success. An experimentation interface has to be purposeful, intuitive, and consistent.</p><h3>Establishing A North Star: “The Moneyball Metric”</h3><p>To continue the baseball analogy, one of the biggest catalysts for Billy Beane’s famous “<a href="https://en.wikipedia.org/wiki/Moneyball">moneyball</a>” team was unlocking their north star metric. In this case, it was optimizing for on-base percentage rather than batting average (like the rest of the league at the time).</p><p>Experimentation programs’ winning streaks start when there is a clear north star metric. In the case of Airbnb it was incremental bookings. For the New York Times it was the lifetime value of a subscriber.</p><p>Why is this so important? It enables more teams to conduct meaningful experiments independently. You don’t have to be prescriptive on the exact route to take when there is a clear <a href="https://en.wikipedia.org/wiki/Intent_(military)">commander’s intent.</a></p><p>A north star metric also lets you compare the impact of different experiments. For example:</p><ul><li>The translation layer for the Chinese version of Airbnb drove bookings, but not as much as…</li><li>The search ranking model, which didn’t drive as much impact as..</li><li>Urgency messaging saying that an Airbnb listing was a rare find.</li></ul><p>Identifying and gaining consensus on a north star metric is easier said than done. Ideally, data leaders should work closely with executives to do this before the heavy lifting starts.</p><p>In some cases, it may be the experimentation process, and the tradeoffs it forces, that make formerly hypothetical questions around north stars more real and urgent.</p><h3>Navigating Tradeoffs: “Framing The Strike Zone”</h3><p>Speaking of tradeoffs, it’s important to realize that any time you are optimizing for a behavior you are, by definition, discouraging others. In the process of democratizing data experimentation it’s important to set guardrails for what types of experiments are outside the strike zone.</p><p>For example, following user visits with aggressive email campaigns would likely have led to more incremental bookings at Airbnb. However, the user’s inbox and other parts of the user experience (like avoiding adversely impacting the star rating) were considered sacrosanct because they would have fundamentally impacted the company brand. Velocity is important, but you never want to experiment with your core identity.</p><p>In a large organization like the New York Times, it’s inevitable that you’ll have different teams (or departments) seeking to improve different metrics through experimentation. For example, media companies are regularly experimenting with tactics to increase signup or subscription, which may come at the expense of further engagement and advertising revenue, at least in the near-term.</p><p>It can be helpful to establish a principle that teams can freely maximize their target metric so long as it does not significantly decline an <em>important</em> counter metric (these metrics should be narrowly defined rather than exhaustive).</p><p>For experiments where there is a clear metric trade-off, align executives on a decision criteria up front. While you may not achieve complete alignment, at least you’ve set the stage for a successful experiment.</p><p>Finally, create an “optimal frontier” based on the experiment results that clearly articulates the trade-off to be made. With the evidence on the table of the best set of options, and the avoidance of inefficient options, you’ll have a better shot at a decision even in the most consensus-driven cultures.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0F_p3ekYxN51emQz" /><figcaption>Image courtesy of Shane Murray.</figcaption></figure><p><em>The optimal frontier displays the set of treatments — i.e. from a multivariate experiment — that result in the most efficient trade-off of two competing outcomes.</em></p><h3>Avoiding Pitfalls: “Creating A Level Playing Field”</h3><p>Your program will not be successful if you don’t create a fair and level playing field for your experiments. At both the New York Times and Airbnb, considerable resources were invested in developing strong experimentation design and data infrastructure to avoid problems like:</p><ul><li><strong>Improper randomization</strong>- Many teams will attempt to randomize their control and variable groups by using problematic methods such as using the last number in the user ID. Shane had one experiment fail because an ad hoc method of randomization was systematically biasing toward more frequent users. He likes to say, “you can’t leave randomization to chance.” Data teams need a well-designed and consistent method of random assignment.</li><li><strong>Product bugs in diverse user bases</strong>- Companies have user bases that are much more diverse than they think. They span languages, time zones, subscription plans, device types, screen sizes, internet latencies. It’s very difficult to design for every user’s context. Experimentation reveals when one of those bugs or misses is going to undo any positives that you might have accomplished.</li><li><strong>Data silos</strong>- Experiments are also great at surfacing the gaps and silos in your data infrastructure. Very often, you are breaking new ground with an experiment, and will identify systems that are incapable of serving the new experience, or issues with delivering a consistent experience across platforms or surfaces.</li><li><strong>Data quality</strong>- Statistical significance is a fragile thing. If you experience data freshness issues or other data anomalies because your data pipelines broke during an experiment, all the effort that has been invested is for naught.</li><li>Unfortunately, bad data is particularly likely to strike during periods of product change that involve multiple simultaneous changes to instrumentation. This is potentially disastrous given how experiments are highly scrutinized; after all, painstakingly crafted code and promotions are often on the line.</li><li>Clearing this bar requires a high degree of data trust. It’s hard to convince departments to launch experiments or executives to trust them if no one believes in the underlying data or the dashboards they look at every day. Oftentimes these ETL systems come under considerable pressure as all of your stakeholders want to look at every metric a million different ways with sub second latency.</li><li><strong>Metric management</strong>- The curation and massaging of a central metrics store is foundational to an experiment program. The metrics layer requires constant addition, pruning, and governance. This can be time intensive, but it can be greatly expedited in a modern data stack with tools built to integrate natively with the warehouse.</li></ul><h3>Getting Started: “Don’t Start Swinging For The Fences”</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*OkFP-9u_xy6XkQVx" /><figcaption>Source: <a href="https://sabr.org/journal/article/home-runs-and-strikeouts-another-look/">Society for Baseball Research</a>.</figcaption></figure><p>Home run hitters can be game changers, but they strike out a lot. When you are starting to build operational muscle around your experimentation program, it’s better to be Ichiro than Mark McGwire.</p><p>Avoid experiments on products that take a long time to develop. These will inherently be more political due to the real and emotional investment made by the business.</p><p>Instead, aim to get a lot of at-bats. Your first few experiments will serve as a kind of spring training. They will reveal many of the context and infrastructure challenges mentioned in the previous section.</p><p>To build up quick wins and momentum, one of the first places to start is by scheduling a conversation with someone who was with the company when it found product market fit. Even if it wasn’t formalized, they were running constant experiments and changing the product to determine what levers led to adoption and growth. They will have a strong, intuitive understanding of the channels that get users hooked.</p><p>Then conduct a complete walk through of how users experience a product. At this point, you are mostly going to be looking at removing friction points or reducing large prominent elements on screen (unless there is rock solid rationale). Experimentation design should start with simple A/B tests rather than more complex Bandit approaches.</p><p>Once you’ve gotten a few singles and ground-rule doubles, now it’s time for the home run derby.</p><p>Reaching this stage is important because while velocity and iterative learning is critical, you can’t iterate your way to a new business model.</p><p>Too often experiments in large organizations are constrained by what’s possible in the current user experience or what is controllable by a single product team, while the task at hand might be to reimagine what the experience could be.</p><p>The narrow mandate of product teams and the lack of good methodology can lead to what are effectively bunts, that are barely discernible from random noise. Bigger swings require teams to think beyond their own scope and define hypotheses that fundamentally alter the user experience and business model.</p><h3>What’s Hard Is What Makes It Great</h3><p>Data teams cannot afford to wait to build great experimentation programs.</p><p>It is an abdication of responsibility to solely be involved in the dashboard creation process and put the onus entirely on business stakeholders to naturally experiment with their tactics and interpret the results.</p><p><a href="https://hbr.org/2017/09/the-surprising-power-of-online-experiments">The experience</a> of the largest and best experimentation teams have shown that about one-third of experiments will be positive while one-third will be negative. If data teams don’t step up to the experimentation plate to call the balls and strikes, then it will be the organization that is striking out.</p><p><em>Chetan Sharma is the CEO of </em><a href="https://www.geteppo.com/about-us"><em>Eppo</em></a><em>, the only warehouse-native experimentation platform. He was one of the first data scientists at Airbnb.</em></p><p><em>Shane Murray is the field CTO of Monte Carlo. He previously served as the Senior VP of Data and Insights at the New York Times. He is an investor in Eppo.</em></p><p><a href="https://medium.com/@shane.murray5">Follow me</a> on Medium for more stories on data leadership, applications of data science, and related topics. <a href="https://medium.com/subscribe/@shane.murray5">Subscribe</a> to get my stories delivered to your inbox.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=658405022fc8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Chaos Data Engineering Manifesto]]></title>
            <link>https://medium.com/data-science/the-chaos-data-engineering-manifesto-5dc09a182e85?source=rss-8aa0d9ae3ebd------2</link>
            <guid isPermaLink="false">https://medium.com/p/5dc09a182e85</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[notes-from-industry]]></category>
            <category><![CDATA[chaos-engineering]]></category>
            <category><![CDATA[data-reliability]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[shane murray]]></dc:creator>
            <pubDate>Fri, 24 Feb 2023 22:26:22 GMT</pubDate>
            <atom:updated>2023-02-24T22:26:22.532Z</atom:updated>
            <content:encoded><![CDATA[<h3>The Chaos Data-Engineering Manifesto</h3><h4>Another lesson we can learn from software engineers: break stuff to make it more reliable.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UHC1t3RNad_hesU8" /><figcaption><em>Photo by </em><a href="https://unsplash.com/@msohebzaidi?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText"><em>Soheb Zaidi</em></a><em> on </em><a href="https://unsplash.com/s/photos/chaos?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText"><em>Unsplash</em></a></figcaption></figure><p>It’s midnight in the dim and cluttered office of <em>The</em> <em>New York Times</em> currently serving as the “situation room.”</p><p>A powerful surge of traffic is inevitable. During every major election, the wave would crest and crash against our overwhelmed systems before receding, allowing us to assess the damage.</p><p>We had been in the cloud for years, which helped some. Our main systems would scale—our articles were always served—but integration points across backend services would eventually buckle and burst under the sustained pressure of insane traffic levels.</p><p>However, this night in 2020 was different from similar election nights in 2014, 2016, and 2018. That’s because this traffic surge was simulated and an election wasn’t taking place.</p><h3>Pushing to the point of failure</h3><p>Simulation or not, this <em>was</em> prod so the stakes were high. There was suppressed horror as J-Kidd–our system that brought ad targeting parameters to the front end–went down hard. It was as if all the ligaments had been ripped from the knees of the pass-first point guard for which it had been named. Ouch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/0*MpgtgxDcgE3e5DIT" /><figcaption><em>Via </em><a href="https://www.google.com/url?sa=i&amp;url=https%3A%2F%2Fcommons.wikimedia.org%2Fwiki%2FFile%3AJason_Kidd_Mike_Woodson.jpg&amp;psig=AOvVaw2eZ0VfIHJlUc8ya-YFm_Rg&amp;ust=1677332195040000&amp;source=images&amp;cd=vfe&amp;ved=0CA8QjRxqFwoTCMDoy9Gjrv0CFQAAAAAdAAAAABAD"><em>Wikimedia Commons</em></a><em>. Credit: KeithAllisonPhoto.com. </em>This file is licensed under the <a href="https://en.wikipedia.org/wiki/en:Creative_Commons">Creative Commons</a> <a href="https://creativecommons.org/licenses/by-sa/2.0/deed.en">Attribution-Share Alike 2.0 Generic</a> license.</figcaption></figure><p>J-Kidd wasn’t the only system that found its way to the disabled list. That was the point of the whole exercise, to push our systems until they failed. We succeeded. Or failed depending on your point of view.</p><p>The next day the team made adjustments. We decoupled systems, implemented failsafes, and returned to the court for game 2. As a result, the 2020 election was the first I can remember where the on-call engineers weren’t on the edge of their seats, white-knuckling their keyboards…At least not for system reliability reasons.</p><h3>Pre-Mortems and Chaos Engineering</h3><p>We referred to that exercise as a “pre-mortem.” Its conceptual roots can be traced back to the idea of <a href="https://principlesofchaos.org/">chaos engineering</a> introduced by site reliability engineers.</p><p>For those unfamiliar, chaos engineering is a disciplined methodology for intentionally introducing points of failure within systems to better understand their thresholds and improve resilience.</p><p>It was popularized in large part by the success of <a href="https://netflixtechblog.com/the-netflix-simian-army-16e57fbab116">Netflix’s Simian Army</a>, a suite of programs that would automatically introduce chaos by removing servers, regions, and introduce other points of failure into production. All in the name of reliability and resiliency.</p><p>While this idea isn’t <a href="https://lakefs.io/blog/chaos-data-engineering/">completely</a> <a href="https://www.linkedin.com/posts/eczachly_dataengineering-activity-7033975323838877696-s4Qj?utm_source=share&amp;utm_medium=member_desktop">foreign</a> <a href="https://medium.com/geekculture/why-you-should-occasionally-kill-your-data-stack-613143c986ea">to</a> data engineering, it can certainly be described as an extremely uncommon practice.</p><p>No data engineer in their right mind has looked at their to-do list, the unfilled roles on their team, the complexity of their pipelines, and then said: “This needs to be harder. Let’s introduce some chaos.” That may be part of the problem.</p><p>Data teams need to think beyond providing snapshots of data quality to the business and start thinking about how to build and maintain reliable data systems at scale.</p><p>We cannot afford to overlook data’s increasingly large role in critical operations. Just this year, we witnessed how the deletion of one file and an out-of-sync legacy database <a href="https://www.nbcnews.com/news/us-news/us-flights-grounded-faa-outage-rcna65243">could ground more than 4,000 flights</a>.</p><p>Of course, you can’t just copy and paste software engineering concepts straight into data engineering playbooks. Data is different. DataOps tweaks DevOps methodology as <a href="https://www.montecarlodata.com/blog-what-is-data-observability/">data observability</a> does to observability.</p><p>Consider this manifesto then a proposal for how to take the proven concepts of chaos engineering and apply them to the eccentric world of data.</p><h3>The 5 Laws Of Data Chaos Engineering</h3><p>The principles and lessons of chaos engineering are a good place to start defining the contours of a data chaos engineering discipline. Our first law combines two of the most important.</p><h4>First Law: Have a bias for production, but minimize the blast radius</h4><p>There is a maxim among site reliability engineers that will ring true for every data engineer who has had the pleasure of the same SQL query return two different results across staging and production environments. That is, “Nothing acts like prod except for prod.”</p><p>To that I would add, “production data too.” Data is just too creative and fluid for humans to anticipate. <a href="https://medium.com/cord-tech/the-advantages-and-disadvantages-of-synthetic-training-data-b3acbb68e116">Synthetic data</a> has come a long way, and don’t get me wrong, it can be a piece of the puzzle, but it’s unlikely to simulate key edge cases.</p><p>Like me, the mere thought of introducing points of failure into production systems probably makes your stomach churn. It’s terrifying. Some data engineers justifiably wonder, “Is this even necessary within a modern data stack where so many tools abstract the underlying infrastructure?”</p><p>I’m afraid so. Remember, as the opening anecdote and J-Kidd’s snapped ligaments illustrated, the elasticity of the cloud is not a cure-all.</p><p>In fact, it’s that abstraction and opacity–along with the multiple integration points–that make it so important to stress test a modern data stack. An on-premise database may be more limiting, but data teams tend to understand its thresholds as they hit them more regularly during day to day operations.</p><p>Let’s move past the philosophical objections for the moment, and dive into the practical. Data is different. Introducing fake data into a system won’t be helpful because the input changes the output. It’s going to get really messy too.</p><p>That’s where the second part of the law comes into play: minimize the blast radius. There is a spectrum of chaos and tools that can be used:</p><ul><li>In words only, “let’s say this failed, what would we do?”</li><li>Synthetic data in production</li><li>Techniques like data diff that allow you to test snippets of SQL code on production data</li><li>Solutions like LakeFS allow you to do this on a bigger scale by creating “chaos branches” or complete snapshots of your production environment where you can use production data but with complete isolation.</li><li>Do it in prod, and practice your backfilling skills. After all, nothing acts like prod, but prod.</li></ul><p>Starting with lesser chaotic scenarios is probably a good idea and will help you understand how to minimize the blast radius in production.</p><p>Deep-diving into real production incidents is also a great place to start. Does everyone really understand what exactly happened? Production incidents are chaos experiments that you’ve already paid for, so make sure that you are getting the most out of them.</p><p>Mitigating the blast radius may also include strategies like backing up applicable systems or having data observability or data quality monitoring solution in place to assist with the detection and resolution of data incidents.</p><h4>Second Law: Understand it’s never a perfect time (within reason)</h4><p>Another chaos engineering principle holds to observe and understand “steady state behavior.”</p><p>There is wisdom in this principle, but it is also important to understand the field of data engineering isn’t quite ready to measure by the standard of “5 9s” or 99.999% uptime.</p><p>Data systems are constantly in flux and there is a wider range of “steady state behavior.” There will be the temptation to delay the introduction of chaos until you’ve reached the mythical point of “readiness.” Well, you <a href="https://www.montecarlodata.com/blog-you-cant-out-architect-bad-data/">can’t out-architect bad data</a> and no one is ever ready for chaos.</p><p>The Silicon Valley cliche of fail fast is applicable here. Or <a href="https://twitter.com/reidhoffman/status/847142924240379904?lang=en">to paraphrase Reid Hoffman</a>, if you aren’t embarrassed by the results of your first pre-mortem/fire drill/chaos introducing event, you introduced it too late.</p><p>Introducing fake data incidents while you are dealing with real ones may seem silly, but ultimately this can help you get ahead by better understanding where you have been putting bandaids on larger issues that may need to be refactored.</p><h4>Third Law: Formulate hypotheses and identify variables at the system, code, and data levels</h4><p>Chaos engineering encourages forming hypotheses of how systems will react to understand what thresholds to monitor. It also encourages leveraging or mimicing past real-world incidents or likely incidents.</p><p>We’ll dive deeper into the details of this in the next section, but the important modification here is to ensure these span the system, code, and data levels. Variables at each level can create data incidents, some quick examples:</p><ul><li><strong>System</strong>: You didn’t have the right permissions set in your data warehouse.</li><li><strong>Code</strong>: A bad left JOIN.</li><li><strong>Data</strong>: A third-party sent you garbage columns with a bunch of NULLS.</li></ul><p>Simulating increased traffic levels and shutting down servers impact data systems, and those are important tests, but don’t neglect some of the more unique and fun ways data systems can break bad.</p><h4>Fourth Law: Everyone in one room (or at least Zoom call)</h4><p>This law is based on the experience of my colleague, site reliability engineer, and chaos practitioner <a href="https://www.linkedin.com/in/timtischler">Tim Tischler</a>.</p><p>“Chaos engineering is just as much about people as it is systems. They evolve together and they can’t be separated. Half of the value from these exercises come from putting all the engineers in a room and asking, ‘what happens if we do X or if we do Y?’ You are guaranteed to get different answers. Once you simulate the event and see the result, now everyone’s mental maps are aligned. That is incredibly valuable,” he said.</p><p>Also, the interdependence of data systems and responsibilities creates blurry lines of ownership even on the most well-run teams. Breaks often happen, and are overlooked, in those overlaps and gaps in responsibility where the data engineer, analytical engineer, and data analyst are pointing at the other.</p><p>In many organizations, the product engineers creating the data and the data engineers managing it are separated and siloed by team structures. They also often have different tools and models of the same system and data. Feel free to pull these product engineers in as well, especially when the data has been generated from internally built systems.</p><p>Good incident management and triage can often involve multiple teams and having everyone in one room can make the exercise more productive.</p><p>I’ll also add from personal experience that these exercises can be fun (in the same weird way putting all your chips on red is fun). I’d encourage data teams to consider a chaos data engineering fire drill or pre-mortem event at the next offsite. It makes for a much more practical team bonding exercise than getting out of an escape room.</p><h4>Fifth Law: Hold off on the automation for now</h4><p>Truly mature chaos engineering programs like Netflix’s Simian Army are automated and even unscheduled. While this may create a more accurate simulation, the reality is that the automated tools don’t currently exist for data engineering. If they did, I’m unsure if I would be brave enough to use them.</p><p>To this point, one of the original Netflix chaos engineers <a href="https://medium.com/@njones_18523/chaos-engineering-traps-e3486c526059">has described</a> how they didn’t always use automation as the chaos could create more problems than they could fix (especially in collaboration with those running the system) in a reasonable period of time.</p><p>Given data engineering’s current reliability evolution and the greater potential for an unintentionally large blast radius, I would recommend data teams lean more towards scheduled, carefully managed events.</p><h3>Data Chaos Engineering Variables, AKA What To Break</h3><p>It would be a bit of a cop out to suggest data engineers overhaul their processes and introduce greater levels of risk without providing more concrete examples of tests or the value that can be derived from them.</p><p>Here are some suggestions.</p><h4>Systems Level</h4><p>At this level you should be testing the integration points across your IT and data systems. This includes your data pipelines from ingestion and storage to orchestration and visualization. You could create some chaos by:</p><ul><li><strong>Creating problems in Postgres</strong>- You may discover you need to decouple operational systems from your analytical systems.</li><li><strong>Killing Airflow jobs or Spark clusters- </strong>You may discover you need alerts to notify the team when these failures occur and<a href="https://www.startdataengineering.com/post/design-patterns/#32-behavioral"> self-healing pipeline design</a>.</li><li><strong>Changing up your number of Spark core executors</strong>-<a href="https://www.linkedin.com/posts/eczachly_dataengineering-activity-7023802704107888640-kAtW?utm_source=share&amp;utm_medium=member_desktop"> Zach Wilson has pointed out </a>how adding more Spark executor cores can negatively impact reliability without boosting performance. Experiment with that.</li><li><strong>Adjust the auto-suspend policy in Snowflake</strong>- A lot of teams are working to optimize their data cloud costs and it’s easy to cut a bit too deep and cause reliability issues. For example, if you set the auto-suspend more tightly than any gaps in your query workload, the warehouse could end up in a continual state of auto-suspend and auto-resume. This might help you discover you need to re-organize and categorize your data workloads by data warehouse size. Or, maybe be a bit more judicious on who has permission and authority to make these types of changes.</li><li><strong>Unpublish a dashboard</strong>- I call this the scream test. If no one says anything weeks after unpublishing a dashboard, chances are you can retire it. This can also help test your detection and communication skills across data engineering and analyst teams.</li><li><strong>Simulate a connection failure with your streaming pipelines</strong>- The Mercari data team <a href="https://www.montecarlodata.com/blog-mercari-data-reliability-engineering/">deftly fixed this issue quickly</a> thanks to data monitoring alerts. They are tentatively planning to simulate similar issues to drill their detection, triage, and resolution processes.</li></ul><p>“The idea is we’ll roleplay an alert from a particular dataset and walk through the steps for triage, root cause analysis, and communicating with the business,” said Daniel, member of the Mercari data reliability engineering team. “By seeing them in action, we can better understand the recovery plans, how we prioritize to ensure the most essential tables are recovered first, and how we communicate this with a team that spans across four time zones.”</p><h4>Code Level</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WaUBes4cV2KFp0qM" /><figcaption><em>What happens when you remove that block (of code) does it all come crashing down? Photo by </em><a href="https://unsplash.com/@mparzuchowski?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText"><em>Michał Parzuchowski</em></a><em> on </em><a href="https://unsplash.com/s/photos/jenga?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText"><em>Unsplash</em></a></figcaption></figure><p>This level is the T (transformation) in the ETL or ELT process. What happens when something goes wrong in your dbt models or SQL queries? Time to find out!</p><ul><li><strong>Archive a dbt model-</strong> Similar to the dashboard scream test or Jenga. Is that model really needed or does the structure hold just fine when you remove a block? This can help evaluate your testing and monitoring; whether there are unknown dependencies; and lines of ownership.</li><li><strong>Adjust how a key metric is calculated-</strong>When there are multiple versions of truth within an organization, trust in data is inevitably undermined by conflicting trends. Do changes to a key metric flow to all your downstream reports? Are there rogue definitions floating around? This can help evaluate the organizational process for defining key metrics and whether you might need to invest in a <a href="https://www.getdbt.com/product/semantic-layer/">sema</a>n<a href="https://www.getdbt.com/product/semantic-layer/">tic layer</a>.</li><li><strong>Have other domains interpret your models or SQL code</strong>- Sure, one engineer or team might understand the models in a particular domain, but what happens when others need to understand? Is it well documented? Can others help triage incidents?</li></ul><h4>Data Level</h4><p>Where software engineers also deal with reliability issues at the code and system level, only data engineers must address potential issues at the data level. Have I mentioned that data is different?</p><p>This level can’t be neglected because your Airflow jobs may be running fine, your data diff may check out, but the data running across them might just be complete garbage. Garbage in, garbage out (and maybe systems down)</p><ul><li><strong>Change the schema-</strong> Software engineers regularly and unintentionally commit code that changes how the data is output in a service, breaking downstream data pipelines. You may need to decouple systems and, for some operational pipelines, require a <a href="https://www.montecarlodata.com/blog-data-contracts/">data contract </a>architecture in place (or some other process for better engineering coordination).</li><li><strong>Create bad data- </strong>This may reveal your models aren’t robust against spikes of volume, NULLs, etc in their source tables.</li><li><strong>Duplicate the data</strong>- Does a spike in data reveal bottlenecks causing jobs to fail? Do you need to microbatch certain pipelines? Are your pipelines idempotent where they need to be?</li></ul><h3>We’re Talking About Practice</h3><p>We started this manifesto with a point guard, perhaps it’s fitting we conclude with <a href="https://www.facebook.com/SportsCenter/videos/allen-iversons-practice-rant/470685314188618/">this infamous 2002 quote</a> from shooting guard Allen Iverson as a warning for those who don’t buy into the “practice like you play” concept:</p><blockquote><em>“We’re in here talking about practice. I mean, listen, we’re talking about practice. Not a game! Not a game! Not a game! We’re talking about practice. Not a game; not the game that I go out there and die for and play every game like it’s my last, not the game, we’re talking about practice, man. I mean, how silly is that? We’re talking about practice.</em></blockquote><blockquote><em>I know I’m supposed to be there, I know I’m supposed to lead by example, I know that. And I’m not shoving it aside like it don’t mean anything. I know it’s important. I do. I honestly do. But we’re talking about practice, man. What are we talking about? Practice? We’re talking about practice, man!</em></blockquote><blockquote><em>We’re talking about practice! We’re talking about practice… We ain’t talking about the game! We’re talking about practice, man!”</em></blockquote><p>That year the 76ers lost in the Eastern Conference First Round of the playoffs to the Boston Celtics in five games. Iveson’s field goal percentage for the series was a tepid 38.1%.</p><p><a href="https://medium.com/@shane.murray5">Follow me</a> on Medium for more stories on data leadership, applications of data science, and related topics. <a href="https://medium.com/subscribe/@shane.murray5">Subscribe</a> to get my stories delivered to your inbox.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5dc09a182e85" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/the-chaos-data-engineering-manifesto-5dc09a182e85">The Chaos Data Engineering Manifesto</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Meaningful Experimentation: 5 Impactful Data Projects To Help Build Better Products]]></title>
            <link>https://medium.com/data-science/meaningful-experimentation-5-impactful-data-projects-to-help-build-better-products-1fb84afdfb2?source=rss-8aa0d9ae3ebd------2</link>
            <guid isPermaLink="false">https://medium.com/p/1fb84afdfb2</guid>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[notes-from-industry]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-platforms]]></category>
            <category><![CDATA[product-management]]></category>
            <dc:creator><![CDATA[shane murray]]></dc:creator>
            <pubDate>Fri, 06 Jan 2023 21:26:48 GMT</pubDate>
            <atom:updated>2023-01-06T21:26:48.924Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Part two in a practical data leader series on how to best work with each department and all their wonderful eccentricities</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aZ3IMwo5wiCGVzgO" /><figcaption>Image courtesy of Shane Murray.</figcaption></figure><p>Understanding and aligning with each business domain’s unique incentives and workflows is what ultimately makes data teams not just efficient, but great.</p><p><a href="https://medium.com/towards-data-science/how-data-and-finance-teams-can-be-friends-and-stop-being-frenemies-7ecc357f51ef">Part one of this series</a> looked at everyone’s favorite spreadsheet power users, the finance team. This article will examine how data teams can better identify the common causes and avoid points of friction with product teams.</p><p>While these interactions will certainly differ based on the nature of the product being delivered, this post will be applicable to a wide range of organizations that primarily offer a digital, consumer-facing product. Typically in these organizations cross-functional product teams (product managers, designers, engineers, etc) collectively own the end-to-end consumer experience and individually own aspects of that experience, such as features or products.</p><p>Let’s start by looking at some examples of impactful projects where data and product teams can find shared wins before diving into tips for driving meaningful collaboration.</p><h3>Projects that will win over your product team</h3><p>Product teams might own anything from an email to an app. They can be narrowly focused on a new feature or broadly evaluating an entirely new monetization strategy. However, there are some commonalities that can be drawn upon for product analytics best practices.</p><p>Here are five of those best practices I’ve seen that have driven results:</p><h4>1. A data-informed product strategy</h4><p>Product teams are tasked with being data-informed at all stages in the product life cycle, from idea generation and product definition, to validating prototypes and building a commercially successful product.</p><p>An unfortunate downside I’ve encountered over the years is the product team who wants data to dictate their strategy, perhaps by asking their data partners, “What feature should we build next?”</p><p>It’s understandable — developing new features is a resource intensive process with high stakes. This puts pressure on PMs to make the right calls, which often creates an existential need for some level of certainty, or at least defensible justification. But the analyst is only armed with data on past experiments and decisions, not the counterfactuals that might allow them to extrapolate to future decisions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AnYAW15sNNqUDfJd" /><figcaption><em>Image by JUVART via </em><a href="https://www.shutterstock.com/image-vector/development-team-creating-mobile-application-set-2108335400"><em>Shutterstock</em></a><em>.</em></figcaption></figure><p>Analytics teams will produce their best work when there are clear product hypotheses to test or explore, and a set of options that might be evaluated against a well-defined outcome or success criteria.</p><p>For example, “A Discover tab will outperform a Recent Plays tab for daily engagement,” is a hypothesis that can be explored either with historical data (and some assumptions around tab usage) or a proposed experiment. On the other hand, if the question is, “Should we build a Discover tab or something else?” you might want to return to user research.</p><p><em>So where does that leave us?</em></p><p>Data teams should take the lead on (~annual) modeling exercises to explore the most important product levers that correlate with growth (i.e. customer acquisition &amp; retention). Early efforts will come with the usual caveat, “it’s correlation, not causation” as a causal understanding of the product mechanics will be built over time and multiple iterations.</p><p>These models can be valuable guideposts for where to invest in the product, but should never be prescriptive. <strong>The bold idea that forces you to throw out the current model is likely more impactful than the one you can deduce from how the current product levers impact the business.</strong></p><p>I’ve found this work will generate demand for more granular models, that dig into the importance of product levers within smaller products (e.g. newsletter, app) or at different stages of the customer journey.</p><p>Additionally, data teams can use this deeper, holistic understanding of the business to define a “north star metric” or the suite of “metrics that matter,” that can act as the compass for product teams as they piece together the product strategy and roadmap.</p><h4>2. The basics: Tracking, reporting and post launch evaluation</h4><p>The most foundational practice in product analytics is the definition of metrics, tracking and reporting that allows the product or feature to be reliably evaluated after launch. But so often data is not a first-class citizen in product launches.</p><p><strong>In fact, data is often the last thing considered before launch, but the first thing asked for after launch.</strong></p><p>It’s incumbent on data leaders and product leaders to make quality data integral to the launch of a product. The best product leaders I’ve worked with have the attitude of, “if we can’t measure it, then we can’t launch it.”</p><p>Ahead of a launch, the discussion typically starts with the data team asking what metrics are important to report and what user segments matter. The answer from the product team is almost invariably, “everything and everyone.”</p><p>But the best product analytics teams are part anthropologist, part engineer. They initially research the behaviors they want to study on the populations exposed to the new feature, define a clear criteria for success, and then translate these into codified outcome metrics and user segments that can be implemented. Doing the work upfront will save a lot of time on the back-end, when the executives ask whether the launch was a success.</p><h4>3. Experimentation</h4><p>Experimentation is the lifeblood of a product manager. With most digital product development teams shipping in an agile fashion, the ability to run experiments is generally considered “table stakes” these days.</p><p>But doing product experimentation <em>well</em> is far from “table stakes.” It isn’t always as simple as splitting an email campaign into control and test groups, adjusting the copy, then measuring response rates.</p><ul><li><strong>The stakes are often higher </strong>— for example, user experience changes can require many days to design, build and deploy.</li><li><strong>There are multiple points of integration</strong> — for example, you may need to deploy your experiment allocation library via SDK or CDN, in addition to offering integration via JavaScript in the browser.</li><li><strong>Outcomes are less immediately known</strong> — for example, a recommendation system might need to be measured based on how it affects user engagement and retention over many weeks or months, rather than its clickability.</li><li><strong>The trade-offs can be more complex </strong>— for example, a home page experiment may require measurement of multiple business outcomes, each with different value to the business and different sets of stakeholders to manage.</li><li><strong>The methods may vary based on the type of product</strong> — for example, you may use discrete (A/B and multivariate) or adaptive (“bandit”) modes of experimentation, depending on the shape of the optimization problem at hand.</li></ul><p>For these reasons, it’s beneficial for data leaders to proactively standardize the platform or tools to support product experimentation. This is especially the case when the execution of experiments will likely be decentralized across many cross-functional product teams.</p><p>Don’t assume you can buy or build the platform to support all use cases. But have a clear set of use cases that the platform will support, and sufficient flexibility in implementation and data collection to allow for less common or more complex experiments to be reliably delivered.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2VXMAp8RYgGkDY-k" /><figcaption>Photo by <a href="https://unsplash.com/@hansreniers?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Hans Reniers</a> on <a href="https://unsplash.com/s/photos/lab-experimentation?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a>.</figcaption></figure><p>Experimentation velocity is often prized by the product teams as the key to building and scaling successful products, often epitomized by Google’s (seriously flawed) “<a href="https://www.theguardian.com/technology/2014/feb/05/why-google-engineers-designers">41 shades of blue” experiment</a>.</p><p>But while velocity and iterative learning is critical, <strong>you can’t iterate your way to a new business model.</strong> Too often experiments in large organizations are constrained by what is possible in the current user experience and / or what is controllable by a single product team, while the task at hand might be to reimagine what the experience could be.</p><p>That’s why the data team should also drive best practices for experiment design and analysis, such as:</p><ul><li>Lead with behavioral hypotheses rather than variations of product features</li><li>Translate hypotheses into independent experiment attributes (and maybe leave the shade of blue up to the design team)</li><li>Consider the right experiment design — is it a series of consecutive A/B tests or does it require a multivariate approach, such that you can understand interactions between experiment attributes?</li></ul><p>Don’t get caught in a local maximum. The most impactful experiments I’ve seen over the last 15 years have required teams to think beyond their own scope and define hypotheses that fundamentally alter the user experience and business model.</p><h4>4. Cohort analysis</h4><p>Just behind running controlled experiments, cohort analysis is the next most useful analytical tool for product teams. This method involves dividing users into groups with common characteristics as a means to evaluate user health and develop product hypotheses.</p><p>There’s no one right way to do cohort analysis — the possibilities are endless — but there are many ways to get it wrong. I’ll explain a few ways I’ve seen this technique misused, and how to do it right.</p><p><em>First, let’s consider a product team seeking to understand the health of their user base, and whether the product experience is getting “stickier” over time.</em></p><p>Here, “health” may be defined by user retention, a more specific level of active engagement, or perhaps a user’s transition to a paying customer.</p><p>Avoid a loose definition of the cohort, such as comparing <em>all users</em> of your app week-by-week, where you are likely to cloud the analysis with return users in different proportions within each cohort.</p><p>For this analysis to be useful, it’s critical that you are comparing apples with apples, such as focusing on <em>new users</em> arriving each week. You might further segment these cohorts (e.g. based on geography or device) in order to diagnose increases and decreases in user health.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZscZYrfdkfcmTKB-" /><figcaption><em>This chart plots User Retention over Time for new users to the product each week, showing the health of each cohort. </em>Image courtesy of Shane Murray.</figcaption></figure><p><em>Second, let’s consider a product team seeking to understand which features are the most valuable in driving healthy users.</em></p><p>Avoid returning with an analysis that says “people that do x are more likely to retain.” These statements are practically useless for a few reasons.</p><p>Firstly, users that perform any action are bound to be more likely to retain — you may just be cherry-picking; secondly, you are not providing any context on the feature’s usage — it could be the most unpopular feature ever; finally, the lack of temporal awareness makes it more likely that the statement is purely correlative.</p><p>Instead, you should:</p><ul><li>Time-box the cohort actions (e.g. first 30 days) to separate potential cause from subsequent effect (e.g. day 30-to-180)</li><li>Compare many actions so as not to cherry-pick features of interest</li><li>Provide context around the feature’s usage along with its impact on user retention</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*i5E_LkQmI7bRWnd6" /><figcaption>Feature Scatterplot. Image courtesy of Shane Murray.</figcaption></figure><p>The Feature Scatterplot (above) plots Feature Usage by Retention Lift for a set of product features labeled A through H. For example, Features A is lightly adopted but correlates to the largest subsequent lift on retention; Feature D is heavily adopted but correlates with a much smaller lift on retention.</p><p>While this analysis does not prescribe where to invest, it can inform future product development:</p><ul><li>You might sunset feature E due to its lack of adoption and impact on retention</li><li>You might choose to drive more adoption of high impact features A, B or F</li><li>You might choose to rebuild a feature that is underperforming its potential adoption or impact</li></ul><p>Product analytics tools, such as Amplitude and Mixpanel, have made easy cohorting their bread-and-butter, provided you’ve gone through the upfront implementation. But you may also choose to build it yourself within a BI tool or custom app, starting with basic behavioral cohorting then adding in dimensions to support more exploration.</p><h4>5. Self-serve access to data</h4><p>Across all of the above product data needs you face a common question — <em>how do we make this self-serve?</em></p><p>While data teams have become increasingly agile, product teams expect the answers to seemingly straightforward questions (e.g. “how does retention differ for users who interacted with push alerts?”) at their fingertips, even if no one has pondered that specific question before.</p><p>Also, analytical work still carries an expectation of <em>do-it-yourself</em>, in a way that design or engineering do not. While tempting to write-off our product stakeholders as not understanding our field, it’s very often because analysts are stuck responding to rudimentary questions that <em>could</em> be addressed with better access to high-quality data, rather than spending time on more sophisticated, thorough, or profound analysis that clearly demonstrates their expertise.</p><p>The problem then for data teams to solve is how to invest in the right self-serve access for their product stakeholders, in order to accomplish:</p><ul><li>Broader access to trustworthy data, for making good decisions faster</li><li>A shift in analyst work away from service-oriented responses, and towards a data insights roadmap</li></ul><p>Time spent on instrumentation and access is usually well-spent, but the pay-off in terms of self-serve use must be greater than the effort spent to build and maintain it. You may find that building self-serve for basic metric access makes a lot of sense, but the further you push towards more fine-grained or sophisticated analysis, the juice is only worth the squeeze if you can generalize the solution and drive significant adoption across many product teams.</p><p>You can take multiple paths to delivering these capabilities — from off-the-shelf to homegrown and somewhere in between. Data teams will want to maintain a single source of truth in their warehouse, in order to minimize the cost to govern and maintain trust in the data. But today’s stack requires a range of tools to deliver on the self-serve promise — BI, product analytics and experimentation are often separate tools worth investing in for their tailored UIs.</p><h3>Tips for success in working with product teams</h3><p>With product teams becoming increasingly data-minded, and data teams becoming increasingly product-minded, the relationship between data and product disciplines can easily become complex or fraught.</p><p>But it doesn’t have to be — this is a pivotal partnership that can hum if you set product teams up for success with the right expectations on what data can bring to the table. Here’s how.</p><p><strong>Channel their energy </strong>— Product managers are rewarded for driving impact, and the path most traveled is through narrowly focusing the product-engineering team and using experimentation to deliver incremental product improvements.</p><p>The analyst or data scientists should become product’s expert partner, ensuring the quality of experiment design, flawless execution, and rigorous interpretation of experiment data. Analysts in these experiment-driven product teams commonly earn a seat at the table, as a strategic partner who will contribute hypotheses, weigh in on the product roadmap, and calibrate the team’s compass on when they should use data versus when they might rely on instincts or qualitative methods.</p><p>The experiments won’t all be ground-breaking, but momentum is a necessary attribute of a successful product team.</p><p><strong>Elevate the strategy with a curious data team </strong>— While it’s critical analysts become an expert partner in support of <em>each</em> product team’s goals, a unique strength of a strong, collaborative data team is their ability to observe overlaps, gaps and trade-offs across many product teams.</p><p>A collaborative team of analysts will see the overlap in metrics and reporting across disconnected product teams, and can build the analytical data product that no one realized they needed.</p><p>Or, they might observe a gap in the metrics used to understand user behavior, then research and implement a new set of metrics to better encapsulate the behaviors that drive the product strategy.For example, a data scientist with visibility across product teams might factor multiple outcomes into their targeting or recommendation algorithms, such as balancing short-term growth versus long-term engagement.</p><p>Cultivate a culture that encourages curiosity and critical thinking across the data team, such that analysts and data scientists can challenge the status quo and make an impact outside of the immediate direction of their product partners.</p><p><strong>There will always be tradeoffs</strong> — There is no free lunch. When you optimize for one behavior, by definition, you are discouraging others. The more ambitious the experiment, the more stakeholders will need to be at the table and the longer the trial.</p><p>Experiments on product home pages are often where these complexities emerge, and where you might expect to run a long experiment on a very small fraction of users (e.g. &lt;1%).</p><p>Understand the stakeholders and their objectives, define success criteria upfront (e.g. how will you value different outcomes related to browsing, navigating, and purchase) and expect to present a range of solutions that allow the product team to make trade-offs.</p><p><strong>Embed data team members for speed, but maintain a strong core — </strong>Product teams are data hungry, experimentation happy, and feel the need for speed. As a result, most data teams will find it helpful to leverage a decentralized approach and embed data analysts to increase proximity to the product teams, avoid bottlenecks and increase velocity.</p><p>However, I recommend only doing so once:</p><ul><li>The data platform is sufficiently mature, in terms of the tools &amp; processes that enable productive analytical work (and similarly, data science / ML work)</li><li>The data team has a sufficient bench to embed directors / managers as well as individual contributors, such that you can manage in a distributed model while continue to build a stronger core (or “<a href="https://medium.com/@shane.murray5/organizing-talent-return-of-the-data-center-of-excellence-3c6a2b4a3190">center of excellence</a>”)</li><li>The product organization has clear and distinct objectives, else your analysts’ work will overlap at best and be meaningless at worst</li></ul><p>Another challenge to consider is once you start embedding analysts across each product team, you shift the management of critical data products that enable those teams. You can inadvertently create a silo and miss the ability to produce analysis or data products that benefit multiple product teams.</p><p>There are synergies that can be captured with a shared platform and services. Questions may arise around chargeback models, the complexity of which is best avoided unless the situation on the ground forces your hand (widely divergent costs across teams).</p><p><strong>Enable self-serve clickstream data, but don’t expect a panacea — </strong>One of the key decisions data teams will need to make is how they treat their <a href="https://www.techtarget.com/searchcustomerexperience/definition/clickstream-analysis-clickstream-analytics">clickstream data</a>, which plays a pivotal role as connective tissue between disparate data sources to understand user behavior.</p><p>This decision is rarely an easy one — while customers of the data team often say they want access to raw data, they usually mean they want access to highly curated, trustworthy data delivered in a way they can easily manipulate.</p><p>Self-serve solutions (e.g. Amplitude, Mixpanel) provide many of the out-of-the-box visualizations and downloadable reports that your product customers desperately want, but will require you to adhere to their data model and result in significantly greater investment from the data team in instrumentation, governance, training and support for the chosen tool.</p><p>On the other hand, building up product-centered reporting on top of your warehouse, via BI tools or more custom-built data apps, may offer data teams the consistency and flexibility they want, but often requires development for each new metric, report or screen.</p><p>Whichever path you choose, it should ultimately be aimed towards better or faster product decisions. And so you should factor in both the benefit of broader access, and the shift in workload of the data team from adopting this solution. A good, lightweight implementation of self-service will free up your analysts to take on more meaningful work.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8r2SQJfR-bHX1JFa" /><figcaption>Image courtesy of Shane Murray.</figcaption></figure><p>The other defining characteristic to keep in mind regarding clickstream data is its fleeting value. Fundamental changes in products and markets make gleaning insights from trends in five year old clickstream data a near-worthless exercise. <a href="https://www.linkedin.com/feed/update/urn:li:activity:6997982529827811328?commentUrn=urn%3Ali%3Acomment%3A%28activity%3A6997982529827811328%2C6998001949992259584%29&amp;dashCommentUrn=urn%3Ali%3Afsd_comment%3A%286998001949992259584%2Curn%3Ali%3Aactivity%3A6997982529827811328%29">Some have suggested</a> the time decay can be expressed as aggressively as 1/x. The takeaway here is that you can often leave it behind in cold storage (if you keep it at all) during a migration or other data reorganization project.</p><h3>“Real artists ship”</h3><p>The famous Steve Jobs quote is an apt summary of an effective data-product team relationship. At its best, both teams are working collaboratively to ship new effective features and products guided by the best data driven decisions… and some artistic intuition.</p><p><a href="https://medium.com/@shane.murray5">Follow me</a> on Medium for more stories on data leadership, applications of data science, and related topics. <a href="https://medium.com/subscribe/@shane.murray5">Subscribe</a> to get my stories delivered to your inbox.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1fb84afdfb2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/meaningful-experimentation-5-impactful-data-projects-to-help-build-better-products-1fb84afdfb2">Meaningful Experimentation: 5 Impactful Data Projects To Help Build Better Products</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Data and Finance Teams Can Be Friends (And Stop Being Frenemies)]]></title>
            <link>https://medium.com/data-science/how-data-and-finance-teams-can-be-friends-and-stop-being-frenemies-7ecc357f51ef?source=rss-8aa0d9ae3ebd------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ecc357f51ef</guid>
            <category><![CDATA[notes-from-industry]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[business]]></category>
            <category><![CDATA[finance]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[shane murray]]></dc:creator>
            <pubDate>Tue, 29 Nov 2022 19:50:59 GMT</pubDate>
            <atom:updated>2022-11-29T19:50:59.558Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Part one in a practical data leader series</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*L2qfZz9cdZFiZ4FQ" /><figcaption><em>Photo credited to </em><a href="https://www.shutterstock.com/g/JUVART"><em>JUVART</em></a><em> via Shutterstock.</em></figcaption></figure><p>Recently I wrote an article about <a href="https://medium.com/towards-data-science/where-data-silos-live-in-your-organization-f51ecdd90809">data silos</a> that form across the organization, often due to lack of alignment with partners. This alignment can be difficult to come by, but is crucial to a data leader’s success. In fact, you could argue this alignment is the single most important task for any data scientist or engineer at the manager level and above.</p><p>With the range of internal customers to support, it can be tempting for data teams to inhabit the principles of an assembly line or even a fry cook at McDonalds. Everyone gets the same cheeseburger with exactly two pickles. Order up.</p><p>But the best data teams are able to inhabit the problem space of different partners (finance, product, marketing, etc), meeting them where they are with tailored data products that go beyond the generic, off-the-shelf solutions.</p><p>This is the first post in what is intended to be a series on best practices for working with each department, and all their wonderful eccentricities.</p><p>These challenges and friction points tend to transcend industries and verticals because while the data types or specific systems may vary, the primary goals of each department do not (and behavior is greatly influenced by incentives). At most companies for example, the marketing team will need to drive growth, the finance team will need to budget for it, and the data team will need to measure it.</p><p>We’ll start with the finance team, diving into how data teams can collaborate with them where priorities intersect. While this is written with midsized companies (approx. 1,000 to 5,000 employees) leveraging a modern data stack in mind, many of these projects and tips can be adjusted according to the specific needs of your organization.</p><h3>Projects that will win over your finance team</h3><p>Here I’ll focus predominantly on strategic finance or FP&amp;A projects such as blending financial and non-financial metrics to inform business growth; strategy; and investment. This is where I see the bulk of project collaborations with finance teams.</p><p>Financial accounting (e.g. tax, payroll), on the other hand, tends to exist in a set of systems largely outside of the data platform — and that’s OK.</p><h4><strong>Financial reporting</strong></h4><p>If you are a public company or approaching an IPO, chances are the data team will partner with finance to deliver quarterly reporting to Wall Street and accompanying data for earnings calls.</p><p>To enable this reporting, the data team will manage complex pipelines that transform transactional data from commerce systems into customer-level data detailing the specific customer relationship or products purchased, and the associated revenue.</p><p>The finance team will be focused on reliability above all else; no CFO wants to restate quarterly earnings. I’ve found this is one of the few places where accuracy can be precisely measured, using the general ledger as a source-of-truth. The finance team will high standards for accuracy as they are accountable up to the penny on their regular reporting — expect their standards for analytical data supporting their team to be no less.</p><p>How timely should the data be? Ideally the same data can be queried and used for financial analytics and reporting on a daily basis. But it’s best to prioritize accuracy over timeliness, potentially offering separate real-time views of sales that don’t require the same accuracy-to-the-penny.</p><p>By extension, the CFO will be focused on related data in the earnings report that may drive the business growth, such as the daily active users (DAUs), app downloads, or views. Accuracy is often more subjective here — relying on the definitions you give to “users,” “active,” and “views.” The definition must be clear and defensible, and critically consistent over time, so that you can be confident in period-over-period trends. In other words, the data should (at least) be directionally accurate.</p><p>It goes without saying that <a href="https://www.montecarlodata.com/blog-what-is-data-observability/">data observability</a> is critical for these metrics. You must proactively detect any shifts in the nature of the input data that could affect the integrity of your outputs. Turnaround times at the end of the quarter are tight, you don’t want to be discovering data issues as these reports are compiled and delivered.</p><h4><strong>Forecasts and growth-driver modeling</strong></h4><p>Finance teams live-and-breathe forecasts. Once data teams are supporting financial reporting, the next logical step is to partner on forecasts for future quarters — sales, new customers acquired, customer churn.</p><p>Data teams can provide expertise on the right modeling method, but you may find that simpler, empirical methods are preferred by your finance colleagues over a more precise but less explainable model, allowing them to more readily adjust assumptions, inputs and model parameters.</p><p>When a forecast is missed, you’ll want to be able to quickly diagnose the contributing reasons and produce the much-loved <a href="https://en.wikipedia.org/wiki/Waterfall_chart">waterfall chart</a>. The reason these charts are so effective is that they can communicate different types of uncertainty in the forecast, such as variance due to the environment, variance due to customer behavior, and unexplainable or random variance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/0*aiuQvREvaG4DJu-k" /><figcaption>An example of a waterfall chart from <a href="https://en.wikipedia.org/wiki/Waterfall_chart">Wikipedia</a>.</figcaption></figure><p>While forecasting is an operating mechanism, its strategic counterpart is the modeling of growth drivers. Data teams should take the lead on modeling exercises to explore the most important product and marketing levers that correlate with growth (i.e. customer acquisition &amp; retention).</p><p>While these models are historical in nature, when done well they will hint at causal relationships. For example, <a href="https://research.atspotify.com/2020/12/algorithmic-effects-on-the-diversity-of-consumption-on-spotify/">analysis may show that I’m more likely to retain my Spotify subscription if I listen to many different genres of music</a>, but is this a behavior Spotify can influence through interventions in the product experience to <em>increase my engagement and likelihood to retain</em>?</p><p>Pairing these models with subsequent product experiments can solidify this causal relationship. Using the example above, Spotify may experiment with the genre diversity of discovery algorithms in order to see whether customers like me increase their engagement and retention when algorithmic programming is more genre-diverse.</p><p>For the finance team, these models can be particularly useful in plotting out longer-term growth scenarios for the business, informing how to set targets for product and marketing metrics that ladder up to the growth targets.</p><h4><strong>Marketing Attribution and Mix</strong></h4><p>Typically, when it comes to questions of marketing spend, you’ll encounter a triumvirate of marketing (determine where and how to spend money), finance (determine the budget and profitability), and data (determine how to measure the ROI on spend).</p><p>Are data teams still attempting to deliver multi-touch attribution, or has this become a non-starter given the platform “walled gardens” and demise of the third-party cookie? I’ve seen data teams chase a holy grail of perfect attribution at the behest of executives, only to fall short due tracking limitations.</p><p><a href="https://en.wikipedia.org/wiki/Marketing_mix_modeling">Marketing mix modeling (MMM)</a> has always been my preferred approach to questions of spend, channel mix and ROI, using time-series models to deliver a holistic view of digital and non-digital marketing channels, accounting for incremental sales over-and-above the base that would be expected if marketing spend were removed from the equation.</p><p>Then, you can focus on individual customer attribution within each marketing channel in order to optimize the specific offers and messaging, without attempting to track customers across multiple external channels.</p><h4><strong>Pricing, Offers &amp; Lifetime Value (LTV)</strong></h4><p>Somewhere between the marketing and finance teams sits the crucial pricing decisions — what are the price-points for different products, what discounts can be profitably offered, and when do you increase prices on existing customers?</p><p>An early partnership might be around pricing experiments, moving these decisions from the realm of executive guessing to a series of pricing experiments, where success is measured beyond the initial sale to account for customer retention and lifetime value (LTV).</p><p>Typically, this LTV metric will be modeled by the finance team on specific campaigns or cohorts of customers, based on aggregated assumptions of customer retention. As you solidify the partnership, a high impact project for the data team is to move the measurement of LTV from the campaign or cohort level to the individual customer level, incorporating what you know about customer behavior that influences churn and upsell.</p><p>Then these LTV scores can live within the data warehouse and be readily applied to future experiments and analyses. Once they see the granularity with which you can measure their pricing strategies, finance teams will return with more hypotheses to be tested.</p><h3>Tips for success in working with finance teams</h3><p>The finance team can be one of the most strategic partners to the data team. The key is to understand their existential need for data accuracy, how this need affects project delivery expectations, how to best enrich data from their systems of record and, in doing so, how to partner with them on transformative data projects. Remember:</p><h4><strong>1. Accuracy and consistency are paramount</strong></h4><p>Financial metrics must meet a high standard of accuracy, and non-financial metrics must be consistent over time, to be useful in financial modeling and reporting.</p><p>Accuracy has long been one of the prized “dimensions of data quality” yet there are very few data products where you can state objectively that the data is accurate or inaccurate. Financial data is a rare exception, where your analytical reporting on revenue can be checked against the general ledger.</p><p>The key here is to set expectations around the use of the financial data product. If it is to be used for external reporting then chances are you need to be accurate to-the-penny of the general ledger over a specified time window. On the other hand, if you intend to use the financial data product for day-to-day analytics or modeling, then your finance partners may accept less precision in favor of more granular or timely data.</p><h4><strong>2. Requirements gathering is critical</strong></h4><p>You may need to bend your agile principles in favor of something closer to a waterfall approach, given that launch cannot happen until data reliability criteria are met.</p><p>While agile processes will include requirements gathering, I’ve found there is a degree of uncertainty in terms of what will be delivered at milestone dates. On the flip side, finance teams are held to stringent reporting windows, requiring a high degree of certainty in what data they are permitted to rely on for reporting.</p><p>Invariably, a new data product or change to financial reporting will need to run parallel to the existing process for a set period of time to build confidence in its accuracy and reliability.</p><p>For these reasons, you may still be able to run an agile process, but be ready to set clear timeline and deliverable expectations that align with the finance team’s reporting cadence.</p><h4><strong>3. Embedded is better than a hand-off</strong></h4><p>If you can embed a finance team member in your project, it will dramatically improve the translation of business rules and metric definitions.</p><p>It can be tempting for data and finance teams to sit in opposite corners of the floor, throwing requirements and solutions back-and-forth, complaining when the other team doesn’t deliver against expected milestones. You could sit for weeks diligently capturing detailed requirements for financial reporting only to miss some critical corner-case that results in a failure.</p><p>Not surprisingly, the better path is usually to embed the subject-matter-expert for the length of the project. You’ll see mutual benefits: the data engineers will be able to collaborate with an expert to define the business rules and expectations on the data, and the finance team will gain a greater appreciation for the technical complexity involved in building a reliable and trustworthy data product.</p><h4><strong>4. Bring data processing upstream</strong></h4><p>Excel may have (long ago) increased its row limit to 2²⁰, but you can handle much heavier workloads in your warehouse and move finance logic from a cohort-level to a customer-level.</p><p>Finance teams are known for building out multi-layered spreadsheets to support business operations, starting with granular data and embedding complex logic to deliver on their weekly reporting of business performance versus forecast.</p><p>Data teams can partner with finance to elaborate on the assumptions, inputs and outputs that enable you to run these computations upstream in the data warehouse, where you can store important business logic, pre-compute scenarios and begin to introduce more automation on the process.</p><p>In doing so, you’ll replace the spreadsheet with a configurable dashboard, taking the weight off your finance partner’s mind (and laptop), and introduce stronger quality control on the data pipeline in the process.</p><h4><strong>5. Empirical models may be preferred</strong></h4><p>Before you go and deliver a black-box model to the finance team to forecast churn, make sure it’ll be useful for them when they need to adjust assumptions week-in, week-out.</p><p>Data teams will often see a problem, such as forecasting churn, and rush to apply advanced techniques for modeling the likelihood that each customer will churn, based on tens or hundreds of predictors. But is that the problem that your finance partner wanted to solve? Will it meet their day-to-day needs?</p><p>If they are forecasting how many customers will churn each week, then you may be better off with a simpler model that applies observed or expected churn rates by tenure — and perhaps one or two other key predictors — such that the forecast can be easily managed and adapted by the finance team.</p><p>Keep these things in mind and you will be fast friends with the finance team.</p><p><a href="https://medium.com/@shane.murray5">Follow me</a> on Medium for more stories on data leadership, applications of data science, and related topics. <a href="https://medium.com/subscribe/@shane.murray5">Subscribe</a> to get my stories delivered to your inbox.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ecc357f51ef" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-data-and-finance-teams-can-be-friends-and-stop-being-frenemies-7ecc357f51ef">How Data and Finance Teams Can Be Friends (And Stop Being Frenemies)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Where Data Silos Live In Your Organization]]></title>
            <link>https://medium.com/data-science/where-data-silos-live-in-your-organization-f51ecdd90809?source=rss-8aa0d9ae3ebd------2</link>
            <guid isPermaLink="false">https://medium.com/p/f51ecdd90809</guid>
            <category><![CDATA[collaboration]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-platforms]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[customer-data-platform]]></category>
            <dc:creator><![CDATA[shane murray]]></dc:creator>
            <pubDate>Mon, 07 Nov 2022 21:52:25 GMT</pubDate>
            <atom:updated>2022-11-08T04:53:57.023Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>You’ve heard of shadow IT, but what about shadow data?</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*J8U-wlvSPg1AayMx" /><figcaption>Photo by <a href="https://unsplash.com/@stephencpedersen?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Stephen Pedersen</a> on <a href="https://unsplash.com/s/photos/silo?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Data silos and dark data: every organization has it—and by definition none of us know the full extent. Sometimes data hides from the scrutiny of the data team similar to <a href="https://en.wikipedia.org/wiki/Shadow_IT">shadow IT</a>.</p><p>Shadow data (or, data silos) arise when:</p><ul><li>Autonomy and speed is prioritized over technical standards;</li><li>Data access or resources are limited, forcing teams to work around existing systems and processes; or</li><li>Data consumers decide to deploy their own point-solutions rather than work with you.</li></ul><p>But instead of waving the white flag, consumers often find a way to get what they need outside of the watchful eye of the data team — and that’s risky.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OzeCqBPi-ES1a2mv" /><figcaption>If your data platform or product is too cumbersome, data consumers, and even members of your data team will find another path–and likely one that is less reliable and compliant. Photo by <a href="https://unsplash.com/@fairfilter?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Oliver Roos</a> on <a href="https://unsplash.com/s/photos/two-paths?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Data silos present numerous risks to data consumers and the data team alike, such as:</p><ul><li><strong>Fragility</strong>: I’d wager John on the finance analytics team doesn’t have systems like dependency management and anomaly detection in place for his data set. And when executives come to you to ask why this metric is wrong, it’s going to be hard to debug when you don’t have any idea how it’s produced. Either way this creates a risk that valuable business data may be lost.</li><li><strong>Knowledge loss</strong>: If a data “power user” leaves, you don’t want to have to figure out how to reverse engineer their esoteric system you have now inherited. Not only will this be a poor allocation of resources, but their system may create untold headaches before it’s discovered as an issue (one company saw millions in revenue impacted when one of their predictive algorithms was discovered to be running on autopilot following the departure of a team member).</li><li><strong>Security</strong>: “I didn’t know about it!” won’t prevent regulators from handing down a fine for improper handling of personal data. For GDPR, that fine can be up to 4% of the company’s revenue–ouch.</li><li><strong>Spotty Access</strong>: Analysts get grumpy, and rightly so, when they can’t get access to what they need because it can’t be joined to their canonical user tables. These data silos create either opacity or the need to duplicate processes–neither of which is a positive outcome.</li><li><strong>Blinkered decisions</strong>: While data consumers can move quickly with user-friendly point solutions, for complex decisions they may need an expert to think through experiment design, sampling bias and confounding factors. An entire department could be circling the wagons and marching towards a goal that won’t fundamentally add value to the business or that’s being measured in an arbitrary fashion.</li></ul><p>So how can data teams identify the data silos that lurk in the shadows? Can they use it as an opportunity to spot weaknesses within their data platform and bring consumers into the fold, either by encouragement or force?</p><p>In my opinion, the answer is yes. Let’s take a look at where the data silos are and how to break them.</p><h3>Data Silo #1: Transformations and Pre-aggregations</h3><p>But before we talk about data consumers, let’s not let ourselves as data professionals <em>completely</em> off the hook. We create our fair share of data silos in our quixotic attempt to meet the ever increasing demand for data while working within the realities of our team’s capacity.</p><p>Our company experienced an incident the other day that was a result of adding DataDog to our tech stack, which introduced some tracing information into the data structures flowing into our application.</p><p>When that data was persisted into S3 and then ingested into Databricks, it was only partially loaded so our pipelines had missing data.</p><p>Our post mortem concluded that our Spark jobs were overly complex with too many transformations. What we should have done as a best practice is to break the job up into a series of smaller transformations at key checkpoints that wrote to tables that could be monitored.</p><p>I’ve seen a similar scenario when dealing with old-school ETL and business intelligence implementations. Data was pre-aggregated for performance and the underlying layers were invisible to an analyst responsible for diagnosing surreptitious declines in key metrics. The analyst was lucky if they knew how the underlying data was transformed, let alone able to successfully find the cause of decline.</p><p>Some of this behavior shifted with the move from ETL to ELT. Now, these days, you are more likely to see data teams establish best practices around breaking up transformations into clear steps (e.g. events =&gt; sessions =&gt; users =&gt; campaigns) rather than writing three pages of SQL that only the owner can decipher.</p><p>While business intelligence tools will tout their data prep capabilities, this can wind up turning into another silo of business logic unavailable to teams outside of the tool. Clearly there are some benefits to last-mile manipulation and maybe rules like “no more than SELECT*” within BI are too draconian, but any reusable semantics must be widely available to your team.</p><p><strong><em>Solution</em></strong><em>: Break up complex transformations or SQL queries into distinct checkpoints that write out to tables that are monitored for data quality.</em></p><p><em>Ensure your business logic isn’t locked into a single tool only accessible to part of your user base. For example, you might de-silo by implementing a metrics layer that can serve business intelligence and analytics use cases; and a feature store that can be used across machine learning applications.</em></p><p><em>Coding languages can also naturally form silos and standardization should be considered wherever possible. The extreme form of this is being played out currently where Tesla engineers who code in Python are trying to integrate within Twitter which is predominantly coded in Scala.</em></p><h3>Data Silo #2: Spreadsheets</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VpiNGcSuTzYRTgJL" /><figcaption><a href="https://www.shutterstock.com/g/sasirin+pamai">sasirin pamai</a> via Shutterstock.</figcaption></figure><p>Spreadsheets remain the most successful way to democratize data in many organizations, even if they may be an occasional source of pain and derision for data teams.</p><p>Our partners in the finance department can do remarkable things with a PC and a VLOOKUP, and in my experience there are few better ways to democratize the prototyping of new data than by dropping a table into Google Sheets and collaborating with partners to manually append tags that are meaningful attributes for them to see in analysis.</p><p>A spreadsheet becomes a data silo when it moves from prototype to production.Or in other words, if you see the same spreadsheet playing a role in a business operation more than once, it’s time to move the logic upstream and create something more systemic.</p><p><strong><em>Solution:</em></strong><em> A hack for data teams is to periodically review the mammoth spreadsheets that are being used to guide the business or may be sitting in a shared drive, spreadsheets that likely have some gnarly formulas or macros sitting on top of data that’s supplied by your team according to their specifications.</em></p><p><em>This review will often reveal ways to address gaps in your data platform and can help you bring transformations upstream to increase observability and scalability. You may even unlock new opportunities to standardize metrics such as lifetime value (LTV) for wider application and produce them at a far more granular level.</em></p><p><em>Data integrity is all important when removing financial data silos. You will not be able to move spreadsheet logic upstream unless you can show finance partners that your systems are accurate to the penny, up-to-date and available when they need to deliver business critical reporting.</em></p><h3>Data Silo #3: The “all-in-one” solution — ESP, CDP, DMP, A/B</h3><p>When done wrong, these can be the hardest silos to reign in.</p><p>Much of the <a href="https://lumapartners.com/content/lumascapes/martech-lumascape/">marketing tech stack</a>, like email service providers and marketing automation platforms, rose to prominence ahead of the advances in modern data warehouses. This meant the most expedient way to drive impact was with an all-in-one solution that collected, managed and served data directly to the marketer. Admittedly, I was once selling the dream of a “single line of javascript” for running experiments that allows you to avoid any further conversations with “IT.”</p><p>Some of the most pressing problems created by this silo are that these systems quickly experience an ungoverned sprawl of customer segments (product page repeat visitors oct 2020) and the segmented campaigns often suffer from lack of measurement.</p><p>Back in my consulting days I assessed that one large telco was “micro-targeting”<strong> less than 1% of their customer base</strong> because they were unknowingly filtering campaigns down to customers that satisfied <strong>every</strong> attribute within their complex segmentation.</p><p>But we turned a corner a few years ago — data teams and technologies are now capable of being fast and flexible enough to operate at the speed-of-business while also unlocking new opportunities with the richness of data in the warehouse (“hey, want to optimize for LTV instead of click-through?”). Marketers and other business partners see the value in going through, rather than around, the data team.</p><p>These marketer-first solutions are adapting. Whether you opt for a modern CDP or reverse ETL to transport data into the hands of marketers, the must-have feature is to have collection, transformation and basic segmentation operating on your enterprise data warehouse.</p><p><strong><em>Solution</em></strong><em>: I’ve found the best way to get marketing teams onboard is to create systems that respect and address their need for speed and autonomy, while partnering to ensure strong governance and measurement are part of the solution.</em></p><p><em>Not only do you need to give them a better tool, at some point you need to decommission the siloed system even if there will be resistance. Before you reach this point, walk a mile in marketing’s shoes. Partner on a campaign launch and use that experience to show you can automate and optimize the process, while delivering higher return on the campaign.</em></p><p><em>Unfortunately, data silos can be inherently political. A marketing automation tool may be funded from marketing budget, of which there is plenty, while a data warehouse will come from the data team’s comparatively smaller budget. Build the business case, but also build consensus with operators and executives.</em></p><h3>My recommendation? Proactively remove silos</h3><p>It can be tempting to let sleeping dogs lie but, in my opinion, data teams should proactively work to remove data silos or any “shadow data” systems.</p><p>I’ve typically opted for the Field of Dreams approach — “if you build it, they will come.” (The approach is probably more akin to gather requirements, scope the project, get approvals, build a minimum viable product, get feedback, iterate and they will come–but that is not nearly as pithy).</p><p>But if you build it and they don’t come, then you need to address whether that’s a failure of your technology solution, a lack of organizational buy-in, or something else altogether. Then you need to find a solution that puts you on the right path to breaking down silos.</p><p>After all, data is ultimately the data team’s responsibility, and consumers will do what they need to do to access it. Our best path forward as data leaders is to accept this reality and take steps to mitigate it.</p><p><a href="https://medium.com/@shane.murray5">Follow me</a> on Medium for more stories on data leadership, applications of data science, and related topics. <a href="https://medium.com/subscribe/@shane.murray5">Subscribe</a> to get my stories delivered to your inbox.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f51ecdd90809" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/where-data-silos-live-in-your-organization-f51ecdd90809">Where Data Silos Live In Your Organization</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>