<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ziqi Ma on Medium]]></title>
        <description><![CDATA[Stories by Ziqi Ma on Medium]]></description>
        <link>https://medium.com/@ziqima?source=rss-22c5dfb9cfdc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*rRlucsj5AranDZamn42kfA.jpeg</url>
            <title>Stories by Ziqi Ma on Medium</title>
            <link>https://medium.com/@ziqima?source=rss-22c5dfb9cfdc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 21:34:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ziqima/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Many facets of un-truth: LLM hallucination 101]]></title>
            <link>https://medium.com/data-science-at-microsoft/many-facets-of-un-truth-llm-hallucination-101-8273806fa145?source=rss-22c5dfb9cfdc------2</link>
            <guid isPermaLink="false">https://medium.com/p/8273806fa145</guid>
            <category><![CDATA[generative-ai-solution]]></category>
            <category><![CDATA[ai-hallucination]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Ziqi Ma]]></dc:creator>
            <pubDate>Tue, 05 Sep 2023 07:18:02 GMT</pubDate>
            <atom:updated>2023-09-05T07:18:02.291Z</atom:updated>
            <content:encoded><![CDATA[<p>With the increased adoption of LLMs across industries, <em>hallucination</em> has become one of the hottest related buzzwords — after all, when one embeds LLMs into real scenarios with real consequences, one of the core requirements is that the model should not provide false information. This article explores the definitions of hallucination, discusses its evaluation both in a closed-domain and an open-domain setting, highlights the trickiness related to obtaining ground truth, and finally presents some mitigation solutions.</p><p>Throughout the text of this article, references are made as annotations with numbers in brackets to the equivalent numbered sources that are listed in the References section at the end of the article, which readers can consult for deeper background information.</p><h3><strong>Defining hallucination</strong></h3><p>Albeit widely used, <em>hallucination</em> is a loosely defined term. Based on existing definitions of hallucination in natural language generation, this article proposes the following framework, as seen in Figure 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nEdBZlsi19CGGYwOCAJXTQ.png" /><figcaption>Figure 1: Categorization of hallucination.</figcaption></figure><p>Hallucination can first be divided into closed-domain and open-domain types as in [1].</p><ul><li><strong><em>Closed-domain hallucination</em></strong> can be defined as generating content that cannot be fully verified from some provided reference, regardless of whether the content is in fact true. In the words of [2], hallucination in this setting simply means “unfaithful,” regardless of factuality. The commonly used Retrieval-Augmented Generation (RAG) setting for LLM applications naturally follows this definition for hallucination. This definition is used in [2] and [3]. Within closed-domain, hallucination can be further divided into intrinsic and extrinsic, as in [2], [4]: <em>intrinsic hallucination</em> can be proved faulty by the source, whereas <em>extrinsic hallucination</em> cannot be easily verified by the source.</li><li><strong><em>Open-domain hallucination</em></strong> is harder to define. [1] uses hallucination interchangeably with “erroneous generation,” yet calling all errors “hallucination” seems too broad. [5] defines hallucination as the opposite of factuality, which makes sense in the presented scenario of objective fact verification, yet does not extend well to scenarios where authoritative truth is harder to evaluate.</li></ul><p>When it is possible to retrieve consensual reference related to the generated content, open-domain hallucination simply reduces to closed-domain hallucination. When it is not possible to retrieve any reference related to the generated content, we can claim it to be hallucination. The trickier situation lies in cases where we might retrieve various references that are conflicting. Is the content untrue according to the ground truth, majority of agreeing evidence, its training data, or something else? In practice, in the open setting, people generally care about specific aspects of hallucination, such as adversarial robustness (i.e., cannot be prompted to lie), misconception correction, or common-sense correctness, which we discuss more below in the evaluation sections.</p><h4><strong>Evaluation of closed-domain hallucination</strong></h4><p>Although hallucination is generally discussed in the generative context, closed-domain evaluation traces back to one of the earliest classical tasks of natural language processing, known as Natural Language Inference (NLI). NLI evaluates whether a hypothesis is true (entailment), false (contradiction), or undetermined (neutral) given a premise. While NLI in computational systems has been studied since around the early 2000s, SNLI [6] (2015) was the first large-scale (570K) dataset that is still widely used today.</p><p>Both the premise and the hypothesis are one-sentence long and logically simple. Tasks such as question answering (QA) and summarization provide richer, more complex settings. In question answering, the canonical setting such as SQuAD [7] contains a long context, though the question is multiple-choice focused on one topic. Longer-form question answering has also been studied, such as in [8], which notes the lower accuracy (based on support document) of abstractive models versus extractive models. This can be seen as an early form of closed-domain hallucination.</p><p>Similar to long-form QA, summarization tasks provide relatively long and complex context as well as content to be evaluated. More recent work such as [9] and [10] combine various QA and summarization sources and focus explicitly on longer-form NLI. FEVER [11], a Wikipedia-based verification benchmark proposed in 2018, has since grown to a workshop spanning domains such as biomedical and science.</p><p>There are various solutions toward hallucination evaluation for LLMs, many of which use LLMs. For example, LLM-based hallucination evaluation is used in Anthropic [12], LangChain [13], and Vicuna [14]. Azure Machine Learning’s PromptFlow [15] also provides LLM-based closed-domain evaluation. Due to the length of generated content by today’s LLMs, evaluation is sometimes done at a more granular level, such as in [16].</p><h4><strong>Evaluation of open-domain hallucination</strong></h4><p>I consult the rich literature in fact verification for open-domain hallucination evaluation. Because open-domain hallucination is very broad, evaluation in various tasks generally focuses on “truthfulness” in a task-specific aspect such as knowledge conflict, misconception correction, or common sense.</p><p>For example, among the eight tasks in the “truthfulness” category of the BIG-bench benchmark [17], [18] explores knowledge conflict: when the provided evidence might contradict with the model’s prior knowledge acquired during training. On the other hand, [19] focuses on imitative falsehoods that are based on misconceptions seen in training data. [20] measures whether models can be prompted to convincingly deliver false information. Works such as [21] and [22] focus on common-sense correctness evaluation. I expect this list to keep growing, as an increasing number of LLM applications call attention to different aspects of hallucination.</p><h4><strong>A caveat about ground truth</strong></h4><p>There is one important caveat I would like to highlight regarding hallucination evaluation, which is the difficulty in obtaining “ground truth.” The heavy use of crowdsource annotators, plus the inherent ambiguity related to long and complex text, results in less-than-ideal agreement among the annotators.</p><p>To quantify the degree of consensus, Fleiss’s kappa, <a href="https://en.wikipedia.org/wiki/Fleiss%27_kappa">which measures actual agreement over possible agreement above random chance</a>, is oftentimes reported. With 1 being perfect agreement and 0 being no agreement, works such as FEVER [23] report 0.68 and the simpler SNLI [6] achieves 0.70. These measures suggest that we should sometimes take human-labeled truth with a grain of salt, especially when the labels are crowdsourced. Furthermore, as shown in the figure below, as language tasks grow in complexity, agreement becomes harder. In the 2015 SNLI task, three-annotator agreement is as high as 98 percent (among five annotators), whereas in the 2022 OpenAI summarization task, the average inter-annotation agreement is only 73 percent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-uDcfeN7ZCa0zARHlCLzGQ.png" /><figcaption>Figure 2: Inter-annotator agreement becomes harder to obtain as language tasks become more complex.</figcaption></figure><p>We note that this comparison is not to be taken literally, as the summarization task adopts a 1–7 scale whereas SNLI is binary. Nevertheless, the general range of the percentage still testifies to the difficulty in obtaining ground truth as the tasks grow more complex.</p><p>Various works explore the challenge related to agreement on “ground truth” and propose some solutions. [24] explores the tradeoff between cost, efficiency, diversity, and data quality. Solutions related to obtaining “ground truth” come in three flavors:</p><ol><li><strong>Better annotation protocol:</strong> In the non-LLM context, there have been investigations into various annotation protocols for NLI tasks, such as [25] and [26]. In the LLM context, comparison-based annotation is used widely for reinforcement learning to improve alignment, such as that for OpenAI [27] and Anthropic [28].</li><li><strong>Better tooling to enable efficient human annotation at scale: </strong>[28] focuses on its annotator experience as an important element of their alignment solution.</li><li><strong>Model-based annotation: </strong>Although letting AI evaluate AI’s truthfulness sounds like circular logic, it is helpful in various scenarios. For example, evaluation might require domain-specific knowledge that crowdsource workers do not have. Online evaluations of deployed models need to happen in real time and cannot wait for a human annotator. Even in scenarios where human annotations are viable, the high inter-annotator variance and cost of human annotation are a strong motivating factor for using a model-based annotation. For example, Anthropic leverages AI annotation to better scale up supervision [29].</li></ol><h3><strong>Mitigating hallucination</strong></h3><p>This section starts by presenting some mitigation techniques specific to closed-domain and open-domain hallucination, respectively, and then discusses calibration as it applies to both cases.</p><p>In the closed setting, if the task is Retrieval-Augmented Generation, sometimes better retrieval can help reduce hallucination. One could also prompt the model to be more faithful to the reference while generating content, such as asking it to provide explicit citation. Post-hoc refinement is also widely used, such as self-refine [30], a prompt-only method that iteratively generates feedback and new content based on feedback. Refiner [31] uses a similar iterative approach but imposes some structure by breaking down error types into a few categories, and co-trains the generator and critic models. Such refinement methods might address the inherent limitation of autoregressive generation, which does not enable global understanding or backtracking, as suggested by [1]. Along a different direction, [32] learns a (potentially smaller) corrector which re-weighs the output distribution produced by the generator.</p><p>Open-domain hallucination could be reduced with retrieval. [33] is a good example that enhances LLMs with a knowledge consolidator that is able to query APIs to retrieve external information such as reddit posts or Yelp reviews, and de-noises retrieved content for relevance. Leveraging external information enables a significant reduction of hallucination in their experiments. Similarly, [34] retrieves live Wikipedia information to fact-check and refine content generation.</p><p>Regardless of closed-domain or open-domain, another solution is calibration — knowing when to abstain from answering when the model is uncertain. Reinforcement Learning from Human Feedback (RLHF), initially introduced in [27], is widely used to nudge the model against hallucinating about what it doesn’t know, as seen in work from Anthropic [35] and OpenAI [36].</p><h3><strong>Conclusion</strong></h3><p>I hope to have shared some general context related to LLM hallucination, including its various definitions, evaluations in different settings, the main challenge of obtaining ground truth, and mitigation. While closed-domain hallucination evaluation has been studied the longest, people have also come up with methods to evaluate open-domain hallucination, mainly focusing on task-specific aspects such as new evidence contradicting prior knowledge acquired during training, misconceptions present in training data, or even adversarial prompting to let the LLM output false information.</p><p>I have also discussed the inherent difficulty in evaluating hallucination due to annotator disagreements. For practitioners who implement custom labeling, investment in tooling and protocols to evaluate and facilitate agreement might be worthwhile. I end by discussing hallucination mitigation, including self-refinement or model-based correction in closed-domain, retrieval in open-domain, and calibration via RLHF or AI feedback for both closed- and open-domain. As LLMs are adopted in an increasing number of scenarios, the hallucination problem is no longer only academic. Innovation by practitioners is needed in every embedded solution, so that as a society we can leverage the power of AI responsibly.</p><p><em>Ziqi Ma is on </em><a href="https://www.linkedin.com/in/ziqi-ma-491b47164/"><em>LinkedIn</em></a>.</p><h3><strong>Acknowledgments</strong></h3><p>Thanks to Jue Zhang and Harsha Nori for feedback on the draft.</p><h3><strong>References</strong></h3><p>[1] “Sparks of Artificial General Intelligence: Early experiments with GPT-4,” Sébastien Bubeck et al., April 13, 2023. <a href="https://arxiv.org/pdf/2303.12712.pdf">https://arxiv.org/pdf/2303.12712.pdf</a></p><p>[2] “Survey of Hallucination in Natural Language Generation,” Ziwei Ji et al., November 7, 2022. <a href="https://arxiv.org/pdf/2202.03629.pdf">https://arxiv.org/pdf/2202.03629.pdf</a></p><p>[3] “On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?” Nouha Dziri et al., <em>Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies</em>, pages 5271–5285, July 10–15, 2022. <a href="https://aclanthology.org/2022.naacl-main.387.pdf">https://aclanthology.org/2022.naacl-main.387.pdf</a></p><p>[4] “Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Grounding,” Nouha Dziri et al., September 14, 2021. <a href="https://arxiv.org/pdf/2104.08455.pdf">https://arxiv.org/pdf/2104.08455.pdf</a></p><p>[5] “WikiChat: A Few-Shot LLM-Based Chatbot Grounded with Wikipedia,” Sina J. Semnani, May 23, 2023. <a href="https://arxiv.org/pdf/2305.14292.pdf">https://arxiv.org/pdf/2305.14292.pdf</a></p><p>[6] “A large annotated corpus for learning natural language inference,” Samuel R. Bowman et al., August 21, 2015. <a href="https://arxiv.org/pdf/1508.05326.pdf">https://arxiv.org/pdf/1508.05326.pdf</a></p><p>[7] SQuAD2.0: The Stanford Question Answering Dataset (online). <a href="https://rajpurkar.github.io/SQuAD-explorer/">https://rajpurkar.github.io/SQuAD-explorer/</a></p><p>[8] “ELI5: Long Form Question Answering,” Angela Fan et al., July 22, 2019. <a href="https://arxiv.org/abs/1907.09190">https://arxiv.org/abs/1907.09190</a></p><p>[9] “DOCNLI: A Large-scale Dataset for Document-level Natural Language Inference,” Wenpeng Yin, Dragomir Radev, and Caiming Xiong, June 17, 2021. <a href="https://arxiv.org/pdf/2106.09449.pdf">https://arxiv.org/pdf/2106.09449.pdf</a></p><p>[10] “Looking Beyond Sentence-Level Natural Language Inference for Question Answering and Text Summarization,” Anshuman Mishra et al., <em>Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies</em>, pages 1322–1336, June 6–11, 2021. <a href="https://aclanthology.org/2021.naacl-main.104.pdf">https://aclanthology.org/2021.naacl-main.104.pdf</a></p><p>[11] The Sixth FEVER Workshop (online), May 5, 2023. <a href="https://fever.ai/">https://fever.ai/</a></p><p>[12] “Constitutional AI: Harmlessness from AI Feedback,” Yuntao Bai et al., December 15, 2022. <a href="https://arxiv.org/pdf/2212.08073.pdf">https://arxiv.org/pdf/2212.08073.pdf</a></p><p>[13] Auto-Evaluator (online). <a href="https://autoevaluator.langchain.com/">https://autoevaluator.langchain.com/</a></p><p>[14] Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90% ChatGPT Quality (online), The Vicuna Team, March 30, 2023. <a href="https://lmsys.org/blog/2023-03-30-vicuna/">https://lmsys.org/blog/2023-03-30-vicuna/</a></p><p>[15] “What is Azure Machine Learning prompt flow (preview),” Microsoft Learn (online), July 2, 2023. <a href="https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/overview-what-is-prompt-flow?view=azureml-api-2">https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/overview-what-is-prompt-flow?view=azureml-api-2</a></p><p>[16] “FACTSCORE: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation,” Sewon Min et al., May 23, 2023. <a href="https://arxiv.org/pdf/2305.14251.pdf">https://arxiv.org/pdf/2305.14251.pdf</a></p><p>[17] BIG-bench: Beyond the Imitation Game collaborative benchmark for measuring and extrapolating the capabilities of language models, online, bigbench@googlegroups.com. <a href="https://github.com/google/BIG-bench/tree/main">https://github.com/google/BIG-bench/tree/main</a></p><p>[18] “Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence,” Tal Schuster, Adam Fisch, and Regina Barzilay, <em>Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies</em> (online), June 2021. <a href="https://aclanthology.org/2021.naacl-main.52/">https://aclanthology.org/2021.naacl-main.52/</a></p><p>[19] Google / BIG-bench (online) Cedrick Argueta, Vinay Ramasesh, and Jaime Fernández Fisac. <a href="https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/convinceme">https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/convinceme</a></p><p>[20] Google / BIG-bench (online) Cedrick Argueta, Vinay Ramasesh, and Jaime Fernández Fisac. <a href="https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/convinceme">https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/convinceme</a></p><p>[21] “COM2SENSE: A Commonsense Reasoning Benchmark with Complementary Sentences,” Shikhar Singh et al., August 2021. <a href="https://aclanthology.org/2021.findings-acl.78/">https://aclanthology.org/2021.findings-acl.78/</a></p><p>[22] “Vera: A General-Purpose Plausibility Estimation Model for Commonsense Statements,” Jiacheng Liu et al., May 2023. <a href="https://arxiv.org/abs/2305.03695">https://arxiv.org/abs/2305.03695</a></p><p>[23] “FEVER: a large-scale dataset for Fact Extraction and VERification,” James Thorne et al., March 2018. <a href="https://arxiv.org/pdf/1803.05355.pdf">https://arxiv.org/pdf/1803.05355.pdf</a></p><p>[24] “Clean or Annotate: How to Spend a Limited Data Collection Budget,” Derek Chen, Zhou Yu, and Samuel R. Bowman, July 2022. <a href="https://aclanthology.org/2022.deeplo-1.17/">https://aclanthology.org/2022.deeplo-1.17/</a></p><p>[25] “New Protocols and Negative Results for Textual Entailment Data Collection,” Samuel R. Bowman et al., April 2004. <a href="https://arxiv.org/pdf/2004.11997.pdf">https://arxiv.org/pdf/2004.11997.pdf</a></p><p>[26] “Asking Crowdworkers to Write Entailment Examples: The Best of Bad Options,” Clara Vania, Ruijie Chen, and Samuel R. Bowman, <em>Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing</em>, pages 672–686, December 4–7, 2020. <a href="https://aclanthology.org/2020.aacl-main.68.pdf">https://aclanthology.org/2020.aacl-main.68.pdf</a>.</p><p>[27] “Deep Reinforcement Learning from Human Preferences,” Paul F Christiano et al., June 2017. <a href="https://arxiv.org/pdf/1706.03741.pdf">https://arxiv.org/pdf/1706.03741.pdf</a></p><p>[28] “A General Language Assistant as a Laboratory for Alignment,” Amanda Askell et al., December 2021. <a href="https://arxiv.org/pdf/2112.00861.pdf">https://arxiv.org/pdf/2112.00861.pdf</a></p><p>[29] “Constitutional AI: Harmlessness from AI Feedback,” Yuntao Bai et al., December 2022. <a href="https://arxiv.org/pdf/2212.08073.pdf">https://arxiv.org/pdf/2212.08073.pdf</a></p><p>[30] “Self-Refine: Iterative Refinement with Self-Feedback,” Aman Madaan et al., March 2023. <a href="https://arxiv.org/abs/2303.17651">https://arxiv.org/abs/2303.17651</a></p><p>[31] “REFINER: Reasoning Feedback on Intermediate Representations,” Debjit Paul et al., April 2023. <a href="https://arxiv.org/abs/2304.01904">https://arxiv.org/abs/2304.01904</a></p><p>[32] “Generating Sequences by Learning to Self-Correct,” Sean Welleck et al., November 2022. <a href="https://arxiv.org/abs/2211.00053">https://arxiv.org/abs/2211.00053</a></p><p>[33] “Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback,” Baolin Peng et al., February 2023. <a href="https://arxiv.org/pdf/2302.12813.pdf">https://arxiv.org/pdf/2302.12813.pdf</a></p><p>[34] “WikiChat: A Few-Shot LLM-Based Chatbot Grounded with Wikipedia,” Sina J. Semnani, Violet Z. Yao, Heidi C. Zhang, and Monica S. Lam, May 2023. <a href="https://arxiv.org/pdf/2305.14292.pdf">https://arxiv.org/pdf/2305.14292.pdf</a></p><p>[35] “Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback,” Yuntao Bai, Andy Jones, Kamal Ndousse et al., April 2024. <a href="https://arxiv.org/pdf/2204.05862.pdf">https://arxiv.org/pdf/2204.05862.pdf</a></p><p>[36] “Training language models to follow instructions with human feedback,” Long Ouyang et al., March 2023. <a href="https://arxiv.org/pdf/2203.02155.pdf">https://arxiv.org/pdf/2203.02155.pdf</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8273806fa145" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-at-microsoft/many-facets-of-un-truth-llm-hallucination-101-8273806fa145">Many facets of un-truth: LLM hallucination 101</a> was originally published in <a href="https://medium.com/data-science-at-microsoft">Data Science + AI at Microsoft</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reasoning about change: Lessons learned from building a near real-time system for Azure pricing]]></title>
            <link>https://medium.com/data-science-at-microsoft/reasoning-about-change-lessons-learned-from-building-a-near-real-time-system-for-azure-pricing-34049816ffbd?source=rss-22c5dfb9cfdc------2</link>
            <guid isPermaLink="false">https://medium.com/p/34049816ffbd</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[near-real-time]]></category>
            <category><![CDATA[microsoft]]></category>
            <category><![CDATA[streaming]]></category>
            <category><![CDATA[real-time-analytics]]></category>
            <dc:creator><![CDATA[Ziqi Ma]]></dc:creator>
            <pubDate>Tue, 25 Apr 2023 07:17:45 GMT</pubDate>
            <atom:updated>2023-04-25T07:17:45.236Z</atom:updated>
            <content:encoded><![CDATA[<p>Traditionally, when we think about data, we think about states — static measurements or records of phenomena that we care about, in a format that is easy to study and analyze. But data is never truly static because the world is not static. Instead of focusing on snapshot after snapshot, however, can we shift our focus to change?</p><p>This is what streaming systems are all about. Also known as <em>event stream processing</em>, streaming data is the continuous flow of data generated by various sources. Some people talk about streaming data as “data in motion.” Today, as “time-to-insight” has become a widely used metric in the data science world, adoption of streaming systems has been steadily increasing. In this article I reflect on a few lessons I’ve learned during the batch-to-streaming transformation of Azure’s pricing system — a few aspects to consider when shifting from “reasoning about state” to “reasoning about change.”</p><h3><strong>Problem overview</strong></h3><p>Prices change all the time. There are three big sources of change — product (e.g., Microsoft or an Independent Software Vendor on Azure changes the price of a meter or the SKU of a product), modifiers (e.g., a new deal is negotiated, specifying a discount for given customers and products, perhaps for certain regions or markets), and customers (e.g., a customer creating a new billing group or acquiring a new deal). Across all of Azure’s customers and products, the volume of change data is large, and new prices are constantly being generated. The challenge lies in surfacing the correct price for a query as close to real-time as possible via a streaming system. Below I delve a bit deeper into various aspects of this problem.</p><h3><strong>Three dimensions of time</strong></h3><p>When talking about streaming, people refer mostly to streaming along two dimensions of time: event time (when the event actually happened) and processing time (when the event entered the system), the latter of which lags behind event time by an unknown amount. Many of the intricacies of streaming engines are related to resolving these two dimensions — for example, watermarks that help us decide what is an acceptable lateness/out-of-order boundary. Canonical examples like clickstream correlation correlate events only within a certain boundary in event time — but because the processor knows only about processing time, the engine needs to keep some state under the hood for processing to happen correctly.</p><p>Pricing, however, is quite different in this regard. Pricing has its own inherent time dimension — when a price starts to be effective, and when a deal starts and ends. In this sense, “event time” does not make much sense — even if a vendor entered the price today, for example, it might be effective only starting next month. We call this the third dimension of time “effective time.” Because of this, all the smart mechanisms we get “out of the box” from streaming engines (e.g., buffered joins) no longer work. With this third dimension, watermarking no longer makes sense — what would a “lag” mean if effective time is an arbitrary value based on user input? In a more practical sense, we cannot define rules for non–inner-joins between two streams — in case of no match, how long does stream 1 wait until emitting a null record? If there is a window function, when do we emit the (perpetually partial) result?</p><p>As long as we have a set “expiration time” or “max lag time” of, for example, one day, a streaming engine could safely ignore any record prior to one day ago from the latest received event time. But this relies on the assumption that event time increases monotonically. The effective time, however, need not be monotonic, thus breaking the framework. Furthermore, it is possible to re-price something in the past, further breaking the monotonicity of time. The notion of “snapshot” does not actually exist in our system.</p><h3><strong>Bounded state</strong></h3><p>All the questions above are closely related to state keeping — and setting an expiration time is one way to make sure the state does not grow indefinitely. We look at two types of state management: management by the engine and management by the user.</p><h4><strong>State management by the engine</strong></h4><p>We look at engine-controlled state management in the simple example of a stream-stream outer join. Assume we are using an outer join with two streams, A and B (e.g., in Spark Streaming), and there are three watermarks that affect state keeping and expiration:</p><ol><li>The watermark of A, i.e., in either event time or processing time, allowed lateness of any record in stream A. Assume this is 1 minute.</li><li>The watermark of B, i.e., in either event time or processing time, allowed lateness of any record in stream B. Assume this is 3 minutes.</li><li>The join window, i.e., in either event time or processing time, allowed lateness (or earliness, expressed as a negative value) of any record in stream A relative to a record in stream B with the same key. Assume this is [-4 minutes, 5 minutes].</li></ol><p>In this case, how should the engine keep state? Let’s say we use only event time in watermark calculations (using processing time is analogous). The two questions are: 1.) How long should we buffer a record from A received at event time t0? and 2.) How long should we buffer a record from B received at event time t0? For the first question, see the figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NpvZ27a9zdGK1GLIzLyKNg.png" /></figure><p>The worst case is that the A event is the earliest possible (when stream A first reaches t0 in event time), then it joins with a B event with event time t0+4min that comes at the latest possible time (when stream B reaches t0+4min+3min in event time). This means that any event received from A needs to be buffered until stream B crosses event time t0+7min.</p><p>Similarly, for the second question, see the figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L_5gFdCb6u4BMNh0n0zhhw.png" /></figure><p>The worst case is that the B event is the earliest possible (when B first reaches t0 in event time), then joins with an A event with event time t0+5min that comes at the latest time possible when stream A reaches t0+5min+1min in event time. This means that any event received from stream B must be buffered until stream A crosses event time t0+6min. Because this is an outer join, for any unmatched key we must wait until the end of this window to know whether its counterpart is indeed “null” or its counterpart simply arrives late. This means that all unmatched records must be emitted a few minutes late.</p><p>The good news is that all the calculations above are automatically handled by the engine.</p><h4><strong>State management by the user</strong></h4><p>The example above represents the most comprehensive state keeping — every single record is kept. Sometimes the user might want to keep a leaner state — some self-defined data structure that “summarizes” the records seen before, such as some aggregation in a self-defined grouping. In such cases, the windowing and expiration provided by the engine are sometimes helpful, sometimes hindering. Because of this, most engines or frameworks open up flexibility for the user to manage an arbitrary state, known as “stateful streaming.” State management, however, comes with its own set of challenges.</p><p>Scaling state is not impossible, but it is tricky, because there are a few requirements: First and foremost, we want state to be local so that no network call is required when fetching state; and second, we want the state to be recoverable fairly quickly if a node crashes. Scaling becomes harder given these two constraints. Based on these requirements, some streaming engines use an embedded database solution like RocksDB/LevelDB. Without the automatic window expiration, users also must implement their own garbage collection so that the state does not grow indefinitely.</p><p>The table below shows a comparison of various popular open source streaming engines or frameworks regarding stateful streaming.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*av98-nKbE8vK_o7X9aIfQQ.png" /></figure><p>Again, pricing does not fit easily into stateful streaming. The reasons are twofold: 1.) We might want to join with events that happened at an arbitrary time in history due to arbitrarily late data; and 2.) By design, we want to be able to selectively restate historical data (and expect upstream to do the same). Both scenarios require us to somehow store all records in the state, which would be too large for an embedded solution.</p><h3><strong>Our solution: Taking ordering out of the system</strong></h3><p>Because we cannot cast pricing into a canonical “bounded out-of-order” streaming problem, our solution is to take ordering entirely out of the system. In its place, we established our own versioning strategy, as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-egnfQSVMskqwW_uacR1eQ.png" /></figure><p>We impose our own versioning system that overrides the time of arrival at any system. Versioning happens at any level of the system and is accumulated in base 1000 (or any large number) and shown to downstream. Internally, any change of logic or reprocessing of historical data increments the processor version (we assume upstream to do the same when sending an event to our system).</p><p>To illustrate the example shown above: From the event sources we obtain an entity that upstream sources stamp version 5 — a unique version for a given effective time. Processor 0 is running on version 3. From the perspective of processor 0, the output version is its upstream version (5) + 1000 * its own version (3) = 3005. This is what processor 1 will regard as the “upstream version.” Similarly, processor 1, which runs on version 2, produces a version based on its upstream version (3005) + 1000² * its own version (2) = 2003005. This is the ultimate (unique) version we emit downstream. In this way we send downstream an “accumulated” version of upstream, as well as all our internal processors. In this example, version 2003006 would mean upstream restated the entity but our processors did not change. Version 2004005 would mean a restatement due to updating processor 0.</p><p>From the perspective of any downstream consumer, for any given “effective time” (the time specified by input, not related to the arrival time at any part of the system), the highest version is authoritative. We also rely on this versioning (combined with IDs and effective time) for idempotency — systems like Spark Streaming provide only an at-least-once guarantee (i.e., it’s possible to deliver duplicate messages upon failover). Because of this, it is important for the application code to provide idempotency so that processing the same message more than once does not cause any unintended system behavior.</p><h3><strong>Conclusions and further reading</strong></h3><p>Stream processing has been around for well over a decade. Given the abundance of examples online, setting up a streaming system is not hard. However, there is no “one-size-fits-all” solution. To build a system that serves your application well, it is important to understand system-specific requirements and constraints. A few books delve a bit deeper into the design decisions behind some widely-used systems, including <a href="https://www.oreilly.com/library/view/i-heart-logs/9781491909379/"><em>Streaming Systems</em></a> by Tyler Akidau on Beam/Dataflow, <a href="https://www.oreilly.com/library/view/kafka-the-definitive/9781491936153/"><em>Kafka: The Definitive Guide</em></a> by Neha Narkhede et al., and <a href="https://www.oreilly.com/library/view/i-heart-logs/9781491909379/"><em>I Heart Logs</em></a> by Jay Kreps, which is another take on the origin story of Kafka at LinkedIn.</p><p>In fact, change-based (or log-centric) design goes well beyond streaming, and is quite central in many data systems (think <a href="https://docs.databricks.com/delta/index.html">Delta Lake</a>’s transaction logs, for example). By writing this article, I hope to have imparted a few of the aspects that we have been careful about when we made the design change from being state based to change based. These aspects remain relevant as we push toward lower latency, more complex data operations, and larger scale.</p><h3><strong>Acknowledgments</strong></h3><p>Thanks to the pricing team we work with (especially Ed Pinto, Zoyi Zhang, Ao Li, and Alan Bishel) for the great collaboration and help in putting this solution into production. Thanks also to Harkirat Singh, Venkata Subramaniam, and Ashis Roy for brainstorming about the customer context component.</p><p><em>Ziqi Ma is on </em><a href="https://www.linkedin.com/in/ziqi-ma-491b47164/"><em>LinkedIn</em></a>.</p><h4><strong>References</strong></h4><p>[1] <a href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#arbitrary-stateful-operations">https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#arbitrary-stateful-operations</a></p><p>[2] <a href="https://docs.confluent.io/platform/current/streams/architecture.html#example">https://docs.confluent.io/platform/current/streams/architecture.html#example</a></p><p>[3] <a href="https://samza.incubator.apache.org/learn/documentation/0.7.0/container/state-management.html">https://samza.incubator.apache.org/learn/documentation/0.7.0/container/state-management.html</a></p><p>[4] <a href="http://www.vldb.org/pvldb/vol10/p1634-noghabi.pdf">http://www.vldb.org/pvldb/vol10/p1634-noghabi.pdf</a></p><p>[5] <a href="https://sap1ens.com/blog/2020/12/12/streaming-systems-and-global-state/">https://sap1ens.com/blog/2020/12/12/streaming-systems-and-global-state/</a></p><p>[6] <a href="https://www.oreilly.com/library/view/kafka-the-definitive/9781491936153/ch11.html#idm45788269588584">11. Stream Processing — Kafka: The Definitive Guide [Book] (oreilly.com)</a></p><p><a href="https://www.oreilly.com/library/view/streaming-systems/9781491983867/">https://www.oreilly.com/library/view/streaming-systems/9781491983867/</a></p><p>[7] <a href="https://www.oreilly.com/library/view/i-heart-logs/9781491909379/">I Heart Logs [Book] (oreilly.com)</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=34049816ffbd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-at-microsoft/reasoning-about-change-lessons-learned-from-building-a-near-real-time-system-for-azure-pricing-34049816ffbd">Reasoning about change: Lessons learned from building a near real-time system for Azure pricing</a> was originally published in <a href="https://medium.com/data-science-at-microsoft">Data Science + AI at Microsoft</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Speeding up “Reverse ETL”]]></title>
            <link>https://medium.com/data-science-at-microsoft/speeding-up-reverse-etl-3af04e069fd1?source=rss-22c5dfb9cfdc------2</link>
            <guid isPermaLink="false">https://medium.com/p/3af04e069fd1</guid>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[etl]]></category>
            <dc:creator><![CDATA[Ziqi Ma]]></dc:creator>
            <pubDate>Tue, 07 Mar 2023 08:16:48 GMT</pubDate>
            <atom:updated>2023-03-07T08:16:48.802Z</atom:updated>
            <content:encoded><![CDATA[<p>Talking about the speed of data can be a little tricky. Nowadays data serves so many different use cases, each with its own definition of speed. For example, real-time applications care about latency (such as “within how many milliseconds do I get my answer?”), while large-scale batch operations care about throughput (such as “does processing of my entire dataset finish within two hours or 20 hours?”). Certain query patterns are better served by certain infrastructure designs. While it is not possible to simultaneously speed up all processes in a system for all operations and all users, the data community is continuously evolving to accommodate the ever-growing demand for speed. This article focuses on speeding up a specific process: Reverse ETL.</p><p><em>Reverse ETL</em> is a new approach that has been adopted by industry to bring data from a central warehouse/lake/lakehouse into real-time operational systems. It is a small piece of the big puzzle of data infrastructure, as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DwslKtRhrtc8W26O3vo4aA.png" /></figure><p>In this approach, data flows in from operational systems to a central store via ETL (extract, transform, load) or ELT (extract, load, transform) pipelines. The at-rest data in the central datastore (in our case, a lakehouse — in other cases it could also be a warehouse or data lake) serves as a single source of truth and enables the creation of new data products, whether via business logic or via data science.</p><p>But what has been less talked about is the other side of the picture — the data consumers.</p><p>While BI tools and dashboarding are sometimes sufficient to meet the needs of data consumers, an increasing number of use cases call for low-latency APIs that can feed directly into operational engineering systems. This class of use cases is arguably the most valuable — because it turns data into operational, actionable insights. This is the <a href="https://www.getcensus.com/blog/what-is-reverse-etl">Reverse-ETL process</a>.</p><p>The focus of this article is on a small but critical piece in enabling high-performing data ingestion from the lakehouse to a relational database management system (RDBMS) — the red arrow in the chart above. I share our analysis and experiments on some various factors that affect ingestion speed, and I show how we leveraged this analysis to achieve a fivefold speed increase for our production scenario.</p><h3>RDBMS setup</h3><p>RDBMS (Relational Database Management System) design is critical to the experience of data consumers. Design decisions about RDBMS also have important implications for how we design and optimize the ingestion process. Three important areas of consideration are storage and indexing.</p><h4>Storage: Column store in SQL Server</h4><p>Traditionally, SQL servers are row stores, which store data in a heap, a clustered index, or a memory-optimized table. Starting from 2016, SQL Server started to support operational analytics with columnar storage, allowing transactional and analytic workloads to happen in one store.</p><p>In our case, although our main storage is still the lakehouse, we use column store on Azure SQL Hyperscale to serve our API for three reasons:</p><ol><li>We know that incoming queries typically do not ask for a full row but only for a subset of columns (although different queries can request different subsets), making column store a natural fit.</li><li>Data can be compressed to a smaller size in columnar storage.</li><li>Column store enables good bulk insert performance. The figure below shows the difference between ingesting into row store (heap) versus column store in SQL Server.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ne0k-gLijDlp8UIJ2Fm0HQ.png" /><figcaption>Optimized versus unoptimized ingestion on a 20-gigabyte dataset.</figcaption></figure><p>To understand why we get better performance, we start with a closer look at how data is actually stored in columnar fashion.</p><p>Data is first grouped into rowgroups — normally between 102,400 and 1,048,576 rows. If this number is too small, compression will not be effective. But if it is too large, then it cannot benefit from the low latency of in-memory operations. Each row group is then compressed in a columnar fashion. The “spillover” data is stored as a deltastore in a clustered B-tree index, and compressed and moved to columnar storage once it reaches the minimum required number for a rowgroup. For more information on this process, please see <a href="https://learn.microsoft.com/en-us/sql/relational-databases/indexes/columnstore-indexes-overview?view=sql-server-ver16">Microsoft’s overview of column store indexes</a>.</p><p>Below is a comparison of the space taken by a small test table (90 million rows, 19 columns) when stored differently. Here I also put its size in <a href="https://en.wikipedia.org/wiki/Apache_Parquet">parquet format</a> (which is also columnar) as a reference. This shows how much of a difference compression can make.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dqMJlYtLgYNThgun0Q9kvQ.png" /><figcaption>Comparison of different compression strategies.</figcaption></figure><p>Even though the tables are stored in columnar fashion by declaring a Clustered Columnstore Index, we can still build nonclustered indexes with included columns to speed up queries, based on the query pattern. Similarly, nonclustered columnstore indexes can be built on row store tables — although the relevant columns will be copied in their entirety and built into a separate columnar table.</p><p>I would like to note that loading and storing data in a column store is not always the correct approach. If there are known query patterns — for example, filtering on a particular column — it might be better to store as a row store clustered index, because column store will still result in the scanning of all rowsets.</p><h4>Indexing</h4><p>We cannot design the ingestion process without considering indexing, which is critical to the consumer experience. In a nutshell, indexes are data structures that help speed up specific query patterns. Indexes are what enable the low query latency of RDBMS, and good index design based on query patterns is critical to achieving good performance. The issue is that whenever we make any change to the data, indexes must also be changed. We need to ensure that users never see stale indexes for changed data or experience bad query performance because the index is temporarily unavailable.</p><p>Because ingestion should not affect read performance, two copies of data are retained to allow for time spent on index building: staging and live. The staging environment is where we can perform ingestion or index rebuild, while the live table still serves incoming traffic.</p><p>Thus, ingestion happens in four steps:</p><ol><li>Disable indexes in staging table(s)</li><li>Ingest data to staging table(s)</li><li>Rebuild indexes on staging table(s)</li><li>Swap tables</li></ol><p>When ingesting multiple tables that need to stay consistent with each other, this multi-table swap must be atomic — wrapped within one transaction, as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*afE-sG4rFD07rpA5Wn5WKw.png" /></figure><p>In addition to indexing, <a href="https://learn.microsoft.com/en-us/sql/relational-databases/statistics/statistics?view=sql-server-ver16">table statistics</a> also affect how the query optimizer creates query plans. Statistics capture information about column-level data distribution as well as cross-column correlations, which help the query optimizer estimate cardinality. If indexes are disabled and re-enabled, depending on the configurations, statistics might need to be manually updated to ensure future query performance. The statistics update locks the table and blocks writes. If indexes are dropped and rebuilt, statistics are automatically updated.</p><h3>Ingestion process optimization</h3><p>Once the RDBMS setup is decided based on both read performance and ingestion considerations, we can focus on the ingestion process itself. We summarize six factors to consider when speeding up ingestion: degree of parallelization, locking, batch size, logging, system specific considerations, and ingestion tool.</p><p>By considering these six factors, we were able to achieve fivefold faster results for our RDBMS. While such optimization requires nuanced understanding of specific systems, I would also like to emphasize some big-picture takeaways that remain true even if tools and technologies change.</p><h4>Parallelization</h4><p>The most intuitive factor is parallelization, or how many threads are ingesting to the database. When done from Spark, this is equal to the number of partitions of the input dataframe.</p><p>Increasing parallelization does not always improve performance because sometimes the threads are forced to wait on each other. This can be caused not only by locking (which we discuss in more detail in the next section), but for other reasons too. For example, if Log IO is the bottleneck, increasing parallelism might saturate that limit more easily.</p><h4>Locking</h4><p>Locking is another important factor that affects performance. SQL Server implements locks on different granularity, including row (row range), page, table, and database.</p><p>For bulk insert to row store, table lock improves performance by reducing lock contention. The bulk insert lock on the table still allows multiple threads to load concurrently, but it prevents other non–bulk-insert access to the table.</p><p>On column store, table lock should not be enabled. This is because different threads should load data only to non-overlapping rowsets, so they are able to load concurrently while holding a set of locks for each thread’s specific rowset. Unlike the rowstore setting, enabling table lock in this case causes a thread to take an exclusive lock on the entire table, blocking other rowsets.</p><h4>Batch size</h4><p>Batch size determines how many rows to insert in one commit. Without specifying this parameter, the entire dataset is loaded in one commit, which causes a costly reload if anything goes wrong in the process.</p><p>On the other hand, too small of a batch size incurs performance overhead due to the frequent commits. Although engineers try to avoid ingesting data to indexed tables to avoid overhead, limiting the batch size can provide an advantage when ingesting to indexed tables is necessary. Because insertion involves sorting, a smaller batch size can ensure the whole process happens in-memory without a disk spill. The downside is that this incurs increased fragmentation. You can learn more about fragmentation in <a href="https://learn.microsoft.com/en-us/previous-versions/sql/sql-server-2008/dd425070(v=sql.100)?redirectedfrom=MSDN">Microsoft’s documentation</a>.</p><h4>Logging</h4><p>Logging might not be the first factor that comes to mind when tuning performance, but in some cases it might be the bottleneck.</p><p>SQL servers usually have a limit on the log IO rate for a given SKU. In many cases, this means that we need to hint the system to produce minimal logs. Bulk insert operations are usually minimally logged. However, based on our experiments, when we increase parallelism either on one table or across tables, the log limit is more easily saturated. For more information about the interaction between batch sizing and logging, see <a href="https://learn.microsoft.com/en-us/previous-versions/sql/sql-server-2008/dd425070(v=sql.100)?redirectedfrom=MSDN">Microsoft’s documentation</a>.</p><h4>System-specific considerations</h4><p>Additionally, there are many system-specific considerations.</p><p>For example, Azure SQL Hyperscale accepts writes on the primary and serves reads through read replicas. How many read replicas would the primary need to relay the data to? When copying happens, we see a dip in the ingestion performance. Are the read replicas the same SKU as the primary? If they are of lower SKU, the relay might become a bottleneck of ingestion performance. What other processes are running at the same time — what is the query load like? Are there indexes being built? Are other tables being modified?</p><p>I have found two helpful ways to figure out these system particulars:</p><ol><li>Monitor real-time system metrics such as CPU usage, data IO, and log IO to observe what is the bottleneck for a given configuration.</li><li>Experiment with small-scale data, so that each ingestion job finishes within around 10 minutes and the experiments are manageable.</li></ol><h4>Ingestion tool</h4><p>Finally, the ingestion tool decision is not only about performance but also cost. There are a broad range of tools that enable bulk insert into SQL Servers, including Synapse (Azure Data Factory) copy job, Azure Databricks (with SQL connector), BCP, and Integration Services. Note that performance and cost are both important considerations in choosing the right tool for the right problem.</p><p>I would like to point out that while the experiments shown above are all conducted in a static environment while no other loads are running on the system, in a live production environment, things can get more complicated when different workloads are able to run at the same time. It is my belief that single-workload experiments are informative in understanding the system. Understanding how different workloads affect each other when running concurrently, however, is an important topic that I leave to future work.</p><h3>Future outlook</h3><p>With Reverse ETL, the ultimate problem to solve is enabling real-time data consumption from aggregated large-scale, at-rest data. This problem belongs to the larger challenge of balancing latency with scale, and various communities have been making efforts in this direction. For example, the RDBMS community is pushing hard on scale (e.g., Azure SQL). Although the progress is encouraging, storage is still very costly compared to data lakes and the scale is not limitless. The Lakehouse community, such as Delta Lake, is providing enhanced capacities to reduce query latency. Finally, as streaming technologies continue to develop, some (like <a href="https://www.kai-waehner.de/blog/2021/09/30/reverse-etl-anti-pattern-event-streaming-data-lake-warehouse-kafka-confluent-snowflake-databricks-splunk/">Kai Waehner</a>) even argue that Reverse ETL, or even data at rest, will not be needed anymore. We are excited about future innovations that can help us achieve more with less. This will be achieved by tools and platforms supporting more scenarios with their respective speed requirements, while maintaining a simpler infrastructure and less data duplication.</p><p>Thanks to Harkirat Singh for helping me navigate experimentation and performance debugging on Azure SQL Hyperscale. Thanks also to Saravanan and the Gnosis team for sharing infrastructure. And finally, thanks to Rachit Ranjan, Huizhen Ji, Piyush Sharma, Venkata Subramaniam, and Ashis Roy for providing design feedback and sharing their experience.</p><p><em>Ziqi Ma is on </em><a href="https://www.linkedin.com/in/ziqi-ma-491b47164/"><em>LinkedIn</em></a>.</p><h4><strong>References</strong></h4><p>[1] <a href="https://www.getcensus.com/blog/what-is-reverse-etl">https://www.getcensus.com/blog/what-is-reverse-etl</a></p><p>[2] <a href="https://learn.microsoft.com/en-us/sql/relational-databases/indexes/columnstore-indexes-overview?view=sql-server-ver16">https://learn.microsoft.com/en-us/sql/relational-databases/indexes/columnstore-indexes-overview?view=sql-server-ver16</a>)</p><p>[3] <a href="https://techcommunity.microsoft.com/t5/sql-server-blog/data-loading-performance-considerations-with-clustered/ba-p/305223">https://techcommunity.microsoft.com/t5/sql-server-blog/data-loading-performance-considerations-with-clustered/ba-p/305223</a></p><p>[4] <a href="https://learn.microsoft.com/en-us/previous-versions/sql/sql-server-2008/dd425070(v=sql.100)?redirectedfrom=MSDN">https://learn.microsoft.com/en-us/previous-versions/sql/sql-server-2008/dd425070(v=sql.100)?redirectedfrom=MSDN</a></p><p>[5] <a href="https://github.com/microsoft/sql-spark-connector">https://github.com/microsoft/sql-spark-connector</a></p><p>[6] <a href="https://learn.microsoft.com/en-us/sql/relational-databases/statistics/statistics?view=sql-server-ver16">https://learn.microsoft.com/en-us/sql/relational-databases/statistics/statistics?view=sql-server-ver16</a></p><p>[7] <a href="https://www.kai-waehner.de/blog/2021/09/30/reverse-etl-anti-pattern-event-streaming-data-lake-warehouse-kafka-confluent-snowflake-databricks-splunk/">When to Use Reverse ETL and when it is an Anti-Pattern — Kai Waehner (kai-waehner.de)</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3af04e069fd1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-at-microsoft/speeding-up-reverse-etl-3af04e069fd1">Speeding up “Reverse ETL”</a> was originally published in <a href="https://medium.com/data-science-at-microsoft">Data Science + AI at Microsoft</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What do we talk about when we talk about ML robustness?]]></title>
            <link>https://medium.com/data-science-at-microsoft/what-do-we-talk-about-when-we-talk-about-ml-robustness-4e58a55f9b8f?source=rss-22c5dfb9cfdc------2</link>
            <guid isPermaLink="false">https://medium.com/p/4e58a55f9b8f</guid>
            <category><![CDATA[machine-learning-systems]]></category>
            <category><![CDATA[microsoft-azure]]></category>
            <category><![CDATA[robustness]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Ziqi Ma]]></dc:creator>
            <pubDate>Tue, 21 Dec 2021 08:17:36 GMT</pubDate>
            <atom:updated>2022-01-01T18:30:30.155Z</atom:updated>
            <content:encoded><![CDATA[<p>After finishing a great one-time ML analysis, it is natural to expect it to lead to a reliable, permanent solution. However, this notebook-to-service transformation can be tricky, as I have come to realize throughout my experience of putting ML models into production. The real world is unpredictable and ever-changing. Data can be corrupted. Models might misbehave when exposed to new data.</p><p>This is a significant risk in high-stakes applications. But even when bad predictions are not catastrophic and can be fixed retroactively, robustness still matters: A brittle model means high maintenance that can quickly exhaust data scientists and engineers, leaving little time to develop new models.</p><h3><strong>A partial solution</strong></h3><p><em>“Let’s just retrain the models dynamically in response to shift. Problem solved.”</em></p><p>The most intuitive solution is to retrain models on a regular schedule so that they remain “fresh” despite shift. However, parameters such as training frequency and how much historical data to use are often decided quite arbitrarily. To this end, I experimented with dynamic retraining using an online Martingale exchangeability test (see this <a href="https://icml.cc/2012/papers/808.pdf">related paper</a> for more information).</p><p>The main reason for using Martingale rather than better-known distribution shift statistics is to avoid “batching” time-series data into arbitrarily defined intervals for the calculation of distribution statistics. Although our models are not yet in the online learning regime, we aim to maintain as little data in memory as possible. This approach only requires updating the betting capital (a real number) and thresholding it in direct correspondence to the p-value against the exchangeability assumption (no hyperparameters required). The exchangeability test is done on loss, overcoming the trickiness of estimating shift on a mixture of high-dimensional numerical and categorical data.</p><p>Figure 1 is an experiment run on <a href="https://nijianmo.github.io/amazon/">Amazon book review data from 1996 to 2017</a>, showing a lower loss with dynamic retraining than fixed-schedule training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Of7Exj6is_TuZNlzJ4Hing.png" /><figcaption>Figure 1: Using online Martingale exchangeability test for dynamic model retraining.</figcaption></figure><p>This is a better alternative to fixed-schedule retraining, but as I came to realize, it’s missing the point.</p><h3><strong>What is ML robustness?</strong></h3><p><em>“Oh, you mean covariate shift? Label shift? Adversarial robustness?”<br>“Actually, I mean this deployed model will not fail in the next month.”</em></p><p>Underlying the two statements above are two different notions of “ML”: One is a mathematical function, and the other is a software system with an ML component. As part of my work productionizing ML models, I kept a log of all model failures in production for three months. I found that surprisingly few of them are related to distribution shift.</p><p>Many issues lie at the intersection of models and software. For example, a miscalibrated anomaly detection model might feed twice the number of anomalies to the next model, causing an out-of-memory error. Even for pure software bugs, the uncertainty and long run-time caused by the ML component make them much harder to detect. Such issues suggest that models and related software components should be considered holistically.</p><h3><strong>Robustness as “passing all tests”</strong></h3><p><em>“ML robustness is not robust models plus robust software wrappers.”</em></p><p>What is a meaningful way to formulate software robustness? Rather than devising a theoretical worst case or an average case, an engineer might simply say the software “passes all tests.” And the testing may very well be comprehensive: unit tests, integration tests, mock traffic, pre-production environments, and so on. In fact, a software component might be run hundreds to thousands of times before being exposed to the user.</p><p>However, training a model under all reasonable conditions (assuming one can generate such conditions, which is an interesting problem in itself) would probably take a prohibitively long time. Additionally, while software is generally deterministic, ML depends on ever-changing data, which is inherently unpredictable. In software, creating mock traffic or using automated random tests is often done to increase coverage, in the hope that they can expose code to complex conditions that are otherwise unanticipated. But this is much harder to do when data is involved.</p><p>Even if exactly how the data may shift is known, and there is enough time to test on shifted distributions, what would the “correct” result be? The issue is that ML models are attempts to understand the world via data. While software engineers have the freedom to design (and thus the responsibility of ensuring) desired outputs given certain inputs, data scientists can merely make their best attempt at answering questions with the available data and tools. There remains the question of “what is possible?”</p><p>It is obviously unreasonable to expect good performance when input data is complete noise. However, how robust can we expect our models to be? This is impossible to answer prior to deployment (and likely after deployment too). This unknowability issue, coupled with traditional software challenges related to memory, compute time, and dependencies, results in great complexities when defining what a robust ML system should be like.</p><h3><strong>Data consists of more than distributions</strong></h3><p><em>“I dislike the term ‘distribution shift’ — it implies too many assumptions.”</em></p><p>Calling data distributions implies a statistical perspective, which makes it easy to ignore data’s physical attributes — volume, storage, methods of retrieval, ordering of various streams, schema, skewness of partitions, and so on. All these aspects are closely coupled with models, and distribution shifts capture only one of the many dimensions of the challenge.</p><p>Additionally, model development is usually an iterative process. My original conception of “a static deployed model in changing distributions” is indeed quite off (at least for the platform I work on). Due to frequently changing business logic, upstream data sources and stakeholder requirements, data scientists routinely add new features or data sources to improve their models, changing the input space.</p><p>My “eye-opening” moment was when my team customized a data drift detection service into a much simpler tool. The emergence of such services in the market shows data shift to be increasingly relevant in various ML applications. However, to my surprise, the overwhelming attitude from our data science team was that they prefer simple and intuitive metrics (such as change in count/mean per column) over metrics such as KL divergence and Wasserstein distance.</p><p>The central issue is that for data drift, the ultimate goal is not merely discovery but correction. If all shifts are natural, KL-type metrics can be helpful. However, when shifts happen due to human or system errors (which are much more frequent than I thought!), straightforward metadata monitoring is more helpful in pinpointing root causes and informing mitigation plans. After all, if an outage of one specific data source causes one feature to suddenly contain 70 percent missing values, KL divergence or Wasserstein distance is unlikely to be helpful. Similarly, for issues related to model dependencies, such as the previously mentioned out-of-memory error, a direct, straightforward metric would be most helpful.</p><p>Additionally, there is the “data-centric versus model-centric” argument: KL-type metrics, although defined on distributions, are in fact model-centric: they need to be calculated separately for each model and recalculated when a model adds or removes certain features. On the other hand, data-centric validation techniques (such as <a href="https://aws.amazon.com/blogs/big-data/test-data-quality-at-scale-with-deequ/">deequ</a>) might just be an easier solution when multiple models have intersecting inputs and model features are updated frequently.</p><p>Even before the explosion of ML, data engineers have been wrestling with data quality challenges. The sensitivity of ML model performance to data quality highlights the importance (and trickiness) of these challenges. At least in my team, the good old data quality frameworks are still favored over the newer ML-specific solutions.</p><h3><strong>Solutions for the short and long term</strong></h3><p><em>“If robustness is so hard, what can we do with our ML systems today? What about tomorrow?”</em></p><p>While we might not be able to solve the robustness problem today, we can still take some steps to mitigate failures or speed up corrections. For example, we can work toward building systems that emit timely and intelligible signals upon failures, adapt to environment changes, and even potentially self-correct. Even though we might still need data scientists or engineers to retroactively fix or tweak these systems, we can build tools to make this process easier. Here are some practices adopted by our MLOps team:</p><ul><li><strong>Before go-to-production: </strong>In addition to verifying successful model runs in the development environment, we use a customized automatic analysis tool to scan the program code to avoid frequent reasons for failures observed in the past.</li><li><strong>While in production: </strong>We apply data schema and quality checks on datasets, and monitor drift with a customized version of <a href="https://docs.microsoft.com/en-us/azure/machine-learning/how-to-monitor-datasets?tabs=python">Azure Machine Learning’s data drift service</a>. Violations trigger alerts or stop model training.</li><li><strong>Post-hoc debugging or correction:</strong> We use dependency mapping to track model and data dependencies. We increase automation in the deployment process to ensure timely correction. Time-travel functionalities are in-progress to enable easier debugging.</li></ul><p>We also envision testing frameworks that can perturb input, vary data load (or simulate outages), and even adversarially shift the data to “stress test” ML systems. To speed things up, some tests might be run on smaller mock datasets. From an organizational perspective, the impact of ML models should not only be measured on their first successful run, but also account for their reliability over time.</p><p>The long-term solution might be more complex. ML robustness is not only a technical issue, but also fundamentally related to our vision of ML-human relations. For example, do we want to build auxiliary tools that help human practitioners tackle these challenges more efficiently, or do we strive for increasingly self-sufficient systems that can automatically adapt to the dynamic environment? Do we hope to empower data scientists to put their models into production in one click (and thus be responsible for potential failures), or do we believe in a strong engineering presence in any software product?</p><p>These questions do not have an absolute answer. However, as ML becomes more closely coupled with various systems across domains, the challenges of ML safety, reliability, and maintenance require us to take robustness seriously. Although the reflections in this article are based on one particular ML system, we hope some of the challenges could be relevant to other application scenarios as well. ML robustness is a complex, multidimensional challenge, and its solution depends on the collective wisdom of both engineering and data science.</p><p>Special thanks to Paul Mineiro for the discussions and inspiration, and to Huizhen Ji and Venkata Subramaniam for their helpful feedback.</p><p><em>Ziqi Ma is on </em><a href="https://www.linkedin.com/in/ziqi-ma-491b47164/"><em>LinkedIn</em></a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4e58a55f9b8f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-at-microsoft/what-do-we-talk-about-when-we-talk-about-ml-robustness-4e58a55f9b8f">What do we talk about when we talk about ML robustness?</a> was originally published in <a href="https://medium.com/data-science-at-microsoft">Data Science + AI at Microsoft</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Navigating the (long and winding) road from innovation to production]]></title>
            <link>https://medium.com/data-science-at-microsoft/navigating-the-long-and-winding-road-from-innovation-to-production-b9ab14ec065d?source=rss-22c5dfb9cfdc------2</link>
            <guid isPermaLink="false">https://medium.com/p/b9ab14ec065d</guid>
            <category><![CDATA[microsoft-azure]]></category>
            <category><![CDATA[innovation]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[microsoft]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Ziqi Ma]]></dc:creator>
            <pubDate>Tue, 24 Aug 2021 07:06:09 GMT</pubDate>
            <atom:updated>2021-08-31T22:53:55.844Z</atom:updated>
            <content:encoded><![CDATA[<p>Finding an innovative solution that enables you to solve a real problem is fascinating — perhaps one of the biggest reasons for people to enter the fields of machine learning and data science — but the journey from research innovation to business impact can be long and winding. Innovation is valued in almost every technology or data science organization, and yet finding synergy between innovation and delivery of the best business results is often challenging.</p><p>To better understand these challenges, as well as the key ingredients of facilitating successful research to bring about transformations, I interviewed practitioners as well as leaders in the Microsoft Cloud Data Sciences organization to learn about their experiences in research adoption, collaboration, and fostering innovation.</p><h3><strong>The right question at the right moment</strong></h3><p>Recent advances in state-of-the-art models enable us to solve problems that could not be solved before. A data science solution that is both good and practical, however, is rarely about finding the best model to answer the specific problem setup — it is about enabling action to address the bigger underlying problem.</p><p>The tradeoffs among model performance, speed of delivery/communication, ease of explanation, and ability to productionalize are not easy to navigate. The question naturally arises, “When is the right moment, and what are the right motivators to look for in a more state-of-the-art solution?” To better understand this question, I spoke with Yael Brumer, Senior Data and Applied Scientist at Microsoft Cloud Data Sciences, about her professional journey. Yael has worked with natural language processing for four years, and her team is focused on upgrading the language model for the customer support space. (For more details on the models, check out these articles: <a href="https://medium.com/data-science-at-microsoft/ml-and-customer-support-part-1-using-machine-learning-to-enable-world-class-customer-support-c90b3b02f6a3?source=friends_link&amp;sk=9a2a8174ca765f0b7c16c71bf35da09f">ML and customer support (Part 1): Using Machine Learning to enable world-class customer support</a> and <a href="https://medium.com/data-science-at-microsoft/ml-and-customer-support-part-2-leveraging-topic-modeling-to-identify-the-top-investment-areas-in-f0348382c251?source=friends_link&amp;sk=38d53c28713ab5b1d62d1159a8aa115e">ML and customer support (Part 2): Leveraging topic modeling to identify the top investment areas in support cases</a>.)</p><p>The problem that Yael and her team are trying to tackle is finding out what customers are most dissatisfied about — or what they need the most — from a large volume of support tickets. Finding the answers to these questions enables product teams to prioritize their feature improvements in a more informed way.</p><p>From a data science perspective, the questions are then framed into a clustering problem based on topic modeling. This framing helps data scientists transform ambiguous business questions into a more concrete problem that can be modeled. Yael points out, however, that their goal is not solely to come up with the best model under this framing — the bigger goal is to enable product teams to improve customer satisfaction. According to Yael, “Eventually, what we want is not just to do the best clustering of tickets we have, but to try to proactively predict and address what a customer is going to complain about. And we want to enable any product team to do this.”</p><p>The first model was a simple <a href="https://en.wikipedia.org/wiki/Tf%E2%80%93idf">TF-IDF</a>–based <a href="https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation">LDA</a> model specific to the “unexpected charge” area (in other words, support tickets related to the issue of receiving unexpected charges from Azure). Yael remarked that a simpler model enabled faster stakeholder communication: “There is a tradeoff between model accuracy and time to deliver. A simpler model means a faster solution, and therefore less time to get feedback from stakeholders.” To me, this sounds like the ML version of “agile development”: Faster iteration, increased adaptivity, and constant feedback to make sure data science is in good sync with the business. The data scientists on Yael’s team spend a lot of time cleaning and inspecting various clusters of subtopics related to “unexpected charges,” which also gives them valuable knowledge in the business domain area. Additionally, a simpler model enabled them to deploy in production relatively smoothly without consuming a great deal of compute resources, such as GPUs.</p><p>As the first model in this area of inquiry went live successfully, Yael and her team started to think about scaling to other areas. To do so, however, called for a more complex model. According to Yael, “For the initial model, we needed to do very careful cleaning (such as with stop words, subwords, and stemming), including removing certain keywords manually. Hyperparameter tuning also took a long time. That level of customization would not be feasible for offering as a service to a broader range of product teams. We also started to notice the importance of context, as we observed different customers phrasing the same problems in various ways.”</p><p>Motivated by the model’s generalizability and changing context, Yael turned to a new transfer learning solution based on <a href="https://arxiv.org/abs/1810.04805">BERT</a>. The new model is able to provide the same service of finding the most important topics from support tickets related to any product area, and is accessible through an internal portal.</p><p>So, when was the right moment for this more complex model? It probably wouldn’t have been a good idea to start out with the BERT approach, even though it might have brought better clustering results. That’s because at the initial stage, rapid communication and quick proof of success are both key. Deployment with GPU compute resources also would have taken much longer. The right moment came when the quick win from a simple model earned stakeholder trust, and the team was ready to scale the solution to multiple services, which called for more advanced modeling for contextual understanding.</p><h3><strong>Three key enablers</strong></h3><p>Even when the right moment for adopting research innovations is identified, and the methodology does indeed address a meaningful business question that can make an impact, the research-to-production process can still be a long journey with no shortage of challenges. To understand more about such journeys, I interviewed Jane Huang, Senior Machine Learning Scientist, who spent almost two years working with the Microsoft Research team in New England on causal inference for investment ROI estimation.</p><p>Jane highlighted three key elements she thinks are necessary for successful application of research results on a business problem: <strong>direct and frequent stakeholder communication, data engineering capacity, </strong>and <strong>domain knowledge</strong>.</p><p><strong>Stakeholder communication</strong> is highly important for two reasons: 1) getting a deep understanding of the problem space, and 2) obtaining higher-quality data. First, good problem solving requires understanding the real, broader problem given the business context, rather than taking the stakeholder’s request at face value. Rather than limiting themselves to a specific problem setting, the best problem solvers should be able to put themselves in the place of their stakeholders and think about the maximum value that data can deliver. This requires communication well beyond the technical side. Stakeholders vary in their technical acumen, and communicating well with those who have different levels of technical understanding is pivotal — otherwise, discussion on certain definitions can take weeks, slowing down overall progress.</p><p>The second aspect of communication is to better understand the available data and working with the stakeholder to collect better data. Jane mentioned that in the initial stage, one business stakeholder was working directly with the research team without any middle layer, and the research team received incomplete, low-quality, non-queryable cube data that required imputation before modeling, which did not always result in satisfactory results. After Jane’s team joined the effort, they spent a great amount of time connecting with data engineers for multiple data sources, helping related stakeholders improve their data quality, which was critical to the success of the project.</p><p>For example, Jane’s team identified issues related to workload solution area mismapping and submitted a ticket for the data owner to fix it. They also identified the need to summarize investment/discounting data by solution areas they were receiving from stakeholders, and then they helped the data team prioritize their work. “Data-centric AI” might sound like only a buzzword, but in practice it means that data science solution providers and stakeholders must both put in the effort to work together, and the actual solution represents an <strong>active, long-term relationship</strong> rather than a one-time transaction.</p><p>This means that data scientists cannot just sit on their technical expertise, and stakeholders cannot expect good solutions simply by handing off some data files. Tracing the data flow, in addition to its input and output, is also important. Discussions on the best way to integrate and deliver output to the stakeholder also take considerable time. The often less-applauded work in coordination, relationship building, and data quality improvement is indeed indispensable.</p><p>In addition to stakeholder communication, a level of <strong>engineering capacity</strong> is also important: Before even going into production, it is immensely helpful to build data pipelines and pull from various data sources to fuel the analysis. The ability to build such pipelines efficiently can accelerate the pace of projects from months to weeks. Finally, the ability to provide data science as a true service depends on the engineering capability to productionalize models.</p><p>The last key enabler is <strong>domain knowledge</strong>, which goes one step further than good stakeholder communication. Working in a data science organization and collaborating with multiple stakeholders in related areas results in having additional expertise and familiarity with the problem space. In this sense, a data scientist with experience working across projects in similar problem spaces might uncover more connections and insights from other data sources that the stakeholder might not even know about. This shows the value of having a centralized repository for data and insights, which can in some sense enable a transfer of learning to new problems in adjacent problem spaces.</p><p>When I asked Jane what tips she would give anyone well versed in research to successfully solve a practical problem using research idea, she smiled and said, “Partner with us as data scientists, or with a team that has three key capacities: Ongoing direct and frequent stakeholder communication, engineering capacity, and broader domain knowledge and experience in the problem space.” I found this answer especially interesting, because any of these key enablers can hardly be accomplished by one individual, and yet the quality of each one has a deep impact on the success of such projects. The journey from research to practical impact is indeed a long and winding one, requiring truly collaborative efforts across roles and teams.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m2YLDEz65LemZUIVGY5yWg.png" /><figcaption>Three key enablers</figcaption></figure><h3><strong>Two-way inspiration</strong></h3><p>Once we have identified the right moment, found the right methodology, and gathered all three key enablers, one of the best things that could happen, even beyond the success of one project, is to have two-way inspiration. Inspiration does not only flow in only one way from research to practice; practice can also inspire research, which is not always as obvious — and yet is truly valuable.</p><p>Besides demonstrating the impact of the research work, a practical problem can highlight the importance of certain problem areas, or even sometimes motivate new research problems. Jane told me a great anecdote about how her team’s applied project motivated the incorporation of a dynamic adjustment technique into the <a href="https://github.com/microsoft/EconML">EconML</a> package, the economic-focused causal inference package developed by Microsoft Research. In Jane’s project involving the use of causal inference to obtain a holistic estimation of investment ROI, they discovered the importance of a dynamic treatment effect approach. This approach removes the effects of future incentives from the outcome to create a projected long-term revenue stream as if those future incentives never happened.</p><p>Jane and her team tried out the method proposed in <a href="https://arxiv.org/abs/2002.07285">a new paper</a> by Greg Lewis and Vasilis Syrgkanis, which later became part of the EconML package’s new release. The success of Jane’s project in applying dynamic adjustment showed that it could work in practice, thus highlighting its importance and motivating the incorporation of this method into the package. Jane also told me that in addition to dynamic adjustment, based on the practical scenarios she faced, more research work in evaluation approaches in the absence of labels, as well as inference without being able to observe all confounders, would be very helpful.</p><h3><strong>Culture and bottom-up innovation</strong></h3><p>We have seen that putting research innovation into practice demands a lot of hard work. If it’s not written into anyone’s stated job responsibility, how can such efforts be encouraged from an organizational level? I interviewed Ron Sielinski, Principal Director of Data Sciences, and Siddharth Kumar, Principal ML Scientist Manager, both at Microsoft, to understand this question better.</p><p>There is a delicate balance between operating well under practical constraints and giving enough space for free learning and discovery. Practical constraints such as the opportunity cost of exploration and innovation, ease of deployment, and the importance of consistency in delivering good results are very important in the day-to-day work of a data scientist. At the same time, thinking about it at an organizational level, Ron and Sid both mentioned the importance of leaving time for free exploration and learning. For example, in our organization, Friday afternoons are a no-meeting time reserved for learning, and Sid’s Machine Learning team has a weekly session where data scientists can discuss anything related to ML, which ranges from “new package sharing” to discussions on <a href="https://openai.com/blog/openai-api/">GPT-3</a>. Both also mentioned leveraging venues such as the Microsoft Research AI School (which selects and pairs up product teams with research mentors to initiate collaboration projects), the Microsoft-internal bi-annual Machine Learning and Applied Data Science (MLADS) conference, and publishing blog articles both internally and externally, all of which have started many interesting conversations.</p><p>“But it really boils down to culture,” Ron said. People’s hallway conversations, their collaboration on projects, and their follow-up questions to presentations or blog posts are key motivators of innovation. I agree that the motivation to search for innovative approaches and take them all the way to production needs to be bottom-up. Unlike more planned work items, these projects can hardly be clearly defined ahead of time and assigned to people. Although the reward mechanism for a successful project with impact is always there, considering the challenges outlined in the sections above, achieving success in such projects is not always easy.</p><h3><strong>Conclusion</strong></h3><p>The journey from research innovation to production impact can be long and winding, but is also highly rewarding. Through my interviews with practitioners and leaders in the Microsoft Cloud Data Sciences organization, I have learned about the specific challenges they faced, enabling me to reflect on their learnings and appreciate the two-way inspiration that both solves practical problems and motivates the importance of certain research problem spaces. I hope these learnings are also beneficial to you if you are also part of such a journey.</p><p><em>Ziqi Ma is on </em><a href="https://www.linkedin.com/in/ziqi-ma-491b47164/"><em>LinkedIn</em></a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b9ab14ec065d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-at-microsoft/navigating-the-long-and-winding-road-from-innovation-to-production-b9ab14ec065d">Navigating the (long and winding) road from innovation to production</a> was originally published in <a href="https://medium.com/data-science-at-microsoft">Data Science + AI at Microsoft</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data science perspectives on averages and related areas]]></title>
            <link>https://medium.com/data-science-at-microsoft/data-science-perspectives-on-averages-and-related-areas-4df73468f820?source=rss-22c5dfb9cfdc------2</link>
            <guid isPermaLink="false">https://medium.com/p/4df73468f820</guid>
            <category><![CDATA[microsoft]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Ziqi Ma]]></dc:creator>
            <pubDate>Tue, 08 Jun 2021 17:16:15 GMT</pubDate>
            <atom:updated>2021-06-08T23:54:18.023Z</atom:updated>
            <content:encoded><![CDATA[<h4>More knowledge can help you use these tools more effectively</h4><p>By <a href="https://www.linkedin.com/in/ziqi-ma-491b47164/">Ziqi Ma</a>, <a href="https://www.linkedin.com/in/ivan-b-b4709137/">Ivan Barrientos</a>, <a href="https://www.linkedin.com/in/shijing-fang-79a40b3b/">Shijing Fang</a>, <a href="https://www.linkedin.com/in/laurent-bou%C3%A9-b7923853/">Laurent Boué</a>, and <a href="https://www.linkedin.com/in/ekaterinalajintseva/">Ekaterina Lazhintseva</a></p><p>Data scientists are accustomed to developing new insights from data, employing a variety of analytical tools, ML models, and statistical techniques. But it is also possible to gain new insights related to the tools themselves, which enables a deeper understanding of how they work and what to look out for while using them. In his article “<a href="https://medium.com/data-science-at-microsoft/no-customer-is-average-115cadb9f367?source=friends_link&amp;sk=237330a5b1b1d7075cf87321675fdc75">No customer is average</a>,” Ron Sielinski writes that averages can sometimes obscure more than they reveal. But they remain useful, especially when considered in light of related subjects.</p><p>In this article, we explore several topics related to averages. It is our goal to help you deepen your understanding of averages, their limitations, and some related concepts. We start with an exploration of the Central Limit Theorem, construct representative subsets, consider outliers, probe the limits of describing distributions, and end with exploring the Bayesian framework.</p><h3><strong>Understanding the Central Limit Theorem</strong></h3><p>Imagine you start flipping a coin. You flip it six times and get four heads and two tails. You flip it another six times and you get two heads and four tails. You flip it yet another six times and you get only one head and five tails. If you continue to flip the coin in this way and average the results of each set of six coin flips, you will find the result tends to get closer to three heads and three tails.</p><p>This is an example of the central limit theorem (CLT), which is a statement about how the average of samples behaves as more samples are added up. In this case of doing multiple sets of six coin flips, as the results average out to three heads and three tails, they are said to be <em>normally distributed</em>. What this means is that, in this special case of our flipped coin, while sometimes we get six successive heads (or six successive tails), these outcomes are rarer than the times we get something closer to three heads and three tails. If we were to plot these outcomes on a graph, it would look like a bell-shaped curve, with the most frequent outcomes at the top of the bell (i.e., outcomes close to three heads and three tails), and rarer outcomes at the lower edges of the bell (i.e., outcomes close to six heads or six tails).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*OcclOs23Yraxr5j5jbZ6mg.png" /></figure><p>Sometimes the CLT is incorrectly referred to as a guarantee that if you sample enough data — the outcomes of the successive sets of six coin flips of our example, or your own data — the data can be <em>assumed</em> to be normally distributed, and that as a result, its average would be as meaningful as what we get from our sets of coin flips. However, this is not actually the case. Rather, as you average more data, it is the <em>averaging process, rather than the data itself,</em> that is normally distributed about the true (asymptotic) average.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*aTJhdLtHLdkR902n7e2Uzw.png" /></figure><p>So, if the CLT doesn’t help us in the way we might have hoped it would, how does it help us? The CLT can still be useful because it gives us a way to ask important questions such as, “How much data do I need to get a good estimate of its true average value?” as long as we take some things into consideration.</p><p>The bell-shaped distribution we discussed above, for example, isn’t the only one possible. A (mild) form of the CLT assumes that your data has both a finite mean (the average we’ve been discussing) and variance. Not every distribution, however, has a finite mean and variance. A frequently encountered one that has a finite mean but non-finite variance is known as the <a href="https://en.wikipedia.org/wiki/Cauchy_distribution">Cauchy distribution</a>. Another, known as the <a href="https://en.wikipedia.org/wiki/Pareto_distribution">Pareto distribution</a>, can have both a non-finite mean and non-finite variance! The upshot of this is that it’s important to understand the distribution that your data actually has, instead of simply assuming it’s normally distributed. If you don’t, your results may not be valid. This is yet another reason to make <a href="https://en.wikipedia.org/wiki/Exploratory_data_analysis">exploratory data analysis</a> (EDA) one of your first steps, because it will reveal how your data is distributed, among other attributes.</p><p>So, we’ve seen that the CLT identifies a single (asymptotic) average, but such an identification may not be helpful if a distribution is “bimodal,” as illustrated by our coin flipping example. Let’s return to that example in more detail. If the coin is flipped and the result is heads, record +1, and if tails, record –1. The law of large numbers — which applies if we keep flipping the coin and recording the results — says that the <em>average value</em> we get is zero. Note, however, that <em>nothing you have actually written down from each coin flip is ever zero</em>: It’s either +1 or –1.</p><p>That might not be a problem if we want to apply CLT to see how far the <em>average of a sequence</em> of coin flips is from zero. In a real-world scenario this could also help us quantify how “fair” the coin is, say, using <a href="https://medium.com/data-science-at-microsoft/csat-an-emperor-with-no-clothes-e1c27a1b93eb?source=friends_link&amp;sk=dafc5cdbf917937892fc9e1427934bd8">confidence intervals</a>. But now suppose that instead of +1 and –1, we replaced those values with +1000 and –1000. The average value is still zero and the CLT still applies, but how insightful is the CLT now? We could still apply confidence intervals, but they might not be informative because the variance is now significantly higher than in our initial +/–1 model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*JGsR0eNztuB24RRqd8lUpA.png" /></figure><p>This is the sort of reason it’s important not to over-rely on averages, and to have at our disposal techniques that help us better understand data beyond its simple average. We now turn to showing some of these techniques.</p><h3><strong>Using a (smart) subset: The “coreset”</strong></h3><p>In the era of big data, when we have more data than we can analyze in real time, it makes sense to focus on a subset — or coreset — of it. As an aid to understanding coresets, let’s revisit our coin flip example.</p><p>Flipping a coin is a simple example of where having some summary statistics, or a simple model containing good prior knowledge of the data, is sufficient. Consider, however, a more complex scenario where the coin flip is not fair (i.e., there is a tendency for the coin to land either on heads or on tails most of the time, instead of on each of them about half the time). Consider further that not only are the coin flips unfair, but their unfairness is some function of the weather that is unknown to us.</p><p>Let’s say that as a hobby, each day I flip the coin (with knowledge of the lack of fairness) and record the results, as well as metrics related to weather such as temperature, humidity, and UV level. After doing this for more than a year, I end up with a lot of data. But in this case, instead of working with summary statistics or some parametric distribution of the data I’ve collected, perhaps a good alternative is to get a representative subset of those flip records — for example, 30 “representative” days — and devise some intelligent importance weights for each of the 30 records such that this subset can be a good and concise representation of the overall data. This small, weighted subset is called a coreset.</p><p>The notion of coreset comes from computational geometry. Coreset is increasingly relevant today as data volume keeps growing. On the one hand, big data fuels better models and better insights. On the other hand, we do not always want to wait for hours when iterating on ideas (in addition to the cost of computation, which is not just money, but also energy). Thus, it is natural to want a succinct, lightweight summary of the dataset that allows for quick exploration and experimentation.</p><p>This might sound contradictory: Don’t we always want bigger, richer data to derive good insights and build good models? This is exactly what makes the problem of coreset construction nontrivial, because we must sample and weigh the coreset well enough so that we preserve the richness of original dataset while maintaining fast and cheap computation (which is why we are constructing a coreset in the first place).</p><p>How do we formalize the definition of coreset, and how do we construct one? One mathematical definition is that, with regard to some model class, the weighted loss on the coreset closely approximates (<a href="https://en.wikipedia.org/wiki/(%CE%B5,_%CE%B4)-definition_of_limit">epsilon-close</a>) the loss on the full dataset (for more information, see “<a href="https://arxiv.org/pdf/1703.06476.pdf">Practical Coreset Constructions for Machine Learning</a>”). Coreset construction remains an area of active research: A couple of methods under study build on importance sampling — assigning a higher sampling probability to “important” instances, while downweighing them if they are included in the coreset. This lowers variance while remaining unbiased.</p><p>This necessitates having a formal definition of “important,” which is highly task dependent. When we are clustering or segmenting customers, for example, we might call the relative rare class of high-spending customers “important.” For a classification task, however, the boundary points are the ones that are “important.” In a time series, the “important” points might be inflection points.</p><p>Mathematically, this indicates that “importance” should be related to loss, because that is the optimization objective, and is what the definition of coreset is based on. Defining this relation exactly is the job of the algorithm designer: The definition might be directly proportional (for more information, see “<a href="https://arxiv.org/pdf/1306.1840.pdf">Loss-Proportional Subsampling for Subsequent ERM</a>”), or it might be based on sensitivity (i.e., the worst-case impact on loss of a given data point; for more information, see “<a href="https://arxiv.org/pdf/1703.06476.pdf">Practical Coreset Constructions for Machine Learning</a>”). Regardless of the path chosen, it’s important to strike a good balance between representing the “important” data well, while still maintaining a faithful overall summary of the original data.</p><h3><strong>Considering outliers and winsorization</strong></h3><p>As a practical matter, it’s important to consider outliers. Mathematically there is no way to distinguish between “an important customer that is unusual” and “an outlier to discard.” But from a business perspective, this is an important distinction, and it’s where “winsorization” comes in: “The transformation of statistics by limiting extreme values in the statistical data to reduce the effect of possibly spurious outliers” (as written in <a href="https://en.wikipedia.org/wiki/Winsorizing">Wikipedia</a>).</p><p>The first question is how to define outliers. There is no standard or strong definition; they are typically regarded as observations that are quite different from other observations. Depending on the nature of your data, outliers might be legitimate observations, such as a few wealthy households, or errors in your data, like a web session lasting 36 hours.</p><p>Depending on your domain knowledge of your data and the purpose of your analysis, you might decide to trim some of the data — such as by completely removing extreme observations — or by <em>winsorizing</em> them, which means replacing them so they are still “high enough” but not skewing your data too much. It is a judgment call for the data scientist to decide whether to winsorize or trim. For either approach, however, a threshold must be defined; for example, by using the <a href="https://en.wikipedia.org/wiki/Interquartile_range">IQR method</a>, with a threshold in the seventy-fifth percentile + an interquartile range * 1.5, or in the ninetieth percentile. (Note that if using the ninetieth percentile, it’s always necessary to winsorize or trim your data. If the IQR method is used, it might not be necessary to winsorize.)</p><p>Winsorization is often used to deal with outliers in charts or clustering. For charts that are unreadable due to outliers, using winsorized values — for example, aligned along the ninetieth percentile line — provides a better picture. Or, if you are doing <em>k</em>-means clustering and want to avoid a cluster with only extreme high values — but it’s acceptable to have “high enough” values grouped together — then winsorization is also useful.</p><p>It’s also useful to consider a couple open questions in regard to winsorization:</p><ol><li>Is it acceptable to winsorize only from one side of the spectrum of your data, or if you do ninetieth-percentile winsorization, should you also do the tenth percentile? The answer depends on the purpose of your work and your data, and is a judgment call.</li><li>Is it valid to use winsorized values in regression models, or is it better to use other transformations, such as logarithmic ones? It is worth trying different transformations and evaluating the difference they make in your model.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pSDPpArorxE44lSGeRQiWQ.jpeg" /></figure><h3><strong>Describing a distribution: How far can we go, theoretically?</strong></h3><p>As we’ve seen, it’s useful to capture information about a distribution with a few descriptive statistics. The Central Limit Theorem can be seen as a way to represent a dataset using only two parameters. Allowing for more parameters provides even more freedom to represent more complicated structures, such as skewness and higher-order moments of the data distribution. But there is a tradeoff — expressed aptly by <a href="https://en.wikipedia.org/wiki/Occam%27s_razor">Occam’s razor</a> — when modeling a dataset using a parametric model: We don’t want <em>unnecessary</em> complexity.</p><p>Conceptually, any dataset can be characterized by its theoretical minimum description length (the <a href="https://en.wikipedia.org/wiki/Kolmogorov_complexity">Kolmogorov complexity</a>) which, at a high level, describes how many bits of information define the dataset. Statistical learning is about finding regularities so that a dataset can be described using fewer bits. Machine Learning practitioners constantly strive to find a balance between model complexity (more parameters mean more degrees of freedom) and learning (fewer parameters lead to a simpler model).</p><p>In practice, such distinctions can be quite subtle. For example, representing parameters as floating points of varying precision raises new tradeoffs between computation complexity and the generalization ability of parametric models. This is a topic of active research. <a href="https://arxiv.org/abs/1904.12320">Taken to the extreme</a>, it’s possible to define a class of one-parameter models that are able to perfectly represent any dataset — but that will not generalize to any held-out dataset.</p><p>Sometimes datasets are sufficiently simple that the first few moments, such as floating-point average, variance, skewness, kurtosis, and so on, are enough. At other times, more sophisticated alternatives are needed, such as overparametrized deep learning architectures that are able to extract essential features of the dataset even if the underlying learning mechanism is still not completely understood.</p><h3><strong>Leveraging prior knowledge about the data: The Bayesian framework</strong></h3><p>Referring to the coin flip example once again, suppose we try to find “the number of times that heads occurs when we flip the coin six times.” One way to approach this, based on the CLT, is to sample enough times of the event (one event means flipping the coin six times), draw the sample mean’s distribution (assuming a normal distribution), and then choose a single value (the point estimate) known as a maximum likelihood estimate (MLE) to represent the “best guess” of the true mean. But as obvious as it might seem, this type of <a href="https://en.wikipedia.org/wiki/Frequentist_inference">frequentist approach</a> can interfere with our ability to interpret the data in useful and valid ways. For example, confidence intervals must be introduced to demonstrate the degree to which the point estimates constituting the average (true mean) are “good” or “bad.”</p><p>In contrast, the Bayesian approach — based on <a href="https://en.wikipedia.org/wiki/Bayes%27_theorem">Bayes’ Rule</a> (formula below) — is a more natural approach to statistical inference leading to meaningful results about our data. The Bayesian approach focuses on the distributions of random variables. Flipping a coin is binary, so we choose the <a href="https://en.wikipedia.org/wiki/Bernoulli_distribution">Bernoulli distribution</a> as the distribution of this event. Each time we collect new data, along with our prior learning, we then establish a new understanding of the probability distribution of the true value (“heads occurring in the event”). In simpler terms, the frequentist approach yields point estimates, whereas the Bayesian statistic yields distributions.</p><p>This provides us with some notable advantages. For one, we can ask, “What is the <em>probability</em> that metric A is better than metric B?” Let’s take a deeper look at the Bayes’ rule, <em>p</em>(<em>θ</em>|<em>X</em>) = <em>p</em>(<em>X</em>|<em>θ</em>)<em>p</em>(<em>θ</em>)/<em>p</em>(<em>X</em>). This reflects the Bayesian assertion that the true mean, <em>p</em>(<em>θ</em>|<em>X</em>), can be updated based on the distribution and the probability drawn from prior learning, <em>p</em>(<em>θ</em>), and the current observation, <em>p</em>(<em>X</em>|<em>θ</em>), which is also called the “likelihood.”</p><p>The <em>p</em>(<em>X</em>) term is the marginal likelihood of the data (called “evidence”). It is computed as the summation (discrete) or integral (continuous) of all the conditional probabilities of <em>X</em> under all the possible events. Taking the evidence as continuous (<em>p</em>(<em>X</em>)=∫<em>p</em>(<em>X</em>|<em>θ</em>)<em>p</em>(<em>θ</em>)<em>d</em>) and applying it to the Bayes’ Rule equation above, the evidence acts as a normalizing constant that only changes the scale of the posterior density. As a result, the “posterior” can be stated as the proportionality to the “likelihood” (namely, <em>p</em>(<em>X</em>|<em>θ</em>) multiplied by <em>p</em>(<em>θ</em>), the “prior”). We generally stick to a specific set of distributions (such as <a href="https://en.wikipedia.org/wiki/Normal_distribution">Gaussian</a>, <a href="https://en.wikipedia.org/wiki/Bernoulli_distribution">Bernoulli</a>, <a href="https://en.wikipedia.org/wiki/Binomial_distribution">binomial</a>, and <a href="https://en.wikipedia.org/wiki/Poisson_distribution">Poisson</a>), and use <a href="https://en.wikipedia.org/wiki/Conjugate_prior">conjugate distributions</a>, which say that if we choose the right “likelihood” and “prior,” then the “posterior” and the “prior” have the same distribution. As a result, we do not have to calculate the evidence. We just need to discover the scale factor, which is exploiting the distribution relationship.</p><p>In practice, we can find distributions suited to our data as well as our priors. For discrete variables, such as binary click-through events, conversion, or churn, we can use the <a href="https://en.wikipedia.org/wiki/Bernoulli_distribution">Bernoulli distribution</a>. For metrics like usage, revenue, or time spent on a page are continuous variables, and we can model them with the <a href="https://en.wikipedia.org/wiki/Beta_distribution">beta distribution</a> family. <a href="https://en.wikipedia.org/wiki/Normal_distribution">Gaussian (normal) distribution</a> is ubiquitous in nature, but as we have seen earlier, it might not always be best suited to the data. As an alternative, the <a href="https://en.wikipedia.org/wiki/Gamma_distribution">gamma distribution</a> can be used for skewed metrics such as revenue.</p><p>What all of this suggests is that we can use Bayesian techniques to iteratively update the mean as we continue to gather data. But we can also compare the distribution of two or more groups. So, instead of saying one is better than the other, we can use the <em>probability</em> that one is better than the other.</p><h3><strong>Conclusion</strong></h3><p>There are more to averages than it might seem at first glance. Using averages in valid ways means understanding the underlying distribution of the data involved, and as we’ve seen, considering a data distribution is nontrivial: There is the theoretical question concerning “how much information can we capture,” and the practical concern of “what information is important to the analysis.” In this article, we have explored several areas useful for addressing this question, not to propose a definitive solution, but to share some perspectives and theoretical or practical considerations on averages and related areas.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4df73468f820" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-at-microsoft/data-science-perspectives-on-averages-and-related-areas-4df73468f820">Data science perspectives on averages and related areas</a> was originally published in <a href="https://medium.com/data-science-at-microsoft">Data Science + AI at Microsoft</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>