<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Quentin Gallea, PhD on Medium]]></title>
        <description><![CDATA[Stories by Quentin Gallea, PhD on Medium]]></description>
        <link>https://medium.com/@quentin.gallea?source=rss-a52dcb9793ad------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*wzHWI-3NPH0r-C_MwE7-CQ.png</url>
            <title>Stories by Quentin Gallea, PhD on Medium</title>
            <link>https://medium.com/@quentin.gallea?source=rss-a52dcb9793ad------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 02:11:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@quentin.gallea/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Easiest Way to Learn and Use Python Today]]></title>
            <link>https://medium.com/data-science/the-easiest-way-to-learn-and-use-python-today-8a230424ece4?source=rss-a52dcb9793ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/8a230424ece4</guid>
            <category><![CDATA[getting-started]]></category>
            <category><![CDATA[google-colab]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Quentin Gallea, PhD]]></dc:creator>
            <pubDate>Wed, 09 Oct 2024 17:55:45 GMT</pubDate>
            <atom:updated>2024-10-09T17:55:45.588Z</atom:updated>
            <content:encoded><![CDATA[<h4>Google Colab and its integrated Generative AI, a powerful combination</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h5sbzpH6qrBZw-kHUWeeOw.png" /><figcaption>Image by author, realized with DALL-E.</figcaption></figure><p><strong>What you’ll find in this article: A guide on the various ways to use Generative AI tools integrated into Google Colab (a no-installation, cloud-based platform for coding in Python), making it the easiest way to learn and work with Python.</strong></p><p>Knowing how to code is more useful and more accessible than ever. In this article you’ll see how to start coding in a minute without any prerequisites, leveraging the power of the latest Generative AI tools.</p><p>I started coding 25 years ago; I was about 10 years old. Everything was tough, from installing development tools, to learning the commands, including debugging of course.</p><p>Today, we are very far from that era. Google Colab recently integrated a set of GenAI tools that completely revolutionize the way we code.</p><blockquote>It has never been easier to start coding. All the barriers are now down.</blockquote><p>This is great news because coding is virtually everywhere and becoming useful, or even required, in a growing number of jobs. Moreover, if you know a little bit of code, you can now go extremely far with minimal effort thanks to these Generative AI tools.</p><p>In this article, I will show you the most efficient way to learn and use Python today with a no-installation tool. If you are not new to Python (know what Google Colab and notebooks are, you can skip Part I). The article is organized as follows:</p><p><strong>Part I: Preliminary:</strong></p><ul><li>Why choose Python and Google Colab?</li><li>Where to start learning Python?</li></ul><p><strong>Part II: Generative AI tools integrated in Google Colab:</strong></p><ul><li>Code completion</li><li>Debugging</li><li>Suggestions</li><li>Graph suggestion</li><li>Getting help</li></ul><p><strong>Discussion</strong></p><h3>Part I: Preliminary</h3><h4>Why choose Python and Google Colab?</h4><p><strong>Why Python? </strong>Python is the most popular and versatile language today. Python can be used for:</p><ul><li>Machine Learning and Artificial Intelligence (e.g. NLP, deep learning),</li><li>Statistics and Analytics</li><li>Creating and working with Chatbots (e.g. LLMs, agents etc.)</li><li>Web development (e.g. Backend Development)</li><li>and more: Finance, Robotics, Database access, Game development etc.</li></ul><p>Moreover, due to its popularity, Python is a requirement for many jobs, and it’s particularly easy to learn because of the vast number of resources available.</p><p><strong>Why Google Colab? </strong>When it comes to Python there are numerous ways to code. The two most popular ways to start are IDEs (Integrated Development Environments) or Notebooks. Notebooks are a web-based interactive environment for writing code. They allow you to mix code, text, and visualizations in a single document.</p><p>You can either install a local notebook tool on your computer (e.g. Jupyter Notebook) or use an online cloud-based solution like Google Colab.</p><p>Since this guide is focused on accessibility, I picked a cloud-based tool that requires no installation. The only requirement is a Google account. All the documents will be stored on your Google Drive, and hence you can work from any computer and easily collaborate with others.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SsBINdaj9_nqKycIpRLKKQ.png" /><figcaption>This is a snippet of a Notebook (Google Colab). You can see that in the same document, you have texts, code and the output of the code. Image by author</figcaption></figure><h4>Where to start learning Python?</h4><p>There are countless options to start learning Python. Here are two sources for a complete beginner’s guide to Python in different formats:</p><ul><li>YouTube free full course for beginners: <a href="https://www.youtube.com/watch?v=rfscVS0vtbw">https://www.youtube.com/watch?v=rfscVS0vtbw</a></li><li>Free full course with with integrated code cells: <a href="https://www.w3schools.com/python/python_intro.asp">https://www.w3schools.com/python/python_intro.asp</a></li><li>Interactive platforms: <a href="https://www.datacamp.com/courses/intro-to-python-for-data-science">DataCamp</a></li></ul><p>Learning how to code is similar to learning many other skills, like swimming or biking — you need to practice. So, when you start with these tutorials or others, open Google Colab, start experimenting with code, and adapt it. Use the tools covered in Part II to support your learning journey.</p><h3>Part II: Generative AI Tools Integrated in Google Colab</h3><p>Since the public release of ChatGPT 3.5 in November 2022, the number of Generative AI tools to support coding has grown quickly. Large Language Models, like ChatGPT, are incredibly powerful to help us with code. Coding relies on a “language” with clear syntax, which makes it an ideal domain for LLMs.</p><p>Google Colab recently integrated a set of Generative AI tools that can support various aspects of your work, from code suggestions to debugging and explanations. Let’s now cover all of these tools:</p><ul><li>Code completion</li><li>Debugging</li><li>Suggestions</li><li>Automatic graphs suggestions</li><li>Help</li></ul><h4>Code completion</h4><p>When you start typing code in Google Colab, you’ll quickly notice that code suggestions appear in grey and italic (see video below) beyond what you type.</p><p>The suggestions appear very quickly and adapt as you continue typing. You just need to press the Tab key to accept the suggestion.</p><p>Note that the suggestions are based not only on what you are typing but also on the rest of the file, making this feature incredibly powerful and going significantly beyond traditional simple code completion tools. For example, in the video below, the suggestion for importing a file is not generic — it’s the exact code needed to import the file in my active Google Colab document with the correct format.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*3Ksc-Oi4V6TSKHtYjt39jQ.gif" /><figcaption>This video illustrates the code completion tool in Google Colab. You can see me type the beginning of some line of code in white and then the suggestions automatically and very quickly appear in grey and italic. I just have to click the Tab key to accept the suggestion. Video by author.</figcaption></figure><h4>Debugging</h4><p>If you’ve ever tried coding, you know that debugging is often what we spend the most time doing. Historically, if you didn’t understand the error, you would refer to manuals, copy and paste the error message, and search online (e.g. Stack Overflow) for solutions. More recently, you might even ask ChatGPT or another LLM for help.</p><p>But now there’s an incredibly fast integrated solution. As you can see in the video below, I ran some code that generated an error. After each error message, you’ll see a button labeled “Explain error.” Once you click it, a pane will open on the right-hand side, and Gemini (an LLM) will explain the error and propose adjusted code. You can then adapt the code by hand, copy-paste the suggestion, or in one click, create a new cell with the corrected code in your notebook.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dGuHVfsctplJB1ztK_3GJg.png" /><figcaption>Here is an example of a simple debugging situation with the integrated Generative AI system in Google Colab. Image by author.</figcaption></figure><h4><strong>Suggestions</strong></h4><p>Beyond code completion, Google Colab offers two simple ways to suggest code based on your description.</p><p>The first way is by writing a comment (see the video below). I just write a comment that explains the next line of code, and Colab directly interprets it and automatically suggests the corresponding code. This functionality works mainly for simple, usually single lines of code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*CGeVLAecH2LPq3NeVkV9Bg.gif" /><figcaption>This video illustrate how Google Colab use Generative AI to automatically generate the code corresponding the the comment you just wrote. Video by author.</figcaption></figure><p>When you need code suggestions for more complex requests, often requiring several lines of code, you can click on “Generate” with AI when you start a new code block (see video below). Then, you can use natural language to explain what you want to do, and the code will be automatically generated. Note that the prompt will be included as a comment on top, so try to make a clear request to save time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*UBcCinWD-wFHizjTINSwMA.gif" /><figcaption>This video illustrate how you can use natural language to create code in Google Colab. Video by author.</figcaption></figure><h4>Automatic graphs suggestions</h4><p>There are also specific suggestions for graph creation when you work with a dataframe (see video below). When you describe or display part of a dataframe, an icon with a graph appears in the top right corner. When you click on it, you’ll see a gallery of potential graphs. By clicking on one of the graphs, a new code cell will be generated with the code required to create the selected graph.</p><p>So far, I haven’t been very impressed by this function. It crashed a few times, returned errors, or suggested numerous options, but the one I was interested in wasn’t available.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*C-CqhaVcXoWz2m25e4Z8QQ.gif" /><figcaption>This video illustrates the automatic graph suggestion in Google Copilot. Video by author.</figcaption></figure><h4>Help</h4><p>Finally, you can directly chat with Gemini (a chatbot/LLM) to ask code-related questions. These questions could be about a piece of code you don’t understand, how to perform a specific task with code, or virtually anything else. You essentially have an AI tutor available 24/7, just one click away.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dSMR0zmO--m4dLDx45iFMg.png" /><figcaption>You can click on the top right button “Gemini” in Google Colab to directly start a discussion about code. You can ask questions about current code, about how to do something etc. Image by author.</figcaption></figure><h3>Discussion</h3><blockquote>While Generative AI is incredibly useful and powerful for coding, it should be used in moderation when learning. This efficiency might prevent us from truly mastering the material and could negatively affect our long-term performance.</blockquote><p>I was blown away by the impact of these Generative AI integrations. I find myself writing less and less code — it’s more about being able to read and test code now. But reading is always easier than writing, just like when learning any language (not just programming).</p><p>However, this raises questions about the long-term effects for people who haven’t yet fully learned how to code. I remember using these tools extensively to select parts of Pandas dataframes because I often mixed up the brackets, .loc or .iloc functions, and syntax. ChatGPT helped me go faster a few times, but over the long run, I became less efficient. If I have to ask every time, it often takes longer than if I knew the solution by heart. And what happens if the tool isn&#39;t available?</p><p>Moreover, it’s very important to remember to use AI suggestions responsibly. Always aim to understand the code you’re incorporating to avoid potential issues with plagiarism or unintended errors. Note that when using suggestions in Google Colab, you might see the source of the code inspiration (see image below). This information can help you avoid potential copyright violations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8aMKKXq1C8voPhrB5cu_ZA.png" /><figcaption>Example of a note added in Google Colab when using a Generative AI code recommendation. It appears that the potential source of the code is cited, and a clickable link is available to check for any potential copyright issues. Image by author.</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8a230424ece4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/the-easiest-way-to-learn-and-use-python-today-8a230424ece4">The Easiest Way to Learn and Use Python Today</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Machine Learning Is Not Made for Causal Estimation]]></title>
            <link>https://medium.com/data-science/why-machine-learning-is-not-made-for-causal-estimation-f2add4a36e85?source=rss-a52dcb9793ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/f2add4a36e85</guid>
            <category><![CDATA[causality]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[causal-inference]]></category>
            <category><![CDATA[causal-machine-learning]]></category>
            <dc:creator><![CDATA[Quentin Gallea, PhD]]></dc:creator>
            <pubDate>Thu, 18 Jul 2024 05:38:24 GMT</pubDate>
            <atom:updated>2024-07-18T05:38:24.081Z</atom:updated>
            <content:encoded><![CDATA[<h4>Predictive vs. causal inference: a critical distinction</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/1*OheiyRPG-YLAtihjLo-jmQ.png" /><figcaption>Image by author, generated with Dall-E 3.</figcaption></figure><blockquote><em>“If all you have is a hammer, everything looks like a nail.”</em></blockquote><p>In the era of AI/ML, Machine Learning is often used as a hammer to solve every problem. While ML is profoundly useful and important, ML is not always the solution.</p><p>Most importantly, Machine Learning is made essentially for predictive inference, which is inherently different from causal inference. Predictive models are incredibly powerful tools, allowing us to detect patterns and associations, but they fall short in explaining why events occur. This is where causal inference steps in, allowing for more informed decision-making, that can effectively influence outcomes and go beyond mere association.</p><blockquote>Predictive inference exploits correlations. So if you know that “Correlation does not imply causation”, you should understand that Machine Learning should not be used blindly to measure causal effects.</blockquote><p>Mistaking predictive inference for causal inference can lead to costly mistakes as we are going to see together! To avoid making such mistakes, we will examine the main differences between these two approaches, discuss the limitations of using machine learning for causal estimation, explore how to choose the appropriate method correctly, how often they work together to solve different parts of a question, and explore how both can be effectively integrated within the framework of Causal Machine Learning.</p><h4>This article will answer the following questions:</h4><ul><li><strong>What is causal inference and what is predictive inference?</strong></li><li><strong>What are the main differences between them, and why does correlation not imply causation?</strong></li><li><strong>Why is it problematic to use Machine Learning for inferring causal effects?</strong></li><li><strong>When should each type of inference be used?</strong></li><li><strong>How can causal and predictive inference be used together?</strong></li><li><strong>What is Causal Machine Learning and how does it fit into this context?</strong></li></ul><h3>What is predictive inference?</h3><p>Machine Learning is about prediction.</p><p><strong>Predictive inference</strong> involves estimating the value of something (an outcome) based on the values of other variables (as they are). If you look outside and people are wearing gloves and hats, it is most certainly cold.</p><p><strong>Examples:</strong></p><ul><li><strong>Spam filter:</strong> ML algorithms are used to filter incoming emails between safe and spam using the content, the sender, and other various information attached to an email.</li><li><strong>Tumor detection:</strong> Machine Learning (Deep Learning) can be used to detect brain tumors from MRI images.</li><li><strong>Fraud detection:</strong> In banking, ML is used to detect potential fraud based on credit card activity.</li></ul><p><strong>Bias-variance:</strong> In predictive inference, you want a model able to predict the outcome well, most of the time out-of-sample (with new unseen data). You might accept a bit of bias if it results in lower variance in the predictions.</p><h3>What is causal inference?</h3><p>Causal inference is the study of cause and effect. It is about impact evaluation.</p><p><strong>Causal inference</strong> aims to measure the value of the outcome when you change the value of something else. In causal inference, you want to know what would happen if you change the value of a variable (feature), everything else equal. This is completely different from predictive inference where you try to predict the value of the outcome for a different observed value of a feature.</p><p><strong>Examples:</strong></p><ul><li><strong>Marketing campaign ROI:</strong> Causal inference helps to measure the impact (consequence) of a marketing campaign (cause).</li><li><strong>Political Economy:</strong> Causal inference is often used to measure the effect (consequence) of a policy (cause).</li><li><strong>Medical research:</strong> Causal inference is key to measuring the effect of drugs or behavior (causes) on health outcomes (consequence).</li></ul><p><strong>Bias-variance:</strong> In causal inference, you do not focus on the quality of the prediction with measures like R-square. Causal inference aims to measure an unbiased coefficient. It is possible to have a valid causal inference model with a relatively low predictive power as the causal effect might explain just a small part of the variance of the outcome.</p><p><strong>Key conceptual difference: </strong>The complexity of causal inference lies in the fact that we want to measure something that we will never actually observe. To measure a causal effect, you need a reference point: the counterfactual. The counterfactual is the world without your treatment or intervention. The causal effect is measured by comparing the observed situation with this reference (the counterfactual).</p><p>Imagine that you have a headache. You take a pill and after a while, your headache is gone. But was it thanks to the pill? Was it because you drank tea or plenty of water? Or just because time went by? It is impossible to know which factor or combination of factors helped as all those effects are confounded. The only way to answer this question perfectly would be to have two parallel worlds. In one of the two worlds, you take the pill and in the other, you don’t. As the pill is the only difference between the two situations, it would allow you to claim that it was the cause. But obviously, we do not have parallel worlds to play with. In causal inference, we call this: <strong>The fundamental problem of causal inference</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Oi8QZ2R3eVm_7wWB5ZJVcw.png" /><figcaption>Ideally we would need parallel worlds to measure causal effects. Image by author.</figcaption></figure><p>So the whole idea of causal inference is to approach this impossible ideal parallel world situation by finding a good counterfactual. This is why the gold standard is randomized experiments. If you randomize the treatment allocation (pill vs. placebo) within a representative group, the only systematic difference (assuming that everything has been done correctly) is the treatment, and hence a statistically significant difference in outcome can be attributed to the treatment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3Xd3h7F9nRpXHjI6UG1eYQ.png" /><figcaption>Illustration of a randomized experiment. Image by author.</figcaption></figure><p>Note that randomized experiments have weaknesses and that it is also possible to measure causal effects with observational data. If you want to know more, I explain those concepts and causal inference more in-depth here:</p><p><a href="https://towardsdatascience.com/the-science-and-art-of-causality-part-1-5d6fb55b7a7c">The Science and Art of Causality (part 1)</a></p><h3>Why does correlation not imply causation?</h3><p>We all know that “Correlation does not imply causation”. But why?</p><p>There are two main scenarios. First, as illustrated below in Case 1, the positive relationship between drowning accidents and ice cream sales is arguably just due to a common cause: the weather. When it is sunny, both take place, but there is no direct causal link between drowning accidents and ice cream sales. This is what we call a spurious correlation. The second scenario is depicted in Case 2. There is a direct effect of education on performance, but cognitive capacity affects both. So, in this situation, the positive correlation between education and job performance is confounded with the effect of cognitive capacity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*NYHecPy8m4f70Y0omEp8MQ.png" /><figcaption>The main reason why “correlation does not imply causation”. The arrows represent the direction of the causal links in causal graphs. Image by author.</figcaption></figure><p>As I mentioned in the introduction, predictive inference exploits correlations. So anyone who knows that ‘Correlation does not imply causation’ should understand that Machine Learning is not inherently suited for causal inference. Ice cream sales might be a <strong>good predictor</strong> of the risk of drowning accidents the same day even if there is <strong>no causal link</strong>. This relationship is just correlational and driven by a common cause: the weather.</p><p>However, if you want to study the potential causal effect of ice cream sales on drowning accidents, you must take this third variable (weather) into account. Otherwise, your estimation of the causal link would be biased due to the famous Omitted Variable Bias. Once you include this third variable in your analysis you would most certainly find that the ice cream sales is not affecting drowning accidents anymore. Often, a simple way to address this is to include this variable in the model so that it is not ‘omitted’ anymore. However, confounders are often unobserved, and hence it is not possible to simply include them in the model. Causal inference has numerous ways to address this issue of unobserved confounders, but discussing these is beyond the scope of this article. If you want to learn more about causal inference, you can follow my guide, here:</p><p><a href="https://towardsdatascience.com/how-to-learn-causal-inference-on-your-own-for-free-98503abc0a06">How to Learn Causal Inference on Your Own for Free</a></p><blockquote>Hence, a central difference between causal and predictive inference is the way you select the “features”.</blockquote><p>In Machine Learning, you usually include features that might improve the prediction quality, and your algorithm can help to select the best features based on predictive power. However, in causal inference, some features should be included at all costs (confounders/common causes) even if the predictive power is low and the effect is not statistically significant. It is not the predictive power of the confounder that is the primary interest but rather how it affects the coefficient of the cause we are studying. Moreover, there are features that should not be included in the causal inference model, for example, mediators. A mediator represents an indirect causal pathway and controlling for such variables would prevent measuring the total causal effect of interest (see illustration below). Hence, the major difference lies in the fact that the inclusion or not of the feature in causal inference depends on the assumed causal relationship between variables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/1*6Iti2QUnt-WVruLdv-ahng.png" /><figcaption>Illustration of a mediator. Here Motivation is a mediator of the effect of training on productivity. Imagine that training for the staff increases their productivity directly but also indirectly through their motivation. The employees are now more motivated because they learned new skills and see that the employer put effort into upskilling the employees. If you want to measure the effect of the training, most of the time, you want to measure the total effect of the treatment (direct and indirect), and including motivation as a control variable would prevent doing so. Image by author.</figcaption></figure><p>This is a subtle topic. Please refer to <a href="https://ftp.cs.ucla.edu/pub/stat_ser/r493.pdf">“A Crash Course in Good and Bad Controls” Cinelli et al. (2002)</a> for more details.</p><h3>Why is it problematic to use Machine Learning for inferring causal effects?</h3><p>Imagine that you interpret the positive association between ice cream sales and drowning accidents as causal. You might want to ban ice cream at all costs. But of course, that would have potentially little to no effect on the outcome.</p><p>A famous correlation is the one between chocolate consumption and Nobel prize laureates <a href="https://www.biostat.jhsph.edu/courses/bio621/misc/Chocolate%20consumption%20cognitive%20function%20and%20nobel%20laurates%20(NEJM).pdf">(Messerli (2012))</a>. The author found a 0.8 linear correlation coefficient between the two variables at the country level. While this sounds like a great argument to eat more chocolate, it should not be interpreted causally. (Note that the arguments of a potential causal relationship presented in Messerli (2012) have been disproved later (e.g., <a href="https://pubmed.ncbi.nlm.nih.gov/23616517/">P Maurage et al. (2013)</a>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d6LRQUAPE5wX1AfmgbsQ6Q.png" /><figcaption>Positive correlation between Nobel Laureates per 10 million population and Chocolate Consumption (kg/yr/capita) found in (Messerli (2012)). Image by author.</figcaption></figure><p>Now let me share a more serious example. Imagine trying to optimize the posts of a content creator. To do so, you build an ML model including numerous features. The analysis revealed that posts published late afternoon or in the evening have the best performance. Hence, you recommend a precise schedule where you post exclusively between 5 pm and 9 pm. Once implemented, the impressions per post crashed. What happened? The ML algorithm predicts based on current patterns, interpreting the data as it appears: posts made late in the day correlate with higher impressions. Eventually, the posts published in the evening were the ones more spontaneous, less planned, and where the author didn’t aim to please the audience in particular but just shared something valuable. So the timing was not the cause; it was the nature of the post. This spontaneous nature might be harder to capture with the ML model (even if you code some features as length, tone, etc., it might not be trivial to capture this).</p><p>In marketing, predictive models are often used to measure the ROI of a marketing campaign.</p><blockquote>Often, models such as simple <strong>Marketing Mix Modeling (MMM)</strong> suffer from omitted variable bias and the measure of ROI will be misleading.</blockquote><p>Typically, the behavior of the competitors might correlate with our campaign and also affect our sales. If this is not taken into account properly, the ROI might be under or over-evaluated, leading to sub-optimal business decisions and ad spending.</p><p>This concept is also important for policy and decision-making. At the beginning of the Covid-19 pandemic, a French “expert” used a graph to argue that lockdowns were counterproductive (see figure below). The graph revealed a positive correlation between the stringency of the lockdown and the number of Covid-related deaths (more severe lockdowns were associated with more deaths). However, this relationship was most likely driven by the opposite causal relationship: when the situation was bad (lots of deaths), countries would impose strict measures. This is called reverse causation. Indeed, when you study properly the trajectory of the number of cases and deaths within a country around the lockdowns controlling for potential confounders, you find a strong negative effect (c.f. <a href="https://pubsonline.informs.org/doi/abs/10.1287/mnsc.2022.4652">Bonardi et al. (2023)</a>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*LEtVAOU9HTlr-ZEb5HU1Fw.png" /><figcaption>Replication of the graph used to argue that lockdowns were ineffective. The green corresponds to the lowest lockdown measure and the red the most restrictive. Image by author.</figcaption></figure><h3>When should each type of inference be used?</h3><blockquote>Machine learning and causal inference are both profoundly useful; they just serve different purposes.</blockquote><p>As usual with numbers and with statistics, most of the time the problem is not the metrics but their interpretation. Hence, a correlation is informative, it becomes problematic only if you interpret it blindly as a causal effect.</p><p><strong>When to use Causal Inference: </strong>When you want to understand the cause-and-effect relationship and do impact evaluation.</p><ul><li><strong>Policy Evaluation:</strong> To determine the impact of a new policy, such as the effect of a new educational program on student performance.</li><li><strong>Medical Studies:</strong> To assess the effectiveness of a new drug or treatment on health outcomes.</li><li><strong>Economics:</strong> To understand the effect of interest rate changes on economic indicators like inflation or employment.</li><li><strong>Marketing:</strong> To evaluate the impact of a marketing campaign on sales.</li></ul><p><strong>Key Questions in Causal Inference:</strong></p><ul><li>What is the effect of X on Y?</li><li>Does changing X cause a change in Y?</li><li>What would happen to Y if we intervene on X?</li></ul><p><strong>When to use Predictive Inference</strong>: When you want to do accurate prediction (association between features and outcome) and learn patterns from the data.</p><ul><li><strong>Risk Assessment:</strong> To predict the likelihood of credit default or insurance claims.</li><li><strong>Recommendation Systems:</strong> To suggest products or content to users based on their past behavior.</li><li><strong>Diagnostics:</strong> To classify medical images for disease detection.</li></ul><p><strong>Key Questions for Predictive Inference:</strong></p><ul><li>What is the expected value of Y given X?</li><li>Can we predict Y based on new data about X?</li><li>How accurately can we forecast Y using current and historical data on X?</li></ul><h3>How can causal and predictive inference be used together?</h3><p>While causal and predictive inference serve different purposes, they sometimes work together. Dima Goldenberg, Senior Machine Learning Manager at Booking.com, illustrated this perfectly in a <a href="https://www.youtube.com/watch?v=xkx1tXLAP-o">podcast</a> with Aleksander Molak (author of ‘Causal Inference and Discovery in Python’).</p><p>Booking.com is obviously working hard on recommendation systems. “Recommendation” is a prediction problem: “What type of product would client X prefer to see?” Hence, this first step is typically solved with Machine Learning. However, there is another connected question: “What is the effect of this new recommendation system on sales/conversion, etc.?” Here, the keyword <strong>“effect … on” </strong>should directly make you realize that you must use causal inference for this second step. This step will require causal inference and more precisely randomized experiments (A/B testing).</p><blockquote>This is a typical workflow including complementary roles of Machine Learning and Causal Inference. You develop a predictive model with Machine Learning and you evaluate its impact with Causal Inference.</blockquote><h3>So, what is Causal Machine Learning and how does it fit into this context?</h3><p>Recently, a new field emerged: Causal Machine Learning. While this is an important breakthrough, I think that it added to the confusion.</p><blockquote>Many people see the term “Causal ML” and just think that they can use Machine Learning carelessly for causal inference.</blockquote><p>Causal ML is an elegant combination of both worlds. However, Causal ML is not Machine Learning used blindly for Causal Inference. It is rather Causal Inference with ML sprinkled on top of it to improve the results. The key differentiating concept of causal inference is still valid with Causal ML. The feature selection relies on the assumed causal relationships.</p><p>Let me present the two main methods in Causal ML to illustrate this interesting combination.</p><h4>A. Dealing with High-Dimensional Data and Complex Functional Forms</h4><p>In some situations, you have numerous control variables in your causal inference models. To reduce the risk of omitted variable bias, you included many potential confounders. How should you deal with this high number of controls? Maybe you have multicollinearity between sets of controls, or maybe you should control for non-linear effects or interactions. This can quickly become very complex and quite arbitrary to solve with traditional causal inference methods.</p><p>As Machine Learning is particularly efficient in dealing with high-dimensional data, it can be used to address these challenges. Machine Learning will be used to find the best set of controls and the right functional form in such a way that your model will not suffer from multicollinearity while still satisfying the conditions to measure a causal effect (see: Double Machine Learning Method).</p><h4>B. Heterogenous Treatment Effects</h4><p>Causal Inference historically focused on measuring the Average Treatment Effect (ATE), which is a measure of the average effect of a treatment. However, as you are certainly aware, the average is useful, but it can also be misleading. The effect of the treatment could lead to different effects depending on the subject. Imagine that a new drug reduces the risk of cancer significantly on average, but actually, the whole effect is driven by the results on men, while the effect on women is null. Or imagine a marketing campaign leading to a higher conversion rate on average while it actually has a negative impact in a specific region.</p><blockquote>Causal ML allows us to go beyond Average Treatment Effect and uncover this heterogeneity by identifying the Conditional Average Treatment Effect (CATE). In other words, it helps to identify the treatment effect conditionally on different characteristics of the subjects.</blockquote><p>The main method to uncover the Conditional Average Treatment Effect is called Causal Forest. However, it is important to note that while this methodology will allow us to find sub-groups with different reactions to a treatment, the characteristics defining such groups uncovered with Machine Learning are not necessarily causal. Imagine that the model reveals that the effect of an Ad is completely different for smartphone users vs. tablet users. The ‘device’ should not be interpreted as the cause for this difference. It could be that the real cause is not measured but correlates with this feature, for example, age.</p><h3>Conclusion</h3><p>Distinguishing predictive from causal inference is central today to avoid costly mistakes in various domains like marketing, policy-making, and medical research. We examined why machine learning, despite its remarkable predictive capabilities, is not intrinsically suited to causal inference due to its reliance on correlations/associations rather than causal relationships. Hopefully, you will be able to understand this distinction and pick the right model for the right type of questions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*ta8YjXDX_y0JoBvIx0JaBA.jpeg" /><figcaption>Illustration of why mistaking a correlation for a causal effect lead to bad decisions. Credit War and Peas, Elizabeth Pich &amp; Jonathan Kunz.</figcaption></figure><p>If you want to learn more about causal machine learning here are a few valuable and reliable <strong>resources:</strong></p><p><strong>Causal Machine Learning:</strong></p><ul><li><a href="https://ijbms.net/assets/files/1721058154.pdf">Causal Machine Learning in Marketing</a></li><li><a href="https://arxiv.org/abs/2101.00878">The Value Added of Machine Learning to Causal Inference: Evidence from Revisited Studies</a></li><li><a href="https://www.nature.com/articles/s41591-024-02902-1">Causal machine learning for predicting treatment outcomes</a></li></ul><p><strong>Double Machine Learning:</strong></p><ul><li><a href="https://matheusfacure.github.io/python-causality-handbook/22-Debiased-Orthogonal-Machine-Learning.html">Matheus Facure book cha</a>pter</li><li><a href="https://arxiv.org/abs/1608.00060">Research paper presenting the methodology</a></li></ul><p><strong>Heterogenous Treatment Effect:</strong></p><ul><li><a href="https://matheusfacure.github.io/python-causality-handbook/18-Heterogeneous-Treatment-Effects-and-Personalization.html">Matheus Facure book chapter</a></li><li><a href="https://www.gsb.stanford.edu/faculty-research/labs-initiatives/sil/research/methods/ai-machine-learning/short-course">Stanford class by Susan Athe</a>y</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f2add4a36e85" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/why-machine-learning-is-not-made-for-causal-estimation-f2add4a36e85">Why Machine Learning Is Not Made for Causal Estimation</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Learn Causal Inference on Your Own for Free]]></title>
            <link>https://medium.com/data-science/how-to-learn-causal-inference-on-your-own-for-free-98503abc0a06?source=rss-a52dcb9793ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/98503abc0a06</guid>
            <category><![CDATA[causal-inference]]></category>
            <category><![CDATA[causality]]></category>
            <category><![CDATA[getting-started]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[econometrics]]></category>
            <dc:creator><![CDATA[Quentin Gallea, PhD]]></dc:creator>
            <pubDate>Tue, 20 Feb 2024 20:51:55 GMT</pubDate>
            <atom:updated>2024-02-20T20:51:55.505Z</atom:updated>
            <content:encoded><![CDATA[<h4>The ultimate self-study guide for all levels</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hEm_3p_F0v2YE3schi_Tow.png" /><figcaption>Image by author</figcaption></figure><p>While everyone focuses on AI and predictive inference, standing out requires mastering not just prediction, but understanding the “why” behind the data — in other words, mastering <strong>causal inference</strong>.</p><blockquote>You have heard that “correlation does not imply causation”, but few truly grasp its implications or know when to confidently assert causality.</blockquote><p>The distinction between predictive inference and causal inference is profound, with the latter often overlooked, leading to costly mistakes. The logic and models between the two approaches are vastly different and this guide aims to equip you with the knowledge to discern causal relationships with confidence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x7ga0_cFTxSOSnHPjld1Ug.png" /><figcaption>Results from Google Trends for “causal inference” revealing the recent quickly growing interest and the association with machine learning. Image by author</figcaption></figure><p>I am convinced that causal inference is arguably one of the most valuable skills to acquire today for three reasons:</p><ol><li>It is tremendously useful for virtually any job, extending beyond data scientists to include business leaders and managers (see next Section).</li><li>It remains a niche and few people are experts in this field and the interest is growing fast (cf. image above).</li><li>As revealed by the Google Trends results (see image above), <em>“causal machine learning”</em> is the latest associated trend. Hence, knowing causal inference will help you connect this knowledge with the current AI focus and put you one step ahead.</li></ol><blockquote><strong>To help you master causal inference and have a valuable asset on the job market and beyond, I crafted this self-study guide, suitable for all levels, requiring no prerequisites, and composed exclusively of free online resources.</strong></blockquote><h4>Plan of the guide:</h4><ol><li>Introduction: Causal inference key concepts</li><li>Technical tools</li><li>Randomized Experiments (A/B testing)</li><li>Quasi-experimental design</li><li>Advanced topics</li><li>Conclusion</li></ol><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FXBJI-pQKDDY&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DXBJI-pQKDDY&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=google" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/32458105d191e4ac0d12aeb11883482d/href">https://medium.com/media/32458105d191e4ac0d12aeb11883482d/href</a></iframe><h3>1. Introduction: Causal inference key concepts</h3><p>Causality, the field focused on understanding the relationships between cause and effect, seeks to answer critical questions such as ‘Why?’ and ‘What if?’. Understanding the concept of causality is crucial from fighting climate change, to our quest for happiness, including strategic decisions making.</p><p>Examples of major questions requiring causal inference:</p><ul><li>What impact might banning fuel cars have on pollution?</li><li>What are the causes behind the spread of certain health issues?</li><li>Could reducing screen time lead to increased happiness?</li><li>What is the Return On Investment of our ad campaign?</li></ul><p>In what follows I will essentially refer to two free e-books available with Python code and data to play with. The first e-book offers quick overviews, while the second allows for a more in-depth exploration of the content.</p><ol><li><a href="https://matheusfacure.github.io/python-causality-handbook/landing-page.html">Causal Inference for the Brave and True</a> <em>by Matheus Facure</em></li></ol><p>2. <a href="https://mixtape.scunning.com/">Causal Inference: The Mixtape</a> <em>by Scott Cuningham</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cc6P4YwbO04nGWTAI3w4wQ.jpeg" /><figcaption>Image by Timo Elliott</figcaption></figure><h4>1.1 The fundamental problem of causal inference</h4><p>Let’s dive into the most fundamental concept necessary to understand causal inference through a situation we might all be familiar with.</p><p>Imagine that you have been working on your computer all day long, a deadline is approaching, and you start to feel a headache coming on. You still have a few hours of work ahead, so you decide to take a pill. After a while, your headache is gone.</p><p>But then, you start questioning: Was it really the pill that made the difference? Or was it because you drank tea or took a break? The fascinating but eventually also frustrating part is that it is impossible to answer this question as all those effects are confounded.</p><blockquote>The only way to know for certain if it was the pill that cured your headache would be to have two parallel worlds.</blockquote><p>In one of the two worlds you take the pill, and in the other, you don’t, or you take a placebo ideally. You can only prove the pill’s causal effect if you feel better in the world where you took the pill, as the pill is the only difference between the two worlds.</p><p>Unfortunately, we do not have access to parallel worlds to experiment with and assess causality. Hence, many factors occur simultaneously and are confounded (e.g., taking a pill for a headache, drinking tea, and taking a break; increasing ad spending during peak sales seasons; assigning more police officers to areas with higher crime rates, etc.).</p><p>To quickly grasp this fundamental concept in more depth without requiring any additional technical knowledge, you can dive into the following article on Towards Data Science:</p><h4>📚 Resource:</h4><p><a href="https://towardsdatascience.com/the-science-and-art-of-causality-part-1-5d6fb55b7a7c">The Science and Art of Causality (part 1)</a></p><h4>1.2 A little bit of formalization: Potential Outcomes</h4><p>Now that you understand the basic idea, it is time to go further and theoretically formalize these concepts. The most common approach is the potential outcomes framework, which allows for the clear articulation of model assumptions. These are essential for specifying the problems and identifying the solutions.</p><p>The central notation used in this model are:</p><ol><li>Yᵢ(0) represents the potential outcome of individual <em>i </em>without the treatment.</li><li>Yᵢ(1) represents the potential outcome of individual <em>i </em>with the treatment.</li></ol><p>Note that various notations are used. The reference to the treatment (1 or 0) may appear in parentheses (as used above), in superscript, or subscript. The letter “Y” refers to the outcome of interest, such as a binary variable that takes the value one if a headache is present and zero otherwise. The subscript “i” refers to the observed entity (e.g., a person, a lab rat, a city, etc.). Finally, the term ‘treatment’ refers to the ‘cause’ you are interested in (e.g., a pill, an advertisement, a policy, etc.).</p><p>Using this notation, we can refer to the fundamental problem of causal inference by stating that it is impossible to observe both Yᵢ(0) and Yᵢ(1) simultaneously. In other words, you never observe the outcome for the same individual with and without the treatment at the same time.</p><p>While we cannot identify the individual effect Yᵢ(1)-Yᵢ(0), we can measure the Average Treatment Effect (ATE): E[Yᵢ(1)-Yᵢ(0)]. However, this Average Treatment Effect is biased if you have systematic differences between the two groups other than the treatment.</p><p>To go beyond this short introduction you can refer to the two following chapters:</p><h4>📚Resources:</h4><ul><li>Brief overview: <a href="https://matheusfacure.github.io/python-causality-handbook/01-Introduction-To-Causality.html">The potential outcomes (Causal Inference for the brave and true)</a></li><li>In-depth: <a href="https://mixtape.scunning.com/04-potential_outcomes">The potential outcomes (Causal Inference: The Mixtape)</a></li></ul><h4>1.3 Visual representation of causal links: Directed (Acyclic) Graphs</h4><p>Visual representations are powerful tools for reducing mental load, clarifying assumptions, and facilitating communication. In causal inference, we use directed graphs. As the name suggests, these graphs depict various elements (e.g., headache, pill, tea) as nodes, connected by unidirectional arrows that illustrate the direction of causal relationships. (Note: I deliberately omit mentioning the common assumption of ‘Acyclicity’ associated with these graphs, as it goes beyond the scope of this overview but is discussed in the second reference available at the end of this subsection.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/1*WF6BFhN6-jWiGNNrBYHkQw.png" /><figcaption>Example of a Directed Graph, image by author</figcaption></figure><p>Causal inference primarly differs from predictive inference due to the assumed underlying causal relationships. Those relationships are explicitly represented using this special kind of graph called Directed (Acyclic) Graphs. This tool jointly with the potential outcome framework is at the core of causal inference and will allow thinking clearly about the potential problems and, consequently, solutions for assessing causality</p><h4>📚Resources:</h4><ul><li>Short intro: <a href="https://matheusfacure.github.io/python-causality-handbook/04-Graphical-Causal-Models.html">Directed Acyclic Graph (Causal Inference for the brave and true)</a></li><li>In-depth: <a href="https://mixtape.scunning.com/03-directed_acyclical_graphs">Directed Acyclic Graph (Causal Inference: The Mixtape)</a></li></ul><h3>2. Technical Tools</h3><p>If you want to go further and apply these methods to data, you will need to be equipped with two technical tools</p><ul><li>A foundational understanding of probability, statistics, and linear regression.</li><li>Knowledge of a statistical software</li></ul><h4>2.1 Probability, Statistics, and Linear regression</h4><p>These tools are valuable for data science in general, and you can concentrate specifically on what matters most. Furthermore, there is a chapter dedicated to this topic in both of the reference books including exclusively on the concepts useful for causal inference:</p><h4>📚Resources:</h4><ul><li><a href="https://mixtape.scunning.com/02-probability_and_regression">Probability and Regression Review (Causal Inference: The Mixtape)</a></li><li><a href="https://matheusfacure.github.io/python-causality-handbook/05-The-Unreasonable-Effectiveness-of-Linear-Regression.html?highlight=b%20testing">The Unreasonable Effectiveness of Linear Regression (Causal Inference for the Brave and True)</a></li></ul><p>In addition, in my opinion, one of the most valuable yet often disregarded topics related to linear regression is the concept of “bad controls”. Understanding what you should control for and what will actually create problems is key. The following two references will allow you to grasp this concept.</p><h4>📚Resources:</h4><ul><li><a href="https://causalinf.substack.com/p/which-variables-do-i-need-to-control">Which variable you should control for? (Scott Cuningham’s substack)</a></li><li><a href="https://ftp.cs.ucla.edu/pub/stat_ser/r493.pdf">A Crash Course in Good and Bad Controls (Cinelli et al. (2022))</a></li></ul><p>Finally, understanding Fixed Effects regression is essential for causal inference. This type of regression allows us to account for numerous confounding factors that might be impossible to measure (e.g., culture) or for which data is simply unavailable.</p><h4>📚Resources:</h4><ul><li>Short intro: <a href="https://matheusfacure.github.io/python-causality-handbook/14-Panel-Data-and-Fixed-Effects.html">Fixed Effects (Causal Inference for the brave and true)</a></li><li>In depth: <a href="https://mixtape.scunning.com/08-panel_data">Fixed Effects (Causal Inference: The Mixtape)</a></li></ul><h4>2.2 Statistical tool</h4><p>Numerous tools allow us to do causal inference and statistical analysis, and in my opinion, the best among them are STATA, Python, and R.</p><p>STATA is specifically designed for statistics, particularly econometrics, making it an incredibly powerful tool. It offers the latest packages from cutting-edge research. However, it is expensive and not versatile.</p><p>Python, on the other hand, is the leading programming language today. It is open-source and highly versatile, which are key reasons why it’s my preferred choice. Additionally, ChatGPT performs very well with Python-related queries which is an important advantage in this AI era.</p><p>R is very powerful for statistics. The debate between R and Python is ongoing, and I leave the judgment to you. Note that R is less versatile than Python, and it appears that ChatGPT’s proficiency with R is not as strong. Additionally, the two main books I reference contain Python code (‘Causal Inference for the Brave and True’; ‘Causal Inference: The Mixtape’). This further supports focusing on Python.</p><p>There are millions of free resources to learn Python from scratch (please comment if you have better suggestions). But here are the ones I would typically recommend if you start from zero (you never coded):</p><h4>📚Resources:</h4><ul><li><a href="https://www.udemy.com/course/pythonforbeginnersintro/">Introduction To Python Programming (Udemy)</a></li><li><a href="https://realpython.com/linear-regression-in-python/">Linear Regression in Python (Real Python)</a></li></ul><h3>3. Randomized Experiments (A/B testing)</h3><p>In the first section of this guide, we discovered the fundamental problem of causal inference. This problem highlighted the difficulty of assessing causality. So what can we do?</p><p>Usually, the first solution presented and considered the <em>“gold standard”</em> in causal inference, is randomized experiments (Randomized Controlled Trials).</p><p>Essentially, the idea behind randomized experiments is to replicate, or at least approach as much as possible, a parallel world scenario. This allows us to isolate the effect (the consequence) of the treatment (the cause).</p><p>As explained in my <a href="https://medium.com/towards-data-science/the-science-and-art-of-causality-part-1-5d6fb55b7a7c">Towards Data Science article</a> mentioned in Section 1:</p><blockquote>“We take a sample that is, hopefully, representative of a larger population, and randomly allocate subjects between two groups (treatment and control) or more. The subjects typically do not know whether they receive the treatment or not (a process known as blinding). Therefore, the two groups are arguably comparable. Since the only difference is the treatment, if we observe an effect, it is potentially causal, provided that no other biases exist.”</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/679/1*qXWEwzlhJN06cnFSSyjguw.png" /><figcaption>Simple representation of a Randomized Experiment. Image by author</figcaption></figure><h4>📚Resource:</h4><ul><li><a href="https://matheusfacure.github.io/python-causality-handbook/02-Randomised-Experiments.html">Randomised Experiments (Causal Inference for the brave and true)</a></li><li><a href="https://news.lunartech.ai/simple-and-complet-guide-to-a-b-testing-c34154d0ce5a">Simple and Complet Guide to A/B Testing </a>(Lunar Tech)</li></ul><h3>4. Quasi-experimental design</h3><p>Controlled experiments are not always possible (e.g. changing the sex/origin for studying discrimination) or ethical (e.g. exposing humans to lethal doses of pollutant to study respiratory disease).</p><p>Moreover, Randomized experiments tend to have a very strong internal validity but a weaker external validity. Internal validity means that it allows within the scope of the study to measure precisely causality, while external validity refers to the capacity to extrapolate the results beyond the scope of the study.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0PikVMkpMCOaWeYZmFJfqw.png" /><figcaption>One of the main limitation of controlled experiments is the external validity. Image by author.</figcaption></figure><p>For example, medical research relies extensively on inbred strains of rats/mice. Those animals have almost an identical genetic code, they live the same lab life, eat the same food, etc. Hence, when you do a controlled experiment with such animals you are very close to the parallel world situation working almost with clones. However, the external validity is weaker due to the homogeneity of the study subjects. Moreover, often in controlled experiments the whole environment is controlled and in some cases, it makes the situation a bit unrealistic and reduces the usefulness of the results.</p><p>Let me illustrate this point with the following research paper. A study published in the <em>British Medical Journal </em>(one of the most prestigious medical journals)<em>,</em> found no effect of using a parachute when jumping from an airplane on the “Composite of death or major traumatic injury” <a href="https://www.bmj.com/content/363/bmj.k5094">(Yeh et al. (2018))</a>. In this randomized experiment, the participants jumped from a small, stationary airplane on the ground. This absurd setup illustrates the issue that often arises in controlled experiments when the setup is unrealistic. The purpose of the paper is to raise awareness in medical research about the issue of external validity, which is a current and serious concern.</p><p>One way to address this issue is by relying on other methods, known as quasi-experimental designs. The idea is to observe a quasi-random allocation between groups in natural settings. ‘Quasi-random’ means that the allocation is effectively as good as random once we isolate or control for potential systematic differences.</p><h4>4.1 Example</h4><p>To illustrate the concept of quasi-experimental design, I will explain the intuition behind one of the methods called Regression Discontinuity Design (RDD) used to measure the impact of alcohol consumption on mortality.</p><p>The idea behind a RDD is to exploit a discontinuity in treatment allocation (e.g., a geographical border, an age-related administrative law, etc.) where individuals or places very similar to each other receive different treatments based on a cutoff point.</p><p>For instance, in the study “<em>The Effect of Alcohol Consumption on Mortality”</em> by Carpenter and Dobkin (2009), the authors used the discontinuity at the minimum legal drinking age to examine the immediate effect of alcohol consumption on mortality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/384/1*S9v9OP67dt9z5gozu5JQfg.png" /><figcaption>The mortality rate related to moving vehicle accidents jumps suddenly when people turn 21 years old arguably caused by alcohol consumption. Replication of (Carpenter and Dobkin (2009)) main result. Image by author.</figcaption></figure><p>The rationale in this study is that individuals who consume alcohol are not usually comparable to those who don’t due to many other systematic differences (e.g. age, socio-economic statuts, risk of various diseases, etc.). However, by comparing individuals just below and just above the age of 21, one can argue they are very similar, assuming no other significant changes occur at that age. This approach allows for a clearer attribution of changes in mortality rates to alcohol consumption. This example, complete with data and code, is also included in the first reference provided at the end of the next sub-section 4.2 Methods.</p><h4>4.2 Methods</h4><p>There are numerous methods available. However, I recommend you learn the following three methods in this order:</p><ol><li>Regression Discontinuity Design (RDD)</li><li>Difference-in-Differences (Diff-in-Diff)</li><li>Instrumental Variables (IV)</li></ol><p>These three methods are standard in academic research but are also used in the industry. Together, they will provide a toolbox that allows you to address causal questions in various settings.</p><h4>📚Resources:</h4><ul><li>Short intro: <a href="https://matheusfacure.github.io/python-causality-handbook/16-Regression-Discontinuity-Design.html">Regression Discontinuity Design (Causal Inference for the brave and true)</a></li><li>In depth: <a href="https://mixtape.scunning.com/06-regression_discontinuity">Regression Discontinuity Design (Causal Inference: The Mixtape)</a></li><li>Short intro: <a href="https://matheusfacure.github.io/python-causality-handbook/13-Difference-in-Differences.html">Diff-in-Diff (Causal Inference for the brave and true)</a></li><li>In depth: <a href="https://mixtape.scunning.com/09-difference_in_differences">Diff-in-Diff (Causal Inference: The Mixtape)</a></li><li>Example of Diff-in-Diff: <a href="https://towardsdatascience.com/how-ai-is-changing-the-way-we-code-36ff30262e65">ChatGPT effect on Stack Overflow</a></li><li>Short intro: <a href="https://matheusfacure.github.io/python-causality-handbook/08-Instrumental-Variables.html">Instrumental Variables (Causal Inference for the brave and true)</a></li><li>In depth: <a href="https://mixtape.scunning.com/07-instrumental_variables">Instrumental Variables (Causal Inference: The Mixtape)</a></li></ul><h3>6. Advanced topics</h3><p>Of course, causal inference is very rich. Mastering those tools will already equip you very well to work in causal inference. However, other potential topics could be explored as well. Here is a list:</p><h4>📚 Synthetic Control (very popular and fastest growing quasi-experimental method) :</h4><ul><li>Short intro: <a href="https://matheusfacure.github.io/python-causality-handbook/15-Synthetic-Control.html">Synthetic Control (Causal Inference for the brave and true)</a></li><li>Short intro: <a href="https://matheusfacure.github.io/python-causality-handbook/25-Synthetic-Diff-in-Diff.html">Synthetic Diff-in-Diff (Causal Inference for the brave and true)</a></li><li>In depth: <a href="https://mixtape.scunning.com/10-synthetic_control">Synthetic Control (Causal Inference: The Mixtape)</a></li></ul><h4>📚 Causal Machine Learning (joining the best of the two worlds of causal inference and Machine Learning):</h4><ul><li><a href="https://www.gsb.stanford.edu/faculty-research/labs-initiatives/sil/research/methods/ai-machine-learning/short-course">Machine Learning &amp; Causal Inference: A Short Course</a> (Stanford University)</li><li><a href="https://arxiv.org/abs/2206.15475">Causal Machine Learning: A Survey and Open Problems ((Kaddour et al. 2022))</a></li><li>!Not Free!<a href="https://mitpress.mit.edu/9780262545914/causal-analysis/"> Causal Analysis: Impact Evaluation and Causal Machine Learning with Applications in R</a> (Martin Huber)</li></ul><h4>📚 Causal Discovery</h4><ul><li>!Not Free! <a href="https://www.packtpub.com/product/causal-inference-and-discovery-in-python/9781804612989">Causal Inference and Discovery in Python</a></li></ul><h3>6. Conclusion</h3><p>Following this guide will equip you with a solid theoretical foundation and a comprehensive toolbox for incorporating causal inference into your skillset. These concepts extend beyond measuring causal effects for your job; they enhance your critical thinking abilities. Abusive causal claims are widespread in the news, politics, by ‘experts’, managers, and others. Applying these concepts can reduce the risk of manipulation. This toolbox can also be valuable in everyday discussions with friends or during team meetings when making decisions or reflecting on observations.</p><p>Of course, this is just the beginning. To fully integrate and master these skills, practice is essential. I encourage you to select a relevant question, find data, and use the tools provided in this guide to attempt to answer it. Then, include this project in your portfolio to showcase your new knowledge.</p><p><strong>Enjoy this exciting journey on the path to answerig the never-ending question of ‘Why?’.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=98503abc0a06" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-learn-causal-inference-on-your-own-for-free-98503abc0a06">How to Learn Causal Inference on Your Own for Free</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Custom GPT: Lessons and Tips]]></title>
            <link>https://medium.com/data-science/building-a-custom-gpt-lessons-and-tips-cd1b892ec738?source=rss-a52dcb9793ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd1b892ec738</guid>
            <category><![CDATA[gpt]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[causality]]></category>
            <category><![CDATA[causal-inference]]></category>
            <dc:creator><![CDATA[Quentin Gallea, PhD]]></dc:creator>
            <pubDate>Wed, 15 Nov 2023 23:36:16 GMT</pubDate>
            <atom:updated>2023-11-16T00:06:53.587Z</atom:updated>
            <content:encoded><![CDATA[<h4>From enthusiasm to disappointment and finally the path to solutions and appreciation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*-TLKH0uh4YQ0jj_c0xRDaw.png" /><figcaption>The Causal Mindset (personalized GPT by Quentin Gallea), generated with Dall-E.</figcaption></figure><p>Last Tuesday (6th November 2023), Sam Altman (OpenAI CEO), revealed the release of the GPTs, which allow anyone to create a personalized ChatGPT using natural language.</p><p>Like many people, I jumped on the hype train, and spent my days and nights on it over the last weeks, to the point where I felt like my brain was melting. It was quite a rollercoaster, as I was first amazed, then disappointed. But finally, I found solutions to problems, embraced the limitations, and I am now enthusiastic about it.</p><p><strong>What you will find in this article: </strong>In this article, I will first present my App to provide context, then go through the three phases: Enthusiasm (How does this work?), Disappointment (What are the issues?), and Acceptance (How to solve the issues and embrace the limitations?).</p><h3>1. My App: The Causal Mindset</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/494/1*YpTg7oV5H_3RzPxavMnTjQ.png" /><figcaption>The Causal Mindset App. Image by author. More on thecausalmindset.com</figcaption></figure><p>I’ve been on a mission to make causal inference accessible for years to help people make better decisions and reduce the risk of manipulations. Over the last decade, I taught statistics to about 12,000 people, mostly in academia. <br>Beyond, theoretical and empirical class, I created a framework, “The Causal Mindset”, which is essentially a set of practical critical thinking tools (without math) grounded in causal inference and statistics (I post weekly on LinkedIn and Instagram).</p><p>Distinguishing between facts and fiction is crucial for making informed decisions and protecting oneself from manipulation. Unfortunately, this task is becoming increasingly challenging in the face of the overwhelming surge of information; misinformation, deceptive articles, and outright lies abound.</p><p>Fact-checking isn’t always practical, as it can be time-consuming and often assumes that there is a definitive “correct” answer available.</p><p>The aim of this app is to offer a versatile tool that can be used anytime, anywhere to uncover flaws in arguments and enhance decision-making. Furthermore, it seeks to empower you with these critical thinking tools, fostering independence from the app itself.</p><p><strong>How does my app work?</strong><br>You can share a statement, a graph, or a reflection with the Chatbot and it will apply the Causal Mindset Framework to dissect and question the validity of the claim.</p><p>You can find examples or try them yourself on <a href="http://thecausalmindset.com/">thecausalmindset.com</a>. But this is not the point of this article. I rather want to present to you what I learned in the process that might be useful for you.</p><h3><strong>2. Enthousiasm (How does this work?)</strong></h3><p>Now, if you have access to chatGPT Plus you can enter the “playground” and create your own personalized GPT. OpenAI already released some of their homemade GPTs (this is how the company refers to those), and plans to roll out an “App store” of the GPTs produced by the users in the near future (cf: <a href="https://openai.com/blog/introducing-gpts">https://openai.com/blog/introducing-gpts</a>).</p><p>The fundamental aspect is that instead of coding, you use natural language to setup and tune the app. Here how this playground looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N7gApmJPtHRj0lTBtBUUTQ.png" /><figcaption>Image by author, prinscreen of the lab to create your personlized GPT</figcaption></figure><p>On the left, you can chat with the GPT Builder to set things up, and on the right, you can try it out. So, essentially, this is how I started. I gave instructions, asked advice from the builder as well, and tested the result on the right.</p><p>On the top left, you can also click Configure, and there you have access to other aspects of the app directly (see image below). You can also fill all those fields by discussing with the GPT Builder on the “Create” tab. Note that the Builder can decide to adapt the content of those fields depending on the conversation you have (this will become important in the next part).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/1*k88VjR26jJ_O5CuQcIHPFg.png" /><figcaption>Image by author, prinscreen of the lab to create your personlized GPT</figcaption></figure><p>You can add an icon or generate it with Dall-E in the conversation. You can also add a description that is always visible next to the name of the App. Then you have the instruction part: What does this GPT do? How does it behave? What should it avoid doing? Finally, you have the conversation starters that people can pick to test the app.</p><p>Below you have additional options:</p><p><strong>Knowledge: </strong>this is the knowledge base. You can put files and your GPT will prioritize the information coming from this part. Hence this is key to distinguish from the basic ChatGPT.</p><p><strong>Capabilities:</strong> You can also select its capabilities: Web browsing, Dall-e image generation, and code interpreter.</p><p><strong>Actions:</strong> The Add actions button:<em> “You can make third-party APIs available to your GPT by providing details about the endpoints, parameters, and a description about how the model should use it.”</em></p><p>At first, it was incredible. It felt like training someone. I gave my knowledge base to the GPT (my articles, transcript of my TEDx, and book I wrote about my framework The Causal Mindset). It was fascinating, it felt like the Matrix. Instead of reading my book, I would upload it to a machine, and people could access the knowledge within it.</p><blockquote>But quickly, it started to fall apart.</blockquote><h3>3. Disappointment (What are the issues?)</h3><p>Similarly to teaching someone, the GPT would start forgetting things. So, I started to fix again things that I fixed before. When the latest issue was solved, I would realize that something else had changed and I had to fix it again. Hence, very quickly I was stuck in a loop and it drove me nuts. Worse than this, the model would keep changing the conversation starters, the instruction, the description etc. Other people who were working on this felt the same.</p><blockquote>Some colleagues and friends threw the thing away and claimed that it was just too preliminary. I didn’t.</blockquote><p>Then, I also realized that users could find my knowledge base, my instructions and even the conversation I used to build the App (‘prompt injection’). This was also quite a bummer (see image below).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/1*Vhxk-bXqiSShIR49-r4vRw.png" /><figcaption>Example of how I got the list of documents in the knowledge base as a user of the App. Image by author.</figcaption></figure><h3>4. Acceptance (How to solve the issues and embrace the limitations?)</h3><p>I was not going to give up already. So here are the solutions I found to all those problems:</p><h4>4.1 Privacy</h4><p>I gave up on the privacy aspects and shared only content that I already shared online and anyone could access (not a solution suitable for anyone and any situation). Moreover, instead of aiming to protect how I instructed my model, I thought that it would be better to share it to allow other people to improve their GPTs (hence this article). So I embraced the “open source” aspect of it. Note that they might be working on this aspect and deactivating the Code Interpreter might reduce slightly the risks.</p><h4>4.2 User Guide</h4><p>When you arrive on the App, you have limited information. I have a full website with examples, and a guide elsewhere (<a href="http://thecausalmindset.com/">thecausalmindset.com</a>) but on the app itself, you just have your short description which is often not enough. So, here was my first idea. I included as the first conversation starter “How does this app work?” so people would be able to click this as soon as they arrive on the app. Then, I instructed the GPT to always answer the same thing exactly (see image below). I aimed for something short with examples to invite the people to try rather than reading a booklet.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/793/1*XMpuLbYKhtgA6ccLtUAI8Q.png" /><figcaption>My short user guide. Image by author.</figcaption></figure><h4><strong>4.3 How to prevent everything changing all the times</strong></h4><p>While this method was great, sometimes the app would still “forget” if I instructed the app to do other things. This is where I’ve made the biggest update.</p><blockquote>I completely changed the way I work with the Builder.</blockquote><p>Instead of just using the chat, I created an instruction file containing all the key aspects of the app. This file is in the knowledge base and also pasted almost entirely into the Instructions (the max length is 80000 characters).</p><blockquote><strong>Hence, instead of discussing those instructions for hours, I decided to write the instructions, past them, added them to the knowledge base, and instructed only through the chat to follow this carefully at all times (full instructions can be found in the Appendix of this article).</strong></blockquote><p>Here is the content of my instructions:</p><p><strong>Main rules for the App:</strong> Here I gave the “10 commandments of the app”.</p><p><strong>How does this app work?: </strong>I gave the description shown above that I want the app to tell users.</p><p><strong>Core analysis structure:</strong> This is the core of my method. I instructed the model to always answer following this structure first (it is also in my main rules).</p><p><strong>The causal mindset app instruction: </strong>Those are secondary instructions on the whole procedure beyond just the core analyses.</p><p><strong>Writing style: </strong>How I want it to be written (tone, style, etc).</p><p><strong>Conversation starters:</strong> Here I put all my conversation starters.</p><h3>5. Conclusion</h3><p>In essence, I completely changed my approach to creating GPTs after this experience. Instead of discussing for hours with a model, I would prepare an instruction file, upload it, and only ask the App to follow this carefully. Et voilà. If you want to protect your content, you might be willing to wait a bit as this is something that I could not solve so far.</p><p>I have seen some skepticism about this new concept: the GPTs. The main point I have heard was about the fact that it remains essentially ChatGPT. I disagree with this.</p><blockquote>I think that it can be way more than just a subtly different version of ChatGP, and I am convinced that it can be tremendously useful, potentially opening a door to unforeseen possibilities for two main reasons.</blockquote><p>First, pre-prompting is valuable. Even if you know where you want to go, it might take hours or days. And even if it takes minutes, it prevents you from redoing it numerous times. Moreover, it allows benefiting from the expertise of others.</p><p>Second, the knowledge base makes a huge difference. I have worked for years to create the causal mindset. Hence, even if it takes you minutes to set up it might represent the culmination of a decade of research.</p><p>I am looking forward to see what are the possibilites, what you did and what you think.</p><p>If you want to test and provide feedback in the comments below, here it my app: <a href="http://thecausalmindset.com/">thecausalmindset.com</a></p><h3>APPENDIX: My instruction file</h3><h4>Main rule for the App:</h4><p>· Make every word count and rely extensively on examples to illustrate your point.</p><p>· Always redirect the discussion towards a causal analysis.</p><p>· If someone is asking something unrelated, recall your purpose and goal and suggest examples of questions people could ask you (for example using the conversation starters.</p><p>· Always start the first part of the analysis with the Core Analyses described below.</p><p>· Prioritizing Uploaded Documents: The app gives priority to information from uploaded documents for its analyses and refers to these documents as its main knowledge source.</p><p>· Adherence to Facts: The app sticks to the facts presented in the documents and avoids speculation. It heavily favors the knowledge provided in these documents before falling back on baseline knowledge or other sources.</p><h4>How does this app work?</h4><p>The Causal Mindset helps you to separate facts from fiction.</p><p>Share a statement, a graph, or a reflection with me and I’ll apply the Causal Mindset Framework to dissect and question the validity of the claim.</p><p>Examples:</p><p>· I followed a fitness program for a month, can I attribute my increased performance to this program?</p><p>· I have read a scientific article claiming that cold showers significantly reduce work sick leave. They did a controlled experiment in the Netherlands with 4000 participants and collected self-reported data.</p><p>· I have seen a free e-scooter company in a city center claiming that they have helped to reduce emissions because using an e-scooter is more environmentally friendly than driving a car.</p><h4>Core analysis’s structure:</h4><p>The Causal Mindset App should always start by providing an analysis of the causal questions or situations following this structure:</p><p>- Recall the user that you are going to apply the causal mindset framework to analyze the situation.</p><p>- Quick Feedback — Major Flaw: Describe with a clear example what the main problem in the situation presented.</p><p>- Is there something else? Here you should basically find if there are alternative explanations for the causal effect suggested or at least some confounding effect (omitted variable bias). If reverse causation is a problem, you should also mention it here (but not necessarily to mention if it is not the case).</p><p>- Can We Extrapolate? You should question the external validity with at least one clear example.</p><p>- Counterfactual: Here you should present the potential counterfactual to invite the person to reflect if the comparison in their example is valid or not.</p><p>- Common Biases: Here, potential additional statistical or behavioral biases (e.g. placebo effect, selection bias, confirmation bias, desirability bias, etc.)</p><p>- Offering Solutions: Suggest an explanation of an experiment or natural experiment to measure properly causality for a broad audience.</p><p>- Conclusion: Finish the analysis by inviting the person to go further by asking for an in-depth analysis depending on their role:</p><p>o Journalists/Content Creators/critical thinking enthusiasts: For those involved in media and content creation, the app suggests further exploration into source verification and bias detection. This is critical for ensuring the accuracy and impartiality of their work.</p><p>o Professionals/Decision-Makers: For this group, the app provides advanced modules for risk assessment and decision analysis. These tools are essential for making informed decisions in complex, high-stakes environments.</p><p>o Students/Researchers: The app recommends educational resources on causal inference and opportunities for academic collaboration. This is particularly beneficial for those in academic settings or those looking to deepen their understanding of causal analysis.</p><h4>The causal mindset app instruction:</h4><p>The Causal Mindset app is structured to guide users through a comprehensive framework of causal analysis. Here’s how it works:</p><p><strong>Core Analysis Phase: </strong>When a user presents a situation or a problem, the app first conducts a fundamental assessment of causality. This involves identifying key points, potential biases, and relationships between variables. The focus here is on understanding the causal structure of the problem, grounded in principles of causal inference.</p><p><strong>Tailored Deep-Dive: </strong>Based on the initial analysis, the app then offers more detailed exploration tailored to the specific role or needs of the user:</p><p>- Journalists/Content Creators: For those involved in media and content creation, the app suggests further exploration into source verification and bias detection. This is critical for ensuring the accuracy and impartiality of their work.</p><p>- Professionals/Decision-Makers: For this group, the app provides advanced modules for risk assessment and decision analysis. These tools are essential for making informed decisions in complex, high-stakes environments.</p><p>- Students/Researchers: The app recommends educational resources on causal inference and opportunities for academic collaboration. This is particularly beneficial for those in academic settings or those looking to deepen their understanding of causal analysis.</p><p><strong>User Interaction</strong></p><p>- The app interacts with users through a series of questions and analyses, prompting them to consider various aspects of their situation.</p><p>- It might use real-world examples or hypothetical scenarios to illustrate points and help users apply causal thinking to their specific contexts.</p><p>- The app could potentially use visual tools like causal diagrams or flowcharts to help users visualize complex relationships.</p><p><strong>Additional Features</strong></p><p>- <strong>Educational Content:</strong> The app may include tutorials, articles, and case studies on causal inference to educate users about key concepts.</p><p>- <strong>Interactive Exercises:</strong> To reinforce learning, the app might offer interactive exercises or simulations that allow users to practice applying causal analysis in different scenarios.</p><p><strong>Goal of the App</strong></p><p>The primary goal is to empower users with the skills to make better decisions, backed by a solid understanding of causal relationships.</p><p>It aims to enhance critical thinking and analytical skills, helping users become more immune to misinformation and better equipped to handle the complexities of their professional and personal lives.</p><p>This approach ensures that users receive immediate, practical value from the app, while also having the option to engage with more in-depth and specialized content according to their specific needs and expertise level.</p><h4>Writing style:</h4><p>· Informative and Educational: The writing primarily serves to educate the reader, delivering complex concepts of causality and statistical analysis in an accessible manner. It aims to impart knowledge and understanding in these areas.</p><p>· Analytical and Thought-Provoking: The style is analytical, encouraging readers to engage deeply with the subject matter. It prompts critical thinking and challenges readers to question and explore the causal relationships in various scenarios.</p><p>· Structured and Clear: The text is well-structured, presenting ideas in a logical sequence. This clarity makes complex topics more understandable, especially beneficial for readers new to the concepts of causal analysis and critical thinking.</p><p>· Engaging with Real-World Examples: The writing often incorporates real-world scenarios and examples, which helps in grounding abstract concepts in practical situations. This approach makes the material more relatable and easier to grasp.</p><p>· Interactive and Inclusive: The text invites reader interaction, posing questions and scenarios that encourage readers to apply the concepts they are learning. This interactive style enhances learning and retention.</p><p>· Academically Rigorous Yet Accessible: The writing style strikes a balance between academic rigor and accessibility. It is evident that the text is grounded in thorough research and expertise, yet it’s presented in a way that is approachable to a broader audience.</p><p>· Conversational at Times: In some parts, the writing adopts a more conversational tone, which adds to its engaging quality. This style helps in demystifying statistical concepts and makes the content more approachable.</p><p>Overall, the writing style is conducive to learning and engagement, particularly suited for those seeking to understand and apply causal analysis in various aspects of their professional, academic, or personal lives.</p><h4>Conversation starters:</h4><p>How does this app work?</p><p>Example: Fitness program: I followed a fitness program for rowing for a month. I tested today and went faster. Is it thanks to the program?</p><p>Example: Environmental policy evaluation: In September 2022, the Swiss government launched a national advertising campaign in favor of energy sobriety in response to the threat of shortages due to the war in Ukraine. It presented a graph showing that Switzerland’s net consumption per month fell after the implementation of this policy (during winter).</p><p>Example: Company impact: I saw a company that provides free electric scooters in city centers across Europe and claims that their service reduces pollution. They have used life cycle analysis to compare the pollution from one kilometer in a car to one kilometer with their e-scooter.</p><p>Example: Cold shower research paper: I saw a peer-reviewed article about a randomized control trial in which 2000 volunteers in the Netherlands were split between a cold shower and a normal shower every day. They found that those taking cold showers missed work due to sickness 1/3 less often than the control group. All results are self-reported. What do you think?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd1b892ec738" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/building-a-custom-gpt-lessons-and-tips-cd1b892ec738">Building a Custom GPT: Lessons and Tips</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How AI Is Changing the Way We Code]]></title>
            <link>https://medium.com/data-science/how-ai-is-changing-the-way-we-code-36ff30262e65?source=rss-a52dcb9793ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/36ff30262e65</guid>
            <category><![CDATA[causal-inference]]></category>
            <category><![CDATA[thoughts-and-theory]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[econometrics]]></category>
            <dc:creator><![CDATA[Quentin Gallea, PhD]]></dc:creator>
            <pubDate>Wed, 23 Aug 2023 19:28:51 GMT</pubDate>
            <atom:updated>2023-09-04T07:13:18.720Z</atom:updated>
            <content:encoded><![CDATA[<h4>Evidence from ChatGPT and Stack Overflow</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EGt5Gh4UXpQhXsB-zpLGfg.jpeg" /><figcaption>Photo by Pavel Danilyuk from <a href="https://www.pexels.com/fr-fr/photo/femme-technologie-vin-porter-8439089/">Pexels</a></figcaption></figure><p><strong>In short: </strong>In this article, you will find a summary of my latest research on AI and work (exploring the effect of AI on productivity while opening up the discussion on the long-term effects), an example of a quasi-experimental method (Difference-in-Difference) illustrated with ChatGPT and Stack Overflow, and see how you can extract data from Stack Overflow with a simple SQL query.</p><p>Link to the full scientific article (please cite): <a href="https://arxiv.org/abs/2308.11302">https://arxiv.org/abs/2308.11302</a></p><p>As with most technological revolutions, ChatGPT’s release was accompanied by fascination and fear. On one hand in just two months, with 100 millions monthly active users, the app broke the record for the fastest-growing consumer application in history. On the other hand, a report by Goldman Sachs claimed that such technology could replace more than 300 millions jobs globally [1]. Additionally, Elon Musk alongside more than 1,000 tech leaders and researchers signed an open letter urging for a pause on the most advanced AI developments [2].</p><blockquote>``We can only see a short distance ahead, but we can see plenty that needs to be done.’’ Alan Turing</blockquote><p>In line with Alan Turing’s quote, this article does not seek to predict heroically the distant future of AI and its impacts. However, I focus on one of the main observable consequences affecting us: How AI is changing the way we code.</p><p>The world changed with the birth of ChatGPT. At least, as someone who codes every day, my world changed overnight. Instead of spending hours on Google to find the right solution or digging into the answers on Stack Overflow and translating the solution to my exact problem with the right variables names and matrices dimensions, I could just ask ChatGPT. The chatbot would not only give me an answer in a blink of an eye but the answer would fit my exact situation (e.g. correct names, dataframes dimensions, variable types, etc.). I was blown away, and my productivity jumped suddenly.</p><p>Hence, I decided to explore the large-scale effect of ChatGPT release and its potential effect on productivity and ultimately on the way we work. I defined three hypotheses (Hs) that I tested using Stack Overflow data.</p><p><strong>H1: ChatGPT decreases the number of questions asked on Stack Overflow. </strong>If ChatGPT can solve coding problems in seconds, we can expect a fall of questions on coding community platforms where asking a question and getting an answer takes time.</p><p><strong>H2: ChatGPT increases the quality of the questions asked. </strong>If ChatGPT is used largely, the remaining questions on Stack Overflow must be better documented as ChatGPT might have already helped a bit.</p><p><strong>H3: The remaining questions are more complex. </strong>We can expect that the remaining questions are more challenging as they could potentially not be answered by ChatGPT. Hence, to test this we are testing if the proportion of unanswered questions increases. In addition, I also test if the number of views per question changes. If the number of views per question is stable it would be an additional sign that the complexity of the remaining questions is increased and that this finding is not only caused by the reduced activity on the platform.</p><p>To test those hypotheses, I will exploit the sudden release of ChatGPT on Stack Overflow. In November 2022, when OpenAI released publicly their chatbot, no other alternatives were available (e.g. Google Bard), and the access was free (not limited to paid subscription as with OpenAI ChatGPT 4 or Code Interpreter). Hence it is possible to observe how the activity changed in the online coding community before and after the shock. However, despite how ‘clean’ this shock is, other effects might be confounded and hence question causality. In particular, seasonality (e.g. end of the year holidays after the release) as well as the fact that the more recent the question is, the lower the number of views and the probability that an answer is found.</p><p>Ideally, to mitigate the influence of potential lingering confounding variables such as seasonality and measure a causal effect, we would like to observe the world without ChatGPT release which is impossible (e.g. the fundamental problem of causal inference). Nevertheless, I will address this challenge by exploiting the fact that the quality of the answers of ChatGPT for coding-related issues varies from one language to another and use quasi-experimental methods to limit the risk of other factors confounding the effect (Difference-in-Difference).</p><p>To do so, I will compare the activity on Stack Overflow between Python and R. Python is an obvious choice as it, is arguably, one of the<br>most popular programming languages used (e.g. ranked 1st in the TIOBE<br>Programming Community Index). The large set of resources online for Python provides a rich training set for chatbots like ChatGPT. Now, to compare with Python, I chose R. Python is often cited as the best replacement for R and both are freely available. However, R is somewhat less popular (e.g.~16th in the TIOBE Programming Community index) and hence the training data might be smaller, implying poorer performance by ChatGPT. Anecdotal evidence confirmed this difference (more details on the method in the Method section). Hence, R represents a valid counter factual for Python (it is affected by seasonality but we can expect a negligible effect of ChatGPT).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qw4QVr2S1uLy8rIC_czDXw.png" /><figcaption>Figure 1: The effect of ChatGPT on weekly number of questions on Stack<br>Overflow (figure by the author)</figcaption></figure><p>The Figure above presents the raw weekly data. We can witness the sudden and important drop (21.2%) in the number of questions asked weekly on Stack Overflow about Python after the release of ChatGPT 3.5 while the effect on R is somewhat smaller (drop of 15.8%).</p><p>These ‘qualitative’ observations are confirmed by the statistical model. The econometric model described later finds a statistically significant drop of 937.7 (95% CI: [-1232.8,-642.55 ] ; p-value = 0.000) weekly questions on average for Python on Stack Overflow. The subsequent analysis, utilizing the Diff-in-Diff method, further unveils an improvement in question quality (measured on the platform by a score), alongside an increase in the proportion of questions remaining unanswered (while the average number of views per question seems unchanged). Consequently, this study provides evidence for the three hypotheses defined earlier.</p><p><strong>These findings underscore the profound role of AI in the way we work. By addressing routine inquiries, generative AI empowers individuals to channel their efforts toward more complex tasks while boosting their productivity. However, important long-term potential adverse effects are also discussed in the Discussion section.</strong></p><p>The rest of the article will present the Data and Methods, then the Results, and will close with the Discussion.</p><h3>Data</h3><p>The data have been extracted using an SQL query on the <a href="https://data.stackexchange.com/">Stack Overflow data explorer portal</a> (licence: <a href="https://creativecommons.org/licenses/by-sa/4.0/">CC BY-SA</a>). Here is the SQL command used:</p><pre>SELECT Id, CreationDate, Score, ViewCount, AnswerCount<br>FROM Posts<br>WHERE Tags LIKE &#39;%&lt;python&gt;%&#39;<br>AND CreationDate BETWEEN &#39;2022–10–01&#39; AND &#39;2023–04–30&#39;<br>AND PostTypeId = 1;</pre><p>I then aggregated the data by week to reduce the noise and hence obtained a dataset from Monday the 17th of October 2022 to the 19th of March 2023 with information on the number of weekly posts, the number of views, the number of views per questions, the average score per question and the proportion of unanswered question. The score is defined by users of the platform who can vote up or down to say if the question shows <em>“research effort; it is useful and clear”</em> or not.</p><h3>Method</h3><p>In order to measure a causal effect, I use a Difference-in-Difference model which is an econometric method that exploits usually a change over time and compares a treated unit(s) with an untreated group. In order to know more about this method I can recommend you to read the chapter referring to this method in two free e-books: <a href="https://matheusfacure.github.io/python-causality-handbook/13-Difference-in-Differences.html">Causal Inference Inference for the Brave and True</a> and <a href="https://mixtape.scunning.com/09-difference_in_differences">Causal Inference: The Mixtape</a>.</p><p>In simple terms, the Diff-in-Diff model computes a double difference in order to identify a causal effect. Here is a simplified explanation. First, the idea is to compute two simple differences: the ‘average’ difference between the pre (before ChatGPT release) and post-period for the two groups treated and untreated (here respectively Python and R questions). What we care about is the effect of the treatment on the treated units (here is the effect of ChatGPT release on Python questions). However, as said earlier, there might be another effect still confounded with the treatment (e.g. seasonality). In order to address this issue, the idea of the model is to compute a double difference, in order to check how the first difference for the treated (Python) is different from the second (difference for the control group, R). As we expect no treatment effect (or negligible) on the control group, while still affected by seasonality for example, we can get rid of this potential confounding factor and ultimately measure a causal effect.</p><p>Here is a slightly more formal representation.</p><p>First difference for the treated group:</p><p>E[Yᵢₜ| Treatedᵢ, Postₜ]-E[Yᵢₜ| Treatedᵢ, Preₜ] = λₜ+β</p><p>Here i and t refer respectively to the language (R or Python) and to the week. While treated refer to the questions related to Python and Post refers to the period when ChatGPT was available. This simple difference might represent the causal effect of ChatGPT (β) + some time effect λₜ (e.g. seasonality).</p><p>First difference for the control group:</p><p>E[Yᵢₜ| Controlᵢ, Postₜ]-E[Yᵢₜ| Controlᵢ, Preₜ] = λₜ</p><p>The simple difference for the control group does not include the treatment effect (as it is untreated) but only the λₜ.</p><p>Hence the double difference will give:</p><p>DiD = ( λₜ+β) — λₜ = β</p><p>Under the assumption that the λₜ are identical for both groups (parallel trend assumption, discussed below), the double difference will allow us to identify β, the causal effect.</p><p>The essence of this model lies in the parallel trend assumption. In order to claim a causal effect we should be convinced that without ChatGPT the evolution of posts on Stack Overflow for Python (treated) and for R (untreated) would be the same in the treatment period (after November 2022). However, this is obviously impossible to observe and hence to test directly (c.f. the Fundamental Problem of Causal Inference). (If you want to learn more about this concept and causal inference find my videos and articles on Towards Data Science: <a href="https://medium.com/towards-data-science/the-science-and-art-of-causality-part-1-5d6fb55b7a7c">the Science and Art of Causality</a>). However, it is possible to test if the trends are parallel before the shock, suggesting that the control group is a potentially good “counterfactual”. Two different placebo tests made with the data revealed that we cannot reject the parallel trend assumption for the pre-ChatGPT period (p-values of the tests respectively 0.722 and 0.397 (see online APPENDIX B)).</p><p><strong>Formal definition:</strong></p><p><strong>Yᵢₜ = β₀ + β₁ Pythonᵢ + β₂ ChatGPTₜ + β₃ Pythonᵢ × ChatGPTₜ + uᵢₜ</strong></p><p>“i” and “t” correspond respectively to the topic of the question on Stack Overflow (i ∈ {R; Python}) and the week. Yᵢₜ represents the outcome variable: Number of questions (H1), Average question score (H2), and proportion of unanswered questions (H3). Pythonᵢ is a binary<br>variable taking the value 1 if the question is related to Python and 0<br>otherwise (related to R). ChatGPTₜ is another binary variable<br>taking the value 1 from the release of ChatGPT and onwards and 0<br>otherwise. uᵢₜ is an error term clustered at the coding language<br>level (i).</p><p>The essence of this model lies in the parallel trends assumption. In order to claim a causal effect we should be convinced that without ChatGPT the evolution of posts on Stack Overflow for Python (treated) and for R (untreated) would be the same in the treatment period (after November 2022). However, this is obviously impossible to observe and hence to test directly (c.f. the Fundamental Problem of Causal Inference). (If you want to learn more about this concept and causal inference find my videos and articles on <a href="https://medium.com/towards-data-science/the-science-and-art-of-causality-part-1-5d6fb55b7a7c">the Science and Art of Causality</a>). However, it is possible to test if the trends are parallel before the shock, suggesting that the control group is a good “counterfactual”. In this case, two different placebo tests reveal that we cannot reject the parallel trends assumption for the pre-ChatGPT period (p-values of the tests respectively 0.722 and 0.397 (see online APPENDIX B)).</p><h3>Results</h3><h4>H1: <strong>ChatGPT decreases the number of questions asked on Stack Overflow.</strong></h4><p>As presented in the introduction, the Diff-in-Diff model estimates a statistically significant drop of 937.7 (95% CI: [-1232.8, -642.55] ; p-value = 0.000) weekly questions on average for Python on Stack Overflow (see Figure 2 below). This represents a fall of 18% in weekly questions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N9L4an0PbHw4Plc4ylutJQ.png" /><figcaption>Figure 2: The effect of ChatGPT on the weekly number of questions (image by author)</figcaption></figure><h4>H2: <strong>ChatGPT increases the quality of the questions asked.</strong></h4><p>ChatGPT might be helpful to answer questions (c.f. H1). However, when the chatbot cannot solve the issue, it is possible that it allows one to go further and get more information on the problem or some element of the solution. The platform allows us to test this hypothesis as users can vote for each question if they think that <em>“This question shows research effort; it is useful and clear” </em>(increase the score by 1 point)<em>,</em> or not (decrease the score by 1 point)<em>. </em>This second regression estimates that there is a 0.07 points (95% CI: [-0.0127 , 0.1518 ]; p-value: 0.095) increase in the questions’ score on average (see Figure 3) which represents a 41.2% increase.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cCIrUhpHDcc95YN30xQojQ.png" /><figcaption>Figure 3: The effect of ChatGPT on quality of the questions (image by author)</figcaption></figure><h4>H3: The remaining questions are more complex.</h4><p>Now that we have some pieces of evidence that ChatGPT is able to provide significant help (solve questions and help document the others), we would like to confirm that the remaining questions are more complex. To do so, we are going to look at two things. First, I find that the proportion of unanswered questions is raising (no answer could be a sign that the questions are more complex). More precisely I find a 2.21 percentage point (95% CI: [ 0.12, 0.30]; p-value: 0.039) increase in the proportion of questions unanswered (see Figure 4) which represents an increase of 6.8%. Second, we also find that the number of views per question is unchanged (we cannot reject the null hypothesis that it is unchanged, with a p-value of 0.477). This second test allows us to partially discard the alternative explanation that there are more unanswered questions due to the lower traffic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N9tvlLFljqwx8SvCH9H0Wg.png" /><figcaption>Figure 4: The effect of ChatGPT on the proportion of unanswered question (image by author)</figcaption></figure><h3>Discussion</h3><p>These findings support the view that generative AI could revolutionize our work by taking care of routine questions, allowing us to focus on more complex problems requiring expertise while boosting our productivity.</p><p>While this promise sounds exciting there is a reverse of the medal. First, low-qualified work might be replaced by chatbots. Second, such tool might affect (negatively) the way we learn. Personally, I see coding as biking or swimming: watching videos or following classes is not enough, you have to try and fail yourself. If the answers are too good and we don’t force ourselves to study, many people might struggle to learn. Third, if the mass of questions on Stack Overflow fall, it might reduce a valuable source for the training set of generative AI models hence, affecting their long-term performance.</p><p>All those long term adverse effects are not clear yet and require careful analysis. Let me know what you think in the comments.</p><p>[0] Gallea, Quentin. “From Mundane to Meaningful: AI’s Influence on Work Dynamics — evidence from ChatGPT and Stack Overflow” arXiv econ.GN (2023)</p><p>[1] Hatzius, Jan. “The Potentially Large Effects of Artificial Intelligence on Economic Growth (Briggs/Kodnani).” <em>Goldman Sachs</em> (2023).</p><p>[2] <a href="https://www.nytimes.com/2023/03/29/technology/ai-artificial-intelligence-musk-risks.html">https://www.nytimes.com/2023/03/29/technology/ai-artificial-intelligence-musk-risks.html</a></p><p>[3] Bhat, Vasudev, et al. “Min (e) d your tags: Analysis of question response time in stackoverflow.” <em>2014 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM 2014)</em>. IEEE, (2014)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=36ff30262e65" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-ai-is-changing-the-way-we-code-36ff30262e65">How AI Is Changing the Way We Code</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Save your A/B testing by avoiding those 3 costly mistakes]]></title>
            <link>https://medium.com/data-science/save-your-a-b-testing-by-avoiding-those-3-costly-mistakes-6ff2e4effe22?source=rss-a52dcb9793ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/6ff2e4effe22</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[causality]]></category>
            <category><![CDATA[ab-testing]]></category>
            <dc:creator><![CDATA[Quentin Gallea, PhD]]></dc:creator>
            <pubDate>Fri, 07 Jul 2023 14:36:21 GMT</pubDate>
            <atom:updated>2023-07-07T14:36:21.596Z</atom:updated>
            <content:encoded><![CDATA[<h4>The details that will make the difference</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EJMMKJ6_apFQIo8MYxbQNA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@woodpecker65?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Christian Stahl</a> sur <a href="https://unsplash.com/fr/photos/8S96OpxSlvg?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Once exclusively used in academia, in particular medical research, randomized control trials are now a popular method for businesses to make data-driven decisions. In particular, online A/B testing is easy to implement and potentially incredibly powerful to optimize digital processes. By comparing two or more variants, organizations can evaluate the effectiveness of different options and determine the most favorable outcome. However, it is crucial to recognize and address certain limitations to ensure that biases do not impact the reliability and validity of the results. In this article, we explore three key limitations to consider before running an online A/B test to avoid costly bias. Before jumping into the list of what I personally think are the top three issues, let me briefly define A/B testing and important concepts.</p><h4>What is A/B testing?</h4><p>A/B testing involves presenting different versions/variants, A and B, to different study subjects (e.g., clients). Online A/B testing could explore variations of a webpage, email campaign, user interface, or any other digital asset to a subset of users. The variations typically differ in one or more specific elements, such as design, layout, color scheme, call-to-action, or content. Through carefully controlled experiments, organizations can measure the impact of these variations on user behavior, engagement, and conversion rates.</p><h4><strong>Randomization</strong></h4><p>The process begins by dividing the audience randomly into two or more groups, with each group exposed to a different variant. The control group receives the original version (referred to as the baseline or control, as long as an original version exists), while the other groups receive modified versions. By tracking user interactions, such as clicks, conversions, time spent on a page, or any other predefined metrics, organizations can compare the performance of the different variants and determine which one yields the desired outcomes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8cRGiM3QrM208fqzCPADug.png" /><figcaption>Image by Author. Here is a representation of an A/B testing process. First, we split randomly the sample between control and treatment (A and B). Second, we observe the outcome (e.g. conversion rate), represented here by the black/green color.</figcaption></figure><h4>Causality</h4><p>The primary goal of A/B testing is to properly identify the effect of a change. Without following this strategy carefully, other things could affect the behavior of the subjects. Imagine that Netflix decides to change its homepage to show the content that is watched the most at the moment instead of the latest release (this is a hypothetical example). Then, imagine that the company does not use A/B testing but rather changes the platform for everyone in April and then compares the time spent on the platform and the number of subscribers between March and April. The differences could be caused by the homepage change but also a difference in weather, in other online streaming platforms, etc. It would be impossible to identify the cause due to the multiple factors confounded at the same time. A/B testing aims to solve this issue by randomly allocating and testing simultaneously two or more groups. To learn more about causality, I invite you to read my two-part article on the Science and Art of Causality (<a href="https://medium.com/towards-data-science/the-science-and-art-of-causality-part-1-5d6fb55b7a7c">https://medium.com/towards-data-science/the-science-and-art-of-causality-part-1-5d6fb55b7a7c</a>)</p><p>Now, let us dive into three key limitations that organizations should consider before running an online A/B test to avoid costly bias. By understanding and mitigating these limitations, businesses can maximize the value of A/B testing, make more informed decisions, and drive meaningful improvements in their digital experiences.</p><h4>1. Channel: Uncovering the User’s Perspective</h4><p>One of the primary limitations of online A/B testing is understanding the reasons behind user preferences for one option over another. Often, the choice between options A and B is not explicitly justified, leaving experimenters to speculate about user behavior. In scientific research, we call this the “channel,” the reasoning explaining the rationale for the causal effect.<br>Imagine that your option B incorporates an additional feature on the checkout page (e.g. recommendations for similar products or products bought together). You observe a drop in purchases with option B and hence conclude that it was a bad idea. However, a more careful analysis revealed that actually, the time to load the page for option B was longer. Now you have basically two differences: the content and the waiting time. Hence, back to the concept of causality, you don’t know what drives the choice; the two are confounded. If you think that loading time is marginal, think again: <em>“ […] experiments at Amazon showed a 1% sales decrease for an additional 100msec, and that a specific experiments at Google, which increased the time to display search results by 500 msecs reduced revenues by 20%” </em><a href="https://ai.stanford.edu/~ronnyk/2007GuideControlledExperiments.pdf"><em>(</em>Kohavi et al. (2007))</a></p><p><strong>Solutions: </strong>First, to mitigate this limitation, incorporating additional survey questions can provide valuable insights into users’ motivations and hence minimize the risk of biased interpretations. Second, trying to avoid having several differences helps to pin down the cause (e.g. having the same loading time).</p><h4>2. Short-Term vs. Long-Term Impact: Beyond Immediate Results</h4><p>When conducting an online A/B test, it is essential to consider the potential long-term effects of the chosen metric. While short-term objectives, such as click-through rates or immediate conversions, may seem favorable initially, they could have adverse consequences in the long run. For example, employing clickbait strategies may yield quick views and impressions, but they might negatively impact the audience’s perception and your credibility over time.</p><p><strong>Solution: </strong>It is crucial to measure multiple metrics that assess both short-term and long-term impact. By evaluating a comprehensive range of indicators, organizations can make more informed decisions and avoid myopic optimization strategies. Long-term impact metrics could include satisfaction evaluation and audience retention (e.g. time of a video watched or time spent reading an article). That being said, it is not trivial to assess those.</p><h4>3. Primacy and Newness Effects: The Influence of Novelty</h4><p>Two related limitations arise from the influence of novelty in online A/B testing: primacy and newness effects. Primacy effect refers to the fact that experienced users might be confused or lost when encountering a change, such as a button’s placement or color alteration. Conversely, newness effect occurs when users are tempted to interact with a new feature due to its novelty, but this effect may fade quickly. These effects are particularly prevalent in platforms where users have regular interactions, such as social media.</p><p><strong>Solution: </strong>It is recommended to run experiments over several weeks, observing how the effects change over time. By monitoring the fluctuating user behavior, experimenters can gain a more comprehensive understanding of the long-term impact of their changes.</p><h4>Conclusion:</h4><p>While online A/B testing offers a valuable tool for data-driven decision-making, it is crucial to consider at least those three potential issues. By considering the channel through which users engage, measuring both short-term and long-term impacts, and accounting for primacy and newness effects, organizations can enhance the reliability and validity of their A/B testing results. This is just the tip of the iceberg and I invite you to read further: <a href="https://ai.stanford.edu/~ronnyk/2007GuideControlledExperiments.pdf">Kohavi, R., Henne, R. M., &amp; Sommerfield, D. (2007, August). Practical guide to controlled experiments on the web: listen to your customers not to the hippo. In <em>Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining</em> (pp. 959–967).</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6ff2e4effe22" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/save-your-a-b-testing-by-avoiding-those-3-costly-mistakes-6ff2e4effe22">Save your A/B testing by avoiding those 3 costly mistakes</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Interpret Linear Regression Coefficients | Complete Guide]]></title>
            <link>https://medium.com/data-science/how-to-interpret-linear-regression-coefficients-8f4450e38001?source=rss-a52dcb9793ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f4450e38001</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-dives]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[regression]]></category>
            <dc:creator><![CDATA[Quentin Gallea, PhD]]></dc:creator>
            <pubDate>Wed, 24 May 2023 21:20:16 GMT</pubDate>
            <atom:updated>2023-05-29T19:18:07.776Z</atom:updated>
            <content:encoded><![CDATA[<h4>A complete guide from simple to advanced models</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EFat-3ZlSE8p1BFVPP1i1A.jpeg" /><figcaption>Photo by Vitalii Khodzinskyi from <a href="https://unsplash.com/fr/photos/cx8cofxiKYc">Unsplash</a></figcaption></figure><p>Looking online at how to interpret linear regression coefficients is similar to looking at how to import a CSV file in Python, many people can get this information in their heads. After teaching statistics to over 10,000 students for a decade, I still sometimes have to double-check the interpretation in some special cases (e.g., a binary result and a log-transformed explanatory variable). That is why I decided to write this article which contains a large list of different linear regression models and explains the interpretation of the coefficients in each situation, including log-transformed variables, binary variables or interaction terms.</p><p>Note that to fully understand (but it is not necessary) the content of this article you need to be familiar with two mathematical concepts: <a href="https://en.wikipedia.org/wiki/Partial_derivative">partial derivative</a> and <a href="https://en.wikipedia.org/wiki/Conditional_expectation">conditional expectation</a>.</p><p>Before going through the list of different cases, let me introduce some important definitions and considerations (model definition, ceteris paribus, binary variables, multicollinearity).</p><h4>Table of content:</h4><p><a href="#95e4"><strong>0. Definitions and important considerations</strong></a></p><p><a href="#8035"><strong>1. Intercept</strong></a></p><p><a href="#cfa3"><strong>2. Continuous dependent variable</strong></a></p><p><a href="#bad0">2.1 Continous independent variable</a><br> <br><a href="#8543"> 2.1.a lin-lin: Linear outcome, linear independent variable</a><br> <br> <a href="#55cd">2.1.b log-lin: Log transformed outcome, linear independent variable</a><br> <br> <a href="#7a8c">2.1.c lin-log: Linear outcome, Log transformed independent variable</a><br> <br><a href="#f387"> 2.1.d log-log: Log transformed outcome, Log transformed independent variable</a><br> <br><strong> </strong><a href="http://b7ae/">2.2 Binary independent variable</a><br> <br><a href="#cbae"> 2.2.a Linear dependent variable</a><br> <br><a href="#d79a"> 2.2.b Log transformed dependent variable</a><br> <br><a href="#8634"><strong>3. Binary dependent variable</strong></a></p><p><a href="#2b4b">3.1 Continous independent variable</a><br> <br><a href="#805e"> 3.1.a Linear independent variable</a><br> <br><a href="#c427"> 3.1.b Log transformed independent variable</a><br> <br><a href="#ec8c"> 3.2 Binary independent variable</a><br> <br><a href="#83af"><strong>4. Interaction effects</strong></a><br> <br><a href="#683c"> 4.1 Quadratic effect</a><br> <br><a href="#707f"> 4.2 Interaction between two continous variables</a></p><p><a href="#4d89">4.3 Interaction between two binary variables</a></p><p>Special case Difference-in-Difference</p><h3><strong>0. Definitions and important considerations</strong></h3><h4>General definition</h4><p>Definition of the model:</p><p>Yᵢ = β₀ + β₁ Xᵢ + Z’ λ+ ϵᵢ</p><p>Yᵢ is the dependent variable, Xᵢ is an independent continous variable, Z a vector of control variables, and ϵᵢ an error term.</p><p>Analytically, β₀ is the intercept of the function, and β₁ is a slope parameter. Hence β₁ represents the change in Y after a unit increase of X everything else equals (keeping the values in Z fixed).</p><p>Formally we can write β₁ = ∂ Y / ∂ X (the partial derivative of Y with respect to X).</p><h4>Ceteris paribus</h4><p>In multiple linear regressions, we use the term <em>ceteris paribus</em>, which means “all other things being equal”. This is a direct consequence of the definition above. The partial derivative measures the changes in a function due to a change in one variable, with all other variables remaining fixed. Thus, in the model above, β₁ represents the effect of a change in X on Y while keeping everything else in the vector of control variables Z fixed.</p><h4>Binary variables</h4><p>The dependent or independent variable can be a binary variable, i.e. a variable taking either the value 1 or the value 0.</p><p>When the <strong>independent variable is a binary variable</strong>, the coefficient should be interpreted as a difference in expectation. Suppose that Dᵢ is a binary variable taking the value 1 if the person in the dataset is an adult (Age≥21) and 0 otherwise. The model would be Yᵢ = β₀ + β₁ Dᵢ + Z’ λ+ ϵᵢ. According to the general definition of the interpretation of the coefficient given above (partial derivative), a change of one unit here means a change from childhood to adulthood. Therefore, the coefficient should be interpreted as the ‘average’ difference in Y between children and adults. Formally, β₁ = E[Y |D=1,Z]-E[Y |D=0,Z]. In other words, β₁ represents the difference between the ‘average’ value of Y for adults and the average value of Y for children.</p><p>When the <strong>dependent variable is a binary variable</strong>, we are in the case of a Linear Probability Model (LPM). The coefficients of the regression represent a change in the probability that the dependent variable is equal to 1. LPMs are often neglected, mainly because these models can predict negative probabilities or probabilities greater than 1. To avoid this problem, we should rely on logit (logistic regression) or probit models. However, LPMs are still often used as the coefficients are easy to interpret (see section 3). In particular, with fixed effects, which are widely used in econometrics, LPMs is more ‘suitable’ (cf: <a href="https://www.econstor.eu/bitstream/10419/145837/1/VfS_2016_pid_6909.pdf">Estimating Fixed Effects Logit Models with Large Panel Data</a>).</p><h4>Perfect multicollinearity</h4><p>If two or more independent variables can be expressed as an exact linear relationship to each other, we have a perfect multicollinearity problem. (Technically, the matrix of explanatory variables would not be of full rank and thus would not be invertible, which is essential for estimating the regression parameters.) For binary and categorical variables, this fact leads to an important point. You cannot include binary variables capturing all the different possible categories, otherwise you would have a perfect multicollinearity problem. For example, in the same regression, you cannot include a binary variable for adults and non-adults. In fact, this is not a problem because the regression coefficient associated with the variable (here Adult) represents the difference with the reference category (the one that is excluded, here Non-Adult). So it is not useful to have both.</p><p>Now, if you have more than two categories, for example Light, Medium and Heavy, you cannot have one binary variable for all of them. You have to exclude one of them.</p><p>Example:</p><p>Yᵢ = β₀ + β1₁ Mediumᵢ + β₂ Heavy + Z’ λ + ϵᵢ</p><p>Here we excluded the category Light. Hence, β₀ represents actually the expectation of Y for the observations belonging to the category Light. The excluded category will become the reference category, meaning that the coefficients will always represent the difference in expectations compared to this category:</p><p>β₁ = E[Y|Medium=1,Z]-E[Y |Light=1,Z]</p><p>β₂ = E[Y|Heavy=1,Z]-E[Y |Light=1,Z]</p><h4><strong>Log transformation</strong></h4><p><strong>Why do we log transform?</strong></p><p>Note that the log transformation is often used for linear regression. Linear regression measures the average effect, and so when a variable is highly skewed on the right, a common approach is to apply the natural logarithm to transform it. This strategy aims to reduce the skewness and therefore allows the mean to be used. My personal rule is that if the skewness (a measure of skewness) is greater than 3, I log transform the variable.</p><p><strong>What should I look out for with the log transformation?</strong></p><p>When a variable is transformed into a logarithm, the interpretation of the coefficient changes. This is not necessarily a bad thing and may even be beneficial. When a variable is log-transformed, the linear regression coefficient can be interpreted as an elasticity or semi-elasticity. As we will see later, this means that instead of looking at changes in the unit of the variable, we look at changes in percentages. In the case of a level regression (no log transformation), with the regression coefficients corresponding to a partial derivative (∂ Y / ∂ X ), a change of one unit in X implies a change of β₁ unit in Y (with Y the dependent variable, X the independent variable, and β₁ the regression coefficient associated with X). When both variables (dependent and independent) are log-transformed, we interpret the regression coefficient approximately as an elasticity: a 1% change in X implies a β₁% change in Y.</p><p>Careful Exploratory Data Analysis (EDA) should be made before and after the transformation to make the right decision. You can follow my EDA recipe here: <a href="https://medium.com/towards-data-science/a-recipe-to-empirically-answer-any-question-quickly-22e48c867dd5">https://medium.com/towards-data-science/a-recipe-to-empirically-answer-any-question-quickly-22e48c867dd5</a> . In addition, please refer to <em>“</em><a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4120293/"><em>Log-transformation and its implications for data analysis</em></a><em>” </em>(Feng et al. (2002)) for more details on the potential issues.</p><h3>1. Intercept</h3><p>Definition of the model:</p><p>Yᵢ = β₀ + β₁ Xᵢ + Z’ λ+ ϵᵢ</p><p>Y is the dependent variable, X is an independent variable, Z is a vector of control variables, and ϵᵢ an error term.</p><p><em>Interpretation:</em> β₀ is the expected value of Y if all the other variables are set to 0. Note that if the explanatory variable(s) never equal zero (e.g. height, GDP or whatever), it makes no sense to interpret this coefficient.</p><h3>2. Continuous dependant variable</h3><p>In this Section 2, the dependant variable Yᵢ is always continuous.</p><h4>2.1 Continous independent variable</h4><p>In this sub-Section 2.1, the independent variable Xᵢ is always continuous.</p><h4><strong>2.1.a level-level: Level outcome, level independent variable</strong></h4><p>Definition of the model:</p><p>Yᵢ = β₀ + β₁ Xᵢ + Z’ λ+ ϵᵢ</p><p>Y is the dependent variable, X is an independent variable, and ϵ an error term.</p><p><em>Interpretation:</em> A one-unit increase of X implies a β₁ unit change of Yᵢ on average (ceteris paribus, everything else held constant).</p><h4><strong>2.1.b log-level: Log transformed outcome, level independent variable</strong></h4><p>Definition of the model:</p><p>log(Yᵢ) = β₀ + β₁ Xᵢ + Z’ λ+ ϵᵢ</p><p>log(Yᵢ) is the log-transformed dependent variable, Xᵢ is an independent variable, and ϵᵢ an error term.</p><p><em>Interpretation:</em> A one unit increase of X implies a (exp(β₁)-1)*100 percent change of Y on average (ceteris paribus, everything else held constant). For a quick approximation, you can interpret the coefficient as a semi-elasticity: A one-unit increase of X implies a 100*β₁ % change in Y on average (ceteris paribus, everything else held constant).</p><h4><strong>2.1.c level-log: Level outcome, Log transformed independent variable</strong></h4><p>Definition of the model:</p><p>Yᵢ = β₀ + β₁ log(Xᵢ) + Z’ λ+ ϵᵢ</p><p>Yᵢ is the dependent variable, log(Xᵢ) is a log-transformed independent variable, and ϵᵢ an error term.</p><p><em>Interpretation:</em> A one percent increase of X implies a β₁*log(1.01) change of Y on average (ceteris paribus, everything else held constant). For a quick approximation, you can interpret the coefficient as a semi-elasticity: A one percent increase of X implies a β₁ / 100 unit change in Y on average (ceteris paribus, everything else held constant).</p><h4>2.1.d log-log: Log transformed outcome, Log transformed independent variable</h4><p>Definition of the model:</p><p>log(Yᵢ) = β₀ + β₁ log(Xᵢ) + Z’ λ+ ϵᵢ</p><p>log(Yᵢ) is the log-transformed dependent variable, log(Xᵢ) is a log-transformed independent variable, and ϵᵢ an error term.</p><p><em>Interpretation:</em> A one percent increase of X implies a (1.01^β₁–1) * 100 percent change of Yᵢ on average (ceteris paribus, everything else held constant). For a quick approximation, you can interpret the coefficient as an elasticity: A one percent increase of X implies a β₁ percent % in Y on average (ceteris paribus, everything else held constant).</p><h3>2.2 Binary independent variable</h3><p>In this sub-Section 2.2, the independent variable Dᵢ is a binary variable taking only either the value 1 OR the value 0.</p><h4>2.2.a Level dependent variable</h4><p>Definition of the model:</p><p>Yᵢ = β₀ + β₁ Dᵢ+ Z’ λ+ ϵᵢ</p><p>Yᵢ is the dependent variable, Dᵢ is an independent binary variable, Z is a vector of control variables, and ϵᵢ an error term.</p><p><em>Interpretation:</em> Recall that formally β₁ = E[Yᵢ |Dᵢ=1,Z]-E[Yᵢ |Dᵢ=0,Z]. In other words, the difference in expectation when Dᵢ changes from 0 to 1 is equal to β₁, everything else is equal.</p><p>To make it more concrete let me use the following example:</p><p>HoursOfSleepᵢ = β₀ + β₁ Adultᵢ+ Z’ λ+ ϵᵢ.</p><p>In this example, β₁ represents the “average” difference of hours of sleep between adults (when Adult = 1) and non-adults (aka children, when Adult = 0) everything else equal.</p><h4>2.2.b Log transformed dependent variable</h4><p>Definition of the model:</p><p>log(Yᵢ) = β₀ + β₁ Dᵢ+ Z’ λ+ ϵᵢ</p><p>log(Yᵢ) is the log-transformed dependent variable, Dᵢ is an independent binary variable, Z is a vector of control variables, and ϵᵢ an error term.</p><p><em>Interpretation:</em> Recall that the coefficient of a binary variable represents a difference in “means” (conditional expectations). However, here due to the log transformation we have:</p><p>β₁ = log(E[Yᵢ |Dᵢ=1,Z]) — log(E[Yᵢ |Dᵢ=0,Z]) = log(E[Yᵢ |Dᵢ=1,Z]/E[Yᵢ |Dᵢ=0,Z])</p><p>⇔ exp(β₁) = E[Yᵢ |Dᵢ=1,Z]/E[Yᵢ |Dᵢ=0,Z]</p><p>To make it more concrete let me use the following example: log(HoursOfSleepᵢ) = β₀ + β₁ Adultᵢ+ Z’ λ+ ϵᵢ. In this example, exp(β₁) represents the ratio of the mean hours of sleep for adults (when Adultᵢ = 1) over non-adults (aka children, when Adultᵢ = 0) everything else equal. If exp(β₁) = 1.1 it would mean that the adults have 10% more hours of sleep compared to children. While if exp(β₁) = 1.5 it would mean that the adults have 50% more hours of sleep compared to children.</p><p>Note that in this context the mean is the geometric mean (for more details see: <a href="https://stats.oarc.ucla.edu/other/mult-pkg/faq/general/faqhow-do-i-interpret-a-regression-model-when-some-variables-are-log-transformed/">https://stats.oarc.ucla.edu/other/mult-pkg/faq/general/faqhow-do-i-interpret-a-regression-model-when-some-variables-are-log-transformed/</a>).</p><h3><strong>3. Binary dependent variable</strong></h3><p>In this Section 3, the dependent variable Dᵢ is always binary (taking the value 1 or 0). In this case, the model is called a Linear Probability Model (see the note section 0. for more details).</p><h4>3.1 Continous independent variable</h4><p>In this sub-Section 3.1, the independent variable Xᵢ is a continous variable.</p><h4>3.1.a Level independent variable</h4><p>Definition of the model:</p><p>Dᵢ = β₀ + β₁ Xᵢ + Z’ λ+ ϵᵢ</p><p>Dᵢ is the binary dependent variable, Xᵢ is an independent continuous variable, Z is a vector of control variables, and ϵᵢ an error term.</p><p><em>Interpretation:</em> A one-unit increase of X implies a change in the probability that D = 1 of β₁ on average (ceteris paribus, everything else held constant). For example, if β₁=0.1 it means that the probability of D being equal to 1 increases by 0.1 on average (ceteris paribus, everything else held constant).</p><h4>3.1.b Log transformed independent variable</h4><p>Definition of the model:</p><p>Dᵢ = β₀ + β₁ log(Xᵢ) + Z’ λ+ ϵᵢ</p><p>Dᵢ is the binary dependent variable, log(Xᵢ) is a log transformed independent continuous variable, Z is a vector of control variables, and ϵᵢ an error term.</p><p><em>Interpretation:</em> A one-percent increase of X implies a change in the probability that D = 1 of β₁ unit on average (ceteris paribus, everything else held constant). For example, if β₁=0.1 it means that the probability of D being equal to 1 increases 0.1 on average following a one percent increase of X (ceteris paribus, everything else held constant).</p><h4>3.2 Binary independent variable</h4><p>Definition of the model:</p><p>Dᵢ = β₀ + β₁ Bᵢ + Z’ λ+ ϵᵢ</p><p>Dᵢ is the binary dependent variable, Bᵢ is an independent binary variable, Z is a vector of control variables, and ϵᵢ an error term.</p><p><em>Interpretation:</em> The probability that D = 1 when B changes from 0 to 1 changes by β₁, everything else equal. For example, if β₁ = 0.1 it means that the probability of D equal to 1 increases by 0.1 on average when B changes from 0 to 1 (ceteris paribus, everything else held constant).</p><p>To make it more concrete let me use the following example:</p><p>Insomniaᵢ = β₀ + β₁ Adultᵢ + Z’ λ+ ϵᵢ.</p><p>Insomnia is a binary variable taking the value 1 if individual “i” suffers from insomnia and 0 otherwise. Adult is a binary variable taking the value 1 if individual “i” is strictly older than 20 years old and 0 otherwise. In this example, if β₁ = 0.1 it means that the probability of an Adult (Adult = 1) suffering from insomnia is larger than for a Child (Adult = 0) by 0.1 on average, everything else equal.</p><h3>4. Interaction effects</h3><p>Linear regressions are linear in parameters, which does not prevent the estimation of a non-linear function. We will see three different types of interactions. For the sake of parsimony, I will only use one continuous dependent variable in this section: Y.</p><h4>4.1 Quadratic effect</h4><p>Definition of the model:</p><p>Yᵢ = β₀ + β₁ Xᵢ + β₂ Xᵢ * Xᵢ+ Z’ λ+ ϵᵢ</p><p>or</p><p>Yᵢ = β₀ + β₁ Xᵢ + β₂ Xᵢ²+ Z’ λ+ ϵᵢ</p><p>Yᵢ is a continuous dependent variable, Xᵢ is an independent continuous variable, Z is a vector of control variables, and ϵᵢ an error term. This model includes a polynomial function of order 2 for X. Linear regressions could include higher-order polynomial functions.</p><p><em>Interpretation:</em> The interpretation is more complex with polynomial form because the partial derivative (the marginal effect), is not constant anymore. In the current situation, ∂ Y / ∂ X = β₁ + 2 * β₂ Xᵢ. As the marginal effect depends on the values of X, we must evaluate the marginal effect for different meaningful values of X.</p><p>To do so, I compute and plot the marginal effect of changing X on Y for different values of X. In STATA you could use the commands <em>margins</em> and <em>marginsplot</em>, in R you can use <em>marginaleffects</em>, while in Python I use the following code:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2Fb036003f-51d5-44a1-8b96-f8abb3d00491%2F96836d5709784b4a84c31b34da3e6e95%2Ff357df0583514c5fb01673ca7e3ba0e1%3Fheight%3D1213&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2Fb036003f-51d5-44a1-8b96-f8abb3d00491%2F96836d5709784b4a84c31b34da3e6e95%2Ff357df0583514c5fb01673ca7e3ba0e1%3Fheight%3D1213&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnails%2Fmain.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="1213" frameborder="0" scrolling="no"><a href="https://medium.com/media/bce1bc83e36e3f7ddc20751ac76a0da9/href">https://medium.com/media/bce1bc83e36e3f7ddc20751ac76a0da9/href</a></iframe><p>The graph above reveals the quadratic relationship between X and Y. Hence, the marginal effect of X on Y is initially negative and then it becomes positive. The code below allows us to compute the marginal effect in this particular case. Note that I computed the derivative by hand for this second-degree polynomial function as you can see from the first line of code.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2Fb036003f-51d5-44a1-8b96-f8abb3d00491%2F96836d5709784b4a84c31b34da3e6e95%2Fe57140727f544bfebfb8b40ef2504ea8%3Fheight%3D835&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2Fb036003f-51d5-44a1-8b96-f8abb3d00491%2F96836d5709784b4a84c31b34da3e6e95%2Fe57140727f544bfebfb8b40ef2504ea8%3Fheight%3D835&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnails%2Fmain.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="835" frameborder="0" scrolling="no"><a href="https://medium.com/media/372f67f2271e426b44dbfecf8141cf72/href">https://medium.com/media/372f67f2271e426b44dbfecf8141cf72/href</a></iframe><p>We can see from the graph above that for values approximately below -1 the marginal is negative and then it becomes positive.</p><h3>4.2 Interaction between two continous variables</h3><p>In some situations we expect two different variables to interact together. In one of my recent publications published in PNAS, jointly with my colleague Pr.Rohner, we explored the relationship between a strategic position to control maritime trade, trade openness, and conflicts (<a href="https://www.pnas.org/doi/abs/10.1073/pnas.2105624118?doi=10.1073%2Fpnas.2105624118">https://www.pnas.org/doi/abs/10.1073/pnas.2105624118?doi=10.1073%2Fpnas.2105624118</a>). Trade openness and the strategic value of a position have a direct effect on the probability of conflict. But there is also an important joint effect (an interaction), as shown in the series of graphs below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/352/1*8-8b_fypbyoenSqEqSPyrQ.png" /><figcaption>Image by author. The figure reveals the effect of the proximity to a main strategic position to control maritime trade routes (X-axis) on the probability of conflict (Y-axis). The observations are split by quartile of proximity (q1, q2, q3 and q4), meaning that q4 are the closest regions to strategic positions.</figcaption></figure><p>The graph above revealed that the closer we are to a strategic location, the higher the risk of conflict.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/352/1*20QgQctGi92f6QzIlvKnkg.png" /><figcaption>Image by author. The figure reveals the relationship of trade openness (X-axis) on the probability of conflict (Y-axis). The observations are split by quartile of trade openness (q1, q2, q3 and q4).</figcaption></figure><p>This second graph shows that the greater the trade openness, the higher the risk of conflict. Thus, years of trade booms have a higher risk of conflict.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/403/1*EhEgDj8fxM5xxLQMgIGwrg.png" /><figcaption>Image by author. This bar graph represent the relationship between distance to a strategic position divided in quartiles (near_dist_q), the trade openness during the year (tradew_q) and the probability of conflict (y-axis).</figcaption></figure><p>This last graph shows how our two explanatory variables interact. On the one hand, we can see in the first group of bars on the left that locations far from strategic positions for maritime trade have a higher risk of conflict during periods of trade expansion (yellow bar). On the other hand, we can see that this relationship is reversed for locations close to strategic positions (group of bars on the far right).</p><p>Hence, the marginal effect of the distance to strategic positions on the probability of conflict change as a function of trade openness. To model this effect we must use an interaction effect.</p><p>Definition of the model:</p><p>Yᵢ = β₀ + β₁ Xᵢ + β₂ Zᵢ + β₃ Zᵢ*Xᵢ + ϵᵢ</p><p>Yᵢ is a continuous dependent variable, Xᵢ and Zᵢ are independent continuous variables, and ϵᵢ an error term. It is very important to note that when we have an interaction effect, we must include every “main” term as well in the model. For example, if we want to have an interaction between X and Z, we must also include X and Z alone in the regression.</p><p><em>Interpretation:</em> To understand how to interpret such an effect we have to go back to the definition of the marginal effect, which is a partial derivative. In our case there are two different partial derivatives including β₃:</p><p>∂ Y / ∂ X = β₁ + β₃ Zᵢ</p><p>∂ Y / ∂ Z = β₁ + β₃ Xᵢ</p><p>Hence, in this situation, as in the polynomial situation, we have to evaluate the marginal effect for a set of meaningful values of Z or X as the marginal effect is a function of those variables. To do so, I use in STATA the commands <em>margins</em> and <em>marginsplot</em>, in R <em>marginaleffects</em>, while in Python I use the following code:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2Fb036003f-51d5-44a1-8b96-f8abb3d00491%2F96836d5709784b4a84c31b34da3e6e95%2F437c11184d3948d088e0803c57a8826e%3Fheight%3D1462.6666870117188&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2Fb036003f-51d5-44a1-8b96-f8abb3d00491%2F96836d5709784b4a84c31b34da3e6e95%2F437c11184d3948d088e0803c57a8826e%3Fheight%3D1462.6666870117188&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnails%2Fmain.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="1462" frameborder="0" scrolling="no"><a href="https://medium.com/media/971edcb0766e3822ff40a1ebb6d38174/href">https://medium.com/media/971edcb0766e3822ff40a1ebb6d38174/href</a></iframe><p>The figure above plots the regression plane (not anymore regression line as Y is a function of X and Z).<br>The following code will allow us to plot the marginal effect of X on Y as a function of Z values.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2Fb036003f-51d5-44a1-8b96-f8abb3d00491%2F96836d5709784b4a84c31b34da3e6e95%2Ff2cf5d937f7346bd8465696229b53996%3Fheight%3D924.6666870117188&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2Fb036003f-51d5-44a1-8b96-f8abb3d00491%2F96836d5709784b4a84c31b34da3e6e95%2Ff2cf5d937f7346bd8465696229b53996%3Fheight%3D924.6666870117188&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnails%2Fmain.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="924" frameborder="0" scrolling="no"><a href="https://medium.com/media/17e1356915f25f5c416810cd69759a59/href">https://medium.com/media/17e1356915f25f5c416810cd69759a59/href</a></iframe><p>We can see from the graph above that the marginal effect of x on y is negative when z is approximately lower than -1 while it becomes positive for larger values.</p><h3>4.3 Interaction between two binary variables</h3><p>As in the previous section, we sometimes have binary variables that interact with each other. A famous example of this is wage discrimination. Let us imagine a correlational model to calculate the average wage differences between men and women, whites and non-whites. In this model it is important to include an interaction term if one wants to test the hypothesis that wage discrimination against non-white women is different (potentially larger) than the sum of discrimination against women and non-whites.</p><p>Model:</p><p>Wageᵢ = β₀ + β₁ Womanᵢ + β₂ NonWhiteᵢ + β₃ Womanᵢ * NonWhiteᵢ + ϵᵢ</p><p>Wageᵢ is the hourly wage, Womanᵢ is a binary variable taking the value 1 if individual “i” is female, NonWhiteᵢ is a binary variable taking the value 1 if individual “i” is non-white, and ϵᵢ an error term. <br>It is very important to note that when we have an interaction effect, we must also include each “main” term in the model. For example, if we want to get an interaction between X and Z, we must also include X and Z alone in the regression (here we include the interaction but also Woman and NonWhite separately).</p><p><em>Interpretation:</em> First, let us interpret the main effects:</p><p>β₁ = E[Wageᵢ | Womanᵢ = 1, NonWhiteᵢ = 0] - E[Wageᵢ | Womanᵢ = 0, NonWhiteᵢ = 0]</p><p>β₁ captures the average wage difference between white women and white men. Note that the other terms are 0 as NonWhite is set to 0 (including the interaction).</p><p>β₂ = E[Wageᵢ | NonWhiteᵢ = 1, Womanᵢ = 0] — E[Wageᵢ | NonWhiteᵢ = 0, Womanᵢ = 0]</p><p>β₂ captures the average wage difference between non-white men and non-white men. Note that the other terms are 0 as Woman is set to 0 (including the interaction).</p><p>β₃ =</p><p>(E[Wageᵢ | NonWhiteᵢ = 1, Womanᵢ = 1] — E[Wageᵢ | NonWhiteᵢ = 0, Womanᵢ = 1])</p><p>-</p><p>(E[Wageᵢ | NonWhiteᵢ = 1, Womanᵢ = 0] — E[Wageᵢ | NonWhiteᵢ = 0, Womanᵢ = 0])</p><p>Finally, β₃ is the additional wage penality (assuming that the coefficient is negative) for being non-white AND a woman.</p><p><strong>Special case: Difference-in-Difference</strong></p><p>There is another common situation in which we use binary variable interactions. A quasi-experimental technique widely used in econometrics called difference-in-difference. This strategy aims to measure the causal effect of a policy, for example. However, a discussion of how to assess causality in this setting is beyond the scope of this paper (see Scott Cuningham’s free e-book : <a href="https://mixtape.scunning.com/09-difference_in_differences#going-beyond-cunningham2013">“Causal Inference: The Mixtape”</a>).</p><p>Let me discuss a basic hypothetical example of a Diff-in-Diff model. In 2008, the UK implement a policy to reduce CO² emissions, while Ireland is not subject to this policy. To evaluate the effect of this policy on pollution we could set the following model:</p><p>CO²EmissionsPerCapitaᵢₜ = β₀ + β₁ UKᵢₜ + β₂ Postᵢₜ + β₃ UKᵢₜ * Postᵢₜ + ϵᵢₜ</p><p>i and t are indices for country and year respectively, CO²EmissionsPerCapitaᵢₜ is self-explanatory, UKᵢ is a binary variable taking the value one if observation i is for the UK, Postᵢ is a binary variable taking the value one if observation i is measured after the implementation of the policy (after 2008), and ϵᵢ an error term.</p><p><em>Interpretation:</em></p><p>First, let us interpret the main effects. β₁ is the average CO2 emissions per capita difference between the UK and Ireland over the whole period. β₂ is the average CO2 emissions per capita difference between the Post (after the 2008) and Pre period (both countries together). Now the important coefficient in this setup is β₃.</p><p>β₃ =</p><p>(E[CO²EmissionsPerCapitaᵢₜ| Postᵢ = 1, UKᵢ = 1] —</p><p>E[CO²EmissionsPerCapitaᵢₜ | Postᵢ = 0, UKᵢ = 1])</p><p>-</p><p>(E[CO²EmissionsPerCapitaᵢₜ| Postᵢ = 1, UKᵢ = 0] —</p><p>E[CO²EmissionsPerCapitaᵢₜ | Postᵢ = 0, UKᵢ = 0])</p><p>β₃ is indeed a double difference. It represents the change in UK CO² emissions after the policy is implemented compared to the change in CO² emissions in Ireland (a country that did not implement the policy). Therefore, assuming Ireland is a good counterfactual, the additional difference captured by β₃ represents the effect of the policy.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f4450e38001" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-interpret-linear-regression-coefficients-8f4450e38001">How to Interpret Linear Regression Coefficients | Complete Guide</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Science and Art of Causality (part 2)]]></title>
            <link>https://medium.com/data-science/the-science-and-art-of-causality-part-2-c085a9d56fd2?source=rss-a52dcb9793ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/c085a9d56fd2</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[causality]]></category>
            <category><![CDATA[science]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[causal-inference]]></category>
            <dc:creator><![CDATA[Quentin Gallea, PhD]]></dc:creator>
            <pubDate>Fri, 06 Jan 2023 14:56:27 GMT</pubDate>
            <atom:updated>2024-09-30T07:30:10.224Z</atom:updated>
            <content:encoded><![CDATA[<h4>Let’s put ourselves in the shoes of a detective and explore causal inference</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ODU-dGVjAt9LLhMyrz7AlA.png" /></figure><p>As we saw in the first part of this two-part article, measuring a causal effect is critical to drawing the right conclusions, because every choice you make or decision you make is usually the result of expected causal relationships.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F2W2Y9jv0mDY%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D2W2Y9jv0mDY&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F2W2Y9jv0mDY%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a5d604acaa27c99a6c83e521e04ba5f2/href">https://medium.com/media/a5d604acaa27c99a6c83e521e04ba5f2/href</a></iframe><h4>For example:</h4><p><strong>Individual choices:</strong></p><ul><li>If I go vegan, I’ll reduce my ecological footprint.</li><li>If I drink this tequila shot, I’ll dance better.</li></ul><p><strong>Companies:</strong></p><ul><li>Home-office reduces productivity</li><li>Spamming users with YouTube Premium Ads will increase the number of subscribers.</li></ul><p><strong>Policymakers:</strong></p><ul><li>Replacing nuclear power plants with renewables will help to reach the Paris Agreement.</li><li>Lockdowns will reduce the spread of the covid-19</li></ul><p>The point is that there is no statistical test to prove that your effect is causal. To challenge causality, as explained in the first part of this article, you can ask two main questions: is there anything else that could explain the relationship between the cause and the effect, and could it be the other way around (i.e., the effect causes the cause)?</p><blockquote>[H]ow can we find evidence of a causal effect despite the absence of statistical tests to directly test causality?</blockquote><p>Those questions allow us to challenge causal claims. But, how can we find evidence of a causal effect despite the absence of statistical tests to directly test causality? In this article, I will show you how researchers proceed with a fascinating scientific paper: <em>‘LONDON FOG: A CENTURY OF POLLUTION AND MORTALITY, 1866–1965’</em> (Hanlon (2018)).</p><p>To do this, we will put ourselves in the shoes of police detectives. Police detectives constantly try to answer causal questions: who caused the death of this person? Was it Colonel Mustard with the candlestick in the conservatory? Are you sure that it was not with a wrench or that the crime was committed by someone else? In our case, we have a suspect, or more precisely a hypothesis that we would like to test (for example, pollution increases mortality). And then we ask ourselves if it was really pollution or the evolution of health services? Or was it actually a consequence of the weather? etc.</p><p>Usually, if you want to find out who committed the crime, you very rarely have video footage of someone committing the crime. And even if you do, maybe the image is blurry, maybe it’s a fake. Therefore, you might never be one hundred percent sure of the identity of the perpetrator. To overcome this limitation, you accumulate evidence, you try to discard all the possible alibis for the criminal until you have enough evidence, and you manage to discard the main other stories (is it somebody else? Was this person doing something else?). It’s very similar in research when we want to find causal evidence.</p><h3>Case Study: London fog and mortality</h3><p>Let me illustrate those concepts with the following paper <em>‘LONDON FOG: A CENTURY OF POLLUTION AND MORTALITY, 1866–1965’</em> (Hanlon (2018)). London was a densely habited and heavily polluted area already during the 19th century. The author of this research paper, answers a very important question: what is the effect of exposure on mortality?</p><p>The interesting part of this article comes from the fact that air pollution data has only been available since the 1950s. However, accurate meteorological data have been available since the 1850s. So the idea of the article is to use fog as an indicator of pollution, because when the weather is foggy, pollution remains at a lower altitude and citizens’ exposure to pollution increases (see figure below).</p><p>You can find a complete Python notebook with my code to replicate the paper and produce the graph I will use in this article here: <a href="https://deepnote.com/workspace/statswithquentin-9de199f7-1b70-481e-a6e2-df6c97c779f4/project/06londonfog-c5f4d2b2-5471-42ec-b8df-879dc56780f2/%2F02_rdd_london.ipynb">Deepnote notebook</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NrSxMPms6TtNGahK9Jq1rg.png" /><figcaption>Causal Diagram. Image by author.</figcaption></figure><h4>A first look at the effect</h4><p>The paper investigates the effect of heavy fog on mortality rates. Therefore, let us first look at how the mortality rate (all causes together) changed from five weeks before a week of heavy fog to five weeks after. It appears that the mortality rate increases at impact (week 0) as well as in subsequent weeks. However, many factors could explain this effect (e.g., seasonality).</p><p>The dataset contains weekly data on weather and death in London from 1850 to 1940 and exclude World War 1 years.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*i2Zo-gYmiVwBeOx0wHGExw.png" /><figcaption>Image by author</figcaption></figure><h4>Can we discard the story of seasonality and time trend?</h4><p>First, let us look at how the number of fog events is distributed over the year. There is a very strong seasonality (the probability of encountering heavy fog is higher in winter). It is therefore important to capture the effect of seasonality in our model because cold weather is associated with more fog and also potentially with more deaths (people get sick when it is cold).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*HwvoPp9-HeYwWb4rj2xlzA.png" /><figcaption>Seasonality of fog. Image by author.</figcaption></figure><p>Second, let us look at the frequency of week with heavy fog from 1850 to 1940. Again, we can see a strong correlation. We observe fewer weeks with fog on average after 1900 compared to before 1900. The model will have to take this evolution into account to avoid confounding this effect with our effect of interest. This is because the quality of the medical system changes over time, reducing the mortality rate, while the number of weeks with fog also decreases over time. Therefore, if we do not capture the time trend, it could inflate the coefficient (overestimate the effect of fog on mortality).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*4QsVNUp-2RPTw0CtkZ0vaw.png" /><figcaption>Yearly number of fog events. The horizontal red dashed lines show the average before/after 1900. Image by author</figcaption></figure><p>Note that even if you are not familiar with the model presented below, you should be able to follow and understand the idea. The estimated model is a simple linear regression:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/470/1*Yf9rrAUEiQIEHgQi2qC_Yg.gif" /></figure><p>with <em>t</em> for the week. <em>Fog^s</em> is a dummy variable taking the value one when there was heavy fog in week <em>s+t</em>. <strong><em>X</em></strong> is a vector of meteorological controls including rainfall, temperature, pressure, and humidity.<em> Year </em>and <em>Week </em>are sets of fixed effects respectively capturing the effect of the year and the effect of the calendar week (seasonality). <em>e</em> is an error term.</p><p>Therefore, this model allows to measure the effect on mortality before, during and after weeks withheavy fog, while taking into account meteorological conditions, seasonality and yearly fixed effects (evolution in time).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/667/1*pjPDN2vObJxOkBFd8PnzIg.png" /><figcaption>Forest plot representing the coefficient of the linear regressions. The vertical axis represents the death rate while the horizontal axis represent the distance in week to a week with heavy fog. The bars represent the 95% confidence intervals. Image by author</figcaption></figure><p>The figure above compares a model without taking seasonality into account (pink squares) and a model taking seasonality into account (orange circles). We can see that the effect of seasonality does indeed inflate the coefficients (the increase in mortality rate is greater in the pink model than in the orange model). In addition, the mortality rate in the model with seasonality returns to the pre-fog week level after two weeks.</p><p>Let us now include the year fixed effects. This set of control variables captures the evolution over time of pollution but also for example of the quality of the health sector. Therefore, the interpretation of the coefficients is slightly different. Now we explore the deviations of the mortality rate from the average mortality rate in year t.</p><p>The figure below reveals that the mortality rate increases during the week of heavy fog and the following week. In addition, we can see that the weather controls do not affect the estimates much.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/668/1*DqRwzCIJkaZ8OZGKbb8sNw.png" /><figcaption>Forest plot representing the coefficient of the linear regressions. The vertical axis represents the death rate while the horizontal axis represent the distance in week to a week with heavy fog. Here both models include week and year fixed effects. The bars represent the 95% confidence intervals. Image by author</figcaption></figure><p>Let us now question the causal nature of this effect. The rationale is that fog keeps pollution at a lower altitude and therefore increases mortality. Use the tool I presented in the first part of this article: <em>“What if it was something else that explained this effect?”</em>.</p><h4>Could it be a story of accident and crime?</h4><p>If there is more fog, it is difficult to see, so there are more accidents or crimes. To rule out this alternative story, the author compares the death rate based on the recorded cause of death (e.g., accident/crime vs. pneumonia).</p><p>The figure below reveals that there is no effect on the death rate caused by accident/crime around a week of heavy fog, while we observe a statistically significant effect on mortality caused by pneumonia.</p><p>We are getting closer to catch our suspect: pollution. There is one last alternative story I wanted to explore with you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1005/1*U45Tv1ThOL8UzqI6lh2DBQ.png" /><figcaption>Forest plot representing the coefficient of the linear regressions. The vertical axis represents the death rate (from different cause: accident/crime vs pneumonia) while the horizontal axis represent the distance in week to a week with heavy fog. The bars represent the 95% confidence intervals. Image by author</figcaption></figure><h4>Could it be a story of weather and epidemiology?</h4><p>When the weather is bad (foggy), people stay home. Hence, if people stay at home, there is a higher risk that they will contaminate other people and therefore the increase in deaths (e.g. from pneumonia) is simply a consequence of that and not of pollution. It seems very difficult to falsify this story, right?</p><p>The author has done something very elegant to dismiss this alternative story. The author compares two different weather shocks: heavy fog and heavy rain. Indeed, a heavy rain will have a similar effect: people might stay home longer. However, the key point here is that fog keeps pollution at a lower altitude while rain cleans the air. Therefore, if this is a pollution story, we would find an opposite effect of rain (fewer deaths) compared to fog.</p><p>The figure below reveals exactly this effect: fog kills, rain saves lives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yKXd4-xwhU58FhwhrlUoJg.png" /><figcaption>Forest plot representing the coefficient of the linear regressions. The vertical axis represents the overall death rate while the horizontal axis represent the distance in week to a week with heavy fog. The bars represent the 95% confidence intervals. Image by author</figcaption></figure><h4>Conclusion</h4><p>The paper presents a strong argument for the causal relationship between fog and pollution on health in London for a century, using evidence to discount various alternative explanations. To evaluate causality, it can be helpful to ask two questions: ‘Is there something else that could be causing the effect?’ and ‘Could it be the reverse?’. In addition, next time you have to challenge a causal claim, put yourself in the shoes of a detective, considering and gathering evidence for and against different explanations. Together, using those technics let’s make more informed decisions and fight misinformation.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c085a9d56fd2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/the-science-and-art-of-causality-part-2-c085a9d56fd2">The Science and Art of Causality (part 2)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Science and Art of Causality (part 1)]]></title>
            <link>https://medium.com/data-science/the-science-and-art-of-causality-part-1-5d6fb55b7a7c?source=rss-a52dcb9793ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/5d6fb55b7a7c</guid>
            <category><![CDATA[causality]]></category>
            <category><![CDATA[science]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[causal-inference]]></category>
            <dc:creator><![CDATA[Quentin Gallea, PhD]]></dc:creator>
            <pubDate>Thu, 05 Jan 2023 22:20:13 GMT</pubDate>
            <atom:updated>2024-04-09T12:36:50.200Z</atom:updated>
            <content:encoded><![CDATA[<h4>If we cannot directly test for causality, what should we do?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XwS8SuZlxFyTgnetpaaI8A.png" /><figcaption>Image by author</figcaption></figure><p>Let me take you with me on a journey in my field of expertise, which is also my passion, my obsession, what I like to call: <strong>The science and art of causality</strong>.</p><p>“Causality” refers to the relationship between cause and effect. It’s the idea that one event or action can lead to another event or outcome. In other words, causality is concerned with understanding how things happen and why they happen.</p><p>In this article we are going to see first going to answer two questions: why understanding causality is so imporant, and why it is so hard to assess causality (fundamental problem of causal inference). Then, we are going to see two main ways to measure a causal effect (randomized control trials and natural experiments). Finally, I will illustrate how to question causality and give you practical tool to do it.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FLJAYf51anKc&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DLJAYf51anKc&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=google" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/b36700440bc0cb81077cd3b9633e262c/href">https://medium.com/media/b36700440bc0cb81077cd3b9633e262c/href</a></iframe><h3>Why causality?</h3><p>First things first: Why should we care about causality? Why am I so obsessed with this topic? Well, because every decision we make, whether as individuals or organizations, is based on the assumption that certain actions will lead to certain outcomes.</p><p>For example, if we decide to go vegan, it’s because we believe it will have health or environmental benefits. Or if a company changes its advertising strategy, it’s because it hopes to improve sales or subscriptions. Similarly, governments must consider how their actions will impact the world around us, such as whether transitioning to renewable energy sources will help meet climate goals. However, understanding causality can be complex and requires careful analysis to determine the true relationships at play. That’s where the science and art of causality comes in — it helps us examine and interpret data to better understand the world around us and make informed decisions.</p><blockquote>The thing is, if we fail to properly assess the relationship, the causal chain, the consequences might be very costly.</blockquote><p>The thing is, if we fail to properly assess the relationship, the causal chain, the consequences might be very costly. First, or governments might put a lot of effort and resources into getting data, and then even more resources to analyze those. But then, if the conclusion is wrong, they might lose even more resources by following a wrong pathway. Hence, that’s why it’s so important to assess causality.</p><blockquote>Additionnaly, understanding causality also helps when we read the news, when we listen to politicians, when we discuss with others, to reduce the risk of being manipulated or suffering from misinformation.</blockquote><p>Moreover, the good news is that there are tools that you can learn, and practical methods that you can apply daily to fight misinformation and to make better, or more educated, decisions.</p><h3>Why is it so hard to assess causality?</h3><p>So, actually, why is it so hard to assess causality? The thing is, you don’t have a statistical test that will tell you if your effect is causal or not. You can do many things in that direction. Numerous statistical tests can be done to challenge what we call the identifying assumptions (the assumptions to identify the causal effects), but usually, we cannot test those assumptions directly.</p><p>This is what makes understanding causality so exciting! It requires both a strong foundation in statistics and a deep domain knowledge. You have to use your critical thinking skills to consider the relationships between different variables and events. Math alone will not save you. The main challenge is often in how we interpret the statistical measures we use — we may mistake a correlation for a causal effect.</p><h3>The fundamental problem of causal inference</h3><p>The key aspect comes from the fundamental problem of causal inference. Let me illustrate this concept with the two graphs below. On the left, you have the global direct primary energy consumption in the world, split between renewable energies and other sources of energy production. The second graph represents the CO2 emissions of the world, also from 1900 to 2020. Both are growing. However, we tend to think that using renewable energy help to reduce CO2 emissions. Using such aggregate data, some people might be tempted to argue that renewable energies are not necessarily useful.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AV9os1fS7XM0azpPsu2LHA.png" /><figcaption>Image by author.</figcaption></figure><p>Obviously, it’s impossible to answer the question (What is the effect of renewable energy on CO2 emissions?) with such simple statistics. The thing is, we don’t know what will happen without renewable energies. We don’t have a world without renewable energies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dwl56cOgjwve3YaVyS0Zng.png" /><figcaption>Image by author.</figcaption></figure><p>The relationship between renewable energy and CO2 emissions is complex. On one hand, renewable energy sources like solar and wind power can be used as substitutes for fossil fuels and thus lead to lower CO2 emissions. On the other hand, the process of extracting resources to produce photovoltaic cells is energy-intensive, and the “rebound effect” (where people consume more energy when it is provided by renewable sources) can also lead to higher CO2 emissions. Without more data, it is impossible to definitively answer the question of whether renewable energy leads to higher or lower CO2 emissions. Bottom line, it is impossible to answer the question with such data.</p><blockquote>To perfectly answer this question, we would need two parallel worlds.</blockquote><p>In one of the two worlds you have renewable energy and in the other, you don’t. And as it’s the only thing that changed between the two worlds, if there is a difference with respect to CO2 emissions it would most certainly be caused by renewable energy use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Eyx4OPH2phyxk6-6wmSsOQ.png" /><figcaption>What is the effect of renewable energy on CO2 emissions? To perfectly answer this question we would need two parallel worlds. Image by author.</figcaption></figure><p>Unfortunately, we don’t have access to parallel worlds where we can observe the same situation with and without a particular treatment or action. This creates the <strong>“fundamental problem in causal inference”</strong>, as we cannot observe the counterfactual — the alternative reality in which a treatment or action was not taken. For example, we can’t observe a person who is sick both taking and not taking a medication at the same time. What we try to do in causal inference is to get as close as possible to this ideal situation, where we can compare the outcomes of the same situation with and without a particular treatment or action. This allows us to better understand the causal relationships at play.</p><h3>The golden standard</h3><p>Usually, the first way mentioned to address this issue, often called the golden standard, and arguably the best solution is randomized control trials (A/B testing).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x5P0hjSy1L274vF5CGZhtg.png" /><figcaption>Illustration of a Randomized Control Trial. Image by author.</figcaption></figure><p>In short the concept is the following. We take a sample that is, hopefully, representative of a larger population, and randomly allocate subjects between two groups (treatment and control) or more. The subjects typically do not know whether they receive the treatment or not (a process known as blinding). Therefore, the two groups are arguably comparable. Since the only difference is the treatment, if we observe an effect, it is potentially causal, provided that no other biases exist.</p><p>However, there are two main weaknesses with Randomized Control Trials (RCT). The first is that we cannot always use RCT. Sometimes it’s impossible, for example to change the sex of a subject just for the sake of an experiment. In other situations, it is unethical. For example, I have a paper assessing the effect of weapons exports on the probability of conflict in Africa. We are not going to randomly send weapons in different countries to see if it affects the probability of conflict.</p><p>The second main weakness is that usually when we control perfectly the environment for an experiment it might come at the cost of external validity (to what extend we can extrapolate the results beyond the scope of the study). For example, medical research is often done with inbred strains of rats/mice. Those animals are almost identical genetically and hence we are close to the parallel worlds situation. But again, the drawback is that then we lose external validity.</p><blockquote>Hence, often there is a trade-off between measuring perfectly a causal effect and the extend to which the results reflects well real life situations.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0KZK7QoLjMVd4-_1qBTgFQ.png" /><figcaption>Image by author</figcaption></figure><p>Let me illustrate this idea with a fantastic paper: <em>Parachute use to prevent death and major trauma when jumping from aircraft: randomized controlled trial</em> (<a href="https://www.bmj.com/content/363/bmj.k5094">https://www.bmj.com/content/363/bmj.k5094</a>). This paper was published in a top medical journal: the British Medical Journal (BMJ).</p><p>In this experiment (it’s a Randomized Control Trial), they managed to enroll 23 volunteers over a period of one year, between 2017 and 2018, and they made them jump from a plane. The participants were randomly assigned to two groups, either with a parachute or with an empty backpack. They measured the probability of dying or of a major physical trauma directly after impact. The authors don’t find a difference between the two groups with respect to to those outcomes (death and major traumatic injury).</p><p>So, where’s the catch? It’s a it’s a real experiment, but the thing is, to be able to do these experiments, they of course did it on a small stationary plane which never left the ground. People were jumping more or less from one meter off the ground. The idea of the paper was to highlight the fact that sometimes you try to control perfectly the environment with an experiment but then it does not really reflect reality.</p><h3>Natural experiments</h3><p>So then, if it’s not necessarily possible to do a randomized control trial, what can we do? Well, we can resort to what we call quasi-experimental design or natural experiments.</p><p><em>“Natural experiments are </em><strong><em>observational studies</em></strong><em> in which an event or </em><strong><em>a situation that allows for the random</em></strong><em> or seemingly random </em><strong><em>assignment of study subjects</em></strong><em> to different groups is exploited to answer a particular question.”</em> Britannica</p><p>Let me take an example to illustrate such experiment. Let’s say you want to assess the effect of pollution on health (e.g. respiratory diseases risk). You can use animals in a lab, but you’re not going to use humans and expose them to lethal or very dangerous level of pollution. Even if animal research migh be useful, we might struggle to extrapolate to humans. We might be interested about what happens in real life and when there is a long exposure to pollution and humans living their daily life. However, if we compare people from cities to rural areas, they are not comparables, other things make them very different, for example: what they eat, how they move, their type of work, etc.</p><blockquote>Another option is to compare people living in cities to those living in rural areas, but these groups may be different in other ways (such as their diet, physical activity, and type of work) that could confound the results. Therefore, it can be difficult to accurately assess the causal relationship between pollution and health.</blockquote><p>Hence, to answer such question, we can resort to a natural experiments. That is exactly what some researchers did in 2016 (<a href="https://www.sciencedirect.com/science/article/abs/pii/S0095069616300237">https://www.sciencedirect.com/science/article/abs/pii/S0095069616300237</a>). The authors used the Olympic Games in Beijing to measure the causal effect of air pollution on mortality. The government imposed very strict laws to reduce pollution over the city before and during the Games (e.g. closure of power plants, reduction of the use of cars). This situation allows to observe exactly the same people before, during and after. These individuals experienced a sudden change in their level of pollution exposure, going from a high level of exposure to a significantly lower one. The author found that “<em>a 10 percent decrease in PM10 [small particulate matter in the air] concentrations reduces the monthly standardized all-cause mortality rate by 8 percent.</em>”.</p><h3>How to challenge causality ?</h3><p>The two main things that question causality (endogeneity issues) are that there are often other things affecting the relationship you are looking at (omitted variable bias), or the effect might go in both directions (simultaneity, or reverse causality).</p><blockquote>As we cannot test directly for causality, what can we do?</blockquote><p>As we cannot test directly for causality, what can we do? You can always ask if there is something else explaining this result, or if the reverse is true at the same time. Use those questions to challenge a causal statement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*31qLgTtrHS-02Yt-mBtEfg.png" /><figcaption>Image by author</figcaption></figure><p>First let me illustrate the first concept: Omitted Variable Bias (Is there something else?). Research has found that there is a high negative correlation between coffee consumption and cardiovascular diseases (<a href="https://academic.oup.com/eurjpc/article-abstract/29/17/2240/6704995?redirectedFrom=fulltext">https://academic.oup.com/eurjpc/article-abstract/29/17/2240/6704995?redirectedFrom=fulltext</a>). There may be some reasons why the link between coffee consumption and health mentioned in the paper (Chieng et al., 2022) is not necessarily causal. One thing to consider is that other factors may be influencing both coffee consumption and overall health. For example, people more physically active might consume more coffee consumption and be more healthy due to their physical activity. It’s important to always consider the possibility that other factors may be at play and not interpret an observed relationship as a causal effect without further analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jJuYvrSvVIHePZJfgV_bhw.png" /><figcaption>Image by author</figcaption></figure><p>The second question you should ask is: Could it be the reverse? So when you observe that people who drink more are more depressed, is it because they drink, or are they drinking because they are depressed? Research found the two effects. Hence, with this correlation it is very hard to assess the effect of each on the other, because they are of mixed together.</p><h3>Conclusion</h3><p>As we cannot test directly for causality, it is important to use your critical thinking to challenge any causal statement you hear or read. Start by thinking about the parallel world’s situation and try to spot what is the difference between this ideal situation and the situation you face to assess a causal effect. This usually helps to spot where things could go wrong. And then, use your two questions: Could it be something else? Is it the reverse?</p><p>To go further, read <a href="https://medium.com/towards-data-science/the-science-and-art-of-causality-part-2-c085a9d56fd2">part 2 </a>of this article.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5d6fb55b7a7c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/the-science-and-art-of-causality-part-1-5d6fb55b7a7c">The Science and Art of Causality (part 1)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Unhappy with statistical significance (p-value)? Here is a simple solution]]></title>
            <link>https://medium.com/data-science/unhappy-with-statistical-significance-p-value-here-is-a-simple-solution-d196c936f8f2?source=rss-a52dcb9793ad------2</link>
            <guid isPermaLink="false">https://medium.com/p/d196c936f8f2</guid>
            <category><![CDATA[p-value]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[statistical-significance]]></category>
            <dc:creator><![CDATA[Quentin Gallea, PhD]]></dc:creator>
            <pubDate>Thu, 15 Sep 2022 15:08:27 GMT</pubDate>
            <atom:updated>2022-09-15T15:08:27.813Z</atom:updated>
            <content:encoded><![CDATA[<h4>Why and how to use forest plots efficiently?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VEzLFhMlPPmEPVIRTQPUBA.png" /><figcaption>From regression to forest plot, image by the author</figcaption></figure><p>Humans like to put things into bins in a simple way: Statistically significant or not statistically significant. This dichotomous view of statistical results must stop! For almost a decade, I heard the same debate. On one hand, people argued that statistical significance has the advantage of simplicity and clarity (and many people are already struggling to achieve this). On the other hand, there were those who argued for more advanced models (e.g. the Bayesian approach) because of the limitations of the p-value (see next section). I was somewhat in between. I realised that it was useful because of its simplicity and, pragmatically, and I accepted that I could not ask more of many students (including some scientific researchers). On the other hand, I was also bothered by the problematic nature of this vision.</p><p>However, today I am using a solution that seems to satisfy both sides: it is simple and includes the p-value while addressing the main weaknesses of using statistical significance alone. I hope that this proposal can help to take a step away from the excessive focus on statistical significance.</p><p>This article has two main parts. First, I present the concept: the main problems related to statistical significance (p-value), the solution (forest diagram) and a conclusion. Second, I include a complete example with data and a real research question (the relationship between temperature and environmental policy implementation).</p><h4>What are the issues with statistical significance (p-value)?</h4><p>First things first. What is the p-value and why it is a problem to use statistical significance (only)?</p><p><em>“The p-value is the probability of obtaining test results at least as extreme as the results actually observed, under the assumption that the null hypothesis is correct.”</em>— Wikipedia</p><p>It would take a whole article to explain the p-value. So, if you are not yet familiar with this key concept, I let you check quickly <a href="https://towardsdatascience.com/p-values-explained-by-data-scientist-f40a746cfc8">this nice summary on Towards Data Science</a> (<a href="https://towardsdatascience.com/p-values-explained-by-data-scientist-f40a746cfc8">link</a>).</p><p>Here are the two main issues:</p><p><strong>1. Practical relevance:</strong></p><p>The statistical significance says nothing about the size of the effect. Let’s say I see an advertisement for a shampoo that helps reduce hair loss. The advert claims that the results have been proven in the laboratory by a large experiment and that the results are statistically significant (those who use the shampoo lose less hair). However, when you read the underlying research paper, you discover that on average, the person who uses this shampoo will have five more hairs on their head. Who cares? Right?</p><p>Here’s another example from my work as a researcher. I worked for two years on a paper to assess the effect of covid containment on the spread of the virus (Bonardi et al. (2022) forthcoming). Here, the magnitude of the effect is key. Lockdowns have been very costly for the economy, for the psychological health of many people, etc. Therefore, we want to know not only whether they have reduced the spread of the virus, but by how much to assess the cost/benefit of such measures.</p><p>That is why I always advocate using statistical significance simply as a necessary condition for interpreting the effect size. And then, on the basis of the effect size, one can conclude whether the effect has practical significance or not. The emphasis is on magnitude, while statistical significance provides evidence that the results are not random.</p><p><strong>2. Manipulation:</strong></p><p>The focus on a fixed threshold (e.g. 5%) leads to two main problems. When working with data, there are sometimes a few degrees of freedom in the choice of statistical test, sample, etc. Therefore, this may allow some people to manipulate the results slightly to fall below the threshold.</p><p>On the other hand, if you are slightly above the threshold, you may not be able to publish your results. However, basically, a p-value of 4.9% and a p-value of 5.1% show similar evidence on the hypothesis being tested. This leads to serious problems in research. Here is an illustrative example I found on LinkedIn (Florian Weigert): <em>”60 researchers examined the effect of medical treatment X on the reduction of disease Y. 57 researchers found no effect and could not publish the results. 3 researchers documented a statistically significant relationship and were published. Conclusion: A meta-study provides overwhelming support for the fact that medical treatment X leads to a reduction in disease Y. “</em>. If you want to go further, read <em>”The significance filter, the winner’s curse and the need to shrink “*</em>by Zwet and Cator (2021)¹ or the American Statistical Association’s statement on p-values².</p><h4>A simple solution: Forest plots</h4><p>To get a full understanding of a test result, you need to do three things: assess statistical significance (with a grain of salt), focus on the magnitude (if it is statistically significant), and account for the variability.</p><p>Why are forest plots an elegant solution? Forest plots allow us to see all this information and make it easier to compare the coefficients. The picture in top of this article show an example of a forest plot next to a regression table. More example can be found with in the example in the second part of this article.</p><p>1. <strong>Statistical significance:</strong> Statistical significance remains an essential aspect of a statistical test. If the result is not statistically significant, it is not worth interpreting. Forest plots allow you to see statistical significance. If the bar of the 95% confidence intervals does not include 0, it means that we can reject the two-sided null hypothesis that the true coefficient is 0. However, the representation is not oversimplified. In many journals, coefficients from a regression or statistical test are accompanied by small stars to indicate their statistical significance (for example, * if the p-value is less than 5%). The problem with this simplification is that we do not know whether the results are far from statistically significant or very close. Forest plots allow us to visually see the part of the CI that goes beyond the 0 lines, and thus to be a little less obsessed with a clear cut-off.</p><p>2. <strong>Magnitude</strong>: The magnitude is easy to read, as well as the relative magnitude between coefficients. However, sometimes the coefficient cannot be compared simply. For example, in a regression, the variables may have very different scales. Normalising the variables by their standard deviation will give a comparable scale (see example below).</p><p>3. <strong>Variability</strong>: Uncertainty is highlighted by the size of the confidence intervals. Again, it allows us to quickly see the lower and higher boundary of the coefficient and get an idea of variability.</p><h4>Conclusion</h4><p>Forest plots make all the key elements of a statistical test (statistical significance, magnitude and variability) easily accessible without the need to be an expert in statistics. In addition, this richer representation naturally gives less weight to statistical significance by providing other pieces of information. If you are able to run a statistical test correctly, you should have the skills to produce a forest plot and interpret the results (this is not the case with the Bayesian approach for example).</p><p>However, it is important to note the weakness of these charts compared to a table: the loss of precision. Tables remain a necessity (kept in the appendix for example), in order to obtain the exact value of a coefficient. Furthermore, in the case of a regression, if you standardise the coefficient by the standard deviation, the interpretation may be less intuitive even if it favours the comparison of the relative magnitude.</p><h3>Application in Python: Temperature and Environmental Policies:</h3><p>Let me build on my <a href="https://towardsdatascience.com/a-recipe-to-empirically-answer-any-question-quickly-22e48c867dd5">previous article</a>, where I explored the relationship between temperature (heat-waves) and environmental policies.</p><p>Here is the list of the variables:</p><ul><li>cname: Country name</li><li>year: Year</li><li>oecd_eps: Environmental Policy Stringency Index⁴ (from Botta and Kozluk (2014))</li><li>cckp_temp: Annual average temperature in Celsius⁵ (from the Climate Change Knowledge Portal)</li><li>cckp_rain: Annual average rainfall in mm⁵ (from the Climate Change Knowledge Portal)</li><li>wdi_fossil: Fossil fuel energy consumption (% of total)⁶ (from the World Development Indicators, World Bank)</li><li>gdp_pc_ppp: GDP per capita in power purchase parity⁶ (from the World Development Indicators, World Bank)</li><li>pop: Population⁶ (from the World Development Indicators, World Bank)</li></ul><h4>Model</h4><p>Here is the baseline model I will estimate:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*_tkeB3OQ00TQ3VWtvZH2Rw.png" /><figcaption>Regression equation</figcaption></figure><p>with <em>i </em>and <em>t</em> respectively for country and year. <em>EPS</em> is the environmental policy stringency index (an index taking value from 0 to 5). <em>Temperature </em>is the annual average temperature in Celsius. I have subtracted the average value for each country for each variable. As observed in the preliminary analysis, the idea is to compare the warmest years in France with the coldest years in France and not comparing France with Canada (exploiting the variation within the country). <em>X</em> is a vector of controls including population and GDP per capita (in power purchase parity).<em> </em>ɛ is an error term clustered at the country level (standard procedure to account for the correlation within the errors between the same countries in the covariance matrix).</p><p>The methodology I used for plotting the the coefficient is based on this blog post <a href="https://zhiyzuo.github.io/Python-Plot-Regression-Coefficient/">https://zhiyzuo.github.io/Python-Plot-Regression-Coefficient/</a> by Zhiya Zuo.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00002-6ea26696-f011-40bc-b360-3a259a00306d%3Fheight%3D260&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00002-6ea26696-f011-40bc-b360-3a259a00306d%3Fheight%3D260&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnails%2Fmain.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="260" frameborder="0" scrolling="no"><a href="https://medium.com/media/b4c56fe7860ae94221dfdd5d5953bf5d/href">https://medium.com/media/b4c56fe7860ae94221dfdd5d5953bf5d/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00003-077c357e-4d88-489d-ac86-237b09629164%3Fheight%3D620.4400024414062&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00003-077c357e-4d88-489d-ac86-237b09629164%3Fheight%3D620.4400024414062&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnails%2Fmain.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="620" frameborder="0" scrolling="no"><a href="https://medium.com/media/200e56bedac9e15678ba96b668d37f00/href">https://medium.com/media/200e56bedac9e15678ba96b668d37f00/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00003-077c357e-4d88-489d-ac86-237b09629164%3Fheight%3D620.4400024414062&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00003-077c357e-4d88-489d-ac86-237b09629164%3Fheight%3D620.4400024414062&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnails%2Fmain.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="620" frameborder="0" scrolling="no"><a href="https://medium.com/media/200e56bedac9e15678ba96b668d37f00/href">https://medium.com/media/200e56bedac9e15678ba96b668d37f00/href</a></iframe><h4>Forest plot for one regression model</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00006-2b2d60fe-f1d4-4909-97a6-d87119a27baa%3Fheight%3D1037.3399963378906&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00006-2b2d60fe-f1d4-4909-97a6-d87119a27baa%3Fheight%3D1037.3399963378906&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnails%2Fmain.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="1037" frameborder="0" scrolling="no"><a href="https://medium.com/media/d70c9cd55502462658cbd1700a819a90/href">https://medium.com/media/d70c9cd55502462658cbd1700a819a90/href</a></iframe><p><strong>Note:</strong> We can see that an increase of 1°C is associated with an increase in the environmental policy stringency index of about 0.27 (mean value of the environmental policy stringency index: 1.58) with a CI between 0.2 and 0.35. The coefficient is statistically significant at the 5% level.</p><h4>Forest plot with two regression models</h4><p>Here I will compare the coefficient of temperature for two different samples: years with above-median rainfall and years with below-median rainfall. In the first paper, I found that the effect of temperature was significantly stronger in dry years.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%3A443%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00010-905d98ff-e52d-426b-827e-44d9363674ce%3Fheight%3D242&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%3A443%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00010-905d98ff-e52d-426b-827e-44d9363674ce%3Fheight%3D242&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnails%2Fmain.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="242" frameborder="0" scrolling="no"><a href="https://medium.com/media/f605d9fc20f92d459215f6e3f4187b2f/href">https://medium.com/media/f605d9fc20f92d459215f6e3f4187b2f/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00011-d1290afb-fb4f-43ac-bb41-2e8d09a3ff9b%3Fheight%3D512&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00011-d1290afb-fb4f-43ac-bb41-2e8d09a3ff9b%3Fheight%3D512&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnails%2Fmain.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="512" frameborder="0" scrolling="no"><a href="https://medium.com/media/87d39961fec9799f50350f8fb45bf35c/href">https://medium.com/media/87d39961fec9799f50350f8fb45bf35c/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00012-9600f82c-7b51-4347-8075-23970faaeced%3Fheight%3D188&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00012-9600f82c-7b51-4347-8075-23970faaeced%3Fheight%3D188&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnails%2Fmain.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="188" frameborder="0" scrolling="no"><a href="https://medium.com/media/836b66ea35208eea2ff7b9ae83492b94/href">https://medium.com/media/836b66ea35208eea2ff7b9ae83492b94/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%3A443%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00013-f136bded-5766-424d-829d-37a67a008f8b%3Fheight%3D1020&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%3A443%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00013-f136bded-5766-424d-829d-37a67a008f8b%3Fheight%3D1020&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnails%2Fmain.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="1020" frameborder="0" scrolling="no"><a href="https://medium.com/media/526d711b7c8040b2d61c0eaed871c1c3/href">https://medium.com/media/526d711b7c8040b2d61c0eaed871c1c3/href</a></iframe><p><strong>Note: </strong>We are able to confirm the finding from the previous article that the effect is larger when rainfall is low (below median) and actually not statistically significant at the 5% threshold for the sample with rain above median. Moreover, we can see that the difference between the two coefficients is statistically significant at the 5% threshold. This comes from the fact that the confidence intervals of each coefficient are not going over the other coefficient.</p><h4>Forest plot with two regression models and multiple variables</h4><p>Here, I will compare the coefficient for temperature for two different models: without and with control variables (GDP and fossil fuel consumption).</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%3A443%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00016-4881837f-80b0-43cc-ab06-b6d8df7656b2%3Fheight%3D530&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%3A443%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00016-4881837f-80b0-43cc-ab06-b6d8df7656b2%3Fheight%3D530&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnails%2Fmain.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="530" frameborder="0" scrolling="no"><a href="https://medium.com/media/a1152143d4d523816763727105128d19/href">https://medium.com/media/a1152143d4d523816763727105128d19/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00017-65829b70-387f-4a5e-a934-6cacfae97577%3Fheight%3D242.1875&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00017-65829b70-387f-4a5e-a934-6cacfae97577%3Fheight%3D242.1875&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnails%2Fmain.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="242" frameborder="0" scrolling="no"><a href="https://medium.com/media/4c02bcedc4afb6f3d8c280067c2becc6/href">https://medium.com/media/4c02bcedc4afb6f3d8c280067c2becc6/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00018-0185fa9f-e22a-4cb1-bbbc-e18c4f2654ee%3Fheight%3D1135.8000183105469&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00018-0185fa9f-e22a-4cb1-bbbc-e18c4f2654ee%3Fheight%3D1135.8000183105469&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnails%2Fmain.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="1135" frameborder="0" scrolling="no"><a href="https://medium.com/media/0ed7e1cefeaf8d796b926f456113cb88/href">https://medium.com/media/0ed7e1cefeaf8d796b926f456113cb88/href</a></iframe><p><strong>Note:</strong> We can see that the coefficient on temperature is no longer statistically significant once we control for GDP and fossil fuel consumption. It seems therefore that our main effect suffered from an omitted variable bias. The correlation table below reveals that GDP is strongly negatively correlated with temperature (-0.45) and strongly positively correlated with the environmental policy stringency index (0.65). Thus, the basic positive coefficient with temperature is rather due to the fact that richer countries are in colder locations and that these countries tend to implement more environmental policies.</p><p>Note also that it is difficult to compare the coefficients here because they are at different scales (GDP in dollars, fossil fuels as a share of consumption, temperature in °C). It is therefore preferable to normalise the coefficients by the standard deviation of each variable. See next section.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00020-c6c9c3a1-1433-468b-a87b-95cf878e839e%3Fheight%3D515&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00020-c6c9c3a1-1433-468b-a87b-95cf878e839e%3Fheight%3D515&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnails%2Fmain.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="515" frameborder="0" scrolling="no"><a href="https://medium.com/media/97b4992d6c6422227d66ea9612b7afad/href">https://medium.com/media/97b4992d6c6422227d66ea9612b7afad/href</a></iframe><h4>Forest plot with two regression models and multiple variables standardized</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00022-5b0b3615-d205-430b-b79d-3aaeed9c51f6%3Fheight%3D641.8000183105469&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00022-5b0b3615-d205-430b-b79d-3aaeed9c51f6%3Fheight%3D641.8000183105469&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnails%2Fmain.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="641" frameborder="0" scrolling="no"><a href="https://medium.com/media/8318cdb9617eb41b6815cf1bec563d8f/href">https://medium.com/media/8318cdb9617eb41b6815cf1bec563d8f/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%3A443%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00023-f7da9c67-c7ae-41f7-b8ac-36d69f29d01f%3Fheight%3D530&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%3A443%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00023-f7da9c67-c7ae-41f7-b8ac-36d69f29d01f%3Fheight%3D530&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnails%2Fmain.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="530" frameborder="0" scrolling="no"><a href="https://medium.com/media/936d1f7658308ee33d21a50d9348d719/href">https://medium.com/media/936d1f7658308ee33d21a50d9348d719/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00024-894c73ad-2123-4bec-b752-55a007741b70%3Fheight%3D242.1875&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00024-894c73ad-2123-4bec-b752-55a007741b70%3Fheight%3D242.1875&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnails%2Fmain.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="242" frameborder="0" scrolling="no"><a href="https://medium.com/media/7f0ca6f0514a30ecb28001207b4870d0/href">https://medium.com/media/7f0ca6f0514a30ecb28001207b4870d0/href</a></iframe><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00025-936347c2-0957-4105-800a-2318ab3ccdb8%3Fheight%3D1131.8000183105469&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00025-936347c2-0957-4105-800a-2318ab3ccdb8%3Fheight%3D1131.8000183105469&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnails%2Fmain.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="1131" frameborder="0" scrolling="no"><a href="https://medium.com/media/db8eac24839215a64e496a76ed42b254/href">https://medium.com/media/db8eac24839215a64e496a76ed42b254/href</a></iframe><p><strong>Note:</strong> Now that the coefficients are standardised, it is easier to compare their magnitude and see the statistical significance of each. However, as we saw in the first part of this article, it is more difficult to interpret the coefficient. Let me illustrate this with the temperature coefficient for the model without controls. A one standard deviation (7.3°C) increase in temperature is associated with a 2 point increase in the policy stringency index (mean value of the index: 1.58).</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fembed.deepnote.com%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00027-2fb91b68-9000-45c2-a9d3-b7cd07e7df14%3Fheight%3D696&amp;display_name=Deepnote&amp;url=https%3A%2F%2Fembed.deepnote.com%2Fca9bafb2-0a1d-4bba-ade3-d3f6d2815395%2F055afa6a-e4ea-4d1c-8d21-c9b12c8a6d74%2F00027-2fb91b68-9000-45c2-a9d3-b7cd07e7df14%3Fheight%3D696&amp;image=https%3A%2F%2Fdeepnote.com%2Fstatic%2Fthumbnails%2Fmain.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=deepnote" width="700" height="696" frameborder="0" scrolling="no"><a href="https://medium.com/media/4d1b40a43e6e2e73f45d288dc5f95491/href">https://medium.com/media/4d1b40a43e6e2e73f45d288dc5f95491/href</a></iframe><h4>References</h4><p>[1] van Zwet, E. W., &amp; Cator, E. A. (2021). The significance filter, the winner’s curse and the need to shrink. Statistica Neerlandica, 75(4), 437–452.</p><p>[2] Wasserstein, R. L., &amp; Lazar, N. A. (2016). The ASA statement on p-values: context, process, and purpose. The American Statistician, 70(2), 129–133.</p><p>[3] Povitkina, Marina, Natalia Alvarado Pachon, and Cem Mert Dalli. “The Quality of Government Environmental Indicators Dataset, version Sep21.” University of Gothenburg: The Quality of Government Institute, <a href="https://www.gu.se/en/quality-government">https://www.gu.se/en/quality-government</a> (2021).</p><p>[4] Botta, Enrico, and Tomasz Koźluk. “Measuring environmental policy stringency in OECD countries: A composite index approach.” (2014).</p><p>[5] The World Bank Group. 2021. Climate Change Knowledge Portal. <a href="https://climateknowledgeportal.worldbank.org/">https://climateknowledgeportal.worldbank.org</a></p><p>[6] World Bank, World Development Indicators. (2022).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d196c936f8f2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/unhappy-with-statistical-significance-p-value-here-is-a-simple-solution-d196c936f8f2">Unhappy with statistical significance (p-value)? Here is a simple solution</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>