<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Shichao Ma on Medium]]></title>
        <description><![CDATA[Stories by Shichao Ma on Medium]]></description>
        <link>https://medium.com/@shichaoma?source=rss-2476826c2c2c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*CCmbo4l3dW_Ybg9y.jpg</url>
            <title>Stories by Shichao Ma on Medium</title>
            <link>https://medium.com/@shichaoma?source=rss-2476826c2c2c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 02:08:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shichaoma/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Use ML Models for Variance Reduction in A/B experiments]]></title>
            <link>https://shichaoma.medium.com/use-ml-models-for-variance-reduction-in-a-b-experiments-b2c66533a668?source=rss-2476826c2c2c------2</link>
            <guid isPermaLink="false">https://medium.com/p/b2c66533a668</guid>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[variance-reduction]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistical-power]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Shichao Ma]]></dc:creator>
            <pubDate>Sat, 19 Jun 2021 11:37:56 GMT</pubDate>
            <atom:updated>2021-06-19T11:37:56.714Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<h3>TL;DR</h3><p>ML models can significantly reduce variance and hence increase the power of your A/B experiments.</p><h3>Introduction</h3><p>Machine Learning (ML) models are good at making predictions. But in general they are not widely used for recovering causality. I have always been wondering if they can do us any good for causal inference. I will explore this very topic by running some simulations and report back to the community with a series of articles.</p><p>In this series, I plan to examine the following cases:</p><ol><li>(This article) The treatment is assigned completely at random (i.e., in an A/B experiment).</li><li>We have observational data and the ignorability assumption holds.</li><li>We have observational data and the exclusion restriction holds.</li></ol><p>Note that I plan to work with binary treatments throughout the whole series. If you have never seen the term “ignorability” or “exclusion restriction,” do not worry about them now. I will explain them when I write about them.</p><p>When the treatment is assigned completely at random, ML models not only <em>can</em> recover causality, but can do it <em>better</em>. You can achieve a higher statistical power if you use a ML model (compared to vanilla t-tests or linear regressions).</p><p>Next I will examine several simulation cases, from simple to difficult for a model to learn, as evidence.</p><h3>Homogeneous Treatment Effect</h3><p>What is the homogeneous treatment effect? Homogeneous treatment effect means each and every sample in your dataset has the exact same effect from the treatment. This possibly will never be true in reality, but it is a nice baseline case. If our model fails in this case, then we do not need to go further.</p><p>Let’s generate some simulation data to work with. We will stick to simulated data so that we can control the true treatment effect. Otherwise we will never verify if our model has recovered the truth because of the fundamental problem in causal inference.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7cd1a7f161c68a1273e6c4830c7ad7ef/href">https://medium.com/media/7cd1a7f161c68a1273e6c4830c7ad7ef/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/659/1*ssWsc7vNTN4bEwTPyKj7Aw.png" /><figcaption>Five generated data samples</figcaption></figure><p>In this dataset, treatment is independently drawn from a Bernoulli distribution with p=0.5. We have 6 features other than the treatment. Features a, b and c influence the outcome in a complex and non-linear way. Feature e and f do not influence the outcome in any way. The treatment effect is 0.1 throughout the board. Our task is to recover this 0.1.</p><p>Let’s take a look at the distribution of the outcome, grouped by the treatment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/416/1*44TVtaWiWYQPa072u3SjbQ.png" /></figure><p>From the plot, we can see the outcome is continuously distributed. The distribution of the control group is not that different from the treated group. If we just eyeball it, we might conclude that there is little to no effect.</p><p>Let’s build a ML model that uses all we have (features a to f and the treatment) to predict outcome.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b49fd2dc16adc5c2ff5a3dc08ac047ce/href">https://medium.com/media/b49fd2dc16adc5c2ff5a3dc08ac047ce/href</a></iframe><p>To estimate the treatment effect, we need to run the prediction twice. We set treatment to 0 and 1 for all samples and predict the outcomes respectively. Then we take the difference of the two sets of predictions and take the average of the differences. If the model can recover the average treatment effect, then we should get a number that is close to 0.1.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/384a565192cf6cef189654e7e8cd6454/href">https://medium.com/media/384a565192cf6cef189654e7e8cd6454/href</a></iframe><p>We get:</p><pre>Predicted Average Treatment Effect: 0.1028</pre><p>0.1028, which is not equal to, but quite close to 0.1. Does this mean we fail? Or is this just noise? To answer this question, we need the standard error and/or the confidence interval of the estimated effect. But unfortunately we do not have a nice formula for the standard error because the model is a black box. We have to find the standard error and the confidence interval through non-parametric bootstrap.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/34ad311af5ab165d500689188dfa72b8/href">https://medium.com/media/34ad311af5ab165d500689188dfa72b8/href</a></iframe><p>The result from the bootstrap goes as follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1010fd032bfd57e00659e6eca550e3f5/href">https://medium.com/media/1010fd032bfd57e00659e6eca550e3f5/href</a></iframe><p>We can see the estimated effect is indeed very close to 0.1 and the 95% confidence interval covers the true value. Let’s also plot the results from all bootstrap runs to check if the distribution looks reasonable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/352/1*wwpK4BKplD9tYoOaGIs11g.png" /></figure><p>Good! Now I am confident to say ML models can recover the causal effect in this case.</p><p>But you probably want to argue this is just the standard A/B test and we don’t need LightGBM to recover the causal effect. You are right. Let’s see how other approaches perform compared to the ML based one. Let’s first do the standard t-test. Because it is so standard, I will omit the code and show you the result here:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ad949fb36b1bfaac802b9ebcd6447a8e/href">https://medium.com/media/ad949fb36b1bfaac802b9ebcd6447a8e/href</a></iframe><p>The vanilla t-test obviously also recovers the effect. However, the most striking difference is the standard error and the confidence interval. Here the standard error is 280% of the one given by the LightGBM bootstrap.</p><p>Is this because the standard error given by the ML based approach is wrong? Let me give you two other standard errors by trying two other approaches.</p><p>The first approach is a linear regression with the treatment indicator and all the linear terms of the features (i.e., treatment + a + b + c + d + e + f + constant). The coefficient on treatment is the estimated effect, packages like statsmodels will give us the associated standard error and the confidence interval. The result goes as follows.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7c317c1c0f5ecc0d9b8cd81a689574c7/href">https://medium.com/media/7c317c1c0f5ecc0d9b8cd81a689574c7/href</a></iframe><p>The estimated effect is still correct, and the standard error is smaller than the t-test one.</p><p>Before I explain why, let’s look at another approach. We will still stick with the linear regression, but this time let’s add some polynomial terms to better capture the non-linear effects we baked in earlier. I will add quadratic and interactive terms on top of existing linear terms (that is, we add a**2 + b**2 + … + f**2 + a*b + a*c + … + e*f). And here is the result:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ff6f19049c6bbb1fe151bb4568b23622/href">https://medium.com/media/ff6f19049c6bbb1fe151bb4568b23622/href</a></iframe><p>The standard error further goes down, but it is still no way near the ML based one.</p><p>What is going on here? Why do we have different standard errors and why the one from ML models is the lowest?</p><p>Let’s answer the former first. All these approaches can give us unbiased and consistent estimates of the causal effect, but the noise level contained in each estimate is different. In the t-test, we only utilize information on the treatment (in practice, this means we only utilize the cohort assignment information). All the contributions from other features to the outcome are considered “noise” under this approach. So the noise level is the highest.</p><p>If we want to reduce the standard error, then we need more data to overcome the noise. We can continue using t-test and gather more data points. This is a perfectly fine strategy if your traffic is cheap.</p><p>But more data does not necessarily imply more data <em>points</em>. We can add more data horizontally by adding more features. Features that help to predict the outcome help us to reduce the noise. With the help of these features, the treatment column does not have to fight alone. In some sense, the noise on the treatment’s shoulder is less.</p><p>With this, we can understand why the standard errors from these approaches rank in this way.</p><p>In the first linear regression, the linear terms take some burden off of the treatment, but these linear terms cannot fully approximate the nonlinear functional forms we cooked up earlier. So the non-linear parts are still “noise” to the treatment. Therefore, we see reduced standard error, but not quite comparable to other approaches.</p><p>In the second linear regression, the polynomial terms help to fit the non-linear contributions of the features. Therefore we see further reduction from the first linear regression. However, these polynomial terms still cannot perfectly approximate the non-linearity we cooked up earlier and the treatment still has quite a bit left-overs to deal with.</p><p>ML models are good at fitting unknown and complex functions. With a sufficiently flexible model, we can fit the true relationship between features (i.e., a to f) and the outcome as much as possible. Therefore, we can leave little noise for the treatment. This is why the ML based standard error is much lower.</p><p>For some not 100% scientific evidence, I fitted all the models without the treatment column (for t-test, this just means a model with a constant term only). Let’s take a look at their residuals’ standard deviations.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e8bb964abc9af47950e3e4c9cdeb137a/href">https://medium.com/media/e8bb964abc9af47950e3e4c9cdeb137a/href</a></iframe><p>Again not 100% scientific, but you can think of these standard deviations as the noise levels that the treatment column needs to fight against. Clearly the one given by the LightGBM model is the lowest.</p><h3>Heterogeneous Causal Effects</h3><p>In the previous example, you might think I’m cheating because the treatment is only linearly added to the outcome. What if we make it more complex so that each sample’s causal effect is different? Let’s give it a try.</p><p>I will examine two cases. The first case is mathematically a bit simpler: each sample’s treatment effect only depends on feature c. In the second case, features a, b and c all influence the treatment effect in different ways. If the ML based approach can take care of both, then I feel comfortable relying on ML (in this case LightGBM) models in practice.</p><p>The following block is the code that generates the outcome for the two cases.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/90199eb7eddcb51a202df5fc869f4225/href">https://medium.com/media/90199eb7eddcb51a202df5fc869f4225/href</a></iframe><p>Case 1’s true average causal effect is about -0.4432 and case 2’s true average causal effect is about -2.9390 (they are computed by numerical integration).</p><p>For case 1, here is the result comparison among all four approaches:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ac78bdae046b72ebe937d85c55dd8e65/href">https://medium.com/media/ac78bdae046b72ebe937d85c55dd8e65/href</a></iframe><p>And here is the result for case 2:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/400b62ebbd04868287f7c7a111e0b47c/href">https://medium.com/media/400b62ebbd04868287f7c7a111e0b47c/href</a></iframe><p>They all manage to recover the true effect. As in the previous case, the standard errors go down when non-treatment features capture more noise. For the highest power, you still want to choose the ML based approach.</p><h3>What have we learned from this exercise?</h3><p>In an experimental setting, the flip side of noise level is statistical power. Higher noise level, lower statistical power, and vice versa.</p><p>When you do a t-test in your A/B test workflow, everything else the same, you need the highest number of data points to fight against the noise. If you have enough traffic, then this is perfectly okay.</p><p>But when you do not have enough traffic and yet you’re not willing to lower the statistical power or increase the effect size, then you can include features that help to predict the outcome variable. One important caveat is that those features cannot be influenced by your treatment, otherwise you have label leakage. So one practical approach to implement in your pipeline is logging some features of your subject at the same time they enter your intervention page.</p><p>In the analysis phase, you can use either linear regressions or ML models to achieve noise reduction. I suggest you implement both. Linear regression, although not as effective, is much faster to compute. You have to do bootstrap when you use the ML based approach, and you need to train a new model in each bootstrap run. In this mock example with only 20k data points, the ML based approach takes about 30 minutes in the free version of Google’s Colab while the linear regression is almost instantaneous.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b2c66533a668" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Propensity Score Matching: A Practitioner’s Appendium]]></title>
            <link>https://shichaoma.medium.com/propensity-score-matching-a-practitioners-appendium-ddbd641363b2?source=rss-2476826c2c2c------2</link>
            <guid isPermaLink="false">https://medium.com/p/ddbd641363b2</guid>
            <category><![CDATA[causal-inference]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[matching]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Shichao Ma]]></dc:creator>
            <pubDate>Mon, 31 May 2021 12:46:32 GMT</pubDate>
            <atom:updated>2021-05-31T12:46:32.348Z</atom:updated>
            <cc:license>http://creativecommons.org/licenses/by/4.0/</cc:license>
            <content:encoded><![CDATA[<h3><strong>TL;DR</strong></h3><p>This article discusses the usage and a common pitfall of the Propensity Score Matching (PSM) from a working data scientist’s perspective in the language that (hopefully) is accessible to most data scientists.</p><ol><li>In a simulation exercise, I show PSM can be used to remove functional form dependence.</li><li>The key assumption in matching is the ignorability assumption. You will see why you want to make sure this assumption holds.</li><li>You will find a user-friendly way to test the ignorability assumption.</li><li>Code snippets using Python libraries that are familiar to most data scientists are provided.</li></ol><p>Check the jupyter notebook version of this article <a href="https://github.com/shichao-ma/propensity_score_matching_notebook/blob/main/test_ignorability.ipynb">here</a>!</p><h3>Introduction</h3><p>It is well known to data scientists that “correlation does not imply causation.” For example, let’s consider a hypothetical situation where a hospital is considering promoting either of the following doctors (I’m sure in reality hospitals do not work this way). One candidate typically ends up with patients that either died or still had ongoing conditions when they left the hospital. The other candidate’s patients mostly fully recovered from their illness. Does this comparison mean the former is incompetent?</p><p>On the contrary, it is possible the former is far more competent. For various reasons, good doctors work with tricky cases more often and tricky cases are tricky to work with even for the best doctors. This is known as the selection effect.</p><p>Selection effect can mute or, in extreme cases like the one above, completely opposite the correlation from causation. Unfortunately, the selection effect is everywhere in human activities. Tech sector is no exception. For instance, hosts that display more photos on Airbnb might have more bookings. But this mere correlation does not imply actionable interventions unless you can establish causation. Encourage hosts of horrible places to upload more photos of their horrible places? Good luck with that.</p><p>Then how to establish the causation that a good doctor <em>is</em> good? To do so, one way is to figure out factors that influence the “trickiness” of an illness and control them in your analysis. Let’s call it controlling confounders. Another way is to find a subset of patients from the two doctors’ patient pools that are similar in their severity before they are admitted and do the comparison only on this subset. The latter is known as the matching method.</p><p>In the rest of this article, I will first introduce a popular matching method called the propensity score matching (PSM), which I think is the most feasible matching method when you work with datasets of many many features. Then I will walk you through the reasons why you want to choose the PSM over controlling confounders. Lastly, I will highlight to you a super duper important assumption of PSM and give you a simple way to test it.</p><h4>A Note on the Terminology</h4><p>We have three important terms in this article that are typically unseen in predictive modeling exercises.</p><ul><li>Treatment: intervention, the item that is different from the status quo you’re contemplating to apply. I will use its verb variants like “treat” or “treated” to describe applying the intervention to a certain population. In this article, I will only discuss the situation where the treatment is binary. That is, we have one intervention and one status quo, the typical A/B case.</li><li>Propensity score, or simply propensity: the probability of being treated. The reason I insist on using this “foreign” term is that it will be easier for you to find relevant literature should you decide to read on.</li><li>Outcome: the metric you care about, typically your primary business metric that you’re hoping to optimize.</li></ul><p>For folks that are more familiar with A/B experiments, treatment means the new item you are testing against the status quo; propensity means the probability of a subject receiving the new item; and outcome is your experiment’s primary metric. Unlike A/B experiments, we are working with the case where the treatment/intervention is not completely randomly assigned.</p><h3>Data Generation</h3><p>Let’s consider a hypothetical situation where you have a binary treatment, a continuous outcome, several features that may or may not be important to the propensity or the outcome. Let’s generate some simulation data to match this situation. The reason we’re working with the generated data is that we can control the data generating process so that we will have a truth to compare against.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a4981a84309a910bfccaa9813b9d3edb/href">https://medium.com/media/a4981a84309a910bfccaa9813b9d3edb/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/699/1*9Lw4PYPHfno-CBFGd_BL9w.png" /><figcaption>Head of the DataFrame</figcaption></figure><p>Note that here I generated 6 features. The true propensity score (true_ps) is decided by a, b, c, and f via a standard logistic form (expit is just the Sigmoid function) so that we do not need a sophisticated non-linear model on the propensity score. The treatment column indicates the binary intervention. It is drawn from a Bernoulli distribution with the success probability being true_ps. The outcome column is the metric we care about. Here it is decided stochastically by a, b, and c in a highly non-linear way. It turns out the non-linearity is necessary for a point I’m about to make. In reality, however, the relationship between your business metric and your features might be more complex than this.</p><p>Importantly, the outcome has nothing to do with the treatment in this very case. In other words, the true treatment effect is zero throughout the board.</p><p>In practice, we can observe all the 6 features, the treatment label, and the outcome column. But we will not know the true propensity score, or the functional forms that generate the true propensity score or the outcome.</p><p>The following is the relationship between the true propensity score and the treatment. Not surprisingly, treated samples have higher propensity scores in general.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/416/1*PQ1ERIAAATmPAXqW1xq5VA.png" /><figcaption>True Propensity Grouping By Treatment</figcaption></figure><h3>Why do we Need Matching?</h3><p>It is no secret that you need to consider confounding factors when you try to make any claim that people might have a causal read.[1]</p><p>Suppose a data scientist has correctly identified a to f as confounding factors and decides to include them in their analysis. Following the common practice, they run a regression on the treatment label and all the confounding features, as follows.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ef531a455aa75bc45072c719d79b10a8/href">https://medium.com/media/ef531a455aa75bc45072c719d79b10a8/href</a></iframe><p>They find the model attributes a -0.1 effect to the treatment, and it is statistically significant under conventional thresholds. If the data scientist is not careful, they would conclude the treatment negatively impacts the outcome.</p><p>Because we know the underlying data generating process, we know this -0.1 effect is false. But why does the model attribute any effect to the treatment? In this model the confounders enter the regression in a linear way. Because the functional form is not able to fully capture the true relationship (the nonlinearity we cooked up in the data generating part) and the treatment label is correlated with those confounding features, some of the nonlinear effects are attributed to the treatment.</p><p>Can a nonlinear model help? In theory, yes. If the model has the correct functional form, then the model is able to find the true treatment effect is zero. But without knowing the true relationship, this is unlikely to happen. Can you guess the true functional form that I arbitrarily wrote down just by looking at the date? Probably not.</p><p>Only if there is a method to incorporate confounders without knowing how they affect the outcome… Lucky you, we do have such a method. This method is, guess what, propensity score matching (or matching in general).[2]</p><p>Before I explain how, let’s first go through the whole matching process. A typical PSM workflow has three steps:</p><ol><li>Build a model for the propensity score.</li><li>Find matched pairs based on the predicted propensities.</li><li>Perform analysis using the matched sample.</li></ol><h4>Build a Propensity Score Model</h4><p>Because we only consider the binary treatment here, a propensity score model is just a binary classifier that can make probability predictions.</p><p>Here I will cheat a little bit. I run a simple (means not regularized) logistic regression as the propensity model. Because we already know the data generating process of the propensity score, this is the most efficient model.</p><p>In a real problem, you may want to use a more capable classifier. If you decide to train a binary classifier like your typical ML project, then you still want to follow the best practices on the bias and variance tradeoff. In this use case, however, we do not care about accuracy/precision/recall/f1 or any other evaluation metrics that only compare <em>binary</em> predictions against their labels. Instead, you want to choose an evaluation metric that captures the goodness of predicted probabilities. The good old “log loss” is usually a good choice.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/24bcccf98a2600ebe6cc84540708dade/href">https://medium.com/media/24bcccf98a2600ebe6cc84540708dade/href</a></iframe><p>The predicted propensity score is simply the probability prediction on the training set. Do not worry about the traditional ML’s insistence on the out-of-sample predictions in this case.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8393481de2eec9ec586a8d649830b2aa/href">https://medium.com/media/8393481de2eec9ec586a8d649830b2aa/href</a></iframe><h4>Pairing up using the Propensity Predictions</h4><p>I use the following algorithm to form matched pairs.[3] In the literature, this method is called the nearest neighbor with a caliper. It is a greedy algorithm. It iteratively forms pairs using the closest matches that are within a certain threshold (caliper) without replacement. You can also go for the optimal matching that minimizes the average distance within pairs, although studies (e.g., Gu and Rosenbaum, 1993) show their empirical performances are typically on par with each other.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/80cfdad0d16c224590c1b2a77866b49f/href">https://medium.com/media/80cfdad0d16c224590c1b2a77866b49f/href</a></iframe><p>Let’s take a peek at the post match data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*_LTwHj3KuUZX2JXW_M2Urw.png" /></figure><p>The _score column is the predicted propensity score. As expected, it is close to the true score. The column _match_id is the id of the pair. Two rows that form a comparable pair in this dataframe share one id.</p><p>Notice that the size of the matched sample is much smaller because we dropped all the single data points.</p><h4>Analysis with the Matched Sample</h4><p>On this matched sample, let’s run the original regression again and see if the result is any different.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b8b55a5dbcfa6c749ea553c6ef0fa593/href">https://medium.com/media/b8b55a5dbcfa6c749ea553c6ef0fa593/href</a></iframe><p>Look at the coefficient on the treatment. This time we correctly fail to reject the treatment effect being zero. This is the magic of PSM.</p><p>But what’s going on? The reason is that PSM removes the mix shift and balances the data in all dimensions. When the data itself is balanced, the functional form does not matter any more.</p><p>For a concrete example, let’s examine the distribution of feature a before and after the matching. Here is its distribution before the matching, grouping by the treatment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*3xUdgFmFyTm-Q-cEjRA0iQ.png" /><figcaption>Distribution of Feature “a” Grouping by Treatment Before Matching</figcaption></figure><p>Now you see it. The feature is not balanced with respect to the treatment before the matching. Recall the doctor example in the introduction, we hope to find two groups that are comparable and now the two groups are not. Because feature a plays a key and non-linear role in producing the outcome, this mix shift effect is wrongly associated with the treatment label.</p><p>To show this visually, let’s check the scatter plot between a and the outcome.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*PdaNSGW1PJUOliw0O-Fkmw.png" /><figcaption>Relationship between “a” and the Outcome Before Matching</figcaption></figure><p>We can see more blue points on the right up corner, which moves up the blue line. Roughly speaking, the gap between the two lines (caused by mix shift) is wrongly attributed to the treatment label.</p><p>Let’s visit the matched sample and see if we can find any mix shift.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*t9t4OViKKJNUfrNmL8h1tA.png" /><figcaption>Distribution of Feature “a” Grouping by Treatment After Matching</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/429/1*wmNdO9HXUCS7sWgUG7hPzg.png" /><figcaption>Relationship between “a” and the Outcome After Matching</figcaption></figure><p>The answer clearly is no.</p><h3>The Ignorability Assumption</h3><p>The ignorability (also known as unconfoundedness, conditional independence, or selection on observables) assumption means we have included all the features that affect both the outcome and the treatment assignment in our models.</p><p>This is a tricky assumption: it is super duper important and yet very hard to test. You can check <a href="https://arxiv.org/pdf/2012.15716.pdf">this</a> 80-page and full-of-math paper if you feel like learning something. I’ll give you a simple procedure to test this assumption only using scikit-learn. But before we do this, let me first emphasize its importance.</p><h4>The Violation of the Ignorability Assumption</h4><p>Not all problems in data science are sensitive to assumptions. Therefore, maybe not all of your fellow data scientists are allergic to skipping tests on assumptions. How bad can it be if we fail to include some features? We lose 1% of model performance? Who cares?</p><p>Well the answer is the estimated treatment effect can be biased. This bias can lead to qualitatively incorrect business decisions that might cost you millions or more.</p><p>In the following simulation, let’s exclude a and b. In practice, maybe you are not aware of their importance; maybe it takes weeks to build the pipeline to process them; maybe you never gather them to begin with; or maybe they are private to your users and you can never observe it (e.g., the true feeling about your product). In this exercise, we know both of them are critical for the propensity and the outcome equations. Therefore, we’re violating the ignorability assumption. Is this a big deal? Let’s find out.</p><p>We run the propensity score model, generate the propensity score prediction, and do the matching as usual.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b7a76bc36b83f830969f4946eb2f9e01/href">https://medium.com/media/b7a76bc36b83f830969f4946eb2f9e01/href</a></iframe><p>The model attributes 1.1 to the treatment (and this number is statistically significant in all conventional significance levels). If we were to make business decisions based on this analysis, we would reach a false conclusion and potentially lose a lot.</p><p>Why do we end up with 1.1 on the treatment’s coefficient? Because a and b are important to the outcome and yet we do not include them, the regression tries to attribute some of their effects on the treatment label, which is correlated with a and b.</p><p>In summary, if we violate the ignorability assumption, then the estimated treatment effect can be much biased.</p><h3>How to Test Ignorability?</h3><p>Now we know we should really treat the ignorability assumption seriously. But how do we know if we meet this assumption or not?</p><p>At first sight, the ignorability assumption is difficult to test. How do you test whether you have missed features that you might have never considered their existence?</p><p>Sometimes you don’t have to. When you control the treatment assignment process, you already know which features influence the propensity. For example, a doctor may decide her prescription based on a set of lab results; your marketing team may decide which users to send a coupon code to based on a spreadsheet. In either case, we only need to grab the information the doctor or your marketing team has.</p><p>But what if you don’t fully understand the treatment assignment process? In this situation, you might have a laundry list of features but you are still worried about the ignorability assumption: your laundry list might just miss a couple of key items.</p><p>To test whether your laundry list is good enough, we need to utilize the residuals. For any model, residuals contain information in the training labels that cannot be attributed to the included features. In other words, residuals summarize the unincluded features that would be relevant to the model if they were included.</p><p>To perform tests on the ignorability assumption, we need to model the outcome as well. If the ignorability assumption holds, then we should expect the residuals of the propensity model and the outcome model to be independent because we have already included features that are critical to both. Conversely, we know we have missed something important.[4]</p><p>Let’s put this idea to the test.</p><h4>The Outcome Model</h4><p>Because the data generating process for the outcome is nonlinear, we need a model that can fit a flexible functional form. Here I choose LightGBM simply because I’m currently playing with LightGBM. You should feel free to choose others as long as they can capture complex nonlinear relationships (e.g., XGBoost or some AutoML framework).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9d48d70b81382c5819d9dc26db25fea0/href">https://medium.com/media/9d48d70b81382c5819d9dc26db25fea0/href</a></iframe><p>Now we have the outcome model. Let’s compute residuals from the two models.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/32dfab4d94d40f9af14c5b277192cfa5/href">https://medium.com/media/32dfab4d94d40f9af14c5b277192cfa5/href</a></iframe><p>Here is a scatter plot of the two sets of residuals.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/385/1*_JzB4mUcKDY0LegRi2UbWw.png" /><figcaption>Residual-Residual Plot</figcaption></figure><p>From the picture it is hard to tell if there is any relationship. Can we have a formal hypothesis test?</p><h4>Test Residual Dependence with Mutual Information</h4><p>It turns out we have an established measure on the dependence between two random variables in information theory. This measure is called <a href="https://en.wikipedia.org/wiki/Mutual_information">mutual information</a> (MI). If two random variables are independent, then their MI is equal to 0. Otherwise it will be above 0.</p><p>Its sample counterpart is a statistic that measures the dependence of two random samples. scikit-learn has an <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.mutual_info_regression.html">implementation</a> under its feature_selection module (yes its intended usage is to do feature selection, feel free to explore this usage).</p><p>I use the following code snippet to compute MI. It is a light wrapper of mutual_info_regression that ensures the input has the desired shape and the output is a scalar.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/939a99d39d623a6a4b4cd673c8a6f61b/href">https://medium.com/media/939a99d39d623a6a4b4cd673c8a6f61b/href</a></iframe><p>Unfortunately, the distribution of the sample MI is unclear (Hutter 2002 has a Bayesian posterior derivation, but its frequentist derivation to my knowledge does not exist yet). But, under the null hypothesis, it is very easy to bootstrap its distribution and compute the p value.</p><p>The idea of the bootstrap is quite simple. Under the null hypothesis, the two residuals are independent. So, any re-sample should not change the statistical behavior of the two residual samples. Re-sample, on the other hand, guarantees independence. So conversely if the two sets of residuals are not independent, then the original MI will be quite different from the re-sampled MI.</p><p>The code snippet for bootstrapping MI is given as follows.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5dc04d7ca457b0b0321e84bcd1b6bd5d/href">https://medium.com/media/5dc04d7ca457b0b0321e84bcd1b6bd5d/href</a></iframe><p>Let’s try it on the models that satisfy the ignorability assumption.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/06df7a54b50cf71b03e6b972133d0ddc/href">https://medium.com/media/06df7a54b50cf71b03e6b972133d0ddc/href</a></iframe><p>As expected, the sample MI of the two residuals is close to zero and the p value is also safely above 0.</p><p>What if we exclude important variables? Let’s try the residuals that come from the wrong models.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ad21893fcb9dde3d90f274f65acf5718/href">https://medium.com/media/ad21893fcb9dde3d90f274f65acf5718/href</a></iframe><p>Compared to the previous case, here the MI is much higher and the p value is close 0. We can safely reject the null that we have included all the relevant features.</p><h3>Summary</h3><p>PSM is a valuable tool to help you establish the causal relationship from observational data. It can take care of confounders even if you do not know how the confounders influence the outcome. However, for the PSM to be successful, you need to make sure the ignorability assumption holds.</p><h3>Notes</h3><p>[1] For example, you may be careful enough to write down sentences like “X is <em>associated</em> with a better outcome”, but you just fool yourself if your stakeholders use it to make decisions as if X <em>causes</em> a better outcome.</p><p>[2] Informed readers may have read King and Nielsen’s (2019) criticism on the PSM. I find their advice hardly applicable on real world datasets where you may have hundreds or thousands of features of mixed types. Because of the curse of dimensionality, you can rarely find counterparts of a data point in its vicinity. So you end up with dropping a big chunk of your dataset if you want to mimic a fully blocked experiment.</p><p>[3] Well I don’t actually use it, but it helps you to understand what the matching actually does. You probably want to use an existing package like <a href="https://cran.r-project.org/web/packages/MatchIt/index.html">MatchIt</a> in your own work. If you need to implement a nearest neighbor matching yourself, you should go for an efficient nearest neighbor algorithm.</p><p>[4] For the informed readers, the idea is inspired by the Heckman selection model.</p><h3>References</h3><p>Gu, Xing Sam, and Paul R. Rosenbaum. “Comparison of multivariate matching methods: Structures, distances, and algorithms.” Journal of Computational and Graphical Statistics 2.4 (1993): 405–420.</p><p>Hutter, Marcus. “Distribution of mutual information.” Advances in neural information processing systems 1 (2002): 399–406.</p><p>King, Gary and Richard Nielsen. “Why Propensity Scores Should Not Be Used for Matching.” Political Analysis 27, 4 (May 2019): 435–454.</p><p>Masten, Matthew A., Alexandre Poirier, and Linqi Zhang. “Assessing Sensitivity to Unconfoundedness: Estimation and Inference.” arXiv preprint arXiv:2012.15716 (2020).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ddbd641363b2" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>