<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Michael Kaminsky on Medium]]></title>
        <description><![CDATA[Stories by Michael Kaminsky on Medium]]></description>
        <link>https://medium.com/@kaminsky.michael?source=rss-4df2abcf2e4c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*_xEFmGKKylQjn_hT.</url>
            <title>Stories by Michael Kaminsky on Medium</title>
            <link>https://medium.com/@kaminsky.michael?source=rss-4df2abcf2e4c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 23:00:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kaminsky.michael/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Matching Cities for Small Sample Experiments]]></title>
            <link>https://medium.com/harrys-engineering/matching-cities-for-small-sample-experiments-harrys-engineering-be4e88c2b112?source=rss-4df2abcf2e4c------2</link>
            <guid isPermaLink="false">https://medium.com/p/be4e88c2b112</guid>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Michael Kaminsky]]></dc:creator>
            <pubDate>Tue, 21 Jun 2016 00:00:00 GMT</pubDate>
            <atom:updated>2018-05-11T19:23:07.495Z</atom:updated>
            <content:encoded><![CDATA[<h4>We use a generalized form of propensity score matching to determine which characteristics of a city are most relevant to us for an experimental design.</h4><p>Our marketing team wanted to run an experiment on the impact of a certain type of advertising. Because of the difficulties in cross-device attribution, we decided to isolate the test to certain geographies and look for the impact not at the user level, but rather at the geography level. This is a pretty straightforward experimental design, except for the wrinkle that we only have budget to run this test in a small number of geographies, maybe 3 or 4 at most.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/0*qXR-A7oe4cygxJhT.png" /></figure><p>There’s lots of literature on how to accurately estimate significance on a small-sample experiment, but in particular we chose to follow the example outlined by <a href="http://www.jakebowers.org/PAPERS/BowPan-Fisher.pdf">Bowers and Panagopoulos</a> and use random inference to determine the statistical significance of the impact. Additionally, we want to match cities into pairs before assigning treatment to one half of the pair. This allows us to use a technique known as ‘block random assignment’ that minimizes the standard error of the response, allowing for a much more precise estimate of the impact of the treatment. (<a href="http://www.amazon.com/Field-Experiments-Design-Analysis-Interpretation/dp/0393979954">Gerber and Green’s</a> <strong>Field Experiments</strong>, Chapter 3 provides an excellent treatment of this subject.)</p><p>Outside of the statistical benefits associated with this matching design, it’s also simply easier for key stakeholders to understand and immediately ‘eyeball’ the impact of the experiment, which in many business applications can be very valuable.</p><p>While we’ll save the implementation of random inference for another post, here we’ll discuss how we chose pairs of cities to select for the experiment. We want to choose cities that 1) minimize the variance in the outcome of interest and 2) maximize the likelihood of observing a statistically significant effect. To do this, we want to find cities that are as similar as possible to be pairs. But what does it mean for cities to be as similar as possible?</p><p>For us, it means we want them to have similar demographics and similar population sizes. We used data from the ACS to pull the following data for every metropolitan / micropolitan statistical area in the united states1:</p><ul><li>Total population</li><li>Share of population that is male</li><li>Median age (of males)</li><li>Race (specifically, we calculated share white, share black, share asian, and share other)</li><li>Employment status (share of total population employed, and share of total population unemployed)</li><li>Median income</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/0*bx3CS4Ptac23fL2h.png" /></figure><p>Once we pull together this data, we have to create a measure of ‘similarity’. This is actually trickier than it seems at first glance because measuring the similarity between two cities in total population is easy, but measuring the similarity between two cities in terms of both population and median age is harder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/290/0*sT52K9YMU23mU5D6.gif" /></figure><p>If you just look at the simple multi-dimensional distance which you might remember from high-school math, you’ll end up treating all of the attributes the same. But that doesn’t make sense in this situation. Not only are the cities’ attributed denominated differently (percents vs total counts vs dollars), but we have many reasons to believe that these variables shouldn’t be treated equally.</p><p>Which is more important, population or age? What if there’s one city that’s closer in total population but farther apart in median age, and for another city it’s reversed? As you add more variables, this problem gets harder and harder, because you have to trade-off between ‘similarity’ in each of these different variables.</p><p>Rather than trying to guess which variables are most important, <em>_we can let the data tell us which variables are most important to us_</em>. The demographics of a city that matter most to us (selling men’s grooming products) might be different than what matters to a company like Casper (selling mattresses) or Glossier (skincare marketed to women). So, the trick here is that we can use our own data to estimate the appropriate weights for each of these variables to determine the overall similarity between cities. This technique is very frequently used in healthcare analytics to match patients who are similarly likely to experience an adverse health event, and is often called <a href="http://www.ncbi.nlm.nih.gov/pubmed/?term=propensity+score+matching">‘propensity score matching’</a>.</p><p>While you can imagine a lot of algorithms for estimating these weights (pretty much any form of regression will work), we went with the most easily interpretable — linear regression. It turns out, that the coefficients estimated in a linear regression are exactly what we’re looking for. They can be interpreted as how the variation in a given variable impacts the variation in the outcome of interest (while fixing variation in the other variables) which tells us precisely how much we should weight differences in that variable when trying to determine overall similarity.</p><p>So we estimate the linear model, predict the responses, sort the data by prediction, and voila! All of the cities in our database are ranked by their similarity, with their characteristics weighted according to their relevance to Harry’s. In particular, the number of orders we sell on average per week in a given city is a very important metric to us, so we use that as independent variable in our regression, and use the predicted values as our key distance metric.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/695/0*eBSIdqo2-oP1pqXx.png" /></figure><p>Note: the data in this example are completely fake, but you get the idea.</p><p>From here, there are a number of algorithms one can imagine for choosing pairs of cities (these algorithms become increasingly important as the sample size you need to pair grows), but we simply ‘eyeballed’ it to choose cities that were adjacent in weighted similarity scores and within a tolerance range of actual observed orders 2. Since we are only choosing a few cities for our experiment, we didn’t see a strong reason to over-complicate this piece.</p><p>So, what we end up with are 3 or 4 pairs of cities that we know are very similar in terms of their demographics, weighted by their relevance to Harry’s. From here, all we have to do is run the experiment (and evaluate the results, but that’s a story for a later blog post).</p><h3>Notes:</h3><p>1: The <a href="https://cran.r-project.org/web/packages/acs/acs.pdf">ACS package for R</a> was indispensable for the munging process, here. As was the <a href="http://censusreporter.org/topics/">census reporter</a></p><p>2: In this example, we want to minimize both the difference in similarity scores as well as actual observed orders. We need to be mindful of the ‘error’ between the similarity score and the actual observed number of orders, because an error could indicate that one city is particularly over-saturated or under-saturated with Harry’s, which could impact the results of our experiment.</p><p><em>Originally published at </em><a href="http://engineering.harrys.com/2016/06/21/matching-cities-for-small-sample-experiments.html"><em>engineering.harrys.com</em></a><em> on June 21, 2016.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=be4e88c2b112" width="1" height="1" alt=""><hr><p><a href="https://medium.com/harrys-engineering/matching-cities-for-small-sample-experiments-harrys-engineering-be4e88c2b112">Matching Cities for Small Sample Experiments</a> was originally published in <a href="https://medium.com/harrys-engineering">Harry&#39;s Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Democratizing Kaplan-Meier]]></title>
            <link>https://medium.com/harrys-engineering/democratizing-kaplan-meier-harrys-engineering-a2505f593825?source=rss-4df2abcf2e4c------2</link>
            <guid isPermaLink="false">https://medium.com/p/a2505f593825</guid>
            <category><![CDATA[analytics]]></category>
            <dc:creator><![CDATA[Michael Kaminsky]]></dc:creator>
            <pubDate>Mon, 18 Apr 2016 00:00:00 GMT</pubDate>
            <atom:updated>2018-05-11T19:22:42.738Z</atom:updated>
            <content:encoded><![CDATA[<h4>We’ve used Looker on PostgreSQL to allow anyone in the company to perform Kaplan-Meier analysis on our retention data to drive insights and make decisions.</h4><p>At Harry’s, we strive to have a robust understanding of our customers, and we’re particularly interested in identifying features of customers and their behavior that increase retention and reduce churn. One of the key difficulties in these time-to-event analyses is that measures of central tendency often give a very incomplete picture of customer behavior and so more sophisticated methods are often required.</p><p>For example, if you want to understand customer retention, you might ask ‘of those who have made one purchase from us, how many have made two’? Unfortunately, this has a lot of potential to be misleading because not all of your customers have the same ‘exposure’. That is, customers acquired 3 years ago have had 3 years to come back, but customers acquired last week probably won’t have already made another purchase. You can fine-tune this line of questioning quite a bit to drive at something that’s pretty reasonable, but any single number is going to be missing key information because time-to-event analysis is fundamentally a two-dimensional problem — you need to know both how many events happened, but also when they are happening.</p><p>Enter the retention curves, which often looks something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/0*tRtQ7oyAI3SHPDjo.png" /></figure><p>This curve is showing the cumulative share of customers who have made a second purchase among those who made a first purchase in Q1 or Q2 respectively. The higher your curves, the more customers are returning to buy your products, and the better your business is doing.</p><p>The catch with the way this is generally done in Excel is that you can only show the curve for as long as you have complete information. That is, if you want to show a retention curve for 52 weeks, everyone in the cohort has to have made their first purchase at least 52 weeks ago. This is sort of a bummer, because you are throwing away good information you have about the people who started at the beginning of the quarter and have made it to 52 weeks. Additionally, these curves don’t generalize well to comparing cohorts that aren’t based on time (e.g., gender) because it’s difficult to know what the correct denominator is for any given week since your cohorts didn’t all make their first purchases at the same time.</p><p>It turns out that this problem can be solved with Kaplan Meier analysis. Kaplan Meier analysis is a well-known statistical tool for understanding differences in survival rates across cohorts. Kaplan Meier is a well-studied technique and is very commonly used in <a href="http://www.ncbi.nlm.nih.gov/pubmed/?term=kaplan+meier">medical research</a> to assess the differential impact of treatment interventions on patient outcomes. As it turns out, Kaplan Meier analysis can also be used to effectively understand retention and churn among customers.</p><p>The key feature of Kaplan Meier analysis is that it is well suited to handling a phenomenon known as data ‘censoring’. Censoring occurs when you don’t have the same amount of observation time for all of the individuals in your study. In clinical trials, for example, this happens when patients ‘drop out’ of the study early: they move, or they stop being interested, or simply get sick of participating.</p><p>Imagine a year long study of 100 people with terminal cancer who are given a new treatment. The ‘event’ of interest in this study is death, so the doctors are interested in whether or not there is a difference in the survival rates of those who get the treatment vs those who received a placebo. Now imagine one of the patients receiving the treatment drops out of the study after 6 months. On the one hand, the researchers can’t assumed that the patient survived a whole year because that might over-state the impact of the treatment. On the other hand, if the researchers throw out that patient’s data they might accidentally under-estimate the impact of the treatment because that patient did survive <em>for at least 6 months</em>.</p><p>Kaplan Meier then actually takes into account two types of events (events of study and censoring events) when generating survival curves. It turns out that we can generalize this technique to our analysis of the behavior of customers by treating the current date as the censoring date for all retrospective analysis. That is, we say that customers who made their first purchase 30 days ago but haven’t repurchased yet as being censored at 30 days.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/0*3hM4tZkgK3DjfHEc.png" /></figure><p>Circles are ‘censoring events’, and Xs are ‘events of study’</p><p>Once you have your data re-aligned to use analytic time, and you’ve identified censored data appropriately, applying the Kaplan Meier computation is actually <a href="https://en.wikipedia.org/wiki/Kaplan%E2%80%93Meier_estimator">pretty straightforward</a>. The key insight of Kaplan Meier is that cumulative probability of an event having occurred by time T is the cumulative product of the marginal probabilities of the events occurring from time 0 to time T-1. That is, if you know the probability of customers returning between weeks 0 and 1, and between weeks 1 and 2, and between weeks 2 and 3, and … it’s trivial to calculate the cumulative probability of a customer having returned by week N, even though you may have a different sample of customers in each of those marginal probability calculations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/0*hMvXLFMVAMvZk6Oo.png" /></figure><p>The thing that’s a bummer about Kaplan Meier analysis is that it’s typically been limited to use by data scientists and analysts via packages in <a href="http://www.uni-kiel.de/psychologie/rexrepos/posts/survivalKM.html">R</a> or <a href="http://lifelines.readthedocs.org/en/latest/">Python</a>. This goes against Harry’s philosophy of democratizing data analysis and causes a bottleneck in the agile analytics feedback loop. I, personally, don’t want to have to write a python script any time our business analyst wants to compare retention rates between two cohorts.</p><p>Enter <a href="http://looker.com/">Looker</a>, our BI tool. Looker is a nifty interface for allowing data exploration for business users via parameterized SQL in the backend. As it turns out, the Kaplan Meier calculation can actually be written in SQL 1. We created a user-defined function in our PostgreSQL data warehouse that will calculate Kaplan Meier curves for any groupings of customers, and plugged this into the Looker backend. With a little bit 2of hacking, we were able to get the Looker interface to generate Kaplan Meier curves for arbitrary cohort definitions which analysts and business users can the explore via Looker’s drag-and-drop style GUI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/0*BR8_Tt0fFqEzMjW1.png" /></figure><p>(The lines that are lower than all of the others are holiday gift-giving cohorts)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/0*AUXycZVRe4vRphvj.png" /></figure><p>(Unsurprisingly, people who shave more tend to retain at higher levels!)</p><p>This is just a small sample of all of the possible analyses this opens up for business owners and less-technical analysts. Most importantly, it frees up our data science and analytics team to focus on harder problems while the rest of the company is able to ask and answer their own questions quickly and easily.</p><p>1: Assuming your SQL flavor can perform window functions, and you recognize that a cumulative product can be calculated for a series x using exp(log(sum(x)). You can see our implementation in <a href="https://gist.github.com/mikekaminsky/3daf5f9ab2901ec5149964bbe32e8aa8">this gist</a>. We based this off of the content of <a href="https://www.youtube.com/watch?v=dLPzx6qEIJM">this talk by Gleb Arshinov</a> and a number of other people who have written about this problem.</p><p>2: A lot. <a href="https://discourse.looker.com/t/templated-pivots-for-custom-aggregates/1304">A lot of hacking</a></p><p><em>Originally published at </em><a href="http://engineering.harrys.com/2016/04/18/democratizing-kaplan-meier.html"><em>engineering.harrys.com</em></a><em> on April 18, 2016.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a2505f593825" width="1" height="1" alt=""><hr><p><a href="https://medium.com/harrys-engineering/democratizing-kaplan-meier-harrys-engineering-a2505f593825">Democratizing Kaplan-Meier</a> was originally published in <a href="https://medium.com/harrys-engineering">Harry&#39;s Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>