<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jireh Tan on Medium]]></title>
        <description><![CDATA[Stories by Jireh Tan on Medium]]></description>
        <link>https://medium.com/@jireh?source=rss-52ce2c80b91c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*NbTQSi5Zbblp2fRQ7usojw.jpeg</url>
            <title>Stories by Jireh Tan on Medium</title>
            <link>https://medium.com/@jireh?source=rss-52ce2c80b91c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 17:49:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@jireh/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A clever use of ggplot internals!]]></title>
            <link>https://medium.com/@jireh/a-clever-use-of-ggplot-internals-bbb168133909?source=rss-52ce2c80b91c------2</link>
            <guid isPermaLink="false">https://medium.com/p/bbb168133909</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Jireh Tan]]></dc:creator>
            <pubDate>Thu, 15 Mar 2018 03:19:08 GMT</pubDate>
            <atom:updated>2018-03-15T03:20:34.393Z</atom:updated>
            <content:encoded><![CDATA[<p>One question that keeps on cropping up now that I tend to use a lot of Bayesian estimation is how to shade the area under a density curve. This is useful for</p><ul><li>visualizing a credible interval (the Bayesian version of a confidence interval)</li><li>understanding how skewed a distribution might be</li></ul><p>It would be very convenient if ggplot2::geom_density provided you with optional arguments to shade the area under the density curve. But it doesn’t.</p><p>Enter ggplot_build. It’s a ggplot internal that is used to prepare the data for plotting purposes. In particular, whenever you ggplot_build a geom_density, you’ll get list with an element called data. This element contains a list of the (x,y) coordinates used to plot the density curve.</p><pre>library(ggplot2)</pre><pre>random_numbers &lt;- data.frame(x=rbeta(1000, 41, 921))<br>build_object &lt;- ggplot_build(<br>  ggplot(data=random_numbers, aes(x=x)) + geom_density())</pre><pre># Access the x, y coordinates in the build object<br>build_object$data[[1]]$x <br>build_object$data[[1]]$y</pre><p>The good news is that you can totally use these to shade the area under the density curve: you simply have to intercept the build object:</p><pre>plot_credible_interval &lt;- function(<br>  gg_density,  # ggplot object that has geom_density<br>  bound_left,<br>  bound_right<br>) {</pre><pre>  build_object &lt;- ggplot_build(gg_density)<br>  x_dens &lt;- build_object$data[[1]]$x<br>  y_dens &lt;- build_object$data[[1]]$y<br>  <br>  index_left &lt;- min(which(x_dens &gt;= bound_left))<br>  index_right &lt;- max(which(x_dens &lt;= bound_right))<br>  <br>  gg_density + geom_area(<br>    data=data.frame(<br>      x=x_dens[index_left:index_right],<br>      y=y_dens[index_left:index_right]), <br>    aes(x=x,y=y),<br>    fill=&quot;grey&quot;,<br>    alpha=0.6)<br>}</pre><p>Then you can combine this with magrittr and it’s pretty cute!</p><pre>gg_density &lt;- ggplot(data=random_numbers, aes(x=x)) + <br>  geom_density()</pre><pre>gg_density %&gt;% plot_credible_interval(0.03, 0.05)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QAiFM0pAhv8vicE66rccTg.png" /><figcaption>voilà!</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bbb168133909" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[User Accounting: A SQL Story]]></title>
            <link>https://medium.com/@jireh/user-accounting-a-sql-story-bbf959375e1e?source=rss-52ce2c80b91c------2</link>
            <guid isPermaLink="false">https://medium.com/p/bbf959375e1e</guid>
            <category><![CDATA[growth-hacking]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[growth]]></category>
            <dc:creator><![CDATA[Jireh Tan]]></dc:creator>
            <pubDate>Sun, 11 Mar 2018 14:33:30 GMT</pubDate>
            <atom:updated>2018-03-11T14:33:30.847Z</atom:updated>
            <content:encoded><![CDATA[<p>All consumer apps must perform user accounting. Without it no VC will invest, and you should be suspicious of any VC that will throw money at you without understanding what your growth curves look like.</p><p>What do I mean by user accounting? I simply mean understanding how many users you have, and where they’re coming from. Behold the <a href="https://medium.com/swlh/diligence-at-social-capital-part-1-accounting-for-user-growth-4a8a449fddfc">usual formulae</a> (which I’ve translated for weekly active users):</p><blockquote>WAU(t) = new(t) + retained(t) + resurrected(t)</blockquote><blockquote>WAU(t-1) = retained(t) + churned(t)</blockquote><p>Some algebraic rearrangement yields:</p><blockquote>WAU(t) — WAU(t-1) = new(t) + resurrected(t) — churned(t)</blockquote><p>So the rate of change of MAU is simply equal to the number of new users and the number of resurrected users, less the number of churned users. Note that this is extremely useful, because it gives everyone just a couple numbers to rally around.</p><p>For example, if you consider the quick ratio (defined as (new + resurrected) / churned), your user base is growing if that value is greater than one, and shrinking if that value is smaller than one. This makes it super straightforward to assess the health of your consumer app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-xTa0ighhKJXRrQ-0C-mKg.png" /><figcaption>The app represented by the green curve has quick ratios that are generally above one, so its user base is growing. The app represented by the red curve has quick ratios that are generally below one, so its user ase is shrinking.</figcaption></figure><p>So far, so good. <em>Badabing. Badaboom. The rest of the proof is left to the reader.</em></p><p>In my experience, however, many startups struggle to get user accounting done because they get stuck in the implementation phase. There’s a mire of questions. How do you define a week — calendar week, or week of year? Do you calculate the weekly quick ratio on a daily basis? Do you need a separate script written in [Python/R/hottest data science language of the minute] to difference these values based on timestamp? Questions on questions. 😱 😱</p><p>Stop. Take a breath. You can do this. Let’s go through these step by step.</p><p>1: <em>daily metrics should be computed every day</em>, <em>weekly metrics every week</em>, <em>monthly metrics every month</em>. Because of overlapping intervals, it doesn’t make sense to compute weekly metrics every day, and then comparing the numbers absolutely.</p><p>You might (and indeed you should) look at MAU/WAU every day, but drilling down beyond that makes little practical sense.</p><p>2: don’t overthink it — let <em>weekly metrics be Sunday-Sunday</em>, so your metrics are available every Monday in time for the week’s action; let <em>monthly metrics be from the 1st till the end of the month</em>.</p><p>This helps to keep your time intervals non-overlapping, thereby making comparisons across time useful. For weekly metrics, any weekly interval contains intra-week variation, so I advise you to choose the one that makes <em>calendar</em> sense. Most teams have planning meetings early on in the week, so it makes a lot of sense to have last week’s numbers ready in time for Monday’s stand-up.</p><p>For most small to mid-sized startups, the weekly metrics are probably the most useful. The <strong>daily metrics are too noisy</strong> and susceptible to random deviations. The <strong>monthly metrics are too slow</strong>, and could mask interesting time-trends in the data. That’s why I’ve focused on weekly for this post.</p><p>3: You can kick these off using pure SQL, which means that you can do so quite easily with Redshift/BigQuery/HiveQL. Thus, you don’t need a fully functioning data science team™️ with 3 PhDs to do user accounting I assure you. I’ll even show you!</p><h3>User Accounting: A SQL Tutorial</h3><p>This tutorial is written in general SQL, and it can be adapted to your dialect of SQL using the appropriate functions. It assumes you have three things:</p><p>First, a <strong>user table</strong>. This table could have other things in it, but it <em>must </em>contain the <em>userid</em>, and the associated <em>signup time</em>.</p><p>(Note: Your userid could be anything your app uses to identify users: phone numbers, email addresses, generated integers, etc. As long as they uniquely identify users, you’re fine.)</p><p>Without loss of generality, we will call this table dim_users, and we will assume that it has the following schema: userid: BIGINT; signuptime TIMESTAMP.</p><p>Second, an <strong>activity table</strong>. This table could also have other things in it, but it <em>must </em>contain the userid, and the times at which a user was active.</p><p>(Note: “Activity” is a term that’s up to your team to decide. It could be something that’s super restrictive, like “signed up for a paid subscription”, or it could be something permissive, like “opened up the app”. Whatever it is, if you’ve decided it’s business-critical, you should be collecting data specifically for that purpose.)</p><p>Without loss of generality, we will call this table user_activity, and we will assume that it has the following schema: userid: BIGINT; active_time TIMESTAMP.</p><p>Third, <strong>the ability to run templatized SQL queries on a regular basis</strong> (in this case, every Monday morning). This can be whatever is most convenient: a simple CRON service you write yourself, <a href="https://airflow.apache.org/">Airflow</a>, <a href="https://github.com/spotify/luigi">Luigi</a>. Picking between any of these is an argument to have somewhere else. (Below, I assume a <a href="https://github.com/janl/mustache.js/">mustache-ish</a> style of templating.) So let’s begin!</p><h4><strong>Step 0</strong>: create your tables.</h4><p>We’ll create a table that contains the week, userid and status:</p><pre>CREATE TABLE IF NOT EXISTS user_accounting_weekly (<br>    week_ending    DATE<br>    userid         BIGINT<br>    status         VARCHAR(25)<br>    PRIMARY KEY(week, userid)<br>)</pre><p>Note that I’ve added a primary key constraint here because within each week of the year, the user has to be mapped to a <em>unique</em> state, either one of new, retained, resurrected or churned. Not all SQL dialects/implementations will have primary key constraints, but I’ve done so here just to make thing super obvious.</p><p>Now we will populate this table every Monday morning, with data from Sunday-Sunday.</p><h4><strong>Step 1</strong>: Create the job that populates the new users.</h4><p>This is the easiest query to write, since it only requires a lookup in the users table.</p><pre>INSERT INTO user_accounting_weekly (week_ending, userid)<br>SELECT<br>  DATE({YESTERDAY_SUNDAY}) AS week_ending,<br>  userid AS userid,<br>  &quot;new&quot; AS status<br>FROM<br>  dim_users<br>WHERE <br>  signuptime &gt; {PREVIOUS_SUNDAY} <br>  AND signuptime &lt;= {YESTERDAY_SUNDAY}</pre><p><strong><em>Common pitfall</em></strong>: You <em>must</em> exclude on the previous Sunday, and include on yesterday’s Sunday. That’s because we want to split the timeline into non-overlapping intervals. If you use SQL’s BETWEEN semantics, you’ll get wonky results that won’t add up, because BETWEEN is inclusive on both sides.</p><h4>Step 2a: Create the job that populates the retained users.</h4><p>To do so, you’ll need to join the activity table to itself.</p><p>We need to split the timeline up into mutually exclusive weeks. Graphically, what we’re doing is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L-FdNSIY7WDP3qUHA-oF4A.png" /><figcaption>dividing the timeline up into non-overlapping weeks</figcaption></figure><pre>INSERT INTO user_accounting_weekly (week_ending, userid)<br>SELECT<br>  DATE({YESTERDAY_SUNDAY}) AS week_ending,<br>  userid AS userid,<br>  &quot;retained&quot; AS status<br>FROM<br>  (SELECT DISTINCT userid from user_activity<br>   WHERE activity_time &gt; {TWO_SUNDAYS_AGO} <br>   AND activity_time &lt;= {PREVIOUS_SUNDAY}) active_week_before_last <br>INNER JOIN<br>  (SELECT DISTINCT userid from user_activity<br>   WHERE activity_time &gt; {PREVIOUS_SUNDAY} <br>   AND activity_time &lt;= {YESTERDAY_SUNDAY}) active_last_week<br>ON<br>  active_week_before_last.userid = active_last_week.userid</pre><p><strong><em>Common pitfall</em></strong>: your activity logs are pretty large, so you should make them DISTINCT before you join them. The result is the same, but you’ll use far fewer resources!</p><h4>Step 2b: Augment the previous query to add churned users.</h4><p>The previous query can capture churned users too, if you change the INNER JOIN to a LEFT OUTER.</p><pre>INSERT INTO user_accounting_weekly (week_ending, userid)<br>SELECT<br>  DATE({YESTERDAY_SUNDAY}) AS week_ending,<br>  active_week_before_last.userid AS userid,<br>  IF(active_last_week.userid IS NOT NULL,<br>     &quot;retained&quot; <br>     &quot;churned&quot;) AS status<br>FROM<br>  (SELECT DISTINCT userid from user_activity<br>   WHERE activity_time &gt; {TWO_SUNDAYS_AGO} <br>   AND activity_time &lt;= {PREVIOUS_SUNDAY}) active_week_before_last <br>LEFT OUTER JOIN<br>  (SELECT DISTINCT userid from user_activity<br>   WHERE activity_time &gt; {PREVIOUS_SUNDAY} <br>   AND activity_time &lt;= {YESTERDAY_SUNDAY}) active_last_week<br>ON<br>  active_week_before_last.userid = active_last_week.userid</pre><p>This might require a moment’s thought, but it’s pretty basic. If you saw someone the previous week but not last week, then last week they churned. That means they aren’t in theactive_last_week subset, so their userid after the LEFT JOIN should be NULL.</p><p>(At this point, I’m sometimes asked why I haven’t bothered to exclude the new users from this query. After all, the new users from last week <em>could</em> have been active last week, and we might be double counting them. But note that we’ve put the users from the week before last on the <em>left</em> <em>side</em> <em>of the query</em>. Thus, it cannot be the case that this query is capturing new users from last week!)</p><h4>Step 3: Add the resurrected users.</h4><p>Here I simply like to use an accounting trick. If the following is true:</p><blockquote><em>WAU(t) = new(t) + retained(t) + resurrected(t)</em></blockquote><p>Then the following is also true:</p><blockquote><em>resurrected(t) = WAU(t) — new(t) — retained(t)</em></blockquote><p>But recall that we already have new users and retained users from steps 1 &amp; 2. So this boils down to removing the new and retained users from our weekly actives:</p><pre>INSERT INTO user_accounting_weekly (week_ending, userid)<br>SELECT<br>  DATE({YESTERDAY_SUNDAY}) AS week_ending,<br>  active_week_before_last.userid AS userid,<br>  &quot;resurrected&quot; AS status<br>FROM<br>  (SELECT DISTINCT userid from user_activity<br>   WHERE activity_time &gt; {PREVIOUS_SUNDAY} <br>   AND activity_time &lt;= {YESTERDAY_SUNDAY}) all_active_last_week<br>LEFT OUTER JOIN<br>  (SELECT userid FROM user_accounting_weekly <br>   WHERE week_ending = {YESTERDAY_SUNDAY}<br>   AND status IN (&#39;new&#39;, &#39;retained&#39;)) new_and_retained_only<br>WHERE<br>  new_and_retained_only.userid IS NULL</pre><p><strong><em>Common pitfall</em></strong>: you <em>must</em> run the queries in this order, because we’re relying on the accounting identity above. So whatever system you’re using to kick off these templatized queries must be able to model the sequence as well. That is, if step 2b fails, you should stop entirely and not run step 3, until you’ve gone and fixed step 2b. Airflow and Luigi can do these as dependency graphs.</p><h4>Step 4 (optional): Run checks against the accounting identities.</h4><p>This <em>must be true</em> if you’ve done steps 1, 2 and 3 correctly:</p><pre>SELECT<br>  COUNT(DISTINCT userid) AS WAU_t<br>FROM<br>  user_accounting_weekly<br>WHERE week_ending = {YESTERDAY_SUNDAY}</pre><pre>SELECT<br>  COUNT(DISTINCT userid) AS WAU_tminus1<br>FROM<br>  user_accounting_weekly<br>WHERE week_ending = {PREVIOUS_SUNDAY}</pre><pre>SELECT<br>  status AS status,<br>  COUNT(DISTINCT userid)<br>FROM<br>  user_accounting_weekly<br>WHERE week_ending = {YESTERDAY_SUNDAY}</pre><pre># Then check that WAU_t - WAU_tminus1 = new + resurrected - churned<br># And check that WAU_tminus1 = retained + churned</pre><p><strong><em>Common pitfall</em></strong>: Not actually running any checks. The query flow is pretty finicky, so it’s prone to mistakes. Double-checking never hurt anyone. Bonus points if you actually run <a href="https://github.com/apache/incubator-airflow/blob/master/airflow/operators/check_operator.py">automated checks</a>.</p><h3>Closing words and caveats</h3><p>Here are some common problems I’ve seen.</p><ul><li><strong>No activity logs</strong>. The user table usually comes first, since that’s crucial application data. But everyone will log whether a user is active or not. If you don’t already have a dedicated activity log table, <em>consider looking at the full dump of your requests</em> to determine if a user was active or not.</li><li><strong>Activity table and user tables have different types of userids</strong>. For example, the activity table uses phone numbers to identify users, but the users table uses userid. In this case, <em>simply disambiguate the users by mapping their phone numbers to their userids</em>. If each user can be mapped onto several phone numbers, <strong>be careful</strong>! You need to make sure that you preserve the uniqueness of users.</li><li><strong>Want to do user accounting by country</strong>? Be careful, because usually the country produced by activity logs is linked to IP, and those can be notoriously unhelpful. You shouldn’t group by those, because a person could jump from country to country within the same week. Instead, you should build intermediary tables, where you augment the user table by choosing a country for the user that makes the most sense. (A sensible heuristic you might use is “the country that the user makes the most requests from”.) Whatever it is, you determine the user’s country by that <em>unique</em> mapping in the augmented user table. This avoid the problem of overlapping users that make user accounting numbers uninterpretable.</li><li><strong>Keep shining, keep optimizing</strong>. Once your app gets to a certain size, running DISTINCT queries over and over again probably doesn’t make that much sense. You should consider running another job that DISTINCT-s the weekly active users first, and then you could just refer to that table instead!</li></ul><p><em>I am a data scientist with a lot of experience teaching SQL. I was once </em><a href="https://www.facebook.com/518332116/posts/10152601675007117/?match=aGl2ZQ%3D%3D"><em>declared awesome at HiveQL</em></a><em>. In previous lives I did growth accounting for Flipboard and Facebook Groups. If this post was helpful to you, please clap or </em><a href="http://www.jireh.io/"><em>get in touch</em></a><em>!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bbf959375e1e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Proportion Estimation and its Discontents]]></title>
            <link>https://medium.com/@jireh/proportion-estimation-and-its-discontents-c88828a8ba43?source=rss-52ce2c80b91c------2</link>
            <guid isPermaLink="false">https://medium.com/p/c88828a8ba43</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statistical-significance]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[statistical-analysis]]></category>
            <dc:creator><![CDATA[Jireh Tan]]></dc:creator>
            <pubDate>Tue, 06 Feb 2018 23:14:27 GMT</pubDate>
            <atom:updated>2018-02-06T23:14:27.282Z</atom:updated>
            <content:encoded><![CDATA[<p>One of the first things we learn in Stats 101 is how to estimate proportions using p-values and confidence intervals. A toy example is usually offered, with the TA taking a conspiratorial — perhaps even slightly apologetic — tone: <em>imagine we’re tossing a coin, and we want to know whether it’s fair </em>(p=0.5).</p><p>We then go through some rote. The TA’s eyes may gloss over as he starts chanting, much like a priest performing the last rites 😵</p><ol><li><em>Toss the coin many times, noting the number of successes and failures</em></li><li><em>With these quantities, use the </em><a href="http://www.stat.yale.edu/Courses/1997-98/101/binom.htm"><em>normal approximation to the binomial</em></a></li><li><em>(mumble something about n and p)</em></li><li><em>Look up the values in a z-table to get confidence intervals and p-values</em></li></ol><p>And we’re done! (If you’re fancy you might apply some <a href="https://en.wikipedia.org/wiki/Continuity_correction">continuity corrections</a> for “small” <em>n, </em>but basically you’re Stats-101-done.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oO6fTC6ScrDnvx4nY3Zd8A.jpeg" /><figcaption>Do you approve of my frown turned upside down, turned upside down?</figcaption></figure><p>Because I enjoy being extra petty, here’s an example. Imagine you’re interested in Donald Trump’s popularity, so you call up 2000 people and ask them if they approve of his presidency. Only 378 people say that they approve. So what’s the approval rate? You apply the TA’s formula. A flurry of <em>p</em>-hats and <em>n</em>s later, you get a mean of <strong>0.189</strong> and a confidence interval of (<strong>0.1718, 0.2062</strong>). Big!</p><p>Armed with this knowledge, you go out into industry and start estimating proportions. In particular, instead of moving to Pew Research, or Gallup, you decide on going to a startup, and you’re tasked with estimating clickthrough rates on a given button on a website. <em>How droll</em>. You get these numbers: 19902 people visited the website. Only 2 people clicked on the button. You apply the formula, and you get a mean of <strong>0.0001004924</strong>, and a confidence interval of (-<strong>0.000038773408, 0.0002397582</strong>). What? Isn’t a proportion supposed to lie between zero and one? Sad!</p><p>What’s going on here? To understand why this is the case, we have to take a step back and look at point 3, the TA’s mumblings about <em>n</em> and <em>p.</em> But before we do that, let’s take a step back and think about what a confidence interval is.</p><h3>Confidence Intervals: The Recap No One Wants to Read 😢</h3><p>Let’s first consider what a confidence interval is. The textbook definition (according to <a href="https://en.wikipedia.org/wiki/Confidence_interval">Wikipedia</a> as of January 2018) is</p><blockquote>is a type of <a href="https://en.wikipedia.org/wiki/Interval_estimation">interval estimate</a> (of a <a href="https://en.wikipedia.org/wiki/Population_parameter">population parameter</a>) that is computed from the observed data. The <strong>confidence level</strong> is the frequency (i.e., the proportion) of possible confidence intervals that contain the true value of their corresponding parameter. In other words, if confidence intervals are constructed using a given confidence level in an infinite number of independent experiments, the proportion of those intervals that contain the true value of the parameter will match the confidence level.</blockquote><p>Thanks, Wikipedia, super helpful. Not opaque <em>in the slightest</em>! 🙄</p><p>What Wikipedia means to say is as follows. Let’s say we’re interested in some unknown quantity, call it <em>X.</em> So you collect <em>data</em> regarding <em>X. </em>Imagine, then, that you have a <em>procedure</em> that, given any set of data, generates an interval on the real line. But you could always have gotten a different set of data! (Going back to our Donald Trump example, you could have phoned a different set of 2000 people.) If you were to repeat your data collection exercise a large number of times and apply the same procedure, you’d get a different interval each time. As long these intervals resulting from the different data cover the true quantity <em>X</em> at least θ% of the time, these intervals may be called θ% confidence intervals.</p><p>The issue is that any given procedure has its benefits and drawbacks. Let’s take a simple example. Say we’re interested in a 95% confidence interval for Trump’s approval rating. What’s a procedure that that would generate intervals covering the true approval rating at least 95% of the time? Here’s one: <strong>regardless of the data you get, return the interval [0,1]</strong>. Since the true approval rating must lie between zero and 1 inclusive, you’re definitely going to cover the true mean 100% of the time.</p><p>What’s the issue with this procedure? It’s… not very informative. 😑 But it is a confidence interval!</p><p>(I like giving this example of the [0,1] interval because it defamiliarizes the concept of a confidence interval, which is a pretty counterintuitive idea that people <em>think</em> that they understand, but they don’t really.)</p><h3>Confidence Intervals Are Not God Given: Assess Them!</h3><p>We’ve implicitly one of the most important aspects of confidence intervals: <em>coverage</em>. That is, if you advertise a 95% confidence interval, <strong>it had <em>better</em> cover the true mean about 95% of the time</strong>.</p><p>Any more, and you could probably stand to gain by making the intervals <em>narrower. </em>Remember, the widest interval for estimating a proportion is [0,1], but that’s not interesting, because it’s so wide. You should figure out a way to make the intervals smaller, so that you have a better idea of where to believe the proportion is.</p><p>Any less, and you fail the definition of confidence interval, and you are more likely to make a false positive, because your intervals are <em>too narrow.</em> You need to figure out why your confidence intervals aren’t doing as they’re supposed to.</p><p>Unfortunately, it’s hard to assess true coverage, unless you’re running simulations. (<a href="https://www.optimizely.com/optimization-glossary/aa-testing/">A/A tests</a>, for example, are one way to assess coverage.) But here’s one thing I’m telling you now: if you are assessing a proportion, <em>p</em>, where <em>p </em>is super tiny, like a baseline rate for a conversion, or sudden cardiac death in healthy adults, <em>n</em> has to be <em>extremely large</em> in order for the confidence interval produced by the normal approximation to have an appropriate coverage. That’s because the underlying variance is <em>so large</em> relative to <em>p </em>that you need to accumulate a lot of information before you have the advertised coverage.</p><p>(Intuitively speaking, if an interval places any confidence at all that a proportion <em>could be negative</em>, then that interval can be improved by figuring out strategies to shift that confidence into the [0, 1].)</p><h3>Further Readings and More Suggestions</h3><p>I find the work of <a href="http://www-stat.wharton.upenn.edu/~tcai/">Tony Cai</a> at Penn really useful at concretizing understanding of confidence intervals for proportions, and there’s a really cool <a href="https://repository.upenn.edu/cgi/viewcontent.cgi?article=1440&amp;context=statistics_papers">discussion</a> on coverage and best practices for <em>p</em>.</p><p>When I find the time, I’ll discuss the Bayesian approach to estimating <em>p</em>, which turns out to have a more intuitive understanding for the statistical dabbler: <a href="https://en.wikipedia.org/wiki/Credible_interval">credible intervals</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c88828a8ba43" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Revenue Analysis: A Bayesian Approach]]></title>
            <link>https://medium.com/@jireh/revenue-analysis-a-bayesian-approach-867bbd2c979c?source=rss-52ce2c80b91c------2</link>
            <guid isPermaLink="false">https://medium.com/p/867bbd2c979c</guid>
            <category><![CDATA[bayesian-statistics]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Jireh Tan]]></dc:creator>
            <pubDate>Fri, 05 Jan 2018 22:24:41 GMT</pubDate>
            <atom:updated>2018-01-06T19:09:54.075Z</atom:updated>
            <content:encoded><![CDATA[<p>Recently, one of my client companies, <a href="http://love.hornetapp.com/">Hornet</a>, posed me an interesting question that gave me pause. They were testing a couple of revenue strategies and were wondering which one they should pick. I can’t go into specifics, but this is actually a pretty common problem in behavioral economics.</p><p>Say you’re selling apples, and say you’re exploring different pricing options for your customers. You give them the choice of buying 1 apple, 5 apples, or 10 apples at once. Obviously, the unit price of the apple has to go down if you buy in bulk — that’s why we love Costco!</p><p>But <em>how much</em> should each option be priced? The problem doesn’t reduce simply to <a href="https://en.wikipedia.org/wiki/Price_elasticity_of_demand#Optimal_pricing">drawing demand elasticity curves</a>. Pricing psychology has shown, <a href="https://en.wikipedia.org/wiki/Anchoring">again</a> <a href="https://en.wikipedia.org/wiki/Framing_effect_(psychology)">and</a> <a href="https://en.wikipedia.org/wiki/Mental_accounting">again</a>, that humans are notoriously bad at being rational agents when it comes to picking between prices. So a relatively high price for a single apple could drive people to pick the 10-apple option. Conversely, a relatively high unit price for 10 apples could drive people to pick the 1-apple option.</p><p>Add to this the fact that you’re testing between several strategies, and you’ve got yourself a quandary. Standard hypothesis testing fails for several reasons. Standard hypothesis testing isn’t great at estimating binomial or multinomial proportions, where the baseline non-conversion rate is pretty high (remember, there’s an implicit 4th option: the customer could walk away without buying any apples, and this might actually be the most common scenario!). Furthermore, you’re picking between many pricing strategies, and thus the problem of <a href="https://en.wikipedia.org/wiki/Multiple_comparisons_problem">multiple comparisons</a> rears its ugly head. So how do we proceed?</p><h3>Setting Up the Problem</h3><p>Let’s begin with a hypothetical dataset. You first decide on your pricing strategies. For each strategy, you give the customer 3 choices, which result in the following revenue (represented in cents):</p><pre>┌─────────┬──────────────────────┬─────────┬──────────┬───────────┐<br>│         │       Strategy       │ 1 Apple │ 5 Apples │ 10 Apples │<br>├─────────┼──────────────────────┼─────────┼──────────┼───────────┤<br>│ Store A │ High-High-Low        │       6 │       25 │        20 │<br>│ Store B │ Medium-Medium-Medium │       4 │       20 │        40 │<br>│ Store C │ Low-Low-Low          │       3 │     12.5 │        20 │<br>└─────────┴──────────────────────┴─────────┴──────────┴───────────┘</pre><p>You then sell apples at different stores for a couple days, and you see that the customers choose the options accordingly:</p><pre>┌─────────┬─────────┬──────────┬───────────┬─────────────────┐<br>│         │ 1 Apple │ 5 Apples │ 10 Apples │ Total Customers │<br>├─────────┼─────────┼──────────┼───────────┼─────────────────┤<br>│ Store A │      15 │        5 │         1 │           74    │<br>│ Store B │      10 │       10 │         0 │           69    │<br>│ Store C │       8 │        7 │         6 │           65    │<br>└─────────┴─────────┴──────────┴───────────┴─────────────────┘</pre><p>Doing some simple multiplication, we get the following revenue:</p><ul><li><strong>High-high-low</strong>: 235 cents</li><li><strong>Medium-medium-medium</strong>: 240 cents</li><li><strong>Low-low-low</strong>: 231.5 cents</li></ul><p>But does that mean that we should opt for the one with the highest revenue in this given mini-experiment?</p><p>Categorically, <strong>no</strong>. Note, first of all, that these revenue points ignore the total number of customers that came in — so you aren’t taking into account the number of customers that turned away because they found the apples too expensive. More importantly, these points don’t provide bounds for uncertainty. So even if you pick the MMM option, you don’t know, in the long run, if it’s the winning strategy, and why, and by how much.</p><h3>Framing the Problem Probabilistically</h3><p>As I alluded to earlier, a statistician might look at this problem and see that the problem reduces to estimating the parameters of a <a href="https://en.wikipedia.org/wiki/Multinomial_distribution">multinomial distribution</a>. The multinomial distribution is a generalization of the <a href="https://en.wikipedia.org/wiki/Binomial_distribution">binomial distribution</a>, which is what you get when you try to answer the question “what’s the probability of getting 5 heads if you flip a fair coin 12 times?”</p><p>In this case, we’re trying to answer the question: if <em>N</em> customers come into the store, what’s the probability of these customers selecting the 1-apple, 5-apple, 10-apple or no-apple options respectively? With these probabilities, we are then able to calculate the expected revenue.</p><p>I’ve found that the easiest way to estimate these multinomial probabilities is to adopt a Bayesian approach and to sample directly from a <a href="https://en.wikipedia.org/wiki/Dirichlet_distribution">Dirichlet distribution</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/350/1*OofKz8cZAhfLG67USZX24w.jpeg" /><figcaption>Forrest Gump sampling from a distribution of distributions</figcaption></figure><p>The Dirichlet distribution is a distribution of distributions. This sounds a little opaque, but a good analogy is to think about chocolates. Imagine that there are 100 boxes of chocolates, each with a different proportion of white, milk and dark chocolate. Then each box of chocolates has a distribution of chocolates. So when you’re picking your box of chocolates, you’re actually picking from a distribution of distributions. Though you never know what you’re gonna get, a Bayesian might insist that you can place bounds on your uncertainty.</p><p>That’s where the Dirichlet distribution comes in. Given your data, it allows you to generate plausible probability distributions that generated the data.</p><h3>The Dirichlet Distribution</h3><p>Let’s first zoom in on the customers going to store A, which adopted the <strong>High-High-Low</strong> strategy. Recall that 15 customers chose the 1-apple option, 5 customers chose the 5-apple option, 1 customer chose the 10-apple option, and 53 customers didn’t buy any apples.</p><p>Then consider this distribution: <em>Dirichlet</em>(15, 5, 1, 53). Let’s sample from this distribution:</p><pre>+-----------+------------+--------------+-----------+<br>|  1 apple  |  5 apples  |  10 apples   | No apples |<br>+-----------+------------+--------------+-----------+<br>| 0.1856628 | 0.05850445 | 0.0001780732 | 0.7556546 |<br>| 0.1499370 | 0.04898182 | 0.0058574549 | 0.7952237 |<br>| 0.2159189 | 0.10980787 | 0.0094055983 | 0.6648676 |<br>| 0.1575091 | 0.02458244 | 0.0037588271 | 0.8141497 |<br>| 0.2843948 | 0.02787938 | 0.0103459431 | 0.6773799 |<br>| 0.2014919 | 0.11635238 | 0.0132865587 | 0.6688691 |<br>+-----------+------------+--------------+-----------+</pre><p>Note that <strong>each row sums to 1</strong>. The interpretation of this: when you sample from <em>Dirichlet</em>(15, 5, 1, 53), you get probability distributions that could plausibly result in your 60 customers partitioned into 4 buckets, with each bucket having 15, 5, 1 and 53 customers, that correspond to them choosing the 1-, 5-, 10- and no-apple option respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qGY5Mn_K2tzKK7B-p_NRcg.png" /></figure><p>Why is this helpful? Because it allows us to generate a <strong>distribution over the expected revenues</strong>! Recall that the revenue vector for this strategy is (6, 25, 20, 0). Then we can simply perform simple matrix multiplication to get a sequence of expected revenues. The graph to the left was generated by taking 30,000 samples from <em>Dirichlet</em>(15, 5, 1, 53), multiplying it by the revenue vector, and then summarizing it using a density plot.</p><p>We can repeat this for all of the strategies that we tried, namely:</p><ul><li><strong>High-High-Low: </strong><em>Dirichlet</em>(15, 5, 1, 53)</li><li><strong>Medium-Medium-Medium: </strong><em>Dirichlet</em>(10, 10, 0, 49)</li><li><strong>Low-Low-Low: </strong><em>Dirichlet</em>(8, 7, 6, 44)</li></ul><p>This results in the following expected revenue densities:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i9tHPGoxLqUM0N4tCYsOmA.png" /></figure><p>How do we know which strategy is the best? We can simply break each of the 30,000 samples down into how frequently they result in the highest expected revenue.</p><pre>+---------------+----------------------+-------------+<br>| high-high-low | medium-medium-medium | low-low-low |<br>+---------------+----------------------+-------------+<br>|          6519 |                11036 |       12445 |<br>+---------------+----------------------+-------------+</pre><p>This implies that, in the long run, low-low-low produces the highest expected revenue 41.5% of the time, followed by medium-medium-medium at 36.8%, and in last place is high-high-low at 21.7%. So all other things being equal, you should choose the <strong>low-low-low</strong> option!</p><h3>Summary</h3><p>In this post, I showed you how you can use the Dirichlet distribution to estimate multinomial probabilities to calculate expected revenue:</p><ol><li>Collect data for each strategy, <strong>breaking each strategy down into the number of times</strong> people chose each option</li><li>Use the number of times people chose each option to <strong>draw a large number of samples from a Dirichlet distribution</strong> (say 30,000)</li><li>Use these samples to <strong>calculate expected revenue </strong>by multiplying by the relevant revenue vector</li><li><strong>Compare these samples to each other</strong> to see which strategy results in the highest revenue most of the time</li></ol><p>In future posts, I’ll elaborate on how you can use prior information (for example, historical conversion rates) in order to pool your estimates.</p><p><em>I am a data scientist interested in Bayesian approaches to statistical problems. If these problems are interesting to you, </em><a href="http://www.jireh.io/"><em>get in touch</em></a><em>!</em></p><p><em>Hornet is a gay social network where the gay community can explore the world together with pride and confidence. Founded in 2011 with the mission to build the seminal digital home for the gay community, Hornet has grown to 25 million diverse community members strong. If you’re interested in </em><a href="https://hornet.recruiterbox.com/"><em>working with us</em></a><em>, drop me a line!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=867bbd2c979c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>