<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Preeti Semwal on Medium]]></title>
        <description><![CDATA[Stories by Preeti Semwal on Medium]]></description>
        <link>https://medium.com/@preeti-semwal?source=rss-62e228735f49------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*N0josgWN3_H3Wt_g</url>
            <title>Stories by Preeti Semwal on Medium</title>
            <link>https://medium.com/@preeti-semwal?source=rss-62e228735f49------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 09:29:16 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@preeti-semwal/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[25 A/B Testing Concepts — Interview Cheat Sheet]]></title>
            <link>https://medium.com/data-science/25-a-b-testing-concepts-interview-cheat-sheet-c998a501f911?source=rss-62e228735f49------2</link>
            <guid isPermaLink="false">https://medium.com/p/c998a501f911</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[interview-questions]]></category>
            <category><![CDATA[hypothesis-testing]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[statistical-significance]]></category>
            <dc:creator><![CDATA[Preeti Semwal]]></dc:creator>
            <pubDate>Mon, 14 Nov 2022 19:45:04 GMT</pubDate>
            <atom:updated>2022-11-15T01:06:01.651Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>25 A/B Testing Concepts You Must Know: Interview Refresher</strong></h3><h4><strong>All-in-one cheat sheet to ace interviews</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uB5i1y5Z_qdVqNWNTjKH9Q.png" /><figcaption>Photo by Dan Cristian Pădureț on Unsplash</figcaption></figure><p>Questions on A/B testing are being increasingly asked in interviews but reliable resources to prepare for these are still far and few. Let’s say you completed a <a href="https://www.udemy.com/course/product-experimentation-ab-testing-in-r-with-real-examples/learn/practice/1238666?referralCode=C0EC84F644B6BCB10FD0#overview">great course on A/B testing </a>sometime back and felt confident in your understanding of A/B testing statistics. It’s been a while now, you have an interview coming up and for the love of god, you can’t seem to remember what statistical power means or what SUTVA stands for. OR you have been using experimentation in your current role and have automated most processes. Without the need for manual steps, you have become rusty and need a quick cheat sheet to recall important concepts and intuition behind them so that you can ace an upcoming interview.</p><p>Use this post as a quick resource for the most important concepts in A/B testing you need to know before interviews. You will find a summary of the most important concepts along with examples to build your intuition.</p><h4><strong>Table of contents (click below to skip to specific topics)</strong></h4><ul><li><a href="#a7ad">The Very Basics</a> 📚<br><em>Population, sample, sample mean, sample variability</em></li><li><a href="#4746">Experiment Design</a> 🎭<br><em>Null Hypothesis, Key Metrics, Overall Evaluation Criteria (OEC), Guardrail Metrics, Randomization Unit, Interference</em></li><li><a href="#281f">A/B Test Statistics &amp; Sample Size Calculation</a> 🧮<br><em>Confidence Level, Margin of error, Confidence Interval, Type 1 Error, Type 2 Error, p-value, Statistical Significance, Statistical Power, Minimum Detectable Effect, Practical Significance, Sample Size &amp; Duration</em></li><li><a href="#aec6">Threats to Experiment Validity</a> 🚧<br><em>Novelty Effect, Primacy Effect, Seasonality, Day of Week Effect</em></li></ul><p>Before we begin, let’s establish an example of an A/B test we will utilize as we go over the concepts. Lets say you are running an A/B test on a web page with the goal to improve click-through rates (CTRs) — your original web page, which is the control offers a $25 saving using the text ‘Save $25’; through this A/B test you are going to test a variation of this web page that presents the same offer (value is still $25) in a different way using the text ‘Save 15%’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/1*Ziq-YsCfqs4lNU1H6-63nQ.png" /><figcaption>Image by Author</figcaption></figure><p>Now onto the concepts…</p><h3>The Very Basics</h3><ol><li><strong>Population</strong> — It is the entire group that you want to draw conclusions about. In statistics, a population is a set of similar items or events which is of interest for some question or experiment. <em>In our example above, the true population is every future individual who will visit the web page.</em></li><li><strong>Sample</strong> — A sample is a small portion of a population that is representative of the characteristics of the larger population. With statistical analysis, you can use data collected from samples to make estimates or test hypotheses about population. <em>In A/B testing, a sample is the randomly selected set of visitors we display each of our page variations to — control is exposed to one sample and treatment is exposed to another.</em></li><li><strong>Sample Mean</strong> — For a given metric, this is the mean or average based on data collected for the sample. <em>For our A/B test example that is aiming to optimize CTR (click-through rate), this is nothing but the average CTR for users in each sample.</em></li><li><strong>Sample Variability</strong> — Two randomly selected samples from a population may be different from each other. Whatever conclusions we are drawing about the population through a sample, because of sample variability, there is likely to be error in our estimations. Sampling variability will decrease as sample size increases. <em>In A/B testing, the sampling variability affects the sample size we need in order to have a chance of deriving statistically significant results.</em></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fz5VUGZH6NQrOer3tCaCww.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bbWBfsDjcF-VdU752vsiew.png" /><figcaption>Image by Author</figcaption></figure><h3>Experiment Design</h3><p><strong>5. Null Hypothesis</strong> — Inferential statistics is based on the premise that you cannot prove something to be true but you can disprove something by finding an exception. You decide what you are trying to provide evidence for — which is the alternate hypothesis, then you set up the opposite as the null hypothesis and find evidence to disprove that. <em>In our A/B test example, the null hypothesis is that the population CTR on the original page and the page variation are not different.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xJB94vyySjMIsBksdxAl7w.png" /><figcaption>Image by Author</figcaption></figure><p><strong>6. Key Metric/s</strong> — A set of metrics that you are trying to optimize through the experiment. <em>Some commonly used metrics are click through rates (CTR), sign up rate, engagement rate, average revenue per order, retention rates etc</em>. As you can imagine key metrics are going to relate to the priorities of the business, OKRs and goals. Many organizations examine multiple key metrics, and have a mental model of the trade offs they are willing to accept when they see a particular combination. <em>For e.g., they may have a good idea about how much they are willing to lose users (increase in churn) if the remaining users increase their engagement and revenue.</em> This brings us to OEC explained below.</p><p><strong>7. Overall Evaluation Criteria (OEC) </strong>— When there are multiple metrics to be optimized through the experiment, it is helpful to formulate trade-offs by devising a single metric called an Overall Evaluation Criteria (OEC) — which is essentially a weighted combination of such objectives. <em>One way to do this is to normalize each metric to a predefined range, say 0–1, and assign each a weight. Your OEC then is the weighted some of the normalized metrics. In the example above with needing to evaluate trade-off between churn and revenue, LTV can be used as an OEC.</em></p><p><strong>8. Guardrail Metrics</strong> — These are metrics that are important for the company and should not be negatively impacted by the experiment. <em>For e.g. our goal may be to get as many users as possible to register, but we don’t want the per-user engagement level to drop drastically. Or we may want to increase app engagement but at the same time ensure that app uninstalls do not increase.</em></p><p><strong>9. Randomization Unit</strong> — This is the unit e.g. users or pages that a randomization process is applied to map them to either control or treatment. Proper randomization is important to ensure that populations assigned to the different variants are similar statistically. Randomization unit should be chosen such that Stable unit treatment value assumptions (SUTVA) are satisfied. SUTVA states that experiment units do not interfere with one another i.e. the behavior of units in test and control is independent of each other. User-level randomization is the most common as it avoids inconsistent experience for the user and allows for long-term measurement such as user retention.</p><p><strong>10. Interference</strong> — Sometimes also called spillover or leakage occurs when the behavior of the control group is influenced by the treatment given to the test group. This leads to a violation of SUTVA assumption which results in potentially incorrect conclusions. There are two ways inference may arise —</p><ul><li>Direct — two units can be directly connected if they are friends on a social network or if they visited the same physical space at the same time. If one of these is assigned to Treatment and other to control, this will cause interference between variants</li><li>Indirect — indirect connections are connections that exist because of certain shared resources. For e.g. If the Airbnb marketplace improved conversion flow for treatment users, resulting in more bookings, it would naturally lead to less inventory for Control users. Similarly marketplaces such as Lyft/Uber/Doordash where users in control and treatment might share the same pool of drivers/dashers will also face interference</li></ul><h3>Sample size calculation</h3><p>11.<strong> Confidence Level </strong>— Confidence level refers to the percentage or probability or certainty, that the confidence interval would contain the true population parameter when you draw a random sample many times. In the tech world of online A/B Testing, a 95% confidence level is chosen most often but you can choose different levels depending on the situation. <em>A 95% confidence level means that the confidence interval around sample mean is expected to include the true mean value 95% of the time.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8NVfLkzhDS-EEkjFgGTn7w.png" /></figure><p>12. <strong>Margin of error</strong> — As we noted earlier, due to sampling variability, it is possible that the conclusions your draw about the population based on samples is inaccurate. A margin of error tells you how many percentage points your results will differ from the real population value. <em>For example, a 95% confidence interval with a 4 percent margin of error means that your statistic will be within 4 percentage points of the real population value 95% of the time. The margin of error is added to and subtracted from the mean to determine the confidence interval (discussed below).</em></p><p>13. <strong>Confidence Interval</strong> — In statistical inference, we aim to estimate population parameters using observed sample data. A confidence interval gives an estimated range of values which is likely to include an unknown population parameter, the estimated range being calculated from a given set of sample data. <em>The width of confidence interval depends on 3 things — the variation within the population of interest, the size of the sample and the confidence level we are seeking.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UhVzAcRSIXdLoRbcw5D16g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2d10kxFTdpQscCX7wVhGKQ.png" /><figcaption>Image by Author</figcaption></figure><p>14. <strong>Type 1 Error</strong> — A type I error occurs when we incorrectly reject the null hypothesis.<em> In our A/B test example, a type I error would occur if we concluded that population mean of treatment is different from population mean of control when in reality they were the same. </em>Type I error is avoided by achieving statistically significant results.</p><p>15. <strong>Type 2 Error </strong>— A type II error occurs when the null hypothesis is false, but we incorrectly fail to reject it. <em>In our A/B test example, a type II error would occur if we concluded that the population mean of Variation B is not different than the mean of Variation A when it actually was different.</em> These errors are avoided by running tests with a high statistical power.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3UaBmW9YQ9GGNyYU7COLQw.png" /><figcaption>Image by Author</figcaption></figure><p>16. <strong>P-value</strong> — p-value is the probability of obtaining at least as extreme results as we are seeing if the null hypothesis of the test is true. p-value basically tells you whether your evidence makes your null hypothesis look ridiculous.</p><p>17. <strong>Statistical Significance </strong>— Statistical significance is attained when the p-value is less than the significance level. The significance level (𝛂), is the threshold you want to use for the probability of making Type 1 error i.e. concluding that population mean of Control and Treatment are different when in fact they are the same. In other words, statistical significance is another way of saying that the p-value of a statistical test is small enough to reject the null hypothesis. The scientific standard is to use a p-value &lt; 0.05 i.e. 𝛂 = 5%.</p><p>18. <strong>Statistical Power </strong>— Statistical Power, which as we know is the probability that a test correctly rejects the null hypothesis i.e. the percentage of time the minimal effect will be detected, if it exists.</p><p>19. <strong>Minimum Detectable Effect (MDE) </strong>— is the smallest change in conversion rate you are interested in detecting. In the example where you are optimizing for CTR, let’s say the CTR for the control is 20%. And the smallest change you would like to detect at the minimum is 5% absolute lift relative to control i.e. if the CTR for the treatment is 25% or more. In this case 5% is the MDE.</p><p>20. <strong>Practically significant — </strong>It’s possible for hypothesis tests to produce results that are statistically significant, despite having a small effect size. This usually happens due to 2 reasons — 1) Low sampling variance and 2) Large sample size. In both these cases, we may be able to detect even small differences between test and control with statistical significance. However, these may not be significant in the real world. <em>Let’s take an example of an A/B test which shows a new module to users and is able to detect a difference of .05% in the CTR — however, is that cost of building that module justified for such a small lift. What is the practically significant lift that would be feasible in this case.</em></p><p>21. <strong>Sample size </strong>— The number of units per variation needed to reach statistical significance, given the baseline conversion, minimum detectable difference (MDE), significance level &amp; statistical power chosen. ‘Duration’ refers to how long you need to run the test to reach adequate sample size per variation.</p><h3>Threats to experiment validity</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T3s0vHQYOLl0fpLCHCWo7A.png" /><figcaption>Image by Author</figcaption></figure><p>22. <strong>Novelty Effect </strong>— “Sometimes there’s a “novelty effect” at work. Any change you make to your website will cause your existing user base to pay more attention. Changing that big call-to-action button on your site from green to orange will make returning visitors more likely to see it, if only because they had tuned it out previously. This type of effect is not likely to last in the long run — but it can artificially impact your test results.</p><p>23.<strong> Primacy effect </strong>— When there’s a change in the product, people react to it differently. Some users may be used to the way a product works and are reluctant to change. This is called primacy effect. The primacy effect is nothing but the tendency to remember the first piece of information we encounter better than information presented later on. This can be thought of as an <strong><em>opposite phenomenon to novelty effect</em></strong>.</p><p>24. <strong>Seasonality</strong> — Businesses may have different user behavior say on 1st of a month and 15th of a month. For some eCommerce sites, their traffic and sales are not stable all over the year, they tend to peak on Black Friday and Cyber Mondays for example. Variability due to these factors could influence your test results.</p><p>25. <strong>Day of week effect </strong>— Similar to seasonality, a metric may have cyclicity based on day of week. For e.g. say conversion rates are much higher on Thursdays than they are on the weekends. In this case, it is important to run tests for full-week increments, so you are including every day of the week.</p><h3>Conclusion</h3><p>A/B Testing is one of the most important and widely applied data science concepts with numerous applications in growth optimization — be it product experimentation <em>(optimizing onboarding flows, increasing CTRs, server side optimizations etc.) </em>OR for marketing acquisition <em>(creative testing, incrementality testing, geo-split testing) </em>and many more<em>. </em>With so many potential applications, it is very likely that you will be gauged on your knowledge of A/B testing in interviews — specifically for product data science/analytics or marketing data science/analytics roles.</p><p>If you would like to continue learning A/B testing concepts and applications, check out <a href="https://www.udemy.com/course/product-experimentation-ab-testing-in-r-with-real-examples/?referralCode=C0EC84F644B6BCB10FD0">Complete Course on A/B Testing with Interview Guide</a>.</p><h3>Thanks for reading!</h3><p><em>Paradoxically, acing data science interviews is more an art than a science! For more content from me on data science interviews or career growth in data science, </em><a href="https://preeti-semwal.medium.com/subscribe"><em>subscribe and follow</em></a><em>!</em></p><p><em>Follow on </em><a href="https://www.linkedin.com/in/preeti-semwal/"><em>LinkedIn!</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c998a501f911" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/25-a-b-testing-concepts-interview-cheat-sheet-c998a501f911">25 A/B Testing Concepts — Interview Cheat Sheet</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Top 3 Tools to Promote Your Work in Analytics & Data Science]]></title>
            <link>https://medium.com/data-science/top-3-tools-to-promote-your-work-in-analytics-data-science-7b87d80615fc?source=rss-62e228735f49------2</link>
            <guid isPermaLink="false">https://medium.com/p/7b87d80615fc</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[promotion]]></category>
            <category><![CDATA[communication]]></category>
            <category><![CDATA[office-hours]]></category>
            <category><![CDATA[career-advice]]></category>
            <dc:creator><![CDATA[Preeti Semwal]]></dc:creator>
            <pubDate>Tue, 25 Oct 2022 04:40:43 GMT</pubDate>
            <atom:updated>2022-11-09T14:40:06.393Z</atom:updated>
            <content:encoded><![CDATA[<h3>Top 3 Tools to Promote Your Work in Analytics and Data Science</h3><h4>Get the credit you deserve</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5m0ua46tbDN3yFEI" /><figcaption>Photo by <a href="https://unsplash.com/es/@patrickian4?utm_source=medium&amp;utm_medium=referral">Patrick Fore</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Introduction</h3><blockquote>Are you tired of Analytics &amp; Data Science feeling like a support function with little to no recognition of the value your team drives?</blockquote><blockquote>Do you constantly feel under-appreciated for all the work you and your team puts in?</blockquote><blockquote>Do you want to learn tools that help advocate and evangelize your team’s contributions?</blockquote><p>In this blog post we will discuss the 3 most effective ways to get you and your teams the recognition they deserve.</p><h3>Promoting your work</h3><p>The days of ‘the work will speak for itself’ are long gone. Being able to communicate the value you are driving is important in any job but it is <em>crucial</em> in Analytics because in many cases you are enabling other functions to do their job effectively. For e.g. you partnered with Product Manager on initial strategy for a product feature launch. After 3 months the feature was launched and while you had contributed heavily during the ideation phase, with lesser interaction closer to launch, the PM misses mentioning you when doling out credit for the launch. This is just one scenario, but there are several such scenarios that add up and lead to missed opportunities for recognition.</p><blockquote>Being able to <em>promote your work</em> is extremely important in Analytics — without this skill, it is easy for the impact you are driving to go unnoticed.</blockquote><p>As a shy, nerdy and self conscious child, I have been terrible at self-promotion well into my early 30&#39;s. However as I grew as a leader, I had to specially pay attention and learn this skill —not just for my sake but also for the sake of the teams I was leading.</p><p>Here are 3 of the most effective strategies I learnt and used to promote Analytics and DS teams and demonstrate the impact of the work being done —</p><h4>1. Announcement emails</h4><p>While working at a tech startup, not long ago, my team and I thought we were doing a good job of broadcasting our work. But then something happened that turned our perception on its head. We had worked very hard to create a new Customer Retention Dashboard — creating new data pipelines, defining metrics that helped surface crucial insights, and iterating to eventually achieve great visualizations. I sent out an email sharing the dashboard with a small group of audience and we got a decent response as expected.</p><p><em>‘This looks great! Thanks.’</em></p><p><em>‘Very helpful — thanks for sharing.’</em></p><p>Then our VP of Marketing stepped in and blew us away with what can be achieved with the right packaging. She sent out another communication ‘announcing’ (not just sharing) the dashboard to a broader audience. The response was spectacular! She generated far more engagement and excitement around the dashboard than we did with our comms.</p><p>‘<em>This is incredible. Thank you</em> 🙌’</p><p>‘<em>What a masterpiece. 🔥 This level of analytical rigor will be instrumental for unlocking future growth! I’d like to put a request in for a lunch-n-learn dashboard demo ;)’</em></p><p><em>‘Love, love, love — amazing work!!’</em></p><p>In the end, this was one of the most bookmarked dashboards and obviously created a lot of brand equity for the authors and the team.</p><p>It made me realize the enormous opportunity for packaging our announcement or distribution emails better. Here are some tips that I have learnt along the way :</p><ul><li><strong>Use catchy subject lines</strong> — e.g. ‘Retention Dashboard is here!’ or ‘Introducing Retention Dashboard’</li><li>In addition to stating what the dashboard contains, tie it to <strong>key insights, recommendations and next steps</strong></li><li><strong>Use icons &amp; visuals</strong> — Adding relevant icons and visuals makes the email easier to consume and provides a nice break from all the heavy text. <em>Caution: Do not overuse!</em></li></ul><blockquote>John, Ramya and Kevin have been developing a fantastic one-stop-shop for consolidating trends in customer sentiment into a centralized Customer Feedback Dashboard. This will be our primary resource to support the company’s focus on Know Your Customer that could unlock many levers of growth for us. 🚀</blockquote><blockquote><strong>Within this dashboard you will find -</strong></blockquote><blockquote>— Customer sentiment by business line: <em>💡 XXX has the highest negative sentiment while YYY has the lowest negative sentiment across all customer cohort</em></blockquote><blockquote>— Sentiment breakdown over time:<em>💡Customer sentiment has been worsening in the last 6 months with sharp uptick during the winter storm a few weeks back</em></blockquote><blockquote><strong>Recommendations &amp; Next Steps -</strong></blockquote><blockquote>— Conduct user study to dig deep into opportunities to improve Customer Service</blockquote><blockquote>I highly encourage you all to bookmark this resource, share with your teams, and leverage the great work the team has driven to surface customer insights using previously untapped data sources. Be on a lookout for a demo on how to use this dashboard. Until then feel free to reach out on #Customer-Insights slack channel with any questions.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mcmQ3L-3-dClVtp7Dufz2g.png" /><figcaption>Source: Author</figcaption></figure><h4>2. Readouts</h4><p>For those of you who may not have heard the term — ‘readout’ is another term for an analysis that is packaged in a way that is easy to read through. Typically I have seen readouts done by teams when they do a one-time analysis — be it a <strong>deep dive into a topic</strong> e.g. what drives customer retention or <strong>root cause analysis</strong> e.g. why did top of funnel conversion decline or <strong>analyzing an experiment / launch / campaign performance.</strong></p><p>In addition to one-time readouts, what will earn you a lot of credit is to do recurring readouts on topics important to your stakeholders. Depending on business needs, you can choose a weekly or monthly cadence.</p><h4><strong>Recurring readouts from dashboards</strong></h4><p>How often have you heard the need for ‘More insight and less dashboards’? I have seen many Analytics teams get flustered with this one — you NEED dashboards to create insights, they would counter. While that may be true, what ends up happening is without a continual stream of insights from those dashboards, they are just seen as another source of data. Actively sharing summarized findings from these dashboards to the stakeholders will change this perception.</p><blockquote>We need more insights and less dashboards</blockquote><h4>Monthly or Quarterly Business Reviews</h4><p>These are another great way of maintaining a recurring flow of information to your stakeholders. Business Reviews are presentations where you review health of business based on trends in key metrics. These give a chance for senior leadership to step back from the day today humdrum and look at the business performance month over month or quarter rover quarter.</p><p>Analytics teams are best positioned to present business reviews for multiple reasons — <br>1) They are the experts on the source of truth data and metrics<br>2) They are more likely to take an analytically rigorous and hypothesis driven approach<br>3) They are impartial</p><h4>Customer’s perspective</h4><p>At the end of the day, your business partners are in some sense your customers so let’s step into their shoes and think about their needs. Imagine doing your day-to-day job as a Marketer, Product Manager or Operations Manager and then also having to dig into and stitch together information from dashboards. This is a big time sink for them and frankly not what they are best at. If you can push this information to them instead of them coming to you constantly, not only will this alleviate the pain point for them, they will also start seeing you as a trusted partner. Realigning this push-pull dynamic will also help you be more in control of your time and work priorities.</p><blockquote>We need to change the dynamic such that the team is pushing insights to Execs versus them asking for it</blockquote><blockquote>~ CFO of a tech startup on Analytics &amp; DS teams</blockquote><h4>3. Newsletter</h4><p>This is an easy one and yet so effective. You don’t need to get a lot of cross-functional alignment for this one in terms of prep or set up. You can simply start publishing the newsletter to the intended audience expressing your goal and iterate based on feedback. The trick here is to highlight goals for ongoing work-streams and outcomes for those completed, always connecting to business outcomes or stakeholder needs. Here is a sample layout that will earn you many brownie points:</p><ul><li><strong>Summary</strong> — Key Wins &amp; What’s Coming</li><li><strong>Detailed updates by themes</strong></li><li><strong>Newsletter FAQ’s —</strong><br>- <strong>Goals</strong>: What is the goal of this newsletter. e.g. Providing visibility and aligning on prioritization<br>- <strong>Cadence</strong>: Weekly / Bi-weekly / Monthly<br>- <strong>Audience</strong>: Sr. Leadership of company<br>- <strong>Team members</strong><br>- <strong>POC: </strong>Who should they reach out to if they have questions</li></ul><blockquote><strong>Summary:</strong> The team was primarily focused on (1) a framework for customer sentiment analysis, (2) revising north star metrics definitions . Beyond that we’ve been investigating a new Experimentation Platform POC (A/B testing) and scoping churn modeling.</blockquote><blockquote><strong>Next up:</strong> Finalizing north star metric definitions and pipelines, customer segmentation framework and goal setting sessions (in the next 1 month).</blockquote><blockquote><strong>1. Customer Sentiment Dashboard:</strong></blockquote><blockquote><strong><em>Goal: </em></strong>Create a consolidated view to surface actionable insights for growth sr. leadership</blockquote><blockquote>— Developed framework approach, roadmap and executed on it</blockquote><blockquote>— Identified initial set of metrics and conducted feasibility study, stakeholder interviews</blockquote><blockquote>— Built new data pipelines to surface to Tableau and generated insights &amp; recommendations</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*-dCvhfPmUCj_5eS8KTsg6w.png" /><figcaption>Source: Author</figcaption></figure><blockquote><em>Bonus Tip</em></blockquote><blockquote>After you have sent all this communication, look at the communication as the start of a dialogue and not the end of a conversation. The idea should not be “I have provided them with the tool and they will take it from there,” but instead it should be a conversation starter. You should always have enough information in your communication on who they can reach out to for more questions and collaboration.</blockquote><h3>Conclusion</h3><p><em>Think about a project you are working on or a team you are leading. Can you see yourself utilizing these tools to evangelize your work?</em></p><p>There are over a million job posts on LinkedIn for Analytics &amp; Data Science roles. Most non-analytics roles also emphasize the importance of data driven skills. And yet, many Analytics &amp; Data Science teams struggle to demonstrate the value they are driving. With added skills on effectively communicating your work, you will not just be good at your work but be recognized for it.</p><p><em>If you would like to continue seeing content from me on career growth in Analytics &amp; Data Science, you can follow &amp; subscribe to me on </em><a href="https://preeti-semwal.medium.com/subscribe"><em>Medium</em></a><em>.</em></p><p>Other resources from me to advance your career in Analytics &amp; DS—</p><p><a href="https://www.udemy.com/course/product-experimentation-ab-testing-in-r-with-real-examples/?couponCode=3B2B9F1E96F8605CD651"><strong>Complete Course on Product A/B Testing with Interview Guide</strong></a> <em>(Use this link to avail discounted price)</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7b87d80615fc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/top-3-tools-to-promote-your-work-in-analytics-data-science-7b87d80615fc">Top 3 Tools to Promote Your Work in Analytics &amp; Data Science</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to ace A/B Testing Data Science Interviews (Doordash Case Example)]]></title>
            <link>https://medium.com/data-science/cracking-a-b-testing-data-science-interviews-bc66e399b109?source=rss-62e228735f49------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc66e399b109</guid>
            <category><![CDATA[product-experimentation]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[data-science-interview]]></category>
            <category><![CDATA[doordash]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Preeti Semwal]]></dc:creator>
            <pubDate>Wed, 17 Mar 2021 20:02:47 GMT</pubDate>
            <atom:updated>2022-11-16T18:22:41.381Z</atom:updated>
            <content:encoded><![CDATA[<h4>Mock Interview with step by step guide</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-ceHJjq8wNgI5zUk1u9nTw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@brucemars?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">bruce mars</a> on <a href="https://unsplash.com/s/photos/happy?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h3>Introduction</h3><p>A/B Testing is an in-demand skill that is often tested in data science interviews. At the same time, there are very few resources out there to help prepare for A/B testing interviews. In my 15 year career and as a hiring manager in Data Science, I have found that most candidates perform poorly in these interviews. In fact, the field of experimentation has been evolving and there are new concepts and approaches that are becoming more relevant each year. Which means that even seasoned data scientists who may have done A/B testing some years back, often find themselves stumped in interviews.</p><p>In this post, we will go through a mock interview that will help you understand what the interviewer is looking for and how to approach these interviews. Why mock interview, you might ask? Well, as data scientists, we sometimes struggle with communication and having a template in mind helps tremendously. Personally, I have also found that when I can visualize a high stake situation and how it may play out, it helps me be better prepared mentally, handle pressure well and perform better overall.</p><p>We will use an example from <strong>Doordash</strong>, a food delivery company, whose mobile app currently ranks #1 on the iPhone App Store. They are constantly improving their app through experimentation and look for strong experimentation skills during their data science interviews — especially for product data science or product analytics roles.</p><h3>Question</h3><p>INTERVIEWER — Doordash is expanding into other categories such as convenience store delivery. Their notifications have had good success in the past and they are considering sending an in app notification to promote this newly launched category.</p><p>How would you design and analyze an experiment to decide if they should roll out the notification?</p><h3>Solution</h3><h3>Part 1 — Ask clarifying questions to understand business goals and product feature details well</h3><p><em>What Interviewer is looking for -</em></p><ul><li><em>Did you begin by stating the product/business goal before diving into the experiment details? Talking about the experiment without knowing the product goal is a red flag.</em></li></ul><p>INTERVIEWEE — Before we begin with the experiment details, I would like to make sure my understanding of the background is clear. There could be multiple goals with a feature like this one — such as increasing new user acquisition, increase conversion for this category, increasing # of orders in the category or increasing total order value. Can you help me understand what the <strong>goal</strong> is here?</p><p>INTERVIEWER — That’s a fair question. With the in-app notification, we are primarily trying to increase the conversion rate for the new category — i.e. % of users that place an order in the new category out of all users that login to the app.</p><p>INTERVIEWEE — Ok, that’s helpful. Now I would like to also understand more about the notification — what is the messaging and who is the intended audience?</p><p>INTERVIEWER — We are not offering any discount at this point. The messaging is simply going to be to let them know we have a new category that they can start ordering from. If the experiment is successful, we intend to roll out the notification to all users.</p><p>INTERVIEWEE — Ok. Thanks for that background. I am now ready to dive into the experiment details.</p><h3>Part 2 — State Business Hypothesis, Null Hypothesis &amp; define metrics to be evaluated</h3><p><em>What Interviewer is looking for -</em></p><ul><li><em>That you think through secondary metrics and guardrail metrics in addition to the primary metrics</em></li></ul><p>INTERVIEWEE — So to state the <strong>business hypothesis </strong>-we expect that if we send in-app notification, then the daily number of orders in the new category will increase. That means our <strong>Null Hypothesis (Ho) </strong>is that there is no change in the conversion rate due to the notification.</p><p>Now let me state the different metrics that we will want to include for the experiment — Since the goal of the notifications is to increase the conversion rate in the new category. That will be our <strong>primary metric</strong>. In terms of <strong>Secondary metrics, </strong>we should also watch the average order value to see what the impact is. It is possible that the conversion rate increases but the average order value decreases such that the resulting impact is lower overall revenue. That is something we may want to watch out for.</p><p>We should also consider <strong>guardrail metrics</strong> — these are metrics that are critical to the business that we do not want to impact through the experiment such as time spent on app or app uninstalls for example. Are there any such metrics that we should include in this case?</p><p>INTERVIEWER — I agree with your choice of primary metric but you can ignore the secondary metrics for this exercise. And you are spot on in terms of guardrail metrics — Doordash wants to be judicious about any features or releases when it comes to their app because we know that the LTV of a customer who has installed the app is much higher. We want to be careful so as not to drive users to uninstall the app.</p><p>INTERVIEWEE — Ok — that’s good to know. So we will include % of uninstalls as our guardrail metric.</p><h3>Part 3 — Choose significance level, power, MDE and calculate the required sample size and duration for the test</h3><p><em>What Interviewer is looking for -</em></p><ul><li><em>Your knowledge of the statistical concepts and the calculation for sample size and duration</em></li><li><em>Whether you consider factors such as network effect (common in two sided marketplaces such as Doordash, Uber, Lyft, Airbnb or social networks such as FB, LinkedIn), day of week effect, seasonality or novelty effect that may affect the validity of the test and need to be considered while arriving at the experiment design</em></li></ul><p>INTERVIEWEE — Now I would like to get into the design of the experiment.</p><p>Let’s first see if we need to consider <strong>network effects </strong>— these occur when the behavior of the control is influenced by the treatment given to the test group. Since Doordash is a double sided marketplace, it is more prone to seeing network effects. In this specific case, it is possible that if the treatment given to the test increases the demand from the test group, that may result in a deficit of supply (i.e. dashers) which could in-turn affect the performance of the control group.</p><p>To account for network effects, we will need to choose the <strong>randomization unit</strong> differently than we would normally do. There are many ways to do this — we could do<strong> geo-based randomization or time-based randomization or network-cluster randomization or network ego-centric randomization. </strong>Would you like me to go into the details for these?</p><p>INTERVIEWER — I am glad you brought up network effects as it is in fact something we carefully look for for our experiments in Doordash. In the interest of time, let’s assume there are no network effects in play here and move on.</p><p>INTERVIEWEE — So if we are assuming there are no network effects to be accounted for, the <strong>randomization unit</strong> for the experiment is simply the user — i.e. we will randomly select users and assign them to treatment and control. Treatment will receive notifications while control will not receive any notifications. Next, I would like to calculate the sample size and duration. For this I need a few inputs.</p><ul><li><strong>Baseline conversion </strong>— which is the existing conversion of the control before changes are made</li><li><strong>Minimum detectable difference or MDE</strong> — which is the smallest change in conversion rate we are interested in detecting. A smaller change than this will not be practically significant to the business — it is typically chosen such that the improvement in the desired outcome will justify the cost of implementing and maintaining the feature</li><li><strong>Statistical Power</strong> — which is the probability that a test correctly rejects the null hypothesis</li><li><strong>Significance Level</strong> — which is the probability of rejecting a null hypothesis when it is true</li></ul><p>A 5% significance level and power of 80% are usually chosen and I will assume these unless you say otherwise. Also I will assume a 50–50 split between the control and treatment. Once I have these inputs finalized, I will use power analysis to calculate the sample size. I would use a programming language for this. For e.g. In R, there is a package called ‘pwr’ that can be used for this.</p><p>INTERVIEWER — Yes, let’s say based on the analysis we get a sample size of 10,000 users per variation needed. How will you calculate the duration for the test?</p><p>INTERVIEWEE — Sure, for this we will need the daily number of users that login to the app.</p><p>INTERVIEWER — Assume we have 10,000 users that login to the app daily.</p><p>INTERVIEWEE — Ok, in that case, we would need at a minimum 2 days to run the experiment — I arrived at this by taking the total sample size of Control &amp; Treatment and dividing by daily user count. However, there are other factors we should consider when finalizing the duration -</p><ul><li><strong>Day of week effect — </strong>You may have a different population of users on weekend than weekdays — hence it is important to run long enough to capture weekly cycles.</li><li><strong>Seasonality — </strong>There can be times when users behave differently that are important to consider, such as holidays.</li><li><strong>Novelty effect </strong>— When you introduce a new feature, especially one that’s easily noticed, initially it attracts users to try it. A test group may appear to perform well at first, but the effect will quickly decline over time.</li><li><strong>External effects — </strong>for e.g. let’s say the market is doing really well and more people are likely to ignore the notification with the expectation of making high returns. This will lead us to draw spurious conclusions from the experiment</li></ul><p>Due to the above, I would recommend running the experiment for at least one week.</p><p>INTERVIEWER — Ok, that’s fair. How would you analyze the test results?</p><h3>Part 4 — Analyze the results and draw valid conclusions</h3><p><em>What interviewer is looking for -</em></p><ul><li><em>Your knowledge of the appropriate statistical tests to be used in different scenarios (for e.g. t-test for sample mean and z-test for sample proportions)</em></li><li><em>You check for randomization — this will get you some brownie points</em></li><li><em>You provide a final recommendation (or a framework to get there)</em></li></ul><p>INTERVIEWEE — Sure. There are two key parts to the analysis -</p><ul><li><strong>Check for randomization</strong> — As best practice, we should check that the randomization was done correctly when assigning test and control. For this, we can look at some baseline metrics that we do not expect to be influenced by the test and compare them for the two groups. We can do this comparison by comparing the histograms or density curves for these metrics between the two groups. If there is no difference, we can conclude that randomisation was done correctly.</li><li><strong>Significance test for all metrics</strong> (including primary and guardrail metrics) — Both our primary metric (conversion rate) and guardrail metric (uninstall rate) are proportions. We can use the z-test to test for statistical significance. We can do this using a programming language such as R or Python.</li></ul><p>If there is a statistically significant increase in conversion rate, and uninstall rate is not impacted negatively, I would recommend implementing the test.</p><p>If there is a statistically significant increase in conversion rate, and uninstall rate is impacted negatively, I would recommend not implementing the test.</p><p>And lastly, if there is no statistically significant increase in conversion rate — I would recommend not implementing the test.</p><p>INTERVIEWER — That all sounds good. Thanks for your response.</p><h3>Conclusion</h3><p>Doing well in A/B testing or experimentation interviews will provide you with an edge in the hiring process and set you apart from other candidates. So I would highly recommend spending focused time to learn key concepts in A/B testing and prepare well for these interviews. I have put together a 2.5 hour course that covers the concepts in depth with real world examples and assignments. Check it out — <a href="https://www.udemy.com/course/product-experimentation-ab-testing-in-r-with-real-examples/?referralCode=C0EC84F644B6BCB10FD0">Complete Course on Product A/B Testing with Interview Guide </a><em>(</em><strong><em>Use this link to avail discounted price)</em></strong></p><p>Happy Learning and Interviewing!</p><p><em>If you would like to continue seeing content from me on data science and analytics interviews prep, you can follow me on </em><a href="https://preeti-semwal.medium.com/"><em>Medium</em></a><em>.</em></p><p><em>If you are looking for data science career related guidance or 1–1 mentorship, you can reach out to me on </em><a href="https://www.linkedin.com/in/preeti-semwal/"><em>Linkedin</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc66e399b109" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/cracking-a-b-testing-data-science-interviews-bc66e399b109">How to ace A/B Testing Data Science Interviews (Doordash Case Example)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>