<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Phani Chundru on Medium]]></title>
        <description><![CDATA[Stories by Phani Chundru on Medium]]></description>
        <link>https://medium.com/@phanikumarc24?source=rss-1ddb54a16b1e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*M594MxG14zAUJgt_fl5OlA.jpeg</url>
            <title>Stories by Phani Chundru on Medium</title>
            <link>https://medium.com/@phanikumarc24?source=rss-1ddb54a16b1e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 09:19:31 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@phanikumarc24/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Unlocking the Power of A/B Testing: The Ultimate Guide to Making Data-Driven Decisions]]></title>
            <link>https://medium.com/@phanikumarc24/unlocking-the-power-of-a-b-testing-the-ultimate-guide-to-making-data-driven-decisions-e1eda91f9c87?source=rss-1ddb54a16b1e------2</link>
            <guid isPermaLink="false">https://medium.com/p/e1eda91f9c87</guid>
            <category><![CDATA[research]]></category>
            <category><![CDATA[product]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[decision-making]]></category>
            <category><![CDATA[product-management]]></category>
            <dc:creator><![CDATA[Phani Chundru]]></dc:creator>
            <pubDate>Sun, 17 Sep 2023 11:33:36 GMT</pubDate>
            <atom:updated>2023-09-17T12:22:44.590Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/801/1*eC5KjDlfVvwPUAvt7vP3ew.jpeg" /></figure><h3>What’s the Hype Around A/B Testing?</h3><p>First things first — why should you even care about A/B testing? Well, it’s the closest thing to a crystal ball that Product Managers have. It allows you to peek into alternative futures to know which version of your website or app would be the most effective in achieving your goals — before committing to changes.</p><p>Today, I’m going to demystify A/B testing, starting from the absolute basics to running your own experiments and finally making sense of the data you gather. So whether you’re a complete newbie or a seasoned pro, this guide is here to empower you to make better, data-driven decisions. Let’s jump right in!</p><h3>Step 1: The Hypothesis — Where It All Begins</h3><p>Every good experiment starts with a question or a hypothesis. Imagine you’re the proud owner of an e-commerce website, and you’re not satisfied with the current Click-Through Rate(CTR) of your “Buy Now” button. You think to yourself, “Hey, what if the button were green instead of red?” Boom! You’ve got yourself a hypothesis: “Changing the ‘Buy Now’ button from red to green will improve the CTR.”</p><p>To put this hypothesis to test, you’ll need to establish a Null Hypothesis(H0) and an Alternative Hypothesis(Ha​):</p><blockquote><strong><em>Null Hypothesis(H0​):</em></strong><em> Changing the “Buy Now” button from red to green will have no effect on the CTR.</em></blockquote><blockquote><strong><em>Alternative Hypothesis (Ha​):</em></strong><em> Changing the “Buy Now” button from red to green will improve the CTR.</em></blockquote><p><strong>Terms to Know:</strong></p><p><strong>Click-Through Rate(CTR):</strong> It’s the percentage of people who click on an element (like your “Buy Now” button) divided by the total number of people who view the page or element.</p><h3>Step 2: Calculating Sample Size</h3><p>This is the number of people you need to participate in your experiment to get statistically significant results. The size depends on several factors, including your baseline conversion rate, desired level of significance, and the minimum detectable effect(MDE).</p><p>Before moving into the calculation of sample size, let’s understand some terminology:</p><p><strong>Baseline Metric:</strong> This is the current value of your metric before the test begins. It serves as a point of comparison.</p><ul><li>Example: Suppose the CTR of your red “Buy Now” button is currently 2%. That’s your baseline metric.</li></ul><p><strong>Minimum Detectable Effect (MDE):</strong> This is the smallest change in the metric that you consider meaningful or valuable. It’s expressed as a percentage.</p><ul><li>Example: Let’s say you’d be happy if the green “Buy Now” button increases the CTR by at least 0.5%. This 0.5% increase is your MDE.</li></ul><p><strong>Statistical Significance (Alpha Level):</strong> This is the probability of rejecting the null hypothesis when it’s actually true, commonly set at 0.05 (or 5%). Lower values mean you’ll be more sure of your results, but they require a larger sample size.</p><ul><li>Example: If you set the alpha at 0.05, there’s a 5% chance you could be wrong in thinking the green button is better when it’s not.</li></ul><p><strong>Statistical Power (Beta Level):</strong> Power is the probability that you’ll correctly reject the null hypothesis when the alternative hypothesis is true. It’s commonly set at 0.8 (or 80%).</p><ul><li>Example: With a beta level of 0.8, you’re 80% confident that if the green button really is better, your test will show it.</li></ul><p><strong>Two-Tailed or One-Tailed Test:</strong> A two-tailed test checks for differences in both directions (increases and decreases), whereas a one-tailed test checks in just one direction.</p><ul><li>Example: If you only care about whether the green button increases the CTR and aren’t interested in knowing if it could make it worse, you would use a one-tailed test.</li></ul><p><strong>Control Group: </strong>The control group is the segment of users who are exposed to the current version of the product or feature. They serve as a benchmark for evaluating how well the new variation performs in comparison.</p><ul><li>Example: In our “Buy Now” button color change scenario, the control group would continue to see the red “Buy Now” button, just as they always have. Any metrics, such as CTR collected from this group represent what you can expect if no changes are made.</li></ul><p><strong>Treatment Group: </strong>The treatment group is the segment of users who are exposed to the new variation that you’re testing. The metrics collected from this group will tell you how the new change impacts user behavior.</p><ul><li>Example: In the case of changing the “Buy Now” button color, the treatment group would see the new green “Buy Now” button. Metrics like CTR would be collected from this group to evaluate whether the green button leads to better engagement, conversion, or other targeted outcomes.</li></ul><p>To calculate your sample size, you’d generally use a textbook formula (which can vary depending on the parameters mentioned above). You can also find online calculators like the one mentioned below to make this process easier.</p><p><a href="https://www.evanmiller.org/ab-testing/sample-size.html">Sample Size Calculator</a></p><p>Using these variables — a 0.05 alpha level, a 0.8 power level, a 2% baseline conversion rate, and an MDE of 0.5% — you might find that you need about 13,000 users for each group (control and treatment) for your experiment to be statistically valid.</p><h3>Step 3: Running the Experiment</h3><p>Once you’ve set up your hypothesis and calculated the sample size, you’re ready to roll. But there are some do’s and don’ts you need to follow:</p><h4>Do’s:</h4><ul><li>Run the Test Simultaneously: Make sure you’re comparing apples to apples.</li><li>Keep Time Constants: Run your A/B test for a complete week to account for day-to-day variations.</li></ul><h4>Don’ts:</h4><ul><li>Peek at Results: Resist the urge to stop the experiment early if things look promising. Stick to your sample size!</li></ul><h3>Step 4: Analyzing the Results</h3><p>You’ve done the legwork: set your hypothesis, calculated the sample size, and ran the test. Now comes the moment of truth. It’s time to analyze the data and draw conclusions.</p><p><strong>Gather Your Data</strong></p><p>First, compile the results for both your treatment group (users who saw the green “Buy Now” button) and control group (users who saw the red “Buy Now” button).</p><ul><li>Treatment Group (Green Button) — CTR: 2.5%</li><li>Control Group (Red Button) — CTR: 2%</li></ul><p><strong>Calculate Point Estimate</strong></p><p>The point estimate gives you the best single guess for a parameter. In our example, the point estimate would be the difference between the CTR of the treatment group and the control group. As per our experiment, Point Estimate = 2.5% — 2% = 0.5%.</p><p><strong>Calculate Z-Score</strong></p><p>The Z-score tells you how many standard errors there are between the point estimate and the null hypothesis value (which is zero if you’re measuring the difference). A high absolute value of the Z-score implies that the observed difference is less likely due to random chance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/0*KZE1k7-Fb2z0JrER.png" /></figure><p>From our experiment, we have already established:</p><ul><li>Treatment group (new “Buy Now” button) conversion rate, p1 = 2.5% or 0.025</li><li>Control group (old “Buy Now” button) conversion rate, p2 = 2% or 0.02</li><li>Sample size for both groups, n1=n2=13000</li></ul><p>Plugging in the values in the above formula or using an <a href="https://www.socscistatistics.com/tests/ztest/default2.aspx">online calculator</a>, we get Z-score as <strong>2.72</strong></p><p><strong>Make The Decision</strong></p><p>When it comes to making a decision, it’s crucial to approach it from multiple angles for comprehensive analysis. This includes not only statistical significance but also practical significance. We’ll explore both these concepts using Z-score, P-value, and the Minimum Detectable Effect (MDE).</p><p><strong>Using Z-score</strong></p><p>First, let’s talk about the Z-score. In our example, we’ve calculated a Z-score of 2.72. This figure is then compared with a critical Z-value, which, for a two-tailed test with 95% confidence, is typically 1.96. Our Z-score of 2.72 exceeds the critical Z-value of 1.96, meaning that the result of our experiment is statistically significant at a 95% confidence level. In other words, we can reject the null hypothesis that says changing the “Buy Now” button from red to green will have no effect on the CTR. We’ve got enough evidence to say that changing the button color does make a difference.</p><p><strong>Using P-value</strong></p><p>Now, let’s delve into the P-value. The P-value is the probability of obtaining test results as extreme as the ones observed, under the assumption that the null hypothesis is true. A low P-value indicates that the results are statistically significant and that the null hypothesis can be rejected.</p><p>To find the P-value from a Z-score, you can use an <a href="https://www.socscistatistics.com/pvalues/normaldistribution.aspx">online calculator</a>. Here, the P-value corresponding to a Z-score of 2.72 is 0.0065. This is below the commonly accepted alpha level of 0.05, providing additional evidence against the null hypothesis.</p><p><strong>Considering Practical Significance: The MDE</strong></p><p>While it’s critical to establish statistical significance, it’s equally important to consider practical significance. In our example, we set an MDE of 0.5%, which is the smallest change in the metric that we consider valuable. Having achieved this MDE in our experiment, we can say that the change is not just statistically significant but also practically significant for our business goals.</p><p><strong>Conclusion</strong></p><p>Both our Z-score and P-value indicate that we should reject the null hypothesis, and the achieved MDE confirms the practical value of this change. In summary, changing the “Buy Now” button from red to green improved the CTR in a manner that is both statistically and practically significant. Therefore, you can go ahead and implement the change on your website.</p><p><strong>About The Author</strong></p><p>Hello! I’m Phani, currently working as a Product Manager in an edtech startup. My journey has taken me from coding to an MBA at IIM Kozhikode, and now to the exciting world of startups. I’m passionate about building products and sharing my experiences. You can expect posts about product management, startup challenges, and lots more. Feel free to connect with me on <a href="https://www.linkedin.com/in/phanichundru">LinkedIn</a>/<a href="https://www.x.com/phani_chundru_">Twitter</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e1eda91f9c87" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction: The Start of a New Journey]]></title>
            <link>https://medium.com/@phanikumarc24/introduction-the-start-of-a-new-journey-378860868c74?source=rss-1ddb54a16b1e------2</link>
            <guid isPermaLink="false">https://medium.com/p/378860868c74</guid>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[startup-lessons]]></category>
            <dc:creator><![CDATA[Phani Chundru]]></dc:creator>
            <pubDate>Mon, 28 Aug 2023 08:41:11 GMT</pubDate>
            <atom:updated>2023-08-28T08:41:11.537Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*z1Df1BasykxcxbIgCWsxwQ.png" /></figure><h4>From Coding to Creating</h4><p>Like many engineers in India, I began my career as a software developer in a service-based company. After three years, I felt a calling for more than just coding. I wanted to be part of the process where businesses are built, where decisions about what, when, and how to build are made.</p><h4>The Road to Product Management</h4><p>I found my path by pursuing an MBA from a top institute in the country, with the aim of breaking into product management. Preparing for the CAT exam was challenging but rewarding, leading me to IIM Kozhikode.</p><p>Post MBA, I worked as a program manager at Flipkart, designing supply chain processes. Although the work was significant, I felt drawn to building products for direct customers.</p><h4>Taking the Plunge</h4><p>When an opportunity arose with an edtech startup in its ideation stage, I knew it was my chance. I left my stable job to build products from scratch in a startup environment. It was a bold move, but one that felt right for my career.</p><h4>Embracing the Chaos</h4><p>Since joining the startup, the learnings have been immense. In just a few months, I’ve experienced growth that feels more substantial than an entire year at Flipkart. It’s about clarity in chaos, prioritization, and understanding various perspectives.</p><h3>Stay Tuned!</h3><p>I’ll be sharing my insights and learnings as I navigate the fascinating journey of building products at a startup from “-1 to 1”. It’s a thrilling and inspiring experience that I’m eager to share with you all. Stay tuned and wish me luck!</p><h3>About the Author</h3><p>Hello! I’m Phani, currently working as a Product Manager in an edtech startup. My journey has taken me from coding to an MBA at IIM Kozhikode, and now to the exciting world of startups. I’m passionate about building products and sharing my experiences. You can expect posts about product management, startup challenges, and lots more. Feel free to connect with me on <a href="https://www.linkedin.com/in/phanichundru">LinkedIn</a>/<a href="https://www.x.com/phani_chundru_">Twitter</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=378860868c74" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>