<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kavikumaran R on Medium]]></title>
        <description><![CDATA[Stories by Kavikumaran R on Medium]]></description>
        <link>https://medium.com/@kavikumaran?source=rss-f6917b2a396d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*JZHOrG1uIRiWk39ufgj68A.jpeg</url>
            <title>Stories by Kavikumaran R on Medium</title>
            <link>https://medium.com/@kavikumaran?source=rss-f6917b2a396d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 23:02:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kavikumaran/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[4 Powerful Prioritization Frameworks for every Product Manager]]></title>
            <link>https://medium.com/design-bootcamp/4-powerful-prioritization-frameworks-for-every-product-manager-558de71a11dc?source=rss-f6917b2a396d------2</link>
            <guid isPermaLink="false">https://medium.com/p/558de71a11dc</guid>
            <category><![CDATA[product-manager]]></category>
            <category><![CDATA[framework]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[product-development]]></category>
            <category><![CDATA[product]]></category>
            <dc:creator><![CDATA[Kavikumaran R]]></dc:creator>
            <pubDate>Fri, 17 Dec 2021 13:17:38 GMT</pubDate>
            <atom:updated>2021-12-17T14:51:06.402Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*otse8d76pjbvocCX" /><figcaption>Photo by <a href="https://unsplash.com/@andretaissin?utm_source=medium&amp;utm_medium=referral">Andre Taissin</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>As a product manager, with only a few chances to make an impact you got to make the right choices while prioritizing what to build next from a long list of options. The first and foremost choice is to choose between <strong><em>value and innovation</em></strong> and honestly, the vision and mission of the company have a large role to play in this. If you’re a startup trying to build an MVP (Minimum Viable Product), you must be focussing on solving a real problem faced by the user whereas if you have already found the product-market fit, you may focus on innovating by adding new features and thereby can delight your users.</p><p>Another key challenge faced by PMs is figuring out a way to <strong><em>balance the feedbacks and re-prioritize</em></strong> stuff. Interpretation of your users (includes customers, partners, and internal stakeholders) is a gold mine of information but it’s still from a limited perspective they have. Reacting to every response could prevent you from making actual advancements. That is why we have a wide number of frameworks that address all these challenges and help to incorporate strategic thinking in the product prioritization process. <strong>In this article, we will be discussing 4 of the most widely used product prioritization frameworks in detail.</strong></p><h4>Eisenhower Matrix — Urgency vs Importance</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FtimB7TP6hLecE5phaiIYg.png" /><figcaption>Picture from techtello.com</figcaption></figure><p>The Eisenhower matrix, also known as Eisenhower Box or Urgent-Important matrix, is a framework developed by President Dwight Eisenhower which helps in prioritizing high-stakes features or backlogs by categorizing them based on their urgency and importance.</p><ol><li><strong><em>Quadrant I</em></strong><em>: Urgent and Important. Do it right away.</em><strong><em> Example: A Client Projec</em></strong><em>t</em></li><li><strong><em>Quadrant II</em></strong><em>: Important and Not Urgent. Plan and Schedule to do it.</em><strong><em> Example: Internal Training Programs</em></strong></li><li><strong><em>Quadrant III</em></strong><em>: Not Important but Urgent. Delegate it to the team.</em><strong><em> Example: Responding to emails</em></strong></li><li><strong><em>Quadrant IV</em></strong><em>: Neither Important nor Urgent. Ignore or Drop it.</em></li></ol><h4>Value vs Complexity</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/431/1*e84iTdxrzMD7N0IhM9m_Ug.png" /><figcaption>Picture from: productfolio.com</figcaption></figure><p>The Value vs Complexity framework views each product initiative or feature under 2 distinct lenses: the <strong><em>potential value</em></strong> it could deliver for the client as well as the company and the effort (<strong><em>time and finite development resources at your disposal</em></strong>) it would take to accomplish/execute the initiative successfully. Once plotted inside a quadrant structure your Value vs Complexity matrix will have the following 4 quadrants.</p><ol><li><strong><em>Quick Wins</em></strong><em>: High Value and Low Complexity. There is a very high chance that these initiatives have already been completed or taken care of in some form.</em></li><li><strong><em>Major Projects &amp; Strategic Initiatives:</em></strong><em> High Value and High Complexity. Most of your company &amp; product initiatives fall under this category</em></li><li><strong><em>Fill-Ins</em></strong><em>: Low Value and Low Complexity. Doesn’t require immediate attention and can be pursued later when there are adequate resources available.</em></li><li><strong><em>Thankless Tasks</em></strong><em>: Low Value and High Complexity. Simply Scrap it.</em></li></ol><h4>MoSCoW Analysis</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*5xEjGQ8ocnWNZw6NeJHd1A.png" /><figcaption>Picture from: online.visual-paradigm.com</figcaption></figure><p>The MoSCoW Analysis framework as depicted in the self-explanatory table above classifies the product features into 4 major categories. More often than not cross-functional teams participate in brainstorming activities to decide and agree upon the place of each feature inside the framework as different features could have a different value prop to each of them.</p><h4>RICE Scoring</h4><p>RICE is a scoring model used in product prioritization that takes 4 things into account: reach, impact, confidence, and effort.</p><ol><li><strong><em>Reach</em></strong><em>: Number of users who would be directly impacted by this feature in a definite timeframe. This could be the number of new leads or visits to a website.</em></li><li><strong><em>Impact</em></strong><em>: What impact or experience does it leave to the customers? What’s the scale we are planning to quantify the same?</em></li><li><strong><em>Confidence</em></strong><em>: How confident are we about the reach and impact scores? We use percentages to answer this question.</em></li><li><strong><em>Effort</em></strong><em>: How much effort can we dedicate to this initiative? The effort could be man-hours/ weeks taken to accomplish the task.</em></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*KYjuQ4vqpbTJPJsrmhjvtg.jpeg" /><figcaption>Picture from airfocus.com</figcaption></figure><p>The key benefits of using a scoring model are that it eliminates personal biases, empowers us in making better informed (data-informed) decisions, and also enables us to justify the choices made using proper facts.</p><p><strong>If this article piqued your interest and would like to see more such content on analytics, product management, and more, please feel free to follow me! Thanks for reading! :)</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=558de71a11dc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/design-bootcamp/4-powerful-prioritization-frameworks-for-every-product-manager-558de71a11dc">4 Powerful Prioritization Frameworks for every Product Manager</a> was originally published in <a href="https://medium.com/design-bootcamp">Bootcamp</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Successful Data Science Professionals keep these 4 Non-Technical Qualities in check]]></title>
            <link>https://medium.com/@kavikumaran/successful-data-science-professionals-keep-these-4-non-technical-qualities-in-check-8623e39f193d?source=rss-f6917b2a396d------2</link>
            <guid isPermaLink="false">https://medium.com/p/8623e39f193d</guid>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data]]></category>
            <dc:creator><![CDATA[Kavikumaran R]]></dc:creator>
            <pubDate>Wed, 08 Dec 2021 19:14:30 GMT</pubDate>
            <atom:updated>2021-12-08T19:14:30.712Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*StLxdtXnAg1it-ht" /><figcaption>Photo by <a href="https://unsplash.com/@markusspiske?utm_source=medium&amp;utm_medium=referral">Markus Spiske</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Every single <strong><em>data guy</em></strong> out there, irrespective of their formal education and technical background, would have come across situations that in a sense helped them evolve into a much better and efficient professionals. In this article, I would like to highlight such key areas and scenarios.</p><p>To make it more memorable and catchy for you, below are the <strong>A, B, C, D</strong>s of it:</p><h4>Assumption:</h4><p>In Analytics and Data Science, you spend a fair amount of time and effort in making assumptions, which definitely must be calculated and have a context. But the assumptions that we are talking about here are the ones you make about your stakeholders, target audience, and yourself too.</p><p>Especially while you are working with cross-functional teams or business units, it is important to ensure that all of them are on the same page as far as the topic of discussion is concerned. This could save you from lots of misunderstandings that otherwise could have potentially led to chaos and eventually to the downfall of your project.</p><p>To overcome this scenario, you may consider conducting separate interviews or discussions at a business unit level, get their inputs and feedback and later build on top of it. As far as the technical side of things is concerned, you can ask a peer of yours to review your work.</p><h4>Bias:</h4><p>We all are guilty as charged, as some amount of bias always creeps into our analysis. Why is that happening?</p><ol><li>We wouldn’t want to highlight the low performance of the company this quarter as we feel bad about it. “<strong><em>There has to be something wrong with the data, right?”</em></strong></li><li>In a few cases we might have anticipated a different outcome and we don’t want to be proven otherwise.<strong><em> “Of course, my solution can’t be wrong.”</em></strong></li></ol><h4>Communication:</h4><p>By that, I definitely don’t plan on lecturing you how important your soft skills are and how engaging your pitch should be. Of course, that’s supercritical not just for data science but for any profession.</p><p>Rather I would like to highlight the importance of how timely response and ensuring transparency throughout your process can lead you right into the good books of your stakeholders. You must also make sure that you are <strong><em>not being too technical or statistical</em></strong> while delivering your presentation. But why?</p><p>For one, they don’t have the time to listen to how cool your solution is or how you arrived at it? Two, in most cases they may not have the same level of technical expertise as yours. So if they can’t relate your solution to the business context, there is only one way it ends.</p><h4>Delivery:</h4><p>You are the go-to guy, a ray of hope, or whatever they call you to get the job done ASAP. So when they come up with an idea and want it to be implemented, you guessed it, you raise your hands exclaiming ‘Boss, I can get it done in the next x days’ when it could actually take weeks to deliver something like that.</p><p>Don’t do that. As a data science professional, you got to have a <strong><em>solid sense of what has to be done and be realistic about how long is it going to take</em></strong> for you or the team. Your stakeholders may not be pleased with an ETA greater than they expected but think of a scenario where you overpromised and weren’t able to deliver it. Which scenario looks worse for you?</p><p>Make no mistake, this doesn’t mean you can take forever to share your analysis as you feel like you can still improve its quality or scope.</p><p>If you enjoy reading stories like these and would like to support me as a writer, please feel free to follow me and let me know your thoughts in the comments. Thanks for reading! :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8623e39f193d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[3 Reasons why your next Data Product could be a complete disaster]]></title>
            <link>https://medium.com/design-bootcamp/why-your-next-data-product-could-be-a-complete-disaster-d7e754c85e19?source=rss-f6917b2a396d------2</link>
            <guid isPermaLink="false">https://medium.com/p/d7e754c85e19</guid>
            <category><![CDATA[product]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Kavikumaran R]]></dc:creator>
            <pubDate>Sat, 04 Dec 2021 08:16:54 GMT</pubDate>
            <atom:updated>2021-12-17T14:45:01.442Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XCENunNQjxS0u0js" /><figcaption>Photo by <a href="https://unsplash.com/@adamaszczos?utm_source=medium&amp;utm_medium=referral">Adam Nowakowski</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>It’s a Monday morning and your boss has called you to discuss the <strong><em>next big thing</em></strong> to be built for your <em>data-driven</em> company. He goes on to explain the impact it could have on your team and company’s mission and how your stakeholders could potentially gain value from it.</p><p>You tune in to your work mode and after putting in hours of thinking and getting your hands dirty with data and tech, you’re all set to showcase your findings &amp; solution to the executive audience. You are super confident and can’t wait to explain the cool features and benefits of your new data product. Everything goes as anticipated and the end outcome is a HUGE SUCCESS!</p><p>A few days later you receive an ad-hoc request via email that reads;</p><blockquote>“Require x report for y scenario to understand the impact in z market”</blockquote><p>You just can’t help but wonder that the solution you had built and presented earlier was supposed to solve the exact same problem for them. In a sense, it gets you thinking about where exactly things went wrong for you. In spite of putting a lot of effort into building the data product, how can they not see its value proposition?</p><p>If you are in the data analytics space for a few years now this scenario definitely might seem very familiar and for someone who is trying to break through into this highly glorified field, here’s a word of caution:</p><blockquote>There’s a very high chance of your Data product/solution ending up as a failure or a complete disaster, regardless of how good you think it is.</blockquote><h4>With that said, let’s look at possible reasons that could have led to this misery (from a Data Analyst’s POV):</h4><h4>The solution was built for a person, not for personas</h4><p>If you’re going to build a data product for specific people (like you’re boss, highly successful and acclaimed leaders of your company, the ultimate decision-makers), your product leaves the company when they do. Data products are meant to solve problems of personas as they are ultimately going to use them in their day-to-day tasks.</p><blockquote>A persona is not a person itself but it represents a group of people having distinct characteristics and needs</blockquote><p>You may start creating different personas by defining who will be your end-users and providing equal importance for each of their views and expectations of your data product.</p><h4>Self-serve shouldn’t prevent you from selling</h4><p>Most of us truly believe that a 30–45min demo session on the usage of your data product should be good enough to convince your stakeholders and have them get started with it. These sessions often end with a “Please feel free explore the XYZ tool and let us know your feedback on the same”. But unfortunately, that’s not how things work. Let me explain:</p><ol><li>People might have used a different solution (that might look a little less lucrative to you) to solve the same challenges in the past so rigorously that becomes a habit for them. In those cases, it usually takes a lot more convincing to have them transitioned to your new data product.<em> </em><strong><em>For instance, hardcore excel users love playing around with raw data and hence may find tableau dashboards less attractive.</em></strong></li><li>With too many things to do at work, people easily forget, and sometimes it’s your responsibility to emphasize the importance of your new data product by <strong><em>conducting multiple sessions and followups</em></strong>, encouraging them to use more of it, and gathering continuous feedback and work towards it. <em>One effective way to tackle this scenario is by building a tracking mechanism inside your data product and understanding how and whether your stakeholders are using it as expected.</em></li></ol><h4>Build vs Buy</h4><p>This choice is not as simple as it sounds. For every tech-savvy data guy out there, it’s simply a <em>“Need it? then Build it ” attitude. </em>However, having some sense of familiarity with what the market outside has to offer (in terms of data product) is as critical as understanding your internal capabilities.</p><p>Being fully aware of all data products available to solve your business problem is easier said than done. One can leverage tech blogs, webinars, networking, and LinkedIn feeds to stay updated with current market trends. But why is it important to choose between build vs buy? Simply put, It saves you a lot of investment (money and resources), and sometimes it’s just too complex and challenging to build something new that is of enterprise standards.</p><h4>Key Considerations</h4><ol><li>Your data product is just as good as your underlying data. Lack of quality data leads to a bad data product which in turn results in losing your user’s trust in making a data-driven decision.</li><li>It is also important to build a sustainable data product by making it more dynamic and adaptable to any changes or updates.</li></ol><p>If you enjoy reading stories like these and would like to support me as a writer, please feel free to follow me and let me know your thoughts in the comments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d7e754c85e19" width="1" height="1" alt=""><hr><p><a href="https://medium.com/design-bootcamp/why-your-next-data-product-could-be-a-complete-disaster-d7e754c85e19">3 Reasons why your next Data Product could be a complete disaster</a> was originally published in <a href="https://medium.com/design-bootcamp">Bootcamp</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[4 Analytics Frameworks you must know as a Data Analytics Practitioner]]></title>
            <link>https://medium.com/swlh/4-analytics-frameworks-you-must-know-as-a-data-analytics-practitioner-1ec7c4d4603f?source=rss-f6917b2a396d------2</link>
            <guid isPermaLink="false">https://medium.com/p/1ec7c4d4603f</guid>
            <category><![CDATA[business]]></category>
            <category><![CDATA[startup]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Kavikumaran R]]></dc:creator>
            <pubDate>Mon, 29 Nov 2021 08:35:46 GMT</pubDate>
            <atom:updated>2021-11-29T08:35:46.814Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TfcUqW-gwbxv7JCW" /><figcaption>Photo by <a href="https://unsplash.com/@kellysikkema?utm_source=medium&amp;utm_medium=referral">Kelly Sikkema</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>As a data analytics professional, you are required to perform a lot of analysis and experiments for informed decision making and every new problem poses a new challenge that needs to be solved. However different a problem might be, there are proven methodologies to approach them and not to lose sight of the bigger picture, and I’ll try to highlight the same in this blog.</p><h4>What is a Framework?</h4><p>Lately, we have all had enough of this buzzword, “Framework”. Although this particular jargon comes up in our day-to-day discussions every now and then, each one of us has our own definition of it. To get us all on the same page, here’s my take on Frameworks;</p><blockquote>“ A Framework is a set of guidelines or a structure that forms a base of any business or a process. It takes different forms depending upon the context it is used in.”</blockquote><p>For instance,</p><ol><li>C#.Net framework in the software development world is a collection of technologies for building and running applications on Microsoft Windows</li><li>In the business context, it lists the duties and responsibilities (like Brainstorming, Interviewing, Building prototypes, etc.) of a BA professional to ensure the successful completion of a project activity</li></ol><p>In Analytics, a framework helps a startup (or any organization for that matter) to find answers to the following queries at a high level;</p><ol><li>Current stage in the startup lifecycle</li><li>One metric that matters (now!)</li></ol><h4>4 Key Types of Analytical Frameworks:</h4><ol><li>Dave McClure’s Pirate Metrics</li><li>Eric Ries’s Engines of Growth</li><li>Ash Maurya’s Lean Canvas</li><li>Sean Ellis’s Startup Growth Pyramid</li></ol><h4>Dave McClure’s Pirate Metrics</h4><p>Pirate Metrics? Seriously? Now, we just can’t help but wonder,</p><blockquote>What’s that one thing that pirates who are after valuable goods have in common with metrics that basically are critical for a startup’s success? <strong>It’s AARRR!</strong></blockquote><p>A term coined by Venture Capitalist Dave McClure in 2007, <strong><em>AARRR aka Acquisition, Activation, Retention, Revenue, and Referral</em></strong> are the 5 distinctive focus areas that describe the customer journey inside of your product or service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*63XRYVcKH5m56cXtSw3whA.png" /><figcaption>The AARRR Pirate framework</figcaption></figure><h4>Eric Ries’s Engines of Growth</h4><p>In The Lean Startup, Eric Ries emphasizes the importance of 3 key drivers that are critical for a startup environment to thrive; <strong><em>Virality, Stickiness, and Paid Engines</em></strong>.</p><p>While each of these engines is proven to be a great growth mechanism, it’s highly essential for your startup to focus on any one engine at a given time, depending upon the offering or business strategy of yours and the current stage of your startup.</p><h4>Ash Maurya’s Lean Canvas</h4><p>A well-known framework, widely used to validate a business plan and helps in understanding the competition and its positioning in the market. It’s a <strong><em>one-page visual plan</em></strong> that, unlike a traditional business plan, is an ongoing activity requiring continuous updates and hence also known as a ‘<strong><em>living, evolving</em></strong>’ plan.</p><p><em>Note: I will dedicate a separate blog for explaining this framework in much detail.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*UXTDy4RUXUHK6oed.jpg" /><figcaption>Photo from Adobe Stock</figcaption></figure><h4>Sean Ellis’s Startup Growth Pyramid</h4><p>A fairly simple and easy-to-implement framework, vital for Go-To-Market and other growth activities. Coined by serial entrepreneur and marketer Sean Ellis, it primarily <strong><em>focuses on the next steps once the product/market fit is achieved</em></strong>.</p><p>But the challenge here is to figure out how to know whether or not the product/market fit is achieved. Answers to the following questions can help you come to a conclusion on the same.</p><ol><li>What problem of the customer’s does it currently solve and How well it does? <strong>Highlights our Unique Value Proposition</strong></li><li>How many others are solving the same problem? <strong>Competitor Analysis and Understanding our Positioning in the Market</strong></li><li>What impact (both quantitative and sentimental) will it have on our users if we were to shut down our services? The question has to be answered only by the customers and if nearly half of them (40% or above) say that they would be disappointed, you’ve found your fit.</li></ol><p>As you might have understood by now, there are a good number of analytical frameworks available and by no means this blog represents the exhaustive list of them. Also, there is no way to rank which of these frameworks are best as each of them gives a unique perspective of your business’s current standing and growth trajectory. It’s also a common practice among businesses to use a mix of these frameworks.</p><p>If this article piqued your interest and would like to see more such content on analytics, product management, and more, please feel free to follow me! Thanks for reading! :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1ec7c4d4603f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/4-analytics-frameworks-you-must-know-as-a-data-analytics-practitioner-1ec7c4d4603f">4 Analytics Frameworks you must know as a Data Analytics Practitioner</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Intuitive Approach towards Understanding A/B Testing-II]]></title>
            <link>https://medium.com/swlh/an-intuitive-approach-towards-understanding-a-b-testing-ii-250a9dc68802?source=rss-f6917b2a396d------2</link>
            <guid isPermaLink="false">https://medium.com/p/250a9dc68802</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[user-experience]]></category>
            <category><![CDATA[hypothesis-testing]]></category>
            <category><![CDATA[user-testing]]></category>
            <category><![CDATA[ab-testing]]></category>
            <dc:creator><![CDATA[Kavikumaran R]]></dc:creator>
            <pubDate>Wed, 01 Jul 2020 10:58:14 GMT</pubDate>
            <atom:updated>2020-07-02T15:32:13.419Z</atom:updated>
            <content:encoded><![CDATA[<p>In this article, I will show you how A/B Testing is carried out in a planned and controlled manner with a real-world example.</p><p><em>(Quick note: You might want to check out </em><a href="https://medium.com/p/an-intuitive-approach-towards-understanding-a-b-testing-i-71571c5da7a1?source=email-f6917b2a396d--writer.postDistributed&amp;sk=306e60506d62bed4eba61b36e29b3b78"><strong><em>Part 1</em></strong></a><strong><em> </em></strong><em>of this tutorial series which explains the basics of A/B Testing and the key terminologies used.)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*q2sNVmagqC6UeTRSFo0P7Q.jpeg" /><figcaption>Image Source: mlm.com</figcaption></figure><h4>Identify where there is the most opportunity:</h4><p>In any business, there is always room for experimentation and improvement. In fact, there will be plenty of such opportunities. Does that imply all those should be worked upon? Let me remind you that A/B Testing is time-consuming and has costs associated with it and hence a good <strong>ROI (Return on Investment)</strong> is a must to convince the higher management into this. For instance, if you are trying to increase the sales of your e-commerce site and for that, you are buying an ad-space (for say $1000) and see an increase of $250 in sales. Is that effort worth the spend and experiment? The next question would be ‘How do we decide or identify the right opportunity?’</p><p>Try starting from any one of these:</p><ul><li><strong>The top-level goal</strong>: which is the prime objective of your organization such as the number of leads generated, revenue generated, etc.</li><li><strong>Dig into performance metrics</strong>: Investigate your operations and check how well the business has been performing recently across various functional areas and divisions. Start with the one which requires the most attention (<strong>need of the hour</strong>).</li><li><strong>Do user research: </strong>Gather user inputs and feedback on how they like the services provided and what kind of changes would make their experience better.</li><li><strong>Do market research</strong>: Check how your competitors are performing, their new products and services.</li></ul><h4>Know your Business and Customers.</h4><p>Before you begin an experiment, be very sure about how your business currently performs. For instance, if you are running an e-commerce site, you need to have an understanding of the following metrics (not limited to).</p><ul><li>Website Traffic</li><li>Traffic source</li><li>Revenue per Traffic Source</li><li>Average Order value</li><li>Customer Lifetime Value</li><li>New Customers and Returning Customers</li><li>Conversion Rate etc.</li></ul><h4>Select your target metric:</h4><p>While designing an experiment there could be 2 or more metrics that we would be interested in</p><ul><li>Metrics that are directly impacted by the experiment</li><li>Metrics that are at the bottom of the funnel (our top-level goal)</li></ul><p>Let us consider an example of <strong>making the category menu of our e-commerce site more prominent</strong> which would directly impact the number of interactions made by the user. However with increased engagement, one would expect a higher revenue. It’s always better to choose a metric that is directly impacted by the experiment, as choosing our top-line goal might lead to incorrect inferences or in fact, might need a longer duration to execute. In general, the various stages the customers go through on the site are treated as a funnel, and an example of it shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/1*CNmgdcHpcNuRZkHnVcaW3A.png" /><figcaption>Image Source: neilpatel.com</figcaption></figure><h4>Define your Target Audience:</h4><p>While performing experimentation, it is highly essential for us to be more specific about our target audience. For instance, In our e-commerce site example, we might want to define the specific characteristics like geographical region, type of user (new or existing users), age group of users, etc. of the sample of visitors for the experiment to be more impactful.</p><h4>State your hypothesis</h4><p>A scientific hypothesis is just an idea or a proposal that can be tested scientifically. It is something more than a wild guess and less than an established truth. The Hypothesis or Idea must be very clear and concise for the experiment to be successful.</p><blockquote>In our case, <strong>‘Revamping the Category menu of our e-commerce site could lead to an increased conversion rate (Directly impacted metric)’ </strong>could be one way to start.</blockquote><blockquote>Just for the sake of better understanding our target metric selection, let us also test if <strong>‘Revamping the Category Menu leads to increased revenue per customer (Our Top line goal)’</strong>.</blockquote><h3>Testing Conversion Rate Hypothesis with Chi-Square Test:</h3><p>As shown below, we check for statistical significance using <a href="https://www.evanmiller.org/ab-testing/chi-squared.html">Evan’s Awesome A/B Tools Chi-Square Test</a> for the number of clicks made by users. Why the Chi-Square Test? Recalling from the previous article, since we are dealing with clicks made by the user which is either a click or no-click event (boolean outcome) we make use of it.</p><p>We see that there were a total of 300 clicks made by users from the control (sample 1)and 350 clicks made by users from treatment (sample 2)groups respectively. We see that with 95% confidence, the conversion rate of users from the treatment group is sufficiently higher (p=0.017). Ok, what is this p-value here? To know that, Just be a little more patient and keep reading!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HlDkkHSih1KGoK__DYyImg.png" /><figcaption><strong>Evan’s Awesome A/B Tools: Chi-Square Test</strong></figcaption></figure><h3>Testing Revenue per user Hypothesis with 2-Sample T-Test:</h3><h4>Estimating the Sample Size:</h4><p>Suppose you get 1000 visitors for your e-commerce site every day. Out of those 1000 visitors, 200 interact with the category menu and the conversion rate for the same is 40%. The conversion rate of the remaining 800 users who don’t make use of the category menu is 20%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eZ7QD7QtOChHKqob8miLGg.png" /></figure><p>The Overall Conversion Rate of the site is calculated as 24%. Since our Hypothesis is on the number of visitors who make use of the category menu, we can expect a lift on the conversion rate of those 200 visitors only. Let’s say we are anticipating a conversion rate of 80% as a result of the UI change. Our new conversion rate summary looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h1wm-keWDWdHpT6Vc75N1g.png" /></figure><p>We see that the overall conversion rate has increased to 32% from 24%. The percentage difference can be computed as Absolute and Relative difference.</p><p><strong>Absolute % Difference = 32–24 = 8%</strong></p><p><strong>Relative % Difference = [(32–24)/(24)]*100 = 33.33%</strong></p><p>Why do we need to calculate these differences? To calculate the size of the sample that we would require for our experimentation process and here’s how you use the percentage difference to calculate the sample size using <a href="https://www.evanmiller.org/ab-testing/sample-size.html">Evan’s Sample Size Calculator</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*56rtbJnxTssBJa-hJH1kzg.png" /><figcaption>Evan’s Sample Size Calculator (<strong>Absolute</strong>)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AJH4wmyvQS6M6g2M5Lf9YQ.png" /><figcaption>Evan’s Sample Size Calculator <strong>(Relative)</strong></figcaption></figure><p>As we can see, we get the same number of samples while trying with both absolute and relative % lifts. By the way, don’t be bothered about the column chart or any additional detail that you see on the page. However, we need to have an understanding of what <strong>significance level</strong> means.</p><blockquote>Mathematically, confidence level and significance level add up to 1.</blockquote><blockquote><strong>confidence level = 1-significance level</strong></blockquote><blockquote><strong>(or) significance level = 1-confidence level</strong></blockquote><p>So a significance level of 5% displayed on the Sample Calculator (as shown above) implies that the confidence level is 95%. Anyone who has a little bit of introduction to the hypothesis testing must have come across this phrase.</p><blockquote>If the P <strong>value</strong> is less than your <strong>significance</strong> (alpha) <strong>level</strong>, the hypothesis <strong>test</strong> is statistically <strong>significant</strong>.</blockquote><p>Back to the cricket example but in this scenario, there are 4 friends (say A, B, C, and D) playing together and wants to decide who should keep wickets. Each of their names is written on a piece of paper and put inside a bowl and randomly one piece of paper is drawn out of it to decide who is gonna keep wickets.</p><p>Let’s say in the first three matches, A never got to keep wickets as the paper with his name was never drawn from the bowl. The question is,</p><blockquote><strong>Did that happen by chance? Or is something not right?</strong></blockquote><p>Our Null Hypothesis or Status Quo is <strong>‘The draw of names made from the bowl is a fair event’</strong></p><p>Let us test it out by using probability.</p><p>P(A not getting chance to keep wickets) = 3/4</p><p>P(A not getting chance to keep wickets in 3 matches)</p><p>= (3/4) * (3/4) * (3/4) =27/64 = 0.421</p><p>From the above probabilistic calculation, we see that there is a 42% chance of A not getting to keep wickets at all in the first 3 matches which <strong>seems fair enough</strong>.</p><p>What if A never gets to keep wickets in all 10 matches played? Can it also happen by chance?</p><p>P(A not getting chance to keep wickets in 10 matches)</p><p>=(3/4)¹⁰=0.056 which is 5.6%.</p><p>Assuming the Significance level as 5%, we still conclude that A not getting to keep wickets in all his 10 matches is <strong>by chance and is fair enough since p-value (5.6%) &gt; significance level (5%).</strong></p><p>if A never gets to keep wickets in all 15 matches played then</p><p>P(A not getting chance to keep wickets in 15 matches)</p><p>=(3/4)¹⁵=0.0133 which is 1.3%.</p><p>In this case, the chances of A not getting to keep wickets <strong>doesn’t seem like happening by chance since p-value(1.3%) &lt; Significance level (5%). Hence we reject the Null Hypothesis.</strong></p><blockquote><strong>Significance level, in general, is a threshold set by the statistician to draw conclusions from the Hypothesis Testing. A threshold of 5% is arbitrarily defined for most cases and for few cases a significance level of 1% is also used.</strong></blockquote><blockquote><strong>the p-value, on the other hand, refers to the probability of obtaining test results at least as extreme as the results actually observed. In our example, the probabilities of player keeping wickets we calculated are p-values.</strong></blockquote><h4>How long should we run the experiments?</h4><p>For our example, we calculated the sample size required for each variant (Control and Treatment)to be 460. Since we know that the number of visitors to the site every day is 1000, does that mean running the test for a day would suffice?</p><p>Often, the number of visits to the site varies from each weekday and one has to be aware of this trend. Hence the <strong>experiment has to be carried out for a week at least</strong> to reduce the effect of variability caused by weekdays. It is also crucial to ensure that all other factors surrounding the control and treatment groups remain constant. That is, there shouldn’t be any promotion events or big sale events happening during the time of experimentation as it would introduce a lot of probabilities.</p><h4>Understanding the right usage of A/B tests:</h4><p>Once the experimentation process is complete, checking for statistical significance is made simple using <a href="https://www.evanmiller.org/ab-testing/t-test.html">Evan’s Awesome A/B Tools: Two-Sample T-Test</a>, since we are experimenting on revenue generated which is a continuous value. You can either paste your raw sample data or the summary of it by selecting an appropriate option in the tool. In our example, we have revenue generated by each user in both control and treatment groups, all we have to do is check if their average values differ significantly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L4uArypNWwSUtYEAIJGbRQ.png" /><figcaption><strong>Evan’s Awesome A/B Tools-2 Sample T-test</strong></figcaption></figure><p>As shown above, the mean, standard deviation, and count(number of samples) for Sample 1(Control) and Sample 2(Treatment) are given as input. The hypothesis selection d=0 implies that our Null hypothesis or Status Quo is ‘There is no significant difference or the average of both samples is equal’.Once we have all the inputs in place, we will have the verdict (which is <strong>‘No significant difference’</strong> in our example).</p><blockquote><strong>As discussed earlier, it is always best to select the metric that is directly impacted by the change as a Target metric instead of going for our top-line goal. If at all you want to experiment with the top-line goal only, you may need a larger sample size to get anticipated results.</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TlvbE9HfaYyn4GJLluh9KQ.png" /><figcaption><strong>2 Sample T-test with a higher sample size</strong></figcaption></figure><h4>Interpreting Results:</h4><p>Remember, Statistics is just a tool for gathering evidence and not the ultimate truth.</p><blockquote>“There are three types of lies — lies, damn lies, and statistics.”<br>― <strong>Benjamin Disraeli</strong></blockquote><p>Even after you have conclusive evidence on your new feature or variant performing significantly well, you would still need to devise a proper execution plan to go further. Review meetings with cross-functional teams and having decision plans place would also help in eliminating bias during the decision-making process.</p><h4>EndNote:</h4><p>I know this might be a lot of information to take in but, we haven’t discussed in detail what a Chi-Squared test or 2 Sample T-Test is for a good reason. The main purpose of this article is to educate professionals who are not well versed with Stats to leverage A/B testing for making better business decisions and validating the same.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=250a9dc68802" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/an-intuitive-approach-towards-understanding-a-b-testing-ii-250a9dc68802">An Intuitive Approach towards Understanding A/B Testing-II</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Intuitive Approach towards Understanding A/B Testing-I]]></title>
            <link>https://medium.com/swlh/an-intuitive-approach-towards-understanding-a-b-testing-i-71571c5da7a1?source=rss-f6917b2a396d------2</link>
            <guid isPermaLink="false">https://medium.com/p/71571c5da7a1</guid>
            <category><![CDATA[user-engagement]]></category>
            <category><![CDATA[user-experience]]></category>
            <category><![CDATA[ab-test]]></category>
            <category><![CDATA[experimentation]]></category>
            <category><![CDATA[ab-testing]]></category>
            <dc:creator><![CDATA[Kavikumaran R]]></dc:creator>
            <pubDate>Wed, 24 Jun 2020 09:52:48 GMT</pubDate>
            <atom:updated>2020-07-02T15:32:11.448Z</atom:updated>
            <content:encoded><![CDATA[<p>In our day-to-day life, be it personal or at work, we are always presented with a lot of options to choose from. And we always tend to believe that we have made the best possible choice. But have we?</p><h3><strong>What’s so unique about this article?</strong></h3><p>There are many articles out there that clearly explains everything about A/B Testing, so why this? People tend to shy away from learning as soon as they see this word. <strong>STATISTICS!</strong> Here’s a simpler and precise explanation of it without including any fancy jargon or Greek alphabets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*6oLN3TnonAOvLQyV7t5ylw.png" /><figcaption>Image by JB Shreve from theendofhistory.net</figcaption></figure><p>Does that mean there are no statistics covered in this article? Yes and No. We are just going to use the term <strong>INTUITION</strong> in place of statistics and better understand the mechanics of A/B Testing from the scratch.</p><h3><strong>What is A/B Testing?</strong></h3><p>In short, we are presented with 2 choices A and B(<strong>known as ‘variants’</strong>) out of which we have to figure out the best-suited option for our use case. It’s a relatively straightforward concept which aids the <strong>Conversion Rate Optimisation (CRO) </strong>and Funnel Analysis in most businesses<strong>. </strong>In general, A/B Testing is about introducing a new feature or conducting a campaign and analyzing whether or not it had the expected impact on the target audience in a controlled way. By the controlled way we mean, the users in the Control and Treatment group are randomly selected which eliminates the bias in the experimentation process, and all other factors (<strong>known as guardrail metrics</strong>) that may influence our Target Metric are maintained constant. Not familiar with what a Control and Treatment group means? Keep reading!</p><h3><strong>Understanding the Terminologies:</strong></h3><p><strong>Control Group:</strong> refers to a segment of users who are shielded from the experiment/ variant (<strong>often known as the placebo group</strong>). Why do we need it? It acts as a benchmark to measure the success of the test and eliminates bias in decision-making process.</p><p><strong>Treatment Group:</strong> refers to the segment of users who are subjected to the experiment/ new variant.</p><p><strong>Variance:</strong> is the measure of spread between the numbers in a dataset. Ahh! Let’s make it sound a little less statistical.</p><p>Consider the following scenario of determining the better player amongst the two:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EgSvKanH13biFSUzHUGOBQ.png" /></figure><p>For comparison sake, the total scores of both the players in our data add up to 490. But does it really look like both players have been in the same form? Give it a thought and see what differentiates both these players. <strong>Consistency</strong> it is!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*XywzaJoysD4fkmRbuGwu9A.png" /></figure><p>Both players share the same average of 49.0 in 10 matches, but the difference between scores made each match and their average score <strong>(quantified as Standard Deviation) </strong>is high for Player 2 which is definitely not a good sign. Player 1 has managed to score 30+ scores in all his matches whereas Player 2’s performance is kind of unpredictable. One can argue that Player 2 still manages to make big scores which Player 1 didn’t. But for understanding what variance is we shall agree that Player 1 is a more bankable player.</p><p><strong>Confidence and Sample Size:</strong></p><p>To make this more interesting, Here comes Player 3 with match scores 10, 120. Now is it right to compare Player 3 having played just 2 matches with the other 2 players? We may never know how Player 3 will perform in the upcoming matches and from our example, we would require <strong>at least 10 match scores (sample size) </strong>to do a comparison and arrive at a conclusion. Let’s dive deeper into this for better understanding.</p><p>Considering only the first 2 scores made by Player 2 would lead to an average score of <strong>39 (calculated as (66+12)/2)</strong> which is a little lower than the average calculated using 10 match scores. On including the third match score (82) the average rises to 53. The key takeaway for us here is that <strong>with more matches played the fluctuation in the average score made by the player tends to stabilise</strong> which in-turn helps us to gain <strong>confidence</strong> in the test results we get at the end.</p><blockquote>Now here is the question for you, In a Cricket tournament, there is Team A which has won 2 of 2 matches and Team B which has won 20 of 20 matches. Which of these two teams are impressive? I’m hoping you would agree that it is Team B. Hence the key takeaway here is</blockquote><blockquote><strong>“With smaller sample size there is more variability and less confidence and with larger sample size there is less variability and more confidence”</strong></blockquote><h4>Understanding your Target Metric:</h4><p>A Target Metric is simply the change that you want to witness as a result of your experiment or the campaign. It can be the click-through rate of your marketing emails, revenue generated in an e-commerce site, etc. Basically, your target metric can be either continuous or discrete in nature. Not so clear?</p><blockquote>Consider the first scenario where you are basically trying out a new email template or a subject line for marketing. In order to quantify the impact of the change, we just have to figure out the number of clicks and no clicks but nothing else.</blockquote><blockquote>The Yes/No kind of an outcome is termed as a Discrete or Boolean outcome</blockquote><p>On the Other hand, Suppose you want to see if there is an increase in the revenue generated by your e-commerce website after introducing a UI change for better navigation. In this scenario, the revenue generated could be any number, and range of it may also vary depending upon the size of business.</p><blockquote><strong>Why is it so important to understand this difference?</strong></blockquote><blockquote><strong>It kind of decides what statistical test should be carried to check and quantify the significance of the impact. One can also decide beforehand how much of that impact is expected at the end of the experiment.</strong></blockquote><p>In my next <a href="https://medium.com/@kavikumaran/an-intuitive-approach-towards-understanding-a-b-testing-ii-250a9dc68802?source=friends_link&amp;sk=1728061208efd9a0614e09b187dd436b">post</a>, we shall see</p><ul><li>An end to end flow on ‘How A/B Testing is carried out’ with an example</li><li>Best practices of A/B Testing</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=71571c5da7a1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/an-intuitive-approach-towards-understanding-a-b-testing-i-71571c5da7a1">An Intuitive Approach towards Understanding A/B Testing-I</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>