<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pritul Patel on Medium]]></title>
        <description><![CDATA[Stories by Pritul Patel on Medium]]></description>
        <link>https://medium.com/@pritulp?source=rss-89c0c9586b74------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*6HwOpjpm-RdSmEO3lPhUBA.png</url>
            <title>Stories by Pritul Patel on Medium</title>
            <link>https://medium.com/@pritulp?source=rss-89c0c9586b74------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 09:42:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pritulp/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How the Online Sample Size calculators are silenty limiting and killing your Experimentation…]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/@pritulp/how-the-online-sample-size-calculators-are-silenty-limiting-and-killing-your-experimentation-b90911559c30?source=rss-89c0c9586b74------2"><img src="https://cdn-images-1.medium.com/max/704/1*OlyQI7XowUbYQnfgHw0kUA.png" width="704"></a></p><p class="medium-feed-snippet">Would you trust a doctor who brings just the thermometer to diagnose every ailment?</p><p class="medium-feed-link"><a href="https://medium.com/@pritulp/how-the-online-sample-size-calculators-are-silenty-limiting-and-killing-your-experimentation-b90911559c30?source=rss-89c0c9586b74------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://medium.com/@pritulp/how-the-online-sample-size-calculators-are-silenty-limiting-and-killing-your-experimentation-b90911559c30?source=rss-89c0c9586b74------2</link>
            <guid isPermaLink="false">https://medium.com/p/b90911559c30</guid>
            <category><![CDATA[conversion-optimization]]></category>
            <category><![CDATA[experimentation-culture]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[experimentation]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Pritul Patel]]></dc:creator>
            <pubDate>Thu, 10 Apr 2025 15:12:08 GMT</pubDate>
            <atom:updated>2026-08-10T15:19:32.734Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[The Truth About Holdout Groups in A/B Testing]]></title>
            <link>https://medium.com/@pritulp/the-truth-about-holdout-groups-in-a-b-testing-a1a13940a92c?source=rss-89c0c9586b74------2</link>
            <guid isPermaLink="false">https://medium.com/p/a1a13940a92c</guid>
            <category><![CDATA[experimentation]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[conversion-optimization]]></category>
            <category><![CDATA[holdouts]]></category>
            <dc:creator><![CDATA[Pritul Patel]]></dc:creator>
            <pubDate>Wed, 05 Mar 2025 23:08:33 GMT</pubDate>
            <atom:updated>2026-08-07T13:45:12.465Z</atom:updated>
            <content:encoded><![CDATA[<h4>Why Experimentation Holdouts May Be Fundamentally Flawed</h4><p>In the world of digital experimentation and A/B testing, holdout groups have become a popular method for measuring long-term impacts of experiments or entire programs. But despite their prevalence, there are serious questions about their validity and usefulness. Having worked with numerous experimentation teams, I have come to a controversial conclusion - <em>holdout groups are often fundamentally biased from the start</em>.</p><h3>Understanding the Three Types of Holdout Tests</h3><p>Before diving into their flaws, let meclarify what holdout tests actually are.</p><ol><li><strong>Experiment Holdouts</strong>: Here I am essentially talking about <strong>reverse tests</strong> conducted after an experiment concludes, testing the opposite of what was proven in the forward test.</li><li><strong>Feature Holdouts</strong>: These combine winners from multiple experiments related to a specific feature (e.g., personalization) and hold back traffic from experiencing those improvements.</li><li><strong>Universal Holdouts</strong>: A more comprehensive approach where a segment of users is completely shielded from all experiments and improvements.</li></ol><p><strong>EDIT: </strong>Holdout tests BEFORE any experiment launch, or release is still a good practice. Say you start a holdout on Jan 1 to measure everything you’d be doing for the rest of the quarter or year, will atleast have a utility value in quantifying effect of stacking the changes. In the rest of the post, I’ll be talking about the questions business as after the fact, which is more often than you’d anticipate.</p><h3>The Inherent Bias Problem</h3><p>The central issue with holdout tests lies in their methodology and implementation. Here is why they are problematic.</p><p><strong>Timeframe and Traffic Allocation Discrepancies</strong></p><p>In a typical scenario, you might run a forward A/B test for 4 weeks with a 50/50 traffic split, then spend a week setting up a holdout, followed by running the holdout for 13 weeks (since holdouts typically run longer to measure long-term effects).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ekSt8PGmrNiQr6pwZ_ptPA.png" /><figcaption>Holdout group timelines in forward and holdback setting</figcaption></figure><p>But unlike your original test’s 50/50 split, holdouts usually allocate only about 5% of traffic to the holdout group and 95% to the treatment group.</p><p>While your 5% holdout group remains static, the other 95% of traffic continues experiencing new experiments and improvements. This means your holdout is no longer representative of the conditions during your original test — it is comparing against a constantly evolving experience.</p><p>This fundamental difference creates several problems.</p><p><strong>A. The Reshuffling Dilemma</strong></p><p>When creating a holdout, you face an impossible choice.</p><ul><li><strong>Option 1</strong>: Reshuffle all users and start a new holdout experiment (very common)<br>Problem: Some users who experienced the test variant will now be in the holdout group, creating bias as they are suddenly deprived of features they have grown accustomed to. Example — if the forward test was 50/50 split, and the new holdout setup is 5/95, then the 5 is actually having equal representation of the previous 50/50 split (if reshuffling works as expected). So the 5% holdout has 2.5% users from previous Control AND the remaining 2.5% users from previous Treatment. These 2.5% (HALF of the holdout variant) will be deprived of the feature that had received in the forward test. So the holdout is biased from the get go.</li><li><strong>Option 2</strong>: Keep the original 5% of control users in the holdout. <br>Problem — This means these users will be continuing to not see the new feature for very long time. If the holdout is Feature or Universal holdout, then all the new features launched in prod will be held back from these users for very long time. This creates what I call “The Interstellar Problem.”</li></ul><p><strong>The “Interstellar Problem”</strong></p><p>In the movie Interstellar, the main character experiences time differently than everyone else. When a holdout experiment ends after, say, 13 weeks, the holdout users suddenly experience a massive influx of changes that everyone else received gradually over that period.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v79qul-6wfh_zeK93vkkqQ.png" /></figure><p>This creates a shock effect — these users haven’t evolved with your product and will behave differently than your regular users, rendering post-holdout measurements equally suspect.</p><h3>The Technical Implementation Challenge</h3><p>Setting up proper holdouts is technically challenging, especially with third-party experimentation platforms. You may need to:</p><ul><li>Repurpose experiment IDs</li><li>Reduce sample sizes</li><li>Configure mutually exclusive conditions</li><li>Layer additional experiments correctly</li></ul><p>This complexity often leads to implementation errors that further compromise validity.</p><h3>Questions Worth Asking</h3><p>Before implementing holdouts, consider:</p><ol><li>Why don’t you trust your forward test results enough that you feel compelled to run reverse tests after every winner? The exceptions are you need to collect long term data for this cohort, in which case, don’t rely on statistical measures. Just track descriptive statistics of the cohort but the data is not conducive for inferential statistics.</li><li>Do you have the infrastructure necessary to properly implement feature or universal holdouts? (It is a significant undertaking.)</li><li>What are the business costs of holding back improvements from a segment of your users? For example, advertising teams often use holdouts to measure ad program impact — but at what cost to revenue and user experience?</li></ol><h3>Better Alternatives for Measuring Long-Term Impact</h3><p>If holdouts aren’t ideal for understanding long-term impacts, what alternatives exist? While some practitioners may insist that A/B testing remains the “gold standard,” some other approaches can provide meaningful insights.</p><ol><li><strong>Causal Inference methods</strong>: Techniques like Bayesian time structural series analysis can examine impacts before and after an intervention. Other methods like difference-in-differences (DiD) or regression discontinuity can also provide causal insights without the biases of holdouts. They are not the gold standard though.</li><li><strong>Cohort analysis</strong>: Following control and treatment cohorts offline over time can yield valuable long-term insights. However, it is worth acknowledging that these cohorts will inevitably be affected by future experiments, making complete isolation impossible.</li><li><strong>Extended experiment duration</strong>: Sometimes the simplest solution is best — if you are concerned about long-term effects, consider running your original A/B test for a longer period rather than rushing to a conclusion and then trying to validate with a flawed holdout.</li></ol><p>It is important to recognize that the pursuit of perfect long-term measurement can become a distraction that adds friction to your experimentation program. Teams can get bogged down in measuring past experiment value instead of running new experiments and generating fresh insights.</p><h3>Conclusion</h3><p>While holdout testing remains popular, my experience suggests we should be much more skeptical of the results these tests provide. The methodological biases, technical challenges, and fundamental asymmetry between holdout groups and the rest of your user base make it difficult to draw reliable conclusions.</p><p>Instead of relying on inherently flawed holdout methodologies, consider investing in more sophisticated causal inference techniques, better forward test design, and improved metrics that can capture long-term impacts without the need for problematic holdout groups.</p><p>What has your experience been with holdout testing? Have you found ways to overcome these inherent biases, or have you moved away from holdouts altogether?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a1a13940a92c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Internet Might have Never Existed]]></title>
            <link>https://medium.com/@pritulp/internet-might-have-never-existed-7ef0001a3c91?source=rss-89c0c9586b74------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ef0001a3c91</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[communication-technology]]></category>
            <category><![CDATA[internet]]></category>
            <category><![CDATA[history-of-technology]]></category>
            <dc:creator><![CDATA[Pritul Patel]]></dc:creator>
            <pubDate>Sat, 05 Oct 2024 20:01:20 GMT</pubDate>
            <atom:updated>2024-10-05T20:01:20.536Z</atom:updated>
            <content:encoded><![CDATA[<p>The fascinating story of the birth of the Internet</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Co4VL__S4mcPVaHp" /><figcaption>Photo by <a href="https://unsplash.com/@yapics?utm_source=medium&amp;utm_medium=referral">Leon Seibert</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>April 1968. A very bold article was published in the Science and Technology journal by two men, <strong>Joseph Carl Robnett</strong> <strong>Licklider</strong> and <strong>Robert Taylor</strong>. The first line of the article read “In a few years, men will be able to communicate more effectively through a machine than face to face.” The title of the article “The Computer as a Communication Device”.</p><p><em>I also fed this article to NotebookLM to create a 2 person podcast episode. The AI generated podcast episode has turned out pretty well. Its not perfect and there are issues but those are minor.</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FhsNmbSbLn_0%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DhsNmbSbLn_0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FhsNmbSbLn_0%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/674a916e94c2b85d7acd9b0dfaa3cfb6/href">https://medium.com/media/674a916e94c2b85d7acd9b0dfaa3cfb6/href</a></iframe><p><strong>Lets step back a few years into early 60s</strong></p><p>At that time, telephone lines (landlines) had become very ubiquitous as a communication device in the United States. Just 10 years before this article was published, 75% of households in the US had a telephone. Meaning most people had become quite comfortable with the idea of communicating over long distance using telephones with the voice. The company responsible for such a wide scale adoption of telephones in the US was AT&amp;T (American Telephone and Telegraph company). This was the subsidiary of Bell Telephone company, yes Bell telephone company started by Alexander Graham Bell, who is often credited for the invention of the telephone. Bell Telephone company had multiple companies — Bell Labs for technology research, Western Electric as their telephone and other commercial equipment manufacturing arm related to telephone infrastructure, like switches.</p><p>While Western Electric provided hardware, Bell Labs came up with telecommunication methods that made possible voice telephones from coast to coast. One of the most widely adopted methods for making a call between two people was made possible with a connection method called “Circuit switching”.</p><h4>What is circuit switching?</h4><p>Imagine you want to host a private event like a birthday or wedding. You reach out to a hotel and book a large hall that they provide on rent for holding such events. The rental charge is huge but it includes dedicated services like food, decor, waiting staff and services, DJ, beverages, security/gating of guests so that only authorized guests can enter. Now imagine this is the only hall they have. Now I call them up and ask to book the same hall for the same date and time that you’ve already booked. They’ve already dedicated full resources to your event so they’ll tell me sorry but either postpone your event or go elsewhere. So in circuit switching when one person initiates a call to speak to another person on a telephone it is basically like dedicating a single line of circuit between two phone numbers. The switching part is basically all the resources like chef, decorator, waiting staff, DJ, beverages, etc. Even if one of them is missing or failed to show up it will screw up the entire event. Similarly, a circuit gets completed by several different switches that are between the two phone lines and even if one switch is busy or failed, the circuit cannot be completed and the call cannot be made. The exact switches between a given two phone numbers was always the same. Meaning, everytime same two people talked, the circuit will be established by the exact same switches in the path between the two. The best advantage of circuit switching is that as long as the circuit is established, all the resources and bandwidth are dedicated to that circuit. Someone else tries to connect the same person or even trying to use the same switch beyond a load they will NOT be able to interrupt established line. This made sure that the quality and reliability of call was amazing.</p><h4>Cold War</h4><p>1950s and 60s were the early decades that the US was going through a cold war with Russia. This required that any communication between military bases be extremely flawless. As I told you, calls between two people going through the exact same switches posed a huge threat. If the enemy found out the exact location of the switches that facilitates connection to military bases and destroys it, the military would suddenly become blind.</p><p>This problem was amplified by <strong>Paul Baran</strong>, an engineer researching communication networks at the RAND corporation, one of the major defense contractors for the US. He pointed out that not only the circuit switching system was vulnerable to attack, but in the event of a switch failure the call cannot be immediately completed by another switch in the network, ie. <strong>there is no redundancy</strong>. He also pointed out all the inefficiencies and scalability issues with the system.</p><p>He wanted to prove his idea of <strong>network survivability</strong> in the event of an attack. So he took advantage of AM radio network nodes. He created a simulation where multiple connections are established through nodes of a radio network. The simulation then randomly “killed” some nodes and subsequently tested the percentage of nodes that remained connected. The simulation showed that if a node is connected with 3 or more other nodes then killing even half the nodes would make sure the system kept functioning showing <strong>resilience</strong> through <strong>redundancy</strong>. This proved that a distributed network was survivable.</p><p>This was a huge proof of concept and a foundation to many modern computing principles. He also proved that the nodes need not be as expensive equipment as circuit switching equipment was. <strong>Circuit switching required some of the components to be gold plated.</strong> The Distributed Network that Baran introduced was intended to route around damage. It provided connection to others through many points, not one centralized connection. Fundamental to the scheme was the division of the information into “blocks” before they were sent out across the network.</p><p>Around 1964 he was trying to explain all this to AT&amp;T engineers to make their infrastructure more resilient to all the downsides of circuit switching <strong><em>but they completely ignored him.</em></strong></p><h4>Packet Switching</h4><p>Around this time, completely independent of Baran’s work, another person named Donald Davies, from a government lab called “National Physical Laboratory” in the UK also proposed the same method and called it as sending “packets” over the network. This created a method called “packet switching”. I will explain packet-switching but first don’t confuse <em>this</em> computer network with the internet. Already at this time there were several small local isolated computer networks created that involved connecting computers to mainframe computers or connecting to other local computers in a very short distance (same room) but not farther.</p><p>Now let me explain what packet switching is. Imagine you have to parcel a chair from your city to a city 1000 miles away. You take the chair to your local post office and say you want to send this chair to x address. The post office looks at the size of the chair and says okay we will send it. Since the size is too big, post office will not send entire chair in one parcel. So they dismantle the chair into 4 legs, seat, and back rest — total 6 parts. Then wrap up each part individually and puts destination address and sent to their dispatch. The dispatch doesn’t know that all 6 packs are part of one chair however, the label on each has more details noted, like the part number out of total parts etc. The dispatch takes first few packages and sends them via a truck. The truck is full so they wait for next day to send via another truck the rest of the 2 parts. The first truck goes to another city and exchanges good with another truck that is supposed to go further. The second truck is going to go directly to the destination. After 3 days, the second truck reaches the destination post office. Parts 3 and 5 arrive before every other parts. The post master can read the label and understand that each of these are part of other items shipped so they keep it on the side. After 2 more days go by, the remaining parts arrive. Now post master knows the part numbers, they open up and assemble and deliver the chair to the final destination.</p><p>Similarly, packet switching involves breaking down data into small packets. So an email or image or document can be broken down into packets with each packet containing the header instructions for origin-destination and the sequence in which the packets were broken and needs to be assembled. Each packet may take different routes and reach the final destination out of order but the instructions in the header contains assembly instructions so the message is fully delivered as intended.</p><p>In 1967, Donald Davies’ work of “packet switching” caught the eye of ARPANET researchers. ARPA issued an RFQ to build such a system and the contract was awarded to a research lab and also a defense contractor BBN (<a href="https://en.wikipedia.org/wiki/BBN_Technologies">Bolt, Beranek and Newman</a>).</p><p>Now combining the work of Paul Baran and Donald Davies, and their original attempt to solve a problem of communicating between people with more than just voice without having them to be in the same room, <strong>JCR Licklider</strong> and <strong>Robert Taylor</strong> from ARPA published their article in 1968 “The Computer as a Communication Device”.</p><h4><strong>So how did everything come into practicality?</strong></h4><p>This is where the role of BBN and <strong>Leonard Kleinrock</strong> comes into picture. Leonard Kleinrock is a faculty and professor at UCLA. He is best-known for his most-significant work on queueing theory. Queuing theory is the branch of mathematics that studies waiting in lines or queues. He analyzed delays in a message switching system operated by Western Union for processing telegrams. He published his thesis and also a book on this topic.</p><p>BBN were awarded the contract to build the Interface Message Processors (IMPs), which were the packet-switching nodes of ARPANET. The IMPs are the initial version of what we know today as “Routers”. They also helped in the development of software to manage data transmission between these nodes, and handling the initial network setup and testing.</p><p>Leonard Kleinrock and his team at UCLA focused on the theoretical underpinnings of packet-switching technology which was incorporated into building Network Control Protocol (NCP). NCP is the precursor to what is known as TCP/IP protocol today.</p><p>NCP was the first host-to-host communication protocol for ARPANET, implemented in 1970. It allowed different types of computers on the network to communicate reliably, handling issues like establishing and maintaining a connection between host computers, data transfer, and controlling the flow of data so that slower devices weren’t overwhelmed by faster ones. NCP provided the rules and standard procedures for sending and receiving data on ARPANET. It operated above the IMP level, directly on the host computers.</p><p>Klienrock established the first ARPANET host at the main building of UCLA which connected to a second host in Stanford Research Institute (about 350 miles north).</p><p>On Oct 29th, 1969 at 10:30pm pacific time, UCLA undergraduate student, Charley Kline, who was supervised by Kleinrock, sent the first message ever over the ARPANET. They chose a word “LOGIN” to be transmitted from UCLA computer to SRI’s computer. Each letter was entered one at a time and would be received by SRI computer and acknowledge the receipt back to UCLA for next letter. Letters L and O were received and acknowledge by SRI and the third letter G was sent from UCLA but the system at SRI crashed due to a software bug that caused memory overflow. It took them about an hour to reboot, identify, and fix the bug and again attempted to send LOGIN. This time the system worked like a charm.</p><p><strong>And this is how the Internet was born.</strong></p><p>After a month, a permanent node was established between these two places for exchange of ideas and information. Then it was expanded to 4 nodes adding two more university research labs (UCSB and University of Utah). The beauty of all of these 4 nodes is that none of them had same computer or operating system yet the message transfer could be made possible without crashing the system. This is possible through amazing invention of the IMPs (routers) and NCP.</p><p>Today the Internet has proliferated over billions of networks of routers and uses TCP/IP protocol to communicate between networks and devices.</p><h3>About the author</h3><p><em>My name is Pritul Patel. I am a Data scientist with 15+ years of experience in Product analytics and building world-class Experimentation platforms for big tech companies like Apple, eBay, Yahoo, and Peacock TV. I have a knack for understanding complex topics in a manner I can explain to anyone. I read the history and the motivation behind an invention when the topics get confusing. Feel free to </em><a href="https://www.linkedin.com/in/pritul-patel"><em>connect with me on LinkedIn</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ef0001a3c91" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Are your A/B test results “neutral” experiment after experiment?]]></title>
            <link>https://medium.com/@pritulp/are-your-a-b-test-results-neutral-experiment-after-experiment-d94379428b05?source=rss-89c0c9586b74------2</link>
            <guid isPermaLink="false">https://medium.com/p/d94379428b05</guid>
            <category><![CDATA[experimentation]]></category>
            <category><![CDATA[measurement]]></category>
            <category><![CDATA[conversion-optimization]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <dc:creator><![CDATA[Pritul Patel]]></dc:creator>
            <pubDate>Mon, 30 Sep 2024 14:41:09 GMT</pubDate>
            <atom:updated>2024-09-30T14:41:09.579Z</atom:updated>
            <content:encoded><![CDATA[<p>OEC might be the culprit here</p><p>Overall Evaluation Criteria (OEC) are a set of key metrics used to assess the success of experiments and business initiatives. In experimentation, OEC serve as a critical framework for decision-making, helping teams determine whether a change should be implemented or not. These criteria typically encompass multiple dimensions of business performance, such as user engagement, revenue, and long-term customer value. The importance of OEC lies in its ability to provide a holistic view of an experiment’s impact, aligning short-term actions with long-term business goals. By using OEC, organizations can avoid the pitfall of optimizing for a single metric at the expense of overall business health.</p><p>However, OEC is not always the best measure to optimize for in every experiment.</p><figure><img alt="A two-panel cartoon metaphor for misaligned Overall Evaluation Criteria (OEC) in business. Top panel: Sea creatures at a race starting line — octopus, eel with shrimp, crab, and pufferfish. Bottom panel: Race started, each moving ineffectively — octopus upward, eel downward, shrimp backward, crab sideways, pufferfish inflated. Represents how poorly chosen OEC can lead to actions that don’t progress toward business goals, with each metric (creature) excelling in its own direction without overall" src="https://cdn-images-1.medium.com/max/550/1*3qV2nVJI-N_T0ajOcIHAMA.png" /></figure><p>If the chosen metrics within OEC are misaligned the outcome might just be “neutral”, experiment after experiment.</p><p>Many companies adopt “launch at neutral” policy leading to risk-aversion behavior. This will ultimately be detrimental to the overall outcome of experimentation program for several reasons:</p><ol><li><strong>False sense of progress — </strong>It can create an illusion of productivity, while your company is essentially treading water.</li><li><strong>Opportunity costs — </strong>Resources are spent on implementing and maintaining neutral changes instead of pursuing potentially impactful initiatives.</li><li><strong>Increased complexity — </strong>Each neutral launch adds to product or service complexity without clear benefits, potentially degrading user experience.</li><li><strong>Missed optimization opportunities — </strong>It may prevent teams from iterating further to find truly positive improvements.</li><li><strong>Risk of cumulative negative effects — </strong>Multiple neutral changes could have unforeseen negative interactions over time.</li><li><strong>Complacency — </strong>It can foster a culture that settles for mediocrity rather than striving for significant improvements.</li><li><strong>Misaligned incentives — </strong>Teams might focus on launching rather than creating real value.</li><li><strong>Data noise — </strong>Accumulating neutral changes can make it harder to detect meaningful signals in future experiments.</li><li><strong>Neglect of negative signals — </strong>Subtle negative impacts might be overlooked if they fall within the “neutral” range.</li><li><strong>Stifled innovation — </strong>It may discourage bold, potentially breakthrough ideas in favor of safe, incremental changes.</li></ol><p>I have seen all of this at work before we revisited the overall business outcomes and its correlation with launch policies of the experiments ran.</p><p>This approach can ultimately lead to stagnation and a failure to achieve meaningful progress or competitive advantage.</p><p>Sometimes, optimizing for just one critical metric can drive significant progress and breakthroughs. Some experiments are designed to test specific features, or need shorter data collection loops. To push the business in a specific direction, it may be necessary to prioritize and optimize for metrics that align with that particular goal, even if they’re not part of the standard OEC.</p><p>Adhering too rigidly to the “best practice” of always using comprehensive OEC can stifle innovation and limit strategic shifts.</p><h3>About the author</h3><p><em>I am a Data scientist with 15+ years of experience in Product analytics and building world-class Experimentation platforms for big tech companies like Apple, eBay, Yahoo, and Peacock TV. I have a knack for understanding complex topics in a manner I can explain to anyone. I read the history and the motivation behind an invention when the topics get confusing. Feel free to </em><a href="https://www.linkedin.com/in/pritul-patel"><em>connect with me on LinkedIn</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d94379428b05" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part 2 — The True History of P-values and Hypothesis Testing: What Every A/B Testing Practitioner…]]></title>
            <link>https://medium.com/@pritulp/part-2-the-true-history-of-p-values-and-hypothesis-testing-what-every-a-b-testing-practitioner-baf7fd50f7e2?source=rss-89c0c9586b74------2</link>
            <guid isPermaLink="false">https://medium.com/p/baf7fd50f7e2</guid>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[experimentation]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[conversion-optimization]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Pritul Patel]]></dc:creator>
            <pubDate>Thu, 11 Jul 2024 01:56:06 GMT</pubDate>
            <atom:updated>2024-07-11T01:56:06.289Z</atom:updated>
            <content:encoded><![CDATA[<h3>Part 2 — The True History of P-values and Hypothesis Testing: What Every A/B Testing Practitioner Should Know</h3><h4>Understanding this crucial piece of history will remove the confusion from your experimentation results</h4><p>If you haven’t read <a href="https://medium.com/@pritulp/the-true-history-of-p-values-and-hypothesis-testing-what-every-a-b-testing-practitioner-should-a82ffe35dec9">Part 1</a> yet, I highly suggest you do so to get complete context.</p><p><a href="https://medium.com/@pritulp/the-true-history-of-p-values-and-hypothesis-testing-what-every-a-b-testing-practitioner-should-a82ffe35dec9">The True History of P-values and Hypothesis Testing: What Every A/B Testing Practitioner Should…</a></p><p>You saw at the end of Part 1 the dilemma that can occur to businesses with significance testing only in their toolkit.</p><p>This was a short-coming of the significance testing as pointed out by Jerzey Neyman and Egon Pearson, the two statisticians who challenged significance testing with their own method called “Hypothesis testing”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/532/1*qH9xoR0PauLeOTnj6xzUKw.jpeg" /><figcaption>Jerzey Neyman (source: wikipedia)</figcaption></figure><p>Jerzy Neyman was a Polish mathematician with a flair for practical problem-solving. Picture a man with a quick mind and an even quicker smile, always ready to tackle the next big statistical challenge. Neyman had a knack for seeing how abstract mathematical concepts could be applied to real-world problems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/175/1*BmBfdznaQ3d5cDIUBzI-Ww.jpeg" /><figcaption>Egon Pearson (source: wikipedia)</figcaption></figure><p>Egon Pearson, on the other hand, was the son of the famous statistician Karl Pearson (who invented chi-square test and p-values). Imagine a thoughtful, methodical man, determined to make his own mark on the field while carrying on his father’s legacy. Egon had grown up surrounded by statistics, but he wasn’t content to simply follow in his father’s footsteps.</p><p>Together, Neyman and Pearson formed a dynamic duo in the world of statistics. They wanted to take Fisher’s significance testing futher. With their proposal of Hypothesis testing method, they introduced <strong>five new concepts.</strong></p><h4>1. Alternate Hypothesis</h4><p>They believed that focusing solely on the null hypothesis wasn’t enough. So they introduced the concept of an <strong>alternate hypothesis. </strong>The alternative hypothesis is like the “what if” scenario. If null hypothesis was “our current understanding of the world” then alternate hypothesis was “what if our understanding of the world <strong><em>was xyz</em></strong>”.</p><p>It gives you a clear idea of what you’re looking for. You’re not just asking “Is my old understanding wrong?” but also “Is this new idea right?”. This will get more clear as we visit other concepts.</p><h4>2. Effect size (a.k.a. Minimum detectable effect or MDE)</h4><p>As part of alternate hypothesis it is imperative to specify the change that you seek to observe in the alternate world. If the null hypothesis was no change then in the alternate hypothesis it was required to specify to observe the change by &lt;some number&gt;. If you know what change you’re expecting, you can design your test to detect that specific difference.</p><h4>3. Type I and Type II Errors</h4><p>They defined these two types of errors that can occur in hypothesis testing.</p><p><strong>Type I Error:</strong> Rejecting the null hypothesis when it’s actually true. The probability of making a Type I error is specified by <strong>α</strong> (alpha). The widely acceptable number in online controlled experimentation is 0.05 (or 5%). Sometimes, it is relaxed to 0.1 depending on the use case and population size of the business. In the example of A/B test, Type I error is defined as incorrectly saying that our belief about our world needs to be updated because the data shows that version B is definitely different than version A as p-value &lt;0.05.</p><p><strong>NOTE:</strong> <em>this is </em><strong><em>NOT</em></strong><em> the same as p-value guideline of 0.05. However, in the application of statistics in the modern experimentation platforms, these two seem to have converged.</em></p><p><strong>Type II Error:</strong> Failing to reject the null hypothesis when it’s actually false. The probability of making a Type II error is specified by <strong>β</strong> (beta). The widely acceptable number in online controlled experimentation is 0.2 (or 20%). In the example of A/B test, Type II error is defined as incorrectly saying that there is no difference between version B and version A, when actually there is a difference but the data failed to detect it.</p><p><strong>Power of a Test — </strong>Defined as the probability of correctly rejecting a false null hypothesis. It is the complement of Type II Error and denoted as (1-<strong>β</strong>) and the acceptable threshold is (1–0.2) = 0.8 or 80%. So if there is a specified difference (see Effect size above) between version B and version A, the <strong>Power</strong> of the test is the probability of correctly detecting the difference in the observed data.</p><h4>4. Decision Boundary</h4><p>The predetermined effect size, <strong>α</strong> (alpha), and distribution of test statistic under null hypothesis determine critical value of the test statistic (t-test or z-test) which defines Decision boundary beyond which is the rejection zone. After the experiment, the observed test statistic (t or z) is compared to the predefined boundaries:</p><ul><li>If the test statistic falls within the rejection region then the null hypothesis is rejected</li><li>If it falls outside the rejection region, the null hypothesis is not rejected</li></ul><h4>5. Long-term Frequency Interpretation [most important]</h4><p>This is the concept that tackles the problem arising from Fisher’s approach which was hiding in plain sight — <em>Hypothesis testing is about controlling long-term Type I errors over hundreds and thousands of experiments.</em></p><p>Hypothesis testing is a <strong>true</strong> <strong>objective</strong> <strong>approach</strong> to interpreting experiment results. NOTE — there is no mention of p-value at all. In fact, you do not need p-values if you follow Hypothesis testing approach. And every modern experimentation platform follows Hypothesis testing approach by having alpha and beta thresholds specified. The slight difference or evolution from the original approach is instead of comparing the test statistic with the decision boundaries we translate the Type I and Type II boundaries into sample size and read the data at the end of sample size collection.</p><p><strong>So how does this control long-term Type I errors?</strong></p><p>Let us go to the example where you and your colleague collected samples. Under alternate hypothesis say you both had selected 5% effect size, alpha and power of 0.05 and 0.8. In your 100 samples, you observed 5% lift BUT the test statistic would NOT have crossed the critical value, and in your colleague’s case of 300 samples the test statistic would have crossed the critical value. You would determine that you cannot reject the null hypothesis whereas your colleague would state that they have rejected the null hypothesis in favor of alternate hypothesis. <em>But the main principle that clears the air is the test statistic crossing the decision boundary. This can happen only under certain circumstances (in this case more samples).</em> The business would have a clear indication of what to do next instead of being in a dilemma.</p><p>Let us scale this example up. Imagine a large-scale tech company with 100 product managers running 10,000 experiments annually. Each starts and stops experiments at different times with different effect sizes in different geographies on different targeted demographics. Each product manager has their own understanding of statistics and moods and whims and pressure of delivering. Under such circumstances, there is a <strong>must need for an objective decision-making framework. </strong>And Hypothesis testing provides this exact framework.</p><p>Frequentist test statistics (read Hypothesis testing) are confusing and somewhat restrictive, but they provide a robust framework for controlling long-term Type I errors. <strong>Bayesians</strong> gloat on their approach but their method ultimately succumbs to long-term Type I error control. It is a highly subjective interpretation of how much variant B is better than variant A. Some could accept 55%, some 60% while some stick to 70%. The bigger problem with Bayesian is estimating the priors and there are various approaches and one can pick whatever they can float.</p><blockquote>Despite the fiery opposition these two schools of thought have concentrated against each other for more than 70 years, the two approaches nowadays are embedded in a single exercise that often leads to misuse of the original approaches by naïve researchers and sometimes even statisticians [1]</blockquote><p>In the modern experimentation stats engine, we have combined three separate statistical approaches — sample size calculation, significance testing (p-values), and Hypothesis testing (effect size, errors, and power).</p><p><strong>Why can’t we keep just one — Hypothesis testing in the stats engine if it is an objective framework?</strong></p><p>If you only kept Hypothesis testing in the stats engine, you’d still end up peeking at the results. The peeking in this case is not as bad because if you kept your eye on the goal — the effect size — then any observed lift less than the effect size would need to be ignored. Only read into the data when the observed lift is atleast equal to the effect size.</p><p>Hypothesis testing also poses a problem of when to stop the experiment — if the observed lift is less than the effect size the decision boundary will take a long time to cross. This would be the case of — if you keep an experiment running forever it will be significant and have enough power.</p><p>So translating the decision boundaries into Sample size makes it easier to determine when to stop the experiment and read the data. We could certainly get rid of the p-value from the scorecard. In fact, at two of my jobs we have seriously considered removing it from the scorecard and only showing if certain conditions are met.</p><p>Hope this history helps you get behind the true principles of these theories created at different times, and how blending them in a modern experiment framework creates confusion, especially among those untrained in statistics.</p><p>Further Technical Reading: <br>[1] <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2816758/">https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2816758/</a></p><h3>About the author</h3><p><em>I am a Data scientist with 15+ years of experience in Product analytics and building world-class Experimentation platforms for big tech companies like Apple, eBay, Yahoo, and Peacock TV. I have a knack for understanding complex topics in a manner I can explain to anyone. I read the history and the motivation behind an invention when the topics get confusing. Feel free to </em><a href="https://www.linkedin.com/in/pritul-patel"><em>connect with me on LinkedIn</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=baf7fd50f7e2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Part 1 — The True History of P-values and Hypothesis Testing: What Every A/B Testing Practitioner…]]></title>
            <link>https://medium.com/@pritulp/the-true-history-of-p-values-and-hypothesis-testing-what-every-a-b-testing-practitioner-should-a82ffe35dec9?source=rss-89c0c9586b74------2</link>
            <guid isPermaLink="false">https://medium.com/p/a82ffe35dec9</guid>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[conversion-optimization]]></category>
            <category><![CDATA[experimentation]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Pritul Patel]]></dc:creator>
            <pubDate>Tue, 09 Jul 2024 03:28:34 GMT</pubDate>
            <atom:updated>2024-07-11T01:57:22.441Z</atom:updated>
            <content:encoded><![CDATA[<h3>Part 1 — The True History of P-values and Hypothesis Testing: What Every A/B Testing Practitioner Should Know</h3><h4>Understanding this crucial piece of history will remove the confusion from your experimentation results</h4><ul><li><em>Have you ever been confused by your A/B testing dashboard when p-values are sometimes less than 0.05 even though you have not yet collected required samples?</em></li><li><em>Do your stakeholders get agitated when you stop them from making any decision at this point?</em></li><li><em>Heck, do you have an existential crisis while trying to explain the convoluted statistics to them?</em></li></ul><p>I hope to expose the source of this confusion in this two-part story…</p><p><strong>NOTE:</strong> <em>The post is relevant for Frequentist statistics like t-test. But Bayesians should absolutely read Part 2 before gloating about their approach</em></p><p>It’s the 1920s. The world is recovering from a devastating war. Jazz is all the rage. And in the quiet halls of academia, a statistical revolution is brewing. This story begins with a cast of characters who wouldn’t look out of place in a quirky British detective novel.</p><p>Meet <strong>Sir Ronald Fisher</strong>, a brilliant British statistician and one of the most influential scientists of the 20th century. Fisher had a razor-sharp mind and an intense passion for numbers. He wasn’t just smart; he was a game-changer in the world of statistics. His work laid the foundation for much of modern statistical theory and practice.</p><figure><img alt="Portrait of Sir Ronald Fisher from the year 1912." src="https://cdn-images-1.medium.com/max/291/1*EBFNxketyXo5QPlFUfD7iw.jpeg" /><figcaption>By Unknown author — <a href="https://www.adelaide.edu.au/">https://www.adelaide.edu.au</a>, Public Domain, <a href="https://commons.wikimedia.org/w/index.php?curid=42616717">https://commons.wikimedia.org/w/index.php?curid=42616717</a></figcaption></figure><p>Fisher’s influence on the field of statistics was so profound and wide-ranging that it is challenging to enumerate all of his inventions and developments comprehensively. Here are some of his inventions and contributions — ANOVA, Maximum likelihood method, Fisher information, Fisher’s exact test, F-test, Fisher’s z-transformation, Linear discriminant, and Design of Experiments, to name a few.</p><p>He also widely popularized the <strong>Null hypothesis</strong> and <strong>P-values</strong> in statistical significance testing.</p><h4>Null Hypothesis</h4><p>The Null hypothesis, as Fisher explained it, is like your current understanding of the world. For example, in an A/B test, your null hypothesis might be that there’s no difference between version A and version B because that’s your current understanding of the world. Remember, <strong>we specify the null hypothesis before we start any data collection of the experiment</strong>. There is no new data yet stating that version B is better. So version B is neither better nor worse than version A.</p><h4>p-value</h4><p>He introduced the p-value as a way to measure how surprised we should be by our data if the null hypothesis were true. This means, <strong>the null hypothesis is already assumed to be true</strong>. In the case of A/B test example, we assume that there is no difference between version A and version B. The p-value helps us decide if the results we see (that version B is better than version A) is unusual enough to make us question that assumption (the assumption being null hypothesis is true). Fisher’s approach was straightforward: he suggested that if the p-value is very small (typically less than 0.05), we should consider the results as statistically significant and question our assumption about the null hypothesis being true. In other words, and you might have already heard this that, if p-value is low the null must go. Or as statisticians like to say “we reject the null hypothesis”.</p><p>If it was confusing, let me try to explain in a different manner.</p><p>The null hypothesis is basically your current understanding of the world. So p-value tells you what is the probability that whatever data you’ve observed is so extreme given your current understanding of the world is true. In A/B test example, if your <em>understanding</em> of the world is version B is neither better nor worse than version A, but the p-value is small (say &lt;0.05) which indicates that the probability of <em>observing</em> that version B is actually better than version A is very small (&lt;0.05). In that case, my assumption about null hypothesis (that version B is neither better nor worse than version B) is FALSE. So I need to reject the assumption about the null hypothesis. And my NEW understanding of the world is that version B is better (or worse depending on data) than version A.</p><h4>But the one fact about p-values hidden in plain sight…</h4><p>Fisher’s approach to p-values and significance testing <em>was designed for use with a single hypothesis or experiment at a time</em>. Fisher’s method focused on testing the null hypothesis in isolation. He wasn’t concerned with making decisions across multiple hypotheses whether in tandem or overlapping. His method was more about the strength of evidence in individual cases, rather than decision-making procedures over multiple tests.</p><p>You may ask — how does this matter?</p><p>Picture this — you are in 1930s. There is no Internet. The data-pipelines are not feeding your experiment scorecard. Everything is offline.</p><p>Now you collected some data for your experiment (forget the required sample size). Say you collected 100 observations from each variant. You found that variant B is 5% better than variant A and p-value &gt; 0.05. So you would conclude that variant B is statistically not significant. But your colleague sees your data and presents his data to you which he had independently collected for 300 observations from each variant. His data also shows that variant B is 5% better than variant A with p-value &lt; 0.05.</p><p>Both are correct as per Fisher’s p-value of statistical significance test. Both hypothesis were evaluated independently of each other. Both results are true.</p><p>But how can both results be true?</p><p>This poses a dilemma for the business whether to launch variant B or continue with variant A? This was likely not a concern in old day because the data was always collected in offline mode. Doing multiple or duplicate studies (like the example above) was costly. The world went forward using p-values and the &lt;0.05 guideline (btw… Fisher never meant for 0.05 to be a strict rule, just a guideline) for decision-making for their experiments.</p><p>Just like you are wondering — how can both results be true — two other statisticians also wondered and understood that <strong>significance testing can be a big problem for businesses.</strong></p><p>Hope that I was able to explain the history behind p-values and how every hypothesis was treated in isolation in the offline world.</p><p>The above example has already set the tone for what is coming in part 2. So see you there.</p><p><a href="https://medium.com/@pritulp/part-2-the-true-history-of-p-values-and-hypothesis-testing-what-every-a-b-testing-practitioner-baf7fd50f7e2">Part 2 — The True History of P-values and Hypothesis Testing: What Every A/B Testing Practitioner…</a></p><h4><em>About the author</em></h4><p><em>I am a Data scientist with 15+ years of experience in Product analytics and building world-class Experimentation platforms for big tech companies like Apple, eBay, Yahoo, and Peacock TV. I have a knack for understanding complex topics in a manner I can explain to anyone. I read the history and the motivation behind an invention when the topics get confusing. Feel free to </em><a href="https://www.linkedin.com/in/pritul-patel"><em>connect with me on LinkedIn</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a82ffe35dec9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why you need to truly understand the randomization algorithm of your Experimentation Platform?]]></title>
            <link>https://medium.com/@pritulp/why-you-need-to-truly-understand-the-randomization-algorithm-of-your-experimentation-platform-08a2d9b5cb0b?source=rss-89c0c9586b74------2</link>
            <guid isPermaLink="false">https://medium.com/p/08a2d9b5cb0b</guid>
            <category><![CDATA[ab-testing-software]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[experimentation]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[ab-testing-tools]]></category>
            <dc:creator><![CDATA[Pritul Patel]]></dc:creator>
            <pubDate>Sun, 07 Jul 2024 15:01:44 GMT</pubDate>
            <atom:updated>2024-07-07T15:01:44.618Z</atom:updated>
            <content:encoded><![CDATA[<h4>This is the first component that needs to be made absolutely robust before building sophisticated features of a scalable experimentation platform</h4><figure><img alt="A meme showing a complex stack of blocks with only a thin vertical block supporting one side at the bottom. It conveys that if that thin vertical block fails, the entire stack will collapse." src="https://cdn-images-1.medium.com/max/500/1*kWlCbwFLq0SmYH47YX9n5A.jpeg" /></figure><p>The randomization (or hashing) algorithm is an undervalued pillar of the entire experimentation stack and a key to truly scaling your experimentation platform</p><p>In my 10 years as an Experimentation Data Scientist helping build large-scale experimentation platforms for big tech companies like Yahoo, eBay, Apple, and Peacock TV — the foundational step has always been auditing the capabilities of the “hashing algorithm” — a powerhouse for scaling experimentation capabilities.</p><h3>What is a hashing algorithm?</h3><p>The hashing algorithm consistently turns an input, like a user ID (or browser cookie ID), into a pseudo-random fixed-length (2 to 4 digit) output (e.g. <strong>0 to 99</strong> OR <strong>0 to 9999</strong> buckets). It is crucial for experimentation platforms to randomly but deterministically divide the users into different variants.</p><p>The most popular hashing algorithms are either MD5 or SHA256. These are seemingly inconspicuous in their capabilities but the versatility lies in how the algorithm is designed for accepting various inputs to generate a particular output.</p><p>Below are important capabilities that are directly enabled by designing and handling of a truly versatile hashing algorithm.</p><h4>👥 User Assignment/Sticky bucketing</h4><p>Hashing assigns users to experiment groups such as control or treatment. Robust hashing ensures deterministic user experience meaning the same user will always fall in the same variant as long as that experiment is live, eliminating chances of fragmented encounters and biases.</p><h4>🔗 User Experience</h4><p>The deterministic nature of hashing algorithm keeps users consistently in their experiment groups, preventing random shifts that could distort user experience and give biased results.</p><h4>🔀 Overlapping Experiments</h4><p>Using different “salts” (or hashes or seeds) as an input enables overlapping experiment infrastructure. In overlapping experiments, you can randomly reassign the same user to participate in multiple experiments at the same time as long as the experiments are not on the same web/app element or component.</p><p>Microsoft has built a seed-finder capability in their platform that runs simulations for different hash values and picks the one for the experiment where the user distribution among every bucket value is statistically a uniform distribution.</p><h4>🏊 Swim Lanes</h4><p>Fixing salt values can be used to create “swim lanes”. Within these lanes, experiments can conducted in a mutually-exclusive manner without the risk of overlap or cross-contamination when these experiments are testing on the same feature or features that can create user experience issues (e.g. black text on navy blue background).</p><h4>🚦 Traffic Allocation</h4><p>Can control the percentage of total users to be exposed to each experiment and also individual variants. This is very basic and standard feature in every platform.</p><h4>🚫 Inclusion/Exclusion Groups</h4><p>Apart from swim lanes for setting up mutually exclusive experiments, it can also help control which user groups partake in experiments through inclusion and exclusion groups across various salt values, ensuring clear boundaries between who does and doesn’t see specific features.</p><h4>⚖️ Bias Mitigation</h4><p>One of the core components of the hashing algorithm should be automated rehashing (or resalting). Doing this diminishes pre-existing biases by reshuffling users from a concluded experiment to a new experiment.</p><h4>🥪 Experiment Layering</h4><p>In sophisticated platforms, hashing ensures that multiple layers of experiments run smoothly without interference.</p><p>No matter how sophisticated an experimentation platform&#39;s capabilities are, if the hashing algorithm goes unaudited then it will lead to major problems like:</p><ul><li>Experiment samples not representative of the Population</li><li>Pre-existing biases</li><li>Decision Errors due to biased data</li><li>Trust erosion</li><li>Wasted resources</li></ul><p>Below are the two most common issues I have uncovered while auditing the hashing algorithm:</p><ol><li>Non-deterministic assignments leading to the sample being not representative of the population</li><li>No automated rehashing/reshuffling leading to biased outcomes</li></ol><p>These two alone can completely destroy all the value of experimentation in product development because the data obtained will be highly unreliable.</p><h3>What should you do?</h3><ol><li>For an in-house platform, sit with your developers to truly understand the various capabilities of the hashing algorithm to ensure that there is an automated reshuffling mechanism as well as the assignment is deterministic for every experiment.</li><li>For 3rd party platform, ask your tech rep to walk you through the various components of the hashing algorithm.</li></ol><p><em>If you need a professional help in auditing your experimentation platform capabilities then feel free to reach out to me at sigmasquaredco@gmail.com or send me a </em><a href="https://www.linkedin.com/in/pritul-patel"><em>message on LinkedIn</em></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=08a2d9b5cb0b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Wordle inspired games you should definitely give a try]]></title>
            <link>https://medium.com/@pritulp/wordle-inspired-games-23ea8e2b996a?source=rss-89c0c9586b74------2</link>
            <guid isPermaLink="false">https://medium.com/p/23ea8e2b996a</guid>
            <category><![CDATA[wordle]]></category>
            <category><![CDATA[word-guessing-game]]></category>
            <dc:creator><![CDATA[Pritul Patel]]></dc:creator>
            <pubDate>Tue, 15 Feb 2022 02:11:31 GMT</pubDate>
            <atom:updated>2022-02-15T02:13:47.105Z</atom:updated>
            <content:encoded><![CDATA[<p>Like you I’ve been hooked on Wordle. It took a long time for me to discover this game but has quickly become a daily habit. Much has been said about the game so that is not the topic of this post.</p><p>While I’ve been playing wordle I have come across 3 games that are inspired from Wordle. Below are the games and some details for you to try. I will add more to the list as I come across more brilliant adaptations of the game.</p><h3><strong>Worldle</strong></h3><p>Notice the subtle spelling difference. As the name suggests, the game is testing <strong>world</strong> map knowledge. The game shows you an outline of a country and you have to guess which country it is. You get 6 guesses. For each incorrect guess, it tells you how far away the subject country is in km (or miles in settings) and the direction where the subject country is. It is now my favorite as I am a big world map fan. In hard mode, you can either decide to hide the country outline (for blindly guessing) as well as you can randomly make the game rotate the country outline so its hard to guess. All the time that I’ve played the countries have been small. You have high chance of losing on this game than in the classic Wordle.</p><p>You can play Worldle at <a href="https://worldle.teuteuf.fr/">worldle.teuteuf.fr</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*RO3zEqedJzelFPu2LX_Lxw.png" /><figcaption>Worldle game UI showing two guesses with distance and direction of shown country map</figcaption></figure><h3><strong>Semantle</strong></h3><p>This one is built on a machine learning algorithm known as word2vec. It uses neural networks to make word associations from a large collection of text. This game specifically utilizes ‘news articles’ as the collection. How to play? I won’t go in full details as you can read them on the game website. Just start guessing random words. Literally. The guessed word will show a similarity score and ‘Getting Close’ indicator to the target word. The getting close indicator is telling the score of the word you’ve guessed compared to the target word. The target word is always at 1000. You can improve your guesses to get the similarity score and getting close indicator higher and higher. There is no limit to number of guesses. I’ve only played this once and I guessed the correct word in 46 guesses. Time to play again tomorrow. You can play Semantle at <a href="https://semantle.novalis.org/">https://semantle.novalis.org/</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/559/1*m6nzclEXw5zG8mhITXMELg.png" /><figcaption>UI is very rudimentary with a simple box to type in the middle of text filled screen. Disclaimer: I took the screenshot with fake entries after I had already won.</figcaption></figure><h3><strong>Quordle</strong></h3><p>If Wordle is too easy for you, here’s Wordle x 4. Yes, you have to guess 4-five letter words all in 9 guesses. This is obviously a hard one but I have to appreciate the keyboard design for this complex game. I won’t go into details here and leave it up to you to play the game. But I wanted to share how I did playing the game the first time. I could solve only 2 out of 4 words in lower quadrants at 4th and 6th attempts.</p><p>You can play the game at <a href="https://www.quordle.com/#/">quordle.com</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*BcO99347nQsQRrw-pw8-7w.png" /><figcaption>Quordle results take up some space but the results are elegantly summarized.</figcaption></figure><p>Hope you like this list. Please share with me if there are any other interesting twists added on Wordle.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=23ea8e2b996a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is the most important factor to succeed in Data Science interviews?]]></title>
            <link>https://medium.com/@pritulp/what-is-the-most-important-factor-to-succeed-in-data-science-interviews-48ff7b2b65ae?source=rss-89c0c9586b74------2</link>
            <guid isPermaLink="false">https://medium.com/p/48ff7b2b65ae</guid>
            <category><![CDATA[self-improvement]]></category>
            <category><![CDATA[self-awareness]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[interview-tips]]></category>
            <category><![CDATA[job-hunting]]></category>
            <dc:creator><![CDATA[Pritul Patel]]></dc:creator>
            <pubDate>Tue, 04 Jan 2022 19:40:09 GMT</pubDate>
            <atom:updated>2022-01-04T19:40:09.014Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pg8ZuKyc2oSt6pGIuxRCbw.jpeg" /></figure><p>Being on both side of the interview tables for years here’s what I believe is the most important factor to conquering data science interviews.</p><p><strong>Approaching interview with a right mindset and not afraid of failing spectacularly.</strong></p><p>One statement that has fixated in my head for years is ‘the person who is most qualified to do the job doesn’t get the job, the person who interviewed best for the job gets the job’. So if believe you’re qualified to do the job you’ll still have to learn to master the interview techniques.</p><blockquote><em>“the person who is most qualified to do the job doesn’t get the job, the person who interviewed best for the job gets the job”</em></blockquote><p>You can prepare as much technical skills by solving hundreds of leetcode questions, solving every SQL question bank out there, and memorizing solutions to a list of 100+ stats and probability questions. But going to an interview with a right mindset will go a long way than any amount of technical and soft-skill preparation. In fact, after a short preliminary preparation, the goal should be diving straight into giving interviews and failing spectacularly. The failures will lead you to a path of becoming a better interviewee. You might learn that you don’t need to go deep into a particular topic and maybe spend that time elsewhere. Alternately, you can spend hundreds of dollars on mock interview services which absolutely helps cut down this process for you but then it does take away the fun in recognizing and conquering your inner demons especially in real-life job situations that keeps throwing curve-balls!</p><p>Now you may ask — What is a right mindset? Here are a few top tips from my experience to cultivating right mindset:</p><h3>1. Embracing interviews</h3><p>Most people dread the interview process. The thought of conversing with so many unknown people who’ll judge you based on what you say is nerve-wracking. But pay attention to the keyword “conversing”. Interviewing is a two-way street. Think of it as a conversation rather than one sided bombardment of questions. In conversations, you want to communicate your thought process with clarity. To bring out that clarity you will have to ask counter-questions. This is especially true during product case study rounds, as well as leetcode style and SQL programming rounds. There are no right or wrong answers in an interview but not showing your thought-process will absolutely guarantee rejections. Personally, before the interview, I spend 10–15 minutes doing a mindfulness exercise that calms by nerves and clears the clouded brain to bring out clarity in thoughts. Outcome doesn’t matter after this point.</p><h3>2. Using silence to your advantage</h3><p>Listen to the interviewer carefully and don’t think what you’re going to say next after they stop speaking, or worse, cutting them off to get your thoughts out. Having a moment of silence after interviewer has spoken will give you a chance to gather some thoughts around the question. If needed, write down few words so you can frame cohesive sentences around it. This will help you build patience, calm your nerves, and cultivate presence of mind. This also works when interviewer throws a wrench in your thought-process which is typical in a working environment as well.</p><h3>3. Accepting Rejections or Failures</h3><p>This one connects back to the first point. Embracing interview process also means embracing rejections as part of the package. Failures provide opportunities to address your blind-spots. Think of it like stock market where you’re absolutely bound to make losses if you made only a few trades. But as you make more and more trades you learn to put stop-losses giving much higher dollar wins even if the win-rate didn’t improve i.e. number of losses might still be high but losses in dollar value will be restricted to a minimal. In the context of interviews, your stop-losses come in the form of showcasing your thought-process, ability to have a good two-way conversation, and projecting confidence of your soft and hard skills. This means you’re building confidence among various members of the interview panel to reducing your individual rejects. If you have convinced 4 out of 5 interviewers to hire you then there is a very high chance of getting an offer or a possibility of scoring an additional interview. If you’re rejected then you’ll almost never know the reason which brings me to the last point.</p><h3>4. Deconstructing the Interview</h3><p>Companies rarely provide any feedback for rejections. So after the interview, spend some time replaying it in your head. Recall your answers to the questions, the interviewer’s reaction and counter-questions, and putting yourself in the interviewer’s shoe to determine whether would you hire you? If not, think how you could have better approached a question and write it down to address it better next time. This exercise helps reveal blind spots especially when you get a rejection when you were least expecting.</p><p>Anything else that you feel helps with approaching the interview with a right mindset? Feel free to share your comments below.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=48ff7b2b65ae" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I conducted my first ever Year-in-Review and was shocked]]></title>
            <link>https://medium.com/@pritulp/i-conducted-my-first-ever-year-in-review-and-was-shocked-af3bddbcb21e?source=rss-89c0c9586b74------2</link>
            <guid isPermaLink="false">https://medium.com/p/af3bddbcb21e</guid>
            <category><![CDATA[year-in-review]]></category>
            <category><![CDATA[self-improvement]]></category>
            <category><![CDATA[motivation]]></category>
            <category><![CDATA[self-reflection]]></category>
            <category><![CDATA[goals]]></category>
            <dc:creator><![CDATA[Pritul Patel]]></dc:creator>
            <pubDate>Sat, 05 Jan 2019 03:48:34 GMT</pubDate>
            <atom:updated>2019-01-07T01:35:40.813Z</atom:updated>
            <content:encoded><![CDATA[<h3>I had an epiphany after I conducted my first ever Year-in-Review</h3><p><em>“A good plan, violently executed now, is better than a perfect plan tomorrow.” — George S Patton, General, US Army</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FrDceIp-Kg1_gi8QKNYvYA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/photos/gNUMZjxd89w?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">rawpixel</a> on <a href="https://unsplash.com/collections/2207641/gettings-things-done?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><h4>Why I opted to do a year-in-review?</h4><p>Throughout the month of December I came across several posts like <a href="https://www.gatesnotes.com/About-Bill-Gates/Year-in-Review-2018">this</a> (blog), <a href="https://www.getrichslowly.org/2018-year-review/">this</a> (blog) and <a href="https://www.youtube.com/watch?v=3V_e3_KHgeg&amp;t=412s">this</a> (YouTube) talking about their year-in-review. What I noticed is at the end of these reviews the subjects had so much clarity on what they wanted to achieve next year. Even if they did not have full clarity at least they know what they wanted to do the next year. I digged that! If you asked me what I wanted to do next year I would have no clue apart from some stupid answer like make more money. Duh!</p><h4>What happened after I conducted the review?</h4><p>With inspiration, I decided to conduct my year-in-review. Worst case it would be just a list of things I did that I’d never look at again. But I was wrong.</p><p>I was shocked with the long list of experiences and accomplishments that I was able to achieve in just one year. Why was I shocked considering that there are 365 days in a year and I was bound to achieve or experience those things? Because when I started writing, the only thing that I could recollect initially were the endless hours of wasted time mindlessly watching TV, browsing Reddit, Instagram, Twitter and YouTube! This realization paralyzed me. I had dug deep into the crevices of my brains for extracting the list.</p><p>What was more shocking is realizing how much significantly more I could have achieved! The endless hours wasted on several media started to replay in my head. I was cringing with realization my 100 point bucket-list hadn’t materialized as if it was going to happen by itself!</p><p>I am highly grateful for what I was able to accomplish and experience — personal and mental growth, fulfilling relationships, home-improvement projects, starting a new job and overcoming most of its challenges, exploring the world of behavioral economics through books, podcasts, papers, etc. And there were those <em>first-evers</em> — ski-trip, Euro-trip through 9 countries, reading books, carpooling, 200+ audience teaching session that helped me grow tremendously.</p><p>The above shortlist expands into 40+ detailed bullets. It is shocking for most to have done so much in a year but for high-performing individuals who live by the <a href="https://www.amazon.com/4-Hour-Workweek-Escape-Live-Anywhere/dp/0307465357/ref=sr_1_2?ie=UTF8&amp;qid=1546659839&amp;sr=8-2&amp;keywords=4+hour+work+week">4-hour work-week</a> (Amazon) principles my achievements are, let’s just say, cute! And rightfully so.</p><p>For example, I wanted to start a blog since a few years now but procrastinated with excuses like “my thoughts are not unique”, “there’s already material about this topic a Google search away”, “I don’t have Internet clout, who’ll read my post?”, “I lack articulation”, “I don’t have time writing”. But this writing, <em>my very first ever long form blog post</em>, is coming after conducting my year-in-review which revealed my lack of achievements and left me unfulfilled.</p><p>Another example. When I look back at my goals of 2016, I had a goal of purchasing 4 rental properties, each year, from 2017 onwards. I achieved half of my 2017 goal and was so proud about it but the same goal fell flat on its face in 2018. I live in CA but purchase rental properties out of state. I had excuses from “not having money” to “I don’t have a real estate agent in that state”. Any real estate investor knows that both those excuses cannot stop you from your goals. Don’t have money then find investors who’ll lend you money, or be creative with banks; don’t have an agent or property manager then just Google! All I had to do is take action. Any action. I tapped into my existing resources on the last working day of 2018 (Dec 31st). Lo and behold. I had a list of 13 rental properties with rental potential from my agent I worked with before on the first working day of 2019 (Jan 2)! And I had found a lending company ready to handle the finance should I choose to buy one or more from the list. The result was that quick.</p><h4>My Inspiration</h4><p>One of the books I read this year was Mark Manson’s <a href="https://www.amazon.com/Subtle-Art-Not-Giving-Counterintuitive/dp/0062457713/ref=sr_1_2?ie=UTF8&amp;qid=1546657924&amp;sr=8-2&amp;keywords=subtle+art+of+not+giving+a+f">The Subtle Art of Not Giving a F*ck</a> (Amazon). The first half of the book is hilarious and packed with wisdom! Another book I read was <a href="https://www.amazon.com/Work-System-Mechanics-Working-Revised/dp/160832253X/ref=sr_1_1?ie=UTF8&amp;qid=1546657968&amp;sr=8-1&amp;keywords=Work+The+System">Work The System</a> (Amazon) by Sam Carpenter. Both talked about executing in the face of procrastination. The results will be multi-fold. This year I’ve already been introduced to a book titled <a href="https://www.amazon.com/10X-Rule-Difference-Between-Success/dp/0470627603/ref=sr_1_2?ie=UTF8&amp;qid=1546658031&amp;sr=8-2&amp;keywords=the+10x+rule+book">The 10X Rule</a> (Amazon) by Grant Cardone which talks about amplifying your goals for growing at break-neck speed.</p><p>I look forward to choke-slam 2019 with this new found zing for accomplishing my long term goals of blogging and starting a YouTube channel, while continuing investing in real estate. By the way, these are outside of my full-time job. At work my goals are all related to the field of Data Science — technical goals on researching and implementing statistical processes at large scale; and growing into management level — communication, compassion, empathy, planning and strategy, helping others grow, etc.</p><h4>Take Action. Now.</h4><p>It’s not late to review your 2018. Before the stresses of 2019 haze the memories of 2018, grab a pen and a paper and start writing your achievements and every memorable experience of last year and how they helped towards your goals. If you’ve never written goals then I recommend creating a massive list of at least 100 things that you want to achieve in this life-time — small or big — across all avenues from self-development, relationships, travel, career, personal, spiritual, technical, financial, etc. Then check the ones out that you already achieved in 2017 and 2018. You will instantly realize that a year is a very very very long time and we can easily accomplish several things on our checklist if we take action while not fully giving up on fun.</p><p>Hope this post helps making your 2019 action-packed and filled with rewards.</p><p><em>P.S.: This is my first blog post. If you are a writing expert, I’d highly value your critique. I intend to articulate better in my future posts and it cannot happen without getting some feedback.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af3bddbcb21e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>