<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Itamar Faran on Medium]]></title>
        <description><![CDATA[Stories by Itamar Faran on Medium]]></description>
        <link>https://medium.com/@itamar.faran?source=rss-153789513a7b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Z3UTOYqSNPzMzW5KIW04_g.png</url>
            <title>Stories by Itamar Faran on Medium</title>
            <link>https://medium.com/@itamar.faran?source=rss-153789513a7b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 23:56:18 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@itamar.faran/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[CRPS — A Scoring Function for Bayesian Machine Learning Models]]></title>
            <link>https://medium.com/data-science/crps-a-scoring-function-for-bayesian-machine-learning-models-dd55a7a337a8?source=rss-153789513a7b------2</link>
            <guid isPermaLink="false">https://medium.com/p/dd55a7a337a8</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[model-evaluation]]></category>
            <category><![CDATA[probability]]></category>
            <category><![CDATA[bayesian-statistics]]></category>
            <category><![CDATA[bayesian-machine-learning]]></category>
            <dc:creator><![CDATA[Itamar Faran]]></dc:creator>
            <pubDate>Sat, 28 Jan 2023 01:27:52 GMT</pubDate>
            <atom:updated>2023-01-28T09:21:37.522Z</atom:updated>
            <content:encoded><![CDATA[<h3>CRPS — A Scoring Function for Bayesian Machine Learning Models</h3><h4>The Continuous Ranked Probability Score is a statistical metric that compares distributional predictions to ground-truth values</h4><p>An important part of the machine learning workflow is the model evaluation. The process itself can be considered common knowledge: split the data into train and test sets, train the model on the train set, and evaluate its performance on the test set using a score function.</p><p>The score function (or <a href="https://scikit-learn.org/stable/modules/model_evaluation.html">metric</a>) is a mapping of the ground truth values and their predictions into a single and comparable value [1]. For example, for continuous predictions one could use score functions such as the RMSE, MAE, MAPE or R-squared. But what if the prediction is not a point-wise estimate, but a distribution?</p><p>In Bayesian machine learning, the predictions are often not point-wise estimates but distributions of values. For example, the prediction could be estimated parameters of a distribution, or, in the non-parametric case—an array of samples from an MCMC method.</p><p>In these cases, traditional score functions do not suit the statistical design; one could aggregate the predicted distributions into their mean or median values, but that would result with a great loss of information regarding the dispersion and shape of the predicted distribution.</p><h4>The Continuous Ranked Probability Score</h4><p>The CRPS — Continuous Ranked Probability Score — is a score function that compares a single ground truth value to a Cumulative Distribution Function (CDF):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/438/1*UxE3bHk_j6WdZ5kO1g5Q7w.png" /><figcaption>Definition of the CRPS [1]. Image by author.</figcaption></figure><p>First introduced in the 70’s [4] and primarily used in weather forecasts, it is now gaining renewed attention in the literature and industry [1] [6]. It can be used as a metric to evaluate a model’s performance when the target variable is continuous and the model predicts the target’s distribution; Examples include Bayesian Regression or Bayesian Time Series models [5].</p><p>The fact that the theoretical definition includes the CDF makes the CRPS useful for both parametric and non-parametric predictions: for many distributions there is an analytic expression for the CRPS [3], and for non-parametric predictions, one could use the CRPS with the Empirical Cumulative Distribution Function (eCDF).</p><p>After computing the CRPS for each observation in our test set, we are left to aggregate the results into a single value. Similarly to the RMSE and MAE, we’ll aggregate them using a (possibly weighted) average:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/531/1*3LkUGvalIpLBursH_9I8CQ.png" /><figcaption>Aggregation of CRPS over the test set, with empirical CDFs. Image by author.</figcaption></figure><h4>Intuition</h4><p>The main challenge of comparing a single value to a distribution is how to translate the single value into the domain of distributions. The CRPS deals with that by translating the ground truth value into a <a href="https://en.wikipedia.org/wiki/Degenerate_distribution">degenerate distribution</a> with the indicator function. For example, if our ground truth value is 7, we can translate it with:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/1*m9-gsDfMaWBlNou-Z174jg.png" /><figcaption>Example of a <a href="https://en.wikipedia.org/wiki/Degenerate_distribution">degenerate distribution</a> with an indicator function. Image by author.</figcaption></figure><p>The indicator function is a valid CDF answering all the <a href="https://en.wikipedia.org/wiki/Cumulative_distribution_function#Properties">requirements of a CDF</a>. Now we are left with comparing the predicted distribution to the degenerate distribution of the ground truth value. Clearly, we want the predicted distribution to be as close as possible to the ground truth; this is expressed mathematically by measuring the (squared) area trapped between these two CDFs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TiwKzA_kQ5wZNit4pc4r2g.png" /><figcaption>Visualization of the CRPS. The predicted distribution is marked in red, and the ground truth’s degenerate distribution is marked in blue. The CRPS is the (squared) area trapped between the two CDFs. Image by author.</figcaption></figure><h4>Relation to the MAE</h4><p>The CRPS is closely related to the well-known <a href="https://en.wikipedia.org/wiki/Mean_absolute_error">MAE</a> (Mean Absolute Error). If we take a point-wise prediction, treat it as a degenerate CDF and inject it into to the CRPS equation, we get:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/361/1*Fb9MOEV5i_uEnGP_kZyE6A.png" /><figcaption>Relation between the CRPS and the MAE. Image by author.</figcaption></figure><p>So, if the predicted distribution is a degenerate distribution (e.g. a point-wise estimate), the CRPS reduces to the MAE. This helps to get another intuition for the CRPS: it can be viewed as a <strong>generalization of the MAE into distributional predictions</strong>: <strong>The MAE is a special case of the CRPS</strong> when the predicted distribution is degenerate.</p><h4>Empirical Evaluation</h4><p>When the model’s prediction is a parametric distribution (e.g. the model predicts the distribution’s parameters), the CRPS has an analytic expression for some common distributions [3]. For example, if the model predicts the parameters <em>μ</em> &amp; <em>σ </em>of the Normal distribution, the CRPS can be calculated with:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*poiwNJtvoZRPAIWSRM3AxQ.png" /><figcaption>Analytical solution of the CRPS for the normal distribution [3]. Image by author.</figcaption></figure><p>Analytic solutions are known for distributions such as Beta, Gamma, Logistic, Log-Normal and others [3].</p><p>When the prediction is non-parametric, or more specifically — the prediction is an array of simulations, calculating the integral over the eCDF is a hefty task. However, the CRPS can also be analytically expressed by:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/637/1*eox6dP-XV0k1S5xzOj8R1A.png" /><figcaption>Different forms of the CRPS and their names [2]. Image by author.</figcaption></figure><p>Where <em>X, X’</em> are independently and identically distributed according to <em>F</em>. These expressions, while still a bit computationally intensive, are simpler to estimate:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fe0681c95b590cc4bbc0ccbb94de7973/href">https://medium.com/media/fe0681c95b590cc4bbc0ccbb94de7973/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/de3e7d505844e5388125b247d09704a4/href">https://medium.com/media/de3e7d505844e5388125b247d09704a4/href</a></iframe><p>You can check out an example on a Bayesian Ridge Regression in a Jupyter notebook <a href="https://github.com/itamarfaran/public-sandbox/blob/master/crps.ipynb">here</a>, where I demonstrate the usage of both the parametric and non-parametric CRPS.</p><h4>Summary</h4><p>The Continuous Ranked Probability Score (CRPS) is a scoring function that compares a single ground-truth value to its predicted distribution. This property makes it relevant to Bayesian machine learning, where models usually output distributional predictions rather than point-wise estimates. It can be viewed as a generalization of the well known MAE to distributional predictions.</p><p>It has analytical expressions for parametric predictions, and can be simply computed for non-parametric predictions. All together, the CRPS emerges as the new standard way to evaluate the performance of Bayesian machine learning models with a continuous target.</p><h4>References</h4><ol><li><em>Strictly Proper Scoring Rules, Prediction, and Estimation, </em>Gneiting &amp; Raftery (2007)</li><li><em>Estimation of the Continuous Ranked Probability Score with Limited Information and Applications to Ensemble Weather Forecasts</em>, Zamo &amp; Naveau (2017)</li><li><em>Calibrated Ensemble Forecasts Using Quantile Regression Forests and Ensemble Model Output Statistics</em>, Taillardat, Zamo &amp; Naveau (2016)</li><li><em>Scoring Rules for Continuous Probability Distributions</em>, Matheson &amp; Winklers (1976)</li><li><em>Distributional Regression and its Evaluation with the CRPS: Bounds and Convergence of the Minimax Risk,</em> Pic, Dombry, Naveau &amp; Taillardat (2022)</li><li><a href="https://docs.pyro.ai/en/stable/ops.html#pyro.ops.stats.crps_empirical">CRPS Implementation in Pyro-PPL</a>, Uber Technologies, Inc.</li><li><a href="https://github.com/properscoring/properscoring">CRPS Implementation in properscoring</a>, The Climate Corporation</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dd55a7a337a8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/crps-a-scoring-function-for-bayesian-machine-learning-models-dd55a7a337a8">CRPS — A Scoring Function for Bayesian Machine Learning Models</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dask or Spark? A Comparison for Data Scientists]]></title>
            <link>https://medium.com/geekculture/dask-or-spark-a-comparison-for-data-scientists-d4cba8ba9ef7?source=rss-153789513a7b------2</link>
            <guid isPermaLink="false">https://medium.com/p/d4cba8ba9ef7</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[dask]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[spark]]></category>
            <dc:creator><![CDATA[Itamar Faran]]></dc:creator>
            <pubDate>Sat, 04 Jun 2022 07:32:30 GMT</pubDate>
            <atom:updated>2022-06-04T07:32:30.416Z</atom:updated>
            <content:encoded><![CDATA[<h4>3 Reasons Why Dask is Better Suited for Data Science Projects (And When it is Not)</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*dbUA5oJxia41XAL3.jpg" /><figcaption>By <a href="https://imgflip.com/memegenerator">imgflip.com</a></figcaption></figure><h3>Introduction: Big Data in Data Science</h3><p>As a data scientist, you might encounter projects where your data doesn’t fit into memory. In these cases, popular tools such as <a href="https://pandas.pydata.org/">pandas</a> and <a href="https://numpy.org/">NumPy</a> cannot support your operations, even <a href="https://medium.com/bigdatarepublic/advanced-pandas-optimize-speed-and-memory-a654b53be6c2">after optimization</a>. Before going to Big Data solutions such as <a href="https://spark.apache.org/docs/latest/api/python/">PySpark</a>, there are 3 question you should ask yourself:</p><ol><li><strong>Why do I have Big Data? </strong>Do I really need all the data? Is there any way to reduce its size by sampling or filtering?</li><li><strong>How can I make it small again? </strong>Do I really need this level of aggregation? Can I reduce its size by aggregation or using chunks?</li><li><strong>What are my Big Data needs and which tools support it? </strong>Where is my bottleneck?<br>• Do I train or predict on Big Data?<br>• Do I train one model on a large dataset, or do I train many models on small datasets?</li></ol><p>If after answering the first two questions your data still doesn’t fit in memory, then the purpose of this post is to present the advantages (and disadvantages) of <a href="https://dask.org/">Dask</a> in comparison to <a href="https://spark.apache.org/docs/latest/api/python/">Apache Spark</a> for your data science needs and to advise when to choose which.</p><h3>What is Dask?</h3><blockquote>Dask is a Pure Python Big-Data solution that integrates with the Python Data Science ecosystem.</blockquote><p><a href="https://dask.org/">Dask</a> is a Python module and Big-Data tool that enables scaling pandas and NumPy. Like Spark, Dask supports parallel execution and handles out-of-memory data frames and arrays. But while Spark is written in Scala and offers a Python API (PySpark), Dask is a Pure Python solution that is part of the Python data science ecosystem.</p><p>Dask shares many characteristics with Spark. Like Spark, Dask has a lazy execution. That means that when you run a command in the console, the actual action isn’t executed — but appended to an execution graph. When the execution command is called, Dask will optimize the graph before executing it. You can <a href="http://distributed.dask.org/en/stable/manage-computation.html">manage Dask’s execution graph</a> with the .compute() and .persist() methods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/997/1*M19s5GddXfmgnDjcsd9YFw.png" /><figcaption>High level collections are used to generate task graphs which can be executed by schedulers on a single machine or a cluster. Image taken with permission from <a href="https://docs.dask.org/en/latest/">dask.org</a>.</figcaption></figure><p>Another resemblance is the ability to scale by deploying clusters. Like Spark, a <a href="https://docs.dask.org/en/stable/scheduler-overview.html">Dask scheduler</a> can be deployed to a cluster of machines using various technologies. In this case you can work with many Dask workers, and their resources will be managed by the scheduler. Unlike Spark, you can work <a href="https://docs.dask.org/en/stable/deploying.html">without a scheduler</a>; You can just import dask.dataframe as dd and get to work.</p><h3>3 Reasons to Choose Dask</h3><h4>I: Integration with pandas and NumPy</h4><p>pandas and NumPy are the bread &amp; butter of any Data Scientist, and Dask is built upon these libraries. This fact results with a couple of positive outcomes for us. First of all is the common API; if you have good control over the pandas and NumPy API, you’ve already half way migrated to Dask as Dask inherits most of these modules’ API.</p><p>The second is the ability to work with these libraries in the same project. As pandas dataframes are the “building blocks” of Dask dataframes, Dask knows how to handle them both. For example, if I deal with both large and small dataframes in my project, I can manage the big ones using Dask and the small ones using pandas. I can concatenate and merge them together easily without any preparation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/569/1*F7YlsjUq2NUgd-8YPvWjIw.png" /><figcaption>Dask scales NumPy arrays and pandas dataframes. Images taken with permission from <a href="https://dask.org/">dask.org</a>.</figcaption></figure><p>However, it is important to remember that <strong>Dask is not pandas</strong>. Unlike pandas, Dask objects are immutable, and don’t support inplace operations. This can be very frustrating, as for example you cannot assign new values using the .loc method. However, Dask offers the .map_partitions method that does support pandas operations on each partition.</p><p>These properties allow you to write projects that can work on both Dask and pandas dataframes with very little effort. In turn, this makes it very simple to develop and debug your project: You can develop in pandas, debug with a simple scheduler, and deploy with a distributed scheduler.</p><h4>II: Integration with SciKit-Learn and JobLib</h4><p>Dask is closely integrated with SciKit-Learn and inherits it’s conventional API; this is a huge advantage over Spark, as pyspark.ml uses a whole new unconventional API that we need to learn.</p><p>Dask offers some in-house preprocessing and machine learning algorithms, ranging from linear models and Naive Bayes to clustering, decomposition and XG-Boost. Nonetheless, Dask supports the parallelization of most SKLearn models as a backend for the JobLib library.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/0*af4t59aV49gyNb0T.jpg" /><figcaption>Image taken with permission from <a href="https://github.com/dask/dask/issues/4471">github.com/dask</a></figcaption></figure><p>Finally, in cases when you need to train your model on a small dataset and predict on a large one, Dask offers the ParallelPostFit wrapper over SciKit-Learn models. This wrapper allows any SKLearn model to support Dask dataframes and arrays in their .predict method.</p><h4>III: A Pure Python Solution</h4><p>Dask is implemented purely in Python, so it has no ex-pythonic dependencies. This has implications for both us data scientists, and the engineers supporting our operations.</p><p>For us data scientists, this means that the developing and debugging becomes much simpler. In order to run your project locally, you don’t need to install non-pythonic dependencies such as Scala, Java and Spark. If you’re working with a simple scheduler, you don’t need to deal with raising a local cluster, and the project dependencies can be easily managed with an environment manager (such as Anaconda). This makes it much simpler to research, develop and debug our projects.</p><p>For data engineers, this means that deploying projects becomes much simpler. If your data is medium sized and you use a simple scheduler in your project, it can be deployed with a virtual environment, instead of building a Spark cluster (with a YARN scheduler, for example).</p><h4>Spark’s Advantages over Dask</h4><p>While Dask suits data science projects better and is integrated within the Python ecosystem, Spark has many major advantages, including:</p><ol><li><em>Spark is able to deal with much bigger work loads than Dask.</em> If your data is larger than 1TB, Spark is probably the way to go.</li><li><em>Dask’s SQL engine is premature.</em> Unlike Spark, you can’t manipulate your data with SQL queries (<a href="https://dask-sql.readthedocs.io/en/latest/">for now</a>).</li><li><em>Spark is part of the Apache eco-system.</em> Unlike Dask, it integrates with other Apache tools such as Hive and Iceberg.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*LF4iXrriwzb1gnye" /><figcaption>The Apache Eco-System. Image taken from <a href="https://dzone.com/articles/example-of-etl-application-using-apache-spark-and">dzone.com</a>.</figcaption></figure><h3>Summary</h3><blockquote>You should save the big guns for the big wars — and Spark is a <strong>really big gun</strong>.</blockquote><p>Remember the 3 questions you need to ask yourself as a data scientist dealing with big data:</p><ol><li>Why do I have Big Data?</li><li>How can I make it small again?</li><li>What is the correct Big Data Tool for my project’s needs?</li></ol><p>In this blog post, I presented 3 reasons why Dask can be more suitable for data science projects: it integrates with pandas and NumPy, it integrates with SciKit-Learn, and it is a Pure Python solution — making life much easier for us data scientists and the engineers that support us. While Spark intends to be an all-in-one solution, Dask intends to blend in the Python data science ecosystem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fxmt4LG0j4CObyLoQIZ7Rg.png" /><figcaption>Dask, Modin, Vaex, Ray, and CuDF are often considered potential alternatives to each other. Image generated with <a href="https://anvaka.github.io/vs/?query=Dask">this tool</a>, cited also in <a href="https://www.datarevenue.com/en-blog/pandas-vs-dask-vs-vaex-vs-modin-vs-rapids-vs-ray">datarevenue.com</a>.</figcaption></figure><p>However, Dask might also not be the best suited tool for your project; think about what you need before choosing a solution. There are other Pythonic solutions for Big Data, such as <a href="https://www.ray.io/">Ray</a> and <a href="https://modin.readthedocs.io/en/stable/">Modin</a>, <a href="https://vaex.io/">Vaex</a> and <a href="https://rapids.ai/">Rapids</a>; all have their <a href="https://www.datarevenue.com/en-blog/pandas-vs-dask-vs-vaex-vs-modin-vs-rapids-vs-ray">pros and cons</a>. And if you have more than a couple of terabytes of data, Spark is still probably the way to go.</p><h3>References</h3><ol><li><a href="https://docs.dask.org/en/stable/spark.html">Comparison to Spark</a>, <em>dask.org</em></li><li><a href="https://databricks.com/session/dask-and-apache-spark">Dask and Apache Spark</a>, <em>databricks.com</em></li><li><a href="https://www.datarevenue.com/en-blog/pandas-vs-dask-vs-vaex-vs-modin-vs-rapids-vs-ray">Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and RAPIDS</a>, <em>datarevenue.com</em></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d4cba8ba9ef7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/geekculture/dask-or-spark-a-comparison-for-data-scientists-d4cba8ba9ef7">Dask or Spark? A Comparison for Data Scientists</a> was originally published in <a href="https://medium.com/geekculture">Geek Culture</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How To Do Bayesian A/B Testing, FAST!]]></title>
            <link>https://medium.com/data-science/how-to-do-bayesian-a-b-testing-fast-41ee00d55be8?source=rss-153789513a7b------2</link>
            <guid isPermaLink="false">https://medium.com/p/41ee00d55be8</guid>
            <category><![CDATA[notes-from-industry]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[bayesian-statistics]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[a-b-testing]]></category>
            <dc:creator><![CDATA[Itamar Faran]]></dc:creator>
            <pubDate>Mon, 21 Jun 2021 12:52:06 GMT</pubDate>
            <atom:updated>2021-11-04T08:46:23.733Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/notes-from-industry">Notes from Industry</a></h4><h3>How To Do Bayesian A/B Testing at Scale</h3><h4>Analyzing Bayesian A/B Tests Without Compromising on Performance</h4><p>If you’ve read <a href="https://towardsdatascience.com/why-you-should-switch-to-bayesian-a-b-testing-364557e0af1a">my previous post</a>, you already know why I think you should move to Bayesian A/B testing. In this post, I give a short overview over the statistical models behind Bayesian A/B tests, and present the ways we implemented them at Wix.com — where we deal with a massive scale of A/B tests.</p><p>I wrote some practical examples in Python along this post. You can easily replicate them by copy-pasting the code from <a href="https://github.com/itamarfaran/public-sandbox/blob/master/bayesian_blog/main.py">here</a> and <a href="https://github.com/itamarfaran/public-sandbox/blob/master/bayesian_blog/orthogonal.py">here</a>. While the code is rather simple, the formulas I derive can be a little advanced; however I don’t think it is critical to understand all the derivations to use the code.</p><p>If you don’t have a background in Statistics and <a href="https://en.wikipedia.org/wiki/Bayesian_inference">Bayesian inference</a>, I recommend you read the articles in my references section. Also, a lot of concepts I use have links to Wikipedia, where they are explained in more detail.</p><h3>The Bayesian A/B Test Model</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*N9sXeyes6ImYAjfPqujj4w.png" /><figcaption>Visualization of results of a Bayesian A/B test. Left: the posterior distribution of two rates. Right: The posterior distribution of the uplift (ratio), approximated using the Central Limit Theorem. <em>Image by Author</em></figcaption></figure><p>Bayesian statistics is built on two main concepts: the <a href="https://en.wikipedia.org/wiki/Prior_probability">prior distribution</a> — what we “know” about the KPI before the test, and the <a href="https://en.wikipedia.org/wiki/Posterior_probability">posterior distribution</a> — what we know after gathering data. We find the posterior by “<a href="https://en.wikipedia.org/wiki/Bayes%27_theorem">updating</a>” the prior with our data. In an A/B test context, we try to keep it simple and use prior distributions that can be easily updated (<a href="https://en.wikipedia.org/wiki/Conjugate_prior">conjugate priors</a>), like the Beta-Binomial model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cG0bB1Go8u9ELbAh9PY9hA.jpeg" /><figcaption>Eq. 1: <a href="https://en.wikipedia.org/wiki/Bayes%27_theorem">Bayes’ Theorem</a>. In an A/B test, <em>θ</em> is the KPI in question, such as the Conversion Rate or the Average Revenue per User. Image by Author</figcaption></figure><p>The <a href="https://en.wikipedia.org/wiki/Beta-binomial_distribution">Beta-Binomial</a> model is used to model binary data such as conversions or clicks (“did the user convert or not?”). I’ll also review the <a href="https://en.wikipedia.org/wiki/Multivariate_normal_distribution#Bivariate_case_2">Normal-Normal</a> model, which is used for continuous data (for example revenue per user).</p><p>In the Beta-Binomial model we assume that the conversion rate <em>Pᴀ</em> has a <a href="https://en.wikipedia.org/wiki/Beta_distribution">Beta distribution</a> with parameters <em>α</em> and <em>β</em>. A <a href="https://en.wikipedia.org/wiki/Beta_distribution#Bayes-Laplace_prior_probability_(Beta(1,1))">common choice</a> for <em>α</em> and <em>β</em> is 1, which results with a uniform distribution (sometimes referred to as an “<a href="https://en.wikipedia.org/wiki/Prior_probability#Uninformative_priors">uninformative prior</a>”). I discuss the choice of the prior in more detail in the appendix, but for now let’s continue as if they have already been chosen.</p><p>We denote the count of converted users with <em>Xᴀ </em>and the count of all users (converted or not) with <em>nᴀ</em>, and we model <em>Xᴀ | Pᴀ ~ Bin(nᴀ, pᴀ). </em>Since <em>Pᴀ ~ Beta(α, β)</em>, by using <a href="https://en.wikipedia.org/wiki/Bayes%27_theorem">Bayes’ theorem</a> we get:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6UfZ_xwwpQKgR_1hlIR69Q.jpeg" /><figcaption>Eq. 2: Posterior of the Beta Distribution</figcaption></figure><p>Meaning we “update” our prior by adding the number of successes to <em>α</em> and the number of failures to <em>β</em>. Pretty simple, don’t you think?</p><p>In the Normal-Normal model, we assume that the expected revenue per user (or any other continuous metric) <em>μᴀ</em> has a <a href="https://en.wikipedia.org/wiki/Normal_distribution">Gaussian distribution</a> with parameters <em>μ₀</em> and <em>σ²₀/n₀</em>. Here we denote the sample average with <strong><em>X</em></strong><em>̄ᴀ</em> and the sample standard deviation with <em>sᴀ</em>, and we assume that <strong><em>X</em></strong><em>̄ᴀ</em> is approximately Normally distributed as well. This time, the update rule is a bit more tricky:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bwwBLgeuR2y9AcVtK05XMg.jpeg" /><figcaption>Equation 3: Posterior of the Gaussian distribution</figcaption></figure><p>The posterior’s expected value is a weighted average of the Prior mean and the sample mean, and the weights are inverse to their variances.</p><p>All these formulas may seem a little overwhelming, but the calculations in NumPy are pretty simple:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a80bff1393f686be143f6b1f224e17e5/href">https://medium.com/media/a80bff1393f686be143f6b1f224e17e5/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/65d1ef2c1ca366a58d338f20ad03ca4d/href">https://medium.com/media/65d1ef2c1ca366a58d338f20ad03ca4d/href</a></iframe><h3>How To Calculate (Most Metrics)</h3><p>Now that we’ve found the posterior distributions of <em>Pᴀ</em> and <em>Pʙ</em>, we want to calculate inferential metrics, such as Credibility Intervals, the Probability B is Better and each version’s Risk. The most common and simple way to do this is by using <a href="https://en.wikipedia.org/wiki/Monte_Carlo_method">Monte Carlo simulations</a>. But at Wix.com we have thousands of KPIs in hundreds of A/B tests every day, and using simulations isn’t scalable. Instead, we use two methods: <a href="https://en.wikipedia.org/wiki/Gaussian_quadrature">Gaussian Quadratures</a> (more on that later) and… <a href="https://en.wikipedia.org/wiki/Central_limit_theorem">Central Limit Theorem</a> (CLT) approximations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/260/1*7Dq1zJo1jkNU6bMEtUPG9w.jpeg" /><figcaption>Image by <a href="https://memegenerator.net/instance/58400233/tropic-thunder-downey-you-went-full-bayesian-never-go-full-bayesian">memegenerator.net</a></figcaption></figure><p>You might be thinking — isn’t the CLT more of a frequentist kind of thing? And can the CLT even be applied to the Beta distribution? You’ll be right to question that. However, when you have a sample size of thousands in every A/B test, the CLT approximation for the Beta distribution is “good enough”. You can validate me by doing your own simulations (or looking at <a href="https://github.com/itamarfaran/public-sandbox/blob/master/bayesian_blog/compare.py">mine</a>).</p><p>Here we’re going to approximate the Credibility Intervals and the Probability B is Better with the CLT. Again, while the formulas can be a bit exhaustive, the code with SciPy is almost a one-liner. We denote the difference <em>Pʙ - Pᴀ </em>with<em> D₁</em> and the log-ratio <em>log Pʙ/Pᴀ</em> with <em>D₂</em>. The latter is used when we report the <a href="https://en.wikipedia.org/wiki/Relative_risk">relative uplift</a> instead of the percentage-point difference. We approximate <em>D₁</em>’s<em> </em>and <em>D₂</em>‘s distribution with:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fQ0daEuiwWjpqIp7HkQ0fg.jpeg" /><figcaption>Eq. 4: CLT approximation of the difference and ratio between two Beta random variables</figcaption></figure><p>To complete the formulas above, we denote the <a href="https://en.wikipedia.org/wiki/Digamma_function">digamma function</a> and the first <a href="https://en.wikipedia.org/wiki/Polygamma_function">polygamma function</a> with <em>ψ</em> and <em>ψ</em>₁ respectively. You don’t really need to know what they do, just where to find them in SciPy! The equations below are simply copied from Wikipedia:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pFupCKlp02zbt0xkQV22aw.jpeg" /><figcaption>Eq. 5: Moments of a Beta variable and it’s logarithm, taken from Wikipedia</figcaption></figure><p>Using these formulas, we can easily calculate the probability B is better with <em>P(D₁ &gt; 0)</em>, and the Credibility Interval with the quantiles of <em>D₁</em> or <em>D₂:</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8c98f4e30f70a5f4598a72997098aba9/href">https://medium.com/media/8c98f4e30f70a5f4598a72997098aba9/href</a></iframe><p>For the Normal-Normal case, <em>D₁</em> is pretty <a href="https://en.wikipedia.org/wiki/Normal_distribution#Operations_on_two_independent_normal_variables">straightforward</a>. But what if we want to use the relative difference instead? We’ve decided to use the <a href="https://en.wikipedia.org/wiki/Delta_method">Delta method</a> to find the approximate distribution of <em>ln μᴀ</em>. Now you might be thinking — how the heck can I take the logarithm of a Gaussian random variable? And (again) you’ll be completely right — the Gaussian distribution’ support includes negative numbers and its logarithm isn’t properly defined. But (again) with a sample size of tens of thousands, it’s a “good enough” approximation as <em>μᴀ</em>’s distribution is “pretty far away” from 0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tLKLDB8E4g7PkgaeqI041w.jpeg" /><figcaption>Eq. 6: Distribution of the difference between two Gaussian variables, and the approximate distribution of their ratio found with the Delta method.</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e34bf29931166878676dc53aaa576f16/href">https://medium.com/media/e34bf29931166878676dc53aaa576f16/href</a></iframe><h3>Calculating the Risk</h3><p>Now we have finally arrived to the important part: The Risk measure is the most important measure in Bayesian A/B testing. It replaces the P-value as a <a href="https://en.wikipedia.org/wiki/Decision_rule">decision rule</a>, but also serves as a <a href="https://en.wikipedia.org/wiki/Stopping_time">stopping rule</a> — since the Bayesian A/B test has a dynamic sample size.</p><p>It is interpreted as “When B is worse than A, If I choose B, how many conversions am I expected to lose?”, and formally, it is defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XaSEpR4MCYcdmeJ3turQdg.jpeg" /><figcaption>Eq. 7: Formal definition of the Risk in a Bayesian A/B test</figcaption></figure><p>with <strong><em>1</em></strong> as an indicator function. Notice the integral on the third line — I really hate that integral! It doesn’t have an analytical solution, and <strong>we can’t approximate it with the CLT</strong>. But as I previously mentioned, Monte-Carlo simulations are NOT an option for us. So what did we do?</p><p><a href="https://en.wikipedia.org/wiki/Gaussian_quadrature">Gaussian quadratures</a> (GQ) are a cool way of approximating integrals with a weighted sum of a small number of nodes. The nodes and weights are calculated by the GQ algorithm. Formally, we find <em>n</em> nodes (<em>x</em>) and weights (<em>w</em>) that best approximate the integral of <em>g </em>with a summation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EooZbABDUiuMTLbYl2bL5A.jpeg" /><figcaption>Eq. 8: A Gaussian quadrature approximates an integral with a weighted sum of selected nodes. <em>-∞ ≤ a &lt; b ≤ +</em>∞</figcaption></figure><p>As we saw earlier, the Risk metric is an integral — so we can try to approximate it with a GQ. First, let’s simplify it’s expression:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TiiVJHcujcJFnh1r8JS1iQ.jpeg" /><figcaption>Eq. 9: Simplification of the Risk metric</figcaption></figure><p>Now <em>ξᴀ </em>is an integral we <em>can</em> approximate with a Gaussian quadrature! We can accurately calculate the Risk with about 20 nodes, instead of thousands in a Monte-Carlo simulation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pmIo07XnkOFLlU7SHIaxOw.jpeg" /><figcaption>Eq. 10: Simplified expression of the Risk metric</figcaption></figure><p>We implemented this approximation using <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.roots_hermitenorm.html">scipy.special.roots_hermitnorm</a> and a work-around over <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.roots_sh_jacobi.html">scipy.special.roots_sh_jacobi</a> (more on the work-around in my notes). This isn’t <em>very</em> fast, but it’s the fastest way we found.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4f76c0aa7363d136986d4255d16e13a9/href">https://medium.com/media/4f76c0aa7363d136986d4255d16e13a9/href</a></iframe><h3>Summary</h3><p>The mathematics and programming in Bayesian A/B testing are a bit more challenging than in the Frequentist framework. However, as I argued in <a href="https://towardsdatascience.com/why-you-should-switch-to-bayesian-a-b-testing-364557e0af1a">my previous post</a>, I think it is worth it. Although it is more computationally expensive, the added clarity and interpretability of the A/B test gives an enormous value for anyone who uses them.</p><p>In this post, I reviewed the basics of the Beta-Binomial and Normal-Normal models for Bayesian A/B testing, and presented some of the approximations we implemented at Wix.com. While these approximations may not be 100% accurate, they are “good enough” when experiments have thousands of users, and they allowed us to support Bayesian A/B testing on a large scale.</p><p>This post focused on how to calculate Bayesian metric of an A/B test, and not on how to analyze it or when to stop it. If you want to read more on those issues, the posts in my references give an excellent overview.</p><h3>References</h3><p>Here are some of the posts that I read when I was first learning about Bayesian A/B testing. The two latter posts are a bit more critical on the framework, and I recommend reading them especially.</p><ul><li><em>“</em><a href="https://medium.com/convoy-tech/the-power-of-bayesian-a-b-testing-f859d2219d5"><em>The Power of Bayesian A/B Testing</em></a><em>”</em> by Michael Frasco</li><li><em>“</em><a href="https://towardsdatascience.com/exploring-bayesian-a-b-testing-with-simulations-7500b4fc55bc"><em>Bayesian A/B testing — a practical exploration with simulations</em></a><em>”</em> by Blake Arnold</li><li><em>“</em><a href="https://cdn2.hubspot.net/hubfs/310840/VWO_SmartStats_technical_whitepaper.pdf"><em>Bayesian A/B Testing at VWO</em></a><em>”</em> by Chris Stucchio</li><li><em>“</em><a href="https://cxl.com/blog/bayesian-frequentist-ab-testing/"><em>Bayesian vs. Frequentist A/B Testing: What’s the Difference?</em></a><em>”</em> by Alex Birkett</li><li><em>“</em><a href="http://varianceexplained.org/r/bayesian-ab-testing/"><em>Is Bayesian A/B Testing Immune to Peeking? Not Exactly</em></a><em>”</em> by David Robinson</li></ul><p>The last post is probably my favorite, I’ve read it a couple of times. Particularly, by recreating it’s simulations, I’ve learnt about the effect of choosing different priors on the error rate when stopping the Bayesian A/B test dynamically (more on that subject in the appendix).</p><p><strong>I’ve also built some cool R Shiny apps related to Bayesian A/B testing, using <em>only </em>the formulas presented in this post:</strong></p><ul><li><a href="http://itamar-faran.shinyapps.io/bayesian-binary-calculator-app/">Beta-Binomial A/B Test Calculator</a></li><li><a href="http://itamar-faran.shinyapps.io/bayesian-continuous-calculator-app">Normal-Normal A/B Test Calculator</a></li><li><a href="http://itamar-faran.shinyapps.io/bayesian-runtime-simulator">A cool app comparing Runtime &amp; Accuracy between Bayesian &amp; Frequentist A/B testing</a></li></ul><p>All the code I used in this post can be found in <a href="https://github.com/itamarfaran/public-sandbox/tree/master/bayesian_blog">this directory</a>.</p><h3>Appendix: On The Choice of the Prior</h3><p>There is a niche debate about the importance of choosing a prior in Bayesian A/B testing. To my opinion, using an informative and suitable prior has great importance, and here&#39;s why:</p><ol><li>I view the prior as a sort of “<a href="https://en.wikipedia.org/wiki/Regularization_(mathematics)">regularization</a>” on the A/B test, and it plays a major role when dealing with the <a href="https://en.wikipedia.org/wiki/Multiple_comparisons_problem">multiple comparisons issue</a>. Yes — it is <a href="http://www.stat.columbia.edu/~gelman/research/published/multiple2f.pdf">still an issue</a>, even though we’re not using the Frequentist framework..</li><li>Since we use Bayesian A/B testing <a href="https://gopractice.io/blog/peeking-problem/">sequentially</a> (looking on the results every day and stopping once the Risk is below the threshold), using a “weak” (low) prior increases our error rate. See David Robinson’s post in my references.</li><li>One cannot avoid the dilemma of choosing a prior by using an uninformative prior. <strong>Choosing an uninformative prior is still a choice of a prior</strong>, just a very lousy one.</li></ol><p>I think it is important to find and use an appropriate prior in a Bayesian model. At Wix, we did so by automating a “prior builder” process, where we automatically query historical data from the past few weeks and build a prior from that data, as Frasco and Arnold suggest in their posts. Only when there isn’t enough data, we fallback to an “uninformative” prior (<em>α=1, β=1</em> for the Beta distribution, <em>μ₀=0, σ₀=1, n₀=0</em> for the Gaussian distribution).</p><h3>Notes</h3><p><strong><em>Ordered by descending level of importance</em></strong></p><ul><li>scipy.special.roots_sh_jacobi encounters an integer overflow error when <em>α</em> or <em>β</em> are too large. I’ve copied the source code and did the “log trick” to make it work, you can see it <a href="https://github.com/itamarfaran/public-sandbox/blob/master/bayesian_blog/orthogonal.py">here</a>. I also have a PR (<a href="https://github.com/scipy/scipy/pull/12580">here</a>) for that issue.</li><li>If you’re programming in R, you can approximate the Risk with a Gaussian quadrature using the <a href="https://www.rdocumentation.org/packages/statmod/versions/0.5/topics/gauss.quad.prob">statmod::gauss.quad.prob</a> function.</li><li>The parameters <em>μ₀=0, σ₀=1, n₀=0</em> <strong>aren’t </strong>the Gaussian distribution’s uninformative prior, but result with a very “weak” prior. The reason we don’t use the <a href="https://en.wikipedia.org/wiki/Jeffreys_prior#Examples">actual uninformative prior</a> is because it doesn’t result with a conjugate prior, which makes calculations much more difficult.</li><li>We found that the Normal-Normal model suits most of our continuous KPIs. While there are more sophisticated models (for example a product of an Exponential and Beta distribution, <a href="https://towardsdatascience.com/bayesian-ab-testing-part-ii-revenue-1fbcf04f96cd">here</a>), we found that in a large sample the Gaussian approximation yields similar results — and its much simpler to calculate (Occam’s razor to the rescue). Once again I invite you to validate me by doing your own simulations, or look at mine <a href="https://github.com/itamarfaran/public-sandbox/blob/master/bayesian_blog/normal_approx.py">here</a>.</li><li>In my previous post I wrote that a lot of metrics aren’t that numerically different between the Bayesian and Frequentist models. Indeed, they have much resemblance, and in fact — if they didn’t we would have been worried!<br>Here’s a nice example: In a large sample the moments of the Beta distribution in Eq. 5 are almost identical to the CLT approximations of the rate’s estimate and <a href="https://en.wikipedia.org/wiki/Relative_risk">its logarithm</a> in a proportion test. Setting <em>α, β = 0 </em>for simplicity, one gets:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*labY2qR8xeqPpQ0ciKAZZA.jpeg" /><figcaption>Eq. 11: Resemblance between the Beta distribution’s moments and the CLT approximations of the sample rate</figcaption></figure><ul><li>Notice that in the Normal-Normal model I presented, the standard deviation <em>σ₀</em> doesn’t have a posterior. The reason is that at Wix we don’t have an interest in inferring the standard deviation, and we treat it as a nuisance parameter. If you do need to infer it, I refer you to the <a href="https://en.wikipedia.org/wiki/Normal-inverse-gamma_distribution#Posterior_distribution_of_the_parameters">Normal-Inverse-Gamma</a> prior.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=41ee00d55be8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-do-bayesian-a-b-testing-fast-41ee00d55be8">How To Do Bayesian A/B Testing, FAST!</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why You Should Switch to Bayesian A/B Testing]]></title>
            <link>https://medium.com/data-science/why-you-should-switch-to-bayesian-a-b-testing-364557e0af1a?source=rss-153789513a7b------2</link>
            <guid isPermaLink="false">https://medium.com/p/364557e0af1a</guid>
            <category><![CDATA[a-b-testing]]></category>
            <category><![CDATA[product-management]]></category>
            <category><![CDATA[bayesian-statistics]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[hands-on-tutorials]]></category>
            <dc:creator><![CDATA[Itamar Faran]]></dc:creator>
            <pubDate>Mon, 07 Jun 2021 12:22:54 GMT</pubDate>
            <atom:updated>2021-06-08T18:51:57.234Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/hands-on-tutorials">Hands-on Tutorials</a></h4><h4>When Statistics &amp; Business share the same language</h4><h3>What is A/B Testing?</h3><p>Almost everyone hated learning statistics (well, maybe except <em>some</em> statisticians). With all those distributions and critical values that we needed to memorize, we just ended up with a headache. You might have swore not to ever touch the subject again; that is, until you had to analyze an A/B test.</p><p>A/B testing is the “fun” name for <a href="http://en.wikipedia.org/wiki/Randomized_controlled_trial">Randomized Controlled Trials</a>, where we randomly assign users to two (or more) groups and measure a metric (<em>KPI</em>) in each of them. For example, we can randomly present users a green “buy now!” button or a red one, and measure whether the click rate (or purchase rate) is higher in one of them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/536/1*tLlNupSyD4Q5OlW45s-FwA.jpeg" /><figcaption><em>Image by author</em></figcaption></figure><p>Eventually, we need statistics when we analyze the differences between the two groups. When we see that one button is performing better, we can’t know if it’s “real” or just by chance; because even if the color doesn’t affect the click rate at all, we don’t expect the groups to have <em>exactly</em> the same rate. Statistics help us distinguish between these two options.</p><p>The question we ask the A/B test is “which version is better?”. If you’re using a <a href="http://en.wikipedia.org/wiki/Frequentist_inference">Frequentist</a> framework, the answer you’ll get when looking at the P-value is “well, if there is no difference between the buttons, then the probability to see an uplift such as yours (or more extreme) is <em>x%</em>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*UO2TF_i1TO-FGEHSk2PryQ.jpeg" /><figcaption>A typical reaction when hearing the definition of the P-value for the first time. Image taken from <a href="https://en.meming.world/wiki/Confused_Nick_Young">en.meming.world</a></figcaption></figure><h3>It Doesn’t Have to be Complicated</h3><p>Statistics doesn’t have to be that complicated. It turns out that an alternative framework can give us the following answer: “The probability that the green button is better is <em>y%</em>. This is what you’re risking by choosing it”. This framework is called the <a href="http://en.wikipedia.org/wiki/Bayesian_inference">Bayesian</a> framework, and it is gaining more and more popularity in the online industries.</p><blockquote>The most important advantage of Bayesian statistics is that it is understandable</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*N9sXeyes6ImYAjfPqujj4w.png" /><figcaption>Visualization of results from a Bayesian A/B test. Left: The uncertainty in the rates is easily visualized (“The <strong>real</strong> rates are more probable to be where the curves are higher”). Right: The <em>probability that B</em> is winning is easy to comprehend (“The green area is much larger than the red, but the latter isn’t negligible”). <em>Image by author.</em></figcaption></figure><p>While there are many advantages (and some disadvantages) for the framework, I think the most important advantage of Bayesian A/B testing is that it is <em>understandable</em>. It answers exactly what we need to know in the case of uncertainty: What are the chances we are wrong, and what are the risks in that case.</p><h3>What’s the Difference?</h3><p>In short, in the frequentist framework we assume that there are two alternative worlds: One where there isn’t a difference between the red and green buttons (<em>the null hypothesis</em>), and one where there is. We assume we live in the first world (<em>we </em><strong><em>assume</em></strong><em> the null hypothesis is true</em>), and we want to disprove that, with a certain level of certainty (<em>the significance level</em>). What if we didn’t manage to prove that? That doesn’t necessarily mean that we’re in the first world (<em>an insignificant result doesn’t prove the null hypothesis</em>).</p><p>The Bayesian framework has a completely different point of view. We start by saying “<em>the click rate of the red and green buttons can be any rate between 0% and 100%, with equal chance</em>” (this is what we call <em>the prior</em>). This means that each button initially has a 50% chance to be better than the other. As we start gathering data, we update our knowledge, and we can say things like “<em>Given the data I have observed, I now think there is a 70% chance that the green button is better</em>”. We call this <em>the posterior</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*maBYke4rgJgQQeNIgsT40Q.png" /><figcaption>The posterior distribution of the uplift (the difference between rates in percentage) between A and B. The sample size increases from left to right. Even though the uplift is kept the same, the increased sample size results with higher confidence in our results. With Bayesian A/B testing, it is easy to communicate this confidence. <em>Image by author.</em></figcaption></figure><p>What I love about the Bayesian framework is that it embraces uncertainty. To my view, the common interpretation of frequentist inference is almost “deterministic”. It’s funny to say that on a statistical framework, but think about the way we <em>interpret</em> hypothesis testing — if the P-value is lower than .05, the result is significant (“real”); if it’s higher, then the result “isn’t real”. But real life <em>is</em> uncertain; we can’t just set a .05 threshold and say “the P-value told us so”. The Bayesian framework embraces uncertainty — and tells you “there’s a 95% chance the green button is better — but there’s a 5% chance it’s not. The choice is yours”.</p><h3>Bayesian Metrics</h3><p>The funny thing is that most metrics aren’t that numerically different between the two methods, at least in the A/B test context; however they completely differ in their interpretation. In the frequentist method we have a Confidence Interval; the matching Bayesian metric is the Credibility Interval. I personally view the “Probability B is Better” as the matching metric of the P-value: both are computed very similarly, but their meanings are totally different!</p><p>However, Bayesian A/B testing does have a metric that does not have a parallel in the frequentist framework: The <em>Risk</em>. We calculate the risk for both A and B, and it’s interpretation is: “<em>If I chose B when B is actually worse than A, how much will I lose?</em>”. This metric is also used as a decision rule in the A/B test.</p><p>The Risk has two major advantages over the P-value. First of all, it’s threshold is in business jargon (“what is the cost I am willing to pay if I’m wrong?”), while the frequentist significance level is statistical gibberish (“what is the type-I error rate I am willing to accept?”). Second, it’s much more robust to sequential examination (“peaking”) than the P-value, and as a result <strong><em>the sample size doesn’t have to be pre-determined when using the Bayesian framework</em></strong>. This is also a major advantage of Bayesian A/B testing.</p><h3>Summary</h3><p>A/B testing doesn’t have to be confusing. Using the Bayesian framework, we can answer exactly the question we ask ourselves when we need to make a decision: “Which version is better, what is the chance I’m wrong, and what is the price I will pay if that is the case”. This answer is much more business oriented, and <em>any</em> colleague in our company can comprehend it easily, no matter what is their role or expertise.</p><p>People are Bayesian. We ask ourselves “<em>will the chicken cross the road?</em>”, not “<em>Am I in the null reality where the chicken doesn’t cross the road or the alternative one where it does with at least 95% confidence?</em>”. Why should we treat online A/B testing differently?</p><h3>References</h3><p>I really tried to keep it simple in this post and not to get too deep into the maths. If you want to dig (a little bit) deeper, I really recommend these posts:</p><ul><li><em>“</em><a href="https://medium.com/convoy-tech/the-power-of-bayesian-a-b-testing-f859d2219d5"><em>The Power of Bayesian A/B Testing</em></a><em>”</em> by Michael Frasco</li><li><em>“</em><a href="https://towardsdatascience.com/exploring-bayesian-a-b-testing-with-simulations-7500b4fc55bc"><em>Bayesian A/B testing — a practical exploration with simulations</em></a><em>”</em> by Blake Arnold</li></ul><h4>Teaser</h4><p>Stay tuned to my next post, where I describe how we implemented Bayesian A/B testing at Wix, where performance at a large scale is critical.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=364557e0af1a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/why-you-should-switch-to-bayesian-a-b-testing-364557e0af1a">Why You Should Switch to Bayesian A/B Testing</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>