<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Meraldo Antonio on Medium]]></title>
        <description><![CDATA[Stories by Meraldo Antonio on Medium]]></description>
        <link>https://medium.com/@meraldo.antonio?source=rss-65b4354e2ed8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*UvZGaLbkd2zrM3Y8xdfkHg.jpeg</url>
            <title>Stories by Meraldo Antonio on Medium</title>
            <link>https://medium.com/@meraldo.antonio?source=rss-65b4354e2ed8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 21:45:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@meraldo.antonio/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Bayesian Customer Lifetime Values Modeling using PyMC3]]></title>
            <link>https://medium.com/data-science/bayesian-customer-lifetime-values-modeling-using-pymc3-d770676f5c06?source=rss-65b4354e2ed8------2</link>
            <guid isPermaLink="false">https://medium.com/p/d770676f5c06</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[business-development]]></category>
            <category><![CDATA[editors-pick]]></category>
            <dc:creator><![CDATA[Meraldo Antonio]]></dc:creator>
            <pubDate>Thu, 21 Apr 2022 17:10:20 GMT</pubDate>
            <atom:updated>2022-04-25T19:24:56.160Z</atom:updated>
            <content:encoded><![CDATA[<h4>Implementing BG-NBD, a probabilistic hierarchical model, using PyMC3 to analyze customer purchase behavior</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*N89xjx6J1K2wWF5K" /><figcaption>Source: <a href="https://unsplash.com/photos/pQyTChJwEDI">Unsplash</a></figcaption></figure><p>Customer lifetime value (CLV) is the total worth of a customer to a company over the length of their relationship. The collective CLV of a company&#39;s customer base reflects its economic value and is often measured to evaluate its future prospects.</p><p>While many ways to estimate CLV exist, one of the most influential models to surface in the past couple of decades is the Beta-Geometric Negative Binomial Distribution (BG-NBD).¹ <strong>This framework models customers’ repeat purchasing behavior using the </strong><a href="https://en.wikipedia.org/wiki/Gamma_distribution"><strong>Gamma</strong></a><strong> and </strong><a href="https://en.wikipedia.org/wiki/Beta_distribution"><strong>Beta</strong></a><strong> distributions. </strong><a href="https://medium.com/p/d5111cb52dd">Part 1</a> of this series of articles already explored the mathematics behind this model. Meanwhile, <a href="https://medium.com/towards-data-science/modeling-customer-lifetime-value-with-lifetimes-71171a35f654">part 2</a> introduced <a href="https://lifetimes.readthedocs.io/en/latest/"><em>lifetimes</em></a>, a library that allows us to conveniently fit a BG-NBD model and obtain its maximum likelihood estimate (MLEs) parameters. I’d suggest you read these parts if you haven’t as the upcoming content builds on top of them.</p><p><strong>This last part of the series presents an alternative way to implement BG-NBD — one that relies on Bayesian principles. </strong>We’ll be using <a href="https://docs.pymc.io/en/v3/"><em>PyMC3</em></a> to code this implementation. While the principal aim of this article is to elucidate BG-NBD through the Bayesian lens, some general points on the Bayesian worldview and <em>PyMC3</em> framework will also be presented. Here is our agenda:</p><ol><li>First, we’ll argue that the Bayesian implementation offers some advantages over the frequentist.</li><li>Second, we’ll provide an implementation ‘blueprint’ that will guide our coding steps.</li><li>Third, we’ll spend some time looking into prior selection<strong> </strong>— a key component in Bayesian modeling.</li><li>Next, we’ll go over the <em>PyMC3 </em>code and its intricacies, including the famous Monte-Carlo Markov Chain (MCMC) algorithm.</li><li>Finally, we’ll analyze the output of our Bayesian BG-NBD model and compare it with the output of <em>lifetimes</em>.</li></ol><p>Let’s get started!</p><h3>Why Bayesian?</h3><p>You might be wondering as to why there is a need for another implementation if we already have the user-friendly <em>lifetimes </em>in our arsenal. The answer lies in the contrast between frequentist and Bayesian inference and the advantages offered by the latter.</p><p>To illustrate these differences, let’s imagine the frequentist and Bayesian frameworks as two programming functions that attempt to model statistical problems (such as CLV estimation). We’ll see that these functions have crucial differences in their inputs and outputs.</p><p>Both functions take as an input the dataset (the observations). <strong>However, the Bayesian function requires additional inputs in the form of <em>prior</em> distributions.</strong> Priors reflect our initial assumptions of the parameters of the models and will be updated after the observations have been taken into account.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fgyg0a3BEMGxeEX9qatajQ.png" /></figure><p>The inclusion of priors is a double-edged sword.<strong> On the one hand, priors offer a way for users to use their domain knowledge to ‘guide’ the model into finding the correct parameters.</strong> In BG-NBD for example, if we’re sure that most of our customers make purchases at a rate λ of 4 transactions/week, we can supply priors that would lead to a Gamma distribution with a mean of 4. If this intuition is only slightly off-the-mark (for example, if the real λ is 3.5 transactions/week), the Bayesian framework will allow us to arrive at the correct values with only few data points.</p><blockquote><strong>On the other hand, since there isn’t a single ‘correct’ way to choose the priors, the Bayesian modeling process is subjective and variable.</strong> Two statisticians sharing the same data but employing different priors could end up getting completely different outcomes. This subjectivity is among the chief reasons why some practitioners eschew Bayesian. Nevertheless, as more data points are included in the modeling, the influence of priors diminishes.</blockquote><p>The two functions also differ in their outputs. They both return the parameter estimates of the model but in different forms. The frequentist returns them as point estimates. For example, we’ve seen how the frequentist library<em> </em><a href="https://lifetimes.readthedocs.io/en/latest/"><em>lifetimes</em></a> uses MLE to calculate the point estimates of <em>r, α</em>, <em>a</em> and <em>b</em>.<strong> In contrast, the Bayesian returns them as probability distributions.</strong> These distributions, called posteriors, represent our updated beliefs about the parameters after taking into account the observations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Th9GhwKWzqIOF53pw-0q0g.png" /></figure><p>The obtainment of posteriors is another aspect where the Bayesian framework outshines the frequentist. <strong>With posteriors, it is easy to visualize the most plausible values for the parameters and quantify any uncertainty in our estimates.</strong> Additionally, the prior-to-posterior cycle provides a principled way to update the parameters if additional observations become available.</p><h3>Implementation Blueprint</h3><p>We’ve seen in Part 1 that the BG-NBD model is a hierarchical probabilistic model. <em>Hierarchical </em>here refers to the fact that the parameters of some distributions are modeled by other distributions (‘parent’ distributions). We’ll soon see that <em>PyMC3 </em>allows us to express these parent-child relationships in code.</p><p>A blueprint is provided below to guide the coding steps. This blueprint walks through the model’s logical flow, starting from its inputs (the data and the prior distributions) and ending with its outputs (the posteriors).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3r1OAAXD3BOmjDiZ" /></figure><p>Some remarks on the above blueprint:</p><ol><li><strong>The Gamma distribution represents the distribution of transaction rate λ in the customer population.</strong> It is parameterized by two parameters: <em>r</em><strong> </strong>and <em>α</em>. We’ll supply priors for both <em>r</em><strong> </strong>and <em>α</em> that<strong> </strong>will be updated by our observations.</li><li><strong>The Beta distribution represents the distribution of deactivation probability <em>p</em> in the customer population.</strong> It is parameterized by two shape parameters: <em>a</em><strong> </strong>and <em>b</em><strong>.</strong> Similarly, we’ll provide priors for these parameters.</li><li><strong>The <em>λ</em> from the Gamma distribution and the <em>p</em> from the Beta distribution contribute to the likelihood function, which represents the probability of observing the transactions in the dataset. </strong>This likelihood is <em>the</em> equation that binds our data and our theory.</li><li>An MCMC algorithm in <em>PyMC3</em> will enable us to sample the posterior distributions of <em>r, α, a</em> and<em> b</em>. We’ll talk more about this algorithm later.</li></ol><p>Details about the above distributions were provided in <a href="https://towardsdatascience.com/customer-lifetime-value-estimation-via-probabilistic-modeling-d5111cb52dd">part 1</a>; take a look at it if you’re unfamiliar with them!</p><h3>Selection of priors</h3><h4>Consideration</h4><p>As discussed previously, we’ll need to supply a prior each for <em>r, α, a</em> and<em> b</em>. <strong>We can think about these priors and their respective parameters as the ‘hyperparameters’ of our model. </strong>There are various considerations to keep in mind when choosing a prior:</p><ol><li>A prior can be formulated from existing knowledge, which in turn can come from domain expertise, historical observations or past modeling experiments.</li><li>When no information is available, an uninformative prior can be chosen to reflect an indifference among possible parameter values.</li><li>We can impose some constraints to guide our selection. For example, we can require that our prior distributions be symmetrical, strictly positive, heavy tailed, etc.</li></ol><p>In our case, the following are our considerations and assumptions:</p><ol><li>As for the purchase rate λ, we&#39;ll assume that we have a strong intuition that the bulk of our customer transact at a rate of 4 purchases/week. We’ll select Gamma priors that reflect this prior belief.</li><li>In contrast, we’ll assume no prior knowledge on the deactivation probability (<em>p</em>) of our customers. As such, we’ll choose uninformative Beta priors.</li><li>Because all of our parameters (<em>r, α, a</em> and <em>b)</em> can only be positive real numbers, the prior distributions we choose must assign zero probability density for negative numbers.</li></ol><h4>Truncated Normal Distribution</h4><p>A good candidate for the priors is the <a href="https://en.wikipedia.org/wiki/Truncated_normal_distribution">Truncated Normal distribution</a>. As its name suggests, the Truncated Normal is a Gaussian distribution that has been ‘truncated’. That is, its density <em>beyond</em> a specified left and a right limit is set to zero and its density <em>between</em> the limits is re-normalized to sum to 1. The distribution has four parameters: the left and right limits (often referred to as <em>A </em>and <em>B)</em>, the mean <em>μ</em> and the standard deviation <em>σ</em>.</p><blockquote>Truncated Normal is frequently used as prior as its four parameters provide an intuitive way to ‘guess’ a value. <em>μ</em> represents our most probable guess, <em>σ</em> reflects our uncertainty level and <em>A </em>and <em>B</em> designate the lower and the upper constraint, respectively.</blockquote><p>Here are some examples of the Truncated Normal:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B_H-XNOHQwopRnTYD7Xr9A.png" /></figure><h4>Gamma and Beta Priors</h4><p>With these considerations, let&#39;s now construct our Gamma priors. Here are the steps:</p><p>1. As mentioned before, we have a prior belief that the most likely value of the Gamma distribution (i.e. its mean) is 4 transactions/week.</p><p>2. <strong>We then select a specific combination of the Gamma parameters <em>r </em>and<em> α </em>that leads to a Gamma distribution with the desired mean of 4. </strong>We know that a <a href="https://en.wikipedia.org/wiki/Gamma_distribution">Gamma distribution&#39;s mean</a> can be calculated by the simple formula <em>μ</em> = <em>rα; </em>as such, any pair of positive numbers whose product is 4 can be chosen as candidates for <em>r </em>and<em> α</em>. Let&#39;s set <em>r </em>to be 8 and <em>α </em>to be 0.5. The Gamma distribution built using these parameters is plotted below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xu0rhifMDI_S27cpkRphBA.png" /></figure><p>3. These <em>r </em>and<em> α </em>values represent our <em>initial guesses</em>. Since we&#39;re not 100% sure of these guesses, we should present them as <em>distributions </em>as opposed to point estimates. It is these distributions that we refer to as <em>priors. </em>For reasons previously mentioned, we&#39;re going to use the Truncated Normal to represent these priors.</p><p>4. Now let’s focus on <em>r. </em>The Truncated Normal for <em>r</em> will have the following parameters:</p><ul><li><em>μ</em>: 8 (our most probable initial guess)</li><li><em>σ</em>: 7 (a rather wide standard deviation that reflects our high uncertainty level)</li><li><em>A</em>: 0 (since we know that <em>r </em>can&#39;t be negative)</li><li><em>B</em>: 40 (here, we pick an extreme right limit which we are <em>r </em>won’t exceed).</li></ul><p>The Truncated Normal built using these parameters is plotted below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZSS1ftpP7m3J9K_lZy-Saw.png" /></figure><p>5. The above process is repeated to get the priors for <em>α, a </em>and <em>b. </em>The workflow is visualized below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1LHL5znZJ4Tp2o8sWEjheQ.png" /></figure><p>Note that for the Beta parameters <em>a </em>and <em>b, </em>we have chosen the uninformative uniform distribution as priors to reflect that we have no prior knowledge on the deactivation probability (<em>p</em>) of our customers.</p><h3>Coding everything in PyMC3</h3><h4>Dataset</h4><p>Throughout this article, we will be using the sample dataset (<a href="https://github.com/CamDavidsonPilon/lifetimes">MIT licensed</a>) provided by <em>lifetimes</em>. This table records the transactions of <a href="https://en.wikipedia.org/wiki/CDNow">CDNow</a>, a dot-com era company that sold CDs. Here is how we load it:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1140bfecc33465e243cb1347f22b5736/href">https://medium.com/media/1140bfecc33465e243cb1347f22b5736/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*1QPCJ-m6wdp6lS1x3m_pdQ.png" /><figcaption><strong>transactions</strong></figcaption></figure><p>As discussed <a href="https://medium.com/@meraldo.antonio/71171a35f654#2b5f-e66d9b5b6df0">in part 2</a>, prior to modeling, this transactions table should be subjected to two prerequisite steps:</p><ol><li>converted to the canonical ‘RFM’ format, and</li><li>split into the calibration and holdout sets.</li></ol><p>We’ll employ the same splitting regime we used in part 2 to facilitate the comparison between the <em>lifetimes </em>and the <em>PyMC3 </em>estimates.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ce8d9760b43edc6bec5b3b94e8eb06fe/href">https://medium.com/media/ce8d9760b43edc6bec5b3b94e8eb06fe/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*keRVFfxZ3euukvyn5q9nPg.png" /><figcaption><strong>rfm_cal_holdout</strong></figcaption></figure><h4><em>PyMC3</em> in action</h4><p>With this theoretical framework set up, we’re now ready to start coding. We’ll use <em>PyMC3, </em>a powerful Bayesian modeling library. While we’ll provide a few pointers on <em>PyMC3 </em>itself, our focus will be on the BG-NBD implementation².</p><p>In <em>PyMC3</em>, a probabilistic model is represented by a <em>pm.Model()</em> object which is usually coded as a context manager. This model will contain several distributions represented by <em>Distribution </em>objects. These <em>Distribution </em>objects can have parent-child relationships among themselves, where the value of the parent object influences/determines the value of the child object.</p><p>The following is the BG-NBD implementation in <em>PyMC3</em>. As you can see, the code closely mirrors the theory presented earlier.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/65d4a80681ece82ea6776d5d18dfc5b8/href">https://medium.com/media/65d4a80681ece82ea6776d5d18dfc5b8/href</a></iframe><p>Having said that, two parts of the code might not be as straightforward and warrant further discussion. These parts, the custom likelihood function (line 30–41) and the MCMC sampling (line 43–62), are elaborated below.</p><h4>Custom likelihood function</h4><p><em>PyMC3</em> is equipped with many <em>Distribution </em>objects, each of which corresponds to a well known probability distribution such as the Beta or Gamma distribution. However, when it comes to a non-standard probability distribution such as the BG-NBD likelihood function, custom implementation is required.</p><p>As a reminder, the likelihood function in the BG-NBD model is shown below (its derivation was provided in <a href="https://medium.com/p/d5111cb52dd">part 1</a>):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*E0fs2Weup7B6PpUr" /></figure><p>You might remember from your statistics class that the maximum likelihood estimation (MLE) involves the multiplication, over our entire dataset, of the outputs of the likelihood function. Since each of these multiplication steps outputs a fraction, a direct calculation of likelihood might lead to floating point problems. To avoid this, it is customary to convert the likelihood function into a log-likelihood function, which is more convenient to optimize. The log-likelihood version of the above function is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*0566F1M2ZP6S9vOv" /></figure><p>It was <em>this</em> above formula that we implemented in the <em>logp </em>function. We then turned this function into a <em>Distribution </em>object by instantiating a new <a href="https://docs.pymc.io/en/v3/api/distributions/utilities.html#pymc3.distributions.DensityDist"><em>DensityDist</em></a><em> </em>object and passing on the callable during the instantiation. Because this <em>Distribution </em>object is one whose values are <em>known</em>, we also supplied these known values the <em>x </em>, <em>tₓ</em> and <em>T</em> from our data) during its instantiation.</p><h4>The MCMC algorithm</h4><p>After describing the model, we can execute an MCMC algorithm to sample the posterior distributions. MCMC is a family of simulation algorithms that allows the sampling from a distribution whose closed-form PDF or PMF is unknown. It works by constructing a Markov chain — a multi-state system that allows transitions from one state to another according to specific probabilistic rules. The chain is designed so that its steady state matches the distribution to be sampled.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*19p90LbT_pe98C76.gif" /><figcaption>Animation of MCMC from <a href="https://blog.stata.com/2016/11/15/introduction-to-bayesian-statistics-part-2-mcmc-and-the-metropolis-hastings-algorithm/">the Stata Blog</a>³</figcaption></figure><p>We start the chain construction and sampling using <em>pm.sample().</em> Upon completion, the function returns a <em>trace</em> object that contains the drawn samples. If the sampling process went well (i.e. a steady state was reached), these samples would represent our desired posteriors.</p><p>Before further analysis, we should confirm the convergence (the steady-state attainment) of the algorithm, as using samples obtained <em>before </em>convergence could lead to erroneous downstream analyses. To ascertain convergence, we can plot <em>trace</em> using <a href="https://arviz-devs.github.io/arviz/api/generated/arviz.plot_trace.html"><em>pm.trace_plot()</em></a>⁴; a ‘meandering’ pattern indicates non-convergence and is no bueno.</p><p>Here is the trace plot for our experiment:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w6zh5VnIh7IiYfHbeuLqyA.png" /></figure><p>We see that no obvious sign of non-convergence was detected.</p><h3>Posterior distribution analysis</h3><h4>Priors vs posteriors</h4><p>Since we’re confident that the sampling process ran properly, we can proceed to analyze the posteriors. A common starting point is <em>pm.summary(),</em> which returns summary statistics of our posteriors. Let’s print their means.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ef2164101a2ed73a80bf5467fa2b230c/href">https://medium.com/media/ef2164101a2ed73a80bf5467fa2b230c/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/284/1*HTS2-_mGyEt4a099rNbWFQ.png" /></figure><p>We note that these posterior means are quite different from the prior means. For instance, <em>b </em>goes from 1 (prior mean) to 2.736 (posterior mean). This indicates that our prior beliefs were not fully supported by the data and thus were ‘updated’.</p><h4>Comparison with <em>lifetimes outputs</em></h4><p>In the previous article of the series, we’ve seen the use of <em>lifetimes</em> library can also be used to estimate the parameters <em>r, α</em>, <em>a</em> and <em>b</em>. Let’s now compare the result from the two implementations:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/358/1*4Fa_AYjlb0mGuUkJ_dBLSA.png" /></figure><p>We see that despite coming from different frameworks (frequentist vs Bayesian), the estimated parameters are not that different.</p><h4>Analysis of estimated Beta and Gamma distributions</h4><p>We can then use the mean (i.e. the most probable) parameter values of the Bayesian posteriors to construct and visualize the Gamma and Beta distributions.</p><p>Let’s start from Gamma — here is the graph:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0BdsAtWvDOUamYjnOuFVHQ.png" /></figure><p>As we discussed in part 1, the Gamma distribution has a practical significance — it quantitatively describes the collective purchasing behavior of our customer base. Specifically, it indicates the distribution of purchasing rates within the population. The plot above displays a relatively desirable Gamma distribution with most of the 𝜆 found near 2. This means our customers are expected to shop at a rate of 2 transactions per week — not a bad rate for a CD company!</p><p>Now let’s look at the Beta distribution, which describes the distribution of the deactivation probabilities <em>p </em>within our customer base:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ybDasDopsx1vkUf9tihAiw.png" /></figure><p>The plot displays a relatively healthy Beta distribution that concentrates most of its <em>p </em>near 0. This implies that as a whole, our customers are unlikely to deactivate.</p><h3>Conclusion</h3><p>In this series of articles, we’ve discussed the fundamentals BG-NBD model and showcased its value in tackling real-world business problems.</p><p><strong>Specifically, in article 1, we learned that BG-NBD models the repeat purchase behavior and the deactivation probability of the customer population. </strong>The resulting probability distributions are not only theoretically sound; they also present insightful quantitative representations of our individual customers and customer base.</p><p>In article 2, we explored <em>lifetimes</em>, the frequentist implementation of BG-NBD. <em>lifetimes </em>allows us to fit a BG-NBD model with several lines of code. We’ve also seen a couple of practical ways to use the outputs from <em>lifetimes.</em> <strong>I’d recommend <em>lifetimes</em> for analysts who don’t wish to manually code the BG-NBD equations and want to start deriving business values immediately.</strong></p><p>This third and last article implemented BG-NBD using the Bayesian <em>PyMC3 </em>library. This implementation involves coding BG-NBD from scratch and as such requires a more in-depth understanding of the model. The advantage is that it allows us to specify the priors, and this allows us to incorporate our expert knowledge into the modeling process. <strong>I’d recommend this approach for practitioners having both deep familiarity with BG-NBD and domain knowledge of the customer base they are trying to model.</strong></p><h3>References</h3><p><a href="http://brucehardie.com/papers/018/fader_et_al_mksc_05.pdf">[1] “Counting Your Customers” the Easy Way: An Alternative to the Pareto/NBD Model (Bruce Hardie <em>et. al</em>, 2005)</a></p><p>[2] If a deep dive of <em>PyMC3 </em>interests you, do check out the awesome book <a href="http://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/"><em>Probabilistic Programming &amp; Bayesian Methods for Hackers</em></a>.</p><p><a href="https://blog.stata.com/2016/11/15/introduction-to-bayesian-statistics-part-2-mcmc-and-the-metropolis-hastings-algorithm/">[3] Stata Blog: Introduction to Bayesian Statistics (part 2)—MCMC and Metroplis Hastings Algorithm</a></p><p>[4] Trace plotting is just <em>one </em>of the many ways to confirm convergence. The <em>PyMC3 </em><a href="https://pymcmc.readthedocs.io/en/latest/modelchecking.html">documentation</a> provides an extensive treatment of the topic.</p><p><strong><em>Note</em></strong><em>: All images, diagrams, tables and equations were created by me unless indicated otherwise.</em></p><p>If you have any comments about the article or would like to reach out to me, feel free to send me a connection through <a href="https://www.linkedin.com/in/meraldo/">LinkedIn</a>. Also, I&#39;d be very grateful if you could support me by becoming a Medium member through <a href="https://medium.com/@meraldo.antonio/membership">my referral link</a>. As a member, you&#39;ll be able to read all my writings on data science and personal development and have full access to all stories on Medium.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d770676f5c06" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/bayesian-customer-lifetime-values-modeling-using-pymc3-d770676f5c06">Bayesian Customer Lifetime Values Modeling using PyMC3</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Modeling Customer Lifetime Value with lifetimes]]></title>
            <link>https://medium.com/data-science/modeling-customer-lifetime-value-with-lifetimes-71171a35f654?source=rss-65b4354e2ed8------2</link>
            <guid isPermaLink="false">https://medium.com/p/71171a35f654</guid>
            <category><![CDATA[business]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[probability]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Meraldo Antonio]]></dc:creator>
            <pubDate>Thu, 24 Mar 2022 08:16:05 GMT</pubDate>
            <atom:updated>2022-04-25T19:29:09.814Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Modeling Customer Lifetime Values with L<em>ifetimes</em></strong></h3><h4>Estimating your customers’ worth and formulating proactive actions with a few lines of code</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*8bBcSP9jsgjPZdzI" /><figcaption>Source: <a href="https://unsplash.com/photos/odxB5oIG_iA">Unsplash</a></figcaption></figure><p><strong>The Beta-Geometric Negative Binomial Distribution (BG-NBD) model is an influential probabilistic model for describing customer behavior and for predicting customer lifetime value (CLV)¹. </strong>In the <a href="https://medium.com/towards-data-science/customer-lifetime-value-estimation-via-probabilistic-modeling-d5111cb52dd">previous article</a> of the series, we&#39;ve explored the intuition, assumptions and mathematical derivation of this model. I’d suggest you take a look at it if you haven’t!</p><blockquote>As you might have inferred from that article, it’s not easy to code all those BG-NBD equations manually. Luckily, there is a Python library called <a href="https://lifetimes.readthedocs.io/en/latest/"><em>lifetimes</em></a> that can do all the heavy lifting for us. <strong><em>lifetimes</em> abstracts away the internal complexity of BG-NBD and allows us to focus on deriving insights and business values out of the model.</strong></blockquote><p>This awesome library was created by <a href="https://github.com/CamDavidsonPilon">Cam-Davidson Pillon</a>, who is also the author of the book <a href="http://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/"><em>Probabilistic Programming &amp; Bayesian Methods for Hackers</em></a>, a must-read for all Bayesian enthusiasts.</p><p>This article explores the ins and outs of <em>lifetimes</em>. Here is our agenda:</p><ol><li>First, we’ll look at the requisite table format that our dataset needs to have prior to modeling.</li><li>Second, we’ll see how train and assess a BG-NBD model using <em>lifetimes.</em></li><li>Third, we’ll understand the various insights and analyses that <em>lifetimes</em> facilitates.</li><li>Finally, we’ll discuss several practical business applications made possible by <em>lifetimes</em>.</li></ol><p>Let’s get started!</p><h3>Data preparation</h3><h4>Data transformation</h4><p>Usually, businesses record their transactions in a “transactions” table. Such a table has rows that represent individual transactions and columns that present different aspects of the transactions such as the customer involved, the timestamp of the transaction and its value.</p><p>An example transactions table is provided by <em>lifetimes</em>. This table records the transactions of <a href="https://en.wikipedia.org/wiki/CDNow">CDNow</a>, a dot-com era company that sold CDs and music products. It contains 6,919 transactions from 2,357 customers from 1 January 1997 to 30 June 1998 (a total of 78 weeks). We&#39;ll be using it throughout this article. Here is how to load it:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1140bfecc33465e243cb1347f22b5736/href">https://medium.com/media/1140bfecc33465e243cb1347f22b5736/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*1QPCJ-m6wdp6lS1x3m_pdQ.png" /><figcaption><strong>transactions</strong></figcaption></figure><p><strong>Before fitting the BG-NBD model, we need to first restructure this table to the canonical “RFM” format.</strong> A table in the RFM format has rows that represent individual customers and the following columns:</p><ul><li><em>frequency</em><strong>,</strong> which represents the number of repeat purchases the customer has made.</li><li><em>recency</em>, which represents the “age” of the customer when they made their most recent purchases.</li><li><em>T</em>, which represents the “age” of the customer at the end of the observation period.</li><li><em>monetary_value, </em>an optional column which<strong> </strong>represents the average value of a given customer’s purchases, excluding the first purchase.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/0*54bt4m67wTiv1mjJ" /><figcaption>Image by Author</figcaption></figure><p><em>lifetimes</em> provides a utility function that facilitates the conversion from the &quot;transactions&quot; format to the RFM format.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8eea1d0a9e04c76ae15d3c6150abc666/href">https://medium.com/media/8eea1d0a9e04c76ae15d3c6150abc666/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*JFmYyuDxCp6lMt_jF8_k7w.png" /><figcaption><strong>‘rfm’</strong></figcaption></figure><p>The way we’d interpret the first row of the RFM dataframe is as follows:</p><ul><li>During the observation period, customer 4 made a total of 4 purchases (thus 3 repeat purchases).</li><li>The time span between his first and last purchases was 49 weeks.</li><li>The time span between his first purchase and the end of the observation period was 78 weeks.</li><li>The average of his transactions, excluding his first one, is USD 23.73.</li></ul><p>Note that when a zero value for the <em>frequency</em>, <em>recency</em> and <em>monetary_value</em> columns (such as for customer 18) indicates that the customer only made one purchase in the observation period.</p><h4>Calibration-holdout split</h4><blockquote>When reformating our table into the RFM format, we have the option to also perform a calibration-holdout split, which is similar in spirit to the train-test split procedure we’re all familiar with. The calibration-holdout split divides our transactions into two depending on whether they fall into the <em>calibration period </em>or the <em>observation period. </em>The transactions that took place during the calibration period are used to train the model, whereas those occurring during the observation period (“holdout’”<em> </em>transactions) are used to validate the model.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YUvxLkMUNx2jOIek" /><figcaption>Image by Author</figcaption></figure><p>In this example, we’ll set the calibration and the observation periods to be 52 and 26 weeks, respectively. Here is how we can do the split using <em>lifetimes</em>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ce8d9760b43edc6bec5b3b94e8eb06fe/href">https://medium.com/media/ce8d9760b43edc6bec5b3b94e8eb06fe/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*keRVFfxZ3euukvyn5q9nPg.png" /><figcaption><strong>‘rfm_cal_holdout’</strong></figcaption></figure><p>We see that now each customer is associated with both calibration features for model training and holdout (observation period) features for model validation.</p><h3>Training, Prediction and Evaluation</h3><h4>Fitting</h4><p>In <em>lifetimes</em>, the model fitting follows the familiar <em>scikit-learn</em> steps of instantiating a model object (which optionally includes hyperparameter setup) and fitting it to the calibration (training) data:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8ea6907a098223ebf5b0206474c8c26b/href">https://medium.com/media/8ea6907a098223ebf5b0206474c8c26b/href</a></iframe><p>That’s it — two lines and we have a working BG-NBD model!</p><p><strong>As you might remember from the </strong><a href="https://medium.com/towards-data-science/customer-lifetime-value-estimation-via-probabilistic-modeling-d5111cb52dd"><strong>first article</strong></a><strong>, a BG-NBD model is composed of a Gamma and a Beta distribution.</strong> The parameters <em>r </em>and<em> α</em> of<em> </em>the Gamma distribution and <em>a </em>and <em>b </em>of the Beta distribution are accessible through the <em>.summary</em> attribute of the fitted model object:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d03284d3efceb886708ca267b365ef7c/href">https://medium.com/media/d03284d3efceb886708ca267b365ef7c/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/716/1*VABXGuRzbjtM4TDWdmJd7g.png" /><figcaption>parameters of the fitted model</figcaption></figure><p>In the next section, we&#39;ll derive some business insights from these parameters.</p><h4>Assessment of model fit by comparing model-generated artificial data and real data</h4><p>The adage of “garbage-in, garbage-out” applies to BG-NBD modeling: relying on a poorly-fitted model can lead to disastrous business decisions. <strong>As such, before making predictions and interpretations using our model, we should first assess the model’s performance.</strong></p><p><em>lifetimes</em> provides a couple of ways of assessing our model. The first is to compare the frequencies between our real calibration data and artificial data sampled from the distributions generated by the BG-NBD model. The function <em>plot_period_transactions</em> does this in one line:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7e40fd7bb74ccb2f543e1e0032726b55/href">https://medium.com/media/7e40fd7bb74ccb2f543e1e0032726b55/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BnbBnD7JEPk8SBKkketNZg.png" /></figure><p>The plot shows that the distribution of frequencies in the training data largely agrees with that artificially generated by our fitted model. <strong>This suggests that the physical process that we assume to have generated our data is captured pretty well by our model.</strong></p><h4>Making predictions</h4><p>One of the main goals behind fitting the BG-NBD model is to make predictions. In this section, we&#39;ll see how predictions can be generated and subsequently used to further assess the performance of the model.</p><p>When given a customer with specific <em>frequency</em>, <em>recency</em> and <em>T,</em> a fitted model object can generate two types of predictions:</p><ol><li>The number of purchases he&#39;ll make in <em>k </em>future periods</li><li>The probability of him being active at the end of the observation period.</li></ol><p>Let’s see an example of how this can be done. First, we’ll select a sample customer and inspect his <em>frequency</em>, <em>recency</em> and <em>T</em> for both the calibration and the observation periods:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cabf0d36d51ea4d688444ed2828bd147/href">https://medium.com/media/cabf0d36d51ea4d688444ed2828bd147/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*Xl2shHB7_IQf0VkiaqnTIA.png" /></figure><p>We see that during the observation period, the customer made 2 + 1 = 3 transactions (the “<em>+ </em>1” comes from converting the number of <em>repeat </em>transactions, which exclude the first transaction, to <em>total</em> transactions).</p><p>Let&#39;s now compare this “real” number to the prediction generated by BG-NBD. The following code yields an expected number of transactions the customer will make in the next 26 weeks (the length of the observation period)²:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e33fc85e846caee4129277089b8216ce/href">https://medium.com/media/e33fc85e846caee4129277089b8216ce/href</a></iframe><p>We see that the predicted transaction number (0.76 transactions) is lower than the actual one (3 transactions).</p><p>We can similarly predict the probability that a certain customer is still active/alive at the end of the calibration period/start of the observation period).³</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a609ff2c89c5195febef1f27fc128a90/href">https://medium.com/media/a609ff2c89c5195febef1f27fc128a90/href</a></iframe><p>Since the customer <em>did</em> make some purchases in the observation period, we know with absolute certainty that he <em>was </em>active at the end of the calibration period. As such, the predicted probability of 0.57 is an underestimate.</p><h4>Comparison of real and predicted number of transactions</h4><p>Having seen how to make number-of-transactions prediction for <em>one</em> individual, we can extend the procedure to our whole customer base. The resulting predictions can then be compared with the real transaction data to gauge the accuracy of our model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/db915cb7c51bac3bd08fcf984dc76351/href">https://medium.com/media/db915cb7c51bac3bd08fcf984dc76351/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/1*mJglfrr4qkXdxiK7wfDadQ.png" /></figure><p>If a more rigorous assessment is desired, these two columns can be subjected to typical regression metrics such as RMSE:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3859766c8a4b7446a3c09fa9b4c6eef3/href">https://medium.com/media/3859766c8a4b7446a3c09fa9b4c6eef3/href</a></iframe><h3>Insights and Interpretation</h3><h4>Visualizing the MLE Gamma and Beta distributions</h4><p>Once we are satisfied with the performance of our model, we can proceed to derive insights from it.</p><p>A good starting point would be the extraction and visualization of the estimated Gamma and Beta parameters. As elaborated in the <a href="https://towardsdatascience.com/customer-lifetime-value-estimation-via-probabilistic-modeling-d5111cb52dd">previous article</a>, the Gamma and Beta distributions carry important business significance. <strong>The Gamma distribution tells us about the distribution of the transaction rates of our customer base, while the Beta distribution reflects the distribution of probability-to-deactivate.</strong></p><p>The code below shows how to extract the Gamma and Beta parameters from the model and plot the distribution:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2ae12e37e8df0e58dda5f46bfa0817ef/href">https://medium.com/media/2ae12e37e8df0e58dda5f46bfa0817ef/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w-m-j0PcYArYWYVcNSzMjg.png" /></figure><p>From the plot, we can see that the Gamma distribution is relatively healthy, with most of the 𝜆 found near 2. This means our customers are expected to shop at a rate of 2 transactions per week.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7yCPj501FBkM2QJ6SJ54dA.png" /></figure><p>The Beta distribution also seems healthy, with most of the <em>p </em>found near 0. This implies that our customers are unlikely to deactivate soon.</p><h4>Frequency/Recency/Future Purchases Matrix</h4><p>In addition to the model-fit assessment described, the fitted model object can also be used for some interpretative analysis.</p><p>For example, we can look at its Frequency/Recency/Future Purchases Matrix. This matrix shows how different frequency-recency combinations give rise to different expected numbers of future purchases.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cb13923450ad01d6fb85ef5e2f3f7990/href">https://medium.com/media/cb13923450ad01d6fb85ef5e2f3f7990/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B3_FnMlMxWPckArBtvShMg.png" /><figcaption><strong>Frequency/Recency/Future Purchases Matrix</strong></figcaption></figure><p>As we can see, our best customers are in the bottom right corner — these peeps made frequent purchases pretty recently, so we have a high expectation that they’ll come back.</p><p>On the contrary, our least promising customers are in the top right corner — they made frequent purchases and then stopped coming and we haven’t seen them in months. It is likely that they’ve found another store (i.e. they’ve deactivated).</p><h4>Frequency/Recency/Probability Alive Matrix</h4><p>Similarly, we can also produce the Frequency/Recency/Probability Alive Matrix. This matrix shares the same axes as the previous one, but now the shade of each cell tells us the alive probability of customers with various frequency-recency combinations.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/043ed2ad6f08b6824342110f419b9adb/href">https://medium.com/media/043ed2ad6f08b6824342110f419b9adb/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9O7SJ0NQDx63rn-b" /><figcaption><strong>Frequency/Recency/Probability Alive Matrix</strong></figcaption></figure><p>Unsurprisingly, we can see similar patterns, in which the best and the worst customers are in the top right and the bottom right of the matrix, respectively.</p><h3>Business applications</h3><h4>CLV estimation using BG-NBD</h4><p>We’ve seen how we could use the BG-NBD model to predict the probability-of-being-alive <em>p </em>as well as number of purchases in the next <em>k </em>periods. These two predictions can in turn be used to calculate a rough estimate of the customer’s value for the next <em>k </em>periods. The estimate can be calculated as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xYj8MnFdm2hVzRP4" /></figure><p>This estimate relies on the following two simplistic assumptions:</p><ol><li>The probability-of-being-alive <em>p </em>remains unchanged in the next <em>k </em>periods</li><li>The average value of purchases in the next <em>k </em>periods equals the average purchase values in the observation period.</li></ol><p>Do note that these naive assumptions have weak theoretical underpinnings — in reality, <em>p </em>usually changes after every purchase and the future purchase value might diverge significantly from the past values. As such, the resulting estimate is a rough one. <strong>Should you be interested, more sophisticated probabilistic models such as the Gamma-Gamma model exist that can calculate the future purchase value in a more rigorous manner⁴</strong>.</p><p>This is how we perform the above calculation in Pandas:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/06a35a013e254c7cb0bfc901a8f879e4/href">https://medium.com/media/06a35a013e254c7cb0bfc901a8f879e4/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/938/1*_pC11mYHgc9WOfG_aWoFIQ.png" /></figure><p>The resulting <em>value_10_pred</em> column encapsulates the estimated value of our customers for the next 10 weeks. We can then use this estimate to measure the health of our business. For example, we can obtain summary statistics of <em>value_10_pred</em>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/235121ab60869a7e4aed226140202416/href">https://medium.com/media/235121ab60869a7e4aed226140202416/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/546/1*-rLd6xyA_XVhlFnAGdLVgg.png" /></figure><p>We see that on average, we’d expect customers to spend around $5.18 in the next 10 weeks.</p><p>We can also plot a histogram:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1f698196e13e6e9c29fe1dda4850d086/href">https://medium.com/media/1f698196e13e6e9c29fe1dda4850d086/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6x6WCTJU2D716cU603tyrA.png" /></figure><p>We see that a big chunk of our customers have <em>value_10_pred </em>close to 0. The resulting average of $5.18 was driven by a few outliers (a handful of customers with a very high <em>value_10_pred</em>). In the next section, we&#39;ll see how to make further use of this result.</p><h4>Proactive intervention</h4><p>These three new columns, <em>alive_prob</em>, <em>n_transactions_10_pred</em> and <em>value_10_pred</em>, crystalize and quantify aspects of our customers that were previously invisible. Apart from being informational, these new features can be used to drive concrete actions. <strong>For example, we can use these new features to selectively and proactively reach out to different segments of our customers with the goal of increasing their expected CLV</strong>.</p><p>We&#39;ve previously concluded that our overall customer value is largely attributable to a few high value customers. As these customers are our revenue drivers, we&#39;d want to pay special attention to them. One idea is to occasionally send them congratulatory messages/vouchers to encourage their continuous loyalty.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6f4c0fbeda4295ed35fdff77697caa9b/href">https://medium.com/media/6f4c0fbeda4295ed35fdff77697caa9b/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*O0v-OEMyNcYBiFQyeg83GQ.png" /></figure><p>Another idea is to contact customers with high probability of churning (i.e. those with low probability of being alive) to discourage them from abandoning our business.</p><h4>Demand prediction</h4><p>As we&#39;ve seen before, <em>value_10_pred</em> estimates the amount of purchases that our customers will make in the next ten periods. <strong>Another way to make use of this is to regard this value as predicted demand from the customer. </strong>This prediction can in turn support our supply chain decision (e.g. stock replenishing or manufacturing schedule).</p><p>Words of caveat: the BG-NBD was not conceived as a time-series model; it is not equipped with time-series features such as the accounting of seasonality and trend. As such, when it comes to forecasting, it would be prudent to not <em>only </em>rely on the forecasts made by BG-NBD and rather combine it with time series models such as ARIMA.</p><h3>Conclusion</h3><p>We’ve seen how <em>lifetimes </em>allows us to conveniently understand the purchasing behavior of our customer base and derive powerful and actionable insights from it.</p><p>While this is awesome, keep in mind that the estimations from <em>lifetimes</em> are maximum likelihood estimations that are inferred purely from the data we supply.</p><p>In the <a href="https://medium.com/p/d770676f5c06">third article</a> of the series, I will show you an alternative way to implement BG-NBD, this time using Bayesian hierarchical modeling powered by <a href="https://docs.pymc.io/en/v3/"><em>PyMC3</em></a>. We&#39;ll see that this implementation offers more flexibility as it allows us to introduce domain knowledge into the modeling step.</p><p>I hope to see you there!</p><h3>References</h3><p><a href="http://brucehardie.com/papers/018/fader_et_al_mksc_05.pdf">[1] “Counting Your Customers” the Easy Way: An Alternative to the Pareto/NBD Model (Bruce Hardie <em>et. al</em>, 2005)</a></p><p><a href="http://brucehardie.com/papers/018/fader_et_al_mksc_05.pdf">[2] This function comes from equation (10) of the BG-NBD paper (source [1])</a></p><p><a href="http://www.brucehardie.com/notes/021/palive_for_BGNBD.pdf">[3] Computing P(alive) Using the BG/NBD Model (Bruce Hardie <em>et. al</em>, 2008)</a></p><p><a href="http://www.brucehardie.com/notes/025/gamma_gamma.pdf">[4] The Gamma-Gamma Model of Monetary Value (Bruce Hardie <em>et. al</em>, 2013)</a></p><p><strong><em>Note</em></strong><em>: All images, diagrams, tables and equations were created by me unless indicated otherwise.</em></p><p>If you have any comments about the article or would like to reach out to me, feel free to send me a connection through <a href="https://www.linkedin.com/in/meraldo/">LinkedIn</a>. Also, I’d be very grateful if you could support me by becoming a Medium member through <a href="https://medium.com/@meraldo.antonio/membership">my referral link</a>. As a member, you’ll be able to read all my writings on data science and personal development and have full access to all stories on Medium.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=71171a35f654" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/modeling-customer-lifetime-value-with-lifetimes-71171a35f654">Modeling Customer Lifetime Value with lifetimes</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Customer lifetime value estimation via probabilistic modeling]]></title>
            <link>https://medium.com/data-science/customer-lifetime-value-estimation-via-probabilistic-modeling-d5111cb52dd?source=rss-65b4354e2ed8------2</link>
            <guid isPermaLink="false">https://medium.com/p/d5111cb52dd</guid>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[metrics]]></category>
            <category><![CDATA[bayesian-statistics]]></category>
            <dc:creator><![CDATA[Meraldo Antonio]]></dc:creator>
            <pubDate>Wed, 23 Feb 2022 17:22:44 GMT</pubDate>
            <atom:updated>2023-03-27T10:20:58.034Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Customer Lifetime Value Estimation via Probabilistic Modeling</strong></h3><h4>A deep-dive into BG-NBD, an influential hierarchical model that facilitates a understanding of customers’ purchase behavior</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Fn-bGnjcfUggiyIK" /><figcaption>Photo by <a href="https://unsplash.com/@chewy?utm_source=medium&amp;utm_medium=referral">Chewy</a> on <a href="https://unsplash.com/photos/1p7TrM0LkXc">Unsplash</a></figcaption></figure><h3><strong>Importance of customer lifetime value</strong></h3><p>Customer lifetime value (CLV) is the total worth of a customer to a company over the length of his relationship. In practice, this “worth” can be defined as revenue, profit, or other metrics of an analyst’s choosing.</p><p>CLV is an important metric to track for two reasons. First, the totality of a company’s CLV over its entire customer base gives a rough idea of its market value. Thus, a company with a high total CLV will appear attractive to investors. Second, a CLV analysis can guide the formulation of customer acquisition and retention strategies. For example, special attention could be given to high-value customers to ensure that they stay loyal to the company.</p><blockquote>Many CLV models have been developed with different levels of sophistication and accuracy, ranging from rough heuristics to the use of complex probabilistic frameworks. In this article series, we delve into one of them: <strong>the Beta Geometric Negative Binomial Distribution (BG-NBD) model</strong>. This model, developed by <a href="http://brucehardie.com/papers/018/fader_et_al_mksc_05.pdf">Fader, Hardie, and Lee in 2005</a>¹, has been one of the most influential models in the domain, thanks to its interpretability and accuracy.</blockquote><p>This series consists of three articles that build on top of one another. Our game plan is as follows:</p><ol><li>In Part 1, (this one), we’ll achieve an ELI-5 understanding of the BG-NBD model and its assumptions.</li><li>In <a href="https://medium.com/@meraldo.antonio/71171a35f654#2b5f-e66d9b5b6df0">Part 2</a>, we’ll look into the Python library <a href="https://lifetimes.readthedocs.io/en/latest/"><em>lifetimes</em></a> that allows us to conveniently fit a BG-NBD model to a dataset in a <em>scikit-learn</em>-like fashion and almost immediately receive the maximum likelihood estimate of the model’s parameters. We’ll also explore the various downstream analyses that <em>lifetimes </em>enables.</li><li>In <a href="https://medium.com/p/d770676f5c06">Part 3</a>, we’ll look at an alternative way to implement the BG-NBD model, this time from a Bayesian perspective. We’ll see how the Bayesian hierarchical BG-NBD model allows us to inject our prior intuition of customer behavior into the model. To this end, we will be using the Python library <a href="https://docs.pymc.io/en/v3/"><em>PyMC3</em></a>.</li></ol><h3><strong>The scope of BG-NBD</strong></h3><p>Before digging deeper into the mathematics of BG-NBD, we need to understand what it can and cannot do. There are two major limitations to keep in mind:</p><blockquote>The model is only applicable to non-contractual, continuous purchases.</blockquote><blockquote>The model only tackles one component of CLV calculation, which is the prediction of the number of purchases.</blockquote><p>Let’s learn about these limitations in greater detail.</p><h4>BG-NBD applies to non-contractual, continuous purchases</h4><p>Depending on the relationship between the sellers and the buyers, a business can either be a <em>contractual </em>business or a <em>non-contractual </em>business<em>.</em></p><ul><li>A <em>contractual</em> business, as its name suggests, is one where the buyer-seller relationship is governed by contracts. When either party no longer wants to continue this relationship, the contract is terminated. Thanks to the contract, there is no ambiguity as to whether someone is a customer of the business at a given point.</li><li>In a <em>non-contractual </em>business, on the other hand, purchases are made on a per-need basis without any contract.</li></ul><p>We can further distinguish between <em>continuous</em> and <em>discrete</em> settings:</p><ul><li>In a <em>continuous</em> setting, purchases can occur at any given moment. The majority of purchase situations (e.g. grocery purchases) fall under this category.</li><li>In a <em>discrete</em> setting, purchases usually occur periodically with some degree of regularity. An example of this is weekly magazine purchase.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vlp-o9Ek_XlfsaRzxYTpBw.png" /></figure><p><strong>The BG-NBD model tackles the non-contractual, continuous situation, which is the most common yet most challenging of the four to analyze.</strong> Under such a setting, customer attrition is not explicitly observable and can happen anytime. This makes it harder to differentiate between customers who have indefinitely churned and those who will return in the future. As we will see later, the BG-NBD model is capable of assigning probabilities to each of these two options.</p><h4><strong>BG-NBD focuses on predicting the number of transactions</strong></h4><p>A customer’s CLV for a given period can be calculated by multiplying two numbers:</p><ol><li>The customer&#39;s predicted number of transactions within this period.</li><li>The predicted value of each purchase.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c1pANnxkb2qlqf1CY4R8EQ.png" /></figure><p>Usually these two components are tackled and modeled separately. <strong>The BG-NBD model addresses the first — predicting the number of transactions, which in many regards is the more difficult of the two.</strong></p><p>The second component, the expected value of the purchases, can be found either by using simple heuristics, such as taking the average of all past purchases, or by a more sophisticated probabilistic model, such as the <a href="http://www.brucehardie.com/notes/025/gamma_gamma.pdf">Gamma-Gamma model</a>² (which was also created by the authors of BG-NBD).</p><h3>Intuition</h3><p>Before going into the mathematics of the model, let’s try to understand how the model works on a conceptual level.</p><p>Let’s imagine the following scenario. The date is 31 December 2021 and you’re the manager of a cake shop. You’ve carefully kept track of all transactions that happened this year and you want to predict how many transactions you can expect your customers to make in 2022.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*iL6ewFtQsW0hpN4Z.png" /><figcaption>Source: <a href="https://pixabay.com/vectors/icon-small-isolated-front-cafe-4423853/">pixabay</a></figcaption></figure><p>You also happen to be a skilled data scientist and you plan to achieve this prediction by fitting a model to your data. This model should be able to describe your customers’ purchasing behavior in an interpretable way.</p><p>There are some assumptions you could consider when developing the model.</p><h4>Each customer has different purchasing rate</h4><p>You’ve noted that some people buy cakes every day and some every weekend. Others only buy them on special occasions that take place every six months on average. <strong>Your model will need a way to assign a different purchasing rate to each customer.</strong></p><h4>Each customer can stop being your customer at any time</h4><p>In the fiercely competitive cake business, loyalty isn’t guaranteed. <strong>At any point, your customer can leave your business for another one.</strong> Let’s refer to this departure as the “deactivation” of a previously active customer.</p><p>To conveniently model a deactivation, we can assume that it can only happen <em>after</em> a successful purchase. That is to say, after every purchase in your store, a customer will decide whether he’ll continue buying at your shop or to abandon it. Deactivation happens when the customer decides for the latter.</p><blockquote>We’ll assume that a deactivation is both <strong>permanent</strong> and <strong>latent</strong>. <strong>Permanent, </strong>because once a customer decides to churn, he will never return. <strong>Latent</strong>, because he won’t explicitly let you know that he will no longer be your customer.</blockquote><h3>Illustration</h3><p>With these assumptions in place, let’s consider the following scenario, where we have two customers, A and B. Each of them has made some transactions in 2021, and each transaction is indicated by a red dot.</p><p>Can we tell which customers have deactivated and which ones will still frequent your store and contribute to your future revenue?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/658/0*H72eOxIIrEXOAnNn" /></figure><p>The answer is yes — to a certain extent. For example, looking at A’s pattern above, we see that he used to shop pretty frequently, but it’s been a while since we’ve seen him. Because his inter-transaction time is so much shorter than the time that’s passed since his last transaction, it is quite likely that A has deactivated.</p><p>On the other hand, B is an infrequent shopper and her most recent purchase is not that long ago compared to her average between-purchase period. It is pretty likely that she’ll come back.</p><p>Let’s now develop these assumptions and intuitions into a more sophisticated model!</p><h3>Mathematical model</h3><h4>Probabilistic modeling: an introduction</h4><p>Traditionally, CLV was calculated using a simple function of the past data. For example, we can estimate the value of future transactions by taking a fixed fraction of the value of past transactions. Such a calculation, unsurprisingly, is simplistic, unreliable, and uninterpretable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EGQ0MgirX-pVT27FZVg2uQ.png" /><figcaption>Source: the BG-NBD paper¹</figcaption></figure><p>The BG-NBD model, on the other hand, is a <em>probabilistic model</em>. In a probabilistic model, we assume that our observations (i.e. the transactions) are generated by a physical process that we can model using probability distributions. Our task is to estimate the parameters that best explain our existing observations. One commonly used option is to find maximum likelihood estimators of these parameters. We can then use these estimated parameters to perform future predictions. Compared to the first one, this probabilistic framework is usually more robust, accurate and interpretable.</p><p>With that introduction, let’s now convert the assumptions qualitatively described above into a solid probabilistic framework.</p><h4>Poisson process to model transactions and exponential distribution to model time between purchases</h4><p>First, let’s focus on the repeat purchasing behaviors of active customers. We can assume that as long as a customer is still active, their transactions follow a <a href="https://en.wikipedia.org/wiki/Poisson_point_process">Poisson process</a> with a constant purchase rate 𝜆. With this assumption, we can model the time-to-next-purchase Δ<em>t </em>as an exponential distribution parameterized by 𝜆. The PDF of this distribution is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/344/0*20NAs06r0sPx-1dr" /></figure><p>Each active customer will have his own exponential distribution that we can use to predict the probability of the time of the next purchase.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DPAkY_936mXBpzaU" /></figure><p>The graph above shows the PDF of two exponential distributions associated with two customers. The first customer (the blue curve) generally buys a cake every day (his purchasing rate 𝜆 is 1 cake/day). The probability that his next purchase takes place within one day of his current purchase can be found by taking the area under the blue curve between 0 and 1 and calculate to 0.63</p><p>The second customer only buys a cake every week (his 𝜆 is 1/7 cake/day). After performing the same integration, we see that it is much less probable (P = 0.13) that his next purchase will happen sometime before tomorrow.</p><h4>Gamma distribution to describe the variation in buying behavior across population</h4><p>It is useful to think that all these customers, with their differing 𝜆’s, contribute to a store-wide 𝜆 distribution. Our task now is to model this 𝜆 distribution. In doing so, we’ll need to comply with the following requirements:</p><ul><li>The distribution should preferably be one that is well-studied.</li><li>Since 𝜆 can only take values in positive real numbers, the chosen distribution must only have positive values.</li><li>The distribution needs to be flexible enough to model different customer bases with different purchasing behaviors.</li></ul><p><a href="https://en.wikipedia.org/wiki/Gamma_distribution">The Gamma distribution</a> ticks all those boxes and is the one used in BG-NBD to model 𝜆. It is parameterized by the shape parameter<em> r</em> and the scale parameter α<strong>; </strong>different combinations of these two parameters result in the gamma distribution taking distinct shapes. Here is the PDF of the distribution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/0*TNL2kZ_h88_sEP2A" /></figure><p>It is important to note that this Gamma distribution is not just some theoretical mumbo-jumbo. <strong>In fact, a particular Gamma distribution quantitatively describes the collective purchasing behavior of a specific customer base and carries important business implications.</strong></p><p>For example, the blue line in the graph below shows a downward sloping, left-leaning Gamma distribution that results from setting both <em>r</em> and α to equal 1. If this distribution corresponded to my customer base, I wouldn’t be too happy — the heavy left skew means that the bulk of my customers have purchasing rates 𝜆 that are close to zero. That is, they barely purchase any cake!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*608lFDJ97FbCrUkn" /></figure><p>Another Gamma distribution is shown in orange. This is a healthier distribution in which the 𝜆 peaks around 2, which means that a considerable chunk of this customer base buys two cakes per day. Not too shabby!</p><p>Now, a little nerdy note — the combination of Poisson/Gamma distributions, which we’ve been using to model our customers’ purchasing behavior, is also known as<a href="https://en.wikipedia.org/wiki/Negative_binomial_distribution#Gamma%E2%80%93Poisson_mixture"> the Negative Binomial Distribution (NBD</a>). Yep, this is where the name of our model comes from.</p><h4>Deactivation of a customer is modeled as a geometric process</h4><p>Now let’s deal with the deactivation process. As mentioned earlier, after each purchase, a customer will make a decision on whether or not to deactivate. We can assign a probability <em>p </em>for this deactivation. Consequently, the transaction after which a customer deactivates is distributed according to <a href="https://en.wikipedia.org/wiki/Geometric_distribution">the shifted geometric distribution</a>. The PMF of this discrete distribution is shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*V9Eba2oJYit0GXg4" /></figure><p>This PMF is very intuitive — it comes from noting that (1) if a customer deactivates after the xᵗʰ transaction, he must have survived the preceding x-1 transactions and (2) each of this survival carries the probability (1-<em>p</em>). Do note that by definition, a customer must have performed <em>at least one</em> transaction before deactivating (otherwise he wouldn&#39;t have become our customer in the first place!).</p><p>The graph below compares two customers with <em>p</em> = 0.01 and <em>p</em> = 0.1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ToJ-gDopSdn15dWp" /></figure><p>We can see that the higher the <em>p</em>, the more likely it is that the deactivation happens earlier. The customer with <em>p</em> = 0.01 (blue) has a much lower probability of deactivating early compared to the customer with <em>p</em> = 0.1 (orange).</p><h4>Beta distribution to describe the variation in deactivation probability</h4><p>Similar to 𝜆, it is useful to consider that a population of customers is associated with a distribution<em> p</em>. This time, however, we can’t use the Gamma distribution, which has no upper bound. We’ll need another distribution that is equally flexible but whose value range from 0 to 1 (because <em>p</em> can only be between 0 to 1).</p><p>This time, <a href="https://en.wikipedia.org/wiki/Beta_distribution">the Beta distribution</a> fits our needs. Here is the PDF of the Beta distribution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/0*H29v15gSysNUNhyM" /></figure><p>We can see that the distribution is parameterized by two positive shape parameters, <em>a</em> and <em>b</em>. Here are some examples of Beta distribution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*O5nbw9t8iUTAFtRM" /></figure><p>Similar to the Gamma distribution, this Beta distribution also carries business implications. You’d want to see a left-skewed Beta distribution that puts most of its weight near 0, which suggests that most of your customers have low <em>p</em> and aren’t likely to deactivate early. The orange line in the graph above is an example.</p><p>Another quick note — it is this combination of Beta/Geometric distributions that gives rise to the “BG” in the BG-NBD model. Now you know!</p><h4>Tying everything together: mathematical model of likelihood on an individual level</h4><p>We’ve looked at all the distributions with which we quantitatively describe the behavior of our customers. How do we then obtain the best parameters for these distributions?</p><p>One way is to get the maximum likelihood estimators (MLE), which are parameter estimators that maximize the likelihood that the model produced the data that were actually observed.</p><p>Let’s make it more concrete. Suppose that we currently are at time <em>T </em>and we’re looking back at the historical transactions of a particular customer that has a purchase rate 𝜆. He made his first transaction at <em>t</em>₁<em> </em>and his last at <em>t</em>ₓ<em> </em>. These points, drawn on a timeline, look like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Hf9FNehPepwXW8xa" /></figure><p>We can derive the individual-level likelihood function of this person by following the below steps:</p><ul><li>The likelihood of the first transaction occurring at <em>t</em>₁<em> </em>is described using the exponential distribution we elaborated earlier:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QT5NPDsah3ypJPbq" /></figure><ul><li>The likelihood of the <em>second</em> transaction occurring at <em>t</em>₂ is the probability of the customer remaining active after <em>t</em>₁ — (1-<em>p</em>) — multiplied by the standard exponential likelihood component:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZG1lIcjmns7gyWJ4" /></figure><ul><li>Such a likelihood pattern is repeated for each subsequent transaction, that is, the likelihood of the xᵗʰ transaction happening at <em>t</em>ₓ is:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/324/0*FLlKpncto0u4lB30" /></figure><ul><li>Now, let’s analyze what happens after the last transaction at <em>t</em>ₓ. We didn’t observe any transaction between <em>t</em>ₓ and <em>T; </em>this absence can be due to either of the following two scenarios:</li></ul><ol><li><strong>The customer deactivated after his last transaction at <em>t</em>ₓ</strong>. As we know, the probability of this happening is <em>p</em>.</li><li><strong>He remained active yet didn’t make any transaction at this interval</strong>. The probability of this happening is:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/272/0*h48yY-BkKyDZCDBW" /></figure><ul><li>The likelihood of observing the transaction pattern that we observed is simply the multiplication of all the likelihood for earlier transactions times the sum of the likelihood of the two scenarios:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FxLVeGNykiirVVa8" /></figure><ul><li>The above defined likelihood formula is applicable to customers who made <em>some</em> purchase in the observation period. Since we assume that all customers were active to begin with, the likelihood that a customer will not make any purchase between the time [0, T] is the standard exponential function:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vdHv-icT4kMlKi7E4v7Njw.png" /></figure><ul><li>Lastly, combining the above two likelihood formulas, we obtain a generalized formula for all customers notwithstanding the number of transactions they made (or lack thereof):</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tVuAB7Sx3UYXMhcr" /></figure><p>We can then programmatically try out different values of <em>p </em>and 𝜆 and choose a (<em>p, </em>𝜆) combination that maximizes this likelihood. These parameter values, referred to as Maximum Likelihood Estimators (MLE), represent the “best” parameters that describe the purchasing behavior and the deactivation probability of this individual.</p><p>Do note that this individual-level likelihood function only involves three unknowns variables that need to be supplied by the data:</p><ul><li><em>x</em>: t<strong>he number of repeat transactions.</strong> This is also called the (repeat) <strong>frequency</strong>.</li><li><em>t</em>ₓ: <strong>the age of the customer at his last transaction time, </strong>that is, the time that had passed between his first and his last transaction. This is also called <strong>recency</strong>.</li><li><em>T</em>: <strong>the age of the customer at the point of analysis, </strong>that is, the time that had passed between his first transaction and the time of analysis.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/0*yHeIs2v4Pv_1kXyX" /></figure><p>Interestingly enough, we can see that the time of the earlier transactions is not part of the formula.</p><p><strong>A dataset whose rows correspond to different customer IDs and whose columns indicate the <em>x, t</em>ₓ, and <em>T</em> of each customer is referred to as being in the “RFM format”. </strong>The &quot;R&quot; and &quot;F&quot; here stand for recency and (repeat) frequency, respectively. Meanwhile, &quot;M&quot; stands for monetary value; this is a column we won&#39;t use in our analysis since we&#39;re not concerned about the transaction values. The RFM format is the canonical format used frequently in CLV analysis.</p><h4>Zooming out: mathematical model of likelihood on a population level</h4><p>As a company with (hopefully) many customers, oftentimes, we’re not too interested in looking at individual customers. Rather, we’d like to analyze our customer base as a whole. Specifically, we’re interested in obtaining the best Gamma and Beta distributions that describe the performance of our entire business.</p><p><strong>Just like how we can use MLE to get the best <em>p </em>and 𝜆 for an individual, we can also use MLE to get the best <em>r, </em></strong><em>α</em><strong>, <em>a, </em>and<em> b </em>for the population. </strong>I won’t be deriving the population-level likelihood equation in this article; it’s long enough as it is. However, if you’ve understood the math above, you should be in good shape to dive into the derivation that is clearly explained in the <a href="http://brucehardie.com/papers/018/fader_et_al_mksc_05.pdf">BG-NBD paper.</a></p><h3>Outro</h3><h4>Other applications of BG-NBD</h4><p>We’ve so far framed our discussion around CLV calculation, which is what BG-NBD was initially intended for. However, BG-NBD is more versatile than that. It can in fact be used to model any phenomenon that involves different “users” making repeated “transactions” and predict (1) how many future &quot;transactions&quot; will be made by those &quot;users&quot; provided that they are still “active” and (2) the probability that they are still &quot;active&quot; during the time of analysis. For example:</p><ul><li>Predicting the future usage frequency of a mobile app by exploring users’ usage history.</li><li>Calculating the probability that your distant relative who used to call you periodically is still alive, literally, by analyzing her call pattern.</li><li>Checking if your Tinder dates have become disinterested in you by looking at their texting frequency.</li></ul><h4>Going forward</h4><p>Alright, I know that we’ve gone through a lot of math, which can be challenging. You might be thinking: is there a way to skip all these equations and use a ready-made implementation of this model to start deriving business values out of it?</p><p>I hear you! In <a href="https://medium.com/@meraldo.antonio/71171a35f654#2b5f-e66d9b5b6df0">Part 2 of the series</a>, we’ll look at the Python library <em>lifetimes</em> that allows us to use a couple lines of code to get the MLE of <em>r, α</em><strong>, </strong><em>a, </em>and<em> b </em>from a given record of past transactions. This library also contains other useful analytical and plotting functions that will allow us to derive business insights from the BG-NBD model and other related models.</p><p>Afterwards, in <a href="https://medium.com/p/d770676f5c06">Part 3</a>, we&#39;ll check out an alternative implementation of BG-NBD, which approaches the parameter estimation from the Bayesian perspective. This Bayesian framework will allow us to &quot;inject&quot; our domain knowledge and/or beliefs into the modeling process.</p><p>I hope to see you there!</p><h3>References</h3><p><a href="http://brucehardie.com/papers/018/fader_et_al_mksc_05.pdf">[1] “Counting Your Customers” the Easy Way: An Alternative to the Pareto/NBD Model (Bruce Hardie <em>et. al</em>, 2005)</a></p><p><a href="http://www.brucehardie.com/notes/025/gamma_gamma.pdf">[2] The Gamma-Gamma Model of Monetary Value (Bruce Hardie <em>et. al</em>, 2013)</a></p><p><strong><em>Note</em></strong><em>: All images, diagrams, tables and equations belong to me unless indicated otherwise.</em></p><p>If you have any comments about the article or would like to reach out to me, feel free to send me a connection through <a href="https://www.linkedin.com/in/meraldo/">LinkedIn</a>. Also, I’d be very grateful if you could support me by becoming a Medium member through <a href="https://medium.com/@meraldo.antonio/membership">my referral link</a>. As a member, you’ll be able to read all my writings on data science and personal development and have full access to all stories on Medium.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d5111cb52dd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/customer-lifetime-value-estimation-via-probabilistic-modeling-d5111cb52dd">Customer lifetime value estimation via probabilistic modeling</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Live Prediction of Traffic Accident Risks Using Machine Learning and Google Maps]]></title>
            <link>https://medium.com/data-science/live-prediction-of-traffic-accident-risks-using-machine-learning-and-google-maps-d2eeffb9389e?source=rss-65b4354e2ed8------2</link>
            <guid isPermaLink="false">https://medium.com/p/d2eeffb9389e</guid>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[google]]></category>
            <category><![CDATA[transportation]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Meraldo Antonio]]></dc:creator>
            <pubDate>Mon, 14 Oct 2019 15:41:32 GMT</pubDate>
            <atom:updated>2022-04-25T19:30:02.302Z</atom:updated>
            <content:encoded><![CDATA[<h4>Here, I describe the creation and deployment of an interactive traffic accident predictor using <em>scikit-learn, </em>Google Maps API, Dark Sky API, Flask and PythonAnywhere.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*xE7aSbp_Vd6gWNUH" /><figcaption>Source: <a href="https://unsplash.com/photos/ZWXx6uERI6M">unsplash</a></figcaption></figure><p>Traffic accidents are extremely common. If you live in a sprawling metropolis like I do, chances are that you’ve heard about, witnessed, or even involved in one. Because of their frequency, traffic accidents are a major cause of death globally, cutting short millions of lives per year. Therefore, a system that can predict the occurrence of traffic accidents or accident-prone areas can potentially save lives.</p><p>Although difficult, traffic accident prediction is not impossible. Accidents don’t arise in a purely random manner; their occurrence is influenced by a multitude of factors such as drivers’ physical conditions, car types, driving speed, traffic condition, road structure and weather. Studying historical accident records would help us understand the (potentially causative) relationships between these factors and road accidents, which would in turn allow us to build an accident predictor.</p><p>Fortunately, several of such accident records are publicly available! For instance, the UK government has published detailed records of traffic accidents in the country dating back to 2002. <strong>With this data set, my teammates and I have created and deployed a </strong><a href="https://kteo7.pythonanywhere.com/"><strong>machine learning model </strong></a><strong>that predicts with high accuracy as to when and where accidents are likely to occur in Greater London.</strong></p><blockquote>In this article, I will take you through the process of completing this project. The complete code can be found in my <a href="https://github.com/meraldoantonio/AccidentPredictor">GitHub repository</a>, while the live model can be accessed at this <a href="https://kteo7.pythonanywhere.com/">website</a>.</blockquote><p><a href="https://kteo7.pythonanywhere.com/">Road Traffic Accident</a></p><h3>Objective</h3><p>Before embarking on this project, we set ourselves a clear objective: <strong>we wanted to create an interactive traffic accident predictor that would be easily accessible by anyone. </strong>We decided that the best way to achieve this goal was to deploy a trained predictor on a website. This predictor-website should be capable of doing the following:</p><ul><li>Allow users to input an origin and a destination (both of which have to be in greater London) and find the best driving route that connects the two.</li><li>Allow user to pick the date/time they plan to make the trip and identify areas along the route that are particularly accident-prone within that time window.</li></ul><p>The steps we undertook to achieve this objective are depicted in the diagram below. I will elaborate them in the sections that follow.</p><figure><img alt="A diagram showing the major steps in this project" src="https://cdn-images-1.medium.com/max/1024/1*Rxom3C2iIyeApNVocwRKcw.jpeg" /><figcaption>Steps in this project</figcaption></figure><h3>Data collection</h3><p>Having defined an objective, we proceeded to look for the necessary data. The following are the two data sets we finally chose:</p><h4>Kaggle</h4><p><strong>The main data set we used for this project was a detailed accident record published by the UK government and hosted on </strong><a href="https://www.kaggle.com/daveianhickey/2000-16-traffic-flow-england-scotland-wales/version/8"><strong>Kaggle.</strong></a><strong> T</strong>his data set contains details about 1.6 million traffic accidents that took place in the UK between 2000 and 2014. It consists of 33 columns which capture details such as the location, time, severity of the accidents as well as various meteorological and traffic backdrops. In this project, we restricted ourselves to analyzing traffic accidents in Greater London between 2012 and 2014.</p><h4>Dark Sky</h4><p>We believe that weather is a particularly important factor in road accidents. Although the aforementioned Kaggle data set contains meteorological information, we didn’t think that it’s sufficient. There were the two issues about using the weather_condition column of the Kaggle dataset:</p><ul><li><strong>It assumes that weather is constant throughout the day. </strong>If you’ve ever been to London, you know why this assumption is problematic — the weather there changes very frequently!</li><li><strong>It only contains historical weather records (obviously!).</strong> In order to use weather_condition as a predictor for <em>future </em>accidents, we need a way to get a weather <em>forecast</em>.</li></ul><p>Because of these reasons, <strong>we decided to instead make use of the meteorological data provided by </strong><a href="https://kteo7.pythonanywhere.com/darksky.net"><strong>Dark Sky</strong></a><strong>, an American company that specializes in weather forecasting. </strong>Dark Sky provides both past records and future weather forecast updated on a half-hourly basis, which perfectly serves our purpose.</p><h3>Data processing</h3><h4>DBSCAN Clustering</h4><p>Upon exploring the data set, we found that almost every street in London has once been an accident site. This makes sense— London is a bustling city where traffic accidents (including minor ones) happen <em>very</em> frequently. The picture below, created using Tableau, superimposes accident locations (red dots) on the map of London. Notice how the whole city is bathed in red!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*J492T-9x9zV3Dl5oRai12A.png" /><figcaption>The city of London. Each red spot indicates an accident between 2012 and 2014. As we can see, London is full of (historical) car accidents!</figcaption></figure><p>Although accidents have happened all across the city, some areas are more accident-prone than the others. In order to systematically identify these accident hot spots, we grouped the aforementioned accident locations into clusters. We defined a cluster to be an area with a cross-section of 25 meters in which at least 14 accidents had occurred within the two-year time window.</p><p>We used the DBSCAN algorithm to perform this clustering. <strong>DBSCAN was chosen due to its speed, its ability to find arbitrarily-shaped clusters and its robustness to outlier. </strong>The accident spots that fall <em>outside</em> of the clusters were regarded as outliers and were excluded from our subsequent analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/1*3Fm9f_PwvgLd9YKvgUeVzg.gif" /><figcaption>A depiction of how DBSCAN works</figcaption></figure><p>The DBSCAN clustering step resulted in the discovery of 473 accident hot spots. These hot spots were visualized below using ArcGIS, a software for geographical analysis. We can see the hot spots are concentrated on the major roads of London such as A11 and A13.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AIDS7v_c6V77XLH0ize7BA.png" /><figcaption>Accident hot spots in London</figcaption></figure><h4>Negative Sampling</h4><p>At this point, we had several hundreds of accident hot spots. One thing I should clarify — these hot spots are not accident-prone 100% of the time. For instance, accident hot spots along London’s main arteries might be risky during the rush hours but relatively safe during quieter times. On the other hand, hot spots in a sharp turn might be pretty harmless on a bright, sunny day but might become mortal on a snowy or rainy day.</p><p>Thus, our next task was to find out under which conditions these hot spots are “activated”. Framed this way, <strong>our project now became a classic binary classification problem: we needed to find out the combination of factors (e.g. weather, time, etc) that would “activate” a hot spot.</strong></p><p>However, our data at this point weren’t yet sufficient to create a classification model! Training a classifier requires both positive and negative samples, but we only had positive ones (i.e. records whose is_accident target label is 1). <strong>Thus, we needed a way to generate negative samples (i.e. records of “non-accidents” )</strong>.</p><p>The generation of negative samples using positive samples had previously been described. We followed a method described by <em>Yuan et al.</em>¹ <strong>In essence, this method entails the random generation of three negative samples for every positive sample in the cluster. </strong>For instance, if we had a cluster located on Abbey Road in which 15 accidents had occurred, we would randomly synthesize 45 records of non-accidents for this very spot. Except for their location, other predictive features of these negative samples, such as date and time of the non-accident events were selected at random. Caution was taken so that these randomly-synthesized negative samples did not coincidentally match the real positive samples.</p><h3>Exploration and Observation</h3><p>Upon exploring our cleaned data set, we made several interesting observations. Some of the more interesting observations are elaborated below.</p><p><strong><em>Which months are road traffic accidents most likely to occur?</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/739/1*g94Bcjbk0Xd1zOtxj6soAg.png" /></figure><p>The heat map above shows how many accidents have occurred for each day of the week in all calendar months. We can see that the months of September until December are characterized by a relatively high frequency of accident. These are the months of autumn in London, which are characterized by low temperature, fog and rain. Counter-intuitively, fewer accidents take place in the chilly winter months of January and February. This might suggest that people are more reluctant to drive during this time.</p><p><strong><em>What time of the day are road traffic accidents most likely to occur?</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oM6Gz0rXyqPD_Z0fc1u1Xw.png" /></figure><p>The heat map above shows how many traffic accidents have occurred in each hour for each day of the week. We observed that unsurprisingly, most accidents transpire during morning rush hour from 8 am to 9 am and evening rush hour from 3pm to 7pm (purple blocks). Friday is particularly bad.</p><p><strong><em>How many vehicles are involved per accident?</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3YKyfL6lpv183rKzIsX6sA.png" /></figure><p>The bar graph above shows the distribution of accidents based on the number of vehicles involved. It indicates that most accidents involve one or two vehicles. This intuitively makes sense; most accidents happen as a result of a c ar veering off the road and/or collision between two cars.</p><p><strong><em>In which boroughs do traffic accidents occur most often?</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ng2yYYqt2Au6AJ1GxxhfGA.png" /></figure><p>The bar chart above shows how many accidents take place in different boroughs of London. We see that Westminster takes the number one spot and Kingston Upon Thames comes in last place with the least traffic accidents. The map below presents the same information in a different way: it highlights the accident-prone boroughs in darker shades of purple.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JIbLM1l85B3jFU8LZWr-7g.png" /></figure><h3>Supervised learning</h3><p>With the aforementioned pre-processing steps, we were finally ready to proceed to the modeling stage! We split our data set in a 70:30 ratio into a training and testing data set.</p><p>Python’s <em>scikit-learn </em>library was used for model training and evaluation. We attempted common classification models such as SVM, logistic regression and random forest. The models’ relative performance was measured and compared using accuracy and area under Receiver Operating Characteristic curve (AUC-ROC). The results of our modeling steps are summarized in the table below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*896RRh9tFAeWI59jehbb6w.png" /></figure><p>We found that among the models we tested, the best performing model was random forest trained on only numerical features. This was the model we selected for deployment, a process I shall describe below.</p><h3>Deployment</h3><p>We then packaged our <em>scikit-learn </em>model into a web application using Flask, a Python web framework. The front-end display of the website was built off of a free template taken from the website HTML5UP . All the html, javascript and CSS codes were integrated in to the Flask application.</p><p>The application was then is then made online by PythonAnywhere, a Python-focused web hosting service. You can visit the website on <a href="http://kteo7.pythonanywhere.com/">kteo7.pythonanywhere.com</a>. Here is a screenshot of the website:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nlV-qdGW4IKX57iXCK0WPg.png" /></figure><p>Now I’d like to provide more details into the workings of the Flask application.</p><p>On the front end, there are fields where user can input the origin and destination of their choosing. These text fields are equipped with Google Places API that provide autocomplete feature. There is also a drop down menu with which the user can pick a date/time (can be past, present, or within 48 hours into the future).</p><p>Once these inputs are made, a POSTrequest is sent to the backend framework. Here, these inputs act as arguments to the function call_google. <strong>Given an origin and destination, this function will call a Google Maps API that will return the best driving route connecting the two. </strong>More specifically, the function returns the latitudes and longitudes of the route’s <em>waypoints, </em>which are regularly-spaced points along the route.</p><p>The latitudes and longitudes serve as inputs to another function called calc_distance. <strong>This function “draws” imaginary circles with a radius of 50 meters from these waypoints and checks whether or not there are any accident hot spots that fall within these waypoints.</strong></p><p>For each cluster falling inside the circles, yet another function, call_darksky, makes a request to the Dark Sky API. <strong>This function will return the weather forecast for the spot in the specified time.</strong></p><p>With the weather data, together with the information on the date/time information, <strong>the final predict function will perform a binary prediction on whether or not the hot spots of interest are “activated” under the chosen weather/time.</strong> These predictions are then sent to the front-end, which will display them on the screen.</p><p>The gif below shows our interactive model in action. Here, I picked a future date and time (6 pm on Saturday, October 12th), chose an origin (Hyde Park) and a destination (King’s Cross). We can see that the model chose a route that passes through Gloucester Place and A501 and flagged the busy intersection between these two roads as a potential accident site.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*U3GPUb0dUheOr4wb83q6Bw.gif" /></figure><p>Again, you can access this model on <a href="http://kteo7.pythonanywhere.com/">kteo7.pythonanywhere.com</a>. I encourage you to play around with it while it’s still active (it will be active until early 2020).</p><p><em>Note: all images and animations above belong to me unless specified otherwise.</em></p><p>If you have any comments about the article or would like to reach out to me, feel free to send me a connection through <a href="https://www.linkedin.com/in/meraldo/">LinkedIn</a>. Also, I’d be very grateful if you could support me by becoming a Medium member through <a href="https://medium.com/@meraldo.antonio/membership">my referral link</a>. As a member, you’ll be able to read all my writings on data science and personal development and have full access to all stories on Medium.</p><p><em>This project was a part of a course, “Data and Visual Analytics”, that I took as a part of my Master’s in Analytics from Georgia Institute of Technology. Special thanks to my teammates in this project</em> — <em>Katannya, Roger, Siew Lee and Yeok! You guys rock!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d2eeffb9389e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/live-prediction-of-traffic-accident-risks-using-machine-learning-and-google-maps-d2eeffb9389e">Live Prediction of Traffic Accident Risks Using Machine Learning and Google Maps</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Interview with Apache Pig: Theory and Practical Guide]]></title>
            <link>https://medium.com/data-science/interview-with-apache-pig-theory-and-practical-guide-6c44f06a4ca9?source=rss-65b4354e2ed8------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c44f06a4ca9</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Meraldo Antonio]]></dc:creator>
            <pubDate>Tue, 24 Sep 2019 16:34:37 GMT</pubDate>
            <atom:updated>2022-04-25T19:30:33.080Z</atom:updated>
            <content:encoded><![CDATA[<h4>In this exclusive interview, Apache Pig spoke about his role in the Hadoop ecosystem and elaborated on his use cases and his differences from other Hadoop components such as Hive and Spark. He also provided some practical guidance on how to start using him. Let’s hear what Pig has to say!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*k8W0iFiZrzH3nKNu" /><figcaption>Source: Kenneth Schipper Vera from <a href="https://unsplash.com/photos/6y7jACxmhP8">Unsplash</a></figcaption></figure><p><em>Apache Pig is an integral component of the Hadoop ecosystem. Since his birth more than a decade ago, he has helped thousands of analysts around the world make sense of massive data sets. Despite his importance, few people outside of the big data sector have heard of him, and some who are acquainted with him have only vague idea about his usage.</em></p><p><em>A few days ago, I had the pleasure to sit down and talk with Apache Pig himself. Our discussion covered many topics — from his founding philosophy to practical guidance on writing his language, Pig Latin. Below is the excerpt of this interview.</em></p><h4><strong>Hi Pig! Thank you for this interview opportunity!</strong></h4><p>Thank <em>you</em>! It’s my pleasure to give this interview and gain some additional exposure for myself.</p><p>You know, I’m tired of people confusing me with my buddies in the Hadoop ecosystem. To many, we are just a mishmash of colorfully-named tools that somehow fit together to facilitate the management of Big Data. This has got to change! We play distinct roles in the Hadoop ecosystem and a familiarity with each of us is key to understanding Big Data. It’s about time that I cleared this confusion and told the world who I really am!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/258/0*KauzwM211rZFxp5G" /><figcaption>C’est moi — Apache Pig!</figcaption></figure><h4><strong>Great, that’s exactly what I’m here for! Now for the sake of our casual readers who are just getting started to the world of Big Data, could you please introduce yourself?</strong></h4><p>Of course! My name is Apache Pig, but most people just call me Pig.<strong> </strong>I am an open-source tool for analyzing large data sets.</p><p>I was conceived by Yahoo in 2006. My conception was driven by Yahoo’s need to transform and analyze massive amount of data without having to write complex MapReduce programs. In 2007, Yahoo open-sourced me under Apache Software Foundation (hence my first name). Since then, I have been guzzling and analyzing petabytes worth of data all around the world.</p><h4>I like your name! Can you elaborate on its origin?</h4><p>Thanks! It is an interesting name, isn’t it? Many people think that my name stems from my creators’ effort to stick to the animal theme that permeates the Hadoop world. This is partly true — my name was indeed inspired by the eponymous animal🐷. This name, in turn, led to the adoption of cute nomenclatures for concepts related to me such as <em>Pig Latin</em>⁠—my programming language — and <em>Grunt</em>⁠ — my interactive shell (more on these later).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*ecxWn0F-KcSvNVIU" /><figcaption>Apache Pig&#39;s similarity to these cute fellas are more than skin deep! (Image source: <a href="https://unsplash.com/photos/6y7jACxmhP8">Unsplash)</a></figcaption></figure><p>However, apart from it being catchy, there is actually an interesting philosophy that underlies my name. This philosophy, which is somewhat related to the characteristics of pig the animal, consists of four principles that guide how I should be developed:</p><blockquote><strong><em>1. Pigs eat anything.</em></strong><em> I am a voracious data-guzzler. I can process any type of data, be it relational, flat, nested, structured or unstructured. I’m also capable of reading data from multiple sources, such as Hive, HBase and Cassandra.</em></blockquote><blockquote><strong><em>2. Pigs live anywhere.</em></strong><em> Although I was first implemented on Hadoop, I’m not intended to be tied to this framework. For instance, I can be run in a single computer to analyze local files.</em></blockquote><blockquote><strong><em>3. Pigs are domestic animals.</em></strong><em> I was designed to be easily understandable and modifiable by users. My programming language, Pig Latin, is easy to wrap one’s head around. In addition, its capabilities can be easily extended by importing various user defined functions (UDFs).</em></blockquote><blockquote><strong><em>4. Pigs fly. </em></strong><em>This motto speaks to the speed with which I can process data. It also embodies my ability to help my users express complex logic in just a few lines of Pig Latin code.</em></blockquote><h4>That’s very interesting. Now, you mentioned that you are part of the Hadoop ecosystem. Can you tell us a bit about how you fit in this ecosystem?</h4><p>Certainly! To start off, for the sake of those who are new to this field, let me give you a quick explanation about Hadoop.</p><blockquote><strong>Hadoop is a collection of open-source tools with which we can harness the combined power of several computers to store and process massive amounts of data.</strong></blockquote><p>The diagram below depicts how I relate to other components within the ecosystem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0eSXK6JvX3A0z22FUPZpDg.png" /><figcaption>Pig and other Hadoop components</figcaption></figure><p>The core of Hadoop is <strong>HDFS (Hadoop Distributed File System) </strong>and<strong> YARN (Yet Another Resource Negotiator)</strong>. I work closely with these two guys.</p><p>HDFS is a distributed storage system. It breaks files into blocks, makes redundant copies of these blocks and distributes them across different machines in the cluster. <strong>You can think of HDFS as my data manager.</strong> I read input files from HDFS, temporarily store my intermediate calculation results in HDFS and write the output of my analysis to HDFS.</p><p>On the other hand, <strong>you can think of YARN as my computing resource manager. </strong>YARN negotiates computing resources on my behalf and schedules the tasks I need to execute. In doing so, YARN works closely with HDFS. YARN makes sure that the computing resources I get are located close (in terms of network topology) to the machines that store the data that I have to analyze.</p><h4>In the diagram above, I see that there is an execution engine layer between YARN and yourself. Could you explain what an execution engine is?</h4><p>Good question! YARN is actually as my <em>indirect </em>computing resource manager — I don’t interface directly with him. Instead, this interaction takes place through an <em>execution engine</em> layer. In Hadoop’s context, an execution engine is a software system or a framework that runs across a cluster and gives the illusion of the cluster being a single giant machine. Most execution engines run on top of YARN. <strong>As of today, there are three execution engines I can run on: </strong><a href="https://hadoop.apache.org/docs/r1.2.1/mapred_tutorial.html"><strong>MapReduce</strong></a><strong>, </strong><a href="https://tez.apache.org/"><strong>Tez</strong></a><strong> and </strong><a href="https://spark.apache.org/"><strong>Spark</strong></a><strong>.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/1*D3U2w0zAe6qwKfcEsolb8g.png" /><figcaption>The three execution engines that Pig can run on: Hadoop, Tez and Spark.</figcaption></figure><h4>You’ve previously singled out MapReduce and said that you obviate the need for writing MapReduce code. Can you elaborate more on what MapReduce is?</h4><p>Sure! As I mentioned earlier, MapReduce is one of the three execution engines that I can run on. It is not a piece of software <em>per se; </em>rather,<em> </em>it is a simple yet powerful <em>programming paradigm </em>(i.e. a way of designing a program). <strong>Every MapReduce program consists of three phases: map, shuffle, and reduce.</strong></p><p>I shall first define these terms conceptually. Suppose we have a giant file on which we’d like a MapReduce program to operate. <strong>In the map phase, the program will look at this giant file line by line. For each line, the program will output a key-value pair. </strong>The exact identity of these keys and values will depend on the objective of the program.</p><p><strong>In the shuffle phase, the key-value pairs with the same keys are grouped together. </strong>The resulting groups will be ordered according to their keys (e.g. if the keys are strings, then the groups will be sorted by the alphabetical structure of the keys). The output of the shuffle phase is an ordered bunch of groups. These groups are then distributed to different machines for the subsequent reduce phase.</p><p><strong>In the reduce phase, an aggregation function is applied to the bundled values in each group produced in the shuffle phase. </strong>This process is performed in parallel in many machines. The reduce phase will output the result of the program.</p><p>An example will make this clearer. Let’s say I have a text file and I want to count the occurrence of each distinct word in the text. Here is how a MapReduce job for this task would look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6AFn9DCVIi0ydCZFoAUGeg.png" /><figcaption>MapReduce for word count</figcaption></figure><p>The above example looks simple, doesn’t it? Unfortunately, this is not the case for a great majority of MapReduce programs! Any big data professional can tell you that writing MapReduce programs is cumbersome and that not all data procedures can be easily written using the MapReduce paradigm.</p><p>This is where I come in! It turns out that my language, Pig Latin, is much easier to write than MapReduce (one estimate points out that 10 lines of Pig Latin are equivalent to 200 lines of MapReduce code in Java!). When I run on MapReduce as the execution engine, under the hood, all these Pig Latin codes are converted into their MapReduce programs. Pretty neat, huh?</p><h4>Cool! What about the two other execution engines, Tez and Spark? Are they similar to MapReduce?</h4><p>Not quite! The MapReduce paradigm requires that a program be broken down into a series of mapping and reducing tasks. This requirement can sometimes lead to a convoluted program that consists of a long linear series of mappers and reducers.</p><p>On the other hand, <strong>when I run on either Tez or Spark, I will convert the logic expressed in a Pig Latin code into a data structure known as Directed Acyclic Graphs (DAGs). </strong>The word “directed” in DAG means that all the data flow to <em>one</em> direction, while the word “acyclic” means that DAGs can’t contain any loop. Here is an illustration of DAG:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/817/1*yDfPvEyimzfQZt0p8rkcaA.png" /><figcaption>Example of DAG</figcaption></figure><p>When creating a DAG, I will ensure that the steps are ordered in a way that optimizes computation. For example, let’s say we have a script that contains a JOIN operation followed by a FILTER step. I will investigate whether the FILTER step can be done before the JOIN step. If yes, I will perform the FILTER step prior to the JOIN step, thus reducing the number of data points to be joined (JOIN is a computationally expensive operation!)<strong> Because of the power of DAGs, my running on Tez or Spark always leads to better performance compared to MapReduce.</strong></p><h4><strong>Dope! Now let’s talk about your use case. You told me that you facilitate the analysis of Big Data. Could you go into specifics</strong> — what are scenarios where someone should use you?</h4><p>In my experience, people use me for two things: extract-transform-load (ETL) pipelines and ad-hoc queries.</p><p><strong>A typical ETL pipeline loads in a data from a source, clean it, transform it and store it somewhere.</strong> An example is the processing of log data of a particular website. Using my FILTER function, one can clean these logs by filtering out data coming from automatic bots and internal views. I can also perform transformations that enrich the original log data. For instance, I can group these log data by the countries that the visitors hail from to provide more fine-grained view of the data.</p><p><strong>My other common use case is research and ad-hoc queries. </strong>Traditionally, these are done using SQL. However, if the data set to be queried is unstructured and contains “complications” such as nested structures and missing values, I may be a better tool. My flexibility and extendability lend themselves well for such cases.</p><h4>That sounds a whole lot like what your friend Hive can do, too. What’s the difference between you and Hive?</h4><p>This is a question I receive a lot. Before I go into the details, let me quickly introduce Hive 🐝 to you. Like myself, <strong>Hive is a data analysis tool that is part of the Hadoop ecosystem and can run on either MapReduce, Tez or Spark.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*tBYZ3fJLcAHTZiNunNC7-Q.png" /></figure><p>Now let’s go to the meat of the question. If both Hive and I can perform data analysis, why is there a need to have both of us? In other words, what are the differences between us? Here are my answer to that:</p><ol><li><strong>Language. </strong>Hive uses HiveQL, a variant of the familiar query language a programming language SQL while I use Pig Latin, a procedural language. Pig Latin is more extendable than SQL⁠⁠ — one can import Python or Java functions to broaden Pig Latin’s functions. There are other differences between Pig Latin and SQL, which I will elaborate later.</li><li><strong>Input data. </strong>Hive imposes schema on the data it processes, thus can only handle structured data. Meanwhile, as I mentioned before, I can handle structured, semi-structured or unstructured data.</li><li><strong>Target user. </strong>Because Hive uses a language similar to standard SQL, it is targeted for users who are already comfortable with the latter, namely data analysts. These data analysts mainly use Hive for periodic reporting. Meanwhile, Pig Latin is more powerful but takes a bit more time to get used to. As such, I am more suited to programmers, data engineers and researchers that can apply me for the use cases I mentioned previously.</li></ol><h4><strong>Ok, we have talked quite a bit about your strengths and use cases. I don’t want to sound like a clueless HR guy, but I have a rather uncomfortable question that I believe to be of interest to our readers. What are your weaknesses?</strong></h4><p>Ah I see, the dreaded “weakness” question 😓. Well let me be frank — I do have some limitations that might make me unsuitable in some cases. Here are my main limitations:</p><ol><li>My language, Pig Latin, is a procedural language that describes data transformation. It does not natively support loops and only have limited support for IF/THEN constructs, which are sometimes necessary.</li><li>I can only perform basic data analysis and can only return textual/numerical results. If you want to perform more sophisticated data exploration, such as ones that involve making visualizations, or build a machine learning model, then I am not the right guy! For this, my buddy Spark will be more appropriate.</li><li>Lastly, I am rather slow 😓 and thus not suitable for low latency queries.</li></ol><h4><em>That brings me to my next awkward question. Word has it that some people are ditching you in favor of Spark. What say you?</em></h4><p>Sadly, this is true. Spark is quickly gaining ground in the industry. In the rapidly evolving Big Data sector, it is only natural that the community regularly phases out older technologies in favor of newer, more powerful tools. To my chagrin, it feels that <em>I </em>am the one currently being phased out 😭.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VXMtFp0sG40OmVabEZFoug.png" /><figcaption>Relative interest in Pig vs Spark as indicated by Google searches of these terms</figcaption></figure><p>Let me quickly tell you why Spark is in many ways superior to myself. While my language Pig Latin provides several high-level operators and is extendable to a certain extent, Spark is a full-blown programming language. Thanks to tight integration to its “host” language (usually Scala or Python), Spark comes with complete control statements and unlimited ability to define functions and objects. All these allow you to transform your data in almost any way you like!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/633/0*mvegzgsAhx1F26Ik.png" /><figcaption>Spark and its core libraries</figcaption></figure><p>On top of that, Spark also has several libraries that can perform specialized procedures on massive data sets. For example, Spark’s MLlib allows for pre-processing, model training and making predictions at scale. With these abilities, Spark serves as a unified platform for the processing and analysis of Big Data. These are all things that are out of my reach!</p><h4><strong>Thank you for your honesty! Well, although you might now be slightly outdated, you are still an integral component in many legacy systems…</strong></h4><h4><strong>Ok, now let’s talk application. Let’s say someone is interested in using you. Where do they start?</strong></h4><p>I’d recommend them to download <a href="https://www.cloudera.com/downloads/hortonworks-sandbox.html">Hortonworks Data Platform (HDP) Sandbox.</a> The HDP Sandbox is a pre-configured learning environment that contains the latest developments from Apache Hadoop, including myself! The Sandbox comes packaged in a virtual environment that one can run in their personal machine, allowing them to get a hold of me in no time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/0*UNHChv4YEu2AwS-0.png" /><figcaption>A screenshot of Ambari, the dashboard-interface of HDP.</figcaption></figure><p>Once you download the Sandbox, you can run me through a couple of ways:</p><ul><li><strong>You can run me interactively using the Grunt shell. </strong>After ssh-ing to your Hadoop environment, simply type pig in your terminal to launch Grunt. Once Grunt is activated, you can start exploring Pig Latin by entering one line at a time.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/1*c2QBsDAqUQ6eHyVfglzPFg.png" /></figure><ul><li><strong>You can run a pig script. </strong>A pig script is just a text file that contains a series of Pig Latin commands. To run a script, you can simply type pig followed by the path to your Pig script.</li><li><strong>Pig View. </strong>Earlier versions of HDP (e.g. HDP 2.6.5) provides a web-UI called Pig View in which you can interactively run Pig Latin commands or scripts.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lB4nBOWl2qRtFD8z.png" /><figcaption>Pig View in Ambari</figcaption></figure><h4><strong>Cool! Now it’s time to talk about Pig Latin, your language. Can you give us some pointers about this language?</strong></h4><p>Sure! Pig Latin is a relatively simple scripting language. Through Pig Latin, users can command me to perform common data operations such as SELECT,JOIN and FILTER. Pig Latin is also extendable; users can develop and import UDFs to expand Pig Latin’s capability.</p><p>Here is an example of Pig Latin. The script below is the Pig Latin equivalent of the MapReduce program we saw earlier, which counts the occurrence of each distinct word in a text file. Take a look:</p><pre>lines = LOAD &#39;sourcefile.txt&#39; AS (line:chararray);<br>words = FOREACH lines GENERATE FLATTEN(TOKENIZE(line)) as word;<br>grouped = GROUP words BY word;<br>wordcount = FOREACH grouped GENERATE group, COUNT(words);<br>STORE wordcount INTO &#39;result.txt&#39;;</pre><h4><strong>Hmm… It looks a bit like SQL…</strong></h4><p>That seems to be a common opinion among my users — that Pig Latin is a procedural version of SQL. I don’t necessarily agree with this view. Although there are certain similarities, there are also some salient differences.</p><p><strong>SQL is a descriptive language.</strong> It lets users describe the information (i.e. the table) they need and will get it for them. <strong>Pig Latin, on the other hand, is a procedural language. </strong>In Pig Latin, users have to specify the steps that will lead them to their desired table (or <em>relation</em>, in Pig Latin terminology). The procedural nature of Pig Latin commands an advantage: by forcing users to write each step atomically, Pig Latin reduces the risk of conceptual bugs that often creep up in complicated SQL scripts.</p><p>While we are on the topic of language comparison, let me quickly point out that <strong>Pig Latin has also been described to be rather similar to Python’s Pandas </strong>🐼. Such a comparison stems from the fact that both Pig Latin and Pandas are procedural languages. IMO, we shouldn’t overstate the similarity of the two — Pandas is a much more powerful language equipped with extensive in-built functions and easy integration to Numpy and other Python tools. However, on the basic level, Pig Latin <em>does</em> have some overlap with Pandas.</p><h4>I think it is great that Pig Latin is similar to SQL and pandas—two languages with which most data analysts are already familiar. This overlap supports your earlier claim that you are easy to use! Do you have any source <strong>that compares Pig Latin and the two?</strong></h4><p>Certainly! In the table below, I have listed out common Pig Latin commands and their equivalents in SQL and Pandas. Do note that all there are lots and lots of variations to all these commands that can be invoked with different keywords and arguments, and the ones I show are just one possibility.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yy8vUG6EjQFp6pycUVUb-Q.png" /><figcaption>Common commands in Pig Latin and their SQL/Pandas equivalents</figcaption></figure><h4><strong>Wow, that’s awesome! That table will certainly help people who are familiar with SQL or Pandas get up to speed with writing Pig Latin!</strong></h4><p>Yep, that is my hope when I compiled that table 😇.</p><h4>Well Pig, this has been a rather long interview. To conclude our talk, do you have any last message for our readers?</h4><p>Sure! I hope that our conversation has given insight into who I am and how I fit in the Hadoop ecosystem. Although I may no longer be the sexiest Big Data tool in the market, I am still widely used in the industry and academia. I hope you will take the time to learn to use me 🙂.However, if you are pressed for time and want to simply master a single, Swiss-army-knife tool that can be applied to a wide range of Big Data problems, Spark might be your best bet.</p><p><em>I hope you enjoyed the interview. In case you were wondering, </em><strong>I</strong> <em>supplied both sides of the above dialogue. The personification of Apache Pig was intended to facilitate learning and this character does not in any way represent the opinions of the creators of Apache Pig.</em></p><p>If you have any comments about the article or would like to reach out to me, feel free to send me a connection through <a href="https://www.linkedin.com/in/meraldo/">LinkedIn</a>. Also, I’d be very grateful if you could support me by becoming a Medium member through <a href="https://medium.com/@meraldo.antonio/membership">my referral link</a>. As a member, you’ll be able to read all my writings on data science and personal development and have full access to all stories on Medium.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c44f06a4ca9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/interview-with-apache-pig-theory-and-practical-guide-6c44f06a4ca9">Interview with Apache Pig: Theory and Practical Guide</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Modeling and Output Layers in BiDAF — an Illustrated Guide with Minions!]]></title>
            <link>https://medium.com/data-science/modeling-and-output-layers-in-bidaf-an-illustrated-guide-with-minions-f2e101a10d83?source=rss-65b4354e2ed8------2</link>
            <guid isPermaLink="false">https://medium.com/p/f2e101a10d83</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Meraldo Antonio]]></dc:creator>
            <pubDate>Tue, 03 Sep 2019 16:34:51 GMT</pubDate>
            <atom:updated>2024-03-16T12:23:48.075Z</atom:updated>
            <content:encoded><![CDATA[<h4>The Definitive Guide to BiDAF — part 4 of 4</h4><h3><strong>Modeling and Output Layers in BiDAF</strong> — an Illustrated Guide with Minions</h3><h4>BiDAF is a popular machine learning model for Question and Answering tasks. This article explains the modeling layer of BiDAF with the help of some cute Minions.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*cUrRfbRQ2Sr75fFz" /><figcaption>Source: <a href="https://unsplash.com/photos/ltR0yi_YwFI">Unsplash</a></figcaption></figure><p>This article is the last in a series of four articles that aim to illustrate the working of <a href="https://arxiv.org/abs/1611.01603"><strong>Bi-Directional Attention Flow (BiDAF)</strong></a><strong>,</strong> a popular machine learning model for question and answering (Q&amp;A).</p><p>To recap, BiDAF is an <em>closed-domain, extractive</em> Q&amp;A model. This means that to be able to answer a <strong>Query</strong>, BiDAF needs to consult an accompanying text that contains the information needed to answer the Query. This accompanying text is called the <strong>Context. </strong>BiDAF works by extracting a substring of the Context that best answers the query — this is what what we refer to as the <strong>Answer</strong><em> </em>to the Query. I intentionally capitalize the words Query, Context and Answer to signal that I am using them in their specialized technical capacities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/607/1*zSsrgM-btCoduYDHR6MsMg.png" /><figcaption>An example of Context, Query and Answer</figcaption></figure><p>A quick summary of the three preceding articles are as follows:</p><ul><li><a href="https://medium.com/@meraldo.antonio/the-definitive-guide-to-bi-directional-attention-flow-d0e96e9e666b">Part 1</a> of the series provides a high-level overview of BiDAF.</li><li><a href="https://towardsdatascience.com/the-definitive-guide-to-bidaf-part-2-word-embedding-character-embedding-and-contextual-c151fc4f05bb">Part 2</a> explains how BiDAF uses 3 embedding algorithms to get the vector representations of words in the Context and the Query.</li><li><a href="https://towardsdatascience.com/the-definitive-guide-to-bidaf-part-3-attention-92352bbdcb07">Part 3</a> explores BiDAF’s attention mechanism that combines the information from the Context and the Query.</li></ul><p>The output of the aforementioned attention step is a giant matrix called <strong>G</strong>. <strong>G</strong> is a <strong>8d</strong>-by-<strong>T</strong> matrix that encodes the Query-aware representations of Context words. <strong>G </strong>is the input to the modeling layer, which will be the focus of this article.</p><h3>What Have We Been Doing? What Does G Actually Represent? <em>Minions to the Rescue!</em></h3><p>Ok, so I know we’ve been through a lot of steps in the past three articles. It is <em>extremely</em> easy to get lost in the myriad of symbols and equations, especially considering that the choices of symbols in the BiDAF paper aren’t that “user friendly.” I mean, do you even remember what each of <strong>H, U, Ĥ </strong>and <strong>Ũ </strong>represents?</p><p>Hence, let’s now step back and try to get the intuition behind all these matrix operations we have been doing so far.</p><blockquote><em>Practically, all the previous steps can be broken down to two collection of steps: </em><strong><em>the embedding steps</em></strong><em> and </em><strong><em>the attention steps</em></strong><em>.</em> As I mentioned above, the result of all these steps is an <strong>8d</strong>-by-<strong>T</strong> matrix called <strong>G</strong>.</blockquote><p>An example of <strong>G </strong>can be seen below. Each column of <strong>G </strong>is an <strong>8d</strong>-by-<strong>1</strong> vector representation of a word in the Context.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/444/1*0cF6HuD4WlKq7HcEH2Arww.png" /><figcaption>An example of <strong>G</strong>. The length of the matrix, <strong>T</strong>, equals the number of words in the Context (9 in this example). Its height is <strong>8d</strong>; <strong>d </strong>is a number that we preset in the word embedding and character embedding steps.</figcaption></figure><p>Let’s now play a little game that (hopefully) can help you understand all the mathematical mumbo-jumbo in the previously articles. Specifically, <strong>let’s think of the words in the Context as an ordered bunch of Minions.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sxtJALXTcR5v39kxy2FDsw.png" /><figcaption>Think of our Context as a bunch of Minions, with each Context word corresponding to one Minion</figcaption></figure><p>Each of our Minions has a brain in which he can store some information. Right now, our Minions’ brains are already pretty cramped. <strong>The current brain content of each Minion is equivalent to the 8d-by-1 column vector of the Context word that the Minion represents.</strong> Here I present the brain scan of the “Singapore” Minion:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/304/1*tYGi1OwYGpjAYlVKv0j2_g.png" /></figure><p>The Minions’ brains haven’t always been this full! In fact, when they came into being, their brains were pretty much empty. Let’s now go back in time and think about what were the “lessons” that the Minions went through to acquire their current state of knowledge.</p><p>The first two lessons the Minions had were “Word Embedding” and “Character Embedding.” <strong>During these lessons, the Minions learned about their own identities. </strong>The teacher in the “Word Embedding” class, Prof. GloVe, teaches the Minions basic information about their identities. On the other hand, the “Character Embedding” class is an anatomy class in which the Minions gained an understanding of their body structure through repeated scans.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*_fnYbhMSeIDa4whe.gif" /><figcaption>Minions in the “Character Embedding” class</figcaption></figure><p>Here is the brain scan of the “Singapore” Minion after these two lessons.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/1*mfrZNAza-Vi5GMuOocuECQ.png" /><figcaption>The “Singapore” Minion understands his identity after attending the “Word Embedding” and “Character Embedding” lessons</figcaption></figure><p>Right after, the Minions moved on and attended the “Contextual Embedding” lesson. <strong>This lesson is a conversational lesson during which the Minions had to talk to one another through a messenger app called bi-LS™. </strong>The bi-LS™-facilitated convo allows the Minions to learn each other’s identities<strong> </strong>—which they learned in the previous two lessons. Pretty neat, huh?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*hfo4osImRi7Unequ.gif" /><figcaption>Two Minions having a fun conversation through bi-LS™, sharing information about themselves. Source: Giphy</figcaption></figure><p>I took another MRI scan of the “Singapore” Minion right after the “Contextual Embedding” class. As you can see, now our little guy knows a bit more stuff!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/513/1*PKE4EIRDg9BP1A9YAyHsgA.png" /><figcaption>Now “Singapore” knows both his and his neighbors’ identities!</figcaption></figure><p>Our Minions were happily studying when suddenly a man barged into their school😱 It turns out that his name is <strong>Mr. Query</strong> and he is a journalist. Here he is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/0*grdcUkvOuO7FnAh2" /><figcaption>The inquisitive Mr. Query. He has an urgent question<strong> —</strong><em>”Where is Singapore situated”</em> — and he knows some of our Minions hold relevant information for this question.</figcaption></figure><p>Mr. Query urgently needs to collect some information for an article he is writing. Specifically, he wants to know “<em>where is Singapore situated.” </em>Mr. Query knows that <em>some</em> of our Minions hold this information in their brains.</p><p>Our Minions, helpful as they are, want to help Mr. Query out. To do so, they will need to select several members of their team to meet with Mr. Query and deliver the information he’s seeking. <strong>This bunch of Minions that have relevant information for Mr. Query and will be dispatched to him is known as the Answer Gang.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*i0dJoGxq2dV-NrLHAx5lNw.png" /><figcaption>The Answer Gang, which collectively holds the answer to Mr. Query’s question. Only relevant Minions can join the Answer Gang!</figcaption></figure><p><strong>Now, our Minions have a task to do — they need to collectively decide who should and shouldn’t join the Answer Gang. </strong>They need to be careful when doing so! If they leave out from the Answer Gang too many Minions that should’ve been included, Mr. Query won’t get all the information he needs. This situation is called <em>Low Recall</em> and Mr. Query hates that.</p><p>On the other hand, if too many unnecessary Minions join the Answer Gang, Mr. Query will be inundated with superfluous information. He calls such situation <em>Low Precision</em> and he doesn’t like that either! Mr. Query is known to have some anger management issues 👺 so it’s in our Minions’ best interest to supply Mr. Query with just the right amount of information.</p><p><em>So how do the Minions know which of them should join the Answer Gang?</em></p><p>The answer to this is by organizing several meet-up sessions that are collectively called “Attention.” During these sessions, each Minion gets to talk to Mr. Query separately and understand his needs. <strong>In other words, the Attention sessions allow the Minions to measure their importance to Mr. Query’s question.</strong></p><p>This is the MRI scan of the “Singapore” Minion’s brain as he saunters away from the Attention sessions. This is equivalent to the first brain scan image I showed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*7Rgj1vy10-mFVD7oap8GXA.png" /><figcaption>Singapore’s current brain content. He knows quite a bit— but he is still missing one thing!</figcaption></figure><p>As we can see, our Minions’ brain are now pretty full. With the Minions’ current state of knowledge, are they now in position to start choosing the members of the Answer Gang? Nope, not quite! They are still missing one key piece of information. Each of our Minions knows<em> his own </em>importance to Mr. Query. However, before they can make this important nomination,<strong> they will also need to be aware of <em>everyone&#39;s relative importance</em> to Mr. Query.</strong></p><p>As you might’ve guessed, this implies that the Minions have to talk to each other for the second time! And now you know that this conversation is done through the bi-LS™ app<strong>.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*zz_4Hb8ppc3AaJoO.png" /><figcaption>The Minions during the <strong>modeling step </strong>meeting. Here, they talk to each other through bi-LS™ and share their relative importance to Mr. Query. Source: <a href="https://www.freepnglogos.com/images/minions-9701.html">Free PNG Logo</a></figcaption></figure><p><strong>This bi-LS™-facilitated conversation is also known as the ‘modeling step’ and is the focus of our current article. </strong>Let’s now learn this step in detail!</p><h3><strong>Step 10. Modeling Layer</strong></h3><p>Okay, let’s leave our Minions for a while and get back to symbols and equations, shall we? It’s not <em>that </em>complicated, I promise!</p><p>The modeling layer is relatively simple. It consists of two layers of bi-LSTM. As mentioned above, the input to the modeling layer is <strong>G. </strong>The first bi-LSTM layer converts <strong>G </strong>into <strong>2d</strong>-by-<strong>T</strong> matrix called <strong>M1</strong>.</p><p><strong>M1 </strong>then acts as an input to the second bi-LSTM layer, which converts it to another <strong>2d</strong>-by-<strong>T </strong>matrix called <strong>M2</strong>.</p><p>The formation of <strong>M1 </strong>and <strong>M2 </strong>from <strong>G </strong>is illustrated below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ojdqFQ74YY0aC-_rkCSsOQ.jpeg" /><figcaption>Step 10. In the modeling layer, <strong>G </strong>is passed through two bi-LSTM layers to form <strong>M1 </strong>and <strong>M2</strong></figcaption></figure><p><strong>M1 </strong>and <strong>M2</strong> are yet another matrix representation of Context words. The difference between <strong>M1 </strong>and <strong>M2 </strong>and the previous representations of Context words are that <strong>M1 </strong>and <strong>M2 </strong>have embedded in them information about the entire Context paragraph as well as the Query.</p><p>In Minion-speak, this means that our Minions now have all the information they need to make the decision about who should be in the Answer Gang.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/1*misbSI-B7odDdxVpOp8JVA.png" /><figcaption>The “Singapore” guy now has all he needs to decide if he should join the Answer Gang.</figcaption></figure><h3><strong>Step 11. Output Layer</strong></h3><p>Okay, now we’ve reached the finale! Just one more step and then we’re done!</p><p>For each word in the Context, we have in our disposal two numeric vectors that encode the word’s relevance to the Query. That’s awesome! The very last thing we need is to<strong> convert these numeric vectors to two probability values so that we can compare the Query-relevance of all Context words</strong>. And this is exactly what the output layer does.</p><p>In the output layer, <strong>M1 </strong>and <strong>M2 </strong>are first vertically concatenated with <strong>G </strong>to form [<strong>G</strong>; <strong>M1</strong>] and [<strong>G</strong>; <strong>M2</strong>]. Both [<strong>G</strong>; <strong>M1</strong>] and [<strong>G</strong>; <strong>M2</strong>] have a dimension of <strong>10d</strong>-by-<strong>T</strong>.</p><p>We then obtain <strong>p1</strong>, the probability distribution of the start index over the entire Context, by the following steps:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ol4_FDFa_IVm653fNVbfOQ.png" /></figure><p>Similarly, we obtain <strong>p2</strong>, the probability distribution of the end index, by the following steps:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uiHEJ0Esl-rIryS2JRM12Q.png" /></figure><p>The steps to get <strong>p1 </strong>and <strong>p2 </strong>are depicted in the diagram below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IohHc3HjsJSOuTj5zsil4g.jpeg" /><figcaption>Step 11. The output layer, which converts <strong>M1 </strong>and <strong>M2 </strong>to two vector of probabilities, <strong>p1 </strong>and <strong>p2</strong>.</figcaption></figure><p><strong>p1 </strong>and <strong>p2 </strong>are then used to find the best Answer span. The best Answer span is simply a substring of the Context with the highest <em>span score</em>. The span score, in turn, is simply the product of the <strong>p1 </strong>score of the first word in that span and the <strong>p2</strong> score of the last word in the span. We then return the span with the highest span score as our Answer.</p><p>An example will make this clear. As you know, we are currently dealing with the following Query/Context pair:</p><ul><li><strong>Context</strong>: <em>“Singapore is a small country located in Southeast Asia.”</em> (<strong>T </strong>= 9)</li><li><strong>Query</strong>: <em>“Where is Singapore situated?”</em> (<strong>J</strong> = 4)</li></ul><p>After running this Query/Context pair through BiDAF, we obtain two probability vectors — <strong>p1 </strong>and <strong>p2 .</strong></p><p>Each word in the Context is associated with one <strong>p1 </strong>value and one <strong>p2 </strong>value. <strong>p1 </strong>values indicate the probability of the words being the start word of the answer span. Below are the the <strong>p1 </strong>values for our example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*qmMO65RS5jADlOY21nbxQg.png" /></figure><p>We see that the model thinks that the most probable start word for our Answer span is “Southeast.”</p><p><strong>p2 </strong>values indicate the probability of the words being the last word of the Answer span. Below are the the <strong>p2 </strong>values for our example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/943/1*0IAbK-YSscu8mc6jiuZjEQ.png" /></figure><p>We see that our model is very sure, with almost 100% certainty, that the most probable end word for our Answer span is “Asia.”</p><p>If in the original Context the word with the highest <strong>p1 </strong>comes <em>before</em> the word with the highest <strong>p2</strong>, then we have our best Answer span already — it will simply one that begins with the former and ends with the later. This is the case in our example. As such, the Answer returned by the model will simply be “Southeast Asia.”</p><p><em>That’s it, ladies and gentlemen — finally after 11 long steps we obtain the Answer to our Query!</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/1*RAsDiEatQ1s-WwRN_KTeug.png" /><figcaption>Here is Mr. Query with “Southeast” and “Asia”, both of whom have been selected to join the Answer Gang. It turns out that the information provided by “Southeast” and “Asia” is just what Mr. Query needs! Mr. Query is happy🎊</figcaption></figure><p>Okay, one caveat before we end this series. In the hypothetical case that the Context word with the highest <strong>p1 </strong>comes <em>after </em>the Context word with the highest <strong>p2</strong>, we still have a bit of work to do. In this case, we’d need to generate <em>all</em> possible answer spans and calculate the span score for each of them. Here are some examples of possible answer spans for our Query/Context pair:</p><ul><li><strong>Possible answer span: “</strong>Singapore” ; <strong>span score</strong>: 0.0000031</li><li><strong>Possible answer span: “</strong>Singapore is” ; <strong>span score</strong>: 0.00000006</li><li><strong>Possible answer span: “</strong>Singapore is a” ; <strong>span score</strong>: 0.0000000026</li><li><strong>Possible answer span: “</strong>Singapore is a small” ; <strong>span score</strong>: 0.0000316</li></ul><p>We then take the span with the highest span score to be our Answer.</p><p>So that was it — a detailed illustration of each step in BiDAF, from start to finish (sprinkled with a healthy dose of Minion-joy). I hope that this series has helped you in understanding this fascinating NLP model!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*Ak1LA_uH_Wco8SOG" /><figcaption>Source: <a href="https://giphy.com/gifs/minions-iUOzkJmvnFfqM">Giphy</a></figcaption></figure><p>If you have any questions/comments about the article or would like to reach out to me, feel free to do so either through <a href="https://www.linkedin.com/in/meraldo/">LinkedIn</a> or via email at meraldo.antonio AT gmail DOT com.</p><h3>Glossary</h3><ul><li><strong>Context : </strong>the accompanying text to a query that contains an answer to that query.</li><li><strong>Query : </strong>the question to which the model is supposed to give an answer.</li><li><strong>Answer : </strong>a substring of the Context that contains information that can answer the query. This substring is to be extracted out by the model.</li><li><strong>Span score : </strong>the product of <strong>p1 value </strong>of the first word in an answer span and <strong>p2 value </strong>of the last word in the answer span.</li><li><strong>T </strong>: the number of words in the Context.</li><li><strong>J </strong>: the number of words in the Query.</li><li><strong>G </strong>: a big, <strong>8d</strong>-by-<strong>T</strong> matrix that contains Query-aware Context representations. <strong>G </strong>is the input to the modeling layer.</li><li><strong>M1: </strong>a <strong>2d</strong>-by-<strong>T </strong>matrix obtained by passing <strong>G </strong>through a bi-LSTM. <strong>M1 </strong>contain vector representation of Context words that have information about the entire Context paragraph as well as the Query.</li><li><strong>M2: </strong>a <strong>2d</strong>-by-<strong>T </strong>matrix obtained by passing <strong>M1 </strong>through a bi-LSTM. <strong>M2</strong>, just like <strong>M1, </strong>contains vector representation of Context words that have information about the entire Context paragraph as well as the Query.</li><li><strong>p1: </strong>a probability vector with length <strong>T</strong>. Each Context word has its own <strong>p1 value. </strong>This <strong>p1 value </strong>indicates the probability of the word being the <strong>first word</strong> in the answer span.</li><li><strong>p2: </strong>a probability vector with length <strong>T</strong>. Each Context word has its own <strong>p2 value. </strong>This <strong>p2 value </strong>indicates the probability of the word being the <strong>last</strong> <strong>word</strong> in the answer span.</li></ul><h3>References</h3><p>[1] <a href="https://arxiv.org/abs/1611.01603">Bi-Directional Attention Flow for Machine Comprehension (Minjoon Seo <em>et. al</em>, 2017)</a></p><p>If you have any comments about the article or would like to reach out to me, feel free to send me a connection through <a href="https://www.linkedin.com/in/meraldo/">LinkedIn</a>. Also, I’d be very grateful if you could support me by becoming a Medium member through <a href="https://medium.com/@meraldo.antonio/membership">my referral link</a>. As a member, you’ll be able to read all my writings on data science and personal development and have full access to all stories on Medium.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f2e101a10d83" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/modeling-and-output-layers-in-bidaf-an-illustrated-guide-with-minions-f2e101a10d83">Modeling and Output Layers in BiDAF — an Illustrated Guide with Minions!</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Attention Mechanism in Seq2Seq and BiDAF — an Illustrated Guide]]></title>
            <link>https://medium.com/data-science/the-definitive-guide-to-bidaf-part-3-attention-92352bbdcb07?source=rss-65b4354e2ed8------2</link>
            <guid isPermaLink="false">https://medium.com/p/92352bbdcb07</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[coding]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Meraldo Antonio]]></dc:creator>
            <pubDate>Thu, 29 Aug 2019 18:00:28 GMT</pubDate>
            <atom:updated>2022-04-25T19:31:43.579Z</atom:updated>
            <content:encoded><![CDATA[<h4>The Definitive Guide to BiDAF — Part 3 of 4</h4><h3>Attention Mechanism in Seq2Seq and BiDAF — an Illustrated Guide</h3><h4>Sequence-to-sequence (seq2seq) and <strong>Bi-Directional Attention Flow (BiDAF) are influential NLP models. These models make use of a technique called “attention” that involves the comparison of two sequences. In this article, I explain how the attention mechanism works in these two models.</strong></h4><p>This article is the third in a series of four articles that aim to illustrate the working of <a href="https://arxiv.org/abs/1611.01603"><strong>Bi-Directional Attention Flow (BiDAF)</strong></a><strong>,</strong> a popular machine learning model for question and answering (Q&amp;A).</p><p>To recap, BiDAF is an <em>closed-domain, extractive</em> Q&amp;A model. This means that to be able to answer a <strong>Query</strong>, BiDAF needs to consult the an accompanying text that contains the information needed to answer the Query. This accompanying text is called the <strong>Context. </strong>BiDAF works by extracting a substring of the Context that best answers the query — this is what what we refer to as the <strong>Answer</strong><em> </em>to the Query. I intentionally capitalize the words Query, Context and Answer to signal that I am using them in their specialized technical capacities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*t0x7MJnEbpY4DEIqDvB1_w.png" /><figcaption>An example of <strong>Context, Query, </strong>and <strong>Answer. </strong>Notice how the Answer can be found verbatim in the Context.</figcaption></figure><p>In the <a href="https://medium.com/@meraldo.antonio/the-definitive-guide-to-bi-directional-attention-flow-d0e96e9e666b">first article</a> in the series, I presented a high level overview of BiDAF. In the <a href="https://medium.com/@meraldo.antonio/the-definitive-guide-to-bidaf-part-2-word-embedding-character-embedding-and-contextual-c151fc4f05bb#59ee-8e19973bf213">second article</a>, I talked about how BiDAF uses 3 embedding layers to get vector representations of the Context and the Query. The final outputs of these embedding steps are two matrices — <strong>H</strong> (which represent words in the Context) and <strong>U</strong> (which represent words in the Query). <strong>H</strong> and <strong>U</strong> are the inputs of the attention layers, whose function is to combine their informational content.</p><p>These attention layers are the core component in BiDAF that differentiates it from earlier models and enables it to score highly in the SQuAD leaderboard. The workings of the attention layers will be the focus of this article. Let’s dive in!</p><h3><strong>Conceptual Introduction to Attention Mechanism</strong></h3><p>Before delving into the details of the attention mechanism used in BiDAF, it behooves us to first have an understanding of what attention is. Attention was first introduced in 2016 as a part of a <a href="https://arxiv.org/abs/1409.0473">sequence-to-sequence</a> (seq2seq) model. <strong>As its name suggests, seq2seq is a neural network model whose aim is to convert one sequence to another sequence. </strong>An example application of seq2seq is the translation of a 🇫🇷 French sentence to an English 🇬🇧 sentence, such as the one below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tTvdvb7SftMMKLmueyrU7w.png" /><figcaption>A sample translation task for seq2seq model.</figcaption></figure><p>A seq2seq model consists of two Recurrent Neural Networks (RNNs):</p><ul><li>The first RNN, called <em>“encoder”</em>, is responsible for understanding the input sequence (in our example, a French sentence) and converting its informational content into a fixed-size intermediary vector.</li><li>The second RNN, called <em>“decoder”</em>, then uses this intermediary vector to generate an output sequence (in our example, the English translation of the French sentence).</li></ul><p><strong>Prior to the incorporation of attention mechanism, seq2seq models can only deal with short sequences.</strong> This restriction arises because the “vanilla” seq2seq models can only fit a limited amount of information into the intermediary vector and some informational content is loss in the process.</p><p>This situation is akin to an attempt to translate a French book by first reading the whole book, memorizing its content and <em>then</em> translating it into English just from memory. As you can imagine, such an attempt is bound to fail— our poor translator will forget most of the book by the time he starts writing the English translation!</p><blockquote>The attention mechanism was developed to solve this information bottleneck. The core idea of attention is that on each step of the decoding process, we are to make a direct connection to specific parts of the encoder.</blockquote><p>In the context of our French-English translation task, this means that at e<strong>very time our model is to generate the next English word, it will only focus on the most relevant portions of the input French sentence.</strong></p><p>Conceptually, a seq2seq translation model with attention works just like how a normal human translator would translate a French text. He would read the first paragraph of the French text and translate it into English, move on to the second paragraph and translate this one, and so on. By doing so, he doesn’t have to commit the whole content of the book into his memory and run the risk of forgetting most of its content.</p><h3>Implementation of Attention in Sequence-to-Sequence Model</h3><p>Practically, we can include an attention mechanism in a seq2seq model by performing the following three steps that are depicted in the diagram below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1V221DO9QIafh4htkwVBYw.jpeg" /><figcaption>Attention mechanism in seq2seq</figcaption></figure><h4><strong>1. Comparison of Sequences and Calculation of Attention Scores</strong></h4><p><em>At each time step during the decoding process, we will compare the decoder hidden state with all of the encoder hidden states.</em><strong> </strong>This comparison can be done using any function that takes two vectors and outputs a scalar that reflects their similarity. The simplest of such function is a simple dot product. The scalar output of the similarity function is called an <strong><em>“attention score”; </em></strong>these attention scores are depicted by blue circles 🔘 in the diagram above.</p><h4><strong>2. Conversion of Attention Scores to Attention Distribution</strong></h4><p><em>We then take the softmax of all these attention scores.</em><strong> </strong>The softmax function normalizes these attention scores into a probability distribution (a set of numbers that sum up to one). This probability distribution is called the <em>“</em><strong><em>attention distribution”</em></strong> ; it signals the parts of the input sequence that are most relevant to the decoding process at hand.</p><p>The blue bars in the diagram above show the attention distribution. We see that the bar corresponding to the second 🇫🇷 French word, “<em>me”,</em> is the tallest ; this is because this word translates to “I” in 🇬🇧 English, which is the first word in our output sequence.</p><h4><strong>3. Multiplying Attention Distribution with Encoder Hidden States to Get Attention Output</strong></h4><p><em>We then multiply each element of the attention distribution with its corresponding encoder hidden states and sum up all of these products to produce a single vector called the </em><strong><em>“attention output”.</em></strong><em> </em>You can think of the attention output as a selective summary of the input sequence. The attention output will then become the input to the next decoding step.</p><p>Although the three attention steps above were first applied to seq2seq, they are generalizable to other applications. As we will see later, BiDAF uses the same three steps in its implementation of attention, albeit with some minor modifications.</p><p>With this quick overview of the attention mechanism and its implementation in seq2seq, we are now ready to see how this concept is applied in BiDAF. <em>C’est parti!</em></p><h3><strong>Step 6. Formation of Similarity Matrix</strong></h3><p>Just to remind you where we left off in <a href="https://medium.com/@meraldo.antonio/the-definitive-guide-to-bidaf-part-2-word-embedding-character-embedding-and-contextual-c151fc4f05bb">the last article</a> — our last step (step 5) was a contextual embedding step that produced two output matrices: <strong>H</strong> for the Context and <strong>U</strong> for the Query. Our overarching goal for the attention steps (steps 6 to 9) is to fuse together information from <strong>H </strong>and <strong>U</strong> to create several matrix representations of the Context that also contain information from the Query.</p><p><strong><em>Our sixth step — the first attention-related step — is the formation of the so-called similarity matrix </em>S<em>.</em></strong> <strong>S</strong> is a tall skinny matrix with a dimension of <strong>T</strong>-by-<strong>J </strong>(number of words in Context by number of words in the Query).</p><p>The generation of the similarity matrix <strong>S</strong> corresponds to the first step in the seq2seq attention mechanism discussed above. It entails applying a comparison function to each column in <strong>H</strong> and each column in <strong>U</strong>.<strong> </strong><em>The value in row </em><strong>t<em> </em></strong><em>and column </em><strong>j<em> </em></strong><em>of the matrix </em><strong>S </strong><em>represents the similarity of</em> <strong>t</strong><em>-th Context word and </em><strong>j</strong><em>-th Query word.</em></p><p>Let’s take a look at an example of similarity matrix <strong>S. </strong>Suppose we have this Query/Context pair:</p><ul><li><strong>Context</strong>: <em>“Singapore is a small country located in Southeast Asia.”</em> (<strong>T </strong>= 9)</li><li><strong>Query</strong>: <em>“Where is Singapore situated?”</em> (<strong>J</strong> = 4)</li></ul><p>The similarity matrix <strong>S </strong>produced from the above Query/Context pair is shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/757/1*mgA8fZSwsoGg772bVSBcQw.png" /><figcaption>An example of similarity matrix <strong>S</strong></figcaption></figure><p>We can make a couple of observations from the matrix <strong>S </strong>above:</p><ul><li>As we expect, the matrix has a dimension of 9-by-4, 9 being the length of the Context(<strong>T</strong>)<strong> </strong>and 4 being the length of the Query (<strong>J</strong>).</li><li>The cell in row 1, column 3 contains a relatively high value as indicated by its bright yellow color. This implies that the Query word and Context word associated with this coordinate are highly similar to each other. These words turn out to be the exact same word — “Singapore” — hence it makes sense that their vector representations are very similar.</li><li><strong>Just because a Context word and a Query word are identical doesn’t necessarily imply that their vector representations are highly similar! </strong>Look at the cell in row 2, column 2 — this cell encodes the similarity of the Context word “is” and the identical Query word “is”. However, its value is not as high as the “Singapore” par above. This is because these vector representations also incorporates information from the surrounding phrases. This contextual contribution is especially important for small copulas such as “is”.</li><li><strong>On the other hand, we can see that the similarity value of two distinct words with close semantic and grammatical meaning, such as “situated” and “located” is relatively high.</strong> This is thanks to our word and character embedding layers, which can generate vector representations that pretty accurately reflect a word’s meaning.</li></ul><p>Now let me tell you about how we calculate values in <strong>S</strong>. <em>The comparison function used to perform this calculation is called </em><strong>α</strong><em>.</em> <strong>α</strong><em> </em>is more complex than the dot product used in the seq2seq; here is the equation for <strong>α</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/1*wNIggOqRtfUs9Tn1JfRKHg.png" /></figure><p>As the function <strong>α</strong> comprises a multiplication of a row vector and a equally-sized column vector, it always returns a scalar.</p><p>The diagram below shows all the matrix operations performed in this step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aOTpjr7kRtTkt4Hv8or1BQ.jpeg" /><figcaption>Step 6. Using the similarity function <strong>α</strong>, we combine context matrix <strong>H </strong>and query matrix <strong>U </strong>to form similarity matrix <strong>S.</strong></figcaption></figure><h3><strong>Step 7. Context-to-Query </strong>Attention <strong>(C2Q)</strong></h3><p>The similarity matrix <strong>S </strong>serves as an input to the next two steps: Context-to-Query attention (C2Q) and Query-to-Context attention (Q2C).</p><blockquote><strong><em>In this section, we will focus on C2Q. The goal of this step is to find which Query words are most relevant to each Context words.</em></strong></blockquote><p>Performing C2Q is similar to performing the second and the third steps of the seq2seq attention. First, we use the scalar values in <strong>S </strong>to calculate the attention distribution. This is done by taking the row-wise softmax of <strong>S</strong>. The result is another matrix. This matrix is not explicitly named in the BiDAF paper, but let’s call it matrix <strong>A</strong>.</p><p>The matrix <strong>A</strong>, whose dimension is the same as <strong>S</strong>, indicates which Query words are the most relevant to each Context word. Let’s look at an example of <strong>A</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/605/1*Cag5weayf5MyrP-YjnzEzA.png" /><figcaption>An example of matrix <strong>A</strong>, the row-wise softmaxed version of <strong>S</strong></figcaption></figure><p>By observing the heatmap above, we can conclude that:</p><ul><li><strong>Semantic similarity strongly contributes to relevance. </strong>We see that for the Context words [“Singapore”, “is”, “located”], the most relevant Query words are [“Singapore”, “is”, “situated”] . These are also words with which they share strong semantic similarity.</li><li><strong>Context words “understand” the information requested by query words. </strong>We see that the Query word “Where” is the most relevant query word for the Context words [“a”, “small”, “country”, “in” ,“Southeast”, “Asia”]— words related to geographical location.</li></ul><p>We then take every row of <strong>A </strong>to get a the attention distribution <strong>At: </strong>which has a dimension of 1-by-<strong>J</strong>. <strong>At:<em> </em></strong><em>reflects the relative importance of each Query word for the t-th Context word.</em></p><p>We then calculate the weighted sum of the query matrix <strong>U </strong>with respect to each element in the attention distribution <strong>At: . </strong>The result of this step is the attention output matrix called <strong>Ũ</strong>, which is a <strong>2d</strong>-by-<strong>T </strong>matrix.</p><blockquote><strong>Ũ</strong>, just like <strong>H</strong>, is a matrix representation of the Context. However, <strong>Ũ</strong> contains different information from <strong>H</strong>! Whereas <strong>H</strong> encapsulates the semantic, syntactic and contextual meaning of each Context word, <strong>Ũ</strong> encapsulates the information about the relevance of each query word to each Context word.</blockquote><p>The whole process to generate <strong>Ũ </strong>from similarity matrix <strong>S </strong>and query matrix <strong>U </strong>is depicted below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*keaPvJWv0qJRHL4UvaDcjQ.jpeg" /><figcaption>Step 7. Context-to-Query attention</figcaption></figure><h3><strong>Step 8. Query-to-Context (Q2C) Attention</strong></h3><blockquote>The next step is Q2C, which like C2Q also starts with the similarity matrix <strong>S</strong>.<strong> <em>In this step, our goal is to find which Context word is most similar to either one of the Query words hence are critical for answering the Query.</em></strong></blockquote><p>We first take the maximum across row of the similarity matrix <strong>S</strong> to get a column vector. This column vector is not explicitly named in the paper, but let’s call it <strong>z</strong>.</p><p>Let’s now step back and think about what <strong>z </strong>symbolizes. Remember, our similarity matrix <strong>S</strong> records the similarity between each Context word and each Query word. Let’s take a second look at the example <strong>S</strong> we created above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/757/1*mgA8fZSwsoGg772bVSBcQw.png" /><figcaption>Similarity matrix <strong>S</strong></figcaption></figure><p>Now let’s focus our attention to the the fourth row of this matrix, which corresponds to the Context word “small”. <em>One can see that there isn’t any bright cell across this row! </em>This indicates that there is no word in the query that is similar in meaning to the Context word “small”. When we take the maximum across this row, the maximum value obtained will be close to zero.</p><p>Contrast this with the word “Singapore” and “situated”, in whose rows we do find at least one bright cell, indicating the existence of query word(s) similar to these words. When we take the maximum of these two rows, the corresponding value in the vector <strong>z </strong>for these rows will also be relatively high.</p><p>Here is the <strong>z </strong>obtained for our example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/263/1*EysU0QBkZx7EDWvhf-17SQ.png" /></figure><p>In Q2C, the values in <strong>z </strong>serve our attention values. We apply softmax on <strong>z </strong>to get an attention distribution called <strong>b</strong>. We then use <strong>b </strong>to take a weighted sum of the Context matrix <strong>H. </strong>The resulting attention output is a <strong>2d</strong>-by-1 column vector called <strong>ĥ.</strong></p><blockquote>The last step of Q2C is copying-and-pasting <strong>ĥ </strong>T times and combine these copies into a <strong>2d</strong>-by-<strong>T</strong> matrix called <strong>Ĥ. Ĥ is yet another representation of the Context that encapsulates the information about the most important words in the Context with respect to the Query.</strong></blockquote><p>The whole process to generate <strong>Ĥ </strong>from similarity matrix <strong>S </strong>and Context matrix <strong>H </strong>is depicted below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ct8c5ytfmFJ71nEz_YUfcw.jpeg" /><figcaption>Step 8. Query-to-Context attention</figcaption></figure><h3><strong>Step 9. “Megamerge”</strong></h3><p>The matrices produced in steps 5, 7 and 8 are then combined to form a giant matrix <strong>G. </strong>To refresh your memory, these three matrices are as follows:</p><ul><li><strong>H</strong> : the original Context matrix that encapsulates <em>the semantic, syntactic and contextual meaning of Context words.</em></li><li><strong>Ũ</strong> : Context matrix that encapsulates <em>the relevance of each Query word to each Context word.</em></li><li><strong>Ĥ </strong>: Context matrix that encapsulates <em>the information about the most important words in the context with respect to the Query.</em></li></ul><p>These three matrices have the same dimension — <strong>2d</strong>-by-<strong>T.</strong></p><p>This “megamerging”, unfortunately, is not as simple as stacking these three matrices vertically! Here, we make use of another custom function called <strong>β</strong>. Here is the equation for <strong>β</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/443/1*dyfJPkDigVC8H6oTWLzJHw.png" /></figure><p>We define our megamerged matrix <strong>G</strong> by <strong>G:t</strong> = β(<strong>H</strong>:t, <strong>Ũ</strong>:t, <strong>Ĥ</strong>:t) where <strong>G:t</strong> is the t-th column vector of <strong>G </strong>that corresponds to t-th Context word. <strong>G </strong>has a dimension of <strong>8d</strong>-by-<strong>T.</strong></p><p>The whole process to generate <strong>G </strong>from <strong>H</strong>, <strong>Ũ </strong>and <strong>Ĥ </strong>is depicted below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UrSetp-cxMiJ9jOe4F0t6A.jpeg" /><figcaption>Step 9. Merging of the three Context matrices <strong>H</strong>, <strong>Ũ </strong>and <strong>Ĥ </strong>to form <strong>G</strong></figcaption></figure><p>The giant matrix <strong>G </strong>contains all information in <strong>H</strong>, <strong>Ũ </strong>and <strong>Ĥ. </strong><em>That is, you can think of each column vector in </em><strong><em>G </em></strong><em>as a vector representation of a Context word that is “aware” of the existence of the Query and has incorporated relevant information from it.</em></p><p>So that’s all there is to it about the attention layers in BiDAF! I know this might be a lot to take in, especially with the myriad of symbols and matrix operations involved. If you really want to study BiDAF in detail, I recommend you to print out the glossary below as well as all the diagrams above and study them side-by-side.</p><p>In the <a href="https://towardsdatascience.com/modeling-and-output-layers-in-bidaf-an-illustrated-guide-with-minions-f2e101a10d83">next article</a>, which will be the last article of the series, I will discuss about how <strong>G </strong>serves as an input to the modeling and output layer. The output layer will be the one that gives out probabilities for each Context word being included in the Answer span. I hope to see you in that <a href="https://towardsdatascience.com/modeling-and-output-layers-in-bidaf-an-illustrated-guide-with-minions-f2e101a10d83">last article</a>!</p><h3>Glossary</h3><ul><li><strong>Context : </strong>the accompanying text to a query that contains an answer to that query</li><li><strong>Query : </strong>the question to which the model is supposed to give an answer</li><li><strong>Answer : </strong>a substring of the Context that contains information that can answer the query. This substring is to be extracted out by the model.</li><li><strong>T </strong>: the number of words in the Context</li><li><strong>J </strong>: the number of words in the Query</li><li><strong>U </strong>: the original Context matrix that encapsulates<em> </em>the semantic, syntactic and contextual meaning of query words. <strong>U </strong>has a dimension of <strong>2d</strong>-by-<strong>J</strong></li><li><strong>H</strong> : the original Context matrix that encapsulates<em> </em>the semantic, syntactic and Contextual meaning of Context words. <strong>H </strong>has a dimension of <strong>2d</strong>-by-<strong>T</strong></li><li><strong>S</strong> : the similarity matrix that records the similarity between each Context word and each query word. <strong>S </strong>has a dimension of <strong>T</strong>-by-<strong>J </strong>(number of words in the Context by number of words in the Query).</li><li><strong>α : </strong>the comparison function used to get similarity values in <strong>S.</strong></li><li><strong>A : </strong>the matrix that results from the row-wise softmax of <strong>A </strong>and indicates which query words are the most relevant to each Context word. <strong>A </strong>has a dimension of <strong>T</strong>-by-<strong>J.</strong></li><li><strong>z : </strong>the column vector obtained by taking the maximum across row of the similarity matrix <strong>S. z </strong>has a dimension of <strong>T</strong>-by-1.</li><li><strong>b : </strong>an attention distribution vector that comes as the result of applying softmax to <strong>z. b </strong>has a dimension of <strong>T</strong>-by-1.</li><li><strong>ĥ : </strong>an attention output obtained by multiplying <strong>b </strong>with Context matrix <strong>H. ĥ </strong>has a dimension of <strong>2d</strong>-by-1.</li><li><strong>Ũ</strong> : Context matrix that encapsulates the relevance of each query word to each Context word. <strong>Ũ </strong>has a dimension of <strong>2d</strong>-by-<strong>T.</strong></li><li><strong>Ĥ </strong>: Context matrix that encapsulates the information about the most important words in the Context with respect to the query. <strong>Ĥ </strong>has a dimension of <strong>2d</strong>-by-<strong>T.</strong></li><li><strong>G </strong>: a big, <strong>8d</strong>-by-<strong>T</strong> matrix that contains all information in <strong>H</strong>, <strong>Ũ </strong>and <strong>Ĥ. G </strong>is the input to the modeling layer.</li><li><strong>β : </strong>a custom concatenation function used to build <strong>G</strong></li></ul><h3>References</h3><p>[1] <a href="https://arxiv.org/abs/1611.01603">Bi-Directional Attention Flow for Machine Comprehension (Minjoon Seo <em>et. al</em>, 2017)</a></p><p><a href="https://arxiv.org/abs/1409.0473">[2] Neural Machine Translation by Jointly Learning to Align and Translate (Bahdanau <em>et. al</em>, 2015)</a></p><p>All images were created/designed by me unless specified otherwise.</p><p>If you have any comments about the article or would like to reach out to me, feel free to send me a connection through <a href="https://www.linkedin.com/in/meraldo/">LinkedIn</a>. Also, I’d be very grateful if you could support me by becoming a Medium member through <a href="https://medium.com/@meraldo.antonio/membership">my referral link</a>. As a member, you’ll be able to read all my writings on data science and personal development and have full access to all stories on Medium.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=92352bbdcb07" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/the-definitive-guide-to-bidaf-part-3-attention-92352bbdcb07">Attention Mechanism in Seq2Seq and BiDAF — an Illustrated Guide</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Word Embedding, Character Embedding and Contextual Embedding in BiDAF — an Illustrated Guide]]></title>
            <link>https://medium.com/data-science/the-definitive-guide-to-bidaf-part-2-word-embedding-character-embedding-and-contextual-c151fc4f05bb?source=rss-65b4354e2ed8------2</link>
            <guid isPermaLink="false">https://medium.com/p/c151fc4f05bb</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[computer-science]]></category>
            <dc:creator><![CDATA[Meraldo Antonio]]></dc:creator>
            <pubDate>Wed, 28 Aug 2019 18:29:21 GMT</pubDate>
            <atom:updated>2022-04-25T19:32:14.612Z</atom:updated>
            <content:encoded><![CDATA[<h4>The Definitive Guide to BiDAF — Part 2 of 4</h4><h3><strong>Word Embedding, Character Embedding and Contextual Embedding in BiDAF</strong> — <strong>an Illustrated Guide</strong></h3><h4><strong>BiDAF is </strong>a popular machine learning model for Question and Answering tasks. This article illustrates how BiDAF uses three embedding mechanisms to convert words into their vector representations.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*8crcJbkfEzl4kfRm" /><figcaption>Source: Brett Jordan from <a href="https://unsplash.com/photos/7PYqjNzvrc4">Unsplash</a></figcaption></figure><p>This article is the second in a series of four articles that aim to illustrate the working of <a href="https://arxiv.org/abs/1611.01603"><strong>Bi-Directional Attention Flow (BiDAF)</strong></a><strong>,</strong> a popular machine learning model for question and answering (Q&amp;A).</p><p>To recap, BiDAF is an <em>closed-domain, extractive</em> Q&amp;A model. This means that to be able to answer a <strong>Query</strong>, BiDAF needs to consult the an accompanying text that contains the information needed to answer the Query. This accompanying text is called the <strong>Context. </strong>BiDAF works by extracting a substring of the Context that best answers the query — this is what what we refer to as the <strong>Answer</strong><em> </em>to the Query. I intentionally capitalize the words Query, Context and Answer to signal that I am using them in their specialized technical capacities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*t0x7MJnEbpY4DEIqDvB1_w.png" /><figcaption>An example of <strong>Context, Query, </strong>and <strong>Answer. </strong>Notice how the Answer can be found Verbatim in the Context.</figcaption></figure><p><a href="https://medium.com/@meraldo.antonio/the-definitive-guide-to-bi-directional-attention-flow-d0e96e9e666b">The first article</a> in the series provided a high-level overview of BiDAF. In this article, we will focus on the first portion of the BiDAF architecture — the first thing that takes place when the model receives an incoming Query and its accompanying Context. To facilitate your learning, a glossary containing the mathematical notations involved in these steps is provided at the end. Let’s dive in!</p><h3><strong>Step 1. Tokenization</strong></h3><p><strong>In BiDAF, the incoming Query and its Context are first tokenized, i.e. these two long strings are broken down into their constituent words.</strong> In the BiDAF paper, the symbols <strong>T </strong>and <strong>J </strong>are used to denote the number of words in Context and Query, respectively. Here is a depiction of the tokenization:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*P1xy5WJMBQSu5WKD" /><figcaption>Step 1. The incoming Query and its accompanying Context are tokenized into their constituent words.</figcaption></figure><h3><strong>Step 2. Word Level Embedding</strong></h3><p><strong>The resulting words are then subjected to the embedding process, where they are converted into vectors of numbers.</strong> These vectors capture the grammatical function (<em>syntax</em>) and the meaning (<em>semantics</em>) of the words, enabling us to perform various mathematical operations on them. In BiDAF, embedding is done on three levels of granularity: on the character, word and contextual levels. Let’s now focus on the first embedding layer — the word embedding.</p><p><strong>The word embedding algorithm used in the original BiDAF is </strong><a href="https://nlp.stanford.edu/projects/glove/"><strong>GloVe</strong></a><strong>.</strong> In this article, I will only give a brief overview of GloVe because there already exist several <a href="http://mlexplained.com/2018/04/29/paper-dissected-glove-global-vectors-for-word-representation-explained/">excellent</a> resources that explain how the algorithm works. But if you are short on time, here is a very simplified summary of GloVe:</p><blockquote><strong>GloVe is an unsupervised learning algorithm that uses co-occurrence frequencies of words in a corpus to generate the words’ vector representations. </strong>These vector representations numerically represent various aspects of the words’ meaning.</blockquote><p>As the numbers that make up GloVe vectors encapsulate semantic and syntactic information about the words they represent, we can perform some cool stuff using these vectors! For instance, we can use subtraction of GloVe vectors to find word analogies, as illustrated below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*O5JMFxfwgeJcYDMQ.jpg" /><figcaption>Pairwise distances of concepts such as aunt-uncle and woman-man are similar. Source: <a href="https://nlp.stanford.edu/projects/glove/">Stanford&#39;s NLP Group</a></figcaption></figure><p><strong>BiDAF uses pre-trained GloVe embeddings to get the vector representation of words in the Query and the Context. </strong>“Pre-trained” means that the GloVe representations used here have already been trained; their values are frozen and won’t be updated during training. Thus, you can think of BiDAF’s word embedding step as a simple dictionary lookup step where we substitute words (the “keys” of the GloVe “dictionary”) with vectors (the “values” of the “dictionary”).</p><p>The output of the word embedding step is two matrices — one for the Context and one for the Query. The lengths of these matrices equal the number of words in the Context and the Query (<strong>T </strong>for the Context matrix and <strong>J </strong>for the Query matrix). Meanwhile, their height , <strong>d1</strong>, is a preset value that is equal to the vector dimension from GloVe; this can either be 50, 100, 200 or 300. The figure below depicts the word embedding step for the Context:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xPHimSKPamAmoKAeQNM7yw.jpeg" /><figcaption>Step 2. The word embedding step converts Context tokens into a <strong>d1</strong>-by-<strong>T</strong> matrix and Query tokens into a <strong>d1</strong>-by-<strong>J</strong> matrix</figcaption></figure><h3><strong>Step 3. Character Level Embedding</strong></h3><p>Okay, so with GloVe, we obtain the vector representations of most words. However, the GloVe representations are not enough for our purpose!</p><p>The pretrained GloVe “dictionary” is huge and contains millions of words; however, there will come a time where we encounter a word in our training set that is not present in GloVe’s vocabulary. Such a word is called an out-of-vocabulary (OOV) word. <strong>GloVe deals with these OOV words by simply assigning them some random vector values.</strong> If not remedied, this random assignment would end up confusing our model.</p><p>Therefore, we need another embedding mechanism that can handle OOV words. This is where the character level embedding comes in. <strong>Character level embedding uses one-dimensional convolutional neural network (1D-CNN) to find numeric representation of words by looking at their character-level compositions.</strong></p><blockquote>You can think of 1D-CNN as a process where we have several scanners sliding through a word, character by character. <strong>These scanners can focus on several characters at a time. As these scanners sweep along, they extract the information from the characters they are focusing on.</strong> At the end of these scanning process, information from different scanners are collected to form the representation of a word.</blockquote><p>The output of the character embedding step is similar to the output of the word embedding step. We obtain two matrices, one for the Context and the other for the Query. The lengths of these matrices equal the number of words in the Context and in the Query — <strong>T </strong>and <strong>J </strong>— while their height depends on the number of <em>convolutional filters</em> used in 1D-CNN (to know what a “convolutional filter” is, do read the next section). The height is denoted as <strong>d2 </strong>in the diagram. These two matrices will be concatenated with the matrices that we obtained from the word embedding step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dnS6itQPZwj_DnVWsdh0ig.jpeg" /><figcaption>Step 3. The character embedding step converts Context tokens into a <strong>d2</strong>-by-<strong>T</strong> matrix and Query tokens into a <strong>d2</strong>-by-<strong>J</strong> matrix</figcaption></figure><h4><strong>Additional Details on 1D-CNN</strong></h4><p>The section above only presents a very conceptual overview of the workings of 1D-CNN. In this section, I will explain how 1D-CNN works in details. Strictly speaking, these details are not necessary to understand how BiDAF works; as such, feel free to jump ahead if you are short on time. However, if you are the type of person who can’t sleep well without understanding every moving part of an algorithm you are learning about, this section is for you!</p><blockquote>The idea that motivates the use of 1D-CNN is that not only words as a whole have meanings — <strong><em>word parts can carry meaning, too!</em></strong></blockquote><p>For example, if you know the meaning of the word “underestimate”, you will understand the meaning of “misunderestimate”, although the latter isn’t actually a real word. Why? Because you know from your knowledge of the English language that the prefix “mis-” usually indicates the concept of “mistaken”; this allows you to deduce that “misunderestimate” refers to “mistakenly underestimate” something.</p><p>1D-CNN is an algorithm that mimics this human capability to understand word parts. <strong>More broadly speaking, 1D-CNN is an algorithm capable of extracting information from shorter segments of a long input sequence.</strong> This input sequence can be music, DNA, voice recording, weblogs, etc. In BiDAF, this “long input sequence” is words and the “shorter segments” are the letter combinations and morphemes that make up the words.</p><p>To understand how 1D-CNN works, let’s look at the series of illustrations below, which are taken from <a href="https://nlp.seas.harvard.edu/slides/aaai16.pdf">slides</a> by Yoon Kim et. al., a group from Harvard University.</p><ol><li>Let’s say we want to apply 1D-CNN on the word “absurdity”. The first thing we do is represent each character in that word as a vector of dimension <strong>d</strong>. These vectors are randomly initialized. Collectively, these vectors form a matrix <strong>C. <em>d </em></strong>is the height of this matrix, while its length, <strong><em>l</em></strong><em>, </em>is simply the number of characters in the word. In our example, <strong><em>d </em></strong>and <strong><em>l </em></strong>are 4 and 9, respectively.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/0*CX27cQ0fROrkYX1m" /></figure><p>2. Next, we create a <em>convolutional filter</em> <strong>H</strong>.<strong> </strong>This convolutional filter (also known as “kernel”) is a matrix with which we will “scan” the word. Its height, <strong><em>d</em></strong><em>, </em>is the same as the height of <strong>C</strong> but its width <strong>w </strong>is a number that is smaller than <strong><em>l</em></strong><em>. </em>The values within <strong>H </strong>are randomly initialized and will be adjusted during model training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/0*a3L5elzVXysBDWRe" /></figure><p>3. We overlay <strong>H </strong>on the leftmost corner of <strong>C</strong> and take an element-wise product of <strong>H </strong>and its projection on <strong>C </strong>(a fancy word to describe this process is taking a <em>Hadamard product </em>of <strong>H </strong>and its projection on <strong>C</strong>). This process outputs a matrix that has the same dimension as <strong>H </strong>— a <strong><em>d</em> </strong>x <strong><em>l</em> </strong>matrix. We then sum up all the numbers in this output matrix to get a scalar. In our example, the scalar is 0.1. This scalar is set as the first element of a new vector called <strong>f</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/547/0*nyu2nN9jPdWBVcp4" /></figure><p>4. We then slide <strong>H </strong>one character to the right and perform the same operations (get the Hadamard product and sum up the numbers in the resulting matrix) to get another scalar, 0.7. This scalar is set as the second element of <strong>f.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/553/0*thPjPYoXzgKBRDty" /></figure><p>5. We repeat these operations character by character until we reach the end of the word. In each step, we add one more element to <strong>f </strong>and lengthen the vector until it reaches its maximum length which is <strong>l - w+1. </strong>The vector<strong> f </strong>is a numeric representation of the word “absurdity” obtained when we look at this word <em>three characters at a time</em>. One thing to note is that the values within the convolution filter <strong>H </strong>don’t change as <strong>H</strong> slides through the word. In fancier terms, we call <strong>H</strong> “position invariant”. <strong>The position invariance of the convolutional filters enables us to capture the meaning of a certain letter combination no matter where in the word such combination appears.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*jludjFa2-Xd9DFsB" /></figure><p>6. We record the maximum value in <strong>f. </strong>This maximum can be thought of as the “summary” of <strong>f. </strong>In our example, this number is 0.7.<strong> </strong>We shall refer to this number as the <em>“summary scalar” </em>of f. This process of taking a maximum value of the vector <strong>f </strong>is also referred to as <em>“max-pooling”</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/0*ZLr-ze2Rruq604PY" /></figure><p>7. We then repeat all of the above steps with yet another convolutional filter (yet another <strong>H</strong>!). This convolutional filter might have a different width. In our example below, our second <strong>H</strong>, denoted <strong>H’</strong>, has a width of 2. As with the first filter, we slide along <strong>H’ </strong>across the word to get the vector <strong>f </strong>and then perform max-pooling on <strong>f </strong>(i.e. get its summary scalar).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/0*M7mExowu6zCt6R2p" /></figure><p>8. We repeat this scanning process several times with different convolutional filters, with each scanning process resulting in one summary scalar. <strong>Finally, the summary scalars from these different scanning processes are collected to form the character embedding of the word.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/0*-rleIVeJNKYA2bXg" /></figure><p>So that’s it — now we’ve obtained a character-based representation of the word that can complement is word-based representation. That&#39;s the end of this little digression on 1D-CNN; now let&#39;s get back to talking about BiDAF.</p><h3><strong>Step 4. Highway Network</strong></h3><p>At this point, we have obtained two sets of vector representations for our words — one from the GloVe (word) embedding and the other from 1D-CNN (character) embedding. <strong>The next step is to vertically concatenate these representations.</strong></p><p>This concatenation produces two matrices, one for the Context and the other for the Query. Their height is <strong>d</strong>, which is the sum of <strong>d1</strong> and <strong>d2</strong>. Meanwhile, their lengths are still the same as their predecessor matrices (<strong>T </strong>for the Context matrix and <strong>J </strong>for the Query matrix).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xRDdUlAV3qHSvFDYElwN6w.jpeg" /><figcaption>Step 4. The concatenated matrices from the word embedding and the character embedding steps are passed into a highway network</figcaption></figure><p><strong>These matrices are then passed through a so-called highway network. </strong>A highway network is very similar to a feed forward neural network. You guys are probably familiar with feed forward neural network already. To remind you, if we insert an input vector <strong>y </strong>into a single layer of feed forward neural network, three things will happen before the output <strong>z</strong> is produced:</p><ol><li><strong>y </strong>will be multiplied with <strong>W</strong>, the weight matrix of the layer</li><li>A bias, <strong>b</strong>, will be added to <strong>W*y</strong></li><li>A nonlinear function <strong><em>g</em></strong>, such as ReLU or Tanh will be applied to<strong> W*y + b</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/281/1*4zwWTnuLKGE-wpV_nkdNnA.png" /></figure><p><strong>In a highway network, only a fraction of the input will be subjected to the three aforementioned steps; the remaining fraction is permitted to pass through the network untransformed. </strong>The ratio of these fractions is managed by <strong>t</strong>, the <em>transform gate </em>and by (<strong>1-t</strong>),the <em>carry gate. </em>The value of <strong>t </strong>is calculated using a sigmoid function and is always between 0 and 1. Now, our equation becomes as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*dp9fqZTrHEx8ooMMi9YCMw.png" /></figure><p>Upon exiting the network, the transformed fraction of the input is summed with its untransformed fraction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/569/1*foO4g9u3kGY9goYvF-LBsw.png" /></figure><p><strong>The highway network’s role is to adjust the relative contribution from the word embedding and the character embedding steps. </strong>The logic is that if we are dealing with an OOV word such as “misunderestimate”, we would want to increase the relative importance of the word’s 1D-CNN representation because we know that its GloVe representation is likely to be some random gibberish. On the other hand, when we are dealing with a common and unambiguous English word such as “table”, we might want to have more equal contribution from GloVe and 1D-CNN.</p><p>The outputs of the highway network are again two matrices, one for the Context (a <strong>d</strong>-by-<strong>T </strong>matrix)<strong> </strong>and one for the Query (a <strong>d</strong>-by-<strong>J </strong>matrix). They represent the adjusted vector representations of words in the Query and the Context from word and character embedding steps.</p><h4><strong>Step 5. Contextual Embedding</strong></h4><p>It turns out that these representations are still insufficient for our purpose! <strong>The problem is that these word representations don’t take into account the words’ contextual meaning — the meaning derived from the words&#39; surroundings. </strong>When we rely on word and character embedding alone, a pair of homonyms such as the words “tear” (watery excretion from the eyes) and “tear” (rip apart) will be assigned the exact same vector representation although these are actually different words. This might confuse our model and reduce its accuracy.</p><p>Thus, we need an embedding mechanism that can understand a word in its context. This is where the contextual embedding layer comes in. <strong>The contextual embedding layer consists of Long-Short-Term-Memory (LSTM) sequences.</strong> Here is a quick introduction to LSTM:</p><blockquote><strong>An LSTM is a neural network architecture that can memorize long-term dependencies. </strong>When we enter an input sequence (such as a string of text) into a normal forward LSTM layer, the output sequence for each timestep will encode information from that timestep as well as past timesteps. In other words, the output embedding for each word will contain contextual information from words that came before it.</blockquote><p><strong>BiDAF employs a bidirectional-LSTM (bi-LSTM), which is composed of both forward- as well as backward-LSTM sequences.</strong> The combined output embeddings from the forward- and backward-LSTM simultaneously encode information from both past (backwards) and future (forward) states. Put differently, each word representation coming out of this layer now includes include contextual information about the surrounding phrases of the word.</p><p>The output of the contextual embedding step is two matrices — one from the Context and the other from the Query. The BiDAF paper refers to these matrices as <strong>H</strong> and <strong>U</strong>, respectively (terminologies alert — this matrix <strong>H </strong>is distinct from the convolutional matrix <strong>H</strong> mentioned earlier; it is an unfortunate coincidence that the two sources use the same symbol for two different concepts!). The Context matrix <strong>H </strong>is a <strong>d</strong>-by-<strong>T </strong>matrix while the Query matrix <strong>U </strong>is a <strong>d</strong>-by-<strong>J </strong>matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pMxCPR_U8jb_jGskXNS_gQ.jpeg" /><figcaption>Part 5. The contextual embedding step uses bi-LSTM to embed contextual information into the output matrices H and U.</figcaption></figure><p>So that’s all there is to it about the embedding layers in BiDAF! Thanks to the contribution from the three embedding layers, the embedding outputs <strong>H </strong>and <strong>U </strong>carry within them the syntactic, semantic as well as contextual information from all words in the Query and the Context. We will use <strong>H </strong>and <strong>U </strong>in the next step — the attention step — in which we will fuse together the information from them. The attention step, which is the core technical innovation of BiDAF, will be the focus of the <a href="https://towardsdatascience.com/the-definitive-guide-to-bidaf-part-3-attention-92352bbdcb07">next article</a> in the series — do check it out!</p><h4>Glossary</h4><ul><li><strong>Context : </strong>the accompanying text to the Query that contains an answer to that Query</li><li><strong>Query : </strong>the question to which the model is supposed to give an answer</li><li><strong>Answer : </strong>a substring of the Context that contains information that can answer Query. This substring is to be extracted out by the model</li><li><strong>T </strong>: the number of words/tokens in the Context</li><li><strong>J </strong>: the number of words/tokens in the Query</li><li><strong>d1 </strong>: the dimension from the word embedding step (GloVe)</li><li><strong>d2 </strong>: the dimension from the character embedding step</li><li><strong>d </strong>: the dimension of the matrices obtained by vertically concatenating word and character embeddings. <strong>d </strong>is equal to <strong>d1 </strong>+ <strong>d2.</strong></li><li><strong>H </strong>: the Context matrix outputted by the contextual embedding step. <strong>H </strong>has a dimension of <strong>2d</strong>-by-<strong>T</strong></li><li><strong>U </strong>: the Query matrix outputted by the contextual embedding step. <strong>U </strong>has a dimension of <strong>2d</strong>-by-<strong>J</strong></li></ul><h3>References</h3><p>[1] <a href="https://arxiv.org/abs/1611.01603">Bi-Directional Attention Flow for Machine Comprehension (Minjoon Seo <em>et. al</em>, 2017)</a></p><p><a href="https://arxiv.org/abs/1508.06615">[2] Character-Aware Neural Language Models (Yoon Kim <em>et. al</em>, 2015)</a></p><p>All images and diagrams above belong to me unless indicated otherwise.</p><p>If you have any comments about the article or would like to reach out to me, feel free to send me a connection through <a href="https://www.linkedin.com/in/meraldo/">LinkedIn</a>. Also, I’d be very grateful if you could support me by becoming a Medium member through <a href="https://medium.com/@meraldo.antonio/membership">my referral link</a>. As a member, you’ll be able to read all my writings on data science and personal development and have full access to all stories on Medium.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c151fc4f05bb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/the-definitive-guide-to-bidaf-part-2-word-embedding-character-embedding-and-contextual-c151fc4f05bb">Word Embedding, Character Embedding and Contextual Embedding in BiDAF — an Illustrated Guide</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Illustrated Guide to Bi-Directional Attention Flow (BiDAF)]]></title>
            <link>https://medium.com/data-science/the-definitive-guide-to-bi-directional-attention-flow-d0e96e9e666b?source=rss-65b4354e2ed8------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0e96e9e666b</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Meraldo Antonio]]></dc:creator>
            <pubDate>Wed, 28 Aug 2019 09:38:43 GMT</pubDate>
            <atom:updated>2022-04-25T19:32:37.017Z</atom:updated>
            <content:encoded><![CDATA[<h4>The Definitive Guide to BiDAF — part 1 of 4</h4><h4>This article illustrates the workings of BiDAF, an NLP model that has pushed the envelope in the Question and Answer domain.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*95qP_V3IGhweqZDE" /><figcaption>Source: <a href="https://unsplash.com/photos/12ea-y_1-UE">Unsplash</a></figcaption></figure><p>The year 2016 saw the publication of BiDAF by a team at University of Washington. BiDAF handily beat the best Q&amp;A models at that time and for several weeks topped the leaderboard of <a href="https://rajpurkar.github.io/SQuAD-explorer/">the Stanford Question and Answering Dataset (SQuAD)</a>, arguably the most well-known Q&amp;A dataset. Although BiDAF’s performance has since been surpassed, the model remains influential in the Q&amp;A domain. The technical innovation of BiDAF inspired the subsequent development of competing models such as <a href="https://arxiv.org/abs/1802.05365">ELMo</a> and <a href="https://arxiv.org/abs/1810.04805">BERT</a>, by which BiDAF was eventually dethroned.</p><p>When I first read the <a href="https://arxiv.org/abs/1611.01603">original BiDAF paper</a>, I was rather overwhelmed by how seemingly complex it was.</p><blockquote><strong>BiDAF exhibits a modular architecture — think of it as a composite structure made out of lego blocks with the blocks being “standard” NLP elements such as GloVe, CNN, LSTM and attention</strong>. The problem with understanding BiDAF is that there are just so many of these blocks to learn about and the ways they are combined can seem rather “hacky” at times. This complexity, coupled with the rather convoluted notations used in the original paper, serves as a barrier to understanding the model.</blockquote><p>In this article series, I will deconstruct how BiDAF is assembled and describe each component of BiDAF in (hopefully) an easy-to-digest manner. Copious amount of pictures and diagrams will be provided to illustrate how these components fit together.</p><p>Here is the plan:</p><ul><li>Part 1 (this article) will provide an overview of BiDAF.</li><li><a href="https://medium.com/@meraldo.antonio/the-definitive-guide-to-bidaf-part-2-word-embedding-character-embedding-and-contextual-c151fc4f05bb">Part 2</a> will talk about <strong>the embedding layers</strong></li><li><a href="https://medium.com/@meraldo.antonio/the-definitive-guide-to-bidaf-part-3-attention-92352bbdcb07">Part 3</a> will talk about<strong> the attention layers</strong></li><li><a href="https://towardsdatascience.com/modeling-and-output-layers-in-bidaf-an-illustrated-guide-with-minions-f2e101a10d83">Part 4</a> will talk about <strong>the modeling and output layers. </strong>It will also include a recap of the whole BiDAF architecture presented in a very easy language. If you aren’t technically inclined, I recommend you to simply jump to part 4.</li></ul><h3><strong>BiDAF vis-à-vis Other Q&amp;A Models</strong></h3><p>Before delving deeper into BiDAF, let’s first position it within the broader landscape of Q&amp;A models. There are several ways with which a Q&amp;A model can be logically classified. Here are some of them:</p><ul><li><strong>Open-domain </strong><em>vs</em><strong> closed-domain. </strong>An <em>open-domain</em> model has access to a knowledge repository which it will tap on when answering an incoming <em>Query</em>. The famous <a href="https://www.ibm.com/watson">IBM-Watson</a> is one example. On the other hand, a <em>closed-form</em> model doesn’t rely on pre-existing knowledge; rather, such a model requires a <em>Context</em> to answer a Query. A quick note on terminology here — a “Context”<em> </em>is an accompanying text that contains the information needed to answer the Query, while “Query” is just the formal technical word for question.</li><li><strong>Abstractive </strong><em>vs</em><strong> extractive.</strong> An <em>extractive</em> model answers a Query by returning the substring of the Context that is most relevant to the Query. In other words, the answer returned by the model can always be found verbatim within the Context. An <em>abstractive </em>model, on the other hand, goes a step further: it paraphrases this substring to a more human-readable form before returning it as the answer to the Query.</li><li><strong>Ability to answer non-factoid queries. </strong><em>Factoid</em> Queries are questions whose answers are short factual statements. Most Queries that begin with “who”, “where” and “when” are factoid because they expect concise facts as answers. <em>Non-factoid</em> Queries, simply put, are all questions that are not factoids. The non-factoid camp is very broad and includes questions that require logics and reasoning (e.g. most “why” and “how” questions) and those that involve mathematical calculations, ranking, sorting, etc.</li></ul><p>So where does BiDAF fit in within these classification schemes?<strong> BiDAF is a closed-domain, extractive Q&amp;A model that can only answer factoid questions. </strong>These characteristics imply that BiDAF requires a Context to answer a Query. The Answer that BiDAF returns is always a substring of the provided Context.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*bwHzr3hBsXMaTyzA-c0J2g.png" /><figcaption>An example of <strong>Context</strong>, <strong>Query</strong> and <strong>Answer. </strong>Notice how the Answer can be found verbatim in the Context.</figcaption></figure><p>Another quick note: as you may have noticed, I have been capitalizing the words “Context”, “Query” and “Answer”. This is intentional. These terms have both technical and non-technical meaning and the capitalization is my way of indicating that I am using these words in their specialized technical capacities.</p><p>With this knowledge at hand, we’re now ready to explore how BiDAF is structured. Let’s dive in!</p><h3><strong>Overview of BiDAF Structure</strong></h3><blockquote>BiDAF’s ability to pinpoint the location of the Answer within a Context stems from its layered design. Each of these layers can be thought of as a transformation engine that <em>transforms</em> the vector representation of words; each transformation is accompanied by the inclusion of additional information.</blockquote><p>The BiDAF paper describes the model as having 6 layers, but I’d like to think of BiDAF as having 3 parts instead. These 3 parts along with their functions are briefly described below.</p><h4><strong>1. Embedding Layers</strong></h4><p>BiDAF has 3 embedding layers whose function is to <em>change the representation of words</em> in the Query and the Context from strings into vectors of numbers.</p><h4><strong>2. Attention and Modeling Layers</strong></h4><p>These Query and Context representations then enter the attention and modeling layers. These layers use several matrix operations to<em> fuse the information contained in the Query and the Context.</em> The output of these steps is another representation of the Context that contains information from the Query. This output is referred to in the paper as the “Query-aware Context representation.”</p><h4><strong>3. Output Layer</strong></h4><p>The Query-aware Context representation is then passed into the output layer, which will <em>transform it to a bunch of probability values</em>. These probability values will be used to determine where the Answer starts and ends.</p><p>A simplified diagram that depicts the BiDAF architecture is provided below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*28SxjikSqEx92liYeC1yjg.jpeg" /><figcaption>Architecture of BiDAF. Source: author</figcaption></figure><p>If all these don’t make sense yet, don’t worry; in the next articles, I will delve into each BiDAF component in detail. See you in <a href="https://medium.com/@meraldo.antonio/the-definitive-guide-to-bidaf-part-2-word-embedding-character-embedding-and-contextual-c151fc4f05bb">part 2</a>!</p><h3>References</h3><p>[1] <a href="https://arxiv.org/abs/1611.01603">Bi-Directional Attention Flow for Machine Comprehension (Minjoon Seo <em>et. al</em>, 2017)</a></p><p>If you have any comments about the article or would like to reach out to me, feel free to send me a connection through <a href="https://www.linkedin.com/in/meraldo/">LinkedIn</a>. Also, I’d be very grateful if you could support me by becoming a Medium member through <a href="https://medium.com/@meraldo.antonio/membership">my referral link</a>. As a member, you’ll be able to read all my writings on data science and personal development and have full access to all stories on Medium.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0e96e9e666b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/the-definitive-guide-to-bi-directional-attention-flow-d0e96e9e666b">An Illustrated Guide to Bi-Directional Attention Flow (BiDAF)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>