<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Gerry Ongko on Medium]]></title>
        <description><![CDATA[Stories by Gerry Ongko on Medium]]></description>
        <link>https://medium.com/@gerrychrist?source=rss-dd67f049f099------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*bpXENWzQZWYXjcUd10ATdA.jpeg</url>
            <title>Stories by Gerry Ongko on Medium</title>
            <link>https://medium.com/@gerrychrist?source=rss-dd67f049f099------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 06:40:46 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gerrychrist/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Iterated Reweighted Least Squares and GLMs Explained]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/iterated-reweighted-least-squares-and-glms-explained-9c0cc0063526?source=rss-dd67f049f099------2"><img src="https://cdn-images-1.medium.com/max/2600/1*0O7VNtg__LIHgU8QjcR-Rg.jpeg" width="6000"></a></p><p class="medium-feed-snippet">With a detailed implementation in Python</p><p class="medium-feed-link"><a href="https://medium.com/data-science/iterated-reweighted-least-squares-and-glms-explained-9c0cc0063526?source=rss-dd67f049f099------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/iterated-reweighted-least-squares-and-glms-explained-9c0cc0063526?source=rss-dd67f049f099------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c0cc0063526</guid>
            <category><![CDATA[generalized-linear-model]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[probability-distributions]]></category>
            <category><![CDATA[algorithms]]></category>
            <dc:creator><![CDATA[Gerry Ongko]]></dc:creator>
            <pubDate>Tue, 19 Jul 2022 04:02:05 GMT</pubDate>
            <atom:updated>2022-07-20T16:53:06.726Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Implementing Various NLP Text Representation in Python]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/implementing-various-nlp-text-representation-in-python-84781da0ec2d?source=rss-dd67f049f099------2"><img src="https://cdn-images-1.medium.com/max/2600/1*SevZ61QnLIxrvtYNZ4DenQ.jpeg" width="3872"></a></p><p class="medium-feed-snippet">One-Hot Encoding, Bag-of-Words, N-Grams, and TF-IDF</p><p class="medium-feed-link"><a href="https://medium.com/data-science/implementing-various-nlp-text-representation-in-python-84781da0ec2d?source=rss-dd67f049f099------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/implementing-various-nlp-text-representation-in-python-84781da0ec2d?source=rss-dd67f049f099------2</link>
            <guid isPermaLink="false">https://medium.com/p/84781da0ec2d</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[tf-idf]]></category>
            <category><![CDATA[bag-of-words]]></category>
            <category><![CDATA[ngrams]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Gerry Ongko]]></dc:creator>
            <pubDate>Mon, 16 May 2022 03:59:49 GMT</pubDate>
            <atom:updated>2022-05-16T18:09:48.186Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Implementing Various Root-Finding Algorithms in Python]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/implementing-various-root-finding-algorithms-in-python-67917ef090b3?source=rss-dd67f049f099------2"><img src="https://cdn-images-1.medium.com/max/1125/1*EiY4HEL_3n_TPxu_oYR_aA.jpeg" width="1125"></a></p><p class="medium-feed-snippet">With an actual application in data science and logistic regression</p><p class="medium-feed-link"><a href="https://medium.com/data-science/implementing-various-root-finding-algorithms-in-python-67917ef090b3?source=rss-dd67f049f099------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/implementing-various-root-finding-algorithms-in-python-67917ef090b3?source=rss-dd67f049f099------2</link>
            <guid isPermaLink="false">https://medium.com/p/67917ef090b3</guid>
            <category><![CDATA[logistic-regression]]></category>
            <category><![CDATA[newton-raphson-method]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Gerry Ongko]]></dc:creator>
            <pubDate>Mon, 11 Apr 2022 11:36:15 GMT</pubDate>
            <atom:updated>2022-04-12T10:36:36.490Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Building a Machine Learning Web Application Using Streamlit]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/building-a-machine-learning-web-application-using-streamlit-8c3d942f7b35?source=rss-dd67f049f099------2"><img src="https://cdn-images-1.medium.com/max/1646/1*08FLOVoSjKr4wgi6zZAuIA.png" width="1646"></a></p><p class="medium-feed-snippet">An easy tutorial on building Streamlit machine learning applications</p><p class="medium-feed-link"><a href="https://medium.com/data-science/building-a-machine-learning-web-application-using-streamlit-8c3d942f7b35?source=rss-dd67f049f099------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/building-a-machine-learning-web-application-using-streamlit-8c3d942f7b35?source=rss-dd67f049f099------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c3d942f7b35</guid>
            <category><![CDATA[streamlit]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Gerry Ongko]]></dc:creator>
            <pubDate>Fri, 18 Feb 2022 13:37:24 GMT</pubDate>
            <atom:updated>2022-02-18T16:14:52.996Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Building a Machine Learning Web Application Using Flask]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/building-a-machine-learning-web-application-using-flask-29fa9ea11dac?source=rss-dd67f049f099------2"><img src="https://cdn-images-1.medium.com/max/2066/1*o1kMGvK-fN0RY96_VKtZaw.png" width="2066"></a></p><p class="medium-feed-snippet">Simple tutorial on implementing machine learning APIs using Python and Flask</p><p class="medium-feed-link"><a href="https://medium.com/data-science/building-a-machine-learning-web-application-using-flask-29fa9ea11dac?source=rss-dd67f049f099------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/building-a-machine-learning-web-application-using-flask-29fa9ea11dac?source=rss-dd67f049f099------2</link>
            <guid isPermaLink="false">https://medium.com/p/29fa9ea11dac</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[flask]]></category>
            <category><![CDATA[web-development]]></category>
            <dc:creator><![CDATA[Gerry Ongko]]></dc:creator>
            <pubDate>Thu, 03 Feb 2022 15:02:40 GMT</pubDate>
            <atom:updated>2022-02-18T21:34:33.197Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Implementing Expectation-Maximisation Algorithm from Scratch with Python]]></title>
            <link>https://medium.com/data-science/implementing-expectation-maximisation-algorithm-from-scratch-with-python-9ccb2c8521b3?source=rss-dd67f049f099------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ccb2c8521b3</guid>
            <category><![CDATA[expectation-maximization]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[unsupervised-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Gerry Ongko]]></dc:creator>
            <pubDate>Wed, 19 Jan 2022 05:08:19 GMT</pubDate>
            <atom:updated>2024-09-25T03:00:21.285Z</atom:updated>
            <content:encoded><![CDATA[<h4>Demystifying the horrors of the EM Algorithm by building one from scratch</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7LLvnhAm1Ax-CwiIE1PFvg.png" /><figcaption>A mixture model. Created using Tableau.</figcaption></figure><p>The Expectation-Maximisation (EM) Algorithm is a statistical machine learning method to find the maximum likelihood estimates of models with unknown latent variables. I am sure that that sentence will make no sense to some of you. Not to worry! This article aims to demystify the horror equations and puzzling vocabularies of the EM Algorithm. Unfortunately, without proper mathematics, one can not truly understand the essence of the EM algorithm. So this is what I am going to do. Every section of this article will include a <strong>Full Mathematics </strong>section and an <strong>All You Need to Know </strong>section. The <strong>Full Mathematics </strong>section includes complete derivations of the algorithm, while the <strong>All You Need to Know </strong>section summarises essential details without the derivations. To follow the <strong>Full Mathematics </strong>section, I will assume that you are familiar with probability theory, statistical theory, and fundamental calculus. There will still be some maths in the <strong>All You Need to Know </strong>section, but not as complex.</p><p>To understand the algorithm, we must first understand the data we are dealing with. The EM algorithm deals with data from multiple clusters, i.e. a mixture model. Each cluster has a probability of data being from that cluster, but we do not know these probabilities. Each data cluster has its distribution, which we know of, but we do not know the distribution parameters. See the example below for a Poisson mixture model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/852/1*7vEFTEl5n9yaSnT1grp-ww.png" /></figure><p>In this example, there are <em>2</em> clusters. <em>Zᵢ </em>denotes the cluster of the data <em>Xᵢ</em>. The π denotes the probability of being from a certain cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/514/1*c-3FqkLHJrLgUgHEXsDrXg.png" /></figure><p>Hence the sum of all the π values should equal one. For this tutorial, I will assume these parameter values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*3Ftsf90we0IPGZ9jdm3bhQ.png" /></figure><p>If a data point is from the first cluster, it comes from a Poisson distribution with parameter λ = 2 , and if the data point is from the second cluster, it comes from a Poisson distribution with λ = 6. Remember that we do not know these parameter values and are trying to estimate them.</p><p><strong>The purpose of the EM algorithm is to find the parameter estimates of this mixture model</strong>. In the case above, the EM algorithm would estimate <em>π₁, π₂, λ₁, and λ₂</em>. The <em>π</em> is the probability of the cluster, and the <em>λ</em> is the parameter of the Poisson distributions. Consequently, if the distribution of the clusters is Gaussian (Normal) instead of Poisson, the EM algorithm would estimate <em>μ</em> and <em>σ²</em> instead of <em>λ. </em>Let me leave this Poisson pdf function here. Please adjust it to fit your own desired distribution.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/365e5a2d270a23b96aa16def1137ae4c/href">https://medium.com/media/365e5a2d270a23b96aa16def1137ae4c/href</a></iframe><p>In practice, there may be as many clusters as the users want, and the distribution of each cluster can be anything we want it to be. Once again, I will remind you that we do not know any of the parameter values, nor do we know from which cluster each data point is from. This unknown cluster allocation is why the clusters are called latent variables. The EM algorithm is very useful for dealing with unobservable variables like this.</p><p>Since the EM algorithm is iterative, I will use θ to denote the new parameter estimates and θ⁰ to indicate the previous iteration estimates. Some other variables; N denotes the total number of samples, and K represents the total number of clusters.</p><p>You should skip to section <strong>4. Algorithm</strong> to go right to the demystifying part or<strong> </strong>if you are looking for the non-maths summary of the algorithm.</p><h3>1. Expectation of Complete Log-Likelihood</h3><p>The complete log-likelihood is equivalent to the log-likelihood of the data. We used the term ‘complete’ because, in later sections, there will be another statistic called the ‘<em>incomplete log-likelihood’</em>. I will use ‘<em>complete log-likelihood</em>’ and ‘<em>log-likelihood</em>’ interchangeably <strong>for this section only</strong>.</p><h4><strong>Full Mathematics</strong></h4><p>Skip to the <strong>All You Need to Know </strong>section if you are not interested in the derivations.</p><p>To find the distribution parameters of some data, we usually take the parameters that maximise the likelihood of that data. The EM Algorithm applies the same concept here, trying to maximise the complete likelihood.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G-DY-_GSTx10iQzzcAR19A.png" /><figcaption>Complete Likelihood</figcaption></figure><p>We can expand the formula above as below,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YNmYypIXPFvXcfeBOZ2Alg.png" /><figcaption>Complete Likelihood</figcaption></figure><p>The function on the exponent is an indicator function, returning one if the condition inside is true and 0 otherwise. To simplify later calculations, we prefer this indicator function to not be in the exponent. Let’s take the log of the likelihood.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8dPUrWM0a56Yyok7v8Y95Q.png" /><figcaption>Complete Log-Likelihood</figcaption></figure><p>This is a more straightforward form to maximise. Unfortunately, we cannot calculate this log-likelihood because we do not know which clusters each data comes from. Therefore, we do not know the output of the indicator function and distribution parameters to use. A solution to this problem is to use the expectation of the log-likelihood instead of the actual log-likelihood. Let’s define the expectation of the log-likelihood as below,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Oq3bHXp0JG9drplbi40gaw.png" /><figcaption>Expectations of Log-Likelihood</figcaption></figure><p>The expectation of the log-likelihood replaces the indicator function with the posterior probabilities. We will calculate the values of the posterior probabilities in later sections.</p><p>For our Poisson example above, the expectation becomes the following,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y4y_EVcTIsbiWQ-yY1LyKw.png" /><figcaption>Poisson Expectations of Log-Likelihood</figcaption></figure><h4>All You Need to Know</h4><p>All you need to know about the complete log-likelihood is that we want to find parameters that maximise it. Unfortunately, the complete log-likelihood is difficult to calculate because of the unknown clusters. To get around this, we calculate the expectation of the log-likelihood and maximise it instead of the actual log-likelihood.</p><p>The expectation of the complete log-likelihood is denoted as <em>Q(θ, θ⁰)</em>. The formula for <em>Q(θ, θ⁰) </em>is given below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TsSU_0_he_3wnZPJiF4MXQ.png" /><figcaption>The expectation of Complete Log-Likelihood</figcaption></figure><p>For our Poisson example, this becomes,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y4y_EVcTIsbiWQ-yY1LyKw.png" /><figcaption>Poisson Expectations of Log-Likelihood</figcaption></figure><p>For now, this is all you need to know about the <strong>expectation of the complete log-likelihood</strong>.</p><h3>2. Posterior Probability</h3><h4>Full Mathematics</h4><p>Skip to the <strong>All You Need to Know </strong>section if you are not interested in the derivations.</p><p>The posterior probability is the probability of obtaining a particular data point from a cluster given the value of that data point and the parameters in the current iteration. This is not to be confused with the probability denoted by π. Let’s define them,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Rv12mj5qPaNiV9Y4CoKt6g.png" /><figcaption>Posterior definition</figcaption></figure><p>It would help if you realised that I used the conditional probability definition to obtain this. We can expand this expression to the following,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kV35LVTnDELSP9eOjCJWVw.png" /></figure><p>I will not go into the probabilistic properties that I applied here, but it is the exact conditional probability definition above. If you notice, the numerator is the likelihood of data and cluster. We have seen these probabilities in the previous section. The denominator is just the sum of all clusters’ likelihood.</p><p>For our Poisson example, the posterior probability becomes,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FTMOyku_HM358MOVkAaqMg.png" /><figcaption>Poisson Posterior Formula</figcaption></figure><p>Note that the parameters used to calculate the posterior probabilities are the previous iteration’s parameters. Therefore, the posterior probability will be considered a constant in the expectation of complete log-likelihood.</p><h4>All You Need to Know</h4><p>All you need to know about the posterior probability is that it is the probability of obtaining a cluster given the current data and parameters. We need this probability to calculate the expectation of complete log-likelihood. Thankfully, It is not too difficult to find, so I will just give out the formula.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_7p8YzoNuzdd9dnrwdcGHw.png" /><figcaption>Posterior Probability</figcaption></figure><p>This formula is simple because we already know everything. The numerator is the likelihood of a data and cluster, while the denominator is the sum of likelihoods of the same data and all possible clusters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FTMOyku_HM358MOVkAaqMg.png" /><figcaption>Poisson Posterior Formula</figcaption></figure><p>In our previous example, we only have two possible clusters. Hence the posterior probability for the first cluster of that example is just the likelihood of the first cluster divided by the sum of the first cluster’s likelihood and the second cluster’s likelihood.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nEJtOg1Yhs2F3E1QWpKq5g.png" /><figcaption>Poisson Posterior Formula for k = 1</figcaption></figure><p>I hope that you can extrapolate this example to other examples.</p><h4>Python Code</h4><p>Using the Poisson mixture example, below is a function to calculate the posterior probability.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/74da36b26469df4dcc23230f9b92831f/href">https://medium.com/media/74da36b26469df4dcc23230f9b92831f/href</a></iframe><p>The function above returns a list of lists, where each inner list denotes a cluster, and the content of the inner list is the posterior probabilities. Try to match this Python code with the<em> Poisson Posterior Formula</em> image above.</p><h3>3. Maximisation</h3><h4>Full Mathematics</h4><p>Skip to the <strong>All You Need to Know </strong>section if you are not interested in the derivations.</p><p>Maximisation refers to obtaining parameters by maximising the expectation of complete log-likelihood. We do this by finding the derivative of the expectation and setting it to zero. Let’s go back to the formula.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y4y_EVcTIsbiWQ-yY1LyKw.png" /><figcaption>Poisson Expectations of Log-Likelihood</figcaption></figure><p>I will use this notation to denote the posterior probability to simplify the equations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/840/1*OByH1Sfgqm4iGMY2_8YRog.png" /></figure><p>We will start with finding the optimal π₁. Because we only have 2 clusters, we can substitute π₂ using 1 - π₁. Now let’s remove all the terms in <em>Q(θ, θ⁰) </em>that do not contain π₁.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*ySi17rN_NxkYy_Bof8q8RA.png" /></figure><p>We are allowed to do this because we are about to derive this equation with respect to π₁. Meaning that all the other terms that do not contain π₁ will become 0 anyway, so let’s remove them so that the equation will become easier to the eye. The next step is to expand the inner sigma.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/1*MbfYvBXRz9Tkthj6wCkbpw.png" /></figure><p>At this point, taking the derivatives is relatively simple.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*97hO3XkbW6vCk-etEOrHEg.png" /></figure><p>Setting the derivative to 0 will yield the following results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/564/1*wu6HeljhaQe-Clb-tN4lAA.png" /></figure><p>With some simple algebra, solving for π₁ will get us,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*GFb2XK9DzsYjAR_dEl8t-w.png" /><figcaption>Optimum Formulas</figcaption></figure><p>The interpretation of this result is that, with the current obtained posterior probability, we can update π₁ to whatever the result of this equation is.</p><p>I will not repeat the process for λ because it will be too long. If you are interested, see my calculations <a href="https://drive.google.com/file/d/1a5mxg1RlGpyTMwsZGa6hn_DCHeTxO1xR/view?usp=sharing">here</a>. Alternatively, visit the <strong>All You Need to Know </strong>section for the final estimates only.</p><h4>All You Need to Know</h4><p>All you need to know about the maximisation process is that to obtain the parameter estimates, we need to maximise the complete log-likelihood expectation. Let’s go back to the formula.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y4y_EVcTIsbiWQ-yY1LyKw.png" /><figcaption>Poisson Expectations of Log-Likelihood</figcaption></figure><p>The way to maximise this is to take the derivatives of <em>Q(θ, θ⁰) </em>with respect to each parameter and set it to 0. Remember that the posterior probability is constant when taking derivatives because it is calculated using the previous iteration’s estimates.</p><p>I will show you the result of the derivation for the Poisson example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gmF8MRsgHLgEAyCyMBIvnQ.png" /><figcaption>Optimum Formulas</figcaption></figure><p>Hence to obtain the new parameter estimates of every iteration, we need to calculate these equations.</p><h4>Python Code</h4><p>Using the Poisson mixture example, below are functions to calculate the optimum parameter estimates.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/49b9b7c5d30b9bc7af47bbc6028cc4cb/href">https://medium.com/media/49b9b7c5d30b9bc7af47bbc6028cc4cb/href</a></iframe><p>The functions above return two lists for the π and λ estimates, respectively. Try to match this Python code with the<em> Optimum Formulas</em> image above.</p><h3>4. Algorithm</h3><p>So far, we have not gotten much to the demystifying part of the article. I understand that everything might still be a blur. So let’s wrap up everything we know and put the algorithm to live.</p><p>Steps of an EM Algorithm:</p><ol><li>Initialise random parameter values.</li><li>Derive the expectation of complete log-likelihood, <em>Q(θ, θ⁰).</em></li><li>Calculate the posterior probabilities.</li><li>Given the posterior probability, find optimal parameters by differentiating <em>Q(θ, θ⁰) </em>w.r.t each parameter, set the derivatives to 0, and solve for the parameter.</li><li>Change current parameters to the one obtained in step 4.</li></ol><p>Notice that after changing the parameter estimates in step 5, recalculating the posterior probabilities would yield different results (See <em>Poisson Posterior Formula </em>image). But if the posterior probabilities are now different, then surely redoing step 4 would generate different parameter estimates (See <em>Optimum Formulas </em>image). And that is the essence of the EM algorithm. Let’s repeat that in English. The EM algorithm is,</p><ol><li>Using current parameters, calculate posterior probability.</li><li>Using current posterior probability, update the parameters.</li></ol><p>And we iterate these steps over and over again. That’s not so mystical, isn’t it? Sure the maths might still be heavy, but the essence of the algorithm is just these two little steps here. Let’s try to translate this to code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/53ec81c2a2d37dbecbb32c38705422cf/href">https://medium.com/media/53ec81c2a2d37dbecbb32c38705422cf/href</a></iframe><p>Now without needing to understand the contents of each function, this python code should be able to highlight the core of the EM algorithm. The heart of it is mathematical and complicated, but the algorithm is quite simple.</p><p>The only unsolved piece of the puzzle left is the stopping criteria. Now that we understand what is done in each iteration, we want to know when to stop the iteration. The following section will explain this stopping criterion.</p><h3>5. Incomplete Log-Likelihood</h3><p>In this section, I will introduce the concept of incomplete log-likelihood. The idea of the incomplete log-likelihood is similar to the complete log-likelihood. I will not go through the mathematical details, but essentially, the incomplete log-likelihood does not care about the data cluster. It calculates the likelihood from all clusters and sums them up. Whereas the complete log-likelihood only takes the likelihood of the cluster from which the data originates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DMvk3pmOtLL4SpcK1z_7mw.png" /><figcaption>Incomplete log-likelihood</figcaption></figure><p>I am going to translate this formula into Python.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fbde03ced9e8ce111809ca8409e52be3/href">https://medium.com/media/fbde03ced9e8ce111809ca8409e52be3/href</a></iframe><p>The function above returns the value of the incomplete log-likelihood.</p><p>Due to some properties I will not explore, the incomplete log-likelihood will always increase on every iteration.<strong> The EM algorithm iteration stops when the change in incomplete log-likelihood is no longer significant</strong><em>.</em></p><h3>6. Summary</h3><p>Let’s put everything in code.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/603528a3ee5d6f29584789ba7eac130e/href">https://medium.com/media/603528a3ee5d6f29584789ba7eac130e/href</a></iframe><p>The output of this function is the final parameter estimate.</p><p>Now you might be wondering why I said that the EM algorithm is a machine learning method, whereas it is clearly an estimation method. We have been calculating the posterior probabilities along the way, and we can use this to obtain the clusters of each data point. We do this by calculating the posterior probability of a data point belonging to each cluster and assigning the class to the cluster with the highest probability. Hence an unsupervised machine learning method.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b4401119424e0c2f442791a5ce617564/href">https://medium.com/media/b4401119424e0c2f442791a5ce617564/href</a></iframe><p>The above parameters lambdas and probs refers to the final estimate from the EM algorithm. Running this function would return a NumPy array of the clusters of the data X.</p><h3>7. Congratulations</h3><p>You have successfully finished this article. If you managed to understand and recreate the algorithm, I would applaud you. But if not, this is an advanced algorithm, and you’ll get it right with more practice. See my complete code on <a href="https://github.com/gerchristko/Expectation-Maximisation">GitHub</a> to see how to use our functions for practical usage. If you enjoy this article, consider giving a clap and a follow. Please do share any feedback and thoughts in the comments. Thank you for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ccb2c8521b3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/implementing-expectation-maximisation-algorithm-from-scratch-with-python-9ccb2c8521b3">Implementing Expectation-Maximisation Algorithm from Scratch with Python</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implementing Naïve Bayes Classification from Scratch with Python]]></title>
            <link>https://blog.devgenius.io/implementing-na%C3%AFve-bayes-classification-from-scratch-with-python-badd5a9be9c3?source=rss-dd67f049f099------2</link>
            <guid isPermaLink="false">https://medium.com/p/badd5a9be9c3</guid>
            <category><![CDATA[naive-bayes]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Gerry Ongko]]></dc:creator>
            <pubDate>Mon, 03 Jan 2022 03:42:32 GMT</pubDate>
            <atom:updated>2022-03-31T10:06:21.807Z</atom:updated>
            <content:encoded><![CDATA[<h4>Understanding Naïve Bayes classification by building one from scratch</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*XvnRrwFcyF9g7VTLunYeig.png" /><figcaption>The image is taken from Scienceprog.com</figcaption></figure><h3>Naïve Bayes</h3><p>In machine learning and data science, Naïve Bayes is a nice simple yet widely used algorithm. Unfortunately, due to the many simplifications life has given us, many in this profession tend to underestimate the need for elaborate algorithm understanding. This article aims to teach or refresh your knowledge of the Naïve Bayes classifier by building one from scratch in Python, using only <em>Pandas</em> and <em>Numpy</em>. It will benefit your understanding if you are familiar with probability theory, as I will be talking about some probabilistic properties.</p><p>Naïve Bayes itself is a probability-based classifier algorithm. The foundation of this model is Bayes’ theorem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6BsgZiGoibNqEFvQFlz9zw.png" /><figcaption>Bayes’ Theorem</figcaption></figure><p>In simple words, the Naïve Bayes classifier classifies an instance by calculating the posterior of each class, given the instance; <em>P(C ∣ x)</em>, and assigning the prediction to the class with the largest posterior. In practice, the posterior probability is quite tricky to calculate. Some strong assumptions need to be made to get around this obstacle, hence the name; <strong>Naïve</strong> Bayes. We will explore this obstacle and any mathematical assumptions made further down below. Let’s begin.</p><h3>Data Preprocessing</h3><p>Let’s start with our libraries.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e22d2f8a7ead7bfd2d05faacf3fbc92e/href">https://medium.com/media/e22d2f8a7ead7bfd2d05faacf3fbc92e/href</a></iframe><p>As promised, we will only use <em>Pandas</em> and <em>Numpy </em>for this tutorial.</p><p>Next, we are going to load the data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1f8fdce4f8d2f51e612dae5664f35fb0/href">https://medium.com/media/1f8fdce4f8d2f51e612dae5664f35fb0/href</a></iframe><p>I am using the famous iris dataset, downloaded from <a href="https://www.kaggle.com/uciml/iris">Kaggle</a>. You could also follow this tutorial using your data, with the following conditions; numerical data as predictors and categorical data as predicted.</p><p>Like all machine learning developments, we will need to split the data into training and testing sets. This is conventionally done using the <em>Sklearn </em>library. I will show a neat trick to achieve train-test-split using <em>Pandas</em>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2aeb4160f723831a23f64340c189ebf7/href">https://medium.com/media/2aeb4160f723831a23f64340c189ebf7/href</a></iframe><h3>Model Training</h3><p>Training a Naïve Bayes model is very simple. But before we get there, we have to establish the distribution on which we base the probabilities. Recall Bayes’ Theorem,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6BsgZiGoibNqEFvQFlz9zw.png" /></figure><p>To find the posterior probabilities <em>P(C ∣ x)</em>, we need to calculate the likelihood <em>P(x ∣ C) </em>and the prior<em> P(C)</em>.</p><p>We can obtain this likelihood quite quickly if we know the distribution of <em>x</em>. The distribution of x is, unfortunately, something that the data scientist must figure out themselves. You can use any mixture of distributions for different classes, but note that this selection impacts the model performance significantly. This tutorial will assume a gaussian (normal) distribution for all classes, as the gaussian distribution is a good approximation for many other distributions.</p><p>The gaussian distribution takes two parameters, <em>μ</em> and <em>σ²</em>. We do not know the actual values of these parameters, so we have to estimate them from the current samples.</p><p>We also need to find the prior probability <em>P(C)</em>. We will estimate the prior probability of a class <em>C </em>by counting all occurrences of class <em>C</em> and dividing it by the total number of samples.</p><p>There is also a <em>P(x) in the formula</em>, which do not need to be calculated. We will get to the reason for this later on.</p><p>All calculations are done only on the training data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fc6992bb479b3e0c1a83c32f0d88487e/href">https://medium.com/media/fc6992bb479b3e0c1a83c32f0d88487e/href</a></iframe><p>We now have everything we need to calculate the posterior. This concludes the training process.</p><h3>Classification</h3><p>The classification process in a Naïve Bayes classifier is done by calculating the posterior for all classes given the current data. I will go into detail on how the posterior is calculated. We will, of course, start with Bayes’ Theorem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PKD2BDj12LYQra4aaQFxvg.png" /></figure><p>In line 2, the <em>P(x) </em>is removed. This is because the probability of observing <em>x </em>is the same across all classes. This will not affect the calculation because we do not need to find the actual posterior probability of the classes. Instead, we only need to see which class has the largest posterior probability. Dividing all the posterior probabilities by the same constant would return the same results on whose value is largest.</p><p>In line 3, <em>P(x ∣ C)</em> is expanded to include each column. The initial <em>P(x ∣ C)</em> is the probability of observing <em>x </em>(the entire row), given the current class. The expanded version calculates the probability of each column of the given row and class (individual data).</p><p>If you are familiar with your probabilities, you might notice something entirely fallacious about the expansion on line 3. Here is the breakdown of it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/1*vO2dQ4LWdQOvVJ7eEnSHsA.png" /></figure><p>From this point, the mathematicians and the statisticians would start to go wild and say,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/1*x4Deq6FRagcou4FVEgnHRg.jpeg" /><figcaption>Generated from <a href="https://imgflip.com/">https://imgflip.com/</a></figcaption></figure><p>and I would say I naïvely did.</p><p>This is one of the essential assumptions of the Naïve Bayes model, which assumes all features are independent of each other. In practice, this is rarely the case. This is why they named the model <strong>Naïve</strong> Bayes.</p><p>Back to classification, some people like to calculate the log-posterior instead of the posterior itself. This will not change any predictions as <em>log(x)</em> is a strictly increasing function. This means that If we have <em>a &gt; b</em>, then we will have <em>log(a) &gt; log(b)</em>. Hence the order of posterior probability by size will not change. I will use the log transformation in my code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8-DZUh-6xdqD-P9zXAJblA.png" /></figure><p>The classification process of this model is done by simply calculating the log-posterior and observing which class has the largest. The code below shows the process of classification.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8dfb36079d0c2239ab5a9120371f98f6/href">https://medium.com/media/8dfb36079d0c2239ab5a9120371f98f6/href</a></iframe><p>In this code, the log-prior, <em>log P(Cᵢ), </em>is calculated in line 20, and the log-likelihoods, <em>log P(xₖ ∣ Cᵢ), </em>are calculated in line 29. The log-posterior of the class given the data, <em>log P(Cᵢ ∣ x)</em>, is calculated by summing the log-prior and the log-likelihoods in line 38. To assign the prediction, lines 41 and 42 search for the class with the largest log-posterior and record it. The variable Predictions store all predictions made by the model.</p><p>To assess the model’s performance, I will test the model on both training and testing sets.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0c67003ae5e42af3bcc901ddc86b4136/href">https://medium.com/media/0c67003ae5e42af3bcc901ddc86b4136/href</a></iframe><p>The prediction for the training set is stored in PredictTrain and for the testing set, in PredictTest. This concludes the classification process.</p><h3>Performance</h3><p>There are many ways to assess the performance of a multi-class classifier, but in this article, I will only calculate the accuracy.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d607ad7c07e1bda54986e1db6a15ce64/href">https://medium.com/media/d607ad7c07e1bda54986e1db6a15ce64/href</a></iframe><p>If you are using the same data and have followed everything correctly, you should obtain the following results.</p><p>For the training set,</p><p>round(Accuracy(y_train, PredictTrain), 5)</p><p>Output:</p><pre>0.98095</pre><p>For the testing set,</p><p>round(Accuracy(y_test, PredictTest), 5)</p><p>Output:</p><pre>0.91111</pre><p>Overall, I would say that this is a pretty good fit for the data. You could further improve the performance by parameters tuning, i.e. using different distributions or applying smoothing methods.</p><h3>Comparison</h3><p>Let’s compare the model we just created to the typical way of doing it using <em>Sklearn</em>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/765ee17b211156a6b94c0adbca09c410/href">https://medium.com/media/765ee17b211156a6b94c0adbca09c410/href</a></iframe><p>Comparing our training set to <em>Sklearn’s</em> training set,</p><p>round(Accuracy(PredictTrain, SkTrain), 5)</p><p>Output:</p><pre>1</pre><p>Comparing our testing set to <em>Sklearn’s</em> testing set,</p><p>round(Accuracy(PredictTest, SkTest), 5)</p><p>Output:</p><pre>1</pre><p>This shows that our model made identical predictions to <em>Sklearn’s</em> Gaussian Naïve Bayes library.</p><h3>Congratulations</h3><p>We successfully constructed a Naïve Bayes classifier from scratch using <em>Pandas </em>and<em> Numpy</em>. Consider redoing the tutorial using your data for better understanding. You can find the code and dataset on <a href="https://github.com/gerchristko/Gaussian-Naive-Bayes">GitHub</a>. If you enjoy this article, consider giving a clap and a follow. Please do share any feedback and thoughts in the comments. Thank you for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=badd5a9be9c3" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/implementing-na%C3%AFve-bayes-classification-from-scratch-with-python-badd5a9be9c3">Implementing Naïve Bayes Classification from Scratch with Python</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Implementing Vector Autoregression (VAR) from Scratch with Python]]></title>
            <link>https://blog.devgenius.io/implementing-vector-autoregression-from-scratch-with-python-b12eedbf35ad?source=rss-dd67f049f099------2</link>
            <guid isPermaLink="false">https://medium.com/p/b12eedbf35ad</guid>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[vector-autoregression]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <dc:creator><![CDATA[Gerry Ongko]]></dc:creator>
            <pubDate>Fri, 31 Dec 2021 15:12:17 GMT</pubDate>
            <atom:updated>2022-02-09T08:42:30.183Z</atom:updated>
            <content:encoded><![CDATA[<h4>Understanding VAR by building one from scratch</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-HQDjvH1nCwfR5seQ2aReA.jpeg" /><figcaption>Multivariate Time Series</figcaption></figure><h3>VAR</h3><p>When talking about time series analysis and forecasting, there are many models and algorithms to choose from. One of the base models of the time series domain is the <em>vector autoregression </em>(VAR), which is the simplest version of most multivariate time series models that we know of. This article aims to explain the core of the VAR by coding it in Python. I will assume that you are familiar with obtaining the estimated parameters of linear regression, mainly the normal equation. I will also assume that you are familiar with time series features (i.e., lags in time series).</p><p>To start, an <em>N</em>-variate VAR model is just <em>N </em>linear regression models stacked into 1. These <em>N </em>linear regressions use the same design matrix (simplified: same predictor data) and use it on <em>N </em>different time series. The predictor data is those <em>N </em>time series on previous time points or lags. Hence the estimated parameters of a VAR model is just the estimated parameters of all those linear regressions. You should have no trouble understanding this content if you know how linear regression works.</p><h3>Preprocessing</h3><p>Let’s start with our libraries.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e22d2f8a7ead7bfd2d05faacf3fbc92e/href">https://medium.com/media/e22d2f8a7ead7bfd2d05faacf3fbc92e/href</a></iframe><p>And that’s it. We want to create a VAR model without external libraries.</p><p>Next, we load the data. The data I am using is electricity prices and demands in Victoria, Australia. It is openly available on the <a href="https://aemo.com.au/energy-systems/electricity/national-electricity-market-nem/data-nem/aggregated-data">AEMO website</a>. For simplicity, I have changed the column names so that it is easier for users to read. You can download my version of the data <a href="https://drive.google.com/file/d/1CLfYy4KWbGsMnxPwXtMJdBT4hW1yvI0q/view?usp=sharing">here</a>, though you can use any data of your own for this tutorial. Below is a snippet of my version of the data. Please set any time indicator in the index for this tutorial only or remove them.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2bf7f9c80429d18670b0d47528d74e01/href">https://medium.com/media/2bf7f9c80429d18670b0d47528d74e01/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xYetF_gs2cIiqEfHlMf8jA.png" /><figcaption>First data frame output</figcaption></figure><p>We will now get a little technical. There is something called an optimal lag structure in every time series model. Let me give you an example to illustrate this. Let’s say we are predicting a daily time series data and we want to predict tomorrow’s value. To predict tomorrow, I used today’s value, yesterday’s value and the value of the day before yesterday as predictors. This model has a lag structure of 3, seeing that we used 3 past data to predict the next value.</p><p>Each model has an optimal lag structure where it maximises (or minimises) a statistic. Generally, the choice of statistic is between information criterions; AIC (Akaike), SIC (Schwarz), or HQIC (Hannan-Quinn), but there are also other methods to determine the optimal structure. When using the three previous criterion, the best practice is to calculate all of them and choose a ‘majority rules’ lag, regardless of which criterions are conforming. In the case of a tie, arbitrary selection will not impact the model significantly. For those mathematically interested, I provided the formula of the criterions I mentioned.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rYex73Z_z9F0hxJUBY-mxw.png" /><figcaption>Information criterion formulas</figcaption></figure><p>Where <em>L</em>(θ) is the likelihood of the <em>K</em>-variate VAR model, <em>N</em> is the sample size, and <em>p</em> is the number of estimated parameters. This process might seem complicated, but Python has libraries for this specific task, namely <em>statsmodel</em>. If you don’t understand much about the lag structure, don’t worry. For the sake of this article, I will choose an arbitrary lag of 100.</p><p>To put the lags as features, we will loop through each lag of each feature. The method df[col].shift(i) is a convenient way to obtain the <em>i</em>-th lag of any feature. If you are observant, you might notice that this maneuver would produce NaN values in the first 100 rows. We will not worry about this and simply drop these first 100 rows. Note that each time I mention 100, this refers to your optimal lag structure.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/18ccd81287b852d58425df338402b45f/href">https://medium.com/media/18ccd81287b852d58425df338402b45f/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uxpa0yKpsePhllWt5uoeyg.png" /><figcaption>A snippet of the resulting data frame. You could also use this image to understand what lags are.</figcaption></figure><p>The next step is to take the original lag zero data (in this case, columns Price and Demand), store them somewhere else and remove them from the data frame. This is because these original data are predicted variables, and it makes no sense to train the model with them in the data.</p><p>We should also add an intercept column on the first entry of the data frame. In statistical terms, the intercept is there to adjust for the bias in predictions. But all you need to know is that in every regression model, we need an intercept and hence an intercept here. Putting the intercept in the first column is just conventional.</p><p>The final step is to transform the data into matrices.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cdcafbcf8b485b1070aad0e551721191/href">https://medium.com/media/cdcafbcf8b485b1070aad0e551721191/href</a></iframe><h3>Model Training</h3><p>This is where we get to the regression part. If you recall, I mentioned that an <em>N</em>-variate VAR model is simply <em>N</em> linear regressions simultaneously. In our case, the time series is bivariate (or a 2-variate); hence the VAR model is just two linear regressions.</p><p>To solve a linear regression is to obtain the parameter estimates for each variable, usually denoted as the <em>β</em>-matrix. If we have our predictor and predicted matrix prepared, finding the <em>β</em>-matrix will be simple. In our code, the predictor matrix is the variable X and the predicted matrix is y_price and y_demand.</p><p>To obtain the estimated <em>β</em>-matrix, we need to do a series of matrix multiplications. Mainly, the normal equation,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/482/1*hHZtRjHnUuQmblYv75BqDA.png" /><figcaption>The normal equation</figcaption></figure><p>Here, <em>X </em>refers to the predictor matrix X and <em>y </em>refers to the predicted matrix y_price and y_demand. Matrix multiplication in Python is made easy with <em>Numpy</em>. I will give out some commands for matrix manipulation in Python that will be useful for our purposes,</p><ul><li>Let A and B be <em>Numpy</em> matrices.</li><li>np.matmul(A, B) is used to multiply the matrix A and matrix B. Note that np.matmul(B, A) will yield different results due to the nature of matrix multiplications.</li><li>A.T is used to obtain the transpose of the matrix A.</li><li>np.linalg.inv(A) is used to obtain the inverse of the matrix A .</li></ul><p>Let’s create a function to obtain parameter estimates from the normal equations.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cd6db7f4c7ebd260a502e5078e55a8bc/href">https://medium.com/media/cd6db7f4c7ebd260a502e5078e55a8bc/href</a></iframe><p>As simple as that, we have obtained the regression parameters (and the VAR parameters) in just these few lines of code. This concludes the training of the model. The parameter estimates, which are stored in b_price and b_demand. We can now use it to make forecasts.</p><h3>Forecasting</h3><p>Let’s proceed to the forecasting process. In time series forecasting, there are two types of forecast.</p><ul><li><strong>Ex Ante</strong>: Forecasting presently available data. This means testing the model on the training data. Useful for model diagnostics.</li><li><strong>Ex Post</strong>: Forecasting using future data. Future data means forecasted data. In an ex post forecast, the forecasted data at any time <em>t </em>becomes a predictor at time <em>t + n</em>. Useful for practical uses.</li></ul><h4>Ex Ante</h4><p>Now let’s review the maths for forecasting from a regression. If you already have the parameter estimates for the regression, you can find the forecast by calculating the dot product of the parameter estimates and the data. Here’s a simple explanation,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/956/1*8vaxDaYiYxnzu8IfehQouw.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4a9a283409d8bac1a2bcf7ae9507c1c9/href">https://medium.com/media/4a9a283409d8bac1a2bcf7ae9507c1c9/href</a></iframe><p>We can actually simplify this process to just two lines of matrix multiplication price_estimates = np.matmul(X, b_price) and demand_estimates = np.matmul(X, b_demand). But I decided to break it down anyway for better understanding.</p><p>This concludes the ex ante forecasting process. Ex ante forecast is helpful for model diagnostics, and I will demonstrate how to calculate some of them. To assess the performance of a VAR model, we will use statistics that are typically used to evaluate linear regression models.</p><ul><li>R²: Measures how much variance is explained by the data</li><li>MSE: Measures level of error</li><li>F-test: Measures significance of predictors.</li></ul><p>I will not detail the interpretation of these statistics, but I will provide the formula for those who are mathematically interested.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UQCsiHEraHdoMgz_n-EZvg.png" /></figure><p>Let’s do some diagnostics. I am going to copy the above formulas as Python functions.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c3d71b452feb5fa825c87c496b673269/href">https://medium.com/media/c3d71b452feb5fa825c87c496b673269/href</a></iframe><p>Running the Diagnostics() function would return these outputs for our data.</p><p>For price,</p><p>Diagnostics(y_price, X, b_price, N, p)</p><p>Output:</p><pre>The R-squared is: 0.66<br>The MSE is: 28262.74<br>The F-statistic is: 686.42</pre><p>For demand,</p><p>Diagnostics(y_demand, X, b_demand, N, p)</p><p>Output:</p><pre>The R-squared is: 1.0<br>The MSE is: 3318.01<br>The F-statistic is: 2517997.46</pre><p>You could further explore the diagnostics by calculating the adjusted R² or a p-value for the F statistic. There are many more diagnostic methods available for regression.</p><h4>Ex Post</h4><p>Ex post forecast is forecasting the future. The gist is that if <em>t </em>represents the present time point, then to predict a time point <em>t + n</em>, where n is a number larger than 1, we would need to use the lags at <em>t + n </em>as predictors. At least one of the lags at <em>t + n </em>is after <em>t </em>if <em>n</em> is larger than 1. In other words, some of the lags of <em>t + n </em>are future forecasted time points. Observe the table below to visualise.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/45108802c007188a2efd2365445be59f/href">https://medium.com/media/45108802c007188a2efd2365445be59f/href</a></iframe><p>Here, any entries with an asterisk (*) represent data we do not have at time <em>t </em>and hence forecasted. For a large <em>n</em>, we will have all forecasted values as predictors. The Python code below shows ex post forecasting. The variable PointsAhead represents how far ahead you would like to forecast, you can change it to fit your requirements.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b44f42e9a0af73457f817d4ea87a77b0/href">https://medium.com/media/b44f42e9a0af73457f817d4ea87a77b0/href</a></iframe><p>The final forecast will be stored in ExPostPrice and ExPostDemand. This concludes the ex post forecasting process. Because we are practically predicting the future, we can’t run diagnostics on our prediction. A standard solution to run diagnostics on ex post forecasts is to train the model using data only up to a specific time <em>t - n</em>, and then conduct ex post forecast from time <em>t - n</em> up to time <em>t</em>. You can then run the diagnostics on the testing set because, this time, you have the actual values of the testing set.</p><h3>Congratulations</h3><p>We have successfully constructed a VAR model from start to finish using only <em>Pandas</em> and <em>Numpy</em>. I hope you can recreate the model with your data, preferably with a 3-variate time series just for complete generalised understanding. You can also find the code and dataset on <a href="https://github.com/gerchristko/Vector-Autoregression">GitHub</a>. If you enjoy this article, consider giving a clap. Please do share any feedback and thoughts in the comments. Thank you for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b12eedbf35ad" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/implementing-vector-autoregression-from-scratch-with-python-b12eedbf35ad">Implementing Vector Autoregression (VAR) from Scratch with Python</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>