<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Hamed Zakerzadeh on Medium]]></title>
        <description><![CDATA[Stories by Hamed Zakerzadeh on Medium]]></description>
        <link>https://medium.com/@hamed-zakerzadeh?source=rss-70119830737e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*QefNWfvfpS8huh3f_EDWkg.png</url>
            <title>Stories by Hamed Zakerzadeh on Medium</title>
            <link>https://medium.com/@hamed-zakerzadeh?source=rss-70119830737e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 23:54:37 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@hamed-zakerzadeh/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[In the Search of the “Logistic Regression”]]></title>
            <link>https://medium.com/swlh/in-the-search-of-the-logistic-regression-f763894b4235?source=rss-70119830737e------2</link>
            <guid isPermaLink="false">https://medium.com/p/f763894b4235</guid>
            <category><![CDATA[logistic-regression]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[maximum-likelihood]]></category>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[statisics]]></category>
            <dc:creator><![CDATA[Hamed Zakerzadeh]]></dc:creator>
            <pubDate>Fri, 08 Jan 2021 13:26:06 GMT</pubDate>
            <atom:updated>2021-01-12T14:25:31.640Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Unfortunately, if you ask a “data scientist” to explain the real machinery and rationale behind the logistic regression, his answer can hardly convince anyone, even if he (which I doubt)! They repeat the same “ambiguous” explanations about logit function and logodds and how much it looks like a line! “Look! It is so simple: ln(p/(1-p))=β₀+β₁ x”. Things even get worse when they want to justify the strange form of the “cost function”. My first goal in this post is to justify the cost function in a simple, but mathematically valid, language. Moreover, I will explain it from another point of view: </em><strong><em>maximum likelihood estimation</em></strong><em>.</em></p><p>The dean of the Department of Economics and Business in a respectable university would like to predict whether an application submitted for their graduate program will be accepted or rejected, based on the applicant’s GPA. She reviews 20 applications from the last year, and she puts her findings in a scatter plot, where 0 and 1 correspond, respectively, to rejection or acceptance of an application. (<em>In French Mention means grade!</em>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/573/0*udDXLfObyDv9XdIU" /></figure><p>Even though the correlation between GPA and the application decisions is not that strict, she can still say that applicants with higher GPAs <strong><em>“seem more likely”</em></strong> to get accepted. But, rather than such rough judgments, the dean prefers to be more precise and get a model describing this<strong> “likeliness”</strong> or <strong>“probability”</strong>, a model which provides the acceptance probability given a particular GPA. What can she do?</p><h4><strong>Linear regression? Not a good idea!</strong></h4><p>One may rapidly suggest the linear regression (seriously?!). It looks a bit strange, however, let’s give it a chance. Linear regression will provide this yellow line which gives the line with the smallest residual:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/589/0*1Z0UT7LUpoa4mWCo" /></figure><p>One may say that it is better than nothing, and I agree! But notice that the values the linear model predicts are not in the interval [0,1], so one cannot assign them directly to a valid probability between [0% — 100%].</p><h4><strong>Transforming the linear regression? Not a good idea neither!</strong></h4><p>One may think of <em>“transforming”</em> the linear model to models living in [0,1]. A very common and well-known trick is to use the <strong><em>logistic function</em></strong> which has an “S”-shaped curve <strong><em>(sigmoid function)</em></strong> :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/161/0*d2XTo4avddVqIlg7.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/0*VA9HI0ny-dguORd3" /></figure><p>In fact, the logistic function transforms the variable z living in [-∞,+∞] to y which lives in [0,1]. <strong>This property is exactly what we were looking for!</strong></p><p>So, it seems that the problem is solved and we can transform the output of our linear regression from [-∞,+∞] to [0,1]. To do this, we still need the linear estimator η=β₀+β₁ x but what we change is the estimated response yhat</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/46/0*7SMh2h4gxjm014OR.png" /></figure><p>which should be modified as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/69/0*Cf4LltjJemq5WZNw.png" /></figure><p>Let’s look at what we get. The red line is the result of applying the logistic function on the yellow line. As we wished, the estimated values are in [0,1] which is great. But does this model make any sense? The yellow line minimizes the residuals min||η-y|| among all other linear models. But what can we say about the red line? Does it reach the minimum of min||g(z)-y||? No! In fact, we cannot find much mathematical sense for this transformed model unless we make a small modification.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/0*w3G-e4tBXRw7RkXW" /></figure><h4><strong>Least-square of residuals? We are quite there!</strong></h4><p>Our persistent dean does not lose her hope and, this time she tries finding the solution of min||g(z)-y||. This is what she gets (the red line):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/0*9KZ2XTBQG8QKFosf" /></figure><p>Now, the values are in [0,1] and she is sure that this logistic model is the best model in the sense that it minimizes the norm of residuals. Seems much better, no? But she should not be so excited yet!</p><p>Do you remember that for finding the minimum of a function, we find the points where the derivative of the function is zero? In fact, this zero-derivative condition is satisfied not only for the global minimum of the function, which is our objective but also for all local minima and maxima. If the function is <em>convex</em>, one can be sure that there is only one minimum, but what if it is <em>“non-convex”</em> (as you see in the picture)?</p><p>When you ask your computer to solve the minimization problem, it searches for the points which satisfy the zero-derivative condition, starting from the initial guess you feed it with, and it gets back to you as soon as it finds such a point. But, can one guarantee that what is founded is the global minimum?</p><p>Unless the function is convex, this point can even be a global maximum! In fact, for non-convex problems, your computer may get trapped in another point other than the global minimum and consider it as the solution you asked for. This is the drawback of min||g(z)-y||; it is non-convex! It would be insightful to check this graphically:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/1*CD9x-mPAIN5Ce4zd6OxJ0A.png" /></figure><p>The absolute difference of the estimated response yhatₖ (for every data points k) and the true response yₖ is called the residual eₖ. Now, consider three residuals: e₁ and e₃ when the response is 1 and e₂ when it is zero:</p><ul><li>e₁=|1-g(η₁)| with η₁=β₀–0.3β₁</li><li>e₂=|0-g(η₂)| with η₂=β₀+0.2β₁</li><li>e₃=|1-g(η₃)| with η₃=β₀+0.8β₁</li></ul><p>For simplicity, we fix β₀=1.5. So, we plot each residual as a function of β₁: e₁² in (red), e₂² (blue), and e₃² (green). The sum e₁² +e₂²+ e₃² (cost function) is also plotted in orange, which is clearly non-convex!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/0*MXIN0M5KN24-3IBq" /></figure><p>Notice that the minimization problem here is much more difficult than the one in the linear regression. There, the regression was <strong>linear</strong>, so we could employ linear algebra (QR decomposition) to simplify the problem. Here, the logistic function makes the problem <strong>non-linear</strong>. So, we do not have any other choice but to use the zero-derivative condition.</p><p>But this is not the whole story. Even if God solve this minimization problem for you (which he do not!) it is not plausible to define the residuals similar to linear regression. Here, there are only two possible outputs: 0 (rejection) or 1 (acceptance). If the response is 0 and the estimate is 1, the residual is 1, but this <em>“1”</em> is really huge, it is the largest error one could make, it is not <em>“only 1”</em>! So, it makes sense to define residual differently such that the cost function shows the <em>“huge”</em> difference between the two possible outputs.</p><h4><strong>Re-defining the residuals? Finally … the salvation!</strong></h4><p>Rather than the usual residual eₖ=|yhatₖ-yₖ|, we want to define a new residual ϵₖ to be huge when eₖ=1. What about this?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/155/0*2m-OSgpSRYspYt_f.png" /></figure><p>As we can see in the graph</p><ul><li>if e=0 (smallest difference), ϵ=0</li><li>If e=1 (largest difference), ϵ=+∞</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/0*A0XDxkKt9NGnm8tP" /></figure><p>So, for this new residual, having a difference of 1 is so huge, which is exactly what we were looking for. Then, instead of minimizing ||e|| or ∑ₖ eₖ², we consider another minimization problem min ∑ₖϵₖ. One can show that the function min ∑ₖϵₖ convex, so has only <strong>one local (and global) minimum</strong>. The more common form of writing such a cost function is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/377/0*mMXun9meO8e1hMvK.png" /></figure><ul><li>When yₖ=0, the residual will be -ln(1-yhatₖ)</li><li>When yₖ=1, the residual will be -ln(yhatₖ))</li></ul><p>which gives the same things as the other formula min ∑ₖϵₖ.</p><p>Considering the same example as before, we compute the new residuals: ϵ₁ and ϵ₃ when the response is 1 and ϵ₂ when it is zero:</p><ul><li>ϵ₁=-ln(1-e₁)</li><li>ϵ₂=-ln(1-e₂)</li><li>ϵ₃=-ln(1-e₃)</li></ul><p>So, we plot each residual as a function of 1: ϵ₁ (red), ϵ₂ (blue), and ϵ₃ (green). The sum (cost function) ϵ₁+ϵ₂+ϵ₃ is also plotted in orange, which is convex!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/0*-08MG9bo-thE1nn6" /></figure><p>Let us solve min ∑ₖ ϵₖ and obtain the logistic model as this new red line, which has values in [0,1], whose residuals are based on a correct intuition, and is much easier to be computed due to convexity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/611/0*xPspPFVI90OOCGRJ" /></figure><p>Having probabilities is nice but sometimes we want to be more strict, we want to decide: 0 or 1! In such cases, we often consider the threshold probability of 50% based on the logistic model: if the probability is more than 50% the application will be accepted and rejected otherwise.</p><p>Mathematically speaking, this is related to the sign of our linear estimator η=β₀+β₁ x:</p><ul><li>If η&lt;0, the probability will be more than 50%</li><li>If η&gt;0, the probability will be more than 50%</li><li>If η=0, the probability is 50%. This gives us the <strong><em>“decision boundary”</em></strong> which is colored black in the graph for Mention = 17.29983.</li></ul><h4><strong>Who came up with such a strange name, <em>logistic</em> regression?</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/129/0*R2wOURwGGJYMqP_T.png" /></figure><p>In 1798 Thomas Malthus published his book <em>“An Essay on the Principle of Population”</em>, warning of<em> “future difficulties, on an interpretation of the population increasing in geometric progression (so as to double every 25 years) while food production increased in an arithmetic progression, which would leave a difference resulting in the want of food and famine, unless birth rates decreased”</em>. It was then in 1838 that <strong>Pierre François Verhulst</strong>, a Belgian mathematician, modified Malthus’s model and obtained a more realistic function like g(z)=1/(1+e⁻ᶻ) predicting the population of countries quite well. He called it the logistic function but did not provide any justification for this name!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/719/0*SXFytkveIxVrc93m" /></figure><h4><strong>Maximum likelihood estimation: the hidden calculation behind the logistic regression</strong></h4><p><strong><em>Maximum likelihood estimation</em></strong><em>.</em> or MLE, as we often call it, is a way to find the probability distribution function fitted to some data. <a href="https://www.youtube.com/watch?v=XepXtl9YKwc&amp;ab_channel=StatQuestwithJoshStarmer">Here</a> is a good introduction.</p><p>For example, if we assume that the <strong><em>Gaussian normal distribution</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/228/0*b5sA-TPMdOlruFz7.png" /></figure><p>is to describe the data’s distribution, we need to find the best two parameters, mean μ and the standard deviation σ, to describe the data. But what if the data (the event or output) is binomial (0 and 1, or rejection and acceptance)? We need the <strong><em>binomial distribution</em></strong>.</p><p>Consider</p><ul><li>m as the total number of observations (the total number of acceptances and rejections)</li><li>p is the (average) <strong>acceptance</strong> chance of an application</li><li>1-p is the (average) <strong>rejection</strong> chance of an application</li></ul><p>The probability of having x accepted applications is pˣ while (m-x) rejections occur with the probability of (1-p)ᵐ⁻ˣ. So do we multiply these two probabilities to get the probability of getting x acceptances and (m-x) rejections? No! The x acceptances and (m-x) rejections can occur in different orders, that is, there are several different ways of distributing them. The <strong>binomial coefficient</strong> Cᵐₓ gives the number of possible combinations of having x acceptances and (m-x) rejections, so that the probability writes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/274/0*NB2HFeSoH9vkX48s.png" /></figure><p>where Cᵐₓ reads</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/149/0*1GfV2-sctFVxZci6.png" /></figure><h4><strong>Example (binomial coefficient):</strong></h4><p>If m=4 and x=2 (2 acceptances), then there are 6 different combinations in which these 2 acceptances occur (❌ is the rejection and ✔ is the acceptance):</p><p>✔✔❌❌</p><p>❌❌✔✔</p><p>✔❌❌✔</p><p>❌✔✔❌</p><p>❌✔❌✔</p><p>✔❌✔❌</p><p>The binomial coefficient will be then 6 as the formula suggests: C⁴₂=6</p><p>Now that we reviewed the basics of the binomial distribution, we can see how the maximum likelihood estimation works for the binomial distribution: it finds the acceptance chance p which maximizes the probability of having the observed number of acceptance and rejections. For example, assume that 7 out of 10 applications are rejected. So, the probability will be C¹⁰₃ p³ (1-p)⁷.</p><p>MLE looks for the p which makes this probability or likelihood of our result as large as possible. Simple calculations show that it is p=0.3 which does this job! You can see that from the plot as well. Notice for example that if we considered p=0.6, which assumes 60% acceptance chance, it would be quite unlikely (less than 5%) that we got such a result (only 3 acceptances).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*sx2_mX3r6J5aTtmC.png" /></figure><p>However, in our problem of predicting the decision based on GPA, things are not as easy as this because p is clearly not the same for all applications; p is not a constant anymore but a function of independent variables of the model (like GPA). Thus, instead of p, we have pₖ for each application k=1,2,…,m. Ignoring the constant Cᵐₓ, we will write the binomial distribution as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/211/0*gFhaFuHBcFY0a6bh.png" /></figure><p>where yₖ is the decision. And what is pₖ? Similar to our discussion in the first part, we consider this function to be the logistic function pₖ=g(ηₖ)=1/(1+exp(-ηₖ)). So, the objective is to find the maximum of this likelihood function L.</p><p>As <em>“logarithm of products is a summation of logarithms”</em>, people often work with log-likelihood l, which is the logarithm of the likelihood, that is l=ln(L) or</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/363/0*B7YNcfgyQu2SEfCM.png" /></figure><p>Don’t you think that this log-likelihood looks a bit familiar? ;) <strong>This is exactly the cost function we wanted to minimize in the first part of this article, but with a negative sign!</strong> So, maximizing the likelihood function is, in fact, minimizing the cost function! Now, we have a clear idea of where that function originated from.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f763894b4235" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/in-the-search-of-the-logistic-regression-f763894b4235">In the Search of the “Logistic Regression”</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>