<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Midhilesh elavazhagan on Medium]]></title>
        <description><![CDATA[Stories by Midhilesh elavazhagan on Medium]]></description>
        <link>https://medium.com/@midhileshelavazhagan?source=rss-1652b74e929b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*IC_XGEZgfIk6QiLLXE93Tg.jpeg</url>
            <title>Stories by Midhilesh elavazhagan on Medium</title>
            <link>https://medium.com/@midhileshelavazhagan?source=rss-1652b74e929b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 14:05:20 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@midhileshelavazhagan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Linear Regression — Deep View (Part 2)]]></title>
            <link>https://medium.com/@midhileshelavazhagan/linear-regression-deep-view-part-2-55d26ac51b13?source=rss-1652b74e929b------2</link>
            <guid isPermaLink="false">https://medium.com/p/55d26ac51b13</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Midhilesh elavazhagan]]></dc:creator>
            <pubDate>Wed, 20 Feb 2019 11:45:46 GMT</pubDate>
            <atom:updated>2019-02-20T11:45:46.216Z</atom:updated>
            <content:encoded><![CDATA[<h3>Linear Regression — Deep View (Part 2)</h3><p>I have discussed the basic concept of linear regression in <a href="https://medium.com/@midhileshelavazhagan/linear-regression-deep-view-part-1-fa1682ed8d78">part 1</a>. Let’s start answering the questions in part 1. Before getting into that question let’s understand what is curve fitting?</p><p>Curve fitting is basically trying to fit a linear line or non-linear curve that better fits the data points.</p><p><strong>Curve fitting, a probabilistic view</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*H_4wN1OzPhODyimPdvYpjw.png" /></figure><p>The goal in curve fitting problem is to make a prediction of the target variable Y given some new value of the input variable <em>X</em> on the basis of a training data comprising n input values <strong>X</strong>=(x₁,….xₙ<em>) </em>where xᵢ ’s are drawn from known probability density family <em>p(</em><strong>X</strong><em>|θ) </em>and the target value <strong>Y=</strong>(<em>y₁,….,yₙ</em>). We can express our uncertainty over the values of the target variable using a probability distribution. For example, we assume that given the value of <em>x, </em>the corresponding value of <em>t </em>has a Gaussian distribution. Thus finding the <strong>optimal value for θ </strong>is basically what curve fitting. Here θ is the parameter.</p><p>How to find optimal values for the parameter?</p><p>We cannot find the exact values of the parameters, but we can estimate values that are close to the actual parameter using a sample drawn from the population.</p><p>Methods to estimate the parametric values:</p><ol><li><strong>Maximum likelihood estimation</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/409/1*_H7xupNc1upJBOAPTtfD2Q.png" /></figure><p><em>l(θ|X) </em>is basically the likelihood of the parameter <em>θ </em>given a sample X. Since all xᵢ ’s are iid we can take the product over all the xᵢ. In maximum likelihood estimation, we are interested in finding the value <em>θ </em>that makes <strong>X </strong>more likely to be drawn. Thus, we need to search for <em>θ </em>that maximizes <em>l(θ|X). </em>We take log(.) over the likelihood and maximizes this log likelihood by making it’s derivative equal to zero. Thus,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/409/1*WUcWOND3Q2rN8vgUdFQyAA.png" /></figure><p>Why do we need to take a log transformation?</p><p>Taking log is basically a monotonic transformation which converts the multiplication into addition since derivative is not linear operator in multiplication.</p><p>Consider the distribution for <em>p(Y|X) </em>as Gaussian, then the <strong>mean is basically <em>θᵀ.X</em></strong>. As I mentioned in <a href="https://medium.com/@midhileshelavazhagan/linear-regression-deep-view-part-1-fa1682ed8d78">part 1</a> <em>θᵀ.X </em>is what our hypothesis and it the predicted value for given xi. The following image will depict it</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/623/1*cW5wyLHO5m0rs3B3qu1LrQ.png" /></figure><p>Since probability distribution of <em>y </em>given <em>x </em>follows Gaussian distribution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/518/1*K5LYw9JVFDJKc79o7m-l4Q.png" /></figure><p>We need to find the optimal value for <em>θ </em>by using maximum likelihood.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/345/1*JY3MA1wJbZa8ScDIPtQ5VA.png" /></figure><p>Since all the values of X are iid, we have</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/285/1*eA5sIQg7Na57wAG_viJOUg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*Q3Xc1Viw7O5uQcVqHYyEdA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*A4NI1jxneJB4dYtOzQIzFw.png" /></figure><p>By applying the log product gets transformed to sum</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/1*pJ9jldlW-CSWOBXdRG4pVg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/659/1*OCiAF0gl4JzkxGrUm3zyxw.png" /></figure><p>Hence maximizing the <em>l(θ) </em>gives the same as minimizing</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/270/1*BKdQZSOT-KbQCk83RM7QqA.png" /></figure><p>which we recognize to be <em>J(θ), </em>our original least-squares cost function. Thus under our previous assumption of data, least squares regression as just doing maximum likelihood estimation.</p><p>There are other two ways for estimating the parametric value</p><p>2. Maximum posterior estimation</p><p>3. Bayes’ estimation</p><p>You can refer to these concepts in my reference book.</p><p>My next blog deals with how to use matrix derivatives to find the minimum of the cost function <em>J(θ).</em></p><p>Reference</p><ol><li>Introduction to machine learning by Ethem Alpaydin</li><li>CS229 Andrew Ng course</li><li>Pattern Recognition and Machine Learning by Christopher M. Bishop</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=55d26ac51b13" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Linear Regression- Deep View(Part 1)]]></title>
            <link>https://medium.com/@midhileshelavazhagan/linear-regression-deep-view-part-1-fa1682ed8d78?source=rss-1652b74e929b------2</link>
            <guid isPermaLink="false">https://medium.com/p/fa1682ed8d78</guid>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Midhilesh elavazhagan]]></dc:creator>
            <pubDate>Sun, 17 Feb 2019 08:53:10 GMT</pubDate>
            <atom:updated>2019-02-20T11:49:18.349Z</atom:updated>
            <content:encoded><![CDATA[<p>Linear Regression is a supervised learning algorithm<strong> </strong>where we are trying to predict continuous values.</p><p>What is Supervised Learning?</p><p>In Supervised Learning, we are given a training set with both features as <em>x</em> and labels as <em>y </em>and the algorithm learns a function <em>h: x→y </em>so that <em>h(x) </em>is a good predictor for the corresponding value <em>y</em>. Here <em>h </em>is called a <strong>hypothesis.</strong></p><p>What is the hypothesis in linear regression?</p><p>To perform supervised learning we need to decide how we are going to represent the function <em>h.</em> Let’s decide to approximate <em>y </em>as a linear function of <em>x</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/319/1*xFk3unq9aRDNi8y8rQEXzg.png" /></figure><p>where <em>θi’s </em>are <strong>parameters </strong>parameterizing the space of linear function mapping <em>x to y.</em></p><p>Given a training set, how are we going to learn the parameter <em>θ?</em></p><p>One reasonable method makes <em>h(x) </em>close to <em>y </em>at least for the training example. For this let’s formulate <em>J(θ) </em>that measures how close h(xi)’s are to the corresponding yᵢ for each value of <em>θ’s. </em>Let’s call <em>J(θ) </em>as <strong>cost function:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/410/1*QKNacSrSyta0edAvRHc0lg.png" /></figure><p>How to update the parameters?</p><p>we want to choose <em>θ </em>so as to minimize <em>J(θ). </em>To do this let’s use a search algorithm called <a href="https://towardsdatascience.com/gradient-descent-simply-explained-1d2baa65c757"><strong>gradient descent</strong></a><strong>.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/248/1*jRvepApkRPFcAsdxrgeYqA.png" /></figure><p>where <em>α</em> is called the <strong>learning rate. </strong>After taking partial derivate on <em>J(θ)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/359/1*yk0INcRI9QvYaw3sLdXMzA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/315/1*cfg9mBIv5F33--T-WqsWFg.png" /></figure><p>For a single training example, the update rule is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/377/1*hY1GEdEaxPSCLhTTBKmTgQ.png" /></figure><p>This rule is called the <strong>LMS </strong>update rule (“LMS” — ”Least Mean Square”) or <strong>Widrow-Hoff </strong>learning rule.</p><p>How to train the model using LMS rule?</p><p><strong>Batch gradient descent: </strong>(updating <em>θ </em>only after the whole dataset )</p><p>Repeat until convergence {</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*C4SzHvhdmh8xRME_4doYIw.png" /></figure><p>( for every j ) }</p><p><strong>Stochastic gradient descent: </strong>(update <em>θ </em>for every data point)</p><p>Loop{</p><p>for i=1 to m {</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/377/1*hY1GEdEaxPSCLhTTBKmTgQ.png" /></figure><p>(for every j) }</p><p>}</p><p><strong>Mini-batch gradient descent: </strong>(It is a trade-off between batch gradient descent and stochastic gradient descent. we split the dataset into mini-batches and for every mini batch, we update the weights<strong>. </strong>The cost function is averaged over all the batches)</p><p>Here J is a convex quadratic function. The ellipse shown below is the contours of a quadratic function and a sample trajectory taken by gradient descent for the above three methods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*3L2t1Da4M3ztbB0I1Torhw.png" /></figure><p>From the above image, it is clear that the stochastic gradient descent takes a long time to reach optimum with a zig-zag path. Batch gradient descent takes a smooth path but not guaranteed to reach optimum. Mini-batch gradient descent takes less time, less zig-zag path when compared to stochastic gradient descent but reaches the optimum.</p><p>“Why might least-square cost function <em>J, </em>be a reasonable choice?.” Let’s see the answer to these questions in <a href="https://medium.com/@midhileshelavazhagan/linear-regression-deep-view-part-2-55d26ac51b13">part2</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fa1682ed8d78" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>