<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Benuri Edirisinghe on Medium]]></title>
        <description><![CDATA[Stories by Benuri Edirisinghe on Medium]]></description>
        <link>https://medium.com/@benuriedirisinghe?source=rss-63fdb62bae92------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ejhxW4MGoCQj4TKXIt5iAg.jpeg</url>
            <title>Stories by Benuri Edirisinghe on Medium</title>
            <link>https://medium.com/@benuriedirisinghe?source=rss-63fdb62bae92------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 12:27:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@benuriedirisinghe/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Second Law of Thermodynamics and Entropy]]></title>
            <link>https://medium.com/@benuriedirisinghe/second-law-of-thermodynamics-and-entropy-3bbe742ecc6a?source=rss-63fdb62bae92------2</link>
            <guid isPermaLink="false">https://medium.com/p/3bbe742ecc6a</guid>
            <category><![CDATA[second-law]]></category>
            <category><![CDATA[entropy]]></category>
            <category><![CDATA[carnot]]></category>
            <category><![CDATA[thermodynamics]]></category>
            <dc:creator><![CDATA[Benuri Edirisinghe]]></dc:creator>
            <pubDate>Wed, 09 Jul 2025 06:28:52 GMT</pubDate>
            <atom:updated>2025-07-09T06:28:52.666Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Second Law of Thermodynamics and Entropy</strong></p><p>The second law of thermodynamics states that, in a process that occurs inside a closed system, there is always wasted energy, which is not used for the process directly (The Second Law of Thermodynamics, n.;d.). This can be expressed in a more generalized form using the concept of entropy. Which is the entropy of the universe is always increasing (Frautschi, 1982). The concept of entropy, which some may refer to as randomness, is always addressed when it comes to the second law. It is known to be a ‘misunderstood’ concept in physics because of its vast generalization due to having many definitions. In this essay, the connection between the second law and entropy will be discussed using examples.</p><p>The history of the second law runs back to the 19th century, when Sadie Carnot, a 28-year-old Frenchman, who published a work which included the quote:</p><p>“Everyone knows that heat can produce motion. That it possesses vast motive power no one can doubt in these days when steam engines are everywhere so well known…. Notwithstanding the work of all kinds done by steam engines, notwithstanding the satisfactory condition to which they have been brought today, their theory is very little understood, and the attempts to improve them are still directed almost by chance.” (Raman, 1970, p.331)</p><p>Carnot had realized that approaching this problem with a mathematical way would gain him more fruitful results. With the abstract approach without using the exact mechanisms used in engines or materials, he was successful with the foundation for the second law. But this was not noticed until Rudolf Clausius in 1850 published it in a paper, referring to them as the ‘Carnot’s principles’ (Raman, 1970). Carnot’s principles refer to the efficiency of heat engines, saying that irreversible heat engines always have less efficiencies than the reversible ones. This is because the fact, ‘part of energy is always wasted’ that was mentioned in the introduction. Therefore, a need of a heat sink is a must because the <em>perfect </em>process, which can also be known as the reversible process, is not practical in almost every process. For an example, take a hot cup of tea is set on a room at 25 degrees of Celsius. The tea is cooling down with time, but can the tea be hot again by keeping it inside the room? The answer would be no. But isn’t that what happens inside a fridge? It cools down the food inside by transferring heat from a cold temperature medium to a hot temperature medium. The only difference is that it is supported by an external power source. The idea of the fridge is sparked by the statement that was declared by Rudolf Clausius: heat cannot of itself pass from a colder to a hotter body (Raman, 1970). This led to another interesting concept which was also discovered by him. Which he named as ‘entropy’ (Cengel &amp; Boles, 2014).</p><p>As mentioned before, the second law talks about the perfection of a process. Which is a qualitative measure of energy. This can be explained by the example that was used before. The teacup, which is cooling down inside the room, transfers its heat to the room. The energy is conserved here but it has also expanded in a way that it cannot be used again. The quality of it is lost and we cannot reverse it. This is the simple explanation of entropy, which we can also name as ‘randomness’ or ‘disorder’. The heat loss from the tea had made an increasement of the entropy of the surroundings. This leads to the idea that is mentioned in the second law: the entropy of the universe tends towards a maximum (Raman, 1970, p.333). Because the entropy loss from a system increases the entropy somewhere else and it goes on until some of it is released to the universe. The oldest and the easiest idea associated with entropy is ‘disorder’, identified by Boltzmann. But he never <em>defined </em>entropy as disorder, he simply <em>associated </em>entropy with disorder. He used the term to describe the state of the process (Ben-Naim, 2013). The concept was also used by Atkins, who wrote:</p><p>“We shall identify entropy with disorder’’ and “With disorder in mind, we shall explore the implication of Clausius’ expression and verify that it is plausible in capturing the entropy as a measure of the disorder in a system.” (Ben-Naim, 2013, p.24)</p><p>But a proper proof to use disorder in terms of describing entropy has never been made. Yet, it is one of the most used descriptive words used when it comes to entropy. And the much needed mathematical approach to entropy was first done by Clausius (Raman,1970) again, which gives a much more solid idea about the definition of entropy.</p><p>The Clausius relation is the mathematical approach to the second law, and it is further derived to obtain the Clausius inequality, which is,</p><p>Where <em>dS </em>is the entropy change and <em>dq</em> is the heat transferred from the system or to the system. And <em>T</em> is the temperature of the surroundings in Kelvin. This inequality between the entropy change and the heat transferred, is due to the entropy generation during the irreversible process is not added (Cengel &amp; Boles, 2014). This principle is also known as an “echo of the Aristotle’s concept: the whole is greater than the sum of its parts” (Clausius, n.d.). Which leads to the meaning that it is better to analyse entropy as a whole than in parts. Clausius’ initial works led to the second law of thermodynamics making his name a definite factor when it came to thermodynamics, even though he is the most forgotten nineteenth century scientist. This is due to his disappearance from history with no biographical details or any detailed descriptions, the known facts about him are very little (Cropper, 1986). While the generalized idea of entropy dissipates to ‘chaos’ or ‘randomness’, Clausius managed to find a mathematical approach to get the definition of entropy using differential calculus (Balibrea, 2016). This helped in understanding entropy as a scientific concept to many scientists as well as students.</p><p>The concept of entropy is known as one of the most misunderstood concepts in physics. But with time it has evolved and the idea of it is much more graspable after watching a few YouTube videos nowadays. But it is also interesting to delve into the classical-thermodynamics-days and go through the true works of the revolutionary physicists who helped building these concepts like, Ludwig Boltzmann, Sadie Carnot, Rudolph Clausius and William Thompson. It is a well-known fact that entropy and the second law of thermodynamics are always involved with one-another. One’s discovery led to another. This essay was written in hopes of giving the idea of the connection between the two concepts, with the examples of the true works of early developers. Learning the history behind the concepts would give the reader a much clear view to the background of the problem, hence making understanding it easier. And understanding the connection between the second law and entropy is explained in the next paragraph, with an understandable day-to-day example. The article then tends to the mathematical approach, describing the Clausius inequality. In the author’s opinion, the second law of thermodynamics is a concept that opens the doors to many more areas in classical physics, and also can be a foundation for the world of quantum physics.</p><p>Reference List</p><p>[1] THE SECOND LAW OF THERMODYNAMICS. (n.d.).https://bazaarmodel.net/phorum/read.php?1,90</p><p>[2] Frautschi, S. (1982). Entropy in an Expanding Universe. <em>Science</em>, <em>217</em>(4560), 593–599. <a href="https://doi.org/10.1126/science.217.4560.593">https://doi.org/10.1126/science.217.4560.593</a></p><p>[3] Raman, V. V. (1970). Evolution of the second Law of Thermodynamics. <em>Journal of Chemical Education</em>, <em>47</em>(5), 331. <a href="https://doi.org/10.1021/ed047p331">https://doi.org/10.1021/ed047p331</a></p><p>[4] Cengel, Y. A., &amp; Boles, M. A. (2014). <em>Thermodynamics : An Engineering Approach 8th Edition</em>. <a href="https://sirkulasi-lib.unesa.ac.id/index.php?p=show_detail&amp;id=88668">https://sirkulasi-lib.unesa.ac.id/index.php?p=show_detail&amp;id=88668</a></p><p>[5] Ben-Naim, A. (2013). Entropy and the second law: interpretation and misss-interpretationsss. <em>Choice Reviews Online</em>, <em>50</em>(08), 50–4496. <a href="https://doi.org/10.5860/choice.50-4496">https://doi.org/10.5860/choice.50-4496</a></p><p>[6]Clausius, R. (n.d.). <em>Entropy Basics</em>. <a href="http://akademe.org/pubs/QTT/0entropy.pdf">http://akademe.org/pubs/QTT/0entropy.pdf</a></p><p>[7]Cropper, W. H. (1986). Rudolf Clausius and the road to entropy. <em>American Journal of Physics</em>, <em>54</em>(12), 1068–1074. <a href="https://doi.org/10.1119/1.14740">https://doi.org/10.1119/1.14740</a></p><p>[8]Balibrea, F. (2016). On Clausius, Boltzmann and Shannon Notions of Entropy. <em>Journal of Modern Physics</em>, <em>07</em>(02), 219–227. <a href="https://doi.org/10.4236/jmp.2016.72022">https://doi.org/10.4236/jmp.2016.72022</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3bbe742ecc6a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Writer’s Block]]></title>
            <link>https://medium.com/@benuriedirisinghe/writers-block-2ad33e3c212c?source=rss-63fdb62bae92------2</link>
            <guid isPermaLink="false">https://medium.com/p/2ad33e3c212c</guid>
            <category><![CDATA[stroy]]></category>
            <category><![CDATA[writers-block]]></category>
            <category><![CDATA[writing]]></category>
            <category><![CDATA[writer]]></category>
            <category><![CDATA[write]]></category>
            <dc:creator><![CDATA[Benuri Edirisinghe]]></dc:creator>
            <pubDate>Mon, 23 Dec 2024 13:16:46 GMT</pubDate>
            <atom:updated>2024-12-23T13:16:46.108Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*92e2O_eXVr4-dg6f" /><figcaption>Photo by <a href="https://unsplash.com/@kaitlynbaker?utm_source=medium&amp;utm_medium=referral">Kaitlyn Baker</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The temptation to write is still itching my finger tips. But whenever I sit down to write something I cannot think of a thing to write about. My head feels like a jumbled mess of words and phrases, which I cannot string on comprehensively. Not to brag but I used to write novels from a pencil in an exercise book as a kid and always seemed to be brimming with ideas. Getting hit with a writer’s block on break simply is the unluckiest thing to happen to a ‘supposedly’ writer.</p><p>Whenever I have some important assignment or exam lined up I always have some ideas that I swear to myself that I would get myself to write right after the exam, but it rarely ever happens.</p><p>I think it is because my brain is automatically trying to give myself something to think about that makes me happy to do after the actual work that I am supposed to do. It’s like giving myself a treat. But the treat actually never happens. It’s just a false reassurance that I give myself in order to complete the work that I have to do.</p><p>I used to think that I would be a big writer someday, but right now, that dream is nowhere to be seen. The writing I do here and there are just my outlets for any problem that I am going through at that moment.</p><p>I used to maintain a diary since I was twelve years old. I used to complain about things and write down the minimal details that I had with my crushes on those days. It was inspired by Anne Frank, a young girl who maintained a diary during the days of World War II when she and her family were in hiding. It fascinated me as a young girl too. I am an avid reader, who can read through almost anything and everything. Anne’s story had a real impact on me, and for once, I was actually grateful for what I had when I was thirteen years old, compared to her.</p><p>Getting out of a writer’s block is hard work. For me, I have to find an earth-bending, mind-shattering literature piece to read. And soon enough, I would be in the delusion that I can write something like this if I was really motivated. And I would try and then give up, thinking that I can never really finish something I started. And I would end up being jealous of the author, who managed to conjure such a brilliant piece, during their lifetime.</p><p>I would try until I convince myself that I was not born to write.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ad33e3c212c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Adding 9s]]></title>
            <link>https://medium.com/@benuriedirisinghe/adding-9s-68881cfc8308?source=rss-63fdb62bae92------2</link>
            <guid isPermaLink="false">https://medium.com/p/68881cfc8308</guid>
            <category><![CDATA[maths-tricks]]></category>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[addition]]></category>
            <category><![CDATA[math-tricks]]></category>
            <category><![CDATA[fun-math-games]]></category>
            <dc:creator><![CDATA[Benuri Edirisinghe]]></dc:creator>
            <pubDate>Sun, 18 Aug 2024 15:51:43 GMT</pubDate>
            <atom:updated>2024-08-18T15:51:43.852Z</atom:updated>
            <content:encoded><![CDATA[<p>When I was a kid, my dad showed me a trick on addition.</p><p>He was driving me to school, my favorite radio station buzzing in the background, when he decided to strike up a conversation with me.</p><p>“Do you see that number plate on that red car? Add it up and tell me the answer,” I took a minute, but managed to find an answer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nsbJlQ5Vk_ge_o2a" /><figcaption>Photo by <a href="https://unsplash.com/@ross_sokolovski?utm_source=medium&amp;utm_medium=referral">Ross Sokolovski</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>“1?”</p><p>“Correct! The blue one next, but be faster than the last time,”</p><p>But it took me more time than the last one because the number plate was filled with 9’s[FRT — 9999].</p><p>“9?”</p><p>“Correct! But 9 is already on the plate, isn’t it?” I was confused but nodded anyway.</p><p>“Look at that one[CJ — 9349], I am not going to add the 9’s, Ok? So, the answer is three plus four equals seven. It gives the same answer when you add it up with or without 9’s,”</p><p>No way that was true right? So, I decided to add it up again with the nines just to prove him wrong.</p><p>9+3+4+9= 25</p><p>2+5=7</p><p>I got 7! I was fascinated but on the same note I wanted to see if he had been lucky just this once. So I decided to have a go at it again with a different number plate.</p><p>I solved it with both methods, but the answer was the same! Now this time I was <em>truly </em>fascinated.</p><p>How did that happen?</p><p>Why doesn’t adding number 9 affect when you add up a number until a single digit remains?</p><p>When simply explained, it happens because when adding a 9 to some number, the value that the 10&#39;s place goes up is the same as the value that the 1’s place goes down, which is 1, when adding a single 9.</p><p>We can explain this by the multiplication table of 9,</p><p>1 x 9 = 09</p><p>2 x 9 = 18</p><p>3 x 9 = 27</p><p>4 x 9 = 36</p><p>As you can see, the first digit moves up by one while the second digit moves down by 1 at the same time, when we add 9. So when we add all the digits together the effect of adding a 9 cancels out because of this. Therefore there’s no point in adding a 9, when you are going to add up all the digits.</p><p>But note that it is important to add 9 in every other occasion, where you cannot ignore them!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=68881cfc8308" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Learning Regression in Machine Learning: A Simple Guide for Beginners]]></title>
            <link>https://medium.com/@benuriedirisinghe/understanding-learning-regression-in-machine-learning-a-simple-guide-for-beginners-45df52adcbcd?source=rss-63fdb62bae92------2</link>
            <guid isPermaLink="false">https://medium.com/p/45df52adcbcd</guid>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[linear-regression-model]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[regression]]></category>
            <category><![CDATA[mathematics]]></category>
            <dc:creator><![CDATA[Benuri Edirisinghe]]></dc:creator>
            <pubDate>Sun, 18 Aug 2024 14:07:27 GMT</pubDate>
            <atom:updated>2024-08-18T14:07:27.288Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>What is Machine Learning?</em></strong></p><p>When asked “What is machine learning?”, most people would probably imagine a set of machines, or we could say robots, sitting in a classroom learning things. It <em>is </em>kind of accurate, considering what machine learning actually is. According to Arthur Samuel, “machine learning is the field of study that gives computers the ability to learn without being explicitly programmed”. This quote dates back to 1959, proving that machine learning has been around for nearly seventy years. It is not a futuristic fantasy that is ages away and is deemed unreachable. It is a technology that is already here, without a doubt.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LjTgEhu-ZYT8hiKo" /></figure><p><em>Figure 1 — ‘Machine’ Learning</em></p><p>A common example of machine learning being used in our day today devices is the “spam” filter in our emails. The filter sorts the mail into spam or ham (non-spam or regular) using machine learning algorithms. The feature was first introduced in 1990 and has been improved ever since then. Now there’s barely a miss when it comes to our emails, and you must find that the accuracy of sorting is nearly perfection.</p><p>To explain this process in an easier way, let’s say the developers write a self-learning algorithm, where the machine can identify the spam email, even if the senders keep changing the words or phrases. A basic code for identifying would specify a list of words to avoid but a machine learning algorithm would even identify variations of the words and patterns, eliminating the task of re-writing and updating the code furthermore.</p><p>In this article, we are going to talk about one of the simplest ML models that exist, which is linear regression. So let’s get started.</p><p><strong><em>Where does Linear Regression fall in ML?</em></strong></p><p>Machine learning is a vast subject that has a lot of branches spread about, classifying the ML algorithms in different categories. The three main classifications are,</p><ul><li>Whether or not the algorithm is trained with human supervision</li><li>Whether or not they can learn in a step by step basis</li><li>Whether they function by comparing new data sets to known ones, or instead by detecting patterns in the training data and building a predictive model</li></ul><p>And linear regression would fall under supervised learning, implying that these tasks are done under human supervision. In this method, unlike unsupervised learning, we feed the algorithm with a labeled data set to train with and to clearly identify the relationship between the input and the outputs. This is called training the model and it will be further explained later in the article.</p><p><strong><em>What is Linear Regression?</em></strong></p><p>When simply put, regression means the connection between the two or more variables, namely input(s) or outputs, in this case. And the word ‘linear’ is there because we assume that the relationship between the variables is linear (additive).</p><p>For an example, y=mx+c is the simplest example that can be given for the simple linear regression model.</p><p>And y = m1x1 +m2x2 + m3x3 + …… + mnxn + c is an example for multiple linear regression model.</p><p>What linear regression actually do is make a prediction by simply computing a weighted sum of the input features, plus a constant called the <em>bias term/ intercept term</em>. Bias term is the term we include to represent the tendency of the data to have a distribution centered about a given value that is offset from an origin.</p><p>Example 1 :-</p><p>ŷ = θ0 +θ1x1 +θ2x2 +……+θnxn</p><p>In this equation,</p><p>ŷ is the value predicted by the model.</p><p>n is the number of features.</p><p><em>xi</em> is the <em>ith</em> feature value.</p><p>θj is the <em>jth </em>model parameter (including the bias term θ0 and the feature weights θ1,θ2,….,θn )</p><p>This also can be written in a vectorized form,</p><p>yˆ = hθ (x) = θ ⋅ x</p><p>In this equation:</p><ul><li>θ represents the model’s ‘parameter vector’, containing the bias term θ0 and the feature weights θ1 to θn</li><li><em>X</em> is the instance’s ‘feature vector’, containing x0 to xn, with x0</li><li>θ.x is the dot product of the vectors θ and x, which is of course equal to θ0 x0<strong>+</strong>θ1 x1 +θ2 x2<strong>+…..+</strong>θn xn.</li><li>hθ is the hypothesis function, using the model parameters θ .</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*9Heh-T-GESAh24uv" /></figure><p><em>Figure 2 — A graphical representation of the linear regression model</em></p><p><strong><em>What Are the Applications of Linear Regression?</em></strong></p><p>Modern world uses linear regression in many instances, for example, if you want to predict your company’s revenue next year, it can be predicted by using the Linear Regression Model.</p><p>And to classify these tasks in a more generalized manner;</p><p>We can do,</p><ul><li>Market analysis by using marketing strategies and maximizing sales</li><li>Sports analysis (Predicting the attendance of the game, the chances of winning or losing,etc.)</li><li>Predicting the impact of environmental pollution (Increase of global warming)</li><li>Health predictions (Risk of diseases, identifying high-risk patients, etc.)</li></ul><p>Next up is how do we train the linear regression model using a dataset.</p><p><strong><em>How Do We Train the Model?</em></strong></p><p>In machine learning, a training data set is the data set we use to train the model. And training the model means setting its parameters so that the model best fits the training set. To do this we need to measure how well or how poorly the model fits the training data.</p><p>Most common performance measure for a regression model is the Root Mean Square Error(RMSE) method.It is also one of the two main performance indicators for the linear regression model. It measures the average difference of the values predicted by a model and the actual values. It provides an estimation of how well the model is able to predict.</p><p>Lower the RMSE value, the better the model. Perfection would mean the value is zero.</p><p>Therefore to train a linear regression model, we need to find a parameter vector(θ) value that minimizes the RMSE value. But in practice it is much easier to minimize the mean squared error (MSE) than the RMSE, and it leads to the same result we need.</p><p>Because,</p><p>f(x) ↓ → √f(x) ↓</p><p>(minimizing a function also implies that it minimizes its square root)</p><p>The MSE of a Linear Regression hypothesis hθ on a training set X is calculated using the below equation,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/466/1*H2fakD3SrzZJq5CTRr4qsw.png" /></figure><p>Notations -</p><p>m is the number of instances in the dataset you are measuring the MSE on.</p><p>For example, if you are evaluating the MSE on a validation set of 2,000 districts, then m = 2,000.</p><p>x(i) is a vector of all the feature values (excluding the label) of the ith instance in the dataset, and y(i) is its label (the desired output value for that instance).</p><p>For example, if the first district in the dataset is located at longitude –118.29°, latitude 33.91°, and it has 1,416 inhabitants with a median income of $38,372, and the median house value is $156,400 (ignoring the other features for now), then:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/334/1*m2NgP0h7dQmXyXweNbSG5Q.png" /></figure><p>and:</p><p>y (1) = 156,400</p><p>X is a matrix containing all the feature values (excluding labels) of all instances in the dataset. There is one row per instance, and the ith row is equal to the transpose of x(i) , noted (x(i) )T .</p><p>For example, if the first district is as just described, then the matrix X looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/745/1*nuytkwWGzpe7iUJfdUsdyQ.png" /></figure><p>h is your system’s prediction function, also called a hypothesis.</p><p><strong><em>The Normal Equation</em></strong></p><p>To find the value of the parameter vector that minimizes the RMSE function. There is a ‘closed-form solution’, in mathematics, an equation that gives a direct result. This is what we call the ‘Normal Equation’.</p><p>The Normal Equation:</p><p>θˆ = (XT X)−1 XT y</p><p>In this equation:</p><ul><li>θˆ is the value of θ that minimizes the cost function.</li><li>y is the vector of target values containing y(1) to y(m).</li></ul><p>Now let’s try this out in python. For this you need to install the Scikit-Learn library.</p><p>First we are going to generate a data set, a linear-looking one at that, to test this out.</p><pre>import numpy as np<br>X = 2 * np.random.rand(100, 1)<br>y = 4 + 3 * X + np.random.randn(100, 1)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Bva9imhU5bcu8cBb" /></figure><p><em>Figure 3-Randomly generated linear dataset</em></p><p>Now let’s compute θˆ using the Normal Equation. We will use the inv() function from NumPy’s linear algebra module (np.linalg) to compute the inverse of a matrix, and the dot() function for matrix multiplication:</p><p>X_b = np.c_[np.ones((100, 1)), X] # add x0 = 1 to each instance <br>theta_best = np.linalg.inv(X_b.T.dot(X_b)).dot(X_b.T).dot(y)</p><p>The function that we used to generate the data is y = 4 + 3x1, + Gaussian noise (to cause disturbance to the data set). Let’s see what the equation found:</p><pre>&gt;&gt;&gt; theta_best <br>array([[4.21509616],<br>[2.77011339]])</pre><p>We would have hoped for θ0 = 4 and θ1 = 3 instead of θ0 = 4.215 and θ1 = 2.770. Close enough, but the noise made it impossible to recover the exact parameters of the original function.</p><p>Now we can make predictions using θˆ:</p><pre>&gt;&gt;&gt; X_new = np.array([[0], [2]])<br>&gt;&gt;&gt; X_new_b = np.c_[np.ones((2, 1)), X_new] # add x0 = 1 to each instance <br>&gt;&gt;&gt; y_predict = X_new_b.dot(theta_best)<br>&gt;&gt;&gt; y_predict<br>array([[4.21509616], <br>[9.75532293]])</pre><p>Let’s plot this model’s predictions:</p><pre>plt.plot(X_new, y_predict, &quot;r-&quot;)<br>plt.plot(X, y, &quot;b.&quot;)<br>plt.axis([0, 2, 0, 15])<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hLQxF0EwyNukJ4-I" /></figure><p><em>Figure 4 — Linear Regression model predictions</em></p><p>Performing Linear Regression using Scikit-Learn is simple:</p><pre>&gt;&gt;&gt; from sklearn.linear_model import LinearRegression <br>&gt;&gt;&gt; lin_reg = LinearRegression()<br>&gt;&gt;&gt; lin_reg.fit(X, y)<br>&gt;&gt;&gt; lin_reg.intercept_, lin_reg.coef_ <br>(array([4.21509616]), array([[2.77011339]])) <br>&gt;&gt;&gt; lin_reg.predict(X_new)<br>array([[4.21509616], <br>[9.75532293]])</pre><p>The LinearRegression class is based on the scipy.linalg.lstsq() function (the name stands for “least squares”), which you could call directly:</p><pre>&gt;&gt;&gt; theta_best_svd, residuals, rank, s = np.linalg.lstsq(X_b, y, rcond=1e-6) <br>&gt;&gt;&gt; theta_best_svd<br>array([[4.21509616], <br>[2.77011339]])</pre><p>This function computes θˆ = X+y, where X+ is the ‘pseudoinverse’ of X (specifically, the Moore-Penrose inverse). You can use np.linalg.pinv() to compute the pseudoinverse directly:</p><pre>&gt;&gt;&gt; np.linalg.pinv(X_b).dot(y) <br>array([[4.21509616],<br>[2.77011339]])</pre><p>The pseudoinverse itself is computed using a standard matrix factorization technique called Singular Value Decomposition (SVD) that can decompose the training set matrix X into the matrix multiplication of three matrices U Σ VT (see numpy.linalg.svd()). The pseudoinverse is computed as X+ = VΣ+UT. To compute the matrix Σ+, the algorithm takes Σ and sets to zero all values smaller than a tiny threshold value, then it replaces all the nonzero values with their inverse, and finally it transposes the resulting matrix.</p><p>This approach is more efficient than computing the Normal Equation, plus it handles edge cases nicely: indeed, the Normal Equation may not work if the matrix XT X is not invertible (i.e., singular), such as if m &lt; n or if some features are redundant, but the pseudoinverse is always defined.</p><p><strong><em>Computational Complexity</em></strong></p><p>The Normal Equation computes the inverse of XTX, which is an (n+1)x(n+1) matrix, where n is the number of features. The ‘computational complexity’ of inverting such a matrix is typically about O(n2.4) to O(n3), depending on the implementation. In other words, if you double the number of features, you multiply the computation time by roughly 22.4 = 5.3 to 23 = 8.</p><p>In a more generalized manner ‘computational complexity’ means a measure of the amount of computing resources, such as time and space, an algorithm uses when it runs. In this case, it’s the number of features and the computation time.</p><p>The SVD approach used by Scikit-Learn’s Linear Regression class is about O(n2)1. If you double the number of features, you multiply the computation time by roughly 4.</p><p>Once you have trained your Linear Regression model using any type of algorithm(Normal Equation or any other), the prediction rate is fast.</p><p>That means the computational complexity is linear with regard to both the number of instances you want to make predictions on and the number of features. In other words, making predictions on twice as many instances(or twice as many features) will take roughly twice as time.</p><p>Note that this training method can be used for a smaller number of features or training instances. Because the Normal Equation and the SVD approach get very slow when the number of features grows large (ex:- 100,000).</p><p>Notation-</p><p>O(n2)1 = The Big O notation — Describes the complexity of the cord in algebraic terms. For an example, in O(n2), n is the input size and n2 gives us the complexity of the algorithm with respect to its input size.</p><p>With this we come to the end of this article. Hope you learnt something new or verified your facts with this piece on linear regression! I have added the references for this article below if you want to know more on the subject.</p><p>Thank you for reading and happy learning!</p><p>References</p><p>[1]A. Géron, <em>Hands-on machine learning with Scikit-Learn and TensorFlow concepts, tools, and techniques to build intelligent systems</em>, 2nd ed. O’Reilly Media, Inc., 2019.</p><p>‌</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=45df52adcbcd" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>