<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tanya Gupta on Medium]]></title>
        <description><![CDATA[Stories by Tanya Gupta on Medium]]></description>
        <link>https://medium.com/@tanya-gupta18?source=rss-5659a588d90b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*76WUybwQfMQFx9NP.jpg</url>
            <title>Stories by Tanya Gupta on Medium</title>
            <link>https://medium.com/@tanya-gupta18?source=rss-5659a588d90b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 17:31:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tanya-gupta18/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A Noob’s Guide to Random Forest]]></title>
            <link>https://levelup.gitconnected.com/a-noobs-guide-to-random-forest-d7398d56b01c?source=rss-5659a588d90b------2</link>
            <guid isPermaLink="false">https://medium.com/p/d7398d56b01c</guid>
            <category><![CDATA[random-forest]]></category>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[decision-tree]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[implementation]]></category>
            <dc:creator><![CDATA[Tanya Gupta]]></dc:creator>
            <pubDate>Thu, 28 Jan 2021 13:36:07 GMT</pubDate>
            <atom:updated>2021-01-29T14:14:53.726Z</atom:updated>
            <content:encoded><![CDATA[<p>As a noob, it is very easy to get lost while understanding random forests. But fear not my fellow noobs cuz this blog will give you a starter guide to survive this tangled mess of a topic. We will also be looking at the implementation of random forest as well as decision trees on a dataset which will help you further in understanding this algorithm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/677/0*NRpxi_ezJ4GfVTNY.jpg" /><figcaption>Source:www.fromthegenesis.com</figcaption></figure><h3>Introduction</h3><p>Random Forest is an <strong><em>ensemble learning</em></strong> method used mainly for classification problems but it can also be used for regression as well. It combines the decision making of multiple decision trees and then takes the majority response out of all the decision trees as the final verdict/output.</p><p><strong>Ensembles </strong>are basically predictive models that combines the predictions from the “weak learners” to create one “strong predictor”.</p><p><em>In order to understand and implement random forest you should first understand the concept of decision trees. If you would like, I have covered the theory for this topic in another blog whose link you can find </em><a href="https://medium.com/analytics-vidhya/a-noobs-guide-to-decision-trees-171252c5b653"><strong><em>here</em></strong></a><em>.</em></p><h3>Working</h3><p>In order to understand the working of random forest we need to understand the meaning of <strong>“bagging” or “bootstrap aggregating”</strong>.</p><p><strong><em>Bagging</em></strong> is a technique wherein we select a number of samples for each tree with replacement. <strong><em>Sampling with replacement</em></strong> suggests that when choosing dataset rows for sampling, we can choose some or all of those rows again for other decision tree as well.</p><p>After assigning samples to these trees, we then train the trees using these samples and then proceed to predict for a testing instance/row using all of them.</p><p>After we get the results from all the trees, we find out the majority label out of all the outcomes from the pool of results in case of classification and average the results in case of regression.</p><p>This entire process can be depicted using the following image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/0*rakt_8G6Aftn7qxn.png" /><figcaption>Source: towardsdatascience.com</figcaption></figure><p>If you still don’t understand this process we can take an example of a fruit basket containing cherries, apples and oranges. Assume we are to find the label for a certain fruit in the image given below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/0*oaPMwL4GvDRjv2Vb.png" /><figcaption>Source:pinterest.com</figcaption></figure><p>This unknown fruit has the following attributes: diameter = 3, it is orange in colour, it grows in summer and shape is circular.</p><p>Now we will use the random forest algorithm to find out where it belongs.</p><ul><li>So first we see that from our dataset, we take one sample each for our three decision trees and train them using their respective samples as shown in the image below:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/0*C9aoWNGLIgw0PQF4.png" /><figcaption>Source:pinterest.com</figcaption></figure><ul><li>After training them, we “pass” our test instance through each of these trees and see their prediction on it.</li><li>Taking first tree, we see that first it asks whether the diameter ≥3? for the test instance it is true so, it goes down the right path. Then the tree asks whether the color is orange. Again it is true, so we go down the right path which ends up in prediction being an orange.</li><li>Similarly, for tree 2 and tree 3 we get predicted label as “orange”.</li><li>So, the label which comes maximum number of times from all three predictions would be “orange”.</li></ul><p>The working is pretty straightforward and with the knowledge and implementation of decision trees, random forest becomes fairly easy.</p><h3>Implementation</h3><p>I have implemented this algorithm on breast cancer dataset which you can find <a href="https://github.com/tanya12181/Machine-Learning-Internship-at-Internity/blob/main/Task%209/Breast_cancer_data%5B1%5D.csv">here</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b900c6079d41ad334b29eea7844ee323/href">https://medium.com/media/b900c6079d41ad334b29eea7844ee323/href</a></iframe><ul><li>Import numpy and pandas libraries.</li><li>pandas.read_csv() is used to read the csv file and store the dataframe in the variable dataset.</li><li>I checked for null values using numpy.isnull().sum(). In this dataset there aren’t any null values.</li><li>I converted my dataframe into a numpy array because it is easier for me personally to use its slicing operations<em> ( I can also shuffle the rows easily using np.random.shuffle(&lt;array&gt;) and I don’t have to use dataframe.iloc everytime I want to access a column or a row).</em></li><li>I divided my dataset up in the ratio of 75:25 for training and testing set. I did this by first finding out the index at 3/4th of my dataset. Since, I am using Python 3.x, I would have to use integer division which uses “//” instead of “/” for normal division. This division would give me an integral value as opposed to float one.</li><li>train = dataset[:idx , : ] and test = dataset[idx: , : ]. Notice I have not divided train to x_train and y_train. Similar case for testing set. As we will go through the functions you will see why.</li></ul><h4>Decision Tree Functions</h4><p><strong>define count_label( &lt;section of dataset&gt;)</strong>: It creates a dictionary for different type of values in our response variable. In our case different types of values in “diagnosis” is either 0 or 1.</p><p>In “cnt” dictionary, for each type it stores number of times it occurred in the response column.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/659/1*fzbRoszoaT_Uk8I4XgBJbA.png" /></figure><p><strong>define entropy( &lt;section of dataset&gt;)</strong>: It finds the entropy or randomness in the entire dataset (specifically the response variable which in our case “diagnosis”).</p><p>It first gets the number of occurrences for each type of value in response variable column, then it calculates entropy by formula:</p><p><strong>entropy = </strong>Σ <strong>(-n/total)* log(n/total)</strong> where,</p><ul><li>total = number of rows in that section and,</li><li>n = number of occurrences for a certain type.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/529/1*RzFcMW19ZPaLvnWwIOQp4g.png" /></figure><p><strong>define info_gain(left_split, right_split, parent_entropy): </strong>where, left and right are the left and right portion of the dataset after partitioning.</p><p>It tells us how much information is gained after the splitting by:</p><ul><li>first, finding the summation of product of entropies and their weighted average.</li><li>then, subtracting this information with the parent entropy.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/803/1*DutazDwjE-EY9zBiVjH6XQ.png" /></figure><p><strong>class Node:</strong></p><p>This class was made to facilitate the storage of the best splitting feature, the node’s children (i.e., left and right split), the best question for that given feature and the final predicted label.</p><p>So, for all the nodes which aren’t leaf nodes we will store everything except predicted label.</p><p>To find out whether a node is leaf or not, we use the method <strong><em>is_leaf()</em></strong> which returns true if predicted label isn’t none.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MBNE_PR7I6ueviZNbmKwDg.png" /></figure><p><strong>define partition(&lt;rows of the section&gt;, question value, feature):</strong></p><p>So, the question value corresponds to the feature or the column based on which we are splitting. It creates a left and right partition based on this “threshold value” or question value.</p><p>If the column value of that row has value ≥ question value , then it is appended to right partition. Else, it is appended to the left partition.</p><p>It returns partitions as numpy arrays.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/553/1*0jNWmgHr0bhnAS3FFWz2wg.png" /></figure><p><strong>define best_split(rows, column):</strong></p><p>here, row defines the section of dataset to be split and column is a list of features which are already used up in splitting.</p><p>It returns the best question value and the best attribute or feature based on which we will gain maximum information gain and the entropies of the resulting children would be minimum.</p><p>It first checks whether the feature is already used for splitting in early iterations. If it is, then we skip this feature else, we continue to find the best possible question value for this feature.</p><p>We find the information gain for each and every value in that feature and store the best possible value in ques variable which gives maximum value for information gain.</p><p><strong><em>It returns ques and attr variables None, if there is no split with information gain &gt;0.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/617/1*n-RlT4LyfzfYkhjwxPkanQ.png" /></figure><p><strong>define build_tree(rows, max_depth, column):</strong></p><ul><li>column is a list of features which are already used up in splitting.</li><li>max_depth gives us the limit till which we will let our tree grow.</li><li>rows is the dataset.</li></ul><p><strong>Tasks it performs:</strong></p><ul><li>First it finds the best split using best_split function.</li><li>If the attribute value returned is not None, it then proceeds to find whether, our response variable contains only one type of value or not. It also updates the column list.</li><li>It returns the leaf node if max_depth is 0 or attribute/feature is None or splitting is pure (no. of types = 1).</li><li>Else, we partition the rows into left and right partitions and recursively call build_tree for both partitions, having max_depth decremented by 1 and with our updated column list.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/663/1*wIFJiN04o5QK6cmcz8wAwg.png" /></figure><p><strong>define find_common_label(column): </strong>For a given feature it returns the most common type of value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/1*Q3LcYLzTdsNAME46C33soQ.png" /></figure><p><strong>define_root(rows, max_depth):</strong> It calls build_tree() function and stores the root of our decision tree.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/559/1*QYT7s47jIT0c9AglH5g_kg.png" /></figure><p><strong>define predict(test_set, root): </strong>It takes the root of the decision tree modelled using our training data and for each test row in test set it gives predictions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/670/1*uOwVMpKYBQAd-ePDrAgntA.png" /></figure><p>define traverse_tree(test_row,node): It compares the feature value in the test row with the question value stored in our decision tree.</p><ul><li>If the feature value ≥ question value for that node, traverses the right path</li><li>Else, traverses the left path.</li></ul><p>If it reaches the leaf node then, it returns the predicted label.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*eBo_yN_1evzhE8khWX9R-w.png" /></figure><p>Oof that’s a lot of functions for decision trees.</p><h4>Random Forest Functions</h4><p><strong>define random_forest_classifier(): </strong>It takes number of trees, the training set, the test row (excluding the response column), maximum depth of tree as arguments.</p><ul><li>Each tree is assigned with samples from training set.</li><li>They are trained using our decision tree functions.</li><li>We append each of the tree roots to “trees” list and tree predictions to “predictions_per_tree” list.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/914/1*74FJe2IhJMYOlLBpQp2uKw.png" /></figure><p><strong>define get_samples(training set):</strong></p><p>We choose random row indices using np.random.choice(). I have chosen sample size to be one third of train set.</p><p>These randomly selected rows are then appended to sample list and then return.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/671/1*J5ZD7Wa1J9ojiLUj99l0xw.png" /></figure><p><strong>define random_forest_predict( tree_predictions, actual_test_response_labels):</strong> We count predictions for each row of each tree and find the majority label which is then returned as the final output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/1*uxqn61s_wFvhNn2daPJ6Uw.png" /></figure><p>I took number of trees = 5 and max_depth = 100. These values gave me an accuracy of 92.3%.</p><p>****************************************************************</p><h3>Advantages:</h3><ul><li>It is robust to multicollinearity among predictors.</li><li>It takes care of missing data effectively.</li><li>It reduces overfitting problem of decision trees.</li><li>It is useful for both classification and regression problems.</li><li>It doesn’t require us to normalize values in our dataset, since it is rule-based search.</li></ul><h3><strong>Disadvantages:</strong></h3><ul><li>Large number of trees might slow down the algorithm’s speed and thus, limit its usage in real-time applications.</li><li>It makes it harder for us to interpret each decision, which was a great advantage in case of decision trees.</li><li>It requires a lot of time for training multiple trees and requires more computational power to build them up.</li></ul><p>************************************************************</p><p>So, that wraps up the blog. The decision tree part of this blog might seem to be quite a mouthful. And I totally agree with it. It took me a lot of time to understand the algorithm and implementation. And I hope that this blog has helped you (even by a little bit) to understand the working behind decision trees and random forest.</p><p>In no way, I am saying this is a complete guide. All I can say is I tried.</p><p>So, thank you to all the readers who took the time of their day to read this blog. Have a wonderful day 😄!!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d7398d56b01c" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/a-noobs-guide-to-random-forest-d7398d56b01c">A Noob’s Guide to Random Forest</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Intern Diaries: Mathematics for ML]]></title>
            <link>https://medium.com/analytics-vidhya/intern-diaries-mathematics-for-ml-138c3d81b4c9?source=rss-5659a588d90b------2</link>
            <guid isPermaLink="false">https://medium.com/p/138c3d81b4c9</guid>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[calculus]]></category>
            <category><![CDATA[probability]]></category>
            <category><![CDATA[beginner-guide]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Tanya Gupta]]></dc:creator>
            <pubDate>Wed, 27 Jan 2021 11:12:57 GMT</pubDate>
            <atom:updated>2021-01-27T16:56:12.439Z</atom:updated>
            <content:encoded><![CDATA[<p>Today’s blog will give you a warm-up for mathematical concepts involved in machine learning. So, let’s relive those days in high-school where you absolutely “adored” mathematics and wondered how learning this stuff would be useful in life.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*z7JgaNFrSnPM-Pfe.jpg" /><figcaption>source: plus.maths.org</figcaption></figure><h3>Why do we need to learn maths?</h3><p>Firstly, what people need to understand is that knowing libraries required for machine learning algorithms is not enough. In order to improve your implementation and grip on machine learning, you need to understand how an algorithm works and that’s where mathematics comes in.</p><ul><li>It helps you understand how certain factors,parameters and features affect the results the way they do.</li><li>It helps you to choose the correct metrics to evaluate the working of your implementation.</li><li>It helps in expressing a relationship between the response and predictor variables (well, at least in supervised learning) and analyzing how much your model “fits” with the dataset.</li></ul><p>In conclusion: high school mathematics wasn’t useless after all…</p><p>Now that we have taken our time to appreciate the beauty of mathematics, let’s get into the meat of the matter.</p><h4>The four key areas to focus on for machine learning:</h4><ul><li><strong>Linear Algebra</strong></li><li><strong>Statistics</strong></li><li><strong>Calculus</strong></li><li><strong>Probability</strong></li></ul><p>This blog will introduce you to calculus and probability. You can find my blogs for linear algebra and statistics below:</p><ul><li><a href="https://medium.com/analytics-vidhya/intern-diaries-linear-algebra-24b996b0806c">https://medium.com/analytics-vidhya/intern-diaries-linear-algebra-24b996b0806c</a></li><li><a href="https://medium.com/analytics-vidhya/intern-diaries-statistics-67c7d2e75192">https://medium.com/analytics-vidhya/intern-diaries-statistics-67c7d2e75192</a></li></ul><h3>Multivariate Calculus</h3><p>Instead of judging the effect of a single factor on our output variable, we are looking at multiple factors in multivariate calculus. Calculus helps in optimizing our implementation of the machine learning models.</p><p>Most of the time we use differentiation in machine learning for example during gradient descent ( used to find the minima in our loss function in linear and logistic regression).</p><h4>Differentiation</h4><p>It is the subfield of calculus that helps us in finding out how much sensitive our output variable y is w.r.t multiple factors x₁,x₂,…,xₙ where y = f(x₁,x₂,…,xₙ). If multiple factors confuses you, then just understand that w.r.t one factor differentiation means,</p><p>(change in y)/(change in x) = ∆y/∆x where, y = f(x) = function of x</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/286/0*xKGIyBtKuQf-Xnm8.gif" /><figcaption>∆y/∆x</figcaption></figure><p>The above image shows us the amount of change in y (∆y) given the change in x (∆x). For example, the change in distance w.r.t the change in time gives us speed. In a way, speed is derivative of distance w.r.t time.</p><p>If the change in x is <em>infinitesimally</em> small ( ∆x -&gt;0 ) then, we call the above expression as a derivative and it is then represented as dy/dx.</p><p>In other words, derivative gives us the instantaneous change in y w.r.t x which is slope to the tangent of the curve y = f(x) at point x:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/243/0*iMxFOTOqiapiPDIc" /><figcaption>derivative depicted as a slope of tangent at the point A . In this case ∆x = h. So for a derivative, h tends to 0.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*xnxlhmVV_HCIXP5emTP1qg.png" /></figure><p>In case of multivariate differentiation, we see the change in our function f(x₁,x₂,…,xₙ) w.r.t any one of the factors x₁,x₂,…,xₙ by using partial derivative.</p><h3>Quick refresher for differentiation</h3><p>We will first learn w.r.t univariate calculus and then later extend it for multivariate calculus.</p><h4><strong>List of derivatives for different y</strong></h4><ul><li>y = x, <strong>dy/dx = 1</strong></li><li>y = constant, <strong>dy/dx = 0</strong></li><li>y = xⁿ, <strong>dy/dx = n*xⁿ⁻¹</strong></li><li>y = a*xⁿ, <strong>dy/dx = a*n*xⁿ⁻¹</strong></li><li>y = eˣ, <strong>dy/dx = eˣ</strong></li><li>y = log(x), <strong>dy/dx = 1/x</strong></li><li>y = x⁻ⁿ, <strong>dy/dx = -n*x⁻⁽ⁿ⁺¹⁾</strong></li><li>y = 2ˣ, <strong>dy/dx = 2ˣ(log 2)</strong></li><li>y = cos(x), <strong>dy/dx = -sin(x)</strong></li><li>y = sin(x), <strong>dy/dx = cos(x)</strong></li></ul><h4><strong>Sum rule for differentiation:</strong></h4><p>If y = first + second then, dy/dx = d(first)/dx + d(second)/dx</p><p>For example let, y = x³ + 2x⁵ then,</p><p>dy/dx = d( x³ + 2x⁵)/dx = d(x³)/dx +d(2x⁵)/dx</p><p>dy/dx = 3x² + 2*5*x⁴</p><p>dy/dx = 3x² + 10x⁴</p><h4><strong>Product rule for differentiation:</strong></h4><p>If y = first*second then dy/dx = (second)*d(first)/dx + (first)*d(second)/dx</p><p>For example, y = x* (cos(x)) then,</p><p>dy/dx = cos(x) * dx/dx + x* d(cos(x))/dx = cos(x) + x(-sin(x))</p><h4><strong>Chain rule for differentiation:</strong></h4><p><strong>Let’s understand with an example of y = sin(x²).</strong></p><ol><li>dy/dx = d(sin(x²))/dx = dsin(z)/dx where, <strong>z= x²</strong>.</li><li>First, we find the derivative of sin(z) w.r.t z= <strong>cos(z)</strong></li><li>Then, we differentiate z(= x²) w.r.t x which gives us <strong>2*x</strong></li><li>d(sin(x²)) = product of steps (2) and (3) = 2*x*cos(z)=<strong> 2*x*cos(x²)</strong>.</li></ol><p>In short, d(sin x² )/dx = d(sin z)/dz * dz/dx</p><p><strong>Another example:</strong></p><ol><li>Let y = (sin x)² then, we assume that<strong> z = sin(x)</strong></li><li>We find derivative of y (= z²) w.r.t z which is <strong>2*z</strong></li><li>We find derivative of z w.r.t x now, which gives us = <strong>cos(x)</strong></li><li>dy/dx = dy/dz * dz/dx = 2*z * cos(x) = <strong>2* sin(x)*cos(x).</strong></li></ol><p>Extending the above concepts to multivariate calculus we get the concept of partial differentiation.</p><h4>Partial differentiation and how to find it</h4><p>Partial differentiation can be understood with the example of housing prices. We can see that house prices are affected by a lot of factors like number of rooms ,number of floors, total area etc.</p><p>But if we want to find out how much a particular factor (say, total area) affects our house price (given everything else is the same) then, we see the change in prices w.r.t the change in total area. This effect by a factor can be found out using partial differentiation.</p><p>It’s very simple to calculate if you know differentiation. Let’s assume we differentiate f(x₁,x₂,…,xₙ) w.r.t x₁. All you gotta do is assume that all independent variables are constants (except for x₁ in this case) and only differentiate the terms containing x₁. For example,</p><p>y = 3*(x₁)² + 4*x₂</p><p>so we partial differentiation w.r.t x₁ would be like,</p><p>∂y/∂x₁ = ∂(3*(x₁)² + 4*x₂)/∂x₁</p><p>= ∂(3*(x₁)²)/∂x₁ + ∂(4*x₂)/∂x₁ = 6*x₁</p><p>the result of first term would be 6*x₁ and second term would be 0. Let’s take another example.</p><p>Let y = 3*(x₁)² + 4*x₂*x1. Then,</p><p>∂y/∂x₁ = ∂(3*(x₁)² + 4*x₂*x1)/∂x₁ = ∂(3*(x₁)²)/∂x₁ + ∂(4*x₂*x1)/∂x₁</p><p>∂y/∂x₁ = 6*x₁ + 4*x₂</p><h3>Probability</h3><p>It is the branch of mathematics which deals with calculating the likelihood of an event occurring. The value of this likelihood lies between 0 and 1 where, 0 defines the event not occuring at all while 1 defines that the event would definitely occur.</p><p><strong><em>Probability of event occurring = desired outcome/ total outcomes</em></strong></p><p>All probabilities sum up to 1.</p><p>For example when we toss a coin,</p><p>the probability of getting a head = the probability of getting a tail = 1/2.</p><p>This can be found as a coin has only 2 sides and thus, total outcome = 2 while desired outcome for both of these cases = 1 which gives us 1/2 when plugging these values in above formula.</p><h4>Important Terms</h4><ul><li><strong>Random Experiment:</strong> a process whose outcome is “random” or uncertain.</li><li><strong>Sample Set:</strong> set of all possible outcomes for that random experiment.</li><li><strong>Event: </strong>outcome(s) for a given experiment.</li></ul><h4>Types of events</h4><ul><li><strong>Joint events (A∩B):</strong> two events (A and B) can have common outcomes. For example, we roll two dice. Let event A be getting an even number and event B be getting a number smaller than 5. Outcomes for event A:{ 2, 4, 6 } and that of B: {1, 2, 3, 4 } so A∩B = {2, 4}.</li><li><strong>Disjoint events:</strong> They do not have any common outcomes. Taking the dice example, if event A is getting odd number and B is getting even number then, A∩B = ∅.</li><li><strong>Independent events: </strong>The outcome of first event doesn’t affect the outcome of the second event and vice-versa. The difference between independent and mutually exclusive events is that, for mutually exclusive two events can’t take place at the same time.</li></ul><p>For example, on tossing a coin, we can’t get a head and a tail at the same time…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/0*vN56KrmLzGhC904U.jpg" /><figcaption>Source: imgflip.com</figcaption></figure><p>Welp, nevermind then. As for independent events, even if they did occur at the same time, their respective outcomes don’t affect each other.</p><h4>Types of Probabilities</h4><ul><li><strong>Marginal probability: </strong>It gives the probability of a single event with no strings attached i.e., we don’t care about the outcome of some other event. For example, probability of choosing a king from a deck of cards = 4/52=1/13.</li><li><strong>Joint probability:</strong> It gives the probability of two events occurring simultaneously. For example, when rolling a dice we want to find probability of getting an even number which is less than 5. The two events here are- getting even number(event A) and getting number &lt;5 (event B). So, P(A∩B) = 2/6 = 1/3.</li><li><strong>Conditional probability: </strong>probability of an event given some other event has already occurred. For example, probability of a kid playing outside when it is raining. Here event A is “playing outside” and B is “raining”. Conditional probability is denoted as P(A|B) or P(“playing outside”|“raining”) for our example.</li></ul><h4>Taking a detour to learn conditional probability:</h4><p>If the two events A and B are dependent then their conditional probability is,</p><p><strong>P(A|B) = P(A∩B)/P(B)</strong></p><p>If A and B are independent events (you could multiply the probabilities of A and B):</p><p><strong>P(A|B) = P(A)*P(B)/P(B) = P(A)</strong></p><h4>Bayes Theorem</h4><p>It gives us a way of finding conditional probability with the following formula:</p><p><strong>P(A|B) = P(B|A)* P(A)/P(B)</strong> here,</p><ul><li>P(B|A) gives us the <strong>likelihood ratio</strong></li><li>P(A) is known as <strong>prior</strong> and P(A|B) is known as <strong>posterior value</strong></li><li>P(B) is known as <strong>evidence</strong> (many times it is not given to us)</li></ul><p>Evidence can be calculated using the following equation,</p><p><strong>P(B) = P(B|A)*P(A) + P(B|not A)*P(not A)</strong></p><h3>Probability Density Functions(PDF)</h3><p>This is nothing but the graphical representation of probabilities and probabilities can now be described as a function. PDF has the following properties:</p><ul><li>The area under the curve = 1</li><li>The values are all continuous.</li><li>The probability that a random variable assumes a value between x1 and x2 is the area under the curve bounded by x1 and x2 from the sides. This can be found by integrating the pdf from x = x1 till x = x2 as given below.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Z1L-qBEgdvVO-2yC.png" /></figure><h4>Binomial distribution</h4><p>It gives the probability of success or failure of an event which is being repeated multiple times. The formula for finding probability here is given by,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/517/1*EnGrL6YKb1uRPSmbj4IgXQ.png" /></figure><p>For instance, if a student has given 10 mock tests then, what is the probability that this student passes 8 tests?</p><p>So, for this we have x = 8, p = 0.5( because, a student can either pass or fail exam), n = 10. We plug in these values to our formula,</p><p>P = ⁿCₓ .pˣ.(1-p)ⁿ⁻ˣ = ¹⁰C₈.(0.5)⁸.(1–0.5)² = 45*(0.5)¹⁰ = 0.0439</p><p><strong>Conditions for this distribution:</strong></p><ul><li>The number of trials is finite and fixed.</li><li>The events must be independent.</li><li>The probability of success must be same for each and every event.</li></ul><p><em>Details about Normal and standard distribution curve is given in my statistics blog.</em></p><h3>Central Limit Theorem</h3><p>It states that,</p><blockquote>The sampling distribution of the mean of any independent, random variable will be normal or nearly normal, if the sample size is large enough.</blockquote><p>In other words, if we take a sample size which is large enough from our population, then the sample means (mean of each sample) will be approximately normally distributed. Increasing our sample size leads our probability distribution of sample means to look more and more like normal distributions.</p><p><strong><em>As a rule of thumb, it is said that the sample size must be at least 30 for CLT to work.</em></strong></p><h3>What are it’s practical implications?</h3><p>It is useful in that even though your sample’s distribution is in any random shape, your sample mean distribution would be nearly a gaussian curve(given the above sample size limit).</p><p>This can be very useful in finding out nature of the population without accessing each and every data point from the population.</p><blockquote>The only condition CLT has, besides the sample size, is that the initial distribution must be such that we are able to find out the sample’s mean. Cauchy distribution is one of those examples where, we can’t find out the sample mean.</blockquote><h4>Let’s take a real-world example:</h4><p>Suppose you are conducting a survey to determine the income of people throughout a state. However, we all know it ain’t feasible for us to ask each and every person in the state about their income. This is where we can apply CLT.</p><p>So, what we can do is take a few samples from some cities in the state. Here the samples would be the number of people from each of the cities. Given, this we can then, calculate the mean of each of these samples and plot a probability distribution curve out of this. We will be able to see then the resemblance of the curve to normal distribution and hence, use its properties to find the required data.</p><h3>Applications of Probability:</h3><ul><li>It helps us to optimize our model</li><li>Loss can be calculated using probability which further helps us in classifying data points correctly.</li></ul><p>*****************************************************************</p><p>That concludes our blog, I hope it helped you to understand these mathematical concepts (even if it is by a little amount). Thank you for reading my blog and have a wonderful day 😄!!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=138c3d81b4c9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/intern-diaries-mathematics-for-ml-138c3d81b4c9">Intern Diaries: Mathematics for ML</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Logistic Regression]]></title>
            <link>https://medium.com/swlh/understanding-logistic-regression-3cd1a69e070b?source=rss-5659a588d90b------2</link>
            <guid isPermaLink="false">https://medium.com/p/3cd1a69e070b</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[implementation]]></category>
            <category><![CDATA[logistic-regression]]></category>
            <category><![CDATA[from-scratch]]></category>
            <dc:creator><![CDATA[Tanya Gupta]]></dc:creator>
            <pubDate>Sun, 24 Jan 2021 16:57:48 GMT</pubDate>
            <atom:updated>2021-01-25T00:20:35.939Z</atom:updated>
            <content:encoded><![CDATA[<p>Today we will be learning about a probabilistic classification algorithm known as logistic regression and its implementation.</p><figure><img alt="Logistic Regression： Concept &amp; Application | SunJackson Blog" src="https://cdn-images-1.medium.com/proxy/0*dvLU5DDWSnHdQqdC.jpg" /><figcaption>Source: dimensionless.in</figcaption></figure><h3>Introduction</h3><p>Logistic regression comes under the supervised machine learning algorithms. Its name is a bit misleading as it is used for <em>binary classification</em> however, the regression part stems from its similarity to linear regression. As we have learnt, linear regression involves around finding out a regression line which “fits” our dataset. For this we find out the parameters for our line which gives the minimum value of loss function. Logistic regression is pretty similar to that. However, it includes just one more thing, namely sigmoid or logit function which takes a value and results in a number lying between 0 and 1.</p><h4>But then why use logistic and not linear regression?</h4><p>This is because the regression line can be affected a lot due to outliers. Let’s take an example. In the image given below, without outliers the regression line is pretty acceptable. If we take a threshold value like y=0.5 and if we get a y value ≥0.5 then it is classified in the “Malignant” group else, it would be classified as “Benign”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*gjLV_NjRAaLgYOuXjTeF4Q.png" /></figure><p>But with introduction of an outlier, we see that in order to decrease the loss function and “fit” with dataset better, the regression line must adjust itself. This causes some points to be misclassified as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1003/1*2GqaVg3wwlLJTnQgzr5lrA.png" /></figure><p>We can see that the points circled in blue are now, classified incorrectly. Another flaw here is, that linear regression might result in y value &gt;1 or &lt;0. In both cases we won’t be able to determine exactly in which class should our data point lie in.</p><h3>How logistic regression works?</h3><p>Apart from the procedure followed in linear regression, we have a small addition in case of logistic regression known as sigmoid function which helps in normalizing the dot product of θ (the line parameters) and x (data points) . This is important because we have seen above that outliers might drastically change the line and it may lead to mis-classifications.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AHHd73qUwTxa-KQ2.png" /><figcaption>Sigmoid function (Source: oreilly.com)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*z2N-W48P-DOPB7Ag.png" /><figcaption>Graphical representation of the sigmoid function (Source : en.wikipedia.org)</figcaption></figure><p>where, <strong>z = θᵀx </strong>.</p><p>But the above function σ(z) lies between 0 and 1 thus, nullifying any effect that the outliers might have.</p><p><strong><em>So, the hypothesis of logistic regression (hθ(x) )is σ(z) which gives the probability of our data point belonging in a certain class.</em></strong></p><p>After calculating the probability from σ(z), we use a threshold value (= 0.5, generally) to ultimately predict where our data point lies. So for σ(z)≥ 0.5, the data point belongs to the class y =1. Else, it belongs to y=0.</p><p>This also implies that, σ(z)≥ 0.5 =&gt; z (= θᵀx)≥ 0 and similarly, σ(z)&lt; 0.5 if z &lt; 0.</p><h4>The Cost Function</h4><p><strong>Cost = -y*log(hθ(x))-(1-y)*log(1-hθ(x))</strong></p><p>The above function gives a convex shape which makes it easier for us to use gradient descent to find the convergence (point where cost function is minimum). Notice if y=1, then the second term vanishes while for y=0 the first term vanishes.</p><p><strong><em>For every right classification cost = 0 while for wrong ones it penalizes with cost tends to ∞. This happens as for y = 1 and hθ(x) is very close to 0 then, log(0)-&gt;∞. Similarly, for y = 0 and hθ(x) very close to 1 then, log(1-hθ(x))≈log(0)-&gt;∞.</em></strong></p><h4>Applying gradient descent</h4><ul><li>Partial differentiating the cost function w.r.t θ will result in,</li></ul><p><strong>∂ (cost)/∂θ = [hθ(x)-y]*x</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZX6p5NKRQ1s-75gJRp-NHQ.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QR4GJzN0fbzZ9sGjDR_cDA.jpeg" /><figcaption>derivation for the above expression for the curious.</figcaption></figure><ul><li>Update θ for all the features from 1≤j≤n ( α is the learning rate) until we achieve convergence:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/0*JQN8JWASqpUnqEtL.png" /><figcaption>Source: stats.stackexchange.com</figcaption></figure><h3>Implementation on breast cancer dataset</h3><p>You can find the dataset <a href="https://github.com/tanya12181/Machine-Learning-Internship-at-Internity/blob/main/Task%2010/Breast_cancer_data%5B1%5D.csv">here</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fa537df9bb088081c3880e82f7a0e937/href">https://medium.com/media/fa537df9bb088081c3880e82f7a0e937/href</a></iframe><ul><li>Import numpy, matplotlib.pyplot , pandas libraries.</li><li>Convert the dataframe to numpy array. I did this so that I can shuffle up the rows of the dataset by using np.random.shuffle(dataset).</li><li>I split my dataset in 75: 25 ratio for train and test set. Only after splitting should we normalize our data. This is because, if we find the minimum and maximum values of the entire dataset and use these to normalize the dataset then, we might leak some of the test data to our train data when we split it. Also, remember to use the same minimum and maximum value of training data to normalize the testing data.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/508/1*8pjnSJDZg3HsZS1KShAd-A.png" /><figcaption>normalization of dataset</figcaption></figure><ul><li>I concatenate a column of ones (as the first column) to our training and testing set for the intercept value in the line equation y = θ₁ + θ₂*x+…+θₙ*x.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/1*_N_ZJyNIYcoyUNn-sChvRw.png" /></figure><ul><li>The gradient descent function is given below</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/543/1*vU4L__ei6HtthxdbA8Z1kQ.png" /></figure><p>alpha and epoch are the learning rate and number of iterations respectively.</p><p>r and c store the number of rows and columns for the dataset X respectively.</p><p>theta (or parameter vector) is initialized with ones and it has shape (c,1).</p><p>I have initialized min_cost with an empty list and min_theta with None. These variables store the minimum cost possible during the iterations and min_theta stores the parameters corresponding to the minimum cost.</p><p>At each iteration, we calculate the hypothesis using the functions below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/306/1*gl42dY-63DL3g8NboMtHNw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/346/1*e974VgBdsqbQkp5RL65xeQ.png" /></figure><ul><li>gradient or partial derivative is found by finding the dot product between the transpose of dataset(X) and (h-y) where, h is hypothesis and y is the response variable (for our dataset, it is “diagnosis”). The dot product is then divided by the number of rows.</li><li>Then we update each theta by subtracting alpha*gradient.</li><li>I find the value of cost function at each iteration for comparison and to find the theta having minimum cost using the function below:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/585/1*HOWXtx-BJ_xuAtoXcaz_cA.png" /></figure><p>To predict for the values in our test set:</p><p>First find the hypothesis using the theta found using training dataset and the test dataset. This will give you the probability for each row in our testing data. Then for each row, check whether it is greater than or equal to our threshold value (i.e., 0.5). If it is then the predicted value is 1 else, it is 0.</p><p>For every match between our predicted and actual y label, increment the correct variable and then find the percentage by dividing the value stored in the correct variable with the number of rows in our test set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/539/1*20NLEs4e0UVgOxt_Rr1bDw.png" /></figure><p>So, that wraps up the blog. Hope you have learnt something out of this.</p><p>Thank you for taking some time off your day to read this blog. Have a wonderful day 😄!!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3cd1a69e070b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/understanding-logistic-regression-3cd1a69e070b">Understanding Logistic Regression</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Noob’s Guide to Naive Bayes]]></title>
            <link>https://medium.datadriveninvestor.com/a-noobs-guide-to-naive-bayes-2ef0d8c131a6?source=rss-5659a588d90b------2</link>
            <guid isPermaLink="false">https://medium.com/p/2ef0d8c131a6</guid>
            <category><![CDATA[beginner-guide]]></category>
            <category><![CDATA[classification-algorithms]]></category>
            <category><![CDATA[naive-bayes]]></category>
            <category><![CDATA[from-scratch]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Tanya Gupta]]></dc:creator>
            <pubDate>Sat, 23 Jan 2021 12:21:07 GMT</pubDate>
            <atom:updated>2024-12-14T13:29:28.840Z</atom:updated>
            <content:encoded><![CDATA[<p>Today, we will learn about the competence of a very “naive” algorithm known as Naive Bayes. We would also be discussing its implementation (from scratch) on a dataset containing both categorical and continuous values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2GIyGE5-ezaw6JXd.jpg" /><figcaption>Source:analyticsprofile.com</figcaption></figure><h3>Introduction</h3><p>Naive Bayes is a supervised learning algorithm used for classification problems. It comes from the family of “probabilistic classifiers” and applies Bayes theorem for classifying objects with the assumption that all the features are independent of each other. This is the assumptions that makes it “naive” because in real life, it is highly unlikely to find datasets which has no multicollinearity or zero correlation amongst the predictors.</p><p>Despite this assumption, it does a pretty good job of classifying objects.</p><p>So before we implement our naive bayes algorithm, we should first understand what Bayes Theorem is all about.</p><h3>Bayes Theorem</h3><p>Before we learn about Bayes theorem, we need to understand the concept of conditional probability.</p><h4>Conditional Probability:</h4><blockquote>Conditional probability means the probability of an event occurring given that some condition is true. It is represented as P(event|condition).</blockquote><p>Let’s understand it using a “very sophisticated” example.</p><p>Suppose a person wants to punch a wall (I don’t know why though but moving on). What is the probability that it would hurt given he punches the wall? Here, the event of which we find the probability is “hurt hands” and the condition is “punching the wall”.</p><p>Our example would be represented as P(“hurt hands”|“punching a wall”).</p><blockquote>It should be noted that,</blockquote><blockquote>P(“hurt hands”|“punching a wall”) ≠P(“punching a wall”|“hurt hands”)</blockquote><blockquote>The probability of a person hurting his hand after punching a wall is not the same as the probability of him punching a wall while his hand is hurting. (Just read it a few times and it would make sense to you)</blockquote><h4>Coming back to Bayes Theorem…</h4><p>This theorem was introduced by this gentleman below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/225/0*bEWsMGdHhVcd4Cu0.gif" /><figcaption>Thomas Bayes</figcaption></figure><p>It gives us a way for calculating the conditional probability.</p><p>Mathematically this can be described as,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*B4yufzfUN6pD9Ov6.jpg" /><figcaption>Source: iq.opengenus.org</figcaption></figure><p>Many times, we are not given the value of the denominator P(B)<em>(this term is also known as evidence)</em>. We can calculate this by using the following equation:</p><p>P(B) = P(B|A)*P(A) + P(B|not A)*P(not A)</p><p><strong>Note that:</strong></p><ul><li>P(not A) = 1-P(A)</li><li>P(B|not A) = 1-P(not B|not A)</li></ul><p>Using the above equation, we have a new formulation of the Bayes theorem as,</p><p>P(A|B) = P(B|A)*P(A)/(P(B|A)*P(A) + P(B|not A)*P(not A))</p><h3>Working of Naive Bayes using an example:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/378/0*R8_mVmVaq81uShm2" /><figcaption>Train set</figcaption></figure><ul><li>Find different classes/groups in our response variable. In our example, the response variable is “flu?” and it is divided into 2 classes: “Yes” and “No”.</li><li>For each predictor variable, we determine the different types of values it has. For example, for “chills” , “fever” and “runny nose” columns they have values of types: “Yes” and “No”; for “headache” column it has “Mild”, “No”and “Strong”.</li><li>For each type of value in a particular predictor, we find out their “likelihood” of being classified into a certain class of response variable. For example,</li></ul><blockquote>In case someone has flu what is the probability that they have a mild headache or P(headaches= “mild”|flu = “yes”) ?</blockquote><blockquote>Total number of rows having (headaches = “mild” ) is 3. Out of these three rows, we see that 2 of them are classified into “yes” class in the “flu?” column. So, out of total yes values in “flu” column i.e., 5, we see that 2 entries has mild headache.</blockquote><blockquote>Hence, P(headaches= “mild”|flu = “yes”) = 2/5</blockquote><p>Similarly, we calculate the others as follows:</p><h4><strong>Given that flu = “yes” (likelihoods):</strong></h4><ul><li><em>P(headaches= “mild”|flu = “yes”) = 2/5,</em></li><li><em>P(headaches= “no” |flu = “yes”) = 1/5</em></li><li><em>P(headaches= “strong”|flu = “yes”)=2/5</em></li><li><em>P(chills= “yes”|flu = “yes”) = 3/5 and P(chills= “no”|flu = “yes”) = 2/5</em></li><li><em>P(runny nose= “yes”|flu = “yes”) = 4/5 and P(runny nose= “no”|flu = “yes”) = 1/5</em></li><li><em>P(fever= “yes”|flu = “yes”) = 4/5 and P(fever= “no”|flu = “yes”) = 1/5</em></li></ul><h4><strong>Given that flu = “no” </strong>(likelihoods)<strong>:</strong></h4><ul><li><em>P(headaches= “mild”|flu = “no”)</em> <em>= 1/3</em></li><li><em>P(headaches= “no”|flu = “no”)</em> <em>= 1/3</em></li><li><em>P(headaches= “strong”|flu = “no”)=1/3</em></li><li><em>P(chills= “yes” |flu = “no”) = 1/3 </em>and <em>P(chills= “no”|flu = “no”)</em> <em>= 2/3</em></li><li><em>P(runny nose= “yes” |flu = “no”) = 1/3</em> and <em>P(runny nose= “no”|flu = “no”) = 2/3</em></li><li><em>P(fever= “yes”|flu = “no”) = 1/3</em> and <em>P(fever= “no”|flu = “no”) = 2/3</em></li></ul><h4>Other probabilities:</h4><ul><li><em>P(flu = “yes”) = 5/8</em> and <em>P(flu = “no”) = 3/8</em></li><li><em>P(chills = “yes”) = 4/8 </em>and <em>P(chills= “no”) = 4/8</em></li><li><em>P(headache= “mild”) = 3/8</em> ; <em>P(headache= “strong”) = 3/8</em> and <em>P(headache= “no”) = 2/8</em></li><li><em>P(runny nose = “yes”) = 5/8 </em>and <em>P(runny nose = “no”) = 3/8</em></li><li><em>P(fever = “yes”) = 5/8 </em>and <em>P(fever= “no”) = 3/8</em></li></ul><h4>Example: To find out the likelihood of getting a flu given a runny nose (posterior) using Bayes Theorem</h4><ul><li><strong>P(flu= “yes”|runny nose= “yes”)</strong> =</li></ul><p>P(runny nose: “yes”|flu= “yes”)*P(flu: “yes”) /P(runny nose: “yes”)</p><p>= ((4/5)*(5/8))/(5/8)=4/5</p><h4>Now, we will predict whether for the given conditions we will have a flu or not</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/369/0*tQBHzuqv_JmEjrou" /><figcaption>Test case</figcaption></figure><p>Let the above condition be X. We need to calculate:</p><ul><li>P(flu = “yes”|X) = P(X|flu = “yes”)* P(flu= “yes”)/P(X)</li><li>P(flu = “no”|X) = P(X|flu = “no”)* P(flu= “no”)/P(X)</li></ul><p>Since P(X) is common just ignore it.</p><p><strong>Calculating P(flu= “yes”|X) = 5/8 * (3/5* 1/5 * 2/5 * 1/5) = 0.006 by multiplying the values given below:</strong></p><ul><li><em>P(flu = “yes”) = 5/8</em></li><li><em>P(chills= “yes”|flu = “yes”) = 3/5</em></li><li><em>P(runny nose= “no”|flu = “yes”) = 1/5</em></li><li><em>P(headaches= “mild”|flu = “yes”) = 2/5</em></li><li><em>P(fever= “no”|flu = “yes”) = 1/5</em></li></ul><blockquote>The above values was calculated using the equation below:</blockquote><blockquote>P(X₁, X₂, X₃,…Xₙ| “yes”) = P(X₁|“yes”)*P(X₂|“yes”)*P(X₃|“yes”)….*P(Xₙ|“yes”)</blockquote><p>Similarly we calculate for flu = “no”,</p><p>Calculating P(flu= “no”|X) = (3/8)*(1/3)*(2/3)*(1/3)*(2/3) = 0.01852 (approx)</p><ul><li><em>P(flu = “no”) = 3/8</em></li><li><em>P(chills= “yes” |flu = “no”) = 1/3</em></li><li><em>P(runny nose= “no”|flu = “no”) = 2/3</em></li><li><em>P(headaches= “mild”|flu = “no”)</em> <em>= 1/3</em></li><li><em>P(fever= “no”|flu = “no”) = 2/3</em></li></ul><blockquote>Since, P(flu= “no”| X) &gt; P(flu= “yes”| X), it is relatively unlikely that a person will get a flu for the given set of conditions.</blockquote><p>Oof that is a lot of calculations.</p><h3>Gaussian Naive Bayes</h3><p>All the calculations we have done for finding conditional probability is done for categorical data. The above won’t apply for continuous values for obvious reasons. This is where we introduce gaussian naive bayes.</p><p>If we want to calculate likelihood of a point given a condition y =c :</p><ul><li>First find out rows which satisfy y=c</li><li>calculate the mean and standard deviation for each feature or column for this given set of rows.</li><li>For each data point in this set of rows, plug in the value of x (which is the feature value of that data point), the column’s mean and standard deviation in the expression given below.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/0*_MDkD8Bg5VTg0uTL" /><figcaption>Source: <a href="http://www.machinelearningplus.com/">www.machinelearningplus.com</a></figcaption></figure><ul><li>These values for each column is then multiplied to find out the likelihood of that data point by applying , P(X|Y=c) = P(X₁, X₂, X₃,…Xₙ| Y = c) = P(X₁|Y = c)*P(X₂|Y = c)*P(X₃|Y = c)….*P(Xₙ|Y = c)</li></ul><p>where, X={X₁, X₂, X₃,…Xₙ} are the features/independent variables.</p><h3>Implementation on Penguins dataset</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e23c20bc73b0fdd073cb7c51528e6767/href">https://medium.com/media/e23c20bc73b0fdd073cb7c51528e6767/href</a></iframe><ul><li>Importing numpy, math and seaborn libraries.</li><li>I have loaded the dataset “penguins” using seaborn.load_dataset(“penguins”).</li><li>For this dataset, our response variable is “species” and rest of the columns are feature values.</li></ul><p>Before I implement naive bayes on the dataset, I checked for null values:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/421/1*SQ_YhkSIk522NNQNpTH2JA.png" /></figure><p>Some rows had 5 values as Null and some rows only had the values under “sex” column as null.</p><ul><li>First, I found out the mode value in the “sex” column and replace null values in this column with this value. In my case, mode was “Male”.</li><li>If some of the rows still have null values, then I dropped it.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/539/1*biIDlHR38YH9Wjvhjv-04w.png" /><figcaption>cnt stores the mode of a column and fillna(cnt) fills the null values.</figcaption></figure><p>I divided the dataset in a 80:20 ratio for training and testing dataset after converting dataset into a numpy array (I just find it easier to work with) :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/873/1*efG4m_es_OCW_0KNTf8-Qg.png" /></figure><h3>The functions</h3><p>We find the mean,standard deviation for continuous data and likelihoods for categorical data. These values are then used on testing data to find the predicted values.</p><h4>stats_by_class(x,y):</h4><p>divides dataset by class type of response variable and then returns the mean, standard deviation for continuous data and finds log likelihood for categorical data.</p><p>arguments: x_train set and y_train set.</p><ul><li>It first finds out all the unique class types for our response variable.</li><li>Initializing dictionary “groups” for each class type with all the rows which has y = &lt;Class_name&gt;.</li><li>“stats” stores the mean and standard deviation (both for each column), column index ( to identity columns with continuous values) and number of rows for each class.</li><li>“category_prob” stores the log likelihoods for the columns containing categorical data.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/651/1*5df1AitLep7M6zehdxjvgQ.png" /></figure><h4>find_stats(x):</h4><p>argument: x is the dataset whose corresponding response variable belongs to a particular class type.</p><p>It checks each column whether they contain data of string class or not. If it is not of string type then, we find mean and standard deviation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/515/1*AduSvZEb0nCOZ8hbTAp2lQ.png" /></figure><h4>gaussian_prob(x_i,mean,std):</h4><p>finds the probability using gaussian distribution expression for x_i where, x_i is equivalent to x[row][column] and mean and standard deviation is passed for that particular column.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*12uFyvfomsrqp5TET0NM4A.png" /></figure><h4>find_category_prob(x,y,cls):</h4><p>argument: here, cls is class type.</p><p>finds the log-likelihood for all predictors given that it is classified under a class type of response variable.</p><p>Again, we check for data type of the column, if it contains data of string class then, we find:</p><ul><li>unique types in that column.</li><li>find the rows in that column where y_train[row] = cls for each type</li><li>divide the above count with the total count of that type in that column.</li><li>Then find its log and store it in probabilities dictionary where key value is the column index.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/1*bIi59n8SHNR4uoZ4GSt2ow.png" /></figure><h4>find_prob_for_class(stats, test_row, category_prob=None):</h4><p>This function finds the posterior values for P(Y=cls|X) where X is the set of conditions presented by our test row.</p><p>We call the gaussian_prob() to find likelihoods for continuous columns and find its log value. We add all these log values with log likelihood values provided by category_prob dictionary and the log value of our class probability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*tbMTduNYfOywrYDG4D3bvA.png" /><figcaption>stats[class][0] contains mean for each column, stats[class][1] contains standard deviation for each column, stats[class][2] has column indices, stats[class][3] has number of rows for that class.</figcaption></figure><p>we find probability of that class by dividing stats[cls][3] with total_rows in the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/1*Dr9IO9I3y7rTikL0nbWE0A.png" /></figure><h4>predict(stats, test_row, category_prob=None):</h4><p>It gets log value of posteriors from find_prob_for_class() function and returns the class which has the maximum posterior value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*emzUU_C5yZsuI4sqNKp7_g.png" /></figure><p>Anyways, that wraps up our blog. I hope it gives you a bit of a head start about this topic. Thank you for taking your time to read this!!!</p><p>Have a good day 😄!!!</p><p>P.S. I am all ears for all the corrections and suggestions. Feel free to point out if I did anything wrong and how I can do better.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ef0d8c131a6" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/a-noobs-guide-to-naive-bayes-2ef0d8c131a6">A Noob’s Guide to Naive Bayes</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Noob’s Guide to SVM]]></title>
            <link>https://medium.com/swlh/a-noobs-guide-to-svm-e7c68854345a?source=rss-5659a588d90b------2</link>
            <guid isPermaLink="false">https://medium.com/p/e7c68854345a</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[implementation]]></category>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[from-scratch]]></category>
            <category><![CDATA[svm]]></category>
            <dc:creator><![CDATA[Tanya Gupta]]></dc:creator>
            <pubDate>Wed, 20 Jan 2021 13:58:28 GMT</pubDate>
            <atom:updated>2021-01-21T09:04:58.413Z</atom:updated>
            <content:encoded><![CDATA[<p>“Support Vector Machine” seems like handful of a name, but it is pretty simple if you understand it nicely. Today’s blog will give you a run through regarding what SVM is, how useful it is and how to implement linear SVM from scratch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*heqYTVcbrXcBDkeP.png" /><figcaption>Souce: medium.com</figcaption></figure><h3>Introduction</h3><blockquote>It is a supervised-learning algorithm used for classification and regression problems, primarily used for the former though. It can be used as a non-probabilistic linear classifier.</blockquote><p>This means that unlike logistic regression, it doesn’t find the probability of where a data point lies. It is more straightforward than that and just gives the class in which the data point belongs. That being said, it is pretty similar to our logistic regression in the sense that we still need to introduce a line (or a hyperplane for higher dimensions) to create a separation between the classes.</p><p>Another interesting thing is, it could also be used for non-linear classification with the help of kernels which will be touched upon later.</p><p>Before we move on, we get ourselves equipped with the necessary terms.</p><h3>Terminology</h3><ul><li><strong>Support Vectors:</strong> The points nearest to the hyperplane. They are used to define the margins for each class and also affect the position of the hyperplane.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/410/0*nXcqarwqKE1r-kUT.png" /><figcaption>The middle line here, is our decision boundary(hyperplane). The data points which pass through the hyperplane for their respective classes are the support vectors. The distance between these hyperplanes is the margin width.</figcaption></figure><ul><li><strong>Hyperplane: </strong>It is a subspace whose dimension is one less than that of the surrounding space. For instance, for a 2-D plane, this hyperplane would be a line. In SVM, this hyperplane acts as a best possible separator(or decision boundary) for different classes.</li><li><strong>Margin:</strong>It is the distance between a data point and the decision boundary.<strong> </strong>In SVM, we aim to increase this as much as we can.</li><li><strong>Kernels: </strong>These are mathematical functions which are used to process the input and convert it in a form so as to make classification easier. These are particularly useful in non-linear classifications. They convert the given n-dimensional space to a higher dimensional space to make classification easier for us. Examples: <em>linear, nonlinear, polynomial, radial basis function (RBF), and sigmoid.</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/0*U9-ucnz9XlTSnxbU.png" /><figcaption>The left image shows a 2-D plane and all the data points on it. If we use a linear classifier on such data, the accuracy would be very poor. To tackle this, we change our perspective of viewing the data points by converting this plane into 3D space. Here, defining decision boundary becomes a lot easier.</figcaption></figure><h3>Linear SVM</h3><p>We classify data which is linearly separable, or can be classified effectively using a straight line as our decision boundary. <em>Our aim (for SVM in general) here is to increase the margin width as much as we can.</em> The hyperplane is defined as,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/213/1*xshQ1PQQJQaMjSFw1-A96g.png" /><figcaption>Souce: en.wikipedia.org</figcaption></figure><p>where,</p><p>w is the set of parameters for our set of independent variables x and</p><p>b is the bias.</p><p>In order to understand the implementation we will go through more of the thesaurus. (I know its getting a bit verbose but bare with me)</p><h4>Idea behind maximizing margins:</h4><p>Imagine you are driving a car and there are many obstacles here and there. If these obstacles are closer to the area you are driving, there is a big chance for you to collide with them.</p><p>Now, take the lane you are driving as a decision boundary and all these obstacles as data points. The more closer your decision boundary is to the data points, the more chance there is for mis-classification or “collision” as far as our example goes.</p><p>In order to avoid such mistakes, we increase our “margin of error” .</p><h4>Types of Margins:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*h_PM_T4S_cwqyEIp.png" /><figcaption>source: wikipedia.org</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/0*rexrHQ4xVRuMura5.png" /><figcaption>source: medium.com</figcaption></figure><ul><li><strong>Hard margins: </strong>This kind of margin is very inflexible and intolerant w.r.t margin violations ( <em>case where</em> <em>a data point is classified within the margin) </em>and mis-classifications. There is a higher chance of over-fitting here.</li></ul><p>The equations for the said margins are,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*Y8Rufy0Sm1_VwIzsh9sLdA.png" /><figcaption>Equation 1</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/845/1*Cs3jbkextakOV-5M9QoTBA.png" /><figcaption>Equation 2 (Both images courtesy of en.wikipedia.org)</figcaption></figure><p>At the end of both equations, it could be any value a and -a respectively. What the above equations convey is that 2 classes are defined as 1 and -1. So, a correct classification is done if,</p><p>for class(y) = +1, our point lies above wᵀx-b or wᵀx-b≥1</p><p>for class(y) = -1, our point lies above wᵀx-b or wᵀx-b≤-1</p><p>We can also say,</p><p>yᵢ*( wᵀx-b)≥1 where yᵢ is the response variable for iᵗʰ row and 1≤ i ≤ n (total number of data points).</p><p>From the first image, we can also see that the distance between these two margins is 2/∥w∥ where, ∥w∥ is the norm of w. In order to maximize the margin width, we need to minimize ∥w∥ or ∥w∥². This width could be obtained by subtracting equations 1 and 2 with each other.</p><p><strong><em>So for hard margins, the optimization goal is to minimize </em></strong>∥w∥<strong><em>² for yᵢ*( wᵀx-b)≥1 for every 1≤i≤n.</em></strong></p><ul><li><strong>Soft margins:</strong> These margins are tolerant to a certain number of violations and mis-classifications. <em>This is generally used in cases where the data might not be linearly separable.</em></li></ul><blockquote>To achieve this we use a <strong>hinge loss function</strong>. It can be defined as,</blockquote><blockquote>max(0, 1- yᵢ*( wᵀx-b)).</blockquote><blockquote>For yᵢ*( wᵀx-b)≥1, the classification is correct and thus, above loss function gives us 0 penalty.</blockquote><blockquote>yᵢ*( wᵀx-b)&lt;1 gives us wrong classification and it gives us penalty. The more further our point is mis-classified from margin, the more penalty it would give.</blockquote><p>The optimization goal is to minimize,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/1*6WZcULf43FJAXvG1OLUYlA.png" /></figure><p>where, n is number of data points and λ is the regularization coefficient. λ provides a trade-off between increasing margin-size and correct classification of data points. So for smaller values of λ , we will have the same objective as the hard margin, which is to classify the data points as correctly as possible. It shifts its focus from maximizing margin width to increasing accuracy.</p><h3>Implementation</h3><p>I have implemented linear SVM with soft margins using gradient descent to optimize the cost function. You can find the breast cancer dataset <a href="https://www.kaggle.com/merishnasuwal/breast-cancer-prediction-dataset">here</a>.( In the dataset 1 is for benign and 0 is for malignant).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/144261a61d95292cc0b9a49c79537a6a/href">https://medium.com/media/144261a61d95292cc0b9a49c79537a6a/href</a></iframe><p>The given dataset was divided into train and test dataset in an 80 to 20 ratio. I normalized the data because this algorithm involves us to find out the distance between our support vectors.</p><p>Any algorithm, which involves us to find distance can be heavily affected if our dataset have difference in scale among its features. For example, we might have a feature which ranges between 0 to 1 and other might have it between 10 to 100. We use the same min and max parameters we used to normalize train set for our test set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/868/1*5-lZLUphe5QUDG39uHY5xw.png" /></figure><p>Remember, to change the labels of your dataset if necessary from 0 to -1. This is because we are using the inequality , yᵢ*( wᵀx-b)≥1 for checking our classifications. For this to work properly, we need to change our 0 label to -1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/504/1*b-G_gDj6m_J2xVPvbnY1iQ.png" /></figure><h4>Some common arguments:</h4><ul><li>x : the matrix all the independent variables or predictors of our dataset</li><li>y: it contains all the response values for each dataset.</li><li>alpha: it is the learning rate</li><li>epoch: number of iterations</li><li>reg: regularization coefficient(λ)</li><li>pred: prediction list</li><li>x_ : for x testing data</li><li>y_ : for y testing data</li></ul><h4>Functions:</h4><p><strong>grad_descent(x,y,reg, alpha,epoch):</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/669/1*GAA-sLwgn_yirnsIQ7V5uw.png" /></figure><p>We run epoch number of iterations, and find the values of our parameters w and b for which the cost is minimum. The cost/optimization function is the same as defined for soft margin. So,</p><p>if yᵢ*( wᵀx-b)≥1, then cost = λ*∥w∥² — — (1)</p><p>its derivatives would be,</p><ul><li>∂(cost)/∂wᵢ =2*λ*wᵢ</li><li>∂(cost)/∂b = 0</li></ul><p>else, cost = λ*∥w∥² + 1- yᵢ*( wᵀxᵢ-b) — —(2)</p><p>its derivatives would be,</p><ul><li>∂(cost)/∂wᵢ =2*λ*wᵢ -xᵢ*yᵢ</li><li>∂(cost)/∂b = yᵢ</li></ul><p>It calculates the above derivatives and find the cost at each iterations. We check for minimum cost each time and store the corresponding w and b values. It stores every costs in the cost_list. This function returns the optimized w and b and the cost_list.</p><p><strong>plotCost(cost): </strong>It plots a graph of cost v.s. number of iterations,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/1*6Qlm65aY7o1n-Rs2cHfL3g.png" /></figure><p><strong>calAccuracy(pred , y_ ):</strong> It takes our prediction list and the response values of our testing dataset. It gives % accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/401/1*kzaNh2U9Oca8nzB-BvIVcA.png" /></figure><p><strong>predict(x_ , y_ , w, b): </strong>It takes the x and y testing data along with our calculated w and b values. It performs a dot product of w and x_ and adds b to it. If this value is ≥1 then, we assign it to class = 1. Else, we assign it to class=-1.</p><p><strong>showPredictions(pred , y_ ):</strong> It prints our predicted and actual response values side by side.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*a9JnD8rEymQuyNKFyIefGw.png" /></figure><h3>Some Applications:</h3><ul><li>Face Expression Recognition</li><li>Image classification</li><li>Handwriting recognition</li></ul><h3>Advantages:</h3><ul><li>Suitable for datasets having more features than the training examples</li><li>Effective in a higher dimension</li><li>Works well with unstructured data like text and images.</li></ul><h3>Disadvantages:</h3><ul><li>Difficult to choose appropriate kernel</li><li>Feature scaling is required</li><li>Difficulty in interpreting the final model.</li></ul><p>That wraps up our blog. I hope you find this blog useful. Thank you for reading!!!</p><p>If you have any suggestions and corrections, I am all ears.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e7c68854345a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/a-noobs-guide-to-svm-e7c68854345a">A Noob’s Guide to SVM</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Noob’s Guide to K-Means]]></title>
            <link>https://medium.com/swlh/a-noobs-guide-to-k-means-215a600bd1eb?source=rss-5659a588d90b------2</link>
            <guid isPermaLink="false">https://medium.com/p/215a600bd1eb</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[unsupervised-learning]]></category>
            <category><![CDATA[k-means-clustering]]></category>
            <dc:creator><![CDATA[Tanya Gupta]]></dc:creator>
            <pubDate>Mon, 18 Jan 2021 22:48:21 GMT</pubDate>
            <atom:updated>2021-01-19T08:18:02.467Z</atom:updated>
            <content:encoded><![CDATA[<p>Today we will be learning together a very interesting algorithm that uses similarity among items to group them. The idea seems pretty similar to that of KNN algorithm but the algorithm is not. Let’s get started with the k-means algorithm</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*Edfide-ylgyTSgRr5jX4WA.png" /></figure><h3>What is clustering and what are its types?</h3><p>Clustering is a process of grouping up data points or items based on their similar features. It does not involve labelled data and it is left on the machine to find a pattern according to which we cluster. There are 3 types of clustering:</p><ul><li><strong>Exclusive Clustering:</strong> The clusters are non-overlapping ( example: K-means)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/439/0*zQkeCx6VezCR2t-s.jpg" /><figcaption>Source: analyticsprofile.com</figcaption></figure><ul><li><strong>Overlapping Clustering: </strong>some data points may belong to one or more clusters(example: Fuzzy or C-means clustering)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/447/0*axWe15JIcPvXI8vn.jpg" /><figcaption>Source: analyticsprofile.com</figcaption></figure><ul><li><strong>Hierarchical Clustering: </strong>creates an hierarchy of clusters. Creation of such hierarchy could be either in a bottom-up approach(agglomerative) or top-down approach(divisive).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/0*yBqC6ytiEP0YhwH4.png" /><figcaption>Source: <a href="https://en.wikipedia.org/wiki/Hierarchical_clustering">https://en.wikipedia.org/wiki/Hierarchical_clustering</a></figcaption></figure><h3>Introduction</h3><p>K-means clustering is an unsupervised learning algorithm. Unlike in KNN algorithm, we do not group them by choosing the group that majority of the K neighbours belong to. Instead, we choose random k centroids, find the data points near those centroid and group them accordingly. We will get more into this as we will move to its implementation.</p><p>Let’s take an example to understand the idea of clusters:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*7UhFo-zMaD5m7QYe.jpg" /><figcaption>Source:lbb.in</figcaption></figure><p>Imagine you are at a restaurant. You will see that a bunch of people are sharing a table. This suggests that these “clusters” of people, are familiar with each other more than the people sharing another table.</p><p>Another example could be that if you are in an “all-you-could-eat-buffet”, you would always see the dessert, main course and the appetizers separated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*_gl-FManlFFksIfb.jpg" /><figcaption>Source:theatlantic.com</figcaption></figure><p>From both of these examples, we can see some properties of the clusters formed via any clustering algorithm:</p><ul><li>Within a cluster, the data points are as similar to each other as possible.</li><li>If we compare each point within one cluster to each point within any other cluster, then, the data points should be as dissimilar as possible.</li></ul><h3>Working and Implementation:</h3><p>Steps to perform K-means:</p><ol><li>Assuming we have selected the number of clusters, k, we choose any k random positions as our centroids.</li><li>For each of our centroids, we find their distance from each of our data points. If a centroid has least distance from a point, that point is grouped with that centroid.</li><li>After grouping each point of our dataset, we find the mean of points in each of the clusters. This mean becomes our new centroid.</li><li>If our new centroid values and old centroid values are equal then, this means we have reached “convergence” or the process of clustering is over.</li><li>Else we would repeat steps 2 to 4 until we reach convergence.</li></ol><h4>Given below is the implementation on crime.csv dataset:</h4><p>You can find the dataset <a href="https://raw.githubusercontent.com/namanvashistha/doctor_strange/master/crime.csv">here</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ebc8524e77458040c1e900e4cc9e1512/href">https://medium.com/media/ebc8524e77458040c1e900e4cc9e1512/href</a></iframe><p>You might find this code implementation a bit intimidating but don’t worry I will explain each and every function in this code as best as I can. That being said, given below is the description of this code:</p><ul><li>We will be importing all the important libraries to process and visualize our data like as given in the first cell in our embed.</li><li>For clustering data, I found it easier to remove the columns “Date” and “Magnitude” as they didn’t seem to have any significance to our data. I just dropped the “Date” column here. Feel free to do so for both.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/1*pAmzIqAUEX2frmBuC7BL9w.png" /></figure><ul><li>For clustering algorithms, you might find scatterplot from seaborn or scatter() from matplotlib.pyplot a very convenient tool for visualizing data points and seeing how similar they are to each other.</li><li>After removing all the unnecessary columns, I converted our dataframe into a numpy array which I find easier to work with.</li></ul><p><strong>Some common arguments and their meanings:</strong></p><ul><li>df = dataset</li><li>k = number of clusters</li><li>centroid = center (or centroid) for each and every group. The number of centers is equal to k.</li></ul><h4>randomCentroid(df , k):</h4><p>The above function is used for finding the random centroids required for our first step. It uses uniform() function from the random library to generate a random float value in the range of the minimum and maximum elements of a particular column.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*XOXnqpnWWVNNEtI8IwtGyg.png" /><figcaption>I used df.shape[1]-1 here because I am only concerned with the first and second column and not the “Magnitude” column. If you are removing that column all together, just remove the -1 here.</figcaption></figure><h4>FindDistance( ):</h4><p>Generates the square of Euclidean distance. It takes 2 rows as the argument. In our case each row would only contain the elements of “Latitude” and “Longitude” columns.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/477/1*KYK-nNKAtnG70m6EbJiFmQ.png" /></figure><h4>group_data(df , k , centroid) :</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/542/1*ZlTlaOFhjnwNg_oXvB3ZWw.png" /></figure><ul><li>It groups the data points based on their closest centroid values.</li><li>For example, if a particular point is closest to the centroid associated with group 1 then, this data point would be added to the said group.</li><li>It creates a dictionary named “groups” which has the group number/label as the key and for each group it stores the row of all those data points which are closest to the associated centroids.</li></ul><h4>KMeans(df , k):</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/587/1*lelQJcc0TxesNb5ig5o6Cg.png" /></figure><ul><li><strong>cen:</strong> it is a list storing the initial set of randomized centroids for each group.</li><li><strong>ncen:</strong> stores the list of new centroids that we calculate by finding the mean of each point in each cluster.</li><li>If the values of ncen and cen are equal then, we have reached a point of convergence and we return the group details and the final centroid values.</li></ul><p>For k=5 we get the following result,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*IHLJrARMSgvsbtK-SRmoXg.png" /></figure><h3><strong>Applications:</strong></h3><ul><li>used in recommendation systems in online shopping and our “beloved” Netflix.</li><li>You can find dominant colours in an image</li><li>Is used in feature-learning step in either semi-supervised learning or unsupervised learning.</li></ul><h3>Advantages:</h3><ul><li>It works for dataset where the data points cannot be separated by a straight line.</li><li>It works for unlabelled data which is pretty common in the real-world.</li><li>It is pretty easy to implement.</li><li>It is pretty efficient on large datasets.</li></ul><h3>Disadvantages:</h3><ul><li>Finding the appropriate k value is difficult.</li><li>The clustering is dependent on the initial set of centroids especially for high values of k.</li><li>Outliers might affect the centroid positions which inadvertently affects the clustering.</li><li>If the shape of the clusters are complex, then k-means might not be able to give satisfactory results.</li></ul><p>That wraps our blog. Hope it helps you to understand k-means algorithm a little bit better. Thank you very much for reading!!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=215a600bd1eb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/a-noobs-guide-to-k-means-215a600bd1eb">A Noob’s Guide to K-Means</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Noob’s Guide to Decision Trees]]></title>
            <link>https://medium.com/analytics-vidhya/a-noobs-guide-to-decision-trees-171252c5b653?source=rss-5659a588d90b------2</link>
            <guid isPermaLink="false">https://medium.com/p/171252c5b653</guid>
            <category><![CDATA[decision-tree]]></category>
            <category><![CDATA[noob]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[basics]]></category>
            <category><![CDATA[beginner]]></category>
            <dc:creator><![CDATA[Tanya Gupta]]></dc:creator>
            <pubDate>Wed, 13 Jan 2021 20:03:29 GMT</pubDate>
            <atom:updated>2021-01-15T16:45:12.746Z</atom:updated>
            <content:encoded><![CDATA[<p>Greetings, my fellow noobs. Today I have compiled a blog for decision trees, a concept which at first glance may seem like a “tangled mess” but is much simpler than that. Here, we will grasp the idea and working of the decision tree and how it makes your life a tad bit easier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*3AK84eVYvmsRWlzd.jpg" /><figcaption>Source: better.sg</figcaption></figure><h3>Introduction</h3><blockquote>It is a supervised learning algorithm used in classification and regression problems. It uses its tree-like representation to give visuals of all the decisions that can be made given the problem.</blockquote><p>In very basic, noobish terms, a decision trees is a tree which represents all possible solutions to a problem. For example,</p><p>Suppose we are given a group containing birds , fishes, cats and dogs. We want to classify all these animals in their respective groups. So what questions can we ask to answer this problem?</p><p>We can first ask, “does the animal have fur?”. If it is true then, cats and dogs come in that group. The animals without fur would then have a seperate group containing birds and fishes.</p><p>Now, lets focus on animals with fur. We now ask, “whether that animal barks?”. If it does it is classified as a dog. If it doesn’t then it is a cat. Now, that this group is classified we move to the next one.</p><p>In case of animals without fur, we can ask, “whether that animal flies?” If it does then it is a bird, else it is a fish. This decision making can be described in the form of the tree given below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DFT6N3aAbBWAsTWnxh82GQ.png" /><figcaption>Source: wiki.tum.de</figcaption></figure><h4>There are two types of decision trees:</h4><ul><li><strong>Categorical Variable Decision Tree: </strong>The target value is a categorical value. For example, if we have a dataset for diagnosing diabetes then, the target value is either “yes” or “no” for diagnosis.</li><li><strong>Continuous Variable Decision Tree: </strong>The target value is a continuous value like in regression. For example, prediction of house prices based on the square feet of basement, first floor ,second floor, number of bathrooms, number of bedrooms etc.</li></ul><h3>Applications</h3><ul><li>It is used for data-mining and for classification in multivariate systems.</li><li>Used for developing prediction algorithms for the target variables.</li><li>Used in demographic data to find potential clients.</li><li>It helps you to map out different alternatives and choose the best one based on different situations.</li></ul><h3>Basic Terms</h3><ul><li><strong>Root node:</strong> This is where we start making decisions. It represents the entire dataset and then based on the selected question, we split it.</li><li><strong>Leaf node:</strong> This is where we reach a dead end. It can also be said that we cannot obtain any new information using such node and thus, can’t be divided.</li><li><strong>Pruning:</strong> process of removing unwanted nodes in a tree. It’s the opposite of splitting.</li><li><strong>Splitting: </strong>It is the process of division of our dataset based on some question or condition.</li><li><strong>Parent/child node:</strong> the node from which divisions are made is the parent node while, child node is the sub-node originating from the parent node.</li><li><strong>Branch or sub-tree: </strong>the result of splitting the tree/node.</li></ul><h3>How does it work?</h3><p><em>We will be implementing the </em><strong><em>ID3 algorithm</em></strong><em> for decision trees which uses entropy as one of its metrics. </em><strong><em>We will be discussing about the metrics for decision trees involving categorical target variables.</em></strong></p><p>Being a decision tree, it involves asking a lot of questions. But the real task is to find the order of questions that needs to be asked.</p><p><strong>How to select a question in such a way that making decisions further down the line becomes easier for us? </strong>For this we have to introduce certain <strong><em>metrics </em></strong>through which we could evaluate our questions and order them up.</p><p>But to understand these terms, we need to understand some more basic terms:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/0*Zma5Ea_Smz00ywe6.gif" /></figure><ul><li><strong>Impurity:</strong> it is a measure of how homogeneous our node labels are. Let’s take the same group we defined previously, as an example. If we were to put any label (e.g. cat) on any of these animals randomly(for e.g., fish) <strong>“what is the probability that it would be a wrong classification?” . This question defines the concept of impurity.</strong></li><li><strong>Entropy: </strong>It is the measure of impurity and defines randomness in a dataset. If some part of dataset falls under one category and other part falls in other category, so we can say our dataset is random as it becomes inconclusive for us to make any decision based on this. For example,</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VBCRudaHMqvFmf01.png" /><figcaption>Source: www. dataversity.net</figcaption></figure><p>In the above figure, we can see that labelling the first image (impure) into groups in tough compared to the second one (pure). It can be calculated as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/264/0*iHWSSXpBsms9tB71.png" /><figcaption>p is the probability of an item with a label i from i=1 to c where, c is the no. of groups</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/0*Cah6QBus9zll1h91.png" /></figure><p>If you want to make it more poetic then entropy means adding salt in distilled water and then segrating them into groups. It is dissolved so it ain’t gonna show differently by itself. You gotta work to separate them up!!!</p><p>Now that the thesaurus and my cringiness is over, we will be explaining the metrics for selecting the question based on an attribute,</p><ul><li><strong>Gini Index</strong>: It is a measure of impurity and its value lies between 0 and 1. It is calculated as,</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*Ct5iiyX9_hBH90Zo.png" /></figure><p>where, pi is the probability of an object classified into a group. Gini index only works for binary splits. Higher the value of Gini index, higher is the homogeneity. This metric is used in CART (Classification And Regression Tree) implementation for decision trees.</p><ul><li><strong>Information Gain</strong>: It gives a measure of how much information an attribute can give about a class/group. It specifies how much entropy can be decreased using an attribute.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nCRe7C6Ell3773n-.png" /></figure><p>As we go down our decision tree, we want to go from high entropy ( highly inconclusive) to a lower entropy ( where classification becomes straightforward). We can also see intuitively see that, if the weighted average of the entropy of children is less then, information gain must be high i.e., we have gained a lot of new information to classify items easily.</p><p>That being said, the attribute with the highest information gain is chosen first and the same process is repeated as we go down our decision tree.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gmjiRIAiNpbgtXuN.png" /><figcaption>(Source: towardsdatascience.com) Going from high to low entropy.</figcaption></figure><blockquote>Gini index is preferred more over entropy and information gain as it doesn’t require intense computations as in the case of both.</blockquote><h4>Now we will see the steps to implement decision tree:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/658/0*e57puoJopcq_aNZ4.png" /><figcaption>Our target variable here is ‘Return’. (Source:blog.quantinsti.com)</figcaption></figure><ol><li>For each attribute of the dataset we will find the information gain from it. For example, for the attribute “past trend” there are two types of values here i.e., positive and negative.</li><li>For each type we find out their respective classification based on the target variable. Here for positive type in “past trend”, we have 4 “up” and 2 “down”. For negative type, we have all values of “down”. So, negative type is homogenous.</li><li>For each type we will find the entropy and then find their weighted average. In this example, for positive value our entropy would be: (-4/6)log₂(4/6)+(-2/6)log₂(2/6) = (-2/3)log₂(2/3)+(-1/3)log₂(1/3)=0.3899+0.5283=0.9182 ; Similarly for negative we have entropy =0</li><li>Now, we find it’s weighted average which is : (6/10)*(0.9182) + (4/10)*0 = 0.5509</li><li>After finding this weighted average we subtract it from the entropy of our parent. For root node however we find entropy of the dataset. This is found by finding the probability of each type of values in the target variable and then applying entropy formula on each of those probabilities.</li><li>Here, for our target variable “return” there are 4 “up”s and 6 “downs”.</li><li>Applying entropy formula, (-4/10)*log₂(4/10)+(-6/10)*log₂(6/10) = 0.9709</li><li>Subtracting weighted average from entropy of dataset, we get 0.42 which is the information gained from “past trends”.</li><li>Steps (1) to (4) are performed for all other attributes and then we select the attribute with maximum information gain as the root.</li><li>For “Open Interest”, (-4/6)log₂(4/6)+(-2/6)log₂(2/6) =0.9182 entropy for value of type “low” and for “high” we have 1. Weighted average would be (6/10)*0.9182 + (4/10)*1 = 0.9509. Information gained would be 0.02. For “Trading value” we have information gained = 0.9709-0.6896=0.2813.</li><li>So, we choose the “past trends” as our root and split our dataset accordingly. We perform all these steps for the child nodes of “past trends” and build a tree until we either cannot obtain any information anymore from the nodes or, we have reached our maximum specified depth.</li></ol><h4>Advantages:</h4><ul><li>We do not need to preprocess dataset for this algorithm. Outliers don’t have much effect on its results.</li><li>Performs well with large datasets.</li><li>Is able to handle both numeric and categorical data.</li><li>It is easy to interpret its decision rules.</li></ul><h4>Disadvantages:</h4><ul><li>They tend to overfit and thus, do not generalize well with the entire dataset. We use pruning to avoid this.</li><li>A small change in the dataset could lead to large changes in the tree and hence the final predictions.</li><li>It can predict only within the minimum and maximum range of the response variable in the training data.</li></ul><p>So, with this we wrap up our lesson. I hope you have come a little bit closer to understanding the decision tree. Thank you for reading and have a good day.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=171252c5b653" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/a-noobs-guide-to-decision-trees-171252c5b653">A Noob’s Guide to Decision Trees</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding KNN Algorithm]]></title>
            <link>https://medium.com/swlh/understanding-knn-algorithm-4970d4b71ce3?source=rss-5659a588d90b------2</link>
            <guid isPermaLink="false">https://medium.com/p/4970d4b71ce3</guid>
            <category><![CDATA[classification-algorithms]]></category>
            <category><![CDATA[implementation]]></category>
            <category><![CDATA[knn-algorithm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[basics]]></category>
            <dc:creator><![CDATA[Tanya Gupta]]></dc:creator>
            <pubDate>Mon, 11 Jan 2021 19:00:42 GMT</pubDate>
            <atom:updated>2021-01-12T00:04:04.381Z</atom:updated>
            <content:encoded><![CDATA[<p>Today, we will be learning about KNN algorithm, its applications, intuition behind it and its implementation with MNIST dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*y3bQJA5ff2isL_VI.png" /><figcaption>Source: edureka.co</figcaption></figure><h3>Introduction</h3><p>KNN or K-Nearest Neighbours is a supervised classification algorithm and is as straightforward of an algorithm as its name. This algorithm classifies a given instance based on the groups of its surrounding “K” number of neighbours. For instance,</p><p>Assume you are given 2 groups : food and vehicle. You were given the task of classifying apple in any one of the groups, then the distance between apple and the other food items would be less compared to the vehicle group. This is because they would have more similarity to the features than that when compared to the vehicle group. If you don’t get it, then don’t worry. We will elucidate it further down the road. For now, you can admire the nice visuals given below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/0*iPqWzRdaMwmafhO1.jpg" /><figcaption>Classification of flowers based on their appearance. (source: researchgate.net)</figcaption></figure><h4>Where is it used?</h4><p>It can be best utilized if the dataset :</p><ul><li>has labelled data ( otherwise you won’t be able to group an item)</li><li>is small because it involves a lot of computations for calculating distance( although you can do it with fairly large datasets)</li><li>If data is noise free, i.e., the columns are not filled with random values irrelevant to the datatype of the column.</li></ul><h4>Applications:</h4><ul><li><strong>Used in face-recognition. </strong>You could use this algorithm to judge the similarity between your test cases and training set and accordingly “label” or “classify” the test image.</li><li><strong>Used in recommendation systems: </strong>during online shopping it sorts out similar items for you based on your past purchases or search values.</li><li>Used in retail for <strong>pattern recognition</strong> to avoid credit card fraud and theft.</li><li><strong>Image classification</strong></li></ul><p>Notice that most of the above applications are using KNN algorithms ability to find some commonality among instances of objects</p><h4>Why is it known as “Instance-based learning” ?</h4><p>Before we get into this question, let’s take a small detour to learn what is “instance-based learning”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/0*N0wYSN2wfRrlQLaN.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rJdHYAH6HpleIHX4.png" /><figcaption>Source:blogs.sas.com</figcaption></figure><p>Take for instance, if you were to teach a kid the difference between a cat and a dog, you would emphasize on their appearance like the shape of ears, their paws etc. So, you would be explaining the differences between them by giving instances of cats and dogs and the child would learn from them.</p><p>This is what we call instance-based learning. This type of learning is also known as <strong>“lazy learning”</strong> because they delay all the calculations until we find a new instance to classify.</p><p>Similarly, in KNN algorithm we compare a test set and the training set based on their features. This comparison is made mathematically by calculating the Euclidean distances between them.</p><h4>What is K and how to find it ?</h4><figure><img alt="KNN using scikit-learn | LaptrinhX" src="https://cdn-images-1.medium.com/proxy/1*hncgU7vWLBsRvc8WJhxlkQ.png" /><figcaption>For K=3, The star would be classified as class B but for K=6 it would have been classified as class A.</figcaption></figure><p>K just defines the number of nearest neighbours for classifying the test instance. K can be chosen as :</p><ul><li>square root of the number of instances in our set.</li><li>It must be an odd number. This is because if it were even, then there might be a case where there are equal number of neighbours belonging to each group. For odd K-value, you would always have a majority.</li></ul><p>The K-value should not be too large because this would lead to increase in computations and longer time to process. However, if we take it too small then, it might lead to unstable predictions.</p><h3>The Implementation and steps</h3><p>Let’s get into the steps for understanding the implementation better:</p><ul><li>We are given a dataset and test instance to classify. So we would calculate the differences in features for every row in our training set with our test instance.</li><li>These differences are squared and added in together for every row to find how “close” each instance in training data is to our test instance. This is Euclidean distance.</li><li>We sort these distances in ascending order and choose the first K neighbours.</li><li>From these neighbours we see which group has the majority of neighbours. That group is our result.</li></ul><h4>Given below is the implementation of KNN algorithm on MNIST dataset</h4><p><strong>MNIST (Modified National Institute of Standards and Technology) </strong>dataset provides us with images of digits 0 to 9 in different handwritings. It has 60,000 images in the training set and 10,000 images in testing set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/566/0*Z3g3lxcT38kuKJJO.png" /><figcaption>MNIST dataset ( Source:researchgate.net)</figcaption></figure><p>(the screenshots below are from google colab)</p><ul><li>We first import the necessary libraries for implementation.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/1*SupVoYuDxzW8XXlEb9zQbg.png" /></figure><ul><li>Incase there are null values, you could fill them up with 0’s or the mean of values from 0 to 255 i.e., 128.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*1wJjYsLEy9Cw-NWhSGlXSQ.png" /><figcaption>I sorted based on any column value to create more randomness in label values.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fB3dJ0VygrVLnNHOR51LwQ.png" /><figcaption>The position of the pixels are the column labels for this dataset</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/539/1*maDlmh6VRwe5B8LY27C1YQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*IqEUBL16bBDrrQuhwwhpvA.png" /></figure><ul><li><strong>The KNNClassifier( )</strong> function calculates the distance between our test instance and each of our training instance.</li><li><strong>neighbours</strong> list stores the tuple of (label, distances). Here label is a number from 0 to 9 represented by the values in that row.</li><li>We sort the neighbours list by distance and check the label where majority of our K neighbours belong.</li><li><strong>count </strong>is a dictionary which stores the number of instances of each label among those K neighbours.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/547/1*kH8MrTwsRz5WCa9v8YvFxw.png" /></figure><ul><li><strong>testAccuracy() </strong>sees how many of our predictions are the same as in our test set and then calculates the percentage accuracy.</li><li><strong>plotAccuracy()</strong> plots the ‘accuracy’ vs ‘K-values’ curve using matplotlib library.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/1*_IFvq38RnDt4_AO2tRp-BA.png" /></figure><p>In the above code snippet,</p><ul><li>Predictions list stores all the labels predicted by our classifier</li><li>for every row in our test set, we will initialize values in count dictionary to 0. The keys here are numbers 0 to 9.</li><li>After all iterations are over for a particular K, we will test its accuracy and store it in our accuracy_val list.</li><li>Notice that I have used all odd values for K.</li></ul><p>The output for above is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/557/1*b4eSdyRk87_QnvsRcS4CAw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*Bt1lUaXmVhlw_PfDD48R-Q.png" /></figure><h4>Confusion Matrix</h4><p>This matrix tells us the ratio of false positives, false negatives, true positives and true negatives. It is a very good tool to judge the accuracy of our classifier.</p><p>We will explain the above terms with an example. Assume that a doctor is diagnosing patients with diabetes. In this scenario,</p><p><strong>False positives: </strong>doctor predicted that a patient has diabetes while actually it is not true.</p><p><strong>False negatives: </strong>doctor predicted that a patient doesn’t have diabetes but the prediction is wrong.</p><p><strong>True positive:</strong> doctor diagnoses that patient has diabetes and it turns out to be true.</p><p><strong>True negative: </strong>doctor diagnoses that patient doesn’t have diabetes and it turns out to be true.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*znTDNUgV2VxadQ4k.png" /></figure><p>That’s all for today. I hope that you learned something out of this. Thank you for reading my blog!!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4970d4b71ce3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/understanding-knn-algorithm-4970d4b71ce3">Understanding KNN Algorithm</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Linear Regression]]></title>
            <link>https://medium.com/codex/understanding-linear-regression-402df5a4667e?source=rss-5659a588d90b------2</link>
            <guid isPermaLink="false">https://medium.com/p/402df5a4667e</guid>
            <category><![CDATA[basics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[beginners-guide]]></category>
            <category><![CDATA[linear-regression]]></category>
            <dc:creator><![CDATA[Tanya Gupta]]></dc:creator>
            <pubDate>Sun, 10 Jan 2021 08:07:13 GMT</pubDate>
            <atom:updated>2021-01-12T07:43:44.192Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dl-IaecaQ7Q7o8VQ" /><figcaption>Photo by <a href="https://unsplash.com/@tannerboriack?utm_source=medium&amp;utm_medium=referral">Tanner Boriack</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4><a href="https://medium.com/codex">CODEX</a></h4><h4>In this blog, we will understand probably the most basic algorithm in machine learning which is linear regression</h4><h3>Introduction</h3><p>Linear regression is an algorithm that models the relationship variables linearly. When we say relationship, it is between our dependent variable and the independent variable(s). In this algorithm, we create a “linear” regression line in such a way that it “fits” our dataset. This is a regression algorithm which means that it is used to find out continuous values for our dependent variable. This should not be confused with our classification problem.</p><p><strong>For example,</strong></p><p>if you are predicting whether it would rain today or not, your answer would be either “yes” or “no”. So your answer is classified into 2 groups namely, “yes” or “no”.</p><p>However, for the house pricing dataset, we are predicting house prices which can not be classified into groups, and thus, it is a type of regression problem.</p><p>Now that we have warmed up a bit we will get into the meat of the matter.</p><h3>Explaining the Basics: Simple Linear Regression</h3><p>To understand linear regression, we will first start with simple linear regression i.e., which involves one independent and one dependent variable. <em>Our independent variable is also known as the </em><strong><em>“predictor variable”</em></strong><em> and the dependent variable is known as the </em><strong><em>“response variable”</em></strong><em>.</em> The linear regression model, in general, involves us to compare two models:</p><ul><li>a model containing only the dependent variable</li><li>a model containing both independent and dependent variable</li></ul><p>If the first model, the only way we can have somewhat of a “prediction” is by finding the mean of our dependent variable values. This is the base model with which we will compare our regression line from our second model. If after comparison we don’t see any improvement, this means that our independent variable offers no new information, and discarding it won’t matter much.</p><h4>But, how do we compare the two models in the first place?</h4><p><em>Well for that, we will introduce the concept of </em><strong><em>residuals </em></strong><em>which is a fancy word for the </em><strong><em>difference between the original and the values predicted by our regression line</em></strong><em>. Residuals are also known as errors.</em></p><p>So for comparison, we follow the following steps for both of the models:</p><ul><li>We would first find the residuals between the original values and the values predicted by our regression line.</li><li>We will square all of these residuals and add them up.</li></ul><p>This summation is called the <strong>“sum of squared of errors ”</strong> or <strong>SSE.</strong> This is used as a criterion to compare the above two models. SSE is also known as “unknown errors” and if a model has more of it, then it requires more optimization.</p><p>Besides SSE, we also have SSR and SST.</p><p><strong>SST</strong> is <strong>“sum of squares total”</strong> which accounts for all the errors whether it is known or unknown. It is the sum of squares of differences between the dependent variable predicted from our regression line and the mean of the dependent variable.</p><p><strong>SSR </strong>is <strong>“sum of squares regression”</strong> which accounts for all the known causes for errors and it describes how our line “fits” with our dataset. It is the sum of squares of differences between dependent variable values and the mean of the dependent variable.</p><p>These 3 terms have the following relationship:</p><p><strong>SST = SSE + SSR</strong></p><p><strong><em>Note: </em></strong><em>For our model with the only dependent variable, SSE = SST</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*gYarZmFvCQht3i5U.png" /><figcaption>Source: Vitalflux.com</figcaption></figure><p><strong>R-square value ( Coefficient of Determination): </strong>This value is used to judge how much our regression line fits with our dataset. This is found out by the expression,</p><p><strong>R² = 1-( SSE/SST) = SSR/SST</strong></p><p>If <strong>R²</strong> is near to 1, then this suggests that the regression line fits nicely with the dataset. Else if it is near to 0 or even negative, it fits poorly.</p><p><strong>Code Snippet:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/1*yNlxtzMQW9JVucwx8KVNLQ.png" /></figure><h3>Regression Line</h3><p>Since this is “linear” regression, our regression line would be of the form:</p><p><strong>y = b₀+ b₁*x</strong></p><p>For multiple linear regression our line would be of the form:</p><p><strong>y = b₀+ b₁*x₁+ b₂*x₂ + …. + bₙ*xₙ</strong></p><p>where,</p><ul><li>y is our dependent (response/outcome) variable</li><li>b₁,b₂,…,bₙ are the parameters and b₀ is our intercept</li><li>x₁, x₂,…, xₙ are independent variables</li></ul><p>To find our regression line, we need to first the parameters and the line intercept. For finding the above things we have two methods:</p><ol><li>Least Square Method</li><li>Gradient Descent</li></ol><h4>Least Squares Method</h4><p>The steps for the same are given as follows:</p><ol><li>Find the mean for the dependent and independent variables. Let’s call it yₘₑₐₙ and xₘₑₐₙ.</li><li>For each value of x, subtract xₘₑₐₙ from them and for each value of y subtract yₘₑₐₙ from them.</li><li>Find the product for each (xᵢ-xₘₑₐₙ) and (yᵢ-yₘₑₐₙ), where 1≤i≤n and n is the no. of values for the variables. Then add these products.</li><li>Then, square (xᵢ-xₘₑₐₙ) and add them up together.</li><li>Divide the result of (3) with the result of (4) and this gives you the slope for the regression line.</li><li>For finding the intercept, we solve <strong>b₀ = y - b₁*x </strong>by substituting y with yₘₑₐₙ and x with xₘₑₐₙ.</li></ol><p>Note that we are basically finding slope in the following manner:</p><p><strong><em>slope for regression line (</em>y = b₀+ b₁*x)</strong></p><p><strong><em>= [Σ covariance(xi , yi)/variance(x)] for 1≤i≤n</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*KceVc-CCq6eEwXBiNNbP1A.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/895/1*NdeOsa2w0oJIDfyDR5oqfQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*SqoGccmTnmCohcnVGsFOlQ.png" /><figcaption>Green line is found using least squares method for simple linear regression.</figcaption></figure><h4>Gradient Descent</h4><p>Before we understand this method we need to know what a cost function is.</p><p><strong>Cost Function: </strong>also known<strong> as Mean Squared Error, </strong>it is the summation of squares of differences between the original dependent variable and dependent variable predicted by our regression line.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/765/1*Mf58YX2naGhj_kYbXygLEQ.png" /></figure><p>This method involves us to find the first-order partial derivative of our cost function w.r.t the slope m and the intercept b. But, why do we need to find it?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*DoDlzg9kggewSy6g.jpeg" /><figcaption>(Source: morioh.com) Notice that the step size keeps decreasing. This is because as we move down, our slope also decreases.</figcaption></figure><p>In the above picture, we have a cost curve. Let’s assume that we have slope m on our x-axis. To find the global minimum (where our cost function is minimum), we keep stepping downwards little-by-little by a variable factor. Notice that global (or even local) minimum is a point where the slope of the cost curve is 0. This is the <strong>point of convergence</strong>. This point can be found via partial derivative w.r.t m. Similarly, for intercept b, we can find partial derivative w.r.t b.</p><p><em>Partial derivative w.r.t to some variable suggests the rate of change of cost function w.r.t only that variable.</em></p><p>Calculations are given below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/0*Tskkk4ObOJpHrNeA.png" /></figure><p><strong>Steps to do gradient descent:</strong></p><ol><li>Find the partial derivative of the cost function J(b,m) w.r.t m and b.</li><li>Decrement m and b by the product of <strong>learning rate (α)</strong> and their respective partial derivatives.</li><li>We do the (1) and (2) steps for a specified amount of <strong>iterations (epoch)</strong> until we reach convergence.</li><li>Adjust α and epoch values until you reach the convergence.</li></ol><p><strong><em>Note: α value should not be too large, otherwise we might bypass the convergence point as shown in the picture below. It would take a long time before it reaches the convergence point if it is too small.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/524/1*5CmoAdCjckOlCfse_4xX2A.png" /><figcaption>Source:jeremyjordan.me</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/531/1*sZSw4TnHPw5lP4SWWlPujw.png" /><figcaption>Source:jeremyjordan.me</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/521/1*q8NS8OPqKUahhiMpTVGKxA.png" /><figcaption>Source:jeremyjordan.me</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wfXtTDE-yqqYPLM-.png" /><figcaption>theta 0 is b and theta 1 is m</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/696/0*gf46c-B71Ujdn9HG.png" /><figcaption>For multiple linear regression.</figcaption></figure><h3>Multicollinearity with multiple predictors:</h3><p>Whenever we are performing multiple linear regression, there is a chance that apart from the “correlation” between independent and dependent variables, there is a correlation among independent variables as well. This makes it a “blur” that how much effect a variable has on our dependent variable.</p><p>To get a basic idea of how much a predictor “correlates” with other predictors, we can use the Pearson correlation coefficient about which you can read in this <a href="https://medium.com/@tanya-gupta18/intern-diaries-statistics-67c7d2e75192">blog</a>.</p><p>Using a heatmap gives you a good visual for this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/667/1*ZqQyKqKgF8y6-Gc6sbjjxw.png" /><figcaption>Taking an example from the house prediction dataset, we see that all three of them have a very high correlation with each other and it would be preferable to use one of them to predict the price of the house. Especially the one which has more correlation with our response variable ‘price’.</figcaption></figure><p><strong>Code snippet:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tCyjq0P-982t2mnB5NZYBQ.png" /><figcaption>For finding gradient descent for multiple linear regression from scratch. This method can also be used in simple linear regression, it’s just your x matrix would contain only one column instead of multiple.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/455/1*2MWO-JQcbNsmEBHEqRP8Fg.png" /></figure><p>where,</p><ul><li><strong>cost:</strong> a list that stores all the cost values in each iteration; used later on to plot a cost vs no. of iterations (or epoch).</li><li><strong>min_cost:</strong> initialized with None, it checks for minimum values during iterations.</li><li><strong>b_min:</strong> used to store the parameters for the min_cost value during iterations.</li><li><strong>h: </strong>hypothesis value, found out by performing x values with the parameter values.</li><li><strong>b: </strong>parameter array initialized with ones.</li></ul><p><strong><em>Note:</em></strong><em> using np.sum() increases the computation speed of the code and do use feature scaling on your dataset to bring every predictor values in the same range, this way we will actually see how a predictor “contributes” towards our response variable.</em></p><p><strong>Code snippet for plotting graph:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/1*g90BrfA5-fh0WNhcRnbSAg.png" /></figure><p><strong>Output:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/608/1*u6N7zg-8cVa337Nu6JFvDQ.png" /></figure><p>That’s it for today, thank you for giving your time to read it!!!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=402df5a4667e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codex/understanding-linear-regression-402df5a4667e">Understanding Linear Regression</a> was originally published in <a href="https://medium.com/codex">CodeX</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Intern Diaries: Linear Algebra]]></title>
            <link>https://medium.com/analytics-vidhya/intern-diaries-linear-algebra-24b996b0806c?source=rss-5659a588d90b------2</link>
            <guid isPermaLink="false">https://medium.com/p/24b996b0806c</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[linear-algebra]]></category>
            <dc:creator><![CDATA[Tanya Gupta]]></dc:creator>
            <pubDate>Wed, 06 Jan 2021 14:26:11 GMT</pubDate>
            <atom:updated>2021-01-07T16:16:10.103Z</atom:updated>
            <content:encoded><![CDATA[<p>In today’s blog, we will get into the nitty gritty of things and go down memory lane to revisit our “beloved” high school mathematics topic of linear algebra. We will also discuss why studying linear algebra is essential for getting started with machine learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/0*DJNRL9_CK_oiQJWd.jpg" /><figcaption>Source: acc.fibfl.ru</figcaption></figure><h4><strong>What is Linear Algebra?</strong></h4><p>It is a branch of mathematics dealing with the study and operations of vectors and matrices. It is a form of continuous mathematics and thus, isn’t paid much attention to as much as discrete mathematics by many computer scientist.</p><p>However, despite this one cannot ignore its importance in machine learning. It is mostly used to format data in such a way so that a machine learning algorithm could understand and work on it. We will discuss more applications of linear algebra as we move on with this blog.</p><h3>Basic Terminology</h3><p>Here, we will first learn about some basic terms which are used commonly in linear algebra:</p><ul><li><strong>Vector:</strong> It is an array of numbers which can be either in a column or a row format. This means they can be represented either vertically or horizontally respectively.</li></ul><p><em>You can also say that the row vector could be obtained from the column vector by performing transpose operation on it and vice-versa.</em></p><p>It has both direction and magnitude. When we talk of vectors, its order gives us the dimensionality of our dataset. For example,</p><p><strong>[ x , y , z ] =&gt;</strong> this row vector can be represented as an arrow with its tail at origin and head pointing towards a coordinate (x,y,z) in 3-D space. In other words, each element of this vector gives us a value along different coordinate axes.</p><ul><li><strong>Matrix:</strong> It is a 2 dimensional array of scalars/numbers. You can even say that its a collection of vectors. Each element of the matrix can be allocated using a row and a column index as opposed to only one used in case of vectors. The order of the matrix for example n x m can be defined as n rows and m columns.</li><li><strong>Tensor: </strong>It is a generalized term for vectors , matrices and arrays having a dimensionality greater than 2. So here, a 1-D tensor represents a vector and a 2-D tensor represents a matrix.</li><li><strong>Scalars:</strong> it just means “number” in linear algebra.</li><li><strong>Trace:</strong> sum of all diagonal elements of a square matrix.</li></ul><h3>Operations on Vectors</h3><h4>Addition of two or more vectors:</h4><p>It is a really basic operation. For example there are two row vectors a and b-</p><p>a= [ x1, y1, z1] and b=[x2 , y2 , z2] then the addition would be like</p><p>a+b = [ x1+x2 , y1+y2 , z1+z2 ]</p><p>So, we are just adding the elements at the same index location of each vectors.</p><h4>But why is it important to add vectors in context with ML?</h4><p>It is because, many times in datasets we might end up having multiple features which are of similar type. So, adding the vectors of such features will help us create one vector which can then be used to predict the output feature. This helps us to reduce the number of features in that dataset, making computations easy for our algorithm.</p><p>For example,</p><p>In dataset for “housing price predictions” we find that there are columns like “1stFlrSF” (or first floor square feet) , “2ndFlrSF” (or second floor square feet) and GrLivArea (or group livable area). They all define the area at different locations of the house.</p><p>But our aim is just to predict the price of a house, so we don’t need the above features separately. Instead, we could just add them up and define the summation under the new column vector “TotalAreaSF”.</p><p>In this way, the number of features are reduced and this new feature would still have the same relationship with the output feature as the original ones i.e., the price of house increases with increase in area of the house.</p><p>In this example, what we are doing is numeric grouping using sum operation which is a type of “feature engineering” technique.</p><h4>Scalar Multiplication</h4><p>It is nothing but, number x [ each element in the given vector]. For example, if we have num= 2 and vector [2 , 1 , 3 ] so the result of the scalar multiplication would result into,</p><p>[ 2*2 , 1*2 , 3*2] = [ 4 , 2 , 6 ]</p><p>Earlier we stated that, that the vector [x, y , z] represents an arrow with its head towards the point (x, y , z) in 3D. So, if we imagine doing scalar multiplication on such vector, all we are doing is elongating or shrinking of that vector along its original direction. This can be illustrated with the following image:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*DRFIfvzq8PN3A_QWMDXMcw.png" /><figcaption>The vector [ 1 ,2 ] (blue arrow) is multiplied by 5. The purple vector represents the elongated vector.</figcaption></figure><p>Below is the code snippet to create above image,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GUtyfuG9sDztnCdOVUGGNg.png" /></figure><h3>Magnitude of a vector</h3><p>“Norm” is a term used for defining the magnitude or length of a vector. It can also be said as a function which maps the vectors to a non-negative value. It is denoted by the symbol ∥v∥ where, v is a vector. There are 3 properties of norm:</p><ul><li>It is always a non-negative value.</li><li>Scalar multiplication of a vector either decreases or increases the norm.</li><li>Triangle inequality: This means that if there are <strong>two vectors a and b</strong> and their <strong>addition results in a vector c</strong>, then the sum of magnitudes of a and b is greater than or equal to the magnitude of vector c i.e., ∥<strong>a</strong>∥<strong>+</strong>∥<strong>b</strong>∥<strong>≥</strong>∥<strong>c</strong>∥.</li></ul><h4>Kinds of Norms</h4><p>The most generalized notation of the norm (also known as Lᴾ norm) is given as below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/1*eMG-mLm32rSO_bTQbXt04A.png" /><figcaption>Source: <a href="https://www.datacamp.com/community/tutorials/tutorial-machine-learning-basics-norms">https://www.datacamp.com/community/tutorials/tutorial-machine-learning-basics-norms</a></figcaption></figure><p>where,</p><p>|xᵢ| is the absolute value of the iᵗʰ element of the vector.</p><p>We have taken the absolute value because if we didn’t then, when our p is odd and xᵢ negative, our distances might end up negative which goes against the property of the norm.</p><p>Based on this generalized definition, we have different types of norm:</p><ul><li><strong>L² norm: (also known as Euclidean norm) </strong>which is obtained when p=2. It is the most commonly used norm and defines the Euclidean distance from origin to a point defined by the n-dimensional vector.</li><li><strong>L¹ norm:</strong> It is also known as <strong>“Manhattan distance”</strong> or <strong>“Taxicab geometry”</strong>. It just adds up the absolute values of all the elements in a vector. This is useful in scenarios where, a grid is given and the grid lines represent the streets. We can only walk along the “streets” and so finding the smallest path possible from point A to B is possible using this norm. While L² norm gives us the direct shortest path from A to B, it doesn’t apply in this scenario since we can’t walk outside of the grid lines.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*L_IWDd7L1Kj0TiyUSqxMJg.png" /><figcaption>(Source: en.wikipedia.org) red, blue and yellow paths conform with L1 norm while green path is the direct path given by the L2 norm.</figcaption></figure><ul><li><strong>L-inf norm(L-infinity norm): </strong>This norm gives the maximum absolute value from the vector. For example, in an error vector, it tells which element contributes more to the error.</li><li><strong>L⁰ norm:</strong> It is not a norm technically since it doesn’t conform with the properties of norm. It just gives us the number of non-zero elements in the vector. If we multiply L⁰<strong> </strong>norm<strong> </strong>with any scalar a, then the no. of non-zero elements won’t change, which doesn’t satisfy the 2nd property of norm.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/1*j1oVhnFMtCXO_S6ryGHHdg.png" /></figure><h4>Angles between two vectors</h4><p>This can be found out using dot product. Given two vectors <strong>v</strong> and <strong>w</strong></p><p>v = [v1, v2 , v3 ,…., vn] and w = [ w1, w2 , w3 ,…., wn] the dot product would be,</p><p>v•w = (v1* w1) +(v2*w2) + (v3*w3) +…..+ (vn* wn) = ∥v∥.∥w∥.cosΘ</p><p>where, ∥v∥ and ∥w∥ are the magnitudes of the vectors<strong> v </strong>and <strong>w.</strong></p><p>We could also write it as, <strong>vᵀ.w</strong> where, vᵀ is the transpose of the vector v and v and w were originally column vectors. Needless to say, their dimensionality must be same.</p><ul><li><strong>Orthogonality: </strong>If two vectors are perpendicular to each other or their dot product is 0 then they are “orthogonal” to each other.</li><li><strong>Hyperplane:</strong> is a part of space which is of (n-1) dimensions in a n-dimensional space. So, if we are in 2-D, it is a line. If we are in 3-D, then it is a plane. For a particular vector, if some line, plane or sub-space is perpendicular to it, then that line, plane or sub-space is known as its hyperplane. These are used as decision boundaries for classifier problems.</li></ul><h3>A quick refresher for types of matrices:</h3><ul><li><strong>Square matrix: </strong>has equal no. of rows and columns</li><li><strong>Zero or null matrix: </strong>has all elements as 0</li><li><strong>Diagonal matrix: </strong>has non-zero elements along its diagonal.</li><li><strong>Identity matrix:</strong> It is a square matrix which when multiplied with a matrix does not change the values of that matrix. So, we can say it retains its “identity”.</li><li><strong>Symmetric matrix: </strong>the transpose of a matrix is equal to the original matrix</li><li><strong>Skew-symmetric matrix:</strong> the transpose of a matrix is equal to the original matrix multiplied with scalar -1.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/0*3vqHth_ZpUP_SkK4.png" /><figcaption>Source: onlinemathlearning.com</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/0*O8xft2uCzyjJYGoz.png" /><figcaption>Source: flexiprep.com</figcaption></figure><h3>Matrix Operations</h3><h4><strong>Sum of two matrices</strong></h4><p>We do element-wise addition of 2 matrices:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/1*wG1Wqg9ZBddFs2bk8L2VoQ.png" /></figure><h4>Addition of vector and matrix:</h4><p>Normally if we have a matrix of order n x m and a vector of order 1 x m, then in order to add them together, we would need to create a matrix of the same order out of our vector by repeating the row n times. But <strong>“broadcasting” </strong>creates a shortcut for us, in which the vector is added to every row implicitly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/1*j2U6QCi8bzofT2Tks8dU-A.png" /></figure><h4>Scalar Multiplication</h4><p>Similar to vector, we can also multiply a matrix by a scalar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/474/1*fPtnVQ4M3UVsFnppT73DZw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/359/1*WuYxTy5u1qsiLvWrzs5EuA.png" /></figure><h4><strong>Multiplication of 2 Matrices</strong></h4><p>For the two matrices to be applicable for multiplication, the no. of columns of the first matrix must be equal to the no. of rows of the second matrix. Mathematically it is defined as ,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/184/1*O7tR5bBJ7cGMm3QpzFTWvQ.png" /></figure><p>This kind of element-wise multiplication is known as Hadamard product denoted by A⊙B. It is similar to the dot product of the iᵗʰ row of A and jᵗʰ column of B.</p><blockquote><strong>Note:</strong> matrix multiplication of A and B is not commutative or, AB ≠ BA (not always) but it is distributive [For example : A(B+C) = AB+AC ] and associative [Example: A(BC) = (AB)C ].</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Uu5GAh9QsILKhYnHWdm8HA.png" /></figure><h4>Identity Matrix</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/358/1*f8qr3Xo2EdE_G1nt4Fs_qA.png" /><figcaption>Creating a 4x4 identity matrix using numpy library</figcaption></figure><h4>Inverse matrix</h4><p>If for a matrix A we have AB = I where, I is the identity matrix, then we say that B is the inverse of A. Only square matrices can have an inverse matrix and even then, not all of them have an inverse. Square matrices which do not have an inverse matrix, are known as “singular matrices”. Such matrices have a determinant value of 0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/749/1*upuXLnS51zE_9KTreIavmQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*AhtzVzWwNAgMopH9YGPTrg.png" /><figcaption>np.solve() solves a linear matrix equation or a system of linear scalar equations.</figcaption></figure><h4>Transposing a Matrix</h4><p>In this operation, we just interchange the rows and columns. In this way the dimension of a matrix gets flipped.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/965/1*jzqDd0jVZqzi1MZtIPa9YA.png" /></figure><h4>Eigenvalues and Eigenvectors:</h4><p>“Eigen decomposition” is a method of decomposing a square matrix into eigenvalues and eigenvectors.</p><p><strong><em>AX = λX</em></strong></p><p>where, A is a matrix , X is our eigenvector and λ is a scalar known as eigenvalue. With this definition we see that multiplying vector X by A only leads to the change in its magnitude or scale, not the direction. This is a property of eigenvector.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/816/1*ZmdJj_Kt9q68YpmBx9uGmQ.png" /></figure><h4>Determinants</h4><p>When we multiply a matrix and a vector we see that our vectors gets transformed. What this means is, that our vector gets scaled , might also get rotated. Even in the case of eigenvalues and eigenvectors we saw that a matrix multiplying with a vector leads to the scaling of that vector. This scalar quantity that changes the form of that vector, hidden underneath our matrix is what we call our determinant. Well, this is the intuitive idea.</p><p>Its formal definition is,</p><blockquote>In linear algebra, the <strong>determinant</strong> is a scalar value that can be computed from the elements of a square matrix and encodes certain properties of the linear transformation described by the matrix. The determinant of a matrix <em>A</em> is denoted det(<em>A</em>), det <em>A</em>, or |<em>A</em>|.</blockquote><p>So, for a vector lying on a 2-D plane, multiplying by matrix might lead to shearing and scaling of a certain area. For 3-D plane it might lead to the change in the volume or configuration of a unit volume cube.</p><p>Anyways, enough with the intuitive idea… Now we will see how it is calculated.</p><p>For a 2 x 2 matrix it is defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/318/1*Tpa9iKylVkWlqMoBz1K0vQ.png" /></figure><p>For a 3 x 3 matrix it is defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/1*xzLQUMNtYBCGo6WwYqYXgQ.png" /></figure><p>One interesting thing to notice about determinants are that, if upon multiplying with a matrix, a vector merges with one of the coordinate axis or even merges into a point, the features of the vectors are lost. You cannot bring back the original features by multiplication with any matrix. This leads to invertibility. So a matrix with |A|=0, by that sense couldn’t be inverted back and thus, doesn’t have an inverse.</p><p><strong>Code snippet for calculating determinant using NumPy:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/611/1*rIuFRg3iVDpUQGIaK0Gqlg.png" /></figure><h4>Linear Dependency:</h4><p>It is the idea that a vector can be represented as a linear combination of a set of vectors. In order to understand this concept we need to understand what a span is.</p><p><strong>Span:</strong> for a given set of vectors , refers to all the points in the space covered by all possible scalar manipulations and combinations of vectors in that set. For a single vector a span would constitute a straight line along which it slithers.</p><p>So if a vector a lies on the span of the set of vectors V , then that means it is one of the combination of vectors V in which some(or all) might have been multiplied by scalars.</p><p>Mathematical definition of linear dependency,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/1*WFp6R5DGqdeW5u8yNDWiTw.png" /><figcaption>where a1, a2, …, ak are scalars while, v1,v2,…,vk are vectors.</figcaption></figure><p>This concept is important because linear dependency often leads to redundant features which offer no information regarding our dataset.</p><p>That’s it for today’s lesson. I hope you find this blog beneficial. Thank you for reading!!</p><h3>References and useful links:</h3><ul><li><a href="https://www.deeplearningbook.org/contents/linear_algebra.html">https://www.deeplearningbook.org/contents/linear_algebra.html</a></li><li><a href="http://datahacker.rs/linear-algebra-for-machine-learning/">http://datahacker.rs/linear-algebra-for-machine-learning/</a></li><li><a href="https://en.wikipedia.org/wiki/Linear_independence#Geometric_meaning">https://en.wikipedia.org/wiki/Linear_independence#Geometric_meaning</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=24b996b0806c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/intern-diaries-linear-algebra-24b996b0806c">Intern Diaries: Linear Algebra</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>