<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Khyati Mahendru on Medium]]></title>
        <description><![CDATA[Stories by Khyati Mahendru on Medium]]></description>
        <link>https://medium.com/@khyatimahendru?source=rss-78433997a4a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*iouBOfVcAhhsCXcHEnmmyw.jpeg</url>
            <title>Stories by Khyati Mahendru on Medium</title>
            <link>https://medium.com/@khyatimahendru?source=rss-78433997a4a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 01:22:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@khyatimahendru/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A Detailed Guide to 7 Loss Functions for Machine Learning Algorithms]]></title>
            <link>https://medium.com/analytics-vidhya/a-detailed-guide-to-7-loss-functions-for-machine-learning-algorithms-26e11b6e700b?source=rss-78433997a4a------2</link>
            <guid isPermaLink="false">https://medium.com/p/26e11b6e700b</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[regression]]></category>
            <category><![CDATA[loss-function]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[classification]]></category>
            <dc:creator><![CDATA[Khyati Mahendru]]></dc:creator>
            <pubDate>Wed, 14 Aug 2019 02:50:26 GMT</pubDate>
            <atom:updated>2024-07-19T05:34:11.003Z</atom:updated>
            <content:encoded><![CDATA[<h4>with Python Code</h4><h3>Overview</h3><ul><li>What are loss functions? And how do they work in machine learning algorithms? Find out in this article</li><li>Loss functions are actually at the heart of these techniques that we regularly use</li><li>This article covers multiple loss functions, where they work, and how you can code them in Python</li></ul><h3>Introduction</h3><p>Picture this — you’ve trained a machine learning model on a given dataset and are ready to put it in front of your client. But how can you be sure that this model will give the optimum result? Is there a metric or a technique that will help you quickly evaluate your model on the dataset?</p><p>Yes — and that, in a nutshell, is where loss functions come into play in machine learning.</p><p>Loss functions are at the heart of the <a href="https://www.analyticsvidhya.com/blog/2017/09/common-machine-learning-algorithms/?utm_source=blog&amp;utm_medium=detailed-guide-understanding-loss-functions-machine-learning-python-code">machine learning algorithms</a> we love to use. But I’ve seen the majority of beginners and enthusiasts become quite confused regarding how and where to use them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/0*A5X8coeaIhXLTTlK.jpg" /></figure><p>They’re not difficult to understand and will enhance your understand of machine learning algorithms infinitely. So, what are loss functions and how can you grasp their meaning?</p><p>In this article, I will discuss 7 common loss functions used in <a href="https://courses.analyticsvidhya.com/courses/applied-machine-learning-beginner-to-professional?utm_source=blog&amp;utm_medium=detailed-guide-understanding-loss-functions-machine-learning-python-code">machine learning</a> and explain where each of them is used. We have a lot to cover in this article so let’s begin!</p><h3>Table of Contents</h3><h4>What are Loss Functions?</h4><h4>Regression Loss Functions</h4><ul><li>Squared Error Loss</li><li>Absolute Error Loss</li><li>Huber Loss</li></ul><h4>Binary Classification Loss Functions</h4><ul><li>Binary Cross-Entropy</li><li>Hinge Loss</li></ul><h4>Multi-class Classification Loss Functions</h4><ul><li>Multi-class Cross Entropy Loss</li><li>Kullback Leibler Divergence Loss</li></ul><h3>What are Loss Functions?</h3><p>Let’s say you are on the top of a hill and need to climb down. How do you decide where to walk towards?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Rd-DYURq14Zg_G3l.jpg" /></figure><p>Here’s what I would do:</p><ul><li>Look around to see all the possible paths</li><li>Reject the ones going up. This is because these paths would actually cost me more energy and make my task even more difficult</li><li>Finally, take the path that I <strong><em>think</em> </strong>has the most slope downhill</li></ul><p>This intuition that I just judged my decisions against? This is exactly what a loss function provides.</p><blockquote><em>A loss function maps decisions to their associated costs.</em></blockquote><p>Deciding to go up the slope will cost us energy and time. Deciding to go down will benefit us. Therefore, it has a negative cost.</p><p>In <a href="https://www.analyticsvidhya.com/blog/2017/09/common-machine-learning-algorithms/?utm_source=blog&amp;utm_medium=detailed-guide-understanding-loss-functions-machine-learning-python-code">supervised machine learning</a> algorithms, we want to <strong>minimize</strong> the error for each training example during the learning process. This is done using some optimization strategies like <a href="https://www.analyticsvidhya.com/blog/2017/03/introduction-to-gradient-descent-algorithm-along-its-variants/?utm_source=blog&amp;utm_medium=detailed-guide-understanding-loss-functions-machine-learning-python-code">gradient descent</a>. <em>And this error comes from the loss function.</em></p><h3>What’s the Difference between a Loss Function and a Cost Function?</h3><p>I want to emphasize this here — although <strong><em>cost function</em></strong> and <strong><em>loss function</em></strong> are synonymous and used interchangeably, they are different.</p><p>A loss function is for a single training example. It is also sometimes called an <strong>error function</strong>. A cost function, on the other hand, is the <strong>average loss</strong> over the entire training dataset. The optimization strategies aim at minimizing the cost function.</p><h3>Regression Loss Functions</h3><p>You must be quite familiar with linear regression at this point. It deals with modeling a linear relationship between a <strong>dependent variable</strong>, Y, and several <strong>independent variables,</strong> X_i’s. Thus, we essentially fit a line in space on these variables.</p><pre>Y = a0 + a1 * X1 + a2 * X2 + ....+ an * Xn</pre><p>We will use the given data points to find the coefficients a0, a1, …, an.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*fKxHQFuuarziUmTq.png" /><figcaption><em>Source: Wikipedia</em></figcaption></figure><p>We will use the famous <a href="https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_boston.html">Boston Housing Dataset</a> for understanding this concept. And to keep things simple, we will use only one feature — the <em>Average number of rooms per dwelling</em> (X) — to predict the dependent variable — <em>Median Value</em> (Y) of houses in $1000′ s.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/0*SxDdLHETJ_tWUpyN.png" /></figure><p>We will use <strong>Gradient Descent</strong> as an optimization strategy to find the regression line. I will not go into the intricate details about Gradient Descent, but here is a reminder of the Weight Update Rule:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*5MWbA8WaPSmmOWI2.png" /></figure><p>Here, theta_j is the weight to be updated, alpha is the learning rate and J is the cost function. The cost function is parameterized by theta. <strong>Our aim is to find the value of theta which yields minimum overall cost.</strong></p><p><em>You can get an in-depth explanation of Gradient Descent and how it works </em><a href="https://www.analyticsvidhya.com/blog/2017/03/introduction-to-gradient-descent-algorithm-along-its-variants/"><em>here</em></a><em>.</em></p><p>I have defined the steps that we will follow for each loss function below:</p><ol><li>Write the expression for our predictor function, f(X) and identify the parameters that we need to find</li><li>Identify the loss to use for each training example</li><li>Find the expression for the Cost Function — the average loss on all examples</li><li>Find the gradient of the Cost Function with respect to each unknown parameter</li><li>Decide on the learning rate and run the weight update rule for a fixed number of iterations</li></ol><h3>1. Squared Error Loss</h3><p>Squared Error loss for each training example, also known as <strong>L2 Loss</strong>, is the square of the difference between the actual and the predicted values:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/289/0*vl7jaf0v9AZ_U-5Y.jpg" /></figure><p>The corresponding cost function is the <strong>Mean</strong> of these<strong> Squared Errors (MSE)</strong>.</p><p>I encourage you to try and find the gradient for gradient descent yourself before referring to the code below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/11e75aa608d6f2cbb32dd14101b8f8b4/href">https://medium.com/media/11e75aa608d6f2cbb32dd14101b8f8b4/href</a></iframe><p>I used this code on the Boston data for different values of the learning rate for 500 iterations each:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/0*PUOCZIWEPArSGg7S.png" /></figure><p><em>Here’s a task for you. Try running the code for a learning rate of 0.1 again for 500 iterations. Let me know your observations and any possible explanations in the comments section.</em></p><p>Let’s talk a bit more about the MSE loss function. It is a positive quadratic function (of the form ax² + bx + c where a &gt; 0). Remember how it looks graphically?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*repsk6oCB9YFNZVp.png" /></figure><p>A quadratic function only has a global minimum. Since there are no local minima, we will never get stuck in one. Hence, it is always guaranteed that Gradient Descent will converge ( <em>if it converges at all</em>) to the global minimum.</p><p>The MSE loss function penalizes the model for making large errors by squaring them. Squaring a large quantity makes it even larger, right? But there’s a caveat. This property makes the MSE cost function less robust to outliers. Therefore,<strong><em> it should not be used if our data is prone to many outliers.</em></strong></p><h3>2. Absolute Error Loss</h3><p>Absolute Error for each training example is the distance between the predicted and the actual values, irrespective of the sign. Absolute Error is also known as the <strong>L1 loss:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/277/0*M-U9_kRegbIrNA8R.jpg" /></figure><p>As I mentioned before, the cost is the <strong>Mean</strong> of these <strong>Absolute Errors (MAE)</strong>.</p><p><strong><em>The MAE cost is more robust to outliers as compared to MSE</em>.</strong> However, handling the absolute or modulus operator in mathematical equations is not easy. I’m sure a lot of you must agree with this! We can consider this as a disadvantage of MAE.</p><p>Here is the code for the <em>update_weight</em> function with MAE cost:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a1658b9017870a98cc5a4683737b1e21/href">https://medium.com/media/a1658b9017870a98cc5a4683737b1e21/href</a></iframe><p>We get the below plot after running the code for 500 iterations with different learning rates:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/0*NsoM8QVpoNihV41G.png" /></figure><h3>3. Huber Loss</h3><p>The Huber loss combines the best properties of MSE and MAE. It is quadratic for smaller errors and is linear otherwise (and similarly for its gradient). It is identified by its <em>delta</em> parameter:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/513/0*-tdPFMV1Q0TFFgaO.jpg" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0fcf5d7386ffdbe952716b9caccc6817/href">https://medium.com/media/0fcf5d7386ffdbe952716b9caccc6817/href</a></iframe><p>We obtain the below plot for 500 iterations of weight update at a learning rate of 0.0001 for different values of<em> the delta </em>parameter:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/0*DOB2Bs9Di4lhwwuk.png" /></figure><p>Huber loss is more robust to outliers than MSE. <strong><em>It is used in </em></strong><a href="https://en.wikipedia.org/wiki/Robust_regression"><strong><em>Robust Regression</em></strong></a><strong><em>, </em></strong><a href="https://en.wikipedia.org/wiki/M-estimator"><strong><em>M-estimation</em></strong></a><strong><em> and </em></strong><a href="https://en.wikipedia.org/wiki/Additive_model"><strong><em>Additive Modelling</em></strong></a><strong><em>. A variant of Huber Loss is also used in classification.</em></strong></p><h3>Binary Classification Loss Functions</h3><p>The name is pretty self-explanatory. Binary Classification refers to assigning an object into one of two classes. This classification is based on a rule applied to the input feature vector. For example, classifying an email as <em>spam</em> or <em>not spam</em> based on, say its subject line, is binary classification.</p><p>I will illustrate these binary classification loss functions on the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_breast_cancer.html">Breast Cancer dataset</a>.</p><p><em>We want to classify a tumor as</em><strong><em> ‘Malignant’</em></strong><em> or</em><strong><em> ‘Benign’</em></strong><em> based on features like average radius, area, perimeter, etc. For simplification, we will use only two input features (X_1 and X_2) namely </em><strong><em>‘worst area’</em></strong><em> and</em><strong><em> ‘mean symmetry’</em></strong><em> for classification. The target value Y can be 0 (Malignant) or 1 (Benign).</em></p><p>Here is a scatter plot for our data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/0*GP-a6ZY0oVfjDvRy.png" /></figure><h3>1. Binary Cross Entropy Loss</h3><p><strong>Let us start by understanding the term ‘entropy’.</strong> Generally, we use entropy to indicate disorder or uncertainty. It is measured for a random variable X with probability distribution p(X):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/418/0*0-osPu2NVYSvWvFX.jpg" /></figure><p>The negative sign is used to make the overall quantity positive.</p><blockquote><em>A greater value of entropy for a probability distribution indicates a greater uncertainty in the distribution. Likewise, a smaller value indicates a more certain distribution.</em></blockquote><p>This makes binary cross-entropy suitable as a loss function — <strong>you want to minimize its value. </strong>We use <strong>binary cross-entropy</strong> loss for classification models which output a probability <em>p</em>.</p><pre>Probability that the element belongs to class 1 (or positive class) = p <br>Then, the probability that the element belongs to class 0 (or negative class) = 1 - p</pre><p>Then, the cross-entropy loss for output label y (can take values 0 and 1) and predicted probability p is defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/0*STviM21QPOdQtHR1.jpg" /></figure><p>This is also called <strong>Log-Loss. </strong>To calculate the probability p, we can use the sigmoid function. Here, z is a function of our input features:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/134/0*PHLugd2orEiFGmYl.jpg" /></figure><p>The range of the sigmoid function is [0, 1] which makes it suitable for calculating probability.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/275/0*dmvVWgOeRPI3sLY1.png" /></figure><p>Try to find the gradient yourself and then look at the code for the <em>update_weight</em> function below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/912c347773e5f8208b1965bc19c42b6b/href">https://medium.com/media/912c347773e5f8208b1965bc19c42b6b/href</a></iframe><p>I got the below plot on using the weight update rule for 1000 iterations with different values of alpha:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/0*VRix3T3d3-oPl-NE.png" /></figure><h3>2. Hinge Loss</h3><p><strong><em>Hinge loss is primarily used with </em></strong><a href="https://www.analyticsvidhya.com/blog/2017/09/understaing-support-vector-machine-example-code/"><strong><em>Support Vector Machine (SVM)</em></strong></a><strong><em> Classifiers with class labels -1 and 1</em></strong>. So make sure you change the label of the ‘Malignant’ class in the dataset from 0 to -1.</p><blockquote><em>Hinge Loss not only penalizes the wrong predictions but also the right predictions that are not confident.</em></blockquote><p>Hinge loss for an input-output pair (x, y) is given as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/421/0*VOoMkump1JaNWojT.jpg" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2f386ee4ac0fbc8fe5eb1c4d376d7d22/href">https://medium.com/media/2f386ee4ac0fbc8fe5eb1c4d376d7d22/href</a></iframe><p>After running the update function for 2000 iterations with three different values of alpha, we obtain this plot:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/0*_BYAQ5AxkoCUvbFB.png" /></figure><p><strong><em>Hinge Loss simplifies the mathematics for SVM while maximizing the loss (as compared to Log-Loss). It is used when we want to make real-time decisions with not a laser-sharp focus on accuracy.</em></strong></p><h3>Multi-Class Classification Loss Functions</h3><p>Emails are not just classified as spam or not spam (this isn’t the 90s anymore!). They are classified into various other categories — Work, Home, Social, Promotions, etc. This is a <a href="https://www.analyticsvidhya.com/blog/2017/08/introduction-to-multi-label-classification/">Multi-Class Classification</a> use case.</p><p>We’ll use the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_iris.html">Iris Dataset</a> for understanding the remaining two loss functions. We will use 2 features X_1, <strong>Sepal length</strong> and feature X_2, <strong>Petal width,</strong> to predict the class (Y) of the Iris flower — <strong>Setosa, Versicolor or Virginica</strong></p><p>Our task is to implement the classifier using a neural network model and the in-built Adam optimizer in Keras. This is because as the number of parameters increases, the math, as well as the code, will become difficult to comprehend.</p><p>If you are new to Neural Networks, I highly recommend reading <a href="https://medium.com/analytics-vidhya/build-your-first-neural-network-model-on-a-structured-dataset-using-keras-d9e7de5c6724">this article</a> first.</p><p>Here is the scatter plot for our data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/0*xs4dY2yVl-keQSd8.png" /></figure><h3>1. Multi-Class Cross Entropy Loss</h3><p>The multi-class cross-entropy loss is a generalization of the Binary Cross Entropy loss. The loss for input vector X_i and the corresponding one-hot encoded target vector Y_i is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/679/0*BcorbZXeGD-BRKiq.jpg" /></figure><p>We use the softmax function to find the probabilities p_ij:</p><blockquote><em>“Softmax is implemented through a neural network layer just before the output layer. The Softmax layer must have the same number of nodes as the output layer.”</em></blockquote><blockquote>Google Developer’s Blog</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/0*quNwGtzt7OCm_act.jpg" /></figure><p>Finally, our output is the class with the maximum probability for the given input.</p><p>We build a model using an input layer and an output layer and compile it with different learning rates. Specify the loss parameter as <strong>‘categorical_crossentropy’</strong> in the model.compile() statement:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4734f5df71864c8ef00cfc566af73ba4/href">https://medium.com/media/4734f5df71864c8ef00cfc566af73ba4/href</a></iframe><p>Here are the plots for cost and accuracy respectively after training for 200 epochs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/0*pPvwazKtvNwe3xkr.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/0*l6N3IBKHeKeN_f4B.png" /></figure><h3>2. KL-Divergence</h3><p>The <strong>Kullback-Liebler</strong> <strong>Divergence</strong> is a measure of how a probability distribution differs from another distribution. A KL-divergence of zero indicates that the distributions are identical.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/0*9j5MJ00NShW6G_BY.jpg" /></figure><p>Notice that the divergence function is not symmetric.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/214/0*ekk5uJxhfltJXK6k.jpg" /></figure><p>This is why KL-Divergence cannot be used as a distance metric.</p><p>I will describe the basic approach of using KL-Divergence as a loss function without getting into its math. We want to approximate the <strong><em>true</em></strong> probability distribution P of our target variables with respect to the input features, given some <strong><em>approximate</em></strong> distribution Q. Since KL-Divergence is not symmetric, we can do this in two ways:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/0*XliwyNXsT74y0Fl9.jpg" /></figure><p>The first approach is used in Supervised learning, the second in Reinforcement Learning. KL-Divergence is functionally similar to multi-class cross-entropy and is also called relative entropy of P with respect to Q:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/0*qM3WavYEb63AH17a.jpg" /></figure><p>We specify the <strong>‘kullback_leibler_divergence’</strong> as the value of the loss parameter in the compile() function as we did before with the multi-class cross-entropy loss.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/25b7a5cb9deb90506212613a0b44e95b/href">https://medium.com/media/25b7a5cb9deb90506212613a0b44e95b/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/0*V3N7Lpe8QWTjT9zI.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/0*b71oD5XLNXp19WuA.png" /></figure><p><strong><em>KL-Divergence is used more commonly to approximate complex functions than in multi-class classification. We come across KL-Divergence frequently while playing with deep-generative models like Variational Autoencoders (VAEs).</em></strong></p><h3>End Notes</h3><p>Woah! We have covered a lot of ground here. Give yourself a pat on your back for making it all the way to the end. This was quite a comprehensive list of loss functions we typically use in machine learning.</p><p>I would suggest going through this article a couple of times more as you proceed with your machine learning journey. This isn’t a one-time effort. It will take a few readings and experience to understand how and where these loss functions work.</p><p>Make sure to experiment with these loss functions and let me know your observations down in the comments. Also, let me know other topics that you would like to read about. I will do my best to cover them in future articles.</p><p><em>Originally published at </em><a href="https://www.analyticsvidhya.com/blog/2019/08/detailed-guide-7-loss-functions-machine-learning-python-code/"><em>https://www.analyticsvidhya.com</em></a><em> on August 14, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=26e11b6e700b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/a-detailed-guide-to-7-loss-functions-for-machine-learning-algorithms-26e11b6e700b">A Detailed Guide to 7 Loss Functions for Machine Learning Algorithms</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Master Dimensionality Reduction with these 5 Must-Know Applications of Singular Value…]]></title>
            <link>https://medium.com/analytics-vidhya/master-dimensionality-reduction-with-these-5-must-know-applications-of-singular-value-777299940b89?source=rss-78433997a4a------2</link>
            <guid isPermaLink="false">https://medium.com/p/777299940b89</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[linear-algebra]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[svd]]></category>
            <category><![CDATA[dimensionality-reduction]]></category>
            <dc:creator><![CDATA[Khyati Mahendru]]></dc:creator>
            <pubDate>Mon, 05 Aug 2019 03:21:04 GMT</pubDate>
            <atom:updated>2024-07-23T08:28:46.558Z</atom:updated>
            <content:encoded><![CDATA[<h3>Master Dimensionality Reduction with these 5 Must-Know Applications of Singular Value Decomposition (SVD) in Data Science</h3><h3>Overview</h3><ul><li>Singular Value Decomposition (SVD) is a common dimensionality reduction technique in data science</li><li>We will discuss 5 must-know applications of SVD here and understand their role in data science</li><li>We will also see three different ways of implementing SVD in Python</li></ul><h3>Introduction</h3><blockquote><em>“Another day has passed, and I still haven’t used </em>y = mx + b.<em> “</em></blockquote><p>Sounds familiar? I often hear my school and college acquaintances complain that the algebra equations they spent so much time on are essentially useless in the real world.</p><p>Well — I can assure you that’s simply not true. Especially if you want to carve out a career in data science.</p><p>Linear algebra bridges the gap between theory and practical implementation of concepts. A healthy understanding of linear algebra opens doors to <a href="https://courses.analyticsvidhya.com/courses/applied-machine-learning-beginner-to-professional/?utm_source=blog&amp;utm_medium=5-applications-singular-value-decomposition-svd-data-science">machine learning</a> algorithms we thought were impossible to understand. And one such use of linear algebra is in Singular Value Decomposition (SVD) for dimensionality reduction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/0*qWhMvb4nemrCJie3.jpg" /></figure><p>You must have come across SVD a lot in data science. It’s everywhere, especially when we’re dealing with dimensionality reduction. But what is it? How does it work? And what are SVD’s applications?</p><p>I briefly mentioned SVD and its applications in my article on the <a href="https://www.analyticsvidhya.com/blog/2019/07/10-applications-linear-algebra-data-science/?utm_source=blog&amp;utm_medium=5-applications-singular-value-decomposition-svd-data-science">Applications of Linear Algebra in Data Science</a>. In fact, SVD is the foundation of <a href="https://www.analyticsvidhya.com/blog/2018/06/comprehensive-guide-recommendation-engine-python/?utm_source=blog&amp;utm_medium=5-applications-singular-value-decomposition-svd-data-science">Recommendation Systems</a> that are at the heart of huge companies like Google, YouTube, Amazon, Facebook and many more.</p><p>We will look at five super useful applications of SVD in this article. But we won’t stop there — we will explore how we can use SVD in Python in three different ways as well.</p><p><em>And if you’re looking for a one-stop-shop to learn all machine learning concepts, we have put together one of the </em><a href="http://courses.analyticsvidhya.com/courses/applied-machine-learning-beginner-to-professional/?utm_source=blog&amp;utm_medium=5-applications-singular-value-decomposition-svd-data-science"><em>most comprehensive courses</em></a><em> available anywhere. Make sure you check it out (and yes, SVD is in there as part of the dimensionality reduction module).</em></p><h3>Table of Contents</h3><h4>Applications of Singular Value Decomposition (SVD)</h4><ul><li>Image Compression</li><li>Image Recovery</li><li>Eigenfaces</li><li>Spectral Clustering</li><li>Background Removal from Videos</li></ul><h4>What is Singular Value Decomposition?</h4><ul><li>Rank of a Matrix</li><li>Singular Value Decomposition</li><li>Why is SVD used in Dimensionality Reduction?</li></ul><h4>3 Ways to Perform SVD in Python</h4><h3>Applications of Singular Value Decomposition (SVD)</h3><p>We are going to follow a top-down approach here and discuss the applications first. I have explained the math behind SVD after the applications for those interested in how it works underneath.</p><p>You just need to know four things to understand the applications:</p><ol><li>SVD is the <strong>decomposition</strong> of a matrix A <strong>into 3 matrices — U, S, and V</strong></li><li>S is the diagonal matrix of singular values. Think of <strong>singular values</strong> as the importance values of different features in the matrix</li><li>The <strong>rank</strong> of a matrix is a measure of the unique information stored in a matrix. Higher the rank, more the information</li><li><strong>Eigenvectors</strong> of a matrix are directions of maximum spread or variance of data</li></ol><p>In most of the applications, the basic principle of <a href="https://dimensionality-reduction-techniques-python/">Dimensionality Reduction</a> is used. <em>You want to reduce a high-rank matrix to a low-rank matrix while preserving important information</em>.</p><h3>SVD for Image Compression</h3><p>How many times have we faced this issue? We love clicking images with our smartphone cameras and saving random photos off the web. And then one day — no space! Image compression helps deal with that headache.</p><p>It minimizes the size of an image in bytes to an acceptable level of quality. This means that you are able to store more images in the same disk space as compared to before.</p><p>Image compression takes advantage of the fact that only a few of the singular values obtained after SVD are large. You can trim the three matrices based on the first few singular values and obtain a compressed approximation of the original image. Some of the compressed images are nearly indistinguishable from the original by the human eye.</p><p>Here’s how you can code this in Python:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7ea6bff066904224bf75d100782fd12f/href">https://medium.com/media/7ea6bff066904224bf75d100782fd12f/href</a></iframe><p><strong>Output:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/933/0*yst9E9jdAOl5z5ed.png" /></figure><p>If you ask me, even the last image (with n_components = 100) is quite impressive. I would not have guessed that it was compressed if I did not have the other images for comparison.</p><h3>SVD for Image Recovery</h3><p>Ever clicked an image in low light? Or had an old image become corrupt? We assume that we cannot get that image back anymore. It’s surely lost to the past. Well — not anymore!</p><p>We’ll understand image recovery through the concept of matrix completion (and a cool Netflix example).</p><p>Matrix Completion is the process of filling in the missing entries in a partially observed matrix. The Netflix problem is a common example of this.</p><blockquote><em>Given a ratings-matrix in which each entry represents the rating of movie</em><strong> j</strong><em> by customer</em><strong> i, </strong><em>if customer has watched movie</em><strong> j</strong><em> and is otherwise missing, we would like to predict the remaining entries in order to make good recommendations to customers on what to watch next.</em></blockquote><p>The basic fact that helps to solve this problem is that <strong>most users have a pattern</strong> in the movies they watch and in the ratings they give to these movies. So, the ratings-matrix has little unique information. This means that a low-rank matrix would be able to provide a good enough approximation for the matrix.</p><p>This is what we achieve with the help of SVD.</p><p>Where else do you see this property? Yes, in matrices of images! <strong>Since an image is contiguous, the values of most pixels depend on the pixels around them.</strong> So a low-rank matrix can be a good approximation of these images.</p><p>Here is a snapshot of the results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/0*NnGxMVGIHfLUX_WN.jpg" /><figcaption><em>Chen, Zihan. “Singular Value Decomposition and its Applications in Image Processing.” ACM, 2018</em></figcaption></figure><p>The entire formulation of the problem can be complex to comprehend and requires knowledge of other advanced concepts as well. You can read the paper that I referred to <a href="https://dl.acm.org/doi/10.1145/3274250.3274261">here</a>.</p><h3>SVD for Eigenfaces</h3><p>The original paper <a href="http://www.face-rec.org/algorithms/pca/jcn.pdf">Eigenfaces for Recognition</a> came out in 1991. Before this, most of the approaches for facial recognition dealt with identifying individual features such as the eyes or the nose and developing a face model by the position, size, and relationships among these features.</p><blockquote><em>The Eigenface approach sought to extract the relevant information in a face image, encode it as efficiently as possible, and compare one face encoding with a database of models encoded similarly.</em></blockquote><p>The encoding is obtained by expressing <strong>each face as a linear combination of the selected eigenfaces in the new face space</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/430/0*QimLhqx_jZPKdwK6.png" /></figure><p>Let me break the approach down into five steps:</p><ol><li>Collect a training set of faces as the training set</li><li>Find the most important features by finding the <strong>directions of maximum variance</strong> — the <strong>eigenvectors or the eigenfaces</strong></li><li>Choose top M eigenfaces corresponding to the highest eigenvalues. These eigenfaces now define a new <em>face space</em></li><li>Project all the data in this <em>face space</em></li><li>For a new face, project it into the new <em>face space</em>, find the closest face(s) in the space, and classify the face as <strong>a</strong> <strong>known or an unknown face</strong></li></ol><p>You can find these eigenfaces using both PCA and SVD. Here is the first of several eigenfaces I obtained after performing SVD on the <a href="http://vis-www.cs.umass.edu/lfw/">Labelled Faces in the Wild</a> dataset:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ALBwngsRpqdbrBUz.jpg" /></figure><p>As we can see, only the images in the first few rows look like actual faces. Others look noisy and hence I discarded them. I preserved a total of 120 eigenfaces and transformed the data into the new <em>face space</em>. Then I used the <a href="https://www.analyticsvidhya.com/blog/2018/03/introduction-k-neighbours-algorithm-clustering/?utm_source=blog&amp;utm_medium=5-applications-singular-value-decomposition-svd-data-science"><strong><em>k-nearest neighbors</em></strong></a><strong><em> classifier</em></strong><em> </em>to predict the names based on the faces.</p><p>You can see the classification report below. Clearly, there is scope for improvement. You can try adjusting the number of eigenfaces to preserve and experiment with different classifiers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/482/0*U6gsLdxzH1aeC8Ma.jpg" /></figure><p>Have a look at some of the predictions and their true labels:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/0*OYPQD15xGmtEvuPC.png" /></figure><p>You can find my attempt at Facial Recognition using Eigenfaces <a href="https://github.com/KhyatiMahendru/EigenfacesWithSVD/blob/master/EigenfacesSVD.ipynb">here</a>.</p><h3>SVD for Spectral Clustering</h3><p><a href="https://www.analyticsvidhya.com/blog/2016/11/an-introduction-to-clustering-and-different-methods-of-clustering/?utm_source=blog&amp;utm_medium=5-applications-singular-value-decomposition-svd-data-science">Clustering</a> is the task of grouping similar objects together. It is an unsupervised <a href="https://courses.analyticsvidhya.com/courses/applied-machine-learning-beginner-to-professional/?utm_source=blog&amp;utm_medium=5-applications-singular-value-decomposition-svd-data-science">machine learning</a> technique. For most of us, clustering is synonymous with K-Means Clustering — a simple but powerful algorithm. However, it is not always the most accurate.</p><p>Consider the below case:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/383/0*uA5vuPyR2xAa0PmJ.png" /></figure><p>Clearly, there are 2 clusters in concentric circles. But KMeans with n_clusters = 2 gives the following clusters:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/383/0*xkr8a2GJAp2IloPo.png" /></figure><p>K-Means is definitely not the appropriate algorithm to use here. Spectral clustering is a technique that combats this. It has roots in <a href="https://www.analyticsvidhya.com/blog/2018/09/introduction-graph-theory-applications-python/?utm_source=blog&amp;utm_medium=5-applications-singular-value-decomposition-svd-data-science">Graph theory</a>. These are the basic steps:</p><ul><li>Start with the <strong>Affinity matrix (A)</strong> or the <strong>Adjacency matrix</strong> of the data. This represents how similar one object is to another. In a graph, this would represent if an edge existed between the points or not</li><li>Find the <strong>Degree matrix (D)</strong> of each object. This is a diagonal matrix with entry equal to the number of objects object <strong><em>i</em></strong> is similar to</li><li>Find the <strong>Laplacian (L)</strong> of the Affinity Matrix: <strong>L = A — D</strong></li><li>Find the highest <strong><em>k</em></strong> eigenvectors of the Laplacian Matrix depending on their eigenvalues</li><li>Run<strong><em> k-means</em></strong> on these eigenvectors to cluster the objects into k classes</li></ul><p>You can read about the complete algorithm and its math <a href="https://www.cs.cmu.edu/~aarti/Class/10701/readings/Luxburg06_TR.pdf">here</a>. The implementation of Spectral Clustering in scikit-learn is similar to KMeans:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/696cbb0aee01cbe6624351a93f42ab23/href">https://medium.com/media/696cbb0aee01cbe6624351a93f42ab23/href</a></iframe><p>You will obtain the below perfectly clustered data from the above code:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/383/0*2Wnp83P0Cir5j5qh.png" /></figure><h3>SVD for Removing Background from Videos</h3><p>I have always been curious how all those TV commercials and programs manage to get a cool background behind the actors. While this can be done manually, why put in that much manual effort when you have machine learning?</p><p>Think of how you would distinguish the background of a video from its foreground. The <strong>background of a video is essentially static — it does not see a lot of movement. All the movement is seen in the foreground.</strong> This is the property that we exploit to separate the background from the foreground.</p><p>Here are the steps we can follow for implementing this approach:</p><ul><li>Create matrix M from video — This is done by sampling image snapshots from the video at regular intervals, flattening these image matrices to arrays, and storing them as the columns of matrix M</li><li>We get the following plot for matrix M:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*mjAWYILljQfbVcOq.png" /></figure><p>What do you think these horizontal and wavy lines represent? Take a moment to think about this.</p><p><strong>The horizontal lines represent the pixel values that do not change throughout the video. So essentially, these represent the background in the video. The wavy lines show movement and represent the foreground.</strong></p><ul><li>We can, therefore, think of M as being the sum of two matrices — one representing the background and other the foreground</li><li>The <strong>background matrix</strong> does not see a variation in pixels and is thus redundant i.e. it does not have a lot of unique information. So, it is a <strong>low-rank</strong> matrix</li><li>So, a low-rank approximation of M is the background matrix. We use SVD in this step</li><li>We can obtain the foreground matrix by simply subtracting the background matrix from the matrix M</li></ul><p>Here is a frame of the video after removing the background:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*jgJqQDvmjVj_URMo.jpg" /></figure><p>Pretty impressive, right?</p><p>You can find the original notebook <a href="https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/3.%20Background%20Removal%20with%20Robust%20PCA.ipynb">here</a>.</p><p>We have discussed five very useful applications of SVD so far. But how does the math behind SVD actually work? And how useful is it for us as data scientists? Let’s understand these points in the next section.</p><h3>What is Singular Value Decomposition (SVD)?</h3><p>I have used the term <strong>rank</strong> a lot in this article. In fact, through all the literature on SVD and its applications, you will encounter the term “rank of a matrix” very frequently. So let us start by understanding what this is.</p><h3>Rank of a Matrix</h3><p>The rank of a matrix is the maximum number of linearly independent row (or column) vectors in the matrix. A vector <strong>r</strong> is said to be linearly independent of vectors <strong>r1</strong> and<strong> r2</strong> if it cannot be expressed as a linear combination of <strong>r1</strong> and <strong>r2</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/261/0*sWcZzo0WPSjaPrCV.jpg" /></figure><p>Consider the three matrices below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/699/0*PfD8XjZpXDpiPQIq.jpg" /></figure><ul><li>In matrix A, row <strong>r2</strong> is a multiple of <strong>r1</strong>, <strong><em>r2 = 2 r1</em></strong>, so it has only one independent row. <strong>Rank(A) = 1</strong></li><li>In matrix C, all 3 rows are independent of each other. <strong>Rank(C) = 3</strong></li></ul><blockquote><em>The rank of a matrix can be thought of as a representative of the amount of unique information represented by the matrix. Higher the rank, higher the information.</em></blockquote><h3>Singular Value Decomposition (SVD)</h3><p>So where does SVD fit into the overall picture? SVD deals with decomposing a matrix into a product of 3 matrices as shown:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/212/0*X2pRFxZtv-MkF3_Y.png" /></figure><p>If the dimensions of A are m x n:</p><ul><li>U is an m x m matrix of <strong>Left Singular Vectors</strong></li><li>S is an m x n rectangular diagonal matrix of <strong>Singular Values</strong> arranged in decreasing order</li><li>V is an n x n matrix of <strong>Right Singular Vectors</strong></li></ul><h3>Why is SVD used in Dimensionality Reduction?</h3><p>You might be wondering why we should go through with this seemingly painstaking decomposition. The reason can be understood by an alternate representation of the decomposition. See the figure below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*MMh9iM41jsj890tm.jpg" /></figure><p>The decomposition allows us to express our <strong><em>original matrix as a linear combination of low-rank matrices</em></strong>.</p><p>In a practical application, you will observe that only the first few, say k, singular values are large. The rest of the singular values approach zero. As a result, terms except the first few can be ignored without losing much of the information. See how the matrices are truncated in the figure below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*arP2ZshqOKgXss-h.png" /></figure><p>To summarize:</p><ul><li>Using SVD, we are able to represent our large matrix A by 3 smaller matrices U, S and V</li><li>This is helpful in large computations</li><li>We can obtain a <strong>k-rank approximation</strong> of A. To do this, select the first k singular values and truncate the 3 matrices accordingly</li></ul><h3>3 Ways to Perform SVD in Python</h3><p>We know what SVD is, how it works, and where it is used in the real world. But how can we implement SVD on our own?</p><p>The concept of SVD sounds complex enough. You might be wondering how to find the 3 matrices U, S, and V. It is a long process if we were to calculate these by hand.</p><p>Fortunately, we do not need to perform these calculations manually. We can implement SVD in Python in three simple ways.</p><h3>SVD in NumPy</h3><p>NumPy is the fundamental package for Scientific Computing in Python. It has useful Linear Algebra capabilities along with other applications.</p><p>You can obtain the complete matrices U, S, and V using SVD in <strong>numpy.linalg</strong>. Note that S is a diagonal matrix which means that most of its entries are zeros. This is called a <strong>sparse matrix</strong>. To save space, S is returned as a 1D array of singular values instead of the complete 2D matrix.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fd81e0d6e8a07b710d48b37013b01b74/href">https://medium.com/media/fd81e0d6e8a07b710d48b37013b01b74/href</a></iframe><h3>Truncated SVD in scikit-learn</h3><p>In most common applications, we do not want to find the complete matrices U, S and V. We saw this in dimensionality reduction and Latent Semantic Analysis, remember?</p><p>We are ultimately going to trim our matrices, so why find the complete matrices in the first place?</p><p>In such cases, it is better to use <strong>TruncatedSVD</strong> from <strong>sklearn.decomposition</strong>. You specify the number of features you want in the output as the <strong>n_components</strong> parameter. <em>n_components should be strictly less than the number of features in the input matrix</em>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b08d9770b693be03b18ca0f5c1723abe/href">https://medium.com/media/b08d9770b693be03b18ca0f5c1723abe/href</a></iframe><h3>Randomized SVD in scikit-learn</h3><p>Randomized SVD gives the same results as Truncated SVD and has a faster computation time. While Truncated SVD uses an exact solver ARPACK, Randomized SVD uses approximation techniques.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b4b285de5da550378ed0652ead6e9cd7/href">https://medium.com/media/b4b285de5da550378ed0652ead6e9cd7/href</a></iframe><h3>End Notes</h3><p>I really feel Singular Value Decomposition is underrated. It is an important fundamental concept of Linear Algebra and its applications are so cool! Trust me, what we saw is just a fraction of SVD’s numerous uses.</p><p>I encourage you to check out this <a href="https://www.analyticsvidhya.com/blog/2018/06/comprehensive-guide-recommendation-engine-python/?utm_source=blog&amp;utm_medium=5-applications-singular-value-decomposition-svd-data-science">Comprehensive Guide to build Recommendation Engine from scratch</a> to realize the power of SVD for yourself. Building this project will surely add value to your resume (and enhance your own skillset!).</p><p>Which SVD application(s) impressed you the most? Use the comments section below to let the community know.</p><p><em>Originally published at </em><a href="https://www.analyticsvidhya.com/blog/2019/08/5-applications-singular-value-decomposition-svd-data-science/"><em>https://www.analyticsvidhya.com</em></a><em> on August 5, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=777299940b89" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/master-dimensionality-reduction-with-these-5-must-know-applications-of-singular-value-777299940b89">Master Dimensionality Reduction with these 5 Must-Know Applications of Singular Value…</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[10 Powerful Applications of Linear Algebra in Data Science]]></title>
            <link>https://medium.com/analytics-vidhya/10-powerful-applications-of-linear-algebra-in-data-science-c9c0b6018074?source=rss-78433997a4a------2</link>
            <guid isPermaLink="false">https://medium.com/p/c9c0b6018074</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[linear-algebra]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Khyati Mahendru]]></dc:creator>
            <pubDate>Tue, 23 Jul 2019 03:17:55 GMT</pubDate>
            <atom:updated>2019-07-23T12:17:06.253Z</atom:updated>
            <content:encoded><![CDATA[<h4>with Multiple Resources</h4><h3>Overview</h3><ul><li>Linear algebra powers various and diverse data science algorithms and applications</li><li>Here, we present 10 such applications where linear algebra will help you become a better data scientist</li><li>We have categorized these applications into various fields — Basic Machine Learning, Dimensionality Reduction, Natural Language Processing, and Computer Vision</li></ul><h3>Introduction</h3><p>If Data Science was Batman, Linear Algebra would be Robin. This faithful sidekick is often ignored. But in reality, it powers major areas of Data Science including the hot fields of Natural Language Processing and Computer Vision.</p><p>I have personally seen a LOT of data science enthusiasts skip this subject because they find the math too difficult to understand. When the programming languages for data science offer a plethora of packages for working with data, people don’t bother much with linear algebra.</p><p>That’s a mistake. Linear algebra is behind all the powerful machine learning algorithms we are so familiar with. It is a vital cog in data scientists’ skillset. As we will soon see, you should consider linear algebra as a must-know subject in data science.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uL41IHtWh-1pngu1.png" /></figure><p>And trust me, Linear Algebra really is all-pervasive! It will open up possibilities of working and manipulating data you would not have imagined before.</p><p>In this article, I have explained in detail ten awesome applications of Linear Algebra in Data Science. I have broadly categorized the applications into four fields for your reference:</p><p>I have also provided resources for each application so you can deep dive further into the one(s) which grabs your attention.</p><p><em>Note: Before you read on, I recommend going through this superb article — </em><a href="https://www.analyticsvidhya.com/blog/2017/05/comprehensive-guide-to-linear-algebra/?utm_source=blog&amp;utm_medium=10-applications-linear-algebra-data-science"><em>Linear Algebra for Data Science</em></a><em>. It’s not mandatory for understanding what we will cover here but it’s a valuable article for your budding skillset.</em></p><h3>Table of Contents</h3><ul><li>Why study Linear Algebra?</li></ul><h4>Linear Algebra in Machine Learning</h4><ul><li>Loss functions</li><li>Regularization</li><li>Covariance Matrix</li><li>Support Vector Machine Classification</li></ul><h4>Linear Algebra in Dimensionality Reduction</h4><ul><li>Principal Component Analysis (PCA)</li><li>Singular Value Decomposition (SVD)</li></ul><h4>Linear Algebra in Natural Language Processing</h4><ul><li>Word Embeddings</li><li>Latent Semantic Analysis</li></ul><h4>Linear Algebra in Computer Vision</h4><ul><li>Image Representation as Tensors</li><li>Convolution and Image Processing</li></ul><h3>Why Study Linear Algebra?</h3><p>I have come across this question way too many times. Why should you spend time learning Linear Algebra when you can simply import a package in Python and build your model? It’s a fair question. So, let me present my point of view regarding this.</p><p>I consider Linear Algebra as one of the foundational blocks of Data Science. You cannot build a skyscraper without a strong foundation, can you? Think of this scenario:</p><blockquote><em>You want to reduce the dimensions of your data using Principal Component Analysis (PCA). How would you decide how many Principal Components to preserve if you did not know how it would affect your data? Clearly, you need to know the mechanics of the algorithm to make this decision.</em></blockquote><p>With an understanding of Linear Algebra, you will be able to develop a better intuition for machine learning and deep learning algorithms and not treat them as black boxes. This would allow you to choose proper hyperparameters and develop a better model.</p><p>You would also be able to code algorithms from scratch and make your own variations to them as well. Isn’t this why we love data science in the first place? The ability to experiment and play around with our models? Consider linear algebra as the key to unlock a whole new world.</p><h3>Linear Algebra in Machine Learning</h3><p>The big question — where does linear algebra fit in machine learning? Let’s look at four applications you will all be quite familiar with.</p><h3>1. Loss Functions</h3><p>You must be quite familiar with how a model, say a Linear Regression model, fits a given data:</p><ul><li>You start with some arbitrary prediction function (a linear function for a Linear Regression Model)</li><li>Use it on the independent features of the data to predict the output</li><li>Calculate how far-off the predicted output is from the actual output</li><li>Use these calculated values to optimize your prediction function using some strategy like Gradient Descent</li></ul><p><strong>But wait — how can you calculate how different your prediction is from the expected output? Loss Functions, of course.</strong></p><p>A loss function is an application of the <strong>Vector Norm</strong> in Linear Algebra. The norm of a vector can simply be its magnitude. There are many types of vector norms. I will quickly explain two of them:</p><ul><li><strong>L1 Norm</strong>: Also known as the Manhattan Distance or Taxicab Norm. The L1 Norm is the distance you would travel if you went from the origin to the vector if the only permitted directions are parallel to the axes of the space.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*BL8jsAXIjLoD91OL.jpg" /></figure><p>In this 2D space, you could reach the vector (3, 4) by traveling 3 units along the x-axis and then 4 units parallel to the y-axis (as shown). Or you could travel 4 units along the y-axis first and then 3 units parallel to the x-axis. In either case, you will travel a total of 7 units.</p><ul><li><strong>L2 Norm</strong>: Also known as the Euclidean Distance. L2 Norm is the shortest distance of the vector from the origin as shown by the red path in the figure below:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*zbJQyroGOLrp2frP.jpg" /></figure><p>This distance is calculated using the Pythagoras Theorem (I can see the old math concepts flickering on in your mind!). It is the square root of (3² + 4²), which is equal to 5.</p><p><em>But how is the norm used to find the difference between the predicted values and the expected values?</em> Let’s say the predicted values are stored in a vector and the expected values are stored in a vector <strong><em>E</em></strong>. Then <strong><em>P-E</em></strong> is the difference vector. And the norm of <strong><em>P-E</em></strong> is the total loss for the prediction.</p><h3>2. Regularization</h3><p>Regularization is a very important concept in data science. It’s a technique we use to prevent models from overfitting. <em>Regularization is actually another application of the Norm.</em></p><p>A model is said to overfit when it fits the training data too well. Such a model does not perform well with new data because it has learned even the noise in the training data. It will not be able to generalize on data that it has not seen before. The below illustration sums up this idea really well:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/555/0*VZ5Jx0akxgzkwMSE.png" /></figure><p>Regularization <strong>penalizes overly complex models by adding the norm of the weight vector to the cost function.</strong> Since we want to minimize the cost function, we will need to minimize this norm. This causes unrequired components of the weight vector to reduce to zero and prevents the prediction function from being overly complex.</p><p>You can read the below article to learn about the complete mathematics behind regularization:</p><ul><li><a href="https://www.analyticsvidhya.com/blog/2015/02/avoid-over-fitting-regularization/?utm_source=blog&amp;utm_medium=10-applications-linear-algebra-data-science">How to Avoid Overfitting using Regularization?</a></li></ul><p>The L1 and L2 norms we discussed above are used in two types of regularization:</p><ul><li>L1 regularization used with <strong>Lasso</strong> <strong>Regression</strong></li><li>L2 regularization used with <strong>Ridge Regression</strong></li></ul><p>Refer to our <a href="https://www.analyticsvidhya.com/blog/2016/01/complete-tutorial-ridge-lasso-regression-python/?utm_source=blog&amp;utm_medium=10-applications-linear-algebra-data-science">complete tutorial on Ridge and Lasso Regression in Python</a> to know more about these concepts.</p><h3>3. Covariance Matrix</h3><p>Bivariate analysis is an important step in <strong>data exploration</strong>. We want to study the relationship between pairs of variables. Covariance or Correlation are measures used to study relationships between<strong> two continuous variables</strong>.</p><p><strong>Covariance indicates the direction of the linear relationship between the variables.</strong> A positive covariance indicates that an increase or decrease in one variable is accompanied by the same in another. A negative covariance indicates that an increase or decrease in one is accompanied by the opposite in the other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/776/0*p3srqvhRS62cQ7wv.png" /></figure><p>On the other hand, <strong>correlation is the standardized value of Covariance</strong>. A correlation value tells us both the strength and direction of the linear relationship and has the range from -1 to 1.</p><p>Now, you might be thinking that this is a concept of Statistics and not Linear Algebra. Well, remember I told you Linear Algebra is all-pervasive? Using the concepts of <strong>transpose and matrix multiplication</strong> in Linear Algebra, we have a pretty neat expression for the covariance matrix:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/201/0*w6h8gz-7O79vJkX4.jpg" /></figure><p>Here, X is the standardized data matrix containing all numerical features.</p><p>I encourage you to read our <a href="https://www.analyticsvidhya.com/blog/2016/01/guide-data-exploration/?utm_source=blog&amp;utm_medium=10-applications-linear-algebra-data-science">Complete Tutorial on Data Exploration</a> to know more about the Covariance Matrix, Bivariate Analysis and the other steps involved in Exploratory Data Analysis.</p><h3>4. Support Vector Machine Classification</h3><p>Ah yes, support vector machines. One of the most common classification algorithms that regularly produces impressive results. It is an application of the concept of <strong>Vector Spaces</strong> in Linear Algebra.</p><p>Support Vector Machine, or SVM, is a discriminative classifier that works by finding a decision surface. It is a supervised machine learning algorithm.</p><p>In this algorithm, we plot each data item as a point in an <strong>n-dimensional space</strong> (where n is the number of features you have) with the value of each feature being the value of a particular coordinate. Then, we perform classification by finding the <strong>hyperplane</strong> that differentiates the two classes very well i.e. with the <strong>maximum margin</strong>, which is C is this case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/0*SSnyrcPRDBaieqgq.png" /></figure><p>A hyperplane is a <strong>subspace</strong> whose dimensions are one less than its corresponding vector space, so it would be a straight line for a 2D vector space, a 2D plane for a 3D vector space and so on. Again Vector Norm is used to calculate the margin.</p><p>But what if the data is not linearly separable like the case below?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/0*KzWb0I8wkFeElHdb.png" /></figure><p>Our intuition says that the decision surface has to be a circle or an ellipse, right? But how do you find it? Here, the concept of <strong>Kernel Transformations</strong> comes into play. The idea of transformation from one space to another is very common in Linear Algebra.</p><p>Let’s introduce a variable<strong><em> z = x² + y²</em></strong>. This is how the data looks if we plot it along the z and x-axes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/435/0*4R9m3SdQM7gkVSRA.png" /></figure><p>Now, this is clearly linearly separable by a line<strong><em> z = a</em></strong>, where <em>a</em> is some positive constant. On transforming back to the original space, we get as the decision surface, which is a circle!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/441/0*ifV_FjbvpqVrLinF.png" /></figure><p>And the best part? We do not need to add additional features on our own. SVM has a technique called the <strong>kernel trick</strong>. Read this <a href="https://www.analyticsvidhya.com/blog/2017/09/understaing-support-vector-machine-example-code/?utm_source=blog&amp;utm_medium=10-applications-linear-algebra-data-science">article on Support Vector Machines</a> to learn about SVM, the kernel trick and how to implement it in Python.</p><h3>Dimensionality Reduction</h3><p>You will often work with datasets that have hundreds and even thousands of variables. That’s just how the industry functions. Is it practical to look at each variable and decide which one is more important?</p><p>That doesn’t really make sense. We need to bring down the number of variables to perform any sort of coherent analysis. This is what dimensionality reduction is. Now, let’s look at two commonly used dimensionality reduction methods here.</p><h3>5. Principal Component Analysis (PCA)</h3><p>Principal Component Analysis, or PCA, is an unsupervised dimensionality reduction technique. PCA finds the <strong>directions of maximum variance</strong> and projects the data along them to reduce the dimensions.</p><p>Without going into the math, these directions are the <a href="https://www.youtube.com/watch?v=PFDu9oVAE-g"><strong>eigenvectors</strong></a><strong> of the covariance matrix</strong> of the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/225/0*8Mz0sZBrEhpdAI9Z.jpg" /></figure><p>Eigenvectors for a square matrix are special non-zero vectors whose direction does not change even after applying linear transformation (which means multiplying) with the matrix. They are shown as the red-colored vectors in the figure below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*y8rls0eGrub-3qlM.png" /></figure><p>You can easily implement PCA in Python using the PCA class in the scikit-learn package:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f41db636e89d56d1b2bc1fe04da6ee76/href">https://medium.com/media/f41db636e89d56d1b2bc1fe04da6ee76/href</a></iframe><p>I applied PCA on the <a href="https://scikit-learn.org/stable/auto_examples/datasets/plot_digits_last_image.html">Digits dataset</a> from sklearn — a collection of 8×8 images of handwritten digits. The plot I obtained is rather impressive. The digits appear nicely clustered:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*N-WZjimWMM2MaAIL.png" /></figure><p>Head on to our <a href="https://www.analyticsvidhya.com/blog/2018/08/dimensionality-reduction-techniques-python/?utm_source=blog&amp;utm_medium=10-applications-linear-algebra-data-science">Comprehensive Guide to 12 Dimensionality Reduction techniques with code in Python</a> for a deeper insight into PCA and 11 other Dimensionality Reduction techniques. It is honestly one of the best articles on this topic you will find anywhere.</p><h3>6. Singular Value Decomposition</h3><p>In my opinion, Singular Value Decomposition (SVD) is underrated and not discussed enough. It is an amazing technique of <strong>matrix decomposition</strong> with diverse applications. I will try and cover a few of them in a future article.</p><p>For now, let us talk about SVD in Dimensionality Reduction. Specifically, this is known as <strong>Truncated SVD</strong>.</p><ul><li>We start with the large m x n numerical data matrix A, where m is the number of rows and n is the number of features</li><li>Decompose it into 3 matrices as shown here:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/294/1*wATkNo-sNRt3g9aA5X_3xg.png" /><figcaption><em>Source: hadrienj.github.io</em></figcaption></figure><ul><li>Choose k singular values based on the diagonal matrix and truncate (trim) the 3 matrices accordingly:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*VZayMAFvPAItLOtZoh806w.png" /><figcaption><em>Source: researchgate.net</em></figcaption></figure><ul><li>Finally, multiply the truncated matrices to obtain the transformed matrix <em>A_k</em>. It has the dimensions m x k. So, it has k features with k &lt; n</li></ul><p>Here is the code to implement truncated SVD in Python (it’s quite similar to PCA):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3bb8f30be6f16a23c828489fdc4c9b63/href">https://medium.com/media/3bb8f30be6f16a23c828489fdc4c9b63/href</a></iframe><p>On applying truncated SVD to the Digits data, I got the below plot. You’ll notice that it’s not as well clustered as we obtained after PCA:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*OrELdiQMAXz_oVwi.png" /></figure><h3>Natural Language Processing (NLP)</h3><p><a href="https://courses.analyticsvidhya.com/courses/natural-language-processing-nlp?utm_source=blog&amp;utm_medium=10-applications-linear-algebra-data-science">Natural Language Processing (NLP)</a> is the hottest field in data science right now. This is primarily down to major breakthroughs in the last 18 months. If you were still undecided on which branch to opt for — you should strongly consider NLP.</p><p>So let’s see a couple of interesting applications of linear algebra in NLP. This should help swing your decision!</p><h3>7. Word Embeddings</h3><p>Machine learning algorithms cannot work with raw textual data. We need to convert the text into some numerical and statistical features to create model inputs. There are many ways for engineering features from text data, such as:</p><ol><li>Meta attributes of a text, like word count, special character count, etc.</li><li>NLP attributes of text using Parts-of-Speech tags and Grammar Relations like the number of proper nouns</li><li>Word Vector Notations or Word Embeddings</li></ol><p>Word Embeddings is a way of representing words as <strong>low dimensional vectors</strong> of numbers while preserving their context in the document. These representations are obtained by training different neural networks on a large amount of text which is called a <strong>corpus</strong>. They also help in analyzing syntactic similarity among words:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YgCW96SbuWxSml8x.png" /></figure><p><strong>Word2Vec</strong> and <strong>GloVe</strong> are two popular models to create Word Embeddings.</p><p>I trained my model on the <a href="https://norvig.com/ngrams/shakespeare.txt">Shakespeare corpus</a> after some light preprocessing using Word2Vec and obtained the word embedding for the word ‘world’:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/526/0*fp999PYqnziwQgnT.jpg" /></figure><p>Pretty cool! But what’s even more awesome is the below plot I obtained for the vocabulary. Observe that <em>syntactically similar</em> words are closer together. I have highlighted a few such clusters of words. The results are not perfect but they are still quite amazing:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/0*wkGrjPBR8VWLCo18.png" /></figure><p>There are several other methods to obtain Word Embeddings. Read our article for <a href="https://www.analyticsvidhya.com/blog/2017/06/word-embeddings-count-word2veec/?utm_source=blog&amp;utm_medium=10-applications-linear-algebra-data-science">An Intuitive Understanding of Word Embeddings: From Count Vectors to Word2Vec</a>.</p><h3>8. Latent Semantic Analysis (LSA)</h3><p>What is your first thought when you hear this group of words — “prince, royal, king, noble”? These very different words are <strong>almost synonymous</strong>.</p><p>Now, consider the following sentences:</p><ul><li>The <em>pitcher</em> of the Home team seemed out of form</li><li>There is a <em>pitcher</em> of juice on the table for you to enjoy</li></ul><p>The word ‘pitcher’ has <strong>different meanings based on the other words in the two sentences</strong>. It means a baseball player in the first sentence and a jug of juice in the second.</p><p>Both these sets of words are easy for us humans to interpret with years of experience with the language. But what about machines? Here, the NLP concept of Topic Modeling comes into play:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/0*yToAxym8EsEnGNlM.png" /></figure><p><a href="https://www.analyticsvidhya.com/blog/2016/08/beginners-guide-to-topic-modeling-in-python/?utm_source=blog&amp;utm_medium=10-applications-linear-algebra-data-science"><strong>Topic Modeling</strong></a><strong> is an unsupervised technique to find topics across various text documents.</strong> These topics are nothing but clusters of related words. Each document can have multiple topics. The topic model outputs the various topics, their distributions in each document, and the frequency of different words it contains.</p><p>Latent Semantic Analysis (LSA), or Latent Semantic Indexing, is one of the techniques of Topic Modeling. It is another application of <strong>Singular Value Decomposition</strong>.</p><p>Latent means ‘hidden’. True to its name, LSA attempts to capture the hidden themes or topics from the documents by leveraging the context around the words.</p><p>I will describe the steps in LSA in short so make sure you check out this <a href="https://www.analyticsvidhya.com/blog/2018/10/stepwise-guide-topic-modeling-latent-semantic-analysis/?utm_source=blog&amp;utm_medium=10-applications-linear-algebra-data-science">Simple Introduction to Topic Modeling using Latent Semantic Analysis with code in Python</a> for a proper and in-depth understanding.</p><ul><li>First, generate the <strong>Document-Term</strong> matrix for your data</li><li>Use SVD to decompose the matrix into 3 matrices:</li></ul><ol><li><strong>Document-Topic</strong> matrix</li><li><strong>Topic Importance</strong> Diagonal Matrix</li><li><strong>Topic-term</strong> matrix</li></ol><ul><li>Truncate the matrices based on the importance of topics</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/48033b1b99b1d162f5814a1eb9e5ea64/href">https://medium.com/media/48033b1b99b1d162f5814a1eb9e5ea64/href</a></iframe><p>For a hands-on experience with Natural Language Processing, you can check out our course on <a href="https://courses.analyticsvidhya.com/courses/natural-language-processing-nlp?utm_source=blog&amp;utm_medium=10-applications-linear-algebra-data-science">NLP using Python</a>. The course is beginner-friendly and you get to build 5 real-life projects!</p><h3>Computer Vision</h3><p>Another field of deep learning that is creating waves — <a href="https://courses.analyticsvidhya.com/courses/computer-vision-using-deep-learning-version2?utm_source=blog&amp;utm_medium=10-applications-linear-algebra-data-science">Computer Vision</a>. If you’re looking to expand your skillset beyond tabular data (and you should), then learn how to work with images.</p><p>This will broaden your current understanding of machine learning and also help you crack interviews quickly.</p><h3>9. Image Representation as Tensors</h3><p>How do you account for the ‘vision’ in Computer Vision? Obviously, a computer does not process images as humans do. Like I mentioned earlier, machine learning algorithms need numerical features to work with.</p><p>A digital image is made up of small indivisible units called pixels. Consider the figure below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*UZdjlDBhI6Y0lm2_.png" /></figure><p>This <strong>grayscale image</strong> of the digit zero is made of 8 x 8 = 64 pixels. Each pixel has a value in the range 0 to 255. A value of 0 represents a black pixel and 255 represents a white pixel.</p><p>Conveniently, an <em>m x n</em> grayscale image can be represented as a <strong>2D matrix</strong> with <em>m</em> rows and <em>n</em> columns with the cells containing the respective pixel values:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*oYG7rPneZFiBTEDm.png" /></figure><p>But what about a <strong>colored image</strong>? A colored image is generally stored in the RGB system. Each image can be thought of as being represented by three 2D matrices, one for each R, G and B channel. A pixel value of 0 in the R channel represents zero intensity of the Red color and of 255 represents the full intensity of the Red color.</p><p>Each pixel value is then a combination of the corresponding values in the three channels:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*pZEj9v9pLxANtyVW.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/213/0*ibLyY6J8gSNE9PT8.png" /></figure><p>In reality, instead of using 3 matrices to represent an image, a <strong>tensor</strong> is used. A tensor is a <strong>generalized n-dimensional matrix</strong>. For an RGB image, a 3rd ordered tensor is used. Imagine it as three 2D matrices stacked one behind another:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/570/1*r05c4OUyw1AaFSdMCUWCIg.jpeg" /><figcaption><em>Source: slidesharecdn</em></figcaption></figure><h3>10. Convolution and Image Processing</h3><p><strong>2D Convolution</strong> is a very important operation in image processing. It consists of the below steps:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*Fy0zaPzaHcOkGx1h.gif" /></figure><ol><li>Start with a small matrix of weights, called a <strong>kernel</strong> or a filter</li><li>Slide this kernel on the 2D input data, performing element-wise multiplication</li><li>Add the obtained values and put the sum in a single output pixel</li></ol><p>The function can seem a bit complex but it’s widely used for performing various image processing operations like <strong>sharpening and blurring the images and edge detection</strong>. We just need to know the right kernel for the task we are trying to accomplish. Here are a few kernels you can use:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/623/0*JC4I2syY1TAGuLe4.jpg" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fe0eb3d590b415d40981e2943e2bfb28/href">https://medium.com/media/fe0eb3d590b415d40981e2943e2bfb28/href</a></iframe><p>You can download <a href="https://drive.google.com/file/d/1aM4otWKSsDz1Rof3LZkY055YkYXeO-vf/view">the image I used</a> and try these image processing operations for yourself using the code and the kernels above. Also, try this <a href="https://www.analyticsvidhya.com/blog/2019/04/introduction-image-segmentation-techniques-python/?utm_source=blog&amp;utm_medium=10-applications-linear-algebra-data-science">Computer Vision tutorial on Image Segmentation techniques</a>!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1023/1*35H1dewfnvIoZoVi-wm0Mg.png" /></figure><p>Amazing, right? This is by far my most favorite application of Linear Algebra in Data Science.</p><p>Now that you are acquainted with the basics of Computer Vision, it is time to <a href="https://www.analyticsvidhya.com/blog/2019/03/opencv-functions-computer-vision-python/?utm_source=blog&amp;utm_medium=10-applications-linear-algebra-data-science">start your Computer Vision journey with 16 awesome OpenCV functions</a>. We also have a comprehensive course on <a href="https://courses.analyticsvidhya.com/courses/computer-vision-using-deep-learning-version2?utm_source=blog&amp;utm_medium=10-applications-linear-algebra-data-science">Computer Vision using Deep Learning</a> in which you can work on real-life Computer Vision case studies!</p><h3>End Notes</h3><p>My aim here was to make Linear Algebra a bit more interesting than you might have imagined previously. Personally for me, learning about applications of a subject motivates me to learn more about it.</p><p>I am sure you are as impressed with these applications as I am. Or perhaps you know of some other applications that I could add to the list? Let me know in the comments section below.</p><p><em>Originally published at </em><a href="https://www.analyticsvidhya.com/blog/2019/07/10-applications-linear-algebra-data-science/"><em>https://www.analyticsvidhya.com</em></a><em> on July 23, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c9c0b6018074" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/10-powerful-applications-of-linear-algebra-in-data-science-c9c0b6018074">10 Powerful Applications of Linear Algebra in Data Science</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Heading for a Data Science Interview?]]></title>
            <link>https://medium.com/analytics-vidhya/heading-for-a-data-science-interview-95eeab94ba1d?source=rss-78433997a4a------2</link>
            <guid isPermaLink="false">https://medium.com/p/95eeab94ba1d</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[interview-tips]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[interview]]></category>
            <category><![CDATA[interview-preparation]]></category>
            <dc:creator><![CDATA[Khyati Mahendru]]></dc:creator>
            <pubDate>Wed, 10 Jul 2019 03:07:52 GMT</pubDate>
            <atom:updated>2019-07-11T05:33:59.262Z</atom:updated>
            <content:encoded><![CDATA[<h4>6 Things to Do 1 Day Before your Interview</h4><h3>Introduction</h3><p>You’ve finally done it! You have landed an <a href="https://courses.analyticsvidhya.com/courses/ace-data-science-interviews">interview for a data science role</a>. Now, a day before your interview, you’re not sure what to study. The day is almost here but there is so much to cover!</p><p>Sound familiar?</p><p>Interviews can be daunting — I completely get that. Add in Data Science, and you’ve got yourself a nerve-racking cocktail. Data Science professionals need to combine their technical skills with their soft skills. It’s a tough landscape to navigate.</p><p>Landing the interview is great — but cracking it? That’s where things get <em>really</em> interesting. What should you study? What should you leave out? Is there any cheat code you can apply and simply plug and play it in during the interview?</p><p>If you are in a similar situation — you’ve come to the right place!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MNaLnWgUm-EyNU_N.png" /></figure><p>In this article, I will focus on 6 key things to do a day before your big Data Science Interview, apart from the obvious revision of concepts, to make sure you absolutely nail this opportunity. I will not cover the entire preparation process which should ideally begin months in advance of the actual interview.</p><p><em>Personally, I have often felt under-prepared for interviews because I did not know what to expect out of them. If you have ever felt similarly, the ‘</em><a href="https://courses.analyticsvidhya.com/courses/ace-data-science-interviews"><em>Ace Data Science Interviews</em></a><em>’ course can help. The course breaks the complete </em><a href="https://www.analyticsvidhya.com/blog/2019/02/7-step-process-ace-data-science-interviews/"><em>Interview Process into 7 steps</em></a><em> by taking into account the experiences of taking hundreds of interviews and guides on how to excel at each of these steps.</em></p><h3>Be Thorough with your Data Science Resume</h3><p>The absolute basic of any interview, and especially a data science one. <strong>You should be able to explain everything listed on your resume.</strong> Anything that you could possibly reference, you should be able to speak about it.</p><p>If you’ve listed an NLP project for example, and are unable to explain the details — that’s a MAJOR red flag for the interviewer.</p><p>Use the day before the interview to edit and revise your resume. Cut details that are not required and add new ones if required. Think about each experience and project that you list — does it add something relevant?</p><p>That means your experience with a marketing firm as a non-technical person might not be very relevant for a data science role. You should consider keeping details like that off your resume. <em>Mentioning it will just give the interviewer a sense that you are not clear about what you want from the job.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EDlH55oPtageXcCJ.jpg" /></figure><p>Also, think of how you will go about explaining your work experience. Your account should depict your skills and how they led to progress. Consider the following statements:</p><ul><li>“Used LSTM’s to predict the company’s stock prices.”</li><li>“Used LSTM’s to predict the company’s stock prices with 40% more accuracy than the historical average.”</li></ul><p>Doesn’t the second statement sound way more impressive than the first?</p><p><strong>Make sure to make your achievements are measurable and quantifiable</strong>. This will leave a better impression on the minds of your data science interviewer(s).</p><p><em>I recommend reading our </em><a href="https://www.analyticsvidhya.com/blog/2019/07/how-to-build-effective-data-science-resume-4-key-aspects/"><em>guide to building an effective Data Science Resume</em></a><em>. It mentions the 4 key aspects that will make or break your data science application.</em></p><h3>Study up on your Data Science Projects</h3><p>Much like the other details on your resume, deciding what projects to talk about in your interview is also crucial. If there are any projects irrelevant to the role you’ve applied to, adding it in any way isn’t a great practice. This just shows your interviewer that you cannot prioritize well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/1*zXL3btlFjSvabNqortyXWQ.jpeg" /><figcaption><em>Source: DataOptimal</em></figcaption></figure><p>Shortlist 3 to 4 projects which showcase your best work and prepare yourself to talk about them. These projects could be from your current organization, internships, from some coursework or even independent projects using <a href="https://datahack.analyticsvidhya.com/contest/all/">datasets from Analytics Vidhya</a> or Kaggle. Also, keep in mind that these projects should be relevant to your job profile.</p><p>I keep reiterating this because it is THAT important.</p><p>Let me give you my own example. I had listed a research project on my resume which I had done two years back. In hindsight, I should have left it off since it had nothing relevant to the internship role I was interviewing for — a data analytics intern.</p><p>As I went on explaining what I did in this project, I made the mistake of mentioning the term ‘cubic splines’. The interviewer immediately wanted me to elaborate on cubic splines and I realized that I had dug myself into a hole. And no, I did not get the internship.</p><p>There’s a lesson there for all of you data science enthusiasts! <em>If you are looking for projects, refer to our list of </em><a href="https://www.analyticsvidhya.com/blog/2018/05/24-ultimate-data-science-projects-to-boost-your-knowledge-and-skills/"><em>24 Ultimate Data Science Projects to boost your Knowledge and Skills</em></a><em>.</em></p><h3>Practice Solving Puzzles — A Key Data Science Skill</h3><p>Puzzles are a fairly popular way of evaluating a candidate’s quick thinking and analytical acumen. You need to be logical, creative and good with numbers to solve puzzles.</p><p>Many organizations use puzzles for testing their candidates on their problem-solving skills. They want to know about your thought process and how you approach a problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*rdGKcHbWSJW7u6i0.jpg" /></figure><p>I cannot give you a complete guide to solving each puzzle, but I do have a few tips for you to proceed towards puzzle-solving:</p><ul><li>Approach the problem slowly and understand all the details. Ask for any assumptions if they are not explicitly mentioned</li><li>These are meant to showcase your thought process. So make sure to walk your interviewer through your solution while you think</li><li>Do not stick with an approach for too long. Take cues from your interviewer and modify your approach accordingly</li><li>Realize that it is okay if you were not able to completely solve the puzzle. Different puzzles have different levels of difficulty and not all of them are meant to be solved in one sitting</li></ul><p><em>Try solving the puzzles in our list of </em><a href="https://www.analyticsvidhya.com/blog/2016/07/20-challenging-job-interview-puzzles-which-every-analyst-solve-atleast/"><em>20 Hard Data Science Interview Puzzles that every analyst should solve at least once</em></a>.</p><h3>Prepare to Face Case Studies</h3><p>Organizations use case studies as a means of evaluating candidates on how they approach real-life problems. Case studies are the closest thing to the problems that you would be encountering in your role later on. I have seen freshers struggle the most with this part of the data science interview process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/0*64TZr9XoXC96bzvR.jpg" /></figure><p>The tricky aspect of a case study is that it might not be directly related to data science. For example, I got a case study around how to predict the number of black cars in Delhi NCR right now. It’s a tricky one — but if you have a structured mindset — you’ll knock it out of the park!</p><p>Approaching a case study can appear hard since there is no fixed formula to solve them. But you can use the below points to guide yourself through them:</p><ul><li><strong>Ask a lot of questions.</strong> Whatever questions pop in your head, ask away! It will help you uncover a lot of details that you will require for the solution</li><li><strong>Structure the problem.</strong> This could be organizing all available data into a table. Structuring might unveil some hidden patterns in the data</li><li><strong>Practice</strong>! Try case studies from different domains like retail, healthcare, business, etc. The more you practice, the easier a new problem will feel</li><li>Remember what is important is <strong>good brainstorming and a great discussion</strong>. The goal is not to reach a fixed or pre-defined solution, but rather to find a path to it and show your thought process</li></ul><p>Have a look at some of the case studies on Analytics Vidhya (practice each of them and you’ll be interview-ready in a jiffy):</p><ul><li><a href="https://www.analyticsvidhya.com/blog/2016/04/operational-analytics-case-study-freshers-call-center-optimization/?utm_source=blog&amp;utm_medium=6-essential-tips-should-know-day-before-data-science-interview">Call Center optimization</a></li><li><a href="https://www.analyticsvidhya.com/blog/2016/04/case-study-analytics-interviews-dawn-taxi-aggregators/?utm_source=blog&amp;utm_medium=6-essential-tips-should-know-day-before-data-science-interview">Dawn of Taxi Aggregators</a></li><li><a href="https://www.analyticsvidhya.com/blog/2016/07/solving-case-study-optimize-products-price-online-vendor-level-hard/?utm_source=blog&amp;utm_medium=6-essential-tips-should-know-day-before-data-science-interview">Optimize the Products Price for an Online Vendor</a></li></ul><h3>Research the Job Profile and the Organization</h3><p>Researching the job profile has obvious benefits. You would be able to streamline your preparation based on what is required from the role.</p><p>Sometimes, employers may even ask candidates a question or use a keyword to make sure they read the job description carefully:</p><ul><li>“What technologies do we work with?”</li><li>“What are you expecting from this role?”</li><li>“Can you tell us the latest project our data science team open-sourced?”</li></ul><p>These questions will be dreadful if you didn’t read up on the company and the role.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TFFVXx6y7FEZP90R.jpg" /></figure><p>I highly recommend spending some time reading about the company’s mission, vision and core values. Find out about their key achievements. Try and find the data science set up that they have and what kind of projects they work on. If possible, find out about the hierarchy of the organization and how the data science team fits into it.</p><p>Studying the organization and its structure will help you frame better questions for your interviewers. This shows your enthusiasm and curiosity towards the organization and leaves your interviewers impressed.</p><h3>Review confusing terms</h3><p>Are there any data science terms that have bamboozled you before? I’m sure there are a few — this is true for even experienced data scientists.</p><p>A few confusing terms or concepts that I encourage you to read up on a day before your interview:</p><ul><li>Type I and Type II errors</li><li>Precision and Recall</li><li>False Positive Rate and True Negative Rate</li><li>Business metrics v STatistical metric</li><li>Model deployment</li></ul><p>I frequently have to look up the difference between these terms and I am sure most of you do as well. These can stump you if asked in an interview. You know the answer, but the slight differences just aren’t coming to you.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/559/1*t32ioFWL6qvh37wuhsf9oA.png" /><figcaption>Source: AB Tasty</figcaption></figure><p>Make sure to revise such terms a day before the interview. Refer to our glossary of <a href="https://www.analyticsvidhya.com/glossary-of-common-statistics-and-machine-learning-terms/">common machine learning and data science terms</a> for a quick idea around these concepts.</p><h3>End Notes</h3><p>These are just some last-minute tips. The entire data science interview preparation is a long process. You need to start months in advance and build your profile. There are also multiple rounds in a data science hiring process, including:</p><ol><li>Telephonic Screening</li><li>Assignments</li><li>On-site interview, which has several rounds like technical, case studies, puzzles, guesstimate, and more.</li></ol><p>The ‘ <a href="https://courses.analyticsvidhya.com/courses/ace-data-science-interviews">Ace Data Science Interviews</a> ‘ course covers all of these rounds in detail. The course also has a rich collection of Interview Questions along with many helpful tips and tricks. This could significantly increase your chances of acing your next Data Science Interview. So make sure to check it out!</p><p><em>Originally published at </em><a href="https://www.analyticsvidhya.com/blog/2019/07/6-essential-tips-should-know-day-before-data-science-interview/"><em>https://www.analyticsvidhya.com</em></a><em> on July 10, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=95eeab94ba1d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/heading-for-a-data-science-interview-95eeab94ba1d">Heading for a Data Science Interview?</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[11 Superb Data Science Videos Every Data Scientist Must Watch]]></title>
            <link>https://medium.com/analytics-vidhya/11-superb-data-science-videos-every-data-scientist-must-watch-2ef092d39cb2?source=rss-78433997a4a------2</link>
            <guid isPermaLink="false">https://medium.com/p/2ef092d39cb2</guid>
            <category><![CDATA[youtube]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Khyati Mahendru]]></dc:creator>
            <pubDate>Thu, 04 Jul 2019 03:24:01 GMT</pubDate>
            <atom:updated>2019-07-11T05:35:04.716Z</atom:updated>
            <content:encoded><![CDATA[<h3>Overview</h3><ul><li>Presenting 11 data science videos that will enhance and expand your current skillset</li><li>We have categorized these videos into three fields — Natural Language Processing (NLP), Generative Models, and Reinforcement Learning</li><li>Learn how the concepts in these videos work and build your own data science project!</li></ul><h3>Introduction</h3><p>I love learning and understanding <a href="https://courses.analyticsvidhya.com/courses/applied-machine-learning-beginner-to-professional?utm_source=blog&amp;utm_medium=11-data-science-videos-every-data-scientist-must-know">data science</a> concepts through videos. I simply do not have the time to pour through books and pages of text to understand different ideas and topics. Instead, I get a much better overview of concepts via videos and then pick and choose the topics I want to learn more about.</p><p>The sheer quality and diversity of topics available on platforms like YouTube never ceases to amaze. I recently learned about the amazing XLNet framework for NLP from a video (which I have mentioned below for your consumption). This helped me grasp the concept so I could explore more about XLNet!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/0*YBXmbDTA4fgYmYT5.jpg" /></figure><p>I strongly believe structure is very necessary when we’re learning any concept or topic. I follow that approach each time I write an article as well. That’s why I’ve categorized these videos into their respective domains, primarily Natural Language Processing (NLP), Generative Models and Reinforcement Learning.</p><p>So are you ready to dive in and explore the length and breadth of data science through these fascinating videos?</p><h3>Without any further ado, here are 11 awesome Data Science Videos:</h3><h4>Natural Language Processing (NLP)</h4><ul><li>XLNet explained</li><li>How does Google Duplex work?</li><li>Google’s POEMPORTRAITS: Combining Art and AI</li></ul><h4>Generative Models</h4><ul><li>Dive into Variational Autoencoders!</li><li>Create Facial Animation from Audio</li><li>MuseNet Learned to Compose Mozart, Bon Jovi, and More</li></ul><h4>Reinforcement Learning</h4><ul><li>Teaching the Computer to drive</li><li>Learn how AlphaGo Zero works</li><li>Google DeepMind AI learns to walk</li><li>AI learns to play 2048</li></ul><h4>Bonus</h4><ul><li>Adobe develops AI to detect Photoshopped Faces</li></ul><h3>Natural Language Processing (NLP)</h3><p>Check out Analytics Vidhya’s awesome course on NLP <a href="https://courses.analyticsvidhya.com/courses/natural-language-processing-nlp?utm_source=blog&amp;utm_medium=11-data-science-videos-every-data-scientist-must-know">here</a>.</p><h4>XLNet Explained</h4><p>XLNet is the hottest framework in NLP right now. You simply <em>must</em> be aware of what it is and how it works if you want to carve out a career in this field. I came across this video recently and wanted to share it with the community as soon as possible.</p><p>XLNet is the latest state-of-the-art NLP framework. It has outperformed Google’s BERT on 20 NLP tasks and achieved state-of-the-art results on 18 of them. That is very, very impressive.</p><p><strong>Make sure you check out our article covering XLNet and it’s powerful ability </strong><a href="https://www.analyticsvidhya.com/blog/2019/07/6-powerful-open-source-machine-learning-github-repositories-data-scientists/?utm_source=blog&amp;utm_medium=11-data-science-videos-every-data-scientist-must-know"><strong>here</strong></a><strong>.</strong></p><p>The below video provides a clear explanation of the original XLNet research paper. <em>Note: You might need to know a few </em><a href="https://www.analyticsvidhya.com/blog/2019/06/understanding-transformers-nlp-state-of-the-art-models/?utm_source=blog&amp;utm_medium=11-data-science-videos-every-data-scientist-must-know"><em>NLP concepts</em></a><em> beforehand to truly grasp the inner workings of XLNet.</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FnaOuE9gLbZo%3Ffeature%3Doembed%26start%3D8&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DnaOuE9gLbZo&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FnaOuE9gLbZo%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/b43787b262bff660e740cf214ec67b23/href">https://medium.com/media/b43787b262bff660e740cf214ec67b23/href</a></iframe><h4>How does Google Duplex work?</h4><p>Remember when Sundar Pichai went on stage and sent the whole world into a frenzy when he unveiled Google Duplex in his keynote at Google I/O 2018? I remember listening in complete awe to the super-realistic calls that the AI made.</p><p>It took a bit of time for the data science and NLP community to come up with an explanation as to how Google Duplex actually works. It’s pretty powerful and has the potential to change how we interact with machines.</p><p>So the million dollar question — did Google Duplex pass the Turing Test!? You decide after watching this video:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FIuIpgArEZig%3Ffeature%3Doembed%26start%3D508&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DIuIpgArEZig&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FIuIpgArEZig%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/bc48ec71104b03fec2ea887d488deb51/href">https://medium.com/media/bc48ec71104b03fec2ea887d488deb51/href</a></iframe><h4>Google’s POEMPORTRAITS: Combining Art and AI</h4><p>I am an artist and the prospect of combining any art form with Artificial Intelligence is extremely enticing. In a world where there is so much fear around AI, such applications are more than welcome.</p><p>Google’s POEMPORTRAITS AI has been trained on nineteenth-century poetry using NLP techniques. You can contribute and donate a word to generate your own POEMPORTRAIT. Check out how this awesome concept works:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FL04O83xz42s&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DL04O83xz42s&amp;image=http%3A%2F%2Fi.ytimg.com%2Fvi%2FL04O83xz42s%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/c9e32f0b31ced6f6d572e75fd76aabc4/href">https://medium.com/media/c9e32f0b31ced6f6d572e75fd76aabc4/href</a></iframe><h3>Generative Models</h3><h4>Dive into Variational Autoencoders!</h4><p>Here’s one of our favorite <a href="https://www.analyticsvidhya.com/blog/tag/reinforcement-learning/?utm_source=blog&amp;utm_medium=11-data-science-videos-every-data-scientist-must-know">reinforcement learning</a> experts Xander Streenbrugge from his wonderful ArxivInsights channel.</p><p><a href="https://www.analyticsvidhya.com/blog/2018/05/essentials-of-deep-learning-trudging-into-unsupervised-deep-learning/?utm_source=blog&amp;utm_medium=11-data-science-videos-every-data-scientist-must-know">Variational Autoencoders (VAEs)</a> are powerful generative models with diverse applications. You can generate human faces or synthesize your own music or use VAEs for removing noise from images.</p><p>I like this video a lot. Xander begins with an introduction to basic autoencoders and then goes into VAEs and disentangled beta-VAEs. Quite technical, but explained beautifully and concisely, in typical Xander style.</p><p><em>Xander is coming back to </em><a href="https://www.analyticsvidhya.com/datahack-summit-2019/?utm_source=blog&amp;utm_medium=11-data-science-videos-every-data-scientist-must-know"><em>DataHack Summit this year</em></a><em> so you can hear from him and meet him in person!</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F9zKuYvjFFS8%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D9zKuYvjFFS8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F9zKuYvjFFS8%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/96f7a858ebed359bfc6c58c7a7591d88/href">https://medium.com/media/96f7a858ebed359bfc6c58c7a7591d88/href</a></iframe><h4>Create Facial Animation from Audio</h4><p>I was immediately drawn to the video when I read the title. This is Generative Models at their best! You can not only generate facial animation from audio but also generate different emotions for the same audio. And the facial expressions look incredibly natural.</p><p>If you aren’t following Two Minute Papers, you’re missing out. They regularly churn out videos breaking down the latest developments in easy-to-understand fashion. It’s a gem of a channel.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FZtP3gl_2kBM%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DZtP3gl_2kBM&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FZtP3gl_2kBM%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/d67100d5f79c7f27a2e31fbed969b6e3/href">https://medium.com/media/d67100d5f79c7f27a2e31fbed969b6e3/href</a></iframe><h4>MuseNet Learned to Compose Mozart, Bon Jovi, and More</h4><p>Another entry from the Two Minute Papers archive.</p><p>OpenAI’s MuseNet is a deep neural network that generates musical compositions with different instruments and combines different styles. It uses the same general-purpose unsupervised technology as GPT-2 and the results are amazing.</p><p>Never heard of GPT-2? It’s an NLP framework on par with XLNet. Check out how MuseNet works here:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FpQA8Wzt8wdw%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DpQA8Wzt8wdw&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FpQA8Wzt8wdw%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/0135b47a0c83b8b785bdee0b69bf02c6/href">https://medium.com/media/0135b47a0c83b8b785bdee0b69bf02c6/href</a></iframe><h3>Reinforcement Learning</h3><h4>Teaching a Computer to Drive</h4><p>Self-driving cars have always fascinated me. The sheer scale of an autonomous vehicle’s project is staggering. There are so many components, both on the hardware side as well as the data science side, that need to align for this project to work.</p><p>This is a perfect video for beginners to learn about <a href="https://www.analyticsvidhya.com/blog/2017/07/introduction-to-genetic-algorithm/?utm_source=blog&amp;utm_medium=11-data-science-videos-every-data-scientist-must-know">Genetic Programming</a> and Reinforcement Learning and how they are used to create powerful applications. Simon’s personality kept me hooked until the very end.</p><p>And I am definitely trying the project on my own.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FQv7dhms5dDQ%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DQv7dhms5dDQ&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FQv7dhms5dDQ%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/d494b1d25e1549e41c91a67deb81cd1c/href">https://medium.com/media/d494b1d25e1549e41c91a67deb81cd1c/href</a></iframe><h4>Learn how Google DeepMind’s AlphaGo Zero works</h4><p>Another great video by Xander. He explains Google DeepMind’s popular paper on AlphaGo Zero.</p><p>AlphaGo Zero is a new version of the original AlphaGo program that beat human champion Lee Sedol comprehensively. I recommend reading our article on <a href="https://www.analyticsvidhya.com/blog/2019/01/monte-carlo-tree-search-introduction-algorithm-deepmind-alphago/?utm_source=blog&amp;utm_medium=11-data-science-videos-every-data-scientist-must-know">Monte Carlo Tree Search</a>, the algorithm behind AlphaGo before proceeding to learn about AlphaGo Zero.</p><p>AlphaGo Zero uses Reinforcement Learning to beat the world’s leading Go players without using data from human games.</p><blockquote><em>“AlphaGo Zero surpassed the strength of AlphaGo Lee in three days by winning 100 games to 0, reached the level of AlphaGo Master in 21 days, and exceeded all the old versions in 40 days.”</em></blockquote><blockquote><em>Source: </em><a href="https://en.wikipedia.org/wiki/AlphaGo_Zero"><em>Wikipedia</em></a></blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FMgowR4pq3e8%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DMgowR4pq3e8&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FMgowR4pq3e8%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/297f35b39093126c4600e5232f593944/href">https://medium.com/media/297f35b39093126c4600e5232f593944/href</a></iframe><h4>Google DeepMind’s AI learns to Walk</h4><p>This video is both hilarious and informative. Exactly the type of video I like when I’m learning new things! It was funny to watch the AI learn to walk. But at the same time, it left me marveling at the power of Reinforcement Learning.</p><p>The video discusses 3 papers to try and explain how the AI learned to walk and it is surprisingly simple to understand.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fvideoseries%3Flist%3DPLxt59R_fWVzQIWYn5R-9s8Gy45_5L9fCi%26start%3D0&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D89-5x8iTH3s&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F89-5x8iTH3s%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/668e4b3dd598fb323604fb757a902cd6/href">https://medium.com/media/668e4b3dd598fb323604fb757a902cd6/href</a></iframe><h4>AI learns to play 2048</h4><p>Have you ever played the 2048 game? It is super addictive once you get the hang of it. I used to easily finish games earlier but not anymore. Being a data science enthusiast, I am going to train my computer to play it with the help of this awesome video.</p><p>This is another example of the use of Genetic Programming and Evolutionary Algorithms.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F1g1HCYTX3Rg%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D1g1HCYTX3Rg&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F1g1HCYTX3Rg%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/22017ee8874f1f279f7fb01796d20d91/href">https://medium.com/media/22017ee8874f1f279f7fb01796d20d91/href</a></iframe><h3>BONUS: Adobe develops AI to detect Photoshopped Faces</h3><p>Adobe is a market leader in image and video manipulation software. Other companies have tried, but not many have even gotten close to Adobe’s level.</p><p>Last month, Adobe announced its research efforts to detect manipulated images. High time someone did that! It will soon be impossible to tell real from fake given how quickly GANs have taken over the world.</p><p>Imagine Donald Trump challenging Kim Jong Un to a nuclear war and then claiming that it was a deepfake and shrugging off all responsibility! We need to avoid those situations turning into reality. This video shows how Adobe’s algorithm works and tried to combat fake images:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FTUootD36Xm0%3Ffeature%3Doembed&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Ftime_continue%3D155%26amp%3Bv%3DTUootD36Xm0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FTUootD36Xm0%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/2cb4f95f67e05845704bbfb8b3f9d422/href">https://medium.com/media/2cb4f95f67e05845704bbfb8b3f9d422/href</a></iframe><h3>End Notes</h3><p>I love knowing about the latest research in data science, machine learning and AI. But I find it hard to read papers. It takes a lot of time and effort — something not every data science professional has. I am sure many of you struggle with the same. Consuming videos is the ideal way to get an overview of these concepts.</p><p>You can then pick and choose where your interests lie and try to spin up a project or blog post on it. Trust me, it’s a wonderful way to learn and ingrain new data science concepts.</p><p>What are some of your favorite channels or videos on data science? Let’s discuss in the comments below.</p><p><em>Originally published at </em><a href="https://www.analyticsvidhya.com/blog/2019/07/11-data-science-videos-every-data-scientist-must-watch/"><em>https://www.analyticsvidhya.com</em></a><em> on July 4, 2019.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ef092d39cb2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/11-superb-data-science-videos-every-data-scientist-must-watch-2ef092d39cb2">11 Superb Data Science Videos Every Data Scientist Must Watch</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My Experience at the Google India Tech Intern Connect 2019]]></title>
            <link>https://khyatimahendru.medium.com/my-experience-at-the-google-india-tech-intern-connect-2019-49b88c0db190?source=rss-78433997a4a------2</link>
            <guid isPermaLink="false">https://medium.com/p/49b88c0db190</guid>
            <category><![CDATA[google]]></category>
            <category><![CDATA[internships]]></category>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[google-india]]></category>
            <category><![CDATA[networking-event]]></category>
            <dc:creator><![CDATA[Khyati Mahendru]]></dc:creator>
            <pubDate>Sun, 30 Jun 2019 09:16:20 GMT</pubDate>
            <atom:updated>2019-07-03T11:28:21.315Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B6xLDndMpkiQTuCxs6XbQg.png" /></figure><p>A few months back, I filled a Google form I received in my inbox from Google India. There was a question that really stumped me.</p><p>Which of the following events or programs from Google India are you aware of?</p><p>Here is my answer:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*UvjQrKsrcY6JSWlKsGYjdw.jpeg" /></figure><p>It is highly likely that most of you reading this are unaware of at least one of the programs from the list.</p><p>I immediately took a screenshot and set on researching each of these events.</p><p>I decided to try my level best to attend all of these events this year. So naturally, I was elated when <strong>I received an email from Google India with an invitation to register for the Tech Intern Connect 2019.</strong></p><p>On 24th June 2019, I received yet another email saying that <strong>my application had been accepted and that I was invited for the Tech Intern Connect on 29th June 2019, at the Gurugram office!</strong></p><h4>What is Tech Intern Connect?</h4><p>Tech Intern Connect is a <strong>networking </strong>event organized by Google Interns every year. It is an opportunity to network with Googlers, other tech interns from different organizations and universities and to learn more about Google’s culture. The event is held in the four Google offices — Bangalore, Gurugram, Hyderabad, and Mumbai — simultaneously.</p><p>Here is the agenda that I received in the email:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*a6Xm3Gbodo9hWK14Wba4SA.png" /></figure><h3>My Experience</h3><p>I reached the Google Office sharp at 8:30 AM. The building is so beautiful and definitely took my breath away.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s_2lJS6mbajbQLIMi5DmPg.jpeg" /></figure><p>I went inside and registered. Then I was led to the Cafeteria for breakfast.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GTwRmDFqYc1VPLJid6R2-A.jpeg" /></figure><p>The Cafeteria was huge! My phone camera has not even captured half its beauty. The ambiance was amazing and I felt so lucky to have got the opportunity.</p><p>I chatted up with a few people — quick introductions followed by what everyone was working on. It was definitely a diverse group and I would be lying if I said that I did not feel a little intimidated.</p><h3>Meet the Interns and Google India Overview</h3><p>At 9 AM, all of us went to the hall where the entire event was to happen. There were 2 large projector screens on which we could see the people from the other 3 locations.</p><p>Our <strong>Google swag</strong> waited for us on our seats! I am going to be sneaky and not tell you what was in it :)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vigPF0qO_qskYv7s7knoNg.jpeg" /></figure><p>The event started with a game of <strong>People Bingo</strong> to get us all warmed up to each other. It was a nice ice-breaker.</p><p>This was followed by <strong>a talk by Rohit, the Director of Google Docs</strong> from the Bangalore Office. He shared the details of his journey to Google which surprisingly were relatable. He then discussed some projects that Google Interns had worked in the past and how they were now used in Google Products by people all over the world. It was so inspiring to hear how their work touched the lives of millions. I was particularly motivated by a quote that he shared, which I might have phrased incorrectly.</p><blockquote>If you tell someone where to go but not how to reach there, you’ll be surprised with results.</blockquote><p>He used this to emphasize how his interns were just explained about the problem statement and they worked on the solutions all on their own.</p><h3>Google Codelab</h3><p>This session was taken by two engineers from Google Bangalore who had flown in to be at the Gurugram Office. The session started with an introduction to <a href="https://codingcompetitions.withgoogle.com/kickstart"><strong>Kick Start</strong></a>. And they cleared up a widespread myth.</p><blockquote>No, performing badly in previous Kick Start rounds DOES NOT affect your chances of getting into Google if you perform well in the current round!</blockquote><p>They then taught us the dreaded <strong>Dynamic Programming</strong> which I have always struggled with.</p><p>Due to the time constraint, the session was not as effective as I would have liked it to be. Nonetheless, we got some pointers on how to identify and solve DP problems.</p><p>I was not able to solve the Coding Challenge that they had for us. But I can’t wait to try my hands on it again after I thoroughly revise on Dynamic Programming.</p><h3>Networking Lunch</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3ijxO84TYwCfM4v2aHi3HA.jpeg" /></figure><p>The food was amazing and, of course, free. I got the chance to sit with <strong>Kathita, Product Technical Manager</strong> at Google. She was very nice and interacted with the whole table. She shared her journey to Google and talked to us about our internships. It was a fun, enthusiastic and enlightening conversation.</p><h3>#IamRemarkable Workshop</h3><blockquote>#IamRemarkable is a Google initiative empowering women and underrepresented groups to celebrate their achievements in the workplace and beyond.</blockquote><p>This was <strong>my favorite session </strong>of the day. And it was taken by Kathita!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SR4XJ1R5Pc4jF5M1j-ILkg.jpeg" /></figure><p>I would not be able to describe my feelings and experience in words. I resonated with others’ stories and what they went through.</p><p><strong>Thinking about why I am remarkable and what I am proud of was nerve-wracking but talking about it was incredibly liberating.</strong> Hearing others talk about themselves was amazing and inspirational.</p><p>The energy in the room was very different after the session. It ended with the following video and left me a little tearful.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FYKBgEsH0PJc%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DYKBgEsH0PJc&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FYKBgEsH0PJc%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/16d7f8f2ad76bfe97eb526b93ec39c11/href">https://medium.com/media/16d7f8f2ad76bfe97eb526b93ec39c11/href</a></iframe><h3>3 PM Surprise — Google Theater Company!</h3><p>This was an extremely fun session. Two Googlers — Luba and Siddharth conducted the session on <strong>Improv Comedy</strong>!</p><p>I am a huge fan of “Whose Line is it anyway” and was so excited to witness improv IRL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*G4mHxyvatq8RZU-v" /><figcaption>Source: variety.com</figcaption></figure><p>The session constituted of many, many fun games which us attendees got a chance to participate in. All of us had a blast and laughed our hearts out.</p><p>A perfect commencement to a perfect event!</p><h4>The day’s not over yet…</h4><p>I stuck around for another hour or so to pick off the minds of Googlers Rahul and Basu, present there. They were so warm and encouraging. It was the best pep talk anyone could receive!</p><p>I left the Google office feeling so grateful for the opportunity and so incredibly motivated. This day was definitely the highlight of my year :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=49b88c0db190" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Deal with Imbalanced Data using SMOTE]]></title>
            <link>https://medium.com/analytics-vidhya/balance-your-data-using-smote-98e4d79fcddb?source=rss-78433997a4a------2</link>
            <guid isPermaLink="false">https://medium.com/p/98e4d79fcddb</guid>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[sampling]]></category>
            <category><![CDATA[classification-algorithms]]></category>
            <dc:creator><![CDATA[Khyati Mahendru]]></dc:creator>
            <pubDate>Tue, 25 Jun 2019 03:58:46 GMT</pubDate>
            <atom:updated>2019-06-26T12:21:26.917Z</atom:updated>
            <content:encoded><![CDATA[<h4>With a Case Study in Python</h4><p>With libraries like <strong><em>scikit-learn</em></strong> at our disposal, building classification models is just a matter of minutes. However, building models without properly examining the structure of your data can lead to disastrous results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*yFPrRPSONNyngGNbo243Ag.jpeg" /></figure><p>Imagine that you are a doctor. Sadly, you have discovered a tumor in one of your patients. And you have to investigate if it is cancerous or not. To give a primary diagnosis, you note down some features of the tumor and feed it to your trusted classification model for prediction.</p><p>What if your model predicts that the tumor is benign while in reality, it is cancerous? Tragic! There is a huge cost of this wrong prediction — it might delay treatment and even lead to the death of the patient.</p><h3>The problem with Imbalanced Data</h3><p>In classification problems, balancing your data is absolutely crucial. Data is said to be imbalanced when instances of one class outnumber the other(s) by a large proportion.</p><blockquote>Feeding imbalanced data to your classifier can make it biased in favor of the majority class, simply because it did not have enough data to learn about the minority.</blockquote><p>There are several sampling methods to deal with this. You can (read should) check out the articles below to learn about all of them in detail:</p><ul><li><a href="https://www.analyticsvidhya.com/blog/2016/03/practical-guide-deal-imbalanced-classification-problems/"><em>Practical Guide to Deal with Imbalanced Classification Problems in R</em></a></li><li><a href="https://www.analyticsvidhya.com/blog/2017/03/imbalanced-classification-problem/"><em>How to handle Imbalanced Classification Problems in machine learning?</em></a></li><li><a href="https://www.analyticsvidhya.com/blog/2016/09/this-machine-learning-project-on-imbalanced-data-can-add-value-to-your-resume/"><em>This Machine Learning Project on Imbalanced Data Can Add Value to Your Resume</em></a></li></ul><p>In this article, I will discuss one of the sampling techniques — <strong>S</strong>ynthetic <strong>M</strong>inority <strong>O</strong>ver-Sampling <strong>Te</strong>chnique, abbreviated as <strong>SMOTE</strong>.</p><h3>What is SMOTE?</h3><p>Just like the name suggests, the technique generates synthetic data for the minority class.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*Hjp5xV1i3x0j1g7AZMKiwQ.png" /></figure><p>SMOTE proceeds by joining the points of the minority class with line segments and then places artificial points on these lines.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*nHwVl6AgcP_ym8RbXuKvSg.png" /></figure><p>Under the hood, the <strong>SMOTE algorithm</strong> works in 4 simple steps:</p><ol><li>Choose a minority class input vector</li><li>Find its k nearest neighbors (<strong><em>k_neighbors</em></strong> is specified as an argument in the <strong><em>SMOTE()</em></strong> function)</li><li>Choose one of these neighbors and place a synthetic point anywhere on the line joining the point under consideration and its chosen neighbor</li><li>Repeat the steps until data is balanced</li></ol><p>SMOTE is implemented in Python using the <strong><em>imblearn</em></strong> library.</p><p>I would recommend reading the <a href="https://imbalanced-learn.readthedocs.io/en/stable/generated/imblearn.over_sampling.SMOTE.html">documentation for SMOTE</a> to get acquainted with its various parameters.</p><h3>Credit Card Fraud Detection: SMOTE in Python</h3><p>We will work with <a href="https://www.kaggle.com/mlg-ulb/creditcardfraud/downloads/creditcardfraud.zip/3">this data</a> available at Kaggle.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1d9888aabc86e9d1011831407cf0bddf/href">https://medium.com/media/1d9888aabc86e9d1011831407cf0bddf/href</a></iframe><p>The Python notebook may take time to render. You can also view it <a href="https://nbviewer.jupyter.org/gist/KhyatiMahendru/734da147385bcf630a42ba7c11cfc6c4">here</a>.</p><h3>End Notes</h3><p>There are many sampling techniques for balancing data. SMOTE is just one of them. But, there’s no single best technique. Generally, you need to experiment with a few of them before deciding on one. Make sure to check out the resources I attached above to learn about all the sampling techniques.</p><p>In the end, I leave you with a challenge. Try improving the accuracy of the classifier by using different classification algorithms in combination with SMOTE. Let me know of your results down in the comments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=98e4d79fcddb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/balance-your-data-using-smote-98e4d79fcddb">How to Deal with Imbalanced Data using SMOTE</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Measuring the Goodness of Fit: R² versus Adjusted R²]]></title>
            <link>https://medium.com/analytics-vidhya/measuring-the-goodness-of-fit-r%C2%B2-versus-adjusted-r%C2%B2-1e8ed0b5784a?source=rss-78433997a4a------2</link>
            <guid isPermaLink="false">https://medium.com/p/1e8ed0b5784a</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[linear-regression]]></category>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[regression]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Khyati Mahendru]]></dc:creator>
            <pubDate>Fri, 21 Jun 2019 11:36:01 GMT</pubDate>
            <atom:updated>2019-08-13T12:16:42.851Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Linear Regression</strong> deals with modeling a linear relationship between a dependent variable and several explanatory variables. Essentially, we want to fit a line through our data points in space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*rQoiu4Y51IUMDsOE8aGCyw.jpeg" /></figure><p>But there can be multiple possible lines. Thus to find the best line, there is a need for a measure of goodness of fit, ergo the <strong>Coefficient of Determination — R².</strong></p><p><em>Before you proceed, here is a list of </em><strong><em>5 must-read articles</em></strong><em> on Regression to brush up your basics:</em></p><ul><li><a href="https://www.analyticsvidhya.com/blog/2016/02/complete-tutorial-learn-data-science-scratch/"><em>A Complete Tutorial to learn Data Science in R from scratch</em></a></li><li><a href="https://www.analyticsvidhya.com/blog/2015/10/regression-python-beginners/"><em>5 Questions which can teach you Multiple Regression (with Python and R)</em></a></li><li><a href="https://www.analyticsvidhya.com/blog/2015/08/comprehensive-guide-regression/"><em>7 Types of Regression Techniques you should know!</em></a></li><li><a href="https://www.analyticsvidhya.com/blog/2016/01/complete-tutorial-ridge-lasso-regression-python/"><em>A Complete Tutorial on Ridge and Lasso Regression in Python</em></a></li><li><a href="https://www.analyticsvidhya.com/blog/2016/12/45-questions-to-test-a-data-scientist-on-regression-skill-test-regression-solution/"><em>45 Questions to test a Data Scientist on Regression</em></a></li></ul><p>You can also check out (and participate in) a detailed discussion on this question with answers from other data science experts here:</p><p><a href="https://discuss.analyticsvidhya.com/t/difference-between-r-square-and-adjusted-r-square/264/15"><em>https://discuss.analyticsvidhya.com/t/difference-between-r-square-and-adjusted-r-square/264/15</em></a></p><h3>Coefficient of Determination — R²</h3><p>Consider the case of <strong>simple regression</strong> as shown in the figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*LabByiMu_qYfdBgxucZ8HA.png" /><figcaption>As explained in Gujarati, Damodar N. <em>Basic econometrics</em>. Tata McGraw-Hill Education, 2009.</figcaption></figure><p>Circle Y represents the <strong>variation</strong> in our dependent variable, circle X, in our independent variable.</p><p>The overlap of the circles represents the <strong>extent to which the variation in Y is explained by the variation in X.</strong></p><p>The coefficient <em>r²</em> is the numerical measure of this overlap and is called the <strong>Coefficient of Determination</strong>. This becomes the <strong>Multiple Coefficient of Determination</strong> <em>R²</em> in the case of Multiple Regression.</p><p>To state formally,</p><blockquote>R² measures the proportion of the total variation in Y explained by the regression model.</blockquote><p>This variation is measured in terms of the sum of squared errors from the mean.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/1*2cSAB2SRrR2Gz-9rLv3lvQ.png" /></figure><p>If we have <strong><em>n</em></strong> observations <strong><em>(X_i, Y_i)</em></strong> and <strong><em>f(.)</em></strong> is our predictor function,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/448/1*XsaopfnMdjd2R8paaf2RLA.png" /></figure><p>Here are some key points about R²:</p><ul><li>It is a non-negative quantity with range <strong><em>0 ≤ R² ≤ 1</em></strong></li><li>R² = 0 implies that the regression line does not fit the data at all.</li><li>R² = 1 implies that the regression line is a perfect fit.</li></ul><h4>Problem with R² — Value increases with the number of explanatory variables</h4><p>Think about it. R² is the ratio of the explained variance to the total variance. On adding a new variable the explained variance and hence the value of R² <strong><em>will increase, or at least, will not decrease</em></strong>.</p><p>However, this <strong><em>does not at all</em></strong> mean that the model with the added variable is better than the model without it. R² can be misleading if used to compare models with a different number of predictors.</p><h3>Adjusted R²</h3><p>Adjusted R² is a modified version of R² adjusted with the number of predictors. It penalizes for adding unnecessary features and allows a comparison of regression models with a different number of predictors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/565/1*KJlygowBOFqPmWnz9zvclQ.png" /></figure><p>Here <strong><em>k</em></strong> is the number of explanatory variables in the model and <strong><em>n</em></strong> is the number of observations.</p><p><strong>The value of adjusted R² is always less than that of R².</strong></p><blockquote>The adjusted R-squared increases only if the new term improves the model more than would be expected by chance. It decreases when a predictor improves the model by less than expected by chance.</blockquote><p>Also, note that the value of adjusted R² can be negative.</p><p>Obtaining a negative value for Adjusted R² can indicate few or all of the following:</p><ul><li>The linear model is a poor fit for the data</li><li>The number of predictors is large</li><li>The number of samples is small</li></ul><h3>R² and Adjusted R² in Python</h3><p>Generate a random dataset first</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e7190de68b0a3223d6b70570f65a87fc/href">https://medium.com/media/e7190de68b0a3223d6b70570f65a87fc/href</a></iframe><p><strong><em>X</em></strong> has 6 features. This is how the dependent variable <strong><em>y</em></strong> varies with each of these features:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/847/1*ZGgNRxB-iTkJ7uT0IDqaSg.png" /></figure><p><em>y versus X[2]</em> or <em>y versus X[4] </em>could possibly be linear relationships. Let us fit a line using <strong>Ordinary Least Squares</strong> Regression between <strong><em>y</em></strong> and <strong><em>X[4] </em></strong>using the<strong><em> statsmodels</em></strong> library in Python. The summary() function can be used to view the R² and Adjusted R² coefficients.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e115047c96077203e87f115fcf2532e8/href">https://medium.com/media/e115047c96077203e87f115fcf2532e8/href</a></iframe><p>Here is the summary:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/438/1*xSxRKaTT4FqG6jVjGFQZew.jpeg" /></figure><p>Now add another explanatory variable, <strong><em>X[3]</em></strong>, to see the effect on both the coefficients.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*5j1qbjfUxxycRlMm1o5O8w.jpeg" /></figure><p>Interesting! The value of R² increased while the value of adjusted R² dropped. <strong><em>X[3]</em></strong> is an insignificant feature to add to the linear relationship. This is further confirmed on finding the adjusted R² for the OLS regression on <strong><em>Y</em></strong> versus <strong><em>X[3]</em></strong> individually. We observe that it is negative.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/417/1*xLk1L3REnKI4z5FG6AEb-Q.jpeg" /></figure><p>You can also use the <strong><em>r2_score</em></strong> function from the metrics module in <strong>sklearn.</strong> However, there is no such function to find the adjusted r2_score. We need to use its formula to calculate it.</p><p>Here is the regression analysis of the famous <a href="https://www.kaggle.com/ruiromanini/mtcars"><strong><em>mtcars</em></strong></a> dataset without the categorical variables in sklearn:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3c576a65f93ec586d563a59dd837b277/href">https://medium.com/media/3c576a65f93ec586d563a59dd837b277/href</a></iframe><p>We obtain an R² score of about 0.81 and an adjusted R² score of about 0.77</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*h0Z8ZKtTPW9G-TOP00FWOw.png" /></figure><h3>End Notes</h3><p>Although both R² and adjusted R² are measures of goodness of fit, linear regression models should not be built with the sole goal of maximizing these coefficients. This will eventually tempt us to introduce many insignificant predictors in our model. As a result, our model will be inaccurate and unreliable.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1e8ed0b5784a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/measuring-the-goodness-of-fit-r%C2%B2-versus-adjusted-r%C2%B2-1e8ed0b5784a">Measuring the Goodness of Fit: R² versus Adjusted R²</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decision Trees: Which feature to split on?]]></title>
            <link>https://medium.com/analytics-vidhya/decision-trees-which-feature-to-split-on-91083fc32279?source=rss-78433997a4a------2</link>
            <guid isPermaLink="false">https://medium.com/p/91083fc32279</guid>
            <category><![CDATA[decision-tree]]></category>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Khyati Mahendru]]></dc:creator>
            <pubDate>Wed, 19 Jun 2019 05:47:45 GMT</pubDate>
            <atom:updated>2019-08-13T11:51:23.529Z</atom:updated>
            <content:encoded><![CDATA[<p>Decision Tree classification is perhaps one of the most intuitive and easy-to-interpret classification algorithms we have today. It can easily model non-linear relationships and results in predictive models that are quite accurate and stable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/1*KbYxWwafL6qSm3yUuCWyaQ.jpeg" /></figure><p>Learning in <a href="https://courses.analyticsvidhya.com/courses/applied-machine-learning-beginner-to-professional?utm_source=medium&amp;utm_medium=DecisionTreesWhichFeatureToSplitOn">Decision Tree Classification</a> has the following <strong>key features:</strong></p><ul><li>We recursively <strong>split</strong> our population into two or more sub-populations based on a <strong>feature</strong>. This can be visualized as a tree, with its root node representing the entire population and its subnodes representing the subpopulations</li><li>With each subnode that is added to the tree, the <strong>homogeneity or purity increases</strong> with respect to the target variable (or class)</li><li>The leaf nodes are completely pure and contain objects of a single class</li></ul><p>In this article, I will discuss the following metrics used for deciding which feature to split on:</p><ol><li><strong>Gini Index</strong></li><li><strong>Entropy</strong></li></ol><p><em>Before proceeding, I </em><strong><em>strongly</em></strong><em> recommend you to read this </em><a href="https://www.analyticsvidhya.com/blog/2016/04/complete-tutorial-tree-based-modeling-scratch-in-python/#three"><em>complete tutorial on Tree-based modeling from scratch (in R and Python)</em></a><em>.</em></p><p>We will use the two metrics on the following example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*Hdf_qBtDfwcryc5Nhw8cng.png" /></figure><p><strong><em>Note:</em></strong> We are using only categorical variables for a better illustration. However, continuous variables are also possible. In the case of categorical variables, we need to find a threshold value for splitting. For example, age ≤ 18 and age&gt;18 is a possible splitting if age was a continuous variable.</p><h3>Gini Index</h3><p>Gini Index is used with binary splits, where one class can be considered a <strong>success</strong> and other a <strong>failure</strong>.</p><blockquote>A higher value of the Gini Index indicates more homogeneity in the sub-nodes.</blockquote><p>The Gini-Index for a split is calculated in two steps:</p><ol><li>For each <strong>subnode</strong>, calculate Gini as <strong>p² + q²</strong>, where p is the probability of success and q of failure</li><li>Then for the <strong>split</strong>, Gini is equal to the <strong>weighted average</strong> of the Gini scores of the subnodes</li></ol><p>Let us calculate the Gini Scores for our example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*Gsb8_lw-WUmRyzbvbPl8pw.png" /></figure><p>Here, our <strong><em>success</em></strong> class is defined as <strong><em>using the computer daily</em></strong>. The corresponding probabilities of success and failure are calculated for both subnodes as shown.</p><p>Then Gini for Male subnode = 0.5² + 0.5² = 0.5</p><p>Gini for Female subnode = 0.33² + 0.67² = 0.56</p><p>Finally,<strong> Gini for the split on gender</strong> = (20 x 0.5 + 30 x 0.56) / 50 = 0.536</p><p>Similarly, we calculate the Gini for splitting on our categorical age variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*ZvBiVjvQXKP9A4HCeL_3Ig.png" /></figure><p>Gini for Adult subnode = 0.48² + 0.42² = 0.4068</p><p>Gini for Minor subnode = 0.32² + 0.68² = 0.5648</p><p>Thus, <strong>Gini for split on age</strong> = (25 x 0.4068 + 25 x 0.5648) / 50 = 0.4856</p><p>Since, Gini for split on gender is greater than on age, <strong>gender is chosen as the feature to split on.</strong> The resultant subnodes — Males and Females — divide the population into more homogenous subpopulations.</p><p>You might come across the term <strong><em>Gini Impurity</em></strong>. It is defined as:</p><blockquote>Gini Impurity = 1 -Gini</blockquote><p>Thus equivalently, we need to find the feature that <strong>minimizes the Gini Impurity of the split.</strong></p><p>We can easily implement Decision Trees with the Gini Index using the <em>sklearn</em> library in Python. This is done by specifying the value of the <strong><em>criterion</em></strong> attribute as <strong>‘<em>gini’</em></strong> in the DecisionTreeClassifier() function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/25942891e08f793323463236915589d8/href">https://medium.com/media/25942891e08f793323463236915589d8/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fV4Ymcmq7lqYrfaBl_T_vw.png" /><figcaption>Decision Tree for the Iris Dataset with gini value at each node</figcaption></figure><h3>Entropy</h3><p>The entropy is a metric frequently used to measure the <strong>uncertainty</strong> in a distribution. For a binary classification with the probability of success p and of failure q, entropy is calculated as follows:</p><p>Entropy(p, q) = -plog2(p)-qlog2(q), where log2(.) is log with base 2.</p><blockquote>Since we want our subnodes to be homogenous, we desire to <strong>split on the feature with the minimum value for entropy.</strong></blockquote><p>Here’s how you calculate the value for the entropy of a split:</p><ol><li>For each <strong>subnode</strong>, calculate entropy using the formula mentioned above.</li><li>Then the entropy for the <strong>split</strong> is the weighted average of entropy for all the subnodes.</li></ol><p>Let’s calculate the entropy values for our example.</p><h4><strong>Split on Gender:</strong></h4><p>Entropy for the Male subnode = -0.5 x log2(0.5)-0.5 x log2(0.5) = 1</p><p>Entropy for the Female subnode = -0.33 x log2(0.33)-0.67 x log2(0.67) = 0.91</p><p>Then, <strong>entropy for split on gender </strong>= (20 x 1 + 30 x 0.91) / 50 = 0.946</p><h4><strong>Split on Age:</strong></h4><p>Entropy for the Adult subnode = -0.48 x log2(0.48)-0.42 x log2(0.42) = 1.033</p><p>Entropy for the Minor subnode = -0.32 x log2(0.32)-0.68 x log2(0.68) = 0.904</p><p>Then, <strong>entropy for split on age</strong> = (25 x 1.033 + 25 x 0.904) / 50 = 0.968</p><p>Again, we find that Gender is a better feature for the split than Age.</p><p>Just like Gini, DecisionTreeClassifer() can use the <strong>‘entropy’ criterion</strong> to decide the splitting.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a7e31764da71d4c297347f6892d1c19d/href">https://medium.com/media/a7e31764da71d4c297347f6892d1c19d/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mJ04dy02Khstb_rl48_NHw.png" /><figcaption>Decision Tree for the Iris Dataset with entropy value at each node</figcaption></figure><h3>End Notes</h3><p>It is not coincidental that the Gini Index and Entropy suggest the same feature to split on. The two objectives are:</p><ol><li>Minimize Gini Impurity = 1- (p² + q²)</li><li>Minimize Entropy = -plog2(p)-qlog2(q)</li></ol><p>These are more or less the same. However, since entropy requires logarithm to be computed, it will be slower in some cases. Generally, though, they will provide similar performance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=91083fc32279" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/decision-trees-which-feature-to-split-on-91083fc32279">Decision Trees: Which feature to split on?</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to determine the optimal K for K-Means?]]></title>
            <link>https://medium.com/analytics-vidhya/how-to-determine-the-optimal-k-for-k-means-708505d204eb?source=rss-78433997a4a------2</link>
            <guid isPermaLink="false">https://medium.com/p/708505d204eb</guid>
            <category><![CDATA[unsupervised-learning]]></category>
            <category><![CDATA[analytics-vidhya]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Khyati Mahendru]]></dc:creator>
            <pubDate>Mon, 17 Jun 2019 12:51:35 GMT</pubDate>
            <atom:updated>2019-06-17T12:51:35.163Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to Determine the Optimal K for K-Means?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/1*OJ-dzKJdbwVJu_P7INWuyg.png" /></figure><h3>Introduction</h3><p>The K-Means algorithm needs no introduction. It is simple and perhaps the most commonly used algorithm for clustering.</p><p>The basic idea behind k-means consists of defining k clusters such that total<strong> within-cluster variation (or error) is minimum</strong>.</p><p><em>I encourage you to check out the below articles for an in-depth explanation of different methods of clustering before proceeding further:</em></p><ul><li><a href="https://www.analyticsvidhya.com/blog/2016/11/an-introduction-to-clustering-and-different-methods-of-clustering/"><em>An Introduction to Clustering and different methods of Clustering</em></a></li><li><a href="https://www.analyticsvidhya.com/blog/2019/05/beginners-guide-hierarchical-clustering/"><em>A Beginner’s Guide to Hierarchical Clustering and how to perform it in Python</em></a></li></ul><p>A cluster center is the representative of its cluster. The squared distance between each point and its cluster center is the required variation. The aim of k-means clustering is to find these k clusters and their centers while reducing the total error.</p><p>Quite an elegant algorithm. But there is a catch. How do you decide the number of clusters?</p><p>In this article, I will explain in detail two methods that can be useful to find this mysterious k in k-Means.</p><p>These methods are:</p><ol><li><strong>The Elbow Method</strong></li><li><strong>The Silhouette Method</strong></li></ol><p>We will use our own dataset generated by the code below for an illustration of the two methods:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8199e578ef9158a87288f4c07db77bf0/href">https://medium.com/media/8199e578ef9158a87288f4c07db77bf0/href</a></iframe><p>This is how the data looks graphically:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/1*B1zpDSV9aG-c8jVoJ8ttvg.png" /></figure><p>Clearly, the dataset has 3 clusters. We will validate both our methods on this dataset.</p><h3>The Elbow Method</h3><p>This is probably the most well-known method for determining the optimal number of clusters.<em> It is also a bit naive in its approach.</em></p><blockquote>Calculate the <strong>Within-Cluster-Sum of Squared</strong> Errors (WSS) for <strong>different values of k</strong>, and choose the k for which WSS becomes first starts to diminish. In the plot of WSS-versus-k, this is visible as an <strong>elbow.</strong></blockquote><p>Within-Cluster-Sum of Squared Errors sounds a bit complex. Let’s break it down:</p><ul><li>The Squared Error for each point is the square of the distance of the point from its representation i.e. its predicted cluster center.</li><li>The WSS score is the sum of these Squared Errors for all the points.</li><li>Any distance metric like the Euclidean Distance or the Manhattan Distance can be used.</li></ul><p>Let us implement this in Python using the<strong><em> sklearn</em></strong> library and our own function for calculating WSS for a range of values for k.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8c3e5d478722135f8a1d4e3527e8a003/href">https://medium.com/media/8c3e5d478722135f8a1d4e3527e8a003/href</a></iframe><p>We obtain the following plot for WSS-vs-k for our dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/407/1*O_JmBi6rM6PlrPFx_uNrVQ.png" /></figure><p>As expected, the <strong>plot looks like an arm with a clear elbow</strong> <strong>at k = 3.</strong></p><p>Unfortunately, we do not always have such clearly clustered data. This means that the elbow may not be clear and sharp.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*t7VgOuC_Yp-Y1ND8z-eFtQ.png" /><figcaption>Source: bl.ocks.org/</figcaption></figure><p>For Dataset A, the elbow is clear at k = 3. However, this choice is ambiguous for Dataset B. We could choose k to be either 3 or 4.</p><p>In such an ambiguous case, we may use the Silhouette Method.</p><h3>The Silhouette Method</h3><blockquote>The silhouette value measures how similar a point is to its own cluster (cohesion) compared to other clusters (separation).</blockquote><blockquote>Source: Wikipedia</blockquote><p>The range of the Silhouette value is between +1 and -1. A <strong>high value is desirable</strong> and indicates that the point is placed in the correct cluster. If many points have a negative Silhouette value, it may indicate that we have created too many or too few clusters.</p><p>The Silhouette Value <strong><em>s(i)</em></strong> for each data point <strong><em>i</em></strong> is defined as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/295/1*vFXfyIjIjI_kbZNWODiuxQ.jpeg" /><figcaption>Source: Wikipedia</figcaption></figure><p><strong><em>Note</em></strong>: s(i) is defined to be equal to zero if <strong><em>i</em></strong> is the only point in the cluster. This is to prevent the number of clusters from increasing significantly with many single-point clusters.</p><p>Here, <strong><em>a(i)</em></strong> is the measure of similarity of the point <strong><em>i</em></strong> to its own cluster. It is measured as the average distance of <strong><em>i</em></strong> from other points in the cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/403/1*OFOXVdMIwldl8diJvscw_A.jpeg" /><figcaption>Source: Wikipedia</figcaption></figure><p>Similarly, <strong><em>b(i)</em></strong> is the measure of dissimilarity of <strong><em>i</em></strong> from points in other clusters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/293/1*fGgqT3IHwpEOiDbCtN5dAQ.jpeg" /><figcaption>Source: Wikipedia</figcaption></figure><p><strong><em>d(i, j)</em></strong> is the distance between points <strong><em>i</em></strong> and <strong><em>j</em></strong>. Generally, <strong>Euclidean Distance</strong> is used as the distance metric.</p><p>The Silhouette score can be easily calculated in Python using the metrics module of the <strong><em>sklearn</em></strong> library.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0b23366cdaea99c76e200472de5ead71/href">https://medium.com/media/0b23366cdaea99c76e200472de5ead71/href</a></iframe><p>I mentioned before that a high Silhouette Score is desirable. The Silhouette Score reaches its <strong><em>global maximum at the optimal k</em></strong>. This should ideally appear as a peak in the Silhouette Value-versus-k plot.</p><p>Here is the plot for our own dataset:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/1*Otg9XRLPqOjb80Png6B9Og.png" /></figure><p>There is a clear peak at k = 3. Hence, it is optimal.</p><p>Finally, the data can be <strong><em>optimally</em></strong> clustered into 3 clusters as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/1*EN3KSIqQGQNhqmEcbA1pbw.png" /></figure><h3>End Notes</h3><p>The Elbow Method is more of a decision rule, while the Silhouette is a metric used for validation while clustering. Thus, it can be used in combination with the Elbow Method.</p><p>Therefore, the Elbow Method and the Silhouette Method are not alternatives to each other for finding the optimal K. Rather they are tools to be used together for a more confident decision.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=708505d204eb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/how-to-determine-the-optimal-k-for-k-means-708505d204eb">How to determine the optimal K for K-Means?</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>