<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Venkata Sandeep on Medium]]></title>
        <description><![CDATA[Stories by Venkata Sandeep on Medium]]></description>
        <link>https://medium.com/@venkatasandeepd?source=rss-db1a063707cb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*X6s-u0QcLdsInBRq1LcESw.jpeg</url>
            <title>Stories by Venkata Sandeep on Medium</title>
            <link>https://medium.com/@venkatasandeepd?source=rss-db1a063707cb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 04:16:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@venkatasandeepd/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Principal Component Analysis — Inside Out]]></title>
            <link>https://venkatasandeepd.medium.com/principal-component-analysis-inside-out-a8f1f522c37d?source=rss-db1a063707cb------2</link>
            <guid isPermaLink="false">https://medium.com/p/a8f1f522c37d</guid>
            <category><![CDATA[principal-component]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[dimensionality-reduction]]></category>
            <dc:creator><![CDATA[Venkata Sandeep]]></dc:creator>
            <pubDate>Wed, 10 Jun 2020 04:02:19 GMT</pubDate>
            <atom:updated>2020-06-10T04:02:19.397Z</atom:updated>
            <content:encoded><![CDATA[<h3>Principal Component Analysis — Inside Out</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BSfcwLMa5rBVSr6I" /><figcaption>Photo by <a href="https://unsplash.com/@lishakov?utm_source=medium&amp;utm_medium=referral">Andrej Lišakov</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Before we move into the internals of Principal Component Analysis(PCA), it is important to understand the concepts of linear algebra such as coordinate systems, <a href="https://www.youtube.com/watch?v=PFDu9oVAE-g">Eigen Vectors and Values</a>, <a href="https://www.youtube.com/watch?v=P2LTAUO1TdA">linear transformations</a> and Eigendecomposition etc. Please brush them up before jumping further into this post.</p><p>Before going to learn PCA, we need to understand what lead to this and why.</p><h3>Exploratory Data Analysis(EDA)</h3><p>EDA is the first and foremost step in data analysis. To put it simply, EDA is a process of taking a peek into your data to find any patterns, correlations and anomalies and even to assess assumptions. It gives us preliminary understanding of our data.</p><p>Remember, we assume all the features are IID in an ideal regression setup. If there is any correlation between variables, then our prediction model might not be good.</p><p><strong>Problems in EDA : Curse of Dimensionality <br></strong> It refers to the problems associated with multivariate data analysis as the dimensionality increases. It means, for a given sample size, there is a maximum number of features above which the performance of our classifier<br>will degrade rather than improve. How can we handle this ?</p><p>The number of regions of a regular grid grows exponentially with the dimensionality D of the space:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ClIp_3xT8fCoEgr2L9yh9Q.png" /></figure><p><strong>Problems in EDA : Multi-collinearity</strong><br>One of the processes involved in EDA is to find any correlation between variables or features. A pair plot produces visualization of correlation between variables in pairs of two over all the variables available in the dataset. Suppose we have 20 variables, then we have to plot 190 plots. It is very hard to visualize or to interpret the relation. How can we visualize data in higher dimensions?</p><p>How can we solve these problems? One of the obvious ways is to reduce the features in the dataset. But how can we do that without losing the information in the data ? Well, there are few techniques. They are</p><blockquote><strong>Feature Extraction</strong> - creating new features by combining the existing features.</blockquote><blockquote><strong>Feature Selection</strong> - which includes dropping some features etc. We may lose some information by doing this.</blockquote><p><em>Feature extraction</em> is commonly limited to linear projections. Two main concepts of this are</p><blockquote><strong>Signal representation -</strong> the goal is to represent the objects accurately in a lower dimensional space; e.g., <strong>principal component analysis (PCA)</strong>.</blockquote><blockquote><strong>Classification -</strong> The goal is to enhance the class-discriminatory information in the lower-dimensional space; e.g., <strong>linear discriminant analysis (LDA)</strong>.</blockquote><p><strong>Properties and informal goals of PCA:</strong></p><ol><li>Reducing the dimensions or number of features such as they retain most of the information as of original dataset.</li><li>The derived features must be uncorrelated with/ independent of each other.</li><li>Principal components are linear combination of original variables.</li><li>PCA is an <strong><em>unsupervised</em></strong> technique: there is no ‘Y’ or dependent/response variable.</li></ol><p><strong>Features of PCA:</strong></p><ol><li>Dimensionality reduction.</li><li>Data visualization and Exploratory Data Analysis.</li><li>Create uncorrelated features/variables that can be an input to a prediction model.</li><li>Uncovering latent variables/themes/concepts.</li><li>Noise reduction in the dataset.</li></ol><h3>Building PCA</h3><p>To understand how PCA is build, we need to understand the concepts of <strong><em>basis of a space</em>, <em>basis transformation</em> </strong>and<strong> <em>variance as information</em></strong>.</p><p>The non-linear coordinate system can be transformed into linear space using transformation. The new space obtained can be called as feature space. More about this can be found in my previous<strong><em> </em></strong><a href="https://medium.com/@venkatasandeepd/all-about-support-vector-machines-4b43b920db4c"><strong><em>post about SVM</em></strong></a>. <strong><em>Basis of space and transformation</em></strong> can be <strong><em>better understood by this </em></strong><a href="https://www.youtube.com/watch?v=P2LTAUO1TdA"><strong><em>video</em></strong></a>.</p><p>In short, the vectors(column/features) we have and the rows (axis/dimensions) will be represented in other set of basis(for eg., cartesian coordinates can be represented in polar coordinates form). That is, we apply linear transformation such that the new feature we derived is a <strong><em>linear combination of our basis vectors</em></strong>.</p><p><strong><em>Variance as information</em></strong> can be defined as variables which captures the variance in the data are the ones that holds the information in the data. If we have a variable which has only one common value in the dataset, it really doesn’t add any value to the prediction model. Further, if two variables are very highly correlated, they together don’t add a lot information than they do individually. So we can drop one of them.</p><h4>Step 1:</h4><p>Choose how many principal components we want, let’s call it <strong><em>M</em></strong>. And we have <strong><em>p</em></strong> features.</p><p>The first principal component is calculated such that it captures maximum variance in the data. Let’s call it z1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6jmiKH0wbHQIvf7aWs2S-g.png" /></figure><blockquote>The first principal component can be defined as finding a hyper plane that is close to the data.</blockquote><p>Then we calculate our second principal component(z2) such that it captures maximum variance of all the linear combination of variables which are uncorrelated with z1. That is <strong><em>z1 and z2 are</em></strong> <strong><em>orthogonal</em></strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lLsZcVKsBG9fm4Dygg6hGg.png" /></figure><p>Similarly we calculate principal components upto M i.e <strong><em>zM, where M &lt;&lt; p.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/334/0*3O_Rfc48woxB_nFO.gif" /><figcaption><a href="https://online.stat.psu.edu/stat508/book/export/html/639?source=post_page-----8489b46c2ae0----------------------">Source</a></figcaption></figure><h4>Step 2 : Calculating Principal Components</h4><p>There are two ways in which we can calculate the principal components.</p><p><strong>2.1 Eigen Decomposition of covariance matrix</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NTNLKjWC2vUVMR1P9qbxoA.png" /></figure><p><strong>2.2 Singular Value Decomposition(SVD)</strong></p><p>SVD is general form of Eigen Decomposition. It breaks the single matrix A(original dataset mxn) into three parts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cj7ZNLCyfZxTI01JMI2tbg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*9l-GbGecWPrj9e-j.png" /><figcaption>Source: <a href="https://intoli.com/blog/pca-and-svd/">https://intoli.com/blog/pca-and-svd/</a></figcaption></figure><p>Based on our M value i.e number of components, we ignore the zero and lower order rows and columns and reconstruct the matrix. We will get lower dimension matrix.</p><p>While using SVD, PCA is scale sensitive, data standardization is mandatory.</p><p>More about SVD can be found in this post.</p><p><a href="https://blog.statsbot.co/singular-value-decomposition-tutorial-52c695315254">Singular Value Decomposition (SVD) Tutorial: Applications, Examples, Exercises</a></p><h4>Step 3 : How to choose number of Principal Components ?</h4><p>Based on how much variance we capture, we choose the number of components.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/1*c7EuXhT-5TsmF7h-Cv3mLA.png" /></figure><ul><li>Against the number of principal components, we look at the cumulative proportion of variance captured.</li><li>Choose <strong><em>k</em></strong> after which incremental benefit negligible.</li><li>Common choices: 80%, 90%, 95% variance.</li></ul><h4>Non-Linear PCA</h4><p>Traditional PCA applies linear transformations. It may not be effective for nonlinear data. So we apply a nonlinear transformation to map objects<br>to a potentially very high-dimensional space. We can do this by using <strong><em>Kernel Trick </em></strong>which was discussed in this <a href="https://medium.com/@venkatasandeepd/all-about-support-vector-machines-4b43b920db4c"><strong><em>post about SVM</em></strong></a><strong><em>.</em></strong></p><p>Or as an alternative, we can go for <strong>t-SNE. </strong>t-Distributed Stochastic Neighbor Embedding (t-SNE) is a non-linear technique for dimensionality reduction that is particularly well suited for the visualization of high-dimensional datasets. It is extensively applied in image processing, NLP, genomic data and speech processing.</p><ul><li>t-SNE calculates the probability of similarity of points in high-dimensional space and calculating the probability of similarity of points in the corresponding low-dimensional space. The similarity of points is calculated by using conditional probability under Normal distribution.</li><li>It then tries to minimize the difference between these conditional probabilities (or similarities) in higher-dimensional and lower-dimensional space for a perfect representation of data points in lower-dimensional space.</li></ul><p>More about t-SNE can be found <a href="http://www.jmlr.org/papers/volume9/vandermaaten08a/vandermaaten08a.pdf"><strong><em>here</em></strong></a></p><h4>Notes</h4><p>The main limitation of PCA is that it does not consider class separability</p><ul><li>PCA simply performs a coordinate rotation that aligns the transformed axes with the directions of maximum variance. There is no guarantee that the directions of maximum variance will contain good features for discrimination.</li></ul><h4>Implementation of Image compression and recovery through dimensionality reduction using PCA in Matlab</h4><blockquote>Note: Eigen Decomposition is used in the repo below.</blockquote><p><a href="https://github.com/venkatasandeepd/pca_matlab">venkatasandeepd/pca_matlab</a></p><h4>References</h4><ol><li><a href="https://arxiv.org/pdf/1404.1100.pdf">https://arxiv.org/pdf/1404.1100.pdf</a></li><li><a href="https://intoli.com/blog/pca-and-svd/">https://intoli.com/blog/pca-and-svd/</a></li><li><a href="https://blog.statsbot.co/singular-value-decomposition-tutorial-52c695315254">https://blog.statsbot.co/singular-value-decomposition-tutorial-52c695315254</a></li><li><a href="http://www.jmlr.org/papers/volume9/vandermaaten08a/vandermaaten08a.pdf">http://www.jmlr.org/papers/volume9/vandermaaten08a/vandermaaten08a.pdf</a></li><li><a href="https://medium.com/@venkatasandeepd/all-about-support-vector-machines-4b43b920db4c">https://medium.com/@venkatasandeepd/all-about-support-vector-machines-4b43b920db4c</a></li></ol><p><strong><em>Thanks for reading :-)</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a8f1f522c37d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[All about Support Vector Machines]]></title>
            <link>https://venkatasandeepd.medium.com/all-about-support-vector-machines-4b43b920db4c?source=rss-db1a063707cb------2</link>
            <guid isPermaLink="false">https://medium.com/p/4b43b920db4c</guid>
            <category><![CDATA[support-vector-machine]]></category>
            <category><![CDATA[machıne]]></category>
            <dc:creator><![CDATA[Venkata Sandeep]]></dc:creator>
            <pubDate>Wed, 08 Apr 2020 03:10:34 GMT</pubDate>
            <atom:updated>2020-04-08T03:10:34.893Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uFglSvEgnctDoq1QgRankw.png" /><figcaption>Source : <a href="https://analyticsprofile.com/wp-content/uploads/2019/06/10-1-1024x500.png">https://analyticsprofile.com/wp-content/uploads/2019/06/10-1-1024x500.png</a></figcaption></figure><p><strong><em>Support Vector Machines</em></strong>, or SVMs were conceived by Vladimir Vapnik as competitor for neural nets (and in his book he explained that neural nets with one hidden layer can be considered to be SVMs). For a while they completely eclipsed neural nets. But with the rise of Deep Learning, neural nets are in the game again.</p><p>So, what really is <strong><em>SVM</em></strong>, here it goes…</p><h3>All about SVM</h3><p>This article assumes you know classification models, i.e. <strong>logistic regression </strong>and<strong> naive Bayes</strong>. If you don’t, please read them first. Though these models are widely used in industries such as banking, e-commerce etc. and applications such as default prediction and text classification, they find limited use in more complex classification problems, such as image classification.</p><p><strong><em>SVMs</em></strong>, a class of extremely popular classification models, are capable of dealing with quite complex problems such as image recognition, voice detection etc. where models such as logistic regression typically fail. Besides their ability to solve complex machine learning problems, they have numerous other advantages over other classification problems, such as the ability to deal with computationally heavy data sets, classifying non-linearly separable data, etc.</p><p>SVMs are mostly used for <strong>classification tasks</strong>, but they can also be used for regression. The topics covered in this article are limited to <strong>classification tasks</strong>. In this article, we will see how SVMs can be implemented in email spam detection problem.</p><h4>1. Concept of Hyperplane</h4><p>As told above, lets take an email dataset and our task is to separate spam and ham emails. For our convenience lets assume the dataset features and labels as below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FgjwRuCjO6pgZtNuW1R1_A.png" /></figure><p>We have plotted two features of email dataset, typically this is text corpus of words we use to classify a mail is a spam or not, the numbers are word counts. We represent each data point <em>(x,y)</em> as a feature vector of two features in 2 dimension. We draw a line that separates the green pints(spam) and the red points(ham). A<strong> line </strong>that is used to classify one class from another is also called a <strong>hyperplane</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ozj_2ICxkxRaMkCZKSWZUg.png" /></figure><p>The standard equation of a line is given by <strong>ax+by+c = 0</strong>. You could generalize it as W0+W1x1+W2x2=0, where x1 and x2 are the features and W1 and W2 are the coefficients. For any line with W coefficients, substituting the value of features x1 and x2 in the equation of the line determined by its W coefficients, will return a value.</p><p>A <strong>positive value</strong> (green points in the plot above) would mean that the set of values of the features is in one class; however, a <strong>negative value</strong> (red points in the plot above) would imply it belongs to the other class. A value of zero would imply that the point lies on the line (hyperplane) because any point on the line will satisfy the equation: W0+W1x1+W2x2=0.</p><p>This is what a <strong>2-dimensional hyperplane </strong>looks like. Now, what do you think a hyperplane will look like in <strong>3 dimensions</strong>? In general, we can conclude that the <strong>hyperplane</strong> in <strong>2D</strong> can be represented by a <strong>line</strong>, whereas a <strong>hyperplane</strong> in <strong>3D </strong>can be represented by a <strong>plane</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/1*n0RFL_SyeO79THX2tZmkBw.jpeg" /></figure><p>In 3 dimensions the <strong>hyperplane</strong> (light orange) will be a plane with an expression of <strong>ax+by+cz+d = 0</strong>. The plane divides the data set into two halves. Data points <strong>above the plane</strong> represent <strong>one class (green)</strong>, while data points <strong>below the plane</strong> represent the <strong>other class (red)</strong>.</p><p>Imagine we have a datapoint in d- dimensions, the equation goes like this…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VMhIa-TfRfM4XY4vZ6sYjA.png" /></figure><p>We need to find the coefficients in such a way that the equation satisfies the following conditions for every data point in the set. This is called <strong><em>Linear Discriminator.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/1*uTqqOhkKSsGHpyi0GmThIQ.png" /></figure><p>After finding the coefficients, we can find the separating hyperplane that separates two classes of emails. Now what ? The problem is we get more than one hyperplane.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HNTcQziPW_my2_M3UoxZyQ.png" /></figure><p>So which hyperplane we can use for classification. Which is the best <strong><em>line </em></strong>? Here <strong><em>Maximal Margin Classifier</em></strong> comes into play.</p><h4>2. Maximal Margin Classifier</h4><p>As we have seen above, there are multiple hyperplanes.</p><blockquote>The best hyperplane is one that maintains <strong>largest possible equal distance</strong> from the nearest points of both the classes.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/673/1*jUdxuPFOCvjB9QtLntA9NA.png" /></figure><p>We call this equidistance as <strong><em>margin. </em></strong>The maximum margin hyperplane represents the mid-line of the widest <strong><em>slab </em></strong>that we can insert between the two classes. In the figure above, three training samples are equidistant from the maximum margin hyperplane and lie along the dashed lines indicating the width of the margin. These three samples are known as <strong><em>support vectors</em></strong>.</p><p><strong>Support vectors </strong>are the points that <strong>lie close to the hyperplane</strong>. In fact, they are <strong>the only points that are used in constructing the hyperplane</strong>. So why these are called support vectors. Lets say, I remove all the data points except the closest points to the hyperplane we calculated. Then we calculate the hyperplane only with the remaining points. Still we get the same hyperplane we got earlier. So these are the only points that matter for us the most.</p><p>Again, if we take the farthest points from the hyper plane, it’s very easy to classify those extremities even without looking at hyperplane. These are of less importance to us.</p><p>How can we maximize the margin. Certainly there are two constraints we should follow to maximize the margin.</p><p><strong><em>Constraint 1 : </em></strong><em><br>Standardizing /Normalizing the coefficients such that the summation of the square of the coefficients of all the attributes is equal to 1.This objective means that we are minimizing the </em><strong><em>Euclidean length of w</em></strong><em> which means we are maximizing the margin.</em></p><p><strong><em>Constraint 2 :<br></em></strong><em>The hyperplane should be a safe distance of more than or equal to the </em><strong><em>margin ‘M’ </em></strong><em>from both the labels.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n4GQrQcMApgKL9NImElnkw.png" /></figure><p>This guarantees that each sample will be on the correct side of the hyperplane with some cushion.</p><p>The <strong><em>maximal margin line (hyperplane)</em></strong>, although it separates the two classes perfectly, is very sensitive to the training data. This means that the <strong>Maximal Margin Classifier</strong> will perform perfectly on the training data set. But on the unseen data, it may perform poorly. Also, there are cases where the classes cannot be perfectly separated. For example, if you look at figure below, the maximal margin can not classify the data points perfectly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RX2qqTvmvyNw-omgnVHOFw.png" /></figure><p>It seems that the maximal margin line (hyperplane) is not even possible in this case. In this case, if you want to create a linear hyperplane, you will inevitably need to misclassify a few data points. In other words, some <strong>points will need to fall on the wrong side</strong> of the hyperplane.</p><p>This problem led to the idea of the <strong>Soft Margin Classifier</strong>.</p><h4>3. Soft Margin Classifier</h4><p>The hyperplane that allows certain points to be <strong>deliberately</strong> misclassified is also called the <strong>Support Vector Classifier</strong>. The support vector classifier works well when the data is <strong>partially intermingled</strong> (i.e. most of the data can be classified correctly with some misclassifications).</p><p>Similar to the Maximal Margin Classifier, the Support Vector Classifier also maximizes the margin; but it will also allow some points to be misclassified.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bHbuexIUxKwX5VCpdd0frg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vCx0-8BAVFVGaZqNWVI4uw.png" /></figure><p>Real world data is not clean and neatly separated. To the figure on the right, we can estimate a hyperplane which allows some misclassifications. But we can not even draw an optimal hyperplane to the data on the left. We will discuss on how to apply classification to that problems later (in the kernel module).</p><p>Lets come to problem at hand, we need to estimate an optimal hyperplane which maximizes the margin and keeps the misclassification as low as possible. For this purpose slack variable is introduced.</p><h4>4. Slack variable (<strong>ε</strong>)</h4><p>A slack variable is used to control <strong>misclassifications</strong>. It tells you where an observation is located relative to the margin and hyperplane. For points which are at a distance of more than M, i.e. at a safe distance from the hyperplane, the value of the slack variable is 0.</p><blockquote><strong><em>ε</em></strong> can be simply defined as tolerance level we allow to every data point.</blockquote><p>Lets see how this effects our equation of hyperplane</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bb9PZCcMBeR7hO7oGgdRVg.png" /></figure><p>But we want to ensure that the cumulative error we commit with every data point is something we can control that we don’t want that to go out of hand.</p><p>So, we put one more constraint along with two constraints we discussed in maximal margin classifier, which is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z_DqfbpJr2CnDJ7U3A5sBg.png" /></figure><p>If we allowing C to be large, you are allowing the model to make lot of errors.</p><p>If we allowing C to be very small, we are pushing the optimization routine to find strict separator.</p><p>But there should be a bias variance trade-off on how we choose C value. If <strong>C is large</strong>, the model will be having <strong>high bias </strong>as<strong> </strong>the model is<strong> flexible, more generalisable, and less likely to overfit</strong> and vice versa if we choose small C value. On other notes, we can find best C for the model using Cross-Validation and GridSearch.</p><blockquote>Note : <strong>The hyper-parameter C has a different meaning in the python SVC() implementation. </strong>The parameter C that we discussed and the C in the SVC() function in Python are the<strong> inverse of each other.</strong></blockquote><blockquote>In the SVC() implementation of python that you will use, the <strong>hyper-parameter</strong> C is analogous to the <strong>penalty imposed for misclassification</strong>, i.e. a higher C will force the model to classify most (training) data points correctly (and thus, overfit). In sklearn, <strong>high value of C</strong> implies a <strong>high cost of making errors </strong>or misclassifications.</blockquote><h4>5. Kernels</h4><p>So far, we have learnt about hyperplanes, the Maximal Margin Classifier, and the Support Vector Classifier. All of these are <strong>linear models</strong> (since they use linear hyperplanes to separate the classes).</p><p>However, many real-world data sets are not separable by linear boundaries. For instance, what if the distribution of data points looks like the figure given below?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bHbuexIUxKwX5VCpdd0frg.png" /></figure><p>We can not separate the data points using a linear hyperplane. We need to tweak the linear SVM model and enable it to incorporate non-linearity in some way. <strong>Kernels</strong> serve this purpose — they enable the linear SVM model to separate non-linearly separable data points.</p><h4>5.1. Mapping Nonlinear Data to Linear Data</h4><p>The figure shown above is non-linear data. Do you have any idea on how to separate them? Of course the span data points form an ellipse/circle. But how to transform elliptical data into linear space. Remember the ellipse equation from high school ?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HSxkhYVHoWRSzovO3LnqMg.png" /></figure><p>The new data points obtained after transformation is called <strong><em>feature space. </em></strong>In this case the we can see the data is in circular or elliptical form, but it is highly uncommon that real world data looks like that. So we can assume a basic quadratic equation for the transformation. Considering we have only two features, the possible quadratic equation of degree 2 will be</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JBUmzmL66tmHgCnON-qzCg.png" /></figure><p>But we have a problem here, after transformation, our features have increased 3 fold in degree 2. But in general we have 100’s of attributes in the dataset. Assume how many features we might get after transforming them to get a linear model. This is where <strong><em>Kernel</em></strong> comes into play.</p><p><strong>5.2 . Kernel Trick</strong></p><p>A <strong><em>kernel</em></strong> is a function that quantifies the similarity between two samples. We take a feature mapping F : X -&gt;H of the sample space <strong><em>X</em></strong> into a feature space <strong><em>H</em></strong><br>equipped with dot product (formally, a Hilbert space). This gives us the kernel<br><br>In general, a kernel is any function that can be obtained this way. Any such kernel can be used in the <strong><em>“kernel trick”</em></strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/394/1*ZDElI7rHNYR4xTM7XDTDEw.png" /></figure><blockquote><strong>Kernel trick: </strong>Write the algorithm in such a way that all x’s(input space) only appear in dot products and then replace all dot products with kernels.</blockquote><p>The three most popular types of kernel functions are:</p><ol><li><strong>The linear kernel:</strong> This is the same as the support vector classifier, or the hyperplane, without any transformation at all</li><li><strong>The polynomial kernel:</strong> It is capable of creating nonlinear, polynomial decision boundaries</li><li><strong>The radial basis function (RBF) kernel: </strong>This is the most complex one, which is capable of transforming highly nonlinear feature spaces to linear ones. It is even capable of creating elliptical (i.e. enclosed) decision boundaries. In RBF, you’ll need to choose two tuning parameters: <strong>gamma</strong> and ‘<strong>C</strong>’. The hyper-parameter gamma<strong> controls the amount of non-linearity </strong>in the model — as gamma increases, the model becomes more non-linear, and thus model complexity increases.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SO8tSq_0IWAWDSSMlDfbsg.png" /><figcaption><a href="http://www.3leafnodes.com/support-vector-machines-classification">Source : http://www.3leafnodes.com/support-vector-machines-classification</a></figcaption></figure><p>Kernels are little tricky to understand. Kernel functions bypass the steps of finding a suitable transformation function ϕ by doing the same thing implicitly. The advantage of such a kernel function is that the complexity of the optimization problem remains only dependent on the dimensionality of the input space and not of the feature space.</p><p>For better understanding of kernels, please read <br><a href="https://cogsys.uni-bamberg.de/teaching/ss06/hs_svm/slides/SVM_Seminarbericht_Hofmann.pdf">https://cogsys.uni-bamberg.de/teaching/ss06/hs_svm/slides/SVM_Seminarbericht_Hofmann.pdf</a></p><h4>6. Strengths and Weakness of SVM</h4><p><strong><em>Pros</em></strong> : <br> → Support vector machines are powerful models and perform well on a variety of datasets. They allow for complex decision boundaries, even if the data has only a few features. They work well on low-dimensional and high-dimensional data (i.e., few and many features).</p><p><strong><em>Cons : <br> → </em></strong>They don’t scale very well with the number of samples.<br> → They require careful pre-processing of the data and tuning of the parameters.<br> → SVM models are hard to inspect; it can be difficult to understand why a particular prediction was made.<br> → Like neural nets, they work best with homogeneous data.</p><h3>Resources</h3><ol><li><a href="https://www.mathworks.com/help/stats/support-vector-machines-for-binary-classification.html#bsr5oqx">https://www.mathworks.com/help/stats/support-vector-machines-for-binary-classification.html#bsr5oqx</a></li><li><a href="http://cs229.stanford.edu/notes/cs229-notes3.pdf">http://cs229.stanford.edu/notes/cs229-notes3.pdf</a></li><li><a href="http://pyml.sourceforge.net/doc/howto.pdf">http://pyml.sourceforge.net/doc/howto.pdf</a></li><li><a href="https://www.ugpti.org/smartse/resources/downloads/support-vector-machines.pdf">https://www.ugpti.org/smartse/resources/downloads/support-vector-machines.pdf</a></li></ol><p><strong><em>Thanks for reading :)</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b43b920db4c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Guideline to Conformal Prediction]]></title>
            <link>https://medium.com/analytics-vidhya/a-guideline-to-conformal-prediction-7a392fc29bc1?source=rss-db1a063707cb------2</link>
            <guid isPermaLink="false">https://medium.com/p/7a392fc29bc1</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[conformal-prediction]]></category>
            <dc:creator><![CDATA[Venkata Sandeep]]></dc:creator>
            <pubDate>Sun, 29 Mar 2020 03:42:29 GMT</pubDate>
            <atom:updated>2020-11-26T08:31:42.790Z</atom:updated>
            <content:encoded><![CDATA[<p>While predicting an event or a point in vector space, we need a guaranteed validity of the outcome. This already exists in statistics as confidence intervals or prediction intervals. It can be embedded into machine learning by conformal prediction, which is relatively recent.</p><blockquote>The idea of conformal prediction is to to predict a label to the given test observation based on past experiences.</blockquote><p>Conformal prediction uses past experiences to determine precise levels of confidence to new predictions. We calculate non-conformity score for each observation, which measures how unusual our observation looks relative to previous examples. The conformal prediction algorithm, using the nonconformity measure, produces a prediction region for every probability of error ε.</p><p>In layman’s terms, if given a training set and test set, we try in turn each potential label for the test set. For each postulated label, we look at how plausible the extended training set is. We can make a confident prediction if all but one completion looks implausible. To evaluate the implausibility of the extended training set we use the statistical notion of a <em>p-value.</em></p><h4>Steps</h4><blockquote>Calculating conformity measure</blockquote><p>Conformity measure is a function which assigns a conformity score to every sample in the dataset. A conformity score(<em>∝</em>) is what defines how well a sample(<em>z</em>) in the dataset conforms to rest of the dataset. If <em>∝</em> is small, we say <em>z</em> is <em>non-conforming</em> or <em>strange</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h69ItjKhRq0PJBSz5AzQ8A.png" /></figure><h4>Validity and efficiency</h4><p>Conformal predictors satisfy the following property of validity automatically</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/165/1*r_GID6LdxsAeE-J24QH-yg.png" /><figcaption>i.e.the predictor makes a mistake with probability <strong><em>ε</em></strong> at most (provided the labelled samples are Independent and Identically Distributed)</figcaption></figure><p>Conformal prediction is usually very inefficient if the prediction set/dataset is very large as we have to calculate conformity measure for every sample in the test set and make predictions. If we need to increase efficiency, the prediction set must be small.</p><h4>Conformal Prediction for classification (Nearest Neighbor implementation)</h4><p>In nearest neighbor classification we can use conformal prediction idea in two ways.</p><ol><li>Calculating the distance to the nearest sample of different class</li><li>One over the distance to the nearest sample of same class</li></ol><p>And combining the two ways,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V72ISKDtSCLaZgJJq23UqA.png" /></figure><h4>Non-Conformity measures</h4><p>Formally, nonconformity measures are defined in the same way as conformity measures. But their interpretation is different i.e</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aeepndtz4-vFmYDYGGtpIg.png" /></figure><p>In principle, it does not matter whether you use non-conformity measures (the Royal Holloway convention) or conformity measures (the Carnegie Mellon convention). But in application to regression non-conformity scores are often more convenient.</p><p>Let’s do an exercise.</p><p><strong><em>Training set:</em></strong><br> 1. Positive samples: (0, 3), (2, 2), (3, 3)<br> 2. Negative samples: (-1, 1), (-1, -1), (0, 1)<br><strong><em>Test sample:</em></strong> (0, 0)<br>Let us calculate the euclidean distance from the new sample to each training sample.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jwHYxYJPse4E0WcBb4rJPw.png" /></figure><p>As conformity measure, use the distance to the nearest sample of a different class divided by the distance to the nearest sample of the same class.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jKqtJ5ePgFKNRPo_EJhlVw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dPLV8VitXcKlmhfw_QStxg.png" /></figure><blockquote>1. Assume the label of (0, 0) is +1. The test sample is the strangest, so the p-value is 1/7 = 0.143.</blockquote><blockquote>2. Assume the label of (0, 0) is -1. The test sample is the second most conforming; its rank is 6, so the p-value is 6/7 = 0.857.</blockquote><p>The <em>p-values</em> are 0.143 (for +1) and 0.857 (for -1). We can predict -1, but our prediction does not achieve statistical significance of 5% for that we should at least have 20 training observations.</p><p>Conformal prediction can also be applied for regression models, which is not covered in this article.</p><h4>Implementation of Conformal Prediction in Python</h4><pre>def calculate_distance(X_tr, y_tr):<br>    train_length = X_tr.shape[0]<br>    same_class_dist, other_class_dist = [[math.inf for i in range(train_length)] for j in range(2)]<br><br>    for i in range(train_length-1):<br>        for j in range(i+1,train_length):<br>            distance = np.linalg.norm(X_tr[i]-X_tr[j])<br><br>            if y_tr[i]==y_tr[j]:<br>                if distance &lt; same_class_dist[i]:<br>                    same_class_dist[i] = distance<br>                if distance &lt; same_class_dist[j]:<br>                    same_class_dist[j] = distance<br>            else:<br>                if distance &lt; other_class_dist[i]:<br>                    other_class_dist[i] = distance<br>                if distance &lt; other_class_dist[j]:<br>                    other_class_dist[j] = distance<br><br>    return [same_class_dist, other_class_dist]</pre><pre>def conformal_prediction(X, y, dataset, test_size=0.3, train_size=0.7, random_state=3006):<br>    predicted_list, p_values = [[] for i in range(2)]<br>    # splitting the data<br>    X_train,X_test,y_train,y_test = train_test_split(X, y, test_size=test_size, train_size=train_size, random_state=random_state)<br>    lenrange = len(list(set(y_train)))<br>    same_class_dist, other_class_dist = calculate_distance(X_train, y_train)<br><br>    for i in range(len(X_test)):<br>        conformity_scores = [[] for j in range(lenrange)]<br>        curr_testXval = X_test[i]<br>        for j in range(lenrange):<br>            new_same_dist = np.append(same_class_dist, math.inf)<br>            new_other_class_dist = np.append(other_class_dist, math.inf)<br>            extended_X = np.concatenate((X_train, [curr_testXval]), axis = 0)<br>            extended_y = np.concatenate((y_train, [j]), axis = 0)<br><br>            for curr_idx, curr_elem in enumerate(extended_X):<br>                distance = np.linalg.norm(curr_elem - curr_testXval)<br>                idx = len(extended_X)-1<br><br>                if distance != 0: #to avoid duplicate value<br>                    if j == extended_y[curr_idx]:<br>                        if distance &lt; new_same_dist[idx]:<br>                            new_same_dist[idx] = distance<br>                    else:<br>                        if distance &lt; new_other_class_dist[idx]:<br>                            new_other_class_dist[idx] = distance<br><br>                if new_same_dist[curr_idx] == 0: #to avoid duplicate value<br>                    conformity_scores[j].append(0)<br>                else:<br>                    conformity_scores[j].append(new_other_class_dist[curr_idx]/new_same_dist[curr_idx])<br><br>        p_vals = []<br>        for k in range(lenrange):<br>            p_vals.append(np.mean(conformity_scores[k]&lt;=conformity_scores[k][X_train.shape[0]]))<br><br>        predicted_list.append(p_vals.index(max(p_vals)))<br>        p_values.append(p_vals)<br><br>    falsep = []<br>    for i, p in enumerate(p_values):<br>        sumval = 0;<br>        for j, q in enumerate(p):<br>            if j != y_test[i]:<br>                sumval += q<br>        falsep.append(sumval)<br><br>    false_p_value = np.sum(falsep)/(len(falsep)*2)<br>    accuracy = np.mean(predicted_list == y_test)<br><br>    print(&quot;For {}, &quot;<br>          &quot;The average false p-value : {} \n&quot;<br>          &quot;The accuracy of prediction : {} \n&quot;<br>          &quot;The test error rate is : {}&quot;<br>          .format(dataset, false_p_value, accuracy, 1-accuracy))</pre><p>This conformal prediction is implemented for both Iris and Ionosphere datasets. Full context of code can be found in the repository.</p><p><a href="https://github.com/venkatasandeepd/conformal_prediction">venkatasandeepd/conformal_prediction</a></p><p>There is also framework implementation of conformal prediction which can be used as extension to the scikit-learn library.</p><h4>Python Packages</h4><ol><li>Conformal Prediction add-on in Orange 3 (Version 1.1.3, May 2019)<br>https://pypi.org/project/Orange3-Conformal/</li><li>Henrik Linusson, nonconformist (Version 2.1.0, June 2017)<br>https://pypi.org/project/nonconformist/<br><a href="https://github.com/donlnz/nonconformist">https://github.com/donlnz/nonconformist</a><br><a href="http://donlnz.github.io/nonconformist/">http://donlnz.github.io/nonconformist/</a></li></ol><h4>R packages</h4><ol><li>Carnegie Mellon Conformal Inference Team (Max G’Sell, Jing Lei, Alessandro Rinaldo, Ryan Tibshirani, and Larry Wasserman), conformalInference (Version 1.0.0, June 2018)<br>https://github.com/ryantibs/conformal/</li><li>Niharika Gauraha and Ola Spjuth, conformalClassification (Version 1.0.0, December 2017)<br>https://cran.r-project.org/web/packages/conformalClassification/</li></ol><h4>Resources:</h4><ol><li><a href="http://jmlr.csail.mit.edu/papers/volume9/shafer08a/shafer08a.pdf">http://jmlr.csail.mit.edu/papers/volume9/shafer08a/shafer08a.pdf</a></li><li><a href="https://cml.rhul.ac.uk/copa2017/presentations/CP_Tutorial_2017.pdf">https://cml.rhul.ac.uk/copa2017/presentations/CP_Tutorial_2017.pdf</a></li></ol><p>Thank you for reading :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7a392fc29bc1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/a-guideline-to-conformal-prediction-7a392fc29bc1">A Guideline to Conformal Prediction</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Hadoop]]></title>
            <link>https://venkatasandeepd.medium.com/introduction-to-hadoop-b62a3d7519fd?source=rss-db1a063707cb------2</link>
            <guid isPermaLink="false">https://medium.com/p/b62a3d7519fd</guid>
            <category><![CDATA[hadoop]]></category>
            <category><![CDATA[yarn]]></category>
            <category><![CDATA[mapreduce]]></category>
            <dc:creator><![CDATA[Venkata Sandeep]]></dc:creator>
            <pubDate>Thu, 19 Mar 2020 15:34:51 GMT</pubDate>
            <atom:updated>2020-03-19T15:34:51.737Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2fWojRsxZDsfMBx5" /></figure><p><strong><em>Hadoop, </em></strong>a fault tolerant distributed system, is an opensource framework used to manage, store and process huge amounts of data, which we call as big data. There are certain characteristics to identify big data. They are 3V’s</p><p><strong><em>Volume — Huge amounts of data in terms of petabytes</em></strong></p><p><strong><em>Variety — Structured, Semi-Structured and Unstructured data</em></strong></p><p><strong><em>Velocity— Rate at which the data is generating</em></strong></p><h3>Why Hadoop instead of SQL ?</h3><p>As one of the characteristics of Big Data, data includes semi structured and unstructured data. NoSQL can somehow manage semi-structured data but there is no way SQL can handle unstructured data which includes audio, video and texts.</p><p>There are three main differences between <strong><em>SQL</em></strong> and <strong><em>Hadoop</em></strong></p><ol><li><strong>Schema-On-Write vs Schema-On-Read</strong><br>In traditional SQL we need to have some information on structure of database. We have to ensure the data types of data that we are going to write data to that database where as in HDFS we just write data to database, but while reading the data we apply our structure/rules to the code that is going to read the data.</li><li><strong>Storage and Scalability</strong><br>In SQL, the data is stored in logical forms in the form of interrelated tables and columns where as in HDFS, it is stored in the form of text files, compressed files etc.<br>Whenever there is increase in the data, in Hadoop we increase number of nodes in the cluster instead of static memory allocation in the case of SQL.</li><li><strong>Fault Tolerance</strong><br>Hadoop is always in High Availability (HA) mode due to its 3 level replication of data using slave nodes but if we lost data in RDBMS, it’ll take more time and resource to get back the lost data.</li></ol><h3>Under the Hadoop</h3><p>Hadoop consists of three main parts:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tFA3u4yynMrWxXfk.png" /><figcaption>Source: GeeksForGeeks</figcaption></figure><h3>1. HDFS</h3><blockquote>Self-healing high-bandwidth clustered storage</blockquote><p>With the rampant increase in the data and evolving workloads businesses can’t cope up with the traditional databases.</p><p><strong><em>Hadoop Distributed File System</em></strong>(HDFS) is a file storage system used by Hadoop to store huge amounts of data. It contains a software layer that controls how data is stored and retrieved from physical storage media. It also enables to store the data across multiple nodes in a cluster for security and fault tolerance. It also exposes an abstract view of data organization i.e File and Folder directory structure.</p><p>The data we are going to write in HDFS is divided in to chunks of data and these chunks are stored in different nodes in the clusters of different nodes called as data nodes. The information related to data(meta-data) of where the data is stored is stored in name node.</p><h4>HDFS Design Overview</h4><ol><li>Single hierarchical namespace for entire cluster</li><li>Data coherency<br>2.1 Write-once-read-many access model <br>2.2 Client can only append to existing files</li><li>Files are broken up into blocks<br>3.1 Typically 128MB block size<br>3.2 Each block replicated on multiple Data nodes</li><li>Separation of data and metadata (namespace)</li><li>Intelligent client (library)<br>5.1 Client can find location of blocks<br>5.2 Client accesses data directly from Data node</li></ol><h3>2. MapReduce</h3><blockquote>Scalable data programming abstraction</blockquote><p>MapReduce is a process inspired from Google File System(GFS) and Google’s MapReduce platform.</p><p>The core idea behind MapReduce is mapping your data set into a collection of &lt;key, value&gt; pairs, and then reducing over all pairs with the same key. When we start a map/reduce workflow, the framework will split the input into segments, passing each segment to a different machine. Each machine then runs the map script on the portion of data attributed to it. The reduce script takes a collection of &lt;key, value&gt; pairs and “reduces” them according to the user‐specified reduce script.</p><p>Let us discuss an example.</p><p>If MapReduce job is given a sentence and the expected result is the word counts in that sentence.</p><p><strong><em>Phase 1 : Splitting and Mapping</em></strong></p><p>Splitting the data in to smaller chunks, that is splitting the sentence into words in this case.</p><p>In this stage, each smaller chunk of data is passed to a mapping function which is, counting the number of occurrences of each word in that chunk of data.</p><p><strong><em>Phase 2 : Shuffling and Reducing</em></strong></p><p>This phase consumes the output of Mapping phase. Its task is to consolidate the relevant records from Mapping phase output. In our example, the same words are clubbed together along with their respective frequency.</p><p>In this phase, output values from the Shuffling phase are aggregated. This phase combines values from Shuffling phase and returns a single output value containing the all the different words and their frequencies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JJxwQmTjxoxgLaithnZYew.png" /><figcaption>Source:Stanford Education</figcaption></figure><h3>3. YARN</h3><blockquote>Distributed fault-tolerant resource manager and scheduler</blockquote><p>YARN stands for <strong><em>Yet Another Resource Negotiator. </em></strong>It was introduced in Hadoop 2.x and allows graph, stream, interactive and batch processing on the data stored in HDFS clusters. YARN also manages the resources of the cluster and also functions as a framework for job scheduling in Hadoop.</p><p>YARN Resource Manger(RM) contains two main systems.</p><ol><li><strong><em>Application Manager</em></strong><br>It manages running Application Masters in the cluster, i.e., it is responsible for starting application masters and for monitoring and restarting them on different nodes in case of failures.</li><li><strong><em>Scheduler<br></em></strong>The scheduler is responsible for allocating the resources to the running application. The scheduler is pure scheduler it means that it performs no monitoring no tracking for the application and even doesn’t guarantees about restarting failed tasks either due to application failure or hardware failures. The various types of scheduling are First Come First Serve, Fair Share Scheduler and Capacity Scheduler etc. The First Come First Serve scheduling is set by default in YARN.</li></ol><h3>How it all works together ?</h3><p>For example we are loading some twitter data into Hadoop system with a cluster of some thousand servers. Our twitter data might be replicated in under 60 systems along with other twitter profiles on the data nodes. As told above, Hadoop keeps track of the meta-data of where the profile data is in the name nodes. One might think it is waste of resources, but here is the scalability takes center stage in Hadoop system. Suppose we are searching a tag or a topic in twitter data, usually it takes a lot of time in traditional RDBMS system, but as our data is replicated in many servers, the internals of Hadoop distributes/maps the search/query/workload across all the servers along the history of timeline. This is YARN’s job scheduling the resources and assigning workloads. When the results are available, these are sorted and then reducing takes place. The results are consolidated and then presented to the client. This is how Hadoop scales storing and processing of huge amounts of data.</p><h3>Resources and References</h3><ol><li><a href="https://www.youtube.com/watch?v=MfF750YVDxM">https://www.youtube.com/watch?v=MfF750YVDxM</a></li><li><a href="https://towardsdatascience.com/an-introduction-to-big-data-apache-spark-and-rdds-975ef4d91be2">https://towardsdatascience.com/an-introduction-to-big-data-apache-spark-and-rdds-975ef4d91be2</a></li><li><a href="https://www.geeksforgeeks.org/hadoop-introduction/">https://www.geeksforgeeks.org/hadoop-introduction/</a></li><li><a href="https://data-flair.training/blogs/hadoop-yarn-tutorial/">https://data-flair.training/blogs/hadoop-yarn-tutorial/</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b62a3d7519fd" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>