<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Srivignesh Rajan on Medium]]></title>
        <description><![CDATA[Stories by Srivignesh Rajan on Medium]]></description>
        <link>https://medium.com/@srivigneshrajan?source=rss-517b08d1ae3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*rzz01K1bFE0xFxP3UMziYA.jpeg</url>
            <title>Stories by Srivignesh Rajan on Medium</title>
            <link>https://medium.com/@srivigneshrajan?source=rss-517b08d1ae3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 09:23:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@srivigneshrajan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Analyzing the Performance of the Classification Models in Machine Learning]]></title>
            <link>https://medium.com/data-science/analyzing-the-performance-of-the-classification-models-in-machine-learning-ad8fb962e857?source=rss-517b08d1ae3------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad8fb962e857</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[cost-function]]></category>
            <category><![CDATA[confusion-matrix]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[classification]]></category>
            <dc:creator><![CDATA[Srivignesh Rajan]]></dc:creator>
            <pubDate>Mon, 27 Jul 2020 16:33:12 GMT</pubDate>
            <atom:updated>2020-07-27T16:52:48.978Z</atom:updated>
            <content:encoded><![CDATA[<h4>Probing the the basics of Confusion Matrix, ROC-AUC curve, and Cost Functions for Classification in Machine Learning.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BpPPlhIRgU036foL" /><figcaption>Photo by <a href="https://unsplash.com/@codestorm?utm_source=medium&amp;utm_medium=referral">Safar Safarov</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Confusion Matrix</h3><p>Confusion matrix (also called Error matrix) is used to analyze how well the Classification Models (like Logistic Regression, Decision Tree Classifier, etc.) performs. Why do we analyze the performance of the models? Analyzing the performance of the models helps us to find and eliminate the bias and variance problem if exist and it also helps us to fine-tune the model so that the model produces more accurate results. Confusion Matrix is usually applied to Binary classification problems but can be extended to Multi-class classification problems as well.</p><h3>Terminologies of Confusion Matrix</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/1*lK17g_LKDGcU7UjKlZn1tQ.png" /><figcaption>Confusion Matrix for Binary Classification. <a href="https://commons.wikimedia.org/wiki/File:ConfusionMatrixRedBlue.png">Image Source</a></figcaption></figure><p>Concepts are comprehended better when illustrated with examples so let us consider an example. Let us assume that a family went to test for COVID19.</p><p><strong>True Positive (TP): </strong>True Positives are the cases that have been predicted as positive and they indeed have that disease.</p><p><strong>False Positive (FP): </strong>False Positives are the cases that have been predicted as positive but they do not have that disease.</p><p><strong>True Negative (TN):</strong> True Negatives are the cases that have been predicted as negative and they indeed do not have that disease.</p><p><strong>False Negative (FN):</strong> False Negatives are the cases that have been predicted as negative but they have that disease.</p><p><strong>Sensitivity: </strong>Sensitivity is also called as <strong>Recall and True Positive Rate</strong>. Sensitivity is the proportion of actual positives that are correctly predicted as positives. In other words, Sensitivity is the ratio of True Positives to the Sum of True Positives and False Negatives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/632/1*kfGwwhSFYZbSHGmGhIvPrg.png" /><figcaption>PC: Author</figcaption></figure><p><strong>Specificity: </strong>Specificity is also called <strong>True Negative Rate</strong>. Specificity is the proportion of actual negatives that are correctly predicted as negatives. In other words, Specificity is the ratio of the True Negatives to the Sum of True Negatives and False Positives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*GN3ffx5aRP5Wz13usuFJVg.png" /><figcaption>PC: Author</figcaption></figure><p><strong>Precision:</strong> Precision is the proportion of predicted positives that are correctly predicted as positives. In other words, Precision is the ratio of the True Positives to the Sum of True Positives and False Positives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/1*QYb0mAP7j9d6FoRbJnrnLg.png" /><figcaption>PC: Author</figcaption></figure><p><strong>F1 Score: </strong>F1 Score is defined as the harmonic mean of Precision and Recall. F1 Score scales from 0 to 1 with 0 as the worst score and 1 as the highest score. F1 Score can be used when the data is suffering from class imbalance since it considers both False Positives and False Negatives.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/1*WfYHVQ2ZsCT5r7kheaVRzw.png" /><figcaption>PC: Author</figcaption></figure><p><strong>Accuracy: </strong>Accuracy of a model is defined as the ratio of Sum of True Positives and True Negatives to the Total Number of Predictions. Accuracy scales from 0 to 100. Accuracy can be used when obtaining True Positives and True Negatives are imperative.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/718/1*Go8jEcaXvt3QqTLoAjYg6A.png" /><figcaption>PC: Author</figcaption></figure><h3>ROC- AUC Curve</h3><p>ROC-AUC curve (Receiving Operator Characteristics- Area Under Curve) helps to analyze the performance of classification at various threshold settings. High True Positive Rates (TPR/ Sensitivity) of a class describes that the model has performed well in classifying that particular class. ROC-AUC curves can be compared for various models and the model that possesses high AUC (Area Under Curve) is considered to have performed well. In other words, the model has performed significantly well producing high TPR (True Positive Rate) at various threshold settings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QHIpEetX9cBRQ0QYLwgUDQ.png" /><figcaption>PC: Author</figcaption></figure><h3>Cost functions for Classification</h3><p>Cost functions assist in measuring how well a model performs by considering actual values and predicted values.</p><h4>Cross-Entropy loss</h4><p>Cross-Entropy loss is also called as <strong>Log loss</strong>. Log loss can be applied to Binary classification problems where the targets are binary and to Multi-class classification problems as well. Let us consider C to be the number of classes in the target variable.</p><blockquote><em>If C = 2 (binary classification) the log loss or binary cross-entropy loss is calculated as follows,</em></blockquote><ul><li>When the actual value y = 0, [(1-y) * log(1- 𝑦̂)] is applied where 𝑦̂ is the prediction of y.</li><li>When the actual value y = 1, [y * log(𝑦̂)] is applied 𝑦̂ is the prediction of y.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PNznVVF7InHjRJnDfHXzAw.png" /><figcaption>PC: Author</figcaption></figure><p><strong>Graph for -y * log(𝑦̂) when y = 1 (y is the actual value)</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*oS2aeqioaKcIg1Mnb3NhdQ.png" /><figcaption>PC: Author. Done using Desmos</figcaption></figure><p><strong>Graph for -[(1- y) * log(1- 𝑦̂)] when y = 0</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/899/1*mEm8-__D1PL3D7fY4RCEFw.png" /><figcaption>PC: Author. Done using Desmos</figcaption></figure><blockquote><em>If C &gt; 2 (multi-class classification) the log loss or </em><strong><em>multi-class cross-entropy loss</em></strong><em> is calculated as follows,</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pk4DHDFJhur5Gl2iPA9lfw.png" /><figcaption>PC: Author</figcaption></figure><p><strong>Multi-Class Cross-Entropy Loss</strong> is defined for a single instance of data and <strong>Multi-Class Cross-Entropy Error</strong> is defined for the entire set of instances of data.</p><h4>Sparse Multi-class Cross-Entropy Loss</h4><p>Sparse Multi-class Cross-Entropy Loss is very similar to Multi-class Cross-Entropy loss except for the representation of the true labels that differ from the Multi-class Cross-Entropy Loss.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C2BqazlFJH74MF1rDBJaPw.png" /><figcaption>PC: Author</figcaption></figure><p>In Multi-class Cross-Entropy loss, the true labels are one-hot encoded whereas, in Sparse Multi-class Cross-Entropy loss, the true labels are left as such thereby reducing the computation time.</p><p>Representation of true labels y in Multi-class Cross-Entropy loss,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/1*Hb1BgZCpvUBxVnFfTzyf6Q.png" /><figcaption>PC: Author</figcaption></figure><p>Representation of true labels y in Multi-class Sparse Cross-Entropy loss,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/1*5HbVXtU_NaZtcVZVkSZEmw.png" /><figcaption>PC: Author</figcaption></figure><h3>Summary</h3><ul><li>F1 Score can be used when the dataset is imbalanced. A dataset is said to be imbalanced when the number of samples of one class is more than the number of samples of another class.</li><li>ROC-AUC curve is plotted using True Positive Rates and False Positive Rates at different threshold settings. ROC-AUC curve helps to find the optimal threshold for classification.</li><li>Cross-Entropy loss can be applied to both binary and multi-class classification problems.</li><li>Sparse Multi-class Cross-Entropy Loss is computationally faster than Multi-class Cross-Entropy Loss.</li></ul><h4>References</h4><p>[1] Jason Brownlee, <a href="http://how%20to%20choose%20loss%20functions%20when%20training%20deep%20learning%20neural%20networks/">How to Choose Loss Functions When Training Deep Learning Neural Networks</a>.</p><p>[2] Scikit-learn, <a href="https://scikit-learn.org/stable/auto_examples/model_selection/plot_roc.html#sphx-glr-auto-examples-model-selection-plot-roc-py">Receiving Operator Characteristics</a>.</p><p>[3] ML- cheatsheet, <a href="https://ml-cheatsheet.readthedocs.io/en/latest/loss_functions.html">Loss functions — ML glossary documentation</a>.</p><p><em>Connect with me on </em><a href="https://www.linkedin.com/in/srivignesh-rajan-123569151/"><em>LinkedIn</em></a><em>, </em><a href="https://twitter.com/RajanSrivignesh"><em>Twitter</em></a><em>!</em></p><p><strong>Happy Machine Learning!</strong></p><h4>Thank you!</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad8fb962e857" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/analyzing-the-performance-of-the-classification-models-in-machine-learning-ad8fb962e857">Analyzing the Performance of the Classification Models in Machine Learning</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Overview of Clustering Algorithms]]></title>
            <link>https://medium.com/data-science/overview-of-clustering-algorithms-27e979e3724d?source=rss-517b08d1ae3------2</link>
            <guid isPermaLink="false">https://medium.com/p/27e979e3724d</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[clustering]]></category>
            <category><![CDATA[k-means]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Srivignesh Rajan]]></dc:creator>
            <pubDate>Mon, 20 Jul 2020 17:33:06 GMT</pubDate>
            <atom:updated>2020-07-20T18:03:41.083Z</atom:updated>
            <content:encoded><![CDATA[<h4>Hands-on Clustering Algorithms: A Walkthrough in Python!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oXAR7zbmngd-bieH" /><figcaption>Photo by <a href="https://unsplash.com/@kellysikkema?utm_source=medium&amp;utm_medium=referral">Kelly Sikkema</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Clustering</h3><p>Clustering is an unsupervised technique in which the set of similar data points is grouped together to form a cluster. A Cluster is said to be good if the intra-cluster (the data points within the same cluster) similarity is high and the inter-cluster (the data points outside the cluster) similarity is low. Clustering could also be viewed as a <strong>Data Compression</strong> technique in which the data points of a cluster can be treated as a group. Clustering is also called <strong>Data Segmentation </strong>because it partitions the data such that a group of similar data points forms a cluster.</p><h4>How does Clustering differ from Classification?</h4><p>Classification Algorithms are good techniques to distinguish between groups and classify. Classification requires manual labeling of data which is a tiring process when a dataset is huge. How about the reverse process? viz., partitioning the similar data points together and assigning cluster labels to them. A Clustering Algorithm does not require labels to further proceed since it is an unsupervised technique.</p><h3>Requirements of Clustering</h3><ul><li>Scalability</li><li>Potential to cope up with distinct attributes</li><li>Robustness towards noise and outliers</li><li>Capability to face high-dimensional data</li></ul><h3>Methods of Clustering</h3><p>The most common methods of Clustering are,</p><ul><li>Partitioning methods</li><li>Hierarchical methods</li><li>Density-based methods</li><li>Model-based methods</li></ul><blockquote><strong><em>Partitioning methods: </em></strong><em>Partitioning methods involve partitioning the data and clustering the group of similar items. Common Algorithms used in this method are,</em></blockquote><ul><li>K-Means</li><li>K-Medoids</li><li>K-Modes</li></ul><blockquote><strong><em>Hierarchical methods: </em></strong><em>Hierarchical methods involve decomposing the data hierarchically. Two types of hierarchical methods exist,</em></blockquote><ul><li>Agglomerative (bottom-up approach)</li><li>Divisive (top-down approach)</li></ul><blockquote><strong><em>Density-based methods: </em></strong><em>Density-based methods are used in Anomaly detection. The data points that are highly dense are grouped together leaving the data points with low density.</em></blockquote><ul><li>DBSCAN (Density-Based Spatial Clustering of Application with Noise)</li><li>OPTICS (Ordering Points To Identify Cluster Structures)</li></ul><blockquote><strong><em>Model-based methods: </em></strong><em>Model-based methods involves applying a model to find the best cluster structures.</em></blockquote><ul><li>EM (Expectation-Maximization) Algorithm</li></ul><p>In this post, we concentrate on some of the algorithms of <strong>Partitioning methods and Density-based methods.</strong></p><h3>Load the required Libraries in Python</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3a91ed943d20506ec2ce1880708a9b95/href">https://medium.com/media/3a91ed943d20506ec2ce1880708a9b95/href</a></iframe><h3>Partitioning methods</h3><h4>1. K-Means Clustering</h4><p>K-Means Clustering is a classical approach to Clustering. K-Means iteratively relocates the cluster centers by computing the mean of a cluster.</p><ul><li>Initially, K-Means chooses <em>k </em>cluster centers randomly.</li><li>Distance is calculated between each data point and cluster centers (Euclidean distance is commonly used). A data point is assigned to a Cluster to which it is very close.</li><li>After all the data points are assigned to a cluster the algorithm computes the mean of the cluster data points and relocates the cluster centers to its corresponding mean of the cluster.</li><li>This process is continued until the cluster centers do not change.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f99d7b84b5b1ae1058977efa3174044d/href">https://medium.com/media/f99d7b84b5b1ae1058977efa3174044d/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fCorh8GSH3OrpJqAwIs9Fw.png" /><figcaption>PC: Author</figcaption></figure><p>The advantage of using K-Means is Scalability. K-Means perform well on huge data. The main disadvantage of using K-Means is its sensitivity to outliers. The outliers have a severe impact while computing means of the cluster. Cluster results vary according to <em>k</em> value and initial choice of cluster centers. K-Means algorithm works well only for spherical data and fails to perform well on arbitrary shapes of data.</p><h4>2. K-Modes Clustering</h4><p>K-Means works well for continuous data. What about categorical data? K-Modes Clustering comes to the rescue. The algorithm is very similar to K-Means but instead of calculating the mean of the cluster, K-Modes calculate the mode (a value that occurs frequently) of the cluster.</p><ul><li>Initially, K-Mode chooses <em>k </em>cluster centers randomly.</li><li>The similarity is calculated between each data point and cluster centers. A data point is assigned to a Cluster to which it has high similarity.</li><li>After all the data points are assigned to a cluster the algorithm computes the mode of the cluster data points and relocates the cluster centers to its corresponding mode of the cluster.</li><li>This process is continued until the cluster centers do not change.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5b85c8546621a1c6640c142a475b9824/href">https://medium.com/media/5b85c8546621a1c6640c142a475b9824/href</a></iframe><h4>How do you find an optimal value for <em>k</em>?</h4><p>Elbow method and Silhouette index are the most commonly used methods to find an optimal value for k.</p><h3>Density-based methods</h3><h4>1. DBSCAN</h4><p>DBSCAN (Density-Based Spatial Clustering of Applications with Noise) is a density-based clustering algorithm that has the ability to perform well on data with arbitrary shapes. DBSCAN finds the data points that are dense and assigns it to a cluster leaving the less dense data points apart from the cluster</p><p><strong>Terminologies of DBSCAN</strong></p><ul><li>If a data point <em>q</em> is within the <strong>radius ϵ</strong> of another data point <em>p </em>then the data point <em>q</em> is considered to be <strong>ϵ-neighborhood (epsilon neighborhood) </strong>of the data point <em>p</em>.</li><li>A data point <em>p</em> is said to be a <strong>core object</strong> if the ϵ-neighborhood of point <em>p </em>consists of the value mentioned in <strong><em>MinPts (Minimum Points). </em></strong>For example, consider <strong>MinPts = 5, </strong><em>p </em>is said to be a core object if the ϵ-neighborhood of <em>p </em>consists of 5 data points.</li><li>A data point <em>p</em> is said to be <strong>directly density-reachable</strong> from point <em>q </em>if the point <em>p</em> is within the ϵ-neighborhood of <em>q</em>.</li></ul><p>How does it work? DBSCAN checks the ϵ-neighborhood of each data point. If <em>p</em> is a core object and its data points in ϵ-neighborhood are higher than the value of MinPts it forms a new cluster around the core object <em>p</em>. DBSCAN finds directly density-reachable data points iteratively and may amalgamate other few clusters. The process is continued until there are no more points left to analyze.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f8e78a3b43c37ecaa9d681ec95a7e4be/href">https://medium.com/media/f8e78a3b43c37ecaa9d681ec95a7e4be/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B79_1IVWCFSAlatOw3zQ8g.png" /><figcaption>PC: Author</figcaption></figure><p>The disadvantage of using DBSCAN is the presence of hyperparameters ϵ and <em>MinPts. </em>The results produced vary according to the values opted for hyperparameters.</p><h4>2. OPTICS</h4><p><strong>OPTICS (Ordering Points To Identify the Clustering Structure) </strong>is a clustering algorithm that surmounts the disadvantages faced by the usage of hyperparameters in DBSCAN. OPTICS is very similar to DBSCAN but it uses a flexible value for the radius ϵ.</p><p><strong>Terminologies of OPTICS</strong></p><ul><li>The smallest value for 𝜖′ (less than the value for 𝜖′) that makes the point<em> p</em> as a core object is called <strong>core-distance</strong> of <em>p</em>.</li><li>The maximum of the core-distance of p and the Euclidean distance between p and q is called the <strong>reachability-distance</strong> of a point q with respect to another object p.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8f3f4c11f54184791935461e91ca6de3/href">https://medium.com/media/8f3f4c11f54184791935461e91ca6de3/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6GlagN7uCUrVEM8v2IDcGw.png" /><figcaption>PC: Author</figcaption></figure><h3>Summary</h3><p>Clustering is used in the sales and marketing industry to identify the right segment of customers. Netflix uses Clustering algorithms to group viewers of similar interests. Clustering is a field of research work and many variants of the available algorithms are in the development phase.</p><p><strong>Find this post in my Kaggle notebook: </strong><a href="https://www.kaggle.com/srivignesh/overview-of-clustering-algorithms">https://www.kaggle.com/srivignesh/overview-of-clustering-algorithms</a></p><p><strong>References:</strong></p><p>[1] Jiawei Han and Micheline Kamber, Data Mining: Concepts and Techniques Second Edition (2006)</p><p><em>Connect with me on </em><a href="https://www.linkedin.com/in/srivignesh-rajan-123569151/"><em>LinkedIn</em></a><em>, </em><a href="https://twitter.com/RajanSrivignesh"><em>Twitter</em></a><em>!</em></p><p><strong>Happy Machine Learning!</strong></p><h4><strong>Thank you!</strong></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=27e979e3724d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/overview-of-clustering-algorithms-27e979e3724d">Overview of Clustering Algorithms</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Foundational  Concepts of Machine Learning]]></title>
            <link>https://medium.com/data-science/the-basic-underlying-concepts-of-machine-learning-28282c8421f4?source=rss-517b08d1ae3------2</link>
            <guid isPermaLink="false">https://medium.com/p/28282c8421f4</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[beginners-guide]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Srivignesh Rajan]]></dc:creator>
            <pubDate>Sat, 18 Jul 2020 12:59:27 GMT</pubDate>
            <atom:updated>2020-07-20T15:48:01.761Z</atom:updated>
            <content:encoded><![CDATA[<h3>Foundational Concepts of Machine Learning</h3><h4>Probing deep into the fundamental concepts of Machine Learning</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bSiTDvHdt3e0sozc" /><figcaption>Photo by <a href="https://unsplash.com/@tetrakiss?utm_source=medium&amp;utm_medium=referral">Arseny Togulev</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Machine Learning</h3><p>Machine Learning is an enticing and alluring field of study. Machine Learning is a subset of Artificial Intelligence and its algorithms revolve around mathematics. Machine Learning has transformed many industries. The industries that leverage Machine Learning are healthcare industries, finance, transportation industries, manufacturing industries, advertising industries, automobile industries, etc. The impact of Machine Learning is such that many industries couldn’t function without Machine Learning.</p><p>The term “Machine Learning” was coined by Arthur Samuel in 1952. Arthur Samuel describes Machine Learning as</p><blockquote>“Field of study that gives computers the ability to learn without being explicitly programmed”.</blockquote><p>Machine Learning Algorithms are very different from Traditional Programming Algorithms. Traditional programming algorithms take in rules and data as inputs and produces answers as output, whereas Machine Learning takes in answers and data as inputs and produces rules as output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GJkgbvabn3bxCrAArpwmUw.png" /><figcaption>Difference between Traditional and Machine Learning Algorithms. PC: Author</figcaption></figure><p>You may wonder what is the use of producing rules as output. The rules produced by the Machine Learning algorithms are utilized to make predictions on new data.</p><h3>Types of Learning</h3><p>Machine Learning has diverse types of learning which are applicable to different types of real-world problems. The different types of learning available are,</p><ul><li>Supervised Learning</li><li>Unsupervised Learning</li><li>Reinforcement Learning</li></ul><h4>Supervised Learning</h4><p>Supervised learning attempts to learn the relationship between the features and the labeled response variable (or target variable). Consider X as features and Y as response variable then supervised learning tries to learn the X to Y mapping in the data.</p><p>Supervised learning is further divided into,</p><ul><li><strong>Regression</strong></li><li><strong>Classification</strong></li></ul><h4>Regression</h4><p>Regression is a statistical method used to find the relationship between the features and the <strong>continuous target variable</strong> and predict future samples. A variable is considered to be continuous if it can take any values between an interval. The values it can take are infinite.</p><p>For example, 0.1, 2.2, 3.1 are continuous values. Heights and weights are considered to be continuous values.</p><p>Some of the available Regression Algorithms are,</p><ul><li>Linear Regression</li><li>Polynomial Regression</li><li>Decision Tree Regressor</li><li>Extreme Gradient Boost Regressor</li><li>Adaptive Boost Regressor</li><li>Artificial Neural Networks with linear activation function in the last layer of the neural network.</li></ul><p>For more details on Artificial Neural Networks refer to my article: <a href="https://towardsdatascience.com/an-introduction-to-artificial-neural-networks-5d2e108ff2c3">An Introduction to Artificial Neural Networks</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/1*QnnELxDwmFySEZEXlnLkww.png" /><figcaption>Linear Regression. PC: Author</figcaption></figure><h4>Classification</h4><p>The classification task attempts to classify an instance to any of the available categories. A task is considered to be a classification task if the response variable (or target) is <strong>categorical</strong>. A variable is considered to categoric if we can make that fall into a defined category.</p><p>For example, yes or no, colors like red, green, etc.. are considered to be categorical variables.</p><p>Classification is further divided into,</p><ul><li>Binary classification</li><li>Multi-class classification</li></ul><h4>Binary classification</h4><p>A problem is said to be a binary classification problem when the target variable is binary (like yes or no, true or false, etc).</p><p><strong>Logistic Regression</strong> is an algorithm that is widely used in binary classification problems. Logistic Regression is different from Linear regression. Logistic Regression is a classification algorithm and Linear regression is a regression algorithm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pJF7IjTKQRLb5BZBEBmL0Q.png" /><figcaption>Logistic Regression. PC: Author</figcaption></figure><h4>Multi-class classification</h4><p>A problem is said to be a multi-class classification problem when the target variable consists of multiple classes.</p><p>Some of the algorithms available for Multi-class classification are,</p><ul><li>Decision Tree Classifier</li><li>Support Vector Machines</li><li>Gradient Boosting Classifier</li><li>Random Forest Classifier</li><li>Ada Boost Classifier</li><li>Artificial Neural Networks with Softmax activation function in the last layer of the neural architecture.</li></ul><h4>Reinforcement Learning</h4><p>Reinforcement learning attempts to make a sequence of actions in an interactive environment such that the reward is maximized. A reward is provided whenever the desired set of actions are performed and similarly, a penalty is accounted whenever undesired actions are performed.</p><p>The most commonly used Reinforcement algorithms are Q-learning and SARSA (State-Action-Reward-State-Action).</p><h3>Pipeline for approaching a Machine Learning problem</h3><h4>Data Collection</h4><p>Before approaching any machine learning problem data collection is the most vital part. Collecting quality data is the first predominant step taken in any Machine Learning problem. Collecting relevant data assists in producing reliable results. Data is then split into training and test set. The model is trained on the training set and tested on the test set.</p><h4>Exploratory Data Analysis</h4><p>Exploratory Data Analysis (EDA) helps you to analyze the quality and distribution of data. Data may contain <strong>Outliers and Skewness</strong> that affect the statistical tests conducted on the data. Outliers are data that do not comply with the general behavior of the other data. Skewness is when the frequency distribution of the data is distorted from the normal distribution. Many statistical tests work under the assumption that the data is normally distributed so it is very important to remove outliers and transform the skewed data.</p><h4>Data Preprocessing</h4><p>Data preprocessing is an inevitable step in machine learning. Highly insightful results can be obtained if the data is preprocessed before data modeling. We can obtain reliable results if the quality of the data is ensured. <strong>Incomplete, noisy, and inconsistent data</strong> are the inherent nature of real-world datasets. Data must be preprocessed before feeding it into the model.</p><p>Common steps involved in data preprocessing are,</p><ol><li><strong>Data Cleaning</strong></li><li><strong>Data Integration</strong></li><li><strong>Data Transformation</strong></li><li><strong>Data Reduction</strong></li></ol><p>For more details on data preprocessing, refer to my article: <a href="https://medium.com/swlh/data-preprocessing-and-data-modeling-for-kaggle-house-price-prediction-data-in-python-c04055ded258">Data Preprocessing Pipeline in Machine Learning</a></p><h4>Model Training</h4><p>The model training is the learning process in which the relationships between the features and the target are computed. The relationships between the features and the target are represented as model parameters. The model parameters are then used to predict the test data. The training process for each algorithm differs.</p><ul><li>The Decision Tree Algorithm and the Random Forest Algorithm are called non-parametric algorithms which means that it doesn’t make use of parameters (or coefficients) to predict the new data instead it constructs a tree-like structure to make predictions.</li></ul><h4>Cost functions and its Optimizations</h4><p>A Cost function is used to measure the performance of the model. It is futile to deploy a model to production without optimizing a cost function. Cost function tells us how well a model performs. Cost function compares the actual value of the target with the predicted values of the target. It is important to devote our time to choose an appropriate Cost function specific to the data. Optimization of the cost function can also be considered as a model training.</p><p>For more details on cost functions and its optimizations, refer to my article: <a href="https://towardsdatascience.com/cost-functions-of-regression-and-its-optimization-techniques-in-machine-learning-2f5931cd33f1">Cost functions of Regression and its Optimization Techniques in Machine Learning</a></p><h4>Regularizations</h4><p>Regularizations are techniques to avoid Overfitting. Overfitting is when the model memorizes the relationship between features and target and fails to generalize to new, unseen data. Overfitting can be caused due to flexible models like a decision tree model and can also be caused due to the Curse of Dimensionality. Dimensionality Reduction techniques like <a href="https://medium.com/analytics-vidhya/an-introduction-to-feature-selection-in-machine-learning-9d6f2d5e47">Feature Selection</a> and Feature Extraction could prevent Overfitting as well. Overfitting is when there exists high variance and low bias.</p><p>Regularizations are usually done by applying a penalty to the cost function. The most commonly used Regularization techniques are,</p><ul><li>LASSO regression</li><li>Ridge regression</li><li>Dropout for Artificial Neural Networks</li></ul><h4>Hyperparameter Optimization</h4><p>Hyperparameters are tunable parameters that have an impact on model training. Hyperparameters are different from Model parameters as Model parameters are results of the training phase and the Hyperparameters are provided manually by the user before a training phase.</p><p>Most commonly used Hyperparameter optimization techniques are,</p><ul><li>Grid Search</li><li>Random Search</li><li>Bayesian optimization</li></ul><h4>Model Interpretation</h4><p>What is the fidelity of the model? Why should I believe the results produced by the model? These questions are answered by model interpretation techniques that attempt to explain why the model arrived at this conclusion.</p><ul><li>SHAP (Shapley Additive exPlanations)</li><li>LIME (Linearly Interpretable Model-Agnostic Explanations)</li></ul><h4>Model Deployment</h4><p>The trained model can be deployed in a production environment. A production environment is where a user can access and avail the provisioned services. The role of a Data Engineer is to handle the intricacies of deploying a model and abstracting it from the Data Scientists and Machine Learning Engineers. A model can be deployed using a Python web framework like Flask, CherryPy, FastAPI.</p><h3>Summary</h3><p>Machine Learning is a fascinating field of study that has the potential to revolutionize our future. Many Machine Learning algorithms are still in the area of research and development. The demand for Machine Learning Engineers is growing day by day and the demand will reach its top in the near future. Machine Learning never fails to excite you! It is one of a kind!</p><p><strong>Happy Machine Learning!</strong></p><h4><strong>Thank you!</strong></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=28282c8421f4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/the-basic-underlying-concepts-of-machine-learning-28282c8421f4">Foundational  Concepts of Machine Learning</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cost functions for Regression and its Optimization Techniques in Machine Learning]]></title>
            <link>https://medium.com/data-science/cost-functions-of-regression-and-its-optimization-techniques-in-machine-learning-2f5931cd33f1?source=rss-517b08d1ae3------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f5931cd33f1</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[cost-function]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[optimization]]></category>
            <dc:creator><![CDATA[Srivignesh Rajan]]></dc:creator>
            <pubDate>Fri, 17 Jul 2020 17:50:19 GMT</pubDate>
            <atom:updated>2020-07-24T08:12:54.652Z</atom:updated>
            <content:encoded><![CDATA[<h4>Probing deep into cost functions of Regression and its Optimization Techniques: A Walkthrough in Python</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zsPVQNw1ytflVuV7" /><figcaption>Photo by <a href="https://unsplash.com/@alexandermils?utm_source=medium&amp;utm_medium=referral">Alexander Mils</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Cost Function</h3><p>A Cost function is used to gauge the performance of the Machine Learning model. A Machine Learning model devoid of the Cost function is futile. Cost Function helps to analyze how well a Machine Learning model performs. A Cost function basically compares the predicted values with the actual values. Appropriate choice of the Cost function contributes to the credibility and reliability of the model.</p><h4>Loss function vs. Cost function</h4><ul><li>A function that is defined on a single data instance is called Loss function.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/492/1*TtJ9LjicBwJ1ddrMD99eug.png" /><figcaption>Absolute loss of Regression</figcaption></figure><ul><li>A function that is defined on an entire data instance is called the Cost function.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/664/1*hJ8y-Z5CJiQUMgE9LG0cag.png" /><figcaption>Mean Absolute Error of Regression</figcaption></figure><h3>Cost functions of Regression</h3><p>Regression tasks deal with continuous data. Cost functions available for Regression are,</p><ul><li>Mean Absolute Error</li><li>Mean Squared Error</li><li>Root Mean Squared Error</li><li>Root Mean Squared Logarithmic Error</li></ul><h4>Mean Absolute Error</h4><p><strong>Mean Absolute Error(MAE)</strong> is the mean absolute difference between the actual values and the predicted values.</p><ul><li>MAE is more robust to outliers. The insensitivity to outliers is because it does not penalize high errors caused by outliers.</li><li>The drawback of MAE is that it isn’t differentiable at zero and many Loss function Optimization algorithms involve differentiation to find optimal values for Parameters.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*Lp3cA5fYLtMT1ruvr5h3GQ.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3e4659dbfa9141e6cdfed665df6e1c3d/href">https://medium.com/media/3e4659dbfa9141e6cdfed665df6e1c3d/href</a></iframe><h4>Mean Squared Error</h4><p><strong>Mean Squared Error(MSE)</strong> is the mean squared difference between the actual and predicted values. MSE penalizes high errors caused by outliers by squaring the errors. The optimization algorithms benefit from penalization as it is helpful to find the optimal values for parameters.</p><ul><li>The drawback of MSE is that it is very sensitive to outliers. When high errors (which are caused by outliers in the target) are squared it becomes, even more, a larger error.</li><li>MSE can be used in situations where high errors are undesirable.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*J0D080xocWZGZ4ADH5NWIQ.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/34bd0bb6c1219455f5f56749873cf60f/href">https://medium.com/media/34bd0bb6c1219455f5f56749873cf60f/href</a></iframe><h4>Root Mean Squared Error</h4><p><strong>Root Mean Squared Error (RMSE) </strong>is the root squared mean of the difference between actual and predicted values. RMSE can be used in situations where we want to penalize high errors but not as much as MSE does.</p><ul><li>RMSE is highly sensitive to outliers as well. The square root in RMSE makes sure that the error term is penalized but not as much as MSE.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*c61g9CvHR4xYO1tbNBScAw.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/01f95241b1ea904a6393cf200ee112b3/href">https://medium.com/media/01f95241b1ea904a6393cf200ee112b3/href</a></iframe><h4>Root Mean Squared Logarithmic Error</h4><p><strong>Root Mean Squared Logarithmic Error (RMSLE)</strong> is very similar to RMSE but the log is applied before calculating the difference between actual and predicted values. The large errors and small errors are treated equally. RMSLE can be used in situations where the target is not normalized or scaled.</p><ul><li>RMSLE is less sensitive to outliers as compared to RMSE. It relaxes the penalization of high errors due to the presence of the log.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GyLbnf_moZPHsyToi5hNjQ.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1ccca41d10074ee57a99e5ebd146de6a/href">https://medium.com/media/1ccca41d10074ee57a99e5ebd146de6a/href</a></iframe><h3>Cost function Optimization Algorithms</h3><p>Cost function optimization algorithms attempt to find the optimal values for the model parameters by finding the global minima of cost functions. The various algorithms available are,</p><ul><li>Gradient Descent</li><li>RMS Prop</li><li>Adam</li></ul><h4>Load the preprocessed data</h4><p>The data you feed to the ANN must be preprocessed thoroughly to yield reliable results. The training data has been preprocessed already. The preprocessing steps involved are,</p><ul><li>MICE Imputation</li><li>Log transformation</li><li>Square root transformation</li><li>Ordinal Encoding</li><li>Target Encoding</li><li>Z-Score Normalization</li></ul><blockquote>For the detailed implementation of the above-mentioned steps refer my Kaggle notebook on data preprocessing. <a href="https://www.kaggle.com/srivignesh/data-preprocessing-for-house-price-prediction">Notebook Link</a></blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2cd451017e3526cd4215285f2d7f9925/href">https://medium.com/media/2cd451017e3526cd4215285f2d7f9925/href</a></iframe><h4>Train the model with ANN</h4><p>Refer to my Kaggle notebook on <a href="https://www.kaggle.com/srivignesh/introduction-to-ann-in-tensorflow">Introduction to ANN in Tensorflow</a> for more details.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4d30a2916dbd86e3268175a36f3405e2/href">https://medium.com/media/4d30a2916dbd86e3268175a36f3405e2/href</a></iframe><h4>Gradient Descent</h4><p>Gradient Descent algorithm makes use of gradients of the cost function to find the optimal value for the parameters. Gradient descent is an iterative algorithm. It attempts to find a global minimum.</p><p>On each iteration t,</p><ul><li>The cost of the data is found.</li><li>The partial differentiation of cost function with respect to weights and bias is computed.</li><li>The weights and bias are then updated by making use of gradients of the cost function and learning rate 𝛼. The value of 𝛼 can range from 0.0 to 1.0. Greater the value of 𝛼 greater is the number of steps taken to find the global minimum of the cost function.</li><li>Continue the above-mentioned steps until a specified number of iterations are completed or when a global minimum is reached.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t_vY-BtfUK4wTnt_avlUKA.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4e062e39155d0da27ff801261cbfddfe/href">https://medium.com/media/4e062e39155d0da27ff801261cbfddfe/href</a></iframe><h4>RMS Prop (Root Mean Squared Prop)</h4><p>RMS Prop is an optimization algorithm that is very similar to Gradient Descent but the gradients are smoothed and squared and then updated to attain the global minimum of the cost function soon.</p><p>On each iteration t,</p><ul><li>The cost of the data is found.</li><li>The partial differentiation of cost function with respect to weights and bias is computed.</li><li>The weights and bias parameters are smoothed and then updated by making use of gradients of cost function and 𝛼 (learning rate).</li><li>Continue the above-mentioned steps until a specified number of iterations are completed or when a global minimum is reached.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1UL3xGPa2Jfymy0-nFAdJA.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/db6f7e44f777047d8fb08b41e983ff60/href">https://medium.com/media/db6f7e44f777047d8fb08b41e983ff60/href</a></iframe><h4>Adam (Adaptive Moment Estimation)</h4><p>Adam (Adaptive Moment Estimation) is an algorithm that emerged by combining Gradient Descent with momentum and RMS Prop.</p><p>On each iteration t,</p><ul><li>The cost of the data is found.</li><li>The partial differentiation of cost function with respect to weights and bias is computed.</li><li>The weights and bias are smoothed with the technique used in RMS Prop and Gradient Descent with momentum and then the weights and bias are updated by making use of gradients of cost function and 𝛼 (learning rate).</li><li>Continue the above-mentioned steps until a specified number of iterations are completed or when a global minimum is reached.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SbfKEgz_GQGNCW0Wx00K0Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r7wZVNd_KY1YBWk4kgPm0A.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bac3bd816cb95ecac71fe9d34e9cbc79/href">https://medium.com/media/bac3bd816cb95ecac71fe9d34e9cbc79/href</a></iframe><h3>Summary</h3><ul><li>Mean Absolute Error is robust to outliers whereas Mean Squared Error is sensitive to outliers</li><li>Gradient descent algorithm attempts to find the optimal values for parameters such that the global minimum of the cost function is found.</li><li>The algorithms like RMS Prop and Adam can be thought of as variants of Gradient descent algorithm.</li></ul><p><strong>Find this post in my Kaggle notebook: </strong><a href="https://www.kaggle.com/srivignesh/cost-functions-of-regression-its-optimizations">https://www.kaggle.com/srivignesh/cost-functions-of-regression-its-optimizations</a></p><p><strong>References:</strong></p><p>[1] Andrew Ng, <a href="https://www.coursera.org/specializations/deep-learning">Deep Learning Specialization</a>.</p><p><em>Connect with me on </em><a href="https://www.linkedin.com/in/srivignesh-rajan-123569151/"><em>LinkedIn</em></a><em>, </em><a href="https://twitter.com/RajanSrivignesh"><em>Twitter</em></a><em>!</em></p><p><strong>Happy Machine Learning!</strong></p><h4><strong>Thank you!</strong></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f5931cd33f1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/cost-functions-of-regression-and-its-optimization-techniques-in-machine-learning-2f5931cd33f1">Cost functions for Regression and its Optimization Techniques in Machine Learning</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Beginners Guide to Apache Pyspark]]></title>
            <link>https://medium.com/data-science/an-introduction-to-apache-pyspark-4454cc03bdfa?source=rss-517b08d1ae3------2</link>
            <guid isPermaLink="false">https://medium.com/p/4454cc03bdfa</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[pyspark]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Srivignesh Rajan]]></dc:creator>
            <pubDate>Thu, 16 Jul 2020 12:22:12 GMT</pubDate>
            <atom:updated>2020-07-20T15:46:25.058Z</atom:updated>
            <content:encoded><![CDATA[<h4>Boost your data processing performance with Apache Pyspark!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_5rG4C0-rZQv2zw1" /><figcaption>Photo by <a href="https://unsplash.com/@krisroller?utm_source=medium&amp;utm_medium=referral">Kristopher Roller</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Apache Spark</h3><p>Apache Spark is an open-source analytics engine and cluster-computing framework that boosts your data processing performance. As they claim, Spark is a lightning-fast unified analytics engine. Spark is entirely written in Scala.</p><p>Spark is effectively used in the field of <strong>Big-Data and Machine Learning</strong> for analytical purposes. Spark has been adopted by various companies like Amazon, eBay, and Yahoo.</p><h4>Features of Spark</h4><ul><li>Spark is <strong>polyglot</strong> which means you can utilize Spark using one or more programming languages. Spark provides you with high-level APIs in Java, Python, R, SQL, and Scala. Apache Spark package written in Python is called <strong>Pyspark</strong>.</li><li>Spark supports multiple <strong>data formats</strong> such as Parquet, CSV (Comma Separated Values), JSON (JavaScript Object Notation), ORC (Optimized Row Columnar), Text files, and RDBMS tables.</li><li>Spark has low latency because of its <strong>in-memory computation. </strong>Spark has been designed to deal with humongous data, so scalability is an inherent feature of Spark.</li><li>Spark can be seamlessly integrated with Hadoop and has the capability of running on top of Hadoop clusters.</li></ul><h4>How Spark works</h4><ul><li>Spark uses <strong>Master-Slave architecture</strong>. The Master node assigns tasks to the slave nodes that reside across the cluster and the slave nodes would execute them.</li><li>A <strong>Spark Session</strong> must be created to utilize all the functionalities provided by Spark. A Spark Session is created inside the Driver program. The Driver program resides inside the Master node.</li></ul><pre><strong># Example of creating a Spark Session in Pyspark<br></strong>spark = SparkSession.\<br>builder.master(&quot;local&quot;).\<br>appName(&quot;AppName&quot;).getOrCreate()</pre><ul><li>When you read the DataFrame using Spark Session the DataFrame will be partitioned and stored across the cluster nodes so that it can be operated parallelly. The partitions of the DataFrame are collectively called <strong>RDD (Resilient Distributed Datasets)</strong>. RDDs are <strong>fault-tolerant </strong>which means it is resilient to failures.</li><li>When an Action is invoked through the Spark Session, the Spark creates <strong>DAG (Directed Acyclic Graph)</strong> of transformations (which would be applied to the partitions of data) and implements them by assigning tasks to the Executors in the slave nodes. A transformation is never implemented until an Action is invoked. The tendency of implementing a transformation only when an Action is invoked is called <strong>Lazy Evaluation</strong>.</li><li>The Driver Program running in the Master node assigns Spark jobs to the slave nodes when an action is invoked. The <strong>spark jobs</strong> are split into <strong>stages</strong> and those stages are further split into <strong>tasks</strong>.</li><li>The slave nodes contain many <strong>Executors </strong>that receive tasks and execute them parallelly on the partitions of the data<strong>. </strong>The executors are the ones which cache data for in-memory computation.</li></ul><h3><strong>Spark Transformations</strong></h3><p>The Spark Transformations perform some operations on RDDs and produce new RDD. Various Spark transformations include <em>map</em>, <em>flatMap, filter, groupBy, reduceBy, and join.</em></p><p><em>Spark Transformations are further classified into two types,</em></p><ul><li>Narrow transformations</li><li>Wide transformations</li></ul><h4>Narrow transformations</h4><p>Spark transformations are called narrow transformations when the operation <strong>does not require Shuffling. </strong>A Narrow transformation does not require partitions of data to be shuffled across nodes in the cluster.</p><p>Examples of Narrow transformations are <em>map, flatMap, filter, sample, </em>etc<em>.</em></p><h4>Wide transformations</h4><p>Spark transformations are called wide transformations when the operation <strong>requires Shuffling</strong>. Shuffling is an operation that involves shuffling the partitions of the data across the nodes of the cluster to perform an operation.</p><p>Examples of Wide transformations are <em>groupBy, reduceBy, join,</em> etc.</p><ul><li>The <em>groupBy</em> is a transformation in which the values of the column are grouped to form a unique set of values. To perform this operation is costly in distributed environments because all the values to be grouped must be collected from various partitions of data that reside in nodes of the cluster.</li></ul><h3>Actions in Spark</h3><p>Actions are operations that trigger Spark Jobs. Spark does not immediately perform transformations. It requires an Action to trigger the implementation of the Spark transformations.</p><p>Examples of Spark actions are <em>collect</em>, <em>count, take, first, saveAsTextFile, etc.</em></p><ul><li><em>Collect</em> is an action that collects all the partitions of data that resides across the nodes of the cluster and stores them in the Driver that resides in the Master node.</li></ul><h3>Spark Jobs</h3><p>Spark jobs are triggered when an Action is invoked. Spark jobs are further divided into <strong>Stages and Tasks.</strong></p><h4>Stages</h4><p>Spark jobs involving wide transformations are grouped together as a stage and the jobs involving narrow transformations are grouped as another stage.</p><pre><strong># A Spark Job</strong><br>df.filter(col(&#39;A&#39;)).groupBy(&#39;A&#39;)</pre><p>The entire code above is considered to be a Spark job, in this <em>filter </em>is a separate stage and <em>groupBy </em>is a separate stage because <em>filter </em>is a narrow transformation and <em>groupBy</em> is a wide transformation.</p><pre><strong># Stage A<br>df.filter(col(&#39;A&#39;))</strong>.groupBy(&#39;A&#39;)</pre><pre><strong># Stage B</strong><br>df.filter(col(&#39;A&#39;)).<strong>groupBy(&#39;A&#39;)</strong></pre><h4>Tasks</h4><p>The stages of Spark Jobs are further divided into tasks. The tasks are the operations that are applied to each of the partitions across the nodes of the cluster.</p><h3>Data preparation using Pyspark</h3><h4>Install Pyspark</h4><p>Pyspark can be installed by executing the following command</p><pre>pip install pyspark</pre><h4>Import the required libraries</h4><pre>import math<br>import numpy as np <br>import pandas as pd  <br>import pyspark<br>from pyspark.sql import SparkSession<br>from pyspark.sql.functions import isnan, when, count, col, isnull, asc, desc, mean</pre><pre><strong>&#39;&#39;&#39;Create a spark session&#39;&#39;&#39;</strong><br>spark = SparkSession.\<br>builder.\<br>master(&quot;local&quot;).appName(&quot;DataWrangling&quot;).getOrCreate()<br><strong>&#39;&#39;&#39;Set this configuration to get output similar to pandas&#39;&#39;&#39;</strong><br>spark.conf.set(&#39;spark.sql.repl.eagerEval.enabled&#39;, True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qjqkSug-tft-w0KW0xG6-Q.png" /><figcaption>Output</figcaption></figure><pre><strong>&#39;&#39;&#39;Find the count of a dataframe&#39;&#39;&#39;</strong><br>df.count()<strong><br>&quot;&quot;&quot;OUTPUT:</strong>891&quot;&quot;&quot;</pre><h4>Count of values in a column</h4><pre>df.groupBy(&#39;Sex&#39;).count()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/242/1*WTJjfTK-fRabE4t9FgzwMQ.png" /><figcaption>Output</figcaption></figure><h4>Find distinct values of a column in a Dataframe</h4><pre>df.select(&#39;Embarked&#39;).distinct()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/164/1*Jy9QRagq7GQ3ednk_StGtg.png" /><figcaption>Output</figcaption></figure><h4>Select a specific set of columns in a Dataframe</h4><pre>df.select(&#39;Survived&#39;, &#39;Age&#39;, &#39;Ticket&#39;).limit(5)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*VZcGUiwkkcZevw_ZMx1v3Q.png" /><figcaption>Output</figcaption></figure><h4>Find the count of missing values</h4><pre>df.select([count(when(isnull(column), column)).alias(column) \<br><strong>for</strong> column <strong>in</strong> df.columns])</pre><h4>Filtering null and not null values</h4><pre><strong>&#39;&#39;&#39;Find the null values of &#39;Age&#39; &#39;&#39;&#39;</strong><br>df.filter(col(&#39;Age&#39;).isNotNull()).limit(5)</pre><pre><strong>&#39;&#39;&#39;Another way to find not null values of &#39;Age&#39; &#39;&#39;&#39;</strong><br>df.filter(&quot;Age is not NULL&quot;).limit(5)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cYY8jQYpKXXFkwrE6Wcbyw.png" /><figcaption>Output</figcaption></figure><pre><strong>&#39;&#39;&#39;Find the null values of &#39;Age&#39; &#39;&#39;&#39;</strong><br>df.filter(col(&#39;Age&#39;).isNull()).limit(5)</pre><pre><strong>&#39;&#39;&#39;Another way to find null values of &#39;Age&#39; &#39;&#39;&#39;</strong><br>df.filter(&quot;Age is NULL&quot;).limit(5)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bhzBxKF-gNbotjXhbYOOGw.png" /><figcaption>Output</figcaption></figure><h4>Sorting columns</h4><pre><strong>&#39;&#39;&#39;Sort &quot;Parch&quot; column in ascending order and &quot;Age&quot; in descending order&#39;&#39;&#39;</strong><br>df.sort(asc(&#39;Parch&#39;),desc(&#39;Age&#39;)).limit(5)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L2s7okPvh6dBo6Y97trxYw.png" /><figcaption>Output</figcaption></figure><h4>Dropping columns</h4><pre><strong>&#39;&#39;&#39;Drop multiple columns&#39;&#39;&#39;</strong><br>df.drop(&#39;Age&#39;, &#39;Parch&#39;,&#39;Ticket&#39;).limit(5)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ft1N1q9DbFZFWk4OsLINfg.png" /><figcaption>Output</figcaption></figure><h4>Groupby and aggregation</h4><pre><strong>&#39;&#39;&#39;Finding the mean age of male and female&#39;&#39;&#39;</strong><br>df.groupBy(&#39;Sex&#39;).agg(mean(&#39;Age&#39;))</pre><h3><strong>Summary</strong></h3><ul><li>Spark is a lightning-fast cluster computing framework used for analytical purposes.</li><li>SparkSession is the entry point for all the functionalities in Spark and is responsible for creating and scheduling Spark Jobs.</li><li>The Executors implement the transformations on all the partitions of data available in RDD.</li></ul><p><strong>Find this post in my Kaggle notebook: </strong><a href="https://www.kaggle.com/srivignesh/an-introduction-to-pyspark-apache-spark-in-python">https://www.kaggle.com/srivignesh/an-introduction-to-pyspark-apache-spark-in-python</a></p><p><em>Connect with me on </em><a href="https://www.linkedin.com/in/srivignesh-rajan-123569151/"><em>LinkedIn</em></a><em>, </em><a href="https://twitter.com/RajanSrivignesh"><em>Twitter</em></a><em>!</em></p><p><strong>Happy Learning!</strong></p><h4><strong>Thank you!</strong></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4454cc03bdfa" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/an-introduction-to-apache-pyspark-4454cc03bdfa">Beginners Guide to Apache Pyspark</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Introduction to Artificial Neural Networks]]></title>
            <link>https://medium.com/data-science/an-introduction-to-artificial-neural-networks-5d2e108ff2c3?source=rss-517b08d1ae3------2</link>
            <guid isPermaLink="false">https://medium.com/p/5d2e108ff2c3</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-neural-network]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Srivignesh Rajan]]></dc:creator>
            <pubDate>Wed, 15 Jul 2020 14:56:59 GMT</pubDate>
            <atom:updated>2020-07-20T15:46:41.269Z</atom:updated>
            <content:encoded><![CDATA[<h4>Skyrocket your model performance with Artificial Neural Networks. A Walkthrough in Tensorflow!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rf8XW5C8yPOXsyvU" /><figcaption>Photo by <a href="https://unsplash.com/@moritz_photography?utm_source=medium&amp;utm_medium=referral">Moritz Kindler</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Artificial Neural Network (ANN)</h3><p>Artificial Neural Network (ANN) is a deep learning algorithm that emerged and evolved from the idea of <strong>Biological Neural Networks of human brains</strong>. An attempt to simulate the workings of the human brain culminated in the emergence of ANN. ANN works very similar to the biological neural networks but doesn’t exactly resemble its workings.</p><p>ANN algorithm would accept only numeric and structured data as input. To accept unstructured and non-numeric data formats such as Image, Text, and Speech, <strong>Convolutional Neural Networks (CNN),</strong> and <strong>Recursive Neural Networks (RNN)</strong> are used respectively. In this post, we concentrate only on Artificial Neural Networks.</p><h3>Biological neurons vs Artificial neurons</h3><h4>Structure of Biological neurons and their functions</h4><ul><li><strong>Dendrites</strong> receive incoming signals.</li><li><strong>Soma</strong> (cell body) is responsible for processing the input and carries biochemical information.</li><li><strong>Axon</strong> is tubular in structure responsible for the transmission of signals.</li><li><strong>Synapse</strong> is present at the end of the axon and is responsible for connecting other neurons.</li></ul><h4>Structure of Artificial neurons and their functions</h4><ul><li>A neural network with a single layer is called a <strong>perceptron</strong>. A multi-layer perceptron is called <strong>Artificial Neural Networks.</strong></li><li>A Neural network can possess any number of layers. Each layer can have one or more neurons or units. Each of the neurons is interconnected with each and every other neuron. Each layer could have different <strong>activation functions</strong> as well.</li><li>ANN consists of two phases <strong>Forward propagation and Backpropagation.</strong> The forward propagation involves multiplying weights, adding bias, and applying activation function to the inputs and propagating it forward.</li><li>The backpropagation step is the most important step which usually involves finding optimal parameters for the model by propagating in the backward direction of the Neural network layers. The backpropagation requires <strong>optimization function</strong> to find the optimal weights for the model.</li><li>ANN can be applied to both <strong>Regression and Classification tasks</strong> by changing the activation functions of the output layers accordingly. (Sigmoid activation function for binary classification, Softmax activation function for multi-class classification and Linear activation function for Regression).</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nIg6rUi5Mr0Co8NqcErSMA.png" /><figcaption>Perceptron. <a href="https://commons.wikimedia.org/wiki/File:Artificial_Neuron.svg">Image Source</a></figcaption></figure><h3>Why Neural Networks?</h3><ul><li>Traditional Machine Learning algorithms tend to perform at the same level when the data size increases but ANN outperforms traditional Machine Learning algorithms when the data size is huge as shown in the graph below.</li><li><strong>Feature Learning</strong>. The ANN tries to learn hierarchically in an incremental manner layer by layer. Due to this reason, it is not necessary to perform feature engineering explicitly.</li><li>Neural Networks can handle <strong>unstructured data</strong> like images, text, and speech. When the data contains unstructured data the neural network algorithms such as CNN (Convolutional Neural Networks) and RNN (Recurrent Neural Networks) are used.</li></ul><h3>How ANN works</h3><p>The working of ANN can be broken down into two phases,</p><ul><li>Forward Propagation</li><li>Back Propagation</li></ul><h4>Forward Propagation</h4><ul><li>Forward propagation involves multiplying feature values with weights, adding bias, and then applying an activation function to each neuron in the neural network.</li><li>Multiplying feature values with weights and adding bias to each neuron is basically applying <strong>Linear Regression</strong>. If we apply Sigmoid function to it then each neuron is basically performing a <strong>Logistic Regression.</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZyHN5j51J92nwzdVcaFrtQ.png" /></figure><h4>Activation functions</h4><ul><li>The purpose of an activation function is to introduce <strong>non-linearity</strong> to the data. Introducing non-linearity helps to identify the underlying patterns which are complex. It is also used to scale the value to a particular interval. For example, the sigmoid activation function scales the value between 0 and 1.</li></ul><h4>Logistic or Sigmoid function</h4><ul><li>Logistic/ Sigmoid function scales the values between 0 and 1.</li><li>It is used in the output layer for Binary classification.</li><li>It may cause a <strong>vanishing gradient</strong> problem during backpropagation and slows the training time.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*NSdUKVuBQt5vokwJHCwKCw.png" /><figcaption>Sigmoid function</figcaption></figure><h4>Tanh function</h4><ul><li>Tanh is the short form for <strong>Hyperbolic Tangent</strong>. Tanh function scales the values between -1 and 1.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*tAE9A9tDlhnAGXq_-5y2JQ.png" /><figcaption>Hyperbolic Tangent function</figcaption></figure><h4>ReLU function</h4><ul><li><strong>ReLU (Rectified Linear Unit)</strong> outputs the same number if x&gt;0 and outputs 0 if x&lt;0.</li><li>It prevents the <strong>vanishing gradient</strong> problem but introduces an <strong>exploding</strong> <strong>gradient problem</strong> during backpropagation. The exploding gradient problem can be prevented by capping gradients.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*fALttJAfgDC1xxPgHoTpIQ.png" /><figcaption>ReLU function</figcaption></figure><h4>Leaky ReLU function</h4><ul><li>Leaky ReLU is very much similar to ReLU but when x&lt;0 it returns (0.01 * x) instead of 0.</li><li>If the data is normalized using Z-Score it may contain negative values and ReLU would fail to consider it but leaky ReLU overcomes this problem.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*Uvovl617zz6eJVjQTVi3hQ.png" /><figcaption>Leaky ReLU function</figcaption></figure><h4>Backpropagation</h4><ul><li>Backpropagation is done to find the <strong>optimal value for parameters</strong> for the model by iteratively updating parameters by partially differentiating <strong>gradients of the loss function</strong> with respect to the <strong>parameters</strong>.</li><li>An optimization function is applied to perform backpropagation. The objective of an optimization function is to find the optimal value for parameters.</li></ul><p>The optimization functions available are,</p><ul><li>Gradient Descent</li><li>Adam optimizer</li><li>Gradient Descent with momentum</li><li>RMS Prop (Root Mean Square Prop)</li></ul><p>The <strong>Chain rule of Calculus</strong> plays an important role in backpropagation. The formula below denotes partial differentiation of Loss (L) with respect to Weights/ parameters (w).</p><p>A small change in weights ‘w’ influences the change in the value ‘z’ (∂𝑧/∂𝑤). A small change in the value ‘z’ influences the change in the activation ‘a’ (∂a/∂z). A small change in the activation ‘a’ influences the change in the Loss function ‘L’ (∂L/∂a).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*jV3Ph2feCAOEZDi9-skl1Q.png" /><figcaption>Chain rule</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9PPYx9CIhxDQ5gF6WNE9TA.png" /><figcaption>Description of the values in the Chain rule</figcaption></figure><h3><strong>Terminologies:</strong></h3><h4>Metrics</h4><ul><li>A metric is used to gauge the performance of the model.</li><li>Metric functions are similar to cost functions, except that the results from evaluating a metric are not used when training the model. Note that you may use any cost function as a metric.</li><li>We have used Mean Squared Logarithmic Error as a metric and cost function.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*bgzceS9jybK6YzuRrGTbPw.png" /><figcaption>Mean Squared Logarithmic Error (MSLE) and Root Mean Squared Logarithmic Error(RMSLE)</figcaption></figure><h4>Epoch</h4><ul><li>A single pass through the training data is called an epoch. The training data is fed to the model in mini-batches and when all the mini-batches of the training data are fed to the model that constitutes an epoch.</li></ul><h4>Hyperparameters</h4><p>Hyperparameters are the <strong>tunable parameters</strong> that are not produced by a model which means the users must provide a value for these parameters. The values of hyperparameters that we provide affect the training process so hyperparameter optimization comes to the rescue.</p><p>The Hyperparameters used in this ANN model are,</p><ul><li>Number of layers</li><li>Number of units/ neurons in a layer</li><li>Activation function</li><li>Initialization of weights</li><li>Loss function</li><li>Metric</li><li>Optimizer</li><li>Number of epochs</li></ul><h3>Coding ANN in Tensorflow</h3><h4>Load the preprocessed data</h4><p>The data you feed to the ANN must be preprocessed thoroughly to yield reliable results. The training data has been preprocessed already. The preprocessing steps involved are,</p><ul><li>MICE Imputation</li><li>Log transformation</li><li>Square root transformation</li><li>Ordinal Encoding</li><li>Target Encoding</li><li>Z-Score Normalization</li></ul><p>For the detailed implementation of the above-mentioned steps refer my notebook on data preprocessing</p><p><a href="https://www.kaggle.com/srivignesh/data-preprocessing-for-house-price-prediction">Notebook Link</a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c18289ae4e7be0687e0269669e5e172c/href">https://medium.com/media/c18289ae4e7be0687e0269669e5e172c/href</a></iframe><h4>Neural Architecture</h4><ul><li>The ANN model that we are going to use, consists of seven layers including one input layer, one output layer, and five hidden layers.</li><li>The first layer (input layer) consists of 128 units/ neurons with the ReLU activation function.</li><li>The second, third, and fourth layers consist of 256 hidden units/ neurons with the ReLU activation function.</li><li>The fifth and sixth layer consists of 384 hidden units with ReLU activation function.</li><li>The last layer (output layer) consists of one single neuron which outputs an array with the shape (1, N) where N is the number of features.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c5507426149388c41966d8a23742ba84/href">https://medium.com/media/c5507426149388c41966d8a23742ba84/href</a></iframe><p><strong>Find this post in my Kaggle notebook: </strong><a href="https://www.kaggle.com/srivignesh/introduction-to-ann-in-tensorflow">https://www.kaggle.com/srivignesh/introduction-to-ann-in-tensorflow</a></p><p><strong>References:</strong></p><p>[1] Andrew Ng, <a href="https://www.coursera.org/specializations/deep-learning">Deep Learning Specialization</a>.</p><p><em>Connect with me on </em><a href="https://www.linkedin.com/in/srivignesh-rajan-123569151/"><em>LinkedIn</em></a><em>, </em><a href="https://twitter.com/RajanSrivignesh"><em>Twitter</em></a><em>!</em></p><p><strong>Happy Deep Learning!</strong></p><h4><strong>Thank you!</strong></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5d2e108ff2c3" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/an-introduction-to-artificial-neural-networks-5d2e108ff2c3">An Introduction to Artificial Neural Networks</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hands-on Categorical Feature Encoding in Machine Learning]]></title>
            <link>https://medium.com/swlh/an-introduction-to-categorical-feature-encoding-in-machine-learning-cd0ca08c8232?source=rss-517b08d1ae3------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd0ca08c8232</guid>
            <category><![CDATA[categorical-data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[beginners-guide]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Srivignesh Rajan]]></dc:creator>
            <pubDate>Tue, 14 Jul 2020 14:52:11 GMT</pubDate>
            <atom:updated>2020-07-20T15:46:58.755Z</atom:updated>
            <content:encoded><![CDATA[<h4>Categorical Feature Encoding: A Walkthrough in python!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MlYtI_x9imQ4lCOg" /><figcaption>Photo by <a href="https://unsplash.com/@maxcodes?utm_source=medium&amp;utm_medium=referral">Maxwell Nelson</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Categorical Feature Encoding</h3><p>Categorical feature encoding is an important step in data preprocessing. Categorical feature encoding is the process of converting the categorical features into numeric features. Categorical variables are also called as <strong>Qualitative</strong> variables. The results produced by the model varies when different encoding techniques are used.</p><p>Two types of categorical features exist,</p><ul><li><strong>Ordinal Features</strong></li><li><strong>Nominal Features</strong></li></ul><h3>Ordinal Features:</h3><ul><li>Ordinal features are the features that have inherent ordering.</li><li>Eg: Ratings such as Good, Bad.</li></ul><h3>Nominal Features:</h3><ul><li>Nominal features are the features that don’t have any inherent ordering as opposed to Ordinal features.</li><li>Eg: Names of persons, gender, yes, or no.</li></ul><h3>Need for categorical feature encoding</h3><ul><li>Categorical features must be encoded before feeding it to the model because many Machine Learning algorithms don’t support categorical features as their input.</li><li>Machine Learning algorithms and Deep Learning algorithms would support only <strong>numerical variables/ quantitative variables</strong>.</li></ul><h3>Ordinal Encoding Techniques</h3><ul><li>Label Encoding or Ordinal Encoding</li></ul><h3>Nominal Encoding Techniques</h3><ul><li>Frequency Encoding</li><li>Target Encoding</li><li>One-hot Encoding</li><li>Leave One Out Encoding</li><li>M-Estimate Encoding</li></ul><h3>Load the required libraries</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5c10ba84a28113dc5c46096bd8411edf/href">https://medium.com/media/5c10ba84a28113dc5c46096bd8411edf/href</a></iframe><h3>Ordinal Encoding</h3><h3>1. Ordinal Encoding using OrdinalEncoder in sci-kit learn</h3><p><strong>OrdinalEncoder</strong> is used to assign numerical values to the categories in the ordinal features.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0aa1fb7b634ab187e7915b8b755d0b7f/href">https://medium.com/media/0aa1fb7b634ab187e7915b8b755d0b7f/href</a></iframe><h3>2. Ordinal Encoding using LabelEncoder in sci-kit learn</h3><p><strong>LabelEncoder</strong> would also produce the same result produced by the <strong>OrdinalEncoder</strong>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1ea38bc00e819d126dcdaa27a9c2db47/href">https://medium.com/media/1ea38bc00e819d126dcdaa27a9c2db47/href</a></iframe><h3>Nominal Encoding</h3><h3>1. Frequency Encoding</h3><p>In frequency encoding, each of the categories in the feature is replaced with the frequencies of categories.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0J73Hd4I4nvWmC5ikUN5-A.png" /><figcaption>The formula for frequency encoding</figcaption></figure><p>Category refers to each of the unique values in a feature.</p><ul><li><strong><em>Frequency(category)</em> </strong>= Number of values in that category</li><li><strong><em>Size(data)</em> </strong>= Size of the entire dataset.</li></ul><p><strong>Disadvantage</strong>: If two categories have the same frequency then it is hard to distinguish between them.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a9d722a43b81fd78f323cb3e78a60aff/href">https://medium.com/media/a9d722a43b81fd78f323cb3e78a60aff/href</a></iframe><h3>2. Target Encoding</h3><p>In target encoding, each of the categories is replaced with the mean of the target variable. Target encoding is one of the most used categorical encoding techniques in Kaggle.</p><ul><li><em>Target Encoding = mean(target of a category)</em></li></ul><p><strong>Disadvantage</strong>: Tends to overfit the data if some of the categories have a low number of occurrences.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d473f2771befac0d4c5a0a946f5732b2/href">https://medium.com/media/d473f2771befac0d4c5a0a946f5732b2/href</a></iframe><h3>3. One-Hot Encoding</h3><p>One Hot Encoding replaces the categories with binary values. ‘N’ number of features is created if the unique values in a feature are equal to ‘N’.</p><p><strong>Disadvantages</strong>:</p><ul><li>Tree algorithms cannot be applied to one-hot encoded data since it creates a sparse matrix.</li><li>When the feature contains too many unique values, that many features are created which may result in overfitting.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/128ae259aee1b7bd2cd842137079c7b8/href">https://medium.com/media/128ae259aee1b7bd2cd842137079c7b8/href</a></iframe><h3>4. Leave One Out Encoding</h3><p>Leave One Out Encoding(LOOE) is very similar to Target Encoding but the difference is LOOE doesn’t consider the current row while calculating the mean of the target.</p><p><strong>Disadvantage</strong>: Tends to overfit to the data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/af8997e280e539df57ff3593e0743607/href">https://medium.com/media/af8997e280e539df57ff3593e0743607/href</a></iframe><h3>5. M-Estimate Encoding</h3><p>M-Estimate Encoding is also called as additive smoothing overcomes the disadvantages of the Target Encoding (overfitting) by considering a smoothing factor M to encode.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FbwSpUcWbGBfSpgJmf3h4Q.png" /><figcaption>The formula for M-Estimate encoding</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ef71273c761f7f0e1aa469990e2ec221/href">https://medium.com/media/ef71273c761f7f0e1aa469990e2ec221/href</a></iframe><p><strong>Find this post in my Kaggle notebook: </strong><a href="https://www.kaggle.com/srivignesh/categorical-feature-encoding-techniques">https://www.kaggle.com/srivignesh/categorical-feature-encoding-techniques</a></p><p><strong>References:</strong></p><p>[1] Patricio Cerda, Ga¨el Varoquaux, and Bal´azs K´egl, <a href="https://arxiv.org/pdf/1806.00979.pdf">Similarity encoding for learning with dirty categorical variables</a> (2018).</p><p><em>Connect with me on </em><a href="https://www.linkedin.com/in/srivignesh-rajan-123569151/"><em>LinkedIn</em></a><em>, </em><a href="https://twitter.com/RajanSrivignesh"><em>Twitter</em></a><em>!</em></p><p><strong>Happy Machine Learning!!</strong></p><h4><strong>Thank you!</strong></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd0ca08c8232" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/an-introduction-to-categorical-feature-encoding-in-machine-learning-cd0ca08c8232">Hands-on Categorical Feature Encoding in Machine Learning</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Techniques of Feature Selection in Machine Learning]]></title>
            <link>https://medium.com/analytics-vidhya/an-introduction-to-feature-selection-in-machine-learning-9d6f2d5e47?source=rss-517b08d1ae3------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d6f2d5e47</guid>
            <category><![CDATA[dimensionality-reduction]]></category>
            <category><![CDATA[beginners-guide]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <dc:creator><![CDATA[Srivignesh Rajan]]></dc:creator>
            <pubDate>Tue, 14 Jul 2020 13:37:49 GMT</pubDate>
            <atom:updated>2020-07-20T15:47:16.702Z</atom:updated>
            <content:encoded><![CDATA[<h4>Facing issues with Overfitting and Low accuracy? Feature Selection comes to the rescue</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4u0tDZP8pAvzWVuq" /><figcaption>Photo by <a href="https://unsplash.com/@hharritt?utm_source=medium&amp;utm_medium=referral">Hunter Harritt</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Dimensionality Reduction</h3><ul><li>Dimensionality reduction is the process of reducing the set of available features in the dataset.</li><li>The model could not be applied to the entire set of features directly which may lead to spurious predictions and generalization issues which in turn makes the model unreliable.</li><li>In order to prevent these issues dimensionality reduction is applied.</li></ul><h3>Need for dimensionality reduction</h3><p>Dimensionality reduction prevents overfitting.</p><ul><li>Overfitting is when the model memorizes the data and fails to generalize. Overfitting can be caused by flexible models (like decision tree) and high dimensional data as well.</li><li>The Overfitted model could not be applied to real-world problems due to the problem of generalization.</li></ul><h3>Types of Dimensionality Reduction</h3><ul><li><strong>Feature Selection</strong>: Feature selection methods attempt to reduce the features by discarding the least important features.</li><li><strong>Feature Extraction</strong>: Feature extraction methods attempt to reduce the features by combining the features and transforming it to the specified number of features.</li></ul><h3>Feature Selection</h3><ol><li>Filter methods</li><li>Wrapper methods</li><li>Embedded methods</li><li>Feature Importance</li></ol><h3>Import the required libraries</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8df612e0b44d8bb30c13624e10ebc75c/href">https://medium.com/media/8df612e0b44d8bb30c13624e10ebc75c/href</a></iframe><h3>Load the preprocessed data</h3><p>The training data has been preprocessed already. The preprocessing steps involved are,</p><ol><li>MICE Imputation</li><li>Log transformation</li><li>Square root transformation</li><li>Ordinal Encoding</li><li>Target Encoding</li><li>Z-Score Normalization</li></ol><p>For the detailed implementation of the above-mentioned steps refer to my Kaggle notebook on data preprocessing:</p><p><a href="https://www.kaggle.com/srivignesh/data-preprocessing-for-house-price-prediction">Notebook Link</a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7a1287077c2f8a8b1730f39650a3fcc4/href">https://medium.com/media/7a1287077c2f8a8b1730f39650a3fcc4/href</a></iframe><h3>Feature Selection</h3><h3>1. Filter Methods</h3><p>Filter methods select the features independent of the model used. It can use the following methods to select a useful set of features,</p><ul><li>Correlation for numeric columns</li><li>Chi2 association for categoric columns</li></ul><h3>Select K Best in sci-kit learn</h3><p><strong>F_Regression:</strong></p><p>F_Regression is used for numeric variables. F_Regression consists of 2 steps:</p><ul><li>Correlation is computed using each feature with the target.</li><li>The correlation is then converted to an F score and then to a p-value.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/948/1*TyJFq-lLIcSSWeR06labmA.png" /><figcaption>Correlation formula</figcaption></figure><p><strong>Chi2:</strong></p><ul><li>Chi2 is used for testing the association between categorical variables.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AgFUo5da5JgUmlRemjO9kg.png" /><figcaption>Chi2 formula</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/df8b052c03ba10734eb64feede6db1fd/href">https://medium.com/media/df8b052c03ba10734eb64feede6db1fd/href</a></iframe><h3>2. Wrapper methods</h3><p>Wrapper methods make use of an estimator to select a useful set of features. The techniques available are,</p><ul><li>Recursive Feature Elimination</li><li>Recursive Feature Elimination Cross-Validation</li></ul><h3>Recursive Feature Elimination (RFE)</h3><ul><li>The estimator that is provided to RFE assigns weights to features (e.g., the coefficients), RFE recursively eliminates a subset of features which have low weights assigned to it.</li><li>The estimator is trained with the initial set of features. The estimator might have attributes such as coef_ or feature_importances_. With that attribute of the estimator, we find the weights of each feature.</li><li>The least weighted features are removed from the current set of features. This procedure is repeated on the removed set until the specified number of features to select is finally reached.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4d674407f7d8fd2d04defad0d6818d57/href">https://medium.com/media/4d674407f7d8fd2d04defad0d6818d57/href</a></iframe><h3>Recursive Feature Elimination Cross-Validation (RFECV)</h3><ul><li>RFECV is very similar to RFE but it uses Cross Validation at each training phase and finally outputs the optimal number of columns to select.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/efd5f72376caa9394456b4d468c58916/href">https://medium.com/media/efd5f72376caa9394456b4d468c58916/href</a></iframe><h3>3. Embedded Methods</h3><p>Embedded methods select features during the training process itself.</p><ul><li>The coefficients of features become zero when the importance of that feature is low and therefore that feature is not utilized to make predictions.</li></ul><h3>LASSO regression</h3><p>LASSO stands for Least Absolute Shrinkage and Selection Operator</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/696/1*NVjc67ZRhVyz0qElUTFtsA.png" /><figcaption>LASSO cost function</figcaption></figure><p>λ = Penalty (Tuning Parameter)</p><p>When λ = 0 no parameters are eliminated and when λ = 1 it is equal to linear regression.</p><ul><li>The parameter estimates are found by minimizing this cost function.</li><li>When the coefficient estimates are less than λ/2 the coefficients become zero.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4600b180cffd58eb4056896d035fce67/href">https://medium.com/media/4600b180cffd58eb4056896d035fce67/href</a></iframe><h3>4. Feature Importance</h3><p>The feature importances is calculated after fitting the model to the entire set of features which assigns weights to each of the features.</p><ul><li>The model might have attributes such as coef_ or feature_importances_ which help to select the subset of features. Using this the least important features are pruned.</li></ul><h3>Select From Model in sci-kit learn</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6e884c9a26e0484190bb0deed3db076e/href">https://medium.com/media/6e884c9a26e0484190bb0deed3db076e/href</a></iframe><p><strong>Find this post in my Kaggle notebook: </strong><a href="https://www.kaggle.com/srivignesh/feature-selection-techniques">https://www.kaggle.com/srivignesh/feature-selection-techniques</a></p><p><strong>References:</strong></p><p>[1] M. Ramaswami and R. Bhaskaran, <a href="https://arxiv.org/pdf/0912.3924.pdf">A Study on Feature Selection Techniques in Educational Data Mining</a> (2009)</p><p><em>Connect with me on </em><a href="https://www.linkedin.com/in/srivignesh-rajan-123569151/"><em>LinkedIn</em></a><em>, </em><a href="https://twitter.com/RajanSrivignesh"><em>Twitter</em></a><em>!</em></p><h4><strong>Thank you!</strong></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d6f2d5e47" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/an-introduction-to-feature-selection-in-machine-learning-9d6f2d5e47">Techniques of Feature Selection in Machine Learning</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Methods to Handle Missing Data in Machine Learning]]></title>
            <link>https://medium.com/analytics-vidhya/techniques-for-handling-the-missing-data-in-machine-learning-a-walkthrough-in-python-fdc37103a66b?source=rss-517b08d1ae3------2</link>
            <guid isPermaLink="false">https://medium.com/p/fdc37103a66b</guid>
            <category><![CDATA[data-imputation]]></category>
            <category><![CDATA[data-cleaning]]></category>
            <category><![CDATA[data-preprocessing]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Srivignesh Rajan]]></dc:creator>
            <pubDate>Mon, 13 Jul 2020 17:48:34 GMT</pubDate>
            <atom:updated>2020-07-20T15:47:32.417Z</atom:updated>
            <content:encoded><![CDATA[<h4>Techniques for handling the Missing data in Machine Learning: A Walkthrough in Python</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*INNOb7JtIeEZylCo" /><figcaption>Photo by <a href="https://unsplash.com/@franki?utm_source=medium&amp;utm_medium=referral">Franki Chamaki</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Why handling missing data is important?</h3><p>The problem of missing data is prevalent in most of the research areas. Missing data produces various problems.</p><ul><li>First, the missingness of data reduces the power of statistical methods.</li><li>Second, the missing data can cause bias in the model.</li><li>Third, many machine learning packages in python does not accept missing data. It needs the missing data to be treated first.</li></ul><p>Each of these problems may lead to spurious conclusions that reduce the reliability of the model.</p><h3>Missing data mechanisms</h3><ul><li><strong>Missing Completely At Random (MCAR):</strong> The values are Missing Completely At Random (MCAR) if the missing data is completely not related to both observed and missing instances.</li><li><strong>Missing At Random (MAR):</strong> Missing At Random (MAR) is when the missing data is related to the observed data but not to the missing data.</li><li><strong>Missing Not At Random (MNAR):</strong> Missing Not At Random (MNAR) is data that is neither MAR nor MCAR. This implies that the missing data is related to both observed and missing instances.</li></ul><h3>Handling Missing data</h3><ol><li>Dropping Variables</li><li>Partial Deletion</li><li>Data Imputation</li></ol><h3>1. Dropping Variables</h3><p>Delete the column if it consists of more than <strong>70% of missing values</strong> otherwise, Data Imputation is the most preferred method than deleting it. Greater the information to the model, the greater is the reliability of the model’s results.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bfd2ca8c36b742e78446f195c6f8c09b/href">https://medium.com/media/bfd2ca8c36b742e78446f195c6f8c09b/href</a></iframe><h3>2. Partial Deletion</h3><h3>2.1 Listwise Deletion</h3><p><strong>Listwise deletion</strong> is a technique in which the rows that contain missing values are deleted.</p><p><strong>Disadvantages:</strong> Listwise deletion diminishes the intensity of the statistical tests performed. More samples of data are the key factor in the analysis. Because listwise deletion drops the data with missing values, it diminishes the data size. Listwise deletion may cause bias in the data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c6f9931363c24e36281dd1a836b4ea82/href">https://medium.com/media/c6f9931363c24e36281dd1a836b4ea82/href</a></iframe><h3>3. Data Imputation</h3><h3>3.1 Single Imputation</h3><p>Single Imputation attempts to impute the missing data by a single value as opposed to Multiple Imputation which replaces the missing data with multiple values.</p><blockquote><strong><em>3.1.1 Single Imputation for Numeric columns</em></strong></blockquote><p>3.1.1.1 Mean Imputation</p><p>3.1.1.2 Regression Imputation</p><blockquote><strong><em>3.1.2 Single Imputation for Categoric columns</em></strong></blockquote><p>3.1.2.1 Mode Imputation</p><h4>3.1.1 Single Imputation for Numeric columns</h4><p><strong>3.1.1.1 Mean Imputation</strong></p><ul><li>Mean Imputation is the process of imputing the missing data by the mean of the variable and can be done only to numeric columns.</li></ul><p><strong>Disadvantage:</strong> Mean Imputation is more likely to introduce bias in the model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4027b2cff63be62fbab1c752f01c6004/href">https://medium.com/media/4027b2cff63be62fbab1c752f01c6004/href</a></iframe><p><strong>3.1.1.2 Regression Imputation</strong></p><ul><li>A Regression model is fitted where the predictors are the features without missing values and the targets are the features with missing values.</li><li>The missing values are then replaced with the predictions. Regression imputation is less likely to introduce bias in the model.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/836981f1354c7b636bf265b88bb99af6/href">https://medium.com/media/836981f1354c7b636bf265b88bb99af6/href</a></iframe><h4>3.1.2 Single Imputation for categoric columns</h4><p><strong>3.1.2.1 Mode Imputation</strong></p><ul><li>Mode Imputation is the process of imputing the missing data by the mode of the variable and can be done only to categoric columns.</li></ul><p><strong>Disadvantage:</strong> More likely to introduce bias in the model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d944aa0e144af90abc6225929512f21e/href">https://medium.com/media/d944aa0e144af90abc6225929512f21e/href</a></iframe><h3>3.2 Multiple Imputation</h3><ul><li>In multiple imputations, the missing data is imputed with multiple values multiple times culminating in a multitude of imputed datasets.</li></ul><h4>3.2.1 MICE (Multiple Imputation by Chained Equation)</h4><h4>MICE Algorithm:</h4><ul><li><strong>Step 1:</strong> Initially, the dataset is imputed with mean which acts as the “placeholders”.</li><li><strong>Step 2:</strong> A missing variable is chosen at random and entitled as the target and the other features except that will act as features. A regression model is trained on these features and the target.</li><li><strong>Step 3:</strong> The missing values of the target are then replaced with predictions (imputations) from the regression model.</li><li><strong>Step 4:</strong> Steps 2–3 are then repeated for each feature that has unobserved data. At the end of a single cycle, all the missing values would have been imputed with the predictions.</li><li><strong>Step 5:</strong> Steps 2–4 are repeated for a specified number of cycles.</li></ul><h4>MICE Algorithm for Categorical data:</h4><p>Before going through steps 1 to 6 in the MICE algorithm the following steps must be done in order to impute categorical data.</p><ul><li><strong>Step 1:</strong> Ordinal Encode the non-null values</li><li><strong>Step 2:</strong> Use MICE imputation with Gradient Boosting Classifier to impute the ordinal encoded data</li><li><strong>Step 3:</strong> Convert back from ordinal values to categorical values.</li><li><strong>Step 4:</strong> Follow steps 1 to 5 in MICE Algorithm. Instead of using Mean imputation for initial strategy use <strong>Mode imputation</strong>.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c3041b4d9abca06567c6127092438077/href">https://medium.com/media/c3041b4d9abca06567c6127092438077/href</a></iframe><p><strong><em>Find this post in my Kaggle notebook: </em></strong><a href="https://www.kaggle.com/srivignesh/techniques-for-handling-the-missing-data"><em>https://www.kaggle.com/srivignesh/techniques-for-handling-the-missing-data</em></a></p><p><strong>References:</strong></p><p>[1] Hyun Kang<strong>, </strong><a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3668100/">The prevention and handling of the missing data</a> (2013), National Center for Biotechnology Information.</p><p>[2]<strong> </strong>Melissa J. Azur, Elizabeth A. Stuart, Constantine Frangakis, and Philip J. Leaf, <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3074241/">Multiple imputation by chained equations: what is it and how does it work?</a> (2011), National Center for Biotechnology Information.</p><p><em>Connect with me on </em><a href="https://www.linkedin.com/in/srivignesh-rajan-123569151/"><em>LinkedIn</em></a><em>, </em><a href="https://twitter.com/RajanSrivignesh"><em>Twitter</em></a><em>!</em></p><p><strong>Happy Machine Learning!</strong></p><h4><strong>Thank you!</strong></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fdc37103a66b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/techniques-for-handling-the-missing-data-in-machine-learning-a-walkthrough-in-python-fdc37103a66b">Methods to Handle Missing Data in Machine Learning</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Preprocessing Pipeline in Machine Learning]]></title>
            <link>https://medium.com/swlh/data-preprocessing-and-data-modeling-for-kaggle-house-price-prediction-data-in-python-c04055ded258?source=rss-517b08d1ae3------2</link>
            <guid isPermaLink="false">https://medium.com/p/c04055ded258</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-preprocessing]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Srivignesh Rajan]]></dc:creator>
            <pubDate>Fri, 10 Jul 2020 15:20:55 GMT</pubDate>
            <atom:updated>2020-07-20T15:41:17.280Z</atom:updated>
            <content:encoded><![CDATA[<h4>Data Preprocessing pipeline for Machine Learning using Kaggle House Price Prediction competition’s data: A walkthrough in Python</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*egNuwDxtmGTPRp-B" /><figcaption>Photo by <a href="https://unsplash.com/@grakozy?utm_source=medium&amp;utm_medium=referral">Greg Rakozy</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Data Preprocessing:</h3><p>Data preprocessing is a predominant step in machine learning to yield highly accurate and insightful results. Greater the quality of data, the greater is the reliability of the produced results. <strong>Incomplete, noisy, and inconsistent data</strong> are the inherent nature of real-world datasets. Data preprocessing helps in increasing the quality of data by filling in missing incomplete data, smoothing noise, and resolving inconsistencies.</p><ul><li><strong>Incomplete data</strong> can occur due to many reasons. Appropriate data may not be persisted due to a misunderstanding, or because of instrument defects and misfunctions.</li><li><strong>Noisy data</strong> can occur for a number of reasons (having incorrect feature values). The instruments used for the data collection might be faulty. Data entry may contain human or instrument errors. Data transmission errors might occur as well.</li></ul><p>There are many stages involved in data preprocessing,</p><ol><li><strong>Data Cleaning</strong></li><li><strong>Data Integration</strong></li><li><strong>Data Transformation</strong></li><li><strong>Data Reduction</strong></li></ol><p><strong>Data cleaning</strong> attempts to impute missing values, smooth out noise, resolve inconsistencies, removing outliers in the data.</p><p><strong>Data integration</strong> integrates data from a multitude of sources into a single data warehouse.</p><p><strong>Data transformations</strong>, such as normalization, may be applied. For example, normalization may improve the accuracy and efficiency of mining algorithms involving distance measurements.</p><p><strong>Data reduction</strong> can reduce the data size by dropping out redundant features. Feature selection and feature extraction techniques can be used.</p><h3>Import the required libraries and load the dataset for training and testing</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8c2074f2be68780664123e627618ce8f/href">https://medium.com/media/8c2074f2be68780664123e627618ce8f/href</a></iframe><h3>1. Data Cleaning</h3><h3>1.1 Find the missing percentage of each column in the training set.</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8741066267d4dbbfc435d02605b77948/href">https://medium.com/media/8741066267d4dbbfc435d02605b77948/href</a></iframe><h3>1.2 Drop the columns which have more than 70% of missing values</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ad8417f70b95afa7193787b88a1f43c9/href">https://medium.com/media/ad8417f70b95afa7193787b88a1f43c9/href</a></iframe><h3>1.3 MICE (Multiple Imputation by Chained Equation)</h3><p>Imputation of missing values can be done using two techniques,</p><ul><li><strong>Single Imputation</strong></li><li>Single Imputation attempts to impute the missing data by a single value as opposed to Multiple Imputation which replaces the missing data with multiple values.</li><li><strong>Multiple Imputation</strong></li><li>In multiple imputations, the missing data is imputed with multiple values multiple times culminating in a multitude of imputed datasets.</li></ul><h4>MICE Algorithm:</h4><ul><li><strong>Step 1:</strong> Initially, the dataset is imputed with mean which acts as the “placeholders”.</li><li><strong>Step 2:</strong> A missing variable is chosen at random and entitled as the target and the other features except that will act as features. A regression model is trained on these features and the target.</li><li><strong>Step 3:</strong> The missing values of the target are then replaced with predictions (imputations) from the regression model.</li><li><strong>Step 4:</strong> Steps 2–3 are then repeated for each feature that has unobserved data. At the end of a single cycle, all the missing values would have been imputed with the predictions.</li><li><strong>Step 5:</strong> Steps 2–4 are repeated for the specified number of cycles.</li></ul><h4>MICE Algorithm for Categorical data:</h4><p>Before going through steps 1 to 5 in the MICE algorithm the following steps must be done in order to impute categorical data.</p><ul><li><strong>Step 1:</strong> Ordinal Encode the non-null values</li><li><strong>Step 2:</strong> Use MICE imputation with Gradient Boosting Classifier to impute the ordinal encoded data</li><li><strong>Step 3:</strong> Convert back from ordinal values to categorical values.</li><li><strong>Step 4:</strong> Follow steps 1 to 6 in MICE Algorithm. Instead of using Mean imputation for initial strategy use <strong>Mode imputation</strong>.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/006113f35bba622c31b402609161f408/href">https://medium.com/media/006113f35bba622c31b402609161f408/href</a></iframe><h3>2. Data Visualization</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fa512cd7136bef711241a623a07e7b99/href">https://medium.com/media/fa512cd7136bef711241a623a07e7b99/href</a></iframe><h3><strong>Sample Output:</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v7GUHmWt3Dg9irM5HlR55w.png" /><figcaption>Histogram of numeric columns.</figcaption></figure><h3>3. Data Transformation</h3><h3>3.1 Skewed data:</h3><p>Skewness is the distortion of data from normality. It may act as outliers and produce unreliable results. The skewed data must be transformed back to normal distribution.</p><ul><li>The distribution is <strong>highly skewed</strong> if the skewness is less than -1 or greater than 1.</li><li>The distribution is <strong>moderately skewed</strong> if the skewness is between -1 and -0.5 or between 0.5 and 1</li><li>The distribution is <strong>approximately symmetric </strong>if the skewness is between -0.5 and 0.5.</li><li>The distribution is <strong>symmetric </strong>if the skewness is 0.</li></ul><h4>3.1.1 Positively skewed data:</h4><blockquote><strong>Log transformation</strong> (when the data is highly skewed)</blockquote><ul><li>log(X) — if no zero values are present, log(C + X) — if zero values are present</li><li>C is a constant added so that the smallest value will be equal to 1.</li></ul><blockquote><strong>Square root transformation</strong> (when the data is moderately skewed)</blockquote><ul><li>sqrt(X)</li></ul><h4>3.1.2 Negatively skewed data:</h4><blockquote><strong>Reflect and Log transformation</strong></blockquote><ul><li>log(K — X) — K is a constant from which the values are subtracted so that the smallest value is 1.</li><li>(K — X) makes the large number small and the small number large so the negatively skewed data becomes positively skewed.</li></ul><blockquote><strong>Reflect and Square root transformation</strong></blockquote><ul><li>sqrt(K — X)</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2cf7b8c4a448132e4cff7cd2148b63ca/href">https://medium.com/media/2cf7b8c4a448132e4cff7cd2148b63ca/href</a></iframe><h3>3.2 Categorical Encoding</h3><h4>3.2.1 Ordinal Encoding:</h4><p>Ordinal columns are the ones that have ordinality or inherent order in themselves. Example ratings and feedback like excellent, good, fair, poor.</p><ul><li>Various <strong>Ordinal Encoding techniques</strong> are,</li><li>Label Encoding</li><li>Binary Encoding</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7108b704cf08be97193eaeca6e160fc7/href">https://medium.com/media/7108b704cf08be97193eaeca6e160fc7/href</a></iframe><h4>3.2.2 Nominal Encoding:</h4><p>Nominal columns are the ones that do not have any ordinality or inherent order. Example country names, gender (male, female).</p><ul><li>Various <strong>Nominal Encoding techniques</strong> available are,</li><li>Frequency Encoding</li><li>Target Encoding</li><li>M-Estimate Encoding</li><li>Leave One Out Encoding</li><li>One-Hot Encoding</li></ul><h4>3.2.2.1 Target Encoding:</h4><p>Target encoding is the process of encoding a qualitative/ categoric value with the mean of the target variable.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cf960fb30a50662a2efb40109edb9a1e/href">https://medium.com/media/cf960fb30a50662a2efb40109edb9a1e/href</a></iframe><h3>3.3 Normalization:</h3><ul><li>Normalization is also called as <strong>Feature Scaling</strong>. Normalization scales the values of features between a certain interval. Eg: [0,1]</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/1*cZu9voYAc8Gjnj5R8bRt_Q.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/320380ed4cbb1f84dac563a88930a300/href">https://medium.com/media/320380ed4cbb1f84dac563a88930a300/href</a></iframe><h3>4. Data Modeling</h3><p>Fit XGBoost Regressor model to the preprocessed data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9c63af100935702e2e85bc613f9f60a6/href">https://medium.com/media/9c63af100935702e2e85bc613f9f60a6/href</a></iframe><p><strong>Find this post in my Kaggle Notebook</strong>: <a href="https://www.kaggle.com/srivignesh/data-preprocessing-for-house-price-prediction">https://www.kaggle.com/srivignesh/data-preprocessing-for-house-price-prediction</a></p><p><strong>Data Source: </strong><a href="https://www.kaggle.com/c/house-prices-advanced-regression-techniques/data">https://www.kaggle.com/c/house-prices-advanced-regression-techniques/data</a></p><p><strong>References</strong>:</p><p>[1] Jiawei Han and Micheline Kamber, Data Mining: Concepts and Techniques Second Edition (2006)</p><p>[2]<strong> </strong>Melissa J. Azur, Elizabeth A. Stuart, Constantine Frangakis, and Philip J. Leaf, <a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3074241/">Multiple imputation by chained equations: what is it and how does it work?</a> (2011), National Center for Biotechnology Information.</p><p>[3] Suraj Donthi,<a href="https://projector-video-pdf-converter.datacamp.com/17404/chapter4.pdf"> Dealing with missing data in python.</a></p><p><em>Connect with me on </em><a href="https://www.linkedin.com/in/srivignesh-rajan-123569151/"><em>LinkedIn</em></a><em>, </em><a href="https://twitter.com/RajanSrivignesh"><em>Twitter</em></a><em>!</em></p><h4>Thank you!</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c04055ded258" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/data-preprocessing-and-data-modeling-for-kaggle-house-price-prediction-data-in-python-c04055ded258">Data Preprocessing Pipeline in Machine Learning</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>