<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Umang Saluja on Medium]]></title>
        <description><![CDATA[Stories by Umang Saluja on Medium]]></description>
        <link>https://medium.com/@umangsaluja?source=rss-13b6cdff6950------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*raq4rfdoP-pu6fRwSICI3w.png</url>
            <title>Stories by Umang Saluja on Medium</title>
            <link>https://medium.com/@umangsaluja?source=rss-13b6cdff6950------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 04:06:33 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@umangsaluja/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Quickest Way To Cross-Validate Machine Learning Model Using Python.]]></title>
            <link>https://blog.devgenius.io/quickest-way-to-cross-validate-machine-learning-model-using-python-a55b07eedeef?source=rss-13b6cdff6950------2</link>
            <guid isPermaLink="false">https://medium.com/p/a55b07eedeef</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[machine-learning-ai]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-validation]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Umang Saluja]]></dc:creator>
            <pubDate>Mon, 22 Jul 2024 09:00:00 GMT</pubDate>
            <atom:updated>2024-07-25T10:10:05.228Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>Introduction</strong></h4><p>Whenever we build any machine learning model, we feed it with initial data to train the model. And then we feed some unknown data (test data) to understand how well the model performs and generalized over unseen data. If the model performs well on the unseen data, it’s consistent and is able to predict with good accuracy on a wide range of input data; then this model is stable.</p><p>But this is not the case always! Machine learning models are not always stable and we have to evaluate the stability of the machine learning model. That is where <strong>Cross Validation</strong> comes into the picture.</p><blockquote><strong><em>“In simple terms, Cross-Validation is a technique used to assess how well our Machine learning models perform on unseen data”</em></strong></blockquote><p>According to Wikipedia, Cross-Validation is the process of assessing how the results of a statistical analysis will generalize to an independent data set.</p><p><strong>There are many ways to perform Cross-Validation and we will learn about 4 methods in this article.</strong></p><p>Let’s first understand the need for Cross-Validation!</p><h4><strong>Why do we need Cross Validation?</strong></h4><p>Suppose you build a machine learning model to solve a problem, and you have trained the model on a given dataset. When you check the accuracy of the model on the training data, it is close to 95%. Does this mean that your model has trained very well, and it is the best model because of the high accuracy?</p><p>No, it’s not! Because your model is trained on the given data, it knows the data well, captured even the minute variations(noise), and has generalized very well over the given data. If you expose the model to completely new, unseen data, it might not predict with the same accuracy and it might fail to generalize over the new data. This problem is called over-fitting.</p><p>Sometimes the model doesn’t train well on the training set as it’s not able to find patterns. In this case, it wouldn’t perform well on the test set as well. This problem is called Under-fitting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/0*BBdH7ocHw_gB5vEy.png" /></figure><p>Image Source: fireblazeaischool.in</p><p>To overcome over-fitting problems, we use a technique called Cross-Validation.</p><h3>What is Cross Validation?</h3><p><strong>Cross-Validation</strong> is a resampling technique with the fundamental idea of splitting the dataset into 2 parts- training data and test data. Train data is used to train the model and the unseen test data is used for prediction. If the model performs well over the test data and gives good accuracy, it means the model hasn’t overfitted the training data and can be used for prediction.</p><p>Let’s dive deep and learn about some of the model evaluation techniques.</p><h3>1. Hold Out method</h3><p>This is the simplest evaluation method and is widely used in Machine Learning projects. Here the entire dataset(population) is divided into 2 sets — train set and test set. The data can be divided into 70–30 or 60–40, 75–25 or 80–20, or even 50–50 depending on the use case. As a rule, the proportion of training data has to be larger than the test data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/857/0*22e044NZOKyuU3k-.jpg" /></figure><p>Image Source: DataVedas</p><p>The data split happens randomly, and we can’t be sure which data ends up in the train and test bucket during the split unless we specify random_state. This can lead to extremely high variance and every time, the split changes, the accuracy will also change.</p><p>There are some drawbacks to this method:</p><ul><li>In the Hold out method, the test error rates are highly variable (<strong>high variance</strong>) and it totally depends on which observations end up in the training set and test set</li><li>Only a part of the data is used to train the model (<strong>high bias</strong>) which is not a very good idea when data is not huge and this will lead to overestimation of test error.</li></ul><p>One of the major advantages of this method is that it is computationally inexpensive compared to other cross-validation techniques.</p><p><strong>Quick implementation of Hold Out method in Python</strong></p><p><strong>Python Code:</strong></p><pre>from sklearn.model_selection import train_test_split<br>X = [10, 20, 30, 40, 50, 60, 70, 80, 90, 100]<br>X_train, X_test= train_test_split(X, test_size=0.3, random_state=1)<br>print(&#39;Train:&#39;, X_train, &#39;Test:&#39;, X_test)</pre><pre>Output<br>Train: [50, 10, 40, 20, 80, 90, 60] Test: [30, 100, 70]</pre><p>Here, <em>random_state</em> is the seed used for reproducibility.</p><h3>2. Leave One Out Cross-Validation</h3><p>In this method, we divide the data into train and test sets — but with a twist. Instead of dividing the data into 2 subsets, we select a single observation as test data, and everything else is labeled as training data and the model is trained. Now the 2nd observation is selected as test data and the model is trained on the remaining data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/0*SmFv-ssR1QTUcfPV.png" /></figure><p>Image Source: ISLR</p><p>This process continues ’n’ times and the average of all these iterations is calculated and estimated as the test set error.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/197/0*jHxp6nITxdd_djYH.png" /></figure><p>When it comes to test-error estimates, LOOCV gives unbiased estimates (<strong>low bias</strong>). But bias is not the only matter of concern in estimation problems. We should also consider variance.</p><p>LOOCV has an extremely <strong>high variance</strong> because we are averaging the output of n-models which are fitted on an almost identical set of observations, and their outputs are highly positively correlated with each other.</p><p>And you can clearly see this is computationally expensive as the model is run ’n’ times to test every observation in the data. Our next method will tackle this problem and give us a good balance between bias and variance.</p><p><strong>Quick implementation of Leave One Out Cross-Validation in Python</strong></p><pre>from sklearn.model_selection import LeaveOneOut</pre><pre>X = [10,20,30,40,50,60,70,80,90,100]</pre><pre>l = LeaveOneOut()</pre><pre>for train, test in l.split(X):</pre><pre>    print(&quot;%s %s&quot;% (train,test))</pre><pre>Output</pre><pre>[1 2 3 4 5 6 7 8 9] [0]<br>[0 2 3 4 5 6 7 8 9] [1]<br>[0 1 3 4 5 6 7 8 9] [2]<br>[0 1 2 4 5 6 7 8 9] [3]<br>[0 1 2 3 5 6 7 8 9] [4]<br>[0 1 2 3 4 6 7 8 9] [5]<br>[0 1 2 3 4 5 7 8 9] [6]<br>[0 1 2 3 4 5 6 8 9] [7]<br>[0 1 2 3 4 5 6 7 9] [8]<br>[0 1 2 3 4 5 6 7 8] [9]</pre><p>This output clearly shows how LOOCV keeps one observation aside as test data and all the other observations go to train data.</p><h3>3. K-Fold Cross-Validation</h3><p>In this resampling technique, the whole data is divided into k sets of almost equal sizes. The first set is selected as the test set and the model is trained on the remaining k-1 sets. The test error rate is then calculated after fitting the model to the test data.</p><p>In the second iteration, the 2nd set is selected as a test set and the remaining k-1 sets are used to train the data and the error is calculated. This process continues for all the k sets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/555/0*QOlco0-S8jl8R8Ty.jpg" /></figure><p>Image Source: Wikipedia</p><p>The mean of errors from all the iterations is calculated as the CV test error estimate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/241/0*eTS7tYzhU6AmKFj3.png" /></figure><p>In K-Fold CV, the no of folds k is less than the number of observations in the data (k&lt;n) and we are averaging the outputs of k fitted models that are somewhat less correlated with each other since the overlap between the training sets in each model is smaller. This leads to <strong>low variance</strong> then LOOCV.</p><p>The best part about this method is each data point gets to be in the test set exactly once and gets to be part of the training set k-1 times. As the number of folds k increases, the variance also decreases (low variance). This method leads to <strong>intermediate bias</strong> because each training set contains fewer observations (k-1)n/k than the Leave One Out method but more than the Hold Out method.</p><p>Typically, K-fold Cross Validation is performed using k=5 or k=10 as these values have been empirically shown to yield test error estimates that neither have high bias nor high variance.</p><p>The major disadvantage of this method is that the model has to be run from scratch k-times and is computationally expensive than the Hold Out method but better than the Leave One Out method.</p><p><strong>Simple implementation of K-Fold Cross-Validation in Python</strong></p><pre>from sklearn.model_selection import KFold</pre><pre>X = [&quot;a&quot;,&#39;b&#39;,&#39;c&#39;,&#39;d&#39;,&#39;e&#39;,&#39;f&#39;]</pre><pre>kf = KFold(n_splits=3, shuffle=False, random_state=None)</pre><pre>for train, test in kf.split(X):</pre><pre>    print(&quot;Train data&quot;,train,&quot;Test data&quot;,test)</pre><pre>Output</pre><pre>Train: [2 3 4 5] Test: [0 1]<br>Train: [0 1 4 5] Test: [2 3]<br>Train: [0 1 2 3] Test: [4 5]</pre><h3>4. Stratified K-Fold Cross-Validation</h3><p>This is a slight variation from K-Fold Cross Validation, which uses <strong>‘stratified sampling’</strong> instead of ‘random sampling.’</p><p>Let’s quickly understand what stratified sampling is and how is it different from random sampling.</p><p>Suppose your data contains reviews for a cosmetic product used by both the male and female population. When we perform random sampling to split the data into train and test sets, there is a possibility that most of the data representing males is not represented in training data but might end up in test data. When we train the model on sample training data that is not a correct representation of the actual population, the model will not predict the test data with good accuracy.</p><p>This is where Stratified Sampling comes to the rescue. Here the data is split in such a way that it represents all the classes from the population.</p><p>Let’s consider the above example which has a cosmetic product review of 1000 customers out of which 60% is female and 40% is male. I want to split the data into train and test data in proportion (80:20). 80% of 1000 customers will be 800 which will be chosen in such a way that there are 480 reviews associated with the female population and 320 representing the male population. In a similar fashion, 20% of 1000 customers will be chosen for the test data ( with the same female and male representation).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*F8LNuOpDTlDKPH4O.png" /></figure><p>Image Source: stackexchange.com</p><p>This is exactly what stratified K-Fold CV does and it will create K-Folds by preserving the percentage of sample for each class. This solves the problem of random sampling associated with Hold out and K-Fold methods.</p><p><strong>Quick implementation of Stratified K-Fold Cross-Validation in Python</strong></p><pre>from sklearn.model_selection import StratifiedKFold</pre><pre>X = np.array([[1,2],[3,4],[5,6],[7,8],[9,10],[11,12]])<br>y= np.array([0,0,1,0,1,1])</pre><pre>skf = StratifiedKFold(n_splits=3,random_state=None,shuffle=False)</pre><pre>for train_index,test_index in skf.split(X,y):<br>    print(&quot;Train:&quot;,train_index,&#39;Test:&#39;,test_index)<br>    X_train,X_test = X[train_index], X[test_index]<br>    y_train,y_test = y[train_index], y[test_index]</pre><pre>Output</pre><pre>Train: [1 3 4 5] Test: [0 2]<br>Train: [0 2 3 5] Test: [1 4]<br>Train: [0 1 2 4] Test: [3 5]</pre><p>The output clearly shows the stratified split done based on the classes ‘0’ and ‘1’ in ‘y’.</p><h3>Bias — Variance Tradeoff</h3><p>When we consider the test error rate estimates, K-Fold Cross Validation gives more accurate estimates than Leave One Out Cross-Validation. Whereas Hold One Out CV method usually leads to overestimates of the test error rate, because in this approach, only a portion of the data is used to train the machine learning model.</p><p>When it comes to bias, the Leave One Out Method gives unbiased estimates because each training set contains n-1 observations (which is pretty much all of the data). K-Fold CV leads to an intermediate level of bias depending on the number of k-folds when compared to LOOCV but it’s much lower when compared to the Hold Out Method.</p><p>To conclude, the Cross-Validation technique that we choose highly depends on the use case and bias-variance trade-off.</p><p>If you have read this article this far, here is a quick bonus for you</p><p><strong>sklearn.model_selection</strong> has a method <strong>cross_val_score</strong> which simplifies the process of cross-validation. Instead of iterating through the complete data using the ‘split’ function, we can use <em>cross_val_score</em> and check the accuracy score for the chosen cross-validation method</p><p>If you would like to share your thoughts, you can connect with me on <a href="https://www.linkedin.com/in/umang-saluja-bb864926a/"><strong>LinkedIn</strong></a>.</p><p><strong>Happy Learning!!</strong></p><p>Reference:-<a href="https://www.analyticsvidhya.com/blog/2021/05/4-ways-to-evaluate-your-machine-learning-model-cross-validation-techniques-with-python-code/">https://www.analyticsvidhya.com/blog/2021/05/4-ways-to-evaluate-your-machine-learning-model-cross-validation-techniques-with-python-code/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a55b07eedeef" width="1" height="1" alt=""><hr><p><a href="https://blog.devgenius.io/quickest-way-to-cross-validate-machine-learning-model-using-python-a55b07eedeef">Quickest Way To Cross-Validate Machine Learning Model Using Python.</a> was originally published in <a href="https://blog.devgenius.io/">Dev Genius</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Linear Regression with used case.]]></title>
            <link>https://medium.com/@umangsaluja/introduction-to-linear-regression-with-used-case-2069a8e37da1?source=rss-13b6cdff6950------2</link>
            <guid isPermaLink="false">https://medium.com/p/2069a8e37da1</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Umang Saluja]]></dc:creator>
            <pubDate>Sun, 19 Nov 2023 14:24:46 GMT</pubDate>
            <atom:updated>2023-11-19T14:24:46.300Z</atom:updated>
            <content:encoded><![CDATA[<p>Linear regression is a statistical method that is used to model the relationship between two or more variables. It is one of the most widely used statistical techniques, and it is often used as a first step in more complex analyses.</p><h3><strong>What is Linear Regression?</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/250/0*WQTfVX0oXWSCOApF" /><figcaption>Linear regression diagram</figcaption></figure><p>Linear regression is a supervised machine learning algorithm that assumes a linear relationship between one or more independent variables and a dependent variable. The goal of linear regression is to find the linear equation that best fits the data. This equation can then be used to predict the value of the dependent variable for new values of the independent variables.</p><h3>How Does Linear Regression Work?</h3><p>Linear regression works by fitting a straight line to the data. The equation of this line is:</p><pre>y = mx + b</pre><p>where:</p><ul><li>y is the dependent variable</li><li>x is the independent variable</li><li>m is the slope of the line</li><li>b is the y-intercept of the line</li></ul><p>The slope of the line tells us how much the dependent variable changes for a one-unit change in the independent variable. The y-intercept of the line tells us where the line crosses the y-axis.</p><h3><strong>When is Linear Regression Used?</strong></h3><p>Linear regression is used in a variety of applications, including:</p><ul><li>Predicting sales</li><li>Predicting stock prices</li><li>Modeling customer behavior</li><li>Analyzing medical data</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/696/0*4bLt-AdJtKXXED_e.png" /><figcaption>Linear Regression: Predicting House Prices</figcaption></figure><p>To predict the price of a house based on its size, a linear regression model is built. The model fits a straight line to the data points, aiming to capture the underlying trend. By extending this line, you can estimate the price for a given house size. Linear regression falls under the category of regression models, as it predicts numerical values, such as house prices in dollars.</p><h3>Training a Linear Regression Model</h3><p>To train the linear regression model, a training set is used. The training set comprises pairs of <strong>input features</strong><em> (e.g : square feet) </em>and corresponding <strong>output targets</strong><em> (e.g.: house price)</em>.</p><p>Let:</p><ul><li><em>“m” be the number of training examples.</em></li><li><em>“x” be the input feature representing a specific property of the house, such as its size.</em></li><li><em>“y” be the output target, representing the actual price of the house.</em></li><li><em>“i” be the index of a specific training example.</em></li></ul><p>A specific training example is denoted as (x^i, y^i).</p><p>Therefore, the equation can be written as:</p><pre>Training set: {(x¹, y¹), (x², y²), …, (x^m, y^m)}</pre><p><em>Where each pair </em><strong><em>(x^i, y^i)</em></strong><em> represents a specific house in the dataset, and “m” denotes the number of training examples.</em></p><h3>The Function f(x) and Model Representation</h3><p>When training the model, a learning algorithm produces a function (f) that takes <strong>an input feature (x)</strong> and <strong>outputs an estimate or prediction (</strong>ŷ<strong>)</strong>. In linear regression, the function f(x) is defined as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/217/0*MBxC8DB4zHrRW-0V.png" /></figure><p><em>where:</em></p><ul><li><em>“w” and “b” are numeric values that determine the line’s </em><strong><em>slope</em></strong><em> and </em><strong><em>intercept</em></strong><em>, respectively</em></li></ul><h3>What are the Advantages of Linear Regression?</h3><p><strong>Linear regression is a popular statistical technique because it is:</strong></p><ul><li>Easy to understand and implement</li><li>Interpretable</li><li>Robust</li><li>Efficient</li></ul><h4>A crucial aspect of linear regression is the cost function. The cost function measures the model’s performance by quantifying the difference between predicted values and actual targets. This concept is widely applicable in machine learning and plays a vital role in training advanced AI models.</h4><h3><strong>What are the Disadvantages of Linear Regression?</strong></h3><p>Linear regression has several disadvantages, including:</p><ul><li>It assumes a linear relationship between the variables.</li><li>It is sensitive to outliers.</li><li>It can be overfitted to the data.</li></ul><p>Despite its limitations, linear regression is a valuable tool for understanding the relationships between variables. It is a powerful and versatile technique that can be used in a wide variety of applications.</p><p>With respect to our used case for predicting house prices based on specific features. Its simplicity and effectiveness make it widely used in the field of machine learning. By fitting a straight line to the data, the linear regression model can estimate the price of a house based on its size. This example demonstrates the concept of supervised learning, where the model is trained using labeled data to predict numeric outputs.</p><p><em>References: </em><a href="https://medium.com/@yennhi95zz/2-introduction-to-linear-regression-predicting-house-prices-a36f6172030">https://medium.com/@yennhi95zz/2-introduction-to-linear-regression-predicting-house-prices-a36f6172030</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2069a8e37da1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decoding the Language of Machines: A Journey into the World of Natural Language Processing (NLP).]]></title>
            <link>https://medium.com/@umangsaluja/decoding-the-language-of-machines-a-journey-into-the-world-of-natural-language-processing-nlp-e94cacd4faca?source=rss-13b6cdff6950------2</link>
            <guid isPermaLink="false">https://medium.com/p/e94cacd4faca</guid>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Umang Saluja]]></dc:creator>
            <pubDate>Sun, 19 Nov 2023 14:01:26 GMT</pubDate>
            <atom:updated>2023-11-19T14:01:26.101Z</atom:updated>
            <content:encoded><![CDATA[<p>In the realm of artificial intelligence, <strong>Natural Language Processing (NLP)</strong> stands as a beacon of innovation, bridging the gap between human language and the computational prowess of machines. NLP empowers computers to comprehend, interpret, and generate human language, enabling them to engage in meaningful interactions with us. This transformative technology is revolutionizing industries and shaping the future of human-computer communication.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/295/1*q29TtGnJlPBelwqBrAFoug.jpeg" /><figcaption>NLP</figcaption></figure><p><strong>Delving into the Essence of NLP</strong></p><p>At its core, NLP encompasses a suite of techniques that enable computers to process and analyze text-based data. These techniques encompass tasks such as:</p><ul><li>Natural Language Understanding (NLU): Extracting meaning from human language, including identifying entities, relationships, and sentiment.</li><li>Natural Language Generation (NLG): Converting structured data into human-readable text, such as generating summaries or creating reports.</li><li>Machine Translation: Translating text from one language to another, breaking down linguistic barriers and fostering global communication.</li><li>Speech Recognition: Converting spoken language into text, enabling machines to understand and respond to human speech.</li></ul><p><strong>The Impact of NLP in Our Daily Lives</strong></p><p>NLP’s influence permeates various aspects of our lives, often unnoticed yet profoundly impactful. From the moment we wake up and check our smartphones for news updates and personalized recommendations to the way we interact with chatbots for customer service or virtual assistants for scheduling tasks, NLP is seamlessly woven into our digital experiences.</p><p>In the healthcare sector, NLP is revolutionizing medical diagnosis and treatment. By analyzing patient records, medical images, and clinical notes, NLP algorithms can identify patterns and correlations, assisting doctors in making informed decisions and providing personalized care.</p><p>In the legal field, NLP is transforming document review and analysis, enabling lawyers to efficiently sift through vast amounts of text, identify relevant information, and streamline legal processes.</p><p>In the education sector, NLP-powered tools are enhancing personalized learning experiences. Adaptive learning systems use NLP to tailor instruction to individual student needs, providing targeted feedback and support.</p><p><strong>The Future of NLP: A World of Possibilities</strong></p><p>As NLP research continues to advance, we can anticipate a future where machines become even more adept at understanding and responding to human language. This progress will lead to breakthroughs in various domains, including:</p><ul><li>Human-computer interaction: Natural and intuitive communication between humans and machines, revolutionizing the way we interact with technology.</li><li>Content creation: AI-powered tools capable of generating creative content, such as writing novels, composing music, or designing art.</li><li>Augmented decision-making: NLP algorithms providing insights and recommendations based on vast amounts of unstructured data, aiding decision-makers in various fields.</li></ul><p><strong>Conclusion: Embracing the NLP Revolution</strong></p><p>NLP is poised to play an increasingly transformative role in our lives, shaping the way we communicate, work, and learn. As we embrace the technological revolution, it is crucial to consider the ethical implications and ensure that NLP is used responsibly and for the benefit of humanity. With careful consideration and responsible development, NLP holds the potential to enhance our understanding of the world, empower individuals, and shape a brighter future.</p><p>Image Source:- <a href="https://www.google.com/url?sa=i&amp;url=https%3A%2F%2Fmedium.com%2Fmlearning-ai%2Fbuilding-an-end-to-end-nlp-application-an-overview-ef0221c4ab1f&amp;psig=AOvVaw1DkSIFTAg0ph36ln3S670n&amp;ust=1700488459644000&amp;source=images&amp;cd=vfe&amp;opi=89978449&amp;ved=2ahUKEwjMm_alm9CCAxULQWwGHULkCHsQr4kDegQIARBS">https://www.google.com/url?sa=i&amp;url=https%3A%2F%2Fmedium.com%2Fmlearning-ai%2Fbuilding-an-end-to-end-nlp-application-an-overview-ef0221c4ab1f&amp;psig=AOvVaw1DkSIFTAg0ph36ln3S670n&amp;ust=1700488459644000&amp;source=images&amp;cd=vfe&amp;opi=89978449&amp;ved=2ahUKEwjMm_alm9CCAxULQWwGHULkCHsQr4kDegQIARBS</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e94cacd4faca" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[EDA(Exploratory Data Analysis) Practice on Adult Income Dataset.]]></title>
            <link>https://towardsdev.com/eda-exploratory-data-analysis-practice-on-adult-income-dataset-f15661961b5e?source=rss-13b6cdff6950------2</link>
            <guid isPermaLink="false">https://medium.com/p/f15661961b5e</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Umang Saluja]]></dc:creator>
            <pubDate>Mon, 12 Sep 2022 15:20:55 GMT</pubDate>
            <atom:updated>2022-09-12T15:20:55.599Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>Hi Folks, Greetings!!</em></strong></p><h4>Let’s see the content we’re going to cover through this blog.</h4><ol><li><strong>Understand the Dataset</strong></li><li><strong>Measures of Central Tendency</strong></li><li><strong>Measures of Dispersion</strong></li><li><strong>Distribution of the Data</strong></li><li><strong>Correlation</strong></li></ol><p>The dataset contains information about the individuals’ age, gender, and several attributes affecting the income of a person.<br><strong>Attribute information:</strong></p><p><strong>age:</strong> Age of an individual; <strong>workclass:</strong> Working class of the individual; <strong>fnlwgt:</strong> Weights based on the demographic characteristics; <strong>education:</strong> Highest education of the individual; <strong>education.num:</strong> Education encoded with unique numbers; <strong>marital.status:</strong> Marital status of the individual; <strong>occupation:</strong> Occupation of the individual; <strong>relationship:</strong> Relation of the individual in the family;<strong>race:</strong> Race of the individual;<strong>sex:</strong> Gender of the individual (0 = Male, 1 = Female);<strong>capital.gain:</strong> Gain amount of the individual;<strong>capital.loss:</strong> Loss amount of the individual;<strong>hours.per.week:</strong> Working hours per week;<strong>native.country:</strong> The native country of the individual;<strong>income:</strong> Income of the individual.</p><blockquote>To download the data set, please visit:-<em> </em><a href="https://www.kaggle.com/datasets/umangsaluja/adult-income-dataset">https://www.kaggle.com/datasets/umangsaluja/adult-income-dataset</a></blockquote><blockquote><strong>## I’m doing the EDA with the sample of questions that I have designed, so let’s dirty our hands on Python.</strong></blockquote><p><strong>Import the required libraries.</strong></p><pre>import numpy as np<br>import pandas as pd<br>import matplotlib.pyplot as plt<br>import seaborn as sns<br>import warnings<br>from warnings import filterwarnings<br>filterwarnings(&#39;ignore&#39;)</pre><p><strong>Load the adult&#39;s income dataset and print first five observations.</strong></p><pre><strong># loading the dataset</strong><br>df= pd.read_csv(&#39;adult_income.csv&#39;)<br>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LjQUye39-OA9r6zLCR5VPA.png" /></figure><h3><strong>1. Understand the Dataset.</strong></h3><ol><li><strong>Find out the type of variables in the Dataset.</strong></li></ol><pre>df.info()</pre><pre><strong>#we can see that dataset has,32560 observations and 15 variables(columns).<br>#We have 7 int type variables and 8 object type variables.</strong></pre><pre><strong>Output:-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/1*41XP6_EGOgR1wDsQp5P7fQ.png" /></figure><p><strong>2. Check for data description?</strong></p><pre>df.describe().T<br><strong>Output:-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*30KfiPhKylVZApbCYcQkWw.png" /></figure><h3><strong>2. Measures of Central Tendency.</strong></h3><ol><li><strong>What is the average age of males and females in the data?</strong></li></ol><pre><strong># As given in data that 0 stands for Male and 1 stands for Female</strong>.<br>df.groupby(&#39;sex&#39;)[&#39;age&#39;].mean()</pre><pre><strong>Output:-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/262/1*lt1R6ekixrBM3hKQbglZCA.png" /></figure><p><strong>2. What is the value under which 95% of the capital gain would lie?</strong></p><pre>df[&#39;capital.gain&#39;].quantile(0.95)</pre><pre><strong>Output:-<br>5013.0</strong></pre><p><strong>3. Find the qualification of most of the Individuals.</strong></p><pre>df[&#39;education&#39;].mode()</pre><pre><strong>Output:-<br>0    HS-grad<br>dtype: object</strong></pre><p><strong>4. Find the middle most observation of age for different working class.</strong></p><pre>df.groupby(&#39;workclass&#39;)[&#39;age&#39;].median()<br><strong># Here ? Represent missing data.</strong></pre><pre><strong>Output:-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/243/1*q1TnN3yXieSzVv91iXndTg.png" /></figure><p>5. <strong>Compare the mean and trimmed mean of capital gain and comment on the presence of outliers.</strong></p><pre>df[&#39;capital.gain&#39;].mean()<br><strong>Output:-<br>1077.6488437087312</strong></pre><pre><strong># for trimmed mean</strong><br>import scipy<br>from scipy import stats</pre><pre>scipy.stats.trim_mean(df[&#39;capital.gain&#39;], proportiontocut=0.01)</pre><pre><strong># It Returns the mean of array after trimming distribution from both tails.</strong></pre><pre><strong>Output:- 493.7085331077058</strong></pre><p><strong>6. Find the percentage of the Asian-Pac-Islander race people in the data.</strong></p><pre>(df.race.value_counts()/len(df))*100<br><strong># Asian-Pac-Islander are approx 3.19%</strong></pre><pre><strong>Output:-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/311/1*-ypdO16ozzHMMwI4-XHKjw.png" /></figure><h3>3. Measures of Dispersion.</h3><ol><li><strong>Which occupation has more variability in the working hours?</strong></li></ol><pre>df.groupby(&#39;occupation&#39;)[&#39;hours.per.week&#39;].std().sort_values(ascending= False).head(5)<br><strong>Output:-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/365/1*fhwL1ZH3lw7HFBd6kj6xOQ.png" /></figure><p><strong>2.</strong> <strong>Are there any variables which can be excluded from further analysis based on their variability?</strong></p><pre>df.std()<br><strong># education.num can be dropped in terms of variability.<br><br>Output:-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/328/1*aqlPA4tfk7-KkFghchYVRw.png" /></figure><p><strong>3. Calculate the percentage of variation in work hours for each work class.</strong></p><pre>(df.groupby(&#39;workclass&#39;)[&#39;hours.per.week&#39;].std()/df.groupby(&#39;workclass&#39;)[&#39;hours.per.week&#39;].mean()) * 100</pre><pre><strong>Output:-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/367/1*mVBM16Qkco5DFPo_TGq5CQ.png" /></figure><h3>4. Distribution of the Data.</h3><ol><li><strong>Make a function to plot the categories of the variable and find the category with the highest frequency.</strong></li></ol><pre># <strong>Here we prepare the function by which we can analyze our categorical columns. It&#39;s best practice to have function for Univariate Analysis. </strong><br>def fun(columns):<br>    sns.countplot(data=df, x=columns)<br>    plt.xticks(rotation=90)<br>    plt.show()<br>    print(&#39;Mode is&#39;,df[columns].mode()[0])</pre><p>Recall the function (fun).</p><pre>fun(&#39;sex&#39;)</pre><pre><strong>Output:-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/530/1*oUwSyE_KMA0NiVcCmNP57A.png" /></figure><pre>fun(&#39;education&#39;)<br>Output:-</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*b74cL94R3SupaHScI8aNLQ.png" /></figure><pre>fun(&#39;relationship&#39;)</pre><pre><strong>Output:-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/511/1*2kRqAsekT_14gQ5wNnhRlw.png" /></figure><pre>fun(&#39;workclass&#39;)<br><strong>Output:-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/530/1*btoHVtjETeqF2_h7zK2lLQ.png" /></figure><p>2. Find the variables which have normal distribution.</p><pre><strong># To check for Normal distributions among variable</strong><br>plt.figure(figsize=(20,15))<br>j=1<br>for i in df.columns:<br>    if df[i].dtypes == &#39;float64&#39; or df[i].dtypes == &#39;int64&#39;:<br>        plt.subplot(5,4,j)<br>        sns.distplot(df[i])<br>        j+=1<br>        print(df[i].name,df[i].skew())<br>plt.tight_layout()<br>plt.show()</pre><pre><strong># we can see that variables like fnlwgt,capital.gain,capital.loss are highly skewed.</strong></pre><pre><strong>Output:-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5Fkk8RqFqkPQPHHaNRFAkA.png" /></figure><p><strong>2. Find the working class for most of the population in the data.</strong></p><pre>df.workclass.value_counts().plot(kind=&#39;bar&#39;,color=&#39;red&#39;)<br><strong># Majority of people are from Private workclass.</strong></pre><pre><strong>Output:-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/488/1*Ov_GWX1H-Oj90CNnkhpThg.png" /></figure><p><strong>3. How much time are most of the individuals working per week?</strong></p><pre>df[&#39;hours.per.week&#39;].value_counts()<br><strong># Majority of people are working for 40 hours per week.</strong></pre><pre><strong>Output:-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/1*tUOh1IfMqlAP62TdMF3gHg.png" /></figure><h3>5. Correlation.</h3><ol><li><strong>Find the effect of age on the working hours of a person.</strong></li></ol><pre>df[[&#39;age&#39;,&#39;hours.per.week&#39;]].corr()</pre><pre><strong>Output:-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/339/1*kHfyzzBa1YHYjjaJsz4f7A.png" /></figure><p><strong>2. Create the upper triangular correlation matrix</strong></p><pre>array = np.zeros_like(df.corr())<br>array[np.tril_indices_from(array,k=-1)]=True<br>sns.heatmap(df.corr(),mask= array,annot=True)<br>plt.show()</pre><pre><strong>Output:-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/557/1*zXQzo9lDaV6bc35kvf3Teg.png" /></figure><p><strong><em>If you like this article, make sure to clap until your hand start paining :) Take care, stay healthy, stay safe and keep learning.</em></strong></p><p>Thanks.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f15661961b5e" width="1" height="1" alt=""><hr><p><a href="https://towardsdev.com/eda-exploratory-data-analysis-practice-on-adult-income-dataset-f15661961b5e">EDA(Exploratory Data Analysis) Practice on Adult Income Dataset.</a> was originally published in <a href="https://towardsdev.com/">Towards Dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The 6 principles of persuasion that skyrocket your product or services.]]></title>
            <link>https://medium.com/@umangsaluja/the-6-principles-of-persuasion-that-skyrocket-your-product-or-services-a4b9f87ebb51?source=rss-13b6cdff6950------2</link>
            <guid isPermaLink="false">https://medium.com/p/a4b9f87ebb51</guid>
            <category><![CDATA[market-research-reports]]></category>
            <category><![CDATA[research]]></category>
            <category><![CDATA[marketing]]></category>
            <category><![CDATA[marketing-strategies]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Umang Saluja]]></dc:creator>
            <pubDate>Mon, 12 Sep 2022 06:24:25 GMT</pubDate>
            <atom:updated>2022-09-12T07:30:36.923Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>Hi Folks! Before we get into the topic, I pray for your good health and well-being.</em></strong></p><p><strong>Do you want to boost your conversions?</strong></p><p>If you would like to trap someone into the physiology of persuasion an influence for your business, then these significant principles are what you really want.</p><p>The marketing physiology says we must know the behavior and mindset of any person before targeting it as a potential customer.</p><p>Indeed, the 6 principle of persuasion given by <a href="https://www.referralcandy.com/blog/persuasion-marketing-examples/"><strong>Dr.Robert Cialdian </strong></a>is amply recommended to increase conversion of potential customer, no matter the industry.</p><p><strong>1. The Reciprocity Principle.</strong> — When you get a favor from someone you feel obligation to help them. In many social situations whenever we receive a favor, we want to return that obligation.</p><p>You might be wondering why? When you favor someone, this generates a feeling of indebted for you, in them and make them feel special. As people want to get respect go giving creates reciprocity.</p><p><strong>For Instance</strong>- like food outlets they give free sample to try for different food items so in the end we order for one or more.</p><p><strong>2. The Commitment Principle </strong>— This principle demonstrated that when we make commitment publicly, we are more likely to go through and deliver that commitment in order to maintain consistency.</p><p>When we make commitment it’s harder to change the decision, the psychological fact proves that people have aligned commitment with their self- respect. This principle of commitment tends to bring a higher rate of conversions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*IzRudwxGilBzaVu2uekI6A.png" /><figcaption><a href="https://www.osiaffiliate.com/blog/boost-conversions-using-e-commerce-tips/">https://www.osiaffiliate.com/blog/boost-conversions-using-e-commerce-tips/</a></figcaption></figure><p><strong>3. The Social Proof Principle-</strong> We humans always want to be a part of community however, it’s like showing prove to the customer to make him believe that he is at a right place.</p><p>Social proof is even known as <strong>“wisdom-of-the-crowds”</strong>. Social proof by the other customer report or result can help to hit the risk of clients and clarify the dilemmas. Therefore, it becomes crucial to add customer reviews on the website, so the visitors can know how the other feels about your product and services.</p><p><strong>For Instance</strong> — Before buying a product from Amazon or any other website, we generally seek for the “Social Prove” by checking for its reviews given by other customers, as their results.</p><p>This is my personal social proof, if an eatery or restaurant is always full of people, I will always like to give their food a try.</p><p><strong>4. The Authority Principle</strong>- Human have a tendency to obey authority figures. Its human nature! That authority for the product and services boost customer confidence to buy with credibility. Thus, if your product has any certification like getting an award or certified by any job titles, (e.g. Dr.) or by eminent resource personalities, this increases the product credibility.</p><p><strong>For Instance —</strong> when we visit the doctor, we see its degrees, then from where he had studied and seeing all that hanging frames in the clinic, mesmerize and satisfied us that we are at the right place. This even happens during advertisement, the specialist demonstrating about the product and services like, actor as (DOCTOR) recommending for toothpaste on TV during advertisement.</p><p><strong>5. The liking Principle</strong> — Does like to someone matters? Liking effects, the chances of being influenced by others. We like people who are similar to us as this give us common interest.</p><p>“People love to buy from people they like”. <strong>For Instance</strong> — That sounds absurd, but it makes sense when you understand that a company’s “About Us” page is an opportunity to tell buyers about the similarities between its staff and site visitors.</p><p>Thus, it is very crucial to cooperate with people and making them feel of helping them either by giving compliments of what they like, this hit their oxytocin in the brain, and they never stop buying from you.</p><p><strong>6. The Scarcity Principle</strong>- “Fear of Missing” Scarcity influence our decision in quick time. When we see product is short in supply or last day sale, we get more attractive and likely to buy more quantity in the fear of missing.</p><p>Scarcity is perception that motivates and encourage us to act instantly, the moment the brain sees the scarcity, the cortisol (the fear chemical) reaches the summit, ‘Do some action’.</p><p><strong>For Instance </strong>— Companies use scarcity principle to boost up their sales and conversions, as it’s a common tactic used over by many ways to attract and create “Fear of Missing”.</p><p><strong>It is mainly used on booking sites in many ways.</strong></p><p><strong>· 60% off on fresh arrivals.</strong></p><p><strong>· Hurry! 5 pieces left.</strong></p><p><strong>· Buy one, get one free.</strong></p><p><strong>· Festival bonanza, get it before it’s too late</strong>.</p><p>Such tactics are commonly played to that tends to boost sales and conversions.</p><p><strong>Hope you find this informative. Thanks!!</strong></p><p><strong><em>If you like this article, make sure to clap until your hand start paining :) Take care, stay healthy, stay safe and keep learning.</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a4b9f87ebb51" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python built-in function, eval().]]></title>
            <link>https://medium.com/@umangsaluja/python-built-in-function-eval-892686ee42c6?source=rss-13b6cdff6950------2</link>
            <guid isPermaLink="false">https://medium.com/p/892686ee42c6</guid>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Umang Saluja]]></dc:creator>
            <pubDate>Mon, 12 Sep 2022 06:06:03 GMT</pubDate>
            <atom:updated>2022-09-12T07:29:46.743Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong><em>Hi Folks! Before we get into the topic, I pray for your good health and well-being.</em></strong></h4><p>This would be a small article about eval() function, after going through this blog you will understand the usage of this function. I would explain this function with the examples so, that at the end of the blog you will be clear with its usage and syntax.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_1lMRxlGd0DTpEtMjdtjNg.png" /><figcaption><a href="https://www.schemecolor.com/green-python.php">https://www.schemecolor.com/green-python.php</a></figcaption></figure><p>Let’s get started!!</p><p><strong>What is eval() function?</strong></p><p>In python, eval() is a build in function that is used to evaluate the “String” python expression and return the result as an integer.</p><p>The Syntax for eval() function</p><pre>eval(expression, [globals[, locals]])</pre><p>NOTE-The argument and parameters in eval() function are “String” however, the value return would be the result of the evaluated expression that would be an integer.</p><p>USES- eval() function used to evaluate mathematical expression. The user can also use this function to evaluate string, as it’s the property of this function to convert the result into integer further examples will clearly demonstrate the property of this function.</p><p>To make it simple lets us look with an example and difference between input() and eval(), say we want to add 40+40 =80, let see</p><p>The use of input() function returned a string of the same input entered.</p><pre>user= input(&#39;enter the values-&#39;)<br>print(user)<br>type(user)</pre><pre><strong>Output:</strong>-<br><strong>enter the values-40+40</strong></pre><pre><strong>str  # we can see that datatype is string</strong></pre><p>The use of eval() function that convert string input to result integer.</p><pre>user= eval(input(&#39;enter the values-&#39;))<br>print(user)<br>type(user)</pre><pre><strong>Output:-<br>enter the values-40+40<br>80</strong></pre><pre><strong>int # we can see that data type is integer.</strong></pre><p>If you’re wandering to solve mathematical operations using eval() function, yes, you can perform that function for mathematical operation. Let’s look at an example for mathematical operation using eval().</p><pre>ex= eval(input(&#39;input the mathematical expression- &#39;)<br>print(ex)<br>print(type(ex)))</pre><pre><strong>Output:-</strong></pre><pre><strong>input the mathematical expression- 5000*2<br>100000<br>&lt;class &#39;int&#39;&gt;</strong></pre><p>I can assure you this function going to help you when you practice EDA however, the point is to know the concepts better, then you can answer any questions. If you want to spend some time reading through some material regarding the eval() function, <strong>I recommend you guys to read the documentation of eval ()function link shown below:</strong></p><p><a href="https://docs.python.org/3/library/functions.html?source=post_page-----601f87db191----------------------%23eval"><strong>https://docs.python.org/3/library/functions.html?source=post_page-----601f87db191----------------------%23eval</strong></a></p><p>I know that it’s very crisp blog, however my point was to make you aware about the eval() function properties, alright time to pen down!!</p><p><strong><em>If you like this article, make sure to clap until your hand start paining :) Take care, stay healthy, stay safe and keep learning.</em></strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=892686ee42c6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Explanation of Decision Tree From Scratch with Examples.]]></title>
            <link>https://medium.com/@umangsaluja/explanation-of-decision-tree-from-scratch-with-examples-559557e9ae9d?source=rss-13b6cdff6950------2</link>
            <guid isPermaLink="false">https://medium.com/p/559557e9ae9d</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Umang Saluja]]></dc:creator>
            <pubDate>Sun, 11 Sep 2022 17:02:01 GMT</pubDate>
            <atom:updated>2022-09-12T11:41:46.471Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Decision tree </strong>is one of the popular used models in the data science fields. Decision tree also known as management tool, where management have to make the decision based on rational decisions. It is a supervised machine learning algorithm used for classification as well as regression problems.</p><p>Most often, decision tree is used for classification concerns like binary classification(<strong>example-<em>whether the applicant for loan turns out to be defaulter or not</em></strong>). It can also be used in multi-class classification.</p><p>Decision tree algorithms being part of <strong>Supervised Learning</strong> take dependent and independent variables and try to find out the relationship between independent variable and dependent variable and express it as a tree structure.</p><p>In decision tree, use a flowchart like tree structure, and it shows predictions based on splits it starts with the root node and end the with the leaf node.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GXgzyd6EGwblDOAZTE2Qqw.png" /></figure><p>Let’s understand how decision tree works by splitting into nodes, let say we are given a data about cars and our objective is to find out the pattern exist with the <strong>Horse-Power, Weight and Car Type</strong>. Our target variable will be a <strong>Car Type</strong> having binary values <strong>L</strong> and<strong> S</strong>.(<strong>Figure-A</strong>)</p><blockquote>Now, what decision tree do is that it take our dataset and apply function on this dataset which is given as(<strong>Weight&gt;2000</strong>), on the basis of this our table will be bifurcated with data into 2 different tables, where the first table will be having the weights more than 2000 and another with weights less than 2000.</blockquote><p>Our goal is to have the <strong>leaf node</strong> where our<strong> Car Type </strong>become completely<strong> </strong>homogeneous, to reach to that level this tree further distribute our data on the basis of <strong>Horse-Power</strong> after that we can see in the below table that our data further need not to split as it’s perfectly homogeneous. Thus, in a very same fashion, our decision tree works.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*JZqIjv7Y-6xWhZ9VNLg-pQ.png" /><figcaption><strong>Figure-A</strong></figcaption></figure><p><strong><em>Decision Tree normally suffers from the problem of overfitting if it’s allowed to grow till is maximum depth.</em></strong></p><p>When we grow our decision tree to its depth we get <strong>low bias and high variance</strong>, we can say that our model will perform perfectly on the training dataset however, its efficiency falls on test dataset.</p><p>To prevent decision tree also known as <strong>Greedy Model </strong>from overfitting, we apply regularization, this is a way of controlling its behavior of overfitting.</p><p><strong><em>In decision tree, the Loss Function represented by impurity /purity of the target column. The loss function help in reduction of impurity of the target column, that is, by increasing the homogeneity of the target column.</em></strong></p><p>To understand the loss function, we need to understand how it imputes the purity /impurity of the target column. There are 2 ways to measure impurity.</p><ol><li><strong>Entropy:-</strong> Entropy is borrowed from physics, in decision tree, entropy is majorly used for selecting the best splitter, It helps us to measure the purity of the splits. A <strong>pure split</strong> means either you should get <em>yes </em>or <em>no.</em></li></ol><p>The formula for Entropy is given as:-</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/564/1*oZxJSkT-ooIkwcIMet6_0w.png" /></figure><p>Here, p+ is the probability of positive class</p><p>p– is the probability of negative class</p><p>S is the subset of the training example</p><p><strong>How Decision Tree use Entropy?</strong></p><p>Let’s suppose we have <strong>shipping Mode </strong>in data by which we have to deliver customer parcel either by <strong>Regular Air</strong> or <strong>Express Air, in the tree below we can see we have made splits in order to make decision tree.</strong></p><p>We see here the split is not pure because, we still have some negative class in our decision tree.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*CqptQ3yhHTvshwWCcny1Iw.png" /><figcaption>Figure-2</figcaption></figure><blockquote><strong><em>Always keep in mind higher the Entropy, lower the purity and higher will be the impurity.</em></strong> <strong>Keeping note</strong>, the goal of machine learning is to lower the uncertainty or impurity in the dataset. For this, a new metric called <strong>Information Gain(Info Gain)</strong> in the above figure-2 tells us how much the parent entropy has decreased after splitting the features.</blockquote><blockquote><strong>NOTE-:</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GyxLKfC4xLBYfsbXzbrDKw.png" /></figure><p>For more mathematical concepts, visit <a href="https://towardsdatascience.com/entropy-how-decision-trees-make-decisions-2946b9c18c8">://towardsdatascience.com/entropy-how-decision-trees-make-decisions-2946b9c18c8</a></p><p>2. <strong>Gini Index:- </strong>The Gini is also used to measure the impurity of our split, to select the feature to split we need to know how much pure or impure our split will be. <strong>It is calculated by subtracting the sum of the squared probabilities of each class from one. It is denoted by the given formula.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/357/1*tsfPtOjKvGHpCKR6YKF8yw.png" /></figure><p>The pure split means either you get <strong>Yes</strong> or <strong>No</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*CUfzXK2bI5LCWTKf0IlLZg.png" /><figcaption>Figure 3</figcaption></figure><p>we usually use Gini Index since it is computationally effective and take shorter period of time for execution since there is no logarithmic like their in entropy, thus the impurity of our dataset can be easily understood by Gini Index.</p><h3>When to stop splitting?</h3><p>You must be wandering to yourself that when do we stop growing tree, typically in real-world we have data that are large enough, which will lead to large number of splits and eventually result in overfitting. That means the tree will give very good accuracy on training data set and flops on test dataset.</p><p>To overcome that here I introduce a new term known as <strong><em>hyperparameter tuning also known as regularization</em></strong></p><p><strong>Regularization parameters includes:-</strong></p><p><strong>1.max_depth </strong>— Its the maximum length of a path from root to leaf (in terms of number of decision points). The leaf node is not split further. It could lead to a tree with leaf node containing many observations on one side of the tree, whereas on the other side, nodes containing much fewer observations get further split.</p><p><strong>2.min_sample_split</strong> — A limit to stop further splitting of nodes when the number of observations in the node is lower than this value.</p><p><strong>3.min_sample_leaf</strong> — Minimum number of samples a leaf node must have. When a leaf contains too few observations, further splitting will result in overfitting (modeling of noise in the data).</p><p><strong>4.min_weight_fraction_leaf </strong>— Same as min_sample_leaf, but expressed in fraction of total number of weighted instances.</p><p><strong>5.max_leaf_nodes</strong> — maximum number of leaf nodes in a tree.</p><p><strong>6.max_feature_size</strong> — max number of features that are evaluated for splitting each node.</p><p><strong>Now let&#39;s work on Python to see the practical implementation of Decision Tree. Here to showcase you, I am using Credit Card Defaulter dataset.</strong></p><p>For the dataset, please visit:- <a href="https://www.kaggle.com/umangsaluja/credit-card-defaulters">https://www.kaggle.com/umangsaluja/credit-card-defaulters</a></p><p><strong>Import the required libraries.</strong></p><pre>%matplotlib inline<br>import numpy as np<br>import pandas as pd<br># calculate accuracy measures and confusion matrix<br>from sklearn import metrics</pre><pre>from sklearn.tree import DecisionTreeClassifier<br>from sklearn.feature_extraction.text import CountVectorizer<br>import pydotplus<br>from IPython.display import Image<br>from sklearn.metrics import classification_report</pre><p><strong>Load the dataset and check the first five observations.</strong></p><pre><strong># reading the dataset<br></strong>credit_df = pd.read_csv(&quot;credit.csv&quot;)<br>credit_df.head()<br><strong>Output:-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZvrpuNGfb2JH_ERT1PfNpg.png" /></figure><pre><strong># data Info<br></strong>credit_df.info()<br><strong># many columns are of type object, i.e., strings. These need to be converted to ordinal type.</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/488/1*FQuJz3IgnaCNM4EqtyYVgw.png" /><figcaption>Figure 4</figcaption></figure><p>After performing EDA steps, we capture the target column (“default”) into separate vectors for training set and test set.</p><pre><strong># capture the target column (“default”) into separate vectors for training set and test set</strong></pre><pre>X = credit_df.drop(&quot;default&quot; , axis=1)</pre><pre>y = credit_df.pop(&quot;default&quot;)</pre><p>Using (“test_train_split”) we split our data with 70:30 for training and testing.</p><pre><strong># splitting data into training and test set for independent attributes</strong><br>from sklearn.model_selection import train_test_split</pre><pre>xtrain, xtest, ytrain, ytest = train_test_split(X, y, test_size=.30, random_state=1)</pre><p>Our next step is to apply<strong> Decision tree algorithm.</strong></p><pre><strong># invoking the decision tree classifier function. Using &#39;entropy&#39; method of finding the split columns. Other option <br># could be gini index.  Restricting the depth of the tree to 5 (no particular reason for selecting this)<br></strong>                                  <br>dt_model = DecisionTreeClassifier(criterion = &#39;entropy&#39; )<br>DT=dt_model.fit(xtrain, ytrain)</pre><pre><strong>Output:-<br>DecisionTreeClassifier(criterion=&#39;entropy&#39;)</strong></pre><p>Let’s build the tree.</p><pre>labels = xtrain.columns</pre><pre><strong># export a decision tree in DOT format<br># pass the &#39;decision_tree&#39; to export it to Graphviz<br># pass the column names to &#39;feature_names&#39;<br># pass the required class labels to &#39;class_names&#39;</strong><br>dot_data = tree.export_graphviz(DT, feature_names = labels, class_names = [&quot;0&quot;,&quot;1&quot;])</pre><pre><strong># plot the decision tree using DOT format in &#39;dot_data&#39;</strong><br>graph = pydotplus.graph_from_dot_data(dot_data)</pre><pre><strong># display the decision tree</strong><br>Image(graph.create_png())</pre><pre><strong>Output:-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/973/1*ivadD2ArGmYaRbz4VJO09g.png" /><figcaption>Figure 5, Full Grown Decision Tree.</figcaption></figure><pre>y_predict = DT.predict(xtrain)<br>print(dt_model.score(xtrain , ytrain))<br>print(dt_model.score(xtest , ytest))</pre><pre><strong>Output:-<br>1.0    # Training <br>0.6533333333333333 # Testing</strong></pre><p><strong>Here, we can see that the full-grown tree is prone to overfitting. Overfitting refers to the condition when the model completely fits the training data but fails to generalize the testing unseen data</strong>. Overfit condition arises when the model memorizes the noise of the training data and fails to capture important patterns.</p><p><strong><em>If you like this article, make sure to clap until your hand start paining :) Take care, stay healthy, stay safe and keep learning.</em></strong></p><p>Thanks.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=559557e9ae9d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[EDA(Exploratory Data Analysis) Practice on Health Insurance Data.]]></title>
            <link>https://towardsdev.com/eda-exploratory-data-analysis-practice-on-health-insurance-data-844782cc4943?source=rss-13b6cdff6950------2</link>
            <guid isPermaLink="false">https://medium.com/p/844782cc4943</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[greatlearning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <dc:creator><![CDATA[Umang Saluja]]></dc:creator>
            <pubDate>Thu, 17 Mar 2022 15:03:49 GMT</pubDate>
            <atom:updated>2022-03-17T15:03:49.880Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>Hi Folks! Before we get into the topic, I pray for your good health and well-being.</em></strong></p><h3>Let’s see the content we’re going to cover through this blog.</h3><ol><li><strong>Univariate Analysis.</strong></li><li><strong>Multivariate Analysis.</strong></li><li><strong>Handle Non-Numeric Data.</strong></li><li><strong>Normalization.</strong></li><li><strong>Transformation Techniques.</strong></li></ol><p>The dataset contains information about 1300 individuals regarding insurance charges. The data contains an individual’s basic information like age, sex, bmi, the region of residence and so on.</p><p><strong>Attribute Information:</strong></p><p><strong>age</strong>: Age of the individual; <strong>gender</strong>: Gender of the individual; <strong>bmi</strong>: Body mass index of the individual; <strong>bp</strong>: Systolic blood pressure in mm/hg; <strong>children</strong>: Number of children of the individual;<strong>smoker</strong>: Individual is smoker or not;<strong>region</strong>: Individual’s region of residence; <strong>medical_cost</strong>: Individual’s medical costs billed by the insurance company.</p><blockquote>To download the data set, please visit:-<a href="https://www.kaggle.com/umangsaluja/health-insurance-data">https://www.kaggle.com/umangsaluja/health-insurance-data</a></blockquote><blockquote>## I’m doing the EDA with the sample of questions that I have designed##, so lets dirty our hands on Python.</blockquote><p><strong>Import the required libraries</strong></p><pre>import numpy as np<br>import pandas as pd<br>import matplotlib.pyplot as plt<br>import seaborn as sns<br>import warnings<br>warnings.filterwarnings(&#39;ignore&#39;)</pre><h4>Load the health insurance dataset and check the first five observations</h4><pre><strong># reading the dataset</strong><br>df= pd.read_csv(&#39;health_insurance.csv&#39;)<br>df.head(5)</pre><pre><strong>Output-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/1*d0e1l-t-7jqfKv_OT_xnlA.png" /></figure><h3><strong>1. Univariate Analysis.</strong></h3><p>Let’s begin with some hands-on practice exercises</p><ol><li><strong><em>Are there any extreme observations present in ‘bmi’? Support your answer with a plot?</em></strong></li></ol><pre>sns.boxplot(df.bmi)<br>plot.show()</pre><pre><strong># yes, extreme values are present in the column bmi.</strong></pre><pre><strong>Output-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/421/1*1FVrDVN0WGMk3wKaKAg1bw.png" /></figure><p><strong><em>2. Is the variable ‘age’ skewed? Support your answer with a plot and metric?</em></strong></p><pre>sns.distplot(df.age)<br>print(df.age.skew())<br><strong># column age has 2 peaks, and data is slightly skewed.</strong></pre><pre><strong>Output-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/495/1*B0TG9T4MRLYeEfgIqbNGxA.png" /></figure><p><strong><em>3. Draw the line for mean and median on the distribution plot and identify the type of skewness for BM?</em></strong></p><pre>sns.distplot(df.bmi)<br>plt.axvline(df.bmi.mean(),color=&#39;r&#39;)<br>plt.axvline(df.bmi.median(),color=&#39;r&#39;)<br>plt.show()<br><strong># we can say that BMI is normally distributed since mean and median are almost equal.</strong></pre><pre><strong>Output-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/1*Y5vmpEE1_OT3Yx6cNmYSZw.png" /></figure><p><strong><em>4. Visualize the distribution of medical cost using violin plot and describe its distribution?</em></strong></p><pre>sns.violinplot(df.medical_cost)<br>plt.show()</pre><pre><strong>Output-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/423/1*59oWyP1wV0hmifSKm6BhJA.png" /></figure><h3><strong>2. Multivariate Analysis.</strong></h3><ol><li><strong><em>Is there any individual with age less than 30 and medical cost greater than 40k?</em></strong></li></ol><pre>df[(df.age&lt;30)&amp;(df.medical_cost&gt;40000)]</pre><pre><strong>Output</strong>-</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/1*vSW9apRM-vjd4yGQJN6zBQ.png" /></figure><p><strong><em>2. Does the scatter plot of BMI and medical cost reveal potential outliers?</em></strong></p><pre>sns.scatterplot(df.bmi,df.medical_cost,color=&#39;green&#39;)<br>plt.show()<br><strong># we can say that values above 50,000 our the outliers present.</strong></pre><pre><strong>Output-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/494/1*qFRxKpd-NK3IyzbmJXPAAA.png" /></figure><p><strong><em>3. Determine the percentage variation in BMI for males and females?</em></strong></p><pre>df.groupby([&#39;gender&#39;])[&#39;bmi&#39;].sum()</pre><pre><strong>Output-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/260/1*MwJVEUB_j0W-pwPPEU-2sA.png" /></figure><p><strong><em>4. Which region has the highest number of males?</em></strong></p><pre>df.groupby((&#39;region&#39;),[&#39;gender&#39;]==&#39;M&#39;).max()</pre><pre><strong>Output-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/536/1*EgpA0WcNbLCVAFtcf_ebEA.png" /></figure><p><em>5. </em><strong><em>Analyze the pairwise relationship between the numeric variables?</em></strong></p><pre>df_num= df.select_dtypes(np.number)<strong>#selecting the numeric columns</strong><br>sns.pairplot(df_num)</pre><pre><strong>Output-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*Pykeke_A2vRVHJ3STWaf1Q.png" /></figure><p><strong><em>6. Are the medical costs affected by an individual’s smoking habits?</em></strong></p><pre>df.groupby(&#39;smoker&#39;)[&#39;medical_cost&#39;].max()<br><strong># yes we can see that medical_cost of smokers is way too high</strong></pre><pre><strong>Output-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/254/1*Ny93CTZaskJmbAXvrzOFjw.png" /></figure><p><strong><em>7. Check if the region of residence and age of the individuals have an impact on the medical cost?</em></strong></p><pre>sns.scatterplot(df.age,df.medical_cost,hue=df.region)<br>plt.show()</pre><pre><strong>Output-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/367/1*bkbaAxd55z5ks75Yg0ezDQ.png" /></figure><h3>3. Handle Non-Numeric Data.</h3><ol><li><strong><em>Create a dummy variable for each category in ‘region’?</em></strong></li></ol><pre># import your code here<br>df_region=pd.get_dummies(df.region,drop_first=True)</pre><pre><strong>Output-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/286/1*wMGgj9wEgpwpFOwQFmzcIw.png" /></figure><p><strong><em>2. Create a variable ‘smoker_label’ containing unique label for each category in the variable ‘smoker’?</em></strong></p><pre>import scipy as stats<br>from sklearn.preprocessing import<br>LabelEncoder,StandardScaler,MinMaxScaler<br>le=LabelEncoder()<br>ss=StandardScaler()<br>ms=MinMaxScaler()</pre><p>After importing Library, Sklearn</p><pre>df[&#39;smoker&#39;]=le.fit_transform(df[&#39;smoker&#39;])<br>df[&#39;smoker&#39;] <br><strong># it converts smoker as 1 and non-smoker as 0</strong></pre><pre><strong>Output-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/358/1*Nyfr18tuvezX3mf-SYzbtA.png" /></figure><p><strong>3. Create a variable which will contain the frequency of an individual’s gender.</strong></p><pre>df[&#39;gender&#39;]= df[&#39;gender&#39;].replace(df[&#39;gender&#39;].value_counts(normalize=True))</pre><pre><strong>Output-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/374/1*KMj-pHaheEYB1J3g91nqaw.png" /></figure><h3>4. Normalization.</h3><ol><li><strong><em>Transform BMI such that it will have mean zero and variance one.</em></strong></li></ol><pre><strong># before we normalize our data, we need to make sure that all our columns are numeric in nature.</strong><br>df.drop(&#39;region&#39;,axis=1,inplace=True)</pre><pre>newdf= pd.concat([df,df_region],axis=1)<br><strong># as now we have to normalize the data, so we concatenate the columns on which feature engineering was performed.</strong></pre><p>Applying Standard Scaler to the entire dataset (<strong><em>scaling the dataset is needed for making data points generalized so that the distance between them will be lower.</em></strong></p><pre>from sklearn.preprocessing import StandardScaler<br>SS=StandardScaler()<br>scaled= SS.fit_transform(newdf)<br>scaled_df= pd.DataFrame(scaled,columns=newdf.columns)<br>scaled_df</pre><pre><strong>Output-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*P_ked1Ue7t2h8-SiBaXAPA.png" /></figure><h3>5. Transformation Techniques.</h3><ol><li><strong><em>Check the skewness for ‘medical_cost’. If the data is skewed, then use the appropriate technique to reduce the skewness in the data.</em></strong></li></ol><pre>print(&#39;Skewness-&#39;,scaled_df.medical_cost.skew())<br>sns.distplot(scaled_df.skew())<br>plt.show()<br><strong># we can see even after scaling the entire data, we have skewed values in our dataset<br># to remove this skewness we can use various transformation technique includes log transformation, sq rt and cube root transformation</strong></pre><pre><strong>Output-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/1*UvWugzzfr7ASIanAFPWCrA.png" /></figure><p>By seeing the above graph, we can notice that the above data is highly skewed, hence forth we will apply data transformation technique to reduce the scale of data</p><pre>scaled_df.skew()</pre><pre><strong>Output-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/222/1*AlkBaa0f7Uxykqzf8JrEwg.png" /></figure><pre>scaled_df[&#39;medical_cost&#39;]= np.cbrt(scaled_df[&#39;medical_cost&#39;])<br><strong># now check the skewness</strong></pre><pre>print(scaled_df.skew())<br>sns.distplot(scaled_df)<br><strong>#you can see that after transformation, skewness has reduced.</strong></pre><pre><strong>Output-</strong></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/454/1*x3rNYER-hWySATwpX4hawA.png" /></figure><blockquote>This was a brief introduction to Exploratory Data Analysis.Link for dataset-<a href="https://www.kaggle.com/umangsaluja/health-insurance-data">https://www.kaggle.com/umangsaluja/health-insurance-data</a></blockquote><p><strong><em>If you like this article, make sure to clap until your hand start paining :). Take care, stay healthy, stay safe and keep learning.</em></strong></p><p><strong><em>If you would like to share your thoughts, you can connect with me on </em></strong><a href="https://www.linkedin.com/in/umang-saluja-36713b171/"><strong><em>LinkedIn.</em></strong></a></p><p>Thanks.</p><blockquote>Please visit my portfolio- <a href="https://eportfolio.mygreatlearning.com/umang21">https://eportfolio.mygreatlearning.com/umang21</a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=844782cc4943" width="1" height="1" alt=""><hr><p><a href="https://towardsdev.com/eda-exploratory-data-analysis-practice-on-health-insurance-data-844782cc4943">EDA(Exploratory Data Analysis) Practice on Health Insurance Data.</a> was originally published in <a href="https://towardsdev.com/">Towards Dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[In-Depth Understanding of K-Means Clustering in Machine Learning.]]></title>
            <link>https://towardsdev.com/in-depth-understanding-of-k-means-clustering-in-machine-learning-c01ef7df9e5b?source=rss-13b6cdff6950------2</link>
            <guid isPermaLink="false">https://medium.com/p/c01ef7df9e5b</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Umang Saluja]]></dc:creator>
            <pubDate>Mon, 14 Mar 2022 16:23:15 GMT</pubDate>
            <atom:updated>2024-07-22T16:00:26.188Z</atom:updated>
            <content:encoded><![CDATA[<p><strong><em>Hi Folks! Before we get into the topic, I pray for your good health and well-being.</em></strong></p><h3><strong>Let’s see the content we&#39;re going to cover through this blog.</strong></h3><ol><li>What is Clustering?</li><li>Why Clustering is an Unsupervised Learning Problem?</li><li>What is K-Means Clustering?</li><li>How to choose the Right Number of Clusters in K-Means?</li><li>Implementing K-Means Clustering in Python.</li></ol><h3><strong>1. What is Clustering?</strong></h3><p>Primarily, clustering is an exploratory technique used to analyze hidden structure of the data. It is a way to decompose the dataset into a subset, with each subset representing as group of similar characteristics, these groups also known as <strong>C<em>lusters</em></strong>. Let&#39;s understand this with an example.</p><p>A bank wants to give loan offers to its customers. Currently, they look at the details of each customer and based on customer&#39;s details, they decided to gave them an offer.</p><p>You know, the bank can potentially have millions of customers. Does it make sense to look at the details of each customer separately and then make a decision? Hell no! The manual process and will take huge amount of time.</p><p>So what can the bank do? One option is to segment its customers into different Clusters. For instance, the bank can group the customers based on their income:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1008/1*blzZUmbZquolVo_5dtjEpg.png" /></figure><p>Now you can see bank can use 3 different strategies for their each group of customers. Thus, clustering is an Unsupervised Learning technique where we divide the dataset into groups and each group/cluster possess the same characteristics.</p><h3><strong>2. Why Clustering is an Unsupervised Learning Problem?</strong></h3><p>Now you know, clustering is a process of dividing a dataset into clusters, where each cluster represents unique characteristics.</p><p>It is important to understand the difference between <strong>supervised learning</strong> and <strong>unsupervised learning.</strong></p><p>In supervised learning, we always have our target variable(y) as dependent variable and (x) as independent variable, and we split our data using the independent variables in the supervision of the target variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/316/0*J08SwamRTyMDN_RN.png" /></figure><p>However, in unsupervised learning we do not have any <strong><em>target variable/Dependent variable, all we have left is Independent Variables.</em></strong></p><p><strong>So in clustering, we do not have a target to predict. We divide the data into groups/ cluster having similar characteristics. </strong>Hence, it is an unsupervised learning problem.</p><h3><strong>3. What is K-Means Clustering?</strong></h3><blockquote>It is a commonly used method, <strong>where K (specifies the number of clusters) and means(specify the average centroid of the cluster).</strong></blockquote><p>K-Means, each cluster is associated with a specific centroid due to the reason it is also known as centroid based algorithm or distance based algorithm. <em>The main objective of the K-Means algorithm is to minimize the sum of distances between the points and their respective cluster centroid.</em></p><p>The number of <strong>K</strong> can be easily analyzed using an Elbow Method that we cover further in this article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/531/1*PxOb4wCJdEJtqyjbfjKqkw.png" /></figure><p>Let’s go through the steps involved in K means clustering for a better understanding.</p><p><strong>Step1-</strong> Select the number of clusters for the dataset ( K ).</p><p><strong>Step2-</strong> Select K number of centroids</p><p><strong>Step3 -</strong>By calculating the Euclidean distance or Manhattan distance assign the points to the nearest centroid, thus creating K groups</p><p><strong>Step4-</strong> Now find the original centroid in each group</p><p><strong>Step5-</strong> Again reassign the whole data point based on this new centroid, then repeat S<strong>tep 4 </strong>until the position of the centroid doesn’t change.</p><h3><strong>Let’s understand the above steps with an example.</strong></h3><p>Consider the data of flowers petal length and petal width in millimeters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*D1vv0rYWsFzwnuW8PTq9iw.png" /></figure><blockquote><strong><em>Can we cluster the flower petals into distinct groups?</em></strong></blockquote><p><strong>Rather than getting to the python code, I would like you to understand the concept.</strong></p><p><strong>Step1- </strong>With the help of scatter plot, we plot the above given data, which visualize like this. From the plot, we can see that the dataset is divided into 3 groups/clusters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/501/0*_OnFMQW2-UBwWzhA" /></figure><p>Now, use K-means clustering to check whether the algorithm can form such three clusters (K = 3) from the given data. Consider the Euclidean distance as a distance measure.</p><blockquote><strong>I insist don’t worry about the code, we will do that in later part, just understand the steps that need to be carried out</strong></blockquote><p><strong>Step 2 -</strong>Initial assignment: Forgy method. Randomly choose 3 points as cluster centroids. Consider the below observations as initial centroids.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*MYgbNlb8NUgsw5fF" /></figure><p><strong>Step 3-</strong>Calculate the Euclidean distance of each data point from the cluster centroids.</p><p><strong>Step4</strong>- Assign the data to the nearest cluster. The plot shows that the 1st cluster contains 5 points, the 2nd cluster contains 2 points, and the 3rd cluster contains 7 points.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/0*oFMohLRMCoqCX93x" /></figure><p><strong>Step-5 </strong>In step 3, we have obtained 3 clusters based on the initial centroid assignment. Now calculate the means of these clusters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/713/1*yLna77tKXpXqRI9YrRreoQ.png" /></figure><p><strong>Repeat steps 3, 4 and 5, until the cluster centroids remains the same.</strong></p><p><strong>Inference-</strong>The <strong>first cluster </strong>contains 5 flowers with the least average petal length and width. This represents the group of small-sized flowers.</p><p>The <strong>second cluster</strong> represents 5 medium-sized flowers. <strong>The third cluster</strong> consists of 4 flowers with the highest average petal length and width.</p><p>Thus, K-means has clustered the data into 3 clusters based on the length and width of each flower petal.</p><p><strong>Summary- </strong>It<strong> </strong>Iterates these centroids until no change happens to the position of the centroid i.e K=3</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*FRy3b-_Tee8IDJOeO39ulQ.png" /></figure><h3><strong>4. How to choose the Right Number of Clusters in K-Means?</strong></h3><p>Before we dive into Python, here comes a concept of <strong><em>ELBOW METHOD, </em></strong>a commonly used method for finding <strong>optimal </strong>K value.</p><ul><li><strong>In the previous example</strong>, it was obvious that the observations were divided into 3 groups; thus we considered K = 3. But in general, it is not easy to decide the optimal value of K.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/505/0*j0Sqj_LrIrd5BBRI" /></figure><blockquote>The plot shows that the WCSS is decreasing rapidly for the K values less than the optimal K value.</blockquote><blockquote>After the elbow point, the WCSS is steadily decreasing, which implies that more clusters are formed by dividing the large clusters into subgroups.</blockquote><blockquote>Selecting the K greater than optimal K leads to overfitting.</blockquote><p><strong>WCSS</strong> -In python, the attribute<strong> ‘inertia_’</strong> returns the WCSS for a specific value of K. We use the different WCSS and K values to plot the elbow plot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yQciH0REtVYq92nX" /></figure><blockquote><strong>‘inertia_’ actually calculates the sum of distances of all the points within a cluster from the centroid of that cluster.</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/270/0*dH1-HesNn5XDQPk-.png" /></figure><h3><strong>5. Implementing K-Means Clustering in Python.</strong></h3><p>You can get the dataset from this <a href="https://github.com/Umangavin/umang.machine.learning/blob/main/Customer%20Unsupervised%20Learning.ipynb">link</a> with its solution at my GitHub.</p><p>About the data- A financial firm in New York tracks the overall spending behavior of all their clients. The firm has collected their data like age, gender, annual income and the spending score. The firm is interested in grouping clients with similar spending habits. It can be done using collected information regarding salary and spending score. The machine learning clustering algorithms can provide an analytical method to cluster clients with similar interests.</p><blockquote>I have done EDA and scaling of the Data, We scale the variables to get all the variables in the same range. With this, we can avoid a problem in which some features come to dominate solely because they tend to have larger values than others.</blockquote><p><strong>To check the whole script please do visit-</strong><a href="https://github.com/Umangavin/umang.machine.learning/blob/main/Customer%20Unsupervised%20Learning.ipynb">https://github.com/Umangavin/umang.machine.learning/blob/main/Customer%20Unsupervised%20Learning.ipynb</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MotX_5ss01s1GjullnHQgQ.png" /></figure><p>Let us plot the elbow plot and identify the elbow point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1013/1*v4YF3JbaBk14LArgkgJI8Q.png" /></figure><p><strong>Interpretation:</strong> We can see that the for K = 5, there is an elbow in the plot. Before this elbow point, the WCSS is decreasing rapidly and after K = 5, the WCSS is decreasing slowly.</p><p>Building the Clusters with optimum K value, i.e <strong>5</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/924/1*tyZsNjRiQDOfIUAhXv7Xbg.png" /></figure><p>Let us visualize the clusters by considering the variables ‘Cust_Spend_Score’ and ‘Yearly_Income’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1018/1*XABw6PCWc6sdOJS0TjZN9g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*Caib2z-xKECWA7UdH8obow.png" /></figure><p>This is all about the basic concept of the K-Means Clustering algorithm in Machine Learning. In the upcoming articles, we can learn more about different ML Algorithms.</p><p><strong><em>If you like this article, make sure to clap until your hand start paining. Take care, stay healthy, stay safe and keep learning.</em></strong></p><p><strong><em>If you would like to share your thoughts, you can connect with me on </em></strong><a href="https://www.linkedin.com/in/umang-saluja-36713b171/"><strong><em>LinkedIn.</em></strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c01ef7df9e5b" width="1" height="1" alt=""><hr><p><a href="https://towardsdev.com/in-depth-understanding-of-k-means-clustering-in-machine-learning-c01ef7df9e5b">In-Depth Understanding of K-Means Clustering in Machine Learning.</a> was originally published in <a href="https://towardsdev.com/">Towards Dev</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fundamental Script of Pandas Library]]></title>
            <link>https://medium.com/@umangsaluja/fundamental-script-of-pandas-library-30644bdeb52?source=rss-13b6cdff6950------2</link>
            <guid isPermaLink="false">https://medium.com/p/30644bdeb52</guid>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[pandas-dataframe]]></category>
            <dc:creator><![CDATA[Umang Saluja]]></dc:creator>
            <pubDate>Tue, 08 Feb 2022 14:14:41 GMT</pubDate>
            <atom:updated>2022-02-08T14:14:41.393Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/259/1*l-t9ZK7JoJtRnVejQU_ZHA.jpeg" /></figure><p>Pandas is the most crucial library for Data Science beginners. I used it practically every day while exploring the data, so I decided to write a brief summary in the form of Python Snippets.</p><p>In this article, I will share some main functionality of pandas library python snippets Moreover, you will be seeing examples related to that functionality.</p><p>Let’s get started!!</p><p>Working With Data Frames</p><p>Let’s begin with creating a basic dataframe with the help of dictionary:</p><pre>import numpy as n<br>import pandas as pd<br>df= pd.DataFrame({&#39;Name&#39;:[&#39;Jon&#39;,&#39;Linda&#39;,&#39;Gavin&#39;, &#39;Peter&#39;],<br>                  &#39;Marks&#39;:[45,65,75,80],<br>                 &#39;Subject&#39;:[&#39;English&#39;,&#39;Maths&#39;,&#39;Science&#39;,&#39;History&#39;]})<br>df</pre><p>After creating the dataframe, the keys from the dictionary will be the columns header and the corresponding will be the value of the dataframe.</p><p>To check the technical summary and data type of the columns in dataframe for that, the function is df.info():</p><pre>df.info() </pre><pre>#Output:-<br></pre><pre>0   Name     4 non-null      object<br> 1   Marks    4 non-null      int64 <br> 2   Subject  4 non-null      object<br>dtypes: int64(1), object(2)</pre><p>Reading and Writing Data by Pandas</p><p>Pandas provide read_csv() function which assist to read data stored in CSV format however, there are many other formats that help to read data are(csv, excel, sql ect).</p><pre>df=pd.read_csv(&#39;Student_result&#39;)<br>df</pre><p>If you want to see the top rows of your data that you have uploaded, you can do df.head()</p><pre>df.head(10)</pre><p>In the same fashion for the end rows of your data, you use df.tail()’</p><pre>df.tail(6)</pre><p>Filtering Rows in Data Frame</p><p>Filtering is mostly done with dataframes to get the desired subset from data. For instance, if we are working on the titanic dataset, and we want information of people survived with age greater than 25.</p><pre>df= pd.read_csv(&#39;titanic.csv&#39;)<br>df</pre><p>After uploading the titanic dataset, we apply the filter condition for the above criteria, here two conditions are given that is combined with operator (&amp;)</p><pre>df[(df[&#39;Age&#39;] &gt;25) &amp; (df[&#39;Survived&#39;]==1)]</pre><p>To check for the Null Values in the column, we can use isnull() function.</p><pre>df[&#39;Age&#39;].isnull().sum()</pre><pre>#output:-</pre><pre>177</pre><p>The sum() function will count the total null values, as shown above. For more on conditional expressions and filtering, see this section of the Pandas Documentation.</p><p>Selecting Specific Rows and Column.</p><p>There are 2 functions, loc() and iloc() widely used in pandas for an instance if we need the name of passengers whose age is greater than 50 in titanic dataset than:</p><pre>df.loc[df.Age&gt;50, &#39;Name&#39;]</pre><pre>#Output:-</pre><pre>6                                McCarthy, Mr. Timothy <br>11                              Bonnell, Miss. Elizabeth<br>15                      Hewlett, Mrs. (Mary D Kingcome) <br>33                                 Wheadon, Mr. Edward H<br>54                        Ostby, Mr. Engelhart Cornelius<br>                             ...                        <br>820    Hays, Mrs. Charles Melville (Clara Jennings Gr...<br>829            Stone, Mrs. George Nelson (Martha Evelyn)<br>851                                  Svensson, Mr. Johan<br>857                               Daly, Mr. Peter Denis <br>879        Potter, Mrs. Thomas Jr (Lily Alexenia Wilson)<br>Name: Name, Length: 64, dtype: object</pre><p>As mentioned in the documentation of pandas:</p><p>When using loc/iloc, the part before the comma is the rows you want, and the part after the comma is the columns you want to select.</p><p>Creating a New Column with the Existing Column.</p><p>Creating a new column in dataframe using pandas is like a piece of cake, just use [] with the new column name, and you are all set to go.</p><pre>df[&#39;New_Age&#39;]=df[&#39;Age&#39;].replace(np.nan,0)</pre><p>However, what if we want to create a column with a non-existing column in data? For instance, say we have a titanic dataset, and we want to add a Date column to it.</p><pre>df[&quot;Date&quot;] = pd.date_range(&quot;1965-01-01&quot;, periods=len(df)</pre><pre>df</pre><p>If you wanted to apply a more complex function to a column and have the result saved as a new column of the dataframe, you could use the .apply() method.</p><p>To rename the column we use .rename() function, for instance, say want to rename sex column in titanic as gender then.</p><pre>df.rename(columns={&#39;Sex&#39;:&#39;Gender&#39;})</pre><h3>Calculating Summary Statistics</h3><p>Describe method df.describe() provides an overview of numerical columns in the dataframe and is very handy to speed up the things.</p><pre>df.describe()</pre><pre>#Output:-</pre><pre>PassengerId	Survived	Pclass	Age	SibSp	Parch	Fare	New_Agcount	891.000000	891.000000	891.000000	714.000000	891.000000	891.000000	891.000000	891.000000mean	446.000000	0.383838	2.308642	29.699118	0.523008	0.381594	32.204208	23.799293std	257.353842	0.486592	0.836071	14.526497	1.102743	0.806057	49.693429	17.596074min	1.000000	0.000000	1.000000	0.420000	0.000000	0.000000	0.000000	0.00000025%	223.500000	0.000000	2.000000	20.125000	0.000000	0.000000	7.910400	6.00000050%	446.000000	0.000000	3.000000	28.000000	0.000000	0.000000	14.454200	24.00000075%	668.500000	1.000000	3.000000	38.000000	1.000000	0.000000	31.000000	35.000000max	891.000000	1.000000	3.000000	80.000000	8.000000	6.000000	512.329200	80.000000e</pre><p>String Method</p><p>str() method work with strings that help in manipulation of strings given in the columns, these two cases str.lower() and str.upper() are widely used to change string to upper and lower case.</p><pre>df[&#39;Name&#39;].str.lower()</pre><p>We can even check if string column contain certain strings with the help of str. contains(). For instance, say in titanic data set we want to check names that have master in it.</p><pre>df[df[&#39;Name&#39;].str.contains(&#39;Master&#39;)] [&#39;Name&#39;]</pre><p>You can learn more about text manipulation in Pandas.</p><p>To recapitulate, pandas is the most widely used library in the context of Data Science and Machine Learning, with its spectacular functionality it is extensively used as it increases the speedy workflow with respect to data analysis.</p><p>References</p><p>· <a href="https://pandas.pydata.org/pandas-docs/stable/getting_started/intro_tutorials/01_table_oriented.html#min-tut-01-tableoriented">pandas documentation</a></p><p>· <a href="https://pandas.pydata.org/pandas-docs/stable/user_guide/index.html#user-guide">pandas user guide</a></p><p>· <a href="https://excalidraw.com/">excalidraw</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=30644bdeb52" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>