<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by ITBodhi on Medium]]></title>
        <description><![CDATA[Stories by ITBodhi on Medium]]></description>
        <link>https://medium.com/@itbodhi?source=rss-e420879ba37------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*UB2QWzjEmj-_aDl9</url>
            <title>Stories by ITBodhi on Medium</title>
            <link>https://medium.com/@itbodhi?source=rss-e420879ba37------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 05:41:02 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@itbodhi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Handling Imbalanced data sets in Machine Learning]]></title>
            <link>https://medium.com/@itbodhi/handling-imbalanced-data-sets-in-machine-learning-5e5f33c70163?source=rss-e420879ba37------2</link>
            <guid isPermaLink="false">https://medium.com/p/5e5f33c70163</guid>
            <category><![CDATA[machine-learning-course]]></category>
            <category><![CDATA[handling-imbalanced-data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[imbalanced-data-set]]></category>
            <dc:creator><![CDATA[ITBodhi]]></dc:creator>
            <pubDate>Tue, 21 Jul 2020 17:14:34 GMT</pubDate>
            <atom:updated>2020-07-21T17:14:34.890Z</atom:updated>
            <content:encoded><![CDATA[<p>What are the Best Practices, Techniques and Tools to make Right Model with Imbalanced data set?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IulFfGY3Qq_D1LsR" /><figcaption>Photo by <a href="https://unsplash.com/@coltonsturgeon?utm_source=medium&amp;utm_medium=referral">Colton Sturgeon</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Let’s Start</h4><p>Suppose that you have to build a classification model on the given Covid-19 data set.You have applied your favourite algorithm and achieved 94% accuracy in predicting Covid Positive or Covid Negative.But still your boss is not happy and threw your model into the trash. You are surprised that model is working fine without any visible defect and with higher accuracy.</p><p>After close analysis you found that Covid19 infection rate is only 6–7% and that is why in given data set Positive class is only 7% and Negative class is around 93%. That is highly Imbalanced data set and my model is behaving in a very interesting way and predicting every case as Negative class and giving accuracy as high as 94% but not predicting any positive class correctly.That is actually a blunder not predicting Covid19 patient as positive while it is.</p><h4>Why my Model is behaving insanely?</h4><p>No, model is actually doing the right thing but my way of training the model is wrong. I am focusing on the wrong thing !</p><h3>What am I doing wrong?</h3><p>My Approach is biased with my knowledge of evaluation matrices which are actually befooling me.ACCURACY is not the right matrix when working for the Imbalanced data set.</p><h4>Let’s refresh the memory:Confusion matrix, Precision, Recall and F1</h4><p>Confusion matrix gives an interesting overview of how well a model is doing. Thus, it is a great starting point for any classification model evaluation. We summarise most of the metrics that can be derived from the confusion matrix in the following graphic:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*_mO22Txi4K3KvTTg.png" /></figure><p>Let us give a short description of these metrics.</p><ul><li><strong><em>The accuracy </em></strong><em>of the model is basically the total number of correct predictions divided by total number of predictions.</em></li><li><strong>The precision</strong> of a class define how trustable is the result when the model answer that a point belongs to that class.</li><li><strong>The recall </strong>of a class expresses how well the model is able to detect that class.</li><li><strong><em>The F1 score </em></strong><em>of a class is given by the harmonic mean of precision and recall (2×precision×recall / (precision + recall)), it combines precision and recall of a class in one metric.</em></li></ul><p>For a given class, the different combinations of recall and precision have the following meanings :</p><ul><li>high recall + high precision : the class is perfectly handled by the model</li><li>low recall + high precision : the model can’t detect the class well but is highly trustable when it does</li><li>high recall + low precision : the class is well detected but the model also include points of other classes in it</li><li>low recall + low precision : the class is poorly handled by the model</li></ul><p>In our introductory example, we have the following confusion matrix for 10000 products.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NO15SfKSns_0_s7O.png" /></figure><p>The accuracy is 96.2% as said earlier. The non defective class precision is 96.2% and the defective class precision is not computable. The recall of the non defective class is 1.0 which is perfect (all the non defective products have been labelled as such). But the recall of the defective class is 0.0 which is the worse case (no defective products were detected). Thus, we can conclude our model is not doing well <strong>for this class</strong>. The F1 score is not computable for the defective products and is 0.981 for the non defective products. In this example, looking at the confusion matrix could have led to re-think our model or our goal (as we will see in the following sections). It could have prevented using a useless model.</p><h3>Challenges with Imbalanced data Set?</h3><p>The conventional model evaluation methods do not accurately measure model performance when faced with imbalanced datasets.</p><p>Standard classifier algorithms like Decision Tree and Logistic Regression have a bias towards classes which have number of instances. They tend to only predict the majority class data. The features of the minority class are treated as noise and are often ignored. Thus, there is a high probability of misclassification of the minority class as compared to the majority class.</p><p>Imbalanced datasets can be found for different use cases in various domains:</p><ul><li>Finance: Fraud detection datasets commonly have a fraud rate of ~1–2%</li><li>Ad Serving: Click prediction datasets also don’t have a high clickthrough rate.</li><li>Transportation/Airline: Will Airplane failure occur?</li><li>Medical: Does a patient has cancer?</li><li>Content moderation: Does a post contain NSFW content?</li></ul><h3>Handling Imbalanced Data: Best Practices and Approaches</h3><h4><strong>1. Collect More Data:</strong></h4><p>A larger dataset might expose a different and perhaps more balanced perspective on the classes.</p><h4><strong>2. Try Changing Your Performance Metric:</strong></h4><p>Accuracy is not the metric to use when working with an imbalanced dataset. We have seen that it is misleading.</p><p>Looking at the following performance measures that can give more insight into the accuracy of the model than traditional classification accuracy:</p><ul><li><strong>Confusion Matrix:</strong> A breakdown of predictions into a table showing correct predictions (the diagonal) and the types of incorrect predictions made (what classes incorrect predictions were assigned).</li><li><strong>Precision:</strong> A measure of a classifiers exactness.</li><li><strong>Recall</strong>: A measure of a classifiers completeness</li><li><strong>F1 Score (or F-score):</strong> A weighted average of precision and recall.</li><li><strong>Kappa (or </strong><a href="https://en.wikipedia.org/wiki/Cohen%27s_kappa"><strong>Cohen’s kappa</strong></a><strong>):</strong> Classification accuracy normalized by the imbalance of the classes in the data.</li><li><a href="http://sklearn.calibration.calibratedclassifiercv/"><strong><em>Adjust the decision threshold</em></strong></a></li><li><strong>Adjusting misclassification costs</strong></li></ul><h4><strong>3. Cost-sensitive classifiers</strong></h4><p>May be used for unbalanced data sets by setting a high cost to the misclassifications of a minority class example.</p><h4>4. Boosting Algorithm</h4><p><strong><em>AdaCost, WEKA, AdaBoost, Gradient Boost, XGBoost</em>:</strong>xgboost offers parameters to balance positive and negative weights using <em>scale_pos_weight</em>(<a href="https://stats.stackexchange.com/questions/171043/how-to-tune-hyperparameters-of-xgboost-trees">https://stats.stackexchange.com/questions/171043/how-to-tune-hyperparameters-of-xgboost-trees</a>)</p><h4><strong>5. Weighting of examples</strong></h4><p>It involves the creation of specific weight vectors in order to improve minority class predictions</p><p>The <strong>class-specific weights(</strong>class_weight parameter)<strong> </strong>are calculated per class whereas the <strong>test-case-specific weights </strong>are calculated for each single instance.<a href="https://scikit-learn.org/stable/auto_examples/svm/plot_separating_hyperplane_unbalanced.html">https://scikit-learn.org/stable/auto_examples/svm/plot_separating_hyperplane_unbalanced.html</a></p><h4><strong>6. Try Different Algorithms</strong></h4><p>Run a lot of tests on multiple models. Intuition can take you a long way in data-science — if your gut tells you that an ensemble of classifiers will give you the best results, go ahead and try it.</p><h4><strong>7. Use Stratified CV</strong></h4><h4><strong>8. Penalized SVM</strong></h4><p>In SVM where it is desired to give more importance to certain classes or certain individual samples, the parameters <strong><em>class_weight and sample_weight</em></strong> can be used.</p><h4><strong>9. Bagging may give interesting results.</strong></h4><h4><strong>10. Data Level approach</strong></h4><p>Let’s apply some of resampling techniques using the Python library <a href="http://contrib.scikit-learn.org/imbalanced-learn/stable/"><strong><em>imbalanced-learn</em></strong></a><strong><em>. </em></strong>It is compatible with scikit-learn and is part of scikit-learn-contrib projects.</p><p>For visualization, let’s create a small unbalanced sample dataset using the make_classificationmethod:</p><pre>import imblearn</pre><pre>from sklearn.datasets import make_classification<br><br>X, y = make_classification(<br>    n_classes=2, class_sep=1.5, weights=[0.9, 0.1],<br>    n_informative=3, n_redundant=1, flip_y=0,<br>    n_features=20, n_clusters_per_class=1,<br>    n_samples=100, random_state=10<br>)<br><br>df = pd.DataFrame(X)<br>df[&#39;target&#39;] = y<br>df.target.value_counts().plot(kind=&#39;bar&#39;, title=&#39;Count (target)&#39;);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BF1vdw9326YdiLdPFhTKSQ.png" /></figure><p>We will also create a 2-dimensional plot function, plot_2d_space, to see the data distribution using PCA (Because the dataset has many dimensions and our graphs will be 2D, we will reduce the size of the dataset using Principal Component Analysis (PCA)</p><pre>def plot_2d_space(X, y, label=&#39;Classes&#39;):   <br>    colors = [&#39;#1F77B4&#39;, &#39;#FF7F0E&#39;]<br>    markers = [&#39;o&#39;, &#39;s&#39;]<br>    for l, c, m <strong>in</strong> zip(np.unique(y), colors, markers):<br>        plt.scatter(<br>            X[y==l, 0],<br>            X[y==l, 1],<br>            c=c, label=l, marker=m<br>        )<br>    plt.title(label)<br>    plt.legend(loc=&#39;upper right&#39;)<br>    plt.show()</pre><pre>from sklearn.decomposition import PCA<br><br>pca = PCA(n_components=2)<br>X = pca.fit_transform(X)<br><br>plot_2d_space(X, y, &#39;Imbalanced dataset (2 PCA components)&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iKMqAiQpkX17GXNmP1hCsw.png" /></figure><h4><strong>Resampling</strong></h4><p>You can change the dataset that you use to build your predictive model to have more balanced data.This change is called sampling your dataset and there are two main methods that you can use to even-up the classes:</p><ul><li><strong>Over-sampling: </strong>You can add copies of instances from the under-represented class called over-sampling (or more formally sampling with replacement), or</li><li><strong>Under-sampling:</strong> You can delete instances from the over-represented class, called under-sampling.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/725/1*H6XodlitlGDl9YdbwaZLMw.png" /><figcaption>Sampling for imbalanced data set</figcaption></figure><h4>A. Under-sampling</h4><p>It<strong> </strong>aims to balance class distribution by randomly eliminating majority class examples. This is done until the majority and minority class instances are balanced out.</p><blockquote><strong><em>Advantages</em></strong></blockquote><ul><li>It can help improve run time and storage problems by reducing the number of training data samples when the training data set is huge.</li></ul><blockquote><strong><em>Disadvantages</em></strong></blockquote><ul><li>It can discard potentially useful information which could be important for building rule classifiers.</li><li>The sample chosen by random under sampling may be a biased sample. And it will not be an accurate representative of the population. Thereby, resulting in inaccurate results with the actual test data set.</li></ul><pre>from imblearn.under_sampling import RandomUnderSampler<br><br>rus = RandomUnderSampler()<br>X_rus, y_rus, id_rus = rus.fit_sample(X, y)<br><br><br>plot_2d_space(X_rus, y_rus, &#39;Random under-sampling&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*le_GDgYUfiUYLaCmbT0lzA.png" /><figcaption>Random Under Sampling</figcaption></figure><h4>B. Random over-sampling</h4><p>It increases the number of instances in the minority class by randomly replicating them in order to present a higher representation of the minority class in the sample.</p><p><strong><em>Advantages</em></strong></p><ul><li>Unlike under sampling this method leads to no information loss.</li><li>Outperforms under sampling</li></ul><p><strong><em>Disadvantages</em></strong></p><ul><li>It increases the likelihood of overfitting since it replicates the minority class events.</li><li>It increases the size of the training set and the time to build a classifier.</li><li>Furthermore, in case of decision tree learning, the decision region for the minority class becomes very specific through the replication of the minority class and this causes new splits in the decision tree, which can lead to overfitting.</li></ul><pre>from imblearn.over_sampling import RandomOverSampler<br><br>ros = RandomOverSampler()<br>X_ros, y_ros = ros.fit_sample(X, y)<br><br>print(X_ros.shape[0] - X.shape[0], &#39;new random picked points&#39;)<br><br>plot_2d_space(X_ros, y_ros, &#39;Random over-sampling&#39;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OT1JUU0koZ9wpQ8mPzCCiQ.png" /><figcaption>Random Over Sampling</figcaption></figure><h4>C. Under-sampling: Tomek links</h4><p>Tomek links are pairs of very close instances, but of opposite classes. Removing the instances of the majority class of each pair increases the space between the two classes, facilitating the better classification.</p><p>In this algorithm, we end up removing the majority element from the Tomek link, which provides a better decision boundary for a classifier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*KxFmI15rxhvKRVl-febp-Q.png" /></figure><pre>from imblearn.under_sampling import TomekLinks</pre><pre>tl = TomekLinks(sampling_strategy=&#39;majority&#39;)<br>X_tl, y_tl= tl.fit_sample(X, y)</pre><pre>plot_2d_space(X_tl, y_tl, &#39;Tomek links under-sampling&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G_lY4BrF0y27_Ir4SZAnkQ.png" /><figcaption>TOMEK Links Under Sampling</figcaption></figure><p><strong>D. Under-sampling: Cluster Centroids</strong></p><p>This technique performs under-sampling by generating centroids based on clustering methods. The data will be previously grouped by similarity, in order to preserve information.</p><p>Makes use of <strong>K-means</strong> to reduce the number of samples</p><pre>from imblearn.under_sampling import ClusterCentroids<br><br>cc = ClusterCentroids()<br>X_cc, y_cc = cc.fit_sample(X, y)<br><br>plot_2d_space(X_cc, y_cc, &#39;Cluster Centroids under-sampling&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hKIQDoDMJCJYrzIDSSw5AQ.png" /><figcaption>KMeans Under Sampling</figcaption></figure><h4>E. Generate Synthetic Samples</h4><p>A simple way to generate synthetic samples is to randomly sample the attributes from instances in the minority class.</p><h4><strong>E.1 Over-sampling: SMOTE</strong></h4><p>SMOTE <strong>(Synthetic Minority Oversampling Technique) </strong>consists of synthesizing elements for the minority class, based on those that already exist. It works randomly picking a point from the minority class and computing the k-nearest neighbours for this point. The synthetic points are added between the chosen point and its neighbours.</p><blockquote><strong><em>Advantages</em></strong></blockquote><ul><li>Mitigates the problem of overfitting caused by random oversampling as synthetic examples are generated rather than replication of instances</li><li>No loss of useful information</li></ul><blockquote><strong><em>Disadvantages</em></strong></blockquote><ul><li>While generating synthetic examples SMOTE does not take into consideration neighbouring examples from other classes. This can result in increase in overlapping of classes and can introduce additional noise</li><li>SMOTE is not very effective for high dimensional data</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/734/1*yRumRhn89acByodBz0H7oA.png" /><figcaption>SMOTE Over Sampling</figcaption></figure><pre>from imblearn.over_sampling import SMOTE</pre><pre>X_resampled, y_resampled = SMOTE().fit_resample(X, y)</pre><pre>plot_2d_space(X_resampled, y_resampled, &#39;SMOTE over-sampling&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5ZaEkNlnOERGpVFjGoOpKQ.png" /></figure><p><strong>E.2 Adaptive Synthetic Sampling (Borderline-SMOTE and ADA-SYN)</strong></p><p>The essential idea of ADASYN is to use a weighted distribution for different minority class examples according to their level of difficulty in learning, where more synthetic data is generated for minority class examples that are harder to learn compared to those minority examples that are easier to learn.</p><p>As a result, the ADASYN approach improves learning with respect to the data distributions in two ways: (1) reducing the bias introduced by the class imbalance, and (2) adaptively shifting the classification decision boundary toward the difficult examples.</p><pre>from imblearn.over_sampling import ADASYN</pre><pre>X_resampled, y_resampled = ADASYN().fit_resample(X, y)</pre><pre>plot_2d_space(X_resampled, y_resampled, &#39;ADASYN over-sampling&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p-2FJnwx8O8-QZLOIgbcog.png" /><figcaption>ADASYN Over sampling</figcaption></figure><h4>F. Over-sampling followed by under-sampling</h4><p>We previously presented SMOTE and showed that this method can generate noisy samples by interpolating new points between marginal outliers and inliers. This issue can be solved by cleaning the space resulting from over-sampling.</p><p>In this regard, Tomek’s link is one of the cleaning method that has been a problem solver.</p><pre>from imblearn.combine import SMOTETomek</pre><pre>smt = SMOTETomek()<br>X_smt, y_smt = smt.fit_sample(X, y)</pre><pre>plot_2d_space(X_smt, y_smt, &#39;SMOTE + Tomek links&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MFMJVAeWbCW9CuZyqUmSWA.png" /></figure><h3>Important Points to Note</h3><ul><li>Both SMOTE and ADASYN use the KNN algorithm to generate new samples</li><li>The other SMOTE variants and ADASYN differ from each other by selecting the sample ahead of generating the new samples.</li><li><a href="https://imbalanced-learn.org/stable/generated/imblearn.over_sampling.SVMSMOTE.html">SVMSMOTE</a> — uses an SVM classifier to find support vectors and generate samples considering them. Note that the Cparameter of the SVM classifier allows to select more or less support vectors.</li><li><a href="https://imbalanced-learn.org/stable/generated/imblearn.over_sampling.KMeansSMOTE.html">KMeansSMOTE</a> — uses a KMeans clustering method before to apply SMOTE. The clustering will group samples together and generate new samples depending of the cluster density.</li><li>All algorithms can be used with multiple classes as well as binary classes classification</li><li><strong><em>When dealing with mixed data type such as continuous and categorical features, none of the presented methods (apart of the class </em></strong><a href="https://imbalanced-learn.org/stable/generated/imblearn.over_sampling.RandomOverSampler.html"><strong><em>RandomOverSampler</em></strong></a><strong><em>) can deal with the categorical features. The </em></strong><a href="https://imbalanced-learn.org/stable/generated/imblearn.over_sampling.SMOTENC.html"><strong><em>SMOTENC</em></strong></a><strong><em> is an extension of the </em></strong><a href="https://imbalanced-learn.org/stable/generated/imblearn.over_sampling.SMOTE.html"><strong><em>SMOTE</em></strong></a><strong><em> algorithm for which categorical data are treated differently.</em></strong></li></ul><p><strong>11. Integration of sampling and boosting-SMOTEBoost, RUSBoost</strong></p><p><a href="https://github.com/dialnd/imbalanced-algorithms/blob/master/smote.py"><strong>SMOTEBoost</strong></a> is an oversampling method based on the SMOTE algorithm (Synthetic Minority Oversampling Technique). SMOTE uses k-nearest neighbours to create synthetic examples of the minority class. SMOTEBoostthen injects the SMOTE method at each boosting iteration. The advantage of this approach is that while standard boosting gives equal weights to all misclassified data, SMOTE gives more examples of the minority class at each boosting step.</p><p><a href="https://github.com/dialnd/imbalanced-algorithms/blob/master/rus.py"><strong>RUSBoost </strong></a>achieves the same goal by performing random undersampling (RUS) at each boosting iteration instead of SMOTE.</p><p><strong>12. One interesting approach to solving the imbalance problem is to discard the minority examples and treat it as a single-class (or anomaly detection) problem. </strong>Isolation Forests that attempted to identify anomalies in data by learning random forests and then measuring the average number of decision splits required to isolate each particular data point. The resulting number can be used to calculate each data point’s anomaly score, which can also be interpreted as the likelihood that the example belongs to the minority class. Indeed, the authors tested their system using highly imbalanced data and reported very good results. Nearest Neighbour Ensembles as a similar idea that was able to overcome several shortcomings of Isolation Forests.</p><p>I Hope that this article will help you to understand and use best practices to handle imbalanced data sets.</p><p><strong><em>Yet nothing is comparable to ‘Experience and Learn’. Try different things and you never know which will actually work for you.</em></strong></p><p>Happy Reading and Keep Learning….</p><h3>Recommended reading</h3><p><a href="https://towardsdatascience.com/handling-imbalanced-datasets-in-machine-learning-7a0e84220f28">Handling imbalanced datasets in machine learning</a></p><p>The imbalanced-learn documentation:</p><p><a href="http://contrib.scikit-learn.org/imbalanced-learn/stable/index.html">Welcome to imbalanced-learn documentation!</a></p><p>The imbalanced-learn GitHub:</p><p><a href="https://github.com/scikit-learn-contrib/imbalanced-learn">GitHub - scikit-learn-contrib/imbalanced-learn: A Python Package to Tackle the Curse of Imbalanced Datasets in Machine Learning</a></p><p>Examples:</p><p><a href="https://imbalanced-learn.org/stable/auto_examples/index.html">General examples - imbalanced-learn 0.7.0 documentation</a></p><p>Research Papers:</p><p><a href="https://www.researchgate.net/publication/220543125_SMOTE_Synthetic_Minority_Over-sampling_Technique">(PDF) SMOTE: Synthetic Minority Over-sampling Technique</a></p><p><a href="https://www.researchgate.net/publication/224330873_ADASYN_Adaptive_Synthetic_Sampling_Approach_for_Imbalanced_Learning">https://www.researchgate.net/publication/224330873_ADASYN_Adaptive_Synthetic_Sampling_Approach_for_Imbalanced_Learning</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e5f33c70163" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Overfitting and Underfitting in Machine Learning Models]]></title>
            <link>https://medium.com/@itbodhi/overfitting-and-underfitting-in-machine-learning-models-76cb60dbdaf6?source=rss-e420879ba37------2</link>
            <guid isPermaLink="false">https://medium.com/p/76cb60dbdaf6</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[machine-learning-course]]></category>
            <category><![CDATA[bias-and-variance]]></category>
            <category><![CDATA[bias-variance-tradeoff]]></category>
            <category><![CDATA[overfitting]]></category>
            <dc:creator><![CDATA[ITBodhi]]></dc:creator>
            <pubDate>Thu, 02 Jul 2020 11:41:37 GMT</pubDate>
            <atom:updated>2020-07-02T12:40:17.752Z</atom:updated>
            <content:encoded><![CDATA[<h3>Overfitting and Underfitting</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*OCUH6dKUG-IGUfqqxOn-0A.png" /></figure><p>In Machine Leaning, model performance is evaluated on the basis of two important parameters. <strong><em>Accuracy and Generalisation</em></strong>. Accuracy means how well model predicts the right target value and Generalisation means how well model behaves on seen and unseen data.</p><p>Machine learning models are trained on given training data and performance is evaluated on the unseen test data. <strong>Model is considered right when it behaves nearly same way on training and test data with highest accuracy.</strong></p><h3>What is Underfitting?</h3><blockquote>Underfitting means model has low accuracy score on training data and test data both.</blockquote><p>Underfitting happens when algorithm used to build prediction model is very simple and not able to learn complex pattern from the training data. In that case accuracy will be on lower side on seen training data as well as unseen test data.</p><p><strong><em>Underfitting also referred as High Bias</em></strong></p><blockquote><a href="https://medium.com/@itbodhi/bias-and-variance-trade-off-542b57ac7ff4">Check Bias and Variance Trade off</a></blockquote><p><strong>Generally,</strong> It happens with Linear Algorithms. A underfit model makes incorrect assumptions about the dataset to make the target function easier to learn. If training data distribution is non linear and you apply linear algorithm to build the prediction model, in that case model would not be able to learn non linear relationship between target value and predictors( features) and accuracy will suffer.</p><p><strong><em>Underfit Model is very simple</em></strong> because of the assumptions made about the data( which may or may not be true) <strong><em>pays very little attention to the training data and oversimplifies the model. High Bias model always leads to high error on training as well as on test data.</em></strong></p><blockquote>In a simple language, your model is very simple and not able to learn the complex pattern (or relationship in data) from the data to make right prediction.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M3Sq4zOG8nB7tPhPesbQuQ.jpeg" /><figcaption>Picture1 — Regression Example for Overfitting and Underfitting</figcaption></figure><p><strong>Regression</strong> example <strong><em>Picture1</em></strong> — Regression Example for Overfitting and Underfitting, first Image represents model is Underfit. Data given for training is non linear and you are applying linear regression on the data for modelling. Model has high error because maximum of the data points are far from the line so not able to capture the existing pattern from the data.In this case model built is not right and will have low prediction accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*67HOjsCZVzE6FFM7TuSBmw.jpeg" /><figcaption>Picture2 — Classification Example for Overfitting and Underfitting</figcaption></figure><p><strong>Classification</strong> example <strong><em>Picture2</em></strong> — Regression Example for Overfitting and Underfitting, first Image represents model is Underfit. Data given for training is non linear and you are trying linear model for classification. When data distribution is non linear than linear model ( in above example a Line) can not work as decision boundry to classify data in to different classes. Model has high error because many of the data points classified incorrectly so will have low prediction accuracy.</p><h3>What is Overfitting?</h3><blockquote>Overfitting means model has High accuracy score on training data but low score on test data.</blockquote><p><strong>Overfitting means your model is not Generalised.</strong></p><p><strong><em>Overfitting happens when algorithm used to build prediction model is very complex and it has over learned the underlying patterns in training data</em></strong>.Overfitting is error from sensitivity to small fluctuations in the training set. Overfitting can cause an algorithm to model the random noise in the training data, rather than the intended result.</p><p><strong><em>Underfitting also referred as High Variance</em></strong></p><blockquote><a href="https://medium.com/@itbodhi/bias-and-variance-trade-off-542b57ac7ff4">Check Bias and Variance Trade off</a></blockquote><p>In supervised learning, <strong>overfitting</strong> happens when algorithms(Non Linear Algorithms) are strongly influenced by the specifics of the training data and try to learn patterns which are noisy and not generalized and only limited to training data set.</p><blockquote>In a simple language, your model is very complex and strongly influenced by the training data and will result in high score on training data but low score on unseen test data.</blockquote><p><strong>Regression</strong> example <strong><em>Picture-1</em></strong>, third Image represents model is Overfit. Data given for training is non linear and you are using higher degree features for modelling. As you can observe liner model ( line in above case) has been curved too much to go through each and every data points. In this case there will be minimum error on training data because model has over learned the patterns. But on test data when data distribution will differ from the training data, prediction error will increase because line is curved to suit the training data only and not generalised to fit the unseen data.</p><p><strong>Classification</strong> example <strong><em>Picture-2</em></strong>, Third Image represents model is Overfit. Data given for training is non linear and you are trying linear model for classification. As you can observe, because of higher degree features, decision boundary has curved too much to classify each and every training example correctly. In this case there will be minimum error on training data because model has over learned the patterns and resulting in to zero classification error. But on test data when data distribution will differ from the training data, prediction error will increase because line is curved to suit the training data only and not generalised to classify the unseen data.</p><h3>What is Right Fit?</h3><p>Model is considered to be Right Fit when it is generalised and behaves more or less in the same way as on training data and on test data.</p><p><strong>Model accuracy should be almost same on training and test data.</strong></p><p>Right fit model is neither Underfit and nor Overfit, it is a generalised model that does not change for seen and unseen data. Middle image in Picture1–2 is right fit model.</p><blockquote><strong><em>Practically, Overfitting and Underfitting are inverse to each other and you have to take trade of between Overfit and Underfit (</em></strong><a href="https://medium.com/@itbodhi/bias-and-variance-trade-off-542b57ac7ff4"><strong><em>Bias &amp; variance</em></strong></a><strong><em>) to get the right model.</em></strong></blockquote><p>Generally, You can see a general trend in the examples above:</p><ul><li><strong>Linear</strong> machine learning algorithms often are <strong>Underfit</strong>. <em>Example:Linear Regression, Logistic Regression</em></li><li><strong>Nonlinear</strong> machine learning algorithms often are <strong>Overfit</strong>. <em>Example: Decision Tree, SVM, Neural Networks</em></li></ul><h3>How to find the Right Balance?</h3><p><strong>Lowering high Bias or Underfitting:</strong></p><ol><li>Use non Parameterised Algorithms</li></ol><p>2. Make model more complex with more features</p><p>3. Use Non Linear Algorithms <em>Example( Polynomial Regression, Kernel Function in SVM</em></p><p><strong>Lowering high Variance or Overfitting:</strong></p><ol><li>Use More Data for training to make model learn maximum hidden pattern from the training data and model becomes generalised.</li></ol><p>2. Use Regularization Techniques <em>Example: L1 , L2, Drop Out, Early Stopping( in case of Neural Networks)etc.</em></p><p>3. Hyper Parameter Tuning to avoid Overfitting <em>Example: Higher value of K in KNN, Tuning of C and Gama for SVM, Depth of Tree in Decision Tree</em></p><p>4. Use less number of features — <em>Manual or Feature Selection Algorithms or automated using L1, L2 Regularization</em></p><p>5. Reduce complexity of Model — <em>Reduce polynomial degree in case of Polynomial regression and Logistic regression</em></p><p>6. Use Advance techniques like Cross Validation, Stratified Cross Validation etc.</p><blockquote>Finding the right balance is an iterative process where model is trained with different combination of features, Hyperparameters, different set of data set for training and test to find the right combination. We stop when we reach the point where Low Bias and Low variance is achieved and model is neither underfit and nor overfit.i.e Prediction accuracy is same on train and test data.</blockquote><h3>Let’s Visualise the things</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lsg_bRYztO3D19w36_g7BQ.jpeg" /><figcaption>Picture — 3 Accuracy Score on Training and Testing data during Cross Validation</figcaption></figure><p>In above picture we can have visualisation of Model Accuracy while Training/Validating on Train/Validation data.</p><p><strong>First Image : </strong>Model has low accuracy on train and validation data + Model has difference in accuracy score in train and validation → Underfit with High Bias and High Variance</p><p><strong>Second Image : </strong>Model has high accuracy on train and validation data + Model has difference in accuracy score in train and validation → Overfit with Low Bias and High Variance</p><p><strong>Third Image : </strong>Model has high accuracy on train and validation data + Model has little difference in accuracy score in train and validation → RIGHT FIT with Low Bias and Low Variance</p><p><strong>Fourth Image : </strong>Model has low accuracy on train and validation data + Model has little difference in accuracy score in train and validation → Underfit with High Bias and Low Variance</p><p><strong>Thank you for reading this article and I hope it has helped you to understand the concept and will surely help you to make right model.</strong></p><p>Happy Reading….</p><p><a href="http://www.itbodhi.com/"><strong><em>www.itbodhi.com</em></strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=76cb60dbdaf6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Bias and Variance Trade off]]></title>
            <link>https://medium.com/@itbodhi/bias-and-variance-trade-off-542b57ac7ff4?source=rss-e420879ba37------2</link>
            <guid isPermaLink="false">https://medium.com/p/542b57ac7ff4</guid>
            <category><![CDATA[bias-variance-tradeoff]]></category>
            <category><![CDATA[bias-and-variance]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[machine-learning-training]]></category>
            <category><![CDATA[data-science-training]]></category>
            <dc:creator><![CDATA[ITBodhi]]></dc:creator>
            <pubDate>Wed, 01 Jul 2020 09:24:10 GMT</pubDate>
            <atom:updated>2020-07-03T10:37:23.361Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Ai3XTuIEAaYc5SyF" /><figcaption>Photo by <a href="https://unsplash.com/@markusspiske?utm_source=medium&amp;utm_medium=referral">Markus Spiske</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In Machine Learning, when we want to optimise model prediction, it is very important to understand the parameters which describe prediction errors and accuracy . In Machine Learning, any model’s performance is based on its correct predictions and how well it is generalised i.e how accurately model gives correct predictions on seen(Training data) and unseen data (Test data, Validation data and Real time data). Best way to measure and optimise a model’s accuracy is by keeping account of the <strong>B<em>ias and Variance</em></strong> in the model.</p><p>So let’s start with the basic understanding of the concepts and see how they help to make model which higher accuracy and minimum error.</p><h4><strong>Error in Machine Learning Model</strong></h4><p>In supervised machine learning, an algorithm is trained on the training data to build model that is well designed to make correct prediction on the unseen data that is not available for training.</p><p>A machine learning model is nothing but a mathematical function which describes relationship between Predictors ( Features and Machine Learning terminology) and Traget variable.</p><p>Suppose that we have a training set consisting of a set of points x₁, x₂,x₃….xn and labelled output as y₁,y₂,y₃…..yn associcated with each record. We assume that there is a function which describes this relationship as follows:</p><blockquote>y=f(x) + e</blockquote><blockquote>where e: — is noise in the data due to various reasons and that has 0 mean and some varaince.</blockquote><p><strong><em>Now purpose of model creation is to learn pattern from the given data and approximate this function f(x) using various machine learning algorithms. </em></strong>This is kind of reverse engineering where data is given and by merely looking at the data we need to find some function f^(x) that approximate the behaviour of true function f(x). That is always not possible to due to limitation of algorithms and limited size of given data to learn pattern.</p><p>On given training data set D=(x₁,y₁), (x₂,y₂) ,(x₃,y₃)……(xn,yn) we need to create a Machine Learning model that predicts correct ( approx.) value y for given data (xᵢ,yᵢ) from <strong><em>seen data</em></strong> (Training data) and <strong><em>unseen data</em></strong>( Test/Real Time).</p><p>There would always besome error between actual value of y and predicted y^ since real data contain noise e so we has to ready to accept some irreducible predcition error. Let us measure it using <a href="https://en.wikipedia.org/wiki/Mean_squared_error">mean squared error</a> ( For Regression problem when using Linear Regression or SVM Algorithm) that is difference between actual and prediction value.</p><blockquote>MSE = (y — y^)²</blockquote><p>If we further decompose this equation, we reach at</p><blockquote>MSE = Bias² + Variance + Irreducible error</blockquote><p>Irreducible error is the error that can’t be ZERO by creating any good models. It is a measure of the amount of noise in our training data. No matter how good we make our prediction model using any robust algorithm, irreducible error that can not be removed.</p><blockquote><strong><em>Here we can see prediction error is made of two components which can be controlled: Bias and Variance</em></strong></blockquote><h3>What is Bias?</h3><p>Bias is the difference between the average prediction of our model and the correct target value which model is trying to predict.</p><p>Bias is inherent to the algorithm we choose to make the Model. A biased model is one that makes incorrect assumptions about the dataset to make the target function easier to learn.For example, suppose that we use a linear regression model on a data that has cubic relationhip. This model will be biased because we have taken wrong assumption about the data and model would be trained using this wrong assumptions about data.</p><blockquote><strong><em>Low Bias is desired : High Bias means UnderFitting of the model on training data.</em></strong></blockquote><blockquote>Given points x, a true value f(x), and a model f̂ (x), bias can be expressed mathematically as:</blockquote><blockquote>Bias[f̂ (x)]=E[f̂ (x)−f(x)]</blockquote><blockquote>Where E[⋅] is the <a href="https://en.wikipedia.org/wiki/Expected_value">expected value</a> function (e.g. the mean value).</blockquote><p><strong><em>Model with high bias is very simple</em></strong> because of the assumptions made about the data( which may or may not be true) <strong><em>pays very little attention to the training data and oversimplifies the model. High Bias model always leads to high error on training as well as on test data.</em></strong></p><p>In supervised learning, <strong>Underfitting/High Bias </strong>happens when algorithms are simple( Linear Algorithms) making them fast to learn and easier to understand but generally less flexible to learn complex patterns in data. They have lower predictive performance on complex problems that fail to meet the simplifying assumptions of the algorithms bias.</p><ul><li><strong>Low Bias </strong>: Suggests less assumptions about the relation in data to predict the target function. <em>Example: SVM, Decision Tree, KNN</em></li><li><strong>High-Bias </strong>: Suggests more assumptions about the relation in data to predict the target function. <em>Example: Linear Regression, Logistic Regression</em></li></ul><h3>What is Variance?</h3><p>Variance is the amount that the estimate of the target function will change if different training data was used.</p><p>The <em>variance</em> of the learning method, how much the learning method will move around its mean.</p><p><strong>Variance</strong> is error from sensitivity to small fluctuations in the training set. High variance can cause an algorithm to model the random noise in the training data, rather than the intended result.</p><blockquote><strong><em>Low Variance is desired : High Variance means OverFitting of the model on training data.</em></strong></blockquote><blockquote>It can expressed mathematically as:</blockquote><blockquote><em>Var(f̂ (x))=E[f̂ (x)²]−E[f̂ (x)]²</em></blockquote><blockquote>Where E[⋅] is the <a href="https://en.wikipedia.org/wiki/Expected_value">expected value</a> function (e.g. the mean value).</blockquote><p><strong><em>Model with high variance pays a lot of attention to training data and fail to generalize on the unseen data. As a result, such models perform very well on training data but has high prediction error on unseen/test data.</em></strong></p><p>We are doing the reverse engineering and trying to learn true relationship function from the training data by a machine learning algorithm, so we should expect the algorithm to have some variance. Ideally, it should not change too much from one training dataset to the next, meaning that the algorithm is good at picking out the hidden underlying mapping between the inputs and the output variables.</p><p>In supervised learning, <strong>overfitting</strong> happens when algorithms(Non Linear Algorithms) have a high variance are strongly influenced by the specifics of the training data and try to learn patterns which are noisy and not generalized and only limited to training data set.</p><ul><li><strong>Low Variance</strong>: Small changes to the estimate of the target function with changes to the training dataset. <em>Example: Linear Regression, Logistic Regression</em></li><li><strong>High Variance</strong>: Large changes to the estimate of the target function with changes to the training dataset.<em>Example: SVM, Decision Tree, KNN</em></li></ul><h3>Bias-Variance Trade-Off</h3><blockquote><strong><em>Ultimate goel of any machine learning algorithm is to build a prediction model with Low Bias and Low Variance</em></strong> that is measure of reduced error and correct prediction power of any model.</blockquote><ol><li>High Variance-High Bias — The model is inconsistent and also inaccurate on prediction</li><li>Low Variance-High Bias — Models is consistent but low on prediction</li><li>High Variance-Low Bias — Somewhat accurate but inconsistent on prediction</li><li>Low Variance-Low Bias — ideal scenario, the model is consistent and highly accurate on prediction</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t2mjAZLP7bctrCeIYXFkHA.png" /><figcaption>Bias and Variance in Machine Learning Models</figcaption></figure><p>Generally, You can see a general trend in the examples above:</p><ul><li><strong>Linear</strong> machine learning algorithms often have a high bias but a low variance. <em>Example:Linear Regression, Logistic Regression</em></li><li><strong>Nonlinear</strong> machine learning algorithms often have a low bias but a high variance. <em>Example: Decision Tree, SVM, Neural Networks</em></li></ul><p><em>The parameterization of machine learning algorithms is often a battle to balance out bias and variance.</em></p><p><strong>Finding the right balance between the bias and variance of the model is called the Bias-Variance trade-off.</strong></p><p><strong><em>There is inverse relationship between bias and variance in machine learning.</em></strong></p><ul><li>I<em>ncreasing the bias will decrease the variance.</em></li><li><em>Increasing the variance will decrease the bias.</em></li></ul><p>There is a trade-off at play between these two parameters and the algorithms you choose and the way you choose to configure them are finding different balances in this trade-off for your problem</p><blockquote><strong>High Model Complexity : High Variance , Low Bias</strong></blockquote><blockquote><strong>Low Model Complexity : High Bias , Low Variance</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/614/1*n0t9AG2CpFN1TSrKbGshmg.png" /><figcaption><a href="https://ai-pool.com/a/s/bias-variance-tradeoff-in-machine-learning">https://ai-pool.com/a/s/bias-variance-tradeoff-in-machine-learning</a>.</figcaption></figure><p>In a hypothetical example when model is perfect with ZERO error, will have perfect combination of Zero Bias and Zero Variance. In reality, it is very difficult to calculate the real bias and variance because we do not know the actual underlying target function.</p><p>If the model is too simple and has very few parameters, it will suffer from high bias and low variance. On the other hand, if the model has a large number of parameters, it will have high variance and low bias. This trade-off should result in a perfectly balanced relationship between the two.</p><h3>Bulls Eye Diagram</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/978/1*flnS5NmHx41uDh60ZO-KXg.png" /></figure><p>In the above diagram, center of the target is a model that perfectly predicts correct values. As we move away from the bulls-eye our predictions become get worse and worse. We can repeat our process of model building to get hits on the target.</p><h3>How to find the Right Balance?</h3><blockquote>Ideally, low bias and low variance is the target for any Machine Learning model.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/1*ePMqfevdkn4GBgvsZKrCcw.png" /></figure><p><strong>Lowering high Bias or Underfitting:</strong></p><ol><li>Use non Parameterised Algorithms</li></ol><p>2. Make model more complex with more features</p><p>3. Use Non Linear Algorithms <em>Example( Polynomial Regression, Kernel Function in SVM</em></p><p><strong>Lowering high Variance or Overfitting:</strong></p><ol><li>Use More Data for training to make model learn maximum hidden pattern from the training data and model becomes generalised.</li></ol><p>2. Use Regularization Techniques <em>Example: L1 , L2, Drop Out, Early Stopping( in case of Neural Networks)etc.</em></p><p>3. Hyper Parameter Tuning to avoid Overfitting <em>Example: Higher value of K in KNN, Tuning of C and Gama for SVM, Depth of Tree in Decision Tree</em></p><p>4. Use less number of features —<em> Manual or Feature Selection Algorithms or automated using L1, L2 Regularization</em></p><p>5. Reduce complexity of Model — <em>Reduce polynomial degree in case of Polynomial regression and Logistic regression</em></p><p>6. Use Advance techniques like Cross Validation, Stratified Cross Validation etc.</p><blockquote>Finding the right balance with Low Bias and Low Variance is an iterative process where model is trained with different combination of features, Hyperparameters, different set of data set for training and test to find the right combination. We stop when we reach the point where Low Bias and Low variance is achieved and model is neither underfit and nor overfit.i.e Prediction accuracy is same on train and test data.</blockquote><p>Understanding of the Bias and variance concept is fundamental to train Right prediction Model for your problem.</p><p>Thank you for reading this article and I hope it has helped you to understand the concept.</p><p>Happy Reading….</p><p><a href="http://www.itbodhi.com/">www.itbodhi.com</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=542b57ac7ff4" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>