<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Hasan Ersan YAĞCI on Medium]]></title>
        <description><![CDATA[Stories by Hasan Ersan YAĞCI on Medium]]></description>
        <link>https://medium.com/@hersanyagci?source=rss-7c248ad78b88------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*uyw7aT7kQ7SBMMWNWuqOcA@2x.jpeg</url>
            <title>Stories by Hasan Ersan YAĞCI on Medium</title>
            <link>https://medium.com/@hersanyagci?source=rss-7c248ad78b88------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 22:32:49 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@hersanyagci/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Under-Sampling Methods for Imbalanced Data (ClusterCentroids, RandomUnderSampler, NearMiss)]]></title>
            <link>https://hersanyagci.medium.com/under-sampling-methods-for-imbalanced-data-clustercentroids-randomundersampler-nearmiss-eae0eadcc145?source=rss-7c248ad78b88------2</link>
            <guid isPermaLink="false">https://medium.com/p/eae0eadcc145</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[imbalanced-data]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[scikit-learn]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Hasan Ersan YAĞCI]]></dc:creator>
            <pubDate>Thu, 15 Jul 2021 14:27:36 GMT</pubDate>
            <atom:updated>2021-08-23T12:18:26.546Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/425/0*OwxdfaWrrYYb1M99.png" /><figcaption><a href="https://hersanyagci.medium.com/random-resampling-methods-for-imbalanced-data-with-imblearn-1fbba4a0e6d3">https://hersanyagci.medium.com/random-resampling-methods-for-imbalanced-data-with-imblearn-1fbba4a0e6d3</a></figcaption></figure><p>The imbalance of data is a big problem for classification tasks. In python, there is a library to allow to use of many algorithms to handle this imbalanced state of the data and its harms.</p><p>imbalanced-learn is a python package offering a several re-sampling techniques commonly used in datasets showing strong between-class imbalance. It is compatible with scikit-learn and is part of <a href="https://github.com/scikit-learn-contrib/imbalanced-learn#id3">scikit-learn-contrib projects.</a> There is an <a href="https://hersanyagci.medium.com/random-resampling-methods-for-imbalanced-data-with-imblearn-1fbba4a0e6d3">introduction</a>, you can read the imbalanced-learn library and get some idea about its outline.</p><p>In this series, each technique will be handled separately. Re-sampling techniques are divided into two categories mainly;</p><p>● Under-sampling the majority class(es).<br>● Over-sampling the minority class.</p><p>There are others; ‘combining over and under sampling’ and ‘create ensemble balanced sets’.</p><p>This is about under-sampling;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Aom5pMJJOF038TQOYaoVIg.png" /><figcaption><a href="https://github.com/hersany/Imbalanced-Data">https://github.com/hersany/Imbalanced-Data</a></figcaption></figure><p>Under-sampling techniques are two types, prototype generation, and prototype selection.</p><h3><strong>A. Prototype Generation</strong></h3><p>Prototype generation algorithms will reduce the number of samples while generating a new set according to the given original data set. It means that it will remove the majority class and generating a new set. There is one algorithm in the imbalanced-learn library, which is ClusterCentroids.</p><h3><strong>ClusterCentroids</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/1*EaLiJ5afqPgp9DUU9ykNCg.png" /></figure><p>This technique makes undersampling by generating a new set based on centroids by clustering methods. The algorithm is generating a new set according to the cluster centroid of a KMeans algorithm.</p><p>A method that under samples the majority class by replacing a cluster of majority samples with the cluster centroid of a KMeans algorithm.</p><p>The newly generated set is synthesized with the centroids of the K-means method instead of the original samples. It transforms the majority class(es), the minority class preserves itself.</p><p>We will use the travel_insurance dataset for the examination.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/1*aMYh-0R_bf5A3hhcmiHlEw.png" /><figcaption><a href="https://www.kaggle.com/ersany/undersampling-methods-part-1">https://www.kaggle.com/ersany/undersampling-methods-part-1</a></figcaption></figure><p>There is an imbalance between 0 and 1 at the target feature.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/948/1*YYJFQxBFMo7IlL3_HGfGeg.png" /><figcaption><a href="https://www.kaggle.com/ersany/undersampling-methods-part-1">https://www.kaggle.com/ersany/undersampling-methods-part-1</a></figcaption></figure><p>After the undersampling technique (ClusterCentroids), the number of majority classes (it is 0) decreased to the number of minority classes (it is 1). The data are balanced now.</p><p>We can check the original data and undersampling data;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*PSuozpx30KBLeJzQaiSdPg.png" /><figcaption><a href="https://www.kaggle.com/ersany/undersampling-methods-part-1">https://www.kaggle.com/ersany/undersampling-methods-part-1</a></figcaption></figure><p>At the original data set, there are some instances which are Claim is 0, Duration is 46 and Age is 33. After ClusterCentroids, all of them are removed and generated a new one. Net Sales and Commission features which are the continuous variables are synthesized based on centroids.</p><p>For the instance which is Claim is 1, Duration is 76 and Age is 36, there is nothing to change. The minority class preserves its original form.</p><pre>from imblearn.under_sampling import ClusterCentroids<br>undersampler = ClusterCentroids()</pre><pre>X_smote, y_smote = undersampler.fit_resample(X_train, y_train)</pre><p>There are some parameters at ClusterCentroids, with sampling_strategy we can adjust the ratio between minority and majority classes. We can change the algorithm of the estimator by the estimator parameter. Its default value is KMeans. Check this <a href="https://imbalanced-learn.org/stable/references/generated/imblearn.under_sampling.ClusterCentroids.html">link</a> for the parameters.</p><h3>B. Prototype Selection</h3><p>Unlike to prototype generation algorithm, prototype selection algorithms will select samples from the original set.</p><p>Prototype selection algorithms will reduce the number of majority class(es), but they won’t generate new instances.</p><p>Prototype selection algorithms are divided into two groups:</p><p>● Controlled under-sampling techniques.<br>● Cleaning under-sampling techniques.</p><p>At the controlled under-sampling techniques, the user can define the number of samples. The algorithms are worked under control.</p><p>At the cleaning under-sampling techniques, the user can’t define the number of samples. The algorithms work themselves, clean the feature space and try to reduce noise.</p><h3><strong>B.1. Controlled under-sampling techniques</strong></h3><p>There are two algorithms at the imbalanced-learn library as controlled under-sampling techniques</p><h3>RandomUnderSampler</h3><p>RandomUnderSampler randomly deletes the rows of the majority class(es) according to our sampling strategy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/817/1*c_NxNUqce3dGXsCiqcdqkA.png" /></figure><p>RandomUnderSampler is a fast and easy way to balance the data by randomly selecting a subset of data for the targeted classes.</p><pre>from imblearn.under_sampling import RandomUnderSampler</pre><pre>under = RandomUnderSampler(sampling_strategy=1) <br>                     <br>X_smote, y_smote = under.fit_resample(X_train, y_train)</pre><p>This topic has been covered before at <a href="https://hersanyagci.medium.com/random-resampling-methods-for-imbalanced-data-with-imblearn-1fbba4a0e6d3">here</a>. For more detail, you can check this <a href="https://imbalanced-learn.org/stable/references/generated/imblearn.under_sampling.RandomUnderSampler.html">link</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/1*YgqRsCRckI6riogX2KmTOQ.png" /><figcaption><a href="https://www.kaggle.com/ersany/undersampling-methods-part-1">https://www.kaggle.com/ersany/undersampling-methods-part-1</a></figcaption></figure><h3>NearMiss</h3><p>NearMiss is a kind of controlled under-sampling technique. It has some rules additional to RandomUnderSampler. NearMiss has two important parameters; these are version and n_neighbors.</p><p>NearMiss has 3 versions. We can define the version with the version parameter.</p><p>n_neighbors refer to the size of the neighborhood to consider to compute the average distance to the minority point samples.</p><p>NearMiss-1 selects samples from the majority class for which the average distance of the k nearest samples of the minority class is the smallest. NearMiss-2 selects the samples from the majority class for which the average distance to the farthest samples of the negative class is the smallest. NearMiss-3 is a 2-step algorithm: first, for each minority sample, their m nearest-neighbors will be kept; then, the majority samples selected are the on for which the average distance to the k nearest neighbors is the largest.</p><p>We can look at them on plots.</p><p><strong>NearMiss-1</strong> selects the positive samples for which the average distance to the k closest samples of the negative class is the smallest.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*mQ-vU2Q976Yl3Dm8r_WRPw.png" /></figure><p><strong>NearMiss-2</strong> selects the positive samples for which the average distance to the k farthest samples of the negative class is the smallest.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/597/1*52CR0BCJXwWMvPolv3WYYg.png" /></figure><p><strong>NearMiss-3</strong> is a 2-steps algorithm. First, for each negative sample, their m nearest-neighbors will be kept. Then, the positive samples selected are the ones for which the average distance to the k nearest-neighbors is the largest.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*VX_aldBnPLRkvPN2Njkodg.png" /></figure><pre>from imblearn.under_sampling import NearMiss</pre><pre>undersampler = NearMiss(version = 1, n_neighbors = 3) <br>                     <br>X_smote, y_smote = undersampler.fit_resample(X_train, y_train)</pre><p>For more detail, you can check this <a href="https://imbalanced-learn.org/stable/under_sampling.html#controlled-under-sampling">link</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*e1WHty-f_MNlitqPLRp5cQ.png" /><figcaption><a href="https://www.kaggle.com/ersany/undersampling-methods-part-1">https://www.kaggle.com/ersany/undersampling-methods-part-1</a></figcaption></figure><h3>References</h3><ul><li><a href="https://www.kaggle.com/ersany/undersampling-methods-part-1">Undersampling Methods Part-1</a></li><li><a href="https://github.com/hersany/Imbalanced-Data">hersany/Imbalanced-Data</a></li><li><a href="https://hersanyagci.medium.com/random-resampling-methods-for-imbalanced-data-with-imblearn-1fbba4a0e6d3">Random Resampling Methods for Imbalanced Data with Imblearn</a></li><li><a href="https://github.com/scikit-learn-contrib/imbalanced-learn#id3">scikit-learn-contrib/imbalanced-learn</a></li><li><a href="https://imbalanced-learn.org/stable/install.html">Getting Started - Version 0.8.0</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eae0eadcc145" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Yellowbrick; Machine Learning Visualization]]></title>
            <link>https://hersanyagci.medium.com/yellowbrick-machine-learning-visualization-a8c2e9cae78e?source=rss-7c248ad78b88------2</link>
            <guid isPermaLink="false">https://medium.com/p/a8c2e9cae78e</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[yellow-brick]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[scikit-learn]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Hasan Ersan YAĞCI]]></dc:creator>
            <pubDate>Mon, 19 Apr 2021 22:27:48 GMT</pubDate>
            <atom:updated>2021-04-19T22:49:07.211Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cPzqREq02Dav_5S4.png" /><figcaption><a href="https://github.com/DistrictDataLabs/yellowbrick/tree/develop/examples">https://github.com/DistrictDataLabs/yellowbrick/tree/develop/examples</a></figcaption></figure><p>Visualization is essential to make our analysis or modeling process understandable. We need visualization to see results or workflow, especially in machine learning algorithms.</p><p><a href="https://www.scikit-yb.org/en/latest/">Yellowbrick</a> was created for this. Yellowbrick is a visualization library that can work with Scikit Learn machine learning algorithms. It is not a part of <a href="https://github.com/scikit-learn-contrib">scikit-learn-contrib</a> projects, but it uses Scikit-Learn API to make classification, clustering, hyperparameter selection, model selection, etc. It helps the user in many areas. Yellowbrick generates visualizations by wrapping <a href="https://matplotlib.org/">Matplotlib,</a> the most prominent Python scientific visualization library.</p><p>When we want to add a good visual to our workbook, or when we can’t decide on selection parameters or interpret scores, we can rely on Yellowbrick. Yellowbrick not only provides good visualization but also provides advice on parameters.</p><h3><strong>Using Yellowbrick</strong></h3><pre>pip install yellowbrick</pre><p>The Yellowbrick API is specially designed to play nicely with scikit-learn. The workflow is very similar to using a scikit-learn transformer and the visualizers are intended to be integrated with scikit-learn utilities. In Yellowbrick, the primary interface is a visualizer. Visualizers are scikit-learn estimator objects.</p><p>To use the visualizers, simply use the same workflow as with a scikit-learn model. Import the visualizer, instantiate it, call the visualizer’s fit() method, then to render the visualization, call the visualizer’s show() method.</p><pre>from sklearn.linear_model import LinearRegression<br>model = LinearRegression()</pre><pre>from yellowbrick.regressor import ResidualsPlot</pre><pre>visualizer = ResidualsPlot(model)</pre><pre>visualizer.fit(X_train, y_train)  # Fit the training data<br>visualizer.score(X_test, y_test)  # Evaluate the model on the test <br>visualizer.show()                 # Finalize and render the figure</pre><p>In the above example, we instantiate LinearRegression from scikit-learn. Next, for the ResidualsPlot, we instantiate the visualizer with LinearRegression from Yellowbrick. Then we use fit, score, and show methods such as scikit-learn workflow. We don’t fit LinearRegression into train data. After dividing the data, we apply Yellowbrick directly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/1*Y1M-VOE0XhO0QgVhfRI4AA.png" /></figure><p>The image can be saved as follows In addition to the .png extension, .pdf is also commonly used for high-quality publication-ready images.</p><pre>visualizer.show(outpath=&quot;pcoords.png&quot;)</pre><h3>Usage areas of Yellowbrick</h3><ul><li>Classification Visualization</li><li>Clustering Visualization</li><li>Feature Visualization</li><li>Model Selection Visualization</li><li>Regression Visualization</li><li>Target Visualization</li><li>Text Visualization</li></ul><p>We can use Yellowbrick visualizers in different areas and at different stages in the machine learning pipeline. It helps the user to make a more accurate decision with good visualization. It makes the process easier and better.</p><p>For example; Defining numbers of the cluster is a difficult choice in clustering problems. If we do not have any prior knowledge, we need to find the number of clusters for the K-mean clustering algorithm. We use the elbow method, but in this method, it is our job to choose the number of clusters. But Yellowbrick can decide instead of us with good visualization.</p><p>You can see the API documentation for Yellowbrick <a href="http://documentation for Yellowbrick">here</a>.</p><p>Let’s give a few examples with Yellowbrick;</p><p><strong>Elbow Method</strong></p><p>In the clustering problem, we need the number of clusters and we can use the elbow method. We generate a dataset with 8 random clusters and apply KElbowVisualizer of Yellowbrick. We use the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.calinski_harabasz_score.html">calinski_harabasz</a> score to obtain the k parameter.</p><pre>from sklearn.cluster import KMeans<br>from sklearn.datasets import make_blobs<br><br>from yellowbrick.cluster import KElbowVisualizer<br><br>X, y = make_blobs(n_samples=1000, n_features=12, centers=8, random_state=42)<br><br><em># Instantiate the clustering model and visualizer</em></pre><pre>model = KMeans()<br>visualizer = KElbowVisualizer(<br>    model, k=(4,12), metric=&#39;calinski_harabasz&#39;, timings=False)<br><br>visualizer.fit(X)        <em># Fit the data to the visualizer</em><br>visualizer.show()        <em># Finalize and render the figure</em></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*XDBfehNymu72cKgd.png" /></figure><p>We got the plot above. It gave us the parameter k correctly. If we use the classical elbow method, our plot will be like the below. 8 seems to be the breaking point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*uVBXFVUJ758vjxKj1dbidA.png" /></figure><p><strong>Prediction Error Plot</strong></p><p>A prediction error plot shows the actual targets in the data set and the predicted values generated by our model. We can see the variance of the model. This plot also gives us the score.</p><pre>from sklearn.linear_model import Lasso<br>from sklearn.model_selection import train_test_split<br><br>from yellowbrick.datasets import load_concrete<br>from yellowbrick.regressor import PredictionError<strong><br></strong><br><em># Load a regression dataset</em></pre><pre>X, y = load_concrete()<br><br><em># Create the train and test data</em></pre><pre>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)<br><br><em># Instantiate the linear model and visualizer</em></pre><pre>model = Lasso()<br>visualizer = PredictionError(model)<br><br>visualizer.fit(X_train, y_train)  <em># Fit the training data</em><br>visualizer.score(X_test, y_test)  <em># Evaluate the model </em><br>visualizer.show()                 <em># Finalize and render the figure</em></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*LY2aGRci0puopT7Q.png" /></figure><p><strong>Alpha Selection</strong></p><p>Regularization is designed to penalize model complexity, therefore the higher the alpha, the less complex the model, decreasing the error due to variance (overfit). Alphas that are too high on the other hand increase the error due to bias (underfit). It is important, therefore to choose an optimal alpha such that the error is minimized in both directions.</p><p>The AlphaSelection Visualizer demonstrates how different values of alpha influence model selection during the regularization of linear models. Generally speaking, alpha increases the effect of regularization, e.g. if alpha is zero there is no regularization, and the higher the alpha, the more the regularization parameter influences the final model.</p><pre>import numpy as np<br><br>from sklearn.linear_model import LassoCV<br>from yellowbrick.datasets import load_concrete<br>from yellowbrick.regressor import AlphaSelection<br><br><em># Load the regression dataset</em><br>X, y = load_concrete()<br><br><em># Create a list of alphas to cross-validate against</em><br>alphas = np.logspace(-10, 1, 400)<br><br><em># Instantiate the linear model and visualizer</em><br>model = LassoCV(alphas=alphas)<br>visualizer = AlphaSelection(model)<br>visualizer.fit(X, y)<br>visualizer.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*LeMIB-Wjqa45g2K0.png" /></figure><p><strong>Rank Features</strong></p><p>A two-dimensional ranking of features utilizes a ranking algorithm that takes into account pairs of features at a time (e.g. joint plot analysis). The pairs of features are then ranked by score and visualized using the lower-left triangle of a feature co-occurrence matrix.</p><p>By default, the Rank2D visualizer utilizes the Pearson correlation score to detect colinear relationships.</p><pre>from yellowbrick.datasets import load_credit<br>from yellowbrick.features import Rank2D<br><br><em># Load the credit dataset</em><br>X, y = load_credit()<br><br><em># Instantiate the visualizer with the Pearson ranking algorithm</em><br>visualizer = Rank2D(algorithm=&#39;pearson&#39;)<br><br>visualizer.fit(X, y)           <em># Fit the data to the visualizer</em><br>visualizer.transform(X)        <em># Transform the data</em><br>visualizer.show()              <em># Finalize and render the figure</em></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*NWBawfempCPj_ZPT.png" /></figure><h3>Conclusion</h3><p>Yellowbrick is the machine learning library that makes things easy. Making the machine learning model in Yellowbrick is not very effective because we can set parameters in scikit-learn meaningfully, but it can be used as a consultant in the visualization and machine learning process.</p><p><a href="https://www.scikit-yb.org/en/latest/">Yellowbrick official website</a> has all codes you can take them and apply to your project easily.</p><h3>References</h3><ul><li><a href="https://www.scikit-yb.org/en/latest/index.html">Yellowbrick: Machine Learning Visualization - Yellowbrick v1.5 documentation</a></li><li><a href="https://pypi.org/project/yellowbrick/">yellowbrick</a></li><li><a href="https://github.com/DistrictDataLabs/yellowbrick/tree/develop/examples">DistrictDataLabs/yellowbrick</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a8c2e9cae78e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Feature Selection with BorutaPy, RFE and Univariate Feature Selection]]></title>
            <link>https://hersanyagci.medium.com/feature-selection-with-borutapy-rfe-and-univariate-feature-selection-f1779ca4bd01?source=rss-7c248ad78b88------2</link>
            <guid isPermaLink="false">https://medium.com/p/f1779ca4bd01</guid>
            <category><![CDATA[feature-selection]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[boruta]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[scikit-learn]]></category>
            <dc:creator><![CDATA[Hasan Ersan YAĞCI]]></dc:creator>
            <pubDate>Sun, 04 Apr 2021 19:26:17 GMT</pubDate>
            <atom:updated>2021-04-06T19:51:38.287Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*PXK6dkYKYVBEqSllnIfpVw.png" /><figcaption><a href="https://freerangestock.com/">https://freerangestock.com/</a></figcaption></figure><p><a href="https://en.wikipedia.org/wiki/Feature_selection">Feature selection</a> is one of the most critical stages of a machine learning pipeline. We may have to struggle with a lot of features or useless features, so we have to do some elimination of features. So why is feature selection important? Because we can increase the performance and quality of our model, we can shorten the training time. We do not want to train our model on unnecessary features, we eliminate noisy (non-informative) features that do not contribute to our model or cause overfitting. The feature selection can be the key to a successful machine learning process.</p><p>Feature selection is a technique where we choose features from our data that contribute most to the target variable. In other words, we choose the best predictors for the target variable so we can predict the target with higher accuracy and with lower variance, preferably.</p><p>Although the terms feature selection and feature extraction are sometimes used interchangeably, they are indeed two separate techniques. Feature selection gives us exactly the features themselves. It does not perform any conversion or transformation; it only removes unnecessary ones according to given constraints. On the other hand, with feature extraction, we do not know which features are more important than others. Feature extraction transforms features, creating a combination of features that represent all features. Feature importance is another term, often appears as a sub-stage within feature selection methods where features are sorted according to their importance level, i.e., their contribution to the model (output).</p><p>In this article we will focus on feature selection, feature extraction and feature importance will be the topic of another article.</p><p>Sci-kit learn library, the most popular library for data science in Python offers some useful methods for feature selection. Boruta library also provides a handy, scikit-learn compatible api for Boruta feature selection algorithm. We will be mainly focusing on techniques mentioned above.</p><p>Feature selection techniques will be applied to the diamond dataset from Seaborn. For the sake of simplicity, we have removed the categorical features and split our data. Price is the target variable.</p><pre>import pandas as pd<br>import seaborn as sns<br>from sklearn.model_selection import train_test_split</pre><pre>df = sns.load_dataset(&#39;diamonds&#39;)<br>df = df.drop([&#39;cut&#39;, &#39;color&#39;, &#39;clarity&#39;], axis = 1)</pre><pre>X = df.drop(&#39;price&#39;, axis = 1)<br>y = df[&#39;price&#39;]</pre><pre>X_train, X_test, y_train, y_test = train_test_split(X, y,test_size=0.2)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*hJcPuBjCpt3hQicXN7kolQ.png" /></figure><p>All the methods below have a statistical and mathematical background that could be explored in-depth, yet we will just give a simple introduction.</p><p><strong>1 — BorutaPy</strong></p><p>Boruta is an improved Python implementation of the Boruta R package. We will use BorutaPy from the Boruta library. BorutaPy is a feature selection algorithm based on NumPy, SciPy, and Sklearn.</p><p>We can use BorutaPy just like any other scikit learner: fit, fit_transform and transform are all implemented similarly. BorutaPy needs an estimator and an ensemble method could be used: random forest, extra trees classifier, even gradient boosted trees.</p><p>BorutaPy generates shadow features (shuffled copies of all features) of the original data and calculates the threshold based on the estimator’s feature importance of shadow features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/815/1*svIBPaEd69WCE7vzm_t3DQ.png" /></figure><p><a href="https://www.semanticscholar.org/paper/Boruta-A-System-for-Feature-Selection-Kursa-Jankowski/85a8b1d9c52f9f795fda7e12376e751526953f38?p2df">It compares, in the iterative fashion, the importances of original attributes with importances of their randomised copies.</a></p><pre>from sklearn.ensemble import RandomForestRegressor<br>from boruta import BorutaPy</pre><pre>forest = RandomForestRegressor(max_depth = 5)</pre><pre>feat_selector = BorutaPy(forest, n_estimators=&#39;auto&#39;, verbose=2, random_state=1)</pre><pre>feat_selector.fit(np.array(X_train), np.array(y_train))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*3cx1UvLWw06CTwizTylDqg.png" /></figure><p>After fitting BorutaPy it provides the user with ranking of features. Confirmed ones are 1, tentatives are 2, and the rejected ones are 3, based on their feature importance history through the iterations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/708/1*wTV2d8EEcStARRVtGfW29g.png" /></figure><p>The important point is for BorutaPy, multicollinearity should be removed before running it.</p><p><strong>2— Recursive Feature Elimination (RFE)</strong></p><p>RFE makes an elimination with the estimator which provides feature importances to RFE. For example, if the estimator is Linear Regression, RFE uses coefficients of the linear model; if the estimator is Random Forest, then RFE uses feature importance method of Random Forest, etc.</p><p>RFE filters the features according to a number that the user wants to select, by the weights which are assigned by the external estimator (supervised learning algorithm).</p><p>There are other versions of recursive feature elimination in the sklearn.feature_importance;</p><pre><a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.RFECV.html#sklearn.feature_selection.RFECV"><strong>RFECV</strong></a></pre><pre>Recursive feature elimination with a built-in cross-validated selection of the best number of features.</pre><pre><a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectFromModel.html#sklearn.feature_selection.SelectFromModel"><strong>SelectFromModel</strong></a></pre><pre>Feature selection based on thresholds of importance weights.</pre><pre><a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SequentialFeatureSelector.html#sklearn.feature_selection.SequentialFeatureSelector"><strong>SequentialFeatureSelector</strong></a></pre><pre>Sequential cross-validation based feature selection. Does not rely on importance weights.</pre><p>All these methods mentioned above use the same source: estimator assigned weights on features (feature importance). They just need extra parameters. We will apply RFE to the diamonds dataset.</p><pre>from sklearn.feature_selection import RFE<br>from sklearn.linear_model import LinearRegression</pre><pre>model = LinearRegression()<br>rfe = RFE(estimator = model, n_features_to_select = 5)<br>fit = rfe.fit(X_train, y_train)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*G3ELkccl5jn6J0vGUhJ0Hw.png" /></figure><p>RFE recursively refit the model and eliminated the worst feature at each step and added back eliminated features one feature (if not specified otherwise by user) at a time to search for the best combination of features until the number of features fell down to a preset limit. We defined the n_features_to_select as 5, so it gave us 5 features out of 6 initial features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*GXVdTqe6xoLUs4SgBScTuA.png" /></figure><p>We can control the RFE according to the coefficients of the linear regression model (we can say feature importance). The ‘z’ feature is the last. RFE has removed the ‘z’ feature.</p><p><strong>3 — Univariate Feature Selection</strong></p><p>Univariate feature selection works by selecting the best features based on bivariate statistical tests or univariate statistics.</p><p>There are different univariate feature selection methods in sklearn, we will focus on the most commonly used <strong>SelectKBest</strong>, there are other methods in sklearn.feature_selection;</p><pre><a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectPercentile.html#sklearn.feature_selection.SelectPercentile"><strong>SelectPercentile</strong></a></pre><pre>Select features based on the percentile of the highest scores.</pre><pre><a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectFpr.html#sklearn.feature_selection.SelectFpr"><strong>SelectFpr</strong></a></pre><pre>Select features based on a false positive rate test.</pre><pre><a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectFdr.html#sklearn.feature_selection.SelectFdr"><strong>SelectFdr</strong></a></pre><pre>Select features based on an estimated false discovery rate.</pre><pre><a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectFwe.html#sklearn.feature_selection.SelectFwe"><strong>SelectFwe</strong></a></pre><pre>Select features based on family-wise error rate.</pre><pre><a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.GenericUnivariateSelect.html#sklearn.feature_selection.GenericUnivariateSelect"><strong>GenericUnivariateSelect</strong></a></pre><pre>Univariate feature selector with configurable mode.</pre><p>There is also another method, <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.VarianceThreshold.html"><strong>removing features with low variance.</strong></a> It is not widely used, however might come in handy in when looking for a method in our pipeline to quickly remove constants.</p><p>We use the SelectKBest method, but we also need the score function, each method above needs the score function.</p><ul><li><strong>For regression:</strong> f_regression, mutual_info_regression</li><li><strong>For classification:</strong> chi2, f_classif, mutual_info_classif</li></ul><p>These functions use some tests, f_regression uses univariate linear regression tests, f_classif uses ANOVA F-value method, chi2 k-uses chi-square statistics. mutual_info_regression and mutual_info_classif functions based on entropy estimation from k-nearest neighbors distances.</p><p>SelectKBest takes another parameter, k, besides the score function. SelectKBest gives scores based on the score function and selects k number of features in turn.</p><pre>from sklearn.feature_selection import SelectKBest, f_regression</pre><pre>selector = SelectKBest(score_func = f_classif, k = 5)</pre><pre>X_new = selector.fit_transform(X_train, y_train)<br>names = X_train.columns.values[selector.get_support()]<br>scores = selector.scores_[selector.get_support()]<br>names_scores = list(zip(names, scores))</pre><pre>ns_df = pd.DataFrame(data = names_scores, columns=[&#39;Feat_names&#39;, &#39;F_Scores&#39;])</pre><pre>ns_df.sort_values(&#39;F_Scores&#39;, ascending=False)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*SazG6fb2PfSCGcLWnseZcw.png" /></figure><p>SelectKBest labels the features as True and False, in the example above we set k as 5. With the True tag, we will get the best 5 attributes (strongest relationship to output). The ‘depth’ feature is labeled False because we have 6 features and the ‘depth’ is the last one in terms of association determined by chosen test.</p><p><strong>Conclusion</strong></p><p>Feature selection methods select the features according to our decisions that include parameters like the number of the features or a threshold. It is the main difference between feature selection and feature importance.</p><p>BorutaPy is a more robust solution for feature selection because it doesn’t need any parameters or threshold. But for other methods, we have to give a number or threshold.</p><p>All methods have different properties, strengths, and weaknesses for different types of datasets. There is no one-works-for all feature selection method, so we must choose functional and true one for our data.</p><p><strong>References</strong></p><p><a href="https://www.semanticscholar.org/paper/Boruta-A-System-for-Feature-Selection-Kursa-Jankowski/85a8b1d9c52f9f795fda7e12376e751526953f38?p2df">https://www.semanticscholar.org/paper/Boruta-A-System-for-Feature-Selection-Kursa-Jankowski/85a8b1d9c52f9f795fda7e12376e751526953f38?p2df</a></p><p><a href="https://scikit-learn.org/stable/modules/feature_selection.html#feature-selection-as-part-of-a-pipeline">https://scikit-learn.org/stable/modules/feature_selection.html#feature-selection-as-part-of-a-pipeline</a></p><p><a href="https://github.com/hersany/DataScience/blob/master/Other/Feature_Selection.ipynb">https://github.com/hersany/DataScience/blob/master/Other/Feature_Selection.ipynb</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f1779ca4bd01" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Random Resampling Methods for Imbalanced Data with Imblearn]]></title>
            <link>https://hersanyagci.medium.com/random-resampling-methods-for-imbalanced-data-with-imblearn-1fbba4a0e6d3?source=rss-7c248ad78b88------2</link>
            <guid isPermaLink="false">https://medium.com/p/1fbba4a0e6d3</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[imbalanced-data]]></category>
            <dc:creator><![CDATA[Hasan Ersan YAĞCI]]></dc:creator>
            <pubDate>Thu, 25 Mar 2021 08:19:30 GMT</pubDate>
            <atom:updated>2021-07-15T12:43:21.630Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ty32tHfHSz-0ofre.png" /><figcaption><a href="https://imbalanced-learn.org/stable/auto_examples/over-sampling/plot_comparison_over_sampling.html#sphx-glr-auto-examples-over-sampling-plot-comparison-over-sampling-py">Reference</a></figcaption></figure><p>We want the data we prepared or analyzed for the model to be perfect. However, data may have missing values, outliers, complex data types. We always do some preprocessing. This process is normal. However, sometimes we encounter imbalanced data in classification tasks. This is unusual for algorithms and this imbalanced situation needs to be fixed.</p><p>For example, you have a binary classification task and you have 100 records (rows) in your data, of which 90 rows are labeled as 1 and the remaining 10 rows are labeled as 0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*a5JJ7kPOI0V6o5fA.png" /></figure><p>With this data, our model would be biased. The predictions will be dominated by the majority class.</p><p>To prevent this, we can refer to the Imbalanced-learn Library.</p><blockquote><a href="https://imbalanced-learn.org/stable/">Imbalanced-learn (imported as </a><a href="https://imbalanced-learn.org/stable/"><strong><em>imblearn</em></strong>) is an open source, MIT-licensed library relying on scikit-learn (imported as </a><a href="https://imbalanced-learn.org/stable/"><strong><em>sklearn</em></strong>) and provides tools when dealing with classification with imbalanced classes.</a></blockquote><p>The Imbalanced-learn library includes some methods for handling imbalanced data. These are mainly; <strong>under-sampling, over-sampling, a combination of over-and under-sampling, and an ensemble of samplers.</strong> You can take the relevant method or algorithm and apply it to your data.</p><p>Each method mentioned above contains many techniques and algorithms. You can use boosting or bagging algorithms in ensemble methods. You can create a pipeline with SMOTE and under-sampling methods. It depends on your data, analysis, and approach. See the <a href="https://imbalanced-learn.org/stable/user_guide.html">documentation</a> for details.</p><p>To give you an idea, we will apply random resampling techniques, naive over_sampling and under_sampling methods, which are the most common imblearn library implementations. These are <a href="https://imbalanced-learn.org/stable/references/generated/imblearn.over_sampling.RandomOverSampler.html#imblearn.over_sampling.RandomOverSampler">RandomOverSampler</a> and <a href="https://imbalanced-learn.org/stable/references/generated/imblearn.under_sampling.RandomUnderSampler.html#imblearn.under_sampling.RandomUnderSampler">RandomUnderSampler</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*PkM2asY31sx2rSkR.png" /><figcaption><a href="https://www.kdnuggets.com/">Reference</a></figcaption></figure><p><strong>What do these methods do?</strong></p><p>We can define;</p><ul><li>RandomOverSampler duplicates rows of the minority class.</li><li>RandomUnderSampler deletes rows of the majority class.</li></ul><p>These two methods make the duplicating and deleting randomly. Naive resampling methods are best if we want to get balanced data quickly and easily.</p><p>We should note that we only apply it to training data. We just balance our training data, our test data remain as is (original distribution). It means that we apply resampling methods after dividing our data into training and testing.</p><p>We will analyze the travel insurance data to apply our resampling methods, the data is as follows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/1*X6DG2zYqko5yD5c2-pPmvA.png" /></figure><p>We have a binary classification problem. Our target feature is “Claim”. 0 is the majority, 1 is the minority class. The target distribution is like that;</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*gWCAznM14-rmDKiikmOzOw.png" /></figure><p>We will apply Logistic Regression to compare the results between imbalanced data and resampling data. This dataset is from the kaggle, its fame comes from being a formidable unbalanced dataset, and the low scores. We don’t have an exploratory data analysis process to better see the comparison, so scores may below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*YAeY6UIfGB7dUXip4wJzmQ.png" /></figure><p>We applied Logistic Regression to the data before resampling methods. According to the classification report; precision, recall, and f1-score of 1 are 0 because the model couldn’t learn. The model predicted all records to be 0 in favor of the majority class. It gives us a prediction model that always returns the majority class. It ignores the minority class.</p><p>The f1 score is the most suitable measure for unbalanced dataset models. So we use the f1 score to compare.</p><p>Now we will apply in order, RandomOverSampler, RandomUnderSampler, and the combination of over-under sampler.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/564/1*3W3fw5k7QhaEKZHY6k0lmQ.png" /><figcaption><a href="https://www.hindawi.com/">Reference</a></figcaption></figure><p><strong>Oversampling</strong></p><p>We add synthetic rows to the data with the RandomOverSampler. We make the number of target values equal by increasing the minority class. This can be beneficial or harmful, depends on the quantities. However, we do not want to make our data synthetic. If there is a significant difference between the majority and minority class, apply this method carefully, maybe we can adjust the sampling strategy parameter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*5IiXjuE1GwSebUbauz27qQ.png" /></figure><p>We set the sampling strategy to 1. It means that the minority class will be the same amount (1 to 1) as the majority class, the minority class will copy their rows. Check y_smote’s value_counts (y_train converted to y_smote with resampling method)</p><p>We split our data into training and testing and apply RandomOverSampler only to training data (X_train and y_train). If we resampling the test data or all data, it may cause data leakage.</p><pre>from imblearn.over_sampling import RandomOverSampler</pre><pre>over = RandomOverSampler(sampling_strategy=1)</pre><pre>X_smote, y_smote = over.fit_resample(X_train, y_train)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/1*aLtQ3BB5YonObkf5ODphEw.png" /></figure><p>After Logistic Regression, the metrics of the data are above. The score increases by 9.52% after RandomOverSampler.</p><p><strong>Undersampling</strong></p><p>RandomUnderSampler randomly deletes the rows of the majority class according to our sampling strategy. This resampling method deletes the actual data consider this situation. We don’t want to lose or compress our data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*PO81ywzI-ByPiT4T4Y2ycg.png" /></figure><p>We adjust the sampling strategy as 1. It means that the majority class will be the same amount as the minority class (1 to 1), the majority class will lose rows. Check y_smote’s value_counts (y_train converted to y_smote by resampling method).</p><p>We split our data into training and testing and apply RandomUnderSampler only to training data (X_train and y_train).</p><pre>from imblearn.under_sampling import RandomUnderSampler</pre><pre>under = RandomUnderSampler(sampling_strategy=1) <br>                     <br>X_smote, y_smote = under.fit_resample(X_train, y_train)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/1*FYIYLeRA-BSzb5KTsvcmow.png" /></figure><p>After Logistic Regression, the metrics of the data are above. The score increases by 9.37% after RandomUnderSampler.</p><p>The common use of these resampling methods is to combine them in a pipeline. It is not recommended to use just one of these in big datasets, which is an important difference between the majority and minority class.</p><p><strong>Pipeline</strong></p><p>As mentioned above, it is not recommended to apply only oversampling or undersampling methods to big data which has a significant difference between classes. We have an extra option, we can apply oversampling and undersampling methods simultaneously with the pipeline. We will combine these two methods with adjusting the sampling strategy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/989/1*wIBzytxldAPEzv5k0ICz1A.png" /></figure><p>We create a pipeline with imblearn.pipeline, give the steps (over and under) sequentially. RandomOverSampler with a 0.1 sampling strategy raises the minority class to “0.1 * majority class”. Next, RandomUnderSampler with a 0.5 sampling strategy reduces the majority class quantity to “2 * minority class”. At the end of the pipeline, the ratio between minority and majority classes will be 0.5.</p><pre>from imblearn.over_sampling import RandomOverSampler<br>from imblearn.under_sampling import RandomUnderSampler</pre><pre>over = RandomOverSampler(sampling_strategy=0.1)<br>under = RandomUnderSampler(sampling_strategy=0.5)</pre><pre>from imblearn.pipeline import Pipeline</pre><pre>steps = [(&#39;o&#39;, over), (&#39;u&#39;, under)]<br>pipeline = Pipeline(steps=steps)</pre><pre>X_smote, y_smote = pipeline.fit_resample(X_train, y_train)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/671/1*Vlrd-kJKl7J-mNnKpYqiZg.png" /></figure><p>After Logistic Regression, the metrics of the data are above. After the pipeline, the score increased by 11.83%. We get the best score with Pipeline.</p><p><strong>Conclusion</strong></p><p>We should note that we only apply these methods to training data. We just balance our training data, our test data remain as is (original distribution).</p><p>There are other techniques and algorithms in the imblearn library, check this <a href="https://imbalanced-learn.org/stable/user_guide.html">library</a> to solve imbalanced dataset problems.</p><p>We should use these techniques carefully because they manipulate data.</p><p><strong>References</strong></p><ul><li><a href="https://imbalanced-learn.org/stable/user_guide.html">User guide: contents - Version 0.12.3</a></li><li><a href="https://github.com/scikit-learn-contrib/imbalanced-learn">GitHub - scikit-learn-contrib/imbalanced-learn: A Python Package to Tackle the Curse of Imbalanced Datasets in Machine Learning</a></li><li><a href="https://github.com/hersany/Datasets/tree/master/travel_insurance">hersany/Datasets</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1fbba4a0e6d3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Feature Scaling with Scikit-Learn for Data Science]]></title>
            <link>https://hersanyagci.medium.com/feature-scaling-with-scikit-learn-for-data-science-8c4cbcf2daff?source=rss-7c248ad78b88------2</link>
            <guid isPermaLink="false">https://medium.com/p/8c4cbcf2daff</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[scaling]]></category>
            <category><![CDATA[scikit-learn]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <dc:creator><![CDATA[Hasan Ersan YAĞCI]]></dc:creator>
            <pubDate>Mon, 22 Feb 2021 21:16:25 GMT</pubDate>
            <atom:updated>2021-02-22T22:28:50.863Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/379/1*qRkOjYmADydP8qojPv6iEg.png" /><figcaption>Do you really want to this happen?</figcaption></figure><p>In the data science process, we need to do some preprocessing before machine learning algorithms. These can be some basic data analysis processes such as handling missing values and outliers and data cleaning. We also apply scaling (data transformation) for some data.</p><p>Scaling is not mandatory, but it performs better to scale the data before some machine learning algorithms.</p><p>The main purpose of scaling is to avoid the effects of greater numeric ranges. Especially it is so important to machine learning algorithms which the distance is important, such as KNN (k Nearest Neighbor), K-Means Clustering, SVM (Support Vector Machine), PCA (Principal Component Analysis) algorithms.</p><p>In the scatter plots below, you can check the difference between scaling and non-scaling. Scaling was used in the PCA process. The range(scale) is very wide without scaling so it is difficult to separate points. When scaling is used, it provides efficiency and good performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/745/1*BdWfAJd1N7eJFjcYZNAcIQ.png" /></figure><p>As stated before, the purpose of scaling is to bring each data in the dataset closer together. The other goal is to avoid some types of numerical difficulties during the calculation. For example, Gradient Descent based optimization algorithm likes to scale and performs better with scaling. Linear Regression and Logistic Regression, common algorithms using Gradient Descent, can be scaled too. We can also apply scaling for other machine learning algorithms and Neural Networks. We can achieve better performance and better results. We can compare results with scaling and without scaling.</p><p>Some algorithms like Decision Tree, Random Forest, Adaboost do not need to scale.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*XM86Qdk5C5GSKM-W.png" /><figcaption><a href="https://stackoverflow.com/questions/46686924/why-scaling-data-is-very-important-in-neural-networklstm/46688787#46688787">Reference</a></figcaption></figure><p>We will focus on the most common; StandardScaler, RobustScaler, MinMaxScaler, and MaxAbsScaler.</p><p><strong>1 — StandardScaler</strong></p><pre>from sklearn.preprocessing import StandardScaler</pre><blockquote><a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html#sklearn.preprocessing.StandardScaler">Standardize features by removing the mean and scaling to unit variance</a>.</blockquote><p>StandardScaler is a mean-based scaling method. The formula of StandardScaler is <strong>(Xi-Xmean)/Xstd,</strong> so it adjusts the mean as a 0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/466/0*WrXe0uhKc-Yb4H8G.jpg" /><figcaption><a href="https://www.geeksforgeeks.org/ml-feature-scaling-part-2/">Reference</a></figcaption></figure><p>The StandardScaler is vulnerable to outliers because outliers affect the mean. If you have a normal distribution or have near-normal data, StandardScaler moves your data closer to the standard normal distribution. However, StandardScaler will not perform well if you have outliers.</p><p>Let’s check the StandardScaler on charts;</p><pre>from sklearn.preprocessing import StandardScaler</pre><pre>df_scale = StandardScaler().fit_transform(df)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/1*1C-7C_EZObVL3PgmI9iIxA.png" /></figure><p>We have right-skewed data. After StandardScaler implementation our data are clustered around 0, but it is still right-skewed and has a large scale (-1 and 30). The data are clustered and do not have large outliers. If we have larger values as outliers, we cannot get good results with StandardScaler. Because outliers will affect all scaling data. It also affected this data. In such cases, it is better to remove outliers firstly. For now, we will use log transformation. You can check this <a href="https://hersanyagci.medium.com/detecting-and-handling-outliers-with-pandas-7adbfcd5cad8">link</a> for the methods of dealing with outliers.</p><pre>from sklearn.preprocessing import StandardScaler<br>import numpy as np</pre><pre>df_log = np.log(df)</pre><pre>df_scale = StandardScaler().fit_transform(df_log)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/1*u0WRXf0FHmcrlmZf800enA.png" /></figure><p>The log transformation changes data distribution to the normal distribution. After the log transformation, the StandardScaler implementation makes our data standard normal distribution (near standard normal distribution). This method performs better and removes the effects of outliers. It is better to apply StandardScaler to data that are not outliers.</p><p><strong>2 — RobustScaler</strong></p><pre>from sklearn.preprocessing import RobustScaler</pre><blockquote><a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.RobustScaler.html#sklearn.preprocessing.RobustScaler">This Scaler removes the median and scales the data according to the quantile range (defaults to IQR: Interquartile Range). The IQR is the range between the 1st quartile (25th quantile) and the 3rd quartile (75th quantile).</a></blockquote><p>RobustScaler is a median-based scaling method. The formula of RobustScaler is <strong>(Xi-Xmedian) / Xiqr</strong>, so it is not affected by outliers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*-FQhpFiURH9dQA6z" /><figcaption><a href="https://seleritysas.com/blog/2020/12/18/techniques-of-feature-scaling-with-sas-custom-macro/">Reference</a></figcaption></figure><p>Since it uses the interquartile range, it absorbs the effects of outliers while scaling. The interquartile range (Q3 — Q1) has half the data point. If you have outliers that might affect your results or statistics and don’t want to remove them, RobustScaler is the best choice.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/882/1*8xcyqQ9XCwDFgK0bycteKQ.png" /></figure><p>Above, the original data (without scale) have outliers at points 250000 and 200000, after RobustScaler they change to 150 and 200, but StandardScaler makes them 25 and 35. Outliers affect the StandardScaler and get too close to the mean point. RobustScaler keeps outliers where they should be away from the data mean.</p><p><strong>3 — MinMaxScaler</strong></p><pre>from sklearn.preprocessing import MinMaxScaler</pre><blockquote><a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html#sklearn.preprocessing.MinMaxScaler">This estimator scales and translates each feature individually such that it is in the given range on the training set, e.g. between zero and one.</a></blockquote><p>The formula of MinMaxScaler is <strong>(Xi-Xmin) / (Xmax-Xmin)</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*jLXgL0vdAo43ua0P.jpg" /><figcaption><a href="https://www.geeksforgeeks.org/ml-feature-scaling-part-2/">Reference</a></figcaption></figure><p>MinMaxScaler sets data from 0 to 1. Scaling is calculated with minimum and maximum points, therefore data can be affected by outliers. It is not suitable for outliers. Before implementing MinMaxScaler, it is good to handle outliers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/1*AOQtkCREyClDv0-CPS8QEQ.png" /></figure><p><strong>4— MaxAbsScaler</strong></p><pre>from sklearn.preprocessing import MaxAbsScaler</pre><blockquote><a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MaxAbsScaler.html#sklearn.preprocessing.MaxAbsScaler">This estimator scales and translates each feature individually such that the maximal absolute value of each feature in the training set will be 1.0. It does not shift/center the data, and thus does not destroy any sparsity</a>.</blockquote><p>The formula of MaxAbsScaler is Xi /|Xmax|. If the data has negative values, MaxAbsScaler sets the data between -1 and 1. It scales data according to the absolute maximum, so it is not suitable for outliers. It needs pre-processing like handling with outliers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/1*zQe_gog8aC0cMGMxoEMT9Q.png" /></figure><p><strong>Conclusion</strong></p><p>The principal purpose of scaling is to restate each variable on a standardized scale and to avoid numerical instabilities potentially caused by too large numbers, to speed up optimization algorithms. And the most significant issue about scaling is that be sure that apply scaling both of training and test data. We want to scale all data to the same scale.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*sc3itKWQzIUgbgxg.png" /><figcaption><a href="https://medium.com/lets-talk-ml/scaling-the-data-for-machine-learning-standardisation-and-normalisation-26ae02b1d101">Reference</a></figcaption></figure><p><strong>StandardScaler</strong></p><ul><li>It is not suitable for data that have outliers.</li><li>It adjusts the mean to 0.</li></ul><p><strong>RobustScaler</strong></p><ul><li>It is more suitable for data that have outliers.</li></ul><p><strong>MinMaxScaler</strong></p><ul><li>It is not suitable for data that have outliers.</li><li>It adjusts the data between 0 and 1.</li></ul><p><strong>MaxAbsScaler</strong></p><ul><li>It is not suitable for data that have outliers.</li><li>It adjusts the data between -1 and 1.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8c4cbcf2daff" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Model Deployment with Streamlit on AWS EC2]]></title>
            <link>https://hersanyagci.medium.com/model-deployment-with-streamlit-on-aws-ec2-f52ab42d7813?source=rss-7c248ad78b88------2</link>
            <guid isPermaLink="false">https://medium.com/p/f52ab42d7813</guid>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[web-app-development]]></category>
            <category><![CDATA[streamlit]]></category>
            <dc:creator><![CDATA[Hasan Ersan YAĞCI]]></dc:creator>
            <pubDate>Fri, 12 Feb 2021 11:14:52 GMT</pubDate>
            <atom:updated>2021-02-27T07:51:09.902Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*m3_ROPZEL0o3EYaU.jpg" /><figcaption><a href="https://www.youtube.com/watch?v=Klqn--Mu2pE&amp;t=189s">Tutorial Series</a></figcaption></figure><p>We have already touched on the importance of model deployment and sharing this model with others. We need to share our model with stakeholders to collaborate or get feedback. Therefore, we need a web app with powerful and interactive content for our colleagues or clients for who we want to showcase our work. Streamlit is the most practical and fastest way to create a web app. You can browse the below to learn more about Streamlit and see simple model deployment examples.</p><p><a href="https://hersanyagci.medium.com/introduction-to-streamlit-for-machine-learning-web-app-cd89c4181c33">Introduction to Streamlit for Machine Learning Web App</a></p><p>In this tutorial, we will create a virtual environment. In this virtual environment, we will prepare a python file for the web app and a requirements file for the list of libraries. Then we will import the model (trained before in the same virtual environment), python and requirements files to Github. We will launch a virtual machine (EC2) on AWS and pull the files and model from Github. Then we will build our web app with these files on EC2. Our web app will run on AWS EC2 even though we shut down our computer.</p><p>We will use the car price prediction model for this tutorial. It is not real data so the prices can be different and some features may not affect the price as expect as normal. You can download the model from this <a href="https://github.com/NavarraB/Streamlit.git">link</a>.</p><p>We will go step by step. At the end of the steps, we will reach the web app that estimates the car prices according to the user’s choices, and we will be able to cooperate and get feedback by sharing this link with others.</p><p><strong>1 — Creating Virtual Environment</strong></p><p>First of all, we need a virtual environment. We can think of the virtual environment as an isolated space. Libraries we use when training a model or creating a Streamlit python file are kept in this virtual environment with their current versions, and updates to the libraries will not affect our model or files when we run them later in this virtual environment. We can keep a list of libraries and if we want to run our works in other environments or platforms, we can install versions of the libraries on the list. When working on AWS EC2 we will need this list of libraries called the requirements.txt file.</p><p>Let’s create a virtual environment;</p><p>I am an ubuntu user, this process is similar to me for MacOs and Linux distributions. You can follow me or you can check this <a href="https://www.geeksforgeeks.org/creating-python-virtual-environment-windows-linux/">link</a> for windows and other details.</p><p>First of all, we need the <a href="https://docs.python.org/3/library/venv.html">venv</a> module on the computer;</p><pre>sudo apt install python3-venv</pre><p>After installing the venv module, we can easily create our environment. I will create an environment named web_app_env.</p><pre>python3 -m venv web_app_env</pre><p>And activate it;</p><pre>source web_app_env/bin/activate</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*7byL2ywgNG0IwE7qa1ASnw.png" /></figure><p>Our virtual environment is ready to use. We can understand that we are in a virtual environment from the brackets in front of our username.</p><p><strong>2 — Python File</strong></p><p>We need a python file (py extension) for an app. We will write the codes on this file for a web app. We will build our app on this file. It can be created and edited with a text editor but requires a python interpreter to run. py files are often used for programming web servers and other administrative computer systems.</p><p>First, we will create a python file, I will do this on the terminal. I go to the relevant directory where my model is located (we will use the car price prediction model trained before with random forest regression.) I create a py file with <em>touch my_first_app.py.</em></p><p>If you don’t have Streamlit, you need to install Streamlit. You just need to type <em>pip install streamlit</em> into your terminal and it is ready to use. (If you don’t have pip, <a href="https://pip.pypa.io/en/stable/installing/">install pip first.</a>)</p><pre>pip install streamlit</pre><p>Then type <em>streamlit run my_first_app.py</em> in the relevant directory in the terminal and run it. Python file works even if it is empty and gives you a URL Our Streamlit web app is ready, now we can fill this file.</p><pre>streamlit run my_first_app.py</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*B-S0EVA42eGEuBFiUc334A.png" /></figure><p>Click on the URL (it doesn’t matter which one), this link will open in your browser.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gP18QqvuxrKhlpmi.png" /></figure><p>When we write something to the Python file (my_first_app.py) and every time we save the file, the web app will change at the same time. You can save the file (ctrl + s or command + s) and the web app asks you “<strong><em>Stream </em>file changed rerun or rerun always.</strong>”. Click the <strong>rerun always</strong> so it doesn’t warn us every time and also changes the view.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LWRTpk_T8eak3iVV.png" /></figure><p>First, we will import the libraries and model and putting a nice title.</p><pre>import streamlit as st<br>import pickle<br>import pandas as pd</pre><pre>html_temp = &quot;&quot;&quot;<br>&lt;div style=&quot;background-color:tomato;padding:10px&quot;&gt;<br>&lt;h1 style=&quot;color:white;text-align:center;&quot;&gt;Car Price Prediction &lt;/h1&gt;<br>&lt;/div&gt;&quot;&quot;&quot;<br>st.markdown(html_temp,unsafe_allow_html=True)</pre><pre>model = pickle.load(open(&quot;my_model&quot;, &quot;rb&quot;))</pre><p>We will now add some interactivity to our web app. We will put slider, selection box and drop down menu for user choices. So user can select values and control the price based on their input.</p><p>Our original dataset is as follows. There are 7 columns outside of our target column (price). That’s why we will add 7 interactive boxes/sliders to our app.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*aFate-sQZ_E-fIRVVkoo-g.png" /></figure><p>We will put sliders for hp and km columns.</p><p>We will put in selection boxes for the make_model, body_type, Body Color, and Gearing Type columns. We will add all the unique values found in these columns as a selection in these selection boxes.</p><p>We will put a multi-selection box for the Extras column. Because there are a total of 16 unique features in the entire dataframe in the Extras column, and each car has a different number of features. Some cars can have one while some have several.</p><p>The user will select the desired features and the price will be shown.</p><pre>hp=st.slider(&quot;What is the hp of your car?&quot;, 60, 200, step=5)<br>km=st.slider(&quot;What is the km of your car&quot;, 0,100000, step=500)<br>car_model=st.selectbox(&quot;Select model of your car&quot;, (&#39;Audi A1&#39;, &#39;Audi A2&#39;, &#39;Audi A3&#39;))<br>body_type=st.selectbox(&quot;Select body type of your car&quot;, (&#39;Convertible&#39;, &#39;Coupe&#39;, &#39;Off-Road&#39;, &#39;Other&#39;, &#39;Sedans&#39;, &#39;Station wagon&#39;, &#39;Compact&#39;))<br>color=st.selectbox(&quot;What is the color of your car&quot;, (&#39;Black&#39;, &#39;Red&#39;, &#39;Brown&#39;,&#39;White&#39;,&#39;Grey&#39;,&#39;Blue&#39;,&#39;Silver&#39;,&#39;Beige&#39;,&#39;Violet&#39;,&#39;Yellow&#39;,&#39;Green&#39;,&#39;Bronze&#39;,&#39;Orange&#39;))<br>gear=st.selectbox(&quot;What is the gearing type of your car&quot;, (&#39;Automatic&#39;, &#39;Manual&#39;, &#39;Semi-automatic&#39;))<br>extras=st.multiselect(&quot;Select extras for your car&quot;, (&#39;Alloy wheels&#39;, &#39;Cab or rented Car&#39;, &#39;Catalytic Converter&#39;,&#39;Handicapped enabled&#39;,&#39;Right hand drive&#39;,&#39;Roof rack&#39;,&#39;Shift paddles&#39;,&#39;Ski bag&#39;,&#39;Sport package&#39;,&#39;Sport seats&#39;,&#39;Sport suspension&#39;,&#39;Touch screen&#39;,&#39;Trailer hitch&#39;,&#39;Tuned car&#39;,&#39;Voice Control&#39;,&#39;Winter tyres&#39;))</pre><p>Now we will convert our dataframe into model structure.</p><p>First, we will create a dataframe with no price column, then add all unique values for categorical columns as column names. We will implement one hot encoding to the dataframe. Because we will get categorical values from users and this python file will work as an interface between the model and the web app.</p><p>For one hot encoding, we will use the ‘Pandas get_dummies’ function. We will use the ‘Fill_value = 0’ parameter. That is, the selected value/column will be 1, the others will be 0. We use str.get_dummies for the Extras column because there are multiple selections. Check this <a href="https://hersanyagci.medium.com/encoding-with-pandas-get-dummies-2f95709adfd">article</a> for details on get_dummies.</p><pre>my_dict = {<br>    &quot;hp&quot;: hp,<br>    &quot;km&quot;: km,<br>    &quot;make_model&quot;: car_model,<br>    &quot;body_type&quot;: body_type,<br>    &quot;Body Color&quot;: color,<br>    &quot;Gearing Type&quot;: gear,<br>    &quot;Extras&quot;:extras<br>}</pre><pre>df = pd.DataFrame.from_dict([my_dict])</pre><pre>columns=[&#39;hp&#39;,<br>         &#39;km&#39;,<br> &#39;make_model_A1&#39;, &#39;make_model_A2&#39;, &#39;make_model_A3&#39;,<br> &#39;body_type_Convertible&#39;, &#39;body_type_Coupe&#39;, &#39;body_type_Off-Road&#39;, &#39;body_type_Other&#39;, &#39;body_type_Coupe&#39;, &#39;body_type_Sedans&#39;,<br> &#39;body_type_Station wagon&#39;, &#39;body_type_Compact&#39;<br> &#39;Body Color_Black&#39;, &#39;Body Color_Blue&#39;, &#39;Body Color_Bronze&#39;, &#39;Body Color_Brown&#39;, &#39;Body Color_Green&#39;, &#39;Body Color_Grey&#39;, &#39;Body Color_Orange&#39;, &#39;Body Color_Red&#39;, &#39;Body Color_Silver&#39;, &#39;Body Color_Violet&#39;, &#39;Body Color_White&#39;, &#39;Body Color_Yellow&#39;, &#39;Body Color_Beige&#39;,<br> &#39;Gearing Type_Manual&#39;, &#39;Gearing Type_Semi-automatic&#39;, &#39;Gearing Type_Automatic&#39;]</pre><pre>df1 = df.drop([&#39;Extras&#39;], axis = 1)</pre><pre>df1 = pd.get_dummies(df1).reindex(columns=columns, fill_value=0)</pre><pre>df_extra = df[&#39;Extras&#39;].str.get_dummies(&#39;,&#39;)</pre><pre>df = df1.join(df_extra)</pre><p>In the last part of the file, we will make prediction and give an output.</p><pre>prediction = model.predict(df)</pre><pre>st.success(&quot;The estimated price of your car is €{}. &quot;.format(int(prediction[0])))</pre><p>The latest version of the my_first_app.py file will be as follows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*fD241-bse34zYD4OM9Za0A.png" /></figure><p>Our web app and python file are ready.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/869/1*oa1mbqo3SPlNb4B_bXNG3g.png" /></figure><p><strong>3 — requirements.txt</strong></p><p>We save the libraries we use when training the model and creating the Python file, along with their versions, in a requirements.txt file. If we want to run our works in another environment or platform, we can install versions of the libraries on the list. When working on AWS EC2, we will use the requirements.txt file.</p><p>It is very easy to create the requirements.txt file, we will write <em>pip freeze&gt; requirements.txt</em> in the terminal.</p><pre>pip freeze &gt; requirements.txt</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/1*k64KExikrqoB-THxtr3Mhw.png" /></figure><p>We created this in the web_app_env virtual environment, so there are only a few libraries.</p><p>Finally, I have pushed <strong>my_model, my_first_app.py, and requirements.txt</strong> file to Github for later use on AWS EC2. We can now start our virtual machine on AWS.</p><p><strong>4 — Run the EC2 on AWS</strong></p><p>If you do not have an account on AWS, you can create an AWS free tier from this <a href="https://aws.amazon.com/free">link</a>.</p><p>If you have an account on AWS, follow it,</p><p>Open your AWS Management Console and select <strong>Launch a virtual machine</strong>;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_9ELlOMq8-vJDb8xPmzjQA.png" /></figure><p>Then choose <strong>Ubuntu Server 20.04 LTS</strong> from the AMI list for our virtual machine;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1SPxLXcT7xcPKeR-Hi5pPg.png" /></figure><p>Go directly to step 6, leaving steps 2, 3, 4, and 5 at the default values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NdSIQIMddrW07ArrO7RYQQ.png" /></figure><p>In step 6, in the Configure Security Group section, we will add the port range for Streamlit. For <strong>TCP Protocols</strong>, we will add a <strong>port range of 8501, 8502</strong> and <strong>Source</strong> will be <strong>Anywhere</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GQ-XWDWAKLSrCs3fA74bKw.png" /></figure><p>In step 7, create a <strong>key pair</strong> if you don’t have a <strong>key pair.</strong> Then select it and <strong>Launch Instances.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_TNjwYGFA3gSBjuTKQh4KQ.png" /></figure><p>Our EC2 is ready, then go to the instances page and select the relevant instance. Then click the <strong>connect,</strong> it will give you <strong>code snippets</strong> to connect the virtual machine. Take these and paste them into your terminal to connect to EC2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RirnJQFUasgD2BCFvt56Qg.png" /></figure><p>After connecting to the virtual machine, type the following commands to install the latest updates, python3, pip, and git.</p><pre>sudo apt-get update -y<br>sudo apt-get install python3-pip -y<br>sudo apt-get install git -y</pre><p>Then we will clone the python file we prepared, the model, and the requirements.txt files we created.</p><pre>git clone <a href="https://github.com/NavarraB/Streamlit.git">https://github.com/NavarraB/Streamlit.git</a></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DpPZDbMwhDt7MjT34JT_Vg.png" /></figure><p>The files we need are in the Car_Price directory in the Streamlit repository. So we will go to this directory and install the libraries with requirements.txt.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*gflUiXQvIC_zhmxw5dbKdg.png" /></figure><pre>sudo pip3 install -r requirements.txt</pre><p>We installed the libraries for the web app. You can control what you have;</p><pre>sudo pip3 freeze</pre><p>Finally, we can run the web app;</p><pre>streamlit run my_first_app.py &amp;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/774/1*6xlEgDVE61eOMaEssbhqXA.png" /></figure><p>With ampersand (&amp;), our web app will run in AWS EC2 even if we turn off our computer. You can exit AWS EC2 with ctrl + d or command + d and test whether your web app is working. Just let EC2 run.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fmGiZOMxzQpaQBGF_vCYrA.png" /></figure><p>Now that the app is deployed you can easily share it and collaborate on it.</p><p>You can build a machine learning web app for your model and show people how is it works, about its predictions, about its accuracy. You can share your link and get feedback and collaborate with others. Your web app link will remain active as long as you do not stop EC2.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f52ab42d7813" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Streamlit for Machine Learning Web App]]></title>
            <link>https://hersanyagci.medium.com/introduction-to-streamlit-for-machine-learning-web-app-cd89c4181c33?source=rss-7c248ad78b88------2</link>
            <guid isPermaLink="false">https://medium.com/p/cd89c4181c33</guid>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[streamlit]]></category>
            <category><![CDATA[ml-model-deployment]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Hasan Ersan YAĞCI]]></dc:creator>
            <pubDate>Fri, 05 Feb 2021 10:47:18 GMT</pubDate>
            <atom:updated>2021-02-05T10:47:18.908Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7mUI9yTv9TUXCco3" /></figure><blockquote><a href="https://streamlit.io/">Streamlit</a> is an open-source Python library that makes it easy to build beautiful custom web-apps for machine learning and data science.</blockquote><p>Data scientists run the data science process to arrive at a solution, creativity, and a model. They create a product and at the end of this process, they have to share their product with their stakeholders in order to collaborate or get feedback. Stakeholders can be customers, colleagues, or anyone else, anyone located in another city or perhaps another country.</p><p>To share the model, it is necessary to deploy it over the web. Deploying the model is the most important part, you can build a good model and accomplish the successful data science process. However, you should show this work to others in a beautiful presentation.</p><p>With Streamlit, you can create your own interactive environment, you can create a web app with powerful content for your colleagues or customers you want to show your work. You can get values from users and change the result interactively according to their inputs. Streamlit is an absolute tool that brings your idea to the internet. It is very easy to install and create a web app with Streamlit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*jKavjeukIU66AT-D.gif" /><figcaption><a href="https://www.streamlit.io/gallery">https://www.streamlit.io/gallery</a></figcaption></figure><p>In Streamlit, you mainly use markdown and python. HTML and CCS are also supported. You can create a good-looking visual web app. You can design your app, deploy your model, and someone can test your model.</p><p><strong>Installing the Streamlit</strong></p><p>You just need to type <em>pip install streamlit</em> into your terminal and it is ready to use. (If you don’t have pip, <a href="https://pip.pypa.io/en/stable/installing/">install pip first.</a>)</p><pre>pip install streamlit</pre><p>Then we just need a python file (py extension) for an app. We will write the codes on this file for a web app. We will build our app on this file. It can be created and edited with a text editor but requires a python interpreter to run. py files are often used for programming web servers and other administrative computer systems.</p><p>Let’s create our first demo app;</p><p>First, we will create a python file, I will do this on the terminal. I go to the relevant directory where my model is located (if we want to deploy a model, we have to do it in the relevant directory. This is not required for this demo.). I create a py file with <em>touch demo_app.py.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*G9Y6L7bDq96VIUFyXUtEXA.png" /></figure><p>Then type <em>streamlit run demo_app.py</em> in the relevant directory in the terminal and run it. Python file works even if it is empty and gives you a URL. Our Streamlit web app is ready, it’s all that easy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/1*DWERKWTDunz4Fr9DER9bJQ.png" /></figure><p>Click on the URL (it doesn’t matter which one), this link will open in your browser.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/951/1*bC2lS83DeJzuetghqvLf0Q.png" /></figure><p>We have a web app. We can fill this now. I will open the demo_app.py, in Jupyter Notebook to overwrite.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gX5kAfr-yU-JxsLVFqmXMA.png" /></figure><p>We will write something in the python file (demo_app.py) and the web app will change at the same time each time we save the file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ipSN30Ahzobly5_eI1C6Gg.png" /></figure><p>First, we need to import Streamlit. Then we write our first code <em>st.title (“Web App”)</em> and save the file (ctrl + s or command + s) and the web app asks you “<strong><em>Stream </em>file changed rerun or rerun always.</strong>”. Click the <strong>rerun always</strong> so it doesn’t warn us every time and also changes the view.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pCVEgEmil5MCL-qHrPWAhA.png" /></figure><p>Now we can add some interactivity and creativity to your web app. Below are some examples.</p><p>We can write title and text;</p><pre>st.title(&#39;Web App&#39;)<br>st.text(&#39;Hello Streamlit&#39;)</pre><p>We can write headers like that;</p><pre>st.header(&#39;This is a header&#39;)<br>st.subheader(&#39;This is a subheader&#39;)</pre><p>We can use markdown and its features on streamlit;</p><pre>st.markdown(&#39;This is a normal Markdown&#39;)<br>st.markdown(&#39;# This is a bold Markdown&#39;)<br>st.markdown(&#39;## This is a thin-bold Markdown&#39;)<br>st.markdown(&#39;* This is a Markdown with point&#39;)<br>st.markdown(&#39;** This is a small bold Markdown **&#39;)</pre><p>We can make colorful our text;</p><pre>st.success(&#39;Successful&#39;)<br>st.markdown(&#39;`This is a markdown`&#39;)<br>st.info(&quot;This is an information&quot;)<br>st.warning(&#39;This is a warning&#39;)<br>st.error(&#39;This is an error&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/845/1*nx3gcjj6Q120jDR6cciBiQ.png" /></figure><p>After these codes we get;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/942/1*qrvFPB_8lPAQoe6dBmmiAg.png" /></figure><p>We can add many interactive elements such as buttons, sliders, selection boxes to select or reference values.</p><p>We can add single-select box. It will make dropdown menu;</p><pre>st.selectbox(&#39;Your Occupation&#39;, [&#39;Programmer&#39;, &#39;Data Scientist&#39;])</pre><p>We can add multi-select box. It will make dropdown menu;</p><pre>st.multiselect(&#39;Where do you work&#39;, (&#39;London&#39;,&#39;Istanbul&#39;,&#39;Berlin&#39;))</pre><p>We can add buttons,</p><pre>st.button(&#39;Simple Button&#39;)</pre><p>We can add sliders and some features,</p><pre>st.slider(&#39;What is your level&#39;, 0,40, step = 5)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/853/1*yxng6gWXRJ5IMX8ODy7Ofg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/1*pPW-0JehmqcwYj_3tVi1EQ.png" /></figure><p>As you can see there is no need for HTML or CSS but you can add some extra features if you want.</p><pre>html_temp = &quot;&quot;&quot;<br>&lt;div style=&quot;background-color:tomato;padding:1.5px&quot;&gt;<br>&lt;h1 style=&quot;color:white;text-align:center;&quot;&gt;Demo Web App &lt;/h1&gt;<br>&lt;/div&gt;&lt;br&gt;&quot;&quot;&quot;<br>st.markdown(html_temp,unsafe_allow_html=True)<br></pre><pre>st.title(&#39;This is for a good design&#39;)<br>st.markdown(&#39;&lt;style&gt;h1{color: red;}&lt;/style&gt;&#39;, unsafe_allow_html=True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*jcAvKWSjcgSRYibr3eeZQg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*JDODcBMlzuygZ_rqgHwPOA.png" /></figure><p>We can also add images and graphics;</p><pre>#images<br>from PIL import Image<br>im = Image.open(&quot;winter.jpg&quot;)<br>st.image(im, width=700, caption=&quot;Winter&quot;)</pre><pre>#Graphs<br>import matplotlib.pyplot as plt<br>import numpy as np</pre><pre>arr = np.random.normal(1, 1, size=100)<br>fig, ax = plt.subplots()<br>ax.hist(arr, bins=20)</pre><pre>st.pyplot(fig)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*nlC5yE4gWPSQRTXV4Cs2fQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*NSjVGMY_qcixXxU2vMxyHw.png" /></figure><p>Python libraries are supported. You may get this error message if there is no library on your computer, then you just need to install that library on the computer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/931/1*RqoxxeU0HSvxJjEsIXFgew.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/1*ve-5TxVGNIubOFXX3R4KyQ.png" /></figure><p>There are many items in Streamlit that help you build a good web app. You can add many features depending on your creativity and needs. Videos, sounds, emojis, charts, graphics, etc. It is the fastest and most practical way to create a web app for machine learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*0Jui13L-_9aqamoA.gif" /><figcaption><a href="https://www.streamlit.io/gallery">https://www.streamlit.io/gallery</a></figcaption></figure><p>Just go to <a href="https://docs.streamlit.io/en/stable/">Streamlit web page</a> and check the <a href="https://docs.streamlit.io/_/downloads/en/stable/pdf/">documentation</a>, get your needs.</p><p>In our next post, we will deploy a model in AWS EC2.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cd89c4181c33" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Encoding with Pandas get_dummies]]></title>
            <link>https://hersanyagci.medium.com/encoding-with-pandas-get-dummies-2f95709adfd?source=rss-7c248ad78b88------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f95709adfd</guid>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[pandas]]></category>
            <dc:creator><![CDATA[Hasan Ersan YAĞCI]]></dc:creator>
            <pubDate>Sun, 24 Jan 2021 13:46:35 GMT</pubDate>
            <atom:updated>2021-01-24T13:49:35.259Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dhOQhd1V5YjuepdQ.jpg" /></figure><p>We previously covered the issue of encoding and its importance. In short, machine learning models are mathematical models that use algorithms that work with numerical data types, and neural networks also work with numerical data types. Therefore, we need encoding methods to convert non-numerical data to meaningful numerical data. We have covered the encoding methods and the options that we can apply these encoding methods at this <a href="https://hersanyagci.medium.com/label-encoding-vs-one-hot-encoding-e8a10916da5b">link</a>.</p><p><a href="https://hersanyagci.medium.com/label-encoding-vs-one-hot-encoding-e8a10916da5b">Label Encoding vs One Hot Encoding</a></p><p>In this story, we will look at the Pandas get_dummies method. Pandas get_dummies is the easiest way to implement one hot encoding method and it has very useful parameters, of which we will mention the most important ones. With get_dummies we can get a hot encoder data frame (dummy variables) in one row.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MgMIDllcHOeR9G9K.png" /><figcaption>Two Methods of One Hot Encoding</figcaption></figure><p>We will use <a href="https://github.com/NavarraB/DataScience/tree/master/Other">car_price</a> data for this demo. You can download data and notebook from this <a href="https://github.com/NavarraB/DataScience/tree/master/Other">link</a>.</p><p>The purpose of this data science process is to predict the prices of cars. We will use Linear Regression for this data, but the data is not ready for the machine learning model. Because there are categorical (non-numerical) columns and we need to transform them. For this, we will implement get_dummies.</p><p>Before diving into Get_dummies, let’s check the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/845/1*K1ADSId0RnjqEqmkSzu7kA.png" /></figure><p>As you can see, the data has no missing values and I acquired this structure after handling outliers. You can check <a href="https://hersanyagci.medium.com/detecting-and-handling-outliers-with-pandas-7adbfcd5cad8">this link</a> to see how to detect and handle outliers with pandas.</p><p>We have 5 categorical columns that we need to make numerical before machine learning algorithm. These columns are; “make_model”, “body_type”, “Body Color”, “Gear Type” and “Extras”, all of them are nominal data. For example, for “Body Color” there is no hierarchy between colors. Black has no superiority over red. That’s why we have to use one hot encoder.</p><p>Firstly we will just apply get_dummies to the “Body Color” column to see details of get_dummies, then we will use all dataframe;</p><p><strong>1 — get_dummies()</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/887/1*VTaePI9Zjw9VBh_dcGjYKA.png" /></figure><p>The “Body Color” column has 13 unique values, which means we will get 13 columns after applying get_dummies.</p><p>With this syntax we can apply get_dummies to a column of dataframe;</p><pre>pd.get_dummies(df[&#39;Body Color&#39;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*3xdEj6ejMKfNW_ueEXX0Uw.png" /></figure><p>We didn’t use any parameters, get_dummies has default parameters. As you can see we got a 13 column dataframe after get_dummies. There are 4800 rows. We can see the color of the car. For example; the first car color black, the second car color red, etc.</p><p>Our original data frame, df, keeps its shape. We must merge these dataframes. Now we can assign “Body Color”s dummy variables (one hot encoder) to a new dataframe to merge with the main dataframe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/876/1*HoTux_kKfdrR7CybbqapSg.png" /></figure><p>We will now merge them into data frames using the join method. But we should drop the “Body Color” column, we don’t need it anymore because it’s not numerical.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/839/1*xjxKEDe-LMlLnr17hzvOxg.png" /></figure><p>With this syntax, we dropped the “Body Color” column and added our dummy variables. As you can see, all colors have a column and column types are numerical.</p><p><strong>2 — get_dummies with ‘drop_first’ parameter</strong></p><p>One column transformed into 13 columns. We can use the ‘drop_first’ parameter and decrease one column. We can take 12 columns. Normally the default value of this parameter is ‘False’, we just set it to ‘True’. Let’s see how it works.</p><pre>pd.get_dummies(df[&#39;Body Color&#39;], drop_first = True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*OFuL_yf7-9dK44PyJYTmTw.png" /></figure><p>Check the number of columns, instead of 13 we got 12 columns.</p><p>It removes the first column of the get_dummies dataframe. The first column for the “Body Color” column is Beige. If there is a beige car, all columns are 0. When all columns are 0, the model knows it’s a beige car. Check out the example below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/813/1*3QHZzeZmr2ao4fcHPnaYXA.png" /></figure><p>More columns mean less performance and more training time. Imagine we have 20 columns that are not numerical. If we use ‘drop_first’, we get 20 columns less. So it is useful to use the drop_first parameter for model performance.</p><p><strong>3 — get_dummies with ‘prefix’ parameter</strong></p><p>If the dataframe had the “Upholstery Color” column, we would also get a black or a brown column for the tile color after get_dummies except “Body Color”. Multiple columns with the same name can cause problems. We can use the ‘prefix’ parameter to avoid this situation.</p><pre>pd.get_dummies(df[&#39;Body Color&#39;], drop_first = True, prefix = &#39;BC&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/923/1*I6cnmtdcmNMhhtiPmzPQ8Q.png" /></figure><p>This parameter adds the word as a prefix with an underscore. For the example above, we have used the BC prefix.</p><p><strong>4— get_dummies with ‘columns’ parameter</strong></p><p>We can apply get_dummies directly to a dataframe instead of applying it individually. It will automatically add the column name as a prefix for each dummy variable.</p><pre>pd.get_dummies(df, drop_first = True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/1*npW8IAz1Y2TMKbxGCS1rhw.png" /></figure><p>But the dataframe evolved to 349 columns because the “Extras” column has many unique values. We have to consider this column individually. In this case, we can use the columns parameter.</p><pre>pd.get_dummies(df, columns = [&#39;make_model&#39;, &#39;body_type&#39;, &#39;Body Color&#39;, &#39;Gearing Type&#39;], drop_first = True)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/993/1*ZjbXo0RjW9R6MwfQkpZ9Dw.png" /></figure><p>With the columns parameter, we can apply the columns we want. So we can handle custom columns later with str.get_dummies.</p><p><strong>4 — str.get_dummies</strong></p><p>The str.get_dummies method is a version of get_dummies that can be applied to a series. It is a string-handling version. The str.get_dummies () method divides each string in the given series with the separator. There is only one separator parameter.</p><p>In this dataframe, there are some features for cars in the “Extras” column. For example; alloy wheels, sports seats, voice control, etc. These features are important in terms of price. Check below, it says there are 325 unique values. However, there are 16 extra features. Cars have a different number of extra features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/1*h9Tn2PqoM2kcpR6l8T78Ug.png" /></figure><p>If we apply get_dummies directly, it adds 325 more columns. So we have to use str.get_dummies. We use a comma (,) as the separator because the values in the data are separated by commas (,).</p><pre>df[&#39;Extras&#39;].str.get_dummies(&#39;,&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*IkKLbdhF0qKFrVcTpXBgyw.png" /></figure><p>As you can see, there are 16 extra features. Check for the first row. It has alloy wheels, catalytic converter, and sound control; these are 1, others are 0.</p><p>Now let’s get the latest version of the dataframe by merging them;</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/1*xlgiybHm4J6adli9HcMdsg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/988/1*YkEIbRSXqsCnbQlzAXsaOw.png" /></figure><p>All columns are numeric. Our data is now ready for the model.</p><p><strong>Conclusion</strong></p><p>Pandas get_dummies is the final touch on data before modeling. Because we have to make all non-numerical columns numerical. We have to use it or some other method to get the encoding tables / dummy variables. But as you can see, get_dummies is the easiest way and it has many parameters that make our model more readable and smoother.</p><p>Import the data, handle with missing values and outliers, then apply get_dummies. It is ready for the model.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f95709adfd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Label Encoding vs One Hot Encoding]]></title>
            <link>https://hersanyagci.medium.com/label-encoding-vs-one-hot-encoding-e8a10916da5b?source=rss-7c248ad78b88------2</link>
            <guid isPermaLink="false">https://medium.com/p/e8a10916da5b</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[scikit-learn]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Hasan Ersan YAĞCI]]></dc:creator>
            <pubDate>Mon, 18 Jan 2021 20:43:01 GMT</pubDate>
            <atom:updated>2021-01-24T13:51:17.354Z</atom:updated>
            <content:encoded><![CDATA[<p>Label Encoding vs One Hot Encoding</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/613/1*tUu7iR1xzAeIxJKTlVHpLw.png" /></figure><p>We need numerical data in data science techniques such as machine learning and deep learning models. We start our analysis with categorical and numerical data types. When preparing the data for the model, we drop some categorical data types if we don’t need them, or we use some techniques such as regex and get numerical values. I refer to non-numeric data such as text, object, datetime, etc. with <strong>“categorical data”</strong>. There will always be some columns we need and, there won’t be any numerical values we can get in regex or another function. Therefore, to use data science techniques, we need to convert this non-numerical data to numerical data.</p><p>First of all, we must decide whether these qualitative data are nominal or ordinal. This point is important. It will define the style of our approach. If there is a hierarchy among data, it is ordinal. If there is no hierarchy among the data, it is nominal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/760/1*2dp_x-D1QHMtrI7MISfdaA.png" /><figcaption>Qualitative Data</figcaption></figure><p>Take car colors for example; there is no hierarchy, the car can be black, white, or red. It doesn’t make any order between cars. These are nominal data.</p><p>Now let’s look at the ordinal data, there are shirts in the store and the shirts have sizes such as large, medium, and small. There is a rank. Large is bigger than medium. These are ordinal data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/845/1*a1uJTmDfEiL-UH6n2LPfTw.png" /></figure><p><strong>Encoding Methods</strong></p><p>There are some encoding methods, but the most common and the easiest to implement are Label Encoding and One Hot Encoding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ieFTW63rKI2IyWBP-BU0JQ.png" /><figcaption>Label Encoding and One Hot Encoding</figcaption></figure><p><strong>1 — Label Encoding</strong></p><p>Label encoding is mostly suitable for ordinal data. Because we give numbers to each unique value in the data. If we use label encoding in nominal data, we give the model incorrect information about our data. The model algorithm can act as if there is a hierarchy among the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/289/1*RX9lkufRn7_MYB7Y7WhrVg.png" /><figcaption>Label Encoding for Nominal Data</figcaption></figure><p>Let’s look at the example. There is a column with bridge types. It is nominal. Because there is no ranking among bridge types. In this example, label encoding has been implemented, and with this new column, the model algorithm can act as if there is a hierarchy between bridge types.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/294/1*f7M6th5blh54wwCXT2BPtg.png" /><figcaption>Label Encoding for Ordinal Data</figcaption></figure><p>This is the correct use case. There is a Safety-Level column. There is a hierarchy between data, it is ordinal. Medium is low from high, so it may take 2 for medium and 3 for high. We can implement label encoding and it won’t be a problem for the model algorithm.</p><p>There are different options for applying label encoding to data. Some examples;</p><ul><li>The Scikit-learn, the most useful data science library of python, has methods for this.</li></ul><pre>from sklearn.preprocessing import OrdinalEncoder<br>from sklearn.preprocessing import LabelEncoder</pre><p>OrdinalEncoder is for 2D data, LabelEncoder is for 1D data. For details visit <a href="https://datascience.stackexchange.com/questions/39317/difference-between-ordinalencoder-and-labelencoder">https://datascience.stackexchange.com/questions/39317/difference-between-ordinalencoder-and-labelencoder</a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c04483cba5f380767ea76ad7e76dc408/href">https://medium.com/media/c04483cba5f380767ea76ad7e76dc408/href</a></iframe><ul><li>We can use the Pandas <strong>map</strong> function to implement label encoding. We convert non-numerical values to numerical values manually.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*pKEouBSKF5Gc4QN5zZt4yw.png" /><figcaption>For Data and Notebook <a href="https://github.com/NavarraB/DataScience/tree/master/Other">https://github.com/NavarraB/DataScience/tree/master/Other</a></figcaption></figure><p><strong>2 — One Hot Encoding</strong></p><p>One hot encoding is the common one and the best option, especially for nominal data. We can also use it for ordinal data. It doesn’t matter if there is a hierarchy or not.</p><p>One hot encoding converts the non-numeric column to columns containing 1s and 0s.</p><p>In this method, a non-numerical column is converted into n * columns. n refers to unique values in a non-numerical column.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TuPcb9C8rIkmpP3dZ3fp_w.png" /><figcaption>One Hot Encoding</figcaption></figure><p>In the example above, the color column has 3 unique values; red, blue, and green. It turned into 3 columns after the one hot encoding was applied. One column for each value; color_red, color_blue, and color_green. The blue colored IDs are 2 and 4. In our converted dataframe, note the lines 2 and 4; color_red and color_green are 0 but color_blue is 1.</p><p>It can slows down models because it creates multiple columns. More columns mean less performance and more training time. It’s better to use label encoding if we have ordinal data if we don’t need it.</p><p>There are two methods to implement one hot encoding;</p><ul><li>The Scikit-learn has OneHotEncoder to apply one hot encoding.</li></ul><pre>from sklearn.preprocessing import OneHotEncoder</pre><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/640f9b642f6e9874984aec53526a8aaf/href">https://medium.com/media/640f9b642f6e9874984aec53526a8aaf/href</a></iframe><ul><li>We can use the get_dummies method from Pandas.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*u9qrKm3hXcN9rqrfjhls_Q.png" /><figcaption>One Hot Encoding with Pandas get_dummies</figcaption></figure><p>We can use these two methods but applying get_dummies is easier than Scikit-Learn OneHotEncoder and the get_dummies method has very useful parameters. We covered Pandas get_dummies method at this <a href="https://hersanyagci.medium.com/encoding-with-pandas-get-dummies-2f95709adfd">link</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sHwumEVIG-kpdJTFca1suw.png" /><figcaption>Two Methods of One Hot Encoding</figcaption></figure><p><strong>Conclusion</strong></p><p>As a result, machine learning models are mathematical models that use algorithms that work with numerical data types, and neural networks also work with numerical data types. Therefore, we need to transform categorical data into meaningful numerical data.</p><p>There are some techniques we have seen above for obtaining numerical data from non-numeric data. But there are essentially two results we can get, label encoding or one hot encoding.</p><p>Check the data, decide the encoding method and apply the function.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e8a10916da5b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Detecting and Handling Outliers with Pandas]]></title>
            <link>https://hersanyagci.medium.com/detecting-and-handling-outliers-with-pandas-7adbfcd5cad8?source=rss-7c248ad78b88------2</link>
            <guid isPermaLink="false">https://medium.com/p/7adbfcd5cad8</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Hasan Ersan YAĞCI]]></dc:creator>
            <pubDate>Fri, 15 Jan 2021 08:44:45 GMT</pubDate>
            <atom:updated>2021-01-15T09:14:12.382Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/275/1*kvqLIKkuJMYyTxxn6_U8dQ.jpeg" /></figure><p>Data analysis is a long process. There are some steps to do this. First of all, we need to recognize the data. We have to know every feature in the dataset. Then we must detect the missing values and clear our dataset from these NaN values. We can fill these NaN values with some values (mean, median, etc.) or we can create our function to fill these missing values. We can also drop some columns that are not helpful or have more NaN values than others.</p><p>This process can change. It depends on the data and target. But we must finally fight the outliers. We have to detect and handle them. Each data has different types of outliers, whether they are within 1.5 IQR or not. Sometimes these outliers aren’t harmful, so we don’t deal with them. But if we want to get good results in models or our analysis, we need to handle outliers. There are 3 commonly used methods to deal with outliers.</p><p>1. Dropping the outliers.</p><p>2. Winsorize method.</p><p>3. Log transformation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/481/1*n_ms1q5YoHAQXXUIfeADKQ.png" /></figure><p>Let’s look at these methods with Python,</p><p>In this demo, we will use the Seaborn diamonds dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IhAKeVNPlkn4eJt6Mjbs-g.png" /></figure><p>We will handle the table feature of the diamonds dataset and assume all NaN values have been processed (we just dropped them). Let’s look at the graphs boxplot and histogram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZzC-SUD623uEoX7hTtGDrA.png" /></figure><p>As you can see this column has outliers (it is shown at boxplot) and it is right-skewed data(it is easily seen at histogram). Boxplot is the best way to see outliers.</p><p>Before handling outliers, we will detect them. We will use Tukey’s rule to detect outliers. It is also known as the IQR rule. First, we will calculate the Interquartile Range of the data (IQR = Q3 — Q1). Later, we will determine our outlier boundaries with IQR.</p><p>We will get our lower boundary with this calculation Q1–1.5 * IQR. We will get our upper boundary with this calculation Q3 + 1.5 * IQR.</p><p>According to this rule, the data between boundaries are acceptable but the data outside of the between lower and upper boundaries are outliers. We can use 2.5 or 2 to detect IQR. It depends on our data and analysis. But the most commonly used is 1.5 and we will use 1.5 IQR in this analysis.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jHijFMuswcgEruUZEK4tQg.png" /></figure><p>With the describe method of pandas, we can see our data’s Q1 (%25) and Q3 (%75) percentiles. We can calculate our IQR point and boundaries (with 1.5).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wwdOhCihhK8mXwMuuZb_Rw.png" /></figure><p>Our upper boundary is 63.5 and our lower boundary is 51.5. This means that these values between 51.5 and 63.5 are acceptable but those outside mean there are outliers. So we need to handle them because they corrupt our data.</p><p>Let’s handle outliers.</p><p><strong>1 — Dropping the outliers;</strong></p><p>We can easily remove outliers, but this narrows our data. If we have a lot of rows, big data, maybe we can take risks. But remember, if we drop the value, we delete all records (row). If we have vulnerable records, they can get lost.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j_3a_P89hbIysZ5oguQXcQ.png" /></figure><p>We make some equations to reach outliers and index. As you can see, if we drop outliers, we will lose 605 records. Line 27 shows us the outliers. Line 28 shows us the data without outliers. Check the length of the tables. In the beginning, we had 53940 rows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*z3hOsJzLxUmgg_oGLvNgSg.png" /></figure><p>We removed the outliers and our data rows drop to 53335. After dropping outliers, let’s check the boxplot and histogram of our data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*USZg5fcbSRW0_MllN5Vx3g.png" /></figure><p>Now, we don’t have any outliers.</p><p><strong>2 — Winsorize Method;</strong></p><p>Our second method is the Winsorize Method. In the Winsorize Method, we limit outliers with an upper and lower limit. We will set the limits. We will make our upper and lower limits for data our new maximum and minimum points.</p><p>We will use the table column of the diamonds dataset again. Let’s check the boxplot again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hgjJSK0bSvjJusOG9s-VGw.png" /></figure><p>We have outliers, we detected them at the beginning. For outliers, our upper limit is 63.5 and our lower limit is 51.5.</p><p>For the Winsorize Method, we have to import winsorize from Scipy. We need boundaries to apply winsorize. We will limit our data between 53 and 63. These values are within limits for outliers. We need precise points of these values in the percentile and we can use the quantile method of Pandas.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DP3VaJU92NY4xI-ko3iX7g.png" /></figure><p>We will create a new variable with Winsorize Method. To implement the Winsorize Method, we write the exact boundary points as a tuple on the percentile. For example, we will write (0.01, 0.02). This means we want to apply quantile(0.01) and quantile(0.98) as a boundary. The first one is the exact point on percentile from the beginning, the second one is exact point on percentile from the end.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kCqMbE0S3PQZcg3dw-YKfQ.png" /></figure><p>We applied Winsorize Method, let’s check data on graphs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aM5wvx351gq9_R4WN76UBQ.png" /></figure><p>As you can see, there are no outliers. Focus on this, our maximum and minimum values on the boxplot; 53 and 63. We applied them as boundaries. Now, we can look at descriptive statistical values of old and new data with the describe method.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CApjnuKAPnmT620PtlukMA.png" /></figure><p>I changed df_table_win to a series to implement the describe method. Notice that, the mean and standard deviation of df_table_win have changed. Minimum and maximum points also changed but these changes do not affect median-logic descriptive statistical values. Therefore, we should apply the Winsorize Method carefully because as you can see, mean-logic descriptive statistical values can change. It can corrupt our data, damage our analysis, or have adverse effects on models.</p><p><strong>3 — Log Transformation;</strong></p><p>Our last method is Log Transformation. We use log transformation on skewed data. Log transformation reduces the skewness of data and tries to make it normal. Log transformation doesn’t always make it normal, sometimes makes data more skewed. So it depends on the data. We have to apply transformation and control the result.</p><p>For this method, we will use the carat column of diamonds dataset. Let’s check the data and graphs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kacf9m4vDkbw0oipbnWhBw.png" /></figure><p>There are many outliers and the data is right-skewed. Log transformation will transform data to normal or close to normal. Let’s apply the log transformation to reduce the variability of data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RfYTpWp6sSiWUKXkZFE2Cg.png" /></figure><p>We implemented log transformation from NumPy with np.log. It completely changed our data and it removed outliers, we can see this in the boxplot.</p><p>Log transformation is commonly used for machine learning algorithms. Be careful, it changes our values but removes outliers. It makes our model normal and the machine learning algorithm likes normal distribution data. There are some methods and features in machine learning algorithms such as scaling and normalization. We will talk about these terms in our next stories.</p><p><strong>Conclusion</strong></p><p>It takes a long time to deal with outliers but it’s worth it. We have to choose the best method for the data. It is important to know the data very well. With this domain knowledge, we will decide on the method of handling outliers. Apart from these methods, we can consider these outliers as missing values. We can use the methods of filling missing values to get rid of outliers.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7adbfcd5cad8" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>