<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Chandra Sekhar on Medium]]></title>
        <description><![CDATA[Stories by Chandra Sekhar on Medium]]></description>
        <link>https://medium.com/@chandrasekhar_ism?source=rss-2926b588c328------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*ktvv8rcRLNQRNtEV.jpg</url>
            <title>Stories by Chandra Sekhar on Medium</title>
            <link>https://medium.com/@chandrasekhar_ism?source=rss-2926b588c328------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 21:57:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@chandrasekhar_ism/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Outlier Detection and Removal in Data Science and Analytics projects]]></title>
            <link>https://medium.com/@chandrasekhar_ism/outlier-detection-and-removal-in-data-science-and-analytics-projects-9a18a8835970?source=rss-2926b588c328------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a18a8835970</guid>
            <category><![CDATA[data-mining]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Chandra Sekhar]]></dc:creator>
            <pubDate>Tue, 10 Sep 2024 17:05:50 GMT</pubDate>
            <atom:updated>2024-09-11T09:46:34.097Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>What are outliers in data?</strong></p><p><strong>Outliers are the datapoints that do not follow the general pattern and differ considerably from the rest of the data. They can arise in data due to several reasons. Anomaly in the data or error in measurement are some of the major reasons.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*YYvRR7qAb816OGLO.png" /><figcaption>Outliers</figcaption></figure><p>But, when we are analyzing data in any data science or analytics project, our goal is to uncover patterns that lead to actionable insights. However, outliers are a common challenge, both in the real world and in datasets. When working with real-world data, encountering outliers is inevitable. These outliers can obscure the underlying patterns, making it difficult to draw accurate conclusions and build reliable predictive models.</p><p>So, we must know how to detect them and treat them before we build any data science model or analyze the data to derive meaningful insights. We can follow the following steps:-</p><blockquote><strong>Step1 : </strong>The first step is to check the distribution of the data. The data could be normally distributed or skewed. The approach will differ based on the distribution.</blockquote><blockquote><strong>Step 3:</strong> If the distribution is normal, we apply <strong>Z score method</strong> to find the outliers and if the distribution is skewed, we can apply <strong>IQR method.</strong></blockquote><blockquote><strong>Step 4:</strong> Once the outliers are detected, we treat them either by <strong>trimming</strong> them from the dataset or <strong>capping</strong> them in the dataset. The details on trimming and capping are mentioned below.</blockquote><h3>Normal Distribution</h3><p>We have taken a dataset and plotted a histogram and boxplot.</p><pre>df = pd.read_excel(&#39;Outlier.xlsx&#39;, index_col=0)<br>df.head()<br><br><br>plt.figure(figsize=(12,8))<br>plt.subplot(2,2,1)<br>sns.histplot(df[&#39;Price&#39;], kde=True)<br><br>plt.subplot(2,2,2)<br>sns.boxplot(x=df[&#39;Price&#39;])<br><br><br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oThvVnzL7b07u6aK_-UzaA.png" /></figure><p>Looking at the plots, we can see that the distribution is close to normal. Skewness in price is almost 0.2 and hence the distribution can be considered normal.</p><pre>print(&quot;Skewness in the Data: &quot;,df[&#39;Price&#39;].skew())<br>print(df[&#39;Price&#39;].describe())<br><br>Output: <br>Skewness in the Data:  0.2063491684645793<br>count    100.000000<br>mean      63.730000<br>std       25.117443<br>min      -20.000000<br>25%       54.000000<br>50%       66.000000<br>75%       77.000000<br>max      160.000000<br>Name: Price, dtype: float64</pre><p><strong>Identifying the Outliers: </strong>For identifying the outliers, we must define the range of values that should be considered as normal. We can define the upper and lower limit of the data using the formula below. This is the Z score method of identifying the outliers.</p><blockquote>Upper Limit = Mean + 3 * std deviation</blockquote><blockquote>Lower Limit = Mean — 3 * std deviation</blockquote><pre>#Determining the upper and lower limit of the data<br>upper= df[&#39;Price&#39;].mean()+3*df[&#39;Price&#39;].std()<br>lower= df[&#39;Price&#39;].mean()-3*df[&#39;Price&#39;].std()<br>print(&#39;upper: &#39;,upper)<br>print(&#39;lower: &#39;,lower)<br><br>df[(df[&#39;Price&#39;]&gt;upper) | (df[&#39;Price&#39;]&lt; lower)]<br><br>Output:<br><br>upper:  139.08232999956695<br>lower:  -11.62232999956695<br>            <br>The outlier rows are:-<br>Product ID |Price |Sales<br>2 -20 14<br>33 158 25<br>48 160 37</pre><p><strong>Removal of Outliers: </strong>The methods for removal are:-</p><ol><li><strong>Trimming</strong>: Removing the rows that have data that is beyond the 3 standard deviation from the mean value. .</li><li><strong>Capping : </strong>In this method, we can replace the values of the outliers with the upper limit and lower limit</li></ol><pre>#Applying Trimming method<br><br>df_trimmed= df[(df[&#39;Price&#39;]&lt;upper) &amp; (df[&#39;Price&#39;]&gt; lower)]<br><br>#Applying Capping method<br>df_capped= df.copy()<br>df_capped[&#39;Price&#39;]= np.where(df_capped[&#39;Price&#39;]&gt;upper, upper,<br>                            np.where(df_capped[&#39;Price&#39;]&lt;lower,lower,<br>                            df_capped[&#39;Price&#39;]))<br></pre><p>In the plot we can look at the 3 graphs after the outlier treatment.</p><pre>plt.figure(figsize=(20,12))<br>plt.subplot(2,3,1)<br>sns.histplot(df[&#39;Price&#39;], kde=True)<br>plt.title(&#39;Original&#39;)<br><br>plt.subplot(2,3,2)<br>sns.histplot(df_trimmed[&#39;Price&#39;], kde=True)<br>plt.title(&#39;Trimmed&#39;)<br><br>plt.subplot(2,3,3)<br>sns.histplot(df_capped[&#39;Price&#39;], kde=True)<br>plt.title(&#39;Capped&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*TRky3RTtmIh6Bi2JsiwDpQ.png" /><figcaption>After Treatment of outliers</figcaption></figure><p>In the trimmed plot, we can see that all the 3 outlier rows have been removed. While in the capped plot, we have re-assigned the lower and upper limit values to the outliers.</p><h3>Non- Normal Distribution</h3><p>This method should be used when the data is skewed. The data could be right skewed or left skewed. We can look at the given data for sales in the same dataset as discussed above. The distribution plot is right skewed. we can observe some outliers in the boxplot as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aNJ-dvYMn5FZGuM8A5Fwhw.png" /></figure><p><strong>Inter Quartile Range method: (IQR):</strong></p><p>Here, we would be using the IQR method to determine the upper and lower limit for the dataset. The upper and lower limit can be defined by using the following formula</p><p>upper= 75th percentile + 1.5 * IQR</p><p>lower= 25th percentile — 1.5* IQR</p><pre>iqr= df[&#39;Sales&#39;].quantile(.75)-df[&#39;Sales&#39;].quantile(.25)<br><br>upper= df[&#39;Sales&#39;].quantile(.75) + 1.5*iqr<br>lower= df[&#39;Sales&#39;].quantile(.25) - 1.5*iqr<br>print(iqr)<br>print(upper)<br>print(lower)<br><br>Out:<br>18.0<br>55.0<br>-17.0</pre><p>We can now perform the same set of operations ie trimming or capping in the dataset to remove the outliers.</p><pre>#Applying Trimming method<br><br>df_trimmed= df[(df[&#39;Sales&#39;]&lt;upper) &amp; (df[&#39;Sales&#39;]&gt; lower)]<br><br>df_capped= df.copy()<br>df_capped[&#39;Sales&#39;]= np.where(df_capped[&#39;Sales&#39;]&gt;upper, upper,<br>                            np.where(df_capped[&#39;Sales&#39;]&lt;lower,lower,df_capped[&#39;Sales&#39;]))<br><br>plt.figure(figsize=(20,12))<br>plt.subplot(2,3,1)<br>sns.histplot(df[&#39;Sales&#39;], kde=True)<br>plt.title(&#39;Original&#39;)<br><br>plt.subplot(2,3,2)<br>sns.histplot(df_trimmed[&#39;Sales&#39;], kde=True)<br>plt.title(&#39;Trimmed&#39;)<br><br>plt.subplot(2,3,3)<br>sns.histplot(df_capped[&#39;Sales&#39;], kde=True)<br>plt.title(&#39;Capped&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ftDU3hmf8ji9YiuffuE3JQ.png" /><figcaption>Outlier treatment using IQR method</figcaption></figure><p>The plots here just represent the way in which we can treat the outliers. Data can be different in different use cases. Hence, the method used for removing the outliers can be different.</p><blockquote>The objective to be kept in mind while treating the outliers is that the overall data must make sense and it should be such that we can derive insights from it. We should not just mechanically follow a strict rule for the treatment of outliers.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a18a8835970" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>