<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nindya Isdiarti on Medium]]></title>
        <description><![CDATA[Stories by Nindya Isdiarti on Medium]]></description>
        <link>https://medium.com/@nindyais?source=rss-6e5dcd8d298a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*PADKTdvWKs0DWJkB5FuGYA.jpeg</url>
            <title>Stories by Nindya Isdiarti on Medium</title>
            <link>https://medium.com/@nindyais?source=rss-6e5dcd8d298a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 19:57:58 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nindyais/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Data Cleaning: Inconsistent Data Entry]]></title>
            <link>https://medium.com/@nindyais/data-cleaning-inconsistent-data-entry-7731ac3c52c7?source=rss-6e5dcd8d298a------2</link>
            <guid isPermaLink="false">https://medium.com/p/7731ac3c52c7</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[text-preprocessing]]></category>
            <category><![CDATA[pandas-dataframe]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-cleaning]]></category>
            <dc:creator><![CDATA[Nindya Isdiarti]]></dc:creator>
            <pubDate>Thu, 24 Jun 2021 08:52:25 GMT</pubDate>
            <atom:updated>2021-06-28T00:29:55.102Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/626/0*ra0utGoOQQh_P90U.png" /><figcaption>Photo by <a href="http://www.intellspot.com/what-is-data-cleansing/">intellspot</a></figcaption></figure><p>There are several ways to cleaned up data entry inconsistencies and ensure that the data is cleaned up and ready for analysis. The first way is to check each row by hand, of course, and fix inconsistencies if found. But, there is a more efficient way to do this like doing some preliminary text pre-processing.</p><h3>Get our environment set up</h3><p>The first thing that needs to be done is to load the libraries and datasets that will be used. In this notebook will use the airlines DataFrame from the Datacamp course.</p><pre># import modules<br>import pandas as pd</pre><pre># assign url of file: url<br>url = &#39;&lt;https://assets.datacamp.com/production/repositories/5737/datasets/ba95dfa6d750e4bf2ddda2349cfe0af80ab765ff/airlines_final.csv&gt;&#39;</pre><pre># read file airlines<br>airlines = pd.read_csv(url, sep=&#39;,&#39;)</pre><pre># print the head of the DataFrame<br>print(airlines.head())</pre><p>Now we’re ready to get started! You can, as always, take a moment here to look at the data and get familiar with it</p><h3>Do some preliminary text pre-processing</h3><p>For this exercise, Datacamp provides an example using two categorical columns from an airline DataFrame, dest_region and dest_size, respectively. Categorical and text data can often be some of the messiest parts of a dataset due to their unstructured nature. Do some preliminary text pre-processing in this chapter means how to fix whitespace and capitalization inconsistencies in category labels.</p><h3>Fix capitalization inconsistencies</h3><pre># get all the unique values in the &#39;dest_region&#39; column<br>dest_region = airlines[&#39;dest_region&#39;].unique()</pre><pre># sort them alphabetically and then take a closer look<br>dest_region.sort()<br>dest_region</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/646/1*da9vnf7xyJ3bCshVFqsbFw.png" /><figcaption>Output: Unique values in the ‘dest_region’</figcaption></figure><p>From looking at the output, it shows that the dest_region column has inconsistent values due to capitalization. For example ‘EAST US’ and ‘East US’, ‘Middle East’ and ‘middle east’. The first thing that can be done is to make everything lowercase.</p><h3>Lowercase</h3><pre># Lower dest_region column<br>airlines[&#39;dest_region&#39;] = airlines[&#39;dest_region&#39;].str.lower()</pre><pre># Verify changes have been effected<br>print(airlines[&#39;dest_region&#39;].unique())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/678/1*RhUV37JX45uykKqO5J6Feg.png" /><figcaption>Output: Unique values in the ‘dest_region’ after make everything lowercase</figcaption></figure><p>From looking at the output, it shows that the dest_region column has no inconsistent values due to capitalization.</p><h3>Fix whitespace</h3><pre># get all the unique values in the &#39;dest_size&#39; column<br>dest_size = airlines[&#39;dest_size&#39;].unique()</pre><pre># sort them alphabetically and then take a closer look<br>dest_size.sort()<br>dest_size</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/651/1*GCJDw0FHLTAGRk2q7D1kZQ.png" /><figcaption>Output: Unique values in the ‘dest_size’</figcaption></figure><p>From looking at the output, it shows that the dest_size column has only inconsistent values due to leading and trailing spaces. For example ‘ Hub’ and ‘Hub’. The first thing that can be done is to remove the whitespace at the beginning and end of the cell.</p><h3>Remove whitespace</h3><pre># Remove white spaces from `dest_size`<br>airlines[&#39;dest_size&#39;] = airlines[&#39;dest_size&#39;].str.strip()</pre><pre># Verify changes have been effected<br>print(airlines[&#39;dest_size&#39;].unique())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/316/1*oWpJgBKJLe2UX9nqzlTFsg.png" /><figcaption>Output: Unique values in the ‘dest_size’ after removing whitespace</figcaption></figure><p>From looking at the output, it shows that the dest_size column has no inconsistent values due to leading and trailing spaces. Inconsistencies in capitalization and trailing whitespace are very common in text data and can fix 80% of your text data entry inconsistencies by doing this.</p><h3>Use fuzzy matching to correct inconsistent data entry</h3><p>Alright, let’s take another look at the dest_region column and see if there’s another inconsistency.</p><pre># get all the unique values in the &#39;dest_region&#39; column<br>dest_region = airlines[&#39;dest_region&#39;].unique()</pre><pre># sort them alphabetically and then take a closer look<br>dest_region.sort()<br>dest_region</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*6mykapXAMwmvnUWUUL929A.png" /><figcaption>Output: Unique values in the ‘dest_region’</figcaption></figure><p>It does look like there are some remaining inconsistencies: ‘europe’ and ‘eur’ should probably be the same.</p><p><a href="https://github.com/seatgeek/fuzzywuzzy">fuzzywuzzy</a> package to help identify which string are closest to each other. This dataset is small enough that we could probably could correct errors by hand, but that approach doesn’t scale well. (Would you want to correct a thousand errors by hand? What about ten thousand? Automating things as early as possible is generally a good idea.</p><blockquote><strong>Fuzzy matching:</strong> The process of automatically finding text strings that are very similar to the target string. In general, a string is considered “closer” to another one the fewer characters you’d need to change if you were transforming one string into another. So “apple” and “snapple” are two changes away from each other (add “s” and “n”) while “in” and “on” and one change away (rplace “i” with “o”). You won’t always be able to rely on fuzzy matching 100%, but it will usually end up saving you at least a little time.</blockquote><p>Fuzzywuzzy returns a ratio given two strings. The closer the ratio is to 100, the smaller the edit distance between the two strings. Here, we’re going to get the five strings from our list of dest_region that have the closest distance to ‘europe’.</p><pre># get the top 5 closest matches to &quot;europe&quot;<br>matches = fuzzywuzzy.process.extract(&quot;europe&quot;, dest_region, limit=5, scorer=fuzzywuzzy.fuzz.token_sort_ratio)</pre><pre># take a look at them<br>matches</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/212/1*mrjukvStj3Lq9KjiVEoukQ.png" /><figcaption>Output: top 5 closest matches to “europe”</figcaption></figure><p>From looking at the output, it shows that two of the items in the dest_region are very close to “europe”: “europe” and “eur”. Let’s replace all rows in dest_region column that have a ratio of &gt; 65 with “europe”.</p><pre>def replace_matches_in_column(df, column, string_to_match, min_ratio = 65):<br>    # get a list of unique strings<br>    strings = df[column].unique()<br>    <br>    # get the top 5 closest matches to our input string<br>    matches = fuzzywuzzy.process.extract(string_to_match, strings, <br>                                         limit=5, scorer=fuzzywuzzy.fuzz.token_sort_ratio)</pre><pre># only get matches with a ratio &gt; 65<br>    close_matches = [matches[0] for matches in matches if matches[1] &gt;= min_ratio]</pre><pre># get the rows of all the close matches in our dataframe<br>    rows_with_matches = df[column].isin(close_matches)<br>    <br>     # replace all rows with close matches with the input matches <br>    df.loc[rows_with_matches, column] = string_to_match<br>    <br>    # let us know the function&#39;s done<br>    print(&quot;All done!&quot;)</pre><p>Now let’s can check the unique values in dest_region column again and make sure we’ve tidied up europe correctly.</p><pre># use the function to replace close matches to &quot;europe&quot; with &quot;europe&quot;<br>replace_matches_in_column(df=airlines, column=&#39;dest_region&#39;, string_to_match=&quot;europe&quot;)</pre><pre># get all the unique values in the &#39;dest_region&#39; column<br>dest_region = airlines[&#39;dest_region&#39;].unique()</pre><pre># sort them alphabetically and then take a closer look<br>dest_region.sort()<br>dest_region</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*ju21v1-r9ndqvP2cYhX9LQ.png" /></figure><p>From looking at the output, it seems that there is no problem in dest_region column. Now we only have “europe” in dataframe and we didn’t have to change anything by hand.</p><p>Source:</p><p><a href="https://campus.datacamp.com/courses/cleaning-data-in-python/text-and-categorical-data-problems?ex=6">Datacamp Course: Cleaning Data in Python</a></p><p><a href="https://www.kaggle.com/rtatman/data-cleaning-challenge-inconsistent-data-entry">Rachael Tatman: Data Cleaning Challenge Inconsistent Data Entry</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7731ac3c52c7" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>