<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ronald Wahome on Medium]]></title>
        <description><![CDATA[Stories by Ronald Wahome on Medium]]></description>
        <link>https://medium.com/@wahomeron?source=rss-ec73e9699e73------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*_vTGn62Cvww8QNDw-BQYwQ.jpeg</url>
            <title>Stories by Ronald Wahome on Medium</title>
            <link>https://medium.com/@wahomeron?source=rss-ec73e9699e73------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 08:38:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@wahomeron/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Downloading Craigslist Images in Python— Data Science With Craigslist Data Part 2.2]]></title>
            <link>https://medium.com/@wahomeron/downloading-craigslist-images-in-python-data-science-with-craigslist-data-part-2-2-2c286130a510?source=rss-ec73e9699e73------2</link>
            <guid isPermaLink="false">https://medium.com/p/2c286130a510</guid>
            <category><![CDATA[beautifulsoup]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Ronald Wahome]]></dc:creator>
            <pubDate>Mon, 08 Nov 2021 13:33:24 GMT</pubDate>
            <atom:updated>2021-11-08T15:29:07.656Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EgkSFOMU1283lKgJ" /></figure><p>This post is part of the Data Science with Craigslist Data series which is available here. In part 2.1, I demonstrated how to scrape from Craigslist vehicle ads from 15 top cities in the US using python and Beautiful Soup. In this post I’ll go over how to download the images that come with each post as highlighted below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zYoRvLBAxE8Z6g1CYtcKzA.png" /></figure><p>There are many ways to go about this but my workflow is as follows. I scraped 34,000 car ads and saved the <em>post IDs</em>. Then I created a different script to get the images based on this post IDs only which was more efficient than getting the post data and images together.</p><h4>Images Downloads Workflow</h4><ol><li>Create a list with links to all available pages in 15 cities.</li><li>Loop through each page and access each post. There’s 120 posts per page.</li><li>For each vehicle post, create a dictionary with <em>key </em>as postID and <em>values </em>as all available images links.</li><li>Create another list and append<em> postID, ImageLinks </em>dictionary.</li><li>Save the list of dictionaries to a JSON file for later.</li><li>Read the JSON file with python and iterate through it.</li><li>For each postID, create a local folder.</li><li>For each image link associated with the postID, download image, rename and store in the postID folder.</li><li>Repeat this process for all 34,000 keys in the dictionary.</li></ol><p>Ok lets look at the actual code. The Jupyter notebook is posted on <a href="https://github.com/RonKG/Craigslist-Cars-For-Sale-Data-Project/blob/71778f28b1a2733348981bb19d2410084b2e20a5/Craigslist_Part_2_2_Downloading_Images_from_Craigslist.ipynb">my GitHub</a> if you’d like to follow along.</p><p>The first thing we do is load all the necessary python libraries we’ll need including BeautifulSoup.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dfdd81a7fcdafb6f220ba5fed63ee8b6/href">https://medium.com/media/dfdd81a7fcdafb6f220ba5fed63ee8b6/href</a></iframe><p>Next we read a csv file that has links to all the pages available for all 15 cities in our data. There’s a total of 318 pages for cities like Chicago, Dallas, New York etc.. Each pointing to a page on the vehicle got sale with owner and a <em>containing images</em> filter. They look as seen below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4iNUU01XjizLnbzfb4jZrA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*rD9BOPUSSewvLkn-zkgtZA.png" /><figcaption>Returned list is 318 in length</figcaption></figure><p>We’ll now cycle through these list and for each vehicle in each page, get the embedded image links. First create a list which you’ll use to store the image dictionaries. Then use <em>Requests method </em>to navigate to the link and <em>BS </em>to get a BS object of the page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*faew9TgW1HoJcCIi3GLFYA.png" /></figure><p>You can then use BS method to find the<em> html classes and objects </em>that contains the relevant data.</p><p>The postID to each vehicle is found in the <em>result-row</em> html class object and stored as <em>data-pid</em>. so go ahead and grab this text. It is indicated with <strong>arrow 1</strong> in the image above.</p><p>The image data for each vehicle is stored as <em>data-ids </em>in <em>result-image-gallery</em> class object shows by <strong>arrow 2 </strong>in the above image<em>.</em></p><p>The data stored in the data-ids variable is the id to each image for that vehicle. We grab these <em>data ids</em> and pass them to an image link place holder which we looks like this<em> ‘https://images.craigslist.org/{}_300x300.jpg&#39;</em><strong>.</strong> See <strong>arrow 3</strong> above. These should give you all the links to the images which you store in a (postID: imgLinks) dictionary.</p><p>Then whole code is shown below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d863eff182eb004c9cc2ecd2df7a48c7/href">https://medium.com/media/d863eff182eb004c9cc2ecd2df7a48c7/href</a></iframe><p>After we’ve stored the list of dictionaries in a JSON file. We can retrieve it at a later time to download the actual images. To do this, we just loop through the list and for each dictionary value, we first create a directory that has the same name as the dictionary key.</p><p>Then we access the image link with <em>requests.get(imgLink)</em>. If the request returns a status code of 200, then the image is present so go ahead and save it locally directory we just created and rename it to postID +=1.</p><p>Repeat this steps for all dictionaries and you’ll have all the images downloaded. We can then link the images back to the scraped vehicle data stored in a table using the postID. This will come in handy when doing some machine learning project like image recognition because we can attach labels to the data etc..</p><p>Just as a side note, the Craigslist server will lock you out after about 30,000 images or 12 hours. Even after using a delay time filter. So plan ahead and expect to do this in a few days if you have more than 30K images. Also utilize error handling when writing you code because unexpected stuff will happen. <br>Thanks for reading and if you have any questions let me know below.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2c286130a510" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploratory Data Analysis (EDA) — Data Science With Craigslist Data Part 4]]></title>
            <link>https://medium.com/@wahomeron/exploratory-data-analysis-eda-data-science-with-craigslist-data-part-4-169079a555ab?source=rss-ec73e9699e73------2</link>
            <guid isPermaLink="false">https://medium.com/p/169079a555ab</guid>
            <category><![CDATA[data-exploration]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <dc:creator><![CDATA[Ronald Wahome]]></dc:creator>
            <pubDate>Mon, 01 Nov 2021 12:31:05 GMT</pubDate>
            <atom:updated>2021-11-01T14:49:09.949Z</atom:updated>
            <content:encoded><![CDATA[<h3>Exploratory Data Analysis (EDA) — Data Science With Craigslist Data Part 4</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*93O7lL_BQq98svsH" /></figure><p>When you get your hands on some new data, some of the the first questions go something like this. How much data is there, as in rows and columns? How much is missing? How many variables are there? How many numerical fields? What relationships exist in the data? Is there a skew in the data to avoid making the wrong conclusions? etc. The process of getting answers to these kinds of questions is exploratory data analysis.</p><p>In previous posts of this series, we web <a href="https://medium.com/mlearning-ai/a-step-by-step-how-to-do-a-data-science-project-with-craigslist-data-part-2-website-scraping-d352351655ee"><strong>scraped vehicle ads from Craigslist in part 2</strong></a> using python and then performed some <a href="https://medium.com/@wahomeron/craigslist-cars-for-sale-data-project-part-3-data-cleaning-2d49bf39dbde"><strong>data cleaning in part 3</strong></a>. In this section we’ll perform some EDA on the resulting dataset from the previous two sections. You can find a copy of the Jupyter notebook on my <a href="https://github.com/RonKG/Craigslist-Cars-For-Sale-Data-Project/blob/b0faa022e35b4832128c03e08ee447a1aa5cbee8/Craigslist%20Part%204_Exploratory_Data_Analysis_(EDA).ipynb"><strong>GitHub </strong></a>if you’d like to follow along.</p><p>First things first, we load the necessary python libraries and then read the cleaned data into a Jupyter notebook environment, check out the data types and reformat them as needed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/571/1*d5hjqq7JIvQindN9QMRPYw.png" /></figure><p>In total, we have 35,887 rows and 25 columns. We’ll start our exploration by seeing how many cities there are in the dataset and how much data we have on each city. A quick way to visualize this is using a histogram which we’ll create using the <em>matplotlib </em>library.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3a10cc0128221d093314111a339190af/href">https://medium.com/media/3a10cc0128221d093314111a339190af/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*lO9TFt52EKxkyQmsw9it1A.png" /></figure><p>Most of the cities have around 3000 records. The exceptions are, Philadelphia, San Antonio, Washington DC, Boston and Nashville which have records ranging from 800 to 1800 records. The reason is that Craigslist maintains a maximum of 3000 records per city for each post category. We see that some cities do not have enough posts in the vehicle ads category to reach 3000.</p><p>This data was downloaded in late October 2021 but we can also see just what date each vehicle ad was posted. Pandas are quite handy at handling time series data so we’ll just group the data by year, month and day to see the trends.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d9db707d12b7580e010e058d7f73de61/href">https://medium.com/media/d9db707d12b7580e010e058d7f73de61/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/753/1*faiscZAJMc6UAJHZuY9HSA.png" /></figure><p>The earliest post was on 9/24/2021 and the last one was on 10/24/2021. That&#39;s a months worth of vehicle ads. But on what days were the ads posted?</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5f5e544e7fc9c8dfe12b991e472f11af/href">https://medium.com/media/5f5e544e7fc9c8dfe12b991e472f11af/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/1*TyGR9SqZsbnPWMGvNH_dAA.png" /></figure><p>Saturdays are the most popular to post vehicle ads on Craigslist followed by Saturday, Sunday, Thursday, Wednesday, Monday respectively and finally Tuesday is the slowest day for new ads posts. That’s great to know but what time do these posts happen?</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/86175c2362a34854c22be1fbd732196e/href">https://medium.com/media/86175c2362a34854c22be1fbd732196e/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*8YISpLspgziLRV1xKwajbw.png" /></figure><p>It look like the hours between 9am and 6pm are the most popular for posting vehicle ads with a peak time between 10am and 3pm.</p><p>Now that we’ve examined the time components of the data, let’s look at some more variables. Let’s start with the price distribution across the data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/af190478555f0bfb857e8ce0541833f2/href">https://medium.com/media/af190478555f0bfb857e8ce0541833f2/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/755/1*7I5Kvvc2IvATfwclTlBLXw.png" /></figure><p>There’s 34K plus vehicles with roughly 11K in the range of $0 to $5,000 and another 11K in the range of $5,000 to $10,000. This means that roughly 65% of all vehicle ads are below ten thousand dollar. Now I’m just burning to know when these cars were made.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/52d28e97aec1f7a63d7e143b770780f1/href">https://medium.com/media/52d28e97aec1f7a63d7e143b770780f1/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*2X2pvqz9z3OazMch4VSIsQ.png" /></figure><p>It turns out that most of these cars were made between 2004 and 2016 with a peak year of 2008. The mode age of a car on Craigslist is around 12 years old but I wonder which vehicle brands dominate these ads.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ebf43cfa7a33984c6f66d7a063bd8a89/href">https://medium.com/media/ebf43cfa7a33984c6f66d7a063bd8a89/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/726/1*8Jy4sf0hTipw1AJGzwfxTA.png" /></figure><p>In all the 15 cities in our dataset, Ford, Chevrolet and Toyota are the top three car makes/brands on Craigslist respectively making up about 40% of all vehicles in the data. The top 2 are American made while the ranks 3, 4 and 5 are Japanese made. These numbers might change a bit if you group them by parent company but Ford and Chevrolet parent companies would still have a significant lead over the other brands. We also have the color attribute so we’ll go ahead and see what the most popular colors are.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/85f738ac51b359c45d548217068b7570/href">https://medium.com/media/85f738ac51b359c45d548217068b7570/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*6EYIUSgdwob2ahWvYsObaQ.png" /></figure><p>Well, the color white is the most popular followed by black and in third place we have silver. Blue and red also appear a lot in ads making up around 5,000 of the 34K records.</p><p>We have three variables left to explore which are vehicle transmission, number of engine cylinders and title status. Let’s visualize those as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/508/1*DKhA-7SGyfAP3MiF-Ws8yg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/475/1*11ssri8hhLGDFrEQaSsuyw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/592/1*Pg1eyb5rBe93s-sBeKDGrw.png" /></figure><p>Automatic transmission is the most popular type of transmission while a clean title status occurs the most in the title status category. Also, 6 cylinder engines are the most prominent but not by much compared to the second ranked 8 cylinders and in third place 4 cylinders. These top 3 account for more than 90% of all records. All other cylinder configurations appear in very small numbers.</p><p>So far, we have familiarized ourselves with the data variables and know where to start when we’re asking deeper questions from the data like what is the relationship of one variable to another. Or are cars cheaper in Miami than say Chicago. But more importantly, we also have an inkling of the data limitations from our EDA. Let’s take a look at price and vehicle age for example.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c8633ba36139b05d9ca80e2daea2219c/href">https://medium.com/media/c8633ba36139b05d9ca80e2daea2219c/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/773/1*fx03VXDDrrlzdkxVOsbuwA.png" /></figure><p>While not enough to make a solid decision, we can see from the boxplots above that the average price of a car is lowest in Atlanta, Philadelphia and Washington DC. Phoenix and Los Angeles have the highest average price. Most of the high averages are driven by the high variance that exists in price within each city.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*jFv1Swdba3KxdmIdUcHQrQ.png" /></figure><p>Looking at the vehicle ages, we can observe that on average, Los Angeles and Seattle have the most oldest cars in the group at 6 years old average. Miami and New York have the newest average age of a car at around 4½ years old. That’s not much of a difference from the latter but it does give you an idea of which city has more new cars than old ones. A point which is further revealed by the boxplot above which shows that cars from the 25th percentile of all cars within those cities were manufactured around the year 2000.</p><p>The histogram and KDE plots below show the same information as above but from a different angle.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5d801fbb3ffdbed13d2d32480d95af7c/href">https://medium.com/media/5d801fbb3ffdbed13d2d32480d95af7c/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lPn6NAxq8qy8nK3f6tPfWQ.png" /></figure><p>The vehicle age distribution of cars in all 15 cities is skewed to the left with most cars made between 2004 and 2015. Philadelphia, San Antonio, Boston and Nashville have fewer cars overall which is explained by a much smaller sample size.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c62ea128eb62b277e6cee2749e8b2802/href">https://medium.com/media/c62ea128eb62b277e6cee2749e8b2802/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tnURI2VlMuHG158gICvvMQ.png" /></figure><p>The price in all 15 cities is skewed to the right with most cars ranging from $5,000 to $10,000 in price. Price and vehicle age combined yield the scatterplot below which helps us understand some of the relationship that exists between these two factors.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8ab9a695d4947d9a963b5bb40974d585/href">https://medium.com/media/8ab9a695d4947d9a963b5bb40974d585/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f4HK4uvJhRbxpC-DL-_RpQ.png" /></figure><p>By analyzing the price vs vehicle age relationship, we can see that the newer car models fetch a higher price than older ones. Furthermore, the worse the vehicle condition the lower the price on average. We can also observe that most of the vehicles that are very old, between 1925 and 1975, these car tend to fetch a high price with Los Angeles and Phoenix leading the count in these older vehicles.</p><p>Finally, we’ll create a correlation matrix to give us a higher understanding of the numerical variables in the data, i.e. columns. Some of the categorical variables can also be encoded into numerical variables for the correlation analysis. The columns that fit this categorical encoding are:- year, condition, cylinders, drive(manual or automatic), fuel, size, title status, transmission, make. We use the <em>pandas.Categorical </em>method to do the encoding and create new numerical columns while keeping the old categorical ones.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fe12dc609fb0e1e8741eec99894ad1f1/href">https://medium.com/media/fe12dc609fb0e1e8741eec99894ad1f1/href</a></iframe><p>Then create a correlation matrix plot with <em>Seaborn</em>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/20f6f11fe1c2babaa8d8dae7ffca3713/href">https://medium.com/media/20f6f11fe1c2babaa8d8dae7ffca3713/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/745/1*pOMNsONFAOUbXm7CnGyrdA.png" /></figure><p>On a high level, it seems that none of the variable have a strong correlation with each other and while this may be true, we can now examine closer using variable values for hidden correlations that may not immerge at the moment because of the noise in the data.</p><p>When shopping for a car, everyone wants good value for their money. Value is mostly determined by looking a four basic things. Price, age, mileage and vehicle condition. We also have drivetrain, fuel type, transmission among other data points which further inform on the price. In the next section, <strong>part 5</strong> of this series, we’ll do a thorough deep dive into these fields and elicit any interrelationships that exist between them. As always, please let me know if you have any questions in the comment section below.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=169079a555ab" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Cleaning in Python — Data Science With Craigslist Data Part 3]]></title>
            <link>https://medium.com/@wahomeron/craigslist-cars-for-sale-data-project-part-3-data-cleaning-2d49bf39dbde?source=rss-ec73e9699e73------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d49bf39dbde</guid>
            <category><![CDATA[pandas-dataframe]]></category>
            <category><![CDATA[data-cleaning]]></category>
            <category><![CDATA[pandas-tutorial]]></category>
            <category><![CDATA[data-cleansing]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Ronald Wahome]]></dc:creator>
            <pubDate>Mon, 25 Oct 2021 12:02:57 GMT</pubDate>
            <atom:updated>2021-10-31T17:21:50.518Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EZBAKuNpSK7YYCHyNaYu2Q.jpeg" /></figure><p>Hello again Medium. This is the third installment of a six part series that goes over the entire data analytics process of some 20K plus Craigslist vehicle ads data from 20 cities. In the <a href="https://medium.com/@wahomeron/a-step-by-step-how-to-do-a-data-science-project-with-craigslist-data-part-2-website-scraping-d352351655ee">previous section</a>, I covered the actual web scraping process using python and BeautifulSoup.</p><p>Data cleansing is the technique of fixing or removing incorrect, corrupted, incorrectly formatted, duplicate, or incomplete statistics inside a dataset. If the data is incorrect, outcomes and algorithms become unreliable, even though they may appear to be correct. In most cases though, having a domain knowledge helps to avoid under cleaning or over cleaning the data.</p><p>In this section, we’ll go over the data cleaning procedure in python as well. The Jupyter Notebook is available on <a href="http://Craigslist Part 2 Web Scrapping.ipynb">my GitHub</a> if you would like to follow along.</p><p><strong>We’ll use the following workflow albeit loosely.</strong></p><ol><li><strong>Load raw data from the web scraping exercise</strong></li><li><strong>Create new columns as needed</strong></li><li><strong>Dropping unnecessary columns</strong></li><li><strong>Check for erroneous data and clean</strong></li><li><strong>Check for outliers and clean</strong></li><li><strong>Fix incorrect data types</strong></li><li><strong>Save clean data frame to csv</strong></li></ol><p>Ok, let’s get started. First we import all libraries that we will need into our Jupyter notebook environment. Please see the code below for the imports.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7ae8612e18c81189f51f937a017fce35/href">https://medium.com/media/7ae8612e18c81189f51f937a017fce35/href</a></iframe><p>Next, we load and preview the data we obtained from our web scraping session. My data is stored as csv on google drive and I am running a notebook on colab.research.google.com which is Google platform that lets you run notebooks on the cloud. To read data, you just run <em>from google.colab import drive </em>then<em> drive.mount() </em>to mount your google drive and read data from it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8a69034912cfd5f1bab83df8a9e6b6ed/href">https://medium.com/media/8a69034912cfd5f1bab83df8a9e6b6ed/href</a></iframe><p>Here is the snapshot of the imported data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s72W2BJGlvx-Z1kZ6vmsoA.png" /></figure><p>First thing that jumps out is the data time column. We’ll want to create weekdays which will come in handy when we analyze the data. While we’re at it we can create a separate <em>date </em>column as well as <em>timestamp </em>column. Not entirely necessary but useful when you are looking at the data from these two dimensions separately. To create the two new columns, we’re going to use the string function split but first we have to convert the data type to a string.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9d7bd63f61a5711b3936160605eb4022/href">https://medium.com/media/9d7bd63f61a5711b3936160605eb4022/href</a></iframe><p>The resulting columns look as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/374/1*zATkgDHpwr5MdK6ncfLmbw.png" /></figure><p>Next we’ll create both the numeric and categorical week days. The numeric version comes in handy when doing correlations with other columns. Once we have numerical weekdays, we can create a dictionary with day number day name key values and then map the key values to the numerical labels.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9f353319a309674f1e211660aeeaea40/href">https://medium.com/media/9f353319a309674f1e211660aeeaea40/href</a></iframe><p>All the resulting new columns from the <em>data time</em> column are shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*B46Gq2WIa_jeHEDtBxh6Rw.png" /></figure><p>Next column that needs our attention is the <em>year make model </em>column which contains all three attributes in the same string. We’ll need to split this into three separate columns. Here is what the <em>year make model </em>column looks like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/322/1*17Eqbj0ftxem6SJc66zn0w.png" /></figure><p>Immediately, we see another issue that will cause problems down the road. The year occurs twice in the string so splitting say on the first space will create some bad data for second and third columns. We need to remove the extra year where it occurs. To do this we use <em>regular expressions</em> which detect repeated words and remove duplicate words in the <em>year make model </em>column.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/53518ed59fd5e26e8d0b6420d5224b6c/href">https://medium.com/media/53518ed59fd5e26e8d0b6420d5224b6c/href</a></iframe><p>The resulting column after removing duplicates looks as shown below. You can see the duplicate year from row 9 above is now shown once below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/295/1*B1OfpvrgiLu4Lq-2xCBp3g.png" /></figure><p>We can now split the year make model column and grab the first three words for <em>year, make</em> and <em>model </em>columns respectively.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1435d61778063e42af15ae7686adba4a/href">https://medium.com/media/1435d61778063e42af15ae7686adba4a/href</a></iframe><p>Here are the resulting new columns from the code above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/460/1*vedguY1xsrXcaGe9zxijTw.png" /></figure><p>As a final check, we’ll create a histogram using <em>matplotlib.pyplot</em> for all year occurrences in the data to see if our cleaning worked as intended.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1fYwLLXFCPF1XCXalCoAag.png" /></figure><p>It almost worked perfectly if it were not for the few erroneous years shown above. We have to do some further cleaning of the year column. There are many ways to do this but I created a helper column that stores the length of the year column. A proper year is four characters long so we’ll find out which year values don&#39;t conform to this length by examining the values stored in the new helper column called <em>length</em>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3caa2d0b6d8b109ccc0533693cdbb797/href">https://medium.com/media/3caa2d0b6d8b109ccc0533693cdbb797/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2rSTaMyksMgziNnQBHD_0Q.png" /></figure><p>We see that there are three bad rows with the year length as 4,5 and 9. We‘ll just drop these rows all together from the data frame. We should also drop[ the helper column at this stage since we’re done with it. Now we see our histogram doesn’t detect any more bad years formats.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v9qe4RMeuoEYKyy-PE20gw.png" /></figure><p>Next, we look at the data types for each column and change as needed. We just run the <em>pandas.dtypes</em> method to retrieve the data types.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/310/1*HV_9-oR6LauROhPpBGr-qg.png" /></figure><p>The date time and date columns have the right data type of <em>datetime</em>. Lat, long, odometer and price have the <em>float </em>and <em>int </em>data types. So these don&#39;t need any changing. The categorical columns are however stored as <em>object </em>type which would work but we need to tell pandas to convert this to <em>categorical </em>data type categorical variable which is more efficient when working with pandas.</p><p>We should also drop any columns we no longer need at this stage. Like the ‘<em>year make model’</em> column.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/35c80e6d9c8f0eca444f21f62d9d535c/href">https://medium.com/media/35c80e6d9c8f0eca444f21f62d9d535c/href</a></iframe><p>The resulting data types are shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/259/1*wZSbyHZnQcLCw2VHxfizXA.png" /></figure><p>The next thing is to examine the data frame to see how many null values we have in each column.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/edc98b8e6362f8b62139497c274716e6/href">https://medium.com/media/edc98b8e6362f8b62139497c274716e6/href</a></iframe><p>The table below shows the null and the percentage of data they represent for each respective column right besides it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/337/1*LNBj_V_l0yHrZj4QTyeSnw.png" /><figcaption>The null occurrences are sorted from least occurring to the most.</figcaption></figure><p>We see that the VIN number column has the most null values at 85% of all vehicles because most people don’t input this on the craigslist ad. The other columns with a lot of nulls are size, type, drive paint color, cylinders and condition column respectively. In some data cleaning cases we might have to drop these rows all together but in this case we’ll keep them because we’ll lose more than we gain. For example, it’s best to retain the price and odometer information of cars without a VIN number because the price data is more important. This is where a familiarity with the domain kicks in. The key thing here is that the most important columns have almost no null values.</p><p>We’ll now get the basic descriptive statistics of the numerical columns to get an idea on how the data is leaning. We use the <em>pandas.describe</em> method for this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*Qpz1RCqTNknsSBdV_XgKYQ.png" /></figure><p>While lat, long and pID are shown here we can ignore them as they are more or less just labels. We see that the odometer column has a maximum value of 10,000,000 and price a maximum of $123 million. At first glance, this obviously does not make sense so we’ll want to see the data distribution and find the outliers. The weekday column shows a maximum of 6 which is the last day of the week and minimum of 0 being the first day of the week. This makes total sense.</p><p>So let’s create a boxplot of both price and odometer and find the outliers in the data and remove them. Outliers tend to heavily distort the measure of central tendency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/553/1*frpcapQv_mjPQrCdtZfRAg.png" /><figcaption>The outliers are so far out that the boxplot is barely visible.</figcaption></figure><p>Let’s try another type of plot, a histogram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*s2rYv0R6Hnf9jD9W74wfLw.png" /><figcaption>Nearly all of the data falls in one bar which further highlights the outliers.</figcaption></figure><p>We’re going to remove these outliers by using a quartile filter which will eliminate the top 1% and bottom 1% of data. Pandas the elegant <em>pandas.DataFrame.quantile</em> method to help us handle this.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/db9a389b362db4de3a81d9577945f595/href">https://medium.com/media/db9a389b362db4de3a81d9577945f595/href</a></iframe><p>After removing the outliers we can now take a look at the new price distribution using a histogram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*GG2SEVaexaEkjGRqbOwkqQ.png" /></figure><p>The cleaned up price column still has a right skew but is reflective of the real world. We use the same steps to clean up the odometer column.</p><p>Removing the outliers resulted in some lost data so we’ll just examine how much. We started with 24,427 rows of data and ended up with 23,442. Almost 1000 records short or about 4% of the original data. Not a bad compromise for good data as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/629/1*DvixrOaYsfIcjUX0s8HGJw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/1*unenxJ26DvK2Hivon5S0rQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*LBXtxusKyvnoNHUe6Xx0nQ.png" /></figure><p>Finally we have a dataset that we can analyze in confidence. For now we’ll just save it to csv but in the <strong>next blog post we’ll do a proper data exploration</strong>. Just remember that you might have to reassign the right data types when this clean data is loaded from csv.</p><p>In conclusion, I know that we can find more ways to clean the data at hand but most of that will depend on what insights we want to generate from the data. Thanks for following along and please, do add your comments or questions below.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d49bf39dbde" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Website Scraping Using Python in Five Steps — Data Science With Craigslist Data Part 2]]></title>
            <link>https://medium.com/@wahomeron/a-step-by-step-how-to-do-a-data-science-project-with-craigslist-data-part-2-website-scraping-d352351655ee?source=rss-ec73e9699e73------2</link>
            <guid isPermaLink="false">https://medium.com/p/d352351655ee</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[craigslist]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Ronald Wahome]]></dc:creator>
            <pubDate>Tue, 19 Oct 2021 12:51:28 GMT</pubDate>
            <atom:updated>2021-10-31T17:20:54.074Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ce9QS__fDzsELbhaOA6MVg.png" /></figure><p>In this post I’ll walk you step by step through the steps I took to scrape vehicle postings data from Craigslist. I used Python and the BeautifulSoup library to web scrape which is just one of the many ways to access data from a website. Ideally, an API is best for this purpose but Craigslist does not have one that would let me get the data on I needed.</p><p>The full code is posted on my <a href="https://github.com/RonKG/Craigslist-Cars-For-Sale-Data-Project">GitHub </a>if you would like to follow along on a Jupyter Notebook.</p><p>If navigate to <em>car &amp; trucks</em> for any city on Craigslist you’ll have the option to view ads from both private sellers and dealers. I chose to only scrape data from private sellers as these posts contain more reliable information relating to an actual vehicle up for sale. I also chose posts with images since I wanted to download the images for the advanced project section like image recognition or NLP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qckyptsUV0aS63kff-d-Cw.png" /></figure><p>The data I needed is then embedded within each vehicle post page as seen on the right in the image below. Don’t worry if not everything in the following image makes sense. We’ll get to that soon.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bCE9xAq3AG5-woOvpvuCGA.png" /></figure><p>After visually inspecting the website I was able to come up with game plan of how I would structure my code.</p><h3>On a high level, my five steps workflow for scraping the website will be as follows:</h3><ol><li><strong>Step One:</strong> Create a list of cities which I would like to get the data for</li><li><strong>Step Two: </strong>Navigate to the <em>cars &amp; trucks for sale by owner</em> for each city</li><li><strong>Step Three: </strong>For each city, navigate to each page available</li><li><strong>Step Four: </strong>For each page available, navigate to each post</li><li><strong>Step Five: </strong>For each post, get post attributes</li></ol><p>That&#39;s the working sudo code version.</p><h3>The Code</h3><p>First things first, I fired up an instance Jupyter notebook and imported the necessary libraries. We import the <em>requests </em>module to handle http requests and <em>BeautifulSoup </em>from <em>bs4 </em>which is the module for getting data from the website.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/29888a2c614120ac9f090a514c24951b/href">https://medium.com/media/29888a2c614120ac9f090a514c24951b/href</a></iframe><p>The next step is to create a list of cities that we want to navigate to on Craigslist. These city names must match the same spelling as is on Craigslist. For example San Francisco is sfbay otherwise you’ll get a page not found error.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4df60b330622c53c5d1fde651f5f2c40/href">https://medium.com/media/4df60b330622c53c5d1fde651f5f2c40/href</a></iframe><p>We’re now going to get the links to each page within the city then we’ll have a list of all pages for each city. Each page has 120 posts. Therefore, to go to the next page we just need to create and increment a page placeholder by 120 to get the next page. We continue this iteration until we get to page 1800 or less. I chose 1800 because posts on that page are usually around the thirty day mark. The final result is a list with links to all pages for each city.</p><p>To get link to each city page, we’ll use a loop to change the city name in the base city link from the <em>cities </em>list and store it in a variable named <em>city_link. </em>What we’re doing is just taking any city link from the website and creating a placeholder link with an option to change the city name and page number. Something like this “<a href="https://sfbay.craigslist.org/d/cars-trucks-by-owner/search/cto?s=120&amp;hasPic=1">https://<strong>sfbay</strong>.craigslist.org/d/cars-trucks-by-owner/search/cto?s=<strong>1</strong>&amp;hasPic=1</a>” we change into the code shown below on line 12.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7320257dd7352450a5ea4a0cd137ce5d/href">https://medium.com/media/7320257dd7352450a5ea4a0cd137ce5d/href</a></iframe><p>After we run the above code we now have a list of links to pages available in each of the cities in out <em>cities </em>list. Below is an slice of what the page links look like. For all 15 cities we get a total of 225 page links, each containing 120 individual vehicle posts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*U-okCwzpxMW7PK-QyJzEdg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/1*X1HDqHvVI-qc4Guhmrfaiw.png" /></figure><p>Now that we have this long list of all pages contained in every city for the <em>car for sale by owner</em> section, we need to navigate to each page and get the links to each individual vehicle post. That’s 120 posts by 225 pages. In all we should end up with about 27,000 links. All 27,000 represent a specific vehicle post on Craigslist.</p><p>To get the vehicle links, we loop through each page link from the list we created above and get all 120 posts per page. We store these links in a list variable called <em>car_urls</em>.</p><p>To get the link to each post, we’re going to use BeautifulSoup to parse the links from the webpage. To do this, we pass a link to the get method from the request library which should return a response code <em>&lt;Response [200]&gt;</em> if the link is present. We then pass the request object to an instance of BeautifulSoup and add a <em>html.parser</em> flag. This then returns a html object as seen below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qAGzMJ6paIr_37oXKsO10A.png" /><figcaption>This is the html object returned by BeautifulSoup</figcaption></figure><p>We can then tell the <em>soup </em>object to find our vehicle links in the returned <em>html </em>object by using the <em>find_all </em>method.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cu7bSt0fEunZn1X2KrUEug.png" /><figcaption>This is the find_all results</figcaption></figure><p>You can easily figure out where the links are by right clicking on your chrome browser and selecting inspect. This brings up the panel shown below with the html structure of the page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e07RxjrbZtaP_6E53bq7xg.png" /><figcaption>The class_= ‘result-image gallery’ houses all vehicle links in the page</figcaption></figure><p>A html link is denoted by the <em>href</em> tag but since there are many hrefs in the html object, we also want to tell soup which <em>class </em>to look in. From the example above, the <em>class </em>we want to look into is <em>result-image gallery </em>class. All the hrefs contained lead to the individual vehicle posts on that page as seen below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vj-o26-u3rUtXEkoimO_Lg.png" /><figcaption>hrefs contained within the “result-image gallery” class</figcaption></figure><p>We can then store these links into a data frame or csv for later viewing. The full code to get all the vehicle links is shown below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/db8ac905909c3a644be59695545f5090/href">https://medium.com/media/db8ac905909c3a644be59695545f5090/href</a></iframe><p>We see that now we have links to 24706 vehicle posts that we can now point BeautifulSoup to scrape data from but first we save a copy so we can always get back to it later. This step is easy and we just save it as a csv file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/1*EYWfCEPW3NRh6fdcWsIttg.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/89b95975196e13bff40c37cb6448dd85/href">https://medium.com/media/89b95975196e13bff40c37cb6448dd85/href</a></iframe><p>Then read the file back out to a pandas data frame.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1f2693c12567f41e434d98e1dca27cae/href">https://medium.com/media/1f2693c12567f41e434d98e1dca27cae/href</a></iframe><p>Well, now that we have all the links we need, lets get to scraping the vehicle details. We are going to use BeautifulSoup again for this but this time we’ll look through each vehicle post page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ce9QS__fDzsELbhaOA6MVg.png" /><figcaption>Some visible vehicle attributes that we want to scrape</figcaption></figure><p>Again, we pass each vehicle link to the <em>request </em>module and the result to a BeautifulSoup object. Then we use the find methods of the BS object to grab the vehicle attributes and any other data from the page. For example, for the price we find the price <em>class </em>as follows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/511/1*nDla-rYk4Y_4GiakIVMRyw.png" /></figure><p>To find the date time we use the following.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*R3oXdyasjgyXqWWKobZGBg.png" /></figure><p>For the geolocations, we parse them from the <em>mapbox class </em>of the html like so.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/439/1*lerMZ4Rf4q__atUOWMuAeg.png" /></figure><p>Here is the full code for getting vehicle attributes from the each vehicle link.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d2393a0ae4ddd9fb2a7090b874267103/href">https://medium.com/media/d2393a0ae4ddd9fb2a7090b874267103/href</a></iframe><p>Now that we got all the vehicle attributes from our 27K vehicle links and stored it in a list of lists. Some minimal cleaning was also done on the fly like creating attribute labels and removing the dollar sign and commas from the price.</p><p>Each vehicle and it’s attributes is stored in the list <em>car_final .</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*VjLWxGarJh2K29aleqYRcQ.png" /><figcaption>Each vehicle’s attributes stored in a list</figcaption></figure><p>And each list is stored in the list <em>cars</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x47SO4x0yoAUMY54Yyimbg.png" /></figure><p>Next, we have to store this data for future analysis. What I did was create dictionaries with attribute labels acting as key names. To do this I created a dictionary of key, value pairs where each key represented a column name and then stored the dictionaries in the list <em>car_dicts</em>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/45e32814479238b139ca75ae4cd30daf/href">https://medium.com/media/45e32814479238b139ca75ae4cd30daf/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*Hc_2dKG-bs6n5yIcifsCQA.png" /></figure><p>Finally write the list of dictionaries in to a pandas data frame.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/75daf95cf2b0bf7a5a0f4e7b636fd22a/href">https://medium.com/media/75daf95cf2b0bf7a5a0f4e7b636fd22a/href</a></iframe><p>Then preview the data frame to see if our data was stored properly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oR-K1hVimAB-iB7-lHn9Ag.png" /></figure><p>And finally save the data frame to csv or dump to a data base.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e858bcc0eddb71bbc5386422c3ae5be9/href">https://medium.com/media/e858bcc0eddb71bbc5386422c3ae5be9/href</a></iframe><p>That’s all folks. For now at least. We have successfully scraped data from Craigslist. We now have data to 24 thousand plus vehicles from 15 cities across the US from a period of one month.</p><p>Please join me in the next section, <strong>C</strong><a href="https://medium.com/p/2d49bf39dbde"><strong>raigslist Cars For Sale Data Project: Part 3 — Data Cleaning</strong></a>, as we perform some much needed data cleaning like fixing the data types and creating new columns. Also let me know if you have any feedback on the process above. Thank you!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d352351655ee" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Step By Step How To Do A Data Science Project With Craigslist Data: Part 1 — Some Background]]></title>
            <link>https://medium.com/@wahomeron/a-step-by-step-how-to-do-a-data-science-project-with-craigslist-data-part-1-some-background-215b4d1efd5f?source=rss-ec73e9699e73------2</link>
            <guid isPermaLink="false">https://medium.com/p/215b4d1efd5f</guid>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-cleaning]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Ronald Wahome]]></dc:creator>
            <pubDate>Tue, 19 Oct 2021 12:44:06 GMT</pubDate>
            <atom:updated>2021-10-31T20:34:58.890Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Science With Craigslist Data <strong>Part 1 — Some Background</strong></h3><p>In my regular day job as a data professional in an insurance company, I engage in the usual data science processes to create value for my employer but I am always looking for side projects where I can enhance my data science skills as well as learn new ones. In the business world, data analysis requirements tend to be very clear but to find the answers within the data, a sense of curiosity is a must.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cTiJ3OyUWF_lxCTf" /></figure><p>This sense of curiosity creeps into my personal life a lot including one of my biggest hobbies which is cars. Really, I watch more YouTube car videos than I do Netflix and other video streaming services combined. I also browse car websites for cool cars but also browse Craigslist to see which ones are on sale near me. Looking at all this Craigslist data got me asking more questions. Like if I see a car I like, can I get it in another city for cheaper, better maintenance history, better mileage or should I wait a week or a month for a better deal. You get the picture.</p><p>This got me thinking about how I could get to the raw data to answer all this questions in my mind. You see, the thing is, if you work with data a lot, then you always want the raw data. Looking through the Craigslist web interface just gives you access to the a filtered view of the raw data. A very skilled person put that view together but it doesn’t answer all my questions. I want more data.</p><p>I did a bit of research and found out that Craigslist does have an API but you cannot pull data from it. You can however create post through it. This was not ideal as I wanted to get my hands on the raw data. However, you can legally scrape data from the website as long as you are responsible about it.</p><p>For my data requirements, I needed all available data points for each vehicle classified ad. These data points are:</p><ol><li>All vehicle attributes available. Make, model, year, cylinders, etc.</li><li>Geographical attributes of each post. City, Longitude and latitude, etc.</li><li>Date and timestamp of post</li><li>The text body of each post</li><li>Images for each vehicle post</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*z_DYkCuaO2-6khx1" /></figure><h4><strong>In the next few blog posts</strong>,<strong> I am going to document each stage of the data science process from end to end as follows</strong>:</h4><ol><li><a href="https://medium.com/@wahomeron/a-step-by-step-how-to-do-a-data-science-project-with-craigslist-data-part-2-website-scraping-d352351655ee">Webs Scraping Using Python</a></li><li>Downloading Craigslist Images in Python</li><li><a href="https://medium.com/p/2d49bf39dbde">Data Cleaning</a></li><li><a href="https://medium.com/p/169079a555ab">Exploratory Data Analysis (<strong>EDA</strong>)</a></li><li>Data Insights and Visualization</li><li>Advanced Analytics (from images and posts body)</li><li>Conclusion and Lessons Learned</li></ol><p>All code will be posted on <a href="https://github.com/RonKG/Craigslist-Cars-For-Sale-Data-Project">my personal GitHub</a> repo.</p><p>Please join me in the next post, <a href="https://medium.com/@wahomeron/a-step-by-step-how-to-do-a-data-science-project-with-craigslist-data-part-2-website-scraping-d352351655ee"><strong>Part 2 — Website Scraping Using Python</strong></a>, where I’ll walk you step by step on how I scraped 24,000 vehicle ads postings from Craigslist using python.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=215b4d1efd5f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[This Is How Twitter Sees The World : Sentiment Analysis Part Two]]></title>
            <link>https://medium.com/data-science/the-real-world-as-seen-on-twitter-sentiment-analysis-part-two-3ed2670f927d?source=rss-ec73e9699e73------2</link>
            <guid isPermaLink="false">https://medium.com/p/3ed2670f927d</guid>
            <category><![CDATA[sentiment-analysis]]></category>
            <category><![CDATA[scikit-learn]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Ronald Wahome]]></dc:creator>
            <pubDate>Fri, 07 Sep 2018 22:53:06 GMT</pubDate>
            <atom:updated>2018-09-09T16:18:50.113Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1004/1*Ffl2ZZPfmF7L7K04AIbpuw.png" /><figcaption>image by <a href="https://commons.wikimedia.org/wiki/File:State_Choropleth.png">Deepthiyathiender</a></figcaption></figure><p>This blog post is a continuation from a <a href="https://medium.com/@wahomeron/the-real-world-as-seen-on-twitter-sentiment-analysis-part-one-5ac2d06b63fb">previous post</a> that goes over the inner workings of text preprocessing for a machine learning(ML) sentiment analysis task. In this post we’ll split the data from <a href="http://www.sentiment140.com/">setiment140</a> into training and test sets. After training the model, we will then use it to classify sentiment on unseen twitter data that has been preprocessed in the same manner as the training data.</p><h3>Section 1 : Model Training</h3><h4>Naive Bayes Classifier</h4><p>In this classification task we are going to use a <strong>Naive Bayes (NB)</strong> classifier which is based on the <a href="https://en.wikipedia.org/wiki/Bayes%27_theorem">Bayes Theorem</a>. In a nutshell, the NB classifier assumes that the presence of a particular feature in a class is unrelated to the presence of any other feature. So for example, a fruit may be considered to be an orange if it is orange in color, round, and about 3″ in diameter. A Naive Bayes classifier considers each of these “features” (orange, round, 3” in diameter) to contribute independently to the probability that the fruit is an apple, regardless of any correlations between features. Features, however, aren’t always independent which is often seen as a shortcoming of the Naive Bayes algorithm and this is why it’s labeled “naive”. However it is relatively simple to understand, build and train and mostly very fast all which makes it a good candidate for sentiment classification.</p><p>Bayes theorem provides a way of calculating posterior probability P(c|x) from P(c), P(x) and P(x|c). Look at the equation below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*1a8BdMzkO0IKMNna.png" /></figure><pre>- P(c|x) is the posterior probability of class (c, target) given predictor (x, attributes).<br>- P(c) is the prior probability of class.<br>- P(x|c) is the likelihood which is the probability of predictor given class.<br>- P(x) is the prior probability of predictor.</pre><h4>Create Pipeline to manage the preprocessing steps in one step</h4><ul><li><a href="http://scikit-learn.org/stable/">Scikit Learn</a> provides a <a href="http://scikit-learn.org/stable/modules/pipeline.html">pipeline</a> capability that lets you define a pipeline workflow which will take all the above steps and even a classifier and grid search parameters.</li><li>Pipelines make code more readable and makes it easy to swap out pipeline pieces (which can include other ML algorithms and try different configurations).</li><li>Pipelines also allow for Cross Validation on to the model workflow.</li><li>Pipelines also help avoid leaking statistics from your test data into the trained model in cross-validation, by ensuring that the same samples are used to train the transformers and predictors.</li></ul><h4>Cross Validation:</h4><ul><li>The recommended method for training a good model is to first cross-validate using a portion of the training set itself to check if you have used a model with too much capacity (i.e. if the model is overfitting the data).</li><li>To cross-validate and select the best parameter configuration at the same time, we use <a href="http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html">GridSearchCV.</a></li><li>This allows us to easily test out different hyperparameter configurations using for example the KFold strategy to split the model into random parts to find out if it’s generalizing well or if it’s overfitting.</li><li>GridSearchCV allows you do define a ParameterGrid with hyperparameter configuration values to iterate over. All combinations are tested and scored and the best model returned.</li><li>For our case, there are 4 + 2 + 2 parameter combinations to test and 10 kfold validations, so the model will be trained and tested on the validation set 8 x 10 = 80 times.</li></ul><pre><em># Run Train Data Through Pipeline analyzer=text_process</em><br><em># uncomment below to train on a larger dataset but it&#39;s very slow for a slower machine.</em></pre><pre><em># X_train, X_test, y_train, y_test = train_test_split(data[&#39;text&#39;], data[&#39;label&#39;], test_size=0.2)</em><br>X_train, X_test, y_train, y_test = train_test_split(data[&#39;text&#39;][:5000], data[&#39;label&#39;][:5000], test_size=0.2)</pre><pre><em># create pipeline</em><br>pipeline = Pipeline([<br>    (&#39;bow&#39;, CountVectorizer(strip_accents=&#39;ascii&#39;,<br>                            stop_words=&#39;english&#39;,<br>                            lowercase=<strong>True</strong>)),  <em># strings to token integer counts</em><br>    (&#39;tfidf&#39;, TfidfTransformer()),  <em># integer counts to weighted TF-IDF scores</em><br>    (&#39;classifier&#39;, MultinomialNB()),  <em># train on TF-IDF vectors w/ Naive Bayes classifier</em><br>])</pre><pre><em># this is where we define the values for GridSearchCV to iterate over</em><br>parameters = {&#39;bow__ngram_range&#39;: [(1, 1), (1, 2)],<br>              &#39;tfidf__use_idf&#39;: (<strong>True</strong>, <strong>False</strong>),<br>              &#39;classifier__alpha&#39;: (1e-2, 1e-3),<br>             }</pre><pre><em># do 10-fold cross validation for each of the 6 possible combinations of the above params</em><br>grid = GridSearchCV(pipeline, cv=10, param_grid=parameters, verbose=1)<br>grid.fit(X_train,y_train)</pre><pre><em># summarize results</em><br>print(&quot;<strong>\n</strong>Best Model: <strong>%f</strong> using <strong>%s</strong>&quot; % (grid.best_score_, grid.best_params_))<br>print(&#39;<strong>\n</strong>&#39;)<br>means = grid.cv_results_[&#39;mean_test_score&#39;]<br>stds = grid.cv_results_[&#39;std_test_score&#39;]<br>params = grid.cv_results_[&#39;params&#39;]<br><strong>for</strong> mean, stdev, param <strong>in</strong> zip(means, stds, params):<br>    print(&quot;Mean: <strong>%f</strong> Stdev:(<strong>%f</strong>) with: <strong>%r</strong>&quot; % (mean, stdev, param))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*HrWfjwtGIbHHCcfIYGPv4Q.png" /></figure><ul><li>We passed a combined 8 parameters to the GridsearchCV object and 10 folds for the cross validation which means that for every parameter combination, the grid will run 10 different iterations with a different test set every time.</li><li>After trying out the different model parameter combinations, the GridsearchCV returns the best performing model which we can use to classify new (twitter) data. We save the model below to the working directory for when we want to retrieve the trained model in the future without retraining it.</li><li>Saving the model is also a necessary step if you plan to deploy the model somewhere else like a mobile app or web app.</li></ul><h3>Section 2 : Model Evaluation</h3><p>In the code below we will test our trained model on the test holdout dataset.</p><pre><em># save best model to current working directory</em><br>joblib.dump(grid, &quot;twitter_sentiment.pkl&quot;)</pre><pre><em># load from file and predict using the best configs found in the CV step</em><br>model_NB = joblib.load(&quot;twitter_sentiment.pkl&quot; )</pre><pre><em># get predictions from best model above</em><br>y_preds = model_NB.predict(X_test)</pre><pre>print(&#39;accuracy score: &#39;,accuracy_score(y_test, y_preds))<br>print(&#39;<strong>\n</strong>&#39;)<br>print(&#39;confusion matrix: <strong>\n</strong>&#39;,confusion_matrix(y_test,y_preds))<br>print(&#39;<strong>\n</strong>&#39;)<br>print(classification_report(y_test, y_preds))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*7oFfls2X_3HKCxurTtJZDQ.png" /></figure><ul><li>In the cell above I used the best model to perform predictions on the unseen test data which lets us grade and retrieve the performance metrics.</li><li>Among some of the performance metrics we can get back from above is the classification report and a confusion matrix.</li><li>A <a href="https://en.wikipedia.org/wiki/Confusion_matrix">Confusion Matrix (CM)</a> is pretty straight forward but below is a key for interpreting it as well as the deriving a classification report:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/0*yWQLruLQDbHTaqmK.png" /></figure><p><strong>From the accuracy metric above:</strong></p><ul><li>95% is the number of times the model predicted the correct label over all labels in the dataset.</li><li>When the distribution of the classes in data is well balanced, accuracy can give you a good picture of how the model is performing.</li></ul><p><strong>From the confusion matrix:</strong></p><ul><li>The model predicted 553 labels correctly as Negative and 799 labels correctly as Positive.</li><li>We also got 4 labels predicted as Positives even though they are Negatives (False Negatives).</li><li>Another thing we can tell from the CM is that the model predicted 62 labels as Negatives but they turned out to be Positives (False Positives).</li></ul><p><strong>From the classification report:</strong></p><p>The classification report which can be obtained from the metrics in the confusion matrix, gives up more detailed information on model performance.</p><ul><li><strong>Precision</strong>: 99% for label 0 and 93% for label 1. This number tells us what proportion of the labels were predicted correctly out of the total predictions for that class.</li><li><strong>Recall</strong>: 90% for label 0 and 100% for label 1. This is the number of correct predictions out of true labels for that class.</li><li><strong>f1 -score</strong>: This is the weighted average of precision and recall for that class. It generally gives the bigger picture of how the model is performing for that label and obviously the higher this number is the better. 94 % for label 0 and 96% for label 1. 0 being negative sentiment and 1 being positive sentiment.</li><li>From the above metrics, the model seems to be performing relatively well even though it could do better on predicting the right labels for class 0.</li><li>According to research human raters typically agree 80%[Roebuck, K. (2012–10–24)] of the time because of <a href="https://en.wikipedia.org/wiki/Inter-rater_reliability">Inter-rater reliability issues</a>.</li><li>Therefore a 70% accurate program is doing almost as good as humans, even though such accuracy may not seem all that impressive. If a program were “right” 100% of the time, humans would still disagree with it about 20% of the time, since they disagree that much about any answer.</li><li>We can also benchmark our model against similar work that performed <a href="https://www.sciencedirect.com/science/article/pii/S1877050917328089">an extensive study of sentiment analysis and binary classification of sentiment labels</a>. The final model was based on Support Vector Machine(<a href="https://en.wikipedia.org/wiki/Support_vector_machine">SVM</a>) and went on to achieve an accuracy of 79.08%.</li><li>So far we have taken steps to avoid over fitting with the way we built our model pipeline but ultimately the true test is testing the model on unseen data.</li></ul><h3>Section 3: Real-world Model Performance</h3><p>In the next step, I would like to run sentiment predictions with our model on the data we acquired through the twitter API and evaluate it’s performance. Predicting and previewing the results is done in the code below.</p><pre><em># run predictions on twitter data</em><br>tweet_preds = model_NB.predict(df_twtr[&#39;message&#39;])</pre><pre><em># append predictions to dataframe</em><br>df_tweet_preds = df_twtr.copy()<br>df_tweet_preds[&#39;predictions&#39;] = tweet_preds<br>df_tweet_preds.shape</pre><pre># Output<br>(4164, 8)</pre><p>View some sample predictions below.</p><p>0 = Positive and 1 = Negative.</p><pre><em># print text and sentiment</em></pre><pre>index = random.sample(range(tweet_preds.shape[0]), 20)<br><strong>for</strong> text, sentiment <strong>in</strong> zip(df_tweet_preds.message[index],<br>                           df_tweet_preds.predictions[index]):<br>    print (sentiment, &#39;--&#39;, text, &#39;<strong>\n</strong>&#39;)</pre><pre>#------------------------------<br># SAMPLE PREDICTIONS BELOW    #<br>#------------------------------</pre><pre>0 -- Spot on correct. The Paul Ryan Story: From Flimflam to Fascism <a href="https://t.co/BPwrobl0aS">https://t.co/BPwrobl0aS</a></pre><pre>0 -- .@Varneyco @charleshurt   Paul Ryan didn&#39;t get Tax Reform.  Donald Trump got Tax Reform.  Boy George could have bee… <a href="https://t.co/vGmRJWQnlG">https://t.co/vGmRJWQnlG</a></pre><pre>1 -- @HumanBeings1st @LotraineH Paul Ryan is cold-hearted👎🏼👹He hates Social Security and Medicare and Medicaid and has t… <a href="https://t.co/oQAXvfEuLI">https://t.co/oQAXvfEuLI</a></pre><pre>0 -- Paul Ryan is an unconvincing charlatan whose primary goal in life has been to make life harder for poor people. The… <a href="https://t.co/yyK80Tsgmz">https://t.co/yyK80Tsgmz</a></pre><pre>0 -- absolutely delish...#winning</pre><pre>Meghan McCain Erupts After Audience Cheers Paul Ryan Retirement: ‘You Deserve Trump’ <a href="https://t.co/7s2AUVGrgp">https://t.co/7s2AUVGrgp</a></pre><pre>0 -- Is anyone surprised Paul Ryan isn’t going to run? #morningjoe</pre><pre>0 -- @CNN If the Repubs stand by and continue to let the lunatic run the asylum, then they are accountable, if the Repub… <a href="https://t.co/uNNkd9yljp">https://t.co/uNNkd9yljp</a></pre><pre>0 -- .@IronStache wins eight months early <a href="https://t.co/7txWs0YvER">https://t.co/7txWs0YvER</a></pre><p>Below I tested the model on some text from the internet to get a better feel of how it would perform on some random data.</p><pre><em># Testing random text from the internet</em></pre><pre><em># load model</em><br>model_NB = joblib.load(&quot;twitter_sentiment.pkl&quot; )</pre><pre><em># test string</em><br>sample_str = &quot;&quot;&quot;While ride-sharing first mover Uber has fallen on tough times with<br>scandal and abyssal track records of leadership, and cash burning<br>growth-orientated practices, the world has caught up with self-driving<br>tech with many players now in the race.&quot;&quot;&quot;</pre><pre>p = model_NB.predict([sample_str])</pre><pre><em># formatting helper</em><br><strong>def</strong> sentiment_str(x):<br>    <strong>if</strong> x==0:<br>        <strong>return</strong> &#39;Negative&#39;<br>    <strong>else</strong>:<br>        <strong>return</strong> &#39;Positive&#39;<br><em>#_____________________________________________</em></pre><pre><em># test result ___ 0=Negative, 1=Positive</em><br>print(&quot;the sentence: <strong>\n\n</strong>&#39;<strong>{}</strong>&#39; <strong>\n\n</strong>has a <strong>{}</strong> sentiment&quot;.format(sample_str,sentiment_str(p[0])))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/558/1*WuQRydHSb8FRUP_jxC8ntw.png" /></figure><h3>Observations</h3><ul><li>I have visualized 10 examples of tweets classified by our model in the cells above which seems to perform mostly right but sometimes fails embarrassingly.</li><li>When tested on non-twitter text, the model seems to pick up a lot of the embedded sentiment but needs more work to extract the general conclusion from said text. For example, we may need to train it on an entire paragraph as opposed to just one sentence.</li><li>One tricky thing about sentiment analysis is that even if the model performs well by some benchmark, the final performance will have to be left to actual human judges which are hard to quantify.</li><li>Having said that, I viewed lots of random negatively classified and positively classified tweets and there are some really good classifications and then some classifications that don’t seem to belong anywhere. Which also makes me think that we should explore a third classification class. So we’d have <strong>Positive, Negative and Neutral sentiment</strong>.</li><li>While I think the model has done some learning, I think we can do much better by tweaking the model’s parameters, exploring other algorithms or create an ensemble of algorithms.</li></ul><p>Compare model prediction stats on twitter data</p><pre>pos = df_tweet_preds.predictions.value_counts()[0]<br>neg = df_tweet_preds.predictions.value_counts()[1]</pre><pre>print(&#39;Model predictions: Positives - <strong>{}</strong>, Negatives - <strong>{}</strong>&#39;.format(neg,pos))</pre><pre><em># save dataframe with appended preditions </em><br>df_tweet_preds.to_pickle(&#39;paulry_predicts_df.p&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/482/1*HbqefawEtuH8HOszf4O3dg.png" /></figure><h3>Section 7: Geoplots, Final Thoughts and Conclusions</h3><ul><li>In this section we have also created geographic visualizations of the downloaded tweets which is an important part of data storytelling.</li><li>While training any ML model an important part is to be able to present our findings in ways that facilitate decision making by the stakeholders.</li><li>We will load the dataframe containing predictions and explore country and US states distribution and especially since our twitter data revolve around US news.</li><li>It is worth mentioning that there are limitations with our dataset going forward. Majority of twitter users do not broadcast their geo location but our search criteria only pulled tweets that had location information enabled.</li><li>It is therefore possible that we are missing a lot of tweets that would paint a different picture than we come up with when plotted on a map. Knowing this is crucial in interpreting results but also but also understanding how we can make the model more robust.</li></ul><pre><em># load dataframe with predictions</em><br>df = pd.read_pickle(&#39;paulry_predicts_df.p&#39;)</pre><pre><em># get all the countries in dataset</em><br>df.country.unique()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/697/1*4-vxDFXQchd2SOTmEqOwlA.png" /></figure><pre><em># get top countries in the dataset by percentage of tweets</em><br>df.country.value_counts(1).head(10)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/248/1*vMnG0Fud18u022BJOSVTyg.png" /></figure><p>In [132]:</p><pre><em># plot histogram of tweets counts by country of origin</em><br>sns.set_style(&quot;darkgrid&quot;)<br>x = df.country.value_counts(1).head(20)<br>x.plot(kind=&#39;bar&#39;,figsize=(10,6),fontsize=13,color=&#39;steelblue&#39;)<br>plt.ylabel(&#39;<strong>% o</strong>f Total Tweets&#39;, fontsize=13)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/637/1*VF5hS_-ugTESTDPIH75r5Q.png" /></figure><ul><li>Just like we suspected, an overwhelming majority of tweets originate from the US. Next we will get the state by state counts.</li></ul><pre><em># get latitudes and longitudes</em></pre><pre><em># some helper funtions to get longs and lats</em><br><strong>def</strong> lats(x):<br>    <strong>return</strong> x[1]</pre><pre><strong>def</strong> longs(x):<br>    <strong>return</strong> x[0]</pre><pre><em># --------------------------------------------------------#</em><br><em># append longs and lats to dframe</em><br>df[&#39;latitude&#39;] = df[&#39;geo_code&#39;].apply(lats)<br>df[&#39;longitude&#39;] = df[&#39;geo_code&#39;].apply(longs)<br>df.columns</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/1*elBtGS_AuZEBsILhA4B-Wg.png" /></figure><pre><em># for US tweets extract state abbreviations for a new STATE column</em><br><em># helper function to extract state origin of every tweet</em><br><strong>def</strong> get_state(x):<br>    <br>    states = [&quot;AL&quot;, &quot;AK&quot;, &quot;AZ&quot;, &quot;AR&quot;, &quot;CA&quot;, &quot;CO&quot;, &quot;CT&quot;, &quot;DC&quot;, &quot;DE&quot;, &quot;FL&quot;, &quot;GA&quot;, <br>              &quot;HI&quot;, &quot;ID&quot;, &quot;IL&quot;, &quot;IN&quot;, &quot;IA&quot;, &quot;KS&quot;, &quot;KY&quot;, &quot;LA&quot;, &quot;ME&quot;, &quot;MD&quot;, <br>              &quot;MA&quot;, &quot;MI&quot;, &quot;MN&quot;, &quot;MS&quot;, &quot;MO&quot;, &quot;MT&quot;, &quot;NE&quot;, &quot;NV&quot;, &quot;NH&quot;, &quot;NJ&quot;, <br>              &quot;NM&quot;, &quot;NY&quot;, &quot;NC&quot;, &quot;ND&quot;, &quot;OH&quot;, &quot;OK&quot;, &quot;OR&quot;, &quot;PA&quot;, &quot;RI&quot;, &quot;SC&quot;, <br>              &quot;SD&quot;, &quot;TN&quot;, &quot;TX&quot;, &quot;UT&quot;, &quot;VT&quot;, &quot;VA&quot;, &quot;WA&quot;, &quot;WV&quot;, &quot;WI&quot;, &quot;WY&quot;]</pre><pre>states_dict = {<br>            &#39;AK&#39;: &#39;Alaska&#39;,&#39;AL&#39;: &#39;Alabama&#39;,&#39;AR&#39;: &#39;Arkansas&#39;,&#39;AS&#39;: &#39;American Samoa&#39;,<br>            &#39;AZ&#39;: &#39;Arizona&#39;,&#39;CA&#39;: &#39;California&#39;,&#39;CO&#39;: &#39;Colorado&#39;,&#39;CT&#39;: &#39;Connecticut&#39;,<br>            &#39;DC&#39;: &#39;District of Columbia&#39;,&#39;DE&#39;: &#39;Delaware&#39;,&#39;FL&#39;: &#39;Florida&#39;,&#39;GA&#39;: &#39;Georgia&#39;,<br>            &#39;GU&#39;: &#39;Guam&#39;,&#39;HI&#39;: &#39;Hawaii&#39;,&#39;IA&#39;: &#39;Iowa&#39;,&#39;ID&#39;: &#39;Idaho&#39;,&#39;IL&#39;: &#39;Illinois&#39;,<br>            &#39;IN&#39;: &#39;Indiana&#39;,&#39;KS&#39;: &#39;Kansas&#39;,&#39;KY&#39;: &#39;Kentucky&#39;,&#39;LA&#39;: &#39;Louisiana&#39;,<br>            &#39;MA&#39;: &#39;Massachusetts&#39;,&#39;MD&#39;: &#39;Maryland&#39;,&#39;ME&#39;: &#39;Maine&#39;,&#39;MI&#39;: &#39;Michigan&#39;,<br>            &#39;MN&#39;: &#39;Minnesota&#39;,&#39;MO&#39;: &#39;Missouri&#39;,&#39;MP&#39;: &#39;Northern Mariana Islands&#39;,<br>            &#39;MS&#39;: &#39;Mississippi&#39;,&#39;MT&#39;: &#39;Montana&#39;,&#39;NA&#39;: &#39;National&#39;,&#39;NC&#39;: &#39;North Carolina&#39;,<br>            &#39;ND&#39;: &#39;North Dakota&#39;,&#39;NE&#39;: &#39;Nebraska&#39;,&#39;NH&#39;: &#39;New Hampshire&#39;,&#39;NJ&#39;: &#39;New Jersey&#39;,<br>            &#39;NM&#39;: &#39;New Mexico&#39;,&#39;NV&#39;: &#39;Nevada&#39;,&#39;NY&#39;: &#39;New York&#39;,&#39;OH&#39;: &#39;Ohio&#39;,&#39;OK&#39;: &#39;Oklahoma&#39;,<br>            &#39;OR&#39;: &#39;Oregon&#39;,&#39;PA&#39;: &#39;Pennsylvania&#39;,&#39;PR&#39;: &#39;Puerto Rico&#39;,&#39;RI&#39;: &#39;Rhode Island&#39;,<br>            &#39;SC&#39;: &#39;South Carolina&#39;,&#39;SD&#39;: &#39;South Dakota&#39;,&#39;TN&#39;: &#39;Tennessee&#39;,&#39;TX&#39;: &#39;Texas&#39;,<br>            &#39;UT&#39;: &#39;Utah&#39;,&#39;VA&#39;: &#39;Virginia&#39;,&#39;VI&#39;: &#39;Virgin Islands&#39;,&#39;VT&#39;: &#39;Vermont&#39;,<br>            &#39;WA&#39;: &#39;Washington&#39;,&#39;WI&#39;: &#39;Wisconsin&#39;,&#39;WV&#39;: &#39;West Virginia&#39;,&#39;WY&#39;: &#39;Wyoming&#39;<br>    }</pre><pre>abv = x.split(&#39;,&#39;)[-1].lstrip().upper()<br>    state_name = x.split(&#39;,&#39;)[0].lstrip()<br>    <strong>if</strong> abv <strong>in</strong> states:<br>        state = abv<br>    <strong>else</strong>:<br>        <strong>if</strong> state_name <strong>in</strong> states_dict.values():<br>            state = list(states_dict.keys())[list(states_dict.values()).index(state_name)]<br>        <strong>else</strong>:<br>            state = &#39;Non_USA&#39;    <br>    <strong>return</strong> state</pre><pre><em># ____________________________________________________________________________</em></pre><pre><em># create abreviated states column</em><br>df = df.copy()<br>df[&#39;states&#39;] = df[&#39;full_name&#39;].apply(get_state)<br>list(df[&#39;states&#39;].head())</pre><p>Output:</p><pre>[&#39;Non_USA&#39;, &#39;PA&#39;, &#39;FL&#39;, &#39;NY&#39;, &#39;FL&#39;]</pre><p>Create some visualizations below.</p><pre><em># save updated dataframe</em><br>df.to_pickle(&#39;df_paulry_longs_lats.p&#39;)</pre><pre><em># retrive updated dataframe</em><br>df = pd.read_pickle(&#39;df_paulry_longs_lats.p&#39;)</pre><pre><em># plot tweets distribution by state</em></pre><pre>plt.style.use(&#39;seaborn-darkgrid&#39;)<br>df_states = df[df.country==&#39;United States&#39;]<br>df_states = df_states[df_states.states!=&#39;Non_USA&#39;]</pre><pre>x = df_states.states.value_counts()<br>x.plot(kind=&#39;bar&#39;,figsize=(14,6),fontsize=12,color=&#39;steelblue&#39;)<br>plt.ylabel(&#39;Origin of Tweets&#39;, fontsize=13)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/865/1*JANu0XHlOUlCEw7Vj8OkvA.png" /></figure><ul><li>From above we can see where majority of the tweets are coming from and it looks like it may be correlated to the size of the population and size of the state.</li><li>Next we extract the total sentiment from each state by adding positives to negatives and the final number is the indicator.</li><li>We will use this total sentiment from each state to create a heat map showing the most negative to the most positive states.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/326/1*NTBWP726DNgaIzVF5nyR3g.png" /></figure><pre><em># Create the sentiment by state Heat Map</em></pre><pre>colorscale=[<br>            [0, &#39;rgb(31,120,180)&#39;], <br>            [0.35, &#39;rgb(166, 206, 227)&#39;], <br>            [0.75, &#39;rgb(251,154,153)&#39;], <br>            [1, &#39;rgb(227,26,28)&#39;]<br>           ]</pre><pre>data = dict(type=&#39;choropleth&#39;,<br>            colorscale = colorscale,<br>            reversescale=<strong>True</strong>,<br>            locations = df_state_sentiment[&#39;states&#39;],<br>            z = df_state_sentiment[&#39;total_sentiment&#39;],<br>            locationmode = &#39;USA-states&#39;,<br>            text = df_state_sentiment[&#39;states&#39;],<br>            marker = dict(line = dict(color = &#39;rgb(255,255,255)&#39;,width = 2)),<br>            colorbar = {&#39;title&#39;:&quot;Twitter Sentiment&quot;}<br>            )</pre><pre>layout = dict(title = &#39;Twitter Sentiment: GOP House Speaker: Paul Ryan&#39;,<br>              geo = dict(scope=&#39;usa&#39;<br>                        )<br>             )</pre><pre>choromap_us = go.Figure(data = [data],layout = layout)</pre><pre><em># plotly.offline.plot(choromap_us, filename=&#39;img_map.html&#39;)  # save html map</em><br>IFrame(&#39;img_map.html&#39;, width=950, height=700)  <em># view saved map html file</em></pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EtQD9IFqQhBXvCI5w8naxQ.png" /></figure><ul><li>The total sentiment is obtained by summing up the positives and negatives for each state to come up with a final number.</li><li>From the heat map above we can make some interesting observations.</li><li>From the color scale to the right of the map, the blues transition to reds indicating positive to negative sentiment. The scale does actually start at 0 since no states muster a total sentiment in the positive side. So the more we go in the red, the more negative the sentiment gets.</li><li>New York and California have the most negative sentiment on this keyword search. Keeping in mind that NY and CA have the highest twitter using population which is very likely to skew the image above.</li><li>As such, I feel that this heatmap is not very reliable but it is a good starting point to exploring <strong><em>sentiment vs states</em></strong> visualizations.</li><li>Next we will see an alternative heat map where we will map every tweet in the dataset from the respective geo-location data.</li></ul><pre><em># use the folium library to create all tweet origins in the dataset on map of US</em></pre><pre>geoplots = []<br><strong>for</strong> index, row <strong>in</strong> df_states[[&#39;latitude&#39;,&#39;longitude&#39;,&#39;predictions&#39;]].iterrows():<br>    geoplots.append([row[&#39;latitude&#39;],row[&#39;longitude&#39;],row[&#39;predictions&#39;]])</pre><pre>mus = folium.Map(location=[39, -99], zoom_start=4)<br>plugins.Fullscreen(<br>    position=&#39;topright&#39;,<br>    title=&#39;Expand me&#39;,<br>    title_cancel=&#39;Exit me&#39;,<br>    force_separate_button=<strong>True</strong>).add_to(mus)</pre><pre>mus.choropleth(<br>    geo_data=&#39;us_states.geojson&#39;,<br>    fill_color=&#39;red&#39;, <br>    fill_opacity=0.1, <br>    line_opacity=0.2,<br>    name=&#39;US States&#39;)<br>    <br>mus.add_child(plugins.HeatMap(geoplots,<br>                            name=&#39;Twitter HeatMap&#39;,<br>                            radius=10,<br>                            max_zoom=1,<br>                            blur=10, <br>                            max_val=3.0))<br>folium.TileLayer(&#39;cartodbpositron&#39;).add_to(mus)<br>folium.TileLayer(&#39;cartodbdark_matter&#39;).add_to(mus)<br>folium.TileLayer(&#39;Mapbox Control Room&#39;).add_to(mus)<br>folium.LayerControl().add_to(mus)<br>mus.save(&quot;twitter_us_map.html&quot;) <br>IFrame(&#39;twitter_us_map.html&#39;, width=960, height=520)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/950/1*vggDbv58dN8yzL7iAwEL9g.png" /></figure><ul><li>The above map of the US shows a lot of tweets originate from these high population density areas.</li><li>At best, the map gives us a way to see where the tweets are coming from and the next step would be to map them according to their sentiment value for a better geo representation of sentiment.</li><li>We can also add a time dimension to the map to visualize tweets as they happen in a time series and in real time which would be a great tool for tracking a topic of interest for the parties involved. Such tools have been used before during presidential campaigns and during natural disasters.</li></ul><h3>Conclusions</h3><ul><li>So far our model has performed relatively well for a sentiment analysis model with an <em>accuracy of 76%</em> but a lot can be done to improve our confidence in this performance.</li><li>I think it is a good base model whose performance can be improved by experimenting with the suggestions below:</li><li>Additional data normalization can be performed on the training corpus. We removed stop words from our training data but we can also experiment with <a href="https://en.wikipedia.org/wiki/Stemming">stemming</a> and <a href="http://nbviewer.jupyter.org/github/RonKG/machine-learning-portfolio-projects/blob/master/3.%20NLP_twitter_sentiment_analysis/FINAL____twitter_sentiment_twitter.ipynb">lemmatization</a> which are both ways of analyzing words in a corpus based on their root or base word.</li><li>We could also try out different ways of tokenizing the words with special attention to capitalized words and special characters.</li><li>Emojis could also be used to extract more context to the text.</li><li>Experiment with multi-class labels to accommodate all human emotions that can be conveyed through text. <em>e.g.: sad, happy, excited, bored, etc..</em></li><li>Consider other ML algorithms like SVMs, Decision Trees, Neural Nets etc and even ensembles of these algorithms.</li><li>Finally I wonder if it would be beneficial to train a model on different topics and pick the most relevant one for different datasets. For example, a model for <em>Law</em> trained on an extensive library of law documents or a model for <em>Sports</em> trained on sports related text like tweets associated with sports and sports personalities. The ground is definitely rich for experimentation and subsequent evaluation of course.</li></ul><p>The code and more for this write up is available at my <a href="https://github.com/RonKG/machine-learning-projects-2/tree/master/3.%20NLP_twitter_sentiment_analysis">GitHub</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3ed2670f927d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/the-real-world-as-seen-on-twitter-sentiment-analysis-part-two-3ed2670f927d">This Is How Twitter Sees The World : Sentiment Analysis Part Two</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[This Is How Twitter Sees The World : Sentiment Analysis Part One]]></title>
            <link>https://medium.com/data-science/the-real-world-as-seen-on-twitter-sentiment-analysis-part-one-5ac2d06b63fb?source=rss-ec73e9699e73------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ac2d06b63fb</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[naive-bayes]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Ronald Wahome]]></dc:creator>
            <pubDate>Fri, 07 Sep 2018 22:50:15 GMT</pubDate>
            <atom:updated>2018-09-08T19:45:25.289Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Cby8nTrXuoNc4Vl9" /><figcaption>by<a href="https://unsplash.com/@hughhan?utm_source=medium&amp;utm_medium=referral"> Hugh Ha</a>n on<a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral"> Unsplas</a>h</figcaption></figure><p>Twitter is an online social network with over 330 million active monthly users as of February 2018. Users on twitter create short messages called tweets to be shared with other twitter users who interact by retweeting and responding. Twitter employs a message size restriction of 280 characters or less which forces the users to stay focused on the message they wish to disseminate. This very characteristic makes messages on twitter very good candidates for the Machine Learning (ML) task of sentiment analysis. Sentiment Analysis falls under <a href="https://en.wikipedia.org/wiki/Natural_language_processing">Natural Language Processing</a> (NLP) which is a branch of ML that deals with how computers process and analyze human language.</p><p>The training data was obtained from <a href="http://help.sentiment140.com/">Sentiment140</a> and is made up of about 1.6 million random tweets with corresponding binary labels. 0 for Negative sentiment and 1 for Positive sentiment. In this blog post, we’ll use a <a href="https://en.wikipedia.org/wiki/Naive_Bayes_classifier">Naive Bayes Classifier</a> to learn the correct labels from this training set and do a binary classification. In a nut shell, the Naive Bayes theorem calculates the probability of a certain event happening based on the joint probabilistic distributions of certain other events. I downloaded the test dataset using twitter’s API and will be use to test the model’s real world performance. Full documentation and terms of the API are available at <a href="https://developer.twitter.com/en/docs">developer.twitter.com/en/docs</a>.</p><p>In the first section of this blog post we’ll go through standard preprocessing steps performed on text data for a sentiment analysis ML task. The <a href="https://medium.com/p/3ed2670f927d">second section</a> is covered in a subsequent blog post where we combine the preprocessing into one step within a ML pipeline. This first post goes through the trouble of explaining what is happening under the hood when we use a pipeline to handle preprocessing.</p><h3>Section 1: Explore Labeled Training Dataset</h3><p>We’ll use Jupyter Notebook with Python for our workflow.</p><ul><li>Import the python libraries used in this project.</li><li>Import the data to a pandas dataframe and do some exploratory data analysis.</li></ul><pre><strong># you can see the full list of imports on GitHub!</strong></pre><pre><strong># Machine Learning imports<br>import</strong> <strong>nltk<br>from</strong> <strong>sklearn.pipeline</strong> <strong>import</strong> Pipeline<br><strong>from</strong> <strong>sklearn.model_selection</strong> <strong>import</strong> train_test_split<br><strong>from</strong> <strong>sklearn.metrics</strong> <strong>import</strong> classification_report, confusion_matrix, accuracy_score<br><strong>from</strong> <strong>sklearn.naive_bayes</strong> <strong>import</strong> MultinomialNB<br><strong>from</strong> <strong>sklearn.model_selection</strong> <strong>import</strong> KFold, cross_val_score<br><strong>from</strong> <strong>sklearn.ensemble</strong> <strong>import</strong> RandomForestClassifier<br><strong>from</strong> <strong>sklearn.feature_extraction.text</strong> <strong>import</strong> CountVectorizer<br><strong>from</strong> <strong>sklearn.feature_extraction.text</strong> <strong>import</strong> TfidfTransformer<br><strong>from</strong> <strong>sklearn.model_selection</strong> <strong>import</strong> GridSearchCV<br><strong>from</strong> <strong>sklearn.externals</strong> <strong>import</strong> joblib<br><strong>from</strong> <strong>nltk.corpus</strong> <strong>import</strong> stopwords<br><strong>from</strong> <strong>nltk.tokenize</strong> <strong>import</strong> TweetTokenizer<br><strong>from</strong> <strong>nltk.stem.wordnet</strong> <strong>import</strong> WordNetLemmatizerLoad training dataset to Pandas and preview the top rows.</pre><pre><em># load train data</em><br>data = pd.read_csv(&#39;Sentiment Analysis Dataset.csv&#39;,<br>                   error_bad_lines=<strong>False</strong>)<br>data.columns = [&#39;id&#39;,&#39;label&#39;,&#39;source&#39;,&#39;text&#39;]<br>data.head(2)</pre><pre><em># get text and matching label columns</em><br>data = data.drop([&#39;id&#39;,&#39;source&#39;],axis=1)<br>data.head(10)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/617/1*BFV4ghaFg6BZ9IQ4yh_7JQ.png" /></figure><ul><li>We can observe that the data is indeed from tweet messages posted on twitter.</li><li>The labels and the text do not seem to be in any listed order. This can be a problem if data is not randomly distributed as it can introduce biases to a learning model. In any case, we are going to use the <a href="http://scikit-learn.org/stable/documentation.html">Scikit Learn</a> library which has a function to split our training and testing data and shuffle the data at the same time.</li><li>Shuffling data reduces variance and makes sure our model can generalize better on the data and do less overfitting. We want to make sure the train and test dataset are representative of the overall distribution of the data.</li><li>To that note, we also want to check the label frequency distribution in the data which is done below.</li><li>Another observation we can make is that the text contains varying formats. Some words contain mixed case letters which need to be normalized to their base word. e.g.: ‘CRy’ changed to ‘cry’.</li><li>Leaving words with first letter capitalized can be experimented with as they may hold a different feature space like the name of a person or country etc.</li></ul><p>Explore distribution of label types.</p><pre><em># check the number of positive vs. negative tagged sentences</em><br>positives = data[&#39;label&#39;][data.label == 0]<br>negatives = data[&#39;label&#39;][data.label == 1]</pre><pre>print(&#39;number of positve tagged sentences is:  <strong>{}</strong>&#39;.format(len(positives)))<br>print(&#39;number of negative tagged sentences is: <strong>{}</strong>&#39;.format(len(negatives)))<br>print(&#39;total length of the data is:            <strong>{}</strong>&#39;.format(data.shape[0]))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/379/1*SeVM35bbuBIsjVwSKZzNKA.png" /><figcaption>positive VS negative</figcaption></figure><ul><li>Considering the size of the dataset, the labels seem to be ‘about’ evenly distributed at 788435 vs. 790177 for positive and negative respectively.</li><li>Next, we want to see the number of words contained in every sentence so I created a function to extract this information and appended it to a column next to the <em>text column</em>. Below is a sample output.</li></ul><pre><em># get a word count per sentence column</em><br><strong>def</strong> word_count(sentence):<br>    <strong>return</strong> len(sentence.split())<br>    <br>data[&#39;word count&#39;] = data[&#39;text&#39;].apply(word_count)<br>data.head(3)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/363/1*M92j9faYXiv7NFRfKOUejw.png" /><figcaption>word count per message</figcaption></figure><pre><em># plot word count distribution for both positive and negative sentiments</em><br>x = data[&#39;word count&#39;][data.label == 1]<br>y = data[&#39;word count&#39;][data.label == 0]<br>plt.figure(figsize=(12,6))<br>plt.xlim(0,45)<br>plt.xlabel(&#39;word count&#39;)<br>plt.ylabel(&#39;frequency&#39;)<br>g = plt.hist([x, y], color=[&#39;r&#39;,&#39;b&#39;], alpha=0.5, label=[&#39;positive&#39;,&#39;negative&#39;])<br>plt.legend(loc=&#39;upper right&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*rBPVELAq1GmfZG5W4Q5p9w.png" /><figcaption><em>word count distribution for both positive and negative</em></figcaption></figure><ul><li>From the graph above, most sentences fall between 5–10 words but it’s fair to say that majority of text on twitter falls between 1 and 25 words. This is no wonder considering that twitter has a limit of how many characters one can use in a message. 280 characters is the limit at the time of this writing.</li><li>In all, it looks like 1–20 words covers more than 80% of all sentences which makes this dataset set a good training candidate.</li><li>There are more positive sentences with 5 words or less than there are negative ones which does not seem like a big enough difference to cause any concern at the moment.</li></ul><pre><em># get most common words in training dataset</em><br>all_words = []<br><strong>for</strong> line <strong>in</strong> list(data[&#39;text&#39;]):<br>    words = line.split()<br>    <strong>for</strong> word <strong>in</strong> words:<br>        all_words.append(word.lower())<br>    <br>    <br>Counter(all_words).most_common(10)</pre><p>Output below:</p><pre>[(&#39;i&#39;, 741876),<br> (&#39;to&#39;, 556149),<br> (&#39;the&#39;, 516654),<br> (&#39;a&#39;, 374024),<br> (&#39;my&#39;, 309966),<br> (&#39;and&#39;, 294805),<br> (&#39;you&#39;, 236109),<br> (&#39;is&#39;, 229444),<br> (&#39;for&#39;, 212852),<br> (&#39;in&#39;, 209009)]</pre><ul><li>In the cell above we extracted the most common words in the dataset and listed the top ten.</li><li>Perhaps to no surprise we encounter words like <strong>i, and</strong> and <strong>is</strong> as they are very highly used in human expressions. These kind of words usually appear equally in both negative and positive oriented expressions and as such they bring very little information that can be incorporated in the model so we will have to get rid of them down the road.</li><li>In the text preprocessing steps later on we will learn how to deal with these common words that don’t add much to the feature space.</li><li>Below is a code to output a graph showing the frequency of the first 25 words.</li></ul><pre><em># plot word frequency distribution of first few words</em><br>plt.figure(figsize=(12,5))<br>plt.title(&#39;Top 25 most common words&#39;)<br>plt.xticks(fontsize=13, rotation=90)<br>fd = nltk.FreqDist(all_words)<br>fd.plot(25,cumulative=<strong>False</strong>)</pre><pre><em># log-log plot</em><br>word_counts = sorted(Counter(all_words).values(), reverse=<strong>True</strong>)<br>plt.figure(figsize=(12,5))<br>plt.loglog(word_counts, linestyle=&#39;-&#39;, linewidth=1.5)<br>plt.ylabel(&quot;Freq&quot;)<br>plt.xlabel(&quot;Word Rank&quot;)<br>plt.title(&#39;log-log plot of words frequency&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*XTWJB1M5VOb2fYCkEiQX0w.png" /></figure><ul><li>I also created a <a href="https://en.wikipedia.org/wiki/Log%E2%80%93log_plot"><strong>log-log</strong></a> plot for the words frequency which is similar to the previous frequency graph but includes all words and is plotted on a base 10 logarithmic scale which helps us visualize the rapidly diminishing frequency of words as their rank drops.</li><li>The word distribution present in this data dictionary is a very common phenomenon in large samples of words as shown by <a href="https://simple.wikipedia.org/wiki/Zipf%27s_law">Zipf’s law</a> where the most frequent word will occur about twice as often as the second most frequent word, three times as often as the third most frequent word, etc.</li><li>It would be interesting to see if this holds true after we remove words like <strong>i, and</strong> and <strong>is</strong> from the observation above.</li></ul><h3>Section 2: Explore Raw Twitter Data</h3><p>After training the model we make sentiment predictions on the raw twitter data but before we can do that, we need to download and do some basic data cleaning. You download tweets off the twitter RESTful API by <em>keyword</em> and get a stream of historical tweets back along with the metadata. I used <a href="https://tweepy.readthedocs.io/en/v3.6.0/">Tweepy</a> which is a python wrapper library for the twitter API that gives you more control on how you query the API. You can see the code on <a href="https://github.com/RonKG/machine-learning-portfolio-projects.......in-progress/tree/master/3.%20NLP_twitter_sentiment_analysis">GitHub</a>.</p><p>The keyword used to download historical tweets was “Paul Ryan” who is the speaker of the US House of Representative at the time of writing. My choice for this keyword is arbitrary but I feel is good enough to return some polarizing tweets. This is important because the tweets themselves can act as their own benchmark. One of the obstacles of NLP is evaluating a model’s performance considering that language is relative and always changing.</p><ul><li>Import and preview raw data.</li></ul><pre><em># create column names</em><br>col_names=[&#39;date&#39;,&#39;user_loc&#39;,&#39;followers&#39;,&#39;friends&#39;,&#39;message&#39;,&#39;bbox_coords&#39;,\<br>           &#39;full_name&#39;,&#39;country&#39;,&#39;country_code&#39;,&#39;place_type&#39;]</pre><pre><em># read csv</em><br>df_twtr = pd.read_csv(&#39;paul_ryan_twitter.csv&#39;, names=col_names)</pre><pre><em># check head</em><br>df_twtr.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/1*we0gOjLE1h5IQrahRis7yg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/301/1*nrOtNlxzHgxsXU_WbEDDVg.png" /></figure><p>We’ll keep the location data and use it later to map out our results on a geographic map.</p><h3>Section 3: Text Pre-processing</h3><ul><li>We want to remove any unnecessary qualities in the data which would make the trained model a poor generalizer.</li><li>Text preprocessing involves many things like removing emojis, properly formatting the text to remove extra spaces or any other information in the text that we don’t believe would add information to our model. We’ll see some examples below.</li><li>We also have to make sure that the information we pass the model is in a format that computers can understand. We’ll also go through some of these steps below.</li><li>After this pre-processing step, our data should be ready to use for a machine learning classification task.</li></ul><p>NOTE : Whatever text preprocessing we do on the train data must also be done on the test and our raw data.</p><h4>Pre-processing 1: Clean tweets text by removing links, special characters</h4><ul><li>This step is one that we could spend a lot of time on but the goal is always to find the best balance.</li><li>The bulk of the work in NLP is done on feature engineering. For now we will get rid of the links and emoji characters. It is important to mention that we can use them as features but for now we’re just building a basic model.</li></ul><p>I created a function that uses <em>regex</em> to do bulk formatting for every tweet in the dataset. Sample output is included under the code.</p><pre><em># helper function to clean tweets</em><br><strong>def</strong> processTweet(tweet):<br>    <em># Remove HTML special entities (e.g. &amp;amp;)</em><br>    tweet = re.sub(r&#39;\&amp;\w*;&#39;, &#39;&#39;, tweet)<br>    <em>#Convert @username to AT_USER</em><br>    tweet = re.sub(&#39;@[^\s]+&#39;,&#39;&#39;,tweet)<br>    <em># Remove tickers</em><br>    tweet = re.sub(r&#39;\$\w*&#39;, &#39;&#39;, tweet)<br>    <em># To lowercase</em><br>    tweet = tweet.lower()<br>    <em># Remove hyperlinks</em><br>    tweet = re.sub(r&#39;https?:\/\/.*\/\w*&#39;, &#39;&#39;, tweet)<br>    <em># Remove hashtags</em><br>    tweet = re.sub(r&#39;#\w*&#39;, &#39;&#39;, tweet)<br>    <em># Remove Punctuation and split &#39;s, &#39;t, &#39;ve with a space for filter</em><br>    tweet = re.sub(r&#39;[&#39; + punctuation.replace(&#39;@&#39;, &#39;&#39;) + &#39;]+&#39;, &#39; &#39;, tweet)<br>    <em># Remove words with 2 or fewer letters</em><br>    tweet = re.sub(r&#39;\b\w{1,2}\b&#39;, &#39;&#39;, tweet)<br>    <em># Remove whitespace (including new line characters)</em><br>    tweet = re.sub(r&#39;\s\s+&#39;, &#39; &#39;, tweet)<br>    <em># Remove single space remaining at the front of the tweet.</em><br>    tweet = tweet.lstrip(&#39; &#39;) <br>    <em># Remove characters beyond Basic Multilingual Plane (BMP) of Unicode:</em><br>    tweet = &#39;&#39;.join(c <strong>for</strong> c <strong>in</strong> tweet <strong>if</strong> c &lt;= &#39;<strong>\uFFFF</strong>&#39;) <br>    <strong>return</strong> tweet<br><em># ______________________________________________________________</em></pre><pre><em># clean dataframe&#39;s text column</em><br>df_paulry[&#39;message&#39;] = df_paulry[&#39;message&#39;].apply(processTweet)<br><em># preview some cleaned tweets</em><br>df_paulry[&#39;message&#39;].head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*AZD6musAuOVGAR-RJKy3wQ.png" /></figure><ul><li>Also note that you cannot have a perfect set of clean up steps as some cleanup steps will ultimately introduce some flaws in the data.</li><li>Below is a before and after example where <em>‘re’</em> has been removed from <em>re-election</em> which alters the meaning of the word and perhaps the user’s intended meaning. The word <em>won’t</em> also changes to <em>won</em> which obviously has a totally different meaning.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/639/1*3HGtyfEDoJq6oyW6JP-Xsw.png" /></figure><p>We drop duplicate tweets as they bring no new information to the dataset and are also computationally inefficient. We also visualize the most common words in the corpus.</p><pre><em># most common words in twitter dataset</em><br>all_words = []<br><strong>for</strong> line <strong>in</strong> list(df_paulry[&#39;message&#39;]):<br>    words = line.split()<br>    <strong>for</strong> word <strong>in</strong> words:<br>        all_words.append(word.lower())</pre><pre><em># plot word frequency distribution of first few words</em><br>plt.figure(figsize=(12,5))<br>plt.xticks(fontsize=13, rotation=90)<br>fd = nltk.FreqDist(all_words)<br>fd.plot(25,cumulative=<strong>False</strong>)</pre><pre><em># log-log of all words </em><br>word_counts = sorted(Counter(all_words).values(), reverse=<strong>True</strong>)plt.figure(figsize=(12,5))</pre><pre>plt.loglog(word_counts, linestyle=&#39;-&#39;, linewidth=1.5)<br>plt.ylabel(&quot;Freq&quot;)<br>plt.xlabel(&quot;Word Rank&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/613/1*JJ3Uhh2LM-9WTgITXevWsw.png" /></figure><h4>Pre-processing 2: Tokenize without the Stop-Words</h4><ul><li>Our training data has now been transformed into a much leaner body of text that is much cleaner for feature extraction. As we mentioned before, we do have some words in the dataset that are common in natural human language but used in most sentence compositions that we would be better left off since they bring no useful features to our model.</li><li>These words are commonly known as <a href="https://en.wikipedia.org/wiki/Stop_words">stop-words</a> in NLP and the <a href="http://text-processing.com/demo/sentiment/">NLTK library</a> comes with a function that can filter out these words from the dataset. Below is the actual words contained in the stop_words list. We can also make our own special list of stop words to fit any unique case. For example if you are doing sentiment analysis on law documents, you would probably need a special set considering the law jargon contained is unique in nature.</li></ul><pre><em># show stop words examples</em></pre><pre>(&quot;i , me , my , myself , we , our , ours , ourselves , you , you&#39;re , you&#39;ve , you&#39;ll , you&#39;d , your , yours , yourself , yourselves , he , him , his , himself , she , her , hers , herself , it &quot;)</pre><ul><li>After removing stop-words we split all the sentences in the dataset to get individual words <strong>(tokens)</strong> which is basically a list of words per sentence contained in the newly processed tweet. Now we can see that we have two new columns in the dataframe that contains these tokenized versions of a tweet.</li><li>A sample output is included below.</li></ul><pre><em># tokenize helper function</em><br><strong>def</strong> text_process(raw_text):<br>    <em>&quot;&quot;&quot;</em><br><em>    Takes in a string of text, then performs the following:</em><br><em>    1. Remove all punctuation</em><br><em>    2. Remove all stopwords</em><br><em>    3. Returns a list of the cleaned text</em><br><em>    &quot;&quot;&quot;</em><br>    <em># Check characters to see if they are in punctuation</em><br>    nopunc = [char <strong>for</strong> char <strong>in</strong> list(raw_text) <strong>if</strong> char <strong>not</strong> <strong>in</strong> string.punctuation]</pre><pre>    <em># Join the characters again to form the string.</em><br>    nopunc = &#39;&#39;.join(nopunc)<br>    <br>    <em># Now just remove any stopwords</em><br>    <strong>return</strong> [word <strong>for</strong> word <strong>in</strong> nopunc.lower().split() <strong>if</strong> word.lower() <strong>not</strong> <strong>in</strong> stopwords.words(&#39;english&#39;)]<br></pre><pre><strong>def</strong> remove_words(word_list):<br>    remove = [&#39;paul&#39;,&#39;ryan&#39;,&#39;...&#39;,&#39;“&#39;,&#39;”&#39;,&#39;’&#39;,&#39;…&#39;,&#39;ryan’&#39;]<br>    <strong>return</strong> [w <strong>for</strong> w <strong>in</strong> word_list <strong>if</strong> w <strong>not</strong> <strong>in</strong> remove]</pre><pre><em># -------------------------------------------</em></pre><pre><em># tokenize message column and create a column for tokens</em><br>df_paulry = df_paulry.copy()<br>df_paulry[&#39;tokens&#39;] = df_paulry[&#39;message&#39;].apply(text_process) <em># tokenize style 1</em><br>df_paulry[&#39;no_pauls&#39;] = df_paulry[&#39;tokens&#39;].apply(remove_words) <em>#tokenize style 2</em><br>df_paulry.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*c37PEaifaYzhKVot6Bu57A.png" /><figcaption>Compare the message column VS tokens column VS no_pauls.</figcaption></figure><ul><li>There are additional normalization techniques like <a href="http://www.nltk.org/howto/stem.html"><strong>stemming</strong></a> and <a href="http://www.nltk.org/api/nltk.stem.html?highlight=lemmatizer#nltk.stem.wordnet.WordNetLemmatizer"><strong>lemmatizing</strong></a> that we can try on our data but twitter messages are short by design and the above methods may not work so well because they essentially shorten words to their base words. e.g.: <em>running</em> to <em>run</em>.</li><li>For now we will stick with the current normalized state of our message data which we can now convert into a vector that can be fed into the appropriate ML algorithim.</li><li>I have also created a word cloud depicting the most common words in the entire twitter dataset after normalization.</li><li>We can see our keywords — paul &amp; ryan — are obviously very prominent but some domain knowledge would be helpful in making sense of why some of the other words are there. It is twitter after all so you can always expect a wide range of emotions. Some words in the dataset do use very strong language.The word cloud also excludes the words <em>paul</em> and <em>ryan</em> as they are disproportionately common in the dataset.</li></ul><pre><em># split sentences to get individual words</em><br>all_words = []<br><strong>for</strong> line <strong>in</strong> df_paulry[&#39;no_pauls&#39;]: <em># try &#39;tokens&#39;</em><br>    all_words.extend(line)<br>    <br><em># create a word frequency dictionary</em><br>wordfreq = Counter(all_words)</pre><pre><em># draw a Word Cloud with word frequencies</em><br>wordcloud = WordCloud(width=900,<br>                      height=500,<br>                      max_words=500,<br>                      max_font_size=100,<br>                      relative_scaling=0.5,<br>                      colormap=&#39;Blues&#39;,<br>                      normalize_plurals=<strong>True</strong>).generate_from_frequencies(wordfreq)</pre><pre>plt.figure(figsize=(17,14))<br>plt.imshow(wordcloud, interpolation=&#39;bilinear&#39;)<br>plt.axis(&quot;off&quot;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/1*05LInvAGSD9iB8mpudrXxQ.png" /><figcaption>Tokenized words word cloud</figcaption></figure><h4>Pre-processing 3: Feature Extraction</h4><h4>Vectorization — (Bag Of Words)</h4><p>We’ll convert each message which is represented by a list of tokens into a vector that a machine learning model can understand.</p><p>To do this we use the <a href="https://en.wikipedia.org/wiki/Bag-of-words_model">Bag Of Words</a> model which is a three step process.</p><ul><li>1.Count how many times does a word occur in each message (Known as term frequency)</li><li>2.Weigh the counts, so that frequent tokens get lower weight (inverse document frequency)</li><li>3.Normalize the vectors to unit length, to abstract from the original text length (L2 norm)</li></ul><p>Each vector will have as many dimensions as there are unique words in the tweeter corpus.</p><p>We will first use SciKit Learn’s CountVectorizer function which converts a collection of text documents to a matrix of token counts.</p><p>Imagine this as a 2-D matrix where 1-D is the entire vocabulary contained in the messages and the other dimension is one column per tweet.</p><p>Since there are so many messages, we can expect a lot of zero counts for the presence of every word in the data but SciKit Learn will output a <a href="https://en.wikipedia.org/wiki/Sparse_matrix">Sparse Matrix</a>. Below is an example — code and output — of a vectorized sentence.</p><pre><em># vectorize</em><br>bow_transformer = CountVectorizer(analyzer=text_process).fit(df_paulry[&#39;message&#39;])<br><em># print total number of vocab words</em><br>print(len(bow_transformer.vocabulary_))</pre><pre># output<br>6865</pre><pre><em># example of vectorized text</em><br>sample_tweet = df_paulry[&#39;message&#39;][111]<br>print(sample_tweet)<br>print(&#39;<strong>\n</strong>&#39;)</pre><pre><em># vector representation</em><br>bow_sample = bow_transformer.transform([sample_tweet])<br>print(bow_sample)<br>print(&#39;<strong>\n</strong>&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/261/1*XS8M302NwFVY9s4cOYJf6w.png" /><figcaption>Sample sentence in vector representation</figcaption></figure><pre><em># transform the entire DataFrame of messages</em><br>messages_bow = bow_transformer.transform(df_paulry[&#39;message&#39;])</pre><pre><em># check out the bag-of-words counts for the entire corpus as a large sparse matrix</em><br>print(&#39;Shape of Sparse Matrix: &#39;, messages_bow.shape)<br>print(&#39;Amount of Non-Zero occurences: &#39;, messages_bow.nnz)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/309/1*lDxkYWC5p7CMvazLVHmxnQ.png" /><figcaption>Whole corpus in vector representation</figcaption></figure><h4>Term Frequency, Inverse Document Frequency</h4><ul><li>TF-IDF stands for term frequency-inverse document frequency, and the tf-idf weight is a weight often used in information retrieval and text mining. This weight is a statistical measure used to evaluate how important a word is to a document in a collection or corpus.</li><li>The importance increases proportionally to the number of times a word appears in the document but is offset by the frequency of the word in the corpus. Variations of the tf-idf weighting scheme are often used by search engines as a central tool in scoring and ranking a document’s relevance given a user query.</li><li>One of the simplest ranking functions is computed by summing the tf-idf for each query term; many more sophisticated ranking functions are variants of this simple model.</li><li>Typically, the tf-idf weight is composed by two terms: the first computes the normalized Term Frequency (TF), aka. the number of times a word appears in a document, divided by the total number of words in that document; the second term is the Inverse Document Frequency (IDF), computed as the logarithm of the number of the documents in the corpus divided by the number of documents where the specific term appears.</li><li>TF: Term Frequency, which measures how frequently a term occurs in a document. Since every document is different in length, it is possible that a term would appear much more times in long documents than shorter ones. Thus, the term frequency is often divided by the document length (aka. the total number of terms in the document) as a way of normalization:</li></ul><p>TF(t) = (Number of times term t appears in a document) / (Total number of terms in the document).</p><ul><li>IDF: Inverse Document Frequency, which measures how important a term is. While computing TF, all terms are considered equally important. However it is known that certain terms, such as “is”, “of”, and “that”, may appear a lot of times but have little importance. Thus we need to weigh down the frequent terms while scale up the rare ones, by computing the following:</li></ul><p>IDF(t) = log_e(Total number of documents / Number of documents with term t in it).</p><ul><li>See below for a simple example.</li><li>Example: Consider a document containing 100 words wherein the word cat appears 3 times. The term frequency (i.e., tf) for cat is then (3 / 100) = 0.03. Now, assume we have 10 million documents and the word cat appears in one thousand of these. Then, the inverse document frequency (i.e., idf) is calculated as log(10,000,000 / 1,000) = 4. Thus, the Tf-idf weight is the product of these quantities: 0.03 * 4 = 0.12.</li></ul><pre><em># from sklearn.feature_extraction.text import TfidfTransformer</em><br>tfidf_transformer = TfidfTransformer().fit(messages_bow)<br>tfidf_sample = tfidf_transformer.transform(bow_sample)<br>print(tfidf_sample)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/232/1*Xi9PkqanqeKBBC48zEHTeQ.png" /></figure><ul><li>We’ll go ahead and check what is the IDF (<a href="https://en.wikipedia.org/wiki/Tf%E2%80%93idf">inverse document frequency</a>) of the word “trump” and of word “gop”?</li></ul><pre><em># some IDF (inverse document frequency) example</em><br>print(tfidf_transformer.idf_[bow_transformer.vocabulary_[&#39;trump&#39;]])<br>print(tfidf_transformer.idf_[bow_transformer.vocabulary_[&#39;gop&#39;]])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/126/1*6Sh3ebR0F80pdP-6fESe1w.png" /></figure><ul><li>To transform our entire twitter bag-of-words corpus into TF-IDF corpus at once we’ll use the code below:</li></ul><pre><em># to transform the entire bag-of-words corpus</em><br>messages_tfidf = tfidf_transformer.transform(messages_bow)<br>print(messages_tfidf.shape)</pre><pre># Output<br>(4164, 6865)</pre><ul><li>After preprocessing the data, we are now ready to pass it through a ML classification algorithim.</li><li>This ends the text preprocessing stages of our workflow but we will revisit these steps as <strong>one step</strong> in the pipeline in the <a href="https://medium.com/p/3ed2670f927d">next section</a> continued here.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ac2d06b63fb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/the-real-world-as-seen-on-twitter-sentiment-analysis-part-one-5ac2d06b63fb">This Is How Twitter Sees The World : Sentiment Analysis Part One</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Homicide in Chicago: Data Storytelling Part Two]]></title>
            <link>https://medium.com/data-science/homicide-in-chicago-data-stroytelling-part-two-e8748602daca?source=rss-ec73e9699e73------2</link>
            <guid isPermaLink="false">https://medium.com/p/e8748602daca</guid>
            <category><![CDATA[data-storytelling]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Ronald Wahome]]></dc:creator>
            <pubDate>Tue, 21 Aug 2018 14:15:22 GMT</pubDate>
            <atom:updated>2021-05-10T23:48:03.698Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bBCu34hfDd9ZxbQYaskAJw.jpeg" /><figcaption>Image via <a href="https://commons.wikimedia.org/wiki/File:Chicago_police_pan.jpg">Wikipedia</a></figcaption></figure><h4>Data Exploration and Visualization</h4><p>This is the second part of the Data Storytelling write up about homicides in Chicago which has been a hot topic lately including mentions by President Trump when he was campaigning for office in 2017. Being from Chicago, every time I meet people from out of state, they can’t help but talk to me about murder rates in the city but very few people bring this topic up from a point of data awareness.</p><p>In <a href="https://medium.com/@wahomeron/homicide-in-chicago-data-storytelling-part-one-e6fbd77afc07">part one</a> of this series, I cleaned the crime data but now I will attempt to extract some insights from the data on the ground. In this section I will use the newly created pandas dataframe from the previous section to paint a picture of homicides across the city of Chicago.</p><blockquote>the most dangerous place in Chicago is on the streets, in the summer time, on a weekend, between 11 pm and 2 am.</blockquote><h4>Crime distribution across the City</h4><p>The first thing we want to see is the homicide crime distribution for the whole city grouped by police districts.</p><p>We will use <a href="http://seaborn.pydata.org/">Seaborn</a> for this. We’ll draw inspiration from the earlier scatter plots in part one to visualize all homicides from 2001 to February 2018 by districts. Below is the code and the resulting plot.</p><pre><em># plot all homicides in dataset by location per District</em></pre><pre>df_homicide = df_homicide.loc[(df_homicide[&#39;X Coordinate&#39;]!=0)]</pre><pre>sns.lmplot(&#39;X Coordinate&#39;,<br>           &#39;Y Coordinate&#39;,<br>           data=df_homicide[:],<br>           fit_reg=<strong>False</strong>, <br>           hue=&quot;District&quot;, <br>           palette=&#39;Dark2&#39;,<br>           size=12,<br>           ci=2,<br>           scatter_kws={&quot;marker&quot;: &quot;D&quot;, <br>                        &quot;s&quot;: 10}) <br>ax = plt.gca()<br>ax.set_title(&quot;All Homicides (2001-2018) per District&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/913/1*j-6f6NbN8E1btC7WUr8Ngg.png" /></figure><p>From the visual above we can observe the following:</p><ul><li>high homicide rates are clustered on the top left and middle bottom of the scatter plot.</li><li>homicide rates in the top right and some areas in the middle are more freely scattered.</li></ul><p>Since each district is a different color, the big insight we can get from this chart is that the districts with a lot of points clustered together have more homicide rates than the ones that do not. So now, to some degree we have an idea of homicide from a spatial dimension but I would also wanted to understand these homicide rates from a time dimension.</p><p>Below I created some homicide vs time visualization and made notes on the observations thereafter.</p><h4>Yearly Homicide Rates :</h4><pre><em># plot bar chart of homicide rates for all years</em></pre><pre>plt.figure(figsize=(12,6))<br>sns.barplot(x=&#39;Year&#39;,<br>            y=&#39;HOMICIDE&#39;,<br>            data=df_homicide.groupby([&#39;Year&#39;])[&#39;Primary Type&#39;].value_counts().\<br>                 unstack().reset_index(),<br>            color=&#39;steelblue&#39;).\<br>            set_title(&quot;CHICAGO MURDER RATES: 2001 - 2018&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*L_3cKRjLaZ_u7g7SUjqRWg.png" /></figure><p>From the above barchart visualization we can draw the following conclusions:</p><ul><li>For the range of data available, homicides in Chicago had a short uptrend spike from 2001 to 2003 from 460 to 586.</li><li>This was followed by a a good drop into the 400 range which was maintained for four years until 2008.</li><li>The homicide rate then immediately dropped back into the 400’s for seven years until 2016.</li><li>However, the murder rate dramatically increased in 2016 to an all time high of 749, 282 from 2015’s rate of 467 .</li><li>2017 also saw a sustained high homicide rate albeit a drop of 90 to 659.</li><li>We can say that 2016 and 2017 had remarkable increase in homicide rates for Chicago.</li><li>I think we can begin to make sense of why so much has been in the news about murders in the Windy city.</li><li>In light of our observations about the last two years in our dataset, I wanted to keep this in mind and take a deeper look along the way as I explored the rest of the data.</li></ul><h4>Monthly Homicide Rates :</h4><pre><em># plot homicides sorted by month</em></pre><pre>fig, ax = plt.subplots(figsize=(14,6))<br>month_nms = [&#39;January&#39;,&#39;February&#39;,&#39;March&#39;,&#39;April&#39;,&#39;May&#39;,&#39;June&#39;,&#39;July&#39;,&#39;August&#39;\<br>             ,&#39;September&#39;,&#39;October&#39;,&#39;November&#39;,&#39;December&#39;]    <br>fig = sns.barplot(x=&#39;Month&#39;,<br>                  y=&#39;HOMICIDE&#39;,<br>                  data=df_homicide.groupby([&#39;Year&#39;,&#39;Month&#39;])[&#39;Primary Type&#39;].\<br>                  value_counts().unstack().reset_index(),<br>                  color=&#39;#808080&#39;)<br>ax.set_xticklabels(month_nms)<br>plt.title(&quot;CHICAGO MURDER RATES by MONTH -- All Years&quot;)</pre><pre><em># -------------------------------------------</em></pre><pre><em># plot average monthly temps in Chicago</em><br><em># source of data:  ncdc.noaa.gov</em></pre><pre>mntemp = [26.5,31,39.5,49.5,59.5,70,76,75.5,68,56,44,32]<br>df_temps = pd.DataFrame(list(zip(month_nms,mntemp)),<br>                       columns=[&#39;Month&#39;,&#39;AVERAGE TEMPERATURE&#39;])<br>fig, ax = plt.subplots(figsize=(14,6))<br>sns.barplot(x=&#39;Month&#39;, y=&#39;AVERAGE TEMPERATURE&#39;, data=df_temps,color=&#39;steelblue&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/1*FoC_moS3VwMmSIGOqjeZcw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/1*07oFqgutyhMXn7mTxeRfOA.png" /></figure><ul><li>From the month to month homicide rate, we see that on average homicides start steadily rising during the month of March and peaked during the month of July.</li><li>January was slightly higher than February but that seems like a continuation from the month of December.</li><li>A lot as been attributed to this cyclical trend but the best explanation seems to be tied to average monthly temperatures and weather in general.</li><li>Chicago winters tend to be very cold and that keeps most people from venturing outside unless they really have to.</li><li>Another possible reason for this cyclical trend is the fact that schools are out during the summer months and hence more young people with time on their hands to get into more trouble. Considering that the the most at risk age group is between ages 15 to 30. (Source: <a href="http://homicides.redeyechicago.com/ages/">Chicago Tribune</a>)</li></ul><h4>Day of the Week Homicide Rates :</h4><pre><em># plot homicide rates vs. day of the week</em></pre><pre>fig, ax = plt.subplots(figsize=(14,6))<br>week_days = [&#39;Sunday&#39;,&#39;Monday&#39;,&#39;Tuesday&#39;,&#39;Wednesday&#39;,&#39;Thursday&#39;,&#39;Friday&#39;,&#39;Saturday&#39;]    <br>fig = sns.barplot(x=&#39;Weekday&#39;,<br>                  y=&#39;HOMICIDE&#39;,<br>                  data=df_homicide.groupby([&#39;Year&#39;,&#39;Weekday&#39;])[&#39;Primary Type&#39;].\<br>                       value_counts().unstack().reset_index(),<br>                  color=&#39;steelblue&#39;)<br>ax.set_xticklabels(week_days)<br>plt.title(&#39;HOMICIDE BY DAY OF THE WEEK -- All Years&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/833/1*m9byYmi587MONXhXGouewQ.png" /></figure><p>Next we digger deeper for a more granular look from the monthly to the weekly time frame.</p><ul><li>At this level, Monday and Tuesday had the lowest homicide rates at around 60 murders a day on average.</li><li>Wednesday had a slight uptick followed by another slight uptick on Thursday. This small change is barely noticeable but considering this is a 18 year average, we can assume that some of this uptick is more pronounced during some weeks more than others.</li><li>This small up trend seen earlier in the week made a 25% jump from the 61 homicides a day average to 81 homicides a day. We have to remember though that these averages are not adjusted to the seasonal nature of these killing. So in reality the numbers are much higher in the warm months and lower in the colder months.</li><li>Saturday saw a sustained high rate but only a small increase from Friday.</li><li>Sunday did experience a drop in murder rates back into the 60’s from the 80’s which continued to drop into Monday and settle at the upper 50’s before weekly cycle repeated all over again.</li><li>We don’t know what is causing this cycle but we can hypothesize that they are tied to time away from school or work and more people tend to engage in drugs and gang activity and other crimes during this time.</li></ul><h4>Hour Of Day Homicide Rates :</h4><pre><em># use seaborn barplot to plot homicides vs. hour of the day </em></pre><pre>fig, ax = plt.subplots(figsize=(14,6))<br>fig = sns.barplot(x=&#39;HourOfDay&#39;,<br>                  y=&#39;HOMICIDE&#39;,<br>                  data=df_homicide.groupby([&#39;Year&#39;,&#39;HourOfDay&#39;])[&#39;Primary Type&#39;].\<br>                       value_counts().unstack().reset_index(),<br>                  color=&#39;steelblue&#39;,<br>                  alpha=.75)<br>plt.title(&#39;HOMICIDE BY HOUR OF THE DAY -- All Years&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/1*mpkwEigpKpsSyVnUyNjZew.png" /></figure><p>Continuing with the time frame analysis, it makes sense to look at homicide rates in Chicago from a time of the day perspective. I think it is easy to make assumptions that murders occur more at night than during the day but at best it’s just a guess so it was wise to look at the numbers no matter how obvious we think the conclusion is.</p><p>One way to look at the above plot is from a normal work day or school day perspective.</p><ul><li>7 am has the lowest numbers at about 9 homicides but as the morning progresses we see a steady rise in the number of homicides.</li><li>At 2 pm we cross the 15 mark which give way to another steady rise with a bit of volatility where 6 pm spikes up and 7 pm drops but not enough to erase previous rate gains.</li><li>9 pm is when things start getting alarming. Averages are can sometimes fail to give a clear picture but they do report on that one simple metric of the sum of all observations divided by number of observations. With that in mind, 25 murders an hour is quite high.</li><li>From 9 pm till a peak at 12 am hour we see a very sustained rise in these numbers. 12 am and 1 am rates stay above 30 murders an hour and only at 2 am do we see a drop below 30 followed a by a rapid drop until 7 am back to below 10.</li><li>A likely reason for this drop maybe because everyone tends to go to sleep the later into the night it gets. Even drug dealers and gang members have to fall asleep sometime.</li><li>A final observation from the above chart that I would like to make is that by the error bars for every hourly bar, there is more variance in the night hours than the day light hours indicating more active periods of homicides at night.</li></ul><h4>Yearly Homicide Rates by Domestic Involvement :</h4><pre><em># plot domestic variable vs. homicide variable</em></pre><pre>fig, ax = plt.subplots(figsize=(14,6))<br>df_arrest = df_homicide[[&#39;Year&#39;,&#39;Domestic&#39;]]<br>ax = sns.countplot(x=&quot;Year&quot;,<br>                   hue=&#39;Domestic&#39;,<br>                   data=df_arrest,<br>                   palette=&quot;Blues_d&quot;)<br>plt.title(&#39;HOMICIDE - DOMESTIC STATS BY YEAR&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/833/1*qywlKSTtKJONwKX5-FTuUQ.png" /></figure><ul><li>The level of domestic involvement for the entire range of the data does not seem to have any noticeable influence on the rate of homicides. The spike in murders for 2016 till 2017 does not have any noticeable correlation with domestic numbers.</li><li>Considering that the original dataset contained more than 40 types of crime from arson, sexual assault, domestic violence, stalking etc, then it makes sense that this variable was included in the dataset but it is not useful for our particular case involving homicides even though it must have a weight in domestic violence.</li></ul><h4>Homicide Rates by Crime Scene:</h4><pre><em># visualize the &quot;scene of the crime&quot; vs. number of occurences at such scene</em></pre><pre>crime_scene = df_homicide[&#39;Primary Type&#39;].\<br>            groupby(df_homicide[&#39;Location Description&#39;]).\<br>            value_counts().\<br>            unstack().\<br>            sort_values(&#39;HOMICIDE&#39;,ascending=<strong>False</strong>).\<br>            reset_index()<br>    <br><em># Top Homicide Crime Scene Locations</em><br>crime_scene.head(10)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/388/1*3FDyu8Vho2FAR5CpGyfxJw.png" /></figure><pre><em># create a count plot for all crime scene locations</em></pre><pre>g = sns.factorplot(x=&#39;Location Description&#39;,<br>                   y=&#39;HOMICIDE&#39;,<br>                   data=crime_scene,<br>                   kind=&#39;bar&#39;,<br>                   size=10,<br>                   color=&#39;steelblue&#39;, <br>                   saturation=10)</pre><pre>g.fig.set_size_inches(15,5)<br>g.set_xticklabels(rotation=90)<br>plt.title(&#39;CRIME SCENE BY LOCATION FREQUENCY&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/925/1*vr_Hm9zBxVrZXSnk8QbmvQ.png" /></figure><ul><li>From our Seaborn FactorPlot above, we can observe that majority of the murders (more than 60%) happen on the street.</li><li>The 2nd most common place is in cars followed by apartment, alley, house, porch , yard and parking lot in that order.</li><li>The rest of the locations are present in very low amounts to have a significant statistical impact. It is however worth looking more into the data collection methods as some of the other locations (as specialized as the labels may be) may be labeled after a larger generalized label like street.</li><li>So far from all the above observations we have made so far, we can begin to see a picture emerge which is; the most dangerous place in Chicago is on the streets, in the summer time, on a weekend, between 11 pm and 2 am.</li><li>However we want to know if this is true for all parts of Chicago. So often I get asked by out of town friends if I feel safe living in Chicago and I tell them that it is safe as long as you do not go to the wrong places. Next I show my exploration to find out if this is true.</li></ul><h4>Heat Map — Homicide Rates per Police District :</h4><pre><em># create a heatmap showing crime by district by year</em></pre><pre>corr = df_homicide.groupby([&#39;District&#39;,&#39;Year&#39;]).count().Date.unstack()<br>fig, ax = plt.subplots(figsize=(15,13))<br>sns.set(font_scale=1.0)<br>sns.heatmap(corr.dropna(axis=1),<br>            annot=<strong>True</strong>,<br>           linewidths=0.2,<br>           cmap=&#39;Blues&#39;,<br>            robust=<strong>True</strong>,<br>           cbar_kws={&#39;label&#39;: &#39;HOMICIDES&#39;})<br>plt.title(&#39;HOMICIDE vs DISTRICT vs YEAR&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/823/1*MdXM1hVA1PjjoNznvWwiBw.png" /></figure><p>Above I created a heatmap comparing all Chicago Police Districts vs. the number of Homicides per district from 2001 to 2017.</p><ul><li>As the heatmap shows, the murders are not evenly distributed across the districts. Darker area correspond to higher murder rates.</li><li>The darker the region on the heat map, the more murders in that district at the corresponding year.</li><li>Districts 1, 16 and 20 have low homicide rates at <strong>10 or less</strong> a year for all the years in the dataset.</li><li>Districts 17, 18, 19 and 24 all have murder rates at <strong>20 or less</strong> murders a year even though there are some years when they fall below 10 a year.</li><li>Districts 2, 12, 14 and 22 fall at <strong>30 or less</strong> murders a year.</li><li>The rest of the districts have <strong>more than 30 murders</strong> a year with some seeing murder rates well above 50 a year.</li><li>District 7 and 11 see consistently high numbers through the 17 years in the dataset. We could go as far as declare them the most deadly police districts for homicides in Chicago.</li><li>The <strong>highest murder rate</strong> for the entire dataset was in district 11 in 2016 with 92 homicides.</li><li>2016 was the <strong>worst year</strong> for homicides in the dataset with 6 districts seeing murders above 50.</li><li>The heat map brings this data to life and is such a great tool to have in our data exploration toolkit.</li></ul><h4>Most Dangerous &amp; Least Dangerous Police Districts</h4><pre><strong>with</strong> sns.plotting_context(&#39;notebook&#39;,font_scale=1.5):<br>    sorted_homicides = df_homicide[df_homicide[&#39;Year&#39;]&gt;=2016].groupby([&#39;District&#39;]).count()\<br>                    .Arrest.reset_index().sort_values(&#39;Arrest&#39;,ascending=<strong>False</strong>)<br>    fig, ax = plt.subplots(figsize=(14,6))<br>    sns.barplot(x=&#39;District&#39;,<br>                y=&#39;Arrest&#39;,<br>                data=sorted_homicides,<br>                color=&#39;steelblue&#39;,<br>                order = list(sorted_homicides[&#39;District&#39;]),<br>                label=&#39;big&#39;)<br>    plt.title(&#39;HOMICIDES PER DISTRICT (2016-2017) - Highest to Lowest&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*YTReKfrVNOPqt69U9YGq1w.png" /></figure><p>As we observed from the heat map earlier, some districts are more dangerous than others but now we can explore this observation more closely. From the various visualizations we have created, we can at least conclude that 2016 and 2017 have been the most active years for homicides in Chicago.</p><ul><li>In the barplot above I visualize the most and least murders per district in this two year period to see their relationship.</li><li>The top 10 districts all had more than 90 homicides in this two year period.</li><li>District 11 had 30 more murders than the next group in this list.</li><li>The first five out of 22 groups account for more than 50% of all homicides in the city.</li><li>The first 10 groups account for more than 80% of all homicides in Chicago.</li><li>8 out of 25 districts have very low to relatively low homicide rates.</li></ul><h4>Arrests vs Non-Arrests for Homicides:</h4><p>Next I looked at the relationship between arrest and homicides by plotting some relationships below.</p><blockquote>It looks like you could get away with murder in Chicago.</blockquote><pre><em># create seaborn countplots  for whole dataset</em></pre><pre>fig, ax = plt.subplots(figsize=(14,6))<br>df_arrest = df_homicide[[&#39;Year&#39;,&#39;Arrest&#39;]]<br>ax = sns.countplot(x=&quot;Year&quot;,<br>                   hue=&#39;Arrest&#39;,<br>                   data=df_arrest,<br>                   palette=&quot;PuBuGn_d&quot;)<br>plt.title(&#39;HOMICIDE - ARRESTS STATS BY YEAR&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/833/1*33tEnv5DNAoohXfU04PIVw.png" /></figure><p>The chart above compares arrest vs. non-arrests for all homicides in the dataset for all 17 years.</p><ul><li>In the early part of the 2000’s, police made more arrests than no arrests.</li><li>From 2001 till 2008, police made arrests on more than half of the homicides that happened.</li><li>From 2009 to date, police have made arrests less than half the time.</li><li>In fact this trend looks like a downward trend towards police making arrests may have started around 2004.</li><li>In 2016 and 2017, the arrests rate were less than 30%.</li></ul><pre><em># create seaborn countplots for 2016 and 2017 -- high crime rate spike years</em></pre><pre>fig, ax = plt.subplots(figsize=(14,6))<br>ax = sns.countplot(x=&quot;Month&quot;,<br>                   hue=&#39;Arrest&#39;,<br>                   data=df_homicide[df_homicide[&#39;Year&#39;]&gt;=2016][[&#39;Month&#39;,&#39;Arrest&#39;]],<br>                   palette=&quot;PuBuGn_d&quot;)<br>month_nms = [&#39;January&#39;,&#39;February&#39;,&#39;March&#39;,&#39;April&#39;,&#39;May&#39;,&#39;June&#39;,&#39;July&#39;,\<br>             &#39;August&#39;,&#39;September&#39;,&#39;October&#39;,&#39;November&#39;,&#39;December&#39;]    <br>ax.set_xticklabels(month_nms)<br>plt.title(&#39;HOMICIDE - ARRESTS STATS BY MONTH -- (2016-2018)&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/833/1*kZsxkT5qMdw7BJYJyC3law.png" /></figure><ul><li>What’s really surprising to me here is that even after crime spiked in 2016 followed by 2107, the police arrests have continued to fall for murder incidents. The figure above attempts to look at month characteristics but just confirms our previous observations.</li><li>I have not uncovered a good reason for this decline yet, but after a little bit of digging I found media reports on a policy change by CPD to concentrate less on arresting low level criminals and concentrating more on the bigger fish in terms of crime magnitude. Which begs the question, isn’t murder a high profile crime?</li><li>Another interesting thing that happened in 2016 was the public focus on shootings by police towards people of color and that leaves me wondering if it may have played a big role in police not being too aggressive in their arrests. Most of the high crime districts are in predominantly minority neighborhoods and in 2016, a judge ordered the release of a video showing Chicago police shooting dead a member of the black community.</li></ul><pre><em># create seaborn lmplot to compare arrest rates for different districts</em></pre><pre>dfx = df_homicide[df_homicide[&#39;District&#39;].\<br>                isin(list(sorted_homicides.head(10)[&#39;District&#39;]))].\<br>                groupby([&#39;District&#39;,&#39;Year&#39;,&#39;Month&#39;,&#39;Arrest&#39;])[&#39;Primary Type&#39;].\<br>                value_counts().unstack().reset_index()</pre><pre><strong>with</strong> sns.plotting_context(&#39;notebook&#39;,font_scale=1.25):<br>    sns.set_context(&quot;notebook&quot;, font_scale=1.15)</pre><pre>    g = sns.lmplot(&#39;Year&#39;,&#39;HOMICIDE&#39;,<br>                   col=&#39;District&#39;,<br>                   col_wrap=5,<br>                   size=5,<br>                   aspect=0.5,<br>                   sharex=<strong>False</strong>,<br>                   data=dfx[:],<br>                   fit_reg=<strong>True</strong>,<br>                   hue=&quot;Arrest&quot;, <br>                   palette=sns.color_palette(&quot;seismic_r&quot;, 2),<br>                   scatter_kws={&quot;marker&quot;: &quot;o&quot;,<br>                            &quot;s&quot;: 7},<br>                   line_kws={&quot;lw&quot;:0.7})</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/959/1*AAIVN4cBmVtdd3Dxfjn7MA.png" /></figure><p>A further break down of the arrest vs. non-arrest on a district level, of just the high homicide districts, added more weight to my previous observation of decreasing number of arrests over the years in the dataset. The plots above show regression lines of both arrests and non-arrests on the same axis for the top 10 most dangerous districts.</p><p>The trend of police making less arrests than the previous year had already began before 2006 and by 2010 the tables flipped in all districts to making arrests on less than half of all homicides.</p><p>It may well be that the odds of getting arrested for a homicide are less than half. The odds get much better if you are a criminal living in a high crime district.</p><p>Next I used <a href="https://github.com/python-visualization/folium">Folium </a>which is a python visualization album to create some maps illustrating some of my findings and make notes below. The maps on the <a href="http://nbviewer.jupyter.org/github/RonKG/machine-learning-portfolio-projects.......in-progress/blob/master/2.%20data_storytelling_project/homicide.data.cityofchicago.org.ipynb#">original code</a> are user responsive so you can zoom in and out and move them around.</p><h4>Chrolopleth Map of Homicide vs District</h4><h4>(2001–2017)</h4><pre><em># plot chloropleth maps 2001 - 2017<br></em><strong>def</strong> toString(x):<br>    <strong>return</strong> str(int(x))</pre><pre>df_homicide_allyears = df_homicide.groupby([&#39;District&#39;]).count().Arrest.reset_index()<br>df_homicide_allyears[&#39;District&#39;] = df_homicide_allyears[&#39;District&#39;].apply(toString)</pre><pre><em># ______________________________________________________#</em></pre><pre>chicago = location=[41.85, -87.68]<br>m = folium.Map(chicago,<br>               zoom_start=10)</pre><pre>plugins.Fullscreen(<br>    position=&#39;topright&#39;,<br>    title=&#39;Expand me&#39;,<br>    title_cancel=&#39;Exit me&#39;,<br>    force_separate_button=<strong>True</strong>).add_to(m)</pre><pre>m.choropleth(<br>    geo_data=&#39;chicago_police_districts.geojson&#39;,<br>    name=&#39;choropleth&#39;,<br>    data=df_homicide_allyears,<br>    columns=[&#39;District&#39;, &#39;Arrest&#39;],<br>    key_on=&#39;feature.properties.dist_num&#39;,<br>    fill_color=&#39;YlOrRd&#39;, <br>    fill_opacity=0.4, <br>    line_opacity=0.2,<br>    legend_name=&#39;Choropleth of Homicide per Police District : 2001-2017&#39;,<br>    highlight=<strong>True</strong><br>    )<br>folium.TileLayer(&#39;openstreetmap&#39;).add_to(m)<br>folium.TileLayer(&#39;cartodbpositron&#39;).add_to(m)<br>folium.LayerControl().add_to(m)<br>m.save(&quot;map1.html&quot;) <br>IFrame(&#39;map1.html&#39;, width=990, height=700)</pre><pre><em># plot 2016-2018 chloropleth map</em></pre><pre># code for <em>second map </em>is available on github</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KyTMpfAQzf6kTpm02jiAYA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1015/1*Obg8UDTiv3Bmcn8lRWJJeQ.png" /><figcaption>2001–2017 vs 2018</figcaption></figure><ul><li>I created 2 choropleth maps for two different time frames. One for the entire dataset and another for 2016–2017 period when we saw a spike in murders.</li><li>These maps are great way of showing how the homicide crimes vary across the different police districts.</li><li>From both maps, we can see that crime is concentrated around police districts on the west side of the city and south side of the city.</li><li>There are two districts that especially stand out from there darkened shades. These are districts 11 and 7.</li><li>Furthermore, districts with higher crime rates are geographically connected / contagious.</li></ul><h4>Heat Map of Homicide vs District — (2016–2017)</h4><pre><em># plot heatmap all districts -- (2016-2018)</em></pre><pre>after_2015_geo = []<br><strong>for</strong> index, row <strong>in</strong> df_homicide[df_homicide[&#39;Year&#39;]&gt;=2016][[&#39;Latitude&#39;,&#39;Longitude&#39;,&#39;District&#39;]].dropna().iterrows():<br>    after_2015_geo.append([row[&quot;Latitude&quot;], row[&quot;Longitude&quot;],row[&#39;District&#39;]])<br><em># ___________________________________________________________________</em><br>chicago = location=[41.85, -87.68]<br>m = folium.Map(chicago, zoom_start=9.5,control_scale = <strong>False</strong>)</pre><pre>plugins.Fullscreen(<br>    position=&#39;topright&#39;,<br>    title=&#39;Expand me&#39;,<br>    title_cancel=&#39;Exit me&#39;,<br>    force_separate_button=<strong>True</strong>).add_to(m)</pre><pre>m.choropleth(<br>    geo_data=&#39;chicago_police_districts.geojson&#39;,<br>    name=&#39;choropleth&#39;,<br>    data=df_homicide_after_2015,<br>    columns=[&#39;District&#39;, &#39;Arrest&#39;],<br>    key_on=&#39;feature.properties.dist_num&#39;,<br>    fill_color=&#39;YlOrRd&#39;, <br>    fill_opacity=0.4, <br>    line_opacity=0.2,<br>    legend_name=&#39;HeatMap Homicides : 2016-2017&#39;,<br>    highlight=<strong>True</strong><br>    )<br>m.add_child(plugins.HeatMap(after_2015_geo,<br>                            name=&#39;all_homicides_2016_to_2017&#39;,<br>                            radius=5,<br>                            max_zoom=1,<br>                            blur=10, <br>                            max_val=3.0))<br>folium.TileLayer(&#39;openstreetmap&#39;).add_to(m)<br>folium.TileLayer(&#39;cartodbpositron&#39;).add_to(m)<br>folium.LayerControl().add_to(m)<br>m.save(&quot;map3.html&quot;) <br>IFrame(&#39;map3.html&#39;, width=990, height=700)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/792/1*5bzVRKOHgQm7p6H2whH6ww.png" /><figcaption>Chicago Homicide Heat Map</figcaption></figure><ul><li>The heat mapping visualizations above were created with pythons <a href="https://folium.readthedocs.io/en/latest/">Folium</a> visualization library that gives great control on plotting dynamic maps on a <em>notebook</em>.</li><li>The heat map is for the 2016–2017 time frame. The individual crime scene locations from this time frame are mapped on the city’s canvas and represented as colors.</li><li>The heatmap can help us visualize general locations on a map where there is a high frequency or low frequency of crimes. The dark orange represent areas where crime is high and the disconnected light yellow / green shades show locations on a map where crime is low.</li><li>The immediate impression is that homicides are higher on the west side and south side of Chicago. We can zoom in closer for more details per area where with every zoom, more clusters are revealed within that sub-region.</li></ul><p><em>Note: I also created time lapse heat maps and cluster maps which are more interactive and can be </em><a href="http://nbviewer.jupyter.org/github/RonKG/machine-learning-portfolio-projects.......in-progress/blob/master/2.%20data_storytelling_project/homicide.data.cityofchicago.org.ipynb#"><em>viewed here</em></a><em> and on </em><a href="https://github.com/RonKG/machine-learning-portfolio-projects.......in-progress/tree/master/2.%20data_storytelling_project"><em>github</em></a><em>.</em></p><p>Finally, I created cluster maps that show clusters of murder locations. A live interactive notebook rendered on <a href="http://nbviewer.jupyter.org/github/RonKG/machine-learning-portfolio-projects.......in-progress/blob/master/2.%20data_storytelling_project/homicide.data.cityofchicago.org.ipynb#">nbviewer</a> is located here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/1*-VBsUdcp3KrbQxopuYeN5g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*69-PIEakkERx6p9-hVT1KQ.png" /><figcaption>(2001–2017) vs 2018</figcaption></figure><p>These map individual crime scenes which is more or less like the heat mapping without the color added to these geo locations. One of the maps is for the 2016–2017 period and the other is for the 2 month data available for 2018 (January and February).</p><p>The January and February map is particularly interesting because it shows where crime is already starting to appear in larger numbers at the beginning of 2018. Sadly, the homicides follow the same pattern as previous years with an overwhelming majority of the crimes happening in the same districts.</p><p>The map would be a great tool for anyone in the right field trying to identify which problem spots need more resources. The maps can also be great tool for visually inspecting individual crime locations for people in the community and combined with their knowledge of the area could bring new insights that can be explored and acted upon to make the community safer. An example would be community members identifying a problem spots and establishing a safe passage route for children to and from school. The map would be a great aid in helping them identify blocks that are less dangerous.</p><h3>Conclusion</h3><p>Throughout this notebook, I used data compiled by the Chicago police department to extract some insights on homicides in Chicago. While the data analysis performed here is devoid of a national perspective my key findings can be distilled to a few key points below:</p><ul><li>The number of homicide crimes vary greatly by police district. Districts with high numbers seem to always have high numbers year after year and vice versa.</li><li>Homicide numbers were largely unchanged from 2004 to 2015 with 400 and 500 homicides a year. These numbers suddenly saw a jump in 2016 to 700 homicides a year.</li><li>This new numbers were maintained into 2017 at over 650 homicides a year.</li><li>There is a correlation between weather temperatures and number of homicides per month. The warmer the month, the more homicide crimes are committed.</li><li>Day of the week also affects how many murders are committed with weekends seeing higher numbers.</li><li>Time of the day also affects the amount of homicides committed with 9pm to 2am being the most dangerous and 7am to 1pm being the safest hours.</li><li>District 11, 7 and 15 had the highest homicides in 2016–2017 period with over 120 homicides each over the same time period.</li><li>An overwhelming majority of homicides are committed on the street accounting for more than 60% of all homicide crime scenes.</li><li>District’s 20,16, 18, 1, 19, 17, 24 and 14 had less than 20 homicides each for the 2016–2017 period making them the <em>safest </em>police districts in Chicago.</li><li>Arrests for homicide crimes have been declining since 2003 and in 2008 the number of murders surpassed the number of arrests made for the murders. As of 2018, less than 3 arrests are made for every 10 homicides.</li></ul><p>While these observations are very illuminating, I should mention that they don’t paint the whole picture and comparisons should be made with national or other comparable data to give them more perspective. For example, while the number of homicides per year seem high, we don’t know if that is within the national average or too high. Such comparisons would help us build more compelling arguments in our data story telling by using them to support our local findings.</p><p>With that being said, our findings are still very relevant to local decision makers because they are very clear on the where and the when but not the why.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e8748602daca" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/homicide-in-chicago-data-stroytelling-part-two-e8748602daca">Homicide in Chicago: Data Storytelling Part Two</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Homicide in Chicago: Data Storytelling Part One]]></title>
            <link>https://medium.com/data-science/homicide-in-chicago-data-storytelling-part-one-e6fbd77afc07?source=rss-ec73e9699e73------2</link>
            <guid isPermaLink="false">https://medium.com/p/e6fbd77afc07</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[chicago]]></category>
            <category><![CDATA[storytelling]]></category>
            <category><![CDATA[data-cleaning]]></category>
            <dc:creator><![CDATA[Ronald Wahome]]></dc:creator>
            <pubDate>Tue, 21 Aug 2018 14:13:02 GMT</pubDate>
            <atom:updated>2018-08-25T17:26:32.800Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bBCu34hfDd9ZxbQYaskAJw.jpeg" /></figure><h4>Data Cleaning</h4><p>Chicago has been in the news a lot in the last few years especially due to the recent up surge of murders resulting from gun violence. As a resident of Chicago, I am aware that most of these shootings are confined to certain parts of the city but to truly understand the the true nature of this violence, I decided to dig deeper into the data available from the City of Chicago.</p><blockquote>It may well be that the odds of getting arrested for a homicide are less than half. The odds get much better if you are a criminal living in a high crime district.</blockquote><p>This blog post is divided into two parts. Part one covers data cleaning and part two which can be <a href="https://medium.com/@wahomeron/homicide-in-chicago-data-stroytelling-part-two-e8748602daca">found here</a> goes into data exploration and visualization.</p><p>I performed the data cleaning, exploration and visualization using Python in a Jupyter Notebook with the hope of uncovering some interesting insights along the way. I have also used the python libraries below. The full notebook, supporting code and files can be found in my <a href="https://github.com/RonKG/machine-learning-portfolio-projects.......in-progress/tree/master/2.%20data_storytelling_project">GitHub</a>.</p><pre><em># import modules</em><br><strong>import</strong> <strong>numpy</strong> <strong>as</strong> <strong>np</strong><br><strong>import</strong> <strong>pandas</strong> <strong>as</strong> <strong>pd</strong><br><strong>from</strong> <strong>pandas</strong> <strong>import</strong> *<br><strong>import</strong> <strong>os</strong><br><strong>import</strong> <strong>matplotlib.pyplot</strong> <strong>as</strong> <strong>plt</strong><br><strong>import</strong> <strong>seaborn</strong> <strong>as</strong> <strong>sns</strong><br>%<strong>matplotlib</strong> inline<br><strong>import</strong> <strong>datetime</strong><br><strong>from</strong> <strong>scipy</strong> <strong>import</strong> stats<br>sns.set_style(&quot;darkgrid&quot;)<br><strong>import</strong> <strong>matplotlib.image</strong> <strong>as</strong> <strong>mpimg</strong><br><strong>from</strong> <strong>IPython.display</strong> <strong>import</strong> IFrame<br><strong>import</strong> <strong>folium</strong><br><strong>from</strong> <strong>folium</strong> <strong>import</strong> plugins<br><strong>from</strong> <strong>folium.plugins</strong> <strong>import</strong> MarkerCluster, FastMarkerCluster, HeatMapWithTime</pre><p>I used publicly available data from the City of Chicago’s data portal linked below to explore crime in Chicago from January 2001 to February 2018.</p><p>Data Sources:</p><ul><li><a href="https://data.cityofchicago.org/Public-Safety/Crimes-2001-to-present/ijzp-q8t2">https://data.cityofchicago.org/Public-Safety/Crimes-2001-to-present/ijzp-q8t2</a></li><li><a href="https://data.cityofchicago.org/Public-Safety/Boundaries-Police-Beats-current-/aerh-rz74">https://data.cityofchicago.org/Public-Safety/Boundaries-Police-Beats-current-/aerh-rz74</a></li></ul><p>The notebook work flow is as follows:</p><ol><li>Loading dataset to pandas dataframe</li><li>Data exploration and cleaning</li><li>Insights extraction</li><li>Conclusion</li></ol><p>The original dataset from the city data portal was in .CSV format but was too large (at 1.5 GB) for my laptop’s resources. I could not fit the whole file in memory so I used pandas&#39; <em>TextFileReader </em>function which allowed me to load the large file in chunks of 100,000 rows and then concatenate the chunks back together into a new dataframe.</p><p>After loading the dataset, I used some of panda&#39;s built in functions to explore the characteristics of the data as shown below.</p><pre><em><br># use TextFileReader iterable with chunks of 100,000 rows.</em><br>tp = read_csv(&#39;Crimes_-_2001_to_present.csv&#39;, iterator=<strong>True</strong>, chunksize=100000)  <br>crime_data = concat(tp, ignore_index=<strong>True</strong>)  <br><br><em># print data&#39;s shape</em><br>crime_data.shape</pre><pre># Output:<br>(6546899, 22)</pre><p>Next we view the basic statistics on columns</p><pre>crime_data.info()</pre><pre>Output:</pre><pre>&lt;class &#39;pandas.core.frame.DataFrame&#39;&gt;<br>RangeIndex: 6546899 entries, 0 to 6546898<br>Data columns (total 22 columns):<br>ID                      int64<br>Case Number             object<br>Date                    object<br>Block                   object<br>IUCR                    object<br>Primary Type            object<br>Description             object<br>Location Description    object<br>Arrest                  bool<br>Domestic                bool<br>Beat                    int64<br>District                float64<br>Ward                    float64<br>Community Area          float64<br>FBI Code                object<br>X Coordinate            float64<br>Y Coordinate            float64<br>Year                    int64<br>Updated On              object<br>Latitude                float64<br>Longitude               float64<br>Location                object<br>dtypes: bool(2), float64(7), int64(3), object(10)<br>memory usage: 1011.5+ MB</pre><p>The next line of code previews the first 5 lines of the dataframe.</p><pre>crime_data.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UabEUt-wF3B1neNicb2dMg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/1*DDni5OtLV0fIWdnD6woILg.png" /></figure><h4>Observations about the data.</h4><ul><li>From the above cells we can observe that there are 22 columns and well over 6 million rows but just as important we can see the data type used in every column.</li><li>This step is important in helping us find out if we may need to change the data type to a more appropriate format for our specific needs. For example the <em>Date</em> column may need to be changed to python&#39;s <em>datatime.datetime</em> format to extract the month, time and day of the week information.</li><li>Also we can see from the <em>Primary Type</em> column that out research subject (homicide) is mixed in with other crimes so at some point we are going to have to put homicide in a separate table.</li><li>Another piece of information we can get from observing the outputs above is that some of the columns in the dataset overlap in importance so it would not make sense to keep all of them in our final dataframe configuration.</li><li>It would also make our dataframe less bulky from a memory standpoint. It is bad practice to feed this kind of bloated configuration into a machine learning (ML) task as it does not help the model generalize well on the data.</li></ul><pre><em># preview all crime variables in the &quot;Primary Type&quot; column</em><br><br>crimes = crime_data[&#39;Primary Type&#39;].sort_values().unique()<br>crimes, len(crimes)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/710/1*lehs2O_iNe0rzf3pGW_2XA.png" /></figure><h4>Map of Chicago Police Districts VS Scatter Plot of all crime geo-locations</h4><ul><li>I was interested in the <em>Homicide</em> numbers but I still wanted to know how all crime in general is distributed across the whole city. There are multiple ways to do this but I went ahead and created a scatter plot mapping all crime geo locations (X and Y coordinates) in the dataset and plotted it on the city&#39;s geographic map.</li><li>I also used the <em>Police Districts</em> as a hue to get a better understanding of the crime distributions across different communities in the city.</li><li>I could have used <em>Community Area</em> or <em>Ward</em> to sub-divide the city into different regions but these two variables were missing a lot of data points for the duration of the available time frame (2001-2018).</li><li>Using the alternate columns would mean that we would have to drop a lot of the data in other rows to the tune of 600,000 plus rows and that happened to be 10 % of the dataset. A bit too much data to be loosing and along with it possible important information.</li></ul><pre><em># Created a scatter plot of X and Y coordinates vs all crime data available in the dataset</em><br><br>crime_data = crime_data.loc[(crime_data[&#39;X Coordinate&#39;]!=0)]<br><br>sns.lmplot(&#39;X Coordinate&#39;, <br>           &#39;Y Coordinate&#39;,<br>           data=crime_data[:],<br>           fit_reg=<strong>False</strong>, <br>           hue=&quot;District&quot;,<br>           palette=&#39;Dark2&#39;,<br>           size=12,<br>           ci=2,<br>           scatter_kws={&quot;marker&quot;: &quot;D&quot;, <br>                        &quot;s&quot;: 10})<br>ax = plt.gca()<br>ax.set_title(&quot;All Crime Distribution per District&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/913/1*6FyN0QNZWVFQPU7te97CEQ.png" /></figure><ul><li>From the visualizations above, we can see an image from the web of the districts and a scatter plot of all crime data geo-locations in the dataset which produced a copy image of the Chicago map.</li><li>There were some gaps in the scatterplot but those are locations which have no continuous human occupation like rivers, waterports and airports.</li><li>The Chicago police have in the past attributed the vast majority of homicides in Chicago to gang violence so I grouped 4 crimes that are commonly associated with gang activity from the <em>Primary Type</em> column and created scatter plots to see if they have the have similar geographic distributions across the city. These four crimes are homicide, concealed weapon violations, narcotics and weapons violation and their plots are shown below.</li></ul><pre><em># create and preview dataframe containing crimes associated with gang violence</em><br><br>col2 = [&#39;Date&#39;,&#39;Primary Type&#39;,&#39;Arrest&#39;,&#39;Domestic&#39;,&#39;District&#39;,&#39;X Coordinate&#39;,&#39;Y Coordinate&#39;]<br>multiple_crimes = crime_data[col2]<br>multiple_crimes = multiple_crimes[multiple_crimes[&#39;Primary Type&#39;]\<br>                  .isin([&#39;HOMICIDE&#39;,&#39;CONCEALED CARRY LICENSE VIOLATION&#39;,&#39;NARCOTICS&#39;,&#39;WEAPONS VIOLATION&#39;])]<br><br><em># clean some rouge (0,0) coordinates</em><br>multiple_crimes = multiple_crimes[multiple_crimes[&#39;X Coordinate&#39;]!=0]<br><br><br>multiple_crimes.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/842/1*O4R9Ab_qzb2GK9Z1Jajuww.png" /></figure><p>Next I visualized each geographical distribution scatter plot for each of the crimes in the group to understand how the 4 crimes are distributed across the city.</p><pre># geographical distribution scatter plots by crime<br>g = sns.lmplot(x=&quot;X Coordinate&quot;,<br>               y=&quot;Y Coordinate&quot;,<br>               col=&quot;Primary Type&quot;,<br>               data=multiple_crimes.dropna(), <br>               col_wrap=2, size=6, fit_reg=<strong>False</strong>, <br>               sharey=<strong>False</strong>,<br>               scatter_kws={&quot;marker&quot;: &quot;D&quot;,<br>                            &quot;s&quot;: 10})</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/1*D9XeluzJANIU_t-3iJpMHw.png" /></figure><ul><li>It looks like <em>Narcotics</em> and <em>Weapons Violation</em> crimes are common all over the city but <em>Homicide</em> and <em>Concealed Carry Violation</em> crimes have a specific geographic pattern starting to emerge.</li><li>There were two general areas to the top left and bottom where clusters of crimes seemed to occur which was a great observation but too early to have drawn any confidence inspiring conclusions.</li><li>It could also be that narcotics crimes are more common outside of gang activity and don&#39;t really tell us anything as far as homicide is concerned. This would make a good topic for a machine learning prediction task.</li></ul><p>After a few basic observations, it was time to retrieve a dataframe with the homicide data and do some data cleaning before I could go on to explore it at depth.</p><pre><em># create a dataframe with Homicide as the only crime</em><br><br>df_homicideN = crime_data[crime_data[&#39;Primary Type&#39;]==&#39;HOMICIDE&#39;]<br>df_homicideN.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/938/1*G6HQ04qmWR9Ep8bBKoa6ug.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/787/1*bfN50d-Uk2mXsqXJhX2kWQ.png" /></figure><pre><em># print some attributes of our new homicide dataframe</em><br><br>df_homicideN.info()</pre><pre>&lt;class &#39;pandas.core.frame.DataFrame&#39;&gt;<br>Int64Index: 8975 entries, 641 to 6546894<br>Data columns (total 22 columns):<br>ID                      8975 non-null int64<br>Case Number             8975 non-null object<br>Date                    8975 non-null object<br>Block                   8975 non-null object<br>IUCR                    8975 non-null object<br>Primary Type            8975 non-null object<br>Description             8975 non-null object<br>Location Description    8975 non-null object<br>Arrest                  8975 non-null bool<br>Domestic                8975 non-null bool<br>Beat                    8975 non-null int64<br>District                8975 non-null float64<br>Ward                    8594 non-null float64<br>Community Area          8594 non-null float64<br>FBI Code                8975 non-null object<br>X Coordinate            8893 non-null float64<br>Y Coordinate            8893 non-null float64<br>Year                    8975 non-null int64<br>Updated On              8975 non-null object<br>Latitude                8893 non-null float64<br>Longitude               8893 non-null float64<br>Location                8893 non-null object<br>dtypes: bool(2), float64(7), int64(3), object(10)<br>memory usage: 1.5+ MB</pre><p>Find all the null values in the dataframe for further cleaning.</p><pre><em># find null values in our dataframe</em><br><br>df_homicideN.isnull().sum()</pre><pre># OUTPUT</pre><pre>ID                        0<br>Case Number               0<br>Date                      0<br>Block                     0<br>IUCR                      0<br>Primary Type              0<br>Description               0<br>Location Description      0<br>Arrest                    0<br>Domestic                  0<br>Beat                      0<br>District                  0<br>Ward                    381<br>Community Area          381<br>FBI Code                  0<br>X Coordinate             82<br>Y Coordinate             82<br>Year                      0<br>Updated On                0<br>Latitude                 82<br>Longitude                82<br>Location                 82<br>dtype: int64</pre><p>From the cells above we can observe:</p><ul><li>that we now have a total of 8981 rows of data and 7 columns contain null values.</li><li><em>Ward</em> and <em>Community Area</em> columns have 386 null values while <em>District</em> has zero null values. In terms of drawing geographic comparisons, I chose to go with the district column since it seemed to be more consistently available. However, with data cleaning one always has to weigh pros and cons of keeping or dropping data and the rules are not set in stone but rather we have to use our best judgement and test test test.</li><li>I have a total of 463 rows containing one or more null values.</li><li>Next I dropped all the rows containing null values from the dataframe to be able to perform different data transformations to the dataset.</li></ul><pre><em># drop null values and confirm</em><br><br>df_homicide = df_homicideN.dropna()<br>df_homicide.isnull().sum().sum()</pre><pre># OUTPUT<br>0</pre><p>Next I created a list of columns to keep and updated the dataframe with the new columns.</p><pre><em># create a list of columns to keep and update the dataframe with new columns</em><br><br>keep_cols = [&#39;Year&#39;,&#39;Date&#39;,&#39;Primary Type&#39;,&#39;Arrest&#39;,&#39;Domestic&#39;,&#39;District&#39;,&#39;Location Description&#39;,<br>             &#39;FBI Code&#39;,&#39;X Coordinate&#39;,&#39;Y Coordinate&#39;,&#39;Latitude&#39;,&#39;Longitude&#39;,&#39;Location&#39;]<br><br>df_homicide = df_homicide[keep_cols].reset_index()<br>df_homicide.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZeXBsY1brphbQ2Px8b_M8Q.png" /></figure><p>After dropping the NULL columns, I updated the dataframe with the columns that I thought to be most relevant in helping me understand the homicide picture in Chicago.</p><ul><li>I should mention that new insight may cause us to go back and re-examine columns that we dropped but for now we should be OK.</li><li>Below I used pandas <em>datetime </em>function to extract some time variables from the <em>Date</em> column. These new columns gave me extra dimensions with which to visualize the data and maybe new insights.</li></ul><pre><em># change string Date to datetime.datetime format</em><br><br>df_homicide[&#39;Date&#39;] = df_homicide[&#39;Date&#39;].apply(<strong>lambda</strong> x: datetime.datetime.strptime(x,&quot;%m/<strong>%d</strong>/%Y %I:%M:%S %p&quot;))<br>df_homicide.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1-JQmZe0UCahfJPW15E-4g.png" /></figure><pre><em># create new columns from date column -- Year, Month, Day, Hour, Minutes, DayOfWeek </em><br><br>df_homicide[&#39;Year&#39;] = df_homicide[&#39;Date&#39;].dt.year<br>df_homicide[&#39;Month&#39;] = df_homicide[&#39;Date&#39;].dt.month<br>df_homicide[&#39;Day&#39;] = df_homicide[&#39;Date&#39;].dt.day<br>df_homicide[&#39;Weekday&#39;] = df_homicide[&#39;Date&#39;].dt.dayofweek<br>df_homicide[&#39;HourOfDay&#39;] = df_homicide[&#39;Date&#39;].dt.hour<br><br>df_homicide = df_homicide.sort_values(&#39;Date&#39;)<br><em># print columns list and info</em><br><br>df_homicide.info()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/1*22sEKyP1o648wyweSVjbcA.png" /></figure><p>So far I had created a leaner pandas dataframe with just the relevant data from the original crime dataframe. In the next section we will get our hands dirty exploring and visualizing the new homicide dataframe.</p><h4>Pickle Data</h4><p>Now that we have performed some data cleaning, we can start to visualize it to get the bigger picture of crime in Chicago. To be able to pick up from where we left off, I stored the new data in pickle form which is a way in Python of saving an object to file for later retrieval. Pickling accomplishes this by writing the object as one long stream of bytes.</p><pre><em># save cleaned data to pickle file </em><br>df_homicide.to_pickle(&#39;df_homicide.pkl&#39;) <br>print(&#39;pickle size:&#39;, os.stat(&#39;df_homicide.pkl&#39;).st_size)</pre><pre># load pickled data <br>df_homicide = pd.read_pickle(&#39;df_homicide.pkl&#39;)</pre><p>The data exploration and visualization of this Chicago crime dataset is continued in the <a href="https://medium.com/@wahomeron/homicide-in-chicago-data-stroytelling-part-two-e8748602daca">second part</a> of this post.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e6fbd77afc07" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/homicide-in-chicago-data-storytelling-part-one-e6fbd77afc07">Homicide in Chicago: Data Storytelling Part One</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Cleaning Financial Time Series data with Python]]></title>
            <link>https://medium.com/data-science/cleaning-financial-time-series-data-with-python-f30a3ed580b7?source=rss-ec73e9699e73------2</link>
            <guid isPermaLink="false">https://medium.com/p/f30a3ed580b7</guid>
            <category><![CDATA[fintech]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-cleaning]]></category>
            <category><![CDATA[timeseries]]></category>
            <category><![CDATA[pandas]]></category>
            <dc:creator><![CDATA[Ronald Wahome]]></dc:creator>
            <pubDate>Sun, 19 Aug 2018 06:32:54 GMT</pubDate>
            <atom:updated>2018-08-20T13:49:27.440Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lpYAnDef4KkR4AHQ" /><figcaption>“Black and white photo of the street sign for Wall St in New York City” by <a href="https://unsplash.com/@ricktap?utm_source=medium&amp;utm_medium=referral">Rick Tap</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>A hypothetical company, <strong><em>ABC Financial Services Corp</em></strong> makes financial investments decisions on behalf of it’s clients based on the company’s economic research. A lot of these decisions involve speculating on future prices of financial instruments. ABC Corp utilizes several economic indicators but there is one in particular that is heavily weighted in their analysis and that is the <a href="https://en.wikipedia.org/wiki/University_of_Michigan_Consumer_Sentiment_Index">University of Michigan’s Consumer Sentiment Survey</a> (CSI).</p><p>The only problem is that they have to wait for the release of this data (released once a month) which erodes some of ABC’s edge in the market. To stay competitive, they would like a way to predict this number ahead of time. I propose using a form of Machine Learning (ML) to make time series predictions on the final Consumer Sentiment number that’s yet to be released.</p><p>To do this we are going to use other economic data (as features for the ML algorithm) which is released before the CSI. We’ll use this collection of data to construct a final dataset that is ready for a predictive algorithm.</p><h3>Multiple datasets</h3><p>The historical datasets that we’ll use are listed below and can be downloaded from the following links:</p><ul><li><a href="https://finance.yahoo.com/quote/%5EDJI/history/">The Dow Jones Index</a> : Source (Yahoo Finance)</li><li><a href="https://fred.stlouisfed.org/series/UNRATE">US Unemployment (Jobless Claims) data from the US Department of Labor</a> : Source (Federal Reserve)</li><li><a href="https://fred.stlouisfed.org/series/MCOILBRENTEU">Historical price of Crude Oil in the open market</a> : Source (Federal Reserve)</li><li><a href="https://fred.stlouisfed.org/series/HOUST#0">New Housing Starts from US Census Bureau</a> : Source (Federal Reserve)</li><li><a href="https://fred.stlouisfed.org/series/TOTALSA">Total Vehicles Sold</a> : Source (Federal Reserve)</li><li><a href="https://fred.stlouisfed.org/series/RSXFS">Retail Sales data from US Census Bureau</a> : Source (Federal Reserve)</li><li><a href="https://fred.stlouisfed.org/series/FEDFUNDS%5D">Federal Interest Rates</a> : Source (Federal Reserve)</li><li><a href="http://www.sca.isr.umich.edu/">The University of Michigan’s Consumer Sentiment Survey</a> — data to predict : Source (University of Michigan)</li></ul><h3>Tools</h3><p>We’ll use Python with the Pandas library to handle our data cleaning task. We are going to use can use Jupyter Notebook which is an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. It is a really great tool for data scientists. You can head over to <a href="http://anaconda.org/">Anaconda.org</a> to download the latest version which comes pre-loaded with most data science libraries.</p><p>We will combine the above datasets into one table using pandas and then do the necessary cleaning. Some of the above datasets have been seasonally adjusted to remove the influences of predictable seasonal patterns. In actual prediction learning/testing, we would experiment with both types of datasets.</p><p>Data cleaning is highly dependent on the type of data and the task you’re trying to achieve. In our case we combine data from different sources and clean up the resulting dataframe. In image classification data, we may have to reshape and resize the images and create labels while a sentiment analysis task may need to be checked for grammatical errors and keyword extraction.</p><h3>Visually inspect the dataframes</h3><p>To do this, we’ll need a few imports from the python library as shown below.</p><pre><em># Import necessary modules</em></pre><pre><strong>import</strong> <strong>numpy</strong> <strong>as</strong> <strong>np</strong><br><strong>import</strong> <strong>pandas</strong> <strong>as</strong> <strong>pd</strong><br><strong>import</strong> <strong>matplotlib.pyplot</strong> <strong>as</strong> <strong>plt</strong><br><strong>import</strong> <strong>seaborn</strong> <strong>as</strong> <strong>sns</strong><br>%matplotlib inline<br><strong>from</strong> <strong>scipy</strong> <strong>import</strong> stats<br><strong>from</strong> <strong>datetime</strong> <strong>import</strong> datetime<br><strong>from</strong> <strong>functools</strong> <strong>import</strong> reduce<br><strong>import</strong> <strong>datetime</strong></pre><p>Import the data tables into Pandas Dataframe.</p><pre><em># load all the datasets to pandas DataFrames</em></pre><pre>dow     = pd.read_csv(&#39;data/Dow Jones Industrial Average DJI.csv&#39;)<br>unemp   = pd.read_csv(&#39;data/Civilian Unemployment Rate UNRATE.csv&#39;)<br>oil     = pd.read_csv(&#39;data/Crude Oil Prices MCOILBRENTEU.csv&#39;)<br>hstarts = pd.read_csv(&#39;data/Housing Starts HOUST.csv&#39;)<br>cars    = pd.read_csv(&#39;data/Total Vehicle SalesTOTALSA .csv&#39;)<br>retail  = pd.read_csv(&#39;data/Advance Retail Sales_RSXFS.csv&#39;)<br>fedrate = pd.read_csv(&#39;data/Federal Interest Rates FEDFUNDS.csv&#39;)<br>umcsi   = pd.read_excel(&#39;data/consumer_sent_UMCH_tbmics.xls&#39;,header=3)</pre><p>After loading up the data, the first thing we do is visually inspect the data to understand how it is structured, it’s contents and to note anything out of the ordinary. Most of the data that you are going to come across is at least thousands of rows long so I like to inspect random chucks of rows at a time.</p><p>We use the <em>head()</em> and <em>tail()</em> functions to inspect the top and the bottom sections of the table respectively.</p><pre># view the top of the dow jones table</pre><pre>dow.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*16ZVajQnFAAs_wHM99jiaA.png" /></figure><pre># view the top of the umcsi table</pre><pre>umcsi.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/493/1*D8ZOoD-Jj0bIXdKzl1PrxQ.png" /></figure><pre># view the bottom of the data table</pre><pre>umcsi.tail()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/504/1*b7ZJzC-gvn0bydC87a4WuA.png" /></figure><p>We can also use a loop to iterate through all tables and get back the sizes. From this step we can start to anticipate the kinds of joins we need to do or decide whether we statistically have enough data to get started. Remember bad data is worse than no data.</p><pre><em># get the shape of the different datasets</em><br><br>dflist = [dow, unemp, oil, hstarts, cars, retail, fedrate, umcsi]<br><strong>for</strong> i, dfr <strong>in</strong> enumerate(dflist):<br> print(dflist[i].shape)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/242/1*xt6JODJei787VXkV8s5PaA.png" /></figure><p>Another useful pandas tool when we are inspecting data is the <em>describe()</em> function that gives a snapshot of the general statistics of all the numerical columns in the data.</p><pre><em># we look at the statistical charateristics of the datsets</em><br><br><strong>for</strong> i, dfr <strong>in</strong> enumerate(dflist):<br>    print(dflist[i].describe())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/656/1*8kDV4pf0rVTmkPRtAeZ0Vw.png" /><figcaption>The table above contains more columns as indicated by the back slash at the top right</figcaption></figure><p>We also want to know if we are dealing with data containing null values as this can result in bad data if ignored. One way to get the null values is to use the <em>isnull()</em> function to extract this information. We use a loop below to iterate over all the data tables.</p><pre><em># see which datasets have null values</em><br><br><strong>for</strong> i, dfr <strong>in</strong> enumerate(dflist):<br>    print(dflist[i].isnull().sum().sum())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/1*7NAPlqXI9fZ-J806MzbKVw.png" /></figure><h3>Making observations about the data</h3><ul><li>Some of the observations below are not apparent from the images above but are visible from the original notebook available <a href="https://github.com/RonKG/machine-learning-portfolio-projects.......in-progress/tree/master/1.%20data_cleaning_project">here</a> in <a href="https://github.com/RonKG/machine-learning-portfolio-projects.......in-progress/tree/master/1.%20data_cleaning_project">my GitHub</a> repository.</li><li>In this case, the data we acquired is not overly complicated with hundreds of columns but it is good to bear in mind that this is not always the case and we must be comfortable handling really large messy data.</li><li>From the inspections above, there are some things that need to be rectified before we can our achieve a final clean dataset.</li><li>The date formats need to be converted to a uniform format across all datasets.</li><li>The date ranges are also very inconsistent. The start dates range from 1947 to 1992.</li><li>The dates in umcsi are in two columns (String <strong>month</strong> &amp; float <strong>year</strong>) making it hard to merge with other datsets on that column.</li><li>umcsi also has 3 null values so we have to remove entire rows where this exists.</li><li>For our purposes, the dow dataframe also has extra columns so we will need to get rid of some of them.</li><li>These are just some of the modifications that I can observe but there is likely going to be other complications.</li><li>It is also necessary to comment your code so your colleagues can understand what you are doing.</li><li>At some point we also have to change the date format from String to a format that supports plotting.</li></ul><h3>Data Cleaning</h3><p>The Dow Jones data comes with a lot of extra columns that we don’t need in our final dataframe so we are going to use<em> pandas drop function</em> to loose the extra columns.</p><pre><em># drop the unnecessary columns</em><br><br><em> dow.drop([&#39;Open&#39;,&#39;High&#39;,&#39;Low&#39;,&#39;Adj Close&#39;,&#39;Volume&#39;],axis=1,inplace=True)</em></pre><pre># view the final table after dropping unnecessary columns</pre><pre>dow.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*3LJiRC0MXaD4syq_a6KMZQ.png" /></figure><pre><em># rename columns to upper case to match other dfs</em><br><br>dow.columns = [&#39;DATE&#39;, &#39;OPEN&#39;, &#39;HIGH&#39;, &#39;LOW&#39;, &#39;CLOSE&#39;, &#39;ADJ CLOSE&#39;, &#39;VOLUME&#39;]</pre><pre># view result after renaming columns<br>dow.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/639/1*bLwldYfMVbx3plm2nQ6D0A.png" /></figure><p>Most of these steps can be combined into fewer steps but I break them down so we can follow along and also we can confirm that we are achieving the intended results. Next we drop those columns that have null values from the data table. There are times when we may need to combine these null columns and then drop them later where we can fill in the values from other tables or gain additional columns (information).</p><p>The <em>inplace</em> flag below permanently removes the dropped rows.</p><pre><em># drop NaN Values</em><br><br>umcsi.dropna(inplace=<strong>True</strong>)</pre><p>The umcsi table contains year values as data type float which may be problematic when we start getting decimals numbers for years. I created a function that creates a new integer column from the float column. We can then drop the old float year column <em>inplace()</em>. Sometimes the date columns are in string format and we have to parse the date using pandas built in functions or we can create our own columns for those unique cases. And there will be many of those cases when you’re cleaning data.</p><pre><em># create &#39;Year&#39; column with int values instead of float</em></pre><pre># casting function<br><strong>def</strong> to_int(x):<br>    <strong>return</strong> int(x)</pre><pre># use function to convert floats to int<br><br>umcsi[&#39;Year&#39;] = umcsi[&#39;Unnamed: 1&#39;].apply(to_int)<br>umcsi.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/533/1*sjmItT4URSg0TOcdlHFG6A.png" /></figure><p>Observe that we have month and year as separate columns which need to be combined to match the date format from the rest of the data tables. For that we use pandas <em>datetime </em>functions which are very capable of handling most date and time manipulations. As it turns out, the other tables have dates in string data type so we will also have to change the umcsi date column to string. This will make sense because as a time series, any table joinings will be on the date column as the key.</p><pre><em># combine year columns to one column format</em><br><br>umcsi[&#39;DATE&#39;] = umcsi.apply(<strong>lambda</strong> x: datetime.datetime.strptime(&quot;<strong>{0}</strong> <strong>{1}</strong>&quot;.format(x[&#39;Year&#39;],x[&#39;DATE OF SURVEY&#39;]), &quot;%Y %B&quot;),axis=1)</pre><pre><em># turn date format to string to match other DATE&#39;s. We&#39;ll merge the data on this column so this is a vital step.</em><br><br><strong>def</strong> to_str(x):<br>    <strong>return</strong> str(x)[:10]<br><br>umcsi[&#39;DATE&#39;] = umcsi[&#39;DATE&#39;].apply(to_str)<br>umcsi.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*Or96eARB8NyS0rqj950nUw.png" /></figure><p>Our umcsi table is looking good with the exception of the old float date column and the month column so we have to get rid of that. We should also move the final date column to the front column for the sake of staying organized.</p><pre><em># drop unneeded columns column</em><br>umcsi.drop([&#39;Unnamed: 1&#39;,&#39;Year&#39;,&#39;DATE OF SURVEY&#39;], axis=1, inplace=<strong>True</strong>)<br><br><em># move &#39;DATE&#39; column to the front</em><br>cols = list(umcsi)<br>cols.insert(0, cols.pop(cols.index(&#39;DATE&#39;)))<br>umcsi = umcsi.reindex(columns = cols)<br>umcsi.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/351/1*NYdFB0DGhMdg650S01JpLw.png" /><figcaption>A more useful table than we started of with.</figcaption></figure><p>With all tables in a cohesive format, we can go ahead and join them and do some final cleanup steps. We shall concatenate the tables with date column as the key. We’ll use the all powerful lambda function for this one to get it done on the fly. Actually we will wrap a few more functions to demonstrate just how powerful pandas is for data manipulations.</p><pre><em># concatenate all dataframes into one final dataframe</em>  </pre><pre>dfs = [dow,unemp,oil,hstarts,cars,retail,fedrate,umcsi] </pre><pre># we perform the joins on DATE column as key and drop null values</pre><pre>df = reduce(<strong>lambda</strong> left,right: pd.merge(left,right,on=&#39;DATE&#39;, how=&#39;outer&#39;), dfs).dropna() df.head(5)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/1*U-Hjgd1_dYddh04257IXzw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/1*fI7DAfR_nITMqANqdz1FGw.png" /></figure><p>We now have a final pandas dataframe even though it still needs a bit more cleanup. Next we have to remove outliers from our final table since these outliers are likely to introduce a lot of noise to our machine learning task later on.</p><pre><em># remove all rows with outliers in at least one row</em><br><br>df = df[(np.abs(stats.zscore(df.drop([&#39;DATE&#39;], axis=1))) &lt; 3).all(axis=1)]</pre><pre># show final size after removing outliers<br>df.shape</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/151/1*rHLiaIya9QODcgE5_6bB2g.png" /><figcaption>Final dataframe shape</figcaption></figure><p>Python has a specialized format for dealing with time columns which is very efficient. We can extract the <em>datetime.datetime </em>format from current string format using the <em>strip() </em>function. Again we’ll use the lambda function to apply it to all rows on the fly.</p><pre><em># change the DATE column from String to python&#39;s <br># datetime.datetime format</em>  </pre><pre>df[&#39;DATE&#39;] = df[&#39;DATE&#39;].apply(<strong>lambda</strong> x: datetime.datetime.strptime(x,&quot;%Y-%m-<strong>%d</strong>&quot;))</pre><p>The final step is to rename the columns to more user friendly names for those that go on to consume this data.</p><pre><em># rename columns to more user friendly names</em><br><br>df.columns = [&#39;DATE&#39;, &#39;OPEN&#39;, &#39;HIGH&#39;, &#39;LOW&#39;, &#39;CLOSE&#39;, &#39;ADJ CLOSE&#39;, &#39;VOLUME&#39;, &#39;UNEMP %&#39;,&#39;OIL PRICE&#39;,&#39;NEW HOMES&#39;,&#39;NEW CARS SOLD&#39;,<br>                    &#39;RETAIL SALES&#39;,&#39;FED INTRST %&#39;,&#39;CSI&#39; ]</pre><pre># preview final table<br>df.head(20)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/1*DJI5FQixagz1ufNFFE0joA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/219/1*GebIBZ576bv1wpcNEetI7w.png" /><figcaption>FINAL pandas dataframe</figcaption></figure><h3>Data cleaning conclusions</h3><ul><li>Data cleaning comes in all shapes and sizes and there is no one template to handle all situations.</li><li>While we do not know how the data will perform for the task of predicting the CSI, we do know that the data supplied has been processed to facilitate fast adoption and testing in a ML enviroment.</li><li>Sure we could have engineered more features and processed the current ones some more but that would be too presumptuous on our part as to how the ML team would proceed. For example we could have normalized the features to a uniform scale but did not.</li><li>In conclusion, like most tasks in the data science world, the best we can do is keep asking questions in the right team and experiment deeper based on those questions.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f30a3ed580b7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/cleaning-financial-time-series-data-with-python-f30a3ed580b7">Cleaning Financial Time Series data with Python</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>