<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Brian Chan on Medium]]></title>
        <description><![CDATA[Stories by Brian Chan on Medium]]></description>
        <link>https://medium.com/@brian_chan?source=rss-58fefbfa684a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*v9paInoA6-zWfzFwQr_D0w.jpeg</url>
            <title>Stories by Brian Chan on Medium</title>
            <link>https://medium.com/@brian_chan?source=rss-58fefbfa684a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 16:19:49 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@brian_chan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Collect and Visualize COVID-19 Data]]></title>
            <link>https://medium.com/data-science/collect-and-visualize-covid-19-data-336e4dc9ead?source=rss-58fefbfa684a------2</link>
            <guid isPermaLink="false">https://medium.com/p/336e4dc9ead</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[beautifulsoup]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[folium]]></category>
            <dc:creator><![CDATA[Brian Chan]]></dc:creator>
            <pubDate>Sat, 11 Apr 2020 18:18:21 GMT</pubDate>
            <atom:updated>2020-04-21T21:49:02.071Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Beginner’s guide to Webscraping and Exploratory Data Analysis</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HCq-jpiNcFV-2KkEoAw1mA.png" /></figure><p>Needless to say, everyone is well aware of the devastating effect COVID-19 has on the entire world. There are countless datasets made available to capture trends regarding the spread of this virus worldwide.</p><p>Personally, I am more interested in analyzing the COVID-19 cases in my home state of Pennsylvania (USA). I stumbled upon the <a href="https://www.health.pa.gov/topics/disease/coronavirus/Pages/Cases.aspx">Pennsylvania Department of Health Website</a>, which publishes daily COVID-19 cases in Pennsylvania broken down by counties. Unfortunately, each daily update is presented in a separate section as HTML tables. As we scroll through this webpage, it becomes difficult to form a mental image of the daily trends. As a result, I decided to take on the challenge to solve this problem. I embarked on a weekend mini-project to implement solutions to the following problems:</p><ul><li>Extract Data from the Webpage</li><li>Perform Data Analysis</li><li>Create Data Visualizations</li></ul><p>The result is presented in a <a href="https://github.com/bchan01/covid-19-data-analysis/blob/master/data-analysis.ipynb">Jupyter Notebook</a> in my <a href="https://github.com/bchan01/covid-19-data-analysis">GitHub Repo</a>. Unfortunately, GitHub is unable to render the interactive data map, please visit <a href="https://nbviewer.jupyter.org/github/bchan01/covid-19-data-analysis/blob/master/data-analysis.ipynb">Jupyter NBViewer</a> if you wish to see it.</p><blockquote>Without further ado, let’s get started ! I will walk you through my learning journey in details.</blockquote><p>We will start with how to turn the html tables into datasets, followed by data manipulation, plots and charts. We will conclude with creating an interactive map to show the confirmed cases by county.</p><h3><strong>Extracting Data through Webscraping</strong></h3><p>I would like to briefly touch on how I extract the datasets from the <a href="https://www.health.pa.gov/topics/disease/coronavirus/Pages/Archives.aspx">PA Department of Health</a> web page, which is updated daily. As you scroll through the page, you will notice each daily section starts with a date header (example: April 8, 2020). I am interested in aggregating data from the below two tables, namely <em>Total Counts Table</em> and <em>Total Counts by County Table</em>.</p><p><strong>Total Counts Table: </strong>shows total negative, positive cases and deaths to date.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jat-KO94c8WOuytw_UFthQ.png" /></figure><p><strong>Total Counts by County Table: </strong>shows total positive cases and deaths to date for each Pennsylvania county.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/278/1*TjQfrxf9qq1vU55mw59Xug.png" /></figure><p>Below is an outline of the webscraping strategy, the <a href="https://github.com/bchan01/covid-19-data-analysis/blob/master/data_extractor_pa.py">complete source code </a>is available in my GitHub Repository.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OitjgeRH7TU27xV--V0aCw.png" /></figure><p>You need to install these dependencies:</p><pre>pip install beautifulsoup4<br>pip install lxml</pre><p><strong>Step 1.</strong> Extract the dates between the &lt;h4&gt; headers and store a list.</p><p><strong>Step 2.</strong> Extract data in each <em>Total Counts Table</em> and match it to the dates in step 1. A sample of the output file is shown below.</p><pre><strong>pa_summary.csv:<br></strong>Date,Negative,Positive,Deaths<br>2020–04–08,82299,16239,309<br>2020–04–07,76719,14559,240</pre><p><strong>Step 3.</strong><em> </em>Extract data in each Total Counts by County Table and match it to the dates in step 1. A sample of the output file is shown below.</p><pre><strong>pa_county.csv:<br></strong>Date,County,Cases,Deaths<br>2020–04–08,Adams,33,0<br>2020–04–08,Allegheny,720,1</pre><p><strong>Step 4. </strong>Invoke <a href="https://developer.mapquest.com/documentation/geocoding-api/">MapQuest Geocoding API</a> to obtain latitude and longitude for the counties. You can setup a free account with 15K transactions per month. Obtain an API Key from MapQuest and update API_KEY value in <a href="https://github.com/bchan01/covid-19-data-analysis/blob/master/data_extractor_pa.py">data_extractor_pa.py</a>. A sample of the output file is shown below:</p><pre><strong>pa_county_geolocation.csv</strong><br>County,Latitude,Longitude<br>Bradford,41.774669,-76.526461<br>Butler,40.861021,-79.895225</pre><p><strong>Step 5 (Optional).</strong> If you wish to get the latest data, Run extractor to re-generate the CSV files.</p><pre>python data_extractor_pa.py</pre><h3><strong>Performing Exploratory Data Analysis</strong></h3><p>With these three datasets in hand, let’s fire up <a href="https://github.com/bchan01/covid-19-data-analysis/blob/master/data-analysis.ipynb">Jupyter Notebook</a>.</p><p>First order of business, import these dependencies:</p><pre>import datetime as dt<br>import matplotlib.pyplot as plt<br>from matplotlib import style<br>import pandas as pd</pre><p>Load Summary Dataset into a Panda DataFrame:</p><pre>df = pd.read_csv(&quot;pa_summary.csv&quot;)<br>df.head(5)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CL5eIerlpWLuvFmu2Mqc1g.png" /></figure><p>Compute total tested and percentages of negatives/positives/deaths:</p><pre>df[&#39;Tested&#39;] = df[&#39;Positive&#39;] + df[&#39;Negative&#39;]<br># Percentages<br>df[&#39;Positive_Percent&#39;] = round(100*df[&#39;Positive&#39;]/df[&#39;Tested&#39;], 1)<br>df[&#39;Negative_Percent&#39;] = round(100*df[&#39;Negative&#39;]/ df[&#39;Tested&#39;], 1)<br>df[&#39;Death_Percent&#39;] = round(100*df[&#39;Deaths&#39;]/ df[&#39;Tested&#39;], 3)<br># sort by date<br>df = df.sort_values(&#39;Date&#39;, ascending = True)<br>df.head(5)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_ZnHC48LNC1BQhaQnOo0gQ.png" /></figure><p>Plot the total tested, positive, negative and death counts to date:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/35ff360c75d819c34a6a8cd8a60a049e/href">https://medium.com/media/35ff360c75d819c34a6a8cd8a60a049e/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lepiyfl1cRYL8CepLJ917Q.png" /></figure><blockquote>What are the percentages of positives vs negatives out of the tested population?</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1bb304dd7a5fd7d1d877ee26cef8490e/href">https://medium.com/media/1bb304dd7a5fd7d1d877ee26cef8490e/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NFWI96ZjA5CwHkgWYal1rA.png" /></figure><blockquote>Perhaps it’s more meaningful to look at the daily changes.</blockquote><p>The following code snippets show how to calculate daily changes in cases utilizing the <em>diff function</em> available in panda. Note that we need to fill in <em>0</em> for the first row to avoid getting <em>NaN.</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3ff1370bcd06aaaa20eb31379651c9f4/href">https://medium.com/media/3ff1370bcd06aaaa20eb31379651c9f4/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/848/1*oOItbQ0pheP9n2bJCGMuBw.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dc53f9984461a2817cc8e05679ee40b9/href">https://medium.com/media/dc53f9984461a2817cc8e05679ee40b9/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c3iy0VPjg25HgiSd-mhewg.png" /></figure><h3><strong>Looking at County Breakdown</strong></h3><pre>df_counties = pd.read_csv(&quot;pa_county.csv&quot;)<br>df_counties.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/806/1*4A_XiX70rbIGt7gQcJd0Qw.png" /></figure><blockquote>How are the confirmed cases trending in the city of Philadelphia?</blockquote><pre># Filter Data for a County<br>county_name = &#39;Philadelphia&#39;<br>df_county = df_counties[df_counties[&#39;County&#39;]== county_name]<br>df_county = df_county.sort_values(&#39;Date&#39;, ascending = True)</pre><pre># Line Plot<br>style.use(&#39;ggplot&#39;)<br>ax = plt.gca()<br>df_county.plot(kind=&#39;line&#39;,x=&#39;Date&#39;,y=&#39;Cases&#39;, color=&#39;green&#39;, ax=ax, figsize=(20,10))<br>plt.title(&#39;Number of Cases In &#39; + county_name +&#39; County&#39;)<br>plt.xlabel(&#39;Date&#39;)<br>plt.ylabel(&#39;Cases&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o8NBGshXSDlbQtDnvaY30g.png" /></figure><h3>Creating an Interactive Data Map</h3><p>Let’s start to prepare a DataFrame for visualizing confirmed cases by county in an interactive map.</p><blockquote>How do we pinpoint each county in a map?</blockquote><p>First, load the county geolocation dataset containing latitude/longitude pair for each county</p><pre>df_geo = pd.read_csv(&quot;pa_county_geolocation.csv&quot;)<br>df_geo.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/918/1*w0QAg0loEtRYFyuUhBJdUQ.png" /></figure><p>Perform left outer-join to add lat/long columns to the county DataFrame.</p><pre>df_county_with_geo = df_counties.merge(df_geo, how=&#39;left&#39;, on=[&#39;County&#39;])<br>df_county_with_geo.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zyJ4TNoLGaAhwmvpppwTrg.png" /></figure><p>Finally, we only want to keep the records with the latest date.</p><pre>df_latest_county = df_county_with_geo[df_county_with_geo[&#39;Date&#39;] == df_county_with_geo[&#39;Date&#39;].max()]</pre><pre># drop unnecessary columns<br>del df_latest_county[&#39;Date&#39;]<br>del df_latest_county[&#39;Deaths&#39;]</pre><pre>df_latest_county.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*2PNjUgWYOrojg1eK5z44XQ.png" /></figure><blockquote>We are ready to create an interactive map</blockquote><p>I chose Folium, which has rich features for rendering interactive maps. This is not intended to be a tutorial on Folium. Please visit <a href="https://python-visualization.github.io/folium/index.html">here</a> to learn more about it.</p><p>You need to install Folium before proceeding.</p><pre>pip install folium</pre><blockquote>With a few lines of codes, we can build an awesome interactive data map to display COVID-19 confirmed cases across Pennsylvania counties.</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c513dfab3ffb4a85cc95524840ebebf3/href">https://medium.com/media/c513dfab3ffb4a85cc95524840ebebf3/href</a></iframe><p>Let’s walk through what we are doing here</p><ul><li>Center the map on a location so that most data points are visible within the viewing area.</li><li>The function get_icon_color() determines the color of the icon to display on the map based on the value of the positive cases.</li><li>Create a map with center point and proper zoom level.</li><li>Decorate each county data point as a color-coded icon and popup text to show the count and county name.</li><li>Create a legend section for showing what the color means. Kudos to this great <a href="https://medium.com/@bobhaffner/creating-a-legend-for-a-folium-map-c1e0ffc34373">Medium blog post</a> on how to add legend to Folium Map.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*88jTcOHnfdGHOConsXQZPA.png" /></figure><blockquote><strong>That’s all folks!!</strong></blockquote><p>The code from this post is available on <a href="https://github.com/bchan01/covid-19-data-analysis">GitHub</a>. I hope this would inspire you to get out of your comfort zone to learn something new. Until next time, keep learning and getting better each and every day.</p><h3>Resources:</h3><ul><li><a href="https://pandas.pydata.org/docs/reference/frame.html">Pandas DataFrame API Reference</a></li><li><a href="https://matplotlib.org/3.2.1/contents.html">Matplotlib Documentation</a></li><li><a href="https://www.crummy.com/software/BeautifulSoup/bs4/doc/#">Beautifulsoup Documentation</a></li><li><a href="https://python-visualization.github.io/folium/">Folium Documentation</a></li><li><a href="https://medium.com/@bobhaffner/creating-a-legend-for-a-folium-map-c1e0ffc34373">Creating a legend for a Folium map</a></li><li><a href="https://developer.mapquest.com/documentation/geocoding-api/">Map Quest Geocoding API</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=336e4dc9ead" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/collect-and-visualize-covid-19-data-336e4dc9ead">Collect and Visualize COVID-19 Data</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>