<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Wenjun on Medium]]></title>
        <description><![CDATA[Stories by Wenjun on Medium]]></description>
        <link>https://medium.com/@wenjun.sarah.sun?source=rss-c8f8d68255b0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*OOP3gml0fr82ioO0BQQsyA.png</url>
            <title>Stories by Wenjun on Medium</title>
            <link>https://medium.com/@wenjun.sarah.sun?source=rss-c8f8d68255b0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 09:57:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@wenjun.sarah.sun/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Visualize Google’s GDPR Fines with Waffle Chart in R]]></title>
            <link>https://medium.com/@wenjun.sarah.sun/visualize-googles-gdpr-fines-with-waffle-chart-in-r-1b75d8e51535?source=rss-c8f8d68255b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/1b75d8e51535</guid>
            <category><![CDATA[google]]></category>
            <category><![CDATA[r]]></category>
            <category><![CDATA[gdpr]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[data-privacy]]></category>
            <dc:creator><![CDATA[Wenjun]]></dc:creator>
            <pubDate>Wed, 22 Apr 2020 15:07:07 GMT</pubDate>
            <atom:updated>2020-04-22T15:32:38.915Z</atom:updated>
            <content:encoded><![CDATA[<p>Google is fined twice under Europe’s General Data Protection Regulation (GDPR) for a total of 57 million euros (about 63 million USD). One of which is the largest GDPR penalty to date. However, this is less than 0.2% of its net income in 2019.</p><p>So I think it would be interesting to visualize the size of the penalty compared to Google’s annual net income.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y1AeHt18274HP3fr71WNXQ.png" /></figure><p>I choose a waffle chart for this visualization because I think waffle charts can do a good job when there are only a few categories and it allows us to use one category as “a unit” (see below for an example) — so I can use it to emphasize the huge gap between the penalty and Google’s profit.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/514/1*Xe4uy1OxdgQejWyGOR97WA.jpeg" /><figcaption>The graph is created by <a href="https://www.nytimes.com/2008/07/20/business/20debt.html">the New York Times</a>, I stole it from <a href="https://github.com/hrbrmstr/waffle">Bob Rudis’s repo for the Waffle package</a>.</figcaption></figure><p>The package I used to make this viz is waffle, which is created by <a href="https://rud.is/b/">Bob Rudis</a>. You can find more about this amazing package at <a href="https://github.com/hrbrmstr/waffle">his github</a>.</p><h3>Data</h3><p>The GDPR Fines data comes from <a href="https://www.privacyaffairs.com/gdpr-fines/">Privacy Affairs</a>. Thank <a href="https://github.com/rfordatascience/tidytuesday/blob/master/data/2020/2020-04-21/readme.md">#TidyTuesday</a> for tidying the data and making it available for download. The net income data is from <a href="https://abc.xyz/investor/static/pdf/2019Q4_alphabet_earnings_release.pdf">Alphabet’s Fiscal Year 2019 Results</a>.</p><p>All code can be found <a href="https://github.com/SUN-Wenjun/data_viz/tree/master/GDPR">here</a>.</p><p>This is the 8th post of my #100dayproject. Hope you enjoyed it! Comments/thoughts/feedback would be welcomed ❤</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b75d8e51535" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Visualize the Greatest Hip Hop Songs with Wordcloud in R]]></title>
            <link>https://medium.com/@wenjun.sarah.sun/visualize-the-greatest-hip-hop-songs-with-wordcloud-in-r-6b0f24aa8a24?source=rss-c8f8d68255b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/6b0f24aa8a24</guid>
            <category><![CDATA[hip-hop]]></category>
            <category><![CDATA[rap]]></category>
            <category><![CDATA[word-cloud]]></category>
            <category><![CDATA[r]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Wenjun]]></dc:creator>
            <pubDate>Sun, 19 Apr 2020 02:26:57 GMT</pubDate>
            <atom:updated>2020-04-19T02:26:57.455Z</atom:updated>
            <content:encoded><![CDATA[<p>In 2019, <a href="http://www.bbc.com/culture/story/20191007-the-greatest-hip-hop-songs-of-all-time-who-voted">BBC</a> polled 108 critics from 15 countries to find out the best hip-hop song. Based on the list, I use wordcloud to visualize the top frequent words in songs from female and male artists.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YJocqDrQWmXYCbttLU1Vsg.png" /></figure><p>From the wordcloud, we can see that women rap way less than men. In fact, out of the 311 songs, there are only 23 songs by women rappers. (BBC made a follow-up <a href="http://www.bbc.com/culture/story/20191007-why-are-there-so-few-women-in-best-of-hip-hop-polls">article</a> on why there are so few women in this polls and other “best-of” lists. )</p><h3>how I made this plot …</h3><p>The whole source file can be found <a href="https://github.com/SUN-Wenjun/data_viz/blob/master/best_hip_hop/best_hip_hop.Rmd">here</a>. To sum up, there are three main steps, and the code for each step is also linked below.</p><ol><li>Get lyrics <a href="https://github.com/SUN-Wenjun/data_viz/blob/3899626d320ad5b7de468eee864802ee67663f04/best_hip_hop/best_hip_hop.Rmd#L51">(with package </a><a href="https://github.com/SUN-Wenjun/data_viz/blob/3899626d320ad5b7de468eee864802ee67663f04/best_hip_hop/best_hip_hop.Rmd#L51">genius and </a><a href="https://github.com/SUN-Wenjun/data_viz/blob/3899626d320ad5b7de468eee864802ee67663f04/best_hip_hop/best_hip_hop.Rmd#L51">geniusr)</a> and more lyrics <a href="https://github.com/SUN-Wenjun/data_viz/blob/3899626d320ad5b7de468eee864802ee67663f04/best_hip_hop/best_hip_hop.Rmd#L155">(by hand)</a>;</li><li><a href="https://github.com/SUN-Wenjun/data_viz/blob/3899626d320ad5b7de468eee864802ee67663f04/best_hip_hop/best_hip_hop.Rmd#L199">Clean</a> the text while keeping the vibe of Hip Hop;</li><li>Make the <a href="https://github.com/SUN-Wenjun/data_viz/blob/3899626d320ad5b7de468eee864802ee67663f04/best_hip_hop/best_hip_hop.Rmd#L225">wordcloud</a> (with package ggwordcloud)</li></ol><h3>Source</h3><p>Greatest hip-hop song list: <a href="http://www.bbc.com/culture/story/20191007-the-greatest-hip-hop-songs-of-all-time-who-voted">BBC Music</a> by way of <a href="https://blog.datawrapper.de/best-hip-hop-songs-of-all-time-visualized/">Simon Jockers at Datawrapper</a>. I downloaded the data from <a href="https://github.com/rfordatascience/tidytuesday/blob/master/data/2020/2020-04-14/readme.md">#TidyTuesday</a>.</p><p>This is the 7th post of my #100dayproject. Hope you enjoyed it! Feel free to tell me your thoughts/comments/feedback :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6b0f24aa8a24" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Visualizing Mountain and Flat Stages in Tour De France]]></title>
            <link>https://medium.com/@wenjun.sarah.sun/visualizing-mountain-and-flat-stages-in-tour-de-france-cde227e98145?source=rss-c8f8d68255b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/cde227e98145</guid>
            <category><![CDATA[tour-de-france]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[area-chart]]></category>
            <category><![CDATA[ggplot2]]></category>
            <dc:creator><![CDATA[Wenjun]]></dc:creator>
            <pubDate>Wed, 08 Apr 2020 02:09:32 GMT</pubDate>
            <atom:updated>2020-04-08T02:16:48.263Z</atom:updated>
            <content:encoded><![CDATA[<p>Today I looked in Tour de France, a men’s multiple stage cycling race held in France every year since 1903.</p><p>The data used for this visualization comes from <a href="https://github.com/rfordatascience/tidytuesday/blob/master/data/2020/2020-04-07/readme.md">TidyTuesday</a>, which nicely combines the data from <a href="https://github.com/alastairrushworth/tdf">Alastair Rushworth’s data package </a><a href="https://github.com/alastairrushworth/tdf">tdf</a> and <a href="https://www.kaggle.com/jaminliu/a-brief-tour-of-tour-de-france-in-numbers/data">Kaggle</a>.</p><p>I used an<strong> area plot </strong>to visualize the change of proportion of mountain/flat trails over the years. Perhaps surprisingly, the plot part is the easiest part, a simple geom_area() does the trick. In contrast, I spent a lot more time to add the annotation (using geom_point() and geom_text() ) and fit it to the image (with the help of scale_x_continuous(..., expand=...)).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9ERof_rhJweNuwo0u7VekQ.png" /><figcaption><a href="https://github.com/SUN-Wenjun/data_viz/blob/a6fb59640fde45b9b1043d19da9ad62d14c6a2b1/Tour_de_France/Tour_de_France.Rmd#L57">code</a></figcaption></figure><p>As you can probably see from the plot above, the proportion of mountain trails in Tour De France shown an increasing trend since the 1940s. Does it mean that Tour De France is more mountainous now than before? The answer is yes and no.</p><p>Instead of using proportion, the plot below visualizes the total distance of mountain stages and flat stages over time. We can find that: the total distance of mountain stages remains roughly the same, but that of flat stages is decreasing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1QjO2j77ILWzLhncZlDbWA.png" /><figcaption><a href="https://github.com/SUN-Wenjun/data_viz/blob/a6fb59640fde45b9b1043d19da9ad62d14c6a2b1/Tour_de_France/Tour_de_France.Rmd#L104">code</a></figcaption></figure><p>While this plot explains the increasing proportion of mountain stages in Tour de France, it brings another question — what happened around 1927? Why both mountain stages and flat stages plunged at that time?</p><p>To answer this question, I further divided the stage types — it turned out the sudden change was because of <a href="https://en.wikipedia.org/wiki/1927_Tour_de_France">the introduction of “team time trial” and its quick abolition in 1929</a>. Cool.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qz7Ny1F-Dge-J7oNt4qcFA.png" /><figcaption><a href="https://github.com/SUN-Wenjun/data_viz/blob/a6fb59640fde45b9b1043d19da9ad62d14c6a2b1/Tour_de_France/Tour_de_France.Rmd#L140">code</a></figcaption></figure><h4>A side note: How to remove gaps or breaks in area chart?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/695/1*zcz8tv1iTLPdF-rnTYs76g.png" /></figure><p>By the way, it was kind of tricky to draw a <em>beautiful</em> area plot when categories emerge and disappear. (Not doing it right will result in an ugly plot like the left one.) I dealt with it by first using pivot_wider() to make one column for each category, then using replace_na() to add zeros, and using pivot_longer() to turn it back to the original shape. (Below is part of the code. For more details, check <a href="https://github.com/SUN-Wenjun/data_viz/blob/a6fb59640fde45b9b1043d19da9ad62d14c6a2b1/Tour_de_France/Tour_de_France.Rmd#L153">here</a>.)</p><p>Code can be found <a href="https://github.com/SUN-Wenjun/data_viz/tree/master/Tour_de_France">here</a>. This is the 6th post of my #100dayproject. Hope you enjoyed it! Thoughts/comments/feedbacks are always super welcomed :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cde227e98145" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Visualize Beer Production with Bump Chart in R with ggplot2]]></title>
            <link>https://medium.com/@wenjun.sarah.sun/visualize-beer-production-with-bump-chart-in-r-with-ggplot2-c72fccdf2250?source=rss-c8f8d68255b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/c72fccdf2250</guid>
            <category><![CDATA[bump-chart]]></category>
            <category><![CDATA[ggplot2]]></category>
            <category><![CDATA[tidytuesday]]></category>
            <category><![CDATA[r]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Wenjun]]></dc:creator>
            <pubDate>Wed, 01 Apr 2020 19:04:55 GMT</pubDate>
            <atom:updated>2020-04-01T19:04:55.173Z</atom:updated>
            <content:encoded><![CDATA[<h3>Visualize Beer Production with Bump Chart in R</h3><p>This is my first submission to #TidyTuesday, a weekly data visualization project focuses on ggplot2 and other tools in the tidyverse ecosystem.</p><h3>Data</h3><p>The data comes from the <a href="https://www.ttb.gov/beer/statistics">Alcohol and Tobacco Tax and Trade Bureau (TTB)</a>. A clean version of the data can be found at <a href="https://github.com/rfordatascience/tidytuesday/blob/master/data/2020/2020-03-31/readme.md">TidyTuesday github</a>.</p><h3>My viz</h3><h4>First attempt:</h4><p>Highlight all states whose rank in 2019 is higher than that in 2008 — but the resulted graph was too noisy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*pvCNErUY9PoW05HD4vJfOw.png" /><figcaption>Code for <a href="https://github.com/SUN-Wenjun/data_viz/blob/d3e2cbc17e69fb363b38fce9e0cc4c6290b9c0f1/beer_production/beer_production.Rmd#L41">preprocessing</a> and <a href="https://github.com/SUN-Wenjun/data_viz/blob/d3e2cbc17e69fb363b38fce9e0cc4c6290b9c0f1/beer_production/beer_production.Rmd#L76">plotting</a> is available on github</figcaption></figure><p>But from this plot, we can see some states had a larger increase in ranking. So I focus on this point in my final plot.</p><h4>Final plot:</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7pFIT8ZNZYxrxYdl87lZHg.png" /><figcaption><a href="https://github.com/SUN-Wenjun/data_viz/blob/d3e2cbc17e69fb363b38fce9e0cc4c6290b9c0f1/beer_production/beer_production.Rmd#L76">code can be found here</a></figcaption></figure><p>Because we already got a lots of information to plot (50+ lines), this plot focused on a fewer number of states and used only two colors. Size is also adjusted to a longer one, so there’s more space between lines.</p><p>Code can be found <a href="https://github.com/SUN-Wenjun/data_viz/blob/master/beer_production/beer_production.Rmd">here</a>. This is the 5th post of my #100dayproject. Hope you enjoyed it! Thoughts/comments/feedbacks are always super welcomed :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c72fccdf2250" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Visualize UK’s Pizza Topping]]></title>
            <link>https://medium.com/@wenjun.sarah.sun/visualize-uks-pizza-topping-f23793bfb1f4?source=rss-c8f8d68255b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/f23793bfb1f4</guid>
            <category><![CDATA[pizza]]></category>
            <category><![CDATA[tableau]]></category>
            <category><![CDATA[makeovermonday2020]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Wenjun]]></dc:creator>
            <pubDate>Sun, 29 Mar 2020 21:31:42 GMT</pubDate>
            <atom:updated>2020-04-17T15:34:06.604Z</atom:updated>
            <content:encoded><![CDATA[<h3>Visualize UK’s Favorite Pizza Topping</h3><p>My first entry for the <strong>#makeovermonday</strong> is about — UK’s favorite pizza toppings!</p><p>Original Viz:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jPqvRw82HDO8kQkPSDqAYA.png" /><figcaption>source: <a href="https://yougov.co.uk/topics/politics/articles-reports/2017/03/06/does-pineapple-belong-pizza">YouGov</a></figcaption></figure><p>While it may look delicious, pie chart is a bad idea because people are bad at reading and comparing angles. For this chart, it’s even worse because the size of each pizza is not related to the labeled percentage at all.</p><p>My makeover:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*cDy0LMFdM9_ojvifK3zD5g.png" /></figure><p>It’s interesting to find that: while mushrooms, ham, peppers, pepperoni, and onion are popular for both men and women, men show more love to meat and spice and women show more love to veggies.</p><p>I think a proportional area chart like below can also do the work, maybe I’ll give it a try next time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/532/1*sOAXDuE-2Aa-IMZdFcZrvA.png" /><figcaption>source: <a href="https://datavizproject.com/data-type/proportional-area-chart-half-circle/">datavizproject</a></figcaption></figure><p>A public link for my visualization can be found <a href="https://public.tableau.com/profile/wenjun.sun#!/vizhome/MOM_W13_pizza_topping/Dashboard1?publish=yes">here</a>. This is the 4th post of my #100dayproject. Hope you enjoyed it! Thoughts/comments/feedbacks are always super welcomed :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f23793bfb1f4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Maps of where the Fortune 500 companies in R]]></title>
            <link>https://medium.com/@wenjun.sarah.sun/maps-of-where-the-fortune-500-companies-in-r-6ee6e98c64f8?source=rss-c8f8d68255b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/6ee6e98c64f8</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[leaflet]]></category>
            <category><![CDATA[interactive]]></category>
            <category><![CDATA[r]]></category>
            <category><![CDATA[maps]]></category>
            <dc:creator><![CDATA[Wenjun]]></dc:creator>
            <pubDate>Sat, 28 Mar 2020 17:26:51 GMT</pubDate>
            <atom:updated>2020-03-29T15:22:23.500Z</atom:updated>
            <content:encoded><![CDATA[<p>This is the 3rd project of my #100dayproject on data visualization. I made maps of where the Fortune 500 companies have their headquarters and how headquarter locations are related to state corporate income tax rates — statically and interactively (!).</p><h3>Data</h3><p>Data on Fortune 500 (in 2018) headquarter addresses is from <a href="https://www.geographyrealm.com/map-and-list-of-fortune-1000-companies-for-2018/">Geography Realm</a>. Data on state corporate income tax rates is from <a href="https://taxfoundation.org/state-corporate-income-tax-rates-brackets-2018/">Tax Foundation</a>.</p><h3>Fortune 500 Companies by State</h3><h4>Choropleth map made with ggplot2</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TDS427uIRJYaSHAsVfS5FA.png" /><figcaption>Package &quot;albersusa” provides a neat composite projection of the US where Alaska and Hawaii are placed closer to the continent. Find how I made this map <a href="https://github.com/SUN-Wenjun/data_viz/blob/4b6351a9f7a21f75be74b442b8eb9a5f102c865e/fortune_500_headquarters/fortune_500_hq.Rmd#L56">here</a>.</figcaption></figure><h4>But choropleth maps can be misleading …</h4><p>Although choropleth maps are visually appealing, they are also criticized for being misleading — they falsely imply an association between the area of a place and its importance in the display.</p><p>One alternative is to use same-sized shapes to represent different states， make sure all states are equal. See below for an example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z8hD7LxDbul2CU9fxqwIKg.png" /><figcaption>Created with package “<a href="https://github.com/hrbrmstr/statebins">statebins</a>&quot; in R. Find code <a href="https://github.com/SUN-Wenjun/data_viz/blob/4b6351a9f7a21f75be74b442b8eb9a5f102c865e/fortune_500_headquarters/fortune_500_hq.Rmd#L84">here</a>.</figcaption></figure><h4>Let’s go interactive!</h4><p>Mapping is no fun without interactivity. The leaflet package in R enables us to build interactive maps within R — this package a htmlwidget which allows users to build interactive maps using the Javascript library.</p><p>Here, I built the same HQs by state map again but in an interactive way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nwj-JyszcorqrPQAxZM9ew.gif" /><figcaption>Find out how I made it<a href="https://github.com/SUN-Wenjun/data_viz/blob/4b6351a9f7a21f75be74b442b8eb9a5f102c865e/fortune_500_headquarters/fortune_500_hq.Rmd#L115"> here</a></figcaption></figure><h3>Are companies headquartered in low tax states?</h3><p>Beyond the number of HQs of each state, I am also interested in the relation between state tax rates and headquarter locations.</p><h4>Top corporate income tax rate by state</h4><p>First, I plotted a map shaded by top corporate income tax rates. This map is similar to the previous one, but it has a continuous scale (tax rate). It’s a bit tricky to sort a continuous scale descendingly in legend in Leaflet R, but I found this function wrote by <a href="https://github.com/rstudio/leaflet/issues/256">mpriem89</a> that solves this problem perfectly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ny_eFwVMw8hk-0rG3QyQRQ.gif" /><figcaption><a href="https://github.com/SUN-Wenjun/data_viz/blob/4b6351a9f7a21f75be74b442b8eb9a5f102c865e/fortune_500_headquarters/fortune_500_hq.Rmd#L192">code</a></figcaption></figure><h4>Is mapping the only (and best) way?</h4><p>While it’s a huge temptation to map geographical data when we have them, mapping is not necessarily the best way. In this case, what matters is the <strong>association</strong> between tax rates and the number of HQs, so a scatter plot is more suitable.</p><p>The first scatter plot shows the relationship between the number of Fortune 500 HQs and the state corporate tax rate. The second scatter plot adjusted to state population (scraped from <a href="https://en.wikipedia.org/wiki/List_of_U.S._states_and_territories_by_population">wiki</a>), so Y becomes the number of HQs per million people.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ITCB_BVksNu8zMLBYUl9bg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nvfG8sUnUE3NgdjgqA1Oww.png" /><figcaption>No relationship between tax rate and HQs is found. (Find code <a href="https://github.com/SUN-Wenjun/data_viz/blob/4b6351a9f7a21f75be74b442b8eb9a5f102c865e/fortune_500_headquarters/fortune_500_hq.Rmd#L344">here</a>)</figcaption></figure><h3>State tax rates, HQ locations, and Fortune 500 rank</h3><p>Finally, I want to know if company of different sizes will show a difference in HQ locations and have more fun with Leaflet. So I add locations of all HQs to the state map shaded by tax rates and size them according to the company’s ranking on the Fortune 500 list.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5tBF7frlNVD9dQxH5OQ5SA.gif" /><figcaption><a href="https://github.com/SUN-Wenjun/data_viz/blob/4b6351a9f7a21f75be74b442b8eb9a5f102c865e/fortune_500_headquarters/fortune_500_hq.Rmd#L436">code</a></figcaption></figure><h3>On a side note:</h3><ol><li><strong>State name and abbreviation</strong>. Dealing with state name and its abbreviation can get troublesome, especially when working with several different datasets. I found the abbr2state() and state2abbr() function from package openintro very handy for this problem — it swift an input state name to its abbreviation and vice versa.</li><li><a href="https://rstudio.github.io/leaflet/"><strong>THE</strong></a><strong> website about Leaflet for R</strong>. This website provides almost everything you need to know about creating interactive maps with Leaflet in R. Very helpful.</li></ol><p>Code and gifs can be found at this <a href="https://github.com/SUN-Wenjun/data_viz/tree/master/fortune_500_headquarters">github</a> repo. This is the third post of my #100dayproject. Hope you enjoyed it! Thoughts/comments/feedbacks are always super welcomed :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6ee6e98c64f8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Choropleth Map of World Measles Vaccinations with ggplot2]]></title>
            <link>https://medium.com/@wenjun.sarah.sun/choropleth-map-of-world-measles-vaccinations-with-ggplot2-d9bc7684bc47?source=rss-c8f8d68255b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/d9bc7684bc47</guid>
            <category><![CDATA[maps]]></category>
            <category><![CDATA[world]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[r]]></category>
            <category><![CDATA[ggplot2]]></category>
            <dc:creator><![CDATA[Wenjun]]></dc:creator>
            <pubDate>Tue, 17 Mar 2020 00:03:36 GMT</pubDate>
            <atom:updated>2020-03-17T00:03:36.566Z</atom:updated>
            <content:encoded><![CDATA[<p>This is the second day of my #100dayproject on data visualization. I will make a choropleth map of measles vaccinations around the world with ggplot2 in R.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*hEwampiWs1QFU7NIh5_e0w.png" /></figure><h4>Load World Development Indicators data</h4><p>World Development Indicators provide data on measles vaccinations for more than 200 countries and regions from 1980. To access the data, I used an R package named ‘<a href="https://cran.r-project.org/web/packages/WDI/WDI.pdf">WDI</a>’, a cool package created by Vincent Arel-Bundock.</p><pre>library(WDI)</pre><pre>df &lt;- WDI(indicator = &quot;SH.IMM.MEAS&quot;, extra = TRUE)<br>df &lt;- df %&gt;% <br>  filter(!is.na(SH.IMM.MEAS)) %&gt;%<br>  filter(region != &quot;Aggregates&quot;)</pre><pre>df_2018 &lt;- df %&gt;%<br>  filter(year==2018) %&gt;%<br>  select(country, SH.IMM.MEAS, iso3c) # this viz will focus on 2018</pre><h4>Load world map data</h4><p>There are many types of spatial objects and many ways to load them in R. For this viz, because I want to stay in the kingdom of ggplot2, I will load map data in sf format and use geom_sf() to implement them.</p><pre># download mapping data from <a href="http://www.naturalearthdata.com/">Natural Earth</a> <br>library(&quot;rnaturalearth&quot;)<br>library(&quot;rnaturalearthdata&quot;)</pre><pre>world &lt;- ne_countries(scale = &quot;medium&quot;, returnclass = &quot;sf&quot;)</pre><h4>Merge measles vaccinations data to mapping data</h4><p>To plot measles vaccinations data on a map, we need to merge it to the mapping data. A simple left_join will do the task.</p><pre>world_merged &lt;- left_join(world, df_2018, by = c(&quot;iso_a3&quot;=&quot;iso3c&quot;))</pre><h4>Finally, let’s make the plot</h4><p>This first part tells ggplot2 to 1) color countries with their share of infants vaccinated against measles, 2) do not color borderline, and 3) make the map a bit more transparent.</p><pre>p &lt;- ggplot(data = world_merged) + <br>     geom_sf(aes(fill = SH.IMM.MEAS), color = NA, alpha = 0.8)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*w0e613Og0Vk4w11WKZ8zlA.jpeg" /></figure><p>This part defines which color palette to use. na.value sets the default color for regions with no data on vaccination.</p><pre>p &lt;- p + <br>     scale_fill_viridis_c(option = &quot;plasma&quot;, na.value=&quot;gray90&quot;, breaks=seq(0,100,20), limits=c(20,100))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*M9pq40rP9TZdqyDuHck36g.jpeg" /></figure><p>The last part is adding titles and styling the graph.</p><pre>p &lt;- p + <br>     labs(title = &quot;Fighting Measles with Vaccination&quot;,<br>          subtitle = &quot;a world map of the share of infants vaccinated against measles&quot;,<br>          caption = &quot;Source: World Development Indicators, World Bank 2018&quot;,<br>          fill = &quot;% of infants \nvaccinated against measles&quot;) +</pre><pre>     theme_map() +<br>     theme(plot.subtitle=element_text(size=10, color=&quot;grey40&quot;, lineheight=.9, face=&quot;italic&quot;), <br>           plot.caption=element_text(size=7, color=&quot;grey40&quot;),<br>           legend.title = element_text(size=7, color=&quot;grey40&quot;))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*hEwampiWs1QFU7NIh5_e0w.png" /></figure><p>This is the 2nd day of my #100dayproject. Hope you enjoyed it. Code can be found at this <a href="https://github.com/SUN-Wenjun/data_viz/tree/master/map_measles_vaccinations">github</a> repo. If you like it, please share it or give me a clap. Comments and feedbacks are always super welcomed :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d9bc7684bc47" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Visualize Student Loans with ggplot2]]></title>
            <link>https://medium.com/@wenjun.sarah.sun/visualize-student-loans-with-ggplot2-c44ea0cfd911?source=rss-c8f8d68255b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/c44ea0cfd911</guid>
            <category><![CDATA[100dayproject]]></category>
            <category><![CDATA[r]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[ggplot2]]></category>
            <category><![CDATA[student-loans]]></category>
            <dc:creator><![CDATA[Wenjun]]></dc:creator>
            <pubDate>Sun, 01 Mar 2020 00:18:54 GMT</pubDate>
            <atom:updated>2020-03-01T01:39:26.474Z</atom:updated>
            <content:encoded><![CDATA[<p>Making progress in data visualization is one of my new year resolutions for 2020, and I decided to start a <strong>#100dayproject</strong> on data visualization. Today is the first day and I am going to visualize student loans in the US with ggplot2.</p><p>The data<strong> </strong>comes from the Survey of Consumer Finances which is conducted every three years from 1989 onwards.</p><h3>Student loans over time</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*daD6LiiJbUbjXqS7eL8VhA.png" /></figure><p>The first data viz is a line chart overlaying on a bar chart. It shows the increasing trend of student loans in terms of number of borrowers and the average value of student loans among US households.</p><p>I put the legend to the top-left corner and decorated them the same way as they appear in the graph so that readers would know <em>how to</em> read the graph before they start reading it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QvjI5rC86YmZFgAMTJwRiQ.png" /></figure><p>The second graph focuses on the change of loan-to-income ratio over time. My aim is to emphasize the steep increase after 2007, and I hope the change of background color and line color does make you notice the trend.</p><h3>Compare student loan debt to other types of debt</h3><p>I am also interested in comparing student loans to other types of debt in terms of absolute value and change over time. So I created line plots for each type of debt and combine them together in one graph with facet_grid().</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EcmI1OF2vIFSvBTPK2KeNw.png" /></figure><p>Because line plots share the same x scale (i.e. year), so I only note it once in the top-left corner.</p><h3>Relate household characteristics to student loan indebtedness</h3><p>Although the survey is taken at a household level which kind of limits our inference about individuals, I still found it interesting to explore the relationship between some household characteristics to indebtedness.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*li-lRCXGdyDlHEDImhFdCg.png" /><figcaption>Btw, does anyone find a violin plot without outline looks like skate fish too…?</figcaption></figure><p>This graph combines a violin plot and a box plot to show that people with higher education levels tend to have more student loans.</p><p>The data is highly skewed as many data related to money, which makes it difficult to plot. But scale_y_continuous(trans=&#39;log10&#39;, labels=dollar) helps to transform the scale of the y-axis (plus add a cute dollar sign to each number), which saves the plot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8mli2wY_HY3APUlo5lHOOw.png" /></figure><p>This is a waterfall chart which categorizes student loan borrowers according to their graduate year and visualizes the increase/decrease in average student loan. Although it looks like a bar chart, all ‘bars’ here are actually created by geom_rect().</p><h3>Income, Education, and Student Loans</h3><p>The relationship between wealth, education, and student loans is a complicated one. On the one hand, we may expect households with higher incomes to generally be less indebted. On the other hand, higher incomes may be the result of longer (and potentially more expensive) education.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cSY-cky_m4lz3ls0eXuzdA.png" /></figure><p>This graph is created with geom_jitter(), it aims to visualize the complicated relationship between income, education, and student loans. Dots with a deeper blue color means a higher education level, and we can see more deep blue dots on the top (i.e. more in debt) and on the right (i.e. higher income level).</p><h3>Going broke</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xdkst9VRzpEKN-d9_PrHDw.png" /></figure><p>The last visualization is a grouped bar chart. All debts are marked with grey except for student loans, bars are kept in the same order as the legend so if anyone is interested in another type of debt (although it’s not what I want to emphasize), he or she can also get to know its type by referring to the legend.</p><p>This is the first day of my #100dayproject on data visualization. Hope you enjoyed it. Code can be found at my <a href="https://github.com/SUN-Wenjun/data_viz/tree/master/student_loans">github</a>. If you like it, please share it or give me a clap. Comments and feedbacks are super welcomed.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c44ea0cfd911" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Trumpy Tweets — Trump Tweet Classifiers and Generators]]></title>
            <link>https://medium.com/analytics-vidhya/trumpy-tweets-trump-tweet-classifiers-and-generators-9ccfddc7764e?source=rss-c8f8d68255b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ccfddc7764e</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[donald-trump]]></category>
            <category><![CDATA[text-classification]]></category>
            <category><![CDATA[text-generation]]></category>
            <category><![CDATA[tweet]]></category>
            <dc:creator><![CDATA[Wenjun]]></dc:creator>
            <pubDate>Wed, 18 Dec 2019 20:30:39 GMT</pubDate>
            <atom:updated>2019-12-23T08:53:38.646Z</atom:updated>
            <content:encoded><![CDATA[<h3>Trumpy Tweets — Trump Tweet Classifiers and Generators</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g1Nx3ZZ9UQaw0v614813KA.png" /></figure><h3>Abstract</h3><ul><li>Used about 24k tweets of Trump to train Trump tweet generators.</li><li>Added another 24k tweets from Obama, Hillary, and Kim Kardashian to train tweet classifiers.</li><li>Compared the performance of different classifiers, used the better-performed one to evaluate the performance of the tweet generators and picked the most ‘Trumpy’ fake tweets.</li></ul><h3>Motivation</h3><p>I am a fan of The Daily Show hosted by Trevor Noah and I love watching his Trump impersonation. His shows remind me that Trump has a very characterized way of talking — which could hopefully be recognized and mimic by NLP models. Besides, Trump is so addicted to Twitter that it provides me lots of data for training my bot.</p><h3>Plan</h3><p>I plan to build two tweet generators and two tweet classifiers, and use the better-performed classifier to evaluate the performance of the generators and select the most ‘Trumpy’ made-up tweets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Uc3DzvymNHrafRYppl5OQA.png" /><figcaption>the plan</figcaption></figure><p>I plan to try out a traditional statistical approach as well as a modern neural network approach. So for the classifier, I will train a Multinomial Naive Bayes model and a long short-term memory (LSTM) model; and for the generator, I will train a Markov Chain model and an LSTM model.</p><h4>Classifier</h4><ul><li>Multinomial Naive Bayes</li><li>LSTM</li></ul><h4>Generator</h4><ul><li>Markov Chain Model</li><li>LSTM</li></ul><h3>Data</h3><h4>Trump’s tweets</h4><p>The Twitter API has many limits — it only gives access to the last 3200 tweets of a user, but Brendan Brown made a great website called ‘<a href="http://www.trumptwitterarchive.com/">Trump Twitter Archive</a>’ that collects almost all of Trump’s tweets. So I used this archive to get data of Trump’s tweet instead. The raw .csv file is included in the data folder of the GitHub repo [add a link here] of this project, but the definitive source is <a href="https://github.com/bpb27/trump_tweet_data_archive">Brown&#39;s Github repository.</a></p><h4>Other people’s tweets</h4><p>To train a classifier that can classify Trump and non-Trump tweets, I need some non-Trump data. This <a href="https://www.kaggle.com/speckledpingu/RawTwitterFeeds/data">Kaggle dataset</a> collects tweets from several politicians and celebrities. For this project, I took tweets from Obama, Hillary, and Kim Kardashian as ‘non-Trump’ tweets.</p><h3>Data Preprocessing</h3><p>I did two levels of data preprocessing: 1) basic cleaning for all tweets and 2) further preprocessing for training classifiers. The reason why it is done in two steps is that the data I want the classifiers to learn is slightly different from that for the generators. For example, for the generator, I want to keep @mention, #tag, and special characters (like !?,.) in our dataset, because we do want our bot to be able to @mention, #tag, or using exclamation marks. However, things like @, #, and special characters create many difficulties in tokenizing, so I need to remove them before training the classifiers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XovfFZCFxwfANbvit4ChUw.png" /><figcaption>There are two parts of data preprocessing.</figcaption></figure><h4>Data Preprocessing Part 1</h4><p>First, all retweets are removed. This leaves us 24595 tweets from Trump and 24127 tweets from Obama, Hillary, and Kim Kardashian combined.</p><p>Then I removed quotations, URLs, ‘\n’s from tweets and replaced ‘&amp;amp’ with ‘and’.</p><pre>def clean_tweet(t):<br>    # remove quotes<br>    t = re.sub(r&#39;&quot;@.*&#39;, &#39;&#39;, t)<br>    t = re.sub(r&#39;^“.*”$&#39;, &#39;&#39;, t)</pre><pre>    # remove URLs<br>    t = re.sub(r&#39;https*:\/\/\S*&#39;, &#39;&#39;, t)<br>    t = re.sub(r&#39;pic\.twitter\.com\/\S*&#39;, &#39;&#39;, t)</pre><pre>    # remove \n<br>    t = re.sub(&#39;\n&#39;, &#39; &#39;, t)</pre><pre>    # remove extra whitespaces<br>    t = re.sub(r&#39;\s+&#39;, &#39; &#39;, t)</pre><pre>    # replace &#39;&amp;amp&#39; with &#39;and&#39;<br>    t = re.sub(&#39;&amp;amp;&#39;, &#39;and&#39;, t) </pre><pre>    return(t)</pre><p>The data cleaned above were later used to train the generators.</p><h4>Data Preprocessing Part 2</h4><p>In addition to the cleaning above, for training the classifiers, we need to remove things like @ and # first.</p><pre>def prepare_tweet_clf(t):<br>    # clean<br>    t = t.lower()<br>    t = re.sub(&quot;&#39;ll&quot;, &#39; will&#39;, t) # replace abbreviations<br>    t = re.sub(&quot;won&#39;t&quot;, &#39;will not&#39;, t)<br>    t = re.sub(&quot;n&#39;t&quot;, &#39; not&#39;, t) <br>    t = re.sub(r&#39;@[A-Za-z0-9_]+&#39;, &#39;&#39;, t) # remove @mention<br>    t = re.sub(r&#39;#[A-Za-z0-9_]+&#39;, &#39;&#39;, t) # remove #tag<br>    t = re.sub(r&#39;[^a-zA-Z ]&#39;, &#39;&#39;, t) # remove special characters</pre><pre>    # remove stopwords <br>    stop = set(stopwords.words(&#39;english&#39;)) <br>    stop.update([&#39;rt&#39;, &#39;cc&#39;]) <br>    stop = stop - set([&#39;no&#39;, &#39;not&#39;, &#39;never&#39;]) <br>    t = [word for word in t.split(&#39; &#39;) if word not in stop] <br>    t = &#39; &#39;.join(t)<br>    <br>    return t</pre><p>The data that went through the second cleaning step were used to train the classifiers.</p><h3>Models</h3><p>There are two kinds of models in this project:</p><ol><li>text classifiers that classify Trump’s tweets and other people’s tweets</li><li>text generators that generate made-up tweets in the style of Trump</li></ol><p>The classifier with the highest prediction accuracy on the test set is considered the best classifier and is used to evaluate the text generators.</p><h3>Tweet Classifier</h3><h4>Split data</h4><p>Trump and others’ tweets are first merged into one dataset, then the dataset is randomly split into a training set (80%) and a test set (20%).</p><pre>df_train, df_test = train_test_split(df_combine, test_size=0.2)</pre><pre>x_train, y_train = df_train[&#39;cleaned_text_clf&#39;], df_train[&#39;author&#39;]<br>x_test, y_test = df_test[&#39;cleaned_text_clf&#39;], df_test[&#39;author&#39;]</pre><h4>Metric</h4><p>Classifiers are optimized towards precision, which equals to</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dW6zOSSMYuZ1lPDBGqIk_g.png" /></figure><p>In this case, it can be written as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Spm6fccUpvVDejeXhs3sHw.png" /></figure><p>Intuitively, precision is the ability of the classifier not to label a fake tweet as Trump’s.</p><h4>Classifier 1 — Multinomial Naive Bayes</h4><p><a href="https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.MultinomialNB.html">The Multinomial Naive Bayes classifier</a> is suitable for classification with discrete features, for example, text classification.</p><p>The pipeline of this model consists of three parts: vectorize with CountVectorizer(), transform with TfidfTransformer(), and model with MultinomialNB(). I used a 10-fold cross-validated grid-search to optimize the parameters, and the best set of hyperparameters for the model are: using unigram, bigram, and trigram; using raw counts instead of tf-idf, and using a smoothing coefficient of 0.1.</p><p>The model was trained on the training set, and its performance was evaluated with the rest of the dataset. As the confusion matrix shows below, the Multinomial Naive Bayes classifier is pretty impressive — its accuracy is about <strong>0.9791</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/318/1*5gm-0mw0Tf7rXYn2TBo94Q.png" /><figcaption>confusion matrix of the Multinomial Naive Bayes classifier</figcaption></figure><h4>Classifier 2 — LSTM</h4><p>The second classifier is based on LSTM. Long short-term memory (LSTM) is a recurrent neural network (RNN) architecture that has feedback networks. It was developed to deal with the exploding and vanishing gradient problems that can be encountered when training traditional RNNs. As a variation of RNN, LSTM is well-suited to classifying, processing, and making predictions on serial data.</p><p>The model implemented in this project has two Bidirectional LSTM Layers, two Dense Layers, and an output layer. This model is inspired by the <a href="https://github.com/manashpratim/Tweet-Classification">hate tweets detector</a> created by Manash Pratim Barman.</p><p>I vectorized tokens with <a href="https://nlp.stanford.edu/projects/glove/">Twitter pre-trained word embedding from GloVe</a> (200d vectors) and trained the model for 20 epochs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/323/1*rTFCUsOb93kpYcTCT9AWVw.png" /><figcaption>confusion matrix of the LSTM classifier</figcaption></figure><p>The LSTM model got an accuracy of 0.8955 on the test set, which is not bad. But since the Multinomial Naive Bayes classifier gets a higher accuracy, I will use it to evaluate the performance of tweet generators.</p><h4>Model Comparison</h4><p>For this given task and dataset, the Multinomial Naive Bayes classifier outperformed the LSTM one. I think it is because text classification is a relatively easy task and our dataset is not large enough, so complex models like LSTM might suffer from overfitting. In other words, if we have more data, the performance of LSTM might get better.</p><h3>Tweet Generator</h3><h4>Generator 1 — Markov Chain Model</h4><p>Markov Chain is based on the idea that: the possible next states are entirely dependent on the present state. Based on this idea, we can train a Markov Chain model to learn things like: how likely you are going to find ‘bananas’/‘oranges’/‘apples’ after ‘like’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-MahqW4TBtrvslC14TNNSg.png" /><figcaption>a Markov Chain example</figcaption></figure><p>In other words, when generating text, we randomly pick the first word, then choose the ‘next word’ continuously on a probabilistic basis.</p><p>For the first tweet generator, I used a package called <a href="https://github.com/jsvine/markovify">Markovify</a>. It is a text generator package based on Markov Chain developed by Jeremy Singer-Vine. It gives users the options to set state_size (unigram/bigram/trigram) and to utilize a part-of-speech (POS) tagger.</p><p>Three models are trained: A bigram baseline model, a trigram one, and a trigram+POS one. I then generated 1000 tweets with the three models respectively and evaluate their performance in terms of how well their tweets fool the classifier trained earlier.</p><p><strong>Bigram Markov Chain model: </strong>The proportion of tweets fooled the classifier is 0.93. That is to say, among the 1000 fake tweets generated, 930 tweets are classified as ‘Trump’s’ by our Mutilnomial classifier. The most ‘Trumpy’ tweet (probability = 0.9999) is:</p><blockquote>From a large shipment of military weapons and supplies to my GREAT members at Trump International Golf Links in Scotland!</blockquote><p>The tweet that has a 75th percentile ‘Trumpy’ probability (probability = 0.98) is:</p><blockquote>Watch it, should be ashamed for using the celebrity’s name followed by new champion Chris Weidman!</blockquote><p>And the median ‘Trumpy’ tweet (probability =0.94) is:</p><blockquote>#Debates .USA has the guts to even greater unemployment.</blockquote><p><strong>Trigram Markov Chain model: </strong>The proportion of tweets generated by the trigram generator that fooled our classifier is 0.946. The most ‘Trumpy’ tweet we got (probability = 0.9999) is:</p><blockquote>The Rigged Witch Hunt headed by 13 Angry Democrats and others who are totally corrupt and/or conflicted.</blockquote><p>The 75th percentile ‘Trumpy’ tweet (probability = 0.99):</p><blockquote>Thank you, a very wise move that Ted Cruz FORGOT to file.</blockquote><p>The median tweet (probability = 0.96):</p><blockquote>In the meantime they continue to be drawn to it.</blockquote><p><strong>Trigram + POS Markov Chain model: </strong>the proportion of tweets fooled our classifier is 0.947 for this generator. The most ‘Trumpy’ tweet (probability = 0.9999) is:</p><blockquote>We want victory .. Trump International Golf Links Scotland .</blockquote><p>The 75th percentile ‘Trumpy’ tweet (probability = 0.99):</p><blockquote>I will beat Hillary easily , but Lindsey Graham says I wo n’t be around much longer , it ‘s called Envy .</blockquote><p>The median tweet (probability = 0.95):</p><blockquote>.On behalf of an entire Nation , THANK YOU for today ‘s update and GREAT WORK !</blockquote><p>— it seems that more complex models did get (sightly) better at fooling the classifier, and the fake tweets we got are pretty good even for the ‘median Trumpy’ ones.</p><h4>LSTM</h4><p>In this part, I utilized the <a href="https://github.com/minimaxir/textgenrnn">textgenrnn</a> packaged developed by Max Woolf to build the generator. <a href="https://github.com/minimaxir/textgenrnn">textgenrnn</a> is a Python 3 module on top of Keras/TensorFlow for creating char-rnns, it can generate text on character-level or on word-level.</p><p>Similar to the previous model, I generated 1000 tweets with each of the generators and calculate their success rate of fooling the classifier. Here are the tweets —</p><p><strong>Character-level LSTM model: </strong>the proportion of tweets fooled our classifier is 0.985. The most ‘Trumpy’ tweet (probability = 0.99999) is:</p><blockquote>.@marklevinshow interview discussing the Celebrity Apprentice and Crooked Hillary Clinton will only get born in the world. I will be the best thing to have the haters and the truth in the New York Times — just reported that I am the only one that the Fake News Media will do a g</blockquote><p>The 75th percentile ‘Trumpy’ tweet (probability = 0.99595):</p><blockquote>My @SquawkCNBC interview discussing the boardroom of the United States of Americans will be a great deal for the fact that China is always going to be the greatest of the world, and all others, get ready for a winner!</blockquote><p>The median tweet (probability = 0.97986):</p><blockquote>A MUST has spent records and positive process of Edwarding to President Obama for a failed policy in the tragic deal done by @SecretaryZinkens will be a great guy who spends and the field in the U.S. Military will be a disgrace to the American people and the victims of the Stoc</blockquote><p><strong>Word-level LSTM model: </strong>the proportion of tweets fooled our classifier is #. The most ‘Trumpy’ tweet (probability = 0.99996) is:</p><blockquote>he big story that the fake news media refuses to report me for @ cnn . my great honor to speak at the @ ussarizona . thank you ! # trump2016</blockquote><p>The 75th percentile ‘Trumpy’ tweet (probability = 0.97720):</p><blockquote>“ i have great respect for the people that represent the united us care . remember that the great president ? he is a nice guy !</blockquote><p>The median tweet (probability = 0.89935):</p><blockquote>. @ hillaryclinton is on the front page of the illinois . . . but write a . that ’ s because it should be an economic failure ‘ s to mexico . the problem ‘ 12 . you are both more .</blockquote><h4>Model comparison</h4><p>By reading the generated tweets, I think both the Markov Chain models and the LSTM models did a pretty good job at mimicking Trump’s tweet. Their performance of fooling the classifier confirmed my observation — at least 93% of their tweets fooled the classifier. Although by my human judgment, tweets generated by Markov Chain and LSTM are all pretty good, the LSTM generator seems to be better at fooling the classifier — a higher proportion of its tweets successfully fooled the classifier.</p><p>I think it might because that the Markov Chain models only consider the current token when predicting the next one, which tends to result in loss of context (in other words, the Markov Chain model is ‘memoryless’). On the other hand, LSTM models are particularly good at remembering information for long periods of time. This behavior may make LSTM models outperform the Markov Chain models on this task.</p><h3>Future plans</h3><ul><li>Polish the output sentences, e.g. spelling correction, uppercase when necessary, and punctuation correction.</li><li>Use Generative Adversarial Network (GAN) to boost the performance of both the text generator and the text classifier</li></ul><h3>Notes</h3><p>All the data and code for this package can be found on this <a href="https://github.com/SUN-Wenjun/Trump_Tweet_Generator">GitHub repo</a> and this <a href="https://github.com/SUN-Wenjun/Trump_Tweet_Generator/blob/master/trumpy_tweet.ipynb">notebook</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ccfddc7764e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/trumpy-tweets-trump-tweet-classifiers-and-generators-9ccfddc7764e">Trumpy Tweets — Trump Tweet Classifiers and Generators</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An R Wrapper for FBI Crime API]]></title>
            <link>https://medium.com/@wenjun.sarah.sun/an-r-wrapper-for-fbi-crime-api-a80f8586e0ff?source=rss-c8f8d68255b0------2</link>
            <guid isPermaLink="false">https://medium.com/p/a80f8586e0ff</guid>
            <category><![CDATA[api]]></category>
            <category><![CDATA[crime]]></category>
            <category><![CDATA[package]]></category>
            <category><![CDATA[wrapper]]></category>
            <category><![CDATA[r]]></category>
            <dc:creator><![CDATA[Wenjun]]></dc:creator>
            <pubDate>Wed, 11 Dec 2019 22:33:17 GMT</pubDate>
            <atom:updated>2019-12-13T03:19:33.555Z</atom:updated>
            <content:encoded><![CDATA[<p>This short article introduces an R package that enables users to access the most updated crime data from the FBI Crime API with ease.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*W7gzdqO84I2WCrL4UPgYQQ.png" /></figure><p><a href="https://crime-data-explorer.fr.cloud.gov/api">FBI Crime Data API</a> is an API maintained by the FBI. This API provides information about offenses, offenders, victims, and arrests in the US.</p><h3>What does this package do?</h3><p>Long story short. It helps you to download the crime data you want as a dataframe in R, so that you can have fun with these data.</p><p>There are five useful functions included in this package:</p><h4>A function for setting API Key as a global option</h4><p>A function to set FBI Crime API Key as a global option. This facilitates API calls without the need to specify the key every time. It is recommended to run this function first before running querying functions. However, alternatively, you can also pass your API key to the api_key parameter each time when you run a query.</p><pre>set_fbi_crime_api_key(&#39;theapikeyyougotfromtheirwebsite&#39;)</pre><h4>Four querying functions that get crime-related data</h4><p>Four functions for querying offense, offender, victim, and arrest data. The output of each function is a dataframe. Here are some examples.</p><pre>count_offense(offense = &#39;larceny&#39;, level = &#39;agencies&#39;, level_detail = &#39;MA0010100&#39;)</pre><pre>summarize_offender(offense = c(&#39;burglary&#39;,&#39;arson&#39;), level = &#39;regions&#39;, level_detail = c(&#39;Northeast&#39;,&#39;South&#39;), variable = &#39;sex&#39;)</pre><pre>summarize_victim(offense = &#39;motor-vehicle-theft&#39;, level = &#39;states&#39;, level_detail = &#39;DC&#39;, variable = &#39;sex&#39;)</pre><pre>summarize_arrest(by_offense_type = TRUE, offense = c(&#39;aggravated-assault&#39;,&#39;rape&#39;), variable = &#39;race&#39;, since = 2010, until = 2011)</pre><p>You can find more about what arguments to feed each parameter in the help document for each function (find them by run ?function_name in R).</p><h4>A function for validating arguments of the querying functions</h4><p>This function is called within each querying function, it’s for internal use only, so you shouldn’t be worried about this. It outputs an error and stops the querying function if any of the arguments are invalid.</p><h3>How to install it?</h3><p>You can follow the following code to install the package. You do need to have devtools package installed first in order to download packages from GitHub.</p><pre># install.packages(&quot;devtools&quot;)<br># library(devtools)<br><br>install_github(&quot;SUN-Wenjun/fbicrime&quot;)<br>library(fbicrime)</pre><h3>Highlights</h3><p>This section is about why this package is cool.</p><ul><li>Created a function called set_fbi_crime_api_key() that <strong>sets the input API key as a global option</strong> so that the user does not need to specify the API key every time. Alternatively, users can also use the api_key parameter in every querying function to input their key for one time use.</li><li><strong>Generalized queries of different endpoints with one function</strong>. This is specifically challenging for arrest<strong> </strong>data, because the construct of its endpoints and response vary a lot — depends on whether the end-user wants to specify one kind of offense or wants to have data about all offense.</li><li><strong>Allow users to give a list</strong> of offense and/or a list of states (or regions, depends on what level the user chooses to aggregate), and the functions return a data frame including all query results based on the given list of parameters.</li><li>The summarize_arrest() function can deal with two different categories of endpoints in API — one specifies offense type, the other does not. These two categories of endpoints have different structures, which add complexity to the function.</li><li><strong>Validate input arguments</strong>. I created a function called valid_arg_level , it produces an error when the input arguments are not valid. First, it outputs an error when the level argument is illegal. There are only four kinds of aggregate levels available for the endpoints — agencies, national, regions, and states, so if the input level argument is not one of them, an error will be produced. Second, it produces an error when level and level_detail are mismatched. In other words, if level is ‘national’ but level_detail is not null, or, if level is not ‘national’ but level_detail is missing, then the validate_arg_level() function will stop the querying and warn the end-user.</li><li><strong>Unit tests for querying functions</strong>. The testings help to check the dimension, missing values, and type of the output dataframe.</li><li><strong>Wrote documents </strong>for each function and for the package. Created a <strong>vignette</strong> for the package.</li></ul><h3>Github link</h3><p>All the code for this package can be found on<a href="https://github.com/SUN-Wenjun/fbicrime"> this GitHub repo</a>, star me if you find this helpful.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a80f8586e0ff" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>