<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Noah Muluberhan-Berhe on Medium]]></title>
        <description><![CDATA[Stories by Noah Muluberhan-Berhe on Medium]]></description>
        <link>https://medium.com/@noah.berhe?source=rss-36db8d98670a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*-V3a1KNiYJzZBaif</url>
            <title>Stories by Noah Muluberhan-Berhe on Medium</title>
            <link>https://medium.com/@noah.berhe?source=rss-36db8d98670a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 01:09:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@noah.berhe/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Text Classification using Word Similarity]]></title>
            <link>https://medium.com/data-science/text-classification-using-word-similarity-2f0c5fc9f365?source=rss-36db8d98670a------2</link>
            <guid isPermaLink="false">https://medium.com/p/2f0c5fc9f365</guid>
            <category><![CDATA[nltk]]></category>
            <category><![CDATA[banking]]></category>
            <category><![CDATA[text-classification]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Noah Muluberhan-Berhe]]></dc:creator>
            <pubDate>Mon, 12 Oct 2020 15:08:37 GMT</pubDate>
            <atom:updated>2020-10-12T18:17:35.020Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YbM1SSUqznl2Bi6D" /><figcaption>Photo by <a href="https://unsplash.com/@tjump?utm_source=medium&amp;utm_medium=referral">Nik Shuliahin</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>NLP / NLTK Wordnet / Text Classification</h4><h4>Using Data Science to spot the FCA’s vulnerable customers</h4><h3>Background</h3><p>One of the key areas that the UK financial services regulator, the FCA, has focused on in their <a href="https://www.fca.org.uk/publication/business-plans/business-plan-2020-21.pdf">2020/21 Business Plan</a> is the fair treatment of banking customers. Following initial consultations last summer, they have <a href="https://www.fca.org.uk/publications/guidance-consultations/gc20-3-guidance-firms-fair-treatment-vulnerable-customers">recently updated</a> the guidance to better clarify the drivers of vulnerability:</p><blockquote>We have identified 4 key drivers which may increase the risk of vulnerability. These are:<br>- <strong>health</strong> — disabilities or illnesses that affect the ability to carry out day-to-day tasks.<br>- <strong>life events </strong>— major life events such as bereavement, job loss or relationship breakdown<br>- <strong>resilience</strong> — low ability to withstand financial or emotional shocks<br>- <strong>capability</strong> — low knowledge of financial matters or low confidence in managing money (financial capability) and low capability in other relevant areas such as literacy, or digital skills</blockquote><p>With COVID-related illness and job loss creating fertile grounds for these drivers to flourish, I wanted to look at how a Data Scientist could help a bank identify such vulnerability so they can more proactively offer services of support.</p><p>I imagined a transcript of a phone conversation between customer and bank, or a typed conversation with a chatbot or customer service agent. And looked to NLP techniques to help automatically detect these drivers.</p><p><strong>Potential Strategies:</strong><br><strong>1. Word similarity:</strong> scanning the passage of text for keywords (e.g. furlough) or their synonyms.<br><strong>2. Bag-of-Words:</strong> derive n-gram features from labelled examples, and use that model to classify future text.</p><p>Given the lack of training data, we’re going with Strategy 1.</p><h3>Implementation</h3><h4>1. Normalize vulnerability drivers</h4><p>First we unpack the 4 drivers of vulnerability into a longer list of distinct categories that we can then explicitly search for in the text:<br>1. Death,<br>2. Redundancy,<br>3. Illness,<br>etc…</p><h4>2. Translating categories for NLTK Wordnet</h4><p>Since we are going to be using similarity scorer available in NLTK we will need to translate these categories into the correct definition as described in Wordnet (NLTK’s lexical database/dictionary). For this we can use a simple function that gives a list of synonyms for that word and their definition:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/960058c90027a3b2683f6bfc2ad84de1/href">https://medium.com/media/960058c90027a3b2683f6bfc2ad84de1/href</a></iframe><p>which for <strong>word_scorer(‘redundancy’)</strong> would output:</p><pre>[(&#39;redundancy.n.01&#39;,<br>  &#39;repetition of messages to reduce the probability of errors in transmission&#39;),<br> (&#39;redundancy.n.02&#39;, &#39;the attribute of being superfluous and unneeded&#39;),<br> (&#39;redundancy.n.03&#39;,<br>  &#39;(electronics) a system design that duplicates components to provide alternatives in case one component fails&#39;),<br> (&#39;redundancy.n.04&#39;, &#39;repetition of an act needlessly&#39;)]</pre><p>and would therefore help us create the topic dictionary we will search against (where the keys here are the user-friendly topic, and the values are the Wordnet-friendly representation):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8ae829d8adc65a1c68b8bd81d5256677/href">https://medium.com/media/8ae829d8adc65a1c68b8bd81d5256677/href</a></iframe><h4>3. Tokenize passage</h4><p>Here’s a passage we may want to analyse (clearly about redundancy):</p><blockquote>I worked in Adult Learning for a county council and my job was paid for by government funding. When the funds ran out, halfway through the financial year, there was suddenly no money to pay my wages. I was called to a meeting with four others (whose jobs were also dependant on the funding) and the news was broken to us. There would be a 12 week consultation period before the redundancy became final. We could choose to take the redundancy payment and go, or we could take another job in a different part of the council. If we refused the job offered to us, we could lose the redundancy settlement.</blockquote><p>which we will tokenize, such that we can iterate through a list of word tokens</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/de0f67bdbcbf2e033b2d26ded89ac815/href">https://medium.com/media/de0f67bdbcbf2e033b2d26ded89ac815/href</a></iframe><h4>4. Score passage against a single topic</h4><p>We are going to implement a function that will follow these steps:</p><ul><li>Iterate over the passage’s word tokens, feed each into wordnet.synsets() to list all the possible usages of the word (e.g. see the possible usages of the word ‘redundancy’ listed above)</li><li>Iterate through that word’s synsets and compare them for high similarity with the topic of interest (we will be using the <a href="https://www.geeksforgeeks.org/nlp-wupalmer-wordnet-similarity/">wup_similarity</a>).</li><li>If any of that word’s possible usages has a high similarity with the topic (greater than a similarity threshold we define), we say that topic has been mentioned, then we move onto the next word.</li><li>We can also add a return_hits option, which will enable us to report which words triggered the similarity threshold.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/845adc5e9a4e4e3a564c8e90ffe39c73/href">https://medium.com/media/845adc5e9a4e4e3a564c8e90ffe39c73/href</a></iframe><h4>5. Score passage against the whole topic dictionary</h4><p>We can then reuse our function to iterate through each topic in our topic dictionary:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e53ddea7d5466deb3fde61b472b4ce6c/href">https://medium.com/media/e53ddea7d5466deb3fde61b472b4ce6c/href</a></iframe><p>and therefore compute that passage’s similarity with each topic and return the words that were deemed similar.</p><p>After packaging the functions in a module I called synonym.py, running this final function was simple:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6d20ff97af3a377cbd7db366a12a34f5/href">https://medium.com/media/6d20ff97af3a377cbd7db366a12a34f5/href</a></iframe><p>which outputs the below dictionary of results, where we can see the topic of redundancy has been mentioned 3 times in the passage, all times due to the mention of the word ‘redundancy’ although it could have been any other similar word or similar that could have added to the score had that been used instead:</p><pre><br>{&#39;disability&#39;: (0, []),<br> &#39;death&#39;: (0, []),<br> &#39;health problems&#39;: (0, []),<br> &#39;being a carer&#39;: (0, []),<br> &#39;living alone&#39;: (0, []),<br> &#39;job loss (fired)&#39;: (0, []),<br> &#39;job loss (redundancy)&#39;: (3, [&#39;redundancy&#39;, &#39;redundancy&#39;, &#39;redundancy&#39;]),<br> &#39;job loss (furlough)&#39;: (0, [])}</pre><h4>6. Checking results</h4><p>We can also use our previously mentioned word_scorer to compare the words listed above (in this case it’s just ‘redundancy’) with the topic’s wordnet definition (ie. ‘redundancy.n.02’) to see which usage of the word ‘redundancy’ was deemed to be similar</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3ac56690c49cc18fe58f4ec24d2c0029/href">https://medium.com/media/3ac56690c49cc18fe58f4ec24d2c0029/href</a></iframe><pre>[(&#39;redundancy.n.01&#39;,<br>  &#39;repetition of messages to reduce the probability of errors in transmission&#39;,<br>  0.25),<br> (&#39;redundancy.n.02&#39;, &#39;the attribute of being superfluous and unneeded&#39;, 1.0),<br> (&#39;redundancy.n.03&#39;,<br>  &#39;(electronics) a system design that duplicates components to provide alternatives in case one component fails&#39;,<br>  0.2222222222222222),<br> (&#39;redundancy.n.04&#39;, &#39;repetition of an act needlessly&#39;, 0.2222222222222222)]</pre><p>The similarity scores returned point to the 2nd entry (itself) so in this case it’s a redundant test, but in cases where a word is listed as similar which isn’t expected, running this test can uncover which usage of that word exactly was deemed to be similar.</p><h3>Conclusion</h3><ul><li>We have created a simple script which is able to classify a piece of text against pre-defined topics of interest (in this case: banks’ vulnerable customers).</li><li>This approach requires no labelled training data as we’re calculating similarity scores between precomputed word vectors.</li><li>This approach is also able to perform a multi label text classification, where a passage of text (and even each word in that text) can potentially be assigned to more than one topic if it’s similar enough. This is important in identifying customers with multiple vulnerabilities.</li></ul><p>All in all: a simple, elegant, flexible solution!</p><p>The GitHub repo is included below, any feedback is appreciated.</p><p><a href="https://github.com/noahberhe/Vulnerable-Customers">noahberhe/Vulnerable-Customers</a></p><p>… Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2f0c5fc9f365" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/text-classification-using-word-similarity-2f0c5fc9f365">Text Classification using Word Similarity</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[COVID-19 Sentiment Analysis Series, Part 3: Winners and Losers]]></title>
            <link>https://medium.com/data-science/covid-19-sentiment-analysis-series-part-3-winners-and-losers-c00f384080a8?source=rss-36db8d98670a------2</link>
            <guid isPermaLink="false">https://medium.com/p/c00f384080a8</guid>
            <category><![CDATA[twitter-sentiment]]></category>
            <category><![CDATA[matplotlib]]></category>
            <category><![CDATA[bayesian-statistics]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Noah Muluberhan-Berhe]]></dc:creator>
            <pubDate>Sat, 27 Jun 2020 09:45:39 GMT</pubDate>
            <atom:updated>2021-09-06T12:14:43.276Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*uA5M2cyBfk4M5hTn" /><figcaption>Photo by <a href="https://unsplash.com/@peterampazzo?utm_source=medium&amp;utm_medium=referral">Pietro Rampazzo</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Twitter Sentiment Analysis / Bayesian Inference</h4><h3>COVID-19 Sentiment Analysis Series, Part 3: Bayesian Sentiment Analysis</h3><h4>Understanding Londoners in lockdown</h4><p><strong><em>Note from the editors:</em></strong><em> </em><a href="http://towardsdatascience.com/"><em>Towards Data Science</em></a><em> is a Medium publication primarily based on the study of data science and machine learning. We are not health professionals or epidemiologists, and the opinions of this article should not be interpreted as professional advice. To learn more about the coronavirus pandemic, you can click </em><a href="https://www.who.int/emergencies/diseases/novel-coronavirus-2019/situation-reports"><em>here</em></a><em>.</em></p><p>In this series, we have been using Londoners’ tweets to understand how emotions evolved in the COVID-19 lockdown, and which public figures and businesses were the winners and losers in their eyes.</p><p>In <a href="https://towardsdatascience.com/covid-19-sentiment-analysis-series-part-1-getting-started-70892122630c">Part 1: “Getting Started”</a> we got our Twitter data and processed it, using entity recognition and sentiment scoring.</p><p>In <a href="https://towardsdatascience.com/covid-19-sentiment-analysis-series-part-2-hypothesis-testing-3db511875165">Part 2: “Hypothesis Testing”</a>, using various different types of hypothesis tests we explained why sentiment towards the NHS was so much more positive than towards the Government.</p><p>Today in the 3rd (and final) part, we will be casting the net wider to find more winners and losers from the lockdown:</p><ul><li><strong>The Simple Method:</strong> We will start with the simple yet cumbersome approach: where we repeat the trick from Part 2 using wordclouds and hypothesis testing to highlight and then make inferences about who were the winners and losers.</li><li><strong>Frequentist Modelling:</strong> We will then look at how to streamline this process by using a Logistic Regression model to give a holistic view.</li><li><strong>Bayesian Updating:</strong> Finally, I will explain <em>how</em> and, importantly, <em>why</em> a Bayesian approach could help us follow the evolving mood of a city, and we will visualise the results:</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FQTsBKl7jWOylhZVNd9%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2FQTsBKl7jWOylhZVNd9&amp;image=https%3A%2F%2Fmedia1.giphy.com%2Fmedia%2FQTsBKl7jWOylhZVNd9%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="335" frameborder="0" scrolling="no"><a href="https://medium.com/media/47717253d86dfa879ac92abc5b6efa62/href">https://medium.com/media/47717253d86dfa879ac92abc5b6efa62/href</a></iframe><h3>The Simple Method</h3><p>Picking up where we left off in <a href="https://towardsdatascience.com/covid-19-sentiment-analysis-series-part-2-hypothesis-testing-3db511875165">Part 2: “Hypothesis Testing”</a>, let’s start by visualising the most frequently mentioned organisations in our dataset of tweets. We have also narrowed our search just on those tweets that were scored to be either positive or negative, as we’re interested in finding winners and losers:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*cjhUqcRczUIKGM0toqNlzQ.png" /></figure><p>We can see Amazon and the Treasury very clearly come to the fore as the most frequently mentioned organisations. However it’s difficult to see beyond these two as the wordcloud is cluttered with capitalised words that spaCy mistakenly labelled as entities.</p><p>Nonetheless we can still look at Amazon and make the statement:</p><blockquote><em>Due to Londoners switching to online shopping during the lockdown, Amazon will be a big winner</em>.</blockquote><p>It’s not enough to conduct a 1-sample test to show that Amazon’s sentiment score was positive, we will need to show Amazon’s score was more positive vs. a sensible baseline. For this, we will conduct two <em>2-sample means</em> t-tests:</p><p><strong>Hypothesis 1: </strong>Amazon sentiment improved on a pre-lockdown baseline.</p><p><strong>Hypothesis 2: </strong>Amazon sentiment in lockdown was higher than the baseline average across all entities.</p><h3>Hypothesis 1: Amazon vs. Amazon</h3><p>We can create the 2 distributions: <em>amazon_prelockdown</em> and <em>amazon_postlockdown</em> very easily, and run SciPy’s t-test:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9630003fce288238c43cdfdd4f3381fa/href">https://medium.com/media/9630003fce288238c43cdfdd4f3381fa/href</a></iframe><pre>Ttest_indResult(statistic=0.10878616367025529, pvalue=0.9134795533240211)</pre><p>which shows us that any difference in the overall mean isn’t statistically significant. Why?</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2dd8e68916b8fc33a661a13be88a909d/href">https://medium.com/media/2dd8e68916b8fc33a661a13be88a909d/href</a></iframe><pre>Number of pre_lockdown Amazon tweets = 3<br>Number of post_lockdown Amazon tweets = 201</pre><p>Because our data only has 3 pre-lockdown tweets mentioning Amazon, leading to a standard error too high to be able to reject the null.</p><h3>Hypothesis 2: Amazon vs the Rest</h3><p>Similarly creating the relevant distributions, and running the same t-test:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/075eb77327f6a0e5573d2c3aa371c69c/href">https://medium.com/media/075eb77327f6a0e5573d2c3aa371c69c/href</a></iframe><pre>Ttest_indResult(statistic=2.469890113757005, pvalue=0.01351783280956628)</pre><p>We have a low p-value which would lead us to reject the null, however let’s check the distributions and the sample sizes to check that the assumption of normality holds, for us to have performed this parametric statistical test:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*CM0z27Ha1fECo5qS33mlvQ.png" /><figcaption>Amazon vs the Rest: distributions</figcaption></figure><pre>Amazon sample size 243<br>Other entities sample size 94912</pre><p>We can see 2 issues here:</p><ol><li>The Amazon distribution looks curious in that it has two peaks located at ~0.2 and ~0.6 along the x-axis, respectively. Running some quick EDA, we can see that the peak at 0.6 needs to be corrected, as it contains many non-independent tweets from a viral campaign:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*e26KsFV0yA8Zy7D-HYkeyA.png" /></figure><p><strong>Lesson:</strong> make sure to remove all duplicate texts from now on.</p><p>2. The sample size for ‘Other Entities’ is much too large to represent the baseline I was hoping it would represent, ie. only for other organisations. It appears that SpaCy incorrectly tagging capitalised words as entities has led to this distribution becoming polluted. Therefore allowing this to more closely represent the broader sample set. I manually created a good subset of the most-mentioned organisations that could make up the ‘Other Entities’:</p><p><em>WHO, Facebook, FDA, UN, Whatsapp, Tesco, Premier League, Google, FDA, Apple, CDC, Parliament, White House</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*slfPVtNj98evfUIynoaGOA.png" /><figcaption>Amazon vs the Rest: distributions (cleaned)</figcaption></figure><pre>Amazon sample size 183<br>Other entities sample size 520</pre><p>And rerunning our t-test:</p><pre>Ttest_indResult(statistic=1.1270992972809608, pvalue=0.26018755847340846)</pre><p>We now fail to reject the null because:</p><ul><li>removing the duplicates saw the removal of the 35 tweets and a reduction in Amazon’s mean score from 0.15 to 0.11</li><li>although standard deviation of ‘Other Entities’ was little changed, the large drop in sample size from restricting on the most-mentioned organisations saw a drop in standard error from 0.0019 to 0.02, therefore reducing the statistical power of the test.</li></ul><p>In essence, if we wanted to be sure that the difference between Amazon’s positive mean and the Others’ neutral mean was statistically significant, we would need to collect more samples to increase the power of our test to reject the null. We can measure this using a Power Analysis…</p><h4>Power Analysis</h4><p>How many tweets would we have needed in order to conclude with 95% confidence that the two sample means were indeed different?</p><p>Put another way: how large a sample size would have been needed to reduce the Type II error rate to below a certain threshold. We will choose a threshold of 20%, which translates to a Power (the ability to correctly reject the null) of 80%.</p><p>We can conduct a <a href="https://machinelearningmastery.com/statistical-power-and-power-analysis-in-python/">Power Analysis</a> (the details of which we won’t go into here) but the code is quite simple and utilises Statsmodel’s TTestIndPower function:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7301a62f760f528abca26bc3c3263dfe/href">https://medium.com/media/7301a62f760f528abca26bc3c3263dfe/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*w7c9PWyTnnPj6hrw18L5Vg.png" /><figcaption>Power Analysis for an Effect Size = 0.13</figcaption></figure><p>which shows that a very large sample size of almost 1000 tweets would have been needed for our test to correctly reject the null 80% of the time.</p><p>So, even a commercial entity that had a relatively large number of tweets (Amazon) tested vs. ALL the other most tweeted about commercial entities still doesn’t give us enough statistical power to differentiate between their means.</p><p>However we know from a 1-sample test that we’re confident that sentiment towards Amazon was positive:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/238b308f5a4533049d3e80624e5447f0/href">https://medium.com/media/238b308f5a4533049d3e80624e5447f0/href</a></iframe><pre>Ttest_1sampResult(statistic=4.298040028883434, pvalue=2.616546144158821e-05)</pre><p>So 2 sample tests don’t have enough statistical power, whereas looking at each entity alone can.</p><p>How do we then present this 1-sample view across all entities to see which ones were indeed the winners / losers, without having to re-run a t-test for each entity individually?</p><h3>Frequentist Modelling</h3><p>To do this we can reframe the question into a supervised learning problem. We can apply Statsmodel’s Logistic Regression classifier predicting for one of two label classes <em>positive</em> or <em>negative,</em> with the features being the organisations that were recognised by SpaCy.</p><p>I used the scikit-learn’s <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html">TfidfVectorizer</a> to encode the entities into a features array with weightings derived using it’s TF-IDF methodology. I also used this function to remove stopwords and other unwanted words, and restricted this to the Top 70 features. The results returned were:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e940610e642e2d48285212458987037e/href">https://medium.com/media/e940610e642e2d48285212458987037e/href</a></iframe><p>The overall model, particularly when restricted to just the 70 top features, couldn’t predict if a tweet would be positive or negative (looking at the R²). However we can learn a lot at the individual feature level: we can see which were net positive (because coef &gt; 0) and which were net negative (coef &lt; 0).</p><p>The model results also show us the certainty of our sample statistic (the coefficient) using: the standard-error for that distribution (influenced by sample size), the z-score and the p-value which both show whether this sample statistic was statistically significant, and then the 95% confidence interval for the sample statistic.</p><h4>Winners:</h4><ul><li><strong>Tech companies.</strong> A positive coefficient and a p-value close to 0 leaves us in no doubt that companies like Amazon, Apple, Whatsapp, Instagram were perceived positively by tweeters.</li><li><strong>stayhomesavelives</strong>, the UK Government’s slogan, also had a very positive coefficient. Along these lines self isolation was another positive story, and social distancing. The public took on board these efforts positively.</li></ul><h4>Losers:</h4><ul><li><strong>Dominic Cummings.</strong> The polarising figure had a very negative coefficient. This deteriorated significantly around the time of the controversy surrounding his decision to break social-distancing rules.</li><li><strong>Keir Starmer</strong> surprisingly has a negative coefficient, could this be due to criticisms of the government approach? No, looking at the data it was actually due to people’s criticism of him not holding the government to account strongly enough.</li><li><strong>PPE</strong> was also negatively portrayed, owing to tweets pointing out the shortages seen in NHS hospitals.</li></ul><h4>Other notes:</h4><ul><li>Words describing government figures and departments generally had a high p-value and wide confidence intervals crossing the 0 threshold, indicating mixed views from the public on their performance.</li></ul><h4>Conclusion:</h4><p>We have used Frequentist statistics to create a top-down approach to finding the organisational drivers of good and bad sentiment, however this relies on the 3 months of data we have already collected.</p><p>At the start of the observation period when the situation was evolving fast, we saw large swings in sentiment towards an entity coupled with lower sample sizes leading to much more uncertainty. This is where a Bayesian model can help…</p><h3>Frequentist to Bayesian</h3><p>In the Frequentist view of the world, we have a broader population that, through sampling, we’re trying to understand. As a result there is one true population parameter which by sampling we can estimate through our sample statistic.</p><ul><li>With this approach we are still attempting to come to one single number for our estimate.</li><li>With this approach we are also allowing our view to be informed only by our observation period, and not by any prior knowledge.</li></ul><p>In the Bayesian view of the world, we don’t think of there being a broader population. We only have our sample. However we do concede that, because our sample size is finite, there is uncertainty in our sample statistic so we should model this as a <strong><em>distribution</em></strong> not as a single number.</p><ul><li>With this approach we take a prior assumption, and update this with new observations using Bayes’ Theorem, to come to a current view of the world.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/323/1*oNEKQxEhdQY3r-VoIuwWkA.png" /><figcaption>Bayes’ Theorem</figcaption></figure><p>where:</p><ul><li><em>P(p) </em>= prior assumption</li><li><em>P(data|p) </em>= new observations (as in Frequentist approach), also called the likelihood function</li><li><em>P(p|data)</em> = latest (or: ‘posterior’) assumption, after we’ve updated the prior with our latest observations.</li></ul><p>Given the ‘live’ nature of most data collection, particularly after a unique event such as COVID-19, it makes sense to look at how a Bayesian approach can help us answer the question.</p><h3>Bayesian Modelling</h3><p>Let’s first outline the approach:</p><ol><li>We have 2 options for modelling the elements of Bayes’ Theorem (above):</li></ol><ul><li><strong>Conjugacy</strong> (simpler and preferred if possible): The posterior will be the same type of distribution as the prior if the prior is a conjugate prior to the likelihood function. This has the benefit of making Bayes’ Theorem algebraically solvable, with the parameters for the posterior (unknown) expressed in terms of the parameters of the prior and likelihood (known).</li><li><strong>Sampling techniques (</strong>e.g. Markov Chain Monte Carlo): There are a finite number of conjugate priors, if our data doesn’t fit any of these, then we’d need to use sampling techniques.</li></ul><p>2. Luckily as we’re only interested in a binary outcome (our tweets will be either positive or negative, as we will omit neutral tweets), we can achieve conjugacy using perhaps the simplest conjugate pair: <strong><em>the beta-binomial pair</em></strong></p><ul><li><strong>Likelihood function:</strong> sampling a stream of tweets and tagging them as success (positive) or failure (negative) is a binomial distribution:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/190/1*JYZ6SyXdkpfq_CHOvsiXAQ.png" /><figcaption>Binomial distribution</figcaption></figure><ul><li><strong>Prior:</strong> the beta distribution describes the range of probabilities for two binary outcomes:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/260/1*m5BswPm0FPUEI7P8bpJBIA.png" /><figcaption>Beta distribution</figcaption></figure><ul><li><strong>Posterior:</strong> … to which we can add the <em>n</em> trials and <em>k</em> successes to derive the posterior:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/737/1*Aa_bJ668IIGM2PuQLTIalg.png" /><figcaption>Posterior Beta distribution</figcaption></figure><p>A long and complicated equation, but the important bits of intuition to take away from this are:</p><ul><li>The Binomial distribution can be thought of as merely the discrete version of the continuous Beta distribution. The Bayesian updating we proved above mimics what is really happening: as the new data comes in this is represented as discrete data points by the Binomial likelihood function, which go into updating the smooth prior distribution.</li><li>α and β can now be expressed very simply in terms of the prior and the likelihood:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/275/1*W_P07c0XjIR5Z28rsA-W5g.png" /></figure><h4>Application</h4><p>We can use the output of the Logistic Regression to give us a list of the most-tweeted entities we can look at.</p><p>The process for building the Bayesian Modelling pipeline for a specific entity can be divided into 2 stages:</p><ol><li>The Prior:</li></ol><ul><li>Count how many positive vs negative tweets there were in the period prior to the lockdown:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/446581fd8d93061154cabcaaa4dd1ae5/href">https://medium.com/media/446581fd8d93061154cabcaaa4dd1ae5/href</a></iframe><ul><li>Use this data to calculate the α and β for the prior’s beta distribution:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/101180faf97c2c527b5dc0ed6a5ec333/href">https://medium.com/media/101180faf97c2c527b5dc0ed6a5ec333/href</a></iframe><p>2. The Posterior:</p><ul><li>Record the evolution of the α and β parameters as new data comes in using the formulae:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/275/1*W_P07c0XjIR5Z28rsA-W5g.png" /></figure><p>I also created an animation in Matplotlib that can show us this evolution:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/476dc6b9e57e94ea867966b1e342d60f/href">https://medium.com/media/476dc6b9e57e94ea867966b1e342d60f/href</a></iframe><p>Let’s look at the evolution in sentiment for a few entities of interest:</p><p><strong>Amazon</strong></p><p>There were 183 tweets referring to Amazon in this period, the orange distribution shows the probability distribution for sentiment before the lockdown, ie. the 2 weeks leading up to it. This is our prior belief of the probability distribution for Amazon sentiment.</p><p>The distribution in this case is quite flat and shallow, reflecting the fact there were relatively few tweets informing our prior belief: only 2 positive tweets and 1 negative tweet.</p><p>Once we hit lockdown, that’s when we update the prior with tweets gathered from lockdown-period tweets. This enables us then to model the Posterior (in blue). Notice how the distribution narrows and peaks as days pass (denoted by the slider) as we harvest more tweets to inform our posterior distribution.</p><ul><li>Amazon’s prior was in neutral territory: 0.5. But jumped into positive territory almost immediately after the lockdown, and stayed there.</li><li>The final Maximum a Posteriori (MAP) probability estimate, ie. where the posterior peaks, of 0.6 shows Amazon’s sentiment was more likely to be positive (denoted by a score of 1.0 on the x-axis) than negative (0 on the x-axis) based on the balance of data we found to update our prior belief.</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FWPzaJskE4dHr4xrnj9%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2FWPzaJskE4dHr4xrnj9&amp;image=https%3A%2F%2Fmedia1.giphy.com%2Fmedia%2FWPzaJskE4dHr4xrnj9%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="335" frameborder="0" scrolling="no"><a href="https://medium.com/media/0ca07da0d1adc2f6354f6e2466b7ed24/href">https://medium.com/media/0ca07da0d1adc2f6354f6e2466b7ed24/href</a></iframe><p><strong>Keir Starmer</strong></p><p>Prior to the lockdown, I had no tweets about Keir Starmer to inform any prior belief about him. This is where you can set α and β both to 0.5 therefore adopting Jeffreys’ Prior to model an uninformed distribution. The prior takes on a U-shape.</p><p>Over the lockdown period we collected 66 tweets, and throughout the entire period Starmer’s MAP remained in negative period, hovering around 0.2.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FQTsBKl7jWOylhZVNd9%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2FQTsBKl7jWOylhZVNd9&amp;image=https%3A%2F%2Fmedia1.giphy.com%2Fmedia%2FQTsBKl7jWOylhZVNd9%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="335" frameborder="0" scrolling="no"><a href="https://medium.com/media/674dff1737977754e1ebfcef8c3715eb/href">https://medium.com/media/674dff1737977754e1ebfcef8c3715eb/href</a></iframe><p><strong>Dominic Cummings</strong></p><p>We collected 1003 tweets for Dominic Cummings, who started off before lockdown with nothing but positive tweets in my dataset.</p><p>After about 20 days into the lockdown period, the posterior distribution had shifted towards the left, with the MAP ending at just under 0.5.</p><p>This shift from positive to negative sentiment, from looking at the actual dates, coincided with the controversy around his breaking of social distancing rules.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FU4SjJLttUsHwZTFCT3%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2FU4SjJLttUsHwZTFCT3&amp;image=https%3A%2F%2Fmedia2.giphy.com%2Fmedia%2FU4SjJLttUsHwZTFCT3%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="335" frameborder="0" scrolling="no"><a href="https://medium.com/media/3b8b2cc52ef6b13bb8e1c0ad605c34d7/href">https://medium.com/media/3b8b2cc52ef6b13bb8e1c0ad605c34d7/href</a></iframe><h3>Conclusions / Final Thoughts</h3><p>In this blog series: we showed that it’s possible to set up a feed in from Twitter, apply some text processing, conduct sentiment scoring, and analyse the per-entity sentiment in a way that clarifies the winners and losers.</p><h4><strong>Hypothesis Testing:</strong></h4><p>I used this extensively to understand bring some statistical clarity to my findings, which taught me a few areas to be cautious about:</p><ul><li><strong>p-values</strong>: can be easily made to be statistically significant if the sample size if large enough to ensure a low enough standard error. Ask yourself if your test is specific enough to ask a pointed question? Ask yourself if your filtering is set up correctly, ensuring your sample isn’t inflated.</li><li><strong>normality assumption</strong>: this generally held in most cases, thanks to the Central Limit Theorem, but we can always use non-parametric versions of our hypothesis tests like Mann-Whitney or Wilcoxon Signed-Rank tests.</li><li><strong>mean reversion / neutralisation</strong>: for any slice of the data that’s large enough or not specific enough, sentiment would generally cluster around 0 and not be meaningful. View the distribution of your subsample, and if it has a high peak at 0 sentiment then likely you need to be more specific with filtering for a meaningful hypothesis test.</li></ul><h4>Holistic Winners/Losers:</h4><p>To get an overall view, I used Logistic Regression to model positive vs negative labelled tweets, using the vectorized entity column as the predictor matrix.</p><p>This gave some interesting results:</p><ul><li>The overall R² was low, suggesting I needed more features than the few I included to be able to model accurately whether a tweet would be positive or negative.</li><li>However, this wasn’t what I wanted… I was interested in how those features I did include contribute to positivity or negativity, which I could get from their coefficients. The nice things about using the statsmodel’s package vs scikit-learn’s was being able to see the statistical confidence of each coefficient.</li><li>Because we actually had a constantly evolving view of sentiment, we then turned to a Bayesian approach…</li></ul><h4>Bayesian Winners/Losers:</h4><p>Here I set up an animated slider to show the prior distribution, and then how the posterior distribution evolved after that as new data came in, which could be configured to the entity of choice.</p><ul><li>This showed clearly how and when <strong>Cummings</strong> and <strong>Starmer</strong> sentiment started to deteriorate, giving some good explanation. Had we been running live, our model would have given us early clues for this deterioration which we would have been able to track tweet-by-tweet.</li><li>It also clearly showed certainty of our posterior beliefs, by attaching a probability distribution to the sentiment at each stage.</li><li>The Beta-Binomial model was used which simplified the parameters greatly, a good way to view this conjugacy is that the binomial distribution is just the discrete version of the beta distribution, and therefore simply reflects the addition of individual new data-points to the continuous prior. This is solved mathematically as the update of the beta parameter by addition of the binomial parameter.</li></ul><p>To those who have made it this far, thanks for reading and of course all feedback is welcome and appreciated, you can find the repo here:</p><p><a href="https://github.com/noahberhe/COVID-19">noahberhe/COVID-19</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c00f384080a8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/covid-19-sentiment-analysis-series-part-3-winners-and-losers-c00f384080a8">COVID-19 Sentiment Analysis Series, Part 3: Winners and Losers</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[COVID-19 Sentiment Analysis Series, Part 2: Hypothesis Testing]]></title>
            <link>https://medium.com/data-science/covid-19-sentiment-analysis-series-part-2-hypothesis-testing-3db511875165?source=rss-36db8d98670a------2</link>
            <guid isPermaLink="false">https://medium.com/p/3db511875165</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[mathematics]]></category>
            <dc:creator><![CDATA[Noah Muluberhan-Berhe]]></dc:creator>
            <pubDate>Tue, 09 Jun 2020 22:30:20 GMT</pubDate>
            <atom:updated>2020-06-27T09:47:35.414Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1CCq7wZ1hhWYi8VT" /><figcaption>Photo by <a href="https://unsplash.com/@sctgrhm?utm_source=medium&amp;utm_medium=referral">Scott Graham</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Twitter Sentiment Analysis / Hypothesis Testing</h4><h4>Understanding Londoners in lockdown</h4><p>In this series, we will be using Londoners’ tweets to understand how emotions evolved in the COVID-19 lockdown, and which public figures and businesses were the winners and losers in their eyes.</p><p>In <a href="https://towardsdatascience.com/covid-19-sentiment-analysis-series-part-1-getting-started-70892122630c">Part 1: “Getting Started”</a> we got our Twitter data and processed it, using entity recognition and sentiment scoring.</p><p>Today in Part 2, we will look at the data and through hypothesis testing verify that <strong><em>statistically significant</em></strong> differences exist in the sentiment scores for different entities. But first…</p><h3>Laying the Groundwork</h3><p>Underpinning hypothesis testing is the following thought process:</p><ol><li><strong>Normal Distribution</strong>: We concede that we can’t possibly capture all of the tweets sent in this period, so the data points we do have are <em>samples</em> taken from a broader population. If this sampling is random you can expect your measured observation (in this case we are measuring sentiment) to be normally distributed due to the <a href="https://towardsdatascience.com/central-limit-theorem-explained-with-examples-4c10377ee58c">Central Limit Theorem </a><em>unless the sample size is &lt; ~30.</em> For example, if we had a sample mean of 5 the distribution is assumed to be:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*11q7emfeJg3ICgM4sI6WUw.png" /><figcaption>Normal distribution, with sample mean = 5</figcaption></figure><p><strong>2. Setting up the test</strong>: To test if this sample mean is different to another sample mean (let’s say 2, but it can be any number), we would calculate how many <em>standard deviations</em> (or, in sampling statistics, <em>standard errors</em>) would separate the two if they were part of the same distribution. That is to say, we would hypothesise they were not derived from different distributions of sentiment but from the same distribution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/454/1*fJlbIaK5z83vSlqRbADoQQ.png" /></figure><p><strong>3. Standardising the Norm</strong>: Visualising this operation, what we are doing is shifting the normal distribution by -2 along the x-axis and dividing by the standard error. It becomes a Standard Normal distribution centred on 2, and the equation can be rewritten:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/331/1*LAnJSE1TUQ8-AhiUCEi4Hw.png" /><figcaption>I’ve also generalised the terms to highlight the hypothesis test being conducted</figcaption></figure><ul><li>the <em>z-score</em> quantifies how many standard errors the <em>Alternate Hypothesis </em>lies away from the new mean (in this case, this is the <em>Null Hypothesis</em>). This is powerful because the p-value for any z-score on a standard normal distribution is <a href="https://en.wikipedia.org/wiki/Standard_normal_table#Table_examples">known</a>. So we’re able to describe the likelihood that the <em>Alternate</em> would have been observed from the same population distribution as the <em>Null.</em> We would then be able to reject or accept our null hypothesis that the alternate was drawn the same distribution as the null by comparing this p-value with our significance level (we’ll be sticking with a 0.05 significance level).</li></ul><p>3. <strong>Student’s t-distribution</strong>. In the following tests, we will be calling the z-score a <em>t-statistic.</em> This is because we’ll be assuming a Student’s t-distribution, which is fatter-tailed than the standard normal, thereby introducing some conservatism against a <a href="https://en.wikipedia.org/wiki/Type_I_and_type_II_errors">Type I error</a>, due to having a finite number of observations:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/1*MX9vdG6UQ3AqCbQgEP9-Bw.png" /></figure><ul><li>the p-value for this distribution is also <a href="https://en.wikipedia.org/wiki/Student%27s_t-distribution#Table_of_selected_values">known</a>.</li></ul><p>Let’s proceed with our tests…</p><h3>Hypothesis 1: Government vs NHS</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HJidcIouP5Ijz4lk" /><figcaption>Photo by <a href="https://unsplash.com/@groovelanddesigns?utm_source=medium&amp;utm_medium=referral">Grooveland Designs</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>We start by looking at two entities that will be well represented in the data: the Government and the NHS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*FqYLE7z9ixa3UaJzDmId-Q.png" /><figcaption>Histogram: Sentiment</figcaption></figure><p>Sentiment for the NHS appears to be more positive with the distribution more heavily-weighted to the right, which we can confirm by looking at the means:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/535c082d0e1bad12c15a39a67329ff5d/href">https://medium.com/media/535c082d0e1bad12c15a39a67329ff5d/href</a></iframe><pre>Avg Gov sentiment -0.036<br>Avg NHS sentiment 0.171</pre><p>You may be wondering why there’s a 2nd spike in the NHS histogram at ~0.5. I thought these were repeated tweets, however, found they were scored the same because they all used the phrase “THANK YOU”, e.g….</p><blockquote>THANK YOU #ClapForCarers #ClapForNHS #NHS #KeyWorkers #Lockdown #DisneylandParis #DisneyWorld #Disney #TomorrowWillBeAGoodDay #ColonelTomMoore #Coronavirus</blockquote><blockquote>A BIG THANK YOU TO hollandandbarret &amp; @ray_electronica FOR TODAY’S EASTER EGG DONATION @ST GEORGES HOSPITAL #NHS #NHSWorkers #LondonNHS #FrontlineNHS #MedicalTeam #StayAtHome #coronavirus…</blockquote><blockquote>THANK YOU NHS biglovenhs #thankyounhs #clappingfornhs #covid19 #weareinittogether #stayathome #bandanavixenv @Surbiton, London</blockquote><p><strong>Assumption</strong>:</p><ol><li><strong>Standard Deviation for each distribution is the same.</strong> The boxplot above helps us confirm this visually by comparing the interquartile ranges:</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*Z29PBl4Rxdvd3zS_zYmxNw.png" /><figcaption>Violinplot (l) and Boxplot (r)</figcaption></figure><p>we also confirm this using NumPy:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bd93253d6c75f77a0e293120c56c69be/href">https://medium.com/media/bd93253d6c75f77a0e293120c56c69be/href</a></iframe><pre>StDev of Gov sentiment: 0.508<br>StDev of NHS sentiment: 0.527</pre><p>Confirming the validity of this assumption enables us to use a <a href="https://en.wikipedia.org/wiki/Pooled_variance">pooled variance</a> approach so that implementation of SciPy’s 2 sample t-test boils down to:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ff3b921bfc1c70444949cb689c00ecf3/href">https://medium.com/media/ff3b921bfc1c70444949cb689c00ecf3/href</a></iframe><p>Had standard deviations been different, we would have used <a href="https://en.wikipedia.org/wiki/Welch%27s_t-test">Welch’s t-test</a> to calculate the standard deviation across both samples. In practise, this would have meant simply setting equal_var =False.</p><p><strong>Result:</strong></p><pre>Ttest_indResult(statistic=43.62708002444751, pvalue=0.0)</pre><p>Recall the sample means we were comparing…</p><pre>Avg Gov sentiment -0.036<br>Avg NHS sentiment 0.171</pre><p>A p-value~0 tells us that it’s highly improbable a Student’s t-distribution centred on a mean of -0.036, would also contain an observation of 0.171. Put another way, the test statistic calculated above is stating that a mean of 0.171 would lie 43 standard errors away from the mean of -0.036</p><p><strong>Verdict: Reject the Null. </strong>We can say with a high degree of confidence that sentiment towards the NHS is more favourable than for the Government.</p><h4>When did sentiment start to diverge?</h4><p>We saw a difference in average sentiment for the entire 3 month period, but when did this start to diverge?</p><p>We can plot the daily difference in average sentiment, however, we would need to adjust for the sample size and the standard deviation of the data for each day to measure our confidence in our daily sample means.</p><p>Instead, we can calculate 95% confidence intervals for the daily differences in average sentiment which incorporates our confidence in our sampling process: <em>we create an upper bound and a lower bound which we’re 95% confident will contain the true difference in average sentiment for that day.</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2c7d2bc2f8efe506991b2a35bc330e84/href">https://medium.com/media/2c7d2bc2f8efe506991b2a35bc330e84/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*n3f9oBDLhxLKpg3puF-u6Q.png" /></figure><p>Each vertical black line is a confidence interval constructed for that day. Each day’s confidence interval marks the region where we’re 95% confident that the true difference in Government vs NHS sentiment lies (because of course our data is inherently just a sample of an overall population).</p><p>At the start of the observation period, there were relatively few tweets made per day. These small daily sample sizes gave rise to larger standard errors because:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/137/1*aV2b3KNiDi86Gnveb8nxWw.png" /></figure><p>and therefore wider confidence intervals. As time progressed, the daily sample sizes increased allowing our confidence intervals to narrow.</p><p>The confidence intervals were for the most part <em>completely below the red dashed zero line</em>, indicating a 95% confidence that average sentiment towards the government was lower than for the NHS throughout this entire period. There are some rare moments, like the end of April, when the intervals crossed over the zero line… for those days we can’t say with the confidence level we have chosen that government sentiment was lower than for the NHS.</p><h3>Hypothesis 2: Government vs NHS… positive or negative feeling?</h3><p>We saw there was a difference in average sentiment, but was this due to more positivity towards the NHS (compound &gt; 0.05), or negativity towards the Government (compound &lt; -0.05)?</p><p>Visually it appears both statements could be true, but we will test for this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*vEmjxCC10LVuwalRoYmYTQ.png" /><figcaption>Distplot comparison using Seaborn</figcaption></figure><p><strong>Positive Tweets:</strong></p><p>The difference in the ratios is 0.159, ie. the NHS has a 0.159 higher ratio of positive tweets than the Government.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4d5744e8f078353c5c8feaa5e3016d52/href">https://medium.com/media/4d5744e8f078353c5c8feaa5e3016d52/href</a></iframe><pre>Difference in (Positive Tweets / Total Tweets): 0.15881778668707092</pre><p>We can test this difference for statistical significance by running a “difference of two <em>proportions </em>test”, which again will rely on calculating a t-statistic for the hypothesis that both ratios came from the same distribution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/265/1*aBbKM8OBNqSzWiJezVAbHQ.png" /></figure><p>where, the standard error for a difference in proportions <em>p_hat</em> is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/362/1*6U75o9RmpqkMp3ZxHzbjlg.png" /></figure><p>so we can easily manually calculate the t-stat and associated p-value:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/779422e6f37838d58b8147e43e995327/href">https://medium.com/media/779422e6f37838d58b8147e43e995327/href</a></iframe><pre>t-statistic 47.33409611769171 p-value 0.0</pre><p><strong>Verdict: Reject the Null.</strong> The alternate hypothesis (NHS’s ratio of positive tweets) lies 47 standard errors away from the null (the Government’s ratio), and with a p-value of 0, we say it’s highly unlikely that both came from the same population.</p><p><strong>Negative Tweets:</strong></p><p>We repeat the analysis for negative tweets:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8d97dcf0a83109bc89b9fa53e53da0f7/href">https://medium.com/media/8d97dcf0a83109bc89b9fa53e53da0f7/href</a></iframe><pre>Difference in (Negative Tweets / Total Tweets): 0.14177193931000753</pre><p>We can see that the Government’s negative tweets ratio is 0.14 higher than the NHS’s, and we test to see if the difference is statistically significant:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e44b8e0e11febc738debcaa924ab8759/href">https://medium.com/media/e44b8e0e11febc738debcaa924ab8759/href</a></iframe><pre>t-statistic 38.561197004624 p-value 0.0</pre><p><strong>Verdict: Reject the Null.</strong> Like with the positive ratio, the difference is statistically significant.</p><p>Alternatively, we could have used the Pandas crosstab function to create a table that shows the number of tweets in each category. Then applied the <a href="https://en.wikipedia.org/wiki/Chi-squared_test">chi-squared test</a> to verify if any row contains an imbalance in the number of pos, neu, or neg tweets:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/288839e1b78676d00469ced692135fdf/href">https://medium.com/media/288839e1b78676d00469ced692135fdf/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/234/1*LmdmpLtUnHv5zGxdUFGCZA.png" /><figcaption>crosstab table</figcaption></figure><p>This returns:<br>- the chi_squared statistic<br>- the p-value<br>- the degrees of freedom<br>- the expected frequencies for an even distribution</p><pre>(1047.7937788707934,<br> 2.981754146928481e-228,<br> 2,<br> array([[6978.92922191, 3234.01294658, 9538.05783151],<br>        [6640.07077809, 3076.98705342, 9074.94216849]]))</pre><p>We can tell from this that the distribution of tweets are imbalanced compared to what an even distribution would look like for both positive and negative.</p><h3>Hypothesis 3: Government vs NHS…were Londoners divided in their opinions?</h3><p>Were some users tweeting more positively about the NHS, while others were more negative pointing to a split amongst Londoners? Or was this favourable view of the NHS also true at the individual user level, and therefore held by each of us?</p><p>This is also a simple test to set up called a <a href="https://en.wikipedia.org/wiki/Student%27s_t-test#Dependent_t-test_for_paired_samples">paired-means test</a>, where we start with using Pandas crosstab again to create a user level view of average sentiment:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3be79b306441d9c40f700281a0ce4acb/href">https://medium.com/media/3be79b306441d9c40f700281a0ce4acb/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/321/1*2lpzbOn96XZuuj5uImp_Mg.png" /></figure><p>As with the other parametric tests, we rely on the assumption of normality in the distribution of the variable of interest (in this case the <em>differences</em> column), which we can see roughly holds:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c9ebaf88daaeee339a1a287d30f233fd/href">https://medium.com/media/c9ebaf88daaeee339a1a287d30f233fd/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/864/1*-qBuJ3hrKqkitQt87zPf9g.png" /><figcaption>Histogram and QQ Plot for the differences column</figcaption></figure><p>We use the same framework to calculate the t-statistic:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/265/1*HVzr2wW_zNsaxgxy-GXUrA.png" /></figure><p>except now the Alternate Hypothesis is the mean of differences column, the Null Hypothesis is that there is no overall difference (ie, zero), and as with Hypothesis 1 (which was a 2 sample means test) the standard error is calculated by scaling the standard deviation (this time of the differences):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/156/1*trxaaSmDWovIwvsrfJfqQQ.png" /></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cac18d49d00ad07d14e59bbfc8654efe/href">https://medium.com/media/cac18d49d00ad07d14e59bbfc8654efe/href</a></iframe><p>or simply using SciPy:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4d6c5d897fba8386c2421395e6e2a132/href">https://medium.com/media/4d6c5d897fba8386c2421395e6e2a132/href</a></iframe><pre>Ttest_relResult(statistic=-9.035954747008454, pvalue=3.4616270088977246e-19)</pre><p><strong>Verdict: Reject the Null. </strong>Even at the individual user level, we detect diverging sentiment towards the Government and the NHS. It appears we weren’t divided in holding these divergent views.</p><p><em>Note: we didn’t need the normality assumption to be able to test for this, the non-parametric analogue of the paired-means test (the </em><a href="https://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test"><em>Wilcoxon Signed-Rank</em></a><em> test) could have been used instead which looks at the median, not the mean:</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8011658861716e59bd1ef31b25982dd6/href">https://medium.com/media/8011658861716e59bd1ef31b25982dd6/href</a></iframe><pre>WilcoxonResult(statistic=934184.0, pvalue=1.2233021976129815e-18)</pre><p><strong>Verdict: </strong>Same outcome.</p><h3>Why was the sentiment more positive towards the NHS?</h3><p>While we now know that public sentiment was warmer towards the NHS than the Government, it’s hard to discern what emotions exactly were stirred when the public thought about one vs. the other. We can use a wordcloud to visualise what words were used:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cffd5211b5198ac5607d8a6f04b99c5a/href">https://medium.com/media/cffd5211b5198ac5607d8a6f04b99c5a/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*yLroGy-cdor5KVHnBrnbMw.png" /></figure><p>We are no clearer as the picture is obscured because the most frequently occurring words were neutral and didn’t contribute to the valence. We need to filter the Government and NHS corpora for the most emotive words:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/245e4698c1020e56547d7c2ae2e98509/href">https://medium.com/media/245e4698c1020e56547d7c2ae2e98509/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*uyKCL-a3EwSr5IbZzZjKZA.png" /></figure><p>We now have a clearer picture of why the NHS was seen more positively than the Government:</p><p><strong>Government:</strong> tweeters referenced the Government when highlighting perceived failures more often than when referencing the NHS:</p><ul><li><em>crisis, failure, died, die, vulnerable, stop, failed, seriously, lack, emergency, hard, warning, failing</em></li></ul><p><strong>NHS:</strong> tweets generally refer to the NHS as heroes for the support and care they gave in such trying circumstances.</p><ul><li><em>thank, care, support, well, help, safe, good, protect, love, helping, hope, friend, heroes, proud, great</em></li></ul><p>It’s interesting therefore that despite ostensibly working towards a common goal, the public it seems saw the NHS as fighting its battles better than the Government did.</p><h3>NHS sentiment over time:</h3><p>How did this NHS sentiment evolve over time? Did the public grow to love the NHS, or did positivity-fatigue set in?</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9503749084535ce82090fd0ab85fe238/href">https://medium.com/media/9503749084535ce82090fd0ab85fe238/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*e2H01m1yreX5O8Rn98M_Xg.png" /><figcaption><em>(Note: the increase you see in num_tweets in May is due to an improvement in my Twitter query as I mentioned in </em><a href="https://towardsdatascience.com/covid-19-sentiment-analysis-series-part-1-getting-started-70892122630c"><em>Part 1</em></a><em>, with this in mind however we can still make the below points)</em></figcaption></figure><p>There were 3 noticeable weekly peaks most clearly seen in the number of tweets but also apparent in the compound sentiment score around the end of March/start of April, coinciding with the start of the #ClapforCarers campaign.</p><p>While sentiment towards the NHS was positively impacted at this time these weekly peaks have since largely subsided, perhaps Londoners have been feeling #ClapforCarers fatigue?</p><h3>Summary</h3><p>We’ve shown how you can apply hypothesis testing to make statistically robust inferences about Twitter sentiment data. We’ve demonstrated how different types of hypothesis tests can help shed more light on the nature of any differences in sentiment, a recap:</p><ol><li><strong>Hypothesis 1:</strong> A 2-sample t-test comparing for <em>sample means</em>, allowed us to conclude there was on average a difference in Gov vs. NHS sentiment. <em>We also drew confidence intervals to track how this evolved over time.</em></li><li><strong>Hypothesis 2:</strong> 2-sample <em>difference of proportions t-test</em> confirmed that this high level difference was driven both by more positivity towards the NHS and also more negativity towards Government.</li><li><strong>Hypothesis 3:</strong> A <em>paired-means</em> t-test showed us that users were not split in this respect, ie. this divergence in Gov vs. NHS sentiment held at the individual user level as well.</li><li><strong>Visualisation:</strong> We used wordclouds to find what emotive words were being used to describe the Gov and the NHS to get further clarity as to what the public actually thought. We even showed the impact that #ClapforCarers had on warmth of feeling towards the NHS.</li></ol><p>In the next (and final) blog in this series, we will be casting the net wider to find more winners and losers of the lockdown period. We will take it step-by-step going deeper into Machine Learning territory to develop interesting ways to approach this problem…</p><ul><li>We will start by repeating the wordcloud/hypothesis testing trick to find new winners and losers.</li><li>We will then look at how to streamline this process by using a Logistic Regression model to give a holistic view.</li><li>Finally, I will explain <em>how</em> and, importantly, <em>why</em> a Bayesian approach could help us follow the evolving mood of a city.</li></ul><p>See you in <a href="https://medium.com/@noah.berhe/covid-19-sentiment-analysis-series-part-3-winners-and-losers-c00f384080a8?source=friends_link&amp;sk=f5436c1fd12af1e4e948880dae09c3cd">Part 3</a>…!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3db511875165" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/covid-19-sentiment-analysis-series-part-2-hypothesis-testing-3db511875165">COVID-19 Sentiment Analysis Series, Part 2: Hypothesis Testing</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[COVID-19 Sentiment Analysis Series, Part 1: Getting Started]]></title>
            <link>https://medium.com/data-science/covid-19-sentiment-analysis-series-part-1-getting-started-70892122630c?source=rss-36db8d98670a------2</link>
            <guid isPermaLink="false">https://medium.com/p/70892122630c</guid>
            <category><![CDATA[covid19]]></category>
            <category><![CDATA[twitter-sentiment]]></category>
            <category><![CDATA[correlation]]></category>
            <category><![CDATA[vader]]></category>
            <category><![CDATA[spacy]]></category>
            <dc:creator><![CDATA[Noah Muluberhan-Berhe]]></dc:creator>
            <pubDate>Tue, 02 Jun 2020 17:45:03 GMT</pubDate>
            <atom:updated>2020-06-09T22:32:23.800Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fKCe23zIH9dlAjaD" /><figcaption>Photo by <a href="https://unsplash.com/@purzlbaum?utm_source=medium&amp;utm_medium=referral">🇨🇭 Claudio Schwarz | @purzlbaum</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4>Twitter Sentiment Analysis</h4><h4>Understanding Londoners in lockdown</h4><p>What I found quite striking about the lockdown is the loss of community that came with meeting friends for a drink, or talking to colleagues around someone’s computer.</p><p>With that, we lost a sense of how everyone was doing, a feeling of being heard, and the shared voice that brought society together.</p><p>I wanted to counteract this in some small way by looking to Twitter to understand how Londoners were reacting to events. Who have we warmed to? And what has brought us anger and frustration?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OUMAt61gjHq8D_0e" /><figcaption>Photo by <a href="https://unsplash.com/@kmitchhodge?utm_source=medium&amp;utm_medium=referral">K. Mitch Hodge</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This will be a 3-part series where we will look at a few ways one could approach this question…</p><p><strong>Part 1: Getting started:</strong> how to get your tweets, how to score tweets for sentiment, conducting some high-level correlation analysis.</p><p><strong>Part 2: Hypothesis testing:</strong> how to confirm to a high degree of statistical confidence differences in sentiment for organisations, using word-clouds to see what the public <em>really </em>thought of someone!</p><p><strong>Part 3: Modelling:</strong> showing step-by-step how to apply Frequentist and Bayesian approaches to modelling the winners and losers of the lockdown, I will also be outlining clearly what each approach offers us in answering this question.</p><h3>Part 1: Getting started</h3><h4>Getting the tweets:</h4><p>I attempted to use the Twitter API to retrieve tweets about the virus, however, the Search API restricts historical calls to 7 days, luckily I found a Python Twitter scraper called GetOldTweets3 which helped me round this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d35ae7112a8aee918519ec9866dbad7e/href">https://medium.com/media/d35ae7112a8aee918519ec9866dbad7e/href</a></iframe><p>I found there were stability issues when scraping Twitter, resulting in “HTTP Error 429: Too Many Requests” when I hit Twitter quotas or “HTTP Error 503: Service Temporarily Unavailable” when I lost connection with their server.</p><p>Both errors will cause the data collected to be lost,<strong> </strong><em>so try/except is your friend</em>… in the case of these exceptions, I asked the function to wait 15 minutes before re-trying, and made sure I was printing plenty of progress updates for the user… including a timer when the function was in time-out…</p><blockquote>Attempt 1 of 3 retrieving 2020–05–30<br>An error occurred during an HTTP request: HTTP Error 429: Too Many Requests</blockquote><blockquote>Error retrieving 2020–05–30 on attempt 1 of 3. Sleeping for 15 minutes.<br>Minutes Slept:<br>0.. 1.. 2.. 3.. 4.. 5.. 6.. 7.. 8.. 9.. 10.. 11.. 12..</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ee74a7adee571c9a74f1aee2727d7d9c/href">https://medium.com/media/ee74a7adee571c9a74f1aee2727d7d9c/href</a></iframe><p>At the time of writing, I have collected 550,000 tweets over a 3 month period from Londoners with the search-string “coronavirus OR covid OR covid-19 OR covid19”, and saved these to my GitHub as daily csv files.</p><p><strong>Parsing text:</strong></p><p>We want to extract who/what Londoners were tweeting about so we can find the main drivers of sentiment…</p><p>I experimented a little with spaCy’s NLP pipeline:</p><p>First using the Part-of-Speech Tagger to retrieve proper nouns (although it worked well, I found this unsuited to my goal as it retrieved many non-entities).</p><p>Then using the Entity Recognizer to retrieve anything the pre-trained pipeline recognised as a ‘PERSON’ or an ‘ORG’.</p><p>Then (out of interest) I used the EntityRuler to over-ride the Entity Recognition with my own custom list (I looked at two: Boris Johnson and the NHS).</p><p>Although the EntityRuler was the most precise it was quite tedious and required manually setting up a rule for each entity, so I opted for the Entity Recognizer:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/333088385fc4e49fca16dcbdc94c0e26/href">https://medium.com/media/333088385fc4e49fca16dcbdc94c0e26/href</a></iframe><h4>Scoring Sentiment</h4><p>I used the VADER (Valence Aware Dictionary and sEntiment Reasoner) Python library specifically because it has been tuned to deal with sentiment expressed on social media. It combines a lexicon approach with a rule-based approach to provide a high level of accuracy when scoring sentiment on social media.</p><p>Their GitHub provides more details from the research paper that gave us VADER, but in summary:</p><p><strong>The Lexicon approach</strong>:</p><ul><li>Sentiment ratings, on a scale of -4 (extremely negative) to +4 (extremely positive), collected for over 9,000 words from 10 independent judges, who had themselves undergone a vetting process.</li><li>Features that had a 0 average rating, or a standard deviation of ratings &gt; 2.5 were discarded, leaving us with a lexicon of 7,500 validated features.</li></ul><p>VADER then goes beyond just looking at the sentiment rating of individual words, the study also looked at the impact of syntax on sentence-level sentiment…</p><p><strong>The Rule-based approach:</strong></p><ul><li>For example, “good” vs “extremely good” is an example of the degree-modifier “extremely” increasing the intensity of sentiment in the sentence.</li><li>Rules have been defined in VADER that detect structural elements such as this, and apply empirically-derived weightings to the overall sentiment.</li></ul><p>The code is very simple to implement: initialise the SentimentIntensityAnalyzer() class then apply the polarity_scores() method (which I did inside a map function).<br>This returned an array containing 4 different sentiment measures per tweet: POS, COMPOUND, NEU, NEG.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4517f501f71ab0c63949bc5fb2dadf25/href">https://medium.com/media/4517f501f71ab0c63949bc5fb2dadf25/href</a></iframe><p>3 of the scores (POS/NEU/NEG) quantified what fraction of that tweet contained POS/NEU/NEG words (e.g. POS = no. positive words in tweet/total no. words in tweet).</p><p>I decided, however, to use COMPOUND (or, valence) in the following analyses as it explains succinctly the degree of polarity for BOTH poles: positive and negative. It’s also normalised to between -1 to +1 (+1 most positive, -1 most negative) with respect to the length of that text, enabling use as a comparison measure. Researchers have typically classed a score &gt; 0.05 as positive, and &lt;-0.05 as negative, which I will also be adopting.</p><h4>Correlation: Did sentiment vary with the spread of the virus?</h4><p>I found a dataset on Kaggle that had collated Public Health England’s statistics on the cases and deaths from Coronavirus. I started to plot this in Matplotlib together with the average daily sentiment observed from the tweets:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/703/1*YjSh-xT-RgIfJjHgLx-GXg.png" /><figcaption>Plotted: Negative Sentiment and Cases / Deaths</figcaption></figure><p>I learned a couple of things from seeing the data like this:</p><ol><li><strong>A.B.C. (Always Be Checking)</strong><br>At the start of the period, I noticed the no. tweets I was collecting reduce dramatically, and soon found my query had become obsolete…<br>- Don’t search for hashtags, search instead for text as most tweeters don’t use hashtags frequently.<br>- “COVID” replaced “Coronavirus” as the convenient shorthand for frequent tweeters.<br>So, changing the search string to <em>“coronavirus OR covid OR covid-19 OR covid19”</em> saw an immediate improvement (see from May).<br>Thankfully I still had a large enough sample size despite this.</li><li><strong>High-level data yields low-level insight</strong><br>At such a high level of aggregation, sentiment is often range-bound around a global average.</li></ol><h4>Did we see more negative tweets as the virus progressed?</h4><p>We can see that broadly, yes.</p><p>For example, sentiment (ie, compound score) vs daily deaths has a negative correlation of -41%. This increases to -50% when you add a 1-day lag to the death toll, suggesting there could have been some delay in people reacting to the previous day’s news.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*rvLNqud3TtVsf9wCOX89Ww.png" /></figure><p>After some early instability in the correlation, we were able to see something interesting:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*OfCn_VImxe_ZZPrY5j_4sg.png" /><figcaption>Correlation History: Negative Tweets vs. Daily Deaths</figcaption></figure><p>The correlation was largely negative through March and April i.e. <em>“an increase in the no. cases /deaths (more positive) happens in line with a deterioration in overall sentiment (more negative score)</em>.<em>”</em> which is also what we saw looking at the sentiment and cases/deaths trends over time.</p><p>The correlation has since turned positive, with sentiment remaining relatively unchanged even as daily death numbers became minimal, suggesting that Londoners have found other reasons now for their ire.</p><h4>Conclusions</h4><p>Having collected our data, we’re now ready to dive deeper and understand Londoners a bit better.</p><p>In the next post, I will be using hypothesis testing <em>(and lots of it!)</em> to verify to a high degree of statistical confidence if there really was a difference in attitude towards different organisations. I will show how to set the tests up, and explain what assumptions we are relying on in each test (and how to test for these).</p><p>We will also see how to visualise people’s REAL opinions in wordcloud form… e.g. we saw the NHS as <em>“caring”, “supportive”, “heroic”, “protective”, “helpful”</em>… but did we think the same of the government? And what effect did the #ClapforCarers have on this?</p><p>See you in <a href="https://medium.com/@noah.berhe/covid-19-sentiment-analysis-series-part-2-hypothesis-testing-3db511875165">Part 2</a>…!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=70892122630c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/covid-19-sentiment-analysis-series-part-1-getting-started-70892122630c">COVID-19 Sentiment Analysis Series, Part 1: Getting Started</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Congressional Tweets: Using Sentiment Analysis to Cluster Members of Congress in PySpark]]></title>
            <link>https://medium.com/analytics-vidhya/congressional-tweets-using-sentiment-analysis-to-cluster-members-of-congress-in-pyspark-10afa4d1556e?source=rss-36db8d98670a------2</link>
            <guid isPermaLink="false">https://medium.com/p/10afa4d1556e</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[sentiment-analysis]]></category>
            <category><![CDATA[clustering]]></category>
            <dc:creator><![CDATA[Noah Muluberhan-Berhe]]></dc:creator>
            <pubDate>Wed, 06 May 2020 12:46:49 GMT</pubDate>
            <atom:updated>2020-05-10T16:29:04.364Z</atom:updated>
            <content:encoded><![CDATA[<h3>Twitter Sentiment Analysis: Using PySpark to Cluster Members of Congress</h3><p>Having used quite a lot of Python in my previous projects I wanted to take on a large dataset which would require PySpark’s parallelised computing abilities. I found a large dataset on tweets sent by Members of the US Congress which looked interesting, and came up with an idea…</p><p><strong>My hypothetical client:<br></strong>I’m helping Lobbyists4America better understand relationships between congress members through their tweets: the dataset is comprised of tweets sent between 2008–2017. The aim being to use this intelligence to help them better target their clients’ lobbying efforts in Congress.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sYZYkrtR1BlyNgmY_M2mYQ.jpeg" /><figcaption>Which of these members of Congress are most alike?</figcaption></figure><p><strong>Hypotheses:<br></strong>1. There will be clusters of like-minded politicians that can be drawn largely along party lines.<br>2. There will be centres of influence (loud / influential voices) in these clusters that clients can target.<br>3. Some themes will emerge as more topical in this time period</p><p><strong>Approach:</strong><br>1. Import data and conduct EDA.<br>2. Theme extraction: Bag-of-Words<br>3. Entity Recognition: Spark-NLP<br>4. Sentiment Analysis: using TextBlob for sentiment scoring<br>5. Spark-ML to cluster like-minded members.</p><h3><strong>Step 1: Import data and conduct EDA.</strong></h3><p>Import Python libraries</p><pre>import pandas as pd<br>import numpy as np<br>import matplotlib.pyplot as plt<br>import seaborn as sns</pre><p>Import Spark functions and Spark-NLP, in the process building a SparkSession that ensures Spark-NLP jars files will be found in our module (I found I had to do this otherwise there was a No Module Found error)</p><pre>import sparknlp<br>from pyspark.sql import SparkSession<br>spark = SparkSession.builder \<br>    .master(&quot;local[4]&quot;)\<br>    .config(&quot;spark.driver.memory&quot;,&quot;4G&quot;)\<br>    .config(&quot;spark.driver.maxResultSize&quot;, &quot;2G&quot;) \<br>    .config(&quot;spark.jars.packages&quot;, &quot;com.johnsnowlabs.nlp:spark-nlp_2.11:2.4.5&quot;)\<br>    .config(&quot;spark.kryoserializer.buffer.max&quot;, &quot;1G&quot;)\<br>    .getOrCreate()<br>from sparknlp.pretrained import PretrainedPipeline<br>from pyspark.sql.functions import from_unixtime, to_date, year, udf, explode, split, col, length, rank, dense_rank, avg, sum<br>from pyspark.sql.window import Window</pre><p>Spark ML to be used later when applying Clustering</p><pre>from pyspark.ml.linalg import Vectors<br>from pyspark.ml.feature import VectorAssembler, StandardScaler<br>from pyspark.ml.stat import Correlation<br>from pyspark.ml.clustering import BisectingKMeans<br>from pyspark.ml.evaluation import ClusteringEvaluator<br>from pyspark.ml import Pipeline<br>from pyspark.ml.tuning import CrossValidator, ParamGridBuilder</pre><p>NLP libraries I will implement:</p><pre>import nltk<br>from nltk.corpus import stopwords<br>from collections import Counter<br>from wordcloud import WordCloud</pre><pre>from textblob import TextBlob</pre><p><strong>Import data into Spark dataframe:</strong></p><p>It’s relatively simple to read in JSON files as Spark Dataframes, although there were some considerations to be made to enable more efficient processing when working with the dataframes.<br>1. A Spark dataframe should be split into partitions = 2–3 times the number of threads available in your CPU or cluster. My machine has 4 logical cores so I chose 3x, ie. 12 partitions, based on experimentation.<br>2. Then I cached the tables (‘persist’) to improve query performance later: you can check the Storage tab of the Spark GUI that 12 partitions have indeed been cached for each file.<br>3. The Shuffle Read partitions parameter is default to 200, we don’t want this to be the bottleneck, so we set this equal to partitions in our data, using spark.sql.shuffle.partitions. This will help specifically with wide shuffle transformations (e.g. GROUP BY or ORDER BY) that may be performed later on.</p><pre># Tweets<br>tweets = spark.read.json(&#39;../Lobbyists4America1/US_PoliticalTweets/tweets.json&#39;).repartition(12).persist()</pre><pre># Users<br>users = spark.read.json(&#39;../Lobbyists4America1/US_PoliticalTweets/users.json&#39;).repartition(12).persist()</pre><p>I used Spark-SQL to create different slices of the data to get a better understanding of it.<br>One observation pertinent to my hypotheses is that few members have a large following on Twitter, these would powerful advocates for my clients’ policies</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/1*KxfbkJG5edbMBnYkn6RX7Q.png" /></figure><p>Namely…</p><pre>+------------------+---------------+<br>|              name|followers_count|<br>+------------------+---------------+<br>|   Donald J. Trump|       31712585|<br>|   President Trump|       18545354|<br>|    Bernie Sanders|        5072538|<br>|       Cory Booker|        3094413|<br>|       Marco Rubio|        2554822|<br>|  Elizabeth Warren|        2412087|<br>|       John McCain|        2274034|<br>|      Nancy Pelosi|        1126205|<br>|       Jerry Brown|        1119275|<br>|  Senator Ted Cruz|         960413|<br>| Governor Christie|         869256|<br>|       John Kasich|         755475|<br>|     Chuck Schumer|         591618|<br>| Senator Tim Kaine|         553125|<br>|        Trey Gowdy|         552930|<br>|        John Lewis|         512433|<br>|Kirsten Gillibrand|         396587|<br>|     Maxine Waters|         379015|<br>|          Mike Lee|         314821|<br>|Rep. Keith Ellison|         313271|<br>+------------------+---------------+<br>only showing top 20 rows</pre><h3>Step 2: Theme extraction: Bag-of-Words</h3><p>After having a quick look at the data and some descriptive stats, I wanted to go a little deeper and understand what the main themes were.</p><ol><li>NLTK has a corpus of stopwords that I downloaded to help me remove them from the tweets.</li><li>Spark RDDs can be manipulated such that we can derive a word count from a collection of documents / tweets: using flatMap, reduceByKey and sort.</li><li>I then fed this into WordCloud to visualise this for each year in the dataset</li></ol><p><em>Note: I focused on years 2013 onwards, as they had large enough sample sizes</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*U6uVjyRMf7cfXbTUFoeTMg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*vBuEqD9wzYe5HIUg4Q9WSA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*k1Y2NydO1NuuKDWxD4QMeg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*A974frtjB7t-NEM8SwPZTw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*Sy6APeunpX3xV9j8LzvNWQ.png" /></figure><h3>Step 3: Entity Recognition: Spark-NLP</h3><p>The problem with the Bag-of-Words approach is that there were many words that didn’t constitute topics or themes, so I fed the corpus generated above into Spark-NLP’s pre-trained pipeline and essentially asked it whether each word was an entity.<br>Recreating the wordclouds on this cleaned corpus, it’s much clearer to see the hot topics at this time:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*8kqN5019YPbo9R8mAU1bVA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*Cav5X0aVZEDx2l3j9Bhahg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*kEkl5p_QdIh54MKk_dLNzA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*mT8ql3Dtyo2rbiHhxXmnuQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*JiJE2AaohfbQB_bIQzBLow.png" /></figure><p>This step gave me some comfort in my direction of travel: I am going to focus on Healthcare as the main theme for analysis…</p><h3>Step 4: Sentiment Analysis: using TextBlob for sentiment scoring</h3><p>Assigning each tweet a sentiment score was incredibly simple with TextBlob, simply 3 lines of code which took less than a second to run over the entire dataset:</p><pre>sentiment = udf(lambda x: TextBlob(x).sentiment[0])<br>spark.udf.register(“sentiment”, sentiment)<br>tweets = tweets.withColumn(‘sentiment’,sentiment(‘text’).cast(‘double’))</pre><p>I looked at the overall trend in sentiment and also number of tweets. Using a 90 day daily moving average we can see that Twitter started to gain popularity as a medium for communication by members of Congress from 2013 onwards. It’s also interesting to see a very large increase at the back end of 2016: perhaps to do with Trump’s high engagement in Twitter and what appears to be other politicians responding to his tactic and increasing their own presence on Twitter. This increase was accompanied by a slight drop in sentiment, can we infer that tweets started becoming more confrontational in tone?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*X-E3WeDWIdV5o1Eo5eUFfA.png" /></figure><p>Looking at tweets containing “Obamacare” we see an interesting trend: in 2013 the number of these tweets really started to pick up sharply and fall away just as sharply in 2014, was this a deliberate tactic? Also interestingly: during this pick up there was a drop in sentiment to neutral, looks like if you’re tweeting about “Obamacare” instead of “ACA” then you’re likely to be negative about it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/1*5q7wASxjngGnRN3TVXz9Vw.png" /></figure><p>This is borne out when looking at those members who tended to use “Obamacare”: they were Republicans.<br>Something else interesting: was this drop in “Obamacare” sentiment also the cause of the overall drop in sentiment at this time? <br>Explaining overall changes in sentiment by theme could be an interesting way to shed light on overall trends, perhaps creating some sort of weighted sentiment measure at the thematic level: but that’s for another time (unrelated to my hypotheses).</p><h3>Step 5: Spark-ML to cluster like-minded members</h3><p>Focusing on Healthcare, I wanted to get a more nuanced idea of likeminded and similarly-energised Congress members. It’s easy enough to split them into R vs D, but what if you wanted to know which of those members of Congress held the strongest opinions on the topic, and which of them were also energised by the issue to be active on Twitter about it. This would be valuable intel for a lobbyist.<br>So I used Spark-ML’s unsupervised learning models (namely Bisecting K-Means) to create these clusters based on the number of tweets and sentiment expressed by members in tweets containing either “ACA” or “Obamacare”</p><ol><li>VectorAssembler to create the features column</li><li>Applied StandardScaler to said features column</li><li>Optimise for k in Bisecting K-Means, by iterating through different options and evaluating using the silhouette score.</li><li>Chose k=6 as this had the highest score: 0.502</li><li>Scatterplot…</li></ol><p>I chose to annotate each point with the member’s name and also that member’s ranking based on number of followers. So, based on number of followers Cory Booker appears to be the most influential member within his cluster being the one with the 4th highest number of followers among all members of Congress on Twitter.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/1*KsZB4vDjqF9guvkohRjT9w.png" /></figure><h3>Conclusions</h3><ol><li>Very interesting to see clusters based on the views / strength of opinion expressed on Healthcare reforms. The clusters appear to make sense and give us a more nuanced view than just Republican vs Democrat.</li><li>Here we now have 6 clusters of likeminded (sentiment) members who are also similarly motivated (num_tweets) by the issue at hand.</li><li>The green cluster appears to be highly opposed to Obamacare, among them Mike Lee appears to have the highest following, and very vocal on the issue as well.</li><li>The red cluster are even more negative in their sentiment, although have tweeted far fewer times than those in the green cluster. This could be to do with sample size: the smaller the sample size the more susceptible it is to extremes in sentiment, while the larger the sample size the more it tends towards neutral. This is something I saw in different slices of the data: the more you zoom out, the more sentiment neutralises.</li></ol><h3>Hypotheses: Verdict</h3><ol><li><strong>There will be clusters of like-minded politicians that can be drawn largely along party lines.<br></strong>Correct</li><li><strong>There will be centres of influence (loud / influential voices) in these clusters that clients can target.<br></strong>Correct</li><li><strong>Some themes will emerge as more topical in this time period<br></strong>Correct</li></ol><h3>Potential Next Steps</h3><p>I have created these clusters using the views of politicians across two different themes: ACA and Obamacare, the potential is there to create different clusters on the issue of Foreign Policy, or Energy, or Tax Reforms. Or even one set of clusters across different issues: to see which members are likeminded in general, and not just on specific issues…. this could lead us to extrapolate that … “if politician A is like politician B on this issue, then they may also come round on this issue as well”</p><p>Thanks for reading, for those interested… you can find the code on my GitHub, any feedback for this budding Data Scientist would be greatly appreciated!</p><p><a href="https://github.com/noahberhe/Lobbyists4America">noahberhe/Lobbyists4America</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=10afa4d1556e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/congressional-tweets-using-sentiment-analysis-to-cluster-members-of-congress-in-pyspark-10afa4d1556e">Congressional Tweets: Using Sentiment Analysis to Cluster Members of Congress in PySpark</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>