<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Lakmali Shashika Karunarathne on Medium]]></title>
        <description><![CDATA[Stories by Lakmali Shashika Karunarathne on Medium]]></description>
        <link>https://medium.com/@shashika.lakmali96?source=rss-1563541f871f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*np20PMxOaosHuK98nUvR_Q.jpeg</url>
            <title>Stories by Lakmali Shashika Karunarathne on Medium</title>
            <link>https://medium.com/@shashika.lakmali96?source=rss-1563541f871f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 20:37:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@shashika.lakmali96/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Sentiment Analysis and Predictive Analytics of COVID-19 Tweets Using Machine Learning Techniques.]]></title>
            <link>https://medium.com/@shashika.lakmali96/sentiment-analysis-and-predictive-analytics-of-covid-19-tweets-using-machine-learning-techniques-5c27fff95c13?source=rss-1563541f871f------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c27fff95c13</guid>
            <dc:creator><![CDATA[Lakmali Shashika Karunarathne]]></dc:creator>
            <pubDate>Mon, 09 Sep 2024 10:24:30 GMT</pubDate>
            <atom:updated>2024-09-09T10:24:30.447Z</atom:updated>
            <content:encoded><![CDATA[<h3>1. Acquisition And Understanding the Dataset</h3><p>The selected dataset is COVID 19 Tweet.</p><p>The Twitter data set of Covid 19 is contained the entire tweets from all over the countries in the world like UK, USA, Australia, Canada, Philippines, Ireland, India, Istanbul etc. which is the duration for the pandemic to perform the analysis of the behavioural and the prediction of the outbreak. Since the number of data is limited which can be gathered by the Twitter API endpoints for the purpose of focusing on researching about the different cities in different countries like the England is conducted. The cities in different countries which are having UK in London, Manchester, Liverpool, and the largest cities by the population in UK and other countries are found the initial exploratory analysis. To lead nearby assessment, the region of a Twitter supporter can be gathered in unmistakable procedures. Ordinarily tweet devices involve none, one or everything about ensuing region related attributes, district or geotagged. A geotagged tweet brings reach and longitude facilitates labelled through the man or lady empowering their area to be to be had. Though the spot characteristic is an individual inputted free-shape string, generally demonstrating a purchaser’s home region on their profile.</p><p>A fundamental investigation showed that around 2% of the tweet objects totalled contained a non-invalid geotagged trait. This gave a solid source to decide local reliance, but the volume of geotagged tweets acquired was deficient to separate any significant patterns. Roughly 97% of tweet objects were found to contain a non-invalid area characteristic, where a client’s home area could be derived. Trivial travel in the UK halted on March sixteenth, 2020, close by the primary public lockdown which started on March 23rd, 2020. This implied that most of the UK populace were fixed somewhat in their day-to-day developments. Consequently, a heuristic methodology was taken where the area of a client was surmised by means of their profile area. More modern strategies would be required to work on the unwavering quality of client area to lead further examination on shifting degrees of area granularity, whether that be for the size of a town, city, or region. A methodology that might be valuable in this setting could be to investigate different approaches by which a client’s area could be gathered or removed from their organization of companions or the printed content of a subset of their tweets. The complete volume of tweets matching a specific question can be recovered each moment, hour, or day through the count’s endpoint of the Twitter API.</p><p>This gives a period series conveyance of the recurrence of a specific keyword(s) or hashtag(s) inside a given area. Utilizing on the web hashtag trackers, it is found that the most often utilized hashtags and watchwords throughout the pandemic in the UK (February first — August nineteenth, 2020) were #covid and #coronavirus, where hashtags are regularly used to give setting and sort the topic or normal patterns inside a tweet. Side effect related catchphrases were likewise recognized as high recurrence use inside tweets, such watchwords included hack, fever, and side effect. Accordingly, questions were made against the Twitter API for tweets containing these hashtags and catchphrases, zeroing in on tweets situated in the urban communities of London, Manchester, Liverpool, Birmingham, and Leicester, between February first — August nineteenth, 2020. These areas were picked to research the main significant top in the episode in the four biggest urban communities in the UK considering populace size (London, Manchester, Liverpool, and Birmingham).</p><p>The city of Leicester was additionally picked, which saw a critical second top if numbers during June 2020. It was quite compelling as it was perhaps the earliest city in England to have a second wave while local area testing was altogether increase. Thusly, there was an interest to check whether this subsequent episode could be seen in Twitter information. Tweets were likewise sifted to be of English language and retweets were discredited. A comparative methodology was taken to decide the most well-known catchphrases utilized with regards to masks, where the most often utilized watchwords were facial covering and mask. To fabricate a corpus of tweets, the information endpoint of the Twitter API was utilized. This endpoint returns a limit of 500 tweet objects for every API demand matching a specific question, where each tweet object contains the printed content of the tweet and client metadata. To test the printed content of COVID-19 related tweets spreading over the span of the primary significant flare-up in the UK, the information endpoint was questioned for UK starting tweets containing #covid OR #coronavirus OR Coronavirus OR Covid, as well as a different inquiry containing facial covering OR mask. The time frame of the API question was organized to acquire a limit of 500 tweets each hour and every day between 1st of February — 19th of August 2020.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/845/1*v9_fB7M5ag7fMhzca--KGg.png" /><figcaption>Figure 1 The data is collected from the Twitter API to use the workflow to acquire, store and process</figcaption></figure><p>Figure 1 The data is collected from the Twitter API to use the workflow to acquire, store and process</p><h3>I. Resources Collection Details</h3><p>The data is collected from</p><p>Coronavirus tweets NLP — Text Classification (<a href="https://www.kaggle.com/datasets/datatattle/covid-19-nlp-text-classification">https://www.kaggle.com/datasets/datatattle/covid-19-nlp-text-classification</a>)</p><p>The hardware: Google Cloud Platform</p><p>The software: Anaconda with Jupyter- The Python data science platform</p><h3>1. Storage</h3><p>The Covid 19 Tweet dataset was taken from the following link <a href="http://insideairbnb.com/get-the-data.html">http://insideairbnb.com/get-the-data.html</a> which was provided by the Big Data module assignment and the following .ipynb files were downloaded to the computer. The downloaded .ipynb files were,</p><ol><li>Corona_NLP_train.csv</li></ol><p>2. Corona_NLP_test.csv</p><p>After that, the downloaded csv files were uploaded to the Jupyter Notebook to start the analysing and getting the final conclusion.</p><p>The following screenshots (figures) are included the information about storage under the Jupyter Notebook.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vhXuMigdXOb1K8KR_OFt_g.png" /><figcaption>Figure 2 Uploaded two .csv files under the Covid 19 Tweet Dataset to the Jupyter Notebook</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p7YK8FiVm2RNy2O10NmXbA.png" /><figcaption>Figure 3 Created new Jupyter Notebook as Student ID 210164938 (Python 3- ipykernel)</figcaption></figure><p>In figure 3 is displayed the student.ipynb file which is used to evaluate the output using the appropriate coding with analysing requirements.</p><h3>1. Analysis</h3><h3>A. Data Wrangling</h3><p><strong>1.</strong> <strong>The some of libraries are installed before data importing and loading libraries.</strong></p><p>The required packages like textblob, Adam, gensim and textstat are installed via !pip install before importing libraries to start the coding. The installed libraries are <strong><em>teststat</em></strong>, <strong><em>genism</em></strong> &amp; <strong><em>Adam.</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1015/1*FWKa-FahUW94l0GsrgT3zw.png" /><figcaption>Figure 4 Installed libraries</figcaption></figure><p>After the appropriate libraries are successfully downloaded and installed, the required libraries are imported in the relevant code sections to get the output for analysing purposes and the rest of the requirements to be done.</p><p><strong>2.</strong> <strong>Loading the data from test.csv and train.csv files.</strong></p><p>The two objects are used to load the data to display the information/ data. The used objects are,</p><ol><li>The <strong><em>train</em></strong> <a href="#_ftn1">[1]</a>object is for <em>Corona_NLP_train.csv file</em> and</li></ol><p>2. The <strong><em>test </em></strong><a href="#_ftn2">[2]</a>object is for <em>Corona_NLP_test.csv</em> <em>file</em>.</p><p><a href="#_ftnref1">[1]</a> Corona_NLP_train.csv file</p><p><a href="#_ftnref2">[2]</a> Corona_NLP_test.csv file</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0gPuAuOuGznmUX1eripx0w.png" /><figcaption>Figure 5 Load Data</figcaption></figure><p><strong>3.</strong> <strong>Print the details like shape, memory usage of train and test csv files using the format() function and</strong> <strong>get the train information vis info() function as well.</strong></p><p>Under this it is displayed the total columns which is included and displayed the non-value records in the separate column as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/815/1*5TGt2kFvOPRGYo0AytfK5Q.png" /><figcaption>Figure 6 Format Details</figcaption></figure><p><strong>4.</strong> <strong>Drop the duplicates values.</strong></p><p>To check the duplicated values are inside the data frame and if there’s any duplicate value inside the data frame, then it will be removed from the data frame.</p><p>As per the mentioned the objects that used to called two csv files such as <strong><em>train</em></strong> object is for <em>Corona_NLP_train.csv file and </em><strong><em>test</em></strong> object is for <em>Corona_NLP_test.csv file.</em> At the very first the duplicated values are dropped inside the Corona_NLP_train.csv file (data frame).</p><p>After the drop_duplicate() function called, there are 44955 rows and 6 columns under the Corona_NLP_train.csv file after the dropped duplicated rows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pdvladNceXrsRbh-NUC79Q.png" /><figcaption>Figure 7 Drop the duplicates</figcaption></figure><p><strong>5.</strong> <strong>Get the null values of each column</strong></p><p>Under this section, it is returned the number of missing values in the dataset.</p><p>In the Location column is having 9424 total missing values and 0.209632 precent missing and none of the other columns are not having any missing value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SzG1u8Slv7ERM-GMu9Ryzg.png" /><figcaption>Figure 8 Null values of each column</figcaption></figure><p><strong>6.</strong> <strong>Missing data as the white lines</strong></p><p>The <strong>missingno</strong> library is imported to get the matrix ratio to get the missing values in Location column as white lines to get a clear idea.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mhaLfohTUbeIHbwkJTy5RQ.png" /><figcaption>Figure 9 Missing data</figcaption></figure><p><strong>7.</strong> <strong>Null values in Location column</strong></p><p>Inside the location column there are 44955 total tweets and 44955 total unique users in this data set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HAFsfBFFCseYegUD9TwqTQ.png" /><figcaption>Figure 10 Null Values in location</figcaption></figure><p><strong>8. NaN values in Location column</strong></p><p>In the first 5 records are having two NaN values in the <strong><em>location</em></strong> column.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*jyh6Xz_rMyLLFyiZJXDtdA.png" /><figcaption>Figure 11 NaN values</figcaption></figure><h3>B. Descriptive Analytics</h3><p><strong>1.</strong> <strong>To compare with the original text the separate column is created for the text.</strong></p><p>Since the both Corona_NLP_test.csv and Corona_NLP_train.csv files are having five different sort of sentiments classes like Extremely Negative, Extremely Positive, Negative, Positive and Neutral are sorted into 3 classes ,</p><ol><li><strong><em>Extremely Positive</em></strong> and <strong><em>Positive</em></strong> classes as one class called <strong><em>Positive</em></strong> Class</li><li><strong><em>Extremely Negative</em></strong> and <strong><em>Negative </em></strong>classes as one class called <strong><em>Negative</em></strong> Class</li><li><strong><em>Neutral</em></strong> Class</li></ol><p>Once the five classes are converted into three classes, then normalize the value count of the each classes separately by converting integer values into string values using <strong>astype(str).</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hNZa3uP9WWzY8lhBxOYDtQ.png" /><figcaption>Figure 12 To compare with the original text the separate column is created for the text.</figcaption></figure><p><strong>2.</strong> <strong>Descriptive statistical summary data</strong></p><h3>I.<strong> Get the count</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*suOWP_eb3bGjEZxAoC4EzQ.png" /><figcaption>Figure 13 Get the count</figcaption></figure><p>The total count of different sort of sentiment of Covid 19 Tweet are, <strong>Positive sentiment: 19592, Negative sentiment: 17031 and neutral sentiment: 8332. </strong>To get further understanding purposes, the visualization of all the sentiments is displayed as a percentage in a pie chart using the plot library as well. The 43.6% are having positive sentiments of Covid 19 Tweet, 37.9% from Negative and the rest of the percentage 18.5% from Neutral as per the Covid 19 Tweet according to the Corona_NLP_test.csv and Corona_NLP_train.csv files in the given dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/953/1*VnKFi-9fea3CzM-_REQKZw.png" /><figcaption>Figure 14 Class Distribution</figcaption></figure><h3>II.<strong> Get the number of characters of each positive, negative, and neutral tweets</strong></h3><p>According to the histogram, which is having in below, is described the quantity of characters that used for each sentiment.</p><p><strong>According to the histogram of Positive Sentiments,</strong></p><p>The maximum frequency level of Positive Sentiments is above 4000 according to the 250 to 300 characters. Explaining about the shape of the Positive Sentiment histogram,</p><p>from the beginning value of 0 to 275 is having left skewed shape which is having the “tail” on the left of the distribution also known as “negatively” skewed as well. After the value of 275 is having long down to the bottom so the conclusion of the shape of the Positive sentiment histogram is having Random shape since the pattern is not crystal clear to identify.</p><p>The minimum frequency level of Positive characters is near to 100 according to the 0 to 50 characters are used.</p><p><strong>According to the histogram of Negative Sentiments,</strong></p><p>The maximum frequency level of Positive Sentiments is above 4000 according to the 250 to 300 characters. Explaining about the shape of the Positive Sentiment histogram,</p><p>from the beginning value of 0 to 275 is having left skewed shape which is having the “tail” on the left of the distribution also known as “negatively” skewed as well. After the value of 275 is having long down to the bottom so the conclusion of the shape of the Negative sentiment histogram is having Random shape since the pattern is not crystal clear to identify.</p><p>The minimum frequency level of Negative characters is below 1000 according to the 0 to 50 characters are used.</p><p><strong>According to the histogram of Neutral Sentiments,</strong></p><p>The maximum frequency level of Neutral Sentiments is above 1200 according to the 100 to 150 characters. Explaining about the shape of the Neutral Sentiment histogram, from the beginning it’s having a random skewed shape.</p><p>The minimum frequency level of Neutral sentiments, the characters are around 100 according to the 10 to 50 characters are used.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/889/1*xU17lylr71kxPD2UFDGDeg.png" /><figcaption>Figure 15 Number of characters</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*7pE1ptgkB3cA1CiOLwZo1Q.png" /><figcaption>Figure 16 Number of words in tweet</figcaption></figure><h3>III<strong>. Get the average word length in tweet</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/931/1*LxgRjCqCiA5KxWMH4aGB5Q.png" /><figcaption>Figure 17 Average word length</figcaption></figure><p>The average word length of positive, negative, and neutral sentiments is having different sort of density with texts. Each and every sentiment is described separately as per the below mentioned.</p><p>In the Positive sentiment,</p><p>Around 10 to 15 texts are having the maximum average word length that the users considered are around 0.30 to 0.35 which is nearly 0.34 density, and the density level is decreased as a direct flow to the 0 density level for the text from 15 to 290 as well.</p><p>In the Negative sentiment,</p><p>Around 0 to 5 texts are having an incremental flow line of maximum average word length that the users considered are above 0.35 which is nearly 0.38 density, and the decremental flow line is raised to the 5 in the texts and to the 0 density level for the texts from 10 to 80 as well.</p><p>In the Neutral sentiment,</p><p>Around 0 to 10 texts are having an incremental flow line of maximum average word length that the users considered are above 0.23 which is nearly 0.25 density, and the decremental flow line is raised to the 10 in the texts and to the 0 density level for the texts from 15 to 80 as well.</p><h3>C. Diagnostic Analytics</h3><ol><li><strong>Plot the wordcloud image for stopwords</strong></li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z8zRrRL1iEDMQGzsUoc0JA.png" /><figcaption>Figure 18 Wordcloud for stopword code and output</figcaption></figure><p>The wordcloud is used to demonstrate the most used stop words which are the words that mentioned recently on the tweet. The most used stop words are <strong>Needn’t, Mustn’t, Won’t</strong> and secondarily used stop words are <strong>didn’t, should’ve, doesn’t haven’t</strong> and <strong>mightn’t</strong> in the Covid 19 Tweet dataset. According to the analysing most of the Twitter users are used stopwords like as per the above mentioned.</p><p><strong>2. Wordclouds for Negative, positive and neutral tweets</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1003/1*nSyQ6PsJR9OkBPFMxWkRUw.png" /><figcaption>Figure 19 Wordcloud for Negative, positive and neutral tweets</figcaption></figure><p><strong>According to the wordcloud of Positive Sentiment,</strong></p><p>The most of the Twitter users are used corona virus, hand sanitizer, consumer, customer, grocery store, help, covid, thank, home and etc seems like the most of the Twitter users are mentioned about the facilities which are provided to the customer from supermarket due to the pandemic situation. The facilities like providing sanitizer to the customer at the entrance, online shopping while staying safe at home, work from home, maintain the social distance. While the employees are getting fired due to the pandemic situation and the government is offered money, food and also hope to the families who lost their jobs due to the pandemic situation issues like quarantine, economic unbalance of the companies as well. Due to the pandemic situation, the amazon is increased the quantity of online customers for the purpose of maintaining social distance. The customers get more facilities to shop online while they are staying at home without putting their lives into danger because of the Covid 19.</p><p><strong>According to the wordcloud of Negative Sentiment,</strong></p><p>The most of the Twitter users are used Covid, people, oil price, grocery store, stock, corona virus, supermarket, panic, buying, gas price, stay home, stock food, home, toilet paper, social distancing and seems like people are disappointed about being stay at home without having any social gathering or physical community in between the relevant parties. At the same time people are getting panic due to the corona virus spreading today than yesterday. As a result of that majority of the people are bought toilet papers more than they want. Therefore, the lack of toilet papers issue is raised due to the panic situation. At the same time scams are raised to take money because of people staying home while loosing their jobs that they used to do before the pandemic. At the very first lockdown is issued more problems to the people since it’s a new version of experience to the people who live all over the world.</p><p><strong>According to the wordcloud of Neutral Sentiment,</strong></p><p>The most of the Twitter users are used online shopping, covid, business, restaurant, stock, consumer, supermarket, toilet papers, corona virus, covid, grocery shopping, covid, pandemic, worker, read, people, customer, panic, buying, crisis, social distancing, stay at home and seems like people having neutral feelings about the main topics like social distancing, online shopping, work from home, customer needs supplying through online as a new business platform, the videos about corona virus are updated the people day by day including the social distancing, economic, crisis and safety as well. At the same time they are having bad perspective of buying a lot of toilet papers from grocery store and super market because of having panic due to the pandemic situation. Some of the people are satisfied for being at home and doing their work as a work from home facility while some of the people are having bad perspective of being at home while working at home without having any community with others and need to maintain the social distancing to stop spreading corona virus.</p><p><strong>03. Scatterplot</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KiNGvztBZBr54DUZxe4GcA.png" /><figcaption>Figure 20 ScatterPlot</figcaption></figure><p>According to the figure 20. Scatter plot, the most sentiments reviews are shared on Twitter by Asian countries like Japan, Hon Kon, Israel, Taiwan, and Korea. Apart from that, people who live in Europe countries like Spain, United Kingdom, Germany, and Finland are posted sentiment Covid 19 tweets in Twitter less than Asian countries but more than other African, Americas and Oceania countries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/734/1*v5p124qs_L18nK-PPh2hIQ.png" /><figcaption>Figure 21 Number of listings by sentiment</figcaption></figure><p>According to the figure 21, the most sentiment reviews are about Positive sentiments which is having around 19,000 tweet posts in Twitter. And the quantity of Negative reviews are less than Positive sentiments but more than neutral sentiments. There’re around 17,000 tweets are in Twitter regarding Negative sentiments and apparently neutral sentiments are having around 8,000 tweets in Twitter as well.</p><h3>D. Predictive Analytics</h3><h3>I. Decision tree classifier and getting the accuracy</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9SZv5YBV-TgbRcqhm8_akw.png" /><figcaption>Figure 22 Decision Tree classifier with model accuracy</figcaption></figure><p>According to the figure 20 which is about decision tree classifier and predicted responses of test dataset are displayed the accuracy of the decision tree algorithm is having 56.66% . That means the value is high usually considered as good.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*Krc1N66dzAn79MEHqLjJwQ.png" /><figcaption>Figure 23 Decision Tree</figcaption></figure><p>According to the Decision tree chart,</p><p>the internal nodes are interconnected and having the decision rule to split the specific data in Covid 19 Tweet sentiments. In this, the Gini is called as the Gini Ratio that is measured the impurity of entire nodes in the decision tree. Even if the node is pure then the entire records which are inside it is considered as owned to the same class. Apart from that, rest of the nodes are considered as leaf nodes. Based on the output, the possibility rate of interpreting the decision tree is difficult due to the output’s complexity. Either pruning or shortening of a tree is required to facilitate the understandability of the output and the optimise as well. The optimization is contained three options to operate it.</p><p>01. Criterion: Optional(default=gini)- Selection attributes are measured to choose.</p><p>02. Splitter: string OR optional(default=best) — split the strategy</p><p>03. Max_depth:int OR none, optional(default= none)- The tree maximum depth</p><h3>II. Random forest classifier</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ApXag7ykSLnQ167XwPqTIQ.png" /><figcaption>Figure 24 Random Forest classifier</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c27fff95c13" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>