Natural Language Processing

Techniques, Applications, and Practical example (Sentiment analysis of the 2023 Nigerian general elections)

What is Natural Language Processing ?

This is a field of study, under data science and artificial intelligence that helps machines interpret, understand and manipulate human language, the way a human being would.

It uses techniques such as data mining, statistical modelling, machine and deep learning models, to extrapolate meaning out of structured, semi-structured and unstructured data.

As mentioned above there are 3 major types of data NLP models work with;

Structured data: This is data that is properly organized, stored in well-defined formats, and is easy to read and understand. E.g. Excel files, SQL databases etc.

Semi-structured data: This is data that does not conform to a particular format, cannot be stored in rows and columns, and uses metadata (tags and elements) as some form of structure. E.g. emails, web pages, markup languages etc.

Unstructured data: This is data that is not organized in a pre-defined model, therefore cannot be stored in a structured/relational database. E.g. Multimedia files, social media content etc.

NLP Techniques

There are many techniques used in NLP, and an even larger amount of research going into NLP today. Some of its examples are:

1. Sentiment Analysis: This is the process of categorizing opinions in text format, as either positive, negative, or neutral.

This technique is used by businesses to measure customer feedback, gain market insight, and identify areas where customers are not satisfied, which can further be used for market research and product development.

It can also be used by government agencies to identify ineffective public policies and dysfunctional public services.

2. Speech and Voice Recognition: This process allows machines to identify spoken words and convert them to text.

While speech and voice recognition sound similar, there is a difference in the application of these two methods. Speech Recognition is the ability to recognize any speech regardless of language or accents, voice recognition is the ability to recognize a specific users voice. They can be used in A.I assistant, robots etc.

3. Language Detection and Machine Translation: Although these are two separate techniques, they are sometimes used together. Language detection is the process of identifying the language written in a piece of text, while Machine translation interprets it into the desired language of choice.

For businesses operating across multilingual ecosystems, language detection and translation is essential for reaching a larger audience, interpreting data and preparing data for more advanced NLP analysis.

4. Text summarization: Text summarization is a procedure that involves extracting important information from a long complex publication, breaking it into short paragraphs or sentences, and creating a brief, concise summary.

In financial research, text summarization can be used to process long financial reports, and quickly extract the most important information.

5. Web scraping: This is the process of collecting data from the internet via web pages.

It can be used to monitor product prices, news sources, and other relevant information stored on websites, in real-time.

Real-Life Applications

A combination of one or many of these techniques is used to create value in the real world. Below are five examples:

1. Conversational A.I: This is the art of simulating human-like conversations with users/customers, using NLP and A.I. Two examples are chatbots and virtual assistants.

2. Fraud/Crime detection: With advanced NLP analysis and machine learning, we can easily flag fraudulent and other criminal kinds of communications in online conversations. Governments can also use this to protect their countries from potential security threats.

3. Branding and Marketing: NLP helps businesses increase revenue by targeting advertisement to users who have shown interest in similar products via their social media posts, offering a range of options for them to try. Additionally, by using sentiment analysis, businesses can detect negative reviews and identify areas where they need to improve their products.

4. Grammar Checkers: This is an implementation of NLP that helps find grammatical errors in a text then correct them.

5. Recruitment: NLP simplifies the tedious tasks involved in hiring and recruiting top talent by automating the entire process using chatbots and automated evaluation of candidate resumes.

Practical Example:

Sentiment analysis of the 2023 Nigerian general elections using python:

Although the 2023 Nigerian general elections were held on February 25th and 26th, the hashtag #NigerianGeneralElections continued to trend for over a week. This was because the National Collation Center took until March 2nd, 2023 to compile and announce the final results of the election.

The goal of this analysis is to analyze Twitter users’ sentiment on the election, by categorizing all tweets with the hashtag #2023GeneralElections between 25–02–2023 12:00am and 28–02–2023 11:59pm as either positive or negative.

Methodology:

· Import relevant libraries

· Data scraping

· Data preprocessing

· Sentiment analysis

1. Import relevant libraries

Python libraries are reusable modules of code, that help developers create solutions quickly and easily.

import pandas as pd
import numpy as np
import datetime
import matplotlib.pyplot as plt
import seaborn as sns
import plotly.express as px

import nltk
import re
import string
from nltk.corpus import stopwords
from nltk.sentiment.vader import SentimentIntensityAnalyzer
from nltk.sentiment.util import *
from matplotlib import ticker

from transformers import AutoModelForSequenceClassification, AutoConfig
from transformers import TFAutoModelForSequenceClassification
from transformers import AutoTokenizer
import numpy
from scipy.special import softmax
import tensorflow as tf

2. Scrape twitter data using the snscrape library.

Get Ovbude Ehi’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Specify the request parameters:

• Input the hashtag and the time frame into the “query” variable.
• Specify the number of tweets you would like to scrape by using the “limit” variable.
• Create an empty list to store the tweets.

# Required Library
from snscrape.modules import twitter as scraper

# Request Parameters
query = "(#2023GeneralElections) until:2023-03-01 since:2023-02-25"
tweets = []
limit = 100000

Create a for loop to scrape the required number of tweets, then save as csv:

for tweet in scraper.TwitterHashtagScraper(query).get_items():
    
    # Stop scraping if it has reached the required number of tweets desired
    if len(tweets) == limit:
        break
    # Srape the following data from the tweet   
    else:
        tweets.append([tweet.date, tweet.url, tweet.user.username, tweet.sourceLabel, tweet.user.location,
                       tweet.content, tweet.likeCount, tweet.retweetCount,  tweet.quoteCount, tweet.replyCount])


        
df = pd.DataFrame(tweets, columns=['Date', 'TweetURL','User', 'Source', 'Location', 'Tweet', 'Likes_Count',
                                   'Retweet_Count', 'Quote_Count', 'Reply_Count'])
# Convert and save as csv
df.to_csv('Data/2023Elections.csv')

I intended to scrape as many as one hundred thousand public tweets between February 25, 2023, and March 1, 2023, but was limited to only 6208 tweets made between February 25, 2023, at 12:00am and February 28, 2023, at 11:59pm.

3. Data preprocessing:

This is the most important part of this analysis. Below are the various steps taking to prepare the dataset for the classifier:

Parse date: This is the process of converting a “date” feature often represented as an object/string data type, into a datetime format. This is essential to allow python recognize the date as composed of time, day, month and year.

Replace special characters: Tweets are composed of different characters such as emojis, hashes, asperands and links. To obtain only the words from these tweets, we need to remove these characters.

To do this, we replace these characters with empty spaces “ ”. I created a loop and a function to solve this problem.

# For loop to replace tweet's special characters with ""
special_char = ["~", "|", "-", "\n", "`", "!", "@", "#", "$", "%", "^", "&", "*", "(", ")","|", "-", "_", "+", "=", "{", "}", "[", "]", "|", str('\\'), "/", ':', ';', '"', "'", "<", ">", ",", '.', '?']
for item in special_char:
    data["Tweet"] = data["Tweet"].str.replace('item', ' ')

# Function to replace other special characters like links etc.
def clean_text(text):
    text = re.sub(r"@[A-Za-z0-9]+", ' ', text)
    text = re.sub(r"https?://[A-Za-z0-9./]+", ' ', text)
    text = re.sub(r"[^a-zA-z.!?'0-9]", ' ', text)
    text = re.sub('\t', ' ',  text)
    text = re.sub(r" +", ' ', text)
    return text

data['Tweet'] = data['Tweet'].map(lambda x: clean_text(x))

Before:

After:

Remove stop words: Stop words are a list of words used to form a sentence, such as articles, prepositions, conjunctions, and pronouns, that have little or no meaning.

However, removing stop words is not always a good idea. When using a pre-trained model, it is essential that you have an idea of how the data used to train the model was preprocessed in order to get the best results from the model.

The pre-trained model I will be using, was trained to predict tweets even with stop words.

I did not remove stop words for this sentiment analysis, but here’s how to remove them:

# Function to remove stopwords
def remove_stopwords(text):
    words = [word for word in text.split() if word.lower() not in stopwords.words('english')]
return ' '.join(words)

4. Sentiment Analysis

The model we will be using is the ‘twitter-roberta-base-sentiment-latest’, developed by the University of Cardiff’s NLP group, and it is trained on 58 million tweets.

# Initialize necessary objects
MODEL = f"cardiffnlp/twitter-roberta-base-sentiment-latest"
tokenizer = AutoTokenizer.from_pretrained(MODEL)
config = AutoConfig.from_pretrained(MODEL)
model = AutoModelForSequenceClassification.from_pretrained(MODEL)

# Features we are interested in
rb_data = cleaned_data[["Date", "TweetURL", "Tweet", "Likes_Count", "Retweet_Count"]]
rb_data.head()

Create a function to predict the labels:

Below are the various steps taking:

1. Function takes a tweet as input.
2. Encodes it using a tokenizer.
3. Passes the encoded input into the pre-trained classifier and returns an output.
4. Extracts scores for each sentiment class (Positive, Negative and Neutral) from the pre-trained model output.
5. Applies the softmax function to get a probability distribution of the sentiment classes.
6. Ranks the scores in descending order of their probabilities.
7. Returns the sentiment label of the score, with the highest probability.

def sentiment_labels(text):
    # Encodes the input text using a tokenizer
    encoded_input = tokenizer(text, padding=True,truncation=True,max_length=512, return_tensors='pt')
    
    # Gets the output of the model
    output = model(**encoded_input)

    # Gets scores of each sentiment class
    scores = output[0][0].detach().numpy()

    # Get probability distribution of scores
    scores = softmax(scores)

    # Sort the scores and return their indices
    ranking = np.argsort(scores)

    # Rank the indices in descending order
    ranking = ranking[::-1]

    # Return the sentiment label of the index with the highest probability
    return config.id2label[ranking[0]]

Predict the labels:

This block of code, predicts the sentiment labels for each tweet using the sentiment_labels() function, stores the predicted labels in a list, and adds the list to the dataframe as predictions.

rb_label = []

# Predict the labels
for text in rb_data["Tweet"]:
    rb_sent_label = sentiment_labels(text)
    rb_label.append(rb_sent_label)

# Add predictions to dataframe
rb_data["label"] = rb_label

Check the Distribution of the labels

rb_data["label"].value_counts()

Result:

neutral     3563
negative    1711
positive    920 
Name: label, dtype: int64

Conclusion

Based on this analysis, we can conclude that the 2023 Nigerian General Elections received more negative reviews than positive ones, with a ratio of 1.86 negative reviews for every 1 positive review.