Tweets Preprocessing
Learn to preprocess textual data using Python
As a budding Data Science-Machine Learning Enthusiast, the initial journey might be overwhelming!!!. Hey! you are not alone. Anything worthwhile is always intriguing and scary at the same time. Let’s elevate feelings a little bit!
Preprocessing forms important aspect in the data domain. This article deals with preprocessing of tweets. The tweets I have used is from pharmaceutical domain. Generalize to the data you wish to!
Let’s import important libraries.
import spacy
import pandas as pd
import numpy as np
import seaborn as sns
import tensorflow as tf
import os
import nltk
import textsearch
from wordcloud import WordCloud, STOPWORDS, ImageColorGenerator
from nltk.util import ngrams
from collections import Counter
import csv
import matplotlib.pyplot as plt; plt.rcdefaults()
from nltk.corpus import stopwords
import warnings
warnings.filterwarnings("ignore",category=DeprecationWarning)Let’s initialize the folder folder path for easy access and load the dataset. As the dataset is in json format we load it through json.load.
folder_path = '/content/drive/MyDrive/Projects/{your folder path}/'
import json
with open(folder_path+'tweets.json') as jfile:
d = json.load(jfile)Display the Data.
df= pd.DataFrame.from_dict(d)
df = df.T
df.head()
df.shape It has 43k rows and 2 columns
(43347, 2)The rough view of our tweets (using markdown cell) seemed complicated.
It’s observed that data contains tweets in different languages. Let’s examine the languages present.
Here I have used ‘langdid’ library. Do dive into other options as well. Steps followed are as follows:
- Get English language id.
- Calculate the percentage of English language with respect to other languages.
# examine the percentage of other language
import langid
# get language id for each text
ids_langid = df['tweet_text'].apply(langid.classify)
print(ids_langid[0])
# get just the language label
langs = ids_langid.apply(lambda tuple: tuple[0])
# unique languages present
print(f"no of unique languages estimated: {len(langs.unique())}")
print('unique languages:')
print(langs.unique())
print(f"percentage of English: {(sum(langs=='en')/len(langs))*100}")
- Thus as 91% of the data is in English we will ignore other language tweets.
Assign the language label as a seperate column in dataframe and filter only English tweets.
# Assign the language to dataframe
df['language'] = langs
# query for only english tweets
df_e = df[df['language'] == 'en']
df_e.shapeOutput:
(39313, 3)
Remove duplicate tweets if any.
# removing duplicate tweets
df_e = df_e.drop_duplicates()Let’s create function to visualize data time to time to verify the preprocessing and visualize nature of different data.
def tweet_visualizing(data, m,n):
for i in range((n-m)+1):
print('-',m+1,'------------------------------------------------------------------------------------------')
print(data[m])
m = m+1
tweet_visualizing(df_e['tweet_text'], 0, 10)After visualizing few range of tweets, we observe that:
- Tweets are repeated(promotions and announcements) with different links associated with them. Thus drop_duplicates() becomes less effective.
- Data contains emojis, links, hashtags, organizations.
- Few abbreviations has meaning of adjectives. Thus it’s important to modify them if you have plans of sentimental analysis.
- Thus data need to be preprocessed with careful insights.
We use regex to perform specific tasks.
Get Pooja’s stories in your inbox
Join Medium for free to get updates from this writer.
Non ascii characters.
def remove_non_ascii(text):
return re.sub(r'[^\x00-\x7f]',r' ', text)Special characters
def remove_special_characters(text):
"""
Remove special special characters, including symbols, emojis, and other graphic characters
"""
emoji_pattern = re.compile(
'['
u'\U0001F600-\U0001F64F' # emoticons
u'\U0001F300-\U0001F5FF' # symbols & pictographs
u'\U0001F680-\U0001F6FF' # transport & map symbols
u'\U0001F1E0-\U0001F1FF' # flags (iOS)
u'\U00002702-\U000027B0'
u'\U000024C2-\U0001F251'
']+',
flags=re.UNICODE)
return emoji_pattern.sub(r'', text)Single character function
- Few preprocessing steps result in single characters. Thus it’s important to identify and remove.
def remove_single_char_func(text, threshold=2):
threshold = threshold
words = word_tokenize(text)
texts = ' '.join([word for word in words if len(word)>threshold])
return textsExpand contractions.
- We expand the contractions in order to preserve the meaning.
import contractions
df_e['tweet_text'] = df_e['tweet_text'].apply(lambda x:contractions.fix(str(x))Custom Stop Words
- Here I have used custom stop words. nltk corpus includes stopwords like ‘not’ which upon removal might result in changed meaning of the statement.
custom_stops = ['a','able','about','across','after','all','almost','also','am','among','an','and','any',
'are','as','at','be','because','been','but','by','could','dear','did','do',
'does','either','else','ever','every','for','from','get','got','had','has','have','he','her',
'hers','him','his','how','however','i','if','in','into','is','it','its','just','let','like',
'likely','may','me','might','most','must','my','neither','no','nor','of','often','on',
'only','or','other','our','own','rather','said','say','says','she','should','since','so','some',
'than','that','the','their','them','then','there','these','they','this','tis','to','too','twas',
'us','wants','was','we','were','what','when','where','which','while','who','whom','why','will','with','would','yet','you','your']df_e['tweet_text'] = df_e['tweet_text'].apply(lambda x: " ".join([i for i in x.split() if i not in custom_stops]))
Punctuations
def remove_punct(text):
"""
Remove the punctuation
"""
return text.translate(str.maketrans('', '', string.punctuation))Other common preprocessing steps:
- Converting into lowercase
- Removing @ mentions
- Removing common urls and tags
- Remove multiple spaces
- Removing numbers
# lowercase
df_e['tweet_text'] = df['tweet_text'].str.lower() # remove @ mentions
df_e['tweet_text'] = df_e['tweet_text'].apply(lambda x: re.sub("@[A-Za-z0-9_]+",'', str(x))) # remove url and tags
df_e['tweet_text'] = df_e['tweet_text'].apply(lambda x: re.sub(r'(http|https|ftp|ssh)://([\w_-]+(?:(?:\.[\w_-]+)+))([\w.,@?^=%&:/~+#-]*[\w@?^=%&/~+#-])?', '' , str(x)))#remove multiple spaces
df_e['tweet_text'] = df_e['tweet_text'].apply(lambda x: " ".join(x.split())) # removing numbers
df_e['tweet_text'] = df_e['tweet_text'].apply(lambda x: "".join([i for i in x if not i.isdigit()]))
Hashtags
Hashtags might contain important information. Based on the end goal you can choose to preserve or remove hastags. Here we have stored hashtag as a column and then removed hash symbol from the tweets.
# preserve hastags
df_e['hashtag'] = df_e['tweet_text'].apply(lambda x:re.findall(r'#(\w+)',str(x)))
# remove hash but keep tags
df_e['tweet_text'] = df_e['tweet_text'].apply(lambda x: re.sub("[^A-Za-z0-9_]+"," ", x))Data Specific modifications:
Removing URL
This dataset is medical research based. It has special type of urls. Thus we need to create regex according to the nature of the data in hand.
def remove_URL(text):
"""
Remove URLs from a sample string
"""
return re.sub(r"https://t.co/([A-Za-z0-9_]+)", '', text)This data has organization named ‘NICE’ which might be falsely detected as adjective. Thus ‘NICE’ is changes to ‘NIXX’ for convenience.
df_e['tweet_text'] = df_e['tweet_text'].apply(lambda x: re.sub('NICE', 'NXXX',str(x)))Thus overall preprocessing is included in the below code. One must be careful regarding the order of preprocessing steps. It is worth experimenting. For example, removing punctuation before removal of url might result in breakage of url symbols, resulting in undetected urls and noise.
def remove_non_ascii(text):
return re.sub(r'[^\x00-\x7f]',r' ', text)
def remove_special_characters(text):
"""
Remove special special characters, including symbols, emojis, and other graphic characters
"""
emoji_pattern = re.compile(
'['
u'\U0001F600-\U0001F64F' # emoticons
u'\U0001F300-\U0001F5FF' # symbols & pictographs
u'\U0001F680-\U0001F6FF' # transport & map symbols
u'\U0001F1E0-\U0001F1FF' # flags (iOS)
u'\U00002702-\U000027B0'
u'\U000024C2-\U0001F251'
']+',
flags=re.UNICODE)
return emoji_pattern.sub(r'', text)
def remove_single_char_func(text, threshold=2):
threshold = threshold
words = word_tokenize(text)
texts = ' '.join([word for word in words if len(word)>threshold])
return texts
def remove_punct(text):
"""
Remove the punctuation
"""
return text.translate(str.maketrans('', '', string.punctuation))
# remove_custom_stopwords:
custom_stops = ['a','able','about','across','after','all','almost','also','am','among','an','and','any',
'are','as','at','be','because','been','but','by','could','dear','did','do',
'does','either','else','ever','every','for','from','get','got','had','has','have','he','her',
'hers','him','his','how','however','i','if','in','into','is','it','its','just','let','like',
'likely','may','me','might','most','must','my','neither','no','nor','of','often','on',
'only','or','other','our','own','rather','said','say','says','she','should','since','so','some',
'than','that','the','their','them','then','there','these','they','this','tis','to','too','twas',
'us','wants','was','we','were','what','when','where','which','while','who','whom','why','will','with','would','yet','you','your']
def remove_URL(text):
"""
Remove URLs from a sample string
"""
return re.sub(r"https://t.co/([A-Za-z0-9_]+)", '', text)
# function to calldef basic_preprocessing_tweets():
df_e['tweet_text'] = df_e['tweet_text'].apply(lambda x: re.sub('NICE', 'NXXX',str(x)))
# lowercase
df_e['tweet_text'] = df['tweet_text'].str.lower()
# remove @ mentions
df_e['tweet_text'] = df_e['tweet_text'].apply(lambda x: re.sub("@[A-Za-z0-9_]+",'', str(x)))
# remove special urls and tags
df_e['tweet_text'] = df_e['tweet_text'].apply(lambda x:remove_URL(str(x)))
# remove url and tags
df_e['tweet_text'] = df_e['tweet_text'].apply(lambda x: re.sub(r'(http|https|ftp|ssh)://([\w_-]+(?:(?:\.[\w_-]+)+))([\w.,@?^=%&:/~+#-]*[\w@?^=%&/~+#-])?', '' , str(x)))
# preserve hastags
df_e['hashtag'] = df_e['tweet_text'].apply(lambda x:re.findall(r'#(\w+)',str(x)))
# remove hash but keep tags
df_e['tweet_text'] = df_e['tweet_text'].apply(lambda x: re.sub("[^A-Za-z0-9_]+"," ", x))
#remove multiple spaces
df_e['tweet_text'] = df_e['tweet_text'].apply(lambda x: " ".join(x.split()))
# expand contractions
df_e['tweet_text'] = df_e['tweet_text'].apply(lambda x:contractions.fix(str(x)))
# remove non ascii characters
df_e['tweet_text'] = df_e['tweet_text'].apply(lambda x:remove_non_ascii(x))
# removing numbers
df_e['tweet_text'] = df_e['tweet_text'].apply(lambda x: "".join([i for i in x if not i.isdigit()]))
# remove punctuation
df_e["tweet_text"] = df_e["tweet_text"].apply(lambda x: remove_punct(x))
# # remove single character
df_e['tweet_text'] = df_e['tweet_text'].apply(lambda x: remove_single_char_func(x))
# # remove selected stopwords
df_e['tweet_text'] = df_e['tweet_text'].apply(lambda x: " ".join([i for i in x.split() if i not in custom_stops]))
# # # #remove multiple spaces
df_e['tweet_text'] = df_e['tweet_text'].apply(lambda x: " ".join(x.split()))
Call the function:
basic_preprocessing_tweets() Initial Sample Data:
⚕️ Scientists conducted a Phase II study of acalabrutinib in patients with relapsed/refractory #CLL who were ibrutinib-intolerant, and found an overall response rate of 73%. https://t.co/eJ6m4QpC5P https://t.co/kuZz6ZO47r
After Preprocessing:
scientists conducted phase study acalabrutinib patients relapsed refractory cll ibrutinib intolerant found overall response rate
Congrats! Thus ends the preprocessing journey. Let’s explore and visualize the data in Part 2. Until then enjoy the data!
References:
