Processing text data for Machine Learning
What is NLP?
Natural Language Processing (NLP) is a field of artificial intelligence (AI) concerned with the interaction between computers and humans using natural language. It is the technology used to help machines to understand and learn text and language. It encompasses a variety of tasks, ranging from simple tasks like language translation and sentiment analysis to more complex tasks like text summarization, question answering, and language generation. It is used to apply machine learning algorithms to text and speech.
Key components of NLP include:
Tokenization: Breaking text into individual words or tokens.
Part-of-Speech (POS) Tagging: Assigning grammatical tags (e.g., noun, verb, adjective) to words in a sentence.
Named Entity Recognition (NER): Identifying and classifying entities mentioned in text, such as people, organizations, or locations.
Parsing: Analyzing the grammatical structure of sentences to understand their meaning.
Sentiment Analysis: Determining the sentiment or opinion expressed in text (e.g., positive, negative, neutral).
Machine Translation: Translating text from one language to another automatically.
Text Summarization: Generating concise summaries of longer texts.
Question Answering: Providing accurate answers to questions posed in natural language.
To convert string data into numerical data one can use following methods
· Bag of words
· TFIDF
Text Preprocessing Raw data contain numerical value, punctuation , special character etc. These value can hamper the performance of model so before applying any text featurization first we need to convert raw data into meaningful data which is also called as text preprocessing . This can be done by following ways:
1.Remove Noisy Data
In regular sentences Noisy data can be defined as text file header, footer, HTML , XML, markup data. As these type of data are not meaningful and does not provide any information so it is mandatory to remove these type of noisy data. In python HTML,XML can be removed by BeautifulSoup library while markup,header can be removed by using regular expression
2 .Tokenization
In tokenization we convert group of sentence into token . It is also called text segmentation or lexical analysis. It is basically splitting data into small chunk of words. For example- We have sentence — “Ross 128 is earth like planet. Can we survive in that planet?”. After tokenization this sentence will become -[‘Ross’, ‘128’, ‘is’, ‘earth’, ‘like’, ‘planet’, ‘.’, ‘Can’, ‘we’, ‘survive’, ‘in’, ‘that’, ‘planet’, ‘?’]. Tokenization in python can be done by python’s NLTK library’s word_tokenize() function
3.Normalization
Before going to normalization first closely observe output of tokenization. Will tokenization output can be considered as final output? Can we extract more meaningful information from tokenize data ?
In tokenaization we came across various words such as punctuation , stop words(is ,in ,that ,can etc),upper case words and lower case words.After tokenization we are not focused on text level but on word level. So by doing stemming , lemmatization we can convert tokenize word to more meaningful words .
IS IT IMPORTANT TO US?
We can use NLP to create systems like speech recognition, machine translation, spam detection, text simplifications, question answering, autocomplete, predictive typing, sentiment analysis, document summarization and many more. Few examples include:
Google Translate — Language translation applications
Grammarly — Employ NLP to check grammatical accuracy of texts
Personal assistance — Applications such as OK Google, Siri, Cortana, and Alexa.
Keyword search, spelling check, synonyms search.
Queries like product price, location, company names etc. on search engines.
Reading text data & why do we need to clean the text?
Text data can be in a structured or unstructured format.
A structured format has a well-defined pattern whereas unstructured data has no proper structure. In between the 2 structures, we have a semi-structured format which is a comparably better structured than unstructured format.
In python, we have several libraries to work with text.
- Scikit-learn, Keras, TensorFlow — has some text processing capabilities
- NLTK — Natural language toolkit.
- SpaCy — is an industrial strength NLP package with many practical tools in a nice API.
- Other libraries — TextBlob, gensim, Stanford CoreNLP, OpenNLP.
Natural Language Toolkit (NLTK)
(Python Library for Text Processing)
Natural Language Toolkit (NLTK) is a known open-source package in Python which allows us to run all common NLP tasks. It provides easy-to-use interfaces and a suite of text processing libraries valorous steps involved during preprocessing like classification, tokenization, stemming, tagging, parsing, and semantic reasoning.
Need for preprocessing the text data
- Real data — often incomplete, inconsistent, and filled with a lot of noise and errors
- Data Structure — Approx. 90% of the data is unstructured
- Variety — Text can come from a list of individual words, sentences, multiple paragraphs (with and without correct spellings and punctuation)
- Noise — And this data is never clean and consists of a lot of noise.
It is a necessary to preprocess the data before building models and analysis. In simple words preprocessing transforms raw text data into an understandable format for the computer.
Steps For Data Preprocessing
we will code common steps involved in text preprocessing.
1) Lower Case
Converting the text into lower case letters.
sent_0 =sent_0.lower()2) Removing all URLs from data
we are removing URLs(‘http\’) from text using python regular expression.
sent_0 = re.sub(r"http\S+", "", sent_1000)3) Removing all tags from data
For removing tags we use the Beautiful Soup python package.
soup = BeautifulSoup(sent_0, 'lxml')
text = soup.get_text()4) Removing words with numbers from data
we are removing words containing numbers from text using python regular expression.
sent_0 = re.sub("\S*\d\S*", "", sent_0).strip()5)Removing special character from data
Get Kavyasrirelangi’s stories in your inbox
Join Medium for free to get updates from this writer.
we are removing all the special character(‘*,&,#,@…’) from the text.
sent_0 = re.sub('[^A-Za-z0-9]+', ' ', sent_0)6)Decontracting the words
In this step, we are expanding English language contractions using python.
def decontracted(phrase):
phrase = re.sub(r"won't", "will not", phrase)
phrase = re.sub(r"can\'t", "can not", phrase)
phrase = re.sub(r"n\'t", " not", phrase)
phrase = re.sub(r"\'re", " are", phrase)
phrase = re.sub(r"\'s", " is", phrase)
phrase = re.sub(r"\'d", " would", phrase)
phrase = re.sub(r"\'ll", " will", phrase)
phrase = re.sub(r"\'t", " not", phrase)
phrase = re.sub(r"\'ve", " have", phrase)
phrase = re.sub(r"\'m", " am", phrase)
return phrase7) Removing stop words
stop words are words which are filtered out before or after processing of natural language data Though “stop words” usually refers to the most common words in a language, there is no single universal list of stop words used by all natural language processing tools.
nltk.download('stopwords')
eng_stopwords = set(stopwords.words('english'))
8) Stemming
Stemming is the process of reducing a word to its word stem that affixes to suffixes and prefixes or to the roots of words known as a lemma. Stemming is important in natural language understanding (NLU) and natural language processing (NLP).
def stemmer(data):
review_clean_ps = []
ps = PorterStemmer()
for sentance in tqdm(data['review'].values):
ps_stems = []
for w in sentance.split():
if w == 'oed':
continue
ps_stems.append(ps.stem(w))
review_clean_ps.append(' '.join(ps_stems))
data['review']=review_clean_ps
return data9) Lemmatization
Lemmatization is the process of grouping together the different inflected forms of a word so they can be analysed as a single item. Lemmatization is similar to stemming but it brings context to the words. So it links words with similar meaning to one word.
def get_wordnet_pos(treebank_tag): if treebank_tag.startswith('J'):
return wordnet.ADJ
elif treebank_tag.startswith('V'):
return wordnet.VERB
elif treebank_tag.startswith('N'):
return wordnet.NOUN
elif treebank_tag.startswith('R'):
return wordnet.ADV
else:
return 'n'nltk.download('averaged_perceptron_tagger')
nltk.download('wordnet')def lemmatization(data):
review_clean_wnl = []
wnl = WordNetLemmatizer()
for sentance in tqdm(data['review'].values):
wnl_stems = []
token_tag = pos_tag(sentance.split())
for pair in token_tag:
res = wnl.lemmatize(pair[0],pos=get_wordnet_pos(pair[1]))
wnl_stems.append(res)
review_clean_wnl.append(' '.join(wnl_stems))
data['review']=review_clean_wnl
return data10) Vectorization
Word Embeddings or Word vectorization is a methodology in NLP to map words or phrases from vocabulary to a corresponding vector of real numbers which used to find word predictions, word similarities/semantics.
There are different types of vectorization techniques ,we mainly speaks about bag of words and Tfidf vectorizer .
#bag of words
vectorizer_bow = CountVectorizer(min_df=10,ngram_range=(1,4), max_features=5000)
vectorizer_bow.fit(X_train.values) #fitting
x_train_bow= vectorizer_bow.transform(X_train.values)
x_test_bow= vectorizer_bow.transform(X_test.values)
#tfidf
tfidfvectorizer = TfidfVectorizer(min_df=10,max_features=5000)
text_tfidf = tfidfvectorizer.fit(X_train.values) #fittingX_train_tfidf =tfidfvectorizer.transform(X_train.values)
X_test_tfidf =tfidfvectorizer.transform(X_test.values)What is Bag-of-Words?
We need a way to represent text data for machine learning algorithm and the bag-of-words model helps us to achieve that task. The bag-of-words model is simple to understand and implement. It is a way of extracting features from the text for use in machine learning algorithms.
The process of converting NLP text into numbers is called vectorization in ML. Different ways to convert text into vectors are:
Counting the number of times each word appears in a document.
Calculating the frequency that each word appears in a document out of all the words in the document.
Advantages:
Versatility: BoW can be used for a wide range of text processing tasks, including sentiment analysis, text classification, and document clustering. It provides a flexible and adaptable framework for representing textual data.
Disadvantages:
Sparse Representation: In datasets with a large vocabulary and many unique words, the BoW representation can lead to high-dimensional and sparse feature vectors. This increases computational complexity and memory requirements, especially for algorithms sensitive to data sparsity.
CountVectorizer works on Terms Frequency, i.e. counting the occurrences of tokens and building a sparse matrix of documents x tokens.
TF-IDF Vectorizer TF-IDF stands for term frequency-inverse document frequency. TF-IDF weight is a statistical measure used to evaluate how important a word is to a document in a collection or corpus. The importance increases proportionally to the number of times a word appears in the document but is offset by the frequency of the word in the corpus.
- Term Frequency (TF): is a scoring of the frequency of the word in the current document. Since every document is different in length, it is possible that a term would appear much more times in long documents than shorter ones. The term frequency is often divided by the document length to normalize.
- Inverse Document Frequency (IDF): is a scoring of how rare the word is across documents. IDF is a measure of how rare a term is. Rarer the term, more is the IDF score.
Thus,
Advantages:
Handles Common Terms: TF-IDF down weights common terms that occur frequently across documents, such as “the” or “and”, by assigning them lower weights. This helps in focusing on the more meaningful and distinctive terms within each document.
Disadvantages:
Vocabulary Size: Like BoW, TF-IDF relies on a predefined vocabulary of terms, which may not capture all relevant information in the text. Out-of-vocabulary terms or rare terms may be ignored or poorly represented, leading to information loss.
Types of Stemming
Stemming is a text normalization technique used in natural language processing to reduce words to their root or base form. There are different types of stemming algorithms, each with its own approach to stemming. Some common types of stemming include:
- Porter Stemmer: The Porter Stemmer, developed by Martin Porter in 1980, is one of the most widely used stemming algorithms. It applies a series of heuristic rules to remove suffixes from words to obtain their stems. It is known for its simplicity and efficiency but may produce stems that are not actual words.
- Lancaster Stemmer: The Lancaster Stemmer, also known as the Lancaster stemming algorithm, is another popular stemming algorithm. It is more aggressive than the Porter Stemmer and tends to produce shorter stems by applying a set of iterative rules. However, it can be overly aggressive and may produce stems that are too short or not meaningful.
- Snowball Stemmer (Porter2): The Snowball Stemmer, also known as the Porter2 Stemmer, is an improvement over the original Porter Stemmer. It is implemented in the Snowball programming language and offers better performance and accuracy compared to the Porter Stemmer. It supports multiple languages and provides more control over stemming parameters.
- Lovins Stemmer: The Lovins Stemmer is a stemming algorithm developed by Julie Beth Lovins in 1968. It is based on a set of transformation rules that operate on the endings of words to derive their stems. It is less commonly used compared to the Porter and Lancaster Stemmers but may be suitable for certain applications.
Conclusion
The text data depends on what you’re using the data for. Here are a couple of approaches:
For analysis and machine learning:
- Summarize key findings: Briefly restate the main points you discovered from analyzing the text data. This could be the sentiment of the text, recurring themes, or identified topics.
- Propose next steps: Depending on your goals, you might suggest further analysis techniques or applications based on your findings from the text data.
For written communication (like essays or reports):
- Restate the thesis and connect it to findings: Briefly remind the reader of your main point, and how the text data you analyzed supports it.
- Leave a lasting impression: Offer a final thought, implication, or call to action based on your analysis of the text data.
Here are some additional tips for strong text data conclusions:
- Avoid introducing new information.
- Maintain the same tone and style as the rest of your work.
- Use strong transitions to connect your conclusion to the body of your work.
References:
Text processing applications in machine learning: https://towardsdatascience.com/natural-language-processing-a1496244c15c
