Text Data Preprocessing-Machine Learning

Author: Meghamala Baipothu

Introduction: What is Text Data?

Text data is unstructured human language — such as sentences, reviews, emails, or tweets. Unlike clean, structured numerical data (like age, salary, or temperature), text data consists of open-ended written communication.

How Text Lives Inside Real-World Datasets (CSVs/DataFrames)

When working on Natural Language Processing (NLP) or Machine Learning tasks, text data usually arrives inside a tabular format like a .csv file or a Pandas DataFrame.

Here is a common example of how raw text lives inside a Customer Feedback Dataset:

In this dataset:

• Structured Columns (Feedback_ID, Sentiment): Easy to store, index, or convert directly into numeric labels (Positive → 1, Negative → 0).
• Unstructured Column (User_Comment): A string column containing punctuation, web links, numbers, mixed casing, and varied sentence lengths.

Understanding Documents vs. Corpus

When working with text datasets, two key terms appear frequently:

1. Document (d): A single unit of text. Depending on the project, a document can be:

• A single sentence (e.g., one row in your User_Comment column).
• A full email, a product review, or an entire news article.

2. Corpus (C): The entire collection of all documents in your dataset.

If your dataset has 10,000 customer reviews, the collection of all 10,000 reviews together forms your Corpus.

Mathematically, a corpus is represented as a set of N documents:

C = {d_1, d_2, d_3, …….., d_N}

Why Computers Struggle with Natural Language (0s and 1s vs. Words)

Machine learning models (like Logistic Regression, Random Forests, or Neural Networks) are mathematical functions. They perform matrix multiplications, calculate gradients, and update weights using numbers.

Computers do not natively understand human language:

• No Direct Math on Strings: A computer cannot multiply “GREAT” by “product” or calculate the distance between “slow” and “fast” without numeric representation.
• Ambiguity & Context: Words like “bank” can mean a financial institution or a riverbank.
• Lack of Fixed Length: Sentences in a text dataset vary in length, whereas ML models expect inputs with fixed dimensions.

Before any text can be fed into a model, it must undergo Preprocessing to clean the noise, followed by Vectorization to convert cleaned words into numerical vectors (0s and 1s).

Why Text Data Needs Preprocessing

Raw text data generated from social media, emails, web scraping, or documents is inherently “dirty” and unstructured. Machine learning and deep learning models operate on numerical data and fixed-dimensional vectors, making preprocessing an essential first step in Natural Language Processing (NLP).

1. The Problem with “Dirty” Text

• Case Sensitivity: Words like "Running", "running", and "RUNNING" carry the exact same semantic meaning, but untreated algorithms will treat them as three distinct features, diluting word statistics.
• Noise: Text often contains irrelevancies such as HTML tags (<br>), URLs (http://...), punctuation, special symbols (@, #, $), digits, and excessive white space that add no semantic value to standard downstream tasks.
• High Dimensionality: Uncleaned text results in a massive vocabulary size. This exponential increase in unique tokens leads to high-dimensional feature spaces, causing severe computational overhead and increased risk of overfitting (the Curiosity of Dimensionality).

2. Out-of-Vocabulary (OOV) Challenges

• Unseen Words & Slang: Real-world test data frequently introduces misspellings, typos, emojis, new jargon, or domain-specific terms absent from the training vocabulary.
• Embedding Misses: Traditional lookup models or word embeddings (e.g., Word2Vec, GloVe) assign an generic <UNK> (Unknown) token to OOV words, losing critical context unless subword tokenization or robust normalization is applied.

The 4-Step Text Preprocessing Pipeline

Step 1: Lowercasing & Noise Removal

• Lowercasing: Converts all characters to lowercase to unify vocabulary entries.
• Stripping Noise:
• Removing HTML tags and URLs via regular expressions (regex).
• Removing punctuation and special characters.
• Removing digits (when numbers do not contain domain-critical value).

Step 2: Tokenization & Stop-Word Removal

• Tokenization: Splitting continuous text streams into individual discrete units called tokens (words, subwords, or characters).
• Stop-Word Removal: Filtering out extremely frequent filler words (e.g., “the”, “is”, “at”, “which”, “and”) that carry minimal domain-specific semantic weight.

Step 3: Normalization (Stemming vs. Lemmatization)

Normalization reduces variant forms of a word (inflections, tenses, plurals) to a single common root.

• Stemming: A fast, rule-based heuristic that chops off word prefixes and suffixes. It operates without context and may produce non-dictionary base forms (e.g., “studies” →“studi”).
• Lemmatization: A morphology-aware technique using vocabulary dictionaries and Part-of-Speech (POS) tagging to return the grammatically correct base word (lemma).

Step 4: Text Vectorization: Converting Text to Numbers

Machine learning algorithms and deep learning models operate on numerical vectors, not raw strings. Text vectorization is the foundational natural language processing (NLP) technique of transforming unstructured text data into numerical vectors that mathematical models can parse, analyze, and learn from.

1. Bag of Words (BoW): Binary vs. Count-Based Approaches

The Bag of Words (BoW) model is one of the simplest and most intuitive methods for vectorizing text. It represents text as an unordered collection (“bag”) of words, disregarding grammar, word order, and syntax, but keeping track of vocabulary presence or multiplicity.

How BoW Works

1. Build a vocabulary of all unique words present across the entire dataset (corpus).
2. Represent each document as a vector where the length equals the size of the vocabulary.

Binary vs. Count-Based

Binary BoW (Presence/Absence):

• Each feature in the vector is binary (1 or 0).
• 1 indicates that the word appears at least once in the document; 0 indicates it is absent.
• Best for: Short text classification tasks (e.g., sentiment analysis on tweets) where word presence matters more than frequency.

Count-Based BoW (Term Frequency):

• Each feature represents the raw integer count (frequency) of how many times a word appears in the document.
• Best for: Longer documents where repetition conveys emphasis or topic relevance.

Example

Consider two documents:

Get Meghamalabaipothu’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Doc 1: “data science is fun science”

Doc 2: “data algorithms and data structure”

Vocabulary: [“algorithms”, “and”, “data”, “fun”, “is”, “science”, “structure”]

2. TF-IDF: Term Frequency — Inverse Document Frequency

While Count-Based BoW captures word frequency, it suffers from a major flaw: high-frequency words like “the”, “is”, or “and” dominate feature vectors across all documents without carrying meaningful semantic information.

TF-IDF solves this by evaluating how important a word is to a specific document relative to the entire corpus.

Why and How It Weights Words

• Common Words (Low TF-IDF): Words appearing in almost every document (e.g., “the”) receive an IDF = 0, driving their overall TF-IDF weight close to zero
• Rare, Specific Words (High TF-IDF): Words that appear frequently in a specific document but rarely across the rest of the corpus receive a high TF score and a high IDF score, resulting in a large TF-IDF weight.

Sparse Matrix Visualization

Visual Concept: A side-by-side comparison of a Sparse Matrix representation comparing Raw Word Counts (BoW) vs. Scaled TF-IDF Weights for a document corpus.

• Left Panel (CountVectorizer): Heatmap showing discrete integer counts (0, 1, 2, 3). Common stop words (“the”, “is”) stand out with bright, high-value colors despite having low semantic value.
• Right Panel (TF-IDF Vectorizer): Continuous decimal weights ($0.00$ to 0.85). Common words are suppressed down to near zero (0.05), while domain-specific, rare keywords (e.g., “neural”, “quantum”) light up with high float values.

Putting It All Together: Complete End-to-End Python Pipeline

Below is a complete, clean, and reusable Python script that processes raw text using Pandas, NLTK (for tokenization, stop-word removal, and lemmatization), and Scikit-Learn (for vectorization).

import re
import pandas as pd
import nltk
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer
from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer

# Download required NLTK resources
nltk.download('stopwords', quiet=True)
nltk.download('wordnet', quiet=True)

# ---------------------------------------------------------------------------
# 1. Text Preprocessing Function
# ---------------------------------------------------------------------------
def preprocess_text(text: str) -> str:
    """
    Cleans raw text by lowering case, removing punctuation/digits,
    filtering out stop words, and applying lemmatization.
    """
    if not isinstance(text, str):
        return ""
    
    # 1. Lowercase
    text = text.lower()
    
    # 2. Remove punctuation and numbers
    text = re.sub(r'[^a-z\s]', '', text)
    
    # 3. Tokenize words
    tokens = text.split()
    
    # 4. Remove Stopwords & Lemmatize
    stop_words = set(stopwords.words('english'))
    lemmatizer = WordNetLemmatizer()
    
    cleaned_tokens = [
        lemmatizer.lemmatize(word) 
        for word in tokens 
        if word not in stop_words and len(word) > 1
    ]
    
    return " ".join(cleaned_tokens)

# ---------------------------------------------------------------------------
# 2. Complete End-to-End Vectorization Pipeline
# ---------------------------------------------------------------------------
def text_vectorization_pipeline(corpus: list, max_features: int = 100):
    """
    Takes a raw text corpus, applies preprocessing, and returns Pandas DataFrames
    for Binary BoW, Count BoW, and TF-IDF representations.
    """
    # Step A: Load into Pandas DataFrame
    df = pd.DataFrame({'raw_text': corpus})
    
    # Step B: Preprocess Text
    df['cleaned_text'] = df['raw_text'].apply(preprocess_text)
    
    # Step C: Binary Bag of Words
    binary_vectorizer = CountVectorizer(binary=True, max_features=max_features)
    binary_matrix = binary_vectorizer.fit_transform(df['cleaned_text'])
    df_binary = pd.DataFrame(
        binary_matrix.toarray(), 
        columns=binary_vectorizer.get_feature_names_out()
    )
    
    # Step D: Count-based Bag of Words
    count_vectorizer = CountVectorizer(binary=False, max_features=max_features)
    count_matrix = count_vectorizer.fit_transform(df['cleaned_text'])
    df_count = pd.DataFrame(
        count_matrix.toarray(), 
        columns=count_vectorizer.get_feature_names_out()
    )
    
    # Step E: TF-IDF Vectorization
    tfidf_vectorizer = TfidfVectorizer(max_features=max_features)
    tfidf_matrix = tfidf_vectorizer.fit_transform(df['cleaned_text'])
    df_tfidf = pd.DataFrame(
        tfidf_matrix.toarray(), 
        columns=tfidf_vectorizer.get_feature_names_out()
    )
    
    return df, df_binary, df_count, df_tfidf


# ---------------------------------------------------------------------------
# Example Execution
# ---------------------------------------------------------------------------
if __name__ == "__main__":
    sample_corpus = [
        "Machine learning algorithms build a model based on sample data.",
        "Data science and machine learning are revolutionizing technology.",
        "Deep learning is a subset of machine learning using neural networks."
    ]
    
    df, df_bin, df_cnt, df_tfidf = text_vectorization_pipeline(sample_corpus)
    
    print("--- Cleaned Text ---")
    print(df[['raw_text', 'cleaned_text']])
    
    print("\n--- TF-IDF Matrix Preview ---")
    print(df_tfidf.round(3))

Conclusion

Text vectorization is a vital bridge connecting raw unstructured text to numerical machine learning algorithms:

• Bag of Words (Binary & Count) provides a quick baseline for feature representation, but treats all words equally and ignores term importance.
• TF-IDF enhances basic count models by penalizing ubiquitous non-informative terms while highlighting rare, high-signal keywords.
• Pipeline Integration: Combining text preprocessing (lowercasing, stopword removal, lemmatization) with Scikit-Learn vectorizers provides a robust, scalable workflow for tabular text classification and NLP tasks.

References

1. https://www.analyticsvidhya.com/blog/2018/08/k-nearest-neighbor-introduction-regression-python/
2. https://scikitlearn.org/1.4/tutorial/text_analytics/working_with_text_data.html
3. OpenAI — ChatGPT (Used for AI-assisted diagrams and visual content creation)
   https://chat.openai.com/