NLP Preprocessing Part-II
In the previous blog, I had covered the foundational NLP preprocessing steps used to clean and normalize text. Building on that knowledge, this post focuses on more advanced techniques that help capture the semantic meaning and contextual relationships within language. If you haven’t checked out the earlier part yet, I recommend revisiting it to see how these concepts connect and progress together.
1. Bag of Words (BoW) : Treat text as a “bag” of words, ignoring grammar and word order. This methods simply count how many times each word appears.
How it works:
Build a vocabulary of all unique words in the dataset.
Represent each document as a vector of word counts.
Example:
Sentences: “I love NLP” , “NLP is fun”
Vocabulary = {I, love, NLP, is, fun}
Pros: Simple and easy to implement
Cons: Ignores context (word order and meaning), High-dimensional and sparse vectors.
2. TF–IDF (Term Frequency — Inverse Document Frequency) : Not all words are equally important.
TF-IDF increases the weight of important words and reduces the weight of common ones.
Formula logic: TF (Term Frequency): How often a word appears in a document. IDF (Inverse Document Frequency): Rarer words get higher importance
TF-IDF = TF * IDF
Example:
- “the” appears in almost every document → low weight
- “diabetes” appears rarely but is meaningful → high weight
Pros: More informative than raw counts, Reduces noise from common words
Cons: Still ignores context (word order & meaning).
Get Pranav Srinivas’s stories in your inbox
Join Medium for free to get updates from this writer.
3. Word Embeddings (Word2Vec, GloVe, FastText): Words with similar meanings should have similar vector representations. Instead of counts, embeddings represent words as dense vectors in continuous space (e.g., 100–300 dimensions).
Key property:
Words close in meaning are close in vector space.
vector(“king”) — vector(“man”) + vector(“woman”) ≈ vector(“queen”)
Advantages:vCaptures semantic meaning, Words that appear in similar contexts have similar vectors
Limitations: One vector per word → doesn’t understand context
e.g., bank (river bank vs. financial bank) has same embedding.
4. Transformer Embeddings (BERT, RoBERTa, GPT Models) : Context matters — the meaning of a word depends on the surrounding words.
Transformer-based models generate contextual embeddings: Each instance of a word can have a different vector depending on the sentence.
“I sat on the river bank” → “bank” = river meaning
“I deposited money in the bank” → “bank” = financial meaning
Transformers produce different embeddings for both — unlike Word2Vec.
Why they work:
They use self-attention to understand relationships between words in a sentence.
Advantages: Captures deep semantic meaning, State-of-the-art in NLP tasks like classification, sentiment, summarization, Q&A
Limitations: Computationally heavier than BoW, TF-IDF, or Word2Vec
Use case scenarios:
TF-IDF: Works best with ML based model like SVM, Logistic Regression.
Word Embeddings: Suitable for NN models like LSTM, GRU, RNN and CNN.
Transformer Embeddings: Mainly for transformer models.
With this, I am wrapping up Part 2 of our NLP Preprocessing Techniques. In this section, we explored how raw text can be transformed into meaningful numerical representations using methods like TF–IDF, Word Embeddings, and Transformer-based Embeddings. Understanding when and why to use each of these techniques forms the foundation for building effective NLP models.
