NLP Token Classification

Introduction to NLP

Have you ever wondered how NLP works or What is NLP? . So let’s clearly get in to that .

Natural Language Processing(NLP) is a sub field of Artificial Intelligence(AI) and used by the Machine Learning to help the machine to understand the context of the text.

Basically it is like helping the machine to understand the each and every word meaning and reason behind the usage of the word , so that it can understand and give the relevant context.

What is Token Classification?

Token Classification in simple words it means nothing but dividing or parting the each word in the sentence or paragraph and then each word is assigned a specific label.

Example: John works at Google

• John → Person
• works → Verb
• Google → Organization

Why is Token-Level Understanding Important?

The Token -Level Understanding is very important because it helps the machine to understand each and every word and also it meaning and the propose the word being their.

• Identify important words and entities
• Understand sentence structure and grammar
• Extract meaningful information from text
• Improve accuracy in NLP tasks

The Real-world examples which uses NLP to make the machine understand the text are:
Chatbots
Search engines
Information extraction

Named Entity Recognition (NER)

Definition:

Named Entity Recognition (NER) is a token classification task in NLP that helps to identify and classify important words in a sentence into predefined categories such as:

• Person (PER)
• Organization (ORG)
• Location (LOC)
• Miscellaneous (MISC)

In simple terms, NER helps machines understand “who”, “what”, and “where” in a sentence.

BIO Tagging Concept:

NER uses the BIO Tagging to label the tokens so that machines can clearly understand starting of the tokens along with continuation of the tokens.

• B (Beginning) → Marks the start of an entity
• I (Inside) → Marks words inside the same entity
• O (Outside) → Words that are not part of any entity

Example: John works at Google

• John → B-PER
• works → O
• at → O
• Google → B-ORG

Applications :

Named Entity Recognition is widely used in real-world applications such as:

• Information Extraction → Extract names, locations, and organizations from text
• Chatbots & Virtual Assistants → Understand user queries better
• Search Engines → Improve search results by identifying key entities
• Healthcare Systems → Extract patient information from medical records
• Finance & News Analysis → Identify companies, events, and important entities

Parts Of Speech(POS) Tagging:

Definition:

Parts of Speech Tagging is an NLP token classification task which labels each word with the grammatical label in a sentence. These labels tells the role of the word in the sentence, such as whether the word is noun ,pronoun ,adjective ,verb, adverb, etc.

In simple terms, POS tagging helps amchines understand the grammar and structure of a sentence.

Types of POS Tags :

Some of the most commonly used POS tags include:

• NOUN → Names of people, places, or things
• VERB → Action words
• ADJ (Adjective) → Describes a noun
• ADV (Adverb) → Describes a verb
• PRON (Pronoun) → Replaces a noun
• DET (Determiner) → Specifies a noun (a, an, the)
• ADP (Preposition) → Prepositions like in, on, at
• CONJ (Conjunction) → Connects words (and, but)
• PUNCT → Punctuation marks

Example: John eats an apple

• John → NOUN
• eats → VERB
• an → DET
• apple → NOUN

Importance of POS Tagging

POS tagging plays a crucial role in NLP because it helps the machines:

• Understand the grammatical structure of sentences
• Improve text parsing and analysis
• Support tasks like machine translation and speech recognition
• Enhance the performance of chatbots and search engines.

Chunking (Phrase Detection):

Definition:

Chunking is a technique that groups the words in a sentence into meaning phrases. Instead of labeling individual words it just gather the words that belong together in a sentence. Chunkin is also called as Shallow parsing.

Get Giluguaakanksha’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

In simple terms, chunking helps machines understand “groups of words that belong together.”

Example: The quick brown fox jumps over the lazy dog

• [The quick brown fox] → Noun Phrase (NP)
• [jumps] → Verb Phrase (VP)
• [over the lazy dog] → Prepositional Phrase (PP)

Difference from NER:

Although both are token classification tasks, they serve different purposes:

• NER (Named Entity Recognition) identifies specific entities like names, places, and organizations
• Chunking identifies grammatical phrases such as noun phrases and verb phrases

In Quick Terms:

• NER → “What is this word?” (entity type)
• Chunking → “Which words belong together?” (phrase structure)

Use Cases:

Chunking is widely used in the below some of the use cases:

• Text Summarization → Identifying key phrases in a document
• Question Answering Systems → Understanding sentence structure
• Information Extraction → Extracting meaningful chunks of data
• Grammar Analysis → Understanding how sentences are formed
• Speech Processing → Improving natural language understanding

Comparison of Techniques:

NER vs POS Tagging vs Chunking:

• NER identifies real-world entities, POS tagging identifies grammatical roles of words, while chunking identifies meaningful phrases in a sentence.
• NER and POS tagging both operate at the word level, whereas chunking works at the phrase level by grouping words together.
• POS tagging is generally easier, while NER and chunking are moderately complex as they require contextual understanding.
• NER outputs labels like B-PER and I-LOC, POS tagging outputs labels like NOUN and VERB, while chunking outputs phrase tags like NP and VP.
• NER is mainly used for information extraction, POS tagging is used for grammar analysis, and chunking is used for understanding sentence structure.
• While POS tagging focuses on individual word roles, NER focuses on identifying important entities, and chunking focuses on grouping related words into phrases.

Code Implementation :

In this section, we can see how token classification tasks like Named Entity Recognition (NER), POS Tagging, and Chunking can be implemented using popular NLP libraries.

Named Entity Recognition(NER):

from transformers import pipeline
ner = pipeline(“ner”, grouped_entities=True)

text = “John works at Google in California”
result = ner(text)

print(“NER Output:”)
for entity in result:
print(entity)

POS Tagging:

import spacy
nlp = spacy.load(“en_core_web_sm”)

text = “John works at Google in California”
doc = nlp(text)

print(“\nPOS Tagging:”)
for token in doc:
print(token.text, “→”, token.pos_)

Chunking(Phrase Detection):

print(“\nChunking (Noun Phrases):”)

for chunk in doc.noun_chunks:
print(chunk.text, “→ NP”)

Transformer-Based Approach (BERT for Token Classification)

Definition:

Modern NLP tasks such as Named Entity Recognition (NER), POS Tagging, and Chunking are significantly improved using transformer-based models like BERT (Bidirectional Encoder Representations from Transformers).

How BERT Works

BERT is a deep learning model that reads the entire sentence at once and understands the context of each word by looking at both the left and right sides (bidirectional context).

For example:
Sentence: “He went to the bank”
BERT understands whether “bank” refers to a financial institution or a river bank based on context.

Token Classification using BERT

In token classification tasks, BERT processes a sentence and produces a contextual embedding (vector representation) for each token. These embeddings are then passed through a classification layer to assign labels.

Steps Involved

1. Tokenization
   The input sentence is split into tokens using a BERT tokenizer. Special tokens like [CLS] and [SEP] are added.
2. Embedding Generation
   Each token is converted into embeddings that capture its meaning in context.
3. Contextual Understanding
   BERT processes all tokens simultaneously using self-attention, allowing it to understand relationships between words.
4. Classification Layer
   A fully connected layer is added on top of BERT to assign labels (e.g., NOUN, VERB, B-PER).

Why BERT is Powerful

• Understands context (bidirectional learning)
• Handles ambiguity in language
• Works well for multiple tasks (NER, POS, Chunking)
• Provides high accuracy compared to traditional models

Example: John works at Google

BERT Output:

• John → B-PER
• works → O
• Google → B-ORG

Conclusion

In conclusion, token classification plays a vital role in helping machines understand human language at a deeper level. Tasks like Named Entity Recognition (NER), POS tagging, and chunking each contribute uniquely by identifying entities, grammatical roles, and phrase structures within text.

With the advancement of transformer-based models like BERT, these tasks have become more accurate and efficient, enabling powerful real-world applications such as chatbots, search engines, and information extraction systems. Overall, understanding token classification provides a strong foundation for building intelligent NLP solutions.