The Art of Preprocessing
This article is the first part of the “Teach Your Computer to Read and Write” series to understand core NLP concepts in a clear and accessible way.
If you’d like to explore the concepts from this article in a hands-on way, I’ve included a companion notebook on GitHub with simple examples. If you enjoyed the tutorial, a ⭐ on the repo is always appreciated!
Natural language processing (NLP) is a branch of artificial intelligence that focuses on how computers analyze and understand human language. Today, NLP is integrated into nearly all modern automation workflows that involve any communication. The world generates vast volumes of voice and text data that can be analyzed to uncover valuable insights. While NLP focuses on building systems, linguistics helps us understand the structure of language itself.
Linguistics covers the structure of language (morphology and syntax), the sounds of language (phonetics and phonology), and the meaning of language in and out of context (semantics and pragmatics). This leads naturally to computational linguistics, which sits at the intersection of both worlds. Computational linguistics is a branch of linguistics that studies how computers can understand both written and spoken language, and how we can build systems that process or generate language. Since language reflects how we think, understanding it computationally also helps us understand the nature of thought and intelligence.
In simple terms, the distinction between the two can be seen as follows:
- NLP = “How do we build systems that use language?”
- Computational Linguistics = “How do we understand language in a way computers can model?”
Once we understand what NLP tries to achieve, the next step is exploring how it actually works in practice.
To build NLP applications, we typically move through a series of processing stages called an NLP pipeline.
Key Components of the NLP Preprocessing Pipeline
Before any model can understand language, raw text must go through a series of preprocessing steps. These operations clean, structure, and annotate the input so it becomes machine-interpretable. In classical NLP, tasks like tokenization, POS tagging, and parsing are considered part of the preprocessing pipeline because they prepare text for downstream applications.
Get Gomathi L’s stories in your inbox
Join Medium for free to get updates from this writer.
Sample Input: “I watched Amy playing badminton at the country club yesterday.”
- Text Normalization: Transforming text into a consistent, standard format, such as converting to lowercase, removing punctuation, or expanding abbreviations.
- Sample Output: [“i watched amy playing badminton at the country club yesterday”] - Tokenization: Segmenting text into words, punctuation marks etc.
- Sample Output: [“I”, “watched”, “Amy”, “playing”, “badminton”, “at”, “the”, “country”, “club”, “yesterday”, “.”] - Parts-of-speech (POS) Tagging: Assigning word types to tokens, like verbs or nouns.
- Sample Output: [PRON, VERB, PROPN, VERB, NOUN, ADP, DET, ADJ, NOUN, NOUN, PUNCT] - Dependency Parsing: Assigning syntactic dependency labels, describing the relations between individual tokens, like subject or object.
- Sample Output: “watched” — main verb, “I” — subject, “Amy” — object, etc. - Named Entity Recognition (NER): Labelling named “real-world” objects, like persons, companies or locations.
- Sample Output: “Amy” — PERSON, “country club” — LOCATION, “yesterday” — DATE. - Lemmatization: Assigning the base forms of words.
- Sample Output: [“I”, “watch”, “Amy”, “play”, “badminton”, “at”, “the”, “country”, “club”, “yesterday”, “.”]
Together, these steps help computers handle language more effectively, stripping away the noise so they can focus on what actually matters.
Note: How Modern NLP Pipelines Differ
While the classical NLP pipeline is essential for understanding how language is processed step-by-step, modern transformer-based models handle these stages quite differently:
- Contemporary NLP systems often avoid heavy normalization (like lowercasing or removing punctuation) because neural models learn directly from raw text. Reducing casing or punctuation can remove useful semantic and syntactic cues.
- Instead of splitting text only by spaces or punctuation, modern LLMs use subword tokenization (e.g., BPE, WordPiece). This lets rare or complex words be broken into smaller learned units
(e.g., “playing” → “play” + “ing”), improving flexibility and vocabulary efficiency. - In models like BERT or GPT, tasks such as POS tagging, dependency parsing, and NER are not executed as separate steps. Instead, the model implicitly learns these linguistic patterns within its transformer layers.
Understanding the classical pipeline remains important because it shows what these models must learn behind-the-scenes.
Once you understand these core building blocks, the next practical question becomes: Which tools or libraries should you use to implement them?
This is where two of the most widely used NLP libraries, spaCy and NLTK, come into the picture.
- spaCy and NLTK are two popular NLP libraries, but they each shine in different ways. spaCy is newer and built for speed, so it handles large amounts of text quickly and comes with strong built-in tools like tokenization, named entity recognition, and dependency parsing. It also supports many major languages, though its pretrained pipelines cover fewer languages than the broad rule-based resources available in NLTK. spaCy is highly optimized but follows a more structured pipeline design, so customization is possible but less free-form than in NLTK. NLTK, on the other hand, has been around much longer and is extremely flexible. It includes a wide range of linguistic resources, teaching materials, and tools for working with many languages, but it generally runs slower and often requires more manual setup. If you’re experimenting, learning NLP concepts, or working with various languages and rule-based tools, NLTK is often the preferred choice. If you need something fast, efficient, and ready for production-level tasks, spaCy is usually the more effective choice.
Understanding how machines process language from the ground up is the first step toward building smarter NLP-based applications. Once you grasp the basics of how text is broken down, cleaned, and interpreted by a machine, the rest of NLP begins to fall naturally into place. In the upcoming articles of “Teach Your Computer to Read and Write,” we will explore the next layers of the NLP pipeline and look at how each component works under the hood.
References:
