<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Abhijit Gupta on Medium]]></title>
        <description><![CDATA[Stories by Abhijit Gupta on Medium]]></description>
        <link>https://medium.com/@avgupta456?source=rss-83dc37433c1b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*askT4D8A6WXaoNVvTWTcgQ.jpeg</url>
            <title>Stories by Abhijit Gupta on Medium</title>
            <link>https://medium.com/@avgupta456?source=rss-83dc37433c1b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 04:19:37 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@avgupta456/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Natural Language Processing (NLP): Don’t Reinvent the Wheel]]></title>
            <link>https://medium.com/data-science/natural-language-processing-nlp-dont-reinvent-the-wheel-8cf3204383dd?source=rss-83dc37433c1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/8cf3204383dd</guid>
            <category><![CDATA[libraries]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[natural-language-process]]></category>
            <category><![CDATA[algorithms]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Abhijit Gupta]]></dc:creator>
            <pubDate>Wed, 12 Aug 2020 15:57:01 GMT</pubDate>
            <atom:updated>2020-08-14T04:05:44.527Z</atom:updated>
            <content:encoded><![CDATA[<h4>Tools and techniques to accelerate your Python NLP projects</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/997/1*jawjUxVmzZFZm3i_qgGpXg.png" /><figcaption>(Image by Author)</figcaption></figure><h3>Introduction</h3><p><strong>Natural language processing (NLP) </strong>is an intimidating name for an intimidating field. Generating useful insight from unstructured text is <em>hard</em>, and there are countless techniques and algorithms out there, each with their own use-cases and complexities. As a developer with minimal NLP exposure, it can be difficult to know which methods to use, and how to implement them. I’m here to help.</p><p>If I were to offer perfect results for minimal effort, you would be right to be skeptical. Instead, using the <strong>80/20 Principle</strong>, I’ll show you how to quickly (20%) deliver solutions, without significantly sacrificing outcomes (80%).</p><blockquote>“The 80/20 Principle exerts that a minority of causes, inputs, or efforts usually leads to a majority of results, outputs, or rewards”</blockquote><blockquote><strong>-Richard Koch, author of The 80/20 Principle</strong></blockquote><p>How exactly will we achieve this goal? With some fantastic Python libraries! Instead of reinventing the wheel, we may stand on the shoulders of giants and innovate quickly. With pre-tested implementations and pre-trained models, we will focus on applying these methods and creating value.</p><p>This article is intended for developers looking to quickly integrate natural language processing into their projects. With an emphasis on ease of use and rapid results comes the downside of reduced performance. In my experience, 80% of cutting-edge is plenty for projects, but look elsewhere for NLP research :)</p><p>Without further ado, let’s begin!</p><h3>What is NLP?</h3><p>Natural language processing is a subfield of linguistics, computer science, and artificial intelligence, allowing for the automatic processing of text by software. NLP gives machines the ability to read, understand, and respond to messy, unstructured text.</p><p>People often treat NLP as a subset of machine learning, but the reality is more nuanced.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*lbT7OXMA4J2ZwY809HklCw.png" /><figcaption>(Image by Author)</figcaption></figure><p>Some NLP tools rely on machine learning, and some even use deep learning. However these methods often rely on large datasets and are difficult to implement. Instead, we will focus on simpler, rule-based methods to speed up the development cycle.</p><h4>Terminology</h4><p>Starting with the smallest unit of data, a <strong>character </strong>is a single letter, number, or punctuation. A <strong>word </strong>is a list of characters, and a <strong>sentence </strong>is a list of words. A <strong>document </strong>is a list of sentences, and a <strong>corpus </strong>is a list of documents.</p><h4>Pre-Processing</h4><p>Pre-processing is perhaps the most important step to a NLP project, and involves cleaning your inputs so your models can ignore the noise and focus on what matters most. <strong>A strong pre-processing pipeline will improve the performance of all your models, so I cannot stress it’s value enough.</strong></p><p>Below are some common pre-processing steps:</p><ul><li><strong>Segmentation</strong>: Given a long list of characters, we might separate documents by white space, sentences by periods, and words by spaces. Implementation details will vary based on the dataset.</li><li><strong>Make Lowercase</strong>: Capitalization generally does not add value, and makes string comparison trickier. Just make everything lowercase.</li><li><strong>Remove Punctuation</strong>: We may want to remove commas, quotes, and other punctuation that does not add to the meaning.</li><li><strong>Remove Stopwords</strong>: Stopwords are words like ‘she’, ‘the’, and ‘of’ that do not add to the meaning of a text, and can distract from the more relevant keywords.</li><li><strong>Remove Other</strong>: Depending on your application, you may want to remove certain words that do not add value. For example, if evaluating course reviews, words like ‘professor’ and ‘course’ may not be useful.</li><li><strong>Stemming/Lemmatization</strong>: Both stemming and lemmatization generate the root form of inflected words (ex: ‘running’ to ‘run’). Stemming is faster, but does not guarantee the root is an English word. Lemmatization uses a corpus to ensure the root is a word, at the expense of speed.</li><li><strong>Part of Speech Tagging</strong>: POS tagging marks words with their part of speech (nouns, verbs, prepositions) based on definition and context. For example, we can focus on nouns for keyword extraction.</li></ul><p>For a more thorough introduction to these concepts, check out this amazing guide:</p><p><a href="https://towardsdatascience.com/a-practitioners-guide-to-natural-language-processing-part-i-processing-understanding-text-9f4abfd13e72">A Practitioner&#39;s Guide to Natural Language Processing (Part I) — Processing &amp; Understanding Text</a></p><p>These steps are the foundation of a successful pre-processing pipeline. Depending on your dataset and task, you may skip certain steps or add new ones. Manually observe your data through pre-processing, and correct issues as they arise.</p><h3>Python Libraries</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/0*KDCgV7tszyaM-X64.png" /><figcaption>Source: <a href="http://python.org/">python.org</a></figcaption></figure><p>Let’s take a look at a couple leading Python libraries for NLP. These tools will handle most of the heavy lifting during and especially after pre-processing</p><h4>NLTK</h4><p>The Natural Language Tool Kit is the most widely-used NLP library for Python. Developed at UPenn for academic purposes, NLTK has a plethora of features and corpora. NLTK is great for playing with data and running pre-processing.</p><p><a href="https://www.nltk.org/">NLTK</a></p><p>Here is an example from the NLTK website showing how simple it is to tokenize a sentence and tag parts of speech.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/66633f7dbaa2e8a86d09f8d718d87907/href">https://medium.com/media/66633f7dbaa2e8a86d09f8d718d87907/href</a></iframe><h4>SpaCy</h4><p>SpaCy is a modern and opinionated package. While NLTK has multiple implementations of each feature, SpaCy keeps only the best performing ones. Spacy supports a wide range of features, read the docs for more details:</p><p><a href="https://spacy.io/">spaCy · Industrial-strength Natural Language Processing in Python</a></p><p>In just a few lines, we are able to perform Named Entity Recognition with SpaCy. Many other tasks can be accomplished quickly using the SpaCy API.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ae508ec57a5a842c39d6f4f5c2922853/href">https://medium.com/media/ae508ec57a5a842c39d6f4f5c2922853/href</a></iframe><h4>GenSim</h4><p>Unlike NLTK and SpaCy, GenSim specifically tackles the problem of information retrieval (IR). Developed with an emphasis on memory management, GenSim contains many models for document similarity, including Latent Semantic Indexing, Word2Vec, and FastText.</p><p><a href="https://github.com/RaRe-Technologies/gensim">RaRe-Technologies/gensim</a></p><p>Below is an example of a pre-trained GenSim Word2Vec model that finds word similarities. Without worrying about the messy details, we can quickly get results.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8121bd28f5683577016ee9fa3f622b2e/href">https://medium.com/media/8121bd28f5683577016ee9fa3f622b2e/href</a></iframe><h4>And More…</h4><p>This list is by no means comprehensive, but covers a range of features and use-cases. I recommend checking <a href="https://github.com/keon/awesome-nlp">this repository</a> for more tools and references.</p><h3>Applications</h3><p>Now that we have discussed pre-processing methods and Python libraries, let’s put it all together with a few examples. For each, I’ll cover a couple of NLP algorithms, pick one based on our rapid development goals, and create a simple implementation using one of the libraries.</p><h4>Application #1: Pre-Processing</h4><p>Pre-processing is a critical part of any NLP solution, so let’s see how we can speed up the process with Python libraries. In my experience, NLTK has all the tools we need, with customization for unique use cases. Let’s load a sample corpus.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d1e4d8879dc1333cd8624859318f9798/href">https://medium.com/media/d1e4d8879dc1333cd8624859318f9798/href</a></iframe><p>Following the pipeline defined above, we can use NLTK to implement segmentation, removing punctuation and stopwords, performing lemmatization, and more. Look how easy it is to remove stopwords:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b11bb648608c285f0b7883126568f28e/href">https://medium.com/media/b11bb648608c285f0b7883126568f28e/href</a></iframe><p>The entire pre-processing pipeline took me less than 40 lines of Python. See the full code <a href="https://github.com/avgupta456/medium_nlp/blob/master/Pre-Processing.ipynb">here</a>. Remember, this is a generalized example, and you should modify the process as needed for your specific use case.</p><h4>Application #2: Document Clustering</h4><p>Document clustering is a common task in natural language processing, so let’s discuss some ways to do it. The general idea here is to assign each document a vector representing the topics discussed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*Gm6BpirIv5fGY2dmr1RoYQ.png" /><figcaption>(Image by Author)</figcaption></figure><p>If the vectors are two dimensional, we can visualize the documents, like above. In this example, we see documents A and B are closely related, while D and F are loosely related. Using a distance metric, we can calculate similarity even when these vectors are three, one hundred, or one thousand dimensional.</p><p>The next question is how to construct these vectors for each document, using the unstructured text input. Here there are a few options, from simplest to most complex:</p><ul><li><strong>Bag of Words</strong>: Assign each unique word a dimension. The vector for a given document is the frequency each word occurs.</li><li><strong>Term Frequency — Inverse Document Frequency (TF-IDF)</strong>: Scale the Bag of Words representation by how common a word is in other documents. If two documents share a rare word, they are more similar than if they share a common one.</li><li><strong>Latent Semantic Indexing (LSI)</strong>: Bag of Words and TF-IDF can create highly dimensional vectors, which makes distance measures less accurate. LSI collapses these vectors to a more manageable size while minimizing information loss.</li><li><strong>Word2Vec</strong>: Using a neural network, learn word associations from a large text corpus. Then add up the vectors for each word to get a document vector.</li><li><strong>Doc2Vec</strong>: Building upon Word2Vec but using a better method to approximate the document vector from its list of word vectors.</li></ul><p>Word2Vec and Doc2Vec are quite complicated and require large datasets to learn word embeddings. We could use pre-trained models, but they may not scale well to tasks within niche fields. Instead, we will use Bag of Words, TF-IDF, and LSI.</p><p>Now to choose our library. GenSim is specifically built for this task and contains easy implementations of all three algorithms, so let’s use GenSim.</p><p>For this example, let’s use the Brown corpus again. It has 15 documents for categories of text, such as ‘adventure’, ‘editorial’, ‘news’, etc. After running our NLTK pre-processing routine, we can begin applying the GenSim models.</p><p>First, we create a dictionary mapping tokens to unique indexes.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aa4107fa291a8df312303bd1a8d0c79c/href">https://medium.com/media/aa4107fa291a8df312303bd1a8d0c79c/href</a></iframe><p>Next, we iteratively apply Bag of Words, Term Frequency — Inverse Document Frequency, and Latent Semantic Indexing:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/828843e6b0cbe2ff527733fdd3214562/href">https://medium.com/media/828843e6b0cbe2ff527733fdd3214562/href</a></iframe><p>In just ~10 lines of Python, we handled three separate models, and extracted vector representations for our documents. Using cosine similarity for vector comparison, we can find the most similar documents.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/498e974958e14a8c0a75c4b799819b3c/href">https://medium.com/media/498e974958e14a8c0a75c4b799819b3c/href</a></iframe><p>And just like that, we have results! Adventure texts are most similar to fiction and romance, while editorials are similar to news and government. It checks out. Check the full code <a href="https://github.com/avgupta456/medium_nlp/blob/master/Similarity.ipynb">here</a>.</p><h4>Application #3: Sentiment Analysis</h4><p>Sentiment analysis is the interpretation of unstructured text as positive, negative, or neutral. Sentiment analysis is a useful tool for analyzing reviews, measuring brand, building AI chatbots, and more.</p><p>Unlike document clustering, where pre-processing was applied, we <strong>do not use pre-processing in sentiment analysis</strong>. The punctuation, flow, and context of a passage can reveal a lot about the sentiment, so we do not want to remove them. Instead, we jump straight into the models.</p><p>Keeping things simple and effective, I recommend using pattern-based sentiment analysis. By searching for specific keywords, sentence structure, and punctuation marks, these models measure the text’s polarity. Here are two libraries with built-in sentiment analyzers:</p><p><strong>VADER Sentiment Analysis</strong>:</p><p>VADER stands for Valence Aware Dictionary and sEntiment Recognizer, and is an extension of NLTK for sentiment analysis. It uses patterns to calculate sentiment, and works especially well with emojis and texting slang. It’s also super easy to implement.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/98f27537648ce292b5da5b52827ff624/href">https://medium.com/media/98f27537648ce292b5da5b52827ff624/href</a></iframe><p><strong>TextBlob Sentiment Analysis</strong>:</p><p>A similar tool is TextBlob for sentiment analysis. TextBlob is actually a versatile library similar to NLTK and SpaCy. Regarding its sentiment analysis tool, it differs from VADER in reporting both polarity and subjectivity. From my personal experience, I prefer VADER, but each has its own strengths and weaknesses. TextBlob is also exceedingly easy to implement:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bfd402654f9ea69fd145bf28f9de35b5/href">https://medium.com/media/bfd402654f9ea69fd145bf28f9de35b5/href</a></iframe><p><strong>Note: </strong>Pattern based models do not perform well on such small texts as in the examples above. <strong>I recommend sentiment analysis on texts averaging four sentences.</strong> For a quick demonstration of this, refer to <a href="https://github.com/avgupta456/medium_nlp/blob/master/Sentiment.ipynb">the Jupyter Notebook</a>.</p><h4>Other Applications</h4><p>Here are a couple of additional topics and some useful algorithms and tools to accelerate your development.</p><ul><li><strong>Keyword Extraction:</strong> Named Entity Recognition (NER) using SpaCy, Rapid Automatic Keyword Extraction (RAKE) using ntlk-rake</li><li><strong>Text Summarization</strong>: TextRank (similar to PageRank) using PyTextRank SpaCy extension, TF-IDF using GenSim</li><li><strong>Spell Check</strong>: PyEnchant, SymSpell Python ports</li></ul><p>Hopefully, these examples help demonstrate the plethora of resources available for natural language processing in Python. Regardless of the problem, chances are someone has developed a library to streamline the process. Using these libraries can yield great results in a short time frame.</p><h3>Tips and Tricks</h3><p>With an introduction to NLP, an overview of Python libraries, and some example applications, you’re almost ready to tackle your own challenges. Finally, I have a few tips and tricks to make the most of these resources.</p><ul><li><strong>Python Tooling</strong>: I recommend Poetry for dependency management, Jupyter Notebook for testing new models, Black and/or Flake8 for linting, and GitHub for version management.</li><li><strong>Stay organized</strong>: It can be easy to jump around from library to library, copying in code to test a dozen ideas. Instead, I recommend a more measured approach. You don’t want to miss a great solution in your haste.</li><li><strong>Pre-Processing</strong>: Garbage in, garbage out. It’s super important to implement a strong pre-processing pipeline to clean your inputs. Visually check the processed text to ensure everything is working as expected.</li><li><strong>Presenting Results</strong>: Choosing how to present your results can make a huge difference. If your outputted text look a little rough, consider presenting aggregate statistics or numeric results instead.</li></ul><p>You should be well equipped to tackle some real-world NLP projects now. Good luck, and happy coding :)</p><p><em>If you do anything cool with this information, leave a response in the comments. If you have any feedback or insights, feel free to connect with me on </em><a href="https://www.linkedin.com/in/avgupta456/"><em>LinkedIn</em></a><em>. Thanks for reading!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8cf3204383dd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/natural-language-processing-nlp-dont-reinvent-the-wheel-8cf3204383dd">Natural Language Processing (NLP): Don’t Reinvent the Wheel</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>