<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pashupati Gupta on Medium]]></title>
        <description><![CDATA[Stories by Pashupati Gupta on Medium]]></description>
        <link>https://medium.com/@pashupatigupta1998?source=rss-57485a806d0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*bSwIxH3QVpb84szNZyqzJQ.png</url>
            <title>Stories by Pashupati Gupta on Medium</title>
            <link>https://medium.com/@pashupatigupta1998?source=rss-57485a806d0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 00:59:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pashupatigupta1998/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A statistical auto-correct system from scratch]]></title>
            <link>https://pashupatigupta1998.medium.com/a-statistical-auto-correct-system-from-scratch-e891ddfe141?source=rss-57485a806d0------2</link>
            <guid isPermaLink="false">https://medium.com/p/e891ddfe141</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[text]]></category>
            <dc:creator><![CDATA[Pashupati Gupta]]></dc:creator>
            <pubDate>Sun, 14 Mar 2021 20:06:23 GMT</pubDate>
            <atom:updated>2021-03-14T20:06:23.000Z</atom:updated>
            <content:encoded><![CDATA[<p>The task of an auto-correct system is finding out which words in a document are misspelled. These misspelled words might be presented to a user by underlining that words. Correction is the task of substituting the well-spelled word for misspellings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*0P6i3YcXW8_M_rvzeEGw0A.png" /></figure><p>Modern auto-correct systems use high-end deep learning techniques that are a little complicated to understand. If interested, you can look into some of these papers</p><ul><li><a href="http://static.googleusercontent.com/media/research.google.com/en/us/pubs/archive/36180.pdf">Using the Web for Language-Independent Spellchecking and Autocorrection</a></li><li><a href="http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=52A3B869596656C9DA285DCE83A0339F?doi=10.1.1.146.4390&amp;rep=rep1&amp;type=pdf">How Difficult is it to Develop a Perfect Spell-checker? A Cross-linguistic Analysis through Complex Network Approach</a></li><li><a href="https://arxiv.org/abs/1908.05263">AutoCorrect: Deep Inductive Alignment of Noisy Geometric Annotations</a></li><li><a href="https://arxiv.org/abs/1709.06429">Neural Networks for Text Correction and Completion in Keyboard Decoding</a></li></ul><p>In this blog post, we’ll look into a simple and intuitive auto-correct system based on probabilities. More details in the next sections.</p><blockquote>View this project on -&gt; <a href="https://www.kaggle.com/pashupatigupta/project-statistical-auto-correct-system">Kaggle Link</a>, <a href="https://github.com/pashupati98/statistical-autocorrect">GitHub Link</a></blockquote><p>Note: If you are interested in looking into the code parallelly, I highly recommend viewing it on Kaggle.</p><h4><strong>Data Requirement</strong></h4><p>The very first requirement of the auto-correct system that we’ll be building is data. We need a trusted (good English and grammar) text corpus. There are some public domain text corpuses available. Since it’s an unsupervised type of problem here what we need is just text. One can use any competition data or any other public dataset that has a text field column. In the current version, I have used a small fraction of the wiki corpus.</p><h3><strong>Architecture</strong></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*ZpPa5v65cb2zSEzBW8wYMA.png" /></figure><p>This auto-correct architecture has 4 components -<br>1) <strong>Filtering Misspells</strong>: One simple approach could be checking if a word is there in the vocabulary or not. We’ll use the text corpus to define a vocabulary.<br>2) <strong>Word Suggestion Mechanism</strong>: This mechanism suggests candidate words based on deletion, insertion, switch, or replacement of one/two characters in the original word.<br>3) <strong>Probability Distribution Mechanism</strong>: The probability distribution {key(word) : value(probability)} is created calculated using the text corpus. The probability of each candidate is found using this distribution and the most probable candidate is the final one.<br>4) <strong>Replace Misspells</strong>: Simply replace the misspelled word with the most probable suggestion.</p><p><strong><em>This is a very simplified architecture compared to what is used in reality. Hence its performance will not be very good. But we can make some improvements in this architecture to get better results.</em></strong></p><p><strong>Drawbacks </strong><br>- It has a fixed outcome. i.e. ‘hime’ will be converted to ‘time’ only (because ‘time’ is a more frequent word hence a more probable one) not ‘home’ or anything else.<br>- It is solely based on the frequency of words in the corpus.<br>- Doesn’t care about the context.<br>- Doesn’t care about part of speech, grammar, or punctuations.<br>- Can’t suggest something which is not in the vocabulary</p><p><strong>Improvements</strong><br>1) It can be further improved by introducing bi-gram probabilities. Hence, it will get some inference from previous words.<br>2) The suggestions that are less distance away from the misspelled word are more likely. Hence, the system can be further improved by introducing dynamic programming-based min edit distance functionality.</p><h4><strong>Improvement 1: Introducing n-gram probabilities to get context from previous words</strong></h4><p>This idea is taken from the n-grams based language models. In an n-gram language model<br>- Assume the probability of the next word depends only on the previous n-gram.<br>- The previous n-gram is the series of the previous ’n’ words.</p><p>The conditional probability for the word at position ‘t’ in the sentence, given that the words preceding it are w(t-1), w(t-2) ….. w(t-n) is:</p><blockquote>P( w(t) | w(t-1)….w(t-n) )</blockquote><p>This probability can be estimated by counting the occurrences of these series of words in the training data.<br>- The probability can be estimated as a ratio, where<br>- The numerator is the number of times word (t) appears after words (t-1) through (t-n) appear in the training data.<br>- The denominator is the number of times word t-1 through t-n appears in the training data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/406/1*pedcx6d1HbDFIGcNQ-z8vw.png" /></figure><p>In other words, to estimate probabilities based on n-grams, first, find the counts of n-grams (for the denominator) then divide it by the count of (n+1)-grams (for numerator).</p><p>- The function <strong>C(….)</strong> denotes the number of occurrences of the given sequence. <br>- <strong>P hat</strong> means the estimation of <strong>P</strong>. <br>- The denominator of the above equation is the number of occurrences of the previous <strong>n</strong> words, and the numerator is the same sequence followed by the word <strong>w(t)</strong>.</p><p>Now the issue with the above formula is that it doesn’t work when a count of an n-gram is zero.<br>- Suppose we encounter an n-gram that did not occur in the training data. <br>- Then, the later equation cannot be evaluated (it becomes zero divided by zero).</p><p>A way to handle zero counts is to add k-smoothing. <br>- K-smoothing adds a positive constant <strong>k</strong> to each numerator and <strong>k * |V|</strong> in the denominator, where <strong>|V|</strong> is the number of words in the vocabulary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/462/1*8pto83WCqfC9ziCW7nTNag.png" /></figure><p>For n-grams that have a zero count, the above equation becomes <strong>1/|V|</strong>.<br>- This means that any n-gram with zero counts has the same probability of <strong>1/|V|</strong>.</p><h4>Improvement 2 : Introducing min_edit_diatsnce functionality</h4><p>The idea is derived from the intuition that the suggestions that are less distance away from the misspelled word are more likely. Hence, the system can be further improved by introducing dynamic programming-based min edit distance functionality.</p><p>So, given a string source[0..i] and a string target[0..j], we will compute all the combinations of substrings[i, j] and calculate their edit distance. To do this efficiently, we will use a table to maintain the previously computed substrings and use those to calculate larger substrings.</p><p>We’ll first create a matrix and update each element in the matrix as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*KNt1nEKh3hTZCrrpzG9-MA.png" /></figure><p><strong>Implementing these improvements will make our auto-correct system better</strong>. Let’s look at a demo of how well it is working.</p><h4>Demo</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/617/1*jnyoyVnTTjC1560oGicEdA.png" /></figure><h4>Evaluation</h4><p>Let’s do a unit testing of this system to see how accurately it is performing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/519/1*5vnf5WgQNAWsK6tcy6SnKg.png" /></figure><h4>Conclusion</h4><p>This project is an implementation of a statistical auto-correct system. The architecture that has been developed gives the accuracy around 52% — 55%. The improved version of this architecture could get inference from the previous words by using bi-gram probabilities and min edit distance functionality provided further enhancement. Overall, this simple probability-based auto-correct system performed okay. In order to get better performance one can go for deep learning-based auto-correct systems.</p><p>References</p><ul><li><a href="https://norvig.com/spell-correct.html">How to Write a Spelling Corrector</a></li><li><a href="https://www.coursera.org/learn/probabilistic-models-in-nlp/home/welcome">Coursera NLP Specialization</a></li></ul><p>PS: <strong>Thanks</strong> for reading through this. Any suggestion, feedback, criticism, appreciation is most welcome. If interested, follow me on <a href="https://www.kaggle.com/pashupatigupta">Kaggle</a>, <a href="https://www.linkedin.com/in/pashupati-gupta-372b2312a/">Linked In</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e891ddfe141" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Text Classification model using BiLSTM]]></title>
            <link>https://medium.com/analytics-vidhya/building-a-text-classification-model-using-bilstm-c0548ace26f2?source=rss-57485a806d0------2</link>
            <guid isPermaLink="false">https://medium.com/p/c0548ace26f2</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[lstm]]></category>
            <category><![CDATA[text-classification]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Pashupati Gupta]]></dc:creator>
            <pubDate>Mon, 01 Jun 2020 15:25:33 GMT</pubDate>
            <atom:updated>2020-06-04T16:59:10.958Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uxC3xMt5H4pk_92aytZeZg.jpeg" /><figcaption>Credits : <a href="https://enterprisetalk.com/featured/ibm-integrates-watson-platform-in-project-debater-nlp-technology-2/">EnterpriseTalk</a></figcaption></figure><p>Text classification is one of the fundamental tasks in NLP. Almost every NLP system uses text classification somewhere in its backend. For example - The intent classifier of a chatbot, named-entity recognition, auto-tagging, etc.</p><p>There are many approaches to this problem from statistical machine learning models (Logistic, Naive Bays, SVM, etc.) to high-end deep learning models (CNN, RNN, Transformers, etc.). This blog covers the practical aspects (coding) of building a text classification model using a recurrent neural network (BiLSTM). We will use Python and <a href="https://jupyter.org/">Jupyter Notebook</a> along with several libraries to build an <strong>offensive language/text classification</strong> model. It has three parts.</p><ol><li>Data Preparation</li><li>Model Building</li><li>Training and Evaluation</li></ol><h3>Data Preparation</h3><p>Selecting the right dataset is the key to build a state-of-the-art model but finding/making one is a hectic task. Fortunately, we have a readily available dataset for our task released by Harvard University.</p><p>Download Offensive Language Identification Dataset (OLID) <a href="https://scholar.harvard.edu/malmasi/olid">from here</a>. You can also use any other dataset for this task. Let’s look at the dataset using the Pandas library.</p><pre>import pandas as pd<br>url = &#39;olid-training-v1.0.tsv&#39;<br>df = pd.read_csv(url, sep=&quot;\t&quot;)<br>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/1*DaTvDHsu-my0N1RY4-20Yg.png" /><figcaption>originally downloaded dataset</figcaption></figure><p>This is dataset is a collection of tweets classified as offensive (OFF) and non-offensive (NOT) in the column ‘subtask_a’. We need only ‘tweet’ and ‘subtask_a’ column renamed as ‘label’.</p><pre>del df[&#39;subtask_b&#39;]<br>del df[&#39;subtask_c&#39;]<br>del df[&#39;id&#39;]<br>df.columns = [&#39;tweet&#39;, &#39;label&#39;]<br>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/508/1*f5MAFLsgXTy_pcfzcdErwg.png" /><figcaption>modified dataset</figcaption></figure><p>The next step of data preparation is processing the tweets. We will use NLTK, BS4, Contractions, and Regex(re) libraries for this. Let’s import the libraries and write the required functions. Each function’s objective is commented on in the function itself. The first function is denoise_text used to remove noise from the text.</p><pre>#importing required libraries</pre><pre>import nltk<br>import inflect<br>import contractions<br>from bs4 import BeautifulSoup<br>import re, string, unicodedata<br>from nltk import word_tokenize, sent_tokenize<br>from nltk.corpus import stopwords<br>from nltk.stem import LancasterStemmer, WordNetLemmatizer<br>from sklearn.preprocessing import LabelEncoder</pre><pre># First function is used to denoise text<br>def denoise_text(text):<br>    # Strip html if any. For ex. removing &lt;html&gt;, &lt;p&gt; tags<br>    soup = BeautifulSoup(text, &quot;html.parser&quot;)<br>    text = soup.get_text()<br>    # Replace contractions in the text. For ex. didn&#39;t -&gt; did not<br>    text = contractions.fix(text)<br>    return text</pre><pre># Check the function <br>sample_text = &quot;&lt;p&gt;he didn&#39;t say anything &lt;/br&gt; about what&#39;s gonna &lt;html&gt; happen in the climax&quot;<br>denoise_text(sample_text)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/793/1*4q78Qk85Dl9ZwIrYShbK8Q.png" /></figure><p>The next function is normalize_text which includes many steps. A separate function is written for each step and then they are compiled in one function.</p><pre># Text normalization includes many steps.<br># Each function below serves a step.</pre><pre>def remove_non_ascii(words):<br>    &quot;&quot;&quot;Remove non-ASCII characters from list of tokenized words&quot;&quot;&quot;<br>    new_words = []<br>    for word in words:<br>        new_word = unicodedata.normalize(&#39;NFKD&#39;, word).encode(&#39;ascii&#39;, &#39;ignore&#39;).decode(&#39;utf-8&#39;, &#39;ignore&#39;)<br>        new_words.append(new_word)<br>    return new_words</pre><pre>def to_lowercase(words):<br>    &quot;&quot;&quot;Convert all characters to lowercase from list of tokenized words&quot;&quot;&quot;<br>    new_words = []<br>    for word in words:<br>        new_word = word.lower()<br>        new_words.append(new_word)<br>    return new_words</pre><pre>def remove_punctuation(words):<br>    &quot;&quot;&quot;Remove punctuation from list of tokenized words&quot;&quot;&quot;<br>    new_words = []<br>    for word in words:<br>        new_word = re.sub(r&#39;[^\w\s]&#39;, &#39;&#39;, word)<br>        if new_word != &#39;&#39;:<br>            new_words.append(new_word)<br>    return new_words</pre><pre>def replace_numbers(words):<br>    &quot;&quot;&quot;Replace all interger occurrences in list of tokenized words with textual representation&quot;&quot;&quot;<br>    p = inflect.engine()<br>    new_words = []<br>    for word in words:<br>        if word.isdigit():<br>            new_word = p.number_to_words(word)<br>            new_words.append(new_word)<br>        else:<br>            new_words.append(word)<br>    return new_words</pre><pre>def remove_stopwords(words):<br>    &quot;&quot;&quot;Remove stop words from list of tokenized words&quot;&quot;&quot;<br>    new_words = []<br>    for word in words:<br>        if word not in stopwords.words(&#39;english&#39;):<br>            new_words.append(word)<br>    return new_words</pre><pre>def stem_words(words):<br>    &quot;&quot;&quot;Stem words in list of tokenized words&quot;&quot;&quot;<br>    stemmer = LancasterStemmer()<br>    stems = []<br>    for word in words:<br>        stem = stemmer.stem(word)<br>        stems.append(stem)<br>    return stems</pre><pre>def lemmatize_verbs(words):<br>    &quot;&quot;&quot;Lemmatize verbs in list of tokenized words&quot;&quot;&quot;<br>    lemmatizer = WordNetLemmatizer()<br>    lemmas = []<br>    for word in words:<br>        lemma = lemmatizer.lemmatize(word, pos=&#39;v&#39;)<br>        lemmas.append(lemma)<br>    return lemmas</pre><pre>def normalize_text(words):<br>    words = remove_non_ascii(words)<br>    words = to_lowercase(words)<br>    words = remove_punctuation(words)<br>    words = replace_numbers(words)<br>    words = remove_stopwords(words)<br>    #words = stem_words(words)<br>    words = lemmetize_verbs(words)<br>    return words</pre><pre># Testing the functions<br>print(&quot;remove_non_ascii results: &quot;, remove_non_ascii([&#39;h&#39;, &#39;ॐ&#39;, &#39;©&#39;, &#39;1&#39;]))<br>print(&quot;to_lowercase results: &quot;, to_lowercase([&#39;HELLO&#39;, &#39;hiDDen&#39;, &#39;wanT&#39;, &#39;GOING&#39;]))<br>print(&quot;remove_punctuation results: &quot;, remove_punctuation([&#39;hello!!&#39;, &#39;how?&#39;, &#39;done,&#39;]))<br>print(&quot;replace_numbers results: &quot;, replace_numbers([&#39;1&#39;, &#39;2&#39;, &#39;3&#39;]))<br>print(&quot;remove_stopwords results: &quot;, remove_stopwords([&#39;this&#39;, &#39;and&#39;, &#39;amazing&#39;]))<br>print(&quot;stem_words results: &quot;, stem_words([&#39;beautiful&#39;, &#39;flying&#39;, &#39;waited&#39;]))<br>print(&quot;lemmatize_verbs results: &quot;, lemmatize_verbs([&#39;hidden&#39;, &#39;walking&#39;, &#39;ran&#39;]))<br>print(&quot;normalize_text results: &quot;, normalize_text([&#39;hidden&#39;, &#39;in&#39;, &#39;the&#39;, &#39;CAVES&#39;, &#39;he&#39;, &#39;WAited&#39;, &#39;2&#39;, &#39;ॐ&#39;, &#39;hours!!&#39;]))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*dnh7BhU7aWolSqEPnrZdjw.png" /></figure><p>We can see that each function works correctly. You may have noticed that each function takes a list of tokens (words) as an argument. So, we need a function to tokenize the text. We will use word_tokenize from NLTK for this.</p><pre># Tokenize tweet into words<br>def tokenize(text):<br>    return nltk.word_tokenize(text)</pre><pre># check the function<br>sample_text = &#39;he did not say anything  about what is going to  happen&#39;<br>print(&quot;tokenize results :&quot;, tokenize(sample_text))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*60V2Rh--iSxuioPfxny4dw.png" /></figure><p>This also works fine. Now, we have all the functions required for processing the tweets. Let’s apply them to our dataset using a text_prepare function. We will also LabelEncode our target variable in this step.</p><pre>def text_prepare(text):<br>    text = denoise_text(text)<br>    text = &#39; &#39;.join([x for x in normalize_text(tokenize(text))])<br>    return text</pre><pre>df[&#39;tweet&#39;] = [text_prepare(x) for x in df[&#39;tweet&#39;]]<br>le = LabelEncoder()<br>df[&#39;label&#39;] = le.fit_transform(df[&#39;label&#39;])<br>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/1*ilSb_iycXUJJ1UiVb-3uzg.png" /><figcaption>processed dataset</figcaption></figure><p>Yay! This looks great. We are done with the data preparation step. Note that I haven’t used stem_words function while normalizing the text since it leads to better results in this particular case. Let’s move to the model building.</p><h3>Model Building</h3><p>Model building is a crucial step but thanks to deep learning frameworks that made it easier. We will use <a href="https://keras.io/">Keras</a> library to build a recurrent neural network based on bidirectional LSTMs. Read about LSTMs <a href="https://colah.github.io/posts/2015-08-Understanding-LSTMs/">here</a>. These models take <a href="https://machinelearningmastery.com/what-are-word-embeddings/">word embeddings</a> as input so we will use pre-trained <a href="https://nlp.stanford.edu/pubs/glove.pdf">GloVe embeddings</a> to make the embedding dictionary. Download glove embeddings from <a href="https://nlp.stanford.edu/projects/glove/">here</a>.</p><p>Before doing so we first need to tokenize our whole text and turn them into sequences where each word is assigned to an integer. Also, all the sequences must be of the same length so we need to pad the sequences. Let’s write a function which will take X_train, X_test, MAX_NB_WORDS (maximum number of words in the vocabulary), MAX_SEQUENCE_LENGTH (maximum length of text sequences) as input and will perform the above-mentioned steps to build the embedding dictionary. First, import all required libraries.</p><pre>from keras.layers import Dropout, Dense, Embedding, LSTM, Bidirectional<br>from keras.preprocessing.text import Tokenizer<br>from keras.preprocessing.sequence import pad_sequences<br>from keras.models import Sequential<br>from sklearn.metrics import matthews_corrcoef, confusion_matrix<br>from sklearn.model_selection import train_test_split<br>from sklearn import metrics<br>from sklearn.utils import shuffle<br>import numpy as np<br>import pickle<br>import matplotlib.pyplot as plt<br>import warnings<br>import logging<br>logging.basicConfig(level=logging.INFO)</pre><p>Now the implement a prepare_model_input function as below. This returns X_train_Glove, X_test_Glove, word_index (word assigned with integers), and the embedding_dict.</p><pre>def prepare_model_input(X_train, X_test,MAX_NB_WORDS=75000,MAX_SEQUENCE_LENGTH=500):<br>    np.random.seed(7)<br>    text = np.concatenate((X_train, X_test), axis=0)<br>    text = np.array(text)<br>    tokenizer = Tokenizer(num_words=MAX_NB_WORDS)<br>    tokenizer.fit_on_texts(text)<br>    # pickle.dump(tokenizer, open(&#39;text_tokenizer.pkl&#39;, &#39;wb&#39;))<br>    # Uncomment above line to save the tokenizer as .pkl file <br>    sequences = tokenizer.texts_to_sequences(text)<br>    word_index = tokenizer.word_index<br>    text = pad_sequences(sequences, maxlen=MAX_SEQUENCE_LENGTH)<br>    print(&#39;Found %s unique tokens.&#39; % len(word_index))<br>    indices = np.arange(text.shape[0])<br>    # np.random.shuffle(indices)<br>    text = text[indices]<br>    print(text.shape)<br>    X_train_Glove = text[0:len(X_train), ]<br>    X_test_Glove = text[len(X_train):, ]<br>    embeddings_dict = {}<br>    f = open(&quot;glove.6B.50d.txt&quot;, encoding=&quot;utf8&quot;)<br>    for line in f:<br>        values = line.split()<br>        word = values[0]<br>        try:<br>            coefs = np.asarray(values[1:], dtype=&#39;float32&#39;)<br>        except:<br>            pass<br>        embeddings_dict[word] = coefs<br>    f.close()<br>    print(&#39;Total %s word vectors.&#39; % len(embeddings_dict))<br>    return (X_train_Glove, X_test_Glove, word_index, embeddings_dict)</pre><pre>## Check function<br>x_train_sample = [&quot;Lorem Ipsum is simply dummy text of the printing and typesetting industry&quot;, &quot;It is a long established fact that a reader will be distracted by the readable content of a page when looking at its layout&quot;]<br>x_test_sample = [&quot;I’m creating a macro and need some text for testing purposes&quot;, &quot;I’m designing a document and don’t want to get bogged down in what the text actually says&quot;]<br>X_train_Glove_s, X_test_Glove_s, word_index_s, embeddings_dict_s = prepare_model_input(x_train_sample, x_test_sample, 100, 20)<br>print(&quot;\n X_train_Glove_s \n &quot;, X_train_Glove_s)<br>print(&quot;\n X_test_Glove_s \n &quot;, X_test_Glove_s)<br>print(&quot;\n Word index of the word testing is : &quot;, word_index_s[&quot;testing&quot;])<br>print(&quot;\n Embedding for thw word want \n \n&quot;, embeddings_dict_s[&quot;want&quot;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/690/1*yDfIMSc7PBcsipRq3RFzaA.png" /></figure><p>Everything is working fine. Now, let’s implement a build_bilstms helper function that will return the BiLSTM model. We will use Embedding, Dense, Dropout, LSTM, Bidirectional layers from keras.layers to build a sequential model.</p><pre>def build_bilstm(word_index, embeddings_dict, nclasses,  MAX_SEQUENCE_LENGTH=500, EMBEDDING_DIM=50, dropout=0.5, hidden_layer = 3, lstm_node = 32):<br>    # Initialize a sequebtial model<br>    model = Sequential()<br>    # Make the embedding matrix using the embedding_dict<br>    embedding_matrix = np.random.random((len(word_index) + 1, EMBEDDING_DIM))<br>    for word, i in word_index.items():<br>        embedding_vector = embeddings_dict.get(word)<br>        if embedding_vector is not None:<br>            # words not found in embedding index will be all-zeros.<br>            if len(embedding_matrix[i]) != len(embedding_vector):<br>                print(&quot;could not broadcast input array from shape&quot;, str(len(embedding_matrix[i])),<br>                      &quot;into shape&quot;, str(len(embedding_vector)), &quot; Please make sure your&quot;<br>                                                                &quot; EMBEDDING_DIM is equal to embedding_vector file ,GloVe,&quot;)<br>                exit(1)<br>            embedding_matrix[i] = embedding_vector<br>            <br>    # Add embedding layer<br>    model.add(Embedding(len(word_index) + 1,<br>                                EMBEDDING_DIM,<br>                                weights=[embedding_matrix],<br>                                input_length=MAX_SEQUENCE_LENGTH,<br>                                trainable=True))<br>    # Add hidden layers <br>    for i in range(0,hidden_layer):<br>        # Add a bidirectional lstm layer<br>        model.add(Bidirectional(LSTM(lstm_node, return_sequences=True, recurrent_dropout=0.2)))<br>        # Add a dropout layer after each lstm layer<br>        model.add(Dropout(dropout))<br>    model.add(Bidirectional(LSTM(lstm_node, recurrent_dropout=0.2)))<br>    model.add(Dropout(dropout))<br>    # Add the fully connected layer with 256 nurons and relu activation<br>    model.add(Dense(256, activation=&#39;relu&#39;))<br>    # Add the output layer with softmax activation since we have 2 classes<br>    model.add(Dense(nclasses, activation=&#39;softmax&#39;))<br>    # Compile the model using sparse_categorical_crossentropy<br>    model.compile(loss=&#39;sparse_categorical_crossentropy&#39;,<br>                      optimizer=&#39;adam&#39;,<br>                      metrics=[&#39;accuracy&#39;])<br>    return model</pre><p>We have implemented the helper function to build the model. Let’s build the actual model for our task.</p><pre>X = df.tweet<br>y = df.label<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2)<br>print(&quot;Preparing model input ...&quot;)<br>X_train_Glove, X_test_Glove, word_index, embeddings_dict = prepare_model_input(X_train,X_test)<br>print(&quot;Done!&quot;)<br>print(&quot;Building Model!&quot;)<br>model = build_bilstm(word_index, embeddings_dict, 2)<br>model.summary()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*e-HwXpOHDihrPMFRfSsH2Q.png" /><figcaption>summary of the model</figcaption></figure><p>Yay again! we are done with the model-building part. Let’s train and evaluate the model.</p><h3>Training and Evaluation</h3><p>Training is the part where one can experiment and find best hyper-parameters for his/her model. First, let’s implement some utility functions for training and checking the model performance.</p><pre>def get_eval_report(labels, preds):<br>    mcc = matthews_corrcoef(labels, preds)<br>    tn, fp, fn, tp = confusion_matrix(labels, preds).ravel()<br>    precision = (tp)/(tp+fp)<br>    recall = (tp)/(tp+fn)<br>    f1 = (2*(precision*recall))/(precision+recall)<br>    return {<br>        &quot;mcc&quot;: mcc,<br>        &quot;true positive&quot;: tp,<br>        &quot;true negative&quot;: tn,<br>        &quot;false positive&quot;: fp,<br>        &quot;false negative&quot;: fn,<br>        &quot;pricision&quot; : precision,<br>        &quot;recall&quot; : recall,<br>        &quot;F1&quot; : f1,<br>        &quot;accuracy&quot;: (tp+tn)/(tp+tn+fp+fn)<br>    }</pre><pre>def compute_metrics(labels, preds):<br>    assert len(preds) == len(labels)<br>    return get_eval_report(labels, preds)</pre><pre>def plot_graphs(history, string):<br>  plt.plot(history.history[string])<br>  plt.plot(history.history[&#39;val_&#39;+string], &#39;&#39;)<br>  plt.xlabel(&quot;Epochs&quot;)<br>  plt.ylabel(string)<br>  plt.legend([string, &#39;val_&#39;+string])<br>  plt.show()</pre><p>Now, we are ready for training the model. We will train our model for 5 epochs with a batch size of 128.</p><pre>history = model.fit(X_train_Glove, y_train,<br>                           validation_data=(X_test_Glove,y_test),<br>                           epochs=5,<br>                           batch_size=128,<br>                           verbose=1)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ucz7rDhfXTkVH8ows_kByg.png" /><figcaption>model training logs</figcaption></figure><p>Well, the accuracy achieved by our model seems good but let’s look at the graph of loss and accuracy to see clearly what&#39;s going on.</p><pre>plot_graphs(history, &#39;accuracy&#39;)<br>plot_graphs(history, &#39;loss&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/489/1*shzVMFtKyScnSJP8nIJgrg.png" /></figure><p>In the accuracy vs epochs graph, we observe that validation accuracy is maintained around 0.74 whereas training accuracy increases continuously. In the loss vs epoch graph as well validation loss is maintained around 0.50 whereas training loss decreases continuously. This is a sign of slight overfitting. We can reduce the complexity of the model, increase dropout probability, and use regularization to reduce overfitting. Let’s finally evaluates the model we just trained.</p><pre>print(&quot;\n Evaluating Model ... \n&quot;)<br>predicted = model.predict_classes(X_test_Glove)<br>print(metrics.classification_report(y_test, predicted))<br>print(&quot;\n&quot;)<br>logger = logging.getLogger(&quot;logger&quot;)<br>result = compute_metrics(y_test, predicted)<br>for key in (result.keys()):<br>    logger.info(&quot;  %s = %s&quot;, key, str(result[key]))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*kbp6iCizqrMp9-F84K4kkw.png" /><figcaption>evaluation results</figcaption></figure><p>We can see that despite being slightly overfitted the model performs pretty well on test data with an accuracy of 75% and F1-score 0.62.</p><p>Finally Yay! we have successfully built and trained a BiLSTM model for text classification. Kudos! Find all the codes in this <a href="https://github.com/pashupati98/Text-Classification">GitHub repository</a>.</p><p><em>Optional</em> -: You may consider saving the model and tokenizer as a .pkl file for deployment purposes. (Next part of this blog)</p><pre>#To save the tokenizer follow instructions in prepare_model_input function i.e. uncomment this line #pickle.dump(tokenizer, open(&#39;text_tokenizer.pkl&#39;, &#39;wb&#39;)) in that function</pre><pre># To save the model run this line<br>pickle.dump(model, open(&#39;model.pkl&#39;, &#39;wb&#39;))</pre><pre># you are ready for deployment!</pre><p>PS :- Thanks for reading till here. Any correction, critic, or compliment is most welcomed.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c0548ace26f2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/building-a-text-classification-model-using-bilstm-c0548ace26f2">Building a Text Classification model using BiLSTM</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>