<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nimesh Sinha on Medium]]></title>
        <description><![CDATA[Stories by Nimesh Sinha on Medium]]></description>
        <link>https://medium.com/@nimesh280?source=rss-f3a561171127------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*2lsrnSkXFL6celMtqlOlWA.jpeg</url>
            <title>Stories by Nimesh Sinha on Medium</title>
            <link>https://medium.com/@nimesh280?source=rss-f3a561171127------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 01:43:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nimesh280/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Understanding LSTM and its quick implementation in keras for sentiment analysis.]]></title>
            <link>https://medium.com/data-science/understanding-lstm-and-its-quick-implementation-in-keras-for-sentiment-analysis-af410fd85b47?source=rss-f3a561171127------2</link>
            <guid isPermaLink="false">https://medium.com/p/af410fd85b47</guid>
            <category><![CDATA[towards-data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[lstm]]></category>
            <category><![CDATA[neural-networks]]></category>
            <dc:creator><![CDATA[Nimesh Sinha]]></dc:creator>
            <pubDate>Mon, 19 Feb 2018 22:39:26 GMT</pubDate>
            <atom:updated>2018-03-03T04:41:05.571Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*voNageVB1gI8Dfrmr-U3Ew.png" /></figure><p>Long Short Term Memory networks, usually called “LSTMs” , were introduced by Hochreiter and Schmiduber. These have widely been used for speech recognition, language modeling, sentiment analysis and text prediction. Before going deep into LSTM, we should first understand the need of LSTM which can be explained by the drawback of practical use of Recurrent Neural Network (RNN). So, lets start with RNN.</p><h3>Recurrent Neural Networks (RNN)</h3><p>Being human, when we watch a movie, we don’t think from scratch every time while understanding any event. We rely on the recent experiences happening in the movie and learn from them. But, a conventional neural network is unable to learn from the previous events because the information does not pass from one step to the next. On contrary, RNN learns information from immediate previous step.</p><p>For example, there is a scene in a movie where a person is in a basketball court. We will improvise the basketball activities in the future frames: an image of someone running and jumping probably be labeled as <em>playing basketball</em>, and an image of someone sitting and watching is probably <em>a spectator watching the game.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/129/1*DvlB9rtndUHwtri4E2P-bg.png" /><figcaption>A typical RNN (Source: <a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/">http://colah.github.io/posts/2015-08-Understanding-LSTMs/</a>)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*xTKE0g6XNMLM8IQ4aFdP0w.png" /></figure><p>A typical RNN looks like above-where X(t) is input, h(t) is output and A is the neural network which gains information from the previous step in a loop. The output of one unit goes into the next one and the information is passed.</p><p>But, sometimes we don’t need our network to learn only from immediate past information. Suppose we want to predict the blank word in the text ‘ David, a 36-year old man lives in San Francisco. He has a female friend Maria. Maria works as a cook in a famous restaurant in New York whom he met recently in a school alumni meet. Maria told him that she always had a passion for _________ . Here, we want our network to learn from dependency ‘cook’ to predict ‘cooking. There is a gap between the information what we want to predict and from where we want it to get predicted . This is called long-term dependency. We can say that anything larger than trigram as a long term dependency. Unfortunately, RNN does not work practically in this situation.</p><h3>Why RNN does not work practically</h3><p>During the training of RNN, as the information goes in loop again and again which results in very large updates to neural network model weights. This is due to the accumulation of error gradients during an update and hence, results in an unstable network. At an extreme, the values of weights can become so large as to overflow and result in NaN values.The explosion occurs through exponential growth by repeatedly multiplying gradients through the network layers that have values larger than 1 or vanishing occurs if the values are less than 1.</p><h3>Long Short Term Memory</h3><p>The above drawback of RNN pushed the scientists to develop and invent a new variant of the RNN model, called Long Short Term Memory. LSTM can solve this problem, because it uses gates to control the memorizing process.</p><p>Let’s understand the architecture of LSTM and compare it with that of RNN:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/1*Niu_c_FhGtLuHjrStkB_4Q.png" /><figcaption>A LSTM unit (Source : <a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/">http://colah.github.io/posts/2015-08-Understanding-LSTMs</a>)</figcaption></figure><p>The symbols used here have following meaning:</p><p>a) X : Scaling of information</p><p>b)+ : Adding information</p><p>c) σ : Sigmoid layer</p><p>d) tanh: tanh layer</p><p>e) h(t-1) : Output of last LSTM unit</p><p>f) c(t-1) : Memory from last LSTM unit</p><p>g) X(t) : Current input</p><p>h) c(t) : New updated memory</p><p>i) h(t) : Current output</p><h3>Why tanh?</h3><p>To overcome the vanishing gradient problem, we need a function whose second derivative can sustain for a long range before going to zero. <em>tanh</em> is a suitable function with the above property.</p><h3>Why Sigmoid?</h3><p>As Sigmoid can output 0 or 1, it can be used to forget or remember the information.</p><p>Information passes through many such LSTM units.There are three main components of an LSTM unit which are labeled in the diagram:</p><ol><li>LSTM has a special architecture which enables it to forget the unnecessary information .The sigmoid layer takes the input X(t) and h(t-1) and decides which parts from old output should be removed (by outputting a 0). In our example, when the input is ‘He has a female friend Maria’, the gender of ‘David’ can be forgotten because the subject has changed to ‘Maria’. This gate is called forget gate f(t). The output of this gate is f(t)*c(t-1).</li><li>The next step is to decide and store information from the new input X(t) in the cell state. A Sigmoid layer decides which of the new information should be updated or ignored. A <em>tanh</em> layer creates a vector of all the possible values from the new input. These two are multiplied to update the new cell sate. This new memory is then added to old memory c(t-1) to give c(t). In our example, for the new input ‘ He has a female friend Maria’, the gender of Maria will be updated. When the input is ‘Maria works as a cook in a famous restaurant in New York whom he met recently in a school alumni meet’, the words like ‘famous’, ‘school alumni meet’ can be ignored and words like ‘cook, ‘restaurant’ and ‘New York’ will be updated.</li><li>Finally, we need to decide what we’re going to output. A sigmoid layer decides which parts of the cell state we are going to output. Then, we put the cell state through a <em>tanh</em> generating all the possible values and multiply it by the output of the sigmoid gate, so that we only output the parts we decided to. In our example, we want to predict the blank word, our model knows that it is a noun related to ‘cook’ from its memory, it can easily answer it as ‘cooking’. Our model does not learn this answer from the immediate dependency, rather it learnt it from long term dependency.</li></ol><p>We just saw that there is a big difference in the architecture of a typical RNN and a LSTM. In LSTM, our model learns what information to store in long term memory and what to get rid of.</p><h3>Quick implementation of LSTM for Sentimental Analysis</h3><p>Here, I used LSTM on the reviews data from <a href="https://www.yelp.com/dataset">Yelp open dataset</a> for sentiment analysis using keras.</p><p>This is what my data looks like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pOZ3Dcc27oPlbt6dE-FbSg.png" /><figcaption>Dataset</figcaption></figure><p>I used Tokenizer to vectorize the text and convert it into sequence of integers after restricting the tokenizer to use only top most common 2500 words. I used pad_sequences to convert the sequences into 2-D numpy array.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4bbe47ce2c1381b9a24822d63cb6210c/href">https://medium.com/media/4bbe47ce2c1381b9a24822d63cb6210c/href</a></iframe><p>Then, I built my LSTM network.There are a few hyper parameters:</p><ol><li>embed_dim : The embedding layer encodes the input sequence<br>into a sequence of dense vectors of dimension embed_dim.</li><li>lstm_out : The LSTM transforms the vector sequence into a single vector of size lstm_out, containing information about the entire sequence.</li></ol><p>The other hyper parameters like dropout, batch_size are similar to that of CNN.</p><p>I used softmax as activation function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/317/1*GsT2fwE_39fgmtC735924Q.png" /><figcaption>LSTM network</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6c8f8fdc71929378c9e1f36eda4370ce/href">https://medium.com/media/6c8f8fdc71929378c9e1f36eda4370ce/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PCpvPF7eBi07rRc0ayFqXw.png" /></figure><p>Now, I fit my model on training set and check the accuracy on validation set.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5a15f23f41270d3370aeedd4bcd9b1cf/href">https://medium.com/media/5a15f23f41270d3370aeedd4bcd9b1cf/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/1*XfPXSNqVb3vc5_jTRl-Q3w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dmZz0dp17eqlcrAWFtk2AA.png" /></figure><p>I got a validation accuracy of 86% in just one epoch while running on a small dataset which includes all the businesses.</p><h3>Future Work:</h3><ol><li>We can filter the specific businesses like restaurants and then use LSTM for sentiment analysis.</li><li>We can use much larger dataset with more epochs to increase the accuracy.</li><li>More hidden dense layers can be used to improve the accuracy. We can tune other hyper parameters as well.</li></ol><h3>Conclusion</h3><p>LSTM outperforms the other models when we want our model to learn from long term dependencies. LSTM’s ability to forget, remember and update the information pushes it one step ahead of RNNs.</p><h3>References and other useful resources:</h3><ol><li><a href="https://github.com/nsinha280/lstm">My Github repo</a></li><li><a href="http://colah.github.io/posts/2015-08-Understanding-LSTMs/">Understanding LSTM</a></li><li><a href="https://deeplearning4j.org/lstm.html">Beginner’s guide to RNN and LSTM</a></li></ol><p>4. <a href="http://blog.echen.me/2017/05/30/exploring-lstms/">Exploring LSTMs</a></p><p>5. <a href="http://www.bioinf.jku.at/publications/older/2604.pdf">Research paper on LSTM</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af410fd85b47" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/understanding-lstm-and-its-quick-implementation-in-keras-for-sentiment-analysis-af410fd85b47">Understanding LSTM and its quick implementation in keras for sentiment analysis.</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>