<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by SWAYAM MITTAL on Medium]]></title>
        <description><![CDATA[Stories by SWAYAM MITTAL on Medium]]></description>
        <link>https://medium.com/@swayammittal65_2667?source=rss-84cdd99683f9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*qiSVvQdpdiVEBD9DOctP4w@2x.jpeg</url>
            <title>Stories by SWAYAM MITTAL on Medium</title>
            <link>https://medium.com/@swayammittal65_2667?source=rss-84cdd99683f9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 19:51:01 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@swayammittal65_2667/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Deep Learning Techniques for Text Classification]]></title>
            <link>https://medium.datadriveninvestor.com/deep-learning-techniques-for-text-classification-9392ca9492c7?source=rss-84cdd99683f9------2</link>
            <guid isPermaLink="false">https://medium.com/p/9392ca9492c7</guid>
            <category><![CDATA[nltk]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[keras]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[SWAYAM MITTAL]]></dc:creator>
            <pubDate>Fri, 16 Aug 2019 19:21:03 GMT</pubDate>
            <atom:updated>2019-08-18T11:21:28.331Z</atom:updated>
            <content:encoded><![CDATA[<p>The exponential growth in the number of complex datasets every year requires more enhancement in machine learning methods to provide robust and accurate data classification. Lately, deep learning approaches are achieving better results compared to previous machine learning algorithms on tasks like image classification, natural language processing, face recognition, etc. The success of these deep learning algorithms relies on their capacity to model complex and non-linear relationships within the data.</p><p>However, finding suitable structures for these models has been a challenge for researchers which are addressed leveraging. Transfer Learning techniques where instead of training a model from scratch, we reuse a pre-trained model and then fine-tune it for another related task which have resulted in a massive leap in state-of-the-art results for many of the NLP tasks through various approaches such as <a href="http://nlp.fast.ai/classification/2018/05/15/introducting-ulmfit.html">ULMFiT</a>, the <a href="https://blog.openai.com/language-unsupervised/">OpenAI Transformer,</a> <a href="https://allennlp.org/elmo">ELMo</a> and Google AI’s <a href="https://arxiv.org/pdf/1810.04805.pdf">BERT</a>.</p><p><a href="https://www.datadriveninvestor.com/2019/01/07/the-ultimate-learning-path-for-deep-learning-in-2019-more/">The Ultimate Learning Path for Deep Learning in 2019 &amp; more... | Data Driven Investor</a></p><h3>Topics you’ll find on this post include the following:</h3><ul><li>Deep Neural Networks</li><li>Recurrent Neural Networks (RNN)</li><li>Gated Recurrent Unit (GRU)</li><li>Long Short-Term Memory (LSTM)</li><li>Convolutional Neural Networks (CNN)</li><li>Hierarchical Attention Networks</li><li>Recurrent Convolutional Neural Networks (RCNN)</li><li>Random Multimodel Deep Learning (RMDL)</li><li>Hierarchical Deep Learning for Text (HDLTex)</li></ul><p><strong>Deep Neural Networks</strong></p><p>Deep Neural Networks architectures are designed to learn through multiple connections of layers where every single layer only receives a connection from previous and provides connections only to the next layer in the hidden part. The input layer is embedding vectors as shown in Figure below. The output layer neurons equal to the number of classes for multi-class classification and only one neuron for binary classification. Here, we have multi-class DNNs where the number of nodes in each layer as well as the number of layers are randomly assigned. The implementation of Deep Neural Network (DNN) is basically a discriminatively trained model that uses the standard back-propagation algorithm and sigmoid or ReLU as activation functions. The output layer for multi-class classification should use Softmax.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/677/0*XgRryZAtv6zqNwid.png" /></figure><p>import packages:</p><pre>from sklearn.datasets import fetch_20newsgroups<br>from keras.layers import  Dropout, Dense<br>from keras.models import Sequential<br>from sklearn.feature_extraction.text import TfidfVectorizer<br>import numpy as np<br>from sklearn import metrics</pre><p>convert text to TF-IDF:</p><pre>def TFIDF(X_train, X_test,MAX_NB_WORDS=75000):<br>    vectorizer_x = TfidfVectorizer(max_features=MAX_NB_WORDS)<br>    X_train = vectorizer_x.fit_transform(X_train).toarray()<br>    X_test = vectorizer_x.transform(X_test).toarray()<br>    print(&quot;tf-idf with&quot;,str(np.array(X_train).shape[1]),&quot;features&quot;)<br>    return (X_train,X_test)</pre><p>Build a DNN Model for Text:</p><pre>def Build_Model_DNN_Text(shape, nClasses, dropout=0.5):<br>    &quot;&quot;&quot;<br>    buildModel_DNN_Tex(shape, nClasses,dropout)<br>    Build Deep neural networks Model for text classification<br>    Shape is input feature space<br>    nClasses is number of classes<br>    &quot;&quot;&quot;<br>    model = Sequential()<br>    node = 512 # number of nodes<br>    nLayers = 4 # number of  hidden layer</pre><pre>    model.add(Dense(node,input_dim=shape,activation=&#39;relu&#39;))<br>    model.add(Dropout(dropout))<br>    for i in range(0,nLayers):<br>        model.add(Dense(node,input_dim=node,activation=&#39;relu&#39;))<br>        model.add(Dropout(dropout))<br>    model.add(Dense(nClasses, activation=&#39;softmax&#39;))</pre><pre>    model.compile(loss=&#39;sparse_categorical_crossentropy&#39;,<br>                  optimizer=&#39;adam&#39;,<br>                  metrics=[&#39;accuracy&#39;])</pre><pre>    return model</pre><p>Load text dataset (20newsgroups):</p><pre>newsgroups_train = fetch_20newsgroups(subset=&#39;train&#39;)<br>newsgroups_test = fetch_20newsgroups(subset=&#39;test&#39;)<br>X_train = newsgroups_train.data<br>X_test = newsgroups_test.data<br>y_train = newsgroups_train.target<br>y_test = newsgroups_test.target</pre><p>run DNN and see result:</p><pre>X_train_tfidf,X_test_tfidf = TFIDF(X_train,X_test)<br>model_DNN = Build_Model_DNN_Text(X_train_tfidf.shape[1], 20)<br>model_DNN.fit(X_train_tfidf, y_train,<br>                              validation_data=(X_test_tfidf, y_test),<br>                              epochs=10,<br>                              batch_size=128,<br>                              verbose=2)</pre><pre>predicted = model_DNN.predict(X_test_tfidf)</pre><pre>print(metrics.classification_report(y_test, predicted))</pre><p>Model summary:</p><pre>_________________________________________________________________<br>Layer (type)                 Output Shape              Param #<br>=================================================================<br>dense_1 (Dense)              (None, 512)               38400512<br>_________________________________________________________________<br>dropout_1 (Dropout)          (None, 512)               0<br>_________________________________________________________________<br>dense_2 (Dense)              (None, 512)               262656<br>_________________________________________________________________<br>dropout_2 (Dropout)          (None, 512)               0<br>_________________________________________________________________<br>dense_3 (Dense)              (None, 512)               262656<br>_________________________________________________________________<br>dropout_3 (Dropout)          (None, 512)               0<br>_________________________________________________________________<br>dense_4 (Dense)              (None, 512)               262656<br>_________________________________________________________________<br>dropout_4 (Dropout)          (None, 512)               0<br>_________________________________________________________________<br>dense_5 (Dense)              (None, 512)               262656<br>_________________________________________________________________<br>dropout_5 (Dropout)          (None, 512)               0<br>_________________________________________________________________<br>dense_6 (Dense)              (None, 20)                10260<br>=================================================================<br>Total params: 39,461,396<br>Trainable params: 39,461,396<br>Non-trainable params: 0<br>_________________________________________________________________</pre><p>Output:</p><pre>Train on 11314 samples, validate on 7532 samples<br>Epoch 1/10<br> - 16s - loss: 2.7553 - acc: 0.1090 - val_loss: 1.9330 - val_acc: 0.3184<br>Epoch 2/10<br> - 15s - loss: 1.5330 - acc: 0.4222 - val_loss: 1.1546 - val_acc: 0.6204<br>Epoch 3/10<br> - 15s - loss: 0.7438 - acc: 0.7257 - val_loss: 0.8405 - val_acc: 0.7499<br>Epoch 4/10<br> - 15s - loss: 0.2967 - acc: 0.9020 - val_loss: 0.9214 - val_acc: 0.7767<br>Epoch 5/10<br> - 15s - loss: 0.1557 - acc: 0.9543 - val_loss: 0.8965 - val_acc: 0.7917<br>Epoch 6/10<br> - 15s - loss: 0.1015 - acc: 0.9705 - val_loss: 0.9427 - val_acc: 0.7949<br>Epoch 7/10<br> - 15s - loss: 0.0595 - acc: 0.9835 - val_loss: 0.9893 - val_acc: 0.7995<br>Epoch 8/10<br> - 15s - loss: 0.0495 - acc: 0.9866 - val_loss: 0.9512 - val_acc: 0.8079<br>Epoch 9/10<br> - 15s - loss: 0.0437 - acc: 0.9867 - val_loss: 0.9690 - val_acc: 0.8117<br>Epoch 10/10<br> - 15s - loss: 0.0443 - acc: 0.9880 - val_loss: 1.0004 - val_acc: 0.8070<br></pre><pre>               precision    recall  f1-score   support</pre><pre>          0       0.76      0.78      0.77       319<br>          1       0.67      0.80      0.73       389<br>          2       0.82      0.63      0.71       394<br>          3       0.76      0.69      0.72       392<br>          4       0.65      0.86      0.74       385<br>          5       0.84      0.75      0.79       395<br>          6       0.82      0.87      0.84       390<br>          7       0.86      0.90      0.88       396<br>          8       0.95      0.91      0.93       398<br>          9       0.91      0.92      0.92       397<br>         10       0.98      0.92      0.95       399<br>         11       0.96      0.85      0.90       396<br>         12       0.71      0.69      0.70       393<br>         13       0.95      0.70      0.81       396<br>         14       0.86      0.91      0.88       394<br>         15       0.85      0.90      0.87       398<br>         16       0.79      0.84      0.81       364<br>         17       0.99      0.77      0.87       376<br>         18       0.58      0.75      0.65       310<br>         19       0.52      0.60      0.55       251</pre><pre>avg / total       0.82      0.81      0.81      7532</pre><p><strong>Recurrent Neural Networks (RNN)</strong></p><p>Recurrent Neural Networks (RNN) is another neural network architecture that is addressed by the researchers for text mining and classification. RNN assigns more weights to the previous data points of sequence. Therefore, this technique is a powerful method for text, string, and sequential data classification. In RNN, the neural net considers the information of previous nodes in a very sophisticated method which allows for better semantic analysis of the structures in the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/0*zIfk6jFWnTvBI-d8.png" /></figure><p><strong>Gated Recurrent Unit (GRU)</strong></p><p>Gated Recurrent Unit (GRU) is a gating mechanism for RNN which was introduced by <a href="https://arxiv.org/abs/1412.3555">J. Chung et al.</a> and <a href="https://arxiv.org/abs/1406.1078">K.Cho et al.</a>. GRU is a simplified variant of the LSTM architecture, but there are differences as follows: GRU contains two gates and does not possess any internal memory (as shown in Figure; and finally, a second non-linearity is not applied (tanh in Figure).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/1*JNPTzWqduMiuiiekKUOZ8g.png" /></figure><p><strong>Long Short-Term Memory (LSTM)</strong></p><p>Long Short-Term Memory~(LSTM) was introduced by <a href="https://www.mitpressjournals.org/doi/abs/10.1162/neco.1997.9.8.1735">S. Hochreiter and J. Schmidhuber</a> and developed by many research scientists.</p><p>To deal with these problems Long Short-Term Memory (LSTM) is a special type of RNN that preserves long term dependency in a more effective way compared to the basic RNNs. This is particularly useful to overcome vanishing gradient problem as LSTM uses multiple gates to carefully regulate the amount of information that will be allowed into each node state. The figure shows the basic cell of a LSTM model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/598/1*Mt0LHT1uI6Gl6vGsE0A0TA.png" /></figure><p>import packages:</p><pre>from keras.layers import Dropout, Dense, GRU, Embedding<br>from keras.models import Sequential<br>from sklearn.feature_extraction.text import TfidfVectorizer<br>import numpy as np<br>from sklearn import metrics<br>from keras.preprocessing.text import Tokenizer<br>from keras.preprocessing.sequence import pad_sequences<br>from sklearn.datasets import fetch_20newsgroups</pre><p>convert text to word embedding (Using GloVe):</p><pre>def loadData_Tokenizer(X_train, X_test,MAX_NB_WORDS=75000,MAX_SEQUENCE_LENGTH=500):<br>    np.random.seed(7)<br>    text = np.concatenate((X_train, X_test), axis=0)<br>    text = np.array(text)<br>    tokenizer = Tokenizer(num_words=MAX_NB_WORDS)<br>    tokenizer.fit_on_texts(text)<br>    sequences = tokenizer.texts_to_sequences(text)<br>    word_index = tokenizer.word_index<br>    text = pad_sequences(sequences, maxlen=MAX_SEQUENCE_LENGTH)<br>    print(&#39;Found %s unique tokens.&#39; % len(word_index))<br>    indices = np.arange(text.shape[0])<br>    # np.random.shuffle(indices)<br>    text = text[indices]<br>    print(text.shape)<br>    X_train = text[0:len(X_train), ]<br>    X_test = text[len(X_train):, ]<br>    embeddings_index = {}<br>    f = open(&quot;C:\\Users\\swayam\\Documents\\Glove\\glove.6B.50d.txt&quot;, encoding=&quot;utf8&quot;)<br>    for line in f:</pre><pre>        values = line.split()<br>        word = values[0]<br>        try:<br>            coefs = np.asarray(values[1:], dtype=&#39;float32&#39;)<br>        except:<br>            pass<br>        embeddings_index[word] = coefs<br>    f.close()<br>    print(&#39;Total %s word vectors.&#39; % len(embeddings_index))<br>    return (X_train, X_test, word_index,embeddings_index)</pre><p>Build a RNN Model for Text:</p><pre>def Build_Model_RNN_Text(word_index, embeddings_index, nclasses,  MAX_SEQUENCE_LENGTH=500, EMBEDDING_DIM=50, dropout=0.5):<br>    &quot;&quot;&quot;<br>    def buildModel_RNN(word_index, embeddings_index, nclasses,  MAX_SEQUENCE_LENGTH=500, EMBEDDING_DIM=50, dropout=0.5):<br>    word_index in word index ,<br>    embeddings_index is embeddings index, look at data_helper.py<br>    nClasses is number of classes,<br>    MAX_SEQUENCE_LENGTH is maximum lenght of text sequences<br>    &quot;&quot;&quot;</pre><pre>    model = Sequential()<br>    hidden_layer = 3<br>    gru_node = 32</pre><pre>    embedding_matrix = np.random.random((len(word_index) + 1, EMBEDDING_DIM))<br>    for word, i in word_index.items():<br>        embedding_vector = embeddings_index.get(word)<br>        if embedding_vector is not None:<br>            # words not found in embedding index will be all-zeros.<br>            if len(embedding_matrix[i]) != len(embedding_vector):<br>                print(&quot;could not broadcast input array from shape&quot;, str(len(embedding_matrix[i])),<br>                      &quot;into shape&quot;, str(len(embedding_vector)), &quot; Please make sure your&quot;<br>                                                                &quot; EMBEDDING_DIM is equal to embedding_vector file ,GloVe,&quot;)<br>                exit(1)<br>            embedding_matrix[i] = embedding_vector<br>    model.add(Embedding(len(word_index) + 1,<br>                                EMBEDDING_DIM,<br>                                weights=[embedding_matrix],<br>                                input_length=MAX_SEQUENCE_LENGTH,<br>                                trainable=True))<br></pre><pre>    print(gru_node)<br>    for i in range(0,hidden_layer):<br>        model.add(GRU(gru_node,return_sequences=True, recurrent_dropout=0.2))<br>        model.add(Dropout(dropout))<br>    model.add(GRU(gru_node, recurrent_dropout=0.2))<br>    model.add(Dropout(dropout))<br>    model.add(Dense(256, activation=&#39;relu&#39;))<br>    model.add(Dense(nclasses, activation=&#39;softmax&#39;))<br></pre><pre>    model.compile(loss=&#39;sparse_categorical_crossentropy&#39;,<br>                      optimizer=&#39;adam&#39;,<br>                      metrics=[&#39;accuracy&#39;])<br>    return model</pre><p>run RNN and see result:</p><pre>newsgroups_train = fetch_20newsgroups(subset=&#39;train&#39;)<br>newsgroups_test = fetch_20newsgroups(subset=&#39;test&#39;)<br>X_train = newsgroups_train.data<br>X_test = newsgroups_test.data<br>y_train = newsgroups_train.target<br>y_test = newsgroups_test.target</pre><pre>X_train_Glove,X_test_Glove, word_index,embeddings_index = loadData_Tokenizer(X_train,X_test)<br></pre><pre>model_RNN = Build_Model_RNN_Text(word_index,embeddings_index, 20)</pre><pre>model_RNN.fit(X_train_Glove, y_train,<br>                              validation_data=(X_test_Glove, y_test),<br>                              epochs=10,<br>                              batch_size=128,<br>                              verbose=2)</pre><pre>predicted = Build_Model_RNN_Text.predict_classes(X_test_Glove)</pre><pre>print(metrics.classification_report(y_test, predicted))</pre><p>Model summary:</p><pre>_________________________________________________________________<br>Layer (type)                 Output Shape              Param #<br>=================================================================<br>embedding_1 (Embedding)      (None, 500, 50)           8960500<br>_________________________________________________________________<br>gru_1 (GRU)                  (None, 500, 256)          235776<br>_________________________________________________________________<br>dropout_1 (Dropout)          (None, 500, 256)          0<br>_________________________________________________________________<br>gru_2 (GRU)                  (None, 500, 256)          393984<br>_________________________________________________________________<br>dropout_2 (Dropout)          (None, 500, 256)          0<br>_________________________________________________________________<br>gru_3 (GRU)                  (None, 500, 256)          393984<br>_________________________________________________________________<br>dropout_3 (Dropout)          (None, 500, 256)          0<br>_________________________________________________________________<br>gru_4 (GRU)                  (None, 256)               393984<br>_________________________________________________________________<br>dense_1 (Dense)              (None, 20)                5140<br>=================================================================<br>Total params: 10,383,368<br>Trainable params: 10,383,368<br>Non-trainable params: 0<br>_________________________________________________________________</pre><p>Output:</p><pre>Train on 11314 samples, validate on 7532 samples<br>Epoch 1/20<br> - 268s - loss: 2.5347 - acc: 0.1792 - val_loss: 2.2857 - val_acc: 0.2460<br>Epoch 2/20<br> - 271s - loss: 1.6751 - acc: 0.3999 - val_loss: 1.4972 - val_acc: 0.4660<br>Epoch 3/20<br> - 270s - loss: 1.0945 - acc: 0.6072 - val_loss: 1.3232 - val_acc: 0.5483<br>Epoch 4/20<br> - 269s - loss: 0.7761 - acc: 0.7312 - val_loss: 1.1009 - val_acc: 0.6452<br>Epoch 5/20<br> - 269s - loss: 0.5513 - acc: 0.8112 - val_loss: 1.0395 - val_acc: 0.6832<br>Epoch 6/20<br> - 269s - loss: 0.3765 - acc: 0.8754 - val_loss: 0.9977 - val_acc: 0.7086<br>Epoch 7/20<br> - 270s - loss: 0.2481 - acc: 0.9202 - val_loss: 1.0485 - val_acc: 0.7270<br>Epoch 8/20<br> - 269s - loss: 0.1717 - acc: 0.9463 - val_loss: 1.0269 - val_acc: 0.7394<br>Epoch 9/20<br> - 269s - loss: 0.1130 - acc: 0.9644 - val_loss: 1.1498 - val_acc: 0.7369<br>Epoch 10/20<br> - 269s - loss: 0.0640 - acc: 0.9808 - val_loss: 1.1442 - val_acc: 0.7508<br>Epoch 11/20<br> - 269s - loss: 0.0567 - acc: 0.9828 - val_loss: 1.2318 - val_acc: 0.7414<br>Epoch 12/20<br> - 268s - loss: 0.0472 - acc: 0.9858 - val_loss: 1.2204 - val_acc: 0.7496<br>Epoch 13/20<br> - 269s - loss: 0.0319 - acc: 0.9910 - val_loss: 1.1895 - val_acc: 0.7657<br>Epoch 14/20<br> - 268s - loss: 0.0466 - acc: 0.9853 - val_loss: 1.2821 - val_acc: 0.7517<br>Epoch 15/20<br> - 271s - loss: 0.0269 - acc: 0.9917 - val_loss: 1.2869 - val_acc: 0.7557<br>Epoch 16/20<br> - 271s - loss: 0.0187 - acc: 0.9950 - val_loss: 1.3037 - val_acc: 0.7598<br>Epoch 17/20<br> - 268s - loss: 0.0157 - acc: 0.9959 - val_loss: 1.2974 - val_acc: 0.7638<br>Epoch 18/20<br> - 270s - loss: 0.0121 - acc: 0.9966 - val_loss: 1.3526 - val_acc: 0.7602<br>Epoch 19/20<br> - 269s - loss: 0.0262 - acc: 0.9926 - val_loss: 1.4182 - val_acc: 0.7517<br>Epoch 20/20<br> - 269s - loss: 0.0249 - acc: 0.9918 - val_loss: 1.3453 - val_acc: 0.7638<br></pre><pre>               precision    recall  f1-score   support</pre><pre>          0       0.71      0.71      0.71       319<br>          1       0.72      0.68      0.70       389<br>          2       0.76      0.62      0.69       394<br>          3       0.67      0.58      0.62       392<br>          4       0.68      0.67      0.68       385<br>          5       0.75      0.73      0.74       395<br>          6       0.82      0.74      0.78       390<br>          7       0.83      0.83      0.83       396<br>          8       0.81      0.90      0.86       398<br>          9       0.92      0.90      0.91       397<br>         10       0.91      0.94      0.93       399<br>         11       0.87      0.76      0.81       396<br>         12       0.57      0.70      0.63       393<br>         13       0.81      0.85      0.83       396<br>         14       0.74      0.93      0.82       394<br>         15       0.82      0.83      0.83       398<br>         16       0.74      0.78      0.76       364<br>         17       0.96      0.83      0.89       376<br>         18       0.64      0.60      0.62       310<br>         19       0.48      0.56      0.52       251</pre><pre>avg / total       0.77      0.76      0.76      7532</pre><p><strong>Convolutional Neural Networks (CNN)</strong></p><p>Convolutional Neural Networks (CNN) is Another deep learning architecture that is employed for hierarchical document classification. Although originally built for image processing with architecture similar to the visual cortex, CNNs have also been effectively used for text classification. In a basic CNN for image processing, an image tensor is convolved with a set of kernels of size <em>d by d</em>. These convolution layers are called feature maps and can be stacked to provide multiple filters on the input. To reduce the computational complexity, CNNs use pooling which reduces the size of the output from one layer to the next in the network. Different pooling techniques are used to reduce outputs while preserving important features.</p><p>The most common pooling method is max pooling where the maximum element is selected from the pooling window. In order to feed the pooled output from stacked featured maps to the next layer, the maps are flattened into one column. The final layers in a CNN are typically fully connected dense layers. In general, during the back-propagation step of a convolutional neural network not only the weights are adjusted but also the feature detector filters. A potential problem of CNN used for text is the number of ‘channels’, <em>Sigma</em> (size of the feature space). This might be very large (e.g. 50K), for text but for images this is less of a problem (e.g. only 3 channels of RGB). This means the dimensionality of the CNN for text is very high.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mX5VQEqPtAhevsy0.png" /></figure><p>import packages:</p><pre>from keras.layers import Dropout, Dense,Input,Embedding,Flatten, MaxPooling1D, Conv1D<br>from keras.models import Sequential,Model<br>from sklearn.feature_extraction.text import TfidfVectorizer<br>import numpy as np<br>from sklearn import metrics<br>from keras.preprocessing.text import Tokenizer<br>from keras.preprocessing.sequence import pad_sequences<br>from sklearn.datasets import fetch_20newsgroups<br>from keras.layers.merge import Concatenate</pre><p>convert text to word embedding (Using GloVe):</p><pre>def loadData_Tokenizer(X_train, X_test,MAX_NB_WORDS=75000,MAX_SEQUENCE_LENGTH=500):<br>    np.random.seed(7)<br>    text = np.concatenate((X_train, X_test), axis=0)<br>    text = np.array(text)<br>    tokenizer = Tokenizer(num_words=MAX_NB_WORDS)<br>    tokenizer.fit_on_texts(text)<br>    sequences = tokenizer.texts_to_sequences(text)<br>    word_index = tokenizer.word_index<br>    text = pad_sequences(sequences, maxlen=MAX_SEQUENCE_LENGTH)<br>    print(&#39;Found %s unique tokens.&#39; % len(word_index))<br>    indices = np.arange(text.shape[0])<br>    # np.random.shuffle(indices)<br>    text = text[indices]<br>    print(text.shape)<br>    X_train = text[0:len(X_train), ]<br>    X_test = text[len(X_train):, ]<br>    embeddings_index = {}<br>    f = open(&quot;C:\\Users\\swayam\\Documents\\Glove\\glove.6B.50d.txt&quot;, encoding=&quot;utf8&quot;)<br>    for line in f:<br>        values = line.split()<br>        word = values[0]<br>        try:<br>            coefs = np.asarray(values[1:], dtype=&#39;float32&#39;)<br>        except:<br>            pass<br>        embeddings_index[word] = coefs<br>    f.close()<br>    print(&#39;Total %s word vectors.&#39; % len(embeddings_index))<br>    return (X_train, X_test, word_index,embeddings_index)</pre><p>Build a CNN Model for Text:</p><pre>def Build_Model_CNN_Text(word_index, embeddings_index, nclasses, MAX_SEQUENCE_LENGTH=500, EMBEDDING_DIM=50, dropout=0.5):</pre><pre>    &quot;&quot;&quot;<br>        def buildModel_CNN(word_index, embeddings_index, nclasses, MAX_SEQUENCE_LENGTH=500, EMBEDDING_DIM=50, dropout=0.5):<br>        word_index in word index ,<br>        embeddings_index is embeddings index, look at data_helper.py<br>        nClasses is number of classes,<br>        MAX_SEQUENCE_LENGTH is maximum lenght of text sequences,<br>        EMBEDDING_DIM is an int value for dimention of word embedding look at data_helper.py<br>    &quot;&quot;&quot;</pre><pre>    model = Sequential()<br>    embedding_matrix = np.random.random((len(word_index) + 1, EMBEDDING_DIM))<br>    for word, i in word_index.items():<br>        embedding_vector = embeddings_index.get(word)<br>        if embedding_vector is not None:<br>            # words not found in embedding index will be all-zeros.<br>            if len(embedding_matrix[i]) !=len(embedding_vector):<br>                print(&quot;could not broadcast input array from shape&quot;,str(len(embedding_matrix[i])),<br>                                 &quot;into shape&quot;,str(len(embedding_vector)),&quot; Please make sure your&quot;<br>                                 &quot; EMBEDDING_DIM is equal to embedding_vector file ,GloVe,&quot;)<br>                exit(1)</pre><pre>            embedding_matrix[i] = embedding_vector</pre><pre>    embedding_layer = Embedding(len(word_index) + 1,<br>                                EMBEDDING_DIM,<br>                                weights=[embedding_matrix],<br>                                input_length=MAX_SEQUENCE_LENGTH,<br>                                trainable=True)</pre><pre>    # applying a more complex convolutional approach<br>    convs = []<br>    filter_sizes = []<br>    layer = 5<br>    print(&quot;Filter  &quot;,layer)<br>    for fl in range(0,layer):<br>        filter_sizes.append((fl+2))</pre><pre>    node = 128<br>    sequence_input = Input(shape=(MAX_SEQUENCE_LENGTH,), dtype=&#39;int32&#39;)<br>    embedded_sequences = embedding_layer(sequence_input)</pre><pre>    for fsz in filter_sizes:<br>        l_conv = Conv1D(node, kernel_size=fsz, activation=&#39;relu&#39;)(embedded_sequences)<br>        l_pool = MaxPooling1D(5)(l_conv)<br>        #l_pool = Dropout(0.25)(l_pool)<br>        convs.append(l_pool)</pre><pre>    l_merge = Concatenate(axis=1)(convs)<br>    l_cov1 = Conv1D(node, 5, activation=&#39;relu&#39;)(l_merge)<br>    l_cov1 = Dropout(dropout)(l_cov1)<br>    l_pool1 = MaxPooling1D(5)(l_cov1)<br>    l_cov2 = Conv1D(node, 5, activation=&#39;relu&#39;)(l_pool1)<br>    l_cov2 = Dropout(dropout)(l_cov2)<br>    l_pool2 = MaxPooling1D(30)(l_cov2)<br>    l_flat = Flatten()(l_pool2)<br>    l_dense = Dense(1024, activation=&#39;relu&#39;)(l_flat)<br>    l_dense = Dropout(dropout)(l_dense)<br>    l_dense = Dense(512, activation=&#39;relu&#39;)(l_dense)<br>    l_dense = Dropout(dropout)(l_dense)<br>    preds = Dense(nclasses, activation=&#39;softmax&#39;)(l_dense)<br>    model = Model(sequence_input, preds)</pre><pre>    model.compile(loss=&#39;sparse_categorical_crossentropy&#39;,<br>                  optimizer=&#39;adam&#39;,<br>                  metrics=[&#39;accuracy&#39;])</pre><pre>    return model</pre><p>run CNN and see result:</p><pre>newsgroups_train = fetch_20newsgroups(subset=&#39;train&#39;)<br>newsgroups_test = fetch_20newsgroups(subset=&#39;test&#39;)<br>X_train = newsgroups_train.data<br>X_test = newsgroups_test.data<br>y_train = newsgroups_train.target<br>y_test = newsgroups_test.target</pre><pre>X_train_Glove,X_test_Glove, word_index,embeddings_index = loadData_Tokenizer(X_train,X_test)<br></pre><pre>model_CNN = Build_Model_CNN_Text(word_index,embeddings_index, 20)<br></pre><pre>model_CNN.summary()</pre><pre>model_CNN.fit(X_train_Glove, y_train,<br>                              validation_data=(X_test_Glove, y_test),<br>                              epochs=15,<br>                              batch_size=128,<br>                              verbose=2)</pre><pre>predicted = model_CNN.predict(X_test_Glove)</pre><pre>predicted = np.argmax(predicted, axis=1)<br></pre><pre>print(metrics.classification_report(y_test, predicted))</pre><p>Model:</p><pre>__________________________________________________________________________________________________<br>Layer (type)                    Output Shape         Param #     Connected to<br>==================================================================================================<br>input_1 (InputLayer)            (None, 500)          0<br>__________________________________________________________________________________________________<br>embedding_1 (Embedding)         (None, 500, 50)      8960500     input_1[0][0]<br>__________________________________________________________________________________________________<br>conv1d_1 (Conv1D)               (None, 499, 128)     12928       embedding_1[0][0]<br>__________________________________________________________________________________________________<br>conv1d_2 (Conv1D)               (None, 498, 128)     19328       embedding_1[0][0]<br>__________________________________________________________________________________________________<br>conv1d_3 (Conv1D)               (None, 497, 128)     25728       embedding_1[0][0]<br>__________________________________________________________________________________________________<br>conv1d_4 (Conv1D)               (None, 496, 128)     32128       embedding_1[0][0]<br>__________________________________________________________________________________________________<br>conv1d_5 (Conv1D)               (None, 495, 128)     38528       embedding_1[0][0]<br>__________________________________________________________________________________________________<br>max_pooling1d_1 (MaxPooling1D)  (None, 99, 128)      0           conv1d_1[0][0]<br>__________________________________________________________________________________________________<br>max_pooling1d_2 (MaxPooling1D)  (None, 99, 128)      0           conv1d_2[0][0]<br>__________________________________________________________________________________________________<br>max_pooling1d_3 (MaxPooling1D)  (None, 99, 128)      0           conv1d_3[0][0]<br>__________________________________________________________________________________________________<br>max_pooling1d_4 (MaxPooling1D)  (None, 99, 128)      0           conv1d_4[0][0]<br>__________________________________________________________________________________________________<br>max_pooling1d_5 (MaxPooling1D)  (None, 99, 128)      0           conv1d_5[0][0]<br>__________________________________________________________________________________________________<br>concatenate_1 (Concatenate)     (None, 495, 128)     0           max_pooling1d_1[0][0]<br>                                                                 max_pooling1d_2[0][0]<br>                                                                 max_pooling1d_3[0][0]<br>                                                                 max_pooling1d_4[0][0]<br>                                                                 max_pooling1d_5[0][0]<br>__________________________________________________________________________________________________<br>conv1d_6 (Conv1D)               (None, 491, 128)     82048       concatenate_1[0][0]<br>__________________________________________________________________________________________________<br>dropout_1 (Dropout)             (None, 491, 128)     0           conv1d_6[0][0]<br>__________________________________________________________________________________________________<br>max_pooling1d_6 (MaxPooling1D)  (None, 98, 128)      0           dropout_1[0][0]<br>__________________________________________________________________________________________________<br>conv1d_7 (Conv1D)               (None, 94, 128)      82048       max_pooling1d_6[0][0]<br>__________________________________________________________________________________________________<br>dropout_2 (Dropout)             (None, 94, 128)      0           conv1d_7[0][0]<br>__________________________________________________________________________________________________<br>max_pooling1d_7 (MaxPooling1D)  (None, 3, 128)       0           dropout_2[0][0]<br>__________________________________________________________________________________________________<br>flatten_1 (Flatten)             (None, 384)          0           max_pooling1d_7[0][0]<br>__________________________________________________________________________________________________<br>dense_1 (Dense)                 (None, 1024)         394240      flatten_1[0][0]<br>__________________________________________________________________________________________________<br>dropout_3 (Dropout)             (None, 1024)         0           dense_1[0][0]<br>__________________________________________________________________________________________________<br>dense_2 (Dense)                 (None, 512)          524800      dropout_3[0][0]<br>__________________________________________________________________________________________________<br>dropout_4 (Dropout)             (None, 512)          0           dense_2[0][0]<br>__________________________________________________________________________________________________<br>dense_3 (Dense)                 (None, 20)           10260       dropout_4[0][0]<br>==================================================================================================<br>Total params: 10,182,536<br>Trainable params: 10,182,536<br>Non-trainable params: 0<br>__________________________________________________________________________________________________</pre><p>Output:</p><pre>Train on 11314 samples, validate on 7532 samples<br>Epoch 1/15<br> - 6s - loss: 2.9329 - acc: 0.0783 - val_loss: 2.7628 - val_acc: 0.1403<br>Epoch 2/15<br> - 4s - loss: 2.2534 - acc: 0.2249 - val_loss: 2.1715 - val_acc: 0.4007<br>Epoch 3/15<br> - 4s - loss: 1.5643 - acc: 0.4326 - val_loss: 1.7846 - val_acc: 0.5052<br>Epoch 4/15<br> - 4s - loss: 1.1771 - acc: 0.5662 - val_loss: 1.4949 - val_acc: 0.6131<br>Epoch 5/15<br> - 4s - loss: 0.8880 - acc: 0.6797 - val_loss: 1.3629 - val_acc: 0.6256<br>Epoch 6/15<br> - 4s - loss: 0.6990 - acc: 0.7569 - val_loss: 1.2013 - val_acc: 0.6624<br>Epoch 7/15<br> - 4s - loss: 0.5037 - acc: 0.8200 - val_loss: 1.0674 - val_acc: 0.6807<br>Epoch 8/15<br> - 4s - loss: 0.4050 - acc: 0.8626 - val_loss: 1.0223 - val_acc: 0.6863<br>Epoch 9/15<br> - 4s - loss: 0.2952 - acc: 0.8968 - val_loss: 0.9045 - val_acc: 0.7120<br>Epoch 10/15<br> - 4s - loss: 0.2314 - acc: 0.9217 - val_loss: 0.8574 - val_acc: 0.7326<br>Epoch 11/15<br> - 4s - loss: 0.1778 - acc: 0.9436 - val_loss: 0.8752 - val_acc: 0.7270<br>Epoch 12/15<br> - 4s - loss: 0.1475 - acc: 0.9524 - val_loss: 0.8299 - val_acc: 0.7355<br>Epoch 13/15<br> - 4s - loss: 0.1089 - acc: 0.9657 - val_loss: 0.8034 - val_acc: 0.7491<br>Epoch 14/15<br> - 4s - loss: 0.1047 - acc: 0.9666 - val_loss: 0.8172 - val_acc: 0.7463<br>Epoch 15/15<br> - 4s - loss: 0.0749 - acc: 0.9774 - val_loss: 0.8511 - val_acc: 0.7313<br></pre><pre>               precision    recall  f1-score   support</pre><pre>          0       0.75      0.61      0.67       319<br>          1       0.63      0.74      0.68       389<br>          2       0.74      0.54      0.62       394<br>          3       0.49      0.76      0.60       392<br>          4       0.60      0.70      0.64       385<br>          5       0.79      0.57      0.66       395<br>          6       0.73      0.76      0.74       390<br>          7       0.83      0.74      0.78       396<br>          8       0.86      0.88      0.87       398<br>          9       0.95      0.78      0.86       397<br>         10       0.93      0.93      0.93       399<br>         11       0.92      0.77      0.84       396<br>         12       0.55      0.72      0.62       393<br>         13       0.76      0.85      0.80       396<br>         14       0.86      0.83      0.84       394<br>         15       0.91      0.73      0.81       398<br>         16       0.75      0.65      0.70       364<br>         17       0.95      0.86      0.90       376<br>         18       0.60      0.49      0.54       310<br>         19       0.37      0.60      0.46       251</pre><pre>avg / total       0.76      0.73      0.74      7532</pre><p><strong>Hierarchical Attention Networks</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/515/0*LbxRBk1DJi_6GwYf.png" /></figure><p><strong>Recurrent Convolutional Neural Networks (RCNN)</strong></p><p>Recurrent Convolutional Neural Networks (RCNN) is also used for text classification. The main idea of this technique is capturing contextual information with the recurrent structure and constructing the representation of text using a convolutional neural network. This architecture is a combination of RNN and CNN to use the advantages of both technique in a model.</p><p>import packages:</p><pre>from keras.preprocessing import sequence<br>from keras.models import Sequential<br>from keras.layers import Dense, Dropout, Activation<br>from keras.layers import Embedding<br>from keras.layers import GRU<br>from keras.layers import Conv1D, MaxPooling1D<br>from keras.datasets import imdb<br>from sklearn.datasets import fetch_20newsgroups<br>import numpy as np<br>from sklearn import metrics<br>from keras.preprocessing.text import Tokenizer<br>from keras.preprocessing.sequence import pad_sequences</pre><p>Convert text to word embedding (Using GloVe):</p><pre>def loadData_Tokenizer(X_train, X_test,MAX_NB_WORDS=75000,MAX_SEQUENCE_LENGTH=500):<br>    np.random.seed(7)<br>    text = np.concatenate((X_train, X_test), axis=0)<br>    text = np.array(text)<br>    tokenizer = Tokenizer(num_words=MAX_NB_WORDS)<br>    tokenizer.fit_on_texts(text)<br>    sequences = tokenizer.texts_to_sequences(text)<br>    word_index = tokenizer.word_index<br>    text = pad_sequences(sequences, maxlen=MAX_SEQUENCE_LENGTH)<br>    print(&#39;Found %s unique tokens.&#39; % len(word_index))<br>    indices = np.arange(text.shape[0])<br>    # np.random.shuffle(indices)<br>    text = text[indices]<br>    print(text.shape)<br>    X_train = text[0:len(X_train), ]<br>    X_test = text[len(X_train):, ]<br>    embeddings_index = {}<br>    f = open(&quot;C:\\Users\\swayam\\Documents\\Glove\\glove.6B.50d.txt&quot;, encoding=&quot;utf8&quot;)<br>    for line in f:<br>        values = line.split()<br>        word = values[0]<br>        try:<br>            coefs = np.asarray(values[1:], dtype=&#39;float32&#39;)<br>        except:<br>            pass<br>        embeddings_index[word] = coefs<br>    f.close()<br>    print(&#39;Total %s word vectors.&#39; % len(embeddings_index))<br>    return (X_train, X_test, word_index,embeddings_index)</pre><pre>def Build_Model_RCNN_Text(word_index, embeddings_index, nclasses, MAX_SEQUENCE_LENGTH=500, EMBEDDING_DIM=50):</pre><pre>    kernel_size = 2<br>    filters = 256<br>    pool_size = 2<br>    gru_node = 256</pre><pre>    embedding_matrix = np.random.random((len(word_index) + 1, EMBEDDING_DIM))<br>    for word, i in word_index.items():<br>        embedding_vector = embeddings_index.get(word)<br>        if embedding_vector is not None:<br>            # words not found in embedding index will be all-zeros.<br>            if len(embedding_matrix[i]) !=len(embedding_vector):<br>                print(&quot;could not broadcast input array from shape&quot;,str(len(embedding_matrix[i])),<br>                                 &quot;into shape&quot;,str(len(embedding_vector)),&quot; Please make sure your&quot;<br>                                 &quot; EMBEDDING_DIM is equal to embedding_vector file ,GloVe,&quot;)<br>                exit(1)</pre><pre>            embedding_matrix[i] = embedding_vector</pre><pre>    model = Sequential()<br>    model.add(Embedding(len(word_index) + 1,<br>                                EMBEDDING_DIM,<br>                                weights=[embedding_matrix],<br>                                input_length=MAX_SEQUENCE_LENGTH,<br>                                trainable=True))<br>    model.add(Dropout(0.25))<br>    model.add(Conv1D(filters, kernel_size, activation=&#39;relu&#39;))<br>    model.add(MaxPooling1D(pool_size=pool_size))<br>    model.add(Conv1D(filters, kernel_size, activation=&#39;relu&#39;))<br>    model.add(MaxPooling1D(pool_size=pool_size))<br>    model.add(Conv1D(filters, kernel_size, activation=&#39;relu&#39;))<br>    model.add(MaxPooling1D(pool_size=pool_size))<br>    model.add(Conv1D(filters, kernel_size, activation=&#39;relu&#39;))<br>    model.add(MaxPooling1D(pool_size=pool_size))<br>    model.add(LSTM(gru_node, return_sequences=True, recurrent_dropout=0.2))<br>    model.add(LSTM(gru_node, return_sequences=True, recurrent_dropout=0.2))<br>    model.add(LSTM(gru_node, return_sequences=True, recurrent_dropout=0.2))<br>    model.add(LSTM(gru_node, recurrent_dropout=0.2))<br>    model.add(Dense(1024,activation=&#39;relu&#39;))<br>    model.add(Dense(nclasses))<br>    model.add(Activation(&#39;softmax&#39;))</pre><pre>    model.compile(loss=&#39;sparse_categorical_crossentropy&#39;,<br>                  optimizer=&#39;adam&#39;,<br>                  metrics=[&#39;accuracy&#39;])</pre><pre>    return model</pre><pre>newsgroups_train = fetch_20newsgroups(subset=&#39;train&#39;)<br>newsgroups_test = fetch_20newsgroups(subset=&#39;test&#39;)<br>X_train = newsgroups_train.data<br>X_test = newsgroups_test.data<br>y_train = newsgroups_train.target<br>y_test = newsgroups_test.target</pre><pre>X_train_Glove,X_test_Glove, word_index,embeddings_index = loadData_Tokenizer(X_train,X_test)</pre><p>Run RCNN :</p><pre>model_RCNN = Build_Model_CNN_Text(word_index,embeddings_index, 20)<br></pre><pre>model_RCNN.summary()</pre><pre>model_RCNN.fit(X_train_Glove, y_train,<br>                              validation_data=(X_test_Glove, y_test),<br>                              epochs=15,<br>                              batch_size=128,<br>                              verbose=2)</pre><pre>predicted = model_RCNN.predict(X_test_Glove)</pre><pre>predicted = np.argmax(predicted, axis=1)<br>print(metrics.classification_report(y_test, predicted))</pre><p>summary of the model:</p><pre>_________________________________________________________________<br>Layer (type)                 Output Shape              Param #<br>=================================================================<br>embedding_1 (Embedding)      (None, 500, 50)           8960500<br>_________________________________________________________________<br>dropout_1 (Dropout)          (None, 500, 50)           0<br>_________________________________________________________________<br>conv1d_1 (Conv1D)            (None, 499, 256)          25856<br>_________________________________________________________________<br>max_pooling1d_1 (MaxPooling1 (None, 249, 256)          0<br>_________________________________________________________________<br>conv1d_2 (Conv1D)            (None, 248, 256)          131328<br>_________________________________________________________________<br>max_pooling1d_2 (MaxPooling1 (None, 124, 256)          0<br>_________________________________________________________________<br>conv1d_3 (Conv1D)            (None, 123, 256)          131328<br>_________________________________________________________________<br>max_pooling1d_3 (MaxPooling1 (None, 61, 256)           0<br>_________________________________________________________________<br>conv1d_4 (Conv1D)            (None, 60, 256)           131328<br>_________________________________________________________________<br>max_pooling1d_4 (MaxPooling1 (None, 30, 256)           0<br>_________________________________________________________________<br>lstm_1 (LSTM)                (None, 30, 256)           525312<br>_________________________________________________________________<br>lstm_2 (LSTM)                (None, 30, 256)           525312<br>_________________________________________________________________<br>lstm_3 (LSTM)                (None, 30, 256)           525312<br>_________________________________________________________________<br>lstm_4 (LSTM)                (None, 256)               525312<br>_________________________________________________________________<br>dense_1 (Dense)              (None, 1024)              263168<br>_________________________________________________________________<br>dense_2 (Dense)              (None, 20)                20500<br>_________________________________________________________________<br>activation_1 (Activation)    (None, 20)                0<br>=================================================================<br>Total params: 11,765,256<br>Trainable params: 11,765,256<br>Non-trainable params: 0<br>_________________________________________________________________</pre><p>Output:</p><pre>Train on 11314 samples, validate on 7532 samples<br>Epoch 1/15<br> - 28s - loss: 2.6624 - acc: 0.1081 - val_loss: 2.3012 - val_acc: 0.1753<br>Epoch 2/15<br> - 22s - loss: 2.1142 - acc: 0.2224 - val_loss: 1.9168 - val_acc: 0.2669<br>Epoch 3/15<br> - 22s - loss: 1.7465 - acc: 0.3290 - val_loss: 1.8257 - val_acc: 0.3412<br>Epoch 4/15<br> - 22s - loss: 1.4730 - acc: 0.4356 - val_loss: 1.5433 - val_acc: 0.4436<br>Epoch 5/15<br> - 22s - loss: 1.1800 - acc: 0.5556 - val_loss: 1.2973 - val_acc: 0.5467<br>Epoch 6/15<br> - 22s - loss: 0.9910 - acc: 0.6281 - val_loss: 1.2530 - val_acc: 0.5797<br>Epoch 7/15<br> - 22s - loss: 0.8581 - acc: 0.6854 - val_loss: 1.1522 - val_acc: 0.6281<br>Epoch 8/15<br> - 22s - loss: 0.7058 - acc: 0.7428 - val_loss: 1.2385 - val_acc: 0.6033<br>Epoch 9/15<br> - 22s - loss: 0.6792 - acc: 0.7515 - val_loss: 1.0200 - val_acc: 0.6775<br>Epoch 10/15<br> - 22s - loss: 0.5782 - acc: 0.7948 - val_loss: 1.0961 - val_acc: 0.6577<br>Epoch 11/15<br> - 23s - loss: 0.4674 - acc: 0.8341 - val_loss: 1.0866 - val_acc: 0.6924<br>Epoch 12/15<br> - 23s - loss: 0.4284 - acc: 0.8512 - val_loss: 0.9880 - val_acc: 0.7096<br>Epoch 13/15<br> - 22s - loss: 0.3883 - acc: 0.8670 - val_loss: 1.0190 - val_acc: 0.7151<br>Epoch 14/15<br> - 22s - loss: 0.3334 - acc: 0.8874 - val_loss: 1.0025 - val_acc: 0.7232<br>Epoch 15/15<br> - 22s - loss: 0.2857 - acc: 0.9038 - val_loss: 1.0123 - val_acc: 0.7331<br></pre><pre>             precision    recall  f1-score   support</pre><pre>          0       0.64      0.73      0.68       319<br>          1       0.45      0.83      0.58       389<br>          2       0.81      0.64      0.71       394<br>          3       0.64      0.57      0.61       392<br>          4       0.55      0.78      0.64       385<br>          5       0.77      0.52      0.62       395<br>          6       0.84      0.77      0.80       390<br>          7       0.87      0.79      0.83       396<br>          8       0.85      0.90      0.87       398<br>          9       0.98      0.84      0.90       397<br>         10       0.93      0.96      0.95       399<br>         11       0.92      0.79      0.85       396<br>         12       0.59      0.53      0.56       393<br>         13       0.82      0.82      0.82       396<br>         14       0.84      0.84      0.84       394<br>         15       0.83      0.89      0.86       398<br>         16       0.68      0.86      0.76       364<br>         17       0.97      0.86      0.91       376<br>         18       0.66      0.50      0.57       310<br>         19       0.53      0.31      0.40       251</pre><pre>avg / total       0.77      0.75      0.75      7532</pre><p><strong>Random Multimodel Deep Learning (RMDL)</strong></p><p>A new ensemble, deep learning approach for classification. Deep learning models have achieved state-of-the-art results across many domains. RMDL solves the problem of finding the best deep learning structure and architecture while simultaneously improving robustness and accuracy through ensembles of different deep learning architectures. RDMLs can accept a variety of data as input including text, video, images, and symbols.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fEJ9gteqg8VJl_gQ" /></figure><p>Random Multimodel Deep Learning (RDML) architecture for classification. RMDL includes 3 Random models, oneDNN classifier at left, one Deep CNN classifier at middle, and one Deep RNN classifier at right (each unit could be LSTMor GRU).</p><p>Installation</p><p>There are pip and git for RMDL installation:</p><p>Using pip</p><pre>pip install RMDL</pre><p>Using git</p><pre>git clone --recursive <a href="https://github.com/kk7nc/RMDL.git">https://github.com/kk7nc/RMDL.git</a></pre><p>The primary requirements for this package are Python 3 with Tensorflow. The requirements.txt file contains a listing of the required Python packages; to install all requirements, run the following:</p><pre>pip -r install requirements.txt</pre><p>Or</p><pre>pip3  install -r requirements.txt</pre><p>Or:</p><pre>conda install --file requirements.txt</pre><p><strong>Hierarchical Deep Learning for Text (HDLTex)</strong></p><p>Increasingly large document collections require improved information processing methods for searching, retrieving, and organizing text documents. Central to these information processing methods is document classification, which has become an important task supervised learning aims to solve. Recently, the performance of traditional supervised classifiers has degraded as the number of documents has increased. This exponential growth of document volume has also increased the number of categories. <a href="https://arxiv.org/abs/1709.08267">This paper</a> approaches this problem differently from current document classification methods that view the problem as multi-class classification. Instead, we perform hierarchical classification using an approach we call Hierarchical Deep Learning for Text classification (HDLTex). HDLTex employs stacks of deep learning architectures to provide a hierarchical understanding of the documents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/0*AjBSNmof53jwsTP9" /></figure><h3>Conclusion</h3><p>This should give you a good idea of how you can leverage deep learning for text classification task! The list of techniques here are not exhaustive but definitely cover some of the most popular and widely used methods to train neural network models for the text classification task. I recommend you to try these out with your own data.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fb2a0d6%3Fas_embed%3Dtrue&amp;dntp=1&amp;display_name=Upscribe&amp;url=https%3A%2F%2Fupscri.be%2Fb2a0d6%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/0707f5c806284d01a4a13c7b13a91ce3/href">https://medium.com/media/0707f5c806284d01a4a13c7b13a91ce3/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9392ca9492c7" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/deep-learning-techniques-for-text-classification-9392ca9492c7">Deep Learning Techniques for Text Classification</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python Data Science Getting Started Tutorial: NLTK]]></title>
            <link>https://medium.datadriveninvestor.com/python-data-science-getting-started-tutorial-nltk-2d8842fedfdd?source=rss-84cdd99683f9------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d8842fedfdd</guid>
            <category><![CDATA[nltk]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[sklearn]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[SWAYAM MITTAL]]></dc:creator>
            <pubDate>Mon, 11 Feb 2019 07:30:49 GMT</pubDate>
            <atom:updated>2019-03-07T10:33:31.755Z</atom:updated>
            <content:encoded><![CDATA[<figure><a href="http://www.track.datadriveninvestor.com/1B9E"><img alt="" src="https://cdn-images-1.medium.com/max/700/0*_Msuja4EQ-rqSsTr" /></a></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XhLgddp7k5YE_UOm" /><figcaption>Photo by <a href="https://unsplash.com/@ffstop?utm_source=medium&amp;utm_medium=referral">Fotis Fotopoulos</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>I. Analyze words and sentences using NLTK</h3><p>Welcome to the Natural Language Processing series of tutorials, using Python’s natural language toolkit NLTK module.</p><p>The NLTK module is a huge toolkit designed to help you with the entire Natural Language Processing (NLP) approach. NLTK will provide you with everything from splitting paragraphs to sentences, splitting words, identifying the part of speech, highlighting themes, and even helping your machine understand what the text is about. In this series, we will address the areas of opinion mining or sentiment analysis.</p><p>As we learn how to use NLTK for sentiment analysis, we will learn the following:</p><ul><li>Participle — Splits the text body into sentences and words.</li><li>Part of speech tagging</li><li>Machine learning and naive Bayes classifier</li><li>How to use Scikit Learn (sklearn) with NLTK together</li><li>Training classifiers with data sets</li><li>Real-time streaming sentiment analysis with Twitter.</li><li>…and more.</li></ul><p>To get started, you need the NLTK module, as well as Python.</p><blockquote><a href="http://go.datadriveninvestor.com/pybootcamp/matf">DDI Editor’s Pick — Python for Data Science and Machine Learning Bootcamp</a></blockquote><p>If you don’t already have Python, go to python.org and download the latest version of Python (if you are on Windows). If you are on Mac or Linux, you should be able to run apt-get install python3 .</p><p>Next, you need NLTK 3. The easiest way to install an NLTK module is to use pip .</p><p>For all users, this is done by opening cmd.exe , bash, or any shell you use and typing the following command:</p><pre>pip install nltk</pre><p>Next, we need to install some components for NLTK. Open python in any of your usual ways and type:</p><pre>Import nltk<br>Nltk.download()</pre><p>Select Download All for all packages and click Download. This will give you all the word breakers, blockers, other algorithms and all the corpora. If space is an issue, you can choose to manually download all content. The NLTK module will take up approximately 7MB and the entire nltk_data directory will occupy approximately nltk_data , including your nltk_data , parser and corpus.</p><p>If you are running a headless version with VPS, you can install everything by running Python and doing the following:</p><pre>Import nltk<br>Nltk.download()<br>d (for download)<br>All (for download everything)</pre><p>This will download everything for you.</p><p>Now that you have all the things you need, let’s type some simple words:</p><ul><li>Corpus — The body of the text, singular. Corpora is its plural. Example: A collection of medical journals .</li><li>Lexicon — vocabulary and its meaning. For example: English dictionary. However, considering that there are different thesaurus in each field. For example, for financial investors, the first meaning of the word Bull is the person who is confident in the market. Compared with the “common English vocabulary”, the first meaning of the word is animal. Therefore, financial investors, doctors, children, mechanics, etc. all have a special vocabulary.</li><li>Token — Each “entity” is part of a rule based split. For example, when a sentence is “split” into words, each word is a tag. If you split a paragraph into sentences, each sentence can also be a marker.</li></ul><p>These are the most frequently heard words when entering the natural language processing (NLP) field, but we will cover more words in time. So, let’s show an example of how to split something into tags with the NLTK module.</p><pre>From nltk.tokenize import sent_tokenize, word_tokenize</pre><pre>EXAMPLE_TEXT = &quot;Hello Mr. Smith, how are you doing today? The weather is great, and Python is awesome. The sky is pinkish-blue. You shouldn&#39;t eat cardboard.&quot;</pre><pre>Print(sent_tokenize(EXAMPLE_TEXT))</pre><p>At first, you might think that word segmentation by word or sentence is quite a trivial matter. For many sentences, it may be. The first step might be to execute a simple .split(&#39;. &#39;) , or by a period, followed by a space split. Then maybe you will introduce some regular expressions, divided by periods, spaces, and then uppercase letters. The problem is something like Mr. Smith , and there are many other things that will cause you trouble. Segmentation by word is also a challenge, especially when considering abbreviations, such as we and we&#39;re . NLTK saves you a lot of time with this seemingly simple but very complicated operation.</p><p>The above code will output the sentence, divided into a list of sentences, you can use the for loop to traverse.</p><pre>[ &#39;Hello Mr. Smith, how are you doing today?&#39;, &#39;The weather is great, and Python is awesome.&#39;, &#39;The sky is pinkish-blue.&#39;, &quot;You shouldn&#39;t eat cardboard.&quot; ]</pre><p>So here we create the tags, they are all sentences. Let us divide the words by word this time.</p><pre>Print (word_tokenize(EXAMPLE_TEXT))</pre><pre>[ &#39;Hello&#39; , &#39;Mr.&#39; , &#39;Smith&#39; , &#39;,&#39; , &#39;how&#39; , &#39;are&#39; , &#39;you &#39; , &#39;doing &#39; , &#39;today &#39; , &#39;?&#39; , &#39;The&#39; , &#39;weather&#39; , &#39;is&#39; , &#39;great&#39; , &#39;,&#39; , &#39;and&#39; , &#39;Python&#39; , &#39;is&#39; , &#39;awesome &#39; , &#39;.&#39; , &#39;The&#39; , &#39;sky&#39; , &#39;is&#39; , &#39;pinkish-blue&#39; , &#39;.&#39; , &#39;You&#39; , &#39;should&#39; , &quot;n&#39;t&quot; , &#39;eat&#39; , &#39;cardboard&#39; , &#39;.&#39; ]</pre><p>There are a few things to note here. First, notice that punctuation is treated as a separate tag. Also, note that the word shouldn&#39;t separated into should and n&#39;t . The last thing to note is that pinkish-blue is indeed treated as &quot;a word&quot;, which is what it is. Cool!</p><p>Now, looking at the words after these participles, we must start thinking about what our next step might be. We began to think about how to get the meaning by observing these words. We can think about how to put value on many words, but we also see some words that are basically worthless. This is a form of “stop word” that we can also handle. This is what we will discuss in the next tutorial.</p><h3>II. NLTK and stop words</h3><p>The idea of ​​natural language processing is to perform some form of analysis or processing. The machine can at least understand the meaning, representation or suggestion of the text to some extent.</p><p>This is obviously a huge challenge, but there are steps that anyone can follow. However, the main idea is that computers don’t understand words directly. It is shocking that humans will not. In humans, memory is broken down into electrical signals in the brain in the form of a neural group that emits patterns. There are still many unknown things about the brain, but the more we break down the human brain into basic elements, we will find the basic elements. Well, it turns out that computers store information in a very similar way! If we want to mimic how humans read and understand text, we need a way that is as close as possible. In general, computers use numbers to represent everything, but we often see the use of binary signals directly in programming ( True or False , which can be directly converted to 1 or 0, directly from the presence of an electrical signal (True, 1) or not Exist (False, 0) ). To do this, we need a way to convert words to numeric or signal patterns. Converting data into something that a computer can understand is called &quot;preprocessing.&quot; One of the main forms of preprocessing is to filter out useless data. In natural language processing, useless words (data) are called stop words.</p><p>We can immediately realize that some words are more meaningful than others. We can also see that some words are useless and are filled words. For example, we use them in English to fill sentences, so there is no such strange sound. One of the most common, unofficial, useless examples is the word umm . People often use umm to fill, more than other words. This word is meaningless unless we are looking for someone who may lack confidence, confusion, or not much. We all do this, there are... oh... many times, you can hear me say umm or uhh in the video. For most analyses, these words are useless.</p><p>We don’t want these words to take up space in our database or take up valuable processing time. Therefore, we call these words “useless words” because they are useless and we want to treat them. Another version of the word “stop word” can be written more: the words we stop at.</p><p>For example, if you find words that are often used for satire, you may want to stop immediately. Satirical words or phrases will vary depending on the thesaurus and corpus. For the time being, we will treat stop words as words that do not contain any meaning, and we will remove them.</p><p>You can do it easily by storing a list of words that you think are stop words. NLTK uses a bunch of words that they think are stop words to get you started, you can access it through the NLTK corpus:</p><pre>From nltk.corpus import stopwords</pre><p>Here is the list:</p><pre>&gt;&gt;&gt; set (stopwords. words ( &#39;english&#39; ))<br> { &#39;ourselves&#39; , &#39;hers&#39; , &#39;between&#39; , &#39;yourself&#39; , &#39;but&#39; , &#39;again &#39; , &#39;there &#39; , &#39;about &#39; , &#39;once &#39; , &#39;during &#39; , &#39; out &#39; , &#39;very &#39; , &#39; Having&#39; , &#39;with&#39; , &#39;they&#39; , &#39;own&#39; , &#39;an&#39; , &#39;be&#39; , &#39;some&#39; , &#39;for&#39; , &#39;do&#39; , &#39;its&#39; , &#39;yours &#39; , &#39;such&#39; , &#39;into&#39; , &#39;of&#39; , &#39;most&#39; , &#39;itself&#39; , &#39;other&#39; , &#39;off&#39; , &#39;is&#39; , &#39;s&#39; , &#39;am&#39; , &#39;or&#39; , &#39;who &#39; , &#39;as&#39; , &#39;from &#39; , &#39; Him&#39; , &#39;each&#39; , &#39;the&#39; , &#39;themselves&#39; , &#39;until&#39; , &#39;below&#39; , &#39;are&#39; , &#39;we&#39; , &#39;these &#39; , &#39;your &#39; , &#39;his&#39; , &#39;through&#39; , &#39;don&#39; , &#39;nor&#39; , &#39;me&#39; , &#39;were&#39; , &#39;her&#39; , &#39;more&#39; , &#39;himself &#39; , &#39;this&#39; , &#39;down&#39; , &#39;should&#39; , &#39;our&#39; , &#39;their &#39; , &#39;while &#39; , &#39; Above&#39; , &#39;both&#39; , &#39;up&#39; , &#39;to&#39; , &#39;ours&#39; , &#39;had&#39; , &#39;she&#39; , &#39;all&#39; , &#39;no&#39; , &#39;when&#39; , &#39;at&#39; , &#39;any&#39; , &#39;before&#39; , &#39;them&#39; , &#39;same&#39; , &#39;and&#39; , &#39;been&#39; , &#39;have&#39; , &#39;in&#39; , &#39;will&#39; , &#39;on&#39; , &#39;does &#39; , &#39;yourselves &#39; , &#39;then &#39; , &#39;that &#39; , &#39; Because&#39; , &#39;what&#39; , &#39;over&#39; , &#39;why&#39; , &#39;so&#39; , &#39;can&#39; , &#39;did &#39; , &#39;not&#39; , &#39; now&#39; , &#39;under&#39; , &#39;he&#39; , &#39;you&#39; , &#39;herself&#39; , &#39;has&#39; , &#39;just&#39; , &#39;where&#39; , &#39;too&#39; , &#39;only&#39; , &#39;myself &#39; , &#39;which&#39; , &#39;those &#39; , &#39;i&#39; , &#39;after&#39; , &#39;few &#39; , &#39;whom&#39;  , &#39;t&#39; , &#39;being&#39; , &#39;if&#39; , &#39;theirs&#39; , &#39;my&#39; , &#39;against &#39; , &#39;a&#39; , &#39;by &#39; , &#39;doing &#39; , &#39;it &#39; , &#39;how&#39; , &#39;further &#39; , &#39; Was&#39; , &#39;here&#39; , &#39;than&#39; }</pre><p>Here’s how to use the stop_words collection to remove stop words from the text:</p><pre>From nltk.corpus import stopwords<br>From nltk.tokenize import word_tokenize</pre><pre>Example_sent = &quot;This is a sample sentence, showing off the stop words filtration.&quot;</pre><pre>Stop_words = set(stopwords.words( &#39;english&#39; ))</pre><pre>Word_tokens = word_tokenize(example_sent)</pre><pre>Filtered_sentence = [w for w in word_tokens if not w in stop_words]</pre><pre>Filtered_sentence = []</pre><pre>For w in word_tokens:<br>   If w not in stop_words:<br>      Filtered_sentence.append(w)</pre><pre>Print(word_tokens)<br>Print(filtered_sentence)</pre><p>Our output is:</p><pre>[&#39;This&#39;, &#39;is&#39;, &#39;a&#39;, &#39;sample&#39;, &#39;sentence&#39;, &#39;,&#39;, &#39;showing&#39;, &#39;off&#39;, &#39;the&#39;, &#39;stop&#39;, &#39;words&#39;, &#39;filtration&#39;, &#39; .&#39;]<br>[&#39;This&#39;, &#39;sample&#39;, &#39;sentence&#39;, &#39;,&#39;, &#39;showing&#39;, &#39;stop&#39;, &#39;words&#39;, &#39;filtration&#39;, &#39;.&#39;]</pre><p>Thanks to our database. Another form of data preprocessing is “Stemming,” which is what we’ll discuss next.</p><h3>III. NLTK stem extraction</h3><p>The concept of stemming is a standardized approach. In addition to the tense, many variations of the word have the same meaning.</p><p>The reason we extract the stem is to shorten the time of the search and normalize the sentence.</p><p>consider:</p><pre>I was taking a ride in the car.<br>I was riding in the car.</pre><p>These two sentences mean the same thing. in the car (in the car) is the same. I (I) is the same. In both cases, ing clearly expresses the past tense, so in the case of trying to figure out the meaning of this past-style activity, is it really necessary to distinguish between riding and taking a ride ?</p><p>No, not.</p><p>This is just a small example, but imagine every word in English that can be placed on every possible tense and affix on the word. Each version has a separate dictionary entry that will be very redundant and inefficient, especially since once we convert to numbers, the “value” will be the same.</p><p>One of the most popular porcelain extraction algorithms is Porter, which existed in 1979.</p><p>First, we have to crawl and define our stems:</p><pre>From nltk.stem import PorterStemmer<br>From nltk.tokenize import sent_tokenize, word_tokenize</pre><pre>Ps = PorterStemmer()</pre><p>Now let’s choose some words with similar stems, for example:</p><pre>Example_words = [ &quot;python&quot; , &quot;pythoner&quot; , &quot;pythoning&quot; , &quot;pythoned&quot; , &quot;pythonly&quot; ]</pre><p>Below, we can do this to easily extract stems:</p><pre>For w in example_words:<br>   Print (ps.stem(w))</pre><p>Our output:</p><pre>Python<br>Python<br>Python<br>Python<br>Pythonli</pre><p>Now let’s try to extract stems from a typical sentence instead of some words:</p><pre>New_text = &quot;It is important to by very pythonly while you are pythoning with python. All pythoners have pythoned poorly at least once.&quot;<br>Words = word_tokenize(new_text)</pre><pre>For w in words :<br>   Print(ps.stem(w))</pre><p>Our results now are:</p><pre>It<br>Is<br>Import<br>To<br>By<br>Veri<br>Pythonli<br>While<br>You<br>Are<br>Python<br>With<br>Python<br>.<br>All<br>Python<br>Have<br>Python<br>Poorli<br>At<br>Least<br>Onc<br>.</pre><p>Next, we’ll discuss some of the more advanced content of the NLTK module, part-of-speech tagging, where we can use the NLTK module to identify the part of speech of each word in the sentence.</p><h3>IV. NLTK part-of-speech tagging</h3><p>A more powerful aspect of the NLTK module is that it can be used for your part-of-speech tagging. It means to mark words in a sentence as nouns, adjectives, verbs, etc. Even more impressive is that it can also be marked by tense, and others. This is a list of tags, their meaning and some examples:</p><pre>POS tag list:</pre><pre> CC coordinating conjunction<br> CD cardinal digit<br> DT determiner<br> EX existential there ( like : &quot;there is&quot; ... think of it like &quot;there exists&quot; )<br> FW foreign word<br> IN preposition/subordinating conjunction<br> JJ adjective &#39;big&#39;<br> JJR adjective, comparative &#39;bigger&#39;<br> JJS adjective, superlative &#39;biggest&#39;<br> LS list marker 1 )<br> MD modal could, will<br> NN noun, singular &#39;desk&#39;<br> NNS noun plural &#39;desks&#39;<br> NNP proper noun, singular &#39;Harrison&#39;<br> NNPS proper noun, plural &#39;Americans&#39;<br> PDT predeterminer &#39;all the kids&#39;<br> POS possessive ending parent &#39;s<br> PRP personal pronoun I, he, she<br> PRP$ possessive pronoun my, his, hers<br> RB adverb very, silently,<br> RBR adverb, comparative better<br> RBS adverb, superlative best<br> RP particle give up<br> TO to go &#39;to&#39; the store.<br> UH interjection errrrrrrrm<br> VB verb, base form take<br> VBD verb, past tense took<br> VBG verb, gerund/present participle taking<br> VBN verb, past participle taken<br> VBP verb, sing. present, non- 3 d take<br> VBZ verb, 3 rd person sing. present takes<br> WDT wh-determiner which<br> WP wh-pronoun who, what<br> WP$ possessive wh-pronoun whose<br> WRB wh-abverb where , when</pre><p>How do we use this? When we deal with it, we have to explain a new sentence marker called PunktSentenceTokenizer . This marker enables unsupervised machine learning, so you can actually train on any text you use. First, let&#39;s get some imports that we plan to use:</p><pre>Import nltk<br>From nltk.corpus import state_union<br>From nltk.tokenize import PunktSentenceTokenizer</pre><p>Now let’s create the training and test data:</p><pre>Train_text = state_union.raw( &quot;2005-GWBush.txt&quot; )<br>Sample_text = state_union.raw( &quot;2006-GWBush.txt&quot; )</pre><p>One is the State of the Union speech since 2005, and the other is the speech of President George W. Bush since 2006.</p><p>Next, we can train the Punkt marker as follows:</p><pre>Custom_sent_tokenizer = PunktSentenceTokenizer(train_text)</pre><p>After that we can actually divide the word and use:</p><pre>Tokenized = custom_sent_tokenizer.tokenize(sample_text)</pre><p>Now we can complete the part-of-speech tagging script by creating a function that will traverse and mark the part of speech of each sentence as follows:</p><pre>Def process_content () :<br>   Try :<br>      For i in tokenized[: 5 ]:<br>         Words = nltk.word_tokenize(i)<br>         Tagged = nltk.pos_tag(words)<br>         Print(tagged)</pre><pre>   Except Exception as e:<br>        Print(str(e))<br></pre><pre>Process_content()</pre><p>The output should be a tuple list, with the first element in the tuple being a word and the second element being a part of speech tag. It should look like:</p><pre>[(&#39;PRESIDENT&#39;, &#39;NNP&#39;), (&#39;GEORGE&#39;, &#39;NNP&#39;), (&#39;W.&#39;, &#39;NNP&#39;), (&#39;BUSH&#39;, &#39;NNP&#39;), ( &quot;&#39;S&quot; , &#39;POS &#39;), (&#39;ADDRESS&#39;, &#39;NNP&#39;), (&#39;BEFORE&#39;, &#39;NNP&#39;), (&#39;A&#39;, &#39;NNP&#39;), (&#39;JOINT&#39;, &#39;NNP&#39;), (&#39;SESSION&#39;, &#39;NNP &#39;), (&#39;OF&#39;, &#39;NNP&#39;), (&#39;THE&#39;, &#39;NNP&#39;), (&#39;CONGRESS&#39;, &#39;NNP&#39;), (&#39;ON&#39;, &#39;NNP&#39;), (&#39;THE&#39;, &#39;NNP &#39;), (&#39;STATE&#39;, &#39;NNP&#39;), (&#39;OF&#39;, &#39;NNP&#39;), (&#39;THE&#39;, &#39;NNP&#39;), (&#39;UNION&#39;, &#39;NNP&#39;), (&#39;January&#39;, &#39;NNP &#39;), (&#39; 31 &#39;, &#39;CD&#39;), (&#39;,&#39;, &#39;,&#39;), (&#39; 2006 &#39;, &#39;CD&#39;), (&#39;THE&#39;, &#39;DT&#39;), (&#39;PRESIDENT&#39;, &#39;NNP &#39;), (&#39;:&#39;, &#39;:&#39;), (&#39;Thank&#39;, &#39;NNP&#39;), (&#39;you&#39;, &#39;PRP&#39;), (&#39;all&#39;, &#39;DT&#39;), (&#39;.&#39;, &#39;. &#39;)] [(&#39;Mr.&#39;, &#39;NNP&#39;), (&#39;Speaker&#39;, &#39;NNP&#39;), (&#39;,&#39;, &#39;,&#39;), (&#39;Vice&#39;, &#39;NNP&#39;), (&#39;President&#39;, &#39;NNP&#39;), (&#39;Cheney&#39;, &#39;NNP&#39;), (&#39;,&#39;, &#39;,&#39;), (&#39;members&#39;, &#39;NNS&#39;), (&#39;of&#39;, &#39;IN&#39;), (&#39;Congress&#39;, &#39;NNP&#39;), (&#39;,&#39;, &#39;,&#39;), (&#39;members&#39;, &#39;NNS&#39;), (&#39;of&#39;, &#39;IN&#39;), (&#39;the&#39;, &#39;DT&#39;), (&#39;Supreme&#39;, &#39;NNP&#39;), (&#39;Court&#39;, &#39;NNP&#39;), (&#39;and&#39;, &#39;CC&#39;), (&#39;diplomatic&#39;, &#39;JJ&#39;), (&#39;corps&#39;, &#39;NNS&#39;), (&#39;,&#39;, &#39;,&#39;), (&#39;distinguished&#39;, &#39;VBD&#39;), (&#39;guests&#39;, &#39;NNS&#39;), (&#39;,&#39;, &#39;,&#39;), (&#39;and&#39;, &#39;CC&#39;), (&#39;fellow&#39;, &#39;JJ&#39;), (&#39;citizens&#39;, &#39;NNS&#39;), (&#39;:&#39;, &#39;:&#39;), (&#39;Today&#39;, &#39;NN&#39;), (&#39;our&#39;, &#39;PRP  $&#39;), (&#39;nation&#39;, &#39;NN&#39;), (&#39;lost&#39;, &#39;VBD&#39;), (&#39;a&#39;, &#39;DT&#39;), (&#39;beloved&#39;, &#39;VBN&#39;), (&#39;,&#39;, &#39; ,&#39;), (&#39;graceful&#39;, &#39;JJ&#39;), (&#39;,&#39;, &#39;,&#39;), (&#39;courageous&#39;, &#39;JJ&#39;), (&#39;woman&#39;, &#39;NN&#39;), (&#39;who&#39;, &#39; WP&#39;), (&#39;called&#39;, &#39;VBN&#39;), (&#39;America&#39;, &#39;NNP&#39;), (&#39;to&#39;, &#39;TO&#39;), (&#39;its&#39;, &#39;PRP$&#39;), (&#39;founding&#39;, &#39;NN&#39;), (&#39;ideals&#39;, &#39;NNS&#39;), (&#39;and&#39;, &#39;CC&#39;), (&#39;carried&#39;, &#39;VBD&#39;), (&#39;on&#39;, &#39;IN&#39;), (&#39;a&#39;, &#39;DT&#39;), (&#39;noble&#39;, &#39;JJ&#39;), (&#39;dream&#39;, &#39;NN&#39;), (&#39;.&#39;, &#39;.&#39;)] [(&#39;Tonight&#39;, &#39;NNP&#39;), (&#39;we&#39; , &#39;PRP&#39;), (&#39;are&#39;, &#39;VBP&#39;), (&#39;comforted&#39;, &#39;VBN&#39;), (&#39;by&#39;, &#39;IN&#39;), (&#39;the&#39;, &#39;DT&#39;), (&#39;hope&#39; , &#39;NN&#39;), (&#39;of&#39;, &#39;IN&#39;), (&#39;a&#39;, &#39;DT&#39;), (&#39;glad&#39;, &#39;NN&#39;), (&#39;reunion&#39;, &#39;NN&#39;), (&#39;with&#39; , &#39;IN&#39;), (&#39;the&#39;, &#39;DT&#39;), (&#39;husband&#39;, &#39;NN&#39;), (&#39;who&#39;, &#39;WP&#39;), (&#39;was&#39;, &#39;VBD&#39;), (&#39;taken&#39; , &#39;VBN&#39;), (&#39;so&#39;, &#39;RB&#39;), (&#39;long&#39;, &#39;RB&#39;), (&#39;ago&#39;, &#39;RB&#39;), (&#39;,&#39;, &#39;,&#39;), (&#39;and&#39; , &#39;CC&#39;), (&#39;we&#39;, &#39;PRP&#39;), (&#39;are&#39;, &#39;VBP&#39;), (&#39;grateful&#39;, &#39;JJ&#39;), (&#39;for&#39;, &#39;IN&#39;), (&#39;the&#39; , &#39;DT&#39;), (&#39;good&#39;, &#39;NN&#39;), (&#39;life&#39;, &#39;NN&#39;), (&#39;of&#39;, &#39;IN&#39;), (&#39;Coretta&#39;, &#39;NNP&#39;), (&#39;Scott&#39; , &#39;NNP&#39;), (&#39;King&#39;, &#39;NNP&#39;), (&#39;.&#39;, &#39;.&#39;)] [(&#39;(&#39;, &#39;NN&#39;), (&#39;Applause&#39;, &#39;NNP&#39;), (&#39;. &#39;, &#39;.&#39;), (&#39;)&#39;, &#39;:&#39;)] [(&#39;Pres  Ident&#39;, &#39;NNP&#39;), (&#39;George&#39;, &#39;NNP&#39;), (&#39;W.&#39;, &#39;NNP&#39;), (&#39;Bush&#39;, &#39;NNP&#39;), (&#39;reacts&#39;, &#39;VBZ&#39;), ( &#39;to&#39;, &#39;TO&#39;), (&#39;applause&#39;, &#39;VB&#39;), (&#39;during&#39;, &#39;IN&#39;), (&#39;his&#39;, &#39;PRP$&#39;), (&#39;State&#39;, &#39;NNP&#39;), (&#39;of&#39;, &#39;IN&#39;), (&#39;the&#39;, &#39;DT&#39;), (&#39;Union&#39;, &#39;NNP&#39;), (&#39;Address&#39;, &#39;NNP&#39;), (&#39;at&#39;, &#39;IN&#39;), (&#39;the&#39;, &#39;DT&#39;), (&#39;Capitol&#39;, &#39;NNP&#39;), (&#39;,&#39;, &#39;,&#39;), (&#39;Tuesday&#39;, &#39;NNP&#39;), (&#39;,&#39;, &#39;,&#39;), (&#39;Jan&#39;, &#39;NNP&#39;), (&#39;.&#39;, &#39;.&#39;)]</pre><p>When we get here, we can start to get the meaning, but there is still some work to be done. The next topic we will discuss is chunking, where we follow the part of the word, and divide the words into meaningful groups.</p><h3>V. NLTK block</h3><p>Now that we know the part of speech, we can pay attention to the so-called block, which divides the word into meaningful blocks. One of the main goals of blocking is to group so-called “noun phrases”. These are phrases that contain one or more words of a noun, which may be descriptive words, a verb, or an adverb. The idea is to combine nouns and words related to them.</p><p>In order to block, we combine the part-of-speech tag with a regular expression. Mainly from regular expressions, we want to take advantage of these things:</p><pre>+ = match 1 or more<br>? = match 0 or 1 repetitions.<br>* = match 0 or MORE repetitions   <br>. = Any character except a new line</pre><p>If you need help with regular expressions, see the tutorial linked above. The last thing to note is that the word tag is represented by &lt; and &gt; . We can also place a regular expression in the tag itself to express &quot;all nouns&quot; ( &lt;N.*&gt; ).</p><pre>Import nltk<br>From nltk.corpus import state_union<br>From nltk.tokenize import PunktSentenceTokenizer</pre><pre>Train_text = state_union.raw( &quot;2005-GWBush.txt&quot; )<br>Sample_text = state_union.raw( &quot;2006-GWBush.txt&quot; )</pre><pre>Custom_sent_tokenizer = PunktSentenceTokenizer(train_text)</pre><pre>Tokenized = custom_sent_tokenizer.tokenize(sample_text)</pre><pre>Def process_content () :<br>    Try :<br>        For i in tokenized:<br>            Words = nltk.word_tokenize(i)<br>            Tagged = nltk.pos_tag(words)<br>            chunkGram = r&quot;&quot;&quot;Chunk: {&lt;RB.?&gt;*&lt;VB.?&gt;*&lt;NNP&gt;+&lt;NN&gt;?}&quot;&quot;&quot;<br>            chunkParser = nltk.RegexpParser(chunkGram)<br>            Chunked = chunkParser.parse(tagged)<br>            Chunked.draw()     </pre><pre>    Except Exception as e:<br>        Print(str(e))</pre><pre> Process_content()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1/0*ilfwrPNbRjkwuDJ1.png" /></figure><p>The main line here is:</p><pre>chunkGram = r&quot;&quot;&quot;Chunk: {&lt;RB.?&gt;*&lt;VB.?&gt;*&lt;NNP&gt;+&lt;NN&gt;?}&quot;&quot;&quot;</pre><p>Split this line apart:</p><p>&lt;RB.?&gt;* : Zero or more adverbs of any tense, followed by:</p><p>&lt;VB.?&gt;* : Zero or more verbs of any tense, followed by:</p><p>&lt;NNP&gt;+ : One or more reasonable nouns, followed by:</p><p>&lt;NN&gt;? : Zero or one noun singular.</p><p>Try a fun mix to group the various instances until you feel familiar.</p><p>Not covered in the video, but there is also a reasonable task to actually access a specific block. This is rarely mentioned, but depending on what you are doing, this can be an important step. Suppose you print out the block and you will see the following output:</p><pre>( S ( Chunk PRESIDENT/NNP GEORGE/NNP W./NNP BUSH/NNP) &#39;S/POS ( Chunk ADDRESS/NNP BEFORE/NNP A/NNP JOINT/NNP SESSION/NNP OF/NNP THE/NNP CONGRESS/NNP ON/ NNP THE/NNP STATE/NNP OF/NNP THE/NNP UNION/NNP January/NNP) 31 /CD , /, 2006 /CD THE/DT ( Chunk PRESIDENT/NNP ) :/ : ( Chunk Thank/NNP) you/PRP All/DT ./.)</pre><p>Cool, this helps us visualize, but what if we want to access this data through our program? So what happens here is that our “blocking” variable is an NLTK tree. Each “block” and “non-block” is the “subtree” of the tree. We can refer to them by something like chunked.subtrees . Then we can iterate through these subtrees like this:</p><pre>For subtree in chunked.subtrees():<br>        Print (subtree)</pre><p>Next, we may only care about getting these blocks and ignoring the rest. We can use the filter parameter in the chunked.subtrees() call.</p><pre>For subtree in chunked.subtrees(filter= lambda t: t.label() == &#39;Chunk&#39; ):<br>        Print(subtree)</pre><p>Now we perform filtering to display the subtree with the label “Block”. Keep in mind that this is not a “block” in the NLTK block attribute… this is a literal “block” because this is the label we gave it: chunkGram = r&quot;&quot;&quot;Chunk: {&lt;RB.?&gt;*&lt;VB.?&gt;*&lt;NNP&gt;+&lt;NN&gt;?}&quot;&quot;&quot; .</p><p>If we write something like chunkGram = r&quot;&quot;&quot;Pythons: {&lt;RB.?&gt;*&lt;VB.?&gt;*&lt;NNP&gt;+&lt;NN&gt;?}&quot;&quot;&quot; , then we can pass &quot;Pythons.&quot; Tags to filter. The result should be like this:</p><pre>- ( Chunk PRESIDENT / NNP GEORGE / NNP W . / NNP BUSH / NNP ) ( Chunk ADDRESS / NNP BEFORE / NNP A / NNP JOINT / NNP SESSION / NNP OF / NNP THE / NNP CONGRESS / NNP ON / NNP THE / NNP STATE / NNP OF / NNP THE / NNP UNION / NNP January / NNP ) ( Chunk PRESIDENT / NNP ) ( Chunk Thank / NNP )</pre><p>The complete code is:</p><pre>Import nltk<br>From nltk.corpus import state_union<br>From nltk.tokenize import PunktSentenceTokenizer</pre><pre>Train_text = state_union.raw( &quot;2005-GWBush.txt&quot; )<br>Sample_text = state_union.raw( &quot;2006-GWBush.txt&quot; )</pre><pre>Custom_sent_tokenizer = PunktSentenceTokenizer(train_text)</pre><pre>Tokenized = custom_sent_tokenizer.tokenize(sample_text)</pre><pre>Def process_content () :<br>    Try :<br>        For i in tokenized:<br>            Words = nltk.word_tokenize(i)<br>            Tagged = nltk.pos_tag(words)<br>            chunkGram = r&quot;&quot;&quot;Chunk: {&lt;RB.?&gt;*&lt;VB.?&gt;*&lt;NNP&gt;+&lt;NN&gt;?}&quot;&quot;&quot;<br>            chunkParser = nltk.RegexpParser(chunkGram)<br>            Chunked = chunkParser.parse(tagged)</pre><pre>            Print(chunked)<br>            For subtree in chunked.subtrees(filter= lambda t: t.label() == &#39;Chunk&#39; ):<br>                Print(subtree)</pre><pre>            Chunked.draw()</pre><pre>    Except Exception as e:<br>        Print(str(e))</pre><pre>Process_content()</pre><h3>VI. NLTK adds a gap (Chinking)</h3><p>You may find that after a lot of chunking, there are some words in your block that you don’t want, but you don’t know how to get rid of them by blocking. You may find that adding a gap is your solution.</p><p>Adding a gap is similar to a block, which is basically a way to remove a block from a block. The block you removed from the block is your gap.</p><p>The code is very similar, you only need to use }{ to code the gap, behind the block, not the {} block.</p><pre>Import nltk<br>From nltk.corpus import state_union<br>From nltk.tokenize import PunktSentenceTokenizer</pre><pre>Train_text = state_union.raw( &quot;2005-GWBush.txt&quot; )<br>Sample_text = state_union.raw( &quot;2006-GWBush.txt&quot; )</pre><pre>Custom_sent_tokenizer = PunktSentenceTokenizer(train_text)</pre><pre>Tokenized = custom_sent_tokenizer.tokenize(sample_text)</pre><pre>Def process_content () :<br>    Try :<br>        For i in tokenized[ 5 :]:<br>            Words = nltk.word_tokenize(i)<br>            Tagged = nltk.pos_tag(words)</pre><pre>            chunkGram = r&quot;&quot;&quot;Chunk: {&lt;.*&gt;+} }&lt;VB.?|IN|DT|TO&gt;+{&quot;&quot;&quot;</pre><pre>            chunkParser = nltk.RegexpParser(chunkGram)<br>            Chunked = chunkParser.parse(tagged)</pre><pre>            Chunked.draw()</pre><pre>    Except Exception as e:<br>        Print(str(e))</pre><pre>Process_content()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1/0*4S8Tht6BakAT8gcb.png" /></figure><p>Now, the main difference is:</p><pre>}&lt;VB.?| IN |DT| TO &gt;+{</pre><p>This means that we want to remove one or more verbs, prepositions, qualifiers or to words from the gap.</p><p>Now that we have learned how to perform some custom partitioning and adding gaps, let’s discuss the block form that comes with NLTK, which is named entity recognition.</p><h3>VII. NLTK named entity recognition</h3><p>One of the most important forms of blocking in natural language processing is called “named entity recognition.” The idea is to let the machine immediately pull out “entities” such as people, places, things, locations, currencies, and more.</p><p>This can be a challenge, but NLTK is built for us. NLTK’s named entity recognition has two main options: identifying all named entities, or identifying named entities as their respective types, such as person, location, location, etc.</p><p>This is an example:</p><pre>Import nltk<br>From nltk.corpus import state_union<br>From nltk.tokenize import PunktSentenceTokenizer</pre><pre>Train_text = state_union.raw( &quot;2005-GWBush.txt&quot; )<br>Sample_text = state_union.raw( &quot;2006-GWBush.txt&quot; )</pre><pre>Custom_sent_tokenizer = PunktSentenceTokenizer(train_text)</pre><pre>Tokenized = custom_sent_tokenizer.tokenize(sample_text)</pre><pre>Def process_content () :<br>    Try :<br>        For i in tokenized[ 5 :]:<br>            Words = nltk.word_tokenize(i)<br>            Tagged = nltk.pos_tag(words)<br>            namedEnt = nltk.ne_chunk(tagged, binary= True )<br>            namedEnt.draw()<br>    Except Exception as e:<br>        Print(str(e))<br></pre><pre>Process_content()</pre><p>You can see something right away. When binary is false, it also picks the same thing, but breaks the term White House into White and House as if they are different, and we can see it in the option binary = True , named entity The identification says that White House is part of the same named entity, which is correct.</p><p>According to your goal, you can use the binary option. If your binary is false , here is what you can get, the type of named entity:</p><pre>NE Type and Examples<br>ORGANIZATION - Georgia -Pacific Corp . , WHO<br>PERSON - Eddy Bonte, President Obama<br>LOCATION - Murray River, Mount Everest<br>DATE - June, 2008 - 06 - 29<br>TIME - two fifty am, 1 : 30 p . m .<br>MONEY - 175 million Canadian Dollars, GBP 10.40<br>PERCENT - twenty pct, 18.75 %<br>FACILITY - Washington Monument, Stonehenge<br>GPE - South East Asia, Midlothian</pre><p>Either way, you may find that you need to do more work to get it right, but this feature is very powerful.</p><p>In the next tutorial, we will discuss something similar to stem extraction, called “lemmatizing”.</p><h3>VIII. NLTK morphological restoration</h3><p>An operation that is very similar to stemming is called a morphological restoration. The main difference between the two is that, as you have seen before, stemming powers often create words that don’t exist, while word forms are actual words.</p><p>So, your stem, that is, the word you end up with, is not something you can look up in the dictionary, but you can find a word form.</p><p>Sometimes you end up with very similar words, but sometimes you get completely different words. Let’s look at some examples.</p><pre>From nltk.stem import WordNetLemmatizer</pre><pre>Lemmatizer = WordNetLemmatizer()</pre><pre> Print(lemmatizer.lemmatize( &quot;cats&quot; ))<br> Print(lemmatizer.lemmatize( &quot;cacti&quot; ))<br> Print(lemmatizer.lemmatize( &quot;geese&quot; ))<br> Print(lemmatizer.lemmatize( &quot;rocks&quot; ))<br> Print(lemmatizer.lemmatize( &quot;python&quot; ))<br> Print(lemmatizer.lemmatize( &quot;better&quot; , pos= &quot;a&quot; ))<br> Print(lemmatizer.lemmatize( &quot;best&quot; , pos= &quot;a&quot; ))<br> Print(lemmatizer.lemmatize( &quot;run&quot; ))<br> Print(lemmatizer.lemmatize( &quot;run&quot; , &#39;v&#39; ))</pre><p>Here we have some examples of the form of the word we use. The only thing to note is that lemmatize accepts the part-of-speech parameter pos . If not provided, the default is &quot;noun&quot;. This means that it will try to find the closest noun, which can cause you trouble. If you use morphological restoration, please remember!</p><p>In the next tutorial, we’ll dive into the NTLK corpus that comes with the module to see all the great documentation, and they are waiting for us there.</p><h3>IX. NLTK Corpus</h3><p>In this part of the tutorial, I want to take a moment to dive into the corpus we downloaded all! The NLTK corpus is a collection of natural language data that is definitely worth a look.</p><p>Almost all files in the NLTK corpus follow the same rules, accessing them by using the NLTK module, but they are nothing magical. Most of these files are plain text files, some of which are XML files, others are other format files, but can be accessed manually or in modules and Python. Let’s talk about viewing them manually.</p><p>Depending on your installation, your nltk_data directory may be hidden in multiple locations. To find out where it is, go to your Python directory, where the NLTK module is located. If you don&#39;t know where, please use the following code:</p><pre>Import nltk<br>Print (nltk. __file__ )</pre><p>Run it and the output will be the location of the NLTK module __init__.py . data.py to the NLTK directory and look for the data.py file.</p><p>The important part of the code is:</p><pre>If sys.platform.startswith( &#39;win&#39; ):<br>    # Common locations on Windows:<br>    Path += [<br>       Str( r&#39;C:\nltk_data&#39; ), str( r&#39;D:\nltk_data&#39; ), str( r&#39;E:\nltk_data&#39; ),<br>        Os.path.join(sys.prefix, str( &#39;nltk_data&#39; )),<br>        Os.path.join(sys.prefix, str( &#39;lib&#39; ), str( &#39;nltk_data&#39; )),<br>        Os.path.join(os.environ.get(str( &#39;APPDATA&#39; ), str( &#39;C:\\&#39; )), str( &#39;nltk_data&#39; ))<br>     ]<br>Else :<br>    # Common locations on UNIX &amp; OS X:<br>    Path += [<br>        Str( &#39;/usr/share/nltk_data&#39; ),<br>        Str( &#39;/usr/local/share/nltk_data&#39; ),<br>        Str( &#39;/usr/lib/nltk_data&#39; ),<br>        Str( &#39;/usr/local/lib/nltk_data&#39; )<br>    ]</pre><p>There, you can see the various possible directories for nltk_data . If you&#39;re on Windows, it&#39;s probably in your appdata , in a local directory. To do this, you need to open your file browser, go to the top, and type %appdata% .</p><p>Next click on roaming and find the nltk_data directory. There, you will find your corpus file. The complete path is like this:</p><pre>C: \Users \swayam.mittal\AppData \Roaming \nltk _data \corpora</pre><p>Here you have all available corpora, including books, chats, movie reviews and more.</p><p>We will now discuss accessing these documents through NLTK. As you can see, these are primarily text documents, so you can open and read documents using plain Python code. That said, the NLTK module has some nice ways to handle the corpus, so you may find that the way to use them is practical. Here’s an example of how we opened the Gutenberg Bible and read the first few lines:</p><pre>From nltk.tokenize import sent_tokenize, PunktSentenceTokenizer<br>From nltk.corpus import gutenberg</pre><pre># sample text<br>Sample = gutenberg.raw( &quot;bible-kjv.txt&quot; )</pre><pre>Tok = sent_tokenize(sample)</pre><pre>For x in range( 5 ):<br>    Print(tok[x])</pre><p>One of the more advanced data sets is wordnet . Wordnet is a collection of words, definitions, examples of their use, synonyms, antonyms, and so on. Next we will use wordnet in depth.</p><h3>X. NLTK and Wordnet</h3><p>WordNet is an English vocabulary database created by Princeton and part of the NLTK corpus.</p><p>You can use WordNet and NLTK modules together to find word meanings, synonyms, antonyms, and more. Let’s introduce some examples.</p><p>First, you will need to import wordnet :</p><pre>From nltk.corpus import wordnet</pre><p>Then we plan to use the word program to find synonyms:</p><pre>Syns = wordnet.synsets( &quot;program&quot; )</pre><p>An example of a synonym:</p><pre>Print (syns[ 0 ].name())</pre><pre># plan.n.01</pre><p>Just the word:</p><pre>Print(syns[ 0 ] .lemmas ()[ 0 ] .name ())</pre><pre># plan</pre><p>The definition of the first synonym:</p><pre>Print(syns[ 0 ].definition())</pre><pre># a series of steps to be carried out or goals to be accomplished</pre><p>Example of the use of words:</p><pre>Print(syns[ 0 ].examples())</pre><pre># [ &#39;they drew up a six-step plan&#39;, &#39;they discussed plans for a new bond issue&#39;]</pre><p>Next, how do we identify synonyms and antonyms of a word? These forms are synonymous, and then you can use .antonyms find the antonym of the form. So we can populate some lists like:</p><pre>Synonyms = []<br>Antonyms = []</pre><pre>For syn in wordnet.synsets( &quot;good&quot; ):<br>    For l in syn.lemmas():<br>        Synonyms.append(l.name())<br>        If l.antonyms():<br>            Antonyms.append(l.antonyms()[ 0 ].name())</pre><pre> Print( set (synonyms))<br> Print( set (antonyms))</pre><pre> &#39;&#39; &#39; {&#39; beneficial &#39;, &#39; just &#39;, &#39; upright &#39;, &#39; thoroughly &#39;, &#39; in _force &#39;, &#39; well &#39;, &#39; skilful &#39;, &#39; skillful &#39;, &#39; sound &#39;, &#39; unspoiled &#39;, &#39; expert &#39;, &#39; Proficient &#39;, &#39; in _effect &#39;, &#39; honorable &#39;, &#39; adept &#39;, &#39; secure &#39;, &#39; commodity &#39;, &#39; estimable &#39;, &#39; soundly &#39;, &#39; right &#39;, &#39; respectable &#39;, &#39; good &#39;, &#39; serious &#39;, &#39; ripe &#39;, &#39; salutary &#39;, &#39; dear &#39;, &#39; practiced &#39;, &#39; goodness &#39;, &#39; safe &#39;, &#39; effective &#39;, &#39; unspoilt &#39;, &#39; dependable &#39;, &#39; undecomposed &#39;, &#39; honest &#39;, &#39; full &#39;, &#39; near &#39;, &#39; trade_good &#39;} {&#39; evil &#39;, &#39; evilness &#39;, &#39; bad &#39;, &#39; badness &#39;, &#39; ill &#39;} &#39; &#39;&#39;</pre><p>As you can see, our synonym is more than an antonym, because we only look up the antonym of the first form, but you can easily balance this by performing the exact same process for the word bad .</p><p>Next, we can easily use WordNet to compare the similarities of two words and their tenses, and combine the Wu and Palmer methods for semantic relevance.</p><p>Let’s compare nouns shipand boat:</p><pre>W1 = wordnet.synset( &#39;ship.n.01&#39; )<br>W2 = wordnet.synset( &#39;boat.n.01&#39; )<br>print (w1.wup_similarity(w2))</pre><pre># 0.9090909090909091</pre><pre>W1 = wordnet.synset( &#39;ship.n.01&#39; )<br>W2 = wordnet.synset( &#39;car.n.01&#39; )<br>print (w1.wup_similarity(w2))</pre><pre># 0.6956521739130435</pre><pre>W1 = wordnet.synset( &#39;ship.n.01&#39; )<br>W2 = wordnet.synset( &#39;cat.n.01&#39; )<br>print (w1.wup_similarity(w2))</pre><pre># 0.38095238095238093</pre><p>Next, we will discuss some issues and start discussing the topic of text categorization.</p><h3>XI. NLTK text classification</h3><p>Now that we are familiar with NLTK, let’s try to deal with text categorization. The goal of text categorization can be quite broad. Maybe we are trying to classify the text as political or military. Maybe we try to classify by the gender of the author. A fairly popular text categorization task is to identify the body of the text as spam or non-spam, such as an email filter. In our case, we will try to create a sentiment analysis algorithm.</p><p>To do this, we first try to use a movie review database that belongs to the NLTK corpus. From there, we will try to use vocabulary as a “feature”, which is part of a “positive” or “negative” movie review. The NLTK corpus movie_reviewsdataset has comments and they are marked as positive or negative. This means we can train and test this data. First, let&#39;s preprocess our data.</p><pre>Import nltk<br>import random<br>from nltk.corpus import movie_reviews</pre><pre>Documents = [(list(movie_reviews.words(fileid)), category)<br>             For category in movie_reviews.categories()<br>              for fileid in movie_reviews.fileids(category)]</pre><pre>Random.shuffle(documents)</pre><pre>Print(documents[ 1 ])</pre><pre>All_words = []<br>For w in movie_reviews.words():<br>    All_words.append(w.lower())</pre><pre>All_words = nltk.FreqDist(all_words)<br>Print(all_words.most_common( 15 ))<br>Print(all_words[ &quot;stupid&quot; ])</pre><p>It may take some time to run this script because the movie review dataset is a bit large. Let us introduce what happened here.</p><p>After importing the data set we want, you will see:</p><pre>Documents = [( list (movie_reviews. words (fileid)), category)<br>              for category in movie_reviews.categories()<br>              for fileid in movie_reviews.fileids(category)]</pre><p>Basically, in simple English, the above code is translated into: in each category (we have forward and exclusive), select all file IDs (each comment has its own ID), then store the word_tokenizedversion for the file ID (word list) followed by a positive or negative label in a large list.</p><p>Next, we use randomto disrupt our files. This is because we are going to train and test. If we order them in order, we may train all negative comments, and some positive comments, and then test on all positive comments. We don&#39;t want this, so we messed up the data.</p><p>Then, in order for you to see the data you are using, we print out documents[1]that this is a large list where the first element is a list of words and the second element is posor neglabel.</p><p>Next, we want to collect all the words we found, so we can have a huge list of typical words. From here, we can perform a frequency distribution and then find the most common words. As you can see, the most popular “words” are actually punctuation, the, aetc., but soon we will get valid vocabulary. We are going to store thousands of the most popular words, so this shouldn&#39;t be a problem.</p><pre>Print(all_words. most_common( 15 ) )</pre><p>The 15 most commonly used words are given above. You can also find out the number of occurrences of a word by following the steps below:</p><pre>Print  (all_words[ &quot;stupid&quot; ])</pre><p>Next, we begin to store our words as features of positive or negative movie reviews.</p><h3>XII. use NLTK to convert words into features</h3><p>In this tutorial, we build on previous videos and compile a list of features for words in positive and negative comments to see trends in specific types of words in positive or negative comments.</p><p>Initially, our code:</p><pre>Import nltk<br>Import random <br>from nltk.corpus import movie_reviews</pre><pre>Documents = [(list(movie_reviews. words (fileid)), category)<br>              for category in movie_reviews.categories()<br>              for fileid in movie_reviews.fileids(category)]</pre><pre>Random .shuffle(documents)</pre><pre>All_words = []</pre><pre>For w in movie_reviews. words ():<br>    All_words.append(w. lower ())</pre><pre>All_words = nltk.FreqDist(all_words)</pre><pre>Word_features = list(all_words. keys ())[: 3000 ]</pre><p>Almost as before, just now there is a new variable that word_featurescontains the top 3000 most commonly used words. Next, we&#39;ll build a simple function that finds these top 3000 words in our positive and negative documents, marking their presence as yes or no:</p><pre>Def  find_features  (document) :<br>    Words = set(document)<br>    Features = {}<br>    For w in word_features:<br>        Features[w] = (w in words)</pre><pre>    Return features</pre><p>Below, we can print out the feature set:</p><pre>Print(( find_features(movie_reviews. words( &#39;neg/cv000_29416.txt&#39; ) ) ) ) )</pre><p>We can then do this for all of our documentation by saving the feature boolean values ​​and their respective positive or negative categories by doing the following:</p><pre>Featuresets = [(find_features(rev), category) for (rev, category) in documents]</pre><p>Awesome, now we have features and tags, what’s next? Usually, the next step is to continue and train the algorithm and then test it. So let’s continue to do this, starting with the naive Bayes classifier in the next tutorial!</p><h3>XIII. NLTK Naive Bayes Classifier</h3><p>It’s time to choose an algorithm that divides our data into training and test sets and then starts! The algorithm we first use is the naive Bayes classifier. This is a very popular text categorization algorithm, so we can only try it first. However, before we can train and test our algorithm, we need to first break the data into a training set and a test set.</p><p>You can train and test the same data set, but this will give you some serious deviations, so you shouldn’t train and test the exact same data. For this reason, since we have disrupted the data set, we will first use 1900 out-of-order comments with positive and negative comments as the training set. Then we can test on the last 100 to see how accurate we are.</p><p>This is called supervised machine learning because we are presenting data to the machine and telling it “this data is positive” or “this data is negative.” Then, after the training is complete, we show the machine some new data and ask the computer what we think of the new data based on what we have taught before.</p><p>We can split the data in the following ways:</p><pre># set that we&#39;ll train our classifier with <br>training_ set = featuresets[: 1900 ]</pre><pre># set that we&#39;ll test against. <br>testing_ set = featuresets[ 1900 :]</pre><p>Below, we can define and train our classifiers:</p><pre>Classifier = nltk .NaiveBayesClassifier  .train (training_set)</pre><p>First, we simply call the naive Bayes classifier and use it in a row .train()for training.</p><p>It’s simple enough, now it’s trained. Next, we can test it:</p><pre>Print ( &quot;Classifier accuracy percent:&quot; ,(nltk.classify.accuracy(classifier, testing_set)) *100 )</pre><p>Hey, you got your answer. If you missed it, the reason we can “test” the data is that we still have the right answer. So, in the test, we present the data to the computer without providing the correct answer. If it correctly guesses what we know, then the computer is correct. Considering the disruption we’ve done, you and I may be different in accuracy, but you should see an average of 60–75% accuracy.</p><p>Next, we can learn more about the most valuable words in positive or negative comments:</p><pre>classifier the .Show _most_informative_features ( 15 )</pre><p>This is different for everyone, but you should see something like this:</p><pre>Most Informative Features<br>Insulting = True neg : pos = 10.6 : 1.0 <br>ludicrous = True neg : pos = 10.1 : 1.0 <br>winslet = True pos : neg = 9.0 : 1.0 <br>detract = True pos : neg = 8.4 : 1.0 <br>breathtaking = True pos : neg = 8.1 : 1.0 <br>Silverstone = True neg : pos = 7.6 : 1.0 <br>excruciatingly = True neg : pos = 7.6 : 1.0 <br>warns =True pos : neg = 7.0 : 1.0 <br>tracy = True pos : neg = 7.0 : 1.0 <br>insipid = True neg : pos = 7.0 : 1.0 <br>freddie = True neg : pos = 7.0 : 1.0 <br>damon = True pos : neg = 5.9 : 1.0 <br>debate = True pos : neg = 5.9 : 1.0 <br>ordered = True pos : neg = 5.8 : 1.0 <br>lang = True pos : neg = 5.7: 1.0</pre><p>What this tells you is that each word’s negative to positive appearance, or vice versa. So here we can see that the insultingword in the negative comment appears 10.6 times more than the positive comment. LudicrousIt is 10.1.</p><p>Now let’s assume that you are completely satisfied with your results, that you want to continue, and perhaps use this classifier to predict what is going on. It is very impractical to train the classifier and retrain it whenever you need to use the classifier. Therefore, you can use the picklemodule to save the classifier. We will do it next.</p><h3>XIV. Save the classifier with NLTK</h3><p>Training classifiers and machine learning algorithms can take a long time, especially if you train on a larger data set. Ours is actually very small. Can you imagine that you have to train the classifier every time you want to start using the classifier? So horrible! Instead, we can use the picklemodule and serialize our classifier object so that all we have to do is simply load the file.</p><p>So what should we do? The first step is to save the object. To do this, you first need to import at the top of the script pickle, and then after .train()training with the classifier, you can call the following lines:</p><pre>Save_classifier = open ( &quot;naivebayes.pickle&quot; , &quot;wb&quot; )<br>Pickle. dump (classifier, save_classifier)<br>Save_classifier. close ()</pre><p>This opens a picklefile and is ready to write some data in bytes. Then we use pickle.dump()to dump the data. pickle.dump()The first argument is what you write, and the second argument is where you write it.</p><p>After that, we closed the file as we requested, which means that we now have a pickleserialized object in the script&#39;s directory!</p><p>Next, how do we start using this classifier? .pickleFiles are serialized objects, all we need to do now is to read them into memory, which is as simple as reading any other normal file. this way:</p><pre>Classifier_f = open ( &quot;naivebayes.pickle&quot; , &quot;rb&quot; )<br>Classifier = pickle. load (classifier_f)<br>Classifier_f. close ()</pre><p>Here we have performed a very similar process. We open the file to read the bytes. Then we use pickle.load()to load the file and save the data to the classifier variable. Then we close the file, that&#39;s it. We now have the same classifier object as before!</p><p>Now we can use this object, and whenever we want to use it for classification, we no longer need to train our classifier.</p><p>Although it’s all good, we may not be happy with the 60–75% accuracy we get. What about other classifiers? In fact, there are many classifiers, but we need the scikit-learn (sklearn) module. Fortunately, NLTK employees recognized the value of incorporating the sklearn module into NLTK, and they built a small API for us. This is what we will do in the next tutorial.</p><h3>XV. NLTK and Sklearn</h3><p>Now we have seen how easy it is to use a classifier, now we want to try more! The best module for Python is the Scikit-learn (sklearn) module.</p><p>If you want to learn more about the Scikit-learn module, I have some tutorials on Scikit-Learn machine learning.</p><p>Fortunately, for us, the people behind NLTK value the value of incorporating the sklearn module into the NLTK classifier approach. In this way, they created various SklearnClassifierAPIs. To use it, you just need to import it like this:</p><pre>From nltk.classify.scikitlearn import SklearnClassifier</pre><p>From here on, you can use any sklearnclassifier. For example, let&#39;s introduce more variants of the naive Bayesian algorithm:</p><pre>From sklearn.naive_bayes import MultinomialNB , BernoulliNB</pre><p>After how do you use them? As a result, this is very simple.</p><pre>MNB_classifier = SklearnClassifier(MultinomialNB())<br>MNB_classifier .train (training_set)<br>Print( &quot; MultinomialNB accuracy percent:&quot; ,nltk .classify  .accuracy (MNB_classifier, testing_set))</pre><pre>BNB_classifier = SklearnClassifier(BernoulliNB())<br>BNB_classifier .train (training_set)<br>Print( &quot;BernoulliNB accuracy percent:&quot; ,nltk .classify  .accuracy (BNB_classifier, testing_set))</pre><p>It’s that simple. Let us introduce more things:</p><pre>From sklearn.linear_model import LogisticRegression,SGDClassifier<br> from sklearn.svm import SVC, LinearSVC, NuSVC</pre><p>Now all our classifiers should look like this:</p><pre>Print( &quot;Original Naive Bayes Algo accuracy percent:&quot; , (nltk .classify  .accuracy (classifier, testing_set))* 100 )<br>classifier the .Show _most_informative_features ( 15 )</pre><pre>MNB_classifier = SklearnClassifier(MultinomialNB())<br>MNB_classifier .train (training_set)<br>Print( &quot;MNB_classifier accuracy percent:&quot; , (nltk .classify  .accuracy (MNB_classifier, testing_set))* 100 )</pre><pre>BernoulliNB_classifier = SklearnClassifier(BernoulliNB())<br>BernoulliNB_classifier .train (training_set)<br>Print( &quot;BernoulliNB_classifier accuracy percent:&quot; , (nltk .classify  .accuracy (BernoulliNB_classifier, testing_set))* 100 )</pre><pre>LogisticRegression_classifier = SklearnClassifier(LogisticRegression())<br>LogisticRegression_classifier .train (training_set)<br>Print( &quot;LogisticRegression_classifier accuracy percent:&quot; , (nltk .classify  .accuracy (LogisticRegression_classifier, testing_set))* 100 )</pre><pre>SGDClassifier_classifier = SklearnClassifier(SGDClassifier())<br>SGDClassifier_classifier .train (training_set)<br>Print( &quot;SGDClassifier_classifier accuracy percent:&quot; , (nltk .classify  .accuracy (SGDClassifier_classifier, testing_set))* 100 )</pre><pre>SVC_classifier = SklearnClassifier(SVC())<br>SVC_classifier .train (training_set)<br>Print( &quot;SVC_classifier accuracy percent:&quot; , (nltk .classify  .accuracy (SVC_classifier, testing_set))* 100 )</pre><pre>LinearSVC_classifier = SklearnClassifier(LinearSVC())<br>LinearSVC_classifier .train (training_set)<br>Print( &quot;LinearSVC_classifier accuracy percent:&quot; , (nltk .classify  .accuracy (LinearSVC_classifier, testing_set))* 100 )</pre><pre>NuSVC_classifier = SklearnClassifier(NuSVC())<br>NuSVC_classifier .train (training_set)<br>Print( &quot;NuSVC_classifier accuracy percent:&quot; , (nltk .classify  .accuracy (NuSVC_classifier, testing_set))* 100 )</pre><p>The result of running it should look like this:</p><pre>Original Naive Bayes Algo accuracy percent: 63.0 <br>Most Informative Features<br>Thematic = True pos : neg = 9.1 : 1.0 <br>secondly = True pos : neg = 8.5 : 1.0 <br>narrates = True pos : neg = 7.8 : 1.0 <br>rounded = True pos : neg = 7.1 : 1.0 <br>supreme = True pos : neg = 7.1 : 1.0 <br>Layered = True pos : neg = 7.1 : 1.0 <br>crappy = True Neg : pos = 6.9 : 1.0 <br>uplifting = True pos : neg = 6.2 : 1.0 <br>ugh = True neg : pos = 5.3 : 1.0 <br>mamet = True pos : neg = 5.1 : 1.0 <br>gaining = True pos : neg = 5.1 : 1.0 <br>wanda = True Neg : pos = 4.9 : 1.0 <br>onset = True neg : pos = 4.9 :1.0 <br>fantastic = True pos : neg = 4.5 : 1.0 <br>kentucky = True pos : neg = 4.4 : 1.0 <br>MNB_classifier accuracy percent: 66.0 <br>BernoulliNB_classifier accuracy percent: 72.0 <br>LogisticRegression_classifier accuracy percent: 64.0 <br>SGDClassifier_classifier accuracy percent: 61.0 <br>SVC_classifier accuracy percent: 45.0 <br>LinearSVC_classifier accuracy percent: 68.0 <br>NuSVC_classifier accuracy percent: 59.0</pre><p>So, we can see that SVC errors are more common than correct, so we should probably discard it. but? Next we can try to use all of these algorithms at once. An algorithmic algorithm! To do this, we can create another classifier and generate the results of the classifier based on the results of other algorithms. It’s a bit like a voting system, so we only need an odd number of algorithms. This is what we will discuss in the next tutorial.</p><h3>XVI. Using the NLTK combination algorithm</h3><p>Now we know how to use a bunch of algorithm classifiers, like a child on Candy Island, telling them that they can only choose one, and we may find it difficult to select only one classifier. The good news is that you don’t have to! The combined classifier algorithm is a commonly used technique, which is implemented by creating a voting system. Each algorithm has one vote and the most votes are selected.</p><p>To do this, we want our new classifier to work like a typical NLTK classifier and have all the methods. Quite simply, with object-oriented programming, we can ensure that we inherit from the NLTK classifier class. To do this we will import it:</p><pre>From nltk.classify import ClassifierI<br>from statistics import mode</pre><p>We also import mode(the majority) because this will be the way we choose the maximum count.</p><p>Now let’s build our classifier class:</p><pre>Class  VoteClassifier  (ClassifierI) : <br>    def  __init__  (self, *classifiers) : <br>        self._classifiers = classifiers</pre><p>We call our class VoteClassifier, we inherit NLTK ClassifierI. Next, we assign the list of classifiers passed to our class self._classifiers.</p><p>Next, we will continue to create our own classification method. We intend to call it .classifyso that we can call it later .classify, just like the traditional NLTK classifier.</p><pre>Def  classify  (self, features) :<br>        Votes = []<br>        For c in self._classifiers:<br>            v = c.classify(features)<br>            Votes.append(v)<br>        Return mode(votes)</pre><p>Quite simply, what we are doing here is to iterate through our list of classifier objects. Then, for each one, we ask it to be based on feature classification. Classification is considered a vote. After the traversal is complete, we return mode(votes), this is just the mode to return the vote.</p><p>This is what we really need, but I think another parameter, confidence is useful. Since we have a voting algorithm, we can also count the number of support and negative votes, and call it “confidence.” For example, the confidence of a 3/5 vote is weaker than a 5/5 vote. Therefore, we can literally return the voting ratio as a measure of confidence. This is our confidence method:</p><pre>Def  confidence  (self, features) :<br>        Votes = []<br>        For c in self._classifiers:<br>            v = c.classify(features)<br>            Votes.append(v)</pre><pre>        Choice_votes = votes.count(mode(votes))<br>        Conf = choice_votes / len(votes)<br>        Return conf</pre><p>Now let’s put things together:</p><pre>Import nltk<br>Import random<br>From nltk.corpus import movie_reviews<br>From nltk.classify.scikitlearn import SklearnClassifier<br>Import pickle</pre><pre>From sklearn.naive_bayes import MultinomialNB, BernoulliNB<br>From sklearn.linear_model import LogisticRegression, SGDClassifier<br>From sklearn.svm import SVC, LinearSVC, NuSVC</pre><pre>From nltk.classify import ClassifierI<br>From statistics import mode<br></pre><pre>Class VoteClassifier(ClassifierI):<br>Def __init__(self, *classifiers): <br>self._classifiers = classifiers</pre><pre>Def classify(self, features): <br>votes = [] <br>for c in self._classifiers: <br>v = c.classify(features) <br>votes.append(v) <br>return mode(votes)</pre><pre>Def confidence(self, features): <br>votes = [] <br>for c in self._classifiers: <br>v = c.classify(features) <br>votes.append(v)</pre><pre>Choice_votes = votes.count(mode(votes)) <br>conf = choice_votes / len(votes) <br>return conf</pre><pre>Documents = [(list(movie_reviews.words(fileid)), category)<br>For category in movie_reviews.categories() <br>for fileid in movie_reviews.fileids(category)]</pre><pre>Random.shuffle(documents)</pre><pre>All_words = []</pre><pre>For w in movie_reviews.words():<br> All_words.append(w.lower())</pre><pre>All _words = nltk.FreqDist(all_ words)</pre><pre>Word _features = list(all_ words.keys())[:3000]</pre><pre>Def find_features(document):<br>Words = set(document) <br>features = {} <br>for w in word_features: <br>features[w] = (w in words)</pre><pre> Return features</pre><pre>#print((find_features(movie_reviews.words(&#39;neg/cv000_29416.txt&#39;)))))</pre><pre>Featuresets = [(find_features(rev), category) for (rev, category) in documents]</pre><pre>Training_set = featuresets[:1900]<br>Testing_set = featuresets[1900:]</pre><pre>#classifier = nltk.NaiveBayesClassifier.train(training_set)</pre><pre>Classifier_f = open(&quot;naivebayes.pickle&quot;,&quot;rb&quot;)<br>Classifier = pickle.load(classifier_f)<br>Classifier_f.close()<br></pre><pre>Print(&quot;Original Naive Bayes Algo accuracy percent:&quot;, (nltk.classify.accuracy(classifier, testing_set))*100)<br>classifier.show _most_ informative_features (15)</pre><pre>MNB_classifier = SklearnClassifier(MultinomialNB())<br>MNB _classifier.train(training_ set)<br>Print(&quot;MNB _classifier accuracy percent:&quot;, (nltk.classify.accuracy(MNB_ classifier, testing_set))*100)</pre><pre>BernoulliNB_classifier = SklearnClassifier(BernoulliNB())<br>BernoulliNB _classifier.train(training_ set)<br>Print(&quot;BernoulliNB _classifier accuracy percent:&quot;, (nltk.classify.accuracy(BernoulliNB_ classifier, testing_set))*100)</pre><pre>LogisticRegression_classifier = SklearnClassifier(LogisticRegression())<br>LogisticRegression _classifier.train(training_ set)<br>Print(&quot;LogisticRegression _classifier accuracy percent:&quot;, (nltk.classify.accuracy(LogisticRegression_ classifier, testing_set))*100)</pre><pre>SGDClassifier_classifier = SklearnClassifier(SGDClassifier())<br>SGDClassifier _classifier.train(training_ set)<br>Print(&quot;SGDClassifier _classifier accuracy percent:&quot;, (nltk.classify.accuracy(SGDClassifier_ classifier, testing_set))*100)</pre><pre>##SVC_classifier = SklearnClassifier(SVC()) <br>##SVC_classifier.train(training_set) <br>##print(&quot;SVC_classifier accuracy percent:&quot;, (nltk.classify.accuracy(SVC_classifier, testing_set))*100)</pre><pre>LinearSVC_classifier = SklearnClassifier(LinearSVC())<br>LinearSVC _classifier.train(training_ set)<br>Print(&quot;LinearSVC _classifier accuracy percent:&quot;, (nltk.classify.accuracy(LinearSVC_ classifier, testing_set))*100)</pre><pre>NuSVC_classifier = SklearnClassifier(NuSVC())<br>NuSVC _classifier.train(training_ set)<br>Print(&quot;NuSVC _classifier accuracy percent:&quot;, (nltk.classify.accuracy(NuSVC_ classifier, testing_set))*100)<br></pre><pre>Voted_classifier = VoteClassifier(classifier,<br>NuSVC_classifier, <br>LinearSVC_classifier, <br>SGDClassifier_classifier, <br>MNB_classifier, <br>BernoulliNB_classifier, <br>LogisticRegression_classifier)</pre><pre>Print(&quot;voted _classifier accuracy percent:&quot;, (nltk.classify.accuracy(voted_ classifier, testing_set))*100)</pre><pre>Print(&quot;Classification:&quot;, voted _classifier.classify(testing_ set[ 0 ][ 0 ]), &quot;Confidence %:&quot;,voted _classifier.confidence(testing_ set[ 0 ][ 0 ])*100)<br>Print(&quot;Classification:&quot;, voted _classifier.classify(testing_ set[ 1 ][ 0 ]), &quot;Confidence %:&quot;,voted _classifier.confidence(testing_ set[ 1 ][ 0 ])*100)<br>Print(&quot;Classification:&quot;, voted _classifier.classify(testing_ set[ 2 ][ 0 ]), &quot;Confidence %:&quot;,voted _classifier.confidence(testing_ set[ 2 ][ 0 ])*100)<br>Print(&quot;Classification:&quot;, voted _classifier.classify(testing_ set[ 3 ][ 0 ]), &quot;Confidence %:&quot;,voted _classifier.confidence(testing_ set[ 3 ][ 0 ])*100)<br>Print(&quot;Classification:&quot;, voted _classifier.classify(testing_ set[ 4 ][ 0 ]), &quot;Confidence %:&quot;,voted _classifier.confidence(testing_ set[ 4 ][ 0 ])*100)<br>Print(&quot;Classification:&quot;, voted _classifier.classify(testing_ set[ 5 ][ 0 ]), &quot;Confidence %:&quot;,voted _classifier.confidence(testing_ set[ 5 ][ 0 ])*100)</pre><p>So at the end, we run some sorter examples for the text. All our output:</p><pre>Original Naive Bayes Algo accuracy percent: 66.0<br>Most Informative Features<br>                Thematic = True pos : neg = 9.1 : 1.0 <br>                secondly = True pos : neg = 8.5 : 1.0 <br>                narrates = True pos : neg = 7.8 : 1.0 <br>                 layered = True pos : neg = 7.1 : 1.0 <br>                 rounded = True pos : neg = 7.1 : 1.0 <br>                 Supreme = True pos : neg = 7.1 : 1.0 <br>                  crappy = True neg: pos = 6.9 : 1.0 <br>               uplifting = True pos : neg = 6.2 : 1.0 <br>                     ugh = True neg : pos = 5.3 : 1.0 <br>                 gaining = True pos : neg = 5.1 : 1.0 <br>                   mamet = True pos : neg = 5.1 : 1.0 <br>                   wanda = True neg : pos = 4.9 : 1.0 <br>                   onset = True neg : pos = 4.9 :1.0 <br>               fantastic = True pos : neg = 4.5 : 1.0 <br>                   milos = True pos : neg = 4.4 : 1.0 <br>MNB_classifier accuracy percent: 67.0 <br>BernoulliNB_classifier accuracy percent: 67.0 <br>LogisticRegression_classifier accuracy percent: 68.0 <br>SGDClassifier_classifier accuracy percent: 57.99999999999999 <br>LinearSVC_classifier accuracy percent: 67.0 <br>NuSVC_classifier accuracy percent: 65.0 <br>voted_classifier accuracy percent: 65.0 <br>Classification:  neg Confidence %:100.0 <br>Classification: pos Confidence %: 57.14285714285714 <br>Classification:  neg Confidence %: 57.14285714285714 <br>Classification:  neg Confidence %: 57.14285714285714 <br>Classification: pos Confidence %: 57.14285714285714 <br>Classification: pos Confidence %: 85.71428571428571</pre><h3>XVII. Investigate bias using NLTK</h3><p>In this tutorial we will discuss some issues. The main problem is that we have a fairly biased algorithm. You can test it by commenting out the scramble of the document, then training with the first 1900 and leaving the last 100 (all positive) comments. Test it and you will find that your accuracy is very poor.</p><p>Instead, you can test with the top 100 data, all of which are negative and use 1900 training. Here you will find that the accuracy is very high. This is a bad sign. This can mean a lot of things, and we have a lot of options to solve it.</p><p>In other words, the project we are considering suggests that we continue and use different data sets, so we will do so. Finally, we will find that there is still some deviation in this new data set, that is, it chooses negative things more often. The reason is that the negatives of negative reviews tend to be more positive than positive ones. This can be done with some simple weighting, but it can also be complicated. Maybe it’s another day’s tutorial. Now we are going to grab a new data set, which we will discuss in the next tutorial.</p><h3>XVIII. Using NLTK to improve training data for sentiment analysis</h3><p>So now is the time to train on the new data set. Our goal is to analyze Twitter’s sentiment, so we want every positive and negative statement in the dataset to be short. It happens that I have 5300+ positive and 5300+ negative movie reviews, which is a much shorter data set. We should be able to get more accuracy from a larger training set and fit Twitter tweets better.</p><p>I hosted these two files here, you can find them by downloading a short comment . Save these files as positive.txtand negative.txt.</p><p>Now we can build new data sets as before. What needs to be changed?</p><p>We need a new way to create our “document” variable, and then we need a new way to create the all_wordsvariable. Really no problem, I did this:</p><pre>Short_pos = open ( &quot;short_reviews/positive.txt&quot; , &quot;r&quot; ). read ()<br>Short_neg = open ( &quot;short_reviews/negative.txt&quot; , &quot;r&quot; ). read ()</pre><pre>Documents = []</pre><pre>For r in short_pos. split ( &#39;\n&#39; ):<br>    Documents.append( (r, &quot;pos&quot; ) )</pre><pre>For r in short_neg. split ( &#39;\n&#39; ):<br>    Documents.append( (r, &quot;neg&quot; ) )<br></pre><pre>All_words = []</pre><pre>Short_pos_words = word_tokenize(short_pos)<br>Short_neg_words = word_tokenize(short_neg)</pre><pre>For w in short_pos_words:<br>    All_words.append(w. lower ())</pre><pre>For w in short_neg_words:<br>    All_words.append(w. lower ())</pre><pre>All_words = nltk.FreqDist(all_words)</pre><p>Next, we also need to adjust our feature lookup function, mainly based on the words in the document, because our new sample has no beautiful .words()features. I continued and added the most common words:</p><pre>Word_features = list(all_words.keys())[: 5000 ]</pre><pre>Def  find_features  (document) :<br>    Words = word_tokenize(document)<br>    Features = {}<br>    For w in word_features:<br>        Features[w] = (w in words)</pre><pre>    Return features</pre><pre>Featuresets = [(find_features(rev), category) for (rev, category) in documents]<br>Random.shuffle(featuresets)</pre><p>Other than that, the rest are the same. This is a complete script just in case you or I missed something:</p><p>This process takes a while. You may want to do something else. It took me about 30–40 minutes to get it all running, and I ran it on the i7 3930k. At the time of writing this article (2015), a general processor might take several hours. But this is a one-off process.</p><pre>Import nltk<br>import random<br>from nltk.corpus import movie_reviews<br>from nltk.classify.scikitlearn import SklearnClassifier<br>import pickle</pre><pre>From sklearn.naive_bayes import MultinomialNB , BernoulliNB<br>from sklearn.linear_model import LogisticRegression, SGDClassifier<br>from sklearn.svm import SVC, LinearSVC, NuSVC</pre><pre>From nltk.classify import ClassifierI<br>from statistics import mode</pre><pre>From nltk.tokenize import word_tokenize<br></pre><pre>Class  VoteClassifier  (ClassifierI) : <br>    def  __init__  (self, *classifiers) :<br>        Self._classifiers = classifiers</pre><pre>    Def  classify  (self, features) :<br>        Votes = []<br>        For c in self._classifiers:<br>            v = c.classify(features)<br>            Votes.append(v)<br>        Return mode(votes)</pre><pre>    Def  confidence  (self, features) :<br>        Votes = []<br>        For c in self._classifiers:<br>            v = c.classify(features)<br>            Votes.append(v)</pre><pre>        Choice_votes = votes.count(mode(votes))<br>        Conf = choice_votes / len(votes)<br>        Return conf</pre><pre>Short_pos = open( &quot;short_reviews/positive.txt&quot; , &quot;r&quot; ).read()<br>Short_neg = open( &quot;short_reviews/negative.txt&quot; , &quot;r&quot; ).read()</pre><pre>Documents = []</pre><pre>For r in short_pos.split( &#39;\n&#39; ):<br>    Documents.append( (r, &quot;pos&quot; ) )</pre><pre>For r in short_neg.split( &#39;\n&#39; ):<br>    Documents.append( (r, &quot;neg&quot; ) )<br></pre><pre>All_words = []</pre><pre>Short_pos_words = word_tokenize(short_pos)<br>Short_neg_words = word_tokenize(short_neg)</pre><pre>For w in short_pos_words:<br>    All_words.append(w.lower())</pre><pre>For w in short_neg_words:<br>    All_words.append(w.lower())</pre><pre>All_words = nltk.FreqDist(all_words)</pre><pre>Word_features = list(all_words.keys())[: 5000 ]</pre><pre>Def  find_features  (document) :<br>    Words = word_tokenize(document)<br>    Features = {}<br>    For w in word_features:<br>        Features[w] = (w in words)</pre><pre>    Return features</pre><pre>#print((find_features(movie_reviews.words(&#39;neg/cv000_29416.txt&#39;)))))</pre><pre>Featuresets = [(find_features(rev), category) for (rev, category) in documents]</pre><pre>Random.shuffle(featuresets)</pre><pre># positive data example: <br>training_set = featuresets[: 10000 ]<br>Testing_set = featuresets[ 10000 :]</pre><pre>## <br>### negative data example: <br>##training_set = featuresets[100:] <br>##testing_set = featuresets[:100]<br></pre><pre>Classifier = nltk.NaiveBayesClassifier.train(training_set)<br>Print( &quot;Original Naive Bayes Algo accuracy percent:&quot; , (nltk.classify.accuracy(classifier, testing_set))* 100 )<br>Classifier.show_most_informative_features( 15 )</pre><pre>MNB_classifier = SklearnClassifier(MultinomialNB())<br>MNB_classifier.train(training_set)<br>Print( &quot;MNB_classifier accuracy percent:&quot; , (nltk.classify.accuracy(MNB_classifier, testing_set))* 100 )</pre><pre>BernoulliNB_classifier = SklearnClassifier(BernoulliNB())<br>BernoulliNB_classifier.train(training_set)<br>Print( &quot;BernoulliNB_classifier accuracy percent:&quot; , (nltk.classify.accuracy(BernoulliNB_classifier, testing_set))* 100 )</pre><pre>LogisticRegression_classifier = SklearnClassifier(LogisticRegression())<br>LogisticRegression_classifier.train(training_set)<br>Print( &quot;LogisticRegression_classifier accuracy percent:&quot; , (nltk.classify.accuracy(LogisticRegression_classifier, testing_set))* 100 )</pre><pre>SGDClassifier_classifier = SklearnClassifier(SGDClassifier())<br>SGDClassifier_classifier.train(training_set)<br>Print( &quot;SGDClassifier_classifier accuracy percent:&quot; , (nltk.classify.accuracy(SGDClassifier_classifier, testing_set))* 100 )</pre><pre>##SVC_classifier = SklearnClassifier(SVC()) <br>##SVC_classifier.train(training_set) <br>##print(&quot;SVC_classifier accuracy percent:&quot;, (nltk.classify.accuracy(SVC_classifier, testing_set))*100)</pre><pre>LinearSVC_classifier = SklearnClassifier(LinearSVC())<br>LinearSVC_classifier.train(training_set)<br>Print( &quot;LinearSVC_classifier accuracy percent:&quot; , (nltk.classify.accuracy(LinearSVC_classifier, testing_set))* 100 )</pre><pre>NuSVC_classifier = SklearnClassifier(NuSVC())<br>NuSVC_classifier.train(training_set)<br>Print( &quot;NuSVC_classifier accuracy percent:&quot; , (nltk.classify.accuracy(NuSVC_classifier, testing_set))* 100 )<br></pre><pre>Voted_classifier = VoteClassifier(<br>                                  NuSVC_classifier,<br>                                  LinearSVC_classifier,<br>                                  MNB_classifier,<br>                                  BernoulliNB_classifier,<br>                                  LogisticRegression_classifier)</pre><pre>Print( &quot;voted_classifier accuracy percent:&quot; , (nltk.classify.accuracy(voted_classifier, testing_set))* 100 )</pre><p>Output:</p><pre>Original Naive Bayes Algo accuracy percent: 66.26506024096386 <br>Most Informative Features<br>              Refreshing = True pos : neg = 13.6 : 1.0 <br>                captures = True pos : neg = 11.3 : 1.0 <br>                  stupid = True neg : pos = 10.7 : 1.0 <br>                  tender = True pos : neg = 9.6 : 1.0 <br>              meandering = True neg : pos = 9.1 : 1.0 <br>                      Tv = True neg : pos = 8.6 : 1.0 <br>                 low-key = TruePos : neg = 8.3 : 1.0 <br>              thoughtful = True pos : neg = 8.1 : 1.0 <br>                   banal = True neg : pos = 7.7 : 1.0 <br>              amateurish = True neg : pos = 7.7 : 1.0 <br>                terrific = True pos : neg = 7.6 : 1.0 <br>                  record = True Pos : neg = 7.6 : 1.0 <br>             captivating = True pos : neg = 7.6 :1.0 <br>                portrait = True pos : neg = 7.4 : 1.0 <br>                 culture = True pos : neg = 7.3 : 1.0 <br>MNB_classifier accuracy percent: 65.8132530120482 <br>BernoulliNB_classifier accuracy percent: 66.71686746987952 <br>LogisticRegression_classifier accuracy percent: 67.16867469879519 <br>SGDClassifier_classifier accuracy percent: 65.8132530120482 <br>LinearSVC_classifier accuracy percent: 66.71686746987952 <br>NuSVC_classifier accuracy percent: 60.09036144578314 <br>voted_classifier accuracy percent: 65.66265060240963</pre><p>Yes, I bet you spent a while, so in the next tutorial we will talk about pickleeverything!</p><h3>XIX. Use NLTK to create modules for sentiment analysis</h3><p>With this new data set and new classifiers, we can move on. As you may have noticed, this new data set takes longer to train because it is a larger collection. I have shown you that by pickelserializing or serializing the trained classifiers, we can actually save a lot of time. These classifiers are just objects.</p><p>I have shown you how to use it pickelto implement it, so I encourage you to try it yourself. If you need help, I will paste the complete code... but be careful, do it yourself!</p><p>This process takes a while. You may want to do something else. It took me about 30–40 minutes to get it all running, and I ran it on the i7 3930k. At the time of writing this article (2015), a general processor might take several hours. But this is a one-off process.</p><pre>Import nltk<br> import random<br> #from nltk.corpus import movie_reviews <br>from nltk.classify.scikitlearn import SklearnClassifier<br> import pickle<br> from sklearn.naive_bayes import MultinomialNB , BernoulliNB<br> from sklearn.linear_model import LogisticRegression, SGDClassifier<br> from sklearn.svm import SVC, LinearSVC, NuSVC<br> from nltk. Classify import ClassifierI<br> from statistics import mode<br> from nltk.tokenize import Word_tokenize</pre><pre>Class  VoteClassifier  (ClassifierI) : <br>    def  __init__  (self, *classifiers) :<br>        Self._classifiers = classifiers</pre><pre>    Def  classify  (self, features) :<br>        Votes = []<br>        For c in self._classifiers:<br>            v = c.classify(features)<br>            Votes.append(v)<br>        Return mode(votes)</pre><pre>    Def  confidence  (self, features) :<br>        Votes = []<br>        For c in self._classifiers:<br>            v = c.classify(features)<br>            Votes.append(v)</pre><pre>        Choice_votes = votes.count(mode(votes))<br>        Conf = choice_votes / len(votes)<br>        Return conf</pre><pre>Short_pos = open( &quot;short_reviews/positive.txt&quot; , &quot;r&quot; ).read()<br>Short_neg = open( &quot;short_reviews/negative.txt&quot; , &quot;r&quot; ).read()</pre><pre># move this up here<br>All_words = []<br>Documents = []<br></pre><pre># j is adject, r is adverb, and v is verb <br>#allowed_word_types = [&quot;J&quot;,&quot;R&quot;,&quot;V&quot;] <br>allowed_word_types = [ &quot;J&quot; ]</pre><pre>For p in short_pos.split( &#39;\n&#39; ):<br>    Documents.append( (p, &quot;pos&quot; ) )<br>    Words = word_tokenize(p)<br>    Pos = nltk.pos_tag(words)<br>    For w in pos:<br>         if w[ 1 ][ 0 ] in allowed_word_types:<br>            All_words.append(w[ 0 ].lower())<br></pre><pre>For p in short_neg.split( &#39;\n&#39; ):<br>    Documents.append( (p, &quot;neg&quot; ) )<br>    Words = word_tokenize(p)<br>    Pos = nltk.pos_tag(words)<br>    For w in pos:<br>         if w[ 1 ][ 0 ] in allowed_word_types:<br>            All_words.append(w[ 0 ].lower())<br></pre><pre>Save_documents <br>= open( &quot;pickled_algos/documents.pickle&quot; , &quot;wb&quot; )<br>Pickle.dump(documents, save_documents)<br>Save_documents.close()<br></pre><pre>All_words = nltk.FreqDist(all_words)<br></pre><pre>Word_features = list(all_words.keys())[: 5000 ]<br></pre><pre>Save_word_features = open( &quot;pickled_algos/word_features5k.pickle&quot; , &quot;wb&quot; )<br>Pickle.dump(word_features, save_word_features)<br>Save_word_features.close()<br></pre><pre>Def  find_features  (document) :<br>    Words = word_tokenize(document)<br>    Features = {}<br>    For w in word_features:<br>        Features[w] = (w in words)</pre><pre>    Return features</pre><pre>Featuresets = [(find_features(rev), category) for (rev, category) in documents]</pre><pre>Random.shuffle(featuresets)<br>Print(len(featuresets))</pre><pre>Testing_set = featuresets[ 10000 :]<br>Training_set = featuresets[: 10000 ]<br></pre><pre>Classifier = nltk.NaiveBayesClassifier.train(training_set)<br>Print( &quot;Original Naive Bayes Algo accuracy percent:&quot; , (nltk.classify.accuracy(classifier, testing_set))* 100 )<br>Classifier.show_most_informative_features( 15 )</pre><pre>############### <br>save_classifier = open( &quot;pickled_algos/originalnaivebayes5k.pickle&quot; , &quot;wb&quot; )<br>Pickle.dump(classifier, save_classifier)<br>Save_classifier.close()</pre><pre>MNB_classifier = SklearnClassifier(MultinomialNB())<br>MNB_classifier.train(training_set)<br>Print( &quot;MNB_classifier accuracy percent:&quot; , (nltk.classify.accuracy(MNB_classifier, testing_set))* 100 )</pre><pre>Save_classifier = open( &quot;pickled_algos/MNB_classifier5k.pickle&quot; , &quot;wb&quot; )<br>Pickle.dump(MNB_classifier, save_classifier)<br>Save_classifier.close()</pre><pre>BernoulliNB_classifier = SklearnClassifier(BernoulliNB())<br>BernoulliNB_classifier.train(training_set)<br>Print( &quot;BernoulliNB_classifier accuracy percent:&quot; , (nltk.classify.accuracy(BernoulliNB_classifier, testing_set))* 100 )</pre><pre>Save_classifier = open( &quot;pickled_algos/BernoulliNB_classifier5k.pickle&quot; , &quot;wb&quot; )<br>Pickle.dump(BernoulliNB_classifier, save_classifier)<br>Save_classifier.close()</pre><pre>LogisticRegression_classifier = SklearnClassifier(LogisticRegression())<br>LogisticRegression_classifier.train(training_set)<br>Print( &quot;LogisticRegression_classifier accuracy percent:&quot; , (nltk.classify.accuracy(LogisticRegression_classifier, testing_set))* 100 )</pre><pre>Save_classifier = open( &quot;pickled_algos/LogisticRegression_classifier5k.pickle&quot; , &quot;wb&quot; )<br>Pickle.dump(LogisticRegression_classifier, save_classifier)<br>Save_classifier.close()<br></pre><pre>LinearSVC_classifier = SklearnClassifier(LinearSVC())<br>LinearSVC_classifier.train(training_set)<br>Print( &quot;LinearSVC_classifier accuracy percent:&quot; , (nltk.classify.accuracy(LinearSVC_classifier, testing_set))* 100 )</pre><pre>Save_classifier = open( &quot;pickled_algos/LinearSVC_classifier5k.pickle&quot; , &quot;wb&quot; )<br>Pickle.dump(LinearSVC_classifier, save_classifier)<br>Save_classifier.close()<br></pre><pre>##NuSVC_classifier = SklearnClassifier(NuSVC()) <br>##NuSVC_classifier.train(training_set) <br>##print(&quot;NuSVC_classifier accuracy percent:&quot;, (nltk.classify.accuracy(NuSVC_classifier, testing_set))*100)<br></pre><pre>SGDC_classifier = SklearnClassifier(SGDClassifier())<br>SGDC_classifier.train(training_set)<br>Print( &quot;SGDClassifier accuracy percent:&quot; ,nltk.classify.accuracy(SGDC_classifier, testing_set)* 100 )</pre><pre>Save_classifier = open( &quot;pickled_algos/SGDC_classifier5k.pickle&quot; , &quot;wb&quot; )<br>Pickle.dump(SGDC_classifier, save_classifier)<br>Save_classifier.close()</pre><p>Now you only need to run it once. If you wish, you can run it anytime, but now you are ready to create a sentiment analysis module. This is what we call sentiment_mod.pya file:</p><pre>#File: sentiment_mod.py</pre><pre>Import nltk<br> import random<br> #from nltk.corpus import movie_reviews <br>from nltk.classify.scikitlearn import SklearnClassifier<br> import pickle<br> from sklearn.naive_bayes import MultinomialNB , BernoulliNB<br> from sklearn.linear_model import LogisticRegression, SGDClassifier<br> from sklearn.svm import SVC, LinearSVC, NuSVC<br> from nltk. Classify import ClassifierI<br> from statistics import mode<br> from nltk.tokenize import Word_tokenize</pre><pre>Class  VoteClassifier  (ClassifierI) : <br>    def  __init__  (self, *classifiers) :<br>        Self._classifiers = classifiers</pre><pre>    Def  classify  (self, features) :<br>        Votes = []<br>        For c in self._classifiers:<br>            v = c.classify(features)<br>            Votes.append(v)<br>        Return mode(votes)</pre><pre>    Def  confidence  (self, features) :<br>        Votes = []<br>        For c in self._classifiers:<br>            v = c.classify(features)<br>            Votes.append(v)</pre><pre>        Choice_votes = votes.count(mode(votes))<br>        Conf = choice_votes / len(votes)<br>        Return conf<br></pre><pre>Documents_f = open( &quot;pickled_algos/documents.pickle&quot; , &quot;rb&quot; )<br>Documents = pickle.load(documents_f)<br>Documents_f.close()<br></pre><pre>Word_features5k_f = open( &quot;pickled_algos/word_features5k.pickle&quot; , &quot;rb&quot; )<br>Word_features = pickle.load(word_features5k_f)<br>Word_features5k_f.close()<br></pre><pre>Def  find_features  (document) :<br>    Words = word_tokenize(document)<br>    Features = {}<br>    For w in word_features:<br>        Features[w] = (w in words)</pre><pre>    Return features<br></pre><pre>Featuresets_f <br>= open( &quot;pickled_algos/featuresets.pickle&quot; , &quot;rb&quot; )<br>Featuresets = pickle.load(featuresets_f)<br>Featuresets_f.close()</pre><pre>Random.shuffle(featuresets)<br>Print(len(featuresets))</pre><pre>Testing_set = featuresets[ 10000 :]<br>Training_set = featuresets[: 10000 ]<br></pre><pre>Open_file <br>= open( &quot;pickled_algos/originalnaivebayes5k.pickle&quot; , &quot;rb&quot; )<br>Classifier = pickle.load(open_file)<br>Open_file.close()</pre><pre>Open_file <br>= open( &quot;pickled_algos/MNB_classifier5k.pickle&quot; , &quot;rb&quot; )<br>MNB_classifier = pickle.load(open_file)<br>Open_file.close()<br></pre><pre>Open_file <br>= open( &quot;pickled_algos/BernoulliNB_classifier5k.pickle&quot; , &quot;rb&quot; )<br>BernoulliNB_classifier = pickle.load(open_file)<br>Open_file.close()</pre><pre>Open_file <br>= open( &quot;pickled_algos/LogisticRegression_classifier5k.pickle&quot; , &quot;rb&quot; )<br>LogisticRegression_classifier = pickle.load(open_file)<br>Open_file.close()</pre><pre>Open_file <br>= open( &quot;pickled_algos/LinearSVC_classifier5k.pickle&quot; , &quot;rb&quot; )<br>LinearSVC_classifier = pickle.load(open_file)<br>Open_file.close()</pre><pre>Open_file <br>= open( &quot;pickled_algos/SGDC_classifier5k.pickle&quot; , &quot;rb&quot; )<br>SGDC_classifier = pickle.load(open_file)<br>Open_file.close()<br></pre><pre>Voted_classifier = VoteClassifier(<br>                                  Classifier,<br>                                  LinearSVC_classifier,<br>                                  MNB_classifier,<br>                                  BernoulliNB_classifier,<br>                                  LogisticRegression_classifier)<br></pre><pre>Def  sentiment  (text) :<br>    Feats = find_features(text)<br>    Return voted_classifier.classify(feats),voted_classifier.confidence(feats)</pre><p>So here, in addition to the final function, there is nothing new, it is very simple. This function is the key to interacting with us from here. This function, which we call “emotion”, takes a parameter, text. Here, we use the find_featuresfunctions we have created to decompose these features. All we have to do now is to use our voting classifier to return the classification and return the confidence of the classification.</p><p>With this, we can now use this file, as well as the emotion function, as a module. The following is a sample script that uses this module:</p><pre>Import sentiment_mod as s</pre><pre>Print(s.sentiment( &quot;This movie was awesome! The acting was great, plot was wonderful, and there were pythons...so yea!&quot; ))<br>Print(s.sentiment( &quot;This movie was utter junk. There were absolutely 0 pythons. I don&#39;t see what the point was at all. Horrible movie, 0/10&quot; ))</pre><p>As expected, pythonthe comments with the movie are obviously good, and no pythonmovie is junk. Both have 100% confidence.</p><p>It took me about 5 seconds to import the module, because we saved the classifier and it might take 30 minutes without saving. Thanks to pickleyour time, there will be big differences depending on your processor. If you keep going, I will say that you may want to see it too joblib.</p><p>Now that we have this great module, it works very easily, what can we do? I suggest that we go to Twitter for real-time sentiment analysis!</p><h3>XX. NLTK Twitter sentiment analysis</h3><p>Now that we have an sentiment analysis module, we can apply it to any text, but it’s better to have short text, like Twitter! To this end, we will combine this tutorial with the Twitter Streaming API Tutorial.</p><p>The initial code for this tutorial is:</p><pre>From tweepy import Stream<br> from tweepy import OAuthHandler<br> from tweepy.streaming import StreamListener<br></pre><pre>#consumer key, consumer secret, access token, access secret. <br>ckey= &quot;fsdfasdfsafsffa&quot; <br>csecret= &quot;asdfsadfsadfsadf&quot; <br>atoken= &quot;asdf-aassdfs&quot; <br>asecret= &quot;asdfsadfsdafsdafs&quot;</pre><pre>Class  listener  (StreamListener) :</pre><pre>    Def  on_data  (self, data) :<br>        Print(data)<br>        Return ( True )</pre><pre>    Def  on_error  (self, status) : <br>        print status</pre><pre>Auth = OAuthHandler(ckey, csecret)<br>Auth.set_access_token(atoken, asecret)</pre><pre>twitterStream = Stream(auth, listener())<br>twitterStream.filter(track=[ &quot;car&quot; ])</pre><p>This is enough to print carall the data of a streaming live tweet containing words . We can use jsonmodules json.loads(data)to load data variables, and then we can reference specific ones tweet:</p><pre>Tweet = all_data[ &quot;text&quot; ]</pre><p>Since we have a tweet, we can easily pass it to our sentiment_modmodule.</p><pre>From tweepy import Stream<br> from tweepy import OAuthHandler<br> from tweepy.streaming import StreamListener<br> import json<br> import sentiment_mod as s</pre><pre>#consumer key, consumer secret, access token, access secret. <br>ckey= &quot;asdfsafsafsaf&quot; <br>csecret= &quot; <br>asdfasdfsadfsa &quot; atoken= &quot;asdfsadfsafsaf-asdfsaf&quot; <br>asecret= &quot;asdfsadfsadfsadfsadfsad&quot;</pre><pre>From twitterapistuff import *</pre><pre>Class  listener  (StreamListener) :</pre><pre>    Def  on_data  (self, data) :</pre><pre>        All_data = json.loads(data)</pre><pre>        Tweet = all_data[ &quot;text&quot; ]<br>        Sentiment_value, confidence = s.sentiment(tweet)<br>        Print(tweet, sentiment_value, confidence)</pre><pre>        If confidence* 100 &gt;= 80 :<br>            Output = open( &quot;twitter-out.txt&quot; , &quot;a&quot; )<br>            Output.write(sentiment_value)<br>            Output.write( &#39;\n&#39; )<br>            Output.close()</pre><pre>        Return  True</pre><pre>    Def  on_error  (self, status) :<br>        Print(status)</pre><pre>Auth = OAuthHandler(ckey, csecret)<br>Auth.set_access_token(atoken, asecret)</pre><pre>twitterStream = Stream(auth, listener())<br>twitterStream.filter(track=[ &quot;happy&quot; ])</pre><p>In addition to this, we also save the results to the output file twitter-out.txt.</p><p>Next, what is the data analysis without the chart is complete? Let’s combine another tutorial to draw a live streaming graph from sentiment analysis on the Twitter API.</p><h3>XXI. Using NLTK to draw Twitter real-time sentiment analysis</h3><p>Now that we have obtained real-time data from the Twitter Streaming API, why are there no activity diagrams showing emotional trends? To this end, we will combine this tutorial with the matplotlib drawing tutorial.</p><p>If you want to learn more about how the code works, see this tutorial. otherwise:</p><pre>Import matplotlib.pyplot as plt<br> import matplotlib.animation as animation<br> from matplotlib import style<br> import time</pre><pre>Style.use( &quot;ggplot&quot; )</pre><pre>Fig = plt.figure()<br>Ax1 = fig.add_subplot( 1 , 1 , 1 )</pre><pre>Def  animate  (i) : <br>    pullData = open( &quot;twitter-out.txt&quot; , &quot;r&quot; ).read()<br>    Lines = pullData.split( &#39;\n&#39; )</pre><pre>    Xar = []<br>    Yar = []</pre><pre>    x = 0 <br>    y = 0</pre><pre>    For l in lines[- 200 :]:<br>        x += 1 <br>        if  &quot;pos&quot;  in l:<br>            y += 1 <br>        elif  &quot;neg&quot;  in l:<br>            y -= 1</pre><pre>        Xar.append(x)<br>        Yar.append(y)</pre><pre>    Ax1.clear()<br>    Ax1.plot(xar,yar)<br>Ani = animation.FuncAnimation(fig, animate, interval= 1000 )<br>Plt.show()</pre><h3>XXII. Stanford NER marker and named entity recognition</h3><p>The Stanford NER marker provides an alternative to NLTK’s Named Entity Recognition (NER) classifier. This marker is largely seen as a standard for named entity recognition, but because it uses advanced statistical learning algorithms, its computational overhead is greater than the options offered by NLTK.</p><p>One of the big advantages of the Stanford NER marker is that we have several different models to extract named entities. We can use any of the following:</p><ul><li>Three types of models for identifying locations, people, and organizations</li><li>Four types of models for identifying locations, people, organizations, and miscellaneous entities</li><li>Seven types of models, identifying location, people, organization, time, money, percentage and date</li></ul><p>In order to continue, we need to download the model and jarfiles, because the NER classifier is written in Java. These are available free of charge from the Stanford Natural Language Processing Group . NTLK For the convenience of us, NLTK provides a wrapper for the Stanford marker, so we can use it in the best language (of course Python)!</p><p>StanfordNERTaggerThe parameters passed to the class include:</p><ul><li>The path of the classification model (the following three types of models are used)</li><li>jarPath to the Stanford marker file</li><li>Training data encoding (default is ASCII)</li></ul><p>Here’s how we set it up to mark sentences using three types of models:</p><pre># -*- coding: utf-8 -*-</pre><pre>From nltk.tag import StanfordNERTagger<br> from nltk.tokenize import word_tokenize</pre><pre>St = StanfordNERTagger( &#39;/usr/share/stanford-ner/classifiers/english.all.3class.distsim.crf.ser.gz&#39; ,<br>                        &#39;/usr/share/stanford-ner/stanford-ner.jar&#39; ,<br>                       Encoding= &#39;utf-8&#39; )</pre><pre>Text = &#39;While in France, Christine Lagarde discussed short-term stimulus efforts in a recent interview with the Wall Street Journal.&#39;</pre><pre>Tokenized_text = word_tokenize(text)<br>Classified_text = st.tag(tokenized_text)</pre><pre>Print(classified_text)</pre><p>Once we follow the word segmentation and classify the sentences, we will see that the marker produces the following list of tuples:</p><pre>[(&#39;While&#39;, &#39;O&#39;), (&#39;in&#39;, &#39;O&#39;), (&#39;France&#39;, &#39;LOCATION&#39;), (&#39;,&#39;, &#39;O&#39;), (&#39;Christine&#39;, &#39;PERSON&#39;) , (&#39;Lagarde&#39;, &#39;PERSON&#39;), (&#39;discussed&#39;, &#39;O&#39;), (&#39;short-term&#39;, &#39;O&#39;), (&#39;stimulus&#39;, &#39;O&#39;), (&#39;efforts&#39;, &#39;O &#39;), (&#39;in&#39;, &#39;O&#39;), (&#39;a&#39;, &#39;O&#39;), (&#39;recent&#39;, &#39;O&#39;), (&#39;interview&#39;, &#39;O&#39;), (&#39;with&#39;, &#39;O &#39;), (&#39;the&#39;, &#39;O&#39;), (&#39;Wall&#39;, &#39;ORGANIZATION&#39;), (&#39;Street&#39;, &#39;ORGANIZATION&#39;), (&#39;Journal&#39;, &#39;ORGANIZATION&#39;), (&#39;.&#39;, &#39;O &#39;)]</pre><p>Great!Each tag uses PERSON, LOCATION, ORGANIZATIONor Otags (using our three types of models). ORepresents only the other, unnamed entities.</p><p>This list can now be used to test the annotated data, which we will cover in the next tutorial.</p><h3>XXIII. Testing the accuracy of the NLTK and Stanford NER markers</h3><p>We know how to use two different NER classifiers! But which one should we choose, NLTK or Stanford? Let’s do some testing to find out the answer.</p><p>The first thing we need is some labeled reference data to test our NER classifier. One way to get this data is to find a large number of articles and mark each tag as a named entity (for example, people, organization, location) or other non-named entity. Then we can test our separate NER classifier with the correct label we know.</p><p>Unfortunately, this is very time consuming! The good news is that there is a manually annotated data set that is freely available with over 16,000 English sentences. There are also data sets in German, Spanish, French, Italian, Dutch, Polish, Portuguese and Russian!</p><p>This is an annotated sentence from the dataset:</p><pre>Founding O <br>member O <br>Kojima I -PER<br>Minoru I -PER<br>Played O <br>guitar O <br>on O <br>Good I -MISC<br>Day I -MISC<br>, O <br>and O <br>Wardanceis the I -misc<br>Cover O <br>of O <br>A O <br>Song O <br>by O <br>UK the I -LOC<br>Post O <br>punk O <br>industrial O <br>band O <br>Killing I -ORG<br>Joke I -ORG<br>. O</pre><p>Let’s read, split, and manipulate the data to make it a better format for testing.</p><pre>Import nltk<br>from nltk.tag import StanfordNERTagger<br>from nltk.metrics.scores import accuracy</pre><pre>Raw_annotations = open( &quot;/usr/share/wikigold.conll.txt&quot; ).read()<br>Split_annotations = raw_annotations.split()</pre><pre># Amend class annotations to reflect Stanford&#39;s NERTagger <br>for n,i in enumerate(split_annotations):<br>     if i == &quot;I-PER&quot; :<br>        Split_annotations[n] = &quot;PERSON&quot; <br>    if i == &quot;I-ORG&quot; :<br>        Split_annotations[n] = &quot;ORGANIZATION&quot; <br>    if i == &quot;I-LOC&quot; :<br>        Split_annotations[n] = &quot;LOCATION&quot;</pre><pre># Group NE data into tuples <br>def  group  (lst, n) : <br>  for i in range( 0 , len(lst), n):<br>    Val = lst[i:i+n]<br>    If len(val) == n:<br>       yield tuple(val)</pre><pre>Reference_annotations = list(group(split_annotations, 2 ))</pre><p>Ok, it looks good! However, we also need to paste the “clean” form of this data into our NER classifier. Let’s do it.</p><pre>Pure_tokens = split_annotations[::2]</pre><p>This reads in the data, splits it by whitespace, and then takes split_annotationsa subset of everything in increments of two (starting from the zeroth element) . This produces a data set similar to the (smaller) example below:</p><pre>[&#39;Founding&#39;, &#39;member&#39;, &#39;Kojima&#39;, &#39;Minoru&#39;, &#39;played&#39;, &#39;guitar&#39;, &#39;on&#39;, &#39;Good&#39;, &#39;Day&#39;, &#39;,&#39;, &#39;and&#39;, &#39;Wardanceis&#39;, &#39; Cover&#39;, &#39;of&#39;, &#39;a&#39;, &#39;song&#39;, &#39;by&#39;, &#39;UK&#39;, &#39;post&#39;, &#39;punk&#39;, &#39;industrial&#39;, &#39;band&#39;, &#39;Killing&#39;, &#39;Joke&#39;, &#39;.&#39; ]</pre><p>Let’s go ahead and test the NLTK classifier:</p><pre>Tagged_words = nltk.pos_tag(pure_tokens)<br>nltk_unformatted_prediction = nltk.ne_chunk(tagged_words)</pre><p>Since the NLTK NER classifier generates trees (including POS tags), we need to do some extra data manipulation to get the proper form for testing.</p><pre>#Convert prediction to multiline string and then to list (includes pos tags)<br>Multiline_string = nltk.chunk.tree2conllstr(nltk_unformatted_prediction)<br>Listed_pos_and_ne = multiline_string.split()</pre><pre># Delete pos tags and rename<br>Del listed_pos_and_ne[ 1 :: 3 ]<br>Listed_ne = listed_pos_and_ne</pre><pre># Amend class  annotations  for  consistency  with  reference_annotations<br> for n,i in enumerate(listed_ne):<br>     if i == &quot;B-PERSON&quot; :<br>        Listed_ne[n] = &quot;PERSON&quot; <br>    if i == &quot;I-PERSON&quot; :<br>        Listed_ne[n] = &quot;PERSON&quot;     <br>    if i == &quot;B-ORGANIZATION&quot; :<br>        Listed_ne[n] = &quot;ORGANIZATION&quot; <br>    if i == &quot;I-ORGANIZATION&quot; :<br>        Listed_ne[n] = &quot;ORGANIZATION&quot; <br>    if i == &quot;B-LOCATION&quot; :<br>        Listed_ne[n] = &quot;LOCATION&quot; <br>    if i == &quot;I-LOCATION&quot; :<br>        Listed_ne[n] = &quot;LOCATION&quot; <br>    if i == &quot;B-GPE&quot; :<br>        Listed_ne[n] = &quot;LOCATION&quot; <br>    if i == &quot;I-GPE&quot; :<br>        Listed_ne[n] = &quot;LOCATION&quot;</pre><pre># Group prediction into tuples<br>Nltk_formatted_prediction = list(group(listed_ne, 2 ))</pre><p>Now we can test the accuracy of NLTK.</p><pre>Nltk_accuracy = accuracy(reference_annotations, nltk_formatted_prediction) print(nltk_accuracy)</pre><p>Wow, the accuracy rate .8971!</p><p>Let us now test the Stanford classifier. Because this classifier generates output in tuples, testing does not require more data manipulation.</p><pre>St = StanfordNERTagger( &#39;/usr/share/stanford-ner/classifiers/english.all.3class.distsim.crf.ser.gz&#39; ,<br>                        &#39;/usr/share/stanford-ner/stanford-ner.jar&#39; ,<br>                       Encoding= &#39;utf-8&#39; )                  <br>Stanford_prediction = st.tag(pure_tokens)<br>Stanford_accuracy = accuracy(reference_annotations, stanford_prediction)<br>Print (stanford_accuracy)</pre><p>.9223The accuracy rate! better!</p><p>If you want to draw this, here are some extra code. If you want to learn more about how this works, check out the matplotlib series:</p><pre>Import numpy as np<br>Import matplotlib.pyplot as plt<br>from matplotlib import style</pre><pre>Style.use( &#39;fivethirtyeight&#39; )</pre><pre>N = 1 <br>ind = np.arange(N) # the x locations for the groups <br>width = 0.35  # the width of the bars</pre><pre>Fig, ax = plt.subplots()</pre><pre>Stanford_percentage = stanford_accuracy * 100 <br>rects1 = ax.bar(ind, stanford_percentage, width, color= &#39;r&#39; )</pre><pre>Nltk_percentage = nltk_accuracy * 100 <br>rects2 = ax.bar(ind+width, nltk_percentage, width, color= &#39;y&#39; )</pre><pre># add some text for labels, title and axes ticks <br>ax.set_xlabel( &#39;Classifier&#39; )<br>Ax.set_ylabel( &#39;Accuracy (by percentage)&#39; )<br>Ax.set_title( &#39;Accuracy by NER Classifier&#39; )<br>Ax.set_xticks(ind+width)<br>Ax.set_xticklabels( ( &#39;&#39; ) )</pre><pre>Ax.legend( (rects1[ 0 ], rects2[ 0 ]), ( &#39;Stanford&#39; , &#39;NLTK&#39; ), bbox_to_anchor=( 1.05 , 1 ), loc= 2 , borderaxespad= 0. )</pre><pre>Def  autolabel  (rects) : <br>    # attach some text labels <br>    for rect in rects:<br>        Height = rect.get_height()<br>        Ax.text(rect.get_x()+rect.get_width()/ 2. , 1.02 *height, &#39;%10.2f&#39; % float(height),<br>                Ha= &#39;center&#39; , va= &#39;bottom&#39; )</pre><pre>Autolabel(rects1)<br>Autolabel(rects2)</pre><pre>Plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1/0*RJoEUw0f1hCJ0yPH.png" /></figure><h3>XXIV. testing the speed of the NLTK and Stanford NER markers</h3><p>We have tested the accuracy of our NER classifier, but there are more issues to consider when deciding which classifier to use. Let’s test the speed!</p><p>We know that we are comparing the same thing, we will test it in the same article. Use this episode in NBC News:</p><pre>House Speaker John Boehner became animated Tuesday over  the proposed Keystone Pipeline, castigating the Obama administration for  not having the project.</pre><pre>Republican House Speaker John Boehner says there&#39;s &quot;nothing complex about the Keystone Pipeline,&quot;  and  that  it &#39;s time  to build it .</pre><pre>&quot;Complex? You think the Keystone Pipeline is complex?!&quot; Boehner responded to a questioner. &quot;It&#39;s been under study for five years! We build pipelines in America every day. Do you realize that are 200,000 miles of pipelines in the United States? &quot;</pre><pre>At The Speaker Wentworth ON : &quot;And at The only reason at The President is&#39;s Involved in at The Keystone Pipeline IS Because IT Crosses AN International&#39;s boundary the Listen, WE CAN Build IT There&#39;s Nothing Complex the About at The Keystone Pipeline - IT&#39;s Time to Build IT...&quot;</pre><pre>Of Said Boehner at The President is NO excuse HAD AT the this Point to  not give at The Pipeline at The Go-Ahead the After  at The State Department Report Released A ON catalog on Friday Indicating, at The Project Impact Would have have A minimal ON  at The Environment.</pre><pre>Republicans have long pushed for construction of  the project, which enjoys some measure of Democratic support as well. The GOP is  considering conditioning an extension of  the debt limit on approval of  the project by Obama.</pre><pre>The White House, though, has said that  it has no timetable for a final decision on  the project.</pre><p>First, we perform the import and process the article through reading and word segmentation.</p><pre># -*- coding: utf-8 -*-</pre><pre>Import nltk<br> import os<br> Import numpy as np<br>Import matplotlib.pyplot as plt<br> from matplotlib import style<br> from nltk import pos_tag<br> from nltk.tag import StanfordNERTagger<br> from nltk.tokenize import word_tokenize</pre><pre>Style.use( &#39;fivethirtyeight&#39; )</pre><pre># Process text <br>def  process_text  (txt_file) : <br>    raw_text = open( &quot;/usr/share/news_article.txt&quot; ).read()<br>    Token_text = word_tokenize(raw_text)<br>    Return token_text</pre><p>Great! Now let’s write some functions to split our classification task. Because the NLTK NEG classifier requires a POS tag, we will add a POS tag to our NLTK function.</p><pre># Stanford NER tagger <br>def  stanford_tagger  (token_text) : <br>    st = StanfordNERTagger( &#39;/usr/share/stanford-ner/classifiers/english.all.3class.distsim.crf.ser.gz&#39; ,<br>                             &#39;/usr/share/stanford-ner /stanford-ner.jar&#39; ,<br>                            Encoding= &#39;utf-8&#39; )   <br>    Ne_tagged = st.tag(token_text)<br>    Return (ne_tagged)</pre><pre># NLTK POS and NER <br>taggers def  nltk_tagger  (token_text) :<br>    Tagged_words = nltk.pos_tag(token_text)<br>    Ne_tagged = nltk.ne_chunk(tagged_words)<br>    Return (ne_tagged)</pre><p>Each classifier needs to read the article and classify the named entities, so we wrap these functions in a larger function to make timing easier.</p><pre>Def  stanford_main  () :<br>    Print(stanford_tagger(process_text(txt_file)))</pre><pre>Def  nltk_main  () : <br>    print(nltk_tagger(process_text(txt_file)))</pre><p>When we call our program, we call these functions. We will os.times()wrap our stanford_main()sum nltk_main()function in the function call , taking the fourth index, which is the elapsed time. Then we will plot our results.</p><pre>If __name__ == &#39;__main__&#39; :<br>    Stanford_t0 = os .times()[ 4 ]<br>    Stanford_main()<br>    Stanford_t1 = os .times()[ 4 ]<br>    Stanford_total_time = stanford_t1 - stanford_t0</pre><pre>    Nltk_t0 = os .times()[ 4 ]<br>    Nltk_main()<br>    Nltk_t1 = os .times()[ 4 ]<br>    Nltk_total_time = nltk_t1 - nltk_t0</pre><pre>    Time_plot(stanford_total_time, nltk_total_time)</pre><p>For our drawing, we use the time_plot()function:</p><pre>Def  time_plot  (stanford_total_time, nltk_total_time) : <br>    N = 1 <br>    ind = np.arange(N) # the x locations for the groups <br>    width = 0.35  # the width of the bars<br>    Stanford_total_time = stanford_total_time<br>    Nltk_total_time = nltk_total_time   <br>    Fig, ax = plt.subplots()    <br>    Rects1 = ax.bar(ind, stanford_total_time, width, color= &#39;r&#39; )    <br>    Rects2 = ax.bar(ind+width, nltk_total_time, width, color= &#39;y&#39; )</pre><pre>    # Add text for labels, title and axes ticks <br>    ax.set_xlabel( &#39;Classifier&#39; )<br>    Ax.set_ylabel( &#39;Time (in seconds)&#39; )<br>    Ax.set_title( &#39;Speed ​​by NER Classifier&#39; )<br>    Ax.set_xticks(ind+width)<br>    Ax.set_xticklabels( ( &#39;&#39; ) )   <br>    Ax.legend( (rects1[ 0 ], rects2[ 0 ]), ( &#39;Stanford&#39; , &#39;NLTK&#39; ), bbox_to_anchor=( 1.05 , 1 ), loc= 2 , borderaxespad= 0. )</pre><pre>    Def  autolabel  (rects) : <br>        # attach some text labels <br>        for rect in rects:<br>            Height = rect.get_height()<br>            Ax.text(rect.get_x()+rect.get_width()/ 2. , 1.02 *height, &#39;%10.2f&#39; % float(height),<br>                    Ha= &#39;center&#39; , va= &#39;bottom&#39; )</pre><pre>    Autolabel(rects1)<br>    Autolabel(rects2)    <br>    Plt.show()</pre><p>Wow, NLTK is as fast as lightning! It seems that Stanford is more accurate, but NLTK is faster. This is important information to know when balancing the precision of our preferences with the computing resources required.</p><p>But wait, there is still a problem. Our output is ugly! This is a small sample from Stanford University:</p><pre>[( &#39;House&#39; , &#39;ORGANIZATION&#39; ), ( &#39;Speaker&#39; , &#39;O&#39; ), ( &#39;John&#39; , &#39;PERSON&#39; ), ( &#39;Boehner&#39; , &#39;PERSON&#39; ), ( &#39;became&#39; , &#39;O&#39; ) , ( &#39;animated&#39; , &#39;O&#39; ), ( &#39;Tuesday&#39; , &#39;O&#39; ), ( &#39;over&#39; , &#39;O&#39; ), ( &#39;the&#39; , &#39;O&#39; ), ( &#39;proposed&#39; , &#39;O&#39; ) ,( &#39;Keystone&#39; , &#39;ORGANIZATION&#39; ), (&#39;Pipeline&#39; , &#39;ORGANIZATION&#39; ), ( &#39;,&#39; , &#39;O&#39; ), ( &#39;castigating&#39; , &#39;O&#39; ), ( &#39;the&#39; , &#39;O&#39; ), ( &#39;Obama&#39; , &#39;PERSON&#39; ), ( &#39;administration&#39; , &#39;O&#39; ), ( &#39;for&#39; , &#39;O&#39; ), ( &#39;not&#39; , &#39;O&#39; ), ( &#39;having&#39; , &#39;O&#39; ), ( &#39;approved&#39; , &#39;O&#39; ),( &#39;the&#39; , &#39;O&#39; ), ( &#39;project&#39; ,&#39;O&#39; ), ( &#39;yet&#39; , &#39;O&#39; ), ( &#39;.&#39; , &#39;O&#39; )</pre><p>And NLTK:</p><pre>( S  ( ORGANIZATION House/NNP) Speaker/NNP ( PERSON John/NNP Boehner/NNP) became/VBD animated/VBN Tuesday/NNP over/IN the/DT proposed/VBN ( PERSON Keystone/NNP Pipeline/NNP) , /, Castigating/VBG the/DT ( ORGANIZATION Obama/NNP) administration/NN for/IN not/RB having/VBG approved/VBN the/DT project/NN yet/RB ./.</pre><p>Let’s turn them into a readable form in the next tutorial.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1/0*NBmfg1XA1e2gd0dT.png" /></figure><h3>XXV. Create a readable list of named entities using BIO tags</h3><p>Now that we have completed the test, let’s turn our named entity into a good readable format.</p><p>Again, we will use the same news from NBC News:</p><pre>House Speaker John Boehner became animated Tuesday over  the proposed Keystone Pipeline, castigating the Obama administration for  not having the project.</pre><pre>Republican House Speaker John Boehner says there&#39;s &quot;nothing complex about the Keystone Pipeline,&quot;  and  that  it &#39;s time  to build it .</pre><pre>&quot;Complex? You think the Keystone Pipeline is complex?!&quot; Boehner responded to a questioner. &quot;It&#39;s been under study for five years! We build pipelines in America every day. Do you realize that are 200,000 miles of pipelines in the United States? &quot;</pre><pre>At The Speaker Wentworth ON : &quot;And at The only reason at The President is&#39;s Involved in at The Keystone Pipeline IS Because IT Crosses AN International&#39;s boundary the Listen, WE CAN Build IT There&#39;s Nothing Complex the About at The Keystone Pipeline - IT&#39;s Time to Build IT...&quot;</pre><pre>Of Said Boehner at The President is NO excuse HAD AT the this Point to  not give at The Pipeline at The Go-Ahead the After  at The State Department Report Released A ON catalog on Friday Indicating, at The Project Impact Would have have A minimal ON  at The Environment.</pre><pre>Republicans have long pushed for construction of  the project, which enjoys some measure of Democratic support as well. The GOP is  considering conditioning an extension of  the debt limit on approval of  the project by Obama.</pre><pre>The White House, though, has said that  it has no timetable for a final decision on  the project.</pre><p>Our NTLK output is already a tree (just the last step), so let’s take a look at our Stanford output. We will mark the tag with BIO, B for the beginning of the named entity, I for the internal, and O for the other. For example, if our sentence is yes Barack Obama went to Greece today, we should mark it as Barack-B Obama-I went-O to-O Greece-B today-O. To do this, we will write a series of conditions to check the labels of the current and previous Otags.</p><pre># -*- coding: utf-8 -*-</pre><pre>Import nltk<br> import os<br> Import numpy as np<br>Import matplotlib.pyplot as plt<br> from matplotlib import style<br> from nltk import pos_tag<br> from nltk.tag import StanfordNERTagger<br> from nltk.tokenize import word_tokenize<br> from nltk.chunk import conlltags2tree<br> from nltk.tree import Tree</pre><pre>Style.use( &#39;fivethirtyeight&#39; )</pre><pre># Process text <br>def  process_text  (txt_file) : <br>    raw_text = open( &quot;/usr/share/news_article.txt&quot; ).read()<br>    Token_text = word_tokenize(raw_text)<br>    Return token_text</pre><pre># Stanford NER tagger <br>def  stanford_tagger  (token_text) : <br>    st = StanfordNERTagger( &#39;/usr/share/stanford-ner/classifiers/english.all.3class.distsim.crf.ser.gz&#39; ,<br>                             &#39;/usr/share/stanford-ner /stanford-ner.jar&#39; ,<br>                            Encoding= &#39;utf-8&#39; )   <br>    Ne_tagged = st.tag(token_text)<br>    Return (ne_tagged)</pre><pre># NLTK POS and NER <br>taggers def  nltk_tagger  (token_text) :<br>    Tagged_words = nltk.pos_tag(token_text)<br>    Ne_tagged = nltk.ne_chunk(tagged_words)<br>    Return (ne_tagged)</pre><pre># Tag tokens with standard NLP BIO tags <br>def  bio_tagger  (ne_tagged) :<br>        Bio_tagged = []<br>        Prev_tag = &quot;O&quot; <br>        for token, tag in ne_tagged:<br>             if tag == &quot;O&quot; : #O<br>                Bio_tagged.append((token, tag))<br>                Prev_tag = tag<br>                Continue <br>            if tag != &quot;O&quot;  and prev_tag == &quot;O&quot; : # Begin NE <br>                bio_tagged.append((token, &quot;B-&quot; +tag))<br>                Prev_tag = tag<br>            Elif prev_tag != &quot;O&quot;  and prev_tag == tag: # Inside NE <br>                bio_tagged.append((token, &quot;I-&quot; +tag))<br>                Prev_tag = tag<br>            Elif prev_tag != &quot;O&quot;  and prev_tag != tag: # Adjacent NE <br>                bio_tagged.append((token, &quot;B-&quot; +tag))<br>                Prev_tag = tag<br>        Return bio_tagged</pre><p>Now we write the BIO-tagged tags to the tree, so they are in the same format as the NLTK output.</p><pre># Create tree <br>def  stanford_tree  (bio_tagged) :<br>    Tokens, ne_tags = zip(*bio_tagged)<br>    Pos_tags = [pos for token, pos in pos_tag(tokens)]</pre><pre>    Conlltags = [(token, pos, ne) for token, pos, ne in zip(tokens, pos_tags, ne_tags)]<br>    Ne_tree = conlltags2tree(conlltags)<br>    Return ne_tree</pre><p>Traverse and parse out all named entities:</p><pre># Parse named entities from tree <br>def  structure_ne  (ne_tree) :<br>    Ne = []<br>    For subtree in ne_tree:<br>         if type(subtree) == Tree: # If subtree is a noun chunk, ie NE != &quot;O&quot;<br>            Ne_label = subtree.label()<br>            Ne_string = &quot; &quot; .join([token for token, pos in subtree.leaves()])<br>            Ne.append((ne_string, ne_label))<br>    Return ne</pre><p>In our call, we put all the additional functions together.</p><pre>Def  stanford_main  () :<br>    Print(structure_ne(stanford_tree(bio_tagger(stanford_tagger(process_text(txt_file)))))))</pre><pre>Def  nltk_main  () : <br>    print(structure_ne(nltk_tagger(process_text(txt_file))))</pre><p>Then call these functions:</p><pre>If __name__ == &#39;__main__&#39; :<br>    Stanford_main()<br>    Nltk_main()</pre><p>Here is a nice looking output from Stanford:</p><pre>[(&#39;House&#39;, &#39;ORGANIZATION&#39;), (&#39;John Boehner&#39;, &#39;PERSON&#39;), (&#39;Keystone Pipeline&#39;, &#39;ORGANIZATION&#39;), (&#39;Obama&#39;, &#39;PERSON&#39;), (&#39;Republican House&#39;, &#39; ORGANIZATION&#39;), (&#39;John Boehner&#39;, &#39;PERSON&#39;), (&#39;Keystone Pipeline&#39;, &#39;ORGANIZATION&#39;), (&#39;Keystone Pipeline&#39;, &#39;ORGANIZATION&#39;), (&#39;Boehner&#39;, &#39;PERSON&#39;), (&#39;America &#39;, &#39;LOCATION&#39;), (&#39;United States&#39;, &#39;LOCATION&#39;), (&#39;Keystone Pipeline&#39;, &#39;ORGANIZATION&#39;), (&#39;Keystone Pipeline&#39;, &#39;ORGANIZATION&#39;), (&#39;Boehner&#39;, &#39;PERSON&#39;), (&#39;State Department&#39;, &#39;ORGANIZATION&#39;), (&#39;Republicans&#39;, &#39;MISC&#39;), (&#39;Democratic&#39;, &#39;MISC&#39;), (&#39;GOP&#39;, &#39;MISC&#39;), (&#39;Obama&#39;, &#39;PERSON&#39;), (&#39;White House&#39;, &#39;LOCATION&#39;)]</pre><p>And from NLTK:</p><pre>[(&#39;House&#39;, &#39;ORGANIZATION&#39;), (&#39;John Boehner&#39;, &#39;PERSON&#39;), (&#39;Keystone Pipeline&#39;, &#39;PERSON&#39;), (&#39;Obama&#39;, &#39;ORGANIZATION&#39;), (&#39;Republican&#39;, &#39;ORGANIZATION &#39;), (&#39;House&#39;, &#39;ORGANIZATION&#39;), (&#39;John Boehner&#39;, &#39;PERSON&#39;), (&#39;Keystone Pipeline&#39;, &#39;ORGANIZATION&#39;), (&#39;Keystone Pipeline&#39;, &#39;ORGANIZATION&#39;), (&#39;Boehner&#39; , &#39;PERSON&#39;), (&#39;America&#39;, &#39;GPE&#39;), (&#39;United States&#39;, &#39;GPE&#39;), (&#39;Keystone Pipeline&#39;, &#39;ORGANIZATION&#39;), (&#39;Listen&#39;, &#39;PERSON&#39;), (&#39; Keystone&#39;, &#39;ORGANIZATION&#39;), (&#39;Boehner&#39;, &#39;PERSON&#39;), (&#39;State Department&#39;, &#39;ORGANIZATION&#39;), (&#39;Democratic&#39;, &#39;ORGANIZATION&#39;), (&#39;GOP&#39;, &#39;ORGANIZATION&#39;), (&#39;Obama&#39;, &#39;PERSON&#39;), (&#39;White House&#39;, &#39;FACILITY&#39;)]</pre><h4>Work Hard. God Bless.</h4><h4>Related Stories from DDI:</h4><ul><li><a href="https://www.datadriveninvestor.com/2019/01/23/deep-learning-explained-in-7-steps/">Deep Learning Explained in 7 Steps - Updated | Data Driven Investor</a></li><li><a href="https://www.datadriveninvestor.com/2019/02/07/8-skills-you-need-to-become-a-data-scientist/">8 Skills You Need to Become a Data Scientist | Data Driven Investor</a></li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fb2a0d6%3Fas_embed%3Dtrue&amp;dntp=1&amp;display_name=Upscribe&amp;url=https%3A%2F%2Fupscri.be%2Fb2a0d6%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/0707f5c806284d01a4a13c7b13a91ce3/href">https://medium.com/media/0707f5c806284d01a4a13c7b13a91ce3/href</a></iframe><p><em>Editor’s Disclosure: The editors sometimes post affiliate links to useful resources. If you find them useful and make a purchase, we will earn some big bucks. No, I’m not talking about upsizing my fries kind of big. I’m talking about extra pepperoni on a large pizza kind of big. Thank you for your continued support, we will continue to work hard for the p̶e̶p̶p̶e̶r̶o̶n̶i̶ publication.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d8842fedfdd" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/python-data-science-getting-started-tutorial-nltk-2d8842fedfdd">Python Data Science Getting Started Tutorial: NLTK</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ARTIFICIAL INTELLIGENCE TOOLS YOU CAN USE TODAY — FOR PERSONAL USE]]></title>
            <link>https://medium.datadriveninvestor.com/artificial-intelligence-tools-you-can-use-today-for-personal-use-379ad7303ac4?source=rss-84cdd99683f9------2</link>
            <guid isPermaLink="false">https://medium.com/p/379ad7303ac4</guid>
            <category><![CDATA[internet]]></category>
            <category><![CDATA[daily-life]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[SWAYAM MITTAL]]></dc:creator>
            <pubDate>Tue, 22 Jan 2019 16:28:44 GMT</pubDate>
            <atom:updated>2019-01-25T06:48:22.164Z</atom:updated>
            <content:encoded><![CDATA[<figure><a href="http://www.track.datadriveninvestor.com/1B9E"><img alt="" src="https://cdn-images-1.medium.com/max/700/0*0LGPgTELBaHoVGM6" /></a></figure><h3>Artificial Intelligence Tools You Can Use Today — For Personal Use</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*y79mX09ZOY1XNXrVPmjtlQ.jpeg" /></figure><h3>A break down of the series</h3><p>I’ve trawled through literally thousands of websites (6,000+ links) over a few weeks to bring you a comprehensive list of the best AI products and most promising companies in the field.</p><p>I’ve highlighted some of my favourites, enjoy — !</p><h3>Artificial Intelligence for personal use:</h3><h4>🏠 Personal / In-Home</h4><p><a href="http://ems.ai/"><strong>Ems</strong></a> — helps you find the perfect place to live<br><a href="http://bridge.ai/kitchen/"><strong>Bridge Kitchen</strong></a><strong> </strong>— kitchen assistant for turn-by-turn directions in the kitchen<br><a href="https://unify.id/"><strong>UnifyID</strong></a> — verify your identity by the way you walk, type and sit</p><h4>👔 Work</h4><p><a href="http://www.carly.ai/"><strong>Carly</strong></a> — helps you manage phone calls<br><a href="https://etch.ai/"><strong>ETCH</strong></a> — helps you manage your networks into a searchable database<br><a href="https://leap.ai/welcome"><strong>Leap</strong></a> — recommends companies to apply for based on your skills<br><a href="http://www.lomi.ai/"><strong>Lomi</strong></a> — identifies sales leads<br><a href="https://mosaic.ai/"><strong>Mosaic</strong></a> — helps you write better resumes<br><a href="https://newton.ai/"><strong>Newton</strong></a> — helps you find a dream job<br><a href="https://notion.ai/"><strong>Notion</strong></a> — helps with email overload, organisation, and communication<br><a href="https://www.robby.ai/"><strong>Robby</strong></a> — a better and smarter calendar<br><a href="https://www.stella.ai/"><strong>Stella</strong></a> — scans for jobs and helps manage your application process<br><a href="https://woo.io/"><strong>Woo</strong></a> — helps you make a smarter decision for your career</p><h4>👯 Social</h4><p><a href="https://brightcrowd.com/landing"><strong>Brightcrowd</strong></a> — helps you find meaningful professional connections<br><a href="https://capsule.ai/"><strong>Capsule.ai</strong></a> — recall the good times in life<br><a href="http://dating.ai/"><strong>Dating.ai</strong></a> — dating app with Face Search<br><a href="https://etch.ai/"><strong>ETCH</strong></a> — helps you manage your networks into a searchable database<br><a href="http://www.eezy.ai/"><strong>Eezy</strong></a> — let Eezy take care of your evening so you can take care of having fun<br><a href="http://hash.ai/"><strong>Hashley</strong></a> — ironic hashtag and comment generator for your photos<br><a href="http://hotness.ai/"><strong>Hotness.ai</strong></a> — can tell you how hot you are<br><a href="http://rey.ai/"><strong>Rey</strong></a> — introduces you to people you should really meet (work and play)</p><h4>🎓 Education</h4><p><a href="http://thirdleap.ai/"><strong>Thirdleap</strong></a> — helps children to learn maths<br><a href="https://woogie.ai/"><strong>Woogie</strong></a> — the conversational AI robot that makes learning and discovery fun for children</p><h4>🏥 Health / Medical</h4><p><a href="https://abi.ai/"><strong>Abi</strong></a> — your virtual health assistant<br><a href="https://ada.com/"><strong>Ada</strong></a> — can help if you’re feeling unwell<br><a href="http://airi.ai/"><strong>Airi</strong></a><strong> </strong>— personal health coach<br><a href="https://alz.ai/"><strong>Alz.ai</strong> </a>— helps you care for loved ones with Alzheimer’s<br><a href="http://www.amelie.ai/"><strong>Amélie</strong></a> — chatbot for mental health<br><a href="https://getbitesnap.com/"><strong>Bitesnap</strong></a> — food recognition from meal photos to help count calories<br><a href="http://doc.ai/#helloWorld"><strong>doc.ai</strong></a><strong> </strong>— makes lab results easy to understand<br><a href="http://www.hellojoy.ai/"><strong>Joy</strong></a> — helps you track and improve your mental health<br><a href="https://kiwi.ai/#/home"><strong>Kiwi</strong></a> — helps you to reduce and quit smoking<br><a href="https://x2.ai/"><strong>Tess by x2</strong></a> — mental healthcare in your pocket<br><a href="https://sleep.ai/"><strong>Sleep.ai</strong></a> — diagnose snoring and tooth grinding</p><h4>🕵️ Agents — Personal</h4><p><a href="https://www.amazon.com/Amazon-Echo-Bluetooth-Speaker-with-WiFi-Alexa/dp/B00X4WHP5E"><strong>Amazon Echo / Alexa</strong></a> — everyday personal assistant for in-home<br><a href="http://www.apple.com/ios/siri/"><strong>Apple Siri</strong></a> — an everyday personal assistant on iPhone and Mac<br><a href="https://www.microsoft.com/en-us/mobile/experiences/cortana/"><strong>Cortana</strong></a> — an everyday personal assistant on PC and Windows devices<br><a href="https://www.theguardian.com/technology/2015/aug/27/facebook-m-virtual-assistant-siri-google-now"><strong>Facebook M</strong></a> — competitors to Siri, Now and Cortana<br><a href="http://www.focus.ai/"><strong>Focus</strong></a> — helps you focus, get tasks done and prioritise your day<br><a href="http://gatebox.ai/"><strong>Gatebox</strong></a> — a holographic anime assistant in an espresso machine<br><a href="https://assistant.google.com/"><strong>Google Assistant</strong></a> — everyday personal assistant<br><a href="http://hound.ai/"><strong>Hound</strong></a> — everyday personal assistant<br><a href="http://ling.ai/en/"><strong>Ling</strong></a> — similar to Amazon Echo<br><a href="https://mycroft.ai/"><strong>Mycroft</strong></a><strong> </strong>— is the world’s first open source voice assistant<strong><br></strong><a href="http://remi.ai/"><strong>Remi</strong></a> — like Siri with an interface<br><a href="http://spoken.ai/"><strong>Spoken</strong></a> — virtual assistant with an interface<br><a href="http://viv.ai/"><strong>Viv</strong></a> — like Siri but 10x better</p><h4>📅 Agents — Professional</h4><p><a href="https://claralabs.com/"><strong>Clara</strong></a> — meeting scheduling assistant<br><a href="https://www.juliedesk.com/"><strong>Julie Desk</strong></a><strong> </strong>— meeting scheduling assistant (aimed at C-Suite)<br><a href="http://kono.ai/"><strong>Kono</strong></a> — meeting scheduling assistant<br><a href="https://www.mimetic.ai/"><strong>Mimetic</strong></a> — meeting scheduling assistant<br><a href="https://www.myally.ai/"><strong>My Ally</strong></a> — handles meeting scheduling and manages calendar<br><a href="https://skipflag.com/"><strong>SkipFlag</strong></a> — automatically discover and organise your work<br><a href="https://www.vesper.ai/"><strong>Vesper</strong></a> — virtual assistant aimed at C-Suite<br><a href="http://x.ai/"><strong>x.ai</strong></a> — meeting scheduling assistant<br><a href="http://zoom.ai/"><strong>Zoom.ai</strong></a> — personal assistant to help you at work</p><h4>🙍 Agent — Fun</h4><p><a href="http://fembot.ai/"><strong>Fembot</strong></a> — your AI girlfriend<br><a href="http://simples.ai/"><strong>Lifos</strong></a> — dynamic independent entities that interact with the web and social<br><a href="https://replika.ai/"><strong>Replika</strong></a> — your AI friend that you raise through text conversations</p><h4>📝 Agents — Chatbot</h4><p><a href="https://brin.ai/"><strong>Brin</strong></a> — helps you make smarter business decisions<br><a href="https://chatfuel.com/"><strong>Chatfuel</strong></a><strong> </strong>— create a Facebook chatbot in 7 minutes<strong><br></strong><a href="https://luka.ai/"><strong>Luka</strong></a> — chatbot messenger for people and other chatbots<br><a href="https://mywave.me/"><strong>myWave</strong></a> — chatbot to help you throughout your daily life<br><a href="https://lyr.ai/"><strong>Lyra</strong></a> — monitor analyse your carbon emissions</p><h4>🗞️ Entertainment / News</h4><p><a href="http://jottr.ai/"><strong>Jottr</strong></a> — content and news app that learns what you like and don’t like<a href="https://news360.com/"><strong><br>News360 </strong></a>— learns what you enjoy and finds stories you’ll like</p><h4>✈️ Travel</h4><p><a href="http://fido.ai/"><strong>Ada</strong></a> — chatbot that helps you navigate and make decisions<a href="http://ema.ai/"><strong><br>Emma</strong></a> — automatically calculates and adds meeting travel time<a href="https://www.eta.ai/"><strong><br>ETA</strong></a> — helps you manage travel itineraries and meetings<br><a href="http://mezi.com/"><strong>Mezi</strong></a> — helps with booking flights, hotels, restaurant reservations and more<br><a href="http://ready.ai/"><strong>Ready</strong></a> — traffic forecaster and travel time prediction</p><h4>🚗 Vehicle</h4><p><a href="https://www.vin.li/"><strong>Vinli</strong></a> — turns any car into a smart car</p><h4>🏃‍ Fitness</h4><p><a href="http://mara.ai/"><strong>Mara</strong></a> — smarter running app</p><h4>🎵 Music</h4><p><a href="http://aiva.ai/"><strong>Aiva</strong></a> — composes emotional soundtrack music<br><a href="https://www.pandora.com/"><strong>Pandora</strong></a> — finds music you might like</p><h4>🏛️ Insurance / Legal</h4><p><a href="http://aux.ai/"><strong>Docubot</strong></a> — can advise you on legal issues<br><a href="http://www.driveway.ai/"><strong>Driveway</strong></a> — tracks and rewards safe drivers</p><h4>🛒 Shopping</h4><p><a href="https://www.entrupy.com/"><strong>Entrupy</strong></a> — helps detect if high-end designer products are authentic<br><a href="http://fify.ai/"><strong>Fify</strong></a> — helps you shop for clothing<br><a href="http://gofind.ai/"><strong>GoFind</strong></a> — helps you find clothing online by taking a photo<br><a href="http://mode.ai/"><strong>Mode.ai</strong></a><strong> </strong>— helps you find clothing online</p><h4>💰 Finance</h4><p><a href="https://www.abe.ai/"><strong>Abe</strong></a><strong> </strong>— fast answers about your finances<br><a href="http://andy.ai/"><strong>Andy</strong></a> — a personal Tax Accountant<br><a href="https://ara.ai/"><strong>Ara</strong></a> — helps you budget<br><a href="http://bond.ai/"><strong>Bond</strong></a> — helps you achieve your financial goals<br><a href="https://mylo.ai/"><strong>Mylo</strong></a> — rounds up your everyday purchases and invest the spare change<br><a href="https://www.olivia.ai/"><strong>Olivia</strong></a> — helps you manage your finances<br><a href="https://www.responsive.ai/"><strong>Responsive</strong></a> — institutional-grade active portfolio management<br><a href="https://www.roger.ai/"><strong>Roger</strong></a> — helps you pay bills easily<br><a href="http://pocket.ai/"><strong>Wallet</strong></a> — AI for your daily finance decisions<br><a href="http://xoe.ai/"><strong>Xoe.ai </strong></a>— AI lending chatbot</p><h4>📱 Social media / Website management</h4><p><a href="https://firedrop.ai/"><strong>Firedrop</strong></a> — websites designed automatically, just add content and publish<br><a href="http://hash.ai/"><strong>Hashley</strong></a> — ironic hashtag and comment generator for your photos<br><a href="https://million.ai/"><strong>Millions.ai</strong></a> — throw content at it and it builds you a website<br><a href="https://oly.ai/"><strong>Oly</strong></a><strong> </strong>— selects and helps you publish content to your social media<br><a href="http://signature.ai/"><strong>Signature</strong> </a>— build elegant landing pages using your social media content<br><a href="https://zen.ai/"><strong>Zen.ai</strong></a> — personalised recommendations for your Shopify store</p><h4>💡 IoT / IIoT</h4><p><a href="http://www.aerial.ai/"><strong>Aerial</strong></a> — home activity, movement and identity sensor<br><a href="http://bridge.ai/"><strong>Bridge.ai</strong></a> — smart-home platform focused on speech and sound<br><a href="http://cubic.ai/"><strong>Cubic</strong></a> — one place to connect your smart home devices<br><a href="http://grow.ai/"><strong>Grojo</strong></a> — grow room controller and monitoring system<br><a href="http://home.ai/"><strong>Home</strong></a> — autonomous home operations with connected devices<br><a href="http://hello.ai/"><strong>Hello</strong></a> — helps you monitor and improve your sleep<br><a href="https://www.josh.ai/"><strong>Josh</strong></a> — whole house voice control<br><a href="https://mycroft.ai/"><strong>Mycroft</strong></a><strong> </strong>— is the world’s first open source voice assistant<br><a href="https://www.nanit.com/"><strong>Nanit</strong></a> — the baby monitor reinvented<br><a href="https://nest.com/"><strong>Nest</strong></a> — a range of in-home devices such as Thermostat, security and alarms</p><h4>⌚ Wearables</h4><p><a href="http://eli-talk.com/en/"><strong>Eli</strong></a> — helps you learn a new language from conversations through your day<a href="https://www.kickai.com/"><strong><br>Kick.ai</strong></a> — wearable for martial arts to help analyse kicks and fights</p><h4>🌐 Language / Translation</h4><p><a href="https://liv.ai/"><strong>Liv</strong></a> — transcribe conversations in English and Hindi<a href="https://translator.microsoft.com/neural/"><strong><br>Microsoft Translator </strong></a>— language translator powered by neural networks</p><h4>🤖 Robots</h4><p><a href="http://dispatch.ai/"><strong>Dispatch</strong></a> — delivery by robot<a href="http://roboy.org/"><strong><br>Roboy</strong></a> — a humanoid robot aimed to be as capable as a human<br><a href="http://spoon.ai/"><strong>Spoon</strong></a> — a robot (I couldn’t work out its purpose, but it sounded cool!)</p><h4>💌 Useful + Fun + Random</h4><p><a href="https://www.captionbot.ai/"><strong>CaptionBot</strong></a> — Microsoft describes any photo<a href="https://crowdfunding.ai/"><strong><br>Crowdfunding.ai</strong></a> — crowdfunding platform for AI projects<br><a href="http://elevator.ai/"><strong>Elevator</strong></a> — helps you find and buy cannabis (with recommendations)<br><a href="https://fieldguide.net/"><strong>Fieldguide</strong></a> — universal field guide that suggests possible matches<br><a href="http://frankenstein.ai/"><strong>Frankenstain.ai </strong></a>— is a collaborative storytelling experiment using AI, machine learning, robotics, bio-engineering and IoT<br><a href="http://www.laugh.ai/"><strong>#Laugh</strong></a> — laughs visualised in a digital format, that may be sent to space<br><a href="http://intelligentx.ai/"><strong>IntelligentX Brewing Co.</strong></a><strong> </strong>— beer brewed by artificial intelligence<br><a href="https://token.ai/"><strong>Token</strong></a> — helps you send the perfect gift<br><a href="http://wixi.ai/"><strong>Wixi</strong></a> — helps you fix Wifi problems</p><h4>📚 Research</h4><p><a href="https://apollo.ai/"><strong>Apollo</strong></a> — breaks down articles and PDF’s into quick, readable dot points<a href="http://ferret.ai/"><strong><br>Ferret.ai</strong></a><strong> </strong>— helps you research by summarising articles and searchability<br><a href="http://research.ai/"><strong>Iris</strong></a> — helps you research and visualise concepts in research papers</p><h4>Work Hard. God Bless.</h4><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2Fb2a0d6%3Fas_embed%3Dtrue&amp;dntp=1&amp;display_name=Upscribe&amp;url=https%3A%2F%2Fupscri.be%2Fb2a0d6%2F&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/0707f5c806284d01a4a13c7b13a91ce3/href">https://medium.com/media/0707f5c806284d01a4a13c7b13a91ce3/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=379ad7303ac4" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/artificial-intelligence-tools-you-can-use-today-for-personal-use-379ad7303ac4">ARTIFICIAL INTELLIGENCE TOOLS YOU CAN USE TODAY — FOR PERSONAL USE</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>