<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Zuzanna Deutschman on Medium]]></title>
        <description><![CDATA[Stories by Zuzanna Deutschman on Medium]]></description>
        <link>https://medium.com/@zuzannadeutschman?source=rss-dc134fe9f7c3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*qw83_CNzQzX7rfCT</url>
            <title>Stories by Zuzanna Deutschman on Medium</title>
            <link>https://medium.com/@zuzannadeutschman?source=rss-dc134fe9f7c3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 01:32:36 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@zuzannadeutschman/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Predict The Office character with TwoModalBERT]]></title>
            <link>https://medium.com/@zuzannadeutschman/predict-the-office-character-with-twomodalbert-44e575262c7d?source=rss-dc134fe9f7c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/44e575262c7d</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Zuzanna Deutschman]]></dc:creator>
            <pubDate>Sat, 12 Nov 2022 17:27:22 GMT</pubDate>
            <atom:updated>2022-11-26T16:17:01.038Z</atom:updated>
            <content:encoded><![CDATA[<h3>Predict The Office Character with TwoModalBERT</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bdtD06pAXM5aO_iSDpj-VA.jpeg" /><figcaption><a href="https://unsplash.com/photos/_2uar7MGmgw">Source</a></figcaption></figure><h3>Introduction</h3><p>Have you ever dealt with a case when there were multiple text features in your dataset? For example, your task could be to properly classify a message given its context, understood as the previous message.</p><p>Let’s consider whether we want to classify the below sample as positive or negative.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/815/1*BC46iAuBtk8PA2cHCcJd7Q.png" /></figure><p>When only the message is considered, you can see its sentiment is positive due to the word ‘incredible’. However, when the context is considered, sentiment will be rather negative.</p><p>Given the above, ML Engineer needs to find an answer for below questions:</p><ul><li>Is it worth considering context as a separate feature?</li><li>Does concentrating both text features into one improve the model’s performance?</li><li>Should the weighing of context and message be introduced? If so, what are the proper weights?</li></ul><p>As a very first check, I would recommend just concatenating both text fields. If it brings an improvement to the model compared to the usage of only the latest message — we should dive into weights research. How about we create a neural network with two modes and a tunable size of dense layers on top of each mode? In that way, we could find proper weights automatically!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fex1kwgVxTQx2GrdDBAWkg.png" /><figcaption>TwoModalBERT enables finding the proper weights of two text modes in nn!</figcaption></figure><p>This is can be done with the TwoModalBERT package. Let’s see how the neural network inside it is constructed.</p><h3>Neural network architecture</h3><p>Below you can find how TwoModalBERT is constructed and what are the class parameters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/856/1*jHvF6qveJVuIb7buif1IoQ.png" /></figure><p>First of all, on top of the last BERT layer, a linear layer is added. To be precise, it is added on top of the CLS token. As the CLS token aggregates the entire sequence representation, it is often used in a classification task. Let’s see the relevant three lines of code for better understanding.</p><pre>outputs = self.model(input_ids=input_ids, attention_mask=attention_mask)<br>last_hidden_states = outputs[&#39;hidden_states&#39;][-1]<br># last layer size: (batch_size, seq_len, hs)<br>cls_hidden_states = last_hidden_states[:, 0, :]<br>last_hidden_states_reduced = linear_layer(cls_hidden_states)</pre><p>The linear layer transforms input features with hidden size relevant to the BERT model, which is usually 768 for the models available in the <em>transformers</em> package, into features with hidden size equal to predefined context_size and text_size.</p><p>In the next step, the dropout layer is added on the top. Why? Because it makes the neural network less sensitive to the specific weights of neurons and not prone to overfitting.</p><p>Finally, both branches created similarly are combined and followed by another dropout layer. Does it work? Let’s check out a short demo.</p><h3>TwoModalBERT package</h3><p>The TwoModalBERT package allows us to quickly run an experiment with the two-modal neural network architecture described in the above section. It allows quickly constructing a model on top of Pytorch and transformers libraries and enables experimenting with weights of two input texts. So, how to use the package?</p><p>First, we should create the config.ini file with parameters described in the below table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*NOCrWXMV2Cl6XygAzGOPZw.png" /><figcaption>Parameters in the config.ini file</figcaption></figure><p>After the initial parameters are set up, we can go through the example. We will use the dataset containing all lines of dialogue from the series “The Office”. Our goal is going to be: to verify whether it is easier to recognize the character who speaks based on the previous and the current line or only the current line. After small preprocessing the dataset looks as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*amEJxi4EVufQ3vs0nJOmbw.png" /></figure><p>Essential columns for further analysis are:</p><ul><li>line — what the speaker says</li><li>context — what the previous speaker said</li><li>label — a number associated with each speaker</li></ul><pre># intialize modules<br>DataPreparation = TwoModalDataPreparation(config=config)<br>Trainer = TwoModalBertTrainer(device=DEVICE, config=config)</pre><pre># create data loaders<br>(<br>    train_data_loader,<br>    train,<br>    val_data_loader,<br>    val,<br>    test_data_loader,<br>    test,<br>) = DataPreparation.prepare_data(<br>    df,<br>    text_column=&quot;line&quot;,<br>    context_column=&quot;context&quot;,<br>    label_column=&quot;label&quot;,<br>    train_size=0.8,<br>    val_size=0.1,)</pre><p>After we created data loaders, we can train the model. Note, that all the neural network parameters used here are presented in the previous section. Let’s set up text_size = 100 and context_size = 1.</p><pre># train the model <br>model, history = Trainer.train_model(<br> train_data_loader,<br> train,<br> val_data_loader,<br> val,<br> text_size=100,<br> context_size=50,<br> binary=False,<br> text_p=0.3,<br> context_p=0.3,<br> output_p=0.3,<br> )</pre><pre># evaluate the model on a test set<br>y_pred, y_test = test_model(model, test_data_loader)<br>y_pred, y_test = [e.cpu() for e in y_pred], [e.cpu() for e in y_test]</pre><p>Since the model is trained, we may evaluate it on a test set (feel free to apply any selected metric on y_pred and y_test). We will check the confusion matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/401/1*93YaNhprhlGmLJqHgjOOrQ.png" /></figure><p>The model didn’t learn to recognize any character except for Michael. How about we increase context_size to 50 and see how the confusion matrix changes?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/1*tO5fEFxCht7TKRo6oNm4Fw.png" /></figure><p>We can observe that the model learned to recognize mainly Dwight, Michael, and Jim. For less popular characters it doesn’t work very well. However, compared to the previous option there is a significant improvement.</p><p>Finally, let’s do inference on the best model!</p><pre># run on new pair of text inputs<br>line = &quot;Dwight is my best friend.&quot;<br>context = &quot;What do you think about Dwight?&quot;<br>predict_on_text(model, line, context)</pre><p>The model predicts Stanley!</p><h3>Summary</h3><p>This article presented how to train two modal neural networks with TwoModalBERT package. It allows us to quickly check how different weighting modes of two input text fields influence the classification neural network performance. For the full Jupyter Notebook example please visit my <a href="https://github.com/zuzadeu/twomodalbert">GitHub</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=44e575262c7d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Sentiment Analyzer with BERT (build, tune, deploy)]]></title>
            <link>https://medium.com/data-science/sentiment-analyzer-with-bert-build-tune-deploy-da84c0f2366d?source=rss-dc134fe9f7c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/da84c0f2366d</guid>
            <category><![CDATA[deployment]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Zuzanna Deutschman]]></dc:creator>
            <pubDate>Fri, 24 Jul 2020 12:49:44 GMT</pubDate>
            <atom:updated>2022-08-06T06:01:57.295Z</atom:updated>
            <content:encoded><![CDATA[<h4>Brief description of how I developed sentiment analyzer. It covers text preprocessing, model building, tuning, API, frontend creation and containerization.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*_AQGOWAncIy13B7J8qIcbQ.gif" /></figure><h3>Dataset</h3><p>I used the<a href="https://nlp.stanford.edu/sentiment/"> dataset published by The Stanford NLP Group</a>. I merged two files, namely ‘dictionary.txt’ including 239,232 text fragments and ‘sentiment_labels.txt’ containing the sentiment scores assigned to the various text fragments.</p><h3>Text preprocessing with regular expressions</h3><p>To clean the text, I usually use a bunch of functions containing regular expressions. In common.py you can find all of them, for example remove_nonwords described below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/41625a0b7b422cddc31cf08677c32f8c/href">https://medium.com/media/41625a0b7b422cddc31cf08677c32f8c/href</a></iframe><p>Similar functions were used for empty rows, special signs, numbers and html code removal.</p><p>After text cleaning, it’s time for BERT embeddings creation. For that purpose, I used <a href="https://github.com/hanxiao/bert-as-service">bert-as-service</a>. It is very simple and consists of only 3 steps: download a pre-trained model, start the BERT service and use client for sentence encodings of specified length.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b79c66e55c39ed6d2d7013ec23592d9b/href">https://medium.com/media/b79c66e55c39ed6d2d7013ec23592d9b/href</a></iframe><p>There are multiple parameters that can be setup, when running a service. For example, to define max_seq_len, I calculated 0.9 quantile of train data length.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1f7439a60d4cecbbc2a59e87db19e5ba/href">https://medium.com/media/1f7439a60d4cecbbc2a59e87db19e5ba/href</a></iframe><p>Preprocessed data has a form of data frame containing 768 features. For full code, please go to nlp_preprocess.py.</p><h3>Model building with Keras</h3><p>In this part, we build and train the model on different parameters. Let’s assume we want 5-layers neural network as below. We will parametrize batch_size, number of epochs, number of nodes in the first 4 dense layers and 5 dropout layers.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/44dd84687f23483fab27a15d7c9b576b/href">https://medium.com/media/44dd84687f23483fab27a15d7c9b576b/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/275d374ae283bb1429b23c6bfb67f636/href">https://medium.com/media/275d374ae283bb1429b23c6bfb67f636/href</a></iframe><h3>Model tuning with Sacred</h3><p>Now we can tune the parameters. We will use sacred module. Key points here are:</p><p><strong>1. Create an Experiment and add Observer</strong></p><p>First we need to create an experiment and observer that logs all kinds of information. It’s very simple!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/131494e967c889f96c1f7160d12f8e48/href">https://medium.com/media/131494e967c889f96c1f7160d12f8e48/href</a></iframe><p>2.<strong> Define the main function</strong></p><p>The @ex.automain decorator defines and runs the main function of the experiment when we run the Python script.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ee31f44e8534747c845dee5d3525a281/href">https://medium.com/media/ee31f44e8534747c845dee5d3525a281/href</a></iframe><p><strong>3. Add the Configuration parameters</strong></p><p>We will define them through Config Scope.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/770ea9fcd50ac88eb37924ffb8b6510e/href">https://medium.com/media/770ea9fcd50ac88eb37924ffb8b6510e/href</a></iframe><p><strong>4. Add metrics</strong></p><p>In our case here I want to know the<strong> MAE</strong> and<strong> MSE</strong>. We can use the Metrics API for that.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a11682d7ec218c9275e638a37f8c4786/href">https://medium.com/media/a11682d7ec218c9275e638a37f8c4786/href</a></iframe><p><strong>5. Run the experiment</strong></p><p>Functions from the previous steps are stored in model_experiment.py script. In order to run our exepriment for bunch of parameters, we create and run run_sacred.py. For all possible permutations, MAE and MSE will be saved in MongoDB.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/36d9c04efe4bdc3692dc442ccec1e702/href">https://medium.com/media/36d9c04efe4bdc3692dc442ccec1e702/href</a></iframe><p>The best result I got is 9% of MAE score. That means that our sentiment analyzer works pretty good. We can check it with model_inference function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2559871841cd04d3f907d986c4b3f414/href">https://medium.com/media/2559871841cd04d3f907d986c4b3f414/href</a></iframe><p>Please note that the score is normalized so that outlier values can be also obtained. After model is saved, we can build a Web API!</p><h3>Web API creation with Flask</h3><p>Now we want to create an API that runs the code in the function and displays the returned result in the browser.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/155c09a0e3ab38242f78e5d58c95a4b2/href">https://medium.com/media/155c09a0e3ab38242f78e5d58c95a4b2/href</a></iframe><p>The syntax @app.route(&#39;/score&#39;, methods=[&#39;PUT&#39;]) lets Flask know that the function, score, should be mapped to the <em>endpoint/score</em>. The methods list is a keyword argument that tells us what kind of HTTP requests are allowed. We’ll be using PUT requests to receive sentences from a user. In function score, we get a score in dictionary form, since it can be easily converted to a JSON string. Full code is available in api.py.</p><h3>Frontend</h3><p>For web interface, three files were created:</p><ul><li>index.html provides the basic structure of the site: title, description, input text area and circle with score.</li><li>style.css is used to style the website.</li><li>index.js provides interactivity. It is responsible for reading user input, handling API requests and presenting calculated score. Three main functions here are:</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6b41bbc7cc5dcf7012675ba2353c03a1/href">https://medium.com/media/6b41bbc7cc5dcf7012675ba2353c03a1/href</a></iframe><p>For gradient <strong>HSV</strong> model was used. Saturation and Value are constants. Hue corresponds to score value. Changing hue in range [0;120] yields smooth colour change from red to yellow to green.</p><h3>Docker containerization</h3><p>The brilliance of <a href="https://www.simplilearn.com/tutorials/docker-tutorial">Docker</a> is that, once you package an application and all its dependencies into container, you ensure it will run in any environment. It is generally recommended to separate areas of concern by using one service per container. In my small app there are 3 parts that should be combined: bert-as-service, application and frontend. The tool that helps you build Docker images and run containers is <strong>Docker Compose</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/1*VzZrdRgWuBWg_e6tgeIuKA.png" /></figure><p>Steps that we need to do to dockerize our code:</p><ul><li>Create separate folders for bert-as-service, api and frontend,</li><li>Put there relevant files,</li><li>Addrequirenments.txt and Dockerfile to each folder. The first file should cover all needed libraries that will be installed via command in the second file. Its format is described in docker <a href="https://docs.docker.com/engine/reference/builder/">documentation</a></li><li>Create docker-compose.yaml in the 3 folders directory. Define the 3 services that make up the app in this file, so they can be run together in an isolated environment.</li></ul><p>Now we are ready to build and run our application! Please see the sample outputs below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F51u0SJEBnzAZZeUfN2wGg.png" /></figure><p>As usual, please feel free to view the full code on my GitHub.</p><p><a href="https://github.com/zuzadeu/Sentiment-Analyzer-with-BERT">GitHub - zuzadeu/Sentiment-Analyzer-with-BERT</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=da84c0f2366d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/sentiment-analyzer-with-bert-build-tune-deploy-da84c0f2366d">Sentiment Analyzer with BERT (build, tune, deploy)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Generate Nick Cave’s Lyrics in 2 mins]]></title>
            <link>https://medium.com/data-science/generate-nick-caves-lyrics-in-2-mins-e265f8fc78e2?source=rss-dc134fe9f7c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/e265f8fc78e2</guid>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Zuzanna Deutschman]]></dc:creator>
            <pubDate>Fri, 22 May 2020 17:28:11 GMT</pubDate>
            <atom:updated>2020-05-24T03:56:52.509Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>In this article, we will use GPT-2 language model to develop simple songs lyrics generator on the example of Nick Cave, who is one of my favourite artists.</em></h4><h3>GPT-2</h3><p>GPT-2 is a large transformer-based language model with 1.5 billion parameters. It is trained with a simple objective: predict the next word, given all of the previous words within some text. The diversity of the dataset made of 8 million web pages, causes this simple goal to contain naturally occurring demonstrations of many tasks across diverse domains. For details, please go to <a href="https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf">the OpenAI publication</a>. Let’s focus on practise!</p><h3>Parse lyrics</h3><p>First of all, we need to parse lyrics from <em>Nick Cave’s </em><a href="https://www.nickcave.com/lyrics/">official webpage</a> using well known BeautifulSoup library.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/018e89a502cbe05b6f10a020486c0ca5/href">https://medium.com/media/018e89a502cbe05b6f10a020486c0ca5/href</a></iframe><p>After running above code we have 201 songs coming from 20 albums saved in songs.txt file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/301/1*4bRD_H-kz5cRlqhP_iHSfQ.png" /></figure><h3>Train model</h3><h4>Build tokenizer</h4><p>Use <a href="https://github.com/minimaxir/aitextgen">aitextgen library</a> to train a custom tokenizer on a downloaded songs. This will save two files: aitextgen-vocab.json and aitextgen-merges.txt, that are needed to rebuild the tokenizer.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/346e59e0200cd22216aba2e521a7fe3a/href">https://medium.com/media/346e59e0200cd22216aba2e521a7fe3a/href</a></iframe><h4>Train GPT-2 transformer model</h4><p>Use created tokenizer to initiate aitextgen</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0f5ae5ef643a08f30c5df9a781e2e87b/href">https://medium.com/media/0f5ae5ef643a08f30c5df9a781e2e87b/href</a></iframe><p>Create TokenDatasets, that builds datasets for training, processing them with the appropriate size.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c790aa718f2f0a074ae949a0940ab0ea/href">https://medium.com/media/c790aa718f2f0a074ae949a0940ab0ea/href</a></iframe><p>Time for training!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f82d25ed00e0faa6017252243c9e87ca/href">https://medium.com/media/f82d25ed00e0faa6017252243c9e87ca/href</a></iframe><h3>Generate song</h3><p>Generate 3 paragraphs starting from ‘We drowned’ and save them to file. You can modify max_length and temperature parameters. Temperature is randomness related hyper-parameter. When its value is small (e.g. 0,2), the GPT-2 model is more confident but also more conservative. When temperature is a large value (e.g. 1), the GPT-2 model produces more diversity and also more mistakes. In my opinion, the second option is much better when you want to generate a piece of art ;).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ae9c7bab5b38832fe2f9f221c684465d/href">https://medium.com/media/ae9c7bab5b38832fe2f9f221c684465d/href</a></iframe><p>Below you can see sample results. Looks quite promising, especially considering how much coding is required :).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*8O_bWy2Bri87rgwWlCKWJg.png" /></figure><p>Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e265f8fc78e2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/generate-nick-caves-lyrics-in-2-mins-e265f8fc78e2">Generate Nick Cave’s Lyrics in 2 mins</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Replicate your friend with Transformer]]></title>
            <link>https://blog.chatbotslife.com/replicate-your-friend-with-transformer-bc5efe3a1596?source=rss-dc134fe9f7c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc5efe3a1596</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[bots]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Zuzanna Deutschman]]></dc:creator>
            <pubDate>Wed, 15 Apr 2020 11:02:42 GMT</pubDate>
            <atom:updated>2022-08-06T06:14:46.524Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*k_2VQs053y1gpID3UWhqLw.png" /><figcaption><a href="https://www.scnsoft.com/blog-pictures/mobile-apps/live-chats-vs-chatbots-01.png">Source</a></figcaption></figure><p>In this tutorial we will create <a href="https://chatbotslife.com/"><strong>Chatbot</strong></a> based on Messenger conversations downloaded from Facebook. While you miss your friend, you can easily make a bot that talks like them. For demo purpose I used conversation which contains almost 50 thousands exchanges between me and my boyfriend.</p><p>The project is compatibile with Python 3.4+ and TensorFlow 2.</p><h3>Features</h3><ul><li>Fast and reliable — similarly to Google Translator, it uses Transformer model, implemented in TensorFlow</li><li>Each language compatibile — corpus is built based on sent messages. So, your input conversation may be in any language you want! For me, Polish language works fantastic.</li><li>Easy to run</li></ul><h3>Demo</h3><p>Below you can see how sample talk looks like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*41CZzJCq9c9EaVVagt0kZg.gif" /><figcaption>Chatbot demo</figcaption></figure><h3>Quick Guide</h3><p>Firstly, please download Messenger conversation with any friend you want. <a href="https://www.zapptales.com/en/download-facebook-messenger-chat-history-how-to/">Here</a> you can find description how to do it. As a result you should have one or multiple files called message_{n}.html in certain folder.</p><ul><li>Clone <a href="https://github.com/zuzadeu/Messenger-Chatbot">this repository</a></li><li>To build and run Chatbot, you need the following libraries preinstalled: beautifulsoup4, colorama, lxml, pandas, re, requests, tensorflow, tensorflow_datasets, tqdm</li><li>To run your <a href="https://chatbotslife.com/">Chatbot</a>, only those 3 lines have to be written in Command Prompt (in repository path).</li></ul><h3>python conversations_to_csv.py path_to_html_files You will be asked to enter input and target persons fullname. The first person is probably you and the target person is the one you want to make chatbot based on. Please note, that names you enter should be exactly the same as provided on Facebook.</h3><ul><li>python train.py This script is responsible for building and training Transformer model, so it will take some time to complete. If you would like to change some parameters, for example batch size or number of epochs, you can easily do it within the script.</li><li>python talk.py You will be asked to enter your and chatbot name or nick. Then you can start your conversation. Enjoy! For closing, press CTRL + C</li></ul><p>I hope you like chatbot you built. If you are interested in details of how it works, please visit <a href="https://gitlab.com/Deutschman/chat-bot">my repository</a>, go to the <a href="https://gitlab.com/Deutschman/chat-bot/-/blob/master/transformer_chatbot.ipynb">transformer_chatbot.ipynb</a> Notebook for the code description and dive in the next part of this article to understand how Transformer model works.</p><figure><a href="https://www.chatbotconference.com/"><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5QA4jEOW_w_mKaAW" /></a></figure><h3>Transformer</h3><p>Our <a href="https://chatbotslife.com/">Chatbot</a> is based on Transformer model. Transformer is a deep machine learning model introduced in 2017, used in NLP area for various tasks such as machine translation and text summarization. The core idea behind the Transformer model is <em>self-attention</em> — the ability to attend to different positions of the input sequence to compute a representation of that sequence. The two components of Transformer are Encoder and Decoder. Let’s consider the images below, that come from <a href="https://arxiv.org/abs/1706.03762"><em>Attention is All You Need</em></a> paper and describe enumerated parts.</p><h3>Trending AI Articles:</h3><blockquote><a href="https://chatbotslife.com/building-a-news-bot-using-twitter-api-in-10-mins-51cf4eb744b4">1. Building a News Bot using Twitter API in 10 mins!</a></blockquote><blockquote><a href="https://chatbotslife.com/building-alexa-skills-in-python-for-absolute-beginners-8166a7ad01fa">2. Building Alexa skills in Python, for absolute beginners.</a></blockquote><blockquote><a href="https://chatbotslife.com/ultimate-guide-to-leveraging-nlp-machine-learning-for-you-chatbot-531ff2dd870c">3. Ultimate Guide to Leveraging NLP &amp; NLU for you Chatbot</a></blockquote><blockquote><a href="https://chatbotslife.com/chatbot-conference-2020-e56430926042">4. Chatbot Conference 2020</a></blockquote><h4>Encoder</h4><p>Role of Encoder is to encode the input sentence into <strong>context vector</strong>. If we have 5 tokens long sentence, then it’s context vector is <em>Z = (z1, z2, z3, z4, z5)</em>. What is essential, each context vector has seen all the tokens on each positions within the input sentence — not like hidden state in RNN. Let’s dive into details.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/341/1*FLk9d5dXNeMLaBDtmFhpyg.png" /><figcaption>Encoder Architecture</figcaption></figure><ol><li>Tokens are passed through a standard <strong>Embedding Layer</strong>. There is no information about order of tokens in a sequence yet.</li><li>In <strong>Positional Embedding Layer</strong>, input is position of the token in a sentence. Token and positional embeddings are summed. Before, token embeddings are multiplied by scaling factor, which is square root of depth size. That reduces variance. In the next step dropout is applied to the combined embeddings.</li><li>Now the combined layers are passed through encoder layers. At the beginning <strong>source mask </strong>is used in order to mask the Multi-Head Attention mechanism. As we don’t really want the network to pay attention to the padding, we’re going to mask it. 0 is assigned to pad token and 1 otherwise.</li><li><strong>Multi-Head Attention</strong> is applied to the sentence itself (self-attention) in order to focus on the most relevant parts of the input sequence by calculating similarity score between each word and the rest of words in a sentence. In the next part of this article, the mechanism is explained on the example.</li><li>Results from each head are combined and concatenated by weight matrix in order to keep the input vector size. The <strong>residual connection</strong> provides information about embedding position that was trained before. Dropout is performed and <strong>normalization</strong> is applied — each feature of a vector has a mean 0 and standard deviation 1, so transformer can be trained easier.</li><li><strong>Feed Forward</strong> increases the non-linearity and changes representation of data for better generalization over RELu function and dropout. Normalization is applied again and vectors are transformed back into initial representations. They are ready to serve as input for another encoder layer or decoder.</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/71b94ea444a502ba63433aac46c63fbf/href">https://medium.com/media/71b94ea444a502ba63433aac46c63fbf/href</a></iframe><h4>Decoder</h4><p>The purpose of decoder is to take the encoded representations of the source sentence and convert it into predicted tokens. It compares them with target sentence and calculates loss that will be used for further training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/348/1*3mDmkwNx74Q19rTa5aI7RQ.png" /><figcaption>Decoder Architecture</figcaption></figure><ol><li>Combine<strong> Positional Embedding </strong>with embedded target tokens and dropout</li><li><strong>Masked Multi-Head Attention</strong> is applied. Mask is built from two components. The first component is padding mask, similarly to the encoder mask. The second component’s role is to make n-th token see only itself and the previous tokens, in order to prevent decoder from cheating by simply checking at the next token in a sentence. This look-ahead mask has a form of diagonal matrix, where the elements above diagonal will be 0 and the rest is 1. The two components are combined with logical <em>and</em>, creating a mask together. Attention layer is followed by dropout, residual connection and layer normalization.</li><li>In the second <strong>Multi-Head Attention</strong> layer, we take queries from decoder and keys and values are the encoder representations. It is again followed by dropout, residual connection and layer normalization.</li><li>Finally we pass through <strong>Feed Forward</strong> layer and another sequence of dropout, residual connection and layer normalization.</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/93139bec5d87aba61d5c16516cc78195/href">https://medium.com/media/93139bec5d87aba61d5c16516cc78195/href</a></iframe><h4>Multi-Head Attention</h4><p>Crucial part of transformer is Self-Attention. Self-Attention algorithm takes <em>Q (query), K (key), V (value)</em> as an input and is used to generate attention weights <em>(Wq, Wk, Wv)</em>, which are parameter matrices to be learned.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/539/1*BALnYDth3zF9s0hPydp3LQ.png" /><figcaption>Q, K, V creation schema</figcaption></figure><p>After finding <em>Q, K, V</em> we use the following equation that represents the multiplication of the attention weights and the <em>value</em> vector. This ensures that the words we want to focus on are kept as is and the irrelevant words are flushed out.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/1*CfDe2YV6D4RzCbwn21HGPQ.png" /><figcaption>Scaled dot product attention</figcaption></figure><p>The dot-product attention is scaled by a factor of square root of the depth. This is done because for large values of depth, the dot product grows large in magnitude pushing the softmax function where it has small gradients resulting in a very hard softmax.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*6bZrI_RzPNB-Eqy-y10sHA.png" /><figcaption>Step-by-Step sample attention layer output creation</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6b290fe2acbeb29a0af26eb8e96ea524/href">https://medium.com/media/6b290fe2acbeb29a0af26eb8e96ea524/href</a></iframe><p>Transformer uses <em>N</em> parallel attention layers (Multi-Head Attention) with a difference that they are scald by <em>head dimension = depth/N</em>. For each head, <em>Q, K, V</em> matrices are initialized randomly and modified independently by backpropagation during training. As a result, the embedding takes into consideration various contexts at the same time.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cf88e263c5de1ba9f9e8c32609d68477/href">https://medium.com/media/cf88e263c5de1ba9f9e8c32609d68477/href</a></iframe><h4>Transformer</h4><p>Since Encoder, Decoder and Multi-Head Attention are already defined, we are ready to combine everything and build Transformer!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/81f1730371dcdb4c0d10f102ecc0c84e/href">https://medium.com/media/81f1730371dcdb4c0d10f102ecc0c84e/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/862/1*krmQkN8Y3ZhPORfei88-_A.png" /><figcaption>Transformer Model Summary</figcaption></figure><h4>Inference</h4><p>For evalution:</p><ol><li>Apply the same preprocessing method we used to create our dataset for the input sentence.</li><li>Tokenize the input sentence and add padding.</li><li>Calculate the padding masks and the look ahead masks.</li><li>The decoder then outputs the predictions by looking at the encoder output and its own output.</li><li>Predict word based on probabilities (argmax) and pass it to the decoder.</li></ol><p>Note that in this approach, the decoder predicts the next word based on the previous words it predicted.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/644bb7db54eed2b07f177fc7d120c87e/href">https://medium.com/media/644bb7db54eed2b07f177fc7d120c87e/href</a></iframe><h3>Summary</h3><p>We have implemented Chatbot in TensorFlow 2.0 and understood Transformer architecture! Thank you for reading :).</p><h3>Don’t forget to give us your 👏 !</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/0*ZxaGcKC3UQEydQIX" /></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fupscri.be%2F3e59ef%3Fas_embed%3Dtrue&amp;dntp=1&amp;url=https%3A%2F%2Fupscri.be%2F3e59ef&amp;image=http%3A%2F%2Fapi.screenshotlayer.com%2Fapi%2Fcapture%3Faccess_key%3Dfe59908dad3baab69ffab249a2224b03%26viewport%3D1024x612%26width%3D1000%26url%3Dhttps%253A%252F%252Fupscri.be%252F3e59ef%253Fscreenshot&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=upscri" width="800" height="400" frameborder="0" scrolling="no"><a href="https://medium.com/media/7078d8ad19192c4c53d3bf199468e4ab/href">https://medium.com/media/7078d8ad19192c4c53d3bf199468e4ab/href</a></iframe><figure><a href="https://chatbotslife.com/bot-communities-mastermind-group-d2dae9876709#.53x0py6ou"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*6XUspT9JOSq0w0Fi35HIaA.png" /></a></figure><figure><a href="https://m.me/ChatbotsLife"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*c1LDMH5vbnIz9rmAka8Hwg.png" /></a></figure><figure><a href="https://chatbotslife.com/chatbot-development-hire-top-ai-chatbot-developers-c8b45ef7f207"><img alt="" src="https://cdn-images-1.medium.com/max/255/1*D0Jf3dI6ZThtqcfwDYY7mg.png" /></a></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc5efe3a1596" width="1" height="1" alt=""><hr><p><a href="https://blog.chatbotslife.com/replicate-your-friend-with-transformer-bc5efe3a1596">Replicate your friend with Transformer</a> was originally published in <a href="https://blog.chatbotslife.com/">Chatbots Life</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[BERT: Multilabel Text Classification]]></title>
            <link>https://medium.com/data-science/bert-multilabel-text-classification-a7f560db34e5?source=rss-dc134fe9f7c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/a7f560db34e5</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[bert]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Zuzanna Deutschman]]></dc:creator>
            <pubDate>Sun, 09 Feb 2020 22:52:29 GMT</pubDate>
            <atom:updated>2020-04-18T15:58:26.773Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>In <a href="https://towardsdatascience.com/multi-label-text-classification-5c505fdedca8">my previous article</a>, I introduced various machine learning methods that enable assigning a set of relevant genres for a single movie description (please visit the article for dataset). The best F1 score = 0.43 was obtained for Classifier Chain model. My idea to be verified is to train neural network with BERT embeddings.</p><p><a href="https://github.com/google-research/bert">BERT</a> (Bidirectional Encoder Representations from Transformers) is a recent model developed by researchers at Google AI Language.</p><p>It’s main innovation is applying the bidirectional training of Transformer, a popular attention model, to language modelling. This results in a deeper sense of language context and flow than single-direction language models.</p><h3>Code</h3><p><a href="https://pypi.org/project/bert-serving-server/">Bert_serving</a> enables using BERT model as a sentence encoding service for mapping a variable-length sentence to a fixed-length.</p><p>To find the best bunch of parameters I used <a href="https://github.com/IDSIA/sacred">sacred module</a>. Sacred is a tool to help you configure, organize, log and reproduce experiments in order to:</p><ul><li>keep track of all the parameters of your experiment</li><li>easily run your experiment for different settings</li><li>save configurations for individual runs in a database</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b31058fa06209dd8739a876d20163213/href">https://medium.com/media/b31058fa06209dd8739a876d20163213/href</a></iframe><p>Modified parameters were here:</p><ol><li><strong>max_seq_len</strong> — maximum length of sequence. The best found value is <strong>256 </strong>and it required GPU to be used</li><li><strong>batch_size</strong> — number of samples that will be propagated through the network. Chosen number is <strong>128</strong></li><li><strong>gamma</strong> — the focusing parameter in focal loss that smoothly adjusts the rate at which easy examples are down-weighted. The focal loss is designed to address class imbalance by down-weighting inliers (easy examples) such that their contribution to the total loss is small even if their number is large. It focuses on training a sparse set of hard examples. The most optimal value of gamma in our example is <strong>2</strong></li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fb17e236b4004ae4571a6c933fad023a/href">https://medium.com/media/fb17e236b4004ae4571a6c933fad023a/href</a></iframe><p>Obtained F1 = <strong>0.49</strong></p><h3>Labels co-occurrences</h3><p>We need to remember that there are multiple labels in our dataset and sometimes one label indicate occurrence of the other one. In order to check this percentage dependency, I created matrix of co-occurrences. Now we know that, for example:</p><pre>Action Comedy -&gt; Action, 0.9722222222222222<br>Action Comedy -&gt; Comedy, 0.9629629629629629<br>Action Thrillers -&gt; Action, 0.9492537313432836<br>Action Thrillers -&gt; Thriller, 0.9253731343283582<br>Adventure Comedy -&gt; Comedy, 0.9101123595505618</pre><p>The top 100 strongest dependencies represented on graph below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*VgL7xNbkAggCANIhGow6uw.png" /><figcaption>Route chord of labels cooccurrences</figcaption></figure><p>With gained knowledge, we can try to modify our predictions now. After few trials I decided to change value of resulting label for 1, only if its co-occurrence with indicating label was greater than or equal 0.9.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4f9b06ca3c632da2aa914f18193d284e/href">https://medium.com/media/4f9b06ca3c632da2aa914f18193d284e/href</a></iframe><p>Updated F1 score is <strong>0.5</strong>, which is small improvement.</p><h3>Results</h3><p>Sample results look as follows.</p><pre>First Platoon centers around Rock Brannigan ([[Scott Gibson  and his ragtag squad of ex-military zombie hunters trying to make a living in the desert Southwest two years after the zombie apocalypse. Along the way they encounter the grizzled Pa Jericho , and the eccentric Rex Necro .<br><strong>Action, Comedy, Horror, Parody, Science Fiction</strong></pre><pre>The life of the S&amp;M-theme artist and author Seiu Ito is depicted in the film. His artistic life and Sadian philosophy, inspired by his torturing of his two wives and Tae, his favorite prostitute, are portrayed as shown in his journalistic writings. Tae is eventually driven insane due to Ito&#39;s attentions.<br><strong>Drama, Japanese Movies, World cinema</strong></pre><h3>Conclusions</h3><p>Usage of BERT embeddings enabled to gain 7% of F1 score improvement, (50% overall).</p><p>In future work, I think that good idea would be to reduce number of labels keep only the main ones. As ‘Action Comedy’ is in 100% of cases ‘Action’ and in 100% ‘Comedy’, maybe we don’t really need this category.</p><p>Thank you for reading.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a7f560db34e5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/bert-multilabel-text-classification-a7f560db34e5">BERT: Multilabel Text Classification</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multi-Label Text Classification]]></title>
            <link>https://medium.com/data-science/multi-label-text-classification-5c505fdedca8?source=rss-dc134fe9f7c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c505fdedca8</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Zuzanna Deutschman]]></dc:creator>
            <pubDate>Tue, 03 Dec 2019 21:48:50 GMT</pubDate>
            <atom:updated>2021-05-30T14:47:55.099Z</atom:updated>
            <content:encoded><![CDATA[<h4>Assign labels to movies based on descriptions</h4><h3>Introduction</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*c8UlhrYNjf3sLDZ9" /><figcaption><a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>To understand various classification types, let’s consider the picture above.</p><pre>┌──────────────┬────────────────────┬──────────────────────┐<br>│ Is it a hen? │    Which label?    │    Which labels?     │<br>├──────────────┼────────────────────┼──────────────────────┤<br>│ <strong>Yes</strong>/No       │ harmony/<strong>hen</strong>/nature │<strong> harmony, hen, nature</strong> │<br>│ <strong>BINARY</strong>       │ <strong>MULTICLASS</strong>         │ <strong>MULTI-LABEL </strong>         │<br>└──────────────┴────────────────────┴──────────────────────┘</pre><p>In the case of <strong>binary classification</strong> we just ask a yes/no type of question. If there are multiple possible answers and only one to be chosen, then it’s <strong>multiclass classification</strong>. In our example we can’t really select only one label, I would say that all of them match the photo. The goal of <strong>multi-label classification</strong> is to assign a set of relevant labels for a single instance. However, most of widely known algorithms are designed for a single label classification problems. In this article four approaches for multi-label classification available in <em>scikit-multilearn</em> library are described and sample analysis is introduced.</p><h3>Multi-label classification methods</h3><ol><li><strong>Problem transformation methods</strong> divide the multi-label problem into one or more conventional single-label problems.</li><li><strong>Problem adaptation methods</strong> generalize single-label algorithms to cope with multi-labeled data directly.</li><li><strong>Ensemble methods</strong> incorporate the merits of two previous approaches</li></ol><p>In the next sections I will describe methods presented in below diagram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*5RGjPHsTJjTHS33cjjdoGg.png" /><figcaption>Fig. 1 Multi-label classification methods</figcaption></figure><h3>Binary Relevance</h3><p>In the case of Binary Relevance, an ensemble of single-label binary classifiers is trained independently on the original dataset to predict a membership to each class, as shown on the fig. 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*jZosaDviHiPvO7diltUMGw.png" /><figcaption>Fig. 2 Binary Relevance example</figcaption></figure><pre>┌─────────────────────┬─────────────────────────────────────────┐<br>│     <strong>Advantages  </strong>    │              <strong>Disadvantages </strong>             │<br>├─────────────────────┼─────────────────────────────────────────┤<br>│ - linear complexity │ - ignoring correlation between labels   │<br>│ - simplicity        │   by treating each label independently  │<br>└─────────────────────┴─────────────────────────────────────────┘</pre><h3>Label Powerset</h3><p>This approach maps each combination of labels into a single label and trains a single label classifier (fig. 3).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/411/1*2Epku4Rgpc755Ns7K5xyjQ.png" /><figcaption>Fig. 3 Label Powerset example</figcaption></figure><pre>┌────────────────┬───────────────────────────────────┐<br>│   <strong>Advantages </strong>  │           <strong>Disadvantages </strong>          │<br>├────────────────┼───────────────────────────────────┤<br>│ - taking label │ - high computational complexity   │<br>│   correlations │ - can lead to imbalanced dataset  │<br>│   into account │  with multiple classes associated │<br>│                │  with few examples                │<br>└────────────────┴───────────────────────────────────┘</pre><h3>MLkNN</h3><p>For each instance in the test set, its K nearest neighbours in the train set are identified. For each class y in Y the numbers of neighbouring instances belonging to y are used to compute the posterior probabilities that the test instance belongs or does not belong to y. Based on which of these probabilities are greater , we decide to assign or not the class y to a test instance.</p><p>For better understanding, assume that samples below are neighbours found out in the training set for test instance X = 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/218/1*QRYFCxYgkYzbg-xNa_Y2cA.png" /></figure><p>Now we have to calculate prior and posteriori probabilities for each class for test instance X = 1</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*9PATV_TYsMlVAqwh5OEOlA.png" /></figure><p>Maximal values were obtained for f1 and f2. Hence, we assign test instance to those two classes.</p><pre>┌─────────────────────────┬───────────────────────────────────────┐<br>│       <strong>Advantages </strong>       │            <strong> Disadvantages </strong>            │<br>├─────────────────────────┼───────────────────────────────────────┤<br>│ - better accuracy than  │ - with the binary relevance strategy, │<br>│   the previous methods  │   the ratio estimation                │<br>│ - correlations between  │   may not be accurate                 │<br>│   labels are considered │ - data distributions for some labels  │<br>│                         │   are imbalanced                      │<br>└─────────────────────────┴───────────────────────────────────────┘</pre><h3>Classifier Chain</h3><p>Classifier chain model learns classifiers as in binary relevance method. However, all classifiers are linked in a chain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/207/1*XaYsI2aq-zwej1WI3Df9HA.png" /><figcaption>Fig. 4 Classifier Chain rule</figcaption></figure><ol><li>Firstly, all the features (X1, X2, …, Xm) are used to predict y1.</li><li>Then, all the features (X1, X2, …, Xm, y1) are used to predict y2</li><li>Finally, (X1, X2, …, Xm, y1, y2) are applied to predict y3</li></ol><p>The order in which labels are predicted has large impact on the results.</p><pre>┌────────────────────────────┬──────────────────────────┐<br>│         <strong>Advantages </strong>        │      <strong>Disadvantages </strong>      │<br>├────────────────────────────┼──────────────────────────┤<br>│ - label correlation taken  │ - accuracy heavily       │<br>│   into consideration       │   depends on the order   │<br>│ - acceptable computational │ - for n labels there     │<br>│   complexity               │   are n! possible orders │<br>└────────────────────────────┴──────────────────────────┘</pre><h3>Dataset</h3><p>We will use the <a href="http://www.cs.cmu.edu/~ark/personas/">CMU Movie Summary Corpus</a> open dataset for our project. <strong>You can download the dataset directly from this </strong><a href="http://www.cs.cmu.edu/~ark/personas/data/MovieSummaries.tar.gz"><strong>link</strong></a><strong>.</strong></p><p>This dataset contains multiple files, but we’ll need only two of them:</p><ul><li><strong>movie.metadata.tsv:</strong> Metadata for 81,741 movies, extracted from the November 4, 2012 dump of Freebase. The movie genre tags are available in this file</li><li><strong>plot_summaries.txt:</strong> Plot summaries of 42,306 movies extracted from the November 2, 2012 dump of English-language Wikipedia. Each line contains the Wikipedia movie ID (which indexes into <em>movie.metadata.tsv</em>) followed by the plot summary</li></ul><h3>Code example</h3><p>We will create a table containing multiple genres assigned to movies with description. Based on that description we will try to assign proper genres to each movie.</p><p>Let’s start with importing necessary libraries. For described above methods we will use <em>skmultilearn</em> module.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/66b52544a36829e0c8c9b13d1a9e38e9/href">https://medium.com/media/66b52544a36829e0c8c9b13d1a9e38e9/href</a></iframe><p>Now it’s time for cleaning the text from special signs, removing stopwords, stamming and putting genre variable into separate binary columns.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d6c0c0af81b613b623841d338eb0b685/href">https://medium.com/media/d6c0c0af81b613b623841d338eb0b685/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/981/1*tKqp2K6J-y0mE40EzsOheA.png" /></figure><p>As the data is ready, we can split it into training and test sets and convert to a matrix of TF-IDF features. We will take top 10000 of them ordered by term frequency across the corpus (due to computational complexity). Please note that N-grams of maximal length 3 are considered here as potential features. As <strong>n_gram_range</strong> and <strong>max_features</strong> are essential parameters in terms of accuracy, I recommend you to play with them by your own :).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/10c2c3535c702ff5df97409b8800758e/href">https://medium.com/media/10c2c3535c702ff5df97409b8800758e/href</a></iframe><p>Let’s apply models described above and compare the results. For selecting the best model we will measure:</p><ol><li><strong>Accuracy</strong> — accuracy is the most common metric. However, it can be largely contributed by a large number of True Negatives which we have in this case and in the most of businesss cases.</li><li><strong>F1 score</strong> — F1 Score might be a better measure to use if we need to seek a balance between Precision and Recall and there is an uneven class distribution (large number of Actual Negatives) as in our example</li><li><strong>Hamming loss</strong> — the fraction of labels that are incorrectly predicted. Brings additional information in case of unbalanced datasets.</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b70faa438afa429217059678addcfbd8/href">https://medium.com/media/b70faa438afa429217059678addcfbd8/href</a></iframe><p>Accuracy = 0.004, F1 score = 0.222, Hamming loss = 0.033</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9063e1267c4e6472c4ab56e22f47d0a6/href">https://medium.com/media/9063e1267c4e6472c4ab56e22f47d0a6/href</a></iframe><p>Accuracy = 0.090, F1 score = 0.261, Hamming loss = 0.009</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7ea1e91dc38bdd2bd1c62ceee27abafe/href">https://medium.com/media/7ea1e91dc38bdd2bd1c62ceee27abafe/href</a></iframe><p>Accuracy = 0.052, F1 score = 0.235, Hamming loss = 0.01</p><p>As you see, accuracy is extremely low in all the cases since the dataset is strongly unbalanced. The best results were obtained for Label Powerset model. However, we have still Classifier Chain to be applied. I will predict here each class probability assignment and then play with <strong>threshold</strong> parameter a bit (by default it equals <strong>0.5</strong>). It says how to distinguish between 0/1 label assignment. You will see that its value is crucial for the model selection.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2845871e87d022e9e6113b724cc34948/href">https://medium.com/media/2845871e87d022e9e6113b724cc34948/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/1*FfY-QIn-VnX69RNrRlbb7w.png" /></figure><p>The plot shows that the most optimal threshold is around 0.2. This is the point where F1 score is near the maximum point 0.43, accuracy increases and Hamming loss is low.</p><h3>Conclusions</h3><p>To sum up, popular methods for multilabel classification were described and compared on real-life example. The best results were obtained for Classifier Chain method, by modyfing minimal threshold, based on which we assign 1 in binary classification problems. In future work, the order of labels could be shuffled and tested, which is time consuming process.</p><p>In the next article I will use deep learning methods for the same problems. This is the fastest-growing and the most significantt subfield of Natural Language Processing as allows to obtain the best results.</p><p>Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c505fdedca8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/multi-label-text-classification-5c505fdedca8">Multi-Label Text Classification</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>