<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vitalii Koren on Medium]]></title>
        <description><![CDATA[Stories by Vitalii Koren on Medium]]></description>
        <link>https://medium.com/@korenv20?source=rss-ca172c9ea8e2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Mzcn0biF32-qqwC3BgpxVA.jpeg</url>
            <title>Stories by Vitalii Koren on Medium</title>
            <link>https://medium.com/@korenv20?source=rss-ca172c9ea8e2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 06:23:35 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@korenv20/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Do we need to freeze embeddings when fine-tuning our LM?]]></title>
            <link>https://korenv20.medium.com/do-we-need-to-freeze-embeddings-when-fine-tuning-our-lm-c8bccf4ffeba?source=rss-ca172c9ea8e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/c8bccf4ffeba</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[bert]]></category>
            <category><![CDATA[roberta]]></category>
            <dc:creator><![CDATA[Vitalii Koren]]></dc:creator>
            <pubDate>Sun, 10 Jan 2021 14:53:47 GMT</pubDate>
            <atom:updated>2021-01-12T07:05:51.430Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GJdI2lCpPjmycNxl" /><figcaption><a href="https://www.magnoliabox.com/search?q=tag%3Aartist+Corbis">Author</a></figcaption></figure><h3>Intro</h3><p>Somewhen in somewhat video, I heard the idea of freezing the embedding matrix during the fine-tuning of a pre-trained language model. I completely forgot where it was said, but after that, usually do this. I decided to check if it’s needed and will share with you the results in this article. I found a similar experiment described in <a href="https://medium.com/commetric/semi-frozen-embeddings-for-nlp-transfer-learning-49fcfeec3f1b">this</a> post, but there were used GloVe vectors for initializing the embedding matrix of not pre-trained BiLSTM and there was no exploration of the difference between the validation and train vocabulary. Also found some papers: “<a href="https://arxiv.org/pdf/1903.05987.pdf">To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks</a>” and “<a href="https://arxiv.org/pdf/1911.03090.pdf">What Would Elsa Do? Freezing Layers During Transformer Fine-Tuning</a>” that explored the number of frozen layers but there was no analysis especially for embedding matrix.</p><h3>Idea</h3><p>The intuition behind this is that the embedding layer contains an independent representation of each token and the vectors for tokens with similar meanings should be learned to be close in the embeddings space during pre-training to be used in a pretty similar way by the further layers. But when you have not a good amount of labeled data that is common for real-world use cases, most probably the data on which you will do the inference will contain some tokens that were not presented in the train set. Probably some of these new tokens will have the synonym pairs in your labeled data, but gradient updates can destroy this type of connection learned by the language model.</p><h3>Data</h3><p>So, for the experiment, I chose the multilingual data from the current Kaggle competition “<a href="https://www.kaggle.com/c/contradictory-my-dear-watson/">Contradictory, My Dear Watson</a>”. It’s a natural language inferencing task where you have sentence pairs — premise and a hypothesis and you need to predict the relationship between them from entailment, neutral, and contradiction. This data consists of sentences in 15 languages that fit our experiment as it will be easy to get a validation set with a good amount of non-overlapped tokens. The model I chose for the experiment — XLM-RoBERTa that has a large vocabulary size — 250k which also suits our goals. The way I solved the task — simply concatenate the premise and hypothesis via a special token and input to the model as a single sequence.</p><p>First of all, I split the available 12k rows into the train (8120) and validation (4000) sets. Then I created 4 variations of the training dataset using train split: full, 6k, 4k, and 2k rows respectively. Checking the tokens overlap:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/464/1*bCgFUk9QDORURy9iZfPX3Q.png" /></figure><p>we can see that with decreasing the number of training data, the number of unseen tokens during the training increase.</p><h3>Implementation</h3><p>The implementation is based on the Transformers Library. For the optimization — I chose the <a href="https://arxiv.org/pdf/1908.03265.pdf">RAdam</a> as it does not require the warm-up tuning that depends on data size. I used a batch size of 8 with a maximum sequence length of 512, packing examples of similar lengths into one batch to speed up the training process. Accuracy was used as a metric. More details can be found on <a href="https://github.com/koren-v/freezed-embeddings-impact/">GitHub</a>.</p><h3>Results</h3><p>Finally, I did 8 runs (4 train sizes and 2 options: with frozen embedding (<em>model_fr)</em> and not (<em>model_unfr)</em>). I trained each model for the same number of gradients updates. So, the validation metric curves look following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*udNjVnAel5VIeFwoPJGtog.png" /></figure><p>From the plots above, we can see that there is no noticeable difference between the <em>model_fr </em>and the <em>model_unfr</em>. But in all cases up to the end of the training, the <em>model_fr </em>is a little better<em> </em>(for the experiment with 2k rows the <em>model_unfr</em> got the highest metric at 15th (from 40) epoch that is higher than the best metric of <em>model_fr</em>). Also, I logged the gradients per layer expected to get some insights, but for all experiment pairs, they look similar too. You can check the logs in the <a href="https://colab.research.google.com/drive/167cj0ahu5jtzpCpPvyD0oMdPMy2BzjN6?usp=sharing">colab notebook</a> on your own.</p><p>Then I decided to check the correlation between the number of new tokens [set(validation tokens) - set(training tokens)] and the difference between the best validation metrics [best validation metric of <em>model_fr</em> - best validation metric of <em>model_unfr</em>]:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/776/1*oJd76q9m8nMZ1uJzdAR64w.png" /></figure><p>So, as I mentioned above, the 3/4 experiments improve the validation accuracy but not too much. Also, I expected to see some line for the validation points from the middle (0.00) to up right that would indicate the positive correlation between a number of unseen during the training tokens in the validation set and the gap between a score of <em>model_fr</em> and score of <em>model_unfr</em>. Maybe, it’s a good idea to make more experiments as 8 looks like nothing to make some conclusions (having more results it would be possible to make some statistical test to measure the statistical significance), but even they take enough computing time. On the other hand, all experiments with no freezing take ~1.6 times more training time.</p><h3>Conclusion</h3><p>To sum up, I would say that it’s a good idea to freeze the embedding matrix when you fine-tune some pre-trained language models, especially with a large vocabulary. It won&#39;t give you superior improvements but you can try it for some Kaggle competition. If we are talking about the production models — this setup looks the same as with no freezing, but you can save your time for training that is pretty important.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c8bccf4ffeba" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Korean Tokenization & Lemmatization]]></title>
            <link>https://korenv20.medium.com/korean-tokenization-lemmatization-a741fc9939cc?source=rss-ca172c9ea8e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/a741fc9939cc</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[tokenization]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[text-processing]]></category>
            <category><![CDATA[lemmatization]]></category>
            <dc:creator><![CDATA[Vitalii Koren]]></dc:creator>
            <pubDate>Sat, 26 Dec 2020 16:00:23 GMT</pubDate>
            <atom:updated>2020-12-26T16:00:23.975Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Note: If you have the labeled data and you are not planning to operate directly with words, probably your solution </em>could be a<em> pre-trained network </em>such as<em> </em><a href="https://huggingface.co/monologg/kobert"><em>KoBERT</em></a><em> or even fancier — </em><a href="https://huggingface.co/monologg/koelectra-base-v3-discriminator"><em>KoELECTRA</em></a><em>. If you are searching for some “lighter” approaches, you can build your vocabulary using something from HuggingFace’s </em><a href="https://huggingface.co/docs/tokenizers/python/latest/"><em>tokenizers library</em></a><em> and train what you want.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RiuhivXysP-tcnw-.jpg" /><figcaption><a href="https://www.cntraveler.com/gallery/the-most-beautiful-places-in-south-korea">Source</a></figcaption></figure><h3>Intro</h3><p>A few weeks ago I was faced with processing a few Asian languages: Chinese, Japanese, and Korean. To work with them properly I had first to understand their structure (how words are formed and how further they form the sentence). While searching for the answers to my questions I’ve found out two things:</p><ol><li>At first sight, they resemble each other and are different from most European languages but it’s not quite true.</li><li>There are a lot of tools that can be easily found for Chinese and Japanese, but not for Korean.</li></ol><p>Because of the last one, I decided to briefly share with you the knowledge I’ve acquired. I am grateful to people who speak the mentioned languages and could answer some of my questions. Also, If you want to add or rectify something, you are welcome! So, let’s start.</p><h3>Important language points:</h3><ul><li>Korean has an <strong>alphabet </strong>instead of hieroglyphs in writing</li><li>Unlike Chinese and Japanese, Korean uses <strong>whitespaces</strong> to separate words!)</li><li>The pieces separated by spaces — <strong>eojeols</strong>, eojeols consists of <strong>syllables</strong></li><li>Eojeols are formed of <strong>morphemes</strong>, usually, there is a “root” and other morphemes modify it (like prefixes and suffixes modify English words to create verb or adjective). For example, 결정하겠다 — “I will decide” consist of: “결정”, “하”, “겠”, “다”. The first morpheme (“결정”) is a root, that is translated as “decision”. Because of that, <strong>it’s probably the wrong solution to simply divide the sentence by whitespaces and leave tokens as it is* </strong><em>(the example above was taken from really useful </em><a href="https://www.youtube.com/watch?v=kw_EEfJ57WQ"><em>video</em></a><em>, check it out)</em></li><li>The “root” morphemes can have their normalized form (aka. <strong>lemma</strong>)</li></ul><p><em>*According to my experience, be aware that sometimes eojeol is split too much, and in my case, some eojeols were found in word-graph without this kind of pre-processing</em></p><h3>Open-source tools:</h3><p>Taking into consideration the stated above information, it is required to do at least 3 steps: split eojeols into morphemes, mark each morpheme to identify the root, and finally normalize the root. Quick googling gives usually one library that can solve these tasks — <a href="https://konlpy.org/en/latest/"><strong>KoNLPy</strong></a>. It utilizes several morpheme analyzer tools such as JHannanum, Kkma, KOMORAN, MeCab-ko, Open Korean Text. Most of them use Hidden Markov Models or only the Viterbi Algorithm. If you are interested in these models— there is one great <a href="https://towardsdatascience.com/how-japanese-tokenizers-work-87ab6b256984">article</a> that explains how MeCab-like libraries work (originally it was developed for Japanese). I haven`t found any information on how accurate each of them is, so I delved deeper to find something fresher.</p><p>Luckily, I have stumbled upon a paper <a href="https://www.aclweb.org/anthology/D19-1150.pdf"><strong>Korean Morphological Analysis with Tied Sequence-to-Sequence Multi-Task Model</strong></a> [2019] that uses LSTM with a pointer-generator network and CRF on a top. Their approach showed SOTA for the mentioned tasks at that moment. Talking about the reason to use seq2seq architecture — the decoder generated the base forms of each morpheme. So the model does all at once. Another good point of this paper is that they have an open <a href="https://github.com/songhyunje/kma">repository</a> (working) and even link to the pre-trained weights 🙌. You can easily adapt their inference code to your needs (the repo is written using PyTorch).</p><p>The paper references one more open tool <a href="https://github.com/kakao/khaiii"><strong>khaiii</strong></a>, which excludes using RNN architectures motivated by speed and uses CNN, besides, the part of the repository is written in C++, so probably it has to be faster, but a little less accurate. I didn’t try to launch it, BUT there is an important note — khaiii doesn’t provide any English documentation which is a really common problem for Asian NLP repositories.</p><p>There is one more neural-based solution, that is just tokenizer — <a href="https://github.com/haven-jeon/PyKoSpacing"><strong>PyKoSpacing</strong></a>. It’s a little out-dated compared to previous repositories and has pretty interesting architecture, but maybe you would like Keras more)</p><h3>Instead of summary</h3><p>If you happen to be a contributor to some open-source package, doesn&#39;t matter if it’s about NLP or not, duplicate the readme or documentation in English too. It will definitely help the community to grow faster 😉</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a741fc9939cc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Interpretation of HuggingFase’s model decision]]></title>
            <link>https://korenv20.medium.com/interpretation-of-huggingfases-model-decision-9a4100b2fed7?source=rss-ca172c9ea8e2------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a4100b2fed7</guid>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[hugging-face]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Vitalii Koren]]></dc:creator>
            <pubDate>Sun, 05 Jul 2020 12:59:13 GMT</pubDate>
            <atom:updated>2020-08-14T10:29:57.035Z</atom:updated>
            <content:encoded><![CDATA[<h3>Interpretation of HuggingFace’s model decision</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*88ThZDKQpP0zwPwoDlBZ2A.jpeg" /></figure><p><a href="https://arxiv.org/pdf/1706.03762.pdf"><em>Transformer-based</em></a> models have taken a leading role in NLP today. In most cases using pre-trained encoder architectures in solving downstream tasks achieves super high scores. The main idea of ​​this approach is to train the large model on a big amount of unlabeled data and then add few layers to the top of it for text classification, coreference resolution, question answering, and so on. However, although such models give cool results, those models are still black boxes, whereas the interpretability of a model is very important for debugging and understanding how the model makes a decision.</p><p>A couple of weeks ago I faced a <a href="https://demo.allennlp.org/sentiment-analysis"><em>demo</em></a> from Allen NLP. I found a cool feature there that highlighted the words that impacted on the model’s decision. Then I wanted to do this for my models, but after spending some time looking for a tutorial on how to put a PyTorch model into the Allen NLP Predictor, I haven’t found anything useful. Then I decided to rewrite the interpreter to PyTorch and it was not difficult because Allen NLP is built on it, so some parts of the code were taken from <a href="https://github.com/allenai/allennlp/tree/master/allennlp/interpret"><em>here</em></a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/461/1*n5xL1jJ9q_zeH3jealadXQ.png" /></figure><p>I will consider mostly on pieces of code that will help you integrate your own model. Also, I will omit the explanation of some common and clear parts to not make this article too long. The full code you can find on my <a href="https://github.com/koren-v/Interpret"><em>GitHub repository</em></a>.</p><p><strong>Some updates:</strong></p><ul><li>Have been added batching support.</li><li>There is no need to make dataset instance that will return tokens (just usual dataset, that returns token_ids and so on)</li><li>The repository was modified for simplification the integrating of your model not depending on the task it was made for (classification, token classification etc.)</li><li>To avoid redundant computations I decided to exclude making predictions separately. It impacts on the accuracy of predictions (just on few points) as Smooth Grad adds the Gaussian noise and Integrated Gradients doesn’t include the upper limit of the integral (<a href="https://github.com/koren-v/Interpret/blob/master/integrated_gradient.py#L64">see the repo</a>)</li><li>Also, empirically I found that the <a href="https://arxiv.org/pdf/1703.01365.pdf">Integrated Gradients algorithm</a> highlights more “<a href="https://nbviewer.jupyter.org/github/koren-v/Interpret/blob/master/usage_example.ipynb">expected</a>” words that are more relevant to class the model chose.</li></ul><p>There are 3 gradient-based algorithms that are covered by Allen NLP Interpret. I will choose Smooth Grad and a classification task as an example. The idea of this algorithm is simple: we first make a prediction, take argmax (as we usually do in classification), and assuming it as ground truth — make backpropagation. Then we sum the gradients for each embedding but as was shown in <a href="https://arxiv.org/pdf/1706.03825.pdf"><em>the paper</em></a> the values can have outliers so we calculate gradients few times adding Gaussian noise and then take the average. Let’s start from the first step — making a prediction:</p><pre>def saliency_interpret(self, test_dataset):<br><br>    # Convert inputs to labeled instances<br>    predictions = self._get_prediction(test_dataset)</pre><pre>...</pre><p>In my case, test_dataset<em> </em>is the instance of the class that I inherited from PyTorch’s Dataset class. I use it to build the dataloader to make predictions. If you are feed examples to the model in another way, you can rewrite this method:</p><pre>def _get_prediction(self, test_dataset, batch_size=1):</pre><pre>    test_dataloader = DataLoader(test_dataset,<br>                                 batch_size=batch_size,<br>                                 shuffle=False,<br>                                 num_workers=0)<br>    model_inputs = []<br>    input_tokens = []<br>    predictions = torch.tensor([], dtype=torch.float)<br>    model = self.model.to(self.device)<br>    model.eval()</pre><pre>    for inputs, tokens in test_dataloader:<br>        # collecting inputs, as they will be used in _get_gradients<br>        # and tokens to correspond them method output<br>        model_inputs.append(inputs)<br>        input_tokens.append(tokens)<br>        input_ids = inputs.get(&#39;input_ids&#39;)<br>        attention_mask = inputs.get(&quot;attention_mask&quot;)</pre><pre>        input_ids = input_ids.to(self.device)<br>        attention_mask = attention_mask.to(self.device)</pre><pre>        with torch.no_grad():<br>            outputs = model(<br>                input_ids=input_ids,<br>                attention_mask=attention_mask<br>            )</pre><pre>        predictions = torch.cat((<br>            predictions,<br>            softmax(outputs, dim=-1)<br>        ))</pre><pre>    return predictions, model_inputs, input_tokens</pre><p>The model’s outputs are logits, so we are applying the softmax to get probabilities from it. I also collect model_inputs (tokens ids) that will be used in the next steps as well as input_tokens<em> </em>(tokenized text)<em> </em>that are returned by the dataloader. The reason why I save tokens is that the transformer models use special algorithms (such as <a href="https://en.wikipedia.org/wiki/Byte_pair_encoding">bpe</a>) to reduce the vocabulary size and the tokenizers split some words to word pieces, so not always one word is equal to one token. For example:</p><pre>&gt;&gt;&gt; tokenizer.tokenize(&#39;transformer model&#39;)<br>[&#39;transform&#39;, &#39;##er&#39;, &#39;model&#39;]</pre><p>Then we apply the Smooth Grad for each example (didn’t investigate how to do it by batches):</p><pre>...</pre><pre>predictions = self._get_prediction(test_dataset)</pre><pre>    instances_with_grads = dict()<br>    for idx, (prob, inp, tokens) in enumerate(zip(*predictions)):<br>        # Run smoothgrad<br>        label = torch.argmax(prob, axis=0)<br>        grads = self._smooth_grads(label, inp)</pre><pre>...</pre><p>Where<em> </em>_smoth_grads method implements the described algorithm above:</p><pre>def _smooth_grads(self, label, inp):<br>    total_gradients = {}<br>    for _ in range(self.num_samples):<br>        handle = self._register_forward_hook(self.stdev)<br>        grads = self._get_gradients(label, inp)<br>        handle.remove()</pre><pre>        # Sum gradients<br>        if total_gradients == {}:<br>            total_gradients = grads<br>        else:<br>            for key in grads.keys():<br>                total_gradients[key] += grads[key]</pre><pre>    # Average the gradients<br>    for key in total_gradients.keys():<br>        total_gradients[key] /= self.num_samples</pre><pre>    return total_gradients</pre><p>There are some important points in _register_forward_hook and _get_gradients<em>.</em> In the first one, we have to define the embedding layer. Allen NLP has a specific <a href="https://github.com/allenai/allennlp/blob/60deece9fca2da6b66bfcde44484384bdefa3fe7/allennlp/nn/util.py#L1650"><em>method</em></a> for it (it covers GPT and Bert models), but I decided to define the layer directly via keyword arguments or bert attribute by default:</p><pre>...</pre><pre>encoder = self.kwargs.get(&quot;encoder&quot;)<br>if encoder:<br>    embedding_layer = self.model.__getattr__(encoder).embeddings<br>else:<br>    embedding_layer = self.model.bert.embeddings</pre><pre>...</pre><p>While my model’s class looks:</p><pre>class DistilBertForSequenceClassification(nn.Module):</pre><pre>    def __init__(self, config, num_labels=2):<br>        super(DistilBertForSequenceClassification, self).__init__()<br>        self.num_labels = num_labels<br>        self.config = config<br>        self.bert = DistilBertModel.from_pretrained(<br>            &#39;distilbert-base-uncased&#39;, <br>            output_hidden_states=False<br>        )<br>        self.dropout = nn.Dropout(config.dropout)<br>        self.classifier = nn.Linear(config.hidden_size, num_labels)<br>        nn.init.xavier_normal_(self.classifier.weight)</pre><pre>    def forward(<br>            self, <br>            input_ids, <br>            token_type_ids=None, <br>            attention_mask=None<br>    ):<br>        last_hidden = self.bert(<br>            input_ids=input_ids, <br>            attention_mask=attention_mask<br>        )<br>        pooled_output = torch.mean(last_hidden[0], dim=1)<br>        pooled_output = self.dropout(pooled_output)<br>        logits = self.classifier(pooled_output)<br>        return logits</pre><p>In the second one, we need to do the same as we do during the training. In my case, I feed to the model input_ids and attention_mask to mask the paddings:</p><pre>...</pre><pre>embedding_gradients = []<br>hooks = self._register_embedding_gradient_hooks(embedding_gradients)</pre><pre>input_ids = inp.get(&#39;input_ids&#39;)<br>attention_mask = inp.get(&quot;attention_mask&quot;)</pre><pre>outputs = self.model(input_ids=input_ids, attention_mask=attention_mask)<br>batch_losses = self.criterion(outputs, label.unsqueeze(0))<br>loss = torch.mean(batch_losses)</pre><pre>self.model.zero_grad()<br>loss.backward()</pre><pre>...</pre><p>In _register_embedding_gradient_hooks we also need to define the embedding as we did before.</p><p>After normalizing the gradients (to make the sum = 1) we are saving gradients, tokens, predicted label and the probability of it to the dictionary:</p><pre>...</pre><pre>instances_with_grads[&quot;instance_&quot; + str(idx + 1)] = grads<br>instances_with_grads[&quot;instance_&quot; + str(idx + 1)][&#39;tokens_input_1&#39;] = [t[0] for t in tokens]<br>instances_with_grads[&quot;instance_&quot; + str(idx + 1)][&#39;label_input_1&#39;] = label.item()<br>instances_with_grads[&quot;instance_&quot; + str(idx + 1)][&#39;prob_input_1&#39;] = prob.max().item()</pre><pre>...</pre><p>As a result, we get the dictionary with tokens and their weights in the sentence. I add a util method to make a visualization. Let’s have a look at the example of the output.</p><p>I trained DistilBert on the subset of <a href="https://www.kaggle.com/nulldata/medium-post-titles"><em>Medium Post Titles Dataset</em></a> with 93 classes which correspond to categories of the articles. The dataset consists of titles and subtitles, so let’s try to test the model on the title of this article plus the first sentence as the subtitle (the example of usage you can find in <a href="https://nbviewer.jupyter.org/github/koren-v/Interpret/blob/master/usage_example.ipynb"><em>this notebook</em></a>):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*w0f8xVbGBZHF7U04OINrVw.png" /></figure><p>As we can see, the most impact on the model’s decision makes [CLS] token, which is used for Next Sentence Prediction Objective during pre-training. The embedding of this token is fed to the linear layer with softmax activation to train the model to predict if the sentences separated by [SEP] token are from the same document or not. And the interesting point is that I didn’t use common practice when this token is used for classification (I used mean of the output embeddings). Also, we can emphasize the model gives the right prediction.</p><p><strong>To sum up</strong>, I think that these methods a great tool for analyzing your NLP models and can help you to find their weaknesses. There is one more amazing tool — <a href="https://github.com/jessevig/bertviz"><em>BertViz</em></a> that gives the ability to look inside the model, so I suggest to investigate it if you are interested in exploring your models.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a4100b2fed7" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>