<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Orb Intelligence Engineering &amp; Data Science - Medium]]></title>
        <description><![CDATA[Company information for B2B marketing and sales: http://orb-intelligence.com - Medium]]></description>
        <link>https://medium.com/orb-engineering?source=rss----bbdbbc63b960---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Orb Intelligence Engineering &amp;amp; Data Science - Medium</title>
            <link>https://medium.com/orb-engineering?source=rss----bbdbbc63b960---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 05:58:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/orb-engineering" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[NAACL ’19 Notes: Practical Insights for Natural Language Processing Applications — Part II]]></title>
            <link>https://medium.com/orb-engineering/naacl-19-notes-practical-insights-for-natural-language-processing-applications-part-ii-2a2a3dd42d1?source=rss----bbdbbc63b960---4</link>
            <guid isPermaLink="false">https://medium.com/p/2a2a3dd42d1</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Nikita Zhiltsov]]></dc:creator>
            <pubDate>Wed, 31 Jul 2019 17:04:33 GMT</pubDate>
            <atom:updated>2019-07-31T17:04:33.341Z</atom:updated>
            <content:encoded><![CDATA[<h3>NAACL ’19 Notes: Practical Insights for Natural Language Processing Applications — Part II</h3><p>Continuing on the <a href="https://medium.com/orb-engineering/naacl-19-notes-practical-insights-for-natural-language-processing-applications-part-i-5f981c92af80">Part I</a> of this blog post, we survey recent advances in some important NLP tasks, such as text similarity, text classification, sequence labeling, and language generation.</p><h3>Text Similarity</h3><p>A NAACL ’19 paper <a href="https://www.aclweb.org/anthology/N19-1100">“Correlation Coefficients and Semantic Textual Similarity”</a> [<a href="https://github.com/Babylonpartners/corrsim">code</a>] questions the usage of <em>cosine similarity</em> in the word embedding space. The core idea is to consider a word or a sentence embedding as a sample of <em>N</em> observations of some scalar random variable, where <em>N</em> is the embedding size. Then, some classical statistical correlation measures can be applied for pairs of vectors. As their empirical analysis has shown, cosine similarity is equivalent to Pearson’s (linear) correlation coefficient for commonly used word embeddings (<a href="https://nlp.stanford.edu/projects/glove/">GloVe</a>, <a href="https://fasttext.cc/">FastText</a>, <a href="https://code.google.com/archive/p/word2vec/">word2vec</a>). It comes from the fact that the values observed in practice are distributed around the zero mean. In the scenario of word similarity, a violation of the normality assumption makes cosine similarity especially inappropriate for GloVe vectors. For FastText and word2vec, the results of the Pearson coefficient and rank correlation coefficients (Spearman, Kendall) are comparable. However, the choice of cosine similarity is suboptimal for sentence vectors as centroids of word vectors (a widely used baseline for sentence representation), even for FastText. It is caused by stop word vectors behaving as outliers. The rank correlation measures are empirically preferable in this case.</p><h3>Text Classification</h3><h4>Document classification</h4><p><a href="https://www.aclweb.org/anthology/N19-1408">“Rethinking Complex Neural Network Architectures for Document Classification”</a> and its <a href="https://arxiv.org/pdf/1904.08398v1.pdf">follow-up paper </a>compare state-of-the-art (SOTA) document classification models on four accessible datasets (Reuters, Arxiv APD, IMDB, Yelp). The PyTorch implementations of the considered models are available in the authors’ framework <a href="http://hedwig.ca/">Hedwig</a>. While the fine-tuned BERT classifier shows the best results as expected, other findings are somewhat surprising. The second-best model is a simple <a href="https://github.com/castorini/hedwig/tree/master/models/reg_lstm">bi-LSTM classifier</a>, properly regularized and enhanced with max-pooling to get a document feature vector. It leaves behind some complex hierarchical architectures, such as <a href="https://github.com/castorini/hedwig/tree/master/models/han">hierarchical attention networks</a> (HAN) or <a href="https://github.com/castorini/hedwig/tree/master/models/xml_cnn">XML-CNN</a>, questioning the need for such complexity for this task. Moreover, on the datasets (Reuters, Arxiv) with a large number of classes and relatively scarce examples, the last two models are outperformed even by standard one-vs-rest logistic regression and SVM trained on TF-IDF vectors.</p><p><a href="https://www.aclweb.org/anthology/N19-1316">“Mitigating Uncertainty in Document Classification”</a> [<a href="https://github.com/xuczhang/UncertainDC">code</a>] proposes <em>metric learning</em> on feature representations and a dropout-based method to measure uncertainty of a deep learning model for text classification (with possible application in high-accuracy use cases, such as the medical domain). The classifier architecture is pretty standard: a convolutional neural network (CNN) over trainable word embeddings (initialized with GloVe vectors) followed by a dropout and a fully-connected layer and a softmax layer. Metric learning is used to train word embeddings in a way that minimizes the intra-class Euclidean distance and maximizes the inter-class Euclidean distance. Given <em>Sₖ</em> is a set of instances of <em>k</em>-th class, <strong><em>r</em></strong><em>ᵢ, </em><strong><em>r</em></strong><em>ⱼ</em> — feature representations of <em>i</em> and <em>j</em> instances, <em>D</em> — Euclidean distance:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/914/1*yg-bw9gHjzIYT4oTUX6R7g.png" /></figure><p>The incorporation of metric learning can diminish the prediction variance and increase the confidence of the accurate predictions.</p><p>The dropout-based method measures the model uncertainty in terms of the information entropy of multiple dropout evaluations combined with the denoising mask operations. The output vector of predicted classes <strong><em>y</em></strong><em>* = (y*₁,…,y*ₖ)</em> is obtained by applying dropout after CNN <em>k</em> times (<em>k = 100</em> in the experiments). The entropy of this class distribution (after masking 1/3 of the most underrepresented classes to reduce the noise) is calculated as an uncertainty score. We note that the <em>variational dropout method</em> still raises heated theoretical discussions (e.g., see <a href="https://www.reddit.com/r/MachineLearning/comments/7bm4b2/d_what_is_the_current_state_of_dropout_as/">this thread</a> on Reddit; thanks to <a href="https://blog.inten.to/@moocaholic">Grigory Sapunov</a> for pointing this out). Nonetheless, the paper authors have shown that the approach boosted the macro-F1 score from 78% to 92% by assigning 25% of the labeling work to human experts in a 20-class text classification task.</p><h4>Multi-label classification</h4><p><a href="https://www.aclweb.org/anthology/N19-1289">“Ranking-Based Autoencoder for Extreme Multi-label Classification”</a> proposes a novel principled approach to <em>extreme multi-label text classification</em>, i.e., multi-label text classification with a massive number of labels. This task has many real-world applications. For example, we at <a href="https://medium.com/orb-engineering">Orb Intelligence</a> are doing NAICS industry classification (over 2,200 hierarchical classes) of company text descriptions. This task is also characterized by semantic relationships between labels (classes are not exclusive), class imbalance, and label incompleteness.</p><p>The authors developed a new deep learning method <em>Rank-AE</em> that includes (Figure 1):</p><ul><li><em>a self-attention mechanism</em> to learn rich representations of input texts;</li><li><em>an auto-encoder</em> to project both features and labels onto the common latent space wherein correlations between features and labels are exploited. The auto-encoder then reproduces labels by decoding;</li><li><em>a margin-based ranking loss</em> that is more effective for extreme classification settings and more robust than noisy labeling.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*i2expJvcNWMqDxVXKthDjA.png" /><figcaption>Figure 1: Rank-AE (taken from <a href="https://www.aclweb.org/anthology/N19-1289">“Ranking-Based Autoencoder for Extreme Multi-label Classification”</a>)</figcaption></figure><p><strong>ℒ ₕ</strong><em>(xₕ,yₕ)</em> is chosen as the mean squared loss. The architecture is capable of capturing inter-dependencies between labels during training. During inference, the label encoder ℇ is ignored. The reconstruction loss <strong><em>ℒₐₑ</em></strong><em>(y,y’)</em> is a combination of two margin ranking losses, for positive and negative labels:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*WmmeYLuCmxBoVCR80vVU5w.png" /></figure><p>The exploited attention mechanism is dual (Figure 2).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/1*ra34Hzser59uM5eSaXExeQ.png" /><figcaption>Figure 2: Dual attention in Rank-AE</figcaption></figure><p>First, it weighs word embeddings in the text with TF-IDF. Second, the channel attention is designed to weigh different bits in a word embedding (assuming that some of them emphasize, say, the commercial sense of the term “apple” and the other do that for the agricultural one). The channel attention is implemented as an <a href="https://arxiv.org/pdf/1709.01507.pdf">excitation network</a> (two fully-connected layers with non-linear activations). These kind of networks have been previously utilized only for images. After applying these two attention mechanisms to the embedding matrix, average pooling is used to get a feature embedding x’. The conducted ablation study shows that Rank-AE benefits from the margin-ranking loss on noisy datasets and the attention on complex multi-aspect texts. The post-analysis of attention weights provided in the paper is instructive to explain which text spans contributed to the predicted label.</p><h4>Zero-shot classification</h4><p><a href="https://www.aclweb.org/anthology/N19-1108">“Integrating Semantic Knowledge to Tackle Zero-Shot Text Classification”</a> [<a href="https://github.com/JingqingZ/KG4ZeroShotText">code</a>] offers a principled approach to <em>zero-shot text classification</em>, i.e., prediction of classes not represented in the training data. For this scenario, although we assume that we have at least their names, and, presumably, short descriptions, inter-class taxonomic or even semantic relationships. The approach is two-staged (Figure 3).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V4ZvVHe-hlvKAhCjH5ALbQ.png" /><figcaption>Figure 3: Zero-shot text classification (taken from <a href="https://www.aclweb.org/anthology/N19-1108">“Integrating Semantic Knowledge to Tackle Zero-Shot Text Classification”</a>)</figcaption></figure><p>The first phase, coarse-grained classification, predicts if an input document comes from seen or unseen classes. The multi-class classification is broken down to multiple one-vs-rest classification tasks. The authors apply a data augmentation technique to help the classifiers be aware of the existence of unseen classes without accessing their labeled data. Then the second phase, fine-grained classification, finally specifies the class of the input document. It uses either a) a traditional multi-class classifier trained on examples of seen classes only, or b) a zero-shot binary classifier, depending on the coarse-grained prediction given by the first phase. Given a feature vector <em>xᵢ </em>and a class name vector <em>c</em>, the zero-shot classifier takes <em>(xᵢ, c)</em> pairs as input and learns to predict the confidence <em>p(ŷᵢ = c|xᵢ)</em>. Feature augmentation based on semantic knowledge is used to provide additional information which relates the document and the unseen classes to generalize the zero-shot reasoning. More details on data augmentation and feature augmentation used:</p><ol><li><em>topic translation</em>: translating a document word-by-word from its first seen class (represented as a word vector of the class name <em>c</em>) to a new unseen class <em>c’</em> using a word analogy (via <a href="https://aclweb.org/anthology/W14-1618">3COSMUL method</a>):</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*6tKxqa0J2-MBFw1B22GzVg.png" /></figure><p>and preserving the part-of-speech role of the translated word <em>w</em> (nouns ⇒ nouns, verbs ⇒ verbs etc). The translated documents are used to train a zero-shot classifier for a given unseen class. These documents are used as negative examples for learning binary classifiers of seen classes too.</p><p>2. <em>feature augmentation</em>: the embedding of each word <em>wⱼ</em> enhanced with 2 vectors:</p><p>(a) <strong><em>ũ</em></strong><em>(wⱼ,c)</em> is a relationship vector that shows how the word <em>wⱼ</em> and the class <em>c</em> are related considering the relations in a general knowledge graph, such as <a href="http://conceptnet.io/">ConceptNet</a>.</p><p>(b) <strong><em>ṽ</em></strong><em>(c)</em> is a word embedding of the class name <em>c</em>.</p><p>The experiments on the DBpedia ontology dataset and 20 News groups dataset have shown that data augmentation by topic translation improved the accuracy in detecting instances from unseen classes. Moreover, feature augmentation enables knowledge transfer from seen to unseen classes for zero-shot learning. The method has achieved the highest overall accuracy in each phase and overall compared with the competitive baselines.</p><h3>Sequence Labeling</h3><p><a href="http://www.aclweb.org/anthology/N19-1078">“Pooled Contextualized Embeddings for Named Entity Recognition” </a>by Zalando Research takes advantage of character-level LSTM-based contextual embeddings pooled (with min/max/average) across all sentence contexts in a large corpus (Figure 4).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wNn445JxaQ1gpJq-4nudZw.png" /><figcaption>Figure 4: A contextual string representation embedding is obtained for the given word ``Indra’’ and is concatenated with the pooled aggregation of all the embeddings that were produced for this string on previous sentences in the whole corpus (taken from <a href="http://www.aclweb.org/anthology/N19-1078">“Pooled Contextualized Embeddings for Named Entity Recognition”</a>)</figcaption></figure><p>Such global representation features two charming properties:</p><p>• <em>pre-training:</em> the improved representation of rare words in underspecified contexts — it benefits from “memorizing” the representation of words (or entities) in other, presumably richer, contexts;</p><p>• <em>downstream task training:</em> evolution of word representations as more instances of the same word observed in the downstream task data.</p><p>The final string embeddings are formed by concatenating the original contextual embedding and the pooled representation and are also enhanced with standard GloVe or FastText word embeddings. The experiments have shown that the pooled contextualized embeddings boost the performance of BiLSTM-CRF tagger for multilingual named entity recognition (NER) to the new SOTA (outperforming even <a href="https://github.com/kyzhouhzau/BERT-NER">BERT-NER</a>!). The model implementation is available in the <a href="https://github.com/zalandoresearch/flair">Flair framework</a>.</p><h3>Language Generation</h3><p><a href="https://arxiv.org/pdf/1903.09722.pdf">“Pre-trained language model representations for language generation”</a> [<a href="https://github.com/pytorch/fairseq/tree/bi_trans_lm/examples/pretraining">code</a>] by Facebook AI Research explores the different strategies of incorporating pre-trained vectors in <em>seq2seq</em> (encoder-decoder) architecture with applications in machine translation and abstractive summarization. Both the encoder and decoder are implemented as transformers in the <a href="https://github.com/pytorch/fairseq">Fairseq framework</a>. The considered strategies include:</p><p>• adding contextualized word embeddings (ELMo) as input to the encoder or the decoder;</p><p>• fine-tuning: replacing the learned word embeddings in the encoder (or, separately, in the decoder) with the LM representation of the layer before softmax.</p><p>The experiments show that adding pre-trained representations is very effective for the encoder network (at the expense of 5x training slowdown and only 12–14% inference slowdown, though) in both setups. Interestingly, the quality improvements diminish when more labeled data becomes available, which is in line with the sample efficiency of pre-training discussed in the <a href="https://medium.com/orb-engineering/naacl-19-notes-practical-insights-for-natural-language-processing-applications-part-i-5f981c92af80">Part I</a>.</p><p><em>This concludes the Part II. In the final Part III, we’ll give an overview of frameworks and miscellaneous effective techniques (attention and self-attention, model visualization and interpretation, adversarial learning, knowledge distilling, multi-modal learning).</em></p><p><a href="https://medium.com/orb-engineering?source=post_page---------------------------"><em>Follow us</em></a><em> on Medium to stay in touch. Don’t hesitate to share your experience or ask questions in the comment section.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2a2a3dd42d1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/orb-engineering/naacl-19-notes-practical-insights-for-natural-language-processing-applications-part-ii-2a2a3dd42d1">NAACL ’19 Notes: Practical Insights for Natural Language Processing Applications — Part II</a> was originally published in <a href="https://medium.com/orb-engineering">Orb Intelligence Engineering &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NAACL ’19 Notes: Practical Insights for Natural Language Processing Applications — Part I]]></title>
            <link>https://medium.com/orb-engineering/naacl-19-notes-practical-insights-for-natural-language-processing-applications-part-i-5f981c92af80?source=rss----bbdbbc63b960---4</link>
            <guid isPermaLink="false">https://medium.com/p/5f981c92af80</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Nikita Zhiltsov]]></dc:creator>
            <pubDate>Wed, 24 Jul 2019 17:11:51 GMT</pubDate>
            <atom:updated>2019-07-31T18:05:44.665Z</atom:updated>
            <content:encoded><![CDATA[<h3>NAACL ’19 Notes: Practical Insights for Natural Language Processing Applications — Part I</h3><p>The 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL- HLT) was held in Minneapolis, USA June 2–7, 2019. <a href="https://naacl2019.org/">NAACL-HLT</a> is an A-starred peer-reviewed conference, following the Association of Computational Linguistics (<a href="http://www.acl2019.org/EN/index.xhtml">ACL</a>) conference, the main event in the world of computational linguistics, aka natural language processing (NLP). In this series of posts, inspired by this year’s NAACL, we overview the most effective methods, techniques and frameworks used by academic researchers and industry engineers to achieve exceptional performance for different NLP tasks. These posts are aimed at hands-on software engineers, machine learning engineers, data scientists, and research scientists specialized in NLP and, preferably, who have a basic knowledge of modern neural networks. It is not intended as a comprehensive resource to cover the whole scope of what is being done in the NLP community. Instead, we focus only on some topics that are the most relevant to <a href="https://medium.com/orb-engineering">Orb Intelligence</a>, such as <em>language representation learning, transfer learning, multi-language support</em> (Part I), <em>text similarity, text classification, language generation, sequence labeling</em> (<a href="https://medium.com/orb-engineering/naacl-19-notes-practical-insights-for-natural-language-processing-applications-part-ii-2a2a3dd42d1">Part II</a>), <em>frameworks, and miscellaneous techniques</em> (Part III). At the same time, the reader may find fairly generic ideas quite applicable to out-of-scope NLP tasks.</p><h3>Introduction</h3><p>Perhaps the biggest shift in modern NLP is switching from representing each feature as an independent orthogonal dimension (the so-called one-hot representation, which often comes with TF-IDF weights) to dense vector representation (see Figure 1). That is, each raw feature is embedded into a low-dimensional space (which is already convenient from an engineering perspective) and represented as a vector in that space. The main benefit of the dense representations is that the features are no longer independent and captured similarities or other mutual regularities help them generalize better (see <a href="https://en.wikipedia.org/wiki/Distributional_semantics">Distributional semantics hypothesis</a>). Another advantage is that the embeddings can be trained like other parameters of the neural network, optimizing some meaningful objective (prior to that, <a href="https://en.wikipedia.org/wiki/Latent_semantic_analysis">Latent Semantic Analysis</a>, <a href="https://en.wikipedia.org/wiki/Brown_clustering">Brown clustering</a>, and <a href="https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation">Latent Dirichlet Allocation</a> were widely used for this purpose).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X2iUGrTfehszxF6FM5YEGg.png" /><figcaption>Figure 1: One-hot encoding vs. word embeddings (taken from <a href="https://arxiv.org/pdf/1510.00726.pdf">A Primer on Neural Network Models for Natural Language Processing</a>)</figcaption></figure><p>At the same time, the distributional semantics hypothesis has innate limitations:</p><p>• common sense is implicit and usually is not written down;</p><p>• word embeddings are prone to biases (e.g., ethical, social, professional);</p><p>• no grounding from modalities other than text.</p><p>The last point is fascinating because it is the main difference between artificial neural language learners (as slow and data-hungry learners) and human learners (particularly kids) who are capable of learning a language faster with few examples. We tend to think that the line of work on multi-modal learning, i.e., combining various kinds of inputs (text, audio, image, video) during learning, seems like the next potential breakthrough in AI (see examples of its first steps in Part III).</p><p>Back to reality, in this Part I, we cover some essential aspects of modern language representation learning techniques.</p><h3>Transfer Learning</h3><p>S. Ruder, T. Wolf, S. Swayamdipta, and M. Peters gave an excellent tutorial <a href="http://tiny.cc/NAACLTransfer">“Transfer learning in NLP”</a> at NAACL ’19. <em>Transfer learning in NLP</em> is considered a problem of automatically learning representations that transfer across tasks, domains, and languages with neural network-based methods for natural language processing (see Figure 2).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wQJmUk4rPDr0EhyVtuQ67w.png" /><figcaption>Figure 2: Taxonomy of transfer learning tasks (taken from <a href="http://ruder.io/thesis/neural_transfer_learning_for_nlp.pdf">S. Ruder’s Ph.D. thesis</a>)</figcaption></figure><p>In principle, transfer learning hypothetically (and more and more empirically) outperforms supervised learning except for the scenarios where no relevant information is available, or there is already a sufficient number of available training examples.</p><h4>Pre-training</h4><p><em>Pre-training</em> is the first basic step of transfer learning. The core idea of pre-training is that the internal representations trained to solve a primary NLP task with self-supervision (i.e., unlabeled data) could be useful for other tasks as well. For example, word embeddings learned for a simple word prediction task in context, or <a href="https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf">word2vec</a>, have now become essential building blocks in state-of-the-art NLP models. More difficult language modeling (LM) tasks, such as sentence prediction, contextual word prediction, and masking word prediction will be overviewed below in this blog post. There also exists neural machine translation (NMT) (e.g., <a href="https://papers.nips.cc/paper/7209-learned-in-translation-contextualized-word-vectors.pdf">CoVE</a>) and autoencoding-derived embeddings. According to the current consensus on what is known about pre-training, the tutorial authors observe that:</p><ul><li>in general, the choice of a pre-training task and an end task is coupled, i.e., closer pre-training mimics the target task, better results;</li><li>language modeling objective <a href="https://arxiv.org/pdf/1809.10040.pdf">empirically works better</a> than translation and autoencoding for downstream tasks:</li></ul><p>– it’s hard enough, and LM models have to compress any possible context (syntax, semantics, factual knowledge) to generalize over possible completions;</p><p>– it’s scalable, and more data for pre-training and more parameters ⇒ better LM model, better word embeddings;</p><ul><li>pre-training improves sample efficiency, i.e., less annotated data for the end task is usually needed to achieve the same level of quality and faster convergence.</li></ul><h4>Adaptation</h4><p>A pre-trained model needs to be <em>adapted</em> to the target task. This raises three essential questions:</p><p>1. how much to change the pre-trained model architecture for adaptation <em>(architectural modifications)</em>;</p><p>2. which weights to train during adaptation and following what schedule <em>(optimization schemes)</em>;</p><p>3. how to get more supervision signals for the target task <em>(weak supervision, multi-tasking, and ensembling)</em>.</p><p>Firstly, we have two options for architectural modifications:</p><p><strong>1a) Keep the pre-trained model internals unchanged.</strong> Remove a pre-training task head if not useful for a target task. Add transfer task-specific layers (randomly initialized) on top/bottom of the pre-trained model.</p><p><strong>1b) Modify the internals.</strong> This includes adapting to a structurally different target task. For example, pre-training with a single input sequence to a task with several input sequences (translation, language generation), namely, one may use pre-trained weights to initialize multiple layers of the target model (LM for initialization of encoders and decoders in MT). Another direction is task-specific modifications, such as adding skip/residual connections and attention layers. Finally, adding adapters, or bottleneck modules, between layers of the pre-trained model. Adapters reduce the number of parameters for tuning, allowing other “heavy” layers to stay frozen during the transfer. They may contain different operations (convolution, self-attention) and are usually connected with residual connections in parallel to an existing layer, e.g. see <a href="https://arxiv.org/pdf/1902.00751.pdf">this recent paper</a> [<a href="https://github.com/google-research/adapter-bert">code</a>] that introduces adapter modules after multi-head attention and feed-forward layers in <a href="https://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf">Transformer</a>.</p><p>Secondly, the possible decisions about optimization include:</p><p><strong>2a) (not-)tuning of pre-trained weights.</strong> Unless we change pre-trained weights, we end up with options like feature extraction and adapters. If the pre-trained weights change, then <em>fine-tuning</em> is employed. In this case, the pre-trained weights are used for parameter initialization of the end task model. Generally speaking, if a source task and a target task are dissimilar (i.e., the source task does not include relations that are very beneficial for the target task), feature extraction is preferable in practice (see details in <a href="https://arxiv.org/pdf/1903.05987.pdf">this paper</a>). Transformers (e.g. BERT) are usually easier to fine-tune than LSTMs (e.g. ELMo).</p><p><strong>2b) learning schedule.</strong> This is a decision about what weights to update, in what order and at which rate. Our motivation is to prevent overwriting of useful pre-trained knowledge (catastrophic forgetting) and retain transfer benefits. The good techniques include: updating from top to bottom (usually top layers are task-specific, bottom layers convey more general knowledge such as morphology and syntax), varying learning rates between different learning stages, and adding regularization to prevent deviation of parameters from the pre-trained region.</p><p>Thirdly, let’s consider some ideas to get more supervision:</p><p><strong>3a) fine-tuning of a model on a single adaptation task.</strong> For example, for a text classification task, extract a single fixed-length vector from the model (last hidden state or their pooling). Project to the classification space with an extra classifier, extending the top layer. Train with a classification objective.</p><p><strong>3b) related datasets.</strong> Here we have:</p><ul><li><em>sequential adaption</em>: intermediate fine-tuning on related datasets and tasks;</li><li><em>multi-task fine-tuning with related tasks</em>: employ the combination of loss functions, then, for each optimization step, sample a task and a batch for training, and, in the end, fine-tune only on the target task;</li><li><em>dataset slicing</em>: use auxiliary heads that are trained only on particular subsets of the data and detect automatically challenging subsets, on which the model underperforms (see these features in <a href="https://hazyresearch.github.io/snorkel/">Snorkel</a>);</li><li><em>semi-supervised learning</em>: minimize the distance between predictions on original inputs and their skewed versions to make them more consistent with unlabeled data.</li></ul><p><strong>3c) ensembling.</strong> This means ensembling independently fine-tuned models by combining their predictions. To get uncorrelated predictors in the ensemble, models can be trained on different tasks, dataset splits, parameter settings, and variants of pre-trained models. This direction also includes <em>knowledge distilling</em> (see details in Part III).</p><h4>Example</h4><p>A good illustration of these principles is a NAACL ’19 paper entitled, <a href="https://www.aclweb.org/anthology/N19-1213">“An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models”</a> that introduces SiATL [<a href="https://github.com/alexandra-chron/siatl">code</a>], a simple and efficient transfer learning method for text classification tasks. SiATL (see Figure 3) follows a standard approach with pre-training a language model and transferring its weights to a classifier with a task-specific layer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/886/1*Ed-FjKw1IJmUgmi4wlrw6Q.png" /><figcaption>Figure 3: SiATL</figcaption></figure><p>To prevent the catastrophic forgetting of the language distribution, the model incorporates an auxiliary LM loss along with a classification loss. Moreover, the contribution of the auxiliary LM loss is controlled via exponential decay over training epochs, gradually shifting focus from the LM objective to the classification task. Similarly to ULMFiT (see details below), SiATL benefits from a sequential unfreezing of layers: fine-tuning additional parameters, fine-tuning pre-trained parameters without an embedding layer, and train all layers until convergence. The authors also carefully chose optimizers: stochastic gradient descent (SGD) with a small learning rate is used for fine-tuning, Adam is used for randomly initialized LSTM and classification layers for faster training. While not beating the state of the art (SOTA), SiATL outperforms more sophisticated transfer learning approaches, particularly on small datasets.</p><p>Last but not least, we’ll mention a few sources of pre-trained models that can be used for transfer learning:</p><ul><li><a href="https://www.tensorflow.org/hub">TensorFlow Hub</a></li><li><a href="https://pytorch.org/hub">PyTorch Hub</a></li><li><a href="https://allennlp.org/">AllenNLP</a></li><li><a href="http://fast.ai/">Fast.ai</a></li><li><a href="https://huggingface.co/">HuggingFace</a>.</li></ul><h3>Different Granularities</h3><p>An important aspect of representation learning is the basic unit the model operates on. The authors of <a href="https://www.aclweb.org/anthology/N19-1154">“One Size Does Not Fit All: Comparing NMT Representations of Different Granularities”</a> consider four representation units: words, byte-pair encoding (BPE) units, morphological units (e.g. obtained with <a href="https://github.com/aalto-speech/morfessor">Morfessor</a>), and characters. BPE splits words into symbols (a symbol is a sequence of characters) and then iteratively replaces the most frequent sequences of symbols with a new merged symbol. BPE segmentation is very popular in neural machine translation (NMT). Morfessor splits into morphological units, such as roots and suffixes.</p><p>The authors evaluate the quality of NMT-derived embeddings originating from units of different granularity when used for modeling morphology, syntax, and semantics (as opposed to end tasks such as sentiment analysis and question answering). Their approach extracts feature representations from the encoder of a trained LSTM-based NMT model and then training a logistic regression classifier to make predictions for an auxiliary task.</p><p>The authors came to the following conclusions:</p><ul><li>the best-performing representation unit is target task-dependent;</li><li>representations derived from subword units are better for modeling syntax (i.e., long-range dependencies);</li><li>character-based representations are clearly better for modeling morphology;</li><li>character-based representations are very robust to misspellings;</li><li>using a combination of different representations often works best.</li></ul><h3>Misspelling-Tolerant Word Embeddings</h3><p>Standard word2vec approaches often poorly represent malformed words and their corrected counterparts (we usually like to have similar embeddings for them), which is a severe flaw in real-world applications. While <a href="http://fasttext.cc/">FastText</a> incorporates character n-grams for learning word embeddings, by design, it tends to capture morphemes rather than misspellings. In <a href="https://www.aclweb.org/anthology/N19-1326">“Misspelling Oblivious Word Embeddings”</a>, Facebook AI researchers present MOE, a simple method to learn word embeddings that are resilient to misspellings. They extend the FastText objective with a spelling correction objective.</p><p>They extend the FastText objective with a spelling correction objective <em>Lₛ</em> . Denote <em>(wₘ, wₑ) ∈ M</em> is a set of pairs such that <em>wₑ</em> is the correct expected word and <em>wₘ</em> is its misspelling. <em>N</em> is a set of negative samples. Let <em>l(x) = log (1 + e⁻ˣ)</em> is the logistic loss function. The scoring function <em>ŝ</em> is defined over input vectors of subwords as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*mVafNmkeq3uRmky9StSKMQ.png" /></figure><p>Then:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*n_93FFxwVf3Oa79bLBuFvg.png" /></figure><p>The first term raises the likelihood of wₑ given wₘ, pushing their representations closer. It is trained by standard SGD jointly with the regular FastText loss function, combining both of them through a weighted sum. The experiments on the word similarity and word analogy tasks show that while FastText is indeed capable of capturing low edit distance misspellings, MOE is better at capturing more distant examples. The paper also comes with a nice bonus — the <a href="https://bitbucket.org/bedizel/moe/src/master/">published dataset</a> of more than 20M corrections collected from Facebook search query logs. The experiments were conducted on English datasets. The support of multiple languages is left for future work.</p><h3>Contextual Word Embeddings</h3><p>By dynamically linking words to their various contexts, <em>contextual word embeddings</em> provide a richer semantic and syntactic representation than traditional context-independent word embeddings. There are two effective approaches to build and re-use contextual word embeddings: feature-based (e.g., ELMo) and fine-tuning (ULMFiT, OpenAI’s GPT, and Google AI’s BERT, while having the feature-based mode is <a href="https://arxiv.org/pdf/1903.05987.pdf">more effective when fine-tuned</a>).</p><p><a href="https://arxiv.org/pdf/1802.05365.pdf"><strong>ELMo</strong></a> [<a href="https://allennlp.org/elmo">code</a>] pre-trains a bidirectional LSTM-based character-level language model and extracts contextual word vectors as learned combinations of hidden states (see Figure 4). For downstream tasks, these word embeddings are used as inputs without any changes (so, they serve like features). Upon publication in 2018, ELMo has shown state-of-the-art (SOTA) results on 6 diverse NLP tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/1*3YMy180gav6pJqbZ4RcQAA.png" /><figcaption>Figure 4: ELMo</figcaption></figure><p>Hence, we briefly describe the best fine-tuning approaches to pre-training. Unlike feature-based approaches, fine-tuning provides capabilities to fit the language model to the domain-specific corpus and even downstream tasks, retaining the general knowledge that comes from the initial large corpus. For example, <a href="https://arxiv.org/pdf/1801.06146.pdf"><strong>ULMFiT</strong></a> [<a href="https://github.com/fastai/fastai">code</a>, <a href="https://github.com/fastai/fastai/blob/master/examples/ULMFit.ipynb">tutorial</a>] pre-trains Salesforce’s <a href="https://github.com/salesforce/awd-lstm-lm">AWD-LSTM</a> word-level language model (see Figure 5) and fine-tunes the trained language model in two stages with different adaptation techniques (gradual unfreezing of layers and discriminative fine-tuning with slanted triangular learning rates). ULMFiT has shown SOTA on 6 classification datasets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5O8cf_ayl65R4mVvIG8zMQ.png" /><figcaption>Figure 5: ULMFiT</figcaption></figure><p><a href="https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf">Generative Pretrained Transformer</a> (<strong>GPT</strong>) [<a href="https://github.com/openai/gpt-2">code</a>] pre-trains large 12-layer left-to-right Transformer (see Figure 6) and fine-tunes for sentence, sentence pairs, and multiple-choice questions. The flexibility for downstream tasks is achieved through linearizing the structure of the task into a sequence of tokens. GPT reached SOTA for 9 different NLP tasks, and its larger model GPT-2 gained wide publicity with its <a href="https://techcrunch.com/2019/02/17/openai-text-generator-dangerous">shockingly good text generation application</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UQ0OadyCubmR6-hMgRIxKg.png" /><figcaption>Figure 6: GPT</figcaption></figure><p>The best NAACL ’19 paper <a href="https://www.aclweb.org/anthology/N19-1423">“<strong>BERT</strong>: Pre-training of Deep Bidirectional Transformers for Language Understanding”</a> [<a href="https://github.com/google-research/bert">code</a>] describes the following approach as further development of GPT (see Figure 7): pre-train sentence and contextual word representations (the use of subword units) using a masked language model and next sentence prediction. Masking gives the ability to include both left and right context during word prediction. The BERT-large model has an impressive 340M parameters and 24 layers. BERT is currently the most useful pre-training mechanism (however, the recent <a href="https://arxiv.org/abs/1906.08237">XLNet</a> [<a href="https://github.com/zihangdai/xlnet">code</a>], claiming outperformance of BERT on 20 NLP tasks, is worth checking out).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hHnTbHoLPO0_yAlKzrASSQ.png" /><figcaption>Figure 7: BERT</figcaption></figure><h3>Cross-Lingual Word Embeddings</h3><p>Multilingual embeddings have been demonstrated to be a promising means for enabling cross-lingual transfer in many NLP tasks. There are mostly two orthogonal approaches to achieve this goal. The first is cross-lingual polyglot pre-training: sharing vocabulary and representations across languages by training one model in many languages. While it’s easy to implement, it often leads to under-representation of low-resource languages. The notable examples of this approach include Facebook Research’s <a href="https://github.com/facebookresearch/LASER">LASER</a> and <a href="https://github.com/google-research/bert/blob/master/multilingual.md">multilingual BERT</a> from Google AI.</p><p>Secondly, training word embeddings independently for each language of interest and then aligning those monolingual word embeddings. For example, Facebook Research’s <a href="https://github.com/facebookresearch/MUSE">MUSE</a> implements this approach for FastText token-level embeddings. The paper <a href="https://www.aclweb.org/anthology/N19-1162">“Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing”</a> [<a href="https://github.com/TalSchuster/CrossLingualELMo">code</a>] presents a method to align contextual word embeddings based on ELMo. Interestingly, point clouds from different words are well separated in practice, so the authors introduce an embedding anchor <em>eᵢ</em> as a centroid of a point cloud for a word <em>i</em>. Another interesting property of the space structure is multi-modality of homonym point clouds (see Figure 8).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*lF9lutbgpiUilWjyn_E3kQ.png" /><figcaption>Figure 8: Contextual embeddings for the English word ``bear’’ and its two possible translations in Spanish — ``oso’’ (animal) in blue and ``tener’’ (to have) in red (taken from <a href="https://www.aclweb.org/anthology/N19-1162">``Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing’’</a>)</figcaption></figure><p>That is to say when a word <em>i</em> has multiple distinct senses, one might expect the embeddings for <em>i</em> to reflect this by separating into multiple distinct clouds, one for each meaning. The proposed approach adapts two well-known context-independent alignment methods (<a href="https://arxiv.org/pdf/1309.4168.pdf">Mikolov et al (2013)</a> and MUSE), replacing word vectors with embedding anchors:</p><ul><li><em>supervision</em> through a given word dictionary between source and target languages. Then, the problem is reduced to the orthogonal Procrustes problem of finding an optimal linear transformation (i.e., geometrically scaling, rotation, reflection, etc.) between matrices:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*XPtunsw_d4HG9Q8qWKx-eg.png" /></figure><p>which has a closed-form solution <em>W^s→t = UVᵗ</em>, where columns of <em>U</em> and <em>V</em> are the left and right singular vectors of the multiplication of the source and (transposed) target embedding matrices;</p><ul><li><em>the unsupervised setting</em>, when the dictionary is built automatically by the adversarial framework implemented in MUSE, then the first method is applied.</li></ul><p>The authors have shown that these aligned embeddings supply good word translation (including low-resource languages, such as Kazakh) and improve significantly upon state-of-the-art zero-shot and few-shot cross-lingual dependency parsing models.</p><p><em>We would like to thank </em><a href="https://blog.inten.to/@moocaholic"><em>Grigory Sapunov</em></a><em> and </em><a href="https://www.linkedin.com/in/ACoAAAfoKFEBfW0TeYzy9eZb0-Jgj_KeJ4aoGKU"><em>Mikhail Obukhov</em></a><em> for their helpful feedback.</em></p><p><em>This concludes the Part I. In the </em><a href="https://medium.com/orb-engineering/naacl-19-notes-practical-insights-for-natural-language-processing-applications-part-ii-2a2a3dd42d1"><em>Part II</em></a><em>, we’ll present some recent advances in important end tasks, such as text similarity, text classification, language generation, and sequence labeling. </em><a href="https://medium.com/orb-engineering"><em>Follow us</em></a><em> on Medium to stay in touch. Feel free to share your experience or ask questions in the comment section.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5f981c92af80" width="1" height="1" alt=""><hr><p><a href="https://medium.com/orb-engineering/naacl-19-notes-practical-insights-for-natural-language-processing-applications-part-i-5f981c92af80">NAACL ’19 Notes: Practical Insights for Natural Language Processing Applications — Part I</a> was originally published in <a href="https://medium.com/orb-engineering">Orb Intelligence Engineering &amp; Data Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>