<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Maâli Mnasri on Medium]]></title>
        <description><![CDATA[Stories by Maâli Mnasri on Medium]]></description>
        <link>https://medium.com/@maali?source=rss-199eba94f086------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*sf-AqLeaqiVH8KSXGrKB8Q.jpeg</url>
            <title>Stories by Maâli Mnasri on Medium</title>
            <link>https://medium.com/@maali?source=rss-199eba94f086------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 05:25:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@maali/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Transformers made easy: architecture and data flow]]></title>
            <link>https://medium.com/opla/transformers-made-easy-architecture-and-data-flow-f79f11961942?source=rss-199eba94f086------2</link>
            <guid isPermaLink="false">https://medium.com/p/f79f11961942</guid>
            <category><![CDATA[bert]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Maâli Mnasri]]></dc:creator>
            <pubDate>Tue, 29 Oct 2019 15:17:56 GMT</pubDate>
            <atom:updated>2019-10-30T10:54:22.620Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rIgPNlTe1JeTCGQ3eu8S8g.jpeg" /></figure><p>Dear Transformers fans, sorry but here, we’re not talking about the cartoon series either the movies. However, the transformers we’re dealing with are also heroes but in the Artificial Intelligence world.</p><p><strong>A transformer is a Deep Learning model</strong> introduced by Google Brain’s team in 2017 in their paper: Attention is All You Need [1]. It is an evolution over the famous sequence-to-sequence models used mostly as <strong>transduction models that map a sequence of data of type A into another sequence of type B </strong>depending on the final task. In NLP it could be used for translation, summarization, dialog, etc. If you’re new to Deep Learning you can find a short yet clear introduction to sequence-to-sequence learning in our previous article <a href="https://medium.com/opla/chatbots-approaches-battle-part-1-sequence-to-sequence-vs-reinforcement-learning-731d3db75e0e">Chatbots approaches: Sequence-to-Sequence VS Reinforcement Learning</a>.</p><h3>But what’s wrong with sequence-to-sequence?</h3><p>Sequence-to-sequence (seq2seq) models were introduced in 2014 and have shown great success as unlike previous neural networks, <strong>seq2seq models take as input a sequence </strong>of numbers, words or any other type of data. Much of this success is due to the fact that <strong>most data in the real-world come into sequences</strong> and the most salient case is the text data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1ZD4eZvbtSii7CX0GmM8TA.png" /><figcaption>Sequence-to-sequence model architecture</figcaption></figure><p>Seq2seq neural networks are composed mainly of two elements: <strong>an encoder and a decoder</strong>. The encoder is fed up with the input data. It <strong>encodes data into a hidden state called context vector</strong>. Then comes the turn of the decoder that takes that context vector and <strong>decodes it into the desired output</strong>.</p><p>A major drawback of this architecture is that <strong>a single context vector cannot embed all</strong> the important information as well as <strong>the dependencies between the words</strong>, especially when the input sequence is too long.</p><p>In 2015, the seq2seq models were improved using the famous <strong>attention mechanism</strong>. The global architecture of the model remained the same. However, <strong>instead of feeding the decoder only with the final context vector, we feed it with all the resulting context vectors</strong> that come out of all the encoder RNNs. The resulting architecture is shown in the graph below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/766/1*HkMQXYZkYTMey94OLMGSfQ.png" /><figcaption>Sequence-to-sequence model with an attention mechanism</figcaption></figure><p>Another drawback in seq2seq models is related to their efficiency during training and inference. The learning process, as well as the inference is very time-consuming as we cannot encode any word in the sequence unless we have encoded all its previous words. This means, <strong>there is no hope to parallelize computing in such models</strong>.</p><h3>Attention is All You Need [1]</h3><p>Here come the new Transformers models in the Attention is All You Need Paper. The title says it all: let’s <strong>remove the RNNs and just keep attention</strong>.</p><p>Just as seq2seq, transformers do transform sequences of type A into sequences of type B. The difference is that they <strong>do not use any recurrent networks (GRU, LSTM, etc).</strong></p><blockquote>“The Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence aligned RNNs or convolution.” [1]</blockquote><p>The other difference is that unlike seq2seq, in the transformer model the words can be encoded <strong>in parallel and independently</strong>. Each word crosses a preprocessing step where it is represented by a word embedding (a word vector). Since we don’t deal with sequences anymore, word order should be saved somewhere. To this end, during the preprocessing step, the position of each word is encoded in its embedding vector. The architecture of the encoder part is shown in the graph below. We will explain the decoder part later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IjJLVG2QRuFBAMK-zZpowQ.png" /><figcaption>The architecture of the Encoders part of the transformer</figcaption></figure><p>Each encoder stack also receives information about the other elements via its <strong>Self-attention</strong> sublayers. This allows capturing the relationships between words in the sentence. We notice that each encoder is a stack of sublayers. These sublayers are mainly composed of <strong>an attention layer and a simple fully connected feed-forward network</strong>. The stack size is a parameter set to 6 in the original paper and to 2 in the graph above for simplicity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R2WCtMmEsHSDrVvRh67Tug.png" /><figcaption>The Transformer global architecture</figcaption></figure><p>Now let’s move to the remaining part of the architecture to cover all the flow. The global model is illustrated in the graph above. The decoder is also composed of a stack of sublayers, set to 2 in the graph to make it simple. The decoders’ sublayers are similar to those of the encoder: a self-attention layer that takes as input the output of the preceding decoders and a feedforward network. They also have <strong>an additional layer which is the Encoder-Decoder attention</strong> referred to in the paper as the Masked multi-head attention that receives as input the output of the multiple encoders.</p><p>The decoder is designed to produce sequentially the output scores (converted later to words) until the token &lt;EOS&gt; (End Of Sentence) is obtained. In each step, <strong>we need the output value of the preceding decoding steps</strong>. However, during the training, we already have the target values: Output 1, Output 2, … This allows all the steps from 0 until &lt;EOS&gt; to be performed <strong>in parallel and only during the training</strong>. We just need to <strong>mask</strong>, at each step, the future output values by setting them to zero for example. This is why this layer is called <strong>Masked</strong> Attention.</p><h3>How do we turn the output scores into words?</h3><p>Each decoder stack output is a vector of values of the same dimension as the word embeddings used as input. At each step of the inference, this output vector should be converted to a word. Two additional layers: t<strong>he linear layer and the softmax layer</strong> do this job. The output flow is illustrated in the figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r3EQowyPOhdvLAvI277wvg.png" /><figcaption>Transformer inference: the linear and softmax layers</figcaption></figure><p>The linear layer, which is a fully connected network, <strong>projects the output vector on a high dimensional vector space</strong> where each dimension stands for a word in the vocabulary known by the model and each value stands for the score of the corresponding word. This reminds us of the famous bag-of-words language models. Here, the vocabulary holds all the words that the model has seen during the training. The resulting vectors of the linear layer are called <strong>logits vectors</strong>.</p><p>Those logits vectors are then transferred to <strong>the softmax layer that converts the values of each index into probabilities</strong>. The probability in index <em>x</em> tells what chance does the world with index <em>x </em>have to appear as the next output word<em>. </em>You can guess it now, we need just to perform an A<em>rgmax </em>to find the index with the highest probability. Once we have it, we go back to the vocabulary to find out which word corresponds to the chosen index.</p><h3>So what?</h3><p>A legitim question right now is: ok so what? How would this new model change our NLP framework or why should it work better?</p><p>Actually, the main strength of the Transformer models is as we said its low training time compared to its predecessors. Another important breakthrough that you probably have heard about is the <strong>BERT </strong>model which stands for <strong>Bidirectional Encoder Representations from Transformers. </strong>BERT is a pre-trained model on very large text data and that is able to perform different NLP tasks and of course, it is possible to retrain it in order to make it more performant on the target task. BERT uses Transformers with a bidirectional attention mechanism. <strong>The ability to use a pre-trained model in NLP is a very important step forward</strong> that can help a lot with the lack of training data and computing servers.</p><h3>References</h3><p>[1] VASWANI, Ashish, SHAZEER, Noam, PARMAR, Niki, <em>et al.</em> Attention is all you need. In : <em>Advances in neural information processing systems</em>. 2017. p. 5998–6008.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f79f11961942" width="1" height="1" alt=""><hr><p><a href="https://medium.com/opla/transformers-made-easy-architecture-and-data-flow-f79f11961942">Transformers made easy: architecture and data flow</a> was originally published in <a href="https://medium.com/opla">Opla</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Extrinsic Evaluation of Data Augmentation for NLP: Intents Classification case study]]></title>
            <link>https://medium.com/opla/an-extrinsic-evaluation-of-data-augmentation-for-nlp-intents-classification-case-study-955f50594676?source=rss-199eba94f086------2</link>
            <guid isPermaLink="false">https://medium.com/p/955f50594676</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[small-data]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Maâli Mnasri]]></dc:creator>
            <pubDate>Wed, 31 Jul 2019 10:20:40 GMT</pubDate>
            <atom:updated>2019-07-31T10:45:11.970Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mcVFuRIWNuc_kp_lnmigvg.png" /></figure><p>Applying AI algorithms to small data has always been one of our major interests at <a href="https://opla.ai/">Opla</a> and Data Augmentation is among the key solutions to this problem. This article extends the work presented in <a href="https://medium.com/opla/text-augmentation-for-machine-learning-tasks-how-to-grow-your-text-dataset-for-classification-38a9a207f88d"><strong>our previous article on data augmentation.</strong></a></p><h3>1. Previous Research Work</h3><p>Data augmentation has shown noticeable results in cross-cutting fields such as image processing [11, 13] or speech recognition [6, 9]. Recently, research work has been conducted to apply this technique on text data <strong>which is challenging because of the complex structure of text data</strong>. While geometric transformations such as rotation and scaling are applied in image processing, <strong>in NLU, transformations aim at creating new forms of text units through paraphrase generation, morphological transformations</strong>, etc. One of the most used techniques for text data augmentation relies on text translation. Source text is translated into various intermediate languages then translated back to the original language [1]. This transformation allows creating new expressions of the same sentence. Sahin and Steedman [16] used dependency trees to augment data by changing their structure (move or trim tree fragments). Du and Black [5] performed permutation and flipping for neural dialog response selection task. Kurata et al. [12] trained an <strong>encoder-decoder to reconstruct conversation utterances</strong> in the training data. To perform augmentation, <strong>the encoder’s output hidden states are randomly perturbed to generate new utterances.</strong></p><h3>2. How can we enlarge our training data</h3><p>In this work, we try to study the effect of data augmentation in the context of <strong>intent classification for dialog systems</strong>. Our training data consists of a collection of utterances, usually composed of one sentence. Each utterance is tagged with the user intent. Our goal is to create for each utterance in the training data, new similar utterances that inherit the same label as the original one. This task can be considered as the task of paraphrasing the labeled samples. To paraphrase the available utterances, we combined three techniques: words shuffling, word replacement with synonym and word replacement with semantically related words.</p><h4>2.1. Words shuffling</h4><p>This method consists of simply modifying the order of the words in a sentence to create a new one. In this case, the classification algorithms should be sensitive to the order of the words.</p><h4>2.2. Synonym replacement</h4><p>This variant uses an external thesaurus of synonyms or paraphrases. To paraphrase a sentence, for each of its words, we look for a synonym in the synonyms database. If a synonym is found, we use it to replace the original word. The examples below are extracted from our results. The words in bold are those who have been replaced.</p><p>What to use to <strong>quickly</strong> <strong>cut</strong> Audio/Video ➜ What to use to <strong>speedily clip</strong> Audio/Video</p><p>Shut down without <strong>extra question </strong>➜ Shut down without <strong>additional issues</strong></p><p>We choose to replace every word for which a synonym is found in the database to get as different as possible paraphrases.</p><h4>2.3. Similar word replacement</h4><p>In this variant, we chose to use distributional word representations known as word embeddings to enrich our training data. For each word in a sentence, if the word is represented in the word embeddings we replace it by its most similar word that has the same Part Of Speech (POS) tag.</p><h3>3. Experiments settings</h3><p>Since our ultimate goal is to improve the intent classification performance, we conducted an extrinsic evaluation of the described data augmentation methods by comparing the performance of the classification with and without data augmentation.</p><h4>3.1. Datasets</h4><p>We used mainly three publicly available datasets[3]:</p><ul><li>Ask Ubuntu Corpus: gathers 162 questions and five intents extracted from <a href="https://askubuntu.com/">AskUbuntu forum</a></li><li>Web Applications Corpus: composed of 89 questions and eight intents extracted from <a href="https://webapps.stackexchange.com/">StackExchange</a></li><li>Chatbot Corpus: gathers 206 questions and two intents extracted from a Telegram chatbot for public transport in Munich.</li></ul><p>We used <strong>10-folds cross-validation</strong> to split the data into train and test sets.</p><h4>3.2. Classification algorithms</h4><p>We chose to use three state-of-the-art classification algorithms:</p><ul><li>Support Vector Machines (SVM) [4] with a linear kernel</li><li>K-Nearest Neighbors (KNN) [10] with k set to 5</li><li>Extra Tress Classifier (ETREE) [8] with 200 estimators.</li></ul><h4>3.3. Preprocessing</h4><p>During the augmentation process, we remove all stop words. In the synonym replacement approach, we stem both our dataset and the vocabulary of the thesaurus. Finally, during the classification, all the words are stemmed to get more accurate word counts.</p><h4>3.4. Vectorization</h4><p>We used TF.IDF (Term Frequency x Inverse Document Frequency) to represent our training data features since our original training data sets are relatively small.</p><h4>3.5. External resources</h4><p>a- Synonyms dictionary: we used the ParaPhrase DataBase (PPDB)[7]</p><p>b- Word embeddings: we used different pre-trained word vectors:</p><ul><li>GV6B: are pre-trained word vectors using GloVe [15] on Wikipedia 2014 + Gigaword 5 with 400K vocabulary size</li><li>FTXT: are pre-trained with FastText [2] on Common Crawl and Wikipedia</li><li>W2V: are pre-trained with Word2Vec [14] on the Google News corpus with a vocabulary size of 3 million</li></ul><h3>4. Evaluation and results</h3><p>To evaluate the performance of the classification algorithms we measure each time the <strong>micro-averaged F-measure</strong>. The micro-averaged F-measure aggregates the contributions, in terms of precision and recall, of all classes to compute the average metric. This is more relevant if we suspect imbalanced class distribution. Figure 1 shows the class distribution of each dataset. In all data sets, there is not a predominant class. Despite this, we use the F-measure to produce realistic scores.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PQlr-ir5tdYuGt3_f8TI_w.png" /></figure><p>The results of the multiple experiments are presented in tables 1, 2 and 3 using respectively SVM, KNN and ETREE algorithms. The different variants that we evaluated are:</p><ul><li>W/O DG: classification without data augmentation</li><li>WS is the variant when we perform words shuffling. We chose to generate for each sentence n random shuffles where n is the length of the original sentence</li><li>GV6B &amp; FTXT apply the most similar word replacement respectively from GV6B and FastText word vectors</li><li>GV6B+WS is GV6B with the application of word shuffling over the augmented data</li><li>PPDB applies synonym replacement using the PPDB paraphrase database which includes paraphrases of words and phrases. In this work, we use only words (unigram) paraphrases.</li><li>PPDB-W2V applies the most similar word replacement from W2V word vectors over augmented samples using PPDB</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xfZxFxc7VoMS3a7Hpm_ihQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i8iLEm-qKWgx_SUnLrq9dw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rs-2VqLmEzfupzazvbHCDA.png" /></figure><p>We notice, first of all, that without data augmentation SVM and ETREE perform slightly better than KNN. We also note that the performance on the Web Application data set is considerably weaker than on the AskUbuntu and the Chatbot corpora. Now if we take a look at the data augmentation impact, we can see that <strong>word shuffling, as simple as it is, allows to gain some points in the performance</strong>. This gain is more important on the WebApplication dataset and is around +15%.</p><p>Regarding the paraphrasing techniques, all the presented approaches were able to improve the results over the initial results and also over the word shuffling technique. <strong>Performing word shuffling over data already augmented with paraphrases has, overall, led to better scores.</strong></p><p>Among the semantically related word replacement techniques (GV6B, GV6B+WS, and FTXT), it is hard to decide which one performs better since their scores are very close. On top of that, each classifier reached its best performance using a different combination of augmentation methods.</p><p>However, if we examine the scores of the PPDB variant in the three tables, we can conclude that <strong>using word embeddings for data augmentation works better than using paraphrases or synonyms. </strong>We suppose that word embeddings, by providing semantically related words and not exact synonyms, allow generating <strong>more diverse and generalized data</strong>. Another possible reason is the augmented data size. U<strong>sing word embeddings we could multiply the data size by 45 on average</strong>. Using, the synonym replacement method <strong>we only multiplied by 2 the initial data size.</strong></p><p>Overall, the evaluation of the proposed data augmentation techniques shows that they considerably enlarge our datasets and thus improve the classification performance.</p><h3>5. Conclusion</h3><p>We tried through this work to demonstrate the benefit of using data augmentation for the task of intent classification where training data is usually insufficient. The reported results show that <strong>simple data enrichment techniques such as word shuffling, synonym replacement or semantically related word replacement can yield significant improvement.</strong></p><p>Our future research will focus on context-aware data augmentation to handle intent detection systems that exploit context. Future studies will also be conducted to apply data augmentation to generate complete conversations rather than just utterances which should help a lot with machine learning-based answer generation systems.</p><h3>References</h3><p>[1] Segun Taofeek Aroyehun and Alexander Gelbukh. 2018. Aggression detection in social media: Using deep neural networks, data augmentation, and pseudo labeling. In Proceedings of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC-2018). 90–97.</p><p>[2] Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics 5 (2017), 135–146.</p><p>[3] Daniel Braun, Adrian Hernandez-Mendez, Florian Matthes, and Manfred Langen. 2017. Evaluating Natural Language Understanding Services for Conversational Question Answering Systems. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue. Association for Computational Linguistics, SaarbrÃĳcken, Germany, 174–185. <a href="http://www.aclweb.org/anthology/W17-3622">http://www.aclweb.org/anthology/W17-3622</a></p><p>[4] Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine learning 20, 3 (1995), 273–297.</p><p>[5] Wenchao Du and Alan W Black. 2018. Data Augmentation for Neural Online Chat Response Selection. arXiv preprint arXiv:1809.00428 (2018).</p><p>[6] Takashi Fukuda, Raul Fernandez, Andrew Rosenberg, Samuel Thomas, Bhuvana Ramabhadran, Alexander Sorin, and Gakuto Kurata. 2018. Data Augmentation Improves Recognition of Foreign Accented Speech. Proc. Interspeech 2018 (2018), 2409–2413.</p><p>[7] Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. PPDB: The Paraphrase Database. In Proceedings of NAACL-HLT. Association for Computational Linguistics, Atlanta, Georgia, 758–764. <a href="http://cs.jhu.edu/~ccb/">http://cs.jhu.edu/~ccb/</a> publications/ppdb.pdf</p><p>[8] Pierre Geurts, Damien Ernst, and Louis Wehenkel. 2006. Extremely randomized trees. Machine learning 63, 1 (2006), 3–42.</p><p>[9] Tom Ko, Vijayaditya Peddinti, Daniel Povey, and Sanjeev Khudanpur. 2015. Audio augmentation for speech recognition. In Sixteenth Annual Conference of the International Speech Communication Association.</p><p>[10] Flip Korn, Nikolaos Sidiropoulos, Christos Faloutsos, Eliot Siegel, and Zenon Protopapas. 1998. Fast nearest neighbor search in medical image databases. Technical Report.</p><p>[11] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems. 1097–1105.</p><p>[12] Gakuto Kurata, Bing Xiang, and Bowen Zhou. 2016. Labeled Data Generation with Encoder-Decoder LSTM for Semantic Slot Filling… In INTERSPEECH. 725–729.</p><p>[13] Niall McLaughlin, Jesus Martinez Del Rincon, and Paul Miller. 2015. Data augmentation for reducing dataset bias in person re-identification. In 2015 12th IEEE International conference on advanced video and signal based surveillance (AVSS). IEEE, 1–6.</p><p>[14] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems. 3111–3119.</p><p>[15] Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). 1532–1543.</p><p>[16] Gozde Gul Sahin and Mark Steedman. 2018. Data Augmentation via Dependency Tree Morphing for Low-Resource Languages. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 5004–5009.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=955f50594676" width="1" height="1" alt=""><hr><p><a href="https://medium.com/opla/an-extrinsic-evaluation-of-data-augmentation-for-nlp-intents-classification-case-study-955f50594676">An Extrinsic Evaluation of Data Augmentation for NLP: Intents Classification case study</a> was originally published in <a href="https://medium.com/opla">Opla</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Conducting Scientific Research in companies: A new workflow model]]></title>
            <link>https://medium.com/opla/conducting-scientific-research-in-companies-a-new-workflow-model-5ee6cb91b916?source=rss-199eba94f086------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ee6cb91b916</guid>
            <category><![CDATA[workflow]]></category>
            <category><![CDATA[research]]></category>
            <category><![CDATA[scientific-research]]></category>
            <category><![CDATA[research-and-development]]></category>
            <category><![CDATA[science]]></category>
            <dc:creator><![CDATA[Maâli Mnasri]]></dc:creator>
            <pubDate>Tue, 18 Jun 2019 12:44:37 GMT</pubDate>
            <atom:updated>2019-06-18T13:04:07.226Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MpGnUL6xiXBp9B5zgJL7yA.png" /></figure><p>As an <a href="https://opla.ai/">innovative company</a> specialized in chatbot builders, we continuously conduct researches related to NLP and AI. Originally, research has always been associated with research laboratories and academic institutes. Today, the case is different and <strong>research labs and companies work together </strong>to make advances in many different fields. <strong>Companies have proven their ability to produce strong research results</strong>.</p><p>In this article, we will be sharing our experience of how we manage our research team and how it interacts with the other teams. We present a methodology model for conducting research in companies. Note that the presented work could be applied in any innovative company regardless of its domain.</p><h3>Research workflow</h3><p>Now, we will focus on how the research part is organized. The whole workflow is represented in the following diagram.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lq1ZbKCE3lIDop_atRc5Yg.png" /></figure><p><strong><em>Research origin</em></strong></p><p>The first stage, entitled Research origin, is a very important step especially in companies as it aims at defining research axes and orientations. Such orientations should be in line with the company mid-term and long term evolution needs. In practice, we start by defining <strong>the scientific problem</strong> we aim to solve with our research work. Next, we study our <strong>product development ideas </strong>which can be proposed by any member of any team. We join the problems and the development needs to set <strong>clear questions and goals</strong>. We end this process by identifying the <strong>benefits</strong> of such work. This stage does not only help with setting a starting point for research studies but also prevents from wasting time and effort on developing ideas that won’t be of great benefit for the company and/or for the scientific community.</p><p><strong><em>References</em></strong></p><p>The completion of the previous process will lead to a clear goal and a precise research subject. Now, it’s time to explore the scientific landscape that surrounds the subject. To this end, we start by studying the bibliography in order to <strong>set a reference context</strong> and <strong>highlight theoretical and empirical scientifical materials</strong> justifying decisions to be made during development.</p><p>During this process, we can acquire knowledge about:</p><ul><li>the general theories on the research subject</li><li>previous development experiences on the same issue</li><li>the characteristics of the audience targeted by the research</li><li>the characteristics of the concerned domain</li><li>the existing approaches and strategies</li><li>the principles, rules and maybe the laws that govern the design of the research idea</li></ul><p>Once the bibliography has been established, it becomes possible to further <strong>elaborate the research idea</strong> by <strong>providing enrichments</strong> that literature has unveiled.</p><p>As we can see in the diagram, the reference step does feed all the next processes and continues to be performed until the end of the project. This allows <strong>being always up to date with the latest state of the art</strong> results.</p><p><strong><em>Methodology</em></strong></p><p>Now that the idea is clear and the state of the art is established, we can define the methodology which includes the methods and the tools to be used to bring to light the research idea.</p><p>First, we define which <strong>knowledge we should acquire and which skills we should develop</strong>. Next, we set the <strong>deliverables, deadlines as well the quality standards</strong>. An <strong>evaluation strategy</strong> is then set in order to specify which evaluation tools and frameworks we should acquire or create as well as the test sets. We also prepare a dissemination plan at this stage by spotting the upcoming scientific conferences and events on which we can <strong>validate our work and share it with the community</strong>. Finally, we check the availability of the data we need to set up our ideas so we get prepared for data collection.</p><p><strong><em>Operationalization</em></strong></p><p>The design step aims at defining the components and the links that lead to the development of our general model.<br>Then, we move to the realization which includes the implementation, the configuration, the test, etc. During the test &amp; verification process, we <strong>iteratively refine our solution</strong> as shown in the diagram below which brings to mind the Agile methodology.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*jl2DUJcDRTG9I4-bz-xTpw.png" /></figure><p>We create as many prototypes as needed to satisfy the final goals.</p><p><strong><em>Results</em></strong></p><p>Now it’s time to understand and valorize the results. We start with an analytical study which aims at :</p><ul><li>getting prepared to present the results by synthesizing all the work carried out until this point</li><li>explain and justify all the choices made during the project</li><li>identify a set of principles and conclusions of the research experiment</li><li>compare these conclusions with the knowledge identified in the bibliography process (state of the art)</li></ul><p>Such analysis can unveil some weaknesses of the solution. In this case, we go back to the previous process and update it according to the results. We <strong>continue working inside this loop</strong> (operationalization to results) <strong>until delivery of the Proof Of Concept</strong> (POC).</p><p>When the POC is released and the results are stabilized, we produce articles highlighting the progress of the process and the final results.</p><h3>How to characterize a research work?</h3><p>This whole process must meet the distinctive characteristics of a scientific approach! To know :</p><ul><li>holding an <strong>innovative</strong> character</li><li>establishing links between the proposed solution and the <strong>bibliography</strong></li><li>theoretic <strong>justification</strong> for every choice</li><li>rigorous <strong>analysis</strong> of the results</li><li>opening up to research tracks <strong>beyond the current problematic</strong></li><li>publishing the results in <strong>scientific reports</strong></li></ul><h3>Transfer Methodology: Industrialization of Research</h3><p>Now that our research idea is validated, we move to the transfer process. By transfer, we mean transferring the research knowledge and ideas into concrete features in our product. The corresponding workflow is described in the diagram below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/531/1*UoGOVR0OGDH2bs_Hu8h3qg.png" /></figure><p>The advanced research block stands for the research workflow described in the first part. The first thing to do in the industrialization step is to evaluate the research work from a practical point of view. The Research team, as well as the Pre-Prod R&amp;D team, work together to check if the research is mature enough to be industrialized by :</p><ul><li><strong>discover</strong> the scientific POC</li><li>validate the <strong>vital hypotheses</strong></li><li>ensure the <strong>stability</strong> of the research idea</li><li>verify the achievement of the level of <strong>customer satisfaction</strong> if we move to <strong>the solution industrialization</strong></li></ul><p><strong><em>If the answer is NO :</em></strong></p><p>The reasons of inadequacy are raised to the previous step (research team) and according to the reason:<br>- either we work more on the idea to improve it if possible<br>- either we move to another research problem</p><p><strong><em>If the answer is YES :</em></strong></p><p>We move to the pre-development process which aims at:</p><ul><li>specifying the <strong>main features</strong> inspired by the scientific POC</li><li>implementing a development POC: a <strong>minimalist implementation</strong> of the research idea</li></ul><p>After the pre-development phase, the POC (dev) joins the usual feature development lifecycle that we follow at <a href="https://opla.ai/">Opla</a>. In our case, we use Agile project management with Scrum.</p><h3>Teams’ involvement in the workflow</h3><p>The table below shows the involvement of the different teams in the described processes of the workflow. Despite the distribution of the tasks among the teams is very clear, anyone from the start of the process can participate at the end of the process and vice versa. This guarantees<strong> joint progress </strong>in the research and production axes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wboi2aUGcEFnYdzqak2FJQ.png" /></figure><p>So this was the whole workflow from advanced theoretic research to production. If you try this model on, let us know how it worked for you in the comments section.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ee6cb91b916" width="1" height="1" alt=""><hr><p><a href="https://medium.com/opla/conducting-scientific-research-in-companies-a-new-workflow-model-5ee6cb91b916">Conducting Scientific Research in companies: A new workflow model</a> was originally published in <a href="https://medium.com/opla">Opla</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to train word embeddings using small datasets?]]></title>
            <link>https://medium.com/opla/how-to-train-word-embeddings-using-small-datasets-9ced58b58fde?source=rss-199eba94f086------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ced58b58fde</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[small-data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[word-embeddings]]></category>
            <category><![CDATA[transfer-learning]]></category>
            <dc:creator><![CDATA[Maâli Mnasri]]></dc:creator>
            <pubDate>Mon, 20 May 2019 07:44:46 GMT</pubDate>
            <atom:updated>2019-05-20T07:44:46.953Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/926/1*cD1jR-SV7Ko3AzldRt215A.jpeg" /></figure><p>Word embeddings are <strong>words representation in a low dimensional vector space </strong>learned from a large text corpus according to a predictive approach. This words representation is built in such a way that semantically similar/related words have vectors that are close in space vector. The resulting word vectors encode the meaning of words from a distributional point of view : words that tend to occur together in the same context should have some semantic relatedness.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jnrmjaYMF_gq-9zia2HV7g.png" /><figcaption>Figure 1 : example of 2-D word vector space</figcaption></figure><p>Figure 1 shows an example of such word vector representation in a space of dimension 2. In reality, in order to carry enough meaning word vectors come often with 100 to 600 dimensions. The most known embeddings are sized to 300 dimension which experimentally works very well.</p><p>Word embeddings have been well demystified recently and they have become a key ingredient in most of deep learning based NLP tasks basically because Machine Learning (ML) techniques handle numbers only so every thing needs to be converted to numbers in order to be processed by ML functions.</p><h3>How are word embeddings trained?</h3><p>Word2Vec (Mikolov et al., 2013), GloVe (Pennington et al., 2014), FastText (Joulin et al., 2016) are among the most known algorithms to train word embeddings. According to the task and to the training set, one algorithm may lead to better results than another. However, <strong>what really makes the difference </strong>is the training data. To have the best results when using the generated embeddings, the training corpus should satisfy two constraints:</p><p>1. Having a maximal <strong>vocabulary coverage</strong> with your test data .</p><p>2. Being<strong> large enough</strong> to get accurate word representation .</p><h3>What if you don’t have enough data?</h3><p>One of the fast tricks consists of using <strong>pre-trained word vectors</strong>. Actually, many organizations have released pre-trained embeddings on very large text corpora : <a href="https://fasttext.cc/docs/en/crawl-vectors.html">FastText</a>, <a href="https://code.google.com/archive/p/word2vec/">Word2Vec</a>, <a href="https://nlp.stanford.edu/projects/glove/">GloVe</a>.</p><p>These word embeddings are large enough to cover many general NLP tasks. For example, Google News word vectors released by Word2Vec cover <strong>3 million words and phrases.</strong></p><p>Now, what if pre-trained word vectors don’t fit your domain specific text and your training data isn’t large enough to get accurate results?</p><h3>Retrofitting : Adapting pre-trained word embeddings</h3><p>Retrofitting is a technique proposed by Faruqui et al. (2014) that can help with this problem. It consists of <strong>fine tuning pre-trained word vectors </strong>using external knowledge bases or semantic networks. This process looks if words from the embeddings have some semantic relations in the external knowledge base. According to the found relation, it aims at <strong>making their corresponding vectors closer or more distant</strong>. For example, if we use a paraphrase database as a knowledge base, if we find that two words in the word embeddings are paraphrases, retrofitting will attempt to make their vectors closer in the vector space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wZ2w99JLgZyY11CdJttROg.png" /><figcaption>Figure 2 : Example of retrofitting graph (Graph taken from the paper (Faruqui et al., 2014)</figcaption></figure><p>The model of Faruqui et al. (2014) represents the embeddings vocabulary<br>that is covered by the semantic lexicon by graphs where <strong>semantically related words in the lexicon are connected by edges </strong>(<em>cf. Figure 2</em>). The nodes of these graphs represent the <strong>modified vectors which are latent variables</strong>. Each node is connected to a node representing the <strong>original vector</strong> of the concerned word, which corresponds to the <strong>observable</strong>. Then, using an <strong>iterative algorithm of belief propagation</strong>, the new vectors are modified so as to <strong>minimize the distance of each node to its neighbors and to its original vector</strong>. The <strong>code</strong> of this algorithm can be found <a href="https://github.com/mfaruqui/retrofitting">here</a>.</p><p>The described approach is useful to enrich word vectors with semantic information. It could also be used to adapt the embeddings to any specific domain. Meanwhile, some effort needs to be done in order to <strong>create a semantic graph </strong>reproducing the semantic relations between the domain specific words. Another drawback to this method is that <strong>it doesn’t allow to add new vocabulary </strong>to the embeddings.</p><h3>Take away : Future AI, Transfer Learning, Small AI …</h3><p>Obviously, this method isn’t the miracle solution to data lack. It is all about the global idea of <strong>modifying some parts of machine learned models to adapt them to particular tasks</strong>. This problem is among the current <strong>unresolved AI tasks</strong> and represent one of the <strong>most interesting future AI directions</strong>. To be useful for broad range of application, future machine learning technologies <strong>should be able to learn from small data and should also be transferrable from a context to another (transfer learning).</strong></p><p>FARUQUI, Manaal, DODGE, Jesse, JAUHAR, Sujay K., <em>et al.</em>Retrofitting word vectors to semantic lexicons. <em>arXiv preprint arXiv:1411.4166</em>, 2014.</p><p>GLAVAŠ, Goran et VULIĆ, Ivan. Explicit retrofitting of distributional word vectors. In : <em>Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)</em>. 2018. p. 34–45.</p><p>Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119, 2013. (Cité en pages 50 et 70.)</p><p>Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove : Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014. (Cité en pages 51 et 70.)</p><p>JOULIN, Armand, GRAVE, Edouard, BOJANOWSKI, Piotr, <em>et al.</em>Bag of tricks for efficient text classification. <em>arXiv preprint arXiv:1607.01759</em>, 2016.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ced58b58fde" width="1" height="1" alt=""><hr><p><a href="https://medium.com/opla/how-to-train-word-embeddings-using-small-datasets-9ced58b58fde">How to train word embeddings using small datasets?</a> was originally published in <a href="https://medium.com/opla">Opla</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Chatbots approaches battle (Part 1) : Sequence-to-Sequence VS Reinforcement Learning]]></title>
            <link>https://medium.com/opla/chatbots-approaches-battle-part-1-sequence-to-sequence-vs-reinforcement-learning-731d3db75e0e?source=rss-199eba94f086------2</link>
            <guid isPermaLink="false">https://medium.com/p/731d3db75e0e</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[conversational-ai]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[chatbots]]></category>
            <dc:creator><![CDATA[Maâli Mnasri]]></dc:creator>
            <pubDate>Fri, 29 Mar 2019 13:21:31 GMT</pubDate>
            <atom:updated>2019-10-28T11:40:40.751Z</atom:updated>
            <content:encoded><![CDATA[<h3>Chatbots approaches: Sequence-to-Sequence VS Reinforcement Learning</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/1*C92kgL5dPDRV-FFqUzuhyQ.jpeg" /></figure><p>We have been working recently on a <a href="https://arxiv.org/pdf/1903.09025.pdf">paper</a> that addresses the most recent approaches for chatbots building to draw an overview of the <a href="https://arxiv.org/pdf/1903.09025.pdf">current state of the art of chatbots</a>.</p><iframe src="https://drive.google.com/viewerng/viewer?url=https%3A//arxiv.org/pdf/1903.09025.pdf&amp;embedded=true" width="600" height="780" frameborder="0" scrolling="no"><a href="https://medium.com/media/39c80a42e339caea13ac6a4ed30a012f/href">https://medium.com/media/39c80a42e339caea13ac6a4ed30a012f/href</a></iframe><p>During this research, we categorized the most used techniques in both the industrial and academic sectors. The first comparison was between <strong>rule-based and data-driven approaches,</strong> and the second was between Machine learning and Information retrieval. We will discuss those approaches in a further post on <a href="https://medium.com/opla">medium</a>. Here, we want to focus on the comparison of two particular machine learning approaches for chatbots which were widely leveraged in the scientific landscape: Sequence-to-sequence models and Reinforcement Learning.</p><h3><strong>Let’s understand Sequence-to-sequence learning</strong></h3><p>Sequence to sequence (seq2seq) learning (Sutskever et al., 2014) is a way to combine multiple Recurrent Neural Networks (RNN) in a particular architecture to tackle complex sequence-to-sequence prediction problems such as machine translation, image captioning, text summarization, question-answering, etc.</p><p>Seq2seq learning have shown great success when first applied on the <strong>phrase to phrase</strong> <strong>Machine Translation (MT)</strong>(Cho et al., 2014) and therefore, inspired researchers to apply it for other tasks. During the training process, <strong>a seq2seq model learns to map a given input to a given output</strong>. The length of the input and output sequences can be different and this is the strength of seq2seq models in comparison with other neural learning models. So during the test, the trained model is able to generate a new output given new input. The image below shows a simplified layout of a seq2seq model for MT.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/1*DWLHqfd657M8v-IVwMzrcA.png" /><figcaption>source: <a href="https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html">https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html</a></figcaption></figure><p>For building chatbots, the problem was considered as <strong>translating the user utterance to the chatbot answer</strong>.</p><p>Technically, a seq2seq model is composed of an encoder and a decoder. The encoder is a neural network or a sequence of neural networks that read the input sequence and convert it into a <strong>hidden state</strong> called thought vector because it <strong>stores the meaning of the input sequence</strong>, considered as a thought. The decoder is then fed with this thought vector. During the learning stage, <strong>it learns to map the hidden state to the target output sequence</strong>. In the inference stage, the decoder returns the predicted output sequence with respect to the learning goal. The figure below shows an example of a sequence to sequence architecture for generating answers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nGh-7cgJczJXvrenx6I1FA.png" /><figcaption><a href="https://arxiv.org/pdf/1903.09025.pdf">source : (Mnasri, 2019)</a></figcaption></figure><p>The studies conducted on seq2seq-based chatbots have shown that training a straightforward seq2seq model on a <strong>large conversational dataset</strong> is a simple way to create a chatbot that <strong>answers simple questions, extracts relevant information and even perform some shallow reasoning</strong>. Seq2seq models can also learn many aspects besides answer generation. For example, they can <strong>inherit the personality of characters</strong> they learn from, they can <strong>capture the tone</strong> of the speaker and answer in a convenient way.</p><h3><strong>Let’s move to Reinforcement Learning</strong></h3><p>Reinforcement learning isn’t as recent as seq2seq learning. It is an <strong>old learning technique</strong> used firstly to <strong>train robots to move</strong>. The process is inspired by the human learning process. To start learning, children tend to interact with the surrounding environment. After each interaction, <strong>they decide whether to repeat or not an action based on the consequences </strong>of that action. To train a dialogue system with reinforcement learning, the <strong>chatbot interacts with the end-users</strong> and <strong>observes the results</strong> of its actions. It <strong>receives each time a reward</strong> which can be positive or negative. Throughout the conversations, the chatbot becomes increasingly efficient.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/1*bCRwLkeul7XyaIFndQjg6g.png" /><figcaption>Reinforcement learning cycle</figcaption></figure><h3><strong>Sequence-to-Sequence VS Reinforcement Learning: </strong>Who wins?</h3><p>Both approaches have limits and strengths. Seq2seq made a huge step forward when it comes to the <strong>generative</strong> aspect. They are also <strong>simple to train </strong>as they are most of the time <strong>purely data-driven</strong>. Up to now, you may be impressed by these networks. The bad news is that they have many weaknesses also. The first problem to tackle with these models is the <strong>highly general answers they return</strong>. The second drawback is related to the grammatical correctness of the answers. While seq2seq images that we present everywhere show perfect examples of outputs, the reality is slightly different and <strong>the generated answers are not always well-formed</strong>. So more work needs to be done in order to correct the output. Some researchers also highlighted that seq2seq learning <strong>predicts answers only one at a time</strong> and <strong>fails to predict their influence on future utterances</strong>. The last problem is related to the <strong>necessity of having a big amount of data</strong>, a problem we are working on currently (see our <a href="https://medium.com/opla/text-augmentation-for-machine-learning-tasks-how-to-grow-your-text-dataset-for-classification-38a9a207f88d"><strong>previous article on data augmentation</strong></a>).</p><p>Meanwhile, Reinforcement learning leads to <strong>more natural chatbots</strong> as they learn from human feedback and <strong>develop their own control system</strong>. Reinforcement learning enables the chatbot to <strong>handle long conversations</strong> and <strong>take into account the preceding turns</strong>. It is also convenient to use RL to make chatbots<strong> learn to act</strong> and <strong>not only to chat</strong>. For example, a flight booking agent will receive a reward if it properly books a flight according to the user request and will get penalized if it makes an error. However, RL raises two main issues. Firstly, it needs <strong>so much time</strong> and <strong>so many interactions</strong> until the agent is trained which may be restricting if the training is online. Secondly, RL is <strong>not the most suitable option to learn language generation.</strong></p><h3>To end the suspense...</h3><p>In my opinion, both are <strong>winners</strong>. None is perfect on its own and both approaches <strong>complement one another</strong>. So the question is <strong>What should I use?</strong></p><p>In order to take the best part of each method, many studies focused on combining these two approaches. The combination will depend on the end-use. Li et al. (2016) suggested a novel model that leverages the seq2seq ability to represent semantics and combined it to a reinforcement learning policy that optimizes longterm rewards according to the final use. That being said, the right question should be: <strong>How to combine Sequence-to-Sequence learning to Reinforcement Learning</strong>?</p><blockquote>If our state-of-the-art article helps with your work, please cite it as follows:</blockquote><blockquote>MNASRI, Maali. Recent advances in conversational NLP: Towards the standardization of Chatbot building. arXiv preprint arXiv:1903.09025, 2019.</blockquote><p>LI, Jiwei, MONROE, Will, RITTER, Alan, <em>et al.</em> Deep reinforcement learning for dialogue generation. <em>arXiv preprint arXiv:1606.01541</em>, 2016.</p><p>MNASRI, Maali. Recent advances in conversational NLP: Towards the standardization of Chatbot building. <em>arXiv preprint arXiv:1903.09025</em>, 2019.</p><p>SUTSKEVER, Ilya, VINYALS, Oriol, et LE, Quoc V. Sequence to sequence learning with neural networks. In : <em>Advances in neural information processing systems</em>. 2014. p. 3104–3112.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=731d3db75e0e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/opla/chatbots-approaches-battle-part-1-sequence-to-sequence-vs-reinforcement-learning-731d3db75e0e">Chatbots approaches battle (Part 1) : Sequence-to-Sequence VS Reinforcement Learning</a> was originally published in <a href="https://medium.com/opla">Opla</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Text augmentation for Machine Learning tasks: How to grow your text dataset for classification?]]></title>
            <link>https://medium.com/opla/text-augmentation-for-machine-learning-tasks-how-to-grow-your-text-dataset-for-classification-38a9a207f88d?source=rss-199eba94f086------2</link>
            <guid isPermaLink="false">https://medium.com/p/38a9a207f88d</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[training-data]]></category>
            <category><![CDATA[small-data]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Maâli Mnasri]]></dc:creator>
            <pubDate>Fri, 18 Jan 2019 16:40:31 GMT</pubDate>
            <atom:updated>2019-01-18T16:40:31.483Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*bBMS9SW60XozrRbBWyc1Dw.jpeg" /></figure><p>Text classification is the task of tagging automatically text segments with predefined categories. It is a supervised process that learns from observing pre-labelled data. There are multiple known algorithms to achieve this goal like the well known Naive Bayes, SVM and the different Deep Neural Networks.</p><p>Text classification can be used for <strong>spam detection, language recognition, sentiment analysis</strong>, and the list is so long. @<a href="https://medium.com/u/88b902dfb60f">Opla.ai</a>, we deal with text classification, for example, to perform <strong>intent recognition</strong> within our chatbot builder. Actually, having multiple intent categories, <strong>intent classification</strong> allows to assign the right intent to the user utterance.</p><p>Globally, text classification is no more a mysterious task especially using the most basic algorithms. You can easily find excellent examples on internet on how to do it. However, most of these articles/tutorials explain how to learn classification <strong>HAVING</strong> the training data. Usually, we can find standard training data for classification which is great to start with Text Classification and to compare models. Though, many of us are interested in this task because they have a particular need in their company/project. And that’s when we start wondering about the data problem as <strong>text classifiers are worthless without enough training data</strong>. The quality of these also matters.</p><p>Sometimes, we have no data at all. So we need to create it or collect it. But other times, we do have data but not enough to achieve high accuracy classification. This problem arises in particular when dealing with languages other than English or also for uncommon classification labels. In this article we will try to answer partially the question of <strong>how can we train a classification model on little data</strong>? Answering this question is associated to the <strong>Small AI</strong> area of research which tries to apply machine learning models on small datasets. You can find more details on Small AI in our <a href="https://medium.com/opla/small-data-is-big-in-ai-train-spotting-at-france-is-ai-4afb24168e4c"><strong>article</strong></a><strong> on future directions of AI</strong>.</p><h3><strong>Data augmentation, quésaco?</strong></h3><p>It is the process of extending a set of existant data by performing on them some operations which depend on the data type. The goal is to create new labeled data from the existing ones. For example, when we deal with images we can create new data by performing operations like <strong>rotation, translation, scaling up or down</strong>, etc. Data augmentation can also be achieved by enriching the data using <strong>external knowledge sources</strong>.</p><p>When it comes to <strong>text</strong>, there are other ways to create new data and the possibilities are not few.</p><h3>Approaches</h3><p><strong>Word/sentence shuffling</strong></p><p>A simple baseline for data augmentation is <strong>shuffling the text elements to create new text.</strong> For example, if we have labeled sentences and we want to get more, we can shuffle each sentence words to create a new sentence. This option is only valid for classification algorithms that don’t take into account the words order within a sentence. So in practice, we should tokenize each sentence into words. Then we shuffle those words and rejoin them to create new sentences.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ac0891df4abfdbc3a9e0b84338a3ca8d/href">https://medium.com/media/ac0891df4abfdbc3a9e0b84338a3ca8d/href</a></iframe><p><strong>Word replacement</strong></p><p>This technique aims at replacing some words in the sentence with new words while preserving its meaning. For example, having the sentence “Do not park your car in this placement” we can create these new sentences by substituting the word car with another word each time:</p><p>“Do not park your vehicle in this placement”</p><p>“Do not park your motocar in this placement”</p><p>To replace words, there are several ways to find substituents. The most relevant way is to replace words by their <strong>synonyms </strong>and expressions by their <strong>paraphrases</strong>. The code snippet below allows to perform this. I used a paraphrase list from <a href="http://paraphrase.org/#/">PPDB</a>. You can find <strong>all the code along with the used data </strong><a href="https://github.com/Opla/SmallData-Augmentation-MachineLearning"><strong>here</strong></a><strong>.</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/011d0fcb6029e58852f366c60f0df655/href">https://medium.com/media/011d0fcb6029e58852f366c60f0df655/href</a></iframe><p>We ran the code above and these are the results. The words highlighted with the same colors represent the original words and their synonym substitute.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0MGQ0n4RAOLotEKQDtmVSg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6GEv0eMnoEchvIQ4dxu0Sg.png" /><figcaption>Figure 1: Example of a semantic network. <em>Source :</em><a href="https://commons.wikimedia.org/w/index.php?curid=1353062"><em>https://commons.wikimedia.org/w/index.php?curid=1353062</em></a></figcaption></figure><p>Semantic networks can also be used to find substituents. Concepts related by an “is a” relation in these knowledge bases can replace one another. For example, if we have this subgraph of a Semantic Net, we can replace the word “mammal” by the word “animal” as a mammal is an animal.</p><p>More sophisticated models have been proposed to augment text data by word replacement such as <strong>predictive models </strong>trained to predict the next word having the current one(Kobayashi, 2018).</p><p><strong>Syntax-tree manipulation</strong></p><p>This option allows to generate syntactic paraphrases of the source sentence. Many syntactic transformations can be applied to create paraphrases such as <strong>moving from active to passive voice, breaking apart a sentence containing a subordinate clause</strong>, etc. Three examples of syntactic paraphrasing are presented below. They are extracted from (Dras, 1997) article on using synchronous TAGs for paraphrasing.</p><p>(1) a. The salesman made an attempt to wear Steven down.</p><p>(1) b. The salesman attempted to wear Steven down.</p><p>(2) a. The compere who put the contestant to the lie detector gained the cheers of the audience.</p><p>(2) b. The compere put the contestant to the lie detector test. He gained the cheers of the audience.</p><p>(3) a. The smile broke his composure.</p><p>(3) b. His composure was broken by the smile.</p><p>This operation requires, first, to generate for each sentence its syntactic dependency tree. Then, use some rules to make transformations on that tree while preserving the meaning and the grammatical quality. The graph below shows an example of the source sentence syntactic tree as well as the new sentences ones.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/962/1*Xd4R05yAdLxtdfHYbYZUTg.png" /><figcaption>Figure 2 : Transformational Grammar (Chowdhari, 2012).</figcaption></figure><p>While the paraphrases are often of good quality, <strong>this NLP task is computationally expensive</strong> which can be inconvenient in practice.</p><p><strong>Query expansion</strong></p><p>Another approach to text data augmentation is the so called query expansion technique. This method aims at finding new sentences similar to a given sentence <strong>by considering this latter as query and searching for its answer.</strong> Using a generic or a domain specific search engine, we can define a query as our original sentence. Then, we select <strong>the most relevant search results as new sentences</strong>. These new sentences should inherit the same tag as the original sentence (the query). Sentence query expansion can also be performed by requesting knowledge bases such as <a href="https://en.wikipedia.org/wiki/Main_Page">Wikipedia</a> or <a href="https://wordnet.princeton.edu/">WordNet</a> rather than search engines.</p><p><strong>Conclusion</strong></p><p>In machine learning systems, the more data your system observes the more accurate it gets. So, whatever the technique you choose, data augmentation should increase your classifier accuracy if it is done correctly. From another perspective, data augmentation methods do not only enlarge your training set, but also lead to <strong>more generalizable classifiers.</strong> Data augmentation has also been <strong>identified among the best practices</strong> when working with convolutional neural networks applied to document analysis (Simard &amp; al., 2003).</p><p>Despite the ease of some of the presented approaches, it is undeniable that language understanding is categorically harder than image understanding from an algorithmic perspective. This is not necessarily a bad news as it means <strong>there’s more green field to explore in this area</strong>. That’s why we keep this subject among our high priority points of interest in <a href="https://opla.ai/">Opla</a>’s R&amp;D team.</p><p><a href="https://github.com/Opla/SmallData-Augmentation-MachineLearning">Opla/SmallData-Augmentation-MachineLearning</a></p><p><em>CHOWDHARY, K. R. Natural Language Processing. MBM Engineering College, Jodhpur, India, lecture notes, 2012.</em></p><p><em>DRAS, Mark. Representing paraphrases using synchronous tags. In : Proceedings of the eighth conference on European chapter of the Association for Computational Linguistics. Association for Computational Linguistics, 1997. p. 516–518.</em></p><p><em>KOBAYASHI, Sosuke. Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relations. arXiv preprint arXiv:1805.06201, 2018.</em></p><p><em>SIMARD, Patrice Y., STEINKRAUS, Dave, et PLATT, John C. Best practices for convolutional neural networks applied to visual document analysis. In : null. IEEE, 2003. p. 958.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=38a9a207f88d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/opla/text-augmentation-for-machine-learning-tasks-how-to-grow-your-text-dataset-for-classification-38a9a207f88d">Text augmentation for Machine Learning tasks: How to grow your text dataset for classification?</a> was originally published in <a href="https://medium.com/opla">Opla</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Small Data is Big in AI : Train-spotting at France is AI]]></title>
            <link>https://medium.com/opla/small-data-is-big-in-ai-train-spotting-at-france-is-ai-4afb24168e4c?source=rss-199eba94f086------2</link>
            <guid isPermaLink="false">https://medium.com/p/4afb24168e4c</guid>
            <category><![CDATA[data]]></category>
            <category><![CDATA[chatbots]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Maâli Mnasri]]></dc:creator>
            <pubDate>Wed, 24 Oct 2018 10:21:23 GMT</pubDate>
            <atom:updated>2018-10-24T10:38:04.369Z</atom:updated>
            <content:encoded><![CDATA[<h3>Small Data is Big in AI : Train-spotting at France is AI</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GbJAZtjL1BNfGrKErFtxcw.jpeg" /></figure><p>The third edition of <a href="https://franceisai.com/">France Is AI</a>, major meeting of the French ecosystem of artificial intelligence, was held at <a href="https://stationf.co/fr/">Station F</a> Paris on October 17th and 18th and was organized by <a href="http://www.francedigitale.org/">France Digitale</a>. The event gathered more than <strong>65 renowned speakers and 2000 attendees</strong>.</p><h3>OplaAI on Twitter</h3><p>Today we chat from the #FranceIsAI spot! The #AI conversation starts now ;) #DigitalTransformation</p><p>Exchange, discovery and innovation were on the program of this event that brought together researchers, developers, entrepreneurs, managers and enthusiasts.</p><p>The first day was dedicated to technology and science with talks from Heads of AI Labs at large corporations including Twitter, Huawei, Criteo, Netflix, Facebook, Google. While the second was around the AI ecosystem and future strategies.</p><p>As France is AI 2018 Conference closes its doors, the AI community realizes clearly that today’s AI state of the art comes with as much advancements as questions about the future, regarding technology trends and data usage.</p><p>As an AI/NLP researcher, I will be talking about these trends that arise from this conference. I will also try to highlight the future AI trends and issues.</p><h3>Bias/Context aware Machine Learning</h3><p>Despite the different tasks tackled by each company, the issues were not that different. Netflix head of AI, <a href="http://raimond.me.uk/">Yves Raimond</a> talked about recommendation systems and particularly on how we shouldn’t fall in the trap of overfitting the training data which represent the past. When predicting the future we should take into account the fact that everything changes. <strong>One way to avoid overfitting the past is to add contextual variables</strong> so the future will be predicted having the current context and the user contextual variables. In that case, time is embedded as a continuous variable in the system. He also talked about <strong>integrating causality in the recommender system by using the client feedback in the prediction</strong>.</p><p>IBM AI CTO, <a href="https://www.linkedin.com/in/guilhaume/?originalSubdomain=fr">Guilhaume Leroy-Meline </a>talked about <a href="http://aif360.mybluemix.net/">fairness in AI</a> and particularly how to remove and detect bias in machine learning models. He presented their open source library <a href="http://aif360.mybluemix.net/">AIF360</a> which performs this task and allows also to l<strong>earn to act in order to favor the unprivileged features in the learning process </strong>like gender for example.</p><p>Next, the <a href="http://www.centralesupelec.fr/fr/centre-de-vision-numerique-cvn">Digital Vision Center</a> director <a href="https://www.linkedin.com/in/nikos-paragios-20777869/?originalSubdomain=fr">Nikos Paragios</a> explained <strong>why AI can’t, currently, go beyond human in the medical field.</strong> The main reasons are the <strong>variability in cases</strong>, the lack of data and also their quality. He recommended to set 5 features in order to apply ML properly in medicine. You need to have a well defined task, the right data (quantity and quality), the right methods, the right team and the right expectations too. He also talked about the robustness issue of ML systems where <strong>performance drops when the input changes</strong>.</p><p><a href="http://ogrisel.com/">Olivier Grisel</a>, a software engineer at <a href="https://www.inria.fr/">Inria</a>, presented the new release of <a href="http://scikit-learn.org/stable/">Scikit-learn</a> and its major improvements. The main enhancements are about the feature engineering flexibility and the <strong>early stopping feature which has been added to many classifiers</strong>. This last release is also scalable to parallel computing allowing people having clusters to benefit from the parallelism.</p><h3>Is Deep Learning enough for NLU?</h3><p>A common NLU issue was tackled by many speakers : deep learning is purely data driven and performs nor shallow reasoning. As confirmed by <a href="https://mila.quebec/">Mila</a>’s director <a href="https://mila.quebec/personne/bengio-yoshua/">Yoshua Bengio</a>, <strong>learning distributional representation of words will never lead to understand the language</strong>. If we want to learn an alien language we will observe what aliens do, the <strong>context </strong>in which they speak and try to get their <strong>intention</strong>. That’s what lacks actually in current NLU techniques. In the opposite side of ML NLU, we find symbolic AI/NLU which has been among the first techniques used in NLU. Symbolic NLU uses structured data such as knowledge bases. <strong>Models are able to learn from structured data by seeing only few examples</strong>. However, such data is expensive to produce and doesn’t model uncertainty. The current <a href="https://research.fb.com/category/facebook-ai-research/">Facebook AI research</a> conjecture, presented by <a href="https://research.fb.com/people/bordes-antoine/">Antoine Bordes</a>, is to benefit from both worlds.</p><h3>OplaAI on Twitter</h3><p>Parallel worlds? YES! We love connecting people to machines 😎 Thanks to Antoine Bordes, Director of @Facebook #AI Research in Paris. #FranceisAI</p><p>To this end, it is possible to <strong>learn continuous representations of knowledge bases (KB)</strong> in the same way we learn continuous representations of words. In the KB case, the vectors will represent entities and a similarity function will represent the relation such “is part of”.</p><h3>Will Nano Neurons kill Transistors?</h3><p><a href="http://www.cnrs-thales.fr/">CNRS-Thalès</a> were also there and <a href="http://www.cnrs-thales.fr/spip.php?article59">Julie Grollier</a>, a research director, talked about the future hardware nano neurons. Actually, when comparing hardware processors to our brain, hardware is less efficient than our brain. One of the main differences is that in our brain, memory and processing happen in the same place while in machines processors and memory are separated. <strong>So recent works are trying to go beyond the current transistors which are just switches to create new hardware neurons where memory and processing are fully closer. </strong>These new nano neurons are non linear nano radios composed each of two small magnets that transfer information through their spinnings.</p><p>An edgy subject was addressed by the Microsoft France CTO CSO, <a href="https://news.microsoft.com/fr-fr/exec/bernard_ourghanlian/">Bernard Ourghanlian</a>, who talked about quantum computing and how fast it can solve combinatorial problems that current computers take ages to solve. He also thinks that there is no chance to apply deep learning as it is now on quantum computers which perform mainly linear operations. So he recommends, to <strong>start thinking of new techniques or adapt current ones to quantum computing as quantum computers are not too far away.</strong></p><h3>Small AI : the new AI using Small Data</h3><p>The main observation I made during these intensive presentations is that data availability is a major AI issue slowing down advances in different tasks. Researchers can’t agree more on the necessity of <strong>rethinking the current AI and ML techniques in order to exploit smaller datasets.</strong> Deep learning does a good job in many tasks but requires so much data so it is not applicable in all levels/organizations. So <strong>the future direction in AI will definitely be the “small data AI” and the goal will be to achieve firstly, the same performance using less data.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*_lYdseIcNOyfRgjIhiIGaw.png" /></figure><p>On top of that, a clear consensus emerged on the necessity of extending current deep learning NLU approaches using reasoning on knowledge corpora as in Symbolic AI. This latter has a significant ability to achieve language understanding that purely data driven machine learning prevents from exploiting. It seems that some research need to be done in order to <strong>combine both methods while keeping low computational time.</strong></p><p><strong>We are ALL still searching for the</strong> <strong>right tools and concepts to improve AI</strong>. We are all equal in this quest from small startups to big corporations. Events such as France is AI are an opportunity to go further by exchanging points of views and identifying scientific obstacles. <strong>Undoubtedly, everyone has a chance to bring a significant contribution to the scientific community and solve part of the challenge.</strong> That being said, collaboration remains the shortest way to tackle scientific issues.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4afb24168e4c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/opla/small-data-is-big-in-ai-train-spotting-at-france-is-ai-4afb24168e4c">Small Data is Big in AI : Train-spotting at France is AI</a> was originally published in <a href="https://medium.com/opla">Opla</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>