<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rohit Pillai on Medium]]></title>
        <description><![CDATA[Stories by Rohit Pillai on Medium]]></description>
        <link>https://medium.com/@conversationalaiteam?source=rss-7f2959a1653------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*6eJ7aGCDLK0OjViLrbWSqw.jpeg</url>
            <title>Stories by Rohit Pillai on Medium</title>
            <link>https://medium.com/@conversationalaiteam?source=rss-7f2959a1653------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 18:35:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@conversationalaiteam/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[CODIR: Train smaller, faster NLP models]]></title>
            <link>https://medium.com/data-science/codir-train-smaller-faster-nlp-models-bac6318a9950?source=rss-7f2959a1653------2</link>
            <guid isPermaLink="false">https://medium.com/p/bac6318a9950</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[knowledge-distillation]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[compression]]></category>
            <category><![CDATA[bert]]></category>
            <dc:creator><![CDATA[Rohit Pillai]]></dc:creator>
            <pubDate>Mon, 04 Jan 2021 15:03:18 GMT</pubDate>
            <atom:updated>2021-01-04T19:45:51.883Z</atom:updated>
            <content:encoded><![CDATA[<h4>A unique approach to knowledge distillation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*K-sLRvJzxIJQbWIS" /><figcaption>Photo by <a href="https://unsplash.com/@awcreativeut?utm_source=medium&amp;utm_medium=referral">Adam Winger</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The field of natural language processing has been revolutionized with the advent of large pretrained models like BERT and GPT-3. These models have been able to capture incredible amounts of information from the vast amounts of text they’ve been trained on and use this information to reach state of the art performance and continually improve on a wide variety of tasks like classification, summarization and entailment.</p><p>One reason for these models’ great performance is their size. BERT-base has 110 million parameters, BERT-large with 340 million parameters, GPT-2 with 1.5 billion parameters and GPT-3 has a whopping <strong>175 billion</strong> parameters. As these models grow in size, so does the time and resources required to pretrain them. To make these models cost and resource efficient, a stream of research focused on reducing model size has been burgeoning. Several techniques have been adopted to do this:</p><ul><li><strong>Quantization</strong> focuses on reducing the size of the model weights (from 32-bit numbers to 16 or 8-bit numbers).</li><li><strong>Pruning</strong> identifies redundant weights and tries to discard them.</li><li><strong>Knowledge Distillation</strong> focuses on creating smaller finetuned models by trying to match a larger model (teacher) to a smaller one (student). Models like DistilBERT and MobileBERT extended this concept to create smaller pretrained models using a BERT teacher.</li></ul><p>Generally, knowledge distillation consists of 2 parts:</p><ol><li><strong>The original training objective:</strong> This objective focuses on getting the student model to learn the right labels (during finetuning) or the correct pretraining objective</li><li><strong>The distillation objective:</strong> This objective tries to match the output probabilities (and possibly the hidden states) of the student with those of the teacher. This is done by selecting a distance metric and minimizing it between the student and the teacher.</li></ol><p>The researchers at Microsoft Dynamics 365 AI Research try to approach the distillation problem through a different lens. Instead of trying to minimize the distance for all examples between the teacher and the student, why not try to maximize the distance between false examples and minimize the distance between the true examples? This led them to propose <strong>CODIR</strong>, a technique that uses contrastive learning to distil the teacher’s hidden states to the student. CODIR can be applied to both the pretraining and finetuning stage.</p><p>Unlike traditional knowledge distillation techniques, CODIR has 3 components:</p><ol><li><strong>The original training objective:</strong> This is the same as regular KD</li><li><strong>The distillation objective:</strong> Unlike regular KD, this objective uses the KL-divergence loss to teach the student to predict a similar probability distribution as the teacher. It’s only applied to the output of the student model.</li><li><strong>The contrastive learning objective: </strong>This objective focuses on distancing the hidden states of the student from those of the teacher for negative examples and matching them for positive examples.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9nurB0tja8VmhXhLSBIATg.png" /><figcaption>The CODIR training framework from Sun et al.</figcaption></figure><p>Applying CODIR to the finetuning stage is very straightforward. Since the training examples have labels, negative examples are just those which have a different label than the positive example.</p><p>In the pretraining stage, there are no labels so applying the contrasting learning objective becomes a little more challenging. The researchers test CODIR on the RoBERTa model, which uses masked language modeling as the pretraining objective. In this case, negative examples would be masked sentences or text from the same article so that they’re semantically similar to the positive examples. This way, the contrastive learning objective doesn’t become too easy to solve.</p><p>The researchers use CODIR to reduce a Roberta model to 6 layers and verify its effectiveness on the GLUE dataset. This model achieves almost the same accuracy as the original BERT model with <strong>half the inference time</strong>.</p><p>Here’s a <a href="https://arxiv.org/pdf/2009.14167.pdf">link</a> to the paper if you want learn more about CODIR, a <a href="https://github.com/intersun/CoDIR">link</a> to the code for you to distil your own models and click <a href="https://aka.ms/mmai">here</a> to see more of our publications and other work.</p><p><strong>References</strong></p><ol><li>Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. <strong>Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. </strong>arXiv preprint arXiv:1910.01108.</li><li>Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. 2020. <strong>Mobilebert: a compact task-agnostic bert for resource- limited devices.</strong> arXiv preprint arXiv:2004.02984.</li><li>Siqi Sun, Zhe Gan, Yu Cheng, Yuwei Fang, Shuohang Wang, and Jingjing Liu. <strong>“Contrastive Distillation on Intermediate Representations for Language Model Compression.”</strong> <em>arXiv preprint arXiv:2009.14167</em> (2020).</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bac6318a9950" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/codir-train-smaller-faster-nlp-models-bac6318a9950">CODIR: Train smaller, faster NLP models</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SpanFact: Fix your factually incorrect summaries]]></title>
            <link>https://medium.com/data-science/spanfact-fix-your-factually-incorrect-summaries-459481978f86?source=rss-7f2959a1653------2</link>
            <guid isPermaLink="false">https://medium.com/p/459481978f86</guid>
            <category><![CDATA[fact-checking]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[summarization]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[bert]]></category>
            <dc:creator><![CDATA[Rohit Pillai]]></dc:creator>
            <pubDate>Wed, 16 Dec 2020 08:22:47 GMT</pubDate>
            <atom:updated>2020-12-17T05:06:47.959Z</atom:updated>
            <content:encoded><![CDATA[<h4>A suite of models that fixes inaccuracies in summaries of large text</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7K1tE_N6upn8hjMb" /><figcaption>Photo by <a href="https://unsplash.com/@m_rags?utm_source=medium&amp;utm_medium=referral">madeleine ragsdale</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>When I’m trying to decide whether I should read a book, the first thing I look at is the summary on the back. This is also true for when I have to determine whether a scientific paper is relevant to my work, whether an article could be interesting or anything else that involves reading a lot of text. For this to be effective, the summary needs to be concise and accurate.</p><p>Machine learning researchers have focused on summarizing large chunks of text and developed models that can generate grammatically correct and coherent summaries. These summarization models fall into 2 broad categories: extractive and abstractive. Extractive models try to identify the important phrases in the text and stitch them together in a coherent way without modifying these phrases. In contrast, abstractive models try to generate a summary one word at a time after processing and understanding the whole text.</p><p>Historically, abstractive summaries are generally more concise and unique since they’re not limited to just the words that are in the text but they’re more prone to error because they generate the next word based on the words that have already been generated. However, with the advent of large transformer models like BERT which can understand text better, abstractive models are able to outperform extractive models even grammatically. They’re still prone to one error though: inaccuracy. This is because Seq2Seq models (which form the backbone of most abstractive models) are trained to generate coherent and correct sentences. When asked to be accurate as well, these models struggle to find a balance between the two objectives and generally disregard the accuracy objective.</p><p>There are 2 ways to fix this accuracy problem: retrain the model to focus more on generating accurate information or fix the model’s generated summary. Researchers have used multiple approaches to do the former. Cao et al. and Zhu et al. train their summarization model with fact triplets to generate more accurate summaries. Li et al. and Falke et al. use a different approach by trying to include sentences that are relevant to the text to be summarized.</p><p>While retraining seems like a more comprehensive fix, it requires a lot of resources and data to train and still faces the problem of trying to balance accuracy and coherence. The researchers at Microsoft Dynamics AI 365 Research try the latter approach of fixing an already generated summary. Introducing <strong>SpanFact</strong>! It’s two models that use question-answer (QA) style span selection and auto-regression to fix already generated summaries. By doing this, they aim to maintain coherence while improving the accuracy of the summary.</p><p>SpanFact focuses on correcting inaccuracies related to entities in the summary (names, numbers, places and more). Both SpanFact models use BERT to generate initial embeddings for the passage to be summarized and the summary itself. They also iterate through all the entities in the summary, first masking them and then trying to identify them from the passage. To do this, they make unique assumptions and approaches.</p><p>The QA-span model assumes that all the entities in the generated summary are unrelated. This method works for shorter summaries that have a small number of errors. It masks one entity at a time, uses BERT to learn an embedding for the masked summary along with the passage and tries to predict the part of the passage that corresponds to the entity. Once it has made its selection, the model replaces the masked part of the summary with the selection and proceeds to mask the next entity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JtKJh753o8AczB4kppxdDg.png" /><figcaption>The SpanFact model architectures: QA-Span model on the left and Auto-regressive model on the right. <a href="https://arxiv.org/pdf/2010.02443.pdf">Image source</a></figcaption></figure><p>On the other hand, the auto-regressive model assumes that the entities in the summary are related and tries to minimize future errors by correcting earlier ones. To do this, it first masks all the entities before passing the masked summary through a BERT. It then uses the BERT embeddings of the current masked entity combined with all the previously predicted entities to predict the current entity’s start and end position in the passage. The first entity is predicted using the embedding of the [CLS] token. The main difference between this and the QA-span model is that the prediction layer only sees the entities until now while the QA-span model sees the entire summary with just one mask.</p><p>To make sure the SpanFact models work, the researchers test them on summaries generated on 3 major summarization datasets: CNN/DailyMail, Gigaword and XSum. Both the models are able to significantly improve the summary’s accuracy without noticeably sacrificing coherency. In fact, their summaries achieve state of the art accuracy scores on all these datasets.</p><p>Here’s a <a href="https://arxiv.org/pdf/2010.02443.pdf">link</a> to the paper if you want to go into more detail about SpanFact and click <a href="https://aka.ms/mmai">here</a> to see more of our publications and other work.</p><p><strong>References</strong></p><ol><li>Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018. <strong>Faithful to the original: Fact aware neural abstractive summarization.</strong> In Thirty-Second AAAI Conference on Artificial Intelligence.</li><li>Chenguang Zhu, William Hinthorn, Ruochen Xu, Qingkai Zeng, Michael Zeng, Xuedong Huang, and Meng Jiang. 2020. <strong>Boosting factual correctness of abstractive summarization with knowledge graph.</strong> arXiv preprint arXiv:2003.08612.</li><li>Haoran Li, Junnan Zhu, Jiajun Zhang, and Chengqing Zong. 2018. <strong>Ensure the correctness of the summary: Incorporate entailment knowledge into abstractive sentence summarization.</strong> In Proceedings of the 27th International Conference on Computational Linguistics, pages 1430–1441.</li><li>Tobias Falke, Leonardo FR Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. <strong>Ranking generated summaries by correctness: An interesting but challenging application for natural language inference.</strong> In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2214–2220.</li><li>Dong, Yue, Shuohang Wang, Zhe Gan, Yu Cheng, Jackie Chi Kit Cheung, and Jingjing Liu. <strong>“Multi-fact correction in abstractive text summarization.”</strong> <em>arXiv preprint arXiv:2010.02443</em> (2020).</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=459481978f86" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/spanfact-fix-your-factually-incorrect-summaries-459481978f86">SpanFact: Fix your factually incorrect summaries</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[HERO: You’ll never have to watch long videos again]]></title>
            <link>https://medium.com/data-science/hero-youll-never-have-to-watch-long-videos-again-ee0d5e2ba4fd?source=rss-7f2959a1653------2</link>
            <guid isPermaLink="false">https://medium.com/p/ee0d5e2ba4fd</guid>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[using-pretrained-model]]></category>
            <category><![CDATA[multimodal-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Rohit Pillai]]></dc:creator>
            <pubDate>Thu, 17 Sep 2020 03:01:28 GMT</pubDate>
            <atom:updated>2020-09-29T23:30:23.962Z</atom:updated>
            <content:encoded><![CDATA[<h4>A model that understands a video and its subtitles to help you speed through it</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vhoumAZjgskZctfW" /><figcaption>Photo by <a href="https://unsplash.com/@joeynicotra?utm_source=medium&amp;utm_medium=referral">Joey Nicotra</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Have you ever had to watch 15 lectures worth of material the night before an exam? Or submit a report about a 2-hour movie that you haven’t watched in the next hour? I know I have. Luckily, my lecture videos were marked with where content started and ended so I could just skim through them. But what if those markers weren’t there? Fear not: HERO is here to save you!</p><p>“What is HERO and how will it save me?”, you might ask. Your question will be answered but first, let me give you some context. If you’ve read some of my previous posts, you’ll probably know that machine learning is moving towards the direction of large scale pretrained models, which are models that are given a lot of data initially and learn from them. These models can then apply this knowledge to a specific task to achieve better performances than most others.</p><p>Pretrained models like BERT, RoBERTa and ALBERT have taken over the field of NLP and the multimodal field is starting to develop them too. Models like <a href="https://towardsdatascience.com/uniter-d979e2d838f0">UNITER</a> and Unicoder-VL have been big strides in the image+text pretraining space.</p><p>However, one space that hasn’t been explored all that much is the video+text space. Ideally, this should be a very natural extension. After all, videos are just a sequence of images. There have been a few attempts to tackle this space through models like CBT and UniViLM, but they fail to capture a key aspect of videos: their order. Instead they treat videos and their subtitles as a bunch of images and text without considering that a subtitle is only relevant for certain frames and that the order of the frames matter.</p><p>In order to remedy this, the researchers at Microsoft Dynamics 365 AI Research propose <strong>H</strong>ierarchical <strong>E</strong>ncode<strong>R</strong> for <strong>O</strong>mnirepresentation learning (<strong>HERO</strong>). Unlike previous models for video+text, HERO accounts for the sequence of the video and how the text maps to it.</p><p>To do this, HERO uses a 2-layer hierarchical structure. The first layer uses a <em>“cross-modal”</em> transformer which fuses text and image. It matches each sentence from the subtitles to the corresponding part of video. The output of this <em>“cross-modal”</em> transformer is then used by a <em>“temporal” </em>transformer to learn the order of the different frames of the video. This hierarchical learning is made possible through 4 main pre-training tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*Rh44IIcUlCf5uM0hW6d3EA.png" /><figcaption>The model architecture for <a href="https://arxiv.org/abs/2005.00200">HERO</a>.</figcaption></figure><ol><li><em>Masked Language Modeling (MLM)</em></li></ol><p>In <em>MLM</em>, HERO is given a subtitle sentence with some missing words and the corresponding video frames and HERO is expected to generate those missing words.</p><p>2. <em>Masked Frame Modeling (MFM)</em></p><p><em>MFM</em> is like <em>MLM</em> for images except that it is done with the whole video as context. The input is all the subtitle sentences and most of the video frames. HERO’s job is to generate the missing frames.</p><p>3.<em> Video Subtitle Matching (VSM)</em></p><p><em>VSM</em> is what it sounds like for the most part. HERO is given one sentence from the subtitles as a query and the context is the all the subtitle sentences and all the video frames. HERO has to learn a local alignment i.e. identify the frames of the video that the query sentence corresponds to.</p><p>But there’s a catch. The provided video may not even match the subtitles. This helps HERO learn what a sentence should look like and vice versa.</p><p>4.<strong> </strong><em>Frame Order Modeling (FOM)</em></p><p>While <em>MLM</em>, <em>MFM</em> and <em>FOM</em> are all used to train the <em>“cross-modal”</em> transformer, <em>FOM</em> is used to train the <em>“temporal” </em>transformer. It takes the output of the 3 previous tasks and a jumbled set of frames to predict the correct order of frames.</p><p>After the pre-training tasks worked their magic and HERO was trained, the researchers tested its power on several different tasks including Video Captioning, Video Question Answering, Video-Moment Retrieval (retrieving a specific part of the video based on a sentence) and <a href="https://medium.com/swlh/violin-do-you-really-understand-videos-fe07f7affd9e">Video-Language Inference</a>. HERO was able to beat the best models on all these tasks and even achieved the <strong>#1 position on the </strong><a href="https://tvr.cs.unc.edu/index.html"><strong>TVR</strong></a><strong> </strong>(TV-Retrieval) <strong>and </strong><a href="https://tvr.cs.unc.edu/tvc.html"><strong>TVC</strong></a><strong> </strong>(TV-Captioning) <strong>leaderboards.</strong></p><p>So don’t wait, let HERO save you from those last minute lecture/documentary/movie watches!</p><p>Here’s a <a href="https://arxiv.org/pdf/2005.00200.pdf">link</a> to the paper if you want to know more about the HERO model, a <a href="https://github.com/linjieli222/HERO">link</a> to the code for you to train your own and click <a href="https://aka.ms/mmai">here</a> to see more of our publications and other work.</p><p><strong>References</strong></p><ol><li>Lei, Jie, Licheng Yu, Tamara L. Berg, and Mohit Bansal. <strong>Tvr: A large-scale dataset for video-subtitle moment retrieval.</strong> arXiv preprint arXiv:2001.09099 (2020).</li><li>Chen, Yen-Chun, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. <strong>Uniter: Learning universal image-text representations.</strong> arXiv preprint arXiv:1909.11740 (2019).</li><li>Li, Gen, Nan Duan, Yuejian Fang, Ming Gong, Daxin Jiang, and Ming Zhou. <strong>Unicoder-VL: A Universal Encoder for Vision and Language by Cross-Modal Pre-Training.</strong> In <em>AAAI</em>, pp. 11336–11344. 2020.</li><li>Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Xilin Chen, and Ming Zhou. 2020. <strong>Univilm: A unified video and language pre-training model for multimodal understanding and generation.</strong> arXiv preprint arXiv:2002.06353.</li><li>Li, Linjie, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. <strong>HERO: Hierarchical Encoder for Video+ Language Omni-representation Pre-training.</strong> EMNLP<em> </em>2020.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ee0d5e2ba4fd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/hero-youll-never-have-to-watch-long-videos-again-ee0d5e2ba4fd">HERO: You’ll never have to watch long videos again</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[FILTER: Understand Foreign Languages Better]]></title>
            <link>https://medium.com/swlh/filter-understand-foreign-languages-better-4bfa6d12377f?source=rss-7f2959a1653------2</link>
            <guid isPermaLink="false">https://medium.com/p/4bfa6d12377f</guid>
            <category><![CDATA[comprehension]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[multilingual]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <dc:creator><![CDATA[Rohit Pillai]]></dc:creator>
            <pubDate>Wed, 16 Sep 2020 00:46:31 GMT</pubDate>
            <atom:updated>2020-09-16T07:01:07.301Z</atom:updated>
            <content:encoded><![CDATA[<h4>A technique that promotes a deeper understanding between different languages</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fabqJBzlAt7sZhHf" /><figcaption>Photo by <a href="https://unsplash.com/@graesmagazine?utm_source=medium&amp;utm_medium=referral">GRÆS Magazine</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Based on sources from across the internet, the total world population that speaks English is somewhere between 1 in 6 and 1 in 7. Despite this underwhelming minority of the world’s population speaking English, a vast majority of natural language datasets for understanding and generation like the Stanford Question Answering dataset (SQuAD) and GLUE datasets as well as the large scale pretrained models like BERT, RoBERTa and ALBERT that have revolutionized the NLP world are solely based on the English language.</p><p>However, there has been a recent focus on other languages with the creation of multi-lingual large scale pretrained models like XLM and XLM-RoBERTa and the introduction of complex multi-lingual tasks like question answering, document classification, information retrieval and more. Mostly recently, XTREME and XGLUE are 2 collections of multi-lingual datasets that require models to be good at several tasks to perform well on their scoreboards.</p><p>Now let’s dive a little deeper into how these multi-lingual models are created. There’s 2 main schools of thought to approaching this:</p><ol><li><strong>Learning an embedding</strong> that is common across all the languages in the world (or whatever’s available to train with at least). This can be done by either feeding a huge amount of text in a multitude of languages to a large language model like XLM or XLM-RoBERTa to learn an <strong>implicit embedding</strong> (a word has a different embedding based on the context) or by just using a simple neural network to learn an <strong>explicit embedding</strong> like Word2Vec (a word always maps to the same embedding)</li><li>Translating the data either at train <strong>(Translate-train)</strong> or test <strong>(Translate-test)</strong> time. <strong>Translate-train</strong> is when the training data in English is translated to a foreign language and the translated text is added to the training dataset. <strong>Translate-test</strong> is when the foreign language data is converted to English at test time and the model does its prediction on this English data. The prediction can be translated back to the foreign language for tasks like question answering or span selection or is just a class that doesn’t require translation.</li></ol><p>Even though the dataset in the translate-train method has been augmented by the translated English sentences, the input data is still only one language. Nowhere in the translate-train architecture do multiple languages ever interact. Thus, translate-train is more of a data augmentation method than one that promotes an understanding of multiple languages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CWBJmmY9z8RyucAVb3sYGQ.png" /><figcaption>The translate-train architecture. The streams on the left and the right go through the same set of transformers independently</figcaption></figure><p>Inspired by the translate-train method but with a desire to make the model understand the relationships between 2 languages, the researchers at Microsoft Dynamics 365 AI Research propose a new way to train a multi-lingual model called <strong>FILTER</strong>. The input to FILTER is the same as any translate-train model: the sentence or paragraph in English (or any other source language) <strong>(E)</strong> along with its corresponding translated version in the target foreign language <strong>(F)</strong> you want to train.</p><p>FILTER is also a 3 stage architecture like translate-train. But that’s where the similarities stop. Where translate-train uses either <strong>(E) </strong>and <strong>(F) </strong>as input, FILTER uses both <strong>(E) </strong>and <strong>(F)</strong>. These sentences both go through 2 copies (one for each language) of a <em>“local” </em>transformer, which has <em>m</em> layers, to learn unique embeddings for each of the languages.</p><p>The output from both these <em>“local” </em>transformers is then fed into a cross-lingual <em>“fusion”</em> transformer<strong> </strong>with <em>k</em> layers. Here FILTER tries to glean information and learn relationships across <strong>(E)</strong> and <strong>(F)</strong>.</p><p>Finally, there are 2 <em>“domain” </em>transformers (which are once again copies of each other) that have 24-<em>k</em>-<em>m</em> layers and are both task and language specific. The label provided to each <em>“domain” </em>transformer is the corresponding language’s label.</p><p>The numbers <em>m</em> and <em>k</em> are hyper parameters that can be tuned based on the task as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OnvIEv8SS4QYf1C7ePxppQ.png" /><figcaption>The FILTER architecture. The 2 “local” transformers (m layers) and the 2 “domain” transformers (24-k-m layers) share parameters between them i.e. they are copies of one another.</figcaption></figure><p>For simple tasks like classification, the labels remain the same in both languages and it’s easy to train the final language specific layers. But what about for tasks like question answering, entity recognition or part of speech tagging? The labels may not be directly applicable to the target translated text because of the way different languages structure their sentences. How do you train the target language part of the model?</p><p>FILTER has a solution for this too using <a href="https://towardsdatascience.com/patient-knowledge-distillation-88969fffb4ba">knowledge distillation</a>. First train a teacher model using FILTER with just the source language (generally English) label. Once you have this teacher model, train a student model with the target language labels being the output from the task specific target transformer of the teacher model. This way, the student learns all the hidden knowledge that the teacher has as well.</p><p>Since the final transformer is task specific, FILTER can be easily applied to a variety of tasks by just changing the final transformer. To prove the generalizability of FILTER, the researchers applied it to the multi-task datasets of <a href="https://sites.research.google/xtreme">XTREME</a> and <a href="https://microsoft.github.io/XGLUE/">XGLUE</a>. FILTER was able to perform well on a multitude of tasks like multi-language question answering, sentence retrieval, sentence-pair classification, named entity recognition and more to <strong>achieve the #1 position on both these complex leaderboards!</strong></p><p>Here’s a <a href="https://arxiv.org/pdf/2009.05166.pdf">link</a> to the paper if you want to know more about the FILTER model and click <a href="https://aka.ms/mmai">here</a> to see more of our publications and other work.</p><p><strong>References</strong></p><ol><li>Hu, J.; Ruder, S.; Siddhant, A.; Neubig, G.; Firat, O.; and Johnson, M. 2020. <strong>Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization.</strong> In International Conference on Machine Learning.</li><li>Liang Y, Duan N, Gong Y, et al. <strong>XGLUE: A new benchmark dataset for cross-lingual pre-training, understanding and generation[J]</strong>. arXiv preprint arXiv:2004.01401, 2020.</li><li>Yuwei Fang, Shuohang Wang, Zhe Gan, Siqi Sun, Jingjing Liu. 2020. <strong>FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding. </strong>arXiv preprint arXiv:2009.05166</li><li><a href="https://azure.microsoft.com/en-us/services/cognitive-services/translator/?">Translation tool</a> to convert text from English to a foreign language and vice versa</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4bfa6d12377f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/filter-understand-foreign-languages-better-4bfa6d12377f">FILTER: Understand Foreign Languages Better</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DiscoBERT: A BERT that shortens your reading time]]></title>
            <link>https://medium.com/data-science/discobert-a-bert-that-shortens-your-reading-time-be49d03e1ff?source=rss-7f2959a1653------2</link>
            <guid isPermaLink="false">https://medium.com/p/be49d03e1ff</guid>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[bert]]></category>
            <category><![CDATA[summarization]]></category>
            <category><![CDATA[using-pretrained-model]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Rohit Pillai]]></dc:creator>
            <pubDate>Sat, 22 Aug 2020 02:21:14 GMT</pubDate>
            <atom:updated>2020-08-24T20:54:20.771Z</atom:updated>
            <content:encoded><![CDATA[<h4>Use BERT to summarize all those long documents that you don’t want to read</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*II9DRnW7W-1wrMDe" /><figcaption>Photo by <a href="https://unsplash.com/@brady_bellini?utm_source=medium&amp;utm_medium=referral">Brady Bellini</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>With the world being so connected now, we’re constantly being bombarded with information from a variety of different sources and this can be overwhelming. Social media has also transformed the way information is presented to us, with apps like Instagram and Pinterest focusing mostly on visual content over text. This makes it less appealing to read large swathes of text when we have to.</p><p>But what if those large swathes of text could be converted into a summary of just the key points? In comes machine learning to the rescue again.</p><p>Summarization has been a task of interest to the Natural Language Processing community for many years now and there are 2 broad approaches used: <strong>abstractive</strong> and <strong>extractive</strong>. The abstractive method focuses on understanding the entire text and generating a summary one word at a time. Extractive summarization, on the other hand, focuses on selecting important information (typically sentences) from the text and stitching them together to form a summary.</p><p>Abstractive summaries are often more concise than extractive models but may be grammatically incorrect because generating words based on probabilities isn’t the most intuitive way to write summaries. However, both these approaches suffer from a common problem. They can’t keep track of long-range context in a document. For example, if a news article talked about Leo Messi in the 1st paragraph and mentioned him again in the 4th, both these approaches would struggle to link the contexts of those mentions. For abstractive models (which are mostly Seq2Seq models), this is because Seq2Seq models give more importance to more recent words than the others while extractive models are generally meant for a few sentences or a paragraph rather than a whole document.</p><p>The researchers at Microsoft Dynamics 365 AI Research aim to improve these summarization techniques by proposing their own model, <strong>Discourse-Aware BERT for Text Extraction </strong>(<strong>DiscoBERT</strong>). The name gives away the fact that it’s an extractive model, but with a twist. Unlike other extractive models, DiscoBERT extracts <strong>Elementary Discourse Units</strong> (<strong>EDUs</strong>, which are parts of sentences), rather than sentences for its summary. This helps the generated summary be concise because a whole sentence may contain some irrelevant information which don’t exist in EDUs.</p><p>It also solves the long-range context problem by creating 2 different graphs of the document. The first graph, <strong>the</strong> <strong>RST graph</strong>, is based on Rhetorical Structure Theory. This theory claims that the structure of text can be boiled down to a tree of independent units. In a similar fashion, the RST graph tries to analyze the dependency among these units and identify their roles in the sentence. The second graph, <strong>the Coreference graph</strong>, focuses on building an understanding of the key entities (people, organizations and more), events and themes that are being discussed throughout the text.</p><p>Through these 2 graphs, DiscoBERT understands what the important areas of the text are and ranks the EDUs based on this. To generate the summary, DiscoBERT selects the top <em>N</em> ranked EDUs (with dependencies also being taken into account) and stitches them together chronologically.</p><p>The researchers validate DiscoBERT by running it on 2 news datasets, one comprised of articles from The New York Times and the other from CNN and DailyMail which require a model to summarize each article. DiscoBERT achieves new state of the art performance on both these datasets.</p><p>“<em>Most industrial news apps did well on news recommendation, but we want to deliver the news in a more efficient and accessible way.</em> ” said Jingjing Liu, Senior Principal Research Manager from Microsoft Dynamics 365 AI Research who oversees the project. So the next time you’re on a short bus ride and want a quick brief of the daily news, let DiscoBERT come to your rescue!</p><p>Here’s a <a href="https://arxiv.org/pdf/1910.14142.pdf">link</a> to the paper if you want to know more about DiscoBERT, a <a href="https://github.com/jiacheng-xu/DiscoBERT">link</a> to the code if you want to try to train it to summarize your stories, reports and more and click <a href="https://aka.ms/mmai">here</a> to see more of our publications and other work.</p><p><strong>References</strong></p><p>1. William C Mann and Sandra A Thompson, 1988, <strong>Rhetorical structure theory: Toward a functional theory of text organization</strong>, Text-interdisciplinary Journal for the Study of Discourse, 8(3):243–281.</p><p>2. Xu, Jiacheng, Zhe Gan, Yu Cheng, and Jingjing Liu, <strong>Discourse-Aware Neural Extractive Text Summarization</strong>, In <em>Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics</em>, pp. 5021–5031. 2020.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=be49d03e1ff" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/discobert-a-bert-that-shortens-your-reading-time-be49d03e1ff">DiscoBERT: A BERT that shortens your reading time</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Distill-BERT: Using BERT for Smarter Text Generation]]></title>
            <link>https://medium.com/swlh/distill-bert-using-bert-for-smarter-text-generation-aa65ba6facf0?source=rss-7f2959a1653------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa65ba6facf0</guid>
            <category><![CDATA[bert]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[using-pretrained-model]]></category>
            <category><![CDATA[knowledge-distillation]]></category>
            <dc:creator><![CDATA[Rohit Pillai]]></dc:creator>
            <pubDate>Fri, 07 Aug 2020 08:05:43 GMT</pubDate>
            <atom:updated>2020-08-08T06:07:20.944Z</atom:updated>
            <content:encoded><![CDATA[<h4>Generate better stories, essays, answers and more using BERT’s knowledge</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GN3OUaEfVTR0yqmm" /><figcaption>Photo by <a href="https://unsplash.com/@milkovi?utm_source=medium&amp;utm_medium=referral">MILKOVÍ</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The field of natural language processing is now in the age of large scale pretrained models being the first thing to try for almost any new task. Models like BERT, RoBERTa and ALBERT are so large and have been trained with so much data that they are able to generalize their pre-trained knowledge to understand any downstream tasks that you can use them for. But that’s all they can do — understand. If you wanted to answer a question that wasn’t multiple choice, write a story or an essay or anything that required free form writing, you’d be out of luck.</p><p>Now don’t get me wrong, just because BERT-like models can’t write stories, that doesn’t mean that there aren’t models out there that can. Introducing the Sequence to Sequence (Seq2Seq) model. When we write a story, we write the next word, sentence or even paragraph based on what we’ve written so far. This is exactly what Seq2Seq models are designed to do. They predict the most likely next word based on all the words they’ve seen so far by modeling them as a time series i.e. the order of the previous words matters.</p><p>Seq2Seq models have been around for a while and there are several variants that are used for text generation tasks like summarization and translating one language to another. The exploration of Seq2Seq models has culminated in the development of models like GPT-2 and GPT-3, which can complete news snippets, stories, essays and even <a href="https://delian.substack.com/p/quick-thoughts-on-gpt3">investment strategies</a> — all from a few sentences of context! Forewarning though, not all these generated pieces of text make very much sense when you read them — probability distributions over words can only take you so far.</p><p>A few of the fundamental units used in designing these models are Recurrent Neural Networks (RNNs), Long Short Term Memory (LSTM) and Transformers (a combination of an encoder and a decoder that learns a representation for words using numbers) — which also form the backbone of BERT-like models and GPT-2/3.</p><p>A natural question now arises — if a Seq2Seq model is used as the backbone of both BERT-like models and GPT, why can the BERT-like models not generate text? It’s because they’re trained in a way that considers both the future and past context. During training, these models are given sentences with a few missing words as input and they’re expected to predict these missing words. To predict the missing word, they need to know what the words before mean as well as those after. In this spirit, there has been work done on trying to get BERT-like models to work for text generation such as Yang et al.’s CT-NMT</p><p>Another train of thought of BERT-like models for text generation is based on this question: Can the knowledge of future words that these models have from their training help Seq2Seq models formulate more coherent sentences instead of just predicting the next word? This is exactly the problem that the researchers at Microsoft Dynamics 365 AI Research try to answer using Distill-BERT.</p><p>They use <a href="https://towardsdatascience.com/patient-knowledge-distillation-88969fffb4ba">knowledge distillation</a> to transfer the knowledge from a teacher BERT model to a student Seq2Seq model, while also maintaining the original Seq2Seq goal of predicting the most likely next word. This way, the student model retains the best of both worlds. A more formal explanation of this technique is shown in the below equations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P8hA-O16WqzcsAaC--BlPw.png" /><figcaption>The objective used to train the student Seq2Seq model</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ut-bBqpauUVc0QGcKH6Log.png" /><figcaption>BERT part of the objective</figcaption></figure><p>Here <em>PΦ</em>(yt) is a list of probabilities predicted by BERT of all words being relevant at position t in the generated text.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wG97BR60HWZUMGSE08ocYg.png" /><figcaption>Original Seq2Seq objective</figcaption></figure><p>Once the student model is trained, the teacher BERT model is no longer needed and only the student model is used to generate the text. This means that at generation time, there are no additional resources required for Distill-BERT. This technique is also teacher agnostic. This means that any BERT-like model like RoBERTa, ALBERT, BERT and more can be used to distil knowledge to the student.</p><p>To prove their method works, the researchers distil BERT’s knowledge to train a student transformer and use it for German-to-English translation, English-to-German translation and summarization. The student transformer shows significant improvement over a regular transformer without BERT and is even able to achieve state of the art performance on German-to-English translation.</p><p>They also apply this knowledge to a student RNN, showing that the technique is also student agnostic. This RNN is applied for English-to-Vietnamese translation and shows improvement as well.</p><p>Here’s a <a href="https://arxiv.org/pdf/1911.03829.pdf">link</a> to the paper if you want to know more about Distill-BERT , a <a href="https://github.com/ChenRocks/Distill-BERT-Textgen">link</a> to the code if you want to try to train your own Seq2Seq model and click <a href="https://aka.ms/mmai">here</a> to see more of our publications and other work.</p><p><strong>References</strong></p><ol><li>Radford, Alec, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever, <strong>Language models are unsupervised multitask learners</strong>. <em>OpenAI Blog</em> 1, no. 8 (2019): 9.</li><li>Brown, Tom B., Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan et al., <strong>Language models are few-shot learners</strong>, <em>arXiv preprint arXiv:2005.14165</em> (2020).</li><li>Cristian Bucilu, Rich Caruana, and Alexandru Niculescu-Mizil, <strong>Model compression</strong> (2006), In KDD.</li><li>Jiacheng Yang, Mingxuan Wang, Hao Zhou, Chengqi Zhao, Yong Yu, Weinan Zhang and Lei Li, <strong>Towards making the most of bert in neural machine translation</strong> (2019), arXiv preprint arXiv:1908.05672</li><li>Chen, Yen-Chun, Zhe Gan, Yu Cheng, Jingzhou Liu, and Jingjing Liu, <strong>Distilling Knowledge Learned in BERT for Text Generation</strong>, In <em>Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics</em>, pp. 7893–7905. 2020.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aa65ba6facf0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/distill-bert-using-bert-for-smarter-text-generation-aa65ba6facf0">Distill-BERT: Using BERT for Smarter Text Generation</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[VILLA: A Generic Adversarial Training technique for Vision-and-Language]]></title>
            <link>https://medium.com/data-science/villa-a-generic-adversarial-training-technique-for-vision-and-language-5e6cd5d3abd7?source=rss-7f2959a1653------2</link>
            <guid isPermaLink="false">https://medium.com/p/5e6cd5d3abd7</guid>
            <category><![CDATA[using-pretrained-model]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[adversarial-training]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Rohit Pillai]]></dc:creator>
            <pubDate>Mon, 13 Jul 2020 20:16:48 GMT</pubDate>
            <atom:updated>2020-07-14T14:06:26.982Z</atom:updated>
            <content:encoded><![CDATA[<h4>The first multimodal adversarial training technique for image and text models</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VN6bGHQnYkd-l2eJlDA_yA.jpeg" /><figcaption>Image by Gerd Altmann from Pixabay</figcaption></figure><p>Building large scale pretrained models for vision-and-language multimodal learning has been a booming area of machine learning research recently. Models like <a href="https://towardsdatascience.com/uniter-d979e2d838f0">UNITER</a>, ViLBERT and LXMERT are trained on millions of image-text pairs with different objective functions and are able to learn an effective generic joint representation of image and text.</p><p>These representations are then finetuned on a specific task to achieve state of the art performance on a myriad of image and language tasks like image QA, image-text retrieval, image grounding, image inference and more. When finetuning on a specific task, the amount of data available is only a fraction of the pretraining data used. Such small amounts of data often lead to the model <strong>overfitting</strong> and beats the goal of trying to build a generalizable embedding for image and text.</p><p>So how do we prevent this overfitting? Adversarial training techniques for single modal tasks on images and text have been shown to make a model more robust and generalizable. This sort of training can be done by</p><ol><li>Creating human understandable adversarial examples (as in Szegedy et al.)</li><li>Adding adversarial perturbations to the embedding space (as in <a href="https://towardsdatascience.com/freelb-a-generic-adversarial-training-method-for-text-92ac25828495">FreeLB</a>).</li></ol><p>Applying adversarial training techniques to multimodal learning are a natural extension to the single modal techniques and this is exactly what the researchers at Microsoft Dynamics AI 365 Research did.</p><p>They proposed <strong>VILLA (Vision-and-Language Large-scale Adversarial training)</strong>, which is the first known adversarial training technique designed for vision and language. VILLA follows the second method of adversarial training since their goal is to build a robust model, irrespective of whether humans understand the adversarial examples. For the text modality, they add adversarial perturbations to the word embeddings. Previous works on image adversarial training add noise to the image pixels and find that robustness comes at the cost of performance. In VILLA, these adversarial perturbations are added to the extracted image-region features instead of the pixels and this improves both robustness and performance. During training, the perturbations are only added to one modality at a time while the other modality is kept constant as seen in the picture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EPk_eVQg7UAecF1orl8Xxg.png" /></figure><p>VILLA also leverages “free” training strategies (Shafahi et al., 2019; Zhang et al. 2019) obtains the gradients of parameters with almost no extra cost when computing the gradients of inputs. This way, the model sees more examples at no additional cost and training becomes more cost efficient.</p><p>Adversarial training techniques generally ensure that the label of the adversarial example is the same as the original example. VILLA not only tries to do this but also tries to match the predicted class probabilities of both examples using KL-divergence. By doing this, the training loss function becomes smoother and the model can estimate it better. This further increases the model’s performance.</p><p>VILLA can be applied to both pretraining and finetuning stages to improve the generalizability of the learned embedding. When applied to UNITER-large, VILLA further improves its performance from 74.02% to 74.87% on VQA, and from 62.8% to 65.7% on VCR. With ensemble, the VQA performance is further improved to 75.85%. VILLA also improved UNITER’s results to state of the art on 4 additional tasks.</p><p>VILLA is also model agnostic. It was applied to another large scale pretrained model, LXMERT and showed a gain in performance on VQA.</p><p>Here’s a <a href="https://arxiv.org/pdf/2006.06195.pdf">link</a> to the paper if you want to go into more detail about how VILLA works and click <a href="https://aka.ms/mmai">here</a> to see more of their publications and other work.</p><p><strong>References</strong></p><ol><li><a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Chen%2C+Y">Yen-Chun Chen</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Li%2C+L">Linjie Li</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Yu%2C+L">Licheng Yu</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Kholy%2C+A+E">Ahmed El Kholy</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Ahmed%2C+F">Faisal Ahmed</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Gan%2C+Z">Zhe Gan</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Cheng%2C+Y">Yu Cheng</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Liu%2C+J">Jingjing Liu</a>, <strong>UNITER: Learning UNiversal Image-TExt Representations</strong>, European Conference on Computer Vision (ECCV) 2020</li><li>J<a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Lu%2C+J">iasen Lu</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Batra%2C+D">Dhruv Batra</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Parikh%2C+D">Devi Parikh</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Lee%2C+S">Stefan Lee</a>, <strong>ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks</strong>, arXiv preprint, arXiv:1908.02265</li><li>Hao Tan and Mohit Bansal, <strong>Lxmert: Learning cross-modality encoder representations from transformers</strong>, EMNLP 2019.</li><li>C<a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Szegedy%2C+C">hristian Szegedy</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Zaremba%2C+W">Wojciech Zaremba</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Sutskever%2C+I">Ilya Sutskever</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Bruna%2C+J">Joan Bruna</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Erhan%2C+D">Dumitru Erhan</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Goodfellow%2C+I">Ian Goodfellow</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Fergus%2C+R">Rob Fergus</a>, <strong>Intriguing properties of neural networks</strong> (2013), arXiv preprint, arXiv:1312.6199</li><li>Chen Zhu, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Cheng%2C+Y">Yu Cheng</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Gan%2C+Z">Zhe Gan</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Sun%2C+S">Siqi Sun</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Goldstein%2C+T">Tom Goldstein</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Liu%2C+J">Jingjing Liu</a>, <strong>FreeLB: Enhanced Adversarial Training for Language Understanding</strong> (2019), International Conference on Learning Representations, ICLR 2020</li><li>A. Shafahi, M. Najibi, A. Ghiasi, Z. Xu, J. Dickerson, C. Studer, L. Davis, G. Taylor, and T. Goldstein, <strong>Adversarial Training for Free!, Neural Information Processing Systems</strong>, NeurIPS 2019</li><li>Dinghuai Zhang, Tianyuan Zhang, Yiping Lu, Zhanxing Zhu, and Bin Dong, <strong>You only propagate once: Painless adversarial training using maximal principle</strong>, Neural Information Processing Systems, NeurIPS 2019</li><li>Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, Jingjing Liu, <strong>Large-Scale Adversarial Training for Vision-and-Language Representation Learning</strong>, arXiv preprint, arXiv:2006.06195.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e6cd5d3abd7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/villa-a-generic-adversarial-training-technique-for-vision-and-language-5e6cd5d3abd7">VILLA: A Generic Adversarial Training technique for Vision-and-Language</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[BachGAN: Let your imagination run free]]></title>
            <link>https://medium.com/swlh/bachgan-let-your-imagination-run-free-71e6aa66b371?source=rss-7f2959a1653------2</link>
            <guid isPermaLink="false">https://medium.com/p/71e6aa66b371</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[image-generation]]></category>
            <category><![CDATA[generative-adversarial]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Rohit Pillai]]></dc:creator>
            <pubDate>Fri, 15 May 2020 02:12:24 GMT</pubDate>
            <atom:updated>2020-07-13T18:29:51.063Z</atom:updated>
            <content:encoded><![CDATA[<h4>Create a masterpiece without a pencil or paintbrush</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KQhfP-Wl9EcqDcEj" /><figcaption>Photo by <a href="https://unsplash.com/@adliwahid?utm_source=medium&amp;utm_medium=referral">Adli Wahid</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Picture this: One day, you dream up an artwork so awe striking, filled with vibrant colors and beauty that evokes primal feelings in the beholder and can absorb them for hours. You wake up and jump out of bed filled with excitement, ready to bring this magical piece to life. Days later, you realize you’re no Picasso and your execution of this piece could never do justice to what you imagined. You leave your canvas frustrated at your inability to bring magic to the world. But what if this story could go another way? What if you didn’t have to draw this image but could have something draw it for you, something that could bring justice to your creation? Ladies and gentlemen, I present to you the magic of GANs.</p><p>That’s right, Generative Adversarial Networks (GANs) can take in an input and create something completely new or modify it for you. The GAN was first proposed in 2014 and since then it has been used for a multitude of tasks, a lot of which involve creating new images or modifying them. For context, a GAN consists of 2 components: <strong>a generator</strong>, which generates an image from random noise and the input and <strong>a discriminator</strong>, which predicts whether the image generated by the generator is an actual image or not. They both work together in coordination, helping each other get better. The generator is forced to generate better images so that it can fool the discriminator while the discriminator learns to notice even intricacies in an image that could make it “fake”.</p><p>Here are some examples of GANs that function on different inputs.</p><ul><li><strong><em>WarpGAN</em></strong><em> converts a photo of a face to a caricature while </em><strong><em>CP-GAN</em></strong><em> does the inverse, converting a cartoon image to a one of a real face.</em></li><li><strong><em>SketchyGAN</em></strong><em> creates a realistic object or living thing from just a sketch of it.</em></li><li><strong><em>StackGAN</em></strong><em> and </em><strong><em>AttnGAN</em></strong><em> creates images from a text description.</em></li><li><strong><em>StoryGAN</em></strong><em> converts a story (generally a paragraph) into a series of images, one for each sentence in the story.</em></li><li><strong><em>PG²</em></strong><em> uses a person’s picture and a pose map to generate images of the person standing in different positions.</em></li><li><strong><em>Wav2Pix</em></strong><em> recreates a person’s face from just an audio description.</em></li><li><strong><em>SPADE</em></strong><em> takes in a semantic segmentation map (an image which is filled with different colors, one for each object) and fills it with actual pictures of objects.</em></li><li><strong><em>Layout2Im</em></strong><em> converts a layout (a blank image with boxes where objects should be both in the background and foreground) to a complete image.</em></li></ul><p>These GANs are only the tip of the iceberg that is the work that is being done in the image generation and modification domain.</p><p>Now coming back to your masterpiece, you can use any of these GANs or even a combination of them to complete it. Is your masterpiece a breakdown of the human body? You can talk to Wav2Pix and have it create a great face! Why stop there? Convert it to a caricature with WarpGAN. You could even create a piece of someone in multiple poses superimposed on each other with PG2. If you want to create a scenic piece, describe it to either StackGAN or AttnGAN or write a story about it and StoryGAN will do the rest. Think you can sketch it out? Let SketchGAN flesh it out. Feeling lazy? Draw boxes where you want objects to be and let Layout2Im finish the job.</p><p>Feeling even lazier and don’t really want to imagine the background? Here’s the Microsoft Dynamics 365 AI Research team to the rescue. They proposed a new image generation task that requires a model to be able to generate an image from a layout that specifies boxes only in the foreground and no other additional information. Doing this is significantly harder than using richer information like a segmentation map, text or even a complete layout because the model has to be capable of understanding what sort of background cohesively combines all the objects in the image.</p><p>To solve this task, the team built the <strong>Background Hallucination Generative Adversarial Network (BachGAN)</strong>. Its key components are a retrieval module and a fusion module, which when combined can generate a visually consistent background on-the-fly for any foreground object layout. Given a salient object layout, BachGAN generates an image via two steps:</p><p><strong>(i)</strong> <em>The background retrieval module selects a set of segmentation maps most relevant to the given object layout from a large candidate pool</em></p><p><strong>(ii)</strong> <em>These candidate layouts are encoded by the background fusion module to generate a best-matching background.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CLXwf5AXNipwlKuQUAIPFw.png" /><figcaption>Images created using a segmentation map (SPADE, top row) vs foreground layout(BachGAN, bottom row).</figcaption></figure><p>BachGAN isn’t restricted to creating an image from just one foreground layout. In fact, it is capable of handling a modification to the layout provided to it and can regenerate a new background that cohesively ties in both the old objects as well as the new objects added to the foreground if necessary. It also maintains the positions of the old objects in the new image it generates so it looks like nothing has changed except the addition of the new object.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_nPzv6OMTmUBskro9sywQQ.png" /><figcaption>Sequential image generation by BachGAN on an example from Ade20k</figcaption></figure><p>Every model has to be tested on a dataset to prove its ability and BachGAN is no different. It was tested on the CityScapes dataset, which is composed of images of streets taken in cities, and the Ade20k dataset, which consists of a variety of scenes in different settings. BachGAN was able to outperform the baselines on both these datasets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HrEArBzMR-XuKTXK0p6wOA.png" /><figcaption>Sequential image generation by BachGAN on an example from CityScapes</figcaption></figure><p>With the creation of BachGAN, you now have no excuses to stop yourself from creating that masterpiece that you’ve always dreamed of. So go on and unleash all that untapped creativity!</p><p>Here’s a <a href="https://arxiv.org/pdf/2003.11690.pdf">link</a> to our paper if you want to know more about BachGAN or the new task and click here to see more of our publications and other work.</p><p><strong>References</strong></p><ol><li>Shi, Yichun, Debayan Deb, and Anil K. Jain. <strong>Warpgan: Automatic caricature generation</strong>. ICCV, 2019.</li><li>Junhong Huang and Mingkui Tan and Yuguang Yan and Chunmei Qing and Qingyao Wu and Zhu Liang Yu. <strong>Cartoon-to-Photo Facial Translation with Generative Adversarial Networks. </strong>ACML, 2018</li><li>Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. <strong>Attngan: Fine-grained text to image generation with attentional generative adversarial networks.</strong> CVPR, 2018.</li><li>Chen, Wengling, and James Hays. <strong>Sketchygan: Towards diverse and realistic sketch to image synthesis. </strong>ICCV, 2018.</li><li>Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris Metaxas. <strong>Stackgan: Text to photo-realistic image synthesis with stacked generative adversarial networks. </strong>ICCV, 2017</li><li>Yitong Li, Zhe Gan, Yelong Shen, Jingjing Liu, Yu Cheng, Yuexin Wu, Lawrence Carin, David Carlson, Jianfeng Gao. <strong>StoryGAN: A Sequential Conditional GAN for Story Visualization. </strong>CVPR, 2019</li><li>Liqian Ma, Xu Jia, Qianru Sun, Bernt Schiele, Tinne Tuytelaars, and Luc Van Gool. <strong>Pose guided person image generation.</strong> NIPS, 2017.</li><li>Duarte A, Roldan F, Tubau M, et al.<strong> Wav2Pix: speech-conditioned face generation using generative adversarial networks.</strong> ICASSP<em>,</em> Vol. 3. 2019.</li><li>Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. <strong>Semantic image synthesis with spatially-adaptive normalization.</strong> CVPR, 2019.</li><li>Bo Zhao, Lili Meng, Weidong Yin, and Leonid Sigal. <strong>Image generation from layout.</strong> CVPR, 2019.</li><li>Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. <strong>The cityscapes dataset for semantic urban scene understanding.</strong> CVPR, 2016</li><li>Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. <strong>Scene parsing through ade20k dataset.</strong> CVPR, 2017.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=71e6aa66b371" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/bachgan-let-your-imagination-run-free-71e6aa66b371">BachGAN: Let your imagination run free</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[VIOLIN: Do you really understand videos?]]></title>
            <link>https://medium.com/swlh/violin-do-you-really-understand-videos-fe07f7affd9e?source=rss-7f2959a1653------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe07f7affd9e</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[multimodal-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[violin]]></category>
            <dc:creator><![CDATA[Rohit Pillai]]></dc:creator>
            <pubDate>Sun, 03 May 2020 04:50:06 GMT</pubDate>
            <atom:updated>2020-07-13T18:30:56.551Z</atom:updated>
            <content:encoded><![CDATA[<h4>A new task and dataset that requires multi-modal understanding of videos and text</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XDJEBk1ci8LqOwOn" /><figcaption>Photo by <a href="https://unsplash.com/@cameronvisuals?utm_source=medium&amp;utm_medium=referral">Cameron Mourot</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Our media consumption through visual means has grown drastically over the past decade. In fact, <a href="https://www.marketingcharts.com/digital/video-110520">between 2013 and 2018, the rate of video consumption grew by a stunning 32% every year</a> and we’re showing no signs of slowing down. Social media platforms like Tiktok, Pinterest and Instagram are filled exclusively with images and videos and businesses are increasingly switching to visual platforms like PowerBI, Tableau and SAS Visual Analytics. With such a large amount of our information coming from visual cues, the natural next step for machine learning researchers is to build models that can understand and analyze them.</p><p>The field of computer vision has been around to focus on understanding images way before the advent of deep learning and they’ve done a very good job at this. Today’s computer vision models can detect and classify objects in images (even if there are multiple) and even retrieve images that are similar to an input image (think Google image search). They’re even capable of combining their knowledge of images and understanding other modalities like text. An example of this would be the Visual Entailment task. Given an image, a premise and hypothesis (both in text), a model is expected to judge whether the hypothesis can be confirmed based on the image. Similarly, Visual QA and Visual Dialog require models to answer multiple complex questions about an image. Image Captioning is another multi-modal task where models must generate a caption for a provided image.</p><p>While images have been extensively explored in both unimodal and multi-modal contexts, the same cannot be said for videos. There are several challenging unimodal tasks for video like action recognition, object detection and video classification. However, there are only a few tasks that combine videos with other modalities. A few examples of multi-modal tasks include Video QA (answer questions about a video), Video Captioning (caption a video) and Video Reasoning (identifying events in a video, their causes and potential following events).</p><p>To further multi-modal understanding of videos and text, the researchers at Microsoft Dynamics 365 AI Research proposed a new task, Video-and-Language Inference. This is like Visual Entailment but for videos. Given a video clip with aligned subtitles as premise, paired with a natural language hypothesis based on the video content, a model needs to infer whether the hypothesis is entailed or contradicted by the given video clip.</p><p>In lieu of this task, the team also created a new dataset called <strong>VIOLIN (VIdeO-and-Language INference)</strong>. The dataset is comprised of 95,322 video-hypothesis pairs from 15,887 video clips, spanning over 582 hours of video collected from popular TV shows like Friends and How I Met Your Mother as well as from movie clips from YouTube. While datasets like TVQA that use clips from TV shows, they focus on identifying explicit information from the video. However, in order to successfully answer all the examples in the VIOLIN dataset, a model is required to not only be capable of identifying explicit information (e.g., identifying objects and characters in the video) but also must have in-depth commonsense reasoning abilities (e.g., inferring causal relations of events in the video). The explicit information identification comprises 54% of VIOLIN and the remaining 46% requires commonsense reasoning, making this a significantly more challenging dataset than anything currently available.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jzRbhooUybF6gyyeGvm9lw.png" /><figcaption>Some examples of different types of reasoning from the dataset</figcaption></figure><p>The team also built a baseline model to benchmark against for this dataset. It combines video features, that are extracted using a CNN like ResNet, and text features (both subtitles and hypothesis) , encoded by an encoder like the one used by BERT, using 2 fusion modules and a LSTM. There’s an FC layer on top of this and a sigmoid for normalization after which the prediction is made.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sd5_mapw_Z4rFaBPPPLYwA.png" /><figcaption>Baseline model architecture</figcaption></figure><p>So what are you waiting for? Go on and start working on this new challenge and hopefully you’re able to beat this baseline!</p><p>Here’s a <a href="https://arxiv.org/pdf/2003.11618.pdf">link</a> to our paper if you want to know more about VIOLIN’s composition, how it was made or the proposed baseline model, and click <a href="http://aka.ms/mmai">here</a> to see more of our publications and other work.</p><p><strong>References</strong></p><ol><li>Jingzhou Liu, Wenhu Chen, Yu Cheng, Zhe Gan, Licheng Yu, Yiming Yang, Jingjing Liu, <strong>VIOLIN: A new Video-and-Language Inference task</strong>, The Conference on Computer Vision and Pattern Recognition (CVPR) 2020.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe07f7affd9e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/violin-do-you-really-understand-videos-fe07f7affd9e">VIOLIN: Do you really understand videos?</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[UNITER]]></title>
            <link>https://medium.com/data-science/uniter-d979e2d838f0?source=rss-7f2959a1653------2</link>
            <guid isPermaLink="false">https://medium.com/p/d979e2d838f0</guid>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[multimodal-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[uniter]]></category>
            <dc:creator><![CDATA[Rohit Pillai]]></dc:creator>
            <pubDate>Thu, 05 Mar 2020 02:17:37 GMT</pubDate>
            <atom:updated>2020-08-03T02:46:23.407Z</atom:updated>
            <content:encoded><![CDATA[<h3>UNITER: Combining image and text</h3><h4>Learning a joint representation of image and text that everything can use</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fJe01uNm5Bum44-UQcI04w.jpeg" /><figcaption>Image by Patricia Hébert from Pixabay</figcaption></figure><p>Multimodal learning is omnipresent in our lives. Humans absorb content in different ways, whether through pictures (visual), text, spoken explanations (audio) to name a few. Each of these sources of knowledge is known as a mode. In fact, we often learn through a combination of these modes, giving everyone a unique learning experience. The McGurk effect — when seeing a person say something (ga-ga) but hearing different audio (ba-ba) causes the observer to perceive a third sound (da-da) — is a prime example of the ways different modalities interact with one another. Inspired by this, machine learning researchers also started compiling datasets and tasks that required models to understand multiple modes to be successful.</p><p>Since early multimodal tasks were inspired by the McGurk effect, a lot of them involved just audio and images. As time went by, the varieties and number of modalities for tasks become more diverse. Today there are tasks using images and text (Visual QA, Visual Commonsense Reasoning, Image-Text Retrieval) or images, audio and text (Video Sentiment Analysis, CMU-MOSEI). Unimodal tasks have been well explored and good representations can be extracted from each modality. For example, word embeddings like GloVe and Word2Vec can be used to generate embeddings for text while convolutional neural networks like ResNet50 and VGG16 are often used to extract features from images. The challenge that arises is composing and/or using these unimodal features to do multimodal tasks.</p><p>Composing these features poses 2 major challenges: <strong>how</strong> and <strong>when</strong> to fuse them. Baltrusaitis et al. identified 2 major techniques of combining unimodal features: joint representations (using a neural network or some other architecture to combine multiple modalities into the same representation space) and coordinated representations (using some measure like cosine distance or correlation to bring the unimodal representations closer while still in different representation spaces). Both these fusion techniques can also happen at different points in the model architecture: early fusion (combining the features at the start and having the full model architecture working on the combined features) or late fusion (running the individual modalities through their own architecture and combining them at the end).</p><p>A lot of contemporary models focus on learning good multimodal representations that work for their specific tasks. As a result, these models cannot be generalized to slightly different uses of the same modalities. However, making a model for each task takes a lot of time and effort. In this work, Microsoft Dynamics 365 AI Research’s UNITER model focuses on learning a generalizable joint embedding for images and text.</p><figure><img alt="UNITER model that learns joint image — text embeddings." src="https://cdn-images-1.medium.com/max/1024/1*KanlBUAJ7wVE5pcv35wcIg.png" /><figcaption>The UNITER model architecture (from top to bottom)</figcaption></figure><p>UNITER uses self-supervised learning on a lot of data, like BERT, to ensure that the learned embeddings are generic. The model takes in image — text pairs and was trained with around 9.5 million of these pairs from 4 different datasets.</p><p>For each pair, it first extracts text features (by combining word position information with the WordPiece tokenized sentence) and image region features (by combining Faster R-CNN features with location information). Once the features are extracted, they go through several layers of transformers to finally learn the joint embedding,</p><p>There are 3 main objectives used in the self-supervised training of the joint embedding. For these tasks, the input is (<strong>w, v</strong>) where <strong>v</strong> is the input image regions and <strong>w</strong> is the paired text and θ are trainable parameters.</p><p><strong>Masked Language Modeling (MLM)</strong></p><p><strong>MLM</strong> masks each word of the input text with the [MASK] token with a probability of 15%. The goal is to predict the masked words based on the rest of the sentence and the paired image. This is done by minimizing the negative log likelihood as shown in the equation below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/414/1*7tjM9digii29MynwaZkFdA.png" /><figcaption>MLM objective</figcaption></figure><p>where <strong>wm </strong>are the masked words and <strong>w\m </strong>are the surrounding words.</p><p><strong>Image Text Matching (ITM)</strong></p><p>For <strong>ITM</strong>, an extra [CLS] token is appended to the beginning of the input text and much like BERT’s [CLS] token which captures the context of the input text, it captures the joint context of the input image-text pair. This token then goes through a scorer <strong>sθ</strong> to measure how well the image and text match. Since each input pair could be positive or negative, a binary cross-entropy loss is used to optimize this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*YHyuKhq-Svb34C-2mkZUPA.png" /><figcaption>ITM objective</figcaption></figure><p>where <strong>y</strong> is 0 for negative pairs and 1 for positive pairs.</p><p><strong>Masked Region Modeling (MRM)</strong></p><p><strong>MRM</strong> is like <strong>MLM</strong> except instead of replacing a region with the [MASK] token, it’s filled in with zeros. However, image features are continuous making it impossible to maximize the log likelihood. Thus, to implement <strong>MRM</strong>, the paper proposes 3 different objective functions that fit into the general equation below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/342/1*riqCSGw_NtCmnFsoZHhO1A.png" /><figcaption>MRM objective</figcaption></figure><p>where <strong>vm </strong>are the masked image regions and <strong>v\m </strong>are the surrounding regions.</p><p>1. <em>Masked Region Feature Regression (MRFR)</em></p><p><em>MRFR</em> tries to match the final joint representation of each masked image region <strong>vm(i)</strong> to its pretrained image embeddings, so that the information passed into the model is retained.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/1*3Aq-kL1wPb2dQFrqPg6USQ.png" /><figcaption>MRFR objective</figcaption></figure><p>where <strong>hθ </strong>converts the joint image embedding to the same size as the input image embedding.</p><p>2. <em>Masked Region Classification (MRC)</em></p><p>UNITER tries to identify <strong>vm(i) </strong>as a type of object in <em>MRC</em>. Faster R-CNN predicts the probability of each region being a type of object with certain probability. This is used as the ground truth label and cross-entropy loss of the prediction is optimized.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/384/1*_ueLfsUzvcbZvy1FOXq1oA.png" /><figcaption>MRC objective</figcaption></figure><p>where K is the number of object classes predicted by Faster R-CNN, gθ(<strong>vm(i)</strong>) converts the region into a K long vector and c(<strong>vm(i)</strong>) is a one hot vector of the truth label.</p><p>3. <em>Masked Region Classification with KL divergence (MRC-kl)</em></p><p><em>MRC-kl</em> is the same as <em>MRC </em>except that Faster R-CNN’s predicted probabilities for each object type are used instead of a fixed label (0 or 1). KL divergence loss is used instead of cross-entropy to match all the probabilities instead of just the ground truth class as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/412/1*zC5zj0wcNoCv5nWI2wTCyw.png" /><figcaption>MRC-kl objective</figcaption></figure><p>where K is the number of object classes predicted by Faster R-CNN, gθ(<strong>vm(i)</strong>) converts the region into a K long vector and c(<strong>vm(i)</strong>) is Faster R-CNN’s class probabilities.</p><p>The team tried several combinations of these pretraining tasks and found that using all the tasks except MRC resulted in the best embedding.</p><p>The best UNITER embedding was finetuned for a multitude of downstream tasks involving images and text. Some of these tasks were <em>Visual QA</em> (answer a question about an image), <em>Visual Entailment</em> (determine if an image entails a sentence) and <em>Image Text Retrieval</em> (retrieve one given the other). Each of these tasks was tested on several benchmarks and UNITER was able to achieve state of the art performance on 13 different benchmarks, truly showcasing the UNITER embedding’s generalizability.</p><p>Here’s a <a href="https://arxiv.org/pdf/1909.11740.pdf">link</a> to our paper if you want to go into more detail about UNITER and click <a href="http://aka.ms/mmai">here</a> to see more of our publications and other work.</p><p><strong>References</strong></p><ol><li><a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Baltru%C5%A1aitis%2C+T">Tadas Baltrušaitis</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Ahuja%2C+C">Chaitanya Ahuja</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Morency%2C+L">Louis-Philippe Morency</a>, <strong>Multimodal Machine Learning: A Survey and Taxonomy </strong>(2018), IEEE transactions on pattern analysis and machine intelligence 41, no. 2 pp: 423–443.</li><li><a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Chen%2C+Y">Yen-Chun Chen</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Li%2C+L">Linjie Li</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Yu%2C+L">Licheng Yu</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Kholy%2C+A+E">Ahmed El Kholy</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Ahmed%2C+F">Faisal Ahmed</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Gan%2C+Z">Zhe Gan</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Cheng%2C+Y">Yu Cheng</a>, <a href="https://arxiv.org/search/cs?searchtype=author&amp;query=Liu%2C+J">Jingjing Liu</a>, <strong>UNITER: Learning UNiversal Image-TExt Representations</strong> (2019), arXiv preprint arXiv:1909.11740.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d979e2d838f0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/uniter-d979e2d838f0">UNITER</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>