<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pooja Rao on Medium]]></title>
        <description><![CDATA[Stories by Pooja Rao on Medium]]></description>
        <link>https://medium.com/@poojarao126?source=rss-46ba9746950e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*t4ZOKZb8lJ-RLl2iEoWS0w.jpeg</url>
            <title>Stories by Pooja Rao on Medium</title>
            <link>https://medium.com/@poojarao126?source=rss-46ba9746950e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 04:41:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@poojarao126/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Good Citizens of Research Community]]></title>
            <link>https://medium.com/@poojarao126/good-citizens-of-research-community-14bb0da0de04?source=rss-46ba9746950e------2</link>
            <guid isPermaLink="false">https://medium.com/p/14bb0da0de04</guid>
            <category><![CDATA[cvpr2018]]></category>
            <category><![CDATA[research]]></category>
            <category><![CDATA[helping-others]]></category>
            <category><![CDATA[cvpr]]></category>
            <category><![CDATA[research-paper-writing]]></category>
            <dc:creator><![CDATA[Pooja Rao]]></dc:creator>
            <pubDate>Mon, 14 Sep 2020 13:31:14 GMT</pubDate>
            <atom:updated>2020-09-22T10:19:38.150Z</atom:updated>
            <content:encoded><![CDATA[<h4>Highlights of the Workshop — Good Citizen of CVPR</h4><figure><img alt="Silhouette Of People Helping Each Other" src="https://cdn-images-1.medium.com/max/898/0*zyfg9zP2dXK9h8FZ.jpeg" /><figcaption>Image from <a href="https://www.clipart.email/download/2272037.html">clipart.email</a></figcaption></figure><p>On average, I am on the road for 2 hours every day. I have much time commuting to and fro my workplace. Thanks to the infamous traffic of Bangalore. Instead of cribbing (trust me, I did a lot of that) I decided to put my time to good use. I listen to podcasts, sleep (sleeping is a good use of time), watch videos with less visual content and more informational audio, and so on. This blog post is also a result of my yearning to make good use of my travel time.</p><p>In my quest to find informational content, I stumbled across videos of the premier Computer Vision Conference Workshop CVPR 2018 — <a href="https://www.cc.gatech.edu/~parikh/citizenofcvpr/">Good Citizen of CVPR</a>. This workshop is a one-of-a-kind initiative to help the research community, especially novice researchers. The line-up consisted of some of the giants and distinguished scholars in the field. I found the talk relevant to all researchers, irrespective of the area of research. Here is an attempt to summarise it. I hope you like it as much as I enjoyed the videos.</p><p><em>Note: Being a junior researcher, I have outlined points which I think are relevant. I would like to refer the readers to the </em><a href="https://www.youtube.com/watch?v=MKUCz_3Ee0A"><em>original videos</em></a><em> for some more pointers and great talks.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/1*uyZiQg_rhgU21WVJ8d3eWw.png" /><figcaption>Introductory Remarks at the Workshop. Source: <a href="https://www.cc.gatech.edu/~parikh/citizenofcvpr/static/slides/parikh_good_citizen_cvpr18_opening.pdf">Slides</a></figcaption></figure><p>As rightly put in the opening remarks, non-technical factors are the things that we care about the least. Somewhere between the last-minute experiments and the paper drafts, it gets lost. I opine that these talks can act as reference material and guideline.</p><h4>What does it take to write a good research paper?</h4><p>I highly recommend the content by <a href="https://billf.mit.edu/">Bill Freeman</a>, <a href="https://billf.mit.edu/sites/default/files/documents/cvprPapers.pdf">here</a> and <a href="http://people.csail.mit.edu/billf/publications/How_To_Do_Research.pdf">here</a>. I particularly like this graph from the first talk.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/665/1*wd_qDEMXxzLv2004tTHMkQ.png" /><figcaption>How to Write a Good Research Paper by <a href="https://billf.mit.edu/">Bill Freeman</a>. Source: <a href="https://www.cc.gatech.edu/~parikh/citizenofcvpr/static/slides/freeman_how_to_write_papers.pdf">Slides</a></figcaption></figure><blockquote>Always write good, creative and original papers which have a significant impact on your career. Nothing even remotely less.</blockquote><p>In my opinion, this applies to anything we write. Be it blog posts, reports, or white papers. Creative and original work always gains traction.</p><p>Paper organisation is one of the significant non-technical factors that we tend to ignore. The below slide gives a concrete outline of the same. The main idea here is to make the paper an easy-read for anyone interested.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*1IphMpuNFH6RmCL_qXxKEg.png" /><figcaption>How to Write a Good Research Paper by <a href="https://billf.mit.edu/">Bill Freeman</a>. Source: <a href="https://www.cc.gatech.edu/~parikh/citizenofcvpr/static/slides/freeman_how_to_write_papers.pdf">Slides</a></figcaption></figure><ul><li>The speaker stresses on a dynamite introduction. What is the paper about, what problem it solves and why that problem. This point applies to abstract as well to make a compelling case to the reader and pike the interest to read the whole article/paper.</li><li>One teaser figure which encapsulates the idea we are trying to convey.</li><li>Underutilized technique: explain the main idea with a simple, toy example.</li><li>The paper should clearly describe the main idea with ample experiments and factual results in the experiments section. The conclusion should not repeat what’s already said but address what the paper has opened up.</li></ul><p>The speaker feels that future work is a weak way to end a paper. It would be like conveying the things that could not be done before the deadline. I’m guilty of doing this too. It seems like an easy way to end a paper. Maybe if written subtly, it could be a good ending. Would you end a blog post with future work? 🤔 Let me know your thoughts in the comments section.</p><p>General writing tips:</p><ul><li>Keep the reader in mind. Treat the reader like a guest in your house.</li><li>Be succinct. Omit needless words.</li><li>The figures and captions can help tell a story.</li><li>Have a kind and gracious tone. Be clear and reliable.</li></ul><p>The talk also captures some interesting and informative quotes from various researchers. Do check them out!</p><h4>Research Culture</h4><p>The next few researchers talk about the research culture as a whole and particularly in academia and conferences. They talk about how to be welcoming and make sure all the attendees and especially the newbies don’t feel out of place.</p><p>I’m particularly interested in this slide from the talk by Prof. Sven Dickinson.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_QIDXhVDteuXgvAi-TrNxw.png" /><figcaption>Strengthening Our Community by <a href="https://www.cs.toronto.edu/~sven/">Sven Dickinson</a>. Source: <a href="https://www.cc.gatech.edu/~parikh/citizenofcvpr/static/slides/dickinson_strengthening_our_community.pdf">Slides</a></figcaption></figure><p>I can relate to all the points mentioned here. Research, especially in the field of Artificial Intelligence and Machine Learning is overwhelming. There is so much of new ideas and papers coming out that you feel you are sinking in it. I feel both excited and intimidated at the same time.</p><p>The speaker mentions how he coped with it with help from his mentors and role models. The main take away from this slide is that everyone experiences this at one point or other and with help, all these can be overcome.</p><p>Another resource that I like for beginner researchers is <a href="http://joschu.net/blog/opinionated-guide-ml-research.html">this</a> essay from John Schulman of OpenAI. Do check it out. I like the insights about progress via continual review and aiming towards a larger goal.</p><blockquote>Incremental steps should be moving you towards a larger goal that represents a non-incremental advance.</blockquote><p>It is difficult to keep updated with the field especially when a plethora of papers are getting published. Twitter is a good resource to keep updated with new and upcoming research. You can also find hidden gems if following the right people. I also listen to <a href="http://www.buzzsprout.com/632479">Daily Arxiv Radiostation</a> to keep updated with papers. It’s an automated podcast which reads the abstracts of top 10% papers in each field.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m1lcSQdSZDgulTznqpAF7g.png" /><figcaption>Strengthening Our Community by <a href="https://www.cs.toronto.edu/~sven/">Sven Dickinson</a>. Source: <a href="https://www.cc.gatech.edu/~parikh/citizenofcvpr/static/slides/dickinson_strengthening_our_community.pdf">Slides</a></figcaption></figure><h4>How to give a clear talk?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XDjflM-9CPSEziAXCl96vw.png" /><figcaption>Tips for preparing a clear talk by <a href="https://www.cs.utexas.edu/users/grauman/">Kristen Grauman</a>. Source: <a href="https://www.cc.gatech.edu/~parikh/citizenofcvpr/static/slides/grauman_preparing_clear_talks.pdf">Slides</a></figcaption></figure><p>The speaker starts with the point of understanding your audience. The audience plays an important role in determining the direction of the talk. Are they a broad set of people or narrow or specific. The level of detail and use of the language/grammar/jargon depends on this. Rephrasing and reiterating the main take-home message is necessary to hold the attention. A one-liner or punchline for the result is a good way to make sure it is etched in everyone’s mind.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8KIdCRM6meO2IO4USDDOmQ.png" /><figcaption>Tips for preparing a clear talk by <a href="https://www.cs.utexas.edu/users/grauman/">Kristen Grauman</a>. Source: <a href="https://www.cc.gatech.edu/~parikh/citizenofcvpr/static/slides/grauman_preparing_clear_talks.pdf">Slides</a></figcaption></figure><p>Be concise. Though this point depends largely on the time allocated for the presentation, we need not put out all the details. Be it while talking or on the slides. Deciding what to omit forces us to deliver an important message to the audience. It is important to remember that the main aim of the talk must be to pike the interest of the audience on the field or topic or paper. If done successfully, they will go back and find out more about it. Shorter text phrase in the talk and on the slides is the way to go. Technical slides can especially become too wordy. Try to reduce the cognitive load.</p><p>These points perfectly suit for the prep of an elevator speech as well. An elevator speech is a quick synopsis very essential for networking in conferences, meet-ups, events etc., Short. Quick. Concise. It is always good to have one ready at the back of the mind. Never know when it might come in handy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rDjk8hsEYHR-G7vBemPLAg.png" /><figcaption>Tips for preparing a clear talk by <a href="https://www.cs.utexas.edu/users/grauman/">Kristen Grauman</a>. Source: <a href="https://www.cc.gatech.edu/~parikh/citizenofcvpr/static/slides/grauman_preparing_clear_talks.pdf">Slides</a></figcaption></figure><p>My main takeaway here is the <em>flow</em> of slides. I always think of what to say for that particular slide in the talk and don’t explicitly think about leading the current one to the next. The speaker here makes an interesting point of thinking about where did we come into the slide from and where do we lead to. This helps in taking the audience through your chain of thought. Stage fear and being nervous is common.</p><p>I have tried my best to summarize the main takeaways from the tutorial. I have missed many other great talks like <a href="https://youtu.be/imEtTnQKt4M?t=2168">Principles to Thrive in the Research Community</a>, <a href="https://youtu.be/imEtTnQKt4M?t=3176">Calendar. Not to-do lists.</a> as I don’t want to bore you with a very lengthy post. But I recommend listening to all talks in leisure. The last talk also gives a brief insight into the world of reviewers and might be a good starting point to understand how research papers get reviewed, accepted etc.</p><p>There was another workshop at CVPR 2020 on <a href="https://sites.google.com/view/making-reviews-great-again/home"><em>How to write a good review?</em></a> It also includes Bill Freeman’s talk on how to write a good paper. It is always interesting to see such workshops at premier conferences. Looking forward to watching it!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=14bb0da0de04" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Overview of Transfer Learning in NLP — Contextual Word Vectors]]></title>
            <link>https://medium.com/data-science/overview-of-transfer-learning-in-nlp-part-ii-fd2e8c72eb98?source=rss-46ba9746950e------2</link>
            <guid isPermaLink="false">https://medium.com/p/fd2e8c72eb98</guid>
            <category><![CDATA[transfer-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[100daysofmlcode]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Pooja Rao]]></dc:creator>
            <pubDate>Tue, 14 Aug 2018 12:19:54 GMT</pubDate>
            <atom:updated>2018-08-16T15:10:39.983Z</atom:updated>
            <content:encoded><![CDATA[<h4>A review of recent papers on transfer learning for a supervised target task.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YWDKBbmHMMO0LFX16m2Bww.jpeg" /></figure><p>The <a href="https://medium.com/@poojarao126/overview-of-transfer-learning-in-nlp-d74749d00d77">first post</a> of the series discussed transfer learning in NLP and the publication — <a href="https://papers.nips.cc/paper/5949-semi-supervised-sequence-learning.pdf"><em>Semi-supervised Sequence Learning</em></a><em>. </em>We saw how a simple pre-training step using a sequence autoencoder improved the results on all four classification tasks.</p><p>In this post, I would like to give a brief synopsis of the next two publications in the <a href="https://medium.com/@poojarao126/overview-of-transfer-learning-in-nlp-d74749d00d77/#cd96">list</a>. Contextualized word representation is the focus of both the publications. The researchers try to produce word representations commonly known as embeddings, which are aware of the contextual meaning in the sentence. The process of generating the word embeddings is where the difference lies between the two. CoVe uses a supervised task like <strong><em>machine translation</em></strong> while ELMo uses an unsupervised task like <strong><em>language modelling.</em></strong></p><ul><li>Learned in Translation: Contextualized Word Vectors — <a href="https://arxiv.org/abs/1708.00107">CoVe</a></li><li>Deep contextualized word representations — <a href="https://arxiv.org/abs/1802.05365">ELMo</a></li></ul><p>Let’s dive deep into each of the research publication and chalk out the similarities and the differences.</p><h4>Learned in Translation: Contextualized Word Vectors</h4><p><em>Machine translation data can be the potential equivalent to ImageNet in NLP. </em>(owing to the immense success of transfer learning in computer vision using ImageNet). The authors try to prove this hypothesis by adopting the attentional sequence-to-sequence model trained for machine translation. Machine translation sequence-to-sequence models encode words in context and decode them into another language. These models usually contain an LSTM-based encoder. They make use of this encoder to obtain, what they call, the contextual vectors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FuTTnwwUR4liiERRMkZceQ.png" /><figcaption>a) Training an encoder b) Reusing it for a downstream task. Source: “Learned in Translation: Contextualized Word Vectors” by McCann et al. 2017</figcaption></figure><p>This process involves two steps. a) Train the attentional sequence-to-sequence model for machine translation. b) Transfer the learnt information to downstream tasks.</p><p>English-to-German translation data is used to train the machine translation model. They use a standard, two-layer, bidirectional LSTM network as the encoder (MT-LSTM), initialized with GloVe vectors and a two-layer, unidirectional LSTM with attention as the decoder. The outputs of the pre-trained encoder, MT-LSTM, are treated as the context vectors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/408/1*yCu94sPT83f-fBhz_RGcOQ.png" /></figure><p><em>w </em>is the sequence of input words and <em>GloVe(w)</em> is the sequence of word vectors produced by the GloVe model. <em>CoVe(w)</em> is the sequence of context vectors produced by MT-LSTM.</p><p>The authors check the benefits of the CoVe vectors on a range of datasets for classification tasks namely — sentiment analysis, question classification and entailment; and question answering. The classification models use a biattentive classification network [1] and the question answering model uses a dynamic coattention network [2]. The architectures of these networks are beyond the scope of this article.</p><p>Unlike the <a href="https://medium.com/@poojarao126/overview-of-transfer-learning-in-nlp-d74749d00d77">first review </a>where the target task was predefined, there is no targeted downstream task here. The aim is to adapt the vectors to any downstream task.</p><p>Each vector in GloVe(w) is concatenated with its corresponding vector in CoVe(w)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/312/1*Xy5ZJjaZWVu2vLgqfQFT8A.png" /></figure><p>The authors let these concatenated vectors be the inputs to the classification and the question answering networks. These vectors remain fixed throughout the experiments (unlike the <a href="https://medium.com/@poojarao126/overview-of-transfer-learning-in-nlp-d74749d00d77">first review</a>, where the pre-trained weights are fine-tuned on the target task).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o_A_mRka6h38QjqXo513Dg.png" /><figcaption>Validation results: Accuracy for classification tasks and F1 for SQuAD. Source: “Learned in Translation: Contextualized Word Vectors” by McCann et al. 2017</figcaption></figure><p>The models using CoVe along with GloVe consistently perform better than the models using only GloVe. Appending character n-gram embeddings boost performance even further for some tasks. They achieve new state-of-the-art performance (at the time) on the SST-5 and SNLI datasets. One important insight from the experiments is as follows. Larger the machine translation dataset, better the improvement in the downstream target task using CoVe. This is one of the important insights from the experiments. Thus, this correlation can be an evidence of their hypothesis of using machine translation data as an equivalent to ImageNet in NLP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YcdgzpBmyAWrCloiSauKLw.png" /><figcaption>Test performance results. Source: “Learned in Translation: Contextualized Word Vectors” by McCann et al. 2017</figcaption></figure><h4>Deep Contextualized Word Representations</h4><p>This research from the Allen Institute for AI introduces another type of deep contextual word representations. The aim is to learn representations that model the syntax, semantics and polysemy (existence of many possible meanings for a word). The researchers train a bidirectional LSTM network on the language modelling task to derive vectors. These are called ELMo (Embeddings from Language Model). ELMo vectors are a function of all the internal layers of the language model and hence deeper than the CoVe.</p><p>A bidirectional language model (biLM) is a combination of forward language model and backward language model. Forward LM computes the probability of the sequence by modelling the probability of token <em>k</em> provided the history of <em>k-1</em> tokens. Backward LM is similar to forward LM, running the sequence in reverse order predicting the previous token provided the future context. The architecture of biLM is comparable to [3]. It is trained on the 1B Word Benchmark corpus [4].</p><p>The authors fix the pre-trained biLM and use it to compute representations for any downstream task. In some cases, fine-tuning the biLM on the downstream task data leads to performance improvement. Hence, they fine-tune the biLM for one epoch on the task-specific data removing the supervised labels and fix the weights throughout the task experiments thereafter (unlike the <a href="https://medium.com/@poojarao126/overview-of-transfer-learning-in-nlp-d74749d00d77">first review</a> where they fine-tune the weights during the task experiments). They use the fine-tuned biLM in most of the cases.</p><p><em>ELMo is a task-specific combination of the intermediate layer representations in the biLM. </em>A L-layer biLM<em> </em>computes a set of 2L+1 representations as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*hFijURTQco_oAno6zd52EA.png" /></figure><p>ELMo computes a task-specific weighted combination of all the layers in biLM as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/637/1*tI-JbjkAvnJu1LH0Ry8pvQ.png" /></figure><p>s^task are the softmax-normalized weights and the scalar parameter <strong><em>γ</em></strong> is decided experimentally. In the simplest case, it can choose only the last layer like CoVe.</p><p>The pre-trained word embeddings are concatenated with ELMo vectors from the biLM to be passed into the downstream tasks as inputs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/218/1*4Ecv8EZF_kfHSazB71-VUg.png" /></figure><p>They validate the performance of ELMo vectors across a variety of NLP tasks and achieve a new state-of-the-art (at the time) on each. They use different baseline architectures for each of the tasks and add ELMo vectors to the input. They achieve relative error reductions ranging from 6–20% over strong base models. The results are as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YoCli5AdvjTradWKycrwEw.png" /><figcaption>Test set results on 6 benchmark NLP tasks. Source: “Deep contextualized word representations” by ME Peters et al. 2018</figcaption></figure><p>They show that a weighted combination of representation from all layers improves overall performance than just using the last layer. ELMo beats the performance of CoVe in tasks where comparison is possible. Sometimes, including ELMo at the output layer of the downstream task model improves the performance. It also successfully disambiguates a word in context achieving polysemy. See the example below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lCsnSWNg_2VZcjrbB2gJVA.png" /><figcaption>Nearest neighbours to ‘play’ using GloVe and ELMo. Source: “Deep contextualized word representations” by ME Peters et al. 2018</figcaption></figure><p>The experiments reveal that different layers of the biLM depict a different sort of information of a word. The second layer performs better at word sense disambiguation while the first layer achieves higher accuracy at POS tagging. This way, inclusion of all biLM layers is crucial.</p><p>Adding a large unsupervised dataset via ELMo representations is advantageous. The model converges to a higher accuracy faster and requires a smaller training set. It enhances the usage of the training data by the model. It significantly reduces the amount of training data needed to reach a level of performance.</p><p>Thus, the ELMo captures the syntax and semantics of a word in context using the deep embeddings from the language model.</p><p>CoVe and ELMo are similar in the way that they compute contextual word representations. Some of the similarities —</p><ul><li>Try to compute contextual word vectors.</li><li>Use a large corpus for pre-training.</li><li>Use the pre-trained model to transfer the learnt information in the form of word representations.</li><li>Word vectors computed are used as inputs to the downstream task-specific networks.</li><li>Pre-trained weights are fixed during the task training.</li></ul><p>Some of the major differences —</p><ul><li>CoVe uses a machine translation encoder for the pre-training step to generate word representations while ELMo uses a language model.</li><li>CoVe trains on the supervised machine translation dataset while ELMo trains on the unsupervised language model data.</li><li>CoVe treats the output of the last layer machine translation LSTM as the context vectors while ELMo is a weighted combination of all layers in the language model LSTM.</li><li>CoVe vectors are fixed during task training while the weights from the biLM model are fine-tuned with the task-specific data before being fixed for the task training.</li><li>Researchers with CoVe use a common architecture for a variety of downstream classification tasks while ELMo is used with strong baseline architectures specific to each task.</li></ul><p>One significant difference between the first review and this review of the contextual word vectors is that the researchers choose a generic dataset for pre-training independent of the task. This may be regarded as the next step in moving towards an ImageNet like dataset.</p><p>References:</p><p>[1] <a href="https://arxiv.org/abs/1611.01603">Bidirectional Attention Flow for Machine Comprehension</a></p><p>[2] <a href="https://arxiv.org/abs/1603.01417">Dynamic Memory Networks for Visual and Textual Question Answering</a></p><p>[3] <a href="https://arxiv.org/abs/1602.02410">Exploring the limits of language modeling</a></p><p>[4] <a href="https://arxiv.org/abs/1312.3005">One billion word benchmark for measuring progress in statistical language modeling</a></p><p>Further Reading:</p><ol><li>CoVe — <a href="https://arxiv.org/abs/1708.00107">research paper</a>, <a href="https://einstein.ai/research/learned-in-translation-contextualized-word-vectors">official blog post</a>, <a href="https://github.com/salesforce/cove">code</a></li><li>ELMo — <a href="https://arxiv.org/abs/1802.05365">research paper</a>, <a href="https://allennlp.org/elmo">official blog post</a>, <a href="https://github.com/allenai/allennlp/blob/master/tutorials/how_to/elmo.md">code</a></li></ol><p>This is the second post in the series in where I give a brief about recent publications about transfer learning in NLP. You can find the first post <a href="https://medium.com/@poojarao126/overview-of-transfer-learning-in-nlp-d74749d00d77">here</a>. Any feedback/suggestions are highly appreciated. Feel free to comment below or reach out to me on <a href="https://twitter.com/poojaraosb">Twitter</a> or <a href="https://www.linkedin.com/in/pooja-rao/">LinkedIn</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fd2e8c72eb98" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/overview-of-transfer-learning-in-nlp-part-ii-fd2e8c72eb98">Overview of Transfer Learning in NLP — Contextual Word Vectors</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Overview of Transfer Learning in NLP]]></title>
            <link>https://medium.com/@poojarao126/overview-of-transfer-learning-in-nlp-d74749d00d77?source=rss-46ba9746950e------2</link>
            <guid isPermaLink="false">https://medium.com/p/d74749d00d77</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[transfer-learning]]></category>
            <category><![CDATA[100daysofmlcode]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Pooja Rao]]></dc:creator>
            <pubDate>Sat, 04 Aug 2018 12:19:46 GMT</pubDate>
            <atom:updated>2018-08-07T10:11:11.476Z</atom:updated>
            <content:encoded><![CDATA[<h4>A review of recent papers on transfer learning for a supervised target task.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*ypQ_fekpL7qBOkQOFtgCBQ.jpeg" /></figure><p>Knowledge transfer is a mighty aspect for both humans and machines. Human beings can gain more by sharing and teaching each other. I would like to think of machines in the same way.</p><blockquote>Power is gained by sharing knowledge not hoarding it.</blockquote><p>Transfer Learning is the technique of using the knowledge gained by a model trained for a source task to solve a target task. In most of the cases, the target task will be in a related or a similar domain of data. And the knowledge gained refers to the <em>weights</em> learned.</p><p>In layman terms, T<em>ransfer Learning = Knowledge Sharing.</em></p><p>It is particularly useful when labelled data available for the target task is scarce and/or when the computational power is less. This is because the model need not be trained from scratch.</p><p>In this series, I would like to give a brief read about the recent advances in transfer learning related to Natural Language Processing. Especially, an overview of the following five publications.</p><ul><li><a href="https://papers.nips.cc/paper/5949-semi-supervised-sequence-learning.pdf">Semi-supervised sequence learning</a></li><li>Learned in Translation: Contextualized Word Vectors — <a href="https://arxiv.org/abs/1708.00107">CoVe</a></li><li>Deep contextualized word representations — <a href="https://arxiv.org/abs/1802.05365">ELMo</a></li><li>Universal Language Model Fine-tuning for Text Classification — <a href="https://arxiv.org/abs/1801.06146">ULMFiT</a></li><li><a href="https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf">Improving Language Understanding by Generative Pre-Training</a></li></ul><p>It has been a widely accepted norm to use word embeddings like word2vec, GloVe as the equivalent of transfer learning in NLP. These distributed word representations are used as the common weight initialization strategy. But more recently, word embeddings are paving way for more sophisticated transfer learning options, like Language Models. This is the first post in the series briefing about the first paper. Read on to know more.</p><h4>Semi-supervised sequence learning</h4><p>Semi-supervised learning is a technique of solving a supervised task with a small amount of labelled data and a large amount of unlabelled data. It has the same flavour as the transfer learning process and hence included in this list. This is first of the papers where the authors have leveraged language modelling for a supervised target task. In this 2015 paper, the network is pre-trained using sequence autoencoders and language model.</p><p>The authors experiment with text understanding tasks namely the sentiment analysis (IMDB and Rotten Tomatoes) and text classification (20 Newsgroups and DBpedia). They use standard implementation of LSTMs with input, forget and output gates. The output layer of the LSTM network predicts the document label. But what makes the difference in the results is a simple pre-training step — The recurrent language model or the sequence autoencoder is trained on the same dataset (unlabelled) as the target supervised task. The word embedding parameters and the LSTM weights obtained from this pre-training step is used in the initialization of the supervised target task. The network is then trained on the target task, fine-tuning for the weights from the pre-trained model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/623/1*J4nPzx-ZWHxRoC-NRgTtRA.png" /><figcaption>Results of the pre-trained model with autoencoders from Semi-supervised sequence learning paper.</figcaption></figure><p>They obtain better error rates than the previous best results at the time on all the four experiments. The models pre-trained with sequence autoencoders perform better than the models pre-trained with language models. This may be due to the fact that the autoencoders capture the long-term dependencies in the documents. “Language modelling is a short-term objective so that the hidden state only captures the ability to predict the next few words.”</p><p>They have also briefly explored the usage of other datasets for pre-training. (A method used extensively by the other papers). The experiments on Rotten Tomatoes dataset yield the best results when pre-trained with Amazon movie reviews data.</p><p>The important contribution from this paper is the feasibility of usage of LSTM recurrent networks for document classification tasks with a substantial number of words. The pre-trained sequence autoencoder or the language model stabilize the LSTM networks. Further, the authors demonstrated that the unlabelled data can compensate for the lack of the labelled data. This is very significant in terms that the unlabelled data is cheaper than the labelled data.</p><p>Some references and further reading:</p><ol><li>The paper, <a href="https://papers.nips.cc/paper/5949-semi-supervised-sequence-learning">Semi-supervised Sequence Learning</a></li><li><a href="https://thegradient.pub/nlp-imagenet/">NLP&#39;s ImageNet moment has arrived</a> on The Gradient by <a href="https://medium.com/u/e3999e445181">Sebastian Ruder</a></li><li><a href="http://ruder.io/transfer-learning/index.html">Transfer Learning - Machine Learning&#39;s Next Frontier</a> by <a href="https://medium.com/u/e3999e445181">Sebastian Ruder</a></li></ol><p>This is the first post in the series. I will be briefly exploring the other papers in the upcoming posts. Any feedback/suggestions are highly appreciated. Feel free to comment below or reach out to me on <a href="https://twitter.com/poojaraosb">Twitter</a> or <a href="https://www.linkedin.com/in/pooja-rao/">LinkedIn</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d74749d00d77" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Alexa, What is Machine Learning?]]></title>
            <link>https://medium.com/@poojarao126/alexa-what-is-machine-learning-35d2351ae559?source=rss-46ba9746950e------2</link>
            <guid isPermaLink="false">https://medium.com/p/35d2351ae559</guid>
            <category><![CDATA[alexa-skills]]></category>
            <category><![CDATA[canfulfillintentrequest]]></category>
            <category><![CDATA[100daysofmlcode]]></category>
            <category><![CDATA[alexa]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Pooja Rao]]></dc:creator>
            <pubDate>Sun, 08 Jul 2018 13:10:11 GMT</pubDate>
            <atom:updated>2018-08-04T12:17:06.710Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/489/1*Y636O-xmba2cc4Pf90RoKA.png" /></figure><p>Machine learning is the most transformative and evolutionary area in the field of Computer Science today. And like most of the other areas, it has many terms, concepts and jargons which can be hard to remember. This prompted me to build a glossary of terms. I also wanted to have some fun building it by learning something new. And thus born the Alexa skill — <strong>Machine Learning Glossary.</strong></p><p>Creating an Alexa skill is easy. Especially when a skill does not have a multi-turn dialog or a conversation and has none or a single slot to be filled. (Building a skill with multi-turn dialog is also relatively easy, but that’s for another post). Currently, Alexa gives a very crisp and precise answer to the question <strong>‘Alexa, what is Machine Learning?’</strong></p><blockquote>Machine learning is a field of computer science that gives computers the ability to learn without being explicitly programmed.</blockquote><p>It handles the general definition of words exceptionally. But fails to give a definition in the context of Machine Learning and Data Science. Machine Learning Glossary aims to bridge this gap.</p><p>Machine Learning Glossary can help you get acquainted with general Machine Learning terms, defining them with appropriate examples. It consists of over 200 definitions.</p><p>Few of the handy techniques useful while creating a skill:</p><ul><li>Slot type — Amazon.SearchQuery</li><li>S3 buckets</li><li>CanFulfillIntentRequest</li></ul><p>Amazon.SearchQuery is a category of phrase slot type which allows for a less constrained input from the user. This is designed for a search query like input which is perfect to search for machine learning terms in our case.</p><p>Amazon S3 (Simple Storage Service) can be very convenient to retrieve the data for your skill. It can be thought of as an online filesystem. Refer to <a href="https://github.com/alexa/alexa-cookbook/tree/master/aws/Amazon-S3">Alexa cookbook</a> for instructions on how to use it.</p><p>CanFulfillIntentRequest is like a magic tool to be included in your skill to help users discover your skill without consciously calling it. This will be useful to reach out to a wide range of users. CanFulfillIntentRequest is still in its BETA version and available only to customers in US. Expect some surprises there. Please refer to the <a href="https://developer.amazon.com/docs/custom-skills/understand-name-free-interaction-for-custom-skills.html">official documentation</a> for more information.</p><p>You can find the codebase used in the implementation of this skill <a href="https://github.com/poorao/Machine-Learning-Glossary">here</a>. The code is in Python.</p><p>Please do try out the skill available on the Amazon Alexa Skills store in IN, US, UK and AUS, and let me know if it was helpful. For IN store, click <a href="https://alexa.amazon.in/spa/index.html#skills/dp/B07F7377V8/?ref=skill_dsk_skb_sr_1&amp;qid=1531197498">here</a>. Any feedback/suggestions? Feel free to comment below or reach out to me on <a href="https://twitter.com/poojaraosb">Twitter</a> or <a href="https://www.linkedin.com/in/pooja-rao/">LinkedIn</a>.</p><p>This is part of the #100DaysOfMLCode challenge. This is the progress I made on Day 1 and Day 2 towards my commitment of extra ML study/code outside my research curriculum. Though I had built the skill last week, gathering the data for the glossary including their definitions improved my formal understanding of some terms I was already aware of and also many new terms. Day 0 of this challenge can be found <a href="https://medium.com/@poojarao126/100daysofmlcode-8750581e6c9d">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=35d2351ae559" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[100DaysOfMLCode]]></title>
            <link>https://medium.com/@poojarao126/100daysofmlcode-8750581e6c9d?source=rss-46ba9746950e------2</link>
            <guid isPermaLink="false">https://medium.com/p/8750581e6c9d</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[100daysofmlcode]]></category>
            <category><![CDATA[challenge]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Pooja Rao]]></dc:creator>
            <pubDate>Fri, 06 Jul 2018 15:45:48 GMT</pubDate>
            <atom:updated>2018-07-07T07:44:02.155Z</atom:updated>
            <content:encoded><![CDATA[<iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FcuQMBj1cWPo%3Ffeature%3Doembed&amp;url=http%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DcuQMBj1cWPo&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FcuQMBj1cWPo%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/40793f739ff8c975ccc358ad922e3bef/href">https://medium.com/media/40793f739ff8c975ccc358ad922e3bef/href</a></iframe><p><a href="https://medium.com/u/54526471f9bf">Siraj Raval</a>, the popular Youtube star and AI educator has put up an interesting challenge to persuade all of us to code or study Machine Learning everyday for atleast an hour for 100 days. #100DaysOfMLCode</p><p>I solemnly pledge to learn, study and code Machine Learning for an hour everyday for 100 days starting today 😊 . I intend to update my progress by blogging on <a href="https://medium.com/u/504c7870fdb6">Medium</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/396/0*cTVL2VIWFOxyUP_O.jpg" /></figure><p>I am a research student at Multimodal Perception Lab and do ML for a living 😋. So, I plan to work for an extra hour on projects outside of work, improve my fundamentals, write technical blog posts and advance my skills.</p><h3>Day 0:</h3><p>I am announcing my goals publicly so I have a sense of accountability. I intend to code and blog about my progress everyday. Also, plan to learn and understand the area of Deep Reinforcement Learning. I have a few plans in mind. Let’s see how many of them materialise.</p><p>Hoping for a better me in 100 days!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8750581e6c9d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ICMI 2017 — My First Academic Conference]]></title>
            <link>https://medium.com/@poojarao126/icmi-2017-my-first-academic-conference-f4bd031b3ee9?source=rss-46ba9746950e------2</link>
            <guid isPermaLink="false">https://medium.com/p/f4bd031b3ee9</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[conference]]></category>
            <category><![CDATA[research]]></category>
            <category><![CDATA[academia]]></category>
            <dc:creator><![CDATA[Pooja Rao]]></dc:creator>
            <pubDate>Sat, 07 Apr 2018 11:28:15 GMT</pubDate>
            <atom:updated>2018-04-11T16:04:15.985Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nrwkK8bFYRb77RsH9L0fcQ.jpeg" /><figcaption>Glasgow City Chambers. Courtesy: Colin / Wikimedia Commons</figcaption></figure><p>Let me first introduce myself. I’m a research student at the Multimodal Perception Lab pursuing Master’s by Research in Data Science at <a href="https://medium.com/u/5b3fccc75bb0">IIIT Bangalore</a>. This program is pure research-based, more like a bridge to a doctoral program. So publishing papers or journals in an academic conference is a requirement. You have to work on a problem statement towards submitting a substantial and strong thesis. I had joined the program in August 2015 with no idea on how to or even what to work on or publish.</p><p>Fast-forward to August 2017, my work was accepted to be published in ICMI 2017, International Conference on Multimodal Interactions. I was ecstatic. My hard work had paid off. It was a new experience. I was going to meet like-minded people from around the world. I couldn’t wait for 3 months to attend and present my work.</p><p>Even to see our paper in the conference programme list was a joy. And the icing on the cake was it was the first name in the list for that session (though it was only because of alphabetical ordering, but those little blissful things in life). We created the poster with excitement and prepared for the presentation.</p><p>The conference with the all the workshops was scheduled for the third week of Nov 2017. Our presentation slot was in the Tuesday afternoon session. The workshops and the talks were engaging. I was surprisingly calm on the first day of the conference.</p><h3>The day</h3><p>The morning session went well. The keynote was unique and interesting. The papers and the talks were engaging. I revisited our paper and prepped for the poster session. As the time elapsed, it dawned on me that this is it! I was about to describe a 12-page paper in less than 2 mins. It was like the panic button became active all of a sudden. I had prepared two versions for presentation — a shorter version summarizing the work in a minute or so and a little longer detailed version for the interested folks.</p><p>I was at my booth awaiting the audience. My first listener was a well-known person from one of the sponsors of the conference. With the panic monster looming large on me, I presented a mix of both versions I had prepared leaving out the important details. He did not feel convinced about the work and remarked the same. I felt uncomfortable and dejected. I had to pick myself up and explain better as more people were expected to come. I had to let my apprehensions go.</p><p>The next listener was an attendee of the conference whose presentation I had heard in the workshop. That made me feel like I knew him. The panic faded away. My explanation to him was better and we further interacted with questions. As more people started visiting the booth, my presentation got better and very interactive. I got to meet tremendous people, academicians, research students. I even got the privilege to meet my Professor’s supervisor. He was very humble and listened to my explanation with great curiosity. I had a great time and I particularly liked the poster sessions more than the oral sessions because of its interactive nature.</p><h3>Takeaways</h3><ul><li>Don’t be intimidated. — They are all just people wanting to know more.</li><li>There is no room for doubt. — When you are on the stage or in the limelight, take a stand and be confident.</li><li>Let go of the apprehensions. — You know more about your work than anyone else in the room.</li><li>Talk, talk and talk. — This applies to people more like me who consider silence as comfort. People will not know about you or your work unless you tell them.</li><li>Network. — It is imperative to build a strong network with distinguished folks in your area of work while you are at a conference.</li></ul><p>These learnings are far from achieved. My biggest take away was that my networking and communicative skills need improvements. An irony considering that my work is on the same! I have to learn to tame the panic monster.</p><p>The rest of the days of the conference went on smoothly. I had a great time attending the reception and banquet. I got to stroll through the calm and beautiful city of Glasgow. The people are so welcoming and heartwarming. The Glasgow City Chambers is a masterpiece in itself. I even got to travel the beautiful scenic places of Scotland. Perks of attending an international conference! Overall it was indeed an amazing experience with a lot of things to learn back home.</p><p>Our Publication:</p><p><a href="https://dl.acm.org/citation.cfm?id=3136756">Automatic Assessment of Communication Skill in Non-conventional Interview Settings: A Comparative Study</a></p><p>You can also access it <a href="https://www.researchgate.net/publication/320884467_Automatic_assessment_of_communication_skill_in_non-conventional_interview_settings_a_comparative_study">here</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f4bd031b3ee9" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>