<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Prashanth Rao on Medium]]></title>
        <description><![CDATA[Stories by Prashanth Rao on Medium]]></description>
        <link>https://medium.com/@tech_optimist?source=rss-efdb4f4a0d1b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*_V1mmd775bZSipk0BfxQsA.jpeg</url>
            <title>Stories by Prashanth Rao on Medium</title>
            <link>https://medium.com/@tech_optimist?source=rss-efdb4f4a0d1b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 05:48:54 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tech_optimist/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[PaLM on my forehead: Not another large language model??]]></title>
            <link>https://medium.com/@tech_optimist/palm-on-my-forehead-not-another-large-language-model-6dddd641211b?source=rss-efdb4f4a0d1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/6dddd641211b</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Prashanth Rao]]></dc:creator>
            <pubDate>Wed, 06 Apr 2022 19:32:10 GMT</pubDate>
            <atom:updated>2022-04-06T19:32:10.685Z</atom:updated>
            <content:encoded><![CDATA[<h4>A summary on the Pathways Language Model by Google AI, with the good and not-so-good implications</h4><p>The release of the <a href="https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html">Pathways Language Model (PaLM)</a> by Google AI has once again led to that familiar, vapid feeling among the AI community— “Oh no, not another large-scale language model that thinks it can learn grammar and human intent”. Just another scaling-up exercise where we throw a bunch of data at a newish architecture, and report a ton more SOTA results on benchmark datasets. And yet again, the model seems to blow away past models in its performance on a range of tasks, all the way from reasoning to code translation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*H7Kveb3qfgvntF9c.png" /><figcaption>Image source: <a href="https://www.gwern.net/Scaling-hypothesis#">gwern.net</a></figcaption></figure><p>In this post I want to record some of my thoughts on why I think the <a href="https://storage.googleapis.com/pathways-language-model/PaLM-paper.pdf">PaLM paper</a> is a far more interesting one than the others of late (and not just because it’s <strong>83 pages long!!</strong>).</p><h3>Part 1: The good</h3><h4>Builds on past research from explainability, bias, safety and all in one</h4><p>Unlike the pomp and ceremony that came with The GPT-3 paper in 2021, the PaLM model, from the ground up, is designed with explainability in mind. It builds on an excellent paper from Google Brain: “<a href="https://arxiv.org/abs/2201.11903"><em>Chain of Thought Prompting Elicits Reasoning in Large Language Models</em></a>”, in which large language models are given examples with few-shot prompting, allowing them to learn to generate chains of text that explain how the model arrived at a particular answer. This ability — that is, to sequentially arrive at a particular answer via intermediate stems, as opposed to standard prompting where the model directly provides answer (as a black box), is hugely interesting. We can now see a more human-like train of thought emerging from the generative component of a sufficiently large model!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1005/1*v5_bMDD13QP0Dj_yZPiAmQ.png" /><figcaption>Standard vs. chain of thought prompting. Source: <a href="https://storage.googleapis.com/pathways-language-model/PaLM-paper.pdf">PaLM paper</a></figcaption></figure><p>From a bias standpoint, the paper does a fairly good job of describing the different kinds of bias typically seen in language models, from representational bias (gender and occupation) to toxicity bias(race, religion).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/888/1*fa4aZfH2ueYH6MlOqRv6qQ.png" /><figcaption>Analysis of model toxicity toward religions based on prompt language. Source: <a href="https://storage.googleapis.com/pathways-language-model/PaLM-paper.pdf">PaLM paper</a></figcaption></figure><p>From a safety and ethical standpoint, the authors do well to highlight valid issues and concerns regarding using these powerful models in the real world. Important limitations in the model, as well as the datasets used to train and evaluate it, are discussed and acknowledged, with the relevant literature that highlights the impact of training and data bias. However, it can be argued that simply addressing these as concerns without actually opening the model’s internals to public scrutiny, does little to alleviate the concerns of AI safety researchers on the real-world impact of these models.</p><h4>Built on the shoulders of giants</h4><p>Although this statement is true for most good research, the sheer magnitude of the underlying effort that paved the way for the PaLM model cannot be overstated. The original blog post from <a href="https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html">Google AI</a> recognizes the contributions of not only the large number of researchers and scientists who contributed prior work, but also the open-source contributors that have spent eons of time building the large-scale frameworks and infrastructure that allowed such a model to be trained.</p><blockquote><em>PaLM builds on top of work by many, many teams at Google and we would especially like to recognize the T5X team, the Pathways infrastructure team, the JAX team, the Flaxformer team, the XLA team, the Plaque team, the Borg team, and the Datacenter networking infrastructure team.</em></blockquote><p>Also, the fact that the paper has 69 authors points to the fact that most AI research is, in reality, a massive collaboration between huge teams of researchers and industry professionals, and this can only be expected to continue as the field powers forward.</p><h4>Multi-modal and multi-lingual training</h4><p>A common criticism in the field of NLP is that most large works of research (such as GPT-3) focus too heavily on English. In this case, the PaLM researchers conceptualized a framework that was designed from the ground up to be multilingual, both from a data and language processing task standpoint. This draws from the earlier <a href="https://blog.google/technology/ai/introducing-pathways-next-generation-ai-architecture/">blog post by Google AI on the Pathways architecture</a>, designed to allow one initial model to be fine-tuned to perform hundreds (or thousands of downstream tasks).</p><h4>The end of improvement is nowhere in sight</h4><p>Just like prior works have observed, the 540 billion parameter PaLM model shows that we are nowhere near the end in terms of performance improvements for large language language models.</p><p>As mentioned in the blog post, PaLM’s performance on 5-shot evaluation is comparable to that of humans, with potentially larger versions of the model able to generalize better than ever before. With regard to training, larger models, though prohibitively expensive to train as of today, are nowhere close to overfitting, explaining why so much high-profile industry research is still focusing on scaling up large language models.</p><blockquote>Interestingly, we note that PaLM’s performance as a function of scale follows a log-linear behavior similar to prior models, suggesting that <em>performance improvements from scale have not yet plateaued</em>. PaLM 540B 5-shot also does better than the average performance of people asked to solve the same tasks.</blockquote><h3>Part 2: The not-so-good</h3><h4>Centralization of power and resources</h4><p>The flip side of all these developments is that we are more and more entering a world where the only means to build, train and evaluate large language models exist with the biggest industry players. A huge part of the processing power and research funding for large language models (and the overall field of AI in general) rests with the richest companies, Google being just one of them.</p><p>The main justification for companies that avoid releasing these models in the open domain is the potentially large scope for misuse. However, just as in the case with GPT-3, the counterargument to this is that unless such models are subject to public scrutiny (especially with the world’s leading AI safety researchers), how will we ever begin to address how to align incentives such that the large portion of society benefits from these advances? As more and more research on large language models comes out, it’s becoming clear that universities and other academic institutions have little to no means to compete with the kinds of budgets and compute that the big corporations possess. This is already having large implications on the direction of future research.</p><h4>The double-edged sword of industry research</h4><p>While on most counts, having large research groups in industry is a great thing overall (especially in the case of new open-source frameworks, such as JAX) , the general trend nowadays is to see talented graduate students flock to the top companies in industry after their PhDs. This leads to the natural question — will future generations of AI researchers and mentors be incentivized to consider important areas such as causality, safety and ethics? As <a href="https://www.technologyreview.com/2018/11/17/66372/one-of-the-fathers-of-ai-is-worried-about-its-future/">Yoshua Bengio highlighted in an interview</a>, “<em>AI research by itself will tend to lead to concentrations of power, money, and researchers. The best students want to go to the best companies. They have much more money, they have much more data. And this is not healthy. Even in a democracy, it’s dangerous to have too much power concentrated in a few hands</em>.”</p><h3>Conclusions</h3><p>There is no denying that the road towards progress in the field, as it stands today, is open-ended. On one hand you have large companies that help build massive, dynamic open-source ecosystems that enable thousands of researchers build better, faster models, but on the other, you have closed-source models built by the world’s richest companies that might never see the light of day in terms of public scrutiny. As we see the field progress further than ever before, it helps to take a step back and see the bigger picture — that indeed, we as practitioners and researchers can do better, in terms of asking the right questions, and forcing our leaders in our organizations to address these legitimate concerns.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6dddd641211b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build a LIME explainer dashboard with the fewest lines of code]]></title>
            <link>https://medium.com/data-science/build-a-lime-explainer-dashboard-with-the-fewest-lines-of-code-bfe12e4592d4?source=rss-efdb4f4a0d1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/bfe12e4592d4</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[hands-on-tutorials]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Prashanth Rao]]></dc:creator>
            <pubDate>Sat, 24 Oct 2020 05:00:00 GMT</pubDate>
            <atom:updated>2020-11-05T20:33:34.970Z</atom:updated>
            <content:encoded><![CDATA[<h4><a href="https://towardsdatascience.com/tagged/hands-on-tutorials">Hands-on Tutorials</a></h4><h4>A comparison of Flask, Plotly Dash and Streamlit to build dashboards that provide LIME explanations for classification results</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rPfZH7ku8Yrp4yUe.jpg" /><figcaption>Image source: <a href="https://pixabay.com/photos/doors-choices-choose-open-decision-1767563/">qimono on Pixabay</a></figcaption></figure><p>In <a href="https://towardsdatascience.com/fine-grained-sentiment-analysis-in-python-part-2-2a92fdc0160d">an earlier post</a>, I described how to explain a fine-grained sentiment classifier’s results using LIME ( <strong>L</strong>ocal <strong>I</strong>nterpretable <strong>M</strong>odel-agnostic <strong>E</strong>xplanations). To recap, the following six models were used to make fine-grained sentiment class predictions on the Stanford Sentiment Treebank (SST-5) dataset.</p><ul><li>Rule-based models: <strong>TextBlob</strong> and <strong>VADER</strong></li><li>Feature-based models: <strong>Logistic regression</strong> and <strong>Support Vector Machine</strong></li><li>Embedding-based models: <strong>FastText</strong> and <strong>Flair</strong></li></ul><p>A linear workflow was used to analyze and explain the sentiment classification results using each method. Each model was trained on 5 classes of sentiment (1 through 5), with 1 being “strongly negative”, 3 being “neutral” and 5 being “strongly positive”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*A2ox9DtrIamY6DU9.png" /></figure><p>The goal of this post is to show how to build an explainer dashboard (using any one of three frameworks) that takes in a trained model, and outputs LIME explanations for the prediction made by the model.</p><h3>Example LIME explanation</h3><p>Put simply, LIME generates an explanation object containing visualizations (as embedded JavaScript) that can be output to an HTML file, which can then be opened in any browser. A typical output from LIME is shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*bZVB0yUN_d67NK_y.png" /></figure><h3>Why build an interactive dashboard app?</h3><p>To explain a classifier’s results using LIME, it can be cumbersome to have to write out individual HTML files each time an explanation needs to be made. An interactive dashboard that takes in user input is a very effective means to rapidly iterate through multiple test samples in real time, providing the user with immediate feedback. In addition, having a dashboard allows non-technical users (who may or may not know how to execute Python scripts) to be able to make their own LIME explanations on demand.</p><p>The following sections show how we can build LIME explainer dashboards using three different frameworks: Flask, Dash and Streamlit.</p><h3>Explainer class</h3><p>To easily reference each classifier’s predict methods, the below object-oriented template is applied to enable code reuse, available in <a href="https://github.com/prrao87/fine-grained-sentiment/blob/master/explainer.py">the project’s GitHub repo</a>. In a nutshell, a Python class is defined which takes in the list of variations generated by LIME (random text samples with tokens blanked out), following which we output a class probability for each sample as a Numpy array.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/04d82e8d44e5f04eb1bb98ac403f1a4f/href">https://medium.com/media/04d82e8d44e5f04eb1bb98ac403f1a4f/href</a></iframe><p>Once the class probabilities for each variation is returned, this can be fed to the LimeTextExplainer class (shown below). Enabling bag-of-words ( bow) would mean that LIME doesn&#39;t consider word order when generating variations. However, the FastText and Flair models were trained considering n-grams and contextual ordering respectively, so for a fair comparison between models, the bow flag option is disabled for all explanations on SST-5.</p><p>The exp object returned by the LIME explainer is via the explain_instance method internal to LIME, that converts the local linear model&#39;s predictions (in numerical form) to a visual, interpretable form. This can then be output as HTML.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9a966bd66406e3ce0440e9372903450e/href">https://medium.com/media/9a966bd66406e3ce0440e9372903450e/href</a></iframe><p>The below sections describe how to encapsulate all these functions into an interactive dashboard app.</p><h3>Option 1: Flask</h3><p>A Flask version of the LIME explainer dashboard is shown below. The user enters a piece of text, selects the number of random samples to generate for LIME, and then chooses from a given list of classifiers using the dropdown menu. Clicking on the Explain results! button then generates a LIME explanation HTML object, which is rendered in an HTML Iframe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/0*AmFZg-VIVQLOjmPT.gif" /></figure><p>Although Flask isn’t a data dashboarding tool (it is a <a href="https://wsgi.readthedocs.io/">WSGI</a> web framework that began as a wrapper around <a href="https://werkzeug.palletsprojects.com/">Werkzeug</a> and <a href="https://palletsprojects.com/p/jinja/">Jinja</a>), it provides a simple plugin-based architecture from which developers can build and extend interfaces for complex applications. The key strength of Flask is its robustness in a production setting and the numerous extensions around it in the Python ecosystem.</p><p>To build a LIME explainer dashboard using Flask, it is required to know about the following pieces of technology:</p><ul><li><strong>HTML/JavaScript</strong>: A page’s overall structure and content is defined using HTML. Any actions that need to be triggered based on a field’s value or user input needs to be defined using JavaScript, either through the HTML file itself, or loaded from an external source.</li><li><strong>CSS</strong>: A page’s styles and layout are defined using a CSS file.</li><li><strong>Jinja2</strong>: This is a templating engine that generates the page’s HTML dynamically from within Python. It is necessary because of security reasons (not using templates and passing around unescaped, static HTML can result in cross-site scripting attacks). The template engine is controlled from within Python, and the final HTML is rendered using Flask methods.</li></ul><h4>Flask app: directory structure</h4><p>The directory structure used for the Flask app is shown below. The required styles are configured using CSS in the static/style.css file, and the HTML template to be rendered is defined in templates/index.html. Any trained sentiment classifier models go into the models directory. The explainer class is defined in lime_explainer.py and the Flask routes in app.py.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/278/1*Gqep-YG-3My3sHPQCUOSHw.png" /><figcaption>Directory structure for Flask LIME explainer app</figcaption></figure><p>For this use case, the application code in Flask is written in the simplest way possible. Two routes are defined (the default route &#39;/&#39; and the route for the LIME results, &#39;result&#39;). Note that the results route uses a POST request, meaning that it only generates HTML (via Jinja) once the user inputs some information to the app and interacts with it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/da682577f6105f8975e646355bb15c6b/href">https://medium.com/media/da682577f6105f8975e646355bb15c6b/href</a></iframe><p>The code for the Flask LIME explainer app is available here on GitHub:</p><p><a href="https://github.com/prrao87/fine-grained-sentiment-app">prrao87/fine-grained-sentiment-app</a></p><h3>Option 2: Dash</h3><p>An alternate approach to designing a LIME dashboard is using Plotly’s <a href="https://plotly.com/dash/">Dash</a> library. Dash is a framework for building analytical web applications in Python. The benefits to using Dash are twofold: developers can design applications using just Python (no JavaScript required), and they have full control over the design and structure of the app via CSS. A LIME explainer app written using Dash is demonstrated below. Just as in the case of the Flask app, clicking on the Explain results button generates a LIME explanation HTML object, which is rendered via Dash&#39;s wrapper around HTML Iframes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/927/0*n8d-eAj-HRUx6fDK.gif" /></figure><h3>Dash app: directory structure</h3><p>The directory structure used for the Dash app is shown below. The required styles are configured using CSS in the assets/style.css file. Unlike in the Flask example, the HTML layout of the app and the routes/interactions are defined using pure Python, in app.py. Any trained sentiment classifier models go into the models directory. The explainer class is defined in lime_explainer.py.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/267/1*pckrbaov8GK2MUK9U9-fNw.png" /><figcaption>Directory structure for Dash LIME explainer app</figcaption></figure><p>The key components of a Dash application’s code are described below. The first is the application layout, which is written in Python using Dash’s HTML wrappers.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b32b79a89b5de7991f8eb46c273db7c3/href">https://medium.com/media/b32b79a89b5de7991f8eb46c273db7c3/href</a></iframe><p>Common HTML objects such as headers, labels, text inputs and Iframes can be easily added using Python constructs as shown.</p><p>The next component is the <em>callback</em>, which is a reactive, functional piece of code that allows developers to observe, modify and update properties of any component in the UI. In Dash, callbacks are defined using Python’s decorator syntax. Callbacks are extremely powerful ways to encode interactivity in an application, as they track states <em>on button-clicks</em> as well as on data updates. The primary callback used in the LIME explainer is shown below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/06d7646a9aba4f825ea39e580db81fc6/href">https://medium.com/media/06d7646a9aba4f825ea39e580db81fc6/href</a></iframe><p>As per the above construct, we pass the user’s button click (either the “Submit” or the “Clear” button), as well as the <em>states</em> of the dropdown menu, number of samples entered in the text input, and the text sample for which we want to predict the sentiment. Tracking the state (as opposed to just the <em>value</em>) of each text field here is important — it allows us to tie the execution of the callback to the <em>button click</em>, rather than executing it every time a value in the text input is updated.</p><p>The code for the Dash LIME explainer app is also available on GitHub:</p><p><a href="https://github.com/prrao87/fine-grained-sentiment-app-dash">prrao87/fine-grained-sentiment-app-dash</a></p><h3>Option 3: Streamlit</h3><p>Another alternative is to use <a href="https://www.streamlit.io/">Streamlit</a> to build the explainer app. This is by far the fastest approach, requiring a very basic knowledge of web development and the fewest lines of code. Unlike the Flask and Dash approaches, the Streamlit app uses its own styles and layout (no customizations via CSS are possible, at least using conventional means). The Streamlit LIME explainer app is shown in action, below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/266/1*DBnUg3OjQAmwev5HiIX8Bw.png" /><figcaption>Directory structure for Streamlit LIME explainer app</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/0*F1R6jGtk-O9vxaUd.gif" /></figure><p>Because Streamlit is designed from the ground up to assist in the fast creation and sharing of web apps, the LIME dashboard app has an exceptionally simple directory structure. All the code for the web application is written in a single file, app.py - this includes widgets, structure, interactions and all user inputs to make LIME predictions. The remarkable aspect of this design is that even with all this functionality crammed into one file, it is still very concise (~40 lines of code!). As a result, we only use separate directories for application-specific entities, such as models and data.</p><p>The entire code for the Streamlit dashboard is shown below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d67ce78e3e18131067c261997cf3fcd8/href">https://medium.com/media/d67ce78e3e18131067c261997cf3fcd8/href</a></iframe><p>Headers are written in markdown syntax. Note that because Streamlit’s API is designed for speed and ease of use, it does not allow easy access to the underlying HTML, unlike Flask and Dash. As a result, we are required to explicitly use unsafe rendering of raw HTML strings (using the unsafe_allow_html keyword) to obtain centre-aligned headers in this case. The HTML output from the LIME explainer is then rendered using a <a href="https://docs.streamlit.io/en/stable/develop_streamlit_components.html">Streamlit HTML component</a>, which is a custom component that displays the HTML string in an Iframe.</p><p>As always, the code for the Streamlit LIME explainer app is available on GitHub:</p><p><a href="https://github.com/prrao87/fine-grained-sentiment-app-streamlit">prrao87/fine-grained-sentiment-app-streamlit</a></p><h3>Deployment</h3><p>For the purposes of this post, all three apps were deployed using <a href="https://www.heroku.com/what">Heroku</a>, a PaaS system that enables developers to build, run and operate applications on the cloud. Each of the three frameworks used to build the app have good deployment guides for Heroku, shown below.</p><ul><li>Flask: <a href="https://stackabuse.com/deploying-a-flask-application-to-heroku/">Deploying a Flask Application to Heroku</a></li><li>Dash: <a href="https://dash.plotly.com/deployment">Deploying Dash apps</a></li><li>Streamlit: <a href="https://towardsdatascience.com/deploy-streamlit-on-heroku-9c87798d2088">How to deploy Streamlit on Heroku</a></li></ul><p>However, it is also relatively straightforward to deploy Flask or Dash apps using a production-grade WSGI web server, such as <a href="https://gunicorn.org/">gunicorn</a> and a load balancer such as <a href="https://www.nginx.com/">Nginx</a>. During deployment, Nginx <a href="https://www.digitalocean.com/community/tutorials/how-to-deploy-python-wsgi-apps-using-gunicorn-http-server-behind-nginx">acts as a reverse proxy that sits in front of the web server</a>, to handle a large number of requests with a high degree of reliability. Similarly, Streamlit also offers a means to <a href="https://docs.streamlit.io/en/stable/streamlit_faq.html">manually deploy apps</a> using a combination of Docker and/or Nginx.</p><h3>A note on scalability in Python web frameworks</h3><p>The Web Server Gateway Interface (WSGI) is a <a href="https://www.python.org/dev/peps/pep-3333/">Python specification</a> developed as a standard for Python web frameworks to interact with web servers. Having such a system makes it easy for developers to place a service such as Nginx in front of the Python web application (e.g., written in Flask or Dash) as a reverse proxy that forwards all requests to the web application. <strong>A key property of WSGI-compatible servers is that they are synchronous</strong>. This means that each request <em>blocks</em> the server until it receives a response from the application, termed a <em>blocking operation</em>. The typical way WSGI servers incorporate scalability is through the use of multithreading, where a new thread is created for each request in order to be able to handle multiple requests simultaneously. This is then combined with running multiple worker processes of the web server, which does scale well, but is limited by the number of physical cores available on a given machine.</p><p>The WSGI approach, as used by frameworks like Flask and Dash, can cap out in terms of scaling if there are a very large number of requests coming in. The way this is addressed in production systems is by scaling <em>horizontally</em>, i.e., adding more and more servers, and using load balancing services like Nginx that can evenly distribute the load between all servers during times of high request volumes.</p><p>Recent Python web app frameworks like Streamlit, instead rely on a different concurrency system altogether. Streamlit uses a <a href="https://www.tornadoweb.org/en/stable/">Tornado</a> web server under the hood, which was designed from the ground up to use <a href="https://www.tornadoweb.org/en/stable/guide/async.html">asynchronous event loops</a>. In this system, a single thread is used, which implements <em>non-blocking</em> functions that execute in order of arrival. This approach can easily achieve a very high degree of concurrency in the web application, which, in apps that rely largely on I/O bound operations, can really help scale the system to handle a large volume of requests simultaneously.</p><p>As always, there is no one single rule to decide which concurrency approach is better than the other. Depending on the exact use case and the application at hand, either a WSGI-based or asynchronous event loop-driven service can be a suitable option.</p><blockquote><em>Further reading: </em><a href="https://www.475cumulus.com/single-post/2017/04/03/WSGI-Is-Not-Enough-Anymore"><em>WSGI is Not Enough Anymore</em></a><em> — Parts I, II and III</em></blockquote><h3>When does it make the most sense to use each framework?</h3><p>This section discusses the situations in which each framework is the most suitable for the task at hand.</p><h4>Flask</h4><p>The power of Flask lies in its ability to allow developers to use <em>any</em> combination of front-end tools to create a web application. This includes form-entry extensions, such as <a href="https://wtforms.readthedocs.io/en/stable/">WTForms</a> and <a href="https://flask-login.readthedocs.io/en/latest/">Flask-Login</a>, as well as JavaScript visualization libraries ( <a href="https://www.highcharts.com/">Highcharts</a> or <a href="https://d3js.org/">D3</a>). In addition, Flask provides developers full access to the underlying page structure and user interactions, via HTML, CSS, jQuery and <a href="https://pythonhosted.org/Flask-Bootstrap/">Bootstrap</a>, allowing a huge amount of flexibility in building very complex applications based on the project’s requirements.</p><p>With regard to the LIME explainer dashboard shown in this post, Flask is an excellent option for any one of the following scenarios:</p><ul><li>The app is written by a team with expertise in JavaScript, HTML and CSS, as well as the different HTTP request methods (GET, POST, etc.)</li><li>The app is a part of a much larger dashboard powered by a JavaScript framework on the front end</li><li>Multiple RESTful endpoints that process and serve data already exist (following which the LIME explainer can just be written as another endpoint feeding into the front end)</li></ul><h4>Dash</h4><p>Dash is a great option for developers that require a high degree of customizability for their applications while working primarily in a Python environment. A number of powerful front end visualization tools (from the <a href="https://plotly.com/javascript/">Plotly.js</a> JavaScript library) are provided right out of the box, allowing developers to focus primarily on styling their apps and adding user interactions. Because Dash is built on top of Flask, it follows a similar deployment strategy as that of Flask, making it very easy to use for teams that already have experience implementing Flask apps in production.</p><p>In general, Dash is an excellent option to build a LIME explainer dashboard in any one of the following scenarios:</p><ul><li>The app needs to be written to integrate with an existing Flask application</li><li>The team developing and deploying the app is experienced in Python (but not so much in JavaScript)</li><li>A good degree of customization is required for the dashboard (which Dash allows by providing developers access to the underlying CSS)</li></ul><h4>Streamlit</h4><p>As a relative newcomer to the ecosystem, Streamlit shines in situations where data scientists need to quickly share their work with a larger team via an interactive app. It also works fabulously for an individual data scientist to quickly and interactively explore a dataset or the performance of a model on individual samples.</p><p>For the LIME explainer dashboard, Streamlit is a great alternative to use in any of the following scenarios:</p><ul><li>The app is written by a team (or individual) with minimal web development experience</li><li>The app needs to be rapidly built in the least time and using the fewest lines of code possible</li><li>Developers want to spend more time building interactive tools and the least time possible in customizing the appearance of the application</li></ul><h3>Conclusions</h3><p>This post highlighted three different approaches to building a LIME explainer interactive dashboard application. Streamlit is by far the least verbose and easiest to learn among all the options. Flask requires the most upfront time investment, in terms of learning the various pieces that fit together (HTML, CSS, jQuery/JavaScript, Jinja2 and HTTP request methods). Plotly Dash sits nicely in between Flask and Streamlit, in terms of complexity and initial effort to get the dashboard up and running. Depending on the composition and skill set of the team undertaking such a project, any of the three options can be the most suitable.</p><p>The primary reason to build a LIME dashboard as shown is to allow people who aren’t data scientists to examine an NLP classifier’s results. Providing an interactive means to test an individual sample’s results on the fly can help diagnose problems with NLP models and improve model interpretability, at least to some extent. Have fun playing with and customizing the code from each repo (shown below) for your own use case!</p><ul><li><a href="https://github.com/prrao87/fine-grained-sentiment-app">LIME explainer app: Flask</a></li><li><a href="https://github.com/prrao87/fine-grained-sentiment-app-dash">LIME explainer app: Dash</a></li><li><a href="https://github.com/prrao87/fine-grained-sentiment-app-streamlit">LIME explainer app: Streamlit</a></li></ul><p><em>Originally published at </em><a href="https://prrao87.github.io/blog/lime-explainer-dashboards/"><em>https://prrao87.github.io</em></a><em> on October 24, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bfe12e4592d4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/build-a-lime-explainer-dashboard-with-the-fewest-lines-of-code-bfe12e4592d4">Build a LIME explainer dashboard with the fewest lines of code</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An extensible, interactive visualization framework to measure gender bias in the news]]></title>
            <link>https://medium.com/data-science/an-extensible-interactive-visualization-framework-to-measure-gender-bias-in-the-news-56e33ba1655c?source=rss-efdb4f4a0d1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/56e33ba1655c</guid>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[editors-pick]]></category>
            <category><![CDATA[gender-equality]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Prashanth Rao]]></dc:creator>
            <pubDate>Fri, 25 Sep 2020 03:52:07 GMT</pubDate>
            <atom:updated>2021-01-12T19:19:53.112Z</atom:updated>
            <content:encoded><![CDATA[<h4>How we successfully integrated Dash (by Plotly) into our NLP and linguistics research to study women’s portrayal in mainstream Canadian news</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N5z7ADcd__2bEWAFuB1i-w.png" /><figcaption>How balanced is our representation of gender in the news? (Image credits: <a href="https://pixabay.com/illustrations/gender-equality-man-woman-female-1977912/">Tumisu via Pixabay</a>)</figcaption></figure><h3>Background</h3><p>Over the last several months, I’ve been working at the Discourse Processing Lab at Simon Fraser University (under the leadership of <a href="https://www.sfu.ca/~mtaboada/">Dr. Maite Taboada</a>), where we’ve been actively developing and extending <a href="https://gendergaptracker.informedopinions.org/">the Gender Gap Tracker</a>, an automated Natural Language Processing (NLP) system that measures men and women’s voices on seven major Canadian news outlets in real time. The Gender Gap Tracker is a collaboration between <a href="https://informedopinions.org/">Informed Opinions</a>, a non-profit dedicated to amplifying women’s voices in the media and Simon Fraser University, through the <a href="https://www.sfu.ca/discourse-lab.html">Discourse Processing Lab</a> and the <a href="https://www.sfu.ca/big-data/big-data-sfu">Big Data Initiative</a>.</p><p>At a high level, we perform the following steps on a daily basis to generate statistics that allow us to quantify the number of men and women quoted in the news:</p><ul><li>Scrape the text and metadata of news articles from the daily web content of seven Canadian English-language news outlets.</li><li>Process each article’s text using a variety of NLP techniques (syntactic parsing, named entity recognition, quote extraction) and gender services in order to identify quoted people and their genders.</li><li>Compute and store individual/aggregated statistics that measure the number of quoted people (i.e., <em>sources</em>) from either gender.</li></ul><blockquote><strong>Disclaimer</strong>: We acknowledge that gender is non-binary, and that there are different social, cultural, and linguistic conceptualizations of gender. For this project, we rely on name lookups from internal/external gender services in order to classify people mentioned and quoted as female, male, or other.</blockquote><p>In addition to our daily pipeline, we also run a <em>monthly</em> topic modelling pipeline that performs large-scale discovery of topics using Latent Dirichlet Allocation (LDA), on each month’s English news articles. On average, we process 20,000–30,000 articles’ worth of text per month to generate 15 topics (i.e., keyword distributions) for that month, that are then labelled by a human based on real-world knowledge of the month’s events.</p><p>All our NLP utilities (including the upstream tools that perform named entity recognition and quote extraction to extract gender statistics) are developed using Python and <a href="https://spacy.io/">spaCy</a>. Topic modelling, for reasons involving the sheer size of data being handled, is done using <a href="https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/3741049972324885/3783546674231782/4413065072037724/latest.html">PySpark’s scalable implementation of LDA</a>.</p><p>Although we do showcase our primary gender statistics (i.e., the proportion of female/male sources quoted by each outlet) on a live, interactive tracker website (<a href="https://gendergaptracker.informedopinions.org/">hosted by Informed Opinions</a>), we quickly realized the value of building our own custom dashboard for research and exploration purposes. The aim of our research dashboard (hosted <a href="https://gendergaptracker.research.sfu.ca/">here</a>, on Simon Fraser University’s servers) is to provide data scientists and software developers at the Discourse Processing Lab with the means to easily extend the existing data analysis framework, and to help answer new research questions that may arise in the future.</p><p>We chose the <a href="https://plotly.com/dash/">Dash</a> framework (by <a href="https://plotly.com/about-us/">Plotly</a>) to build our own custom interactive applications within a multi-page dashboard, all in Python. In this article, I’ll highlight some of our results, design decisions, and why Dash is a great framework for tasks related to data science research and exploration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ngj7XA8DPXf_NjpMC0LhiQ.png" /><figcaption>Multi-page research dashboard built using Dash: <a href="https://gendergaptracker.research.sfu.ca/">https://gendergaptracker.research.sfu.ca/</a></figcaption></figure><h3>A brief overview of the existing apps</h3><p>This section highlights a range of interactive tools implemented in the dashboard. Each high-level function is separated into its own application, accessible by clicking the respective button on the <a href="https://gendergaptracker.research.sfu.ca/">home page</a>.</p><h4>Text analyzer</h4><p>This app takes in a user’s input text, typically from a news article or blog post, and runs an NLP algorithm on the backend server to extract the text’s quoted sources and people mentioned. The detected names are passed through our internal cache and gender service scripts to assign each name a gender (any one of ‘female’, ‘male’ or ‘unknown’). A doughnut chart showing the proportion of sources from each gender is displayed, along with the detected names and the quotes in the article/blog post.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EOcyEbQ54VMXgmi5wfycZA.gif" /><figcaption><a href="https://gendergaptracker.research.sfu.ca/apps/textanalyzer">Text analyzer dashboard</a> for the Gender Gap Tracker</figcaption></figure><p>Because journalists employ a wide range of styles in naming the persons they quote, a naive system that simply looks for quotation marks in the text is nowhere near sufficient. The <a href="https://en.wikipedia.org/wiki/Parse_tree#Dependency-based_parse_trees">dependency-based parse tree</a> of a text block, obtained using spaCy, plays a key role in our quote extraction system, locating the beginning and end of quotations using a sentence’s syntactic structure. We first identify key verbs that are commonly used in quotation (defined using a custom, hand-curated verb list), and then locate the objects of each verb, i.e., clauses that are dependent on the verb, based on the parse tree. This allows us to find speakers of quotes regardless of the sentence structure and the presence of quotation marks.</p><p>In addition, a coreference resolution algorithm (powered by <a href="https://huggingface.co/coref/">neuralcoref</a>) is used to link pronoun mentions (‘he’ or ‘she’) to a prior named entity of the type ‘PERSON’. This allows us to capture quotes that do not directly name the speaker. The example snippet shown below highlights the range of quotes detected by our system.</p><blockquote>Prime Minister Justin Trudeau said Saturday that Canada will contribute $300 million towards the international effort to battle COVID-19.</blockquote><blockquote>“None of us have been spared from the effects of COVID-19 and none of us can beat it alone.”</blockquote><blockquote>Canada will contribute $180 million to address the immediate humanitarian and development impacts of the pandemic, he said.</blockquote><blockquote>“We are happy to see Minister Gould pushing for more investment because more is desperately needed,” said Stuart Hickox, the Canadian director of the One Campaign.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/927/1*O24d0-ArjnDzdtgC3_NFwQ.png" /><figcaption>Example of the various kinds of quotes and speakers detected by the <a href="https://gendergaptracker.research.sfu.ca/apps/textanalyzer">text analyzer app</a></figcaption></figure><p>Try out your own custom news snippets <a href="https://gendergaptracker.research.sfu.ca/apps/textanalyzer">on the text analyzer app</a> to get the gender breakdown of the people quoted/mentioned!</p><h4>Topic model explorer</h4><p>In this app, we showcase a new way to visualize the relationship between the topics covered in a news corpus and the gender of those quoted. Our primary goal through this app is to analyze whether female or male sources are more likely to be associated with specific topics in the news. As described earlier, we perform large-scale topic discovery on each month’s data using an LDA algorithm implemented in Spark. The topic keywords discovered for that month, along with other relevant metadata, are written to a MongoDB database (which also hosts all the raw text data from the news articles). These keywords are then inspected by a human, at the start of each month, and labelled manually based on knowledge of the month’s events.</p><p>Because we store the topic distribution (as predicted by LDA) on a per-document basis, we can average these values based on the outlet that published the article. We plot the topic names, as labelled by a human, and these mean ‘topic intensities’ by outlet on a heat map, as shown below for the month of August 2019. The darker the colour, the more strongly that topic was covered, on average, by that outlet in that particular month.</p><p>To study the relationship between topics and the gender of those quoted in articles pertaining to that topic, we first divide our article set for a given month into two subsets — those that quote a majority of female sources (i.e., at least one more female source quoted than male sources), and those that quote a majority of male sources. We then perform a similar averaging of topic intensities (i.e., the topic weights per article obtained from LDA) for each outlet. The results are once again plotted as a heat map, this time with a divergent colour scale from red to blue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SEa3JkGSKxLiEJzZNeAciA.png" /><figcaption>Average topic intensity and topic gender prominence for<strong> August 2019</strong>, in <a href="https://gendergaptracker.research.sfu.ca/apps/topicmodel">the topic model dashboard app</a></figcaption></figure><p>The topic intensity plot (green) for August 2019 clearly shows that ‘<em>Sports</em>’ was a major topic that month, as were ‘<em>Crime and police investigations</em>’ and ‘<em>Business and consumer products</em>’. ‘<em>U.S. politics</em>’ was covered particularly extensively by <em>Huffington Post Canada</em>, in comparison to the other outlets. What is more interesting, however, is that in the second heat map, the ‘<em>Sports</em>’ and ‘<em>Healthcare &amp; medical services</em>’ topics exhibit a strong female<em> prominence</em>, which is a term we use to describe topics that are strongly associated with women being quoted the majority of the time on average for that topic. The darker the red colour, the stronger the female prominence for a given topic and a given outlet. Conversely, the darker the blue colour, the stronger the male prominence for that topic.</p><p>A deeper analysis on the ‘<em>Sports</em>’ topic for August 2019 showed that this was the period in the aftermath of the US women’s soccer team winning the FIFA 2019 women’s world cup. In addition, Canadian tennis player Bianca Andreescu had a dominant performance in the U.S. Open (which she later went on to win in September). As a result of numerous female players and coaches being quoted over the course of August 2019, the sports topic, which would normally be dominated by the quotes of male players and coaches, showed a strong female prominence in this particular month. Unsurprisingly, the ‘<em>U.S. politics</em>’ and Canadian ‘<em>Federal politics</em>’ topics exhibit strong male prominence, due to the sheer number of male politicians in both the U.S. and Canada being regularly quoted in articles pertaining to politics.</p><p>Overall, we obtained some very interesting findings from <a href="https://gendergaptracker.research.sfu.ca/apps/topicmodel">the topic model app</a>, which can be explored in more detail for all months dating back to October 2018.</p><h4>Top-quoted sources app</h4><p>In this app, the quoted persons (male or female) from each article for are collected from our MongoDB database (after having been calculated independently by the NLP modules that run on a daily basis), and then aggregated to retrieve the number of quotes per person for any given month. These lists of top-quoted men and women are plotted as lollipops, the top 15 of which are shown below for the month of May 2020. Each dot represents the man/woman’s quote count, and the length of the line connecting them represents the disparity in counts between either gender.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/741/1*BqDtVvwk35mY52sXVtv5GA.png" /><figcaption>Top 15 quoted men and women in May 2020 as per the <a href="https://gendergaptracker.research.sfu.ca/apps/topsources">top-quoted sources app</a></figcaption></figure><p>It is very interesting that 10 of the top 15 quoted women in May 2020 are either medical officers or healthcare experts, as opposed to just 1 of the top 15 quoted men being from the healthcare domain. Due to the COVID-19 pandemic, many expert women (provincial/regional medical officers and physicians) holding prominent positions in various regions of Canada were brought to the forefront in the news. In fact, our data shows that Bonnie Henry, the Chief Provincial Health Officer of British Columbia, was the most quoted woman all through March-July 2020. The top-quoted men during the same period overwhelmingly tended to be politicians, both Canadian and American, presumably due to over-representation of men in this domain.</p><p>Another interesting aspect of the lollipop plot shown above is its distribution — not only are men always quoted much more frequently than women, but also, the top-quoted people (both men and women) tend to be quoted <em>far</em> more often than people in the middle or bottom. This points to the <a href="https://www.investopedia.com/terms/1/80-20-rule.asp">Pareto principle</a>, i.e., that a large proportion of resources is held by a small percentage of the population, also known as the 80/20 rule. While it is certainly possible that some people (especially politicians) get preferential treatment by the media because they already have a voice, recent evidence from the COVID-19 pandemic suggests that having more women in positions of power can increase the likelihood of journalists quoting them and improve gender parity overall.</p><h3></h3><p>Thanks for the shout out to the #GenderGapTracker! The only silver lining is that we have seen the percentage of women quoted increase a bit over the last few months. We think it&#39;s because so many public health officers are being quoted. And many of them happen to be women.</p><p>Similar trends and patterns can be explored in detail using <a href="https://gendergaptracker.research.sfu.ca/apps/topsources">the top-quoted sources app</a>, for all months dating back to October 2018.</p><h4>Monthly trends app</h4><p>This app visualizes a monthly time series of the number of quotes for a given person (male or female). Because of the size of our data (500,000+ articles with thousands of unique male/female sources over two years, and counting), we only display the names of prominent, public-facing men/women that appear in a month’s top 50 quoted people. The trends are displayed using a line chart, as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TA1Xe_tFpdtTSMu3hxHuwQ.png" /><figcaption>Comparing four prominent women in Federal/provincial healthcare positions using the <a href="https://gendergaptracker.research.sfu.ca/apps/topsourcetrends">monthly trends app</a></figcaption></figure><p>Exposing the number of quotes as a time series allows us to study interesting phenomena pertaining to individuals, such as “the rise of Bonnie Henry”. In the early months of the COVID-19 pandemic (March-April 2020), a sharp increase was observed in the quote count for Bonnie Henry, the Chief Provincial Health Officer of British Columbia. Interestingly, her quote count is significantly higher than that of her federal counterparts (Theresa Tam, the Chief Public Health Officer of Canada and Patty Hajdu, the Federal Minister of Health), as well as that of Christine Elliott, Minister of Health for Ontario, a province with a much higher population than British Columbia. We think this could be due to two factors: 1) British Columbia was a success story in the early days of the COVID-19 pandemic, leading to Bonnie Henry being talked about (and hence quoted) more frequently, and 2) both Ontario and Ottawa had multiple prominent public-facing women for journalists to quote, reducing each woman’s overall share.</p><p>Seeing whether <a href="https://gendergaptracker.research.sfu.ca/apps/topsourcetrends">the monthly trends app</a> can help predict future outcomes, for example, the dominant players in elections, is a very interesting prospect, and we are keen to explore this further.</p><h3>A look under the hood</h3><p>In this section, I’ll explain some of the technical details involved in building the dashboard, including some ideological and design decisions.</p><h4>Why Dash?</h4><p>In a research-focused environment such as ours, front-end web development skills (and knowledge of the associated JavaScript tools) are not easily available. Python is the natural choice of language for all aspects of this project, because of its massive, robust NLP and data science ecosystem. However, the Python visualization ecosystem <a href="https://pyviz.org/tools.html">is immensely complex</a>, and it can quickly become overwhelming to decide between the multitude of options for the specific tasks at hand. There exist a number of great libraries for building dashboards and interactive visualizations in Python, such as <a href="https://panel.holoviz.org/index.html">Panel</a>, <a href="https://bokeh.org/">Bokeh,</a> <a href="https://www.streamlit.io/">Streamlit</a>, and <a href="https://dash.plotly.com/introduction">Dash</a>.</p><p><a href="https://panel.holoviz.org/index.html">Panel</a> is a novel, high-level dashboarding solution for Python that allows users to create interactive web apps and connect user-defined widgets to plots, images, tables or text. Unlike many other approaches, Panel attempts to be plotting library and environment-agnostic, i.e., it is designed to support nearly all Python plotting libraries, and work just as well in a Jupyter notebook or on a standalone web server. The key defining feature of Panel is that it allows users to separate their science/engineering/business logic from their GUI implementation, without getting tied to any particular GUI or web frameworks. Developed under the support of <a href="https://anaconda.com/">Anaconda Inc.</a>, Panel is an ambitious project that provides users the flexibility to move expensive computations and data processing between the server side and the client side, depending on the application’s requirements and the size of the data. The <a href="https://panel.holoviz.org/Comparisons.html">Panel documentation</a> contains a rich and detailed explanation of its features in comparison with other Python dashboarding frameworks.</p><p><a href="https://www.streamlit.io/">Streamlit</a> is a light-weight, minimalistic framework to build dashboards in Python. The main focus of Streamlit is to provide users with the ability to rapidly prototype their UI designs using as few lines of code as possible. All the heavy lifting typically required to deploy a web application, such as defining the backend server and its routes, handling HTTP requests, etc., are abstracted away from the user. A certain degree of customizability is traded off for ease of use and rapid setup, making it ideal for early-stage interactive exploration of data and modelling results.</p><p><a href="https://bokeh.org/">Bokeh</a>, also developed with the support of Anaconda Inc., is a visualization library that provides lower-level primitives for charts and widgets that can be used to power other higher-level charting libraries (such as Panel, or <a href="https://github.com/spotify/chartify">chartify</a>). In Bokeh, <a href="https://docs.bokeh.org/en/latest/docs/user_guide/server.html">high-level “model objects” </a>(representing plots, axes, or other attributes related to plotting) are created in Python (or any other language), converted to JSON, and then consumed by a JavaScript client library, BokehJS. This decoupled architecture allows Bokeh to extend to other languages than Python, provide the backend code that runs the server, and ultimately power the UI event in the front-end (browser) via JavaScript.</p><p><a href="https://dash.plotly.com/">Dash</a> is a powerful, highly customizable library for building dashboards in Python. It is written on top of Flask, Plotly.js and React.js, allowing for fluid, <a href="https://www.outsystems.com/blog/posts/reactive-web-applications/">reactive web applications</a> that scale well to multiple client sessions. In terms of its architecture and functionality, Dash is most similar to Bokeh, in that it allows users to define higher-level plot objects in Python (using an underlying Flask web server), whose underlying attributes are then consumed by a JavaScript library (Plotly.js) that fires events on the client side. Unlike Streamlit and Panel, Dash does require users to have some knowledge of the underlying HTML and CSS details, greatly increasing users’ ability to customize the UI to their needs. Although Dash’s web server <a href="https://github.com/plotly/dash-alternative-viz-demo">can support other Python visualization libraries</a> (such as Altair, Seaborn or matplotlib), it is still somewhat opinionated, with users encouraged to stick to the Plotly graph API for the most part, because this API has been tried and tested in production.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GeCjlOjIhHkLHZ60nCJ8wA.png" /><figcaption>Key features of each dashboarding solution in Python</figcaption></figure><p>As can be seen in the above figure, Dash provides the ideal combination of all our primary requirements. To start with, Dash uses a web server built on top of Flask (unlike the other three frameworks, which use <a href="https://www.tornadoweb.org/en/stable/">Tornado</a>), and is deployed just like traditional Flask servers <a href="https://flask.palletsprojects.com/en/1.1.x/deploying/">using WSGI</a>. Because we already had other dashboards and backend APIs based on Flask, it made more sense to build our new system with some level of compatibility. Dash is designed to store a session’s state on a<em> </em>per-user<em> </em>basis, within the client (i.e., the browser). This is apt in terms of scalability for our use case, because we need a system where many simultaneous client sessions are able to view results from the database without necessarily using up server resources.</p><p>We opt to work around any potential computational bottlenecks on the server side by designing our apps such that they never request huge amounts of data from the server at any one time. Any NLP computations (e.g., in our text analyzer app) only request highly specific data (such as gender values for a given name) and run in near real-time, so the server is unlikely to be bogged down due to compute-heavy requests by many simultaneous client sessions. To avoid repeating expensive calculations for each user, a number of options are available for caching intermediate results (on the client side) that can be reused within the session. Most importantly, Dash offers a great deal of freedom in terms of page design and layout— it allows the developer to directly modify the underlying HTML and CSS attributes, providing much greater flexibility in customizing the dashboard’s appearance.</p><h4>Multi-page structure</h4><p>The dashboard is deployed using a multi-page layout, to make it more extensible as time goes on. A simple, responsive HTML layout <a href="https://www.w3schools.com/html/tryit.asp?filename=tryhtml_responsive_media_query3">based on this CSS template</a> is used to allow for flexible rearrangement of components based on the user device’s display resolution. Each app’s layout and source code is self-contained, <a href="https://github.com/sfu-discourse-lab/GenderGapTracker/tree/master/dashboard_for_research/apps">in its own Python file</a>, and within a clearly defined zone outside the sidebar as shown below. An arbitrary number of independent apps can be added using this structure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4qe5JpRkigYw6UCf37zEoQ.png" /><figcaption>Extensible, multi-page layout for the dashboard (designed using CSS, as per <a href="https://github.com/sfu-discourse-lab/GenderGapTracker/blob/master/dashboard_for_research/assets/style.css">this file</a>)</figcaption></figure><h4>Modularity and code readability</h4><p>A Dash app can be conceptualized as being made of three main blocks of code. The first of these is the <em>application layout</em>, which contains the structure of the web page, including all the interactive widgets and core components (dropdowns, text inputs, etc.) required by the user to modify an app’s state. All code is written in Python, using Dash’s wrappers around Plotly’s graph objects and generic HTML.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/90c5a6962d76f9daa33fb186623efdca/href">https://medium.com/media/90c5a6962d76f9daa33fb186623efdca/href</a></iframe><p>The next block defines the <em>callbacks </em>for the app, which are functions (defined using Python’s decorator syntax) that allow users to observe, modify and update the properties of any component in the UI. Because all Dash code is written in Python, it integrates very well with pandas, allowing specific results to be returned on the fly based on user input. In this example, a dropdown’s selected value is passed as an input argument to a callback, which executes asynchronously to filter the contents of a preloaded pandas DataFrame, returning just the required data. This data is output as a Plotly figure object and rendered as a line chart on the browser.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4269295f7acef7a1283eb09c8de52c6e/href">https://medium.com/media/4269295f7acef7a1283eb09c8de52c6e/href</a></iframe><p>The third block contains user-defined, custom <em>functions</em>, primarily written to make the callbacks and the application layout more readable. For example, if we were performing a slightly more complicated action along with filtering the DataFrame, that portion of code could be factored out into a separate function, making the callback much more readable. The example below is the same as the one above, except that the filtering steps are factored out to an external function, greatly improving readability in complex cases.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1962688029f421f0e06051c8d1c8b059/href">https://medium.com/media/1962688029f421f0e06051c8d1c8b059/href</a></iframe><p>Using this 3-block structure, arbitrarily complex apps can be designed to perform data manipulation and NLP tasks on the fly.</p><h4>Intermediate data storage</h4><p>One common problem that arises when performing NLP tasks is that of expensive bottleneck computations that can increase an app’s response time. For example, in the text analyzer app described earlier, we are required to perform named entity recognition, coreference resolution and gender prediction on a potentially large list of names, all in real time, based on arbitrary text input by the user. This is further complicated by the fact that multiple components in the app — the doughnut charts and multiple data tables — all need to access the same intermediate data.</p><p>Rather than perform the same (potentially expensive) calculation multiple times for each callback, we apply Dash’s inbuilt mechanisms to <a href="https://dash.plotly.com/sharing-data-between-callbacks">share state between callbacks</a>. The simplest way to do this is to store expensive-to-compute intermediate data as in-memory JSON, that can be accessed by every subsequent callback without wasteful recomputation. An example is shown below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5b16b3079b6da6d82317c2fffb2b5676/href">https://medium.com/media/5b16b3079b6da6d82317c2fffb2b5676/href</a></iframe><p>In the snippet shown above, a user first inputs a condition (e.g., a date filter), for which we read specific data from a MongoDB database using the pymongo library. Then, we perform some expensive calculation on this data (using the function expensive_calc) within the first callback. The data computed from this callback is stored as JSON and kept in memory on the client side, where it can then be accessed for free by the second callback. It is trivial to then transform the JSON data blob into a pandas DataFrame, following which it can be output to a Plotly figure object and rendered as a chart on the client side.</p><h3>Conclusions</h3><p>In building each application for the Gender Gap Tracker research dashboard, design decisions were taken to minimize the amount of data being transferred between the server and the client. Because our primary data storage layer is in MongoDB, all data must be passed around as JSON. Due to the immense size of the data in this project (500,000+ news articles from over a two year period, and counting), we only work with <strong>aggregated quantities </strong>(precomputed through intermediate operations on these JSON objects) in all our visual components. All charts (heat maps, lollipop plots and line charts) display precomputed values that either sum up or average the individual values over a specified time period. This is mainly done to maintain reasonable response times and to provide a fluid run time experience.</p><p>The Gender Gap Tracker is the result of many months’ work by a team of eight at Simon Fraser University, whose findings draw from various fields, including big data, computer science, social science, media and linguistics. We are excited to present more nuanced results of this interdisciplinary work for consumption by the broader public, and are always open to feedback!</p><h4>Acknowledgements</h4><p>We thank Alexandra Wilson at Simon Fraser University, who provided invaluable guidance on the design, layout and colour styles for the dashboard. We’re also grateful to to <a href="https://www.linkedin.com/in/alexandreaslopes/">Alexandre Lopes</a> at SFU Research Computing, who built and maintains the database and server infrastructure for the Gender Gap Tracker.</p><h4>Resources</h4><p>The research dashboard for the Gender Gap Tracker and our most recent results can be explored in detail here:</p><p><a href="https://gendergaptracker.research.sfu.ca/">Measuring gender bias in media - SFU</a></p><p>The code for the dashboard and the NLP functionality described in this article is available on the project’s GitHub repo, shown below.</p><p><a href="https://github.com/sfu-discourse-lab/GenderGapTracker/tree/master/dashboard_for_research">sfu-discourse-lab/GenderGapTracker</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56e33ba1655c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/an-extensible-interactive-visualization-framework-to-measure-gender-bias-in-the-news-56e33ba1655c">An extensible, interactive visualization framework to measure gender bias in the news</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Turbo-charge your spaCy NLP pipeline]]></title>
            <link>https://medium.com/data-science/turbo-charge-your-spacy-nlp-pipeline-551435b664ad?source=rss-efdb4f4a0d1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/551435b664ad</guid>
            <category><![CDATA[spacy]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Prashanth Rao]]></dc:creator>
            <pubDate>Sat, 02 May 2020 05:00:00 GMT</pubDate>
            <atom:updated>2024-01-24T04:03:29.647Z</atom:updated>
            <content:encoded><![CDATA[<h4>Tips and tricks to significantly speed up text preprocessing in spaCy using custom pipelines and joblib</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cRvMDq5Tbzpq-M7n.jpg" /><figcaption>Source: <a href="https://pixabay.com/illustrations/space-rocket-night-cartoon-3262811/">Pixabay</a></figcaption></figure><p>Consider you have a large text dataset on which you want to apply some non-trivial NLP transformations, such as stopword removal followed by lemmatizing the words (i.e. reducing them to root form) in the text. <a href="https://spacy.io/usage">spaCy</a> is an industrial-strength NLP library designed for just such a task.</p><p>In this post, the <a href="https://www.kaggle.com/nzalake52/new-york-times-articles">New York Times dataset</a> is used to showcase how to significantly speed up a spaCy NLP pipeline. The goal is to take in an article’s text, and speedily return a list of lemmas as well as unnecessary words, i.e. <em>stopwords</em>, removed.</p><p>Pandas DataFrames provide a convenient interface to work with tabular data of this nature — the spaCy NLP methods can be conveniently applied directly to the relevant column of the DataFrame. First, the news data is obtained by running the <a href="https://github.com/prrao87/blog/tree/master/_notebooks/data/spacy_multiprocess">preprocessing notebook</a> (./data/preprocessing.ipynb), which processes the raw text file downloaded from Kaggle and performs some basic cleaning on it. This step generates a file that contains the tabular data (stored as nytimes.tsv). A curated stopword file is also provided in <a href="https://github.com/prrao87/blog/tree/master/_notebooks/data/spacy_multiprocess">the same directory</a>.</p><h3>Load spaCy model</h3><p>Since we will not be doing any specialized tasks such as dependency parsing and named entity recognition in this exercise, these components are disabled when loading the spaCy model.</p><blockquote><strong>Tip: </strong>spaCy has a sentencizer component that can be plugged into a blank pipeline.</blockquote><p>The sentencizer pipeline simply performs tokenization and sentence boundary detection, following which lemmas can be extracted as token properties.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a92a8384a2fd601fcbfe246394cd8e64/href">https://medium.com/media/a92a8384a2fd601fcbfe246394cd8e64/href</a></iframe><p>A method is defined to read in stopwords from a text file and convert it to a set in Python (for efficient lookup).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/70c926f8add0680d37ad76b960963c4d/href">https://medium.com/media/70c926f8add0680d37ad76b960963c4d/href</a></iframe><h3>Read in New York Times Dataset</h3><p>The pre-processed version of the NYT news dataset is read in as a Pandas DataFrame. The columns are named date, headline and content - the text present in the content column is what will be preprocessed to remove stopwords and generate token lemmas.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cf04c7d37669a6ee2d474e3d0dd62dea/href">https://medium.com/media/cf04c7d37669a6ee2d474e3d0dd62dea/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/461/1*Jkt27i1d0A0EYzC1VxwHkQ.png" /></figure><h3>Define text cleaner</h3><p>Since the news article data comes from a raw HTML dump, it is very messy and contains a host of unnecessary symbols, social media handles, URLs and other artifacts. An easy way to clean it up is to use a regex that parses only alphanumeric strings and hyphens (so as to include hyphenated words) that are between a given length (3 and 50). This filters each document down to only meaningful text for the lemmatizer.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/34a1223fc22f32debf30cb7d6e7a60de/href">https://medium.com/media/34a1223fc22f32debf30cb7d6e7a60de/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/618/1*IzSS_Rqknc1TYeQrRbj-vA.png" /></figure><h3>Option 1: Sequentially process DataFrame column</h3><p>Now that we have just the clean, alphanumeric tokens left over, these can be further cleaned up by removing stopwords before proceeding to lemmatization.</p><p>The straightforward way to process this text is to use an existing method, in this case, the lemmatize method shown below, and apply it to the <a href="https://prrao87.github.io/blog/images/copied_from_nb/clean">clean</a> column of the DataFrame using pandas.Series.apply. Lemmatization is done using the spaCy&#39;s underlying <a href="https://spacy.io/usage/spacy-101#annotations">Doc representation</a> of each token, which contains a lemma_ property. Stopwords are removed simultaneously with the lemmatization process, as each of these steps involves iterating through the same list of tokens.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f2207a6d9aa528e291627efa024e80ab/href">https://medium.com/media/f2207a6d9aa528e291627efa024e80ab/href</a></iframe><p>The resulting lemmas are stored as a list in a separate column preproc as shown below.</p><pre>%%time<br>df_preproc[&#39;preproc&#39;] = df_preproc[&#39;clean&#39;].apply(lemmatize)<br>df_preproc[[&#39;date&#39;, &#39;content&#39;, &#39;preproc&#39;]].head(3)</pre><pre>CPU times: user 48.5 s, sys: 146 ms, total: 48.6 s Wall time: 48.6 s</pre><p>Applying this method to the <a href="https://prrao87.github.io/blog/images/copied_from_nb/clean">clean</a> column of the DataFrame and timing it shows that it takes almost a minute to run on 8,800 news articles.</p><h3>Option 2: Use nlp.pipe</h3><p>Can we do better? in the <a href="https://spacy.io/api/language#pipe">spaCy documentation</a>, it is stated that “processing texts as a stream is usually more efficient than processing them one-by-one”. This is done by calling a language pipe, which internally divides the data into batches to reduce the number of pure-Python function calls. This means that the larger the data, the better the performance gain that can be obtained by nlp.pipe.</p><p>To use the language pipe to stream texts, a new lemmatizer method is defined that directly works on a spaCy Doc object. This method is then called in batches to work on a <em>sequence</em> of Doc objects that are streamed through the pipe as shown below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7796ad0fc02e97296cdd2833147d86f0/href">https://medium.com/media/7796ad0fc02e97296cdd2833147d86f0/href</a></iframe><p>Just as before, a new column is created by passing data from the <a href="https://prrao87.github.io/blog/images/copied_from_nb/clean">clean</a> column of the existing DataFrame. Note that unlike in workflow #1 above, we do not use the apply method here - instead, the column of data (an iterable) is directly passed as an argument to the preprocessor pipe method.</p><pre>%%time<br>df_preproc[&#39;preproc_pipe&#39;] = preprocess_pipe(df_preproc[&#39;clean&#39;])<br>df_preproc[[&#39;date&#39;, &#39;content&#39;, &#39;preproc_pipe&#39;]].head(3)</pre><pre>CPU times: user 51.6 s, sys: 144 ms, total: 51.8 s Wall time: 51.8 s</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/462/1*sWWQG-de-DEQcgcIKto9nA.png" /></figure><p>Timing this workflow doesn’t seem to show improvement over the previous workflow, but as per the spaCy documentation, one would expect that as we work on bigger and bigger datasets, this approach should show some timing improvement (on average).</p><h3>Option 3: Parallelize the work using joblib</h3><p>We can do still better! The previous workflows sequentially worked through each news document to produce the lemma lists, which were then appended to the DataFrame as a new column. Because each row’s output is completely independent of the other, this is an <em>embarrassingly parallel</em> problem, making it ideal for using multiple cores.</p><p>The joblib library is recommended by spaCy for processing blocks of an NLP pipeline in parallel. Make sure that you pip install joblib before running the below section.</p><p>To parallelize the workflow, a few more helper methods must be defined.</p><ul><li><strong>Chunking:</strong> The news article content is a list of (long) strings where each document represents a single article’s text. This data must be fed in “chunks” to each worker process started by joblib. Each call of the chunker method returns a generator that only contains that particular chunk&#39;s text as a list of strings. During lemmatization, each new chunk is retrieved based on the iterator index (with the previous chunks being &quot;forgotten&quot;).</li><li><strong>Flattening:</strong> Once joblib creates a set of worker processes that work on each chunk, each worker returns a “list of lists” containing lemmas for each document. These lists are then combined by the executor to provide a 3-level nested final “list of lists of lists”. To ensure that the length of the output from the executor is the same as the actual number of articles, a “flatten” method is defined to combine the result into a list of lists containing lemmas. As an example, two parallel executors would return a final nested list of the form: [[[a, b, c], [d, e, f]], [[g, h, i], [j, k, l]]], where [[a, b, c], [d, e, f]] and [[g, h, i], [j, k, l]] refer to the output from each executor (the final output is then concatenated to a single list by joblib). A flattened version of this result would be [[a, b, c], [d, e, f], [g, h, i], [j, k, l]], i.e. with one level of nesting removed.</li></ul><p>In addition to the above methods, a similar nlp.pipe method is used as in workflow #2, on each chunk of texts. Each of these methods is wrapped into a preprocess_parallel method that defines the number of worker processes to be used (7 in this case), breaks the input data into chunks and returns a flattened result that can then be appended to the DataFrame. For machines with a higher number of physical cores, the number of worker processes can be increased further.</p><p>The parallelized workflow using joblib is shown below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7a563547157e2608d7cbe905a0a96c86/href">https://medium.com/media/7a563547157e2608d7cbe905a0a96c86/href</a></iframe><pre>%%time<br>df_preproc[&#39;preproc_parallel&#39;] = preprocess_parallel(df_preproc[&#39;clean&#39;], chunksize=1000)</pre><pre>CPU times: user 683 ms, sys: 248 ms, total: 932 ms Wall time: 17.2 s</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/462/1*p2Mu31iMUdHGYt3OvChrEA.png" /></figure><p>Timing this parallelized workflow shows significant performance gains (almost <strong>3x</strong> reduction in run time)! As the number of documents becomes larger, the additional overhead of starting multiple worker processes with joblib is quickly paid for, and this method can significantly outperform the sequential methods.</p><h3>Effect of chunk size and batch size</h3><p>Note that in the parallelized workflow, two parameters need to be specified — the optimum number can vary depending on the dataset. The chunksize controls the size of each chunk being worked on by each process. In this example, for 8,800 documents, a chunksize of 1000 is used. Too small a chunksize would mean that a large number of workers would spawn to deal with the large number of chunks overall, which can slow down execution. Generally, a chunksize of several hundred documents to a few thousand is a good starting point (of course, this depends on how big each document in the data is so that the chunks can fit into memory).</p><p>The batch size is parameter specific to nlp.pipe, and again, a good value depends on the data being worked on. For reasonably long-sized text such as news articles, it makes sense to keep the batch size reasonably small (so that each batch doesn&#39;t contain <em>really</em> long texts), so in this case 20 was chosen for the batch size. For other cases (e.g. Tweets) where each document is much shorter in length, a larger batch size can be used.</p><p><strong>It is recommended to experiment with either parameter to see which combination produces the best performance</strong>.</p><h3>Sets vs. Lists</h3><blockquote><strong>Important: </strong>Use sets over lists for lookups wherever possible.</blockquote><p>Note that in the get_stopwords() method defined earlier on, the list of stopwords read in from the stopword file was converted to a set before using it in the lemmatizer method for stopword removal via lookups. This is a very useful trick in general, but specifically for stopword removal, the use of sets becomes <strong>all the more important</strong>. Why?</p><p>In any realistic stopword list, such as this one for a news dataset, it’s reasonable to expect <em>several hundred</em> stopwords. This is because for downstream tasks such as topic modelling or sentiment analysis, there are a number of domain-specific words that need to be removed (very common verbs, useless abbreviations such as timezones, days of the week, etc.). Each word in each and every document needs to be compared against every word in the stopword list, which is an expensive operation over tens of thousands of documents.</p><p>It’s well known that sets have O(1) (i.e. constant) lookup time as opposed to lists, which have O(n) lookup time. In the lemmatize() method, since we&#39;re checking each word for membership in the set of stopwords, we would expect sets to be much better than lists. To test this, we can rerun workflow #1 but this time, use a stopword <em>list</em> instead.</p><pre>stopwords = list(stopwords)</pre><pre>%%time<br>df_preproc[&#39;preproc_stopword_list&#39;] = df_preproc[&#39;clean&#39;].apply(lemmatize)<br>df_preproc[[&#39;date&#39;, &#39;content&#39;, &#39;preproc_stopword_list&#39;]].head(3)</pre><pre>CPU times: user 1min 17s, sys: 108 ms, total: 1min 18s Wall time: 1min 18s</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/476/1*ST6ABVaF4XGimMauLkWweQ.png" /></figure><p>With a stopword list, producing the same result now takes ~ <strong>50% longer</strong> than it did before (with the set), which is a <strong>1.5x</strong> increase in run time! This makes sense because in this case, the stopword list is about 500 words long, and <em>each and every word</em> in the corpus needs to be checked for membership in this reasonable-sized list.</p><h3>Conclusions</h3><p>In this exercise, a news article dataset (NY Times) was processed using a spaCy pipeline to output a list of lemmas representing the useful tokens present in each article’s content. Because real-world news datasets are almost certainly bigger than this one, and can be unbounded in size, a fast, efficient NLP pipeline is necessary to perform any meaningful analysis on the data. The following steps are very useful in speeding up the spaCy pipeline.</p><p><strong>Disable unnecessary components in spaCy model:</strong> The standard spaCy model’s pipeline contains the tagger (to assign part-of-speech tags), the parser (to generate a dependency parse) and named entity recognition components. If any or none of these actions are desired, these components <em>must</em> be disabled immediately after loading the model (as shown above).</p><p><strong>Use sets over lists for lookups:</strong> When performing lookups to compare one set of tokens against another, always perform membership checks using sets — lists are significantly slower for lookups! The larger the list/set of stopwords, the bigger the performance gain seen when using sets.</p><p><strong>Use custom language pipes when possible:</strong> Setting up a language pipe using nlp.pipe is an extremely flexible and efficient way to process large blocks of text. Even better, spaCy allows you to individually disable components for each specific sub-task, for example, when you need to separately perform part-of-speech tagging and named entity recognition (NER). <a href="https://spacy.io/usage/processing-pipelines#disabling">See the spaCy docs</a> for examples on how to disable pipeline components during model loading, processing or handling custom blocks.</p><p><strong>Use multiple cores when possible:</strong> When processing individual documents completely independent of one another, consider parallelizing the workflow by dividing the computation across multiple cores. As the number of documents becomes larger and larger, the performance gains can be tremendous. One just needs to ensure that the documents are divided up into chunks, all of which must fit into memory at any given time.</p><p>I hope this was useful — have fun testing these out in your next NLP project!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=551435b664ad" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/turbo-charge-your-spacy-nlp-pipeline-551435b664ad">Turbo-charge your spaCy NLP pipeline</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fine-grained Sentiment Analysis (Part 3): Fine-tuning Transformers]]></title>
            <link>https://medium.com/data-science/fine-grained-sentiment-analysis-part-3-fine-tuning-transformers-1ae6574f25a6?source=rss-efdb4f4a0d1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/1ae6574f25a6</guid>
            <dc:creator><![CDATA[Prashanth Rao]]></dc:creator>
            <pubDate>Mon, 09 Sep 2019 11:24:10 GMT</pubDate>
            <atom:updated>2019-09-21T01:41:07.176Z</atom:updated>
            <content:encoded><![CDATA[<h4>Hands-on transfer learning using a pretrained transformer in PyTorch</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Anf4qCqSznUgoHTw7O6c2w.jpeg" /><figcaption>Source: <a href="https://pixabay.com/illustrations/light-bulbs-light-bulb-light-energy-1125016/">Pixabay</a></figcaption></figure><p>This is Part 3 of a series on fine-grained sentiment analysis in Python. <a href="https://towardsdatascience.com/fine-grained-sentiment-analysis-in-python-part-1-2697bb111ed4">Parts 1</a> and <a href="https://towardsdatascience.com/fine-grained-sentiment-analysis-in-python-part-2-2a92fdc0160d">2</a> covered the analysis and explanation of six different classification methods on the Stanford Sentiment Treebank fine-grained (SST-5) dataset. In this post, we’ll look at how to improve on past results by building a <strong>transformer-</strong>based model and applying transfer learning, a powerful method that has been <a href="https://hackingsemantics.xyz/2019/leaderboards/">dominating NLP task leaderboards</a> lately.</p><p>From the <a href="https://towardsdatascience.com/fine-grained-sentiment-analysis-in-python-part-1-2697bb111ed4">first article</a> in this series, the below classification accuracies and F1-scores were obtained on the SST-5 dataset:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/457/1*IjuIWu4APXLVUUkGA3BaZA.png" /></figure><p>In the below sections, we’ll discuss the key training, evaluation and explanation steps that show why transformers are inherently better than the above listed approaches for this task.</p><h3>What makes transformers so powerful?</h3><p>At the heart of the transformer architecture are the following key ideas that make it highly suitable for interpreting complex patterns in natural language:</p><ul><li><strong>Self-attention</strong>: This is a mechanism that transformers use to represent different positions of an input sequence and learn relationships between them.</li><li><strong>Multi-headed attention</strong>: Transformers delegate the task of self-attention to multiple “heads”, i.e. <a href="http://nlp.seas.harvard.edu/2018/04/03/attention.html#training-loop">jointly attending to information from different positions in the sequence</a> from different representation subspaces. This allows them to scale efficiently (with large datasets) using <em>unsupervised learning</em>.</li><li><strong>Transfer Learning</strong>: Recent leading transformer-based methods are enabled by <em>transfer learning</em> — that is, using the knowledge extracted from a prior setting (e.g. unsupervised training) and applying it to a different setting (e.g. sentiment classification or question answering). This is achieved via a two-stage process: <em>pretraining</em>, followed by <em>fine-tuning</em>, or <em>adaptation.</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/757/1*SYZT3DkrK-SLuZjw8LwvuQ.png" /><figcaption>Source: <a href="http://ruder.io/state-of-transfer-learning-in-nlp/">“The State of Transfer Learning in NLP” by Sebastian Ruder</a></figcaption></figure><p>In essence, all of the top transformer-based methods of late (<a href="https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf">GPT</a>, <a href="https://arxiv.org/pdf/1810.04805.pdf">BERT</a>, <a href="https://arxiv.org/pdf/1906.08237.pdf">XLNet</a>) use such a sequential transfer learning approach. Internally, they train a <em>language model</em> during the pretraining stage using unlabelled data from a large corpus, following which an additional task-specific module (attached downstream of the language model) is fine-tuned on custom data. Below is a visualization of this approach for BERT on multiple benchmark datasets.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*_G_AGNptaNmjfDOWQkjffQ.png" /><figcaption>Source: <a href="https://arxiv.org/pdf/1810.04805.pdf">Devlin et al. 2019</a></figcaption></figure><h3>Importance of pretraining objectives</h3><p>The pretraining step involves training a language model in an unsupervised manner — this determines how the model learns syntax, semantics and contextual information from the training corpus it is given. Recent evidence (from models such as <a href="https://openai.com/blog/better-language-models/">OpenAI’s GPT-2</a>) indicates that really large language models, given sufficient data and compute, will learn a great deal about language syntax! For the purposes of the transformer model described later in this article, the below pretraining objectives are useful to note.</p><ul><li><strong>OpenAI GPT</strong> uses a <em>left-to-right</em> language modelling objective during pretraining — that is, it learns to predict the most likely next token in the sequence in a left-to-right direction, just as they occur in natural language (at least for English). Such models are typically referred to as <em>classical</em> language models, or <strong><em>causal</em></strong> language models. The word causal here refers to the fact that the likelihood of a token appearing at a specific location was caused by the sequence of tokens that came before it.</li><li><strong>BERT</strong> uses a <em>masked </em>language model<em> </em>at its core, obtained by randomly masking 15% of a sequence’s tokens during pretraining — this allows the model to learn how to predict tokens that came before or after a previous token (it is trained <em>bidirectionally</em>, unlike GPT). In addition to masking, BERT uses a next-sentence prediction objective, where the model learns to predict if a sentence comes after a previous one. This approach comes with a longer training time compared to causal language modelling, since masking a portion of the tokens results in a smaller training signal.</li><li><strong>XLNet</strong> uses a <em>permutation</em> language modelling objective — unlike BERT, it masks each token of a sequence randomly (not just 15% of them) during pretraining. This way, the model learns to predict <em>random tokens</em> of the sequence in both directions, allowing the model to learn <em>dependencies</em> between tokens (not just which tokens are most likely in a given sequence). As one might imagine, this is more expensive in terms of training time and requires a much bigger language model to achieve a good baseline.</li></ul><p>For the rest of this post, we will work with a transformer model trained using a <strong>causal</strong> language modelling objective (similar to, but much smaller than GPT/GPT-2).</p><h3>Building a Transformer</h3><p>With that bit of background behind us, we can move on to some coding!</p><h4>Read Dataset</h4><p>Some basic preprocessing is done on the SST-5 dataset using Pandas. Note that the class labels are decremented by 1 (to be in the range [0, 1, 2, 3, 4]) because PyTorch expects the labels to be zero-indexed.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f984b4d246682a6bd824282449ebeef4/href">https://medium.com/media/f984b4d246682a6bd824282449ebeef4/href</a></iframe><h4>Tokenize and Encode Data</h4><p>The first step in processing the data is to perform tokenization using the WordPiece tokenizer— [<a href="https://arxiv.org/pdf/1609.08144.pdf">see section 4.1 in this paper</a> for more details]. We use the BertTokenizer <a href="https://huggingface.co/pytorch-transformers/model_doc/bert.html?highlight=berttokenize#pytorch_transformers.BertTokenizer">as implemented in HuggingFace’s </a><a href="https://huggingface.co/pytorch-transformers/model_doc/bert.html?highlight=berttokenize#pytorch_transformers.BertTokenizer">pytorch_transformers library</a>. Next, the tokenized text is encoded to sequences of integers to be processed by our transformer model. Following this, a <a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader">PyTorch </a><a href="https://pytorch.org/docs/stable/data.html#torch.utils.data.DataLoader">DataLoader</a> is created to load the samples onto batches for training.</p><p><a href="https://medium.com/swlh/transformer-fine-tuning-for-sentiment-analysis-c000da034bb5">In his Medium post</a> on transformer fine-tuning for sentiment analysis, Oliver Atanaszov wrote a very nice TextProcessor class that encapsulates the tokenization, encoding and data preparation steps (for PyTorch). This class made use of multiple CPU cores to speed up the encoding process, so it is adapted and reused here.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a4a84af3585143b4577f28caa05ee3a0/href">https://medium.com/media/a4a84af3585143b4577f28caa05ee3a0/href</a></iframe><p>Note that in this example, we fix the maximum length of the sequence to be 256 — in theory the <a href="https://github.com/google-research/bert/blob/master/README.md">maximum sequence length that can be handled by BERT</a> and similar model encoders is 512, but since SST-5 is a relatively small benchmark dataset with relatively short samples, we truncate the maximum token sequence length to 256 to reduce memory usage and model size.</p><p>A special classification token ‘[CLS]’ is added at the end of each sequence — this token is used in classification tasks as an aggregate of each sequence’s representation, to learn which class the sequence belongs to. For sequences that are shorter than 256, a padding token ‘[PAD]’ is added to ensure all batches remain the same size to be loaded into GPU memory during training.</p><h4>Build the Transformer Model</h4><p>The general structure of the stack of transformer blocks used for classification tasks is shown below. This is a modified form of the original version used for machine translation by Vaswani et al.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/358/1*Y6Lj-yARi32o4hpw3DZamQ.png" /><figcaption>Source: <a href="https://arxiv.org/pdf/1904.10509.pdf">Child et al.</a></figcaption></figure><p>In PyTorch code, the above structure looks as follows.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c55e23b237e7a2ead5a9bbb939ab7054/href">https://medium.com/media/c55e23b237e7a2ead5a9bbb939ab7054/href</a></iframe><p>A base Transformer class that inherits from PyTorch’s nn.module is defined. The input sequence (in our case, the text for sentiment classification) is fed to the transformer blocks by summing up the sequence’s token and position embeddings. Each successive transformer block is composed of the following modules:</p><ul><li><strong>Layer normalization</strong>: In essence, normalization is applied to mini-batches during training to make optimization easier (mathematically speaking) — this improves the performance of the model compared to an unnormalized network. Layer normalization allows the selection of arbitrarily-sized mini-batches by changing the dimension across which the batch statistics (mean/variance) are computed for each batch — this has been empirically shown to improve performance with sequential inputs in Recurrent Neural Networks (RNNs). For an excellent intuitive explanation of layer normalization, read <a href="https://mlexplained.com/2018/01/13/weight-normalization-and-layer-normalization-explained-normalization-in-deep-learning-part-2/">this blog post by Keita Kurita</a>.</li><li><strong>Self-attention</strong>: Uses PyTorch’s MultiHeadAttention module to encapsulate the transformer’s self-attention logic — that is, the ability of the transformer to encode positional information and learn from it during training.</li><li><strong>Dropout</strong>: Classic regularization technique in neural networks to reduce over-fitting — this is done by <a href="https://machinelearningmastery.com/dropout-for-regularizing-deep-neural-networks/">introducing random noise during the training process</a>.</li><li><strong>Feed-forward module</strong>: Performs the forward pass, including another layer normalization, with a hidden layer and a nonlinearity (typically <a href="https://machinelearningmastery.com/rectified-linear-activation-function-for-deep-learning-neural-networks/">ReLU</a> or <a href="https://datascience.stackexchange.com/questions/49522/what-is-gelu-activation">GELU</a>), followed by dropout.</li></ul><p>Two “masks” are defined in the feed-forward module of our network.</p><ul><li><strong>Padding mask</strong>: This masks the padding tokens ([‘PAD’]) introduced earlier to make each sequence the same length for each batch. Doing this tells the model to mask out these tokens during inference so that they are ignored by the self-attention module. The padding mask is specific to each batch.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*IFFKQqsyD6a_IWdECCz5zw.png" /><figcaption>Padding mask: <a href="https://docs.google.com/presentation/d/1fIhGikFPnb7G5kr58OvYC3GN4io7MznnM0aAgadvJfc/preview?pru=AAABaz2o8Jk*rdDLH7fXP7h4HQFLtzvHNQ#slide=id.g5888218f39_177_4">NAACL 2019 transferlearning tutorial slides</a></figcaption></figure><ul><li><strong>Attention mask</strong>: An attention mask is included since this method uses a causal language model — this masks the previous tokens in the input below as per the causal language model. The mask is specified the same way for all batches, using an upper-triangular matrix (<a href="https://pytorch.org/docs/stable/torch.html#torch.triu">torch.triu</a>) by setting the value of all elements above the main diagonal to be negative infinity.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/481/1*RucpX7srC6LwjS9g8ckB2Q.png" /><figcaption>Attention mask: <a href="https://docs.google.com/presentation/d/1fIhGikFPnb7G5kr58OvYC3GN4io7MznnM0aAgadvJfc/preview?pru=AAABaz2o8Jk*rdDLH7fXP7h4HQFLtzvHNQ#slide=id.g5888218f39_177_4">NAACL 2019 transferlearning tutorial slides</a></figcaption></figure><h4>Add a classification head</h4><p>The downstream portion of the model uses a linear classification layer on top of the existing transformer. The TransformerWithClfHead class inherits from the base Transformer class, and specifies CrossEntropyLoss as the loss function to optimize for. The size of the linear layer is [embedding_dimensions, num_classes] — in this case for the existing pretrained model and the SST-5 dataset, 410×5.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/79be0476df4e95ab27bbf73d21912520/href">https://medium.com/media/79be0476df4e95ab27bbf73d21912520/href</a></iframe><p>The raw outputs, i.e. the logits, are extracted from the classification layer and fed to a softmax function to generate a class probability vector (1×5) as output.</p><h4>Train the Model</h4><p>The model’s weights are initialized from the <a href="https://github.com/huggingface/naacl_transfer_learning_tutorial/blob/master/utils.py">pretrained model provided by HuggingFace,</a> and the training script [<a href="https://github.com/prrao87/fine-grained-sentiment/blob/master/training/train_transformer.py">training/train_transformer.py</a>] is run on the SST-5 dataset.</p><p>The following hyperparameters are used for training the model— note that the number of embedding dimensions, number of attention heads, etc. are not set explicitly — these are inherited from the pretrained model. After 3 training epochs the model is checkpointed and its config parameters are saved.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/351/1*reDlGObMNnKr7_rk9iqu_g.png" /></figure><p>💡 <strong>Linear warmup schedule</strong>: Rather than setting a constant learning rate, a <a href="https://pytorch.org/ignite/contrib/handlers.html#ignite.contrib.handlers.param_scheduler.PiecewiseLinear">piecewise linear scheduler from </a><a href="https://pytorch.org/ignite/contrib/handlers.html#ignite.contrib.handlers.param_scheduler.PiecewiseLinear">pytorch-ignite</a> is defined to ramp up the learning rate for the early portion of training, followed by ramping down linearly to zero. This is typically a good way to ensure good transfer of knowledge during transfer learning (similar to the “slanted-triangular learning rate” in <a href="https://arxiv.org/pdf/1801.06146.pdf">ULMFiT, Howard and Ruder, 2018</a>).</p><p>💡 <strong>Gradient accumulation</strong>: As Thomas Wolf so nicely describes in his post “<a href="https://medium.com/huggingface/training-larger-batches-practical-tips-on-1-gpu-multi-gpu-distributed-setups-ec88c3e51255"><em>Practical tips for training neural nets</em></a>”, a good way to simulate larger batch sizes without running into GPU memory issues is to accumulate gradients. This is done by summing the gradient tensors from multiple backpropagation steps and <em>then</em> calling the optimizer to minimize the loss — note that the loss would then also need to be divided by the number of accumulation steps. Doing this can allow us to train with batch sizes that are larger than what may actually fit in GPU memory, improving the model’s learning.</p><h4>Model Inference</h4><p>In this section, we’ll go line-by-line and dig into how a trained model can be used to make inferences on our SST-5 sentiment dataset, with an example. The input text is tokenized, converted to integer IDs and fed to the model as an appropriately shaped tensor, shown in the below notebook.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ed69acd235d3ab11b0f5e15585f86901/href">https://medium.com/media/ed69acd235d3ab11b0f5e15585f86901/href</a></iframe><p>This process is encapsulated using the TransformerSentiment class, in the file <a href="https://github.com/prrao87/fine-grained-sentiment/blob/master/classifiers.py">classifiers.py</a>, to read in the SST-5 dataset into a Pandas DataFrame and evaluate the transformer’s performance.</p><h3>Evaluation</h3><p>Running the trained transformer model (<a href="https://drive.google.com/open?id=1_6PoXN5THRB9Px-r7GwVx-8H7fNwNAIH">available in this Google drive link</a>) on the SST-5 test, we can see that the causal transformer improves the classification accuracy to nearly 50%!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/1*BtBgYuV7p4W8ZZA8dgS_og.png" /></figure><p>The macro-F1 scores for the transformer are also improved from the other embedding-based methods (FastText and Flair).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*8EI4dvnlUxeJWEYoyT54tw.png" /><figcaption>Each cell in the confusion matrix shows the <strong>percentage</strong> of predictions made for the corresponding true label.</figcaption></figure><p>The confusion matrix of the next-best model, Flair, is placed next to the transformer’s for comparison.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8bvFz3nUft6fYm-U62ebbw.png" /></figure><p>The transformer does make a lot of false predictions belonging to class labels 2 and 4 — however, it gets significantly more labels belonging to the minority classes (1 and 3) correct compared to all other methods. The fact that it is able to classify these minority classes reasonably well with such a limited number of training samples is testament to the power of using pretrained language models and transfer learning for sentiment classification tasks.</p><h3>Explaining the Transformer’s Predictions</h3><p>Following the model explanation method shown in <a href="https://towardsdatascience.com/fine-grained-sentiment-analysis-in-python-part-2-2a92fdc0160d">Part 2</a> of this series, we make LIME explainer visualizations of specific text examples from the SST-5 test set using our trained transformer model. The code used to generate the visualizations below is available in the file <a href="https://github.com/prrao87/fine-grained-sentiment/blob/master/explainer.py">explainer.py</a> — the trained model file is also made <a href="https://drive.google.com/open?id=1_6PoXN5THRB9Px-r7GwVx-8H7fNwNAIH">available in this Google drive link</a>.</p><p><strong>Example 1:</strong> “<em>It’s not horrible, just horribly mediocre.</em>”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/934/1*VQsXeS9XktkYL9HQLazIbw.png" /><figcaption><strong>True: 1 — Predicted: 1</strong></figcaption></figure><p>The transformer appears to correctly identify the words “<em>horrible</em>” and “<em>mediocre</em>” as the two most important features that contribute to this example having the class label 1 (i.e. strongly negative). The word “<em>not</em>” pushes the prediction probability away from 1 (presumably to label 2 or 3), but its effect is not as strong as the negative words in the sentence. The adverb “<em>horribly</em>” also plays a small role in classifying this sentence as strongly negative, meaning that the model has a notion of how modifiers such as adverbs alter the degree of sentiment in a sentence.</p><p><strong>Example 2: </strong>“<em>The cast is uniformly excellent … but the film itself is merely mildly charming.</em>”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/933/1*HBbiSypH-4n9qW_N_Eeiyg.png" /><figcaption><strong>True: 3 — Predicted: 3</strong></figcaption></figure><p>In this example, it is clear that the word “<em>but</em>” has the biggest weight in the transformer’s prediction of a class label 3 (neutral). This is interesting, because while there are a number of words that show varying degrees of polarity in this sentence (“<em>excellent</em>”, “<em>merely”</em>, “<em>mildly</em>” and “<em>charming</em>”), the word “<em>but</em>” acts as a critical modifier in this sentence — it marks the transition of the sentence from being strongly positive to being slightly negative in the latter half. Once again, the transformer seems to have a notion of how modifiers (a conjunction in this case) alter the overall degree of sentiment in the sentence.</p><p>Another interesting observation is that the word “<em>cast</em>” plays no role in the sentiment being predicted as neutral (class 3). <strong>All</strong> the previously used methods (<a href="https://towardsdatascience.com/fine-grained-sentiment-analysis-in-python-part-2-2a92fdc0160d">see Part 2</a>), including Flair, erroneously learned the feature importance of the word “<em>cast</em>” as contributing to sentiment — because this word occurred many times in the training data. The underlying power of the transformer’s causal language model helps it make a <em>more meaningful association between words and the predicted sentiment label</em>, regardless of how many times the word appeared in the training set.</p><h3>Concluding Remarks</h3><p>Through this series, we explored various NLP methods in Python for fine-grained classification on the Stanford Sentiment Treebank (SST-5) dataset. While this dataset is extremely challenging and poses numerous problems for existing text classification methods, it is clear that the current industry standard in NLP — transformers combined with transfer learning — show inherently superior performance on classification tasks compared to other methods. This is mainly due to the transformer’s underlying language model representation, which gives it more contextual awareness and a better syntactic understanding of the training vocabulary.</p><p>The below chart summarizes the sequential improvement (using more and more complex models) made in accuracy and F1 scores on the SST-5 dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b3e5OSL5xNVz0gD26G4Zug.png" /></figure><h4>Additional experiments</h4><p>To squeeze more performance out of the transformer, a few additional experiments (aside from increasing the number of parameters in the model) could be tried:</p><ul><li><strong>Domain-specific data</strong>: The current approach uses a causal language model (i.e. HuggingFace’s pretrained model) that has 50 million trainable parameters. This model was pretrained on Wikitext-103 (i.e. 103 million tokens from Wikipedia articles) as described in the <a href="https://docs.google.com/presentation/d/1fIhGikFPnb7G5kr58OvYC3GN4io7MznnM0aAgadvJfc/preview?pru=AAABaz2o8Jk*rdDLH7fXP7h4HQFLtzvHNQ#slide=id.g5888218f39_177_4">NAACL tutorial</a>. Augmenting the pretraining with more relevant domain-specific data, such as a few hundred thousand movie reviews could help the model gain a better understanding of the typical vocabulary found in SST-5. This step, though expensive — since it requires re-training the language model — is a one-time step (<a href="https://github.com/huggingface/naacl_transfer_learning_tutorial">which can be done using the </a><a href="https://github.com/huggingface/naacl_transfer_learning_tutorial">pretraining_train.py file in HuggingFace’s repository</a>) and can yield significant downstream improvement in classification accuracy.</li><li><strong>Masked Language Modelling</strong>: Using a causal language model as a pretraining objective, though cheaper to train, might not be the best way to achieve highly accurate classification results. Experimenting with masked language modelling as a pretraining objective (while augmenting the training data with movie reviews) could result in a more accurate language model. Indeed, the <a href="https://docs.google.com/presentation/d/1fIhGikFPnb7G5kr58OvYC3GN4io7MznnM0aAgadvJfc/preview?pru=AAABaz2o8Jk*rdDLH7fXP7h4HQFLtzvHNQ#slide=id.g5888218f39_177_4">NAACL tutorial</a> shows that pretraining using a masked language modelling objective produced a lower <a href="https://towardsdatascience.com/perplexity-intuition-and-derivation-105dd481c8f3"><em>perplexity</em></a> than the causal language model, meaning that this model could perform better on downstream tasks like sentiment classification.</li><li><strong>Hyperparameter tuning</strong>: Experimenting with larger batch sizes (for training and testing), increasing gradient accumulation steps and looking at different learning rate warmup schedules could yield additional performance gains on the test set.</li></ul><p>The purpose of building a sentiment analysis framework from the ground up, as described in this series, was to build confidence in our ability to <em>evaluate</em> and <em>interpret</em> different machine learning techniques, as relevant to our problem statement and available data. Making use of large pretrained transformer-based models such as BERT-large, XLNet or RoBERTa would, of course, yield significantly improved performance on real-world datasets — however, it is important to balance the immense computational cost of using these models versus using smaller, simpler models with a good pretraining objective and clean, <em>well-annotated</em>, <em>domain-specific</em> training data. As more and more NLP techniques expand on the huge success of transformers in the months to come, there can only be interesting times ahead!</p><h4>Code and Trained Model</h4><ul><li>Code to train and make predictions on SST-5 using the transformer model is in <a href="https://github.com/prrao87/fine-grained-sentiment">this project’s GitHub repo</a>.</li><li>The trained transformer model is <a href="https://drive.google.com/open?id=1_6PoXN5THRB9Px-r7GwVx-8H7fNwNAIH">available in this Google drive link</a>.</li></ul><p>Feel free to reproduce the results and make your own findings using the transformer!</p><h4>Acknowledgements</h4><p>All the code used for this post was adapted from the excellent example code in the below links:</p><ul><li><a href="https://github.com/huggingface/naacl_transfer_learning_tutorial">GitHub repository</a> for NAACL 2019 transfer learning NLP tutorial</li><li><a href="https://colab.research.google.com/drive/1iDHCYIrWswIKp-n-pOg69xLoZO09MEgf">Google Colab notebook</a> showing transformer model experiments</li><li><a href="https://medium.com/swlh/transformer-fine-tuning-for-sentiment-analysis-c000da034bb5">This Medium post by Oliver Atanaszov</a> on transformer fine-tuning</li></ul><h4>Further Reading</h4><p>For a <strong><em>much</em></strong> more detailed deep-dive into transfer learning in NLP, it is highly recommended to go through the <a href="https://docs.google.com/presentation/d/1fIhGikFPnb7G5kr58OvYC3GN4io7MznnM0aAgadvJfc/preview?pru=AAABaz2o8Jk*rdDLH7fXP7h4HQFLtzvHNQ#slide=id.g5888218f39_177_4">NAACL 2019 tutorial slides</a> by Sebastian Ruder, Matthew Peters, Swabha Swayamdipta and Thomas Wolf.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fdocs.google.com%2Fpresentation%2Fembed%3Fid%3D1fIhGikFPnb7G5kr58OvYC3GN4io7MznnM0aAgadvJfc%26size%3Dl&amp;url=https%3A%2F%2Fdocs.google.com%2Fpresentation%2Fd%2F1fIhGikFPnb7G5kr58OvYC3GN4io7MznnM0aAgadvJfc%2Fpreview%3Fpru%3DAAABaz2o8Jk%2ArdDLH7fXP7h4HQFLtzvHNQ&amp;image=https%3A%2F%2Flh5.googleusercontent.com%2F8eh4LndS6f3-TsJLC-5dodFogKHyMZcuskANq5RH6rfM1ZFvTu9qq_yA4Fo1RW47t7S4IA%3Dw1200-h630-p&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=google" width="700" height="559" frameborder="0" scrolling="no"><a href="https://medium.com/media/7a9a7dd4c4e246ce49b70ba7840c555b/href">https://medium.com/media/7a9a7dd4c4e246ce49b70ba7840c555b/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1ae6574f25a6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/fine-grained-sentiment-analysis-part-3-fine-tuning-transformers-1ae6574f25a6">Fine-grained Sentiment Analysis (Part 3): Fine-tuning Transformers</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fine-grained Sentiment Analysis in Python (Part 2)]]></title>
            <link>https://medium.com/data-science/fine-grained-sentiment-analysis-in-python-part-2-2a92fdc0160d?source=rss-efdb4f4a0d1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/2a92fdc0160d</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Prashanth Rao]]></dc:creator>
            <pubDate>Wed, 04 Sep 2019 23:53:56 GMT</pubDate>
            <atom:updated>2019-09-09T13:37:12.702Z</atom:updated>
            <content:encoded><![CDATA[<h4>In this post, we’ll <em>generate explanations for </em>various classification results for fine-grained sentiment using LIME</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HTczjsfQiliG-hxyIK-I8A.jpeg" /><figcaption>Source: <a href="https://pixabay.com/illustrations/question-mark-important-sign-1872665/">Pixabay</a></figcaption></figure><blockquote>“Why Should I Trust You?” — <a href="https://arxiv.org/abs/1602.04938">Ribeiro et al.</a></blockquote><p>This is Part 2 of a series on fine-grained sentiment analysis in Python. <a href="https://medium.com/@tech_optimist/fine-grained-sentiment-analysis-in-python-part-1-2697bb111ed4">Part 1</a> covered <em>how</em> to train and evaluate various fine-grained sentiment classifiers in Python. In this post, we’ll discuss <em>why</em> a classifier made a specific class prediction — that is, how to <strong>explain</strong> a sentiment classifier’s results using a popular method called LIME.</p><p>To recap, the following six models were used to make fine-grained sentiment class predictions on the Stanford Sentiment Treebank (SST-5) dataset.</p><ul><li>Rule-based models: <strong>TextBlob</strong> and <strong>VADER</strong></li><li>Feature-based models: <strong>Logistic regression</strong> and <strong>Support Vector Machine</strong></li><li>Embedding-based models: <strong>FastText</strong> and <strong>Flair</strong></li></ul><p>A linear process was used to analyze and explain the sentiment classification results using each method.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LzoocmmonSmsyCqwUDYcCQ.png" /><figcaption>Sentiment classification: Training &amp; Evaluation pipeline</figcaption></figure><p>The below sections explain the final step in the process — generating <strong>explanations</strong> for the predictions of each method.</p><h3>Local Interpretable Model-Agnostic Explanations</h3><p>Or, simply put, <strong>LIME</strong>,<strong> </strong>is a technique [<a href="https://arxiv.org/pdf/1602.04938.pdf">original paper</a>] that <em>explains</em> the prediction of a<em> </em>classifier in an interpretable manner. The term “explain” here means generating textual or visual aids, such as highlighted plots and charts, that provide a qualitative understanding of the relationship between a model’s features and its predictions. <a href="https://arxiv.org/pdf/1602.04938.pdf">From the paper</a>, the below key points make LIME very effective at interpreting a complex model’s classification results:</p><blockquote>It is <strong>model-agnostic </strong>— it treats the original model as a blackbox, so it can be applied on pretty much any classifier</blockquote><blockquote>It is <strong>locally faithful </strong>—<strong> </strong>it corresponds to how the model behaves in the vicinity of the instance being predicted. So for a given set of features in a test sample, the explanation is meaningful for the decision space occupied by those features, which may or may not not apply globally.</blockquote><h4>The model explanation process</h4><p>To provide an explanation for any classifier, LIME uses a very clever approach — the only thing it needs from the larger, blackbox classifier is its prediction probabilities for each class (from the model’s output layer after the softmax function is applied). The below steps explain the process in a nutshell.</p><ul><li>Generate thousands of variations of the text sample in which random words are blanked out, as follows:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/279/1*Vb-P82h2sIh7VqyK7LJXHA.png" /></figure><ul><li>Use the original, complex blackbox model to generate class probabilities and labels for each individual variation (whose words are partially blanked out). For example, the case “<em>It _ not horrible _ _ _ _</em>” would have the class probabilities [0, 0, 0.6, 0.3, 0.1] (i.e. it would belong to class 3), whereas “<em>It ‘s _ horrible _ _ _ _</em>” would have the probabilities [0.9, 0.1, 0, 0, 0]and belong to class 1.</li><li>Train a smaller, simpler, linear model on the labels predicted by the blackbox model for each variation.</li><li>Observe the features that carry the most weight in the simple, linear model (which is <strong>much</strong> easier to interpret)</li><li>Finally, generate a per-word (or token) visualization of the words based on feature importance</li></ul><h4>How does explanation work on multiple classes?</h4><p>For multi-class text classification such as this SST-5 dataset, LIME uses the prediction probabilities to highlight the effect of each feature (i.e. token) on the predicted class using a <a href="https://en.wikipedia.org/wiki/Multiclass_classification#One-vs.-rest">one-vs-rest method</a>. An example multi-class explanation from LIME is shown below. The model predicts a class of 2 for this sentence. The highlighted colours in the text being explained are randomly generated, with more intense colours signifying a greater feature importance for that token (in the below example, “dullness”) for the predicted class.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/941/1*5GW8LkEIZzfDespxVc3OEQ.png" /><figcaption>Prediction class probabilities and feature importance plots generated by LIME</figcaption></figure><h4>Explainer Class</h4><p>Just as in the previous post, an object-oriented approach is applied to reuse code where possible. All code for the explainer is provided in this <a href="https://github.com/prrao87/fine-grained-sentiment/blob/master/explainer.py">project’s GitHub repo [explainer.py</a>]. A Python class is defined which takes in the list of variations generated by LIME (random text samples with tokens blanked out), and outputs a class probability for each sample as a <em>numpy</em> array.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2b294e9c029c07cbf04e1cdb43d1deff/href">https://medium.com/media/2b294e9c029c07cbf04e1cdb43d1deff/href</a></iframe><p>Once the class probabilities for each variation is returned, this can be fed to the LimeTextExplainer class. Enabling bag-of-words (bow) would mean that LIME doesn’t consider word order when generating variations. However, the FastText and Flair models were trained considering n-grams and contextual ordering respectively, so for a fair comparison between models, the bow flag option is disabled for all explanations on SST-5.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e4de360ef4b6c55ca32c29c7f6974efa/href">https://medium.com/media/e4de360ef4b6c55ca32c29c7f6974efa/href</a></iframe><p>The exp object returned by the LIME explainer is an internal method that converts the local linear model’s predictions (in numerical form) to a visual form that can be interpreted by humans, which is output as an HTML file.</p><h4>Hack for the Rule-based Methods</h4><p>Since the rule-based methods (TextBlob and VADER) do not output class probabilities (they output just a single sentiment score), in order to explain their results using LIME, we have to artificially generate class probabilities for these methods. Although this isn’t a formal procedure, a simple workaround to simulate class probabilities using a continuous-valued sentiment score (in the range [-1, +1]) would be to normalize the float score to be within the range [0, 1], and then convert it to a discrete integer class by scaling it 5 times in magnitude. This is done for TextBlob and VADER as follows.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/558502078413e1c6eba86c5a0ce63780/href">https://medium.com/media/558502078413e1c6eba86c5a0ce63780/href</a></iframe><p>In addition, a rule-based model outputs one and only one prediction, so to avoid outputting zero probabilities for the other classes, a <a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html"><em>normal distribution </em></a>consisting of 5 points with the mean as the predicted integer class, is used to assign small non-zero probabilities to the remaining classes. The below plots show how this is done. This array of simulated probabilities can now be used by LIME for generating explanations on the rule-based model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZKkLdTe_fao3d-1QqoaMPA.png" /><figcaption>Example of simulated probabilities for rule-based classifier scores</figcaption></figure><h3>What does each classifier focus on?</h3><p>To understand the predictions, the file <a href="https://github.com/prrao87/fine-grained-sentiment/blob/master/explainer.py">explainer.py</a> is run for each of the six trained classifiers — this outputs an HTML file with visual content that helps us interpret the models’ feature understanding.</p><p>From the <a href="https://medium.com/@tech_optimist/fine-grained-sentiment-analysis-in-python-part-1-2697bb111ed4">EDA done in the previous post</a>, we know that classes <strong>1 and 3 are the minority classes</strong> in the SST-5 dataset, so two samples are chosen from these two classes.</p><blockquote>It’s not horrible, just horribly mediocre. <strong><em>(True label</em></strong><em> </em><strong><em>1)</em></strong></blockquote><blockquote>The cast is uniformly excellent … but the film itself is merely mildly charming. <strong><em>(True label 3)</em></strong></blockquote><p>Each of these samples contains modifiers, conflicting vocabulary and rapidly varying polarities across the sentence, so they should in principle, help expose what each classifier is focusing on.</p><h4><strong>TextBlob</strong></h4><p><strong>Example 1:</strong> “<em>It’s not horrible, just horribly mediocre.</em>”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*xloQdwNeNP3_57XVc1f1Hg.png" /><figcaption><strong>True: 1 — Predicted: 3</strong></figcaption></figure><p>For the sentence “<em>It’s not horrible, just horribly mediocre</em>”, the use of a negation term before the word “<em>horrible</em>” convinces TextBlob that the item isn’t horrible — however, the second clause of the sentence “<em>just horribly mediocre</em>” reaffirms the fact that the item is indeed, very mediocre. The sentence as a whole is still overwhelmingly negative, making the true label 1, but TextBlob focuses heavily on the negation term to push the sentiment rating upward to 3.</p><p><strong>Example 2: </strong>“<em>The cast is uniformly excellent … but the film itself is merely mildly charming.</em>”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*tMJEcL0wqW7_APBdYo62Sg.png" /><figcaption><strong>True: 3 — Predicted: 5</strong></figcaption></figure><p>The above sentence is really challenging, because the first half is overwhelmingly positive, while the second half is quite negative — making the overall sentence neutral in sentiment. The words “<em>mildly</em>”, “<em>charming</em>” and “<em>excellent</em>” contribute heavily to TextBlob incorrectly classifying this sentence as strongly positive (label 5). In general, multiple occurrences of strongly positive words tend to push the rule-based algorithm in TextBlob to classify the sentence as positive overall.</p><h4>VADER</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*7kzMtwHgJmyxMDQWAMKofg.png" /><figcaption><strong>True: 1 — Predicted: 5</strong></figcaption></figure><p>In the above example, VADER predicts a completely opposite sentiment rating to what was expected. VADER tends to heavily penalize negative polarity words (such as “<em>horrible</em>”), but also heavily reward negation terms that precede negative polarity words (“<strong><em>not</em></strong> <em>horrible</em>”). The presence of “<em>horribly mediocre</em>” in the latter half of the sentence has no impact whatsoever because the strong positive score in the first half outweighs everything that comes after — the rule-based approach in VADER proved a little too clever for itself in this example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/931/1*7lfpYTR2ReGG7aYuWwwPIw.png" /><figcaption><strong>True: 3 — Predicted: 5</strong></figcaption></figure><p>Once again, the rule-based approach in VADER heavily weights positive-sounding words like “<em>excellent</em>” and “<em>charming</em>” to incorrectly give this sentence a label 5. The words “<em>merely</em>” and “<em>mildly</em>”, which should be bringing down the overall sentence score, are ignored by VADER— this is most likely because these words do not appear in VADER’s sentiment lexicon or rule-based modifiers.</p><h4>Logistic Regression</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/931/1*jR2GVztyIGeMs0x9pTMJXg.png" /><figcaption><strong>True: 1 — Predicted: 2</strong></figcaption></figure><p>Logistic regression does seem to learn individual features (tokens) that modify the overall sentiment rating of the sentence — as can be seen above, the words “<em>just</em>” and “<em>mediocre</em>” contribute more towards label 2, whereas “<em>horrible</em>” and “<em>horribly</em>” contribute to “not 2” (presumably label 1). The prediction probabilities for both labels 1 and 2 are nearly equal, so the feature-based approach in this case errs by a very fine margin towards label 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*hIQftW6AOhnGMS2fu0QEeQ.png" /><figcaption><strong>True: 3 — Predicted: 3</strong></figcaption></figure><p>In the second example, it is clear that the words “<em>but”</em>, “<em>mildly</em>” and “<em>merely</em>” are correctly identified by the logistic regression model as swinging the overall sentence sentiment towards a neutral state.</p><h4>Support Vector Machine</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*UKVWOP5yo9MZsQDmVdrqQg.png" /><figcaption><strong>True: 1 — Predicted: 1</strong></figcaption></figure><p>The SVM, unlike the logistic regression, focuses more heavily on the strongly negative words “<em>horrible</em>” and “<em>horribly</em>” to assign this a label of 1. The negation term “<em>not</em>” and the word “<em>mediocre</em>” are identified correctly as pushing the rating away from 1— but their effect is low and the overall sentiment label is still 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/933/1*LRzrE2zTE9H75PmVv9gaQg.png" /><figcaption><strong>True: 3 — Predicted: 3</strong></figcaption></figure><p>In the second example, it is clear that the SVM learned the right features that make the overall sentence neutral. The words “<em>merely</em>”, “<em>mildly</em>” and “<em>but</em>” decrease the probability that the sentence is assigned labels 2, 4 and 5, making the label 3 the most likely.</p><h4>FastText</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PbN5eJKaz96XTgI9aD2XKA.png" /><figcaption><strong>True: 1 — Predicted: 1</strong></figcaption></figure><p>FastText focuses heavily on the strongly negative words in example 1 — the other class labels have a zero probability. It does get the overall prediction right, but this could just have been because FastText learned that “<em>horrible</em>” is a word that regularly appears in strongly negative sentences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lARGiFrEkiplETwIq1a38g.png" /><figcaption><strong>True: 3 — Predicted: 3</strong></figcaption></figure><p>In example 2, it is more clear why the FastText model correctly predicts the neutral label. Since the model was trained using word-trigrams and a context window of 5 (see the previous post for training parameters) — the model looks at <strong>sequences</strong> of tokens while making a prediction. For example, the words “<em>uniformly excellent</em>” are immediately followed by “<em>but</em>”, and the words “<em>merely mildly</em>” immediately precede “<em>charming</em>”, so looking at the highlighted words in the visualization, the model is aware of the sequences of word co-occurrences (and not individual words) that make this sentence neutral.</p><h4>Flair + ELMo</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/931/1*mTbmVm4JS_ot4N-SoCim3g.png" /><figcaption><strong>True: 1 — Predicted: 2</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/931/1*MDkxwRbY8z5qysscroGuMQ.png" /><figcaption><strong>True: 3 — Predicted: 2</strong></figcaption></figure><p>The Flair + ELMo embedding model gets both example predictions wrong by a fine margin. In both the above visualizations, the model seems to give a high weight to the period (.) token at the end of each sentence— this wasn’t the case with the other classifiers. The probabilities for the predicted label are very close to the correct label in either case, so the model seems to be on the right track in terms of how it learns from the data.</p><p>An important point to note: <strong>The Flair + ELMo model was underfitting </strong>during the training stage, i.e. the validation loss was still decreasing even after 25 epochs of training (see Part 1 of this series) — this means that further training could push the classifier towards the correct probability outputs for these and other examples.</p><h3>Analysis</h3><p>Studying the explained results across the various methods, we can make some observations on the merits and demerits, as well as the effects of key variables on each method’s performance.</p><h4>Effect of strongly polar words</h4><p>TextBlob and VADER tend to heavily weight words with strong polarity, even when there are other words with a milder polarity (or negation terms) that alter the sentiment of the overall sentence. Hardcoded rules work well in many cases, but natural language from the real world has too much variability for these kinds of rules to work well in practice, at least for fine-grained sentiment analysis.</p><p>Embedding-based methods have the best handle on cases that involve strongly polar words. The FastText model was trained with trigrams, so it learned to pick up sequences of words that preceded or succeeded a strongly polar word — so it wasn’t as easily fooled, allowing it to make better predictions on the neutral class compared to the rule-based and feature-based methods. The Flair model, because of its contextual embeddings and strong underlying language model, was able to more accurately identify patterns involving sequences of strong polarity words.</p><h4>Effect of sentence length</h4><p>Long samples, especially multi-sentence samples can cause trouble when using rule-based methods, which tend to apply some kind of weighted averaging to capture overall sentence polarity. Hence, the longer the sentence, the greater the chance of diffusing the actual sentiment of individual clauses in the long sample.</p><p>Really short (single or two-word) samples pose a different challenge to the models — they either contain unseen words, or provide too little context such as punctuation or similar subwords that a model could have used to categorize the word. In general, rule-based models would fare badly in these situations, because the lack of modifiers (“<em>very</em>” or “<em>too</em>”) do not provide enough of a notion of sentiment intensity in our fine-grained classification scenario.</p><h4>Effect of unseen words</h4><p>In samples where there numerous unseen words — or really short samples, where the chances that a word was unseen during training are quite high — models that lack a sequential (n-gram) or contextual representation (embeddings) tend to struggle to make reliable fine-grained sentiment predictions. The feature-based models in scikit-learn showcase this issue quite clearly. Since they rely on word co-occurrence counts during training, if an unseen word shows up in the test set, the word is ignored by the feature-based classifier. This misses out on features that might have given the model more context.</p><p>FastText is better equipped to deal with unseen words due to its character n-gram subword representations, but it doesn’t have as deep a pre-trained representation as the Flair model does, to properly deal with unseen words. The Flair + ELMo embedding model uses contextual representations from a pretrained vocabulary (<a href="https://ai.google/research/pubs/pub41880">1 billion words of news crawl data</a>). This gives it a significant advantage over the shallower models — the contextualized embeddings carry useful information that identify words similar in meaning as well as their subword representations.</p><h4>Model cost (training and/or inference)</h4><p>All rule-based methods involve zero-training time and are very quick during the prediction stage — however this comes at the cost of lack of stability in results with unseen, real-world data. The feature-based methods are also very quick to train and make inferences with, due to highly efficient vectorized representations in scikit-learn. However, these models also come with limitations in terms of not capturing relationships between words and poor handling of unseen words.</p><p>FastText is a good compromise between cost of computation (it is blazing fast due to its underlying C++ and Cython bindings) and classification accuracy. For fine-grained sentiment analysis, training the model using trigram representations is a must, in order to capture the finer gradations involved in sequences of words that occur in long sentences.</p><p>Flair is the most expensive option of all, mainly due to the fact that it is a large deep learning model that uses pre-trained representations from a combination of string and word embeddings. Training this model can take of the order of several hours (or days, depending on the size of the dataset). A part of the reason that the Flair + ELMo model is slow during training and inference is the way that <a href="https://github.com/zalandoresearch/flair/issues/915#issuecomment-513152226">ELMo embeddings are looked up during run time</a> — this can be sped up using mini-batching — however, the reality is that models like Flair might well prove to be too expensive on real-world datasets for fast and efficient inference.</p><h3>Interactive Dashboard for Model Testing</h3><p>To make it easier to see explanations for a range of models on a number of difference examples, a dashboard was created using the Flask micro-framework. The dashboard was deployed using Heroku (<a href="https://medium.com/the-andela-way/deploying-a-python-flask-app-to-heroku-41250bda27d0">in a similar way to this example</a>). To test the dashboard interactively, input a text sample and choose the classifier whose classification prediction we want explained, as shown below.</p><p><a href="https://sst5-explainer.herokuapp.com/">Try out the dashboard here</a> and make your own explanations!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/947/1*0C5oXx9xyY7ASuyAL0CGIw.gif" /><figcaption>Try out the dashboard! <a href="https://sst5-explainer.herokuapp.com/">https://sst5-explainer.herokuapp.com</a></figcaption></figure><h3>Conclusions</h3><p>In this post, we discussed how to generate and interpret LIME explanations for fine-grained sentiment using six different classifiers in Python. The differences in the features focused on by each model are clearly apparent, making it a more straightforward task to choose the right classifier for the task at hand. The more complex the underlying representation of the vocabulary (especially contextual embeddings), the more reliable the model’s predictions across the five sentiment classes for the SST-5 dataset.</p><p>The below accuracy/F1-scores on the SST-5 dataset were obtained thus far:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/457/1*C5TQJLmPJKi542EmQqHUng.png" /></figure><p>While the Flair + ELMo embedding-based model does achieve a decent accuracy of 48.9%, this is still very far from the state-of-the-art (64.4%). In addition, the Flair model is <strong>very</strong> expensive to train and make predictions with, so a more computationally efficient, yet contextually aware model is desirable.</p><p>In <a href="https://medium.com/@tech_optimist/fine-grained-sentiment-analysis-part-3-fine-tuning-transformers-1ae6574f25a6"><strong>Part 3</strong></a> of this series, we’ll see how to improve on these results further using a transformer model with transfer learning. Thanks for reading!</p><h4>Acknowledgements</h4><ul><li>The code for generating LIME explanations borrows from <a href="https://medium.com/@ageitgey/natural-language-processing-is-fun-part-3-explaining-model-predictions-486d8616813c">this excellent post by Adam Geitgey</a>. For an in-depth yet easily understandable explanation on how LIME works, give it a read!</li><li>The Python/Flask dashboard was a quick-and-dirty implementation, whose code was inspired by <a href="https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-ii-templates">this excellent Flask tutorial</a>.</li></ul><h4>Code</h4><ul><li>All code for making sentiment predictions and generating LIME explanations is in <a href="https://github.com/prrao87/fine-grained-sentiment">this project’s GitHub repo</a>.</li><li>All code for the Flask front-end app is available in <a href="https://github.com/prrao87/fine-grained-sentiment-app">this separate repo</a>.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2a92fdc0160d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/fine-grained-sentiment-analysis-in-python-part-2-2a92fdc0160d">Fine-grained Sentiment Analysis in Python (Part 2)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fine-grained Sentiment Analysis in Python (Part 1)]]></title>
            <link>https://medium.com/data-science/fine-grained-sentiment-analysis-in-python-part-1-2697bb111ed4?source=rss-efdb4f4a0d1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/2697bb111ed4</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Prashanth Rao]]></dc:creator>
            <pubDate>Wed, 04 Sep 2019 23:53:07 GMT</pubDate>
            <atom:updated>2019-09-09T13:36:03.208Z</atom:updated>
            <content:encoded><![CDATA[<h4>In this post, we’ll evaluate and compare the results of several text classification results for the 5-class Stanford Sentiment Treebank (SST-5) dataset.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0hBSEJAdDzieBIpWwvryUQ.jpeg" /><figcaption>Source: <a href="https://pixabay.com/photos/smiley-emoticon-anger-angry-2979107/">Pixabay</a></figcaption></figure><blockquote>“Learning to choose is hard. Learning to choose well is harder. And learning to choose well in a world of unlimited possibilities is harder still, perhaps too hard.” — Barry Schwartz</blockquote><p>When starting a new NLP sentiment analysis project, it can be quite an overwhelming task to narrow down on a select methodology for a given application. Do we use a rule-based model, or do we train a model on our own data? Should we train a neural network, or will a simple linear model meet our requirements? Should we spend the time and effort in implementing our own text classification framework, or can we just use one off-the-shelf? How hard<em> </em>is it to interpret the results and understand <em>why</em> certain predictions were made?</p><p>This series aims at answering some of the above questions, with a focus on fine-grained sentiment analysis. Through the remaining sections, we’ll compare and discuss classification results using several well-known NLP libraries in Python. The methods described below fall under three broad categories:</p><p><strong>Rule-based</strong> methods:</p><ul><li><a href="https://textblob.readthedocs.io/en/dev/">TextBlob</a>: Simple rule-based API for sentiment analysis</li><li><a href="https://github.com/cjhutto/vaderSentiment">VADER</a>: Parsimonious rule-based model for sentiment analysis of social media text.</li></ul><p><strong>Feature-based</strong> methods:</p><ul><li><a href="https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html">Logistic Regression</a>: Generalized linear model in Scikit-learn.</li><li><a href="https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html#sklearn.linear_model.SGDClassifier">Support Vector Machine (SVM)</a>: Linear model in Scikit-learn with a stochastic gradient descent (SGD) optimizer for gradient loss.</li></ul><p><strong>Embedding-</strong>based methods:</p><ul><li><a href="https://fasttext.cc/docs/en/supervised-tutorial.html">FastText</a>: An NLP library that uses highly efficient CPU-based representations of word embeddings for classification tasks.</li><li><a href="https://github.com/zalandoresearch/flair">Flair</a>: A PyTorch-based framework for NLP tasks such as sequence tagging and classification.</li></ul><p>Each approach is implemented in an <a href="https://python.swaroopch.com/oop.html"><em>object-oriented</em></a> manner in Python, to ensure that we can easily swap out models for experiments and extend the framework with better, more powerful classifiers in the future.</p><h3>Why Fine-grained Sentiment?</h3><p>In most cases today, sentiment classifiers are used for binary classification (just positive or negative sentiment), and for good reason: fine-grained sentiment classification is a <strong><em>significantly</em></strong><em> </em><strong><em>more challenging</em></strong> task! The typical breakdown of fine-grained sentiment uses<em> five</em> discrete classes, as shown below. As one might imagine, models very easily err on either side of the strong/weak sentiment intensities thanks to the wonderful subtleties of human language.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/705/1*ug8kyqUlnqEuo3LhHfNypg.png" /><figcaption>Typical class labels (or intensities) for fine-grained sentiment classification</figcaption></figure><p>Binary class labels may be sufficient for studying large-scale positive/negative sentiment trends in text data such as Tweets, product reviews or customer feedback, but they do have their limitations. When performing information extraction with comparative expressions, for example: “<em>This</em> <em>OnePlus model X is so much better than Samsung model X.” — </em>a fine-grained analysis can provide more precise results to an automated system that prioritizes addressing customer complaints. In addition, dual-polarity sentences such as “<em>The location was truly disgusting ... but the people there were glorious.</em>” can confuse binary sentiment classifiers, leading to incorrect class predictions.</p><p>The above points provide sufficient motivation to tackle this problem!</p><h4>Stanford Sentiment Treebank</h4><p>The <a href="https://nlp.stanford.edu/sentiment/">Stanford Sentiment Treebank</a> (SST-5, or SST-fine-grained) dataset is a suitable benchmark to test our application, since it was designed to help evaluate a model’s ability to understand representations of <em>sentence structure</em>, rather than just looking at individual words in isolation. SST-5 consists of 11,855 sentences extracted from movie reviews with fine-grained sentiment labels [1–5], as well as 215,154 phrases that compose each sentence in the dataset.</p><p>The raw data with phrase-based fine-grained sentiment labels is in the form of a tree structure, designed to help train a <a href="https://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf"><em>Recursive Neural Tensor Network</em></a> (RNTN) from their 2015 paper. The component phrases were constructed by parsing each sentence using the Stanford parser (section 3 in <a href="https://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf">the paper</a>) and creating a recursive tree structure as shown in the below image. A deep neural network was then trained on the tree structure of each sentence to classify the sentiment of each phrase to obtain a cumulative sentiment of the entire sentence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/714/1*sh9P4hY6oR0mFbzWkUFtNA.png" /><figcaption>Example of Recursive Neural Tensor Network classifying fine-grained sentiment (Source: <a href="https://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf">Original paper</a>)</figcaption></figure><h4>What is the state-of-the-art?</h4><p>The original RNTN implemented in the <a href="https://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf">Stanford paper</a> [Socher et al.] obtained an accuracy of <strong>45.7% </strong>on the full-sentence sentiment classification. More recently, a Bi-attentive Classification Network (BCN) augmented with <a href="https://arxiv.org/pdf/1802.05365v2.pdf">ELMo embeddings</a> has been used to <a href="https://github.com/sebastianruder/NLP-progress/blob/master/english/sentiment_analysis.md#sst">achieve a significantly higher accuracy</a> of <strong>54.7%</strong> on the SST-5 dataset. The current (as of 2019) state-of-the-art accuracy on the SST-5 dataset is <strong>64.4%</strong>, by <a href="https://arxiv.org/pdf/1806.00807.pdf">a method that uses sentence-level embeddings</a> originally designed to solve a paraphrasing task — it ended up doing surprisingly well on fine-grained sentiment analysis as well.</p><p>Although neural language models have been getting increasingly powerful since 2018, it might take far bigger deep learning models (with far more parameters) augmented with knowledge-based methods (such as graphs) to achieve sufficient semantic context for accuracies of 70–80% in fine-grained sentiment analysis.</p><h4>Transform the Dataset to Tabular Form</h4><p>To evaluate our NLP methods and how each one differs from the other, we will use just the <strong><em>complete samples</em></strong> in the training dataset (ignoring the component phrases since we are not using a recursive tree-based classifier like the Stanford paper). The tree structure of phrases is converted to raw text and its associated class label using the <a href="https://pypi.org/project/pytreebank/"><em>pytreebank</em></a> library. The code for this tree-to-tabular transformation is provided in <a href="https://github.com/prrao87/fine-grained-sentiment/blob/master/data/sst/tree2tabular.py">this project’s GitHub repo</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/36fad8823749d459c072580d4904a03e/href">https://medium.com/media/36fad8823749d459c072580d4904a03e/href</a></iframe><p>The full-sentence text and their class labels (for the train, dev and test sets) are written to individual text files using a tab-delimiter between the sentence and class labels.</p><h4>Exploratory Data Analysis</h4><p>We can then explore the tabular dataset in more detail using Pandas. To begin, read in the training set as a DataFrame while specifying the tab-delimiter to distinguish the class label from the text. Note that the class labels in the column “truth” are cast to the data type category in Pandas rather than leaving it as a string.</p><pre>import pandas as pd</pre><pre># Read train data<br>df = pd.read_csv(&#39;../data/sst/sst_train.txt&#39;, sep=&#39;\t&#39;, header=None, names=[&#39;truth&#39;, &#39;text&#39;])</pre><pre>df[&#39;truth&#39;] = df[&#39;truth&#39;].str.replace(&#39;__label__&#39;, &#39;&#39;)<br>df[&#39;truth&#39;] = df[&#39;truth&#39;].astype(int).astype(&#39;category&#39;)<br>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KVb5P0Bt90GeSLezex4pUg.png" /><figcaption>Sample of SST-5 training data</figcaption></figure><p>Using the command df.shape[0] tells us we have 8,544 training samples.</p><h4>Is the dataset balanced?</h4><p>One important aspect to note before analyzing a sentiment classification dataset is the class distribution in the training data.</p><pre>import matplotlib.pyplot as plt</pre><pre>ax = df[‘truth’].value_counts(sort=False).plot(kind=’barh’)<br>ax.set_xlabel(“Number of Samples in training Set”)<br>ax.set_ylabel(“Label”)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/1*rWJM5qc5LkczVqgp_qC0yw.png" /></figure><p>It is clear that most of the training samples belong to classes 2 and 4 (the weakly negative/positive classes). A sizeable number of samples belong to the neutral class. Barely 12% of the samples are from the strongly negative class 1, which is something to keep in mind as we evaluate our classifier accuracy.</p><p>What about the test set? A quick look tells us that we have 2,210 test samples, with a very similar distribution to the training data — again, there are far fewer samples belonging to the strongly negative/positive classes (1 or 5) compared to the other classes. This is desirable, since the test set distribution on which our classifier makes predictions is not too different from that of the training set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/378/1*-EVYoK9CLt3WOhdabWsiSQ.png" /></figure><p>An interesting point mentioned in the <a href="https://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf">original paper</a> is that many of the really short text examples belong to the neutral class (i.e. class 3). This can be easily visualized in Pandas. We can create a new column that stores the string length of each text sample, and then sort the DataFrame rows in ascending order of their text lengths.</p><pre>df[&#39;len&#39;] = df[&#39;text&#39;].str.len()  # Store string length of each sample</pre><pre>df = df.sort_values([&#39;len&#39;], ascending=True)<br>df.head(20)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*jVoanA6hTQ_MzqY16uv2_A.png" /><figcaption>Class labels for the really short examples in the test set</figcaption></figure><p>Samples with clearly polar words, such as “good” and “loved” would offer greater context to a sentiment classifier— however, for neutral sounding words (such as “Hopkins”, or “Brimful”), the classifier would have to not only work with extremely small context, i.e. single word samples, but also be able to deal with ambiguous or unseen words that did not appear in the training vocabulary.</p><h4>The data labels aren’t perfect!</h4><p>As mentioned in the paper, the SST dataset was labelled by human annotators via Amazon Mechanical Turk. Annotators were shown randomly selected phrases for which they chose labels from a <em>continuous</em> slider bar. A <em>discrete</em> sentiment label belonging to one of five classes was reconstructed based on an average of multiple annotators’ chosen labels. Random sampling was used during annotation to ensure that labelling wasn’t influenced by the phrase that preceded it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/587/1*hTNU7OXbm39XWaud91Jjwg.png" /><figcaption>Labelling interface for SST dataset (source: <a href="https://nlp.stanford.edu/~socherr/EMNLP2013_RNTN.pdf">Original Paper</a>)</figcaption></figure><p>The above example makes it clear why this is such a challenging dataset on which to make sentiment predictions. For example, annotators tended to categorize the phrase “<em>nerdy folks</em>” as somewhat negative, since the word “nerdy” has a somewhat negative connotation in terms of our society’s current perception of nerds. However, from a <em>purely linguistic perspective</em>, this sample could just as well be classified as neutral.</p><p>It is thus important to remember that <strong>text classification labels are always subject to human perceptions and biases. </strong>In a real-world application, it absolutely makes sense to look at certain edge cases on a subjective basis. No benchmark dataset — and by extension, classification model — is ever perfect.</p><p>With these points in mind, we can proceed onward to designing our sentiment classification framework!</p><h3>Methodology</h3><p>A general workflow for model training and evaluation is shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LzoocmmonSmsyCqwUDYcCQ.png" /><figcaption>Sentiment classification: Training &amp; Evaluation pipeline</figcaption></figure><p><strong>Model Training: </strong>Each classifier (except for the rule-based ones) is trained on the 8,544 samples from the SST-5 training set using a supervised learning algorithm. Separate training scripts are <a href="https://github.com/prrao87/fine-grained-sentiment/tree/master/training">available in the project’s GitHub repo</a>.</p><p><strong>Prediction: </strong>As per our object-oriented design philosophy, we avoid repeating code blocks that perform the same tasks across the various classification methods. A Base class is defined in Python that contains the commonly used methods: one for reading in the SST-5 data into a Pandas DataFrame (read_data), and another to calculate the model’s classification accuracy and F1-score (accuracy). Storing the dataset in a Pandas DataFrame this way makes it very convenient to apply custom transformations and user-defined functions while avoiding excessive use of for-loops.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b484c624484f4b079cb3e6774f49c6f8/href">https://medium.com/media/b484c624484f4b079cb3e6774f49c6f8/href</a></iframe><p>Next, each individual classifier added to our framework must inherit the Base class defined above. To make the framework consistent, a score method and a predict method are included with each new sentiment classifier, as shown below. The score method outputs a unique sentiment class for a text sample, and the predict method applies the score method to every sample in the test dataset to output a new column, &#39;pred&#39; in the test DataFrame. It is then trivial to compute the model’s accuracy and F1-scores by using the accuracy method defined in the Base class.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7989d68bf094f1d89f7948733bee1abc/href">https://medium.com/media/7989d68bf094f1d89f7948733bee1abc/href</a></iframe><p><strong>Evaluation: </strong>To evaluate the model’s accuracy, a <a href="https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html"><strong>confusion matrix</strong></a> of the model is plotted using scikit-learn and matplotlib (<a href="https://github.com/prrao87/fine-grained-sentiment/blob/master/plotter.py"><em>plotter.py</em></a> on GitHub). The confusion matrix tabulates the number of correct predictions versus the number of incorrect predictions for each class, so it becomes easier to see which classes are the least accurately predicted for a given classifier. Note that the confusion matrix for our 5-class case is a normalized <a href="https://en.wikipedia.org/wiki/Anti-diagonal_matrix">anti-diagonal matrix</a> — ideally, the classifier would get almost 100% of its predictions correct so all elements outside the anti-diagonal would be as close to zero as possible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/273/1*8-r2ydfDNrXzmC-5fxXSPQ.png" /><figcaption>Idealized confusion matrix (normalized) — termed an “<a href="https://en.wikipedia.org/wiki/Anti-diagonal_matrix">anti-diagonal matrix</a>”</figcaption></figure><h3>Training and Model Evaluation</h3><p>In this section, we’ll go through some key points regarding the training, sentiment scoring and model evaluation for each method.</p><h4>1 — TextBlob</h4><p><a href="https://textblob.readthedocs.io/en/dev/">TextBlob</a> is a popular Python library for processing textual data. It is built on top of NLTK<a href="http://www.nltk.org/">,</a> another popular Natural Language Processing toolbox for Python. TextBlob uses a sentiment lexicon (consisting of predefined words) to assign scores for each word, which are then averaged out using a weighted average to give an overall sentence sentiment score. Three scores: “polarity”, “subjectivity” and “intensity” are calculated for each word.</p><pre># A sentiment lexicon can be used to discern objective facts from subjective opinions in text. <br># Each word in the lexicon has scores for: <br># 1) polarity: negative vs. positive (-1.0 =&gt; +1.0) <br># 2) subjectivity: objective vs. subjective (+0.0 =&gt; +1.0) <br># 3) intensity: modifies next word? (x0.5 =&gt; x2.0)</pre><p>Some intuitive rules are hardcoded inside TextBlob to detect modifiers (such as adverbs in English: “<em>very good</em>”) that increase or decrease the overall polarity score of the sentence. A more detailed description of these rules is available <a href="https://planspace.org/20150607-textblob_sentiment/">in this blog post</a>.</p><p><strong>Sentiment Scoring:</strong> To convert the polarity score returned by TextBlob (a continuous-valued float in the range [-1, 1]) to a fine-grained class label (an integer), we can make use of <em>binning</em>. This is easily done in Pandas using the pd.cut function — it allows us to go from a continuous variable to a categorical variable by using equal sized bins in the float interval of all TextBlob scores in the results.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0cb364b0b993f53a1e230b84bfb523a6/href">https://medium.com/media/0cb364b0b993f53a1e230b84bfb523a6/href</a></iframe><p><strong>Evaluation:</strong> Since we are dealing with imbalanced classes during both training and testing, we look at the <em>macro</em> F1 score (which is the harmonic mean of the macro-averaged precision and recall) as well as classification accuracy. As can be seen , the accuracy of the TextBlob classification method is very low, as is the F1 score.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/385/1*DoSOUl6odo7DrAacN7jFJw.png" /></figure><p>The confusion matrix plot shows more detail about which classes were most incorrectly predicted by the classifier.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*K4YT44_Fimj-PnausoVt6g.png" /><figcaption>Each cell in the confusion matrix shows the <strong>percentage</strong> of predictions made for the corresponding true label.</figcaption></figure><p>To read the above confusion matrix plot, look at the cells along the <a href="https://en.wikipedia.org/wiki/Anti-diagonal_matrix">anti-diagonal</a>. Cell [1, 1] shows the percentage of samples belonging to class 1 that the classifier predicted correctly, cell [2, 2] for correct class 2 predictions, and so on. Cells away from the anti-diagonal show the percentage of wrong predictions made for each respective class — for example, looking at the cell [4, 5], we can see that 47% of all samples that actually belong to class 5 are (incorrectly) predicted as class 4 by TextBlob.</p><p>It is clear that our TextBlob classifier predicts most samples as neutral or mildly positive, i.e. of class 3 or 4, which explains why the model accuracy is so low. Very few predictions are strongly negative or positive — this makes sense because TextBlob uses a weighted average sentiment score over all the words in each sample. This can very easily diffuse out the effect of sentences with widely varying polarities between words, such as “<em>This movie is about lying , cheating , but loving the friends you betray</em>.”</p><h4>2 — VADER</h4><p>“<strong>V</strong>alence <strong>A</strong>ware <strong>D</strong>ictionary and s<strong>E</strong>ntiment <strong>R</strong>easoner” is another popular rule-based library for sentiment analysis. Like TextBlob, it uses a sentiment lexicon that contains intensity measures for each word based on human-annotated labels. A key difference however, is that VADER <strong>was designed with a focus on social media texts</strong>. This means that it puts a lot of emphasis on rules that capture the essence of text typically seen on social media — for example, short sentences with emojis, repetitive vocabulary and copious use of punctuation (such as exclamation marks). Below are some examples of the sentiment intensity scores output by VADER.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*thIrVcJmapb5SPLEWn0wCA.png" /></figure><p>In the above text samples, minor variations are made to the same sentence. Note that VADER breaks down sentiment intensity scores into a positive, negative and neutral component, which are then normalized and squashed to be within the range [-1, 1] as a “compound” score. As we add more exclamation marks, capitalization and emojis/emoticons, the intensity gets more and more extreme (towards +/- 1).</p><p><strong>Sentiment scoring:</strong> For returning discrete class values on the SST-5 dataset, we apply a similar technique as done for TextBlob — the continuous “compound” polarity score (float) is converted to a discrete value using binning through the pandas pd.cut function. This returns one of five classes for each test sample, stored as a new column in the resulting DataFrame.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/617ef969e573026f4855c5026dc774c7/href">https://medium.com/media/617ef969e573026f4855c5026dc774c7/href</a></iframe><p><strong>Evaluation: </strong>The binning method used above is a rather crude way to equally divide the continuous (float) value from VADER into one of the five discrete classes we require. However, we do see an overall classification accuracy and macro F1 score improvement compared to TextBlob.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/387/1*YWHzHSbQCqncowVnqlrlHg.png" /></figure><p>The confusion matrix for VADER shows a lot more classes predicted correctly (along the anti-diagonal) — however, the spread of incorrect predictions about the diagonal is also greater, giving us a more “confused” model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*f_9IIZ9CrVKS-Vd8DuVGTA.png" /><figcaption>Each cell in the confusion matrix shows the <strong>percentage</strong> of predictions made for the corresponding true label.</figcaption></figure><p>The greater spread (outside the anti-diagonal) for VADER can be attributed to the fact that it only ever assigns very low or very high compound scores to text that has <strong><em>a</em></strong> <strong><em>lot</em></strong> of capitalization, punctuation, repetition and emojis. Since SST-5 does not really have such annotated text (it is quite different from social media text), most of the VADER predictions for this dataset lie within the range -0.5 to +0.5 (raw scores). This results in a much more narrow distribution when converting to discrete class labels and hence, many predictions can err on either side of the true label.</p><p>Although the result with VADER is still quite low in accuracy, it is clear that its rule-based approach does capture a good amount of fine-gradation in sentiment when compared to TextBlob — fewer cases that are truly negative get classified as positive, and vice versa.</p><h4>3 — <strong>Logistic Regression</strong></h4><p>Moving onward from rule-based approaches, the next method attempted is a logistic regression — among the most commonly used supervised learning algorithms for classification. Logistic regression is a linear model trained on labelled data — the term linear is important because it means the algorithm only uses linear combinations (i.e. sums and not products) of inputs and parameters to produce a class prediction.</p><p>Sebastian Raschka <a href="https://sebastianraschka.com/faq/docs/logisticregr-neuralnet.html">gives a very concise explanation</a> of how the logistic regression equates to a very simple, one-layer neural network in his blog post. The input features and their weights are fed into an activation function (a sigmoid for binary classification, or a softmax for multi-class). The output of the classifier is just the index of the sigmoid/softmax vector with the highest value as the class label.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/1*-fwVAb7DKvTNMBYO5bbylw.png" /><figcaption>Source: <a href="https://sebastianraschka.com/faq/docs/logisticregr-neuralnet.html">Sebastian Raschka’s blog</a></figcaption></figure><p>For multi-class logistic regression, a <a href="https://en.wikipedia.org/wiki/Multiclass_classification#One-vs.-rest">one-vs-rest</a> method is typically used — in this method, we train <em>C</em> separate binary classification models, where <em>C</em> is the number of classes. Each classifier <em>f_c</em>, for <em>c</em> ∈ {1, …, <em>C</em>} is trained to predict whether a sample is part of class <em>c </em>or not.</p><p><strong>Transforming words to features: </strong>To transform the text into features, the first step is to use scikit-learn’s <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html">CountVectorizer</a>. This converts the entire corpus (i.e. all sentences) of our training data into a matrix of token counts. Tokens (words, punctuation symbols, etc.) are created using NLTK’s tokenizer and commonly-used stop words like “a”, “an”, “the” are removed because they do not add much value to the sentiment scoring. Next, the count matrix is converted to a TF-IDF (Term-frequency Inverse document frequency) representation. From the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfTransformer.html">scikit-learn documentation</a>:</p><blockquote>Tf means term-frequency while tf-idf means term-frequency times inverse document-frequency. This is a common term weighting scheme in information retrieval, that has also found good use in document classification. The goal of using tf-idf instead of the raw frequencies of occurrence of a token in a given document is to scale down the impact of tokens that occur very frequently in a given corpus and that are hence empirically less informative than features that occur in a small fraction of the training corpus.</blockquote><p><strong>Sentiment scoring: </strong>Once we obtain the TF-IDF representation of the training corpus, the classifier is trained by fitting it to the existing features. A “newton-cg” solver is used for optimizing the loss in the logistic regression and <a href="https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html">L2 regularization is used by default</a>. A sentiment label is returned for each test sample (using scikit-learn’s learner.predict method) as the index of the maximum class probability in the softmax output vector.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/98b70790547a92d9fdde1ab11775284f/href">https://medium.com/media/98b70790547a92d9fdde1ab11775284f/href</a></iframe><p><strong>Evaluation: </strong>Switching from a rule-based method to a feature-based one shows a significant improvement in the overall classification accuracy and F1 scores, as can be seen below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/385/1*KtPh2wDPyCg5yWXlaM_IWQ.png" /></figure><p>However, the confusion matrix shows why looking at an overall accuracy measure is not very useful in multi-class problems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*NXxx-4gJOJaFvHsLgiR3gA.png" /><figcaption>Each cell in the confusion matrix shows the <strong>percentage</strong> of predictions made for the corresponding true label.</figcaption></figure><p>The logistic regression model classifies a large percentage of true labels 1 and 5 (strongly negative/positive) as belonging to their <em>neighbour</em> classes (2 and 4). Also, hardly any examples are correctly classified as neutral (class 3). Because most of the training samples belonged to classes 2 and 4, it looks like the logistic classifier <em>mostly learned</em> <em>the features that occur in these majority classes</em>.</p><h4>4 — Support Vector Machine</h4><p>Support Vector Machines (SVMs) are very similar to logistic regression in terms of how they optimize a loss function to generate a decision boundary between data points. The primary difference, however, is the use of “kernel functions”, i.e. functions that transform a complex, nonlinear decision space to one that has higher dimensionality, so that an appropriate hyperplane separating the data points can be found. The SVM classifier looks to maximize the distance of each data point from this hyperplane using “support vectors” that characterize each distance as a vector.</p><p>A key feature of SVMs is the fact that it uses a <em>hinge</em> loss rather than a logistic loss. <a href="https://towardsdatascience.com/support-vector-machine-vs-logistic-regression-94cc2975433f">This makes it more robust to outliers in the data</a>, since the hinge loss does not diverge as quickly as a logistic loss.</p><p><strong>Training and sentiment scoring: </strong>The linear SVM in scikit-learn is set up using a similar pipeline as done for the logistic regression described in earlier. Once we obtain the TF-IDF representation of the training corpus, we train the SVM model by fitting it to the training data features. A hinge loss function with a stochastic gradient descent (SGD) optimizer is used, and <a href="https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.SGDClassifier.html">L2 regularization is applied during training</a>. The sentiment label is returned (using scikit-learn’s learner.predict method) as the index of the maximum class probability in the softmax output vector.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8c49e57eafe0e97c9f197993d393b918/href">https://medium.com/media/8c49e57eafe0e97c9f197993d393b918/href</a></iframe><p><strong>Evaluation: </strong>Because quite a few features are likely to be outliers in a realistic dataset, the SVM should in practice produce results that are slightly better than the logistic regression. Looking at the improvement in accuracy and F1 scores, this appears to be true.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/429/1*vKJ5tFIJX37uw15Wyrob5w.png" /></figure><p>The choice of optimizer combined with the SVM’s ability to model a more complex hyperplane separating the samples into their own classes results in a slightly improved confusion matrix when compared with the logistic regression.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*lMZn1l8nPMdHXe-MM9TcYg.png" /><figcaption>Each cell in the confusion matrix shows the <strong>percentage</strong> of predictions made for the corresponding true label.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QE5kxsSZsdQJNwbc0CTAlA.png" /><figcaption>Side by side: Logistic Regression vs. SVM</figcaption></figure><p>The SVM model predicts the strongly negative/positive classes (1 and 5) more accurately than the logistic regression. However, it still fails to predict enough samples as belonging to class 3— a large percentage of the SVM predictions are once again biased towards the dominant classes 2 and 4. This tells us that there is scope for improvement in the way features are defined. A count vectorizer combined with a TF-IDF transformation does not really learn anything about <em>how words are related to one another </em>— they simply look at the number of word co-occurrences in the each sample to make a conclusion.<strong><em> </em></strong><em>Enter word embeddings</em>.</p><h4>5 — FastText</h4><p>FastText, a highly efficient, scalable, CPU-based <a href="https://fasttext.cc/blog/2016/08/18/blog-post.html">library for text representation and classification</a>, was released by the Facebook AI Research (FAIR) team in 2016. A key feature of FastText is the fact that its underlying neural network learns representations, or <em>embeddings</em> that <em>consider similarities between words</em>. While <a href="https://en.wikipedia.org/wiki/Word2vec">Word2Vec</a> (a word embedding technique released much earlier, in 2013) did something similar, there are some key points that stand out with regard to FastText.</p><ul><li>FastText considers subwords using a collection of n-grams: for example, “train” is broken down into “tra”, “rai” and “ain”. In this manner, the representation of a word is more resistant to misspellings and minor spelling variations.</li><li>Unknown words are handled much better in FastText because it is able to break down long words into subwords that might also appear in other long words, giving it better context.</li></ul><p><strong>Python module:</strong> Although the source code for FastText is in C++, an official Python module was released by FAIR in June 2019 (<a href="https://fasttext.cc/blog/2016/08/18/blog-post.html">after several months of confusion within the community</a>). This makes it very convenient to train and test our model completely within Python, without the use of any external binaries. However, to find the optimum hyperparameters, the command line interface for FastText is recommended.</p><p><strong>Training FastText model: </strong>To train the FastText model, use the <a href="https://fasttext.cc/docs/en/support.html#building-fasttext-as-a-command-line-tool">fasttext command line interface</a> (Unix only) — this contains a very useful utility for hyperparameter auto-tuning. As per the <a href="https://fasttext.cc/docs/en/autotune.html">documentation</a>, this utility optimizes all hyper-parameters for the maximum F1 score, so we don’t need to do a manual search for the best hyper-parameters for our specific dataset. This is run using the following command on the terminal, and takes about 5 minutes on CPU.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b386d3739eaf2ec92ff284f7deb87955/href">https://medium.com/media/b386d3739eaf2ec92ff284f7deb87955/href</a></iframe><p>The above command tells FastText to train the model on the training set and validate on the dev set while optimizing the hyper-parameters to achieve the maximum F1-score. The flag -autotune-modelsize 10M tells FastText to optimize the model’s <em>quantization</em> parameters (explained below) such that the final trained model is under 10 MB in size, and the -verbose option is enabled to see which combination of hyper-parameters gives the best results.</p><p>💡 <strong>TIP: Quantize the FastText model: </strong><a href="https://nervanasystems.github.io/distiller/quantization.html">Quantization</a> reduces the number of bits required to store a model’s weights by using 16 or 8-bit integers, rather than standard 32-bit floating points. Doing so vastly reduces model size (by several orders of magnitude). <a href="https://fasttext.cc/docs/en/cheatsheet.html#quantization">FastText makes quantization very convenient</a> in the latest release of its command line interface or its Python module as follows (the extension of the quantized model is <strong>.ftz</strong>, not .bin as the parent model). The cutoff option is set as per the value obtained during hyper-parameter optimization, which ensures that the final model size stays below 10 MB.</p><pre># Quantize model to reduce space usage                           model.quantize(input=train, qnorm=True, retrain=True, cutoff=110539)                           model.save_model(os.path.join(model_path, &quot;sst5.ftz&quot;))</pre><p>The below snippet shows how to train the model from within Python using the optimum hyper-parameters (this step is optional — only the command-line training tool can be used, if preferred).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/556a19a49a822d8ec30f7c7fb9a27d7a/href">https://medium.com/media/556a19a49a822d8ec30f7c7fb9a27d7a/href</a></iframe><p>For more details on the meaning of each hyper-parameter and how FastText works under the hood, <a href="https://towardsdatascience.com/fasttext-under-the-hood-11efc57b2b3">this article gives a good description</a>.</p><p><strong>Sentiment scoring:</strong> Sentiment predictions are made by loading in the trained, quantized (.ftz ) FastText model. The model has a predict method that outputs the most likely labels based on the probabilities extracted from the softmax output layer. For making a class prediction, we simply choose the most likely class label from this list of probabilities, directly extracting it as an integer.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fc0018fba57f4d72e6a8be32f9a48124/href">https://medium.com/media/fc0018fba57f4d72e6a8be32f9a48124/href</a></iframe><p><strong>Evaluation:</strong> It can be seen that the FastText model accuracy and F1 scores do not vastly improve on the SVM for this dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/1*N8GgwwLmBfZS8nIZ9YpIfA.png" /></figure><p>The F1 score for FastText, however, is slightly higher than that for the SVM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*032_hybFUVvwGvbh4mbR4Q.png" /><figcaption>Each cell in the confusion matrix shows the <strong>percentage</strong> of predictions made for the corresponding true label.</figcaption></figure><p>The confusion matrix of both models side-by-side highlights this in more detail.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RAtOI_ouZYrj-svIrEIu7g.png" /><figcaption>Side by side: SVM vs. FastText</figcaption></figure><p>The key difference between the FastText and SVM results is the percentage of correct predictions for the<strong> </strong>neutral class, 3. The SVM predicts more items correctly in the majority classes (2 and 4) than FastText, which highlight the weakness of feature-based approaches in text classification problems with imbalanced classes. Word embeddings and subword representations, as used by FastText, inherently give it additional context. This is especially true when it comes to classifying unknown words, which are quite common in the neutral class (especially the very short samples with one or two words, mostly unseen).</p><p>However, our FastText model was trained using word trigrams, so for longer sentences that change polarities midway, the model is bound to “forget” the context several words previously. A sequential model such as an RNN or an LSTM would be able to much better capture longer-term context and model this transitive sentiment.</p><h4>6 — Flair</h4><p>In 2018, <a href="https://github.com/zalandoresearch/flair">Zalando Research published a state-of-the-art deep learning sequence tagging NLP library</a> called <strong>Flair</strong>. This quickly became a popular framework for classification tasks as well because of the fact that it allowed <em>combining different kinds of word embedding</em>s together to give the model even greater contextual awareness.</p><p>At the heart of Flair is a contextualized representation called <em>string embeddings. </em>To obtain them, sentences from a large corpus are broken down into character sequences to pre-train a bidirectional language model that “learns” embeddings at the character-level. This way, the model learns to disambiguate case-sensitive characters (for example, proper nouns from similar sounding common nouns) and other syntactic patterns in natural language, which makes it very powerful for tasks like named entity recognition and part-of-speech tagging.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/579/1*h5OEkxR9hhbxV_ll4cgCdA.png" /><figcaption>Illustration of a BiLSTM sequence labeller with contextual character embeddings (<a href="https://research.zalando.com/welcome/mission/research-projects/flair-nlp/">Source</a>)</figcaption></figure><p><strong>Training a Flair Model for Classification:</strong> What makes Flair extremely convenient yet powerful is its ability to <a href="https://github.com/zalandoresearch/flair/blob/master/resources/docs/TUTORIAL_4_ELMO_BERT_FLAIR_EMBEDDING.md#combining-bert-and-flair">“stack” word embeddings</a> (such as ELMo or BERT) with “Flair” (i.e. string) embeddings. The below example shows how to instantiate a stacked embedding of BERT (base, cased) or ELMo (original) embeddings with Flair embeddings. The stacked representation is converted to a <em>document embedding</em>, i.e. one that gives a single embedding for an entire text sample (no matter how many sentences). This allows us to condense a complex, arbitrary length representation to a fixed-size tensor representation that we can fit in GPU memory for training.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0041e83215377f79d7984119b7a5a21a/href">https://medium.com/media/0041e83215377f79d7984119b7a5a21a/href</a></iframe><p>The power of stacking embeddings (either BERT or ELMo) this way comes from the fact that <a href="https://github.com/zalandoresearch/flair/blob/master/resources/docs/TUTORIAL_3_WORD_EMBEDDING.md#flair-embeddings">character-level string embeddings capture latent syntactic-semantic information</a> <em>without using the notion of a word </em>(they explicitly focus on subword representations) — while the stacked word embeddings from an external pre-trained neural network model give added word-level context. This enhances the model’s ability to identify a wide range of syntactic features in the given text, allowing it to surpass the performance of classical word embedding models.</p><p><strong>Notes on training: </strong>The Flair model requires a GPU for training, and due to its LSTM architecture does not parallelize as efficiently as compared to transformer architectures — so training time even on this relatively small SST-5 dataset is of the order of<strong> several hours</strong>. For this project, 25 epochs of training were run, and the validation loss was still decreasing when training was stopped, meaning that the model was underfitting considerably. As a result, using Flair on a real-world, large dataset for classification tasks can come with a significant cost penalty.</p><p><strong>Sentiment scoring: </strong>Just as before, a scoring technique is implemented with the existing framework in Pandas. The trained model is first loaded, and the text converted to a Sentence object (which is a tokenized representation of each sentence in a sample). The Flair model’s predict method is called to predict a class label using the maximum index from the softmax output layer, which is then extracted as an integer and stored sample-wise in a Pandas DataFrame. Since model inference can take quite a while even on a GPU, a <a href="https://github.com/tqdm/tqdm/blob/master/examples/pandas_progress_apply.py">tqdm progress bar</a> is implemented to show how many test samples the model finished predicting.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f2a460dc076f3a6c22db1d701b071388/href">https://medium.com/media/f2a460dc076f3a6c22db1d701b071388/href</a></iframe><p><strong>Evaluation: </strong>Two separate stacked representations are used to train two separate models — one using BERT (base, cased) and the other using ELMo (original). Inference is run using each model to give the following results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/457/1*C5TQJLmPJKi542EmQqHUng.png" /></figure><p>There is a sizeable improvement in accuracy and F1 scores over both the FastText and SVM models! Looking at the confusion matrices for each case yields insights into which classes were better predicted than others.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/519/1*ubYG7d7KX7ZquqRiqpVzRA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/515/1*pZXWwFSIl_Ofd0t5sTTwow.png" /></figure><p>The above plots highlight why stacking with BERT embeddings scored so much lower than stacking with ELMo embeddings. The BERT case almost makes no correct predictions for class 1 — however it does get a lot more predictions in class 4 correct. The ELMo model seems to stack much better with the Flair embeddings and generates a larger fraction of correct predictions for the minority classes (1 and 5).</p><p>What went wrong with the Flair + BERT model during training? It could be that re-projecting and decreasing the number of hidden dimensions (during stacking) resulted in a loss of knowledge from the pre-trained BERT model, explaining why this model did not learn well enough on strongly negative samples. It is not exactly clear why stacking ELMo embeddings results in much better learning compared to stacking with BERT. In both cases, however, the Flair models took a large amount of time (several hours) to train, which can be a huge bottleneck in the real-world —yet, they do highlight the power of using contextual embeddings over classical word embeddings for fine-grained classification.</p><h3>Conclusions</h3><p>In this post, six different NLP classifiers in Python were used to make class predictions on the SST-5 fine-grained sentiment dataset. Using progressively more and more complex models, we were able to push up the accuracy and macro-average F1 scores to around 48%, which is not too bad! In a future post, we’ll see how to further improve on these scores using a transformer model powered by transfer learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s0Y00Ve6OTIrdWi8ZF1s6Q.png" /><figcaption>Comparison of results: Fine-grained sentiment classification on SST-5</figcaption></figure><h4>What more can we learn?</h4><p>Plotting normalized confusion matrices give some useful insights as to why the accuracies for the embedding-based methods are higher than the simpler feature-based methods like logistic regression and SVM. It is clear that overall accuracy is a very poor metric in multi-class problems with a class imbalance, such as this one — which is why macro F1-scores are needed to truly gauge which classifiers perform better.</p><p>A key aspect of machine learning models (especially deep learning models) is that they are notoriously hard to interpret. To address this issue, we’ll look at <strong><em>explaining</em></strong> our results and answering the question: “<em>Why did X classifier predict this specific class for this specific sample?</em>”. The <a href="https://github.com/marcotcr/lime">LIME Python library</a> is used for this task, which will be described in the next post.</p><p>If you made it through to the end of this article, thanks for reading!</p><ul><li>This was <strong>Part 1</strong> of a series on fine-grained sentiment analysis in Python.</li><li><a href="https://medium.com/@tech_optimist/fine-grained-sentiment-analysis-in-python-part-2-2a92fdc0160d"><strong>Part 2</strong></a> covers how to build an explainer module using LIME and explain class predictions on two representative test samples.</li><li><a href="https://medium.com/@tech_optimist/fine-grained-sentiment-analysis-part-3-fine-tuning-transformers-1ae6574f25a6"><strong>Part 3</strong></a> covers how to further improve the accuracy and F1 scores by building our own transformer model and using transfer learning.</li></ul><p><strong>NOTE:</strong> All the training and evaluation code for this analysis are <a href="https://github.com/prrao87/fine-grained-sentiment">available in the project’s Github repo,</a> so feel free to reproduce the results and make your own findings!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2697bb111ed4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/fine-grained-sentiment-analysis-in-python-part-1-2697bb111ed4">Fine-grained Sentiment Analysis in Python (Part 1)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Computational Creativity: The Role of the Transformer]]></title>
            <link>https://medium.com/sfu-cspmp/computational-creativity-the-role-of-the-transformer-c3fa20da9c5f?source=rss-efdb4f4a0d1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/c3fa20da9c5f</guid>
            <category><![CDATA[generative-art]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Prashanth Rao]]></dc:creator>
            <pubDate>Wed, 06 Mar 2019 06:15:57 GMT</pubDate>
            <atom:updated>2019-03-07T06:11:06.678Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>An essay to bring to your Attention the recent developments in generative modelling</em>.</h4><p><a href="https://www.linkedin.com/in/prrao87/"><em>Prashanth Rao</em></a> and <a href="https://www.linkedin.com/in/andrew-wesson-7079728b/"><em>Andrew Wesson</em></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4wSbmW9XVEpK6letMoFMDQ.jpeg" /><figcaption>Self-similar fractal patterns (source: <a href="https://pixabay.com/illustrations/fractal-magenta-pattern-geometry-1912482/">Pixabay</a>)</figcaption></figure><p>Over the years, computers have become increasingly sophisticated in their ability to identify more and more complex patterns. The field of <em>computational creativity</em>, a multidisciplinary endeavour to build software that can assist humans in a variety of tasks in the arts, science and the humanities, has seen much progress since the early days of computers where instructions had to be explicitly programmed.</p><p>In this article, we will attempt to unravel some of the recent developments in <em>generative modelling </em>that have shown significant improvements in computers’ ability to generate useful patterns that appeal to human observers. In particular, one type of neural network architecture, the <em>Transformer</em>, will be discussed in detail with regard to its ability to capture longer-term dependencies in text, music and images. Some future directions that this technology could lead to are also discussed.</p><h3>What is creativity, anyway?</h3><p>In <a href="https://nationalhumanitiescenter.org/on-the-human/2010/05/can-computer-models-help-us-to-understand-human-creativity/">her illuminating essay</a>, Margaret Boden describes that creativity is “the ability to come up with ideas or artifacts that are new, surprising, and valuable”. Historically, this has been viewed as a unique aspect of human intelligence. As described by Boden, for something to be deemed creative, it must deliver both <em>novelty</em> and <em>value</em>.</p><p>A common criticism about modelling creativity is that machines (including the most recent deep learning methods) just use “brute-force pattern matching” to achieve an end goal — if the machine is simply identifying existing patterns in the data that it was trained on, then what “novelty” is it truly adding? In addition, “values” are known to be highly variable, and “<a href="https://nationalhumanitiescenter.org/on-the-human/2010/05/can-computer-models-help-us-to-understand-human-creativity/">any arguments about creativity are rooted in disagreements about value</a>”.</p><p>While it may be true that computers might “just be pattern-matching” on a large-scale, it is important to realize that even great scientists and mathematicians are in essence, using their finely tuned pattern-matching abilities (albeit in a highly sophisticated form) to <a href="https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-general-intelligence/">discover patterns in reality that nobody else thought of</a> before. In the below sections, we will look at how transformers could help address both the issues of novelty and value, and how they might combine with other tools to aid humans in <a href="https://www.smithsonianmag.com/innovation/combinatorial-creativity-and-the-myth-of-originality-114843098/">combinatorial creativity</a>.</p><h3>The Transformer</h3><p>The 2017 Google AI paper “<a href="https://arxiv.org/pdf/1706.03762.pdf">Attention is All You Need</a>” first introduced the concept of the transformer neural network architecture. In essence, a transformer is a stack of <em>encoders</em> that can process a sequence of arbitrary length, which connects to a stack of <em>decoders </em>that outputs another sequence. Below is an example of such a structure used for machine translation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*aRAFrbXbkCWG5p2Z5S7vXQ.png" /><figcaption>Image source: <a href="http://jalammar.github.io/illustrated-transformer/">Illustrated Transformer</a> by Jay Alammar</figcaption></figure><p>In recent times, there have been a number of extensions and modifications made to the vanilla transformer that allow it to generalize to other tasks by using encoder-only stacks (<a href="https://arxiv.org/pdf/1810.04805.pdf">BERT</a>), or decoder-only stacks (<a href="https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf">OpenAI GPT</a>), resulting in large-scale improvements to downstream language modelling tasks.</p><p>A deep technical description of the transformer is out of the scope of this article — however there are excellent visual explanations (<a href="http://jalammar.github.io/illustrated-transformer/">Illustrated Transformer</a>) and line-by-line descriptions of the implementation (<a href="http://nlp.seas.harvard.edu/2018/04/03/attention.html">Annotated Transformer</a>) that supplement the original paper.</p><h4>Self-Attention</h4><p>At the core of the transformer is its <em>self-attention </em>mechanism — i.e. the ability to model relationships between parts of a sequence (for example, words in a sentence), regardless of their respective positions in the sequence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/657/1*6uSPs983W-A3dyxpZzk-xA.png" /><figcaption>Image source: <a href="https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html">Google AI blog post on transformers</a></figcaption></figure><p>The above image shows two examples where the meaning of the word “it” could be ambiguous to a machine— in the example on the left, “it” refers to the word “animal”, whereas on the right, “it” refers to “street”. The term self-attention used in this context means that the encoder, while inspecting its input sequence, is able to <em>focus on different parts </em>of the input <em>in relation to itself,</em> allowing it to “attend” to the right representation based on observed patterns in the training data.</p><p>Self-attention is incorporated into an encoder/decoder structure as follows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*ts2aW2ZukB5rCJxmNIAfvA.png" /><figcaption>Image source: <a href="http://jalammar.github.io/illustrated-transformer/">Illustrated Transformer</a> by Jay Alammar</figcaption></figure><p>Via this design, the encoder’s input tokens first flow through the self-attention layer, which allows it to focus on different parts of the sequence in relation to itself. The output of the self-attention layer is then passed to a feed-forward neural network, through which the model “learns” the representation of each word.</p><h4>Multiple Attention Heads</h4><p>The transformer expands on the idea of self-attention by incorporating multiple “attention heads”, i.e. instances of self-attention applied to different parts of the sequence <em>simultaneously</em>. In Google AI’s transformer, there are eight attention heads focusing on different positions at the same time step, applied to <em>each </em>encoder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/405/1*WK9G4xSDgDYroWJbwgEeFw.png" /><figcaption>Image source: <a href="http://jalammar.github.io/illustrated-transformer/">Illustrated Transformer</a> by Jay Alammar</figcaption></figure><p>In essence, having multiple attention heads (eight, in the above image) makes parallelizing the computation relatively trivial, since each head can identify the relationships between words independently of the other. This greatly reduces the amount of training effort, allowing the transformer to <a href="https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html">scale efficiently to very large training sets</a>, unlike more sequential architectures like Recurrent Neural Networks (RNNs).</p><p>Over each training epoch, the individual attention heads update their vector representations, with the final representation being calculated as the <a href="http://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture14-transformers.pdf">weighted average of attentions</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*q8PJY77DfaFZny-KoGAlEw.png" /><figcaption>Image source: <a href="http://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture14-transformers.pdf">Self-Attention for Generative Models</a>, By Ashish Vaswani and Anna Huang</figcaption></figure><p>The decoder side works in a much similar way, and the final output vector representation is fed through a linear layer (which projects the vector output to a much larger “vocabulary space”), followed by a softmax layer (which converts the float representation of the word into a probability distribution for each word in the complete vocabulary).</p><h3>Generative Models with Self-Attention</h3><p>Now that we have identified how self-attention is capable of determining relationships within a sequence, we can understand how this extends to word<em> </em>generation in text, <em>note </em>generation in music, or <em>pixel </em>generation in images.</p><h4>Text Generation: OpenAI GPT-2</h4><p>In February 2019, OpenAI created quite the storm through their release of a new transformer-based language model,<strong> GPT-2</strong> in their blog post “<a href="https://blog.openai.com/better-language-models/">Better Language Models and Their Implications</a>”. At a very high-level, GPT-2 is a transformer-based generative language model that was<a href="https://blog.openai.com/better-language-models/#sample3"> trained on 40GB of curated text from the internet</a>. Being trained in an unsupervised manner, it simply learns to predict a sequence of most likely tokens (i.e. words) that follow a given prompt, based on the patterns it learned to recognize through its training.</p><p>In the below example, GPT-2 (“small”), which consists of 117 million parameters was used to generate text given the prompt “<em>My 10 favourite animals are: 1.” </em>to give the following result.</p><h3>Janelle Shane on Twitter</h3><p>here are gpt-2&#39;s 10 favorite animals</p><p>Although GPT-2 gets the order of numbers (and the actual number of items) completely wrong, the fact that it is able to definitively maintain the list’s structure and specify qualities (such as colour and appearance of each animal) says something about the model’s ability to identify linguistic patterns and syntax at a fundamental level.</p><p>The language model also appears to have an internal representation of what an “animal” actually is — at no point was it defined during training that crocodiles and zebras are kinds of animals — these are <em>learned</em> representations encoded within the model’s parameters.</p><p>While the capabilities of GPT-2 definitely do not supersede those of human imagination, it is quite plausible that similar large-scale language models trained on sufficiently enough compute and data can get to a point where the generated content has a good degree of narrative structure and coherence.</p><p>Indeed, GPT-2 (“large”), which contains 1.5 billion parameters, does show longer term coherence and can formulate an argument while defending its viewpoint over a greater distance compared to smaller models. Below is an example, posted on<a href="https://blog.openai.com/better-language-models/"> OpenAI’s blog</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/809/1*g9Ec9m05qi-AV645q1Js7A.png" /><figcaption>Given a negation prompt, GPT-2 generates an essay that follows a lead and defends its viewpoint</figcaption></figure><h4>Image Generation</h4><p>Traditionally, generative art, or creative image generation has been the domain of <a href="https://hackernoon.com/how-do-gans-intuitively-work-2dda07f247a1"><em>Generative Adversarial Networks</em></a><em> </em>(GANs). These models consist of two components — the generator, which generates images — and the discriminator, which distinguishes the generated images from the ground truth. Both models compete against each other, until the generated images are nearly indistinguishable from those in the training set.</p><p><a href="https://arxiv.org/pdf/1802.05751.pdf">Recent work on image transformers</a> showed that self-attention can also be applied in pixel-space — where an encoder-decoder architecture in a transformer can convert a contextualized, per-pixel representation of a source image into an output image of varying pixel intensities. This is done using “local self-attention”, wherein attention is restricted to windows within local neighbourhoods of a particular query block.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/756/1*2u1B_tx0RFfDvwvsX-FSgA.png" /><figcaption>Image source: <a href="https://arxiv.org/pdf/1802.05751.pdf">Image transformer</a> paper by Parmar et al.</figcaption></figure><p>Mathematically speaking, <a href="http://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture14-transformers.pdf">this approach is described</a> in terms of <em>probabilistic image generation</em>, where an image is viewed as a sequence given by the joint distribution of pixels in vector space. Combining locality with self-attention this way allows <a href="http://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture14-transformers.pdf">image transformers to model longer range dependencies</a> than Convolutional Neural Networks (CNNs) for a lower overall computational cost.</p><p>Modelling long-range dependencies using local self-attention can greatly improve image generation ability, especially with regard to capturing symmetry, or properties like <a href="https://en.wikipedia.org/wiki/Self-similarity">self-similarity</a> (i.e. self-repeating patterns within an entity). <a href="https://www.ranker.com/list/weird-cases-of-self-similarity/rylee_en">Self-similarity is ubiquitous in the natural world,</a> and has great <a href="https://www.ingentaconnect.com/contentone/ist/ei/2017/00002017/00000014/art00034?crawler=true&amp;mimetype=application/pdf">significance in terms of the human perception of artistic beauty</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Cq1B_GkqfPGFf2ZgRAR6Gw.png" /><figcaption>Self-similar fractals in nature (source: <a href="https://pixabay.com/illustrations/forest-path-magic-magical-nature-2497770/">Pixabay</a>)</figcaption></figure><p>As transformers and attention mechanisms become more sophisticated in capturing properties such as self-similarity, it <a href="https://pixabay.com/illustrations/fractal-leaf-green-abstract-630371/">becomes plausible that they will be used as network architectures in future GANs</a>, with the potential to yield much more aesthetically pleasing works of art.</p><h4><strong>Music Generation</strong></h4><p>The task of music generation has historically been of a level of complexity above that of text or images — to generate tunes even somewhat acceptable to human ears, the required sample rate (for the audio) is extremely high, resulting in huge sequences. To put things into perspective, a typical 3-minute pop song sampled at 16 kHz would be a sequence of nearly <strong>3 million</strong> samples. This makes training models on raw audio data extremely impractical.</p><p>In the past, <a href="https://arxiv.org/pdf/1612.07837.pdf">SampleRNN</a> was used to generate raw audio to create a <a href="https://dadabots.bandcamp.com/album/bot-prownies">punk album</a>, but it is noticeably low in fidelity, with believable repetitions for at most a second at a time. Rather than using raw audio, generative techniques can also use intermediate musical representations, i.e. MIDI, as training data and synthesize raw audio — <a href="https://magenta.tensorflow.org/performance-rnn">Performance RNN</a> is an example of an RNN-based model that does this. Although there is a notable increase in structure and musicality compared to SampleRNN, it still largely resembles what musicians would refer to as “noodling” — it sounds musical but feels aimless.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fw.soundcloud.com%2Fplayer%2F%3Furl%3Dhttps%253A%252F%252Fapi.soundcloud.com%252Ftracks%252F339353973%26show_artwork%3Dtrue&amp;url=https%3A%2F%2Fsoundcloud.com%2Fmachinelearningmusic%2Fperformance-rnn-by-magenta&amp;image=http%3A%2F%2Fi1.sndcdn.com%2Fartworks-000239739358-h7778r-t500x500.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=soundcloud" width="800" height="166" frameborder="0" scrolling="no"><a href="https://medium.com/media/d38512abe0bbf2466c22167f54f43897/href">https://medium.com/media/d38512abe0bbf2466c22167f54f43897/href</a></iframe><p>Recent work on the <a href="https://magenta.tensorflow.org/music-transformer">Music Transformer</a> shows that attention better captures the longer-term structure of music, understanding the tendency for music to repeat motifs throughout a composition, varying them appropriately when they return. Unlike the vanilla transformer, the music transformer uses <em>relative self-attention</em>, which allows attention to vary based on how far apart in the sequence elements are. This allows the model to effectively disambiguate long-term and short-term musical structure, giving far more realistic results.</p><h3>Anna Huang on Twitter</h3><p>Check out our new blog post on music transformer that generates minute-long pieces that I find myself listening to again! It also generates compelling harmonizations and realizes chord progressions. Which is your favorite? Ours is the Jazz 🎷 👂https://t.co/orHUISpltW https://t.co/KCs9d0BJdf</p><p>All indications are that the music transformer is the beginning of a new phase in generative music, with <a href="https://magenta.tensorflow.org/assets/music_transformer/transformer_music_performed.mp4">trained musicians expressing their approval and recreating some of the transformer’s compositions</a>.</p><h3>Can Computers Aid Humans in the Creative Process?</h3><p>In this article, we looked at how the attention mechanism in transformers better enables them to generate novel combinations of existing patterns to assist in the so-called task of “<a href="http://v/">combinatorial creativity</a>”. With their ability to capture underlying sub-features in the data such as semantic structure, symmetry and self-similarity, transformers have been shown in recent research to generalize to a number of downstream tasks with excellent results.</p><p>These are still early days in terms of applying transformers and related tools for creative applications — however, upcoming research on <a href="http://web.stanford.edu/class/cs224n/slides/cs224n-2019-lecture14-transformers.pdf">long-range attention</a> could further push the boundaries on their ability to generate vastly more coherent structures in text, images and music. Just as we have relatively primitive “AI” assistants today that can tell us about the weather, will we be using “creative” assistants for idea generation in the future? Will we be teaming up with our computational allies to captivate audiences with vivid stories, paintings and melodies? Time will tell.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c3fa20da9c5f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/sfu-cspmp/computational-creativity-the-role-of-the-transformer-c3fa20da9c5f">Computational Creativity: The Role of the Transformer</a> was originally published in <a href="https://medium.com/sfu-cspmp">SFU Professional Computer Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transfer Learning in NLP for  Tweet Stance Classification]]></title>
            <link>https://medium.com/data-science/transfer-learning-in-nlp-for-tweet-stance-classification-8ab014da8dde?source=rss-efdb4f4a0d1b------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ab014da8dde</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Prashanth Rao]]></dc:creator>
            <pubDate>Tue, 15 Jan 2019 13:58:59 GMT</pubDate>
            <atom:updated>2023-10-06T02:05:05.166Z</atom:updated>
            <content:encoded><![CDATA[<h3>Transfer Learning in NLP for Tweet Stance Classification</h3><h4>A comparison of two transfer learning methods in Natural Language Processing: “<strong>ULMFiT”</strong> and “OpenAI GPT” for a multi-class classification task involving Twitter data</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-hnEkyzxq_ZgOs9jBuGj0g.png" /><figcaption>A <a href="https://github.com/amueller/word_cloud">wordcloud</a> of an interesting Tweet dataset analyzed in this post</figcaption></figure><p>2018 has been a hugely exciting year in the field of Natural Language Processing (NLP), in particular, for <em>transfer learning </em>— a technique where instead of training a model from scratch, we use models pre-trained on a large dataset and then fine-tune them for specific natural language tasks. <a href="http://ruder.io/">Sebastian Ruder</a> provides an excellent account of the past and current state of transfer learning in his post “<a href="http://ruder.io/nlp-imagenet/">NLP’s ImageNet moment has arrived</a>”, explaining why this is such a hot field in NLP right now — his post is a <strong>must read</strong>.</p><p>In recent times, methods such as <a href="http://nlp.fast.ai/classification/2018/05/15/introducting-ulmfit.html">ULMFiT</a>, <a href="https://blog.openai.com/language-unsupervised/">OpenAI GPT,</a> <a href="https://allennlp.org/elmo">ELMo</a> and Google AI’s <a href="https://arxiv.org/pdf/1810.04805.pdf">BERT</a> have revolutionized the field of transfer learning in NLP by using <em>language modelling</em> during pre-training, which has significantly improved on the state-of-the-art for a variety of tasks in natural language understanding. It can be argued that the use of language modelling (which is not without its limitations) is one of the main reasons computers have shown great improvements in their semantic understanding of language.</p><p>One interesting aspect of the transfer learning methods mentioned above is that they use language models pre-trained on well-formed, massive curated datasets that include full sentences with a clear syntax (such as <a href="https://blog.einstein.ai/the-wikitext-long-term-dependency-language-modeling-dataset/">Wikipedia articles</a> and the <a href="https://ai.google/research/pubs/pub41880">1 billion word benchmark</a>). The natural question that arises is — how well can such pre-trained language models generalize to natural language tasks from a different distribution, such as Tweets?</p><h3>Goal of this post</h3><p>In this post we will discuss and compare two modern transfer learning approaches — <a href="http://nlp.fast.ai/classification/2018/05/15/introducting-ulmfit.html">ULMFiT</a> and <a href="https://blog.openai.com/language-unsupervised/">OpenAI’s<strong> GPT</strong></a> (“<strong>G</strong>enerative <strong>P</strong>re-<strong>T</strong>raining”), and show how they can be fine-tuned with relative ease to perform classification tasks from a different distribution — in this case, classifying the stance of Tweets towards a target topic.</p><p>We will aim to develop a modelling approach that can help answer the following questions:</p><ul><li>Does our fine-tuned language model (and classifier) generalize to the unstructured and messy language syntax of Tweets?</li><li>Can we achieve reasonable accuracy (comparable to a benchmark result from 2016) with <em>minimal task-specific customization for each model</em> and with <em>limited computing resources</em>?</li><li>How do the classification results vary based on the model architecture used?</li></ul><p>All the code and data for the results shown below are available in <a href="https://github.com/prrao87/tweet-stance-prediction">this GitHub repo</a>.</p><h4>Why ULMFiT and OpenAI GPT?</h4><p>These two particular transfer learning methods were chosen for this project because they are very similar in terms of how they use language modelling to perform unsupervised pre-training, followed by a supervised fine-tuning step (they are both <em>semi-supervised</em> methods). But they are also different in that they use different network architectures to achieve generalization. ULMFiT uses a 3-layer bi-LSTM architecture, while the OpenAI approach uses a transformer network.</p><p>A full description of ULMFiT and OpenAI’s GPT is too long a topic for this post, but <a href="https://medium.com/explorations-in-language-and-learning/transfer-learning-in-nlp-2d09c3dfaeb6">this article</a> does an excellent job of highlighting the technical details of both model architectures and why they matter, so please do read it!</p><h3>Background</h3><p>Before going into the details of transfer learning for Tweet stance classification, let’s clarify some terminology to understand <em>why</em> transfer learning has so drastically improved state-of-the-art for a variety of natural language tasks in recent times.</p><h4>Moving on from classical word embeddings</h4><p>Historically, pre-trained word embedding techniques such as <a href="http://papers.nips.cc/paper/5021-distributed-representations-of-words-andphrases">word2vec</a> and <a href="https://nlp.stanford.edu/pubs/glove.pdf">GloVe</a>, were heavily used in NLP to initialize the first layer of a neural network before training for a new task. These are<em> shallow</em> representations (a single layer of weights, known as <em>embeddings</em>). Any prior knowledge from the word embeddings are only present in the first layer of the network — the entire network would still need to be trained from scratch for a new target task.</p><p>To derive meaning from<em> sequences of words </em>(such as those seen in natural language), models that utilize word embeddings would still need tremendous amounts of data to disambiguate large sets of words and “learn” from a completely new and unseen vocabulary. As shown in a benchmark result discussed later in this post, the amount of data required for transferring knowledge through word embeddings can be <em>huge, </em>which can result in very large computational costs. Throughout 2018, the advent of <a href="https://www.sysml.cc/doc/50.pdf">powerful pre-trained language models</a> have shown that is possible to gain a much deeper understanding of language semantics and structure for new tasks, especially for long sequences, using the knowledge gained from pre-training on large text corpora.</p><h4>Language Modelling</h4><p>A language model attempts to learn the structure of natural language through hierarchical representations, and thus contains both low-level features (word representations) and high-level features (semantic meaning). A key feature of language modelling is that it is <em>generative</em>, meaning that it aims to predict the next word given a previous sequence of words. It is able to do this because language models are typically trained on very large datasets in an unsupervised manner, and hence the model can “learn” the syntactic features of language in a much deeper way than word embeddings.</p><p><a href="http://ruder.io/nlp-imagenet/">In his post</a>, Sebastian Ruder does a very elegant job of highlighting why language modelling is so powerful for a broad range of NLP tasks. Unlabelled language data is relatively easy to obtain (it is freely available in the form of large text corpora), so by feeding a language model a sufficiently large dataset, it is now possible to perform unsupervised pre-training on billions of words while incorporating a deeper knowledge of language syntax.</p><h4>Semi-supervised learning</h4><p>Transfer learning in NLP is now typically done as a multi-step process— where an <em>entire network</em> is first pre-trained in an unsupervised manner with a language modelling objective. Following this, the model is then fine-tuned on a new task using a supervised approach (with some labelled data), which can then be used for tasks such as text classification. This combination of unsupervised pre-training (using language modelling) followed by supervised fine-tuning is termed as <a href="http://ruder.io/semi-supervised/"><em>semi-supervised</em> learning</a>, and is the approach used to solve our Tweet stance classification problem in this post.</p><h3>Dataset Used</h3><p>The Tweet dataset used in this post comes from <a href="http://alt.qcri.org/semeval2016/task6/">this SemEval 2016 shared task</a>, which contains Tweets that pertain to the following five topics :</p><ul><li>Atheism</li><li>Climate change is a concern</li><li>Feminist movement</li><li>Hillary Clinton</li><li>Legalization of abortion</li></ul><p>The labelled data provided consists of a target topic, a Tweet that pertains to it, and stance of the Tweet towards the target. The data is already split into a training set (containing 2,914 Tweets) and a test set (containing 1,249 Tweets). The stance can be one of three labels: “FAVOUR”, “AGAINST” and “NEITHER”, hence this is a <em>multi-class</em> dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*A4sRK2Vq15ROr7aYjceOWA.png" /><figcaption>Example training data (randomly sampled)</figcaption></figure><h4>The challenge of detecting stance</h4><p>Stance detection is a subcategory of <em>opinion mining</em>, where the task is to automatically determine whether the author of a piece of text is in <em>favour</em> or <em>against</em> a given target. Consider the following two Tweets:</p><blockquote>We don’t inherit the earth from our parents we borrow it from our children</blockquote><blockquote>Last time I checked, Al Gore is a politician, not a scientist.</blockquote><p>To a human observer, it is reasonably clear that both Tweets are relevant to the topic of climate change, and that each expresses a particular stance towards the topic of climate change. However, to a machine, detecting this stance is a difficult problem on multiple fronts. The informal and unstructured syntax of Tweets combined with the fact that machines lack proper contextual awareness and historical knowledge that humans have (for example, knowing who Al Gore is), makes this a challenging problem for machine learning algorithms. In addition, gathering large amounts of labelled data for Tweet stance in order to train a machine learning algorithm is expensive and tedious. It is becoming more and more necessary to develop deep learning methods that can work with limited amounts of training data and still yield useful insights.</p><h4>Stance and sentiment are not the same!</h4><p>Stance detection is related to, but not the same as sentiment analysis. In sentiment analysis, we are interested in whether a piece of text is positive, negative, or neutral based on <em>just</em> the content of the language used. Typically, for sentiment analysis, the choice of positive or negative language correlates with the overall sentiment of the text.</p><p>However, the stance of a piece of text is defined <em>with respect to a target topic</em>, and can be independent of whether positive or negative language was used. The target (topic towards which opinion is expressed) may or may not be mentioned directly in the actual text, and any entities mentioned in the text may or may not be the actual target of opinion. Below is an example from the <a href="https://arxiv.org/pdf/1605.01655.pdf">task creators’ paper.</a></p><blockquote><strong><em>Topic</em></strong>: <em>legalization of abortion</em></blockquote><blockquote><strong><em>Tweet</em></strong>: The pregnant are more than walking incubators. They have rights too!</blockquote><blockquote><strong><em>Stance</em></strong>: <em>Favour</em></blockquote><p>In the above example, since the topic is phrased as “legalization of abortion”, the Tweet can be interpreted as being in <em>favour</em> of the topic. Had it been phrased as “Pro-life movement”, its stance would have been <em>against</em> the topic. It is clear from this example that the language used in the Tweet is only loosely positive in its sentiment; however this sentiment has no bearing on whether it is in favour of, or against the topic.</p><h4>Class Imbalance</h4><p>The below table shows a breakdown of how many Tweets pertain to each topic in the dataset. The 2,914 Tweets are distributed unequally per topic, and there is significant variation in the number of Tweets belonging to each class for each topic. To explore the distribution and inspect the Tweets in more detail, take a look at the <a href="http://www.saifmohammad.com/WebPages/StanceDataset.htm">fully interactive visualization provided by the task creators</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/1*2-sDmSXLreNqfHQK14EJwA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*Qp_wDFPHC8AkmxPJ5HgwMA.png" /><figcaption>Source: <a href="http://www.saifmohammad.com/WebPages/StanceDataset.htm">SemEval-2016 Stance Dataset</a></figcaption></figure><p>Looking at this distribution, the stance classification task appears quite challenging — not only is the dataset small (a couple of thousand training samples in total, ranging from a minimum of 395 to a maximum 689 Tweets per topic) — but there is also a significant class imbalance in the samples. For example, the topic “climate change is a concern” has a larger percentage of training samples classified as “favour”, and a very small percentage (less than <strong>4%</strong>) classified as “against”. On the other hand, the topic “atheism” has a much larger fraction of its samples classified as “against”.</p><p>Any modelling approach for stance classification must be able to capture this class imbalance, both <em>between</em> and <em>within</em> the target classes.</p><h4>Evaluation Metric Used</h4><p>To evaluate classification performance, the creators of the task <a href="http://www.aclweb.org/anthology/S16-1003">use a macro-averaged <strong>F-score</strong></a>, which is the <em>harmonic mean of precision and recall </em>for the two main classes “FAVOUR” and “AGAINST”. An example of how precision and recall are used in sentiment classification is given <a href="https://streamhacker.com/2010/05/17/text-classification-sentiment-analysis-precision-recall/">here</a>. In general, precision is at odds with recall, and hence the F-score provides a good way to gain insights into a classifier’s performance. Although we do not include the third class “NEITHER” in the evaluation, it is implicitly accounted for, since the system has to correctly predict all three classes to avoid being penalized heavily in either of the first two.</p><p>To evaluate the F-score, a <a href="http://alt.qcri.org/semeval2016/task6/index.php?id=data-and-tools">perl script</a> (whose usage will be described in later sections) was provided by the task creators— all we need to do is shape our stance classifier’s prediction output in a way that can be read by the evaluation script. The macro-averaged F-score obtained can then be compared with other models’ results.</p><h3>Benchmark Result for Comparison: MITRE</h3><p>The winning entry for this task in 2016 was from team MITRE, who describe their classification approach <a href="https://arxiv.org/pdf/1606.03784.pdf">in this paper</a>. To detect stance, MITRE used a Recurrent Neural Network (RNN) model organized into 4 layers as shown in the below image. The first layer contained one-hot-encoded tokens (i.e. words from the text) that were projected through a 256-embedding layer called the “projection layer”. The sequence of outputs were then fed into a “recurrent layer” (containing 128 LSTM units), whose output was then connected to a 128-dimensional layer of Rectified Linear Units (ReLUs) with 90% dropout. The final output layer was a 3-dimensional softmax layer representing each output class: FAVOUR, AGAINST and NONE.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/523/1*QxegdHYn8Ja-WmAfODym7w.png" /><figcaption>Image credit: <a href="https://arxiv.org/pdf/1606.03784.pdf">MITRE’s submission to SemEval 2016</a></figcaption></figure><p>MITRE also applied transfer learning (via the use of word embeddings) to reuse prior knowledge of Tweet syntax from a pre-trained model. This was accomplished through a multi-step pre-training process as described below:</p><ol><li><strong>Pre-training the projection layer</strong>: The weights for the projection layer were initialized from 256-dimensional word embeddings learned using the <a href="https://arxiv.org/pdf/1301.3781.pdf">word2vec skip-gram method</a>. To do this, <a href="https://arxiv.org/pdf/1606.03784.pdf">MITRE extracted 218,179,858 English Tweets</a> from Twitter’s public streaming API and then performed weakly supervised learning on this unlabelled dataset (after cleaning the Tweets and lower-casing them). To learn the meaning of compound phrases, they then applied <a href="https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf">word2phrase</a> to identify phrases comprised of up to four words. Out of the 218 million Tweets sampled, 537,366 vocabulary items were used (that appeared at least 100 times in the corpus).</li><li><strong>Pre-training the recurrent layer</strong>: The second layer of MITRE’s network, which consisted of 128 LSTM units, was initialized with weights pre-trained using distant supervision of a hashtag prediction auxiliary task. To begin, 197 hashtags were automatically identified that were relevant to the five topics under consideration (using a nearest-neighbour search of the word embedding space). Then, 298,973 Tweets (out of the total 218 million) were extracted that contained at least one of these 197 hashtags, and the network was trained to tune the word embeddings and the recurrent layer. Like before, the Tweets were lower-cased, stripped of all hashtags and phrase-chunked before tokenization.</li></ol><h4>Results: MITRE</h4><p>The F-scores on unseen test data for each topic and class obtained by MITRE are shown below. Note that for the topic class “Climate change (AGAINST)”, the F-score is zero, meaning that their model did not predict any Tweets from the test set as being against climate change. However, just <strong>3.8%</strong> of the training data from the climate change topic was of the class “AGAINST” (which amounts to just <strong>15</strong> training samples!), so it makes sense that the model might be lacking context for this particular class due to the sheer lack of training samples.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/707/1*_lYamPnTUOBu2KkplFL8QA.png" /><figcaption>Data source: <a href="https://arxiv.org/pdf/1606.03784.pdf">MITRE’s paper submitted to SemEval 2016</a></figcaption></figure><p>As <a href="https://arxiv.org/pdf/1606.03784.pdf">mentioned in their paper</a>, MITRE obtained a macro F-score<strong> </strong>(averaged across all topics) of<strong> 0.68</strong>. This was the best score among all 19 participants who submitted their results to the competition in 2016.</p><p>The approach used by MITRE was a rather elaborate multi-step pre-training procedure using word embeddings that required the use of very large unlabelled datasets (hundreds of millions of samples), significant cleaning of the raw data and separate pre-training steps for each layer in the network. The multitude of hand-crafted steps was mainly because of the limitations of word2vec embeddings (which was the dominant approach to pre-training models in NLP at the time).</p><h3>Method 1: ULMFiT</h3><p>ULMFiT has been entirely implemented in v1 of the fastai library (see <a href="https://github.com/fastai/fastai/tree/master/fastai/text">fastai.text</a> on their GitHub repo). Version 1 of fastai is built on top of <a href="https://pytorch.org/">PyTorch v1</a>, so having some knowledge of PyTorch objects is beneficial to get started. In this post, we cover some of the techniques that fastai has developed that make it very convenient to do transfer learning, even for someone with relatively little deep learning experience.</p><h4>Training Steps</h4><p>As described in the <a href="https://arxiv.org/pdf/1801.06146.pdf">original paper</a>, ULMFiT consists of three stages.</p><ol><li>Training the language model on a general-domain corpus that captures high-level natural language features</li><li>Fine-tuning the pre-trained language model on target task data</li><li>Fine-tuning the classifier on target task data</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w_qNXVr7N2OPCK5iMnHAVQ.png" /><figcaption>Image source: <a href="https://arxiv.org/pdf/1801.06146.pdf">ULMFiT paper by Jeremy Howard and Sebastian Ruder</a></figcaption></figure><p><strong>We only perform steps 2 and 3</strong> during Tweet stance classification. Step 1 is an unsupervised pre-training step, and is <em>really</em> computationally expensive — which is why the models have been <a href="https://github.com/fastai/fastai/blob/master/fastai/datasets.py">made available publicly</a> by fastai so others can benefit from their work. We bank on the pre-trained language model’s ability to capture long-term dependencies in any target text (in English) that we might encounter.</p><p>All the code for this section is available in a Jupyter notebook <a href="https://github.com/prrao87/tweet-stance-prediction/blob/master/ulmfit.ipynb">[ulmfit.ipynb]</a>. For brevity, only the key elements of the approach are discussed in this post — feel free to look through the full notebook and this project’s <a href="https://github.com/prrao87/tweet-stance-prediction">main GitHub repo</a> for a deep-dive into the working code for classifying Tweet stance.</p><h4>Novel learning techniques in ULMFiT</h4><p>The following novel techniques from the ULMFit paper are what allow it to generalize well even on unseen data from a different distribution. It is recommended to read the <a href="https://arxiv.org/pdf/1801.06146.pdf">full paper</a> for a deeper understanding, but a summary is given below.</p><ul><li><strong>Discriminative Fine-tuning: </strong>Each layer of the model captures <em>different types of information</em>. Hence, it makes sense to fine-tune each layer’s learning rates differently, and this is done in ULMFiT based on <a href="https://github.com/cedrickchee/knowledge/blob/master/courses/fast.ai/deep-learning-part-2/2018-edition/lesson-10-transfer-learning-nlp.md">extensive empirical testing and implementation updates</a>. It was empirically found that first fine-tuning only the last layer (with the others frozen), and then unfreezing all the layers and applying a learning rate lowered by a factor of 2.6 for all other layers during language model fine-tuning <a href="https://arxiv.org/pdf/1801.06146.pdf">worked well in most cases</a>.</li><li><strong>1-cycle learning rate policy: </strong>In the fine-tuning stage, we apply 1-cycle learning rates, which comes from this <a href="https://arxiv.org/abs/1803.09820">report by Leslie Smith</a>. It is a modification of the cyclical learning rate policy, which has been around for a long time, but the 1-cycle policy allows a larger initial learning rate (say max_LR = 1e-03), but decreases it by several orders of magnitude just at the last epoch. This seems to provide greater final accuracy. Note that this doesn’t mean we run it for one epoch — the ‘1’ in 1-cycle means it just cycles the learning rate down <em>one epoch before</em> the max epochs that we specify. In the ULMFiT implementation, this 1-cycle policy has been tweaked and is referred to as <em>slanted triangular learning rates</em>.</li><li><strong>Gradual unfreezing: </strong>During classification, rather than fine-tuning all the layers at once, the layers are “frozen” and the last layer is fine-tuned first, followed by the next layer before it, and so on. This avoids the phenomenon known as <em>catastrophic forgetting</em> (by losing all prior knowledge gained from the language model).</li><li><strong>Concatenated pooling: </strong>Pooling is a component of neural networks to aggregate the learned features and reduce the overall computational burden of a large network. In case you’re curious, a good introduction to pooling as applied in LSTMs is <a href="https://arxiv.org/pdf/1706.05157.pdf">given in this paper</a>. In ULMFiT, because an input text can consist of hundreds or thousands of words, information might get lost if we only consider the last hidden state in the LSTM. To avoid this information loss, the hidden state at the last time step is <strong>concatenated</strong> with <em>both</em> the max-pooled and mean-pooled representation of the hidden states over as many time steps as can fit in GPU memory.</li></ul><h4>ULMFiT’s language model</h4><p>ULMFiT’s pre-trained language model was trained on the <a href="https://www.salesforce.com/products/einstein/ai-research/the-wikitext-dependency-language-modeling-dataset/">Wikitext 103 dataset by Stephen Merity.</a> fast.ai <a href="https://github.com/fastai/fastai/blob/master/fastai/datasets.py">provides an API</a> where this pre-trained model (along with some standard datasets for testing) can be conveniently and easily loaded for any target task before fine-tuning. The main thing to note about Wikitext 103 is that it consists of a pre-processed subset of 103 million tokens extracted from Wikipedia. The dataset retains the original case (it was not lower-cased before training the language model), and all punctuation and numbers are included. The type of text data included in this dataset includes sentences from full Wikipedia articles, so the hope is that the language model is able to capture and retain some longer-term dependencies from relatively complex sentences.</p><h4>Pre-process Tweet data for ULMFiT</h4><p>The original Tweets can contain some arbitrary non-English characters, so we take care of this before loading in the language model. Note that this only removes non-English characters, but does <strong>not</strong> do any other kind of aggressive pre-processing (like lower-casing or removing entire words or hashtags). We retain the entirety of the information in the raw data and let the language model do the heavy lifting.</p><p>The ULMFiT framework implemented in fast.ai works very well with Pandas DataFrames, so all data is read in using pandas.read_csv and stored as a DataFrame.</p><pre>train_orig = pd.read_csv(path/trainfile, delimiter=&#39;\t&#39;, header=0, encoding = &quot;latin-1&quot;)</pre><p>We then make sure that the Tweets only contain ASCII characters by applying a simple cleanup function.</p><pre>def clean_ascii(text):<br>    # function to remove non-ASCII chars from data<br>    return &#39;&#39;.join(i for i in text if ord(i) &lt; 128)</pre><pre>train_orig[&#39;Tweet&#39;] = train_orig[&#39;Tweet&#39;].apply(clean_ascii)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*FFHPSO5w9LEIHdQ-B-cf_Q.png" /><figcaption>Training data stored in a Pandas DataFrame</figcaption></figure><p>ULMFiT requires just the stance and the text data (i.e. Tweets) for the language-model finetuning and classification steps, hence we store these in the relevant DataFrame and write the clean data out to a csv file..</p><pre>train = pd.concat([train_orig[&#39;Stance&#39;], train_orig[&#39;Tweet&#39;]], axis=1)<br># Write train to csv<br>train.to_csv(path/&#39;train.csv&#39;, index=False, header=False))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/513/1*s3NwQ5OGEv2VpWF5AgHyng.png" /><figcaption>Clean training data ready to be loaded into ULMFiT</figcaption></figure><h4>Language model fine-tuning in ULMFiT</h4><p>This is the first stage of training, where we use the pre-trained language model weights (Wikitext 103) and fine-tune it with just the provided training data of 2,914 Tweets.</p><p>The published Jupyter notebook (<a href="https://github.com/prrao87/tweet-stance-prediction/blob/master/ulmfit.ipynb">ulmfit.ipynb</a>) describes in detail the pre-processing that is performed under-the-hood by fastai when using ULMFiT. The reader is directed to look at the Jupyter notebook for more code and API details.</p><p>In our case, we specify a minimum word frequency of 1 for our language model fine-tuning step, which tells ULMFiT to only tokenize words in the Tweets that appear more than once with a unique token — all the words that appear once are given the tag &lt;unk&gt; during tokenization. For a very detailed historical account of how and <em>why</em> tokenization is done this way, this <a href="https://github.com/cedrickchee/knowledge/blob/master/courses/fast.ai/deep-learning-part-2/2018-edition/lesson-10-transfer-learning-nlp.md">fastai course documentation page</a> contains some very useful information.</p><pre>data_lm = TextLMDataBunch.from_csv(path, &#39;train.csv&#39;, min_freq=1)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/733/1*caAtcFM-_vkOIfEGRBC0-A.png" /><figcaption>Tokenized and tagged Tweets using ULMFiT + spaCy</figcaption></figure><p>On viewing the tokenized Tweets, we can see that they look markedly different from their original form. The tokenization technique used by fastai.text is quite advanced and obtained after <a href="https://github.com/cedrickchee/knowledge/blob/master/courses/fast.ai/deep-learning-part-2/2018-edition/lesson-10-transfer-learning-nlp.md">months of development by Jeremy Howard and the </a><a href="https://github.com/cedrickchee/knowledge/blob/master/courses/fast.ai/deep-learning-part-2/2018-edition/lesson-10-transfer-learning-nlp.md">fastai team</a>, and thus uses quite a few tricks to capture semantic meaning from the text.</p><p>Note that we are <strong>not</strong> converting the text to lowercase and removing stopwords (which was a common pre-tokenization approach in NLP until recently) — this would result in a <em>tremendous</em> loss of information that the model could instead use to gather an understanding of the new task&#39;s vocabulary. Instead, a number of added tags are applied to each word as shown above so that minimal information is lost. All punctuation, hashtags and special characters are also retained.</p><p>For example, the xxmaj token <a href="https://github.com/fastai/fastai/blob/1c5c007e10a187f4a048a3ee8137016534259768/fastai/text/transform.py#L71">[source]</a> indicates that there is capitalization of the word. &quot;The&quot; will be tokenized as &quot;xxmaj the&quot;. Words that are fully capitalized, such as &quot;I AM SHOUTING&quot;, are tokenized as &quot;xxup i xxup am xxup shouting&quot;. The method still uses <a href="https://spacy.io/usage/spacy-101#annotations-token"><em>spaCy</em>&#39;s underlying tokenizer</a> (including a multi-thread wrapper around spaCy to speed things up), but adds tags in a very smart way . This balances capturing semantic meaning with reducing the number of overall tokens — so it is both powerful and efficient. For a full list of all the token tags generated by ULMFiT’s fastai implementation, see the <a href="https://github.com/fastai/fastai/blob/4cd2e454a735e208c60b2ff7f82bbb4b62e85725/fastai/text/transform.py#L19">source code here</a>.</p><p><strong>Find the optimum learning rate: </strong>We define a learner object that uses the tokenized language model data, that is organized into batches for the GPU, and feed it a pre-trained language model as follows.fastai.train <a href="https://github.com/fastai/fastai/blob/050080be574cb1260462bbd03e9600e43e7a54b1/fastai/train.py#L23">[source</a>] provides a convenient utility to search through a range of learning rates to find the optimum one for our dataset. The idea is that our optimization function needs to use a learning rate that is at least an order of magnitude below the point at which the loss starts to diverge.</p><pre>learn = language_model_learner(data_lm, pretrained_model=URLs.WT103_1,<br>        drop_mult=0.1)</pre><pre>learn.lr_find(start_lr=1e-8, end_lr=1e2)<br>learn.recorder.plot()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/1*RMUuiFk0omtUW9Av0hmYHg.png" /><figcaption>Result of the “learning rate finder” (lr_find) as implemented in fastai.train</figcaption></figure><p>Applying discriminative fine-tuning as per the ULMFiT paper, we run the language model fine-tuning step until the validation loss drops to a low value (close to 0.3).</p><pre>learn = language_model_learner(data_lm, pretrained_model=URLs.WT103_1, <br>                               drop_mult=0.5)<br># Run one epoch with lower layers <br>learn.fit_one_cycle(cyc_len=1, max_lr=1e-3, moms=(0.8, 0.7))</pre><pre># Run for many epochs with all layers unfrozen<br>learn.unfreeze()<br>learn.fit_one_cycle(cyc_len=20, max_lr=1e-3, moms=(0.8, 0.7))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/362/1*lbi3sfZv-4LR80d0_kK1zQ.png" /></figure><p>This fine-tuned encoder layer’s weights are then saved for use during the classification stage.</p><pre># Save the fine-tuned encoder<br>learn.save_encoder(&#39;ft_enc&#39;)</pre><h4>Fine-tuning the classifier</h4><p>This step involves creating a classifier object [<a href="https://github.com/fastai/fastai/blob/4cd2e454a735e208c60b2ff7f82bbb4b62e85725/fastai/text/learner.py#L146">source</a>] that can predict a class label once we re-train the model as a classifier. The same network structure is still used for this task — the output layer is defined in a way that takes into account the number of classes we want to predict in our data.</p><p><strong>Find optimum learning rate for the classifier</strong>: Just as before, the lr_find method is run like before to find an optimum learning rate for the classifier.</p><pre># Classifier databunch<br>data_clas = TextClasDataBunch.from_csv(path, &#39;train_topic.csv&#39;,<br>            vocab=data_lm.train_ds.vocab, min_freq=1, bs=32)</pre><pre># Classifier learner <br>learn = text_classifier_learner(data_clas, drop_mult=0.5)<br>learn.load_encoder(&#39;ft_enc&#39;)<br>learn.freeze()</pre><pre>learn.lr_find(start_lr=1e-8, end_lr=1e2)<br>learn.recorder.plot()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/472/1*ih47bEska7RZgX5B6k1XSA.png" /><figcaption>Result of the “learning rate finder” (lr_find) as implemented in fastai.train</figcaption></figure><p><strong>Carefully train the classifier</strong>: During classification, we first define a classifier learner object, and <em>gradually</em> unfreeze layers while running for one epoch each time as per the ULMFiT paper’s suggestion. This helps us obtain a better classification accuracy than if we were to aggressively train all the layers at once.</p><pre>learn = text_classifier_learner(data_clas, drop_mult=0.5)<br>learn.load_encoder(&#39;ft_enc&#39;)<br>learn.freeze()</pre><pre>learn.fit_one_cycle(cyc_len=1, max_lr=1e-3, moms=(0.8, 0.7))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/371/1*DFGk1kZtZwaf04Eg8Lj4UQ.png" /></figure><pre>learn.freeze_to(-2)<br>learn.fit_one_cycle(1, slice(1e-4,1e-2), moms=(0.8,0.7))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/367/1*xe7YKP8bI_iHuEgSRWzIPg.png" /></figure><pre>learn.freeze_to(-3)<br>learn.fit_one_cycle(1, slice(1e-5,5e-3), moms=(0.8,0.7))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/363/1*6Hox5FU2JWttgnLkOQtpwQ.png" /></figure><pre>learn.unfreeze()<br>learn.fit_one_cycle(4, slice(1e-5,1e-3), moms=(0.8,0.7))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/368/1*t10d9dEbPSJ7wYZNMBIttA.png" /></figure><p>The validation loss for the classifier is much higher than for the language model, which could possibly be linked with the dataset. Since there is significant class imbalance, a significant number of labels during training are expected to be predicted incorrectly. Specific to this Twitter dataset, this problem could be more severe than what we see in the state-of-the-art examples (such as IMDb).</p><h4>Predict stance using the classifier</h4><p>We read in the test set and store it in a Pandas DataFrame as shown.</p><pre>test = pd.read_csv(path/testfile, delimiter=&#39;\t&#39;, header=0,<br>       encoding=&quot;latin-1&quot;)<br>test = test.drop([&#39;ID&#39;], axis=1)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*AAxFTefUHDyp8NW3zrmlLA.png" /></figure><p>We can then apply our classifier learner’s predict method to predict the stance.</p><pre>test_pred[&#39;Stance&#39;] = test_pred[&#39;Tweet&#39;].apply(lambda row: str(learn.predict(row)[0]))</pre><pre># Output to a text file for comparison with the gold reference<br>test_pred.to_csv(path/&#39;eval&#39;/&#39;predicted.txt&#39;, sep=&#39;\t&#39;, index=True,<br>                 header=[&#39;Target&#39;, &#39;Tweet&#39;, &#39;Stance&#39;], index_label=&#39;ID&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/607/1*78rHwE2Dqy0TPzb_wmcVHw.png" /></figure><h4>Evaluate our prediction from ULMFiT</h4><p>The perl script <a href="http://alt.qcri.org/semeval2016/task6/data/uploads/eval_semeval16_task6_v2.zip">provided by the task creators</a> is run on the output and gold reference file (also provided by the creators) to produce a <strong>macro F-score</strong> that we can then compare with the benchmark result from MITRE.</p><pre>cd eval/<br>perl eval.pl gold.txt predicted.txt</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/601/1*jyp9K51VHeOpRP-8XEXwIg.png" /><figcaption>Best obtained result using ULMFiT</figcaption></figure><h4>Best model parameters using ULMFiT</h4><p>The best <strong>average macro F-score of 0.65</strong> using ULMFiT across all topics was obtained using the below approach and parameters:</p><ul><li>Fine-tuning the language model on an <em>augmented</em> Twitter vocabulary: this was done by downloading the <a href="https://www.kaggle.com/kazanova/sentiment140">Twitter sentiment140 dataset from Kaggle</a>, and feeding a subset of its vocabulary (200,000 words) to the language model fine-tuning step. The steps for this are shown in detail the Jupyter <a href="https://github.com/prrao87/tweet-stance-prediction/blob/master/ulmfit.ipynb">notebook </a><a href="https://github.com/prrao87/tweet-stance-prediction/blob/master/ulmfit.ipynb">ulmfit.ipynb</a>. The full Kaggle Twitter dataset has 1.6 million Tweets, which would have taken several hours to fine-tune the language model on (even on an NVIDIA P100 GPU) — so only a subset of 200,000 words were used to augment the language model fine-tuning step.</li><li>Training 5 <em>distinct</em> classifiers (i.e. a separate training task for each topic during classification) and then combining the outputs for comparison with the gold reference— this was a technique similar to that used by MITRE in their best result from 2016, and is <a href="https://arxiv.org/pdf/1606.03784.pdf">explained in their paper</a>.</li><li>An optimum learning rate of 1e-03 for the language model fine-tuning step</li><li>An optimum learning rate in the range of 1e-05 to 1e-03 for the classifier, with gradual unfreezing</li></ul><h3>Method 2: OpenAI GPT</h3><p>OpenAI GPT, as <a href="https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf">described in their paper</a>, is an adaptation of the well-known transformer from Google Brain’s 2017 paper “Attention is All You Need”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*78qmLJHooYLdeMyRmXeq8Q.png" /><figcaption>Image credit: Google Brain’s <a href="https://arxiv.org/pdf/1706.03762.pdf">“Attention is All You Need” paper</a></figcaption></figure><p>While the original version from Google Brain used an identical encoder-decoder 6-layer stack, GPT uses a 12-layer decoder-only stack. Each layer has two sub-layers, consisting of a multi-head self-attention mechanism, and a fully connected (position-wise) feed-forward network. A full description of the transformer architecture used by OpenAI for transfer learning is given in <a href="https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf">their paper</a>.</p><p>All the code for this section is available in a Jupyter notebook <a href="https://github.com/prrao87/tweet-stance-prediction/blob/master/transformer.ipynb">[transformer.ipynb]</a>. Just as before, only the key elements of the model are discussed in this post for brevity — feel free to look through the full notebook and this project’s <a href="https://github.com/prrao87/tweet-stance-prediction">main GitHub repo</a> for a deep-dive into the working code for classifying the Tweets.</p><h4>Training Steps</h4><p>The following steps are used to train the OpenAI transformer:</p><ol><li><strong>Unsupervised pre-training</strong>: The transformer language model was trained in an unsupervised manner on a few thousand books from the Google Books corpus and the pre-trained weights are made publicly available <a href="https://github.com/openai/finetune-transformer-lm/tree/master/model">on the OpenAI GitHub repo</a> for others’ benefit.</li><li><strong>Supervised fine-tuning</strong>: We can adapt the parameters to the supervised target task. The inputs are passed through the pre-trained model to obtain the final transformer block’s activation.</li></ol><p>The first step (unsupervised pre-training) is <em>very</em> expensive, and was done by OpenAI (who trained the model for a month on 8 GPUs!) — thankfully, we can use the downloaded pre-trained model weights and proceed directly to the supervised fine-tuning step.</p><h4>PyTorch Implementation of OpenAI GPT</h4><p><strong>NOTE</strong>: While the original OpenAI transformer was implemented in TensorFlow [<a href="https://github.com/openai/finetune-transformer-lm">GitHub source</a>] — for this project we used <a href="https://github.com/huggingface/pytorch-openai-transformer-lm">this PyTorch port of OpenAI GPT</a>, thanks to the amazing work by the folks at <a href="https://github.com/huggingface">HuggingFace</a>. This is so that we can make a more consistent comparison with the fastai implementation of ULMFiT (which is also PyTorch-based), not to mention the ease of maintaining and distributing the code using one single framework.</p><h4>Novel Techniques introduced in the OpenAI GPT paper</h4><p>To perform out-of-domain target tasks, GPT includes language modelling as an additional objective to the fine-tuning, which helps generalized learning as <a href="https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf">described in their paper</a>. This auxiliary language modelling objective is specified with a weighting parameter as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/253/1*lshCyd-V8owaZHLeF7uQOA.png" /></figure><p>Where L1, L2 and L3 are the likelihoods for the language modelling objective, task-specific objective and combined objectives respectively.</p><p>The transformer has <a href="https://arxiv.org/pdf/1706.03762.pdf">been proven to be a very powerful language modelling tool</a>, especially in machine translation, thanks to its “self-attention mechanisms”. A very intuitive and visual explanation of how the transformer is suited to capturing longer-range linguistic structure (using masked self-attention) <a href="http://jalammar.github.io/illustrated-transformer/">is explained in this article</a>. A key point to note is that OpenAI GPT used a 12-layer decoder-only structure, with 12 attention heads and a <strong>768-</strong>dimensional state. Another key point is that each minibatch was capable of sampling a maximum of <strong>512</strong> contiguous tokens, which, according to the OpenAI authors allows the transformer to <a href="https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf">achieve a much longer-range context than LSTM-based approaches</a>.</p><p><strong>Task-specific input transformations</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JTKGjUBq_mX6Mg3bWNippQ.png" /></figure><p>OpenAI designed their transformer to be <em>task-agnostic </em>and be able to generalize to a range of natural language tasks. To accomplish this, they allow the definition of custom “task-specific heads” as per the above image. The task-specific head acts on top of the base transformer language model, and is defined in the DoubleHeadModel class in model_pytorch.py(<a href="https://github.com/prrao87/tweet-stance-prediction">see the GitHub repo</a>).</p><p>The <a href="https://github.com/huggingface/pytorch-openai-transformer-lm">PyTorch port</a> of GPT originally written by HuggingFace was for a multiple choice classification problem (ROCStories). For this Tweet stance detection task, we use the guidelines mentioned in the OpenAI paper to write a custom input transform for a classification task-head, such that we pad every text (representing each Tweet, in our case) with a start symbol and tokenize them for input to the encoder layer. This is done as follows.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2edcdd8f6bbbc5387fa115c4e4fcbf85/href">https://medium.com/media/2edcdd8f6bbbc5387fa115c4e4fcbf85/href</a></iframe><p>Note that in the classification input transform function above, all we need to do is specify the start token to the encoder, and then append the text to this start token. We maintain the same tensor dimensionalities for each of the other variables as the original PyTorch code.</p><h4>Pre-process Tweet data for OpenAI GPT</h4><p>The dataloader to the training script is defined as follows. Just as in the case of ULMFiT, we clean the data to remove any non-ASCII characters (to avoid issues during the encoding step). The data is then stored as Numpy arrays.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c82506ff8abb59200db24d3bcc20ba13/href">https://medium.com/media/c82506ff8abb59200db24d3bcc20ba13/href</a></iframe><p>To feed the data to the classification transform, we split it into training, validation and test sets using scikit-learn’s train_test_split utility.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/12b376031aa848dca3779b9d7006e6ce/href">https://medium.com/media/12b376031aa848dca3779b9d7006e6ce/href</a></iframe><p>Using the above code, we store the training Tweet (trX) alongside its numericalized stance (trY). We feed in the data to the PyTorch dataloader in train_stance.py as Numpy arrays (not DataFrame columns). Below is an example of the input data once it is shaped as required by the transformer. The stance “0” corresponds to “AGAINST”, “1” to “FAVOUR” and 2 to “NEITHER”. Just as in ULMFiT, no information from the raw data is removed — we rely on the language model objective to identify syntactic relationships between unseen words (once the model fine-tunes).</p><pre>Plenty of stem cells without baby smashing, by the way. #SemST 0</pre><pre>Is there a breeze i can catch Lakefront or will I die of a heat stroke there as well? #heatstroke #SemST 1</pre><pre>Road to #Paris2015 &quot;#ADP2015 co-chairs&#39; new tool will be presented on July 24th&quot; @manupulgarvidal at @UN_PGA event on #SemST 2</pre><pre>Are the same people who are red faced and frothing over abortion also against the death penalty? Just wondering... #deathpenalty #SemST 1</pre><pre>DID YOU KNOW: The 2nd Amendment is in place &#39;cause politicians ignore the #CONSTITUTION. #PJNET #SOT #tcot #bcot #ccot #2ndAmendment #SemST 0</pre><pre>I Appreciate almighty God for waking me up diz beautiful day + giving me brilliant ideas to grow my #Hustle #SemST 0</pre><pre>Speaking from the heart is rarely wise especially if you have a retarded heart like most feminists.   #GamerGate #SemST 0</pre><pre>@Shy_Buffy welcome little sister! Love you! #SemST 2</pre><pre>@GregAbbott_TX which god? Yours? not mine. oh wait i don&#39;t have one. #LoveWins #SemST 1</pre><h4>Fine-tune the language model and classifier</h4><p>In OpenAI GPT, the language model and classifier fine-tuning are both done simultaneously, thanks to its parallelized architecture using multi-headed attention. This makes it very easy to run the training loop and hence a number of experiments were possible.</p><p>The below command was used to run the training loop for 3 epochs, which is what was used for all our experiments. See the Jupyter notebook (<a href="https://github.com/prrao87/tweet-stance-prediction/blob/master/transformer.ipynb">transformer.ipynb</a>) and the file <a href="https://github.com/prrao87/tweet-stance-prediction/blob/master/transformer-openai/train_stance.py">train_stance.py</a> for more details on the default arguments and the various experiments run.</p><pre>python3 train_stance.py --dataset stance --desc stance --submit --data_dir ../data --submission_dir default --n_iter 3</pre><p>The output of the training script is fed to another script parse_output.py, which shapes the output in a way that can be evaluated by the perl script as <a href="http://alt.qcri.org/semeval2016/task6/data/uploads/eval_semeval16_task6_v2.zip">provided by the task creators</a>.</p><h4>Evaluate our GPT model’s prediction</h4><p>The evaluation perl script is run on the output and gold reference file (also provided by the creators) to produce a <strong>macro F-score</strong> that we can then compare with the benchmark result from MITRE.</p><pre>cd eval/<br>perl eval.pl gold.txt predicted.txt</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/667/1*QY2fLSBZDteaMDDRXLjBVA.png" /><figcaption>Best obtained result using the OpenAI Transformer</figcaption></figure><h4>Best model parameters using OpenAI GPT</h4><p>The best <strong>macro F-score of 0.69</strong> obtained from GPT across all topics was obtained using the below approach and parameters:</p><ul><li>Fine-tuning the language model just the provided training Tweets (2,914 of them). The transformer model was able to quickly generalize to the Tweet data in 3 epochs, and was able to obtain a good macro F-score without having to augment the input data in any way.</li><li>Training <em>one</em> <em>single </em>classifier for all topics at once (i.e. running the training loop on the entire training dataset) — it was noticed that when trying to train the transformer on just a single topic (which had &lt; 500 training samples), there was significant over-fitting where the validation accuracy dropped to well below 70%. This could be because the transformer has a high-dimensional embedding layer (768 dimensions) that requires sufficient amount of training data to avoid over-fitting.</li><li>A language modelling weighting function (lambda) of 0.5 as per the OpenAI paper.</li><li>Dropouts of 0.1 on all layers (including the classification layer), once again as per the OpenAI paper.</li></ul><p>Overall, fine-tuning the dropout, the weight of the language modelling objective and changing some of the other default arguments (such as random seed) did little to improve the macro F-score. In general, the transformer was very quick to produce good results and required only some basic customization of the task-head input transforms.</p><h3>Analysis of Results</h3><p>In this section we compare the F-scores obtained by our two approaches: ULMFiT and GPT with the benchmark result by MITRE.</p><h4>Overall F-score</h4><p>When we consider the macro F-scores for stance across all five topics at once, OpenAI GPT is shown to produce the best result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/1*kwRUiB8OTMp_hOO0uRl3tw.png" /><figcaption>Best results compared with the benchmark from MITRE</figcaption></figure><p>What is remarkable about this result is that both ULMFiT and OpenAI GPT utilize pre-trained language models trained on a very different distribution (Wikipedia and Google Books respectively) whereas MITRE used embeddings pre-trained on a massive Twitter dataset (similar distribution).</p><p>It is clear that utilizing language modelling as a training signal during pre-training as well as fine-tuning has its advantages. Even though Tweets are far more informal in their syntax and have shorter sequence lengths than the average sentence from a book corpus, the pre-trained models in either case are able to generalize and make predictions based on some syntactic understanding of the target data.</p><h4>Topic-wise F-score</h4><p>The below image compares the F-score (FAVOUR) and F-score (AGAINST) from our two approaches with MITRE’s, this time on a per-topic basis. Looking at these results, it is once more clear that OpenAI GPT clearly outperforms ULMFiT on most topics, for either class, which explains the higher overall F-score.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YzkQ10JPmmQAYwVjXX62cg.png" /><figcaption>Comparison of topic-wise F-scores from ULMFiT and the OpenAI transformer vs. MITRE’s</figcaption></figure><p>The transformer does really well on all topics, regardless of whether it is predicting the “FAVOUR” or “AGAINST” class.</p><p>One very interesting observation is that both ULMFiT and MITRE produce <em>zero</em> predictions for “Climate change is a concern (AGAINST)” whereas GPT is able to make some correct predictions for this case. On inspecting the training data further (shown earlier in this post), this is the same class for which we have <em>very</em> few training samples (just <strong>15</strong> of them!). It is remarkable that with such few training examples, the transformer is able to achieve some context and still assign some predictions to this class. Judging by its good performance in the many other widely varying classes, this doesn’t seem like a fluke!</p><h4>Cross-tabulated Results: Ours vs. Gold Reference</h4><p>We count the total predictions made by both models in each class and cross-tabulate it against the labels in the gold reference set (i.e. the “absolute truth”) <a href="http://alt.qcri.org/semeval2016/task6/index.php?id=data-and-tools">provided by the task creators</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*FxjKpP6Kvo-KS4IU_-kiBQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/610/1*O_IpylbV7br5auJHU91UOQ.png" /></figure><p>In these tables, the elements along the main diagonal indicate the correctly predicted labels and the off-diagonal elements show where the classifier went astray. It is clear that both methods predict most classes reasonably well, with ULMFiT making more overall correct predictions in the “AGAINST” class, and the transformer making more correct predictions in the “FAVOUR” and “NEITHER” classes.</p><p>We can look at the training data distribution to see if we can draw additional inferences. In the training data, there are far more Tweets belonging to the “AGAINST” class (image below). The transformer seems to be able to achieve a broader overall context across even the minority classes, even when there are very few training samples per topic!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/574/1*qofxAVcgC2SjcPXRhjojkQ.png" /><figcaption>Distribution of classes (overall) from original training data</figcaption></figure><h4>Effect of Augmentation Vocabulary</h4><p>ULMFiT produced relatively poor overall average F-scores (of less than <strong>0.55</strong>) when we fine-tuned the language model on just the original training set (of 2,914 Tweets). <em>Only</em> when we augmented the language model with 200,000 Tweets from the <a href="https://www.kaggle.com/kazanova/sentiment140">Kaggle Sentiment140 dataset</a> did ULMFiT produce overall F-scores that were comparable to MITRE’s benchmark results. The language model fine-tuning of the bi-LSTMs in ULMFiT does seem to require more data to generalize <em>in this case</em> than the transformer does.</p><p>Our GPT model did not require an augmentation vocabulary during the fine-tuning step. The fact that it is able to achieve good generalization across even the minority classes seems to show that it achieves a better general understanding of Twitter-syntax in the presence of limited training samples. All this seems very interesting and worthy of further study over more diverse datasets.</p><h4>Effect of Sequence Length</h4><p>It is well known that ULMFiT <a href="https://github.com/sebastianruder/NLP-progress/blob/master/english/text_classification.md">produces state-of-the-art accuracy on various text classification benchmark datasets such as IMDb and AG News</a>. The common theme across all these benchmark datasets is that they have really <em>long</em> sequence lengths (some of the reviews/news articles are hundreds of words long), so clearly, the language model used by ULMFiT fine-tunes really well for long sequences. Tweets, on the other hand, have a hard-limit of 140 characters, which are rather small sequences compared to full sentences from a movie review or a news article.</p><p>As described in the <a href="https://arxiv.org/pdf/1801.06146.pdf">ULMFiT paper</a>, the classifier uses “concatenated pooling” to help identify context in long sequences when a document contains hundreds or thousands of words. To avoid loss of information in case of really long sequences, the hidden state at the last time step is concatenated with both the max-pooled and mean-pooled representation of the hidden states, as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/1*HfCMHarpHUETFERwTRPNow.png" /><figcaption>Concatenated pooling: H is the vector of all hidden states</figcaption></figure><p>In this study, no changes were made to the model architecture for either method— however, it could be that changing the representation of the hidden state using just one of the two (max or mean) representations could help ULMFiT generalize better to smaller sequences such as this case with Tweets.</p><p>GPT in its original form (with the classification task head) seems to have no trouble generalizing to Tweet syntax even though it was pre-trained on a book corpus that also had possibly long sequences. It could be that the transformer’s self-attention mechanisms and high-dimensional self-attention layers are capable of adapting to varying sequence lengths while learning aspects of Tweet syntax better than the LSTMs with concat pooling in the hidden layer. These are profound concepts and could be studied in greater detail.</p><h4>Effect of Language Model Tuning</h4><p>In ULMFiT, language model fine-tuning is a necessary step before training the classifier. Including an augmented Tweet vocabulary during the language model fine-tuning step seemed to provide the model a better understanding of Tweet syntax, which seemed to improve its performance. Varying the learning rates, momentum and dropouts did have a minor, but negligible overall impact on the F-score during fine-tuning.</p><p>In OpenAI GPT, the language model is fine-tuned simultaneously with the classifier using a weighting parameter, through the auxiliary language model objective specified as per the below equation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/253/1*lshCyd-V8owaZHLeF7uQOA.png" /></figure><p>When the language model fine-tuning objective was turned off (i.e. lambda=0) in the case of the transformer, the averaged macro F-score became markedly <strong>worse </strong>(below 0.6). We can thus reason that the pre-trained model’s weights on their own do not sufficiently capture the syntax of Tweets, and that language model fine-tuning does help our model generalize better to Twitter data. Increasing lambda to a very high value (5) or a very low value (0.1) also did not improve the F-scores in our experiments, so there could be something specific to our Twitter data that makes the lambda value of 0.5 an optimum one. It would be interesting to see whether other values of the LM coefficient (lambda) are favourable when applied on a completely different dataset.</p><h4>Effect of Training Data Size</h4><p>ULMFiT was able to produce good classification performance on <em>each</em> individual topic (for which we had less than 500 samples to train on). In fact, our best result using ULMFiT was obtained (albeit with some data augmentation) by training<em> five distinct</em> <em>classifiers</em> on a per-topic basis. This makes sense because ULMFiT has been shown to produce <a href="http://nlp.fast.ai/classification/2018/05/15/introducting-ulmfit.html">excellent transfer learning performance on very limited training samples</a> (as few as 100!). Hence, in cases where we have a very small number of labelled training samples, ULMFiT would be a good option to attempt transfer learning, at least for classification tasks.</p><p>The transformer on the other hand, requires at least a few thousand training samples to generalize well and avoid over-fitting. The OpenAI paper shows good results for a range of tasks which have <a href="https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf">anywhere from<strong> 5,000</strong> to <strong>550,000</strong> training samples</a>. The transformer has 786 dimensions in the self-attention heads and 3,072 dimensional inner states in the feed-forward networks; hence when we have fewer than 1,000 training samples, the model with its high dimensionality seems to memorize the training data and massively over-fit the data (with validation accuracy dropping well below 70%).</p><h4>Effect of pre-training language model and architecture</h4><p>Both language models were pre-trained on different text corpora (Wikitext 103 and Google Books). Also, ULMFiT used 3-layer bi-LSTM architecture whereas OpenAI used a transformer network. This Tweet by Yoav Goldberg on the topic provides some food for thought.</p><h3>(((ل()(ل() &#39;yoav)))) on Twitter</h3><p>Strong empirical results, but I wish there was more focus on a proper comparison / controlled experiments. Is the improvement due to LSTM -&amp;gt; Transformer or due to Wiki-1B (single sents) -&amp;gt; BooksCorpus (longer context)? https://t.co/L3WrJW3z12</p><p>Based on our Tweet stance classification results, it appears that the transformer seems to have a benefit when it comes to learning case-specific syntax with limited context from a relatively small number of training samples — this could be more to do with the transformer architecture than the pre-trained language model used. However, until a more consistent language model is available on both architectures, it is relatively hard to tell which of these had a bigger impact on the results.</p><h3>Conclusions</h3><p>In this project, we studied the techniques used in two powerful transfer learning approaches (ULMFiT and OpenAI GPT) for a novel task from a different distribution (stance detection of Tweets). We developed a training and classification pipeline for two separate PyTorch-based frameworks and compared the macro F-scores for stance evaluation with the best results for this task from MITRE in 2016.</p><p>Both methods achieved good performance and good generalization with minimal customization of the model, and fine-tuning was achievable in reasonable time (about an hour overall for either method on a single Tesla P100 GPU). Both methods have also been shown to make transfer learning very easy and achievable with a relatively small learning curve and relatively few additional lines of code.</p><p>While we were able to obtain better F-scores with OpenAI GPT across nearly all topics in this case, it does not imply that a transformer-based model is <em>always</em> a better tool for such classification tasks. This could depend very much on the nature of the data and the language models used, as well as the model architecture. There <strong><em>could be many better ways to fine-tune either model </em></strong>for classification, and better hyper-parameter selection (or model customization) could help further improve the results — it just wasn’t possible to try them all for this project.</p><p>It appears that the task-agnostic and highly parallelized architecture of the transformer in GPT allows it to easily achieve rapid generalization in just 2–3 epochs of training for this Tweet stance task; however, GPT is prone to over-fitting, especially when we have fewer than a thousand training samples. ULMFiT is definitely much better at achieving a good performance with very small datasets (of fewer than 500 training samples); however it seems to require augmenting the vocabulary of the training data during the language model fine-tuning stage (considering that this specific Tweet classification task contained data that was <em>very</em> different from the pre-trained language model).</p><p>Overall, this is a very exciting time to be studying deep learning in NLP, and the advent of powerful transfer learning techniques such as these will hopefully open up deep learning applications to a <em>much</em> wider group of practitioners in the near future!</p><p><strong>EDITED on March 24th, 2019</strong>: Since the release of <a href="https://openai.com/blog/better-language-models/">GPT-2</a> by OpenAI in February 2019, it becomes more important to distinguish between various versions of transformer models, so the previous mentions of “OpenAI Transformer” are now updated to “GPT”, which is the transformer model from their 2018 paper.</p><h4>Acknowledgements</h4><p>This work was done in collaboration with <a href="https://www.linkedin.com/in/andrew-wesson-7079728b/">Andrew Wesson</a> and <a href="https://www.linkedin.com/in/abhisheksunnak/">Abhishek Sunnak</a>, who contributed significantly to idea generation as well as data cleaning and training code.</p><p>Also, if you liked this article, please connect with me on <a href="https://www.linkedin.com/in/prrao87/">LinkedIn</a> and <a href="https://twitter.com/tech_optimist">Twitter</a>!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ab014da8dde" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/transfer-learning-in-nlp-for-tweet-stance-classification-8ab014da8dde">Transfer Learning in NLP for  Tweet Stance Classification</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>