<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sooraj Subrahmannian on Medium]]></title>
        <description><![CDATA[Stories by Sooraj Subrahmannian on Medium]]></description>
        <link>https://medium.com/@soorajsubrahmannian?source=rss-958c810106de------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*5Ao2iUNtXCPki_8T.</url>
            <title>Stories by Sooraj Subrahmannian on Medium</title>
            <link>https://medium.com/@soorajsubrahmannian?source=rss-958c810106de------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 13:52:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@soorajsubrahmannian/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Contextual Brand Safety -II]]></title>
            <link>https://medium.com/gumgum-tech/contextual-brand-safety-ii-47c08f4bfe4f?source=rss-958c810106de------2</link>
            <guid isPermaLink="false">https://medium.com/p/47c08f4bfe4f</guid>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[transfer-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Sooraj Subrahmannian]]></dc:creator>
            <pubDate>Tue, 19 May 2020 03:42:38 GMT</pubDate>
            <atom:updated>2020-05-27T17:54:39.667Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Contextual Brand Safety Cover picture" src="https://cdn-images-1.medium.com/max/1024/1*1X5k46ZGK4zDdcT-2p9Xvg.jpeg" /></figure><blockquote>Contextual brand safety is an ongoing series. This is the second blog in this series. Through this series, we talk about steps to be taken to do <strong>multi-label text classification</strong> in the industry. This blog post talks about model training and evaluation.</blockquote><h3>1. Introduction</h3><p>Brand safety is an important offering of GumGum. <a href="https://medium.com/gumgum-tech/contextual-brand-safety-solution-i-b25b12053b65">Contextual Brand Safety-I</a> talks about the problem and data preprocessing techniques in depth. In this blog post, we will discuss model training, evaluation and steps to production.</p><h3><strong>2. Experimental setup</strong></h3><p>We<strong> </strong>set up a multi-step <a href="https://www.mlflow.org/">mlflow</a> project tracking system to track and store artifacts across each step i.e,</p><ol><li>Data loading and preprocessing (1 step)</li><li>Model training (1 step for each layer)</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tTyxg23o6Ow3ThNZxSHwOw.png" /><figcaption>Model training ecosystem</figcaption></figure><p>We used <a href="https://www.wandb.com/">Weights &amp; Biases</a> for hyperparameter tuning and better visualization. We used an AWS GPU enabled EC2 instance, as a compute resource for running parallel experiments, connected to an AWS EFS volume to store our experiment results. We used <a href="https://www.fast.ai/">fastai</a> and <a href="https://pytorch.org/">Pytorch</a> framework for rapid prototyping and experimentation.</p><p>Although we tried and tested several machine learning and deep learning models, we will talk about the two best models; <strong>ULMFiT</strong> and <strong>BERT</strong> and the experiments conducted with them.</p><ul><li><a href="https://arxiv.org/abs/1801.06146">Universal Language Model Fine-tuning for Text Classification</a></li><li><a href="https://arxiv.org/abs/1810.04805">BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding</a></li></ul><h3>3. Experiments with ULMFiT</h3><h4>3.1 Language model experiments:</h4><p>ULMFiT is an LSTM based language model which has broadly three steps:</p><ol><li>Pretraining language model on Wikipedia</li><li>Fine-tuning language model on production data</li><li>Training a classifier on production data</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IKni3ytV6aBKrGFHOsWmeA.png" /><figcaption>ULMFiT approach towards language modeling</figcaption></figure><p>In all our experiments, we used the language model pretrained on <a href="https://www.salesforce.com/products/einstein/ai-research/the-wikitext-dependency-language-modeling-dataset/">Wikitext 103</a>. We conducted experiments while fine tuning the language model with two different hyperparameters:</p><ol><li><strong>drop out multiplier: </strong>This is a multiplier to the dropouts that are default with the model</li><li><strong>vocabulary size:</strong> The number of words used in the vocabulary used to build the model</li></ol><h4><strong>Drop out Multiplier</strong></h4><p>We conducted experiments with two different corpus sizes namely 100,000 and 250,000 records. The plot below shows the variation of Language model performance with respect to the variation of dropout multiplier</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-LPYYPhQa4JfTFxnKYwLJA.png" /></figure><p>In the plot above, we can observe the following things:</p><ol><li>When dropout is increased, with different data sizes, we observe degradation in performance</li><li>We also observe that, with a higher amount of data, the degradation slows down</li></ol><p>Even though, we can see that with higher data size, there is a perceivable difference in performance. Since the evaluation set for both 100k and 200k are different but from the same distribution, it is inconclusive whether there is any difference and if there is by how much.</p><p><strong>Vocabulary size<br></strong>We conducted another experiment to understand the variation of LM performance with respect to data size. The plot below shows the variation of Language model performance with respect to the variation of vocabulary size.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z7U5BmAxn52wmQD2Q44bCg.png" /></figure><p>In the plot above, we can observe the following things:</p><ol><li>With a corpus size of 100K, we observe that performance increase initially but then decrease gradually whereas with a corpus size of 250K, performance decrease monotonically with respect to vocabulary size.</li></ol><p>Increase in vocabulary can be seen as an increase in complexity, therefore, there was an initial increase in LM performance but it later decreased due to overfitting. But in the case of corpus size of 250K, with a large amount of data, the model is able to generalize with fewer features aka small vocabulary.</p><p><strong>3.2 Experiments with Classifier</strong></p><p>We conducted a total of 104 experiments while training our classifier in stage 3 of language model development. During this stage, we used bayesian hyperparameter tuning to come up with the best set of hyperparameters for the task at hand.</p><ol><li>Language model encoders</li><li>Final layer dropouts</li><li>hidden layer size in the classifier</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GIGiwrUvduoQ9aUaopZ4bA.png" /><figcaption>Variation of brand safety performance with a change in drop out, classifier hidden layer size and encoder</figcaption></figure><p><em>Note: Encoder naming convention: An language model trained with a dropout multiplier of </em><strong><em>0.3</em></strong><em> is named as fwd_enc_fp16_0.3_AWD</em></p><p><strong>Key observations</strong></p><ol><li>An encoder trained with a higher value of dropout multiplier has the highest performance when trained with a lower classifier dropout and vice versa</li><li>The best performing encoder may or may not be a part of the best performing classifier.</li></ol><p>Keep diverse equally performing encoders from the finetuning stage and include them as a hyperparameter tuning of the classifier to get the best performance.</p><blockquote>Characteristics of the dataset, encoder and classifier hyperparameter need to be tuned together because even though we develop the classifier in 3 steps. Each step is dependent on the outcomes of previous steps.</blockquote><h3><strong>4. Comparison of ULMFiT Vs BERT</strong></h3><p>We used <a href="https://github.com/kaushaltrivedi/fast-bert">fastbert</a> library for our experimentation. BERT model was fine tuned using the same dataset. Unlike ULMFiT, BERT does not need the unsupervised fine tuning, the fine-tuning in BERT is equivalent to supervised training in ULMFiT. We compared the performance of the best ULMFiT against BERT on two different datasets, one a multi-label dataset that consists of labels not only to distinguish between safe and unsafe but also to detect different types of threat in articles as per the nomenclature in <a href="https://medium.com/gumgum-tech/contextual-brand-safety-solution-i-b25b12053b65">Series-I</a>.</p><p><strong>1.Multi-label performance (Micro-averaged)</strong></p><p>This dataset is less imbalanced with respect to all the classes and labelled for all types of threatening content</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_2Qr4nCMFv1H2_XNuhoSTw.png" /><figcaption>dataset-1: Multilabel threat</figcaption></figure><p><strong>2.Binary performance</strong></p><p>The performance was calculated on a dataset that mimics our production distribution. Only 10%–15% of the entire corpus has threatening content.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Uca9LD8-FK2FSY3X1WGCvA.png" /><figcaption>dataset-2: Binary threat</figcaption></figure><p>In the dataset-1, we can see that BERT has a better recall, which is good for our task but the recall is worse on the production dataset (dataset-2). Even though BERT being the state of the art (SOTA) model, it is still unable to outperform ULMFiT. The performance could be said to be similar if not worse. This trend could be attributed to the following reasons:</p><ol><li>Transformers outshine LSTMs on medium to long text because they are able to capture the longer-term dependencies. The median text length of our production traffic is about 50 words, which is not quite long. LSTMs are able to handle it pretty well.</li></ol><p>2. ULMFiT encoders were fine tuned in an unsupervised way to capture the language differences between its pretrained corpus, Wiki-103 and the production traffic whereas BERT was not given any such special treatment.</p><p><strong>Memory footprint &amp; Throughput comparison</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/534/1*XWMTimjmNR2C__T92BHiIA.png" /></figure><p>Throughput of a system is measured in the number of completed messages per second. ULMFiT is about 10 times faster than BERT. This means that deploying BERT in production will cost 10 times more than ULMFiT.</p><blockquote>Both ULMFiT and BERT were quantized before throughput measurement.</blockquote><p>The memory footprint of BERT is about 2x more than ULMFiT. This is also a cause of concern because when we asynchronously process messages as a batch, we could increase the size of the batch to scale the system better. Thus, having a model with lesser memory footprint will help in scaling.</p><p><strong>Key takeaways:</strong></p><ol><li>By following the fine tuning procedure as per the ULMFiT paper, we were able to achieve performance on par with state of the art models such as BERT.</li><li>Quantized of ULMFiT has a faster inference time than BERT.</li></ol><h3>Future work:</h3><ol><li>Performance of BERT could be improved by unsupervised finetuning on production corpus before supervised training.</li><li>Explore distillation to speed up BERT inference with minimal loss in quality, such as <a href="https://arxiv.org/abs/1910.01108">DistilBERT</a>.</li><li>Amortization of BERT encoder cost by using multi-task learning using BERT as a centralized encoder for all NLP downstream tasks.</li></ol><h3>Conclusion</h3><p>In this blog post, we have discussed the different experiments we conducted and the steps we took to achieve a model that has state of the art performance in quality and which also scales well. In the next blog post under this series, we will discuss the steps we took for putting this model in production. In the meantime, check out my previous blog on contextual brand safety to understand the task better.</p><p><a href="https://medium.com/gumgum-tech/contextual-brand-safety-solution-i-b25b12053b65">Contextual Brand Safety solution- I</a></p><p>STAY TUNED…</p><p><strong>About Me:</strong> Graduated with a Masters in Data Science from the <a href="https://medium.com/usf-msds">University of San Francisco</a>. I am an NLP Scientist at <a href="https://medium.com/u/9d14b5de3c57?source=post_page-----b25b12053b65----------------------">GumGum</a>. Interested in the application of NLP and Speech.</p><p>We’re always looking for new talent! <a href="https://gumgum.com/careers">View jobs</a>.</p><p>Follow us: <a href="https://www.facebook.com/gumguminc">Facebook</a> | <a href="https://twitter.com/gumgum">Twitter</a> | | <a href="https://www.linkedin.com/company/167379/">Linkedin</a> | <a href="https://www.instagram.com/gumguminc/">Instagram</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=47c08f4bfe4f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gumgum-tech/contextual-brand-safety-ii-47c08f4bfe4f">Contextual Brand Safety -II</a> was originally published in <a href="https://medium.com/gumgum-tech">GumGum Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Contextual Brand Safety solution- I]]></title>
            <link>https://medium.com/gumgum-tech/contextual-brand-safety-solution-i-b25b12053b65?source=rss-958c810106de------2</link>
            <guid isPermaLink="false">https://medium.com/p/b25b12053b65</guid>
            <category><![CDATA[threat-detection]]></category>
            <category><![CDATA[data-preprocessing]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[brand-safety]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Sooraj Subrahmannian]]></dc:creator>
            <pubDate>Wed, 15 Apr 2020 17:15:16 GMT</pubDate>
            <atom:updated>2020-04-15T17:15:16.895Z</atom:updated>
            <content:encoded><![CDATA[<h3>Contextual Brand Safety - I</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/765/1*rstsO_2N-7dE62SEorOUlw.png" /></figure><blockquote>Contextual brand safety is an ongoing series. This is the first blog in this series. Through this series, we talk about steps to be taken to do <strong>multi-label text classification</strong> in the industry. This blog post sets the stage by talking about the problem and data collection.</blockquote><h3>Introduction</h3><p>GumGum is dedicated to ensuring a brand-safe environment to all our clients; advertisers and publishers alike. In order to implement brand safety, we have a variety of methods which assist in ensuring that we deliver ads on safe, relevant and high-quality content.</p><p>One of the most important measures taken to ensure brand safety is <strong>Publisher vetting</strong>. We ensure that each new content under every publisher is validated against our brand safety guidelines, where the publisher content is filtered in such a way that it does not consist of the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/278/1*H6zaYisdbSHlf5P8MI2J5g.png" /></figure><p>The prevalence of threatening content is <strong>less than</strong> <strong>10%</strong> of our entire traffic. The problem is also multi-modal. Each page consists of text, images and videos. Hence we tackle this problem with textual and visual intelligence. In this blog post, we will discuss in detail about the evolution of our textual intelligence in the context of brand safety.</p><p>To add fuel to the fire, the scale of this problem makes it even harder. We process about <strong>40 million unique pages</strong> a day. Therefore, the solution needs to scale as well.</p><h3>Goals &amp; metrics</h3><p>Our goal from a modelling perspective is three folds:</p><ol><li>Identify all the threatening content in our traffic.</li><li>Reduce the amount of inventory loss due to a safe page tagged as unsafe (False positives)</li><li>Scalability</li></ol><p>To achieve the goals mentioned above, it&#39;s crucial to evaluate the system using the right metrics. Hence we look at <a href="https://en.wikipedia.org/wiki/Precision_and_recall"><strong>Recall</strong></a> and <a href="https://en.wikipedia.org/wiki/False_positive_rate"><strong>FPR</strong></a><strong> </strong>to evaluate our goals 1 and 2 respectively.</p><p>We use <strong>Requests per second (RPS)</strong>, the number of input requests completely processed in one second, to understand the scalability of the model.</p><h3>Previous work</h3><p>The previous system was developed based on <a href="https://en.wikipedia.org/wiki/Explicit_semantic_analysis">Explicit Semantic Analysis</a> which uses keyword scores derived from Wikipedia to understand whether a page is threatening or not.</p><p><strong>Pros: </strong>The technique is pretty fast during inference because we only have to lookup in a dictionary of keyword scores.</p><p><strong>Cons: </strong>Domain drift from our publisher content with respect to <a href="http://n.wikipedia.org/wiki/Main_Page">Wikipedia</a> causes many ambiguities to creep in. A keyword-based approach often fails at capturing context.</p><blockquote>Why does context matter?</blockquote><p>Let us understand that with an example given below. In the example, the article talks about the impact of coronavirus in several industries. This is not necessarily a threatening article. But if we chose to tag all the articles mentioning COVID/coronavirus we might end up tagging a lot of informative good articles as threatening.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IgikNjWpxeF5oqL3NduAUQ.png" /></figure><blockquote>It is not the keyword that is threatening, it is the context in which the word decides whether an article is threatening or not.</blockquote><p>A study by <a href="https://medium.com/u/9d14b5de3c57">GumGum</a> has shown that 60% of articles that contains COVID related keywords are safe.</p><p><a href="https://www.mediapost.com/publications/article/349892/study-60-of-content-related-to-covid-19-consider.html">Study: 60% Of Content Related To COVID-19 Considered Brand Safe</a></p><p>In addition to context, like any other legacy system, this system had a lot of features and thresholds which makes it difficult to maintain or do any further development on it.</p><p>To alleviate the above problems, we decided to look into language models. As of today, there are a variety of language models. But the two types of language models we explored are:</p><ol><li>LSTM-based — <a href="https://nlp.fast.ai/classification/2018/05/15/introducing-ulmfit.html"><strong>ULMFiT</strong></a></li><li>Transformer-based — <a href="https://arxiv.org/abs/1810.04805"><strong>BERT</strong></a></li></ol><p>We decided to go depth-wise rather than breadthwise in terms of research. Thus, we explored ULMFiT in-depth and compared it with our existing system and the current state-of-the-art BERT.</p><h3>Data collection and preprocessing</h3><p>Data collection is a crucial step. Since we are interested in language models, we need data for two tasks:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/461/1*f1eTKzcJhXfbWrQqbHShFQ.png" /><figcaption>Data requirements for language modelling</figcaption></figure><p>At GumGum, we log all our data for the past couple of months in S3 which is about a couple of billions of documents.</p><blockquote>How do we select representative good quality samples for training and testing?</blockquote><p>To be able to sample more diverse, representative and good quality data for training and testing, we need to enrich the collected data. The data collected above is run through the following steps in <a href="https://spark.apache.org/docs/latest/api/python/index.html">Pypspark</a> using <a href="https://databricks.com/">Databricks</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/509/1*yP1qyxxLK1TA7aavcgSdxA.png" /><figcaption>Data enrichment process</figcaption></figure><ol><li>Exact deduplication based on text</li><li>Filtering based on text-based statistical metrics such as <a href="https://en.wikipedia.org/wiki/Flesch%E2%80%93Kincaid_readability_tests">readability score</a>, <a href="https://en.wikipedia.org/wiki/Lexical_density">type-to-token ratio</a> and text length</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WKSpWUq5F5mYLc_QdpfzTg.png" /></figure><blockquote>Negative values of readability imply bad quality text but positive readability scores does not always imply good text.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ikDWtt-UCO39ici7gPfMDw.png" /><figcaption>Effect of type-to-token ratio score</figcaption></figure><blockquote>Extremely high/low values of type-to-token ratio give you garbage text.</blockquote><p>3. Near duplicate detection and removal using <a href="https://mattilyra.github.io/2017/05/23/document-deduplication-with-lsh.html">MinHash and LSH</a></p><p>To avoid data leakage, training and testing data are separated across time. We chose October to January for training and February to March for testing.</p><h3>Data sampling</h3><p>Different sampling technique is employed for unsupervised and supervised training. Unsupervised training is to perform <a href="https://en.wikipedia.org/wiki/Domain_adaptation">domain adaptation</a> which helps the encoder to learn the nuances in the target domain (production data) that are different from the source domain (Wikipedia). For this purpose, we sampled about 250,000 articles using <strong>restricted random sampling </strong>(randomly sampled from the entire data but with a cap on the number of samples from <a href="https://en.wikipedia.org/wiki/Top-level_domain">top-level domains</a>) to limit the number of contents from an over-represented domain in the enriched data.</p><p>In supervised training, the samples used to train must be diverse, representative and also hard for the model to train. To incorporate difficulty of samples, we move towards active learning approaches in multi-label classification. We have employed different active learning methods such as:</p><ul><li>Buckets based on<strong> confidence score quantiles:</strong> <br>Here we sample from each of the score buckets for all the classes iteratively as shown below such that previously imbalanced dataset becomes more balanced. This naive sampling has proven to be effective in our experiments probably because the new and the previous classifiers were very different and there is a lot more to learn from the previous classifier. The cons of this sampling are that it does not capture the co-occurrence probability of classes well and does not have a direct notion of hard samples.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/427/1*t3mU3qwPaFiXizKpA3NADw.png" /><figcaption>Buckets of score quantiles</figcaption></figure><ul><li>Uncertainty sampling based on <a href="https://www.sciencedirect.com/science/article/abs/pii/S0925231217313371?via%3Dihub"><strong>CVIRS</strong></a> (Category Vector Inconsistency and Ranking of Scores). This paper extends uncertainty sampling to multi-label classification problems. It uses ranking as well as entropy-based measures to calculate uncertainty. This is effective when we have to improve the classifier with the same architecture over and over again.</li></ul><p>There are many more methods out there. A complete comparison and understanding of these methods deserve another blog post.</p><p>Next step is to get the sampled data annotated and use truth inferencing techniques to obtain the ground truth. This step is done iteratively with multiple annotators until the required quality is achieved.</p><h3><strong>Conclusion</strong></h3><p>In this blog post, we have discussed different techniques to enrich text data and sample data for training and testing. We will discuss the experiments we conducted and the steps we took to put the new model in production in the next blog.</p><p>STAY TUNED…</p><p><strong>About Me:</strong> Graduated with a Masters in Data Science from the <a href="https://medium.com/usf-msds">University of San Francisco</a>. I am an NLP Scientist at <a href="https://medium.com/u/9d14b5de3c57">GumGum</a>. Interested in the application of NLP and Speech.</p><p>We’re always looking for new talent! <a href="https://gumgum.com/careers">View jobs</a>.</p><p>Follow us: <a href="https://www.facebook.com/gumguminc">Facebook</a> | <a href="https://twitter.com/gumgum">Twitter</a> | | <a href="https://www.linkedin.com/company/167379/">Linkedin</a> | <a href="https://www.instagram.com/gumguminc/">Instagram</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b25b12053b65" width="1" height="1" alt=""><hr><p><a href="https://medium.com/gumgum-tech/contextual-brand-safety-solution-i-b25b12053b65">Contextual Brand Safety solution- I</a> was originally published in <a href="https://medium.com/gumgum-tech">GumGum Tech Blog</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Learn to Find Topics in a Text Corpus]]></title>
            <link>https://medium.com/@soorajsubrahmannian/extracting-hidden-topics-in-a-corpus-55b2214fc17d?source=rss-958c810106de------2</link>
            <guid isPermaLink="false">https://medium.com/p/55b2214fc17d</guid>
            <category><![CDATA[topic-modeling]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[lda]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[gensim]]></category>
            <dc:creator><![CDATA[Sooraj Subrahmannian]]></dc:creator>
            <pubDate>Mon, 16 Apr 2018 23:49:41 GMT</pubDate>
            <atom:updated>2018-07-19T23:59:54.610Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bQIpNAeHdTPluRT-2J6edg.jpeg" /></figure><p>Be it customer reviews, news articles or conversations between people, when we are tasked with the ordeal of having to figure out what the corpus is about, it is impossible to manually read and summarize them. Topic modeling is a natural language processing technique that extracts latent topics from a corpus of documents. Unlike a classification problem, there are no labels directing this process, hence it is unsupervised. There are many algorithms that perform topic modeling. The most important ones are:</p><ol><li><strong>Latent Dirichlet Allocation (LDA)</strong></li><li><strong>Non Negative Matrix Factorization (NMF)</strong></li></ol><p>In this blog, we will restrict our discussion to topic modeling using LDA.</p><h3><strong>Understanding LDA</strong></h3><p>Let us explore how LDA works.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/623/1*Bp4PNIeW2rNmAU7mVe1grg.png" /><figcaption>Understanding LDA</figcaption></figure><p>The above chart shows how LDA tries to classify documents. Documents are represented as a distribution of topics. Topics, in turn, are represented by a distribution of all tokens in the vocabulary. But we do not know the number of topics that are present in the corpus and the documents that belong to each topic. In other words, we want to treat the assignment of the documents to topics as a random variable itself which is estimated from the data. This sounds complicated, but the process we have discussed above is similar to the Dirichlet process. Curious readers can read the following article to understand the Dirichlet process in detail.</p><p><a href="http://phyletica.org/dirichlet-process/">The Dirichlet process for dummies (i.e., biologists, like me)</a></p><p>Hence, LDA makes a prior assumption that the (documents, topics) and (topics, tokens) mixtures follow Dirichlet probability distributions. This assumption encourages documents to consist mostly of a handful of topics, and topics to consist mostly of a modest set of the tokens.</p><p>Since topics are learned from the model, these topics are formed based on some underlying concepts buried deep in the data.</p><p>Now that we have an understanding of how LDA works, let’s see how it is implemented. The corpus is converted into a bag of words as given below in Fig-1. This when passed through a topic modeling algorithm such as LDA, we identify the following two things:</p><ol><li>The distribution of words in each topic</li><li>The distribution of topics in each document</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K4y2WbiUo_glULHqB4dy3Q.gif" /><figcaption>Fig-1: General flowchart for topic modeling</figcaption></figure><p>From the figure above, it can be easily understood that topic models solve the problem of a large sparse matrix. Imagine we have n documents and m terms in our vocabulary. Now let us compare conventional algorithms against topic models:</p><ol><li>Conventional models: n x m variables</li><li>Topic models: k x (n+m) variables where k &lt;&lt; n&lt;&lt;m where k is the number of topics</li></ol><p>Each document in a corpus can be as short as a single sentence (a tweet) or as long as a chapter in a book.</p><blockquote>Typically a corpus with very short documents tends to be more difficult to build coherent(interpretable) models over than the one with slightly longer documents.</blockquote><p>Let’s get our hands dirty….</p><h3>LDA Implementation</h3><h4>Step-1: Data preprocessing with SpaCy</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MSCewWY6YZvvcFf-5wj9iA.gif" /><figcaption>Fig-2: General flowchart for data processing</figcaption></figure><p>For the purpose of this blog post, I have scraped about 12,000 resumes from the web. I have masked all personal identifiers from the dataset considering its sensitivity. In this dataset, we will be looking at the skills of candidates who are interested in data science and try to understand what clusters of skill sets exist. Please grab the data from <a href="https://github.com/smsubrahmannian/Topic-Modeling/blob/master/data/input_data.csv">this </a>link.</p><p>Let’s look at the snapshot of the resume data below.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fairtable.com%2Fembed%2FshrGb9KVidqxGWeot&amp;url=https%3A%2F%2Fairtable.com%2FshrGb9KVidqxGWeot&amp;image=https%3A%2F%2Fstatic.airtable.com%2Fimages%2Foembed%2Fairtable.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=airtable" width="800" height="533" frameborder="0" scrolling="no"><a href="https://medium.com/media/3aedbdb7f1c50065f43cb01fa1e9e187/href">https://medium.com/media/3aedbdb7f1c50065f43cb01fa1e9e187/href</a></iframe><p>As we can see, the data is very messy. I preprocessed the data using SpaCy. If you are not familiar with SpaCy, I have created an introductory tutorial in <a href="https://github.com/smsubrahmannian/Topic-Modeling/blob/master/code/Preprocessing%20with%20spacy.ipynb">this</a> link. The preprocessing must be customized to a dataset. Our processed data must represent each document in the corpus as a list of words, that represent the ideas that you want to capture from this document.</p><h4>Step-2: Topic modeling</h4><p>Post preprocessing, the corpus was converted into a ‘bag of words’ as shown in Figure 2. I have previously analyzed LDA with Sklearn and Gensim packages in Python. The latter is capable of running on multiple cores and hence it runs much faster. A full listing of the code from preprocessing texts to generating final topic model is given below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c8d665b06d674f85ab344c5145039964/href">https://medium.com/media/c8d665b06d674f85ab344c5145039964/href</a></iframe><h3><strong>Selecting a topic model &amp; topics within it</strong></h3><p>In the example code given above, the number of topics was set to <strong>12</strong>. But how did I chose this magic number 12? The hardest and the most elusive part of topic modeling is the selection of the number of topics. The most popular way of tackling this issue is by eyeballing the topics derived from models trained with different numbers of topics. The most sensible one is selected. But what is ‘sensible’ differs from one person to another. The use of such approaches often leads us to wrong conclusions about the corpus.</p><p>In this section, I will summarize two important techniques used to evaluate topic models, i.e,</p><ol><li>Perplexity (might not be such a great measure)</li><li>Topic coherence</li></ol><p><strong>Perplexity</strong></p><p>Given a trained model, perplexity tries to measure how this model is surprised when it is given a new dataset. This is measured as the normalized log-likelihood of the held out test set. The lower the perplexity, the better the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/694/1*nb_7a1IXhasz7AyZn45QAA.png" /></figure><p>The first equation calculates the log-likelihood; the probability of observing some unseen data given a model learned earlier. This checks whether the model captures the distribution of the held out set. If it doesn’t then the perplexity is very high suggesting that model is bad. However, studies have shown the following:</p><blockquote>Perplexity is not strongly correlated to human judgment and, even sometimes slightly anti-correlated.</blockquote><p>This is a good example of when a popular metric doesn’t fit the business requirement. This is why there is a focus on topic coherence.</p><p><strong>Topic coherence</strong></p><p>Here we quantify the coherence of a topic by measuring the degree of semantic similarity between its high scoring words. These measurements help distinguish between topics that are human interpretable and those that are artifacts of statistical inference. To compute topic coherence of a topic model, we perform the following steps.</p><ol><li>Select the top n frequently occurring words in each topic</li><li>Compute pairwise scores (UCI or UMass) for each of the words selected above and aggregate all the pairwise scores to calculate the coherence score for a particular topic.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/536/1*263cuLYypfzglrarDygNvg.png" /></figure><p>3. Take a mean of the coherence score per topic for all topics in the model to arrive at a score for the topic model.</p><p>To understand more about the score function used in evaluating topic coherence, please read the article given below.</p><p><a href="http://qpleple.com/topic-coherence-to-evaluate-topic-models/">Topic Coherence To Evaluate Topic Models</a></p><p>Using the concept mentioned above, we can shed some light on the main issues:</p><ol><li><strong>Topic model evaluation: to estimate the number of topics in a corpus</strong></li></ol><p>We can evaluate average coherence score per topic for a range of models trained with a different number of topics by following the steps above. The number of topics for which the average score plateaus, as shown in the figure below, is the sweet spot we are looking for.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*TGNLNGoBe_2BbWY5GjzXag.png" /></figure><p>As you can see, the score plateaus around 12–14 topics. So our best guess for the number of topics is in this range. The code for this step can be found on my <a href="https://github.com/smsubrahmannian/Topic-Modeling/blob/master/code/Preparing%20a%20Topic%20model.ipynb">Github</a>.</p><p><strong>2. Topic evaluation: automated selection of important topics</strong></p><p>Once we select a topic model, say with a number of topics as 12, the next step is to distinguish between human interpretable topics and jargons of statistical analysis. We use the same concept as above. We calculate the average coherence score per topic with the same coherence measure.</p><p>Higher the topic coherence, the topic is more human interpretable. We can order the topics based on their decreasing magnitude of topic coherence scores to obtain the most human interpretable topics.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fairtable.com%2Fembed%2FshrbuXenTt5CNMdfj&amp;url=https%3A%2F%2Fairtable.com%2FshrbuXenTt5CNMdfj&amp;image=https%3A%2F%2Fstatic.airtable.com%2Fimages%2Foembed%2Fairtable.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=airtable" width="800" height="533" frameborder="0" scrolling="no"><a href="https://medium.com/media/f8d2f8333f662449cd9be823739568d9/href">https://medium.com/media/f8d2f8333f662449cd9be823739568d9/href</a></iframe><p><strong>How do we decide on a threshold for our coherence score?<br></strong>This is more of an art than science. Normally, we can see a huge drop in coherence score after a particular topic. In the table above, the drop happens after the first 7 topics. In addition to the above, apart from having a good coherence score, a topic must have a significant amount of tokens/words. In our case, the token percentage falls below 10% after the fourth topic from the top (described in the next section).</p><p>Here, we can see that topics 4, 11 and 8 have very less topic coherence score. Hence they can be discarded but topics 9, 10, 12, 7 have good coherence score. They are more interpretable topics compared to the rest. The code for this section can be found on my <a href="http://nbviewer.jupyter.org/github/smsubrahmannian/Topic-Modeling/blob/master/code/Intrepreting%20Topic%20model.ipynb">GitHub</a>.</p><p>So far we have learned how to build a topic model, how to evaluate and get the best topic model and how to evaluate the topics within the selected topic model. Now let’s proceed to understand these topics through visualization.</p><h3>Interactive visualization of topics using pyLDAviz</h3><p>In the previous section, we have defined each topic with top 10 frequently occurring words in that topic. As you would have noticed, the caveat with the above approach is that often a term that is important to one topic might be important to others topics as well. Such words don’t help us in differentiating the topics from one another. For example, in this dataset, since we are talking about data science skills, words such as ‘python’, ‘sql’ etc will find its way into almost all topics.</p><h4>How do we draw out relevant words with respect to a specific topic?</h4><p>We want terms that are not just important for a topic, but it should also be able to distinguish this topic from the rest of the lot. In this context, I introduce the concept of relevance.</p><blockquote>The relevance of a term w.r.t to a topic is defined as weighted sum of the probability of word w given a topic k and the lift. Lift for a word w.r.t to a topic is given by the ratio of the probability of word w given a topic k and the probability of the word in the entire corpus.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/587/1*zPtDgwlUxWnWY8NmOZU2DA.png" /></figure><p>The first term in the RHS is proportional to the frequency of the word occurrence in that topic. Therefore, if a word frequency is high on multiple topics, then the word doesn&#39;t help to describe or distinguish the topic really well. The second term in the RHS indicates the exclusivity of a word to a specific topic; this term is proportional to the ratio of the frequency of a word’s occurrence in a specific topic and frequency of a word in the corpus. The exclusivity term decreases the score of globally frequent terms but it increases the score of rare terms that occur in a topic. Through empirical research, it has been found that an ideal value of the weight, <strong>ℷ </strong>is generally kept around at 0.3.</p><p>The relevant words for the top 4 coherent topics for this dataset are given below. I chose to use 4 topics out of my entire topic list, as the next topic had a token percentage less than 10%. Since my documents are too short and the number of documents is also less, the topics with less than 10% of total tokens in the corpus will most probably lead to incoherent topics.</p><blockquote>The final number of topics must be chosen after taking into account the token percentage and coherence score for each topic</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a699e833f82e3988a6d2c514635a3a61/href">https://medium.com/media/a699e833f82e3988a6d2c514635a3a61/href</a></iframe><p>Now it is easy for us to name these highly coherent (interpretable) topics with the help of relevant words. We have ignored the less coherent topics from the analysis.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/79c67ca248ada7db6bf43aeb28fdec24/href">https://medium.com/media/79c67ca248ada7db6bf43aeb28fdec24/href</a></iframe><p>The main use of this package to provide interactive visualizations to augment our understanding. Let us see how our topics look:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fcodepen.io%2Fsmsubrahmannian%2Fembed%2Fpreview%2FXYymox%3Fheight%3D600%26slug-hash%3DXYymox%26default-tabs%3Dhtml%2Cresult%26host%3Dhttps%3A%2F%2Fcodepen.io%26embed-version%3D2&amp;url=https%3A%2F%2Fcodepen.io%2Fsmsubrahmannian%2Fpen%2FXYymox&amp;image=https%3A%2F%2Fs3-us-west-2.amazonaws.com%2Fm.cdpn.io%2Fscreenshot-coming-soon-small.png&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=codepen" width="800" height="600" frameborder="0" scrolling="no"><a href="https://medium.com/media/78d90557fd32dd2514a32a92050c48f9/href">https://medium.com/media/78d90557fd32dd2514a32a92050c48f9/href</a></iframe><p>We can understand the document from the following perspective.</p><ol><li>Topics are represented as a bubble. The size of the bubble is proportional to its prevalence of the corpus.</li><li>Similar topics appear close together, topics further apart are less similar.</li><li>Upon selecting a topic, the most representative words for the selected topic can be seen. This measure can be a combination of how frequent or how discriminant the word is. You can adjust the weight of each property using the slider.</li><li>When a topic is selected, the percentage of tokens in the topic is also visible. This measure can be used as an additional measure to weed out irrelevant topics. For example, the final list of topics for the above example was cut down to four because of this reason.</li><li>Hovering over a word will adjust the topic sizes according to how representative the word is for the topic as shown in the figure below.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VPFpZORpkTik3bz7Cf9m3g.png" /></figure><p>To play more with this visualization, please download the interactive visualization from <a href="https://github.com/smsubrahmannian/Topic-Modeling/blob/master/pyLDAvis_12.html">this</a> link.</p><h3>Applications of topic modeling</h3><ol><li>LDA model gives a topic distribution of each document. This can be used to predict the most probable topic for a given document</li><li>Other documents similar to a given document can be found using LDA</li><li>Comparing the topics from two different corpora helps us to understand the similarity and differences between the two</li><li>To construct network of topics and documents. Check out <a href="https://smsubrahmannian.github.io/review_networks/">this</a> link.</li></ol><p>Applications 1–3 have been demonstrated in <a href="https://github.com/smsubrahmannian/Topic-Modeling/blob/master/code/Feature%20Engineering%20with%20Topic%20model.ipynb">this</a> notebook in my GitHub repository.</p><h3>References</h3><ol><li><a href="http://aclweb.org/anthology/D/D12/D12-1087.pdf">http://aclweb.org/anthology/D/D12/D12-1087.pdf</a></li><li><a href="http://qpleple.com/topic-coherence-to-evaluate-topic-models/">http://qpleple.com/topic-coherence-to-evaluate-topic-models/</a></li><li><a href="http://qpleple.com/perplexity-to-evaluate-topic-models/">http://qpleple.com/perplexity-to-evaluate-topic-models/</a></li><li><a href="https://github.com/bmabey/pyLDAvis">https://github.com/bmabey/pyLDAvis</a></li><li><a href="https://rare-technologies.com/gsoc17-training-and-topic-visualizations/">https://rare-technologies.com/gsoc17-training-and-topic-visualizations/</a></li></ol><p>Editors : <a href="https://medium.com/@kunalkotian">Kunal Kotian</a>, <a href="https://medium.com/@santhoshhari">Sri Santhosh Hari </a>and <a href="https://towardsdatascience.com/@aswalin">Alvira Swalin</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=55b2214fc17d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>