<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Moiz Saifee on Medium]]></title>
        <description><![CDATA[Stories by Moiz Saifee on Medium]]></description>
        <link>https://medium.com/@moiz.saifee?source=rss-e747ca1923f5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*5XBi-5XkgPrf7kSIvB8wEg.jpeg</url>
            <title>Stories by Moiz Saifee on Medium</title>
            <link>https://medium.com/@moiz.saifee?source=rss-e747ca1923f5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 14:43:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@moiz.saifee/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Kaggle 3rd Place Solution — Jigsaw Multilingual Toxic Comment Classification]]></title>
            <link>https://medium.com/data-science/kaggle-3rd-place-solution-jigsaw-multilingual-toxic-comment-classification-e36d7d194bfb?source=rss-e747ca1923f5------2</link>
            <guid isPermaLink="false">https://medium.com/p/e36d7d194bfb</guid>
            <category><![CDATA[kaggle]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Moiz Saifee]]></dc:creator>
            <pubDate>Fri, 04 Sep 2020 06:39:06 GMT</pubDate>
            <atom:updated>2020-09-06T02:36:48.402Z</atom:updated>
            <content:encoded><![CDATA[<h3>Kaggle 3rd Place Solution — Jigsaw Multilingual Toxic Comment Classification</h3><h4>Approach, Learnings, and Code</h4><p>I had recently participated in the <a href="https://www.kaggle.com/c/jigsaw-multilingual-toxic-comment-classification">Jigsaw Multilingual Toxic Comment Classification</a> challenge at Kaggle and our team (ACE team) secured 3rd place on the final leader board. In this blog, I describe the problem statement, our approach, and the learnings we had from the competition. I have also provided the link to our full code towards the end of the blog.</p><h3>Problem Description</h3><p>The goal of the competition was to detect toxicity (for example rudeness, disrespect, or threat) in user comments during online interaction. This is a very real-life problem given the increase in trolling and hatred on social media etc these days.</p><p>The problem was set up as a vanilla text classification problem where one had to predict the probability of a (text) comment being toxic and the submissions were evaluated on a <a href="http://en.wikipedia.org/wiki/Receiver_operating_characteristic">ROC AUC</a> metric</p><p>The most interesting part of the problem was that — the toxicity had to be identified in 6 different languages, and there was no training data provided at all for 3 of them, and there was very little training data for the remaining 3.</p><p>The competition was organized by <a href="https://jigsaw.google.com/">Jigsaw</a>, which is a Google company, that builds tools to deal with problems like toxicity, disinformation, harassment radicalization in online content. This was the third toxicity detection competition conducted by Jigsaw (although the first one that was multilingual in nature).</p><h3>Why is the problem challenging?</h3><p>Detecting toxicity is a lot more than just detecting abusive words in the text. As an example, consider the following comment, which doesn’t have any abusive words, but is still toxic</p><blockquote>I just wanted to have a little fun, captain buzz kill. I guess you never had a life when you were young. that was probably like, 100000000000000000 years old</blockquote><p>In addition to this, if you are trying to solve the problem across multiple languages, it becomes quite a handful.</p><h3>Solution Overview</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/546/1*XAtKux-jaV3ILjpkwAf4Bg.png" /></figure><p>The figure above shows high-level overview of various components of our final solution, along with their individual scores (ROC) and weights, that led to our final score of 0.9523.</p><p>The final solution was a blend of diverse underlying models trained in a variety of ways. “Roberta” in the above figure represents models derived off RoBERTa XLM architecture and Mono Bert represents Bert Base models pre-trained on the text of a specific language.</p><h3>Technical Overview</h3><p>The flowchart below gives high-level technical overview of our solution. I’ll share various fine details in the section that follows</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*qCDEpiSgWEemgA6LV_3htA.png" /></figure><h3><strong>Training Data / Pre-processing</strong></h3><p>We used the following datasets for model training in the competition:</p><ul><li>Labeled English data provided by the organizer from the past competitions</li><li>The translated version of labeled English data in each target language</li><li>Small validation data provided by organizers in the target language</li><li><a href="http://opus.nlpl.eu/OpenSubtitles-v2018.php">Open Subtitles data</a> with labels generated using <a href="https://www.kaggle.com/cdeotte/pseudo-labeling-qda-0-969">pseudo labeling</a></li><li>Test dataset with pseudo labels*</li></ul><p>Train data was prepared from the above items using stratified sampling ensuring class balance.</p><p>Translating the labeled English data to target languages help in absence of labeled data in the target language, but its not as good as the data in the target language itself because the process of translation strips the text of some subtlety which is at times important to detect toxicity.</p><h3>Pre-trained Models</h3><p>Transformer architecture based pre-trained (transfer learning) models are the rage in NLP since 2018. Conceptually they are deep learning models trained on huge (terabytes) of text data due to which they learn a very good numerical representation of languages (words, text). These representations are useful in a variety of downstream tasks. You can read more details on Transformers and Transfer Learning in one of my <a href="https://towardsdatascience.com/recent-advancements-in-nlp-2-2-df2ee75e189">earlier</a> blogs</p><p>RoBERTa XLM, a variant of Google’s Bert in terms of architecture, and pre-trained on 100+ languages was the workhorse in the competition. It performs surprisingly well out of the box and performed decently even with just English training data on other languages.</p><p>In addition to the multilingual model described above, we also used Bert models pre-trained on specific languages of interest. These mono-lingual models performed better for the individual language they were trained on, compared to the multilingual models as one would expect (for a given model capacity, per language performance goes down as we increase the number of languages). But this method is slightly less elegant in practice as it&#39;s cumbersome to maintain language-specific models in production if you are dealing with a lot of languages.</p><h3>Model Training</h3><p>This is where a lot of art and experience come into play. We used various strategies for model training:</p><ul><li><strong>Multi-stage training: </strong>Training the model on progressively harder tasks, example training on just labeled English data and then gradually moving to the harder task, example pseudo labels and then did the final round (epoch) of training on the validation data as that was the closes to the test data</li><li><strong>Unsupervised Training on Domain-specific data: </strong>The enormous language models like RoBERTa are typically trained on generic datasets like news, Wikipedia — this may be quite different compared to the content/structure/vocabulary of your task-specific data. While getting the labeled data is always an expensive proposition, there is often greater availability of unlabeled data (just the raw text data in this case) which can be used for pre-training. This helps the performance when doing the task-specific supervised training later on</li><li><strong>Multi-Fold Averaging</strong>: The number of training examples putting together all the different types of datasets I described earlier ran into millions. We observed that the model saturated quickly during training and stopped improving after roughly 200K samples. We trained various models using a random subset of data and averaged our predictions on the test dataset using these models</li></ul><h3>Prediction and Blending / Ensembling</h3><p>It&#39;s a well-known fact that blending predictions from various diverse models tend to outperform the underlying individual model predictions, and is often the best way to improve the final predictions.</p><p>Our final solution was a blend of predictions from 4 families of models as described in “Solution Overview”. We trained multi-fold models (training on different subsets of data) within each family to end up with about 30 (!!) odd models.</p><p>We took a straight average of multifold predictions within each family and a weighted average of predictions across different families of models. We didn’t really have a quality dataset, which was unseen by underlying models, to come with the weights across the family, so we used some intuition, leader board feedback, and experimentation to determine these weights.</p><p>Test Time Augmentation (TTA) is a technique where we make the prediction on not just a test example, but also some of its variants, to end up with a better final prediction. We employed this technique in some of our underlying models, and it gave us a minor boost, here is how it worked:</p><ul><li>Make a prediction on the original comment in the test dataset</li><li>Make predictions on the translation of the original comment in various languages</li><li>Take a weighted average of predictions on original comment and its translations</li></ul><h3>Hardware</h3><p>It&#39;s no secret that Deep Learning models run a lot faster on GPU compared to a CPU, but <a href="https://cloud.google.com/tpu">Google’s TPU</a> is way faster than even a GPU. They are just phenomenal in terms of memory (allow higher batch size) and speed (allow quick experimentation) they provide and make the training process a lot less painful.</p><p>We extensively used the free TPU quota provided by Kaggle and Google Colab during the competition to train our models.</p><h3>Code</h3><p>Here is the <a href="https://github.com/moizsaifee/kaggle-jigsaw-multilingual-toxic-comment-classification-3rd-place-solution">link</a> to our complete code</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e36d7d194bfb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/kaggle-3rd-place-solution-jigsaw-multilingual-toxic-comment-classification-e36d7d194bfb">Kaggle 3rd Place Solution — Jigsaw Multilingual Toxic Comment Classification</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pytorch vs Tensorflow in 2020]]></title>
            <link>https://medium.com/data-science/pytorch-vs-tensorflow-in-2020-fe237862fae1?source=rss-e747ca1923f5------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe237862fae1</guid>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Moiz Saifee]]></dc:creator>
            <pubDate>Mon, 22 Jun 2020 06:25:25 GMT</pubDate>
            <atom:updated>2020-06-23T14:23:55.632Z</atom:updated>
            <content:encoded><![CDATA[<h4>How the two popular frameworks have converged</h4><p>Pytorch and Tensorflow are by far two of the most popular frameworks for Deep Learning. It’s always a lot of work to learn and be comfortable with a new framework, so a lot of people face the dilemma of which one to choose out of the two</p><p>The two frameworks had a lot of major differences in terms of design, paradigm, syntax etc till some time back, but they have since evolved a lot, both have picked up good features from each other and are no longer that different.</p><p>A lot of online articles comparing the two are a little old, and do not appropriately capture the present scenario. In fact Soumith Chintala, one of the original authors of PyTorch, also recently tweeted about how the two frameworks are pretty similar now.</p><h3></h3><p>Debates on PyTorch vs TensorFlow were fun in 2017. There was healthy competition to innovate, and philosophical differences like Theano vs Torch, Emacs vs vim, or android vs iOS. Now both products look exactly the same, the debates are nonsense and boring. Please stop.</p><h3>Brief History</h3><p>Tensorflow is from Google and was released in 2015, and PyTorch was released by Facebook in 2017.</p><p>Tensorflow arrived earlier at the scene, so it had a head start in terms of number of users, adoption etc but Pytorch has bridged the gap significantly over the years</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*IsaBkifkc5P7ihRA8IKQ8Q.png" /><figcaption>Source: <a href="https://trends.google.com/trends/explore?date=today%205-y&amp;geo=US&amp;q=%2Fg%2F11bwp1s2k3,%2Fg%2F11gd3905v1">Google Trends</a></figcaption></figure><p>Both work on fundamental data type called <em>Tensors</em> which are nothing but multi-dimensional arrays, amenable to high performance computation. Both represent computation as a directed acyclic graph often called <em>Computation Graph.</em></p><h3>Feature Comparison: Then and Now</h3><p>Let’s look into some of the important aspect about these frameworks, the major differences in the beginning and where things stand as of today.</p><p><strong>Programming API</strong></p><p>The Tensorflow API was very cryptic to start with, it almost felt like learning a new programming language, on top of it, it was also very hard to debug due to its static computation graph approach (more on that below). Pytorch (python) API on the other hand is very Pythonic from the start and felt just like writing native Python code and very easy to debug.</p><p>Tensorflow did a major cleanup of its API with Tensorflow 2.0, and integrated the high level programming API Keras in the main API itself. A combination of these two significantly reduced the cognitive load which one had to undergo while writing Tensorflow code in the past :-)</p><p>The programming APIs (of TensorFlow and PyTorch) in fact look very similar now, so much that the two are indistinguishable a number of times (see example towards the end)</p><h4><strong>Computation Graph</strong></h4><p>Computation graph was a major design difference between the two frameworks to start with.</p><p>Tensorflow adopted a static Computation Graph approach where one defines the sequence of computation that one wants to do, with placeholder for the data. After that for training / running the model you feed in the data. Static computation graph is great for performance and ability to run on different devices (cpu / gpu / tpu), but is a major pain to debug.</p><p>Pytorch on the other hand adopted a dynamic computation graph approach, where computations are done line by line as the code is interpreted. This makes it a lot easier to debug the code, and also offers other benefits — example supporting variable length inputs in models like RNN.</p><p>Fast forward to today, Tensorflow introduced the facility to build dynamic computation graph through its “Eager” mode, and PyTorch allows building of static computational graph, so you kind of have both static/dynamic modes in both the frameworks now.</p><h4><strong>Distributed Computing</strong></h4><p>Both the frameworks provided the facility to run on single / multiple / distributed CPUs or GPUs. In earlier days it used to be a pain to get Tensorflow to work on multiple GPUs as one had to manually code and fine tune performance across multiple devices, things have changed since then and now its almost effortless to do distributed computing with both the frameworks.</p><p>Google made its custom hardware accelerator Tensor Processing Unit (TPU) which can run computation at blazing speed, even a lot faster than GPU, available for 3rd party use in 2018. Since Tensorflow and TPU are both from Google, its far easier to run code on TPUs using Tensorflow as opposed to PyTorch, as PyTorch has a bit of patchy way of working on TPUs using third party libraries like <a href="https://github.com/pytorch/xla">XLA</a></p><h4>Deployment / Production</h4><p>Deployment is something where Tensorflow had a lot of advantage over PyTorch, in part due to better performance due to its Static Computation graph approach, but also due to packages / tools that facilitated quick deployment over cloud, browser or mobile. That’s the reason a lot of companies preferred Tensorflow when it came to production.</p><p>PyTorch has tried to bridge this gap in version 1.5+ with TorchServe, but its yet to mature</p><h4>Code Comparison</h4><p>Its amusing that for a lot of things the APIs are so similar that the codes are almost indistinguishable. Given below are code snippets for the core components on <a href="https://en.wikipedia.org/wiki/MNIST_database">MNIST Digit Recognition</a> (proverbial “Hello World” problem in Computer Vision) for both Tensorflow and Pytorch, try to <strong>guess</strong> which one is which</p><p>The complete Tensorflow and Pytorch code is available at my <a href="https://github.com/moizsaifee/TF-vs-PyTorch">Github Repo</a></p><p><strong>Data Loader</strong></p><pre><em># Download the MNIST Data<br></em>(x_train, y_train), (x_test, y_test) = mnist.load_data()<br>x_train, x_test = x_train / 255.0, x_test / 255.0<br><em># Add a channels dimension<br></em>x_train = x_train[..., tf.newaxis]<br>x_test = x_test[..., tf.newaxis]<br><br>train_ds = tf.data.Dataset.from_tensor_slices((x_train, y_train)).shuffle(10000).batch(32)<br>test_ds = tf.data.Dataset.from_tensor_slices((x_test, y_test)).batch(32)</pre><p><em>Which one is PyTorch code - above or below?</em></p><pre><em># Download the MNIST Data and create dataloader<br></em>transform = transforms.Compose([transforms.ToTensor()])<br>xy_train = datasets.MNIST(<strong>&#39;./&#39;</strong>, download=True, train=True, transform=transform)<br>xy_test = datasets.MNIST(<strong>&#39;./&#39;</strong>, download=True, train=False, transform=transform)<br><br>train_ds = DataLoader(xy_train, batch_size=32, shuffle=True)<br>test_ds = DataLoader(xy_test, batch_size=32, shuffle=True)</pre><p>Okay the method to load the data looks a bit different, but I promise it gets similar from here on :-)</p><p><strong>Model Definition</strong></p><pre><em># Model Definition<br></em>class MyModel(nn.Module):<br>    def __init__(self):<br>        super(MyModel, self).__init__()<br>        self.conv1 = Conv2d(in_channels=1, out_channels=32, kernel_size=3)<br>        self.flatten = Flatten()<br>        self.d1 = Linear(21632, 128)<br>        self.d2 = Linear(128, 10)<br><br>    def forward(self, x):<br>        x = F.relu(self.conv1(x))<br>        x = self.flatten(x)<br>        x = F.relu(self.d1(x))<br>        x = self.d2(x)<br>        output = F.log_softmax(x, dim=1)<br>        return output</pre><p><em>Which one is PyTorch code - above or below?</em></p><pre><em># Model Definition<br></em>class MyModel(Model):<br>    def __init__(self):<br>        super(MyModel, self).__init__()<br>        self.conv1 = Conv2D(filters=32, kernel_size=3, activation=<strong>&#39;relu&#39;</strong>)<br>        self.flatten = Flatten()<br>        self.d1 = Dense(128, activation=<strong>&#39;relu&#39;</strong>)<br>        self.d2 = Dense(10)<br><br>    def call(self, x):<br>        x = self.conv1(x)<br>        x = self.flatten(x)<br>        x = self.d1(x)<br>        output = self.d2(x)<br>        return output</pre><p><strong>Instantiate Model, Loss, Optimizer</strong></p><pre><em># Instantiate Model, Optimizer, Loss<br></em>model = MyModel()<br>optimizer = Adam()<br>loss_object = SparseCategoricalCrossentropy(from_logits=True, reduction=<strong>&#39;sum&#39;</strong>)</pre><p><em>Which one is PyTorch code - above or below?</em></p><pre><em># Instantiate Model, Optimizer, Loss<br></em>model = MyModel()<br>optimizer = Adam(model.parameters())<br>loss_object = CrossEntropyLoss(reduction=<strong>&#39;sum&#39;</strong>)</pre><p><strong>Training Loop</strong></p><pre>for epoch in range(2):<br>    <em># Reset the metrics at the start of the next epoch<br>    </em>train_loss = 0<br>    train_n = 0<br>    for images, labels in train_ds:<br>        with GradientTape() as tape:<br>            <em># training=True is only needed if there are layers with different<br>            # behavior during training versus inference (e.g. Dropout).<br>            </em>predictions = model(images, training=True)<br>            loss = loss_object(labels, predictions)<br>            train_loss += loss.numpy()<br>            train_n += labels.shape[0]<br>        gradients = tape.gradient(loss, model.trainable_variables)<br>        optimizer.apply_gradients(zip(gradients, model.trainable_variables))<br>    train_loss /= train_n</pre><p><em>Which one is PyTorch code - above or below?</em></p><pre>for epoch in range(2):<br>    <em># Train<br>    </em>model.train()<br>    train_loss = 0<br>    train_n = 0<br>    for image, labels in train_ds:<br>        predictions = model(image).squeeze()<br>        loss = loss_object(predictions, labels)<br>        train_loss += loss.item()<br>        train_n += labels.shape[0]<br>        loss.backward()<br>        optimizer.step()<br>        optimizer.zero_grad()<br>    train_loss /= train_n</pre><p>It’s really interesting (and convenient!) how similar the APIs seem to be now.</p><h3>Conclusion</h3><p>Tensorflow and PyTorch are two excellent frameworks for research and development of deep learning applications. They do the heavy lifting in terms of computation, managing the underlying hardware and have huge communities which makes it a lot easier to develop custom application by standing on the shoulder of giants.</p><p>There were some major differences in the two frameworks till a while back, both have since adopted good features from each other and have both become better in the process. There are still things which are slightly easier in one compared to another, but its now also easier than ever to switch back and forth between the two due to increased similarity.</p><h3>References</h3><ul><li><a href="https://github.com/moizsaifee/TF-vs-PyTorch">https://github.com/moizsaifee/TF-vs-PyTorch</a></li><li><a href="https://www.tensorflow.org/guide/effective_tf2">https://www.tensorflow.org/guide/effective_tf2</a></li><li><a href="https://pytorch.org/docs/stable/index.html">https://pytorch.org/docs/stable/index.html</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe237862fae1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/pytorch-vs-tensorflow-in-2020-fe237862fae1">Pytorch vs Tensorflow in 2020</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GPT-3: The New Mighty Language Model from OpenAI]]></title>
            <link>https://medium.com/data-science/gpt-3-the-new-mighty-language-model-from-openai-a74ff35346fc?source=rss-e747ca1923f5------2</link>
            <guid isPermaLink="false">https://medium.com/p/a74ff35346fc</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Moiz Saifee]]></dc:creator>
            <pubDate>Sun, 31 May 2020 13:33:05 GMT</pubDate>
            <atom:updated>2020-05-31T15:20:24.371Z</atom:updated>
            <content:encoded><![CDATA[<h4>Pushing Deep Learning to the Limit with 175B Parameters</h4><h3>Introduction</h3><p><a href="https://openai.com/">OpenAI</a> recently <a href="https://arxiv.org/abs/2005.14165">released</a> pre-print of its new mighty <em>language model</em> GPT-3. Its a much bigger and better version of its predecessor GPT-2. In fact, with close to 175B trainable parameters, GPT-3 is much bigger in terms of size in comparison to anything else out there. Here is a comparison of number of parameters of recent popular pre trained NLP models, GPT-3 clearly stands out.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/582/1*C-KNWQC_wXh-Q2wc6VPK1g.png" /></figure><h3>What’s New?</h3><p>After the success of Bert, the field of NLP is increasingly moving in the direction of creating <em>pre-trained</em> <em>language models</em>, trained on huge text corpus (in an unsupervised way), which are later fine-tuned on specific tasks such as translation, question answering etc using much smaller task specific datasets.</p><p>While this type of <em>transfer learning</em> obviates the need to use task specific model architectures, but you still need task specific datasets, which are a pain to collect, to achieve good performance.</p><p>Humans by contrast learn in a very different way, and have the ability to learn a new task based on very few examples. GPT-3 aims to address this specific pain point, that is, its a task agnostic model, which needs zero to very limited examples to do well and achieve close to state of the art performance on a number of NLP tasks</p><h3>Terminologies</h3><p>Before we deep dive, it may be useful to define some commonly used terminologies:</p><ul><li><strong>NPL Tasks: </strong>These are tasks which have something to do with human languages, example — Language Translation, Text Classification (e.g. Sentiment extraction), Reading Comprehension, Named Entity Recognition (e.g. recognizing person, location, company names in text)</li><li><strong>Language Model</strong>s: These are models which can predict the most likely next words (and their probabilities) given a set of words (think something like Google query auto-complete). Turns out these type of models are useful for a host of other tasks although they may be trained on mundane next word prediction</li><li><strong>Zero / One / Few shot learning: </strong>Refers to model’s ability to learn a new task by seeing zero / one / few examples for that task</li><li><strong>Transfer Learning: </strong>Refers to the notion in Deep Learning where you train a model for one task (example object detection in images) , but the ability to leverage and build upon that for some other different task (example assessing MRI scans). After massive success in Computer Vision, its in vogue in NLP these days.</li><li><strong>Transformer Models</strong>: Deep learning family of models, used primarily in NLP, which forms the basic building block of most of the state-of-the-art NLP architectures these days. You can read more about <em>Transformers</em> at one of my earlier <a href="https://towardsdatascience.com/recent-advancements-in-nlp-2-2-df2ee75e189">blog</a></li></ul><h3>The Approach</h3><p>The model is built using the standard concepts of <em>Transformer</em>, <em>Attention</em> etc and using the typical <em>Common Crawl, Wikipedia, Books</em> and some additional data sources. A lot of things — pre training, model, data are similar to GPT-2, but everything (model size, data size, training time) is just a lot bigger. In fact its humongous size is what drives most of the benefits of the model.</p><p>The following graph shows the benefit in accuracy for various Zero / One / Few shot tasks as a function of number of Model parameters, clearly major gains are achieved due to the scaled up size.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-iY_0W7uX-D30qjCv-OBmg.png" /><figcaption>Source: Paper</figcaption></figure><p>Most of the things used in the model are so huge — example 96 <em>Attention</em> layers, <em>Batch Size</em> of 3.2M, 175B <em>Parameters</em> — that they are unlike anything in the past. The model is ~10x larger in terms of number of parameters to the next closest thing (Microsoft Turing NLG with 17B parameters)</p><p>There is no need to do gradient / parameter updates (fine tuning) for using the GPT-3 model for various tasks. One can just interact with the model using natural language and/or provide some examples of the tasks that you are trying to do and the model will do it!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/506/1*gtoP7YMNNKr0bvN6JxWp5w.png" /><figcaption>Source: Paper</figcaption></figure><h3>What Does All this Mean?</h3><p>The concept of not requiring large custom, task specific datasets, in addition to not requiring task specific model architectures is a huge step in direction of making cutting edge NLP more accessible.</p><p>While GPT-3 delivers great performance on a lot of NLP tasks example — word prediction, common sense reasoning — but it doesn’t do equally well on everything. For instance it doesn’t do great on things like — Text synthesis, some reading comprehension tasks etc. In addition to this, it also suffers from <a href="https://towardsdatascience.com/can-ai-algorithms-be-biased-6ab05f499ed6">bias</a> in the data which may lead the model to generate stereotyped or prejudiced content. So there is more work to be done here.</p><p>In addition to all this, the huge size of GPT-3, makes it out of bounds for almost everyone except a select few companies and research labs in the world. As per the authors, the model is very versatile and contains a very wide range of skills not needed for specific tasks and there might be a scope of creating smaller, more manageable task specific models using the concept of <a href="https://arxiv.org/abs/1503.02531"><em>distillation</em></a>.</p><p>Would be exciting to see how this thing evolves in future.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a74ff35346fc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/gpt-3-the-new-mighty-language-model-from-openai-a74ff35346fc">GPT-3: The New Mighty Language Model from OpenAI</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Formula for Life Long Learning]]></title>
            <link>https://medium.com/swlh/formula-for-life-long-learning-7ff900e2db9e?source=rss-e747ca1923f5------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ff900e2db9e</guid>
            <category><![CDATA[learning-and-development]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[learning]]></category>
            <dc:creator><![CDATA[Moiz Saifee]]></dc:creator>
            <pubDate>Mon, 06 Apr 2020 12:59:50 GMT</pubDate>
            <atom:updated>2020-04-08T19:18:15.428Z</atom:updated>
            <content:encoded><![CDATA[<h4>In Data Science and Beyond</h4><p>Working in Data Science / Artificial Intelligence could be overwhelming — you need a good handle on basic Math / Statistics (example probability, matrix algebra, calculus), familiarity with various off the shelf algorithms (example Regression, Neural Networks, Clustering etc), facility with tools and programming languages (example Python, R, Excel) and good communication and presentation skills .</p><p>Now that’s a lot of things to fit in one body. Not everyone is familiar with all that takes to be a top data scientist. On top of all this, the field itself is evolving at a very rapid pace and you need to catch up with all the new algorithms, frameworks and innovation that is happening. Life long learning thus is not a luxury but a requirement.</p><p>Some natural questions then are:</p><ul><li>Should one care about taking out time to learn new things, especially the ones which have just come up?</li><li>How to appropriately pick and choose what you should learn out of a ton of things out there?</li><li>What are good resources to keep yourself updated?</li></ul><p>I have thought about these questions many times in life, I’ll share my thoughts on this topic in this blog. While most of the things which I have written here are in context of Data Science / AI, but I believe similar principles apply in other areas in life too.</p><h3>Background</h3><p>Before I deep dive, I’ll introduce some terminologies which would be useful later on. In <a href="https://en.wikipedia.org/wiki/Machine_learning"><em>Machine Learning</em></a> there are broadly three paradigms to make a machine / software “learn” what we want them to learn:</p><ol><li>Supervised Learning</li><li>Unsupervised Learning</li><li>Reinforcement Learning</li></ol><p><a href="https://en.wikipedia.org/wiki/Reinforcement_learning"><em>Reinforcement Learning</em></a> is probably closest to how we as humans learn to do things. It entails use of trial &amp; error and use of reward for desirable behavior, and penalty for undesirable to make a software agent learn optimal behavior in an <em>environment</em>.</p><h4>Exploration and Exploitation</h4><p>At the heart of <em>Reinforcement Learning</em> are the concepts of <em>Exploration </em>and <em>Exploitation</em>.</p><p><em>Exploitation</em> is about making the best decision using the information you already know . It involves doubling down on things which have worked in the past and potentially improving them even further. The broad idea is of incremental refinement.</p><p><em>Exploration</em> is about gathering more information to discover other promising things which are yet to be refined. While <em>Exploration </em>may not always be fruitful and may seem sub optimal in the short term, but may help you find an optimal solution in the long run and avoid getting trapped <em>locally.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_5dltx4BcI8rRmCK2Sq_kw.png" /><figcaption><em>A real-life example of the exploration vs exploitation dilemma: where to eat? (Image source: UC Berkeley AI course </em><a href="http://ai.berkeley.edu/lecture_slides.html"><em>slide</em></a><em>, </em><a href="http://ai.berkeley.edu/slides/Lecture%2011%20--%20Reinforcement%20Learning%20II/SP14%20CS188%20Lecture%2011%20--%20Reinforcement%20Learning%20II.pptx"><em>lecture 11</em></a><em>.)</em></figcaption></figure><p>For a <em>Reinforcement Learning</em> based agent to learn things optimally it needs to strike the right balance between <em>Exploitation</em> and <em>Exploration</em>. Too much <em>Exploitation</em> and it may get stuck in a <em>local optimum, </em>unaware of other things which could have worked better. With too much <em>Exploration</em>, it may never exploit or fully utilize all the information it has collected.</p><h3>Applying the Concepts in Real Life</h3><p>To draw a parallel from <em>Reinforcement Learning</em>, for achieving optimal results, we need to strike the right balance between <em>Exploitation</em> and <em>Exploration</em> in real life too.</p><h4>Exploration:</h4><p><em>Exploration</em> in daily life is primarily about reading, listening and quickly trying out something new. There can be many sources to achieve this goal:</p><ul><li>Reading blogs, forums, news, books and papers to keep yourself updated</li><li>Attending conferences and / or meetups, listening to podcasts etc</li><li>Dipping your toe into a new framework, algorithm, pre-trained model to see if it looks interesting.</li></ul><p>Some of the important sources of information that I monitor are:</p><p>Blog / Twitter handles / Website of :</p><ul><li><strong>Big Companies</strong> — Google AI, Facebook AI, Uber Engineering, Microsoft Research, IBM Research, Netflix, DeepMind, Baidu, OpenAI</li><li><strong>Top Academic Groups</strong> — Stanford NLP, Berkeley Research, CMU etc</li><li><strong>Frameworks</strong> — Tensorflow, Pytorch, Huggingface, Fastai</li><li><strong>Forums</strong> — Kaggle, git</li><li><strong>Papers</strong> — Arxiv, Paperswithcode</li><li><strong>Courses</strong>: Coursera, Udemy</li></ul><p>Even the above (small) list may quickly get overwhelming, but remember, the goal of this step is to shortlist a few things that you want to go really deep into rather than being on top of everything of everything which is not possible.</p><h4>Exploitation:</h4><p><em>Exploitation</em> in real life is about putting things into practice. There is always inertia when you try to include a new thing in your workflow, but overcoming initial hurdles and making some short term sacrifices for long term gains may be well worth it.</p><p>Remember everything you Explore would be of little help to you until you really deep dive. Here are some of the ways of <em>Exploitation</em> in real life:</p><ul><li>Re-implementing one of your existing projects using a new technique / algorithm to assess the gains over what you did earlier</li><li>Doing some fun project using an open dataset and/or Kaggle competition to utilize your newly learned tricks</li><li>Creating some open source utility on Git and / or contributing to an existing utility</li></ul><h3>Conclusion</h3><p>The <em>Exploration</em> and <em>Exploitation </em>paradigm described above may be helpful in going about learning new things without compromising on things which you have to do, using judicious time allocation. The short term overhead may well be worth the long term gains. I have personally benefited a lot by striking a balance between <em>Exploration</em> vs <em>Exploitation</em>. If I look at what I do today, I make heavy use of things I learned in last 2–3 years, without them I simply wouldn’t be as effective as I am right now.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ff900e2db9e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/formula-for-life-long-learning-7ff900e2db9e">Formula for Life Long Learning</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Meena: Google’s New Chatbot]]></title>
            <link>https://medium.com/data-science/meena-googles-new-chatbot-657241cf5595?source=rss-e747ca1923f5------2</link>
            <guid isPermaLink="false">https://medium.com/p/657241cf5595</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Moiz Saifee]]></dc:creator>
            <pubDate>Sat, 01 Feb 2020 17:32:38 GMT</pubDate>
            <atom:updated>2020-02-01T18:34:12.001Z</atom:updated>
            <content:encoded><![CDATA[<h4>A more human-like and versatile chatbot</h4><p>Google recently <a href="https://arxiv.org/abs/2001.09977">published</a> a paper on its new chatbot Meena. Google has hit all the right chords in terms of its design and approach. While the underlying techniques are not entirely new, but it seems to be the right direction in terms of building chatbots which are truly versatile and more human-like in terms of their interactions.</p><h3>The Rise of Chatbots</h3><p>Chatbots are AI system which interact with users via text messages or speech. There has been tremendous growth in applications of chatbots off late. The chatbot market, in fact, is expected to grow to $9.4B based on some <a href="https://markets.businessinsider.com/news/stocks/global-chatbot-market-anticipated-to-reach-9-4-billion-by-2024-robust-opportunities-to-arise-in-retail-ecommerce-1028759508">estimates</a>.</p><p>There are a lot of use cases of chatbots — for example, customer service, e-commerce facilitator, food delivery etc. Most of the big companies have their own version of chatbots — example Siri by Apple, Google Assistant by Google, Alexa by Amazon etc.</p><p>There are also a lot of frameworks available to help you build your own chatbot applications, example — open source framework <a href="https://rasa.com/">RASA</a>, Google’s <a href="https://dialogflow.com/">DialogueFlow</a>, IBM’s <a href="https://www.ibm.com/watson/how-to-build-a-chatbot">Watson</a>, or Microsoft’s <a href="https://dev.botframework.com/">Bot Framework</a></p><p>Notwithstanding the growth which chatbots have seen, here are some of the major challenges with most of the current solutions:</p><ol><li><strong>Closed-Domain / Rule Based</strong>: Most of the chatbots are closed-domain, which means they work only within a specific domain. These bots look for certain keywords to figure out the user intent and some rule-based system under the hood determines what action should be taken based on the intent. While these systems are still useful but they are not generic and require a lot of domain specific work to get going</li><li><strong>Not Conversational: </strong>The holy-grail of chatbots is that the interactions should look human-like. With most of the current systems that’s hardly the case</li><li><strong>Not Multi-turn: </strong>Most of the chatbots fail to take the larger context of multiple interactions (where the user and the chatbot interact with each other by taking turns) into account while giving a response, this again leads to non human-like experience while interacting.</li></ol><h3>Meena</h3><p>Meena is a multi-turn, open-domain chatbot based on Transformer seq2seq architecture (you may read <a href="https://towardsdatascience.com/recent-advancements-in-nlp-2-2-df2ee75e189">this</a> for more on Transformers) trained in an end-to-end fashion. It&#39;s huge in size comprising of 2.6B parameters and was trained on 300GB+ text data. Meena tries to address the challenges highlighted in the prior section by its approach and design.</p><p>Here are some of the key highlights from the paper:</p><ol><li><strong>Training Data: </strong>Meena is trained on public domain social media conversations. Each conversation is turned into a (context, response) pair where “context” represents up to 7 messages prior to the “response” message. Total data comprised of 40B words and 341 GB of text data.</li><li><strong>Model Architecture</strong>: Meena is based on Evolved Transformer (ET) architecture, which is an evolutionary NAS architecture based on the original Transformer architecture. The model has 1 ET encoder block and 13 ET decoder block.</li><li><strong>Model Training: </strong>The model was trained for a whopping 30 days on a TPU v3 pod (2,048 TPU cores !!)</li><li><strong>Taking the existing research forward</strong>: Most of the concepts which are used in Meena, example — open-domain nature, end-to-end training, transformer seq2seq architecture etc have been attempted in some form in the past. It&#39;s just a lot bigger (training data, model size) and better (architecture, decoding).</li><li><strong>Evaluation Metric: </strong>It&#39;s hard to evaluate performance of chatbots as there is no single correct answer. One of the major contributions of the paper as per the authors is — design of an evaluation metric which they call <em>Sensibleness and Specificity</em> <em>Average</em> (SSA). It&#39;s supposed to mimic whether the conversation would be called as a good conversation based on human judgement or not.</li><li><strong>Sensibleness and Specificity Average (SSA)<em>: </em></strong>For a given response, SSA is defined as the average of binary indicators of whether the response would be deemed to make sense (sensible) and specific (that is not a generic response like “I don’t know” or “that’s good”) by a human or not.</li><li><strong>Training Objective</strong>: Defining a metric for evaluation of end performance is not enough as one also needs to define a measure which the model would try to optimize for while training. Another major accomplishment as per the authors is demonstrating that the “perplexity”, the commonly used metric in NLP, is correlated with the SSA metric which mimics human judgement. So optimizing the model for decreasing perplexity automatically makes it more desirable by humans.</li></ol><h3>Results</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/721/1*AHhHdDg4EfQ5gKOMkYQICw.png" /><figcaption>Meena vs Humans &amp; other chabots. Source <a href="https://ai.googleblog.com/2020/01/towards-conversational-agent-that-can.html">Google Blog</a></figcaption></figure><p>Meena scores 79% on the SSA metric which the authors have defined, which is below the level of Humans at 86%, but it&#39;s still way better than a lot of other popular chatbots which were evaluated. Of course, other chatbots were not trained keeping the new SSA metric (which the authors have proposed) in mind, but current approach still deserves merit as it&#39;s not handcrafted and is trained in an end-to-end fashion.</p><h3>Summary</h3><p>Google’s approach and results for Meena are exciting. While there is still a lot of work left for being able to build chatbots which are as good as humans in terms of expertise and versatility, but Meena seems to be a step in the right direction.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=657241cf5595" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/meena-googles-new-chatbot-657241cf5595">Meena: Google’s New Chatbot</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Can AI Algorithms be Biased?]]></title>
            <link>https://medium.com/data-science/can-ai-algorithms-be-biased-6ab05f499ed6?source=rss-e747ca1923f5------2</link>
            <guid isPermaLink="false">https://medium.com/p/6ab05f499ed6</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[bias]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Moiz Saifee]]></dc:creator>
            <pubDate>Fri, 17 Jan 2020 12:03:52 GMT</pubDate>
            <atom:updated>2020-01-22T14:33:34.718Z</atom:updated>
            <content:encoded><![CDATA[<h4>Defining, detecting and avoiding bias</h4><p>AI algorithms are increasingly being used in a wide range of areas for making decisions that impact our day to day life. Some examples are —Recruitment, Healthcare, Criminal Justice, Credit Risk Scoring etc. It&#39;s being used by not just private businesses but also governments.</p><p>One of the supposed benefits of using AI or machines in general for making decisions is — they may be impartial, objective and may not carry the same biases as humans do and hence may be more “fair”. Some of the recent studies have shown that AI systems can be biased as well.</p><p><em>Imagenet</em>, the public Image database which feeds into various computer vision applications like Face Detection, recently removed 600K images when an art project <a href="https://hyperallergic.com/518822/600000-images-removed-from-ai-database-after-art-project-exposes-racist-bias/">exposed</a> racial bias in its images. All this has lead to increased awareness about bias in AI and has raised basic questions related to trust against many AI systems.</p><h4>What is Bias?</h4><p>Before we go deeper into this topic, its important to define bias. Here is a definition from Wikipedia</p><blockquote><strong>Bias</strong> is disproportionate weight <em>in favor of</em> or <em>against</em> an idea or thing, usually in a way that is closed-minded, prejudicial, or unfair.</blockquote><p>There has been a lot of studies on the kind of biases humans suffer from, Nobel Laureate Daniel Kahneman talks about a variety of biases in human intuition in his excellent book <a href="https://www.goodreads.com/book/show/11468377-thinking-fast-and-slow">Thinking Fast and Slow</a></p><p>An example of a bias from which humans commonly suffer from is —<strong> Confirmation bias. </strong>It is defined as a tendency to search, interpret and recall information in a way which confirms or strengthen one’s pre-existing beliefs. While one may think that the advent of internet and explosion of information (data) available to us would lead us to the “truth”, confirmation bias is what keeps people selectively interpret information and hinged to what they already believe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/498/1*az0vC9QAO4cczPxMqxz0hA.png" /><figcaption><strong>Confirmation Bias</strong></figcaption></figure><h4>Case Studies of AI Bias</h4><p>A famous case study in AI Bias is the <a href="https://en.wikipedia.org/wiki/COMPAS_(software)">COMPAS</a> system which is used by US courts to assess the likelihood of a defendant becoming a recidivist. An <a href="https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing">investigation</a> of the software found that, while the system was designed for maximizing overall accuracy, the false positive rates for African Americans was twice that of for Caucasians.</p><p>Another popular case study is — Amazon’s AI recruitment tool. Amazon had developed a tool to help with recruitment which helped in screening of resumes. It was trained on 10 years data of Amazon’s internal recruitment decisions. Amazon later decided to <a href="https://in.reuters.com/article/amazon-com-jobs-automation/insight-amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idINKCN1MK0AH">scrap</a> the tool when it was discovered that the tool was more likely to pick men over women</p><h4>Mathematical Definition of Bias</h4><p>While all that we have reviewed so far is good, but the definition of bias and fairness has a long history of debate in law, social science and philosophy. It&#39;s difficult to come to a consensus on exact definition of these. What makes it even more tricky in the context of AI models is we need to define it in mathematical terms as AI systems only understand numbers and mathematical operations.</p><p>It&#39;s important to distinguish biases from random errors. It&#39;s not uncommon for AI systems to make errors, this is because they are a simplification of the complex real world. But biases are systematic errors, which occur in a somewhat predictable way. Biases may cause one to act unfairly against an individual or a group.</p><p>Some people have attempted to define the absence of bias in AI systems by balancing of false positive and false negative rates in predictions across groups example gender, race etc. But this definition is not generic enough due to various forms in which Bias may manifest itself. Frankly it&#39;s still an open research area.</p><h4>How does Bias creep into AI systems?</h4><p>A natural question is how do AI systems get Biased? Can they be Biased even if they are not explicitly codified to be biased? Following is a list of some possible reasons why an AI system can get biased:</p><ul><li><strong>Choice of Attributes: </strong>If one uses attributes like age, gender, ethnicity in algorithms, the algorithm can learn the relationship between these attributes and the target which may cause the algorithms to get biased.</li><li><strong>Data Collection / Sampling: </strong>A lot of times, data may be collected or sampled in a way which leads to over or under-representation of a group which may lead to biases. There is a famous case study about <a href="http://streetbump.org/">Street Bump</a> app by Boston municipal to detect potholes which suggested wealthier neighborhood had more bumps simply because people living in affluent neighborhood were more likely to use the app.</li><li><strong>Implicit Bias in Data: </strong>Most of the AI systems learn patterns from the data which is provided to train them. A number of times the data may be generated by humans with their in-built biases. If this is the cases the resultant AI system would also reflect those biases as an AI system is only as good as the data fed to train them. <em>Imagenet</em> incident which I talked about earlier is a good example of this type.</li><li><strong>Optimization Metric</strong>: AI systems typically train to maximize/minimize a certain metric, example error rate on the whole data. While the AI system is busy doing that, there is no guarantee that it would not do anything which we as humans would not deem “fair”.</li></ul><h4>How can we avoid AI bias?</h4><p>While it&#39;s still an open research area, here are some steps you could take to mitigate bias in your AI applications:</p><ul><li><strong>Use Representative Data</strong>: Try to actively look for sources of bias in the data and/or the way it was collected and sampled. Look carefully into how the data was annotated and motivation &amp; incentive of people annotating the data.</li><li><strong>Do an Audit of the Model: </strong>Scrutinize the predictions of your model, compare and contrast false positive and false negative rates of your predictions across various sub-groups. Legally protected sub-groups could be a good place to start.</li><li><strong>Focus on Model Explainability</strong>: A lot of AI systems can be akin to black boxes with limited insights into why they make a certain prediction. There is a lot of research work happening on model explainability — getting a better handle on why a model is making a certain prediction may help in the detection and removal of biases</li><li><strong>Third Party tools</strong>: There are various third party tools which can help you asses bias at various stages in the AI life cycle. Look at <a href="http://aif360.mybluemix.net/">IBM’s AI Fairness 360 Open Source Tool Kit</a> as an example</li><li><strong>Hire Diverse Teams: </strong>Having diversity in your team may help bring in diverse perspectives in looking for avenues for biases.</li></ul><h4>Summary</h4><p>AI applications have seen a phenomenal growth off late. There have been genuine instances which have raised questions about the fairness of decisions made by AI systems. It&#39;s important to handle the bias issue so that AI systems continue to enjoy the trust of organizations and the masses. There is a lot of promising work going on in the area, let’s hope all that will results in making AI applications fairer and the world a better place to live for everyone.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6ab05f499ed6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/can-ai-algorithms-be-biased-6ab05f499ed6">Can AI Algorithms be Biased?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to squeeze out more from your data when training an AI model]]></title>
            <link>https://medium.com/analytics-vidhya/how-to-squeeze-out-more-from-your-data-when-training-an-ai-model-884f0048feed?source=rss-e747ca1923f5------2</link>
            <guid isPermaLink="false">https://medium.com/p/884f0048feed</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Moiz Saifee]]></dc:creator>
            <pubDate>Tue, 31 Dec 2019 04:11:36 GMT</pubDate>
            <atom:updated>2020-01-03T07:49:20.418Z</atom:updated>
            <content:encoded><![CDATA[<p>All of us have heard phrases like — “Data is the new oil” and what not. It’s also well known that most of the Deep Learning models are pretty data hungry and getting appropriate (labelled) data is tough and expensive. Given all this, a natural thing to do would be to squeeze out as much as possible from the data you already have, I’ll go over some techniques which help us do that</p><p>Following is a list of items that I’ll cover in this blog.</p><ol><li>Augmentation</li><li>Transfer Learning</li><li>Semi Supervised Learning / Pseudo Labeling</li><li>Simulation</li></ol><p>Please note that NOT all these techniques are generic and it depends on the problem area whether they would be applicable or not. I’ll specify some use cases where these are applicable when I deep dive into them</p><h3>Augmentation</h3><p>This technique is very heavily used in Computer Vision. Let’s take the example of image classification — a picture of a car should be classified as a car even if we do various transformation on the original image for example — flip, rotate, change lighting, convert to grey scale etc. It turns out that feeding all these variants of the input images provide incremental information to the model and improves performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*vVluJNclSDpLef8aXFlt6A.png" /><figcaption><strong>Figure 1: Original Image</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*jB34pcKn9oiMvf-EMPOQVw.png" /><figcaption><strong>Figure 2: Augmented Images</strong></figcaption></figure><p>The use case of augmentation is not just limited to <em>Computer Vision</em> (where its a very routinely used fundamental concept) but can be done in other areas also. For instance, in <em>NLP</em> some of the augmentation methods that work are:</p><ul><li><em>Synonym Replacement — </em>Replace some randomly chosen words from a sentence by their synonyms.</li><li><em>Random Word Deletion — </em>Delete some randomly chosen words from a sentence.</li><li><em>Intermediate Language Translation</em> — In this method, one translates text in a certain language, say English, to an intermediate language, say French, and then back to source language (English). The rational being the final English sentence would be slightly different than original one. This cool <a href="https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/48038">trick</a> was used by some of the top teams in Kaggle <a href="https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/overview">Toxic comment classification challenege</a></li></ul><p>The <em>Augmentation</em> is a very powerful technique and requires just creative thinking on the type of augmentation which can work in a problem area and doesn’t have any dependency on any other external data.</p><h3>Transfer Learning</h3><p>This concept again is heavily used in Computer Vision, is picking up in NLP due to success of <a href="https://arxiv.org/abs/1802.05365">ELMo</a> and <a href="https://arxiv.org/abs/1810.04805">BERT</a> and may be applicable in some other areas as well. The core idea is — rather than training a Deep Neural Network model from scratch (with small amount of data that you have), fine-tune an existing model which was trained for a related task (using large amount of data that is publicly available).</p><p>For instance — <a href="https://en.wikipedia.org/wiki/ImageNet"><em>Imagenet</em></a> is an openly available image dataset which contains roughly 14 M images spanning 20K classes. If the class in which you are trying to classify an image into, doesn’t exist in <em>Imagenet</em> data and you just have a few hundred images — you’ll be better off using a model pre-trained on <em>Imagenet</em> data as the starting point and by just fine tuning the last few layers using your few hundred images you can achieve great performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/979/1*wOeCmII1_9gQB3ULhVvCTg.png" /><figcaption><strong>Figure 3: Transfer Learning Process</strong></figcaption></figure><p>The reason why Transfer Learning works is — in a Deep Neural Network with dozens of layers — the earlier layers learn more fundamental patterns for example — learning to identify lines, edges etc, and latter layers learn more task specific patterns for example — key characteristics which define a cat in a picture — so just fine-tuning the last few layers on your task specific data does the trick.</p><h3>Semi Supervised Learning / Pseudo Labeling</h3><p>This is a pretty widely applicable technique, and works well with all type of data — image, text, tabular etc, the only wrinkle is — the prerequisite for being able to apply this is availability of unlabeled data (which is often a lot easier and cheaper to get) in addition to some labelled data. This technique is very often used by top teams in Kaggle to squeeze out additional performance improvement.</p><p>The way it works is — you first train a (relatively smaller) model with all of your labeled data, and generate what is called the “Teacher” model. The “Teacher” model is used to make predictions on the large amount of unlabeled data that you also have. You use the high confidence predictions, sometimes by adding some noise along with your original labeled data to train a bigger model called “Student” model. The resultant model has better performance than the “Teacher” model which was just trained on the original labeled data. <a href="https://arxiv.org/abs/1911.04252">Here is a paper</a> with more details.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/998/1*FmfBtug5XZJzanCRgE3oVg.png" /><figcaption><strong>Figure 4: Semi Supervised Learning</strong></figcaption></figure><h3>Simulation</h3><p>For certain problems, typically in <em>Reinforcement Learning</em>, but sometimes in other ares too, one can generate data in a simulated environment and use that for training or testing. For example — <em>DeepMind’s AlphaStar </em>achieved Grandmaster status in <em>StarCraft</em> II by <a href="https://deepmind.com/blog/article/AlphaStar-Grandmaster-level-in-StarCraft-II-using-multi-agent-reinforcement-learning">training</a> on tens of thousand of years of virtual (simulated) gaming experience.</p><p>Another example where simulated environments are used for training or validation is — models for self driving cars.</p><p>There are also many frameworks for example — <a href="https://aihabitat.org/">Habitat</a> by Facebook, <a href="https://deepmind.com/research/open-source/bsuite">Behavior Suite for Reinforcement Learning</a> by Google and <a href="https://gym.openai.com/">Gym</a> from OpenAI for helping run simulated environments.</p><h3><strong>Summary</strong></h3><p>While more data is always better from the perspective of model training, it may come at a big time and monetary cost. The least one could do is to utilize their existing data most efficiently. The techniques described above are some of the ways in which you could do that.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=884f0048feed" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/how-to-squeeze-out-more-from-your-data-when-training-an-ai-model-884f0048feed">How to squeeze out more from your data when training an AI model</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Recent Advancements in NLP]]></title>
            <link>https://medium.com/data-science/recent-advancements-in-nlp-2-2-df2ee75e189?source=rss-e747ca1923f5------2</link>
            <guid isPermaLink="false">https://medium.com/p/df2ee75e189</guid>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Moiz Saifee]]></dc:creator>
            <pubDate>Wed, 25 Dec 2019 17:37:04 GMT</pubDate>
            <atom:updated>2019-12-26T14:57:55.330Z</atom:updated>
            <content:encoded><![CDATA[<h4>Attention, Transformer, BERT and more.</h4><p>In a <a href="https://medium.com/swlh/recent-advancements-in-nlp-1-2-192ac7eefe3c">previous post</a>, I wrote about two recent important concepts in NLP —<em> Word Embedding </em>and <em>RNN</em>. In this post, I’ll cover the concept of <a href="https://arxiv.org/abs/1409.0473"><em>Attention</em></a> and<em> </em><a href="https://arxiv.org/abs/1706.03762"><em>Transformer</em></a><em> </em>which have become the building blocks for most of the state of the art models in NLP at present. I’ll also review BERT which made the powerful concept of <a href="https://en.wikipedia.org/wiki/Transfer_learning"><em>transfer learning</em></a><em> </em>easier<em> in </em>NLP</p><p>Like my earlier post, I’ll skip most of the mathematics and focus more on intuitive understanding. The reason for this is — use of excessive notations and equations is a turn off for many, myself included. That’s actually not even critical for conceptual understanding which I believe is a lot more important than understanding each and every underlying mathematical equation.</p><p>While variants of RNN like Bi-LSTM give a pretty solid performance on various NLP tasks, following are some of the key challenges that still remain:</p><ol><li><strong>Lack of parallelization hurts performance</strong> — LSTM needs to process the input tokens in a given sequence in a sequential manner as the output of the current step depends on the output of the previous step. There is no way to parallelize this computation, which comes with obvious disadvantages of long training time, not being able to train on very large datasets etc.</li><li><strong>Learning long term dependency remains a challenge</strong> —While LSTMs are better at remembering context from something which appeared a while back in a sequence compared to classic RNNs, remembering the context from a word which occurred much earlier in long sentences remains a challenges, that’s why performance is not as good in longer sentences/sequences compared to shorter ones.</li><li><strong>Linear increase in # operations with distance —</strong>The number of operations required to relate signals from two arbitrary input or output positions grows linearly in the distance between positions</li></ol><p>The above challenges were the motivation for a lot of work which happened in 2017 and forward. The concept of <em>attention </em>addresses challenge #2 above and the <em>transformer </em>architecture addresses challenge # 1 and #3.</p><p>One thing to note is — while the concepts of <em>attention </em>and <em>transformer </em>helped break established benchmarks in NLP tasks, these are generic techniques with wide application in any sequence to sequence task. For instance, the <em>transformer </em>architecture was also a building block for <a href="https://deepmind.com/blog/article/alphastar-mastering-real-time-strategy-game-starcraft-ii"><em>AlphaStar</em></a> the <a href="https://deepmind.com/"><em>DeepMind</em></a> bot that beat the top <em>StarCraft II</em> professional player.</p><h3>Attention</h3><p>The <em>attention</em> is one of the most influential ideas in Deep Learning in general. While the concept was originally developed for machine translation, its use quickly spread to a lot of other areas. It was proposed by <em>Dzmitry Bahdanau</em> et al in <a href="https://arxiv.org/abs/1409.0473">this</a> influential paper.</p><p>The core idea behind it is — when performing a certain task, for example, translation of a sentence from one language to the other, say English to French, each word in the output French sentence would be informed by (relevant context of) all words in the original input English sentence with varying degree of attention or importance rather than a single/constant context generated by processing the whole English sentence.</p><p>Consider the following excellent English to French translation example from <a href="https://research.googleblog.com/2017/08/transformer-novel-neural-network.html"><em>Google Blog</em></a><em>. </em>The “it” in the two sentences refers to different nouns, and its translation to French differs depending on which noun the “it” refers to.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/356/1*gPJ-RIPRCZu39wYcfUzdzA.jpeg" /></figure><p>To a human, It is obvious that in the first sentence pair “it” refers to the animal, and in the second to the street</p><p>The following picture depicts what happens conceptually with (self) <em>attention. </em>Rather than taking a context vector of the whole sentence and that informing the translation of the word “it”, the translation context would be informed by different words in the input sentence by a different amount as shown by the color coding (darker = more important)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CBLPwCVUJHTGSvjYW0hZrQ.jpeg" /><figcaption>Source: <a href="https://research.googleblog.com/2017/08/transformer-novel-neural-network.html"><em>Google Blog</em></a></figcaption></figure><p>One important thing to note is — <em>attention</em> weights in the model are not fixed, like the other parameters which are learned by neural network models, but are calculated as a function of input / hidden states — that’s how the model knows how to put appropriate emphasis on relevant words in the input sentence for any given sentence.</p><p>As illustrated above, the concept of <em>attention </em>addresses the problem of long term dependencies by using more appropriate context at each step (rather than a “constant” context, example in LSTM based encoder-decoder), but the problem of non parallelism in computation remains as the computation still needs to be done sequentially. With <em>attention</em>, if anything, we make the computation even more complex compared to simple LSTMs, this is where the <em>Transformer </em>architecture came to the rescue.</p><h3>Transformer</h3><p><em>Transformer </em>was introduced in 2017 in <a href="https://arxiv.org/abs/1706.03762">this</a> seminal paper by Vaswani et al from Google. The network architecture of <em>Transformer </em>is solely based on <em>attention </em>mechanism and has no RNN or CNN units. Following are some of the key benefits of this architecture:</p><ul><li>Superior quality results on language translation tasks, beating previous benchmarks</li><li>More parallelizable and requires significantly less time to train</li><li>Generalizes well to a lot of other tasks</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*do7YDFF2sads0p9BnjzrWA.png" /><figcaption>Transformer Model Architecture. Source <a href="https://arxiv.org/pdf/1706.03762.pdf">Transformer Paper</a></figcaption></figure><p>The figure above shows the overall architecture of the <em>Transformer</em>, which is a bit scary at first blush. Let’s look at high level details of the architecture:</p><ul><li><strong>Encoder-Decoder Architecture</strong>: It follows the famous Encoder Decoder paradigm, where Encoder translates the input into a useful representation which is used by decoder to generate the output. In the figure above, the Encoder block is on the left and Decoder block is on the right.</li><li><strong>Auto Regressive</strong>: At each step, the model consumes the previously generated output as an additional input</li><li><strong>No RNNs</strong>: There are no RNN units, only a bunch of attention layers.</li><li><strong>Multiple layers</strong>: Each of Encoder / Decoder consist of a stack of 6 (a hyperparameter in the model) identical layers</li><li><strong>Embedding</strong>: The Input / Output embedding converts text or words into numbers.</li><li><strong>Positional Encoding</strong>: The positional encoding is to encode the order of the original sequence as we no longer process the input sequentially like in RNN, so it ensures the information of ordering is not lost.</li><li><strong>Mult-Head attention</strong>: There are multiple attention layers running in parallel, for adding variety (akin to multiple convolution filters in Computer Vision).</li><li><strong>Attention layers</strong>: There are 3 types of attentions — Encoder self-attention, Encoder-Decoder Attention and Decoder self-attention.</li><li><strong>Self Attention</strong>: In case of Encoder self-attention (left-bottom attention block in figure), a layer within the Encoder tries to figure out how much attention it should pay to the output of the previous layer(of the Encoder), that is, tries to learn the relationship /context for each word in the input with the other words in the input. Decoder self-attention (right-bottom attention block in the figure)does the same thing but over Output.</li><li><strong>Encoder-Decoder Attention</strong>: With Encoder-Decoder attention (right-top attention block in the figure) The model figures out how relevant is each word in the input for each word in the output.</li></ul><p>With those details, the results which <em>Transformer </em>achieved were impressive, however, there were some areas of improvement:</p><ul><li>It can only deal with fixed-length sentences.</li><li>Larger sentences need to be broken into smaller ones for feeding into the model, which causes “<em>context fragmentation</em>”</li></ul><p><strong>BERT — Bidirectional Encoder Representations from Transformers</strong></p><p>First, an honest confession — I just can’t memorize the full form for BERT no matter how many times I read it, I think the full form was derived from the acronym rather than the other way around :-)</p><p><a href="https://arxiv.org/abs/1810.04805">BERT</a> was proposed by Jacob et al from Google, its a language representation model based on <em>Transformer. </em>I think the major contribution of BERT, in addition to its novel bi-directional training, etc is that it played a big role in popularizing the concept of pre-training / <em>transfer learning</em> in NLP</p><p>Before we go further — just a few words about <em>transfer learning. </em>Its one of the most elegant concepts in Deep Learning was more popular in <em>ComputerVision </em>compared to NLP. The basic idea is — one can train a (deep, that is having multiple layers) model on a generic but related task, and just fine-tune the last few layers on their task-specific data to achieve great performance.</p><p>The rationale for <em>transfer learning</em> is — the earlier layers in a deep model learn more fundamental patterns, for example, learning to detect lines and curves in a <em>computer vision</em> model, and the latter layers learn the task-specific patters, example learning whether there is a cat in the picture or not. The major benefit of this approach is — you can get great performance even with a modest amount of data on deep learning models which are pretty data hungry in general.</p><p>Following are some important details about BERT:</p><ol><li><strong>Bi-Directional training</strong>: Jointly trained by looking at words in both directions —left-to-right and right-to-left for better context, other language models are typically trained by looking at context in just one direction</li><li><strong>Pre-trained</strong>: Pre-trained on huge text corpses (~3B words), so you just need to fine-tune one additional output layer, that is very few parameters (using your training data) for getting very good performance on a variety of tasks like — sentence classification, question answering system, named entity recognition etc without having to do too many task specific changes in model architecture.</li><li><strong>Training Objective: </strong>Typically language models are trained in an unsupervised fashion, with the goal of predicting a word given words which immediately preceded it. BERT is jointly trained on two tasks — predict a word given the words on its left and right (with some details on masking to avoid leakage) and predicting the next sentence given a sentence. The latter is not standard, and language models typically do not directly capture relationships among sentences, but doing this tends to help applications like <em>Question Answering</em> systems for example.</li><li><strong>Model Size: </strong>The authors of the BERT paper experimented with BERT of various sizes, most notable — Base (110M parameters) and Large (340M parameters). They found that bigger is better, even with the similar model architecture and small task-specific training data, this is largely due to the pre-training on a huge corpus.</li></ol><p>Since the Original paper of BERT in Oct’2018, there have been many variants and modifications which have been proposed. Most notable among them are — RoBERTa and mBERT by Facebook AI, XLNet by researchers at CMU and GPT-2 by Open AI.</p><p>This concludes my roundup of some exciting things off late in NLP, clearly, there is so much happening in the field and also at such a rapid pace. There is a lot which people can do with very little effort these days thanks to the <em>giants</em> in the field which have done the heavy lifting and made it easier.</p><h3>Additional Resources</h3><ol><li>Excellent <a href="https://jalammar.github.io/visualizing-neural-machine-translation-mechanics-of-seq2seq-models-with-attention/">visualization</a> of <em>RNN</em>, <em>Attention</em> etc</li><li><a href="https://github.com/keitakurita/Practical_NLP_in_PyTorch/blob/master/deep_dives/transformer_from_scratch.ipynb">Building</a> a Transformer from scratch</li><li>Excellent <a href="https://github.com/huggingface/transformers">library</a> for using various Transformer architectures</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=df2ee75e189" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/recent-advancements-in-nlp-2-2-df2ee75e189">Recent Advancements in NLP</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Recent Advancements in NLP (1/2)]]></title>
            <link>https://medium.com/swlh/recent-advancements-in-nlp-1-2-192ac7eefe3c?source=rss-e747ca1923f5------2</link>
            <guid isPermaLink="false">https://medium.com/p/192ac7eefe3c</guid>
            <category><![CDATA[word-embeddings]]></category>
            <category><![CDATA[rnn]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Moiz Saifee]]></dc:creator>
            <pubDate>Tue, 10 Dec 2019 04:22:12 GMT</pubDate>
            <atom:updated>2019-12-10T16:57:43.116Z</atom:updated>
            <content:encoded><![CDATA[<p>2018 has been widely touted as the “Imagenet moment” in the field of NLP due to the release of pre-trained models and open source algorithms by big players like Google, Open AI, Facebook etc. I have tried to summarize some of the important advancements in the field of NLP in this blog.</p><p>I have spent a good fifteen years in the field of Data Science and AI and have worked a fair bit on NLP in this period, but the pace with which things have been moving in the last 2–5 years is unprecedented. If you have not caught up with all the exciting work, you may be lagging behind significantly on what’s possible these days.</p><p>NLP is one of my favorite areas given its wide applications. Some of the popular applications by which we interact every day are — search engines, machine translation, chatbots, home assistants etc.</p><p>This is my first blog in a series of two blogs to capture the exciting work in the field of NLP</p><h3><strong>NLP is Hard</strong></h3><p>In addition to being tremendously useful, NLP is also pretty challenging, consider the following few examples:</p><p><a href="https://en.wikipedia.org/wiki/Buffalo_buffalo_Buffalo_buffalo_buffalo_buffalo_Buffalo_buffalo"><em>“Buffalo buffalo Buffalo buffalo buffalo buffalo Buffalo buffalo”</em></a> is a grammatically correct sentence</p><p><em>“John saw the man on the mountain with a telescope”</em> Who is on the mountain? John, the man, or both? Who has the telescope? John, the man, or the mountain?</p><p>These challenges in interpreting language can cause <a href="https://www.apple.com/in/siri/">Siri</a> to do something like this</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/401/1*fyePFVClx6_84OTBLch6IQ.png" /></figure><p>I have summarized some of the standard techniques which were used in the past along with some popular recent ones.</p><h3><strong>Pre 2012:</strong></h3><ul><li>Using the bag of words representation which paid no heed to the ordering of words was the norm</li><li>Latent Semantic Indexing (LSI), SVM (Support Vector Machine) and Hidden Markov Models (HMM) were the popular techniques</li><li>Techniques like LSI, LDA, PCA, helped bring the notion of semantic similarity by focusing on “topics” rather than words</li><li>Standard tricks of the trade were — POS tagging, NER etc</li></ul><h3><strong>2013–2016</strong></h3><p>One of the more important developments in this period was work on “word embedding”.</p><p><strong>Word Embedding</strong></p><p>Word Embedding is nothing but useful numerical representation of words. While the early work on this concept was done as far back as 2003 (Bengio et al.), but it was really work by Tomas Nikolov et al (from Google) in 2013 on <a href="https://arxiv.org/abs/1301.3781">Word2Vec</a> which gave it a real push.</p><p>One important point to note is, for many word embedding algorithms — while the representation is “learned” by training a model on certain task, example — to predict the next word given surrounding words — it turns out these learned representations give great performance in a variety of other NLP tasks</p><h4><strong>Word2Vec (2013) by Google</strong></h4><p>Uses Neural Network to come up with the word representations. There are two popular variants:</p><ol><li>Continuous bag of words (CBOW) — In this approach we try to predict a given word given its neighboring words (context)</li><li>Skip Gram — The approach is similar, but we solve the reverse problem — given target words, try to come up with neighboring words</li></ol><p>Word2Vec has a lot of nice and desirable benefits:</p><ul><li>The original model was trained on 1.6 B word dataset and with millions of words in the vocabulary</li><li>The training is pretty fast</li><li>Even people with modest amount of data could use these pre-calculated embeddings (generated by training on huge data) in their specific tasks and walk away with a lot of improvement</li><li>The Word2Vec training itself was unsupervised, that is did not require pricey annotated data</li><li>The numerical representation of words which were similar in meaning were close to each other (had small cosine distance)</li><li>“Somewhat surprisingly, it was found that similarity of word representations goes beyond simple syntactic regularities. Using a word offset technique where simple algebraic operations are performed on the word vectors, it was shown for example that <em>vector(”King”) — vector(”Man”) + vector(”Woman”)</em> results in a vector that is closest to the vector representation of the word <em>Queen”</em></li></ul><p>There was additional work which was done on Word Embedding after the impetus which it received with Word2Vec. The most notable variants are:</p><p><strong>GloVe — Global Vector for Word Representation (2014) by Stanford</strong></p><ul><li>Attempted to combine benefits of both — global matrix factorization methods (like LSA), which do well on capturing statistical structure and local context windows methods like Word2Vec, which do well on analogy tasks.</li><li>Instead of extracting numerical representations from training a neural network for certain task like predicting the next word, GloVe vectors have inherent meaning, which is derived from word co-occurrences</li></ul><p><strong>FastText (2016) by Facebook</strong></p><ul><li>Its conceptually somewhat similar to Word2Vec, but with a twist — instead using words for creating embeddings, it uses n-gram of characters. For example, fastText representation of word “test” with n=2 would be &lt;t, te, es, st, t&gt;</li><li>One of the major benefits of FastText is, the above approach allows it to generalize to unknown words, or the words which were not part of vocabulary in training</li><li>Requires lesser training data compared to Word2Vec.</li></ul><p>That’s it on embeddings — the key takeaway is you can find powerful numerical representations of words in almost any major language in the world, which are trained on huge text corpses and can give you a massive headstart in the NLP problem you are trying to solve, thanks to the algorithms listed above.</p><h3><strong>2014–2016</strong></h3><p>While a lot of work was happening around 2014 on embeddings, with the onset of Deep Learning era, another old technique was staging a comeback — the RNN</p><p><strong>RNN — Recurrent Neural Network</strong></p><p>RNN are a type of neural network that specialize in dealing with problems whose input / output are sequential in nature. The text data is inherently sequential given the fact that words are not placed randomly in a sentence and are informed by the grammar of the language. The idea of RNN was proposed as far back as 1980s, but they didn’t really take off given the challenges in training the model</p><p><strong>Why RNN?</strong></p><p>The basic challenge of classic feedforward neural network is that it has no memory, that is, each training example given as input to the model is treated independent of each other. In order to work with sequential data with such models — you need to show them the entire sequence in one go as one training example. This is problematic because number of words in a sentence could vary and more importantly this is not how we tend to process a sentence in our head.</p><p>When we read a sentence, we read it word by word, keep the prior words / context in memory and then update our understanding based on the new words which we incrementally read to understand the whole sentence. This is the basic idea behind the RNNs — they iterate through the elements of input sequence while maintaining a internal “state”, which encodes everything which it has seen so far. The “state” of the RNN is reset when processing two different and independent sequences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/991/1*tn_GY4JmlIC8rLAQArKLEg.png" /></figure><p>While all this sounds great, but the reality is simplistic RNN described above are rarely used in practice. The reason is — while in theory the RNN should be able to retain the information seen many timesteps before, in practice this is almost impossible to do due to the <em>vanishing gradient</em> problem. Vanishing gradient is the problem due to which deeper networks take forever to train as the updates in parameters of the network becomes painfully slow. Two variants of RNN — LSTM and GRU were designed to solve this problem.</p><p><strong>LSTM — Long Short Term Memory</strong></p><p>LSTM is a variants of RNN with a way for carrying over information across many timestamps without getting diluted (vanished) due to a lot of processing.</p><p>Without getting into too much mathematics, the way LSTM achieves this is by means of three gates — <em>Forget, Input</em> and <em>Output</em> Gate. The <em>Input</em> gate determines the extent to which the current timestamp input should be used , the <em>Forget</em> gate determines the extent to which output of the previous timestamp state should be used, and the <em>Output</em> gate determines the output of the current timestamp</p><p><strong>GRU — Gated Recurrent Unit</strong></p><p>The GRU introduced in 2014 by Kyunghyun Cho et al was a simplified version of LSTM with just two gates instead of 3 and with a far fewer parameters vs LSTM.</p><p>GRU have been shown to have better performance on certain tasks and smaller datasets, but LSTM tend to outperform GRU in general.</p><p>There were some additional variants like bidirectional LSTM which don’t just process text left to right, but also do it right to left to give additional performance boost.</p><h3>Closing Remarks</h3><p>There was a lot of important work which happened off late on Word Embedding and Neural Network algorithms like RNN. These developments led to some rapid progress in the field of NLP and significantly improved the performance on various benchmark datasets.</p><h3>But ..</h3><p>The last few years have seen the rise of a new family of models called <strong><em>Transformers — </em></strong>These <a href="https://arxiv.org/abs/1706.03762"><em>attention</em></a> based models have completely swamped most of the other things. Many of the leading players have open sourced their version of Transformer architecture based models — BERT (Google), Transformer Xl (Google), GPT-2 (OpenAI), XL Net (CMU), ERNIE (Baidu). I’ll try to cover these new exciting developments in my next blog.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=192ac7eefe3c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/recent-advancements-in-nlp-1-2-192ac7eefe3c">Recent Advancements in NLP (1/2)</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>