<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Charudatta Manwatkar on Medium]]></title>
        <description><![CDATA[Stories by Charudatta Manwatkar on Medium]]></description>
        <link>https://medium.com/@charudattamanwatkar?source=rss-5583c854307e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*SWih8T2kG-v6IOfdLl3CnQ.jpeg</url>
            <title>Stories by Charudatta Manwatkar on Medium</title>
            <link>https://medium.com/@charudattamanwatkar?source=rss-5583c854307e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 00:38:31 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@charudattamanwatkar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Class-Incremental Learning: Challenges and innovations]]></title>
            <link>https://charudattamanwatkar.medium.com/class-incremental-learning-challenges-and-innovations-10e1fa0600d8?source=rss-5583c854307e------2</link>
            <guid isPermaLink="false">https://medium.com/p/10e1fa0600d8</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[incremental-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[image-classification]]></category>
            <dc:creator><![CDATA[Charudatta Manwatkar]]></dc:creator>
            <pubDate>Mon, 08 Jan 2024 17:43:54 GMT</pubDate>
            <atom:updated>2024-01-08T18:01:44.923Z</atom:updated>
            <content:encoded><![CDATA[<h3>Class-Incremental Learning</h3><h4>A survey of challenges and recent innovations</h4><p>Imagine you have a pre-trained image classification model for dog vs. cat classification, but you don’t have access to the original dataset. Now, you have acquired a new dataset containing lions &amp; tigers and you want to extend your model to classify images among <em>all four classes</em>, i.e. dog vs. cat vs. lion vs. tiger. How would you achieve this?</p><p>Keep in mind that at inference time, you will not have the information about whether an unseen image belongs to (cat/dog) pair or (lion/tiger) pair. <em>So you can’t just train 2 separate models and call it a day</em>. Fine-tuning the model naively on the new data is also not a good idea, as we will see later in the article.</p><h3>Why is this problem practically relevant?</h3><h4>Changing business requirements</h4><p>In a real world setting, the problem statement for an ML task is often evolving with time. As your organization grows, you may need to enhance the capabilities of existing models to cater to more clients.</p><h4>Data Silos</h4><p>Often different parts of the relevant data can not be combined directly because of</p><ol><li>Legal Requirements: For example, different organizations may not be allowed to share their data with each other due to privacy laws.</li><li>Conflict of interest: Data may be split between organizations that are in competition with each other. So, they may be unwilling to pool their data directly with each other, but want to reap the benefits of joint training.</li></ol><p>Thus you may have access to only a subset of classes in the data at a time.</p><h4>Training Efficiency</h4><p>Even if you could access previous data, it would waste a lot of time, money, and efforts if you have to retrain your model from scratch every time you want to add new capabilities to it.</p><h3>Some Terminology</h3><p>Let’s look at some common terminology used in the literature.</p><ul><li><em>Task-IL </em>: A subset of incremental learning with the aim to train a single model that can perform multiple related tasks (e.g. Translation, Question-Answering, etc.) where the specific task-ID is known at the inference time.</li><li><em>Class-IL</em> : Another subset of incremental learning with the aim to extend the model capabilities, but the task-ID is <em>not</em> known at the inference time. An example is the problem mentioned at the beginning.</li><li><em>Learning Task </em>: In case of Task-IL, it’s clear what a ‘task’ refers to. In Case of Class-IL, a ‘task’ refers to a set of classes which the model has to learn together at a time. E.g. (cat/dog) is one task and (lion/tiger) is another task in the example at the beginning.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h8ZT95WjBqBdStPzoIAkKg.png" /><figcaption>Image source: <a href="https://arxiv.org/pdf/2010.15277.pdf">https://arxiv.org/pdf/2010.15277.pdf</a></figcaption></figure><h3>Primitive Solutions</h3><p>The following are some solutions that you may come up with, off the top of your head, but wouldn’t work very well</p><h3>Multiple Models</h3><ol><li>Train a separate model for each task</li><li>At inference time, pass the data to each model and take the class with highest output value.</li></ol><h4>Pros</h4><ul><li>Training is fast</li><li>Approach is simple to understand</li></ul><h4>Cons</h4><ul><li>Inference is very slow (need to pass data through several model).</li><li>Several models need to be stored.</li><li>Suffers from Inter Task Confusion</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BQVliBJsyQQ5SqSCN2b6nw.png" /><figcaption>This is an example of Inter-task confusion. A model was first trained on (Square vs rhombus) task and another on (circle vs triangle) task. However, neither model ever learned to discriminate between squares and circles. Source: <a href="https://arxiv.org/pdf/2010.15277.pdf">https://arxiv.org/pdf/2010.15277.pdf</a></figcaption></figure><h3>Fixed Representations</h3><ol><li>Training:<br>- Train a classifier for the very first task from scratch.<br>- For the new classes, remove previous classification nodes/layers and add new nodes/layers (untrained)<br>- Freeze the previous layers and only train the new nodes/layers on new data.</li><li>For inference, append the old nodes/layers back</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HbBh8ov-Alc-RTwHNuFRrg.png" /><figcaption>Source: <a href="https://arxiv.org/pdf/1810.12448.pdf">https://arxiv.org/pdf/1810.12448.pdf</a></figcaption></figure><h4>Pros</h4><ul><li>Training is Fast</li><li>Inference is fast</li><li>Preserves model performance on old classes</li></ul><h4>Cons</h4><ul><li>Model performance is poor for newer classes. This is referred to as <strong>intransigence</strong>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*meD_HWi7_QrjKpQDHeK_eQ.png" /><figcaption>Model performance by class with Fixed Representation approach. Here, a new class was added every 500 epochs. The model performs well for old classes, but struggles to replicate the same performance for newer classes. Source: <a href="https://arxiv.org/pdf/1810.12448.pdf">https://arxiv.org/pdf/1810.12448.pdf</a></figcaption></figure><h3>Fine-tuning</h3><p>We use the same approach as above, but instead we train the whole network (instead of just the last later) for adding new classes.</p><h4>Pros</h4><p>Performance improves for new classes</p><h4>Cons</h4><p>Performance degrades significantly for old classes. This is also called <strong>Catastrophic Forgetting</strong> in the literature.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y_FpklL3Danzl5sDhZRrRg.png" /><figcaption>Model performance by class with naive fine-tuning approach. The model learns to recognize new classes, but its performance degrades slowly for older classes. Source: <a href="https://arxiv.org/pdf/1810.12448.pdf">https://arxiv.org/pdf/1810.12448.pdf</a></figcaption></figure><h3>Smarter Solutions</h3><p>Now, let’s look at some of the smarter solutions that researchers have come up with, in recent years. These solutions fall into 2 broad types:</p><ol><li><strong>Rehearsal Free Solutions</strong>: These solutions assume that we have no access to old data whatsoever.</li><li><strong>Rehearsal Based Solutions</strong>: These solutions store a tiny fraction of old data and use it to balance model training during new tasks.</li></ol><p>Note: In both these categories, we will append new layers for new classes, but otherwise keep the same architecture of the model.</p><h4>A simple baseline</h4><p>Researchers from University of Oxford have come up with a simple but interesting baseline called <a href="https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123470511.pdf">GDumb</a> (Greedy Sampler and Dumb Learner). They store a random fraction of examples from each old class (within a fixed tiny memory budget) and simply retrain a multi-class classifier from scratch for adding a new class. This naive approach beats several sophisticated approaches in their experiments.</p><h3>Elastic Weight Consolidation</h3><p>This is a rehearsal free solution.</p><p><strong>Core idea:</strong> Identify a subset of model weights which are important for a given task and avoid changing these in the subsequent tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*0zc6i3fS_tPT36jObf99CQ.png" /><figcaption>Visualization of EWC in the loss landscape. EWC attempts to learn the new task while still keeping the important weights unchanged for the old task. Source: <a href="https://arxiv.org/pdf/1612.00796.pdf">https://arxiv.org/pdf/1612.00796.pdf</a></figcaption></figure><p>The importance is calculated using <strong>Fisher information Matrix</strong> of the posterior distribution of model weights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/880/1*m__DMpYEN7p9hekdW_S-4A.png" /><figcaption>Here A and B denote the previous and current task respectively. Fᵢ denote the diagonal value from Fisher information matrix corresponding to weight θᵢ (Source: <a href="https://arxiv.org/pdf/1612.00796.pdf">https://arxiv.org/pdf/1612.00796.pdf</a>)</figcaption></figure><p>The Authors use the fact that Fisher matrix is equivalent to the second derivative of the loss near a minimum. They use the diagonal values in this matrix from the old task as the regularization constants in the next task.</p><p>Link to paper: <a href="https://arxiv.org/pdf/1612.00796.pdf">https://arxiv.org/pdf/1612.00796.pdf</a></p><h3>Path Integral Based Approaches</h3><p>This is a group of rehearsal free approaches.</p><p><strong>Core idea:</strong> in EWC, the weight importances are calculated only once the model is fully trained on a task. These approaches instead calculate the weight importance along the entire training trajectory.</p><p>Examples: [<a href="https://arxiv.org/abs/1801.10112">RWalk</a>] [<a href="https://arxiv.org/pdf/1711.09601.pdf">Memory Aware Synapses</a>]</p><h3>Learning Without Forgetting</h3><p>This is a rehearsal-free solution.</p><p><strong>Core idea</strong>: This is a combination of Knowledge Distillation and Fine-tuning.</p><h4>Knowledge distillation</h4><ol><li>Take a trained model for an old task.</li><li>Discard the data for the old task.</li><li>Record the outputs of the trained model over <strong><em>new</em></strong> <strong><em>inputs</em></strong>. These input/output pairs allow us to “distill” the knowledge of the old task into new data.</li></ol><h4>Fine-tuning</h4><ul><li>The outputs of the layers corresponding to old task should match with the “distilled” outputs.</li><li>The layers corresponding to new tasks should minimize classification loss.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*3sy3-f_EO5LnlE03T_C8LQ.png" /><figcaption>Learning without forgetting. Source: <a href="https://arxiv.org/pdf/1606.09282.pdf">https://arxiv.org/pdf/1606.09282.pdf</a></figcaption></figure><p>Source: <a href="https://arxiv.org/pdf/1606.09282.pdf">https://arxiv.org/pdf/1606.09282.pdf</a></p><h3>Learning Without Memorizing</h3><p>This is a rehearsal-free solution.</p><p><strong>Core idea</strong>: Similar to Learning Without Forgetting, but use attention maps in addition to final outputs for knowledge distillation.</p><p>Attention map is a heat-map over the image that tells us which pixels were most important to determine the output. Here, they are generated using the Grad-CAM approach.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*52unyypTbS4ynXLxT7a1tQ.png" /></figure><blockquote>The authors say “We hypothesize that attention regions encode the models’ representation more precisely […] instead of finding which base classes are resembled in the new data, attention maps explain ‘why’ hints of a base class are present.”</blockquote><p>Source: <a href="https://arxiv.org/pdf/1811.08051.pdf">https://arxiv.org/pdf/1811.08051.pdf</a></p><h3>Incremental Classifier and Representation Learning (iCaRL)</h3><p>This is a very important rehearsal-based solution which has had a big impact on the field of Class-IL.</p><h4>Initial Training</h4><ol><li>Train a classifier model.</li><li>Choose a few “exemplar” images of each class (and save them)</li><li>Consider their representation in an intermediate layer of the model (i.e. vector encoding of the image)</li><li>Calculate the mean of such encodings for each class (called “class prototype”)</li></ol><h4>Inference</h4><ol><li>Calculate the vector encoding of the unseen image</li><li>Find the class prototype nearest to this encoding (as measured by L2 distance)</li><li>Classify the image into the class of this prototype.</li></ol><h4>New class addition</h4><ol><li>Fine-tune the model with images from new classes. Also choose their exemplars.</li><li>Choose new exemplars in each old class such that their prototype is as close to their old prototype as possible.</li></ol><h4>Exemplar management</h4><p>This method maintains a few images from each class within a fixed total memory budget. As new classes are added, exemplars from old classes are removed to make room for the exemplars in the new class.</p><p>Source: <a href="https://arxiv.org/pdf/1611.07725.pdf">https://arxiv.org/pdf/1611.07725.pdf</a></p><h3>Extension to other areas of ML</h3><p>The techniques we discussed here were all developed with image classification as the focus. A lot of these ideas are directly portable to other areas like image segmentation, object detection, etc.</p><p>However, these areas also present challenges unique to them. They have given rise to many interesting avenues of research. I will try to discuss them in a separate article.</p><h3>Summary</h3><p>Class-incremental Learning is a fascinating and practically important area of research. It has its own challenges like Catastrophic Forgetting, Intransigence, Inter-task confusion, etc. Researchers have made great progress in dealing with these problems by a combination of various techniques like regularization, knowledge distillation, careful exemplar selection, etc. Learning about these ideas will provide great insights about concepts like model capacity, model compression, generalization, and so on.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=10e1fa0600d8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hidden Gems of Python]]></title>
            <link>https://medium.com/data-science/hidden-gems-of-python-76020b14e42f?source=rss-5583c854307e------2</link>
            <guid isPermaLink="false">https://medium.com/p/76020b14e42f</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[easter-egg]]></category>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[hidden-feature]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Charudatta Manwatkar]]></dc:creator>
            <pubDate>Tue, 06 Apr 2021 13:53:25 GMT</pubDate>
            <atom:updated>2021-04-08T18:16:08.662Z</atom:updated>
            <content:encoded><![CDATA[<h4>Features of Python that I had no idea even existed</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EGGjZrrAvQ9400GZ" /><figcaption>Photo by <a href="https://unsplash.com/@dearseymour?utm_source=medium&amp;utm_medium=referral">Ksenia Makagonova</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>These days I have a newfound pastime — reading Python documentation just for fun! When you read something at leisure, you tend to notice interesting tidbits that you would have missed otherwise. So, here is the list of tidbits that made me go “Oh! You can do that in Python?”</p><h3>1. Function Attributes</h3><p>Similar to how you can set attributes of classes and objects, you can also set attributes to functions.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4f2d99bdc09c42d8639ef95b3b702244/href">https://medium.com/media/4f2d99bdc09c42d8639ef95b3b702244/href</a></iframe><p>We have set the attributes ‘optional_return’ in line 10 and ‘is_awesome’ in line 11. We have accessed those attributes outside of the function later in lines 19 &amp; 20. The output of the code will be:</p><pre>Final answer is 2197<br>Show calculations --&gt; 13<br>Is my function awesome? --&gt; Yes, my function is awesome.</pre><p>This comes in handy when you want the option to retrieve some intermediate variable, but not explicitly return it with the return statement every time the function is called. Also note that attributes can be set from inside the function definition or from outside the function definition.</p><h3>2. For-else loop</h3><p>In Python, you can add an <em>else</em> clause to a <em>for</em> loop. The <em>else </em>clause will be triggered only if no <em>break</em> statement was encountered within the body of the loop during execution.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c484623f9e8153d7c17203b118470d26/href">https://medium.com/media/c484623f9e8153d7c17203b118470d26/href</a></iframe><pre>All elements at least 3 letters long</pre><p>Notice that the <em>else</em> is indented at the level of <em>for</em> and not at the level of <em>if</em>. Here, no element has length shorter than 3. So, the <em>break</em> statement will never be encountered. Hence, the <em>else</em> clause will get triggered (after the <em>for</em> loop is executed) and print the output shown above.</p><p>One could argue that this could be achieved using a separate variable that keeps track of whether the <em>break</em> statement was encountered. And perhaps it would also be less confusing for the next person reading the code. The following is the equivalent way of achieving the same result:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/77141bf17ceaef3705f14479ab0d40d0/href">https://medium.com/media/77141bf17ceaef3705f14479ab0d40d0/href</a></iframe><p>Still, it’s good to know, I guess.</p><h3>3. Separators for ‘int’</h3><p>It’s hard to distinguish visually between integers like 10000000 and 100000000 (are they even different numbers?). We can’t use commas here in Python like we use them in English because Python will interpret it as a <em>tuple </em>of multiple integers.</p><p>Python has a very convenient way of dealing with this: we can use <em>underscore </em>as a separator to improve readability. Thus, 1_000_000 will be interpreted as a single <em>int</em>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6fe1330d30776cf7f7e01df31a0c237e/href">https://medium.com/media/6fe1330d30776cf7f7e01df31a0c237e/href</a></iframe><pre>&lt;class &#39;int&#39;&gt;<br>&lt;class &#39;int&#39;&gt;<br>True</pre><h3>4. eval() and exec()</h3><p>Python has the ability to dynamically read a string and treat it like a piece of Python code. This is achieved using <em>eval</em>() and <em>exec</em>() functions (‘eval’ is for evaluating <em>expressions</em> and ‘exec’ is for executing <em>statements</em>).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dcaafe211434a4bd36f3097d22375fd2/href">https://medium.com/media/dcaafe211434a4bd36f3097d22375fd2/href</a></iframe><pre>b = 5<br>c is 9</pre><p>In line 3, <em>eval</em>() function will read the input string as a Python expression, evaluate it, and assign the result to the variable ‘b’. In line 6, <em>exec</em>() function will read the input string as a Python statement and execute it.</p><p>You can even pass dynamically created strings to these functions. You could, for example, create 1000 variables with names x_0, x_1, …, x_999 without having to write each of those variable declarations manually in your code. This might look like it’s a completely pointless feature, but it’s not.</p><blockquote>In the broader context of programming in general, not just Python, the use of eval/exec is incredibly powerful because it allows you to write dynamic code that uses information available at runtime to solve problems which cannot even be expressed at compile time. […] exec is literally a Python interpreter embedded inside Python, so if you have a particularly hard problem to solve, one of the ways you can solve it is to write a program to *write a program to solve it*, then use exec to run that second program.</blockquote><p>You can read more of <a href="https://mail.python.org/pipermail/tutor/2015-February/104238.html">this beautiful explanation</a> by Steven D’Aprano.</p><h3>5. Ellipsis</h3><p><em>Ellipsis </em>or ‘…’ is a built-in constant of Python similar to built-in constants like <em>None</em>, <em>True</em>, <em>False</em>, etc. It can be used in different ways including, but not limited to:</p><h4>5.1 Placeholder for unwritten code</h4><p>Similar to <em>pass, </em>Ellipsis can be used as a placeholder when the code is not fully written, but requires some placeholder for syntactic accuracy</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1fd068dd9a3d3575e37c4ff48a5e3fba/href">https://medium.com/media/1fd068dd9a3d3575e37c4ff48a5e3fba/href</a></iframe><h4>5.2 Alternative to ‘<em>None’</em></h4><p><em>None </em>is the usual choice when one wants to denote an empty input or return. But sometimes <em>None </em>can be one of the expected inputs or returns of a function. In that case <em>Ellipsis </em>can serve as the placeholder.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d9367b7b4fe80ad959f321df52ed4e08/href">https://medium.com/media/d9367b7b4fe80ad959f321df52ed4e08/href</a></iframe><p>The function <em>nth_odd() </em>calculates the nth odd number, given n. The function <em>original_num</em>() calculates the original number n, given the nth odd number. Here, <em>None </em>is one of the expected inputs to the function <em>original_num</em>() so we can’t use it as the default placeholder for the argument <em>m. </em>The output of the code will be:</p><pre>This function needs some input<br>Non integer input provided to nth_odd() function<br>9 is 5th odd number<br>16 is not an odd number</pre><h4>5.3 Array slicing in NumPy</h4><p>NumPy uses <em>Ellipsis </em>to slice an array. The following code shows two equivalent ways of slicing a NumPy array:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dd762da0f297260c45a7a4bdd3498040/href">https://medium.com/media/dd762da0f297260c45a7a4bdd3498040/href</a></iframe><pre>[ 0  2  4  6  8 10 12 14]<br>[ 0  2  4  6  8 10 12 14]</pre><p>Thus, ‘…’ signifies that there are as many ‘:’s as needed.</p><h4>Boolean value of Ellipsis</h4><p>In contrast to <em>None (</em>whose Boolean values is <em>False), </em>the Boolean value of <em>Ellipsis </em>is taken to be <em>True.</em></p><h3>TL;DR</h3><p>In summary, the interesting features I found thus far are:</p><ol><li>Function Attributes: Assign attributes to functions just like objects</li><li>For-else loop: To keep track of whether the loop was executed without <em>break </em>statement</li><li>Separators for <em>int</em>: 32_534_478 is an <em>int</em></li><li><em>eval</em>() and <em>exec</em>(): read strings as Python code and run it</li><li><em>Ellipsis</em>: A versatile built-in constant.</li></ol><h3>Parting Words</h3><p>Python is not just a useful language, but also a really interesting one. We are all busy in our lives, but it doesn’t hurt to get to know the language better for its own sake. I would love to know about some Easter-eggs that you might have found.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*_NAKVRrH5z7t8W0yruZJZQ.jpeg" /><figcaption>Image posted by <a href="https://www.reddit.com/user/ANewMuleSkinner/">u/ANewMuleSkinner</a> on <a href="https://www.reddit.com/r/seinfeld/comments/2g94iv/are_you_reading_my_vcr_manual/">reddit</a></figcaption></figure><h3>Resources</h3><p>You can read more about the things I mention above on the following Python documentation pages:</p><p>[1] <a href="https://docs.python.org/3/reference/datamodel.html#:~:text=Special%20read%2Donly%20attributes%3A%20__,in%20or%20None%20if%20unavailable.">Function attributes</a></p><p>[2] <a href="https://docs.python.org/3/tutorial/controlflow.html#break-and-continue-statements-and-else-clauses-on-loops">For-else</a></p><p>[3] <a href="https://docs.python.org/3/library/functions.html#eval">Eval/exec</a></p><p>[4] <a href="https://docs.python.org/3/library/constants.html#Ellipsis">Ellipsis</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=76020b14e42f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/hidden-gems-of-python-76020b14e42f">Hidden Gems of Python</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to give a meaningful Data Science interview (or any interview)]]></title>
            <link>https://medium.com/data-science/how-to-give-a-meaningful-data-science-interview-or-any-interview-849eeca8fe9?source=rss-5583c854307e------2</link>
            <guid isPermaLink="false">https://medium.com/p/849eeca8fe9</guid>
            <category><![CDATA[career-advice]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[interview-tips]]></category>
            <category><![CDATA[interview]]></category>
            <category><![CDATA[communication]]></category>
            <dc:creator><![CDATA[Charudatta Manwatkar]]></dc:creator>
            <pubDate>Tue, 22 Dec 2020 18:30:31 GMT</pubDate>
            <atom:updated>2020-12-24T05:19:55.565Z</atom:updated>
            <content:encoded><![CDATA[<h4>Avoid these mistakes that I observed from 100+ interviews</h4><p>Over the course of the last two years, I have interviewed a lot of candidates for entry-level data science roles. I noticed a few completely avoidable mistakes being repeated frequently by many candidates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bGNt38a7MwESqPd2" /><figcaption>Photo by <a href="https://unsplash.com/@mkvandergriff?utm_source=medium&amp;utm_medium=referral">Maranda Vandergriff</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Here are some of those mistakes and ways to avoid them.</p><h3>#1: Diving way too deep, way too early</h3><p>Too often I see the following conversation:</p><p>“<em>Can you tell me a little bit about XYZ project on your resume?</em>”</p><p>“OK, <em>so we start by converting PDF to images using pdf2image library and then run Tesseract OCR on it to convert the image into text. Then we start to preprocess the data by stemming, lemmatizing, blah blah blah …</em>”</p><p>Don’t do this. This is very confusing for the interviewer. They have no context about the project. They are not really even interested in the project (yet).</p><p>They may pretend as if they understand what you are saying, but they don’t. So they just start waiting for you to finish and move onto the their questions.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FNPF4KKIGlxJl0zyoiO%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2FSWR-Kindernetz-feuerwehr-fire-fighter-alarm-die-jungen-retter-NPF4KKIGlxJl0zyoiO&amp;image=https%3A%2F%2Fmedia4.giphy.com%2Fmedia%2FNPF4KKIGlxJl0zyoiO%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/81863b5fcfa5446400d74ca9091bad13/href">https://medium.com/media/81863b5fcfa5446400d74ca9091bad13/href</a></iframe><h4>Start with an elevator pitch</h4><p>The best way to begin to explain a project (in my opinion) is with an elevator pitch. <strong><em>Your opening statement should have enough details to rouse curiosity in the listener, but no more!</em></strong></p><p>This gives the interviewer the choice to dive deep into the project or move onto something else.</p><p>An easy way to do this is to answer each of the following questions in ONE short(ish) sentence:</p><ol><li><em>If the project did not exist, why would the world be worse off?</em></li><li><em>How does the project solve that?</em></li><li><em>How did you measure the project’s impact?</em></li></ol><p>For example:</p><p>“When a new customer joins our client, they have to manually extract a lot of information from poorly scanned and unstructured documents. Our project automates this process by extracting relevant information like names &amp; addresses, etc. We achieved 90% accuracy and saved more than 5000 work-hours per year.”</p><h4>Ask for a deep dive</h4><p>Now that the interviewer is finally in a position to understand your project in depth you can ask if the they want to dive deep in to the project.</p><p>If they say yes, shine bright!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/1*VITweLlR2mrqcJn7T-7oew.jpeg" /><figcaption>This should be the mental state of your interviewer before you bombard them with details. <a href="https://memegenerator.net/instance/48782974/you-had-my-curiosity-dicaprio-madam-you-had-my-curiosity-but-now-you-have-my-attention">Source</a></figcaption></figure><p>Be sure to break down your project into familiar subtasks that the interviewer can understand, e.g. classification, regression, object detection, clustering, etc. For example:</p><p>“We convert the image to text by OCR. Then <strong>classify </strong>the text into several categories. For each category, we perform <strong>Named Entity Recognition</strong> to retrieve person names, followed by <strong>object detection </strong>to extract their signatures.”</p><p>This helps the interviewer to quickly link their existing knowledge to your particular problem. They may choose to focus on the part which seems the most interesting to them.</p><p>For each of those subtasks which you end up discussing, here is a (non-exhaustive) list of things you should convey.</p><ol><li>Data:</li></ol><pre>a) Source: Where did you get the data? Was it an open source dataset? Or did you collect the data yourself?<br>    <br>b) Characteristics: Was there an imbalance in the dataset? Did it have too many missing values?</pre><pre>c) Preprocessing</pre><p>2. Algorithm:</p><pre>a) Well known algorithm (like YoLo) or designed from scratch?</pre><pre>b) Transfer Learning or trained from scratch?</pre><pre>c) How much training?</pre><pre>d) Any pre-training?</pre><p>3. Performance</p><pre>a) Which metrics were used? Why those metrics?</pre><pre>b) How much performance was achieved on those metrics?</pre><pre>c) How did this performance compare with the business requirements? Were all the stakeholders satisfied?</pre><p>4. Unique Constraints</p><pre>a) Data privacy</pre><pre>b) production/development environments constraint</pre><pre>c) Issues with handling scale (of the data or the model)</pre><h4>Think about the project critically from a third person’s POV</h4><p>The thing to constantly keep in mind is that the interviewer here is an outsider looking in. What is obvious to you may be confusing to them. You have lived through the project, they have not.</p><p>The objective here is to explain your work as clearly as possible, not to (just) brag about the complexity of the project or your technical skills.</p><h3>#2: Not understanding the context of the question</h3><p>The same question can have different right answers under different contexts. As they say,</p><blockquote>“Before climbing the ladder to success, make sure that it is leaning against the right building”</blockquote><p>For every question, think of the following:</p><h4>Do I understand the question *<em>exactly* ?</em></h4><p>Answering a question without understanding its contents is worse than giving a wrong answer.</p><p>Unless the question is extremely obvious, rephrase it in your own words and confirm with the interviewer if you are on the same page. If you are not sure about a specific part of the question, ask for clarification.</p><p>This is also helpful in case you don’t know the answer, because you can at least show that you understand the content of the question.</p><h4>Is this an open-ended question?</h4><p>Ideally, the interviewer should make this clear in the question itself. In case they don’t, don’t hesitate to confirm.</p><p>I was myself stumped in an interview when an interviewer asked me how I would build a particular solution. It didn’t occur to me that it was an open ended question and that I was allowed to answer with my imagination.</p><h4>Is this question connected to a previous question?</h4><p>Sometimes an interviewer asks a follow-up question if the candidate was struggling with a previous question. This actually makes your job easier, as the search-space for the answer is reduced if you know that it is linked to a previous question.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FJWF7fOo3XyLgA%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2Ffunny-reaction-JWF7fOo3XyLgA&amp;image=https%3A%2F%2Fmedia4.giphy.com%2Fmedia%2FJWF7fOo3XyLgA%2F200.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="243" frameborder="0" scrolling="no"><a href="https://medium.com/media/53f28d2f2cf21103ef5b3d91c5f5f415/href">https://medium.com/media/53f28d2f2cf21103ef5b3d91c5f5f415/href</a></iframe><h3>#3: Not understanding the context of the interview</h3><p>It’s important to tailor your answers to the interview in which they are asked. <strong><em>Your answers may be technically correct, but not productive for the context of the interview. </em></strong>Keep in mind the following things:</p><h4>Type of interview</h4><p>Is this a brief telephonic interview or is it a deep technical interview? Short &amp; crisp answers are expected in the former while detailed &amp; nuanced answers in the latter.</p><h4>Who is the interviewer?</h4><p>Is the interviewer your potential team-mate or say, the CEO?</p><p>The team-mate may be more interested in the technical details of the answer, but the CEO may be interested in your overall understanding of the problem.</p><p>This is just my guess, so take it with a pinch of salt!</p><h3>#4: Misplaced priorities</h3><h4>The final answer (almost) doesn’t matter</h4><p>A good interviewer is looking for a competent human, not a parrot. The final answers to questions in a (good) interview are almost irrelevant. What matters is how well you can navigate toward the answer with the given information.</p><h4>Don’t bluff</h4><p>I can’t believe I have to spell it out, but DO NOT BLUFF. If you don’t know the answer to a question, just say so plainly! It’s perfectly OK. Nobody has unlimited knowledge.</p><p><strong><em>It’s painfully obvious when someone is bluffing. </em></strong>The interviewer has thought through their own questions carefully. They have asked the same question to multiple other people. They are not under pressure, you are. You can count on them to catch the bluff.</p><h4>Demonstrate you thought process instead</h4><p>That being said, if you don’t know the answer to a question, you don’t need to draw a complete blank either. You can say something like</p><blockquote>“I don’t know the answer exactly, but my best guess is XYZ because of ABC reasons.”</blockquote><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2F6oR9lrRFIGFu8%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fgiphy.com%2Fgifs%2F6oR9lrRFIGFu8&amp;image=https%3A%2F%2Fmedia4.giphy.com%2Fmedia%2F6oR9lrRFIGFu8%2F200.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="261" frameborder="0" scrolling="no"><a href="https://medium.com/media/9a237907a672baaa693d5c5cb0fbf5de/href">https://medium.com/media/9a237907a672baaa693d5c5cb0fbf5de/href</a></iframe><p>Even though you may not know the answer, you may still display your ability to think clearly about a problem and work towards a solution.</p><h3>Conclusion</h3><ol><li>Don’t bombard the interviewer with details without context.</li><li>Understand the question thoroughly.</li><li>Understand the relationship between the question and the interview.</li><li>Demonstrating a well-grounded thought process is more important than knowing the final answer.</li></ol><p>The idea is to make it really easy for the interviewer to hire you. These are simply my views and by no means hard &amp; fast rules. Please do let me know if there is anything you don’t agree with and would like to discuss further with me. Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=849eeca8fe9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-give-a-meaningful-data-science-interview-or-any-interview-849eeca8fe9">How to give a meaningful Data Science interview (or any interview)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A novel evaluation technique for Named Entity Recognition (NER)]]></title>
            <link>https://medium.com/data-science/a-pathbreaking-evaluation-technique-for-named-entity-recognition-ner-93da4406930c?source=rss-5583c854307e------2</link>
            <guid isPermaLink="false">https://medium.com/p/93da4406930c</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[named-entity-recognition]]></category>
            <category><![CDATA[ner]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <dc:creator><![CDATA[Charudatta Manwatkar]]></dc:creator>
            <pubDate>Sat, 12 Dec 2020 17:27:42 GMT</pubDate>
            <atom:updated>2021-05-05T15:07:24.049Z</atom:updated>
            <content:encoded><![CDATA[<h4>New research explained</h4><h3>A better way to evaluate Named Entity Recognition (NER) models</h3><h4>Precision, recall, and F-score won’t get you far. Use these new insightful metrics instead.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*f0XMtMGVUpv1Ii8W" /><figcaption>Interpretable and fine-grained evaluation techniques are crucial for making well-grounded improvements in any ML model. Photo by <a href="https://unsplash.com/@kent_pilcher?utm_source=medium&amp;utm_medium=referral">Kent Pilcher</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Prologue</h3><p>NER is typically approached as a sequence labelling problem and the models for it are usually evaluated through traditional classification metrics such as precision, recall, F-score, etc. You can read a primer on the topic in <a href="https://medium.com/swlh/a-beginners-introduction-to-named-entity-recognition-ner-2002b1a010c1">this post</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/1*2LPvuGgdMkCEo-s-kEJ2nw.png" /><figcaption>Example of Named Entity Recognition in action. Image by author.</figcaption></figure><h3>The problem with traditional evaluation metrics</h3><p>These simple metrics are fine if you just want a quick comparison of different models, but they are really opaque and don’t tell you a lot. They don’t tell you anything about what are the strengths and weaknesses of the model or what characteristics of the data are most influential on the model performance.</p><p>You are left guessing whether the model is suffering due to long sentence length, or is it because there are too many (or too few) entities, the entities are spread out too far (or too close), or the entities are highly ambiguous, etc.</p><h3>New Development</h3><p>To tackle this problem, researchers from Carnegie Mellon University and Fudan University, China have come up with a novel evaluation technique which tackles some of these problems. You can read the <a href="https://arxiv.org/pdf/2011.06854.pdf">original paper here</a>.</p><p>The authors have also released an <a href="https://github.com/neulab/InterpretEval">open source repository</a> that has the implementation of the concepts in this paper.</p><p>This article is my attempt to explain the findings in this paper to a broader audience in a simplified way. It is in no way a complete and exhaustive reproduction of the paper. <strong>The original paper is rich in details and nuances, so do give it a read to know more</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/867/1*G6oUQwldDFSCQNeRaVlD0A.png" /><figcaption>This recent paper by Jinlan Fu (Fudan University), Pengfei Liu and Graham Neubig (Carnegie Mellon University) presents new evaluation techniques for NER that are interpretable and fine-grained.</figcaption></figure><h3>Core idea</h3><p>The main idea in the paper is to divide the data into <strong><em>buckets</em></strong><em> </em>of entities based on <strong><em>attributes</em></strong> such as entity length, label consistency, entity density, sentence length, etc. and then <strong>evaluate the model on each of these buckets</strong> separately.</p><p>This makes it easy to locate the factors where the model is performing (or not performing) well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*O6buZKinUAvBtJ-q" /><figcaption>The core idea of the paper is to divide the dataset into distinct buckets to understand what factors affect the model performance. Photo by <a href="https://unsplash.com/@pascalwhoop?utm_source=medium&amp;utm_medium=referral">Pascal Brokmeier</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Let’s go into some of these details.</p><h3>Attributes</h3><p>Attributes are different characteristics of the training examples, which may affect the performance of the model. These are:</p><h4>1) Local Attributes</h4><ul><li><strong>Entity Length</strong>: String length of the entity</li><li><strong>Sentence Length</strong></li><li><strong>OOV Density</strong>: Fraction of ‘Out of Vocabulary’ words in the sentence</li><li><strong>Entity Density</strong>: Number of entities in a sentence</li></ul><h4>2) Aggregate Attributes</h4><ul><li><strong>Token Label Consistency</strong>: This is a measure of how versatile an entity is. E.g. in</li></ul><blockquote>“I live in New York”</blockquote><p>‘New York’ is labelled as ‘LOC’, but in</p><blockquote>“I study in New York University”</blockquote><p>‘New York’ is a part of entity labelled ‘ORG’.</p><ul><li><strong>Token frequency</strong>: How many times the token appears in the dataset</li><li><strong>Entity Label Consistency<em> </em></strong>: Same as Token Label Consistency, but calculated for entity text instead of tokens.</li><li><strong>Entity Frequency<em> </em></strong>: Same as Token Frequency, but calculated for entity text instead of tokens.</li></ul><h3>Bucketing</h3><p>For any of the above-mentioned attributes, we may divide the dataset into as many buckets as we want (authors prefer 4 buckets). On each of these buckets, the model performance is measured by its micro-average F-1 score.</p><p>After gathering this diverse performance data, it’s time to find out the correlation between attributes and performance.</p><h3>Correlation between attributes and performance</h3><p>A typical reader would infer correlation between the model performance and the attribute values using Pearson Correlation Coefficient.</p><p>However the authors infer correlations not directly between F-1 scores and the attribute values, but between the <em>ranks of the F-1 scores</em> and the <em>ranks of the attribute values</em>.</p><p>My guess is that this allows the authors to create more convenient buckets which may not be strictly in the increasing order of the attribute values.</p><p>For this purpose, the authors use <a href="https://en.wikipedia.org/wiki/Spearman%27s_rank_correlation_coefficient">Spearman’s rank correlation coefficient</a>. To test the significance of these correlations, they use the <a href="https://en.wikipedia.org/wiki/Friedman_test">Friedman Test</a> and <a href="https://en.wikipedia.org/wiki/Wilcoxon_signed-rank_test">Wilcoxon signed-rank test</a> as appropriate.</p><p>To quote <a href="https://en.wikipedia.org/wiki/Andrew_Ng">a wise man</a>, “Don’t worry about it if you don’t understand it right away”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*tMXBqIzsbYwEDOjh" /><figcaption>To Spearman Rank Correlation Coefficient, rank is all that matters, not the actual value. Photo by <a href="https://unsplash.com/@joshgmit?utm_source=medium&amp;utm_medium=referral">Joshua Golde</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>Empirical Results</h3><p>With this framework, the authors carry out an extensive analysis of <strong>12 models, 8 attributes, and 6 datasets</strong>. The paper describes the results in great details. The analysis is roughly divided in 3 types:</p><h4>1] Holistic Analysis</h4><p>This analysis is similar to how traditional multi-dataset analysis are performed (without fine-graining into attributes or buckets). The authors argue that it does not create a very clear picture of what are the crucial factors for the performance.</p><h4>2] Fine-grained analysis</h4><p>This is the analysis which takes full advantage of the new ideas presented in the paper. After measuring the performance of all the models on all the buckets of all the datasets, the authors present 2 methods to make sense of all the results.</p><p><strong>2.1] Model-wise analysis</strong></p><p>Here, the authors talk about the Spearman’s Rank Correlation between model performance (F- score) and the attribute values. The authors make the following observations:</p><ul><li>Observation 1: <strong><em>The performance of character-unaware models is more sensitive to the label consistency.</em></strong></li><li>Observation 2: <strong><em>The influence of entity length varies greatly between different decoders.</em></strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*gENEUrh9YnNRCiqKo80ehw.png" /><figcaption>The table from the paper presents Spearman Rank Correlation Coefficient of model performance with attribute values. eDen, oDen, etc. are shorthand for attribute names like Entity Density, OOV Density, etc. Green/red values indicate correlation with Observation 1/Observation 2. Gray values indicate correlation which did not pass the Friedman Significance test. [Minor modifications to the <a href="https://arxiv.org/pdf/2011.06854.pdf">original table</a> by me for greater clarity]</figcaption></figure><p><strong>2.2] Attribute-wise analysis</strong></p><p>Here the authors discuss how an attribute affects the NER (independent of the model). They make the following observations:</p><ul><li>Observation 1: <strong><em>Label consistency and entity length have a more consistent influence on NER performance.</em></strong></li><li>Observation 2: <strong><em>Frequency and sentence length matter but are minor factors.</em></strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*PacjH_IBopgytyNs-RLFgA.png" /><figcaption>The distance from the center is proportional to the average of Spearman’s Rank Correlation Coefficient across all models. Each color corresponds to a particular dataset as specified by the legend. <a href="https://arxiv.org/pdf/2011.06854.pdf">Source</a></figcaption></figure><p>The authors also characterize the datasets themselves in terms of average values of attribute values (without referring to any NER model at all). This gives us a great way to assess the character of a new dataset (which may be specific to a particular application). This kind of characterization may enable us to choose the model in an informed way (e.g. if your dataset has high OOV Density, then you might choose a CNN based model rather than BERT). The following graph maps this characterization:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/1*O-Tiv_J-5EE7aeajd_8-2Q.png" /><figcaption>Distance from origin denotes the average value of the attribute in a dataset. Colors denote the dataset similar to the image above. <a href="https://arxiv.org/pdf/2011.06854.pdf">Source</a></figcaption></figure><h4><strong>3] Bucket-wise analysis</strong></h4><p>Bucket-wise analysis aims to:</p><ul><li><strong><em>Self Diagnosis: </em></strong>Identify buckets where a specific model shows good/bad performance. E.g. BERT performs best/worst on buckets with most/least label consistency</li><li><strong><em>Comparative Diagnosis: </em></strong>Identify buckets where 2 models show the most/least difference in performance.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/541/1*8xriLAzpfwnIcOeC_Bej8g.png" /><figcaption>From the paper: “M1 and M2 denote two models. We classify the attribute values into four categories: extra-small (XS), small (S), large (L) and extra-large (XL). In the self-diagnosis histogram, green (red) x-tick labels represents the bucket value of a specific attribute on which system achieved best (worst) performance. Gray bins represent worst performance while blue bins denote the gap between best and worst performance. In the comparative diagnosis histogram, green (red) x-tick labels represents the bucket value of a specific attribute on which system M1 surpasses (under-performs) M2 by the largest margin that is illustrated by a green (red) bin”. <a href="https://arxiv.org/pdf/2011.06854.pdf">Source</a></figcaption></figure><h3>Application: Well-grounded Model Improvement</h3><p>With the use of these fine-grained and interpretable evaluation metrics, we can take informed decisions about how to improve model performance.</p><h4>Example: Improving BERT performance by over 10%</h4><p>The authors present the case that since BERT and Flair are most sensitive to Label Consistency, adding more context to each training example would enable the model to make substantial improvement.</p><p>To test this hypothesis, they train the models again with multiple sentences in each training example. As a result, the performance of the model improves substantially.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GkoK89zJWC-OEnKXiNj_mw.png" /><figcaption>The improvement of performance on different datasets as a result of increase in context. Y-axis is the increase in F-score (%). X- axis is the number of sentences in each training example. More than 10% improvement is seen in “WB”, which is a domain from OntoNotes dataset when X=6. <a href="https://arxiv.org/pdf/2011.06854.pdf">Source</a></figcaption></figure><h3>TL;DR</h3><ol><li>Named Entity Recognition (NER) is a sequence labelling task in ML.</li><li>The models performing the task are traditionally evaluated with classification metrics like precision, recall, F-1 score, etc.</li><li>These traditional NER evaluation metrics don’t tell you <em>where </em>the model is failing or <em>what actions </em>you can take to improve performance.</li><li>A <a href="https://arxiv.org/pdf/2011.06854.pdf">new research paper</a> has presented the following attempt to solve this problem:</li></ol><ul><li>Define explainable attributes of data like Entity Density, Label Consistency, token frequency, etc.</li><li>Divide the dataset into different buckets based on these attributes.</li><li>Evaluate the models on each of those buckets separately.</li></ul><p>5. This allows us to:</p><p>— Find out which attributes of the data affect the performance the most.</p><p>— Choose the best suited model for a particular dataset in an informed way.</p><p>— Come up with well-grounded ideas for model improvement.</p><p>So, I hope this article helps you evaluate your NER models in a more meaningful way. If there is any part of the article which is not very clear or needs further discussion, please feel free to reach out to me.</p><p>I intend to write more such posts summarizing awesome new research papers. If that’s your thing, then be sure to follow me <a href="https://charudattamanwatkar.medium.com/">here</a>.</p><p>Thank you for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=93da4406930c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/a-pathbreaking-evaluation-technique-for-named-entity-recognition-ner-93da4406930c">A novel evaluation technique for Named Entity Recognition (NER)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Beginner’s Introduction to Named Entity Recognition (NER)]]></title>
            <link>https://medium.com/swlh/a-beginners-introduction-to-named-entity-recognition-ner-2002b1a010c1?source=rss-5583c854307e------2</link>
            <guid isPermaLink="false">https://medium.com/p/2002b1a010c1</guid>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <category><![CDATA[named-entity-recognition]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[ner]]></category>
            <dc:creator><![CDATA[Charudatta Manwatkar]]></dc:creator>
            <pubDate>Thu, 03 Dec 2020 20:18:56 GMT</pubDate>
            <atom:updated>2020-12-12T17:35:24.154Z</atom:updated>
            <content:encoded><![CDATA[<h3>A Beginner’s Guide to Named Entity Recognition (NER)</h3><h4>What is NER and why should you care?</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qBeBLKWMiriqCVEz" /><figcaption>Photo by <a href="https://unsplash.com/@nathangbingle?utm_source=medium&amp;utm_medium=referral">Nathan Bingle</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h3>What is Named Entity Recognition (NER)?</h3><p>It is the process of<strong> </strong>identifying proper nouns from a piece of text and classifying them into appropriate categories. These categories can be generic like ‘Organization’, ‘Person’, ‘Location’, etc. or they can be tailor-made for a particular application, e.g. ‘Programming Language’, ‘Blogging site’, etc.</p><p><strong>In simpler words, if you want to find out ‘who’, ‘what’, ‘when’, ‘where’ from a sentence, then NER is your task.</strong></p><p>Take the following example (or try it yourself <a href="https://explosion.ai/demos/displacy-ent?text=MDH%20Spices%20owner%20Dharampal%20Gulati%2C%20fondly%20known%20as%20%E2%80%9CMahashayji%E2%80%9D%2C%20died%20on%20Thursday%2C%20ANI%20reported.%20He%20was%2098.%0A%0AGulati%20suffered%20a%20cardiac%20arrest%20on%20Thursday%20morning%2C%20according%20to%20News18.%20He%20had%20been%20receiving%20treatment%20at%20a%20hospital%20in%20Delhi%20for%20the%20past%20three%20weeks.&amp;model=en_core_web_sm&amp;ents=person%2Corg%2Cgpe%2Cloc%2Cproduct%2Cnorp%2Cdate%2Cper%2Cmisc">here</a>):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/921/1*2LPvuGgdMkCEo-s-kEJ2nw.png" /><figcaption>The text is from a real news and the tags were generated by ‘<a href="https://spacy.io/universe/project/video-spacys-ner-model">SpaCy</a>’, one of the most popular libraries for NER.</figcaption></figure><h3>Why is it even a problem?</h3><p>When I first heard about NER, it sounded terribly boring and even trivial. You might be thinking, “Isn’t this a simple lookup problem? Why do we even need ML here?” and so on. Let me try to convince you otherwise:</p><h4><strong>1. “Can’t I Just look up the names?”</strong></h4><p>You can, and often you do. You might think you’ll simply keep a giant list of all the proper nouns that exist in the world. But when you read a sentence, like</p><blockquote>“<em>Mr. Moony presents his compliments to Professor Snape, and begs him to keep his abnormally large nose out of other people’s business</em>”</blockquote><p>you can still infer from the context that ‘Moony’ and ‘Snape’ are names of people, even though you might never have heard these names in real life. So, there must be something more to the problem of NER than just looking it up.</p><h4>2. “Can’t I j<em>ust use handcrafted rules?</em>”</h4><p>Again, you can, and often you do. You might select all the Title-Case words or UPPER-CASE words.</p><p>However, as you can imagine, this is obviously prone to errors if the text is not properly formatted. This can especially be a problem when the text is obtained from applications like speech recognition.</p><p>Even if you manage to isolate all the entities using rules like these, they still have to be classified into proper categories.</p><h3>How do I formulate it in ML terms?</h3><p>So, assuming that you are now convinced that the problem is worth solving, let’s formulate it in more concrete ML terms: Given a sentence,<strong> classify each word into a category of proper nouns. If a word is not a proper noun, then categorize it as ‘Other’.</strong></p><p>Thus, this is a<strong> </strong><em>sequence labelling</em><strong> </strong>problem.</p><p>There are some finer details to be added here. First, the classification will be done on <em>tokens </em>and not <em>words</em>.</p><p>Second, to handle multi-words entities better (and also to distinguish where one entity ends and the next begins), we use multiple fine-grained classes for each category. For example, the sentence</p><blockquote>“Why are Anthony Gonzales, Robert Downey Jr, and Sachin Tendulkar in Paris?”</blockquote><p>will be tagged as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/256/1*PvlSUY17oVc1PtWvXEos9A.png" /><figcaption>Fine-grained classes for better entity resolution</figcaption></figure><p>This is typically called the ‘IOB scheme’, where ‘B’ stands for ‘Beginning’, ‘I’ stands for ‘Inside’, and ‘O’ stands for ‘Outside’. There are even more fine-grained schemes like BILUO scheme, where ‘L’ stands for ‘Last’, ‘U’ stands for ‘Unit’. SpaCy uses the latter scheme. BILUO scheme has the same expressivity as IOB scheme, but <a href="https://www.aclweb.org/anthology/W09-1119/">this</a> paper shows that BILUO is easier to train than IOB.</p><h3>What are the features?</h3><p>What gives us information about whether a word is a Named Entity or not? It can be:</p><ol><li>Capitalization</li><li>Prefix, suffix</li><li>POS Tag</li><li>Relative location in the sentence (beginning, end, etc.)</li><li>Other words in the sentence.</li><li>Whether the word is ‘OOV’ (Out Of Vocabulary)</li></ol><h3>What algorithms can I use?</h3><p>NER being a sequence labelling task, we can use a wide range of algorithms for it. We can roughly categorize them as:</p><h4><strong>1. Traditional ML based:</strong></h4><ul><li>Conditional Random Fields (CRF) [scikit-learn provides a very handy framework called <a href="https://pypi.org/project/sklearn-crfsuite/">sklearn_crfsuite</a>]</li><li>Maximum-entropy Markov model</li></ul><h4>2. <strong>Neural Networks based:</strong></h4><ul><li>LSTMs, bi-LSTM (A good framework is <a href="https://github.com/flairNLP/flair">Flair</a>)</li><li>CNNs (SpaCy uses CNN based architecture)</li><li>Transformers (Spacy has recently launched it <a href="https://spacy.io/universe/project/spacy-transformers">here</a>)</li></ul><h3>Where do I get the data for training?</h3><p>There are several open source datasets available for NER training. Each of them have their own characteristics. Some of them are:</p><ol><li><a href="https://www.clips.uantwerpen.be/conll2003/ner/">CoNLL-2003</a></li><li><a href="https://noisy-text.github.io/2017/emerging-rare-entities.html">W-NUT 2017</a> (This set is more focused on novel, emerging entities)</li><li><a href="https://catalog.ldc.upenn.edu/LDC2013T19">OntoNotes 5</a> (Contains multiple genres of speech)</li></ol><p>For a much more extensive collection of datasets, refer to <a href="https://github.com/juand-r/entity-recognition-datasets">this</a> github repository.</p><h3>How do I know if my model is performing well?</h3><p>When learning about a new ML problem, people often focus a lot on the algorithm, but forget to think about performance metrics. To know which model is performing the best, we need some <em>model-independent</em> metrics for any ML task.</p><h4><strong>1. Traditional Metrics</strong></h4><p>As a classification task, the performance of NER is usually measured in terms of classification metrics (over all the tokens) like <strong>precision, recall, F-score,</strong> <strong>accuracy, </strong>etc. These are fine if you just want to compare different models in a simple way.</p><p>Keep in mind that <strong>there will be a severe class imbalance in any NER data</strong> because most of the words in any corpus are not proper nouns (Typically, ‘O’ tag alone comprises of more than 75% of the tags). So in my opinion, a <strong>macro-average over all the classes is more appropriate</strong> than a micro-average (for any metric).</p><p><a href="http://www.davidsbatista.net/blog/2018/05/09/Named_Entity_Evaluation/">This</a> post gives a great overview of a few different metrics derived from these simple metrics that have been used over the years.</p><h4><strong>2. New Developments</strong></h4><p>However,<strong> the above-mentioned simple metrics do not tell us a lot</strong>. They are aggregated over the entire dataset and we can not characterize the strengths or weaknesses of the model in meaningful ways. <a href="https://arxiv.org/abs/2011.06854">This</a> recent paper tries to address this problem by breaking up the dataset into different chunks based on explainable criteria and examining the model performance on each chunk separately.</p><p>You can read more about it in <a href="https://charudattamanwatkar.medium.com/a-pathbreaking-evaluation-technique-for-named-entity-recognition-ner-93da4406930c">this</a> post that I wrote recently.</p><h3>Summing it up</h3><p>NER is the problem of classifying words in a sentence into categories of proper nouns. It is not completely solvable by lookup or handcrafted rules, but they help. There are various traditional ML based as well as deep learning based algorithms to solve it. Frameworks like <strong>sklearn_crfsuite, SpaCy, Flair,</strong> etc. make it really easy to implement these algorithms.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2002b1a010c1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/a-beginners-introduction-to-named-entity-recognition-ner-2002b1a010c1">A Beginner’s Introduction to Named Entity Recognition (NER)</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>