<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[MLJC - Medium]]></title>
        <description><![CDATA[Machine Learning Journal Club, Turin, Italy - Medium]]></description>
        <link>https://medium.com/mljcunito?source=rss----b7c1069b4442---4</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>MLJC - Medium</title>
            <link>https://medium.com/mljcunito?source=rss----b7c1069b4442---4</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 06:54:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/feed/mljcunito" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[A Bunch of Words: an Introduction to SpaCy on CORD-19]]></title>
            <link>https://medium.com/mljcunito/a-bunch-of-words-an-introduction-to-spacy-on-cord-19-d1587c0743eb?source=rss----b7c1069b4442---4</link>
            <guid isPermaLink="false">https://medium.com/p/d1587c0743eb</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[covid19]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Machine Learning Journal Club (MLJC)]]></dc:creator>
            <pubDate>Thu, 10 Dec 2020 20:05:24 GMT</pubDate>
            <atom:updated>2021-04-22T14:37:46.200Z</atom:updated>
            <content:encoded><![CDATA[<h4>Episode 1: Linguistic Features &amp; Text Processing</h4><blockquote>Authors: <a href="https://medium.com/u/2fefde940982">Simone Azeglio</a> <a href="https://medium.com/u/9b3c8f3dca7c">Marina Rizzi</a> <a href="https://medium.com/u/9c34a633be97">Eugenio Tonanzi</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cKq_1etm6ADORWXxTplLWw.jpeg" /><figcaption>Word Cloud based on words from CORD-19</figcaption></figure><p><strong>Many valuable insights and information might be trapped in huge quantities of raw text, which are usually difficult to process in a smart manner: many words are rare, sometimes different words have almost the same meaning and, frequently, the same words in a different context or order can mean something completely different. To add a solid structure, and therefore useful information, it’s usually better to consider linguistic features. That’s exactly what spaCy is designed to do: extract quantitative information from text and let us play with it by coding. We’re going to see spaCy in action on </strong><a href="https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge"><strong>COVID-19 Open Research Dataset (CORD-19)</strong></a><strong> from </strong><a href="https://allenai.org/"><strong>Allen Institute for AI</strong></a><strong>, which is one of the best resources on the topic, prepared by the White House and a coalition of leading research groups.</strong></p><p>With the onset of the Coronavirus pandemic, a huge number of academic articles related to it has been released. While this huge increase in the amount of research available can lead to a better understanding of the virus and its consequences, the great number of documents generated could be an obstacle in finding the best answer to some questions related to Covid-19. In this set of articles, we try to exploit techniques from Machine Learning and, in particular, from Natural Language Processing (NLP) in order to facilitate the process of information gathering and discovering. We are going to use the <a href="https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge">COVID-19 Open Research Dataset (CORD-19)</a> from <a href="https://allenai.org/">Allen Institute for AI</a>, available from the <a href="https://www.kaggle.com/">Kaggle</a> platform. The CORD-19 database is a resource of over 135,000 scholarly articles, including over 65,000 with full text, about COVID-19, SARS-CoV-2, and related Coronaviruses. It is a freely available dataset, that was provided to the global research community in order to apply natural language processing techniques at the frontier, to generate new insights, suggestions and possible policy actions from the growing literature on coronavirus. As stated in the Kaggle website:</p><blockquote><em>“There is a growing urgency for these approaches because of the rapid acceleration in new coronavirus literature, making it difficult for the medical research community to keep up”.</em></blockquote><h3>Starting from a sample</h3><p>As a first step, we are going to set the environment and to download a tiny part of the dataset. In order to show step by step how the NLP techniques work, we will use only some sentences from an article of the dataset that we’ve already selected for you. We are going to explore the whole dataset in the next articles.</p><pre># Import the necessary libraries</pre><pre>import numpy as np<br>import pandas as pd<br>import spacy<br>import json<br> <br># Load our sample file and take a look at it</pre><pre>path_to_sample = &quot;sample.json&quot;</pre><pre>input_file = open(path_to_sample)<br>with input_file as f:<br>    data = json.load(f)</pre><pre>data[&#39;abstract&#39;] </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YRleHLs8ELElQyv25IwoqA.png" /></figure><p>We are interested in extracting sentences from this part of the <em>.json</em> file. Here is one sentence:</p><pre>ex_sent0 = data[&#39;abstract&#39;][0][&#39;text&#39;] </pre><pre>print(ex_sent0)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-o9xEojcUyVtz-Rnwz7_Dw.png" /></figure><p>Before starting our journey it’s right and proper to take a look at a few concepts from linguistics, in order to better understand how spaCy handles the processing of text data.</p><h3>Linguistics 101: Dependency Grammars vs Phrase Structure Grammars</h3><p>By using spaCy we’ll focus on analyzing sentence structures to identify patterns in word sequences. To understand sentence analysis and patterns, we’ll need some basic knowledge of linguistics. This paragraph contains a linguistic primer you can use as a reference.</p><p>By default, spaCy uses a dependency grammar rather than a phrase structure grammar, which is more commonly used in linguistics. Let’s delve into the difference between these two grammar types. If you haven’t a formal linguistic background, you may find this information helpful.</p><p>Also known as a constituent-based grammar, a <em>phrase structure grammar</em> models natural language based on how words combine to form constituents in a sentence. In syntax, a <em>constituent</em> is a group of words that works as a single unit in a sentence. Phrase structure rules decompose a sentence into its constituent parts, forming a tree structure that begins with individual words and builds up larger and larger constituents.</p><p>On the other side, a dependency grammar is a word-based grammar that focuses on the relations between individual words rather than between constituents. As a result, a dependency parse forms a tree that reflects how words relate to other words in a sentence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/692/1*agC3YjfbS-DyEeOdrbJkDQ.png" /></figure><p>The phrase structure tree breaks up the sentence based on the fact that the sentence consists of a noun phrase and a verb phrase. Those phrases appear on the second level of the hierarchy, directly under the sentence (S) mark — the formal top level. On the bottom level are the individual words that make up those phrases.</p><p>In contrast, the dependency structure uses the verb as the structural center of a sentence. The other words are either directly or indirectly connected to this verb with the help of directed links, known as <em>dependencies</em>. The dependency grammar that spaCy uses by default expresses the grammatical structure of a sentence as a set of one-to-one correspondences between words. Each of these relations represents a grammatical function in which one word is the <em>child,</em> or the dependent word, and the other is the <em>head</em>, or the governor. For example, in “green apple”, the dominant word is “apple”, and “green” is its subordinate. You can think of the head as the word with the most “relative importance” and without which the child doesn’t make sense.</p><p>Each word in a sentence must be connected to exactly one head. But the same word might have none, one, or several children. SpaCy’s grammar assumes that the head of a sentence (the <em>Root</em> token) is its own head. In the example we showed above (“African wild animals run quickly”), the verb “run” is the head of the sentence, so the<em> head</em> property of the Token object representing this word will refer to this same Token object.</p><p>Note that the head/child relationship has nothing to do with linear order in the sentence. For example, the child “wild” comes before its head “animals”, but the child “quickly” comes after its head “run”.</p><p>With this concepts in mind, we can now dive properly into spaCy’s processing pipeline, analyzing all the different tasks that are performed throughout the process.</p><h3>Tokenization</h3><p>Tokenization is the task of splitting a text into meaningful segments, called <em>tokens</em>. A token is everything we can find in our text: a word, a piece of punctuation, a number, a symbol. The input required is a unicode text, and the output is a spaCy’s Doc object.</p><p>The key principle is that spaCy’s tokenization is non-destructive, which means that we will always be able to reconstruct the original input from the tokenized output. Whitespace information is preserved and no information is added or removed during the process.</p><p>But how tokenization really works? The answer is that it works by applying rules that are specific to each language. For example, the english word “don’t” should be split into two tokens representing its meaning, “do” and “n’t” (not), while the acronym “U.K.” should always remain one token. However, if punctuation comes at the end of a sentence it should always be split off to be a token on its own. Generalizing this example, spaCy’s tokenizer proceeds in two steps. First, the raw text is split in substrings on whitespace characters. Then, each substring is processed, from left to right, checking the following questions:</p><ul><li>Does the substring match a language-specific rule?</li><li>Can a prefix, suffix, infix or piece of punctuation be split off?</li></ul><p>If there’s a match, the rule is applied and the tokenizer continues its loop. In this way, spaCy can also deal with nested tokens like combinations of abbreviations and multiple punctuation marks.</p><p>Needless to say, each and every language has plenty of its own grammar and semantic mechanics, and this is why each language that is supported by spaCy has its own subclass, that loads in lists of hard-coded data and rules.</p><p>We can now proceed to tokenize our sample text extracted before:</p><pre># Load English language</pre><pre>nlp = spacy.load(“en_core_web_sm”) </pre><pre># Tokenize the text</pre><pre>doc = nlp(ex_sent0)</pre><pre># Collect the information in a nice-looking Pandas DataFrame</pre><pre>tokens_list = np.array([token for token in doc])<br>tokens_table = pd.DataFrame(tokens_list, columns = [&quot;Token&quot;])<br>tokens_table.head(14)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nnzacJbyqaD8fydAtbuczg.png" /></figure><p>Moreover, spaCy also allows its users to define custom tokenization rules, and to manually build and/or modify the tokenizer or the tokenization output. As you can imagine the possibilities are countless, so please check our references if you’re interested in more details.</p><h3>Lemmatization</h3><p>Our text has now become a collection of tokens, and we’re getting closer to a representation that can allow us to do statistics on it. But we still need to deal with couple of problems.</p><p>First of all, we need to identify together words with different representation but same meaning. For example, “walking” and “walks” are two different tokens and in principle they should be treated as different words. However, when analyzing a text, we don’t really want to consider them as different entities, but rather as different expressions of the same word, the verb “walk”.</p><p>Lemmatization is the process of reducing the inflected forms of a word to their root form so that they can be analyzed as a single item, identified by the word’s lemma, or dictionary form. As it was the case for the tokenizer, this procedure strongly depends on the language at hand. Usually a dictionary for the selected language is loaded, and lemmatization is performed by dictionary lookup, with the addition of several hand-crafted rules.</p><p>Last but not least, we want to distinguish between words that can provide us some information about the text and non-informative words, also called <em>stopwords</em>. For example, every text is usually full of articles, conjunctions, pronouns, auxiliary verbs and so on. However, we cannot expect to extract much information from this kind of words, because they are building blocks of pretty much every sentence in every text. This is why sometimes they need to be kept separated from “real” words. Fortunately, spaCy’s tokenizer does this job for us while splitting the text, and we can recover the features discussed so far by making use of the token.lemma_ and token.is_stop attributes.</p><pre>lemmas_list = np.array([[token, token.lemma_, token.is_stop] for token in doc])</pre><pre>lemmas_table = pd.DataFrame(lemmas, columns = [&quot;Token&quot;, &quot;Lemma&quot;, &quot;Is stopword?&quot;])</pre><pre>lemmas_table.head(14)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SqiNGxt3Dqo7Q1ZGgMJtuA.png" /></figure><p>So far so good, we are now left with a more schematic version of our starting text, that now results segmented into all its words. This can be considered the starting point for building and applying NLP algorithms and machine learning.</p><h3>Part-of-speech Tagging</h3><p>After tokenization and lemmatization, the next phase for word processing is part-of-speech (POS) tagging. Part-of-Speech tagging is the process through which we assign to each token a label that describe the part of speech (such as noun, verb, and so on) the token represents. Here it is where the statistical model comes in.</p><p>SpaCy makes predictions about which tag or label is the most appropriate for a word using neural network models.</p><p>After the model is trained with a good number of examples, it is able to make predictions for each word. SpaCy makes available some pre-trained models for each language, that are already usable on text documents. Note that each trained model is generalizable only within the specific language for which it was trained, since the tag prediction relies on rules that are language-specific. For example, in English a word following “the” is very likely a noun, but this rules is not generalizable to other languages like Spanish or Italian. Some of the POS tag are the following ones:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GzFwpBrjOnAAMfc12Tva_g.png" /><figcaption>Some POS Tagging, available from spaCy documentation: <a href="https://spacy.io/api/annotation#pos-tagging">https://spacy.io/api/annotation#pos-tagging</a></figcaption></figure><p>Together with this, the model is also able to trace back the dependency parse tree structure in our sentence, giving to each word its position. The following image is obtained using the “Visualizer” feature, available with SpaCy:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TnbRmCamethOZFjpzCo8nQ.png" /><figcaption>An example of a structure that represent the “parse dependency”, the dependency between words.</figcaption></figure><p>Now let’s apply POS tagging on the example sentence we already extract.</p><pre>words_list = np.array([[token, token.lemma_, token.is_stop, token.pos_, token.tag_, token.dep_, ] for token in docx])</pre><pre>words_table = pd.DataFrame(words_list, columns = [“Token”, “Lemma”, “Is Stopword?”, “Part of Speech”, “Morphology”, “Dependecy”])</pre><pre>words_table.head(14)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G1dGnfW_q-ShiXGqmYOc1A.png" /></figure><p>The fifth column is the one we haven’t already discussed about. It indicates the Morphological Tag of the considered word. The difference between the Part-of-Speech Tag and the Morphological Tag is that the Morphological Tag code not only the word type (verb, noun, etc), but also other characteristics like whether the word is singular or plural, or for verbs what is the tense.</p><h3>Named Entity Recognition</h3><p>Another interesting feature of SpaCy is that it is also able to recognize which type of “object” are some entities (for example, that “Apple” is a company or an institution, that “U.K” is a geopolitical entity, that “1$ billion” is a monetary value, that 2007 is a date, and so on).</p><p>It is also possible to add arbitrary classes to the entity recognition system, and consequently update the model with new examples. It is important to note that the entity recognition feature strongly rely on the examples it was trained on, and so it could be that it does not always work perfectly. In this case, some tuning and adjustment could be needed afterwards, in order to adjust the entity recognition to the context of the document analyzed.</p><p>Here there is an example in which the SpaCy visualizer is used in order to recognize different entities, that are highlighted and labelled with a short sequence of characters, which represent the category the words belong to.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IaZMxK0tyQJmBU7K4EyBMA.png" /><figcaption>Example of Named Entity Recognition, using the VIsualizer provided by SpaCy.</figcaption></figure><h3>Language Processing Pipelines</h3><p>We have unleashed spaCy’s power, that enable us to recover many information and structure from text documents. The features we saw are the first bricks to construct our journey in the world of Natural Language Processing, applying these techniques in the CORD-19 dataset. However, we would like to understand if it’s possible to automate some processes and build some kind of pipeline.</p><p>Every operation — on text — that we have seen so far can be thought as a LEGO brick itself. We want to pile bricks up and build a re-usable construction: luckily, we are not left alone since at spaCy, developers have built everything in a modular way, with the idea of sequentializing every operation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/633/1*3qQd0HQqSTu6_GfxIFCqYg.png" /><figcaption>spaCy’s language processing pipeline</figcaption></figure><p>The default pipeline consists of a<em> tagger</em>, a <em>parser</em> and an <em>entity recognizer.</em> Each of these components returns the processed Doc, which is then passed on to the next component. As a result, our initial raw text becomes split into a series of tokens, each one associated with its tags and its role in the dependency parse tree, as we saw in the last table presented above.</p><p>It is also possible to design a custom pipeline, which of course, always depends on the statistical model we are considering.</p><p>In this way, we should take heed to the fact that the <em>Tokenizer</em> is considered as a “special” component and isn’t part of the regular pipeline. The technical rigidity here is that there can be only one tokenizer, and while all other pipeline components take a Doc and return it, the tokenizer takes a <strong>string of text</strong> and turns it into a Doc.You can still customize the tokenizer, though: nlp.tokenizer is writable, so you can either create your own <a href="https://spacy.io/usage/linguistic-features#native-tokenizers">Tokenizer class from scratch</a>, or even replace it with an <a href="https://spacy.io/usage/linguistic-features#custom-tokenizer">entirely custom function</a>.</p><p>We’ve kinda understood how to compose different operations but: it’s time to see how to process text.</p><h3>Text Processing</h3><p>Let’s go back to our sample, extracted from one of the articles from CORD-19. We are pretending to have a huge amount of text to process, when processing large volumes of text, these models are more efficient if you let them work on batches of texts. nlp.pipe method takes an iterable of text and yields processed Doc objects. In this way, the batching is done internally.</p><p>Firstly, we can extract a few other sentences from the abstract of the selected article:</p><pre># 2 more sentences</pre><pre>ex_sent1 = data[&#39;abstract&#39;][1][&#39;text&#39;]<br>ex_sent2 = data[&#39;abstract&#39;][2][&#39;text&#39;]</pre><pre># Building texts list<br> <br>texts = [ex_sent0, ex_sent1, ex_sent2]</pre><pre># Building docs in this way is efficient</pre><pre>docs = list(nlp.pipe(texts))</pre><p>For a further in-depth analysis of spaCy’s nlp.pipe method, you can take a look at the <a href="https://spacy.io/api/language#pipe">official documentation</a>, which is definitely exhaustive.</p><p>Let’s take a look at what we get by using the entity linking feature, since we only want entity identifiers, we can remove LEGO bricks with the disable parameter. Our construction is even dynamic and customizable:</p><pre>for doc in nlp.pipe(texts, disable=[&quot;tagger&quot;, &quot;parser&quot;]):<br>    print([(ent.text, ent.label_) for ent in doc.ents])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pruR-AZnPrZe7vT7IpgDFQ.png" /></figure><p>It apparently works but by deeply analyzing each term we see some incongruences. This is a much more serious problem than it seems and it’s related to the fact that scientific terms are not statistically significant in a “regular English” sentence. You can reproduce this code with some non-scientific sentence and you’ll see that everything works fine. How do we get along with scientific articles though? Luckily NLP researchers have worked out some solutions, in the next few articles we are going to pave the way for more complicated language models such as BERT and specifically <a href="https://github.com/allenai/scibert">SciBERT</a> which deals way better with scientific terms.</p><h3>Conclusions</h3><p>In this short introductory article you’ve taken a sip of the very NLP basis with SpaCy. The road to come is still long and winding, but we’re preparing some more tutorials for you, so that you can drive through nicely and easily. Take your time to practice with SpaCy — and this dataset — and stay tuned for further updates.</p><h3>References:</h3><ol><li><a href="https://explosion.ai/blog/dead-code-should-be-buried"><em>Dead Code Should Be Buried</em></a><em>,</em> M. Honnibal</li><li><em>Foundations of Statistical Natural Language Processing, </em>C. D. Manning &amp; H. Schuetze</li><li><em>The Handbook of Computational Linguistics and Natural Language Processing, </em>A. Clark, C. Fox &amp; S. Lappin</li><li><a href="https://arxiv.org/abs/1903.10676"><em>SciBERT: A Pretrained Language Model for Scientific Text</em></a>, I. Beltagy et al</li><li><em>Natural Language Processing and Computational Linguistics</em>, B. Srinivasa-Desikan</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d1587c0743eb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mljcunito/a-bunch-of-words-an-introduction-to-spacy-on-cord-19-d1587c0743eb">A Bunch of Words: an Introduction to SpaCy on CORD-19</a> was originally published in <a href="https://medium.com/mljcunito">MLJC</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A creative introduction to EDA]]></title>
            <link>https://medium.com/mljcunito/a-creative-introduction-to-eda-be9e906cbcb?source=rss----b7c1069b4442---4</link>
            <guid isPermaLink="false">https://medium.com/p/be9e906cbcb</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[advertising]]></category>
            <dc:creator><![CDATA[Jacopo Pasqualini]]></dc:creator>
            <pubDate>Sun, 18 Oct 2020 21:58:21 GMT</pubDate>
            <atom:updated>2020-10-18T21:58:21.807Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/730/1*WKJGmTVgSA73SnTYqK9DYA.jpeg" /></figure><h3>What is the Exploratory Data Analysis (EDA)? <br> <br>What you should do when a new, unknown dataset is given.</h3><p>Here we are going to review some basic techniques to perform a successful exploration of data. There are many motivations behind this set of procedures. If you get meaningful insights you may be able to design outperforming models saving computational efforts. If the intuition about data is clear it will be easier to develop appropriate hypothesis with which build a good model. And so on…<br>This insights will have different consequences on your dataset. You may be lead to keep or modify some feature, remove o create some feature from the old one. Such procedure have different names like feature selection, among others. Here we will use the term EDA as a (positive) umbrella term.<br>A good EDA can help you to decide in which way you should spent more or less time in preprocessing, building baselines, modeling ecc..</p><p>One important counterpart is data visualization. If there is a pattern in the data it may be found, with some luck, with an appropriate visualization. Here we will show some, EDA oriented, visualization techniques. It means that these visualization will be mid-quality but very easy to implement.</p><p>Given a problem which you know nothing about a good strategy is to make up your mind and clarify what you should do. Mostly, we don’t need to know that much about the topic, since data science is a very generic discipline and cover a wide range of topics. Just a couple of tips:</p><p>1 search other people solution to similar problems with github, medium..<br>2 googling and searching on wikipedia, to get the features meaning</p><h3>Disclaimer</h3><p>Keep in mind that EDA is not a linear, sequential procedure. You will never be given a recipe or a list of steps to follow. Here we are in the realm of intuition and the reader should interpret this notebook as a collection of tips, techniques, good practices.</p><p>Some lines of code are not extensively explained and some passages may seem dead leaves.. they are suggestions and is up to you to develop what you find interesting.</p><p>We strongly encourage to use your creativity!</p><h3>LET’S START</h3><p>You can download the notebook from the link:</p><p><a href="https://github.com/MLJCUnito/HowToTackleAMLCompetition">https://github.com/MLJCUnito/HowToTackleAMLCompetition</a></p><p>You will never use one library, but rather a combination of multiple libraries.<br>Be bright, and develop your intuition about how to combine at least the most famous python libraries.</p><p>First we import some libraries</p><pre>import pandas as pd<br>import numpy as np<br>import matplotlib.pyplot as plt<br>import os<br>import seaborn as sns</pre><pre>%matplotlib inline</pre><p>Here we set an environment variable,the path that will tell the computer where data are stored.</p><p>You may want to download the dataset directly from the jupyter notebook, but it is quite a complicated story that depends on the machine you are working with…<br>You can download the dataset from your browser, and move it to your working directory with your favorite procedure.</p><p>You can find the dataset here:<a href="https://www.kaggle.com/fayomi/advertising/download"> https://www.kaggle.com/fayomi/advertising/download</a></p><pre>DATA_FOLDER = # your path here<br>adv = pd.read_csv(os.path.join(DATA_FOLDER, &#39;advertising.csv&#39;))</pre><p>Let’s give a look at the data, and check what features contains.<br>This step is crucial, because if we already know the research field where features come from we could start to figure out how to proceed with our analysis. We may easily figure out what the baselines of the problem are. We need to be as informed as possible about features meaning. It is often useful answer one question in different ways, ie understand different commands can help us to solve the same problem. Maybe, sometimes further, these little differences in the output will be helpful!</p><p>What can we expect to see when exploring the basic features of a dataframe?<br>How many entries are there in the dataset?<br>Which features are contained?</p><pre>print( “ Dataset shape: “,adv.shape[0])<br>print( “ Number of features: “, adv.shape[1] )</pre><pre>print( “ Your features are: \n”)<br>print(adv.columns)</pre><pre>print(“ Features Types: \n “)<br>print(adv.dtypes)</pre><p>The standard access to a dataframe column is dataframe.[“featuename”]<strong><em>.</em></strong> We can have easier access, avoiding strings usage, in the format `dataframe.featurename`, as if it were a object method.</p><p>We can do that if we slightly change the columns names removing all the spaces and renaming the columns with these new strings</p><pre>features = list(adv.columns)</pre><pre>for f in features:<br> <br> phi = f.replace(‘ ‘,’’)<br> adv = adv.rename(columns = {f:phi})</pre><p>The following command just shows the first rows of our dataset. In the first column we have the features names. The remaining columns are the entries of the dataset. This kind of visualization may be more eye pleasant the the former ones if the number of features is not that large</p><pre>adv.head().T</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_0pfRBNfxwbr4TUy9FM05Q.png" /></figure><p>Another important dataframe method is `info`, which summarizes the dataframe content.<br>With this command we can even see that there are not missing values in the data which, as we will see in future posts, may be something important to think about.</p><pre>adv.info()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/475/1*1mLs1_8TtYkwDteU6YLykQ.png" /></figure><p>It is in the spirit of EDA to understand which feature of the raw dataset plays which role.</p><p>Since this is an advertisement dataset we can say that there is one feature more important than all the other, <em>ClickedonAd</em> that is a binary feature. Let’s see what contains:</p><pre>adv.ClickedonAd.head(11)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/335/1*G4eYDoM6ORtN15BF-96V0Q.png" /></figure><p>Now we can check if the dataset is balanced, ie if there are as many samples of one kind as the other:</p><pre>n = len(pd.unique(adv[‘ClickedonAd’])) <br> <br>print(“Number of unique values :”, n)<br>print(adv[‘ClickedonAd’].value_counts())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/327/1*441qiq6bf7Id0S0qSpNmeg.png" /></figure><p>We were lucky! We found that our dataset is perfectly balanced. In real situations this never happens and this fact absolutely needs to be checked every time we analyze a new dataset.</p><p>Now, brief digression. This perfect balance is a feature of `case study` datasets. They are data preprocessed by the provider in order to be easier to analyze. Even if they are `toy problems` they are very helpful because you can learn the basics of EDA without worrying that much about other messy things such as decoding, preprocessing, augmentation and so on.</p><p>For sake of simplicity we can save the target feature in a different object and then remove it from the data.</p><pre>target = adv.ClickedonAd<br>adv = adv.drop([&quot;ClickedonAd&quot;],axis=1)</pre><p>Here is a binary classification problem!</p><p>We can investigate features correlations plotting the 2d histogram of a couple of variables. This exploration may help us to decide how to proceed in data preprocessing or, in general, further investigations.</p><pre>f, ax = plt.subplots(figsize=(10, 10))<br>sns.kdeplot(adv.Age, adv.DailyTimeSpentonSite, color=&quot;b&quot;, ax=ax)<br>sns.rugplot(adv.Age, color=&quot;r&quot;, ax=ax)<br>sns.rugplot(adv.DailyTimeSpentonSite, vertical=True, ax=ax)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/529/1*P_-yZ7bsLC7ysHcGLBUM5Q.png" /></figure><p>From the former plot we see that we may proceed with a clustering analysis or, in general, unsupervised learning procedures.</p><p>We can extend the former reasoning plotting all the couples of numeric features. We could do that for other feature types, and we left it as an exercise for the reader. A reason for our choice is that pandas can encode integers and categorical features in the same way. This happens because it may not be clear for some variables, such as age, if they should be treated as integers or categorical. How to use these amphibious kind of variable is up to you and your intuition /knowledge about data.</p><pre>numeric_adv = adv.select_dtypes(include=[np.float])</pre><pre>numeric_adv.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/1*zrhMdEBMEwOa-dmnCF6JsQ.png" /></figure><pre>from pandas.plotting import scatter_matrix<br>scatter_matrix(numeric_adv, alpha=0.3, figsize=(10,10))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/683/1*FFXJcgRkxYSiIWzqT8wrgg.png" /></figure><p>This plot is not that clear but suggest that we should investigate the correlations between `DailyTimeSpentonSite` and `DailyInternetUsage`. This clustering will identify two communities: the first is composed by users that spent a lot of their daily internet time on the site, the other that do the opposite.</p><pre>f, ax = plt.subplots(figsize=(6, 6))<br>sns.kdeplot(adv.DailyInternetUsage, adv.DailyTimeSpentonSite, color=&quot;b&quot;, ax=ax)<br>sns.rugplot(adv.Age, color=&quot;r&quot;, ax=ax)<br>sns.rugplot(adv.DailyTimeSpentonSite, vertical=True, ax=ax)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/1*CUxtWJv6cMN4w-to3kS6Fw.png" /></figure><p>We can tye to create a new feature, the `faithfulness` of a user. We would like that this feature encodes the clustering structure. We expect that this feature exhibits a bimodal histogram so it will be easier for our algorithm to build correlations between data and target. We try the following combination</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/1*dV-ddmsR2_2jBAvXjvccyQ.png" /></figure><pre>from scipy.stats import norm</pre><pre>adv[&quot;Faithfulness&quot;]=(<br>    adv[&quot;DailyTimeSpentonSite&quot;]*adv[&quot;DailyInternetUsage&quot;]/<br>    (adv[&quot;DailyTimeSpentonSite&quot;]+adv[&quot;DailyInternetUsage&quot;])<br>)</pre><pre>sns.distplot(adv[‘Faithfulness’], hist=False, color=’r’, rug=True);</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*kEvDXfAeL6MYbFnXMKaCdw.png" /></figure><p>We combined the information of two feature into a new, maybe better one. At this point we may be tempted to drop the two old features. This may lead to some issue during training and so on but this is not the place to discuss this problem. We are happy to have found such a new feature and for sake of simplicity we will drop the old features keeping only the new one. This is a procedure that is known as feature reduction.</p><p>Now let’s focus on the features encoded by pandas as objects:</p><pre>object_features = [&#39;AdTopicLine&#39;, &#39;City&#39;, &#39;Country&#39;]<br>adv[object_features].describe(include=[&#39;O&#39;])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/434/1*oK_nOyckEkwnetL5bbp0uQ.png" /></figure><p>As we can see from the table above that all the values in column “Ad Topic Line” is unique, while the “City” column contains 969 unique values out of 1000. There are too many unique elements within these two categorical columns and it is generally difficult to perform a prediction from a flat histogram. We can consider those features as noise since or they have uniform almost distribution or they are constants, which does not carry information too. In the next post we will see some more robust procedure to detect such features. Because of that, they will be omitted from from the analysis.</p><p>The third categorical variable, i.e “Country”, has a unique element (France) that repeats 9 times. Additionally, we can determine countries with the highest number of visitors. The table below shows the 20 most represented countries in our DataFrame.</p><pre>pd.crosstab(index=adv[&#39;Country&#39;],<br>            columns=&#39;count&#39;).sort_values([&#39;count&#39;],<br>            ascending=False).head(10)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/157/1*Gtk3rqHU4RXO1OZTcmGiCQ.png" /></figure><p>We have already seen, there are 237 different unique countries in our dataset and no single country is too dominant. A large number of unique elements will not allow a model to establish meaningful relationships. For that reason, this variable will be excluded too. It is too difficult to learn from (almost) flat distribution that is equivalent to white noise.</p><pre>data = adv.drop([‘AdTopicLine’, ‘City’, ‘Country’], axis=1)</pre><p>Now, let’s focus on a single feature, *Timestamp*. It contains date and time of the day when the entry was recorded. Entries are recorded in the format *date&amp;hour*. It is likely that, if we split this one in two new features, like *day* and *hour* we may be able to observe different kind of pattern occurring in the two new features. The idea here is that individual activities are determined by time management. We manage in different ways our time on the week scale respect to the 24h scale. It is often a good idea to create new features from the “raw”. With a physics analogy, it is like if we are separating the different time scales of our problem. First, we give a look at this “Mother” feature:</p><pre>print ( adv.Timestamp )</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/1*ykEK0O9Adl3svzhO4k-Wbw.png" /></figure><p>Our purpose now is to create three new features by splitting one column in three. If we split features and properly operate on them we may get something useful.</p><p>The “Day of the week” variable contains values from 0 to 6, where each number represents a specific day of the week (from Monday to Sunday). The category values here are the days of the week, which may be useful to detect periodic pattern in our data. Hours will be categorical too. Date will remain in pandas time format.</p><p>We can proceed as follows</p><p>1 convert the timestamp column (string format) into pandas datetime format<br>2 through the dt method split time and date, hour and day of the week and assign them to new columns<br>3 Eliminate the timestamp column. There is no information loss here<br>4 check the output</p><pre>adv[‘Timestamp’] = pd.to_datetime(adv[‘Timestamp’])<br>adv_times=adv[‘Timestamp’]</pre><pre>adv[‘Hour’] = pd.to_datetime(adv_times, format=’%H:%M’).dt.hour<br>adv[‘Date’] = pd.to_datetime(adv_times, format=’%M:%D’).dt.date<br>adv[‘WDay’] = pd.to_datetime(adv_times).dt.weekday</pre><pre>adv[‘Click’] = target</pre><pre>adv = adv.drop([&#39;Timestamp&#39;], axis=1)</pre><pre>adv.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/977/1*yuVQGPImtHx9T6W3ZZ2AMg.png" /></figure><p>We can evaluate how much time our dataset spans and fix our problem timescale</p><pre>print(&#39;Train min/max date:&#39;, adv.Date.min(), adv.Date.max() )<br>print(&#39;Dataset time span: &#39;,adv.Date.max()-adv.Date.min())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/1*EH-3E0KOk7LlqtfHcVSS8Q.png" /></figure><p>First, one very simple histogram: which day of the week carry us more information? Is there a “special day”?<br>It does not seem, but we can detect a periodic pattern in the data. We can do the same for the hours of the day.</p><pre>ax = adv[&#39;WDay&#39;].value_counts(sort=False).plot(kind=&#39;bar&#39;)<br>ax.set_xlabel(&quot;Day of the week&quot;)<br>ax.set_ylabel(&quot;Records&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/390/1*fsM0KJxSdPjqoNvorxM1jQ.png" /></figure><pre>ax = adv[‘Hour’].value_counts(sort=False).plot(kind=’bar’)<br>ax.set_xlabel(“Hour of the day”)<br>ax.set_ylabel(“Records”)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/389/1*eyrG52R98P7kEbpWOzUWRQ.png" /></figure><p>To obtain this result we have never filtered the data or, equivalently, we didn’t transformed our features. They are still raw. Can we extract more information -as in the first feature reduction case- modifying the original data, applying some kind of filtering? If true, this may lead us to meaningful insights. <br>We can try to drop all entries where the used did not clicked on the ad, in order to investigate if there is a daily-based preference in clicking</p><pre>adv_time = pd.DataFrame()</pre><pre>adv_time[‘WDay’] = adv[‘WDay’]<br>#adv_time[‘Hour’] = pd.to_datetime(adv[‘Hour’])<br>adv_time[‘Hour’] = adv[‘Hour’] <br>adv_time[‘Click’] = target</pre><pre>adv_time.info()</pre><p>Can we see if there are hour of the day where we should do targeted advertisement? The idea here is to separate, according to the target feature (ClickedOnAd) a variable of interest. This may not lead to a particular result, but is to enforce our intuition about data. Now let’s focus only on the time features. We can try to see if the appreciation or not of our advertise exhibit different patterns that we can employ in some way. Here we employ the mask concept: it can be thought as a boolean grid that filters our dataframe according to a characteristic condition and help us to split the data in more and more parts</p><pre>mask_one = (adv_time[‘Click’] == 1)<br>adv_time_one = adv_time[mask_one]</pre><pre>mask_zero = (adv_time[‘Click’] == 0)<br>adv_time_zero = adv_time[mask_zero]</pre><p>Let’s see if there are some evident patterns in the features we found, with a simple plot</p><pre>fig = plt.figure(figsize=(20,15))</pre><pre>ax = fig.add_subplot(221)<br>adv_time_one[&#39;WDay&#39;].value_counts(sort=False).plot(kind=&#39;bar&#39;)<br>ax.set_xlabel(&quot;WDay people click&quot;)<br>ax.set_ylabel(&quot;Records&quot;)</pre><pre>ax = fig.add_subplot(222)<br>adv_time_zero[&#39;WDay&#39;].value_counts(sort=False).plot(kind=&#39;bar&#39;)<br>ax.set_xlabel(&quot;WDay people DON&#39;T click&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/975/1*pei9BAqX1CbDDJ7M80pJ6A.png" /></figure><pre>fig = plt.figure(figsize=(20,15))</pre><pre>ax = fig.add_subplot(221)<br>adv_time_one[&#39;Hour&#39;].value_counts(sort=False).plot(kind=&#39;bar&#39;,title=&quot;Hour people click&quot;)<br>ax.set_xlabel(&quot;Hour people click&quot;)<br>ax.set_ylabel(&quot;Records&quot;)</pre><pre>ax = fig.add_subplot(222)<br>adv_time_zero[&#39;Hour&#39;].value_counts(sort=False).plot(kind=&#39;bar&#39;,title=&quot;Hour people DON&#39;T click&quot;)<br>ax.set_xlabel(&quot;Hour people DON&#39;T click&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/1*zwlA_XSf_BIHxPEhfdcUYQ.png" /></figure><p>There may be patterns in data, but it is not obvious how to proceed. It is not obvious which model we should choose, since there are no evident clustering or periodicity in the preference or not for clicking.</p><p>With this little failure we are facing the necessity to introduce an automatic feature extraction mechanics, possibly designed according to some optimization principle. This is the basic idea behind machine learning or, more specifically, neural networks!</p><h3>Conclusions:</h3><p>In this post we introduced some good practices for an efficient EDA. If the number of features is not too large we can try to study little groups of features, according to their type. Once this is done we have a wide range of choices. With some patience we can investigate the basic correlations between our features. For a first step a basic scatter plot is enough. From the insights that we obtain here we can try to investigate some couples of variables that exhibit particular patterns. If this is the case we can try to create new features as combinations of the raw ones and if this combination seems to reasonably resemble the properties of the couple clustering we can replace the old variables with the new one. This procedure is known as feature reduction. We can get feature reduction even investigating the variability of values that our features take. We can group those values in a histogram and if it happens to be flat we can consider it as pure noise and drop these features. We can follow the opposite practice: feature augmentation. This procedure allows us to create new features from a raw one, as happened with temporal data. From a yy/mm/dd hh/mm entry we created tree new features: the hour, the day of the week and the date of the advestising-user interaction, starting from a a physics-like reasoning. Unfortunately, further investigation showed us that such procedure does not lead to clear patterns. This little failure is not purely a bad thing and we can learn something important from that. If data does not exhibit a clear clustering we may imagine to introduce some algorithm able to extract or design with an automatic procedure meaningful combinations of the features. That’s why with highly dimensional problems we need to introduce machine learning.</p><p>I hope this article was useful for you. If you have any, do not hesitate in posting a comment! Thanks!</p><p>References:</p><p><a href="https://www.coursera.org/learn/competitive-data-science">https://www.coursera.org/learn/competitive-data-science</a></p><p><a href="https://stackabuse.com/predicting-customer-ad-clicks-via-machine-learning/">https://stackabuse.com/predicting-customer-ad-clicks-via-machine-learning/</a></p><p>Yaser S. Abu-Mostafa, Malik Magdon-Ismail, Hsuan-Tien Lin, “Learning from Data”</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=be9e906cbcb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mljcunito/a-creative-introduction-to-eda-be9e906cbcb">A creative introduction to EDA</a> was originally published in <a href="https://medium.com/mljcunito">MLJC</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Validation strategies]]></title>
            <link>https://medium.com/mljcunito/validation-strategies-289df1507bd?source=rss----b7c1069b4442---4</link>
            <guid isPermaLink="false">https://medium.com/p/289df1507bd</guid>
            <category><![CDATA[mljcunito]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Arianna Di Bernardo]]></dc:creator>
            <pubDate>Wed, 16 Sep 2020 19:32:15 GMT</pubDate>
            <atom:updated>2020-09-16T19:32:15.382Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qPZtnaYiBC7GMpqxYP7azw.jpeg" /></figure><blockquote>As already mentioned in the <a href="https://medium.com/mljcunito/overfitting-review-and-the-validation-intervention-a3861eda10c7">last article</a>, overfitting is a phenomenon that negatively affects the performance of your model. With validation, one can efficiently prevent it. Let’s discuss more in detail what are the strategies you can adopt to perform this important step and focus on the reasons to use each one of them.</blockquote><h3>Hold-out recap</h3><p>In the previous story we introduced a first way to divide your dataset in order to perform validation, called <strong>Hold-out</strong> strategy. <br>It consists in splitting your data into two parts: the <em>training part</em> is used to fit our model, the <em>validation part</em> to evaluate its performance. Using the scores from this evaluation step, we can choose the best model and the best hyperparameters for the selected model.</p><p>If we have enough data, using Hold-out is usually a good choice. In particular we can use this scheme if we get similar scores for the same model when we try different splits.</p><p>But what is the optimal size of our sets?</p><blockquote>A common split is using 80% of data for training and the remaining 20% of the data for testing.</blockquote><p>Remember these are the default sizes setted in <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html">train_test_split</a>, the command we have used to perform Hold-out in our <a href="https://github.com/MLJCUnito/ProjectX2020/blob/master/HowToTackleAMLCompetition/Lecture3%7COverfittingRevisitedAndValidation.ipynb">notebook</a> (another useful command is <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.ShuffleSplit.html">ShuffleSplit</a>: this method just returns random indices to split data into training and test sets).</p><h3>Leave-One-Out cross validation</h3><p>If your dataset is small, it makes sense the validation set is small, in order to save enough examples for the training... <br>But this means that the validation error will be a bad estimation of the test error, because a too little validation set doesn’t capture the behaviour of the test set! For example, to measure the performance of your model with only one example is too constrictive.</p><p>The solution is the <strong>Leave-one-out</strong> strategy: in that scheme we iterate through every sample in our data, each time using one example for validation and the remaining examples for training. <br>You will need to retrain the model <em>N</em> times (if <em>N</em> is the number of samples in the dataset); in the end, a prediction for every sample in the training set is computed and then you can calculate the global loss by average the single losses (this averaging process is called <em>cross validation</em> ).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/805/1*Ge3H9zpVURBuT4J8em945g.png" /></figure><p>This method can be helpful if we have too little data and just enough models to choose from.<br>In scikit-learn there exists a function able to do this: <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.LeaveOneOut.html#sklearn.model_selection.LeaveOneOut">LeaveOneOut</a>.</p><p>As an example, let’s consider a very small dataset, with only 10 examples:</p><pre>from sklearn.model_selection import LeaveOneOut<br>from sklearn.datasets import make_classification</pre><pre>X_clf, y_clf = make_classification(n_samples=10, n_features=4, random_state=3)</pre><pre>plt.scatter(X_clf[:, 0], X_clf[:, 1], marker=’o’, c=y_clf, s=25, edgecolor=’k’) <br>plt.title(“Data”)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xBfqHNTkI7WCP5sk2S_rjw.png" /></figure><p>Now, let’s find the best classifier to perform a binary classification task on these points: we try with a naive bayes classifier:</p><pre>loo = LeaveOneOut()</pre><pre>from numpy import mean, std </pre><pre>score=[]</pre><pre>from sklearn.naive_bayes import GaussianNB NB2=GaussianNB()</pre><pre>for train_index, valid_index in loo.split(X_clf): </pre><pre>#Generate indices to split data into training and test set<br>print(“TRAIN:”, train_index, “VALID:”, valid_index) X_train, X_valid = X_clf[train_index], X_clf[valid_index] y_train, y_valid = y_clf[train_index], y_clf[valid_index] </pre><pre>  NB2.fit(X_train, y_train) </pre><pre>  print(NB2.score(X_valid, y_valid)) </pre><pre>  score.append(NB2.score(X_valid, y_valid)) </pre><pre>#Average accuracy with cross validation<br>print(“Accuracy: %0.2f (+/- %0.2f)” % (mean(score), std(score) * 2)) </pre><pre>&gt;&gt;&gt; <br>TRAIN: [1 2 3 4 5 6 7 8 9] VALID: [0] 1.0 <br>TRAIN: [0 2 3 4 5 6 7 8 9] VALID: [1] 0.0 <br>TRAIN: [0 1 3 4 5 6 7 8 9] VALID: [2] 1.0 <br>TRAIN: [0 1 2 4 5 6 7 8 9] VALID: [3] 1.0 <br>TRAIN: [0 1 2 3 5 6 7 8 9] VALID: [4] 1.0 <br>TRAIN: [0 1 2 3 4 6 7 8 9] VALID: [5] 1.0 <br>TRAIN: [0 1 2 3 4 5 7 8 9] VALID: [6] 0.0 <br>TRAIN: [0 1 2 3 4 5 6 8 9] VALID: [7] 1.0 <br>TRAIN: [0 1 2 3 4 5 6 7 9] VALID: [8] 1.0 <br>TRAIN: [0 1 2 3 4 5 6 7 8] VALID: [9] 1.0 <br>Accuracy: 0.80 (+/- 0.80)</pre><h3>K-fold cross validation</h3><p><strong>K-Fold</strong> divides data in <em>K</em> groups of samples of equal size, called folds (if<em> K=size of dataset, </em>this is equivalent to the Leave One Out strategy). <br>The prediction function is trained over <em>K-1</em> folds, and then the fold left out is used as a validation set <em>only once.</em> After this procedure, we average scores over these K-folds (<em>cross validation</em>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*JnTW8G1NSqzrBFLEqvMMRg.png" /></figure><p>This method is a good choice when we have enough data and we get different scores and optimal hyperparameters for different splits.</p><p>You can also estimate mean and variance of the loss. This is very helpful in order to understand the significance of improvement!</p><blockquote>Here it is important to understand the difference between K-fold and a K-repeated hold-out. In the first case it is possible to average the scores in order to obtain a mean score indicating the quality of the model. In the second case, this is not possible: some samples may never have been used as validation examples, while others may appear several times. An average of the scores is no longer informative!</blockquote><p>Let’s apply this strategy to our cancer breast classification problem and select the best classifier with the K-Fold strategy (<em>check out the</em><a href="https://medium.com/mljcunito/overfitting-review-and-the-validation-intervention-a3861eda10c7"><em> previous article</em></a><em> for more details about this example</em>).</p><p>Useful commands are <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.KFold.html">KFold</a>, that provides train/test indices to split data in train/test sets, and <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.cross_val_score.html">cross_val_score</a>, for evaluating the scores by cross-validation: let’s try this second one command.</p><pre>from sklearn.model_selection import cross_val_score</pre><pre>for clf in [SVC(random_state=0), RandomForestClassifier(random_state=0), GaussianNB(), DecisionTreeClassifier(random_state=0), LogisticRegression(random_state=0), MLPClassifier(random_state=0)]:<br> <br>   model = clf </pre><pre>   scores = cross_val_score(model, Xtrain, ytrain, cv=5)</pre><pre>print(&quot;Accuracy: %0.2f (+/- %0.2f)&quot; % (scores.mean(), scores.std() * 2))</pre><pre>&gt;&gt;&gt;<br>Accuracy: 0.63 (+/- 0.01)   --&gt; SVC<br>Accuracy: 0.95 (+/- 0.05)   --&gt; RandomForestClassifier<br>Accuracy: 0.93 (+/- 0.06)   --&gt; GaussianNB<br>Accuracy: 0.92 (+/- 0.06)   --&gt; DecisionTreeClassifier<br>Accuracy: 0.94 (+/- 0.10)   --&gt; LogisticRegression<br>Accuracy: 0.94 (+/- 0.06)   --&gt; MLPClassifier</pre><p>This time, the winner is Random Forest Classifier, and it is a more conscious and precise choice.</p><p>But what happens if you are dealing with time series? Or with unbalanced data sets? Is a random split still appropriate?<br>Take a look at the next story to find out how to act in these particular cases and to discover some useful splitting strategies suitable for ML competition!</p><p><em>You can find my notebook with the entire code </em><a href="https://github.com/MLJCUnito/ProjectX2020/blob/master/HowToTackleAMLCompetition/Lecture3%7COverfittingRevisitedAndValidation.ipynb"><em>here</em></a><em>.</em></p><p><em>References:</em></p><ul><li><em>‘Learning From Data, a short course’, Hsuan-Tien Lin, Yaser S. Abu-Mostafa, Malik Magdon-Ismail</em></li><li><a href="https://scikit-learn.org/stable/auto_examples/model_selection/plot_cv_indices.html">Visualizing cross-validation in scikit-learn</a></li><li><a href="https://www.coursera.org/learn/competitive-data-science">How to Win a Data Science Competition: Learn from Top Kagglers</a></li><li><a href="http://www.chioka.in/how-to-select-your-final-models-in-a-kaggle-competitio/">How to Select Your Final Models in a Kaggle Competition</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=289df1507bd" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mljcunito/validation-strategies-289df1507bd">Validation strategies</a> was originally published in <a href="https://medium.com/mljcunito">MLJC</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Overfitting review and the Validation intervention]]></title>
            <link>https://medium.com/mljcunito/overfitting-review-and-the-validation-intervention-a3861eda10c7?source=rss----b7c1069b4442---4</link>
            <guid isPermaLink="false">https://medium.com/p/a3861eda10c7</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[mljcunito]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[overfitting]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Arianna Di Bernardo]]></dc:creator>
            <pubDate>Sat, 05 Sep 2020 13:45:52 GMT</pubDate>
            <atom:updated>2020-09-05T13:45:52.095Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/611/1*UFOWp9mV3P8hFzIKa9v6gQ.png" /><figcaption>Avoiding overfitting is like finding the right direction in a labyrinth.</figcaption></figure><blockquote>The main challenge when designing a ML algorithm is to avoid overfitting, a phenomenon that causes poor performance. Whit validation, one can keep track of the quality of the model, notice when it overfits and prevent the resulting bad solution.</blockquote><blockquote>This article belongs to a series devoted to figuring out a way to tackle a Machine Learning Competition: the first set of articles deals with Data Preprocessing and Features Engineering (if you have missed it, check out the first story <a href="https://medium.com/mljcunito/data-spectrometry-or-how-to-preprocess-your-data-2a6593ee5dbb">here</a>).</blockquote><blockquote>In this article we review the basic but essential concept of <strong><em>overfitting. Then, </em></strong>we will introduce <strong><em>validation </em></strong>and some useful strategies you can use in practice to perform this important step.</blockquote><h3>Overfitting: your worst enemy</h3><p><strong>Overfitting</strong> means <em>fitting the data more than is warranted.</em></p><p>It is the phenomenon where fitting the observed data well no longer indicates that we will get a good performance, and may actually lead to the opposite effect: in fact, many Machine Learning models have the ability to memorize the training set, which can lead to poor efficiency on unseen examples.</p><p>In those cases, we obtain a model that perfectly approximates your data, but it doesn’t have the ability to generalize to new examples.</p><p>The most important part when you design a model is recognizing overfitting and understanding the underlying causes, in order to combat it.</p><p><strong>Underfitting</strong> is the opposite problem: it occurs when we fit data not enough, leading to bad approximation in addition to poor generalization.</p><blockquote>To choose the best model, we want to avoid underfitting on the one side and overfitting on the other.</blockquote><h3>Overfitting causes</h3><p>Let’s start by identifying the causes of overfitting:</p><ul><li><strong>Dataset size</strong> is the first culprit of overfitting: the fewer examples for training, the more models can fit our data. Consider an extreme example, a sample with only one training point: any model will be able to “explain” it! As the training set size increases, fewer models are able to explain them, leading to decreased overfitting.</li><li><strong>Stochastic noise</strong> is due to measurement errors or random fluctuations in your data. It is stochastic in the sense that it is random: each time we generate the sample, the stochastic noise is different. We cannot directly intervene to limit this kind of noise, it doesn’t depend on the model and therefore on our choices.</li><li><strong>Deterministic noise</strong> appears when the phenomenon being modeled is too complex. In contrast with stochastic noise, deterministic noise is fixed for a given sample, but also in this case we can’t capture or model it.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lN9hH443YGHD-UhBPE75yg.png" /></figure><p>One can think to contrast noise (stochastic and deterministic) by using a complex model. But is that really the case? <strong><em>NO!</em></strong></p><blockquote><em>If the level of noise is high and the dataset is small, using a complex model makes it worse. In fact, a </em><strong><em>complex model makes use of its additional degrees of freedom to fit noise</em></strong><em>, which can result in overfitting.</em></blockquote><p>Let’s try to understand this concept in a very simple example of a one dimensional regression problem with polynomials features<em>.</em> <em>(For further details about the code take a look at the notebook here).</em></p><p>Imagine we have noisy samples (<em>the blue points in the plot</em>) from a real function we want to approximate, in this case the <em>sine</em> function (<em>the orange curve</em>).<br>Then, we make use of three regression models with polynomial features of different degrees: 1, 3 and 50.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/809/1*B9oIbK2WSQQnNNsyYLVORA.png" /></figure><p>In the picture above we can visualize the results of our model’s predictions.</p><p>What we see is that a too simple model leads to underfitting: it can’t capture underlined relationships and we will get poor results.</p><p>Instead a complex model causes overfitting: in fact, the fitted curve passes exactly through each data point, but it oscillates wildly and gives a very poor representation of our target function!</p><p>The compromise is a simple model of degree 3: it doesn’t fit perfectly the target function, but it performs significantly better than the complex model of degree 50, that fits noise and is so far from our target function.</p><p>But does this mean that a simple model is always preferred to a complex model? Cleary not, in fact if you are dealing with a large dataset <strong>without noise</strong>, there is no reason to avoid a complex model.</p><p>The same procedure, if the dataset is <em>without noise, </em>leads to different results…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/809/1*V4hnvJk738GtpmJmZ8NXhA.png" /></figure><p>As you can see, in this case a complex model performs better than a simple one!</p><h3>How to combat overfitting?</h3><p>There are two ways to cure overfitting, often used together: <em>regularization</em> and <em>validation</em>.</p><ul><li>With <strong>regularization</strong> one can put constraints to the learning process to improve the generalization performances of the model, and therefore its quality. The need and the amount of regularization depends on the quality and the quantity of your data: too much regularization can lead to underfitting, because the model is not free enough to fit the data well. In most of the models you will use diving in scikit.learn, you can find the regularization term as hyperparameter.</li><li><strong>Validation</strong> allows you to keep an eye on the model performance, in order to prevent overfitting. This will be the central theme of the next section.</li></ul><blockquote>Notice that the setting of optimal regularization parameters is done through validation!</blockquote><h3>Validation</h3><p>To contrast overfitting we need to correctly understand the quality of our model, i.e. we want to check if the model gives expected results on unseen data.</p><p>The problem is that the performance of our model can differ on train data (<em>available</em>) and on the test data (<em>often not available</em>), because the learning algorithm could just memorize all examples from the train data and be completely useless on unknown data.</p><p>So, usually, we divide the data we have into two parts, the training and validation part. We fit our model on the train part and check its quality on the validation part.</p><p>Only in this way we can select the model which will be expected to get the best quality on the test data, and adjust its hyperparameters (and often regularization is represented by one of these parameters): this could mean the choice between a linear or non linear model, the choice of the degree in a polynomial model, the choice of the value of a regularization parameter.</p><blockquote>This step is called <strong>model selection</strong> and it is of paramount importance when you design a ML model: the goal is to select the best model and the optimal parameters for that model.</blockquote><h4>Train, Validation, Test sets</h4><p>In this section we will show how validation works in a practical and interactive way.</p><p>Let’s introduce a useful dataset from <a href="https://scikit-learn.org/stable/modules/classes.html#module-sklearn.datasets">sklearn.datasets</a>: the <em>breast cancer wisconsin dataset,</em> a classic and very easy binary classification dataset. It is a collection of medical images of breast cancer together with their cancer characteristics. The task consists in building a classifier that is able to recognize if a patient has a benignant or malignant breast tumor.</p><pre>from sklearn import datasets</pre><pre>dataset=datasets.load_breast_cancer()  # load dataset</pre><pre>X, y = dataset.data, dataset.target  # create input and target</pre><p>Let’s create our training and test set, splitting the dataset into two random parts:</p><pre>from sklearn.model_selection import train_test_split</pre><pre>Xtrain, Xtest, ytrain, ytest = train_test_split(X, y, random_state=50)   </pre><blockquote>Remember that the <strong>training set</strong> is used to learn the model, instead the <strong>test set</strong> is used to test the quality of your model, and in many ML competitions is hidden to participants.</blockquote><p>Now, let’s try to design our ML model: to perform the binary classification task I have chosen a Support Vector Machine. Is it a good choice?</p><pre>from sklearn.svm import SVC</pre><pre>SV=SVC(random_state=0).fit(Xtrain, ytrain)    #train the model<br>SV.score(Xtrain, ytrain)   #calculate the score</pre><pre>&gt;&gt;&gt; 1.0   # max score!</pre><p>Our model has learned the training examples perfectly! But is that really the case? Have we settled on a perfect classification scheme? <strong><em>No!</em></strong></p><p>To accurately measure the performance, we need to use a separate set of examples, which the model has not yet seen. Of course, this is the role of the test set…</p><pre>SV.score(Xtest, ytest)</pre><pre>&gt;&gt;&gt; 0.6293706293706294    # low score</pre><p>The model is not able to generalize to new examples: this is the well known <strong>OVERFITTING</strong>!</p><p>In this case we learn two important things. <br>First, the fact that the model gives accurate results on training examples <em>(score 1.0)</em>, only means that it has learned completely the training examples, i.e. it fits perfectly our data, and it is okay. <br>But it doesn’t mean it will work equally well on new examples: it doesn’t guarantee a good generalization <em>(score 0.63)</em>. It’s our job to find the right balance between approximation and generalization!</p><p>Second, the main rule you should know: <strong>never use training examples to measure the quality of a model!</strong> It produces a false prediction of the goodness of your work.</p><p>But… that’s what <strong>validation</strong> is for!</p><p>With validation, one can immediately notice if your model is overfitting, and correct it in time.<br>The basic idea is to split the data with labels into two parts, one is used for the training itself, the other for validation. Validation set try to reproduce the test set, but this time, you can use the resulting information to correct the model (generally you can’t do this with the real test set, often it is unknown, mostly in ML competition).</p><blockquote>An important thing to keep in mind when splitting your dataset in train and validation parts is that samples between train and validation must not overlap: if we evaluate our model with an example yet seen, the performance of the model increases, but we well know that is not a good thing. This is sometimes the case when we have repeated samples in the data: be aware to exclude duplicates before starting the validation steps!</blockquote><p>And now, the time has come to create our validation set!</p><p>We apply another time the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html">train_test_split</a> command, but this time in order to divide the set in training and validation parts:</p><pre>train_x, valid_x, train_y, valid_y = train_test_split(Xtrain, ytrain, test_size=0.2, random_state=20)</pre><p>Now, we can train our model on the ‘new’ training set and test its performance on the validation set:</p><pre>SV=SVC(random_state=0).fit(train_x, train_y)    #train<br>SV.score(valid_x, valid_y)                      #calculate the score</pre><pre>&gt;&gt;&gt; 0.627906976744186</pre><p>We can notice in time that this model is not too good, and avoid using it!</p><p>Let’s try with other models… (this is the <em>model selection</em> step)</p><pre>from sklearn.ensemble import RandomForestClassifier<br>from sklearn.linear_model import LogisticRegression<br>from sklearn.tree import DecisionTreeClassifier<br>from sklearn.naive_bayes import GaussianNB<br>from sklearn.neural_network import MLPClassifier</pre><pre>RF=RandomForestClassifier(random_state=0).fit(train_x, train_y)<br>RF.score(valid_x, valid_y)</pre><pre>&gt;&gt;&gt; 0.9767441860465116</pre><pre>clf = LogisticRegression(random_state=0).fit(train_x, train_y) <br>clf.score(valid_x, valid_y)</pre><pre>&gt;&gt;&gt; 0.9534883720930233</pre><pre>DT=DecisionTreeClassifier(random_state=0).fit(train_x, train_y)  <br>DT.score(valid_x, valid_y)</pre><pre>&gt;&gt;&gt; 0.9418604651162791</pre><pre>NB=GaussianNB().fit(train_x, train_y)   <br>NB.score(valid_x, valid_y)</pre><pre>&gt;&gt;&gt; 0.9418604651162791</pre><pre>MLP=MLPClassifier(random_state=0).fit(train_x, train_y)   <br>MLP.score(valid_x, valid_y)</pre><pre>&gt;&gt;&gt; 0.9302325581395349</pre><p>And the winner is… Random Forest Classifier! We can now stay safe and use this classifier to perform our binary classification on the breast cancer dataset.</p><blockquote>Notice that these validation schemes are supposed to be used to estimate the quality of the model. When you’ve found the right model and hyper-parameters and want to get test predictions,<em> </em><strong>don’t forget to retrain your model using all training data.</strong></blockquote><pre>#retrain the model with the entire and original training set</pre><pre>final_model=RandomForestClassifier(random_state=0).fit(Xtrain, ytrain)<br>final_model.score(Xtest, ytest)   #the final evaluation</pre><pre>&gt;&gt;&gt; 0.9300699300699301</pre><p>We have correct overfitting (almost) by using a different model.</p><p>(Another crucial step in model selection is <em>hyperparameter tuning</em>, that consists in adjusting the intrinsic parameters that characterize each model. In this case we are able to combat definitively overfitting! You will see in detail this theme in a future article.)</p><p>This validation strategy is called <strong>Hold Out </strong>and is the first method you can use to perform the validation step. It is useful when you’re dealing with big datasets and in situations in which different splits lead to similar validation scores. But what happens if the dataset size is limited? Or when different ways to divide the dataset recommend to use different models?</p><p>Check out the next article to discover more validation strategies and what to do in such cases!</p><p><em>You can find my notebook with the entire code </em><a href="https://github.com/MLJCUnito/ProjectX2020/blob/master/HowToTackleAMLCompetition/Lecture3%7COverfittingRevisitedAndValidation.ipynb"><em>here</em></a><em>.</em></p><p><em>References:</em></p><ul><li><em>‘Learning From Data, a short course’, Hsuan-Tien Lin, Yaser S. Abu-Mostafa, Malik Magdon-Ismail</em></li><li><a href="https://scikit-learn.org/stable/auto_examples/model_selection/plot_underfitting_overfitting.html#underfitting-vs-overfitting"><em>Underfitting vs. Overfitting</em></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a3861eda10c7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mljcunito/overfitting-review-and-the-validation-intervention-a3861eda10c7">Overfitting review and the Validation intervention</a> was originally published in <a href="https://medium.com/mljcunito">MLJC</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Last Step in Data Preprocessing: Handling Missing Values]]></title>
            <link>https://medium.com/mljcunito/the-last-step-in-data-preprocessing-handling-missing-values-ae8f19dae309?source=rss----b7c1069b4442---4</link>
            <guid isPermaLink="false">https://medium.com/p/ae8f19dae309</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[mljcunito]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Simone Azeglio]]></dc:creator>
            <pubDate>Sun, 30 Aug 2020 11:59:28 GMT</pubDate>
            <atom:updated>2020-08-30T11:59:28.548Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1mLboEEMIdHjHJbI8_Z7Hg.jpeg" /><figcaption>Source: Freepik</figcaption></figure><blockquote>We are at the end of this mini-series on Data Preprocessing, the last step (if you’ve missed the previous article, you can find it <a href="https://medium.com/mljcunito/a-second-step-into-feature-engineering-feature-selection-d597619e6e2b">here</a>) to overcome is how to handle missing values. In this short article, after introducing three categories of missing data, we’ll go trough a few simple techniques which can lead us to a solution.</blockquote><p>“ <em>Missingness</em>” is almost always informative by itself, and we should tell our algorithm if a value is missing. Even if we build a model to impute our values, we are not adding any real information. We’re just reinforcing the patterns already provided by other features.</p><p>Basically, there are three categories of missing data:</p><ul><li><strong>MCAR</strong> (Missing Completely At Random) where the pattern of missinginess is statistically independent of the data record. Example: you have a data set on a piece of paper and you spill coffee on the paper destroying part of the data.</li><li><strong>MAR</strong> (Missing At Random) where the probability distribution of the pattern of missingness is functionally dependent upon the observable component in the record. MCAR is a special case of MAR. Example: if a child does not attend an educational assessment because the child is (genuinely) ill, this might be predictable from other data we have about the child’s health, but it would not be related to what we would have measured had the child not been ill.</li><li><strong>MNAR</strong> (Missing Not at Random) which is defined as the case which is NOT MAR, or when the missingness is specifically related to what is missing. Example: a person does not attend a drug test because the person took drugs the night before.</li></ul><p>Let’s see a few strategies to impute missing values, i.e. to infer them from the known part of the data.</p><h4>Univariate Feature Imputation</h4><p>We can rely on scikit-learn’s <a href="https://scikit-learn.org/stable/modules/generated/sklearn.impute.SimpleImputer.html#sklearn.impute.SimpleImputer">SimpleImputer</a> class, which provides a few strategies for imputing missing values, such as : imputing by a constant value, by using statistics (mean, median, etc).</p><p>Let’s see how we can replace missing values ( np.nan) using the mean value of the columns that contain the missing values.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ff07ed28679637968f2a512528de8a0b/href">https://medium.com/media/ff07ed28679637968f2a512528de8a0b/href</a></iframe><p>SimpleImputer can also be used in conjunction with <em>pandas</em> and in particular with data represented as strings or categoricals, by using most_frequentor constant strategy:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d22d5e9e00bbb95de8bff2294ecf89fb/href">https://medium.com/media/d22d5e9e00bbb95de8bff2294ecf89fb/href</a></iframe><h4>Multivariate Feature Imputation</h4><p>Another approach is to use the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.impute.IterativeImputer.html#sklearn.impute.IterativeImputer">IterativeImputer</a> class, which models each feature with missing values as a function of other features. How does each iteration work? At each step, a feature column is designated as output yand the other feature columns are treated as inputs X. Then, the regressor is used to predict the missing values of y. This is done for each feature in an iterative fashion.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/868c6528965ecf17ed95eb5a4527022b/href">https://medium.com/media/868c6528965ecf17ed95eb5a4527022b/href</a></iframe><p>IterativeImputer is very flexible, it allows you to use a variety of estimators, if you want to further delve into this class take a look at <a href="https://scikit-learn.org/stable/auto_examples/impute/plot_iterative_imputer_variants_comparison.html#sphx-glr-auto-examples-impute-plot-iterative-imputer-variants-comparison-py">Imputing missing values with variants of IterativeImputer</a>.</p><h4>Nearest Neighbors Imputation</h4><p>The <a href="https://scikit-learn.org/stable/modules/generated/sklearn.impute.KNNImputer.html#sklearn.impute.KNNImputer">KNNImputer</a> class offers imputation for filling in missing values using the k-Nearest Neighbors approach. By default it uses an euclidean distance metric that supports missing values, nan_euclidean_distances.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*WpVMZ0_RxWc6MoOH.gif" /></figure><p>The following code snippet shows how to replace missing values using the mean feature value of the two nearest neighbors of samples with missing values:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/891626f9c49edda54fd67ced0be0c0c3/href">https://medium.com/media/891626f9c49edda54fd67ced0be0c0c3/href</a></iframe><p>Thank you for your attention, in the next mini-series of article we’re going to introduce EDA (Exploratory Data Analysis) and some techniques for data visualization.</p><p>Here it follows a short list of additional articles/ tutorials for Data Preprocessing tasks.</p><h3>References &amp; Additional Material</h3><p><strong>Preprocessing Data</strong></p><ul><li><a href="https://sebastianraschka.com/Articles/2014_about_feature_scaling.html">About Feature Scaling by Sebastian Raschka</a></li><li><a href="https://scikit-learn.org/stable/modules/preprocessing.html">Scikit-learn’s Preprocessing</a></li></ul><p><strong>Feature Selection</strong></p><ul><li><a href="https://scikit-learn.org/stable/auto_examples/feature_selection/plot_feature_selection.html#sphx-glr-auto-examples-feature-selection-plot-feature-selection-py">Univariate</a></li><li><a href="https://scikit-learn.org/stable/auto_examples/feature_selection/plot_select_from_model_diabetes.html#sphx-glr-auto-examples-feature-selection-plot-select-from-model-diabetes-py">Feature Selection using SelectFromModel and LassoCV</a></li><li><a href="https://scikit-learn.org/stable/auto_examples/feature_selection/plot_select_from_model_diabetes.html#sphx-glr-auto-examples-feature-selection-plot-select-from-model-diabetes-py">Tree-based feature importance 1</a></li><li><a href="https://scikit-learn.org/stable/auto_examples/ensemble/plot_forest_importances.html#sphx-glr-auto-examples-ensemble-plot-forest-importances-py">Tree-based feature importance 2</a></li></ul><p><strong>Encoding Categorical Features</strong></p><ul><li><a href="https://hackernoon.com/what-is-one-hot-encoding-why-and-when-do-you-have-to-use-it-e3c6186d008f">One Hot Encoding</a></li><li><a href="https://scikit-learn.org/stable/auto_examples/text/plot_hashing_vs_dict_vectorizer.html">Feature Hasher vs DictVectorizer</a></li></ul><p><strong>Feature Importance</strong></p><ul><li><a href="https://machinelearningmastery.com/calculate-feature-importance-with-python/">Calculate Feature Importance by Machinelearningmastery</a></li></ul><p><strong>Feature Selection</strong></p><ul><li><a href="https://en.wikipedia.org/wiki/Feature_selection">Feature Selection, Wikipedia</a></li><li><a href="https://machinelearningmastery.com/an-introduction-to-feature-selection/">Feature Selection 1, by Machinelearningmastery</a></li><li><a href="https://machinelearningmastery.com/feature-selection-machine-learning-python/">Feature Selection 2, by Machinelearningmastery</a></li><li><a href="https://machinelearningmastery.com/feature-selection-with-real-and-categorical-data/">Feature Selection 3, by Machinelearningmastery</a></li></ul><p><strong>Handling Missing Values</strong></p><ul><li><a href="https://scikit-learn.org/stable/modules/impute.html#marking-imputed-values">Marking Imputed Values</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ae8f19dae309" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mljcunito/the-last-step-in-data-preprocessing-handling-missing-values-ae8f19dae309">The Last Step in Data Preprocessing: Handling Missing Values</a> was originally published in <a href="https://medium.com/mljcunito">MLJC</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DeepFake Detection Challenge: an Overview]]></title>
            <link>https://medium.com/mljcunito/deepfake-detection-challenge-an-overview-a8f6fc5f6cfe?source=rss----b7c1069b4442---4</link>
            <guid isPermaLink="false">https://medium.com/p/a8f6fc5f6cfe</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[mljcunito]]></category>
            <category><![CDATA[social-media]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deepfakes]]></category>
            <dc:creator><![CDATA[Simone Azeglio]]></dc:creator>
            <pubDate>Fri, 28 Aug 2020 15:01:18 GMT</pubDate>
            <atom:updated>2020-08-28T15:04:21.971Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yO1hcqKdoGYjsU6pFxdyPg.png" /><figcaption>Source: OpenDemocracy</figcaption></figure><blockquote>At MLJCUnito we deeply believe that, first of all, AI should be ethical. That’s why a few months ago we decided to take part into DeepFake Detection Challenge: a joint contribution from Amazon AWS, Facebook and Microsoft. Deepfake techniques, in short, present realistic AI-generated videos of people doing and saying fictional things. They possess the potential to have a significant impact on how people determine the legitimacy of information presented online, and, as citizens we should be well aware of this technology.</blockquote><h3>DeepFake: a threat to democracy or just a bit of fun?</h3><blockquote><em>“We are already at the point where you can’t tell the difference between deepfakes and the real thing,”</em> <strong>Professor Hao Li, University of Southern California</strong></blockquote><p><strong>Facebook</strong> <a href="https://about.fb.com/news/2020/01/enforcing-against-manipulated-media/">has announced</a> it will remove videos modified by artificial intelligence, known as deepfakes, from its platform.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FcQ54GDm1eL0%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DcQ54GDm1eL0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FcQ54GDm1eL0%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/002d8915c14edd7968069d71086104ba/href">https://medium.com/media/002d8915c14edd7968069d71086104ba/href</a></iframe><h4><a href="http://www.kaggle.com/"><strong>Kaggle</strong></a> is an <strong>AirBnB for Data Scientists</strong></h4><p>This is where they spend their nights and weekends. It’s a crowd-sourced platform to attract, nurture, train and challenge data scientists from all around the world to solve data science, machine learning and predictive analytics problems. It has over 536,000 active members from 194 countries and it receives close to 150,000 submissions per month. Started from Melbourne, Australia Kaggle moved to Silicon Valley in 2011, raised some 11 million dollars from the likes of Hal Varian (Chief Economist at Google), Max Levchin (Paypal), Index and Khosla Ventures and then ultimately been acquired by the Google in March of 2017. Kaggle is the number one stop for data science enthusiasts all around the world who compete for prizes and boost their Kaggle rankings. There are only 94 Kaggle Grandmasters in the world to this date.</p><p>Do you know that most data scientists are only theorists and rarely get a chance to practice before being employed in the real-world? Kaggle solves this problem by giving data science enthusiasts a platform to interact and compete in solving real-life problems. The experience you get on Kaggle is invaluable in preparing you to understand what goes into finding feasible solutions for big data.</p><p>Fine, fine, fine but what do we do on Kaggle? <strong>We Learn</strong></p><p>Deepfakes are fakes generated by deep learning. So far so easy.</p><p>This usually means someone used a generative model like an <a href="http://ufldl.stanford.edu/tutorial/unsupervised/Autoencoders/"><em>AutoEncoder</em></a> or most likely a <a href="https://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf"><em>Generative Adversarial Network</em></a>, short <strong>GAN</strong>. GANs are technically two networks that work against each other, illustrated below. The <em>artist</em> ( <strong>generator</strong>) draws its inspiration from a noise sample and creates a rendering of the data you are trying to generate with said GAN. The private <em>investigator</em> ( <strong>discriminator</strong>) randomly gets assigned real and fake data to investigate.</p><p>The learning process is collaborative. The generator gets better at fooling the discriminator and the discriminator gets better at figuring out which data is real and which isn’t. In mathematical terms they are learning until a <a href="https://en.wikipedia.org/wiki/Nash_equilibrium">Nash equilibrium</a> is reached, which means neither can learn new tricks and get better. They’re a really cool concept and even used in scientific simulation at <a href="https://indico.cern.ch/event/595059/contributions/2497383/attachments/1431666/2199445/gan_presentation_IML.pdf">CERN</a>.</p><p>You can probably guess that they can be tricky to train, due to so many moving parts. This has become a very popular area of research, warranting a <a href="https://github.com/hindupuravinash/the-gan-zoo">GAN Zoo</a> of all named GANs. Some important stuff you may want to check out if your interested are keywords like Wasserstein GANs, Gradient Penalization, Attention, and in this context: Style Transfer (namely face2face).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*l3VxPIN0Cum4vbmD" /></figure><p>It sounds absurd, I know. <a href="https://poloclub.github.io/ganlab/">Here you can find some more practical examples</a>, why don’t you play with them for a while?</p><h4>A few tips for the <a href="https://www.kaggle.com/c/deepfake-detection-challenge/">Official Challenge on Kaggle</a></h4><ul><li>I strongly encourage you to start first with the <a href="https://www.kaggle.com/c/deepfake-detection-challenge/overview/getting-started">official Getting Started guide here</a>.</li><li>What is the goal of the Deepfake Detection Challenge? According to the FAQ “The AI technologies that power deepfakes and other tampered media are rapidly evolving, making deepfakes so hard to detect that, at times, even human evaluators can’t reliably tell the difference. The Deepfake Detection Challenge is designed to incentivize rapid progress in this area by inviting participants to compete to create new ways of detecting and preventing manipulated media.”</li><li>In this Code Competition:</li><li>CPU Notebook &lt;= 9 hours run-time, GPU Notebook &lt;= 9 hours run-time on Kaggle’s P100 GPUs, <strong>No internet access enabled</strong></li><li>External data is allowed <strong>up to 1 GB in size</strong>. External data must be freely &amp; publicly available, including pre-trained models</li><li>This code competition’s training set is not available directly on Kaggle, as its size is prohibitively large to train in Kaggle. Instead, it’s strongly recommended that you train offline and load the externally trained model as an external dataset into Kaggle Notebooks to perform inference on the Test Set. Review Getting Started for more detailed information.</li></ul><p>Submissions are scored on <a href="http://wiki.fast.ai/index.php/Log_Loss">log loss</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DDyk-PzE4fw-jhvK" /></figure><p>where:</p><ul><li><em>n</em> is the number of videos being predicted</li><li>is the predicted probability of the video being FAKE</li><li>is 1 if the video is FAKE, 0 if REAL</li><li><em>log()</em> is the natural (base e) logarithm</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d5cadcc9ef7ebb981810811c17f94c72/href">https://medium.com/media/d5cadcc9ef7ebb981810811c17f94c72/href</a></iframe><h4><a href="https://deepfakedetectionchallenge.ai/faqs">Data</a></h4><ul><li>We have a bunch of <em>.mp4 </em>files, split into compressed sets of ~10GB a piece. A <em>metadata.json</em> accompanies each set of .<em>mp4</em> files, and contains filename, label (REAL/FAKE), original and split columns, listed below under Columns.</li><li>The full training set is just over 470 GB (<strong>Yeah it’s huge !</strong>).</li></ul><p>There are 4 groups of datasets associated with this competition.</p><p><strong>Training Set:</strong> This dataset, containing labels for the target, is available for download for competitors to build their models. It is broken up into 50 files, for ease of access and download. Due to its large size, it must be accessed through a GCS bucket which is only made available to participants after accepting the competition’s rules. Please read the rules fully before accessing the dataset, as they contain important details about the dataset’s permitted use. <em>It is expected and encouraged that you train your models outside of Kaggle’s notebooks environment and submit to Kaggle by uploading the trained model as an external data source.</em></p><p><strong>Public Validation Set:</strong> When you commit your Kaggle notebook, the submission file output that is generated will be based on the small set of 400 videos/ids contained within this Public Validation Set. This is available on the Kaggle Data page as test_videos.zip</p><p><strong>Public Test Set:</strong> This dataset is completely withheld and is what Kaggle’s platform computes the public leaderboard against. When you “Submit to Competition” from the “Output” file of a committed notebook that contains the competition’s dataset, your code will be re-run in the background against this Public Test Set. When the re-run is complete, the score will be posted to the public leaderboard. If the re-run fails, you will see an error reflected in your “My Submissions” page. Unfortunately, we are unable to surface any details about your error, so as to prevent error-probing. <em>You are limited to 2 submissions per day, including submissions with errors.</em></p><p><strong>Private Test Set:</strong> This dataset is privately held outside of Kaggle’s platform, and is used to compute the private leaderboard. It contains videos with a similar format and nature as the Training and Public Validation/Test Sets, but are real, organic videos with and without deepfakes. After the competition deadline, Kaggle transfers your 2 final selected submissions’ code to the host. They will re-run your code against this private dataset and return prediction submissions back to Kaggle for computing your final private leaderboard scores.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/df73bb60368b0fe87f78056988e9e9c0/href">https://medium.com/media/df73bb60368b0fe87f78056988e9e9c0/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/78eb7d2d545a0dca7de5211bb481734c/href">https://medium.com/media/78eb7d2d545a0dca7de5211bb481734c/href</a></iframe><h4>Review of Data Files Accessible within kernel</h4><ul><li><strong>train_sample_videos.zip</strong> — a ZIP file containing a sample set of training videos and a metadata.json with labels. the full set of training videos is available through the links provided above.</li><li><strong>sample_submission.csv</strong> — a sample submission file in the correct format.</li><li><strong>test_videos.zip</strong> — a zip file containing a small set of videos to be used as a public validation set. To understand the datasets available for this competition, review the Getting Started information.</li><li><strong>filename</strong> — the filename of the video</li><li><strong>label</strong> — whether the video is REAL or FAKE</li><li><strong>original</strong> — in the case that a train set video is FAKE, the original video is listed here</li><li><strong>split</strong> — this is always equal to “train”.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e9cb642730cb558ccf3184f11f207058/href">https://medium.com/media/e9cb642730cb558ccf3184f11f207058/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/01277620235c84dba8f4320686e6baa8/href">https://medium.com/media/01277620235c84dba8f4320686e6baa8/href</a></iframe><h4>Detection Starter Kit</h4><blockquote>A <strong>quickstart guide</strong> on DeepFakes: <a href="https://arxiv.org/abs/2001.00179">“DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detection</a></blockquote><p>Here it follows a Deep Fakes video<strong> EDA</strong> (<em>Exploratory Data Analysis</em>). It relies on <a href="https://www.ffmpeg.org/download.html">static FFMPEG</a> to read/extract data from videos.</p><ul><li>It extracts meta-data. They help us to know frame rate, dimensions and audio format (we can forget leak of “display_ratio” as it will be fixed).</li><li>It extracts frames of videos as <em>PNG</em>.</li><li>It extracts audio track as <em>AAC </em>(disabled).</li><li>It compares a few face detectors (OpenCV HaarCascade, MTCNN). In the end we opted for <em>RetinaFace</em> indeed.</li><li>It provides basic statistics on faces per video, face width/height and face detection confidence. It computes an average face width/height.</li></ul><p>We notice that face detection (with OpenCV currently) is far from being perfect. An additional stage to clean-up detected faces is required before training a model! Maybe some kind of <em>votes/ensemble</em> with different detectors would help.</p><p>In this kernel you will see also some interesting edge cases of face detection:</p><ul><li>Face detected on a t-shirt.</li><li>Face detected on a background board.</li><li>Face detected inside a face.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5601802593b8ed8bf693153e68710861/href">https://medium.com/media/5601802593b8ed8bf693153e68710861/href</a></iframe><p>What is <a href="https://ffmpeg.org/ffprobe.html">ffprobe</a> indeed? Basically, ffprobe gathers information from multimedia streams and prints it in human — and machine — readable fashion.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3e7a662148455d414e4cc2ccbd7b2112/href">https://medium.com/media/3e7a662148455d414e4cc2ccbd7b2112/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2164a3e69ada0b61065ba134d5b749be/href">https://medium.com/media/2164a3e69ada0b61065ba134d5b749be/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/68170ebbd5f7fa8d872e7c9f112039ab/href">https://medium.com/media/68170ebbd5f7fa8d872e7c9f112039ab/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c01873e53f7090b38212a1e1879633f7/href">https://medium.com/media/c01873e53f7090b38212a1e1879633f7/href</a></iframe><p>A few info on <a href="https://filmora.wondershare.com/video-editing-tips/what-is-video-bitrate.html">bitrate</a></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e2fe6910267802509094cb90033caa90/href">https://medium.com/media/e2fe6910267802509094cb90033caa90/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0a427e7219c19b081cab98a8ce4297a9/href">https://medium.com/media/0a427e7219c19b081cab98a8ce4297a9/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/caf995a401162663824b4b318b8fc26d/href">https://medium.com/media/caf995a401162663824b4b318b8fc26d/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ce989fdd8bfafa829b3c5d37b61a2b85/href">https://medium.com/media/ce989fdd8bfafa829b3c5d37b61a2b85/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e538c35c12c9d9860a41102642da155a/href">https://medium.com/media/e538c35c12c9d9860a41102642da155a/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c84fe49ddd4f8fa6f69c1b455beb5608/href">https://medium.com/media/c84fe49ddd4f8fa6f69c1b455beb5608/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/54eca9ba1003fae52f671d7e7d8a600d/href">https://medium.com/media/54eca9ba1003fae52f671d7e7d8a600d/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/def9eb46f7a654f3659c611843f2e2b4/href">https://medium.com/media/def9eb46f7a654f3659c611843f2e2b4/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d0ace17cbd2b17f3dc9bd6135f4f4b69/href">https://medium.com/media/d0ace17cbd2b17f3dc9bd6135f4f4b69/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f021cfbe47b4582fcdcf1951d296ce2a/href">https://medium.com/media/f021cfbe47b4582fcdcf1951d296ce2a/href</a></iframe><p>Try to run the code by yourself, you can find a Jupyter notebook file on <a href="https://github.com/sazio/DeepFakeChallenge/blob/master/DeepFake_Overview.ipynb">my Github</a>. If you explore the same repository you’ll find the code we used for the whole pipeline, soon we’re going to convert it in a series of articles.</p><blockquote>Thank you for your attention, if you like our projects follow us on Medium and Github. Soon we’ll be out with more articles.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a8f6fc5f6cfe" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mljcunito/deepfake-detection-challenge-an-overview-a8f6fc5f6cfe">DeepFake Detection Challenge: an Overview</a> was originally published in <a href="https://medium.com/mljcunito">MLJC</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Second Step into Feature Engineering: Feature Selection]]></title>
            <link>https://medium.com/mljcunito/a-second-step-into-feature-engineering-feature-selection-d597619e6e2b?source=rss----b7c1069b4442---4</link>
            <guid isPermaLink="false">https://medium.com/p/d597619e6e2b</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[mljcunito]]></category>
            <dc:creator><![CDATA[Simone Azeglio]]></dc:creator>
            <pubDate>Fri, 21 Aug 2020 12:10:33 GMT</pubDate>
            <atom:updated>2020-08-21T15:56:55.915Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>We are ready to start with the second part of Feature Engineering (if you’ve missed the previous article, you can find it <a href="https://medium.com/mljcunito/an-introduction-to-feature-engineering-feature-importance-7e8265eb3a36">here</a>). In this short article we’ll go trough a few simple techniques in Feature Selection and Extraction.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XHHToil9E5EFeEh0H0rnjA.jpeg" /></figure><blockquote>Not all features are created equal</blockquote><blockquote>Zhe Chen</blockquote><h3>Feature Selection</h3><p>There would always be some features which are less important with respect to a specific problem. Those irrelevant features need to be removed. <em>Feature selection</em> addresses these problems by automatically selecting a subset that is most useful to the problem.</p><p>Most of the times the reduction in the number of input variables shrinks the computational cost of modeling, but sometimes it might happen that it also improves the performance of the model.</p><p>Among the large amount of feature selection methods we’ll focus mainly on statistical-based ones. They involve evaluating the relationship between each input variable and the target variable using statistics. These methods are usually fast and effective, the only issue is that statistical measures depends on the data type of both input and output variables.</p><p>The classes in the sklearn.feature_selection module can be used for feature selection/dimensionality reduction on sample sets.</p><p>Whenever you want to go for a simple approach, there’s always a <em>threshold</em> involved. VarianceThreshold is a simple baseline approach to select features. It removes all features whose variance doesn&#39;t reach a certain threshold.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4b812d22d663cc33cebca2bafd83c3fe/href">https://medium.com/media/4b812d22d663cc33cebca2bafd83c3fe/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cef1c50d785388c8a3fbf16586f58320/href">https://medium.com/media/cef1c50d785388c8a3fbf16586f58320/href</a></iframe><h4>Univariate Feature Selection</h4><p>Univariate feature selection examines each feature individually to determine the strength of the relationship of the feature with the response variable.</p><p>There are a few different options for univariate selection:</p><p>We can perform chi-squared (𝝌²) test to the samples to retrieve only the two best features:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6c3ed19ec548eda19b1834fa6a028c1d/href">https://medium.com/media/6c3ed19ec548eda19b1834fa6a028c1d/href</a></iframe><p>We have different scoring functions for regression and classification, some of them are listed here:</p><ul><li>Regression: <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.f_regression.html#sklearn.feature_selection.f_regression">f_regression</a>, <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.mutual_info_regression.html#sklearn.feature_selection.mutual_info_regression">mutual_info_regression</a></li><li>Classification: <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.chi2.html#sklearn.feature_selection.chi2">chi2</a>, <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.f_classif.html#sklearn.feature_selection.f_classif">f_classif</a>, <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.mutual_info_classif.html#sklearn.feature_selection.mutual_info_classif">mutual_info_classif</a></li></ul><h4>Recursive Feature Elimination</h4><p>Recursive Feature Elimination (RFE) as its name suggests recursively removes features, builds a model using the remaining attributes and calculates model accuracy. RFE is able to work out the combination of attributes that contribute to the prediction on the target variable.</p><p>Given an external estimator that assigns weights to features (e.g., the coefficients of a linear model), the goal of recursive feature elimination (RFE) is to select features by recursively considering smaller and smaller sets of features.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aeb79017aa3e8e5a968b6e25a2b5e41b/href">https://medium.com/media/aeb79017aa3e8e5a968b6e25a2b5e41b/href</a></iframe><h3>Feature Extraction (Bonus)</h3><blockquote>Feature extraction is very different from Feature selection: the former consists in transforming arbitrary data, such as text or images, into numerical features usable for machine learning. The latter is a machine learning technique applied on these features.</blockquote><p>We’ve decided to show you a standard technique from <em>sklearn.</em></p><h4>Loading Features from Dicts</h4><p>The class <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.DictVectorizer.html#sklearn.feature_extraction.DictVectorizer">DictVectorizer</a> transforms lists of feature-value mappings to vectors.</p><p>In particular, it turns lists of mappings (dict-like objects) of feature names to feature values into <em>Numpy</em> arrays or <em>scipy.sparse</em> matrices for use with scikit-learn estimators.</p><p>While not particularly fast to process, Python’s dict has the advantages of being convenient to use, being sparse (absent features need not be stored) and storing feature names in addition to values.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5bfb633d57a1329d3d3e932d9f988e1b/href">https://medium.com/media/5bfb633d57a1329d3d3e932d9f988e1b/href</a></iframe><p>DictVectorizer is also a useful representation transformation for training sequence classifiers in Natural Language Processing (NLP).</p><h4>Feature Hashing</h4><p>Named as one of the best hacks in Machine Learning, <a href="https://scikit-learn.org/stable/modules/feature_extraction.html#feature-hashing">Feature Hashing</a> is a fast and space-efficient way of vectorizing features, i.e. turning arbitrary features into indices in a vector or matrix. For this topic, sklearn’s documentation is exhaustive, you can find it in the link above.</p><h3>Feature Construction</h3><p>There’s no strict recipe for Feature Construction, I personally consider it as 99% creativity. We’re gonna take a look at some use cases in the next lectures though.</p><p>So far, you should take a look at the Feature Extraction part of t<a href="https://www.kaggle.com/gaborfodor/from-eda-to-the-top-lb-0-367">his marvellous notebook</a> from <em>Beluga,</em> one of the best Competitions Grandmasters on Kaggle.</p><p>In the last two articles we’ve been introducing Feature Engineering as a subsequent step to Feature Processing. As you can see, we’re building a data processing pipeline, indeed, the next step would be finding a way to deal with <em>missing values. </em>Stay tuned for the next article and don’t forget to take a look at our <a href="https://github.com/MLJCUnito/ProjectX2020">Github page</a>, you’ll find the code related to this series of articles.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d597619e6e2b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mljcunito/a-second-step-into-feature-engineering-feature-selection-d597619e6e2b">A Second Step into Feature Engineering: Feature Selection</a> was originally published in <a href="https://medium.com/mljcunito">MLJC</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Introduction to Feature Engineering: Feature Importance]]></title>
            <link>https://medium.com/mljcunito/an-introduction-to-feature-engineering-feature-importance-7e8265eb3a36?source=rss----b7c1069b4442---4</link>
            <guid isPermaLink="false">https://medium.com/p/7e8265eb3a36</guid>
            <category><![CDATA[mljcunito]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Simone Azeglio]]></dc:creator>
            <pubDate>Tue, 18 Aug 2020 12:39:49 GMT</pubDate>
            <atom:updated>2020-08-18T12:39:48.930Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vWsOfiQ78s7f6HZ6wpLoHg.png" /><figcaption>Applying Feature Engineering techniques is like building a lego house</figcaption></figure><blockquote>After having read the <a href="https://medium.com/mljcunito/data-spectrometry-or-how-to-preprocess-your-data-2a6593ee5dbb">first article</a> of this series, devoted to figure out a way to tackle a Machine Learning (ML) competition, we are now ready to go on with Feature Engineering, and, in particular Feature Importance.</blockquote><p>The real deal is that nobody explicitly tells you what <strong>feature engineering</strong> is, in some way, you are expected to understand for yourself what are good features.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aiC1GF0XWxz_bHQl.jpg" /></figure><blockquote>Feature engineering is another topic which doesn’t seem to merit any review papers or books, or even chapters in books, but it is absolutely vital to ML success. […] Much of the success of machine learning is actually success in engineering features that a learner can understand.</blockquote><blockquote>(Scott Locklin, in “Neglected machine learning ideas”)</blockquote><p>Let’s try to figure out what feature engineering is.</p><p>In solving such problems, our goal is to get the best possible result from a model. In order to achieve that, we need to extract useful information and get the most from what we have. On one side, this includes getting the best possible result from the algorithms we are employing. On the other side, it also involves getting the most out of the available data.</p><p><em>How do we get the most out of our data for predictive modeling?</em></p><p>Feature engineering tries to find an answer to this question.</p><blockquote>Actually, the success of all Machine Learning algorithms depends on how you present the data.</blockquote><blockquote>( Mohammad Pezeshki, answer to “What are some general tips on feature selection and engineering that every data scientist should know?”)</blockquote><h3>Feature Importance</h3><p>Feature importance refers to a bunch of techniques that assign a score to input features based on how useful they are at predicting a target variable. These scores play an important role in predictive modeling, they usually provide useful insights into the dataset and the basis for dimensionality reduction and feature selection.</p><p>Feature importance scores can be calculated both for regression and classification problems.</p><p>These scores can be used in a range of situations, such as:</p><ul><li><em>Better understanding the data</em>: the relative scorse can highlight which features may be most relevant to the target, and on the other side, which are least relevant. This could be a useful notion for a domain expert and could be used as a basis for gathering more or different data.</li><li><em>Better understanding a model</em>: inspecting the importance score provides insights into the specific model we’re using and which features are the most important to the model when elaborating a prediction.</li><li><em>Reducing the number of input features</em>: we can use the importance scores to select those features to delete (lowest score) and those to keep (highest scores).</li></ul><p>Now let’s jot down a few lines of code in order to grasp this topic in a better way. In order to explore feature importance scores, we’ll import a few test datasets directly from <em>sklearn</em>.</p><p><strong>Classification Dataset</strong></p><p>Easy peasy, we can use the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_classification.html"><em>make_classification()</em></a> function to create a test binary classification dataset.</p><p>We can specify the number of samples and the number of features, some of them are going to be informative and the remaining redundant. (<em>Tip</em>: you should fix the <em>random seed</em>, in this way you’ll get a reproducible result)</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/75bdd5962b4c5cad6a0c80ce35dc1d1d/href">https://medium.com/media/75bdd5962b4c5cad6a0c80ce35dc1d1d/href</a></iframe><p><strong>Regression Dataset</strong></p><p>In a parallel fashion, we’ll use the <a href="https://machinelearningmastery.com/calculate-feature-importance-with-python/#:~:text=Feature%20importance%20refers%20to%20techniques,at%20predicting%20a%20target%20variable."><em>make_regression()</em></a> function to create a regression dataset.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f86af911813405717db1c9b7f33d7944/href">https://medium.com/media/f86af911813405717db1c9b7f33d7944/href</a></iframe><h3>Coefficients as Feature Importance</h3><p>When we think about linear machine learning algorithms, we always fit a model where the prediction is the weighted sum of the input values (e.g. linear regression, logistic regression, ridge regression etc..)</p><p>These coefficients can be used directly as naive feature importance scores. Firstly we’ll fit a model on the dataset to find the coefficients, then summarize the importance scores for each input feature and create a bar chart to get an idea of the relative importance.</p><p><strong>Linear Regression Feature Importance</strong></p><p>It’s time to fit a <a href="https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html"><em>LinearRegression()</em></a> model on the regression dataset and get the <em>coef_</em> property that contains the coefficients. The only assumption is that the input variables have the same scale or have been scaled prior to fitting the model.</p><p>This same approach can be used with regularized linear models, such as Ridge and ElasticNet.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/df81365e34034cd55c21f064352eea0e/href">https://medium.com/media/df81365e34034cd55c21f064352eea0e/href</a></iframe><p><strong>Logistic Regression Feature Importance</strong></p><p>In a similar fashion, we can do the same to fit a <a href="https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html"><em>LogisticRegression()</em></a> model.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0f1633396526b4fdb77281282c91df1f/href">https://medium.com/media/0f1633396526b4fdb77281282c91df1f/href</a></iframe><p>Recall that this is a classification problem with classes 0 and 1 (binary). Notice that the coefficients are both positive and negative, positive scores indicate a feature that predicts class 1 while negative scores indicate a feature that predicts class 0.</p><p><em>Why can’t we analyze a regression problem with Logistic Regression?</em> (A pretty naive question, try to answer tho)</p><h3>Decision Tree Feature Importance</h3><p>Decision Tree algorithms like <strong>C</strong>lassification <strong>A</strong>nd <strong>R</strong>egression <strong>T</strong>rees ( <strong>CART</strong>) offer importance scores based on the reduction in the criterion used to select split points, like Gini or Entropy. This approach can be also used for ensembles of decision trees, such as Random Forest and Gradient Boositng algorithms.</p><p>We can directly use the CART algorithm for feature importance implemented in Scikit-Learn as the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeRegressor.html"><em>DecisionTreeRegressor</em></a> and <a href="https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html"><em>DecisionTreeClassifier</em></a>.</p><p>The model provides a <em>feature_importances_</em> property that tells us the relative importance scores for each feature.</p><p><strong>CART Regression Feature Importance</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c71810588dce0e4dbff3fd9f6d755b9b/href">https://medium.com/media/c71810588dce0e4dbff3fd9f6d755b9b/href</a></iframe><p><strong>CART Classification Feature Importance</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e5d047de2f3fe3ded86d6ab9c9901c87/href">https://medium.com/media/e5d047de2f3fe3ded86d6ab9c9901c87/href</a></iframe><h3>Random Forest Feature Importance</h3><p>Analogously, we can use the RandomForest algorithm for feature importance implemented in scikit-learn as the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html"><em>RandomForestRegressor</em></a> and <a href="https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html"><em>RandomForestClassifier</em></a>.</p><p>As above, the model provides a <em>feature_importances_</em> property.</p><p><strong>Random Forest Regression Feature Importance</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5da269773457f8b117bae6d556eb92e3/href">https://medium.com/media/5da269773457f8b117bae6d556eb92e3/href</a></iframe><p><strong>Random Forest Classification Feature Importance</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5019cbcdcd1bfc105da1ddfd308e6996/href">https://medium.com/media/5019cbcdcd1bfc105da1ddfd308e6996/href</a></iframe><h3>XGBoost Feature Importance</h3><p><a href="https://xgboost.readthedocs.io/en/latest/index.html">XGBoost</a> is a Python library that provides an efficient implementation of the stochastic gradient boostig algorithm. (For an introduction to Boosted Trees, you can take a look <a href="https://xgboost.readthedocs.io/en/latest/tutorials/model.html">here</a>)</p><p>This algorithm can be integrated with Scikit-Learn via the <em>XGBRegressor</em> and <em>XGBClassifier</em> classes.</p><p>Even in this one, we can find the <em>feature_importances_</em> property.</p><p>First, let’s install the XGBoost library, with pip:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1336fea87c083ebf21d576ad8f96c3a4/href">https://medium.com/media/1336fea87c083ebf21d576ad8f96c3a4/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7a74f4e38e89f963ecfe192ac4799142/href">https://medium.com/media/7a74f4e38e89f963ecfe192ac4799142/href</a></iframe><p>Now, let’s take a look at an example of XGBoost for feature importance.</p><p><strong>XGBoost Regression Feature Importance</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/af8d9ca335d5f64dd637f9ac3410656a/href">https://medium.com/media/af8d9ca335d5f64dd637f9ac3410656a/href</a></iframe><p><strong>XGBoost Classification Feature Importance</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4a1d111fdf88e43f8c604bfabfa51262/href">https://medium.com/media/4a1d111fdf88e43f8c604bfabfa51262/href</a></iframe><p>Permutation feature importance is a technique for calculating relative importance scores that is independent of the model used. It measures the increase in the prediction error of the model after we permuted the feature’s values, which breaks the relationship between the feature and the true outcome.</p><p>The concept is really straightforward: we measure the importance of a feature by calculating the increase in the model’s prediction error after permuting the feature. A feature is “important” if shuffling its values increases the model error, because in this case the model relied on the feature for the prediction. A feature is “unimportant” if shuffling its values leaves the model error unchanged, because in this case the model ignored the feature for the prediction.</p><p>Permutation feature selection can be used via the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.inspection.permutation_importance.html"><em>permutation_importance()</em></a> function that take a fit model, a dataset and a scoring function.</p><p>Let’s try this approach with an algorithm that doesn’t support feature selection natively, <a href="https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm">KNN</a> (K-Nearest Neighbors).</p><p><strong>Permutation Feature Importance for Regression</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cae12aeb1112fce6ae8f851043726846/href">https://medium.com/media/cae12aeb1112fce6ae8f851043726846/href</a></iframe><p><strong>Permutation Feature Importance for Classification</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fda780618f72afab51796e4a631282ff/href">https://medium.com/media/fda780618f72afab51796e4a631282ff/href</a></iframe><p><strong>Feature Selection with Importance</strong></p><p>Feature importance scores can be used to find useful insights and interpret the data, but they can also be used directly to help rank and select features that are most useful. This procedure is usually referred as <em>Feature Selection</em>, and we’ll look at it in more detail soon.</p><p>In our case, we can show how is possible to find redundant features by using the previously shown techniques.</p><p>Firstly, we can split the dataset into train and test sets, train a model on the training set, make predictions on the test set and evaluate the results by employing classification accuracy. We’ll use a Logistic Regression model to fit our data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a986db71a6027559185ed21fe7ab7c8a/href">https://medium.com/media/a986db71a6027559185ed21fe7ab7c8a/href</a></iframe><p>In this case, we can see that our model achieved a classification accuracy of about <em>86.67 %</em> using all the features in the dataset.</p><p>Let’s see what happens if we select only relevant features. We could use any of the feature importance scores above, but in this case we’ll use the ones provided by random forest.</p><p>We can use the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.SelectFromModel.html"><em>SelectFromModel</em></a> class to define both the model abd the number of features to select.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/981d44799a94f1e73f28baac02caa7a4/href">https://medium.com/media/981d44799a94f1e73f28baac02caa7a4/href</a></iframe><p>This will calculate the importance scores that can be used to rank all input features. We can then apply the method as a transform to select a subset of 5 most important features from the dataset. This transform will be applied to the training set and the test set.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/72b40c515c12d82e2da56c550049add6/href">https://medium.com/media/72b40c515c12d82e2da56c550049add6/href</a></iframe><p>We can wrap up every piece and get this code snippet.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/838c5aecb5fb274c66fcede8571ecac1/href">https://medium.com/media/838c5aecb5fb274c66fcede8571ecac1/href</a></iframe><p>In this case, we can see that the model achieves the same performance on the dataset, although with almost half of the features.</p><p>It’s time to press the pause button for now, you should take a look at the code above and try to tweak some parameters, you might find better results too. In the next article we’ll cover Feature Selection and Feature Extraction, stay tuned! 👩‍💻👨‍💻</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7e8265eb3a36" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mljcunito/an-introduction-to-feature-engineering-feature-importance-7e8265eb3a36">An Introduction to Feature Engineering: Feature Importance</a> was originally published in <a href="https://medium.com/mljcunito">MLJC</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Spectrometry or How to Preprocess your Data]]></title>
            <link>https://medium.com/mljcunito/data-spectrometry-or-how-to-preprocess-your-data-2a6593ee5dbb?source=rss----b7c1069b4442---4</link>
            <guid isPermaLink="false">https://medium.com/p/2a6593ee5dbb</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[mljcunito]]></category>
            <category><![CDATA[projectx2020]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Simone Azeglio]]></dc:creator>
            <pubDate>Fri, 31 Jul 2020 15:09:01 GMT</pubDate>
            <atom:updated>2020-07-31T16:58:06.303Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rzyaIB4v1udby1uxloxl8g.jpeg" /><figcaption>Chad Hagen, Nonsensical Infographic №5, 2009, digital</figcaption></figure><blockquote>This is the first article of a to-be series devoted to figure out a way to tackle a Machine Learning (ML) competition. A few months ago, the MLJCUnito gained access to an international ML competition on climate change, organized by the University of Toronto: <a href="http://www.projectx2020.com/">www.projectx2020.com</a>. This competition will start on September 2020, so we decided to start gathering some useful material related to competition dynamics such as Data Preprocessing, Feature Engineering and convert them in a lecture-like format.</blockquote><p>In this first part, we’d like to tell you about some practical tricks for making <strong>gradient descent</strong> work well, in particular, we’re going to delve into feature scaling. As an introductory view, it seems reasonable to try to depict an intuition of the concept of <em>scale</em>.</p><h3>Macro, Meso, Micro-scale in Science</h3><p>As scientists, we are well aware of the effects of using a specific measurement tool in order to characterize some quantity and describe reality. As an ideal example we consider the <strong>length scale</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*mxtbVvVB3v5NTbbp.png" /><figcaption>Images of two systems (one screw and one boot) observed from the macroscopic (left), mesoscopic (center) and microscopic (right) point of view.</figcaption></figure><p>We can identify three different points of view: <em>microscopic</em>, <em>mesoscopic</em> and <em>macroscopic</em>; which are intimately related to the adopted lenght scale.</p><p>We usually deal with the <em>macroscopic scale</em> when the observer is in such a position (pretty far, in terms of distance), with respect to the object, that she/he can describe its global characteristics. Instead, we do refer to the <em>microscopic scale</em> when the observer is so close to the object that she/he can describe its atomistic details or elementary parts (e.g. molecules, atoms, quarks). Last but not least, we talk about <em>mesoscopic scale</em> everytime we are in between micro and macro.</p><p>These definitions are deliberately vague, since delineating a precise and neat explanation would be higly difficult and complex, and it’s actually far from our purposes.</p><p>On the other side, this kind of introduction is quite useful, we should take a few minutes to think about the “active” role of the observer and about the fact that, to be honest, for every length scale, there’s some specific theory, i.e. there’s no global theory for a multi-scale description of some phenomenon.</p><h3>Scaling in Data Science</h3><p>If our beloved observer (i.e. the scientist) has some kind of “privilege”, i.e. choosing the right measurement tool, which is nothing but choosing the right scale in the description of some phenomenon, we can’t really say the same for a data scientist.</p><p>It’s a sort of paradox, but a data scientist can’t really deal with data retrieval most of the times. Because of that, a data scientist is often left alone in front of data, without even knowing from which measurement tool they’re coming from. There’s no way to interact with the length scale for example.</p><p>Is there something that we can do about it? The only thing we can do is assuming that features are independent and scale these features in order to have something compatible from one to the other. This procedure is called <strong>feature scaling</strong>, and soon we’ll understand why it is useful even for ML algorithms, such as gradient descent.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*Cn7w5qo3PAd3om6t.png" /></figure><p>If you make sure that features are on similar scales, i.e. features take on similar range of values, then gradient descent can converge more quickly.</p><p>More concretely, let’s say we have a problem with two features where <strong><em>x</em></strong>₁ is the length of a football field and take values between <em>90</em> (meters) and <em>115</em> (meters) and <strong><em>x</em></strong>₂ is the radius of a ball which takes values between <em>10.5</em>(centimeters) to <em>11.5</em> (centimeters). If you plot the countours of the cost function <strong><em>J(ω)</em></strong> then you might get something similar to the <em>left plot</em>, and because of these very skewed elliptical shape, if we run gradient descent on this cost function, it may end up taking a long time and oscillating back and forth before reaching the global minimum.</p><p>In these settings, as stated previously, a useful thing to do is to scale the features. Generally, the idea is to get every feature into approximately a <em>-1</em> to <em>+1</em> range. By doing this, we get the <em>right plot</em>. In this way, you can find a much more direct path to the global minimum rather than taking a much more convoluted path where you’re sort of trying to follow a very complicated trajectory.</p><h3>Preprocessing Data</h3><blockquote>In any Machine Learning process, Data Preprocessing is that step in which the data gets transformed, or encoded, to bring it to such a state that now the machine can easily parse it. In other words, the features of the data can now be easily interpreted by the algorithm.</blockquote><p>We’re going to dive into Scikit-Learn for this section and exploit its powerful <em>processing</em> package.</p><p>We’ve been talking about scaling our data, now it’s time to understand how to put our hands on code and try to do that. Usually, as previously stated, learning algorithms benefit from standardization of the data set. If some outliers are present in the set, robust scalers or transformers might be more appropriate. (Take a look at <a href="https://scikit-learn.org/stable/auto_examples/preprocessing/plot_all_scaling.html#sphx-glr-auto-examples-preprocessing-plot-all-scaling-py">Compare the effect of different scalers on data with outliers</a>, you’ll see the behaviors of the different scalers, transformers and normalizers with outliers).</p><h3>Standardization</h3><p>Many Machine Learning estimators require <em>standardization</em> of datasets, elseways they might behave badly because data are far from a Gaussian (with zero mean and unit variance) distribution.</p><p>Most of the times, we ignore the shape of the distribution and just transform the data by subtracting the mean value of each feature, then scale by dividing features by their standard deviation.</p><p><em>Do you have in mind some models that assume that all features are centered around zero and have variance in the same order of magnitude? Can you think about possible issues related to the objective function in these cases?</em></p><p>A possible answer: many elements used in the objective function of a learning algorithm (such as the <em>RBF kernel</em> of <em>Support Vector Machines</em> or the <em>l1</em> and <em>l2</em> regularizers of linear models) assume that all features are centered around zero and have variance in the same order. If a feature has a variance that is orders of magnitude larger than others, it might dominate the objective function and make the estimator unable to learn from other features correctly as expected.</p><p>There’s a fast way to do that on a single array, by means of the <em>scale</em> function.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a57908529c2d0afe95fc3862c96e6e17/href">https://medium.com/media/a57908529c2d0afe95fc3862c96e6e17/href</a></iframe><p>The <em>preprocessing</em> module provides a utility class <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html#sklearn.preprocessing.StandardScaler"><em>StandardScaler</em></a> that compute the mean and std on a training set so as to be able to later reapply the same transform on the test set.</p><p>(You should be well aware of what <a href="https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html#sklearn.pipeline.Pipeline"><em>sklearn.pipeline.Pipeline</em></a> is, it’s crucial for strategies’ deployment.)</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/efa617fc0cce6c44730e9ca53f3e3449/href">https://medium.com/media/efa617fc0cce6c44730e9ca53f3e3449/href</a></iframe><p>Now we can use the scaler instance on new data, to transform them in the same way we did previously.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/906716a194713981fed547fc336dcffe/href">https://medium.com/media/906716a194713981fed547fc336dcffe/href</a></iframe><p>It is possible to disable centering or scaling by passing <em>with_mean = False</em> or <em>with_std = False</em>. The first one might be particularly useful if applied to sparse CSR or CSC matrices to avoid breaking the sparsity structure of the data.</p><p><strong>Scaling Features to a Range</strong></p><p>Another standardization is scaling features to lie between a given minimum and maximum value, or so that the maximum absolute value of each future is scaled to unit size. This can be achieved with <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html#sklearn.preprocessing.MinMaxScaler"><em>MinMaxScaler</em></a> or <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MaxAbsScaler.html#sklearn.preprocessing.MaxAbsScaler"><em>MaxAbsScaler</em></a>.</p><p>Here you can see how to scale a toy data matrix to the <em>[0,1]</em> range:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1c5015f0213577f079ab72630b8117e2/href">https://medium.com/media/1c5015f0213577f079ab72630b8117e2/href</a></iframe><p>In the same way as above, the same instance of the transformer can be applied to some new test data: same scaling and shifting will be applied to be consistent.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5cf12eb830e0141ef6ad211b5f98459f/href">https://medium.com/media/5cf12eb830e0141ef6ad211b5f98459f/href</a></iframe><p>It’s pretty useful to let the scaler reveal some details about the transformation learned on the training data:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bc007def251c302a93863374ce2d4579/href">https://medium.com/media/bc007def251c302a93863374ce2d4579/href</a></iframe><p>Can you retrieve the explicit formula for <em>MinMaxScaler</em>?</p><p>Here’s the solution</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qycZhu6uSTaACblZjut9hg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hDdvlr-A_4qXkGJPtYOR2w.png" /></figure><p><em>MaxAbsScaler</em> works in a similar fashion, the data will lie in the range <em>[-1,1]</em>. It is meant for data that is already centered at zero or sparse data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/742401f9a3d96e2fba65fa6b873ac75d/href">https://medium.com/media/742401f9a3d96e2fba65fa6b873ac75d/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/964419f1ce87775b20caad2a77d94c03/href">https://medium.com/media/964419f1ce87775b20caad2a77d94c03/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c1071852dafdb8ecdcb1144bf119557c/href">https://medium.com/media/c1071852dafdb8ecdcb1144bf119557c/href</a></iframe><p><strong>Scaling Data with Outliers</strong></p><p>If our data contain many outliers, scaling using the mean and variance of the data is not likely to work well. In this case, we can use <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.RobustScaler.html#sklearn.preprocessing.RobustScaler"><em>RobustScaler</em></a>.</p><p>This scaler removes the <a href="https://en.wikipedia.org/wiki/Median">median</a> and scales data according to the <a href="https://en.wikipedia.org/wiki/Interquartile_range">IQR</a> (InterQuartile Range).</p><p>Centering and scaling happen independently on each feature by computing the relevant statistics on the samples in the training set.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/71a3857832f1a1a19297f4acbafe49e3/href">https://medium.com/media/71a3857832f1a1a19297f4acbafe49e3/href</a></iframe><p>Median and interquartile range are then stored to be used on later data using the <em>transform</em> method.</p><h3>Non-linear transformations</h3><p>It’s possible to generalize to non-linear transformations. We are going to talk about two types of transformations: <em>quantile transforms</em> and <em>power transforms</em>. The main take-home message is that we need <em>monotonic</em> transformations to preserve the rank of the values along each feature.</p><p>Quantile transforms smooth out unusual distributions and are less influenced by outliers than scaling methods. It does distort correlations and distances within and across features.</p><p>Power transforms are, indeed, a family of parametric transformations that aim to map data from any distribution to as close to a Gaussian distribution.</p><p><strong>Mapping to a Uniform Distribution</strong></p><p><a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.QuantileTransformer.html#sklearn.preprocessing.QuantileTransformer"><em>QuantileTransformer</em></a> provides a non-parametric transformation to map the data to a uniform distribution with values between 0 and 1:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aca7f04a650ffcd79e3ee36c6278fefd/href">https://medium.com/media/aca7f04a650ffcd79e3ee36c6278fefd/href</a></iframe><p>This feature corresponds to the sepal length in cm. Once the quantile transform is applied, those landmarks approach closely the percentiles previously defined</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/52f8511822b70b5be3531e8780d9b098/href">https://medium.com/media/52f8511822b70b5be3531e8780d9b098/href</a></iframe><p>Some more applications <a href="https://machinelearningmastery.com/quantile-transforms-for-machine-learning/">here</a>.</p><p><strong>Mapping to a Gaussian Distribution</strong></p><p>Many machine learning algorithms prefer or perform better when numerical input variables and even output variables in the case of regression have a Gaussian distribution. Power transforms are a family of parametric, monotonic transforms that aim to map data from any distribution to as close to a Gaussian distribution as possible, in order to stabilize variance and minimize skewness.</p><p><a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PowerTransformer.html#sklearn.preprocessing.PowerTransformer"><em>PowerTransformer</em></a> provides two transformations, the <em>Yeo-Johnson</em> transform:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o49kyDielDx0_pS-s5FZWg.png" /></figure><p>and the <em>Box-Cox</em> transform:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/957/1*vVNA5FCtIaOKgyWp6yqlvQ.png" /></figure><p>Box-Cox can only be applied to strictly positive data. In both methods, the transformation is parametrized by $\lambda$, which is determined trough maximum-likelihood estimation. Here an example of using Box-Cox to map samples drawn from a lognormal distribution to a normal distribution:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/25251388abc112069683e254bd5a7cac/href">https://medium.com/media/25251388abc112069683e254bd5a7cac/href</a></iframe><p>(Some more applications <a href="https://machinelearningmastery.com/power-transforms-with-scikit-learn/">here</a>)</p><p>Below some examples of the two transforms applied to various probability distributions, <em>any comment</em>?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/0*lIAgUh67_Lgm7yqh.png" /></figure><h3>Normalization</h3><p>As scientists, we feel much more comfortable with Vector Space Models. <em>Normalization</em> is the process of scaling individual samples to have unit norm. This process might be useful if we plan to use a dot-product or some kernel to quantify similarities of pairs of samples.</p><p>The function provides a quick and easy way to perform this operation on a single array, using <a href="https://medium.com/@montjoile/l0-norm-l1-norm-l2-norm-l-infinity-norm-7a7d18a4f40c">L1 or L2 norms</a>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b5e53bd3abc3aed11d09a953d0cced92/href">https://medium.com/media/b5e53bd3abc3aed11d09a953d0cced92/href</a></iframe><p>The <em>preprocessing</em> module provides a utility class <em>Normalizer</em> that implements the same operation using the <em>Transformer</em> API. This class is suitable for <em>sklearn.pipeline.Pipeline</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a11e4c4e21a93e472f491dfbf801d628/href">https://medium.com/media/a11e4c4e21a93e472f491dfbf801d628/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e206595051d49b3dd8689c38216b3572/href">https://medium.com/media/e206595051d49b3dd8689c38216b3572/href</a></iframe><h3>Encoding Categorical Features</h3><p>In many cases, features are not continous values but categorical. E.g. a person could have some features: [&quot;from Italy&quot;, &quot;from France&quot;, &quot;from Germany&quot;], [&quot;play sports&quot;, &quot;doesn&#39;t play sports&quot;], [&quot;uses Firefox&quot;, &quot;uses Opera&quot;, &quot;uses Chrome&quot;, &quot;uses Safari&quot;, &quot;uses Internet Explorer&quot;].</p><p>Such features can be efficiently coded as integers, for instance [&quot;from France&quot;, &quot;play sports&quot;, &quot;uses Chrome&quot;] could be [1,0,2].</p><p>To convert categorical features to such integer codes, we can use the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OrdinalEncoder.html#sklearn.preprocessing.OrdinalEncoder"><em>OrdinalEncoder</em></a>. In this way, we transform each feature to one new feature of integers <em>(0</em> to <em>n_categories-1)</em>:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c2a76a62e1c9d2d1517ebab7449d0efe/href">https://medium.com/media/c2a76a62e1c9d2d1517ebab7449d0efe/href</a></iframe><p>Some scikit-learn estimators expect continuous input, and would interpret categories as ordered, which is usually not desired.</p><p>There’s another way to convert categorical features to features that can be used with scikit-learn estimators: <em>one-hot encoding</em>. It can be obtained with the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.OneHotEncoder.html#sklearn.preprocessing.OneHotEncoder"><em>OneHotEncoder</em></a>, which transforms each categorical feature with n_categoriespossible values into n_categories binary features, with one of them 1, and all others 0.</p><p>Let’s continue with the example above:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1d397649917632b6bde9c1e4d9bef4ba/href">https://medium.com/media/1d397649917632b6bde9c1e4d9bef4ba/href</a></iframe><p>The values each feature can take is inferred automatically from the dataset and can be found in the categories_ attribute:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f892af8f037a20b98896215f268de983/href">https://medium.com/media/f892af8f037a20b98896215f268de983/href</a></iframe><p>We are done with a brief introduction to Data Preprocessing. I’ve tried to show some of the most useful features of <em>scikit-learn, </em>which is definitely one of the cardinal libraries for ML. In the next lecture we are going to dive into Feature Engineering, which I personally consider the most fundamental part of a ML pipeline. Stay tuned for the next article !👩‍💻👨‍💻</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2a6593ee5dbb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mljcunito/data-spectrometry-or-how-to-preprocess-your-data-2a6593ee5dbb">Data Spectrometry or How to Preprocess your Data</a> was originally published in <a href="https://medium.com/mljcunito">MLJC</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DAIgnosis: Exploring the Space of Metrics]]></title>
            <link>https://medium.com/mljcunito/daignosis-exploring-the-space-of-metrics-c6bca5d53acb?source=rss----b7c1069b4442---4</link>
            <guid isPermaLink="false">https://medium.com/p/c6bca5d53acb</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[covid19]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[medicai]]></category>
            <dc:creator><![CDATA[Machine Learning Journal Club (MLJC)]]></dc:creator>
            <pubDate>Fri, 22 May 2020 13:28:15 GMT</pubDate>
            <atom:updated>2020-05-22T13:28:28.000Z</atom:updated>
            <content:encoded><![CDATA[<h4>Episode 1: Useful Metrics in Medical Diagnosis</h4><blockquote><em>Authors:</em> <a href="https://medium.com/u/2fefde940982">Simone Azeglio</a>, <a href="https://medium.com/u/820bad93a005">Arianna Di Bernardo</a>, <a href="https://medium.com/u/e2d829cbd63d">Nicolò Toscano</a>, <a href="https://medium.com/u/e35208ba419a">Carlo Alberto Maria Barbano</a></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3Y3pCTM2_-dCAa96yQtQaA.jpeg" /></figure><p><strong>One of the core tasks in building a machine learning model is to evaluate its performance. It’s fundamental, and it’s also really hard. While working on any project, especially in the context of healthcare, you should ask yourself: “How can I measure success for this project?” and “How would I know if and when I’ve succeeded?”. These questions allow you to set your goals in a realistic manner, so that you know when to stop. Sometimes they prevent you from working on ill-formulated projects where the concept of “good measurement” is vague or infeasible. It’s all a matter of <em>traffic-lights</em>. So how would one measure <em>the green light</em>: the success of a machine learning model? To answer this question, let’s take a sightseeing tour of machine learning model evaluation for disease diagnosis.</strong></p><blockquote>The most serious mistakes are not being made as a result of wrong answers. The truly dangerous thing is asking the wrong questions. <strong><em>Peter Drucker</em></strong></blockquote><p>It’s all about asking the right question: evaluation metrics are tied to machine learning tasks.</p><p>The problem of asking the right question is generally widespread in the Natural Sciences and it’s the very foundation of the scientific method paradigm, firstly introduced by <a href="https://en.wikipedia.org/wiki/Two_New_Sciences">Galileo Galilei</a>.</p><p>Galilei was able to sketch an outline of a completely new framework: experiments as a research tool. Such a little contribution had exponential consequences in many fields, and this is where asking and mathematically formulating questions kicks in. In order to build a successful model, i.e. a satisfactory representation of some phenomenon, we need to find the best <em>metric,</em> the best measurement method.</p><p>Now we are ready to take the baton and rephrase the formulation of “asking the right question” in machine learning terms, i.e. how do we choose the best <em>evaluation metric</em> for our problem?</p><p>This question is of primary importance, since metrics are the only sensory contribute that we have with respect to<a href="https://wp.wwu.edu/machinelearning/2017/01/22/generalization-and-overfitting/"> <em>generalization</em></a>. In better words, metrics will tell us if our model has learned or not, and hence: if our model is overfitting or not.</p><p>What we are going to do now, is a showcase of different evaluation metrics in a specific setting: medical diagnosis. By doing that we hope to shed light on the usefulness of these tools while keeping some amount of concreteness in terms of a widely known use case. But first we’re going to introduce to you the dataset that we selected for today, namely the <a href="https://www.kaggle.com/nih-chest-xrays/data">Chest X-ray 14 Dataset</a>.</p><h4>The Dataset</h4><p>The <a href="https://www.kaggle.com/nih-chest-xrays/data">Chest X-ray 14 Dataset</a> has been released by the NIH Clinical Center and it’s made up by over 100&#39;000 anonymized chest x-ray images, it’s considered a huge contribution to open science and machine learning research.</p><p>Regarding the data, there are <em>15</em> different<em> classes</em> corresponding to 14 different diseases and one class for “no findings”. For completeness we report the name of each class, since we are going to use them later: <em>Atelectasis, Consolidation, Infiltration, Pneumothorax, Edema, Emphysema, Fibrosis, Effusion, Pneumonia, Pleural Thickening, Cardiomegaly, Nodule Mass and Hernia.</em></p><p>That’s all we need to know for our purposes, but if you are interested in further details you can refer to the <a href="https://www.nih.gov/news-events/news-releases/nih-clinical-center-provides-one-largest-publicly-available-chest-x-ray-datasets-scientific-community">NIH News release</a>.</p><p>As far as we’ve seen, we are dealing with a <em>multi-class classification </em>problem (i.e. the output is not <em>binary</em> but <em>categorical</em>). Before starting with different metrics definitions, let’s take a quick peek at our dataset. For the sake of simplicity, we decided to sample the previously cited dataset and precompute model outputs for test cases, so you don’t have to worry about it. In this way you can find two different files <em>train_preds.csv</em> and <em>valid_preds.csv</em> (you can find every resource in our <a href="https://github.com/MLJCUnito/Datameron/tree/master/DAIgnosis/Article1">Github repository</a>)</p><p>By using <a href="https://pandas.pydata.org/">pandas</a> we create two dataframes from our <em>.csv</em> files, correctly name class labels and define <em>predicted labels</em> which correspond to normalized probabilities with respect to a specific class, i.e. the output of the model.</p><pre>#importing .csv files as pandas dataframes<br>train_results = pd.read_csv(&quot;train_preds.csv&quot;)<br>valid_results = pd.read_csv(&quot;valid_preds.csv&quot;)</pre><pre># labels in our dataset<br>class_labels = [&#39;Cardiomegaly&#39;,<br>&#39;Emphysema&#39;,<br>&#39;Effusion&#39;,<br>&#39;Hernia&#39;,<br>&#39;Infiltration&#39;,<br>&#39;Mass&#39;,<br>&#39;Nodule&#39;,<br>&#39;Atelectasis&#39;,<br>&#39;Pneumothorax&#39;,<br>&#39;Pleural_Thickening&#39;,<br>&#39;Pneumonia&#39;,<br>&#39;Fibrosis&#39;,<br>&#39;Edema&#39;,<br>&#39;Consolidation&#39;]</pre><pre># the labels for prediction values in our dataset<br>predicted_labels = [l + &quot;_pred&quot; for l in class_labels]</pre><pre># extract the ground trouth (class_values) and the predictions (pred)<br>class_values = valid_results[class_labels].values<br>pred = valid_results[pred_labels].values</pre><p>So far so good, we have our data correctly represented in pandas but let’s get an overview and check if there’s any class imbalance problem by plotting occurrences for each disease with <a href="https://matplotlib.org/">matplotlib</a>.</p><pre>#Bar Chart <br>#For the sake of visual clarity --&gt; sort bars in descending order<br>class_values = valid_results[class_labels].values #occurences<br>cnt_values = class_values.sum(axis = 0) #occurences x disease </pre><pre>df_to_plot = pd.DataFrame({&quot;Disease&quot;: class_labels, &quot;Count&quot;: cnt_values}) <br>df_sorted = df_to_plot.sort_values(&quot;Count&quot;)<br></pre><pre>#Creating our plot as horizontal bar chart <br>plt.figure(figsize=(12,6))<br>plt.title(&#39;Disease Incidence&#39;, pad=10)<br>plt.xlabel(&quot;Number of Patients&quot;, size=15)<br>plt.barh(&quot;Disease&quot;, &quot;Count&quot;, data = df_sorted, color=(240/255, 167/255, 95/255))<br>plt.grid(alpha = 0.3);<br>plt.tight_layout()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*APomGAFXZr3zgpms9ludUw.jpeg" /></figure><p>It’s pretty clear now: our dataset has an imbalanced population of samples, specifically, it has a small number of patients diagnosed with <em>Hernia</em>. The class imbalance problem affects almost every dataset, but today we’re not going to cope with it since it isn’t strictly correlated to evaluation metrics. Indeed, class imbalance is going to be one of the main topics of the next few articles, where we’re going to explore<em> </em><a href="https://www.geeksforgeeks.org/python-data-augmentation/"><em>data augmentation</em></a> in detail.</p><h4>Basic Statistics</h4><p>Everything starts from four basic statistics that we can compute from the model predictions: <em>true positives</em> (<strong>TP</strong>),<em> true negatives</em> (<strong>TN</strong>), <em>false positives</em> (<strong>FP</strong>) and <em>false negatives</em> (<strong>FN</strong>).</p><p>As the name suggests:</p><ul><li><em>true positive</em>: the model classifies the example as positive, and the actual label is also positive;</li><li><em>true negative</em>: the model classifies the example as negative and the actual label is also negative;</li><li><em>false positive</em>: the model classifies the example as positive, <strong>but</strong> the actual label is negative;</li><li><em>false negative</em>: the model classifies the example as negative,<strong> but </strong>the actual label is positive.</li></ul><p>These four are of unimaginable importance: every metrics can be built off of them.</p><p>Now a little technical rigidity: recall that the model outputs real numbers between 0 and 1, but as you can imagine the four statistics are binary by definition. Don’t you worry, we just need to define a threshold value<em> th</em> and any outputs above<em> th</em> will be set to 1, and below <em>th</em> to 0.</p><p>Here we define the four functions that return our beloved statistics.</p><pre>#-------------- TRUE POSITIVES --------------#<br>def true_positives(y, pred, th=0.5):</pre><pre>TP = 0 #true positives<br>thresholded_preds = pred &gt;= th # get thresholded predictions<br>TP = np.sum((y == 1) &amp; (thresholded_preds == 1)) # compute TP<br>return TP</pre><pre>#-------------- TRUE NEGATIVES --------------#<br>def true_negatives(y, pred, th=0.5):</pre><pre>TN = 0 #true negatives<br>thresholded_preds = pred &gt;= th # get thresholded predictions<br>TN = np.sum((y == 0) &amp; (thresholded_preds == 0)) # compute TN<br>return TN</pre><pre>#-------------- FALSE POSITIVES --------------#<br>def false_positives(y, pred, th=0.5):</pre><pre>FP = 0 # false positives<br>thresholded_preds = pred &gt;= th # get thresholded predictions<br>FP = np.sum((y == 0) &amp; (thresholded_preds == 1)) # compute FP<br>return FP</pre><pre>#-------------- FALSE NEGATIVES --------------#<br>def false_negatives(y, pred, th=0.5):</pre><pre>FN = 0 # false negatives<br>thresholded_preds = pred &gt;= th # get thresholded predictions<br>FN = np.sum((y == 1) &amp; (thresholded_preds == 0)) # compute FN<br>return FN</pre><p>Now let’s create a toy dataframe so we can see if everything works, if it’s the first time for you approaching these concepts you can try to manually fill the <em>category </em>column and double check the results.</p><pre>df = pd.DataFrame({&#39;y_test&#39;: [1,1,0,0,0,0,0,0,0,1,1,1,1,1],<br>&#39;preds_test&#39;: [0.8,0.7,0.4,0.3,0.2,0.5,0.6,0.7,0.8,0.1,0.2,0.3,0.4,0],<br>&#39;category&#39;: [&#39;TP&#39;,&#39;TP&#39;,&#39;TN&#39;,&#39;TN&#39;,&#39;TN&#39;,&#39;FP&#39;,&#39;FP&#39;,&#39;FP&#39;,&#39;FP&#39;,&#39;FN&#39;,&#39;FN&#39;,&#39;FN&#39;,&#39;FN&#39;,&#39;FN&#39;]<br>})</pre><pre>df # Show data </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/1*8EtRvMUsddKvZQx_l0OtUg.png" /></figure><p>We can compare predicted results with the ground truth.</p><pre># take a look at predictions and ground truth <br>y_test = df[&#39;y_test&#39;]<br>preds_test = df[&#39;preds_test&#39;]<br>threshold = 0.5</pre><pre>print(f&quot;&quot;&quot;Predicted results:<br>TP: {true_positives(y_test, preds_test, threshold)}<br>TN: {true_negatives(y_test, preds_test, threshold)}<br>FP: {false_positives(y_test, preds_test, threshold)}<br>FN: {false_negatives(y_test, preds_test, threshold)}<br>&quot;&quot;&quot;)</pre><pre>print(&quot;Expected results:&quot;)<br>print(f&quot;TP: {sum(df[&#39;category&#39;] == &#39;TP&#39;)}&quot;)<br>print(f&quot;TN {sum(df[&#39;category&#39;] == &#39;TN&#39;)}&quot;)<br>print(f&quot;FP {sum(df[&#39;category&#39;] == &#39;FP&#39;)}&quot;)<br>print(f&quot;FN {sum(df[&#39;category&#39;] == &#39;FN&#39;)}&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*9cbUTHRh00592GN8Q94s9g.png" /></figure><p>We get the same results, our model can be considered “qualitatively good”, but how do we <em>quantify</em> <em>how good is our model</em>?</p><p>Now it’s time to take a look at our dataset: let’s compute <strong>TP, TN, FP, FN </strong>for our cases.</p><pre>#TP computation<br>TP=[]<br>for i in range(len(class_labels)):<br>    TP.append(true_positives(class_values[:,i], pred[:,i], 0.5))<br>#TN computation    <br>TN=[]<br>for i in range(len(class_labels)):<br>    TN.append(true_negatives(class_values[:,i], pred[:,i], 0.5))<br>   <br>#FP computation <br>FP=[]<br>for i in range(len(class_labels)):<br>    FP.append(false_positives(class_values[:,i], pred[:,i], 0.5))<br>   <br>#FN computation <br>FN=[]<br>for i in range(len(class_labels)):<br>    FN.append(false_negatives(class_values[:,i], pred[:,i], 0.5))<br>#create a results table<br>table=pd.DataFrame({&#39;category&#39; : class_labels,<br>&#39;TP&#39;: TP,<br>&#39;TN&#39;: TN,<br>&#39;FP&#39;: FP,            <br>&#39;FN&#39;: FN,            <br>})<br>table.set_index(&#39;category&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v1ta3ajQ6M-wds04Q0ZmjA.png" /></figure><h4>The first brick: Accuracy</h4><p>Let’s start introducing the first metric that allows us to measure a model performance in a simple and intuitive way: <em>diagnostic accuracy</em>.</p><p>Accuracy answer to the question “<em>how good is our model?</em>” by only measuring how often our classification model makes the correct prediction:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*C43XLn142EJ1x1-LudXO4g.png" /></figure><p>In a probabilistic way, we can interpret the accuracy as the probability of being correct: in terms of <strong>TP</strong>, <strong>TN</strong>, <strong>FN</strong> and <strong>FP</strong>, accuracy defines the proportion of true positive and true negative individuals in a total group of subjects. Then,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z7CIY2_itRr5Qb1SjPH7YQ.png" /></figure><p>Now let’s work with our dataset: we will illustrate how to compute accuracy and then we’ll test it in the whole dataset.</p><pre>def get_accuracy(y, pred, th=0.5):</pre><pre>accuracy = 0.0<br>TP = true_positives(y, pred, th = th)<br>FP = false_positives(y, pred, th = th)<br>TN = true_negatives(y, pred, th = th)<br>FN = false_negatives(y, pred, th = th)<br>accuracy = (TP + TN)/(TP + TN + FP + FN)  # Accuracy computation return accuracy</pre><pre># Compute accuracy for the dataset classes<br>acc=[]<br>for i in range(len(class_labels)):<br>    acc.append(get_accuracy(class_values[:,i], pred[:,i], 0.5))<br>#create a results table<br>table2=pd.DataFrame({&#39;category&#39; : class_labels,<br>&#39;accuracy&#39;: acc          <br>})<br>table2.set_index(&#39;category&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/820/1*alSE05ZVi1pr57X6vH0DfA.png" /></figure><p>What would we say about the performance of our model? Is it good or not?</p><p>If we consider <em>Pneumonia</em> detection (accuracy = 0.675), the model is certainly not accurate. Take a look at Emphysema detection, instead (accuracy = 0.889): it is a little more precise! So we can say that our classification model is better in detecting<em> Emphysema</em> than<em> Pneumonia</em>.</p><p>What happens if we consider a different model, able to predict if the subject does not have <em>Emphysema</em> disease (i.e. the new model is a simple binary classifier)?</p><p>Compute the accuracy for this simple model:</p><pre>print(&#39;Emphysema disease accuracy =&#39;, get_accuracy(valid_results[&quot;Emphysema&quot;].values, np.zeros(len(valid_results))))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/781/1*-hGqrVdiBGnSjijcEF8usA.png" /></figure><p>It is a great result! This model is clearly better than the first one.</p><p>Let’s stop for a moment and consider the results we have just obtained. The first model discriminates between different kinds of disease; the second one classifies the presence or absence of a certain disease, and it performs significantly better. This gives us a starting point to define more specific evaluation metrics: in order to get more performant results, it is necessary to consider diagnostic metrics that evaluate how well the model predicts positives for patients with a certain disease, and negatives for healthy subjects.</p><h4>Prevalence</h4><p>With <em>prevalence</em> we can focus on the presence of a certain disease: it is, by definition, the probability to have a certain disease.</p><p>This metric relates to the proportion of individuals with disease in a total of subjects (<em>healthy + diseased</em>) and it is easily defined as the ratio between positive examples and the size of the sample:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WOyQkHK6Idr0Bli1qZKWtg.png" /></figure><p>where <em>y</em>ᵢ<em>=1</em> for positive examples.</p><p>Let’s define and compute prevalence for each disease in our dataset:</p><pre>def get_prevalence(y):</pre><pre>prevalence = 0.0<br>prevalence = np.sum(y)/len(y) #Prevalence computation<br>return prevalence</pre><pre># Compute accuracy for the dataset classes<br>prev=[]<br>for i in range(len(class_labels)):<br>    prev.append(get_prevalence(class_values[:,i]))<br>#create a results table<br>table3=pd.DataFrame({&#39;category&#39; : class_labels,<br>&#39;prevalence&#39;: prev         <br>})<br>table3.set_index(&#39;category&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/1*JaPIuaL025-qvkvlKCoEaQ.png" /></figure><p>High scores indicate common diseases (e.g. <em>Infiltration, 0.192</em>); low scores indicate rare diseases (e.g. <em>Hernia, 0.002</em>)</p><h4>Sensitivity and Specificity</h4><p><em>Sensitivity</em> and <em>specificity</em> are two complementary metrics, a little bit more precise than accuracy, but related with it.</p><p><em>Sensitivity</em> is the probability that the model predicts positive if the patient have the disease: it is the proportion of examples classified as positive in a total of positive examples.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aizE-FDIU0L1jhPY2-FnWg.png" /></figure><p><em>Specificity</em> is the probability that the model predicts negative for a subject without the disease: it is the proportion of examples classified as negative in a total of negative cases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uqHq4vV2l-nVAa8QOMVoow.png" /></figure><p>In terms of probability, sensitivity and specificity relate to the following conditional probabilities <strong>P</strong>(+|disease) and<strong> P</strong>(-|normal) (i.e. the probability that the model outputs positive, given a disease-subject, and the probability that the model outputs negative, given a healt subject).</p><p>Now let’s see how to compute this metrics and how they work on our dataset:</p><pre>#------SENSITIVITY------#</pre><pre>def get_sensitivity(y, pred, th=0.5):</pre><pre>sensitivity = 0.0<br>TP = true_positives(y, pred, th = th)<br>FN = false_negatives(y, pred, th = th)<br>sensitivity = TP/(TP + FN)<br>return sensitivity</pre><pre>#------SPECIFICITY------#</pre><pre>def get_specificity(y, pred, th=0.5):</pre><pre>specificity = 0.0<br>TN = true_negatives(y, pred, th = th)<br>FP = false_positives(y, pred, th = th)<br>specificity = TN/(TN + FP)<br>return specificity</pre><pre># Compute accuracy for the dataset classes<br>sens=[]<br>spec=[]<br>for i in range(len(class_labels)):<br>    sens.append(get_sensitivity(class_values[:,i], pred[:,i], 0.5))<br>    spec.append(get_specificity(class_values[:,i], pred[:,i], 0.5))</pre><pre>#create a results table<br>table4=pd.DataFrame({&#39;category&#39; : class_labels,<br>&#39;sensitivity&#39;: sens,<br>&#39;specificity&#39;: spec         <br>})<br>table4.set_index(&#39;category&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*Cc58cDoItCcV3DSd1_HEWg.png" /></figure><p>Specificity and sensitivity do not depend on the prevalence of the positive class in the dataset, but they don’t really tell us anything new. Sensitivity, for example, is the probability that our model outputs positive given that the person already has the condition!</p><h4>PPV and NPV</h4><p>As we’ve just seen, sensitivity and specificity are not diagnostically helpful. In the clinic, a doctor using some model might be interested in a different approach: given the model predicts positive on a patient, what is the probability that he actually has the disease? This is called the <em>positive predictive value</em> (<strong>PPV</strong>) of a model. Similarly, the doctor might want to know the probability that a patient is normal, given the model prediction is negative. This is called the <em>negative predictive value</em> (<strong>NPV</strong>) of a model.</p><p>To get an overall view of the metrics, let’s introduce the<em> confusion matrix,</em> which shows the comparison between the ground truth (the actual data) and what is obtained through the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qu2WMmaUAyCVeqJJi_yReg.png" /></figure><p>In the same way as we did before, let’s see how we can compute <strong>PPV</strong> and <strong>NPV</strong> using our data.</p><pre>#------ PPV ------#</pre><pre>def get_ppv(y, pred, th=0.5):<br>    <br>    PPV = 0.0<br><br>  # get TP and FP using our previously defined functions<br>    TP = true_positives(y, pred, th = th)<br>    FP = false_positives(y, pred, th = th)<br>  # use TP and FP to compute PPV<br>    PPV = TP/(TP + FP)<br>    <br>    return PPV</pre><pre>#------ NPV ------#</pre><pre>def get_npv(y, pred, th=0.5):<br>    <br>    NPV = 0.0<br> <br>  # get TN and FN using our previously defined functions<br>    TN = true_negatives(y, pred, th = th)<br>    FN = false_negatives(y, pred, th = th)<br>  # use TN and FN to compute NPV<br>    NPV = TN/(TN + FN)<br><br>    return NPV</pre><pre># Compute PPV and NPV for the dataset classes<br>PPV=[]<br>NPV=[]<br>for i in range(len(class_labels)):<br>    PPV.append(get_ppv(class_values[:,i], pred[:,i], 0.5))<br>    NPV.append(get_npv(class_values[:,i], pred[:,i], 0.5))</pre><pre>#create a results table<br>table5=pd.DataFrame({&#39;category&#39; : class_labels,<br>&#39;PPV&#39;: PPV,<br>&#39;NPV&#39;: NPV        <br>})<br>table5.set_index(&#39;category&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*XYuGXU6AMGWdeP-WHO5m4g.png" /></figure><h4><strong>Receiver Operating Characteristic (ROC) curve</strong></h4><p>So far we have been operating under the assumption that our model had a prediction threshold of <em>0.5</em>. But what happens if we change it? How this change affects the efficiency of our model?</p><p>A <em>Receiver Operating Characteristic (ROC) curve</em> is a plot that shows us the performance of our model as its prediction threshold is varied. To construct a ROC curve, we plot the true positive rate (<strong>TPR</strong>) againts the false positive rate (<strong>FPR</strong>), at various threshold settings.</p><pre>from sklearn.metrics import roc_curve, roc_auc_score</pre><pre>def get_ROC(y, pred, target_names):<br>    for i in range(len(target_names)):<br>      curve_function = roc_curve<br>      auc_roc = roc_auc_score(y[:, i], pred[:, i])<br>      label = target_names[i] + &quot; AUC: %.3f &quot; % auc_roc<br>      xlabel = &quot;False positive rate&quot;<br>      ylabel = &quot;True positive rate&quot;<br>      a, b, _ = curve_function(y[:, i], pred[:, i])<br>      plt.figure(1, figsize=(7, 7))<br>      plt.plot([0, 1], [0, 1], &#39;k--&#39;)<br>      plt.plot(a, b, label=label)<br>      plt.xlabel(xlabel)<br>      plt.ylabel(ylabel)<br>      plt.legend(loc=&#39;upper center&#39;, bbox_to_anchor=(1.3, 1),<br>                       fancybox=True, ncol=1)</pre><pre>#plot the curve<br>get_ROC(class_values, pred, class_labels)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/1*lJRRy4QmP8iXP5JsmCSdKg.png" /></figure><p>The shape of a ROC curve, and the area under it (A<em>rea Under the Curve </em><strong>AUC</strong>), tell us how performant is our model: a good model is represented by a ROC curve closed to the upper left hand corner (where TPR is approximately 1 and FPR is 0) and by a AUC value of 1; a bad model has a ROC curve that tends to the diagonal and the respective AUC value is about 0.5.</p><h4>Confidence Intervals</h4><p>Our dataset is only a sample of the real world, and our calculated values for all the above metrics are an estimate of real world values. Hence, it would be better to quantify this uncertainty due to the sampling of our dataset. We’ll do this through the use of confidence intervals. A 95% confidence interval for an estimate<strong> 𝑠̂</strong> of a parameter <strong>𝑠</strong> is an interval 𝐼=(𝑎,𝑏) such that 95% of the time, when the experiment is run, the true value <strong>𝑠</strong> is contained in 𝐼. More concretely, if we were to run the experiment many times, then the fraction of those experiments for which 𝐼 contains the true parameter would tend towards 95%. Thus, 95% confidence does not say that there is a 95% probability that <strong>s</strong> lies within the interval. It also does not say that 95% of the sample accuracies lie within this interval.</p><p>While some estimates come with methods for computing the confidence interval analytically, more complicated statistics, such as the <em>AUC</em> for example, are difficult. In these cases, we can use a method called <em>bootstrap</em>. The bootstrap estimates the uncertainty by resampling the dataset with replacement. For each resampling <strong>𝑖</strong>, we will get a new estimate, <strong>𝑠̂<em>(i)</em></strong>. We can then estimate the distribution of <strong>𝑠̂</strong> by using the distribution of <strong>𝑠̂<em>(i)</em></strong> for our bootstrap samples.</p><p>In the code below, we create bootstrap samples and compute sample AUCs from those samples. Note that we use <em>stratified random sampling </em>(sampling from positive and negative classes separately) to make sure that members of each class are represented.</p><pre>#------ computing intervals -------#</pre><pre>def bootstrap_auc(y, pred, classes, bootstraps = 100, fold_size = 1000):<br>    statistics = np.zeros((len(classes), bootstraps))</pre><pre>    for c in range(len(classes)):<br>        df = pd.DataFrame(columns=[&#39;y&#39;, &#39;pred&#39;])<br>        df.loc[:, &#39;y&#39;] = y[:, c]<br>        df.loc[:, &#39;pred&#39;] = pred[:, c]<br>        # get positive examples for stratified sampling<br>        df_pos = df[df.y == 1]<br>        df_neg = df[df.y == 0]<br>        prevalence = len(df_pos) / len(df)</pre><pre>        for i in range(bootstraps):<br>            pos_sample = df_pos.sample(n = int(fold_size * prevalence), replace=True)<br>            neg_sample = df_neg.sample(n = int(fold_size * (1-prevalence)), replace=True)<br>            y_sample = np.concatenate([pos_sample.y.values, neg_sample.y.values])<br>            pred_sample = np.concatenate([pos_sample.pred.values, neg_sample.pred.values])<br>            score = roc_auc_score(y_sample, pred_sample)<br>            statistics[c][i] = score<br>    return statistics</pre><pre>statistics = bootstrap_auc(class_values, pred, class_labels)</pre><pre>#------ printing table ------#</pre><pre>table7 = pd.DataFrame(columns=[&quot;Mean AUC (CI 5%-95%)&quot;])<br>for i in range(len(class_labels)):<br>    mean = statistics.mean(axis=1)[i]<br>    max_ = np.quantile(statistics, .95, axis=1)[i]<br>    min_ = np.quantile(statistics, .05, axis=1)[i]<br>    table7.loc[class_labels[i]] = [&quot;%.2f (%.2f-%.2f)&quot; % (mean, min_, max_)]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3IerKOE42YxccvtIly9rYw.png" /></figure><h4>Precision-Recall Curve</h4><p>Precision and recall are two metrics often used together:</p><ul><li><em>Precision</em> is a measure of result relevancy: it quantify the ability of our model to not label a negative subject as positive. The precision score is equivalent to PPV</li><li><em>Recall</em> is a measure of how many truly relevant results are returned: it is the probability that a positive prediction is actually positive. The recall score is equivalent to sensitivity</li></ul><p>The <em>precision-recall curve</em> (<strong>PRC</strong>) illustrate the relationship between precision and recall for different thresholds. Let’s compute it:</p><pre>#----- PRC -----#<br>from sklearn.metrics import average_precision_score, precision_recall_curve</pre><pre>def get_PRC(y, pred, target_names):<br>    for i in range(len(target_names)):<br>            precision, recall, _ = precision_recall_curve(y[:, i], pred[:, i])<br>            average_precision = average_precision_score(y[:, i], pred[:, i])<br>            label = target_names[i] + &quot; Avg.: %.3f &quot; % average_precision<br>            plt.figure(1, figsize=(7, 7))<br>            plt.step(recall, precision, where=&#39;post&#39;, label=label)<br>            plt.xlabel(&#39;Recall&#39;)<br>            plt.ylabel(&#39;Precision&#39;)<br>            plt.ylim([0.0, 1.05])<br>            plt.xlim([0.0, 1.0])<br>            plt.legend(loc=&#39;upper center&#39;, bbox_to_anchor=(1.3, 1),<br>                       fancybox=True, ncol=1)</pre><pre>#----- plot the curve -----#<br>get_PRC(class_values, pred, class_labels) </pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/655/1*w4qY7PoT8STAyIxiLI2jiw.png" /></figure><p>The area under the curve helps us to estimate once again how good is our model: high area is due to high recall and high precision, i.e. low false positive rate and low false negative rate, and therefore it represents a good performance.</p><h4>F1-Score</h4><p><em>F1-score</em> combines precision and recall trough their armonic mean:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8iaGnvgal6fcT7JMHjYrAA.png" /></figure><pre>#----- F1-score -----#<br>def get_f1(y, pred, th=0.5):<br>  <br>    F1 = 0.0<br>    <br>    # get precision and recall using our previously defined functions<br> <br>    precision=get_ppv(y, pred, th = th)<br>    recall=get_sensitivity(y, pred, th = th)</pre><pre># use precision and recall to compute F1<br>    F1 = 2 * (precision * recall) / (precision + recall)<br>    <br>    <br>    return F1</pre><p>Alternatively, we can simply use sklearn&#39;s utility metric function of f1_score to compute this measure:</p><pre>from sklearn.metrics import f1_score</pre><p>Now, let’s compute F1 score for the classes in our dataset:</p><pre>f1=[]<br>for i in range(len(class_labels)):<br> f1.append(get_f1(class_values[:,i], pred[:,i]))<br> <br>#create a results table<br>table8=pd.DataFrame({‘category’ : class_labels,<br>‘F1’: f1 <br>})<br>table8.set_index(‘category’)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/887/1*zcsbPklgzZFuy27_0Tnnhw.png" /></figure><p>F1-score lies in the range [0.1]: a perfect diagnostic model has a F1-score 1.</p><p>Be careful on overusing the F1-score though, in practice, different types of misclassifications incur different costs. <a href="https://app.dimensions.ai/details/publication/pub.1084928040">David Hand</a> and many others criticize the widespread since it gives equal importance to precision and recall.</p><h4><strong>Conclusion</strong></h4><p>At this point, as the icing on the cake, you can take a look at the final table, which compares results from every metric.</p><pre>table9=pd.DataFrame({&#39;category&#39; : class_labels,<br>&#39;TP&#39;: TP,<br>&#39;TN&#39;: TN,<br>&#39;FP&#39;: FP,            <br>&#39;FN&#39;: FN,<br>&#39;accuracy&#39;: acc,<br>&#39;prevalence&#39;: prev,<br>&#39;sensitivity&#39;: sens,<br>&#39;specificity&#39;: spec, <br>&#39;PPV&#39;: ppv,<br>&#39;NPV&#39;: npv,<br>&#39;AUC&#39;: auc_value,<br>&#39;Mean AUC (CI 5%-95%)&#39; :table7[&#39;Mean AUC (CI 5%-95%)&#39;],                  <br>&#39;F1&#39;: f1       <br>})<br>table9.set_index(&#39;category&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GpmGWKYFQ1M-GjLGjXBw5w.png" /></figure><p>In this article, we have discussed about evaluation metrics that are useful in diagnostic. By running the code throughout each section you should get a more practical sense of each of these metrics. You should be careful in choosing the best metric for some specific problem, in this sense, we hope to have shed light in such a difficult task.</p><p>This is the first episode of a series devoted to the applications of <em>AI</em> in <em>Medical Diagnosis</em>, the story has just begun. In the next one, we’re going to show you how to deal with a dataset and what is <em>Exploratory Data Analysis</em> (<em>EDA</em>). Since we’re going to deal with x-rays images, it will be necessary to introduce a particular kind of neural network, namely the <em>Convolutional Neural Network</em> (<strong>CNN</strong>) architecture.</p><blockquote>Take a look at the full implementation of our code here on <a href="https://github.com/MLJCUnito/Datameron/tree/master/DAIgnosis">Github</a> (and star us if you like our project!)</blockquote><h4>References:</h4><ol><li><a href="https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4975285/pdf/ejifcc-19-203.pdf">Measures of diagnostic accuracy: basic definitions</a>, A. M. Simundic</li></ol><p>2. <a href="https://turi.com/learn/userguide/evaluation/classification.html#classification-metrics">Turi Machine Learning Platform User Guide, Classification metrics</a></p><p>3. <a href="https://www.oreilly.com/library/view/evaluating-machine-learning/9781492048756/">Evaluating Machine Learning Models,</a> A. Zheng, O’Reilly Media Inc.</p><p>4. <a href="https://www.coursera.org/learn/ai-for-medical-diagnosis">AI for Medical Diagnosis</a>, by deeplearning.ai, Coursera, Andrew NG</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c6bca5d53acb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/mljcunito/daignosis-exploring-the-space-of-metrics-c6bca5d53acb">DAIgnosis: Exploring the Space of Metrics</a> was originally published in <a href="https://medium.com/mljcunito">MLJC</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>