<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Amit Gairola on Medium]]></title>
        <description><![CDATA[Stories by Amit Gairola on Medium]]></description>
        <link>https://medium.com/@amitgairola?source=rss-bc9e46c9ed79------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*xj5DKA0x2b4OCykhmnss2A.jpeg</url>
            <title>Stories by Amit Gairola on Medium</title>
            <link>https://medium.com/@amitgairola?source=rss-bc9e46c9ed79------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Mon, 16 Nov 2026 07:34:11 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@amitgairola/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Text Mining: Named Entity Recognition, Parts of Speech and Sentence Structures]]></title>
            <link>https://amitgairola.medium.com/text-mining-named-entity-recognition-parts-of-speech-and-sentence-structures-db46a5008706?source=rss-bc9e46c9ed79------2</link>
            <guid isPermaLink="false">https://medium.com/p/db46a5008706</guid>
            <category><![CDATA[chunking]]></category>
            <category><![CDATA[nltk]]></category>
            <category><![CDATA[parts-of-speech]]></category>
            <category><![CDATA[named-entity-recognition]]></category>
            <category><![CDATA[naturallanguageprocessing]]></category>
            <dc:creator><![CDATA[Amit Gairola]]></dc:creator>
            <pubDate>Sat, 09 Oct 2021 08:39:03 GMT</pubDate>
            <atom:updated>2021-10-09T08:39:03.521Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JcDPTmkDR8atVL2gq0GR7g.jpeg" /></figure><p>This article is in continuation to the previous one, which can be <a href="https://amitgairola.medium.com/introduction-to-text-mining-7-techniques-for-text-analytics-204687f6f05a">read here</a> or <a href="https://arandomvariableai.wordpress.com/2021/10/09/introduction-to-text-mining-7-techniques-for-text-analytics/">here</a>.</p><p>In the earlier article, 7 techniques were explored to extract words out of a document, or even creating a count vector or a term frequency inverse document frequency vector. Here, we will look at Parts of Speech, Named Entity Recognition and how grammar can be used to extract chunks out of a document.</p><p><strong>Parts of Speech</strong></p><p>In any language, Parts of Speech involves categorizing words into similar categories, each category representing a similar grammatical property. For e.g. words can be classified as Nouns, Pronouns, Adverbs, Adjectives, Verbs etc. Using the Spacy library in Python, we can identify the parts of speech quite easily.</p><p>So let’s take the same <a href="https://timesofindia.indiatimes.com/blogs/toi-editorials/missing-skills-low-employability-calls-for-academia-industry-link/">article</a> to identify Parts of Speech. The first paragraph from the document is reproduced below.</p><p><em>“Three IITs, India’s premier higher education institutions (HEIs), figure in the top 200 institutions across the world ranked according to employability of students in the 2022 QS Graduate Employability Rankings. No Indian HEI is in the top 100. In comparison, an HEI each from China and Hong Kong have challenged Anglosphere hegemony in the top 10. That no Indian HEI has been able to breach the top 100 aptly sums up the employability crisis of Indian graduates.”</em></p><p>After parts of speech tagging, this is what the output looks like. Spacy can also provide explanation as to why it attached a specific Parts of Speech tag to the word.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ioBuwgySVBXNnb1uOkMUBA.png" /><figcaption><strong>Parts of Speech Tagging</strong></figcaption></figure><p><strong>Named Entity Resolution</strong></p><p>Named Entities are organizations, people, countries etc which are referenced in a document. In the paragraph, Spacy was able to automatically detect the below mentioned named Entities. It is also able to distinguish between 200 as a number as well as 2022 as Date. GPE is short for Geographical Political Entities and it is able to find India, China, Hongkong!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*CFZSpGG0r5yICvsIm9ukeA.png" /></figure><p>The named entities can be visualized as below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*H30nI6ZBPKNEur_LQqWIuQ.png" /><figcaption><strong>Named Entities</strong></figcaption></figure><p><strong>Sentence Structures</strong></p><p>For any language, grammar is what defines the sentence syntax and the context. For humans, language has evolved over tens of thousands of years and maybe more! Human brain has learnt to understand the sentence grammar and construct sentences as per the grammar. It takes only a few years for a child to learn a language and to be able to communicate fluently.</p><p>Similarly for computers too, grammar has to be defined to understand natural language. For the programming languages such as C or Java or Python there are compilers/interpreters that convert the programming language syntax into machine language instructions.</p><p>To enable the computers to understand sentence structures, it is important to define grammatical rules. Using these rules, computers can understand sentence structures.</p><p>While it is important to extract information such as parts of speech, named entities from text, it is also important to understand sentence structures.</p><p>Consider the sentence “We saw the yellow dog”. The word “we” is a Pronoun, “saw” is a verb, “the” is a determiner or an adverb, “yellow” is an adjective and “dog” is a noun. Without any grammar, these are just a bunch of words without structure.</p><p>However, if we define a grammar rule that segregates noun phrases together (“the yellow dog”), we can create sentences as tree like structures. If our grammar rule is that a Noun Phrase could be constructed as &lt;Adverb&gt;&lt;Adjective&gt;&lt;Noun&gt;, then we can create a group or a chunk out of this sentence and use this to represent a complete sentence as a structure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Zhty7S11afJ1R1Cl.png" /><figcaption><strong>Source: NLTK tutorial</strong></figcaption></figure><p>Let’s consider a sentence — “The little mouse ate the yellow cheese.”.</p><p>Let our grammar be defined as</p><p>Noun Phrase = &lt;Determiner&gt;&lt;Adjective&gt;&lt;Noun&gt;</p><p>Then the above mentioned sentence can be structured as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/919/1*U5zrMdoeOXW_vs6_E95a7A.png" /><figcaption><strong>Sentence Chunking</strong></figcaption></figure><p>The parser has identified two distinct chunks of noun phrase and combined them with a verb to create a sentence.</p><p>Considering another sentence,</p><p>“The black dog snarled at the frightened kitten.”</p><p>Now if we were to apply the same grammar, this would become</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/954/1*4WUmasOzeis09XpnWnBVoA.png" /><figcaption><strong>Another example of Sentence Chunking</strong></figcaption></figure><p>If we were to change the grammar rule as</p><p>Noun Phrase = &lt;Determiner&gt;&lt;Adjective&gt;&lt;Noun&gt;&lt;optional Verb&gt;</p><p>This would interpret the sentence as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IZlsCmm63jtiiIyWv0rPRw.png" /><figcaption><strong>Another example of Chunking</strong></figcaption></figure><p>If we apply the same grammar to say a sentence from the article</p><p>“An employability report in 2019 based on standardised testing by Aspiring Minds termed the challenge as stubborn unemployability”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T9HvtpQBp9JGBdc_gKAwsQ.png" /><figcaption><strong>Sentence structure of a complex sentence from the article</strong></figcaption></figure><p><strong>Chinking</strong> is another option where one would like to define a grammar but remove certain phrases if they exist.</p><p><strong>With these above new concepts, we can identify not only the parts of speech, but also look at Organizations, People, Institutions aka Named Entities in our text. We can also structure sentences and as an example can find out the common noun phrases in a set of tweets which have been classified as positive to understand what drives a positive tweet from a happy customer!</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=db46a5008706" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to Text Mining : 7 techniques for text analytics]]></title>
            <link>https://amitgairola.medium.com/introduction-to-text-mining-7-techniques-for-text-analytics-204687f6f05a?source=rss-bc9e46c9ed79------2</link>
            <guid isPermaLink="false">https://medium.com/p/204687f6f05a</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[bag-of-words]]></category>
            <category><![CDATA[stemming]]></category>
            <category><![CDATA[text-mining]]></category>
            <dc:creator><![CDATA[Amit Gairola]]></dc:creator>
            <pubDate>Sat, 09 Oct 2021 03:54:11 GMT</pubDate>
            <atom:updated>2021-10-09T03:54:11.878Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gcp4Vrgta-b4_jvG1b0yyg.jpeg" /></figure><h3>Introduction to Text Mining : 7 techniques for text analytics</h3><p>Text data is all around us! We see it in news articles, Twitter tweets, Facebook posts and so on. Data is also present in documents (PDF, Word etc) and this could be contractual documents, legal documents, policy documents etc.</p><p>Text Mining or processing text data becomes important as there are actionable insights that could be derived from such data. In the past such data was ignored, but due to emergence of high end low cost computing through technologies such as Big Data, it is now possible to analyze the unstructured data and incorporate this into data science models.</p><p>As an example, an insurance company could include data from policy documents, claim documents, customer’s interaction with call center agents and augment the “Churn” prediction models using this text data.</p><p>Retail customers could include product reviews by customers as part of their pricing strategy or campaign strategy.</p><p>And the list can go on for various such use cases across multiple industries. However it is important to understand that unlike structured data that comes from traditional systems such as Enterprise Resource Planning (ERP) or Customer Relationship Management (CRM) systems, text data could come from non-traditional sources such as social media.</p><p>Another challenge with text data is that while humans can understand the context behind the sentence, it is difficult for a computer to understand the same (atleast the current ones dont!). So there are techniques and methods to extract information out of raw text data so that the computers can understand and intelligently adopt the text data into building data science models.</p><p>In this article, I would like to explain the various techniques through an example. I took <a href="https://timesofindia.indiatimes.com/blogs/toi-editorials/missing-skills-low-employability-calls-for-academia-industry-link/">this</a> article published as an editorial in Times of India as an example to showcase how we can use text mining techniques to analyze this data. I will also post the entire code used for this article on github.</p><p>For Natural Language Processing, there are a few libraries in Python such as NLTK, Spacy. Even the deep learning library such as Tensorflow provides capabilities for Text Processing. Here, we will use NLTK and Spacy for text mining.</p><p><strong>Technique 1: Tokenization</strong></p><p>The first technique is called as Tokenization. Since the entire document or article is composed of sentences (grouped together as paragraphs) and sentences are in turn composed of words, Tokenization is the process of breaking down documents into sentences and then each sentence into words. The first paragraph of the article is</p><p><em>“Three IITs, India’s premier higher education institutions (HEIs), figure in the top 200 institutions across the world ranked according to employability of students in the 2022 QS Graduate Employability Rankings. No Indian HEI is in the top 100. In comparison, an HEI each from China and Hong Kong have challenged Anglosphere hegemony in the top 10. That no Indian HEI has been able to breach the top 100 aptly sums up the employability crisis of Indian graduates.”</em></p><p>This can be broken down into four sentences</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*Pj69W5WcWqYUrKmCs6MMFg.png" /><figcaption><strong>Sentence Breakdown</strong></figcaption></figure><p>Further, each sentence can be broken down into words. The first sentence, for example, can be broken down into the following words.</p><pre>[&#39;Three&#39;, &#39;IITs&#39;, &#39;,&#39;, &#39;India&#39;, &#39;’&#39;, &#39;s&#39;, &#39;premier&#39;, &#39;higher&#39;, &#39;education&#39;, &#39;institutions&#39;, &#39;(&#39;, &#39;HEIs&#39;, &#39;)&#39;, &#39;,&#39;, &#39;figure&#39;, &#39;in&#39;, &#39;the&#39;, &#39;top&#39;, &#39;200&#39;, &#39;institutions&#39;, &#39;across&#39;, &#39;the&#39;, &#39;world&#39;, &#39;ranked&#39;, &#39;according&#39;, &#39;to&#39;, &#39;employability&#39;, &#39;of&#39;, &#39;students&#39;, &#39;in&#39;, &#39;the&#39;, &#39;2022&#39;, &#39;QS&#39;, &#39;Graduate&#39;, &#39;Employability&#39;, &#39;Rankings&#39;, &#39;.&#39;]</pre><p>This has even punctuation marks, commas, brackets, numbers also identified as words. We need to eliminate these before we can progress further. Also, it is important to change the case of the words.</p><p><strong>Technique 2: Lower case and removal of punctuation marks, numbers etc</strong></p><p>We can simply remove the numbers and punctuation marks by checking if each word is alphanumeric or not. After this operation, the list for the above mentioned sentence is</p><pre>&#39;three iits india s premier higher education institutions heis figure in the top institutions across the world ranked according to employability of students in the qs graduate employability rankings&#39;</pre><p><strong>Technique 3: Removal of Stop Words</strong></p><p>Stopwords are commonly used words in any language. These are words such as “a”, “an”, “the”. Usually these dont have a lot of meaning and need to be removed. NLTK and Spacy provide list of stopwords which can be applied to the text to remove such common words and bring out only the relevant words.</p><p>After removing stop words, the list looks like this. It is now evident that the sentence now looks difficult to read though some meaning can still be derived out of it.</p><pre>&#39;three iits india premier higher education institutions heis figure top institutions across world ranked according employability students qs graduate employability rankings&#39;</pre><p><strong>Technique 4: Stemming</strong></p><p>In any language, especially English, words are formed from their stem. For example words — help, helper, helping, helped are all from the same stem — help. More details about Stemming can be found <a href="https://en.wikipedia.org/wiki/Stemming">here on wikipedia.</a></p><p>So to reduce the number of words in a sentence, and to bring the words to their stem, we use stemmers available in NLTK. The most common one is Porter Stemmer.</p><pre>&#39;three iit india premier higher educ institut hei figur top institut across world rank accord employ student qs graduat employ rank&#39;</pre><p>This is how the sentence looks like after applying Porter Stemmer. Education word for example has been reduced to educ. Further duplicate stems can be removed and now the sentence looks like this.</p><pre>&#39;qs institut iit hei student india educ top world figur graduat rank accord higher across three employ premier&#39;</pre><p>This has no meaning from a human perspective, but now the content has been reduced and hopefully this can be used in a machine learning algorithm.</p><p><strong>Technique 5: Lemmatization</strong></p><p>As per wikipedia,</p><p><strong><em>“Lemmatization is the process of grouping together the inflected forms of a word so they can be analysed as a single item, identified by the word’s </em></strong><a href="https://en.wikipedia.org/wiki/Lemma_(morphology)"><strong><em>lemma</em></strong></a><strong><em>, or dictionary form.”</em></strong></p><p>In other words, as an alternative to the Stemming, Lemmatization can be applied to a sentence. Lemmatization derives the stem using the part of speech. It essentially uses the context to derive the stem as compared to Stemming which just finds the stem irrespective of the part of speech the word is used for.</p><p>In NLTK, parts of speech has to be provided otherwise it considers the word as a noun.</p><p>After applying just plain lemmatization, this results into</p><pre>&#39;three iits india premier higher education institution heis figure top institution across world ranked according employability student q graduate employability ranking&#39;</pre><p>Removing duplicate words, this becomes</p><pre>&#39;india top world ranked iits figure ranking according employability education heis q institution three premier student graduate higher across&#39;</pre><p><strong>Technique 6: Count Vectorization</strong></p><p>Count Vectorization is simply counting the number of words in a sentence and creating a vector out of it. During vectorization, the minimum and maximum frequency of words can be defined so that only the relevant words are included in the feature vector. This ensures that neither very frequently occurring words nor very rare words are included in the feature vector.</p><p>If Count Vectorization is applied to the entire text in the article, these are the most frequent words that come out (atleast 5% occurrence and no more than 95% occurrence)</p><pre>&#39;education employability engineers germany graduates hei heis india indian industry japan job level market report role skills students&#39;</pre><p>The vectors look something like this</p><pre>[[0 1 2 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 1]<br> [1 0 0 0 0 0 1 0 0 1 0 0 0 0 0 0 0 0 0]<br> [0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0]<br> [1 0 1 0 0 1 1 0 0 2 0 0 0 0 0 0 0 0 0]<br> [0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0]<br> [0 0 1 1 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0]<br> [0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]<br> [0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0]<br> [0 0 1 0 0 1 0 0 1 0 0 0 1 0 1 1 0 1 0]<br> [0 0 1 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0]<br> [0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]<br> [0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0]<br> [0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]<br> [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1]<br> [0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0]<br> [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0]<br> [0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0]<br> [0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0]<br> [0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0 1 1 0]<br> [0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 1 0 0]<br> [0 0 0 0 0 0 0 1 1 0 0 0 1 0 1 0 0 0 0]]</pre><p>Each row corresponds to the count of words in a sentence for the words given above.</p><p><strong>Technique 7: Term Frequency Inverse Document Frequency Vectorizer</strong></p><p>As an alternative to Count Vectorization which is simply a count of a particular word occurring in a document, Term Frequency Inverse Document Frequency (TF-IDF) simply creates a score for every word, indicating how relevant is that particular word.</p><p><strong>Term Frequency(TF) </strong>is simply the number of times a word occurs in a document. This however has the disadvantage that it favors large documents.</p><p><strong>Inverse Document Frequency(IDF) </strong>is the measure of the information a word provides across the document. IDF is computed as the log of the scaled inverse ratio of the number of documents that contain the word and the total number of documents (if you want a slightly mathematical view !!)</p><p>Now <strong>TF-IDF </strong>is a measure computed by the multiplication of Term Frequency and Inverse Document Frequency.</p><p>Using the TfIdfVectorizer from Scikit-Learn library, we get the following words</p><pre>&#39;employability hei heis india indian level&#39;</pre><p>For the first sentence, the TFIDF vectorization will look like this</p><pre>[0.73872867, 0.        , 0.49663801, 0.45566504, 0.        ,<br>       0.        ]</pre><p><strong>With the above seven techniques, it is easy to see how the text data can be mined to extract features out of data and these can then be used further into building data science models.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=204687f6f05a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An Introduction to Market Basket Analysis]]></title>
            <link>https://amitgairola.medium.com/an-introduction-to-market-basket-analysis-9f4b881bf11?source=rss-bc9e46c9ed79------2</link>
            <guid isPermaLink="false">https://medium.com/p/9f4b881bf11</guid>
            <category><![CDATA[marketing-strategies]]></category>
            <category><![CDATA[market-basket-analysis]]></category>
            <category><![CDATA[association-rule-mining]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[apriori-algorithm]]></category>
            <dc:creator><![CDATA[Amit Gairola]]></dc:creator>
            <pubDate>Wed, 06 Oct 2021 20:53:55 GMT</pubDate>
            <atom:updated>2021-10-06T20:58:34.070Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V3ULcHUrOoxwxEiMLflJfw.jpeg" /></figure><p>Market Basket Analysis is a set of techniques which probably derived its name from the “Shopping Basket”. It is an essential tool for any retailer to understand and identify hidden patterns from the point-of-sale data. While Market Basket Analysis a.k.a MBA originated from the retail industry, it finds widespread application in Banking and Financial Services Industry, Telecom, Insurance and so on.</p><p>Essentially Market Basket Analysis would tell a retail about what two products would be bought together most frequently and this could become a basis for defining a strategy for</p><p>A. Determining store layouts to maximize revenue</p><p>B. Bundling strategies for products</p><p>C. Discounting strategies</p><p>And the list can go on and on…</p><p>Market Basket Analysis analyses the items that are bought as part of a single order and hence analyzes all the orders identifying frequently occurring items.</p><p>For the other industries such as Banking, a marketer can try to identify which products to recommend to a customer or in Telecom, the Telecom provider could identify bundling strategies for services. An insurance provider can look at unusual combinations of claims which could be a sign of fraud.</p><p>An interesting anecdote is that marketing researchers found that on weekends, Beer and Diapers had a good chance of being bought together. While this anecdote was debunked later, but it gives a good understanding of how marketers can identify hidden patterns in data. Beer and Diapers have nothing in common, but it is possible that in families who have newborn babies, the father gets the job of weekend shopping and since diapers are used frequently, they essentially figure in the shopping list and while the father is at the store, the father also picks up a pack of beer.</p><p>Market Basket Analysis uses something called as <strong>Association Rules </strong>which analyze the frequently occurring items in a shoppers cart. The most popular algorithm is called as <strong>Apriori algorithm. </strong>It would come up with rules such as</p><p><em>“If the customer has picked up bread, then there is a good likelihood that the customer would also pickup butter”</em></p><p>What this would mean for a retailer is that they can keep bread and butter together in a store so that the customer would invariably pickup butter along with bread or the retailer can also keep the two products at two extreme corners of the store so that the customer has to walk across the store to pick up the two products. What might happen is that the customer who came for just bread and butter would end up picking up a lot more items, which fancy the eye!</p><p>To understand Association Rules better, it is important to note and understand the three concepts of <strong><em>Support, Confidence and Lift.</em></strong></p><p><strong>Support: </strong>Support is defined as the probability that a particular product or bundle of products will be picked up. Mathematically it is defined as</p><p>Support = Frequency(Product A)/Total Number of Transactions</p><p><strong>Confidence: </strong>Confidence is given by the ratio of frequency of Product A and Product B divided by the frequency of Product A. Mathematically it can also be given as the conditional probability of B given A.</p><p>Confidence = Frequency(A, B)/Frequency(A)</p><p><strong>Lift: </strong>Lift gives the ratio of the support for Product A and B divided by the product of support for A and support for B.</p><p>Lift = Support(A,B)/(Support(A) * Support (B)</p><p>Lift in a way will tell the retailer if it is better to sell Product A and Product B together as compared to selling them independently.</p><p>Let’s take an example and it will clear out the above mentioned concepts</p><p>Transaction 1: Customer bought Milk, Bread and Butter</p><p>Transaction 2: Customer bought Milk, Butter and Eggs</p><p>Transaction 3: Customer bought Bread, Butter and Eggs</p><p>Transaction 4: Customer bought Milk, Eggs and Cheese</p><p>Transaction 5: Customer bought Bread, Butter and Cheese</p><p>Given the above mentioned transactions, we want to calculate support, lift and confidence for the rule which says that if the customer bought milk, the customer is likely to buy eggs!</p><p><strong>Support for Milk and Eggs</strong> = No of times Milk and Eggs occur/Total Transactions = 2/5 {Transaction 2 and 4}</p><p><strong>Confidence for Eggs given Milk</strong> = Frequency(Milk, Eggs)/Frequency (Milk) = 2/3 { T2 and T4 have Milk and eggs, Milk occurs in T1, T2 and T4) = 2/3</p><p><strong>Lift for Eggs and Milk</strong> = Support for Eggs and Milk / (Support of Milk * Support of Eggs) = 2/5/(3/5*3/5)= 10/9</p><p>Now let’s try to understand how the rules are formed. For this we will consider again 5 transactions, each having a atleast 2 products.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/454/1*9p0ZeA7Ul-AaMlrm0AtCxQ.png" /><figcaption><strong>List of Orders/Transactions</strong></figcaption></figure><p>We will try to create <strong>frequent itemsets</strong> which are nothing but grouping of products/items together and we will compute support for each such combination. We will reject any rule which has support below 40% (less than 2/5). This is also called the <strong>Threshold Criteria for Support.</strong></p><p>So first we will create a table of individual products and compute support for each product individually.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/455/1*eAUWsVOHrz6EK3H11tTMMQ.png" /><figcaption><strong>Frequent Itemset considering single Item</strong></figcaption></figure><p>We see that Product D has only 20% support and hence we remove product D. Now only A, B, C and E are remaining. We will create group of two products taking all possible combination from A, B, C, E.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/454/1*1e4B67AZkgQokcXfRXJQrQ.png" /><figcaption><strong>Frequent Itemset considering two items</strong></figcaption></figure><p>We can eliminate A, B given that support is below 40%.</p><p>Repeating the process for three items, we get the following</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/454/1*LIQjt1snn-XsOYsyjAs0Hw.png" /><figcaption><strong>Frequent Itemset containing three items</strong></figcaption></figure><p>After applying the same rules, we are left with two frequent itemsets {A, C, E} and {B, C, E}. We can go upto four products {A, B, C, E} but the support for that is just 1/5. So our process can stop here.</p><p>Since we had just five transactions, it was easy to compute support. However, consider hundreds of millions of transactions! Computing support becomes computationally challenging exercise and we can use a rule called <strong>“Pruning”. </strong>Pruning helps in eliminating groups of items. For e.g. in the above mentioned table {A,B,C} and {A,B,E} can be easily removed because the subset {A, B} was already eliminated earlier. If a subset of a frequent itemset does not meet the threshold for support, then the frequent itemset will also not meet. Logically a subset should have a better support than the parent set. If it is not the case, then the parent set would also not meet the threshold criteria.</p><p>Now that we have two frequent itemsets {A, C, E} and {B, C, E}, we can create now Association Rules.</p><p>To do this, we need to create all possible subsets of {A, C, E} and {B, C, E}. Let’s consider the first set which is {A, C, E}</p><p>Subsets of {A, C, E} are {A, C, E}, {A, C}, {A, E}, {C, E}, {A}, {C}, {E}.</p><p>So let’s say that we want to measure how effective the rule {A,C -&gt; E}. Which means that if the customer has bought products A and C, then how likely is the customer going to buy product E.</p><p>To calculate the effectiveness, we again define a new <strong>Threshold Criteria for Confidence</strong>, saying that we will reject this rule if the confidence on the rule is less than or equal to 50%.</p><p>Confidence (A, C-&gt;E) = Support(A, C, E)/Support(A,C) = (2/5)/(3/5) = 66.67%. Hence we will accept the rule.</p><p>Confidence of (E -&gt; A,C) = Support (A,C,E)/Support(E) = (2/5)/(4/5) = 50%. We will reject this rule, since this does not meet the Confidence Threshold criteria.</p><p>We can form such rules for all combinations and then figure out which rules give the most confidence and hence can be considered.</p><p>Market Basket Analysis essentially generates such rules and allows the marketer to look at these rules and derive insights. By themselves, the rules mean nothing. Most of the time, the rules could be either “<strong>Actionable</strong>” meaning that some action can be taken based on the rule, or “<strong>Trivial</strong>” meaning that it is a known fact and there is nothing much to do or there could be “<strong>Inexplicable</strong>” rules which make no sense. Hence it is upto the marketer to identify the right set of rules and incorporate them into marketing interventions such as creating customized campaigns etc.</p><p><strong>Challenges with Big Data</strong></p><p>It is likely that a retail store is selling tens of thousands of products/Stock Keeping Units and hence it is computationally difficult to identify such frequent itemsets from such ten thousand products or SKUs and considering billions of transactions. In such cases, association rules can be created at category level. This can also give good insights into associations between different categories and marketing interventions can be designed around such association rules.</p><p><strong>References:</strong></p><p>Data Mining Techniques for Marketing, Sales and Customer Relationship Management by Gordon S Linoff and Michael J.A. Berry</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f4b881bf11" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A structured approach to solving data science problems!]]></title>
            <link>https://amitgairola.medium.com/a-structured-approach-to-solving-data-science-problems-3ca3df3cc593?source=rss-bc9e46c9ed79------2</link>
            <guid isPermaLink="false">https://medium.com/p/3ca3df3cc593</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[apc]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[feature-engineering]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Amit Gairola]]></dc:creator>
            <pubDate>Sun, 03 Oct 2021 18:40:06 GMT</pubDate>
            <atom:updated>2021-10-03T18:50:47.883Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*t7lx34vYoJ8lAeNU6un7_g.jpeg" /></figure><p>While working on numerous data science projects, I have seen that that most data scientists adopt a haphazard approach when they work on a data science problem. While it is understandable that Data Science is both art as well as science, it is important to have some method to the madness. Most people I see just take the data and start throwing algorithms, hoping that they would achieve success through brute force. In most cases, this does not result into a favorable outcome. The business users also get frustrated and then it would appear that data science is nothing more than a fad.</p><p>With experience, I have tried to come up with a 16 step approach (could be more, could be less) to apply a structured approach to solving a data science problem.</p><p>I would try to give an overview of what the approach is and yes, nothing is written in stone. The important thing is that this approach can be adopted, modified and tailored to the need of the problem at hand.</p><p><strong>Step 1: Formulate the business problem</strong></p><p>It is extremely important that the data scientist formulates the business problem identifying the primary objective! This primary objective could be as simple as — I want to forecast sales for the next 6 months or it could be as complex as finding out reasons why employees leave an organization. However, it is very important that the data scientist is clear about what he/she is going to solve.</p><p>Also important is to clearly define and agree with business users on how would the success of the solution be measured. It can be either qualitative or in most cases it is quantitative. An example would be — Accuracy of the weekly sales forecast should be atleast 80% averaged over an 8 week period. Other examples could be that we would like to identify the primary reasons due to which there is employee attrition or customer churn. While the first example is a quantitative measure, the second is a qualitative measure and should be address by formulating appropriate hypothesis.</p><p>It is extremely important to have a sign-off from business users to avoid any issues further.</p><p><strong>Step 2: Identify the right infrastructure, the libraries that are going to be used</strong></p><p>It is important to identify the right infrastructure as this becomes crucial in not only achieving the outcomes but also ensuring that the machine learning model learns in a reasonable amount of time. Choosing the right infrastructure — as an example a Sagemaker Environment on AWS or Azure Machine Learning or Google Cloud could be chosen and infrastructure planned accordingly based on the compute and storage required.</p><p>It is also important to identify the libraries and the version that is going to be used for the project. If the project is going to be built using Tensorflow or Scikit Learn on Python, then it is important to identify the right version of libraries being used.</p><p><strong>Step 3: Identify the right datasets and define the dependent and independent variables</strong></p><p>While we can fancy about solving the most complex problems using data science, the core of the solution lies in the data. Most data scientists would spend atleast 70% of their time working on data. So it is important to identify the right datasets or data sources that need to be worked on.</p><p>It is also important to identify how do you identify the “target” variable. In Data Science, target variable is something that you are trying to predict. Hence it is important to identify the variable/column in the dataset that is going to be defined as the target. The rest of the columns/features become independent variables which will be used to predict the “target variable”.</p><p><strong>Step 4: Identify the subset of data that is going to be used for training of the model</strong></p><p>It is important to identify the data that will be used to train the data science model. As an example, all data till 30-Aug-2021 (as an example) will be used for training the model and the data after that will be used for testing the model accuracy. Usually the dataset is partitioned into training dataset and testing dataset and this could be done through either Random Sampling or Stratified Sampling. Other techniques such as K-fold cross validation or Leave One Out Cross validation can also be used to define how is the model trained.</p><p><strong>Step 5: Data Preparation, Cleaning</strong></p><p>Data Preparation and cleaning is a mandatory step that requires preparing the data for the model to learn. Computers can only understand numerical data. So it is important to do data conversions such as One Hot Encoding of categorical data (as an example if Gender column has Male, Female and Others, then this data is one hot encoded to convert to numerical values). Other example is handling of missing values. It is important to define a strategy for missing values as the data scientist is making a best guess estimate of the missing value. Median or Mode are used for Numerical and Categorical values respectively.</p><p><strong>Step 6: Exploratory Data Analysis</strong></p><p>While Data Science talks about algorithms to solve problems, nothing beats a good exploratory data analysis. An Exploratory Data Analysis or EDA involves visual inspection of data through tools such as Heatmaps for correlation analysis, Bar charts and Scatter plots, Frequency charts or Histograms, Box Plots (Outlier detection) and so on.</p><p>EDA gives important insights into relationships that exist within data and is key to feature engineering.</p><p><strong>Step 7: Scaling for numerical data</strong></p><p>Most data science algorithms are sensitive to the scale of the data. For example, if the dataset has two columns — height in centimeters and weight in kilograms, these are on different scales and these need to be brought to the same scale. Most commonly, StandardScaler is used for scaling. This computes a z-value (number of standard deviation times the data point is away from the mean). It is computed as (x — mean)/sigma, where is the data point.</p><p>Other types of scaling include MinMaxScaling and Robust Scaling.</p><p><strong>Step 8: Categorical feature handling</strong></p><p>Categorical features are class values (Gender — Male and Female, Income group — High and Low). Computers do not understand such variables and One Hot Encoding is required. In R, one hot encoding is done by default by most algorithms.</p><p><strong>Step 9: Multi-collinearity and Curse of Dimensionality</strong></p><p>Multi-collinearity arises when independent variables/columns/features are not truly independent. This essentially means that there is inherent relationship between two columns which are assumed to be independent.</p><p>Also, there could be a case where in after all the steps above, the number of features/dimensions increase dramatically. As an example, initially the data scientist started with 10–15 variables and then after all the Feature Engineering and data pre-processing this results in say more than 100 variables. Hence it becomes very important to reduce the number of dimensions.</p><p>Typically this is done through either Principal Component Analysis (Unsupervised Learning) or Linear Discriminant Analysis (LDA).</p><p>This not only removes multicollinearity as well as it also simplifies the processing of data by the algorithm.</p><p><strong>Step 10: Train the model using an algorithm</strong></p><p>This step involves using a Python or R or the programming language library to train the algorithm.</p><p><strong>Step 11: Measure the outcome on the Training Set</strong></p><p>The data scientist should measure the prediction accuracy on the training set. How well the model is able to learn the patterns from the data is determined through this step. This usually uses the metric as defined in <strong>Step 1.</strong></p><p><strong>Step 12: Predicting the outcome using the Testing Set</strong></p><p>The data scientist would use the Testing set to predict the outcomes. The data scientist would also measure the accuracy or use the metric as defined in Step 1 for the success of the project.</p><p><strong>Step 13: Overfitting?</strong></p><p>If the accuracy on the Training set is high (say 95%) and the accuracy on the testing set is poor (say 60%), then it is possible that the model is overfitting (it is not able to generalize and is failing when it encounters unseen data from the testing set).</p><p>In such cases it is important to look at</p><p>a) How the training set and testing set were formed or if cross validation is necessary</p><p>b) Use L1/L2 regularization to reduce overfitting.</p><p><strong>Step 14: Explore other algorithms</strong></p><p>The data scientist can also explore other algorithms of similar nature to look at improving efficiency and accuracy of the model. If the algorithm in Step 10 was say Logistic Regression, then the data scientist can also look at other algorithms such as Decision Tree, Gaussian Naive Bayes, Support Vector Machines etc.</p><p>If the data scientist chooses to explore other algorithms, then Steps 10 to 13 need to be repeated for every algorithm considered.</p><p><strong>Step 15: Consider Ensemble techniques</strong></p><p>The data scientist can also consider ensemble techniques such as Bagging, Boosting or Stacking to improve accuracy of the model. Random Forest (Bagging), Adaboost, Gradient Boosting Machines, XGBoost (Boosting) can also be considered.</p><p><strong>Step 16: Explainability and Visual Output</strong></p><p>Modern times require data scientists to be able to explain their models. Regulations such as GDPR and others grant the right to a person to ask for explanation of an outcome that was determined by a Data Science Model. For example if the model is predicting the likelihood of a loan application defaulting and hence recommending rejection of the loan application, the new laws grant the right to the loan applicant to question the bank on why the loan application was rejected and bank by law is required to provide an explanation.</p><p>Hence explainability of the model becomes very important and it is an arduous exercise to ensure that the model is explainable.</p><p>Most business users do not usually understand the statistical explanations and hence it is important that the outcome of the model is explained in an intuitive manner. This involves building dashboards with intuitive visuals to explain the model and the outcomes. Usually these are developed in specialized tools such as Power BI, Tableau or Qlik. Otherwise open source libraries such as Plotly can also be used to develop such dashboards.</p><p><strong>Note: The above mentioned steps are an attempt to bring a structure to solving a data science problem. And as mentioned earlier, this is not etched in stone. Building a data science model is a very complex exercise and the above mentioned steps are an attempt to put an structured approach to this complex exercise.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3ca3df3cc593" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Time Series Analysis to predict NIFTY50 movements]]></title>
            <link>https://amitgairola.medium.com/using-time-series-analysis-to-predict-nifty50-movements-aa544b5b96fd?source=rss-bc9e46c9ed79------2</link>
            <guid isPermaLink="false">https://medium.com/p/aa544b5b96fd</guid>
            <category><![CDATA[facebook-prophet]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[forecasting]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Amit Gairola]]></dc:creator>
            <pubDate>Sat, 29 May 2021 06:34:55 GMT</pubDate>
            <atom:updated>2021-05-29T06:34:55.829Z</atom:updated>
            <content:encoded><![CDATA[<h3>1. Introduction</h3><p>The NIFTY 50 index is National Stock Exchange of India’s benchmark broad based stock market index for the Indian equity market. Full form of NIFTY is National Stock Exchange Fifty. It represents the weighted average of 50 Indian company stocks in 12 sectors and is one of the two main stock indices used in India, the other being the BSE Sensex. For more details <a href="https://www.blogger.com/blog/post/edit/2703491178401334903/2248411462083565416#">Click here</a></p><p>In this blog, we will see how we can use the various Time Series algorithms to predict how the NIFTY50 index will move over the next 30 days.</p><h3>2. Download Data</h3><p>To download the data, we can go to the Yahoo Finance site and download the historical data for the NIFTY50 index. The URL for downloading the data is given <a href="https://www.blogger.com/blog/post/edit/2703491178401334903/2248411462083565416#">here</a><br>Once the data is downloaded, we will use the Jupyter Notebook for reading the data and applying the algorithms. For this, a Python environment (preferably Anaconda) or Google Colab is required. Here, we will not go through Anaconda installation or the usage of Google Colab.</p><h3>3. Processing the Data</h3><p>First let us begin with importing the various libraries that are required.</p><h3>3.1 Basic Data Processing Libraries</h3><p>In Python, the following two libraries are a must.</p><p>Numpy</p><p>Pandas</p><p>The pandas library is used for managing the data in a dataframe. A dataframe is nothing but a table with rows and columns stored in the computer’s memory (RAM). Numpy library is used for mathematical operations such as taking logarithms, averaging etc.</p><p>Matplotlib</p><p>Seaborn</p><p>Matplotlib and Seaborn libraries are used for plotting the data. These are excellent visualization libraries that are needed to visually analyze the data.<br>Now with the basic libraries imported, we can start importing the various time series libraries</p><p>Statsmodels</p><p>The statsmodels libraries provide various functions and APIs to build time series models. It also provides many utility functions that help in the analysis of Time Series data.</p><p>Pmdarima<br>Facebook Prophet</p><p>We also need to import pmdarima and Facebook Prophet. These provide capabilities to run AUTO ARIMA and Time Series models. Finally we will import the various metrics that are required to measure the accuracy of the time series models.</p><p>Scikit-learn for Metrics</p><p>We will also write a custom function that will compute Mean Absolute Percentage Error.</p><h3>3.2 Loading the data and handling missing values</h3><p>Once we have imported the various libraries, we need to load the data using the Pandas library.<br>The file name is NSEI.csv (downloaded from the URL given above from Yahoo Finance website).</p><p>data = pd.read_csv(“data/NSEI.csv”)</p><p>The above command loads the data into a dataframe. On analyzing the data frame, this is how it appears.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/511/0*unXUolVXHkA2TtAl.png" /></figure><p>The dataframe has multiple columns. The important ones are ‘Date’ and ‘Adj Close’. The adjusted close column adjusts the closing value by considering the dividends, swaps etc.</p><p>When we analyze the dataset, we see that there are a few records that dont have any value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/409/0*_uSeMhz1PEnGweJA.png" /></figure><p>We also see that there are no records for weekends and non trading days. So either we can drop off these records (which will cause a discontinuity) and may cause issues, or we can use the values from the previous day. This works well for the weekends as there is no trading on weekends. For the five records above, we can safely use the values from the previous date. As an example, for the date 01-Jan-2018, we will take value from 31-Dec-2017.</p><p>To do this, we first convert the “Date” column to a date value. Then we set the Date column as the index for this data frame and do a forward fill of the records. This can be done easily in Pandas as</p><p># Set the column type for Date as Datetime</p><p>data[‘Date’] = pd.to_datetime(data[‘Date’],format=’%Y-%m-%d’)</p><p>data[‘Adj Close’] = data[‘Adj Close’].fillna(method=’bfill’)data.set_index(“Date”,inplace=True)</p><p>data = data.resample(“D”).ffill().reset_index() # resample Days and fill values</p><h3>3.3 Plotting the data</h3><p>Now we actually start plotting the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/0*yJNf3ar92gpv1oKh.png" /></figure><p>We can see that the plot shows an upward trend (except in Mar’20/Apr’20) because of the impact of Covid.<br>We also try to look at how the trend appears by 7-day simple moving averages and 30-day Simple Moving Averages.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/606/0*TioVYkqWWNk1mlh4.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/629/0*3k49nxuJ9kYZoEA3.png" /></figure><p>From the charts given above, it is clear that the process is not Stationary. A stationary process is one which has</p><ul><li>Constant Mean across the time period</li><li>Constant Variance</li><li>The covariance of the m-th term and the (m + n) th term should not be a function of time.</li></ul><p>Clearly there is an upward trend, which means that the Mean is not constant and hence can be concluded that this is a non stationary process.</p><h3>3.4 Decomposing Time Series</h3><p>A time series can be decomposed into its Trend, Seasonal and Residual components. Statsmodels provides a simple function “seasonal_decompose” to decompose the time series into its components.</p><h4>Trend</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*evaasKSJArtH0wwf.png" /></figure><h4>Seasonal</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/638/0*uyWRCKW0PqO3CpEt.png" /></figure><h4>Residual</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*d1UA6idqqRSXtrMk.png" /></figure><h3>3.5 Checking for Stationarity</h3><p>To check for stationarity, Augmented Dickey Fuller Test is one of the key tests. This prints the p value. IF p value is above 0.05, we accept the null hypothesis that the process is not stationary and reject the alternate hypothesis.</p><p>print(“Dickey Fuller Test :”+ str(sm.tsa.stattools.adfuller(data[‘Adj Close’])[1]))</p><p>The outcome is</p><pre>Dickey Fuller Test :0.7892836265784806</pre><p>This is way above 0.05 which says that the process is non-stationary (as confirmed by visual inspection earlier)</p><p>One of the easiest ways to make the time series stationary is to take a difference from the previous value.</p><p>df_ts = data[‘Adj Close’] — data[‘Adj Close’].shift(1)print(“Dickey Fuller Test :”+</p><p>str(sm.tsa.stattools.adfuller(df_ts[1:])[1]))</p><p>The outcome is</p><pre>Dickey Fuller Test :1.049447416803228e-10</pre><p>Since the p-value is less than 0.05, we can now consider that the time series is stationary.<br>Now when we plot the Time Series (after differencing) and the ACF and PACF plots, this is what we see…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/628/0*9Suu08N_t7cECOmJ.png" /></figure><h3>4. Testing Accuracy</h3><p>To test the accuracy of the model, we will use a measure — Mean Absolute Percentage Error. This measures the error in prediction as a percentage of the actual value.<br>Wikipedia gives this as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/228/0*UG462h39EyHlIq8G.png" /></figure><p>Since the python libraries dont have a readymade function for MAPE, we will have to write it which is not such a big task.</p><h3>5. Building Time Series Models</h3><p>Now that we have the building blocks ready, we will start by first defining what should be our training data set and our testing data set. The usual random splitting does not work here as the time is a very important factor and order of the data is important. We will simply consider the data till 30-April-2021 as the training data and the May data as the testing data. Further, we will create a forecast for the next 7 days.<br>df_train = data[[‘Date’,’Adj Close’]][data[‘Date’] &lt;= pd.to_datetime(‘2021–04–30’)]df_test = data[[‘Date’,’Adj Close’]][data[‘Date’] &gt; pd.to_datetime(‘2021–04–30’)]</p><h3>5.1 Building an Auto Regressive Integrated Moving Average Model</h3><p>Auto Regressive Integrated Moving Average Models (ARIMA) are one of the most advanced econometric models that are used world wide for Time Series analysis. ARIMA models incorporate the Auto Regressive (AR) and the Moving Average (MA) processes along with differential as a means to achieving stationarity. I will not go into these models in detail as this demands a separate post by itself. However, as a practitioner, we leave the implementation of the model to Python library PMDARIMA which provides a very easy to use method “auto_arima”. ARIMA model has three basic parameters</p><ul><li>p — measuring the parameter of the AR process (AR(p))</li><li>d — measuring the number of differential required for achieving stationarity</li><li>q — measuring the parameter for the MA process (MA(q))</li></ul><p>On running the ARIMA model with seasonality as True, we get almost 99.5% accuracy which is evident from the following graph. Orange denotes the actual and Blue denotes the predicted. This is the in sample accuracy (which is basically how the model is able to predict the actual values on the training sample).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*w5eb3kvX56PaI5cV.png" /></figure><p>The out of sample predictions look the following with accuracy as 99.2%. This is also not very bad.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*TTxinOOg2mL8OD_E.png" /></figure><h3>5.2 Using Facebook Prophet to build a Time Series Model</h3><p><a href="https://www.blogger.com/blog/post/edit/2703491178401334903/2248411462083565416#">Facebook Prophet</a> is a time series forecasting model developed by Facebook research. As per the Facebook Prophet page<br>“<em>Prophet is a procedure for forecasting time series data based on an additive model where non-linear trends are fit with yearly, weekly, and daily seasonality, plus holiday effects. It works best with time series that have strong seasonal effects and several seasons of historical data. Prophet is robust to missing data and shifts in the trend, and typically handles outliers well.”</em></p><p>I will not go into details of Facebook Prophet as that is another post by itself. Our insample accuracy given by Facebook Prophet is almost 98.1%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*-DH6PcSAb_1lJ54q.png" /></figure><p>On the Test Set, the accuracy achieved for the Facebook Prophet model is 95.5%.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*q3Ma4VjmBUyNst9w.png" /></figure><p>So when we compare Facebook Prophet to ARIMA models, it is not so accurate. ARIMA model is able to do a lot better.</p><h3>5.3 Using Holt Winters Model to forecast</h3><p><a href="https://www.blogger.com/blog/post/edit/2703491178401334903/2248411462083565416#">Holt Winters</a> is an exponential smoothing model that uses historical data to train the model. Again, not going into the details, Holt Winters model gives a very high accuracy of 99.3% on the training data set. This is also evident from the graph given below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*RjgNsHRK2b2cTkZs.png" /></figure><p>The out of sample or test data set accuracy is around 98.3% which is evident from the graph given below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/0*ukONm3xFGA3xc8so.png" /></figure><p>So far, the ARIMA model has shown great accuracy in terms of predicting the NIFTY50 movements.</p><h3>6. Conclusion</h3><p>ARIMA models have given a very good accuracy on the NIFTY50 data and while on this dataset, Holt Winters and Facebook Prophet have been lagging, there are situations when these models can perform better. It is always better to evaluate and try multiple models.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=aa544b5b96fd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exploring Cancer Data]]></title>
            <link>https://medium.com/analytics-vidhya/exploring-cancer-data-d57f2d72975c?source=rss-bc9e46c9ed79------2</link>
            <guid isPermaLink="false">https://medium.com/p/d57f2d72975c</guid>
            <category><![CDATA[cancer-analysis]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Amit Gairola]]></dc:creator>
            <pubDate>Sun, 05 Apr 2020 10:08:39 GMT</pubDate>
            <atom:updated>2020-04-09T16:09:28.042Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jfAkNIfGRX5-t5MQ" /><figcaption>Photo by <a href="https://unsplash.com/@nci?utm_source=medium&amp;utm_medium=referral">National Cancer Institute</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Breast cancer is a very serious cancer and is only second to lung cancer for fatality.</p><p>As part of this post, I would like to use Machine Learning Algorithms to explore how can a statistical model learn from data and make predictions. This is a <a href="https://en.wikipedia.org/wiki/Supervised_learning">supervised learning</a> example where in we already have the response variable identified.</p><p>This data set has been taken from the <a href="https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+%28Diagnostic%29">University of California Irvine Machine Learning library</a>.</p><p>While a medical researcher would definitely look at this data set and problem from a medical domain perspective, my intention is to look at application of data science and demonstrate how a machine can learn and predict looking at past historical data. This exercise lacks some of the activities such as <a href="https://en.wikipedia.org/wiki/Feature_engineering">feature engineering</a> which could enrich the model.</p><p>I have used Python as the language and used <a href="https://scikit-learn.org/stable/index.html">Scikit Learn</a>, <a href="https://xgboost.readthedocs.io/en/latest/">XGBoost</a> and <a href="https://keras.io/">Keras</a> libraries to build the statistical models. The same can also be achieved in R and SAS and SPSS and whole lot of other Machine Learning tools.</p><h3>Importing the Required Libraries</h3><pre># Import Basic libraries for Data-frames and manipulation<br>import pandas as pd<br>import numpy as np</pre><pre># Import libraries for visualization<br>import matplotlib.pyplot as plt<br>import seaborn as sns<br>%matplotlib inline</pre><pre># Import libraries for transformation<br>from sklearn.decomposition import PCA<br>from sklearn.impute import SimpleImputer</pre><pre># Import Statistical Modelling libraries<br>from sklearn.linear_model import LogisticRegressionCV<br>from sklearn.model_selection import train_test_split<br>from sklearn.tree import DecisionTreeClassifier<br>from sklearn.ensemble import RandomForestClassifier, <br>from sklearn.ensemble import GradientBoostingClassifier<br>from sklearn.svm import SVC<br>import xgboost as xgb</pre><pre># Import Deep Learning libraries<br>from keras.models import Sequential<br>from keras.layers import Dense<br>from keras.wrappers.scikit_learn import KerasClassifier</pre><pre># Import library to compute performance metrics<br>from sklearn.metrics import accuracy_score, precision_score, recall_score, confusion_matrix</pre><pre>import warnings<br>warnings.filterwarnings(&quot;ignore&quot;)</pre><p>Pandas and Numpy are the standard Dataframe and numerical procedures library. Matplotlib and Seaborn are meant for visualizations. Various classes from the Scikit Learn library have been imported (Linear Model, Decision Tree, Random Forest etc). In addition eXtreme Gradient BOOSTing (XGBOOST) has also been imported.</p><h3>Importing the Data set</h3><p>The data set has been provided by the University of Wisconsin Irvine ML library. Pandas is used to import the data into a data frame. A data frame is a simple (row, columns) data structure which is loaded in memory.</p><pre># Define the column names<br>column_names = [&#39;ID&#39;,&#39;Clump Thickness&#39;,&#39;Uniformity of Cell Size&#39;,&#39;Uniformity of Cell Shape&#39;,&#39;Marginal Adhesion&#39;,&#39;Single Epithelial Cell Size&#39;,&#39;Bare Nuclei&#39;,&#39;Bland Chromatin&#39;,&#39;Normal Nucleoli&#39;,&#39;Mitoses&#39;,&#39;Class&#39;]<br># Import the data<br>data = pd.read_csv(&quot;breast-cancer-wisconsin.data&quot;, header=None, names=column_names)<br># Display the top 5 rows<br>data.head()</pre><p>Since the data file does not have the headers for columns, we can define the column names when importing the data set. The data.head() by default gives top 5 rows of the imported data.</p><p>The output of the data import is shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GO4DStiC7L2rGYh8" /></figure><p>Here the dependent variable or response variable is <strong>Class </strong>which has values</p><p>2 — Benign (Harmless) [66% of data]</p><p>4 — Malignant [34% of data]</p><p>Rest all variables except ID are assumed to be continuous variables with values ranging from 1–10.</p><h3>Data Transformation</h3><p>Data Transformation is a very important step in the Data Science process as without this step, it is near impossible to build a <strong>meaningful </strong>statistical model. Before we move to the Model building process, we need to look at missing values or if we need to transform any columns. On observing the data, we find that the column ‘Bare Nuclei’ has a missing value (identified by a ‘?’). We can look at replacing this with either a Mean or a Median strategy. I have decided to take median as my preferred strategy.</p><pre># ID column is converted to String<br>data[&#39;ID&#39;] = data[&#39;ID&#39;].astype(str)</pre><pre># Replace the ? with NaN value<br>data.replace(to_replace=[&#39;?&#39;],value=pd.np.NaN, inplace=True)</pre><pre># Use the SimpleImputer to replace NaN with median<br>imputer = SimpleImputer(strategy=&#39;median&#39;, missing_values=pd.np.NaN)<br>data[&#39;Bare Nuclei&#39;] = imputer.fit_transform(data[&#39;Bare Nuclei&#39;].values.reshape(-1,1))</pre><pre># Create a new column - Malignant <br># 0- cancer is benign<br># 1- cancer is malignant<br>data[&#39;Malignant&#39;] = data[&#39;Class&#39;].map({2:0,4:1})</pre><pre># Drop the column &#39;Class&#39; as this is no longer required.<br>data.drop([&#39;Class&#39;], axis=1, inplace=True)</pre><h3>Data Visualization</h3><p>To look at the data elements in the data set and how they are correlated, the best plot is <a href="https://en.wikipedia.org/wiki/Heat_map">Heatmap</a>.</p><p>We use Seaborn library to plot the heatmap.</p><pre>plt.figure(figsize=(10,5))<br>sns.heatmap(data.corr(),annot=True, cmap = &quot;YlGnBu&quot;)</pre><p>The heatmap is shown below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/775/0*tI8tywl-U_GwQ_P_" /></figure><p>From the heatmap, we can see that apart from Mitoses, every other attribute has a high positive <a href="https://en.wikipedia.org/wiki/Correlation_and_dependence">correlation</a> with Malignant. In addition, from the heat map it is clear that individual attributes also have a very high correlation among themselves (Unformity of Cell Size and Uniformity of Cell Shape as an example).</p><p>This indicates that there is multicollinearity present we need to perform a <a href="https://en.wikipedia.org/wiki/Principal_component_analysis">Principal Component Analysis</a> to remove this.</p><h3><a href="https://en.wikipedia.org/wiki/Principal_component_analysis">Principal Component Analysis</a></h3><p>Principal Component Analysis is a standard method to remove multicollinearity and remove dimensions (whenever there are a large number of dimensions in the model). Here, we have used PCA to create 5 Principal Components.</p><pre>X = data.drop([&#39;Malignant&#39;,&#39;ID&#39;],axis=1)<br>y = data[[&#39;Malignant&#39;]]<br>pca = PCA(n_components = 5)<br>pca.fit(X)<br>X = pca.transform(X)</pre><p>The above code creates 5 principal components from the 9 variables.</p><p>Post Principal Component Analysis, the heat map of the 5 principal components looks like</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/581/0*ZbVY6FhXXVcB9Luu" /></figure><p>It is evident that there is no correlation among the various principal components and multicollinearity has been removed from our data set.</p><h3>Model Development</h3><p>Now we come to the interesting part of developing the statistical models.</p><p>To do that, first we have to split the data set into training and testing. While the model is trained on the training data set, the performance of the statistical model is tested using the testing data set.</p><p>We use the train_test_split function provided by scikit learn library to split the data into training and testing. 75% of the data is used for training and 25% is used for testing.</p><pre>np.random.seed(100)<br>X_train, X_test, y_train, y_test = train_test_split(X,<br>                                                    y,<br>                                                    test_size=0.25, <br>                                                    random_state=100)</pre><p>We set the seed to 100 to ensure the outcome is repeatable. The ‘test_size’ parameter determines the size of the testing data set (25% in our case).</p><p>Performance of a statistical model is determined by the <a href="https://developers.google.com/machine-learning/crash-course/classification/accuracy">accuracy score</a>, <a href="https://developers.google.com/machine-learning/crash-course/classification/precision-and-recall">precision and recall</a> and <a href="https://en.wikipedia.org/wiki/F1_score">F1 Score</a>. The confusion matrix determines how the actual and predicted classifications are made by the model.</p><p>Now to measure the outcome of the various models, we define a user defined function for printing the metrics score.</p><pre># Function to print the accuracy, precision, recall and confusion matrix<br># Inputs - predicted value (y_pred), actual value (y_true)<br># Output - prints the various metrics in a % form<br>def printmetrics(y_pred, y_true):<br>    acc = np.round(accuracy_score(y_pred=y_pred, y_true=y_true)*100,0)<br>    prec = np.round(precision_score(y_pred=y_pred, y_true=y_true)*100,0)<br>    recall = np.round(recall_score(y_pred=y_pred,y_true=y_true)*100,0)<br>    f1score = np.round(f1_score(y_pred=y_pred, y_true=y_true)*100,0)<br>    print(&quot;Accuracy Score &quot;+ str(acc)+&quot;%&quot;)<br>    print(&quot;Precision Score &quot;+ str(prec)+&quot;%&quot;)<br>    print(&quot;Recall Score &quot;+ str(recall)+&quot;%&quot;)<br>    print(&quot;F1 Score &quot;+ str(f1score)+&quot;%&quot;)<br>    print(&quot;*** Confusion Matrix ***&quot;)<br>    print(confusion_matrix(y_pred=y_pred, y_true=y_true))</pre><h3><a href="https://en.wikipedia.org/wiki/Logistic_regression">Logistic Regression Model with Cross Validation</a></h3><p>Logistic Regression is the first model that we will try to fit onto this data (after PCA). Logistic Regression implements the Sigmoid function to fit a linear model. The sigmoid function is given as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/177/0*z23izabXBb2dfP5s" /></figure><p>Where z = <em>b + w1x1 + w2x2 + … wNxN</em></p><p>The plot for a sigmoid function is given as</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/456/0*H3Io-v8unFuFlMgT" /></figure><p>Source: <a href="https://developers.google.com/machine-learning/crash-course/logistic-regression/calculating-a-probability">Google Class on Machine Learning</a></p><p><a href="https://en.wikipedia.org/wiki/Cross-validation_(statistics)">Cross Validation</a> is a technique in which the data is sampled to create k-folds. The model trains on k-1 folds and is tested on the 1 fold. This particularly useful to improve the accuracy of the model.</p><pre>logmodel = LogisticRegressionCV(cv=5)<br>logmodel.fit(X_train, y_train)<br>pred = logmodel.predict(X_train)<br>print(&quot;#### Training Set Prediction ####&quot;)<br>printmetrics(pred,y_train)<br>print(&quot;#### Testing Set Prediction ####&quot;)<br>printmetrics(logmodel.predict(X_test),y_test)</pre><p>The output of the model is given below</p><pre>Training Set Prediction ####<br>Accuracy Score 98.0%<br>Precision Score 97.0%<br>Recall Score 97.0%<br>F1 Score 97.0%<br>*** Confusion Matrix ***<br>[[347   6]<br> [  5 166]]<br>#### Testing Set Prediction ####<br>Accuracy Score 94.0%<br>Precision Score 92.0%<br>Recall Score 94.0%<br>F1 Score 93.0%<br>*** Confusion Matrix ***<br>[[99  6]<br> [ 4 66]]</pre><p>From the above given output, the accuracy is quite high and even Precision and Recall are high. The model mis-classifies only 11 out of 524 data points in the training set and 10 out of 175 data points.</p><h3><a href="https://en.wikipedia.org/wiki/Decision_tree">Decision Trees</a></h3><p>Decision Trees can be used for both classification and regression and use a tree like structure to arrive at a decision.</p><pre>dtreemodel = DecisionTreeClassifier()<br>dtreemodel.fit(X_train, y_train)<br>pred = dtreemodel.predict(X_train)<br>print(&quot;#### Training Set Prediction ####&quot;)<br>printmetrics(pred,y_train)<br>print(&quot;#### Testing Set Prediction&quot;)<br>printmetrics(dtreemodel.predict(X_test),y_test</pre><p>The output of the above code results in the following</p><pre>#### Training Set Prediction ####<br>Accuracy Score 100.0%<br>Precision Score 100.0%<br>Recall Score 100.0%<br>F1 Score 100.0%<br>*** Confusion Matrix ***<br>[[353   0]<br> [  0 171]]<br>#### Testing Set Prediction<br>Accuracy Score 94.0%<br>Precision Score 92.0%<br>Recall Score 93.0%<br>F1 Score 92.0%<br>*** Confusion Matrix ***<br>[[99  6]<br> [ 5 65]]</pre><p>On the training data set, a decision tree based classifier is able to classify with 100% accuracy, which is quite remarkable. However on the testing set, it is misclassifying 11 data points.</p><p>Anyhow, this is still fantastic performance.</p><h3><a href="https://en.wikipedia.org/wiki/Random_forest">Random Forests</a></h3><p>Random Forests are an ensemble of Decision Trees and use the <a href="https://en.wikipedia.org/wiki/Bootstrap_aggregating">bagging (Bootstrap Aggregating)</a> algorithm for the final prediction.</p><pre>rfmodel = RandomForestClassifier()<br>rfmodel.fit(X_train,y_train)<br>print(&quot;#### Training Set Prediction ####&quot;)<br>printmetrics(rfmodel.predict(X_train),y_train)<br>print(&quot;#### Testing Set Prediction&quot;)<br>printmetrics(rfmodel.predict(X_test),y_test)</pre><p>The output of the Random Forest model is given below.</p><pre>#### Training Set Prediction ####<br>Accuracy Score 100.0%<br>Precision Score 100.0%<br>Recall Score 100.0%<br>#### Confusion Matrix ####<br>[[353   0]<br> [  0 171]]<br>#### Testing Set Prediction<br>Accuracy Score 94.0%<br>Precision Score 92.0%<br>Recall Score 94.0%<br>#### Confusion Matrix ####<br>[[99  6]<br> [ 4 66]]</pre><p>Again, the model is able to fit the training data with 100% accuracy (zero misclassified data points). However on the testing set, it is mis-classifying 10 data points. It is quite similar to decision trees.</p><h3><a href="https://en.wikipedia.org/wiki/Support-vector_machine">Support Vector Machines</a></h3><p>Support Vector Machines are another class of Machine Learning algorithms that can be used for classification as well as regression. I applied SVM to the data set and the outcomes were quite similar.</p><pre>#### Training Set Prediction ####<br>Accuracy Score 100.0%<br>Precision Score 99.0%<br>Recall Score 100.0%<br>#### Confusion Matrix ####<br>[[351   2]<br> [  0 171]]<br>#### Testing Set Prediction<br>Accuracy Score 94.0%<br>Precision Score 89.0%<br>Recall Score 97.0%<br>#### Confusion Matrix ####<br>[[97  8]<br> [ 2 68]]</pre><p>Accuracy levels are quite high on the testing set as well as on the training set. Here, the precision is low but recall is quite high. Overall, this algorithm is just mis classifying only 10 points.</p><p>In addition to the above mentioned, I also ran <a href="https://xgboost.readthedocs.io/en/latest/">XGBoost </a>algorithm which gave similar performance as Decision Trees and Random Forests.</p><h3><a href="https://en.wikipedia.org/wiki/Artificial_neural_network">Neural Networks</a></h3><p>Neural Networks or Artificial Neural Networks are another class of Deep Learning Algorithms that try to mimic the neurons in the human brain.</p><p>A typical neural network has input layer (data), hidden layers and output layer. The architecture of a neural network is shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*3gjHsud_ZaoMzrjQ" /></figure><p>On the Breast Cancer data set, I ran a neural network with 6 hidden layers and each layer having atleast 30 nodes. The activation function used was RELU for the hidden layers.</p><pre>Training Set Prediction ####<br>Accuracy Score 98.0%<br>Precision Score 96.0%<br>Recall Score 99.0%<br>#### Confusion Matrix ####<br>[[346   7]<br> [  1 170]]<br>#### Testing Set Prediction<br>Accuracy Score 95.0%<br>Precision Score 92.0%<br>Recall Score 96.0%<br>#### Confusion Matrix ####<br>[[99  6]<br> [ 3 67]]</pre><p>The accuracy and other performance measures are not so different from the Logistic Regression or Decision Tree/Random Forest algorithms. Which shows that a neural network works best when there are multiple data variables and large number of rows. For a smaller data set (like in this case), the accuracy and performance is no different than regular algorithms.</p><h3>Conclusion</h3><p>From the applications above, it is quite apparent that Machine Learning definitely finds its way in the field of biology and complex cancer detection and possibly treatment. With a limited data set, the machine learning algorithms were able to learn from the training data set and predict with great accuracy on testing data set.</p><p>Machine Learning finds its applications in many domains and with improving computers (cheaper storage, memory, processing power), Machine Learning can be very useful to detect a complex condition such as breast cancer.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d57f2d72975c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/exploring-cancer-data-d57f2d72975c">Exploring Cancer Data</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[An attempt to predict Corona spread using Curve Fitting]]></title>
            <link>https://medium.com/analytics-vidhya/an-attempt-to-predict-corona-spread-using-curve-fitting-f5954cff5e?source=rss-bc9e46c9ed79------2</link>
            <guid isPermaLink="false">https://medium.com/p/f5954cff5e</guid>
            <category><![CDATA[corona]]></category>
            <category><![CDATA[covid19]]></category>
            <category><![CDATA[predict]]></category>
            <category><![CDATA[curves]]></category>
            <dc:creator><![CDATA[Amit Gairola]]></dc:creator>
            <pubDate>Sat, 04 Apr 2020 18:03:08 GMT</pubDate>
            <atom:updated>2020-04-13T08:35:12.398Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*UU7rEF1W_kLHIqx2" /><figcaption>Photo by <a href="https://unsplash.com/@sonance?utm_source=medium&amp;utm_medium=referral">Viktor Forgacs</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Corona Virus or COVID-19 is the latest pandemic that has come to haunt the human race. Whether it is synthetically manufactured or is an oddity of nature or a means of nature to restore balance, we don’t know. We also dont know what is the rate of progression of the spread of the virus, nor do we know when it is going to end.</p><p>So I decided to use a simple method — Curve Fitting to fit a polynomial equation on the Cumulative Corona Confirmed cases and Fatalities.</p><p>I used Microsoft Excel to plot the spread of Corona virus by country and then used Trend Line feature of Microsoft Excel to fit a polynomial equation. The data for this exercise came from <a href="https://www.kaggle.com/c/covid19-global-forecasting-week-2/overview">Kaggle</a> which is running a competition for the same. I did this for China, Italy, India and the US (aggregating data at the country level and then fitting a curve).</p><p><strong>So first let us look at China.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/0*yryi6c0Er-i2bJgV" /></figure><p>When we plot a Trend Line to this using Excel,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/0*olJPS1ZLL5xqXEH8" /></figure><p>The equation for this curve is given below</p><p>y = -0.0016×5 + 0.3116×4–22.452×3 + 655.91×2–4450.9x + 8330.9</p><p>The R-square value is 0.9932.</p><p>Interestingly, this is a fifth order polynomial.</p><p>Now when we look at the fatalities,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/678/0*fGCKwVYPVanqqnV1" /></figure><p>Interestingly this is a fourth order polynomial with the equation as</p><p>y = 0.0014×4–0.2183×3 + 10.288×2–85.324x + 205.12</p><p>The R-square value is 0.9981.</p><p><strong>Italy</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/586/0*UPHLEqwxhqeRBaYl" /></figure><p>This is also a fifth order polynomial with R-square of 0.9981.</p><p>Coming to fatalities, we again see a fourth order polynomial.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/511/0*B3fl5huVudPg5Xn9" /></figure><p><strong>India</strong></p><p>India is no different from the above. Here are the charts for the Confirmed cases and fatalities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/545/0*dkqFk0TeoMhpXc8_" /></figure><p>A fifth order polynomial fits the curve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/591/0*ctckIjZMUOo0iJIA" /></figure><p><strong>United States</strong></p><p>United States is a slightly different case. The Confirmed Cases are a fifth order polynomial while the fatalities are a sixth order one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/0*ltFo2VIIeOIVki0b" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/0*_iSNQ8iPZ9xII9j_" /></figure><p><strong>Conclusion</strong></p><p>Apart from the data from China, most of the countries have shown an upward trend and it appears that there is still some time for the curve to become an S-curve and flatten out.</p><p>Hopefully as more data comes up, better models can be developed to predict the spread and identify the causes for the spread.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f5954cff5e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/an-attempt-to-predict-corona-spread-using-curve-fitting-f5954cff5e">An attempt to predict Corona spread using Curve Fitting</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>