<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Prasad Patil on Medium]]></title>
        <description><![CDATA[Stories by Prasad Patil on Medium]]></description>
        <link>https://medium.com/@theprasadpatil?source=rss-98cd7d3f47d6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*X2WShgpUoh4icYdAlBATxw.jpeg</url>
            <title>Stories by Prasad Patil on Medium</title>
            <link>https://medium.com/@theprasadpatil?source=rss-98cd7d3f47d6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 17:50:05 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@theprasadpatil/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Last Minute Revision: Part I - Machine Learning & Statistics]]></title>
            <link>https://medium.com/@theprasadpatil/last-minute-revision-part-i-machine-learning-statistics-8de23a377987?source=rss-98cd7d3f47d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/8de23a377987</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[unsupervised-learning]]></category>
            <category><![CDATA[data-science-interview]]></category>
            <category><![CDATA[supervised-learning]]></category>
            <category><![CDATA[confusion-matrix]]></category>
            <dc:creator><![CDATA[Prasad Patil]]></dc:creator>
            <pubDate>Mon, 30 May 2022 07:38:04 GMT</pubDate>
            <atom:updated>2022-05-30T07:38:04.728Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mrbuCPVqLxNlqilsFJcWMg.jpeg" /><figcaption>Source : Image by <a href="https://pixabay.com/users/27707-27707/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=951818">Kevin Phillips</a> from <a href="https://pixabay.com/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=951818">Pixabay</a></figcaption></figure><p>Here’s the ready reference for quick go through to some of the key concepts related to Machine Learning and Statistics. This 7 min material curated especially for the readers who are minutes away from appearing before Interview Panel and for those who wish to touch base everything in short time.</p><p>I have attempted to cover short summaries and a very high level overview of few important Machine Learning Algorithms-their basic intuition in gist , confusion metrics and few very common statistical terms in this article.</p><h3>Machine Learning Algorithms</h3><h4>Linear Regression</h4><ul><li>Assuming there is linear relationship between independent feature (X) and target (y), prediction of real continuous target feature by finding best fitting line.</li><li>Representation :</li></ul><pre><strong>y = m*X + c</strong></pre><p>Values of m(slope) and c(intercept) estimated such that cost function is minimized</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/378/1*Ted27kM8fGYKxk0x5scIZQ.png" /></figure><ul><li>Values of coefficients (m &amp; c) are determined using methods : Ordinary least squares, Gradient descent ,Normal Equation.</li></ul><h4>Logistic Regression</h4><ul><li>It is used to estimate discrete values in classification problem.</li><li>It predicts the probability of occurrence of class in target variable .Hence output ranges from 0 to 1.</li></ul><p>Representation :</p><pre><strong>y = 1/(1 + e^(-z))</strong></pre><ul><li>It is an equation of a sigmoid curve.</li><li>Cost function for a binary classification problem:</li></ul><p><strong>-y*log( hθ(x) ) — (1-y)*log( 1- hθ(x) )</strong></p><ul><li>Performance is measured using confusion matrix, AUC-ROC</li></ul><h4>Decision Tree</h4><ul><li>Applicable for both classification &amp; regression problem.</li><li>Entire population is divided into two or more homogeneous sets .</li><li>This is done by using various techniques like <strong>Gini</strong>, <strong>Chi-square</strong>, <strong>entropy</strong> etc.</li></ul><h4>Gini Index :</h4><pre><strong>1-[(P)²+(1-P)²]</strong></pre><h4>Chi-square :</h4><p><strong>(Actual — Expected)²/ Expected)¹/2</strong></p><h4>Entropy :</h4><pre><strong>-p*log(p) — q*log(q)</strong></pre><p>Predictions are made by walking down the splits of the tree from root node until arriving at a leaf node.</p><h4>Output:</h4><p><strong>for classification</strong> — class value at that leaf node</p><p><strong>for Regression</strong> — average of samples at that terminal.</p><h4>Random Forest</h4><ul><li>Bootstrap Aggregation of decision trees. A random subset of the training data (with replacement) selected to train each tree.</li><li>The model randomly restricts the variables which may be used at the splits of each tree.</li><li>Hence, the trees grown are dissimilar, but they still retain certain predictive power.</li><li>Random forest is an ensemble technique where multiple models are trained on same data &amp; their predictions are combined either by majority-voting or by taking averages.</li><li>It is a black box model &amp; more of an engineering hack than algorithm.</li></ul><h4>Adaptive Boosting</h4><ul><li>Ensemble technique where predictions from multiple weak learners (models) combined to form one strong learner.</li><li>Algorithm functions by assigning more weight on miss-classified instances than correctly classified ones.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/155/1*aSF2uPdO4E0BRhsKMSj2cw.png" /></figure><p>α = weight for the classifier</p><p>ε = minimum misclassification error for the model</p><h4>Gradient Boosting</h4><ul><li>Like AdaBoost, Gradient Boosting works by sequentially adding predictors to an ensemble, each one correcting its predecessor.</li><li>However, instead of tweaking the instance weights at every iteration like AdaBoost, it tries to fit the new predictor to the residual errors made by the previous predictor.</li><li>Gradient boosting machines are generally very slow in implementation.</li><li>It is because of sequential model training. Hence, they are not very scalable.</li><li>Remedy to it : eXtreme Gradient Boosting (XGBoost) designed for speed &amp; performance.</li></ul><h4>K-means Clustering</h4><ul><li><strong>STEP 1</strong>: Choose the number K of clusters</li><li><strong>STEP 2</strong>: Select at random K points, the centroids(not necessarily from your dataset)</li><li><strong>STEP 3</strong>: Assign each data point to the closest centroid &gt;That forms K Cluster</li><li><strong>STEP 4</strong>: Compute and place the new centroid of each cluster.</li><li><strong>STEP 5</strong>: Reassign each data point to the new closest centroid. If any reassignment took place, go to STEP 4, otherwise stop</li></ul><p>Optimal K value is obtained from <strong>Elbow method </strong>(aka within cluster variation) and <strong>Silhouette Analysis</strong>.</p><h4>Hierarchical Clustering</h4><ul><li>Unlike K-means, here no need to specify number of clusters beforehand.</li><li><strong>Types</strong>: Agglomerative or Bottom-Up Clustering &amp; Divisive or Top-Down Clustering</li><li>In <strong>agglomerative</strong>, every data point acts as single cluster (singleton) and based on similarities clusters are merged with each other and together they form a single cluster of all instances.</li><li>In <strong>divisive</strong>, exactly reverse process of agglomerative takes place.</li><li>Distance or similarity between two clusters ,determined by single, complete and average linkage.</li></ul><h3>Confusion Matrix</h3><p>It is a N*N matrix where N is number of classes being predicted. Here in our example we have 2 classes(Positive &amp; Negative) ,so it is a 2*2 matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/487/1*Z4ZHPwVnU4aBYI25au34jg.png" /></figure><p>We have data of 165 patients and target variable is whether patient has cancer(+ve class) or not(-ve class).</p><p>Let decipher terminologies in confusion matrix one by one…</p><p><strong>TN(True Negative) :</strong></p><p>Our model predicted that particular patient as non cancer and in reality also he had no cancer. (50 cases/165)</p><p><strong>TP (True Positive)</strong> :</p><p>Our model predicted that a patient has cancer and in reality he actually had a cancer.(100 cases/165)</p><p><strong>FP (False Positive aka Type I error):</strong></p><p>Our model predicted that a patient has cancer but in reality he had no cancer. (10 cases / 165)</p><p><strong>FN(False Negative aka Type II error) :</strong></p><p>Our model predicted that a patient has no cancer but in reality he had cancer. (5 cases / 165)</p><h4>Accuracy :</h4><p>Total correct predictions out of Total predictions made.</p><p>Mathematically,</p><pre><strong>(TP + TN) /(TP+TN+FP+FN)</strong></pre><p>From example,</p><p>(100+50)/ (50+100+10+5)</p><p>= 150/165</p><p>= 0.91</p><h4>Recall :</h4><p>It is often termed as sensitivity aka accuracy of positive class.</p><p>In reality out of 165 patients 105 patients were diagnosed with cancer.</p><p>So recall basically says “Out of 105 cancer patients how many patients our model able to pick correctly”</p><p>Mathematically,</p><pre><strong>(TP) /(TP + FN)</strong></pre><p>From example,</p><p>(100)/(100+5)</p><p>= (100)/(105)</p><p>= 0.95</p><h4>Precision :</h4><p>Precision says “Out of all predicted positives- model said patient has cancer, how many patients actually had cancer in real life.”</p><p>Mathematically,</p><pre><strong>(TP) /(TP + FP)</strong></pre><p>(100)/(100+10)</p><p>= (100)/(110) =0.91</p><h4>Specificity :</h4><p>It is often termed as accuracy of negative class. Actual negative cases correctly identified out of Total.</p><p>Out of 165 patients, there were 60 patients with cancer , so basically specificity means out those 60 patients how many patients our model could pick the non cancer ones correctly.</p><p>Mathematically ,</p><pre><strong>TN / (TN + FP)</strong></pre><p>50 / (50+10)</p><p>= 50 / 60</p><p>= 0.83</p><h3>STATISTICS</h3><h4>Correlation</h4><p><strong>range</strong> = [-1,1]</p><p><strong>Intuition</strong> : Lets say we have 2 features in data set namely x &amp; y. Both x and y correlated when y is moving when x moves. If direction of their movement is same then they are positively correlated otherwise negatively.</p><p>Point to note that positive correlation isn’t mean good correlation and negative correlation does not mean bad correlation. Polarities simply indicate the direction of movement.</p><p>Correlation can be measured using Pearson correlation coefficient(r).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/361/1*KAwMGXzM8BUEZ4im_4Nu5g.jpeg" /></figure><p>r =1 means ,for every +ve increment of 1 in x , there is +ve increment of 1 in y.</p><p>r = -1 means ,for every +ve increment of 1 in x ,there is -ve increment of 1 in y.</p><h4>Permutation</h4><p>A selection of objects in which the order of the object matters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/368/1*Vvii3SHVKg5dIyk2E-fdgQ.png" /></figure><h4>Combination</h4><p>The number of possible combinations of r objects from set of n objects.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*1CA5tUYmoKscexko9tm5BA.png" /></figure><h4>Central Limit Theorem</h4><p>Irrespective of the underlying distribution of population, the sampling distribution of the sample means is always normal distribution.</p><ol><li>Mean of the sampling distribution of sample means(x-bar) is same as population mean(mew)</li><li>Standard deviation of sampling distribution is ratio of standard deviation of population by square root of samples drawn(n)</li></ol><h4>Hypothesis Testing</h4><p>Ho : Null Hypothesis</p><p>H1 : Alternate Hypothesis</p><p>Ho is assumed first to be True and tested later. Based on findings it is either accepted or rejected.</p><p>H1 can only be accepted &amp; if their is any contradiction we say due to insufficient data we accept H0 but we can not say we are rejecting H1 or H1 is not correct.</p><h3>Z-Test</h3><p>Assumptions</p><ol><li>Sample selected randomly</li><li>Observations are independent</li><li>The population standard deviation is known or the sample contains at least 30 observations</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/496/1*aGjJ475UnzJsybiJ-rY10w.png" /></figure><p>Hope you find it useful and helps you relieve some of those brain muscles in jittery situations. Stay tuned for the next article! #HappyLearning</p><p><strong>Previously Published Articles </strong>-</p><ul><li><a href="https://towardsdatascience.com/exploratory-data-analysis-8fc1cb20fd15">What is Exploratory Data Analysis?</a></li><li><a href="https://medium.com/@theprasadpatil/how-to-create-a-pdf-report-from-excel-using-python-b882c725fcf6">How to create a PDF report from Excel using Python</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8de23a377987" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to create a PDF report from Excel using Python]]></title>
            <link>https://medium.com/@theprasadpatil/how-to-create-a-pdf-report-from-excel-using-python-b882c725fcf6?source=rss-98cd7d3f47d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/b882c725fcf6</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[automation]]></category>
            <dc:creator><![CDATA[Prasad Patil]]></dc:creator>
            <pubDate>Sat, 18 Dec 2021 04:10:17 GMT</pubDate>
            <atom:updated>2022-05-30T07:43:16.787Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qluRujC4FLuZTvdbqRiKGA.png" /><figcaption>Source: Pixabay.com</figcaption></figure><p>If you are seeking ways to automate your excel data into PDF report output to dazzle your stakeholders, then you’ve come to the right place!</p><p>This blog post will guide you on how to make a PDF report consisting of a logo, an image along-with textual content of various sizes, shapes, and colors by sourcing information from an excel workbook. To perform this task I am going to use the FPDF library of Python so c’mon you all let&#39;s just hop right in…!</p><p><em># Happy Reading</em></p><h4><a href="https://pdfexpert.com/blog/why-its-better-to-use-pdf-files-instead-of-word-excel-or-powerpoint-documents">Why PDF reports?</a></h4><p>PDF short for Portable Document Format as the name suggests helps you to transfer or move files swiftly with anyone; not needing to worry about the fonts, images, tables, and the formatting it contains. On the contrary, other MS suite file formats such as Word, PowerPoint, or Excel do not provide that level of ease. They often take time to load or consume a lot of space. A few more strong reasons one should favor the use of reports in PDF format over others are — PDFs are</p><ul><li>Aesthetically pleasing</li><li>Reliable + Secured</li><li>Universal compatibility</li><li>Version independence</li><li>Professionally appealing</li></ul><p>You can leverage complete code and learnings from this article to automate and generate PDFs as deemed fit for your requirement.</p><h4><a href="https://pyfpdf.readthedocs.io/en/latest/index.html">FPDF Python Library</a></h4><p>FPDF is a library for PDF document generation under Python, ported from PHP. F from FPDF stands for Free: which means you may use it for any kind of usage and modify it to suit your needs. The library originally offers the following features -</p><ul><li>Choice of measure unit, page format, and margins</li><li>Page header and footer management</li><li>Automatic page break</li><li>Automatic line break and text justification</li><li>Image support (JPEG, PNG, and GIF), colors and links support</li><li>TrueType, Type1, and encoding support</li><li>Page compression</li><li>Extensive <a href="http://www.fpdf.org/en/tutorial/index.php">Tutorial</a> and complete online <a href="http://www.fpdf.org/en/doc/index.php">documentation</a></li></ul><h4>Installation</h4><p>You can install FPDF from the command line using pip.</p><pre>pip install fpdf</pre><p>If you are using a jupyter notebook/Spyder, you can write the same line of code just replacing “pip” with “!pip” in the coding area. Another way would be to search this library in the Environments tab of Anaconda Navigator.</p><h4>Sneak Peek</h4><p>To fast forward you into the future, this is how our final output would look like. Created out of nothing, literally. You can do wonders with Python code, don’t you agree?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xvOPy3TvsyqcjWwjN-eqUQ.jpeg" /></figure><p>Let’s understand from scratch how I went about it.</p><h4>Input Data</h4><p>Starting with what goes in as input, I prepared dummy data points in excel by collating some information from <a href="https://en.wikipedia.org/wiki/Arsenal_F.C.">Wikipedia</a> and <a href="https://www.britannica.com/topic/Arsenal-English-football-club">Britannica</a> about the football club I follow -Arsenal as shown in output PDF, from the famous English Premier League.</p><p>The excel file has 8 attributes in the following order: year of foundation, Name of the club’s current stadium, current manager, club motto, list of trophies along with the years/seasons in which they were won, the nickname supporters widely known by, short summary of cabinet glory and a brief history of the club.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jDgQx5Dur4qqrh8cHx9PWg.png" /><figcaption>Screenshot of the data in excel file</figcaption></figure><h4>Ingredients and the Recipe</h4><p>Moving to the nitty-gritty, now I will take you through each functionality I leveraged from the FPDF library. At first, you need to import the library and initialize it by assigning it to a variable which in our case named pdf.</p><pre>from fpdf import FPDF<br>pdf =<strong> </strong><a href="https://pyfpdf.readthedocs.io/en/latest/reference/FPDF/index.html"><strong>FPDF()</strong></a></pre><p>The FPDF constructor takes 3 parameters -</p><p>Orientation: Portrait(“P”), Landscape (“L”)</p><p>Unit: millimeters (“mm”), centimeter(“cm”), inches(“in”)</p><p>Format :“A3” ,”A4&quot;,”A5&quot;,”Letter”,”Legal”. You can also pass on a custom format for example -(WIDTH=100, HEIGHT=150)</p><p><em>If not given explicitly, the object assumes (“P”, “mm”, “A4”) by default.</em></p><pre><a href="https://pyfpdf.readthedocs.io/en/latest/reference/add_page/index.html"><strong>add_page</strong></a><strong>(orientation </strong>= &#39;&#39;,<strong> format </strong>= &#39;&#39;<strong>, same </strong>= False<strong>)</strong></pre><p>It does exactly what its name says, it adds a page to the document. It also takes 3 parameters. By default, it takes whatever is passed on to the constructor at the beginning. You can choose from the same options that are there in FPDF() for orientation and format. To Same, if True is passed it carry forwards settings of the previous page and other arguments are ignored.</p><pre><a href="https://pyfpdf.readthedocs.io/en/latest/reference/set_font/index.html"><strong>set_font</strong></a><strong>(family </strong>= &quot;&quot;,<strong> style </strong>= &quot;&quot;,<strong>size </strong>=<strong> )</strong></pre><p>Before exporting the output, we must set the font of the document. Otherwise, it will be invalid. You can specify below arguments for the listed parameters</p><p>Family refers to the font name and available case insensitive standard fonts are “Courier”, “Helvetica”, “Arial”, “Times”, “Symbol”, “ZapfDingbats”. You can also explore <a href="https://www.youtube.com/watch?v=o7vixt2mVdo">how to use custom google fonts</a>.</p><p>Style: Bold (“B”), Underline (“U”), Italics (“I”), regular(“ ”). You can also specify a combination of two or more font styles like Bold+Underline as “BU”. Keep in mind bold and italic styles do not apply to Symbol and ZapfDingbats<em>.</em></p><p>Size is nothing but font size. The default value is 12 points.</p><pre><a href="https://pyfpdf.readthedocs.io/en/latest/reference/image/index.html"><strong>image</strong></a><strong>(name</strong>, <strong>x</strong> = 0, <strong>y</strong> = 0, <strong>w</strong> = 0, <strong>h</strong> = 0, <strong>type</strong> = <strong>&#39;&#39;</strong>, <strong>link</strong> = <strong>&#39;&#39;)</strong></pre><p>I have used this one quite a lot to make our report interactive. The FPDF library supports JPEG, PNG, and GIF formats.</p><p>name is the path of the picture.</p><p>x is The distance of a point from the y-axis scaled with the x-axis also known as called <em>abscissa</em> or x coordinate of the point.</p><p>y is the distance of a point from the x-axis scaled with the y-axis and is known as <em>ordinate</em>. Both x and y start at the upper left corner of the page. W is width and h is height. All these four dimension parameters if not specified explicitly, automatically get calculated.</p><p>typer is the case-insensitive format of the image (i.eJPG, JPEG, PNG, and GIF. If not specified, the type is inferred from the file extension and link is URL or identifier returned by <a href="https://pyfpdf.readthedocs.io/en/latest/reference/add_link/index.html"><strong>add_link</strong></a></p><pre><a href="https://pyfpdf.readthedocs.io/en/latest/reference/cell/index.html"><strong>cell</strong></a><strong>(w</strong>,<strong>h</strong>=0,<strong>txt</strong>=&#39;&#39;,<strong>border</strong>=0,<strong>ln</strong>=0,<strong>align</strong> = &#39;&#39;,<strong>fill</strong> = False, <strong>link</strong> = &#39;&#39;<strong>)<br></strong><a href="https://pyfpdf.readthedocs.io/en/latest/reference/multi_cell/index.html"><strong>multi_cell</strong></a><strong>(w</strong>,<strong>h</strong>=0,<strong>txt</strong>=&#39;&#39;,<strong>border</strong>=0,<strong>ln</strong>=0,<strong>align</strong> = &#39;&#39;,<strong>fill</strong> = False, <strong>link</strong> = &#39;&#39;<strong>)</strong></pre><p>The only difference between cell and multi_cell is multi_cell comes with automatic line breaks. Cell inserts rectangular text box in simple terms with border and text alignment options.</p><p>w: Cell width,<em> </em>h: Cell height, text: strings you want to print, border takes 0 for no border and 1 for a frame or it can take some or all combinations of “L”(left), “R”(right), “T”(top), “B”(bottom).</p><p>To align you can pass “L”(left), “R”(right), “C”(center), “J”( justification), and ln indicates where the cursor position should go to. Possible values for which are 0:to the right, 1:to the beginning of the next line, 2:below.</p><p>fill indicates if the cell background is to be filled with color(True) specified in fill_colur or make a transparent (False)and the link is URL or identifier returned by <a href="https://pyfpdf.readthedocs.io/en/latest/reference/add_link/index.html"><strong>add_link</strong></a></p><pre><a href="https://pyfpdf.readthedocs.io/en/latest/reference/set_fill_color/index.html"><strong>set_fill_color</strong></a><strong>(r</strong>: int<strong>, g</strong>: int<strong> , b</strong>: int <strong>)<br></strong><a href="https://pyfpdf.readthedocs.io/en/latest/reference/set_text_color/index.html"><strong>set_text_color</strong></a><strong>(r:</strong> int, <strong>g:</strong> int , <strong>b:</strong> int <strong>)</strong></pre><p>Both of these take RGB components where r: red, g: green, and b: blue. You can specify any value between 0 to 255. text_color defines the color of text and fill_color is for filling up the background of the cell.</p><pre><a href="https://pyfpdf.readthedocs.io/en/latest/reference/set_xy/index.html"><strong>set_xy</strong></a><strong>(x: </strong>float<strong>, y: </strong>float<strong>)</strong></pre><p>x takes abscissa and y ordinates. If provided negative values, the cursor would shift to the right bottom corner of the page relatively.</p><pre><a href="https://pyfpdf.readthedocs.io/en/latest/reference/output/index.html"><strong>output</strong></a><strong>(name </strong>= &#39;&#39;,<strong> dest </strong>= &#39;&#39;<strong>)</strong></pre><p>name: The name of the file. Only used when writing to a file and dest: Destination where the document to be sent</p><h4><strong>Complete Python Code</strong></h4><p>You can check out my <a href="https://github.com/PBPatil/Python-Automation-Projects/tree/main/Automate%20PDF%20using%20Python"><strong><em>Github repository</em></strong></a> to find all assets-Logos, Background Image, stadium wallpaper, and the input excel file I used for this exercise.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b982fe0017a2962d71594aa2cdd7975c/href">https://medium.com/media/b982fe0017a2962d71594aa2cdd7975c/href</a></iframe><p>Stay tuned for the next article! #HappyLearning</p><h4>References</h4><ol><li><a href="https://pdfexpert.com/blog/why-its-better-to-use-pdf-files-instead-of-word-excel-or-powerpoint-documents">Why PDFs are better?</a></li><li><a href="https://pyfpdf.readthedocs.io/en/latest/index.html">FPDF documentation</a></li><li><a href="https://www.youtube.com/watch?v=p1_7t2SOCIs&amp;t=539s">Learn to add tables in PDF from excel</a> (<em>for you to explore further!)</em></li></ol><p><strong>Previously Published Articles </strong>-</p><ul><li><a href="https://towardsdatascience.com/exploratory-data-analysis-8fc1cb20fd15">What is Exploratory Data Analysis?</a></li><li><a href="https://medium.com/@theprasadpatil/last-minute-revision-part-i-machine-learning-statistics-8de23a377987">Last Minute Revision: Part I — Machine Learning &amp; Statistics</a></li><li><a href="https://towardsdatascience.com/decision-tree-data-scientists-magic-bullet-for-hamletian-dilemma-411e0121ba1e?source=your_stories_page----------------------------------------">Decision Tree — Data Scientist’s magic bullet for Hamletian Dilemma</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b882c725fcf6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Shortcuts Excel/Word]]></title>
            <link>https://medium.com/series/shortcuts-excel-word-de476fc53be4?source=rss-98cd7d3f47d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/de476fc53be4</guid>
            <dc:creator><![CDATA[Prasad Patil]]></dc:creator>
            <pubDate>Fri, 07 Aug 2020 13:48:09 GMT</pubDate>
            <atom:updated>2020-08-07T13:48:50.466Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>Useful short cuts in word</strong></h4><p>Fonts = Ctrl + D</p><p>Left Align text = Ctrl + L</p><p>Right Align text = Ctrl + R</p><p>Center Align text = Ctrl +E</p><p>Full Justify text = Ctrl + J</p><h4>Useful short cuts in Excel</h4><p>Save as = F12</p><p>Spelling Check = F7</p><p>Strikethrough = Ctrl +5</p><p>Custom Sort = Alt + A+S+S</p><p>Quick Data Analysis = Ctrl + Q</p><h3>More to come….</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=de476fc53be4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Shortcuts for Jupyter Notebook]]></title>
            <link>https://medium.com/series/shortcuts-for-jupyter-notebook-5485dcfd0ae3?source=rss-98cd7d3f47d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/5485dcfd0ae3</guid>
            <dc:creator><![CDATA[Prasad Patil]]></dc:creator>
            <pubDate>Tue, 04 Jun 2019 04:22:46 GMT</pubDate>
            <atom:updated>2019-06-04T04:22:46.746Z</atom:updated>
            <content:encoded><![CDATA[<h3>To change modes</h3><h3>(edit , command)</h3><pre><strong>Esc</strong> - to change mode to command mode<br><strong>Enter</strong> - to change mode to edit mode</pre><h3>To change content type</h3><h3>(code or markdown)[in command mode]</h3><pre><strong>m</strong> - To change to markdown<br><strong>y </strong>- to change to code</pre><h3>To execute code or markdown</h3><h3>[any mode]</h3><pre><strong>Shift + Enter</strong> - To execute and go to next cell<br><strong>Ctrl + Enter</strong>  - To execute and be in the same cell</pre><h3>To insert cell</h3><h3>[in command mode]</h3><pre><strong>a</strong> - To create cell in above to the cell<br><strong>b</strong> - To create cell in bellow to the cell</pre><h3>To cut copy paste</h3><h3>[in command mode]</h3><pre><strong>x</strong> - To cut the cell that can be paste any where any number of times<br><strong>c</strong> - To copy the cell that can be paste any where and any number of times<br><strong>v</strong> - To paste the cell </pre><h3>More to Come …</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5485dcfd0ae3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine Learning Basic Concepts]]></title>
            <link>https://medium.com/series/machine-learning-basic-concepts-9937783a6a97?source=rss-98cd7d3f47d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/9937783a6a97</guid>
            <dc:creator><![CDATA[Prasad Patil]]></dc:creator>
            <pubDate>Wed, 18 Jul 2018 12:06:44 GMT</pubDate>
            <atom:updated>2018-07-23T14:17:01.008Z</atom:updated>
            <content:encoded><![CDATA[<h4>Confusion Matrix</h4><p>It is a N*N matrix where N is number of classes being predicted.Here in our example we have 2 classes(+ve &amp; -ve) ,so it is a 2*2 matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/487/1*Z4ZHPwVnU4aBYI25au34jg.png" /></figure><p>We have data of 165 patients and target variable is whether patient has cancer(+ve class) or not(-ve class).</p><p>Let decipher terminologies in confusion matrix one by one…</p><p><strong>TN(True Negative) :</strong></p><p>Our model predicted that particular patient as non cancer and in reality also he had no cancer. (50 cases/165)</p><p><strong>TP (True Positive)</strong> :</p><p>Our model predicted that a patient has cancer and in reality he actually had a cancer.(100 cases/165)</p><p><strong>FP (False Positive aka Type I error):</strong></p><p>Our model predicted that a patient has cancer but in reality he had no cancer. (10 cases / 165)</p><p><strong>FN(False Negative aka Type II error) :</strong></p><p>Our model predicted that a patient has no cancer but in reality he had cancer. (5 cases / 165)</p><h4>Accuracy :</h4><p>Total correct predictions out of Total predictions made.</p><p>Mathematically,</p><pre><strong><br>(TP + TN) /(TP+TN+FP+FN)</strong></pre><p>From example,</p><p>(100+50)/ (50+100+10+5)</p><p>= 150/165</p><p>= 0.91</p><h4>Recall :</h4><p>It is often termed as sensitivity aka accuracy of positive class.</p><p>In reality out of 165 patients 105 patients were diagnosed with cancer.</p><p>So recall basically says “Out of 105 cancer patients how many patients our model able to pick correctly”</p><p>Mathematically,</p><pre><strong>   <br>    (TP) /(TP + FN)</strong></pre><p>From example,</p><p>(100)/(100+5)</p><p>= (100)/(105)</p><p>= 0.95</p><h4>Precision :</h4><p>Precision says “Out of all predicted positives- model said patient has cancer, how many patients actually had cancer in real life.”</p><p>Mathematically,</p><pre><strong><br>(TP) /(TP + FP)</strong></pre><p>(100)/(100+10)</p><p>= (100)/(110) =0.91</p><h4>Specificity :</h4><p>It is often termed as accuracy of negative class. Actual negative cases correctly identified out of Total.</p><p>Out of 165 patients, there were 60 patients with cancer , so basically specificity means out those 60 patients how many patients our model could pick the non cancer ones correctly.</p><p>Mathematically ,</p><pre><strong>  <br>    TN / (TN + FP)</strong></pre><p>50 / (50+10)</p><p>= 50 / 60</p><p>= 0.83</p><h3>Bias</h3><p>How much on an average are the predicted values different from the actual values.</p><h3>Variance</h3><p>If different samples are taken from the same population, how different will the predictions of the model be at the same point.</p><h3>More to come…</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9937783a6a97" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Extract Keywords from PDFs and arrange in order of their weights using Python]]></title>
            <link>https://medium.com/data-science/how-to-extract-keywords-from-pdfs-and-arrange-in-order-of-their-weights-using-python-841556083341?source=rss-98cd7d3f47d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/841556083341</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[text-mining]]></category>
            <category><![CDATA[regular-expressions]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Prasad Patil]]></dc:creator>
            <pubDate>Tue, 17 Jul 2018 17:40:20 GMT</pubDate>
            <atom:updated>2022-05-30T07:42:29.750Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tlggbqlcf36P5OHv002yFQ.jpeg" /><figcaption><strong>Batman masked Word cloud : Find code at the bottom</strong></figcaption></figure><p>Keyword extraction is nothing but the task of identification of terms that best describe the subject of a document.</p><p><em>“Key phrases”</em>, “<em>key terms”</em>, “<em>key segments”</em> or just “<em>keywords”</em> are the different nomenclatures often used for defining the terms that represent the most relevant information contained in the document. Although they may sound distinct,but they all serve the same purpose: characterization of the topic discussed in a document.</p><h3>Why extract keywords?</h3><p>Extracting keywords is one of the most important tasks while working with text data in the domain of Text Mining, Information Retrieval and Natural Language Processing.</p><h4>Use cases :</h4><blockquote>Readers benefit from keywords because they can judge more quickly whether the given text is worth reading or not.</blockquote><blockquote>Website makers benefit from keywords because they can group similar content by their topics.</blockquote><blockquote>Algorithm programmers benefit from keywords because they reduce the dimensionality of text to the most important features.</blockquote><p>These are just few examples among many.</p><p>Let’s get down to the nitty-gritty of the topic.Up next follows a tutorial on how you can parse through a PDF file and convert it in to a list of keywords.We will learn and cement our understanding by taking a hands-on problem, so code along!</p><h3>Problem Statement -</h3><h4>Given a particular PDF/Text document ,How to extract keywords and arrange in order of their weightage using Python?</h4><h3>Dependencies :</h3><p>(I have used Python 2.7.15 version for this tutorial.)</p><p>You will need below mentioned libraries installed on your machine for the task.In case you don’t have it,I have inserted codes for each dependency in code block below, which you can type it on command prompt for windows or on terminal for mac operating system.</p><ul><li><strong>PyPDF2 </strong>(To convert simple, text-based PDF files into text readable by Python)</li></ul><pre>pip install PyPDF2</pre><ul><li><strong>textract </strong>(To convert non-trivial, scanned PDF files into text readable by Python)</li></ul><pre>pip install textract</pre><ul><li><strong>re</strong> (To find keywords)</li></ul><pre>pip install regex</pre><p><strong><em>Note</em></strong> : I have attempted three approaches for this task.Above libraries would be suffice for approach 1.However I have just touched upon two other approaches which I found online.Treat them as alternatives. Down below is the jupyter notebook with all three approaches.Take a look!</p><h3>Jupyter Notebook :</h3><p>All necessary remarks are denoted with ‘#’.</p><ul><li><strong>Approach 1 unboxed</strong></li></ul><p><strong>Step 1</strong>: Import all libraries.</p><p><strong>Step 2</strong>: Convert PDF file to txt format and read data.</p><p><strong>Step 3</strong>: Use “<em>.findall()”</em> function of regular expressions to extract keywords.</p><p><strong>Step 4</strong>: Save list of extracted keywords in a DataFrame.</p><p><strong>Step 5 </strong>: Apply concept of TF-IDF for calculating weights of each keyword.</p><p><strong>Step 6</strong> : Save results in a DataFrame and use “<em>.sort_values()” to arrange keywords in order.</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fb09be337304918c4cd4c063d100180c/href">https://medium.com/media/fb09be337304918c4cd4c063d100180c/href</a></iframe><p>For more details find GitHub repo <a href="https://github.com/PBPatil/Keyword-Extracter"><strong>HERE</strong></a><strong> </strong>!</p><p><strong>References</strong> :</p><ol><li><a href="https://en.wikipedia.org/wiki/Keyword_extraction">www.wikipedia.org</a></li></ol><p>2. <a href="https://medium.com/@rqaiserr/how-to-convert-pdfs-into-searchable-key-words-with-python-85aab86c544f">Medium post for PDF to Text Conversion</a></p><p>3. <a href="https://www.airpair.com/nlp/keyword-extraction-tutorial">keyword extraction tutorial</a></p><p>4. <a href="https://www.analyticsvidhya.com/blog/2015/06/regular-expression-python/">Regular expressions</a></p><p>I hope you find this tutorial fruitful and worth reading. Also,I am sure there must be tons of other approaches with which you can perform the said task.Do share them in comment section if you have came across any.</p><p><strong>Code for the Masked Word Cloud</strong> :</p><p>Find GitHub repo <a href="https://github.com/PBPatil/Masked-Word-cloud"><strong>HERE</strong></a><strong> </strong>!</p><pre><em># modules for generating the word cloud</em> <br>from os import path, getcwd <br>from PIL import Image <br>import numpy as np <br>import matplotlib.pyplot as plt <br>from wordcloud import WordCloud</pre><pre>%matplotlib inline <br>d = getcwd()  <br>text = open(&#39;nlp.txt&#39;,&#39;r&#39;).read()</pre><pre>#Image link = &#39;<a href="https://produto.mercadolivre.com.br/MLB-693994282-adesivo-decorativo-de-parede-batman-rosto-e-simbolo-grande-_JM">https://produto.mercadolivre.com.br/MLB-693994282-adesivo-decorativo-de-parede-batman-rosto-e-simbolo-grande-_JM</a>&#39;  </pre><pre>mask = np.array(Image.open(path.join(d, &quot;batman.jpg&quot;)))<br> <br>wc = WordCloud(background_color=&quot;black&quot;,max_words=3000,mask=mask,\<br>               max_font_size=30,min_font_size=0.00000001,\<br>               random_state=42,)    <br>wc.generate(text) <br>plt.figure(figsize=[100,80]) <br>plt.imshow(wc, interpolation=&quot;bilinear&quot;) <br>plt.axis(&quot;off&quot;) <br>plt.savefig(&#39;bat_wordcloud.jpg&#39;,bbox_inches=&#39;tight&#39;,pad_inches=0.3);</pre><p><strong>Read Other Articles I Published </strong>-</p><ol><li><a href="https://medium.com/@theprasadpatil/how-to-create-a-pdf-report-from-excel-using-python-b882c725fcf6">How to create a PDF report from Excel using Python</a></li><li><a href="https://medium.com/@theprasadpatil/last-minute-revision-part-i-machine-learning-statistics-8de23a377987">Last Minute Revision: Part I — Machine Learning &amp; Statistics</a></li><li><a href="https://towardsdatascience.com/exploratory-data-analysis-8fc1cb20fd15">What is Exploratory Data Analysis?</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=841556083341" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-extract-keywords-from-pdfs-and-arrange-in-order-of-their-weights-using-python-841556083341">How to Extract Keywords from PDFs and arrange in order of their weights using Python</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Finding the favorite team in 2018 FIFA World Cup through scraping Tweets]]></title>
            <link>https://medium.com/data-science/estimate-the-favorite-scraping-tweets-using-python-863303384e29?source=rss-98cd7d3f47d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/863303384e29</guid>
            <category><![CDATA[football]]></category>
            <category><![CDATA[world-cup]]></category>
            <category><![CDATA[twitter]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Prasad Patil]]></dc:creator>
            <pubDate>Fri, 15 Jun 2018 12:33:27 GMT</pubDate>
            <atom:updated>2018-06-19T04:00:28.890Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6y7fhU5TD8yyri81t-oWUQ.jpeg" /><figcaption><strong>source</strong>: <a href="http://blog.cpdfootball.de/2017/12/01/fifa-world-cup-russia-2018-final-draw-conducted-in-moscow/">blog.cpdfootball.de</a></figcaption></figure><blockquote>Job of a Footballer and a Data scientist exhibit identical work-life set up which is <strong><em>combination of skill and will</em></strong> .</blockquote><p>On the eve of 14th June,an acclaimed religious festival of all football followers — <strong>FIFA World Cup 2018 </strong>began in Russia.This month long prestigious sports bonanza will be celebrated across the globe till it’s mega finale scheduled on 15th July.</p><p>Seizing this opportunity, I tested my newly found skills to gauge who will be a crowd favorite among the two competing teams in a given fixture.</p><blockquote><strong><em>Things you’ll learn here</em></strong> :</blockquote><ol><li>How to scrape data from Twitter?</li><li>Data Mining</li></ol><blockquote><strong>Prerequisites :</strong></blockquote><ol><li>Tweepy library should be installed on your machine.</li></ol><p>2. Access to Twitter. Make an account if you don’t have already.</p><p>Scraping textual data is an integral part of natural language processing.Twitter is very utilitarian platform when task in hand is analyzing sentiments.</p><blockquote><strong>Thought Process :</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*KmifLlxjHP8c3PUXcMYNPw.png" /><figcaption><strong>source</strong>: <a href="https://nocodewebscraping.com/twitter-json-examples/">www.nocodewebscraping.com</a></figcaption></figure><p>This is how the our approach will be — firstly, we will generate desired credentials to make use of Twitter API, then we will write a python code to extract live tweets, and in the end we will analyze the tweets by selecting pertinent keywords.</p><p>Let the fun begin…</p><blockquote><strong><em>Step-wise Guide for Scraping Tweets</em></strong></blockquote><h4>A. Generating Credentials for Twitter API :</h4><p><strong>I</strong>. Visit <a href="https://apps.twitter.com/"><strong>https://apps.twitter.com/</strong></a><strong> </strong>and log in with your twitter credentials.</p><p><strong>II. </strong>Click on “<strong>Create New App</strong>”</p><p><strong>III</strong>. Enter necessary details and click “<strong>Create your Twitter application</strong>”</p><p><strong>IV</strong>. On next page, click on “<strong>API keys</strong>” tab, and copy your “API key” and “<strong>API secret</strong>”. (Suggestion: Do make a sticky note on desktop of these credentials)</p><p><strong>V</strong>. Scroll down and click “<strong>Create my access token</strong>”, and copy your “Access token” and “<strong>Access token secret</strong>”.</p><h4>B. Connecting to Twitter API through Python code :</h4><p>Below is the code you need in order to connect to live streaming of tweets on Twitter and download them on your machine.</p><ul><li>Note: Python 2.7 version used while coding.</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2f25881410fe1cb129ea96e5170aec6e/href">https://medium.com/media/2f25881410fe1cb129ea96e5170aec6e/href</a></iframe><p>Once you execute the code on command prompt (for windows users), you’ll witness following data flow.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*lB8qHUd0d4VQpeinM9X3aA.gif" /><figcaption><strong>Live Streaming Tweets</strong></figcaption></figure><h4>C. Analyzing Data :</h4><p>Type below line to save downloaded tweets on command line.</p><blockquote>python twitter_streaming.py &gt; twitter_data.txt</blockquote><p>To understand the direction of the polarity of the tournament opener, I ran the program for 2 days in regular intervals (on 13th and 14th June) to get a sizable and meaningful data sample. I could scrape text consisting 79,359 tweets of size is 450 MB in above mentioned time period.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fq0HQTvWpNkTeemjyK4PTg.png" /></figure><p>You can extract as many insights from the raw text as the business case demands. For example.Here is the one -Languages in which tweet was written.Find Jupyter notebook for this exercise in the bottom section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/438/1*zhmxwVDuTalWccRj45GHjw.jpeg" /></figure><p>Since first encounter was between host nation Russia and opponent was Saudi Arabia I selected following keywords.</p><ul><li>FIFA,World,Cup,football,FIFA World Cup,#FIFA2018</li><li>WorldCup,#WorldCup2018,#FifaWorldCup,</li><li>#FIFAWorldCup ,RUSKSA,#RUSKSA,prediction,win,#FIFA.</li></ul><p>To estimate which one among these two is the favorite, targeted tweets with #RUS ,#KSA and #RUSKSA and segregated the result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/432/1*r32-3LdCFLXC7FNm9x-NzA.jpeg" /><figcaption><strong>Final Outcome</strong></figcaption></figure><p>And surprisingly match result also turned out to be siding with popular public sentiment. As Russians thrashed lackluster Saudis !</p><blockquote>Russia 5 — 0 Saudi Arabia</blockquote><blockquote>What a one sided affair !</blockquote><p>So what are you waiting for, there are 63 matches more to go. Have fun !! Down below is the infographic of teams participating in this edition of FIFA World Cup. <a href="https://www.fifa.com/worldcup/matches/"><strong>Here</strong></a> you can find all fixtures of matches.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uIXAC4VyDh2-tlkRu6dC3g.jpeg" /><figcaption><strong>FIFA WC 2018 Qualified Teams</strong> . source: <a href="https://www.ivacy.com/blog/watch-fifa-world-cup-2018-online-free/">www.Ivacy.com</a></figcaption></figure><blockquote><strong>Jupyter Notebook</strong> :</blockquote><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eadb2f32b41fe2b335b911f153ca38ea/href">https://medium.com/media/eadb2f32b41fe2b335b911f153ca38ea/href</a></iframe><p>Hopefully you enjoyed this post.So shed your love and affection, it will be estimated on the basis of number of times you hit the clap icon . <strong>#SeeYouNextTime #Till Then</strong> <strong>#KeepLearningNewThings</strong></p><blockquote><strong>References</strong>:</blockquote><ol><li><a href="http://docs.tweepy.org/en/v3.5.0/">Tweepy Documentation</a></li><li><a href="https://www.youtube.com/watch?time_continue=7&amp;v=bmdM-sezWGg">Twitter Streaming API — Python &amp; Code!</a></li></ol><blockquote><strong>More Reading :</strong></blockquote><p>If you liked my post then spare few minutes for my other blog posts too -</p><ol><li><a href="https://towardsdatascience.com/exploratory-data-analysis-8fc1cb20fd15"><strong>What is Exploratory Data Analysis?</strong></a></li><li><a href="https://towardsdatascience.com/k-means-clustering-identifying-f-r-i-e-n-d-s-in-the-world-of-strangers-695537505d"><strong>K Means Clustering : Identifying F.R.I.E.N.D.S in the World of Strangers</strong></a></li><li><a href="https://towardsdatascience.com/decision-tree-data-scientists-magic-bullet-for-hamletian-dilemma-411e0121ba1e"><strong>Decision Tree — Data Scientist’s magic bullet for Hamletian Dilemma</strong></a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=863303384e29" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/estimate-the-favorite-scraping-tweets-using-python-863303384e29">Finding the favorite team in 2018 FIFA World Cup through scraping Tweets</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Decision Tree - Data Scientist’s magic bullet for Hamletian Dilemma]]></title>
            <link>https://medium.com/data-science/decision-tree-data-scientists-magic-bullet-for-hamletian-dilemma-411e0121ba1e?source=rss-98cd7d3f47d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/411e0121ba1e</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[decision-tree]]></category>
            <dc:creator><![CDATA[Prasad Patil]]></dc:creator>
            <pubDate>Sat, 02 Jun 2018 03:00:03 GMT</pubDate>
            <atom:updated>2022-05-30T07:41:39.810Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kz2PJytwvCOfLshlCHejiw.jpeg" /><figcaption>source: my sketch book</figcaption></figure><p>Decision trees belongs to the family of supervised machine learning<strong> </strong>algorithms and are considered to be a <em>panacea </em>of all data science problems. Data scientist often caught making witty remarks such as, “whenever problem statement puts you in a hamletian dilemma and you can’t think of any algorithm (irrespective of situation), use decision tree!”.</p><p>More often than not ,be it in industry or be it in kaggle competitions, it is seen that decision tree or at least the algorithms that are evolved from it (<a href="https://analyticsindiamag.com/primer-ensemble-learning-bagging-boosting/"><em>Bagging,Boosting ensemble</em></a>), are practiced religiously.</p><p>Decision tree is a versatile machine learning method capable of performing both regression and classification tasks. Almost all real world problems are non-linear in nature and decision trees helps you get rid of non-linearity in the data. This algorithm is very intuitive,simple to understand and can be visually interpretable — something every business would want in first place.</p><blockquote>What else one can desire from a model? Simply Magical ✨</blockquote><p>Decision trees are drawn upside down, meaning root is at the top and the leaves are at the bottom.Primarily decision tree believes in divide &amp; Conquer rule.</p><h3>Basic Terminology</h3><p>Let’s look at the basic terminology used with Decision trees:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/631/1*xYM6q2y9gJctqIudP9NiFw.png" /></figure><ol><li><strong>Root Node: </strong>It represents entire population or sample and this further gets divided into two or more homogeneous sets.</li><li><strong>Splitting: </strong>It is a process of dividing a node into two or more sub-nodes.</li><li><strong>Decision Node: </strong>When a sub-node splits into further sub-nodes, then it is called decision node.</li><li><strong>Leaf/ Terminal Node: </strong>Nodes which do not split further are called Leaf or Terminal nodes.</li><li><strong>Pruning: </strong>When we remove sub-nodes of a decision node, this process is called pruning.</li><li><strong>Branch / Sub-Tree: </strong>A sub section of entire tree is called branch or sub-tree.</li><li><strong>Parent and Child Node: </strong>A node, which is divided into sub-nodes is called parent node of sub-nodes where as sub-nodes are the child of parent node.</li></ol><h3>Intuition</h3><p>There are two type of Decision trees :</p><p><strong>A</strong>.<em>Classification Decision Trees</em> &amp; <strong>B</strong>.<em>Regression Decision Trees</em></p><ul><li>Classification trees helps you classify your data and hence they work on categorical data for example loan status (approved/not approved),spam/not spam etc.</li><li>Regression trees are designed to help you predict outcome which can be a real number for example income of a person, sale price of a house etc.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/636/1*zgeToKkVqDTnqRo0N5oKJg.gif" /><figcaption>source: algobeans.com — Classification Tree illustration</figcaption></figure><p>Suppose we have two features X and Y and in right panel you can observe there are several data points scattered.Green and grey leafs are two classes in dependent variable. So what basically decision tree does is,it cuts entire data set into slices in several iterations. As shown, there is a split 1 at X = 0.5 , split 2 at Y =0.5 and split 3 at X = 0.25.</p><p>Splits are orchestrated to maximize the number of certain category in each of these splits, meaning decision trees tries to accomplish homogeneous distribution at each node.From right panel , you can notice separation of green leaf category and grey leaf category forms homogeneous structure in the end in every compartment.</p><h3>Math behind the algorithm</h3><p>There are various methods adopted by decision tree for splitting nodes. Most commonly followed are - Gini Index,Entropy,Chi-square etc.</p><p><strong>I</strong>. <strong>Gini Index</strong></p><ul><li>According to it if we select two items from a population at random then they must be of same class and probability is 1 if population is pure.</li><li>It is a measure of impurity. Therefore,lower the value of Gini index, higher is the homogeneity.</li><li>Mathematically represented as</li></ul><blockquote>Gini Index = 1-[(P)²+(1-P)²]</blockquote><p>where P is proportion of positive samples in that node.</p><ul><li>Gini Index of ‘0’ indicates that node is pure.Thus it implies no further splitting is needed.</li><li>Steps involved -</li></ul><blockquote>Calculate Gini for sub-nodes, using formula.</blockquote><blockquote>Calculate Gini for split using weighted Gini score of each node of that split</blockquote><p><strong>II</strong>. <strong>Chi Square</strong></p><ul><li>​It helps to find out the statistical significance between the differences among sub-nodes and parent node.</li><li>We measure it by sum of squares of standardized differences between observed and expected frequencies of target variable.</li><li>Mathematically represented as</li></ul><blockquote>Chi-square = ((Actual — Expected)² / Expected)¹/2</blockquote><ul><li>It is a measure of purity. Therefore,higher the value of Chi-square, higher is the statistical significance of differences between sub-node and Parent node.</li><li>Steps involved -</li></ul><blockquote>Calculate Chi-square for individual node by calculating the deviation for Success and Failure both</blockquote><blockquote>Calculated Chi-square of Split using Sum of all Chi-square of success and Failure of each node of the split</blockquote><p><strong>III</strong>. <strong>Entropy</strong></p><ul><li>It is a measure of the randomness in the information being processed.</li><li>Higher the <strong>entropy</strong>, the harder it is to draw any conclusions from that information.</li><li>Mathematically represented as</li></ul><blockquote>Entropy = -p*log(p) - q*log(q)</blockquote><ul><li>Here p and q is probability of success and failure respectively in that node.</li><li>Log is taken at base “2”.</li><li>Steps involved -</li></ul><blockquote>Calculate entropy of parent node</blockquote><blockquote>Calculate entropy of each individual node of split and calculate weighted average of all sub-nodes available in split.</blockquote><p><strong>IV</strong>.<strong>Reduction in Variance</strong></p><ul><li>All the methods mentioned above are pertaining to classification decision trees. In case of regression decision tree where target variable is continuous, reduction in variance method is followed.</li><li>It uses the standard formula of variance to choose the best split. The split with lower variance is selected as the criteria to split the population.</li><li>Mathematically represented as</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/207/1*OhcMuT8b1SZoiDEfFbKnow.png" /><figcaption>Where, X -&gt; actual values, X −&gt; mean and N -&gt; number of observations</figcaption></figure><ul><li>Steps involved -</li></ul><blockquote>Calculate variance for each node.</blockquote><blockquote>Calculate variance for each split as weighted average of each node variance.</blockquote><p>Lets take a discrete example and grip firmly on the topic</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2a3e557c86430e379d2acb37ed535692/href">https://medium.com/media/2a3e557c86430e379d2acb37ed535692/href</a></iframe><h3>Visualization</h3><p>As claimed earlier, decision trees are interpretable and can be visualized.<em> Graphviz</em> library come to rescue data scientists while working on python. <a href="https://inteligencia-analitica.com/wp-content/uploads/2017/08/Installing-Graphviz-and-pydotplus.pdf">HERE</a><strong> </strong>is the guide to download Graphviz <strong>.</strong></p><p>I have attempted <a href="https://archive.ics.uci.edu/ml/datasets/iris">IRIS data set</a> for visualizing decision tree and diagram displayed below obtained as a result.This is achieved using a code where you don’t even have to download Graphviz on your machine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1011/1*ghXpLgWpBKnyy-dbCUFOLA.png" /><figcaption>Find Hands-On-Code for visualizing decision tree<strong> </strong><a href="http://nbviewer.jupyter.org/github/PBPatil/Decision_Trees/blob/master/visualizing_trees.ipynb"><strong>HERE</strong></a></figcaption></figure><h3>What’s the catch?</h3><p>Well, our magic bullet certainly sounds life savior but it is too good to be true,isn’t it?! The biggest peril while modeling decision trees is that their propensity to overfitting. If there is no limit set on a decision tree, it will give you 100% accuracy on training set because in the worse case it will end up making 1 terminal node for each observation.</p><p>Thus, preventing overfitting is pivotal while modeling a decision tree and it can be done in 2 ways:</p><p><strong><em>A. </em></strong><em>Setting constraints on tree size(</em>Hyper-parameter Tuning<em>) &amp; </em><strong><em>B. </em></strong><a href="http://www.saedsayad.com/decision_tree_overfitting.htm"><em>Tree pruning</em></a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*FPunxzsA9-4CH6y4uZr5AA.png" /><figcaption>Illustration of different Hyper-parameters used in Decision Tree</figcaption></figure><h3>Further Reading</h3><p>Do refer below links for deeper understanding of the concept</p><ol><li><a href="https://www.analyticsvidhya.com/blog/2016/04/complete-tutorial-tree-based-modeling-scratch-in-python/">Analytics vidhya</a></li><li>Scikit <a href="http://scikit-learn.org/stable/modules/tree.html">Documentation</a></li><li><a href="https://stackoverflow.com/questions/1859554/what-is-entropy-and-information-gain">Stack-overflow</a></li></ol><p><em>Congratulations!!</em> on mastering one of the age old algorithms in machine learning :)</p><blockquote><strong><em>Other Publushed Articles:</em></strong></blockquote><p>If you liked my post then spare few minutes for my other blog posts too -</p><ol><li><a href="https://towardsdatascience.com/exploratory-data-analysis-8fc1cb20fd15">What is Exploratory Data Analysis?</a></li><li><a href="https://medium.com/@theprasadpatil/last-minute-revision-part-i-machine-learning-statistics-8de23a377987">Last Minute Revision: Part I — Machine Learning &amp; Statistics</a></li><li><a href="https://medium.com/@theprasadpatil/how-to-create-a-pdf-report-from-excel-using-python-b882c725fcf6?source=your_stories_page----------------------------------------">How to create a PDF report from Excel using Python</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=411e0121ba1e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/decision-tree-data-scientists-magic-bullet-for-hamletian-dilemma-411e0121ba1e">Decision Tree - Data Scientist’s magic bullet for Hamletian Dilemma</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[K Means Clustering : Identifying F.R.I.E.N.D.S in the World of Strangers]]></title>
            <link>https://medium.com/data-science/k-means-clustering-identifying-f-r-i-e-n-d-s-in-the-world-of-strangers-695537505d?source=rss-98cd7d3f47d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/695537505d</guid>
            <category><![CDATA[k-means]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[unsupervised-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[clustering]]></category>
            <dc:creator><![CDATA[Prasad Patil]]></dc:creator>
            <pubDate>Sun, 20 May 2018 14:55:07 GMT</pubDate>
            <atom:updated>2022-05-30T07:40:41.112Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/906/1*Aczgwp5UkCIIO7nf9ItLiA.png" /></figure><p><strong><em>Quickie Quiz Time</em></strong><em>:</em> <em>what is it that defines someone as someone’s friend? How do you spot that somebody in the clutter? How do you make friends?</em></p><p>Well, most common answers are,there has to be some basic level tuning ,alliance of opinions,union of thoughts and liking needed between two individuals to form a bond of friendship. Minimum differences would lead to closeness and stronger unbreakable loyal bond. There are gazillion ways to make friends,differs from person-to-person |&amp; situations-to-situations.</p><ul><li>That’s It!! That’s all about it…! Given above context,It will be easier for you to relate and understand somewhat similar analogy presented on a silver platter right next.Without further ado lets dive straight in to the topic of today’s utmost valued discussion -<strong><em>‘K-means Clustering’</em></strong>.</li></ul><p>I know this sounds a bit like a heavy data science jargon with <em>K-means</em>?! and <em>Clustering</em>?!in it but no need to panic as it’s just another fancy term from DScientist’s back pocket. So to understand the intuition behind it,let’s just “<em>.split()</em>” the term,try to “.<em>describe()</em>” each token and the questions arising with it.</p><p>Shall we begin..?!..A louder Yess will help..!!..Yesss!..Cool…Thrusting Pedal!</p><blockquote><strong>What is Clustering?</strong>-It is an unsupervised machine learning technique of dividing data into distinct groups such that observations within each group are similar.Here every group is termed as cluster.</blockquote><blockquote><strong>What is Unsupervised Machine Learning?</strong>-Type of machine learning where output is not known.We try to learn relationships,structure and underlying pattern from the given input data.</blockquote><blockquote><strong>What is K?</strong>-It is the number of how many clusters you want to form from the data.The ‘-means’ part of it will be explained in greater details further ahead.</blockquote><blockquote><strong>So what is K-means Clustering finally?</strong> -It is an unsupervised machine learning algorithm that will attempt to group similar clusters together in your data.</blockquote><p>Now that we got the basic idea , lets get into the nitty-gritty of the topic.</p><h3>K-means Algorithm:</h3><p>Research says that graphical information is easier to remember than textual information.So let’s learn with pictures as they have better <em>Recall</em>!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tWaaZX75oumVwBMcKN-eHA.png" /></figure><p>All right here we’ve got a scatter plot.Say we have two variables plotted along X-axis and Y-axis from our data set (before K-means graph).Question is can we identify certain groups among our variables and how would we go about doing it.How do we identify number of groups?</p><p>So what the K-means does for you is that it takes the complexity from this decision making process and allows you to easily identify those clusters of data points from data set(after K-means graph).There you go, we got three clusters(red,blue,green).It was a very simplified example where we only have two variables here, so two dimensions but note that K-means can work on multi-dimensional data.</p><h3>How it works:</h3><p>I will break it down in simple step-by-step format.</p><p><strong>STEP 1</strong>:Choose the number K of clusters- We will see how to choose optimum number of clusters further down in section.But for now we assume K of our choice.It could be any number(3,5 or 2 etc).</p><p><strong>STEP 2</strong>: Select at random K points, the centroids(not necessarily from your dataset)- Centroid is the central point of cluster.You can select any two points from that first scatter plot, they can be any random X and Y values provided your selected the number of centroids equate with number of clusters picked in step 1.</p><p><strong>STEP 3</strong>: Assign each data point to the closest centroid &gt;That forms K Cluster — Now understand closest is the vague term here.It depends on what kind of distance you’re measuring which will be defined by business problem.Most commonly followed is Euclidean distances. When data has comprised large values Manhattan distances would be a good option and if data set is full of categorical variables then preferred choice is Cosine distance.</p><p><strong>STEP 4</strong>: Compute and place the new centroid of each cluster.- This will be more clear when we study an example.</p><p><strong>STEP 5</strong>: Reassign each data point to the new closest centroid. If any reassignment took place,go to STEP 4, otherwise stop — Clearly it as an iterative process of step 4 and 5.</p><p>I know this might seem bit complex but let’s go through a visual example so that we understand this on very intuitive level and later on you can refer these steps to unearth whats happening in the background.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*zdwKwqL0M0zUSMvt4tVQzg.png" /><figcaption>STEP 1: Choose Number K of Clusters: K=2</figcaption></figure><p>Here, we have the observations in our dataset. They’re plotted against two variables and right away first question is can you visually identify the final clusters that we will end up with. That’s pretty tough isn’t it! That is just for two variables, imagine what challenge would be if we had three variables and five variables. We won’t be able to plot a five dimensional scatter plot like this.So that’s when K-means algorithm comes in the picture and simplifies the process.Now let’s see how it proceed.In this case we’re manually perform the K-means clustering algorithm and see exactly how it works.Something that would normally be done by tools like python, R etc</p><p>Alright,say we somehow identified optimal number of clusters is equal to 2. Again we will discuss how to find optimal number of clusters further down but at the moment lets agree it just 2 clusters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*t23vcjiA0cLnY8Vt0Udz3w.png" /><figcaption>STEP 2: Select at random K points,the centroids(not necessarily from your dataset)</figcaption></figure><p>In this step, we selected two random points outside of our dataset, Blue square and Red Square as our centroids.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/994/1*1GMesCbiatxGlHY8s8couA.png" /><figcaption><strong>STEP 3</strong>: Assign each data point to the closest centroid &gt;That forms K Cluster. (Fig.1b)</figcaption></figure><p>For step 3, basically we have to check that for every data point in our dataset,identify which of the two centroids(Blue or Red) is the closest.This we will determine by refreshing our geometry knowledge.What we’re going to do is we connect those centroids with straight line and bisect that line with a perpendicular(marked in dotted green).Any point on green line is equidistant from both centroids. Points lying above the line belongs to blue cluster centroid and likewise points lying below the line belongs to red cluster centroid. For simplicity purpose points of clusters that they belong will be painted in same colors.Before we continue further again I would like to mention “closest” is an ambiguous term. Because closest when you visualize things on scatter plot is pretty straight forward implies that the distance we’re looking at.From scatter plot we can very well conclude that point x is closer to cluster A and point y is closer to cluster B.<em>(Type of Distance measures:</em><a href="https://en.wikipedia.org/wiki/Euclidean_distance"><em> Euclidean</em></a><em> ,</em><a href="https://www.quora.com/What-is-the-difference-between-Manhattan-and-Euclidean-distance-measures"><em>Manhattan</em></a><em>,</em><a href="https://en.wikipedia.org/wiki/Cosine_similarity"><em>Cosine</em></a><em> etc.)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/890/1*3zXxvE6U5SX4QBY1DDpe4Q.png" /><figcaption><strong>STEP 4</strong>: Compute and place the new centroid of each cluster.(Fig.1a)</figcaption></figure><p>Well keeping that in mind, proceeding ahead into step 4.Imagine every data point has certain weight. So you need to find the center of mass and you need to pinpoint it on your scatter plot(Shown in dotted squares).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/957/1*n87ePQB1WOlbpNfox3p8bg.png" /><figcaption><strong>STEP 5</strong>: Reassign each data point to the new closest centroid. If any reassignment took place,go to STEP 4, otherwise stop.(Fig.1a)</figcaption></figure><p>It is basically achieved by calculating mean of data points belonging particular cluster. For our example lets look at blue cluster points , take y-coordinates and x-coordinates of each,average them out respectively and plot the mean point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/893/1*bs29ECGGsm1dqL6lLYueow.png" /><figcaption><strong>STEP 5</strong>: Reassign each data point to the new closest centroid. If any reassignment took place,go to STEP 4, otherwise stop.(Fig.1b)</figcaption></figure><p>Great! Now we entered in step 5. After plotting green line, we can see three points are misclassified because of previous membership.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/851/1*OXpQcdV5YNIOfk3yuELN8w.png" /><figcaption><strong>STEP 5</strong>: Reassign each data point to the new closest centroid. If any reassignment took place,go to STEP 4, otherwise stop.(Fig.1c)</figcaption></figure><p>So we will shift them and recolor them according to centroid of the cluster.Since reassignment data points took place we will go back to step 4.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/860/1*n4dNJZohfNqIMbZ9PSfzkw.png" /><figcaption><strong>STEP 4</strong>: Compute and place the new centroid of each cluster.(Fig. 2a)</figcaption></figure><p>Based on reassignment, we recompute the centroid and place it for each cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/861/1*lbeWmB2vxhIILI0gH7bOrA.png" /><figcaption><strong>STEP 4</strong>: Compute and place the new centroid of each cluster.(Fig. 2b)</figcaption></figure><p>Identifying new center of mass of each cluster shown in dotted squares and moved the centroids at those locations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/857/1*UJQjXO5duWimvjJwyFhWEQ.png" /><figcaption><strong>STEP 5</strong>: Reassign each data point to the new closest centroid. If any reassignment took place,go to STEP 4, otherwise stop.(Fig.2b)</figcaption></figure><p>Repeating Step 5.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*8eX1-AzxGQ7LhdmztS5vdg.png" /><figcaption><strong>STEP 5</strong>: Reassign each data point to the new closest centroid. If any reassignment took place,go to STEP 4, otherwise stop.(Fig.2a)</figcaption></figure><p>Noticing the red data point above the green line(seen in Fig.2a of step 5) is being reassigned to blue cluster.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/851/1*aGg3Uwhuz5FzGCZtYoDxjw.png" /><figcaption><strong>STEP 4</strong>: Compute and place the new centroid of each cluster.(Fig. 3a)</figcaption></figure><p>Repeating step 4.As we can see it’s an iterative process. We keep on doing this until algorithm converges.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*tlzoDxuJItMrOgT0xLe3Eg.png" /><figcaption><strong>STEP 4</strong>: Compute and place the new centroid of each cluster.(Fig. 3b)</figcaption></figure><p>Step 4 on third iteration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/851/1*h3LqGaS_-xrOcERF0_Lw4g.png" /><figcaption><strong>STEP 5</strong>: Reassign each data point to the new closest centroid. If any reassignment took place,go to STEP 4, otherwise stop.(Fig.3a)</figcaption></figure><p>Step 5 third iteration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/857/1*PYerIixMoAoffDB4flmvBg.png" /><figcaption><strong>STEP 5</strong>: Reassign each data point to the new closest centroid. If any reassignment took place,go to STEP 4, otherwise stop.(Fig.3b)</figcaption></figure><p>There you see again the equidistant green line.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*ymSICYbXAKKUJ0hMPCqxtg.png" /><figcaption><strong>HSTEP 4</strong>: Compute and place the new centroid of each cluster.(Fig. 4a)</figcaption></figure><p>Step 4 on fourth iteration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/1*Wn8F_Zf8Dv_2kb9go3YWOQ.png" /><figcaption><strong>STEP 4</strong>: Compute and place the new centroid of each cluster.(Fig. 5a)</figcaption></figure><p>Reassigning each data point to the new closest centroid in step 5.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/1*BmVfvx4AB3y_d8s309r2uw.png" /><figcaption><strong>STEP 5</strong>: Reassign each data point to the new closest centroid. If any reassignment took place,go to STEP 4, otherwise stop.(Fig.4a)</figcaption></figure><p>Step 4 on fifth iteration.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/1*3O4L6Kdui6CxNHteksVxeQ.png" /><figcaption><strong>STEP 5</strong>: Reassign each data point to the new closest centroid. If any reassignment took place,go to STEP 4, otherwise stop.(Fig.5a)</figcaption></figure><p>We are very close to the final memberships of our clusters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*g_S_0_pTEbS6QDsSLGNaGg.png" /><figcaption><strong>STEP 4</strong>: Compute and place the new centroid of each cluster.(Fig. 6a)</figcaption></figure><p>Repeating step 4 for the sixth time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/1*kKdgS1zV-PdOUh7T7oAM-Q.png" /><figcaption><strong>STEP 4</strong>: Compute and place the new centroid of each cluster.(Fig. 6b)</figcaption></figure><p>Computed and placed the new centroids in each clusters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/814/1*pXGVFmogNK2nfN3E7E2IoQ.png" /><figcaption><strong>STEP 5</strong>: Reassign each data point to the new closest centroid. If any reassignment took place,go to STEP 4, otherwise stop.(Fig.6a)</figcaption></figure><p>Now that when we ran step 5 for the sixth time,please appreciate that there is no change in cluster memberships.Compare Fig.5a and Fig.6a of step 5. Hence we can safely conclude our K-means algorithm has converged and final model is ready.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/724/1*gdqxguk2JbstxWWXdLhIMQ.png" /><figcaption>Finally !!!</figcaption></figure><p>These are the two clusters we’ve got suggested by K-means algorithm.Hopefully I have successfully demystified the algorithm in simple terms.</p><blockquote><a href="http://nbviewer.jupyter.org/github/PBPatil/K-means-Clustering/blob/master/K-means_clustering_1.ipynb"><strong>Here</strong></a> is hands on example of K-means clustering attempted on Iris dataset.</blockquote><h3>Choosing optimal K value -</h3><p>Now one severe drawback of K-means clustering , value of K.</p><p>Now in order to understand,if for instance 2 clusters would have been better in the above scenario or may be 10 clusters would have been better.We need certain metric.We need way to evaluate how a certain number of clusters performs better comparatively different set of clusters and preferably that metric should be quantifiable. So what kind of metric we impose upon our clustering algorithm,that will tell us something about final result?</p><p>And remedy to it is ‘<strong>Within Cluster Variation</strong>’.</p><p>Within Cluster Variation is mathematically represented as Sum Of Squared Errors(WCSS/SSE).It also goes by the name of “Elbow method”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KZ-V9Hj3UjS1ElaaiJ8X5Q.png" /></figure><p>Now at first glance formula bit seem overwhelming but in reality it is super simple. In line with our mission,making complex simple let’s break it all down.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/1*8VBj2ezTT6TBX8H8jWbe_Q.png" /></figure><p>Basically what we’re doing here is …</p><p>Every data point in cluster 1 (Pi in Cluster 1),Measuring difference between Pi data point and C1 centroid of cluster 1 {Distance(Pi,C1)} and summing them up (the big “E” sign is called sigma which denotes summation)</p><p>same goes with other clusters.</p><p>If you plot K against the SSE,you will see that the error decreases as K gets larger; this is because when the number of clusters increases,they should be smaller, so distortion is also smaller.</p><p>The idea of elbow method is to choose the K at which the SSE decreases abruptly.</p><p>This produces an “Elbow effect” in graph.</p><blockquote>Find hands on code with a mall dataset <a href="http://nbviewer.jupyter.org/github/PBPatil/K-means-Clustering/blob/master/K-means_clustering_2.ipynb"><strong>HERE</strong></a><strong> </strong>where I have touched concept of Elbow method and few other concepts.</blockquote><p><strong>References</strong>:</p><ul><li><a href="https://www.udemy.com/machinelearning/">Machine Learning A-Z (Python &amp; R in Data Science Course)</a></li><li><a href="https://www.udemy.com/python-for-data-science-and-machine-learning-bootcamp">Learn Python for Data Structures, Algorithms &amp; Interviews</a></li><li><a href="http://www-bcf.usc.edu/~gareth/ISL/">Introduction to Statistical Learning</a></li></ul><p><em>How indecisive are you,on a scale of 1 to Hamlet? Well, there is one subtle way to kill the life troubling dilemmas ,we will see </em><strong>when we meet next time</strong> !</p><p><strong>Read Other Articles I Published </strong>-</p><ol><li><a href="https://medium.com/@theprasadpatil/how-to-create-a-pdf-report-from-excel-using-python-b882c725fcf6">How to create a PDF report from Excel using Python</a></li><li><a href="https://towardsdatascience.com/exploratory-data-analysis-8fc1cb20fd15">What is Exploratory Data Analysis?</a></li><li><a href="https://medium.com/@theprasadpatil/last-minute-revision-part-i-machine-learning-statistics-8de23a377987">Last Minute Revision: Part I — Machine Learning &amp; Statistics</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=695537505d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/k-means-clustering-identifying-f-r-i-e-n-d-s-in-the-world-of-strangers-695537505d">K Means Clustering : Identifying F.R.I.E.N.D.S in the World of Strangers</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What is Exploratory Data Analysis?]]></title>
            <link>https://medium.com/data-science/exploratory-data-analysis-8fc1cb20fd15?source=rss-98cd7d3f47d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/8fc1cb20fd15</guid>
            <category><![CDATA[exploratory-data-analysis]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[pipeline]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-visualization]]></category>
            <dc:creator><![CDATA[Prasad Patil]]></dc:creator>
            <pubDate>Fri, 23 Mar 2018 16:37:45 GMT</pubDate>
            <atom:updated>2022-05-30T07:39:44.390Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*mW9PgNjfWxHhbve3O2siNA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*id89S3bN6pKX-ZHsns5v3Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*yY5jSLXWZ4d2H6tVtKRSGg.png" /></figure><p>As I was contemplating what could be the maiden topic I should begin writing my blog with, in no time EDA popped up to my mind.Logically apt , isn’t it ?! Why? You’ll find out soon!</p><h3>Intuition</h3><p>Imagine your wolf pack decides to watch a movie you haven’t heard of.There is absolutely no debate about that,it will lead to a state where you find yourself puzzled with lot of questions which needs to be answered in order to make a decision.Being a good chieftain the first question you would ask, what is the cast and crew of the movie?As a regular practice,you would also watch the trailer of the movie on YouTube.Furthermore,you’d find out ratings and reviews the movie has received from the audience.</p><p>Whatever investigating measures you would take before finally buying popcorn for your clan in theater,is nothing but what data scientists in their lingo call ‘Exploratory Data Analysis’.</p><blockquote>Exploratory Data Analysis refers to the critical process of performing initial investigations on data so as to discover patterns,to spot anomalies,to test hypothesis and to check assumptions with the help of summary statistics and graphical representations.</blockquote><p>It is a good practice to understand the data first and try to gather as many insights from it. EDA is all about making sense of data in hand,before getting them dirty with it.</p><h3><strong>EDA explained using sample Data set:</strong></h3><p>To share my understanding of the concept and techniques I know,I’ll take an example of white variant of <a href="https://archive.ics.uci.edu/ml/datasets/wine+quality">Wine Quality data set</a> which is available on UCI Machine Learning Repository and try to catch hold of as many insights from the data set using EDA.</p><p>To starts with,I imported necessary libraries (for this example pandas, numpy,matplotlib and seaborn) and loaded the data set.</p><p>Note : Whatever inferences I could extract,I’ve mentioned with bullet points.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8KJeO0SOlhyxlvXkjbvjlA.png" /></figure><ul><li>Original data is separated by delimiter “ ; “ in given data set.</li><li>To take a closer look at the data took help of “ .head()”function of pandas library which returns first five observations of the data set.Similarly “.tail()” returns last five observations of the data set.</li></ul><p>I found out the total number of rows and columns in the data set using “.shape”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5c02Q6iy2XMghMaojNXiqQ.png" /></figure><ul><li>Dataset comprises of 4898 observations and 12 characteristics.</li><li>Out of which one is dependent variable and rest 11 are independent variables — physico-chemical characteristics.</li></ul><p>It is also a good practice to know the columns and their corresponding data types,along with finding whether they contain null values or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7cSVoZ4OjozPG9vo-yjB-w.png" /></figure><ul><li>Data has only float and integer values.</li><li>No variable column has null/missing values.</li></ul><p>The describe() function in pandas is very handy in getting various summary statistics.This function returns the count, mean, standard deviation, minimum and maximum values and the quantiles of the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*79E_z0OZsqFExkRHm_OM5w.png" /></figure><ul><li>Here as you can notice mean value is less than median value of each column which is represented by 50%(50th percentile) in index column.</li><li>There is notably a large difference between 75th %tile and max values of predictors “residual sugar”,”free sulfur dioxide”,”total sulfur dioxide”.</li><li>Thus observations 1 and 2 suggests that there are extreme values-Outliers in our data set.</li></ul><p>Few key insights just by looking at dependent variable are as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8ElyezKY4WMBiROaH-Jkxg.png" /></figure><ul><li>Target variable/Dependent variable is discrete and categorical in nature.</li><li>“quality” score scale ranges from 1 to 10;where 1 being poor and 10 being the best.</li><li>1,2 &amp; 10 Quality ratings are not given by any observation. Only scores obtained are between 3 to 9.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wfZX4av3BPx_DWNn977mfw.png" /></figure><ul><li>This tells us vote count of each quality score in descending order.</li><li>“quality” has most values concentrated in the categories 5, 6 and 7.</li><li>Only a few observations made for the categories 3 &amp; 9.</li></ul><p>I got a good a glimpse of data. But that’s the thing with Data Science the more you get involved the harder it is for you to stop exploring.Let’s now explore data with beautiful graphs. Python has a visualization library ,<a href="https://seaborn.pydata.org/">Seaborn</a> which build on top of matplotlib. It provides very attractive statistical graphs in order to perform both <a href="http://www.statisticshowto.com/univariate/">Univariate</a> and <a href="http://www.camo.com/multivariate_analysis.html">Multivariate analysis</a>.</p><p>To use linear regression for modelling,its necessary to remove correlated variables to improve your model.One can find correlations using pandas “.corr()” function and can visualize the correlation matrix using a heatmap in seaborn.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YUjtFOKH4TtjP3Ah67GcjA.jpeg" /><figcaption><strong>Heatmap</strong></figcaption></figure><ul><li>Dark shades represents positive correlation while lighter shades represents negative correlation.</li><li>If you set annot=True, you’ll get values by which features are correlated to each other in grid-cells.</li></ul><p>It’s a good practice to remove correlated variables during feature selection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ObHerhSsFvFV6hLSgLMYHg.jpeg" /><figcaption><strong>Correlation Matrix</strong></figcaption></figure><ul><li>Here we can infer that “density” has strong positive correlation with “residual sugar” whereas it has strong negative correlation with “alcohol”.</li><li>“free sulphur dioxide” and “citric acid” has almost no correlation with “quality”.</li><li>Since correlation is zero we can infer there is no linear relationship between these two predictors.However it is safe to drop these features in case you’re applying Linear Regression model to the dataset.</li></ul><p>A box plot (or box-and-whisker plot) shows the distribution of quantitative data in a way that facilitates comparisons between variables.The box shows the quartiles of the dataset while the whiskers extend to show the rest of the distribution.</p><p>The box plot (a.k.a. box and whisker diagram) is a standardized way of displaying the distribution of data based on the five number summary:</p><ul><li>Minimum</li><li>First quartile</li><li>Median</li><li>Third quartile</li><li>Maximum.</li></ul><p>In the simplest box plot the central rectangle spans the first quartile to the third quartile (the interquartile range or IQR).</p><p>A segment inside the rectangle shows the median and “whiskers” above and below the box show the locations of the minimum and maximum.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j1pIrOQDO9N8w27lXeTnGw.jpeg" /><figcaption><strong>Boxplot</strong></figcaption></figure><p>Outliers are either 1.5×IQR or more above the third quartile or 1.5×IQR or more below the first quartile.</p><ul><li>In our data set except “alcohol” all other features columns shows outliers.</li></ul><p>Now to check the linearity of the variables it is a good practice to plot distribution graph and look for skewness of features. Kernel density estimate (kde) is a quite useful tool for plotting the shape of a distribution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CrcsNnhKfN4pWgvAuO5pvQ.jpeg" /><figcaption><strong>Distribution Plot</strong></figcaption></figure><ul><li>“pH” column appears to be normally distributed</li><li>remaining all independent variables are right skewed/positively skewed.</li></ul><blockquote>Lastly, to sum up all Exploratory Data Analysis is a philosophical and an artistical approach to guage every nuance from the data at early encounter.</blockquote><p>You can glance through my jupyter notebook <a href="http://nbviewer.jupyter.org/github/PBPatil/Exploratory_Data_Analysis-Wine_Quality_Dataset/blob/master/winequality_white.ipynb"><strong>here</strong></a> and try-test with different approaches, for eg. try out a pairplot and share what all inferences you could grab from it or if I failed to capture any useful information in my own approach,do share that too in comments.</p><p><strong>Read Other Articles I Published </strong>-</p><ol><li><a href="https://medium.com/@theprasadpatil/last-minute-revision-part-i-machine-learning-statistics-8de23a377987">Last Minute Revision: Part I — Machine Learning &amp; Statistics</a></li><li><a href="https://medium.com/@theprasadpatil/how-to-create-a-pdf-report-from-excel-using-python-b882c725fcf6">How to create a PDF report from Excel using Python</a></li><li><a href="https://towardsdatascience.com/how-to-extract-keywords-from-pdfs-and-arrange-in-order-of-their-weights-using-python-841556083341?source=follow_footer-----b882c725fcf6----0-------------------------------">How to Extract Keywords from PDFs and arrange in order of their weights using Python</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8fc1cb20fd15" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/exploratory-data-analysis-8fc1cb20fd15">What is Exploratory Data Analysis?</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>