<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by DEVI GUSKRA on Medium]]></title>
        <description><![CDATA[Stories by DEVI GUSKRA on Medium]]></description>
        <link>https://medium.com/@datascienceanywhere?source=rss-50179714151b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*W4XnIrKtOUastKqz0YCfZQ.jpeg</url>
            <title>Stories by DEVI GUSKRA on Medium</title>
            <link>https://medium.com/@datascienceanywhere?source=rss-50179714151b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 13:44:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@datascienceanywhere/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[7 Steps to Build Automatic Number Plate Recognition in Python]]></title>
            <link>https://medium.com/codex/7-steps-to-build-automatic-number-plate-recognition-in-python-53e34c9ae583?source=rss-50179714151b------2</link>
            <guid isPermaLink="false">https://medium.com/p/53e34c9ae583</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[DEVI GUSKRA]]></dc:creator>
            <pubDate>Mon, 29 Mar 2021 14:56:21 GMT</pubDate>
            <atom:updated>2021-03-29T14:59:40.966Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xp4lLXDUx0T4IUa9" /><figcaption>Photo by <a href="https://unsplash.com/@tmillot?utm_source=medium&amp;utm_medium=referral">Thomas Millot</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4><a href="http://medium.com/codex">CODEX</a></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZyOc2VQaM4qJ0UzC8iY0gw.png" /></figure><p><strong>Image Processing and Object Detection</strong> is one of the areas of Data Science and has a wide variety of applications in the industries in the current world. Many industries looking for a Data Scientist with these skills.</p><p>This article explains how to develop a number plate object detection model from scratch. We developed even developed an API using Flask. However, in this article, we explained how to train a<strong> customize object detection model</strong> from scratch.</p><p>For building the license plate recognition we need data. For that, we need to collect the vehicle images where the <strong>number plate</strong> appears on it. <a href="https://drive.google.com/drive/folders/1VRqnQnlYYdTMl1icgW0L_rUEmU-E-NHZ?usp=sharing">Here is the sample data that I used for building this project.</a> You can also download it from my google drive.</p><h3>Project Architecture:</h3><p>Now, let&#39;s look into the project architecture of the number plate recognition and OCR we follow to build.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bsxDyw0N5shI4q0_PyXxlQ.png" /><figcaption>Figure-1: Number Plate Recognition Project Architecture</figcaption></figure><p>In the above architecture, there are six modules. Labeling, Training, Save Model, OCR and Pipeline, and RESTful API. But this article limits to only the first three modules. The process is as follows. First, we will collect the image. Then we have to label images for object detection of License Plate or Number Plate using <a href="https://github.com/tzutalin/labelImg">Image Annotation Tool</a> which is open-source software developed in python GUI. Then after labeling the image we will work on data preprocessing, build and train a deep learning object detection model (Inception Resnet V2) in <strong>TensorFlow </strong>2. Once we have done with the Object Detection model training process, then using this model we will crop the image which contains the license plate which is also called the region of interest (ROI), and pass the ROI to Optical Character Recognition API Tesseract in Python (<strong>PyTesseract</strong>). In this module, we will extract text from images. Now, we will put it all together and build a Pipeline Deep Learning model. In the final module, we will learn to create a web app project using FLASK Python. With that, we are finally ready with our App.</p><h4>Step -1: Labeling</h4><p>For building the license plate recognition we need data. For that, we need to collect the vehicle images where the number plate appears on it. <a href="https://drive.google.com/drive/folders/1VRqnQnlYYdTMl1icgW0L_rUEmU-E-NHZ?usp=sharing">Here is the sample data that I used for building this project.</a> You can also download it from my google drive. For label images, I used the <a href="https://github.com/tzutalin/labelImg">LabelImg image annotation tool</a>. Download the labelImg from GitHub and follow the instruction to install the package. After that open, the GUI as give the instruction and click on <strong>CreateRectBox </strong>and draw the rectangle box as shown below and save the output in <strong>XML</strong>.</p><pre>pip install pyqt=5<br>pip install lxml<br>pyrcc5 -o libs/resources.py resources.qrc<br>python labelImg.py<br>python labelImg.py [IMAGE_PATH] [PRE-DEFINED CLASS FILE]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XVVOz85jdug1HGjiDRejRw.png" /><figcaption>Figure-2: Labeling using Image Annotation</figcaption></figure><p>This is a manual process and you need to do it for all the images. Be careful while doing labeling because the labeling process has a direct impact on the accuracy of the model. <a href="https://www.udemy.com/course/deep-learning-web-app-project-number-plate-detection-ocr/?referralCode=BCC3EDB9787790441A19">Click here for a video tutorial</a></p><h4>Step-2: Parsing Information from XML</h4><p>Once you are done with the labeling process now we need to do some data preprocessing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*1gEG_LQrtONphfgMP1XCoA.png" /><figcaption>Figure -3 Output of Image Annotation Tool in XML</figcaption></figure><p>Since the output of the label is XML in order to use this for the training process we need data in array format. For that, we will take the useful information from the label which are the diagonal points of the rectangle box or <strong>bounding box </strong>which are <em>xmin, ymin, xmax, ymax </em>respectively as shown in <strong>figure 3</strong>. This is available in XML. So we need to extract the information and save it in any convenient format, here I will convert bounding information into CSV and later on, I will convert that into an array using <strong>Pandas</strong>. Now let’s see how to <strong>parse </strong>the information using Python.</p><p><strong>Parsing data from XML and Converting it into CSV</strong></p><p>Here I am using <strong>xml.etree </strong>python library to parse the data from XML and also import pandas and glob. Using glob let first get all the XML files that are produced during labeling.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3f0b091fd1549a585b44a384ce0c7cd3/href">https://medium.com/media/3f0b091fd1549a585b44a384ce0c7cd3/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/99b3fc5a82874aa6caa1a6224eb014e4/href">https://medium.com/media/99b3fc5a82874aa6caa1a6224eb014e4/href</a></iframe><p>In the above code, we individually take each file and parse into xml.etree and find the object -&gt; bndbox which is in line 2 to 7. Then we extract xmin,xmax,ymin,ymax and saved those values in the dictionary which is in lines 8 to 17. After then we convert it into a pandas data frame and save that into CSV file as shown below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/401aef4c67c60cd7d6ac8c7328790304/href">https://medium.com/media/401aef4c67c60cd7d6ac8c7328790304/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/370/1*QqqHCSVSwOQH_0yhbVCjrQ.png" /></figure><p>With the above code, we successfully extract the diagonal position of each image and convert the data from an unstructured to a structured format.</p><p>Now also extract the respective image filename of the XML.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/34c96d6973b811a44678d5a2e068cbad/href">https://medium.com/media/34c96d6973b811a44678d5a2e068cbad/href</a></iframe><h4>Step-4: Verify the Data</h4><p>As till now we did the manual process it is important to verify the information is we got is valid or not. For that just verify the bounding box is appearing properly for a given image. Here I consider the image N1.jpeg and the corresponding diagonal position can found in <strong>df.</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c7dfea636319386beaef76bd221319d4/href">https://medium.com/media/c7dfea636319386beaef76bd221319d4/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*WnKKxQdUF6TJ6k6rV2h4RQ.png" /></figure><h4>Step -5: Data Processing:</h4><p>This is a very important step, in this process we will take each and every image and convert it into an array using <strong>OpenCV </strong>and resize the image into 224 x 224 which is the standard compatible size of the pre-trained <strong>transfer learning model.</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a5f11d732b41025fb8f4bcf14f3195dc/href">https://medium.com/media/a5f11d732b41025fb8f4bcf14f3195dc/href</a></iframe><p>After that, we will normalize the image just by dividing with maximum number as we know that the maximum number for an 8-bit image is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/99/0*cip1n22lSK-pT0d-" /></figure><p>That the reason we will divide our image 255.0. The way of diving an array with the maximum value is called <strong>Normalization </strong>(Min-Max Scaler).</p><p>We also need to normalize our labels too. Because for the deep learning model the output range should be between 0 to 1. For normalizing labels, we need to divide the diagonal points with the width and height of the image. And finally values in a python list. In the next step, we will convert the list into an array using Numpy.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/27ee564a3375364fd3279b2a7bb61b4e/href">https://medium.com/media/27ee564a3375364fd3279b2a7bb61b4e/href</a></iframe><p>Now split the data into training and testing set using <strong>sklearn</strong>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1f5618f5f55b77eeb50d9934bffb6886/href">https://medium.com/media/1f5618f5f55b77eeb50d9934bffb6886/href</a></iframe><h4>Step -6: Build and Train Transfer Learning</h4><p>Now we are ready to train a deep learning model for object detection. Here we will use the <strong>InceptionResNetV2 </strong>model with pre-trained weights and train this to our data. Let’s start this by importing necessary libraries from <strong>TensorFlow 2.3.0</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c9e92c63ce5eaf84857a0206a9f10f17/href">https://medium.com/media/c9e92c63ce5eaf84857a0206a9f10f17/href</a></iframe><p>The model we are building is an object detection model and the number of output expected from the is model 4 which are the diagonal points. Hence we will add an embedding neural network layer to the transfer learning model which is shown in lines 5 to 9.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4b9028ca8722fbac36e69b9428f2774b/href">https://medium.com/media/4b9028ca8722fbac36e69b9428f2774b/href</a></iframe><p>Now compile the model and train the model</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/96b19f4850c98704c1bea3f1474854b8/href">https://medium.com/media/96b19f4850c98704c1bea3f1474854b8/href</a></iframe><p>That’s it with that we trained the model. This process usually takes 3 to 4 hours depends upon the speed of the computer. Here I showing you the loss of the model in TensorBoard.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c3_7FZsg48PMVqG-HHqcIQ.png" /></figure><p>It seems that the model converged after 100 epochs.</p><blockquote>Note: For better reduction in loss we need to train model with lots of data at least 10000 images</blockquote><h4>Step-7: Make Bounding Box Prediction</h4><p>This is the final step in object detection. In this step, we will put it all together and get the prediction for a given image.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/161a14af2b232e55ccb1070bf1c12dd9/href">https://medium.com/media/161a14af2b232e55ccb1070bf1c12dd9/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/596/1*Anj6w9rLro52EAqZn1rVyg.png" /></figure><p>That’s it !!!. This is how we can develop license plate detection from Scratch.</p><p>This article explains just 50% of the project architecture. The next process involves extract text from the number plate and developing RestfulAPI in Flask. Here is the output of the project</p><figure><img alt="Number Plate OCR Web App Project" src="https://cdn-images-1.medium.com/max/1024/1*VnDLRhRABHUMDKJyDmk6ZA.gif" /></figure><blockquote><a href="https://www.udemy.com/course/deep-learning-web-app-project-number-plate-detection-ocr/?referralCode=BCC3EDB9787790441A19">Click here for 3 hr Video Project Explanation and building Number Plate Web App from Scratch.</a></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=53e34c9ae583" width="1" height="1" alt=""><hr><p><a href="https://medium.com/codex/7-steps-to-build-automatic-number-plate-recognition-in-python-53e34c9ae583">7 Steps to Build Automatic Number Plate Recognition in Python</a> was originally published in <a href="https://medium.com/codex">CodeX</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deep Dive into Logistic Regression in Python]]></title>
            <link>https://datascienceanywhere.medium.com/deep-dive-into-logistic-regression-in-python-3a79b7b52d85?source=rss-50179714151b------2</link>
            <guid isPermaLink="false">https://medium.com/p/3a79b7b52d85</guid>
            <category><![CDATA[python-pandas]]></category>
            <category><![CDATA[logistic-regression]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[maximum-likelihood]]></category>
            <dc:creator><![CDATA[DEVI GUSKRA]]></dc:creator>
            <pubDate>Sun, 28 Mar 2021 17:43:36 GMT</pubDate>
            <atom:updated>2021-03-28T18:42:10.527Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dX4OIcT1R2A93V8SkBf6yA.png" /></figure><p><strong>Logistic Regression</strong> is supervised classification algorithm where we will used to classify the categories. In logistic regression the output of the estimator is discrete or categorical. The idea of Logistic regression is started from the intuition of Linear regression. For example as from the figure -1 suppose there are two classes, <strong>class 0</strong> and <strong>class 1</strong> and we need to estimate the classes using independent variable <strong>X</strong>. For that we can start with the idea of simple linear regression where we fit a line for the samples independent variable and dependent variable (class). Now with the simple linear regression (solid yellow line) we can estimate the probability. Lower the probability we can estimate as class 0 and higher the probability we can estimate as class 0.</p><p>Now the question is what is the threshold probability that we need set to class of the sample. By default we can consider 0.5 is the threshold probability and estimate the class. Now here there is problem for default threshold probability = 0.5. Suppose that there are extreme values (not outliers) of X, by common sense we can say that these are also belongs to class 1.</p><figure><img alt="Linear Regression for a classification Data" src="https://cdn-images-1.medium.com/max/960/0*otG5f4oFVyG4JSbq.gif" /><figcaption>Figure-1: Linear Regression to Classification Problem</figcaption></figure><p>Now when you build a linear regression with the extreme values the linear regression will looks like yellow doted line. And for now default threshold value 0.5 is not approximation for selection or estimation of classification. This phenomenon occurs because, linear regression fit the based on least square regression and hence least square is not appropriate for this problem. To tackle this problem Logistic regression use <strong>Maximum Likelihood Estimation (MLE)</strong>.</p><p>In MLE, the Goal is to maximize likelihood.</p><ul><li>In most Data Science optimizations, the goal is to find minima using calculus (minimize sum of squared errors in linear regression, and so on) or numerical techniques like Gradient Descent (minimize deviance in logistic regression, and so on)</li><li>Maximum Likelihood =&gt; Minimum of Negative Log-Likelihood.</li></ul><p><strong>Example Problem:</strong></p><blockquote><strong><em>An auto club mails a flier to its members offering to send more information regarding a supplemental health insurance plan if the member returns a brief enclosed form. Can a model be built to predict if a member will return the form or not?</em></strong></blockquote><blockquote><strong><em>URL: </em></strong><a href="https://github.com/datascienceanywhere"><strong><em>GitHub (for code)</em></strong></a></blockquote><p>Figure-2, explain the scatter plot between the independent variable (Age) and dependent variable (yes or no). As linear regression is not appropriate because we might get probability in greater than 1 and less than zero. In order to limit the values between 0 to 1 range, we will transform the output of linear regression using sigmoid function. Hence from the sigmoid function we can calculate profanities. From this output we find the best threshold probability using MLE.</p><p>So, the equation of the transformed linear regression is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/93/1*rLTj80DtdCC5dMot2AZacA.png" /><figcaption>equation — — — — — (1)</figcaption></figure><p>Where,</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/0*BPwgUTuLWY1JIcer" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/0*9ExfwlZNPDh2DSW5.png" /></figure><p>​ (also know as the systematic or the structural component or linear predictor).</p><p>This is a logistic model. The function is also known as the inverse link function, which links the response with the systematic component. p is the probability that a club member fits into group 1 (returns the form; success; P(Y=1|X)).</p><p>If you expand the equation — 1 :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/259/0*cmUVSAIRXa11oual" /></figure><p>Odds Ratio is obtained by the probability of an event occurring divided by the probability that it will not occur. <strong>Logistic model</strong> can be transformed into an odds ratio:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/174/0*GnOB3IilyEsf87t6" /></figure><p>​if we substitute in equation 2 in 3 and solve it then you will get</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/341/0*jWDHJURaXhf1iNoQ" /></figure><p>The log of the odds ratio is called logit, and the transformed model is linear in 𝛽s. That the reason this algorithm is called logistic regression.</p><h3>Logistic Regression in Python</h3><p>As mentioned above logistic regression has two steps.</p><p>Step-1: Develop transformed linear regression and computer probability of each data point</p><p>Step-2: Find the best odd ratio using MLE.</p><p>Now Lets build logistic regression in Python and you can <a href="https://github.com/datascienceanywhere/logistic_regression/blob/main/data/MTcars.xlsx">download the dataset from out github link </a>. Let start by import necessary libraries:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2719d6372471a5d387fdd9610e762645/href">https://medium.com/media/2719d6372471a5d387fdd9610e762645/href</a></iframe><p>This data is about the automobile on different types of cars and their specification. For the sake of simplicity consider let us consider <strong>wt, qsec, am. </strong>In this example we will take two independent features (wt, qsec) and we will predict dependent variable (am — a= automatic transmission and m=manual transmission).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/411/0*3hVYZZKwGoFtMmpv.png" /><figcaption>Figure-3: Scatter Plot between <strong>wt </strong>and <strong>qsec</strong></figcaption></figure><p>Above figure show the scatter plot between <strong>wt </strong>and <strong>qsec </strong>and colored by the label or output. It seems that data is linear separable and let see how logistic regression can able to separate the classes. Let’s first split the data into independent and dependent variable.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1bc9ae47b917b1c5cf58a10bbb09b9f9/href">https://medium.com/media/1bc9ae47b917b1c5cf58a10bbb09b9f9/href</a></iframe><p>Out: ((32, 2), (32,))</p><p>Now import logistic regression model from sklearn and train the model from the above data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3178982919cb00d6077d8be90784de14/href">https://medium.com/media/3178982919cb00d6077d8be90784de14/href</a></iframe><p>Logistic Regression has train successfully. What we have did is that we simple build transformed linear regression (linear regression passes to sigmoid function) and get probabilities for each data point. You can get the probabilities from using following syntax in python and then we will visualize probability of each data point using <strong>matplotlib.</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9d2496abe1d74583fd6ed0480cfb14d8/href">https://medium.com/media/9d2496abe1d74583fd6ed0480cfb14d8/href</a></iframe><p>Out: [0.8138488 0.62844367 0.70587441 0.14634078 0.30573648 0.05604451 0.3992924 0.1144186 0.02357085 0.16761907 0.1224557 0.07223128 0.13407676 0.0971771 0.00327584 0.00238895 0.00367752 0.65373927 0.93147451 0.77590941 0.41927653 0.28516772 0.27301396 0.3124652 0.14147834 0.8341205 0.92213607 0.97901646 0.79631523 0.84616163 0.58652099 0.44669103]</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/773/0*ZUYTOzTT6VraFm-J.png" /></figure><p>Above scatter plot displays the probability of each data plot from logistic regression. Now we need to estimate the best threshold probability to be select above which it is class -1 other wise it is class-0. We can find this using <em>Maximum Likelihood Estimation</em>.</p><h3>Maximum Likelihood Estimate (MLE)</h3><p>Now let see how to find best <strong>oddsratio </strong>using maximum likelihood estimate. MLE is in simple calculating <a href="https://en.wikipedia.org/wiki/Cross_entropy">log loss or categorical cross entropy</a> for different probability. And find the best probability where log loss is minimum.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/60e6ef3a8be8a38aa0e803954fd80e4d/href">https://medium.com/media/60e6ef3a8be8a38aa0e803954fd80e4d/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/885/0*6mBQ4s3mM4jNW3lx.png" /></figure><p>From above output where x axis is probability and y-axis is log loss and you can see that the threshold probability is around <strong><em>p = 0.42</em></strong>. Hence we can find the index position of best probability, probability and corresponding log loss with following code:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ad4519d72cd21219b3d0ee61934b91c4/href">https://medium.com/media/ad4519d72cd21219b3d0ee61934b91c4/href</a></iframe><p><em>Now we have successfully predicted the values using logistic regression.</em></p><h3><em>Model Equation</em></h3><p>Let’s now calculate the equation of the logistic regression. As we know that equation of the logistic regression is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/227/0*2ZAwK1US0VPov5Cf" /></figure><p>for two independent variables and one dependent variables. Lets now calculate find the equation of the model we found.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/db9988b45e5c5db1fa3865463f2ad5c1/href">https://medium.com/media/db9988b45e5c5db1fa3865463f2ad5c1/href</a></iframe><p>Out: (-0.3141153295199363, array([[-2.38172384, -0.61130325]]), array([17.77738323]))</p><blockquote><em>which means,</em></blockquote><blockquote>-0.314 = 17.77–2.3817 x1–0.611 x2</blockquote><blockquote>x2 = (-0.314–17.77 + 2.3817 * x1)/(-0.611)</blockquote><p>Now using the above equation let’s visualize logistic regression line with best threshold value found from MLE.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/741f068f4e72b8527f967944b211a23a/href">https://medium.com/media/741f068f4e72b8527f967944b211a23a/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/773/0*Q3I4pXWVhaZhFE_k.png" /></figure><p>This is how we can able build the logistic regression and use maximum likelihood estimation for best threshold probability in Python.</p><p>To download the code please click our <a href="https://github.com/datascienceanywhere/logistic_regression">GitHub link</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3a79b7b52d85" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Visualizing Time Series Data in Python]]></title>
            <link>https://datascienceanywhere.medium.com/visualizing-time-series-data-in-python-e49fa5d10ea?source=rss-50179714151b------2</link>
            <guid isPermaLink="false">https://medium.com/p/e49fa5d10ea</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[visualization]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[timeseries-forecasting]]></category>
            <dc:creator><![CDATA[DEVI GUSKRA]]></dc:creator>
            <pubDate>Fri, 13 Nov 2020 14:31:34 GMT</pubDate>
            <atom:updated>2020-11-13T14:31:34.684Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J6DNDq1Pk7BklaCfrQ8MQQ.png" /></figure><p>URL: <a href="http://datascienceanywhere.com/timeseries/">http://datascienceanywhere.com/timeseries/</a></p><p>In this article, I will explain how to visualize time series data. In general, there are four kinds of charts that are used to visualize time series data.</p><ol><li>Line Chart</li><li>Area Plot</li><li>Bar Plot</li><li>Heatmaps</li></ol><p><a href="http://datascienceanywhere.com/timeseries/">Click here to download the data and Google Colab notebook</a></p><p>Now let us consider the <a href="https://docs.google.com/spreadsheets/d/1isEzxgZacr5nHvOlSC7k-KV407Jm9MVd/edit#gid=1381125080">“Debourgh-Sales.xlsx”</a> excel sheet where we are having the “Years” and “Sales” information for each month around 10 years starting from 01–01–2001.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8HhJPN6DzXURYhMG" /><figcaption>Debourgh Sale .xlsx</figcaption></figure><h3>1. Line Chart</h3><p>A line chart is the most common way of visualizing the time series data. Line chart particularly on the x-axis, you will place the time and on the y-axis, you will use independent values like the price of the stock price, sale in each quarter of the month, etc.</p><p>Now let’s see how to visualize a line plot in python.</p><pre>import numpy as np # numerical python<br>import pandas as pd # pannel datasets<br>import matplotlib.pyplot as plt <br>import seaborn as sns</pre><p>Once you import all necessary libraries now load the dataset</p><pre># load the data using pandas<br>df = pd.read_excel(&#39;./data/Debourgh-Sales.xlsx&#39;,sheet_name=&#39;Sheet2&#39;)<br>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/284/0*gY_gIOBgGORAto_R" /></figure><p>And the data we have with us is consists of two columns “Years” and “Sales”. If you see the “Years” column it is of date format and “Sales” could be a float value. Now it is important to find the is there are any missing values and data types of each column and number of instances in the dataset. For that, we can use <em>df.info().</em></p><pre>df.info()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Jx2SIDMVfM87Gr-_" /></figure><p>From the results, there are 108 entries with and no missing values. and the data type of Years is datetime64[ns] and Sales is float64. Now it is convenient to plot the line plot with this data. In this example, I will show you the two ways to plot the line plot.</p><ul><li>Using Seaborn</li></ul><pre>plt.figure(figsize=(15,6))<br>sns.lineplot(x=&#39;Years&#39;,y=&#39;Sales&#39;,data=df)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/0*GIpzJU1MFi_5mPjt" /></figure><ul><li>Using Pandas dataframe</li></ul><p>In order to plot with a pandas data frame, we need to set index to DateTime values i.e., Years, and then we need to visualize the graph.</p><pre>df1 = df.set_index(&#39;Years&#39;) # setting &quot;Years&quot; as index</pre><pre>fig,ax = plt.subplots(figsize=(15,6))<br>df1.plot(kind=&#39;line&#39;,ax=ax)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/0*yBJA6kxyPR6h6qqx" /></figure><p><strong>Hue the Line plot</strong></p><p>You can color your line plot based on years, meaning you can color the line plot for each year ie. you can color the line for each year. In order to do that we need to extract years from the date. We will see how to extract the year and quarter from the Years column.</p><pre>df[&#39;year&#39;] = df[&#39;Years&#39;].dt.year  # extracting year<br>df[&#39;quarter&#39;] = df[&#39;Years&#39;].dt.quarter # extracting quarter<br>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*In3sKy286w1hCfaK" /></figure><p>As you see from the results we can easily extract the year and quarter from the Years column using pandas. Now using the above data frame we visualize the line chart by coloring to each year differently.</p><pre>plt.figure(figsize=(15,6))<br>sns.lineplot(x=&#39;Years&#39;,y=&#39;Sales&#39;,data=df,hue=&#39;year&#39;,palette=&#39;Set1&#39;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/891/0*_e_0hFIeQg7JN-S1" /></figure><h3>2. Bar Plot</h3><p>We cannot directly visualize the bar plot in time series data. We need to extract a few more information out of it. As bar plot is used for categorical data. So, here we need to create categories and with those categories, we need to visualize the bar plot. We will see three kinds of bars plot “Bar plot”, “Proposition Bar plot” and “Stacked Bar Plot”. Let’s see one by one in python.</p><p><strong>(a) Bar Plot</strong></p><p>The bar plot is used for categorical data. But here the data is continuous but still, we can make that categorical like getting of average or sum of all sales in each year, and average or sum of all sales in each quarter are some examples of bringing the data into categorical.</p><p>Let visualizing the sum of all sales each year. We can do that using group of “Sales” and “year” and group by “year”. The following code is shown below</p><pre>df_year = df[[&#39;Sales&#39;,&#39;year&#39;]].groupby(by=&#39;year&#39;).sum().reset_index()<br>df_year</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OJDeDUpqH_JgkPlh" /></figure><p>As you can see that we got the sum of sales from each year and now let’s visualize the bar plot with seaborn as shown below.</p><pre>sns.catplot(x=&#39;year&#39;,y=&#39;Sales&#39;,data=df_year,kind=&#39;bar&#39;,aspect=3)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7nBuAeEPyMi3_1TL" /></figure><p>Let’s visualize the bar plot even with a pandas data frame.</p><pre>fig, ax = plt.subplots(figsize=(15,6))<br>df_year.set_index(&#39;year&#39;).plot(kind=&#39;bar&#39;,ax=ax)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/0*UHcThjbeL8YaNJjL" /></figure><p><strong>(b) Proposition Bar Plot</strong></p><p>Now let’s look into the proposition bar plot. The Proposition bar plot is very of comparing the sales based on quarter and year. This kind of bar chart is very useful especially if you’re want to compare more than two variables. In order to compare the results we need to do the group by to the following columns, in this case, it is year and quarter and with those values, we need to reshape into the desired format as shown in the below code.</p><pre>sales = df[[&#39;Sales&#39;,&#39;year&#39;,&#39;quarter&#39;]].groupby(by=[&#39;year&#39;,&#39;quarter&#39;]).sum().values<br>df_sales = pd.DataFrame(sales.reshape(9,4),index=list(range(2001,2010)),columns=           [&#39;Q1&#39;,&#39;Q2&#39;,&#39;Q3&#39;,&#39;Q4&#39;])</pre><pre>df_sales</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*a0xCMzVplCOYzDQP" /></figure><p>Here we bring the data with index is years, column as a quarter, and the values are sums of total sales in the data. With the data above you can plot the proposition bar plot using pandas as shown below.</p><pre>fig,ax = plt.subplots(figsize=(15,6))<br>df_sales.plot(kind=&#39;bar&#39;,ax=ax)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*b31hiO4ynbDLRrXWEFF4aQ.png" /></figure><p><strong>(c) Stacked Bar Plot</strong></p><p>The stacked bar plot is the same as the proposition bar plot. But in the stacked bar plot we put all the bar on top of each bar as shown below. We can plot a stacked bar plot by just set the parameter `stacked` to True.</p><pre>fig,ax = plt.subplots(figsize=(15,6))<br>df_sales.plot(kind=&#39;bar&#39;,ax=ax,stacked=True)<br>plt.show()</pre><p>Add alt text</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/0*y-zCLPn4KSuaulzu" /></figure><h3>3. Area Plot</h3><p>The area plot is a combination of line-chart and bar plot. Just like a bar, we have different types of area plots like Area chart, proposition area chart, and stacked area chart.</p><p>We can plot the area charts using pandas simply setting the parameter kind to “area”. This is much more similar to the line plot but the area plot filled the portion of the area with blue color.</p><pre>fig,ax = plt.subplots(figsize=(15,6))<br>df1.plot(kind=&#39;area&#39;,ax=ax)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/877/0*h7tghiUmztkaBXxc" /></figure><p>Stacked Area Plot</p><p>The stacked area plot is similar to the stacked bar plot and we will use the same data to visualize the stacked area plot. The intuition behind that is also the same as a stacked bar plot i.e., we can visualize each quarter sales with respective color and sum of all quarter values equal to total sales in that year.</p><pre>fig,ax = plt.subplots(figsize=(15,6))<br>df_sales.plot(kind=&#39;area&#39;,ax=ax,stacked=True)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*b31hiO4ynbDLRrXWEFF4aQ.png" /></figure><p>Proposition Area Plot</p><p>In the proposition area plot, we can visualize the sales or values on the same scale wherein the stacked bar plot we sum the area to get the total sales.</p><pre>fig,ax = plt.subplots(figsize=(15,6))<br>df_sales.plot(kind=&#39;area&#39;,ax=ax,stacked=False)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/883/0*iWTUoef-nP9XuxSb" /></figure><h3>4. Heatmap</h3><p>Finally, heatmap is a powerful way to visualize all your data in one chart. From the chart, we are plotting complete data for all the sales in each quarter and year. It the clear that where we are having high sales and least sale can be easily identified with heatmap</p><pre>plt.figure(figsize=(10,8))<br>sns.heatmap(df_sales,annot=True,fmt=&#39;.2f&#39;,cmap=&#39;gnuplot&#39;,linewidths=1)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/559/0*g4xdvI4dBrzLtNqh" /></figure><p>Still, there are a lot more visualization chart is there for time series data. But these are the charts that are mostly used to get insights from the data.</p><p>For more details on time series and forecasting with ARIMA, SARIMA visit</p><p><a href="http://www.datascienceanywhere.com/timeseries">www.datascienceanywhere.com/timeseries</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e49fa5d10ea" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>