<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by KSV Muralidhar on Medium]]></title>
        <description><![CDATA[Stories by KSV Muralidhar on Medium]]></description>
        <link>https://medium.com/@ksvmuralidhar?source=rss-ce697e8ed35c------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*FxMBjlSNZKHMed1QuNQTgg.jpeg</url>
            <title>Stories by KSV Muralidhar on Medium</title>
            <link>https://medium.com/@ksvmuralidhar?source=rss-ce697e8ed35c------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 17:16:24 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ksvmuralidhar/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to add gradient background to Python plots]]></title>
            <link>https://medium.com/data-science/how-to-add-gradient-background-to-python-plots-9beda811ea19?source=rss-ce697e8ed35c------2</link>
            <guid isPermaLink="false">https://medium.com/p/9beda811ea19</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[matplotlib]]></category>
            <category><![CDATA[gradient-background]]></category>
            <dc:creator><![CDATA[KSV Muralidhar]]></dc:creator>
            <pubDate>Wed, 01 Dec 2021 16:54:05 GMT</pubDate>
            <atom:updated>2023-07-07T18:45:55.120Z</atom:updated>
            <content:encoded><![CDATA[<h4>This article discusses a simple way to add gradient background to plots in Python</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CAHu2KVy--2Dk66Sj-OisA.png" /><figcaption>Image by author</figcaption></figure><p>We may at times need to add multi-color background to plots to highlight a few sections of the plot. For example, we may highlight the periods of recession and recovery in red and green respectively. In this article, we’ll discuss a simple yet effective way to do this in Python. For Tableau users, this is similar to adding reference bands.</p><p>We’ll tell a story of how Nifty (a benchmark index of the Indian stock market) performed during Covid-19. We’ll highlight the period of slump and recovery in red and green respectively and also use gradient background to depict the volatility.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/de48314171d71a9f666e3e972b788c9b/href">https://medium.com/media/de48314171d71a9f666e3e972b788c9b/href</a></iframe><p>In the above code snippet, the variable <strong><em>nifty </em></strong>stores the closing values of Nifty from January 1, 2020 to September 18, 2020. The variable <strong><em>slump</em></strong> stores the closing values of Nifty from February 19, 2020 to May 18, 2020, this is the period where Nifty crashed due to Covid-19. The variable <strong><em>recovery</em></strong> stores the closing values of Nifty from May 18, 2020 to September 18, 2020, this is the period where Nifty began recovering from Covid-19. Below is a snapshot of the <strong><em>nifty</em></strong> data frame.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/263/1*kOFlL0mDLNr7joz0-tWoYg.png" /><figcaption>Nifty data frame (Image by author)</figcaption></figure><p>To understand what’s goes into the plot, we’ll plot the time series without any background colors and tell a story about Nifty’s journey through Covid-19.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2397b17943d002300fe605dda3a23da3/href">https://medium.com/media/2397b17943d002300fe605dda3a23da3/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NFpNlwA4AM01mCt0yMDOzQ.png" /><figcaption>Image by author</figcaption></figure><p>In the above plot, the periods of slump and recovery are highlighted in red and green respectively. There are a few annotations to highlight the major events. There is also an annotation in a bigger font that highlights the number of days of carnage. What if we need to depict the volatility during the periods of slump and recovery? We can do this by using gradient background. Though this method might be computationally expensive, using it can add a new dimension to the story. We’ll look at the code to add gradient background to the above plot.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/03925b0ebc17409f43bc732943653449/href">https://medium.com/media/03925b0ebc17409f43bc732943653449/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CAHu2KVy--2Dk66Sj-OisA.png" /><figcaption>Image by author</figcaption></figure><h4>How is the gradient background added?</h4><ol><li>We calculated the 3-day moving average of the values in the period of slump and recovery.</li><li>The moving average values are normalized within a range of 0.15 to 1 to act as alpha values for the red and green vertical lines (axvline) drawn as the background to the plot. For the red lines, the higher the normalized moving average value, the lighter would be the line. For the green lines, the higher the normalized moving average value, the darker would be the line.</li><li>Each normalized value is repeated 10 times as the vertical lines would be plotted with x-tick incrementing by 0.1 i.e. every single x-tick has 10 vertical lines.</li><li>The last value of the normalized moving averages is repeated 20 times to ensure the number of values are same as that of the original time series, as moving average calculation will result in a reduction of number of values in the time series.</li><li>Vertical lines are drawn as the background of the plot with x-tick incrementing by 0.1.</li></ol><p>In the above plot, we can see alternate light and dark green bands at the initial stages of recovery showing that the initial stages of recovery were more volatile than the later ones. We can also see a darker red band where Nifty made its bottom at 7,610.25.</p><p>This brings this article to an end. We’ve discussed how to add a gradient background using multiple colors to a Python plot. This technique can be used to highlight business cycles. Keeping the alpha value of the vertical lines constant will result in a multi-color background instead of a gradient background.</p><p>Know more about my work at <a href="https://ksvmuralidhar.in/">https://ksvmuralidhar.in/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9beda811ea19" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-add-gradient-background-to-python-plots-9beda811ea19">How to add gradient background to Python plots</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[100% stacked charts in Python]]></title>
            <link>https://medium.com/data-science/100-stacked-charts-in-python-6ca3e1962d2b?source=rss-ce697e8ed35c------2</link>
            <guid isPermaLink="false">https://medium.com/p/6ca3e1962d2b</guid>
            <category><![CDATA[matplotlib]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[stacked-bar-chart]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[KSV Muralidhar]]></dc:creator>
            <pubDate>Tue, 19 Oct 2021 16:04:42 GMT</pubDate>
            <atom:updated>2023-07-07T18:47:11.921Z</atom:updated>
            <content:encoded><![CDATA[<h4>Plotting 100% stacked bar and column charts using Matplotlib</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/841/1*-MSLBij5vknV5Tm-G0Rxfw.png" /><figcaption>Image by author</figcaption></figure><p>In this article, we’ll discuss how to plot 100% stacked bar and column charts in Python using Matplotlib. We’ll use the <strong>Netflix Movies and TV Shows</strong> dataset that’s downloaded from <strong>Tableau Public sample datasets (</strong><a href="https://public.tableau.com/en-us/s/resources">https://public.tableau.com/en-us/s/resources</a> and go to Sample Data<strong>). </strong>This dataset consists of TV shows and movies available on Netflix as of 2019 and is sourced by Tableau from <a href="https://www.kaggle.com/shivamb/netflix-shows"><strong>Kaggle</strong></a> and the updated version of the dataset can be found on Kaggle. However, for this article, we’ll use the older version available in <strong>Tableau sample datasets</strong>. Below is the snapshot of the dataset (transposed for better visibility).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/1*sxXfHCQbTleV2KGJmEnIyg.png" /><figcaption>Snapshot of the transposed dataset (Image by author)</figcaption></figure><p>In this article, we’ll plot stacked bar and column charts showing the proportion of the <strong><em>type</em></strong> of shows released in each <strong><em>release_year </em></strong>for the years 2016 to 2019.<strong><em> </em></strong>In the code below, we’ll select the records/rows with <strong><em>release_year</em></strong> between 2016 and 2019 and preprocess the data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d2348e7a88394f42579cfcbec905bdb3/href">https://medium.com/media/d2348e7a88394f42579cfcbec905bdb3/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/230/1*lfmmqc8iquZC97QKh0EuVg.png" /><figcaption>Image by author</figcaption></figure><p>We’ll create a cross tabulation that shows the proportion of the <strong><em>type</em></strong> of shows for each year. We’ll include <strong><em>release_year</em></strong> in the index and <strong><em>type</em></strong> of show in the columns. Using the <strong>normalize=True</strong> argument gives us the index-wise proportion of the data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/04cbc71e63377038e2e831e6787055a0/href">https://medium.com/media/04cbc71e63377038e2e831e6787055a0/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/261/1*JXRGU-posuMgQJrL67U5Ig.png" /><figcaption>Year-wise proportion of type of shows (Image by author)</figcaption></figure><p>We’ll create a similar cross tabulation as the one above, but this time we’ll retain the counts as they are without converting them into proportions. This cross tabulation is used to display the data labels on the plot, which we’ll see later in this article.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4c177d49b592f72f6aa0d07d46ca06a1/href">https://medium.com/media/4c177d49b592f72f6aa0d07d46ca06a1/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/241/1*8MMY1ChFUuGBlS0PN0z3uQ.png" /><figcaption>Year-wise count of type of shows (Image by author)</figcaption></figure><h3>100% stacked column chart</h3><p>Now, we’ll create a stacked column plot showing the proportion of the type of shows each year. We’ll use the cross tabulation having the proportions (that we created earlier) to plot the chart.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/23a28e9dcc043a86d89f9390c5dd4040/href">https://medium.com/media/23a28e9dcc043a86d89f9390c5dd4040/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/770/1*Q9x3gRc8HaLCJpyzvlWMUw.png" /><figcaption>Image by author</figcaption></figure><p>Now we’ll add the proportion data labels to our plot. We’ll create a nested loop in which the first loop iterates through the index of the cross tabulation having the proportions. The second loop iterates through the values by each index. We’ll look at the code and its output before diving deeper.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1537b9e0c2e8c8d0e8572e7a18c39946/href">https://medium.com/media/1537b9e0c2e8c8d0e8572e7a18c39946/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/771/1*Dw4CEfelU5Zk3nEcipaReA.png" /><figcaption>Image by author</figcaption></figure><p>In the above code, the first loop enumerates and iterates through the index of the cross tabulation having proportions. We’ll use the values of the enumeration (0, 1, 2, 3) as the x-position of the data labels. Let’s look at what the first loop returns.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e3d08c3d3fd8989f2b9d5c762177dc3a/href">https://medium.com/media/e3d08c3d3fd8989f2b9d5c762177dc3a/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/140/1*AlMGbe1JOsdJ7MTAmFV1xw.png" /><figcaption>Image by author</figcaption></figure><p>In the second loop, we’ll iterate through the values in each index of the cross tabulation having proportions. Let’s look at what the second loop returns.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d577de6316785388b06c3f0e8462252b/href">https://medium.com/media/d577de6316785388b06c3f0e8462252b/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/435/1*1AiglnE72bmyF7loCsW5xQ.png" /><figcaption>Image by author</figcaption></figure><p>We can see the inner loop iterating through the proportions of each index one by one. The variable <strong><em>proportion </em></strong>is the proportion in the cross tabulation that we created earlier. We’ve also used it as the y-position of the data label (which is fine while plotting counts but incorrect while plotting proportions).</p><p>In the above plot, we see that the data labels are misaligned and are not being displayed at the right position. We can correct the x-position of the labels by subtracting a value (0.17 in this case) from the x-position such that the label moves to the left. We’ll add a <strong><em>y_loc</em></strong> variable to the inner loop which is nothing but the cumulative sum of the proportions in an index. This acts as the correct y-position of the data labels and places a label at the top-most point of its respective bar. We’ll see the code and the output below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/47bb8c4ee1cbd33d6bb0c1d65e2a9678/href">https://medium.com/media/47bb8c4ee1cbd33d6bb0c1d65e2a9678/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/767/1*PwiALKDRwO0XipzEkQCrfg.png" /><figcaption>Image by author</figcaption></figure><p>We may align the data labels such that they appear in the center of each bar by modifying the y-position of the data labels in the plt.text() function as shown in the code below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3cb4668fc3412655a682ddf23b1ef8da/href">https://medium.com/media/3cb4668fc3412655a682ddf23b1ef8da/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*w8zkHm98pXssYlEXfNv0_Q.png" /><figcaption>Image by author</figcaption></figure><p>We may also add the counts of each bar by including them in the inner loop and modifying the string argument (<strong><em>s</em></strong>) in the plt.text() function as shown below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7ed06dd5fe8c86e9b2aed01c03de24e2/href">https://medium.com/media/7ed06dd5fe8c86e9b2aed01c03de24e2/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/782/1*v-iNGqFm6uAoNwFu8kmiEw.png" /><figcaption>Image by author</figcaption></figure><h3>100% stacked bar chart</h3><p>We can create a 100% stacked bar chart by slightly modifying the code we created earlier. We must change the kind of the plot from <strong><em>‘bar’ </em></strong>to <strong><em>‘barh’</em></strong>. Then swap the x and y labels and swap the x and y positions of the data labels in plt.text() function. Everything else stays the same. We’ll look at the code below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/18171d850f2d0941733b540b951c1f23/href">https://medium.com/media/18171d850f2d0941733b540b951c1f23/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*UYF1oHEj4OkYcy5aP3fC-A.png" /><figcaption>Image by author</figcaption></figure><p>This brings the article to an end. We’ve discussed how to create 100% stacked bar and column charts, add data labels to them, properly align and format those data labels. Since, the data is represented in the form of proportion/relative frequency instead of just counts/frequency (which may at times be misleading) 100% stacked charts provide a more reliable view of the data, especially while comparing across groups.</p><p>Know more about my work at <a href="https://ksvmuralidhar.in/">https://ksvmuralidhar.in/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6ca3e1962d2b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/100-stacked-charts-in-python-6ca3e1962d2b">100% stacked charts in Python</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to derive summary statistics using PostgreSQL]]></title>
            <link>https://medium.com/data-science/how-to-derive-summary-statistics-using-postgresql-742f3cdc0f44?source=rss-ce697e8ed35c------2</link>
            <guid isPermaLink="false">https://medium.com/p/742f3cdc0f44</guid>
            <category><![CDATA[database]]></category>
            <category><![CDATA[summary-statistics]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[postgres]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[KSV Muralidhar]]></dc:creator>
            <pubDate>Wed, 06 Oct 2021 12:48:41 GMT</pubDate>
            <atom:updated>2023-07-07T18:49:53.717Z</atom:updated>
            <content:encoded><![CDATA[<h4>Deriving summary statistics of numerical and categorical fields using PostgreSQL</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OyeNmhElsADd237nBkeoMQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@campaign_creators?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Campaign Creators</a> on <a href="https://unsplash.com/s/photos/database?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>In this article, we’ll discuss how to derive summary statistics of numerical and categorical columns/fields using SQL. We’ll use the <strong>Netflix Movies and TV Shows</strong> dataset that’s downloaded from <strong>Tableau Public sample datasets (</strong><a href="https://public.tableau.com/en-us/s/resources">https://public.tableau.com/en-us/s/resources</a> and go to Sample Data<strong>). </strong>This dataset consists of TV shows and movies available on Netflix as of 2019 and is sourced by Tableau from <a href="https://www.kaggle.com/shivamb/netflix-shows"><strong>Kaggle</strong></a> and the updated version of the dataset can be found on Kaggle. However, for this article, we’ll use the older version available in <strong>Tableau sample datasets</strong> and import it into <strong>Postgres</strong>. This article assumes that the reader has basic knowledge of SQL queries and statistics. Below is a snapshot of the <strong><em>netflix_titles </em></strong>that we’ll be using throughout this article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lYAN_nPiFadylMguvB9ihA.png" /><figcaption>netflix_titles table (Image by author)</figcaption></figure><h3>Summary statistics of numerical variables</h3><p>Frequently used summary statistics for numerical variables are mean, median, minimum, maximum, range, standard deviation, variance, Q1, Q3, IQR and skewness.</p><h4><strong>Mean</strong></h4><p>In PostgreSQL, the mean of a numerical field/column is computed using the <strong>AVG() </strong>function. We’ll compute the mean of <strong><em>duration_minutes</em></strong> field as shown below.</p><pre><strong>SELECT</strong> <br> <strong>AVG</strong>(duration_minutes) <strong>AS</strong> mean<br>  <strong>FROM</strong> netflix_titles;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/187/1*7dgBbVWqXYMtE_c9L0Y5Lw.png" /><figcaption>Image by author</figcaption></figure><h4><strong>Median</strong></h4><p>In PostgreSQL, there is no function to directly compute the median of a numerical field/column. However, since median is the 50th percentile, we can use it as a proxy to median. Percentile of a numerical variable is computed using the <strong>PERCENTILE_CONT() </strong>function. We’ll compute the median of <strong><em>duration_minutes</em></strong> field as shown below.</p><pre><strong>SELECT <br> PERCENTILE_CONT</strong>(0.5) <strong>WITHIN GROUP</strong> (<strong>ORDER BY</strong> duration_minutes) <strong>AS</strong> median<br>  <strong>FROM</strong> netflix_titles;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/153/1*xotHffQ8mJzMeH0k6uaEPA.png" /><figcaption>Image by author</figcaption></figure><p><strong>WITHIN GROUP</strong> clause creates an ordered subset of data that can be used to perform aggregations. <strong>PERCENTILE_CONT </strong>takes the percentile required as an argument, in this case it is 0.5 i.e. the 50th percentile.</p><h4><strong>Minimum</strong></h4><p>In PostgreSQL, the minimum value of a numerical field/column is found using the <strong>MIN() </strong>function. We’ll find the minimum value of <strong><em>duration_minutes</em></strong> field as shown below.</p><pre><strong>SELECT </strong><br> <strong>MIN</strong>(duration_minutes) <strong>AS</strong> min <br>  <strong>FROM</strong> netflix_titles;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/86/1*pEMwwPRJq-307lYLPrbOhg.png" /><figcaption>Image by author</figcaption></figure><h4><strong>Maximum</strong></h4><p>In PostgreSQL, the maximum value of a numerical field/column is found using the <strong>MAX() </strong>function. We’ll find the maximum value of <strong><em>duration_minutes</em></strong> field as shown below.</p><pre><strong>SELECT </strong><br> <strong>MAX</strong>(duration_minutes) <strong>AS</strong> max <br>  <strong>FROM</strong> netflix_titles;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/82/1*OZuHHrkjGAD09aqlLU4fwQ.png" /><figcaption>Image by author</figcaption></figure><h4>Range</h4><p>In PostgreSQL, there is no function to directly compute the range of a numerical field/column. However, since range is the difference between maximum and minimum values, we can use it as a proxy to range.</p><pre><strong>SELECT </strong><br> <strong>MAX</strong>(duration_minutes) - <strong>MIN</strong>(duration_minutes) <strong>AS</strong> range<br>    <strong>FROM</strong> netflix_titles;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/85/1*FRd2Q-qFS1i9ttsEDg2gjg.png" /><figcaption>Image by author</figcaption></figure><h4>Standard deviation</h4><p>In PostgreSQL, the standard deviation of a numerical field/column is computed using the <strong>STDDEV() </strong>function. We’ll compute the standard deviation of <strong><em>duration_minutes</em></strong> field as shown below.</p><pre><strong>SELECT</strong> <br> <strong>ROUND</strong>(<strong>STDDEV</strong>(duration_minutes), 2) <strong>AS</strong> standard_deviation<br>    <strong>FROM</strong> netflix_titles;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/172/1*CVrRH4E4C58qzfXKofOI-A.png" /><figcaption>Image by author</figcaption></figure><p><strong>OR</strong></p><p>We can also compute standard deviation as the square root of variance as shown below.</p><pre><strong>SELECT </strong><br> <strong>ROUND</strong>(<strong>SQRT</strong>(<strong>VARIANCE</strong>(duration_minutes)), 2) <strong>AS</strong> stddev_using_variance<br>    <strong>FROM</strong> netflix_titles;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/196/1*Ifk845plNEBHZA2oIeSMZg.png" /><figcaption>Image by author</figcaption></figure><h4>Variance</h4><p>In PostgreSQL, the variance of a numerical field/column is computed using the <strong>VARIANCE() </strong>function. We’ll compute the variance of <strong><em>duration_minutes</em></strong> field as shown below.</p><pre><strong>SELECT </strong><br> <strong>ROUND</strong>(<strong>VARIANCE</strong>(duration_minutes), 2) <strong>AS</strong> variance<br>    <strong>FROM</strong> netflix_titles;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/102/1*EbZx4D4mWVJR56v1BoUrYA.png" /><figcaption>Image by author</figcaption></figure><p><strong>OR</strong></p><p>We can also compute variance as the square of standard deviation as shown below.</p><pre><strong>SELECT </strong><br> <strong>ROUND</strong>(<strong>POWER</strong>(<strong>STDDEV</strong>(duration_minutes), 2), 2) <strong>AS</strong> variance_using_stddev<br>    <strong>FROM</strong> netflix_titles;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/195/1*xiYBV1iL1ucruvqHQd-Mfg.png" /><figcaption>Image by author</figcaption></figure><h4><strong>Q1</strong></h4><p>In PostgreSQL, there is no function to directly compute the first quartile (Q1) of a numerical field/column. However, since Q1 is the 25th percentile, we can use it as a proxy to Q1. We’ll compute the Q1 of <strong><em>duration_minutes</em></strong> field as shown below.</p><pre><strong>SELECT</strong><br> <strong>PERCENTILE_CONT</strong>(0.25) <strong>WITHIN GROUP</strong> (<strong>ORDER BY</strong> duration_minutes) <strong>AS</strong> q1<br>    <strong>FROM</strong> netflix_titles;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/157/1*wRNBzUJnr6rrHR1dl_mzWA.png" /><figcaption>Image by author</figcaption></figure><h4>Q3</h4><p>In PostgreSQL, there is no function to directly compute the third quartile (Q3) of a numerical field/column. However, since Q3 is the 75th percentile, we can use it as a proxy to Q3. We’ll compute the Q3 of <strong><em>duration_minutes</em></strong> field as shown below.</p><pre><strong>SELECT</strong><br> <strong>PERCENTILE_CONT</strong>(0.75) <strong>WITHIN GROUP</strong> (<strong>ORDER BY</strong> duration_minutes) <strong>AS</strong> q3<br>    <strong>FROM</strong> netflix_titles;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/153/1*QidaVm9tBTKbnIunhLsFKw.png" /><figcaption>Image by author</figcaption></figure><h4>IQR</h4><p>In PostgreSQL, there is no function to directly compute the interquartile range (IQR) of a numerical field/column. However, since IQR is the difference between Q3 and Q1, we can use it as a proxy to IQR. We’ll compute the IQR of <strong><em>duration_minutes</em></strong> field as shown below.</p><pre><strong>SELECT</strong><br> <strong>PERCENTILE_CONT</strong>(0.75) <strong>WITHIN GROUP</strong> (<strong>ORDER BY</strong> duration_minutes) -<br> <strong>PERCENTILE_CONT</strong>(0.25) <strong>WITHIN GROUP</strong> (<strong>ORDER BY</strong> duration_minutes) <strong>AS</strong> iqr<br>     <strong>FROM</strong> netflix_titles;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/153/1*Wi5WFJst9aAQ4Z-AOUP8XA.png" /><figcaption>Image by author</figcaption></figure><h4>Skewness</h4><p>In PostgreSQL, there is no function to directly compute the skewness of a numerical field/column. However, since skewness is <strong>3 * (mean-median) / standard_deviation</strong>, we can use it as a proxy to skewness. We’ll compute the skewness of <strong><em>duration_minutes</em></strong> field as shown below. To keep the code more readable we’ll use CTE to compute the mean, median and standard deviation.</p><pre><strong>WITH</strong> mean_median_sd <strong>AS</strong><br>(<br> <strong>SELECT</strong> <br>  <strong>AVG</strong>(duration_minutes) <strong>AS</strong> mean,<br>  <strong>PERCENTILE_CONT</strong>(0.5) <strong>WITHIN GROUP</strong> (<strong>ORDER BY</strong> duration_minutes) <strong>AS</strong> median,<br>  <strong>STDDEV</strong>(duration_minutes) <strong>AS</strong> stddev<br>   <strong>FROM</strong> netflix_titles<br>)</pre><pre><strong>SELECT</strong> <br> <strong>ROUND</strong>(3 * (mean - median)<strong>::NUMERIC</strong> / stddev, 2) <strong>AS</strong> skewness<br>  <strong>FROM</strong> mean_median_sd;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/112/1*q0zjJEkVgxiV1Phc_6ZPHQ.png" /><figcaption>Image by author</figcaption></figure><h4><strong>Putting it all together</strong></h4><p>We’ll put all the functions discussed earlier together and return them in a single table. In the below query, we’ll create two CTEs namely <strong><em>summary_stats </em></strong>(which returns the previously discussed statistics one per each column) and <strong><em>row_summary_stats </em></strong>(which returns the statistics one per each row).</p><pre><strong>WITH RECURSIVE</strong><br>summary_stats <strong>AS</strong><br>(<br> <strong>SELECT</strong> <br>  <strong>ROUND</strong>(<strong>AVG</strong>(duration_minutes), 2) <strong>AS</strong> mean,<br>  <strong>PERCENTILE_CONT</strong>(0.5) <strong>WITHIN GROUP</strong> (<strong>ORDER BY</strong> duration_minutes) <strong>AS</strong> median,<br>  <strong>MIN</strong>(duration_minutes) <strong>AS</strong> min,<br>  <strong>MAX</strong>(duration_minutes) <strong>AS</strong> max,<br>  <strong>MAX</strong>(duration_minutes) - <strong>MIN</strong>(duration_minutes) <strong>AS</strong> range,<br>  <strong>ROUND</strong>(<strong>STDDEV</strong>(duration_minutes), 2) <strong>AS</strong> standard_deviation,<br>  <strong>ROUND</strong>(<strong>VARIANCE</strong>(duration_minutes), 2) <strong>AS</strong> variance,<br>  <strong>PERCENTILE_CONT</strong>(0.25) <strong>WITHIN GROUP</strong> (<strong>ORDER BY</strong> duration_minutes) <strong>AS</strong> q1,<br>  <strong>PERCENTILE_CONT</strong>(0.75) <strong>WITHIN GROUP</strong> (<strong>ORDER BY</strong> duration_minutes) <strong>AS</strong> q3<br>   <strong>FROM</strong> netflix_titles<br>),</pre><pre>row_summary_stats <strong>AS</strong><br>(<br><strong>SELECT</strong> <br> 1 <strong>AS</strong> sno, <br> &#39;mean&#39; <strong>AS</strong> statistic, <br> mean <strong>AS</strong> value <br>  <strong>FROM</strong> summary_stats<br><strong>UNION<br>SELECT</strong> <br> 2, <br> &#39;median&#39;, <br> median <br>  <strong>FROM</strong> summary_stats<br><strong>UNION<br>SELECT</strong> <br> 3, <br> &#39;minimum&#39;, <br> min <br>  <strong>FROM</strong> summary_stats<br><strong>UNION<br>SELECT <br></strong> 4, <br> &#39;maximum&#39;, <br> max <br>  <strong>FROM</strong> summary_stats<br><strong>UNION<br>SELECT</strong> <br> 5, <br> &#39;range&#39;, <br> range <br>  <strong>FROM</strong> summary_stats<br><strong>UNION<br>SELECT <br></strong> 6, <br> &#39;standard deviation&#39;, <br> standard_deviation <br>  <strong>FROM</strong> summary_stats<br><strong>UNION<br>SELECT <br></strong> 7, <br> &#39;variance&#39;, <br> variance <br>  <strong>FROM</strong> summary_stats<br><strong>UNION<br>SELECT <br></strong> 9, <br> &#39;Q1&#39;, <br> q1 <br>  <strong>FROM</strong> summary_stats<br><strong>UNION<br>SELECT <br></strong> 10, <br> &#39;Q3&#39;, <br> q3 <br>  <strong>FROM</strong> summary_stats<br><strong>UNION<br>SELECT <br></strong> 11, <br> &#39;IQR&#39;, <br> (q3 - q1) <br>  <strong>FROM</strong> summary_stats<br><strong>UNION<br>SELECT <br></strong> 12, <br> &#39;skewness&#39;, <br> <strong>ROUND</strong>(3 * (mean - median)<strong>::NUMERIC</strong> / standard_deviation, 2) <strong>AS</strong> skewness <br>  <strong>FROM</strong> summary_stats<br>)</pre><pre><strong>SELECT</strong> * <br> <strong>FROM</strong> row_summary_stats<br>  <strong>ORDER BY</strong> sno;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/395/1*iqTAURv7XpGGTzJmREmiFA.png" /><figcaption>Image by author</figcaption></figure><h3>Summary statistics of categorical variables</h3><p>Frequently used summary statistics for categorical variables are mode, cardinality (number of unique categories), frequency and relative frequency.</p><h4>Mode</h4><p>In PostgreSQL, mode of a categorical field/column is computed using the <strong>MODE() </strong>function. We’ll compute the mode of <strong><em>rating</em></strong> field as shown below.</p><pre><strong>SELECT </strong><br> <strong>MODE</strong>() <strong>WITHIN GROUP</strong> (<strong>ORDER BY</strong> rating) <strong>AS</strong> mode<br>  <strong>FROM</strong> netflix_titles;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/82/1*9E1lTHJBW6iBh4wZUPhL_A.png" /><figcaption>Image by author</figcaption></figure><h4>Cardinality</h4><p>In PostgreSQL, there is no function to directly compute the number of unique values of a categorical field/column. However, we can count the distinct values of a variable as shown below.</p><pre><strong>SELECT</strong> <br> <strong>COUNT</strong>(<strong>DISTINCT</strong> rating) <strong>AS</strong> cardinality<br>  <strong>FROM</strong> netflix_titles;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/113/1*QhsWyKrJO1zB5PKK-vr4hQ.png" /><figcaption>Image by author</figcaption></figure><h4>Frequency and relative frequency</h4><p>In Postgres, we can compute the frequency of the categories in a categorical variable using <strong>GROUP BY </strong>and<strong> COUNT</strong>. For relative frequency, we’ll use a CTE to count the total number of values in rating field and use it to compute the relative frequency. Since, not all databases support window functions we’ll use CTE. We’ll also discuss how to calculate relative frequency using window functions.</p><pre><strong>WITH</strong> total_count <strong>AS</strong><br>(<br><strong>SELECT</strong><br> <strong>COUNT</strong>(rating) <strong>AS</strong> total_cnt<br>  <strong>FROM</strong> netflix_titles<br>)</pre><pre><strong>SELECT</strong> <br> rating,<br> <strong>COUNT</strong>(rating) <strong>AS</strong> frequency,<br> <strong>ROUND</strong>(<strong>COUNT</strong>(rating)<strong>::NUMERIC</strong> / <br>    (<strong>SELECT</strong> total_cnt <strong>FROM</strong> total_count), 4) <strong>AS</strong> relative_frequency<br>    <strong>FROM</strong> netflix_titles<br>     <strong>GROUP BY</strong> rating<br>   <strong>ORDER BY </strong>frequency <strong>DESC</strong>;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/372/1*WRcftJVu02B-T_R--ACfBQ.png" /><figcaption>Image by author</figcaption></figure><p>In the above example, we’ve created a CTE that captures the count of values in the <strong><em>rating</em></strong> field. We then used it to calculate the percentage/relative frequency of each category in the <strong><em>rating</em></strong> field. Since, Postgres supports window functions, we’ll see a simpler way of calculating relative frequency using window functions. We’ll use <strong>OVER() </strong>to compute the sum of the counts of rating across each group which in turn is the total number of values in the <strong><em>rating</em></strong> field.</p><pre><strong>SELECT</strong> <br> rating,<br> <strong>COUNT</strong>(rating) <strong>AS</strong> frequency,<br> <strong>ROUND</strong>(<strong>COUNT</strong>(rating)<strong>::NUMERIC</strong> / <strong>SUM</strong>(<strong>COUNT</strong>(rating)) <strong>OVER()</strong>, 4) <strong>AS</strong> relative_frequency<br>    <strong>FROM</strong> netflix_titles<br>     <strong>GROUP BY</strong> rating<br>      <strong>ORDER BY</strong> frequency <strong>DESC</strong>;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/383/1*75AGq-iX0jsOqvdZ339CSQ.png" /><figcaption>Image by author</figcaption></figure><p>This brings the article to an end. We’ve discussed various functions to compute the frequently used summary statistics of numerical fields like mean, median, min, max, Q1, Q2, IQR, etc. and of categorical fields like cardinality, frequency, etc. Summary statistics can be used to get an idea of the data distribution and also aid in identifying potential outliers.</p><p>Know more about my work at <a href="https://ksvmuralidhar.in/">https://ksvmuralidhar.in/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=742f3cdc0f44" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-derive-summary-statistics-using-postgresql-742f3cdc0f44">How to derive summary statistics using PostgreSQL</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Dual Axis plots in Python]]></title>
            <link>https://medium.com/data-science/dual-axis-plots-in-python-95e5ef24a4e0?source=rss-ce697e8ed35c------2</link>
            <guid isPermaLink="false">https://medium.com/p/95e5ef24a4e0</guid>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[matplotlib]]></category>
            <category><![CDATA[plotting]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[dual-axis]]></category>
            <dc:creator><![CDATA[KSV Muralidhar]]></dc:creator>
            <pubDate>Fri, 24 Sep 2021 10:35:48 GMT</pubDate>
            <atom:updated>2023-07-07T18:50:14.326Z</atom:updated>
            <content:encoded><![CDATA[<h4>This article discusses how to add a secondary axis to Python plots</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IWKTDXElpzLDMMnFLdn4Zw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@chrisliverani?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Chris Liverani</a> on <a href="https://unsplash.com/s/photos/graphs?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>At times, we may need to add two variables with different scale to an axis of a plot. For example, we want to have GDP per capita (in $) and annual GDP growth % in the y-axis and year in the x-axis. In other words, we need to visualize the trend in GDP per capita ($) and GDP growth rate across years. Since, GDP per capita ($) and GDP growth rate have different scale. Plotting both of them using the same y-axis would undermine the other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/918/1*lXbvNrVeojegvqXrTcl9bA.png" /><figcaption>Image by author</figcaption></figure><p>In the above plot, we can see that the trend in Annual Growth Rate is completely undermined by the GDP per capita ($). In the plot below, we see that using a logarithmic scale in y-axis also didn’t help.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*MkJTI1b12KH4ysEP9yWnvA.png" /><figcaption>Logarithmic y-axis (Image by author)</figcaption></figure><p>This makes it essential to have a secondary y-axis for Annual growth rate (%). to illustrate the addition of a secondary axis, we’ll use the data frame (named ‘gdp’) shown below containing GDP per capita ($) and Annual growth rate (%) data from the year 2000 to 2020.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/485/1*NuDpfLGXvWOabjDrZIVVFQ.png" /><figcaption>gdp data frame (Image by author)</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/efde17bb23e610144ea7259b6488144b/href">https://medium.com/media/efde17bb23e610144ea7259b6488144b/href</a></iframe><p>In the above code, we have created a secondary axis named ‘ax2’ using twinx() function. twinx() creates a secondary axes with shared x-axis. There is another function named twiny() used to create a secondary axis with shared y-axis. In the plot shown below, we can clearly see the trend in both GDP per capita ($) and Annual growth rate (%).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jXAYTgS6Wlfbe4koJHn-vg.png" /><figcaption>Plot with secondary axis (Image by author)</figcaption></figure><p>In the next example, we’ll plot the trend in Nifty (a stock index in India) along with the volume. In this example, we’ll use line plot for index value and bar plot for volume. Below are the first few records of the data frame (named ‘nifty_2021’) that we’ll use in this example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/312/1*7RoOWWCap-XsINeaZHXXrA.png" /><figcaption>nifty_2021 data frame (Image by author)</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/00f5c563a78706a30aa43440e70b8df9/href">https://medium.com/media/00f5c563a78706a30aa43440e70b8df9/href</a></iframe><p>The above code is similar to the one we saw previously. However, there are a few differences to note.</p><ol><li>In the above code, we have used pandas plot() to plot the volume bar plot. This is because Matplotlib’s plt.bar() function may not work properly with plots of different types. Hence, I prefer Matplotlib only for a line plot.</li><li>We have used <strong>ax2.plot(ax.get_xticks() </strong>instead of<strong> ax2.plot(nifty_2021[‘Date’]</strong>. This is because using <strong>nifty_2021[‘Date’]</strong> is moving the plot to the right or not displaying the line plot at all.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U_b2K-_cMr_qeflRD-GAUw.png" /><figcaption>Image by author</figcaption></figure><p>This brings this article to an end. We’ve discussed how variables with different scale may pose a problem in plotting them together and saw how adding a secondary axis solves the problem. We’ve also seen how to plot a line and bar plot using secondary axis.</p><p>Know more about my work at <a href="https://ksvmuralidhar.in/">https://ksvmuralidhar.in/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=95e5ef24a4e0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/dual-axis-plots-in-python-95e5ef24a4e0">Dual Axis plots in Python</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Build an RSS feed-based News app with Streamlit, Heroku and Android Studio?]]></title>
            <link>https://medium.com/swlh/how-to-build-an-rss-feed-based-news-app-with-streamlit-heroku-and-android-studio-cc4ebb1077e?source=rss-ce697e8ed35c------2</link>
            <guid isPermaLink="false">https://medium.com/p/cc4ebb1077e</guid>
            <category><![CDATA[streamlit]]></category>
            <category><![CDATA[heroku]]></category>
            <category><![CDATA[android-app-development]]></category>
            <category><![CDATA[news-aggregator]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[KSV Muralidhar]]></dc:creator>
            <pubDate>Tue, 17 Aug 2021 13:19:17 GMT</pubDate>
            <atom:updated>2023-07-07T18:50:59.192Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to build an RSS feed-based News app with Streamlit, Heroku and Android Studio?</h3><h4>Build an Android app that aggregates news from your favorite news websites. This article is for Python developers and needs no Android development experience.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/346/1*Ine0CQqdrcDyb_D9RB0LGA.png" /><figcaption>Image by author</figcaption></figure><h3>What are RSS Feeds?</h3><p>RSS stands for Really Simply Syndication. RSS feeds are standardized XML files that enable users to access updates from websites. Below is the structure of a typical RSS feed. RSS feeds may also have a few additional elements like image of the news article, etc. In this article, we’ll use RSS feeds and aggregate news updates from multiple websites.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/12419b0b57b11eb5af02d30f30ba7f44/href">https://medium.com/media/12419b0b57b11eb5af02d30f30ba7f44/href</a></iframe><h3>Building the News Aggregator App</h3><p>In this article, we’ll build a news aggregator app using Python (Streamlit), deploy it to Heroku and access it from Heroku via an Android app. Below is the sequence of steps in the app development process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/546/1*MA-JmXiPkVSXBLwwpOy3wQ.png" /><figcaption>Sequence of steps (Image by author)</figcaption></figure><p>Before proceeding, we’ll look at how the Android app works in real time on a smartphone. Looking at the app before getting into details may help in a better understanding.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/234/1*7059DQTSKrB2i8ez338-5w.gif" /><figcaption>Android app demo (Image by author)</figcaption></figure><p>Upon opening the Android app, we see that the browser is connecting to the Streamlit app deployed to Heroku and the app is running. We’ll discuss each step of the app development process in detail, starting with Streamlit app development.</p><h4>Step 1: Streamlit app development</h4><p>It is the first and the most important step which does all the processing stuff. We’ll breakdown the to understand it better. <strong>Each snippet is explained clearly using comments and doc strings.</strong></p><ol><li><strong>Importing libraries</strong></li></ol><p>We’ll import the standard libraries for reading RSS feeds. <strong>dateutil </strong>package is used to parse dates from string to timestamps.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7d63c68debdd311652e9694c816e9855/href">https://medium.com/media/7d63c68debdd311652e9694c816e9855/href</a></iframe><p><strong>2. Initialize the list of RSS feeds to capture</strong></p><p>Below is a list of RSS feeds to track. I’m interested in stock markets, so I wanted to track the business news from top business news websites in India.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dd4a42d24cc73401e4c35a6b3de3933b/href">https://medium.com/media/dd4a42d24cc73401e4c35a6b3de3933b/href</a></iframe><p><strong>3. Function to calculate the elapsed time</strong></p><p>Elapsed time is the difference between the current time and the time when the news article was published. This function computes the time elapsed (in minutes) since the article was published.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c03b33b9e7b08865e623e3e587beb745/href">https://medium.com/media/c03b33b9e7b08865e623e3e587beb745/href</a></iframe><p><strong>4. Function to convert the elapsed time into words</strong></p><p>The above function computes the elapsed time in minutes. This function converts the elapsed time in minutes to word form (example: 2 minutes ago, 2 hours ago, etc.).</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/edbba6659ef478e4810e320233b939e4/href">https://medium.com/media/edbba6659ef478e4810e320233b939e4/href</a></iframe><p><strong>5. Function to clean the title and description of a news article</strong></p><p>There may be cases where few special characters in HTML like &amp;nbsp; &amp;quot; may not be properly parsed. This function replaces such unparsed HTML special characters with characters of the English language.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ce9c5508053f0260b2bda919733611f5/href">https://medium.com/media/ce9c5508053f0260b2bda919733611f5/href</a></iframe><p><strong>6. Function to extract the source of the news from RSS feed’s URL</strong></p><p>Every news article has a source i.e. the publisher. This function extracts the source (domain name of the news website) from the RSS feed URL.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d4229a600e3a97a0ecd63a72e56fc4e7/href">https://medium.com/media/d4229a600e3a97a0ecd63a72e56fc4e7/href</a></iframe><p><strong>7. Function to process individual news item of an RSS feed</strong></p><p>RSS feeds embed each news article within the &lt;item&gt; tag. This function extracts the required data (title, description, published date, etc.) from the news item and stores it in a data frame.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2ca7dca0b1fc79bc39569cbf58df9413/href">https://medium.com/media/2ca7dca0b1fc79bc39569cbf58df9413/href</a></iframe><p><strong>8. Function to parse each RSS feed</strong></p><p>The above function parses each news item (&lt;item&gt;) of an RSS feed. While this function processes each RSS feed by calling the functions declared above and stores the results into a data frame.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dfc988308afd59c196fdd75719152d4d/href">https://medium.com/media/dfc988308afd59c196fdd75719152d4d/href</a></iframe><p><strong>9. Snippet to process, store and clean data from all the RSS feeds into a data frame</strong></p><p>This step traverses through all the RSS feeds declared as a list and stores it in a final data frame having all the news articles from all the RSS feeds. This step also does some final preprocessing and deletes few unwanted columns from the data frame.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ceb4f1086985d95f8d672a3e03109b88/href">https://medium.com/media/ceb4f1086985d95f8d672a3e03109b88/href</a></iframe><p><strong>10. Snippet to build the front end of the app</strong></p><p>This step builds the frontend HTML script used to display the news articles. A loop traverses through each row of the data frame containing all the news articles and inserts each article into a row of a HTML table.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7fb70c00645a29172c86eee357f188ab/href">https://medium.com/media/7fb70c00645a29172c86eee357f188ab/href</a></iframe><h4>Step 2: Deploy the Streamlit app to Heroku</h4><p>Deploying the Streamlit app to ‘Streamlit sharing’ and using it in the Android app isn’t working. Hence, we’ll deploy the Streamlit app to Heroku. The article below clearly demonstrates the process of deploying a Streamlit app to Heroku.</p><p><a href="https://towardsdatascience.com/deploying-a-basic-streamlit-app-to-heroku-be25a527fcb3">Deploying a basic Streamlit app to Heroku</a></p><p>The process of deploying the app to Heroku stays the same as demonstrated in the above article. However, there are a few modifications required to the deployment files which are shown below. <strong>setup.sh </strong>remains the same as in the above article.</p><p><strong>Folder Structure</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/196/1*KbWgVvnNjXpJitCv08fcmA.png" /><figcaption>Image by author</figcaption></figure><p><strong>news_rss.py </strong>is the Python script of the Streamlit app built by placing all the snippets of the <strong>Step 1: Streamlit app development</strong> into it, in the same order.</p><p><strong>requirements.txt</strong></p><pre>streamlit==0.86.0<br>regex==2021.8.3<br>lxml==4.6.3<br>numpy==1.21.1<br>python-dateutil==2.8.2<br>pandas==1.3.1<br>requests==2.26.0<br>bs4==0.0.1</pre><p><strong>Procfile</strong></p><pre>web: sh setup.sh &amp;&amp; streamlit run news_rss.py</pre><h4>Step 3: Building the Android app</h4><p><strong>I have no experience with Android app development (hence, this article shouldn’t be considered as an Android app development tutorial)</strong>. However, I know that it is possible to convert a website into an Android app. As a beginner, there were a few questions raised in my mind. We’ll discuss them one by one.</p><p><strong>1. Which IDE do I use to build an Android app?</strong></p><p>After Googling, I found many IDEs for android development. I decided to go with the official IDE for Android development, Android Studio.</p><p><strong>2. How to start building an app in Android Studio that connects to a website?</strong></p><p>After googling, I found a YouTube video and the <a href="https://developer.android.com/guide/webapps/webview#java">official documentation</a> demonstrating it. I created the project as an ‘Empty Activity’ and named it as ‘MyNews’.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FTUXui5ItBkM&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DTUXui5ItBkM&amp;image=http%3A%2F%2Fi.ytimg.com%2Fvi%2FTUXui5ItBkM%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/6f349ea1bfabe7656b23f64ea267ab0e/href">https://medium.com/media/6f349ea1bfabe7656b23f64ea267ab0e/href</a></iframe><p><strong>3. How do I test the app?</strong></p><p>I found the <a href="https://developer.android.com/studio/run/managing-avds">official documentation</a> of creating an emulator to test the app. Emulator is a virtual device that replicates an original one. It enables us to test the app without actually installing it on a physical device.</p><p><strong>3. Why isn’t my app loading the web page?</strong></p><p>I tested the app and found the WebView was not loading the web page. I found the <a href="https://stackoverflow.com/questions/16207094/android-webview-not-loading-url">solution</a> in Stackoverflow.</p><p><strong>4. How do I add a refresh button?</strong></p><p>Once the app is working fine, I wanted to add a refresh button to refresh the page, when intended. Again, I found the <a href="https://stackoverflow.com/questions/32738756/android-webview-refresh-button">solution</a> in Stackoverflow.</p><p><strong>5. How do I change the app’s theme?</strong></p><p>The default theme of an Android app is purple i.e. the refresh button and the action bar look purple. But, I wanted to change it to black. I found the <a href="https://stackoverflow.com/questions/19186235/how-to-change-app-default-theme-to-a-different-app-theme">solution</a> in Stackoverflow. I’ve changed the colors in <strong>themes.xml</strong> file.</p><p><strong>6. How to change the title of the app displayed on the activity bar (banner)?</strong></p><p>I found the <a href="https://stackoverflow.com/questions/34610219/change-app-name-in-android-studio">solution</a> in Stackoverflow.</p><p><strong>7. Finally, how do I create the .apk file of my app?</strong></p><p>As the app was running fine in the emulator tests, I wanted to create an apk file to install it on my smartphone. I found the way <a href="https://www.educative.io/edpresso/extracting-an-apk-file-from-android-studio">here</a>.</p><p>Below are the Android Studio files of the app. These are only the files which I’ve modified (all the other files need not be modified).</p><p><strong>activity_main.xml</strong>: Specifies the layout of the app (interface)</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/250437a8bb7205b215d29e88ebdd013e/href">https://medium.com/media/250437a8bb7205b215d29e88ebdd013e/href</a></iframe><p><strong>MainActivity.java: </strong>Runs when the app is opened. It also adds actions to the components of the interface.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/055aa2385fa4f3431c4de07c17b48e13/href">https://medium.com/media/055aa2385fa4f3431c4de07c17b48e13/href</a></iframe><p><strong>AndroidManifest.xml: </strong>Holds metadata of the app.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f4bb8e5cc5131de4a24051442bd7b046/href">https://medium.com/media/f4bb8e5cc5131de4a24051442bd7b046/href</a></iframe><p><strong>themes.xml: </strong>Specifies the app’s theme.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1473ed7dc8cfb9c669ef3b6dd6add924/href">https://medium.com/media/1473ed7dc8cfb9c669ef3b6dd6add924/href</a></iframe><p><strong>strings.xml: </strong>Specifies app name, developer info, etc.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/703ccc4a32722d2917f298149bc7459a/href">https://medium.com/media/703ccc4a32722d2917f298149bc7459a/href</a></iframe><p>This is an example of building a news aggregator Android app for Python developers with no Android development experience. The same process can be followed to convert any Streamlit app into an Android app. This app is only for one’s personal use and doesn’t have personalized content capabilities (which is required to support multiple users). If the RSS feeds are used for commercial purposes, developers must get the consent of the feed’s publishers before proceeding further.</p><p>Know more about my work at <a href="https://ksvmuralidhar.in/">https://ksvmuralidhar.in/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc4ebb1077e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/how-to-build-an-rss-feed-based-news-app-with-streamlit-heroku-and-android-studio-cc4ebb1077e">How to Build an RSS feed-based News app with Streamlit, Heroku and Android Studio?</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Search Engine Scraper with Streamlit]]></title>
            <link>https://ksvmuralidhar.medium.com/building-a-search-engine-scraper-with-streamlit-b616e5bd293c?source=rss-ce697e8ed35c------2</link>
            <guid isPermaLink="false">https://medium.com/p/b616e5bd293c</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[search-engine-scraper]]></category>
            <category><![CDATA[data-acquisition]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[streamlit]]></category>
            <dc:creator><![CDATA[KSV Muralidhar]]></dc:creator>
            <pubDate>Tue, 10 Aug 2021 15:17:49 GMT</pubDate>
            <atom:updated>2023-07-07T18:51:19.292Z</atom:updated>
            <content:encoded><![CDATA[<h4>Web scraping is an effective technique to acquire data from the web with minimal manual effort</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/901/1*7pjuZu6g9HJUoMBZZz5D_A.png" /><figcaption>Image by author</figcaption></figure><h4>Web scraping</h4><p>Web scraping enables us to acquire data from the web with minimal manual effort. However, this technique must be used with caution such that it doesn’t degrade the performance of a website. Before proceeding, we’ll discuss a few tips to follow to protect ourselves from being blacklisted by a website as a consequence of degrading its performance.</p><ol><li>We shouldn’t overload a website’s server with requests, as in a DDoS attack, which is an unethical practice. If we overload a website’s server, we may get blacklisted. We can use the <strong>sleep()</strong> function of the <strong>time</strong> package in Python to give random pauses between our requests.</li><li>We must keep on switching user-agents. We may maintain a csv file of user-agents and keep switching them often. We’ll discuss more on user agents later in this article.</li><li>We must keep on saving the scraped data. This ensures that we don’t loose our data when a website blacklists us. We can restart scraping from the last saved checkpoint instead of starting over.</li><li>If possible, we may use the <strong>selenium</strong> package of Python which enables us to mimic human activity like scrolling, clicks, etc. This may lower our chance of being blacklisted.</li><li>We must go through and obey a website’s ‘robots.txt’ to identify the pages prohibited from being accessed and also the user agents that aren’t allowed to access the website. We can access the ‘robots.txt’ using ‘domain/robots.txt’. For example, we can access the robots.txt of bing.com using <a href="http://bing.com/robots.txt">bing.com/robots.txt</a>.</li></ol><h4>Search engine scraper</h4><p>In this article, we’ll build a search engine scraper* using BeautifulSoup and Streamlit. The app takes a search string as an input, scrapes the search results from the first page of Bing and displays them. The app also returns a data frame of the search results. Below is the Python code of the Streamlit app and is clearly explained using comments.</p><p><strong>* This app is built only to illustrate the process of web scraping. It is highly recommended to use the search engine’s API for extracting search results. Web scraping should only be the last resort.</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f59cfbc8024370e57e88a867ef02ee77/href">https://medium.com/media/f59cfbc8024370e57e88a867ef02ee77/href</a></iframe><p>The above code can be run by executing the following command in the terminal of a local machine.</p><pre>streamlit run filename.py</pre><p><strong>filename.py</strong> is the python script file containing the above code. Below is a screenshot of the output we get after executing the command. In this case, the name of the file containing the Python script is <strong>bing_scrape_streamlit.py</strong>. Hence, the command to run the code would be as shown below.</p><pre>streamlit run bing_scrape_streamlit.py</pre><p>Below is the output of the app in a local browser after executing the above command in the terminal of a local machine. We can also see a data frame of the results at the bottom of the output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*XCovyuKIlTXU40yMZoiaCg.png" /><figcaption>Image by author</figcaption></figure><h4>How are the extracted search results useful?</h4><ul><li>To find the website of a list of companies, assuming the first search result (of a company) would mostly be a company’s website. This can be achieved by looping the code over a list of companies and storing the scraped results / responses from API requests. Most API also support bulk requests. However, this may need some further validation, as not every first search result contains a company’s website.</li><li>To find the URLs of the social media pages of people/companies. This can be done by including a person/company’s name and the name of the required social media website in the search string. Site search may be more helpful.</li><li>To find the latest news related to a company.</li></ul><p>All the above mentioned tasks demand a lot of manual effort that increases with the size of the entities to search. Hence, web scraping/APIs reduce this manual effort to a greater extent. However, human-in-the-loop is required to validate the results.</p><p>We’ll discuss a few questions that may arise in the mind of a beginner in web scraping.</p><p><strong>1. What is requests.get()?</strong></p><p>‘requests’ package enables us to handle HTTP requests in Python. Its <strong>get()</strong> function sends a query to the web server and returns a response.</p><p><strong>2. What is a user agent?</strong></p><p>User agent describes the browser and operating environment being used. This enables the web server to send the content that best suits our browser/device. A browser on a mobile device has a different user agent form that of Google Chrome on a Windows system. Switching user agents protects us from being blacklisted as the web server gets requests from multiple devices.</p><p><strong>3. What is BeautifulSoup?</strong></p><p>BeautifulSoup parses HTML/XML documents enabling us to extract data from web pages. We must pass the content/text of a request’s response to the BeautifulSoup class for it to parse. We can then extract the elements/tags of the web page’s HTML script.</p><p><strong>4. How do we select the right HTML tag?</strong></p><p>We can identify the right tag by inspecting the source code of a web page. The <strong>‘inspect’</strong> functionality is available in all the popular web browsers. We’ll discuss how to <strong>‘inspect’</strong> a web page’s source code.</p><p><strong>Step 1:</strong> We’ll right click on the element in a web page that we want to scrape and select <strong>‘Inspect’</strong>. For search engine scraper, we want to scrape the title, URL and description of a search result. Hence, we’ll right click on the URL in the first search result.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/808/1*7OMdnWC6qbKO6-g81MwEuQ.png" /><figcaption>Image by author</figcaption></figure><p><strong>Step 2:</strong> We must select a tag (parent tag) that embeds all the tags (child tags) we want to scrape. For example, we need to scrape the search results, so we’ll select a parent tag that embeds all the tags attached to a single search result. In the figure below, &lt;li class=”b_algo”&gt; embeds all the tags attached to a single search result. Now we can extract each child tag from &lt;li class=”b_algo”&gt; by processing each &lt;li class=”b_algo”&gt; (search result) individually.</p><p>We shouldn’t select only &lt;h2&gt; (title of search result) or &lt;p&gt; (description of a search result). This may pick up other &lt;h2&gt;s and &lt;p&gt;s in the page other than the ones associated with the search results. This may also make it difficult for us to map a list of &lt;h2&gt; to a list of &lt;p&gt;s.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RFP7XJumjKusPI8xaMigPg.png" /><figcaption>Image by author</figcaption></figure><p>Hence, web scraping helps us in reducing the manual effort required for acquiring data from the web. We may use the same code multiple times to extract data from a website, provided the structure of the website stays the same. Code to extract the HTML script from a web page is similar for all the websites. What’s different is the process of cleaning and extracting the required data from the HTML script.</p><p>However, I highly recommend using search engine APIs for extracting search results. Web scraping should only be the last resort. Even if you opt for web scraping, it must be done without degrading the performance of a website and should only be used to extract the publicly available information.</p><p>Know more about my work at <a href="https://ksvmuralidhar.in/">https://ksvmuralidhar.in/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b616e5bd293c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[1/2 - Is it the Probability of passing a test?]]></title>
            <link>https://medium.datadriveninvestor.com/1-2-is-it-the-probability-of-passing-a-test-a559cabb5cf9?source=rss-ce697e8ed35c------2</link>
            <guid isPermaLink="false">https://medium.com/p/a559cabb5cf9</guid>
            <category><![CDATA[theoretical-probability]]></category>
            <category><![CDATA[probability]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[empirical-probability]]></category>
            <dc:creator><![CDATA[KSV Muralidhar]]></dc:creator>
            <pubDate>Sat, 07 Aug 2021 17:09:18 GMT</pubDate>
            <atom:updated>2023-07-07T18:51:38.496Z</atom:updated>
            <content:encoded><![CDATA[<h4>Theoretical probability of passing a test is 1/2 as the sample space, S = {pass, fail}, but the empirical probability might be different.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6_SFcCMtgo22pgeuDXOboA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@rihok?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Riho Kroll</a> on <a href="https://unsplash.com/s/photos/game?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Assume a person is about to take a test. Someone asks him a question, <strong>What is the probability of you passing the test?. </strong>We’ll explore the possible answers to this question by looking at two important concepts of probability viz. <strong>theoretical probability</strong> and <strong>empirical probability</strong>.</p><p><strong>Theoretical Probability</strong></p><p>Theoretical probability assumes every outcome in the sample space, S, has an equal chance of occurrence. Theoretical probability of a person passing an exam is 1/2 or 0.5. This experiment has two possible outcomes namely, “pass” or “fail”, therefore, S = {pass, fail}.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/408/1*34clVEMp1B60OoMoUFqBfA.gif" /><figcaption>Image by author</figcaption></figure><p>From the above formula of probability, P(pass) = 1/2 i.e. a person would fail in half of his exams, which is highly unlikely to happen even after taking thousands of exams (infinite trials). A studious person may have P(pass) = 1. <strong>How do we come up with a more reliable probability measure?</strong> We’ll discuss empirical probability to answer this question.</p><h4>Empirical Probability</h4><p>Empirical probability is computed using the historical data or a sample. The probability of a person passing a test is calculated taking into account his performance in previous tests. For example, a person took 100 tests previously and passed in 98 of them. P(pass) now would be 98/100 (0.98). To be more accurate with the probability estimate, we may only consider a person’s tests that are similar to the current one. We’ll look at some more examples.</p><ol><li>The probability of Team X winning a game against Team Y is not 0.5 (S = {win, lose}). We need to look at the past performance of the team against Team Y and estimate the probability. Assume, the team played 1,000 games against Team Y and won in 650 of them, P(win) = 0.65. To be more accurate with the probability estimate, we may only consider the previous games against Team Y in the recent past.</li><li>Probability of a rain today is not 0.5 (S = {rain, no rain}). Look at the historical data of the weather on the same day. Assume, it rained 10 days of the 100 days considered, P(rain) = 0.1. We may consider the same day of only the last 20–30 years for a more reliable probability estimate.</li><li>The probability of a disease from smoking is not 0.5 (S = {disease, no disease}). We need to consider a random sample of people who smoke and use it to estimate the probability of the disease. Assume, we sampled 1,000 people who smoke and found that 10 people have a disease, P(disease) = 0.01. For a more reliable probability estimate, we may only consider a sample of people with the same demographics and health conditions as that of the subject in the question.</li></ol><p>Theoretical probability is quite different from empirical probability. Empirical probability is a more reliable estimate of probability, since, it is computed using historical data. Hence, it is essential to come up with a probability estimate by analyzing a sample. However, one must remember, probability is only an estimate (not a guarantee) of the chance of an event to occur.</p><p>Know more about my work at <a href="https://ksvmuralidhar.in/">https://ksvmuralidhar.in/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a559cabb5cf9" width="1" height="1" alt=""><hr><p><a href="https://medium.datadriveninvestor.com/1-2-is-it-the-probability-of-passing-a-test-a559cabb5cf9">1/2 - Is it the Probability of passing a test?</a> was originally published in <a href="https://medium.datadriveninvestor.com/">DataDrivenInvestor</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding Autocorrelation in Time Series Analysis]]></title>
            <link>https://medium.com/data-science/understanding-autocorrelation-in-time-series-analysis-322ad52f2199?source=rss-ce697e8ed35c------2</link>
            <guid isPermaLink="false">https://medium.com/p/322ad52f2199</guid>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[statsmodels]]></category>
            <category><![CDATA[autocorrelation]]></category>
            <dc:creator><![CDATA[KSV Muralidhar]]></dc:creator>
            <pubDate>Wed, 04 Aug 2021 14:22:36 GMT</pubDate>
            <atom:updated>2023-07-07T18:51:57.366Z</atom:updated>
            <content:encoded><![CDATA[<h4>Understanding how autocorrelation works is essential for beginners to make their journey in time series analysis easier.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IWKTDXElpzLDMMnFLdn4Zw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@chrisliverani?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Chris Liverani</a> on <a href="https://unsplash.com/s/photos/stock-market?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>I began my time series analysis journey two years ago. Initially, I began learning through YouTube videos where I came across autocorrelation, a basic concept of time series analysis. According to a few videos, the definition of autocorrelation was the relationship/correlation of a time series with its previous versions in time. Few others said, since we need two variables to compute correlation, but in a time series we have only one variable, we need to compute the correlation of the time series with a ‘k’th lagged version of itself.</p><h4><strong>What is ‘k’th lag?</strong></h4><p>This was the first question that came into my mind. As I proceeded, I got to know that a time series (y) with ‘k’th lag is its version that is ‘t-k’ periods behind in time. A time series with lag (k=1) is a version of the original time series that is 1 period behind in time, i.e. y(t-1).</p><p>Most of those videos took an example of the stock market daily prices to explain time series analysis. Those prices were recorded daily. They explained, the autocorrelation of the stock prices is the correlation of the current price with the price ‘k’ periods behind in time. So, the autocorrelation with lag (k=1) is the correlation with today’s price y(t) and yesterday’s price y(t-1). Similarly, for k=2, the autocorrelation is computed between y(t) and y(t-2).</p><h4><strong>Now comes the main question</strong></h4><p>How can we compute the correlation (or even covariance for that matter) of today’s price with yesterday’s price? Since correlation can only be computed between variables with multiple values. If I try to compute correlation between two single values, I’d get an ‘NaN’. Also, the two variables must have same lengths (number of values), so, I can’t even compute the correlation between y(starting point to t) and y(starting point to t-1).</p><p>Then, I started searching for a theoretical explanation of autocorrelation and came across the formula of autocorrelation as shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/238/1*ZbVu3HMeFq4MhloqJ3s7-Q.gif" /><figcaption>Image by author</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/243/1*BxHY6US8q8C6zPxxLmU9eQ.gif" /><figcaption>Image by author</figcaption></figure><h3><strong>Understanding the formula</strong></h3><ul><li>The formula of autocorrelation is similar (but not exactly the same) to that of correlation.</li><li>The numerator is similar to covariance between the current and lagged versions of the time series (but doesn’t have ‘N-1’ as denominator). A closer examination of the two components of the numerator shows that the mean of the original time series, mean(y), is being subtracted from them, not mean(y(t)) and mean(y(t-k)), respectively. This makes the numerator of the formula a bit different from covariance.</li><li>The denominator is similar to the square of standard deviation (a.k.a. variance) of the original time series (but doesn’t have ‘N-1’ as denominator).</li></ul><p><strong>Let’s answer the question, How to compute autocorrelation? by implementing it in Python</strong></p><p>We’ll use the Nifty (an Indian stock index tracking 50 stocks) closing price data from 17 September, 2007 to 30 July, 2021. The data is downloaded as a csv from Yahoo Finance. We’ll first prepare the data for time series analysis.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/74ec7651af1b61dbc6c4419d711268db/href">https://medium.com/media/74ec7651af1b61dbc6c4419d711268db/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sLBS2c3n9ZLXtxLyij9ekw.png" /><figcaption>Nifty time series plot (Image by author)</figcaption></figure><p>We’ll define a function called ‘autocorr’ that returns the autocorrelation (acf) for a single lag by taking a time series array and ‘k’th lag value as inputs. This function will be nested inside another function called ‘my_auto_corr’ that returns acf for lags [k,0] by calling ‘autocorr’ function to compute acf for each lag value.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/84413e786cd96ea9ccaf4b99b8db0488/href">https://medium.com/media/84413e786cd96ea9ccaf4b99b8db0488/href</a></iframe><p>Let’s call the ‘my_auto_corr’ function by passing the ‘nifty’ time series data frame and nlags=10 as arguments. We’ll also compare the output of ‘my_auto_corr’ function with that of ‘acf’ method of ‘statsmodels’.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d9292485c73fc2815638b198000212f0/href">https://medium.com/media/d9292485c73fc2815638b198000212f0/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/630/1*76RF-Ynv6LDezBvTfZcHqA.png" /><figcaption>Image by author</figcaption></figure><p>The results of ‘my_auto_corr’ are same as those of the ‘acf’ method of ‘statsmodels’. Let’s once again look at the formula of autocorrelation that we saw earlier and try to understand it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/238/1*ZbVu3HMeFq4MhloqJ3s7-Q.gif" /><figcaption>Image by author</figcaption></figure><ul><li>The denominator is pretty straightforward, it is similar to the variance of the original time series, but doesn’t have ‘N-1&#39; in the denominator. <strong>It is denoted by ‘denominator’ variable in the code.</strong></li><li>As discussed earlier, the numerator is similar to the covariance between the current and lagged versions of the time series (without N-1 as denominator). Let’s understand how to compute the numerator.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jEGClS1yQA-dU2aDuZ9HvQ.png" /><figcaption>Image by author</figcaption></figure><ul><li>The <strong>brown</strong> rectangle represents <strong>y(t)</strong> in the first part of the numerator. It is subtracted from the mean of the original time series, <strong>mean(y)</strong>. <strong>The first part is denoted by ‘numerator_p1’ in the code &amp; y(t)-mean(y) in the formula.</strong> y(t) is fixed at the bottom and its top moves down by 1 for every unit increase in the lag (k).</li><li>Similarly, the <strong>green</strong> rectangle represents <strong>y(t-k)</strong> in the second part of the numerator. It is also subtracted from the mean of the original time series, <strong>mean(y). The second part is denoted by ‘numerator_p2’ in the code &amp; y(t-k)-mean(y) in the formula</strong>. y(t-k) is fixed at the top and its bottom moves up by 1 for every unit increase in the lag (k).</li></ul><p>However, as we saw earlier, the numerator of the formula is not exactly the same as covariance. However, the denominator is similar to the variance of original time series, but without ‘N-1’ in denominator. Hence, computing the covariance of the brown and green rectangle and dividing it by the variance of the original time series doesn’t give us the autocorrelation coefficient.</p><p>Breaking down the autocorrelation formula into fragments and implementing it in Python helped us understand it better. We saw how the covariance in the numerator is calculated between the current and the lagged versions of time series. Hence, it is important to know what’s under the hood to understand a concept better, be it a machine learning algorithm or a concept in statistics.</p><p>Know more about my work at <a href="https://ksvmuralidhar.in/">https://ksvmuralidhar.in/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=322ad52f2199" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/understanding-autocorrelation-in-time-series-analysis-322ad52f2199">Understanding Autocorrelation in Time Series Analysis</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[My Articles]]></title>
            <link>https://ksvmuralidhar.medium.com/list-of-my-articles-8ca24c8ed021?source=rss-ce697e8ed35c------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ca24c8ed021</guid>
            <dc:creator><![CDATA[KSV Muralidhar]]></dc:creator>
            <pubDate>Thu, 29 Jul 2021 14:17:20 GMT</pubDate>
            <atom:updated>2023-03-08T03:46:11.502Z</atom:updated>
            <content:encoded><![CDATA[<h4>Topics include ML, Statistics, Data Analysis, Time Series Forecasting, SQL, MS Excel and Data Extraction.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A2VzMmL7KNJMCHgWId3pMA.png" /></figure><h4><strong>Machine Learning</strong></h4><ol><li><a href="https://towardsdatascience.com/learning-curve-to-identify-overfitting-underfitting-problems-133177f38df5?source=your_stories_page-------------------------------------">Learning Curve to identify Overfitting and Underfitting in Machine Learning</a></li><li><a href="https://towardsdatascience.com/simplifying-machine-learning-model-development-with-columntransformer-pipeline-f09ffb04ca6b?source=your_stories_page-------------------------------------">Simplifying Machine Learning model development with ColumnTransformer and Pipeline</a></li><li><a href="https://towardsdatascience.com/what-is-stratified-cross-validation-in-machine-learning-8844f3e7ae8e?source=your_stories_page-------------------------------------">What is Stratified Cross-Validation in Machine Learning?</a></li><li><a href="https://builtin.com/data-science/adjusted-r-squared">Demystifying R-Squared and Adjusted R-Squared</a></li><li><a href="https://towardsdatascience.com/outlier-detection-methods-in-machine-learning-1c8b7cca6cb8?source=your_stories_page-------------------------------------">Outlier detection methods in Machine Learning</a></li><li><a href="https://towardsdatascience.com/mining-the-2020-kaggle-machine-learning-data-science-survey-data-e4f29188515?source=your_stories_page-------------------------------------">Mining the ‘2020 Kaggle Machine Learning &amp; Data Science Survey’ data</a></li><li><a href="https://towardsdatascience.com/how-to-avoid-data-leakage-while-evaluating-the-performance-of-a-machine-learning-model-ac30f2bb8586?source=your_stories_page-------------------------------------">How to prevent Data Leakage while evaluating the performance of a Machine Learning model</a></li><li><a href="https://towardsdatascience.com/creating-custom-transformers-using-scikit-learn-5f9db7d7fdb5?source=your_stories_page-------------------------------------">Creating Custom Transformers with Scikit-Learn</a></li><li><a href="https://towardsdatascience.com/the-right-way-of-using-smote-with-cross-validation-92a8d09d00c7?source=your_stories_page-------------------------------------">The right way of using SMOTE with Cross-validation</a></li><li><a href="https://towardsdatascience.com/deploying-a-basic-streamlit-app-ceadae286fd0?source=your_stories_page-------------------------------------">Deploying a basic Streamlit app</a></li></ol><h4>Statistics &amp; Data Analysis</h4><ol><li><a href="https://towardsdatascience.com/a-beginner-dilemma-in-exploratory-data-analysis-8919b4e737e8?source=your_stories_page-------------------------------------">A Beginner’s Dilemma in Exploratory Data Analysis</a></li><li><a href="https://towardsdatascience.com/dual-axis-plots-in-python-95e5ef24a4e0?source=your_stories_page-------------------------------------">Dual Axis plots in Python</a></li><li><a href="https://towardsdatascience.com/100-stacked-charts-in-python-6ca3e1962d2b?source=your_stories_page-------------------------------------">100% stacked charts in Python</a></li><li><a href="https://ksvmuralidhar.medium.com/how-to-add-gradient-background-to-python-plots-9beda811ea19?source=user_profile---------1-------------------------------">How to add gradient background to Python plots</a></li><li><a href="https://medium.datadriveninvestor.com/1-2-is-it-the-probability-of-passing-a-test-a559cabb5cf9?source=your_stories_page-------------------------------------">1/2 — Is it the Probability of passing a test?</a></li><li><a href="https://towardsdatascience.com/reshaping-data-with-pandas-ecff92572488?source=your_stories_page-------------------------------------">Reshaping Data with Pandas</a></li><li><a href="https://towardsdatascience.com/deploying-a-basic-streamlit-app-to-heroku-be25a527fcb3?source=your_stories_page-------------------------------------">Deploying a basic Streamlit app to Heroku</a></li><li><a href="https://medium.datadriveninvestor.com/how-to-decide-on-the-number-of-bins-of-a-histogram-3c36dc5b1cd8?source=your_stories_page-------------------------------------">How to decide on the number of bins of a Histogram?</a></li><li><a href="https://medium.com/mlearning-ai/basic-exploratory-data-analysis-template-for-regression-problems-20ca00c58f7d?source=your_stories_page-------------------------------------">Basic Exploratory Data Analysis (EDA) template for Regression problems</a></li><li><a href="https://medium.com/nerd-for-tech/diving-deep-into-python-plotting-6d2baae4fde1?source=your_stories_page-------------------------------------">Diving deep into Python plotting</a></li><li><a href="https://medium.datadriveninvestor.com/how-would-i-know-if-my-variable-distribution-is-gaussian-c3b215383469?source=your_stories_page-------------------------------------">How would I know if my variable’ distribution is Gaussian?</a></li><li><a href="https://medium.datadriveninvestor.com/how-to-transform-the-data-to-look-like-gaussian-distribution-c50ab3fdada5?source=your_stories_page-------------------------------------">How to transform the data to look like Gaussian Distribution?</a></li></ol><h4>Time Series Forecasting</h4><ol><li><a href="https://towardsdatascience.com/understanding-autocorrelation-in-time-series-analysis-322ad52f2199?source=your_stories_page-------------------------------------">Understanding Autocorrelation in Time Series Analysis</a></li></ol><h4>Python</h4><ol><li><a href="https://medium.com/mlearning-ai/understanding-the-scope-of-the-variables-in-python-d83a95b73d90?source=your_stories_page-------------------------------------">Understanding the scope of the variables in Python</a></li><li><a href="https://towardsdatascience.com/understanding-list-comprehensions-in-python-6de57e26631c?source=your_stories_page-------------------------------------">Understanding List Comprehensions in Python</a></li></ol><h4>SQL</h4><ol><li><a href="https://medium.com/codex/beginners-guide-to-sql-for-data-analysis-using-postgresql-4ed443308835?source=your_stories_page-------------------------------------">Beginner’s guide to SQL for Data Analysis using PostgreSQL</a></li><li><a href="https://towardsdatascience.com/merging-tables-using-sql-a2e60ff687e9?source=your_stories_page-------------------------------------">Merging tables using SQL</a></li><li><a href="https://towardsdatascience.com/how-to-derive-summary-statistics-using-postgresql-742f3cdc0f44?source=your_stories_page-------------------------------------">How to derive summary statistics using PostgreSQL</a></li></ol><h4>MS Excel</h4><ol><li><a href="https://towardsdatascience.com/essential-functions-in-excel-for-data-preprocessing-140f002f3ca?source=your_stories_page-------------------------------------">Essential Functions in Excel for Data Preprocessing</a></li></ol><h4>Data Extraction</h4><ol><li><a href="https://medium.com/nerd-for-tech/building-a-search-engine-scraper-with-streamlit-b616e5bd293c?source=your_stories_page-------------------------------------">Building a Search Engine Scraper with Streamlit</a></li><li><a href="https://ksvmuralidhar.medium.com/how-to-build-an-rss-feed-based-news-app-with-streamlit-heroku-and-android-studio-cc4ebb1077e?source=your_stories_page-------------------------------------">How to build an RSS feed-based News app with Streamlit, Heroku and Android Studio?</a></li><li><a href="https://medium.com/nerd-for-tech/extracting-data-from-an-api-using-python-38e9fbd2525c?source=your_stories_page-------------------------------------">Extracting data from an API using Python</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ca24c8ed021" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reshaping Data with Pandas]]></title>
            <link>https://medium.com/data-science/reshaping-data-with-pandas-ecff92572488?source=rss-ce697e8ed35c------2</link>
            <guid isPermaLink="false">https://medium.com/p/ecff92572488</guid>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[data-transformation]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[KSV Muralidhar]]></dc:creator>
            <pubDate>Thu, 29 Jul 2021 10:53:06 GMT</pubDate>
            <atom:updated>2023-07-07T18:52:15.703Z</atom:updated>
            <content:encoded><![CDATA[<h4>While analyzing data, we may need to reshape tabular data. Pandas has two methods that aid in reshaping the data into a desired format.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a3B7uy7EftcBIe5vzvsSWw.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@mjessier?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Myriam Jessier</a> on <a href="https://unsplash.com/s/photos/data-analysis?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText">Unsplash</a></figcaption></figure><p>Pandas has two methods namely, <strong>melt()</strong> and <strong>pivot()</strong>, to reshape the data. These methods work similar to gather() and spread() functions of the ‘tidyr’ package in R, respectively. We’ll consider a balance sheet in the format as reported by companies. The balance sheet is stored in a pandas data frame named ‘df’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/424/1*2Npua8lqQlzu8z-seK2hzA.png" /><figcaption>Image by author</figcaption></figure><h3>melt()</h3><p>This method flattens/melts tabular data such that the specified columns and their respective values are transformed into key-value pairs. ‘keys’ are the column names of the dataset before transformation and ‘values’ are the values in the respective columns. After transformation, ‘keys’ are stored in a column named ‘variable’ and ‘values’ are stored in another column named ‘value’, by default.</p><p>This might look a bit confusing, it happened to me as well at the beginning. However, things get simpler once we look at an example. Let’s flatten/melt the balance sheet that we saw earlier. Initially, we’ll call the melt() method with its default arguments.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/df48d33424b62b3ff6e7620863012dc8/href">https://medium.com/media/df48d33424b62b3ff6e7620863012dc8/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/366/1*oJ35HodMbs3j31gZSS5smQ.png" /><figcaption>Image by author</figcaption></figure><p>As discussed, the columns of the data frame are transformed into key-value pairs. This format may be of no use, so, let’s look at an example where the ‘Particulars’ column stays as it is and the associated yearly data appears as key-value pairs in the adjacent columns. In this case, the ‘Particulars’ column is called an ‘id’ column and should be passed to the “id_vars” argument of melt() method.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3a298065815822e0e4b243cb1a745f2e/href">https://medium.com/media/3a298065815822e0e4b243cb1a745f2e/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/416/1*LKR2PTJzrWSAYTQg-bkAcg.png" /><figcaption>Image by author</figcaption></figure><p>All other columns, except the ‘Particulars’ column (passed to “id_vars” argument) are transformed into key-value pairs. We may also specify a list of columns to be considered as “id” columns. We can rename the ‘value’ and ‘variable’ columns using the ‘var_name’ and ‘value_name’ arguments as shown below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/738b8d182eaec47e80582462b749e974/href">https://medium.com/media/738b8d182eaec47e80582462b749e974/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/416/1*ZI1GG3_SWfLls07ccz439w.png" /><figcaption>Image by author</figcaption></figure><p>This format will be useful when we analyze data in Excel. This format makes it easier to build pivot tables in Excel.</p><h3>pivot()</h3><p>This method does the reverse of what melt() did. It transforms the key-value pairs into columns. Let’s look at an example to convert ‘df_melt’ (shown below) back to the original format.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/416/1*ZI1GG3_SWfLls07ccz439w.png" /><figcaption>df_melt (Image by author)</figcaption></figure><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f67764f45af3539a1a45f3dd6bd43320/href">https://medium.com/media/f67764f45af3539a1a45f3dd6bd43320/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/433/1*wF3XYgMd4dWN98X_WkFtzg.png" /><figcaption>Image by author</figcaption></figure><p>melt() and pivot() are the two methods that come handy when it comes to transforming tabular data into a desired format. melt() converts columns to key-value pairs, while pivot() converts key-value pairs to columns.</p><p>Know more about my work at <a href="https://ksvmuralidhar.in/">https://ksvmuralidhar.in/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ecff92572488" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/reshaping-data-with-pandas-ecff92572488">Reshaping Data with Pandas</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>