<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Padhma Muniraj on Medium]]></title>
        <description><![CDATA[Stories by Padhma Muniraj on Medium]]></description>
        <link>https://medium.com/@padhmamuniraj?source=rss-a5a3c9ba4575------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Dp79dBdKYXAlGDNpDULwnw.png</url>
            <title>Stories by Padhma Muniraj on Medium</title>
            <link>https://medium.com/@padhmamuniraj?source=rss-a5a3c9ba4575------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 17:50:08 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@padhmamuniraj/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Visualizing QS World University Rankings]]></title>
            <link>https://padhmamuniraj.medium.com/visualizing-qs-world-university-rankings-92cb9f4357ef?source=rss-a5a3c9ba4575------2</link>
            <guid isPermaLink="false">https://medium.com/p/92cb9f4357ef</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[ml-so-good]]></category>
            <category><![CDATA[education]]></category>
            <dc:creator><![CDATA[Padhma Muniraj]]></dc:creator>
            <pubDate>Fri, 18 Feb 2022 18:07:58 GMT</pubDate>
            <atom:updated>2022-02-19T18:53:33.107Z</atom:updated>
            <content:encoded><![CDATA[<h4>Exploring the educational institutions around the world from 2017 to 2022</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4niEaYKAbM9E4phY" /><figcaption>Photo by <a href="https://unsplash.com/@napr0tiv?utm_source=medium&amp;utm_medium=referral">Vasily Koloda</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>QS World University Rankings is an annual publication of global university rankings by Quacquarelli Symonds (QS), a UK company specializing in the analysis of higher education institutions around the world. The QS ranking receives approval from the International Ranking Expert Group (IREG) and is viewed as one of the three most widely read university rankings in the world, along with the Academic Ranking of World Universities and Times Higher Education World University Rankings.</p><h3>History</h3><blockquote>Rank does not confer privilege or give power. It imposes responsibility — Peter Drucker</blockquote><p>In December 2003, Richard Lambert, a British journalist and business executive published a report called the <em>Lambert Review of Business-University Collaboration, </em>for the HM Treasury, the finance ministry of the United Kingdom. One of his suggestions was,</p><blockquote>….. to develop a league table of the world’s best research-intensive universities. Such a table would provide academics with a valuable reality check, and would help policymakers to judge the success of their funding strategies.</blockquote><p>Lambert suggested that such a table would help Britain to compare its universities with the top universities around the world. In his book, “<em>The Great Brain Race</em>”, Ben Wildavsky credited the idea of world rankings to John O’Leary, then-editor of <em>THE</em> (Times Higher Education). THE partnered with QS and produced the world university rankings from 2004 to 2009. <em>THE </em>then split from QS and published its over version of the rankings. QS has been publishing its university rankings in partnership with Elsevier.</p><h3>Data!</h3><p>The data used for the analysis and visualization of the rankings were obtained by scraping the <a href="https://www.topuniversities.com/university-rankings/world-university-rankings/2022">QS World University Rankings website</a> with Python, Selenium, and BeautifulSoup. The scraped data required a lot of data cleaning for many of the attributes. For example, there were many null values for the features city and faculty_count. So, I wrote a python script that would automatically search google based on the university feature and scrape values for the missing feature. Below is a general summary of the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/550/1*xkdl4LAnFgYwzeuk_n-FbQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/642/1*MrqEzfsFThjMVpJiQQKsSg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/604/1*O74e8X5n__7RBxxA3ckVIg.png" /></figure><p>From 2017 to 2022, 6482 universities were ranked by QS. This particular dataset contains a total of 15 features for each university. The main challenge in this dataset was missing values. Despite my efforts to expunge them as mentioned above, they decided to pop up. (Sigh!)</p><p>So, before handling the missing values I wanted to understand if there is any correlation between them. I used Python’s <a href="https://github.com/ResidentMario/missingno">missingno</a> package to visualize the <a href="https://towardsdev.com/how-to-identify-missingness-types-with-missingno-61cfe0449ad9">missingness</a> correlation. The missingno correlation heatmap measures how strongly the presence or absence of one feature affects another.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*f0N9O22vcO1o7MOz3xzr9A.png" /><figcaption>Correlation heatmap of missing values</figcaption></figure><p>The correlation heatmap includes only the columns with missing values. The higher the correlation, the higher the missing values in one column are dependent on the missing values with another column. We can see faculty_count has a significant correlation with student_faculty_ratio and international_students. Other columns have little or no significant correlation.</p><p>After exploring the data more, I decided to drop the few rows that were missing more than 4 features. It took care of the 1–2% missing values across 5 columns. The main culprit was the score feature, it was missing 56.5% values. When I looked for these values on the QS website, I could see they have given a score only for the top 500 universities although 1200+ universities have been ranked. So, I decided to drop it as well.</p><p>I did most of the data cleaning and preparation as soon as I scraped the data. So, as soon as I took care of the missing values my dataset was pretty much done and dusted! Now, it’s time for some insights.</p><p>Although QS has been publishing university rankings for almost two decades now, the data we have is from 2017 to 2022.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*nGcysZZkxE-twRAyHMl8bA.png" /><figcaption>Number of universities ranked from 2017 to 2022</figcaption></figure><p>We can see from the above barplot that with each year they’re considering more universities than the previous year for the rankings with 2022 consisting of the highest number of universities among them.</p><p>One of the major headscratchers for students when shortlisting universities is whether to go for a public university or a private university. Both of them have their own fair share of pros and cons. The below graph demonstrates the number of public and private universities ranked by QS over the years.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*d50aIoe5-qUEdLD3I5t-QA.png" /><figcaption>Types of universities</figcaption></figure><p>If you do a simple google search, you can find many websites claiming that private universities are better than public universities because they tend to have better rankings. Well, that’s not the case here. More than 80% of the universities ranked are public considering they have ranked only 1300 universities in 2022 out of tens of thousands of universities across the world.</p><h4>Visualizing universities across the world: by continents, by countries, by cities.</h4><p>Now that we know how many are there and what kind they are. Let’s see how they are spread out across the world.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_Mfc4ZNMaPoDxPE5s_fnWA.png" /><figcaption>Distribution of universities across the world (Ranked by QS)</figcaption></figure><p>Europe tends to be the continent with the most number of ranked universities though we have to consider the fact that they have included Russia in Europe although it belongs to both Europe and Asia. Africa has the least number of universities that are ranked.</p><p>Looking from a country perspective,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d7UUImhPIQcY_cPpehUEcg.png" /><figcaption>Number of universities ranked in each country</figcaption></figure><p>When we take a look at the number of universities by country, the US and the UK are the leaders. The top tier is dominated by universities from Europe, Asia, and North America. Most of the countries in the bottom tier are from Africa, Latin America, and the Middle East.</p><p>While a continent or a country is large enough to hold on to a number of highly ranked universities, some of the advanced cities also tend to be top contributors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4C7ANeuAiLayBf8yim2Hmw.png" /><figcaption>Top 20 cities with a high number of ranked universities</figcaption></figure><p>The above graph considers the top 20 cities with a high number of unique universities. London is an academic hotspot with a whooping 19 universities that are ranked globally! It’s closely followed by Moscow and Seoul.</p><p>After exploring the distribution of universities continent-wise, country-wise, and city-wise, I was curious about the top 10 universities. The best of the best! As far as the top 10 universities were concerned, the number of unique universities over the years was 10, the only difference being their rank.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IEZOSQ3aBzwgqvdsdomYow.png" /><figcaption>Top 10 universities ranked by QS</figcaption></figure><p>The universities from the above graph have a tendency to occupy the top 10 positions consistently. Some of the key insights are,</p><ul><li>MIT tends to be an undisputed king in terms of QS Rankings, ranked number 1 always.</li><li>Stanford and Harvard have dropped down this year (2022) for the first time since 2017.</li><li>The University of Oxford, the oldest university in the English-speaking world, has jumped from Rank 6 to Rank 2.</li><li>On an overall scale, universities from the UK have spiked up on their rankings compared to the US universities most of which have dropped down this year (2022).</li><li>Out of the top 10, only one university, ETH Zurich (Switzerland), is from a country other than US or UK.</li></ul><h3>Methodology of QS Rankings</h3><p><em>What makes a university number 1 or number 1000? How does QS decide which universities are to be considered for the rankings? What metrics are used to gauge the universities against each other?</em></p><p>The answer to these questions lies in the methodology used QS. QS designed its rankings to assess performance according to what it believes to be key aspects of a university’s mission: teaching, research, nurturing employability, and internationalization. The methodological framework it follows assess universities based on six metrics,</p><ul><li>Academic Reputation (40%)</li><li>Employer Reputation (10%)</li><li>Faculty/Student Ratio (20%)</li><li>Citations per faculty (20%)</li><li>International Faculty Ratio (5%)</li><li>International Student Ratio (5%)</li></ul><p>More information about the methodology can be found <a href="https://www.topuniversities.com/qs-world-university-rankings/methodology">here</a>.</p><p>Out of the 6 metrics that QS used to rank the universities, only 3 were present in the dataset — Research Output(Citations per faculty), Student-Faculty Ratio, and International Students — accounting for only 45% of the survey methodology.</p><h4>Research Output</h4><p>Next to teaching, Academic research is viewed as a very important factor in the growth of a university. QS measures the quality of research output in terms of the “Citations per faculty” metric. To quote them,</p><blockquote>To calculate it, we take the total number of citations received by all papers produced by an institution across a five-year period by the number of faculty members at that institution.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*V7gY9Qc0alC3o5NrouAPDQ.png" /><figcaption>The research output of top universities by type</figcaption></figure><p>Clearly, most universities under consideration for the rankings have a “Very High” research output. Public universities outperform private universities in terms of research.</p><p>There is a feature in the data called faculty_count, it contains the number of academic staff each university has. Not every academic staff is going to be engaged in research but most of the academic researchers are faculty in an institution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*QFsp2NgPRQ0-mkcTB9KlcQ.png" /><figcaption>Research Output Vs Faculty Count</figcaption></figure><p>As far as the number of faculty are concerned,</p><ul><li>Universities with “Very High” research output have more staff.</li></ul><p><em>So, does this mean universities with a higher number of faculty do better research? Not necessarily. Universities with “Very High” research output may attract more accomplished academics and researchers because of their reputation along with many other factors.</em></p><ul><li>We can see public universities with research output as “Very High” and “Low” have nearly equal numbers of staff.</li><li>Also, private universities with “Low” research output have more staff than “High” output.</li></ul><p><em>This can mean that not every university puts an emphasis on research although they have more academic staff. And higher number of academic staff doesn’t mean high-quality research.</em></p><p>The below graph depicts a point plot of research output and the size of a university.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/408/1*NIE-FeRASSkD0NgHUB_Qxg.png" /><figcaption>Point plot of research output and size</figcaption></figure><p>The relationship between the size of the university and research output is pretty candid. Universities with “Very High” and “High” research output are larger in size compared to “Medium” and “Low”.</p><p>If you’re unsure how to interpret a point plot, here’s a brief intro</p><blockquote>A pointplot shows an estimate of mean value for a numeric variable by using scatter plot points. This can be particularly useful for comparing different levels of a categorical variable. The lines joining the pointplot can be used to judge the differences between slopes easily.</blockquote><p>For more info, refer <a href="https://seaborn.pydata.org/generated/seaborn.pointplot.html">here</a>.</p><h4>Student-Faculty Ratio</h4><p>Student Faculty Ratio is an interesting measure. According to QS, “It is usually cited by students as a metric of the highest importance to them”. Lesser the ratio, the higher the performance. A faculty with less number of students assigned to them can dedicate more focus and attention to each individual student.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*5D5KG7Oo2QnU6izExMfKkw.png" /><figcaption>Histogram of Student-Faculty Ratio</figcaption></figure><p>Among the top universities,</p><ul><li>There is an average of 13 students per faculty.</li><li>There are universities that have as low as 1 student per faculty.</li><li>While there are also universities that have 67 students per faculty.</li><li>Most of the universities have somewhere between 5 to 20 students per faculty.</li></ul><p>Students also play a vital role in academic research.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*l7rlKIa3iulW-Q7zoPkPVg.png" /><figcaption>Student Faculty Ratio vs Research Output</figcaption></figure><p>Obviously, universities with “Very High” research output have a very less “student-faculty ratio” compared to the rest of them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/438/1*RXnYQzmosWx14yvcI1TKlg.png" /><figcaption>Student Faculty Ratio Vs Size</figcaption></figure><p>Private universities have very less student_faculty_ratio compared to public universities when it comes to size. Another interesting observation is that the average student_faculty_ratio seems to increase with the increase in the size of the university.</p><h4>International Students</h4><p>A university that attracts students from across the world demonstrates a global outlook and possesses a multicultural diversity in its campus.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*BleNjzumAlI1RlNJrNxIEw.png" /><figcaption>Histogram of International students</figcaption></figure><ul><li>On average, ranked universities tend to have 1900+ international students.</li><li>There is a university with international students&#39; intake as high as 31,000+.</li><li>Most of the universities have an intake between 0 to 5000.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/438/1*praY7xJy4dTkgXiuFpITcw.png" /><figcaption>International students Vs Research output</figcaption></figure><p>International students tend to prefer public universities with “Very High” research output. Due to lesser tuition fees compared to private ones? Maybe. Also, as we saw earlier public universities are better at research_output than private ones.</p><p>One more interesting thing I wanted to explore was, the popular country of choice for international students.</p><p>Can you guess before reading further?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/1*d8WPLc-MWjb9pOaxnehWkA.gif" /><figcaption>Gif: <a href="https://giphy.com/gifs/season-17-the-simpsons-17x4-3o6Mbdr0AHyzqCTgHK">Source</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*_zXvxpybzE2rNYow4R92LQ.png" /><figcaption>Country of choice</figcaption></figure><p>Between the years 2017 to 2022, top-ranked universities in the US had an intake of almost 2 Million international students while the UK has nearly 1.9 Million.</p><h4>Criticisms and Limitations</h4><p>This analysis was to explore the rankings published by QS. Many academics, scholars, and economists have <a href="https://en.wikipedia.org/wiki/QS_World_University_Rankings#Criticisms">expressed concerns</a> over the methodology that QS uses for its rankings. So, you should keep in mind that any conclusions or insights drawn from these rankings are equivocal. Also, this dataset was very limited in scope. I had features that contributed to only 45% of the survey methodology.</p><p>I hope you enjoyed reading this article! It was very fun to explore this dataset and I hope it was the same for you while reading it.</p><p><a href="https://www.kaggle.com/padhmam/qs-world-university-rankings-eda-visualization/notebook"><strong><em>You can find the data and the code used for the visualizations here.</em></strong></a></p><p>Let me know in the comments about your views, concerns, or any feedback.</p><p><a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb">Mlearning.ai Submission Suggestions</a></p><p>🔵 <a href="https://medium.com/mlearning-ai/mlearning-ai-submission-suggestions-b51e2b130bfb"><strong>Become a Writer</strong></a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=92cb9f4357ef" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[3 ways to deal with SettingWithCopyWarning in Pandas]]></title>
            <link>https://medium.com/analytics-vidhya/3-ways-to-deal-with-settingwithcopywarning-in-pandas-f06bfbd0a86b?source=rss-a5a3c9ba4575------2</link>
            <guid isPermaLink="false">https://medium.com/p/f06bfbd0a86b</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[education]]></category>
            <dc:creator><![CDATA[Padhma Muniraj]]></dc:creator>
            <pubDate>Wed, 22 Dec 2021 15:48:14 GMT</pubDate>
            <atom:updated>2022-02-25T16:53:50.758Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nQ46cij3YyfarLPbFyoYoQ.png" /><figcaption>Image: <a href="https://storyset.com/illustration/warning">Source</a></figcaption></figure><blockquote>It doesn’t pay to ignore warnings. Even when they don’t make sense. — Debra Doyle</blockquote><p>One of the things I was taught while learning to code was to not be bothered by ‘Warnings’ in your code. “Focus on fixing major bugs and errors, warnings aren’t a big deal”. I realized what a terrible advice this was when I started working on real-world situations. Sometimes warnings can cost you more than you think. One such warning is the SettingWithCopy warning in Pandas.</p><p>No matter how long you’ve worked with pandas, sooner or later you’re bound to encounter the Settingwithcopywarning. If you’re trying to wrap your heads around what it is and why it keeps showing up even when you “<em>do get the output you expected”,</em> then this article is for you.</p><p>In order to explain the logic behind the warning, I have used the <a href="https://www.kaggle.com/gagandeep16/car-sales">Car Sales dataset</a> from Kaggle. This dataset contains information about different types of cars.</p><p>Here is a glimpse of the data and the structure of the dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PoHNlwqmiyzUZ9e9AvITZA.jpeg" /></figure><pre># car_sales is the dataframe containing the dataset<br>car_sales.info()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/952/1*Lttg7v0CvXZO9DIht7fvSA.jpeg" /><figcaption>car sales dataset info</figcaption></figure><p>Let’s assume a scenario where we have received an update that the fuel_capacity<em> </em>of all the Porsche cars have been increased from 17.0 to 18.0 and we have been requested to make the changes. Let’s go ahead and change them.</p><pre>car_sales[car_sales[&#39;Manufacturer&#39;] == &#39;Porsche&#39;][&#39;Fuel_capacity&#39;] = 18.0</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-mS62GErb1Gn8C5RebSltg.png" /></figure><p>Uh-oh! We have triggered the famous SettingWithCopy warning.</p><p>If we take a look at the dataframe now, we can see the values are not updated.</p><pre>car_sales[car_sales[&#39;Manufacturer&#39;] == &#39;Porsche&#39;]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8O2B3gnUF_8suBAebZYXag.png" /></figure><p>We have to understand <em>SettingWithCopy</em> is a warning and not an error. An error breaks your code and prevents you from moving on further without fixing it. But, a warning indicates that there is something wrong with your code while producing the output.</p><p>In this case, sometimes we might get the output we intended to and also be tempted to ignore the warning. But we should never ignore this warning because it means that the operation we are trying to perform may not have worked as we expected and there can be some unexpected issues in the future.</p><p>These are the <a href="https://github.com/pandas-dev/pandas/issues/8730">words of Jeff Reback</a>, one of the core developers of pandas, on why you should never ignore this warning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*didSklby_TijqLpoKp8izA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/200/1*FCuBFPgUTvSnNu8ecPjspg.gif" /></figure><p>In order to understand how to fix this warning and what to do when we face it, it is imperative to know the difference between Views and Copies in Pandas and how they work.</p><h3>Views Vs Copies</h3><p>In the code above where we try to return all the Porsche cars from the data, the result we receive may either be a view or a copy of the dataframe.</p><p>A view (or a shallow copy) is a subset of the original object which doesn’t have its own memory and address space. It is just a projection of the object we are trying to access.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*813kzGHXIxPolUyJjrMZ5w.jpeg" /><figcaption>View of a dataframe</figcaption></figure><p>A copy (or a deep copy) is a duplicate of the original object which has its own memory and address space. It is a separate entity that is thrown away in Pandas once we are done operating on them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*15KsChaCaAOdQ-0o1vVuug.jpeg" /><figcaption>Copy of a dataframe</figcaption></figure><p>One of the main differences between views and copies is that modifying a view modifies the original dataframe and vice versa, whereas modifying a copy doesn’t affect the original dataframe.</p><p>Let’s say we change sales_in_thousands<em> </em>for the car of Model Boxter to 9.35.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*naqZlfQ81u8YF5hjzFBmkw.jpeg" /><figcaption>Modifying a view</figcaption></figure><p>You can see above, modifying a view modifies the original dataframe as well.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N7dlfaWabuKFC06ncjJvsA.jpeg" /><figcaption>Modifying a copy</figcaption></figure><p>On the contrary, modifying a copy doesn’t necessarily modify the original dataframe.</p><p>Pandas got this behavior of views and copies from the underlying Numpy arrays. Numpy arrays are limited to a datatype so whether a view or a copy is returned can be predicted. While Pandas uses its Numpy core, it follows a complex set of rules to optimize space and determine whether to return a view or a copy. Because of that, whenever we are indexing a dataframe, there is no set way to predict whether a view or a copy is returned. To quote the pandas documentation,</p><blockquote><em>Outside of simple cases, it’s very hard to predict whether it will return a view or a copy (it depends on the memory layout of the array, about which pandas makes no guarantees) ………. That’s what SettingWithCopy is warning you about!</em></blockquote><p>To check whether a view or a copy is returned, you can use the internal attributes _is_view or _is_copy. _is_view returns a boolean and _is_copy returns a reference to the original dataframe or None.</p><p>Let’s look at 3 of the most common issues for encountering this warning and how to handle them.</p><h3>1. Chained Assignment</h3><p>One of the most common reasons Pandas generates this warning is when it detects chained assignment or chained indexing.</p><p>There are two things we do with a Pandas dataframe, we either</p><ul><li><strong>Set</strong> — assign a value to something</li><li>Or, <strong>Get</strong> — access values from something</li></ul><p>A chained assignment is when we try to assign(<strong>set</strong>) something by using more than one indexing operation.</p><p>Recall the example below which we used previously.</p><pre>car_sales[car_sales[&#39;Manufacturer&#39;] == &#39;Porsche&#39;][&#39;Fuel_capacity&#39;] = 18.0</pre><p>Here, two indexing operations are combined to set a value. First, we are trying to access (<strong>get</strong>) all the ‘Porsche’ cars from the dataframe, then we try to assign(<strong>set</strong>) a new value to ‘Fuel_capacity’.</p><p>We want to modify the original dataframe but this operation may try to create a copy and modify it. This is what the warning is telling us. <em>‘A value is trying to be set on a copy of a slice of a dataframe’.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-mS62GErb1Gn8C5RebSltg.png" /></figure><p>We discussed above that Pandas can either create a view or a copy when we are trying to access (<strong>get</strong>) a subset of an operation.</p><p>Let’s see if the operation we are trying to perform is on a view or a copy.</p><pre>car_sales[car_sales[&#39;Manufacturer&#39;] == &#39;Porsche&#39;][&#39;Fuel_capacity&#39;]._is_view<br><br># output<br>True</pre><pre>car_sales[car_sales[&#39;Manufacturer&#39;] == &#39;Porsche&#39;][&#39;Fuel_capacity&#39;]._is_copy<br><br>#output<br>&lt;weakref at 0x7fe118b59b80; to &#39;DataFrame&#39; at 0x7fe1187a14f0&gt;</pre><p>_is_view has returned True meaning it’s a view while _is_copy has returned a weakref meaning it’s a copy. Hence, output of the ‘<strong>get</strong>’ operation is ambiguous. It can be anything in the end. This is why ignoring SettingWithCopywarning is a bad idea. It can eventually lead to breaking something in your code when you least expect it.</p><p>The problem of chained assignment can be tackled easily by combining the back-to-back indexing operations into a single operation using .loc.</p><pre>car_sales.loc[car_sales.Manufacturer == &#39;Porsche&#39;, &#39;Fuel_capacity&#39;] = 18.0<br>car_sales[car_sales.Manufacturer == &#39;Porsche&#39;][&#39;Fuel_capacity&#39;]<br><br>#output<br>124    18.0<br>125    18.0<br>126    18.0<br>Name: Fuel_capacity, dtype: float64</pre><h3>2. Hidden Chaining</h3><p>The second most common reason that triggers this warning is Hidden Chaining. It can be tricky and hard to track down the source of this problem as it may span across your entire codebase.</p><p>Let’s look at a scenario for Hidden Chaining. We’ll go ahead and create a new dataframe containing all the ‘<em>Chevrolet</em>’ cars while bearing in mind to use .loc from our previous lesson.</p><pre>chevrolet_cars = car_sales.loc[car_sales.Manufacturer == &#39;Chevrolet&#39;]<br>chevrolet_cars</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1QIBgQCgxtye81AK9vmilA.png" /></figure><p>We do some other operations for some time and play around with our code.</p><pre>chevrolet_cars[&#39;Model&#39;].value_counts()<br>....<br># few lines of code<br>chevrolet_cars[&#39;Sales_in_thousands&#39;].std()<br>....<br>chevrolet_cars[&#39;__year_resale_value&#39;].max()<br>....<br># few lines of code<br>chevrolet_cars.loc[20,&#39;Price_in_thousands&#39;] = 17.638</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gySV9tJHuAB2hXa8bMpNtQ.png" /></figure><p>Boom! This warning again!!</p><p>There was no chained assignment in that last line of code but it still went ahead and triggered that warning. Let’s look at the values in our dataframe.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8o5rNFq8IXTuD9lqTC2qHg.png" /></figure><p>It has updated our value. So, should we go ahead and ignore the warning this time? Probably not.</p><p>There is no obvious chained assignment in this code. In reality, it can occur on one line or even across multiple lines of code. When we created the chevrolet_cars dataframe, we used a <strong>get</strong> operation. So there is no guarantee whether this returned a <em>view</em> or a <em>copy</em>. So, we might be trying to modify the original dataframe as well.</p><p>Identifying this problem can be very tedious in real codebases spanning thousands of lines but it is very simple to tackle this. When you want to create a new dataframe, explicitly make a <strong>copy</strong> using .copy() method. This will make it clear to Pandas that we are operating on a new dataframe.</p><pre>chevrolet_cars = car_sales.loc[car_sales.Manufacturer == &#39;Chevrolet&#39;].copy()<br>chevrolet_cars.loc[20,&#39;Price_in_thousands&#39;] = 17.638<br>chevrolet_cars.loc[20, &#39;Price_in_thousands&#39;]<br><br>#output<br>17.638</pre><h3>3. False Positives</h3><p>False positive is a case when the warning is triggered when it’s not supposed to be. It’s very safe to ignore the warning in this case. Many of the scenarios which caused the “False Positive” warnings have been fixed in Pandas through the years. It’s discussed in the <a href="https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy">Pandas documentation</a> if you want to take a look.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OTIAokBTfZHMZSv41jdpwQ.png" /></figure><p>Let’s say we want only the cars with Vehicle_type as ‘<em>Passenger</em>’ and we would like create a new column which will be a boolean indicating whether the car is available or not.</p><pre>car_sales = car_sales[car_sales[&#39;Vehicle_type&#39;] == &#39;Passenger&#39;]<br>car_sales[&#39;In Stock&#39;] = &#39;True&#39;<br><br>#output<br>:1: SettingWithCopyWarning: <br>A value is trying to be set on a copy of a slice from a DataFrame.<br>Try using .loc[row_indexer,col_indexer] = value instead</pre><p>If you look at the dataframe, it would have updated the new column. In this case, we are not bothered if it overwrites the original dataframe.</p><p>We can go ahead and suppress the warning by changing the default behavior as follows,</p><ul><li>pd.set_option(‘mode.chained_assignment’, None) — suppresses and turn off the warning</li><li>pd.set_option(‘mode.chained_assignment’, ‘Warn’) — issues a warning</li><li>pd.set_option(‘mode.chained_assignment’, ‘Raise’) — raises an exception</li></ul><p>Changing the behavior is not recommended unless you know what you are doing.</p><h4>End notes</h4><p>Most of the scenarios of the SettingWithCopy warning can be avoided by communicating to Pandas clearly. When you want to modify the original dataframe, use .loc or when you want a copy, specify it directly. This will not only prevent future warnings and errors, it will also make your codebase more robust for maintenance.</p><p>You can take a look at these issues on GitHub <a href="https://github.com/pandas-dev/pandas/pull/5390">#5390</a> and <a href="https://github.com/pandas-dev/pandas/issues/5597">#5597</a> for background discussion.</p><p>Thank you for reading all the way down here! Let me know in the comments if you have any feedback, criticism, or concerns. Have a good day!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f06bfbd0a86b" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/3-ways-to-deal-with-settingwithcopywarning-in-pandas-f06bfbd0a86b">3 ways to deal with SettingWithCopyWarning in Pandas</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Loss functions to evaluate Regression Models]]></title>
            <link>https://medium.com/analytics-vidhya/loss-functions-to-evaluate-regression-models-8dac47e327e2?source=rss-a5a3c9ba4575------2</link>
            <guid isPermaLink="false">https://medium.com/p/8dac47e327e2</guid>
            <category><![CDATA[regression]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[education]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[evaluation-metric]]></category>
            <dc:creator><![CDATA[Padhma Muniraj]]></dc:creator>
            <pubDate>Fri, 17 Dec 2021 20:10:36 GMT</pubDate>
            <atom:updated>2022-02-25T17:09:17.980Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5rXejhcKLorK7JiXnQH-tg.jpeg" /><figcaption><a href="https://unsplash.com/photos/5fNmWej4tAA">Image Source</a></figcaption></figure><p>The objective of any machine learning model is to understand and learn patterns from the data which can further be used to make predictions or answer questions or simply just understand the underlying pattern in the data that is otherwise not evident candidly. Most of the time, the learning part is iterative. A model learns some patterns from the data, we test it against some new data that the model did not encounter during training, we see how good or how bad a job it did, we tweak and adjust some parameters, then we put it to test again. This process is repeated until we are presented with a model that is good enough (Although, some real world models can just be satisfactory and make a world of difference). The part where we evaluate and test our model is where the loss functions come into play.</p><p>Loss functions takes the model’s predicted values and compares them against the actual values. It estimates how well (or how bad) the model is, in terms of its ability in mapping the relationship between X (a feature, independent variable, predictor variable) and Y(the target, dependent variable, response variable). Sometimes just knowing how bad the model is performing may not be enough, we might also need to calculate how far off the model is from the actual value. By knowing the amount of deviation between predicted value and the actual value, we can train our model accordingly. This difference between the actual value and the predicted value is called the loss. A high loss value means the model has poor performance.</p><p>There are many loss functions for evaluating regression models.</p><blockquote>There is no <em>“one function to rule them all”.</em></blockquote><p>Choosing the appropriate loss function is very crucial and what makes one desirable depends on the data at hand. Every function has its own properties. There are many factors that contribute to the appropriate choice of a loss function like the choice of algorithm, outliers in the data, being differentiable, etc.</p><p>This article aims to present you with a list of all loss functions for regression with their pros and cons. Although all of them can be implemented using libraries such as SciPy, PyTorch, Scikit Learn, Keras, etc, I have implemented the code using NumPy as it helps in gaining a better understanding about what is happening under the hood.</p><p>Without further ado, let’s get started.</p><h4>Table of Contents</h4><ol><li>Loss function vs Cost function</li><li>Mean Absolute Error (MAE)</li><li>Mean Bias Error (MBE)</li><li>Relative Absolute Error (RAE)</li><li>Mean Absolute Percentage Error (MAPE)</li><li>Mean Squared Error (MSE)</li><li>Root Mean Squared Error (RMSE)</li><li>Relative Squared Error (RSE)</li><li>Normalized Root Mean Squared Error (NRMSE)</li><li>Relative Root Mean Squared Error (RRMSE)</li><li>Root Mean Squared Logarithmic Error (RMSLE)</li><li>Huber Loss</li><li>Log Cosh Loss</li><li>Quantile Loss</li></ol><h4>Loss function vs Cost function</h4><p>A function that calculates loss for 1 data point is called the loss function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/364/1*-bC7eNhPFbpcyBMElNuexw.png" /><figcaption>Loss function</figcaption></figure><p>A function that calculates loss for the entire data being used is called the cost function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/539/1*1raWXWBwd5Z9AeEG2-XSKg.png" /><figcaption>Cost function</figcaption></figure><h4>Mean Absolute Error (MAE)</h4><p>Mean absolute error, also known as L1 loss is one of the simplest loss functions. It is calculated by taking the absolute difference between the predicted values and the actual values and averaging it across the dataset. Mathematically speaking, it is the arithmetic average of absolute errors. MAE measures only the magnitude of the errors and doesn’t concern itself with their direction. The lower the MAE, the higher the accuracy of a model.</p><p>Mathematically, MAE can be expressed as follows,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/361/1*ZMERG6iRzC4KOJ_2IoiRWw.png" /><figcaption>Mean Absolute Error</figcaption></figure><p>where y_i = actual value, y_hat_i = predicted value, n = sample size</p><pre>def mean_absolute_error(true, pred):<br>    abs_error = np.abs(true - pred)<br>    sum_abs_error = np.sum(abs_error)<br>    mae_loss = sum_abs_error / true.size<br>    return mae_loss</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*pbjTFttGG1ajIJ_qGXlNqw.png" /><figcaption>MAE Plot</figcaption></figure><p><strong>Pros</strong></p><ul><li>It is easy to calculate.</li><li>All the errors are weighted on the same scale since absolute values are taken.</li><li>It is useful if the training data has outliers as MAE does not penalise high errors caused by outliers.</li><li>It provides an even measure of how well the model is performing.</li></ul><p><strong>Cons</strong></p><ul><li>Sometimes the large errors coming from the outliers ends up being treated as the same as low errors.</li><li>MAE follows scale-dependent accuracy measure where it uses the same scale as the data being measured. Hence it cannot be used to compare series’ using different measures.</li><li>One of the main disadvantage of MAE is that it is not differentiable at zero. Many optimization algorithms tends to use differentiation to find optimum value for parameters.</li><li>It can be challenging to compute gradients in MAE.</li></ul><h3>Mean Bias Error (MBE)</h3><p><em>Bias</em> in “Mean Bias Error” is the tendency of a measurement process to over estimate or under estimate the value of a parameter. Bias has only direction which can be either positive or negative. A positive bias means the error from the data is overestimated and a negative bias means the error is underestimated. Mean Bias Error (MBE) is the mean of the difference between the predicted values and the actual values. MBE quantifies the overall bias and captures the average bias in the prediction. It is almost similar to MAE, the only difference being the absolute value is not taken here. MBE should be handled carefully as the positive and negative errors can cancel out.</p><p>The formula for MBE,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/345/1*lF98VEKnBzXpmOyOEyQqAA.png" /><figcaption>Mean Bias Error</figcaption></figure><pre>def mean_bias_error(true, pred):<br>    bias_error = true - pred<br>    mbe_loss = np.mean(np.sum(diff) / true.size)<br>    return mbe_loss</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*hHtqUbFlw2K23wY5cup80Q.png" /><figcaption>MBE Plot</figcaption></figure><p><strong>Pros</strong></p><ul><li>MBE is a good measure if you want to check the direction of the model (i.e. whether there is a positive or negative bias) and rectify the model bias.</li></ul><p><strong>Cons</strong></p><ul><li>It is not a good measure in terms of magnitude as the errors tend to compensate each other.</li><li>It is not highly reliable because sometimes high individual errors produce low MBE.</li><li>MBE can be consistently wrong in one direction. For example, if you’re trying to predict traffic patterns and it always shows lower traffic than which is actually observed.</li></ul><h3>Relative Absolute Error (RAE)</h3><p>Relative absolute error is computed by taking the total absolute error and dividing it by the absolute difference between the mean and the actual value.</p><p>RAE is expressed as,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/304/1*YR71DVW0H6_rrjK5kxar9Q.png" /><figcaption>Relative Absolute Error</figcaption></figure><p>where y_bar is the mean of the n actual values.</p><p>RAE measures the performance of a predictive model and is expressed in terms of a ratio. The value of RAE can range from zero to one. A good model will have values close to zero, with zero being the best value. This error shows how the mean residual relates to the mean deviation of the target function from its mean.</p><pre>def relative_absolute_error(true, pred):<br>    true_mean = np.mean(true)<br>    squared_error_num = np.sum(np.abs(true - pred))<br>    squared_error_den = np.sum(np.abs(true - true_mean))<br>    rae_loss = squared_error_num / squared_error_den<br>    return rae_loss</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*Ilrhr5ui6x0ywg0QzPzIOA.png" /><figcaption>RAE Plot</figcaption></figure><p><strong>Pros</strong></p><ul><li>RAE can be used to compare between models where errors are measured in different units.</li><li>In some cases, RAE is reliable as it offers protection from outliers.</li></ul><p><strong>Cons</strong></p><ul><li>One main drawback of RAE is that it can be undefined if the <a href="https://en.wikipedia.org/wiki/Reference_class_forecasting">reference forecast</a> is equal to the ground truth.</li></ul><h3>Mean Absolute Percentage Error (MAPE)</h3><p>Mean absolute percentage error is calculated by taking the difference between the actual value and the predicted value and dividing it by the actual value. An absolute percentage is applied to this value and it is averaged across the dataset. MAPE is also known as Mean Absolute Percentage Deviation (MAPD). It increases linearly with an increase in error. The smaller the MAPE, the better the model performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/409/1*yFroyKz8HekimjsHGCfhLg.png" /><figcaption>Mean Absolute Percentage Error</figcaption></figure><pre>def mean_absolute_percentage_error(true, pred):<br>    abs_error = (np.abs(true - pred)) / true<br>    sum_abs_error = np.sum(abs_error)<br>    mape_loss = (sum_abs_error / true.size) * 100<br>    return mape_loss</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*JI5S_dGfzetjYlRT6YBXnA.png" /><figcaption>MAPE Plot</figcaption></figure><p><strong>Pros</strong></p><ul><li>MAPE is independent of the scale of the variables since its error estimates are in terms of percentage.</li><li>All errors are normalized on a common scale and it is easy to understand.</li><li>As MAPE uses absolute percentage errors, the problem of positive values and the negative values cancelling each other out is avoided.</li></ul><p><strong>Cons</strong></p><ul><li>One main drawback of MAPE is when the denominator value encounters zero. We are faced with the “division by zero” problem as it it not defined.</li><li>MAPE penalises negative errors more than positive errors. Hence it is biased when we compare the accuracy of prediction methods as it will pick a method whose values are too low by default.</li><li>Since division operation is used, for the same error a change in the actual value will cause a difference in the loss. Consider a scenario when the actual value is 100 and the predicted value is 75, the loss would be 25%. While the actual value is 50 and the predicted value is 75, the loss would be 50%. But in both cases the actual error would be the same. i.e., 25.</li></ul><h3>Mean Squared Error (MSE)</h3><p>MSE is one of the most common regression loss functions. In Mean Squared Error also known as L2 loss, we calculate the error by squaring the difference between the predicted value and actual value and averaging it across the dataset. MSE is also known as Quadratic loss as the penalty is not proportional to the error but to the square of the error. Squaring the error gives higher weight to the outliers, which results in a smooth gradient for small errors. Optimization algorithms benefit from this penalisation for large errors as it is helpful in finding the optimum values for parameters. MSE will never be negative since the errors are squared. The value of the error ranges from zero to infinity. MSE increases exponentially with an increase in error. A good model will have MSE value closer to zero.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/348/1*X5q-geTwO6Cfaj_WctiD7w.png" /><figcaption>Mean Squared Error</figcaption></figure><pre>def mean_squared_error(true, pred):<br>    squared_error = np.square(true - pred) <br>    sum_squared_error = np.sum(squared_error)<br>    mse_loss = sum_squared_error / true.size<br>    return mse_loss</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*XWTOBqGhBfHl64qtXba4kw.png" /><figcaption>MSE Plot</figcaption></figure><p><strong>Pros</strong></p><ul><li>MSE values are expressed in quadratic equations. Hence when we plot it, we get a gradient descent with only one global minima.</li><li>For small errors, it converges to the minima effeciently. There is no local minima.</li><li>MSE penalises the model for having huge errors by squaring them.</li><li>It is particularly helpful in weeding out outliers with large errors from the model by putting more weight on them.</li></ul><p><strong>Cons</strong></p><ul><li>One of the advantages of MSE becomes a disadvantage when there is a bad prediction. The sensitivity to outliers magnifies the high errors by squaring them.</li><li>MSE will have the same effect for a single large error as to many smaller errors. But mostly we will be looking for a model which performs good enough on an overall level.</li><li>MSE is scale-dependent as its scale depends on the scale of the data. This makes it highly undesirable for comparing between different measures.</li><li>When a new outlier is introduced into the data, the model will try to take in the outlier. By doing so it will produce a different line of best fit which may cause the final results to be skewed.</li></ul><h3>Root Mean Squared Error (RMSE)</h3><p>RMSE is computed by taking the square root of MSE. RMSE is also called the Root Mean Square <a href="http://deviation.it/">Deviation.It</a> measures the average magniture of the errors and is concerend with the deviations from the actual value. RMSE value with zero indicates that the model has a perfect fit. The lower the RMSE, the better the model and its predictions. A higher RMSE indicates that there is a large deviation from the residual to the ground truth. RMSE can be used with different features as it helps in figuring out if the feature is improving the model’s prediction or not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*-yx7IsF8gEw7-mB7lvPzfQ.png" /><figcaption>Root Mean Squared Error</figcaption></figure><pre>def root_mean_squared_error(true, pred):<br>    squared_error = np.square(true - pred) <br>    sum_squared_error = np.sum(squared_error)<br>    rmse_loss = np.sqrt(sum_squared_error / true.size)<br>    return rmse_loss</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*J2h5qqAUoqEntg7KkHKHJw.png" /><figcaption>RMSE Plot</figcaption></figure><p><strong>Pros</strong></p><ul><li>RMSE is easy to understand.</li><li>It serves as a heuristic for training models.</li><li>It is computationally simple and easily differentiable which many optimization algorithms desire.</li><li>RMSE does not penalize the errors as much as MSE does due to the square root.</li></ul><p><strong>Cons</strong></p><ul><li>Like MSE, RMSE is dependent on the scale of the data. It increaes in magnitude if the scale of the error increases.</li><li>One major drawback of RMSE is its sensitivity to outliers and the outliers have to be removed for it to function properly.</li><li>RMSE increases with an increase in the size of the test sample. This is an issue when we calculate the results on different test samples.</li></ul><h3>Relative Squared Error (RSE)</h3><p>In order to calculate Relative Squared Error, you take the Mean Squared Error (MSE) and divide it by the square of the difference between the actual and the mean of the data. In other words, we divide the MSE of our model by the MSE of a model which uses mean as the predicted value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/305/1*wtDMQdst08mIxKTmFQnv2g.png" /><figcaption>Relative Squared Error</figcaption></figure><pre>def relative_squared_error(true, pred):<br>    true_mean = np.mean(true)<br>    squared_error_num = np.sum(np.square(true - pred))<br>    squared_error_den = np.sum(np.square(true - true_mean))<br>    rse_loss = squared_error_num / squared_error_den<br>    return rse_loss</pre><p>The output value of RSE is expressed in terms of ratio. It can range from zero to infinity. A good model should have value close to zero while a model with value greater than 1 is not reasonable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*LZ8zp7gR_doUD6tvaNSpqA.png" /><figcaption>RSE Plot</figcaption></figure><p><strong>Pros</strong></p><ul><li>RSE is not scale-dependent. Hence it can be used to compare between models where errors are measured in different units.</li><li>RSE is not sensitive to the mean and the scale of predictions.</li></ul><h3>Normalized Root Mean Squared Error (NRMSE)</h3><p>The Normalized RMSE is generally computed by dividing a scalar value. It can be in different ways like,</p><ul><li>RMSE / maximum value in the series</li><li>RMSE / mean</li><li>RMSE / difference between the maximum and the minimum values (if mean is zero)</li><li>RMSE / standard deviation</li><li>RMSE / interquartile range</li></ul><pre># implementation of NRMSE with standard deviation</pre><pre>def normalized_root_mean_squared_error(true, pred):<br>    squared_error = np.square((true - pred))<br>    sum_squared_error = np.sum(squared_error)<br>    rmse = np.sqrt(sum_squared_error / true.size)<br>    nrmse_loss = rmse/np.std(pred)<br>    return nrmse_loss</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*0LBHUYmTluwUjUAfGGiB8A.png" /><figcaption>NRMSE Plot (standard deviation)</figcaption></figure><p>Sometimes choosing the interquartile range maybe the best bet as other methods are prone to outliers. NRMSE is a good measure when you want to compare the models of different dependent variables or when the dependent variables are modified (log transformed or standardized). It overcomes the scale-dependency and eases comparison between models of different scales or even between datasets.</p><h3>Relative Root Mean Squared Error (RRMSE)</h3><p>RRMSE is a dimensionless form of RMSE. Relative Root Mean Square Error (RRMSE) is the root mean squared error normalized by the root mean square value where each residual is scaled against the actual value. While RMSE is restricted by the scale of original measurements, RRMSE can be used to compare different measurement techniques. When your predictions are inaccurate, it results in an increased RRMSE. RRMSE expresses the error relatively or in a percentage form. Model accuracy is,</p><ul><li>Excellent when RRMSE &lt; 10%</li><li>Good when RRMSE is between 10% and 20%</li><li>Fair when RRMSE is between 20% and 30%</li><li>Poor when RRMSE &gt; 30%</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/431/1*GSJvPw6ClWFGHnxj-NfCvw.png" /><figcaption>Relative Root Mean Squared Error</figcaption></figure><pre>def relative_root_mean_squared_error(true, pred):<br>    num = np.sum(np.square(true - pred))<br>    den = np.sum(np.square(pred))<br>    squared_error = num/den<br>    rrmse_loss = np.sqrt(squared_error)<br>    return rrmse_loss</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*amB6XX4Vw-zbr4pZ7ZUFXg.png" /><figcaption>RRMSE Plot</figcaption></figure><h3>Root Mean Squared Logarithmic Error (RMSLE)</h3><p>Root Mean Squared Logarithmic Error is calculated by applying log to the actual and the predicted values and then taking their differences. RMSLE is robust to outliers where the small and the large errors are treated evenly.</p><p>It penalises the model more if the predicted value is less than the actual value while the model is less penalised if the predicted value is more than the actual value. It does not penalise high errors due to the log. Hence the model has a large penalty for underestimation than overestimation. This can be helpful in situations where we are not bothered by overestimation but underestimation is not acceptable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*TQwlWzCcVuwDAfI9yUC17A.png" /><figcaption>Root Mean Squared Logarithmic Error</figcaption></figure><pre>def root_mean_squared_log_error(true, pred):<br>    square_error = np.square((np.log(true + 1) - np.log(pred + 1)))<br>    mean_square_log_error = np.mean(square_error)<br>    rmsle_loss = np.sqrt(mean_square_log_error)<br>    return rmsle_loss</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*G5lUcCZ5zT9F559lHYhetw.png" /><figcaption>RMSLE Plot</figcaption></figure><p><strong>Pros</strong></p><ul><li>RMSLE is not scale-dependent and is useful across a range of scales.</li><li>It is not affected by large outliers.</li><li>It considers only the relative error between the actual value and the predicted value.</li></ul><p><strong>Cons</strong></p><ul><li>It has biased penalty where it penalises underestimation more than overestimation.</li></ul><h3>Huber Loss</h3><p>What if you want a function that learns about the outliers as well as ignores them? Well, Huber loss is the one for you. Huber loss is a combination of both linear and quadratic scoring method. It has a hyperparameter delta (𝛿) which can be tuned according to the data. Loss will be linear (L1 loss) for values above delta and quadratic (L2 loss) for values below it. It balances and combines good properties of both MAE (Mean Absolute Error) and MSE (Mean Squared Error). In other words, for loss values less than delta, MSE will be used and for loss values greater than delta, MAE will be used. The choice of delta (𝛿) is extremely critical because it defines our choice of the outlier. Huber loss reduces the weight we put on outliers for larger loss values by using MAE while for smaller loss values it maintains a quadratic function using MSE.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/609/1*5zxb2i1BAM6wcfkm0DMirw.png" /><figcaption>Huber Loss</figcaption></figure><pre>def huber_loss(true, pred, delta):<br>    huber_mse = 0.5 * np.square(true - pred)<br>    huber_mae = delta * (np.abs(true - pred) - 0.5 * (np.square(delta)))<br>    return np.where(np.abs(true - pred) &lt;= delta, huber_mse, huber_mae)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*xmXTIRPvl_GhLO1r12ADdQ.png" /><figcaption>Huber plot with different (0.5, 1, 5) delta values</figcaption></figure><p><strong>Pros</strong></p><ul><li>It is differentiable at zero.</li><li>Outliers are handled properly due to the linearity above delta.</li><li>The hyperparameter, 𝛿 can be tuned to maximize model accuracy.</li></ul><p><strong>Cons</strong></p><ul><li>The additional conditionals and comparisons makes Huber loss computationally expensive for large datasets.</li><li>In order to maximize model accuracy, 𝛿 needs to be optimized and it is an iterative process.</li><li>It is differentiable only once.</li></ul><h3>Log Cosh Loss</h3><p>Log cosh calculates the logarithm of the hyperbolic cosine of the error. This function is smoother than quadratic loss. It works like MSE but is not affected by the large prediction errors. It is quite similar to huber loss in the sense that it is a combination of both linear and quadratic scoring methods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/473/1*disNhi7tbubxu1fdMC-DVw.png" /><figcaption>Log Cosh Loss</figcaption></figure><pre>def log_cosh(true, pred):<br>    logcosh = np.log(np.cosh(pred - true))<br>    logcosh_loss = np.sum(logcosh)<br>    return logcosh_loss</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*UP3TJmL46ZhkPDqDrCQMuQ.png" /><figcaption>Log-Cosh plot</figcaption></figure><p><strong>Pros</strong></p><ul><li>It has the advantages of Huber loss while being twice differentiable everywhere. Some optimization algorithms like XGBoost favours double differentiables over functions like Huber which can be differentiable only once.</li><li>It requires less computations than Huber.</li></ul><p><strong>Cons</strong></p><ul><li>It is less adaptive as it follows a fixed scale.</li><li>Compared to Huber loss, the derivation is more complex and requires much in depth study.</li></ul><h3>Quantile Loss</h3><p>Quantile regression loss function is applied to predict quantiles. The quantile is the value which determines how many values in the group falls below or above a certain limit. It estimates the conditional median or <em>quantile</em> of the response(dependent) variables across values of the predictor(independent) variables. The loss function is an extension of MAE except the 50th percentile, where it is MAE. It provides prediction intervals even for residuals with non-constant variance and it does not assume a particular parametric distribution for the response.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GSs59BqcTEo0ELhJAGtEQQ.png" /><figcaption>Quantile Loss</figcaption></figure><p>𝛾 represents the required quantile. The quantiles values are selected based on how we want to weigh the positive and the negative errors.</p><p>In the loss function above, 𝛾 has value between 0 and 1. When there is underestimation, the first part of the formula will dominate and for overestimation, the second part will dominate. The chosen value of quantile(𝛾) gives different penalties for over-prediction and under prediction. When 𝛾 = 0.5, underestimation and overestimation is penalised by the same factor and the median is obtained. When the value of 𝛾 is larger, overestimation is penalised more than underestimation. For example, when 𝛾 = 0.75 the model will penalise overestimation and it will cost three times as much as underestimation. Optimization algorithms based on gradient descent learns from the quantiles instead of the mean.</p><pre>def quantile_loss(true, pred, gamma):<br>    val1 = gamma * np.abs(true - pred)<br>    val2 = (1-gamma) * np.abs(true - pred)<br>    q_loss = np.where(true &gt;= pred, val1, val2)<br>    return q_loss</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*g_wN1NkPvMAN6KItw21dkg.png" /><figcaption>Quantile loss plot with different (0.25, 0.5, 0.75) gamma values</figcaption></figure><p><strong>Pros</strong></p><ul><li>It is particularly useful when we are predicting an interval instead of point estimates.</li><li>This function can also be used to calculate prediction intervals in neural nets and tree based models.</li><li>It is robust to outliers.</li></ul><p><strong>Cons</strong></p><ul><li>Quantile loss is computationally intensive.</li><li>If we use squared loss to measure the efficiency or if we are to estimate the mean, then quantile loss will be worse.</li></ul><p>Thank you for reading all the way down here! I hope this article was helpful in your learning journey. I would love to hear in the comments about any other loss functions that I have missed. Happy Evaluating!</p><p>Connect with me on <a href="https://www.linkedin.com/in/padhma-sahithya-muniraj/">LinkedIn</a></p><h4>References</h4><ul><li>Data Mining Algorithms, Explained using R</li><li><a href="https://www.sciencedirect.com/topics/engineering/mean-bias-error">https://www.sciencedirect.com/topics/engineering/mean-bias-error</a></li><li><a href="https://www.sciencedirect.com/science/article/abs/pii/S1364032115013258">https://www.sciencedirect.com/science/article/abs/pii/S1364032115013258</a></li><li><a href="https://journals.plos.org/plosntds/article?id=10.1371/journal.pntd.0005295">https://journals.plos.org/plosntds/article?id=10.1371/journal.pntd.0005295</a></li><li><a href="https://support.sas.com/resources/papers/proceedings17/SAS0525-2017.pdf">https://support.sas.com/resources/papers/proceedings17/SAS0525-2017.pdf</a></li><li><a href="https://stats.stackexchange.com/questions/39002/when-is-quantile-regression-worse-than-ols">https://stats.stackexchange.com/questions/39002/when-is-quantile-regression-worse-than-ols</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8dac47e327e2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/loss-functions-to-evaluate-regression-models-8dac47e327e2">Loss functions to evaluate Regression Models</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Analyzing popular repositories on GitHub]]></title>
            <link>https://medium.com/analytics-vidhya/analyzing-popular-repositories-on-github-f6ee81e7ad4e?source=rss-a5a3c9ba4575------2</link>
            <guid isPermaLink="false">https://medium.com/p/f6ee81e7ad4e</guid>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-visualization]]></category>
            <category><![CDATA[github]]></category>
            <dc:creator><![CDATA[Padhma Muniraj]]></dc:creator>
            <pubDate>Mon, 12 Jul 2021 14:02:32 GMT</pubDate>
            <atom:updated>2021-09-17T05:09:14.233Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DbJrZ73enNNmnB_UxBDcmQ.jpeg" /><figcaption><a href="https://unsplash.com/photos/wX2L8L-fGeA">Source</a></figcaption></figure><p>GitHub is one of the most popular version control and source code management platform. It is also one of the largest social networking sites for programmers. Software developers use it to showcase their skills to recruiters and hiring managers. By analyzing the repositories on GitHub, we can get valuable insights like user behavior, what makes a repository popular or what technologies are trending among developers nowadays, and much more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zqr0LkCS3Qvy_e8ESHO5XQ.jpeg" /><figcaption>Image: <a href="https://www.reddit.com/r/ProgrammerHumor/comments/836tsw/um_actually/">Source</a></figcaption></figure><p><a href="https://www.kaggle.com/padhmam/analysing-popular-repositories-on-github"><strong><em>You can find the entire code used in the article here</em></strong></a></p><p>I have used <em>the </em><a href="https://www.kaggle.com/vatsalparsaniya/github-repositories-analysis">‘GitHub Repositories 2020’</a> dataset from Kaggle as it is more recent.</p><h3>Implementation</h3><p>Let’s <strong><em>git</em></strong> this started by importing necessary libraries and reading the input data,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8f409c08c34fb0ec044c09d5b83e4f04/href">https://medium.com/media/8f409c08c34fb0ec044c09d5b83e4f04/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/0*7XDkVhAdQJVccMEx.PNG" /></figure><p>The dataset contains 19 columns, out of which I have picked 11 columns based on the most popular GitHub terminologies and the ones relevant to the context of this analysis. You can see that there are typos in the column names, I have renamed them for better clarity.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/afeee61991f3f7542b60203e41e290ea/href">https://medium.com/media/afeee61991f3f7542b60203e41e290ea/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*i6x2rJN7r2E-ZgcP.PNG" /></figure><p>A quick brief about the columns in the data,</p><ul><li><strong>Topic</strong> — A label describing the field or domain of the repo</li><li><strong>Repo_Name</strong> — Name of the repo (short name for repository)</li><li><strong>User_Name</strong> — Repository owner’s name</li><li><strong>Star</strong> — Number of stars a repo has received</li><li><strong>Fork</strong> — Number of times a repo has been forked</li><li><strong>Watch</strong> — Number of users watching the repo</li><li><strong>Issues</strong> — Number of open issues</li><li><strong>Pull_Requests</strong> — Total pull requests raised</li><li><strong>Topic_Tags</strong> — List of topic tags added to that repo by user</li><li><strong>Commits</strong> — Total number of commits made</li><li><strong>Contributors</strong> — Number of people contributing to the repo</li></ul><p>Notice how <em>Star</em>, <em>Fork</em>, and <em>Watch</em> columns contain <em>‘k’s </em>to denote thousands, so let us convert them to multiples of 1000. Also, replacing the <em>‘,’</em>(commas) from the <em>Issues</em> and <em>Commits</em> columns.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5f305f5a37fb4133eb749df3773f4842/href">https://medium.com/media/5f305f5a37fb4133eb749df3773f4842/href</a></iframe><p>Now that the columns are numeric, we can get some basic statistical information from them.</p><pre># display basic statistical details about the columns github_df.describe()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/0*haLGVWAbWRq9sI_f.PNG" /></figure><h3>1. Analysis of top repositories based on popularity</h3><p><em>What makes a GitHub repo popular?</em> This question can be answered with 3 Metrics — Star, Watch, and Fork.</p><ul><li>Star — When a user likes your repo or they want to show some appreciation, they Star it.</li><li>Watch — When a user wants to be notified of all the activities in a repo, then they Watch it.</li><li>Fork — When a user wants a copy of the repo or they intend to make any contributions, they Fork it.</li></ul><pre># create a dataframe with average values of the columns across all topics </pre><pre>pop_mean_df = github_df.groupby(&#39;Topic&#39;).mean().reset_index() pop_mean_df</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/717/0*jW32XGv4e72aX4SY.PNG" /></figure><h4>1.1 Analysis of Stars</h4><p>Visualizing the average number of stars across each topic,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/69c8639fc45476a91a18fcb4871b6b88/href">https://medium.com/media/69c8639fc45476a91a18fcb4871b6b88/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/0*RRufcBiif4lc3aeD.PNG" /></figure><pre># top 10 most starred repos<br>github_df.nlargest(n=10, columns=&#39;Star&#39;)[&#39;Repo_Name&#39;,&#39;Topic&#39;,&#39;Star&#39;]]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/0*KqJeVIjJNgVI8TLE.PNG" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/783/0*ORGp_RS-QMPlrgNK.PNG" /></figure><p><strong><em>In the top 10 most starred repositories, 4 are frameworks (Vue, React, TensorFlow, BootStrap) and 6 of them are about JavaScript.</em></strong></p><h4>1.2 Analysis of Watch</h4><p>Visualizing the average number of watchers across each topic,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/738/0*7akOl3oIxPJdbKTG.PNG" /></figure><p><em>Note: Code for the above graph is the same as the ‘Average Stars on each topic’ except for the column names. I have not added the same to avoid redundancy.</em></p><pre># top 10 most watched repos <br>github_df.nlargest(n=10, columns=&#39;Watch&#39;)[[&#39;Repo_Name&#39;,&#39;Topic&#39;,&#39;Watch&#39;]]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/0*j554LxHvoORmfNj_.PNG" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/780/0*1iJu8mY38Dbgrzgr.PNG" /></figure><p><strong><em>In the top 10 most-watched repositories, 4 are frameworks (TensorFlow, BootStrap, React, Vue), 6 are about JavaScript, and 5 of them contain learning content for programmers.</em></strong></p><h4>1.3 Analysis of Fork</h4><p>Visualizing the average number of forks across each topic,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/0*LeGvkhgdZgQfl_yC.PNG" /></figure><pre># top 10 most forked repos <br>github_df.nlargest(n=10, columns=&#39;Fork&#39;)[&#39;Repo_Name&#39;,&#39;Topic&#39;,&#39;Fork&#39;]]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/0*FUp7HSHseetfOMD7.PNG" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/0*9Vhy7tUAST4uENE7.PNG" /></figure><p><strong><em>In the top 10 most forked repositories, 4 are frameworks (TensorFlow, bootstrap, spring-boot, react) and 5 of them contain learning content for programmers.</em></strong></p><h4>1.4 Relationship between Star, Fork, and Watch</h4><p>Oftentimes users fork a repo when they want to contribute to it. So, let’s explore the relationship between star-fork and watch-fork.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/81685ba88385d051f67abde3f1f8eafa/href">https://medium.com/media/81685ba88385d051f67abde3f1f8eafa/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/0*8-iP333JjxManzsP.PNG" /></figure><pre># set figure size and dpi <br>fig, ax = plt.subplots(figsize=(8,4), dpi=100) </pre><pre># set seaborn theme for background grids <br>sns.set_theme(&#39;paper&#39;) </pre><pre># plot the data <br>sns.regplot(data=github_df, x=&#39;Watch&#39;, y=&#39;Fork&#39;, color=&#39;purple&#39;); </pre><pre># set x and y-axis labels and title <br>ax.set_xlabel(&#39;Watch&#39;, fontsize=13, color = &#39;#333F4B&#39;) ax.set_ylabel(&#39;Fork&#39;, fontsize=13, color = &#39;#333F4B&#39;) fig.subtitle(&#39;Relationship between Watch and Fork&#39;,fontsize=18, color = &#39;#333F4B&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/757/0*-TV6Gne-ZNu-PrNc.PNG" /></figure><p>The data points are much more closer to the regression line between Watch and Fork compared to Star and Fork.</p><p><strong><em>From this we can conclude, if a user is watching a repo then they are most likely to fork it.</em></strong></p><h3>2. Analysis of users with more repositories</h3><p>Let’s take a look at users who have more popular repositories.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2a7aa4f2f6969a0e2f164f143adbf0be/href">https://medium.com/media/2a7aa4f2f6969a0e2f164f143adbf0be/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/0*325vu4e9_gziyPoX.PNG" /></figure><p>In the top 10 users with more repositories,</p><ul><li><strong><em>Microsoft tops the list with 17 repos.</em></strong></li><li><strong><em>Google follows with 15 repos.</em></strong></li><li><strong><em>6 of them are companies or owned by a company (Microsoft, Google, Adafruit, Alibaba, PacktPublishing, flutter)</em></strong></li><li><strong><em>3 are individual users (junyanz, rasbt, MicrocontrollersAndMore)</em></strong></li></ul><h3>3.Understanding contribution activities across the repositories</h3><p><em>GitHub is famous for its contribution graph.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/925/0*EGHgAkyWqcbU3fsW.png" /></figure><p>This graph is a record of all the contributions that a user has made. Whenever a user makes a commit, opens an issue, or proposes a pull request, it is considered a contribution. There are four columns related to contributions in our dataset, <em>Issues, Pull_Requests, Commits, Contributors</em>. Let us see if there is any real relationship between them.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5b2c52fac72bb9045c8f3ae73fc123e3/href">https://medium.com/media/5b2c52fac72bb9045c8f3ae73fc123e3/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/619/0*cdyrP4B1vFPKSBSt.PNG" /></figure><p>The number of Commits is not dependent on either issue, pull requests, or contributors. There is a Moderate Positive Relationship between Issues and Pull Requests.</p><p>Let’s explore the top 100 popular repositories and see if it’s the same,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0f6aa88ceb320a38e7bf56212c1bc43c/href">https://medium.com/media/0f6aa88ceb320a38e7bf56212c1bc43c/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/0*xDZhgPUSoKpepQ0G.PNG" /></figure><p>It’s nearly the same in the top 100 popular repositories as in the overall dataset.</p><p>Let’s check for users with more repositories,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7b5b66573aec3b55cdad1b4f137aaac4/href">https://medium.com/media/7b5b66573aec3b55cdad1b4f137aaac4/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*dc7eLwh9JhYmetDD.PNG" /></figure><p>Surprisingly, users with more repos tend to be more active. There is a fairly strong positive correlation between</p><ul><li>Commits and pull requests</li><li>Commits and Issues</li><li>Issues and Pull requests</li></ul><p>As far as contributions are concerned,</p><ul><li><strong><em>There is no real relationship between contribution activities in the overall dataset.</em></strong></li><li><strong><em>There is no correlation among contributions in the Top 100 popular repos either.</em></strong></li><li><strong><em>If the users tend to have more repos, then the chances of contributions are fairly higher.</em></strong></li></ul><h3>4. Analysis of Topic Tags</h3><p>Adding tags to a repository is a way of classifying them with topics. It helps other users find and contribute to that repo and also helps you in exploring topics across the platform by type, domain, technology, etc.,</p><p>The topic_tags column consists of lists. In order to find popular tags, convert the entire column to a list of lists and count the occurrence of each tag. With that, we can visualize some of the most popular topic tags and see which topics tend to be tagged more.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aca4d07874f1a2a79beb35f931d1a7ff/href">https://medium.com/media/aca4d07874f1a2a79beb35f931d1a7ff/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/699/0*k-6sg3_AYHuY57QN.PNG" /></figure><p><strong><em>Out of the top 15 most popular tags, 10 belong to the world of data science.</em></strong></p><pre># length of tags list in each column <br>len_tags = [len(tag) for tag in topic_tags] </pre><pre># create a new column -&gt; total_tags <br>github_df[&#39;Total_Tags&#39;] = len_tags </pre><pre># group based on topic and calculate total_tags in each topic topic_wise_tags = github_df.groupby(&#39;Topic&#39;).sum()[&#39;Total_Tags&#39;].reset_index(name=&#39;Total Tags&#39;) </pre><pre># set figure size and dpi <br>fig, ax = plt.subplots(figsize=(7,4), dpi=100) </pre><pre># remove background grids <br>ax.grid(False) <br>ax.set_facecolor(&#39;white&#39;) <br>sns.despine() </pre><pre># plot the data <br>sns.barplot(data=topic_wise_tags,x=&#39;Total Tags&#39;, y=&#39;Topic&#39;, ci=None, palette=&#39;gist_rainbow&#39;) </pre><pre># set x and y-axis labels and title <br>ax.set_xlabel(&#39;Total Tags&#39;, fontsize=13, color = &#39;#333F4B&#39;) ax.set_ylabel(&#39;Topic&#39;, fontsize=13, color = &#39;#333F4B&#39;) fig.subtitle(&#39;Tags distribution across topics&#39;,fontsize=18, color = &#39;#333F4B&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/823/0*CQOawAgscYlt_Tyc.PNG" /></figure><p><strong><em>Repositories with topics Computer Vision, Data Science, and Machine Learning tend to be tagged more.</em></strong></p><p>Let’s finish up with a wordcloud of topic_tags,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1502bd180155956fce661a3f5c2356c3/href">https://medium.com/media/1502bd180155956fce661a3f5c2356c3/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/688/0*FtBA3DbbTSU5FZ62.PNG" /></figure><h3>Inference:</h3><ul><li>Among the top 10 most starred, watched, and forked repositories, 4 are frameworks.</li><li>Tensorflow is the most-watched and forked repository.</li><li>If a user is watching a repository, then they are most likely to fork it.</li><li>Microsoft and Google tend to be users with more popular repositories.</li><li>In the top 10 users with more popular repositories, 6 of them are companies.</li><li>There is no actual relationship between contribution activities (Issues, Pull Requests, Commits).</li><li>The most used tags are Machine Learning, Deep Learning, Python, Computer Vision, JavaScript.</li><li>Repositories with topics Computer Vision, Data Science, and Machine Learning have more tags.</li></ul><p>If we had analyzed data from a decade ago, these trends would have been completely different. It’s like data science had a monstrous growth in the past years!</p><p>Thank you for viewing all the way down here! I would love to connect on <a href="https://www.linkedin.com/in/padhma-sahithya-muniraj/">LinkedIn</a></p><p>Let me know in the comment section if you have any concerns, feedback, or criticism. Have a good day!</p><p><em>Originally published at </em><a href="https://www.analyticsvidhya.com/blog/2021/07/analyzing-popular-repositories-on-github/"><em>https://www.analyticsvidhya.com</em></a><em> on July 12, 2021.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f6ee81e7ad4e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/analyzing-popular-repositories-on-github-f6ee81e7ad4e">Analyzing popular repositories on GitHub</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[10 TED Talks that will inspire every Data Professional]]></title>
            <link>https://medium.com/analytics-vidhya/10-ted-talks-that-will-inspire-every-data-professional-9b42037bcec1?source=rss-a5a3c9ba4575------2</link>
            <guid isPermaLink="false">https://medium.com/p/9b42037bcec1</guid>
            <dc:creator><![CDATA[Padhma Muniraj]]></dc:creator>
            <pubDate>Mon, 17 May 2021 08:03:32 GMT</pubDate>
            <atom:updated>2021-09-17T04:57:19.619Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pgoMKxVWVPmB8n3pQjduEg.png" /></figure><p>There is a popular phrase in pop culture,</p><blockquote><em>Water, water, every where,<br> Nor any drop to drink.</em></blockquote><p>from the English poem <a href="https://en.wikipedia.org/wiki/The_Rime_of_the_Ancient_Mariner">The Rime of the Ancient Mariner</a> by <a href="https://en.wikipedia.org/wiki/Samuel_Taylor_Coleridge">Samuel Taylor Coleridge</a>.</p><p>There is also a modern version to this phrase quoted by <a href="https://en.wikipedia.org/wiki/John_Allen_Paulos">John Allen Paulos</a>,</p><blockquote><em>Data, data, everywhere, But not a thought to think.</em></blockquote><p>Can this data be used to solve world humanitarian problems? Can we save lives using data? How innovative are we with data? This article draws inspiration from 10 TED Talks that answer some of these questions.</p><p><a href="https://www.ted.com/about/our-organization">TED</a> is a non-profit organization that organizes conferences under the slogan “Ideas worth spreading”. In 1984 when TED began as a conference, the word stood for Technology, Entertainment, and Design but today there are TED talks around almost any area. The talks in this article will inspire you, captivate you, make you think, exhilarate you, or may even serve as an inspiration for your next project. Let’s dive in.</p><h3>The birth of a word — Deb Roy</h3><p>How far will you go to collect data for your research? <a href="https://en.wikipedia.org/wiki/Deb_Roy">Deb Roy</a> who is a Canadian scientist and MIT Researcher wanted to understand how we learn languages. So he set up video cameras all over his home as he welcomes his newborn son home. He shows how his team captured and parsed over 90,000 hours of video from the very first time his son started mumbling to the time when he got to pronounce a whole word — from gaga to water. He also maps out structures that show how the environment and the people we grow up with, have an impact on the learning of a language.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FRE4ce4mexrU%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DRE4ce4mexrU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FRE4ce4mexrU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5a2742c11717366f216a0ac305ff2ad0/href">https://medium.com/media/5a2742c11717366f216a0ac305ff2ad0/href</a></iframe><h3>Your company’s data could help end world hunger — Mallory Freeman</h3><p>Businesses have been involved for so long in philanthropic activities, donating services or products, or even offering financial aid. But with the coming digital age, there are new responsibilities lined up for companies. <a href="https://www.linkedin.com/in/mallory-freeman-91214b6/">Mallory Freeman</a> is the director of Data Science and Machine Learning at UPS. In this sensational talk, she insists that companies should step up and play their roles in fixing the major problems of the world. She puts forth three main ways on how companies can contribute from their end and some strong reasons as to why they should. This is a compelling talk as to how we can save the world with data.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FSa27SUR0Mlo%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DSa27SUR0Mlo&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FSa27SUR0Mlo%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/9b16af29b967728c626d3dbde1fc6837/href">https://medium.com/media/9b16af29b967728c626d3dbde1fc6837/href</a></iframe><h3>Why smart statistics are the key to fighting crime — Anne Milgram</h3><p>When Anne Milgram became the Attorney General for New Jersey, she stumbled upon two questions — who were in their criminal justice system and what data they had about them. She realized judges and prosecutors have been charging people and making decisions based on instincts and experiences and it did not turn out very well. Listen to this talk on how she created a team of data scientists, researchers, and statisticians to build data-driven assessment and statistical modeling tools to bring down crime rates in New Jersey.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FZJNESMhIxQ0%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DZJNESMhIxQ0&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FZJNESMhIxQ0%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/b8f5c9ac9c2a6ae530d81908296ee53e/href">https://medium.com/media/b8f5c9ac9c2a6ae530d81908296ee53e/href</a></iframe><h3>The best stats you’ve ever seen — Hans Rosling</h3><p>Is the world the way we see it? Is it better or worse than it was yesterday? Watch this talk as <a href="https://en.wikipedia.org/wiki/Hans_Rosling">Hans Rosling</a> — a Swedish physician, academic, author of <a href="https://www.goodreads.com/book/show/34890015-factfulness">Factfulness</a> — explains how we underestimate the tremendous changes in the world if we don’t look at the data and what it has to say. Even though this talk was delivered almost a decade ago, it’s still relevant in today’s scenario. They say, <em>“In Hans Rosling’s hands, data sings”</em>. You will be sure of it by the end of the talk.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FhVimVzgtD6w%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DhVimVzgtD6w&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FhVimVzgtD6w%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/72cf832079445b8dbf1e634afe63bd30/href">https://medium.com/media/72cf832079445b8dbf1e634afe63bd30/href</a></iframe><h3>3 ways to spot a bad statistic — Mona Chalabi</h3><p>Whether you know it or not, statistics are a part of your everyday life. From government statistics to weather forecasts, political campaigns, stock markets — they are everywhere. Most statistics raise suspicion and skepticism. How sure can we be in trusting them? Can we go about trusting them blindly or ignoring them altogether? <a href="https://monachalabi.com/">Mona Chalabi</a>, data journalist, and writer, proposes three ways that will help us to interpret, question, understand and infer good statistics from bad ones.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FZwwanld4T1w%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DZwwanld4T1w&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FZwwanld4T1w%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5354a1c14768cc73ccc28b382ef5aa91/href">https://medium.com/media/5354a1c14768cc73ccc28b382ef5aa91/href</a></iframe><h3>What do we do with all this big data? — Susan Etlinger</h3><p>There is a famous proverb in the Spider-Man comic books,</p><blockquote><em>“With great power comes great responsibility”</em></blockquote><p>That is so true for every professional who handles data. You can manipulate data to mean anything. For every great discovery or invention, there’s always a flip side. When we unlock the power of data, which part of it are we tapping in? What does it bring to the world? Are we making fast and bad decisions with it? Is there something we can learn from the past? <a href="https://www.linkedin.com/in/susanetlinger/">Susan Etlinger</a>, industry analyst and an expert on AI and big data, shares her own personal experiences and gives a convincing talk that <em>deep critical thinking</em> is the way to go.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FAWPrOvzzqZk%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DAWPrOvzzqZk&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FAWPrOvzzqZk%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/5658d41951c8ef86d4994aa46de06d39/href">https://medium.com/media/5658d41951c8ef86d4994aa46de06d39/href</a></iframe><h3>How I hacked online dating — Amy Webb</h3><p>Can you use data and analytics to find your soul mate? Well, <a href="https://amywebb.io/">Amy Webb</a>, Futurist, and Author says “ <em>Yes</em>” and she also shows how she used online dating websites in her favor and found her dream partner. Watch this hilarious talk where she crunches numbers to shortlist men in her city, amasses different data points where each data point is a character that she desires in her future husband, collects data from dating sites, and creates her own formula to find a suitable partner. <em>This talk is a fairytale.</em></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fd6wG_sAdP0U&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Dd6wG_sAdP0U&amp;image=http%3A%2F%2Fi.ytimg.com%2Fvi%2Fd6wG_sAdP0U%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/13212793fbe79693dcdfe0d78b209d1b/href">https://medium.com/media/13212793fbe79693dcdfe0d78b209d1b/href</a></iframe><h3>Why everyone should be data literate — Jordan Morrow</h3><p>Every data problem begins with a question. <em>“Why?” “What?” “How?” “Where?” “When?”</em>. Albert Einstein states,</p><blockquote>“If I had an hour to solve a problem and my life depended on the solution, I would spend the first 55 minutes determining the proper question to ask for once I know the proper question, I could solve the problem in less than five minutes”</blockquote><p>Right questioning solves the problem 70% most of the time. Let alone your job or a project, do you ever look at some numbers or statistics in social media or in the news and question yourself whether this information is correct? In this video, <a href="https://www.linkedin.com/in/jordanmorrow/">Jordan Morrow</a>, Head of data at Pluralsight, talks about data literacy and explains why it should be a skill that everyone ought to possess in this <a href="https://en.wikipedia.org/wiki/Fourth_Industrial_Revolution">Fourth Industrial Revolution</a>. He puts forth four key characteristics of data literacy and shows how you can put them into practice.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F8ovyQZ_Z8Xs%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D8ovyQZ_Z8Xs&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F8ovyQZ_Z8Xs%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/918e5119cb0ab94d3d17c25dcaecbd3c/href">https://medium.com/media/918e5119cb0ab94d3d17c25dcaecbd3c/href</a></iframe><h3>Your social media “likes” expose more than you think — Jennifer Golbeck</h3><p>There is an estimate that half of Earth’s internet population uses Facebook. All of us know that social media sites collect and use information about their users. But to what extent is the question at hand. Can you predict something about a person with content that seems totally irrelevant? A study conducted by <a href="https://www.pnas.org/">Proceedings of the national academy of sciences</a> reveals that people with high intelligence tend to like Curly fries. If you’re thinking, <em>“Wait. What?” </em>then go ahead and watch this talk given by <a href="https://en.wikipedia.org/wiki/Jen_Golbeck">Jennifer Golbeck</a>, computer scientist and professor at the University of Maryland, as she demystifies the curly fries conundrum and also shares an interesting way on how users can take control of their data.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FhgWie9dnssU%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DhgWie9dnssU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FhgWie9dnssU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/6145b465dbdf5088bf52eddd38b227bb/href">https://medium.com/media/6145b465dbdf5088bf52eddd38b227bb/href</a></iframe><h3>Art made of storms — Nathalie Mieback</h3><p>What if the numbers and data on your excel sheet had a musical note and could exist in a physical world as a 3D graph in real-time? Sounds intriguing? <a href="https://nathaliemiebach.com/">Nathalie Mieback</a>, who is an artist extracts weather data, weaves them, and translates it into musical scores. She transforms data into art, music, and a 3D visualization.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FMbhNaj88uL4%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DMbhNaj88uL4&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FMbhNaj88uL4%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/a6e070e8f83846cab4c8d57bc0c18208/href">https://medium.com/media/a6e070e8f83846cab4c8d57bc0c18208/href</a></iframe><p>I hope these talks gave you new insights and experiences. The talks in this article are by no means exhaustive. You could find many more such talks at <a href="https://www.ted.com/">TED</a>.</p><p>Thank you for reading all the way down here. Let me know in the comment section if you have any concerns, feedback, or criticism and also you’re favorite talk. Stay safe and healthy. Have a good day!</p><p>Connect with me on <a href="https://www.linkedin.com/in/padhma-sahithya-muniraj/">LinkedIn</a></p><p><em>Originally published at </em><a href="https://www.analyticsvidhya.com/blog/2021/05/10-ted-talks-that-will-inspire-every-data-professional/"><em>https://www.analyticsvidhya.com</em></a><em> on May 17, 2021.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9b42037bcec1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/10-ted-talks-that-will-inspire-every-data-professional-9b42037bcec1">10 TED Talks that will inspire every Data Professional</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python 3.9 is out! Explore 7 Exciting Python 3.9 Features That You Should Know]]></title>
            <link>https://medium.com/analytics-vidhya/python-3-9-is-out-explore-7-exciting-python-3-9-features-that-you-should-know-5300b08eca81?source=rss-a5a3c9ba4575------2</link>
            <guid isPermaLink="false">https://medium.com/p/5300b08eca81</guid>
            <category><![CDATA[python-features]]></category>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Padhma Muniraj]]></dc:creator>
            <pubDate>Sun, 11 Oct 2020 07:33:30 GMT</pubDate>
            <atom:updated>2020-10-31T17:02:58.823Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UIInd-2A3F1jvLDjL58p5w.jpeg" /></figure><p>As famous author Wayne W.Dyer puts it,</p><blockquote><em>Change the way you look at things and the things you look at change.</em></blockquote><p>When the new version of Python came out, many were worried about backward compatibility issues and other things. But if you love Python, you are sure to be excited by the cool features that have been released in the new update.</p><p>The latest version of Python is out on Monday, October 5th, 2020. This article presents you with a list of Python 3.9 features that you can try right now.</p><h3>Updating Python</h3><p>Let’s begin by updating to the new version of python. If you’re unsure of the version you are currently using, then use the code below to check your current version.</p><p>In cmd, type</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mJmIwWS9IUESC0Rk.jpg" /></figure><p>To update your version, <a href="https://www.python.org/downloads/">go to the Python downloads page</a>, get the installation and start installing. <strong>Note:</strong> Make sure to update the PATH in environment variables.</p><p>Now that we have the latest version, it’s time to check what’s new. Let’s dive in.</p><h3>1. Dictionary updates</h3><p>Dictionaries are one of the most useful and used data structures in Python. The new release has optimized the way of merging and updating dictionaries.</p><h4>1.1 Merge dictionaries</h4><p>Let’s say we have two dictionaries dict1 and dict2,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/0*UW-pbUDVxTLXYGQV.jpg" /></figure><p>dict1 contains the name and model of a car whereas dict2 contains the engine and curb weight.</p><p>Now we would like to merge the two dictionaries as they contain information about the same car. In Python 3.8 and earlier versions, to merge two dictionaries we use either</p><p>Built-in <strong>Update</strong> method,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RKMxK7-BcNm80UaU.png" /></figure><p>Or the expression <strong>**,</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EuwH4TzEoBw-T05x.png" /></figure><p>This can be inconvenient and cumbersome at times.</p><p>With Python 3.9.0, we have a sleek syntax update with the | union operator to merge two dicts,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RtU3FO8t0Ooatb2L.png" /></figure><p>This way is very clean, concise, and candid. It also improves the readability of code.</p><p>If two dictionaries have a common key, then the value in the second dictionary will be kept.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7gaK-3Dcm7751rNK.png" /></figure><h4>1.2 Update dictionary</h4><p>In order to update an existing dictionary with a new key-value pair in Python 3.8 or earlier versions, we would</p><p>Use the<strong> Update</strong> method,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aT5uD1oiFdm-KQs_.png" /></figure><p>Or update with an iterable,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*YuU5-Rg_HwrqkNtg.png" /></figure><p>In 3.9, we now have the update operator |= which does the same job in a much simpler way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Oz9DAib0bnnP_7lz.png" /></figure><p>Here, the |= works like the augmented assignment operator.</p><blockquote><em>dict1 |= dict2 means dict1 = dict1 | dict2</em></blockquote><h3>2. Type Hinting</h3><p>Under normal conditions, we don’t specify the datatype in Python. But there can be situations where we might need a variable to behave in a certain type. In such cases, Python’s flexibility can be annoying. Starting with Python 3.5, we could specify types but this update has made things way too easier.</p><p>Consider a function which calculates the diameter of the circle,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rryB6xHIcTrUSNGu.png" /></figure><p>In this scenario, the type of value passed to the function really matters. Although there is no error in the code, passing a string concatenates the same value twice.</p><p>This issue is overcome in the latest version with the help of type hinting where we can specify the expected type as int,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lb1qCSYgsV9EyfK0.png" /></figure><h3>3. String Methods</h3><p>Two new features have been added to the <strong><em>str</em></strong> object. This feature can sometimes come in handy in the Exploratory Data Analysis process.</p><p>One function removes the prefix from a string</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IVYMaG38NIZXm2Oa.png" /></figure><p>Other function removes the suffix from a string</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KrIvhprPs_y9JV0c.png" /></figure><h3>4. Math functions</h3><h3>4.1 GCD</h3><p>A modification to an existing math function has been made. In earlier versions, the function to calculate <em>GCD </em>accepts only two numbers. But now, it can be applied to any number of values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Gj813-pf8JmOkBWQ.png" /></figure><h3>4.2 LCM</h3><p>A new function has been added to the <strong><em>math</em></strong> module to calculate the <em>LCM</em>. Like the <em>GCD</em> function, the <em>LCM</em> function also accepts any number of values.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rEltqd4OyQwQTLJh.png" /></figure><h3>4.3 Nextafter</h3><p>The <em>math.nextafter() function</em> takes two arguments x and y. This Python 3.9 feature is a function that returns the next floating-point value after x going towards y. It takes into account the precision of the floating-point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HCVleW5yD78oGqk8.png" /></figure><h3>4.4 Unit in the Last Place</h3><p>The next Python 3.9 feature is the function to find the units in the last place which is the spacing between floating-point numbers. Consider a number with 3 digits, 8.14. The nearest larger number to this number is 8.15. These two numbers differ by 1 Unit in the Last Place (ULP). The return value of Math.ulp function is equivalent to this example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GgRcFSPOPA0cXXGD.png" /></figure><p>To know in detail about Units in the Last Place(ULP), check <a href="https://matthew-brett.github.io/teaching/floating_error.html">this</a>.</p><h3>5. Consistent Package Import Errors</h3><p>This is more of a fix than a feature. When a relative import went past its top-level package, earlier versions of Python raised <a href="https://bugs.python.org/issue37444">inconsistent import errors</a>.</p><blockquote><em>builtins.__import__() raise ValueError whereas</em></blockquote><blockquote><em>importlib.__import__() raise ImportError</em></blockquote><p><em>__import__()</em> now raises ImportError instead of ValueError which makes more sense.</p><h3>6. Random Bytes Generation</h3><p>A new method called randbytes() has been introduced in the <a href="https://docs.python.org/3/library/random.html"><em>random </em></a>module to generate random bytes. Python already enables the generation of random bytes through 3 different functions</p><p>but they can’t generate pseudo-random patterns.</p><p>The random.Random.randbytes() function can generate random bytes in a controlled manner and can be reproduced from the seed. Although, it can be used only when security doesn’t matter.</p><h3>7. Support for IANA timezone</h3><p>A new module zoneinfo has been introduced and added to the standard library to support the IANA time zone.</p><p>Consider an example to convert from Indian Standard Time to Madrid’s current time. Before 3.9 we would do this by installing <em>pytz</em> through pip,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*PRWzLjeXIv2JDH5u.png" /></figure><p>With the zoneinfo module, this is pretty candid. You can directly import the ZoneInfo class.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vPS0K3MexKqYYtZ_.png" /></figure><p>Apart from these, we now also have the new High-performance PEG-based parser, Graphlib module, AsyncIO and multiprocessing improvements, HTTP status codes and a bunch of redundant features has been removed. Learn <a href="https://docs.python.org/3.9/whatsnew/3.9.html">more</a>. Thank you for reading all the way down here. Let me know in the comment section if you have any concerns, feedback, or criticism. Have a good day!</p><p><em>Originally published at </em><a href="https://www.analyticsvidhya.com/blog/2020/10/7-exciting-python-3-9-feature-to-know/"><em>https://www.analyticsvidhya.com</em></a><em> on October 11, 2020.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5300b08eca81" width="1" height="1" alt=""><hr><p><a href="https://medium.com/analytics-vidhya/python-3-9-is-out-explore-7-exciting-python-3-9-features-that-you-should-know-5300b08eca81">Python 3.9 is out! Explore 7 Exciting Python 3.9 Features That You Should Know</a> was originally published in <a href="https://medium.com/analytics-vidhya">Analytics Vidhya</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Timestamp Vs Timedelta Vs Time Period]]></title>
            <link>https://medium.com/data-science/timestamp-vs-timedelta-vs-time-period-afad0a48a7d1?source=rss-a5a3c9ba4575------2</link>
            <guid isPermaLink="false">https://medium.com/p/afad0a48a7d1</guid>
            <category><![CDATA[time-series-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Padhma Muniraj]]></dc:creator>
            <pubDate>Mon, 15 Jun 2020 17:29:06 GMT</pubDate>
            <atom:updated>2022-02-15T13:41:02.451Z</atom:updated>
            <content:encoded><![CDATA[<h4>Understanding Pandas Time Series data structures</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*goF0VXkYxghil5Zt" /><figcaption>Photo by <a href="https://unsplash.com/@nate_dumlao?utm_source=medium&amp;utm_medium=referral">Nathan Dumlao</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>DATA!</strong> Not just data science but every other field in the software industry handles data. From system software to application software — handling and storing data in an efficient manner is always a challenge. A common and most effective combat strategy is the utilization of <strong><em>data structures.</em></strong></p><p>As the computer scientist <a href="https://en.wikipedia.org/wiki/Fred_Brooks">Fred Brooks</a> puts it,</p><blockquote>The programmer’s primary weapon in the never-ending battle against slow system is to change the intramodular structure. Our first response should be to reorganize the modules’ data structures.</blockquote><p>One can’t stress data structures highly enough — you can have the perfect code, perfect logic, zero errors yet storing data in a clumsy manner can be the downfall of the application.</p><p>Pandas was essentially developed for the <a href="https://en.wikipedia.org/wiki/Pandas_(software)">purpose</a> of analyzing financial time series data. It contains an ample amount of tools to deal with date and time related data. In this article, I’ll explain the basic pandas time series data structures, the type of inputs they can take, and their functionalities.</p><p>Some of the elementary data structures for working with time series data are:</p><ul><li>Time Stamps</li><li>Time Deltas</li><li>Time Periods</li></ul><h4>1) Time Stamps:</h4><p>Python provides the date and time functionality in the <a href="https://docs.python.org/3/library/datetime.html">datetime</a> module that contains three different types,</p><ul><li>date — day, month, year</li><li>time —hours, minutes, seconds, microseconds</li><li>datetime — components of both date and time</li></ul><p>While datetime in python consists of both date and time together, pandas’ alternative is the Timestamp object that encapsulates date and time together. It is a counterpart for python’s datetime but is based on the more efficient numpy.datetime64 data type.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/739/1*8uf_T4BqN0s5gbwWaPmXFg.png" /><figcaption>Time Stamp Illustration</figcaption></figure><p><em>Pandas </em><a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Timestamp.html"><em>Timestamp</em></a><em> references to a specific instant in time that has nanosecond precision(one thousand-millionth of a second).</em></p><p>It is the most basic type of time series data that associates values with specific instants in time. The Timestamp constructor is very flexible, in the sense that it can handle a variety of inputs like strings, floats, ints. Below are the examples of the different types of input that it can accept.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c5c23ae8fed2ffa1651492a594917c26/href">https://medium.com/media/c5c23ae8fed2ffa1651492a594917c26/href</a></iframe><p>In the example above, [7] and [8] contains a single scalar value. A single value such as integer or a float can also be passed to the Timestamp constructor which returns date and time equivalent to the number of seconds after the <a href="https://en.wikipedia.org/wiki/Unix_time">UNIX epoch</a> (January 1, 1970). Also, it allows human-interpretable date objects to be converted to and from a UNIX epoch for ease of computation. When a nan value is passed as in [9], it returns a NaT (not a time) value which is pandas’ null value for timestamp data.</p><p>The Timestamp constructor understands time zone conversions. By default, it is unaware of the time zone but can be made <em>time zone-aware</em> by passing a time zone to the parameter tz when creating the object. This object internally stores a <a href="https://en.wikipedia.org/wiki/Coordinated_Universal_Time">UTC </a>timestamp value that makes conversion between time zones simple.</p><p><strong><em>Note</em></strong><em>: If your column/index/object is not time zone-aware, you will get an error such as this</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/1*ldkpy1ANQVpsQRf59KDgFA.png" /><figcaption>Time Zone Conversion Error</figcaption></figure><p><em>You can overcome this by localizing (making it time-zone aware) and then converting it.</em></p><p>Apart from this, timestamps can store frequency information, has a large number of attributes and methods that are useful for data manipulation purposes. The index structure for Timestamp is the <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DatetimeIndex.html#pandas.DatetimeIndex">DatetimeIndex</a> which is of type datetime64 and contains time series specific methods for easy processing.</p><h4>2) Time Deltas:</h4><p>One of the definitions of the <a href="https://www.google.com/search?rlz=1C1CHZL_enIN707IN707&amp;sxsrf=ALeKk038CgpGyIw-ZTMsIwEuF0GYDUq1-A%3A1591970785866&amp;ei=4YvjXoHHNKOa4-EP1uKuqAQ&amp;q=what+is+delta&amp;oq=what+is+delta&amp;gs_lcp=CgZwc3ktYWIQAzIKCAAQsQMQFBCHAjICCAAyBwgAEBQQhwIyAggAMgIIADICCAAyAggAMgIIADICCAAyAggAOgQIABBHOgQIIxAnOgcIABCxAxBDOgQIABBDOgUIABCxAzoFCAAQgwFQ4g1YxyVg2SloAHABeACAAd4CiAG_F5IBBzAuNS42LjKYAQCgAQGqAQdnd3Mtd2l6&amp;sclient=psy-ab&amp;ved=0ahUKEwjBu7ypufzpAhUjzTgGHVaxC0UQ4dUDCAw&amp;uact=5">delta</a> is that it is a difference between two things or values. Timedelta is nothing but the difference in time which can refer to an amount of time or the exact length of time between two intervals and is based on numpy.timedelta64.</p><p>Similar to the Timestamp constructor, Timedelta also tends to take a variable amount of inputs. An interesting thing about it is that it can take both positive and negative values. Below are some examples to give you a glimpse of it.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/50380436b8aee2a988322329a41bea4a/href">https://medium.com/media/50380436b8aee2a988322329a41bea4a/href</a></iframe><p>Timedeltas are part of both python and pandas. They can be</p><ul><li>added or subtracted from each other</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0605ff26f5951f6f2e3c34679d39b5f4/href">https://medium.com/media/0605ff26f5951f6f2e3c34679d39b5f4/href</a></iframe><ul><li>divided by each other to return a float value</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fe83d7abbed760973b637135b1e169c1/href">https://medium.com/media/fe83d7abbed760973b637135b1e169c1/href</a></iframe><ul><li>added to timestamp</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a0f11f8e31cef47e562f0dfbee727037/href">https://medium.com/media/a0f11f8e31cef47e562f0dfbee727037/href</a></iframe><ul><li>added to datetime</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/47aa144a6e9dedd9d510e2d0186b81de/href">https://medium.com/media/47aa144a6e9dedd9d510e2d0186b81de/href</a></iframe><ul><li>added to date</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8a2a0c0d4cb7e798faf9c54254ae6638/href">https://medium.com/media/8a2a0c0d4cb7e798faf9c54254ae6638/href</a></iframe><ul><li>cannot be added to time</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/785e7daecc0bf43b72ba4c6567957b47/href">https://medium.com/media/785e7daecc0bf43b72ba4c6567957b47/href</a></iframe><p>Similar to Timestamp, Timedelta also has a large number of attributes and methods for manipulating data and the associated index structure is <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.TimedeltaIndex.html">TimedeltaIndex</a> which is of type int64.</p><h4>3) Time Periods:</h4><p>Time <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Period.html">Periods</a> references a specific length of time between a start and end timestamp which is invariable and does not overlap. The Period class takes the Period type which takes a string or an integer and encodes a fixed frequency based on numpy.datetime64.</p><p>In general, a value and a frequency parameter are passed to the Period() constructor, the frequency parameter specified as freq takes a predefined set of strings. Consider the example below,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4bd91d8b81dbc414df0d19d20ee1bf42/href">https://medium.com/media/4bd91d8b81dbc414df0d19d20ee1bf42/href</a></iframe><p>In [1] freq=&#39;A-OCT&#39; means that the frequency is annual, anchored to the end of October whereas in [3] freq=&#39;M&#39; means that the frequency is monthly. You can also specify quarterly, weekly frequencies. To know more, check out <a href="https://pandas.pydata.org/pandas-docs/version/0.12/timeseries.html">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/1*J1WOPRKuctCC_eAgQ-RwTw.png" /><figcaption>Time Period Illustration</figcaption></figure><p>A sequence of Period objects can be generated using the period_range() function that takes a start, end and frequency parameters.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f853c409cd09cb269d48076f93df675a/href">https://medium.com/media/f853c409cd09cb269d48076f93df675a/href</a></iframe><p>Converting between frequencies is a task that tends to be dealt with mostly. This can be done with the <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.asfreq.html">asfreq</a> method. Let’s consider an example to convert a monthly frequency to weekly frequency,</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f937aaa7d60b4f101183466f098fcc4f/href">https://medium.com/media/f937aaa7d60b4f101183466f098fcc4f/href</a></iframe><p>Apart from this, Periods can be converted to Timestamps and vice versa. The index structure associated with the Period objects is the PeriodIndex.</p><p>Thanks for reading all the way down here. Let me know in the comment section if you have any concerns, feedback, or criticism. Have a good day!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=afad0a48a7d1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/timestamp-vs-timedelta-vs-time-period-afad0a48a7d1">Timestamp Vs Timedelta Vs Time Period</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[3 ways to filter Pandas DataFrame by column values]]></title>
            <link>https://medium.com/swlh/3-ways-to-filter-pandas-dataframe-by-column-values-dfb6609b31de?source=rss-a5a3c9ba4575------2</link>
            <guid isPermaLink="false">https://medium.com/p/dfb6609b31de</guid>
            <category><![CDATA[pandas-dataframe]]></category>
            <category><![CDATA[data-manipulation]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[pandas]]></category>
            <dc:creator><![CDATA[Padhma Muniraj]]></dc:creator>
            <pubDate>Tue, 05 May 2020 09:00:30 GMT</pubDate>
            <atom:updated>2022-02-15T13:40:02.843Z</atom:updated>
            <content:encoded><![CDATA[<h4>Some flexible approaches to combine multiple filters</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gDllw_3KFB_6BrPA" /><figcaption>Photo by <a href="https://unsplash.com/@nate_dumlao?utm_source=medium&amp;utm_medium=referral">Nathan Dumlao</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Looking for some good book recommendations in <a href="https://www.goodreads.com/">Goodreads </a>— I found this quote,</p><blockquote>In the Information Age, the first step to sanity is FILTERING. Filter the information: extract for knowledge. — <strong>Marc Stiegler</strong></blockquote><p>I wondered how it applies to Data Science! This quote fits perfectly to the most significant and the most underrated step in the entire Data Science process — <em>Data Preprocessing! </em>Data Scientists enjoy building models so much that they overlook this process. In actual essence, this process can be intriguing. As mentioned in the quote above, filtering is knowledgable. Filtering data can really guarantee some sanity when you are stumbled upon which variables to fit on the model. In this article, I’ll share some quick ways of filtering data using <a href="https://pandas.pydata.org/">Pandas</a>.</p><p>To jump in, I have obtained <a href="https://www.kaggle.com/kyanyoga/sample-sales-data">this</a> dataset from Kaggle. The dataset contains some sample sales data and looks like the one below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/933/1*-0kH-dyBoNb5rEgLErvahQ.png" /><figcaption>Sample Sales Data</figcaption></figure><p>A simple look into the data and what it has to say:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/715/1*aWzfm58AQ7VeLewvqcI4Cg.png" /><figcaption>Sales Data Shape and Columns</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/362/1*jtMUtfPSnmaz8Xekwg_Rdg.png" /><figcaption>Sales Data Info</figcaption></figure><p>Looks good! Let’s dive in.</p><h4>1) Filtering based on one condition:</h4><p>There is a DEALSIZE column in this dataset which is either small or medium or large Let’s say we want to know the details of all the large deals. A simple way would be,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/908/1*RPAPamxnXsHbgICQbve4iA.png" /><figcaption>Large Deals</figcaption></figure><p>Filtering is pretty candid here. You pick the column and match it with the value you want. A common confusion when it comes to filtering in Pandas is the use of conditional operators. Python syntax creates trouble for many.</p><p><strong>Note</strong>: In Pandas, and is replaced with &amp; , or is replaced with | and not is replaced with ~</p><p>I find out that Madrid is the top-ranking city in terms of revenue. I’d like to compare the sales details of Madrid against all the other cities. This can be achieved by assigning conditions to variables.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/854/1*MxxrTxSzVWe56TOZl5F0YQ.png" /><figcaption>Comparing Madrid against other cities</figcaption></figure><p>Assigning conditions to variables can be a good practice since it creates a mask that can be applied whenever we want to access data under such conditions.</p><h4>2) Filtering based on multiple conditions:</h4><p>Let’s see if we can find all the countries where the order is on hold in the year 2005</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/944/1*QYUB2T966_kj7v08_UC-uA.png" /><figcaption>Applying Multiple filters</figcaption></figure><p>Inside .loc , the condition within the parentheses evaluates to a boolean value which is then applied upon the column specified.</p><p>The data returned from multiple filters depends on the operation performed. When &amp; and | operations are performed without an assignment, a series is returned. When the | operation is performed with an assignment, it modifies the DataFrame.</p><p>The list of conditions to be performed upon the DataFrame can increase drastically. Let’s consider a use case. I find out that Spain ranks second in generating total revenue, see if there any orders in Spain where the Sales didn’t cross 5000 and the Quantityordered is less than 50. This can be done in two ways:</p><ul><li>Either hard-code the list</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/524/1*fO4MQO7pBc6CvTgw5jg_XQ.png" /><figcaption>Hard Coding Multiple conditions</figcaption></figure><ul><li>Or build a list that is dynamically evaluated based on the criteria</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/660/1*d496tnNXusTLqspZtzqV7w.png" /><figcaption>Dynamically evaluated list</figcaption></figure><h4>3) Implementing the If then Constructs:</h4><p>When solving problems, a format such as this is most common:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a9c3810f2f073e9176dd296a8e8714e6/href">https://medium.com/media/a9c3810f2f073e9176dd296a8e8714e6/href</a></iframe><p>Consider a case where a new column called Income Statement is created that contains three categories — if sales is greater than 10000 then it’s considered gain, if the sales range is between 5000–10000 then it’s considered as no change and anything less than that is a loss Let’s look at a few ways of implementing this:</p><ul><li>Define a function that executes this logic and apply that to all columns in a DataFrame</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c69b2a3dad392ffb93a650c7065d9481/href">https://medium.com/media/c69b2a3dad392ffb93a650c7065d9481/href</a></iframe><ul><li>Using a lambda function</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/90feb2854d80a523043413d230c47025/href">https://medium.com/media/90feb2854d80a523043413d230c47025/href</a></iframe><ul><li>Implementing a loop can be faster than .apply</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/175449ec550e1d8276c04bd25ba85c46/href">https://medium.com/media/175449ec550e1d8276c04bd25ba85c46/href</a></iframe><p>These three are more pandas-y ways of arriving at the solution. There are many other alternatives to arrive at the solution. Feel free to check my GitHub repo <a href="https://github.com/Padhma/Data-Science-Notebooks/tree/master/Data%20Analysis/Pandas/Filtering%20DataFrame">here</a> that contains all the code mentioned in the article!</p><p>Thank you for reading all the way down here. Let me know in the comment section if you have any concerns, feedback, or criticism. Have a good day!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dfb6609b31de" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/3-ways-to-filter-pandas-dataframe-by-column-values-dfb6609b31de">3 ways to filter Pandas DataFrame by column values</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Nitty-Gritty of Advanced Indexing in Pandas]]></title>
            <link>https://medium.com/data-science/nitty-gritty-of-advanced-indexing-in-pandas-9850b2d50220?source=rss-a5a3c9ba4575------2</link>
            <guid isPermaLink="false">https://medium.com/p/9850b2d50220</guid>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[data-manipulation]]></category>
            <category><![CDATA[index]]></category>
            <category><![CDATA[data-analysis]]></category>
            <dc:creator><![CDATA[Padhma Muniraj]]></dc:creator>
            <pubDate>Mon, 20 Apr 2020 18:28:17 GMT</pubDate>
            <atom:updated>2022-02-15T13:41:28.883Z</atom:updated>
            <content:encoded><![CDATA[<h4>Why Multi-level Indexing is not as daunting as it seems</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*homoXeHnSxOP3U6X" /><figcaption>Photo by <a href="https://unsplash.com/@ozgut?utm_source=medium&amp;utm_medium=referral">Ozgu Ozden</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>There’s this beautiful quote I hear oftentimes,</p><blockquote>“Sometimes we don’t see what is in front of us”</blockquote><p>That’s so true with everything — be it the cause of your bug or the bug itself, misspelled variable names, not setting inplace = True and wondering why my DataFrame doesn’t change and quite a lot. Not just bugs or typos, even functionalities. We don’t see the basic functionalities as the building blocks of a product. One such thing which is overlooked most of the time but plays a crucial role in Data Analysis and Manipulation is <em>Index</em> in Pandas.</p><p>Indexes, Series and DataFrames are the core Pandas data structures. Indexes are the identifiers or the address to a location in a DataFrame. While manipulating one-dimensional and two-dimensional indexes are the most common practice. Learning to work with an arbitrary number of dimensions can be very handy at times. Luckily, Pandas <a href="https://pandas.pydata.org/pandas-docs/stable/user_guide/advanced.html#hierarchical-indexing-multiindex">enables</a> the use and storage of higher-dimensional data into one-dimensional data structures like series and two-dimensional data structures like DataFrames. This is very well known as the Hierarchical indexing, Advanced Indexing, or Multi-level indexing.</p><p>In this post, I’ll explain the basic how-tos of Multi-level Indexing in real-world data.</p><p>To start off, I have obtained <a href="https://www.kaggle.com/jingbinxu/sample-of-car-data">this</a> dataset from Kaggle. The dataset contains some sample data about cars and looks like the one below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/1*nNGFmDJnUGHsZr636240AQ.png" /><figcaption>Sample Car Data</figcaption></figure><h4>The Tedious Way:</h4><p>Let’s say I want to know if there are any Toyota cars with its body style as the sedan and its fuel type as diesel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/976/1*Jekf6tw-39tjIxc161nkPg.png" /></figure><p>This is one of the ways through which we can find out the desired outcome. What if we would like to know the make of all the diesel cars with its body style as the wagon and check if its price falls within $20,000?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZabPwepi_bNwQ5VFzg2PYg.png" /></figure><p>We still obtain the desired output but the code is not quite neat. As the query gets more complicated, the code gets more monotonous and lengthy.</p><h4>Advanced Indexing or Hierarchical Indexing:</h4><p>Hierarchical Indexing can help us work with an arbitrary number of dimensions. It can help us in filtering, aggregating, organizing, manipulating data for really powerful data analysis.</p><h4>1) Manipulating Indexes:</h4><p>Let’s begin by setting indexes for the DataFrame. To create multi-indexed DataFrame we pass a list of values to the index attribute instead of a string. There are so many categorical features in my dataset, but I’ll consider make first as I would like to know the details of the different set of cars. Also, I’ll consider body_style and fuel_type for better organization.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*Oy1KEGNr5PXuh27_PGG6IA.png" /><figcaption>set_index</figcaption></figure><p>Let’s take a look at the indexes and what they’ve got for us.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/1*AaEdh4YMDZ5AimM5jQQshg.png" /><figcaption>multi-level index</figcaption></figure><p>The multi-index contains three different attributes namely levels, codes, names.</p><ul><li>The levels attribute refers to the label names of the indexes. The indexes are ordered from the left. In this case, make is the highest index with it&#39;s level = 0 . It can also be referred to as level = &#39;make&#39;</li><li>The codes attribute refers to the labels for each data point that encode the levels.</li><li>The names attribute contains the string names for the levels.</li></ul><p>There can be multi-level indexes for both rows and columns. We can reset the indexes by simply passing reset_index() function to the DataFrame.</p><p>A caveat when working with multi-level indexing is sorting the indexes before attempting to perform any slicing operation. Slicing operations will deliberately throw a warning like the one below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*1totbdnFoegQrX5W_WST6A.png" /><figcaption>warning for unsorted multi-index data frame</figcaption></figure><p>We can avoid this by, passing the sort_index() function to the multi-indexed DataFrame. Also, Pandas depends on the index being sorted for optimal retrieval and search operations. We can check if the indexes are sorted using the index.is_lexsorted() function which returns a boolean value.</p><p>Other useful functions are</p><ul><li>swap_level()</li><li>stack()</li><li>unstack()</li></ul><p>swap_level() can be used to interchange the levels within the DataFrame. It takes two parameters i and j which are the level names passed as strings. Passing this function doesn’t alter the order of the values.</p><p>stack() and unstack() functions take an additional parameter where you can choose the level to be stacked or unstacked, where the stacked level becomes the new lowest level (rightmost).</p><p>You can go ahead and check the <a href="https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#indexing-label">official</a> Pandas documentation for more index modifications.</p><h4>2) Accessing Data:</h4><p>Let&#39;s see if we can find if there are any Toyota cars with its body style as the sedan and its fuel type as diesel using our Multi-Indexed DataFrame.</p><ul><li>Using .loc</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*q1404Oo3uJRU27ZOLMp65Q.png" /><figcaption>.loc</figcaption></figure><p>.<a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.loc.html">loc</a> is primarily used for label based selection where the label names are passed inside a tuple. Sometimes it is combined with pd.IndexSlice. An example would be finding all diesel cars.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/881/1*VNQztr5tS67S0AIhQROCVw.png" /><figcaption>pd.IndexSlice with .loc</figcaption></figure><p>Notice, inside pd.IndexSlice the first : selects cars of all make and the second : selects cars of all body_style whereas the outer : is to select all columns.</p><ul><li>Using .xs</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*7jv7iCBraRzI5sgDZgRzhw.png" /><figcaption>.xs</figcaption></figure><p>.<a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.xs.html">xs</a> fetches a particular cross-section of a DataFrame where the axis parameter defaults to zero axis=0 and also takes a level parameter which is an indication of which levels to be used.</p><ul><li>Using .query</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*tS0OQYX4seQVdVWhE_ULxQ.png" /><figcaption>.query</figcaption></figure><p>.<a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.query.html">query</a> uses boolean expressions for filtering or slicing. Notice the use of and and or inside the query over &amp; and | . This is because the expressions inside the query method are evaluated dynamically using the <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.eval.html#pandas.eval">eval()</a> function. Also, this method uses modified Python syntax by default.</p><ul><li>Using get_level_values</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/898/1*Rj1Mr9qHquPvoJSxtKsSVg.png" /><figcaption>get_level_values</figcaption></figure><p><a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Index.get_level_values.html">get_level_values</a> uses a vector of label values for the requested level. Here, the level values can either be passed as integers or strings.</p><p>More generally, loc and xs are for label based selection while query and get_level_values are helpful in generating conditional expressions for filtering. Not all of these methods work equally, each works best under different circumstances. It is upon us to select the best method of retrieval!</p><h4>3) Aggregating Data:</h4><p>All the built-in data aggregation methods such as mean(), min(), max(), sum() can be applied to hierarchical DataFrame. It is passed along with level parameter that influences which part of the data the aggregate is computed on. A simple example would be to find the mean price of all Volkswagen cars.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*TyWt0imrngxdEoe4TSi3zw.png" /><figcaption>average price</figcaption></figure><p>Working with Pandas Indexes has always been fun and curious! Feel free to check my GitHub repo <a href="https://github.com/Padhma/Data-Science-Notebooks/tree/master/Data%20Analysis/Pandas/Advanced%20Indexing">here</a> that contains all the code mentioned in the article.</p><p>Thanks for reading all the way down here. Let me know in the comment section if you have any concerns, feedback, or criticism. Have a good day!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9850b2d50220" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/nitty-gritty-of-advanced-indexing-in-pandas-9850b2d50220">Nitty-Gritty of Advanced Indexing in Pandas</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Recreate GitHub’s Contribution Graph Using Flask and Google Sheets]]></title>
            <link>https://medium.com/better-programming/recreating-githubs-contribution-graph-using-python-flask-and-google-sheets-73ce9c096784?source=rss-a5a3c9ba4575------2</link>
            <guid isPermaLink="false">https://medium.com/p/73ce9c096784</guid>
            <category><![CDATA[google-sheets]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[flask]]></category>
            <category><![CDATA[heroku]]></category>
            <dc:creator><![CDATA[Padhma Muniraj]]></dc:creator>
            <pubDate>Thu, 02 Apr 2020 16:24:35 GMT</pubDate>
            <atom:updated>2020-04-14T16:58:15.129Z</atom:updated>
            <content:encoded><![CDATA[<h4>Consistency breeds consistency</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xrVcailxD9lfjXcseQXMQA.png" /><figcaption>GitHub’s Contribution Graph Clone</figcaption></figure><p>I’m a productivity freak. I used to monitor my tasks each day with pen and paper. Then, as the tasks got more complicated, I started logging them in a spreadsheet. That worked fine for four weeks, but then I lost motivation. I needed a push to follow my routine. That’s where the contribution graph comes in.</p><p>If you use <a href="https://github.com/">GitHub</a>, you’ve seen the heat map in your profile that shows your contributions (commits, pull requests, etc) for every day of the year — as shown in the image above. I worked on a project every day to keep up my GitHub streak — all the time loathing that gray square in my profile! I found <a href="https://levelup.gitconnected.com/how-to-create-a-tracker-like-the-github-contribution-graph-with-node-js-and-google-sheets-5e915c668c1">this tutorial</a> in Node.js. But I fancy Python, so I built one on my own using <a href="https://flask.palletsprojects.com/">Flask</a> and Google Sheets API.</p><p>In this tutorial, I’ll walk through each step, from creating the application to deploying it on Heroku. I’ll also share the errors I found, along with their solutions.</p><h4>Set Up Your Project Environment</h4><p>Let’s start by creating an environment to work on the project. I’ll assume you’ve already installed Python (Python 3 is desirable) and pip to your system — otherwise, download Python <a href="https://www.python.org/downloads/">here</a> and follow the instructions. Pip is the package manager for Python and is helpful in installing additional libraries and dependencies required for your app. Check if pip is installed using this command in the command prompt: pip --version.</p><ol><li>You want an isolated environment for your app and its dependencies, so start by setting up a <a href="https://docs.python.org/3/library/venv.html">virtual environment</a>. Navigate to a new directory where you want to set up your project, open the command prompt and install <a href="https://pypi.org/project/virtualenv/">virtualenv</a> using the command pip install virtualenv.</li><li>Install <a href="https://virtualenvwrapper.readthedocs.io/en/latest/">virtualenvwrapper</a>, which is a set of extensions for virtualenv, by typing pip install virtualenvwrapper-win (Note: This command is only for Windows. For Unix based environment, follow <a href="https://medium.com/@gitudaniel/installing-virtualenvwrapper-for-python3-ad3dfea7c717">this</a> tutorial).</li><li>Create a virtual environment by entering mkvirtualenv your-env-name.</li><li>You can activate the virtual environment using workon your-env-name and deactivate it using the deactivate command.</li><li>After the virtual environment is up and running, install flask by entering pip install flask.</li></ol><h3>Create Your Contribution Spreadsheet</h3><p>Now, go ahead to create the spreadsheet that will act as the database. You can take a look at my sheet <a href="https://docs.google.com/spreadsheets/d/13E-axJ2W2t__bU-Y79rUgHKFbScNiyeIUXOx3xWqJs4/edit#gid=0">here</a>. It has four fields:</p><ul><li>Date, I have formatted this field similar to that of GitHub’s. You can format it however you want.</li><li>Task<em>,</em> which contains the name of each task and its description (optional).</li><li>Number of Tasks<em>, </em>which<em> </em>contains the count of the tasks.</li><li>Level<em>,</em> which uses the formula =IF(C2=0,&quot;0&quot;,if(C2&lt;=2,&quot;1&quot;,if(C2&lt;=4,&quot;2&quot;,&quot;3&quot;))) to calculate the level, based on the count of the number of tasks. The entire column is filled with this formula.</li></ul><p>You can customize these fields however you wish, but there must be a field alike level because it’s used to determine the color with which each square is filled in the graph.</p><h3>Obtain the API to Use Your Spreadsheet</h3><p>This step is crucial as it allows our app to connect with Google Sheets:</p><ul><li>Go to <a href="https://console.developers.google.com/">Google Developer’s Console</a> and sign in with your Google account.</li><li>Create a new project and click enable API.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/425/1*-XlrJ8oo-FunNrHmgf_n8g.png" /><figcaption>Enable API Button</figcaption></figure><ul><li>Look for Google Drive API and click “ENABLE”.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/679/1*DfcLiGdB4tqSfwXGLXGcuw.png" /><figcaption>Google Drive API</figcaption></figure><ul><li>After the API is enabled, click “Create Credentials”. A credentials wizard will open. Fill each field as shown below.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/470/1*Z2OBhzfwChkamV_F5m32Tg.png" /><figcaption>Credentials Wizard in Google Developers Console</figcaption></figure><ul><li>In the second step, name your service account and Select Role &gt; Project &gt; Editor. Click “continue” and it generates a .json file which looks like the one below. Rename it to Client_Secret.json.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/732/1*tFi5HjtoNjn4f5xk0kFDtQ.png" /><figcaption>Client_Secret.json</figcaption></figure><ul><li>If you try to access your spreadsheet now, you’ll get Insufficient Permission error.</li><li>In addition to Google Drive API, you’ll also have to enable the Google Sheets API.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/652/1*Nef6G39b-ZmHc0exqPsoOQ.png" /><figcaption>Google Sheets API</figcaption></figure><ul><li>Once you enable the API, go to the sheet from which you want to pull data and click the “share” button.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vWNa7OabhPapBTxZG7B_cg.png" /><figcaption>Share your Sheet</figcaption></figure><ul><li>There will be a client_email in the Client_Secret.json. Copy that email and paste it in the pop-up and hit “Send”.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*vNVcU7rd-emm12kb_ODjIA.png" /></figure><p>That’s it! Now let’s pull data from the spreadsheet and display it in our app.</p><h3>Structuring your Flask Application:</h3><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9a599d434437e5c461efc5ecc9258221/href">https://medium.com/media/9a599d434437e5c461efc5ecc9258221/href</a></iframe><p>Don’t worry, I’ll explain what each file is for and what it does as we go along. This is just to give you a glimpse of what our folder structure is. You can go <a href="https://exploreflask.com/en/latest/organizing.html">here</a> to learn more about structuring Flask apps.</p><h3>Access Google Sheets Data</h3><p>Now let’s see if we can access data from our sheet. There’s this popular Python API called <a href="https://gspread.readthedocs.io/en/latest/">gspread</a> which makes it easy to read and write from Google Sheets and Oauth and enables your application to call the Google API on behalf of the service account. Install gspread and oauth2 using pip, pip install gspread oauth2.</p><p>The JSON file is nested under static (all the static files that do not change, like CSS, JS are stored here) folder, so access it from the root using the <em>os module</em> os.path.realpath(os.path.dirname(__file__)). Then create a scope variable that contains the link to the feeds in the drive and a creds variable to access the JSON file using the service account credentials. A client variable is created to authorize gspread to use the credentials.</p><p>You can now access the sheet using client.open(&#39;Daily Contribution&#39;).sheet1 where Daily Contribution is the name of the file and sheet1 is the name of the first sheet. We’ll store the values of each column in a variable and create a common variable, data, that holds them all assign it to the return value of the function getData(), as shown in the code below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/023f2c0225b4dac2805c5fa3555ba2c3/href">https://medium.com/media/023f2c0225b4dac2805c5fa3555ba2c3/href</a></iframe><p>To check if this works properly, you can print the data in the command prompt with python spreadsheet.py.</p><p>The data for our application is ready. Now let’s make it pretty!</p><h3>Styling the Application</h3><p>Create an index.html file and nest it under the templates folder. Flask generally looks for Html files inside this folder as it makes use of the <a href="https://jinja.palletsprojects.com/en/2.11.x/templates/">Jinja</a> template library to render templates.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/92ea7599223d6a25dc57cadec3d6be1d/href">https://medium.com/media/92ea7599223d6a25dc57cadec3d6be1d/href</a></iframe><p>For CSS, I found <a href="https://bitsofco.de/github-contribution-graph-css-grid/">this</a> fascinating tutorial by Ire Aderinokun to recreate the Graph. It saved me time, but you can implement it however you want.</p><p>I also created a legend, including tooltips and changed the square color to Pink. It’s a playground, be as creative as you want to be!</p><h3>Connect the Data With the Template</h3><p>Create an intermediate file that sends data to the HTML files, I’ll call it app.py. Import spreadsheet.py and assign the value returned by getData() function to a variable. Pass the variable to the render_template() function along with the HTML.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/acada9f03895df17dcacbd1f87647b36/href">https://medium.com/media/acada9f03895df17dcacbd1f87647b36/href</a></iframe><p>The code in the if block automatically reloads the content of the app every time a change is made.</p><p>Now run the app from the command line using the command python app.py (make sure you have activated the virtual environment). If everything went well, your app should be up and running in the localhost <a href="http://127.0.0.1:5000/">http://127.0.0.1:5000</a>/ and I’m sure it looks beautiful!</p><h3>Deploying the Application Using Heroku</h3><p>Using Heroku requires you to have Git installed:</p><ol><li>Download Heroku CLI from <a href="https://devcenter.heroku.com/articles/heroku-cli">here</a> and follow the installation steps.</li><li>Open up your command prompt (Using Git Bash is recommended) and check if Heroku is installed using Heroku --version.</li><li>Create an account for free on the Heroku website <a href="https://signup.heroku.com/">here</a>.</li><li>Once you’ve set up your account, login in using the shell by typingHeroku login -i and enter your credentials.</li><li>We’ll use <a href="https://gunicorn.org/">gunicorn</a> to deploy our application as it is relatively fast and can handle multiple requests at the same time. Install it using the command pip install gunicorn.</li><li>Create a requirements.txt file, a file that informs Heroku about the dependencies for our app, by typing pip freeze &gt; requirements.txt.</li><li>I faced so many errors due to my stupidity. I froze the requirements file without activating the virtual environment. <em>Always </em>make sure that you’re working when the virtual environment is enabled.</li><li>Create a new file in the root directory and type web: gunicorn --bind 0.0.0.0:$PORT app:app. Name it Procfile (Note: Do not use any extensions). A Procfile is something that informs Heroku what commands will be run by the app on startup.</li><li>In app:app the first app refers to the file name (eg: app.py) and the second app refers to app = Flask(__name__). To understand more about the latter app, look <a href="https://teamtreehouse.com/community/can-someone-help-me-understand-flaskname-a-little-better">here</a> (I messed up with this one by not knowing the difference).</li><li>I had an H14 error: no web processes running but I got rid of that by running this command: heroku ps:scale web=1. This informs Heroku that we’re not running any web dynos. If you didn’t get this, great — keep going.</li><li>Now initiate an empty git repository inside your working directory using git init.</li><li>Now add all the files git add . and make a commit git commit -m &quot;Initial Commit&quot;.</li><li>Create a new app on Heroku using heroku create which generates a link for your app and the remote repo URL, then type heroku open which opens the app in your browser.</li><li>Add your remote repo using git remote add heroku &#39;repo-link-generated-on-heroku-create&#39;.</li><li>Now push your changes, git push heroku master</li></ol><p>Your application is now running on the web!</p><p><a href="https://obscure-scrubland-21146.herokuapp.com/">This</a> is the link to mine.</p><p>You can check my final files on my GitHub repo <a href="https://github.com/Padhma/GitHub-Contribution-Graph-Clone">here</a>.</p><p>Thanks for reading all the way down here. Let me know in the comment section if you have any concerns, feedback, or criticism. Have a good day!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=73ce9c096784" width="1" height="1" alt=""><hr><p><a href="https://medium.com/better-programming/recreating-githubs-contribution-graph-using-python-flask-and-google-sheets-73ce9c096784">Recreate GitHub’s Contribution Graph Using Flask and Google Sheets</a> was originally published in <a href="https://betterprogramming.pub/">Better Programming</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>