<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Arvid Eichner on Medium]]></title>
        <description><![CDATA[Stories by Arvid Eichner on Medium]]></description>
        <link>https://medium.com/@arvideichner?source=rss-493985181cd4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*UohMD5yyC3FugqGdTkRXQA.jpeg</url>
            <title>Stories by Arvid Eichner on Medium</title>
            <link>https://medium.com/@arvideichner?source=rss-493985181cd4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 21:54:00 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@arvideichner/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[OpenAI’s ChatGPT Just Revolutionized Data Science]]></title>
            <link>https://python.plainenglish.io/openais-chatgpt-just-revolutionized-data-science-e542385b1f0e?source=rss-493985181cd4------2</link>
            <guid isPermaLink="false">https://medium.com/p/e542385b1f0e</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Arvid Eichner]]></dc:creator>
            <pubDate>Mon, 08 Jan 2024 08:17:28 GMT</pubDate>
            <atom:updated>2024-04-22T14:17:45.196Z</atom:updated>
            <content:encoded><![CDATA[<h4>Here’s What I Learned Testing the Limits of “Natural Language Data Science” Analyzing Movies on Netflix</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IcuSUadibtaBgQ_xE3Korw.png" /></figure><p>Data exploration and breaking down complex datasets used to demand proficiency in Python’s pandas library.</p><blockquote>Never heard of pandas? Start <a href="https://arvideichner.medium.com/list/pandas-acf8bd8f26d8"><strong>here</strong></a>.</blockquote><p>But not everyone’s fluent in coding, and mastering it takes time.</p><p>What if I told you that there’s a way to explore, transform and analyze datasets using nothing but written commands in plain English?</p><h3>A Natural Way to Interact With Data</h3><p><em>Enter </em><a href="https://docs.pandas-ai.com/en/latest/getting-started/"><em>PandasAI</em></a>, a python library by <a href="https://github.com/gventuri">gventuri</a> and collaborators, which promises a paradigm shift in Data Science by combining the power of pandas with OpenAI’s ChatGPT language model.</p><p>PandasAI is designed to be easy to use and — once set up — requires zero experience with generative AI or pandas. The only “skill“ you need is natural language — just formulate a prompt asking for the desired information and PandasAI will translate your request to code and provide the answer to your question.</p><h3>The Promises</h3><p>The developers promise a simplified process that allows users to interact with data naturally, without coding. The proclaimed skillset of PandasAI looks truly impressive:</p><ul><li>Working with pandas DataFrames</li><li>Plotting</li><li>Working with multiple DataFrames (SmartDatalake)</li><li>A chat agent for dynamic, contextual conversations</li><li>and more</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FifJLOFTTo8DjBtwfmq%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2Fv1.Y2lkPTc5MGI3NjExNWUxeGd6Y3psend5YW01a3YyOG80dGppYWFka2dpcXhybXI4aHdldCZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw%2FifJLOFTTo8DjBtwfmq%2Fgiphy.gif&amp;image=https%3A%2F%2Fmedia1.giphy.com%2Fmedia%2Fv1.Y2lkPTc5MGI3NjExbGJndThzanhseTJkamVoZmxqbGo5aGRrdjBhbmh0dzQ5eGw0b3g1ayZlcD12MV9naWZzX2dpZklkJmN0PWc%2FifJLOFTTo8DjBtwfmq%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="246" frameborder="0" scrolling="no"><a href="https://medium.com/media/21d3255458f7788c3d8fdcb40aa7709f/href">https://medium.com/media/21d3255458f7788c3d8fdcb40aa7709f/href</a></iframe><blockquote>Almost sounds too good to be true, doesn’t it?</blockquote><p>If you’re like me, you’re constantly using ChatGPT to review or enhance code, generate and modify complex plots and much more. If PandasAI is as capable as promised, over are the days of switching between your IDE and ChatGPT, copy-pasting codes and commands. You could simply use ChatGPT directly within your Python IDE.</p><p><em>Want to filter data</em>? Simply say, “Show me rows where sales exceed 1000.” and PandasAI filters the data and presents results in an intuitive manner.</p><p>And the best part? Generating plots would be a breeze. No more struggling with Matplotlib or Seaborn. Just ask, “Can you show me a plot of this data?” Whether it’s histograms or scatter plots, PandasAI creates the perfect visual for you.</p><p>Enough day-dreaming, let’s dive in and test whether all these claims are actually true.</p><blockquote>Keen to try it out yourself or follow along? I feature a step-by-step guide on how to install PandasAI, which requires an OpenAI API integration in this <strong>guide</strong>.</blockquote><h3>Getting Started With PandasAI</h3><p>For this demonstration, we’ll be working with the Netflix TV Shows and Movies dataset, sourced from <a href="https://www.kaggle.com/datasets/shivamb/netflix-shows">Kaggle</a>. As the name suggests, the dataset contains information about the title, release year, director, runtime, and rating of shows and movies on Netflix.</p><p>To get started, we’ll import necessary packages and load our dataset from a csv file. Here’s how:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/920/1*PWGLBLNvdQ43q6NYlCGD3g.png" /></figure><p>Next we’re going to instantiate an LLM using our OpenAI API key:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/974/1*BJb2g7-F6LAaUHoc7XYPow.png" /></figure><p><em>The last thing we need to do is to instantiate a new</em> SmartDataFrame. To do so, we simply pass our DataFrame and LLM from before:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1014/1*D_ZDKYVsVE3y3IBXQU8rIA.png" /></figure><h3>Testing the Limits of “Natural Language Data Science”</h3><p>That’s it! Now we can start playing around to find out what’s possible. Let’s start by plotting all imdb_scores of movies on Netflix by release year including confidence intervals:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/1*-EomkFlqCwcfNOahCZjjcA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*N5IU-WiriZZnH3OGKYs-Mg.png" /></figure><p>That seems to have worked. We can immediately observe that the dispersion of imdb_scores increases drastically as the year progress. This is likely caused by the fact that there are much more movies on Netflix which have been released in recent years.</p><p><em>Let’s use ChatGPT to confirm our hunch</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cAmZc2p4JR9O1JbbLZcZgg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*JBZe_gmU-o62CgSwi7-cYw.png" /></figure><blockquote>Our assumption was correct. It’s fair to say that the majority of movies available on Netflix were released after the year 2000.</blockquote><p>One of my good friends is really into movies. He is convinced that newer movies don’t measure up to the classics. He swears by the golden age of film, claiming they made movies differently back then.</p><p>Let’s use this opportunity to find out if more people would agree with him. I will simply plot the average movie score per release year to test his theory:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jV3oYLJWE8GmKg_eXvL6oQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/567/1*zHOljl8OnvpFZzJ5bhDnag.png" /></figure><p>Seems like IMDb scores of shows on Netflix are slowly decreasing over the years… Also, what on earth happened just before 1980? <em>We will explore that later</em> — first, let’s ask ChatGPT whether release_year and imdb_score are correlated:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ym5VuMKaP9Hh4-yRApBbeg.png" /></figure><p><strong>Again, not a problem for PandasAI.</strong></p><p><em>Let’s see if we can slice the data to only contain movies published after the year 2000 and print the correlation between </em><em>release_year and </em><em>imdb_score for this subset, all in a single prompt</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*bvn6-Yser43Tn72doSIaJw.png" /></figure><p>The provided correlation value is different from the last one, that’s a first good sign. <em>Let’s check the correlation ourselves to see if ChatGPT is right</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/1*O3WlTaAyJ53D-3j-h_WhOA.png" /></figure><blockquote>That’s the same result. Impressive.</blockquote><p><em>Now back to the absurdly low rating of movies from just before 1980</em>. Let’s see if we can instruct ChatGPT to identify the exact year and list all movies from that year and their scores:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vuXnVp3zoEyfPNsW-dZAyA.png" /></figure><p>The low point in scores was just before 1980, so 1978 seems to be correct. Let’s list all movies and their scores to find the culprit:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/1*yW1bQ4BP_vwddw40YdNdbA.png" /></figure><p>There it is! It’s just one movie — ‘<a href="https://www.imdb.com/title/tt1219315/?ref_=ttrt_ov">Bandie</a>’ — and apparently, it’s pretty bad. But we’re not here to judge movies…</p><h3>The Limit of Natural Language Data Science</h3><p>Throughout our exploration, PandasAI has effortlessly handled every task we’ve thrown its way. But where is the limit?</p><p>At this point I was certain that, once we move past basic descriptive analysis, we will start to feel the limits of Natural Language Data Science.</p><blockquote>Well, I was wrong…</blockquote><p>Take a look at the following list. These are optional dependencies which can be installed to extend the capabilities of PandasAI far beyond plotting and basic summaries:</p><ul><li>numpy for numerical computing</li><li>ggplot for grammar-based plotting</li><li>seaborn for statistical data visualization</li><li>statsmodels for statistical modeling and testing</li><li>scikit-learn for machine learning toolkit</li></ul><p>As exploring all these dependencies would go beyond the scope of this story, I think it’s fair to say that, today, we won’t encounter the limits of PandasAI.</p><p>Before we end however, let’s at least test a simple linear regression model using the optional statsmodels dependency. First, we need to add the dependency:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*CLHsOHmWfPjjmOVEUTmI8A.png" /></figure><p>Now, PandasAI can perform regression analyses for you —<em> just ask</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JdGhcJFijT33mlhJtUmwDA.png" /></figure><p>And this is the output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WLOa7EMoX8bTu3yJaNDD2w.png" /></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FAjkUQrzvs5OCc%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2Fv1.Y2lkPTc5MGI3NjExcTJwZWtlYm1lMXZjbG9lcDZxdWdzZGNsbmo0OW1hZ2dleGNwM3J2aiZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw%2FAjkUQrzvs5OCc%2Fgiphy-downsized-large.gif&amp;image=https%3A%2F%2Fmedia0.giphy.com%2Fmedia%2Fv1.Y2lkPTc5MGI3NjExN3BrbHJldnRhY3hwaDFhOXl2OXpjYTFzNzRkYThvNThlNHZ6OHh5bSZlcD12MV9naWZzX2dpZklkJmN0PWc%2FAjkUQrzvs5OCc%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="276" frameborder="0" scrolling="no"><a href="https://medium.com/media/77e8be8772fcd90871239d1c5890d5f4/href">https://medium.com/media/77e8be8772fcd90871239d1c5890d5f4/href</a></iframe><blockquote>Truly a taste of the future…</blockquote><p>PandasAI really handled everything we threw at it. Once we get to more advanced models, the results will likely be less spot-on. But while the features we covered in this story were rather basic, many of us would undoubtedly consider it at least a quality of life improvement to perform them using only natural language. And that’s not even considering those of us who can’t code.</p><h3>Why Should I Still Learn Python?</h3><p>Based on the capabilities shown above, it is not hard to imagine a future where we will no longer need programming skills to analyze data sets. We will simply “talk to the data“ in natural language to find out everything we need to know.</p><p>If anything, this will lower the entry barrier into Data Science and thus enable a larger crowd of people to interact with data efficiently.</p><blockquote>Does this mean that we should stop teaching and studying computer science since programming will evidently become obsolete?</blockquote><p><strong>Not at all!</strong></p><p>Under its’ surface, PandasAI is still a python library which simply translates your commands to Python. Therefore, Python is still important and it is now — more than ever — a great idea to learn how to code. Most importantly, Python is fun and easy to learn and there are plenty of great free resources online.</p><h3>Key Takeaways</h3><ol><li><strong>‘Natural Language Data Science’ — </strong>PandasAI enables you to interact with data in Python using only natural language.</li><li><strong>It’s really capable. </strong>It handled everything we threw at it including regression analysis.</li><li><strong>It won’t replace you just yet. </strong>It is still a good idea to learn how to code.</li></ol><h3><a href="https://plainenglish.io/">PlainEnglish.io</a> 🚀</h3><p><em>Thank you for being a part of the In Plain English community! Before you go:</em></p><ul><li><em>Be sure to </em><strong><em>clap</em></strong><em> and </em><strong><em>follow</em></strong><em> the writer</em><strong>️</strong></li><li><em>Learn how you can also </em><a href="https://plainenglish.io/blog/how-to-write-for-in-plain-english"><strong><em>write for In Plain English</em></strong></a><em>️</em></li><li><em>Follow us: </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>X</em></strong></a><strong><em> | </em></strong><a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a><strong><em> | </em></strong><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a><strong><em> | </em></strong><a href="https://discord.gg/in-plain-english-709094664682340443"><strong><em>Discord</em></strong></a><strong><em> | </em></strong><a href="https://newsletter.plainenglish.io/"><strong><em>Newsletter</em></strong></a></li><li><em>Visit our other platforms: </em><a href="https://stackademic.com/"><strong><em>Stackademic</em></strong></a><strong><em> | </em></strong><a href="https://cofeed.app/"><strong><em>CoFeed</em></strong></a><strong><em> | </em></strong><a href="https://venturemagazine.net/"><strong><em>Venture</em></strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e542385b1f0e" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/openais-chatgpt-just-revolutionized-data-science-e542385b1f0e">OpenAI’s ChatGPT Just Revolutionized Data Science</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Step-by-Step Guide to Count Data Analysis in Python]]></title>
            <link>https://python.plainenglish.io/a-step-by-step-guide-to-count-data-analysis-in-python-a981544fc4f0?source=rss-493985181cd4------2</link>
            <guid isPermaLink="false">https://medium.com/p/a981544fc4f0</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[regression]]></category>
            <category><![CDATA[statistics]]></category>
            <dc:creator><![CDATA[Arvid Eichner]]></dc:creator>
            <pubDate>Mon, 27 Nov 2023 18:11:38 GMT</pubDate>
            <atom:updated>2023-11-28T20:32:37.230Z</atom:updated>
            <content:encoded><![CDATA[<h4>Using Generalized Linear Models (GLMs) to Analyze Non-Normal Data</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wW-WD8xjomGi-PN3sqrLYw.png" /></figure><h3>Count Data and its Unique Challenges</h3><p><strong>Count data</strong> represents ocurrences or frequencies of events within a given timeframe. Common examples of count data are:</p><ul><li>The number of traffic accidents at a certain intersection</li><li>The number of visitors accessing a website</li><li>The number of children per couple</li></ul><blockquote>By nature, count data are <strong>discrete </strong>and <strong>non-negative</strong>.</blockquote><p>Due to these unique characteristics, count data rarely follows a normal distribution. This presents unique challenges in data analysis, as common statistical methods — such as linear regression — commonly rely on normally distributed data.</p><p>While you could normalize count data using log- or sqare-root-transformations, choosing adequate models that can handle non-normal data is almost always preferrable as they perform far better than transformations [<a href="https://www.nature.com/articles/npre.2010.4136.1">1</a>].</p><p>In this step-by-step guide, we will take a look at <strong>Generalized Linear Models (GLMs) </strong>and how they can be utilized to analyze count data in Python. I will use the “Insurance in Sweden” sample dataset, which is part of the statsmodels package.</p><h3>Step 1: Exploring the Dataset</h3><p>Let’s start by loading and examining the “Insurance in Sweden” dataset using Python’s statsmodels module:</p><pre>import statsmodels.api as sm<br><br># Load the &quot;Insurance in Sweden&quot; dataset<br>insurance_data = sm.datasets.get_rdataset(&quot;Insurance&quot;, package=&quot;MASS&quot;).data<br><br># Display dataset information and structure<br>print(insurance_data.head())<br>print(insurance_data.info())</pre><p>For the purpose of this guide, we will analyze whether the Age of a person and the District they live in affect the number of insurance Claims they file – a count variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/924/1*fRBMvl9MNEQKm0epMqA3nA.png" /></figure><blockquote>We can see, that both Age and Group are of data type ‘object’, that’s because the two variables are categorical. We will have to keep that in mind for the regression.</blockquote><p>Let’s take a look at the distribution of claims by plotting it using a histogram.</p><pre>import pandas as pd<br><br>histogram = insurance_data[&#39;Claims&#39;].plot.hist()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WPP7eg_DUHh9glziDDd1Uw.jpeg" /></figure><p>As the histogram clearly shows , Claims features no negative values and it’s distribution is skewed. It certainly doesn’t look normally distributed.</p><h3>Step 2: Poisson Regression</h3><p>Generalized Linear Models (GLMs) allow us to model linear regressions for dependent variables of many different non-normal distributions. Analyzing count data specifically involves a systematic approach to model fitting, performance assessment, and handling complexities like overdispersion.</p><p>Let’s start by fittig a GLM based on the poisson distribution. To do so, I use the glm method from the statsmodels api. Ususally statsmodels uses endog and exog to discern predictor variables from response variables. Alternatively, you can fit statistical models using <a href="https://www.statsmodels.org/dev/example_formulas.html">R-style formulas</a>, which I will do here. The main benefit is that categorical variables in our fomula are detected automatically and treated accordingly. Additionally, we don’t have to specify our constant or intercept manually.</p><pre>import statsmodels.api as sm<br>import statsmodels.formula.api as smf<br>import numpy as np<br>import pandas<br><br># defining the poisson glm <br>poisson_glm = smf.glm(formula = &#39;Claims ~ Age + District&#39;, <br>                      data = insurance_data,<br>                      family = sm.families.Poisson())<br><br># fitting the model <br>results = poisson_glm.fit()<br><br># Printing the summary of the regression results<br>print(results.summary())</pre><p>In the above code snippet, I model the effect of Age and District on the Number of Insurance Claims. Let’s take a look at the results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Wd5CrkK0p5V5Qvw31FF-Jw.png" /></figure><p>There are three coefficients for Age, the fourth factor level was omitted and is used as a referece level. Keeping that in mind we can interpret the results as follows. Compared to people aged 20 to 29:</p><ul><li>People aged 30–35 do not file significantly more or fewer insurance claims</li><li>People younger than 25 file significantly fewer insurance claims</li><li>People older than 35 file significantly more insurance claims</li></ul><p>Additionally, the lower the number of the district people live in, the less insurance claims they seem to file.</p><p>Poisson GLMs operate under the assumption that the variance of the response variable is equal to its mean. This assumption is known as <strong>equidispersion</strong>. However, in real-world scenarios, count data often exhibits greater variability than predicted by a Poisson distribution, a phenomenon known as <strong>overdispersion</strong>.</p><p>Therefore, we need to check for equi- / over-dispersion next and take appropriate actions should we discover overdispersion.</p><h3>Step 3: Assessing Equidispersion</h3><pre># Get model deviance residuals<br>deviance_residuals = results.resid_deviance<br><br># Calculate residual deviance<br>residual_deviance = sum(deviance_residuals)<br><br># Calculate degrees of freedom<br>df = results.df_resid<br><br># Calculate the ratio<br>ratio = residual_deviance / df<br><br># Display the ratio<br>print(&quot;Residual Deviance:&quot;, residual_deviance)<br>print(&quot;Degrees of Freedom:&quot;, df)<br>print(&quot;Residual Deviance to Degrees of Freedom Ratio:&quot;, ratio)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UTcq5P6RViZcpBTZ_LlhiQ.png" /></figure><p>In this case, the ratio between the residual deviance and the degrees of freedom (df) is -0.4664, representing underdispersion.</p><p>Technically, we want neither underdispersion nor overdispersion, since the poisson model assumes equidispersion (variance = mean). In reality though, there is much more concern with overdispersion. One reason for that is that underdispersion produces conservative results (lower type 1 errors) and thus it isn’t viewed as prolematic compared to overdispersion.</p><p>There are different ways to address underdispersion, such as quasi-likelihood estimation (quasibinomial family) but we won’t cover those in this story. Rather, we will take a look at how to adress overdispersion next, as overdispersion is far more common and problematic than underdispersion.</p><p>If the Deviance/df ratio was larger than 1, our model would have been overdispersed. To adress overdispersion, we fit a negeative binomial GLM, which is not too different from the poisson GLM.</p><h3>Step 4: Handling Overdispersion with Negative Binomial Regression (Optional)</h3><p>If the poisson GLM is overdispersed, we next fit a negative binomial GLM. To do so, we need the so called alpha paramter, which represents the dispersion, effectively controlling the relationship between the mean and variance.</p><p>Unfortunately, we cannot use our poisson dispersion parameter from before. It can only hint at the presence of overdispersion when significantly greater than 1, but it cannot be directly used as an input for the alpha parameter in a negative binomial regression. Instead, we need to estimate alpha using methods specific to negative binomial models, such as the <a href="https://books.google.de/books?hl=de&amp;lr=&amp;id=qVEwBQAAQBAJ&amp;oi=fnd&amp;pg=PR15&amp;dq=REGRESSION+ANALYSIS+OF+COUNT+DATA+Second+Edition&amp;ots=ROcign98_h&amp;sig=JEPcn2UysZ6B4kS8nvLo1kso2UM&amp;redir_esc=y#v=onepage&amp;q=REGRESSION%20ANALYSIS%20OF%20COUNT%20DATA%20Second%20Edition&amp;f=false"><em>Cameron-Trivedi decomposition</em></a><em>, </em>which in includes the use of an auxiliary OLS regression.</p><blockquote>Here’s how you do it:</blockquote><pre>&#39;&#39;&#39;<br>First, we need a vector of fitted rates, called λ.<br>We can get the vector from the poisson model results using &#39;.mu&#39;<br>Next, we add the λ vector as a new column called &#39;LAMBDA&#39; to our DataFrame<br>&#39;&#39;&#39;<br><br>insurance_data[&#39;LAMBDA&#39;] = results.mu<br><br>&#39;&#39;&#39;<br>Then we add a new column to our dataframe, <br>which is derived from the λ vector.<br>It will serve as the dependent variable for our auxiliary OLS regression<br>&#39;&#39;&#39;<br><br>insurance_data[&#39;AUX_OLS&#39;] = insurance_data.apply(lambda x: ((x[&#39;Claims&#39;] - x[&#39;LAMBDA&#39;])**2 - x[&#39;LAMBDA&#39;]) / x[&#39;LAMBDA&#39;], axis=1)<br><br># Specify the aux. OLS model <br>ols_expr = &quot;&quot;&quot;AUX_OLS ~ LAMBDA - 1&quot;&quot;&quot;<br><br># Fit the aux. OLS model based on above <br># expression and our dataframe insurance_data<br>aux_olsr_results = smf.ols(ols_expr, insurance_data).fit()<br><br># Print regression parameters<br>print(aux_olsr_results.params)</pre><p>This provides us with our alpha value, which we need for our negative binomial model:</p><p>&gt;&gt; LAMBDA 0.313148</p><blockquote>By default, if we don’t specify our alpha value, the model will assume an alpha of 1. In this case, the correct alpha value is 0.31.</blockquote><p>Before we move on, we need to quickly test whether our alpha value is statistically significant:</p><pre># print the value of the test statistic <br>print(aux_olsr_results.tvalues)</pre><p>In our case, the output is:</p><p>&gt;&gt; LAMBDA 7.732053</p><p>Using a <a href="https://goodcalculators.com/student-t-value-calculator/">t-value calculator</a> based on the degrees of freedom:</p><p>64 observations — 1 alpha = 63</p><p>We now caluclate our t-Value (right-tailed):</p><p>&gt;&gt; 1.669402</p><blockquote>This value is smaller than the t-statistic of alpha, which was 7.732053. Therefore, we can conclude that our alpha value of 0.313148 is statistically significant.</blockquote><p>Now, we can finally move on to fit the negative binomial regression. We use aux_olsr_results.params[0] to access our alpha value and define it in the family parameter of the negative binomial model. The rest is identical to our procedure for fitting the poisson GLM.</p><pre># Defining the negative binomial GLM<br>neg_binom_glm = smf.glm(formula=&#39;Claims ~ Age + District&#39;,<br>                        data=insurance_data,<br>                        family=sm.families.NegativeBinomial(alpha=aux_olsr_results.params[0]))<br><br># Fitting the model<br>results_neg_binom = neg_binom_glm.fit()<br><br># Printing the summary of the regression results for negative binomial<br>print(results_neg_binom.summary())</pre><p>Printing the results we get:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*7woVbP069Y2cugJJCmXLKg.png" /></figure><p>Since in our case the poisson GLM was underdispersed, rather than overdispersed, this model and its results are not really sensible. The whole process just served to demonstrate negative binomial regression using a GLM in python.</p><h3>Key-Takeaways:</h3><ol><li>Count data are <strong>discrete occurrences,</strong> characterized by non-negative integers.</li><li>Use <strong>poisson regression first</strong>, assuming equidispersion</li><li>Always <strong>test the dispersion</strong> of your poisson model</li><li>In the case of <strong>overdispersion, try a negative binomial </strong>model.</li></ol><p>It’s important for me to note that, in this story, we just scratched the surface of GLMs and count data analysis. My focus was to present you with an easy to follow step-by-step guide with Python code for you to use. You should definitely read up on count data regression to fully understand all the details I didn’t cover in this guide.</p><h3><a href="https://plainenglish.io/">PlainEnglish.io</a> 🚀</h3><p><em>Thank you for being a part of the In Plain English community! Before you go:</em></p><ul><li><em>Be sure to </em><strong><em>clap</em></strong><em> and </em><strong><em>follow</em></strong><em> the writer</em><strong>️</strong></li><li><em>Learn how you can also </em><a href="https://plainenglish.io/blog/how-to-write-for-in-plain-english"><strong><em>write for In Plain English</em></strong></a><em>️</em></li><li><em>Follow us: </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>X</em></strong></a><strong><em> | </em></strong><a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a><strong><em> | </em></strong><a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a><strong><em> | </em></strong><a href="https://discord.gg/in-plain-english-709094664682340443"><strong><em>Discord</em></strong></a><strong><em> | </em></strong><a href="https://newsletter.plainenglish.io/"><strong><em>Newsletter</em></strong></a></li><li><em>Visit our other platforms: </em><a href="https://stackademic.com/"><strong><em>Stackademic</em></strong></a><strong><em> | </em></strong><a href="https://cofeed.app/"><strong><em>CoFeed</em></strong></a><strong><em> | </em></strong><a href="https://venturemagazine.net/"><strong><em>Venture</em></strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a981544fc4f0" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/a-step-by-step-guide-to-count-data-analysis-in-python-a981544fc4f0">A Step-by-Step Guide to Count Data Analysis in Python</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why You Should Be Nice to ChatGPT]]></title>
            <link>https://medium.com/dare-to-be-better/why-you-should-be-nice-to-chatgpt-b56491f4dad5?source=rss-493985181cd4------2</link>
            <guid isPermaLink="false">https://medium.com/p/b56491f4dad5</guid>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[language-model]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <dc:creator><![CDATA[Arvid Eichner]]></dc:creator>
            <pubDate>Thu, 09 Nov 2023 12:35:29 GMT</pubDate>
            <atom:updated>2024-05-02T09:57:38.524Z</atom:updated>
            <content:encoded><![CDATA[<h4>How treating AI language models well can improve their results</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*Pnxwjno3ZmWtQWIaIElonQ.jpeg" /><figcaption>Source: u/guywithasubwife on reddit</figcaption></figure><h3><strong>How do you talk to chatGPT?</strong></h3><p>A friend recently told me:</p><blockquote>I say ‘thank you’ and ‘please’ when talking to chatGPT, just in case…</blockquote><h3><strong>Just in case?</strong></h3><p>Well, of course he was joking. But what if I told you that <strong>even if you do not believe in a distopian future ruled by AI overlords,</strong> there’s actually a valid reason to treat chatGPT, or any other language model for that matter, like you would treat a real person. Hear me out:</p><p>In this story, I will tell you:</p><ol><li><strong>Why it matters <em>how </em>you treat AI language models</strong> like ChatGPT and</li><li><strong>How you can significantly improve their results</strong>.</li></ol><p>I include a list of custom instructions that you can start using immediately at the end of this story.</p><h3>The Human-AI Relationship</h3><p>Have you ever thought about treating an AI model with respect, much like you would a fellow human being? It might sound peculiar, but it’s a concept worth considering. ChatGPT, despite being a machine, has been trained on vast amounts of data, much like a knowledgeable colleague in a particular field. Just as you would appreciate a colleague’s expertise, showing kindness to ChatGPT can lead to outcomes that are not only more beneficial but also surprisingly profound.</p><h3>The “Take a Deep Breath” Phenomenon</h3><p>Let me take you through the astonishing findings from a recent research <a href="https://arxiv.org/abs/2309.03409">paper</a> that serves as the backdrop to this story. Researchers stumbled upon a peculiar discovery: beginning a prompt with the soothing phrase “take a deep breath” had a profound impact on ChatGPT’s mathematical responses. It was as if the AI model responded with greater thoughtfulness, producing answers that were not only more accurate but also clearer and more insightful.</p><p>Different instructions that also improved responses were:</p><ul><li>“Let’s think carefully about the problem and solve it together.”</li><li>“Let’s break it down!”</li><li>“Let’s do the math!”</li></ul><p>But by far the most effective prompt was to tell chatGPT to:</p><blockquote>“Take a deep breath and work on this problem step-by-step.”</blockquote><p>These rather strange insights ignited a curiosity in me, leading me to explore the depths of <strong>prompt engineering</strong>.</p><h3>The Importance of Prompt Engineering</h3><p>Prompt engineering, as it turns out, is akin to crafting an artful query or instruction to extract the desired responses from AI models. The phrasing of prompts is where the magic happens, as it can remarkably affect the quality of AI-generated content.</p><p>These are the most important insights:</p><ol><li><strong>Context Matters</strong>: Picture this: you’re conversing with ChatGPT, asking a complex question. Start by providing some context. Like a conversation with a friend, give ChatGPT the background information it needs to better understand your query.</li><li><strong>Clarity is Key</strong>: To make things smoother, use clear and straightforward language in your prompts. Avoid vagueness to ensure ChatGPT fully comprehends your request.</li><li><strong>Specify the Output Format</strong>: If you have a specific format in mind for the AI-generated content, let ChatGPT know explicitly. Whether it’s a list, a summary, or a detailed answer, your specifications are essential.</li><li><strong>Experiment and Iterate</strong>: Think of it as a creative process. Don’t be afraid to experiment with different prompt formulations. Refining your prompts can lead to improved results over time.</li></ol><p>Often, it is a good idea to tell chatGPT to take the role of the person or institution that is most knowledgable in the output you’re trying to produce.</p><p>For example, let’s say you’re trying to score a job at Google and want to prepare for the job interview.</p><p>You could ask:</p><blockquote>I want to apply for a software developer positon at Google. How can I prepare for the interview?</blockquote><p>ChatGPT will provide you with a list of things you should keep in mind to be prepared for the interview. Lets look at the results:</p><blockquote>1. Build a Strong Foundation</blockquote><blockquote>2. Review Programming Languages</blockquote><blockquote>3. Practice Coding</blockquote><blockquote>4. Study System Design</blockquote><blockquote>5. Brush Up on Databases and Networking</blockquote><blockquote>6. Learn Google Technologies</blockquote><blockquote>[…]</blockquote><p>Of course, these are all valid points, but they are not very specific and chances are that you are already aware of them.</p><p>Now, let’s see how you can use chatGPT more effectively by utilizing effective <strong>prompt engineering. </strong>Try telling chatGPT to take the role of the person doing the interview. This way, you can train for interviews effectively by simulating the actual job interview:</p><blockquote>Please take the role of an interviewer at Google.</blockquote><blockquote>I am the candidate applying for a enter the desired postion here positon.</blockquote><blockquote>Only reply as the interviewer and don’t mention, that you are a language model. Don’t provide all statements at once but talk to me step-by-step and wait for my responses. Ask me questions that are part of the interviewing process at Google.</blockquote><h3>How to Write Better Prompts</h3><p>Of course I’m not telling you to always be nice when talking to language models, thats up to you. But, although we don’t really know why the human-like way of giving instructions works, we can still make use of it. Here’s how:</p><p><strong>When starting a new chat, always start by giving instructions to chatGPT.</strong></p><blockquote>Note: Each time you start a new chat with chatGPT it starts with a blank sheet. But as you chat back and forth, it will remember your promts and it’s own responses. So you can reference prior information from the same chat or start off a new chat by “setting the tone” to improve following results, more on that later.</blockquote><p>This twitter user provides a useful list of custom instructions you can simply copy and paste to your chat:</p><h3>Nivi on Twitter: &quot;I&#39;ve completely rewritten and expanded my GPT Custom Instructions:- Be highly organized- Suggest solutions that I didn&#39;t think about-be proactive and anticipate my needs- Treat me as an expert in all subject matter- Mistakes erode my trust, so be accurate and thorough-... / Twitter&quot;</h3><p>I&#39;ve completely rewritten and expanded my GPT Custom Instructions:- Be highly organized- Suggest solutions that I didn&#39;t think about-be proactive and anticipate my needs- Treat me as an expert in all subject matter- Mistakes erode my trust, so be accurate and thorough-...</p><blockquote>Did you know that you can save ‘custom instructions’ in your account settings? This way you can add instructions that should always be followed instead of pasting them each time you start a new conversation.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*d837aGEDkv1qKuvO18q2OQ.png" /><figcaption>Custom instructions are prompts that will always be followed by chatGPT</figcaption></figure><h3>Key Takeaway</h3><p>The next time you engage with ChatGPT, think back to this story, and remember that thoughtful phrasing can produce better results. It’s not just about being nice to AI; it’s about understanding that it matters how you phrase your promts!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b56491f4dad5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dare-to-be-better/why-you-should-be-nice-to-chatgpt-b56491f4dad5">Why You Should Be Nice to ChatGPT</a> was originally published in <a href="https://medium.com/dare-to-be-better">Dare To Be Better</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Step-by-Step Guide to Installing MySQL and MySQL Workbench on Windows]]></title>
            <link>https://arvideichner.medium.com/a-step-by-step-guide-to-installing-mysql-and-mysql-workbench-on-windows-c4f2e9e37fb8?source=rss-493985181cd4------2</link>
            <guid isPermaLink="false">https://medium.com/p/c4f2e9e37fb8</guid>
            <category><![CDATA[guides-and-tutorials]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[mysql]]></category>
            <category><![CDATA[installation]]></category>
            <category><![CDATA[sql]]></category>
            <dc:creator><![CDATA[Arvid Eichner]]></dc:creator>
            <pubDate>Mon, 23 Oct 2023 14:47:26 GMT</pubDate>
            <atom:updated>2024-04-22T14:19:23.018Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_DylO3_P0ezJMLbh9fmqAA.png" /></figure><p>MySQL is one of the most popular open-source relational database management systems, and MySQL Workbench is a powerful graphical tool for managing and interacting with MySQL databases. If you’re looking to set up MySQL and MySQL Workbench on your Windows machine, this step-by-step guide will walk you through the installation process.</p><blockquote>Before we begin, make sure you have the following:</blockquote><ul><li>A Windows computer (Windows 7 or newer)</li><li><a href="https://support.microsoft.com/en-us/windows/how-do-i-log-on-as-an-administrator-63267a09-9926-991a-1c77-d203160c8563">Administrative</a> access to your computer.</li></ul><h3><strong>Step 1: Download MySQL Installer</strong></h3><ul><li>Open your web browser and navigate to the MySQL website at <a href="https://dev.mysql.com/downloads/installer/">https://dev.mysql.com/downloads/installer/</a>.</li><li>Scroll down to the “MySQL Installer” section, make sure “Microsoft Windows“ is selected under Select Operating System and click on the “Download” button for the MySQL Installer MSI (both the 2.4M and 331.3M version will work)</li></ul><blockquote>Note: This guide uses screenshots that might be hard to read on mobile.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/987/1*8s60EXuvD97DegGB2wQMqg.png" /></figure><p>On the next page, you’ll be prompted to log in or create a MySQL account. You can choose “No thanks, just start my download.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/1*YKBPwKjwWvhTw194sKjRAw.png" /></figure><h3><strong>Step 2: Install and Configure MySQL</strong></h3><ul><li>Locate the installer file you just downloaded (e.g., mysql-installer-web-community-x.x.xx.x.msi) and double-click it to run the installer.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/548/1*ZvZACf5Fq7cUiGdqiWt0Sw.png" /></figure><ul><li>In the installer window, choose “Full” under “Setup Type.” This option will install MySQL Server, MySQL Workbench and other components, such as documentation and examples. Then click “Next.”</li></ul><blockquote>You can also choose “Custom” and only install MySQL Server and MySQL Workbench.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*uhDJWWWcadHHANdyGu5iVA.jpeg" /></figure><ul><li>In the next window, just click “Execute.”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*UUY6QdsMW_dFqbVkw4wMdw.jpeg" /></figure><ul><li>After all products have been downloaded sucessfully, as indicated by the green checkmark, click “Next.”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*025227pimMTvMxX05IttLw.jpeg" /></figure><ul><li>In the next window, just click “Execute.”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*b7BbJst9pPsiBy6eNElNhw.jpeg" /></figure><ul><li>After all products have been installed sucessfully, click “Next” to start the configuration.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*kzMzjWeAxCuJSs7Ixg8IsQ.jpeg" /></figure><ul><li>Under “Type and Networking” choose “Development Computer” as Config Type and leave everything else as is. Then click “Next.”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*Lyu5VUvJrSSzSESxWq7Z-g.jpeg" /></figure><ul><li>Next, you will be asked to choose an authentication method. For increased compatibility, choose “Use Legacy Authentication Method” and click “Next.” Choosing “Use Strong Password Encryption” will also work.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*I_GlvmCLXCu4nsy3fTOT1Q.jpeg" /></figure><ul><li>In the “Account and Roles” window, choose a new password for your root account. Be sure to choose a strong <a href="https://1password.com/password-generator/">password</a> and enter the password under MySQL Root Password and Repeat Password then click “Next.”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*Lp4X3Lz1ZpuuXVhZEBdixg.jpeg" /></figure><ul><li>In the next window, leave everything as is and click “Next.”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*wXUHzGfSZD0PPM11LIzcNg.jpeg" /></figure><ul><li>Next, grant full access to let the Installer update server file permissions and click “Next.” Click “Execute” to apply the configuration changes.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*_py1vjs7hnUfuSofOA0oaw.jpeg" /></figure><ul><li>Once the configuration is done (green checkmarks), click “Finish.”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*a_fCwqcKLl4nLafJKi3i4Q.jpeg" /></figure><p>That’s the first step done. You just downloaded and installed MySQL Server and MySQL Workbench. In the next step, we will configure MySQL Router and “Samples and Examples”. Both Services are part of the full Installation. If you chose a custom installation earlier, you can skip ahead to Step 3.</p><ul><li>Click “Next” to start the configuration.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*ikNIduvhMp08wlD2FsQJUg.jpeg" /></figure><ul><li>Under “Connect To Server” enter the password you chose earlier under Password and click “Check.”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*oMMnRTfrpFPImpyuZG10_w.jpeg" /></figure><p>If the password was correct, a green checkmark will appear and the status of the connection will change to “Connection succeeded.”</p><ul><li>Click “Next.”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*-finjHjoIjh7lBAQVX8hRw.jpeg" /></figure><ul><li>Click “Execute” to apply the configurations.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*Cnx5GDspWsmJCVDT6XwmpQ.jpeg" /></figure><ul><li>Once the configuration is completed (green checkmarks), click “Finish.”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*rdJ5YBb8XWr7W-7MyDOTNw.jpeg" /></figure><p>That’s it! The installation is complete. You can now choose whether MySQL Workbench and Shell should be launched — we only need MySQL Workbench right now.</p><ul><li>Click “Finish” and MySQL Workbench will launch.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/982/1*z8meYHTStyyu8PhMPEa4kg.jpeg" /></figure><h3>Step 3: Using MySQL Workbench</h3><p>At the end of the installation process, MySQL Workbench should launch automtically. If it doesn’t, start the programm manually. You are greeted by the graphical user interface (GUI) tool for MySQL, the MySQL Workbench.</p><p>In the bottom half of the window under “MySQL Connections” you should see one tile for your local instance connection, Local instance MySQL80.</p><ul><li>Go ahead and double-click it.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7tNCNuvsFnb1zwhW7oj_Sg.jpeg" /></figure><ul><li>In the new window, enter the password you chose earlier and click “OK.”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P5WlElyZL6syUmkuHnNWNw.jpeg" /></figure><p>Aaaand we’re in. You are now connected to your MySQL Server and can start using the MySQL Workbench.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KGBYAhdCup3lweOBZI23SQ.jpeg" /></figure><blockquote>Let’s quickly look at the most important components of the user interface:</blockquote><h4>Navigator</h4><p>The Navigator is typically located on the left-hand side of the MySQL Workbench GUI. It serves as a central hub for database management and navigation. It houses some administrative features and a handy tree-like structure of all your databases under the tab “Schemas.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5IdvaLVsMSGWYftVc2P-3Q.jpeg" /></figure><h4>Query</h4><p>The “Query” tab, located in the middle of the workspace area, is where you can write and execute SQL queries. This is where you interact with your MySQL databases using SQL commands.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xE39GGGMHG8k7WCQCZn2FQ.jpeg" /></figure><h4>Output</h4><p>The “Output” tab, usually positioned at the bottom of the MySQL Workbench interface, displays various types of information and feedback. This includes feedback on the (successful) execution of your SQL queries, any error messages, and status updates regarding your MySQL server.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MgjB_MJHuMKFclNPcC7yMQ.jpeg" /></figure><h3>Step 4: Testing the Connection</h3><p>To test your connection, use the following steps:</p><ul><li>In MySQL Workbench, open a new query tab by clicking the “+ SQL” icon in the top left corner or selecting “File” &gt; “New Query Tab.”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LeV9OWDZhW34pHCUHDzLXg.jpeg" /></figure><p>You can use a basic SQL query to test the connection. For example, you can run a query to display the current date and time:</p><pre>SELECT NOW();</pre><p>This query will retrieve the current date and time from the MySQL server. If you receive a result with the current date and time, it confirms that your connection to the MySQL server is working.</p><ul><li>Enter SELECT NOW() into the new query tab.</li><li>Click the “Execute” button (lightning bolt symbol) or press Ctrl+Enter to execute the query.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*46HzqvwCxCrx7ABPr91KIA.png" /></figure><p>If the query executes successfully and you see the current date and time in the result pane, it means that your connection to the MySQL server is functioning correctly. If you get an error or don’t see any results, it could indicate a connection issue. In that case, double-check your connection settings, including the hostname, port, username, and password, and ensure your MySQL server is running.</p><p>That’s it. With MySQL Server and MySQL Workbench installed and configured, you are now ready to import or create your databases and query them.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c4f2e9e37fb8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Identifying and Handling Outliers in Pandas: A Step-By-Step Guide]]></title>
            <link>https://python.plainenglish.io/identifying-and-handling-outliers-in-pandas-a-step-by-step-guide-fcecd5c6cd3b?source=rss-493985181cd4------2</link>
            <guid isPermaLink="false">https://medium.com/p/fcecd5c6cd3b</guid>
            <category><![CDATA[outliers]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Arvid Eichner]]></dc:creator>
            <pubDate>Thu, 19 Oct 2023 06:16:33 GMT</pubDate>
            <atom:updated>2024-04-22T14:18:06.190Z</atom:updated>
            <content:encoded><![CDATA[<h4>Learn How To Handle Outliers Like a Pro</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aDmNbnJG5bgDcEoi7uyCyw.png" /></figure><p>Outliers are exceptional data points within your dataset, caused by chance, anomalies, or even measurement errors. Effectively identifying and handling outliers is a complex yet critical process because ignoring them can lead to biased results. <a href="https://medium.com/@arvideichner/why-you-should-start-using-pandas-for-data-preparation-today-6111e13a6195">Pandas</a>, the versatile data manipulation library in Python, provides a set of tools for efficiently handling outliers. In this step-by-step guide, we will explore what outliers are, how to detect them, what actions to take when handling them, and how to leverage pandas along the way. We will look at boxplots, z_scores and the interquartile range (IQR) method.</p><h3>Before We Start</h3><p>You should <strong>NEVER </strong>remove outliers without questioning their origin. Not all outliers are created equal and for some types of outliers there are strong arguments against removing them. If you are not certain that your outliers can be removed, or for a more introductory guide on outliers using real-world examples, check out this story first:</p><p><a href="https://arvideichner.medium.com/don-t-just-remove-outliers-from-your-data-think-twice-6508b37920e3">Don‘t Just Remove Outliers From Your Data – Think Twice</a></p><p>Okay, with that out of the way, lets see how we can identify and get rid of outliers in your data using the Python library pandas.</p><h3>Identifying Outliers Using Pandas</h3><p>The first step in handling outliers is to identify their presence within your dataset. Outliers can be identified visually or by using metrics such as z-scores or the interquartile range (IQR) method. First, let’s quickly create some sample data.</p><pre>&#39;&#39;&#39;<br>I use the numpy library to generate normally distributed data, <br>add 5 outliers, then combine everything in a new DataFrame:<br>&#39;&#39;&#39;<br><br># We need pandas and numpy so let&#39;s import both libraries<br>import pandas as pd<br>import numpy as np<br><br># Next, we set a random seed for reproducibility<br>np.random.seed(42)<br><br># Generate random data following a normal distribution <br># [Mean = 50, Standard Deviation = 10]<br>data = np.random.normal(loc=50, scale=10, size=1000)<br>  <br># Introduce 5 outliers<br>outliers = np.random.normal(loc=200, scale=50, size=5)  <br>data = np.concatenate([data, outliers])<br><br># Shuffle the data to randomize the order<br>np.random.shuffle(data)<br><br># Create final DataFrame containing outliers<br>outlier_df = pd.DataFrame({&#39;values&#39;: data})</pre><blockquote>Since <em>np.random.normal</em> creates random data, I first set a seed using <em>np.random.seed(42)</em>. By setting the same seed (42) in your code, you can recreate the same exact data points, even though they are random.</blockquote><p>In this code snippet, we create a DataFrame ’outlier_df’ with one variable called ‘values’which contains 1000 normally distributed data points with mean (M) = 50 and standard deviation (SD) = 10. &#39;values&#39; also contains 5 additional data points that follow a normal distribution with M = 200 and SD = 50; i.e., potential outliers. Let’s try to find them:</p><h4>1) Identifying Outliers: Visual Inspection</h4><p>The easiest method to spot outliers is to visually inspect the data (or plots thereof) to spot values that seem odd or don’t fit in with the rest of the data.</p><pre># create boxplot <br>outlier_df.boxplot()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/557/1*hfmTqSxSiw0JWI-2CIFeiw.png" /></figure><p>We can see that, as expected, most values lie between 0 and 100 and the median is around 50. This makes sense given that we created data that is normally distributed around a mean of 50 with a standard deviation of 10.</p><p>We can also see some values — 5 to be exact — that are above 150. They appear far from all other values, like outcasts struggling to fit in. <strong>We just identified the outliers</strong>!</p><blockquote>In a boxplot, the data is divided into quartiles, with the central box representing the interquartile range (IQR) that spans the 25th to 75th percentiles. The line inside the box marks the median (50th percentile), and the “whiskers” (T-shaped lines) represent a limit of 1.5 * IQR.</blockquote><p>We can also use different plots, such as histograms. Histograms are great to get a general feel for the distribution of a variable. It is often a good idea to start your data exploration by looking at a histogram. As you will see, they are not as powerful for spotting outliers though, especially compared to boxplots.</p><pre># create histogram<br>outlier_df.hist()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/580/1*KmG-FLqd5x9e6edTSkKnXA.png" /></figure><p>Again, we can see that our data seems to be normally distributed around 50 (although it appears a little skewed). Looking at this histogram there don’t seem to be any outliers though, what’s up with that?</p><blockquote>In this histogram, we can’t spot the 5 outliers since they get lost among the 1000 “normal” data points.</blockquote><p>That’s the tricky thing about outliers. They don’t show up in large numbers (or else they wouldn’t be outliers). But even in low numbers they can disproportionately affect your analysis due to their exceptionally high or low values.</p><p>We can combat this by increasing the number of ‘bins’ for the histogram. The hist() function groups values into bins and then plots the bins. We can change the number of bins to use by changing the value for the bins parameter when calling the function:</p><pre># creating another histogram with more bins <br>outlier_df.plot.hist(bins=100)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/579/1*xk87OOgHws9p37ndRum6qQ.png" /></figure><p>Increasing the number of bins made the histogram more granular. Now it looks more like a normal distribution and, more importantly, we can see the outliers. You might have to squint a bit but at the bottom right corner of the plot we can see our 5 outliers as tiny blue bars.</p><blockquote>Using a higher bin count, we can visually identify outliers using a histogram. But as you notice, boxplots are far superior for visual outlier detection.</blockquote><h4>2) Identifying Outliers: z-Score</h4><p>Apart from visual identification, we can also calculate metrics to determine what represents “normal” values in our data. We can then look for data points that lie outside of this normal range of data to find our outliers.</p><p>For instance, we can calculate the so-called z-score for each data point. It measures the distance of a data point from the mean in standard deviations. High absolute z-scores can be considered outliers (usually z-scores greater than 2 or 3).</p><pre># Calculate the z-scores<br>z_scores = np.abs((outlier_df - outlier_df.mean()) / outlier_df.std())<br><br># add z_scores as a new column to our DataFrame<br>outlier_df[&#39;z_scores&#39;] = z_scores</pre><p>We obtain the z_scores by calculating the distance from the mean outlier_df — outlier_df.mean() and dividing it by the standard deviation outlier_df.std() for each data point. We use np.abs to get absolute values. This is necessary, since we want to identify potential outliers both higher <strong>and lower</strong> than the rest of the data. Finally, I added the z_scores to the DataFrame.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/242/1*gh4Zjl_dsi51y8wB6zIJYQ.png" /></figure><p>Each value in our DataFrame now has its own z_score, representing how far off it is from the rest of the data.</p><blockquote>For example, we can see that the z_score 0.246940 of value 54.979983 in row 3 is higher than the z_score 0.127266 of value 49.039401 in row 0. That’s because the mean of &#39;values&#39; is 51.0598 and 54 is further away from 51 than 49 is.</blockquote><pre># quickly checking the mean of &#39;values&#39;<br>outlier_df[&#39;values&#39;].mean()<br><br>&gt;&gt; 51.05976702436777</pre><h4>3) Identifying Outliers: IQR Method</h4><p>The IQR is a statistical measure that provides insight into the spread of data within the interquartile range, which covers the middle 50% of the data. <strong>Calculate the IQR</strong>: Start by computing the IQR, which is the difference between the 75th percentile (Q3) and the 25th percentile (Q1).</p><pre># Calculate 25% percentile and 75% percentile<br>Q1 = outlier_df[&#39;values&#39;].quantile(0.25)<br>Q3 = outlier_df[&#39;values&#39;].quantile(0.75)<br><br># Calculate Interquartile Range (IQR)<br>IQR = Q3 - Q1</pre><p>Next, we define <strong>upper </strong>and <strong>lower bounds</strong>. These are the bounds that define the range of “acceptable” or “normal” values.</p><blockquote>Usually, the lower bound is defined as Q1 - 1.5 * IQR and the upper bound as Q3 + 1.5 * IQR. You can change out the 1.5 if you want your outlier detection to be more sensitive or less sensitive.</blockquote><pre># Define lower and upper bounds<br>lower_bound = Q1 – 1.5 * IQR<br>upper_bound = Q3 + 1.5 * IQR</pre><p>Now that we have the upper and lower bounds, we can identify our outliers: Data points falling below the lower bound or above the upper bound are considered potential outliers.</p><h3>Removing Outliers Using Pandas</h3><p>Now, the only thing left to do is to remove our outliers. Remember that you should always think twice about removing outliers. <a href="https://arvideichner.medium.com/don-t-just-remove-outliers-from-your-data-think-twice-6508b37920e3"><strong>Never remove remove data points </strong>simply because they are outliers</a>.</p><p>If you are certain, that your outliers can be removed, for instance because they are created by measurement errors, you can go ahead and remove them from your data set.</p><h4>1) Removing Outliers Identified by Boxplot</h4><p>First, we used a boxplot to find our outliers. The plot revealed that all but 5 data points are within a range from 0 to 100. All five outliers had values well above 150. Therefore, we can easily remove the outliers like this:</p><pre># Remove outliers (values larger than 150)<br>cleaned_df = outlier_df[outlier_df[&#39;values&#39;] &lt;= 150]</pre><p>To remove the outliers, we create a new DataFrame df_cleaned which only contains values from outlier_df that are smaller or equal to 150. We can confirm, whether the outlier removal was sucessfull by plotting another boxplot:</p><pre># select values column from cleaned_df<br>values = cleaned_df[[&#39;values&#39;]]<br><br># create new boxplot for values<br>values.boxplot()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/1*J2gYTQSAaQxATydGKYY9Uw.png" /><figcaption>No more outliers, success!</figcaption></figure><h4>2) Removing Outliers Identified by z-Score and IQR</h4><p>Removing outliers identified by z-score or IQR is equally straight-forward. To remove z-score outliers, remove all values that feature z-scores above a predefined threshold (usually 2 or 3).</p><pre># Define threshold (here: remove values with z-score &gt; 2)<br>threshold = 2<br><br># Remove outliers based on z-score and threshold<br>cleaned_df = outlier_df[outlier_df[&#39;z_scores&#39;] &lt;= threshold]</pre><p>Similarly for IQR:</p><pre>&#39;&#39;&#39;<br>We already defined lower and upper bounds above <br>So now we only need to remove values that are: <br><br>  1) lower that the lower bound<br>  2) higher than the upper bound<br><br>&#39;&#39;&#39;<br><br># Remove outliers based on lower &amp; upper bounds<br>cleaned_df = outlier_df[(outlier_df[&#39;Values&#39;] &gt;= lower_bound) &amp; <br>                        (outlier_df[&#39;Values&#39;] &lt;= upper_bound)]</pre><p>Handling outliers is a crucial step in data analysis and is essential for producing accurate and meaningful results. With the power of Pandas and statistical methods, you can effectively identify and manage outliers, ensuring that your analyses are based on reliable data. By incorporating these techniques into your data analysis workflow, you’ll be better equipped to derive valuable insights from your datasets.</p><h3>Related Stories</h3><ul><li><a href="https://python.plainenglish.io/mastering-data-aggregation-with-pandas-groupby-an-easy-to-understand-guide-0622a542ccdd">Mastering Data Aggregation with Pandas GroupBy: An Easy-to-Understand Guide</a></li><li><a href="https://arvideichner.medium.com/why-you-should-start-using-pandas-for-data-preparation-today-6111e13a6195">Why You Should Start Using Pandas for Data Preparation Today</a></li></ul><h3>In Plain English</h3><p><em>Thank you for being a part of our community! Before you go:</em></p><ul><li><em>Be sure to </em><strong><em>clap</em></strong><em> and </em><strong><em>follow</em></strong><em> the writer! 👏</em></li><li><em>You can find even more content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><strong><em> 🚀</em></strong></li><li><em>Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. 🗞️</em></li><li><em>Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a><strong><em>(X</em></strong>), <a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a>, <a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a>, and <a href="https://discord.gg/in-plain-english-709094664682340443"><strong><em>Discord</em></strong></a><strong><em>.</em></strong></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fcecd5c6cd3b" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/identifying-and-handling-outliers-in-pandas-a-step-by-step-guide-fcecd5c6cd3b">Identifying and Handling Outliers in Pandas: A Step-By-Step Guide</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Become a Data Preparation Expert using ‘dplyr’ in R]]></title>
            <link>https://arvideichner.medium.com/become-a-data-preparation-expert-using-dplyr-in-r-103fbbc53c9b?source=rss-493985181cd4------2</link>
            <guid isPermaLink="false">https://medium.com/p/103fbbc53c9b</guid>
            <category><![CDATA[dplyr]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[r]]></category>
            <category><![CDATA[data-preprocessing]]></category>
            <dc:creator><![CDATA[Arvid Eichner]]></dc:creator>
            <pubDate>Wed, 18 Oct 2023 18:47:00 GMT</pubDate>
            <atom:updated>2024-06-06T07:55:02.746Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Become a Data Preparation Expert Using ‘dplyr’ in R</strong></h3><p><em>Enhance Your Data Manipulation Workflow With This Comprehensive Guide</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RdBsAf8gT6WMJUacx95INA.png" /></figure><p>Data preparation is the cornerstone of successful data analysis and machine learning. It involves the intricate tasks of cleaning, transforming, and organizing raw data into a format ready for analysis. In the R ecosystem, one tool that is guaranteed to make your life easier is dplyr.</p><h3><strong>The Elegance of dplyr</strong></h3><p>dplyr is a data manipulation package in R. What sets it apart are its elegant and efficient functions designed to streamline data preparation tasks. While it offers a wide range of capabilities, let&#39;s delve into some of the standout features that make using dplyr a no-brainer:</p><pre>install.packages(&#39;dplyr&#39;)<br>library(&#39;dplyr&#39;)</pre><h4>1. Groupby: Powerful Aggregation</h4><p>Data often contains intrinsic categories or groups. For instance, sales data can be grouped by region or product, students’ academic performances can be grouped by districts, schools or teachers, and so on…</p><blockquote>In grouped data, we are commonly interested in <strong>dedicated summary statistics</strong> for these specific groups:</blockquote><ul><li>Which of my products has the highest margin?</li><li>What is the average grade of students taught by a specific teacher?</li><li>Which U.S. state has the lowest levels of air pollution?</li><li>[…]</li></ul><p>These insights require aggregation, that is, grouping data by shared attributes. The dplyr library enables easy aggregation with its group_by() function, allowing you to apply operations to each group independently.</p><pre># Group data by &#39;Category&#39; and calculate average values<br>aggregated_data &lt;- data %&gt;%<br>  group_by(Category) %&gt;%<br>  summarise(avg_value = mean(Value))</pre><p>In the above code snippet, we first group our data by a category and then display the mean of another value for each category. Using the more tangible example of air pollution in U.S. states:</p><pre># Create sample air pollution data with cities from multiple states<br>data &lt;- data.frame(<br>  City = c(&quot;Columbus&quot;, &quot;Cleveland&quot;, &quot;Cincinnati&quot;, <br>           &quot;Toledo&quot;, &quot;Dayton&quot;, &quot;Pittsburgh&quot;, &quot;Erie&quot;, <br>           &quot;Detroit&quot;, &quot;Indianapolis&quot;),<br>  State = c(&quot;Ohio&quot;, &quot;Ohio&quot;, &quot;Ohio&quot;, &quot;Ohio&quot;, <br>            &quot;Ohio&quot;, &quot;Pennsylvania&quot;, &quot;Pennsylvania&quot;, <br>            &quot;Michigan&quot;, &quot;Indiana&quot;),<br>  AirPollution = c(15, 18, 10, 12, 9, 20, 17, 22, 14)<br>)<br><br># Group by &#39;State&#39; and calculate mean of air pollution<br>aggregated_data &lt;- data %&gt;%<br>  group_by(State) %&gt;%<br>  summarise(avg_air_pollution = mean(AirPollution))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/289/1*PwQbrsRV7F_S5--bANomWw.png" /><figcaption>We can see that air pollution is lowest in Ohio — time to move to Ohio</figcaption></figure><h4><strong>2. Piping with </strong><strong>%&gt;%: Streamlining Workflows</strong></h4><p>At the heart of dplyr is the pipe operator %&gt;%. This operator allows you to create smooth, step-by-step data transformation workflows. Instead of nesting functions, you can chain them together, making your code more readable and intuitive.</p><p>Let’s look at a sample DataFrame containing customer ratings of tech products on Amazon and the customer’s ages and gender:</p><pre># Setting a seed for reproducibility<br>set.seed(42)  <br><br># Generate random age ranges<br>age_ranges &lt;- sample(c(&quot;18-24&quot;, &quot;25-34&quot;, &quot;35-44&quot;, &quot;45-54&quot;, &quot;55+&quot;), <br>                       100, replace = TRUE)<br># Generate random ratings on a 5-star scale<br>ratings &lt;- sample(1:5, 100, replace = TRUE)<br><br># Generate random gender <br>gender &lt;- sample(c(&quot;male&quot;, &quot;female&quot;, &quot;other&quot;), 100, replace = TRUE)<br><br># Create a data frame<br>customer_ratings &lt;- data.frame(AgeRange = age_ranges, <br>                               Rating = ratings, <br>                               Gender = gender)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/473/1*_i-EyDaWqfAsf59guaiOrw.png" /></figure><p>Let’s say you’re interested in the <strong>average rating based on the age</strong> of the customer. Your theory is that younger people are more tech-savvy and thus, on average, leave better reviews for tech products on Amazon. However, you want to exclude 5-star ratings as they are often fake.</p><blockquote>Your code might look something like this:</blockquote><pre># Select columns &#39;AgeRange&#39; and &#39;Rating&#39;<br>subset &lt;- customer_ratings[, c(&quot;AgeRange&quot;, &quot;Rating&quot;)]<br><br># Filter the data to remove 5-star ratings<br>filtered_subset &lt;- subset[subset$Rating &lt; 5, ]<br><br># Calculate the mean rating for each age group<br>mean_ratings &lt;- aggregate(Rating ~ AgeRange, <br>                          data = filtered_subset, <br>                          FUN = mean)<br><br># Create a new data frame with the result<br>result &lt;- data.frame(mean_ratings)</pre><blockquote>And it works as intended:</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/304/1*A4NKPmGW_-vB0pDPvK1JyQ.png" /><figcaption>You were right, younger people leave the best reviews.</figcaption></figure><blockquote>Theres a better way though.</blockquote><pre># using dplyr with %&gt;%<br><br>result &lt;- customer_ratings %&gt;%<br>  select(AgeRange, Rating) %&gt;%<br>  filter(Rating &lt; 5) %&gt;%<br>  group_by(AgeRange) %&gt;%<br>  summarise(avg_rating = mean(Rating))</pre><p>Clean, isn’t it? Using the pipe operator %&gt;% from the dplyr library, we can chain all operations together. The pipe operator simplifies your code, making it easier to follow your logic; it enhances readability and the code is more efficient.</p><h4><strong>3. Data Joining: Merging Datasets</strong></h4><p>Data preparation often involves combining different datasets, for instance when you measure user behavior with multiple instruments (e.g., a step-counter and a heart rate monitor). Data from multiple data sources can be joined by shared attributes, such as a user ID. dplyr provides functions to join multiple sources of data, using different types of joins.</p><ul><li><strong>Inner Join</strong>: This type of join returns only the rows that have matching values in both datasets based on the specified key.</li></ul><pre># Inner join two datasets by &#39;ID&#39; <br>merged_data &lt;- inner_join(dataset1, dataset2, by = &quot;ID&quot;)</pre><ul><li><strong>Left Join</strong>: A left join returns all the rows from the left dataset and the matching rows from the right dataset. If there are no matches in the right dataset, it includes NA values for those rows.</li></ul><pre># Left join two datasets by &#39;ID&#39; <br>merged_data &lt;- left_join(dataset1, dataset2, by = &quot;ID&quot;)</pre><ul><li><strong>Right Join</strong>: A right join is similar to a left join, but it returns all the rows from the right dataset and the matching rows from the left dataset.</li></ul><pre># Right join two datasets by &#39;ID&#39; <br>merged_data &lt;- right_join(dataset1, dataset2, by = &quot;ID&quot;)</pre><ul><li><strong>Full Join</strong>: A full join returns all the rows from both datasets, including NA values where there are no matches.</li></ul><pre># Full join two datasets by &#39;ID&#39;<br>merged_data &lt;- full_join(dataset1, dataset2, by = &quot;ID&quot;)</pre><h4>4. Mutate: Create New Variables Easily</h4><p>One of the powerful functions in the dplyr toolkit is mutate. Just like other dplyr functions, mutate fits seamlessly into your data transformation workflows, thanks to the pipe operator %&gt;%. mutate allows you to create new variables or modify existing ones within your dataset.</p><p>Let’s say you have a dataset containing the ages of individuals, and you want to calculate their ages in months:</p><pre># using mutate to create new variable &#39;age_in_months&#39;<br>data &lt;- data %&gt;%<br>  mutate(age_in_months = age * 12)</pre><p>You can also use mutate to modify existing variables in your dataset. For instance, if you have a dataset of product prices, and you want to apply a 10% discount to all prices:</p><pre># using mutate to modify the price variable<br>data &lt;- data %&gt;%<br>  mutate(discounted_price = price * 0.9)</pre><blockquote>If you are interested in a dplyr counterpart in python, check out this story on the python library pandas:</blockquote><p><a href="https://arvideichner.medium.com/why-you-should-start-using-pandas-for-data-preparation-today-6111e13a6195">Why You Should Start Using Pandas for Data Preparation Today</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=103fbbc53c9b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Don‘t Just Remove Outliers From Your Data – Think Twice]]></title>
            <link>https://arvideichner.medium.com/don-t-just-remove-outliers-from-your-data-think-twice-6508b37920e3?source=rss-493985181cd4------2</link>
            <guid isPermaLink="false">https://medium.com/p/6508b37920e3</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[science]]></category>
            <category><![CDATA[outliers]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[lessons-learned]]></category>
            <dc:creator><![CDATA[Arvid Eichner]]></dc:creator>
            <pubDate>Sat, 14 Oct 2023 15:34:27 GMT</pubDate>
            <atom:updated>2024-04-22T14:18:35.387Z</atom:updated>
            <content:encoded><![CDATA[<h4>Rethinking Outliers: Strategies for Informed Data Decisions</h4><p>Outliers can affect the accuracy of statistical analysis and machine learning models. However, not all outliers are created equal. In this guide, we will explore different types of outliers and how to deal wit them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*icnfDx2bOamBLVSbX4D0wQ.png" /><figcaption>Boxplots are a great tool to visually identify outliers. Can you spot them?</figcaption></figure><h3><strong>What Are Outliers?</strong></h3><p>Outliers are data points that are significantly different from other data points in a dataset. They can occur for various reasons, such as measurement errors, data entry mistakes, or genuinely rare events. Identifying and dealing with outliers is crucial as they can severly distort statistical analyses and machine learning models: Due to their exceptionally high or low values, if used as independent variables, outliers disproportionately affect the results – leading to misleading conclusions.</p><h3><strong>Identifying Outliers</strong></h3><p>There are several ways to identify and handle outliers. The most common ones are:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FKbAkMMoVyry2tIcaE0%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2Fv1.Y2lkPTc5MGI3NjExM2VoeDZscmRrbHU2Nzk0cjc3MzlmNTlwZDF1Z2J0NHRtcXM4ZXZnayZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw%2FKbAkMMoVyry2tIcaE0%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FKbAkMMoVyry2tIcaE0%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="183" frameborder="0" scrolling="no"><a href="https://medium.com/media/7f5fa06cf45d899357701472aeeb4121/href">https://medium.com/media/7f5fa06cf45d899357701472aeeb4121/href</a></iframe><ol><li><strong>Visual Inspection</strong>: By far the easiest method to spot outliers. Create box plots, histograms, or scatter plots to visually identify data points that appear far from the main cluster.</li><li><strong>Z-Score Method</strong>: Calculate the so-called z-score for each data point. The z-score measures how many standard deviations a data point is from the mean. Data points with a high absolute z-score (typically greater than 2 or 3) can be considered outliers.</li><li><strong>IQR (Interquartile Range) Method</strong>: Calculate the IQR, which is the range between the 75th and 25th percentiles of your data. Data points outside 1.5 times the IQR can be considered outliers.</li></ol><blockquote>For a step-by-step guide on how to identify and remove outliers using the Python library pandas click <a href="https://medium.com/python-in-plain-english/identifying-and-handling-outliers-in-pandas-a-step-by-step-guide-fcecd5c6cd3b"><strong>here</strong></a>.</blockquote><h3><strong>Handling Outliers</strong></h3><p>But wait! Before you go ahead and remove the data points that appear to be outliers, do yourself a favor and try to answer the following question:</p><p><strong><em>Is the instrument, that was used to collect the data, reliable and valid?</em></strong></p><p>Without going into too much detail, in the context of measurement instruments reliable and valid means:</p><ul><li>Your instrument is <strong>reliable</strong>, when it is consistent and stable, that is, it consistently produces similar results when used multiple times, assuming no actual changes in the underlying construct being measured.</li><li>Your instrument is <strong>valid</strong> when it is accurate – a valid instrument should measure what it is intended to measure.</li></ul><h3>Measurement Errors Cause Outliers</h3><p>Oftentimes, outliers are caused by unreliable or invalid instruments. We call this type of outliers <strong>measurement errors</strong>.<em></em></p><h4>An example:</h4><p>Meet Isabella. Isabella is a researcher in the field of marketing, studying consumer purchase behavior using eye-tracking technology. In her current project, she uses an eye-tracking device to record where on a computer screen participants focus their gaze and for how long. Using this data, she aims to gain insights into their purchase decisions.</p><blockquote>As Isabella reviews her data, she notices something strange — roughly one third of her participants feature oddly high product “gaze times”, almost four times as high as those of remaining participants.</blockquote><p>This extended gaze time would suggest an outstandingly strong interest in the corresponding products, but Isabella can’t seem to explain the substantial difference. After all, there was no intended distinction between participants; all of them underwent the exact same experimental setup.</p><p>Highly sceptical, Isabella re-examines the results, paying particular attention to potential demographic differences between the two groups. Finally, she notices a small but important detail:</p><blockquote>The group of participants with the abnormally high glaze times were all wearing prescription glasses.</blockquote><p>Apparently, the reflective coating of the glasses created glare when hit by light. This glare was then being picked up by the eye-tracking device, causing it to inaccurately register prolonged gaze times on certain products.</p><p>In reality though, these participants may not have been particularly interested in the products, which would, however, have been the fallacious verdict of Isabella’s experiment.</p><h4>Go Ahead And Remove Outliers That Are Measurement Errors</h4><p>If outliers are caused by measurement errors, for instance an unreliable instrument, there is a reasonable case to be made to remove the outliers as they do not represent genuine (reliable) measurements of the underlying construct.</p><blockquote>For a guide on how to remove outliers using pandas click <strong>here</strong>.</blockquote><h3>A Second Type of Oulier</h3><p>If outliers cannot be explained by unreliable or invalid instruments, that is, you can confidently rule out measurement errors, you should, however, think twice about removing them. In this case, they represent <strong>genuine measurements that happen to be out of the ordinary</strong>. There is an ongoing debate about whether such outliers are to be removed, as removing them can lead to:</p><ul><li>The loss of valuable information</li><li>Misrepresentation of the true patterns in the data (Bias)</li><li>Concerns of data manipulation</li><li>Decreased reproducibility of results</li><li>Potential misrepresentation of marginalized or vulnerable populations</li></ul><p>Some very compelling reasons to keep outliers, when they are not caused by measurement errors. Still, many statistical methods, such as linear regression or ANOVA, do require data sets that are free of outliers.</p><blockquote>And now?</blockquote><p>Well, these are your options:</p><ol><li><strong>Choose a different model</strong>: Of course, this is not always an option. But many methods have alternatives, that are less prone to outlier-induced bias, for example non-parametric tests.</li><li><strong>Transformation</strong>: Apply mathematical transformations to the data to reduce the impact of outliers. For example, you can use a square root or log transformation to make the data less skewed.</li><li><strong>Winsorization</strong>: Replace extreme values with the nearest “normal” data point. Winsorizing is less drastic than outright removal but can still reduce the impact of outliers.</li></ol><p>Remember that your approach to handling outliers should be tailored to your data and the objectives of your analysis. Trying to derive insights, that are both meaningful and representative of the underlying reality, requires an approach that makes theoretical sense and fits the context of your research.</p><h3>Key Takeaways:</h3><ol><li><strong>Always check your data for outliers.</strong> They will impact your analyses.</li><li><strong>Think twice! </strong>Before mindlessly removing outliers, consider instrument reliability and validity to rule out potential measurement errors.</li><li><strong>Go ahead and remove outliers</strong> that are probably measurement errors.</li><li><strong>Do not remove remove data points </strong>simply because they are outliers.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6508b37920e3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Real-Life Cheat Code That Makes Self-Control Effortless]]></title>
            <link>https://medium.com/dare-to-be-better/the-real-life-cheat-code-that-makes-self-control-effortless-f559f6ee5e6f?source=rss-493985181cd4------2</link>
            <guid isPermaLink="false">https://medium.com/p/f559f6ee5e6f</guid>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[life]]></category>
            <category><![CDATA[science]]></category>
            <category><![CDATA[health]]></category>
            <category><![CDATA[self-improvement]]></category>
            <dc:creator><![CDATA[Arvid Eichner]]></dc:creator>
            <pubDate>Tue, 10 Oct 2023 14:43:47 GMT</pubDate>
            <atom:updated>2023-11-07T06:17:32.284Z</atom:updated>
            <content:encoded><![CDATA[<p>It is scientifically proven that reading this story will improve your life.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/1*gKbMmwuRz2JKAp_AzKKIDg.png" /><figcaption>Image created by <a href="https://labs.openai.com/">DALL·E</a></figcaption></figure><p>Have you ever looked at someone effortlessly juggling work, a healthy lifestyle, and regular exercise and wondered how they do it? It seems like they have some secret power, an innate ability to make it all look easy. The key to their success might just be what psychologists call “effortless self-control.”</p><h4><strong>The Traditional View of Self-Control</strong></h4><p>Traditionally, self-control has been seen as a <a href="https://journals.sagepub.com/doi/full/10.1177/1745691617716946">resource</a>. Like a rechargeable battery, it depletes with each exertion, making it harder to resist temptations as the day progresses. We’ve all been there: After a long and exhausting day at work, the temptation to order a pizza may feel irresistible, even if you’re committed to a healthier diet.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FaZupxNI7bBNS0%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FaZupxNI7bBNS0%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FaZupxNI7bBNS0%2F200.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="181" frameborder="0" scrolling="no"><a href="https://medium.com/media/7ae4aa93b1813e93528401c6d0acdbb3/href">https://medium.com/media/7ae4aa93b1813e93528401c6d0acdbb3/href</a></iframe><p>‘So all these people, who work long hours and still have the motivation to work out at the end of their, they’re just better? have larger ‘batteries?’ you might ask.</p><p>Well, not really. What if I told you that these individuals use <strong>real-life cheat codes</strong>? For them, self-controlling might even be easy — ‘effortless’.</p><blockquote>Recent <a href="https://compass.onlinelibrary.wiley.com/doi/full/10.1111/spc3.12160">research</a> suggests a paradigm shift in how we understand self-control. Rather than viewing it as a depletable resource, experts are now emphasizing the importance of strategies that prevent conflicting responses.</blockquote><p>Sounds fancy, but what exactly does that mean?</p><h4><strong>Effortless Self-Control: A Paradigm Shift</strong></h4><p>Some people have developed strategies that prevent them from being in situations, where self-control is necessary (so-called response conflicts). Let me give you an example:</p><p>Meet Dylan and Emma. Both are trying to maintain a healthy diet by avoiding sweets for an entire month. Dylan, unaware of effortless self-control, relies solely on willpower. Every time he’s at the convenience store to buy groceries, he stands there and stares at the wall of sweets, his eyes shining bright as he thinks about how good they taste. That’s him:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FFjHxlAfONvpjCiraRY%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FFjHxlAfONvpjCiraRY%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FFjHxlAfONvpjCiraRY%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/fb703ac404be6652f6099b8adb1b7de6/href">https://medium.com/media/fb703ac404be6652f6099b8adb1b7de6/href</a></iframe><p>Eventually, his conscience kicks in, reminding him of his goal and why he chose to avoid sweets (Dylan wants to lose some weight). Winning his internal struggle, Dylan moves on and finishes his errands.</p><p>Now let’s see how Emma is doing: Unlike Dylan, Emma uses effortless self-control. She isn’t even aware of it; she just does it. Emma also has to get groceries, and she even goes to the same store. But she doesn’t feel any struggle and still refrains from buying any sweets. How did she do it?</p><blockquote><strong>Emma developed the habit to avoid the aisle with the sweets.</strong></blockquote><p>She didn’t even have to think about it — its effortless. By staying clear of the sweets-aisle, she avoids the internal battle between her desire for short-term pleasure and long-term health — a battle Dylan faces every day.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FL404FRbWUfPLS4Of0q%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2Fv1.Y2lkPTc5MGI3NjExcDd4MGd4eXM3b2w4ZThtZGJ0MHVmY20xdHYxcXNqMWJsNWd2cHowNCZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw%2FL404FRbWUfPLS4Of0q%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FL404FRbWUfPLS4Of0q%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/784381b83ba281be322a1ba3cd688240/href">https://medium.com/media/784381b83ba281be322a1ba3cd688240/href</a></iframe><p>In the end, Emma succeeded in not eating any sweets. Dylan managed to withstand the urge for a few days — fighting his inner demon every time — but eventually gave in, losing the battle.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FfsE2DJwKiSpnyccqrg%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2Fv1.Y2lkPTc5MGI3NjExMnRwcTBlZzM1ZmZhMWFqcTN4amlkbW5xMWhkeXA0ODR2dHBwODdnMSZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw%2FfsE2DJwKiSpnyccqrg%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FfsE2DJwKiSpnyccqrg%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/81cf72526e8837b08935e75838004b54/href">https://medium.com/media/81cf72526e8837b08935e75838004b54/href</a></iframe><blockquote>The End</blockquote><p>At this point, you might be asking yourself: ‘Are those of us who are like Dylan, who don’t <em>*magically</em>* come up with these strategies, simply out of luck?‘</p><p>Not at all, <strong>there’s hope</strong>!</p><h4><strong>Mastering Effortless Self-Control</strong></h4><p>The cool thing about these strategies is that everyone can learn to apply them. In a study by Duckworth et al. (<a href="https://psycnet.apa.org/record/2016-15978-003">2016</a>), one group of high-school students received an intervention, while the other group did not. The intervention included:</p><ol><li>A short lesson about the efficacy of effortless self-control strategies</li><li>Examples of strategies</li><li>A prompt to modify one’s environment to minimize temptations instead of trying to resist them</li></ol><blockquote><strong>Within only one week</strong>, students who received the intervention, reported significantly higher progress towards their study goals than students who did not receive the intervention.</blockquote><p>Think about it: By simply having read this story you just received the same intervention. You just improved your life — it’s science!</p><p>If you’re struggling with reducing screen time rather than resisting sweets, I have additional advice for you here:</p><p><a href="https://arvideichner.medium.com/is-your-smartphone-working-for-or-against-you-heres-how-to-find-out-3dd360e9d3c1">Is your Smartphone working for or against you? Here’s how to find out.</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f559f6ee5e6f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/dare-to-be-better/the-real-life-cheat-code-that-makes-self-control-effortless-f559f6ee5e6f">The Real-Life Cheat Code That Makes Self-Control Effortless</a> was originally published in <a href="https://medium.com/dare-to-be-better">Dare To Be Better</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Is your Smartphone working for or against you? Here’s how to find out.]]></title>
            <link>https://arvideichner.medium.com/is-your-smartphone-working-for-or-against-you-heres-how-to-find-out-3dd360e9d3c1?source=rss-493985181cd4------2</link>
            <guid isPermaLink="false">https://medium.com/p/3dd360e9d3c1</guid>
            <category><![CDATA[self-improvement]]></category>
            <category><![CDATA[science]]></category>
            <category><![CDATA[mindfulness]]></category>
            <category><![CDATA[smartphones]]></category>
            <category><![CDATA[habits]]></category>
            <dc:creator><![CDATA[Arvid Eichner]]></dc:creator>
            <pubDate>Mon, 09 Oct 2023 12:11:18 GMT</pubDate>
            <atom:updated>2023-10-10T09:35:32.249Z</atom:updated>
            <content:encoded><![CDATA[<h3>Is Your Smartphone Working for or Against You? Here’s How to Find Out.</h3><blockquote><strong>‘You’re always on your damn phone.’</strong></blockquote><p>Chances are, you’ve heard this comment from someone in your life, and perhaps you’ve even told it to yourself. It might have been that one time when you realized you just spent hours scrolling through Instagram instead of studying for an important exam.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DkMKKSa_ZQx8-4m4PLDt3A.png" /><figcaption>Image created by <a href="https://labs.openai.com/">DALL·E</a></figcaption></figure><p>Maybe you even thought about finally deleting the app. Surely, that will solve your problems and you will finally be able to study without distractions.</p><p>‘<em>That can’t be the solution</em>’, I hear you say. And you’re right. Without Instagram, how would you keep up with the lives of your college friends? Their regular story posts are like virtual updates on the chapters of their lives, and you really want to stay in the loop.</p><h3><strong>Telling Good from Bad</strong></h3><p>As it turns out, distinguishing good from bad smartphone habits is more complicated than it seems. Whether habits are good or bad, and to what extent, depends on individual goals and values as well as the situations in which they are performed. Sounds intuitive right? Hear me out though:</p><p>According to <a href="https://misq.umn.edu/constant-checking-is-not-addiction-a-grounded-theory-of-it-mediated-state-tracking.html">research</a>, the consequences of your smartphone habits can be described using two dimensions:</p><ul><li><strong>Problems of Attention:</strong> This measures the extent to which your constant smartphone checking distracts you from your daily tasks. If you regularly use an app in situations where your attention is required, there is a problem of attention.</li></ul><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2F21viT0JqpwnErlzuAl%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2F21viT0JqpwnErlzuAl%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2F21viT0JqpwnErlzuAl%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="310" frameborder="0" scrolling="no"><a href="https://medium.com/media/984534ad6699b3e96610abb0ea640c25/href">https://medium.com/media/984534ad6699b3e96610abb0ea640c25/href</a></iframe><ul><li><strong>Service to Underlying Goals:</strong> This factor considers whether your smartphone use aligns with your long-term objectives and values. If your smartphone habits contribute positively to your goals and aspirations, it’s likely working for you.</li></ul><h3><strong>Ask Yourself These 5 Questions:</strong></h3><p>How can you use this knowledge to find out if your smartphone is working for or against you? It’s simple — for every app that you cannot live without, <strong>ask yourself five questions</strong>. First, to find out if the app causes problems of attentions, ask yourself:</p><ol><li>Do you frequently find yourself using the app when you should be <strong>studying or working</strong>?</li><li>Have you <strong>physically harmed</strong> you or others or caused material damage due to being distracted by the app?</li><li>Have important <strong>people in your life criticized</strong> you for using the app when they expected your undivided attention?</li></ol><p>Next, to find out if your use of the app is aligned with your long-term goals, take a second to reflect on the goal that initially motivated your use of the app. For example, you might have developed the habit of checking the news app first thing in the morning, motivated by the goal to stay informed about current events. Keeping this underlying goal in mind, ask yourself:</p><ol><li>Do you obtain information that is <strong>relevant to your underlying goal</strong> by using the app?</li><li>Do you obtain information that is <strong>important</strong>?</li></ol><p>By honestly answering these questions, you can gain valuable insights into your smartphone habits and their impact on your life. If you find that your answers indicate a healthy balance between smartphone use and your overall well-being, congratulations, your smartphone is likely working for you. However, if your answers suggest that your smartphone is causing problems of attention and hindering your goals by providing no value, it might be time to reassess and make some changes.</p><blockquote>These five questions are based on more detailed measures developed in collaboration with my co-authors in a recently published <a href="https://www.researchgate.net/publication/371417015_Dangerous_Habit_or_Useful_Routine_Developing_Theory-Based_Measures_for_the_Intended_and_Unintended_Consequences_of_State-Tracking_Habits">paper</a>.</blockquote><h3><strong>Applying the Questions: A Real-Life Example</strong></h3><p>I spend a lot of time on Instagram. I scroll through my feed mindlessly for hours at a time. But I only do it when I’m trying to pass time, when I’m bored at home or on a long train ride by myself. Whenever I am with my friends or driving a car, I leave my smartphone in my pocket. Therefore, for me personally, Instagram does not produce a problem of attention.</p><p>It feels useless though. When I first downloaded Instagram all the way back in 2012, I did so because I wanted to see pictures of my friends, their pets and perhaps the food they ate (<em>debatable</em>). However, today, I mostly watch Reels — The majority of the content I consume is short-form videos, which are supposed to entertain, infused with a good amount of ads. Evidently, considering my underlying goal, the way I use Instagram today, it provides little to no value.</p><blockquote><a href="https://www.jstor.org/stable/43825944">Research </a>has shown that individuals are generally not aware of the consequences of their smartphone habits. So, by answering these five questions, you are one step closer to controlling them.</blockquote><p>Remember, your smartphone is a tool, and its impact on your life depends on how you use it. Taking a mindful approach to your smartphone habits can help you harness its potential for good while minimizing its drawbacks. Good luck!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3dd360e9d3c1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mastering Data Aggregation with Pandas GroupBy: An Easy-to-Understand Guide]]></title>
            <link>https://python.plainenglish.io/mastering-data-aggregation-with-pandas-groupby-an-easy-to-understand-guide-0622a542ccdd?source=rss-493985181cd4------2</link>
            <guid isPermaLink="false">https://medium.com/p/0622a542ccdd</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[group-by]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Arvid Eichner]]></dc:creator>
            <pubDate>Mon, 09 Oct 2023 10:17:24 GMT</pubDate>
            <atom:updated>2023-10-09T14:28:27.434Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P89Qu_9e9YEJHz2pUAQvKg.png" /></figure><p>In the world of data analysis, extracting meaningful insights often hinges on one crucial skill: data aggregation. Fortunately, <strong>pandas</strong>, the Python library for data manipulation, equips data enthusiasts with a powerful tool — the <a href="https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.groupby.html">groupby</a> method. Whether you’re new to data aggregation or need a handy reference, this guide has you covered.</p><h3>Understanding the GroupBy Concept</h3><p>At its core, grouping data is like gathering similar items together. Imagine you have a big box of colorful marbles, and you want to group them based on their colors. You pick all the red marbles and put them in one pile, all the blue ones in another, and so on.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FKAJpM1hdmNuMJ27X25%2Ftwitter%2Fiframe&amp;display_name=Giphy&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2Fv1.Y2lkPTc5MGI3NjExZXA0Y3lqemh1M2I4NjZkZjU3aXBpN25ia20yY2wza3luZHJlbjY1ZyZlcD12MV9pbnRlcm5hbF9naWZfYnlfaWQmY3Q9Zw%2FKAJpM1hdmNuMJ27X25%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FKAJpM1hdmNuMJ27X25%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="244" frameborder="0" scrolling="no"><a href="https://medium.com/media/5a413083ec2e9eaee9c74d3188d36699/href">https://medium.com/media/5a413083ec2e9eaee9c74d3188d36699/href</a></iframe><p>In data analysis, it’s a bit like that. You have a bunch of information, and you want to group it based on common characteristics or categories. The pandas groupby method allows you to group data within a DataFrame by one or more columns.</p><p>Once grouped, you can apply various aggregation functions to calculate summary statistics, derive insights, or perform calculations for each group.</p><p><strong>Let’s look at an example:</strong></p><p>The following DataFrame contains e-commerce transactions with categorical variables such as ‘Product_Category’, ‘Payment_Method’, ‘Customer_Type’ (new or returning customer), ‘Location’, and ‘Total_Amount ($)’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3ZtGm4a1jtTjFayYyvT7Gg.png" /></figure><p>Let’s say you need to find out whether customers spend more when they pay their order using PayPal vs. Credit Card.<em> </em>Grouping by Payment_Method and displaying the mean Total_Amount provides the answer:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/702/1*ZVD72iZQTcVsK3y0LsqfKA.png" /><figcaption>Apparently people spend more when they pay by credit card 💸</figcaption></figure><p>To understand how we got there, lets explore how groupby works in pandas. You can group by a single column, as in the above example (Payment_Method), or even use multiple columns to group. Let’s start with a single column:</p><h3>Grouping by a Single Column</h3><p>To group data by a single column, simply use the .groupby() method followed by the name of the column that should be used for grouping:</p><pre>import pandas as pd<br><br># Creating a sample DataFrame<br>data = {&#39;Category&#39;: [&#39;A&#39;,&#39;B&#39;,&#39;A&#39;,&#39;B&#39;,&#39;A&#39;],<br>        &#39;Value&#39;:    [10, 20, 15, 25, 30]}<br>df = pd.DataFrame(data)<br><br># Grouping by &#39;Category&#39; column<br>grouped = df.groupby(&#39;Category&#39;)</pre><p>Congrats! You’ve just created a grouped object. But what can you do with it? Plenty!</p><blockquote>Note: The results of the groupby function are groupby objects not DataFrames, even though they might look like DataFrames when printing them. So using your trusted DataFrame methods, for example, accessing cell contents via <em>.iloc[]</em> will not work.</blockquote><h3>Applying Aggregation Functions</h3><p>Now that you’ve grouped your data, it’s time to apply aggregation functions like sum, mean, min, max, and size:</p><pre># Calculate the sum for each group<br>sum_values = grouped[&#39;Value&#39;].sum()<br><br># Calculate the mean for each group<br>mean_values = grouped[&#39;Value&#39;].mean()<br><br># Find the minimum and maximum values for each group<br>min_max_values = grouped[&#39;Value&#39;].agg([&#39;min&#39;, &#39;max&#39;])<br><br># Calculate the size (number of elements) in each group<br>size_values = grouped[&#39;Value&#39;].size()</pre><p>These commands provide quick summaries for each group within the ‘Category’ column.</p><blockquote>Note: If you are looking to step up your pandas game, you can also group and aggregate at the same time using only one line of code:</blockquote><pre># get the summed value for each category <br>grouped = df.groupby(&#39;Category&#39;)[&#39;Value&#39;].sum()</pre><h3>Grouping by Multiple Columns</h3><p>For more advanced needs, try grouping your data using multiple columns:</p><pre># Group by &#39;Category&#39; and &#39;Subcategory&#39; columns<br>multi_grouped = df.groupby([&#39;Category&#39;, &#39;Subcategory&#39;])</pre><p>This allows you to analyze data at a more granular level, considering multiple attributes simultaneously.</p><p><strong>Let’s look at another practical example:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZKEg-KLJXT0choxw7WRUOg.png" /><figcaption>DataFrame containing departments, experience and performance scores of employees</figcaption></figure><p>We can use groupby using two categories to find out the average performance score of employees conditional on their experience level <strong>and</strong> department:</p><pre># group by department &amp; experience level then aggregate performance score using mean() <br>perf_by_dep_exp = df.groupby([&#39;Department&#39;, &#39;Experience_Level&#39;])[&#39;Performance_Score&#39;].mean()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/667/1*QonmysmARZ765bnxNE0U3g.png" /><figcaption>The resulting grouped and mean-aggregated data</figcaption></figure><h3>Resetting the Index</h3><p>After performing data aggregation with groupby, you might notice that the resulting object looks kind of like a weird version of a DataFrame. By default, pandas sets the group labels as the new index. While this can be useful for some operations, it may not always be desirable, especially if you want to work with the data in a more conventional tabular format. This is where resetting the index comes into play.</p><h4>Why Reset the Index?</h4><ol><li><strong>Enhanced Readability</strong>: Resetting the index transforms the grouped data back into a more straightforward DataFrame structure. It replaces the group labels with a default integer index, making it easier to work with and interpret the data.</li><li><strong>Facilitating Further Analysis</strong>: When you reset the index, you ensure that the grouped data can be seamlessly integrated into subsequent analyses. Many data analysis functions expect data in a tabular format with a consistent integer index.</li></ol><h4>What Does Resetting the Index Do?</h4><p>When you apply the .reset_index() method to a grouped DataFrame, it resets the index and moves the current index labels back to columns. Here&#39;s how it works using our practical example from above:</p><pre># Reset the index of the groupby object<br>perf_by_dep_exp_reset = perf_by_dep_exp.reset_index()</pre><p>After executing this code, grouped data will again have a default integer index, and the &#39;Category&#39; column will contain the original &#39;Category&#39; labels. This transformation ensures that the grouped data remains readily usable for further exploration or export. Maybe more importantly, its a DataFrame again and looks familiar:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/740/1*nA8xJBixEkta-PxsHedfQA.png" /></figure><p>Remember, it’s not just about understanding the data — it’s about presenting it in a way that others can grasp and appreciate. With these groupby principles, you’re well on your way to becoming a data storyteller.</p><p>If you are new to pandas, you should probably check out my other article on the basics of using pandas for data preparation:</p><p><a href="https://medium.com/@eichner.arvid/why-you-should-start-using-pandas-for-data-preparation-today-6111e13a6195">Why You Should Start Using Pandas for Data Preparation Today</a></p><h3>In Plain English</h3><p><em>Thank you for being a part of our community! Before you go:</em></p><ul><li><em>Be sure to </em><strong><em>clap</em></strong><em> and </em><strong><em>follow</em></strong><em> the writer! 👏</em></li><li><em>You can find even more content at </em><a href="https://plainenglish.io/"><strong><em>PlainEnglish.io</em></strong></a><strong><em> 🚀</em></strong></li><li><em>Sign up for our </em><a href="http://newsletter.plainenglish.io/"><strong><em>free weekly newsletter</em></strong></a><em>. 🗞️</em></li><li><em>Follow us on </em><a href="https://twitter.com/inPlainEngHQ"><strong><em>Twitter</em></strong></a><strong><em>(X</em></strong>), <a href="https://www.linkedin.com/company/inplainenglish/"><strong><em>LinkedIn</em></strong></a>, <a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong><em>YouTube</em></strong></a>, and <a href="https://discord.gg/in-plain-english-709094664682340443"><strong><em>Discord</em></strong></a><strong><em>.</em></strong></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0622a542ccdd" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/mastering-data-aggregation-with-pandas-groupby-an-easy-to-understand-guide-0622a542ccdd">Mastering Data Aggregation with Pandas GroupBy: An Easy-to-Understand Guide</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>